Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................................X.] duration: 1.2768246s trace: https://jaeger.dev.storj.io/trace/252dcb40ec352e08 FAIL (2026-03-08 06:31:50 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: encryption: context deadline exceeded ======LOGS====== Mar 8 06:30:49.463071826 (43h30m47.059842858s) [upload=48697]: Starting upload Mar 8 06:30:49.463177095 (43h30m47.059948131s) [upload=48697]: Done starting upload Mar 8 06:30:49.463389200 (43h30m47.060160215s) [upload=48697]: Uploading segments... Mar 8 06:30:49.463684149 (43h30m47.060455181s) [upload=48697]: Got next segment. Inline: false Mar 8 06:30:49.463736570 (43h30m47.060507598s) [upload=48697]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 8 06:30:49.463745131 (43h30m47.060516155s) [upload=48697]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 8 06:31:49.431270256 (43h31m47.028041272s) [upload=48697]: Next segment err: context deadline exceeded Mar 8 06:31:49.431553505 (43h31m47.028324519s) [upload=48697]: Done uploading segments... Mar 8 06:31:49.431606188 (43h31m47.028377206s) [upload=48697]: Upload finished. err: context deadline exceeded Duration: 1.276861191s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................................X] duration: 1m0.003653013s trace: https://jaeger.dev.storj.io/trace/252dcb40ec352e08 FAIL (2026-03-08 06:31:49 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: encryption: context deadline exceeded ======LOGS====== Mar 8 06:30:49.463071826 (43h30m47.059842858s) [upload=48697]: Starting upload Mar 8 06:30:49.463177095 (43h30m47.059948131s) [upload=48697]: Done starting upload Mar 8 06:30:49.463389200 (43h30m47.060160215s) [upload=48697]: Uploading segments... Mar 8 06:30:49.463684149 (43h30m47.060455181s) [upload=48697]: Got next segment. Inline: false Mar 8 06:30:49.463736570 (43h30m47.060507598s) [upload=48697]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 8 06:30:49.463745131 (43h30m47.060516155s) [upload=48697]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 8 06:31:49.431270256 (43h31m47.028041272s) [upload=48697]: Next segment err: context deadline exceeded Mar 8 06:31:49.431553505 (43h31m47.028324519s) [upload=48697]: Done uploading segments... Mar 8 06:31:49.431606188 (43h31m47.028377206s) [upload=48697]: Upload finished. err: context deadline exceeded Duration: 1m0.003673847s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline) antiblip: 1 out of 60 is failed (threshold=3) [..........................................................X.] duration: 249.31343ms trace: https://jaeger.dev.storj.io/trace/71b24bccf9da3041 FAIL (2026-03-08 06:31:34 UTC) orbiter warning: operation: ListObjects ======ERROR====== uplink: metainfo: metaclient: context deadline exceeded ======LOGS====== Mar 8 06:30:34.589783738 (43h30m32.18655477s) [upload=48689]: Starting upload Mar 8 06:30:34.589887627 (43h30m32.186658665s) [upload=48689]: Done starting upload Mar 8 06:30:34.590008691 (43h30m32.186779719s) [upload=48689]: Uploading segments... Mar 8 06:30:34.590094310 (43h30m32.186865342s) [upload=48689]: Got next segment. Inline: true Mar 8 06:30:34.590102615 (43h30m32.18687365s) [upload=48689]: Waiting for error group managing segments... Mar 8 06:30:34.590170261 (43h30m32.186941294s) [upload=48689]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 8 06:30:34.590177637 (43h30m32.186948664s) [upload=48689]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Mar 8 06:30:34.590183313 (43h30m32.18695434s) [upload=48689]: Flush batch item: pb.BatchRequestItem_ObjectCommit Mar 8 06:30:34.641187938 (43h30m32.237958994s) [upload=48689]: Done uploading segments... Mar 8 06:30:34.641211607 (43h30m32.237982637s) [upload=48689]: Upload finished. err: <nil> Duration:* 249.334321ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [..........................................................X.] duration: 448.22441ms trace: https://jaeger.dev.storj.io/trace/6241d904c479a519 FAIL (2026-03-08 06:30:58 UTC) orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded Duration: 448.244931ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [...........................................................X] duration: 5.001196872s trace: https://jaeger.dev.storj.io/trace/6241d904c479a519 FAIL (2026-03-08 06:30:13 UTC) orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded Duration: 5.001235949s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [X...........................................................] duration: 748.734321ms trace: https://jaeger.dev.storj.io/trace/78791d41d75e16a0 FAIL (2026-03-06 20:33:52 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 19:52:18.382132233 (8h52m15.978903242s) [upload=9902]: Uploading segments... Mar 6 19:52:18.382227489 (8h52m15.978998515s) [upload=9902]: Got next segment. Inline: false Mar 6 19:52:18.382313252 (8h52m15.979084279s) [upload=9902]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 6 19:52:18.382319269 (8h52m15.979090296s) [upload=9902]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 6 19:52:18.411471452 (8h52m16.008242482s) [seg_pos={0 0}, upload=9902]: Begin upload segment... Mar 6 19:52:18.411537096 (8h52m16.008308109s) [seg_pos={0 0}, upload=9902]: Done begin upload segment. Mar 6 19:52:18.411564540 (8h52m16.008335586s) [noise=true , address=45.91.116.7:4037 , piece_id=YXDMXUBE7CEWLJZP6FPDCRVIDGG5CJ3Y3IKEKO7EE2DQUSG6W5RA, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.411626862 (8h52m16.008397897s) [noise=true , address=193.149.164.67:4018 , piece_id=GTQQP355TZZF4W7PPBYZS6ZQ6WYK7I3QIE4DZC7STWAKSBAMT4BQ, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.412356605 (8h52m16.009127631s) [noise=true , address=167.17.72.70:4771 , piece_id=TDLPTUHPNXCICPPXRTRL7WCJSXPX22ONIWWPHNXFUNIKRYPYW52Q, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.412900389 (8h52m16.009671415s) [noise=true , address=145.79.227.39:4022 , piece_id=U5GX3GLVWGWLU2BDWTHM7ST3MLLJDEAERWU3UEHO7Z2RXHGZU2DA, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.413467061 (8h52m16.010238103s) [noise=true , address=45.91.116.11:4133 , piece_id=RZWV5JMLOJ36AH7KSDXVTL7CMHI4INEW6YOOQVMCIIC2NGS3XYJA, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.413653431 (8h52m16.010424457s) [noise=true , address=193.149.164.69:4101 , piece_id=BTIBVPSORFJQR3CLA5UV62NXNI6GD3JF4D5Z74KU4R4CXB3WHHKA, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.414227965 (8h52m16.010998999s) [noise=true , address=45.91.116.14:4199 , piece_id=EKGDG7LYV3O5NSDL3UPWFKXINYNAOSPYYOWQPTFN326AJ2ZXAXHQ, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.415177146 (8h52m16.01194818s) [noise=true , address=77.90.31.16:4763 , piece_id=5Q7H3CU3MWSFPQGC67XMLCMTSUIEJQUV7QTO4UZLZY4MIVM76Z6Q, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.415227889 (8h52m16.011998912s) [noise=true , address=45.91.116.10:4096 , piece_id=EIKWFKPE5BYSZGV5T4RPQED55PG2FXU3ZKSNRVCBD363YQDHS4BQ, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.415728547 (8h52m16.012499581s) [noise=true , address=77.90.31.14:4693 , piece_id=S7NAS2ZBHAQPZRHBRGRSZPLE6COQEEN6SZVMB7X2X6IMXTN5HCBQ, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.415910147 (8h52m16.012681174s) [noise=true , address=193.149.164.70:4129 , piece_id=V7WMI7RDYDOAEQYWF2OTZL5LRPUA2S3V5WZN4BKDY6BI4YV3GRPQ, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.416592462 (8h52m16.01336349s) [noise=true , address=77.90.31.18:4831 , piece_id=4PMIUNE4YXKLIJPML3VSUN4PF32TSLPOOSDZZNAXFWH7WICBOW3Q, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.417118897 (8h52m16.01388993s) [noise=true , address=77.90.31.13:4655 , piece_id=47Q27QI7O2IYW7YQKFMJGGMWPVZWWKQTFVYPUMPUI7ORE7236XIQ, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.417185844 (8h52m16.013956873s) [noise=true , address=193.149.164.71:4304 , piece_id=DTCL54GM76GWKAE7A3X5PUNFYO7H5JIKRT5WBEHYEDF2YNFYTOAA, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.417686790 (8h52m16.014457823s) [noise=true , address=167.17.72.71:4810 , piece_id=ANLHBMENEXSGTHBHLOIHMLM6JGHKIU2X6FPQQIUIVMVUOABNK73Q, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.418236951 (8h52m16.015007982s) [noise=true , address=145.79.227.17:4994 , piece_id=TO3EOAILO4PHF6Z7WQA4RWZNNL3JO6EXXEF2RLW7IHAGWUU2TF4Q, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.419293640 (8h52m16.016064672s) [noise=true , address=64.49.13.139:4479 , piece_id=XUYHOCSBATYECYHWLVZ4VBCMT3H4O5ETSQ5HZOB42OR3OY4LWJ6Q, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.420005502 (8h52m16.016776534s) [noise=true , address=45.91.116.8:4066 , piece_id=5MZHIJ3ELVBAJ7ABRKG565TLU4A4NKB7X6WAACY2JBSLACL5LGVQ, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.483654433 (8h52m16.080425441s) [upload=9902]: Next returned nil segment Mar 6 19:52:18.483671157 (8h52m16.08044219s) [upload=9902]: Waiting for error group managing segments... Mar 6 19:52:18.488078321 (8h52m16.084849398s) [noise=true , address=77.90.31.13:4655 , piece_id=47Q27QI7O2IYW7YQKFMJGGMWPVZWWKQTFVYPUMPUI7ORE7236XIQ, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:52:18.498435619 (8h52m16.095206647s) [noise=true , address=145.79.227.17:4994 , piece_id=TO3EOAILO4PHF6Z7WQA4RWZNNL3JO6EXXEF2RLW7IHAGWUU2TF4Q, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:52:18.502979814 (8h52m16.099750841s) [noise=true , address=77.90.31.16:4763 , piece_id=5Q7H3CU3MWSFPQGC67XMLCMTSUIEJQUV7QTO4UZLZY4MIVM76Z6Q, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:52:18.506259140 (8h52m16.103030166s) [noise=true , address=77.90.31.14:4693 , piece_id=S7NAS2ZBHAQPZRHBRGRSZPLE6COQEEN6SZVMB7X2X6IMXTN5HCBQ, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:52:18.514529389 (8h52m16.111300413s) [noise=true , address=145.79.227.39:4022 , piece_id=U5GX3GLVWGWLU2BDWTHM7ST3MLLJDEAERWU3UEHO7Z2RXHGZU2DA, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:52:18.785925494 (8h52m16.382696517s) [noise=true , address=167.17.72.70:4771 , piece_id=TDLPTUHPNXCICPPXRTRL7WCJSXPX22ONIWWPHNXFUNIKRYPYW52Q, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:52:18.810774903 (8h52m16.407545926s) [noise=true , address=77.90.31.18:4831 , piece_id=4PMIUNE4YXKLIJPML3VSUN4PF32TSLPOOSDZZNAXFWH7WICBOW3Q, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:52:19.002746550 (8h52m16.5995176s) [noise=true , address=193.149.164.71:4304 , piece_id=DTCL54GM76GWKAE7A3X5PUNFYO7H5JIKRT5WBEHYEDF2YNFYTOAA, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:52:19.007614901 (8h52m16.604385924s) [noise=true , address=193.149.164.67:4018 , piece_id=GTQQP355TZZF4W7PPBYZS6ZQ6WYK7I3QIE4DZC7STWAKSBAMT4BQ, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:52:19.010443151 (8h52m16.607214176s) [noise=true , address=193.149.164.69:4101 , piece_id=BTIBVPSORFJQR3CLA5UV62NXNI6GD3JF4D5Z74KU4R4CXB3WHHKA, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:52:19.012614014 (8h52m16.609385038s) [noise=true , address=193.149.164.70:4129 , piece_id=V7WMI7RDYDOAEQYWF2OTZL5LRPUA2S3V5WZN4BKDY6BI4YV3GRPQ, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:52:19.042145380 (8h52m16.638916406s) [noise=true , address=64.49.13.139:4479 , piece_id=XUYHOCSBATYECYHWLVZ4VBCMT3H4O5ETSQ5HZOB42OR3OY4LWJ6Q, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:52:21.314990180 (8h52m18.911761213s) [noise=true , address=167.17.72.71:4810 , piece_id=ANLHBMENEXSGTHBHLOIHMLM6JGHKIU2X6FPQQIUIVMVUOABNK73Q, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:53:18.344002671 (8h53m15.9407737s) [noise=true , address=45.91.116.10:4096 , piece_id=EIKWFKPE5BYSZGV5T4RPQED55PG2FXU3ZKSNRVCBD363YQDHS4BQ, seg_pos={0 0}, upload=9902]: Done uploading piece. err: ecclient: upload failed (node:12MZX1GwZGrEi2q44dPiYrwMoum5A2caZtXTS3qiLNcoyHhon7p, address:45.91.116.10:4096): protocol: write tcp 159.223.143.39:44996->45.91.116.10:4096: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 6 19:53:18.344173472 (8h53m15.940944497s) [noise=true , address=45.91.116.7:4037 , piece_id=YXDMXUBE7CEWLJZP6FPDCRVIDGG5CJ3Y3IKEKO7EE2DQUSG6W5RA, seg_pos={0 0}, upload=9902]: Done uploading piece. err: ecclient: upload failed (node:1iNPwE2XSfXcaXAXcrH2jcyNGYeDiiWvzAcDQffrmjPwTVqVmK, address:45.91.116.7:4037): protocol: write tcp 159.223.143.39:52854->45.91.116.7:4037: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 6 19:53:18.344535130 (8h53m15.941306156s) [noise=true , address=45.91.116.14:4199 , piece_id=EKGDG7LYV3O5NSDL3UPWFKXINYNAOSPYYOWQPTFN326AJ2ZXAXHQ, seg_pos={0 0}, upload=9902]: Done uploading piece. err: ecclient: upload failed (node:1d4B1KMLs2annUuJ3v81FJapS9Yg4omBTj48eWoi6HJxnSEjrf, address:45.91.116.14:4199): protocol: write tcp 159.223.143.39:57596->45.91.116.14:4199: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 6 19:53:18.344687364 (8h53m15.941458387s) [noise=true , address=45.91.116.11:4133 , piece_id=RZWV5JMLOJ36AH7KSDXVTL7CMHI4INEW6YOOQVMCIIC2NGS3XYJA, seg_pos={0 0}, upload=9902]: Done uploading piece. err: ecclient: upload failed (node:12e1wX7hiER8sX4MLm5qLue4mayiN4SxXFdZSxfymRaRZxRGU6E, address:45.91.116.11:4133): protocol: write tcp 159.223.143.39:56440->45.91.116.11:4133: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 6 19:53:18.345220236 (8h53m15.941991259s) [noise=true , address=45.91.116.8:4066 , piece_id=5MZHIJ3ELVBAJ7ABRKG565TLU4A4NKB7X6WAACY2JBSLACL5LGVQ, seg_pos={0 0}, upload=9902]: Done uploading piece. err: ecclient: upload failed (node:123uv49k8LraXJfYQg6KRQthrYs1Egc4c3eZWm5JPvDatFbykm8, address:45.91.116.8:4066): protocol: write tcp 159.223.143.39:44136->45.91.116.8:4066: use of closed network connection; context deadline exceeded Mar 6 19:53:18.345277850 (8h53m15.942048865s) [seg_pos={0 0}, upload=9902]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 6 19:53:18.367560006 (8h53m15.964331035s) [upload=9902]: Done uploading segments... Mar 6 19:53:18.367606646 (8h53m15.964377681s) [upload=9902]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 748.758353ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.X..........................................................] duration: 1.319988019s trace: https://jaeger.dev.storj.io/trace/78791d41d75e16a0 FAIL (2026-03-06 20:33:32 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 19:52:18.382132233 (8h52m15.978903242s) [upload=9902]: Uploading segments... Mar 6 19:52:18.382227489 (8h52m15.978998515s) [upload=9902]: Got next segment. Inline: false Mar 6 19:52:18.382313252 (8h52m15.979084279s) [upload=9902]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 6 19:52:18.382319269 (8h52m15.979090296s) [upload=9902]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 6 19:52:18.411471452 (8h52m16.008242482s) [seg_pos={0 0}, upload=9902]: Begin upload segment... Mar 6 19:52:18.411537096 (8h52m16.008308109s) [seg_pos={0 0}, upload=9902]: Done begin upload segment. Mar 6 19:52:18.411564540 (8h52m16.008335586s) [noise=true , address=45.91.116.7:4037 , piece_id=YXDMXUBE7CEWLJZP6FPDCRVIDGG5CJ3Y3IKEKO7EE2DQUSG6W5RA, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.411626862 (8h52m16.008397897s) [noise=true , address=193.149.164.67:4018 , piece_id=GTQQP355TZZF4W7PPBYZS6ZQ6WYK7I3QIE4DZC7STWAKSBAMT4BQ, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.412356605 (8h52m16.009127631s) [noise=true , address=167.17.72.70:4771 , piece_id=TDLPTUHPNXCICPPXRTRL7WCJSXPX22ONIWWPHNXFUNIKRYPYW52Q, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.412900389 (8h52m16.009671415s) [noise=true , address=145.79.227.39:4022 , piece_id=U5GX3GLVWGWLU2BDWTHM7ST3MLLJDEAERWU3UEHO7Z2RXHGZU2DA, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.413467061 (8h52m16.010238103s) [noise=true , address=45.91.116.11:4133 , piece_id=RZWV5JMLOJ36AH7KSDXVTL7CMHI4INEW6YOOQVMCIIC2NGS3XYJA, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.413653431 (8h52m16.010424457s) [noise=true , address=193.149.164.69:4101 , piece_id=BTIBVPSORFJQR3CLA5UV62NXNI6GD3JF4D5Z74KU4R4CXB3WHHKA, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.414227965 (8h52m16.010998999s) [noise=true , address=45.91.116.14:4199 , piece_id=EKGDG7LYV3O5NSDL3UPWFKXINYNAOSPYYOWQPTFN326AJ2ZXAXHQ, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.415177146 (8h52m16.01194818s) [noise=true , address=77.90.31.16:4763 , piece_id=5Q7H3CU3MWSFPQGC67XMLCMTSUIEJQUV7QTO4UZLZY4MIVM76Z6Q, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.415227889 (8h52m16.011998912s) [noise=true , address=45.91.116.10:4096 , piece_id=EIKWFKPE5BYSZGV5T4RPQED55PG2FXU3ZKSNRVCBD363YQDHS4BQ, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.415728547 (8h52m16.012499581s) [noise=true , address=77.90.31.14:4693 , piece_id=S7NAS2ZBHAQPZRHBRGRSZPLE6COQEEN6SZVMB7X2X6IMXTN5HCBQ, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.415910147 (8h52m16.012681174s) [noise=true , address=193.149.164.70:4129 , piece_id=V7WMI7RDYDOAEQYWF2OTZL5LRPUA2S3V5WZN4BKDY6BI4YV3GRPQ, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.416592462 (8h52m16.01336349s) [noise=true , address=77.90.31.18:4831 , piece_id=4PMIUNE4YXKLIJPML3VSUN4PF32TSLPOOSDZZNAXFWH7WICBOW3Q, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.417118897 (8h52m16.01388993s) [noise=true , address=77.90.31.13:4655 , piece_id=47Q27QI7O2IYW7YQKFMJGGMWPVZWWKQTFVYPUMPUI7ORE7236XIQ, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.417185844 (8h52m16.013956873s) [noise=true , address=193.149.164.71:4304 , piece_id=DTCL54GM76GWKAE7A3X5PUNFYO7H5JIKRT5WBEHYEDF2YNFYTOAA, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.417686790 (8h52m16.014457823s) [noise=true , address=167.17.72.71:4810 , piece_id=ANLHBMENEXSGTHBHLOIHMLM6JGHKIU2X6FPQQIUIVMVUOABNK73Q, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.418236951 (8h52m16.015007982s) [noise=true , address=145.79.227.17:4994 , piece_id=TO3EOAILO4PHF6Z7WQA4RWZNNL3JO6EXXEF2RLW7IHAGWUU2TF4Q, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.419293640 (8h52m16.016064672s) [noise=true , address=64.49.13.139:4479 , piece_id=XUYHOCSBATYECYHWLVZ4VBCMT3H4O5ETSQ5HZOB42OR3OY4LWJ6Q, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.420005502 (8h52m16.016776534s) [noise=true , address=45.91.116.8:4066 , piece_id=5MZHIJ3ELVBAJ7ABRKG565TLU4A4NKB7X6WAACY2JBSLACL5LGVQ, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.483654433 (8h52m16.080425441s) [upload=9902]: Next returned nil segment Mar 6 19:52:18.483671157 (8h52m16.08044219s) [upload=9902]: Waiting for error group managing segments... Mar 6 19:52:18.488078321 (8h52m16.084849398s) [noise=true , address=77.90.31.13:4655 , piece_id=47Q27QI7O2IYW7YQKFMJGGMWPVZWWKQTFVYPUMPUI7ORE7236XIQ, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:52:18.498435619 (8h52m16.095206647s) [noise=true , address=145.79.227.17:4994 , piece_id=TO3EOAILO4PHF6Z7WQA4RWZNNL3JO6EXXEF2RLW7IHAGWUU2TF4Q, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:52:18.502979814 (8h52m16.099750841s) [noise=true , address=77.90.31.16:4763 , piece_id=5Q7H3CU3MWSFPQGC67XMLCMTSUIEJQUV7QTO4UZLZY4MIVM76Z6Q, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:52:18.506259140 (8h52m16.103030166s) [noise=true , address=77.90.31.14:4693 , piece_id=S7NAS2ZBHAQPZRHBRGRSZPLE6COQEEN6SZVMB7X2X6IMXTN5HCBQ, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:52:18.514529389 (8h52m16.111300413s) [noise=true , address=145.79.227.39:4022 , piece_id=U5GX3GLVWGWLU2BDWTHM7ST3MLLJDEAERWU3UEHO7Z2RXHGZU2DA, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:52:18.785925494 (8h52m16.382696517s) [noise=true , address=167.17.72.70:4771 , piece_id=TDLPTUHPNXCICPPXRTRL7WCJSXPX22ONIWWPHNXFUNIKRYPYW52Q, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:52:18.810774903 (8h52m16.407545926s) [noise=true , address=77.90.31.18:4831 , piece_id=4PMIUNE4YXKLIJPML3VSUN4PF32TSLPOOSDZZNAXFWH7WICBOW3Q, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:52:19.002746550 (8h52m16.5995176s) [noise=true , address=193.149.164.71:4304 , piece_id=DTCL54GM76GWKAE7A3X5PUNFYO7H5JIKRT5WBEHYEDF2YNFYTOAA, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:52:19.007614901 (8h52m16.604385924s) [noise=true , address=193.149.164.67:4018 , piece_id=GTQQP355TZZF4W7PPBYZS6ZQ6WYK7I3QIE4DZC7STWAKSBAMT4BQ, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:52:19.010443151 (8h52m16.607214176s) [noise=true , address=193.149.164.69:4101 , piece_id=BTIBVPSORFJQR3CLA5UV62NXNI6GD3JF4D5Z74KU4R4CXB3WHHKA, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:52:19.012614014 (8h52m16.609385038s) [noise=true , address=193.149.164.70:4129 , piece_id=V7WMI7RDYDOAEQYWF2OTZL5LRPUA2S3V5WZN4BKDY6BI4YV3GRPQ, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:52:19.042145380 (8h52m16.638916406s) [noise=true , address=64.49.13.139:4479 , piece_id=XUYHOCSBATYECYHWLVZ4VBCMT3H4O5ETSQ5HZOB42OR3OY4LWJ6Q, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:52:21.314990180 (8h52m18.911761213s) [noise=true , address=167.17.72.71:4810 , piece_id=ANLHBMENEXSGTHBHLOIHMLM6JGHKIU2X6FPQQIUIVMVUOABNK73Q, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:53:18.344002671 (8h53m15.9407737s) [noise=true , address=45.91.116.10:4096 , piece_id=EIKWFKPE5BYSZGV5T4RPQED55PG2FXU3ZKSNRVCBD363YQDHS4BQ, seg_pos={0 0}, upload=9902]: Done uploading piece. err: ecclient: upload failed (node:12MZX1GwZGrEi2q44dPiYrwMoum5A2caZtXTS3qiLNcoyHhon7p, address:45.91.116.10:4096): protocol: write tcp 159.223.143.39:44996->45.91.116.10:4096: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 6 19:53:18.344173472 (8h53m15.940944497s) [noise=true , address=45.91.116.7:4037 , piece_id=YXDMXUBE7CEWLJZP6FPDCRVIDGG5CJ3Y3IKEKO7EE2DQUSG6W5RA, seg_pos={0 0}, upload=9902]: Done uploading piece. err: ecclient: upload failed (node:1iNPwE2XSfXcaXAXcrH2jcyNGYeDiiWvzAcDQffrmjPwTVqVmK, address:45.91.116.7:4037): protocol: write tcp 159.223.143.39:52854->45.91.116.7:4037: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 6 19:53:18.344535130 (8h53m15.941306156s) [noise=true , address=45.91.116.14:4199 , piece_id=EKGDG7LYV3O5NSDL3UPWFKXINYNAOSPYYOWQPTFN326AJ2ZXAXHQ, seg_pos={0 0}, upload=9902]: Done uploading piece. err: ecclient: upload failed (node:1d4B1KMLs2annUuJ3v81FJapS9Yg4omBTj48eWoi6HJxnSEjrf, address:45.91.116.14:4199): protocol: write tcp 159.223.143.39:57596->45.91.116.14:4199: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 6 19:53:18.344687364 (8h53m15.941458387s) [noise=true , address=45.91.116.11:4133 , piece_id=RZWV5JMLOJ36AH7KSDXVTL7CMHI4INEW6YOOQVMCIIC2NGS3XYJA, seg_pos={0 0}, upload=9902]: Done uploading piece. err: ecclient: upload failed (node:12e1wX7hiER8sX4MLm5qLue4mayiN4SxXFdZSxfymRaRZxRGU6E, address:45.91.116.11:4133): protocol: write tcp 159.223.143.39:56440->45.91.116.11:4133: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 6 19:53:18.345220236 (8h53m15.941991259s) [noise=true , address=45.91.116.8:4066 , piece_id=5MZHIJ3ELVBAJ7ABRKG565TLU4A4NKB7X6WAACY2JBSLACL5LGVQ, seg_pos={0 0}, upload=9902]: Done uploading piece. err: ecclient: upload failed (node:123uv49k8LraXJfYQg6KRQthrYs1Egc4c3eZWm5JPvDatFbykm8, address:45.91.116.8:4066): protocol: write tcp 159.223.143.39:44136->45.91.116.8:4066: use of closed network connection; context deadline exceeded Mar 6 19:53:18.345277850 (8h53m15.942048865s) [seg_pos={0 0}, upload=9902]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 6 19:53:18.367560006 (8h53m15.964331035s) [upload=9902]: Done uploading segments... Mar 6 19:53:18.367606646 (8h53m15.964377681s) [upload=9902]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 1.320016393s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..X.........................................................] duration: 1.191132036s trace: https://jaeger.dev.storj.io/trace/78791d41d75e16a0 FAIL (2026-03-06 20:32:21 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 19:52:18.382132233 (8h52m15.978903242s) [upload=9902]: Uploading segments... Mar 6 19:52:18.382227489 (8h52m15.978998515s) [upload=9902]: Got next segment. Inline: false Mar 6 19:52:18.382313252 (8h52m15.979084279s) [upload=9902]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 6 19:52:18.382319269 (8h52m15.979090296s) [upload=9902]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 6 19:52:18.411471452 (8h52m16.008242482s) [seg_pos={0 0}, upload=9902]: Begin upload segment... Mar 6 19:52:18.411537096 (8h52m16.008308109s) [seg_pos={0 0}, upload=9902]: Done begin upload segment. Mar 6 19:52:18.411564540 (8h52m16.008335586s) [noise=true , address=45.91.116.7:4037 , piece_id=YXDMXUBE7CEWLJZP6FPDCRVIDGG5CJ3Y3IKEKO7EE2DQUSG6W5RA, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.411626862 (8h52m16.008397897s) [noise=true , address=193.149.164.67:4018 , piece_id=GTQQP355TZZF4W7PPBYZS6ZQ6WYK7I3QIE4DZC7STWAKSBAMT4BQ, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.412356605 (8h52m16.009127631s) [noise=true , address=167.17.72.70:4771 , piece_id=TDLPTUHPNXCICPPXRTRL7WCJSXPX22ONIWWPHNXFUNIKRYPYW52Q, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.412900389 (8h52m16.009671415s) [noise=true , address=145.79.227.39:4022 , piece_id=U5GX3GLVWGWLU2BDWTHM7ST3MLLJDEAERWU3UEHO7Z2RXHGZU2DA, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.413467061 (8h52m16.010238103s) [noise=true , address=45.91.116.11:4133 , piece_id=RZWV5JMLOJ36AH7KSDXVTL7CMHI4INEW6YOOQVMCIIC2NGS3XYJA, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.413653431 (8h52m16.010424457s) [noise=true , address=193.149.164.69:4101 , piece_id=BTIBVPSORFJQR3CLA5UV62NXNI6GD3JF4D5Z74KU4R4CXB3WHHKA, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.414227965 (8h52m16.010998999s) [noise=true , address=45.91.116.14:4199 , piece_id=EKGDG7LYV3O5NSDL3UPWFKXINYNAOSPYYOWQPTFN326AJ2ZXAXHQ, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.415177146 (8h52m16.01194818s) [noise=true , address=77.90.31.16:4763 , piece_id=5Q7H3CU3MWSFPQGC67XMLCMTSUIEJQUV7QTO4UZLZY4MIVM76Z6Q, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.415227889 (8h52m16.011998912s) [noise=true , address=45.91.116.10:4096 , piece_id=EIKWFKPE5BYSZGV5T4RPQED55PG2FXU3ZKSNRVCBD363YQDHS4BQ, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.415728547 (8h52m16.012499581s) [noise=true , address=77.90.31.14:4693 , piece_id=S7NAS2ZBHAQPZRHBRGRSZPLE6COQEEN6SZVMB7X2X6IMXTN5HCBQ, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.415910147 (8h52m16.012681174s) [noise=true , address=193.149.164.70:4129 , piece_id=V7WMI7RDYDOAEQYWF2OTZL5LRPUA2S3V5WZN4BKDY6BI4YV3GRPQ, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.416592462 (8h52m16.01336349s) [noise=true , address=77.90.31.18:4831 , piece_id=4PMIUNE4YXKLIJPML3VSUN4PF32TSLPOOSDZZNAXFWH7WICBOW3Q, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.417118897 (8h52m16.01388993s) [noise=true , address=77.90.31.13:4655 , piece_id=47Q27QI7O2IYW7YQKFMJGGMWPVZWWKQTFVYPUMPUI7ORE7236XIQ, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.417185844 (8h52m16.013956873s) [noise=true , address=193.149.164.71:4304 , piece_id=DTCL54GM76GWKAE7A3X5PUNFYO7H5JIKRT5WBEHYEDF2YNFYTOAA, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.417686790 (8h52m16.014457823s) [noise=true , address=167.17.72.71:4810 , piece_id=ANLHBMENEXSGTHBHLOIHMLM6JGHKIU2X6FPQQIUIVMVUOABNK73Q, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.418236951 (8h52m16.015007982s) [noise=true , address=145.79.227.17:4994 , piece_id=TO3EOAILO4PHF6Z7WQA4RWZNNL3JO6EXXEF2RLW7IHAGWUU2TF4Q, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.419293640 (8h52m16.016064672s) [noise=true , address=64.49.13.139:4479 , piece_id=XUYHOCSBATYECYHWLVZ4VBCMT3H4O5ETSQ5HZOB42OR3OY4LWJ6Q, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.420005502 (8h52m16.016776534s) [noise=true , address=45.91.116.8:4066 , piece_id=5MZHIJ3ELVBAJ7ABRKG565TLU4A4NKB7X6WAACY2JBSLACL5LGVQ, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.483654433 (8h52m16.080425441s) [upload=9902]: Next returned nil segment Mar 6 19:52:18.483671157 (8h52m16.08044219s) [upload=9902]: Waiting for error group managing segments... Mar 6 19:52:18.488078321 (8h52m16.084849398s) [noise=true , address=77.90.31.13:4655 , piece_id=47Q27QI7O2IYW7YQKFMJGGMWPVZWWKQTFVYPUMPUI7ORE7236XIQ, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:52:18.498435619 (8h52m16.095206647s) [noise=true , address=145.79.227.17:4994 , piece_id=TO3EOAILO4PHF6Z7WQA4RWZNNL3JO6EXXEF2RLW7IHAGWUU2TF4Q, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:52:18.502979814 (8h52m16.099750841s) [noise=true , address=77.90.31.16:4763 , piece_id=5Q7H3CU3MWSFPQGC67XMLCMTSUIEJQUV7QTO4UZLZY4MIVM76Z6Q, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:52:18.506259140 (8h52m16.103030166s) [noise=true , address=77.90.31.14:4693 , piece_id=S7NAS2ZBHAQPZRHBRGRSZPLE6COQEEN6SZVMB7X2X6IMXTN5HCBQ, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:52:18.514529389 (8h52m16.111300413s) [noise=true , address=145.79.227.39:4022 , piece_id=U5GX3GLVWGWLU2BDWTHM7ST3MLLJDEAERWU3UEHO7Z2RXHGZU2DA, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:52:18.785925494 (8h52m16.382696517s) [noise=true , address=167.17.72.70:4771 , piece_id=TDLPTUHPNXCICPPXRTRL7WCJSXPX22ONIWWPHNXFUNIKRYPYW52Q, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:52:18.810774903 (8h52m16.407545926s) [noise=true , address=77.90.31.18:4831 , piece_id=4PMIUNE4YXKLIJPML3VSUN4PF32TSLPOOSDZZNAXFWH7WICBOW3Q, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:52:19.002746550 (8h52m16.5995176s) [noise=true , address=193.149.164.71:4304 , piece_id=DTCL54GM76GWKAE7A3X5PUNFYO7H5JIKRT5WBEHYEDF2YNFYTOAA, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:52:19.007614901 (8h52m16.604385924s) [noise=true , address=193.149.164.67:4018 , piece_id=GTQQP355TZZF4W7PPBYZS6ZQ6WYK7I3QIE4DZC7STWAKSBAMT4BQ, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:52:19.010443151 (8h52m16.607214176s) [noise=true , address=193.149.164.69:4101 , piece_id=BTIBVPSORFJQR3CLA5UV62NXNI6GD3JF4D5Z74KU4R4CXB3WHHKA, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:52:19.012614014 (8h52m16.609385038s) [noise=true , address=193.149.164.70:4129 , piece_id=V7WMI7RDYDOAEQYWF2OTZL5LRPUA2S3V5WZN4BKDY6BI4YV3GRPQ, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:52:19.042145380 (8h52m16.638916406s) [noise=true , address=64.49.13.139:4479 , piece_id=XUYHOCSBATYECYHWLVZ4VBCMT3H4O5ETSQ5HZOB42OR3OY4LWJ6Q, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:52:21.314990180 (8h52m18.911761213s) [noise=true , address=167.17.72.71:4810 , piece_id=ANLHBMENEXSGTHBHLOIHMLM6JGHKIU2X6FPQQIUIVMVUOABNK73Q, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:53:18.344002671 (8h53m15.9407737s) [noise=true , address=45.91.116.10:4096 , piece_id=EIKWFKPE5BYSZGV5T4RPQED55PG2FXU3ZKSNRVCBD363YQDHS4BQ, seg_pos={0 0}, upload=9902]: Done uploading piece. err: ecclient: upload failed (node:12MZX1GwZGrEi2q44dPiYrwMoum5A2caZtXTS3qiLNcoyHhon7p, address:45.91.116.10:4096): protocol: write tcp 159.223.143.39:44996->45.91.116.10:4096: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 6 19:53:18.344173472 (8h53m15.940944497s) [noise=true , address=45.91.116.7:4037 , piece_id=YXDMXUBE7CEWLJZP6FPDCRVIDGG5CJ3Y3IKEKO7EE2DQUSG6W5RA, seg_pos={0 0}, upload=9902]: Done uploading piece. err: ecclient: upload failed (node:1iNPwE2XSfXcaXAXcrH2jcyNGYeDiiWvzAcDQffrmjPwTVqVmK, address:45.91.116.7:4037): protocol: write tcp 159.223.143.39:52854->45.91.116.7:4037: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 6 19:53:18.344535130 (8h53m15.941306156s) [noise=true , address=45.91.116.14:4199 , piece_id=EKGDG7LYV3O5NSDL3UPWFKXINYNAOSPYYOWQPTFN326AJ2ZXAXHQ, seg_pos={0 0}, upload=9902]: Done uploading piece. err: ecclient: upload failed (node:1d4B1KMLs2annUuJ3v81FJapS9Yg4omBTj48eWoi6HJxnSEjrf, address:45.91.116.14:4199): protocol: write tcp 159.223.143.39:57596->45.91.116.14:4199: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 6 19:53:18.344687364 (8h53m15.941458387s) [noise=true , address=45.91.116.11:4133 , piece_id=RZWV5JMLOJ36AH7KSDXVTL7CMHI4INEW6YOOQVMCIIC2NGS3XYJA, seg_pos={0 0}, upload=9902]: Done uploading piece. err: ecclient: upload failed (node:12e1wX7hiER8sX4MLm5qLue4mayiN4SxXFdZSxfymRaRZxRGU6E, address:45.91.116.11:4133): protocol: write tcp 159.223.143.39:56440->45.91.116.11:4133: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 6 19:53:18.345220236 (8h53m15.941991259s) [noise=true , address=45.91.116.8:4066 , piece_id=5MZHIJ3ELVBAJ7ABRKG565TLU4A4NKB7X6WAACY2JBSLACL5LGVQ, seg_pos={0 0}, upload=9902]: Done uploading piece. err: ecclient: upload failed (node:123uv49k8LraXJfYQg6KRQthrYs1Egc4c3eZWm5JPvDatFbykm8, address:45.91.116.8:4066): protocol: write tcp 159.223.143.39:44136->45.91.116.8:4066: use of closed network connection; context deadline exceeded Mar 6 19:53:18.345277850 (8h53m15.942048865s) [seg_pos={0 0}, upload=9902]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 6 19:53:18.367560006 (8h53m15.964331035s) [upload=9902]: Done uploading segments... Mar 6 19:53:18.367606646 (8h53m15.964377681s) [upload=9902]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 1.191156175s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...X........................................................] duration: 750.476092ms trace: https://jaeger.dev.storj.io/trace/78791d41d75e16a0 FAIL (2026-03-06 20:31:45 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 19:52:18.382132233 (8h52m15.978903242s) [upload=9902]: Uploading segments... Mar 6 19:52:18.382227489 (8h52m15.978998515s) [upload=9902]: Got next segment. Inline: false Mar 6 19:52:18.382313252 (8h52m15.979084279s) [upload=9902]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 6 19:52:18.382319269 (8h52m15.979090296s) [upload=9902]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 6 19:52:18.411471452 (8h52m16.008242482s) [seg_pos={0 0}, upload=9902]: Begin upload segment... Mar 6 19:52:18.411537096 (8h52m16.008308109s) [seg_pos={0 0}, upload=9902]: Done begin upload segment. Mar 6 19:52:18.411564540 (8h52m16.008335586s) [noise=true , address=45.91.116.7:4037 , piece_id=YXDMXUBE7CEWLJZP6FPDCRVIDGG5CJ3Y3IKEKO7EE2DQUSG6W5RA, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.411626862 (8h52m16.008397897s) [noise=true , address=193.149.164.67:4018 , piece_id=GTQQP355TZZF4W7PPBYZS6ZQ6WYK7I3QIE4DZC7STWAKSBAMT4BQ, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.412356605 (8h52m16.009127631s) [noise=true , address=167.17.72.70:4771 , piece_id=TDLPTUHPNXCICPPXRTRL7WCJSXPX22ONIWWPHNXFUNIKRYPYW52Q, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.412900389 (8h52m16.009671415s) [noise=true , address=145.79.227.39:4022 , piece_id=U5GX3GLVWGWLU2BDWTHM7ST3MLLJDEAERWU3UEHO7Z2RXHGZU2DA, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.413467061 (8h52m16.010238103s) [noise=true , address=45.91.116.11:4133 , piece_id=RZWV5JMLOJ36AH7KSDXVTL7CMHI4INEW6YOOQVMCIIC2NGS3XYJA, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.413653431 (8h52m16.010424457s) [noise=true , address=193.149.164.69:4101 , piece_id=BTIBVPSORFJQR3CLA5UV62NXNI6GD3JF4D5Z74KU4R4CXB3WHHKA, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.414227965 (8h52m16.010998999s) [noise=true , address=45.91.116.14:4199 , piece_id=EKGDG7LYV3O5NSDL3UPWFKXINYNAOSPYYOWQPTFN326AJ2ZXAXHQ, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.415177146 (8h52m16.01194818s) [noise=true , address=77.90.31.16:4763 , piece_id=5Q7H3CU3MWSFPQGC67XMLCMTSUIEJQUV7QTO4UZLZY4MIVM76Z6Q, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.415227889 (8h52m16.011998912s) [noise=true , address=45.91.116.10:4096 , piece_id=EIKWFKPE5BYSZGV5T4RPQED55PG2FXU3ZKSNRVCBD363YQDHS4BQ, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.415728547 (8h52m16.012499581s) [noise=true , address=77.90.31.14:4693 , piece_id=S7NAS2ZBHAQPZRHBRGRSZPLE6COQEEN6SZVMB7X2X6IMXTN5HCBQ, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.415910147 (8h52m16.012681174s) [noise=true , address=193.149.164.70:4129 , piece_id=V7WMI7RDYDOAEQYWF2OTZL5LRPUA2S3V5WZN4BKDY6BI4YV3GRPQ, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.416592462 (8h52m16.01336349s) [noise=true , address=77.90.31.18:4831 , piece_id=4PMIUNE4YXKLIJPML3VSUN4PF32TSLPOOSDZZNAXFWH7WICBOW3Q, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.417118897 (8h52m16.01388993s) [noise=true , address=77.90.31.13:4655 , piece_id=47Q27QI7O2IYW7YQKFMJGGMWPVZWWKQTFVYPUMPUI7ORE7236XIQ, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.417185844 (8h52m16.013956873s) [noise=true , address=193.149.164.71:4304 , piece_id=DTCL54GM76GWKAE7A3X5PUNFYO7H5JIKRT5WBEHYEDF2YNFYTOAA, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.417686790 (8h52m16.014457823s) [noise=true , address=167.17.72.71:4810 , piece_id=ANLHBMENEXSGTHBHLOIHMLM6JGHKIU2X6FPQQIUIVMVUOABNK73Q, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.418236951 (8h52m16.015007982s) [noise=true , address=145.79.227.17:4994 , piece_id=TO3EOAILO4PHF6Z7WQA4RWZNNL3JO6EXXEF2RLW7IHAGWUU2TF4Q, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.419293640 (8h52m16.016064672s) [noise=true , address=64.49.13.139:4479 , piece_id=XUYHOCSBATYECYHWLVZ4VBCMT3H4O5ETSQ5HZOB42OR3OY4LWJ6Q, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.420005502 (8h52m16.016776534s) [noise=true , address=45.91.116.8:4066 , piece_id=5MZHIJ3ELVBAJ7ABRKG565TLU4A4NKB7X6WAACY2JBSLACL5LGVQ, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.483654433 (8h52m16.080425441s) [upload=9902]: Next returned nil segment Mar 6 19:52:18.483671157 (8h52m16.08044219s) [upload=9902]: Waiting for error group managing segments... Mar 6 19:52:18.488078321 (8h52m16.084849398s) [noise=true , address=77.90.31.13:4655 , piece_id=47Q27QI7O2IYW7YQKFMJGGMWPVZWWKQTFVYPUMPUI7ORE7236XIQ, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:52:18.498435619 (8h52m16.095206647s) [noise=true , address=145.79.227.17:4994 , piece_id=TO3EOAILO4PHF6Z7WQA4RWZNNL3JO6EXXEF2RLW7IHAGWUU2TF4Q, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:52:18.502979814 (8h52m16.099750841s) [noise=true , address=77.90.31.16:4763 , piece_id=5Q7H3CU3MWSFPQGC67XMLCMTSUIEJQUV7QTO4UZLZY4MIVM76Z6Q, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:52:18.506259140 (8h52m16.103030166s) [noise=true , address=77.90.31.14:4693 , piece_id=S7NAS2ZBHAQPZRHBRGRSZPLE6COQEEN6SZVMB7X2X6IMXTN5HCBQ, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:52:18.514529389 (8h52m16.111300413s) [noise=true , address=145.79.227.39:4022 , piece_id=U5GX3GLVWGWLU2BDWTHM7ST3MLLJDEAERWU3UEHO7Z2RXHGZU2DA, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:52:18.785925494 (8h52m16.382696517s) [noise=true , address=167.17.72.70:4771 , piece_id=TDLPTUHPNXCICPPXRTRL7WCJSXPX22ONIWWPHNXFUNIKRYPYW52Q, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:52:18.810774903 (8h52m16.407545926s) [noise=true , address=77.90.31.18:4831 , piece_id=4PMIUNE4YXKLIJPML3VSUN4PF32TSLPOOSDZZNAXFWH7WICBOW3Q, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:52:19.002746550 (8h52m16.5995176s) [noise=true , address=193.149.164.71:4304 , piece_id=DTCL54GM76GWKAE7A3X5PUNFYO7H5JIKRT5WBEHYEDF2YNFYTOAA, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:52:19.007614901 (8h52m16.604385924s) [noise=true , address=193.149.164.67:4018 , piece_id=GTQQP355TZZF4W7PPBYZS6ZQ6WYK7I3QIE4DZC7STWAKSBAMT4BQ, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:52:19.010443151 (8h52m16.607214176s) [noise=true , address=193.149.164.69:4101 , piece_id=BTIBVPSORFJQR3CLA5UV62NXNI6GD3JF4D5Z74KU4R4CXB3WHHKA, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:52:19.012614014 (8h52m16.609385038s) [noise=true , address=193.149.164.70:4129 , piece_id=V7WMI7RDYDOAEQYWF2OTZL5LRPUA2S3V5WZN4BKDY6BI4YV3GRPQ, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:52:19.042145380 (8h52m16.638916406s) [noise=true , address=64.49.13.139:4479 , piece_id=XUYHOCSBATYECYHWLVZ4VBCMT3H4O5ETSQ5HZOB42OR3OY4LWJ6Q, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:52:21.314990180 (8h52m18.911761213s) [noise=true , address=167.17.72.71:4810 , piece_id=ANLHBMENEXSGTHBHLOIHMLM6JGHKIU2X6FPQQIUIVMVUOABNK73Q, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:53:18.344002671 (8h53m15.9407737s) [noise=true , address=45.91.116.10:4096 , piece_id=EIKWFKPE5BYSZGV5T4RPQED55PG2FXU3ZKSNRVCBD363YQDHS4BQ, seg_pos={0 0}, upload=9902]: Done uploading piece. err: ecclient: upload failed (node:12MZX1GwZGrEi2q44dPiYrwMoum5A2caZtXTS3qiLNcoyHhon7p, address:45.91.116.10:4096): protocol: write tcp 159.223.143.39:44996->45.91.116.10:4096: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 6 19:53:18.344173472 (8h53m15.940944497s) [noise=true , address=45.91.116.7:4037 , piece_id=YXDMXUBE7CEWLJZP6FPDCRVIDGG5CJ3Y3IKEKO7EE2DQUSG6W5RA, seg_pos={0 0}, upload=9902]: Done uploading piece. err: ecclient: upload failed (node:1iNPwE2XSfXcaXAXcrH2jcyNGYeDiiWvzAcDQffrmjPwTVqVmK, address:45.91.116.7:4037): protocol: write tcp 159.223.143.39:52854->45.91.116.7:4037: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 6 19:53:18.344535130 (8h53m15.941306156s) [noise=true , address=45.91.116.14:4199 , piece_id=EKGDG7LYV3O5NSDL3UPWFKXINYNAOSPYYOWQPTFN326AJ2ZXAXHQ, seg_pos={0 0}, upload=9902]: Done uploading piece. err: ecclient: upload failed (node:1d4B1KMLs2annUuJ3v81FJapS9Yg4omBTj48eWoi6HJxnSEjrf, address:45.91.116.14:4199): protocol: write tcp 159.223.143.39:57596->45.91.116.14:4199: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 6 19:53:18.344687364 (8h53m15.941458387s) [noise=true , address=45.91.116.11:4133 , piece_id=RZWV5JMLOJ36AH7KSDXVTL7CMHI4INEW6YOOQVMCIIC2NGS3XYJA, seg_pos={0 0}, upload=9902]: Done uploading piece. err: ecclient: upload failed (node:12e1wX7hiER8sX4MLm5qLue4mayiN4SxXFdZSxfymRaRZxRGU6E, address:45.91.116.11:4133): protocol: write tcp 159.223.143.39:56440->45.91.116.11:4133: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 6 19:53:18.345220236 (8h53m15.941991259s) [noise=true , address=45.91.116.8:4066 , piece_id=5MZHIJ3ELVBAJ7ABRKG565TLU4A4NKB7X6WAACY2JBSLACL5LGVQ, seg_pos={0 0}, upload=9902]: Done uploading piece. err: ecclient: upload failed (node:123uv49k8LraXJfYQg6KRQthrYs1Egc4c3eZWm5JPvDatFbykm8, address:45.91.116.8:4066): protocol: write tcp 159.223.143.39:44136->45.91.116.8:4066: use of closed network connection; context deadline exceeded Mar 6 19:53:18.345277850 (8h53m15.942048865s) [seg_pos={0 0}, upload=9902]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 6 19:53:18.367560006 (8h53m15.964331035s) [upload=9902]: Done uploading segments... Mar 6 19:53:18.367606646 (8h53m15.964377681s) [upload=9902]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 750.500157ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [....X.......................................................] duration: 1.29216813s trace: https://jaeger.dev.storj.io/trace/78791d41d75e16a0 FAIL (2026-03-06 20:31:09 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 19:52:18.382132233 (8h52m15.978903242s) [upload=9902]: Uploading segments... Mar 6 19:52:18.382227489 (8h52m15.978998515s) [upload=9902]: Got next segment. Inline: false Mar 6 19:52:18.382313252 (8h52m15.979084279s) [upload=9902]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 6 19:52:18.382319269 (8h52m15.979090296s) [upload=9902]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 6 19:52:18.411471452 (8h52m16.008242482s) [seg_pos={0 0}, upload=9902]: Begin upload segment... Mar 6 19:52:18.411537096 (8h52m16.008308109s) [seg_pos={0 0}, upload=9902]: Done begin upload segment. Mar 6 19:52:18.411564540 (8h52m16.008335586s) [noise=true , address=45.91.116.7:4037 , piece_id=YXDMXUBE7CEWLJZP6FPDCRVIDGG5CJ3Y3IKEKO7EE2DQUSG6W5RA, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.411626862 (8h52m16.008397897s) [noise=true , address=193.149.164.67:4018 , piece_id=GTQQP355TZZF4W7PPBYZS6ZQ6WYK7I3QIE4DZC7STWAKSBAMT4BQ, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.412356605 (8h52m16.009127631s) [noise=true , address=167.17.72.70:4771 , piece_id=TDLPTUHPNXCICPPXRTRL7WCJSXPX22ONIWWPHNXFUNIKRYPYW52Q, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.412900389 (8h52m16.009671415s) [noise=true , address=145.79.227.39:4022 , piece_id=U5GX3GLVWGWLU2BDWTHM7ST3MLLJDEAERWU3UEHO7Z2RXHGZU2DA, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.413467061 (8h52m16.010238103s) [noise=true , address=45.91.116.11:4133 , piece_id=RZWV5JMLOJ36AH7KSDXVTL7CMHI4INEW6YOOQVMCIIC2NGS3XYJA, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.413653431 (8h52m16.010424457s) [noise=true , address=193.149.164.69:4101 , piece_id=BTIBVPSORFJQR3CLA5UV62NXNI6GD3JF4D5Z74KU4R4CXB3WHHKA, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.414227965 (8h52m16.010998999s) [noise=true , address=45.91.116.14:4199 , piece_id=EKGDG7LYV3O5NSDL3UPWFKXINYNAOSPYYOWQPTFN326AJ2ZXAXHQ, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.415177146 (8h52m16.01194818s) [noise=true , address=77.90.31.16:4763 , piece_id=5Q7H3CU3MWSFPQGC67XMLCMTSUIEJQUV7QTO4UZLZY4MIVM76Z6Q, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.415227889 (8h52m16.011998912s) [noise=true , address=45.91.116.10:4096 , piece_id=EIKWFKPE5BYSZGV5T4RPQED55PG2FXU3ZKSNRVCBD363YQDHS4BQ, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.415728547 (8h52m16.012499581s) [noise=true , address=77.90.31.14:4693 , piece_id=S7NAS2ZBHAQPZRHBRGRSZPLE6COQEEN6SZVMB7X2X6IMXTN5HCBQ, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.415910147 (8h52m16.012681174s) [noise=true , address=193.149.164.70:4129 , piece_id=V7WMI7RDYDOAEQYWF2OTZL5LRPUA2S3V5WZN4BKDY6BI4YV3GRPQ, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.416592462 (8h52m16.01336349s) [noise=true , address=77.90.31.18:4831 , piece_id=4PMIUNE4YXKLIJPML3VSUN4PF32TSLPOOSDZZNAXFWH7WICBOW3Q, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.417118897 (8h52m16.01388993s) [noise=true , address=77.90.31.13:4655 , piece_id=47Q27QI7O2IYW7YQKFMJGGMWPVZWWKQTFVYPUMPUI7ORE7236XIQ, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.417185844 (8h52m16.013956873s) [noise=true , address=193.149.164.71:4304 , piece_id=DTCL54GM76GWKAE7A3X5PUNFYO7H5JIKRT5WBEHYEDF2YNFYTOAA, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.417686790 (8h52m16.014457823s) [noise=true , address=167.17.72.71:4810 , piece_id=ANLHBMENEXSGTHBHLOIHMLM6JGHKIU2X6FPQQIUIVMVUOABNK73Q, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.418236951 (8h52m16.015007982s) [noise=true , address=145.79.227.17:4994 , piece_id=TO3EOAILO4PHF6Z7WQA4RWZNNL3JO6EXXEF2RLW7IHAGWUU2TF4Q, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.419293640 (8h52m16.016064672s) [noise=true , address=64.49.13.139:4479 , piece_id=XUYHOCSBATYECYHWLVZ4VBCMT3H4O5ETSQ5HZOB42OR3OY4LWJ6Q, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.420005502 (8h52m16.016776534s) [noise=true , address=45.91.116.8:4066 , piece_id=5MZHIJ3ELVBAJ7ABRKG565TLU4A4NKB7X6WAACY2JBSLACL5LGVQ, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.483654433 (8h52m16.080425441s) [upload=9902]: Next returned nil segment Mar 6 19:52:18.483671157 (8h52m16.08044219s) [upload=9902]: Waiting for error group managing segments... Mar 6 19:52:18.488078321 (8h52m16.084849398s) [noise=true , address=77.90.31.13:4655 , piece_id=47Q27QI7O2IYW7YQKFMJGGMWPVZWWKQTFVYPUMPUI7ORE7236XIQ, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:52:18.498435619 (8h52m16.095206647s) [noise=true , address=145.79.227.17:4994 , piece_id=TO3EOAILO4PHF6Z7WQA4RWZNNL3JO6EXXEF2RLW7IHAGWUU2TF4Q, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:52:18.502979814 (8h52m16.099750841s) [noise=true , address=77.90.31.16:4763 , piece_id=5Q7H3CU3MWSFPQGC67XMLCMTSUIEJQUV7QTO4UZLZY4MIVM76Z6Q, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:52:18.506259140 (8h52m16.103030166s) [noise=true , address=77.90.31.14:4693 , piece_id=S7NAS2ZBHAQPZRHBRGRSZPLE6COQEEN6SZVMB7X2X6IMXTN5HCBQ, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:52:18.514529389 (8h52m16.111300413s) [noise=true , address=145.79.227.39:4022 , piece_id=U5GX3GLVWGWLU2BDWTHM7ST3MLLJDEAERWU3UEHO7Z2RXHGZU2DA, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:52:18.785925494 (8h52m16.382696517s) [noise=true , address=167.17.72.70:4771 , piece_id=TDLPTUHPNXCICPPXRTRL7WCJSXPX22ONIWWPHNXFUNIKRYPYW52Q, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:52:18.810774903 (8h52m16.407545926s) [noise=true , address=77.90.31.18:4831 , piece_id=4PMIUNE4YXKLIJPML3VSUN4PF32TSLPOOSDZZNAXFWH7WICBOW3Q, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:52:19.002746550 (8h52m16.5995176s) [noise=true , address=193.149.164.71:4304 , piece_id=DTCL54GM76GWKAE7A3X5PUNFYO7H5JIKRT5WBEHYEDF2YNFYTOAA, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:52:19.007614901 (8h52m16.604385924s) [noise=true , address=193.149.164.67:4018 , piece_id=GTQQP355TZZF4W7PPBYZS6ZQ6WYK7I3QIE4DZC7STWAKSBAMT4BQ, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:52:19.010443151 (8h52m16.607214176s) [noise=true , address=193.149.164.69:4101 , piece_id=BTIBVPSORFJQR3CLA5UV62NXNI6GD3JF4D5Z74KU4R4CXB3WHHKA, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:52:19.012614014 (8h52m16.609385038s) [noise=true , address=193.149.164.70:4129 , piece_id=V7WMI7RDYDOAEQYWF2OTZL5LRPUA2S3V5WZN4BKDY6BI4YV3GRPQ, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:52:19.042145380 (8h52m16.638916406s) [noise=true , address=64.49.13.139:4479 , piece_id=XUYHOCSBATYECYHWLVZ4VBCMT3H4O5ETSQ5HZOB42OR3OY4LWJ6Q, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:52:21.314990180 (8h52m18.911761213s) [noise=true , address=167.17.72.71:4810 , piece_id=ANLHBMENEXSGTHBHLOIHMLM6JGHKIU2X6FPQQIUIVMVUOABNK73Q, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:53:18.344002671 (8h53m15.9407737s) [noise=true , address=45.91.116.10:4096 , piece_id=EIKWFKPE5BYSZGV5T4RPQED55PG2FXU3ZKSNRVCBD363YQDHS4BQ, seg_pos={0 0}, upload=9902]: Done uploading piece. err: ecclient: upload failed (node:12MZX1GwZGrEi2q44dPiYrwMoum5A2caZtXTS3qiLNcoyHhon7p, address:45.91.116.10:4096): protocol: write tcp 159.223.143.39:44996->45.91.116.10:4096: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 6 19:53:18.344173472 (8h53m15.940944497s) [noise=true , address=45.91.116.7:4037 , piece_id=YXDMXUBE7CEWLJZP6FPDCRVIDGG5CJ3Y3IKEKO7EE2DQUSG6W5RA, seg_pos={0 0}, upload=9902]: Done uploading piece. err: ecclient: upload failed (node:1iNPwE2XSfXcaXAXcrH2jcyNGYeDiiWvzAcDQffrmjPwTVqVmK, address:45.91.116.7:4037): protocol: write tcp 159.223.143.39:52854->45.91.116.7:4037: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 6 19:53:18.344535130 (8h53m15.941306156s) [noise=true , address=45.91.116.14:4199 , piece_id=EKGDG7LYV3O5NSDL3UPWFKXINYNAOSPYYOWQPTFN326AJ2ZXAXHQ, seg_pos={0 0}, upload=9902]: Done uploading piece. err: ecclient: upload failed (node:1d4B1KMLs2annUuJ3v81FJapS9Yg4omBTj48eWoi6HJxnSEjrf, address:45.91.116.14:4199): protocol: write tcp 159.223.143.39:57596->45.91.116.14:4199: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 6 19:53:18.344687364 (8h53m15.941458387s) [noise=true , address=45.91.116.11:4133 , piece_id=RZWV5JMLOJ36AH7KSDXVTL7CMHI4INEW6YOOQVMCIIC2NGS3XYJA, seg_pos={0 0}, upload=9902]: Done uploading piece. err: ecclient: upload failed (node:12e1wX7hiER8sX4MLm5qLue4mayiN4SxXFdZSxfymRaRZxRGU6E, address:45.91.116.11:4133): protocol: write tcp 159.223.143.39:56440->45.91.116.11:4133: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 6 19:53:18.345220236 (8h53m15.941991259s) [noise=true , address=45.91.116.8:4066 , piece_id=5MZHIJ3ELVBAJ7ABRKG565TLU4A4NKB7X6WAACY2JBSLACL5LGVQ, seg_pos={0 0}, upload=9902]: Done uploading piece. err: ecclient: upload failed (node:123uv49k8LraXJfYQg6KRQthrYs1Egc4c3eZWm5JPvDatFbykm8, address:45.91.116.8:4066): protocol: write tcp 159.223.143.39:44136->45.91.116.8:4066: use of closed network connection; context deadline exceeded Mar 6 19:53:18.345277850 (8h53m15.942048865s) [seg_pos={0 0}, upload=9902]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 6 19:53:18.367560006 (8h53m15.964331035s) [upload=9902]: Done uploading segments... Mar 6 19:53:18.367606646 (8h53m15.964377681s) [upload=9902]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 1.292191574s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....X......................................................] duration: 786.035603ms trace: https://jaeger.dev.storj.io/trace/78791d41d75e16a0 FAIL (2026-03-06 20:30:28 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 19:52:18.382132233 (8h52m15.978903242s) [upload=9902]: Uploading segments... Mar 6 19:52:18.382227489 (8h52m15.978998515s) [upload=9902]: Got next segment. Inline: false Mar 6 19:52:18.382313252 (8h52m15.979084279s) [upload=9902]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 6 19:52:18.382319269 (8h52m15.979090296s) [upload=9902]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 6 19:52:18.411471452 (8h52m16.008242482s) [seg_pos={0 0}, upload=9902]: Begin upload segment... Mar 6 19:52:18.411537096 (8h52m16.008308109s) [seg_pos={0 0}, upload=9902]: Done begin upload segment. Mar 6 19:52:18.411564540 (8h52m16.008335586s) [noise=true , address=45.91.116.7:4037 , piece_id=YXDMXUBE7CEWLJZP6FPDCRVIDGG5CJ3Y3IKEKO7EE2DQUSG6W5RA, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.411626862 (8h52m16.008397897s) [noise=true , address=193.149.164.67:4018 , piece_id=GTQQP355TZZF4W7PPBYZS6ZQ6WYK7I3QIE4DZC7STWAKSBAMT4BQ, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.412356605 (8h52m16.009127631s) [noise=true , address=167.17.72.70:4771 , piece_id=TDLPTUHPNXCICPPXRTRL7WCJSXPX22ONIWWPHNXFUNIKRYPYW52Q, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.412900389 (8h52m16.009671415s) [noise=true , address=145.79.227.39:4022 , piece_id=U5GX3GLVWGWLU2BDWTHM7ST3MLLJDEAERWU3UEHO7Z2RXHGZU2DA, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.413467061 (8h52m16.010238103s) [noise=true , address=45.91.116.11:4133 , piece_id=RZWV5JMLOJ36AH7KSDXVTL7CMHI4INEW6YOOQVMCIIC2NGS3XYJA, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.413653431 (8h52m16.010424457s) [noise=true , address=193.149.164.69:4101 , piece_id=BTIBVPSORFJQR3CLA5UV62NXNI6GD3JF4D5Z74KU4R4CXB3WHHKA, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.414227965 (8h52m16.010998999s) [noise=true , address=45.91.116.14:4199 , piece_id=EKGDG7LYV3O5NSDL3UPWFKXINYNAOSPYYOWQPTFN326AJ2ZXAXHQ, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.415177146 (8h52m16.01194818s) [noise=true , address=77.90.31.16:4763 , piece_id=5Q7H3CU3MWSFPQGC67XMLCMTSUIEJQUV7QTO4UZLZY4MIVM76Z6Q, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.415227889 (8h52m16.011998912s) [noise=true , address=45.91.116.10:4096 , piece_id=EIKWFKPE5BYSZGV5T4RPQED55PG2FXU3ZKSNRVCBD363YQDHS4BQ, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.415728547 (8h52m16.012499581s) [noise=true , address=77.90.31.14:4693 , piece_id=S7NAS2ZBHAQPZRHBRGRSZPLE6COQEEN6SZVMB7X2X6IMXTN5HCBQ, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.415910147 (8h52m16.012681174s) [noise=true , address=193.149.164.70:4129 , piece_id=V7WMI7RDYDOAEQYWF2OTZL5LRPUA2S3V5WZN4BKDY6BI4YV3GRPQ, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.416592462 (8h52m16.01336349s) [noise=true , address=77.90.31.18:4831 , piece_id=4PMIUNE4YXKLIJPML3VSUN4PF32TSLPOOSDZZNAXFWH7WICBOW3Q, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.417118897 (8h52m16.01388993s) [noise=true , address=77.90.31.13:4655 , piece_id=47Q27QI7O2IYW7YQKFMJGGMWPVZWWKQTFVYPUMPUI7ORE7236XIQ, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.417185844 (8h52m16.013956873s) [noise=true , address=193.149.164.71:4304 , piece_id=DTCL54GM76GWKAE7A3X5PUNFYO7H5JIKRT5WBEHYEDF2YNFYTOAA, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.417686790 (8h52m16.014457823s) [noise=true , address=167.17.72.71:4810 , piece_id=ANLHBMENEXSGTHBHLOIHMLM6JGHKIU2X6FPQQIUIVMVUOABNK73Q, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.418236951 (8h52m16.015007982s) [noise=true , address=145.79.227.17:4994 , piece_id=TO3EOAILO4PHF6Z7WQA4RWZNNL3JO6EXXEF2RLW7IHAGWUU2TF4Q, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.419293640 (8h52m16.016064672s) [noise=true , address=64.49.13.139:4479 , piece_id=XUYHOCSBATYECYHWLVZ4VBCMT3H4O5ETSQ5HZOB42OR3OY4LWJ6Q, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.420005502 (8h52m16.016776534s) [noise=true , address=45.91.116.8:4066 , piece_id=5MZHIJ3ELVBAJ7ABRKG565TLU4A4NKB7X6WAACY2JBSLACL5LGVQ, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.483654433 (8h52m16.080425441s) [upload=9902]: Next returned nil segment Mar 6 19:52:18.483671157 (8h52m16.08044219s) [upload=9902]: Waiting for error group managing segments... Mar 6 19:52:18.488078321 (8h52m16.084849398s) [noise=true , address=77.90.31.13:4655 , piece_id=47Q27QI7O2IYW7YQKFMJGGMWPVZWWKQTFVYPUMPUI7ORE7236XIQ, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:52:18.498435619 (8h52m16.095206647s) [noise=true , address=145.79.227.17:4994 , piece_id=TO3EOAILO4PHF6Z7WQA4RWZNNL3JO6EXXEF2RLW7IHAGWUU2TF4Q, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:52:18.502979814 (8h52m16.099750841s) [noise=true , address=77.90.31.16:4763 , piece_id=5Q7H3CU3MWSFPQGC67XMLCMTSUIEJQUV7QTO4UZLZY4MIVM76Z6Q, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:52:18.506259140 (8h52m16.103030166s) [noise=true , address=77.90.31.14:4693 , piece_id=S7NAS2ZBHAQPZRHBRGRSZPLE6COQEEN6SZVMB7X2X6IMXTN5HCBQ, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:52:18.514529389 (8h52m16.111300413s) [noise=true , address=145.79.227.39:4022 , piece_id=U5GX3GLVWGWLU2BDWTHM7ST3MLLJDEAERWU3UEHO7Z2RXHGZU2DA, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:52:18.785925494 (8h52m16.382696517s) [noise=true , address=167.17.72.70:4771 , piece_id=TDLPTUHPNXCICPPXRTRL7WCJSXPX22ONIWWPHNXFUNIKRYPYW52Q, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:52:18.810774903 (8h52m16.407545926s) [noise=true , address=77.90.31.18:4831 , piece_id=4PMIUNE4YXKLIJPML3VSUN4PF32TSLPOOSDZZNAXFWH7WICBOW3Q, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:52:19.002746550 (8h52m16.5995176s) [noise=true , address=193.149.164.71:4304 , piece_id=DTCL54GM76GWKAE7A3X5PUNFYO7H5JIKRT5WBEHYEDF2YNFYTOAA, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:52:19.007614901 (8h52m16.604385924s) [noise=true , address=193.149.164.67:4018 , piece_id=GTQQP355TZZF4W7PPBYZS6ZQ6WYK7I3QIE4DZC7STWAKSBAMT4BQ, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:52:19.010443151 (8h52m16.607214176s) [noise=true , address=193.149.164.69:4101 , piece_id=BTIBVPSORFJQR3CLA5UV62NXNI6GD3JF4D5Z74KU4R4CXB3WHHKA, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:52:19.012614014 (8h52m16.609385038s) [noise=true , address=193.149.164.70:4129 , piece_id=V7WMI7RDYDOAEQYWF2OTZL5LRPUA2S3V5WZN4BKDY6BI4YV3GRPQ, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:52:19.042145380 (8h52m16.638916406s) [noise=true , address=64.49.13.139:4479 , piece_id=XUYHOCSBATYECYHWLVZ4VBCMT3H4O5ETSQ5HZOB42OR3OY4LWJ6Q, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:52:21.314990180 (8h52m18.911761213s) [noise=true , address=167.17.72.71:4810 , piece_id=ANLHBMENEXSGTHBHLOIHMLM6JGHKIU2X6FPQQIUIVMVUOABNK73Q, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:53:18.344002671 (8h53m15.9407737s) [noise=true , address=45.91.116.10:4096 , piece_id=EIKWFKPE5BYSZGV5T4RPQED55PG2FXU3ZKSNRVCBD363YQDHS4BQ, seg_pos={0 0}, upload=9902]: Done uploading piece. err: ecclient: upload failed (node:12MZX1GwZGrEi2q44dPiYrwMoum5A2caZtXTS3qiLNcoyHhon7p, address:45.91.116.10:4096): protocol: write tcp 159.223.143.39:44996->45.91.116.10:4096: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 6 19:53:18.344173472 (8h53m15.940944497s) [noise=true , address=45.91.116.7:4037 , piece_id=YXDMXUBE7CEWLJZP6FPDCRVIDGG5CJ3Y3IKEKO7EE2DQUSG6W5RA, seg_pos={0 0}, upload=9902]: Done uploading piece. err: ecclient: upload failed (node:1iNPwE2XSfXcaXAXcrH2jcyNGYeDiiWvzAcDQffrmjPwTVqVmK, address:45.91.116.7:4037): protocol: write tcp 159.223.143.39:52854->45.91.116.7:4037: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 6 19:53:18.344535130 (8h53m15.941306156s) [noise=true , address=45.91.116.14:4199 , piece_id=EKGDG7LYV3O5NSDL3UPWFKXINYNAOSPYYOWQPTFN326AJ2ZXAXHQ, seg_pos={0 0}, upload=9902]: Done uploading piece. err: ecclient: upload failed (node:1d4B1KMLs2annUuJ3v81FJapS9Yg4omBTj48eWoi6HJxnSEjrf, address:45.91.116.14:4199): protocol: write tcp 159.223.143.39:57596->45.91.116.14:4199: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 6 19:53:18.344687364 (8h53m15.941458387s) [noise=true , address=45.91.116.11:4133 , piece_id=RZWV5JMLOJ36AH7KSDXVTL7CMHI4INEW6YOOQVMCIIC2NGS3XYJA, seg_pos={0 0}, upload=9902]: Done uploading piece. err: ecclient: upload failed (node:12e1wX7hiER8sX4MLm5qLue4mayiN4SxXFdZSxfymRaRZxRGU6E, address:45.91.116.11:4133): protocol: write tcp 159.223.143.39:56440->45.91.116.11:4133: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 6 19:53:18.345220236 (8h53m15.941991259s) [noise=true , address=45.91.116.8:4066 , piece_id=5MZHIJ3ELVBAJ7ABRKG565TLU4A4NKB7X6WAACY2JBSLACL5LGVQ, seg_pos={0 0}, upload=9902]: Done uploading piece. err: ecclient: upload failed (node:123uv49k8LraXJfYQg6KRQthrYs1Egc4c3eZWm5JPvDatFbykm8, address:45.91.116.8:4066): protocol: write tcp 159.223.143.39:44136->45.91.116.8:4066: use of closed network connection; context deadline exceeded Mar 6 19:53:18.345277850 (8h53m15.942048865s) [seg_pos={0 0}, upload=9902]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 6 19:53:18.367560006 (8h53m15.964331035s) [upload=9902]: Done uploading segments... Mar 6 19:53:18.367606646 (8h53m15.964377681s) [upload=9902]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 786.065959ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [......X.....................................................] duration: 1.193634776s trace: https://jaeger.dev.storj.io/trace/78791d41d75e16a0 FAIL (2026-03-06 20:29:55 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 19:52:18.382132233 (8h52m15.978903242s) [upload=9902]: Uploading segments... Mar 6 19:52:18.382227489 (8h52m15.978998515s) [upload=9902]: Got next segment. Inline: false Mar 6 19:52:18.382313252 (8h52m15.979084279s) [upload=9902]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 6 19:52:18.382319269 (8h52m15.979090296s) [upload=9902]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 6 19:52:18.411471452 (8h52m16.008242482s) [seg_pos={0 0}, upload=9902]: Begin upload segment... Mar 6 19:52:18.411537096 (8h52m16.008308109s) [seg_pos={0 0}, upload=9902]: Done begin upload segment. Mar 6 19:52:18.411564540 (8h52m16.008335586s) [noise=true , address=45.91.116.7:4037 , piece_id=YXDMXUBE7CEWLJZP6FPDCRVIDGG5CJ3Y3IKEKO7EE2DQUSG6W5RA, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.411626862 (8h52m16.008397897s) [noise=true , address=193.149.164.67:4018 , piece_id=GTQQP355TZZF4W7PPBYZS6ZQ6WYK7I3QIE4DZC7STWAKSBAMT4BQ, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.412356605 (8h52m16.009127631s) [noise=true , address=167.17.72.70:4771 , piece_id=TDLPTUHPNXCICPPXRTRL7WCJSXPX22ONIWWPHNXFUNIKRYPYW52Q, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.412900389 (8h52m16.009671415s) [noise=true , address=145.79.227.39:4022 , piece_id=U5GX3GLVWGWLU2BDWTHM7ST3MLLJDEAERWU3UEHO7Z2RXHGZU2DA, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.413467061 (8h52m16.010238103s) [noise=true , address=45.91.116.11:4133 , piece_id=RZWV5JMLOJ36AH7KSDXVTL7CMHI4INEW6YOOQVMCIIC2NGS3XYJA, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.413653431 (8h52m16.010424457s) [noise=true , address=193.149.164.69:4101 , piece_id=BTIBVPSORFJQR3CLA5UV62NXNI6GD3JF4D5Z74KU4R4CXB3WHHKA, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.414227965 (8h52m16.010998999s) [noise=true , address=45.91.116.14:4199 , piece_id=EKGDG7LYV3O5NSDL3UPWFKXINYNAOSPYYOWQPTFN326AJ2ZXAXHQ, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.415177146 (8h52m16.01194818s) [noise=true , address=77.90.31.16:4763 , piece_id=5Q7H3CU3MWSFPQGC67XMLCMTSUIEJQUV7QTO4UZLZY4MIVM76Z6Q, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.415227889 (8h52m16.011998912s) [noise=true , address=45.91.116.10:4096 , piece_id=EIKWFKPE5BYSZGV5T4RPQED55PG2FXU3ZKSNRVCBD363YQDHS4BQ, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.415728547 (8h52m16.012499581s) [noise=true , address=77.90.31.14:4693 , piece_id=S7NAS2ZBHAQPZRHBRGRSZPLE6COQEEN6SZVMB7X2X6IMXTN5HCBQ, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.415910147 (8h52m16.012681174s) [noise=true , address=193.149.164.70:4129 , piece_id=V7WMI7RDYDOAEQYWF2OTZL5LRPUA2S3V5WZN4BKDY6BI4YV3GRPQ, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.416592462 (8h52m16.01336349s) [noise=true , address=77.90.31.18:4831 , piece_id=4PMIUNE4YXKLIJPML3VSUN4PF32TSLPOOSDZZNAXFWH7WICBOW3Q, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.417118897 (8h52m16.01388993s) [noise=true , address=77.90.31.13:4655 , piece_id=47Q27QI7O2IYW7YQKFMJGGMWPVZWWKQTFVYPUMPUI7ORE7236XIQ, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.417185844 (8h52m16.013956873s) [noise=true , address=193.149.164.71:4304 , piece_id=DTCL54GM76GWKAE7A3X5PUNFYO7H5JIKRT5WBEHYEDF2YNFYTOAA, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.417686790 (8h52m16.014457823s) [noise=true , address=167.17.72.71:4810 , piece_id=ANLHBMENEXSGTHBHLOIHMLM6JGHKIU2X6FPQQIUIVMVUOABNK73Q, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.418236951 (8h52m16.015007982s) [noise=true , address=145.79.227.17:4994 , piece_id=TO3EOAILO4PHF6Z7WQA4RWZNNL3JO6EXXEF2RLW7IHAGWUU2TF4Q, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.419293640 (8h52m16.016064672s) [noise=true , address=64.49.13.139:4479 , piece_id=XUYHOCSBATYECYHWLVZ4VBCMT3H4O5ETSQ5HZOB42OR3OY4LWJ6Q, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.420005502 (8h52m16.016776534s) [noise=true , address=45.91.116.8:4066 , piece_id=5MZHIJ3ELVBAJ7ABRKG565TLU4A4NKB7X6WAACY2JBSLACL5LGVQ, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.483654433 (8h52m16.080425441s) [upload=9902]: Next returned nil segment Mar 6 19:52:18.483671157 (8h52m16.08044219s) [upload=9902]: Waiting for error group managing segments... Mar 6 19:52:18.488078321 (8h52m16.084849398s) [noise=true , address=77.90.31.13:4655 , piece_id=47Q27QI7O2IYW7YQKFMJGGMWPVZWWKQTFVYPUMPUI7ORE7236XIQ, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:52:18.498435619 (8h52m16.095206647s) [noise=true , address=145.79.227.17:4994 , piece_id=TO3EOAILO4PHF6Z7WQA4RWZNNL3JO6EXXEF2RLW7IHAGWUU2TF4Q, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:52:18.502979814 (8h52m16.099750841s) [noise=true , address=77.90.31.16:4763 , piece_id=5Q7H3CU3MWSFPQGC67XMLCMTSUIEJQUV7QTO4UZLZY4MIVM76Z6Q, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:52:18.506259140 (8h52m16.103030166s) [noise=true , address=77.90.31.14:4693 , piece_id=S7NAS2ZBHAQPZRHBRGRSZPLE6COQEEN6SZVMB7X2X6IMXTN5HCBQ, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:52:18.514529389 (8h52m16.111300413s) [noise=true , address=145.79.227.39:4022 , piece_id=U5GX3GLVWGWLU2BDWTHM7ST3MLLJDEAERWU3UEHO7Z2RXHGZU2DA, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:52:18.785925494 (8h52m16.382696517s) [noise=true , address=167.17.72.70:4771 , piece_id=TDLPTUHPNXCICPPXRTRL7WCJSXPX22ONIWWPHNXFUNIKRYPYW52Q, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:52:18.810774903 (8h52m16.407545926s) [noise=true , address=77.90.31.18:4831 , piece_id=4PMIUNE4YXKLIJPML3VSUN4PF32TSLPOOSDZZNAXFWH7WICBOW3Q, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:52:19.002746550 (8h52m16.5995176s) [noise=true , address=193.149.164.71:4304 , piece_id=DTCL54GM76GWKAE7A3X5PUNFYO7H5JIKRT5WBEHYEDF2YNFYTOAA, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:52:19.007614901 (8h52m16.604385924s) [noise=true , address=193.149.164.67:4018 , piece_id=GTQQP355TZZF4W7PPBYZS6ZQ6WYK7I3QIE4DZC7STWAKSBAMT4BQ, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:52:19.010443151 (8h52m16.607214176s) [noise=true , address=193.149.164.69:4101 , piece_id=BTIBVPSORFJQR3CLA5UV62NXNI6GD3JF4D5Z74KU4R4CXB3WHHKA, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:52:19.012614014 (8h52m16.609385038s) [noise=true , address=193.149.164.70:4129 , piece_id=V7WMI7RDYDOAEQYWF2OTZL5LRPUA2S3V5WZN4BKDY6BI4YV3GRPQ, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:52:19.042145380 (8h52m16.638916406s) [noise=true , address=64.49.13.139:4479 , piece_id=XUYHOCSBATYECYHWLVZ4VBCMT3H4O5ETSQ5HZOB42OR3OY4LWJ6Q, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:52:21.314990180 (8h52m18.911761213s) [noise=true , address=167.17.72.71:4810 , piece_id=ANLHBMENEXSGTHBHLOIHMLM6JGHKIU2X6FPQQIUIVMVUOABNK73Q, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:53:18.344002671 (8h53m15.9407737s) [noise=true , address=45.91.116.10:4096 , piece_id=EIKWFKPE5BYSZGV5T4RPQED55PG2FXU3ZKSNRVCBD363YQDHS4BQ, seg_pos={0 0}, upload=9902]: Done uploading piece. err: ecclient: upload failed (node:12MZX1GwZGrEi2q44dPiYrwMoum5A2caZtXTS3qiLNcoyHhon7p, address:45.91.116.10:4096): protocol: write tcp 159.223.143.39:44996->45.91.116.10:4096: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 6 19:53:18.344173472 (8h53m15.940944497s) [noise=true , address=45.91.116.7:4037 , piece_id=YXDMXUBE7CEWLJZP6FPDCRVIDGG5CJ3Y3IKEKO7EE2DQUSG6W5RA, seg_pos={0 0}, upload=9902]: Done uploading piece. err: ecclient: upload failed (node:1iNPwE2XSfXcaXAXcrH2jcyNGYeDiiWvzAcDQffrmjPwTVqVmK, address:45.91.116.7:4037): protocol: write tcp 159.223.143.39:52854->45.91.116.7:4037: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 6 19:53:18.344535130 (8h53m15.941306156s) [noise=true , address=45.91.116.14:4199 , piece_id=EKGDG7LYV3O5NSDL3UPWFKXINYNAOSPYYOWQPTFN326AJ2ZXAXHQ, seg_pos={0 0}, upload=9902]: Done uploading piece. err: ecclient: upload failed (node:1d4B1KMLs2annUuJ3v81FJapS9Yg4omBTj48eWoi6HJxnSEjrf, address:45.91.116.14:4199): protocol: write tcp 159.223.143.39:57596->45.91.116.14:4199: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 6 19:53:18.344687364 (8h53m15.941458387s) [noise=true , address=45.91.116.11:4133 , piece_id=RZWV5JMLOJ36AH7KSDXVTL7CMHI4INEW6YOOQVMCIIC2NGS3XYJA, seg_pos={0 0}, upload=9902]: Done uploading piece. err: ecclient: upload failed (node:12e1wX7hiER8sX4MLm5qLue4mayiN4SxXFdZSxfymRaRZxRGU6E, address:45.91.116.11:4133): protocol: write tcp 159.223.143.39:56440->45.91.116.11:4133: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 6 19:53:18.345220236 (8h53m15.941991259s) [noise=true , address=45.91.116.8:4066 , piece_id=5MZHIJ3ELVBAJ7ABRKG565TLU4A4NKB7X6WAACY2JBSLACL5LGVQ, seg_pos={0 0}, upload=9902]: Done uploading piece. err: ecclient: upload failed (node:123uv49k8LraXJfYQg6KRQthrYs1Egc4c3eZWm5JPvDatFbykm8, address:45.91.116.8:4066): protocol: write tcp 159.223.143.39:44136->45.91.116.8:4066: use of closed network connection; context deadline exceeded Mar 6 19:53:18.345277850 (8h53m15.942048865s) [seg_pos={0 0}, upload=9902]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 6 19:53:18.367560006 (8h53m15.964331035s) [upload=9902]: Done uploading segments... Mar 6 19:53:18.367606646 (8h53m15.964377681s) [upload=9902]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 1.193660129s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......X....................................................] duration: 855.999111ms trace: https://jaeger.dev.storj.io/trace/78791d41d75e16a0 FAIL (2026-03-06 20:28:43 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 19:52:18.382132233 (8h52m15.978903242s) [upload=9902]: Uploading segments... Mar 6 19:52:18.382227489 (8h52m15.978998515s) [upload=9902]: Got next segment. Inline: false Mar 6 19:52:18.382313252 (8h52m15.979084279s) [upload=9902]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 6 19:52:18.382319269 (8h52m15.979090296s) [upload=9902]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 6 19:52:18.411471452 (8h52m16.008242482s) [seg_pos={0 0}, upload=9902]: Begin upload segment... Mar 6 19:52:18.411537096 (8h52m16.008308109s) [seg_pos={0 0}, upload=9902]: Done begin upload segment. Mar 6 19:52:18.411564540 (8h52m16.008335586s) [noise=true , address=45.91.116.7:4037 , piece_id=YXDMXUBE7CEWLJZP6FPDCRVIDGG5CJ3Y3IKEKO7EE2DQUSG6W5RA, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.411626862 (8h52m16.008397897s) [noise=true , address=193.149.164.67:4018 , piece_id=GTQQP355TZZF4W7PPBYZS6ZQ6WYK7I3QIE4DZC7STWAKSBAMT4BQ, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.412356605 (8h52m16.009127631s) [noise=true , address=167.17.72.70:4771 , piece_id=TDLPTUHPNXCICPPXRTRL7WCJSXPX22ONIWWPHNXFUNIKRYPYW52Q, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.412900389 (8h52m16.009671415s) [noise=true , address=145.79.227.39:4022 , piece_id=U5GX3GLVWGWLU2BDWTHM7ST3MLLJDEAERWU3UEHO7Z2RXHGZU2DA, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.413467061 (8h52m16.010238103s) [noise=true , address=45.91.116.11:4133 , piece_id=RZWV5JMLOJ36AH7KSDXVTL7CMHI4INEW6YOOQVMCIIC2NGS3XYJA, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.413653431 (8h52m16.010424457s) [noise=true , address=193.149.164.69:4101 , piece_id=BTIBVPSORFJQR3CLA5UV62NXNI6GD3JF4D5Z74KU4R4CXB3WHHKA, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.414227965 (8h52m16.010998999s) [noise=true , address=45.91.116.14:4199 , piece_id=EKGDG7LYV3O5NSDL3UPWFKXINYNAOSPYYOWQPTFN326AJ2ZXAXHQ, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.415177146 (8h52m16.01194818s) [noise=true , address=77.90.31.16:4763 , piece_id=5Q7H3CU3MWSFPQGC67XMLCMTSUIEJQUV7QTO4UZLZY4MIVM76Z6Q, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.415227889 (8h52m16.011998912s) [noise=true , address=45.91.116.10:4096 , piece_id=EIKWFKPE5BYSZGV5T4RPQED55PG2FXU3ZKSNRVCBD363YQDHS4BQ, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.415728547 (8h52m16.012499581s) [noise=true , address=77.90.31.14:4693 , piece_id=S7NAS2ZBHAQPZRHBRGRSZPLE6COQEEN6SZVMB7X2X6IMXTN5HCBQ, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.415910147 (8h52m16.012681174s) [noise=true , address=193.149.164.70:4129 , piece_id=V7WMI7RDYDOAEQYWF2OTZL5LRPUA2S3V5WZN4BKDY6BI4YV3GRPQ, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.416592462 (8h52m16.01336349s) [noise=true , address=77.90.31.18:4831 , piece_id=4PMIUNE4YXKLIJPML3VSUN4PF32TSLPOOSDZZNAXFWH7WICBOW3Q, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.417118897 (8h52m16.01388993s) [noise=true , address=77.90.31.13:4655 , piece_id=47Q27QI7O2IYW7YQKFMJGGMWPVZWWKQTFVYPUMPUI7ORE7236XIQ, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.417185844 (8h52m16.013956873s) [noise=true , address=193.149.164.71:4304 , piece_id=DTCL54GM76GWKAE7A3X5PUNFYO7H5JIKRT5WBEHYEDF2YNFYTOAA, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.417686790 (8h52m16.014457823s) [noise=true , address=167.17.72.71:4810 , piece_id=ANLHBMENEXSGTHBHLOIHMLM6JGHKIU2X6FPQQIUIVMVUOABNK73Q, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.418236951 (8h52m16.015007982s) [noise=true , address=145.79.227.17:4994 , piece_id=TO3EOAILO4PHF6Z7WQA4RWZNNL3JO6EXXEF2RLW7IHAGWUU2TF4Q, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.419293640 (8h52m16.016064672s) [noise=true , address=64.49.13.139:4479 , piece_id=XUYHOCSBATYECYHWLVZ4VBCMT3H4O5ETSQ5HZOB42OR3OY4LWJ6Q, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.420005502 (8h52m16.016776534s) [noise=true , address=45.91.116.8:4066 , piece_id=5MZHIJ3ELVBAJ7ABRKG565TLU4A4NKB7X6WAACY2JBSLACL5LGVQ, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.483654433 (8h52m16.080425441s) [upload=9902]: Next returned nil segment Mar 6 19:52:18.483671157 (8h52m16.08044219s) [upload=9902]: Waiting for error group managing segments... Mar 6 19:52:18.488078321 (8h52m16.084849398s) [noise=true , address=77.90.31.13:4655 , piece_id=47Q27QI7O2IYW7YQKFMJGGMWPVZWWKQTFVYPUMPUI7ORE7236XIQ, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:52:18.498435619 (8h52m16.095206647s) [noise=true , address=145.79.227.17:4994 , piece_id=TO3EOAILO4PHF6Z7WQA4RWZNNL3JO6EXXEF2RLW7IHAGWUU2TF4Q, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:52:18.502979814 (8h52m16.099750841s) [noise=true , address=77.90.31.16:4763 , piece_id=5Q7H3CU3MWSFPQGC67XMLCMTSUIEJQUV7QTO4UZLZY4MIVM76Z6Q, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:52:18.506259140 (8h52m16.103030166s) [noise=true , address=77.90.31.14:4693 , piece_id=S7NAS2ZBHAQPZRHBRGRSZPLE6COQEEN6SZVMB7X2X6IMXTN5HCBQ, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:52:18.514529389 (8h52m16.111300413s) [noise=true , address=145.79.227.39:4022 , piece_id=U5GX3GLVWGWLU2BDWTHM7ST3MLLJDEAERWU3UEHO7Z2RXHGZU2DA, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:52:18.785925494 (8h52m16.382696517s) [noise=true , address=167.17.72.70:4771 , piece_id=TDLPTUHPNXCICPPXRTRL7WCJSXPX22ONIWWPHNXFUNIKRYPYW52Q, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:52:18.810774903 (8h52m16.407545926s) [noise=true , address=77.90.31.18:4831 , piece_id=4PMIUNE4YXKLIJPML3VSUN4PF32TSLPOOSDZZNAXFWH7WICBOW3Q, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:52:19.002746550 (8h52m16.5995176s) [noise=true , address=193.149.164.71:4304 , piece_id=DTCL54GM76GWKAE7A3X5PUNFYO7H5JIKRT5WBEHYEDF2YNFYTOAA, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:52:19.007614901 (8h52m16.604385924s) [noise=true , address=193.149.164.67:4018 , piece_id=GTQQP355TZZF4W7PPBYZS6ZQ6WYK7I3QIE4DZC7STWAKSBAMT4BQ, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:52:19.010443151 (8h52m16.607214176s) [noise=true , address=193.149.164.69:4101 , piece_id=BTIBVPSORFJQR3CLA5UV62NXNI6GD3JF4D5Z74KU4R4CXB3WHHKA, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:52:19.012614014 (8h52m16.609385038s) [noise=true , address=193.149.164.70:4129 , piece_id=V7WMI7RDYDOAEQYWF2OTZL5LRPUA2S3V5WZN4BKDY6BI4YV3GRPQ, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:52:19.042145380 (8h52m16.638916406s) [noise=true , address=64.49.13.139:4479 , piece_id=XUYHOCSBATYECYHWLVZ4VBCMT3H4O5ETSQ5HZOB42OR3OY4LWJ6Q, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:52:21.314990180 (8h52m18.911761213s) [noise=true , address=167.17.72.71:4810 , piece_id=ANLHBMENEXSGTHBHLOIHMLM6JGHKIU2X6FPQQIUIVMVUOABNK73Q, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:53:18.344002671 (8h53m15.9407737s) [noise=true , address=45.91.116.10:4096 , piece_id=EIKWFKPE5BYSZGV5T4RPQED55PG2FXU3ZKSNRVCBD363YQDHS4BQ, seg_pos={0 0}, upload=9902]: Done uploading piece. err: ecclient: upload failed (node:12MZX1GwZGrEi2q44dPiYrwMoum5A2caZtXTS3qiLNcoyHhon7p, address:45.91.116.10:4096): protocol: write tcp 159.223.143.39:44996->45.91.116.10:4096: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 6 19:53:18.344173472 (8h53m15.940944497s) [noise=true , address=45.91.116.7:4037 , piece_id=YXDMXUBE7CEWLJZP6FPDCRVIDGG5CJ3Y3IKEKO7EE2DQUSG6W5RA, seg_pos={0 0}, upload=9902]: Done uploading piece. err: ecclient: upload failed (node:1iNPwE2XSfXcaXAXcrH2jcyNGYeDiiWvzAcDQffrmjPwTVqVmK, address:45.91.116.7:4037): protocol: write tcp 159.223.143.39:52854->45.91.116.7:4037: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 6 19:53:18.344535130 (8h53m15.941306156s) [noise=true , address=45.91.116.14:4199 , piece_id=EKGDG7LYV3O5NSDL3UPWFKXINYNAOSPYYOWQPTFN326AJ2ZXAXHQ, seg_pos={0 0}, upload=9902]: Done uploading piece. err: ecclient: upload failed (node:1d4B1KMLs2annUuJ3v81FJapS9Yg4omBTj48eWoi6HJxnSEjrf, address:45.91.116.14:4199): protocol: write tcp 159.223.143.39:57596->45.91.116.14:4199: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 6 19:53:18.344687364 (8h53m15.941458387s) [noise=true , address=45.91.116.11:4133 , piece_id=RZWV5JMLOJ36AH7KSDXVTL7CMHI4INEW6YOOQVMCIIC2NGS3XYJA, seg_pos={0 0}, upload=9902]: Done uploading piece. err: ecclient: upload failed (node:12e1wX7hiER8sX4MLm5qLue4mayiN4SxXFdZSxfymRaRZxRGU6E, address:45.91.116.11:4133): protocol: write tcp 159.223.143.39:56440->45.91.116.11:4133: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 6 19:53:18.345220236 (8h53m15.941991259s) [noise=true , address=45.91.116.8:4066 , piece_id=5MZHIJ3ELVBAJ7ABRKG565TLU4A4NKB7X6WAACY2JBSLACL5LGVQ, seg_pos={0 0}, upload=9902]: Done uploading piece. err: ecclient: upload failed (node:123uv49k8LraXJfYQg6KRQthrYs1Egc4c3eZWm5JPvDatFbykm8, address:45.91.116.8:4066): protocol: write tcp 159.223.143.39:44136->45.91.116.8:4066: use of closed network connection; context deadline exceeded Mar 6 19:53:18.345277850 (8h53m15.942048865s) [seg_pos={0 0}, upload=9902]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 6 19:53:18.367560006 (8h53m15.964331035s) [upload=9902]: Done uploading segments... Mar 6 19:53:18.367606646 (8h53m15.964377681s) [upload=9902]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 856.023756ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [........X...................................................] duration: 868.99415ms trace: https://jaeger.dev.storj.io/trace/78791d41d75e16a0 FAIL (2026-03-06 20:28:10 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 19:52:18.382132233 (8h52m15.978903242s) [upload=9902]: Uploading segments... Mar 6 19:52:18.382227489 (8h52m15.978998515s) [upload=9902]: Got next segment. Inline: false Mar 6 19:52:18.382313252 (8h52m15.979084279s) [upload=9902]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 6 19:52:18.382319269 (8h52m15.979090296s) [upload=9902]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 6 19:52:18.411471452 (8h52m16.008242482s) [seg_pos={0 0}, upload=9902]: Begin upload segment... Mar 6 19:52:18.411537096 (8h52m16.008308109s) [seg_pos={0 0}, upload=9902]: Done begin upload segment. Mar 6 19:52:18.411564540 (8h52m16.008335586s) [noise=true , address=45.91.116.7:4037 , piece_id=YXDMXUBE7CEWLJZP6FPDCRVIDGG5CJ3Y3IKEKO7EE2DQUSG6W5RA, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.411626862 (8h52m16.008397897s) [noise=true , address=193.149.164.67:4018 , piece_id=GTQQP355TZZF4W7PPBYZS6ZQ6WYK7I3QIE4DZC7STWAKSBAMT4BQ, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.412356605 (8h52m16.009127631s) [noise=true , address=167.17.72.70:4771 , piece_id=TDLPTUHPNXCICPPXRTRL7WCJSXPX22ONIWWPHNXFUNIKRYPYW52Q, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.412900389 (8h52m16.009671415s) [noise=true , address=145.79.227.39:4022 , piece_id=U5GX3GLVWGWLU2BDWTHM7ST3MLLJDEAERWU3UEHO7Z2RXHGZU2DA, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.413467061 (8h52m16.010238103s) [noise=true , address=45.91.116.11:4133 , piece_id=RZWV5JMLOJ36AH7KSDXVTL7CMHI4INEW6YOOQVMCIIC2NGS3XYJA, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.413653431 (8h52m16.010424457s) [noise=true , address=193.149.164.69:4101 , piece_id=BTIBVPSORFJQR3CLA5UV62NXNI6GD3JF4D5Z74KU4R4CXB3WHHKA, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.414227965 (8h52m16.010998999s) [noise=true , address=45.91.116.14:4199 , piece_id=EKGDG7LYV3O5NSDL3UPWFKXINYNAOSPYYOWQPTFN326AJ2ZXAXHQ, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.415177146 (8h52m16.01194818s) [noise=true , address=77.90.31.16:4763 , piece_id=5Q7H3CU3MWSFPQGC67XMLCMTSUIEJQUV7QTO4UZLZY4MIVM76Z6Q, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.415227889 (8h52m16.011998912s) [noise=true , address=45.91.116.10:4096 , piece_id=EIKWFKPE5BYSZGV5T4RPQED55PG2FXU3ZKSNRVCBD363YQDHS4BQ, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.415728547 (8h52m16.012499581s) [noise=true , address=77.90.31.14:4693 , piece_id=S7NAS2ZBHAQPZRHBRGRSZPLE6COQEEN6SZVMB7X2X6IMXTN5HCBQ, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.415910147 (8h52m16.012681174s) [noise=true , address=193.149.164.70:4129 , piece_id=V7WMI7RDYDOAEQYWF2OTZL5LRPUA2S3V5WZN4BKDY6BI4YV3GRPQ, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.416592462 (8h52m16.01336349s) [noise=true , address=77.90.31.18:4831 , piece_id=4PMIUNE4YXKLIJPML3VSUN4PF32TSLPOOSDZZNAXFWH7WICBOW3Q, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.417118897 (8h52m16.01388993s) [noise=true , address=77.90.31.13:4655 , piece_id=47Q27QI7O2IYW7YQKFMJGGMWPVZWWKQTFVYPUMPUI7ORE7236XIQ, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.417185844 (8h52m16.013956873s) [noise=true , address=193.149.164.71:4304 , piece_id=DTCL54GM76GWKAE7A3X5PUNFYO7H5JIKRT5WBEHYEDF2YNFYTOAA, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.417686790 (8h52m16.014457823s) [noise=true , address=167.17.72.71:4810 , piece_id=ANLHBMENEXSGTHBHLOIHMLM6JGHKIU2X6FPQQIUIVMVUOABNK73Q, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.418236951 (8h52m16.015007982s) [noise=true , address=145.79.227.17:4994 , piece_id=TO3EOAILO4PHF6Z7WQA4RWZNNL3JO6EXXEF2RLW7IHAGWUU2TF4Q, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.419293640 (8h52m16.016064672s) [noise=true , address=64.49.13.139:4479 , piece_id=XUYHOCSBATYECYHWLVZ4VBCMT3H4O5ETSQ5HZOB42OR3OY4LWJ6Q, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.420005502 (8h52m16.016776534s) [noise=true , address=45.91.116.8:4066 , piece_id=5MZHIJ3ELVBAJ7ABRKG565TLU4A4NKB7X6WAACY2JBSLACL5LGVQ, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.483654433 (8h52m16.080425441s) [upload=9902]: Next returned nil segment Mar 6 19:52:18.483671157 (8h52m16.08044219s) [upload=9902]: Waiting for error group managing segments... Mar 6 19:52:18.488078321 (8h52m16.084849398s) [noise=true , address=77.90.31.13:4655 , piece_id=47Q27QI7O2IYW7YQKFMJGGMWPVZWWKQTFVYPUMPUI7ORE7236XIQ, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:52:18.498435619 (8h52m16.095206647s) [noise=true , address=145.79.227.17:4994 , piece_id=TO3EOAILO4PHF6Z7WQA4RWZNNL3JO6EXXEF2RLW7IHAGWUU2TF4Q, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:52:18.502979814 (8h52m16.099750841s) [noise=true , address=77.90.31.16:4763 , piece_id=5Q7H3CU3MWSFPQGC67XMLCMTSUIEJQUV7QTO4UZLZY4MIVM76Z6Q, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:52:18.506259140 (8h52m16.103030166s) [noise=true , address=77.90.31.14:4693 , piece_id=S7NAS2ZBHAQPZRHBRGRSZPLE6COQEEN6SZVMB7X2X6IMXTN5HCBQ, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:52:18.514529389 (8h52m16.111300413s) [noise=true , address=145.79.227.39:4022 , piece_id=U5GX3GLVWGWLU2BDWTHM7ST3MLLJDEAERWU3UEHO7Z2RXHGZU2DA, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:52:18.785925494 (8h52m16.382696517s) [noise=true , address=167.17.72.70:4771 , piece_id=TDLPTUHPNXCICPPXRTRL7WCJSXPX22ONIWWPHNXFUNIKRYPYW52Q, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:52:18.810774903 (8h52m16.407545926s) [noise=true , address=77.90.31.18:4831 , piece_id=4PMIUNE4YXKLIJPML3VSUN4PF32TSLPOOSDZZNAXFWH7WICBOW3Q, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:52:19.002746550 (8h52m16.5995176s) [noise=true , address=193.149.164.71:4304 , piece_id=DTCL54GM76GWKAE7A3X5PUNFYO7H5JIKRT5WBEHYEDF2YNFYTOAA, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:52:19.007614901 (8h52m16.604385924s) [noise=true , address=193.149.164.67:4018 , piece_id=GTQQP355TZZF4W7PPBYZS6ZQ6WYK7I3QIE4DZC7STWAKSBAMT4BQ, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:52:19.010443151 (8h52m16.607214176s) [noise=true , address=193.149.164.69:4101 , piece_id=BTIBVPSORFJQR3CLA5UV62NXNI6GD3JF4D5Z74KU4R4CXB3WHHKA, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:52:19.012614014 (8h52m16.609385038s) [noise=true , address=193.149.164.70:4129 , piece_id=V7WMI7RDYDOAEQYWF2OTZL5LRPUA2S3V5WZN4BKDY6BI4YV3GRPQ, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:52:19.042145380 (8h52m16.638916406s) [noise=true , address=64.49.13.139:4479 , piece_id=XUYHOCSBATYECYHWLVZ4VBCMT3H4O5ETSQ5HZOB42OR3OY4LWJ6Q, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:52:21.314990180 (8h52m18.911761213s) [noise=true , address=167.17.72.71:4810 , piece_id=ANLHBMENEXSGTHBHLOIHMLM6JGHKIU2X6FPQQIUIVMVUOABNK73Q, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:53:18.344002671 (8h53m15.9407737s) [noise=true , address=45.91.116.10:4096 , piece_id=EIKWFKPE5BYSZGV5T4RPQED55PG2FXU3ZKSNRVCBD363YQDHS4BQ, seg_pos={0 0}, upload=9902]: Done uploading piece. err: ecclient: upload failed (node:12MZX1GwZGrEi2q44dPiYrwMoum5A2caZtXTS3qiLNcoyHhon7p, address:45.91.116.10:4096): protocol: write tcp 159.223.143.39:44996->45.91.116.10:4096: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 6 19:53:18.344173472 (8h53m15.940944497s) [noise=true , address=45.91.116.7:4037 , piece_id=YXDMXUBE7CEWLJZP6FPDCRVIDGG5CJ3Y3IKEKO7EE2DQUSG6W5RA, seg_pos={0 0}, upload=9902]: Done uploading piece. err: ecclient: upload failed (node:1iNPwE2XSfXcaXAXcrH2jcyNGYeDiiWvzAcDQffrmjPwTVqVmK, address:45.91.116.7:4037): protocol: write tcp 159.223.143.39:52854->45.91.116.7:4037: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 6 19:53:18.344535130 (8h53m15.941306156s) [noise=true , address=45.91.116.14:4199 , piece_id=EKGDG7LYV3O5NSDL3UPWFKXINYNAOSPYYOWQPTFN326AJ2ZXAXHQ, seg_pos={0 0}, upload=9902]: Done uploading piece. err: ecclient: upload failed (node:1d4B1KMLs2annUuJ3v81FJapS9Yg4omBTj48eWoi6HJxnSEjrf, address:45.91.116.14:4199): protocol: write tcp 159.223.143.39:57596->45.91.116.14:4199: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 6 19:53:18.344687364 (8h53m15.941458387s) [noise=true , address=45.91.116.11:4133 , piece_id=RZWV5JMLOJ36AH7KSDXVTL7CMHI4INEW6YOOQVMCIIC2NGS3XYJA, seg_pos={0 0}, upload=9902]: Done uploading piece. err: ecclient: upload failed (node:12e1wX7hiER8sX4MLm5qLue4mayiN4SxXFdZSxfymRaRZxRGU6E, address:45.91.116.11:4133): protocol: write tcp 159.223.143.39:56440->45.91.116.11:4133: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 6 19:53:18.345220236 (8h53m15.941991259s) [noise=true , address=45.91.116.8:4066 , piece_id=5MZHIJ3ELVBAJ7ABRKG565TLU4A4NKB7X6WAACY2JBSLACL5LGVQ, seg_pos={0 0}, upload=9902]: Done uploading piece. err: ecclient: upload failed (node:123uv49k8LraXJfYQg6KRQthrYs1Egc4c3eZWm5JPvDatFbykm8, address:45.91.116.8:4066): protocol: write tcp 159.223.143.39:44136->45.91.116.8:4066: use of closed network connection; context deadline exceeded Mar 6 19:53:18.345277850 (8h53m15.942048865s) [seg_pos={0 0}, upload=9902]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 6 19:53:18.367560006 (8h53m15.964331035s) [upload=9902]: Done uploading segments... Mar 6 19:53:18.367606646 (8h53m15.964377681s) [upload=9902]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 869.019503ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........X..................................................] duration: 1.316194649s trace: https://jaeger.dev.storj.io/trace/78791d41d75e16a0 FAIL (2026-03-06 20:27:00 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 19:52:18.382132233 (8h52m15.978903242s) [upload=9902]: Uploading segments... Mar 6 19:52:18.382227489 (8h52m15.978998515s) [upload=9902]: Got next segment. Inline: false Mar 6 19:52:18.382313252 (8h52m15.979084279s) [upload=9902]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 6 19:52:18.382319269 (8h52m15.979090296s) [upload=9902]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 6 19:52:18.411471452 (8h52m16.008242482s) [seg_pos={0 0}, upload=9902]: Begin upload segment... Mar 6 19:52:18.411537096 (8h52m16.008308109s) [seg_pos={0 0}, upload=9902]: Done begin upload segment. Mar 6 19:52:18.411564540 (8h52m16.008335586s) [noise=true , address=45.91.116.7:4037 , piece_id=YXDMXUBE7CEWLJZP6FPDCRVIDGG5CJ3Y3IKEKO7EE2DQUSG6W5RA, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.411626862 (8h52m16.008397897s) [noise=true , address=193.149.164.67:4018 , piece_id=GTQQP355TZZF4W7PPBYZS6ZQ6WYK7I3QIE4DZC7STWAKSBAMT4BQ, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.412356605 (8h52m16.009127631s) [noise=true , address=167.17.72.70:4771 , piece_id=TDLPTUHPNXCICPPXRTRL7WCJSXPX22ONIWWPHNXFUNIKRYPYW52Q, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.412900389 (8h52m16.009671415s) [noise=true , address=145.79.227.39:4022 , piece_id=U5GX3GLVWGWLU2BDWTHM7ST3MLLJDEAERWU3UEHO7Z2RXHGZU2DA, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.413467061 (8h52m16.010238103s) [noise=true , address=45.91.116.11:4133 , piece_id=RZWV5JMLOJ36AH7KSDXVTL7CMHI4INEW6YOOQVMCIIC2NGS3XYJA, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.413653431 (8h52m16.010424457s) [noise=true , address=193.149.164.69:4101 , piece_id=BTIBVPSORFJQR3CLA5UV62NXNI6GD3JF4D5Z74KU4R4CXB3WHHKA, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.414227965 (8h52m16.010998999s) [noise=true , address=45.91.116.14:4199 , piece_id=EKGDG7LYV3O5NSDL3UPWFKXINYNAOSPYYOWQPTFN326AJ2ZXAXHQ, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.415177146 (8h52m16.01194818s) [noise=true , address=77.90.31.16:4763 , piece_id=5Q7H3CU3MWSFPQGC67XMLCMTSUIEJQUV7QTO4UZLZY4MIVM76Z6Q, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.415227889 (8h52m16.011998912s) [noise=true , address=45.91.116.10:4096 , piece_id=EIKWFKPE5BYSZGV5T4RPQED55PG2FXU3ZKSNRVCBD363YQDHS4BQ, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.415728547 (8h52m16.012499581s) [noise=true , address=77.90.31.14:4693 , piece_id=S7NAS2ZBHAQPZRHBRGRSZPLE6COQEEN6SZVMB7X2X6IMXTN5HCBQ, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.415910147 (8h52m16.012681174s) [noise=true , address=193.149.164.70:4129 , piece_id=V7WMI7RDYDOAEQYWF2OTZL5LRPUA2S3V5WZN4BKDY6BI4YV3GRPQ, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.416592462 (8h52m16.01336349s) [noise=true , address=77.90.31.18:4831 , piece_id=4PMIUNE4YXKLIJPML3VSUN4PF32TSLPOOSDZZNAXFWH7WICBOW3Q, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.417118897 (8h52m16.01388993s) [noise=true , address=77.90.31.13:4655 , piece_id=47Q27QI7O2IYW7YQKFMJGGMWPVZWWKQTFVYPUMPUI7ORE7236XIQ, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.417185844 (8h52m16.013956873s) [noise=true , address=193.149.164.71:4304 , piece_id=DTCL54GM76GWKAE7A3X5PUNFYO7H5JIKRT5WBEHYEDF2YNFYTOAA, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.417686790 (8h52m16.014457823s) [noise=true , address=167.17.72.71:4810 , piece_id=ANLHBMENEXSGTHBHLOIHMLM6JGHKIU2X6FPQQIUIVMVUOABNK73Q, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.418236951 (8h52m16.015007982s) [noise=true , address=145.79.227.17:4994 , piece_id=TO3EOAILO4PHF6Z7WQA4RWZNNL3JO6EXXEF2RLW7IHAGWUU2TF4Q, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.419293640 (8h52m16.016064672s) [noise=true , address=64.49.13.139:4479 , piece_id=XUYHOCSBATYECYHWLVZ4VBCMT3H4O5ETSQ5HZOB42OR3OY4LWJ6Q, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.420005502 (8h52m16.016776534s) [noise=true , address=45.91.116.8:4066 , piece_id=5MZHIJ3ELVBAJ7ABRKG565TLU4A4NKB7X6WAACY2JBSLACL5LGVQ, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.483654433 (8h52m16.080425441s) [upload=9902]: Next returned nil segment Mar 6 19:52:18.483671157 (8h52m16.08044219s) [upload=9902]: Waiting for error group managing segments... Mar 6 19:52:18.488078321 (8h52m16.084849398s) [noise=true , address=77.90.31.13:4655 , piece_id=47Q27QI7O2IYW7YQKFMJGGMWPVZWWKQTFVYPUMPUI7ORE7236XIQ, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:52:18.498435619 (8h52m16.095206647s) [noise=true , address=145.79.227.17:4994 , piece_id=TO3EOAILO4PHF6Z7WQA4RWZNNL3JO6EXXEF2RLW7IHAGWUU2TF4Q, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:52:18.502979814 (8h52m16.099750841s) [noise=true , address=77.90.31.16:4763 , piece_id=5Q7H3CU3MWSFPQGC67XMLCMTSUIEJQUV7QTO4UZLZY4MIVM76Z6Q, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:52:18.506259140 (8h52m16.103030166s) [noise=true , address=77.90.31.14:4693 , piece_id=S7NAS2ZBHAQPZRHBRGRSZPLE6COQEEN6SZVMB7X2X6IMXTN5HCBQ, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:52:18.514529389 (8h52m16.111300413s) [noise=true , address=145.79.227.39:4022 , piece_id=U5GX3GLVWGWLU2BDWTHM7ST3MLLJDEAERWU3UEHO7Z2RXHGZU2DA, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:52:18.785925494 (8h52m16.382696517s) [noise=true , address=167.17.72.70:4771 , piece_id=TDLPTUHPNXCICPPXRTRL7WCJSXPX22ONIWWPHNXFUNIKRYPYW52Q, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:52:18.810774903 (8h52m16.407545926s) [noise=true , address=77.90.31.18:4831 , piece_id=4PMIUNE4YXKLIJPML3VSUN4PF32TSLPOOSDZZNAXFWH7WICBOW3Q, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:52:19.002746550 (8h52m16.5995176s) [noise=true , address=193.149.164.71:4304 , piece_id=DTCL54GM76GWKAE7A3X5PUNFYO7H5JIKRT5WBEHYEDF2YNFYTOAA, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:52:19.007614901 (8h52m16.604385924s) [noise=true , address=193.149.164.67:4018 , piece_id=GTQQP355TZZF4W7PPBYZS6ZQ6WYK7I3QIE4DZC7STWAKSBAMT4BQ, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:52:19.010443151 (8h52m16.607214176s) [noise=true , address=193.149.164.69:4101 , piece_id=BTIBVPSORFJQR3CLA5UV62NXNI6GD3JF4D5Z74KU4R4CXB3WHHKA, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:52:19.012614014 (8h52m16.609385038s) [noise=true , address=193.149.164.70:4129 , piece_id=V7WMI7RDYDOAEQYWF2OTZL5LRPUA2S3V5WZN4BKDY6BI4YV3GRPQ, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:52:19.042145380 (8h52m16.638916406s) [noise=true , address=64.49.13.139:4479 , piece_id=XUYHOCSBATYECYHWLVZ4VBCMT3H4O5ETSQ5HZOB42OR3OY4LWJ6Q, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:52:21.314990180 (8h52m18.911761213s) [noise=true , address=167.17.72.71:4810 , piece_id=ANLHBMENEXSGTHBHLOIHMLM6JGHKIU2X6FPQQIUIVMVUOABNK73Q, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:53:18.344002671 (8h53m15.9407737s) [noise=true , address=45.91.116.10:4096 , piece_id=EIKWFKPE5BYSZGV5T4RPQED55PG2FXU3ZKSNRVCBD363YQDHS4BQ, seg_pos={0 0}, upload=9902]: Done uploading piece. err: ecclient: upload failed (node:12MZX1GwZGrEi2q44dPiYrwMoum5A2caZtXTS3qiLNcoyHhon7p, address:45.91.116.10:4096): protocol: write tcp 159.223.143.39:44996->45.91.116.10:4096: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 6 19:53:18.344173472 (8h53m15.940944497s) [noise=true , address=45.91.116.7:4037 , piece_id=YXDMXUBE7CEWLJZP6FPDCRVIDGG5CJ3Y3IKEKO7EE2DQUSG6W5RA, seg_pos={0 0}, upload=9902]: Done uploading piece. err: ecclient: upload failed (node:1iNPwE2XSfXcaXAXcrH2jcyNGYeDiiWvzAcDQffrmjPwTVqVmK, address:45.91.116.7:4037): protocol: write tcp 159.223.143.39:52854->45.91.116.7:4037: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 6 19:53:18.344535130 (8h53m15.941306156s) [noise=true , address=45.91.116.14:4199 , piece_id=EKGDG7LYV3O5NSDL3UPWFKXINYNAOSPYYOWQPTFN326AJ2ZXAXHQ, seg_pos={0 0}, upload=9902]: Done uploading piece. err: ecclient: upload failed (node:1d4B1KMLs2annUuJ3v81FJapS9Yg4omBTj48eWoi6HJxnSEjrf, address:45.91.116.14:4199): protocol: write tcp 159.223.143.39:57596->45.91.116.14:4199: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 6 19:53:18.344687364 (8h53m15.941458387s) [noise=true , address=45.91.116.11:4133 , piece_id=RZWV5JMLOJ36AH7KSDXVTL7CMHI4INEW6YOOQVMCIIC2NGS3XYJA, seg_pos={0 0}, upload=9902]: Done uploading piece. err: ecclient: upload failed (node:12e1wX7hiER8sX4MLm5qLue4mayiN4SxXFdZSxfymRaRZxRGU6E, address:45.91.116.11:4133): protocol: write tcp 159.223.143.39:56440->45.91.116.11:4133: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 6 19:53:18.345220236 (8h53m15.941991259s) [noise=true , address=45.91.116.8:4066 , piece_id=5MZHIJ3ELVBAJ7ABRKG565TLU4A4NKB7X6WAACY2JBSLACL5LGVQ, seg_pos={0 0}, upload=9902]: Done uploading piece. err: ecclient: upload failed (node:123uv49k8LraXJfYQg6KRQthrYs1Egc4c3eZWm5JPvDatFbykm8, address:45.91.116.8:4066): protocol: write tcp 159.223.143.39:44136->45.91.116.8:4066: use of closed network connection; context deadline exceeded Mar 6 19:53:18.345277850 (8h53m15.942048865s) [seg_pos={0 0}, upload=9902]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 6 19:53:18.367560006 (8h53m15.964331035s) [upload=9902]: Done uploading segments... Mar 6 19:53:18.367606646 (8h53m15.964377681s) [upload=9902]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 1.316214473s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........X.................................................] duration: 1.207095571s trace: https://jaeger.dev.storj.io/trace/78791d41d75e16a0 FAIL (2026-03-06 20:26:03 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 19:52:18.382132233 (8h52m15.978903242s) [upload=9902]: Uploading segments... Mar 6 19:52:18.382227489 (8h52m15.978998515s) [upload=9902]: Got next segment. Inline: false Mar 6 19:52:18.382313252 (8h52m15.979084279s) [upload=9902]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 6 19:52:18.382319269 (8h52m15.979090296s) [upload=9902]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 6 19:52:18.411471452 (8h52m16.008242482s) [seg_pos={0 0}, upload=9902]: Begin upload segment... Mar 6 19:52:18.411537096 (8h52m16.008308109s) [seg_pos={0 0}, upload=9902]: Done begin upload segment. Mar 6 19:52:18.411564540 (8h52m16.008335586s) [noise=true , address=45.91.116.7:4037 , piece_id=YXDMXUBE7CEWLJZP6FPDCRVIDGG5CJ3Y3IKEKO7EE2DQUSG6W5RA, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.411626862 (8h52m16.008397897s) [noise=true , address=193.149.164.67:4018 , piece_id=GTQQP355TZZF4W7PPBYZS6ZQ6WYK7I3QIE4DZC7STWAKSBAMT4BQ, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.412356605 (8h52m16.009127631s) [noise=true , address=167.17.72.70:4771 , piece_id=TDLPTUHPNXCICPPXRTRL7WCJSXPX22ONIWWPHNXFUNIKRYPYW52Q, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.412900389 (8h52m16.009671415s) [noise=true , address=145.79.227.39:4022 , piece_id=U5GX3GLVWGWLU2BDWTHM7ST3MLLJDEAERWU3UEHO7Z2RXHGZU2DA, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.413467061 (8h52m16.010238103s) [noise=true , address=45.91.116.11:4133 , piece_id=RZWV5JMLOJ36AH7KSDXVTL7CMHI4INEW6YOOQVMCIIC2NGS3XYJA, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.413653431 (8h52m16.010424457s) [noise=true , address=193.149.164.69:4101 , piece_id=BTIBVPSORFJQR3CLA5UV62NXNI6GD3JF4D5Z74KU4R4CXB3WHHKA, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.414227965 (8h52m16.010998999s) [noise=true , address=45.91.116.14:4199 , piece_id=EKGDG7LYV3O5NSDL3UPWFKXINYNAOSPYYOWQPTFN326AJ2ZXAXHQ, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.415177146 (8h52m16.01194818s) [noise=true , address=77.90.31.16:4763 , piece_id=5Q7H3CU3MWSFPQGC67XMLCMTSUIEJQUV7QTO4UZLZY4MIVM76Z6Q, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.415227889 (8h52m16.011998912s) [noise=true , address=45.91.116.10:4096 , piece_id=EIKWFKPE5BYSZGV5T4RPQED55PG2FXU3ZKSNRVCBD363YQDHS4BQ, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.415728547 (8h52m16.012499581s) [noise=true , address=77.90.31.14:4693 , piece_id=S7NAS2ZBHAQPZRHBRGRSZPLE6COQEEN6SZVMB7X2X6IMXTN5HCBQ, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.415910147 (8h52m16.012681174s) [noise=true , address=193.149.164.70:4129 , piece_id=V7WMI7RDYDOAEQYWF2OTZL5LRPUA2S3V5WZN4BKDY6BI4YV3GRPQ, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.416592462 (8h52m16.01336349s) [noise=true , address=77.90.31.18:4831 , piece_id=4PMIUNE4YXKLIJPML3VSUN4PF32TSLPOOSDZZNAXFWH7WICBOW3Q, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.417118897 (8h52m16.01388993s) [noise=true , address=77.90.31.13:4655 , piece_id=47Q27QI7O2IYW7YQKFMJGGMWPVZWWKQTFVYPUMPUI7ORE7236XIQ, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.417185844 (8h52m16.013956873s) [noise=true , address=193.149.164.71:4304 , piece_id=DTCL54GM76GWKAE7A3X5PUNFYO7H5JIKRT5WBEHYEDF2YNFYTOAA, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.417686790 (8h52m16.014457823s) [noise=true , address=167.17.72.71:4810 , piece_id=ANLHBMENEXSGTHBHLOIHMLM6JGHKIU2X6FPQQIUIVMVUOABNK73Q, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.418236951 (8h52m16.015007982s) [noise=true , address=145.79.227.17:4994 , piece_id=TO3EOAILO4PHF6Z7WQA4RWZNNL3JO6EXXEF2RLW7IHAGWUU2TF4Q, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.419293640 (8h52m16.016064672s) [noise=true , address=64.49.13.139:4479 , piece_id=XUYHOCSBATYECYHWLVZ4VBCMT3H4O5ETSQ5HZOB42OR3OY4LWJ6Q, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.420005502 (8h52m16.016776534s) [noise=true , address=45.91.116.8:4066 , piece_id=5MZHIJ3ELVBAJ7ABRKG565TLU4A4NKB7X6WAACY2JBSLACL5LGVQ, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.483654433 (8h52m16.080425441s) [upload=9902]: Next returned nil segment Mar 6 19:52:18.483671157 (8h52m16.08044219s) [upload=9902]: Waiting for error group managing segments... Mar 6 19:52:18.488078321 (8h52m16.084849398s) [noise=true , address=77.90.31.13:4655 , piece_id=47Q27QI7O2IYW7YQKFMJGGMWPVZWWKQTFVYPUMPUI7ORE7236XIQ, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:52:18.498435619 (8h52m16.095206647s) [noise=true , address=145.79.227.17:4994 , piece_id=TO3EOAILO4PHF6Z7WQA4RWZNNL3JO6EXXEF2RLW7IHAGWUU2TF4Q, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:52:18.502979814 (8h52m16.099750841s) [noise=true , address=77.90.31.16:4763 , piece_id=5Q7H3CU3MWSFPQGC67XMLCMTSUIEJQUV7QTO4UZLZY4MIVM76Z6Q, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:52:18.506259140 (8h52m16.103030166s) [noise=true , address=77.90.31.14:4693 , piece_id=S7NAS2ZBHAQPZRHBRGRSZPLE6COQEEN6SZVMB7X2X6IMXTN5HCBQ, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:52:18.514529389 (8h52m16.111300413s) [noise=true , address=145.79.227.39:4022 , piece_id=U5GX3GLVWGWLU2BDWTHM7ST3MLLJDEAERWU3UEHO7Z2RXHGZU2DA, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:52:18.785925494 (8h52m16.382696517s) [noise=true , address=167.17.72.70:4771 , piece_id=TDLPTUHPNXCICPPXRTRL7WCJSXPX22ONIWWPHNXFUNIKRYPYW52Q, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:52:18.810774903 (8h52m16.407545926s) [noise=true , address=77.90.31.18:4831 , piece_id=4PMIUNE4YXKLIJPML3VSUN4PF32TSLPOOSDZZNAXFWH7WICBOW3Q, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:52:19.002746550 (8h52m16.5995176s) [noise=true , address=193.149.164.71:4304 , piece_id=DTCL54GM76GWKAE7A3X5PUNFYO7H5JIKRT5WBEHYEDF2YNFYTOAA, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:52:19.007614901 (8h52m16.604385924s) [noise=true , address=193.149.164.67:4018 , piece_id=GTQQP355TZZF4W7PPBYZS6ZQ6WYK7I3QIE4DZC7STWAKSBAMT4BQ, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:52:19.010443151 (8h52m16.607214176s) [noise=true , address=193.149.164.69:4101 , piece_id=BTIBVPSORFJQR3CLA5UV62NXNI6GD3JF4D5Z74KU4R4CXB3WHHKA, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:52:19.012614014 (8h52m16.609385038s) [noise=true , address=193.149.164.70:4129 , piece_id=V7WMI7RDYDOAEQYWF2OTZL5LRPUA2S3V5WZN4BKDY6BI4YV3GRPQ, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:52:19.042145380 (8h52m16.638916406s) [noise=true , address=64.49.13.139:4479 , piece_id=XUYHOCSBATYECYHWLVZ4VBCMT3H4O5ETSQ5HZOB42OR3OY4LWJ6Q, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:52:21.314990180 (8h52m18.911761213s) [noise=true , address=167.17.72.71:4810 , piece_id=ANLHBMENEXSGTHBHLOIHMLM6JGHKIU2X6FPQQIUIVMVUOABNK73Q, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:53:18.344002671 (8h53m15.9407737s) [noise=true , address=45.91.116.10:4096 , piece_id=EIKWFKPE5BYSZGV5T4RPQED55PG2FXU3ZKSNRVCBD363YQDHS4BQ, seg_pos={0 0}, upload=9902]: Done uploading piece. err: ecclient: upload failed (node:12MZX1GwZGrEi2q44dPiYrwMoum5A2caZtXTS3qiLNcoyHhon7p, address:45.91.116.10:4096): protocol: write tcp 159.223.143.39:44996->45.91.116.10:4096: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 6 19:53:18.344173472 (8h53m15.940944497s) [noise=true , address=45.91.116.7:4037 , piece_id=YXDMXUBE7CEWLJZP6FPDCRVIDGG5CJ3Y3IKEKO7EE2DQUSG6W5RA, seg_pos={0 0}, upload=9902]: Done uploading piece. err: ecclient: upload failed (node:1iNPwE2XSfXcaXAXcrH2jcyNGYeDiiWvzAcDQffrmjPwTVqVmK, address:45.91.116.7:4037): protocol: write tcp 159.223.143.39:52854->45.91.116.7:4037: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 6 19:53:18.344535130 (8h53m15.941306156s) [noise=true , address=45.91.116.14:4199 , piece_id=EKGDG7LYV3O5NSDL3UPWFKXINYNAOSPYYOWQPTFN326AJ2ZXAXHQ, seg_pos={0 0}, upload=9902]: Done uploading piece. err: ecclient: upload failed (node:1d4B1KMLs2annUuJ3v81FJapS9Yg4omBTj48eWoi6HJxnSEjrf, address:45.91.116.14:4199): protocol: write tcp 159.223.143.39:57596->45.91.116.14:4199: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 6 19:53:18.344687364 (8h53m15.941458387s) [noise=true , address=45.91.116.11:4133 , piece_id=RZWV5JMLOJ36AH7KSDXVTL7CMHI4INEW6YOOQVMCIIC2NGS3XYJA, seg_pos={0 0}, upload=9902]: Done uploading piece. err: ecclient: upload failed (node:12e1wX7hiER8sX4MLm5qLue4mayiN4SxXFdZSxfymRaRZxRGU6E, address:45.91.116.11:4133): protocol: write tcp 159.223.143.39:56440->45.91.116.11:4133: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 6 19:53:18.345220236 (8h53m15.941991259s) [noise=true , address=45.91.116.8:4066 , piece_id=5MZHIJ3ELVBAJ7ABRKG565TLU4A4NKB7X6WAACY2JBSLACL5LGVQ, seg_pos={0 0}, upload=9902]: Done uploading piece. err: ecclient: upload failed (node:123uv49k8LraXJfYQg6KRQthrYs1Egc4c3eZWm5JPvDatFbykm8, address:45.91.116.8:4066): protocol: write tcp 159.223.143.39:44136->45.91.116.8:4066: use of closed network connection; context deadline exceeded Mar 6 19:53:18.345277850 (8h53m15.942048865s) [seg_pos={0 0}, upload=9902]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 6 19:53:18.367560006 (8h53m15.964331035s) [upload=9902]: Done uploading segments... Mar 6 19:53:18.367606646 (8h53m15.964377681s) [upload=9902]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 1.207125261s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........X................................................] duration: 1.269021171s trace: https://jaeger.dev.storj.io/trace/78791d41d75e16a0 FAIL (2026-03-06 20:25:22 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 19:52:18.382132233 (8h52m15.978903242s) [upload=9902]: Uploading segments... Mar 6 19:52:18.382227489 (8h52m15.978998515s) [upload=9902]: Got next segment. Inline: false Mar 6 19:52:18.382313252 (8h52m15.979084279s) [upload=9902]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 6 19:52:18.382319269 (8h52m15.979090296s) [upload=9902]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 6 19:52:18.411471452 (8h52m16.008242482s) [seg_pos={0 0}, upload=9902]: Begin upload segment... Mar 6 19:52:18.411537096 (8h52m16.008308109s) [seg_pos={0 0}, upload=9902]: Done begin upload segment. Mar 6 19:52:18.411564540 (8h52m16.008335586s) [noise=true , address=45.91.116.7:4037 , piece_id=YXDMXUBE7CEWLJZP6FPDCRVIDGG5CJ3Y3IKEKO7EE2DQUSG6W5RA, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.411626862 (8h52m16.008397897s) [noise=true , address=193.149.164.67:4018 , piece_id=GTQQP355TZZF4W7PPBYZS6ZQ6WYK7I3QIE4DZC7STWAKSBAMT4BQ, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.412356605 (8h52m16.009127631s) [noise=true , address=167.17.72.70:4771 , piece_id=TDLPTUHPNXCICPPXRTRL7WCJSXPX22ONIWWPHNXFUNIKRYPYW52Q, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.412900389 (8h52m16.009671415s) [noise=true , address=145.79.227.39:4022 , piece_id=U5GX3GLVWGWLU2BDWTHM7ST3MLLJDEAERWU3UEHO7Z2RXHGZU2DA, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.413467061 (8h52m16.010238103s) [noise=true , address=45.91.116.11:4133 , piece_id=RZWV5JMLOJ36AH7KSDXVTL7CMHI4INEW6YOOQVMCIIC2NGS3XYJA, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.413653431 (8h52m16.010424457s) [noise=true , address=193.149.164.69:4101 , piece_id=BTIBVPSORFJQR3CLA5UV62NXNI6GD3JF4D5Z74KU4R4CXB3WHHKA, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.414227965 (8h52m16.010998999s) [noise=true , address=45.91.116.14:4199 , piece_id=EKGDG7LYV3O5NSDL3UPWFKXINYNAOSPYYOWQPTFN326AJ2ZXAXHQ, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.415177146 (8h52m16.01194818s) [noise=true , address=77.90.31.16:4763 , piece_id=5Q7H3CU3MWSFPQGC67XMLCMTSUIEJQUV7QTO4UZLZY4MIVM76Z6Q, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.415227889 (8h52m16.011998912s) [noise=true , address=45.91.116.10:4096 , piece_id=EIKWFKPE5BYSZGV5T4RPQED55PG2FXU3ZKSNRVCBD363YQDHS4BQ, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.415728547 (8h52m16.012499581s) [noise=true , address=77.90.31.14:4693 , piece_id=S7NAS2ZBHAQPZRHBRGRSZPLE6COQEEN6SZVMB7X2X6IMXTN5HCBQ, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.415910147 (8h52m16.012681174s) [noise=true , address=193.149.164.70:4129 , piece_id=V7WMI7RDYDOAEQYWF2OTZL5LRPUA2S3V5WZN4BKDY6BI4YV3GRPQ, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.416592462 (8h52m16.01336349s) [noise=true , address=77.90.31.18:4831 , piece_id=4PMIUNE4YXKLIJPML3VSUN4PF32TSLPOOSDZZNAXFWH7WICBOW3Q, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.417118897 (8h52m16.01388993s) [noise=true , address=77.90.31.13:4655 , piece_id=47Q27QI7O2IYW7YQKFMJGGMWPVZWWKQTFVYPUMPUI7ORE7236XIQ, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.417185844 (8h52m16.013956873s) [noise=true , address=193.149.164.71:4304 , piece_id=DTCL54GM76GWKAE7A3X5PUNFYO7H5JIKRT5WBEHYEDF2YNFYTOAA, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.417686790 (8h52m16.014457823s) [noise=true , address=167.17.72.71:4810 , piece_id=ANLHBMENEXSGTHBHLOIHMLM6JGHKIU2X6FPQQIUIVMVUOABNK73Q, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.418236951 (8h52m16.015007982s) [noise=true , address=145.79.227.17:4994 , piece_id=TO3EOAILO4PHF6Z7WQA4RWZNNL3JO6EXXEF2RLW7IHAGWUU2TF4Q, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.419293640 (8h52m16.016064672s) [noise=true , address=64.49.13.139:4479 , piece_id=XUYHOCSBATYECYHWLVZ4VBCMT3H4O5ETSQ5HZOB42OR3OY4LWJ6Q, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.420005502 (8h52m16.016776534s) [noise=true , address=45.91.116.8:4066 , piece_id=5MZHIJ3ELVBAJ7ABRKG565TLU4A4NKB7X6WAACY2JBSLACL5LGVQ, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.483654433 (8h52m16.080425441s) [upload=9902]: Next returned nil segment Mar 6 19:52:18.483671157 (8h52m16.08044219s) [upload=9902]: Waiting for error group managing segments... Mar 6 19:52:18.488078321 (8h52m16.084849398s) [noise=true , address=77.90.31.13:4655 , piece_id=47Q27QI7O2IYW7YQKFMJGGMWPVZWWKQTFVYPUMPUI7ORE7236XIQ, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:52:18.498435619 (8h52m16.095206647s) [noise=true , address=145.79.227.17:4994 , piece_id=TO3EOAILO4PHF6Z7WQA4RWZNNL3JO6EXXEF2RLW7IHAGWUU2TF4Q, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:52:18.502979814 (8h52m16.099750841s) [noise=true , address=77.90.31.16:4763 , piece_id=5Q7H3CU3MWSFPQGC67XMLCMTSUIEJQUV7QTO4UZLZY4MIVM76Z6Q, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:52:18.506259140 (8h52m16.103030166s) [noise=true , address=77.90.31.14:4693 , piece_id=S7NAS2ZBHAQPZRHBRGRSZPLE6COQEEN6SZVMB7X2X6IMXTN5HCBQ, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:52:18.514529389 (8h52m16.111300413s) [noise=true , address=145.79.227.39:4022 , piece_id=U5GX3GLVWGWLU2BDWTHM7ST3MLLJDEAERWU3UEHO7Z2RXHGZU2DA, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:52:18.785925494 (8h52m16.382696517s) [noise=true , address=167.17.72.70:4771 , piece_id=TDLPTUHPNXCICPPXRTRL7WCJSXPX22ONIWWPHNXFUNIKRYPYW52Q, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:52:18.810774903 (8h52m16.407545926s) [noise=true , address=77.90.31.18:4831 , piece_id=4PMIUNE4YXKLIJPML3VSUN4PF32TSLPOOSDZZNAXFWH7WICBOW3Q, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:52:19.002746550 (8h52m16.5995176s) [noise=true , address=193.149.164.71:4304 , piece_id=DTCL54GM76GWKAE7A3X5PUNFYO7H5JIKRT5WBEHYEDF2YNFYTOAA, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:52:19.007614901 (8h52m16.604385924s) [noise=true , address=193.149.164.67:4018 , piece_id=GTQQP355TZZF4W7PPBYZS6ZQ6WYK7I3QIE4DZC7STWAKSBAMT4BQ, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:52:19.010443151 (8h52m16.607214176s) [noise=true , address=193.149.164.69:4101 , piece_id=BTIBVPSORFJQR3CLA5UV62NXNI6GD3JF4D5Z74KU4R4CXB3WHHKA, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:52:19.012614014 (8h52m16.609385038s) [noise=true , address=193.149.164.70:4129 , piece_id=V7WMI7RDYDOAEQYWF2OTZL5LRPUA2S3V5WZN4BKDY6BI4YV3GRPQ, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:52:19.042145380 (8h52m16.638916406s) [noise=true , address=64.49.13.139:4479 , piece_id=XUYHOCSBATYECYHWLVZ4VBCMT3H4O5ETSQ5HZOB42OR3OY4LWJ6Q, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:52:21.314990180 (8h52m18.911761213s) [noise=true , address=167.17.72.71:4810 , piece_id=ANLHBMENEXSGTHBHLOIHMLM6JGHKIU2X6FPQQIUIVMVUOABNK73Q, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:53:18.344002671 (8h53m15.9407737s) [noise=true , address=45.91.116.10:4096 , piece_id=EIKWFKPE5BYSZGV5T4RPQED55PG2FXU3ZKSNRVCBD363YQDHS4BQ, seg_pos={0 0}, upload=9902]: Done uploading piece. err: ecclient: upload failed (node:12MZX1GwZGrEi2q44dPiYrwMoum5A2caZtXTS3qiLNcoyHhon7p, address:45.91.116.10:4096): protocol: write tcp 159.223.143.39:44996->45.91.116.10:4096: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 6 19:53:18.344173472 (8h53m15.940944497s) [noise=true , address=45.91.116.7:4037 , piece_id=YXDMXUBE7CEWLJZP6FPDCRVIDGG5CJ3Y3IKEKO7EE2DQUSG6W5RA, seg_pos={0 0}, upload=9902]: Done uploading piece. err: ecclient: upload failed (node:1iNPwE2XSfXcaXAXcrH2jcyNGYeDiiWvzAcDQffrmjPwTVqVmK, address:45.91.116.7:4037): protocol: write tcp 159.223.143.39:52854->45.91.116.7:4037: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 6 19:53:18.344535130 (8h53m15.941306156s) [noise=true , address=45.91.116.14:4199 , piece_id=EKGDG7LYV3O5NSDL3UPWFKXINYNAOSPYYOWQPTFN326AJ2ZXAXHQ, seg_pos={0 0}, upload=9902]: Done uploading piece. err: ecclient: upload failed (node:1d4B1KMLs2annUuJ3v81FJapS9Yg4omBTj48eWoi6HJxnSEjrf, address:45.91.116.14:4199): protocol: write tcp 159.223.143.39:57596->45.91.116.14:4199: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 6 19:53:18.344687364 (8h53m15.941458387s) [noise=true , address=45.91.116.11:4133 , piece_id=RZWV5JMLOJ36AH7KSDXVTL7CMHI4INEW6YOOQVMCIIC2NGS3XYJA, seg_pos={0 0}, upload=9902]: Done uploading piece. err: ecclient: upload failed (node:12e1wX7hiER8sX4MLm5qLue4mayiN4SxXFdZSxfymRaRZxRGU6E, address:45.91.116.11:4133): protocol: write tcp 159.223.143.39:56440->45.91.116.11:4133: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 6 19:53:18.345220236 (8h53m15.941991259s) [noise=true , address=45.91.116.8:4066 , piece_id=5MZHIJ3ELVBAJ7ABRKG565TLU4A4NKB7X6WAACY2JBSLACL5LGVQ, seg_pos={0 0}, upload=9902]: Done uploading piece. err: ecclient: upload failed (node:123uv49k8LraXJfYQg6KRQthrYs1Egc4c3eZWm5JPvDatFbykm8, address:45.91.116.8:4066): protocol: write tcp 159.223.143.39:44136->45.91.116.8:4066: use of closed network connection; context deadline exceeded Mar 6 19:53:18.345277850 (8h53m15.942048865s) [seg_pos={0 0}, upload=9902]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 6 19:53:18.367560006 (8h53m15.964331035s) [upload=9902]: Done uploading segments... Mar 6 19:53:18.367606646 (8h53m15.964377681s) [upload=9902]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 1.26904692s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [............X...............................................] duration: 1.380992101s trace: https://jaeger.dev.storj.io/trace/78791d41d75e16a0 FAIL (2026-03-06 20:24:49 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 19:52:18.382132233 (8h52m15.978903242s) [upload=9902]: Uploading segments... Mar 6 19:52:18.382227489 (8h52m15.978998515s) [upload=9902]: Got next segment. Inline: false Mar 6 19:52:18.382313252 (8h52m15.979084279s) [upload=9902]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 6 19:52:18.382319269 (8h52m15.979090296s) [upload=9902]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 6 19:52:18.411471452 (8h52m16.008242482s) [seg_pos={0 0}, upload=9902]: Begin upload segment... Mar 6 19:52:18.411537096 (8h52m16.008308109s) [seg_pos={0 0}, upload=9902]: Done begin upload segment. Mar 6 19:52:18.411564540 (8h52m16.008335586s) [noise=true , address=45.91.116.7:4037 , piece_id=YXDMXUBE7CEWLJZP6FPDCRVIDGG5CJ3Y3IKEKO7EE2DQUSG6W5RA, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.411626862 (8h52m16.008397897s) [noise=true , address=193.149.164.67:4018 , piece_id=GTQQP355TZZF4W7PPBYZS6ZQ6WYK7I3QIE4DZC7STWAKSBAMT4BQ, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.412356605 (8h52m16.009127631s) [noise=true , address=167.17.72.70:4771 , piece_id=TDLPTUHPNXCICPPXRTRL7WCJSXPX22ONIWWPHNXFUNIKRYPYW52Q, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.412900389 (8h52m16.009671415s) [noise=true , address=145.79.227.39:4022 , piece_id=U5GX3GLVWGWLU2BDWTHM7ST3MLLJDEAERWU3UEHO7Z2RXHGZU2DA, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.413467061 (8h52m16.010238103s) [noise=true , address=45.91.116.11:4133 , piece_id=RZWV5JMLOJ36AH7KSDXVTL7CMHI4INEW6YOOQVMCIIC2NGS3XYJA, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.413653431 (8h52m16.010424457s) [noise=true , address=193.149.164.69:4101 , piece_id=BTIBVPSORFJQR3CLA5UV62NXNI6GD3JF4D5Z74KU4R4CXB3WHHKA, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.414227965 (8h52m16.010998999s) [noise=true , address=45.91.116.14:4199 , piece_id=EKGDG7LYV3O5NSDL3UPWFKXINYNAOSPYYOWQPTFN326AJ2ZXAXHQ, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.415177146 (8h52m16.01194818s) [noise=true , address=77.90.31.16:4763 , piece_id=5Q7H3CU3MWSFPQGC67XMLCMTSUIEJQUV7QTO4UZLZY4MIVM76Z6Q, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.415227889 (8h52m16.011998912s) [noise=true , address=45.91.116.10:4096 , piece_id=EIKWFKPE5BYSZGV5T4RPQED55PG2FXU3ZKSNRVCBD363YQDHS4BQ, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.415728547 (8h52m16.012499581s) [noise=true , address=77.90.31.14:4693 , piece_id=S7NAS2ZBHAQPZRHBRGRSZPLE6COQEEN6SZVMB7X2X6IMXTN5HCBQ, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.415910147 (8h52m16.012681174s) [noise=true , address=193.149.164.70:4129 , piece_id=V7WMI7RDYDOAEQYWF2OTZL5LRPUA2S3V5WZN4BKDY6BI4YV3GRPQ, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.416592462 (8h52m16.01336349s) [noise=true , address=77.90.31.18:4831 , piece_id=4PMIUNE4YXKLIJPML3VSUN4PF32TSLPOOSDZZNAXFWH7WICBOW3Q, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.417118897 (8h52m16.01388993s) [noise=true , address=77.90.31.13:4655 , piece_id=47Q27QI7O2IYW7YQKFMJGGMWPVZWWKQTFVYPUMPUI7ORE7236XIQ, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.417185844 (8h52m16.013956873s) [noise=true , address=193.149.164.71:4304 , piece_id=DTCL54GM76GWKAE7A3X5PUNFYO7H5JIKRT5WBEHYEDF2YNFYTOAA, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.417686790 (8h52m16.014457823s) [noise=true , address=167.17.72.71:4810 , piece_id=ANLHBMENEXSGTHBHLOIHMLM6JGHKIU2X6FPQQIUIVMVUOABNK73Q, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.418236951 (8h52m16.015007982s) [noise=true , address=145.79.227.17:4994 , piece_id=TO3EOAILO4PHF6Z7WQA4RWZNNL3JO6EXXEF2RLW7IHAGWUU2TF4Q, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.419293640 (8h52m16.016064672s) [noise=true , address=64.49.13.139:4479 , piece_id=XUYHOCSBATYECYHWLVZ4VBCMT3H4O5ETSQ5HZOB42OR3OY4LWJ6Q, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.420005502 (8h52m16.016776534s) [noise=true , address=45.91.116.8:4066 , piece_id=5MZHIJ3ELVBAJ7ABRKG565TLU4A4NKB7X6WAACY2JBSLACL5LGVQ, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.483654433 (8h52m16.080425441s) [upload=9902]: Next returned nil segment Mar 6 19:52:18.483671157 (8h52m16.08044219s) [upload=9902]: Waiting for error group managing segments... Mar 6 19:52:18.488078321 (8h52m16.084849398s) [noise=true , address=77.90.31.13:4655 , piece_id=47Q27QI7O2IYW7YQKFMJGGMWPVZWWKQTFVYPUMPUI7ORE7236XIQ, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:52:18.498435619 (8h52m16.095206647s) [noise=true , address=145.79.227.17:4994 , piece_id=TO3EOAILO4PHF6Z7WQA4RWZNNL3JO6EXXEF2RLW7IHAGWUU2TF4Q, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:52:18.502979814 (8h52m16.099750841s) [noise=true , address=77.90.31.16:4763 , piece_id=5Q7H3CU3MWSFPQGC67XMLCMTSUIEJQUV7QTO4UZLZY4MIVM76Z6Q, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:52:18.506259140 (8h52m16.103030166s) [noise=true , address=77.90.31.14:4693 , piece_id=S7NAS2ZBHAQPZRHBRGRSZPLE6COQEEN6SZVMB7X2X6IMXTN5HCBQ, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:52:18.514529389 (8h52m16.111300413s) [noise=true , address=145.79.227.39:4022 , piece_id=U5GX3GLVWGWLU2BDWTHM7ST3MLLJDEAERWU3UEHO7Z2RXHGZU2DA, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:52:18.785925494 (8h52m16.382696517s) [noise=true , address=167.17.72.70:4771 , piece_id=TDLPTUHPNXCICPPXRTRL7WCJSXPX22ONIWWPHNXFUNIKRYPYW52Q, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:52:18.810774903 (8h52m16.407545926s) [noise=true , address=77.90.31.18:4831 , piece_id=4PMIUNE4YXKLIJPML3VSUN4PF32TSLPOOSDZZNAXFWH7WICBOW3Q, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:52:19.002746550 (8h52m16.5995176s) [noise=true , address=193.149.164.71:4304 , piece_id=DTCL54GM76GWKAE7A3X5PUNFYO7H5JIKRT5WBEHYEDF2YNFYTOAA, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:52:19.007614901 (8h52m16.604385924s) [noise=true , address=193.149.164.67:4018 , piece_id=GTQQP355TZZF4W7PPBYZS6ZQ6WYK7I3QIE4DZC7STWAKSBAMT4BQ, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:52:19.010443151 (8h52m16.607214176s) [noise=true , address=193.149.164.69:4101 , piece_id=BTIBVPSORFJQR3CLA5UV62NXNI6GD3JF4D5Z74KU4R4CXB3WHHKA, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:52:19.012614014 (8h52m16.609385038s) [noise=true , address=193.149.164.70:4129 , piece_id=V7WMI7RDYDOAEQYWF2OTZL5LRPUA2S3V5WZN4BKDY6BI4YV3GRPQ, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:52:19.042145380 (8h52m16.638916406s) [noise=true , address=64.49.13.139:4479 , piece_id=XUYHOCSBATYECYHWLVZ4VBCMT3H4O5ETSQ5HZOB42OR3OY4LWJ6Q, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:52:21.314990180 (8h52m18.911761213s) [noise=true , address=167.17.72.71:4810 , piece_id=ANLHBMENEXSGTHBHLOIHMLM6JGHKIU2X6FPQQIUIVMVUOABNK73Q, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:53:18.344002671 (8h53m15.9407737s) [noise=true , address=45.91.116.10:4096 , piece_id=EIKWFKPE5BYSZGV5T4RPQED55PG2FXU3ZKSNRVCBD363YQDHS4BQ, seg_pos={0 0}, upload=9902]: Done uploading piece. err: ecclient: upload failed (node:12MZX1GwZGrEi2q44dPiYrwMoum5A2caZtXTS3qiLNcoyHhon7p, address:45.91.116.10:4096): protocol: write tcp 159.223.143.39:44996->45.91.116.10:4096: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 6 19:53:18.344173472 (8h53m15.940944497s) [noise=true , address=45.91.116.7:4037 , piece_id=YXDMXUBE7CEWLJZP6FPDCRVIDGG5CJ3Y3IKEKO7EE2DQUSG6W5RA, seg_pos={0 0}, upload=9902]: Done uploading piece. err: ecclient: upload failed (node:1iNPwE2XSfXcaXAXcrH2jcyNGYeDiiWvzAcDQffrmjPwTVqVmK, address:45.91.116.7:4037): protocol: write tcp 159.223.143.39:52854->45.91.116.7:4037: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 6 19:53:18.344535130 (8h53m15.941306156s) [noise=true , address=45.91.116.14:4199 , piece_id=EKGDG7LYV3O5NSDL3UPWFKXINYNAOSPYYOWQPTFN326AJ2ZXAXHQ, seg_pos={0 0}, upload=9902]: Done uploading piece. err: ecclient: upload failed (node:1d4B1KMLs2annUuJ3v81FJapS9Yg4omBTj48eWoi6HJxnSEjrf, address:45.91.116.14:4199): protocol: write tcp 159.223.143.39:57596->45.91.116.14:4199: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 6 19:53:18.344687364 (8h53m15.941458387s) [noise=true , address=45.91.116.11:4133 , piece_id=RZWV5JMLOJ36AH7KSDXVTL7CMHI4INEW6YOOQVMCIIC2NGS3XYJA, seg_pos={0 0}, upload=9902]: Done uploading piece. err: ecclient: upload failed (node:12e1wX7hiER8sX4MLm5qLue4mayiN4SxXFdZSxfymRaRZxRGU6E, address:45.91.116.11:4133): protocol: write tcp 159.223.143.39:56440->45.91.116.11:4133: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 6 19:53:18.345220236 (8h53m15.941991259s) [noise=true , address=45.91.116.8:4066 , piece_id=5MZHIJ3ELVBAJ7ABRKG565TLU4A4NKB7X6WAACY2JBSLACL5LGVQ, seg_pos={0 0}, upload=9902]: Done uploading piece. err: ecclient: upload failed (node:123uv49k8LraXJfYQg6KRQthrYs1Egc4c3eZWm5JPvDatFbykm8, address:45.91.116.8:4066): protocol: write tcp 159.223.143.39:44136->45.91.116.8:4066: use of closed network connection; context deadline exceeded Mar 6 19:53:18.345277850 (8h53m15.942048865s) [seg_pos={0 0}, upload=9902]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 6 19:53:18.367560006 (8h53m15.964331035s) [upload=9902]: Done uploading segments... Mar 6 19:53:18.367606646 (8h53m15.964377681s) [upload=9902]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 1.381010961s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............X..............................................] duration: 1.134544716s trace: https://jaeger.dev.storj.io/trace/78791d41d75e16a0 FAIL (2026-03-06 20:23:37 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 19:52:18.382132233 (8h52m15.978903242s) [upload=9902]: Uploading segments... Mar 6 19:52:18.382227489 (8h52m15.978998515s) [upload=9902]: Got next segment. Inline: false Mar 6 19:52:18.382313252 (8h52m15.979084279s) [upload=9902]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 6 19:52:18.382319269 (8h52m15.979090296s) [upload=9902]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 6 19:52:18.411471452 (8h52m16.008242482s) [seg_pos={0 0}, upload=9902]: Begin upload segment... Mar 6 19:52:18.411537096 (8h52m16.008308109s) [seg_pos={0 0}, upload=9902]: Done begin upload segment. Mar 6 19:52:18.411564540 (8h52m16.008335586s) [noise=true , address=45.91.116.7:4037 , piece_id=YXDMXUBE7CEWLJZP6FPDCRVIDGG5CJ3Y3IKEKO7EE2DQUSG6W5RA, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.411626862 (8h52m16.008397897s) [noise=true , address=193.149.164.67:4018 , piece_id=GTQQP355TZZF4W7PPBYZS6ZQ6WYK7I3QIE4DZC7STWAKSBAMT4BQ, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.412356605 (8h52m16.009127631s) [noise=true , address=167.17.72.70:4771 , piece_id=TDLPTUHPNXCICPPXRTRL7WCJSXPX22ONIWWPHNXFUNIKRYPYW52Q, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.412900389 (8h52m16.009671415s) [noise=true , address=145.79.227.39:4022 , piece_id=U5GX3GLVWGWLU2BDWTHM7ST3MLLJDEAERWU3UEHO7Z2RXHGZU2DA, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.413467061 (8h52m16.010238103s) [noise=true , address=45.91.116.11:4133 , piece_id=RZWV5JMLOJ36AH7KSDXVTL7CMHI4INEW6YOOQVMCIIC2NGS3XYJA, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.413653431 (8h52m16.010424457s) [noise=true , address=193.149.164.69:4101 , piece_id=BTIBVPSORFJQR3CLA5UV62NXNI6GD3JF4D5Z74KU4R4CXB3WHHKA, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.414227965 (8h52m16.010998999s) [noise=true , address=45.91.116.14:4199 , piece_id=EKGDG7LYV3O5NSDL3UPWFKXINYNAOSPYYOWQPTFN326AJ2ZXAXHQ, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.415177146 (8h52m16.01194818s) [noise=true , address=77.90.31.16:4763 , piece_id=5Q7H3CU3MWSFPQGC67XMLCMTSUIEJQUV7QTO4UZLZY4MIVM76Z6Q, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.415227889 (8h52m16.011998912s) [noise=true , address=45.91.116.10:4096 , piece_id=EIKWFKPE5BYSZGV5T4RPQED55PG2FXU3ZKSNRVCBD363YQDHS4BQ, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.415728547 (8h52m16.012499581s) [noise=true , address=77.90.31.14:4693 , piece_id=S7NAS2ZBHAQPZRHBRGRSZPLE6COQEEN6SZVMB7X2X6IMXTN5HCBQ, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.415910147 (8h52m16.012681174s) [noise=true , address=193.149.164.70:4129 , piece_id=V7WMI7RDYDOAEQYWF2OTZL5LRPUA2S3V5WZN4BKDY6BI4YV3GRPQ, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.416592462 (8h52m16.01336349s) [noise=true , address=77.90.31.18:4831 , piece_id=4PMIUNE4YXKLIJPML3VSUN4PF32TSLPOOSDZZNAXFWH7WICBOW3Q, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.417118897 (8h52m16.01388993s) [noise=true , address=77.90.31.13:4655 , piece_id=47Q27QI7O2IYW7YQKFMJGGMWPVZWWKQTFVYPUMPUI7ORE7236XIQ, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.417185844 (8h52m16.013956873s) [noise=true , address=193.149.164.71:4304 , piece_id=DTCL54GM76GWKAE7A3X5PUNFYO7H5JIKRT5WBEHYEDF2YNFYTOAA, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.417686790 (8h52m16.014457823s) [noise=true , address=167.17.72.71:4810 , piece_id=ANLHBMENEXSGTHBHLOIHMLM6JGHKIU2X6FPQQIUIVMVUOABNK73Q, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.418236951 (8h52m16.015007982s) [noise=true , address=145.79.227.17:4994 , piece_id=TO3EOAILO4PHF6Z7WQA4RWZNNL3JO6EXXEF2RLW7IHAGWUU2TF4Q, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.419293640 (8h52m16.016064672s) [noise=true , address=64.49.13.139:4479 , piece_id=XUYHOCSBATYECYHWLVZ4VBCMT3H4O5ETSQ5HZOB42OR3OY4LWJ6Q, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.420005502 (8h52m16.016776534s) [noise=true , address=45.91.116.8:4066 , piece_id=5MZHIJ3ELVBAJ7ABRKG565TLU4A4NKB7X6WAACY2JBSLACL5LGVQ, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.483654433 (8h52m16.080425441s) [upload=9902]: Next returned nil segment Mar 6 19:52:18.483671157 (8h52m16.08044219s) [upload=9902]: Waiting for error group managing segments... Mar 6 19:52:18.488078321 (8h52m16.084849398s) [noise=true , address=77.90.31.13:4655 , piece_id=47Q27QI7O2IYW7YQKFMJGGMWPVZWWKQTFVYPUMPUI7ORE7236XIQ, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:52:18.498435619 (8h52m16.095206647s) [noise=true , address=145.79.227.17:4994 , piece_id=TO3EOAILO4PHF6Z7WQA4RWZNNL3JO6EXXEF2RLW7IHAGWUU2TF4Q, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:52:18.502979814 (8h52m16.099750841s) [noise=true , address=77.90.31.16:4763 , piece_id=5Q7H3CU3MWSFPQGC67XMLCMTSUIEJQUV7QTO4UZLZY4MIVM76Z6Q, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:52:18.506259140 (8h52m16.103030166s) [noise=true , address=77.90.31.14:4693 , piece_id=S7NAS2ZBHAQPZRHBRGRSZPLE6COQEEN6SZVMB7X2X6IMXTN5HCBQ, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:52:18.514529389 (8h52m16.111300413s) [noise=true , address=145.79.227.39:4022 , piece_id=U5GX3GLVWGWLU2BDWTHM7ST3MLLJDEAERWU3UEHO7Z2RXHGZU2DA, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:52:18.785925494 (8h52m16.382696517s) [noise=true , address=167.17.72.70:4771 , piece_id=TDLPTUHPNXCICPPXRTRL7WCJSXPX22ONIWWPHNXFUNIKRYPYW52Q, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:52:18.810774903 (8h52m16.407545926s) [noise=true , address=77.90.31.18:4831 , piece_id=4PMIUNE4YXKLIJPML3VSUN4PF32TSLPOOSDZZNAXFWH7WICBOW3Q, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:52:19.002746550 (8h52m16.5995176s) [noise=true , address=193.149.164.71:4304 , piece_id=DTCL54GM76GWKAE7A3X5PUNFYO7H5JIKRT5WBEHYEDF2YNFYTOAA, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:52:19.007614901 (8h52m16.604385924s) [noise=true , address=193.149.164.67:4018 , piece_id=GTQQP355TZZF4W7PPBYZS6ZQ6WYK7I3QIE4DZC7STWAKSBAMT4BQ, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:52:19.010443151 (8h52m16.607214176s) [noise=true , address=193.149.164.69:4101 , piece_id=BTIBVPSORFJQR3CLA5UV62NXNI6GD3JF4D5Z74KU4R4CXB3WHHKA, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:52:19.012614014 (8h52m16.609385038s) [noise=true , address=193.149.164.70:4129 , piece_id=V7WMI7RDYDOAEQYWF2OTZL5LRPUA2S3V5WZN4BKDY6BI4YV3GRPQ, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:52:19.042145380 (8h52m16.638916406s) [noise=true , address=64.49.13.139:4479 , piece_id=XUYHOCSBATYECYHWLVZ4VBCMT3H4O5ETSQ5HZOB42OR3OY4LWJ6Q, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:52:21.314990180 (8h52m18.911761213s) [noise=true , address=167.17.72.71:4810 , piece_id=ANLHBMENEXSGTHBHLOIHMLM6JGHKIU2X6FPQQIUIVMVUOABNK73Q, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:53:18.344002671 (8h53m15.9407737s) [noise=true , address=45.91.116.10:4096 , piece_id=EIKWFKPE5BYSZGV5T4RPQED55PG2FXU3ZKSNRVCBD363YQDHS4BQ, seg_pos={0 0}, upload=9902]: Done uploading piece. err: ecclient: upload failed (node:12MZX1GwZGrEi2q44dPiYrwMoum5A2caZtXTS3qiLNcoyHhon7p, address:45.91.116.10:4096): protocol: write tcp 159.223.143.39:44996->45.91.116.10:4096: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 6 19:53:18.344173472 (8h53m15.940944497s) [noise=true , address=45.91.116.7:4037 , piece_id=YXDMXUBE7CEWLJZP6FPDCRVIDGG5CJ3Y3IKEKO7EE2DQUSG6W5RA, seg_pos={0 0}, upload=9902]: Done uploading piece. err: ecclient: upload failed (node:1iNPwE2XSfXcaXAXcrH2jcyNGYeDiiWvzAcDQffrmjPwTVqVmK, address:45.91.116.7:4037): protocol: write tcp 159.223.143.39:52854->45.91.116.7:4037: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 6 19:53:18.344535130 (8h53m15.941306156s) [noise=true , address=45.91.116.14:4199 , piece_id=EKGDG7LYV3O5NSDL3UPWFKXINYNAOSPYYOWQPTFN326AJ2ZXAXHQ, seg_pos={0 0}, upload=9902]: Done uploading piece. err: ecclient: upload failed (node:1d4B1KMLs2annUuJ3v81FJapS9Yg4omBTj48eWoi6HJxnSEjrf, address:45.91.116.14:4199): protocol: write tcp 159.223.143.39:57596->45.91.116.14:4199: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 6 19:53:18.344687364 (8h53m15.941458387s) [noise=true , address=45.91.116.11:4133 , piece_id=RZWV5JMLOJ36AH7KSDXVTL7CMHI4INEW6YOOQVMCIIC2NGS3XYJA, seg_pos={0 0}, upload=9902]: Done uploading piece. err: ecclient: upload failed (node:12e1wX7hiER8sX4MLm5qLue4mayiN4SxXFdZSxfymRaRZxRGU6E, address:45.91.116.11:4133): protocol: write tcp 159.223.143.39:56440->45.91.116.11:4133: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 6 19:53:18.345220236 (8h53m15.941991259s) [noise=true , address=45.91.116.8:4066 , piece_id=5MZHIJ3ELVBAJ7ABRKG565TLU4A4NKB7X6WAACY2JBSLACL5LGVQ, seg_pos={0 0}, upload=9902]: Done uploading piece. err: ecclient: upload failed (node:123uv49k8LraXJfYQg6KRQthrYs1Egc4c3eZWm5JPvDatFbykm8, address:45.91.116.8:4066): protocol: write tcp 159.223.143.39:44136->45.91.116.8:4066: use of closed network connection; context deadline exceeded Mar 6 19:53:18.345277850 (8h53m15.942048865s) [seg_pos={0 0}, upload=9902]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 6 19:53:18.367560006 (8h53m15.964331035s) [upload=9902]: Done uploading segments... Mar 6 19:53:18.367606646 (8h53m15.964377681s) [upload=9902]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 1.134575752s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............X.............................................] duration: 1.169893252s trace: https://jaeger.dev.storj.io/trace/78791d41d75e16a0 FAIL (2026-03-06 20:23:19 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 19:52:18.382132233 (8h52m15.978903242s) [upload=9902]: Uploading segments... Mar 6 19:52:18.382227489 (8h52m15.978998515s) [upload=9902]: Got next segment. Inline: false Mar 6 19:52:18.382313252 (8h52m15.979084279s) [upload=9902]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 6 19:52:18.382319269 (8h52m15.979090296s) [upload=9902]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 6 19:52:18.411471452 (8h52m16.008242482s) [seg_pos={0 0}, upload=9902]: Begin upload segment... Mar 6 19:52:18.411537096 (8h52m16.008308109s) [seg_pos={0 0}, upload=9902]: Done begin upload segment. Mar 6 19:52:18.411564540 (8h52m16.008335586s) [noise=true , address=45.91.116.7:4037 , piece_id=YXDMXUBE7CEWLJZP6FPDCRVIDGG5CJ3Y3IKEKO7EE2DQUSG6W5RA, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.411626862 (8h52m16.008397897s) [noise=true , address=193.149.164.67:4018 , piece_id=GTQQP355TZZF4W7PPBYZS6ZQ6WYK7I3QIE4DZC7STWAKSBAMT4BQ, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.412356605 (8h52m16.009127631s) [noise=true , address=167.17.72.70:4771 , piece_id=TDLPTUHPNXCICPPXRTRL7WCJSXPX22ONIWWPHNXFUNIKRYPYW52Q, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.412900389 (8h52m16.009671415s) [noise=true , address=145.79.227.39:4022 , piece_id=U5GX3GLVWGWLU2BDWTHM7ST3MLLJDEAERWU3UEHO7Z2RXHGZU2DA, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.413467061 (8h52m16.010238103s) [noise=true , address=45.91.116.11:4133 , piece_id=RZWV5JMLOJ36AH7KSDXVTL7CMHI4INEW6YOOQVMCIIC2NGS3XYJA, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.413653431 (8h52m16.010424457s) [noise=true , address=193.149.164.69:4101 , piece_id=BTIBVPSORFJQR3CLA5UV62NXNI6GD3JF4D5Z74KU4R4CXB3WHHKA, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.414227965 (8h52m16.010998999s) [noise=true , address=45.91.116.14:4199 , piece_id=EKGDG7LYV3O5NSDL3UPWFKXINYNAOSPYYOWQPTFN326AJ2ZXAXHQ, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.415177146 (8h52m16.01194818s) [noise=true , address=77.90.31.16:4763 , piece_id=5Q7H3CU3MWSFPQGC67XMLCMTSUIEJQUV7QTO4UZLZY4MIVM76Z6Q, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.415227889 (8h52m16.011998912s) [noise=true , address=45.91.116.10:4096 , piece_id=EIKWFKPE5BYSZGV5T4RPQED55PG2FXU3ZKSNRVCBD363YQDHS4BQ, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.415728547 (8h52m16.012499581s) [noise=true , address=77.90.31.14:4693 , piece_id=S7NAS2ZBHAQPZRHBRGRSZPLE6COQEEN6SZVMB7X2X6IMXTN5HCBQ, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.415910147 (8h52m16.012681174s) [noise=true , address=193.149.164.70:4129 , piece_id=V7WMI7RDYDOAEQYWF2OTZL5LRPUA2S3V5WZN4BKDY6BI4YV3GRPQ, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.416592462 (8h52m16.01336349s) [noise=true , address=77.90.31.18:4831 , piece_id=4PMIUNE4YXKLIJPML3VSUN4PF32TSLPOOSDZZNAXFWH7WICBOW3Q, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.417118897 (8h52m16.01388993s) [noise=true , address=77.90.31.13:4655 , piece_id=47Q27QI7O2IYW7YQKFMJGGMWPVZWWKQTFVYPUMPUI7ORE7236XIQ, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.417185844 (8h52m16.013956873s) [noise=true , address=193.149.164.71:4304 , piece_id=DTCL54GM76GWKAE7A3X5PUNFYO7H5JIKRT5WBEHYEDF2YNFYTOAA, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.417686790 (8h52m16.014457823s) [noise=true , address=167.17.72.71:4810 , piece_id=ANLHBMENEXSGTHBHLOIHMLM6JGHKIU2X6FPQQIUIVMVUOABNK73Q, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.418236951 (8h52m16.015007982s) [noise=true , address=145.79.227.17:4994 , piece_id=TO3EOAILO4PHF6Z7WQA4RWZNNL3JO6EXXEF2RLW7IHAGWUU2TF4Q, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.419293640 (8h52m16.016064672s) [noise=true , address=64.49.13.139:4479 , piece_id=XUYHOCSBATYECYHWLVZ4VBCMT3H4O5ETSQ5HZOB42OR3OY4LWJ6Q, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.420005502 (8h52m16.016776534s) [noise=true , address=45.91.116.8:4066 , piece_id=5MZHIJ3ELVBAJ7ABRKG565TLU4A4NKB7X6WAACY2JBSLACL5LGVQ, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.483654433 (8h52m16.080425441s) [upload=9902]: Next returned nil segment Mar 6 19:52:18.483671157 (8h52m16.08044219s) [upload=9902]: Waiting for error group managing segments... Mar 6 19:52:18.488078321 (8h52m16.084849398s) [noise=true , address=77.90.31.13:4655 , piece_id=47Q27QI7O2IYW7YQKFMJGGMWPVZWWKQTFVYPUMPUI7ORE7236XIQ, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:52:18.498435619 (8h52m16.095206647s) [noise=true , address=145.79.227.17:4994 , piece_id=TO3EOAILO4PHF6Z7WQA4RWZNNL3JO6EXXEF2RLW7IHAGWUU2TF4Q, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:52:18.502979814 (8h52m16.099750841s) [noise=true , address=77.90.31.16:4763 , piece_id=5Q7H3CU3MWSFPQGC67XMLCMTSUIEJQUV7QTO4UZLZY4MIVM76Z6Q, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:52:18.506259140 (8h52m16.103030166s) [noise=true , address=77.90.31.14:4693 , piece_id=S7NAS2ZBHAQPZRHBRGRSZPLE6COQEEN6SZVMB7X2X6IMXTN5HCBQ, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:52:18.514529389 (8h52m16.111300413s) [noise=true , address=145.79.227.39:4022 , piece_id=U5GX3GLVWGWLU2BDWTHM7ST3MLLJDEAERWU3UEHO7Z2RXHGZU2DA, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:52:18.785925494 (8h52m16.382696517s) [noise=true , address=167.17.72.70:4771 , piece_id=TDLPTUHPNXCICPPXRTRL7WCJSXPX22ONIWWPHNXFUNIKRYPYW52Q, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:52:18.810774903 (8h52m16.407545926s) [noise=true , address=77.90.31.18:4831 , piece_id=4PMIUNE4YXKLIJPML3VSUN4PF32TSLPOOSDZZNAXFWH7WICBOW3Q, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:52:19.002746550 (8h52m16.5995176s) [noise=true , address=193.149.164.71:4304 , piece_id=DTCL54GM76GWKAE7A3X5PUNFYO7H5JIKRT5WBEHYEDF2YNFYTOAA, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:52:19.007614901 (8h52m16.604385924s) [noise=true , address=193.149.164.67:4018 , piece_id=GTQQP355TZZF4W7PPBYZS6ZQ6WYK7I3QIE4DZC7STWAKSBAMT4BQ, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:52:19.010443151 (8h52m16.607214176s) [noise=true , address=193.149.164.69:4101 , piece_id=BTIBVPSORFJQR3CLA5UV62NXNI6GD3JF4D5Z74KU4R4CXB3WHHKA, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:52:19.012614014 (8h52m16.609385038s) [noise=true , address=193.149.164.70:4129 , piece_id=V7WMI7RDYDOAEQYWF2OTZL5LRPUA2S3V5WZN4BKDY6BI4YV3GRPQ, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:52:19.042145380 (8h52m16.638916406s) [noise=true , address=64.49.13.139:4479 , piece_id=XUYHOCSBATYECYHWLVZ4VBCMT3H4O5ETSQ5HZOB42OR3OY4LWJ6Q, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:52:21.314990180 (8h52m18.911761213s) [noise=true , address=167.17.72.71:4810 , piece_id=ANLHBMENEXSGTHBHLOIHMLM6JGHKIU2X6FPQQIUIVMVUOABNK73Q, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:53:18.344002671 (8h53m15.9407737s) [noise=true , address=45.91.116.10:4096 , piece_id=EIKWFKPE5BYSZGV5T4RPQED55PG2FXU3ZKSNRVCBD363YQDHS4BQ, seg_pos={0 0}, upload=9902]: Done uploading piece. err: ecclient: upload failed (node:12MZX1GwZGrEi2q44dPiYrwMoum5A2caZtXTS3qiLNcoyHhon7p, address:45.91.116.10:4096): protocol: write tcp 159.223.143.39:44996->45.91.116.10:4096: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 6 19:53:18.344173472 (8h53m15.940944497s) [noise=true , address=45.91.116.7:4037 , piece_id=YXDMXUBE7CEWLJZP6FPDCRVIDGG5CJ3Y3IKEKO7EE2DQUSG6W5RA, seg_pos={0 0}, upload=9902]: Done uploading piece. err: ecclient: upload failed (node:1iNPwE2XSfXcaXAXcrH2jcyNGYeDiiWvzAcDQffrmjPwTVqVmK, address:45.91.116.7:4037): protocol: write tcp 159.223.143.39:52854->45.91.116.7:4037: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 6 19:53:18.344535130 (8h53m15.941306156s) [noise=true , address=45.91.116.14:4199 , piece_id=EKGDG7LYV3O5NSDL3UPWFKXINYNAOSPYYOWQPTFN326AJ2ZXAXHQ, seg_pos={0 0}, upload=9902]: Done uploading piece. err: ecclient: upload failed (node:1d4B1KMLs2annUuJ3v81FJapS9Yg4omBTj48eWoi6HJxnSEjrf, address:45.91.116.14:4199): protocol: write tcp 159.223.143.39:57596->45.91.116.14:4199: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 6 19:53:18.344687364 (8h53m15.941458387s) [noise=true , address=45.91.116.11:4133 , piece_id=RZWV5JMLOJ36AH7KSDXVTL7CMHI4INEW6YOOQVMCIIC2NGS3XYJA, seg_pos={0 0}, upload=9902]: Done uploading piece. err: ecclient: upload failed (node:12e1wX7hiER8sX4MLm5qLue4mayiN4SxXFdZSxfymRaRZxRGU6E, address:45.91.116.11:4133): protocol: write tcp 159.223.143.39:56440->45.91.116.11:4133: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 6 19:53:18.345220236 (8h53m15.941991259s) [noise=true , address=45.91.116.8:4066 , piece_id=5MZHIJ3ELVBAJ7ABRKG565TLU4A4NKB7X6WAACY2JBSLACL5LGVQ, seg_pos={0 0}, upload=9902]: Done uploading piece. err: ecclient: upload failed (node:123uv49k8LraXJfYQg6KRQthrYs1Egc4c3eZWm5JPvDatFbykm8, address:45.91.116.8:4066): protocol: write tcp 159.223.143.39:44136->45.91.116.8:4066: use of closed network connection; context deadline exceeded Mar 6 19:53:18.345277850 (8h53m15.942048865s) [seg_pos={0 0}, upload=9902]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 6 19:53:18.367560006 (8h53m15.964331035s) [upload=9902]: Done uploading segments... Mar 6 19:53:18.367606646 (8h53m15.964377681s) [upload=9902]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 1.169934059s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............X............................................] duration: 1.292821557s trace: https://jaeger.dev.storj.io/trace/78791d41d75e16a0 FAIL (2026-03-06 20:22:28 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 19:52:18.382132233 (8h52m15.978903242s) [upload=9902]: Uploading segments... Mar 6 19:52:18.382227489 (8h52m15.978998515s) [upload=9902]: Got next segment. Inline: false Mar 6 19:52:18.382313252 (8h52m15.979084279s) [upload=9902]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 6 19:52:18.382319269 (8h52m15.979090296s) [upload=9902]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 6 19:52:18.411471452 (8h52m16.008242482s) [seg_pos={0 0}, upload=9902]: Begin upload segment... Mar 6 19:52:18.411537096 (8h52m16.008308109s) [seg_pos={0 0}, upload=9902]: Done begin upload segment. Mar 6 19:52:18.411564540 (8h52m16.008335586s) [noise=true , address=45.91.116.7:4037 , piece_id=YXDMXUBE7CEWLJZP6FPDCRVIDGG5CJ3Y3IKEKO7EE2DQUSG6W5RA, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.411626862 (8h52m16.008397897s) [noise=true , address=193.149.164.67:4018 , piece_id=GTQQP355TZZF4W7PPBYZS6ZQ6WYK7I3QIE4DZC7STWAKSBAMT4BQ, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.412356605 (8h52m16.009127631s) [noise=true , address=167.17.72.70:4771 , piece_id=TDLPTUHPNXCICPPXRTRL7WCJSXPX22ONIWWPHNXFUNIKRYPYW52Q, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.412900389 (8h52m16.009671415s) [noise=true , address=145.79.227.39:4022 , piece_id=U5GX3GLVWGWLU2BDWTHM7ST3MLLJDEAERWU3UEHO7Z2RXHGZU2DA, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.413467061 (8h52m16.010238103s) [noise=true , address=45.91.116.11:4133 , piece_id=RZWV5JMLOJ36AH7KSDXVTL7CMHI4INEW6YOOQVMCIIC2NGS3XYJA, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.413653431 (8h52m16.010424457s) [noise=true , address=193.149.164.69:4101 , piece_id=BTIBVPSORFJQR3CLA5UV62NXNI6GD3JF4D5Z74KU4R4CXB3WHHKA, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.414227965 (8h52m16.010998999s) [noise=true , address=45.91.116.14:4199 , piece_id=EKGDG7LYV3O5NSDL3UPWFKXINYNAOSPYYOWQPTFN326AJ2ZXAXHQ, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.415177146 (8h52m16.01194818s) [noise=true , address=77.90.31.16:4763 , piece_id=5Q7H3CU3MWSFPQGC67XMLCMTSUIEJQUV7QTO4UZLZY4MIVM76Z6Q, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.415227889 (8h52m16.011998912s) [noise=true , address=45.91.116.10:4096 , piece_id=EIKWFKPE5BYSZGV5T4RPQED55PG2FXU3ZKSNRVCBD363YQDHS4BQ, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.415728547 (8h52m16.012499581s) [noise=true , address=77.90.31.14:4693 , piece_id=S7NAS2ZBHAQPZRHBRGRSZPLE6COQEEN6SZVMB7X2X6IMXTN5HCBQ, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.415910147 (8h52m16.012681174s) [noise=true , address=193.149.164.70:4129 , piece_id=V7WMI7RDYDOAEQYWF2OTZL5LRPUA2S3V5WZN4BKDY6BI4YV3GRPQ, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.416592462 (8h52m16.01336349s) [noise=true , address=77.90.31.18:4831 , piece_id=4PMIUNE4YXKLIJPML3VSUN4PF32TSLPOOSDZZNAXFWH7WICBOW3Q, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.417118897 (8h52m16.01388993s) [noise=true , address=77.90.31.13:4655 , piece_id=47Q27QI7O2IYW7YQKFMJGGMWPVZWWKQTFVYPUMPUI7ORE7236XIQ, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.417185844 (8h52m16.013956873s) [noise=true , address=193.149.164.71:4304 , piece_id=DTCL54GM76GWKAE7A3X5PUNFYO7H5JIKRT5WBEHYEDF2YNFYTOAA, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.417686790 (8h52m16.014457823s) [noise=true , address=167.17.72.71:4810 , piece_id=ANLHBMENEXSGTHBHLOIHMLM6JGHKIU2X6FPQQIUIVMVUOABNK73Q, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.418236951 (8h52m16.015007982s) [noise=true , address=145.79.227.17:4994 , piece_id=TO3EOAILO4PHF6Z7WQA4RWZNNL3JO6EXXEF2RLW7IHAGWUU2TF4Q, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.419293640 (8h52m16.016064672s) [noise=true , address=64.49.13.139:4479 , piece_id=XUYHOCSBATYECYHWLVZ4VBCMT3H4O5ETSQ5HZOB42OR3OY4LWJ6Q, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.420005502 (8h52m16.016776534s) [noise=true , address=45.91.116.8:4066 , piece_id=5MZHIJ3ELVBAJ7ABRKG565TLU4A4NKB7X6WAACY2JBSLACL5LGVQ, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.483654433 (8h52m16.080425441s) [upload=9902]: Next returned nil segment Mar 6 19:52:18.483671157 (8h52m16.08044219s) [upload=9902]: Waiting for error group managing segments... Mar 6 19:52:18.488078321 (8h52m16.084849398s) [noise=true , address=77.90.31.13:4655 , piece_id=47Q27QI7O2IYW7YQKFMJGGMWPVZWWKQTFVYPUMPUI7ORE7236XIQ, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:52:18.498435619 (8h52m16.095206647s) [noise=true , address=145.79.227.17:4994 , piece_id=TO3EOAILO4PHF6Z7WQA4RWZNNL3JO6EXXEF2RLW7IHAGWUU2TF4Q, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:52:18.502979814 (8h52m16.099750841s) [noise=true , address=77.90.31.16:4763 , piece_id=5Q7H3CU3MWSFPQGC67XMLCMTSUIEJQUV7QTO4UZLZY4MIVM76Z6Q, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:52:18.506259140 (8h52m16.103030166s) [noise=true , address=77.90.31.14:4693 , piece_id=S7NAS2ZBHAQPZRHBRGRSZPLE6COQEEN6SZVMB7X2X6IMXTN5HCBQ, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:52:18.514529389 (8h52m16.111300413s) [noise=true , address=145.79.227.39:4022 , piece_id=U5GX3GLVWGWLU2BDWTHM7ST3MLLJDEAERWU3UEHO7Z2RXHGZU2DA, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:52:18.785925494 (8h52m16.382696517s) [noise=true , address=167.17.72.70:4771 , piece_id=TDLPTUHPNXCICPPXRTRL7WCJSXPX22ONIWWPHNXFUNIKRYPYW52Q, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:52:18.810774903 (8h52m16.407545926s) [noise=true , address=77.90.31.18:4831 , piece_id=4PMIUNE4YXKLIJPML3VSUN4PF32TSLPOOSDZZNAXFWH7WICBOW3Q, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:52:19.002746550 (8h52m16.5995176s) [noise=true , address=193.149.164.71:4304 , piece_id=DTCL54GM76GWKAE7A3X5PUNFYO7H5JIKRT5WBEHYEDF2YNFYTOAA, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:52:19.007614901 (8h52m16.604385924s) [noise=true , address=193.149.164.67:4018 , piece_id=GTQQP355TZZF4W7PPBYZS6ZQ6WYK7I3QIE4DZC7STWAKSBAMT4BQ, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:52:19.010443151 (8h52m16.607214176s) [noise=true , address=193.149.164.69:4101 , piece_id=BTIBVPSORFJQR3CLA5UV62NXNI6GD3JF4D5Z74KU4R4CXB3WHHKA, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:52:19.012614014 (8h52m16.609385038s) [noise=true , address=193.149.164.70:4129 , piece_id=V7WMI7RDYDOAEQYWF2OTZL5LRPUA2S3V5WZN4BKDY6BI4YV3GRPQ, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:52:19.042145380 (8h52m16.638916406s) [noise=true , address=64.49.13.139:4479 , piece_id=XUYHOCSBATYECYHWLVZ4VBCMT3H4O5ETSQ5HZOB42OR3OY4LWJ6Q, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:52:21.314990180 (8h52m18.911761213s) [noise=true , address=167.17.72.71:4810 , piece_id=ANLHBMENEXSGTHBHLOIHMLM6JGHKIU2X6FPQQIUIVMVUOABNK73Q, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:53:18.344002671 (8h53m15.9407737s) [noise=true , address=45.91.116.10:4096 , piece_id=EIKWFKPE5BYSZGV5T4RPQED55PG2FXU3ZKSNRVCBD363YQDHS4BQ, seg_pos={0 0}, upload=9902]: Done uploading piece. err: ecclient: upload failed (node:12MZX1GwZGrEi2q44dPiYrwMoum5A2caZtXTS3qiLNcoyHhon7p, address:45.91.116.10:4096): protocol: write tcp 159.223.143.39:44996->45.91.116.10:4096: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 6 19:53:18.344173472 (8h53m15.940944497s) [noise=true , address=45.91.116.7:4037 , piece_id=YXDMXUBE7CEWLJZP6FPDCRVIDGG5CJ3Y3IKEKO7EE2DQUSG6W5RA, seg_pos={0 0}, upload=9902]: Done uploading piece. err: ecclient: upload failed (node:1iNPwE2XSfXcaXAXcrH2jcyNGYeDiiWvzAcDQffrmjPwTVqVmK, address:45.91.116.7:4037): protocol: write tcp 159.223.143.39:52854->45.91.116.7:4037: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 6 19:53:18.344535130 (8h53m15.941306156s) [noise=true , address=45.91.116.14:4199 , piece_id=EKGDG7LYV3O5NSDL3UPWFKXINYNAOSPYYOWQPTFN326AJ2ZXAXHQ, seg_pos={0 0}, upload=9902]: Done uploading piece. err: ecclient: upload failed (node:1d4B1KMLs2annUuJ3v81FJapS9Yg4omBTj48eWoi6HJxnSEjrf, address:45.91.116.14:4199): protocol: write tcp 159.223.143.39:57596->45.91.116.14:4199: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 6 19:53:18.344687364 (8h53m15.941458387s) [noise=true , address=45.91.116.11:4133 , piece_id=RZWV5JMLOJ36AH7KSDXVTL7CMHI4INEW6YOOQVMCIIC2NGS3XYJA, seg_pos={0 0}, upload=9902]: Done uploading piece. err: ecclient: upload failed (node:12e1wX7hiER8sX4MLm5qLue4mayiN4SxXFdZSxfymRaRZxRGU6E, address:45.91.116.11:4133): protocol: write tcp 159.223.143.39:56440->45.91.116.11:4133: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 6 19:53:18.345220236 (8h53m15.941991259s) [noise=true , address=45.91.116.8:4066 , piece_id=5MZHIJ3ELVBAJ7ABRKG565TLU4A4NKB7X6WAACY2JBSLACL5LGVQ, seg_pos={0 0}, upload=9902]: Done uploading piece. err: ecclient: upload failed (node:123uv49k8LraXJfYQg6KRQthrYs1Egc4c3eZWm5JPvDatFbykm8, address:45.91.116.8:4066): protocol: write tcp 159.223.143.39:44136->45.91.116.8:4066: use of closed network connection; context deadline exceeded Mar 6 19:53:18.345277850 (8h53m15.942048865s) [seg_pos={0 0}, upload=9902]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 6 19:53:18.367560006 (8h53m15.964331035s) [upload=9902]: Done uploading segments... Mar 6 19:53:18.367606646 (8h53m15.964377681s) [upload=9902]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 1.292843154s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [................X...........................................] duration: 960.720001ms trace: https://jaeger.dev.storj.io/trace/78791d41d75e16a0 FAIL (2026-03-06 20:21:45 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 19:52:18.382132233 (8h52m15.978903242s) [upload=9902]: Uploading segments... Mar 6 19:52:18.382227489 (8h52m15.978998515s) [upload=9902]: Got next segment. Inline: false Mar 6 19:52:18.382313252 (8h52m15.979084279s) [upload=9902]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 6 19:52:18.382319269 (8h52m15.979090296s) [upload=9902]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 6 19:52:18.411471452 (8h52m16.008242482s) [seg_pos={0 0}, upload=9902]: Begin upload segment... Mar 6 19:52:18.411537096 (8h52m16.008308109s) [seg_pos={0 0}, upload=9902]: Done begin upload segment. Mar 6 19:52:18.411564540 (8h52m16.008335586s) [noise=true , address=45.91.116.7:4037 , piece_id=YXDMXUBE7CEWLJZP6FPDCRVIDGG5CJ3Y3IKEKO7EE2DQUSG6W5RA, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.411626862 (8h52m16.008397897s) [noise=true , address=193.149.164.67:4018 , piece_id=GTQQP355TZZF4W7PPBYZS6ZQ6WYK7I3QIE4DZC7STWAKSBAMT4BQ, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.412356605 (8h52m16.009127631s) [noise=true , address=167.17.72.70:4771 , piece_id=TDLPTUHPNXCICPPXRTRL7WCJSXPX22ONIWWPHNXFUNIKRYPYW52Q, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.412900389 (8h52m16.009671415s) [noise=true , address=145.79.227.39:4022 , piece_id=U5GX3GLVWGWLU2BDWTHM7ST3MLLJDEAERWU3UEHO7Z2RXHGZU2DA, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.413467061 (8h52m16.010238103s) [noise=true , address=45.91.116.11:4133 , piece_id=RZWV5JMLOJ36AH7KSDXVTL7CMHI4INEW6YOOQVMCIIC2NGS3XYJA, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.413653431 (8h52m16.010424457s) [noise=true , address=193.149.164.69:4101 , piece_id=BTIBVPSORFJQR3CLA5UV62NXNI6GD3JF4D5Z74KU4R4CXB3WHHKA, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.414227965 (8h52m16.010998999s) [noise=true , address=45.91.116.14:4199 , piece_id=EKGDG7LYV3O5NSDL3UPWFKXINYNAOSPYYOWQPTFN326AJ2ZXAXHQ, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.415177146 (8h52m16.01194818s) [noise=true , address=77.90.31.16:4763 , piece_id=5Q7H3CU3MWSFPQGC67XMLCMTSUIEJQUV7QTO4UZLZY4MIVM76Z6Q, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.415227889 (8h52m16.011998912s) [noise=true , address=45.91.116.10:4096 , piece_id=EIKWFKPE5BYSZGV5T4RPQED55PG2FXU3ZKSNRVCBD363YQDHS4BQ, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.415728547 (8h52m16.012499581s) [noise=true , address=77.90.31.14:4693 , piece_id=S7NAS2ZBHAQPZRHBRGRSZPLE6COQEEN6SZVMB7X2X6IMXTN5HCBQ, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.415910147 (8h52m16.012681174s) [noise=true , address=193.149.164.70:4129 , piece_id=V7WMI7RDYDOAEQYWF2OTZL5LRPUA2S3V5WZN4BKDY6BI4YV3GRPQ, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.416592462 (8h52m16.01336349s) [noise=true , address=77.90.31.18:4831 , piece_id=4PMIUNE4YXKLIJPML3VSUN4PF32TSLPOOSDZZNAXFWH7WICBOW3Q, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.417118897 (8h52m16.01388993s) [noise=true , address=77.90.31.13:4655 , piece_id=47Q27QI7O2IYW7YQKFMJGGMWPVZWWKQTFVYPUMPUI7ORE7236XIQ, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.417185844 (8h52m16.013956873s) [noise=true , address=193.149.164.71:4304 , piece_id=DTCL54GM76GWKAE7A3X5PUNFYO7H5JIKRT5WBEHYEDF2YNFYTOAA, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.417686790 (8h52m16.014457823s) [noise=true , address=167.17.72.71:4810 , piece_id=ANLHBMENEXSGTHBHLOIHMLM6JGHKIU2X6FPQQIUIVMVUOABNK73Q, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.418236951 (8h52m16.015007982s) [noise=true , address=145.79.227.17:4994 , piece_id=TO3EOAILO4PHF6Z7WQA4RWZNNL3JO6EXXEF2RLW7IHAGWUU2TF4Q, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.419293640 (8h52m16.016064672s) [noise=true , address=64.49.13.139:4479 , piece_id=XUYHOCSBATYECYHWLVZ4VBCMT3H4O5ETSQ5HZOB42OR3OY4LWJ6Q, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.420005502 (8h52m16.016776534s) [noise=true , address=45.91.116.8:4066 , piece_id=5MZHIJ3ELVBAJ7ABRKG565TLU4A4NKB7X6WAACY2JBSLACL5LGVQ, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.483654433 (8h52m16.080425441s) [upload=9902]: Next returned nil segment Mar 6 19:52:18.483671157 (8h52m16.08044219s) [upload=9902]: Waiting for error group managing segments... Mar 6 19:52:18.488078321 (8h52m16.084849398s) [noise=true , address=77.90.31.13:4655 , piece_id=47Q27QI7O2IYW7YQKFMJGGMWPVZWWKQTFVYPUMPUI7ORE7236XIQ, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:52:18.498435619 (8h52m16.095206647s) [noise=true , address=145.79.227.17:4994 , piece_id=TO3EOAILO4PHF6Z7WQA4RWZNNL3JO6EXXEF2RLW7IHAGWUU2TF4Q, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:52:18.502979814 (8h52m16.099750841s) [noise=true , address=77.90.31.16:4763 , piece_id=5Q7H3CU3MWSFPQGC67XMLCMTSUIEJQUV7QTO4UZLZY4MIVM76Z6Q, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:52:18.506259140 (8h52m16.103030166s) [noise=true , address=77.90.31.14:4693 , piece_id=S7NAS2ZBHAQPZRHBRGRSZPLE6COQEEN6SZVMB7X2X6IMXTN5HCBQ, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:52:18.514529389 (8h52m16.111300413s) [noise=true , address=145.79.227.39:4022 , piece_id=U5GX3GLVWGWLU2BDWTHM7ST3MLLJDEAERWU3UEHO7Z2RXHGZU2DA, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:52:18.785925494 (8h52m16.382696517s) [noise=true , address=167.17.72.70:4771 , piece_id=TDLPTUHPNXCICPPXRTRL7WCJSXPX22ONIWWPHNXFUNIKRYPYW52Q, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:52:18.810774903 (8h52m16.407545926s) [noise=true , address=77.90.31.18:4831 , piece_id=4PMIUNE4YXKLIJPML3VSUN4PF32TSLPOOSDZZNAXFWH7WICBOW3Q, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:52:19.002746550 (8h52m16.5995176s) [noise=true , address=193.149.164.71:4304 , piece_id=DTCL54GM76GWKAE7A3X5PUNFYO7H5JIKRT5WBEHYEDF2YNFYTOAA, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:52:19.007614901 (8h52m16.604385924s) [noise=true , address=193.149.164.67:4018 , piece_id=GTQQP355TZZF4W7PPBYZS6ZQ6WYK7I3QIE4DZC7STWAKSBAMT4BQ, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:52:19.010443151 (8h52m16.607214176s) [noise=true , address=193.149.164.69:4101 , piece_id=BTIBVPSORFJQR3CLA5UV62NXNI6GD3JF4D5Z74KU4R4CXB3WHHKA, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:52:19.012614014 (8h52m16.609385038s) [noise=true , address=193.149.164.70:4129 , piece_id=V7WMI7RDYDOAEQYWF2OTZL5LRPUA2S3V5WZN4BKDY6BI4YV3GRPQ, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:52:19.042145380 (8h52m16.638916406s) [noise=true , address=64.49.13.139:4479 , piece_id=XUYHOCSBATYECYHWLVZ4VBCMT3H4O5ETSQ5HZOB42OR3OY4LWJ6Q, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:52:21.314990180 (8h52m18.911761213s) [noise=true , address=167.17.72.71:4810 , piece_id=ANLHBMENEXSGTHBHLOIHMLM6JGHKIU2X6FPQQIUIVMVUOABNK73Q, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:53:18.344002671 (8h53m15.9407737s) [noise=true , address=45.91.116.10:4096 , piece_id=EIKWFKPE5BYSZGV5T4RPQED55PG2FXU3ZKSNRVCBD363YQDHS4BQ, seg_pos={0 0}, upload=9902]: Done uploading piece. err: ecclient: upload failed (node:12MZX1GwZGrEi2q44dPiYrwMoum5A2caZtXTS3qiLNcoyHhon7p, address:45.91.116.10:4096): protocol: write tcp 159.223.143.39:44996->45.91.116.10:4096: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 6 19:53:18.344173472 (8h53m15.940944497s) [noise=true , address=45.91.116.7:4037 , piece_id=YXDMXUBE7CEWLJZP6FPDCRVIDGG5CJ3Y3IKEKO7EE2DQUSG6W5RA, seg_pos={0 0}, upload=9902]: Done uploading piece. err: ecclient: upload failed (node:1iNPwE2XSfXcaXAXcrH2jcyNGYeDiiWvzAcDQffrmjPwTVqVmK, address:45.91.116.7:4037): protocol: write tcp 159.223.143.39:52854->45.91.116.7:4037: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 6 19:53:18.344535130 (8h53m15.941306156s) [noise=true , address=45.91.116.14:4199 , piece_id=EKGDG7LYV3O5NSDL3UPWFKXINYNAOSPYYOWQPTFN326AJ2ZXAXHQ, seg_pos={0 0}, upload=9902]: Done uploading piece. err: ecclient: upload failed (node:1d4B1KMLs2annUuJ3v81FJapS9Yg4omBTj48eWoi6HJxnSEjrf, address:45.91.116.14:4199): protocol: write tcp 159.223.143.39:57596->45.91.116.14:4199: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 6 19:53:18.344687364 (8h53m15.941458387s) [noise=true , address=45.91.116.11:4133 , piece_id=RZWV5JMLOJ36AH7KSDXVTL7CMHI4INEW6YOOQVMCIIC2NGS3XYJA, seg_pos={0 0}, upload=9902]: Done uploading piece. err: ecclient: upload failed (node:12e1wX7hiER8sX4MLm5qLue4mayiN4SxXFdZSxfymRaRZxRGU6E, address:45.91.116.11:4133): protocol: write tcp 159.223.143.39:56440->45.91.116.11:4133: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 6 19:53:18.345220236 (8h53m15.941991259s) [noise=true , address=45.91.116.8:4066 , piece_id=5MZHIJ3ELVBAJ7ABRKG565TLU4A4NKB7X6WAACY2JBSLACL5LGVQ, seg_pos={0 0}, upload=9902]: Done uploading piece. err: ecclient: upload failed (node:123uv49k8LraXJfYQg6KRQthrYs1Egc4c3eZWm5JPvDatFbykm8, address:45.91.116.8:4066): protocol: write tcp 159.223.143.39:44136->45.91.116.8:4066: use of closed network connection; context deadline exceeded Mar 6 19:53:18.345277850 (8h53m15.942048865s) [seg_pos={0 0}, upload=9902]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 6 19:53:18.367560006 (8h53m15.964331035s) [upload=9902]: Done uploading segments... Mar 6 19:53:18.367606646 (8h53m15.964377681s) [upload=9902]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 960.742919ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................X..........................................] duration: 1.212957301s trace: https://jaeger.dev.storj.io/trace/78791d41d75e16a0 FAIL (2026-03-06 20:21:15 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 19:52:18.382132233 (8h52m15.978903242s) [upload=9902]: Uploading segments... Mar 6 19:52:18.382227489 (8h52m15.978998515s) [upload=9902]: Got next segment. Inline: false Mar 6 19:52:18.382313252 (8h52m15.979084279s) [upload=9902]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 6 19:52:18.382319269 (8h52m15.979090296s) [upload=9902]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 6 19:52:18.411471452 (8h52m16.008242482s) [seg_pos={0 0}, upload=9902]: Begin upload segment... Mar 6 19:52:18.411537096 (8h52m16.008308109s) [seg_pos={0 0}, upload=9902]: Done begin upload segment. Mar 6 19:52:18.411564540 (8h52m16.008335586s) [noise=true , address=45.91.116.7:4037 , piece_id=YXDMXUBE7CEWLJZP6FPDCRVIDGG5CJ3Y3IKEKO7EE2DQUSG6W5RA, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.411626862 (8h52m16.008397897s) [noise=true , address=193.149.164.67:4018 , piece_id=GTQQP355TZZF4W7PPBYZS6ZQ6WYK7I3QIE4DZC7STWAKSBAMT4BQ, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.412356605 (8h52m16.009127631s) [noise=true , address=167.17.72.70:4771 , piece_id=TDLPTUHPNXCICPPXRTRL7WCJSXPX22ONIWWPHNXFUNIKRYPYW52Q, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.412900389 (8h52m16.009671415s) [noise=true , address=145.79.227.39:4022 , piece_id=U5GX3GLVWGWLU2BDWTHM7ST3MLLJDEAERWU3UEHO7Z2RXHGZU2DA, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.413467061 (8h52m16.010238103s) [noise=true , address=45.91.116.11:4133 , piece_id=RZWV5JMLOJ36AH7KSDXVTL7CMHI4INEW6YOOQVMCIIC2NGS3XYJA, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.413653431 (8h52m16.010424457s) [noise=true , address=193.149.164.69:4101 , piece_id=BTIBVPSORFJQR3CLA5UV62NXNI6GD3JF4D5Z74KU4R4CXB3WHHKA, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.414227965 (8h52m16.010998999s) [noise=true , address=45.91.116.14:4199 , piece_id=EKGDG7LYV3O5NSDL3UPWFKXINYNAOSPYYOWQPTFN326AJ2ZXAXHQ, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.415177146 (8h52m16.01194818s) [noise=true , address=77.90.31.16:4763 , piece_id=5Q7H3CU3MWSFPQGC67XMLCMTSUIEJQUV7QTO4UZLZY4MIVM76Z6Q, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.415227889 (8h52m16.011998912s) [noise=true , address=45.91.116.10:4096 , piece_id=EIKWFKPE5BYSZGV5T4RPQED55PG2FXU3ZKSNRVCBD363YQDHS4BQ, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.415728547 (8h52m16.012499581s) [noise=true , address=77.90.31.14:4693 , piece_id=S7NAS2ZBHAQPZRHBRGRSZPLE6COQEEN6SZVMB7X2X6IMXTN5HCBQ, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.415910147 (8h52m16.012681174s) [noise=true , address=193.149.164.70:4129 , piece_id=V7WMI7RDYDOAEQYWF2OTZL5LRPUA2S3V5WZN4BKDY6BI4YV3GRPQ, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.416592462 (8h52m16.01336349s) [noise=true , address=77.90.31.18:4831 , piece_id=4PMIUNE4YXKLIJPML3VSUN4PF32TSLPOOSDZZNAXFWH7WICBOW3Q, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.417118897 (8h52m16.01388993s) [noise=true , address=77.90.31.13:4655 , piece_id=47Q27QI7O2IYW7YQKFMJGGMWPVZWWKQTFVYPUMPUI7ORE7236XIQ, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.417185844 (8h52m16.013956873s) [noise=true , address=193.149.164.71:4304 , piece_id=DTCL54GM76GWKAE7A3X5PUNFYO7H5JIKRT5WBEHYEDF2YNFYTOAA, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.417686790 (8h52m16.014457823s) [noise=true , address=167.17.72.71:4810 , piece_id=ANLHBMENEXSGTHBHLOIHMLM6JGHKIU2X6FPQQIUIVMVUOABNK73Q, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.418236951 (8h52m16.015007982s) [noise=true , address=145.79.227.17:4994 , piece_id=TO3EOAILO4PHF6Z7WQA4RWZNNL3JO6EXXEF2RLW7IHAGWUU2TF4Q, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.419293640 (8h52m16.016064672s) [noise=true , address=64.49.13.139:4479 , piece_id=XUYHOCSBATYECYHWLVZ4VBCMT3H4O5ETSQ5HZOB42OR3OY4LWJ6Q, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.420005502 (8h52m16.016776534s) [noise=true , address=45.91.116.8:4066 , piece_id=5MZHIJ3ELVBAJ7ABRKG565TLU4A4NKB7X6WAACY2JBSLACL5LGVQ, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.483654433 (8h52m16.080425441s) [upload=9902]: Next returned nil segment Mar 6 19:52:18.483671157 (8h52m16.08044219s) [upload=9902]: Waiting for error group managing segments... Mar 6 19:52:18.488078321 (8h52m16.084849398s) [noise=true , address=77.90.31.13:4655 , piece_id=47Q27QI7O2IYW7YQKFMJGGMWPVZWWKQTFVYPUMPUI7ORE7236XIQ, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:52:18.498435619 (8h52m16.095206647s) [noise=true , address=145.79.227.17:4994 , piece_id=TO3EOAILO4PHF6Z7WQA4RWZNNL3JO6EXXEF2RLW7IHAGWUU2TF4Q, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:52:18.502979814 (8h52m16.099750841s) [noise=true , address=77.90.31.16:4763 , piece_id=5Q7H3CU3MWSFPQGC67XMLCMTSUIEJQUV7QTO4UZLZY4MIVM76Z6Q, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:52:18.506259140 (8h52m16.103030166s) [noise=true , address=77.90.31.14:4693 , piece_id=S7NAS2ZBHAQPZRHBRGRSZPLE6COQEEN6SZVMB7X2X6IMXTN5HCBQ, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:52:18.514529389 (8h52m16.111300413s) [noise=true , address=145.79.227.39:4022 , piece_id=U5GX3GLVWGWLU2BDWTHM7ST3MLLJDEAERWU3UEHO7Z2RXHGZU2DA, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:52:18.785925494 (8h52m16.382696517s) [noise=true , address=167.17.72.70:4771 , piece_id=TDLPTUHPNXCICPPXRTRL7WCJSXPX22ONIWWPHNXFUNIKRYPYW52Q, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:52:18.810774903 (8h52m16.407545926s) [noise=true , address=77.90.31.18:4831 , piece_id=4PMIUNE4YXKLIJPML3VSUN4PF32TSLPOOSDZZNAXFWH7WICBOW3Q, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:52:19.002746550 (8h52m16.5995176s) [noise=true , address=193.149.164.71:4304 , piece_id=DTCL54GM76GWKAE7A3X5PUNFYO7H5JIKRT5WBEHYEDF2YNFYTOAA, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:52:19.007614901 (8h52m16.604385924s) [noise=true , address=193.149.164.67:4018 , piece_id=GTQQP355TZZF4W7PPBYZS6ZQ6WYK7I3QIE4DZC7STWAKSBAMT4BQ, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:52:19.010443151 (8h52m16.607214176s) [noise=true , address=193.149.164.69:4101 , piece_id=BTIBVPSORFJQR3CLA5UV62NXNI6GD3JF4D5Z74KU4R4CXB3WHHKA, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:52:19.012614014 (8h52m16.609385038s) [noise=true , address=193.149.164.70:4129 , piece_id=V7WMI7RDYDOAEQYWF2OTZL5LRPUA2S3V5WZN4BKDY6BI4YV3GRPQ, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:52:19.042145380 (8h52m16.638916406s) [noise=true , address=64.49.13.139:4479 , piece_id=XUYHOCSBATYECYHWLVZ4VBCMT3H4O5ETSQ5HZOB42OR3OY4LWJ6Q, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:52:21.314990180 (8h52m18.911761213s) [noise=true , address=167.17.72.71:4810 , piece_id=ANLHBMENEXSGTHBHLOIHMLM6JGHKIU2X6FPQQIUIVMVUOABNK73Q, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:53:18.344002671 (8h53m15.9407737s) [noise=true , address=45.91.116.10:4096 , piece_id=EIKWFKPE5BYSZGV5T4RPQED55PG2FXU3ZKSNRVCBD363YQDHS4BQ, seg_pos={0 0}, upload=9902]: Done uploading piece. err: ecclient: upload failed (node:12MZX1GwZGrEi2q44dPiYrwMoum5A2caZtXTS3qiLNcoyHhon7p, address:45.91.116.10:4096): protocol: write tcp 159.223.143.39:44996->45.91.116.10:4096: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 6 19:53:18.344173472 (8h53m15.940944497s) [noise=true , address=45.91.116.7:4037 , piece_id=YXDMXUBE7CEWLJZP6FPDCRVIDGG5CJ3Y3IKEKO7EE2DQUSG6W5RA, seg_pos={0 0}, upload=9902]: Done uploading piece. err: ecclient: upload failed (node:1iNPwE2XSfXcaXAXcrH2jcyNGYeDiiWvzAcDQffrmjPwTVqVmK, address:45.91.116.7:4037): protocol: write tcp 159.223.143.39:52854->45.91.116.7:4037: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 6 19:53:18.344535130 (8h53m15.941306156s) [noise=true , address=45.91.116.14:4199 , piece_id=EKGDG7LYV3O5NSDL3UPWFKXINYNAOSPYYOWQPTFN326AJ2ZXAXHQ, seg_pos={0 0}, upload=9902]: Done uploading piece. err: ecclient: upload failed (node:1d4B1KMLs2annUuJ3v81FJapS9Yg4omBTj48eWoi6HJxnSEjrf, address:45.91.116.14:4199): protocol: write tcp 159.223.143.39:57596->45.91.116.14:4199: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 6 19:53:18.344687364 (8h53m15.941458387s) [noise=true , address=45.91.116.11:4133 , piece_id=RZWV5JMLOJ36AH7KSDXVTL7CMHI4INEW6YOOQVMCIIC2NGS3XYJA, seg_pos={0 0}, upload=9902]: Done uploading piece. err: ecclient: upload failed (node:12e1wX7hiER8sX4MLm5qLue4mayiN4SxXFdZSxfymRaRZxRGU6E, address:45.91.116.11:4133): protocol: write tcp 159.223.143.39:56440->45.91.116.11:4133: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 6 19:53:18.345220236 (8h53m15.941991259s) [noise=true , address=45.91.116.8:4066 , piece_id=5MZHIJ3ELVBAJ7ABRKG565TLU4A4NKB7X6WAACY2JBSLACL5LGVQ, seg_pos={0 0}, upload=9902]: Done uploading piece. err: ecclient: upload failed (node:123uv49k8LraXJfYQg6KRQthrYs1Egc4c3eZWm5JPvDatFbykm8, address:45.91.116.8:4066): protocol: write tcp 159.223.143.39:44136->45.91.116.8:4066: use of closed network connection; context deadline exceeded Mar 6 19:53:18.345277850 (8h53m15.942048865s) [seg_pos={0 0}, upload=9902]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 6 19:53:18.367560006 (8h53m15.964331035s) [upload=9902]: Done uploading segments... Mar 6 19:53:18.367606646 (8h53m15.964377681s) [upload=9902]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 1.212983777s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................X.........................................] duration: 974.966514ms trace: https://jaeger.dev.storj.io/trace/78791d41d75e16a0 FAIL (2026-03-06 20:20:16 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 19:52:18.382132233 (8h52m15.978903242s) [upload=9902]: Uploading segments... Mar 6 19:52:18.382227489 (8h52m15.978998515s) [upload=9902]: Got next segment. Inline: false Mar 6 19:52:18.382313252 (8h52m15.979084279s) [upload=9902]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 6 19:52:18.382319269 (8h52m15.979090296s) [upload=9902]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 6 19:52:18.411471452 (8h52m16.008242482s) [seg_pos={0 0}, upload=9902]: Begin upload segment... Mar 6 19:52:18.411537096 (8h52m16.008308109s) [seg_pos={0 0}, upload=9902]: Done begin upload segment. Mar 6 19:52:18.411564540 (8h52m16.008335586s) [noise=true , address=45.91.116.7:4037 , piece_id=YXDMXUBE7CEWLJZP6FPDCRVIDGG5CJ3Y3IKEKO7EE2DQUSG6W5RA, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.411626862 (8h52m16.008397897s) [noise=true , address=193.149.164.67:4018 , piece_id=GTQQP355TZZF4W7PPBYZS6ZQ6WYK7I3QIE4DZC7STWAKSBAMT4BQ, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.412356605 (8h52m16.009127631s) [noise=true , address=167.17.72.70:4771 , piece_id=TDLPTUHPNXCICPPXRTRL7WCJSXPX22ONIWWPHNXFUNIKRYPYW52Q, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.412900389 (8h52m16.009671415s) [noise=true , address=145.79.227.39:4022 , piece_id=U5GX3GLVWGWLU2BDWTHM7ST3MLLJDEAERWU3UEHO7Z2RXHGZU2DA, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.413467061 (8h52m16.010238103s) [noise=true , address=45.91.116.11:4133 , piece_id=RZWV5JMLOJ36AH7KSDXVTL7CMHI4INEW6YOOQVMCIIC2NGS3XYJA, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.413653431 (8h52m16.010424457s) [noise=true , address=193.149.164.69:4101 , piece_id=BTIBVPSORFJQR3CLA5UV62NXNI6GD3JF4D5Z74KU4R4CXB3WHHKA, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.414227965 (8h52m16.010998999s) [noise=true , address=45.91.116.14:4199 , piece_id=EKGDG7LYV3O5NSDL3UPWFKXINYNAOSPYYOWQPTFN326AJ2ZXAXHQ, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.415177146 (8h52m16.01194818s) [noise=true , address=77.90.31.16:4763 , piece_id=5Q7H3CU3MWSFPQGC67XMLCMTSUIEJQUV7QTO4UZLZY4MIVM76Z6Q, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.415227889 (8h52m16.011998912s) [noise=true , address=45.91.116.10:4096 , piece_id=EIKWFKPE5BYSZGV5T4RPQED55PG2FXU3ZKSNRVCBD363YQDHS4BQ, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.415728547 (8h52m16.012499581s) [noise=true , address=77.90.31.14:4693 , piece_id=S7NAS2ZBHAQPZRHBRGRSZPLE6COQEEN6SZVMB7X2X6IMXTN5HCBQ, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.415910147 (8h52m16.012681174s) [noise=true , address=193.149.164.70:4129 , piece_id=V7WMI7RDYDOAEQYWF2OTZL5LRPUA2S3V5WZN4BKDY6BI4YV3GRPQ, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.416592462 (8h52m16.01336349s) [noise=true , address=77.90.31.18:4831 , piece_id=4PMIUNE4YXKLIJPML3VSUN4PF32TSLPOOSDZZNAXFWH7WICBOW3Q, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.417118897 (8h52m16.01388993s) [noise=true , address=77.90.31.13:4655 , piece_id=47Q27QI7O2IYW7YQKFMJGGMWPVZWWKQTFVYPUMPUI7ORE7236XIQ, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.417185844 (8h52m16.013956873s) [noise=true , address=193.149.164.71:4304 , piece_id=DTCL54GM76GWKAE7A3X5PUNFYO7H5JIKRT5WBEHYEDF2YNFYTOAA, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.417686790 (8h52m16.014457823s) [noise=true , address=167.17.72.71:4810 , piece_id=ANLHBMENEXSGTHBHLOIHMLM6JGHKIU2X6FPQQIUIVMVUOABNK73Q, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.418236951 (8h52m16.015007982s) [noise=true , address=145.79.227.17:4994 , piece_id=TO3EOAILO4PHF6Z7WQA4RWZNNL3JO6EXXEF2RLW7IHAGWUU2TF4Q, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.419293640 (8h52m16.016064672s) [noise=true , address=64.49.13.139:4479 , piece_id=XUYHOCSBATYECYHWLVZ4VBCMT3H4O5ETSQ5HZOB42OR3OY4LWJ6Q, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.420005502 (8h52m16.016776534s) [noise=true , address=45.91.116.8:4066 , piece_id=5MZHIJ3ELVBAJ7ABRKG565TLU4A4NKB7X6WAACY2JBSLACL5LGVQ, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.483654433 (8h52m16.080425441s) [upload=9902]: Next returned nil segment Mar 6 19:52:18.483671157 (8h52m16.08044219s) [upload=9902]: Waiting for error group managing segments... Mar 6 19:52:18.488078321 (8h52m16.084849398s) [noise=true , address=77.90.31.13:4655 , piece_id=47Q27QI7O2IYW7YQKFMJGGMWPVZWWKQTFVYPUMPUI7ORE7236XIQ, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:52:18.498435619 (8h52m16.095206647s) [noise=true , address=145.79.227.17:4994 , piece_id=TO3EOAILO4PHF6Z7WQA4RWZNNL3JO6EXXEF2RLW7IHAGWUU2TF4Q, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:52:18.502979814 (8h52m16.099750841s) [noise=true , address=77.90.31.16:4763 , piece_id=5Q7H3CU3MWSFPQGC67XMLCMTSUIEJQUV7QTO4UZLZY4MIVM76Z6Q, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:52:18.506259140 (8h52m16.103030166s) [noise=true , address=77.90.31.14:4693 , piece_id=S7NAS2ZBHAQPZRHBRGRSZPLE6COQEEN6SZVMB7X2X6IMXTN5HCBQ, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:52:18.514529389 (8h52m16.111300413s) [noise=true , address=145.79.227.39:4022 , piece_id=U5GX3GLVWGWLU2BDWTHM7ST3MLLJDEAERWU3UEHO7Z2RXHGZU2DA, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:52:18.785925494 (8h52m16.382696517s) [noise=true , address=167.17.72.70:4771 , piece_id=TDLPTUHPNXCICPPXRTRL7WCJSXPX22ONIWWPHNXFUNIKRYPYW52Q, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:52:18.810774903 (8h52m16.407545926s) [noise=true , address=77.90.31.18:4831 , piece_id=4PMIUNE4YXKLIJPML3VSUN4PF32TSLPOOSDZZNAXFWH7WICBOW3Q, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:52:19.002746550 (8h52m16.5995176s) [noise=true , address=193.149.164.71:4304 , piece_id=DTCL54GM76GWKAE7A3X5PUNFYO7H5JIKRT5WBEHYEDF2YNFYTOAA, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:52:19.007614901 (8h52m16.604385924s) [noise=true , address=193.149.164.67:4018 , piece_id=GTQQP355TZZF4W7PPBYZS6ZQ6WYK7I3QIE4DZC7STWAKSBAMT4BQ, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:52:19.010443151 (8h52m16.607214176s) [noise=true , address=193.149.164.69:4101 , piece_id=BTIBVPSORFJQR3CLA5UV62NXNI6GD3JF4D5Z74KU4R4CXB3WHHKA, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:52:19.012614014 (8h52m16.609385038s) [noise=true , address=193.149.164.70:4129 , piece_id=V7WMI7RDYDOAEQYWF2OTZL5LRPUA2S3V5WZN4BKDY6BI4YV3GRPQ, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:52:19.042145380 (8h52m16.638916406s) [noise=true , address=64.49.13.139:4479 , piece_id=XUYHOCSBATYECYHWLVZ4VBCMT3H4O5ETSQ5HZOB42OR3OY4LWJ6Q, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:52:21.314990180 (8h52m18.911761213s) [noise=true , address=167.17.72.71:4810 , piece_id=ANLHBMENEXSGTHBHLOIHMLM6JGHKIU2X6FPQQIUIVMVUOABNK73Q, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:53:18.344002671 (8h53m15.9407737s) [noise=true , address=45.91.116.10:4096 , piece_id=EIKWFKPE5BYSZGV5T4RPQED55PG2FXU3ZKSNRVCBD363YQDHS4BQ, seg_pos={0 0}, upload=9902]: Done uploading piece. err: ecclient: upload failed (node:12MZX1GwZGrEi2q44dPiYrwMoum5A2caZtXTS3qiLNcoyHhon7p, address:45.91.116.10:4096): protocol: write tcp 159.223.143.39:44996->45.91.116.10:4096: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 6 19:53:18.344173472 (8h53m15.940944497s) [noise=true , address=45.91.116.7:4037 , piece_id=YXDMXUBE7CEWLJZP6FPDCRVIDGG5CJ3Y3IKEKO7EE2DQUSG6W5RA, seg_pos={0 0}, upload=9902]: Done uploading piece. err: ecclient: upload failed (node:1iNPwE2XSfXcaXAXcrH2jcyNGYeDiiWvzAcDQffrmjPwTVqVmK, address:45.91.116.7:4037): protocol: write tcp 159.223.143.39:52854->45.91.116.7:4037: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 6 19:53:18.344535130 (8h53m15.941306156s) [noise=true , address=45.91.116.14:4199 , piece_id=EKGDG7LYV3O5NSDL3UPWFKXINYNAOSPYYOWQPTFN326AJ2ZXAXHQ, seg_pos={0 0}, upload=9902]: Done uploading piece. err: ecclient: upload failed (node:1d4B1KMLs2annUuJ3v81FJapS9Yg4omBTj48eWoi6HJxnSEjrf, address:45.91.116.14:4199): protocol: write tcp 159.223.143.39:57596->45.91.116.14:4199: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 6 19:53:18.344687364 (8h53m15.941458387s) [noise=true , address=45.91.116.11:4133 , piece_id=RZWV5JMLOJ36AH7KSDXVTL7CMHI4INEW6YOOQVMCIIC2NGS3XYJA, seg_pos={0 0}, upload=9902]: Done uploading piece. err: ecclient: upload failed (node:12e1wX7hiER8sX4MLm5qLue4mayiN4SxXFdZSxfymRaRZxRGU6E, address:45.91.116.11:4133): protocol: write tcp 159.223.143.39:56440->45.91.116.11:4133: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 6 19:53:18.345220236 (8h53m15.941991259s) [noise=true , address=45.91.116.8:4066 , piece_id=5MZHIJ3ELVBAJ7ABRKG565TLU4A4NKB7X6WAACY2JBSLACL5LGVQ, seg_pos={0 0}, upload=9902]: Done uploading piece. err: ecclient: upload failed (node:123uv49k8LraXJfYQg6KRQthrYs1Egc4c3eZWm5JPvDatFbykm8, address:45.91.116.8:4066): protocol: write tcp 159.223.143.39:44136->45.91.116.8:4066: use of closed network connection; context deadline exceeded Mar 6 19:53:18.345277850 (8h53m15.942048865s) [seg_pos={0 0}, upload=9902]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 6 19:53:18.367560006 (8h53m15.964331035s) [upload=9902]: Done uploading segments... Mar 6 19:53:18.367606646 (8h53m15.964377681s) [upload=9902]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 975.013973ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................X........................................] duration: 1.264463993s trace: https://jaeger.dev.storj.io/trace/78791d41d75e16a0 FAIL (2026-03-06 20:19:58 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 19:52:18.382132233 (8h52m15.978903242s) [upload=9902]: Uploading segments... Mar 6 19:52:18.382227489 (8h52m15.978998515s) [upload=9902]: Got next segment. Inline: false Mar 6 19:52:18.382313252 (8h52m15.979084279s) [upload=9902]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 6 19:52:18.382319269 (8h52m15.979090296s) [upload=9902]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 6 19:52:18.411471452 (8h52m16.008242482s) [seg_pos={0 0}, upload=9902]: Begin upload segment... Mar 6 19:52:18.411537096 (8h52m16.008308109s) [seg_pos={0 0}, upload=9902]: Done begin upload segment. Mar 6 19:52:18.411564540 (8h52m16.008335586s) [noise=true , address=45.91.116.7:4037 , piece_id=YXDMXUBE7CEWLJZP6FPDCRVIDGG5CJ3Y3IKEKO7EE2DQUSG6W5RA, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.411626862 (8h52m16.008397897s) [noise=true , address=193.149.164.67:4018 , piece_id=GTQQP355TZZF4W7PPBYZS6ZQ6WYK7I3QIE4DZC7STWAKSBAMT4BQ, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.412356605 (8h52m16.009127631s) [noise=true , address=167.17.72.70:4771 , piece_id=TDLPTUHPNXCICPPXRTRL7WCJSXPX22ONIWWPHNXFUNIKRYPYW52Q, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.412900389 (8h52m16.009671415s) [noise=true , address=145.79.227.39:4022 , piece_id=U5GX3GLVWGWLU2BDWTHM7ST3MLLJDEAERWU3UEHO7Z2RXHGZU2DA, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.413467061 (8h52m16.010238103s) [noise=true , address=45.91.116.11:4133 , piece_id=RZWV5JMLOJ36AH7KSDXVTL7CMHI4INEW6YOOQVMCIIC2NGS3XYJA, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.413653431 (8h52m16.010424457s) [noise=true , address=193.149.164.69:4101 , piece_id=BTIBVPSORFJQR3CLA5UV62NXNI6GD3JF4D5Z74KU4R4CXB3WHHKA, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.414227965 (8h52m16.010998999s) [noise=true , address=45.91.116.14:4199 , piece_id=EKGDG7LYV3O5NSDL3UPWFKXINYNAOSPYYOWQPTFN326AJ2ZXAXHQ, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.415177146 (8h52m16.01194818s) [noise=true , address=77.90.31.16:4763 , piece_id=5Q7H3CU3MWSFPQGC67XMLCMTSUIEJQUV7QTO4UZLZY4MIVM76Z6Q, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.415227889 (8h52m16.011998912s) [noise=true , address=45.91.116.10:4096 , piece_id=EIKWFKPE5BYSZGV5T4RPQED55PG2FXU3ZKSNRVCBD363YQDHS4BQ, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.415728547 (8h52m16.012499581s) [noise=true , address=77.90.31.14:4693 , piece_id=S7NAS2ZBHAQPZRHBRGRSZPLE6COQEEN6SZVMB7X2X6IMXTN5HCBQ, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.415910147 (8h52m16.012681174s) [noise=true , address=193.149.164.70:4129 , piece_id=V7WMI7RDYDOAEQYWF2OTZL5LRPUA2S3V5WZN4BKDY6BI4YV3GRPQ, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.416592462 (8h52m16.01336349s) [noise=true , address=77.90.31.18:4831 , piece_id=4PMIUNE4YXKLIJPML3VSUN4PF32TSLPOOSDZZNAXFWH7WICBOW3Q, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.417118897 (8h52m16.01388993s) [noise=true , address=77.90.31.13:4655 , piece_id=47Q27QI7O2IYW7YQKFMJGGMWPVZWWKQTFVYPUMPUI7ORE7236XIQ, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.417185844 (8h52m16.013956873s) [noise=true , address=193.149.164.71:4304 , piece_id=DTCL54GM76GWKAE7A3X5PUNFYO7H5JIKRT5WBEHYEDF2YNFYTOAA, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.417686790 (8h52m16.014457823s) [noise=true , address=167.17.72.71:4810 , piece_id=ANLHBMENEXSGTHBHLOIHMLM6JGHKIU2X6FPQQIUIVMVUOABNK73Q, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.418236951 (8h52m16.015007982s) [noise=true , address=145.79.227.17:4994 , piece_id=TO3EOAILO4PHF6Z7WQA4RWZNNL3JO6EXXEF2RLW7IHAGWUU2TF4Q, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.419293640 (8h52m16.016064672s) [noise=true , address=64.49.13.139:4479 , piece_id=XUYHOCSBATYECYHWLVZ4VBCMT3H4O5ETSQ5HZOB42OR3OY4LWJ6Q, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.420005502 (8h52m16.016776534s) [noise=true , address=45.91.116.8:4066 , piece_id=5MZHIJ3ELVBAJ7ABRKG565TLU4A4NKB7X6WAACY2JBSLACL5LGVQ, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.483654433 (8h52m16.080425441s) [upload=9902]: Next returned nil segment Mar 6 19:52:18.483671157 (8h52m16.08044219s) [upload=9902]: Waiting for error group managing segments... Mar 6 19:52:18.488078321 (8h52m16.084849398s) [noise=true , address=77.90.31.13:4655 , piece_id=47Q27QI7O2IYW7YQKFMJGGMWPVZWWKQTFVYPUMPUI7ORE7236XIQ, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:52:18.498435619 (8h52m16.095206647s) [noise=true , address=145.79.227.17:4994 , piece_id=TO3EOAILO4PHF6Z7WQA4RWZNNL3JO6EXXEF2RLW7IHAGWUU2TF4Q, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:52:18.502979814 (8h52m16.099750841s) [noise=true , address=77.90.31.16:4763 , piece_id=5Q7H3CU3MWSFPQGC67XMLCMTSUIEJQUV7QTO4UZLZY4MIVM76Z6Q, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:52:18.506259140 (8h52m16.103030166s) [noise=true , address=77.90.31.14:4693 , piece_id=S7NAS2ZBHAQPZRHBRGRSZPLE6COQEEN6SZVMB7X2X6IMXTN5HCBQ, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:52:18.514529389 (8h52m16.111300413s) [noise=true , address=145.79.227.39:4022 , piece_id=U5GX3GLVWGWLU2BDWTHM7ST3MLLJDEAERWU3UEHO7Z2RXHGZU2DA, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:52:18.785925494 (8h52m16.382696517s) [noise=true , address=167.17.72.70:4771 , piece_id=TDLPTUHPNXCICPPXRTRL7WCJSXPX22ONIWWPHNXFUNIKRYPYW52Q, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:52:18.810774903 (8h52m16.407545926s) [noise=true , address=77.90.31.18:4831 , piece_id=4PMIUNE4YXKLIJPML3VSUN4PF32TSLPOOSDZZNAXFWH7WICBOW3Q, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:52:19.002746550 (8h52m16.5995176s) [noise=true , address=193.149.164.71:4304 , piece_id=DTCL54GM76GWKAE7A3X5PUNFYO7H5JIKRT5WBEHYEDF2YNFYTOAA, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:52:19.007614901 (8h52m16.604385924s) [noise=true , address=193.149.164.67:4018 , piece_id=GTQQP355TZZF4W7PPBYZS6ZQ6WYK7I3QIE4DZC7STWAKSBAMT4BQ, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:52:19.010443151 (8h52m16.607214176s) [noise=true , address=193.149.164.69:4101 , piece_id=BTIBVPSORFJQR3CLA5UV62NXNI6GD3JF4D5Z74KU4R4CXB3WHHKA, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:52:19.012614014 (8h52m16.609385038s) [noise=true , address=193.149.164.70:4129 , piece_id=V7WMI7RDYDOAEQYWF2OTZL5LRPUA2S3V5WZN4BKDY6BI4YV3GRPQ, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:52:19.042145380 (8h52m16.638916406s) [noise=true , address=64.49.13.139:4479 , piece_id=XUYHOCSBATYECYHWLVZ4VBCMT3H4O5ETSQ5HZOB42OR3OY4LWJ6Q, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:52:21.314990180 (8h52m18.911761213s) [noise=true , address=167.17.72.71:4810 , piece_id=ANLHBMENEXSGTHBHLOIHMLM6JGHKIU2X6FPQQIUIVMVUOABNK73Q, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:53:18.344002671 (8h53m15.9407737s) [noise=true , address=45.91.116.10:4096 , piece_id=EIKWFKPE5BYSZGV5T4RPQED55PG2FXU3ZKSNRVCBD363YQDHS4BQ, seg_pos={0 0}, upload=9902]: Done uploading piece. err: ecclient: upload failed (node:12MZX1GwZGrEi2q44dPiYrwMoum5A2caZtXTS3qiLNcoyHhon7p, address:45.91.116.10:4096): protocol: write tcp 159.223.143.39:44996->45.91.116.10:4096: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 6 19:53:18.344173472 (8h53m15.940944497s) [noise=true , address=45.91.116.7:4037 , piece_id=YXDMXUBE7CEWLJZP6FPDCRVIDGG5CJ3Y3IKEKO7EE2DQUSG6W5RA, seg_pos={0 0}, upload=9902]: Done uploading piece. err: ecclient: upload failed (node:1iNPwE2XSfXcaXAXcrH2jcyNGYeDiiWvzAcDQffrmjPwTVqVmK, address:45.91.116.7:4037): protocol: write tcp 159.223.143.39:52854->45.91.116.7:4037: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 6 19:53:18.344535130 (8h53m15.941306156s) [noise=true , address=45.91.116.14:4199 , piece_id=EKGDG7LYV3O5NSDL3UPWFKXINYNAOSPYYOWQPTFN326AJ2ZXAXHQ, seg_pos={0 0}, upload=9902]: Done uploading piece. err: ecclient: upload failed (node:1d4B1KMLs2annUuJ3v81FJapS9Yg4omBTj48eWoi6HJxnSEjrf, address:45.91.116.14:4199): protocol: write tcp 159.223.143.39:57596->45.91.116.14:4199: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 6 19:53:18.344687364 (8h53m15.941458387s) [noise=true , address=45.91.116.11:4133 , piece_id=RZWV5JMLOJ36AH7KSDXVTL7CMHI4INEW6YOOQVMCIIC2NGS3XYJA, seg_pos={0 0}, upload=9902]: Done uploading piece. err: ecclient: upload failed (node:12e1wX7hiER8sX4MLm5qLue4mayiN4SxXFdZSxfymRaRZxRGU6E, address:45.91.116.11:4133): protocol: write tcp 159.223.143.39:56440->45.91.116.11:4133: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 6 19:53:18.345220236 (8h53m15.941991259s) [noise=true , address=45.91.116.8:4066 , piece_id=5MZHIJ3ELVBAJ7ABRKG565TLU4A4NKB7X6WAACY2JBSLACL5LGVQ, seg_pos={0 0}, upload=9902]: Done uploading piece. err: ecclient: upload failed (node:123uv49k8LraXJfYQg6KRQthrYs1Egc4c3eZWm5JPvDatFbykm8, address:45.91.116.8:4066): protocol: write tcp 159.223.143.39:44136->45.91.116.8:4066: use of closed network connection; context deadline exceeded Mar 6 19:53:18.345277850 (8h53m15.942048865s) [seg_pos={0 0}, upload=9902]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 6 19:53:18.367560006 (8h53m15.964331035s) [upload=9902]: Done uploading segments... Mar 6 19:53:18.367606646 (8h53m15.964377681s) [upload=9902]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 1.264491094s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................X.......................................] duration: 724.591362ms trace: https://jaeger.dev.storj.io/trace/78791d41d75e16a0 FAIL (2026-03-06 20:19:25 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 19:52:18.382132233 (8h52m15.978903242s) [upload=9902]: Uploading segments... Mar 6 19:52:18.382227489 (8h52m15.978998515s) [upload=9902]: Got next segment. Inline: false Mar 6 19:52:18.382313252 (8h52m15.979084279s) [upload=9902]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 6 19:52:18.382319269 (8h52m15.979090296s) [upload=9902]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 6 19:52:18.411471452 (8h52m16.008242482s) [seg_pos={0 0}, upload=9902]: Begin upload segment... Mar 6 19:52:18.411537096 (8h52m16.008308109s) [seg_pos={0 0}, upload=9902]: Done begin upload segment. Mar 6 19:52:18.411564540 (8h52m16.008335586s) [noise=true , address=45.91.116.7:4037 , piece_id=YXDMXUBE7CEWLJZP6FPDCRVIDGG5CJ3Y3IKEKO7EE2DQUSG6W5RA, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.411626862 (8h52m16.008397897s) [noise=true , address=193.149.164.67:4018 , piece_id=GTQQP355TZZF4W7PPBYZS6ZQ6WYK7I3QIE4DZC7STWAKSBAMT4BQ, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.412356605 (8h52m16.009127631s) [noise=true , address=167.17.72.70:4771 , piece_id=TDLPTUHPNXCICPPXRTRL7WCJSXPX22ONIWWPHNXFUNIKRYPYW52Q, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.412900389 (8h52m16.009671415s) [noise=true , address=145.79.227.39:4022 , piece_id=U5GX3GLVWGWLU2BDWTHM7ST3MLLJDEAERWU3UEHO7Z2RXHGZU2DA, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.413467061 (8h52m16.010238103s) [noise=true , address=45.91.116.11:4133 , piece_id=RZWV5JMLOJ36AH7KSDXVTL7CMHI4INEW6YOOQVMCIIC2NGS3XYJA, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.413653431 (8h52m16.010424457s) [noise=true , address=193.149.164.69:4101 , piece_id=BTIBVPSORFJQR3CLA5UV62NXNI6GD3JF4D5Z74KU4R4CXB3WHHKA, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.414227965 (8h52m16.010998999s) [noise=true , address=45.91.116.14:4199 , piece_id=EKGDG7LYV3O5NSDL3UPWFKXINYNAOSPYYOWQPTFN326AJ2ZXAXHQ, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.415177146 (8h52m16.01194818s) [noise=true , address=77.90.31.16:4763 , piece_id=5Q7H3CU3MWSFPQGC67XMLCMTSUIEJQUV7QTO4UZLZY4MIVM76Z6Q, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.415227889 (8h52m16.011998912s) [noise=true , address=45.91.116.10:4096 , piece_id=EIKWFKPE5BYSZGV5T4RPQED55PG2FXU3ZKSNRVCBD363YQDHS4BQ, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.415728547 (8h52m16.012499581s) [noise=true , address=77.90.31.14:4693 , piece_id=S7NAS2ZBHAQPZRHBRGRSZPLE6COQEEN6SZVMB7X2X6IMXTN5HCBQ, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.415910147 (8h52m16.012681174s) [noise=true , address=193.149.164.70:4129 , piece_id=V7WMI7RDYDOAEQYWF2OTZL5LRPUA2S3V5WZN4BKDY6BI4YV3GRPQ, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.416592462 (8h52m16.01336349s) [noise=true , address=77.90.31.18:4831 , piece_id=4PMIUNE4YXKLIJPML3VSUN4PF32TSLPOOSDZZNAXFWH7WICBOW3Q, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.417118897 (8h52m16.01388993s) [noise=true , address=77.90.31.13:4655 , piece_id=47Q27QI7O2IYW7YQKFMJGGMWPVZWWKQTFVYPUMPUI7ORE7236XIQ, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.417185844 (8h52m16.013956873s) [noise=true , address=193.149.164.71:4304 , piece_id=DTCL54GM76GWKAE7A3X5PUNFYO7H5JIKRT5WBEHYEDF2YNFYTOAA, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.417686790 (8h52m16.014457823s) [noise=true , address=167.17.72.71:4810 , piece_id=ANLHBMENEXSGTHBHLOIHMLM6JGHKIU2X6FPQQIUIVMVUOABNK73Q, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.418236951 (8h52m16.015007982s) [noise=true , address=145.79.227.17:4994 , piece_id=TO3EOAILO4PHF6Z7WQA4RWZNNL3JO6EXXEF2RLW7IHAGWUU2TF4Q, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.419293640 (8h52m16.016064672s) [noise=true , address=64.49.13.139:4479 , piece_id=XUYHOCSBATYECYHWLVZ4VBCMT3H4O5ETSQ5HZOB42OR3OY4LWJ6Q, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.420005502 (8h52m16.016776534s) [noise=true , address=45.91.116.8:4066 , piece_id=5MZHIJ3ELVBAJ7ABRKG565TLU4A4NKB7X6WAACY2JBSLACL5LGVQ, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.483654433 (8h52m16.080425441s) [upload=9902]: Next returned nil segment Mar 6 19:52:18.483671157 (8h52m16.08044219s) [upload=9902]: Waiting for error group managing segments... Mar 6 19:52:18.488078321 (8h52m16.084849398s) [noise=true , address=77.90.31.13:4655 , piece_id=47Q27QI7O2IYW7YQKFMJGGMWPVZWWKQTFVYPUMPUI7ORE7236XIQ, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:52:18.498435619 (8h52m16.095206647s) [noise=true , address=145.79.227.17:4994 , piece_id=TO3EOAILO4PHF6Z7WQA4RWZNNL3JO6EXXEF2RLW7IHAGWUU2TF4Q, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:52:18.502979814 (8h52m16.099750841s) [noise=true , address=77.90.31.16:4763 , piece_id=5Q7H3CU3MWSFPQGC67XMLCMTSUIEJQUV7QTO4UZLZY4MIVM76Z6Q, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:52:18.506259140 (8h52m16.103030166s) [noise=true , address=77.90.31.14:4693 , piece_id=S7NAS2ZBHAQPZRHBRGRSZPLE6COQEEN6SZVMB7X2X6IMXTN5HCBQ, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:52:18.514529389 (8h52m16.111300413s) [noise=true , address=145.79.227.39:4022 , piece_id=U5GX3GLVWGWLU2BDWTHM7ST3MLLJDEAERWU3UEHO7Z2RXHGZU2DA, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:52:18.785925494 (8h52m16.382696517s) [noise=true , address=167.17.72.70:4771 , piece_id=TDLPTUHPNXCICPPXRTRL7WCJSXPX22ONIWWPHNXFUNIKRYPYW52Q, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:52:18.810774903 (8h52m16.407545926s) [noise=true , address=77.90.31.18:4831 , piece_id=4PMIUNE4YXKLIJPML3VSUN4PF32TSLPOOSDZZNAXFWH7WICBOW3Q, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:52:19.002746550 (8h52m16.5995176s) [noise=true , address=193.149.164.71:4304 , piece_id=DTCL54GM76GWKAE7A3X5PUNFYO7H5JIKRT5WBEHYEDF2YNFYTOAA, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:52:19.007614901 (8h52m16.604385924s) [noise=true , address=193.149.164.67:4018 , piece_id=GTQQP355TZZF4W7PPBYZS6ZQ6WYK7I3QIE4DZC7STWAKSBAMT4BQ, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:52:19.010443151 (8h52m16.607214176s) [noise=true , address=193.149.164.69:4101 , piece_id=BTIBVPSORFJQR3CLA5UV62NXNI6GD3JF4D5Z74KU4R4CXB3WHHKA, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:52:19.012614014 (8h52m16.609385038s) [noise=true , address=193.149.164.70:4129 , piece_id=V7WMI7RDYDOAEQYWF2OTZL5LRPUA2S3V5WZN4BKDY6BI4YV3GRPQ, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:52:19.042145380 (8h52m16.638916406s) [noise=true , address=64.49.13.139:4479 , piece_id=XUYHOCSBATYECYHWLVZ4VBCMT3H4O5ETSQ5HZOB42OR3OY4LWJ6Q, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:52:21.314990180 (8h52m18.911761213s) [noise=true , address=167.17.72.71:4810 , piece_id=ANLHBMENEXSGTHBHLOIHMLM6JGHKIU2X6FPQQIUIVMVUOABNK73Q, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:53:18.344002671 (8h53m15.9407737s) [noise=true , address=45.91.116.10:4096 , piece_id=EIKWFKPE5BYSZGV5T4RPQED55PG2FXU3ZKSNRVCBD363YQDHS4BQ, seg_pos={0 0}, upload=9902]: Done uploading piece. err: ecclient: upload failed (node:12MZX1GwZGrEi2q44dPiYrwMoum5A2caZtXTS3qiLNcoyHhon7p, address:45.91.116.10:4096): protocol: write tcp 159.223.143.39:44996->45.91.116.10:4096: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 6 19:53:18.344173472 (8h53m15.940944497s) [noise=true , address=45.91.116.7:4037 , piece_id=YXDMXUBE7CEWLJZP6FPDCRVIDGG5CJ3Y3IKEKO7EE2DQUSG6W5RA, seg_pos={0 0}, upload=9902]: Done uploading piece. err: ecclient: upload failed (node:1iNPwE2XSfXcaXAXcrH2jcyNGYeDiiWvzAcDQffrmjPwTVqVmK, address:45.91.116.7:4037): protocol: write tcp 159.223.143.39:52854->45.91.116.7:4037: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 6 19:53:18.344535130 (8h53m15.941306156s) [noise=true , address=45.91.116.14:4199 , piece_id=EKGDG7LYV3O5NSDL3UPWFKXINYNAOSPYYOWQPTFN326AJ2ZXAXHQ, seg_pos={0 0}, upload=9902]: Done uploading piece. err: ecclient: upload failed (node:1d4B1KMLs2annUuJ3v81FJapS9Yg4omBTj48eWoi6HJxnSEjrf, address:45.91.116.14:4199): protocol: write tcp 159.223.143.39:57596->45.91.116.14:4199: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 6 19:53:18.344687364 (8h53m15.941458387s) [noise=true , address=45.91.116.11:4133 , piece_id=RZWV5JMLOJ36AH7KSDXVTL7CMHI4INEW6YOOQVMCIIC2NGS3XYJA, seg_pos={0 0}, upload=9902]: Done uploading piece. err: ecclient: upload failed (node:12e1wX7hiER8sX4MLm5qLue4mayiN4SxXFdZSxfymRaRZxRGU6E, address:45.91.116.11:4133): protocol: write tcp 159.223.143.39:56440->45.91.116.11:4133: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 6 19:53:18.345220236 (8h53m15.941991259s) [noise=true , address=45.91.116.8:4066 , piece_id=5MZHIJ3ELVBAJ7ABRKG565TLU4A4NKB7X6WAACY2JBSLACL5LGVQ, seg_pos={0 0}, upload=9902]: Done uploading piece. err: ecclient: upload failed (node:123uv49k8LraXJfYQg6KRQthrYs1Egc4c3eZWm5JPvDatFbykm8, address:45.91.116.8:4066): protocol: write tcp 159.223.143.39:44136->45.91.116.8:4066: use of closed network connection; context deadline exceeded Mar 6 19:53:18.345277850 (8h53m15.942048865s) [seg_pos={0 0}, upload=9902]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 6 19:53:18.367560006 (8h53m15.964331035s) [upload=9902]: Done uploading segments... Mar 6 19:53:18.367606646 (8h53m15.964377681s) [upload=9902]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 724.623346ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................X......................................] duration: 1.156265566s trace: https://jaeger.dev.storj.io/trace/78791d41d75e16a0 FAIL (2026-03-06 20:18:52 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 19:52:18.382132233 (8h52m15.978903242s) [upload=9902]: Uploading segments... Mar 6 19:52:18.382227489 (8h52m15.978998515s) [upload=9902]: Got next segment. Inline: false Mar 6 19:52:18.382313252 (8h52m15.979084279s) [upload=9902]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 6 19:52:18.382319269 (8h52m15.979090296s) [upload=9902]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 6 19:52:18.411471452 (8h52m16.008242482s) [seg_pos={0 0}, upload=9902]: Begin upload segment... Mar 6 19:52:18.411537096 (8h52m16.008308109s) [seg_pos={0 0}, upload=9902]: Done begin upload segment. Mar 6 19:52:18.411564540 (8h52m16.008335586s) [noise=true , address=45.91.116.7:4037 , piece_id=YXDMXUBE7CEWLJZP6FPDCRVIDGG5CJ3Y3IKEKO7EE2DQUSG6W5RA, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.411626862 (8h52m16.008397897s) [noise=true , address=193.149.164.67:4018 , piece_id=GTQQP355TZZF4W7PPBYZS6ZQ6WYK7I3QIE4DZC7STWAKSBAMT4BQ, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.412356605 (8h52m16.009127631s) [noise=true , address=167.17.72.70:4771 , piece_id=TDLPTUHPNXCICPPXRTRL7WCJSXPX22ONIWWPHNXFUNIKRYPYW52Q, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.412900389 (8h52m16.009671415s) [noise=true , address=145.79.227.39:4022 , piece_id=U5GX3GLVWGWLU2BDWTHM7ST3MLLJDEAERWU3UEHO7Z2RXHGZU2DA, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.413467061 (8h52m16.010238103s) [noise=true , address=45.91.116.11:4133 , piece_id=RZWV5JMLOJ36AH7KSDXVTL7CMHI4INEW6YOOQVMCIIC2NGS3XYJA, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.413653431 (8h52m16.010424457s) [noise=true , address=193.149.164.69:4101 , piece_id=BTIBVPSORFJQR3CLA5UV62NXNI6GD3JF4D5Z74KU4R4CXB3WHHKA, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.414227965 (8h52m16.010998999s) [noise=true , address=45.91.116.14:4199 , piece_id=EKGDG7LYV3O5NSDL3UPWFKXINYNAOSPYYOWQPTFN326AJ2ZXAXHQ, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.415177146 (8h52m16.01194818s) [noise=true , address=77.90.31.16:4763 , piece_id=5Q7H3CU3MWSFPQGC67XMLCMTSUIEJQUV7QTO4UZLZY4MIVM76Z6Q, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.415227889 (8h52m16.011998912s) [noise=true , address=45.91.116.10:4096 , piece_id=EIKWFKPE5BYSZGV5T4RPQED55PG2FXU3ZKSNRVCBD363YQDHS4BQ, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.415728547 (8h52m16.012499581s) [noise=true , address=77.90.31.14:4693 , piece_id=S7NAS2ZBHAQPZRHBRGRSZPLE6COQEEN6SZVMB7X2X6IMXTN5HCBQ, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.415910147 (8h52m16.012681174s) [noise=true , address=193.149.164.70:4129 , piece_id=V7WMI7RDYDOAEQYWF2OTZL5LRPUA2S3V5WZN4BKDY6BI4YV3GRPQ, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.416592462 (8h52m16.01336349s) [noise=true , address=77.90.31.18:4831 , piece_id=4PMIUNE4YXKLIJPML3VSUN4PF32TSLPOOSDZZNAXFWH7WICBOW3Q, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.417118897 (8h52m16.01388993s) [noise=true , address=77.90.31.13:4655 , piece_id=47Q27QI7O2IYW7YQKFMJGGMWPVZWWKQTFVYPUMPUI7ORE7236XIQ, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.417185844 (8h52m16.013956873s) [noise=true , address=193.149.164.71:4304 , piece_id=DTCL54GM76GWKAE7A3X5PUNFYO7H5JIKRT5WBEHYEDF2YNFYTOAA, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.417686790 (8h52m16.014457823s) [noise=true , address=167.17.72.71:4810 , piece_id=ANLHBMENEXSGTHBHLOIHMLM6JGHKIU2X6FPQQIUIVMVUOABNK73Q, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.418236951 (8h52m16.015007982s) [noise=true , address=145.79.227.17:4994 , piece_id=TO3EOAILO4PHF6Z7WQA4RWZNNL3JO6EXXEF2RLW7IHAGWUU2TF4Q, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.419293640 (8h52m16.016064672s) [noise=true , address=64.49.13.139:4479 , piece_id=XUYHOCSBATYECYHWLVZ4VBCMT3H4O5ETSQ5HZOB42OR3OY4LWJ6Q, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.420005502 (8h52m16.016776534s) [noise=true , address=45.91.116.8:4066 , piece_id=5MZHIJ3ELVBAJ7ABRKG565TLU4A4NKB7X6WAACY2JBSLACL5LGVQ, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.483654433 (8h52m16.080425441s) [upload=9902]: Next returned nil segment Mar 6 19:52:18.483671157 (8h52m16.08044219s) [upload=9902]: Waiting for error group managing segments... Mar 6 19:52:18.488078321 (8h52m16.084849398s) [noise=true , address=77.90.31.13:4655 , piece_id=47Q27QI7O2IYW7YQKFMJGGMWPVZWWKQTFVYPUMPUI7ORE7236XIQ, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:52:18.498435619 (8h52m16.095206647s) [noise=true , address=145.79.227.17:4994 , piece_id=TO3EOAILO4PHF6Z7WQA4RWZNNL3JO6EXXEF2RLW7IHAGWUU2TF4Q, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:52:18.502979814 (8h52m16.099750841s) [noise=true , address=77.90.31.16:4763 , piece_id=5Q7H3CU3MWSFPQGC67XMLCMTSUIEJQUV7QTO4UZLZY4MIVM76Z6Q, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:52:18.506259140 (8h52m16.103030166s) [noise=true , address=77.90.31.14:4693 , piece_id=S7NAS2ZBHAQPZRHBRGRSZPLE6COQEEN6SZVMB7X2X6IMXTN5HCBQ, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:52:18.514529389 (8h52m16.111300413s) [noise=true , address=145.79.227.39:4022 , piece_id=U5GX3GLVWGWLU2BDWTHM7ST3MLLJDEAERWU3UEHO7Z2RXHGZU2DA, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:52:18.785925494 (8h52m16.382696517s) [noise=true , address=167.17.72.70:4771 , piece_id=TDLPTUHPNXCICPPXRTRL7WCJSXPX22ONIWWPHNXFUNIKRYPYW52Q, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:52:18.810774903 (8h52m16.407545926s) [noise=true , address=77.90.31.18:4831 , piece_id=4PMIUNE4YXKLIJPML3VSUN4PF32TSLPOOSDZZNAXFWH7WICBOW3Q, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:52:19.002746550 (8h52m16.5995176s) [noise=true , address=193.149.164.71:4304 , piece_id=DTCL54GM76GWKAE7A3X5PUNFYO7H5JIKRT5WBEHYEDF2YNFYTOAA, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:52:19.007614901 (8h52m16.604385924s) [noise=true , address=193.149.164.67:4018 , piece_id=GTQQP355TZZF4W7PPBYZS6ZQ6WYK7I3QIE4DZC7STWAKSBAMT4BQ, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:52:19.010443151 (8h52m16.607214176s) [noise=true , address=193.149.164.69:4101 , piece_id=BTIBVPSORFJQR3CLA5UV62NXNI6GD3JF4D5Z74KU4R4CXB3WHHKA, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:52:19.012614014 (8h52m16.609385038s) [noise=true , address=193.149.164.70:4129 , piece_id=V7WMI7RDYDOAEQYWF2OTZL5LRPUA2S3V5WZN4BKDY6BI4YV3GRPQ, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:52:19.042145380 (8h52m16.638916406s) [noise=true , address=64.49.13.139:4479 , piece_id=XUYHOCSBATYECYHWLVZ4VBCMT3H4O5ETSQ5HZOB42OR3OY4LWJ6Q, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:52:21.314990180 (8h52m18.911761213s) [noise=true , address=167.17.72.71:4810 , piece_id=ANLHBMENEXSGTHBHLOIHMLM6JGHKIU2X6FPQQIUIVMVUOABNK73Q, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:53:18.344002671 (8h53m15.9407737s) [noise=true , address=45.91.116.10:4096 , piece_id=EIKWFKPE5BYSZGV5T4RPQED55PG2FXU3ZKSNRVCBD363YQDHS4BQ, seg_pos={0 0}, upload=9902]: Done uploading piece. err: ecclient: upload failed (node:12MZX1GwZGrEi2q44dPiYrwMoum5A2caZtXTS3qiLNcoyHhon7p, address:45.91.116.10:4096): protocol: write tcp 159.223.143.39:44996->45.91.116.10:4096: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 6 19:53:18.344173472 (8h53m15.940944497s) [noise=true , address=45.91.116.7:4037 , piece_id=YXDMXUBE7CEWLJZP6FPDCRVIDGG5CJ3Y3IKEKO7EE2DQUSG6W5RA, seg_pos={0 0}, upload=9902]: Done uploading piece. err: ecclient: upload failed (node:1iNPwE2XSfXcaXAXcrH2jcyNGYeDiiWvzAcDQffrmjPwTVqVmK, address:45.91.116.7:4037): protocol: write tcp 159.223.143.39:52854->45.91.116.7:4037: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 6 19:53:18.344535130 (8h53m15.941306156s) [noise=true , address=45.91.116.14:4199 , piece_id=EKGDG7LYV3O5NSDL3UPWFKXINYNAOSPYYOWQPTFN326AJ2ZXAXHQ, seg_pos={0 0}, upload=9902]: Done uploading piece. err: ecclient: upload failed (node:1d4B1KMLs2annUuJ3v81FJapS9Yg4omBTj48eWoi6HJxnSEjrf, address:45.91.116.14:4199): protocol: write tcp 159.223.143.39:57596->45.91.116.14:4199: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 6 19:53:18.344687364 (8h53m15.941458387s) [noise=true , address=45.91.116.11:4133 , piece_id=RZWV5JMLOJ36AH7KSDXVTL7CMHI4INEW6YOOQVMCIIC2NGS3XYJA, seg_pos={0 0}, upload=9902]: Done uploading piece. err: ecclient: upload failed (node:12e1wX7hiER8sX4MLm5qLue4mayiN4SxXFdZSxfymRaRZxRGU6E, address:45.91.116.11:4133): protocol: write tcp 159.223.143.39:56440->45.91.116.11:4133: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 6 19:53:18.345220236 (8h53m15.941991259s) [noise=true , address=45.91.116.8:4066 , piece_id=5MZHIJ3ELVBAJ7ABRKG565TLU4A4NKB7X6WAACY2JBSLACL5LGVQ, seg_pos={0 0}, upload=9902]: Done uploading piece. err: ecclient: upload failed (node:123uv49k8LraXJfYQg6KRQthrYs1Egc4c3eZWm5JPvDatFbykm8, address:45.91.116.8:4066): protocol: write tcp 159.223.143.39:44136->45.91.116.8:4066: use of closed network connection; context deadline exceeded Mar 6 19:53:18.345277850 (8h53m15.942048865s) [seg_pos={0 0}, upload=9902]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 6 19:53:18.367560006 (8h53m15.964331035s) [upload=9902]: Done uploading segments... Mar 6 19:53:18.367606646 (8h53m15.964377681s) [upload=9902]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 1.156294152s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................X.....................................] duration: 1.129651212s trace: https://jaeger.dev.storj.io/trace/78791d41d75e16a0 FAIL (2026-03-06 20:18:04 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 19:52:18.382132233 (8h52m15.978903242s) [upload=9902]: Uploading segments... Mar 6 19:52:18.382227489 (8h52m15.978998515s) [upload=9902]: Got next segment. Inline: false Mar 6 19:52:18.382313252 (8h52m15.979084279s) [upload=9902]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 6 19:52:18.382319269 (8h52m15.979090296s) [upload=9902]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 6 19:52:18.411471452 (8h52m16.008242482s) [seg_pos={0 0}, upload=9902]: Begin upload segment... Mar 6 19:52:18.411537096 (8h52m16.008308109s) [seg_pos={0 0}, upload=9902]: Done begin upload segment. Mar 6 19:52:18.411564540 (8h52m16.008335586s) [noise=true , address=45.91.116.7:4037 , piece_id=YXDMXUBE7CEWLJZP6FPDCRVIDGG5CJ3Y3IKEKO7EE2DQUSG6W5RA, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.411626862 (8h52m16.008397897s) [noise=true , address=193.149.164.67:4018 , piece_id=GTQQP355TZZF4W7PPBYZS6ZQ6WYK7I3QIE4DZC7STWAKSBAMT4BQ, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.412356605 (8h52m16.009127631s) [noise=true , address=167.17.72.70:4771 , piece_id=TDLPTUHPNXCICPPXRTRL7WCJSXPX22ONIWWPHNXFUNIKRYPYW52Q, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.412900389 (8h52m16.009671415s) [noise=true , address=145.79.227.39:4022 , piece_id=U5GX3GLVWGWLU2BDWTHM7ST3MLLJDEAERWU3UEHO7Z2RXHGZU2DA, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.413467061 (8h52m16.010238103s) [noise=true , address=45.91.116.11:4133 , piece_id=RZWV5JMLOJ36AH7KSDXVTL7CMHI4INEW6YOOQVMCIIC2NGS3XYJA, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.413653431 (8h52m16.010424457s) [noise=true , address=193.149.164.69:4101 , piece_id=BTIBVPSORFJQR3CLA5UV62NXNI6GD3JF4D5Z74KU4R4CXB3WHHKA, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.414227965 (8h52m16.010998999s) [noise=true , address=45.91.116.14:4199 , piece_id=EKGDG7LYV3O5NSDL3UPWFKXINYNAOSPYYOWQPTFN326AJ2ZXAXHQ, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.415177146 (8h52m16.01194818s) [noise=true , address=77.90.31.16:4763 , piece_id=5Q7H3CU3MWSFPQGC67XMLCMTSUIEJQUV7QTO4UZLZY4MIVM76Z6Q, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.415227889 (8h52m16.011998912s) [noise=true , address=45.91.116.10:4096 , piece_id=EIKWFKPE5BYSZGV5T4RPQED55PG2FXU3ZKSNRVCBD363YQDHS4BQ, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.415728547 (8h52m16.012499581s) [noise=true , address=77.90.31.14:4693 , piece_id=S7NAS2ZBHAQPZRHBRGRSZPLE6COQEEN6SZVMB7X2X6IMXTN5HCBQ, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.415910147 (8h52m16.012681174s) [noise=true , address=193.149.164.70:4129 , piece_id=V7WMI7RDYDOAEQYWF2OTZL5LRPUA2S3V5WZN4BKDY6BI4YV3GRPQ, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.416592462 (8h52m16.01336349s) [noise=true , address=77.90.31.18:4831 , piece_id=4PMIUNE4YXKLIJPML3VSUN4PF32TSLPOOSDZZNAXFWH7WICBOW3Q, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.417118897 (8h52m16.01388993s) [noise=true , address=77.90.31.13:4655 , piece_id=47Q27QI7O2IYW7YQKFMJGGMWPVZWWKQTFVYPUMPUI7ORE7236XIQ, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.417185844 (8h52m16.013956873s) [noise=true , address=193.149.164.71:4304 , piece_id=DTCL54GM76GWKAE7A3X5PUNFYO7H5JIKRT5WBEHYEDF2YNFYTOAA, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.417686790 (8h52m16.014457823s) [noise=true , address=167.17.72.71:4810 , piece_id=ANLHBMENEXSGTHBHLOIHMLM6JGHKIU2X6FPQQIUIVMVUOABNK73Q, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.418236951 (8h52m16.015007982s) [noise=true , address=145.79.227.17:4994 , piece_id=TO3EOAILO4PHF6Z7WQA4RWZNNL3JO6EXXEF2RLW7IHAGWUU2TF4Q, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.419293640 (8h52m16.016064672s) [noise=true , address=64.49.13.139:4479 , piece_id=XUYHOCSBATYECYHWLVZ4VBCMT3H4O5ETSQ5HZOB42OR3OY4LWJ6Q, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.420005502 (8h52m16.016776534s) [noise=true , address=45.91.116.8:4066 , piece_id=5MZHIJ3ELVBAJ7ABRKG565TLU4A4NKB7X6WAACY2JBSLACL5LGVQ, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.483654433 (8h52m16.080425441s) [upload=9902]: Next returned nil segment Mar 6 19:52:18.483671157 (8h52m16.08044219s) [upload=9902]: Waiting for error group managing segments... Mar 6 19:52:18.488078321 (8h52m16.084849398s) [noise=true , address=77.90.31.13:4655 , piece_id=47Q27QI7O2IYW7YQKFMJGGMWPVZWWKQTFVYPUMPUI7ORE7236XIQ, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:52:18.498435619 (8h52m16.095206647s) [noise=true , address=145.79.227.17:4994 , piece_id=TO3EOAILO4PHF6Z7WQA4RWZNNL3JO6EXXEF2RLW7IHAGWUU2TF4Q, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:52:18.502979814 (8h52m16.099750841s) [noise=true , address=77.90.31.16:4763 , piece_id=5Q7H3CU3MWSFPQGC67XMLCMTSUIEJQUV7QTO4UZLZY4MIVM76Z6Q, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:52:18.506259140 (8h52m16.103030166s) [noise=true , address=77.90.31.14:4693 , piece_id=S7NAS2ZBHAQPZRHBRGRSZPLE6COQEEN6SZVMB7X2X6IMXTN5HCBQ, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:52:18.514529389 (8h52m16.111300413s) [noise=true , address=145.79.227.39:4022 , piece_id=U5GX3GLVWGWLU2BDWTHM7ST3MLLJDEAERWU3UEHO7Z2RXHGZU2DA, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:52:18.785925494 (8h52m16.382696517s) [noise=true , address=167.17.72.70:4771 , piece_id=TDLPTUHPNXCICPPXRTRL7WCJSXPX22ONIWWPHNXFUNIKRYPYW52Q, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:52:18.810774903 (8h52m16.407545926s) [noise=true , address=77.90.31.18:4831 , piece_id=4PMIUNE4YXKLIJPML3VSUN4PF32TSLPOOSDZZNAXFWH7WICBOW3Q, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:52:19.002746550 (8h52m16.5995176s) [noise=true , address=193.149.164.71:4304 , piece_id=DTCL54GM76GWKAE7A3X5PUNFYO7H5JIKRT5WBEHYEDF2YNFYTOAA, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:52:19.007614901 (8h52m16.604385924s) [noise=true , address=193.149.164.67:4018 , piece_id=GTQQP355TZZF4W7PPBYZS6ZQ6WYK7I3QIE4DZC7STWAKSBAMT4BQ, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:52:19.010443151 (8h52m16.607214176s) [noise=true , address=193.149.164.69:4101 , piece_id=BTIBVPSORFJQR3CLA5UV62NXNI6GD3JF4D5Z74KU4R4CXB3WHHKA, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:52:19.012614014 (8h52m16.609385038s) [noise=true , address=193.149.164.70:4129 , piece_id=V7WMI7RDYDOAEQYWF2OTZL5LRPUA2S3V5WZN4BKDY6BI4YV3GRPQ, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:52:19.042145380 (8h52m16.638916406s) [noise=true , address=64.49.13.139:4479 , piece_id=XUYHOCSBATYECYHWLVZ4VBCMT3H4O5ETSQ5HZOB42OR3OY4LWJ6Q, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:52:21.314990180 (8h52m18.911761213s) [noise=true , address=167.17.72.71:4810 , piece_id=ANLHBMENEXSGTHBHLOIHMLM6JGHKIU2X6FPQQIUIVMVUOABNK73Q, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:53:18.344002671 (8h53m15.9407737s) [noise=true , address=45.91.116.10:4096 , piece_id=EIKWFKPE5BYSZGV5T4RPQED55PG2FXU3ZKSNRVCBD363YQDHS4BQ, seg_pos={0 0}, upload=9902]: Done uploading piece. err: ecclient: upload failed (node:12MZX1GwZGrEi2q44dPiYrwMoum5A2caZtXTS3qiLNcoyHhon7p, address:45.91.116.10:4096): protocol: write tcp 159.223.143.39:44996->45.91.116.10:4096: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 6 19:53:18.344173472 (8h53m15.940944497s) [noise=true , address=45.91.116.7:4037 , piece_id=YXDMXUBE7CEWLJZP6FPDCRVIDGG5CJ3Y3IKEKO7EE2DQUSG6W5RA, seg_pos={0 0}, upload=9902]: Done uploading piece. err: ecclient: upload failed (node:1iNPwE2XSfXcaXAXcrH2jcyNGYeDiiWvzAcDQffrmjPwTVqVmK, address:45.91.116.7:4037): protocol: write tcp 159.223.143.39:52854->45.91.116.7:4037: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 6 19:53:18.344535130 (8h53m15.941306156s) [noise=true , address=45.91.116.14:4199 , piece_id=EKGDG7LYV3O5NSDL3UPWFKXINYNAOSPYYOWQPTFN326AJ2ZXAXHQ, seg_pos={0 0}, upload=9902]: Done uploading piece. err: ecclient: upload failed (node:1d4B1KMLs2annUuJ3v81FJapS9Yg4omBTj48eWoi6HJxnSEjrf, address:45.91.116.14:4199): protocol: write tcp 159.223.143.39:57596->45.91.116.14:4199: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 6 19:53:18.344687364 (8h53m15.941458387s) [noise=true , address=45.91.116.11:4133 , piece_id=RZWV5JMLOJ36AH7KSDXVTL7CMHI4INEW6YOOQVMCIIC2NGS3XYJA, seg_pos={0 0}, upload=9902]: Done uploading piece. err: ecclient: upload failed (node:12e1wX7hiER8sX4MLm5qLue4mayiN4SxXFdZSxfymRaRZxRGU6E, address:45.91.116.11:4133): protocol: write tcp 159.223.143.39:56440->45.91.116.11:4133: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 6 19:53:18.345220236 (8h53m15.941991259s) [noise=true , address=45.91.116.8:4066 , piece_id=5MZHIJ3ELVBAJ7ABRKG565TLU4A4NKB7X6WAACY2JBSLACL5LGVQ, seg_pos={0 0}, upload=9902]: Done uploading piece. err: ecclient: upload failed (node:123uv49k8LraXJfYQg6KRQthrYs1Egc4c3eZWm5JPvDatFbykm8, address:45.91.116.8:4066): protocol: write tcp 159.223.143.39:44136->45.91.116.8:4066: use of closed network connection; context deadline exceeded Mar 6 19:53:18.345277850 (8h53m15.942048865s) [seg_pos={0 0}, upload=9902]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 6 19:53:18.367560006 (8h53m15.964331035s) [upload=9902]: Done uploading segments... Mar 6 19:53:18.367606646 (8h53m15.964377681s) [upload=9902]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 1.129677456s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................X....................................] duration: 896.194819ms trace: https://jaeger.dev.storj.io/trace/78791d41d75e16a0 FAIL (2026-03-06 20:17:48 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 19:52:18.382132233 (8h52m15.978903242s) [upload=9902]: Uploading segments... Mar 6 19:52:18.382227489 (8h52m15.978998515s) [upload=9902]: Got next segment. Inline: false Mar 6 19:52:18.382313252 (8h52m15.979084279s) [upload=9902]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 6 19:52:18.382319269 (8h52m15.979090296s) [upload=9902]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 6 19:52:18.411471452 (8h52m16.008242482s) [seg_pos={0 0}, upload=9902]: Begin upload segment... Mar 6 19:52:18.411537096 (8h52m16.008308109s) [seg_pos={0 0}, upload=9902]: Done begin upload segment. Mar 6 19:52:18.411564540 (8h52m16.008335586s) [noise=true , address=45.91.116.7:4037 , piece_id=YXDMXUBE7CEWLJZP6FPDCRVIDGG5CJ3Y3IKEKO7EE2DQUSG6W5RA, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.411626862 (8h52m16.008397897s) [noise=true , address=193.149.164.67:4018 , piece_id=GTQQP355TZZF4W7PPBYZS6ZQ6WYK7I3QIE4DZC7STWAKSBAMT4BQ, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.412356605 (8h52m16.009127631s) [noise=true , address=167.17.72.70:4771 , piece_id=TDLPTUHPNXCICPPXRTRL7WCJSXPX22ONIWWPHNXFUNIKRYPYW52Q, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.412900389 (8h52m16.009671415s) [noise=true , address=145.79.227.39:4022 , piece_id=U5GX3GLVWGWLU2BDWTHM7ST3MLLJDEAERWU3UEHO7Z2RXHGZU2DA, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.413467061 (8h52m16.010238103s) [noise=true , address=45.91.116.11:4133 , piece_id=RZWV5JMLOJ36AH7KSDXVTL7CMHI4INEW6YOOQVMCIIC2NGS3XYJA, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.413653431 (8h52m16.010424457s) [noise=true , address=193.149.164.69:4101 , piece_id=BTIBVPSORFJQR3CLA5UV62NXNI6GD3JF4D5Z74KU4R4CXB3WHHKA, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.414227965 (8h52m16.010998999s) [noise=true , address=45.91.116.14:4199 , piece_id=EKGDG7LYV3O5NSDL3UPWFKXINYNAOSPYYOWQPTFN326AJ2ZXAXHQ, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.415177146 (8h52m16.01194818s) [noise=true , address=77.90.31.16:4763 , piece_id=5Q7H3CU3MWSFPQGC67XMLCMTSUIEJQUV7QTO4UZLZY4MIVM76Z6Q, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.415227889 (8h52m16.011998912s) [noise=true , address=45.91.116.10:4096 , piece_id=EIKWFKPE5BYSZGV5T4RPQED55PG2FXU3ZKSNRVCBD363YQDHS4BQ, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.415728547 (8h52m16.012499581s) [noise=true , address=77.90.31.14:4693 , piece_id=S7NAS2ZBHAQPZRHBRGRSZPLE6COQEEN6SZVMB7X2X6IMXTN5HCBQ, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.415910147 (8h52m16.012681174s) [noise=true , address=193.149.164.70:4129 , piece_id=V7WMI7RDYDOAEQYWF2OTZL5LRPUA2S3V5WZN4BKDY6BI4YV3GRPQ, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.416592462 (8h52m16.01336349s) [noise=true , address=77.90.31.18:4831 , piece_id=4PMIUNE4YXKLIJPML3VSUN4PF32TSLPOOSDZZNAXFWH7WICBOW3Q, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.417118897 (8h52m16.01388993s) [noise=true , address=77.90.31.13:4655 , piece_id=47Q27QI7O2IYW7YQKFMJGGMWPVZWWKQTFVYPUMPUI7ORE7236XIQ, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.417185844 (8h52m16.013956873s) [noise=true , address=193.149.164.71:4304 , piece_id=DTCL54GM76GWKAE7A3X5PUNFYO7H5JIKRT5WBEHYEDF2YNFYTOAA, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.417686790 (8h52m16.014457823s) [noise=true , address=167.17.72.71:4810 , piece_id=ANLHBMENEXSGTHBHLOIHMLM6JGHKIU2X6FPQQIUIVMVUOABNK73Q, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.418236951 (8h52m16.015007982s) [noise=true , address=145.79.227.17:4994 , piece_id=TO3EOAILO4PHF6Z7WQA4RWZNNL3JO6EXXEF2RLW7IHAGWUU2TF4Q, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.419293640 (8h52m16.016064672s) [noise=true , address=64.49.13.139:4479 , piece_id=XUYHOCSBATYECYHWLVZ4VBCMT3H4O5ETSQ5HZOB42OR3OY4LWJ6Q, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.420005502 (8h52m16.016776534s) [noise=true , address=45.91.116.8:4066 , piece_id=5MZHIJ3ELVBAJ7ABRKG565TLU4A4NKB7X6WAACY2JBSLACL5LGVQ, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.483654433 (8h52m16.080425441s) [upload=9902]: Next returned nil segment Mar 6 19:52:18.483671157 (8h52m16.08044219s) [upload=9902]: Waiting for error group managing segments... Mar 6 19:52:18.488078321 (8h52m16.084849398s) [noise=true , address=77.90.31.13:4655 , piece_id=47Q27QI7O2IYW7YQKFMJGGMWPVZWWKQTFVYPUMPUI7ORE7236XIQ, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:52:18.498435619 (8h52m16.095206647s) [noise=true , address=145.79.227.17:4994 , piece_id=TO3EOAILO4PHF6Z7WQA4RWZNNL3JO6EXXEF2RLW7IHAGWUU2TF4Q, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:52:18.502979814 (8h52m16.099750841s) [noise=true , address=77.90.31.16:4763 , piece_id=5Q7H3CU3MWSFPQGC67XMLCMTSUIEJQUV7QTO4UZLZY4MIVM76Z6Q, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:52:18.506259140 (8h52m16.103030166s) [noise=true , address=77.90.31.14:4693 , piece_id=S7NAS2ZBHAQPZRHBRGRSZPLE6COQEEN6SZVMB7X2X6IMXTN5HCBQ, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:52:18.514529389 (8h52m16.111300413s) [noise=true , address=145.79.227.39:4022 , piece_id=U5GX3GLVWGWLU2BDWTHM7ST3MLLJDEAERWU3UEHO7Z2RXHGZU2DA, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:52:18.785925494 (8h52m16.382696517s) [noise=true , address=167.17.72.70:4771 , piece_id=TDLPTUHPNXCICPPXRTRL7WCJSXPX22ONIWWPHNXFUNIKRYPYW52Q, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:52:18.810774903 (8h52m16.407545926s) [noise=true , address=77.90.31.18:4831 , piece_id=4PMIUNE4YXKLIJPML3VSUN4PF32TSLPOOSDZZNAXFWH7WICBOW3Q, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:52:19.002746550 (8h52m16.5995176s) [noise=true , address=193.149.164.71:4304 , piece_id=DTCL54GM76GWKAE7A3X5PUNFYO7H5JIKRT5WBEHYEDF2YNFYTOAA, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:52:19.007614901 (8h52m16.604385924s) [noise=true , address=193.149.164.67:4018 , piece_id=GTQQP355TZZF4W7PPBYZS6ZQ6WYK7I3QIE4DZC7STWAKSBAMT4BQ, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:52:19.010443151 (8h52m16.607214176s) [noise=true , address=193.149.164.69:4101 , piece_id=BTIBVPSORFJQR3CLA5UV62NXNI6GD3JF4D5Z74KU4R4CXB3WHHKA, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:52:19.012614014 (8h52m16.609385038s) [noise=true , address=193.149.164.70:4129 , piece_id=V7WMI7RDYDOAEQYWF2OTZL5LRPUA2S3V5WZN4BKDY6BI4YV3GRPQ, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:52:19.042145380 (8h52m16.638916406s) [noise=true , address=64.49.13.139:4479 , piece_id=XUYHOCSBATYECYHWLVZ4VBCMT3H4O5ETSQ5HZOB42OR3OY4LWJ6Q, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:52:21.314990180 (8h52m18.911761213s) [noise=true , address=167.17.72.71:4810 , piece_id=ANLHBMENEXSGTHBHLOIHMLM6JGHKIU2X6FPQQIUIVMVUOABNK73Q, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:53:18.344002671 (8h53m15.9407737s) [noise=true , address=45.91.116.10:4096 , piece_id=EIKWFKPE5BYSZGV5T4RPQED55PG2FXU3ZKSNRVCBD363YQDHS4BQ, seg_pos={0 0}, upload=9902]: Done uploading piece. err: ecclient: upload failed (node:12MZX1GwZGrEi2q44dPiYrwMoum5A2caZtXTS3qiLNcoyHhon7p, address:45.91.116.10:4096): protocol: write tcp 159.223.143.39:44996->45.91.116.10:4096: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 6 19:53:18.344173472 (8h53m15.940944497s) [noise=true , address=45.91.116.7:4037 , piece_id=YXDMXUBE7CEWLJZP6FPDCRVIDGG5CJ3Y3IKEKO7EE2DQUSG6W5RA, seg_pos={0 0}, upload=9902]: Done uploading piece. err: ecclient: upload failed (node:1iNPwE2XSfXcaXAXcrH2jcyNGYeDiiWvzAcDQffrmjPwTVqVmK, address:45.91.116.7:4037): protocol: write tcp 159.223.143.39:52854->45.91.116.7:4037: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 6 19:53:18.344535130 (8h53m15.941306156s) [noise=true , address=45.91.116.14:4199 , piece_id=EKGDG7LYV3O5NSDL3UPWFKXINYNAOSPYYOWQPTFN326AJ2ZXAXHQ, seg_pos={0 0}, upload=9902]: Done uploading piece. err: ecclient: upload failed (node:1d4B1KMLs2annUuJ3v81FJapS9Yg4omBTj48eWoi6HJxnSEjrf, address:45.91.116.14:4199): protocol: write tcp 159.223.143.39:57596->45.91.116.14:4199: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 6 19:53:18.344687364 (8h53m15.941458387s) [noise=true , address=45.91.116.11:4133 , piece_id=RZWV5JMLOJ36AH7KSDXVTL7CMHI4INEW6YOOQVMCIIC2NGS3XYJA, seg_pos={0 0}, upload=9902]: Done uploading piece. err: ecclient: upload failed (node:12e1wX7hiER8sX4MLm5qLue4mayiN4SxXFdZSxfymRaRZxRGU6E, address:45.91.116.11:4133): protocol: write tcp 159.223.143.39:56440->45.91.116.11:4133: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 6 19:53:18.345220236 (8h53m15.941991259s) [noise=true , address=45.91.116.8:4066 , piece_id=5MZHIJ3ELVBAJ7ABRKG565TLU4A4NKB7X6WAACY2JBSLACL5LGVQ, seg_pos={0 0}, upload=9902]: Done uploading piece. err: ecclient: upload failed (node:123uv49k8LraXJfYQg6KRQthrYs1Egc4c3eZWm5JPvDatFbykm8, address:45.91.116.8:4066): protocol: write tcp 159.223.143.39:44136->45.91.116.8:4066: use of closed network connection; context deadline exceeded Mar 6 19:53:18.345277850 (8h53m15.942048865s) [seg_pos={0 0}, upload=9902]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 6 19:53:18.367560006 (8h53m15.964331035s) [upload=9902]: Done uploading segments... Mar 6 19:53:18.367606646 (8h53m15.964377681s) [upload=9902]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 896.220719ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................X...................................] duration: 840.704895ms trace: https://jaeger.dev.storj.io/trace/78791d41d75e16a0 FAIL (2026-03-06 20:17:32 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 19:52:18.382132233 (8h52m15.978903242s) [upload=9902]: Uploading segments... Mar 6 19:52:18.382227489 (8h52m15.978998515s) [upload=9902]: Got next segment. Inline: false Mar 6 19:52:18.382313252 (8h52m15.979084279s) [upload=9902]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 6 19:52:18.382319269 (8h52m15.979090296s) [upload=9902]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 6 19:52:18.411471452 (8h52m16.008242482s) [seg_pos={0 0}, upload=9902]: Begin upload segment... Mar 6 19:52:18.411537096 (8h52m16.008308109s) [seg_pos={0 0}, upload=9902]: Done begin upload segment. Mar 6 19:52:18.411564540 (8h52m16.008335586s) [noise=true , address=45.91.116.7:4037 , piece_id=YXDMXUBE7CEWLJZP6FPDCRVIDGG5CJ3Y3IKEKO7EE2DQUSG6W5RA, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.411626862 (8h52m16.008397897s) [noise=true , address=193.149.164.67:4018 , piece_id=GTQQP355TZZF4W7PPBYZS6ZQ6WYK7I3QIE4DZC7STWAKSBAMT4BQ, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.412356605 (8h52m16.009127631s) [noise=true , address=167.17.72.70:4771 , piece_id=TDLPTUHPNXCICPPXRTRL7WCJSXPX22ONIWWPHNXFUNIKRYPYW52Q, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.412900389 (8h52m16.009671415s) [noise=true , address=145.79.227.39:4022 , piece_id=U5GX3GLVWGWLU2BDWTHM7ST3MLLJDEAERWU3UEHO7Z2RXHGZU2DA, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.413467061 (8h52m16.010238103s) [noise=true , address=45.91.116.11:4133 , piece_id=RZWV5JMLOJ36AH7KSDXVTL7CMHI4INEW6YOOQVMCIIC2NGS3XYJA, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.413653431 (8h52m16.010424457s) [noise=true , address=193.149.164.69:4101 , piece_id=BTIBVPSORFJQR3CLA5UV62NXNI6GD3JF4D5Z74KU4R4CXB3WHHKA, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.414227965 (8h52m16.010998999s) [noise=true , address=45.91.116.14:4199 , piece_id=EKGDG7LYV3O5NSDL3UPWFKXINYNAOSPYYOWQPTFN326AJ2ZXAXHQ, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.415177146 (8h52m16.01194818s) [noise=true , address=77.90.31.16:4763 , piece_id=5Q7H3CU3MWSFPQGC67XMLCMTSUIEJQUV7QTO4UZLZY4MIVM76Z6Q, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.415227889 (8h52m16.011998912s) [noise=true , address=45.91.116.10:4096 , piece_id=EIKWFKPE5BYSZGV5T4RPQED55PG2FXU3ZKSNRVCBD363YQDHS4BQ, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.415728547 (8h52m16.012499581s) [noise=true , address=77.90.31.14:4693 , piece_id=S7NAS2ZBHAQPZRHBRGRSZPLE6COQEEN6SZVMB7X2X6IMXTN5HCBQ, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.415910147 (8h52m16.012681174s) [noise=true , address=193.149.164.70:4129 , piece_id=V7WMI7RDYDOAEQYWF2OTZL5LRPUA2S3V5WZN4BKDY6BI4YV3GRPQ, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.416592462 (8h52m16.01336349s) [noise=true , address=77.90.31.18:4831 , piece_id=4PMIUNE4YXKLIJPML3VSUN4PF32TSLPOOSDZZNAXFWH7WICBOW3Q, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.417118897 (8h52m16.01388993s) [noise=true , address=77.90.31.13:4655 , piece_id=47Q27QI7O2IYW7YQKFMJGGMWPVZWWKQTFVYPUMPUI7ORE7236XIQ, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.417185844 (8h52m16.013956873s) [noise=true , address=193.149.164.71:4304 , piece_id=DTCL54GM76GWKAE7A3X5PUNFYO7H5JIKRT5WBEHYEDF2YNFYTOAA, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.417686790 (8h52m16.014457823s) [noise=true , address=167.17.72.71:4810 , piece_id=ANLHBMENEXSGTHBHLOIHMLM6JGHKIU2X6FPQQIUIVMVUOABNK73Q, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.418236951 (8h52m16.015007982s) [noise=true , address=145.79.227.17:4994 , piece_id=TO3EOAILO4PHF6Z7WQA4RWZNNL3JO6EXXEF2RLW7IHAGWUU2TF4Q, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.419293640 (8h52m16.016064672s) [noise=true , address=64.49.13.139:4479 , piece_id=XUYHOCSBATYECYHWLVZ4VBCMT3H4O5ETSQ5HZOB42OR3OY4LWJ6Q, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.420005502 (8h52m16.016776534s) [noise=true , address=45.91.116.8:4066 , piece_id=5MZHIJ3ELVBAJ7ABRKG565TLU4A4NKB7X6WAACY2JBSLACL5LGVQ, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.483654433 (8h52m16.080425441s) [upload=9902]: Next returned nil segment Mar 6 19:52:18.483671157 (8h52m16.08044219s) [upload=9902]: Waiting for error group managing segments... Mar 6 19:52:18.488078321 (8h52m16.084849398s) [noise=true , address=77.90.31.13:4655 , piece_id=47Q27QI7O2IYW7YQKFMJGGMWPVZWWKQTFVYPUMPUI7ORE7236XIQ, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:52:18.498435619 (8h52m16.095206647s) [noise=true , address=145.79.227.17:4994 , piece_id=TO3EOAILO4PHF6Z7WQA4RWZNNL3JO6EXXEF2RLW7IHAGWUU2TF4Q, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:52:18.502979814 (8h52m16.099750841s) [noise=true , address=77.90.31.16:4763 , piece_id=5Q7H3CU3MWSFPQGC67XMLCMTSUIEJQUV7QTO4UZLZY4MIVM76Z6Q, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:52:18.506259140 (8h52m16.103030166s) [noise=true , address=77.90.31.14:4693 , piece_id=S7NAS2ZBHAQPZRHBRGRSZPLE6COQEEN6SZVMB7X2X6IMXTN5HCBQ, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:52:18.514529389 (8h52m16.111300413s) [noise=true , address=145.79.227.39:4022 , piece_id=U5GX3GLVWGWLU2BDWTHM7ST3MLLJDEAERWU3UEHO7Z2RXHGZU2DA, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:52:18.785925494 (8h52m16.382696517s) [noise=true , address=167.17.72.70:4771 , piece_id=TDLPTUHPNXCICPPXRTRL7WCJSXPX22ONIWWPHNXFUNIKRYPYW52Q, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:52:18.810774903 (8h52m16.407545926s) [noise=true , address=77.90.31.18:4831 , piece_id=4PMIUNE4YXKLIJPML3VSUN4PF32TSLPOOSDZZNAXFWH7WICBOW3Q, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:52:19.002746550 (8h52m16.5995176s) [noise=true , address=193.149.164.71:4304 , piece_id=DTCL54GM76GWKAE7A3X5PUNFYO7H5JIKRT5WBEHYEDF2YNFYTOAA, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:52:19.007614901 (8h52m16.604385924s) [noise=true , address=193.149.164.67:4018 , piece_id=GTQQP355TZZF4W7PPBYZS6ZQ6WYK7I3QIE4DZC7STWAKSBAMT4BQ, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:52:19.010443151 (8h52m16.607214176s) [noise=true , address=193.149.164.69:4101 , piece_id=BTIBVPSORFJQR3CLA5UV62NXNI6GD3JF4D5Z74KU4R4CXB3WHHKA, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:52:19.012614014 (8h52m16.609385038s) [noise=true , address=193.149.164.70:4129 , piece_id=V7WMI7RDYDOAEQYWF2OTZL5LRPUA2S3V5WZN4BKDY6BI4YV3GRPQ, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:52:19.042145380 (8h52m16.638916406s) [noise=true , address=64.49.13.139:4479 , piece_id=XUYHOCSBATYECYHWLVZ4VBCMT3H4O5ETSQ5HZOB42OR3OY4LWJ6Q, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:52:21.314990180 (8h52m18.911761213s) [noise=true , address=167.17.72.71:4810 , piece_id=ANLHBMENEXSGTHBHLOIHMLM6JGHKIU2X6FPQQIUIVMVUOABNK73Q, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:53:18.344002671 (8h53m15.9407737s) [noise=true , address=45.91.116.10:4096 , piece_id=EIKWFKPE5BYSZGV5T4RPQED55PG2FXU3ZKSNRVCBD363YQDHS4BQ, seg_pos={0 0}, upload=9902]: Done uploading piece. err: ecclient: upload failed (node:12MZX1GwZGrEi2q44dPiYrwMoum5A2caZtXTS3qiLNcoyHhon7p, address:45.91.116.10:4096): protocol: write tcp 159.223.143.39:44996->45.91.116.10:4096: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 6 19:53:18.344173472 (8h53m15.940944497s) [noise=true , address=45.91.116.7:4037 , piece_id=YXDMXUBE7CEWLJZP6FPDCRVIDGG5CJ3Y3IKEKO7EE2DQUSG6W5RA, seg_pos={0 0}, upload=9902]: Done uploading piece. err: ecclient: upload failed (node:1iNPwE2XSfXcaXAXcrH2jcyNGYeDiiWvzAcDQffrmjPwTVqVmK, address:45.91.116.7:4037): protocol: write tcp 159.223.143.39:52854->45.91.116.7:4037: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 6 19:53:18.344535130 (8h53m15.941306156s) [noise=true , address=45.91.116.14:4199 , piece_id=EKGDG7LYV3O5NSDL3UPWFKXINYNAOSPYYOWQPTFN326AJ2ZXAXHQ, seg_pos={0 0}, upload=9902]: Done uploading piece. err: ecclient: upload failed (node:1d4B1KMLs2annUuJ3v81FJapS9Yg4omBTj48eWoi6HJxnSEjrf, address:45.91.116.14:4199): protocol: write tcp 159.223.143.39:57596->45.91.116.14:4199: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 6 19:53:18.344687364 (8h53m15.941458387s) [noise=true , address=45.91.116.11:4133 , piece_id=RZWV5JMLOJ36AH7KSDXVTL7CMHI4INEW6YOOQVMCIIC2NGS3XYJA, seg_pos={0 0}, upload=9902]: Done uploading piece. err: ecclient: upload failed (node:12e1wX7hiER8sX4MLm5qLue4mayiN4SxXFdZSxfymRaRZxRGU6E, address:45.91.116.11:4133): protocol: write tcp 159.223.143.39:56440->45.91.116.11:4133: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 6 19:53:18.345220236 (8h53m15.941991259s) [noise=true , address=45.91.116.8:4066 , piece_id=5MZHIJ3ELVBAJ7ABRKG565TLU4A4NKB7X6WAACY2JBSLACL5LGVQ, seg_pos={0 0}, upload=9902]: Done uploading piece. err: ecclient: upload failed (node:123uv49k8LraXJfYQg6KRQthrYs1Egc4c3eZWm5JPvDatFbykm8, address:45.91.116.8:4066): protocol: write tcp 159.223.143.39:44136->45.91.116.8:4066: use of closed network connection; context deadline exceeded Mar 6 19:53:18.345277850 (8h53m15.942048865s) [seg_pos={0 0}, upload=9902]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 6 19:53:18.367560006 (8h53m15.964331035s) [upload=9902]: Done uploading segments... Mar 6 19:53:18.367606646 (8h53m15.964377681s) [upload=9902]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 840.733954ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................X..................................] duration: 920.973115ms trace: https://jaeger.dev.storj.io/trace/78791d41d75e16a0 FAIL (2026-03-06 20:16:20 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 19:52:18.382132233 (8h52m15.978903242s) [upload=9902]: Uploading segments... Mar 6 19:52:18.382227489 (8h52m15.978998515s) [upload=9902]: Got next segment. Inline: false Mar 6 19:52:18.382313252 (8h52m15.979084279s) [upload=9902]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 6 19:52:18.382319269 (8h52m15.979090296s) [upload=9902]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 6 19:52:18.411471452 (8h52m16.008242482s) [seg_pos={0 0}, upload=9902]: Begin upload segment... Mar 6 19:52:18.411537096 (8h52m16.008308109s) [seg_pos={0 0}, upload=9902]: Done begin upload segment. Mar 6 19:52:18.411564540 (8h52m16.008335586s) [noise=true , address=45.91.116.7:4037 , piece_id=YXDMXUBE7CEWLJZP6FPDCRVIDGG5CJ3Y3IKEKO7EE2DQUSG6W5RA, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.411626862 (8h52m16.008397897s) [noise=true , address=193.149.164.67:4018 , piece_id=GTQQP355TZZF4W7PPBYZS6ZQ6WYK7I3QIE4DZC7STWAKSBAMT4BQ, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.412356605 (8h52m16.009127631s) [noise=true , address=167.17.72.70:4771 , piece_id=TDLPTUHPNXCICPPXRTRL7WCJSXPX22ONIWWPHNXFUNIKRYPYW52Q, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.412900389 (8h52m16.009671415s) [noise=true , address=145.79.227.39:4022 , piece_id=U5GX3GLVWGWLU2BDWTHM7ST3MLLJDEAERWU3UEHO7Z2RXHGZU2DA, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.413467061 (8h52m16.010238103s) [noise=true , address=45.91.116.11:4133 , piece_id=RZWV5JMLOJ36AH7KSDXVTL7CMHI4INEW6YOOQVMCIIC2NGS3XYJA, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.413653431 (8h52m16.010424457s) [noise=true , address=193.149.164.69:4101 , piece_id=BTIBVPSORFJQR3CLA5UV62NXNI6GD3JF4D5Z74KU4R4CXB3WHHKA, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.414227965 (8h52m16.010998999s) [noise=true , address=45.91.116.14:4199 , piece_id=EKGDG7LYV3O5NSDL3UPWFKXINYNAOSPYYOWQPTFN326AJ2ZXAXHQ, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.415177146 (8h52m16.01194818s) [noise=true , address=77.90.31.16:4763 , piece_id=5Q7H3CU3MWSFPQGC67XMLCMTSUIEJQUV7QTO4UZLZY4MIVM76Z6Q, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.415227889 (8h52m16.011998912s) [noise=true , address=45.91.116.10:4096 , piece_id=EIKWFKPE5BYSZGV5T4RPQED55PG2FXU3ZKSNRVCBD363YQDHS4BQ, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.415728547 (8h52m16.012499581s) [noise=true , address=77.90.31.14:4693 , piece_id=S7NAS2ZBHAQPZRHBRGRSZPLE6COQEEN6SZVMB7X2X6IMXTN5HCBQ, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.415910147 (8h52m16.012681174s) [noise=true , address=193.149.164.70:4129 , piece_id=V7WMI7RDYDOAEQYWF2OTZL5LRPUA2S3V5WZN4BKDY6BI4YV3GRPQ, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.416592462 (8h52m16.01336349s) [noise=true , address=77.90.31.18:4831 , piece_id=4PMIUNE4YXKLIJPML3VSUN4PF32TSLPOOSDZZNAXFWH7WICBOW3Q, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.417118897 (8h52m16.01388993s) [noise=true , address=77.90.31.13:4655 , piece_id=47Q27QI7O2IYW7YQKFMJGGMWPVZWWKQTFVYPUMPUI7ORE7236XIQ, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.417185844 (8h52m16.013956873s) [noise=true , address=193.149.164.71:4304 , piece_id=DTCL54GM76GWKAE7A3X5PUNFYO7H5JIKRT5WBEHYEDF2YNFYTOAA, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.417686790 (8h52m16.014457823s) [noise=true , address=167.17.72.71:4810 , piece_id=ANLHBMENEXSGTHBHLOIHMLM6JGHKIU2X6FPQQIUIVMVUOABNK73Q, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.418236951 (8h52m16.015007982s) [noise=true , address=145.79.227.17:4994 , piece_id=TO3EOAILO4PHF6Z7WQA4RWZNNL3JO6EXXEF2RLW7IHAGWUU2TF4Q, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.419293640 (8h52m16.016064672s) [noise=true , address=64.49.13.139:4479 , piece_id=XUYHOCSBATYECYHWLVZ4VBCMT3H4O5ETSQ5HZOB42OR3OY4LWJ6Q, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.420005502 (8h52m16.016776534s) [noise=true , address=45.91.116.8:4066 , piece_id=5MZHIJ3ELVBAJ7ABRKG565TLU4A4NKB7X6WAACY2JBSLACL5LGVQ, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.483654433 (8h52m16.080425441s) [upload=9902]: Next returned nil segment Mar 6 19:52:18.483671157 (8h52m16.08044219s) [upload=9902]: Waiting for error group managing segments... Mar 6 19:52:18.488078321 (8h52m16.084849398s) [noise=true , address=77.90.31.13:4655 , piece_id=47Q27QI7O2IYW7YQKFMJGGMWPVZWWKQTFVYPUMPUI7ORE7236XIQ, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:52:18.498435619 (8h52m16.095206647s) [noise=true , address=145.79.227.17:4994 , piece_id=TO3EOAILO4PHF6Z7WQA4RWZNNL3JO6EXXEF2RLW7IHAGWUU2TF4Q, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:52:18.502979814 (8h52m16.099750841s) [noise=true , address=77.90.31.16:4763 , piece_id=5Q7H3CU3MWSFPQGC67XMLCMTSUIEJQUV7QTO4UZLZY4MIVM76Z6Q, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:52:18.506259140 (8h52m16.103030166s) [noise=true , address=77.90.31.14:4693 , piece_id=S7NAS2ZBHAQPZRHBRGRSZPLE6COQEEN6SZVMB7X2X6IMXTN5HCBQ, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:52:18.514529389 (8h52m16.111300413s) [noise=true , address=145.79.227.39:4022 , piece_id=U5GX3GLVWGWLU2BDWTHM7ST3MLLJDEAERWU3UEHO7Z2RXHGZU2DA, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:52:18.785925494 (8h52m16.382696517s) [noise=true , address=167.17.72.70:4771 , piece_id=TDLPTUHPNXCICPPXRTRL7WCJSXPX22ONIWWPHNXFUNIKRYPYW52Q, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:52:18.810774903 (8h52m16.407545926s) [noise=true , address=77.90.31.18:4831 , piece_id=4PMIUNE4YXKLIJPML3VSUN4PF32TSLPOOSDZZNAXFWH7WICBOW3Q, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:52:19.002746550 (8h52m16.5995176s) [noise=true , address=193.149.164.71:4304 , piece_id=DTCL54GM76GWKAE7A3X5PUNFYO7H5JIKRT5WBEHYEDF2YNFYTOAA, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:52:19.007614901 (8h52m16.604385924s) [noise=true , address=193.149.164.67:4018 , piece_id=GTQQP355TZZF4W7PPBYZS6ZQ6WYK7I3QIE4DZC7STWAKSBAMT4BQ, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:52:19.010443151 (8h52m16.607214176s) [noise=true , address=193.149.164.69:4101 , piece_id=BTIBVPSORFJQR3CLA5UV62NXNI6GD3JF4D5Z74KU4R4CXB3WHHKA, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:52:19.012614014 (8h52m16.609385038s) [noise=true , address=193.149.164.70:4129 , piece_id=V7WMI7RDYDOAEQYWF2OTZL5LRPUA2S3V5WZN4BKDY6BI4YV3GRPQ, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:52:19.042145380 (8h52m16.638916406s) [noise=true , address=64.49.13.139:4479 , piece_id=XUYHOCSBATYECYHWLVZ4VBCMT3H4O5ETSQ5HZOB42OR3OY4LWJ6Q, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:52:21.314990180 (8h52m18.911761213s) [noise=true , address=167.17.72.71:4810 , piece_id=ANLHBMENEXSGTHBHLOIHMLM6JGHKIU2X6FPQQIUIVMVUOABNK73Q, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:53:18.344002671 (8h53m15.9407737s) [noise=true , address=45.91.116.10:4096 , piece_id=EIKWFKPE5BYSZGV5T4RPQED55PG2FXU3ZKSNRVCBD363YQDHS4BQ, seg_pos={0 0}, upload=9902]: Done uploading piece. err: ecclient: upload failed (node:12MZX1GwZGrEi2q44dPiYrwMoum5A2caZtXTS3qiLNcoyHhon7p, address:45.91.116.10:4096): protocol: write tcp 159.223.143.39:44996->45.91.116.10:4096: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 6 19:53:18.344173472 (8h53m15.940944497s) [noise=true , address=45.91.116.7:4037 , piece_id=YXDMXUBE7CEWLJZP6FPDCRVIDGG5CJ3Y3IKEKO7EE2DQUSG6W5RA, seg_pos={0 0}, upload=9902]: Done uploading piece. err: ecclient: upload failed (node:1iNPwE2XSfXcaXAXcrH2jcyNGYeDiiWvzAcDQffrmjPwTVqVmK, address:45.91.116.7:4037): protocol: write tcp 159.223.143.39:52854->45.91.116.7:4037: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 6 19:53:18.344535130 (8h53m15.941306156s) [noise=true , address=45.91.116.14:4199 , piece_id=EKGDG7LYV3O5NSDL3UPWFKXINYNAOSPYYOWQPTFN326AJ2ZXAXHQ, seg_pos={0 0}, upload=9902]: Done uploading piece. err: ecclient: upload failed (node:1d4B1KMLs2annUuJ3v81FJapS9Yg4omBTj48eWoi6HJxnSEjrf, address:45.91.116.14:4199): protocol: write tcp 159.223.143.39:57596->45.91.116.14:4199: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 6 19:53:18.344687364 (8h53m15.941458387s) [noise=true , address=45.91.116.11:4133 , piece_id=RZWV5JMLOJ36AH7KSDXVTL7CMHI4INEW6YOOQVMCIIC2NGS3XYJA, seg_pos={0 0}, upload=9902]: Done uploading piece. err: ecclient: upload failed (node:12e1wX7hiER8sX4MLm5qLue4mayiN4SxXFdZSxfymRaRZxRGU6E, address:45.91.116.11:4133): protocol: write tcp 159.223.143.39:56440->45.91.116.11:4133: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 6 19:53:18.345220236 (8h53m15.941991259s) [noise=true , address=45.91.116.8:4066 , piece_id=5MZHIJ3ELVBAJ7ABRKG565TLU4A4NKB7X6WAACY2JBSLACL5LGVQ, seg_pos={0 0}, upload=9902]: Done uploading piece. err: ecclient: upload failed (node:123uv49k8LraXJfYQg6KRQthrYs1Egc4c3eZWm5JPvDatFbykm8, address:45.91.116.8:4066): protocol: write tcp 159.223.143.39:44136->45.91.116.8:4066: use of closed network connection; context deadline exceeded Mar 6 19:53:18.345277850 (8h53m15.942048865s) [seg_pos={0 0}, upload=9902]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 6 19:53:18.367560006 (8h53m15.964331035s) [upload=9902]: Done uploading segments... Mar 6 19:53:18.367606646 (8h53m15.964377681s) [upload=9902]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 920.996654ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................X.................................] duration: 1.356409321s trace: https://jaeger.dev.storj.io/trace/78791d41d75e16a0 FAIL (2026-03-06 20:16:01 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 19:52:18.382132233 (8h52m15.978903242s) [upload=9902]: Uploading segments... Mar 6 19:52:18.382227489 (8h52m15.978998515s) [upload=9902]: Got next segment. Inline: false Mar 6 19:52:18.382313252 (8h52m15.979084279s) [upload=9902]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 6 19:52:18.382319269 (8h52m15.979090296s) [upload=9902]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 6 19:52:18.411471452 (8h52m16.008242482s) [seg_pos={0 0}, upload=9902]: Begin upload segment... Mar 6 19:52:18.411537096 (8h52m16.008308109s) [seg_pos={0 0}, upload=9902]: Done begin upload segment. Mar 6 19:52:18.411564540 (8h52m16.008335586s) [noise=true , address=45.91.116.7:4037 , piece_id=YXDMXUBE7CEWLJZP6FPDCRVIDGG5CJ3Y3IKEKO7EE2DQUSG6W5RA, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.411626862 (8h52m16.008397897s) [noise=true , address=193.149.164.67:4018 , piece_id=GTQQP355TZZF4W7PPBYZS6ZQ6WYK7I3QIE4DZC7STWAKSBAMT4BQ, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.412356605 (8h52m16.009127631s) [noise=true , address=167.17.72.70:4771 , piece_id=TDLPTUHPNXCICPPXRTRL7WCJSXPX22ONIWWPHNXFUNIKRYPYW52Q, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.412900389 (8h52m16.009671415s) [noise=true , address=145.79.227.39:4022 , piece_id=U5GX3GLVWGWLU2BDWTHM7ST3MLLJDEAERWU3UEHO7Z2RXHGZU2DA, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.413467061 (8h52m16.010238103s) [noise=true , address=45.91.116.11:4133 , piece_id=RZWV5JMLOJ36AH7KSDXVTL7CMHI4INEW6YOOQVMCIIC2NGS3XYJA, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.413653431 (8h52m16.010424457s) [noise=true , address=193.149.164.69:4101 , piece_id=BTIBVPSORFJQR3CLA5UV62NXNI6GD3JF4D5Z74KU4R4CXB3WHHKA, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.414227965 (8h52m16.010998999s) [noise=true , address=45.91.116.14:4199 , piece_id=EKGDG7LYV3O5NSDL3UPWFKXINYNAOSPYYOWQPTFN326AJ2ZXAXHQ, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.415177146 (8h52m16.01194818s) [noise=true , address=77.90.31.16:4763 , piece_id=5Q7H3CU3MWSFPQGC67XMLCMTSUIEJQUV7QTO4UZLZY4MIVM76Z6Q, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.415227889 (8h52m16.011998912s) [noise=true , address=45.91.116.10:4096 , piece_id=EIKWFKPE5BYSZGV5T4RPQED55PG2FXU3ZKSNRVCBD363YQDHS4BQ, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.415728547 (8h52m16.012499581s) [noise=true , address=77.90.31.14:4693 , piece_id=S7NAS2ZBHAQPZRHBRGRSZPLE6COQEEN6SZVMB7X2X6IMXTN5HCBQ, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.415910147 (8h52m16.012681174s) [noise=true , address=193.149.164.70:4129 , piece_id=V7WMI7RDYDOAEQYWF2OTZL5LRPUA2S3V5WZN4BKDY6BI4YV3GRPQ, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.416592462 (8h52m16.01336349s) [noise=true , address=77.90.31.18:4831 , piece_id=4PMIUNE4YXKLIJPML3VSUN4PF32TSLPOOSDZZNAXFWH7WICBOW3Q, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.417118897 (8h52m16.01388993s) [noise=true , address=77.90.31.13:4655 , piece_id=47Q27QI7O2IYW7YQKFMJGGMWPVZWWKQTFVYPUMPUI7ORE7236XIQ, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.417185844 (8h52m16.013956873s) [noise=true , address=193.149.164.71:4304 , piece_id=DTCL54GM76GWKAE7A3X5PUNFYO7H5JIKRT5WBEHYEDF2YNFYTOAA, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.417686790 (8h52m16.014457823s) [noise=true , address=167.17.72.71:4810 , piece_id=ANLHBMENEXSGTHBHLOIHMLM6JGHKIU2X6FPQQIUIVMVUOABNK73Q, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.418236951 (8h52m16.015007982s) [noise=true , address=145.79.227.17:4994 , piece_id=TO3EOAILO4PHF6Z7WQA4RWZNNL3JO6EXXEF2RLW7IHAGWUU2TF4Q, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.419293640 (8h52m16.016064672s) [noise=true , address=64.49.13.139:4479 , piece_id=XUYHOCSBATYECYHWLVZ4VBCMT3H4O5ETSQ5HZOB42OR3OY4LWJ6Q, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.420005502 (8h52m16.016776534s) [noise=true , address=45.91.116.8:4066 , piece_id=5MZHIJ3ELVBAJ7ABRKG565TLU4A4NKB7X6WAACY2JBSLACL5LGVQ, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.483654433 (8h52m16.080425441s) [upload=9902]: Next returned nil segment Mar 6 19:52:18.483671157 (8h52m16.08044219s) [upload=9902]: Waiting for error group managing segments... Mar 6 19:52:18.488078321 (8h52m16.084849398s) [noise=true , address=77.90.31.13:4655 , piece_id=47Q27QI7O2IYW7YQKFMJGGMWPVZWWKQTFVYPUMPUI7ORE7236XIQ, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:52:18.498435619 (8h52m16.095206647s) [noise=true , address=145.79.227.17:4994 , piece_id=TO3EOAILO4PHF6Z7WQA4RWZNNL3JO6EXXEF2RLW7IHAGWUU2TF4Q, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:52:18.502979814 (8h52m16.099750841s) [noise=true , address=77.90.31.16:4763 , piece_id=5Q7H3CU3MWSFPQGC67XMLCMTSUIEJQUV7QTO4UZLZY4MIVM76Z6Q, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:52:18.506259140 (8h52m16.103030166s) [noise=true , address=77.90.31.14:4693 , piece_id=S7NAS2ZBHAQPZRHBRGRSZPLE6COQEEN6SZVMB7X2X6IMXTN5HCBQ, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:52:18.514529389 (8h52m16.111300413s) [noise=true , address=145.79.227.39:4022 , piece_id=U5GX3GLVWGWLU2BDWTHM7ST3MLLJDEAERWU3UEHO7Z2RXHGZU2DA, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:52:18.785925494 (8h52m16.382696517s) [noise=true , address=167.17.72.70:4771 , piece_id=TDLPTUHPNXCICPPXRTRL7WCJSXPX22ONIWWPHNXFUNIKRYPYW52Q, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:52:18.810774903 (8h52m16.407545926s) [noise=true , address=77.90.31.18:4831 , piece_id=4PMIUNE4YXKLIJPML3VSUN4PF32TSLPOOSDZZNAXFWH7WICBOW3Q, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:52:19.002746550 (8h52m16.5995176s) [noise=true , address=193.149.164.71:4304 , piece_id=DTCL54GM76GWKAE7A3X5PUNFYO7H5JIKRT5WBEHYEDF2YNFYTOAA, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:52:19.007614901 (8h52m16.604385924s) [noise=true , address=193.149.164.67:4018 , piece_id=GTQQP355TZZF4W7PPBYZS6ZQ6WYK7I3QIE4DZC7STWAKSBAMT4BQ, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:52:19.010443151 (8h52m16.607214176s) [noise=true , address=193.149.164.69:4101 , piece_id=BTIBVPSORFJQR3CLA5UV62NXNI6GD3JF4D5Z74KU4R4CXB3WHHKA, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:52:19.012614014 (8h52m16.609385038s) [noise=true , address=193.149.164.70:4129 , piece_id=V7WMI7RDYDOAEQYWF2OTZL5LRPUA2S3V5WZN4BKDY6BI4YV3GRPQ, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:52:19.042145380 (8h52m16.638916406s) [noise=true , address=64.49.13.139:4479 , piece_id=XUYHOCSBATYECYHWLVZ4VBCMT3H4O5ETSQ5HZOB42OR3OY4LWJ6Q, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:52:21.314990180 (8h52m18.911761213s) [noise=true , address=167.17.72.71:4810 , piece_id=ANLHBMENEXSGTHBHLOIHMLM6JGHKIU2X6FPQQIUIVMVUOABNK73Q, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:53:18.344002671 (8h53m15.9407737s) [noise=true , address=45.91.116.10:4096 , piece_id=EIKWFKPE5BYSZGV5T4RPQED55PG2FXU3ZKSNRVCBD363YQDHS4BQ, seg_pos={0 0}, upload=9902]: Done uploading piece. err: ecclient: upload failed (node:12MZX1GwZGrEi2q44dPiYrwMoum5A2caZtXTS3qiLNcoyHhon7p, address:45.91.116.10:4096): protocol: write tcp 159.223.143.39:44996->45.91.116.10:4096: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 6 19:53:18.344173472 (8h53m15.940944497s) [noise=true , address=45.91.116.7:4037 , piece_id=YXDMXUBE7CEWLJZP6FPDCRVIDGG5CJ3Y3IKEKO7EE2DQUSG6W5RA, seg_pos={0 0}, upload=9902]: Done uploading piece. err: ecclient: upload failed (node:1iNPwE2XSfXcaXAXcrH2jcyNGYeDiiWvzAcDQffrmjPwTVqVmK, address:45.91.116.7:4037): protocol: write tcp 159.223.143.39:52854->45.91.116.7:4037: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 6 19:53:18.344535130 (8h53m15.941306156s) [noise=true , address=45.91.116.14:4199 , piece_id=EKGDG7LYV3O5NSDL3UPWFKXINYNAOSPYYOWQPTFN326AJ2ZXAXHQ, seg_pos={0 0}, upload=9902]: Done uploading piece. err: ecclient: upload failed (node:1d4B1KMLs2annUuJ3v81FJapS9Yg4omBTj48eWoi6HJxnSEjrf, address:45.91.116.14:4199): protocol: write tcp 159.223.143.39:57596->45.91.116.14:4199: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 6 19:53:18.344687364 (8h53m15.941458387s) [noise=true , address=45.91.116.11:4133 , piece_id=RZWV5JMLOJ36AH7KSDXVTL7CMHI4INEW6YOOQVMCIIC2NGS3XYJA, seg_pos={0 0}, upload=9902]: Done uploading piece. err: ecclient: upload failed (node:12e1wX7hiER8sX4MLm5qLue4mayiN4SxXFdZSxfymRaRZxRGU6E, address:45.91.116.11:4133): protocol: write tcp 159.223.143.39:56440->45.91.116.11:4133: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 6 19:53:18.345220236 (8h53m15.941991259s) [noise=true , address=45.91.116.8:4066 , piece_id=5MZHIJ3ELVBAJ7ABRKG565TLU4A4NKB7X6WAACY2JBSLACL5LGVQ, seg_pos={0 0}, upload=9902]: Done uploading piece. err: ecclient: upload failed (node:123uv49k8LraXJfYQg6KRQthrYs1Egc4c3eZWm5JPvDatFbykm8, address:45.91.116.8:4066): protocol: write tcp 159.223.143.39:44136->45.91.116.8:4066: use of closed network connection; context deadline exceeded Mar 6 19:53:18.345277850 (8h53m15.942048865s) [seg_pos={0 0}, upload=9902]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 6 19:53:18.367560006 (8h53m15.964331035s) [upload=9902]: Done uploading segments... Mar 6 19:53:18.367606646 (8h53m15.964377681s) [upload=9902]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 1.356439487s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................X................................] duration: 704.781516ms trace: https://jaeger.dev.storj.io/trace/78791d41d75e16a0 FAIL (2026-03-06 20:15:16 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 19:52:18.382132233 (8h52m15.978903242s) [upload=9902]: Uploading segments... Mar 6 19:52:18.382227489 (8h52m15.978998515s) [upload=9902]: Got next segment. Inline: false Mar 6 19:52:18.382313252 (8h52m15.979084279s) [upload=9902]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 6 19:52:18.382319269 (8h52m15.979090296s) [upload=9902]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 6 19:52:18.411471452 (8h52m16.008242482s) [seg_pos={0 0}, upload=9902]: Begin upload segment... Mar 6 19:52:18.411537096 (8h52m16.008308109s) [seg_pos={0 0}, upload=9902]: Done begin upload segment. Mar 6 19:52:18.411564540 (8h52m16.008335586s) [noise=true , address=45.91.116.7:4037 , piece_id=YXDMXUBE7CEWLJZP6FPDCRVIDGG5CJ3Y3IKEKO7EE2DQUSG6W5RA, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.411626862 (8h52m16.008397897s) [noise=true , address=193.149.164.67:4018 , piece_id=GTQQP355TZZF4W7PPBYZS6ZQ6WYK7I3QIE4DZC7STWAKSBAMT4BQ, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.412356605 (8h52m16.009127631s) [noise=true , address=167.17.72.70:4771 , piece_id=TDLPTUHPNXCICPPXRTRL7WCJSXPX22ONIWWPHNXFUNIKRYPYW52Q, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.412900389 (8h52m16.009671415s) [noise=true , address=145.79.227.39:4022 , piece_id=U5GX3GLVWGWLU2BDWTHM7ST3MLLJDEAERWU3UEHO7Z2RXHGZU2DA, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.413467061 (8h52m16.010238103s) [noise=true , address=45.91.116.11:4133 , piece_id=RZWV5JMLOJ36AH7KSDXVTL7CMHI4INEW6YOOQVMCIIC2NGS3XYJA, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.413653431 (8h52m16.010424457s) [noise=true , address=193.149.164.69:4101 , piece_id=BTIBVPSORFJQR3CLA5UV62NXNI6GD3JF4D5Z74KU4R4CXB3WHHKA, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.414227965 (8h52m16.010998999s) [noise=true , address=45.91.116.14:4199 , piece_id=EKGDG7LYV3O5NSDL3UPWFKXINYNAOSPYYOWQPTFN326AJ2ZXAXHQ, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.415177146 (8h52m16.01194818s) [noise=true , address=77.90.31.16:4763 , piece_id=5Q7H3CU3MWSFPQGC67XMLCMTSUIEJQUV7QTO4UZLZY4MIVM76Z6Q, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.415227889 (8h52m16.011998912s) [noise=true , address=45.91.116.10:4096 , piece_id=EIKWFKPE5BYSZGV5T4RPQED55PG2FXU3ZKSNRVCBD363YQDHS4BQ, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.415728547 (8h52m16.012499581s) [noise=true , address=77.90.31.14:4693 , piece_id=S7NAS2ZBHAQPZRHBRGRSZPLE6COQEEN6SZVMB7X2X6IMXTN5HCBQ, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.415910147 (8h52m16.012681174s) [noise=true , address=193.149.164.70:4129 , piece_id=V7WMI7RDYDOAEQYWF2OTZL5LRPUA2S3V5WZN4BKDY6BI4YV3GRPQ, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.416592462 (8h52m16.01336349s) [noise=true , address=77.90.31.18:4831 , piece_id=4PMIUNE4YXKLIJPML3VSUN4PF32TSLPOOSDZZNAXFWH7WICBOW3Q, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.417118897 (8h52m16.01388993s) [noise=true , address=77.90.31.13:4655 , piece_id=47Q27QI7O2IYW7YQKFMJGGMWPVZWWKQTFVYPUMPUI7ORE7236XIQ, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.417185844 (8h52m16.013956873s) [noise=true , address=193.149.164.71:4304 , piece_id=DTCL54GM76GWKAE7A3X5PUNFYO7H5JIKRT5WBEHYEDF2YNFYTOAA, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.417686790 (8h52m16.014457823s) [noise=true , address=167.17.72.71:4810 , piece_id=ANLHBMENEXSGTHBHLOIHMLM6JGHKIU2X6FPQQIUIVMVUOABNK73Q, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.418236951 (8h52m16.015007982s) [noise=true , address=145.79.227.17:4994 , piece_id=TO3EOAILO4PHF6Z7WQA4RWZNNL3JO6EXXEF2RLW7IHAGWUU2TF4Q, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.419293640 (8h52m16.016064672s) [noise=true , address=64.49.13.139:4479 , piece_id=XUYHOCSBATYECYHWLVZ4VBCMT3H4O5ETSQ5HZOB42OR3OY4LWJ6Q, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.420005502 (8h52m16.016776534s) [noise=true , address=45.91.116.8:4066 , piece_id=5MZHIJ3ELVBAJ7ABRKG565TLU4A4NKB7X6WAACY2JBSLACL5LGVQ, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.483654433 (8h52m16.080425441s) [upload=9902]: Next returned nil segment Mar 6 19:52:18.483671157 (8h52m16.08044219s) [upload=9902]: Waiting for error group managing segments... Mar 6 19:52:18.488078321 (8h52m16.084849398s) [noise=true , address=77.90.31.13:4655 , piece_id=47Q27QI7O2IYW7YQKFMJGGMWPVZWWKQTFVYPUMPUI7ORE7236XIQ, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:52:18.498435619 (8h52m16.095206647s) [noise=true , address=145.79.227.17:4994 , piece_id=TO3EOAILO4PHF6Z7WQA4RWZNNL3JO6EXXEF2RLW7IHAGWUU2TF4Q, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:52:18.502979814 (8h52m16.099750841s) [noise=true , address=77.90.31.16:4763 , piece_id=5Q7H3CU3MWSFPQGC67XMLCMTSUIEJQUV7QTO4UZLZY4MIVM76Z6Q, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:52:18.506259140 (8h52m16.103030166s) [noise=true , address=77.90.31.14:4693 , piece_id=S7NAS2ZBHAQPZRHBRGRSZPLE6COQEEN6SZVMB7X2X6IMXTN5HCBQ, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:52:18.514529389 (8h52m16.111300413s) [noise=true , address=145.79.227.39:4022 , piece_id=U5GX3GLVWGWLU2BDWTHM7ST3MLLJDEAERWU3UEHO7Z2RXHGZU2DA, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:52:18.785925494 (8h52m16.382696517s) [noise=true , address=167.17.72.70:4771 , piece_id=TDLPTUHPNXCICPPXRTRL7WCJSXPX22ONIWWPHNXFUNIKRYPYW52Q, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:52:18.810774903 (8h52m16.407545926s) [noise=true , address=77.90.31.18:4831 , piece_id=4PMIUNE4YXKLIJPML3VSUN4PF32TSLPOOSDZZNAXFWH7WICBOW3Q, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:52:19.002746550 (8h52m16.5995176s) [noise=true , address=193.149.164.71:4304 , piece_id=DTCL54GM76GWKAE7A3X5PUNFYO7H5JIKRT5WBEHYEDF2YNFYTOAA, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:52:19.007614901 (8h52m16.604385924s) [noise=true , address=193.149.164.67:4018 , piece_id=GTQQP355TZZF4W7PPBYZS6ZQ6WYK7I3QIE4DZC7STWAKSBAMT4BQ, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:52:19.010443151 (8h52m16.607214176s) [noise=true , address=193.149.164.69:4101 , piece_id=BTIBVPSORFJQR3CLA5UV62NXNI6GD3JF4D5Z74KU4R4CXB3WHHKA, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:52:19.012614014 (8h52m16.609385038s) [noise=true , address=193.149.164.70:4129 , piece_id=V7WMI7RDYDOAEQYWF2OTZL5LRPUA2S3V5WZN4BKDY6BI4YV3GRPQ, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:52:19.042145380 (8h52m16.638916406s) [noise=true , address=64.49.13.139:4479 , piece_id=XUYHOCSBATYECYHWLVZ4VBCMT3H4O5ETSQ5HZOB42OR3OY4LWJ6Q, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:52:21.314990180 (8h52m18.911761213s) [noise=true , address=167.17.72.71:4810 , piece_id=ANLHBMENEXSGTHBHLOIHMLM6JGHKIU2X6FPQQIUIVMVUOABNK73Q, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:53:18.344002671 (8h53m15.9407737s) [noise=true , address=45.91.116.10:4096 , piece_id=EIKWFKPE5BYSZGV5T4RPQED55PG2FXU3ZKSNRVCBD363YQDHS4BQ, seg_pos={0 0}, upload=9902]: Done uploading piece. err: ecclient: upload failed (node:12MZX1GwZGrEi2q44dPiYrwMoum5A2caZtXTS3qiLNcoyHhon7p, address:45.91.116.10:4096): protocol: write tcp 159.223.143.39:44996->45.91.116.10:4096: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 6 19:53:18.344173472 (8h53m15.940944497s) [noise=true , address=45.91.116.7:4037 , piece_id=YXDMXUBE7CEWLJZP6FPDCRVIDGG5CJ3Y3IKEKO7EE2DQUSG6W5RA, seg_pos={0 0}, upload=9902]: Done uploading piece. err: ecclient: upload failed (node:1iNPwE2XSfXcaXAXcrH2jcyNGYeDiiWvzAcDQffrmjPwTVqVmK, address:45.91.116.7:4037): protocol: write tcp 159.223.143.39:52854->45.91.116.7:4037: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 6 19:53:18.344535130 (8h53m15.941306156s) [noise=true , address=45.91.116.14:4199 , piece_id=EKGDG7LYV3O5NSDL3UPWFKXINYNAOSPYYOWQPTFN326AJ2ZXAXHQ, seg_pos={0 0}, upload=9902]: Done uploading piece. err: ecclient: upload failed (node:1d4B1KMLs2annUuJ3v81FJapS9Yg4omBTj48eWoi6HJxnSEjrf, address:45.91.116.14:4199): protocol: write tcp 159.223.143.39:57596->45.91.116.14:4199: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 6 19:53:18.344687364 (8h53m15.941458387s) [noise=true , address=45.91.116.11:4133 , piece_id=RZWV5JMLOJ36AH7KSDXVTL7CMHI4INEW6YOOQVMCIIC2NGS3XYJA, seg_pos={0 0}, upload=9902]: Done uploading piece. err: ecclient: upload failed (node:12e1wX7hiER8sX4MLm5qLue4mayiN4SxXFdZSxfymRaRZxRGU6E, address:45.91.116.11:4133): protocol: write tcp 159.223.143.39:56440->45.91.116.11:4133: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 6 19:53:18.345220236 (8h53m15.941991259s) [noise=true , address=45.91.116.8:4066 , piece_id=5MZHIJ3ELVBAJ7ABRKG565TLU4A4NKB7X6WAACY2JBSLACL5LGVQ, seg_pos={0 0}, upload=9902]: Done uploading piece. err: ecclient: upload failed (node:123uv49k8LraXJfYQg6KRQthrYs1Egc4c3eZWm5JPvDatFbykm8, address:45.91.116.8:4066): protocol: write tcp 159.223.143.39:44136->45.91.116.8:4066: use of closed network connection; context deadline exceeded Mar 6 19:53:18.345277850 (8h53m15.942048865s) [seg_pos={0 0}, upload=9902]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 6 19:53:18.367560006 (8h53m15.964331035s) [upload=9902]: Done uploading segments... Mar 6 19:53:18.367606646 (8h53m15.964377681s) [upload=9902]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 704.805504ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [............................X...............................] duration: 964.730309ms trace: https://jaeger.dev.storj.io/trace/78791d41d75e16a0 FAIL (2026-03-06 20:14:31 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 19:52:18.382132233 (8h52m15.978903242s) [upload=9902]: Uploading segments... Mar 6 19:52:18.382227489 (8h52m15.978998515s) [upload=9902]: Got next segment. Inline: false Mar 6 19:52:18.382313252 (8h52m15.979084279s) [upload=9902]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 6 19:52:18.382319269 (8h52m15.979090296s) [upload=9902]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 6 19:52:18.411471452 (8h52m16.008242482s) [seg_pos={0 0}, upload=9902]: Begin upload segment... Mar 6 19:52:18.411537096 (8h52m16.008308109s) [seg_pos={0 0}, upload=9902]: Done begin upload segment. Mar 6 19:52:18.411564540 (8h52m16.008335586s) [noise=true , address=45.91.116.7:4037 , piece_id=YXDMXUBE7CEWLJZP6FPDCRVIDGG5CJ3Y3IKEKO7EE2DQUSG6W5RA, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.411626862 (8h52m16.008397897s) [noise=true , address=193.149.164.67:4018 , piece_id=GTQQP355TZZF4W7PPBYZS6ZQ6WYK7I3QIE4DZC7STWAKSBAMT4BQ, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.412356605 (8h52m16.009127631s) [noise=true , address=167.17.72.70:4771 , piece_id=TDLPTUHPNXCICPPXRTRL7WCJSXPX22ONIWWPHNXFUNIKRYPYW52Q, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.412900389 (8h52m16.009671415s) [noise=true , address=145.79.227.39:4022 , piece_id=U5GX3GLVWGWLU2BDWTHM7ST3MLLJDEAERWU3UEHO7Z2RXHGZU2DA, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.413467061 (8h52m16.010238103s) [noise=true , address=45.91.116.11:4133 , piece_id=RZWV5JMLOJ36AH7KSDXVTL7CMHI4INEW6YOOQVMCIIC2NGS3XYJA, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.413653431 (8h52m16.010424457s) [noise=true , address=193.149.164.69:4101 , piece_id=BTIBVPSORFJQR3CLA5UV62NXNI6GD3JF4D5Z74KU4R4CXB3WHHKA, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.414227965 (8h52m16.010998999s) [noise=true , address=45.91.116.14:4199 , piece_id=EKGDG7LYV3O5NSDL3UPWFKXINYNAOSPYYOWQPTFN326AJ2ZXAXHQ, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.415177146 (8h52m16.01194818s) [noise=true , address=77.90.31.16:4763 , piece_id=5Q7H3CU3MWSFPQGC67XMLCMTSUIEJQUV7QTO4UZLZY4MIVM76Z6Q, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.415227889 (8h52m16.011998912s) [noise=true , address=45.91.116.10:4096 , piece_id=EIKWFKPE5BYSZGV5T4RPQED55PG2FXU3ZKSNRVCBD363YQDHS4BQ, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.415728547 (8h52m16.012499581s) [noise=true , address=77.90.31.14:4693 , piece_id=S7NAS2ZBHAQPZRHBRGRSZPLE6COQEEN6SZVMB7X2X6IMXTN5HCBQ, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.415910147 (8h52m16.012681174s) [noise=true , address=193.149.164.70:4129 , piece_id=V7WMI7RDYDOAEQYWF2OTZL5LRPUA2S3V5WZN4BKDY6BI4YV3GRPQ, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.416592462 (8h52m16.01336349s) [noise=true , address=77.90.31.18:4831 , piece_id=4PMIUNE4YXKLIJPML3VSUN4PF32TSLPOOSDZZNAXFWH7WICBOW3Q, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.417118897 (8h52m16.01388993s) [noise=true , address=77.90.31.13:4655 , piece_id=47Q27QI7O2IYW7YQKFMJGGMWPVZWWKQTFVYPUMPUI7ORE7236XIQ, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.417185844 (8h52m16.013956873s) [noise=true , address=193.149.164.71:4304 , piece_id=DTCL54GM76GWKAE7A3X5PUNFYO7H5JIKRT5WBEHYEDF2YNFYTOAA, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.417686790 (8h52m16.014457823s) [noise=true , address=167.17.72.71:4810 , piece_id=ANLHBMENEXSGTHBHLOIHMLM6JGHKIU2X6FPQQIUIVMVUOABNK73Q, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.418236951 (8h52m16.015007982s) [noise=true , address=145.79.227.17:4994 , piece_id=TO3EOAILO4PHF6Z7WQA4RWZNNL3JO6EXXEF2RLW7IHAGWUU2TF4Q, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.419293640 (8h52m16.016064672s) [noise=true , address=64.49.13.139:4479 , piece_id=XUYHOCSBATYECYHWLVZ4VBCMT3H4O5ETSQ5HZOB42OR3OY4LWJ6Q, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.420005502 (8h52m16.016776534s) [noise=true , address=45.91.116.8:4066 , piece_id=5MZHIJ3ELVBAJ7ABRKG565TLU4A4NKB7X6WAACY2JBSLACL5LGVQ, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.483654433 (8h52m16.080425441s) [upload=9902]: Next returned nil segment Mar 6 19:52:18.483671157 (8h52m16.08044219s) [upload=9902]: Waiting for error group managing segments... Mar 6 19:52:18.488078321 (8h52m16.084849398s) [noise=true , address=77.90.31.13:4655 , piece_id=47Q27QI7O2IYW7YQKFMJGGMWPVZWWKQTFVYPUMPUI7ORE7236XIQ, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:52:18.498435619 (8h52m16.095206647s) [noise=true , address=145.79.227.17:4994 , piece_id=TO3EOAILO4PHF6Z7WQA4RWZNNL3JO6EXXEF2RLW7IHAGWUU2TF4Q, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:52:18.502979814 (8h52m16.099750841s) [noise=true , address=77.90.31.16:4763 , piece_id=5Q7H3CU3MWSFPQGC67XMLCMTSUIEJQUV7QTO4UZLZY4MIVM76Z6Q, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:52:18.506259140 (8h52m16.103030166s) [noise=true , address=77.90.31.14:4693 , piece_id=S7NAS2ZBHAQPZRHBRGRSZPLE6COQEEN6SZVMB7X2X6IMXTN5HCBQ, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:52:18.514529389 (8h52m16.111300413s) [noise=true , address=145.79.227.39:4022 , piece_id=U5GX3GLVWGWLU2BDWTHM7ST3MLLJDEAERWU3UEHO7Z2RXHGZU2DA, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:52:18.785925494 (8h52m16.382696517s) [noise=true , address=167.17.72.70:4771 , piece_id=TDLPTUHPNXCICPPXRTRL7WCJSXPX22ONIWWPHNXFUNIKRYPYW52Q, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:52:18.810774903 (8h52m16.407545926s) [noise=true , address=77.90.31.18:4831 , piece_id=4PMIUNE4YXKLIJPML3VSUN4PF32TSLPOOSDZZNAXFWH7WICBOW3Q, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:52:19.002746550 (8h52m16.5995176s) [noise=true , address=193.149.164.71:4304 , piece_id=DTCL54GM76GWKAE7A3X5PUNFYO7H5JIKRT5WBEHYEDF2YNFYTOAA, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:52:19.007614901 (8h52m16.604385924s) [noise=true , address=193.149.164.67:4018 , piece_id=GTQQP355TZZF4W7PPBYZS6ZQ6WYK7I3QIE4DZC7STWAKSBAMT4BQ, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:52:19.010443151 (8h52m16.607214176s) [noise=true , address=193.149.164.69:4101 , piece_id=BTIBVPSORFJQR3CLA5UV62NXNI6GD3JF4D5Z74KU4R4CXB3WHHKA, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:52:19.012614014 (8h52m16.609385038s) [noise=true , address=193.149.164.70:4129 , piece_id=V7WMI7RDYDOAEQYWF2OTZL5LRPUA2S3V5WZN4BKDY6BI4YV3GRPQ, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:52:19.042145380 (8h52m16.638916406s) [noise=true , address=64.49.13.139:4479 , piece_id=XUYHOCSBATYECYHWLVZ4VBCMT3H4O5ETSQ5HZOB42OR3OY4LWJ6Q, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:52:21.314990180 (8h52m18.911761213s) [noise=true , address=167.17.72.71:4810 , piece_id=ANLHBMENEXSGTHBHLOIHMLM6JGHKIU2X6FPQQIUIVMVUOABNK73Q, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:53:18.344002671 (8h53m15.9407737s) [noise=true , address=45.91.116.10:4096 , piece_id=EIKWFKPE5BYSZGV5T4RPQED55PG2FXU3ZKSNRVCBD363YQDHS4BQ, seg_pos={0 0}, upload=9902]: Done uploading piece. err: ecclient: upload failed (node:12MZX1GwZGrEi2q44dPiYrwMoum5A2caZtXTS3qiLNcoyHhon7p, address:45.91.116.10:4096): protocol: write tcp 159.223.143.39:44996->45.91.116.10:4096: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 6 19:53:18.344173472 (8h53m15.940944497s) [noise=true , address=45.91.116.7:4037 , piece_id=YXDMXUBE7CEWLJZP6FPDCRVIDGG5CJ3Y3IKEKO7EE2DQUSG6W5RA, seg_pos={0 0}, upload=9902]: Done uploading piece. err: ecclient: upload failed (node:1iNPwE2XSfXcaXAXcrH2jcyNGYeDiiWvzAcDQffrmjPwTVqVmK, address:45.91.116.7:4037): protocol: write tcp 159.223.143.39:52854->45.91.116.7:4037: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 6 19:53:18.344535130 (8h53m15.941306156s) [noise=true , address=45.91.116.14:4199 , piece_id=EKGDG7LYV3O5NSDL3UPWFKXINYNAOSPYYOWQPTFN326AJ2ZXAXHQ, seg_pos={0 0}, upload=9902]: Done uploading piece. err: ecclient: upload failed (node:1d4B1KMLs2annUuJ3v81FJapS9Yg4omBTj48eWoi6HJxnSEjrf, address:45.91.116.14:4199): protocol: write tcp 159.223.143.39:57596->45.91.116.14:4199: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 6 19:53:18.344687364 (8h53m15.941458387s) [noise=true , address=45.91.116.11:4133 , piece_id=RZWV5JMLOJ36AH7KSDXVTL7CMHI4INEW6YOOQVMCIIC2NGS3XYJA, seg_pos={0 0}, upload=9902]: Done uploading piece. err: ecclient: upload failed (node:12e1wX7hiER8sX4MLm5qLue4mayiN4SxXFdZSxfymRaRZxRGU6E, address:45.91.116.11:4133): protocol: write tcp 159.223.143.39:56440->45.91.116.11:4133: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 6 19:53:18.345220236 (8h53m15.941991259s) [noise=true , address=45.91.116.8:4066 , piece_id=5MZHIJ3ELVBAJ7ABRKG565TLU4A4NKB7X6WAACY2JBSLACL5LGVQ, seg_pos={0 0}, upload=9902]: Done uploading piece. err: ecclient: upload failed (node:123uv49k8LraXJfYQg6KRQthrYs1Egc4c3eZWm5JPvDatFbykm8, address:45.91.116.8:4066): protocol: write tcp 159.223.143.39:44136->45.91.116.8:4066: use of closed network connection; context deadline exceeded Mar 6 19:53:18.345277850 (8h53m15.942048865s) [seg_pos={0 0}, upload=9902]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 6 19:53:18.367560006 (8h53m15.964331035s) [upload=9902]: Done uploading segments... Mar 6 19:53:18.367606646 (8h53m15.964377681s) [upload=9902]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 964.774261ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............................X..............................] duration: 1.190011843s trace: https://jaeger.dev.storj.io/trace/78791d41d75e16a0 FAIL (2026-03-06 20:13:58 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 19:52:18.382132233 (8h52m15.978903242s) [upload=9902]: Uploading segments... Mar 6 19:52:18.382227489 (8h52m15.978998515s) [upload=9902]: Got next segment. Inline: false Mar 6 19:52:18.382313252 (8h52m15.979084279s) [upload=9902]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 6 19:52:18.382319269 (8h52m15.979090296s) [upload=9902]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 6 19:52:18.411471452 (8h52m16.008242482s) [seg_pos={0 0}, upload=9902]: Begin upload segment... Mar 6 19:52:18.411537096 (8h52m16.008308109s) [seg_pos={0 0}, upload=9902]: Done begin upload segment. Mar 6 19:52:18.411564540 (8h52m16.008335586s) [noise=true , address=45.91.116.7:4037 , piece_id=YXDMXUBE7CEWLJZP6FPDCRVIDGG5CJ3Y3IKEKO7EE2DQUSG6W5RA, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.411626862 (8h52m16.008397897s) [noise=true , address=193.149.164.67:4018 , piece_id=GTQQP355TZZF4W7PPBYZS6ZQ6WYK7I3QIE4DZC7STWAKSBAMT4BQ, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.412356605 (8h52m16.009127631s) [noise=true , address=167.17.72.70:4771 , piece_id=TDLPTUHPNXCICPPXRTRL7WCJSXPX22ONIWWPHNXFUNIKRYPYW52Q, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.412900389 (8h52m16.009671415s) [noise=true , address=145.79.227.39:4022 , piece_id=U5GX3GLVWGWLU2BDWTHM7ST3MLLJDEAERWU3UEHO7Z2RXHGZU2DA, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.413467061 (8h52m16.010238103s) [noise=true , address=45.91.116.11:4133 , piece_id=RZWV5JMLOJ36AH7KSDXVTL7CMHI4INEW6YOOQVMCIIC2NGS3XYJA, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.413653431 (8h52m16.010424457s) [noise=true , address=193.149.164.69:4101 , piece_id=BTIBVPSORFJQR3CLA5UV62NXNI6GD3JF4D5Z74KU4R4CXB3WHHKA, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.414227965 (8h52m16.010998999s) [noise=true , address=45.91.116.14:4199 , piece_id=EKGDG7LYV3O5NSDL3UPWFKXINYNAOSPYYOWQPTFN326AJ2ZXAXHQ, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.415177146 (8h52m16.01194818s) [noise=true , address=77.90.31.16:4763 , piece_id=5Q7H3CU3MWSFPQGC67XMLCMTSUIEJQUV7QTO4UZLZY4MIVM76Z6Q, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.415227889 (8h52m16.011998912s) [noise=true , address=45.91.116.10:4096 , piece_id=EIKWFKPE5BYSZGV5T4RPQED55PG2FXU3ZKSNRVCBD363YQDHS4BQ, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.415728547 (8h52m16.012499581s) [noise=true , address=77.90.31.14:4693 , piece_id=S7NAS2ZBHAQPZRHBRGRSZPLE6COQEEN6SZVMB7X2X6IMXTN5HCBQ, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.415910147 (8h52m16.012681174s) [noise=true , address=193.149.164.70:4129 , piece_id=V7WMI7RDYDOAEQYWF2OTZL5LRPUA2S3V5WZN4BKDY6BI4YV3GRPQ, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.416592462 (8h52m16.01336349s) [noise=true , address=77.90.31.18:4831 , piece_id=4PMIUNE4YXKLIJPML3VSUN4PF32TSLPOOSDZZNAXFWH7WICBOW3Q, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.417118897 (8h52m16.01388993s) [noise=true , address=77.90.31.13:4655 , piece_id=47Q27QI7O2IYW7YQKFMJGGMWPVZWWKQTFVYPUMPUI7ORE7236XIQ, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.417185844 (8h52m16.013956873s) [noise=true , address=193.149.164.71:4304 , piece_id=DTCL54GM76GWKAE7A3X5PUNFYO7H5JIKRT5WBEHYEDF2YNFYTOAA, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.417686790 (8h52m16.014457823s) [noise=true , address=167.17.72.71:4810 , piece_id=ANLHBMENEXSGTHBHLOIHMLM6JGHKIU2X6FPQQIUIVMVUOABNK73Q, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.418236951 (8h52m16.015007982s) [noise=true , address=145.79.227.17:4994 , piece_id=TO3EOAILO4PHF6Z7WQA4RWZNNL3JO6EXXEF2RLW7IHAGWUU2TF4Q, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.419293640 (8h52m16.016064672s) [noise=true , address=64.49.13.139:4479 , piece_id=XUYHOCSBATYECYHWLVZ4VBCMT3H4O5ETSQ5HZOB42OR3OY4LWJ6Q, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.420005502 (8h52m16.016776534s) [noise=true , address=45.91.116.8:4066 , piece_id=5MZHIJ3ELVBAJ7ABRKG565TLU4A4NKB7X6WAACY2JBSLACL5LGVQ, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.483654433 (8h52m16.080425441s) [upload=9902]: Next returned nil segment Mar 6 19:52:18.483671157 (8h52m16.08044219s) [upload=9902]: Waiting for error group managing segments... Mar 6 19:52:18.488078321 (8h52m16.084849398s) [noise=true , address=77.90.31.13:4655 , piece_id=47Q27QI7O2IYW7YQKFMJGGMWPVZWWKQTFVYPUMPUI7ORE7236XIQ, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:52:18.498435619 (8h52m16.095206647s) [noise=true , address=145.79.227.17:4994 , piece_id=TO3EOAILO4PHF6Z7WQA4RWZNNL3JO6EXXEF2RLW7IHAGWUU2TF4Q, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:52:18.502979814 (8h52m16.099750841s) [noise=true , address=77.90.31.16:4763 , piece_id=5Q7H3CU3MWSFPQGC67XMLCMTSUIEJQUV7QTO4UZLZY4MIVM76Z6Q, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:52:18.506259140 (8h52m16.103030166s) [noise=true , address=77.90.31.14:4693 , piece_id=S7NAS2ZBHAQPZRHBRGRSZPLE6COQEEN6SZVMB7X2X6IMXTN5HCBQ, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:52:18.514529389 (8h52m16.111300413s) [noise=true , address=145.79.227.39:4022 , piece_id=U5GX3GLVWGWLU2BDWTHM7ST3MLLJDEAERWU3UEHO7Z2RXHGZU2DA, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:52:18.785925494 (8h52m16.382696517s) [noise=true , address=167.17.72.70:4771 , piece_id=TDLPTUHPNXCICPPXRTRL7WCJSXPX22ONIWWPHNXFUNIKRYPYW52Q, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:52:18.810774903 (8h52m16.407545926s) [noise=true , address=77.90.31.18:4831 , piece_id=4PMIUNE4YXKLIJPML3VSUN4PF32TSLPOOSDZZNAXFWH7WICBOW3Q, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:52:19.002746550 (8h52m16.5995176s) [noise=true , address=193.149.164.71:4304 , piece_id=DTCL54GM76GWKAE7A3X5PUNFYO7H5JIKRT5WBEHYEDF2YNFYTOAA, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:52:19.007614901 (8h52m16.604385924s) [noise=true , address=193.149.164.67:4018 , piece_id=GTQQP355TZZF4W7PPBYZS6ZQ6WYK7I3QIE4DZC7STWAKSBAMT4BQ, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:52:19.010443151 (8h52m16.607214176s) [noise=true , address=193.149.164.69:4101 , piece_id=BTIBVPSORFJQR3CLA5UV62NXNI6GD3JF4D5Z74KU4R4CXB3WHHKA, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:52:19.012614014 (8h52m16.609385038s) [noise=true , address=193.149.164.70:4129 , piece_id=V7WMI7RDYDOAEQYWF2OTZL5LRPUA2S3V5WZN4BKDY6BI4YV3GRPQ, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:52:19.042145380 (8h52m16.638916406s) [noise=true , address=64.49.13.139:4479 , piece_id=XUYHOCSBATYECYHWLVZ4VBCMT3H4O5ETSQ5HZOB42OR3OY4LWJ6Q, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:52:21.314990180 (8h52m18.911761213s) [noise=true , address=167.17.72.71:4810 , piece_id=ANLHBMENEXSGTHBHLOIHMLM6JGHKIU2X6FPQQIUIVMVUOABNK73Q, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:53:18.344002671 (8h53m15.9407737s) [noise=true , address=45.91.116.10:4096 , piece_id=EIKWFKPE5BYSZGV5T4RPQED55PG2FXU3ZKSNRVCBD363YQDHS4BQ, seg_pos={0 0}, upload=9902]: Done uploading piece. err: ecclient: upload failed (node:12MZX1GwZGrEi2q44dPiYrwMoum5A2caZtXTS3qiLNcoyHhon7p, address:45.91.116.10:4096): protocol: write tcp 159.223.143.39:44996->45.91.116.10:4096: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 6 19:53:18.344173472 (8h53m15.940944497s) [noise=true , address=45.91.116.7:4037 , piece_id=YXDMXUBE7CEWLJZP6FPDCRVIDGG5CJ3Y3IKEKO7EE2DQUSG6W5RA, seg_pos={0 0}, upload=9902]: Done uploading piece. err: ecclient: upload failed (node:1iNPwE2XSfXcaXAXcrH2jcyNGYeDiiWvzAcDQffrmjPwTVqVmK, address:45.91.116.7:4037): protocol: write tcp 159.223.143.39:52854->45.91.116.7:4037: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 6 19:53:18.344535130 (8h53m15.941306156s) [noise=true , address=45.91.116.14:4199 , piece_id=EKGDG7LYV3O5NSDL3UPWFKXINYNAOSPYYOWQPTFN326AJ2ZXAXHQ, seg_pos={0 0}, upload=9902]: Done uploading piece. err: ecclient: upload failed (node:1d4B1KMLs2annUuJ3v81FJapS9Yg4omBTj48eWoi6HJxnSEjrf, address:45.91.116.14:4199): protocol: write tcp 159.223.143.39:57596->45.91.116.14:4199: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 6 19:53:18.344687364 (8h53m15.941458387s) [noise=true , address=45.91.116.11:4133 , piece_id=RZWV5JMLOJ36AH7KSDXVTL7CMHI4INEW6YOOQVMCIIC2NGS3XYJA, seg_pos={0 0}, upload=9902]: Done uploading piece. err: ecclient: upload failed (node:12e1wX7hiER8sX4MLm5qLue4mayiN4SxXFdZSxfymRaRZxRGU6E, address:45.91.116.11:4133): protocol: write tcp 159.223.143.39:56440->45.91.116.11:4133: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 6 19:53:18.345220236 (8h53m15.941991259s) [noise=true , address=45.91.116.8:4066 , piece_id=5MZHIJ3ELVBAJ7ABRKG565TLU4A4NKB7X6WAACY2JBSLACL5LGVQ, seg_pos={0 0}, upload=9902]: Done uploading piece. err: ecclient: upload failed (node:123uv49k8LraXJfYQg6KRQthrYs1Egc4c3eZWm5JPvDatFbykm8, address:45.91.116.8:4066): protocol: write tcp 159.223.143.39:44136->45.91.116.8:4066: use of closed network connection; context deadline exceeded Mar 6 19:53:18.345277850 (8h53m15.942048865s) [seg_pos={0 0}, upload=9902]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 6 19:53:18.367560006 (8h53m15.964331035s) [upload=9902]: Done uploading segments... Mar 6 19:53:18.367606646 (8h53m15.964377681s) [upload=9902]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 1.190065989s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............................X.............................] duration: 1.168855196s trace: https://jaeger.dev.storj.io/trace/78791d41d75e16a0 FAIL (2026-03-06 20:13:01 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 19:52:18.382132233 (8h52m15.978903242s) [upload=9902]: Uploading segments... Mar 6 19:52:18.382227489 (8h52m15.978998515s) [upload=9902]: Got next segment. Inline: false Mar 6 19:52:18.382313252 (8h52m15.979084279s) [upload=9902]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 6 19:52:18.382319269 (8h52m15.979090296s) [upload=9902]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 6 19:52:18.411471452 (8h52m16.008242482s) [seg_pos={0 0}, upload=9902]: Begin upload segment... Mar 6 19:52:18.411537096 (8h52m16.008308109s) [seg_pos={0 0}, upload=9902]: Done begin upload segment. Mar 6 19:52:18.411564540 (8h52m16.008335586s) [noise=true , address=45.91.116.7:4037 , piece_id=YXDMXUBE7CEWLJZP6FPDCRVIDGG5CJ3Y3IKEKO7EE2DQUSG6W5RA, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.411626862 (8h52m16.008397897s) [noise=true , address=193.149.164.67:4018 , piece_id=GTQQP355TZZF4W7PPBYZS6ZQ6WYK7I3QIE4DZC7STWAKSBAMT4BQ, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.412356605 (8h52m16.009127631s) [noise=true , address=167.17.72.70:4771 , piece_id=TDLPTUHPNXCICPPXRTRL7WCJSXPX22ONIWWPHNXFUNIKRYPYW52Q, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.412900389 (8h52m16.009671415s) [noise=true , address=145.79.227.39:4022 , piece_id=U5GX3GLVWGWLU2BDWTHM7ST3MLLJDEAERWU3UEHO7Z2RXHGZU2DA, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.413467061 (8h52m16.010238103s) [noise=true , address=45.91.116.11:4133 , piece_id=RZWV5JMLOJ36AH7KSDXVTL7CMHI4INEW6YOOQVMCIIC2NGS3XYJA, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.413653431 (8h52m16.010424457s) [noise=true , address=193.149.164.69:4101 , piece_id=BTIBVPSORFJQR3CLA5UV62NXNI6GD3JF4D5Z74KU4R4CXB3WHHKA, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.414227965 (8h52m16.010998999s) [noise=true , address=45.91.116.14:4199 , piece_id=EKGDG7LYV3O5NSDL3UPWFKXINYNAOSPYYOWQPTFN326AJ2ZXAXHQ, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.415177146 (8h52m16.01194818s) [noise=true , address=77.90.31.16:4763 , piece_id=5Q7H3CU3MWSFPQGC67XMLCMTSUIEJQUV7QTO4UZLZY4MIVM76Z6Q, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.415227889 (8h52m16.011998912s) [noise=true , address=45.91.116.10:4096 , piece_id=EIKWFKPE5BYSZGV5T4RPQED55PG2FXU3ZKSNRVCBD363YQDHS4BQ, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.415728547 (8h52m16.012499581s) [noise=true , address=77.90.31.14:4693 , piece_id=S7NAS2ZBHAQPZRHBRGRSZPLE6COQEEN6SZVMB7X2X6IMXTN5HCBQ, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.415910147 (8h52m16.012681174s) [noise=true , address=193.149.164.70:4129 , piece_id=V7WMI7RDYDOAEQYWF2OTZL5LRPUA2S3V5WZN4BKDY6BI4YV3GRPQ, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.416592462 (8h52m16.01336349s) [noise=true , address=77.90.31.18:4831 , piece_id=4PMIUNE4YXKLIJPML3VSUN4PF32TSLPOOSDZZNAXFWH7WICBOW3Q, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.417118897 (8h52m16.01388993s) [noise=true , address=77.90.31.13:4655 , piece_id=47Q27QI7O2IYW7YQKFMJGGMWPVZWWKQTFVYPUMPUI7ORE7236XIQ, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.417185844 (8h52m16.013956873s) [noise=true , address=193.149.164.71:4304 , piece_id=DTCL54GM76GWKAE7A3X5PUNFYO7H5JIKRT5WBEHYEDF2YNFYTOAA, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.417686790 (8h52m16.014457823s) [noise=true , address=167.17.72.71:4810 , piece_id=ANLHBMENEXSGTHBHLOIHMLM6JGHKIU2X6FPQQIUIVMVUOABNK73Q, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.418236951 (8h52m16.015007982s) [noise=true , address=145.79.227.17:4994 , piece_id=TO3EOAILO4PHF6Z7WQA4RWZNNL3JO6EXXEF2RLW7IHAGWUU2TF4Q, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.419293640 (8h52m16.016064672s) [noise=true , address=64.49.13.139:4479 , piece_id=XUYHOCSBATYECYHWLVZ4VBCMT3H4O5ETSQ5HZOB42OR3OY4LWJ6Q, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.420005502 (8h52m16.016776534s) [noise=true , address=45.91.116.8:4066 , piece_id=5MZHIJ3ELVBAJ7ABRKG565TLU4A4NKB7X6WAACY2JBSLACL5LGVQ, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.483654433 (8h52m16.080425441s) [upload=9902]: Next returned nil segment Mar 6 19:52:18.483671157 (8h52m16.08044219s) [upload=9902]: Waiting for error group managing segments... Mar 6 19:52:18.488078321 (8h52m16.084849398s) [noise=true , address=77.90.31.13:4655 , piece_id=47Q27QI7O2IYW7YQKFMJGGMWPVZWWKQTFVYPUMPUI7ORE7236XIQ, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:52:18.498435619 (8h52m16.095206647s) [noise=true , address=145.79.227.17:4994 , piece_id=TO3EOAILO4PHF6Z7WQA4RWZNNL3JO6EXXEF2RLW7IHAGWUU2TF4Q, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:52:18.502979814 (8h52m16.099750841s) [noise=true , address=77.90.31.16:4763 , piece_id=5Q7H3CU3MWSFPQGC67XMLCMTSUIEJQUV7QTO4UZLZY4MIVM76Z6Q, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:52:18.506259140 (8h52m16.103030166s) [noise=true , address=77.90.31.14:4693 , piece_id=S7NAS2ZBHAQPZRHBRGRSZPLE6COQEEN6SZVMB7X2X6IMXTN5HCBQ, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:52:18.514529389 (8h52m16.111300413s) [noise=true , address=145.79.227.39:4022 , piece_id=U5GX3GLVWGWLU2BDWTHM7ST3MLLJDEAERWU3UEHO7Z2RXHGZU2DA, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:52:18.785925494 (8h52m16.382696517s) [noise=true , address=167.17.72.70:4771 , piece_id=TDLPTUHPNXCICPPXRTRL7WCJSXPX22ONIWWPHNXFUNIKRYPYW52Q, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:52:18.810774903 (8h52m16.407545926s) [noise=true , address=77.90.31.18:4831 , piece_id=4PMIUNE4YXKLIJPML3VSUN4PF32TSLPOOSDZZNAXFWH7WICBOW3Q, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:52:19.002746550 (8h52m16.5995176s) [noise=true , address=193.149.164.71:4304 , piece_id=DTCL54GM76GWKAE7A3X5PUNFYO7H5JIKRT5WBEHYEDF2YNFYTOAA, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:52:19.007614901 (8h52m16.604385924s) [noise=true , address=193.149.164.67:4018 , piece_id=GTQQP355TZZF4W7PPBYZS6ZQ6WYK7I3QIE4DZC7STWAKSBAMT4BQ, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:52:19.010443151 (8h52m16.607214176s) [noise=true , address=193.149.164.69:4101 , piece_id=BTIBVPSORFJQR3CLA5UV62NXNI6GD3JF4D5Z74KU4R4CXB3WHHKA, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:52:19.012614014 (8h52m16.609385038s) [noise=true , address=193.149.164.70:4129 , piece_id=V7WMI7RDYDOAEQYWF2OTZL5LRPUA2S3V5WZN4BKDY6BI4YV3GRPQ, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:52:19.042145380 (8h52m16.638916406s) [noise=true , address=64.49.13.139:4479 , piece_id=XUYHOCSBATYECYHWLVZ4VBCMT3H4O5ETSQ5HZOB42OR3OY4LWJ6Q, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:52:21.314990180 (8h52m18.911761213s) [noise=true , address=167.17.72.71:4810 , piece_id=ANLHBMENEXSGTHBHLOIHMLM6JGHKIU2X6FPQQIUIVMVUOABNK73Q, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:53:18.344002671 (8h53m15.9407737s) [noise=true , address=45.91.116.10:4096 , piece_id=EIKWFKPE5BYSZGV5T4RPQED55PG2FXU3ZKSNRVCBD363YQDHS4BQ, seg_pos={0 0}, upload=9902]: Done uploading piece. err: ecclient: upload failed (node:12MZX1GwZGrEi2q44dPiYrwMoum5A2caZtXTS3qiLNcoyHhon7p, address:45.91.116.10:4096): protocol: write tcp 159.223.143.39:44996->45.91.116.10:4096: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 6 19:53:18.344173472 (8h53m15.940944497s) [noise=true , address=45.91.116.7:4037 , piece_id=YXDMXUBE7CEWLJZP6FPDCRVIDGG5CJ3Y3IKEKO7EE2DQUSG6W5RA, seg_pos={0 0}, upload=9902]: Done uploading piece. err: ecclient: upload failed (node:1iNPwE2XSfXcaXAXcrH2jcyNGYeDiiWvzAcDQffrmjPwTVqVmK, address:45.91.116.7:4037): protocol: write tcp 159.223.143.39:52854->45.91.116.7:4037: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 6 19:53:18.344535130 (8h53m15.941306156s) [noise=true , address=45.91.116.14:4199 , piece_id=EKGDG7LYV3O5NSDL3UPWFKXINYNAOSPYYOWQPTFN326AJ2ZXAXHQ, seg_pos={0 0}, upload=9902]: Done uploading piece. err: ecclient: upload failed (node:1d4B1KMLs2annUuJ3v81FJapS9Yg4omBTj48eWoi6HJxnSEjrf, address:45.91.116.14:4199): protocol: write tcp 159.223.143.39:57596->45.91.116.14:4199: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 6 19:53:18.344687364 (8h53m15.941458387s) [noise=true , address=45.91.116.11:4133 , piece_id=RZWV5JMLOJ36AH7KSDXVTL7CMHI4INEW6YOOQVMCIIC2NGS3XYJA, seg_pos={0 0}, upload=9902]: Done uploading piece. err: ecclient: upload failed (node:12e1wX7hiER8sX4MLm5qLue4mayiN4SxXFdZSxfymRaRZxRGU6E, address:45.91.116.11:4133): protocol: write tcp 159.223.143.39:56440->45.91.116.11:4133: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 6 19:53:18.345220236 (8h53m15.941991259s) [noise=true , address=45.91.116.8:4066 , piece_id=5MZHIJ3ELVBAJ7ABRKG565TLU4A4NKB7X6WAACY2JBSLACL5LGVQ, seg_pos={0 0}, upload=9902]: Done uploading piece. err: ecclient: upload failed (node:123uv49k8LraXJfYQg6KRQthrYs1Egc4c3eZWm5JPvDatFbykm8, address:45.91.116.8:4066): protocol: write tcp 159.223.143.39:44136->45.91.116.8:4066: use of closed network connection; context deadline exceeded Mar 6 19:53:18.345277850 (8h53m15.942048865s) [seg_pos={0 0}, upload=9902]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 6 19:53:18.367560006 (8h53m15.964331035s) [upload=9902]: Done uploading segments... Mar 6 19:53:18.367606646 (8h53m15.964377681s) [upload=9902]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 1.168881738s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............................X............................] duration: 1.264981651s trace: https://jaeger.dev.storj.io/trace/78791d41d75e16a0 FAIL (2026-03-06 20:12:24 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 19:52:18.382132233 (8h52m15.978903242s) [upload=9902]: Uploading segments... Mar 6 19:52:18.382227489 (8h52m15.978998515s) [upload=9902]: Got next segment. Inline: false Mar 6 19:52:18.382313252 (8h52m15.979084279s) [upload=9902]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 6 19:52:18.382319269 (8h52m15.979090296s) [upload=9902]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 6 19:52:18.411471452 (8h52m16.008242482s) [seg_pos={0 0}, upload=9902]: Begin upload segment... Mar 6 19:52:18.411537096 (8h52m16.008308109s) [seg_pos={0 0}, upload=9902]: Done begin upload segment. Mar 6 19:52:18.411564540 (8h52m16.008335586s) [noise=true , address=45.91.116.7:4037 , piece_id=YXDMXUBE7CEWLJZP6FPDCRVIDGG5CJ3Y3IKEKO7EE2DQUSG6W5RA, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.411626862 (8h52m16.008397897s) [noise=true , address=193.149.164.67:4018 , piece_id=GTQQP355TZZF4W7PPBYZS6ZQ6WYK7I3QIE4DZC7STWAKSBAMT4BQ, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.412356605 (8h52m16.009127631s) [noise=true , address=167.17.72.70:4771 , piece_id=TDLPTUHPNXCICPPXRTRL7WCJSXPX22ONIWWPHNXFUNIKRYPYW52Q, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.412900389 (8h52m16.009671415s) [noise=true , address=145.79.227.39:4022 , piece_id=U5GX3GLVWGWLU2BDWTHM7ST3MLLJDEAERWU3UEHO7Z2RXHGZU2DA, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.413467061 (8h52m16.010238103s) [noise=true , address=45.91.116.11:4133 , piece_id=RZWV5JMLOJ36AH7KSDXVTL7CMHI4INEW6YOOQVMCIIC2NGS3XYJA, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.413653431 (8h52m16.010424457s) [noise=true , address=193.149.164.69:4101 , piece_id=BTIBVPSORFJQR3CLA5UV62NXNI6GD3JF4D5Z74KU4R4CXB3WHHKA, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.414227965 (8h52m16.010998999s) [noise=true , address=45.91.116.14:4199 , piece_id=EKGDG7LYV3O5NSDL3UPWFKXINYNAOSPYYOWQPTFN326AJ2ZXAXHQ, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.415177146 (8h52m16.01194818s) [noise=true , address=77.90.31.16:4763 , piece_id=5Q7H3CU3MWSFPQGC67XMLCMTSUIEJQUV7QTO4UZLZY4MIVM76Z6Q, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.415227889 (8h52m16.011998912s) [noise=true , address=45.91.116.10:4096 , piece_id=EIKWFKPE5BYSZGV5T4RPQED55PG2FXU3ZKSNRVCBD363YQDHS4BQ, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.415728547 (8h52m16.012499581s) [noise=true , address=77.90.31.14:4693 , piece_id=S7NAS2ZBHAQPZRHBRGRSZPLE6COQEEN6SZVMB7X2X6IMXTN5HCBQ, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.415910147 (8h52m16.012681174s) [noise=true , address=193.149.164.70:4129 , piece_id=V7WMI7RDYDOAEQYWF2OTZL5LRPUA2S3V5WZN4BKDY6BI4YV3GRPQ, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.416592462 (8h52m16.01336349s) [noise=true , address=77.90.31.18:4831 , piece_id=4PMIUNE4YXKLIJPML3VSUN4PF32TSLPOOSDZZNAXFWH7WICBOW3Q, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.417118897 (8h52m16.01388993s) [noise=true , address=77.90.31.13:4655 , piece_id=47Q27QI7O2IYW7YQKFMJGGMWPVZWWKQTFVYPUMPUI7ORE7236XIQ, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.417185844 (8h52m16.013956873s) [noise=true , address=193.149.164.71:4304 , piece_id=DTCL54GM76GWKAE7A3X5PUNFYO7H5JIKRT5WBEHYEDF2YNFYTOAA, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.417686790 (8h52m16.014457823s) [noise=true , address=167.17.72.71:4810 , piece_id=ANLHBMENEXSGTHBHLOIHMLM6JGHKIU2X6FPQQIUIVMVUOABNK73Q, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.418236951 (8h52m16.015007982s) [noise=true , address=145.79.227.17:4994 , piece_id=TO3EOAILO4PHF6Z7WQA4RWZNNL3JO6EXXEF2RLW7IHAGWUU2TF4Q, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.419293640 (8h52m16.016064672s) [noise=true , address=64.49.13.139:4479 , piece_id=XUYHOCSBATYECYHWLVZ4VBCMT3H4O5ETSQ5HZOB42OR3OY4LWJ6Q, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.420005502 (8h52m16.016776534s) [noise=true , address=45.91.116.8:4066 , piece_id=5MZHIJ3ELVBAJ7ABRKG565TLU4A4NKB7X6WAACY2JBSLACL5LGVQ, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.483654433 (8h52m16.080425441s) [upload=9902]: Next returned nil segment Mar 6 19:52:18.483671157 (8h52m16.08044219s) [upload=9902]: Waiting for error group managing segments... Mar 6 19:52:18.488078321 (8h52m16.084849398s) [noise=true , address=77.90.31.13:4655 , piece_id=47Q27QI7O2IYW7YQKFMJGGMWPVZWWKQTFVYPUMPUI7ORE7236XIQ, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:52:18.498435619 (8h52m16.095206647s) [noise=true , address=145.79.227.17:4994 , piece_id=TO3EOAILO4PHF6Z7WQA4RWZNNL3JO6EXXEF2RLW7IHAGWUU2TF4Q, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:52:18.502979814 (8h52m16.099750841s) [noise=true , address=77.90.31.16:4763 , piece_id=5Q7H3CU3MWSFPQGC67XMLCMTSUIEJQUV7QTO4UZLZY4MIVM76Z6Q, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:52:18.506259140 (8h52m16.103030166s) [noise=true , address=77.90.31.14:4693 , piece_id=S7NAS2ZBHAQPZRHBRGRSZPLE6COQEEN6SZVMB7X2X6IMXTN5HCBQ, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:52:18.514529389 (8h52m16.111300413s) [noise=true , address=145.79.227.39:4022 , piece_id=U5GX3GLVWGWLU2BDWTHM7ST3MLLJDEAERWU3UEHO7Z2RXHGZU2DA, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:52:18.785925494 (8h52m16.382696517s) [noise=true , address=167.17.72.70:4771 , piece_id=TDLPTUHPNXCICPPXRTRL7WCJSXPX22ONIWWPHNXFUNIKRYPYW52Q, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:52:18.810774903 (8h52m16.407545926s) [noise=true , address=77.90.31.18:4831 , piece_id=4PMIUNE4YXKLIJPML3VSUN4PF32TSLPOOSDZZNAXFWH7WICBOW3Q, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:52:19.002746550 (8h52m16.5995176s) [noise=true , address=193.149.164.71:4304 , piece_id=DTCL54GM76GWKAE7A3X5PUNFYO7H5JIKRT5WBEHYEDF2YNFYTOAA, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:52:19.007614901 (8h52m16.604385924s) [noise=true , address=193.149.164.67:4018 , piece_id=GTQQP355TZZF4W7PPBYZS6ZQ6WYK7I3QIE4DZC7STWAKSBAMT4BQ, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:52:19.010443151 (8h52m16.607214176s) [noise=true , address=193.149.164.69:4101 , piece_id=BTIBVPSORFJQR3CLA5UV62NXNI6GD3JF4D5Z74KU4R4CXB3WHHKA, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:52:19.012614014 (8h52m16.609385038s) [noise=true , address=193.149.164.70:4129 , piece_id=V7WMI7RDYDOAEQYWF2OTZL5LRPUA2S3V5WZN4BKDY6BI4YV3GRPQ, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:52:19.042145380 (8h52m16.638916406s) [noise=true , address=64.49.13.139:4479 , piece_id=XUYHOCSBATYECYHWLVZ4VBCMT3H4O5ETSQ5HZOB42OR3OY4LWJ6Q, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:52:21.314990180 (8h52m18.911761213s) [noise=true , address=167.17.72.71:4810 , piece_id=ANLHBMENEXSGTHBHLOIHMLM6JGHKIU2X6FPQQIUIVMVUOABNK73Q, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:53:18.344002671 (8h53m15.9407737s) [noise=true , address=45.91.116.10:4096 , piece_id=EIKWFKPE5BYSZGV5T4RPQED55PG2FXU3ZKSNRVCBD363YQDHS4BQ, seg_pos={0 0}, upload=9902]: Done uploading piece. err: ecclient: upload failed (node:12MZX1GwZGrEi2q44dPiYrwMoum5A2caZtXTS3qiLNcoyHhon7p, address:45.91.116.10:4096): protocol: write tcp 159.223.143.39:44996->45.91.116.10:4096: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 6 19:53:18.344173472 (8h53m15.940944497s) [noise=true , address=45.91.116.7:4037 , piece_id=YXDMXUBE7CEWLJZP6FPDCRVIDGG5CJ3Y3IKEKO7EE2DQUSG6W5RA, seg_pos={0 0}, upload=9902]: Done uploading piece. err: ecclient: upload failed (node:1iNPwE2XSfXcaXAXcrH2jcyNGYeDiiWvzAcDQffrmjPwTVqVmK, address:45.91.116.7:4037): protocol: write tcp 159.223.143.39:52854->45.91.116.7:4037: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 6 19:53:18.344535130 (8h53m15.941306156s) [noise=true , address=45.91.116.14:4199 , piece_id=EKGDG7LYV3O5NSDL3UPWFKXINYNAOSPYYOWQPTFN326AJ2ZXAXHQ, seg_pos={0 0}, upload=9902]: Done uploading piece. err: ecclient: upload failed (node:1d4B1KMLs2annUuJ3v81FJapS9Yg4omBTj48eWoi6HJxnSEjrf, address:45.91.116.14:4199): protocol: write tcp 159.223.143.39:57596->45.91.116.14:4199: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 6 19:53:18.344687364 (8h53m15.941458387s) [noise=true , address=45.91.116.11:4133 , piece_id=RZWV5JMLOJ36AH7KSDXVTL7CMHI4INEW6YOOQVMCIIC2NGS3XYJA, seg_pos={0 0}, upload=9902]: Done uploading piece. err: ecclient: upload failed (node:12e1wX7hiER8sX4MLm5qLue4mayiN4SxXFdZSxfymRaRZxRGU6E, address:45.91.116.11:4133): protocol: write tcp 159.223.143.39:56440->45.91.116.11:4133: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 6 19:53:18.345220236 (8h53m15.941991259s) [noise=true , address=45.91.116.8:4066 , piece_id=5MZHIJ3ELVBAJ7ABRKG565TLU4A4NKB7X6WAACY2JBSLACL5LGVQ, seg_pos={0 0}, upload=9902]: Done uploading piece. err: ecclient: upload failed (node:123uv49k8LraXJfYQg6KRQthrYs1Egc4c3eZWm5JPvDatFbykm8, address:45.91.116.8:4066): protocol: write tcp 159.223.143.39:44136->45.91.116.8:4066: use of closed network connection; context deadline exceeded Mar 6 19:53:18.345277850 (8h53m15.942048865s) [seg_pos={0 0}, upload=9902]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 6 19:53:18.367560006 (8h53m15.964331035s) [upload=9902]: Done uploading segments... Mar 6 19:53:18.367606646 (8h53m15.964377681s) [upload=9902]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 1.265004654s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [................................X...........................] duration: 1.123163481s trace: https://jaeger.dev.storj.io/trace/78791d41d75e16a0 FAIL (2026-03-06 20:11:31 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 19:52:18.382132233 (8h52m15.978903242s) [upload=9902]: Uploading segments... Mar 6 19:52:18.382227489 (8h52m15.978998515s) [upload=9902]: Got next segment. Inline: false Mar 6 19:52:18.382313252 (8h52m15.979084279s) [upload=9902]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 6 19:52:18.382319269 (8h52m15.979090296s) [upload=9902]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 6 19:52:18.411471452 (8h52m16.008242482s) [seg_pos={0 0}, upload=9902]: Begin upload segment... Mar 6 19:52:18.411537096 (8h52m16.008308109s) [seg_pos={0 0}, upload=9902]: Done begin upload segment. Mar 6 19:52:18.411564540 (8h52m16.008335586s) [noise=true , address=45.91.116.7:4037 , piece_id=YXDMXUBE7CEWLJZP6FPDCRVIDGG5CJ3Y3IKEKO7EE2DQUSG6W5RA, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.411626862 (8h52m16.008397897s) [noise=true , address=193.149.164.67:4018 , piece_id=GTQQP355TZZF4W7PPBYZS6ZQ6WYK7I3QIE4DZC7STWAKSBAMT4BQ, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.412356605 (8h52m16.009127631s) [noise=true , address=167.17.72.70:4771 , piece_id=TDLPTUHPNXCICPPXRTRL7WCJSXPX22ONIWWPHNXFUNIKRYPYW52Q, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.412900389 (8h52m16.009671415s) [noise=true , address=145.79.227.39:4022 , piece_id=U5GX3GLVWGWLU2BDWTHM7ST3MLLJDEAERWU3UEHO7Z2RXHGZU2DA, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.413467061 (8h52m16.010238103s) [noise=true , address=45.91.116.11:4133 , piece_id=RZWV5JMLOJ36AH7KSDXVTL7CMHI4INEW6YOOQVMCIIC2NGS3XYJA, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.413653431 (8h52m16.010424457s) [noise=true , address=193.149.164.69:4101 , piece_id=BTIBVPSORFJQR3CLA5UV62NXNI6GD3JF4D5Z74KU4R4CXB3WHHKA, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.414227965 (8h52m16.010998999s) [noise=true , address=45.91.116.14:4199 , piece_id=EKGDG7LYV3O5NSDL3UPWFKXINYNAOSPYYOWQPTFN326AJ2ZXAXHQ, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.415177146 (8h52m16.01194818s) [noise=true , address=77.90.31.16:4763 , piece_id=5Q7H3CU3MWSFPQGC67XMLCMTSUIEJQUV7QTO4UZLZY4MIVM76Z6Q, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.415227889 (8h52m16.011998912s) [noise=true , address=45.91.116.10:4096 , piece_id=EIKWFKPE5BYSZGV5T4RPQED55PG2FXU3ZKSNRVCBD363YQDHS4BQ, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.415728547 (8h52m16.012499581s) [noise=true , address=77.90.31.14:4693 , piece_id=S7NAS2ZBHAQPZRHBRGRSZPLE6COQEEN6SZVMB7X2X6IMXTN5HCBQ, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.415910147 (8h52m16.012681174s) [noise=true , address=193.149.164.70:4129 , piece_id=V7WMI7RDYDOAEQYWF2OTZL5LRPUA2S3V5WZN4BKDY6BI4YV3GRPQ, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.416592462 (8h52m16.01336349s) [noise=true , address=77.90.31.18:4831 , piece_id=4PMIUNE4YXKLIJPML3VSUN4PF32TSLPOOSDZZNAXFWH7WICBOW3Q, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.417118897 (8h52m16.01388993s) [noise=true , address=77.90.31.13:4655 , piece_id=47Q27QI7O2IYW7YQKFMJGGMWPVZWWKQTFVYPUMPUI7ORE7236XIQ, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.417185844 (8h52m16.013956873s) [noise=true , address=193.149.164.71:4304 , piece_id=DTCL54GM76GWKAE7A3X5PUNFYO7H5JIKRT5WBEHYEDF2YNFYTOAA, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.417686790 (8h52m16.014457823s) [noise=true , address=167.17.72.71:4810 , piece_id=ANLHBMENEXSGTHBHLOIHMLM6JGHKIU2X6FPQQIUIVMVUOABNK73Q, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.418236951 (8h52m16.015007982s) [noise=true , address=145.79.227.17:4994 , piece_id=TO3EOAILO4PHF6Z7WQA4RWZNNL3JO6EXXEF2RLW7IHAGWUU2TF4Q, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.419293640 (8h52m16.016064672s) [noise=true , address=64.49.13.139:4479 , piece_id=XUYHOCSBATYECYHWLVZ4VBCMT3H4O5ETSQ5HZOB42OR3OY4LWJ6Q, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.420005502 (8h52m16.016776534s) [noise=true , address=45.91.116.8:4066 , piece_id=5MZHIJ3ELVBAJ7ABRKG565TLU4A4NKB7X6WAACY2JBSLACL5LGVQ, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.483654433 (8h52m16.080425441s) [upload=9902]: Next returned nil segment Mar 6 19:52:18.483671157 (8h52m16.08044219s) [upload=9902]: Waiting for error group managing segments... Mar 6 19:52:18.488078321 (8h52m16.084849398s) [noise=true , address=77.90.31.13:4655 , piece_id=47Q27QI7O2IYW7YQKFMJGGMWPVZWWKQTFVYPUMPUI7ORE7236XIQ, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:52:18.498435619 (8h52m16.095206647s) [noise=true , address=145.79.227.17:4994 , piece_id=TO3EOAILO4PHF6Z7WQA4RWZNNL3JO6EXXEF2RLW7IHAGWUU2TF4Q, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:52:18.502979814 (8h52m16.099750841s) [noise=true , address=77.90.31.16:4763 , piece_id=5Q7H3CU3MWSFPQGC67XMLCMTSUIEJQUV7QTO4UZLZY4MIVM76Z6Q, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:52:18.506259140 (8h52m16.103030166s) [noise=true , address=77.90.31.14:4693 , piece_id=S7NAS2ZBHAQPZRHBRGRSZPLE6COQEEN6SZVMB7X2X6IMXTN5HCBQ, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:52:18.514529389 (8h52m16.111300413s) [noise=true , address=145.79.227.39:4022 , piece_id=U5GX3GLVWGWLU2BDWTHM7ST3MLLJDEAERWU3UEHO7Z2RXHGZU2DA, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:52:18.785925494 (8h52m16.382696517s) [noise=true , address=167.17.72.70:4771 , piece_id=TDLPTUHPNXCICPPXRTRL7WCJSXPX22ONIWWPHNXFUNIKRYPYW52Q, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:52:18.810774903 (8h52m16.407545926s) [noise=true , address=77.90.31.18:4831 , piece_id=4PMIUNE4YXKLIJPML3VSUN4PF32TSLPOOSDZZNAXFWH7WICBOW3Q, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:52:19.002746550 (8h52m16.5995176s) [noise=true , address=193.149.164.71:4304 , piece_id=DTCL54GM76GWKAE7A3X5PUNFYO7H5JIKRT5WBEHYEDF2YNFYTOAA, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:52:19.007614901 (8h52m16.604385924s) [noise=true , address=193.149.164.67:4018 , piece_id=GTQQP355TZZF4W7PPBYZS6ZQ6WYK7I3QIE4DZC7STWAKSBAMT4BQ, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:52:19.010443151 (8h52m16.607214176s) [noise=true , address=193.149.164.69:4101 , piece_id=BTIBVPSORFJQR3CLA5UV62NXNI6GD3JF4D5Z74KU4R4CXB3WHHKA, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:52:19.012614014 (8h52m16.609385038s) [noise=true , address=193.149.164.70:4129 , piece_id=V7WMI7RDYDOAEQYWF2OTZL5LRPUA2S3V5WZN4BKDY6BI4YV3GRPQ, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:52:19.042145380 (8h52m16.638916406s) [noise=true , address=64.49.13.139:4479 , piece_id=XUYHOCSBATYECYHWLVZ4VBCMT3H4O5ETSQ5HZOB42OR3OY4LWJ6Q, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:52:21.314990180 (8h52m18.911761213s) [noise=true , address=167.17.72.71:4810 , piece_id=ANLHBMENEXSGTHBHLOIHMLM6JGHKIU2X6FPQQIUIVMVUOABNK73Q, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:53:18.344002671 (8h53m15.9407737s) [noise=true , address=45.91.116.10:4096 , piece_id=EIKWFKPE5BYSZGV5T4RPQED55PG2FXU3ZKSNRVCBD363YQDHS4BQ, seg_pos={0 0}, upload=9902]: Done uploading piece. err: ecclient: upload failed (node:12MZX1GwZGrEi2q44dPiYrwMoum5A2caZtXTS3qiLNcoyHhon7p, address:45.91.116.10:4096): protocol: write tcp 159.223.143.39:44996->45.91.116.10:4096: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 6 19:53:18.344173472 (8h53m15.940944497s) [noise=true , address=45.91.116.7:4037 , piece_id=YXDMXUBE7CEWLJZP6FPDCRVIDGG5CJ3Y3IKEKO7EE2DQUSG6W5RA, seg_pos={0 0}, upload=9902]: Done uploading piece. err: ecclient: upload failed (node:1iNPwE2XSfXcaXAXcrH2jcyNGYeDiiWvzAcDQffrmjPwTVqVmK, address:45.91.116.7:4037): protocol: write tcp 159.223.143.39:52854->45.91.116.7:4037: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 6 19:53:18.344535130 (8h53m15.941306156s) [noise=true , address=45.91.116.14:4199 , piece_id=EKGDG7LYV3O5NSDL3UPWFKXINYNAOSPYYOWQPTFN326AJ2ZXAXHQ, seg_pos={0 0}, upload=9902]: Done uploading piece. err: ecclient: upload failed (node:1d4B1KMLs2annUuJ3v81FJapS9Yg4omBTj48eWoi6HJxnSEjrf, address:45.91.116.14:4199): protocol: write tcp 159.223.143.39:57596->45.91.116.14:4199: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 6 19:53:18.344687364 (8h53m15.941458387s) [noise=true , address=45.91.116.11:4133 , piece_id=RZWV5JMLOJ36AH7KSDXVTL7CMHI4INEW6YOOQVMCIIC2NGS3XYJA, seg_pos={0 0}, upload=9902]: Done uploading piece. err: ecclient: upload failed (node:12e1wX7hiER8sX4MLm5qLue4mayiN4SxXFdZSxfymRaRZxRGU6E, address:45.91.116.11:4133): protocol: write tcp 159.223.143.39:56440->45.91.116.11:4133: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 6 19:53:18.345220236 (8h53m15.941991259s) [noise=true , address=45.91.116.8:4066 , piece_id=5MZHIJ3ELVBAJ7ABRKG565TLU4A4NKB7X6WAACY2JBSLACL5LGVQ, seg_pos={0 0}, upload=9902]: Done uploading piece. err: ecclient: upload failed (node:123uv49k8LraXJfYQg6KRQthrYs1Egc4c3eZWm5JPvDatFbykm8, address:45.91.116.8:4066): protocol: write tcp 159.223.143.39:44136->45.91.116.8:4066: use of closed network connection; context deadline exceeded Mar 6 19:53:18.345277850 (8h53m15.942048865s) [seg_pos={0 0}, upload=9902]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 6 19:53:18.367560006 (8h53m15.964331035s) [upload=9902]: Done uploading segments... Mar 6 19:53:18.367606646 (8h53m15.964377681s) [upload=9902]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 1.123198409s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................................X..........................] duration: 1.37170608s trace: https://jaeger.dev.storj.io/trace/78791d41d75e16a0 FAIL (2026-03-06 20:10:28 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 19:52:18.382132233 (8h52m15.978903242s) [upload=9902]: Uploading segments... Mar 6 19:52:18.382227489 (8h52m15.978998515s) [upload=9902]: Got next segment. Inline: false Mar 6 19:52:18.382313252 (8h52m15.979084279s) [upload=9902]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 6 19:52:18.382319269 (8h52m15.979090296s) [upload=9902]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 6 19:52:18.411471452 (8h52m16.008242482s) [seg_pos={0 0}, upload=9902]: Begin upload segment... Mar 6 19:52:18.411537096 (8h52m16.008308109s) [seg_pos={0 0}, upload=9902]: Done begin upload segment. Mar 6 19:52:18.411564540 (8h52m16.008335586s) [noise=true , address=45.91.116.7:4037 , piece_id=YXDMXUBE7CEWLJZP6FPDCRVIDGG5CJ3Y3IKEKO7EE2DQUSG6W5RA, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.411626862 (8h52m16.008397897s) [noise=true , address=193.149.164.67:4018 , piece_id=GTQQP355TZZF4W7PPBYZS6ZQ6WYK7I3QIE4DZC7STWAKSBAMT4BQ, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.412356605 (8h52m16.009127631s) [noise=true , address=167.17.72.70:4771 , piece_id=TDLPTUHPNXCICPPXRTRL7WCJSXPX22ONIWWPHNXFUNIKRYPYW52Q, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.412900389 (8h52m16.009671415s) [noise=true , address=145.79.227.39:4022 , piece_id=U5GX3GLVWGWLU2BDWTHM7ST3MLLJDEAERWU3UEHO7Z2RXHGZU2DA, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.413467061 (8h52m16.010238103s) [noise=true , address=45.91.116.11:4133 , piece_id=RZWV5JMLOJ36AH7KSDXVTL7CMHI4INEW6YOOQVMCIIC2NGS3XYJA, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.413653431 (8h52m16.010424457s) [noise=true , address=193.149.164.69:4101 , piece_id=BTIBVPSORFJQR3CLA5UV62NXNI6GD3JF4D5Z74KU4R4CXB3WHHKA, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.414227965 (8h52m16.010998999s) [noise=true , address=45.91.116.14:4199 , piece_id=EKGDG7LYV3O5NSDL3UPWFKXINYNAOSPYYOWQPTFN326AJ2ZXAXHQ, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.415177146 (8h52m16.01194818s) [noise=true , address=77.90.31.16:4763 , piece_id=5Q7H3CU3MWSFPQGC67XMLCMTSUIEJQUV7QTO4UZLZY4MIVM76Z6Q, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.415227889 (8h52m16.011998912s) [noise=true , address=45.91.116.10:4096 , piece_id=EIKWFKPE5BYSZGV5T4RPQED55PG2FXU3ZKSNRVCBD363YQDHS4BQ, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.415728547 (8h52m16.012499581s) [noise=true , address=77.90.31.14:4693 , piece_id=S7NAS2ZBHAQPZRHBRGRSZPLE6COQEEN6SZVMB7X2X6IMXTN5HCBQ, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.415910147 (8h52m16.012681174s) [noise=true , address=193.149.164.70:4129 , piece_id=V7WMI7RDYDOAEQYWF2OTZL5LRPUA2S3V5WZN4BKDY6BI4YV3GRPQ, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.416592462 (8h52m16.01336349s) [noise=true , address=77.90.31.18:4831 , piece_id=4PMIUNE4YXKLIJPML3VSUN4PF32TSLPOOSDZZNAXFWH7WICBOW3Q, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.417118897 (8h52m16.01388993s) [noise=true , address=77.90.31.13:4655 , piece_id=47Q27QI7O2IYW7YQKFMJGGMWPVZWWKQTFVYPUMPUI7ORE7236XIQ, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.417185844 (8h52m16.013956873s) [noise=true , address=193.149.164.71:4304 , piece_id=DTCL54GM76GWKAE7A3X5PUNFYO7H5JIKRT5WBEHYEDF2YNFYTOAA, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.417686790 (8h52m16.014457823s) [noise=true , address=167.17.72.71:4810 , piece_id=ANLHBMENEXSGTHBHLOIHMLM6JGHKIU2X6FPQQIUIVMVUOABNK73Q, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.418236951 (8h52m16.015007982s) [noise=true , address=145.79.227.17:4994 , piece_id=TO3EOAILO4PHF6Z7WQA4RWZNNL3JO6EXXEF2RLW7IHAGWUU2TF4Q, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.419293640 (8h52m16.016064672s) [noise=true , address=64.49.13.139:4479 , piece_id=XUYHOCSBATYECYHWLVZ4VBCMT3H4O5ETSQ5HZOB42OR3OY4LWJ6Q, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.420005502 (8h52m16.016776534s) [noise=true , address=45.91.116.8:4066 , piece_id=5MZHIJ3ELVBAJ7ABRKG565TLU4A4NKB7X6WAACY2JBSLACL5LGVQ, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.483654433 (8h52m16.080425441s) [upload=9902]: Next returned nil segment Mar 6 19:52:18.483671157 (8h52m16.08044219s) [upload=9902]: Waiting for error group managing segments... Mar 6 19:52:18.488078321 (8h52m16.084849398s) [noise=true , address=77.90.31.13:4655 , piece_id=47Q27QI7O2IYW7YQKFMJGGMWPVZWWKQTFVYPUMPUI7ORE7236XIQ, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:52:18.498435619 (8h52m16.095206647s) [noise=true , address=145.79.227.17:4994 , piece_id=TO3EOAILO4PHF6Z7WQA4RWZNNL3JO6EXXEF2RLW7IHAGWUU2TF4Q, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:52:18.502979814 (8h52m16.099750841s) [noise=true , address=77.90.31.16:4763 , piece_id=5Q7H3CU3MWSFPQGC67XMLCMTSUIEJQUV7QTO4UZLZY4MIVM76Z6Q, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:52:18.506259140 (8h52m16.103030166s) [noise=true , address=77.90.31.14:4693 , piece_id=S7NAS2ZBHAQPZRHBRGRSZPLE6COQEEN6SZVMB7X2X6IMXTN5HCBQ, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:52:18.514529389 (8h52m16.111300413s) [noise=true , address=145.79.227.39:4022 , piece_id=U5GX3GLVWGWLU2BDWTHM7ST3MLLJDEAERWU3UEHO7Z2RXHGZU2DA, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:52:18.785925494 (8h52m16.382696517s) [noise=true , address=167.17.72.70:4771 , piece_id=TDLPTUHPNXCICPPXRTRL7WCJSXPX22ONIWWPHNXFUNIKRYPYW52Q, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:52:18.810774903 (8h52m16.407545926s) [noise=true , address=77.90.31.18:4831 , piece_id=4PMIUNE4YXKLIJPML3VSUN4PF32TSLPOOSDZZNAXFWH7WICBOW3Q, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:52:19.002746550 (8h52m16.5995176s) [noise=true , address=193.149.164.71:4304 , piece_id=DTCL54GM76GWKAE7A3X5PUNFYO7H5JIKRT5WBEHYEDF2YNFYTOAA, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:52:19.007614901 (8h52m16.604385924s) [noise=true , address=193.149.164.67:4018 , piece_id=GTQQP355TZZF4W7PPBYZS6ZQ6WYK7I3QIE4DZC7STWAKSBAMT4BQ, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:52:19.010443151 (8h52m16.607214176s) [noise=true , address=193.149.164.69:4101 , piece_id=BTIBVPSORFJQR3CLA5UV62NXNI6GD3JF4D5Z74KU4R4CXB3WHHKA, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:52:19.012614014 (8h52m16.609385038s) [noise=true , address=193.149.164.70:4129 , piece_id=V7WMI7RDYDOAEQYWF2OTZL5LRPUA2S3V5WZN4BKDY6BI4YV3GRPQ, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:52:19.042145380 (8h52m16.638916406s) [noise=true , address=64.49.13.139:4479 , piece_id=XUYHOCSBATYECYHWLVZ4VBCMT3H4O5ETSQ5HZOB42OR3OY4LWJ6Q, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:52:21.314990180 (8h52m18.911761213s) [noise=true , address=167.17.72.71:4810 , piece_id=ANLHBMENEXSGTHBHLOIHMLM6JGHKIU2X6FPQQIUIVMVUOABNK73Q, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:53:18.344002671 (8h53m15.9407737s) [noise=true , address=45.91.116.10:4096 , piece_id=EIKWFKPE5BYSZGV5T4RPQED55PG2FXU3ZKSNRVCBD363YQDHS4BQ, seg_pos={0 0}, upload=9902]: Done uploading piece. err: ecclient: upload failed (node:12MZX1GwZGrEi2q44dPiYrwMoum5A2caZtXTS3qiLNcoyHhon7p, address:45.91.116.10:4096): protocol: write tcp 159.223.143.39:44996->45.91.116.10:4096: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 6 19:53:18.344173472 (8h53m15.940944497s) [noise=true , address=45.91.116.7:4037 , piece_id=YXDMXUBE7CEWLJZP6FPDCRVIDGG5CJ3Y3IKEKO7EE2DQUSG6W5RA, seg_pos={0 0}, upload=9902]: Done uploading piece. err: ecclient: upload failed (node:1iNPwE2XSfXcaXAXcrH2jcyNGYeDiiWvzAcDQffrmjPwTVqVmK, address:45.91.116.7:4037): protocol: write tcp 159.223.143.39:52854->45.91.116.7:4037: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 6 19:53:18.344535130 (8h53m15.941306156s) [noise=true , address=45.91.116.14:4199 , piece_id=EKGDG7LYV3O5NSDL3UPWFKXINYNAOSPYYOWQPTFN326AJ2ZXAXHQ, seg_pos={0 0}, upload=9902]: Done uploading piece. err: ecclient: upload failed (node:1d4B1KMLs2annUuJ3v81FJapS9Yg4omBTj48eWoi6HJxnSEjrf, address:45.91.116.14:4199): protocol: write tcp 159.223.143.39:57596->45.91.116.14:4199: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 6 19:53:18.344687364 (8h53m15.941458387s) [noise=true , address=45.91.116.11:4133 , piece_id=RZWV5JMLOJ36AH7KSDXVTL7CMHI4INEW6YOOQVMCIIC2NGS3XYJA, seg_pos={0 0}, upload=9902]: Done uploading piece. err: ecclient: upload failed (node:12e1wX7hiER8sX4MLm5qLue4mayiN4SxXFdZSxfymRaRZxRGU6E, address:45.91.116.11:4133): protocol: write tcp 159.223.143.39:56440->45.91.116.11:4133: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 6 19:53:18.345220236 (8h53m15.941991259s) [noise=true , address=45.91.116.8:4066 , piece_id=5MZHIJ3ELVBAJ7ABRKG565TLU4A4NKB7X6WAACY2JBSLACL5LGVQ, seg_pos={0 0}, upload=9902]: Done uploading piece. err: ecclient: upload failed (node:123uv49k8LraXJfYQg6KRQthrYs1Egc4c3eZWm5JPvDatFbykm8, address:45.91.116.8:4066): protocol: write tcp 159.223.143.39:44136->45.91.116.8:4066: use of closed network connection; context deadline exceeded Mar 6 19:53:18.345277850 (8h53m15.942048865s) [seg_pos={0 0}, upload=9902]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 6 19:53:18.367560006 (8h53m15.964331035s) [upload=9902]: Done uploading segments... Mar 6 19:53:18.367606646 (8h53m15.964377681s) [upload=9902]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 1.371738422s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................................X.........................] duration: 1.258893629s trace: https://jaeger.dev.storj.io/trace/78791d41d75e16a0 FAIL (2026-03-06 20:09:45 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 19:52:18.382132233 (8h52m15.978903242s) [upload=9902]: Uploading segments... Mar 6 19:52:18.382227489 (8h52m15.978998515s) [upload=9902]: Got next segment. Inline: false Mar 6 19:52:18.382313252 (8h52m15.979084279s) [upload=9902]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 6 19:52:18.382319269 (8h52m15.979090296s) [upload=9902]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 6 19:52:18.411471452 (8h52m16.008242482s) [seg_pos={0 0}, upload=9902]: Begin upload segment... Mar 6 19:52:18.411537096 (8h52m16.008308109s) [seg_pos={0 0}, upload=9902]: Done begin upload segment. Mar 6 19:52:18.411564540 (8h52m16.008335586s) [noise=true , address=45.91.116.7:4037 , piece_id=YXDMXUBE7CEWLJZP6FPDCRVIDGG5CJ3Y3IKEKO7EE2DQUSG6W5RA, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.411626862 (8h52m16.008397897s) [noise=true , address=193.149.164.67:4018 , piece_id=GTQQP355TZZF4W7PPBYZS6ZQ6WYK7I3QIE4DZC7STWAKSBAMT4BQ, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.412356605 (8h52m16.009127631s) [noise=true , address=167.17.72.70:4771 , piece_id=TDLPTUHPNXCICPPXRTRL7WCJSXPX22ONIWWPHNXFUNIKRYPYW52Q, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.412900389 (8h52m16.009671415s) [noise=true , address=145.79.227.39:4022 , piece_id=U5GX3GLVWGWLU2BDWTHM7ST3MLLJDEAERWU3UEHO7Z2RXHGZU2DA, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.413467061 (8h52m16.010238103s) [noise=true , address=45.91.116.11:4133 , piece_id=RZWV5JMLOJ36AH7KSDXVTL7CMHI4INEW6YOOQVMCIIC2NGS3XYJA, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.413653431 (8h52m16.010424457s) [noise=true , address=193.149.164.69:4101 , piece_id=BTIBVPSORFJQR3CLA5UV62NXNI6GD3JF4D5Z74KU4R4CXB3WHHKA, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.414227965 (8h52m16.010998999s) [noise=true , address=45.91.116.14:4199 , piece_id=EKGDG7LYV3O5NSDL3UPWFKXINYNAOSPYYOWQPTFN326AJ2ZXAXHQ, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.415177146 (8h52m16.01194818s) [noise=true , address=77.90.31.16:4763 , piece_id=5Q7H3CU3MWSFPQGC67XMLCMTSUIEJQUV7QTO4UZLZY4MIVM76Z6Q, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.415227889 (8h52m16.011998912s) [noise=true , address=45.91.116.10:4096 , piece_id=EIKWFKPE5BYSZGV5T4RPQED55PG2FXU3ZKSNRVCBD363YQDHS4BQ, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.415728547 (8h52m16.012499581s) [noise=true , address=77.90.31.14:4693 , piece_id=S7NAS2ZBHAQPZRHBRGRSZPLE6COQEEN6SZVMB7X2X6IMXTN5HCBQ, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.415910147 (8h52m16.012681174s) [noise=true , address=193.149.164.70:4129 , piece_id=V7WMI7RDYDOAEQYWF2OTZL5LRPUA2S3V5WZN4BKDY6BI4YV3GRPQ, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.416592462 (8h52m16.01336349s) [noise=true , address=77.90.31.18:4831 , piece_id=4PMIUNE4YXKLIJPML3VSUN4PF32TSLPOOSDZZNAXFWH7WICBOW3Q, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.417118897 (8h52m16.01388993s) [noise=true , address=77.90.31.13:4655 , piece_id=47Q27QI7O2IYW7YQKFMJGGMWPVZWWKQTFVYPUMPUI7ORE7236XIQ, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.417185844 (8h52m16.013956873s) [noise=true , address=193.149.164.71:4304 , piece_id=DTCL54GM76GWKAE7A3X5PUNFYO7H5JIKRT5WBEHYEDF2YNFYTOAA, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.417686790 (8h52m16.014457823s) [noise=true , address=167.17.72.71:4810 , piece_id=ANLHBMENEXSGTHBHLOIHMLM6JGHKIU2X6FPQQIUIVMVUOABNK73Q, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.418236951 (8h52m16.015007982s) [noise=true , address=145.79.227.17:4994 , piece_id=TO3EOAILO4PHF6Z7WQA4RWZNNL3JO6EXXEF2RLW7IHAGWUU2TF4Q, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.419293640 (8h52m16.016064672s) [noise=true , address=64.49.13.139:4479 , piece_id=XUYHOCSBATYECYHWLVZ4VBCMT3H4O5ETSQ5HZOB42OR3OY4LWJ6Q, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.420005502 (8h52m16.016776534s) [noise=true , address=45.91.116.8:4066 , piece_id=5MZHIJ3ELVBAJ7ABRKG565TLU4A4NKB7X6WAACY2JBSLACL5LGVQ, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.483654433 (8h52m16.080425441s) [upload=9902]: Next returned nil segment Mar 6 19:52:18.483671157 (8h52m16.08044219s) [upload=9902]: Waiting for error group managing segments... Mar 6 19:52:18.488078321 (8h52m16.084849398s) [noise=true , address=77.90.31.13:4655 , piece_id=47Q27QI7O2IYW7YQKFMJGGMWPVZWWKQTFVYPUMPUI7ORE7236XIQ, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:52:18.498435619 (8h52m16.095206647s) [noise=true , address=145.79.227.17:4994 , piece_id=TO3EOAILO4PHF6Z7WQA4RWZNNL3JO6EXXEF2RLW7IHAGWUU2TF4Q, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:52:18.502979814 (8h52m16.099750841s) [noise=true , address=77.90.31.16:4763 , piece_id=5Q7H3CU3MWSFPQGC67XMLCMTSUIEJQUV7QTO4UZLZY4MIVM76Z6Q, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:52:18.506259140 (8h52m16.103030166s) [noise=true , address=77.90.31.14:4693 , piece_id=S7NAS2ZBHAQPZRHBRGRSZPLE6COQEEN6SZVMB7X2X6IMXTN5HCBQ, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:52:18.514529389 (8h52m16.111300413s) [noise=true , address=145.79.227.39:4022 , piece_id=U5GX3GLVWGWLU2BDWTHM7ST3MLLJDEAERWU3UEHO7Z2RXHGZU2DA, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:52:18.785925494 (8h52m16.382696517s) [noise=true , address=167.17.72.70:4771 , piece_id=TDLPTUHPNXCICPPXRTRL7WCJSXPX22ONIWWPHNXFUNIKRYPYW52Q, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:52:18.810774903 (8h52m16.407545926s) [noise=true , address=77.90.31.18:4831 , piece_id=4PMIUNE4YXKLIJPML3VSUN4PF32TSLPOOSDZZNAXFWH7WICBOW3Q, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:52:19.002746550 (8h52m16.5995176s) [noise=true , address=193.149.164.71:4304 , piece_id=DTCL54GM76GWKAE7A3X5PUNFYO7H5JIKRT5WBEHYEDF2YNFYTOAA, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:52:19.007614901 (8h52m16.604385924s) [noise=true , address=193.149.164.67:4018 , piece_id=GTQQP355TZZF4W7PPBYZS6ZQ6WYK7I3QIE4DZC7STWAKSBAMT4BQ, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:52:19.010443151 (8h52m16.607214176s) [noise=true , address=193.149.164.69:4101 , piece_id=BTIBVPSORFJQR3CLA5UV62NXNI6GD3JF4D5Z74KU4R4CXB3WHHKA, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:52:19.012614014 (8h52m16.609385038s) [noise=true , address=193.149.164.70:4129 , piece_id=V7WMI7RDYDOAEQYWF2OTZL5LRPUA2S3V5WZN4BKDY6BI4YV3GRPQ, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:52:19.042145380 (8h52m16.638916406s) [noise=true , address=64.49.13.139:4479 , piece_id=XUYHOCSBATYECYHWLVZ4VBCMT3H4O5ETSQ5HZOB42OR3OY4LWJ6Q, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:52:21.314990180 (8h52m18.911761213s) [noise=true , address=167.17.72.71:4810 , piece_id=ANLHBMENEXSGTHBHLOIHMLM6JGHKIU2X6FPQQIUIVMVUOABNK73Q, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:53:18.344002671 (8h53m15.9407737s) [noise=true , address=45.91.116.10:4096 , piece_id=EIKWFKPE5BYSZGV5T4RPQED55PG2FXU3ZKSNRVCBD363YQDHS4BQ, seg_pos={0 0}, upload=9902]: Done uploading piece. err: ecclient: upload failed (node:12MZX1GwZGrEi2q44dPiYrwMoum5A2caZtXTS3qiLNcoyHhon7p, address:45.91.116.10:4096): protocol: write tcp 159.223.143.39:44996->45.91.116.10:4096: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 6 19:53:18.344173472 (8h53m15.940944497s) [noise=true , address=45.91.116.7:4037 , piece_id=YXDMXUBE7CEWLJZP6FPDCRVIDGG5CJ3Y3IKEKO7EE2DQUSG6W5RA, seg_pos={0 0}, upload=9902]: Done uploading piece. err: ecclient: upload failed (node:1iNPwE2XSfXcaXAXcrH2jcyNGYeDiiWvzAcDQffrmjPwTVqVmK, address:45.91.116.7:4037): protocol: write tcp 159.223.143.39:52854->45.91.116.7:4037: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 6 19:53:18.344535130 (8h53m15.941306156s) [noise=true , address=45.91.116.14:4199 , piece_id=EKGDG7LYV3O5NSDL3UPWFKXINYNAOSPYYOWQPTFN326AJ2ZXAXHQ, seg_pos={0 0}, upload=9902]: Done uploading piece. err: ecclient: upload failed (node:1d4B1KMLs2annUuJ3v81FJapS9Yg4omBTj48eWoi6HJxnSEjrf, address:45.91.116.14:4199): protocol: write tcp 159.223.143.39:57596->45.91.116.14:4199: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 6 19:53:18.344687364 (8h53m15.941458387s) [noise=true , address=45.91.116.11:4133 , piece_id=RZWV5JMLOJ36AH7KSDXVTL7CMHI4INEW6YOOQVMCIIC2NGS3XYJA, seg_pos={0 0}, upload=9902]: Done uploading piece. err: ecclient: upload failed (node:12e1wX7hiER8sX4MLm5qLue4mayiN4SxXFdZSxfymRaRZxRGU6E, address:45.91.116.11:4133): protocol: write tcp 159.223.143.39:56440->45.91.116.11:4133: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 6 19:53:18.345220236 (8h53m15.941991259s) [noise=true , address=45.91.116.8:4066 , piece_id=5MZHIJ3ELVBAJ7ABRKG565TLU4A4NKB7X6WAACY2JBSLACL5LGVQ, seg_pos={0 0}, upload=9902]: Done uploading piece. err: ecclient: upload failed (node:123uv49k8LraXJfYQg6KRQthrYs1Egc4c3eZWm5JPvDatFbykm8, address:45.91.116.8:4066): protocol: write tcp 159.223.143.39:44136->45.91.116.8:4066: use of closed network connection; context deadline exceeded Mar 6 19:53:18.345277850 (8h53m15.942048865s) [seg_pos={0 0}, upload=9902]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 6 19:53:18.367560006 (8h53m15.964331035s) [upload=9902]: Done uploading segments... Mar 6 19:53:18.367606646 (8h53m15.964377681s) [upload=9902]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 1.258923653s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................................X........................] duration: 1.053780632s trace: https://jaeger.dev.storj.io/trace/78791d41d75e16a0 FAIL (2026-03-06 20:08:46 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 19:52:18.382132233 (8h52m15.978903242s) [upload=9902]: Uploading segments... Mar 6 19:52:18.382227489 (8h52m15.978998515s) [upload=9902]: Got next segment. Inline: false Mar 6 19:52:18.382313252 (8h52m15.979084279s) [upload=9902]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 6 19:52:18.382319269 (8h52m15.979090296s) [upload=9902]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 6 19:52:18.411471452 (8h52m16.008242482s) [seg_pos={0 0}, upload=9902]: Begin upload segment... Mar 6 19:52:18.411537096 (8h52m16.008308109s) [seg_pos={0 0}, upload=9902]: Done begin upload segment. Mar 6 19:52:18.411564540 (8h52m16.008335586s) [noise=true , address=45.91.116.7:4037 , piece_id=YXDMXUBE7CEWLJZP6FPDCRVIDGG5CJ3Y3IKEKO7EE2DQUSG6W5RA, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.411626862 (8h52m16.008397897s) [noise=true , address=193.149.164.67:4018 , piece_id=GTQQP355TZZF4W7PPBYZS6ZQ6WYK7I3QIE4DZC7STWAKSBAMT4BQ, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.412356605 (8h52m16.009127631s) [noise=true , address=167.17.72.70:4771 , piece_id=TDLPTUHPNXCICPPXRTRL7WCJSXPX22ONIWWPHNXFUNIKRYPYW52Q, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.412900389 (8h52m16.009671415s) [noise=true , address=145.79.227.39:4022 , piece_id=U5GX3GLVWGWLU2BDWTHM7ST3MLLJDEAERWU3UEHO7Z2RXHGZU2DA, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.413467061 (8h52m16.010238103s) [noise=true , address=45.91.116.11:4133 , piece_id=RZWV5JMLOJ36AH7KSDXVTL7CMHI4INEW6YOOQVMCIIC2NGS3XYJA, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.413653431 (8h52m16.010424457s) [noise=true , address=193.149.164.69:4101 , piece_id=BTIBVPSORFJQR3CLA5UV62NXNI6GD3JF4D5Z74KU4R4CXB3WHHKA, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.414227965 (8h52m16.010998999s) [noise=true , address=45.91.116.14:4199 , piece_id=EKGDG7LYV3O5NSDL3UPWFKXINYNAOSPYYOWQPTFN326AJ2ZXAXHQ, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.415177146 (8h52m16.01194818s) [noise=true , address=77.90.31.16:4763 , piece_id=5Q7H3CU3MWSFPQGC67XMLCMTSUIEJQUV7QTO4UZLZY4MIVM76Z6Q, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.415227889 (8h52m16.011998912s) [noise=true , address=45.91.116.10:4096 , piece_id=EIKWFKPE5BYSZGV5T4RPQED55PG2FXU3ZKSNRVCBD363YQDHS4BQ, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.415728547 (8h52m16.012499581s) [noise=true , address=77.90.31.14:4693 , piece_id=S7NAS2ZBHAQPZRHBRGRSZPLE6COQEEN6SZVMB7X2X6IMXTN5HCBQ, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.415910147 (8h52m16.012681174s) [noise=true , address=193.149.164.70:4129 , piece_id=V7WMI7RDYDOAEQYWF2OTZL5LRPUA2S3V5WZN4BKDY6BI4YV3GRPQ, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.416592462 (8h52m16.01336349s) [noise=true , address=77.90.31.18:4831 , piece_id=4PMIUNE4YXKLIJPML3VSUN4PF32TSLPOOSDZZNAXFWH7WICBOW3Q, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.417118897 (8h52m16.01388993s) [noise=true , address=77.90.31.13:4655 , piece_id=47Q27QI7O2IYW7YQKFMJGGMWPVZWWKQTFVYPUMPUI7ORE7236XIQ, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.417185844 (8h52m16.013956873s) [noise=true , address=193.149.164.71:4304 , piece_id=DTCL54GM76GWKAE7A3X5PUNFYO7H5JIKRT5WBEHYEDF2YNFYTOAA, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.417686790 (8h52m16.014457823s) [noise=true , address=167.17.72.71:4810 , piece_id=ANLHBMENEXSGTHBHLOIHMLM6JGHKIU2X6FPQQIUIVMVUOABNK73Q, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.418236951 (8h52m16.015007982s) [noise=true , address=145.79.227.17:4994 , piece_id=TO3EOAILO4PHF6Z7WQA4RWZNNL3JO6EXXEF2RLW7IHAGWUU2TF4Q, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.419293640 (8h52m16.016064672s) [noise=true , address=64.49.13.139:4479 , piece_id=XUYHOCSBATYECYHWLVZ4VBCMT3H4O5ETSQ5HZOB42OR3OY4LWJ6Q, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.420005502 (8h52m16.016776534s) [noise=true , address=45.91.116.8:4066 , piece_id=5MZHIJ3ELVBAJ7ABRKG565TLU4A4NKB7X6WAACY2JBSLACL5LGVQ, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.483654433 (8h52m16.080425441s) [upload=9902]: Next returned nil segment Mar 6 19:52:18.483671157 (8h52m16.08044219s) [upload=9902]: Waiting for error group managing segments... Mar 6 19:52:18.488078321 (8h52m16.084849398s) [noise=true , address=77.90.31.13:4655 , piece_id=47Q27QI7O2IYW7YQKFMJGGMWPVZWWKQTFVYPUMPUI7ORE7236XIQ, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:52:18.498435619 (8h52m16.095206647s) [noise=true , address=145.79.227.17:4994 , piece_id=TO3EOAILO4PHF6Z7WQA4RWZNNL3JO6EXXEF2RLW7IHAGWUU2TF4Q, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:52:18.502979814 (8h52m16.099750841s) [noise=true , address=77.90.31.16:4763 , piece_id=5Q7H3CU3MWSFPQGC67XMLCMTSUIEJQUV7QTO4UZLZY4MIVM76Z6Q, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:52:18.506259140 (8h52m16.103030166s) [noise=true , address=77.90.31.14:4693 , piece_id=S7NAS2ZBHAQPZRHBRGRSZPLE6COQEEN6SZVMB7X2X6IMXTN5HCBQ, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:52:18.514529389 (8h52m16.111300413s) [noise=true , address=145.79.227.39:4022 , piece_id=U5GX3GLVWGWLU2BDWTHM7ST3MLLJDEAERWU3UEHO7Z2RXHGZU2DA, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:52:18.785925494 (8h52m16.382696517s) [noise=true , address=167.17.72.70:4771 , piece_id=TDLPTUHPNXCICPPXRTRL7WCJSXPX22ONIWWPHNXFUNIKRYPYW52Q, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:52:18.810774903 (8h52m16.407545926s) [noise=true , address=77.90.31.18:4831 , piece_id=4PMIUNE4YXKLIJPML3VSUN4PF32TSLPOOSDZZNAXFWH7WICBOW3Q, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:52:19.002746550 (8h52m16.5995176s) [noise=true , address=193.149.164.71:4304 , piece_id=DTCL54GM76GWKAE7A3X5PUNFYO7H5JIKRT5WBEHYEDF2YNFYTOAA, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:52:19.007614901 (8h52m16.604385924s) [noise=true , address=193.149.164.67:4018 , piece_id=GTQQP355TZZF4W7PPBYZS6ZQ6WYK7I3QIE4DZC7STWAKSBAMT4BQ, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:52:19.010443151 (8h52m16.607214176s) [noise=true , address=193.149.164.69:4101 , piece_id=BTIBVPSORFJQR3CLA5UV62NXNI6GD3JF4D5Z74KU4R4CXB3WHHKA, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:52:19.012614014 (8h52m16.609385038s) [noise=true , address=193.149.164.70:4129 , piece_id=V7WMI7RDYDOAEQYWF2OTZL5LRPUA2S3V5WZN4BKDY6BI4YV3GRPQ, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:52:19.042145380 (8h52m16.638916406s) [noise=true , address=64.49.13.139:4479 , piece_id=XUYHOCSBATYECYHWLVZ4VBCMT3H4O5ETSQ5HZOB42OR3OY4LWJ6Q, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:52:21.314990180 (8h52m18.911761213s) [noise=true , address=167.17.72.71:4810 , piece_id=ANLHBMENEXSGTHBHLOIHMLM6JGHKIU2X6FPQQIUIVMVUOABNK73Q, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:53:18.344002671 (8h53m15.9407737s) [noise=true , address=45.91.116.10:4096 , piece_id=EIKWFKPE5BYSZGV5T4RPQED55PG2FXU3ZKSNRVCBD363YQDHS4BQ, seg_pos={0 0}, upload=9902]: Done uploading piece. err: ecclient: upload failed (node:12MZX1GwZGrEi2q44dPiYrwMoum5A2caZtXTS3qiLNcoyHhon7p, address:45.91.116.10:4096): protocol: write tcp 159.223.143.39:44996->45.91.116.10:4096: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 6 19:53:18.344173472 (8h53m15.940944497s) [noise=true , address=45.91.116.7:4037 , piece_id=YXDMXUBE7CEWLJZP6FPDCRVIDGG5CJ3Y3IKEKO7EE2DQUSG6W5RA, seg_pos={0 0}, upload=9902]: Done uploading piece. err: ecclient: upload failed (node:1iNPwE2XSfXcaXAXcrH2jcyNGYeDiiWvzAcDQffrmjPwTVqVmK, address:45.91.116.7:4037): protocol: write tcp 159.223.143.39:52854->45.91.116.7:4037: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 6 19:53:18.344535130 (8h53m15.941306156s) [noise=true , address=45.91.116.14:4199 , piece_id=EKGDG7LYV3O5NSDL3UPWFKXINYNAOSPYYOWQPTFN326AJ2ZXAXHQ, seg_pos={0 0}, upload=9902]: Done uploading piece. err: ecclient: upload failed (node:1d4B1KMLs2annUuJ3v81FJapS9Yg4omBTj48eWoi6HJxnSEjrf, address:45.91.116.14:4199): protocol: write tcp 159.223.143.39:57596->45.91.116.14:4199: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 6 19:53:18.344687364 (8h53m15.941458387s) [noise=true , address=45.91.116.11:4133 , piece_id=RZWV5JMLOJ36AH7KSDXVTL7CMHI4INEW6YOOQVMCIIC2NGS3XYJA, seg_pos={0 0}, upload=9902]: Done uploading piece. err: ecclient: upload failed (node:12e1wX7hiER8sX4MLm5qLue4mayiN4SxXFdZSxfymRaRZxRGU6E, address:45.91.116.11:4133): protocol: write tcp 159.223.143.39:56440->45.91.116.11:4133: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 6 19:53:18.345220236 (8h53m15.941991259s) [noise=true , address=45.91.116.8:4066 , piece_id=5MZHIJ3ELVBAJ7ABRKG565TLU4A4NKB7X6WAACY2JBSLACL5LGVQ, seg_pos={0 0}, upload=9902]: Done uploading piece. err: ecclient: upload failed (node:123uv49k8LraXJfYQg6KRQthrYs1Egc4c3eZWm5JPvDatFbykm8, address:45.91.116.8:4066): protocol: write tcp 159.223.143.39:44136->45.91.116.8:4066: use of closed network connection; context deadline exceeded Mar 6 19:53:18.345277850 (8h53m15.942048865s) [seg_pos={0 0}, upload=9902]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 6 19:53:18.367560006 (8h53m15.964331035s) [upload=9902]: Done uploading segments... Mar 6 19:53:18.367606646 (8h53m15.964377681s) [upload=9902]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 1.053831005s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................................X.......................] duration: 1.389461366s trace: https://jaeger.dev.storj.io/trace/78791d41d75e16a0 FAIL (2026-03-06 20:07:44 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 19:52:18.382132233 (8h52m15.978903242s) [upload=9902]: Uploading segments... Mar 6 19:52:18.382227489 (8h52m15.978998515s) [upload=9902]: Got next segment. Inline: false Mar 6 19:52:18.382313252 (8h52m15.979084279s) [upload=9902]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 6 19:52:18.382319269 (8h52m15.979090296s) [upload=9902]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 6 19:52:18.411471452 (8h52m16.008242482s) [seg_pos={0 0}, upload=9902]: Begin upload segment... Mar 6 19:52:18.411537096 (8h52m16.008308109s) [seg_pos={0 0}, upload=9902]: Done begin upload segment. Mar 6 19:52:18.411564540 (8h52m16.008335586s) [noise=true , address=45.91.116.7:4037 , piece_id=YXDMXUBE7CEWLJZP6FPDCRVIDGG5CJ3Y3IKEKO7EE2DQUSG6W5RA, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.411626862 (8h52m16.008397897s) [noise=true , address=193.149.164.67:4018 , piece_id=GTQQP355TZZF4W7PPBYZS6ZQ6WYK7I3QIE4DZC7STWAKSBAMT4BQ, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.412356605 (8h52m16.009127631s) [noise=true , address=167.17.72.70:4771 , piece_id=TDLPTUHPNXCICPPXRTRL7WCJSXPX22ONIWWPHNXFUNIKRYPYW52Q, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.412900389 (8h52m16.009671415s) [noise=true , address=145.79.227.39:4022 , piece_id=U5GX3GLVWGWLU2BDWTHM7ST3MLLJDEAERWU3UEHO7Z2RXHGZU2DA, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.413467061 (8h52m16.010238103s) [noise=true , address=45.91.116.11:4133 , piece_id=RZWV5JMLOJ36AH7KSDXVTL7CMHI4INEW6YOOQVMCIIC2NGS3XYJA, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.413653431 (8h52m16.010424457s) [noise=true , address=193.149.164.69:4101 , piece_id=BTIBVPSORFJQR3CLA5UV62NXNI6GD3JF4D5Z74KU4R4CXB3WHHKA, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.414227965 (8h52m16.010998999s) [noise=true , address=45.91.116.14:4199 , piece_id=EKGDG7LYV3O5NSDL3UPWFKXINYNAOSPYYOWQPTFN326AJ2ZXAXHQ, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.415177146 (8h52m16.01194818s) [noise=true , address=77.90.31.16:4763 , piece_id=5Q7H3CU3MWSFPQGC67XMLCMTSUIEJQUV7QTO4UZLZY4MIVM76Z6Q, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.415227889 (8h52m16.011998912s) [noise=true , address=45.91.116.10:4096 , piece_id=EIKWFKPE5BYSZGV5T4RPQED55PG2FXU3ZKSNRVCBD363YQDHS4BQ, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.415728547 (8h52m16.012499581s) [noise=true , address=77.90.31.14:4693 , piece_id=S7NAS2ZBHAQPZRHBRGRSZPLE6COQEEN6SZVMB7X2X6IMXTN5HCBQ, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.415910147 (8h52m16.012681174s) [noise=true , address=193.149.164.70:4129 , piece_id=V7WMI7RDYDOAEQYWF2OTZL5LRPUA2S3V5WZN4BKDY6BI4YV3GRPQ, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.416592462 (8h52m16.01336349s) [noise=true , address=77.90.31.18:4831 , piece_id=4PMIUNE4YXKLIJPML3VSUN4PF32TSLPOOSDZZNAXFWH7WICBOW3Q, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.417118897 (8h52m16.01388993s) [noise=true , address=77.90.31.13:4655 , piece_id=47Q27QI7O2IYW7YQKFMJGGMWPVZWWKQTFVYPUMPUI7ORE7236XIQ, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.417185844 (8h52m16.013956873s) [noise=true , address=193.149.164.71:4304 , piece_id=DTCL54GM76GWKAE7A3X5PUNFYO7H5JIKRT5WBEHYEDF2YNFYTOAA, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.417686790 (8h52m16.014457823s) [noise=true , address=167.17.72.71:4810 , piece_id=ANLHBMENEXSGTHBHLOIHMLM6JGHKIU2X6FPQQIUIVMVUOABNK73Q, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.418236951 (8h52m16.015007982s) [noise=true , address=145.79.227.17:4994 , piece_id=TO3EOAILO4PHF6Z7WQA4RWZNNL3JO6EXXEF2RLW7IHAGWUU2TF4Q, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.419293640 (8h52m16.016064672s) [noise=true , address=64.49.13.139:4479 , piece_id=XUYHOCSBATYECYHWLVZ4VBCMT3H4O5ETSQ5HZOB42OR3OY4LWJ6Q, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.420005502 (8h52m16.016776534s) [noise=true , address=45.91.116.8:4066 , piece_id=5MZHIJ3ELVBAJ7ABRKG565TLU4A4NKB7X6WAACY2JBSLACL5LGVQ, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.483654433 (8h52m16.080425441s) [upload=9902]: Next returned nil segment Mar 6 19:52:18.483671157 (8h52m16.08044219s) [upload=9902]: Waiting for error group managing segments... Mar 6 19:52:18.488078321 (8h52m16.084849398s) [noise=true , address=77.90.31.13:4655 , piece_id=47Q27QI7O2IYW7YQKFMJGGMWPVZWWKQTFVYPUMPUI7ORE7236XIQ, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:52:18.498435619 (8h52m16.095206647s) [noise=true , address=145.79.227.17:4994 , piece_id=TO3EOAILO4PHF6Z7WQA4RWZNNL3JO6EXXEF2RLW7IHAGWUU2TF4Q, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:52:18.502979814 (8h52m16.099750841s) [noise=true , address=77.90.31.16:4763 , piece_id=5Q7H3CU3MWSFPQGC67XMLCMTSUIEJQUV7QTO4UZLZY4MIVM76Z6Q, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:52:18.506259140 (8h52m16.103030166s) [noise=true , address=77.90.31.14:4693 , piece_id=S7NAS2ZBHAQPZRHBRGRSZPLE6COQEEN6SZVMB7X2X6IMXTN5HCBQ, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:52:18.514529389 (8h52m16.111300413s) [noise=true , address=145.79.227.39:4022 , piece_id=U5GX3GLVWGWLU2BDWTHM7ST3MLLJDEAERWU3UEHO7Z2RXHGZU2DA, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:52:18.785925494 (8h52m16.382696517s) [noise=true , address=167.17.72.70:4771 , piece_id=TDLPTUHPNXCICPPXRTRL7WCJSXPX22ONIWWPHNXFUNIKRYPYW52Q, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:52:18.810774903 (8h52m16.407545926s) [noise=true , address=77.90.31.18:4831 , piece_id=4PMIUNE4YXKLIJPML3VSUN4PF32TSLPOOSDZZNAXFWH7WICBOW3Q, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:52:19.002746550 (8h52m16.5995176s) [noise=true , address=193.149.164.71:4304 , piece_id=DTCL54GM76GWKAE7A3X5PUNFYO7H5JIKRT5WBEHYEDF2YNFYTOAA, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:52:19.007614901 (8h52m16.604385924s) [noise=true , address=193.149.164.67:4018 , piece_id=GTQQP355TZZF4W7PPBYZS6ZQ6WYK7I3QIE4DZC7STWAKSBAMT4BQ, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:52:19.010443151 (8h52m16.607214176s) [noise=true , address=193.149.164.69:4101 , piece_id=BTIBVPSORFJQR3CLA5UV62NXNI6GD3JF4D5Z74KU4R4CXB3WHHKA, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:52:19.012614014 (8h52m16.609385038s) [noise=true , address=193.149.164.70:4129 , piece_id=V7WMI7RDYDOAEQYWF2OTZL5LRPUA2S3V5WZN4BKDY6BI4YV3GRPQ, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:52:19.042145380 (8h52m16.638916406s) [noise=true , address=64.49.13.139:4479 , piece_id=XUYHOCSBATYECYHWLVZ4VBCMT3H4O5ETSQ5HZOB42OR3OY4LWJ6Q, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:52:21.314990180 (8h52m18.911761213s) [noise=true , address=167.17.72.71:4810 , piece_id=ANLHBMENEXSGTHBHLOIHMLM6JGHKIU2X6FPQQIUIVMVUOABNK73Q, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:53:18.344002671 (8h53m15.9407737s) [noise=true , address=45.91.116.10:4096 , piece_id=EIKWFKPE5BYSZGV5T4RPQED55PG2FXU3ZKSNRVCBD363YQDHS4BQ, seg_pos={0 0}, upload=9902]: Done uploading piece. err: ecclient: upload failed (node:12MZX1GwZGrEi2q44dPiYrwMoum5A2caZtXTS3qiLNcoyHhon7p, address:45.91.116.10:4096): protocol: write tcp 159.223.143.39:44996->45.91.116.10:4096: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 6 19:53:18.344173472 (8h53m15.940944497s) [noise=true , address=45.91.116.7:4037 , piece_id=YXDMXUBE7CEWLJZP6FPDCRVIDGG5CJ3Y3IKEKO7EE2DQUSG6W5RA, seg_pos={0 0}, upload=9902]: Done uploading piece. err: ecclient: upload failed (node:1iNPwE2XSfXcaXAXcrH2jcyNGYeDiiWvzAcDQffrmjPwTVqVmK, address:45.91.116.7:4037): protocol: write tcp 159.223.143.39:52854->45.91.116.7:4037: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 6 19:53:18.344535130 (8h53m15.941306156s) [noise=true , address=45.91.116.14:4199 , piece_id=EKGDG7LYV3O5NSDL3UPWFKXINYNAOSPYYOWQPTFN326AJ2ZXAXHQ, seg_pos={0 0}, upload=9902]: Done uploading piece. err: ecclient: upload failed (node:1d4B1KMLs2annUuJ3v81FJapS9Yg4omBTj48eWoi6HJxnSEjrf, address:45.91.116.14:4199): protocol: write tcp 159.223.143.39:57596->45.91.116.14:4199: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 6 19:53:18.344687364 (8h53m15.941458387s) [noise=true , address=45.91.116.11:4133 , piece_id=RZWV5JMLOJ36AH7KSDXVTL7CMHI4INEW6YOOQVMCIIC2NGS3XYJA, seg_pos={0 0}, upload=9902]: Done uploading piece. err: ecclient: upload failed (node:12e1wX7hiER8sX4MLm5qLue4mayiN4SxXFdZSxfymRaRZxRGU6E, address:45.91.116.11:4133): protocol: write tcp 159.223.143.39:56440->45.91.116.11:4133: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 6 19:53:18.345220236 (8h53m15.941991259s) [noise=true , address=45.91.116.8:4066 , piece_id=5MZHIJ3ELVBAJ7ABRKG565TLU4A4NKB7X6WAACY2JBSLACL5LGVQ, seg_pos={0 0}, upload=9902]: Done uploading piece. err: ecclient: upload failed (node:123uv49k8LraXJfYQg6KRQthrYs1Egc4c3eZWm5JPvDatFbykm8, address:45.91.116.8:4066): protocol: write tcp 159.223.143.39:44136->45.91.116.8:4066: use of closed network connection; context deadline exceeded Mar 6 19:53:18.345277850 (8h53m15.942048865s) [seg_pos={0 0}, upload=9902]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 6 19:53:18.367560006 (8h53m15.964331035s) [upload=9902]: Done uploading segments... Mar 6 19:53:18.367606646 (8h53m15.964377681s) [upload=9902]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 1.389491621s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................................X......................] duration: 1.116433096s trace: https://jaeger.dev.storj.io/trace/78791d41d75e16a0 FAIL (2026-03-06 20:07:10 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 19:52:18.382132233 (8h52m15.978903242s) [upload=9902]: Uploading segments... Mar 6 19:52:18.382227489 (8h52m15.978998515s) [upload=9902]: Got next segment. Inline: false Mar 6 19:52:18.382313252 (8h52m15.979084279s) [upload=9902]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 6 19:52:18.382319269 (8h52m15.979090296s) [upload=9902]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 6 19:52:18.411471452 (8h52m16.008242482s) [seg_pos={0 0}, upload=9902]: Begin upload segment... Mar 6 19:52:18.411537096 (8h52m16.008308109s) [seg_pos={0 0}, upload=9902]: Done begin upload segment. Mar 6 19:52:18.411564540 (8h52m16.008335586s) [noise=true , address=45.91.116.7:4037 , piece_id=YXDMXUBE7CEWLJZP6FPDCRVIDGG5CJ3Y3IKEKO7EE2DQUSG6W5RA, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.411626862 (8h52m16.008397897s) [noise=true , address=193.149.164.67:4018 , piece_id=GTQQP355TZZF4W7PPBYZS6ZQ6WYK7I3QIE4DZC7STWAKSBAMT4BQ, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.412356605 (8h52m16.009127631s) [noise=true , address=167.17.72.70:4771 , piece_id=TDLPTUHPNXCICPPXRTRL7WCJSXPX22ONIWWPHNXFUNIKRYPYW52Q, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.412900389 (8h52m16.009671415s) [noise=true , address=145.79.227.39:4022 , piece_id=U5GX3GLVWGWLU2BDWTHM7ST3MLLJDEAERWU3UEHO7Z2RXHGZU2DA, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.413467061 (8h52m16.010238103s) [noise=true , address=45.91.116.11:4133 , piece_id=RZWV5JMLOJ36AH7KSDXVTL7CMHI4INEW6YOOQVMCIIC2NGS3XYJA, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.413653431 (8h52m16.010424457s) [noise=true , address=193.149.164.69:4101 , piece_id=BTIBVPSORFJQR3CLA5UV62NXNI6GD3JF4D5Z74KU4R4CXB3WHHKA, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.414227965 (8h52m16.010998999s) [noise=true , address=45.91.116.14:4199 , piece_id=EKGDG7LYV3O5NSDL3UPWFKXINYNAOSPYYOWQPTFN326AJ2ZXAXHQ, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.415177146 (8h52m16.01194818s) [noise=true , address=77.90.31.16:4763 , piece_id=5Q7H3CU3MWSFPQGC67XMLCMTSUIEJQUV7QTO4UZLZY4MIVM76Z6Q, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.415227889 (8h52m16.011998912s) [noise=true , address=45.91.116.10:4096 , piece_id=EIKWFKPE5BYSZGV5T4RPQED55PG2FXU3ZKSNRVCBD363YQDHS4BQ, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.415728547 (8h52m16.012499581s) [noise=true , address=77.90.31.14:4693 , piece_id=S7NAS2ZBHAQPZRHBRGRSZPLE6COQEEN6SZVMB7X2X6IMXTN5HCBQ, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.415910147 (8h52m16.012681174s) [noise=true , address=193.149.164.70:4129 , piece_id=V7WMI7RDYDOAEQYWF2OTZL5LRPUA2S3V5WZN4BKDY6BI4YV3GRPQ, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.416592462 (8h52m16.01336349s) [noise=true , address=77.90.31.18:4831 , piece_id=4PMIUNE4YXKLIJPML3VSUN4PF32TSLPOOSDZZNAXFWH7WICBOW3Q, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.417118897 (8h52m16.01388993s) [noise=true , address=77.90.31.13:4655 , piece_id=47Q27QI7O2IYW7YQKFMJGGMWPVZWWKQTFVYPUMPUI7ORE7236XIQ, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.417185844 (8h52m16.013956873s) [noise=true , address=193.149.164.71:4304 , piece_id=DTCL54GM76GWKAE7A3X5PUNFYO7H5JIKRT5WBEHYEDF2YNFYTOAA, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.417686790 (8h52m16.014457823s) [noise=true , address=167.17.72.71:4810 , piece_id=ANLHBMENEXSGTHBHLOIHMLM6JGHKIU2X6FPQQIUIVMVUOABNK73Q, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.418236951 (8h52m16.015007982s) [noise=true , address=145.79.227.17:4994 , piece_id=TO3EOAILO4PHF6Z7WQA4RWZNNL3JO6EXXEF2RLW7IHAGWUU2TF4Q, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.419293640 (8h52m16.016064672s) [noise=true , address=64.49.13.139:4479 , piece_id=XUYHOCSBATYECYHWLVZ4VBCMT3H4O5ETSQ5HZOB42OR3OY4LWJ6Q, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.420005502 (8h52m16.016776534s) [noise=true , address=45.91.116.8:4066 , piece_id=5MZHIJ3ELVBAJ7ABRKG565TLU4A4NKB7X6WAACY2JBSLACL5LGVQ, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.483654433 (8h52m16.080425441s) [upload=9902]: Next returned nil segment Mar 6 19:52:18.483671157 (8h52m16.08044219s) [upload=9902]: Waiting for error group managing segments... Mar 6 19:52:18.488078321 (8h52m16.084849398s) [noise=true , address=77.90.31.13:4655 , piece_id=47Q27QI7O2IYW7YQKFMJGGMWPVZWWKQTFVYPUMPUI7ORE7236XIQ, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:52:18.498435619 (8h52m16.095206647s) [noise=true , address=145.79.227.17:4994 , piece_id=TO3EOAILO4PHF6Z7WQA4RWZNNL3JO6EXXEF2RLW7IHAGWUU2TF4Q, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:52:18.502979814 (8h52m16.099750841s) [noise=true , address=77.90.31.16:4763 , piece_id=5Q7H3CU3MWSFPQGC67XMLCMTSUIEJQUV7QTO4UZLZY4MIVM76Z6Q, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:52:18.506259140 (8h52m16.103030166s) [noise=true , address=77.90.31.14:4693 , piece_id=S7NAS2ZBHAQPZRHBRGRSZPLE6COQEEN6SZVMB7X2X6IMXTN5HCBQ, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:52:18.514529389 (8h52m16.111300413s) [noise=true , address=145.79.227.39:4022 , piece_id=U5GX3GLVWGWLU2BDWTHM7ST3MLLJDEAERWU3UEHO7Z2RXHGZU2DA, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:52:18.785925494 (8h52m16.382696517s) [noise=true , address=167.17.72.70:4771 , piece_id=TDLPTUHPNXCICPPXRTRL7WCJSXPX22ONIWWPHNXFUNIKRYPYW52Q, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:52:18.810774903 (8h52m16.407545926s) [noise=true , address=77.90.31.18:4831 , piece_id=4PMIUNE4YXKLIJPML3VSUN4PF32TSLPOOSDZZNAXFWH7WICBOW3Q, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:52:19.002746550 (8h52m16.5995176s) [noise=true , address=193.149.164.71:4304 , piece_id=DTCL54GM76GWKAE7A3X5PUNFYO7H5JIKRT5WBEHYEDF2YNFYTOAA, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:52:19.007614901 (8h52m16.604385924s) [noise=true , address=193.149.164.67:4018 , piece_id=GTQQP355TZZF4W7PPBYZS6ZQ6WYK7I3QIE4DZC7STWAKSBAMT4BQ, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:52:19.010443151 (8h52m16.607214176s) [noise=true , address=193.149.164.69:4101 , piece_id=BTIBVPSORFJQR3CLA5UV62NXNI6GD3JF4D5Z74KU4R4CXB3WHHKA, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:52:19.012614014 (8h52m16.609385038s) [noise=true , address=193.149.164.70:4129 , piece_id=V7WMI7RDYDOAEQYWF2OTZL5LRPUA2S3V5WZN4BKDY6BI4YV3GRPQ, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:52:19.042145380 (8h52m16.638916406s) [noise=true , address=64.49.13.139:4479 , piece_id=XUYHOCSBATYECYHWLVZ4VBCMT3H4O5ETSQ5HZOB42OR3OY4LWJ6Q, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:52:21.314990180 (8h52m18.911761213s) [noise=true , address=167.17.72.71:4810 , piece_id=ANLHBMENEXSGTHBHLOIHMLM6JGHKIU2X6FPQQIUIVMVUOABNK73Q, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:53:18.344002671 (8h53m15.9407737s) [noise=true , address=45.91.116.10:4096 , piece_id=EIKWFKPE5BYSZGV5T4RPQED55PG2FXU3ZKSNRVCBD363YQDHS4BQ, seg_pos={0 0}, upload=9902]: Done uploading piece. err: ecclient: upload failed (node:12MZX1GwZGrEi2q44dPiYrwMoum5A2caZtXTS3qiLNcoyHhon7p, address:45.91.116.10:4096): protocol: write tcp 159.223.143.39:44996->45.91.116.10:4096: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 6 19:53:18.344173472 (8h53m15.940944497s) [noise=true , address=45.91.116.7:4037 , piece_id=YXDMXUBE7CEWLJZP6FPDCRVIDGG5CJ3Y3IKEKO7EE2DQUSG6W5RA, seg_pos={0 0}, upload=9902]: Done uploading piece. err: ecclient: upload failed (node:1iNPwE2XSfXcaXAXcrH2jcyNGYeDiiWvzAcDQffrmjPwTVqVmK, address:45.91.116.7:4037): protocol: write tcp 159.223.143.39:52854->45.91.116.7:4037: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 6 19:53:18.344535130 (8h53m15.941306156s) [noise=true , address=45.91.116.14:4199 , piece_id=EKGDG7LYV3O5NSDL3UPWFKXINYNAOSPYYOWQPTFN326AJ2ZXAXHQ, seg_pos={0 0}, upload=9902]: Done uploading piece. err: ecclient: upload failed (node:1d4B1KMLs2annUuJ3v81FJapS9Yg4omBTj48eWoi6HJxnSEjrf, address:45.91.116.14:4199): protocol: write tcp 159.223.143.39:57596->45.91.116.14:4199: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 6 19:53:18.344687364 (8h53m15.941458387s) [noise=true , address=45.91.116.11:4133 , piece_id=RZWV5JMLOJ36AH7KSDXVTL7CMHI4INEW6YOOQVMCIIC2NGS3XYJA, seg_pos={0 0}, upload=9902]: Done uploading piece. err: ecclient: upload failed (node:12e1wX7hiER8sX4MLm5qLue4mayiN4SxXFdZSxfymRaRZxRGU6E, address:45.91.116.11:4133): protocol: write tcp 159.223.143.39:56440->45.91.116.11:4133: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 6 19:53:18.345220236 (8h53m15.941991259s) [noise=true , address=45.91.116.8:4066 , piece_id=5MZHIJ3ELVBAJ7ABRKG565TLU4A4NKB7X6WAACY2JBSLACL5LGVQ, seg_pos={0 0}, upload=9902]: Done uploading piece. err: ecclient: upload failed (node:123uv49k8LraXJfYQg6KRQthrYs1Egc4c3eZWm5JPvDatFbykm8, address:45.91.116.8:4066): protocol: write tcp 159.223.143.39:44136->45.91.116.8:4066: use of closed network connection; context deadline exceeded Mar 6 19:53:18.345277850 (8h53m15.942048865s) [seg_pos={0 0}, upload=9902]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 6 19:53:18.367560006 (8h53m15.964331035s) [upload=9902]: Done uploading segments... Mar 6 19:53:18.367606646 (8h53m15.964377681s) [upload=9902]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 1.116474088s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................................X.....................] duration: 1.314167842s trace: https://jaeger.dev.storj.io/trace/78791d41d75e16a0 FAIL (2026-03-06 20:06:05 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 19:52:18.382132233 (8h52m15.978903242s) [upload=9902]: Uploading segments... Mar 6 19:52:18.382227489 (8h52m15.978998515s) [upload=9902]: Got next segment. Inline: false Mar 6 19:52:18.382313252 (8h52m15.979084279s) [upload=9902]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 6 19:52:18.382319269 (8h52m15.979090296s) [upload=9902]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 6 19:52:18.411471452 (8h52m16.008242482s) [seg_pos={0 0}, upload=9902]: Begin upload segment... Mar 6 19:52:18.411537096 (8h52m16.008308109s) [seg_pos={0 0}, upload=9902]: Done begin upload segment. Mar 6 19:52:18.411564540 (8h52m16.008335586s) [noise=true , address=45.91.116.7:4037 , piece_id=YXDMXUBE7CEWLJZP6FPDCRVIDGG5CJ3Y3IKEKO7EE2DQUSG6W5RA, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.411626862 (8h52m16.008397897s) [noise=true , address=193.149.164.67:4018 , piece_id=GTQQP355TZZF4W7PPBYZS6ZQ6WYK7I3QIE4DZC7STWAKSBAMT4BQ, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.412356605 (8h52m16.009127631s) [noise=true , address=167.17.72.70:4771 , piece_id=TDLPTUHPNXCICPPXRTRL7WCJSXPX22ONIWWPHNXFUNIKRYPYW52Q, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.412900389 (8h52m16.009671415s) [noise=true , address=145.79.227.39:4022 , piece_id=U5GX3GLVWGWLU2BDWTHM7ST3MLLJDEAERWU3UEHO7Z2RXHGZU2DA, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.413467061 (8h52m16.010238103s) [noise=true , address=45.91.116.11:4133 , piece_id=RZWV5JMLOJ36AH7KSDXVTL7CMHI4INEW6YOOQVMCIIC2NGS3XYJA, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.413653431 (8h52m16.010424457s) [noise=true , address=193.149.164.69:4101 , piece_id=BTIBVPSORFJQR3CLA5UV62NXNI6GD3JF4D5Z74KU4R4CXB3WHHKA, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.414227965 (8h52m16.010998999s) [noise=true , address=45.91.116.14:4199 , piece_id=EKGDG7LYV3O5NSDL3UPWFKXINYNAOSPYYOWQPTFN326AJ2ZXAXHQ, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.415177146 (8h52m16.01194818s) [noise=true , address=77.90.31.16:4763 , piece_id=5Q7H3CU3MWSFPQGC67XMLCMTSUIEJQUV7QTO4UZLZY4MIVM76Z6Q, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.415227889 (8h52m16.011998912s) [noise=true , address=45.91.116.10:4096 , piece_id=EIKWFKPE5BYSZGV5T4RPQED55PG2FXU3ZKSNRVCBD363YQDHS4BQ, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.415728547 (8h52m16.012499581s) [noise=true , address=77.90.31.14:4693 , piece_id=S7NAS2ZBHAQPZRHBRGRSZPLE6COQEEN6SZVMB7X2X6IMXTN5HCBQ, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.415910147 (8h52m16.012681174s) [noise=true , address=193.149.164.70:4129 , piece_id=V7WMI7RDYDOAEQYWF2OTZL5LRPUA2S3V5WZN4BKDY6BI4YV3GRPQ, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.416592462 (8h52m16.01336349s) [noise=true , address=77.90.31.18:4831 , piece_id=4PMIUNE4YXKLIJPML3VSUN4PF32TSLPOOSDZZNAXFWH7WICBOW3Q, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.417118897 (8h52m16.01388993s) [noise=true , address=77.90.31.13:4655 , piece_id=47Q27QI7O2IYW7YQKFMJGGMWPVZWWKQTFVYPUMPUI7ORE7236XIQ, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.417185844 (8h52m16.013956873s) [noise=true , address=193.149.164.71:4304 , piece_id=DTCL54GM76GWKAE7A3X5PUNFYO7H5JIKRT5WBEHYEDF2YNFYTOAA, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.417686790 (8h52m16.014457823s) [noise=true , address=167.17.72.71:4810 , piece_id=ANLHBMENEXSGTHBHLOIHMLM6JGHKIU2X6FPQQIUIVMVUOABNK73Q, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.418236951 (8h52m16.015007982s) [noise=true , address=145.79.227.17:4994 , piece_id=TO3EOAILO4PHF6Z7WQA4RWZNNL3JO6EXXEF2RLW7IHAGWUU2TF4Q, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.419293640 (8h52m16.016064672s) [noise=true , address=64.49.13.139:4479 , piece_id=XUYHOCSBATYECYHWLVZ4VBCMT3H4O5ETSQ5HZOB42OR3OY4LWJ6Q, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.420005502 (8h52m16.016776534s) [noise=true , address=45.91.116.8:4066 , piece_id=5MZHIJ3ELVBAJ7ABRKG565TLU4A4NKB7X6WAACY2JBSLACL5LGVQ, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.483654433 (8h52m16.080425441s) [upload=9902]: Next returned nil segment Mar 6 19:52:18.483671157 (8h52m16.08044219s) [upload=9902]: Waiting for error group managing segments... Mar 6 19:52:18.488078321 (8h52m16.084849398s) [noise=true , address=77.90.31.13:4655 , piece_id=47Q27QI7O2IYW7YQKFMJGGMWPVZWWKQTFVYPUMPUI7ORE7236XIQ, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:52:18.498435619 (8h52m16.095206647s) [noise=true , address=145.79.227.17:4994 , piece_id=TO3EOAILO4PHF6Z7WQA4RWZNNL3JO6EXXEF2RLW7IHAGWUU2TF4Q, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:52:18.502979814 (8h52m16.099750841s) [noise=true , address=77.90.31.16:4763 , piece_id=5Q7H3CU3MWSFPQGC67XMLCMTSUIEJQUV7QTO4UZLZY4MIVM76Z6Q, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:52:18.506259140 (8h52m16.103030166s) [noise=true , address=77.90.31.14:4693 , piece_id=S7NAS2ZBHAQPZRHBRGRSZPLE6COQEEN6SZVMB7X2X6IMXTN5HCBQ, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:52:18.514529389 (8h52m16.111300413s) [noise=true , address=145.79.227.39:4022 , piece_id=U5GX3GLVWGWLU2BDWTHM7ST3MLLJDEAERWU3UEHO7Z2RXHGZU2DA, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:52:18.785925494 (8h52m16.382696517s) [noise=true , address=167.17.72.70:4771 , piece_id=TDLPTUHPNXCICPPXRTRL7WCJSXPX22ONIWWPHNXFUNIKRYPYW52Q, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:52:18.810774903 (8h52m16.407545926s) [noise=true , address=77.90.31.18:4831 , piece_id=4PMIUNE4YXKLIJPML3VSUN4PF32TSLPOOSDZZNAXFWH7WICBOW3Q, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:52:19.002746550 (8h52m16.5995176s) [noise=true , address=193.149.164.71:4304 , piece_id=DTCL54GM76GWKAE7A3X5PUNFYO7H5JIKRT5WBEHYEDF2YNFYTOAA, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:52:19.007614901 (8h52m16.604385924s) [noise=true , address=193.149.164.67:4018 , piece_id=GTQQP355TZZF4W7PPBYZS6ZQ6WYK7I3QIE4DZC7STWAKSBAMT4BQ, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:52:19.010443151 (8h52m16.607214176s) [noise=true , address=193.149.164.69:4101 , piece_id=BTIBVPSORFJQR3CLA5UV62NXNI6GD3JF4D5Z74KU4R4CXB3WHHKA, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:52:19.012614014 (8h52m16.609385038s) [noise=true , address=193.149.164.70:4129 , piece_id=V7WMI7RDYDOAEQYWF2OTZL5LRPUA2S3V5WZN4BKDY6BI4YV3GRPQ, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:52:19.042145380 (8h52m16.638916406s) [noise=true , address=64.49.13.139:4479 , piece_id=XUYHOCSBATYECYHWLVZ4VBCMT3H4O5ETSQ5HZOB42OR3OY4LWJ6Q, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:52:21.314990180 (8h52m18.911761213s) [noise=true , address=167.17.72.71:4810 , piece_id=ANLHBMENEXSGTHBHLOIHMLM6JGHKIU2X6FPQQIUIVMVUOABNK73Q, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:53:18.344002671 (8h53m15.9407737s) [noise=true , address=45.91.116.10:4096 , piece_id=EIKWFKPE5BYSZGV5T4RPQED55PG2FXU3ZKSNRVCBD363YQDHS4BQ, seg_pos={0 0}, upload=9902]: Done uploading piece. err: ecclient: upload failed (node:12MZX1GwZGrEi2q44dPiYrwMoum5A2caZtXTS3qiLNcoyHhon7p, address:45.91.116.10:4096): protocol: write tcp 159.223.143.39:44996->45.91.116.10:4096: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 6 19:53:18.344173472 (8h53m15.940944497s) [noise=true , address=45.91.116.7:4037 , piece_id=YXDMXUBE7CEWLJZP6FPDCRVIDGG5CJ3Y3IKEKO7EE2DQUSG6W5RA, seg_pos={0 0}, upload=9902]: Done uploading piece. err: ecclient: upload failed (node:1iNPwE2XSfXcaXAXcrH2jcyNGYeDiiWvzAcDQffrmjPwTVqVmK, address:45.91.116.7:4037): protocol: write tcp 159.223.143.39:52854->45.91.116.7:4037: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 6 19:53:18.344535130 (8h53m15.941306156s) [noise=true , address=45.91.116.14:4199 , piece_id=EKGDG7LYV3O5NSDL3UPWFKXINYNAOSPYYOWQPTFN326AJ2ZXAXHQ, seg_pos={0 0}, upload=9902]: Done uploading piece. err: ecclient: upload failed (node:1d4B1KMLs2annUuJ3v81FJapS9Yg4omBTj48eWoi6HJxnSEjrf, address:45.91.116.14:4199): protocol: write tcp 159.223.143.39:57596->45.91.116.14:4199: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 6 19:53:18.344687364 (8h53m15.941458387s) [noise=true , address=45.91.116.11:4133 , piece_id=RZWV5JMLOJ36AH7KSDXVTL7CMHI4INEW6YOOQVMCIIC2NGS3XYJA, seg_pos={0 0}, upload=9902]: Done uploading piece. err: ecclient: upload failed (node:12e1wX7hiER8sX4MLm5qLue4mayiN4SxXFdZSxfymRaRZxRGU6E, address:45.91.116.11:4133): protocol: write tcp 159.223.143.39:56440->45.91.116.11:4133: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 6 19:53:18.345220236 (8h53m15.941991259s) [noise=true , address=45.91.116.8:4066 , piece_id=5MZHIJ3ELVBAJ7ABRKG565TLU4A4NKB7X6WAACY2JBSLACL5LGVQ, seg_pos={0 0}, upload=9902]: Done uploading piece. err: ecclient: upload failed (node:123uv49k8LraXJfYQg6KRQthrYs1Egc4c3eZWm5JPvDatFbykm8, address:45.91.116.8:4066): protocol: write tcp 159.223.143.39:44136->45.91.116.8:4066: use of closed network connection; context deadline exceeded Mar 6 19:53:18.345277850 (8h53m15.942048865s) [seg_pos={0 0}, upload=9902]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 6 19:53:18.367560006 (8h53m15.964331035s) [upload=9902]: Done uploading segments... Mar 6 19:53:18.367606646 (8h53m15.964377681s) [upload=9902]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 1.314187206s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................................X....................] duration: 1.164313701s trace: https://jaeger.dev.storj.io/trace/78791d41d75e16a0 FAIL (2026-03-06 20:05:04 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 19:52:18.382132233 (8h52m15.978903242s) [upload=9902]: Uploading segments... Mar 6 19:52:18.382227489 (8h52m15.978998515s) [upload=9902]: Got next segment. Inline: false Mar 6 19:52:18.382313252 (8h52m15.979084279s) [upload=9902]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 6 19:52:18.382319269 (8h52m15.979090296s) [upload=9902]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 6 19:52:18.411471452 (8h52m16.008242482s) [seg_pos={0 0}, upload=9902]: Begin upload segment... Mar 6 19:52:18.411537096 (8h52m16.008308109s) [seg_pos={0 0}, upload=9902]: Done begin upload segment. Mar 6 19:52:18.411564540 (8h52m16.008335586s) [noise=true , address=45.91.116.7:4037 , piece_id=YXDMXUBE7CEWLJZP6FPDCRVIDGG5CJ3Y3IKEKO7EE2DQUSG6W5RA, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.411626862 (8h52m16.008397897s) [noise=true , address=193.149.164.67:4018 , piece_id=GTQQP355TZZF4W7PPBYZS6ZQ6WYK7I3QIE4DZC7STWAKSBAMT4BQ, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.412356605 (8h52m16.009127631s) [noise=true , address=167.17.72.70:4771 , piece_id=TDLPTUHPNXCICPPXRTRL7WCJSXPX22ONIWWPHNXFUNIKRYPYW52Q, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.412900389 (8h52m16.009671415s) [noise=true , address=145.79.227.39:4022 , piece_id=U5GX3GLVWGWLU2BDWTHM7ST3MLLJDEAERWU3UEHO7Z2RXHGZU2DA, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.413467061 (8h52m16.010238103s) [noise=true , address=45.91.116.11:4133 , piece_id=RZWV5JMLOJ36AH7KSDXVTL7CMHI4INEW6YOOQVMCIIC2NGS3XYJA, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.413653431 (8h52m16.010424457s) [noise=true , address=193.149.164.69:4101 , piece_id=BTIBVPSORFJQR3CLA5UV62NXNI6GD3JF4D5Z74KU4R4CXB3WHHKA, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.414227965 (8h52m16.010998999s) [noise=true , address=45.91.116.14:4199 , piece_id=EKGDG7LYV3O5NSDL3UPWFKXINYNAOSPYYOWQPTFN326AJ2ZXAXHQ, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.415177146 (8h52m16.01194818s) [noise=true , address=77.90.31.16:4763 , piece_id=5Q7H3CU3MWSFPQGC67XMLCMTSUIEJQUV7QTO4UZLZY4MIVM76Z6Q, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.415227889 (8h52m16.011998912s) [noise=true , address=45.91.116.10:4096 , piece_id=EIKWFKPE5BYSZGV5T4RPQED55PG2FXU3ZKSNRVCBD363YQDHS4BQ, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.415728547 (8h52m16.012499581s) [noise=true , address=77.90.31.14:4693 , piece_id=S7NAS2ZBHAQPZRHBRGRSZPLE6COQEEN6SZVMB7X2X6IMXTN5HCBQ, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.415910147 (8h52m16.012681174s) [noise=true , address=193.149.164.70:4129 , piece_id=V7WMI7RDYDOAEQYWF2OTZL5LRPUA2S3V5WZN4BKDY6BI4YV3GRPQ, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.416592462 (8h52m16.01336349s) [noise=true , address=77.90.31.18:4831 , piece_id=4PMIUNE4YXKLIJPML3VSUN4PF32TSLPOOSDZZNAXFWH7WICBOW3Q, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.417118897 (8h52m16.01388993s) [noise=true , address=77.90.31.13:4655 , piece_id=47Q27QI7O2IYW7YQKFMJGGMWPVZWWKQTFVYPUMPUI7ORE7236XIQ, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.417185844 (8h52m16.013956873s) [noise=true , address=193.149.164.71:4304 , piece_id=DTCL54GM76GWKAE7A3X5PUNFYO7H5JIKRT5WBEHYEDF2YNFYTOAA, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.417686790 (8h52m16.014457823s) [noise=true , address=167.17.72.71:4810 , piece_id=ANLHBMENEXSGTHBHLOIHMLM6JGHKIU2X6FPQQIUIVMVUOABNK73Q, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.418236951 (8h52m16.015007982s) [noise=true , address=145.79.227.17:4994 , piece_id=TO3EOAILO4PHF6Z7WQA4RWZNNL3JO6EXXEF2RLW7IHAGWUU2TF4Q, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.419293640 (8h52m16.016064672s) [noise=true , address=64.49.13.139:4479 , piece_id=XUYHOCSBATYECYHWLVZ4VBCMT3H4O5ETSQ5HZOB42OR3OY4LWJ6Q, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.420005502 (8h52m16.016776534s) [noise=true , address=45.91.116.8:4066 , piece_id=5MZHIJ3ELVBAJ7ABRKG565TLU4A4NKB7X6WAACY2JBSLACL5LGVQ, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.483654433 (8h52m16.080425441s) [upload=9902]: Next returned nil segment Mar 6 19:52:18.483671157 (8h52m16.08044219s) [upload=9902]: Waiting for error group managing segments... Mar 6 19:52:18.488078321 (8h52m16.084849398s) [noise=true , address=77.90.31.13:4655 , piece_id=47Q27QI7O2IYW7YQKFMJGGMWPVZWWKQTFVYPUMPUI7ORE7236XIQ, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:52:18.498435619 (8h52m16.095206647s) [noise=true , address=145.79.227.17:4994 , piece_id=TO3EOAILO4PHF6Z7WQA4RWZNNL3JO6EXXEF2RLW7IHAGWUU2TF4Q, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:52:18.502979814 (8h52m16.099750841s) [noise=true , address=77.90.31.16:4763 , piece_id=5Q7H3CU3MWSFPQGC67XMLCMTSUIEJQUV7QTO4UZLZY4MIVM76Z6Q, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:52:18.506259140 (8h52m16.103030166s) [noise=true , address=77.90.31.14:4693 , piece_id=S7NAS2ZBHAQPZRHBRGRSZPLE6COQEEN6SZVMB7X2X6IMXTN5HCBQ, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:52:18.514529389 (8h52m16.111300413s) [noise=true , address=145.79.227.39:4022 , piece_id=U5GX3GLVWGWLU2BDWTHM7ST3MLLJDEAERWU3UEHO7Z2RXHGZU2DA, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:52:18.785925494 (8h52m16.382696517s) [noise=true , address=167.17.72.70:4771 , piece_id=TDLPTUHPNXCICPPXRTRL7WCJSXPX22ONIWWPHNXFUNIKRYPYW52Q, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:52:18.810774903 (8h52m16.407545926s) [noise=true , address=77.90.31.18:4831 , piece_id=4PMIUNE4YXKLIJPML3VSUN4PF32TSLPOOSDZZNAXFWH7WICBOW3Q, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:52:19.002746550 (8h52m16.5995176s) [noise=true , address=193.149.164.71:4304 , piece_id=DTCL54GM76GWKAE7A3X5PUNFYO7H5JIKRT5WBEHYEDF2YNFYTOAA, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:52:19.007614901 (8h52m16.604385924s) [noise=true , address=193.149.164.67:4018 , piece_id=GTQQP355TZZF4W7PPBYZS6ZQ6WYK7I3QIE4DZC7STWAKSBAMT4BQ, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:52:19.010443151 (8h52m16.607214176s) [noise=true , address=193.149.164.69:4101 , piece_id=BTIBVPSORFJQR3CLA5UV62NXNI6GD3JF4D5Z74KU4R4CXB3WHHKA, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:52:19.012614014 (8h52m16.609385038s) [noise=true , address=193.149.164.70:4129 , piece_id=V7WMI7RDYDOAEQYWF2OTZL5LRPUA2S3V5WZN4BKDY6BI4YV3GRPQ, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:52:19.042145380 (8h52m16.638916406s) [noise=true , address=64.49.13.139:4479 , piece_id=XUYHOCSBATYECYHWLVZ4VBCMT3H4O5ETSQ5HZOB42OR3OY4LWJ6Q, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:52:21.314990180 (8h52m18.911761213s) [noise=true , address=167.17.72.71:4810 , piece_id=ANLHBMENEXSGTHBHLOIHMLM6JGHKIU2X6FPQQIUIVMVUOABNK73Q, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:53:18.344002671 (8h53m15.9407737s) [noise=true , address=45.91.116.10:4096 , piece_id=EIKWFKPE5BYSZGV5T4RPQED55PG2FXU3ZKSNRVCBD363YQDHS4BQ, seg_pos={0 0}, upload=9902]: Done uploading piece. err: ecclient: upload failed (node:12MZX1GwZGrEi2q44dPiYrwMoum5A2caZtXTS3qiLNcoyHhon7p, address:45.91.116.10:4096): protocol: write tcp 159.223.143.39:44996->45.91.116.10:4096: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 6 19:53:18.344173472 (8h53m15.940944497s) [noise=true , address=45.91.116.7:4037 , piece_id=YXDMXUBE7CEWLJZP6FPDCRVIDGG5CJ3Y3IKEKO7EE2DQUSG6W5RA, seg_pos={0 0}, upload=9902]: Done uploading piece. err: ecclient: upload failed (node:1iNPwE2XSfXcaXAXcrH2jcyNGYeDiiWvzAcDQffrmjPwTVqVmK, address:45.91.116.7:4037): protocol: write tcp 159.223.143.39:52854->45.91.116.7:4037: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 6 19:53:18.344535130 (8h53m15.941306156s) [noise=true , address=45.91.116.14:4199 , piece_id=EKGDG7LYV3O5NSDL3UPWFKXINYNAOSPYYOWQPTFN326AJ2ZXAXHQ, seg_pos={0 0}, upload=9902]: Done uploading piece. err: ecclient: upload failed (node:1d4B1KMLs2annUuJ3v81FJapS9Yg4omBTj48eWoi6HJxnSEjrf, address:45.91.116.14:4199): protocol: write tcp 159.223.143.39:57596->45.91.116.14:4199: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 6 19:53:18.344687364 (8h53m15.941458387s) [noise=true , address=45.91.116.11:4133 , piece_id=RZWV5JMLOJ36AH7KSDXVTL7CMHI4INEW6YOOQVMCIIC2NGS3XYJA, seg_pos={0 0}, upload=9902]: Done uploading piece. err: ecclient: upload failed (node:12e1wX7hiER8sX4MLm5qLue4mayiN4SxXFdZSxfymRaRZxRGU6E, address:45.91.116.11:4133): protocol: write tcp 159.223.143.39:56440->45.91.116.11:4133: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 6 19:53:18.345220236 (8h53m15.941991259s) [noise=true , address=45.91.116.8:4066 , piece_id=5MZHIJ3ELVBAJ7ABRKG565TLU4A4NKB7X6WAACY2JBSLACL5LGVQ, seg_pos={0 0}, upload=9902]: Done uploading piece. err: ecclient: upload failed (node:123uv49k8LraXJfYQg6KRQthrYs1Egc4c3eZWm5JPvDatFbykm8, address:45.91.116.8:4066): protocol: write tcp 159.223.143.39:44136->45.91.116.8:4066: use of closed network connection; context deadline exceeded Mar 6 19:53:18.345277850 (8h53m15.942048865s) [seg_pos={0 0}, upload=9902]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 6 19:53:18.367560006 (8h53m15.964331035s) [upload=9902]: Done uploading segments... Mar 6 19:53:18.367606646 (8h53m15.964377681s) [upload=9902]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 1.164404275s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................................X...................] duration: 998.441943ms trace: https://jaeger.dev.storj.io/trace/78791d41d75e16a0 FAIL (2026-03-06 20:04:46 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 19:52:18.382132233 (8h52m15.978903242s) [upload=9902]: Uploading segments... Mar 6 19:52:18.382227489 (8h52m15.978998515s) [upload=9902]: Got next segment. Inline: false Mar 6 19:52:18.382313252 (8h52m15.979084279s) [upload=9902]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 6 19:52:18.382319269 (8h52m15.979090296s) [upload=9902]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 6 19:52:18.411471452 (8h52m16.008242482s) [seg_pos={0 0}, upload=9902]: Begin upload segment... Mar 6 19:52:18.411537096 (8h52m16.008308109s) [seg_pos={0 0}, upload=9902]: Done begin upload segment. Mar 6 19:52:18.411564540 (8h52m16.008335586s) [noise=true , address=45.91.116.7:4037 , piece_id=YXDMXUBE7CEWLJZP6FPDCRVIDGG5CJ3Y3IKEKO7EE2DQUSG6W5RA, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.411626862 (8h52m16.008397897s) [noise=true , address=193.149.164.67:4018 , piece_id=GTQQP355TZZF4W7PPBYZS6ZQ6WYK7I3QIE4DZC7STWAKSBAMT4BQ, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.412356605 (8h52m16.009127631s) [noise=true , address=167.17.72.70:4771 , piece_id=TDLPTUHPNXCICPPXRTRL7WCJSXPX22ONIWWPHNXFUNIKRYPYW52Q, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.412900389 (8h52m16.009671415s) [noise=true , address=145.79.227.39:4022 , piece_id=U5GX3GLVWGWLU2BDWTHM7ST3MLLJDEAERWU3UEHO7Z2RXHGZU2DA, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.413467061 (8h52m16.010238103s) [noise=true , address=45.91.116.11:4133 , piece_id=RZWV5JMLOJ36AH7KSDXVTL7CMHI4INEW6YOOQVMCIIC2NGS3XYJA, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.413653431 (8h52m16.010424457s) [noise=true , address=193.149.164.69:4101 , piece_id=BTIBVPSORFJQR3CLA5UV62NXNI6GD3JF4D5Z74KU4R4CXB3WHHKA, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.414227965 (8h52m16.010998999s) [noise=true , address=45.91.116.14:4199 , piece_id=EKGDG7LYV3O5NSDL3UPWFKXINYNAOSPYYOWQPTFN326AJ2ZXAXHQ, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.415177146 (8h52m16.01194818s) [noise=true , address=77.90.31.16:4763 , piece_id=5Q7H3CU3MWSFPQGC67XMLCMTSUIEJQUV7QTO4UZLZY4MIVM76Z6Q, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.415227889 (8h52m16.011998912s) [noise=true , address=45.91.116.10:4096 , piece_id=EIKWFKPE5BYSZGV5T4RPQED55PG2FXU3ZKSNRVCBD363YQDHS4BQ, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.415728547 (8h52m16.012499581s) [noise=true , address=77.90.31.14:4693 , piece_id=S7NAS2ZBHAQPZRHBRGRSZPLE6COQEEN6SZVMB7X2X6IMXTN5HCBQ, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.415910147 (8h52m16.012681174s) [noise=true , address=193.149.164.70:4129 , piece_id=V7WMI7RDYDOAEQYWF2OTZL5LRPUA2S3V5WZN4BKDY6BI4YV3GRPQ, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.416592462 (8h52m16.01336349s) [noise=true , address=77.90.31.18:4831 , piece_id=4PMIUNE4YXKLIJPML3VSUN4PF32TSLPOOSDZZNAXFWH7WICBOW3Q, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.417118897 (8h52m16.01388993s) [noise=true , address=77.90.31.13:4655 , piece_id=47Q27QI7O2IYW7YQKFMJGGMWPVZWWKQTFVYPUMPUI7ORE7236XIQ, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.417185844 (8h52m16.013956873s) [noise=true , address=193.149.164.71:4304 , piece_id=DTCL54GM76GWKAE7A3X5PUNFYO7H5JIKRT5WBEHYEDF2YNFYTOAA, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.417686790 (8h52m16.014457823s) [noise=true , address=167.17.72.71:4810 , piece_id=ANLHBMENEXSGTHBHLOIHMLM6JGHKIU2X6FPQQIUIVMVUOABNK73Q, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.418236951 (8h52m16.015007982s) [noise=true , address=145.79.227.17:4994 , piece_id=TO3EOAILO4PHF6Z7WQA4RWZNNL3JO6EXXEF2RLW7IHAGWUU2TF4Q, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.419293640 (8h52m16.016064672s) [noise=true , address=64.49.13.139:4479 , piece_id=XUYHOCSBATYECYHWLVZ4VBCMT3H4O5ETSQ5HZOB42OR3OY4LWJ6Q, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.420005502 (8h52m16.016776534s) [noise=true , address=45.91.116.8:4066 , piece_id=5MZHIJ3ELVBAJ7ABRKG565TLU4A4NKB7X6WAACY2JBSLACL5LGVQ, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.483654433 (8h52m16.080425441s) [upload=9902]: Next returned nil segment Mar 6 19:52:18.483671157 (8h52m16.08044219s) [upload=9902]: Waiting for error group managing segments... Mar 6 19:52:18.488078321 (8h52m16.084849398s) [noise=true , address=77.90.31.13:4655 , piece_id=47Q27QI7O2IYW7YQKFMJGGMWPVZWWKQTFVYPUMPUI7ORE7236XIQ, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:52:18.498435619 (8h52m16.095206647s) [noise=true , address=145.79.227.17:4994 , piece_id=TO3EOAILO4PHF6Z7WQA4RWZNNL3JO6EXXEF2RLW7IHAGWUU2TF4Q, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:52:18.502979814 (8h52m16.099750841s) [noise=true , address=77.90.31.16:4763 , piece_id=5Q7H3CU3MWSFPQGC67XMLCMTSUIEJQUV7QTO4UZLZY4MIVM76Z6Q, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:52:18.506259140 (8h52m16.103030166s) [noise=true , address=77.90.31.14:4693 , piece_id=S7NAS2ZBHAQPZRHBRGRSZPLE6COQEEN6SZVMB7X2X6IMXTN5HCBQ, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:52:18.514529389 (8h52m16.111300413s) [noise=true , address=145.79.227.39:4022 , piece_id=U5GX3GLVWGWLU2BDWTHM7ST3MLLJDEAERWU3UEHO7Z2RXHGZU2DA, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:52:18.785925494 (8h52m16.382696517s) [noise=true , address=167.17.72.70:4771 , piece_id=TDLPTUHPNXCICPPXRTRL7WCJSXPX22ONIWWPHNXFUNIKRYPYW52Q, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:52:18.810774903 (8h52m16.407545926s) [noise=true , address=77.90.31.18:4831 , piece_id=4PMIUNE4YXKLIJPML3VSUN4PF32TSLPOOSDZZNAXFWH7WICBOW3Q, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:52:19.002746550 (8h52m16.5995176s) [noise=true , address=193.149.164.71:4304 , piece_id=DTCL54GM76GWKAE7A3X5PUNFYO7H5JIKRT5WBEHYEDF2YNFYTOAA, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:52:19.007614901 (8h52m16.604385924s) [noise=true , address=193.149.164.67:4018 , piece_id=GTQQP355TZZF4W7PPBYZS6ZQ6WYK7I3QIE4DZC7STWAKSBAMT4BQ, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:52:19.010443151 (8h52m16.607214176s) [noise=true , address=193.149.164.69:4101 , piece_id=BTIBVPSORFJQR3CLA5UV62NXNI6GD3JF4D5Z74KU4R4CXB3WHHKA, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:52:19.012614014 (8h52m16.609385038s) [noise=true , address=193.149.164.70:4129 , piece_id=V7WMI7RDYDOAEQYWF2OTZL5LRPUA2S3V5WZN4BKDY6BI4YV3GRPQ, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:52:19.042145380 (8h52m16.638916406s) [noise=true , address=64.49.13.139:4479 , piece_id=XUYHOCSBATYECYHWLVZ4VBCMT3H4O5ETSQ5HZOB42OR3OY4LWJ6Q, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:52:21.314990180 (8h52m18.911761213s) [noise=true , address=167.17.72.71:4810 , piece_id=ANLHBMENEXSGTHBHLOIHMLM6JGHKIU2X6FPQQIUIVMVUOABNK73Q, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:53:18.344002671 (8h53m15.9407737s) [noise=true , address=45.91.116.10:4096 , piece_id=EIKWFKPE5BYSZGV5T4RPQED55PG2FXU3ZKSNRVCBD363YQDHS4BQ, seg_pos={0 0}, upload=9902]: Done uploading piece. err: ecclient: upload failed (node:12MZX1GwZGrEi2q44dPiYrwMoum5A2caZtXTS3qiLNcoyHhon7p, address:45.91.116.10:4096): protocol: write tcp 159.223.143.39:44996->45.91.116.10:4096: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 6 19:53:18.344173472 (8h53m15.940944497s) [noise=true , address=45.91.116.7:4037 , piece_id=YXDMXUBE7CEWLJZP6FPDCRVIDGG5CJ3Y3IKEKO7EE2DQUSG6W5RA, seg_pos={0 0}, upload=9902]: Done uploading piece. err: ecclient: upload failed (node:1iNPwE2XSfXcaXAXcrH2jcyNGYeDiiWvzAcDQffrmjPwTVqVmK, address:45.91.116.7:4037): protocol: write tcp 159.223.143.39:52854->45.91.116.7:4037: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 6 19:53:18.344535130 (8h53m15.941306156s) [noise=true , address=45.91.116.14:4199 , piece_id=EKGDG7LYV3O5NSDL3UPWFKXINYNAOSPYYOWQPTFN326AJ2ZXAXHQ, seg_pos={0 0}, upload=9902]: Done uploading piece. err: ecclient: upload failed (node:1d4B1KMLs2annUuJ3v81FJapS9Yg4omBTj48eWoi6HJxnSEjrf, address:45.91.116.14:4199): protocol: write tcp 159.223.143.39:57596->45.91.116.14:4199: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 6 19:53:18.344687364 (8h53m15.941458387s) [noise=true , address=45.91.116.11:4133 , piece_id=RZWV5JMLOJ36AH7KSDXVTL7CMHI4INEW6YOOQVMCIIC2NGS3XYJA, seg_pos={0 0}, upload=9902]: Done uploading piece. err: ecclient: upload failed (node:12e1wX7hiER8sX4MLm5qLue4mayiN4SxXFdZSxfymRaRZxRGU6E, address:45.91.116.11:4133): protocol: write tcp 159.223.143.39:56440->45.91.116.11:4133: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 6 19:53:18.345220236 (8h53m15.941991259s) [noise=true , address=45.91.116.8:4066 , piece_id=5MZHIJ3ELVBAJ7ABRKG565TLU4A4NKB7X6WAACY2JBSLACL5LGVQ, seg_pos={0 0}, upload=9902]: Done uploading piece. err: ecclient: upload failed (node:123uv49k8LraXJfYQg6KRQthrYs1Egc4c3eZWm5JPvDatFbykm8, address:45.91.116.8:4066): protocol: write tcp 159.223.143.39:44136->45.91.116.8:4066: use of closed network connection; context deadline exceeded Mar 6 19:53:18.345277850 (8h53m15.942048865s) [seg_pos={0 0}, upload=9902]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 6 19:53:18.367560006 (8h53m15.964331035s) [upload=9902]: Done uploading segments... Mar 6 19:53:18.367606646 (8h53m15.964377681s) [upload=9902]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 998.465796ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................X..................] duration: 761.436081ms trace: https://jaeger.dev.storj.io/trace/78791d41d75e16a0 FAIL (2026-03-06 20:04:04 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 19:52:18.382132233 (8h52m15.978903242s) [upload=9902]: Uploading segments... Mar 6 19:52:18.382227489 (8h52m15.978998515s) [upload=9902]: Got next segment. Inline: false Mar 6 19:52:18.382313252 (8h52m15.979084279s) [upload=9902]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 6 19:52:18.382319269 (8h52m15.979090296s) [upload=9902]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 6 19:52:18.411471452 (8h52m16.008242482s) [seg_pos={0 0}, upload=9902]: Begin upload segment... Mar 6 19:52:18.411537096 (8h52m16.008308109s) [seg_pos={0 0}, upload=9902]: Done begin upload segment. Mar 6 19:52:18.411564540 (8h52m16.008335586s) [noise=true , address=45.91.116.7:4037 , piece_id=YXDMXUBE7CEWLJZP6FPDCRVIDGG5CJ3Y3IKEKO7EE2DQUSG6W5RA, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.411626862 (8h52m16.008397897s) [noise=true , address=193.149.164.67:4018 , piece_id=GTQQP355TZZF4W7PPBYZS6ZQ6WYK7I3QIE4DZC7STWAKSBAMT4BQ, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.412356605 (8h52m16.009127631s) [noise=true , address=167.17.72.70:4771 , piece_id=TDLPTUHPNXCICPPXRTRL7WCJSXPX22ONIWWPHNXFUNIKRYPYW52Q, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.412900389 (8h52m16.009671415s) [noise=true , address=145.79.227.39:4022 , piece_id=U5GX3GLVWGWLU2BDWTHM7ST3MLLJDEAERWU3UEHO7Z2RXHGZU2DA, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.413467061 (8h52m16.010238103s) [noise=true , address=45.91.116.11:4133 , piece_id=RZWV5JMLOJ36AH7KSDXVTL7CMHI4INEW6YOOQVMCIIC2NGS3XYJA, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.413653431 (8h52m16.010424457s) [noise=true , address=193.149.164.69:4101 , piece_id=BTIBVPSORFJQR3CLA5UV62NXNI6GD3JF4D5Z74KU4R4CXB3WHHKA, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.414227965 (8h52m16.010998999s) [noise=true , address=45.91.116.14:4199 , piece_id=EKGDG7LYV3O5NSDL3UPWFKXINYNAOSPYYOWQPTFN326AJ2ZXAXHQ, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.415177146 (8h52m16.01194818s) [noise=true , address=77.90.31.16:4763 , piece_id=5Q7H3CU3MWSFPQGC67XMLCMTSUIEJQUV7QTO4UZLZY4MIVM76Z6Q, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.415227889 (8h52m16.011998912s) [noise=true , address=45.91.116.10:4096 , piece_id=EIKWFKPE5BYSZGV5T4RPQED55PG2FXU3ZKSNRVCBD363YQDHS4BQ, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.415728547 (8h52m16.012499581s) [noise=true , address=77.90.31.14:4693 , piece_id=S7NAS2ZBHAQPZRHBRGRSZPLE6COQEEN6SZVMB7X2X6IMXTN5HCBQ, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.415910147 (8h52m16.012681174s) [noise=true , address=193.149.164.70:4129 , piece_id=V7WMI7RDYDOAEQYWF2OTZL5LRPUA2S3V5WZN4BKDY6BI4YV3GRPQ, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.416592462 (8h52m16.01336349s) [noise=true , address=77.90.31.18:4831 , piece_id=4PMIUNE4YXKLIJPML3VSUN4PF32TSLPOOSDZZNAXFWH7WICBOW3Q, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.417118897 (8h52m16.01388993s) [noise=true , address=77.90.31.13:4655 , piece_id=47Q27QI7O2IYW7YQKFMJGGMWPVZWWKQTFVYPUMPUI7ORE7236XIQ, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.417185844 (8h52m16.013956873s) [noise=true , address=193.149.164.71:4304 , piece_id=DTCL54GM76GWKAE7A3X5PUNFYO7H5JIKRT5WBEHYEDF2YNFYTOAA, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.417686790 (8h52m16.014457823s) [noise=true , address=167.17.72.71:4810 , piece_id=ANLHBMENEXSGTHBHLOIHMLM6JGHKIU2X6FPQQIUIVMVUOABNK73Q, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.418236951 (8h52m16.015007982s) [noise=true , address=145.79.227.17:4994 , piece_id=TO3EOAILO4PHF6Z7WQA4RWZNNL3JO6EXXEF2RLW7IHAGWUU2TF4Q, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.419293640 (8h52m16.016064672s) [noise=true , address=64.49.13.139:4479 , piece_id=XUYHOCSBATYECYHWLVZ4VBCMT3H4O5ETSQ5HZOB42OR3OY4LWJ6Q, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.420005502 (8h52m16.016776534s) [noise=true , address=45.91.116.8:4066 , piece_id=5MZHIJ3ELVBAJ7ABRKG565TLU4A4NKB7X6WAACY2JBSLACL5LGVQ, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.483654433 (8h52m16.080425441s) [upload=9902]: Next returned nil segment Mar 6 19:52:18.483671157 (8h52m16.08044219s) [upload=9902]: Waiting for error group managing segments... Mar 6 19:52:18.488078321 (8h52m16.084849398s) [noise=true , address=77.90.31.13:4655 , piece_id=47Q27QI7O2IYW7YQKFMJGGMWPVZWWKQTFVYPUMPUI7ORE7236XIQ, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:52:18.498435619 (8h52m16.095206647s) [noise=true , address=145.79.227.17:4994 , piece_id=TO3EOAILO4PHF6Z7WQA4RWZNNL3JO6EXXEF2RLW7IHAGWUU2TF4Q, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:52:18.502979814 (8h52m16.099750841s) [noise=true , address=77.90.31.16:4763 , piece_id=5Q7H3CU3MWSFPQGC67XMLCMTSUIEJQUV7QTO4UZLZY4MIVM76Z6Q, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:52:18.506259140 (8h52m16.103030166s) [noise=true , address=77.90.31.14:4693 , piece_id=S7NAS2ZBHAQPZRHBRGRSZPLE6COQEEN6SZVMB7X2X6IMXTN5HCBQ, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:52:18.514529389 (8h52m16.111300413s) [noise=true , address=145.79.227.39:4022 , piece_id=U5GX3GLVWGWLU2BDWTHM7ST3MLLJDEAERWU3UEHO7Z2RXHGZU2DA, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:52:18.785925494 (8h52m16.382696517s) [noise=true , address=167.17.72.70:4771 , piece_id=TDLPTUHPNXCICPPXRTRL7WCJSXPX22ONIWWPHNXFUNIKRYPYW52Q, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:52:18.810774903 (8h52m16.407545926s) [noise=true , address=77.90.31.18:4831 , piece_id=4PMIUNE4YXKLIJPML3VSUN4PF32TSLPOOSDZZNAXFWH7WICBOW3Q, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:52:19.002746550 (8h52m16.5995176s) [noise=true , address=193.149.164.71:4304 , piece_id=DTCL54GM76GWKAE7A3X5PUNFYO7H5JIKRT5WBEHYEDF2YNFYTOAA, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:52:19.007614901 (8h52m16.604385924s) [noise=true , address=193.149.164.67:4018 , piece_id=GTQQP355TZZF4W7PPBYZS6ZQ6WYK7I3QIE4DZC7STWAKSBAMT4BQ, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:52:19.010443151 (8h52m16.607214176s) [noise=true , address=193.149.164.69:4101 , piece_id=BTIBVPSORFJQR3CLA5UV62NXNI6GD3JF4D5Z74KU4R4CXB3WHHKA, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:52:19.012614014 (8h52m16.609385038s) [noise=true , address=193.149.164.70:4129 , piece_id=V7WMI7RDYDOAEQYWF2OTZL5LRPUA2S3V5WZN4BKDY6BI4YV3GRPQ, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:52:19.042145380 (8h52m16.638916406s) [noise=true , address=64.49.13.139:4479 , piece_id=XUYHOCSBATYECYHWLVZ4VBCMT3H4O5ETSQ5HZOB42OR3OY4LWJ6Q, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:52:21.314990180 (8h52m18.911761213s) [noise=true , address=167.17.72.71:4810 , piece_id=ANLHBMENEXSGTHBHLOIHMLM6JGHKIU2X6FPQQIUIVMVUOABNK73Q, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:53:18.344002671 (8h53m15.9407737s) [noise=true , address=45.91.116.10:4096 , piece_id=EIKWFKPE5BYSZGV5T4RPQED55PG2FXU3ZKSNRVCBD363YQDHS4BQ, seg_pos={0 0}, upload=9902]: Done uploading piece. err: ecclient: upload failed (node:12MZX1GwZGrEi2q44dPiYrwMoum5A2caZtXTS3qiLNcoyHhon7p, address:45.91.116.10:4096): protocol: write tcp 159.223.143.39:44996->45.91.116.10:4096: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 6 19:53:18.344173472 (8h53m15.940944497s) [noise=true , address=45.91.116.7:4037 , piece_id=YXDMXUBE7CEWLJZP6FPDCRVIDGG5CJ3Y3IKEKO7EE2DQUSG6W5RA, seg_pos={0 0}, upload=9902]: Done uploading piece. err: ecclient: upload failed (node:1iNPwE2XSfXcaXAXcrH2jcyNGYeDiiWvzAcDQffrmjPwTVqVmK, address:45.91.116.7:4037): protocol: write tcp 159.223.143.39:52854->45.91.116.7:4037: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 6 19:53:18.344535130 (8h53m15.941306156s) [noise=true , address=45.91.116.14:4199 , piece_id=EKGDG7LYV3O5NSDL3UPWFKXINYNAOSPYYOWQPTFN326AJ2ZXAXHQ, seg_pos={0 0}, upload=9902]: Done uploading piece. err: ecclient: upload failed (node:1d4B1KMLs2annUuJ3v81FJapS9Yg4omBTj48eWoi6HJxnSEjrf, address:45.91.116.14:4199): protocol: write tcp 159.223.143.39:57596->45.91.116.14:4199: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 6 19:53:18.344687364 (8h53m15.941458387s) [noise=true , address=45.91.116.11:4133 , piece_id=RZWV5JMLOJ36AH7KSDXVTL7CMHI4INEW6YOOQVMCIIC2NGS3XYJA, seg_pos={0 0}, upload=9902]: Done uploading piece. err: ecclient: upload failed (node:12e1wX7hiER8sX4MLm5qLue4mayiN4SxXFdZSxfymRaRZxRGU6E, address:45.91.116.11:4133): protocol: write tcp 159.223.143.39:56440->45.91.116.11:4133: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 6 19:53:18.345220236 (8h53m15.941991259s) [noise=true , address=45.91.116.8:4066 , piece_id=5MZHIJ3ELVBAJ7ABRKG565TLU4A4NKB7X6WAACY2JBSLACL5LGVQ, seg_pos={0 0}, upload=9902]: Done uploading piece. err: ecclient: upload failed (node:123uv49k8LraXJfYQg6KRQthrYs1Egc4c3eZWm5JPvDatFbykm8, address:45.91.116.8:4066): protocol: write tcp 159.223.143.39:44136->45.91.116.8:4066: use of closed network connection; context deadline exceeded Mar 6 19:53:18.345277850 (8h53m15.942048865s) [seg_pos={0 0}, upload=9902]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 6 19:53:18.367560006 (8h53m15.964331035s) [upload=9902]: Done uploading segments... Mar 6 19:53:18.367606646 (8h53m15.964377681s) [upload=9902]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 761.46222ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................X.................] duration: 1.796331287s trace: https://jaeger.dev.storj.io/trace/78791d41d75e16a0 FAIL (2026-03-06 20:03:33 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 19:52:18.382132233 (8h52m15.978903242s) [upload=9902]: Uploading segments... Mar 6 19:52:18.382227489 (8h52m15.978998515s) [upload=9902]: Got next segment. Inline: false Mar 6 19:52:18.382313252 (8h52m15.979084279s) [upload=9902]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 6 19:52:18.382319269 (8h52m15.979090296s) [upload=9902]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 6 19:52:18.411471452 (8h52m16.008242482s) [seg_pos={0 0}, upload=9902]: Begin upload segment... Mar 6 19:52:18.411537096 (8h52m16.008308109s) [seg_pos={0 0}, upload=9902]: Done begin upload segment. Mar 6 19:52:18.411564540 (8h52m16.008335586s) [noise=true , address=45.91.116.7:4037 , piece_id=YXDMXUBE7CEWLJZP6FPDCRVIDGG5CJ3Y3IKEKO7EE2DQUSG6W5RA, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.411626862 (8h52m16.008397897s) [noise=true , address=193.149.164.67:4018 , piece_id=GTQQP355TZZF4W7PPBYZS6ZQ6WYK7I3QIE4DZC7STWAKSBAMT4BQ, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.412356605 (8h52m16.009127631s) [noise=true , address=167.17.72.70:4771 , piece_id=TDLPTUHPNXCICPPXRTRL7WCJSXPX22ONIWWPHNXFUNIKRYPYW52Q, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.412900389 (8h52m16.009671415s) [noise=true , address=145.79.227.39:4022 , piece_id=U5GX3GLVWGWLU2BDWTHM7ST3MLLJDEAERWU3UEHO7Z2RXHGZU2DA, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.413467061 (8h52m16.010238103s) [noise=true , address=45.91.116.11:4133 , piece_id=RZWV5JMLOJ36AH7KSDXVTL7CMHI4INEW6YOOQVMCIIC2NGS3XYJA, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.413653431 (8h52m16.010424457s) [noise=true , address=193.149.164.69:4101 , piece_id=BTIBVPSORFJQR3CLA5UV62NXNI6GD3JF4D5Z74KU4R4CXB3WHHKA, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.414227965 (8h52m16.010998999s) [noise=true , address=45.91.116.14:4199 , piece_id=EKGDG7LYV3O5NSDL3UPWFKXINYNAOSPYYOWQPTFN326AJ2ZXAXHQ, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.415177146 (8h52m16.01194818s) [noise=true , address=77.90.31.16:4763 , piece_id=5Q7H3CU3MWSFPQGC67XMLCMTSUIEJQUV7QTO4UZLZY4MIVM76Z6Q, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.415227889 (8h52m16.011998912s) [noise=true , address=45.91.116.10:4096 , piece_id=EIKWFKPE5BYSZGV5T4RPQED55PG2FXU3ZKSNRVCBD363YQDHS4BQ, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.415728547 (8h52m16.012499581s) [noise=true , address=77.90.31.14:4693 , piece_id=S7NAS2ZBHAQPZRHBRGRSZPLE6COQEEN6SZVMB7X2X6IMXTN5HCBQ, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.415910147 (8h52m16.012681174s) [noise=true , address=193.149.164.70:4129 , piece_id=V7WMI7RDYDOAEQYWF2OTZL5LRPUA2S3V5WZN4BKDY6BI4YV3GRPQ, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.416592462 (8h52m16.01336349s) [noise=true , address=77.90.31.18:4831 , piece_id=4PMIUNE4YXKLIJPML3VSUN4PF32TSLPOOSDZZNAXFWH7WICBOW3Q, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.417118897 (8h52m16.01388993s) [noise=true , address=77.90.31.13:4655 , piece_id=47Q27QI7O2IYW7YQKFMJGGMWPVZWWKQTFVYPUMPUI7ORE7236XIQ, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.417185844 (8h52m16.013956873s) [noise=true , address=193.149.164.71:4304 , piece_id=DTCL54GM76GWKAE7A3X5PUNFYO7H5JIKRT5WBEHYEDF2YNFYTOAA, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.417686790 (8h52m16.014457823s) [noise=true , address=167.17.72.71:4810 , piece_id=ANLHBMENEXSGTHBHLOIHMLM6JGHKIU2X6FPQQIUIVMVUOABNK73Q, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.418236951 (8h52m16.015007982s) [noise=true , address=145.79.227.17:4994 , piece_id=TO3EOAILO4PHF6Z7WQA4RWZNNL3JO6EXXEF2RLW7IHAGWUU2TF4Q, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.419293640 (8h52m16.016064672s) [noise=true , address=64.49.13.139:4479 , piece_id=XUYHOCSBATYECYHWLVZ4VBCMT3H4O5ETSQ5HZOB42OR3OY4LWJ6Q, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.420005502 (8h52m16.016776534s) [noise=true , address=45.91.116.8:4066 , piece_id=5MZHIJ3ELVBAJ7ABRKG565TLU4A4NKB7X6WAACY2JBSLACL5LGVQ, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.483654433 (8h52m16.080425441s) [upload=9902]: Next returned nil segment Mar 6 19:52:18.483671157 (8h52m16.08044219s) [upload=9902]: Waiting for error group managing segments... Mar 6 19:52:18.488078321 (8h52m16.084849398s) [noise=true , address=77.90.31.13:4655 , piece_id=47Q27QI7O2IYW7YQKFMJGGMWPVZWWKQTFVYPUMPUI7ORE7236XIQ, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:52:18.498435619 (8h52m16.095206647s) [noise=true , address=145.79.227.17:4994 , piece_id=TO3EOAILO4PHF6Z7WQA4RWZNNL3JO6EXXEF2RLW7IHAGWUU2TF4Q, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:52:18.502979814 (8h52m16.099750841s) [noise=true , address=77.90.31.16:4763 , piece_id=5Q7H3CU3MWSFPQGC67XMLCMTSUIEJQUV7QTO4UZLZY4MIVM76Z6Q, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:52:18.506259140 (8h52m16.103030166s) [noise=true , address=77.90.31.14:4693 , piece_id=S7NAS2ZBHAQPZRHBRGRSZPLE6COQEEN6SZVMB7X2X6IMXTN5HCBQ, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:52:18.514529389 (8h52m16.111300413s) [noise=true , address=145.79.227.39:4022 , piece_id=U5GX3GLVWGWLU2BDWTHM7ST3MLLJDEAERWU3UEHO7Z2RXHGZU2DA, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:52:18.785925494 (8h52m16.382696517s) [noise=true , address=167.17.72.70:4771 , piece_id=TDLPTUHPNXCICPPXRTRL7WCJSXPX22ONIWWPHNXFUNIKRYPYW52Q, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:52:18.810774903 (8h52m16.407545926s) [noise=true , address=77.90.31.18:4831 , piece_id=4PMIUNE4YXKLIJPML3VSUN4PF32TSLPOOSDZZNAXFWH7WICBOW3Q, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:52:19.002746550 (8h52m16.5995176s) [noise=true , address=193.149.164.71:4304 , piece_id=DTCL54GM76GWKAE7A3X5PUNFYO7H5JIKRT5WBEHYEDF2YNFYTOAA, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:52:19.007614901 (8h52m16.604385924s) [noise=true , address=193.149.164.67:4018 , piece_id=GTQQP355TZZF4W7PPBYZS6ZQ6WYK7I3QIE4DZC7STWAKSBAMT4BQ, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:52:19.010443151 (8h52m16.607214176s) [noise=true , address=193.149.164.69:4101 , piece_id=BTIBVPSORFJQR3CLA5UV62NXNI6GD3JF4D5Z74KU4R4CXB3WHHKA, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:52:19.012614014 (8h52m16.609385038s) [noise=true , address=193.149.164.70:4129 , piece_id=V7WMI7RDYDOAEQYWF2OTZL5LRPUA2S3V5WZN4BKDY6BI4YV3GRPQ, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:52:19.042145380 (8h52m16.638916406s) [noise=true , address=64.49.13.139:4479 , piece_id=XUYHOCSBATYECYHWLVZ4VBCMT3H4O5ETSQ5HZOB42OR3OY4LWJ6Q, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:52:21.314990180 (8h52m18.911761213s) [noise=true , address=167.17.72.71:4810 , piece_id=ANLHBMENEXSGTHBHLOIHMLM6JGHKIU2X6FPQQIUIVMVUOABNK73Q, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:53:18.344002671 (8h53m15.9407737s) [noise=true , address=45.91.116.10:4096 , piece_id=EIKWFKPE5BYSZGV5T4RPQED55PG2FXU3ZKSNRVCBD363YQDHS4BQ, seg_pos={0 0}, upload=9902]: Done uploading piece. err: ecclient: upload failed (node:12MZX1GwZGrEi2q44dPiYrwMoum5A2caZtXTS3qiLNcoyHhon7p, address:45.91.116.10:4096): protocol: write tcp 159.223.143.39:44996->45.91.116.10:4096: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 6 19:53:18.344173472 (8h53m15.940944497s) [noise=true , address=45.91.116.7:4037 , piece_id=YXDMXUBE7CEWLJZP6FPDCRVIDGG5CJ3Y3IKEKO7EE2DQUSG6W5RA, seg_pos={0 0}, upload=9902]: Done uploading piece. err: ecclient: upload failed (node:1iNPwE2XSfXcaXAXcrH2jcyNGYeDiiWvzAcDQffrmjPwTVqVmK, address:45.91.116.7:4037): protocol: write tcp 159.223.143.39:52854->45.91.116.7:4037: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 6 19:53:18.344535130 (8h53m15.941306156s) [noise=true , address=45.91.116.14:4199 , piece_id=EKGDG7LYV3O5NSDL3UPWFKXINYNAOSPYYOWQPTFN326AJ2ZXAXHQ, seg_pos={0 0}, upload=9902]: Done uploading piece. err: ecclient: upload failed (node:1d4B1KMLs2annUuJ3v81FJapS9Yg4omBTj48eWoi6HJxnSEjrf, address:45.91.116.14:4199): protocol: write tcp 159.223.143.39:57596->45.91.116.14:4199: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 6 19:53:18.344687364 (8h53m15.941458387s) [noise=true , address=45.91.116.11:4133 , piece_id=RZWV5JMLOJ36AH7KSDXVTL7CMHI4INEW6YOOQVMCIIC2NGS3XYJA, seg_pos={0 0}, upload=9902]: Done uploading piece. err: ecclient: upload failed (node:12e1wX7hiER8sX4MLm5qLue4mayiN4SxXFdZSxfymRaRZxRGU6E, address:45.91.116.11:4133): protocol: write tcp 159.223.143.39:56440->45.91.116.11:4133: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 6 19:53:18.345220236 (8h53m15.941991259s) [noise=true , address=45.91.116.8:4066 , piece_id=5MZHIJ3ELVBAJ7ABRKG565TLU4A4NKB7X6WAACY2JBSLACL5LGVQ, seg_pos={0 0}, upload=9902]: Done uploading piece. err: ecclient: upload failed (node:123uv49k8LraXJfYQg6KRQthrYs1Egc4c3eZWm5JPvDatFbykm8, address:45.91.116.8:4066): protocol: write tcp 159.223.143.39:44136->45.91.116.8:4066: use of closed network connection; context deadline exceeded Mar 6 19:53:18.345277850 (8h53m15.942048865s) [seg_pos={0 0}, upload=9902]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 6 19:53:18.367560006 (8h53m15.964331035s) [upload=9902]: Done uploading segments... Mar 6 19:53:18.367606646 (8h53m15.964377681s) [upload=9902]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 1.796352705s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................X................] duration: 1.239838624s trace: https://jaeger.dev.storj.io/trace/78791d41d75e16a0 FAIL (2026-03-06 20:02:58 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 19:52:18.382132233 (8h52m15.978903242s) [upload=9902]: Uploading segments... Mar 6 19:52:18.382227489 (8h52m15.978998515s) [upload=9902]: Got next segment. Inline: false Mar 6 19:52:18.382313252 (8h52m15.979084279s) [upload=9902]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 6 19:52:18.382319269 (8h52m15.979090296s) [upload=9902]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 6 19:52:18.411471452 (8h52m16.008242482s) [seg_pos={0 0}, upload=9902]: Begin upload segment... Mar 6 19:52:18.411537096 (8h52m16.008308109s) [seg_pos={0 0}, upload=9902]: Done begin upload segment. Mar 6 19:52:18.411564540 (8h52m16.008335586s) [noise=true , address=45.91.116.7:4037 , piece_id=YXDMXUBE7CEWLJZP6FPDCRVIDGG5CJ3Y3IKEKO7EE2DQUSG6W5RA, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.411626862 (8h52m16.008397897s) [noise=true , address=193.149.164.67:4018 , piece_id=GTQQP355TZZF4W7PPBYZS6ZQ6WYK7I3QIE4DZC7STWAKSBAMT4BQ, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.412356605 (8h52m16.009127631s) [noise=true , address=167.17.72.70:4771 , piece_id=TDLPTUHPNXCICPPXRTRL7WCJSXPX22ONIWWPHNXFUNIKRYPYW52Q, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.412900389 (8h52m16.009671415s) [noise=true , address=145.79.227.39:4022 , piece_id=U5GX3GLVWGWLU2BDWTHM7ST3MLLJDEAERWU3UEHO7Z2RXHGZU2DA, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.413467061 (8h52m16.010238103s) [noise=true , address=45.91.116.11:4133 , piece_id=RZWV5JMLOJ36AH7KSDXVTL7CMHI4INEW6YOOQVMCIIC2NGS3XYJA, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.413653431 (8h52m16.010424457s) [noise=true , address=193.149.164.69:4101 , piece_id=BTIBVPSORFJQR3CLA5UV62NXNI6GD3JF4D5Z74KU4R4CXB3WHHKA, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.414227965 (8h52m16.010998999s) [noise=true , address=45.91.116.14:4199 , piece_id=EKGDG7LYV3O5NSDL3UPWFKXINYNAOSPYYOWQPTFN326AJ2ZXAXHQ, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.415177146 (8h52m16.01194818s) [noise=true , address=77.90.31.16:4763 , piece_id=5Q7H3CU3MWSFPQGC67XMLCMTSUIEJQUV7QTO4UZLZY4MIVM76Z6Q, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.415227889 (8h52m16.011998912s) [noise=true , address=45.91.116.10:4096 , piece_id=EIKWFKPE5BYSZGV5T4RPQED55PG2FXU3ZKSNRVCBD363YQDHS4BQ, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.415728547 (8h52m16.012499581s) [noise=true , address=77.90.31.14:4693 , piece_id=S7NAS2ZBHAQPZRHBRGRSZPLE6COQEEN6SZVMB7X2X6IMXTN5HCBQ, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.415910147 (8h52m16.012681174s) [noise=true , address=193.149.164.70:4129 , piece_id=V7WMI7RDYDOAEQYWF2OTZL5LRPUA2S3V5WZN4BKDY6BI4YV3GRPQ, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.416592462 (8h52m16.01336349s) [noise=true , address=77.90.31.18:4831 , piece_id=4PMIUNE4YXKLIJPML3VSUN4PF32TSLPOOSDZZNAXFWH7WICBOW3Q, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.417118897 (8h52m16.01388993s) [noise=true , address=77.90.31.13:4655 , piece_id=47Q27QI7O2IYW7YQKFMJGGMWPVZWWKQTFVYPUMPUI7ORE7236XIQ, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.417185844 (8h52m16.013956873s) [noise=true , address=193.149.164.71:4304 , piece_id=DTCL54GM76GWKAE7A3X5PUNFYO7H5JIKRT5WBEHYEDF2YNFYTOAA, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.417686790 (8h52m16.014457823s) [noise=true , address=167.17.72.71:4810 , piece_id=ANLHBMENEXSGTHBHLOIHMLM6JGHKIU2X6FPQQIUIVMVUOABNK73Q, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.418236951 (8h52m16.015007982s) [noise=true , address=145.79.227.17:4994 , piece_id=TO3EOAILO4PHF6Z7WQA4RWZNNL3JO6EXXEF2RLW7IHAGWUU2TF4Q, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.419293640 (8h52m16.016064672s) [noise=true , address=64.49.13.139:4479 , piece_id=XUYHOCSBATYECYHWLVZ4VBCMT3H4O5ETSQ5HZOB42OR3OY4LWJ6Q, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.420005502 (8h52m16.016776534s) [noise=true , address=45.91.116.8:4066 , piece_id=5MZHIJ3ELVBAJ7ABRKG565TLU4A4NKB7X6WAACY2JBSLACL5LGVQ, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.483654433 (8h52m16.080425441s) [upload=9902]: Next returned nil segment Mar 6 19:52:18.483671157 (8h52m16.08044219s) [upload=9902]: Waiting for error group managing segments... Mar 6 19:52:18.488078321 (8h52m16.084849398s) [noise=true , address=77.90.31.13:4655 , piece_id=47Q27QI7O2IYW7YQKFMJGGMWPVZWWKQTFVYPUMPUI7ORE7236XIQ, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:52:18.498435619 (8h52m16.095206647s) [noise=true , address=145.79.227.17:4994 , piece_id=TO3EOAILO4PHF6Z7WQA4RWZNNL3JO6EXXEF2RLW7IHAGWUU2TF4Q, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:52:18.502979814 (8h52m16.099750841s) [noise=true , address=77.90.31.16:4763 , piece_id=5Q7H3CU3MWSFPQGC67XMLCMTSUIEJQUV7QTO4UZLZY4MIVM76Z6Q, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:52:18.506259140 (8h52m16.103030166s) [noise=true , address=77.90.31.14:4693 , piece_id=S7NAS2ZBHAQPZRHBRGRSZPLE6COQEEN6SZVMB7X2X6IMXTN5HCBQ, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:52:18.514529389 (8h52m16.111300413s) [noise=true , address=145.79.227.39:4022 , piece_id=U5GX3GLVWGWLU2BDWTHM7ST3MLLJDEAERWU3UEHO7Z2RXHGZU2DA, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:52:18.785925494 (8h52m16.382696517s) [noise=true , address=167.17.72.70:4771 , piece_id=TDLPTUHPNXCICPPXRTRL7WCJSXPX22ONIWWPHNXFUNIKRYPYW52Q, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:52:18.810774903 (8h52m16.407545926s) [noise=true , address=77.90.31.18:4831 , piece_id=4PMIUNE4YXKLIJPML3VSUN4PF32TSLPOOSDZZNAXFWH7WICBOW3Q, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:52:19.002746550 (8h52m16.5995176s) [noise=true , address=193.149.164.71:4304 , piece_id=DTCL54GM76GWKAE7A3X5PUNFYO7H5JIKRT5WBEHYEDF2YNFYTOAA, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:52:19.007614901 (8h52m16.604385924s) [noise=true , address=193.149.164.67:4018 , piece_id=GTQQP355TZZF4W7PPBYZS6ZQ6WYK7I3QIE4DZC7STWAKSBAMT4BQ, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:52:19.010443151 (8h52m16.607214176s) [noise=true , address=193.149.164.69:4101 , piece_id=BTIBVPSORFJQR3CLA5UV62NXNI6GD3JF4D5Z74KU4R4CXB3WHHKA, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:52:19.012614014 (8h52m16.609385038s) [noise=true , address=193.149.164.70:4129 , piece_id=V7WMI7RDYDOAEQYWF2OTZL5LRPUA2S3V5WZN4BKDY6BI4YV3GRPQ, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:52:19.042145380 (8h52m16.638916406s) [noise=true , address=64.49.13.139:4479 , piece_id=XUYHOCSBATYECYHWLVZ4VBCMT3H4O5ETSQ5HZOB42OR3OY4LWJ6Q, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:52:21.314990180 (8h52m18.911761213s) [noise=true , address=167.17.72.71:4810 , piece_id=ANLHBMENEXSGTHBHLOIHMLM6JGHKIU2X6FPQQIUIVMVUOABNK73Q, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:53:18.344002671 (8h53m15.9407737s) [noise=true , address=45.91.116.10:4096 , piece_id=EIKWFKPE5BYSZGV5T4RPQED55PG2FXU3ZKSNRVCBD363YQDHS4BQ, seg_pos={0 0}, upload=9902]: Done uploading piece. err: ecclient: upload failed (node:12MZX1GwZGrEi2q44dPiYrwMoum5A2caZtXTS3qiLNcoyHhon7p, address:45.91.116.10:4096): protocol: write tcp 159.223.143.39:44996->45.91.116.10:4096: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 6 19:53:18.344173472 (8h53m15.940944497s) [noise=true , address=45.91.116.7:4037 , piece_id=YXDMXUBE7CEWLJZP6FPDCRVIDGG5CJ3Y3IKEKO7EE2DQUSG6W5RA, seg_pos={0 0}, upload=9902]: Done uploading piece. err: ecclient: upload failed (node:1iNPwE2XSfXcaXAXcrH2jcyNGYeDiiWvzAcDQffrmjPwTVqVmK, address:45.91.116.7:4037): protocol: write tcp 159.223.143.39:52854->45.91.116.7:4037: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 6 19:53:18.344535130 (8h53m15.941306156s) [noise=true , address=45.91.116.14:4199 , piece_id=EKGDG7LYV3O5NSDL3UPWFKXINYNAOSPYYOWQPTFN326AJ2ZXAXHQ, seg_pos={0 0}, upload=9902]: Done uploading piece. err: ecclient: upload failed (node:1d4B1KMLs2annUuJ3v81FJapS9Yg4omBTj48eWoi6HJxnSEjrf, address:45.91.116.14:4199): protocol: write tcp 159.223.143.39:57596->45.91.116.14:4199: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 6 19:53:18.344687364 (8h53m15.941458387s) [noise=true , address=45.91.116.11:4133 , piece_id=RZWV5JMLOJ36AH7KSDXVTL7CMHI4INEW6YOOQVMCIIC2NGS3XYJA, seg_pos={0 0}, upload=9902]: Done uploading piece. err: ecclient: upload failed (node:12e1wX7hiER8sX4MLm5qLue4mayiN4SxXFdZSxfymRaRZxRGU6E, address:45.91.116.11:4133): protocol: write tcp 159.223.143.39:56440->45.91.116.11:4133: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 6 19:53:18.345220236 (8h53m15.941991259s) [noise=true , address=45.91.116.8:4066 , piece_id=5MZHIJ3ELVBAJ7ABRKG565TLU4A4NKB7X6WAACY2JBSLACL5LGVQ, seg_pos={0 0}, upload=9902]: Done uploading piece. err: ecclient: upload failed (node:123uv49k8LraXJfYQg6KRQthrYs1Egc4c3eZWm5JPvDatFbykm8, address:45.91.116.8:4066): protocol: write tcp 159.223.143.39:44136->45.91.116.8:4066: use of closed network connection; context deadline exceeded Mar 6 19:53:18.345277850 (8h53m15.942048865s) [seg_pos={0 0}, upload=9902]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 6 19:53:18.367560006 (8h53m15.964331035s) [upload=9902]: Done uploading segments... Mar 6 19:53:18.367606646 (8h53m15.964377681s) [upload=9902]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 1.239864997s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [............................................X...............] duration: 1.080100937s trace: https://jaeger.dev.storj.io/trace/78791d41d75e16a0 FAIL (2026-03-06 20:01:45 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 19:52:18.382132233 (8h52m15.978903242s) [upload=9902]: Uploading segments... Mar 6 19:52:18.382227489 (8h52m15.978998515s) [upload=9902]: Got next segment. Inline: false Mar 6 19:52:18.382313252 (8h52m15.979084279s) [upload=9902]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 6 19:52:18.382319269 (8h52m15.979090296s) [upload=9902]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 6 19:52:18.411471452 (8h52m16.008242482s) [seg_pos={0 0}, upload=9902]: Begin upload segment... Mar 6 19:52:18.411537096 (8h52m16.008308109s) [seg_pos={0 0}, upload=9902]: Done begin upload segment. Mar 6 19:52:18.411564540 (8h52m16.008335586s) [noise=true , address=45.91.116.7:4037 , piece_id=YXDMXUBE7CEWLJZP6FPDCRVIDGG5CJ3Y3IKEKO7EE2DQUSG6W5RA, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.411626862 (8h52m16.008397897s) [noise=true , address=193.149.164.67:4018 , piece_id=GTQQP355TZZF4W7PPBYZS6ZQ6WYK7I3QIE4DZC7STWAKSBAMT4BQ, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.412356605 (8h52m16.009127631s) [noise=true , address=167.17.72.70:4771 , piece_id=TDLPTUHPNXCICPPXRTRL7WCJSXPX22ONIWWPHNXFUNIKRYPYW52Q, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.412900389 (8h52m16.009671415s) [noise=true , address=145.79.227.39:4022 , piece_id=U5GX3GLVWGWLU2BDWTHM7ST3MLLJDEAERWU3UEHO7Z2RXHGZU2DA, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.413467061 (8h52m16.010238103s) [noise=true , address=45.91.116.11:4133 , piece_id=RZWV5JMLOJ36AH7KSDXVTL7CMHI4INEW6YOOQVMCIIC2NGS3XYJA, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.413653431 (8h52m16.010424457s) [noise=true , address=193.149.164.69:4101 , piece_id=BTIBVPSORFJQR3CLA5UV62NXNI6GD3JF4D5Z74KU4R4CXB3WHHKA, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.414227965 (8h52m16.010998999s) [noise=true , address=45.91.116.14:4199 , piece_id=EKGDG7LYV3O5NSDL3UPWFKXINYNAOSPYYOWQPTFN326AJ2ZXAXHQ, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.415177146 (8h52m16.01194818s) [noise=true , address=77.90.31.16:4763 , piece_id=5Q7H3CU3MWSFPQGC67XMLCMTSUIEJQUV7QTO4UZLZY4MIVM76Z6Q, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.415227889 (8h52m16.011998912s) [noise=true , address=45.91.116.10:4096 , piece_id=EIKWFKPE5BYSZGV5T4RPQED55PG2FXU3ZKSNRVCBD363YQDHS4BQ, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.415728547 (8h52m16.012499581s) [noise=true , address=77.90.31.14:4693 , piece_id=S7NAS2ZBHAQPZRHBRGRSZPLE6COQEEN6SZVMB7X2X6IMXTN5HCBQ, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.415910147 (8h52m16.012681174s) [noise=true , address=193.149.164.70:4129 , piece_id=V7WMI7RDYDOAEQYWF2OTZL5LRPUA2S3V5WZN4BKDY6BI4YV3GRPQ, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.416592462 (8h52m16.01336349s) [noise=true , address=77.90.31.18:4831 , piece_id=4PMIUNE4YXKLIJPML3VSUN4PF32TSLPOOSDZZNAXFWH7WICBOW3Q, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.417118897 (8h52m16.01388993s) [noise=true , address=77.90.31.13:4655 , piece_id=47Q27QI7O2IYW7YQKFMJGGMWPVZWWKQTFVYPUMPUI7ORE7236XIQ, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.417185844 (8h52m16.013956873s) [noise=true , address=193.149.164.71:4304 , piece_id=DTCL54GM76GWKAE7A3X5PUNFYO7H5JIKRT5WBEHYEDF2YNFYTOAA, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.417686790 (8h52m16.014457823s) [noise=true , address=167.17.72.71:4810 , piece_id=ANLHBMENEXSGTHBHLOIHMLM6JGHKIU2X6FPQQIUIVMVUOABNK73Q, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.418236951 (8h52m16.015007982s) [noise=true , address=145.79.227.17:4994 , piece_id=TO3EOAILO4PHF6Z7WQA4RWZNNL3JO6EXXEF2RLW7IHAGWUU2TF4Q, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.419293640 (8h52m16.016064672s) [noise=true , address=64.49.13.139:4479 , piece_id=XUYHOCSBATYECYHWLVZ4VBCMT3H4O5ETSQ5HZOB42OR3OY4LWJ6Q, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.420005502 (8h52m16.016776534s) [noise=true , address=45.91.116.8:4066 , piece_id=5MZHIJ3ELVBAJ7ABRKG565TLU4A4NKB7X6WAACY2JBSLACL5LGVQ, seg_pos={0 0}, upload=9902]: Uploading piece... Mar 6 19:52:18.483654433 (8h52m16.080425441s) [upload=9902]: Next returned nil segment Mar 6 19:52:18.483671157 (8h52m16.08044219s) [upload=9902]: Waiting for error group managing segments... Mar 6 19:52:18.488078321 (8h52m16.084849398s) [noise=true , address=77.90.31.13:4655 , piece_id=47Q27QI7O2IYW7YQKFMJGGMWPVZWWKQTFVYPUMPUI7ORE7236XIQ, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:52:18.498435619 (8h52m16.095206647s) [noise=true , address=145.79.227.17:4994 , piece_id=TO3EOAILO4PHF6Z7WQA4RWZNNL3JO6EXXEF2RLW7IHAGWUU2TF4Q, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:52:18.502979814 (8h52m16.099750841s) [noise=true , address=77.90.31.16:4763 , piece_id=5Q7H3CU3MWSFPQGC67XMLCMTSUIEJQUV7QTO4UZLZY4MIVM76Z6Q, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:52:18.506259140 (8h52m16.103030166s) [noise=true , address=77.90.31.14:4693 , piece_id=S7NAS2ZBHAQPZRHBRGRSZPLE6COQEEN6SZVMB7X2X6IMXTN5HCBQ, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:52:18.514529389 (8h52m16.111300413s) [noise=true , address=145.79.227.39:4022 , piece_id=U5GX3GLVWGWLU2BDWTHM7ST3MLLJDEAERWU3UEHO7Z2RXHGZU2DA, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:52:18.785925494 (8h52m16.382696517s) [noise=true , address=167.17.72.70:4771 , piece_id=TDLPTUHPNXCICPPXRTRL7WCJSXPX22ONIWWPHNXFUNIKRYPYW52Q, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:52:18.810774903 (8h52m16.407545926s) [noise=true , address=77.90.31.18:4831 , piece_id=4PMIUNE4YXKLIJPML3VSUN4PF32TSLPOOSDZZNAXFWH7WICBOW3Q, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:52:19.002746550 (8h52m16.5995176s) [noise=true , address=193.149.164.71:4304 , piece_id=DTCL54GM76GWKAE7A3X5PUNFYO7H5JIKRT5WBEHYEDF2YNFYTOAA, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:52:19.007614901 (8h52m16.604385924s) [noise=true , address=193.149.164.67:4018 , piece_id=GTQQP355TZZF4W7PPBYZS6ZQ6WYK7I3QIE4DZC7STWAKSBAMT4BQ, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:52:19.010443151 (8h52m16.607214176s) [noise=true , address=193.149.164.69:4101 , piece_id=BTIBVPSORFJQR3CLA5UV62NXNI6GD3JF4D5Z74KU4R4CXB3WHHKA, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:52:19.012614014 (8h52m16.609385038s) [noise=true , address=193.149.164.70:4129 , piece_id=V7WMI7RDYDOAEQYWF2OTZL5LRPUA2S3V5WZN4BKDY6BI4YV3GRPQ, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:52:19.042145380 (8h52m16.638916406s) [noise=true , address=64.49.13.139:4479 , piece_id=XUYHOCSBATYECYHWLVZ4VBCMT3H4O5ETSQ5HZOB42OR3OY4LWJ6Q, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:52:21.314990180 (8h52m18.911761213s) [noise=true , address=167.17.72.71:4810 , piece_id=ANLHBMENEXSGTHBHLOIHMLM6JGHKIU2X6FPQQIUIVMVUOABNK73Q, seg_pos={0 0}, upload=9902]: Done uploading piece. err: <nil> Mar 6 19:53:18.344002671 (8h53m15.9407737s) [noise=true , address=45.91.116.10:4096 , piece_id=EIKWFKPE5BYSZGV5T4RPQED55PG2FXU3ZKSNRVCBD363YQDHS4BQ, seg_pos={0 0}, upload=9902]: Done uploading piece. err: ecclient: upload failed (node:12MZX1GwZGrEi2q44dPiYrwMoum5A2caZtXTS3qiLNcoyHhon7p, address:45.91.116.10:4096): protocol: write tcp 159.223.143.39:44996->45.91.116.10:4096: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 6 19:53:18.344173472 (8h53m15.940944497s) [noise=true , address=45.91.116.7:4037 , piece_id=YXDMXUBE7CEWLJZP6FPDCRVIDGG5CJ3Y3IKEKO7EE2DQUSG6W5RA, seg_pos={0 0}, upload=9902]: Done uploading piece. err: ecclient: upload failed (node:1iNPwE2XSfXcaXAXcrH2jcyNGYeDiiWvzAcDQffrmjPwTVqVmK, address:45.91.116.7:4037): protocol: write tcp 159.223.143.39:52854->45.91.116.7:4037: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 6 19:53:18.344535130 (8h53m15.941306156s) [noise=true , address=45.91.116.14:4199 , piece_id=EKGDG7LYV3O5NSDL3UPWFKXINYNAOSPYYOWQPTFN326AJ2ZXAXHQ, seg_pos={0 0}, upload=9902]: Done uploading piece. err: ecclient: upload failed (node:1d4B1KMLs2annUuJ3v81FJapS9Yg4omBTj48eWoi6HJxnSEjrf, address:45.91.116.14:4199): protocol: write tcp 159.223.143.39:57596->45.91.116.14:4199: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 6 19:53:18.344687364 (8h53m15.941458387s) [noise=true , address=45.91.116.11:4133 , piece_id=RZWV5JMLOJ36AH7KSDXVTL7CMHI4INEW6YOOQVMCIIC2NGS3XYJA, seg_pos={0 0}, upload=9902]: Done uploading piece. err: ecclient: upload failed (node:12e1wX7hiER8sX4MLm5qLue4mayiN4SxXFdZSxfymRaRZxRGU6E, address:45.91.116.11:4133): protocol: write tcp 159.223.143.39:56440->45.91.116.11:4133: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 6 19:53:18.345220236 (8h53m15.941991259s) [noise=true , address=45.91.116.8:4066 , piece_id=5MZHIJ3ELVBAJ7ABRKG565TLU4A4NKB7X6WAACY2JBSLACL5LGVQ, seg_pos={0 0}, upload=9902]: Done uploading piece. err: ecclient: upload failed (node:123uv49k8LraXJfYQg6KRQthrYs1Egc4c3eZWm5JPvDatFbykm8, address:45.91.116.8:4066): protocol: write tcp 159.223.143.39:44136->45.91.116.8:4066: use of closed network connection; context deadline exceeded Mar 6 19:53:18.345277850 (8h53m15.942048865s) [seg_pos={0 0}, upload=9902]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 6 19:53:18.367560006 (8h53m15.964331035s) [upload=9902]: Done uploading segments... Mar 6 19:53:18.367606646 (8h53m15.964377681s) [upload=9902]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 1.08012251s core uplink us-select-1 us1-placement-20

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)