Orbiter: EU/US/AP/EDGE | Code | Config | Playbook | Escalation Procedure
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
41 out of 60 is failed (threshold=6) [...............XXXXXXXXX.XXX...XXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration:
5.001749513s trace:
https://jaeger.dev.storj.io/trace/1c4d2517c500d09e
FAIL (2026-03-18 20:00:14 UTC)
operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.001870307s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
6 out of 60 is failed (threshold=3) [...............................XXX..........X...X.........X.] duration:
17.061517597s trace:
https://jaeger.dev.storj.io/trace/7c859732d77a7da7
FAIL (2026-03-18 19:59:52 UTC)
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 13)
======LOGS======
d=332621]: Uploading segments...
Mar 18 19:58:34.964515697 (296h58m32.561286706s) [upload=332621]: Got next segment. Inline: false
Mar 18 19:58:34.964555310 (296h58m32.561326334s) [upload=332621]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Mar 18 19:58:34.964560227 (296h58m32.561331251s) [upload=332621]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Mar 18 19:58:38.382666205 (296h58m35.979437219s) [seg_pos={0 0}, upload=332621]: Begin upload segment...
Mar 18 19:58:38.382729299 (296h58m35.979500329s) [seg_pos={0 0}, upload=332621]: Done begin upload segment.
Mar 18 19:58:38.382760799 (296h58m35.979531826s) [noise=true , address=145.79.227.46:4607 , piece_id=LXBA2IGR4U6DLXE4NXL2CN23X5HXP7KFN5UCJZH6IBRYAZCQ7N3Q, seg_pos={0 0}, upload=332621]: Uploading piece...
Mar 18 19:58:38.382881088 (296h58m35.979652115s) [noise=true , address=145.79.227.22:4538 , piece_id=ALPGNVFU5TWXZ7D7I5XFQBZ2CAN74VYUBNYDFSPG3HIHO7Z6M3AA, seg_pos={0 0}, upload=332621]: Uploading piece...
Mar 18 19:58:38.383461716 (296h58m35.980232746s) [noise=true , address=167.17.72.70:4770 , piece_id=5H5RNTJDXOLXRNVU4XG5FP4O5YN5PW2XQH4O2UIX5XBKNMO35WKQ, seg_pos={0 0}, upload=332621]: Uploading piece...
Mar 18 19:58:38.384266580 (296h58m35.981037606s) [noise=true , address=45.91.116.10:4092 , piece_id=Y7E2ZWR5D6M7QKRY3I7SL3ZJN32EK36ZWPP3YRW5WCGQFZCNU64Q, seg_pos={0 0}, upload=332621]: Uploading piece...
Mar 18 19:58:38.384321708 (296h58m35.981092731s) [noise=true , address=79.127.205.235:4273 , piece_id=CQEOZ6GWLY5MEVRQZHYH6AP6DXQY5WKIAHCHDBJMAI77STRNINYA, seg_pos={0 0}, upload=332621]: Uploading piece...
Mar 18 19:58:38.385197892 (296h58m35.981968918s) [noise=true , address=193.149.164.66:4989 , piece_id=EDE7OF4G36VJ7GXFMF47ZI73D6I7OWZSC5AXESS6W74ZDWTBDXWA, seg_pos={0 0}, upload=332621]: Uploading piece...
Mar 18 19:58:38.385232613 (296h58m35.982003642s) [noise=true , address=45.91.116.13:4171 , piece_id=74JRIA5AS6SMECS3FFCCVLH3AY5N4EPNWOEDZGITBPCQZEKDGEAQ, seg_pos={0 0}, upload=332621]: Uploading piece...
Mar 18 19:58:38.385334543 (296h58m35.982105567s) [noise=true , address=193.149.164.70:4120 , piece_id=AO6H5CLNNY2CQ27JO57WEKF7OTFQA7IXAZMVG6A4EWJLZT7NIRSQ, seg_pos={0 0}, upload=332621]: Uploading piece...
Mar 18 19:58:38.386120289 (296h58m35.982891315s) [noise=true , address=79.127.219.46:4908 , piece_id=L2EOXVK7S2ZSQNK47543XTNRWJP2PU57DH2XHZ2BO5OZUTDXDQFA, seg_pos={0 0}, upload=332621]: Uploading piece...
Mar 18 19:58:38.386244430 (296h58m35.983015464s) [noise=true , address=79.127.205.237:4236 , piece_id=FKQ3NUK5AUFD35L25PJFPVDLZUVNRURK5RLSPG3GGC75QYTEWP4A, seg_pos={0 0}, upload=332621]: Uploading piece...
Mar 18 19:58:38.387158310 (296h58m35.983929342s) [noise=true , address=64.49.13.140:4516 , piece_id=D2V5WAW6D46VSSNBFGNO5ETQNCOTDHQYWM62BSSVF53X2MWU3TYQ, seg_pos={0 0}, upload=332621]: Uploading piece...
Mar 18 19:58:38.387233109 (296h58m35.984004132s) [noise=true , address=193.149.164.67:4020 , piece_id=NZQGIA6BVAK3Z6SQZA7WNAOAY25VQKBDTWCRV34KNSYTGVNDSWUA, seg_pos={0 0}, upload=332621]: Uploading piece...
Mar 18 19:58:38.388077887 (296h58m35.984848921s) [noise=true , address=145.79.227.49:4343 , piece_id=NCJH6BOJHPW5OTSCNGKTZCZYC3F3LTRJKGVOKCJODWSMHUAATJQQ, seg_pos={0 0}, upload=332621]: Uploading piece...
Mar 18 19:58:38.388338947 (296h58m35.985109978s) [noise=true , address=79.127.219.41:4940 , piece_id=C2ODTVQIR73WTRBM7EDKEC54DOBK67L2RV4UXPHHTRY6KL3EQIKQ, seg_pos={0 0}, upload=332621]: Uploading piece...
Mar 18 19:58:38.388792743 (296h58m35.985563766s) [noise=true , address=145.79.227.39:4043 , piece_id=D757XSQF27CMMI5UPBRB7LLGISWOYLW7OS3UICYF3U6AI62X73PQ, seg_pos={0 0}, upload=332621]: Uploading piece...
Mar 18 19:58:38.389305068 (296h58m35.986076097s) [noise=true , address=45.91.116.9:4280 , piece_id=QECGXDF7C5YUJNURQCU76PMJM5MZ3DCEGPGAJHCZWI3H3OVGERGQ, seg_pos={0 0}, upload=332621]: Uploading piece...
Mar 18 19:58:38.389771675 (296h58m35.986542707s) [noise=true , address=145.79.227.36:4528 , piece_id=FJ5LKOZJYVNWLZ5I6OUCZMPFDOBVZXQRVPHREHPQX6J3YWAI2KFA, seg_pos={0 0}, upload=332621]: Uploading piece...
Mar 18 19:58:38.390707637 (296h58m35.98747867s) [noise=true , address=77.90.31.18:4827 , piece_id=SGKGAEAHEHU2H4YRLYQLQMTS45U6ZNSAYJ6CYJUGGKEDRMQT6BHQ, seg_pos={0 0}, upload=332621]: Uploading piece...
Mar 18 19:58:38.466570556 (296h58m36.063341572s) [upload=332621]: Next returned nil segment
Mar 18 19:58:38.466587006 (296h58m36.063358039s) [upload=332621]: Waiting for error group managing segments...
Mar 18 19:58:38.471637588 (296h58m36.068408605s) [noise=true , address=45.91.116.9:4280 , piece_id=QECGXDF7C5YUJNURQCU76PMJM5MZ3DCEGPGAJHCZWI3H3OVGERGQ, seg_pos={0 0}, upload=332621]: Done uploading piece. err: <nil>
Mar 18 19:58:38.474621081 (296h58m36.071392109s) [noise=true , address=79.127.219.41:4940 , piece_id=C2ODTVQIR73WTRBM7EDKEC54DOBK67L2RV4UXPHHTRY6KL3EQIKQ, seg_pos={0 0}, upload=332621]: Done uploading piece. err: <nil>
Mar 18 19:58:38.479487991 (296h58m36.076259007s) [noise=true , address=77.90.31.18:4827 , piece_id=SGKGAEAHEHU2H4YRLYQLQMTS45U6ZNSAYJ6CYJUGGKEDRMQT6BHQ, seg_pos={0 0}, upload=332621]: Done uploading piece. err: <nil>
Mar 18 19:58:38.479685133 (296h58m36.076456159s) [noise=true , address=45.91.116.10:4092 , piece_id=Y7E2ZWR5D6M7QKRY3I7SL3ZJN32EK36ZWPP3YRW5WCGQFZCNU64Q, seg_pos={0 0}, upload=332621]: Done uploading piece. err: <nil>
Mar 18 19:58:38.485438319 (296h58m36.082209345s) [noise=true , address=45.91.116.13:4171 , piece_id=74JRIA5AS6SMECS3FFCCVLH3AY5N4EPNWOEDZGITBPCQZEKDGEAQ, seg_pos={0 0}, upload=332621]: Done uploading piece. err: <nil>
Mar 18 19:58:38.620281221 (296h58m36.217052236s) [noise=true , address=79.127.219.46:4908 , piece_id=L2EOXVK7S2ZSQNK47543XTNRWJP2PU57DH2XHZ2BO5OZUTDXDQFA, seg_pos={0 0}, upload=332621]: Done uploading piece. err: <nil>
Mar 18 19:58:38.772264604 (296h58m36.369035618s) [noise=true , address=167.17.72.70:4770 , piece_id=5H5RNTJDXOLXRNVU4XG5FP4O5YN5PW2XQH4O2UIX5XBKNMO35WKQ, seg_pos={0 0}, upload=332621]: Done uploading piece. err: <nil>
Mar 18 19:58:38.967982420 (296h58m36.564753435s) [noise=true , address=79.127.205.235:4273 , piece_id=CQEOZ6GWLY5MEVRQZHYH6AP6DXQY5WKIAHCHDBJMAI77STRNINYA, seg_pos={0 0}, upload=332621]: Done uploading piece. err: <nil>
Mar 18 19:58:38.977759060 (296h58m36.57453009s) [noise=true , address=79.127.205.237:4236 , piece_id=FKQ3NUK5AUFD35L25PJFPVDLZUVNRURK5RLSPG3GGC75QYTEWP4A, seg_pos={0 0}, upload=332621]: Done uploading piece. err: <nil>
Mar 18 19:58:39.005320695 (296h58m36.602091708s) [noise=true , address=193.149.164.70:4120 , piece_id=AO6H5CLNNY2CQ27JO57WEKF7OTFQA7IXAZMVG6A4EWJLZT7NIRSQ, seg_pos={0 0}, upload=332621]: Done uploading piece. err: <nil>
Mar 18 19:58:39.025848813 (296h58m36.622619842s) [noise=true , address=193.149.164.67:4020 , piece_id=NZQGIA6BVAK3Z6SQZA7WNAOAY25VQKBDTWCRV34KNSYTGVNDSWUA, seg_pos={0 0}, upload=332621]: Done uploading piece. err: <nil>
Mar 18 19:58:39.065107032 (296h58m36.661878045s) [noise=true , address=64.49.13.140:4516 , piece_id=D2V5WAW6D46VSSNBFGNO5ETQNCOTDHQYWM62BSSVF53X2MWU3TYQ, seg_pos={0 0}, upload=332621]: Done uploading piece. err: <nil>
Mar 18 19:58:39.201750353 (296h58m36.798521377s) [noise=true , address=193.149.164.66:4989 , piece_id=EDE7OF4G36VJ7GXFMF47ZI73D6I7OWZSC5AXESS6W74ZDWTBDXWA, seg_pos={0 0}, upload=332621]: Done uploading piece. err: <nil>
Mar 18 19:59:34.932832250 (296h59m32.529603284s) [noise=true , address=145.79.227.46:4607 , piece_id=LXBA2IGR4U6DLXE4NXL2CN23X5HXP7KFN5UCJZH6IBRYAZCQ7N3Q, seg_pos={0 0}, upload=332621]: Done uploading piece. err: ecclient: upload failed (node:1zDoLfA5bQWM7Uf479gfFVjG3yYvxG5EZS5yZSFGTvoFoPYEAo, address:145.79.227.46:4607): protocol: rpc: dial tcp 145.79.227.46:4607: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 19:59:34.932941758 (296h59m32.529712814s) [noise=true , address=145.79.227.22:4538 , piece_id=ALPGNVFU5TWXZ7D7I5XFQBZ2CAN74VYUBNYDFSPG3HIHO7Z6M3AA, seg_pos={0 0}, upload=332621]: Done uploading piece. err: ecclient: upload failed (node:1kNFzPpkDtwy4VFyzeFQReorJPEVsmdDAd8ULT8ZKoGde57P3v, address:145.79.227.22:4538): protocol: rpc: dial tcp 145.79.227.22:4538: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 19:59:34.933267818 (296h59m32.53003887s) [noise=true , address=145.79.227.49:4343 , piece_id=NCJH6BOJHPW5OTSCNGKTZCZYC3F3LTRJKGVOKCJODWSMHUAATJQQ, seg_pos={0 0}, upload=332621]: Done uploading piece. err: ecclient: upload failed (node:16EsEMRW53sjKVag2WpSDGLhELyE2bSXyW4aAurCvDFKRNkXuf, address:145.79.227.49:4343): protocol: rpc: dial tcp 145.79.227.49:4343: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 19:59:34.933359674 (296h59m32.530130694s) [noise=true , address=145.79.227.36:4528 , piece_id=FJ5LKOZJYVNWLZ5I6OUCZMPFDOBVZXQRVPHREHPQX6J3YWAI2KFA, seg_pos={0 0}, upload=332621]: Done uploading piece. err: ecclient: upload failed (node:12ArcCrcHdpNgBMfTcPdtTXb65RgnTh8N32ihHAx2RrddxnRYgT, address:145.79.227.36:4528): protocol: rpc: dial tcp 145.79.227.36:4528: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 19:59:34.933622410 (296h59m32.530393477s) [noise=true , address=145.79.227.39:4043 , piece_id=D757XSQF27CMMI5UPBRB7LLGISWOYLW7OS3UICYF3U6AI62X73PQ, seg_pos={0 0}, upload=332621]: Done uploading piece. err: ecclient: upload failed (node:12f97TAyWcE4YrxDQCeycC8tnuetQiagu2MV4U5o8ARLq8jGKTX, address:145.79.227.39:4043): protocol: rpc: dial tcp 145.79.227.39:4043: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 19:59:34.933677971 (296h59m32.530448984s) [seg_pos={0 0}, upload=332621]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil>
Mar 18 19:59:34.955309631 (296h59m32.552080664s) [upload=332621]: Done uploading segments...
Mar 18 19:59:34.955416416 (296h59m32.552187448s) [upload=332621]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13)
Duration: 17.061536855s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)antiblip:
5 out of 60 is failed (threshold=3) [................................X..........XX..X....X.......] duration:
16.956630951s trace:
https://jaeger.dev.storj.io/trace/6b5565e31dae52f1
FAIL (2026-03-18 19:59:49 UTC)
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 13)
======LOGS======
Mar 18 19:53:19.380863725 (296h53m16.977634736s) [upload=332531]: Starting upload
Mar 18 19:53:19.380944319 (296h53m16.977715346s) [upload=332531]: Done starting upload
Mar 18 19:53:19.380976457 (296h53m16.977747482s) [upload=332531]: Uploading segments...
Mar 18 19:53:19.381333196 (296h53m16.978104222s) [upload=332531]: Got next segment. Inline: false
Mar 18 19:53:19.381379068 (296h53m16.978150099s) [upload=332531]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Mar 18 19:53:19.381386543 (296h53m16.978157579s) [upload=332531]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Mar 18 19:53:23.490845546 (296h53m21.087616603s) [seg_pos={0 0}, upload=332531]: Begin upload segment...
Mar 18 19:53:23.490917951 (296h53m21.087688988s) [seg_pos={0 0}, upload=332531]: Done begin upload segment.
Mar 18 19:53:23.490955926 (296h53m21.087726957s) [noise=true , address=45.91.116.5:4312 , piece_id=BDB6LYZZH2HXC3WWPDVMDQLCAAGRXBM5OCZU2JQVUJ7E6REVJ5SQ, seg_pos={0 0}, upload=332531]: Uploading piece...
Mar 18 19:53:23.492093139 (296h53m21.088864171s) [noise=true , address=193.149.164.67:4003 , piece_id=SOPPFFG22JZWQA5NHYGA2OFHC2KMALZOR3ATBQ6JYWJVXATWLSRQ, seg_pos={0 0}, upload=332531]: Uploading piece...
Mar 18 19:53:23.492835295 (296h53m21.089606329s) [noise=true , address=77.90.31.18:4833 , piece_id=ON3SKTN2H26XEXC43C44MAJIPY32NZLNWRU4PBSRJS6HLMXVNBMA, seg_pos={0 0}, upload=332531]: Uploading piece...
Mar 18 19:53:23.492958664 (296h53m21.089729699s) [noise=true , address=45.91.116.13:4167 , piece_id=3JWBJU4M3WSUM3D2TVZDGJKBSRXZY5ZQHV3GFSBKOJMLORY5W7IQ, seg_pos={0 0}, upload=332531]: Uploading piece...
Mar 18 19:53:23.493624498 (296h53m21.090395533s) [noise=true , address=145.79.227.13:4814 , piece_id=3KU6BHQSH4U4XYBSV4I77FCLQKR5RKWFY3QKZ26D5QRCJDQPKTKQ, seg_pos={0 0}, upload=332531]: Uploading piece...
Mar 18 19:53:23.493650442 (296h53m21.090421482s) [noise=true , address=79.127.205.237:4230 , piece_id=Y6AOKOD27V5RL6QWUY2ZAJMONWPTYU4TTU5VIAYJ4RVMNQMCRUZQ, seg_pos={0 0}, upload=332531]: Uploading piece...
Mar 18 19:53:23.494253839 (296h53m21.091024872s) [noise=true , address=45.91.116.14:4223 , piece_id=X3NSJBVWEZ3F3DUHQOSQSUCECIN7FARCFMRPJR6N7FHIIL3MS3NQ, seg_pos={0 0}, upload=332531]: Uploading piece...
Mar 18 19:53:23.494267618 (296h53m21.091038645s) [noise=true , address=145.79.227.22:4516 , piece_id=OES5UD4FMNBSPC4ZSDS45SRXLTAQT3QNGR6JBI5D5Y363JFGJ73Q, seg_pos={0 0}, upload=332531]: Uploading piece...
Mar 18 19:53:23.494879425 (296h53m21.091650453s) [noise=true , address=193.149.164.71:4287 , piece_id=ZE2Q4NHL3KUKRB36IDRMIZ7DIKDYZIBCM2L3PTLXALGL3PINBGAQ, seg_pos={0 0}, upload=332531]: Uploading piece...
Mar 18 19:53:23.494911912 (296h53m21.091682944s) [noise=true , address=193.149.164.69:4074 , piece_id=FUGRHBCQHA4YJ2EVCMKVQVITADSVLM3PWZHV2XM3QLRDGU4G2GPA, seg_pos={0 0}, upload=332531]: Uploading piece...
Mar 18 19:53:23.495497539 (296h53m21.09226857s) [noise=true , address=193.149.164.68:4066 , piece_id=FY5RBQUD2JCNXJCY2KIK744F4QZFHTW45PBDMASIIP27RWWB7SSA, seg_pos={0 0}, upload=332531]: Uploading piece...
Mar 18 19:53:23.496168841 (296h53m21.092939872s) [noise=true , address=145.79.227.41:4105 , piece_id=RTYHA5YP42CFAYVM6PHY5RTB47GRMWSZEBBPNLC532CXKPBT2VXA, seg_pos={0 0}, upload=332531]: Uploading piece...
Mar 18 19:53:23.497008065 (296h53m21.093779105s) [noise=true , address=79.127.205.235:4271 , piece_id=5XULCXAVIG3CQFTYA6Q3TFPRVICQFHQI2MSI2VIRIAGSK5UKKFXQ, seg_pos={0 0}, upload=332531]: Uploading piece...
Mar 18 19:53:23.497605357 (296h53m21.094376396s) [noise=true , address=145.79.227.37:4590 , piece_id=ZTQLQ2IZHJFZWBHMRT74V7JDZYXF5BG4656VBZDJ4J6NGKS2TMOA, seg_pos={0 0}, upload=332531]: Uploading piece...
Mar 18 19:53:23.498200357 (296h53m21.094971388s) [noise=true , address=193.149.164.66:4989 , piece_id=DVAJPRB7ABJJGUP4TYMLOGJVDLUARAMSPCG6FE2PVQXIDP2QBPGA, seg_pos={0 0}, upload=332531]: Uploading piece...
Mar 18 19:53:23.498757906 (296h53m21.095528939s) [noise=true , address=79.127.219.45:4868 , piece_id=6NDAYV5PFDBOMZKAXKO6LZUODM46MQPQOQIDKBHER3LG4CBEDHKQ, seg_pos={0 0}, upload=332531]: Uploading piece...
Mar 18 19:53:23.499215136 (296h53m21.095986159s) [noise=true , address=145.79.227.52:4453 , piece_id=YRAA65GHFFR45DX7MKR5BWBXID3YVSA4DMQDRKWKRDTJCHKHGMOA, seg_pos={0 0}, upload=332531]: Uploading piece...
Mar 18 19:53:23.499687663 (296h53m21.096458697s) [noise=true , address=77.90.31.14:4678 , piece_id=NWBLAAUF3GACR3Y6JRTHDEUJ24GMOTZIPSRGJIPTPLIVWBCSYAHA, seg_pos={0 0}, upload=332531]: Uploading piece...
Mar 18 19:53:23.600142173 (296h53m21.196913186s) [upload=332531]: Next returned nil segment
Mar 18 19:53:23.600161212 (296h53m21.196932236s) [upload=332531]: Waiting for error group managing segments...
Mar 18 19:53:23.620885875 (296h53m21.217656899s) [noise=true , address=45.91.116.5:4312 , piece_id=BDB6LYZZH2HXC3WWPDVMDQLCAAGRXBM5OCZU2JQVUJ7E6REVJ5SQ, seg_pos={0 0}, upload=332531]: Done uploading piece. err: <nil>
Mar 18 19:53:23.621154396 (296h53m21.217925412s) [noise=true , address=45.91.116.13:4167 , piece_id=3JWBJU4M3WSUM3D2TVZDGJKBSRXZY5ZQHV3GFSBKOJMLORY5W7IQ, seg_pos={0 0}, upload=332531]: Done uploading piece. err: <nil>
Mar 18 19:53:23.628883995 (296h53m21.225655021s) [noise=true , address=45.91.116.14:4223 , piece_id=X3NSJBVWEZ3F3DUHQOSQSUCECIN7FARCFMRPJR6N7FHIIL3MS3NQ, seg_pos={0 0}, upload=332531]: Done uploading piece. err: <nil>
Mar 18 19:53:23.630970813 (296h53m21.227741838s) [noise=true , address=79.127.219.45:4868 , piece_id=6NDAYV5PFDBOMZKAXKO6LZUODM46MQPQOQIDKBHER3LG4CBEDHKQ, seg_pos={0 0}, upload=332531]: Done uploading piece. err: <nil>
Mar 18 19:53:23.639825090 (296h53m21.236596104s) [noise=true , address=77.90.31.14:4678 , piece_id=NWBLAAUF3GACR3Y6JRTHDEUJ24GMOTZIPSRGJIPTPLIVWBCSYAHA, seg_pos={0 0}, upload=332531]: Done uploading piece. err: <nil>
Mar 18 19:53:23.642754677 (296h53m21.239525692s) [noise=true , address=77.90.31.18:4833 , piece_id=ON3SKTN2H26XEXC43C44MAJIPY32NZLNWRU4PBSRJS6HLMXVNBMA, seg_pos={0 0}, upload=332531]: Done uploading piece. err: <nil>
Mar 18 19:53:24.057422272 (296h53m21.654193306s) [noise=true , address=79.127.205.235:4271 , piece_id=5XULCXAVIG3CQFTYA6Q3TFPRVICQFHQI2MSI2VIRIAGSK5UKKFXQ, seg_pos={0 0}, upload=332531]: Done uploading piece. err: <nil>
Mar 18 19:53:24.074244577 (296h53m21.671015604s) [noise=true , address=193.149.164.67:4003 , piece_id=SOPPFFG22JZWQA5NHYGA2OFHC2KMALZOR3ATBQ6JYWJVXATWLSRQ, seg_pos={0 0}, upload=332531]: Done uploading piece. err: <nil>
Mar 18 19:53:24.097678734 (296h53m21.694449747s) [noise=true , address=79.127.205.237:4230 , piece_id=Y6AOKOD27V5RL6QWUY2ZAJMONWPTYU4TTU5VIAYJ4RVMNQMCRUZQ, seg_pos={0 0}, upload=332531]: Done uploading piece. err: <nil>
Mar 18 19:53:24.109685982 (296h53m21.706457009s) [noise=true , address=193.149.164.68:4066 , piece_id=FY5RBQUD2JCNXJCY2KIK744F4QZFHTW45PBDMASIIP27RWWB7SSA, seg_pos={0 0}, upload=332531]: Done uploading piece. err: <nil>
Mar 18 19:53:24.110659716 (296h53m21.707430744s) [noise=true , address=193.149.164.69:4074 , piece_id=FUGRHBCQHA4YJ2EVCMKVQVITADSVLM3PWZHV2XM3QLRDGU4G2GPA, seg_pos={0 0}, upload=332531]: Done uploading piece. err: <nil>
Mar 18 19:53:24.114541274 (296h53m21.7113123s) [noise=true , address=193.149.164.71:4287 , piece_id=ZE2Q4NHL3KUKRB36IDRMIZ7DIKDYZIBCM2L3PTLXALGL3PINBGAQ, seg_pos={0 0}, upload=332531]: Done uploading piece. err: <nil>
Mar 18 19:53:24.140894470 (296h53m21.737665488s) [noise=true , address=193.149.164.66:4989 , piece_id=DVAJPRB7ABJJGUP4TYMLOGJVDLUARAMSPCG6FE2PVQXIDP2QBPGA, seg_pos={0 0}, upload=332531]: Done uploading piece. err: <nil>
Mar 18 19:54:19.329327996 (296h54m16.926099018s) [noise=true , address=145.79.227.13:4814 , piece_id=3KU6BHQSH4U4XYBSV4I77FCLQKR5RKWFY3QKZ26D5QRCJDQPKTKQ, seg_pos={0 0}, upload=332531]: Done uploading piece. err: ecclient: upload failed (node:12ERtrzMuh8rZPsgebtyScmYvNVwz3UKJr5Nu2hLdu6sn8X3qLa, address:145.79.227.13:4814): protocol: rpc: dial tcp 145.79.227.13:4814: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 19:54:19.329567681 (296h54m16.926338706s) [noise=true , address=145.79.227.52:4453 , piece_id=YRAA65GHFFR45DX7MKR5BWBXID3YVSA4DMQDRKWKRDTJCHKHGMOA, seg_pos={0 0}, upload=332531]: Done uploading piece. err: ecclient: upload failed (node:1BmpsQ9VfhGzmkZHvSeMptcGhzGpdoN4oYvi83qZcC1Vh9anhP, address:145.79.227.52:4453): protocol: rpc: dial tcp 145.79.227.52:4453: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 19:54:19.329755822 (296h54m16.926526848s) [noise=true , address=145.79.227.41:4105 , piece_id=RTYHA5YP42CFAYVM6PHY5RTB47GRMWSZEBBPNLC532CXKPBT2VXA, seg_pos={0 0}, upload=332531]: Done uploading piece. err: ecclient: upload failed (node:1TVtMQNVV2kERp2aZwmS8E9JUiua1RCyFpZgFACRqH1Nfscw4C, address:145.79.227.41:4105): protocol: rpc: dial tcp 145.79.227.41:4105: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 19:54:19.330021146 (296h54m16.926792175s) [noise=true , address=145.79.227.22:4516 , piece_id=OES5UD4FMNBSPC4ZSDS45SRXLTAQT3QNGR6JBI5D5Y363JFGJ73Q, seg_pos={0 0}, upload=332531]: Done uploading piece. err: ecclient: upload failed (node:12Lhy3AxzYoF25AxWdvjrrV5NNCwP1cDqkEvv4tz9t9KZmF8WPK, address:145.79.227.22:4516): protocol: rpc: dial tcp 145.79.227.22:4516: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 19:54:19.330213680 (296h54m16.926984707s) [noise=true , address=145.79.227.37:4590 , piece_id=ZTQLQ2IZHJFZWBHMRT74V7JDZYXF5BG4656VBZDJ4J6NGKS2TMOA, seg_pos={0 0}, upload=332531]: Done uploading piece. err: ecclient: upload failed (node:1KnNqY1WNwMSFXFrQsWPAhivUTgSn7dKTx1mTTzYG2UxGCkasj, address:145.79.227.37:4590): protocol: rpc: dial tcp 145.79.227.37:4590: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 19:54:19.330250832 (296h54m16.927021848s) [seg_pos={0 0}, upload=332531]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil>
Mar 18 19:54:19.351979131 (296h54m16.948750155s) [upload=332531]: Done uploading segments...
Mar 18 19:54:19.352025297 (296h54m16.948796327s) [upload=332531]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13)
Duration: 16.956659168s
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
32 out of 60 is failed (threshold=3) [............................XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration:
30.025311856s trace:
https://jaeger.dev.storj.io/trace/57fa0971ede26b0e
FAIL (2026-03-18 19:59:42 UTC)
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 9)
======LOGS======
Mar 18 19:59:12.944142306 (296h59m10.540913329s) [upload=332633]: Starting upload
Mar 18 19:59:12.944240508 (296h59m10.541011532s) [upload=332633]: Done starting upload
Mar 18 19:59:12.944288725 (296h59m10.541059742s) [upload=332633]: Uploading segments...
Mar 18 19:59:12.944551886 (296h59m10.541322902s) [upload=332633]: Got next segment. Inline: false
Mar 18 19:59:12.944617978 (296h59m10.541389002s) [upload=332633]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Mar 18 19:59:12.944626253 (296h59m10.541397276s) [upload=332633]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Mar 18 19:59:16.095841171 (296h59m13.692612194s) [seg_pos={0 0}, upload=332633]: Begin upload segment...
Mar 18 19:59:16.095939621 (296h59m13.692710634s) [seg_pos={0 0}, upload=332633]: Done begin upload segment.
Mar 18 19:59:16.096152839 (296h59m13.692923873s) [noise=true , address=145.79.227.12:4770 , piece_id=XVXGRV4NOP3GOYM344EEA3AWHIRLRIBUST6GPDHJH32XIUX6JY7A, seg_pos={0 0}, upload=332633]: Uploading piece...
Mar 18 19:59:16.096187990 (296h59m13.692959004s) [noise=true , address=79.127.219.38:4822 , piece_id=3Q74DGUVMF5SH5EN3UWMQTHJ63ZSMD4BQBZYCJ6LG7LAVCREECUQ, seg_pos={0 0}, upload=332633]: Uploading piece...
Mar 18 19:59:16.097218514 (296h59m13.693989527s) [noise=true , address=145.79.227.47:4268 , piece_id=IEXOBIRQUCTG5VPEPKP5DCCN3GOERW3ZHRS2WU5TAHLFMXRBG3XQ, seg_pos={0 0}, upload=332633]: Uploading piece...
Mar 18 19:59:16.098285629 (296h59m13.695056646s) [noise=true , address=79.127.219.46:4892 , piece_id=D6VHKSNID26SHO7R7G3BQYX3DIKNSBE3FG3SYF32XCXINOU6CXPQ, seg_pos={0 0}, upload=332633]: Uploading piece...
Mar 18 19:59:16.099115265 (296h59m13.695886282s) [noise=true , address=77.90.31.10:4517 , piece_id=QRXKORPHUXX2JRV2M4UGXSUMPRUHJ7FKYPMHIO3HW72JMOEEJAXA, seg_pos={0 0}, upload=332633]: Uploading piece...
Mar 18 19:59:16.100072781 (296h59m13.696843797s) [noise=true , address=145.79.227.52:4447 , piece_id=LTY2QVUDAVYX5ZR3LTIGQUF3JGMOI25EAIGB43HVLAZGZRO3XMRA, seg_pos={0 0}, upload=332633]: Uploading piece...
Mar 18 19:59:16.100434721 (296h59m13.697205737s) [noise=true , address=45.91.116.14:4189 , piece_id=EEZVINBVRM55FKO6LCJVPWVIL4LBYAQNA5CW3VYA54LFCU5XTM4Q, seg_pos={0 0}, upload=332633]: Uploading piece...
Mar 18 19:59:16.100924433 (296h59m13.69769545s) [noise=true , address=145.79.227.36:4520 , piece_id=RED5UFSKYB23KUIS6NER43ZW3XQKXT2MBG7MMHMD5CMUD54HCALA, seg_pos={0 0}, upload=332633]: Uploading piece...
Mar 18 19:59:16.101589544 (296h59m13.69836057s) [noise=true , address=145.79.227.2:4119 , piece_id=FDJQVYEWERQWPQEOVZ6F4ULKUS4UDGXXMU435EES5TM6HUZ5PNAQ, seg_pos={0 0}, upload=332633]: Uploading piece...
Mar 18 19:59:16.101799070 (296h59m13.698570098s) [noise=true , address=77.90.31.11:4554 , piece_id=QXQWNC6KPKT3RLWN3B5PEVLMW5XV4SI6LPZUMGZWK2JXVUZHMYLQ, seg_pos={0 0}, upload=332633]: Uploading piece...
Mar 18 19:59:16.102708292 (296h59m13.699479302s) [noise=true , address=45.91.116.7:4011 , piece_id=AJAKY6NQ2UUKJAO4MB5JMHVHPCHBJYTR4MZYHXEJL6S7F77OB47Q, seg_pos={0 0}, upload=332633]: Uploading piece...
Mar 18 19:59:16.102931983 (296h59m13.699703s) [noise=true , address=77.90.31.18:4809 , piece_id=VEHSQO2JFVES25MCTRS7G37Z7C6FQ5N7F3YKAKPW6MUZ6RIGNCOA, seg_pos={0 0}, upload=332633]: Uploading piece...
Mar 18 19:59:16.103735099 (296h59m13.700506115s) [noise=true , address=45.91.116.12:4226 , piece_id=FJQHXY73TKW5DJDVGJINPQP5XEXA37JP2JQ4O2GOXB3FKJU4QFZQ, seg_pos={0 0}, upload=332633]: Uploading piece...
Mar 18 19:59:16.104913539 (296h59m13.701684556s) [noise=true , address=77.90.31.19:4845 , piece_id=FODJWUO232AORBN7Z5IB7GXCFHPDWC4F4A6CJHVXTASIJXDIOD5Q, seg_pos={0 0}, upload=332633]: Uploading piece...
Mar 18 19:59:16.105924389 (296h59m13.702695406s) [noise=true , address=145.79.227.44:4193 , piece_id=GYV4UHB7O6MIQ5NQFPLVNGWVPUOWNML6VGE4HHO3OYS6Q2YX5URA, seg_pos={0 0}, upload=332633]: Uploading piece...
Mar 18 19:59:16.106611801 (296h59m13.703382818s) [noise=true , address=145.79.227.5:4224 , piece_id=HH2LU4HQ3MRMZR64UPNMBG7CXGNFFBWY6YLAK4OVIRT4ET3BD3CQ, seg_pos={0 0}, upload=332633]: Uploading piece...
Mar 18 19:59:16.172191860 (296h59m13.768962893s) [upload=332633]: Next returned nil segment
Mar 18 19:59:16.172212743 (296h59m13.768983776s) [upload=332633]: Waiting for error group managing segments...
Mar 18 19:59:16.175682438 (296h59m13.77245346s) [noise=true , address=45.91.116.12:4226 , piece_id=FJQHXY73TKW5DJDVGJINPQP5XEXA37JP2JQ4O2GOXB3FKJU4QFZQ, seg_pos={0 0}, upload=332633]: Done uploading piece. err: <nil>
Mar 18 19:59:16.176108817 (296h59m13.772879843s) [noise=true , address=45.91.116.14:4189 , piece_id=EEZVINBVRM55FKO6LCJVPWVIL4LBYAQNA5CW3VYA54LFCU5XTM4Q, seg_pos={0 0}, upload=332633]: Done uploading piece. err: <nil>
Mar 18 19:59:16.178055413 (296h59m13.774826445s) [noise=true , address=77.90.31.11:4554 , piece_id=QXQWNC6KPKT3RLWN3B5PEVLMW5XV4SI6LPZUMGZWK2JXVUZHMYLQ, seg_pos={0 0}, upload=332633]: Done uploading piece. err: <nil>
Mar 18 19:59:16.178257555 (296h59m13.775028582s) [noise=true , address=77.90.31.18:4809 , piece_id=VEHSQO2JFVES25MCTRS7G37Z7C6FQ5N7F3YKAKPW6MUZ6RIGNCOA, seg_pos={0 0}, upload=332633]: Done uploading piece. err: <nil>
Mar 18 19:59:16.179238886 (296h59m13.77600992s) [noise=true , address=77.90.31.19:4845 , piece_id=FODJWUO232AORBN7Z5IB7GXCFHPDWC4F4A6CJHVXTASIJXDIOD5Q, seg_pos={0 0}, upload=332633]: Done uploading piece. err: <nil>
Mar 18 19:59:16.179293136 (296h59m13.776064151s) [noise=true , address=77.90.31.10:4517 , piece_id=QRXKORPHUXX2JRV2M4UGXSUMPRUHJ7FKYPMHIO3HW72JMOEEJAXA, seg_pos={0 0}, upload=332633]: Done uploading piece. err: <nil>
Mar 18 19:59:16.179959283 (296h59m13.776730313s) [noise=true , address=79.127.219.46:4892 , piece_id=D6VHKSNID26SHO7R7G3BQYX3DIKNSBE3FG3SYF32XCXINOU6CXPQ, seg_pos={0 0}, upload=332633]: Done uploading piece. err: <nil>
Mar 18 19:59:16.191720734 (296h59m13.788491758s) [noise=true , address=45.91.116.7:4011 , piece_id=AJAKY6NQ2UUKJAO4MB5JMHVHPCHBJYTR4MZYHXEJL6S7F77OB47Q, seg_pos={0 0}, upload=332633]: Done uploading piece. err: <nil>
Mar 18 19:59:16.211735979 (296h59m13.808507035s) [noise=true , address=79.127.219.38:4822 , piece_id=3Q74DGUVMF5SH5EN3UWMQTHJ63ZSMD4BQBZYCJ6LG7LAVCREECUQ, seg_pos={0 0}, upload=332633]: Done uploading piece. err: <nil>
Mar 18 19:59:42.912072381 (296h59m40.508843415s) [noise=true , address=145.79.227.12:4770 , piece_id=XVXGRV4NOP3GOYM344EEA3AWHIRLRIBUST6GPDHJH32XIUX6JY7A, seg_pos={0 0}, upload=332633]: Done uploading piece. err: ecclient: upload failed (node:1Ug6GpYToYDXiHmiJP4GoE7eJV3kqRveix2S5SLDcKQJtEbNPz, address:145.79.227.12:4770): protocol: rpc: dial tcp 145.79.227.12:4770: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 19:59:42.912096113 (296h59m40.508867145s) [noise=true , address=145.79.227.47:4268 , piece_id=IEXOBIRQUCTG5VPEPKP5DCCN3GOERW3ZHRS2WU5TAHLFMXRBG3XQ, seg_pos={0 0}, upload=332633]: Done uploading piece. err: ecclient: upload failed (node:165gcpfWaKHcMWugoiB5cTK8hpFYp1iymG8xnBdLgPuA2o1wkj, address:145.79.227.47:4268): protocol: rpc: dial tcp 145.79.227.47:4268: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 19:59:42.912463296 (296h59m40.50923433s) [noise=true , address=145.79.227.2:4119 , piece_id=FDJQVYEWERQWPQEOVZ6F4ULKUS4UDGXXMU435EES5TM6HUZ5PNAQ, seg_pos={0 0}, upload=332633]: Done uploading piece. err: ecclient: upload failed (node:121xNAnkm9HZXUszp26UmWTsuhp3igyeDVea6gSCdtrQUQDVHpW, address:145.79.227.2:4119): protocol: rpc: dial tcp 145.79.227.2:4119: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 19:59:42.912742760 (296h59m40.509513794s) [noise=true , address=145.79.227.52:4447 , piece_id=LTY2QVUDAVYX5ZR3LTIGQUF3JGMOI25EAIGB43HVLAZGZRO3XMRA, seg_pos={0 0}, upload=332633]: Done uploading piece. err: ecclient: upload failed (node:12fqT9xA7kXTG8K9XwTUxL6FXibV35AkPHDK6grKGHg17fk91fP, address:145.79.227.52:4447): protocol: rpc: dial tcp 145.79.227.52:4447: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 19:59:42.913065012 (296h59m40.509836043s) [noise=true , address=145.79.227.36:4520 , piece_id=RED5UFSKYB23KUIS6NER43ZW3XQKXT2MBG7MMHMD5CMUD54HCALA, seg_pos={0 0}, upload=332633]: Done uploading piece. err: ecclient: upload failed (node:12eNRsboM5zFiyBVPLuQfgb5QSAc5bZFizK9KJHudNd9nstVGyy, address:145.79.227.36:4520): protocol: rpc: dial tcp 145.79.227.36:4520: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 19:59:42.913327892 (296h59m40.510098924s) [noise=true , address=145.79.227.5:4224 , piece_id=HH2LU4HQ3MRMZR64UPNMBG7CXGNFFBWY6YLAK4OVIRT4ET3BD3CQ, seg_pos={0 0}, upload=332633]: Done uploading piece. err: ecclient: upload failed (node:128nLffK44ioiTpjWjAm2vHFTLouLZdw7BijqBAbsdjo99kPYNo, address:145.79.227.5:4224): protocol: rpc: dial tcp 145.79.227.5:4224: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 19:59:42.913415538 (296h59m40.510186572s) [noise=true , address=145.79.227.44:4193 , piece_id=GYV4UHB7O6MIQ5NQFPLVNGWVPUOWNML6VGE4HHO3OYS6Q2YX5URA, seg_pos={0 0}, upload=332633]: Done uploading piece. err: ecclient: upload failed (node:1EXja3EXj5TJmk2j46Bnc6TXT7N3wgxCLjzAEjbawqvtQU7BF8, address:145.79.227.44:4193): protocol: rpc: dial tcp 145.79.227.44:4193: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 19:59:42.913451959 (296h59m40.510222978s) [seg_pos={0 0}, upload=332633]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil>
Mar 18 19:59:42.935231354 (296h59m40.532002389s) [upload=332633]: Done uploading segments...
Mar 18 19:59:42.935273834 (296h59m40.532044866s) [upload=332633]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9)
Duration: 30.025331775s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
6 out of 60 is failed (threshold=3) [................................XXX..........X...X.........X] duration:
1m0.025074876s trace:
https://jaeger.dev.storj.io/trace/7c859732d77a7da7
FAIL (2026-03-18 19:59:34 UTC)
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 13)
======LOGS======
d=332621]: Uploading segments...
Mar 18 19:58:34.964515697 (296h58m32.561286706s) [upload=332621]: Got next segment. Inline: false
Mar 18 19:58:34.964555310 (296h58m32.561326334s) [upload=332621]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Mar 18 19:58:34.964560227 (296h58m32.561331251s) [upload=332621]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Mar 18 19:58:38.382666205 (296h58m35.979437219s) [seg_pos={0 0}, upload=332621]: Begin upload segment...
Mar 18 19:58:38.382729299 (296h58m35.979500329s) [seg_pos={0 0}, upload=332621]: Done begin upload segment.
Mar 18 19:58:38.382760799 (296h58m35.979531826s) [noise=true , address=145.79.227.46:4607 , piece_id=LXBA2IGR4U6DLXE4NXL2CN23X5HXP7KFN5UCJZH6IBRYAZCQ7N3Q, seg_pos={0 0}, upload=332621]: Uploading piece...
Mar 18 19:58:38.382881088 (296h58m35.979652115s) [noise=true , address=145.79.227.22:4538 , piece_id=ALPGNVFU5TWXZ7D7I5XFQBZ2CAN74VYUBNYDFSPG3HIHO7Z6M3AA, seg_pos={0 0}, upload=332621]: Uploading piece...
Mar 18 19:58:38.383461716 (296h58m35.980232746s) [noise=true , address=167.17.72.70:4770 , piece_id=5H5RNTJDXOLXRNVU4XG5FP4O5YN5PW2XQH4O2UIX5XBKNMO35WKQ, seg_pos={0 0}, upload=332621]: Uploading piece...
Mar 18 19:58:38.384266580 (296h58m35.981037606s) [noise=true , address=45.91.116.10:4092 , piece_id=Y7E2ZWR5D6M7QKRY3I7SL3ZJN32EK36ZWPP3YRW5WCGQFZCNU64Q, seg_pos={0 0}, upload=332621]: Uploading piece...
Mar 18 19:58:38.384321708 (296h58m35.981092731s) [noise=true , address=79.127.205.235:4273 , piece_id=CQEOZ6GWLY5MEVRQZHYH6AP6DXQY5WKIAHCHDBJMAI77STRNINYA, seg_pos={0 0}, upload=332621]: Uploading piece...
Mar 18 19:58:38.385197892 (296h58m35.981968918s) [noise=true , address=193.149.164.66:4989 , piece_id=EDE7OF4G36VJ7GXFMF47ZI73D6I7OWZSC5AXESS6W74ZDWTBDXWA, seg_pos={0 0}, upload=332621]: Uploading piece...
Mar 18 19:58:38.385232613 (296h58m35.982003642s) [noise=true , address=45.91.116.13:4171 , piece_id=74JRIA5AS6SMECS3FFCCVLH3AY5N4EPNWOEDZGITBPCQZEKDGEAQ, seg_pos={0 0}, upload=332621]: Uploading piece...
Mar 18 19:58:38.385334543 (296h58m35.982105567s) [noise=true , address=193.149.164.70:4120 , piece_id=AO6H5CLNNY2CQ27JO57WEKF7OTFQA7IXAZMVG6A4EWJLZT7NIRSQ, seg_pos={0 0}, upload=332621]: Uploading piece...
Mar 18 19:58:38.386120289 (296h58m35.982891315s) [noise=true , address=79.127.219.46:4908 , piece_id=L2EOXVK7S2ZSQNK47543XTNRWJP2PU57DH2XHZ2BO5OZUTDXDQFA, seg_pos={0 0}, upload=332621]: Uploading piece...
Mar 18 19:58:38.386244430 (296h58m35.983015464s) [noise=true , address=79.127.205.237:4236 , piece_id=FKQ3NUK5AUFD35L25PJFPVDLZUVNRURK5RLSPG3GGC75QYTEWP4A, seg_pos={0 0}, upload=332621]: Uploading piece...
Mar 18 19:58:38.387158310 (296h58m35.983929342s) [noise=true , address=64.49.13.140:4516 , piece_id=D2V5WAW6D46VSSNBFGNO5ETQNCOTDHQYWM62BSSVF53X2MWU3TYQ, seg_pos={0 0}, upload=332621]: Uploading piece...
Mar 18 19:58:38.387233109 (296h58m35.984004132s) [noise=true , address=193.149.164.67:4020 , piece_id=NZQGIA6BVAK3Z6SQZA7WNAOAY25VQKBDTWCRV34KNSYTGVNDSWUA, seg_pos={0 0}, upload=332621]: Uploading piece...
Mar 18 19:58:38.388077887 (296h58m35.984848921s) [noise=true , address=145.79.227.49:4343 , piece_id=NCJH6BOJHPW5OTSCNGKTZCZYC3F3LTRJKGVOKCJODWSMHUAATJQQ, seg_pos={0 0}, upload=332621]: Uploading piece...
Mar 18 19:58:38.388338947 (296h58m35.985109978s) [noise=true , address=79.127.219.41:4940 , piece_id=C2ODTVQIR73WTRBM7EDKEC54DOBK67L2RV4UXPHHTRY6KL3EQIKQ, seg_pos={0 0}, upload=332621]: Uploading piece...
Mar 18 19:58:38.388792743 (296h58m35.985563766s) [noise=true , address=145.79.227.39:4043 , piece_id=D757XSQF27CMMI5UPBRB7LLGISWOYLW7OS3UICYF3U6AI62X73PQ, seg_pos={0 0}, upload=332621]: Uploading piece...
Mar 18 19:58:38.389305068 (296h58m35.986076097s) [noise=true , address=45.91.116.9:4280 , piece_id=QECGXDF7C5YUJNURQCU76PMJM5MZ3DCEGPGAJHCZWI3H3OVGERGQ, seg_pos={0 0}, upload=332621]: Uploading piece...
Mar 18 19:58:38.389771675 (296h58m35.986542707s) [noise=true , address=145.79.227.36:4528 , piece_id=FJ5LKOZJYVNWLZ5I6OUCZMPFDOBVZXQRVPHREHPQX6J3YWAI2KFA, seg_pos={0 0}, upload=332621]: Uploading piece...
Mar 18 19:58:38.390707637 (296h58m35.98747867s) [noise=true , address=77.90.31.18:4827 , piece_id=SGKGAEAHEHU2H4YRLYQLQMTS45U6ZNSAYJ6CYJUGGKEDRMQT6BHQ, seg_pos={0 0}, upload=332621]: Uploading piece...
Mar 18 19:58:38.466570556 (296h58m36.063341572s) [upload=332621]: Next returned nil segment
Mar 18 19:58:38.466587006 (296h58m36.063358039s) [upload=332621]: Waiting for error group managing segments...
Mar 18 19:58:38.471637588 (296h58m36.068408605s) [noise=true , address=45.91.116.9:4280 , piece_id=QECGXDF7C5YUJNURQCU76PMJM5MZ3DCEGPGAJHCZWI3H3OVGERGQ, seg_pos={0 0}, upload=332621]: Done uploading piece. err: <nil>
Mar 18 19:58:38.474621081 (296h58m36.071392109s) [noise=true , address=79.127.219.41:4940 , piece_id=C2ODTVQIR73WTRBM7EDKEC54DOBK67L2RV4UXPHHTRY6KL3EQIKQ, seg_pos={0 0}, upload=332621]: Done uploading piece. err: <nil>
Mar 18 19:58:38.479487991 (296h58m36.076259007s) [noise=true , address=77.90.31.18:4827 , piece_id=SGKGAEAHEHU2H4YRLYQLQMTS45U6ZNSAYJ6CYJUGGKEDRMQT6BHQ, seg_pos={0 0}, upload=332621]: Done uploading piece. err: <nil>
Mar 18 19:58:38.479685133 (296h58m36.076456159s) [noise=true , address=45.91.116.10:4092 , piece_id=Y7E2ZWR5D6M7QKRY3I7SL3ZJN32EK36ZWPP3YRW5WCGQFZCNU64Q, seg_pos={0 0}, upload=332621]: Done uploading piece. err: <nil>
Mar 18 19:58:38.485438319 (296h58m36.082209345s) [noise=true , address=45.91.116.13:4171 , piece_id=74JRIA5AS6SMECS3FFCCVLH3AY5N4EPNWOEDZGITBPCQZEKDGEAQ, seg_pos={0 0}, upload=332621]: Done uploading piece. err: <nil>
Mar 18 19:58:38.620281221 (296h58m36.217052236s) [noise=true , address=79.127.219.46:4908 , piece_id=L2EOXVK7S2ZSQNK47543XTNRWJP2PU57DH2XHZ2BO5OZUTDXDQFA, seg_pos={0 0}, upload=332621]: Done uploading piece. err: <nil>
Mar 18 19:58:38.772264604 (296h58m36.369035618s) [noise=true , address=167.17.72.70:4770 , piece_id=5H5RNTJDXOLXRNVU4XG5FP4O5YN5PW2XQH4O2UIX5XBKNMO35WKQ, seg_pos={0 0}, upload=332621]: Done uploading piece. err: <nil>
Mar 18 19:58:38.967982420 (296h58m36.564753435s) [noise=true , address=79.127.205.235:4273 , piece_id=CQEOZ6GWLY5MEVRQZHYH6AP6DXQY5WKIAHCHDBJMAI77STRNINYA, seg_pos={0 0}, upload=332621]: Done uploading piece. err: <nil>
Mar 18 19:58:38.977759060 (296h58m36.57453009s) [noise=true , address=79.127.205.237:4236 , piece_id=FKQ3NUK5AUFD35L25PJFPVDLZUVNRURK5RLSPG3GGC75QYTEWP4A, seg_pos={0 0}, upload=332621]: Done uploading piece. err: <nil>
Mar 18 19:58:39.005320695 (296h58m36.602091708s) [noise=true , address=193.149.164.70:4120 , piece_id=AO6H5CLNNY2CQ27JO57WEKF7OTFQA7IXAZMVG6A4EWJLZT7NIRSQ, seg_pos={0 0}, upload=332621]: Done uploading piece. err: <nil>
Mar 18 19:58:39.025848813 (296h58m36.622619842s) [noise=true , address=193.149.164.67:4020 , piece_id=NZQGIA6BVAK3Z6SQZA7WNAOAY25VQKBDTWCRV34KNSYTGVNDSWUA, seg_pos={0 0}, upload=332621]: Done uploading piece. err: <nil>
Mar 18 19:58:39.065107032 (296h58m36.661878045s) [noise=true , address=64.49.13.140:4516 , piece_id=D2V5WAW6D46VSSNBFGNO5ETQNCOTDHQYWM62BSSVF53X2MWU3TYQ, seg_pos={0 0}, upload=332621]: Done uploading piece. err: <nil>
Mar 18 19:58:39.201750353 (296h58m36.798521377s) [noise=true , address=193.149.164.66:4989 , piece_id=EDE7OF4G36VJ7GXFMF47ZI73D6I7OWZSC5AXESS6W74ZDWTBDXWA, seg_pos={0 0}, upload=332621]: Done uploading piece. err: <nil>
Mar 18 19:59:34.932832250 (296h59m32.529603284s) [noise=true , address=145.79.227.46:4607 , piece_id=LXBA2IGR4U6DLXE4NXL2CN23X5HXP7KFN5UCJZH6IBRYAZCQ7N3Q, seg_pos={0 0}, upload=332621]: Done uploading piece. err: ecclient: upload failed (node:1zDoLfA5bQWM7Uf479gfFVjG3yYvxG5EZS5yZSFGTvoFoPYEAo, address:145.79.227.46:4607): protocol: rpc: dial tcp 145.79.227.46:4607: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 19:59:34.932941758 (296h59m32.529712814s) [noise=true , address=145.79.227.22:4538 , piece_id=ALPGNVFU5TWXZ7D7I5XFQBZ2CAN74VYUBNYDFSPG3HIHO7Z6M3AA, seg_pos={0 0}, upload=332621]: Done uploading piece. err: ecclient: upload failed (node:1kNFzPpkDtwy4VFyzeFQReorJPEVsmdDAd8ULT8ZKoGde57P3v, address:145.79.227.22:4538): protocol: rpc: dial tcp 145.79.227.22:4538: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 19:59:34.933267818 (296h59m32.53003887s) [noise=true , address=145.79.227.49:4343 , piece_id=NCJH6BOJHPW5OTSCNGKTZCZYC3F3LTRJKGVOKCJODWSMHUAATJQQ, seg_pos={0 0}, upload=332621]: Done uploading piece. err: ecclient: upload failed (node:16EsEMRW53sjKVag2WpSDGLhELyE2bSXyW4aAurCvDFKRNkXuf, address:145.79.227.49:4343): protocol: rpc: dial tcp 145.79.227.49:4343: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 19:59:34.933359674 (296h59m32.530130694s) [noise=true , address=145.79.227.36:4528 , piece_id=FJ5LKOZJYVNWLZ5I6OUCZMPFDOBVZXQRVPHREHPQX6J3YWAI2KFA, seg_pos={0 0}, upload=332621]: Done uploading piece. err: ecclient: upload failed (node:12ArcCrcHdpNgBMfTcPdtTXb65RgnTh8N32ihHAx2RrddxnRYgT, address:145.79.227.36:4528): protocol: rpc: dial tcp 145.79.227.36:4528: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 19:59:34.933622410 (296h59m32.530393477s) [noise=true , address=145.79.227.39:4043 , piece_id=D757XSQF27CMMI5UPBRB7LLGISWOYLW7OS3UICYF3U6AI62X73PQ, seg_pos={0 0}, upload=332621]: Done uploading piece. err: ecclient: upload failed (node:12f97TAyWcE4YrxDQCeycC8tnuetQiagu2MV4U5o8ARLq8jGKTX, address:145.79.227.39:4043): protocol: rpc: dial tcp 145.79.227.39:4043: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 19:59:34.933677971 (296h59m32.530448984s) [seg_pos={0 0}, upload=332621]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil>
Mar 18 19:59:34.955309631 (296h59m32.552080664s) [upload=332621]: Done uploading segments...
Mar 18 19:59:34.955416416 (296h59m32.552187448s) [upload=332621]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13)
Duration: 1m0.025212286s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
40 out of 60 is failed (threshold=6) [................XXXXXXXXX.XXX...XXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration:
5.001434453s trace:
https://jaeger.dev.storj.io/trace/7ca099a50e3388fe
FAIL (2026-03-18 19:59:34 UTC)
operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.001475732s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
8 out of 60 is failed (threshold=3) [.............................XXXXXX.XX......................] duration:
14.357763547s trace:
https://jaeger.dev.storj.io/trace/4b4d10a8051ad01b
FAIL (2026-03-18 19:59:28 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 14.358618009s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
31 out of 60 is failed (threshold=3) [.............................XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration:
30.025017191s trace:
https://jaeger.dev.storj.io/trace/81d7bf877db5e77
FAIL (2026-03-18 19:59:12 UTC)
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 7)
======LOGS======
19:58:42.853808005 (296h58m40.45057902s) [upload=332623]: Got next segment. Inline: false
Mar 18 19:58:42.853849621 (296h58m40.450620645s) [upload=332623]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Mar 18 19:58:42.853862849 (296h58m40.450633862s) [upload=332623]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Mar 18 19:58:46.229066847 (296h58m43.825837872s) [seg_pos={0 0}, upload=332623]: Begin upload segment...
Mar 18 19:58:46.229232743 (296h58m43.826003766s) [seg_pos={0 0}, upload=332623]: Done begin upload segment.
Mar 18 19:58:46.229281375 (296h58m43.826052401s) [noise=true , address=145.79.227.22:4512 , piece_id=YRJBQTCXG5X6G5V3FQCT2POZHCUMXNCVOMDNJJRBACVJAWPY5CAA, seg_pos={0 0}, upload=332623]: Uploading piece...
Mar 18 19:58:46.230089393 (296h58m43.826860419s) [noise=true , address=145.79.227.45:4234 , piece_id=HF335WOWCKMG45FPC4HRFSWLWT6XXZF6SQ73ZXAVJNB22ZTK33RA, seg_pos={0 0}, upload=332623]: Uploading piece...
Mar 18 19:58:46.230194080 (296h58m43.826965112s) [noise=true , address=145.79.227.24:4585 , piece_id=YHW3ZWDOKCWVKO7JXL4XHYTTI2SFDYDYFJMP26RRG27P75CLIQ6Q, seg_pos={0 0}, upload=332623]: Uploading piece...
Mar 18 19:58:46.230581374 (296h58m43.827352398s) [noise=true , address=45.91.116.10:4081 , piece_id=EF76JRIGJA4FD4AMV24UKMM5XMGCZNHLGBB73BHKCKXMF4TSP7VA, seg_pos={0 0}, upload=332623]: Uploading piece...
Mar 18 19:58:46.230805983 (296h58m43.827577005s) [noise=true , address=145.79.227.46:4603 , piece_id=YDKENB3HNHHRQH6XGTL24AAXI7GV2JW42CHG45QEJ3EL7XTTKFZA, seg_pos={0 0}, upload=332623]: Uploading piece...
Mar 18 19:58:46.230922602 (296h58m43.827693628s) [noise=true , address=45.91.116.11:4152 , piece_id=KDOS4DU3GRCBJGRN4KSNKDAWLQMWKMALPB6NIVXNFHB2E2Y2KXOA, seg_pos={0 0}, upload=332623]: Uploading piece...
Mar 18 19:58:46.231408843 (296h58m43.828179869s) [noise=true , address=77.90.31.17:4774 , piece_id=OI6Y5HIOZ2ZXIYAWKP3SGVS6EP7QBAAH2YTYNRGRXYG55KRBUIOQ, seg_pos={0 0}, upload=332623]: Uploading piece...
Mar 18 19:58:46.231544376 (296h58m43.828315407s) [noise=true , address=145.79.227.4:4191 , piece_id=C7NPTV4OCU7DSK6JWCXSP7CMBL5ABGJYK5HIRNJYIZ2FNIQYLIPA, seg_pos={0 0}, upload=332623]: Uploading piece...
Mar 18 19:58:46.231768858 (296h58m43.828539881s) [noise=true , address=45.91.116.7:4010 , piece_id=SCVINHLW6JSLU5MZL72IAYMXURYVQ2GKI7JVI2TJQETLUG2EWZLQ, seg_pos={0 0}, upload=332623]: Uploading piece...
Mar 18 19:58:46.232024483 (296h58m43.828795512s) [noise=true , address=145.79.227.33:4874 , piece_id=73BH73YP5SJWPJI45TXHDUCSH77PKMVZWIGVDQHRKFMPACOAMB7Q, seg_pos={0 0}, upload=332623]: Uploading piece...
Mar 18 19:58:46.232134808 (296h58m43.828905831s) [noise=true , address=145.79.227.20:4440 , piece_id=WZIZCLSZSK7PAXAWNT74YUVZNUQ5OSSO5ETPB7D57QZPDOLGAVDA, seg_pos={0 0}, upload=332623]: Uploading piece...
Mar 18 19:58:46.232474396 (296h58m43.829245428s) [noise=true , address=79.127.219.41:4929 , piece_id=2L4IZ2TK4B434LWPF74TMUJOE6MJFPH53DUO5MR6TBOJITNYHFVA, seg_pos={0 0}, upload=332623]: Uploading piece...
Mar 18 19:58:46.232512479 (296h58m43.829283502s) [noise=true , address=45.91.116.9:4261 , piece_id=HNSLU5HXPP57HJAWSRS6TV4FXKI7IFZWEHUE5MELPXZCKSHOMTHA, seg_pos={0 0}, upload=332623]: Uploading piece...
Mar 18 19:58:46.232975431 (296h58m43.829746454s) [noise=true , address=77.90.31.18:4809 , piece_id=A3GJ5PAIK3XRYVLI3NVPXBBGRJXBKNP2YTSEWTO7ACRZJNDKFRWQ, seg_pos={0 0}, upload=332623]: Uploading piece...
Mar 18 19:58:46.235286445 (296h58m43.832057489s) [noise=true , address=145.79.227.23:4550 , piece_id=36QDBITILXOENDGIMDXMCETUC6ZUJ7HLMSYRC7ES5NMMKRLZMIXQ, seg_pos={0 0}, upload=332623]: Uploading piece...
Mar 18 19:58:46.235804879 (296h58m43.832575902s) [noise=true , address=145.79.227.30:4762 , piece_id=FSYZBAVZTDS363COI2VXSFQHNAW7EBPIER6RRCN4SC4MUJ3QDUXA, seg_pos={0 0}, upload=332623]: Uploading piece...
Mar 18 19:58:46.298317784 (296h58m43.895088808s) [upload=332623]: Next returned nil segment
Mar 18 19:58:46.298331971 (296h58m43.895102993s) [upload=332623]: Waiting for error group managing segments...
Mar 18 19:58:46.302951696 (296h58m43.89972272s) [noise=true , address=45.91.116.11:4152 , piece_id=KDOS4DU3GRCBJGRN4KSNKDAWLQMWKMALPB6NIVXNFHB2E2Y2KXOA, seg_pos={0 0}, upload=332623]: Done uploading piece. err: <nil>
Mar 18 19:58:46.302972803 (296h58m43.899743827s) [noise=true , address=45.91.116.9:4261 , piece_id=HNSLU5HXPP57HJAWSRS6TV4FXKI7IFZWEHUE5MELPXZCKSHOMTHA, seg_pos={0 0}, upload=332623]: Done uploading piece. err: <nil>
Mar 18 19:58:46.303866368 (296h58m43.900637391s) [noise=true , address=77.90.31.18:4809 , piece_id=A3GJ5PAIK3XRYVLI3NVPXBBGRJXBKNP2YTSEWTO7ACRZJNDKFRWQ, seg_pos={0 0}, upload=332623]: Done uploading piece. err: <nil>
Mar 18 19:58:46.305626773 (296h58m43.902397796s) [noise=true , address=77.90.31.17:4774 , piece_id=OI6Y5HIOZ2ZXIYAWKP3SGVS6EP7QBAAH2YTYNRGRXYG55KRBUIOQ, seg_pos={0 0}, upload=332623]: Done uploading piece. err: <nil>
Mar 18 19:58:46.316831596 (296h58m43.91360262s) [noise=true , address=45.91.116.10:4081 , piece_id=EF76JRIGJA4FD4AMV24UKMM5XMGCZNHLGBB73BHKCKXMF4TSP7VA, seg_pos={0 0}, upload=332623]: Done uploading piece. err: <nil>
Mar 18 19:58:46.322635277 (296h58m43.919406291s) [noise=true , address=45.91.116.7:4010 , piece_id=SCVINHLW6JSLU5MZL72IAYMXURYVQ2GKI7JVI2TJQETLUG2EWZLQ, seg_pos={0 0}, upload=332623]: Done uploading piece. err: <nil>
Mar 18 19:58:46.322663286 (296h58m43.919434315s) [noise=true , address=79.127.219.41:4929 , piece_id=2L4IZ2TK4B434LWPF74TMUJOE6MJFPH53DUO5MR6TBOJITNYHFVA, seg_pos={0 0}, upload=332623]: Done uploading piece. err: <nil>
Mar 18 19:59:12.822935404 (296h59m10.419706455s) [noise=true , address=145.79.227.20:4440 , piece_id=WZIZCLSZSK7PAXAWNT74YUVZNUQ5OSSO5ETPB7D57QZPDOLGAVDA, seg_pos={0 0}, upload=332623]: Done uploading piece. err: ecclient: upload failed (node:12GfuKp6kJEy1NTHpo9i21UDm62hrAtKq7124Z6FUehNyyLQ33i, address:145.79.227.20:4440): protocol: rpc: dial tcp 145.79.227.20:4440: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 19:59:12.823588047 (296h59m10.42035908s) [noise=true , address=145.79.227.23:4550 , piece_id=36QDBITILXOENDGIMDXMCETUC6ZUJ7HLMSYRC7ES5NMMKRLZMIXQ, seg_pos={0 0}, upload=332623]: Done uploading piece. err: ecclient: upload failed (node:12kn3uD4MCE56bN4MYYsGjDJd2qebgZ6VB6mavL711M8hoU8fbE, address:145.79.227.23:4550): protocol: rpc: dial tcp 145.79.227.23:4550: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 19:59:12.823982710 (296h59m10.420753733s) [noise=true , address=145.79.227.24:4585 , piece_id=YHW3ZWDOKCWVKO7JXL4XHYTTI2SFDYDYFJMP26RRG27P75CLIQ6Q, seg_pos={0 0}, upload=332623]: Done uploading piece. err: ecclient: upload failed (node:1TyK6onzVds1kucaD6Y4aPssPDM5FzzQ91dTrez5B7ttfwvXm, address:145.79.227.24:4585): protocol: rpc: dial tcp 145.79.227.24:4585: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 19:59:12.824226833 (296h59m10.420997864s) [noise=true , address=145.79.227.33:4874 , piece_id=73BH73YP5SJWPJI45TXHDUCSH77PKMVZWIGVDQHRKFMPACOAMB7Q, seg_pos={0 0}, upload=332623]: Done uploading piece. err: ecclient: upload failed (node:12YApHMARaC7Wxeqopt72gpnyJS1xYBwhV7wbU9NJdR6W7hukyq, address:145.79.227.33:4874): protocol: rpc: dial tcp 145.79.227.33:4874: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 19:59:12.824240012 (296h59m10.421011041s) [noise=true , address=145.79.227.4:4191 , piece_id=C7NPTV4OCU7DSK6JWCXSP7CMBL5ABGJYK5HIRNJYIZ2FNIQYLIPA, seg_pos={0 0}, upload=332623]: Done uploading piece. err: ecclient: upload failed (node:1sVWXyKPnLTobAhk6GXZTTP67FbyY3XJ1e2svTiUEa3rrghd2q, address:145.79.227.4:4191): protocol: rpc: dial tcp 145.79.227.4:4191: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 19:59:12.824535581 (296h59m10.421306594s) [noise=true , address=145.79.227.30:4762 , piece_id=FSYZBAVZTDS363COI2VXSFQHNAW7EBPIER6RRCN4SC4MUJ3QDUXA, seg_pos={0 0}, upload=332623]: Done uploading piece. err: ecclient: upload failed (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b, address:145.79.227.30:4762): protocol: rpc: dial tcp 145.79.227.30:4762: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 19:59:12.824732220 (296h59m10.421503255s) [noise=true , address=145.79.227.22:4512 , piece_id=YRJBQTCXG5X6G5V3FQCT2POZHCUMXNCVOMDNJJRBACVJAWPY5CAA, seg_pos={0 0}, upload=332623]: Done uploading piece. err: ecclient: upload failed (node:12cnVqYYfo8b6RvRtMykzh8mU7UAqxcqYvWe7H8G7k8DqtNLAR4, address:145.79.227.22:4512): protocol: rpc: dial tcp 145.79.227.22:4512: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 19:59:12.825191902 (296h59m10.421962935s) [noise=true , address=145.79.227.45:4234 , piece_id=HF335WOWCKMG45FPC4HRFSWLWT6XXZF6SQ73ZXAVJNB22ZTK33RA, seg_pos={0 0}, upload=332623]: Done uploading piece. err: ecclient: upload failed (node:12uthP31WaBmn37h54Q6z9ZMtYRY6K8917Ms4kwyyADqu4EXPJy, address:145.79.227.45:4234): protocol: rpc: dial tcp 145.79.227.45:4234: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 19:59:12.825411179 (296h59m10.422182211s) [noise=true , address=145.79.227.46:4603 , piece_id=YDKENB3HNHHRQH6XGTL24AAXI7GV2JW42CHG45QEJ3EL7XTTKFZA, seg_pos={0 0}, upload=332623]: Done uploading piece. err: ecclient: upload failed (node:1ssbeqxAWdq216V5ukNXZ3zvRroDf1x2hSs8cLtepUpefugtkX, address:145.79.227.46:4603): protocol: rpc: dial tcp 145.79.227.46:4603: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 19:59:12.825458453 (296h59m10.422229462s) [seg_pos={0 0}, upload=332623]: Done waiting for segment. successful: 7 optimal: 13 errs: <nil>
Mar 18 19:59:12.845276225 (296h59m10.44204726s) [upload=332623]: Done uploading segments...
Mar 18 19:59:12.845355413 (296h59m10.442126425s) [upload=332623]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 7)
Duration: 30.025029665s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)antiblip:
5 out of 60 is failed (threshold=3) [.................................X..........XX..X....X......] duration:
17.844205027s trace:
https://jaeger.dev.storj.io/trace/6b5565e31dae52f1
FAIL (2026-03-18 19:58:51 UTC)
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 13)
======LOGS======
Mar 18 19:53:19.380863725 (296h53m16.977634736s) [upload=332531]: Starting upload
Mar 18 19:53:19.380944319 (296h53m16.977715346s) [upload=332531]: Done starting upload
Mar 18 19:53:19.380976457 (296h53m16.977747482s) [upload=332531]: Uploading segments...
Mar 18 19:53:19.381333196 (296h53m16.978104222s) [upload=332531]: Got next segment. Inline: false
Mar 18 19:53:19.381379068 (296h53m16.978150099s) [upload=332531]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Mar 18 19:53:19.381386543 (296h53m16.978157579s) [upload=332531]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Mar 18 19:53:23.490845546 (296h53m21.087616603s) [seg_pos={0 0}, upload=332531]: Begin upload segment...
Mar 18 19:53:23.490917951 (296h53m21.087688988s) [seg_pos={0 0}, upload=332531]: Done begin upload segment.
Mar 18 19:53:23.490955926 (296h53m21.087726957s) [noise=true , address=45.91.116.5:4312 , piece_id=BDB6LYZZH2HXC3WWPDVMDQLCAAGRXBM5OCZU2JQVUJ7E6REVJ5SQ, seg_pos={0 0}, upload=332531]: Uploading piece...
Mar 18 19:53:23.492093139 (296h53m21.088864171s) [noise=true , address=193.149.164.67:4003 , piece_id=SOPPFFG22JZWQA5NHYGA2OFHC2KMALZOR3ATBQ6JYWJVXATWLSRQ, seg_pos={0 0}, upload=332531]: Uploading piece...
Mar 18 19:53:23.492835295 (296h53m21.089606329s) [noise=true , address=77.90.31.18:4833 , piece_id=ON3SKTN2H26XEXC43C44MAJIPY32NZLNWRU4PBSRJS6HLMXVNBMA, seg_pos={0 0}, upload=332531]: Uploading piece...
Mar 18 19:53:23.492958664 (296h53m21.089729699s) [noise=true , address=45.91.116.13:4167 , piece_id=3JWBJU4M3WSUM3D2TVZDGJKBSRXZY5ZQHV3GFSBKOJMLORY5W7IQ, seg_pos={0 0}, upload=332531]: Uploading piece...
Mar 18 19:53:23.493624498 (296h53m21.090395533s) [noise=true , address=145.79.227.13:4814 , piece_id=3KU6BHQSH4U4XYBSV4I77FCLQKR5RKWFY3QKZ26D5QRCJDQPKTKQ, seg_pos={0 0}, upload=332531]: Uploading piece...
Mar 18 19:53:23.493650442 (296h53m21.090421482s) [noise=true , address=79.127.205.237:4230 , piece_id=Y6AOKOD27V5RL6QWUY2ZAJMONWPTYU4TTU5VIAYJ4RVMNQMCRUZQ, seg_pos={0 0}, upload=332531]: Uploading piece...
Mar 18 19:53:23.494253839 (296h53m21.091024872s) [noise=true , address=45.91.116.14:4223 , piece_id=X3NSJBVWEZ3F3DUHQOSQSUCECIN7FARCFMRPJR6N7FHIIL3MS3NQ, seg_pos={0 0}, upload=332531]: Uploading piece...
Mar 18 19:53:23.494267618 (296h53m21.091038645s) [noise=true , address=145.79.227.22:4516 , piece_id=OES5UD4FMNBSPC4ZSDS45SRXLTAQT3QNGR6JBI5D5Y363JFGJ73Q, seg_pos={0 0}, upload=332531]: Uploading piece...
Mar 18 19:53:23.494879425 (296h53m21.091650453s) [noise=true , address=193.149.164.71:4287 , piece_id=ZE2Q4NHL3KUKRB36IDRMIZ7DIKDYZIBCM2L3PTLXALGL3PINBGAQ, seg_pos={0 0}, upload=332531]: Uploading piece...
Mar 18 19:53:23.494911912 (296h53m21.091682944s) [noise=true , address=193.149.164.69:4074 , piece_id=FUGRHBCQHA4YJ2EVCMKVQVITADSVLM3PWZHV2XM3QLRDGU4G2GPA, seg_pos={0 0}, upload=332531]: Uploading piece...
Mar 18 19:53:23.495497539 (296h53m21.09226857s) [noise=true , address=193.149.164.68:4066 , piece_id=FY5RBQUD2JCNXJCY2KIK744F4QZFHTW45PBDMASIIP27RWWB7SSA, seg_pos={0 0}, upload=332531]: Uploading piece...
Mar 18 19:53:23.496168841 (296h53m21.092939872s) [noise=true , address=145.79.227.41:4105 , piece_id=RTYHA5YP42CFAYVM6PHY5RTB47GRMWSZEBBPNLC532CXKPBT2VXA, seg_pos={0 0}, upload=332531]: Uploading piece...
Mar 18 19:53:23.497008065 (296h53m21.093779105s) [noise=true , address=79.127.205.235:4271 , piece_id=5XULCXAVIG3CQFTYA6Q3TFPRVICQFHQI2MSI2VIRIAGSK5UKKFXQ, seg_pos={0 0}, upload=332531]: Uploading piece...
Mar 18 19:53:23.497605357 (296h53m21.094376396s) [noise=true , address=145.79.227.37:4590 , piece_id=ZTQLQ2IZHJFZWBHMRT74V7JDZYXF5BG4656VBZDJ4J6NGKS2TMOA, seg_pos={0 0}, upload=332531]: Uploading piece...
Mar 18 19:53:23.498200357 (296h53m21.094971388s) [noise=true , address=193.149.164.66:4989 , piece_id=DVAJPRB7ABJJGUP4TYMLOGJVDLUARAMSPCG6FE2PVQXIDP2QBPGA, seg_pos={0 0}, upload=332531]: Uploading piece...
Mar 18 19:53:23.498757906 (296h53m21.095528939s) [noise=true , address=79.127.219.45:4868 , piece_id=6NDAYV5PFDBOMZKAXKO6LZUODM46MQPQOQIDKBHER3LG4CBEDHKQ, seg_pos={0 0}, upload=332531]: Uploading piece...
Mar 18 19:53:23.499215136 (296h53m21.095986159s) [noise=true , address=145.79.227.52:4453 , piece_id=YRAA65GHFFR45DX7MKR5BWBXID3YVSA4DMQDRKWKRDTJCHKHGMOA, seg_pos={0 0}, upload=332531]: Uploading piece...
Mar 18 19:53:23.499687663 (296h53m21.096458697s) [noise=true , address=77.90.31.14:4678 , piece_id=NWBLAAUF3GACR3Y6JRTHDEUJ24GMOTZIPSRGJIPTPLIVWBCSYAHA, seg_pos={0 0}, upload=332531]: Uploading piece...
Mar 18 19:53:23.600142173 (296h53m21.196913186s) [upload=332531]: Next returned nil segment
Mar 18 19:53:23.600161212 (296h53m21.196932236s) [upload=332531]: Waiting for error group managing segments...
Mar 18 19:53:23.620885875 (296h53m21.217656899s) [noise=true , address=45.91.116.5:4312 , piece_id=BDB6LYZZH2HXC3WWPDVMDQLCAAGRXBM5OCZU2JQVUJ7E6REVJ5SQ, seg_pos={0 0}, upload=332531]: Done uploading piece. err: <nil>
Mar 18 19:53:23.621154396 (296h53m21.217925412s) [noise=true , address=45.91.116.13:4167 , piece_id=3JWBJU4M3WSUM3D2TVZDGJKBSRXZY5ZQHV3GFSBKOJMLORY5W7IQ, seg_pos={0 0}, upload=332531]: Done uploading piece. err: <nil>
Mar 18 19:53:23.628883995 (296h53m21.225655021s) [noise=true , address=45.91.116.14:4223 , piece_id=X3NSJBVWEZ3F3DUHQOSQSUCECIN7FARCFMRPJR6N7FHIIL3MS3NQ, seg_pos={0 0}, upload=332531]: Done uploading piece. err: <nil>
Mar 18 19:53:23.630970813 (296h53m21.227741838s) [noise=true , address=79.127.219.45:4868 , piece_id=6NDAYV5PFDBOMZKAXKO6LZUODM46MQPQOQIDKBHER3LG4CBEDHKQ, seg_pos={0 0}, upload=332531]: Done uploading piece. err: <nil>
Mar 18 19:53:23.639825090 (296h53m21.236596104s) [noise=true , address=77.90.31.14:4678 , piece_id=NWBLAAUF3GACR3Y6JRTHDEUJ24GMOTZIPSRGJIPTPLIVWBCSYAHA, seg_pos={0 0}, upload=332531]: Done uploading piece. err: <nil>
Mar 18 19:53:23.642754677 (296h53m21.239525692s) [noise=true , address=77.90.31.18:4833 , piece_id=ON3SKTN2H26XEXC43C44MAJIPY32NZLNWRU4PBSRJS6HLMXVNBMA, seg_pos={0 0}, upload=332531]: Done uploading piece. err: <nil>
Mar 18 19:53:24.057422272 (296h53m21.654193306s) [noise=true , address=79.127.205.235:4271 , piece_id=5XULCXAVIG3CQFTYA6Q3TFPRVICQFHQI2MSI2VIRIAGSK5UKKFXQ, seg_pos={0 0}, upload=332531]: Done uploading piece. err: <nil>
Mar 18 19:53:24.074244577 (296h53m21.671015604s) [noise=true , address=193.149.164.67:4003 , piece_id=SOPPFFG22JZWQA5NHYGA2OFHC2KMALZOR3ATBQ6JYWJVXATWLSRQ, seg_pos={0 0}, upload=332531]: Done uploading piece. err: <nil>
Mar 18 19:53:24.097678734 (296h53m21.694449747s) [noise=true , address=79.127.205.237:4230 , piece_id=Y6AOKOD27V5RL6QWUY2ZAJMONWPTYU4TTU5VIAYJ4RVMNQMCRUZQ, seg_pos={0 0}, upload=332531]: Done uploading piece. err: <nil>
Mar 18 19:53:24.109685982 (296h53m21.706457009s) [noise=true , address=193.149.164.68:4066 , piece_id=FY5RBQUD2JCNXJCY2KIK744F4QZFHTW45PBDMASIIP27RWWB7SSA, seg_pos={0 0}, upload=332531]: Done uploading piece. err: <nil>
Mar 18 19:53:24.110659716 (296h53m21.707430744s) [noise=true , address=193.149.164.69:4074 , piece_id=FUGRHBCQHA4YJ2EVCMKVQVITADSVLM3PWZHV2XM3QLRDGU4G2GPA, seg_pos={0 0}, upload=332531]: Done uploading piece. err: <nil>
Mar 18 19:53:24.114541274 (296h53m21.7113123s) [noise=true , address=193.149.164.71:4287 , piece_id=ZE2Q4NHL3KUKRB36IDRMIZ7DIKDYZIBCM2L3PTLXALGL3PINBGAQ, seg_pos={0 0}, upload=332531]: Done uploading piece. err: <nil>
Mar 18 19:53:24.140894470 (296h53m21.737665488s) [noise=true , address=193.149.164.66:4989 , piece_id=DVAJPRB7ABJJGUP4TYMLOGJVDLUARAMSPCG6FE2PVQXIDP2QBPGA, seg_pos={0 0}, upload=332531]: Done uploading piece. err: <nil>
Mar 18 19:54:19.329327996 (296h54m16.926099018s) [noise=true , address=145.79.227.13:4814 , piece_id=3KU6BHQSH4U4XYBSV4I77FCLQKR5RKWFY3QKZ26D5QRCJDQPKTKQ, seg_pos={0 0}, upload=332531]: Done uploading piece. err: ecclient: upload failed (node:12ERtrzMuh8rZPsgebtyScmYvNVwz3UKJr5Nu2hLdu6sn8X3qLa, address:145.79.227.13:4814): protocol: rpc: dial tcp 145.79.227.13:4814: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 19:54:19.329567681 (296h54m16.926338706s) [noise=true , address=145.79.227.52:4453 , piece_id=YRAA65GHFFR45DX7MKR5BWBXID3YVSA4DMQDRKWKRDTJCHKHGMOA, seg_pos={0 0}, upload=332531]: Done uploading piece. err: ecclient: upload failed (node:1BmpsQ9VfhGzmkZHvSeMptcGhzGpdoN4oYvi83qZcC1Vh9anhP, address:145.79.227.52:4453): protocol: rpc: dial tcp 145.79.227.52:4453: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 19:54:19.329755822 (296h54m16.926526848s) [noise=true , address=145.79.227.41:4105 , piece_id=RTYHA5YP42CFAYVM6PHY5RTB47GRMWSZEBBPNLC532CXKPBT2VXA, seg_pos={0 0}, upload=332531]: Done uploading piece. err: ecclient: upload failed (node:1TVtMQNVV2kERp2aZwmS8E9JUiua1RCyFpZgFACRqH1Nfscw4C, address:145.79.227.41:4105): protocol: rpc: dial tcp 145.79.227.41:4105: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 19:54:19.330021146 (296h54m16.926792175s) [noise=true , address=145.79.227.22:4516 , piece_id=OES5UD4FMNBSPC4ZSDS45SRXLTAQT3QNGR6JBI5D5Y363JFGJ73Q, seg_pos={0 0}, upload=332531]: Done uploading piece. err: ecclient: upload failed (node:12Lhy3AxzYoF25AxWdvjrrV5NNCwP1cDqkEvv4tz9t9KZmF8WPK, address:145.79.227.22:4516): protocol: rpc: dial tcp 145.79.227.22:4516: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 19:54:19.330213680 (296h54m16.926984707s) [noise=true , address=145.79.227.37:4590 , piece_id=ZTQLQ2IZHJFZWBHMRT74V7JDZYXF5BG4656VBZDJ4J6NGKS2TMOA, seg_pos={0 0}, upload=332531]: Done uploading piece. err: ecclient: upload failed (node:1KnNqY1WNwMSFXFrQsWPAhivUTgSn7dKTx1mTTzYG2UxGCkasj, address:145.79.227.37:4590): protocol: rpc: dial tcp 145.79.227.37:4590: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 19:54:19.330250832 (296h54m16.927021848s) [seg_pos={0 0}, upload=332531]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil>
Mar 18 19:54:19.351979131 (296h54m16.948750155s) [upload=332531]: Done uploading segments...
Mar 18 19:54:19.352025297 (296h54m16.948796327s) [upload=332531]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13)
Duration: 17.844238703s
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
30 out of 60 is failed (threshold=3) [..............................XXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration:
30.025705026s trace:
https://jaeger.dev.storj.io/trace/78b4268f332bc432
FAIL (2026-03-18 19:58:42 UTC)
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 5)
======LOGS======
(296h58m13.001339841s) [seg_pos={0 0}, upload=332615]: Done begin upload segment.
Mar 18 19:58:15.404596858 (296h58m13.001367882s) [noise=true , address=145.79.227.52:4449 , piece_id=P3IHQHOZD753TYIHMRGTOPDYQVMUUWQYCNUAUT6IDXVNA7YGNVNQ, seg_pos={0 0}, upload=332615]: Uploading piece...
Mar 18 19:58:15.404943486 (296h58m13.001714519s) [noise=true , address=145.79.227.43:4158 , piece_id=DJ76CT74L5R2L3ZVKL2VU7DGB53MQDWFHIEQJ6SBXKBFGCRQCATQ, seg_pos={0 0}, upload=332615]: Uploading piece...
Mar 18 19:58:15.405238214 (296h58m13.002009266s) [noise=true , address=45.91.116.10:4085 , piece_id=2BF7BRBHJDZX6UO3FQYBZ542HRSOM24SU4UJEXDCNNLTRYGTKDEQ, seg_pos={0 0}, upload=332615]: Uploading piece...
Mar 18 19:58:15.405673582 (296h58m13.002444608s) [noise=true , address=145.79.227.50:4374 , piece_id=64DSSJA6JEZLBQ5THCECZ5L47R72N3UMNY5ALOLIJWQ7PRP675YA, seg_pos={0 0}, upload=332615]: Uploading piece...
Mar 18 19:58:15.406078897 (296h58m13.002849927s) [noise=true , address=145.79.227.16:4915 , piece_id=U2HZ26TBMQYPS667U7N2AZ3BNB4B6G3D3HCERWRXUGDCCYR4MHVA, seg_pos={0 0}, upload=332615]: Uploading piece...
Mar 18 19:58:15.406277732 (296h58m13.003048764s) [noise=true , address=45.91.116.9:4261 , piece_id=7O6O7VTCFQCASFUYOGCP5LIQYRYDBW6PL3GDDHXDM75TURPNWPRA, seg_pos={0 0}, upload=332615]: Uploading piece...
Mar 18 19:58:15.406475622 (296h58m13.003246652s) [noise=true , address=45.91.116.12:4225 , piece_id=SF5QISA3Q4SLRV7GXVVVRAUWKBDA6T6RFZFAGVJEWZMB2UTJZTCQ, seg_pos={0 0}, upload=332615]: Uploading piece...
Mar 18 19:58:15.406783206 (296h58m13.003554232s) [noise=true , address=145.79.227.34:4905 , piece_id=LRRLKOSNOK2GYT2YVSHXHEMGX62HWCZKIHL2OWHLW2MRIORMQFLA, seg_pos={0 0}, upload=332615]: Uploading piece...
Mar 18 19:58:15.406909092 (296h58m13.003680123s) [noise=true , address=145.79.227.25:4619 , piece_id=NGHMQ3ETO4G53SMS7QUXVFP3MCC3TBDWKVTAVFGLZHULBVNLWBEA, seg_pos={0 0}, upload=332615]: Uploading piece...
Mar 18 19:58:15.407261771 (296h58m13.004032794s) [noise=true , address=145.79.227.14:4842 , piece_id=LQOTZJTDB2SXER7Q3GNJCJPQJSJPH4OKU27V3LQ3COJ3HHEFGORQ, seg_pos={0 0}, upload=332615]: Uploading piece...
Mar 18 19:58:15.407460750 (296h58m13.004231776s) [noise=true , address=145.79.227.2:4118 , piece_id=OROYDJWDJMPCNMBNQNISU5EHWTGAWL34MXBJ4WY3PRR3HPSPPANA, seg_pos={0 0}, upload=332615]: Uploading piece...
Mar 18 19:58:15.407869158 (296h58m13.004640187s) [noise=true , address=145.79.227.11:4734 , piece_id=MYB4ZZNFGG56662IH3BBK453DGA4H7XOA543AN5UDRFRVZL2O4HQ, seg_pos={0 0}, upload=332615]: Uploading piece...
Mar 18 19:58:15.407900752 (296h58m13.004671793s) [noise=true , address=145.79.227.18:4008 , piece_id=ZCAXZW54UYYFB5DWGLTRO2Y2KQRNX4RA2HEU323SRI35KQAF3Y5A, seg_pos={0 0}, upload=332615]: Uploading piece...
Mar 18 19:58:15.408205363 (296h58m13.004976389s) [noise=true , address=77.90.31.21:4880 , piece_id=H4PBO2R3HNFXDVQXDRXZ3GBV46DUCXZJNN6JK5WQVG5NT6LTFIIQ, seg_pos={0 0}, upload=332615]: Uploading piece...
Mar 18 19:58:15.408499977 (296h58m13.005271009s) [noise=true , address=145.79.227.20:4441 , piece_id=IARKU7DCVMVRB7VPFZLLLCWNKCZNRGSZZWLIC45KZLHO7ATYD75Q, seg_pos={0 0}, upload=332615]: Uploading piece...
Mar 18 19:58:15.409185391 (296h58m13.005956425s) [noise=true , address=45.91.116.11:4152 , piece_id=4JJ4I3PJUWTDGS7ZYRJT3YMWWP7JB4KKZTGLDC7GRYIY32X6HWRA, seg_pos={0 0}, upload=332615]: Uploading piece...
Mar 18 19:58:15.473218271 (296h58m13.069989304s) [upload=332615]: Next returned nil segment
Mar 18 19:58:15.473254275 (296h58m13.070025298s) [upload=332615]: Waiting for error group managing segments...
Mar 18 19:58:15.477069711 (296h58m13.073840724s) [noise=true , address=45.91.116.12:4225 , piece_id=SF5QISA3Q4SLRV7GXVVVRAUWKBDA6T6RFZFAGVJEWZMB2UTJZTCQ, seg_pos={0 0}, upload=332615]: Done uploading piece. err: <nil>
Mar 18 19:58:15.477584542 (296h58m13.07435556s) [noise=true , address=45.91.116.9:4261 , piece_id=7O6O7VTCFQCASFUYOGCP5LIQYRYDBW6PL3GDDHXDM75TURPNWPRA, seg_pos={0 0}, upload=332615]: Done uploading piece. err: <nil>
Mar 18 19:58:15.477654632 (296h58m13.074425657s) [noise=true , address=45.91.116.11:4152 , piece_id=4JJ4I3PJUWTDGS7ZYRJT3YMWWP7JB4KKZTGLDC7GRYIY32X6HWRA, seg_pos={0 0}, upload=332615]: Done uploading piece. err: <nil>
Mar 18 19:58:15.477995764 (296h58m13.074766787s) [noise=true , address=45.91.116.10:4085 , piece_id=2BF7BRBHJDZX6UO3FQYBZ542HRSOM24SU4UJEXDCNNLTRYGTKDEQ, seg_pos={0 0}, upload=332615]: Done uploading piece. err: <nil>
Mar 18 19:58:15.478168327 (296h58m13.074939341s) [noise=true , address=77.90.31.21:4880 , piece_id=H4PBO2R3HNFXDVQXDRXZ3GBV46DUCXZJNN6JK5WQVG5NT6LTFIIQ, seg_pos={0 0}, upload=332615]: Done uploading piece. err: <nil>
Mar 18 19:58:42.720035565 (296h58m40.3168066s) [noise=true , address=145.79.227.14:4842 , piece_id=LQOTZJTDB2SXER7Q3GNJCJPQJSJPH4OKU27V3LQ3COJ3HHEFGORQ, seg_pos={0 0}, upload=332615]: Done uploading piece. err: ecclient: upload failed (node:12q4VdT5BcjUb16PBYHK6zrnW8dqENKrcGx9zdLtNefDVB1MUSn, address:145.79.227.14:4842): protocol: rpc: dial tcp 145.79.227.14:4842: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 19:58:42.720306450 (296h58m40.317077478s) [noise=true , address=145.79.227.43:4158 , piece_id=DJ76CT74L5R2L3ZVKL2VU7DGB53MQDWFHIEQJ6SBXKBFGCRQCATQ, seg_pos={0 0}, upload=332615]: Done uploading piece. err: ecclient: upload failed (node:1vJZrSvPvwebw9V2tTbvkPEmLeLjryBUnq9TL6NwdPsHU9xcq2, address:145.79.227.43:4158): protocol: rpc: dial tcp 145.79.227.43:4158: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 19:58:42.720849104 (296h58m40.317620139s) [noise=true , address=145.79.227.2:4118 , piece_id=OROYDJWDJMPCNMBNQNISU5EHWTGAWL34MXBJ4WY3PRR3HPSPPANA, seg_pos={0 0}, upload=332615]: Done uploading piece. err: ecclient: upload failed (node:12ty5LHXwLmHC7nRPmVfj2wVEif6JQAQQ5Z5218vv1fKK5sD3zE, address:145.79.227.2:4118): protocol: rpc: dial tcp 145.79.227.2:4118: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 19:58:42.720985814 (296h58m40.317756845s) [noise=true , address=145.79.227.11:4734 , piece_id=MYB4ZZNFGG56662IH3BBK453DGA4H7XOA543AN5UDRFRVZL2O4HQ, seg_pos={0 0}, upload=332615]: Done uploading piece. err: ecclient: upload failed (node:12GLb5LQje9wwivnj8Wj1tXcnS6mFb2Bco5saNwgmFJBVDujrv7, address:145.79.227.11:4734): protocol: rpc: dial tcp 145.79.227.11:4734: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 19:58:42.721247244 (296h58m40.318018276s) [noise=true , address=145.79.227.16:4915 , piece_id=U2HZ26TBMQYPS667U7N2AZ3BNB4B6G3D3HCERWRXUGDCCYR4MHVA, seg_pos={0 0}, upload=332615]: Done uploading piece. err: ecclient: upload failed (node:12fqZipCJaNd9u5TvdWZ9nJFs26MiGtYX3Bbh41eQWP1Z4eJRBt, address:145.79.227.16:4915): protocol: rpc: dial tcp 145.79.227.16:4915: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 19:58:42.721274509 (296h58m40.318045549s) [noise=true , address=145.79.227.18:4008 , piece_id=ZCAXZW54UYYFB5DWGLTRO2Y2KQRNX4RA2HEU323SRI35KQAF3Y5A, seg_pos={0 0}, upload=332615]: Done uploading piece. err: ecclient: upload failed (node:148CqQmrsJSfoGByHYRPoZmvP1VgQQpaMVLb7RgRQvNBuo7air, address:145.79.227.18:4008): protocol: rpc: dial tcp 145.79.227.18:4008: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 19:58:42.721571050 (296h58m40.318342076s) [noise=true , address=145.79.227.34:4905 , piece_id=LRRLKOSNOK2GYT2YVSHXHEMGX62HWCZKIHL2OWHLW2MRIORMQFLA, seg_pos={0 0}, upload=332615]: Done uploading piece. err: ecclient: upload failed (node:12ffKESNSVpXQzbwJinP6vvgE6zoF3RFz2NRUuuJHvd7bvk1gc9, address:145.79.227.34:4905): protocol: rpc: dial tcp 145.79.227.34:4905: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 19:58:42.721588311 (296h58m40.318359346s) [noise=true , address=145.79.227.52:4449 , piece_id=P3IHQHOZD753TYIHMRGTOPDYQVMUUWQYCNUAUT6IDXVNA7YGNVNQ, seg_pos={0 0}, upload=332615]: Done uploading piece. err: ecclient: upload failed (node:12f14pHDEEamEnHmeZCe38rjaE6EYWYkGMPrZMsbSDmApN5rckq, address:145.79.227.52:4449): protocol: rpc: dial tcp 145.79.227.52:4449: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 19:58:42.721919791 (296h58m40.318690824s) [noise=true , address=145.79.227.50:4374 , piece_id=64DSSJA6JEZLBQ5THCECZ5L47R72N3UMNY5ALOLIJWQ7PRP675YA, seg_pos={0 0}, upload=332615]: Done uploading piece. err: ecclient: upload failed (node:12QRixhWaUwXPdjcWKYHbiknME5JVomWYABY6egDD6VKwaTPB8K, address:145.79.227.50:4374): protocol: rpc: dial tcp 145.79.227.50:4374: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 19:58:42.721963930 (296h58m40.318734962s) [noise=true , address=145.79.227.25:4619 , piece_id=NGHMQ3ETO4G53SMS7QUXVFP3MCC3TBDWKVTAVFGLZHULBVNLWBEA, seg_pos={0 0}, upload=332615]: Done uploading piece. err: ecclient: upload failed (node:1zuLJznAUFHkanuomswSd7JN6CkqPXso8Ak8e5E57eaEqvjodU, address:145.79.227.25:4619): protocol: rpc: dial tcp 145.79.227.25:4619: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 19:58:42.722312844 (296h58m40.319083921s) [noise=true , address=145.79.227.20:4441 , piece_id=IARKU7DCVMVRB7VPFZLLLCWNKCZNRGSZZWLIC45KZLHO7ATYD75Q, seg_pos={0 0}, upload=332615]: Done uploading piece. err: ecclient: upload failed (node:12LjGecRNJE1kq67iMtxZFhWeaUZpUogsGxbz4VXwnHdQtTnbBr, address:145.79.227.20:4441): protocol: rpc: dial tcp 145.79.227.20:4441: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 19:58:42.722352341 (296h58m40.319123356s) [seg_pos={0 0}, upload=332615]: Done waiting for segment. successful: 5 optimal: 13 errs: <nil>
Mar 18 19:58:42.743755481 (296h58m40.340526519s) [upload=332615]: Done uploading segments...
Mar 18 19:58:42.743840731 (296h58m40.340611769s) [upload=332615]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 5)
Duration: 30.025720244s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
39 out of 60 is failed (threshold=6) [.................XXXXXXXXX.XXX...XXXXXXXXXXXXXXXXXXXXXXXXXXX] duration:
5.002439039s trace:
https://jaeger.dev.storj.io/trace/4557383cefe24a34
FAIL (2026-03-18 19:58:42 UTC)
operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.002468784s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
8 out of 60 is failed (threshold=3) [..............................XXXXXX.XX.....................] duration:
15.435152996s trace:
https://jaeger.dev.storj.io/trace/4b4d10a8051ad01b
FAIL (2026-03-18 19:58:30 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 15.435176243s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
38 out of 60 is failed (threshold=6) [..................XXXXXXXXX.XXX...XXXXXXXXXXXXXXXXXXXXXXXXXX] duration:
5.001255244s trace:
https://jaeger.dev.storj.io/trace/65d4a37f17354bcc
FAIL (2026-03-18 19:58:13 UTC)
operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.001287691s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
5 out of 60 is failed (threshold=3) [.................................XXX..........X...X.........] duration:
17.149608759s trace:
https://jaeger.dev.storj.io/trace/6a01088813e3d8b7
FAIL (2026-03-18 19:57:58 UTC)
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 12)
======LOGS======
m: *pb.BatchRequestItem_ObjectBegin
Mar 18 19:49:08.138517471 (296h49m5.735288494s) [upload=332459]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Mar 18 19:49:16.284286759 (296h49m13.881057773s) [seg_pos={0 0}, upload=332459]: Begin upload segment...
Mar 18 19:49:16.284352223 (296h49m13.88112325s) [seg_pos={0 0}, upload=332459]: Done begin upload segment.
Mar 18 19:49:16.284395789 (296h49m13.881166818s) [noise=true , address=193.149.164.68:4051 , piece_id=FCHPX2FRYUVNH4JAAPKI2Y7DSJCUJ5UG3EASXQYB7VEQPNERYUGQ, seg_pos={0 0}, upload=332459]: Uploading piece...
Mar 18 19:49:16.284449689 (296h49m13.881220728s) [noise=true , address=145.79.227.13:4822 , piece_id=Y5GSC3KFFGLK4B3NCXEMOZTAC3ASNGPJJOPOKCUBUJIL75E6T6FA, seg_pos={0 0}, upload=332459]: Uploading piece...
Mar 18 19:49:16.284549974 (296h49m13.881320988s) [noise=true , address=45.91.116.5:4308 , piece_id=XRXYA347E22AHAJ4XDENA5EHCV2F43HXL67MTUGRYSU6B3WQBDNQ, seg_pos={0 0}, upload=332459]: Uploading piece...
Mar 18 19:49:16.285144837 (296h49m13.88191587s) [noise=true , address=193.149.164.70:4117 , piece_id=PL6FWCO6XQRSVU26FH2KESA26D4Z2PCQVBPBDKBWLBPXHBV5LQVQ, seg_pos={0 0}, upload=332459]: Uploading piece...
Mar 18 19:49:16.285821284 (296h49m13.882592317s) [noise=true , address=145.79.227.52:4451 , piece_id=SBMEAZS5RDQW3RCJP2UU5T7BHF6P7VOICLP5P46YJ5LE2GB65CGA, seg_pos={0 0}, upload=332459]: Uploading piece...
Mar 18 19:49:16.286556627 (296h49m13.883327669s) [noise=true , address=79.127.205.235:4259 , piece_id=N7BGDCO2CR6HLPUWYF35T5OQRWQFEWFG7FPZ6Y33J7FGR5H2ZURQ, seg_pos={0 0}, upload=332459]: Uploading piece...
Mar 18 19:49:16.287228779 (296h49m13.883999813s) [noise=true , address=79.127.219.45:4864 , piece_id=UXHLRN5HUN3ZRXOZEYAJOS6QUN3PVIXXXED3TS6RPWSHXJ4WKS6Q, seg_pos={0 0}, upload=332459]: Uploading piece...
Mar 18 19:49:16.287269957 (296h49m13.884040985s) [noise=true , address=77.90.31.21:4894 , piece_id=IQ6775I4HFC5JQVYUCQPXYWPY3X6QRHVXM6IF4JEZRQMMRRWQDUQ, seg_pos={0 0}, upload=332459]: Uploading piece...
Mar 18 19:49:16.287437026 (296h49m13.884208049s) [noise=true , address=5.231.73.4:4010 , piece_id=LTONWX7UJVP3BPHHKKJUN6PTAR5TVPVKF3CD6NBV2MZWTLUOVQZQ, seg_pos={0 0}, upload=332459]: Uploading piece...
Mar 18 19:49:16.287832890 (296h49m13.884603914s) [noise=true , address=167.17.72.68:4657 , piece_id=UUIV3ERJSVOJT4TISEHCMR6EQCG6PMZUR6OWGK3ZVV33B33CUZCA, seg_pos={0 0}, upload=332459]: Uploading piece...
Mar 18 19:49:16.287952922 (296h49m13.884723957s) [noise=true , address=79.127.219.41:4939 , piece_id=WK6HMSTVM7LD2P5MPCKML4MBFZJX2QJWZS6SYPITDKNOLPMRPO6Q, seg_pos={0 0}, upload=332459]: Uploading piece...
Mar 18 19:49:16.288418812 (296h49m13.885189842s) [noise=true , address=145.79.227.44:4213 , piece_id=MK3BJSFCDOG2MAJYO2BAE2JG62EVEZMCCUATBBBT4QU44GLKK3QA, seg_pos={0 0}, upload=332459]: Uploading piece...
Mar 18 19:49:16.288922438 (296h49m13.885693461s) [noise=true , address=145.79.227.3:4163 , piece_id=3R3QBIJ2XDQZQXJ66RGNZBWZB6V4ERS5RJEZFBTS2LFMQ25G4FLA, seg_pos={0 0}, upload=332459]: Uploading piece...
Mar 18 19:49:16.289343580 (296h49m13.886114604s) [noise=true , address=193.149.164.71:4310 , piece_id=BKUA4WC73DSXJMEPCMTXTUD2PVQAAYD44MAGEB3SS3FEDJO7377A, seg_pos={0 0}, upload=332459]: Uploading piece...
Mar 18 19:49:16.289780590 (296h49m13.88655162s) [noise=true , address=145.79.227.49:4358 , piece_id=QSE44UR5YBYTJHWGOCBFVQPAUZ3SJV6P5WKBEF3MTFAKVRH2QSHQ, seg_pos={0 0}, upload=332459]: Uploading piece...
Mar 18 19:49:16.290237004 (296h49m13.887008027s) [noise=true , address=77.90.31.14:4685 , piece_id=3KAIZVB2LH6R57CFO55R7BJV7G4MGHFN6M5VTKT3J6LMI5F7PVAA, seg_pos={0 0}, upload=332459]: Uploading piece...
Mar 18 19:49:16.292879809 (296h49m13.889650843s) [noise=true , address=77.90.31.18:4817 , piece_id=FMVKZ4Z3YDPWZHTXU3ZTSWXMVUWLKQX7TUWXPQ4LOKBQK64PHF5Q, seg_pos={0 0}, upload=332459]: Uploading piece...
Mar 18 19:49:16.293664670 (296h49m13.890435693s) [noise=true , address=145.79.227.15:4908 , piece_id=LG24OMYBUVPVRBFZL4UUJLPBVNDTVVHQ4HAYTW2PGKHUKZPF2WCA, seg_pos={0 0}, upload=332459]: Uploading piece...
Mar 18 19:49:16.351440712 (296h49m13.948211724s) [upload=332459]: Next returned nil segment
Mar 18 19:49:16.351460639 (296h49m13.948231661s) [upload=332459]: Waiting for error group managing segments...
Mar 18 19:49:16.355941640 (296h49m13.952712664s) [noise=true , address=77.90.31.21:4894 , piece_id=IQ6775I4HFC5JQVYUCQPXYWPY3X6QRHVXM6IF4JEZRQMMRRWQDUQ, seg_pos={0 0}, upload=332459]: Done uploading piece. err: <nil>
Mar 18 19:49:16.356991929 (296h49m13.953762955s) [noise=true , address=79.127.219.41:4939 , piece_id=WK6HMSTVM7LD2P5MPCKML4MBFZJX2QJWZS6SYPITDKNOLPMRPO6Q, seg_pos={0 0}, upload=332459]: Done uploading piece. err: <nil>
Mar 18 19:49:16.361809763 (296h49m13.958580786s) [noise=true , address=45.91.116.5:4308 , piece_id=XRXYA347E22AHAJ4XDENA5EHCV2F43HXL67MTUGRYSU6B3WQBDNQ, seg_pos={0 0}, upload=332459]: Done uploading piece. err: <nil>
Mar 18 19:49:16.367502365 (296h49m13.964273396s) [noise=true , address=77.90.31.14:4685 , piece_id=3KAIZVB2LH6R57CFO55R7BJV7G4MGHFN6M5VTKT3J6LMI5F7PVAA, seg_pos={0 0}, upload=332459]: Done uploading piece. err: <nil>
Mar 18 19:49:16.381549888 (296h49m13.97832091s) [noise=true , address=79.127.219.45:4864 , piece_id=UXHLRN5HUN3ZRXOZEYAJOS6QUN3PVIXXXED3TS6RPWSHXJ4WKS6Q, seg_pos={0 0}, upload=332459]: Done uploading piece. err: <nil>
Mar 18 19:49:16.387503503 (296h49m13.984274527s) [noise=true , address=77.90.31.18:4817 , piece_id=FMVKZ4Z3YDPWZHTXU3ZTSWXMVUWLKQX7TUWXPQ4LOKBQK64PHF5Q, seg_pos={0 0}, upload=332459]: Done uploading piece. err: <nil>
Mar 18 19:49:16.498139242 (296h49m14.094910266s) [noise=true , address=5.231.73.4:4010 , piece_id=LTONWX7UJVP3BPHHKKJUN6PTAR5TVPVKF3CD6NBV2MZWTLUOVQZQ, seg_pos={0 0}, upload=332459]: Done uploading piece. err: <nil>
Mar 18 19:49:16.852944491 (296h49m14.449715518s) [noise=true , address=167.17.72.68:4657 , piece_id=UUIV3ERJSVOJT4TISEHCMR6EQCG6PMZUR6OWGK3ZVV33B33CUZCA, seg_pos={0 0}, upload=332459]: Done uploading piece. err: <nil>
Mar 18 19:49:16.878068834 (296h49m14.474839867s) [noise=true , address=79.127.205.235:4259 , piece_id=N7BGDCO2CR6HLPUWYF35T5OQRWQFEWFG7FPZ6Y33J7FGR5H2ZURQ, seg_pos={0 0}, upload=332459]: Done uploading piece. err: <nil>
Mar 18 19:49:16.892446253 (296h49m14.489217282s) [noise=true , address=193.149.164.68:4051 , piece_id=FCHPX2FRYUVNH4JAAPKI2Y7DSJCUJ5UG3EASXQYB7VEQPNERYUGQ, seg_pos={0 0}, upload=332459]: Done uploading piece. err: <nil>
Mar 18 19:49:16.893099142 (296h49m14.48987017s) [noise=true , address=193.149.164.71:4310 , piece_id=BKUA4WC73DSXJMEPCMTXTUD2PVQAAYD44MAGEB3SS3FEDJO7377A, seg_pos={0 0}, upload=332459]: Done uploading piece. err: <nil>
Mar 18 19:49:17.031110208 (296h49m14.627881232s) [noise=true , address=193.149.164.70:4117 , piece_id=PL6FWCO6XQRSVU26FH2KESA26D4Z2PCQVBPBDKBWLBPXHBV5LQVQ, seg_pos={0 0}, upload=332459]: Done uploading piece. err: <nil>
Mar 18 19:50:08.096628364 (296h50m5.693399398s) [noise=true , address=145.79.227.52:4451 , piece_id=SBMEAZS5RDQW3RCJP2UU5T7BHF6P7VOICLP5P46YJ5LE2GB65CGA, seg_pos={0 0}, upload=332459]: Done uploading piece. err: ecclient: upload failed (node:12bC4P3xBiPFkxLvavGFVbWKnbprCStL6UJgfKSA2Wd2fX7Mbfw, address:145.79.227.52:4451): protocol: rpc: dial tcp 145.79.227.52:4451: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 19:50:08.096712565 (296h50m5.693483599s) [noise=true , address=145.79.227.44:4213 , piece_id=MK3BJSFCDOG2MAJYO2BAE2JG62EVEZMCCUATBBBT4QU44GLKK3QA, seg_pos={0 0}, upload=332459]: Done uploading piece. err: ecclient: upload failed (node:129LRnDm7b4Fo5db7pto8GXYEtxHz9qdPAtQvaYH6gqdkMatQ6g, address:145.79.227.44:4213): protocol: rpc: dial tcp 145.79.227.44:4213: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 19:50:08.096943717 (296h50m5.693714752s) [noise=true , address=145.79.227.49:4358 , piece_id=QSE44UR5YBYTJHWGOCBFVQPAUZ3SJV6P5WKBEF3MTFAKVRH2QSHQ, seg_pos={0 0}, upload=332459]: Done uploading piece. err: ecclient: upload failed (node:1dwCNEcA9o51ZN3UaN3Frxdawwf8DXBj86iQMywjW8XP4ESmw1, address:145.79.227.49:4358): protocol: rpc: dial tcp 145.79.227.49:4358: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 19:50:08.097022451 (296h50m5.693793484s) [noise=true , address=145.79.227.3:4163 , piece_id=3R3QBIJ2XDQZQXJ66RGNZBWZB6V4ERS5RJEZFBTS2LFMQ25G4FLA, seg_pos={0 0}, upload=332459]: Done uploading piece. err: ecclient: upload failed (node:12w88UkGu5duYivJ6L6C9iJmTbnB63fDKiUUQprhwM3JshpXTdC, address:145.79.227.3:4163): protocol: rpc: dial tcp 145.79.227.3:4163: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 19:50:08.097155433 (296h50m5.693926468s) [noise=true , address=145.79.227.13:4822 , piece_id=Y5GSC3KFFGLK4B3NCXEMOZTAC3ASNGPJJOPOKCUBUJIL75E6T6FA, seg_pos={0 0}, upload=332459]: Done uploading piece. err: ecclient: upload failed (node:12fgLN9DGBGxWfBEePSsk84FTrnKoKWwNgoET3FpotfJxJ28f75, address:145.79.227.13:4822): protocol: rpc: dial tcp 145.79.227.13:4822: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 19:50:08.097360007 (296h50m5.694131039s) [noise=true , address=145.79.227.15:4908 , piece_id=LG24OMYBUVPVRBFZL4UUJLPBVNDTVVHQ4HAYTW2PGKHUKZPF2WCA, seg_pos={0 0}, upload=332459]: Done uploading piece. err: ecclient: upload failed (node:1pfQ5v1GwHhFTvh5Y7Jpb9qpWhZhFYqWX9PJpLEUzTkqKWcy4x, address:145.79.227.15:4908): protocol: rpc: dial tcp 145.79.227.15:4908: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 19:50:08.097462766 (296h50m5.69423378s) [seg_pos={0 0}, upload=332459]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil>
Mar 18 19:50:08.117913456 (296h50m5.714684468s) [upload=332459]: Done uploading segments...
Mar 18 19:50:08.117999321 (296h50m5.714770348s) [upload=332459]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12)
Duration: 17.149639163s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
37 out of 60 is failed (threshold=6) [...................XXXXXXXXX.XXX...XXXXXXXXXXXXXXXXXXXXXXXXX] duration:
5.001218946s trace:
https://jaeger.dev.storj.io/trace/171d1b72ee0ae39f
FAIL (2026-03-18 19:57:43 UTC)
operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.001243113s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)antiblip:
5 out of 60 is failed (threshold=3) [..................................X..........XX..X....X.....] duration:
18.079808288s trace:
https://jaeger.dev.storj.io/trace/6b5565e31dae52f1
FAIL (2026-03-18 19:57:40 UTC)
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 13)
======LOGS======
Mar 18 19:53:19.380863725 (296h53m16.977634736s) [upload=332531]: Starting upload
Mar 18 19:53:19.380944319 (296h53m16.977715346s) [upload=332531]: Done starting upload
Mar 18 19:53:19.380976457 (296h53m16.977747482s) [upload=332531]: Uploading segments...
Mar 18 19:53:19.381333196 (296h53m16.978104222s) [upload=332531]: Got next segment. Inline: false
Mar 18 19:53:19.381379068 (296h53m16.978150099s) [upload=332531]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Mar 18 19:53:19.381386543 (296h53m16.978157579s) [upload=332531]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Mar 18 19:53:23.490845546 (296h53m21.087616603s) [seg_pos={0 0}, upload=332531]: Begin upload segment...
Mar 18 19:53:23.490917951 (296h53m21.087688988s) [seg_pos={0 0}, upload=332531]: Done begin upload segment.
Mar 18 19:53:23.490955926 (296h53m21.087726957s) [noise=true , address=45.91.116.5:4312 , piece_id=BDB6LYZZH2HXC3WWPDVMDQLCAAGRXBM5OCZU2JQVUJ7E6REVJ5SQ, seg_pos={0 0}, upload=332531]: Uploading piece...
Mar 18 19:53:23.492093139 (296h53m21.088864171s) [noise=true , address=193.149.164.67:4003 , piece_id=SOPPFFG22JZWQA5NHYGA2OFHC2KMALZOR3ATBQ6JYWJVXATWLSRQ, seg_pos={0 0}, upload=332531]: Uploading piece...
Mar 18 19:53:23.492835295 (296h53m21.089606329s) [noise=true , address=77.90.31.18:4833 , piece_id=ON3SKTN2H26XEXC43C44MAJIPY32NZLNWRU4PBSRJS6HLMXVNBMA, seg_pos={0 0}, upload=332531]: Uploading piece...
Mar 18 19:53:23.492958664 (296h53m21.089729699s) [noise=true , address=45.91.116.13:4167 , piece_id=3JWBJU4M3WSUM3D2TVZDGJKBSRXZY5ZQHV3GFSBKOJMLORY5W7IQ, seg_pos={0 0}, upload=332531]: Uploading piece...
Mar 18 19:53:23.493624498 (296h53m21.090395533s) [noise=true , address=145.79.227.13:4814 , piece_id=3KU6BHQSH4U4XYBSV4I77FCLQKR5RKWFY3QKZ26D5QRCJDQPKTKQ, seg_pos={0 0}, upload=332531]: Uploading piece...
Mar 18 19:53:23.493650442 (296h53m21.090421482s) [noise=true , address=79.127.205.237:4230 , piece_id=Y6AOKOD27V5RL6QWUY2ZAJMONWPTYU4TTU5VIAYJ4RVMNQMCRUZQ, seg_pos={0 0}, upload=332531]: Uploading piece...
Mar 18 19:53:23.494253839 (296h53m21.091024872s) [noise=true , address=45.91.116.14:4223 , piece_id=X3NSJBVWEZ3F3DUHQOSQSUCECIN7FARCFMRPJR6N7FHIIL3MS3NQ, seg_pos={0 0}, upload=332531]: Uploading piece...
Mar 18 19:53:23.494267618 (296h53m21.091038645s) [noise=true , address=145.79.227.22:4516 , piece_id=OES5UD4FMNBSPC4ZSDS45SRXLTAQT3QNGR6JBI5D5Y363JFGJ73Q, seg_pos={0 0}, upload=332531]: Uploading piece...
Mar 18 19:53:23.494879425 (296h53m21.091650453s) [noise=true , address=193.149.164.71:4287 , piece_id=ZE2Q4NHL3KUKRB36IDRMIZ7DIKDYZIBCM2L3PTLXALGL3PINBGAQ, seg_pos={0 0}, upload=332531]: Uploading piece...
Mar 18 19:53:23.494911912 (296h53m21.091682944s) [noise=true , address=193.149.164.69:4074 , piece_id=FUGRHBCQHA4YJ2EVCMKVQVITADSVLM3PWZHV2XM3QLRDGU4G2GPA, seg_pos={0 0}, upload=332531]: Uploading piece...
Mar 18 19:53:23.495497539 (296h53m21.09226857s) [noise=true , address=193.149.164.68:4066 , piece_id=FY5RBQUD2JCNXJCY2KIK744F4QZFHTW45PBDMASIIP27RWWB7SSA, seg_pos={0 0}, upload=332531]: Uploading piece...
Mar 18 19:53:23.496168841 (296h53m21.092939872s) [noise=true , address=145.79.227.41:4105 , piece_id=RTYHA5YP42CFAYVM6PHY5RTB47GRMWSZEBBPNLC532CXKPBT2VXA, seg_pos={0 0}, upload=332531]: Uploading piece...
Mar 18 19:53:23.497008065 (296h53m21.093779105s) [noise=true , address=79.127.205.235:4271 , piece_id=5XULCXAVIG3CQFTYA6Q3TFPRVICQFHQI2MSI2VIRIAGSK5UKKFXQ, seg_pos={0 0}, upload=332531]: Uploading piece...
Mar 18 19:53:23.497605357 (296h53m21.094376396s) [noise=true , address=145.79.227.37:4590 , piece_id=ZTQLQ2IZHJFZWBHMRT74V7JDZYXF5BG4656VBZDJ4J6NGKS2TMOA, seg_pos={0 0}, upload=332531]: Uploading piece...
Mar 18 19:53:23.498200357 (296h53m21.094971388s) [noise=true , address=193.149.164.66:4989 , piece_id=DVAJPRB7ABJJGUP4TYMLOGJVDLUARAMSPCG6FE2PVQXIDP2QBPGA, seg_pos={0 0}, upload=332531]: Uploading piece...
Mar 18 19:53:23.498757906 (296h53m21.095528939s) [noise=true , address=79.127.219.45:4868 , piece_id=6NDAYV5PFDBOMZKAXKO6LZUODM46MQPQOQIDKBHER3LG4CBEDHKQ, seg_pos={0 0}, upload=332531]: Uploading piece...
Mar 18 19:53:23.499215136 (296h53m21.095986159s) [noise=true , address=145.79.227.52:4453 , piece_id=YRAA65GHFFR45DX7MKR5BWBXID3YVSA4DMQDRKWKRDTJCHKHGMOA, seg_pos={0 0}, upload=332531]: Uploading piece...
Mar 18 19:53:23.499687663 (296h53m21.096458697s) [noise=true , address=77.90.31.14:4678 , piece_id=NWBLAAUF3GACR3Y6JRTHDEUJ24GMOTZIPSRGJIPTPLIVWBCSYAHA, seg_pos={0 0}, upload=332531]: Uploading piece...
Mar 18 19:53:23.600142173 (296h53m21.196913186s) [upload=332531]: Next returned nil segment
Mar 18 19:53:23.600161212 (296h53m21.196932236s) [upload=332531]: Waiting for error group managing segments...
Mar 18 19:53:23.620885875 (296h53m21.217656899s) [noise=true , address=45.91.116.5:4312 , piece_id=BDB6LYZZH2HXC3WWPDVMDQLCAAGRXBM5OCZU2JQVUJ7E6REVJ5SQ, seg_pos={0 0}, upload=332531]: Done uploading piece. err: <nil>
Mar 18 19:53:23.621154396 (296h53m21.217925412s) [noise=true , address=45.91.116.13:4167 , piece_id=3JWBJU4M3WSUM3D2TVZDGJKBSRXZY5ZQHV3GFSBKOJMLORY5W7IQ, seg_pos={0 0}, upload=332531]: Done uploading piece. err: <nil>
Mar 18 19:53:23.628883995 (296h53m21.225655021s) [noise=true , address=45.91.116.14:4223 , piece_id=X3NSJBVWEZ3F3DUHQOSQSUCECIN7FARCFMRPJR6N7FHIIL3MS3NQ, seg_pos={0 0}, upload=332531]: Done uploading piece. err: <nil>
Mar 18 19:53:23.630970813 (296h53m21.227741838s) [noise=true , address=79.127.219.45:4868 , piece_id=6NDAYV5PFDBOMZKAXKO6LZUODM46MQPQOQIDKBHER3LG4CBEDHKQ, seg_pos={0 0}, upload=332531]: Done uploading piece. err: <nil>
Mar 18 19:53:23.639825090 (296h53m21.236596104s) [noise=true , address=77.90.31.14:4678 , piece_id=NWBLAAUF3GACR3Y6JRTHDEUJ24GMOTZIPSRGJIPTPLIVWBCSYAHA, seg_pos={0 0}, upload=332531]: Done uploading piece. err: <nil>
Mar 18 19:53:23.642754677 (296h53m21.239525692s) [noise=true , address=77.90.31.18:4833 , piece_id=ON3SKTN2H26XEXC43C44MAJIPY32NZLNWRU4PBSRJS6HLMXVNBMA, seg_pos={0 0}, upload=332531]: Done uploading piece. err: <nil>
Mar 18 19:53:24.057422272 (296h53m21.654193306s) [noise=true , address=79.127.205.235:4271 , piece_id=5XULCXAVIG3CQFTYA6Q3TFPRVICQFHQI2MSI2VIRIAGSK5UKKFXQ, seg_pos={0 0}, upload=332531]: Done uploading piece. err: <nil>
Mar 18 19:53:24.074244577 (296h53m21.671015604s) [noise=true , address=193.149.164.67:4003 , piece_id=SOPPFFG22JZWQA5NHYGA2OFHC2KMALZOR3ATBQ6JYWJVXATWLSRQ, seg_pos={0 0}, upload=332531]: Done uploading piece. err: <nil>
Mar 18 19:53:24.097678734 (296h53m21.694449747s) [noise=true , address=79.127.205.237:4230 , piece_id=Y6AOKOD27V5RL6QWUY2ZAJMONWPTYU4TTU5VIAYJ4RVMNQMCRUZQ, seg_pos={0 0}, upload=332531]: Done uploading piece. err: <nil>
Mar 18 19:53:24.109685982 (296h53m21.706457009s) [noise=true , address=193.149.164.68:4066 , piece_id=FY5RBQUD2JCNXJCY2KIK744F4QZFHTW45PBDMASIIP27RWWB7SSA, seg_pos={0 0}, upload=332531]: Done uploading piece. err: <nil>
Mar 18 19:53:24.110659716 (296h53m21.707430744s) [noise=true , address=193.149.164.69:4074 , piece_id=FUGRHBCQHA4YJ2EVCMKVQVITADSVLM3PWZHV2XM3QLRDGU4G2GPA, seg_pos={0 0}, upload=332531]: Done uploading piece. err: <nil>
Mar 18 19:53:24.114541274 (296h53m21.7113123s) [noise=true , address=193.149.164.71:4287 , piece_id=ZE2Q4NHL3KUKRB36IDRMIZ7DIKDYZIBCM2L3PTLXALGL3PINBGAQ, seg_pos={0 0}, upload=332531]: Done uploading piece. err: <nil>
Mar 18 19:53:24.140894470 (296h53m21.737665488s) [noise=true , address=193.149.164.66:4989 , piece_id=DVAJPRB7ABJJGUP4TYMLOGJVDLUARAMSPCG6FE2PVQXIDP2QBPGA, seg_pos={0 0}, upload=332531]: Done uploading piece. err: <nil>
Mar 18 19:54:19.329327996 (296h54m16.926099018s) [noise=true , address=145.79.227.13:4814 , piece_id=3KU6BHQSH4U4XYBSV4I77FCLQKR5RKWFY3QKZ26D5QRCJDQPKTKQ, seg_pos={0 0}, upload=332531]: Done uploading piece. err: ecclient: upload failed (node:12ERtrzMuh8rZPsgebtyScmYvNVwz3UKJr5Nu2hLdu6sn8X3qLa, address:145.79.227.13:4814): protocol: rpc: dial tcp 145.79.227.13:4814: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 19:54:19.329567681 (296h54m16.926338706s) [noise=true , address=145.79.227.52:4453 , piece_id=YRAA65GHFFR45DX7MKR5BWBXID3YVSA4DMQDRKWKRDTJCHKHGMOA, seg_pos={0 0}, upload=332531]: Done uploading piece. err: ecclient: upload failed (node:1BmpsQ9VfhGzmkZHvSeMptcGhzGpdoN4oYvi83qZcC1Vh9anhP, address:145.79.227.52:4453): protocol: rpc: dial tcp 145.79.227.52:4453: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 19:54:19.329755822 (296h54m16.926526848s) [noise=true , address=145.79.227.41:4105 , piece_id=RTYHA5YP42CFAYVM6PHY5RTB47GRMWSZEBBPNLC532CXKPBT2VXA, seg_pos={0 0}, upload=332531]: Done uploading piece. err: ecclient: upload failed (node:1TVtMQNVV2kERp2aZwmS8E9JUiua1RCyFpZgFACRqH1Nfscw4C, address:145.79.227.41:4105): protocol: rpc: dial tcp 145.79.227.41:4105: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 19:54:19.330021146 (296h54m16.926792175s) [noise=true , address=145.79.227.22:4516 , piece_id=OES5UD4FMNBSPC4ZSDS45SRXLTAQT3QNGR6JBI5D5Y363JFGJ73Q, seg_pos={0 0}, upload=332531]: Done uploading piece. err: ecclient: upload failed (node:12Lhy3AxzYoF25AxWdvjrrV5NNCwP1cDqkEvv4tz9t9KZmF8WPK, address:145.79.227.22:4516): protocol: rpc: dial tcp 145.79.227.22:4516: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 19:54:19.330213680 (296h54m16.926984707s) [noise=true , address=145.79.227.37:4590 , piece_id=ZTQLQ2IZHJFZWBHMRT74V7JDZYXF5BG4656VBZDJ4J6NGKS2TMOA, seg_pos={0 0}, upload=332531]: Done uploading piece. err: ecclient: upload failed (node:1KnNqY1WNwMSFXFrQsWPAhivUTgSn7dKTx1mTTzYG2UxGCkasj, address:145.79.227.37:4590): protocol: rpc: dial tcp 145.79.227.37:4590: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 19:54:19.330250832 (296h54m16.927021848s) [seg_pos={0 0}, upload=332531]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil>
Mar 18 19:54:19.351979131 (296h54m16.948750155s) [upload=332531]: Done uploading segments...
Mar 18 19:54:19.352025297 (296h54m16.948796327s) [upload=332531]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13)
Duration: 18.079858587s
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
8 out of 60 is failed (threshold=3) [...............................XXXXXX.XX....................] duration:
13.487988487s trace:
https://jaeger.dev.storj.io/trace/4b4d10a8051ad01b
FAIL (2026-03-18 19:57:29 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 13.488016363s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
29 out of 60 is failed (threshold=3) [...............................XXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration:
30.025760093s trace:
https://jaeger.dev.storj.io/trace/6ca0c26d6a90740e
FAIL (2026-03-18 19:57:28 UTC)
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
Mar 18 19:56:58.333352337 (296h56m55.930123367s) [upload=332594]: Starting upload
Mar 18 19:56:58.333457680 (296h56m55.930228714s) [upload=332594]: Done starting upload
Mar 18 19:56:58.333695711 (296h56m55.930466736s) [upload=332594]: Uploading segments...
Mar 18 19:56:58.334110096 (296h56m55.930881111s) [upload=332594]: Got next segment. Inline: false
Mar 18 19:56:58.334197433 (296h56m55.930968447s) [upload=332594]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Mar 18 19:56:58.334204472 (296h56m55.930975502s) [upload=332594]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Mar 18 19:57:01.291912012 (296h56m58.888683044s) [seg_pos={0 0}, upload=332594]: Begin upload segment...
Mar 18 19:57:01.292004457 (296h56m58.888775492s) [seg_pos={0 0}, upload=332594]: Done begin upload segment.
Mar 18 19:57:01.292060591 (296h56m58.888831619s) [noise=true , address=45.91.116.9:4295 , piece_id=BYU2PXOJBYIBM2OXDCEOSJYNW3NIAQVQVPF62XMNYGJ5FAV4XG2Q, seg_pos={0 0}, upload=332594]: Uploading piece...
Mar 18 19:57:01.293388233 (296h56m58.890159267s) [noise=true , address=77.90.31.11:4554 , piece_id=HI7M3YW6D64JLTX26INOUUPMKI3XW2IZ3EAADJUTAB55YF6FVNYQ, seg_pos={0 0}, upload=332594]: Uploading piece...
Mar 18 19:57:01.293623843 (296h56m58.890394875s) [noise=true , address=145.79.227.24:4586 , piece_id=UMLEVZWOWVPBN72WZPZBQSWTZISAFTCEEUCL54GI7UYSZ2XFYSRA, seg_pos={0 0}, upload=332594]: Uploading piece...
Mar 18 19:57:01.294109016 (296h56m58.890880049s) [noise=true , address=45.91.116.10:4082 , piece_id=CMPPZJDYECJ3OU4QPF4QTDSSMS376U6XETY2Z47UDTSDBOFHTI6Q, seg_pos={0 0}, upload=332594]: Uploading piece...
Mar 18 19:57:01.294341442 (296h56m58.891112483s) [noise=true , address=145.79.227.40:4051 , piece_id=UUL6GAUA3NGKUH7DY77OFAYOI6BZC7FR2XA2PR4HA2PI6RC6P65A, seg_pos={0 0}, upload=332594]: Uploading piece...
Mar 18 19:57:01.294625245 (296h56m58.891396282s) [noise=true , address=145.79.227.34:4906 , piece_id=L4ESJZOHAHNDQ6RWBKA5NVLQKAXYQHRU6TF6XT3SRRO6N5R7YI5Q, seg_pos={0 0}, upload=332594]: Uploading piece...
Mar 18 19:57:01.294928117 (296h56m58.891699142s) [noise=true , address=45.91.116.14:4216 , piece_id=5TUPWDQSBUQW5DKN642FMSL6K42SMUF6NUXYOQTNPZ32BOT4CDQQ, seg_pos={0 0}, upload=332594]: Uploading piece...
Mar 18 19:57:01.295166517 (296h56m58.89193755s) [noise=true , address=45.91.116.5:4299 , piece_id=D5MOGRRWXPCCNMFLZAZC3AGQ5J3NBOD4BRX2NVAMPDCTMEIPTK7A, seg_pos={0 0}, upload=332594]: Uploading piece...
Mar 18 19:57:01.295624169 (296h56m58.892395205s) [noise=true , address=145.79.227.11:4734 , piece_id=XGB6DUC6EBWCCEDE6R22DPYM6JLE3UPNEYX6PNDDA37TDITWCJKQ, seg_pos={0 0}, upload=332594]: Uploading piece...
Mar 18 19:57:01.295732070 (296h56m58.892503105s) [noise=true , address=45.91.116.7:4011 , piece_id=SBXOIZSZ2BWNKJOBO3SR7QW6PIHFTFGITNITOX5FGE6SHUPL5H4Q, seg_pos={0 0}, upload=332594]: Uploading piece...
Mar 18 19:57:01.296195790 (296h56m58.892966824s) [noise=true , address=77.90.31.17:4769 , piece_id=QGIXRXBAPEXJTJKCDBAJBJ76ISTZK3VL5X6DZC42YPN3FYVPGJSQ, seg_pos={0 0}, upload=332594]: Uploading piece...
Mar 18 19:57:01.296267432 (296h56m58.893038468s) [noise=true , address=77.90.31.10:4522 , piece_id=KWQTOSD7JUEJ7CKLIUG3ZFQ2I22EJTKM6UUYYKLFCDEJLBYJLUWA, seg_pos={0 0}, upload=332594]: Uploading piece...
Mar 18 19:57:01.297247059 (296h56m58.894018092s) [noise=true , address=79.127.219.38:4821 , piece_id=I2F5BAYKOZDJVL6L4OIN3FZ5D66KCNR6LCS5RTJSF7ULXVPSLLKQ, seg_pos={0 0}, upload=332594]: Uploading piece...
Mar 18 19:57:01.297861027 (296h56m58.89463206s) [noise=true , address=77.90.31.14:4666 , piece_id=AV3RHEMNC7QU2JDMD5VNOFFVEQ6WTDUFYGNPJFDYCNJ7S4UIFIMA, seg_pos={0 0}, upload=332594]: Uploading piece...
Mar 18 19:57:01.298618616 (296h56m58.895389641s) [noise=true , address=77.90.31.21:4881 , piece_id=6GV5BQGQGSID32OG2XYIUMIXX4BKPFLRUB2VXAHGSEY2AIALLAWA, seg_pos={0 0}, upload=332594]: Uploading piece...
Mar 18 19:57:01.298986093 (296h56m58.895757127s) [noise=true , address=45.91.116.4:4953 , piece_id=P5543OXJZ36F6ZZQIMO6FNXZVUMEJCME73DCBMSBX24S7NC5HSDQ, seg_pos={0 0}, upload=332594]: Uploading piece...
Mar 18 19:57:01.374422430 (296h56m58.971193447s) [upload=332594]: Next returned nil segment
Mar 18 19:57:01.374442215 (296h56m58.971213248s) [upload=332594]: Waiting for error group managing segments...
Mar 18 19:57:01.378871525 (296h56m58.975642553s) [noise=true , address=79.127.219.38:4821 , piece_id=I2F5BAYKOZDJVL6L4OIN3FZ5D66KCNR6LCS5RTJSF7ULXVPSLLKQ, seg_pos={0 0}, upload=332594]: Done uploading piece. err: <nil>
Mar 18 19:57:01.380060891 (296h56m58.976831922s) [noise=true , address=45.91.116.9:4295 , piece_id=BYU2PXOJBYIBM2OXDCEOSJYNW3NIAQVQVPF62XMNYGJ5FAV4XG2Q, seg_pos={0 0}, upload=332594]: Done uploading piece. err: <nil>
Mar 18 19:57:01.380669745 (296h56m58.977440773s) [noise=true , address=45.91.116.5:4299 , piece_id=D5MOGRRWXPCCNMFLZAZC3AGQ5J3NBOD4BRX2NVAMPDCTMEIPTK7A, seg_pos={0 0}, upload=332594]: Done uploading piece. err: <nil>
Mar 18 19:57:01.380949176 (296h56m58.977720209s) [noise=true , address=77.90.31.21:4881 , piece_id=6GV5BQGQGSID32OG2XYIUMIXX4BKPFLRUB2VXAHGSEY2AIALLAWA, seg_pos={0 0}, upload=332594]: Done uploading piece. err: <nil>
Mar 18 19:57:01.381572080 (296h56m58.978343104s) [noise=true , address=77.90.31.10:4522 , piece_id=KWQTOSD7JUEJ7CKLIUG3ZFQ2I22EJTKM6UUYYKLFCDEJLBYJLUWA, seg_pos={0 0}, upload=332594]: Done uploading piece. err: <nil>
Mar 18 19:57:01.382094780 (296h56m58.97886581s) [noise=true , address=45.91.116.14:4216 , piece_id=5TUPWDQSBUQW5DKN642FMSL6K42SMUF6NUXYOQTNPZ32BOT4CDQQ, seg_pos={0 0}, upload=332594]: Done uploading piece. err: <nil>
Mar 18 19:57:01.382613243 (296h56m58.979384275s) [noise=true , address=77.90.31.14:4666 , piece_id=AV3RHEMNC7QU2JDMD5VNOFFVEQ6WTDUFYGNPJFDYCNJ7S4UIFIMA, seg_pos={0 0}, upload=332594]: Done uploading piece. err: <nil>
Mar 18 19:57:01.383311493 (296h56m58.980082526s) [noise=true , address=45.91.116.7:4011 , piece_id=SBXOIZSZ2BWNKJOBO3SR7QW6PIHFTFGITNITOX5FGE6SHUPL5H4Q, seg_pos={0 0}, upload=332594]: Done uploading piece. err: <nil>
Mar 18 19:57:01.384158199 (296h56m58.980929226s) [noise=true , address=45.91.116.4:4953 , piece_id=P5543OXJZ36F6ZZQIMO6FNXZVUMEJCME73DCBMSBX24S7NC5HSDQ, seg_pos={0 0}, upload=332594]: Done uploading piece. err: <nil>
Mar 18 19:57:01.384778598 (296h56m58.981549623s) [noise=true , address=77.90.31.11:4554 , piece_id=HI7M3YW6D64JLTX26INOUUPMKI3XW2IZ3EAADJUTAB55YF6FVNYQ, seg_pos={0 0}, upload=332594]: Done uploading piece. err: <nil>
Mar 18 19:57:01.395984618 (296h56m58.992755644s) [noise=true , address=45.91.116.10:4082 , piece_id=CMPPZJDYECJ3OU4QPF4QTDSSMS376U6XETY2Z47UDTSDBOFHTI6Q, seg_pos={0 0}, upload=332594]: Done uploading piece. err: <nil>
Mar 18 19:57:01.401718155 (296h56m58.998489179s) [noise=true , address=77.90.31.17:4769 , piece_id=QGIXRXBAPEXJTJKCDBAJBJ76ISTZK3VL5X6DZC42YPN3FYVPGJSQ, seg_pos={0 0}, upload=332594]: Done uploading piece. err: <nil>
Mar 18 19:57:28.301191801 (296h57m25.897962833s) [noise=true , address=145.79.227.40:4051 , piece_id=UUL6GAUA3NGKUH7DY77OFAYOI6BZC7FR2XA2PR4HA2PI6RC6P65A, seg_pos={0 0}, upload=332594]: Done uploading piece. err: ecclient: upload failed (node:12rPbTARUrq7YsD8X75ocnyEvC517itk79s3Dp1wUg72fVuBS8n, address:145.79.227.40:4051): protocol: context deadline exceeded; context deadline exceeded
Mar 18 19:57:28.301299982 (296h57m25.898071016s) [noise=true , address=145.79.227.24:4586 , piece_id=UMLEVZWOWVPBN72WZPZBQSWTZISAFTCEEUCL54GI7UYSZ2XFYSRA, seg_pos={0 0}, upload=332594]: Done uploading piece. err: ecclient: upload failed (node:1mkxgR2oLXcTdyCsxmR9Q1an9ZeLgkTSHRCtR4ZJqKbsStNEMh, address:145.79.227.24:4586): protocol: context deadline exceeded; context deadline exceeded
Mar 18 19:57:28.301437819 (296h57m25.898208853s) [noise=true , address=145.79.227.11:4734 , piece_id=XGB6DUC6EBWCCEDE6R22DPYM6JLE3UPNEYX6PNDDA37TDITWCJKQ, seg_pos={0 0}, upload=332594]: Done uploading piece. err: ecclient: upload failed (node:12GLb5LQje9wwivnj8Wj1tXcnS6mFb2Bco5saNwgmFJBVDujrv7, address:145.79.227.11:4734): protocol: context deadline exceeded; context deadline exceeded
Mar 18 19:57:28.301612120 (296h57m25.898383153s) [noise=true , address=145.79.227.34:4906 , piece_id=L4ESJZOHAHNDQ6RWBKA5NVLQKAXYQHRU6TF6XT3SRRO6N5R7YI5Q, seg_pos={0 0}, upload=332594]: Done uploading piece. err: ecclient: upload failed (node:12m9gfwSNSLJB2XyBAQUVtWUpc1jQtr5dvYWzHqXeaLQUzyW9cw, address:145.79.227.34:4906): protocol: context deadline exceeded; context deadline exceeded
Mar 18 19:57:28.301650292 (296h57m25.898421301s) [seg_pos={0 0}, upload=332594]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Mar 18 19:57:28.324470858 (296h57m25.921241889s) [upload=332594]: Done uploading segments...
Mar 18 19:57:28.324536858 (296h57m25.921307888s) [upload=332594]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 30.025800658s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
36 out of 60 is failed (threshold=6) [....................XXXXXXXXX.XXX...XXXXXXXXXXXXXXXXXXXXXXXX] duration:
5.001229475s trace:
https://jaeger.dev.storj.io/trace/f4116ffffcf9199
FAIL (2026-03-18 19:57:15 UTC)
operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.001319218s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)antiblip:
5 out of 60 is failed (threshold=3) [...................................X..........XX..X....X....] duration:
19.363047191s trace:
https://jaeger.dev.storj.io/trace/6b5565e31dae52f1
FAIL (2026-03-18 19:57:04 UTC)
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 13)
======LOGS======
Mar 18 19:53:19.380863725 (296h53m16.977634736s) [upload=332531]: Starting upload
Mar 18 19:53:19.380944319 (296h53m16.977715346s) [upload=332531]: Done starting upload
Mar 18 19:53:19.380976457 (296h53m16.977747482s) [upload=332531]: Uploading segments...
Mar 18 19:53:19.381333196 (296h53m16.978104222s) [upload=332531]: Got next segment. Inline: false
Mar 18 19:53:19.381379068 (296h53m16.978150099s) [upload=332531]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Mar 18 19:53:19.381386543 (296h53m16.978157579s) [upload=332531]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Mar 18 19:53:23.490845546 (296h53m21.087616603s) [seg_pos={0 0}, upload=332531]: Begin upload segment...
Mar 18 19:53:23.490917951 (296h53m21.087688988s) [seg_pos={0 0}, upload=332531]: Done begin upload segment.
Mar 18 19:53:23.490955926 (296h53m21.087726957s) [noise=true , address=45.91.116.5:4312 , piece_id=BDB6LYZZH2HXC3WWPDVMDQLCAAGRXBM5OCZU2JQVUJ7E6REVJ5SQ, seg_pos={0 0}, upload=332531]: Uploading piece...
Mar 18 19:53:23.492093139 (296h53m21.088864171s) [noise=true , address=193.149.164.67:4003 , piece_id=SOPPFFG22JZWQA5NHYGA2OFHC2KMALZOR3ATBQ6JYWJVXATWLSRQ, seg_pos={0 0}, upload=332531]: Uploading piece...
Mar 18 19:53:23.492835295 (296h53m21.089606329s) [noise=true , address=77.90.31.18:4833 , piece_id=ON3SKTN2H26XEXC43C44MAJIPY32NZLNWRU4PBSRJS6HLMXVNBMA, seg_pos={0 0}, upload=332531]: Uploading piece...
Mar 18 19:53:23.492958664 (296h53m21.089729699s) [noise=true , address=45.91.116.13:4167 , piece_id=3JWBJU4M3WSUM3D2TVZDGJKBSRXZY5ZQHV3GFSBKOJMLORY5W7IQ, seg_pos={0 0}, upload=332531]: Uploading piece...
Mar 18 19:53:23.493624498 (296h53m21.090395533s) [noise=true , address=145.79.227.13:4814 , piece_id=3KU6BHQSH4U4XYBSV4I77FCLQKR5RKWFY3QKZ26D5QRCJDQPKTKQ, seg_pos={0 0}, upload=332531]: Uploading piece...
Mar 18 19:53:23.493650442 (296h53m21.090421482s) [noise=true , address=79.127.205.237:4230 , piece_id=Y6AOKOD27V5RL6QWUY2ZAJMONWPTYU4TTU5VIAYJ4RVMNQMCRUZQ, seg_pos={0 0}, upload=332531]: Uploading piece...
Mar 18 19:53:23.494253839 (296h53m21.091024872s) [noise=true , address=45.91.116.14:4223 , piece_id=X3NSJBVWEZ3F3DUHQOSQSUCECIN7FARCFMRPJR6N7FHIIL3MS3NQ, seg_pos={0 0}, upload=332531]: Uploading piece...
Mar 18 19:53:23.494267618 (296h53m21.091038645s) [noise=true , address=145.79.227.22:4516 , piece_id=OES5UD4FMNBSPC4ZSDS45SRXLTAQT3QNGR6JBI5D5Y363JFGJ73Q, seg_pos={0 0}, upload=332531]: Uploading piece...
Mar 18 19:53:23.494879425 (296h53m21.091650453s) [noise=true , address=193.149.164.71:4287 , piece_id=ZE2Q4NHL3KUKRB36IDRMIZ7DIKDYZIBCM2L3PTLXALGL3PINBGAQ, seg_pos={0 0}, upload=332531]: Uploading piece...
Mar 18 19:53:23.494911912 (296h53m21.091682944s) [noise=true , address=193.149.164.69:4074 , piece_id=FUGRHBCQHA4YJ2EVCMKVQVITADSVLM3PWZHV2XM3QLRDGU4G2GPA, seg_pos={0 0}, upload=332531]: Uploading piece...
Mar 18 19:53:23.495497539 (296h53m21.09226857s) [noise=true , address=193.149.164.68:4066 , piece_id=FY5RBQUD2JCNXJCY2KIK744F4QZFHTW45PBDMASIIP27RWWB7SSA, seg_pos={0 0}, upload=332531]: Uploading piece...
Mar 18 19:53:23.496168841 (296h53m21.092939872s) [noise=true , address=145.79.227.41:4105 , piece_id=RTYHA5YP42CFAYVM6PHY5RTB47GRMWSZEBBPNLC532CXKPBT2VXA, seg_pos={0 0}, upload=332531]: Uploading piece...
Mar 18 19:53:23.497008065 (296h53m21.093779105s) [noise=true , address=79.127.205.235:4271 , piece_id=5XULCXAVIG3CQFTYA6Q3TFPRVICQFHQI2MSI2VIRIAGSK5UKKFXQ, seg_pos={0 0}, upload=332531]: Uploading piece...
Mar 18 19:53:23.497605357 (296h53m21.094376396s) [noise=true , address=145.79.227.37:4590 , piece_id=ZTQLQ2IZHJFZWBHMRT74V7JDZYXF5BG4656VBZDJ4J6NGKS2TMOA, seg_pos={0 0}, upload=332531]: Uploading piece...
Mar 18 19:53:23.498200357 (296h53m21.094971388s) [noise=true , address=193.149.164.66:4989 , piece_id=DVAJPRB7ABJJGUP4TYMLOGJVDLUARAMSPCG6FE2PVQXIDP2QBPGA, seg_pos={0 0}, upload=332531]: Uploading piece...
Mar 18 19:53:23.498757906 (296h53m21.095528939s) [noise=true , address=79.127.219.45:4868 , piece_id=6NDAYV5PFDBOMZKAXKO6LZUODM46MQPQOQIDKBHER3LG4CBEDHKQ, seg_pos={0 0}, upload=332531]: Uploading piece...
Mar 18 19:53:23.499215136 (296h53m21.095986159s) [noise=true , address=145.79.227.52:4453 , piece_id=YRAA65GHFFR45DX7MKR5BWBXID3YVSA4DMQDRKWKRDTJCHKHGMOA, seg_pos={0 0}, upload=332531]: Uploading piece...
Mar 18 19:53:23.499687663 (296h53m21.096458697s) [noise=true , address=77.90.31.14:4678 , piece_id=NWBLAAUF3GACR3Y6JRTHDEUJ24GMOTZIPSRGJIPTPLIVWBCSYAHA, seg_pos={0 0}, upload=332531]: Uploading piece...
Mar 18 19:53:23.600142173 (296h53m21.196913186s) [upload=332531]: Next returned nil segment
Mar 18 19:53:23.600161212 (296h53m21.196932236s) [upload=332531]: Waiting for error group managing segments...
Mar 18 19:53:23.620885875 (296h53m21.217656899s) [noise=true , address=45.91.116.5:4312 , piece_id=BDB6LYZZH2HXC3WWPDVMDQLCAAGRXBM5OCZU2JQVUJ7E6REVJ5SQ, seg_pos={0 0}, upload=332531]: Done uploading piece. err: <nil>
Mar 18 19:53:23.621154396 (296h53m21.217925412s) [noise=true , address=45.91.116.13:4167 , piece_id=3JWBJU4M3WSUM3D2TVZDGJKBSRXZY5ZQHV3GFSBKOJMLORY5W7IQ, seg_pos={0 0}, upload=332531]: Done uploading piece. err: <nil>
Mar 18 19:53:23.628883995 (296h53m21.225655021s) [noise=true , address=45.91.116.14:4223 , piece_id=X3NSJBVWEZ3F3DUHQOSQSUCECIN7FARCFMRPJR6N7FHIIL3MS3NQ, seg_pos={0 0}, upload=332531]: Done uploading piece. err: <nil>
Mar 18 19:53:23.630970813 (296h53m21.227741838s) [noise=true , address=79.127.219.45:4868 , piece_id=6NDAYV5PFDBOMZKAXKO6LZUODM46MQPQOQIDKBHER3LG4CBEDHKQ, seg_pos={0 0}, upload=332531]: Done uploading piece. err: <nil>
Mar 18 19:53:23.639825090 (296h53m21.236596104s) [noise=true , address=77.90.31.14:4678 , piece_id=NWBLAAUF3GACR3Y6JRTHDEUJ24GMOTZIPSRGJIPTPLIVWBCSYAHA, seg_pos={0 0}, upload=332531]: Done uploading piece. err: <nil>
Mar 18 19:53:23.642754677 (296h53m21.239525692s) [noise=true , address=77.90.31.18:4833 , piece_id=ON3SKTN2H26XEXC43C44MAJIPY32NZLNWRU4PBSRJS6HLMXVNBMA, seg_pos={0 0}, upload=332531]: Done uploading piece. err: <nil>
Mar 18 19:53:24.057422272 (296h53m21.654193306s) [noise=true , address=79.127.205.235:4271 , piece_id=5XULCXAVIG3CQFTYA6Q3TFPRVICQFHQI2MSI2VIRIAGSK5UKKFXQ, seg_pos={0 0}, upload=332531]: Done uploading piece. err: <nil>
Mar 18 19:53:24.074244577 (296h53m21.671015604s) [noise=true , address=193.149.164.67:4003 , piece_id=SOPPFFG22JZWQA5NHYGA2OFHC2KMALZOR3ATBQ6JYWJVXATWLSRQ, seg_pos={0 0}, upload=332531]: Done uploading piece. err: <nil>
Mar 18 19:53:24.097678734 (296h53m21.694449747s) [noise=true , address=79.127.205.237:4230 , piece_id=Y6AOKOD27V5RL6QWUY2ZAJMONWPTYU4TTU5VIAYJ4RVMNQMCRUZQ, seg_pos={0 0}, upload=332531]: Done uploading piece. err: <nil>
Mar 18 19:53:24.109685982 (296h53m21.706457009s) [noise=true , address=193.149.164.68:4066 , piece_id=FY5RBQUD2JCNXJCY2KIK744F4QZFHTW45PBDMASIIP27RWWB7SSA, seg_pos={0 0}, upload=332531]: Done uploading piece. err: <nil>
Mar 18 19:53:24.110659716 (296h53m21.707430744s) [noise=true , address=193.149.164.69:4074 , piece_id=FUGRHBCQHA4YJ2EVCMKVQVITADSVLM3PWZHV2XM3QLRDGU4G2GPA, seg_pos={0 0}, upload=332531]: Done uploading piece. err: <nil>
Mar 18 19:53:24.114541274 (296h53m21.7113123s) [noise=true , address=193.149.164.71:4287 , piece_id=ZE2Q4NHL3KUKRB36IDRMIZ7DIKDYZIBCM2L3PTLXALGL3PINBGAQ, seg_pos={0 0}, upload=332531]: Done uploading piece. err: <nil>
Mar 18 19:53:24.140894470 (296h53m21.737665488s) [noise=true , address=193.149.164.66:4989 , piece_id=DVAJPRB7ABJJGUP4TYMLOGJVDLUARAMSPCG6FE2PVQXIDP2QBPGA, seg_pos={0 0}, upload=332531]: Done uploading piece. err: <nil>
Mar 18 19:54:19.329327996 (296h54m16.926099018s) [noise=true , address=145.79.227.13:4814 , piece_id=3KU6BHQSH4U4XYBSV4I77FCLQKR5RKWFY3QKZ26D5QRCJDQPKTKQ, seg_pos={0 0}, upload=332531]: Done uploading piece. err: ecclient: upload failed (node:12ERtrzMuh8rZPsgebtyScmYvNVwz3UKJr5Nu2hLdu6sn8X3qLa, address:145.79.227.13:4814): protocol: rpc: dial tcp 145.79.227.13:4814: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 19:54:19.329567681 (296h54m16.926338706s) [noise=true , address=145.79.227.52:4453 , piece_id=YRAA65GHFFR45DX7MKR5BWBXID3YVSA4DMQDRKWKRDTJCHKHGMOA, seg_pos={0 0}, upload=332531]: Done uploading piece. err: ecclient: upload failed (node:1BmpsQ9VfhGzmkZHvSeMptcGhzGpdoN4oYvi83qZcC1Vh9anhP, address:145.79.227.52:4453): protocol: rpc: dial tcp 145.79.227.52:4453: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 19:54:19.329755822 (296h54m16.926526848s) [noise=true , address=145.79.227.41:4105 , piece_id=RTYHA5YP42CFAYVM6PHY5RTB47GRMWSZEBBPNLC532CXKPBT2VXA, seg_pos={0 0}, upload=332531]: Done uploading piece. err: ecclient: upload failed (node:1TVtMQNVV2kERp2aZwmS8E9JUiua1RCyFpZgFACRqH1Nfscw4C, address:145.79.227.41:4105): protocol: rpc: dial tcp 145.79.227.41:4105: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 19:54:19.330021146 (296h54m16.926792175s) [noise=true , address=145.79.227.22:4516 , piece_id=OES5UD4FMNBSPC4ZSDS45SRXLTAQT3QNGR6JBI5D5Y363JFGJ73Q, seg_pos={0 0}, upload=332531]: Done uploading piece. err: ecclient: upload failed (node:12Lhy3AxzYoF25AxWdvjrrV5NNCwP1cDqkEvv4tz9t9KZmF8WPK, address:145.79.227.22:4516): protocol: rpc: dial tcp 145.79.227.22:4516: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 19:54:19.330213680 (296h54m16.926984707s) [noise=true , address=145.79.227.37:4590 , piece_id=ZTQLQ2IZHJFZWBHMRT74V7JDZYXF5BG4656VBZDJ4J6NGKS2TMOA, seg_pos={0 0}, upload=332531]: Done uploading piece. err: ecclient: upload failed (node:1KnNqY1WNwMSFXFrQsWPAhivUTgSn7dKTx1mTTzYG2UxGCkasj, address:145.79.227.37:4590): protocol: rpc: dial tcp 145.79.227.37:4590: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 19:54:19.330250832 (296h54m16.927021848s) [seg_pos={0 0}, upload=332531]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil>
Mar 18 19:54:19.351979131 (296h54m16.948750155s) [upload=332531]: Done uploading segments...
Mar 18 19:54:19.352025297 (296h54m16.948796327s) [upload=332531]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13)
Duration: 19.363067199s
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
8 out of 60 is failed (threshold=3) [................................XXXXXX.XX...................] duration:
19.443319405s trace:
https://jaeger.dev.storj.io/trace/4b4d10a8051ad01b
FAIL (2026-03-18 19:56:54 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 19.443417128s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
5 out of 60 is failed (threshold=3) [..................................XXX..........X...X........] duration:
20.171121856s trace:
https://jaeger.dev.storj.io/trace/6a01088813e3d8b7
FAIL (2026-03-18 19:56:53 UTC)
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 12)
======LOGS======
m: *pb.BatchRequestItem_ObjectBegin
Mar 18 19:49:08.138517471 (296h49m5.735288494s) [upload=332459]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Mar 18 19:49:16.284286759 (296h49m13.881057773s) [seg_pos={0 0}, upload=332459]: Begin upload segment...
Mar 18 19:49:16.284352223 (296h49m13.88112325s) [seg_pos={0 0}, upload=332459]: Done begin upload segment.
Mar 18 19:49:16.284395789 (296h49m13.881166818s) [noise=true , address=193.149.164.68:4051 , piece_id=FCHPX2FRYUVNH4JAAPKI2Y7DSJCUJ5UG3EASXQYB7VEQPNERYUGQ, seg_pos={0 0}, upload=332459]: Uploading piece...
Mar 18 19:49:16.284449689 (296h49m13.881220728s) [noise=true , address=145.79.227.13:4822 , piece_id=Y5GSC3KFFGLK4B3NCXEMOZTAC3ASNGPJJOPOKCUBUJIL75E6T6FA, seg_pos={0 0}, upload=332459]: Uploading piece...
Mar 18 19:49:16.284549974 (296h49m13.881320988s) [noise=true , address=45.91.116.5:4308 , piece_id=XRXYA347E22AHAJ4XDENA5EHCV2F43HXL67MTUGRYSU6B3WQBDNQ, seg_pos={0 0}, upload=332459]: Uploading piece...
Mar 18 19:49:16.285144837 (296h49m13.88191587s) [noise=true , address=193.149.164.70:4117 , piece_id=PL6FWCO6XQRSVU26FH2KESA26D4Z2PCQVBPBDKBWLBPXHBV5LQVQ, seg_pos={0 0}, upload=332459]: Uploading piece...
Mar 18 19:49:16.285821284 (296h49m13.882592317s) [noise=true , address=145.79.227.52:4451 , piece_id=SBMEAZS5RDQW3RCJP2UU5T7BHF6P7VOICLP5P46YJ5LE2GB65CGA, seg_pos={0 0}, upload=332459]: Uploading piece...
Mar 18 19:49:16.286556627 (296h49m13.883327669s) [noise=true , address=79.127.205.235:4259 , piece_id=N7BGDCO2CR6HLPUWYF35T5OQRWQFEWFG7FPZ6Y33J7FGR5H2ZURQ, seg_pos={0 0}, upload=332459]: Uploading piece...
Mar 18 19:49:16.287228779 (296h49m13.883999813s) [noise=true , address=79.127.219.45:4864 , piece_id=UXHLRN5HUN3ZRXOZEYAJOS6QUN3PVIXXXED3TS6RPWSHXJ4WKS6Q, seg_pos={0 0}, upload=332459]: Uploading piece...
Mar 18 19:49:16.287269957 (296h49m13.884040985s) [noise=true , address=77.90.31.21:4894 , piece_id=IQ6775I4HFC5JQVYUCQPXYWPY3X6QRHVXM6IF4JEZRQMMRRWQDUQ, seg_pos={0 0}, upload=332459]: Uploading piece...
Mar 18 19:49:16.287437026 (296h49m13.884208049s) [noise=true , address=5.231.73.4:4010 , piece_id=LTONWX7UJVP3BPHHKKJUN6PTAR5TVPVKF3CD6NBV2MZWTLUOVQZQ, seg_pos={0 0}, upload=332459]: Uploading piece...
Mar 18 19:49:16.287832890 (296h49m13.884603914s) [noise=true , address=167.17.72.68:4657 , piece_id=UUIV3ERJSVOJT4TISEHCMR6EQCG6PMZUR6OWGK3ZVV33B33CUZCA, seg_pos={0 0}, upload=332459]: Uploading piece...
Mar 18 19:49:16.287952922 (296h49m13.884723957s) [noise=true , address=79.127.219.41:4939 , piece_id=WK6HMSTVM7LD2P5MPCKML4MBFZJX2QJWZS6SYPITDKNOLPMRPO6Q, seg_pos={0 0}, upload=332459]: Uploading piece...
Mar 18 19:49:16.288418812 (296h49m13.885189842s) [noise=true , address=145.79.227.44:4213 , piece_id=MK3BJSFCDOG2MAJYO2BAE2JG62EVEZMCCUATBBBT4QU44GLKK3QA, seg_pos={0 0}, upload=332459]: Uploading piece...
Mar 18 19:49:16.288922438 (296h49m13.885693461s) [noise=true , address=145.79.227.3:4163 , piece_id=3R3QBIJ2XDQZQXJ66RGNZBWZB6V4ERS5RJEZFBTS2LFMQ25G4FLA, seg_pos={0 0}, upload=332459]: Uploading piece...
Mar 18 19:49:16.289343580 (296h49m13.886114604s) [noise=true , address=193.149.164.71:4310 , piece_id=BKUA4WC73DSXJMEPCMTXTUD2PVQAAYD44MAGEB3SS3FEDJO7377A, seg_pos={0 0}, upload=332459]: Uploading piece...
Mar 18 19:49:16.289780590 (296h49m13.88655162s) [noise=true , address=145.79.227.49:4358 , piece_id=QSE44UR5YBYTJHWGOCBFVQPAUZ3SJV6P5WKBEF3MTFAKVRH2QSHQ, seg_pos={0 0}, upload=332459]: Uploading piece...
Mar 18 19:49:16.290237004 (296h49m13.887008027s) [noise=true , address=77.90.31.14:4685 , piece_id=3KAIZVB2LH6R57CFO55R7BJV7G4MGHFN6M5VTKT3J6LMI5F7PVAA, seg_pos={0 0}, upload=332459]: Uploading piece...
Mar 18 19:49:16.292879809 (296h49m13.889650843s) [noise=true , address=77.90.31.18:4817 , piece_id=FMVKZ4Z3YDPWZHTXU3ZTSWXMVUWLKQX7TUWXPQ4LOKBQK64PHF5Q, seg_pos={0 0}, upload=332459]: Uploading piece...
Mar 18 19:49:16.293664670 (296h49m13.890435693s) [noise=true , address=145.79.227.15:4908 , piece_id=LG24OMYBUVPVRBFZL4UUJLPBVNDTVVHQ4HAYTW2PGKHUKZPF2WCA, seg_pos={0 0}, upload=332459]: Uploading piece...
Mar 18 19:49:16.351440712 (296h49m13.948211724s) [upload=332459]: Next returned nil segment
Mar 18 19:49:16.351460639 (296h49m13.948231661s) [upload=332459]: Waiting for error group managing segments...
Mar 18 19:49:16.355941640 (296h49m13.952712664s) [noise=true , address=77.90.31.21:4894 , piece_id=IQ6775I4HFC5JQVYUCQPXYWPY3X6QRHVXM6IF4JEZRQMMRRWQDUQ, seg_pos={0 0}, upload=332459]: Done uploading piece. err: <nil>
Mar 18 19:49:16.356991929 (296h49m13.953762955s) [noise=true , address=79.127.219.41:4939 , piece_id=WK6HMSTVM7LD2P5MPCKML4MBFZJX2QJWZS6SYPITDKNOLPMRPO6Q, seg_pos={0 0}, upload=332459]: Done uploading piece. err: <nil>
Mar 18 19:49:16.361809763 (296h49m13.958580786s) [noise=true , address=45.91.116.5:4308 , piece_id=XRXYA347E22AHAJ4XDENA5EHCV2F43HXL67MTUGRYSU6B3WQBDNQ, seg_pos={0 0}, upload=332459]: Done uploading piece. err: <nil>
Mar 18 19:49:16.367502365 (296h49m13.964273396s) [noise=true , address=77.90.31.14:4685 , piece_id=3KAIZVB2LH6R57CFO55R7BJV7G4MGHFN6M5VTKT3J6LMI5F7PVAA, seg_pos={0 0}, upload=332459]: Done uploading piece. err: <nil>
Mar 18 19:49:16.381549888 (296h49m13.97832091s) [noise=true , address=79.127.219.45:4864 , piece_id=UXHLRN5HUN3ZRXOZEYAJOS6QUN3PVIXXXED3TS6RPWSHXJ4WKS6Q, seg_pos={0 0}, upload=332459]: Done uploading piece. err: <nil>
Mar 18 19:49:16.387503503 (296h49m13.984274527s) [noise=true , address=77.90.31.18:4817 , piece_id=FMVKZ4Z3YDPWZHTXU3ZTSWXMVUWLKQX7TUWXPQ4LOKBQK64PHF5Q, seg_pos={0 0}, upload=332459]: Done uploading piece. err: <nil>
Mar 18 19:49:16.498139242 (296h49m14.094910266s) [noise=true , address=5.231.73.4:4010 , piece_id=LTONWX7UJVP3BPHHKKJUN6PTAR5TVPVKF3CD6NBV2MZWTLUOVQZQ, seg_pos={0 0}, upload=332459]: Done uploading piece. err: <nil>
Mar 18 19:49:16.852944491 (296h49m14.449715518s) [noise=true , address=167.17.72.68:4657 , piece_id=UUIV3ERJSVOJT4TISEHCMR6EQCG6PMZUR6OWGK3ZVV33B33CUZCA, seg_pos={0 0}, upload=332459]: Done uploading piece. err: <nil>
Mar 18 19:49:16.878068834 (296h49m14.474839867s) [noise=true , address=79.127.205.235:4259 , piece_id=N7BGDCO2CR6HLPUWYF35T5OQRWQFEWFG7FPZ6Y33J7FGR5H2ZURQ, seg_pos={0 0}, upload=332459]: Done uploading piece. err: <nil>
Mar 18 19:49:16.892446253 (296h49m14.489217282s) [noise=true , address=193.149.164.68:4051 , piece_id=FCHPX2FRYUVNH4JAAPKI2Y7DSJCUJ5UG3EASXQYB7VEQPNERYUGQ, seg_pos={0 0}, upload=332459]: Done uploading piece. err: <nil>
Mar 18 19:49:16.893099142 (296h49m14.48987017s) [noise=true , address=193.149.164.71:4310 , piece_id=BKUA4WC73DSXJMEPCMTXTUD2PVQAAYD44MAGEB3SS3FEDJO7377A, seg_pos={0 0}, upload=332459]: Done uploading piece. err: <nil>
Mar 18 19:49:17.031110208 (296h49m14.627881232s) [noise=true , address=193.149.164.70:4117 , piece_id=PL6FWCO6XQRSVU26FH2KESA26D4Z2PCQVBPBDKBWLBPXHBV5LQVQ, seg_pos={0 0}, upload=332459]: Done uploading piece. err: <nil>
Mar 18 19:50:08.096628364 (296h50m5.693399398s) [noise=true , address=145.79.227.52:4451 , piece_id=SBMEAZS5RDQW3RCJP2UU5T7BHF6P7VOICLP5P46YJ5LE2GB65CGA, seg_pos={0 0}, upload=332459]: Done uploading piece. err: ecclient: upload failed (node:12bC4P3xBiPFkxLvavGFVbWKnbprCStL6UJgfKSA2Wd2fX7Mbfw, address:145.79.227.52:4451): protocol: rpc: dial tcp 145.79.227.52:4451: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 19:50:08.096712565 (296h50m5.693483599s) [noise=true , address=145.79.227.44:4213 , piece_id=MK3BJSFCDOG2MAJYO2BAE2JG62EVEZMCCUATBBBT4QU44GLKK3QA, seg_pos={0 0}, upload=332459]: Done uploading piece. err: ecclient: upload failed (node:129LRnDm7b4Fo5db7pto8GXYEtxHz9qdPAtQvaYH6gqdkMatQ6g, address:145.79.227.44:4213): protocol: rpc: dial tcp 145.79.227.44:4213: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 19:50:08.096943717 (296h50m5.693714752s) [noise=true , address=145.79.227.49:4358 , piece_id=QSE44UR5YBYTJHWGOCBFVQPAUZ3SJV6P5WKBEF3MTFAKVRH2QSHQ, seg_pos={0 0}, upload=332459]: Done uploading piece. err: ecclient: upload failed (node:1dwCNEcA9o51ZN3UaN3Frxdawwf8DXBj86iQMywjW8XP4ESmw1, address:145.79.227.49:4358): protocol: rpc: dial tcp 145.79.227.49:4358: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 19:50:08.097022451 (296h50m5.693793484s) [noise=true , address=145.79.227.3:4163 , piece_id=3R3QBIJ2XDQZQXJ66RGNZBWZB6V4ERS5RJEZFBTS2LFMQ25G4FLA, seg_pos={0 0}, upload=332459]: Done uploading piece. err: ecclient: upload failed (node:12w88UkGu5duYivJ6L6C9iJmTbnB63fDKiUUQprhwM3JshpXTdC, address:145.79.227.3:4163): protocol: rpc: dial tcp 145.79.227.3:4163: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 19:50:08.097155433 (296h50m5.693926468s) [noise=true , address=145.79.227.13:4822 , piece_id=Y5GSC3KFFGLK4B3NCXEMOZTAC3ASNGPJJOPOKCUBUJIL75E6T6FA, seg_pos={0 0}, upload=332459]: Done uploading piece. err: ecclient: upload failed (node:12fgLN9DGBGxWfBEePSsk84FTrnKoKWwNgoET3FpotfJxJ28f75, address:145.79.227.13:4822): protocol: rpc: dial tcp 145.79.227.13:4822: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 19:50:08.097360007 (296h50m5.694131039s) [noise=true , address=145.79.227.15:4908 , piece_id=LG24OMYBUVPVRBFZL4UUJLPBVNDTVVHQ4HAYTW2PGKHUKZPF2WCA, seg_pos={0 0}, upload=332459]: Done uploading piece. err: ecclient: upload failed (node:1pfQ5v1GwHhFTvh5Y7Jpb9qpWhZhFYqWX9PJpLEUzTkqKWcy4x, address:145.79.227.15:4908): protocol: rpc: dial tcp 145.79.227.15:4908: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 19:50:08.097462766 (296h50m5.69423378s) [seg_pos={0 0}, upload=332459]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil>
Mar 18 19:50:08.117913456 (296h50m5.714684468s) [upload=332459]: Done uploading segments...
Mar 18 19:50:08.117999321 (296h50m5.714770348s) [upload=332459]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12)
Duration: 20.171148901s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
35 out of 60 is failed (threshold=6) [.....................XXXXXXXXX.XXX...XXXXXXXXXXXXXXXXXXXXXXX] duration:
5.001285531s trace:
https://jaeger.dev.storj.io/trace/2fa746b388d8584e
FAIL (2026-03-18 19:56:51 UTC)
operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.001396942s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
28 out of 60 is failed (threshold=3) [................................XXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration:
30.028510985s trace:
https://jaeger.dev.storj.io/trace/5c0ff142424b364
FAIL (2026-03-18 19:56:45 UTC)
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
Mar 18 19:56:15.570608696 (296h56m13.167379719s) [upload=332579]: Starting upload
Mar 18 19:56:15.570670794 (296h56m13.167441816s) [upload=332579]: Done starting upload
Mar 18 19:56:15.570931293 (296h56m13.167702318s) [upload=332579]: Uploading segments...
Mar 18 19:56:15.570971739 (296h56m13.167742771s) [upload=332579]: Got next segment. Inline: false
Mar 18 19:56:15.571003071 (296h56m13.167774095s) [upload=332579]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Mar 18 19:56:15.571008219 (296h56m13.167779242s) [upload=332579]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Mar 18 19:56:18.374265711 (296h56m15.971036737s) [seg_pos={0 0}, upload=332579]: Begin upload segment...
Mar 18 19:56:18.374352710 (296h56m15.971123742s) [seg_pos={0 0}, upload=332579]: Done begin upload segment.
Mar 18 19:56:18.374418876 (296h56m15.971189909s) [noise=true , address=145.79.227.10:4406 , piece_id=AVYACDBRNXAHGJ5HLLR62AK4QAVHNPHVYRKK7R3V5BCFDN2ZTREQ, seg_pos={0 0}, upload=332579]: Uploading piece...
Mar 18 19:56:18.375175683 (296h56m15.971946697s) [noise=true , address=77.90.31.11:4557 , piece_id=53GXDXJDSVDMHWVJHTONWCVXFKS7GYM3FFQB2QMWSUNAFFHGOZTQ, seg_pos={0 0}, upload=332579]: Uploading piece...
Mar 18 19:56:18.375534468 (296h56m15.972305497s) [noise=true , address=79.127.219.38:4817 , piece_id=HUTLJUUQN2VHMUZJDIXNYV3FXSZDY2W24MEQS5MQW563OWQKH6CQ, seg_pos={0 0}, upload=332579]: Uploading piece...
Mar 18 19:56:18.375671766 (296h56m15.97244278s) [noise=true , address=45.91.116.7:4012 , piece_id=AGK4ME6W7K73MI5Z3UQRRSB455ICRE5DLKD4PTO2VEQALQDC7OZQ, seg_pos={0 0}, upload=332579]: Uploading piece...
Mar 18 19:56:18.375990996 (296h56m15.972762029s) [noise=true , address=145.79.227.23:4548 , piece_id=LJ3UP2KVMCJXOKQMPQK5XVJHAYW42GTMYOMRDRZ2AZM3D3NVUNLQ, seg_pos={0 0}, upload=332579]: Uploading piece...
Mar 18 19:56:18.376099491 (296h56m15.972870517s) [noise=true , address=45.91.116.12:4244 , piece_id=DXOYJYMUSYX2NQQ55Y2AWK6VMEFKUPZ2PMIUQ7GYCPFOS3F3Z6UA, seg_pos={0 0}, upload=332579]: Uploading piece...
Mar 18 19:56:18.376499963 (296h56m15.973270994s) [noise=true , address=145.79.227.2:4116 , piece_id=ZYBBTMDOFSCE5R2J6FNL3XADPMGFGKLW4HXHBTXG2POKLL4P2FAA, seg_pos={0 0}, upload=332579]: Uploading piece...
Mar 18 19:56:18.376511288 (296h56m15.973282319s) [noise=true , address=145.79.227.29:4942 , piece_id=6A5DN6GRUDCRLT2S3GRIWPXKP24OSA3LCPFO3NT7R5I2SDQOXAJQ, seg_pos={0 0}, upload=332579]: Uploading piece...
Mar 18 19:56:18.377912342 (296h56m15.974683406s) [noise=true , address=45.91.116.9:4261 , piece_id=TA32AYRHU3JQQACYRO23HGVKC55AJ6WZ2I3KH2XNA2XHXFJWEDFQ, seg_pos={0 0}, upload=332579]: Uploading piece...
Mar 18 19:56:18.377961597 (296h56m15.974732639s) [noise=true , address=77.90.31.12:4590 , piece_id=UUC5XRLLQPGBUBADYLNXHEJPH57YYHFH4SYRWELRMQ3L2TLFA36A, seg_pos={0 0}, upload=332579]: Uploading piece...
Mar 18 19:56:18.378812880 (296h56m15.975583915s) [noise=true , address=45.91.116.5:4300 , piece_id=HT4VRLQHUXDW37MJBKWSVIAWREYP43KMI4CM6RSBADYOTOLW2LQQ, seg_pos={0 0}, upload=332579]: Uploading piece...
Mar 18 19:56:18.379494838 (296h56m15.97626588s) [noise=true , address=77.90.31.17:4774 , piece_id=A2HO4QESEUCN43FMIB5ZFRKPXFLPQIBYX3HORU5RBH4ZKXSIB6MQ, seg_pos={0 0}, upload=332579]: Uploading piece...
Mar 18 19:56:18.380114365 (296h56m15.976885391s) [noise=true , address=45.91.116.10:4084 , piece_id=K62QNLIVMZKUEE6VXHXY4OXGEL4LXJW365VKQGZMUX2KC3XDFWUQ, seg_pos={0 0}, upload=332579]: Uploading piece...
Mar 18 19:56:18.381227355 (296h56m15.977998399s) [noise=true , address=77.90.31.14:4662 , piece_id=RGND53OAAJIY3SHNGROZJIIR52ADXIY22EI363PBVJ7GH4ZTDR6Q, seg_pos={0 0}, upload=332579]: Uploading piece...
Mar 18 19:56:18.381677238 (296h56m15.978448273s) [noise=true , address=45.91.116.8:4080 , piece_id=PPQQVNBANXNMIAULK5V3LJMFMYV555R4RFFB7NGHMYV6O3UPJV7Q, seg_pos={0 0}, upload=332579]: Uploading piece...
Mar 18 19:56:18.382599321 (296h56m15.979370358s) [noise=true , address=77.90.31.15:4700 , piece_id=VOMDQHAXZI4AWLPBXED3DTFM5N22DWBYU36NWQJNZWN4IRVBTQNA, seg_pos={0 0}, upload=332579]: Uploading piece...
Mar 18 19:56:18.464471837 (296h56m16.06124286s) [upload=332579]: Next returned nil segment
Mar 18 19:56:18.464493175 (296h56m16.061264215s) [upload=332579]: Waiting for error group managing segments...
Mar 18 19:56:18.470438036 (296h56m16.067209063s) [noise=true , address=77.90.31.17:4774 , piece_id=A2HO4QESEUCN43FMIB5ZFRKPXFLPQIBYX3HORU5RBH4ZKXSIB6MQ, seg_pos={0 0}, upload=332579]: Done uploading piece. err: <nil>
Mar 18 19:56:18.472384538 (296h56m16.069155568s) [noise=true , address=79.127.219.38:4817 , piece_id=HUTLJUUQN2VHMUZJDIXNYV3FXSZDY2W24MEQS5MQW563OWQKH6CQ, seg_pos={0 0}, upload=332579]: Done uploading piece. err: <nil>
Mar 18 19:56:18.472524780 (296h56m16.069295805s) [noise=true , address=77.90.31.15:4700 , piece_id=VOMDQHAXZI4AWLPBXED3DTFM5N22DWBYU36NWQJNZWN4IRVBTQNA, seg_pos={0 0}, upload=332579]: Done uploading piece. err: <nil>
Mar 18 19:56:18.474398616 (296h56m16.071169629s) [noise=true , address=45.91.116.12:4244 , piece_id=DXOYJYMUSYX2NQQ55Y2AWK6VMEFKUPZ2PMIUQ7GYCPFOS3F3Z6UA, seg_pos={0 0}, upload=332579]: Done uploading piece. err: <nil>
Mar 18 19:56:18.475256597 (296h56m16.072027611s) [noise=true , address=45.91.116.10:4084 , piece_id=K62QNLIVMZKUEE6VXHXY4OXGEL4LXJW365VKQGZMUX2KC3XDFWUQ, seg_pos={0 0}, upload=332579]: Done uploading piece. err: <nil>
Mar 18 19:56:18.475309163 (296h56m16.072080183s) [noise=true , address=45.91.116.5:4300 , piece_id=HT4VRLQHUXDW37MJBKWSVIAWREYP43KMI4CM6RSBADYOTOLW2LQQ, seg_pos={0 0}, upload=332579]: Done uploading piece. err: <nil>
Mar 18 19:56:18.476104190 (296h56m16.072875217s) [noise=true , address=45.91.116.9:4261 , piece_id=TA32AYRHU3JQQACYRO23HGVKC55AJ6WZ2I3KH2XNA2XHXFJWEDFQ, seg_pos={0 0}, upload=332579]: Done uploading piece. err: <nil>
Mar 18 19:56:18.476777275 (296h56m16.073548299s) [noise=true , address=77.90.31.14:4662 , piece_id=RGND53OAAJIY3SHNGROZJIIR52ADXIY22EI363PBVJ7GH4ZTDR6Q, seg_pos={0 0}, upload=332579]: Done uploading piece. err: <nil>
Mar 18 19:56:18.476810278 (296h56m16.073581288s) [noise=true , address=77.90.31.11:4557 , piece_id=53GXDXJDSVDMHWVJHTONWCVXFKS7GYM3FFQB2QMWSUNAFFHGOZTQ, seg_pos={0 0}, upload=332579]: Done uploading piece. err: <nil>
Mar 18 19:56:18.476867903 (296h56m16.073638915s) [noise=true , address=77.90.31.12:4590 , piece_id=UUC5XRLLQPGBUBADYLNXHEJPH57YYHFH4SYRWELRMQ3L2TLFA36A, seg_pos={0 0}, upload=332579]: Done uploading piece. err: <nil>
Mar 18 19:56:18.477433283 (296h56m16.074204296s) [noise=true , address=45.91.116.8:4080 , piece_id=PPQQVNBANXNMIAULK5V3LJMFMYV555R4RFFB7NGHMYV6O3UPJV7Q, seg_pos={0 0}, upload=332579]: Done uploading piece. err: <nil>
Mar 18 19:56:18.490704585 (296h56m16.087475608s) [noise=true , address=45.91.116.7:4012 , piece_id=AGK4ME6W7K73MI5Z3UQRRSB455ICRE5DLKD4PTO2VEQALQDC7OZQ, seg_pos={0 0}, upload=332579]: Done uploading piece. err: <nil>
Mar 18 19:56:45.537881974 (296h56m43.134653005s) [noise=true , address=145.79.227.23:4548 , piece_id=LJ3UP2KVMCJXOKQMPQK5XVJHAYW42GTMYOMRDRZ2AZM3D3NVUNLQ, seg_pos={0 0}, upload=332579]: Done uploading piece. err: ecclient: upload failed (node:19rx4nU8dUTtPXvMonzZSBkn5Z5cnBXrndvU7t8ewUmfYLBqid, address:145.79.227.23:4548): protocol: rpc: dial tcp 145.79.227.23:4548: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 19:56:45.538293857 (296h56m43.13506489s) [noise=true , address=145.79.227.10:4406 , piece_id=AVYACDBRNXAHGJ5HLLR62AK4QAVHNPHVYRKK7R3V5BCFDN2ZTREQ, seg_pos={0 0}, upload=332579]: Done uploading piece. err: ecclient: upload failed (node:1KrucHjbKjhBZEUGd2p8cmfyCwm6rx4BVwwVhkpgnEv9EQiJRB, address:145.79.227.10:4406): protocol: rpc: dial tcp 145.79.227.10:4406: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 19:56:45.539244445 (296h56m43.136015469s) [noise=true , address=145.79.227.2:4116 , piece_id=ZYBBTMDOFSCE5R2J6FNL3XADPMGFGKLW4HXHBTXG2POKLL4P2FAA, seg_pos={0 0}, upload=332579]: Done uploading piece. err: ecclient: upload failed (node:123aoo9g11aJP9hiL8shu2dx44tjyKzEWqPUjHUdXtR7LLFyng8, address:145.79.227.2:4116): protocol: context deadline exceeded; context deadline exceeded
Mar 18 19:56:45.540701873 (296h56m43.137472903s) [noise=true , address=145.79.227.29:4942 , piece_id=6A5DN6GRUDCRLT2S3GRIWPXKP24OSA3LCPFO3NT7R5I2SDQOXAJQ, seg_pos={0 0}, upload=332579]: Done uploading piece. err: ecclient: upload failed (node:17o881pUMuG7K2dv83k7RkrJCrmM7kgUiWesgam4GEw21KVUn7, address:145.79.227.29:4942): protocol: context deadline exceeded; context deadline exceeded
Mar 18 19:56:45.540740436 (296h56m43.137511459s) [seg_pos={0 0}, upload=332579]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Mar 18 19:56:45.564090338 (296h56m43.160861353s) [upload=332579]: Done uploading segments...
Mar 18 19:56:45.564140425 (296h56m43.160911451s) [upload=332579]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 30.02853724s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
8 out of 60 is failed (threshold=3) [.................................XXXXXX.XX..................] duration:
19.717125195s trace:
https://jaeger.dev.storj.io/trace/4b4d10a8051ad01b
FAIL (2026-03-18 19:56:30 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 19.717183771s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)antiblip:
5 out of 60 is failed (threshold=3) [....................................X..........XX..X....X...] duration:
13.816465615s trace:
https://jaeger.dev.storj.io/trace/6b5565e31dae52f1
FAIL (2026-03-18 19:56:09 UTC)
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 13)
======LOGS======
Mar 18 19:53:19.380863725 (296h53m16.977634736s) [upload=332531]: Starting upload
Mar 18 19:53:19.380944319 (296h53m16.977715346s) [upload=332531]: Done starting upload
Mar 18 19:53:19.380976457 (296h53m16.977747482s) [upload=332531]: Uploading segments...
Mar 18 19:53:19.381333196 (296h53m16.978104222s) [upload=332531]: Got next segment. Inline: false
Mar 18 19:53:19.381379068 (296h53m16.978150099s) [upload=332531]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Mar 18 19:53:19.381386543 (296h53m16.978157579s) [upload=332531]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Mar 18 19:53:23.490845546 (296h53m21.087616603s) [seg_pos={0 0}, upload=332531]: Begin upload segment...
Mar 18 19:53:23.490917951 (296h53m21.087688988s) [seg_pos={0 0}, upload=332531]: Done begin upload segment.
Mar 18 19:53:23.490955926 (296h53m21.087726957s) [noise=true , address=45.91.116.5:4312 , piece_id=BDB6LYZZH2HXC3WWPDVMDQLCAAGRXBM5OCZU2JQVUJ7E6REVJ5SQ, seg_pos={0 0}, upload=332531]: Uploading piece...
Mar 18 19:53:23.492093139 (296h53m21.088864171s) [noise=true , address=193.149.164.67:4003 , piece_id=SOPPFFG22JZWQA5NHYGA2OFHC2KMALZOR3ATBQ6JYWJVXATWLSRQ, seg_pos={0 0}, upload=332531]: Uploading piece...
Mar 18 19:53:23.492835295 (296h53m21.089606329s) [noise=true , address=77.90.31.18:4833 , piece_id=ON3SKTN2H26XEXC43C44MAJIPY32NZLNWRU4PBSRJS6HLMXVNBMA, seg_pos={0 0}, upload=332531]: Uploading piece...
Mar 18 19:53:23.492958664 (296h53m21.089729699s) [noise=true , address=45.91.116.13:4167 , piece_id=3JWBJU4M3WSUM3D2TVZDGJKBSRXZY5ZQHV3GFSBKOJMLORY5W7IQ, seg_pos={0 0}, upload=332531]: Uploading piece...
Mar 18 19:53:23.493624498 (296h53m21.090395533s) [noise=true , address=145.79.227.13:4814 , piece_id=3KU6BHQSH4U4XYBSV4I77FCLQKR5RKWFY3QKZ26D5QRCJDQPKTKQ, seg_pos={0 0}, upload=332531]: Uploading piece...
Mar 18 19:53:23.493650442 (296h53m21.090421482s) [noise=true , address=79.127.205.237:4230 , piece_id=Y6AOKOD27V5RL6QWUY2ZAJMONWPTYU4TTU5VIAYJ4RVMNQMCRUZQ, seg_pos={0 0}, upload=332531]: Uploading piece...
Mar 18 19:53:23.494253839 (296h53m21.091024872s) [noise=true , address=45.91.116.14:4223 , piece_id=X3NSJBVWEZ3F3DUHQOSQSUCECIN7FARCFMRPJR6N7FHIIL3MS3NQ, seg_pos={0 0}, upload=332531]: Uploading piece...
Mar 18 19:53:23.494267618 (296h53m21.091038645s) [noise=true , address=145.79.227.22:4516 , piece_id=OES5UD4FMNBSPC4ZSDS45SRXLTAQT3QNGR6JBI5D5Y363JFGJ73Q, seg_pos={0 0}, upload=332531]: Uploading piece...
Mar 18 19:53:23.494879425 (296h53m21.091650453s) [noise=true , address=193.149.164.71:4287 , piece_id=ZE2Q4NHL3KUKRB36IDRMIZ7DIKDYZIBCM2L3PTLXALGL3PINBGAQ, seg_pos={0 0}, upload=332531]: Uploading piece...
Mar 18 19:53:23.494911912 (296h53m21.091682944s) [noise=true , address=193.149.164.69:4074 , piece_id=FUGRHBCQHA4YJ2EVCMKVQVITADSVLM3PWZHV2XM3QLRDGU4G2GPA, seg_pos={0 0}, upload=332531]: Uploading piece...
Mar 18 19:53:23.495497539 (296h53m21.09226857s) [noise=true , address=193.149.164.68:4066 , piece_id=FY5RBQUD2JCNXJCY2KIK744F4QZFHTW45PBDMASIIP27RWWB7SSA, seg_pos={0 0}, upload=332531]: Uploading piece...
Mar 18 19:53:23.496168841 (296h53m21.092939872s) [noise=true , address=145.79.227.41:4105 , piece_id=RTYHA5YP42CFAYVM6PHY5RTB47GRMWSZEBBPNLC532CXKPBT2VXA, seg_pos={0 0}, upload=332531]: Uploading piece...
Mar 18 19:53:23.497008065 (296h53m21.093779105s) [noise=true , address=79.127.205.235:4271 , piece_id=5XULCXAVIG3CQFTYA6Q3TFPRVICQFHQI2MSI2VIRIAGSK5UKKFXQ, seg_pos={0 0}, upload=332531]: Uploading piece...
Mar 18 19:53:23.497605357 (296h53m21.094376396s) [noise=true , address=145.79.227.37:4590 , piece_id=ZTQLQ2IZHJFZWBHMRT74V7JDZYXF5BG4656VBZDJ4J6NGKS2TMOA, seg_pos={0 0}, upload=332531]: Uploading piece...
Mar 18 19:53:23.498200357 (296h53m21.094971388s) [noise=true , address=193.149.164.66:4989 , piece_id=DVAJPRB7ABJJGUP4TYMLOGJVDLUARAMSPCG6FE2PVQXIDP2QBPGA, seg_pos={0 0}, upload=332531]: Uploading piece...
Mar 18 19:53:23.498757906 (296h53m21.095528939s) [noise=true , address=79.127.219.45:4868 , piece_id=6NDAYV5PFDBOMZKAXKO6LZUODM46MQPQOQIDKBHER3LG4CBEDHKQ, seg_pos={0 0}, upload=332531]: Uploading piece...
Mar 18 19:53:23.499215136 (296h53m21.095986159s) [noise=true , address=145.79.227.52:4453 , piece_id=YRAA65GHFFR45DX7MKR5BWBXID3YVSA4DMQDRKWKRDTJCHKHGMOA, seg_pos={0 0}, upload=332531]: Uploading piece...
Mar 18 19:53:23.499687663 (296h53m21.096458697s) [noise=true , address=77.90.31.14:4678 , piece_id=NWBLAAUF3GACR3Y6JRTHDEUJ24GMOTZIPSRGJIPTPLIVWBCSYAHA, seg_pos={0 0}, upload=332531]: Uploading piece...
Mar 18 19:53:23.600142173 (296h53m21.196913186s) [upload=332531]: Next returned nil segment
Mar 18 19:53:23.600161212 (296h53m21.196932236s) [upload=332531]: Waiting for error group managing segments...
Mar 18 19:53:23.620885875 (296h53m21.217656899s) [noise=true , address=45.91.116.5:4312 , piece_id=BDB6LYZZH2HXC3WWPDVMDQLCAAGRXBM5OCZU2JQVUJ7E6REVJ5SQ, seg_pos={0 0}, upload=332531]: Done uploading piece. err: <nil>
Mar 18 19:53:23.621154396 (296h53m21.217925412s) [noise=true , address=45.91.116.13:4167 , piece_id=3JWBJU4M3WSUM3D2TVZDGJKBSRXZY5ZQHV3GFSBKOJMLORY5W7IQ, seg_pos={0 0}, upload=332531]: Done uploading piece. err: <nil>
Mar 18 19:53:23.628883995 (296h53m21.225655021s) [noise=true , address=45.91.116.14:4223 , piece_id=X3NSJBVWEZ3F3DUHQOSQSUCECIN7FARCFMRPJR6N7FHIIL3MS3NQ, seg_pos={0 0}, upload=332531]: Done uploading piece. err: <nil>
Mar 18 19:53:23.630970813 (296h53m21.227741838s) [noise=true , address=79.127.219.45:4868 , piece_id=6NDAYV5PFDBOMZKAXKO6LZUODM46MQPQOQIDKBHER3LG4CBEDHKQ, seg_pos={0 0}, upload=332531]: Done uploading piece. err: <nil>
Mar 18 19:53:23.639825090 (296h53m21.236596104s) [noise=true , address=77.90.31.14:4678 , piece_id=NWBLAAUF3GACR3Y6JRTHDEUJ24GMOTZIPSRGJIPTPLIVWBCSYAHA, seg_pos={0 0}, upload=332531]: Done uploading piece. err: <nil>
Mar 18 19:53:23.642754677 (296h53m21.239525692s) [noise=true , address=77.90.31.18:4833 , piece_id=ON3SKTN2H26XEXC43C44MAJIPY32NZLNWRU4PBSRJS6HLMXVNBMA, seg_pos={0 0}, upload=332531]: Done uploading piece. err: <nil>
Mar 18 19:53:24.057422272 (296h53m21.654193306s) [noise=true , address=79.127.205.235:4271 , piece_id=5XULCXAVIG3CQFTYA6Q3TFPRVICQFHQI2MSI2VIRIAGSK5UKKFXQ, seg_pos={0 0}, upload=332531]: Done uploading piece. err: <nil>
Mar 18 19:53:24.074244577 (296h53m21.671015604s) [noise=true , address=193.149.164.67:4003 , piece_id=SOPPFFG22JZWQA5NHYGA2OFHC2KMALZOR3ATBQ6JYWJVXATWLSRQ, seg_pos={0 0}, upload=332531]: Done uploading piece. err: <nil>
Mar 18 19:53:24.097678734 (296h53m21.694449747s) [noise=true , address=79.127.205.237:4230 , piece_id=Y6AOKOD27V5RL6QWUY2ZAJMONWPTYU4TTU5VIAYJ4RVMNQMCRUZQ, seg_pos={0 0}, upload=332531]: Done uploading piece. err: <nil>
Mar 18 19:53:24.109685982 (296h53m21.706457009s) [noise=true , address=193.149.164.68:4066 , piece_id=FY5RBQUD2JCNXJCY2KIK744F4QZFHTW45PBDMASIIP27RWWB7SSA, seg_pos={0 0}, upload=332531]: Done uploading piece. err: <nil>
Mar 18 19:53:24.110659716 (296h53m21.707430744s) [noise=true , address=193.149.164.69:4074 , piece_id=FUGRHBCQHA4YJ2EVCMKVQVITADSVLM3PWZHV2XM3QLRDGU4G2GPA, seg_pos={0 0}, upload=332531]: Done uploading piece. err: <nil>
Mar 18 19:53:24.114541274 (296h53m21.7113123s) [noise=true , address=193.149.164.71:4287 , piece_id=ZE2Q4NHL3KUKRB36IDRMIZ7DIKDYZIBCM2L3PTLXALGL3PINBGAQ, seg_pos={0 0}, upload=332531]: Done uploading piece. err: <nil>
Mar 18 19:53:24.140894470 (296h53m21.737665488s) [noise=true , address=193.149.164.66:4989 , piece_id=DVAJPRB7ABJJGUP4TYMLOGJVDLUARAMSPCG6FE2PVQXIDP2QBPGA, seg_pos={0 0}, upload=332531]: Done uploading piece. err: <nil>
Mar 18 19:54:19.329327996 (296h54m16.926099018s) [noise=true , address=145.79.227.13:4814 , piece_id=3KU6BHQSH4U4XYBSV4I77FCLQKR5RKWFY3QKZ26D5QRCJDQPKTKQ, seg_pos={0 0}, upload=332531]: Done uploading piece. err: ecclient: upload failed (node:12ERtrzMuh8rZPsgebtyScmYvNVwz3UKJr5Nu2hLdu6sn8X3qLa, address:145.79.227.13:4814): protocol: rpc: dial tcp 145.79.227.13:4814: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 19:54:19.329567681 (296h54m16.926338706s) [noise=true , address=145.79.227.52:4453 , piece_id=YRAA65GHFFR45DX7MKR5BWBXID3YVSA4DMQDRKWKRDTJCHKHGMOA, seg_pos={0 0}, upload=332531]: Done uploading piece. err: ecclient: upload failed (node:1BmpsQ9VfhGzmkZHvSeMptcGhzGpdoN4oYvi83qZcC1Vh9anhP, address:145.79.227.52:4453): protocol: rpc: dial tcp 145.79.227.52:4453: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 19:54:19.329755822 (296h54m16.926526848s) [noise=true , address=145.79.227.41:4105 , piece_id=RTYHA5YP42CFAYVM6PHY5RTB47GRMWSZEBBPNLC532CXKPBT2VXA, seg_pos={0 0}, upload=332531]: Done uploading piece. err: ecclient: upload failed (node:1TVtMQNVV2kERp2aZwmS8E9JUiua1RCyFpZgFACRqH1Nfscw4C, address:145.79.227.41:4105): protocol: rpc: dial tcp 145.79.227.41:4105: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 19:54:19.330021146 (296h54m16.926792175s) [noise=true , address=145.79.227.22:4516 , piece_id=OES5UD4FMNBSPC4ZSDS45SRXLTAQT3QNGR6JBI5D5Y363JFGJ73Q, seg_pos={0 0}, upload=332531]: Done uploading piece. err: ecclient: upload failed (node:12Lhy3AxzYoF25AxWdvjrrV5NNCwP1cDqkEvv4tz9t9KZmF8WPK, address:145.79.227.22:4516): protocol: rpc: dial tcp 145.79.227.22:4516: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 19:54:19.330213680 (296h54m16.926984707s) [noise=true , address=145.79.227.37:4590 , piece_id=ZTQLQ2IZHJFZWBHMRT74V7JDZYXF5BG4656VBZDJ4J6NGKS2TMOA, seg_pos={0 0}, upload=332531]: Done uploading piece. err: ecclient: upload failed (node:1KnNqY1WNwMSFXFrQsWPAhivUTgSn7dKTx1mTTzYG2UxGCkasj, address:145.79.227.37:4590): protocol: rpc: dial tcp 145.79.227.37:4590: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 19:54:19.330250832 (296h54m16.927021848s) [seg_pos={0 0}, upload=332531]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil>
Mar 18 19:54:19.351979131 (296h54m16.948750155s) [upload=332531]: Done uploading segments...
Mar 18 19:54:19.352025297 (296h54m16.948796327s) [upload=332531]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13)
Duration: 13.81651104s
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
27 out of 60 is failed (threshold=3) [.................................XXXXXXXXXXXXXXXXXXXXXXXXXXX] duration:
30.049412374s trace:
https://jaeger.dev.storj.io/trace/2158127507e5a11c
FAIL (2026-03-18 19:56:04 UTC)
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 7)
======LOGS======
) [upload=332569]: Uploading segments...
Mar 18 19:55:34.531106601 (296h55m32.127877625s) [upload=332569]: Got next segment. Inline: false
Mar 18 19:55:34.531173958 (296h55m32.127944986s) [upload=332569]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Mar 18 19:55:34.531181730 (296h55m32.127952756s) [upload=332569]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Mar 18 19:55:37.158514722 (296h55m34.75528578s) [seg_pos={0 0}, upload=332569]: Begin upload segment...
Mar 18 19:55:37.158584096 (296h55m34.755355129s) [seg_pos={0 0}, upload=332569]: Done begin upload segment.
Mar 18 19:55:37.158628676 (296h55m34.755399704s) [noise=true , address=145.79.227.3:4149 , piece_id=GJV52NALFMBDQNBUDBQ7F6R27KODRWJGWMTHTUYZZX3J3PSUSTCQ, seg_pos={0 0}, upload=332569]: Uploading piece...
Mar 18 19:55:37.158661004 (296h55m34.755432013s) [noise=true , address=77.90.31.15:4702 , piece_id=UFRX37JYI4BZFLVCNGPPSNFGBVCUN4WXWJSWNLAAC4QSFRDQWMSQ, seg_pos={0 0}, upload=332569]: Uploading piece...
Mar 18 19:55:37.159521878 (296h55m34.756292912s) [noise=true , address=145.79.227.40:4050 , piece_id=LOIYTIDRPCTUCLOKUKGVAGOJELZTEGW6JCMPDL4YMYJH6PZXHS2Q, seg_pos={0 0}, upload=332569]: Uploading piece...
Mar 18 19:55:37.159808970 (296h55m34.756579993s) [noise=true , address=145.79.227.21:4476 , piece_id=ANT7HNYV2OOJKQQU64YFBMEW2EDE7WHQIE73BL6STYW3EQK24MCA, seg_pos={0 0}, upload=332569]: Uploading piece...
Mar 18 19:55:37.160079412 (296h55m34.756850445s) [noise=true , address=145.79.227.8:4334 , piece_id=453YINSLC7FELDNDYNTY7UMIFBLIDQEOOQNNILDTL65NQBGGORMA, seg_pos={0 0}, upload=332569]: Uploading piece...
Mar 18 19:55:37.160451644 (296h55m34.757222685s) [noise=true , address=145.79.227.29:4942 , piece_id=NMQP2PRD7WZ6YNG4W332HDAH7D3LHBWWER7EJEORQSMD22ZPTZ4A, seg_pos={0 0}, upload=332569]: Uploading piece...
Mar 18 19:55:37.160521737 (296h55m34.757292767s) [noise=true , address=45.91.116.8:4046 , piece_id=Y4T345NY7DKOFXHHLOQYIL3X62J6UPWY34C5U22GED6W7DDUWUWQ, seg_pos={0 0}, upload=332569]: Uploading piece...
Mar 18 19:55:37.160957031 (296h55m34.757728067s) [noise=true , address=145.79.227.19:4083 , piece_id=ABKPTUIONCIGCQY63Q7IUSHZ6YLEFOVSTJMKPTKJLMRLPOQZ3DMA, seg_pos={0 0}, upload=332569]: Uploading piece...
Mar 18 19:55:37.161112533 (296h55m34.757883559s) [noise=true , address=45.91.116.14:4189 , piece_id=7Q7AJIS6GQAZMFMREFACCA6EJT3WQTPJ7UGEAXHOD6AT47MMBNGA, seg_pos={0 0}, upload=332569]: Uploading piece...
Mar 18 19:55:37.161404921 (296h55m34.758175951s) [noise=true , address=145.79.227.49:4338 , piece_id=WKQGLT5DYQLY2QHC4YBXQT22WU4U6RKWYXX3C6SG25XANVAWRYCA, seg_pos={0 0}, upload=332569]: Uploading piece...
Mar 18 19:55:37.161654399 (296h55m34.758425414s) [noise=true , address=45.91.116.6:4989 , piece_id=LNVBGOQV6VOM4W7CAEXNOXP2K4DD5MLJO5DRCHHNZNS2JN7VT4IA, seg_pos={0 0}, upload=332569]: Uploading piece...
Mar 18 19:55:37.161775273 (296h55m34.758546303s) [noise=true , address=45.91.116.10:4088 , piece_id=6DTCYTSZ5ZHZFGJCM5VAVOGB53LLBY2KX2CY5JQWR3A5Y7FSTQ4A, seg_pos={0 0}, upload=332569]: Uploading piece...
Mar 18 19:55:37.162143747 (296h55m34.758914775s) [noise=true , address=145.79.227.39:4017 , piece_id=CJ5BKM6JFJHZTTC4TIMP3K2AIJ7FFFNCCXAXFUV6CV6GFENRJFQA, seg_pos={0 0}, upload=332569]: Uploading piece...
Mar 18 19:55:37.162523408 (296h55m34.759294434s) [noise=true , address=45.91.116.5:4299 , piece_id=WK2QGS2CXUOXTTLYG5SKYHRSD6VAT33Y33C4I5SVLUHBBTPPJXKQ, seg_pos={0 0}, upload=332569]: Uploading piece...
Mar 18 19:55:37.162894535 (296h55m34.759665546s) [noise=true , address=145.79.227.2:4116 , piece_id=6JP2ON64ZCKZWRQ2AEXG4MP2FG7HOG47X2KK76CZLSUWDQRIAMMQ, seg_pos={0 0}, upload=332569]: Uploading piece...
Mar 18 19:55:37.164317891 (296h55m34.761088912s) [noise=true , address=77.90.31.18:4808 , piece_id=XOXTQUS6BEV6HQXEI2PLKQ7J7F2J2HK7CWOQP25Y7WU7MDBY5UDA, seg_pos={0 0}, upload=332569]: Uploading piece...
Mar 18 19:55:37.234747636 (296h55m34.831518653s) [upload=332569]: Next returned nil segment
Mar 18 19:55:37.234763264 (296h55m34.831534287s) [upload=332569]: Waiting for error group managing segments...
Mar 18 19:55:37.239679635 (296h55m34.836450659s) [noise=true , address=45.91.116.8:4046 , piece_id=Y4T345NY7DKOFXHHLOQYIL3X62J6UPWY34C5U22GED6W7DDUWUWQ, seg_pos={0 0}, upload=332569]: Done uploading piece. err: <nil>
Mar 18 19:55:37.240295182 (296h55m34.837066194s) [noise=true , address=45.91.116.6:4989 , piece_id=LNVBGOQV6VOM4W7CAEXNOXP2K4DD5MLJO5DRCHHNZNS2JN7VT4IA, seg_pos={0 0}, upload=332569]: Done uploading piece. err: <nil>
Mar 18 19:55:37.240831973 (296h55m34.837603003s) [noise=true , address=77.90.31.15:4702 , piece_id=UFRX37JYI4BZFLVCNGPPSNFGBVCUN4WXWJSWNLAAC4QSFRDQWMSQ, seg_pos={0 0}, upload=332569]: Done uploading piece. err: <nil>
Mar 18 19:55:37.241279426 (296h55m34.838050439s) [noise=true , address=45.91.116.5:4299 , piece_id=WK2QGS2CXUOXTTLYG5SKYHRSD6VAT33Y33C4I5SVLUHBBTPPJXKQ, seg_pos={0 0}, upload=332569]: Done uploading piece. err: <nil>
Mar 18 19:55:37.241963490 (296h55m34.8387345s) [noise=true , address=77.90.31.18:4808 , piece_id=XOXTQUS6BEV6HQXEI2PLKQ7J7F2J2HK7CWOQP25Y7WU7MDBY5UDA, seg_pos={0 0}, upload=332569]: Done uploading piece. err: <nil>
Mar 18 19:55:37.242437010 (296h55m34.839208023s) [noise=true , address=45.91.116.10:4088 , piece_id=6DTCYTSZ5ZHZFGJCM5VAVOGB53LLBY2KX2CY5JQWR3A5Y7FSTQ4A, seg_pos={0 0}, upload=332569]: Done uploading piece. err: <nil>
Mar 18 19:55:37.259770981 (296h55m34.856542009s) [noise=true , address=45.91.116.14:4189 , piece_id=7Q7AJIS6GQAZMFMREFACCA6EJT3WQTPJ7UGEAXHOD6AT47MMBNGA, seg_pos={0 0}, upload=332569]: Done uploading piece. err: <nil>
Mar 18 19:56:04.490244630 (296h56m2.087015664s) [noise=true , address=145.79.227.39:4017 , piece_id=CJ5BKM6JFJHZTTC4TIMP3K2AIJ7FFFNCCXAXFUV6CV6GFENRJFQA, seg_pos={0 0}, upload=332569]: Done uploading piece. err: ecclient: upload failed (node:12ocB4w2b2owjyWVCcq59Ymo2prJfSFmK5YogQDjGBroF11Tegk, address:145.79.227.39:4017): protocol: rpc: dial tcp 145.79.227.39:4017: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 19:56:04.490651032 (296h56m2.087422065s) [noise=true , address=145.79.227.8:4334 , piece_id=453YINSLC7FELDNDYNTY7UMIFBLIDQEOOQNNILDTL65NQBGGORMA, seg_pos={0 0}, upload=332569]: Done uploading piece. err: ecclient: upload failed (node:12KYMZAFsJbhWC4CdKRod2Z6qsC6ANcpqmpWSWx7UYajp6JvKH1, address:145.79.227.8:4334): protocol: rpc: dial tcp 145.79.227.8:4334: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 19:56:04.491142597 (296h56m2.087913631s) [noise=true , address=145.79.227.49:4338 , piece_id=WKQGLT5DYQLY2QHC4YBXQT22WU4U6RKWYXX3C6SG25XANVAWRYCA, seg_pos={0 0}, upload=332569]: Done uploading piece. err: ecclient: upload failed (node:1cTadUV1Zu2Z8MoobXvyZ965fv6jPFhYGsy5qgRZHgWeSwyitN, address:145.79.227.49:4338): protocol: rpc: dial tcp 145.79.227.49:4338: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 19:56:04.491674230 (296h56m2.088445263s) [noise=true , address=145.79.227.21:4476 , piece_id=ANT7HNYV2OOJKQQU64YFBMEW2EDE7WHQIE73BL6STYW3EQK24MCA, seg_pos={0 0}, upload=332569]: Done uploading piece. err: ecclient: upload failed (node:12NKwRvjiEpY7DrbqFGaDHQ8no5YV4cUALnpsvQ8icUPoLvMYen, address:145.79.227.21:4476): protocol: rpc: dial tcp 145.79.227.21:4476: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 19:56:04.492104226 (296h56m2.088875259s) [noise=true , address=145.79.227.3:4149 , piece_id=GJV52NALFMBDQNBUDBQ7F6R27KODRWJGWMTHTUYZZX3J3PSUSTCQ, seg_pos={0 0}, upload=332569]: Done uploading piece. err: ecclient: upload failed (node:12qexRLJM6TY935UaEWsj1X9ViyBvk2BM1XGjnq8kapnXXXmgbm, address:145.79.227.3:4149): protocol: rpc: dial tcp 145.79.227.3:4149: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 19:56:04.492422815 (296h56m2.089193848s) [noise=true , address=145.79.227.40:4050 , piece_id=LOIYTIDRPCTUCLOKUKGVAGOJELZTEGW6JCMPDL4YMYJH6PZXHS2Q, seg_pos={0 0}, upload=332569]: Done uploading piece. err: ecclient: upload failed (node:12Vw9PtVFLQPpm2uGV55tCoK7VfopaBU4EcWNBUyqhNeJjgjG64, address:145.79.227.40:4050): protocol: rpc: dial tcp 145.79.227.40:4050: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 19:56:04.492857380 (296h56m2.089628418s) [noise=true , address=145.79.227.29:4942 , piece_id=NMQP2PRD7WZ6YNG4W332HDAH7D3LHBWWER7EJEORQSMD22ZPTZ4A, seg_pos={0 0}, upload=332569]: Done uploading piece. err: ecclient: upload failed (node:17o881pUMuG7K2dv83k7RkrJCrmM7kgUiWesgam4GEw21KVUn7, address:145.79.227.29:4942): protocol: rpc: dial tcp 145.79.227.29:4942: i/o timeout; conn closed: op
Mar 18 19:56:04.493016741 (296h56m2.089787775s) [noise=true , address=145.79.227.2:4116 , piece_id=6JP2ON64ZCKZWRQ2AEXG4MP2FG7HOG47X2KK76CZLSUWDQRIAMMQ, seg_pos={0 0}, upload=332569]: Done uploading piece. err: ecclient: upload failed (node:123aoo9g11aJP9hiL8shu2dx44tjyKzEWqPUjHUdXtR7LLFyng8, address:145.79.227.2:4116): protocol: rpc: dial tcp 145.79.227.2:4116: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 19:56:04.493169341 (296h56m2.089940386s) [noise=true , address=145.79.227.19:4083 , piece_id=ABKPTUIONCIGCQY63Q7IUSHZ6YLEFOVSTJMKPTKJLMRLPOQZ3DMA, seg_pos={0 0}, upload=332569]: Done uploading piece. err: ecclient: upload failed (node:12KZ5a6CQ9tUTrC6EYE7M8yg6p9fK95CN8zQLQmP4tdyz4rBfGh, address:145.79.227.19:4083): protocol: rpc: dial tcp 145.79.227.19:4083: i/o timeout; conn closed: op; piecestore: piecestore close: context deadline exceeded
Mar 18 19:56:04.493222705 (296h56m2.089993715s) [seg_pos={0 0}, upload=332569]: Done waiting for segment. successful: 7 optimal: 13 errs: <nil>
Mar 18 19:56:04.537336436 (296h56m2.134107554s) [upload=332569]: Done uploading segments...
Mar 18 19:56:04.537382081 (296h56m2.134153114s) [upload=332569]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 7)
Duration: 30.049439558s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
8 out of 60 is failed (threshold=3) [..................................XXXXXX.XX.................] duration:
20.408002156s trace:
https://jaeger.dev.storj.io/trace/4b4d10a8051ad01b
FAIL (2026-03-18 19:56:00 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 20.408033885s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
34 out of 60 is failed (threshold=6) [......................XXXXXXXXX.XXX...XXXXXXXXXXXXXXXXXXXXXX] duration:
5.001744418s trace:
https://jaeger.dev.storj.io/trace/3ed8ebfab3f4cbae
FAIL (2026-03-18 19:55:58 UTC)
operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.00202576s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
5 out of 60 is failed (threshold=3) [...................................XXX..........X...X.......] duration:
19.299810725s trace:
https://jaeger.dev.storj.io/trace/6a01088813e3d8b7
FAIL (2026-03-18 19:55:57 UTC)
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 12)
======LOGS======
m: *pb.BatchRequestItem_ObjectBegin
Mar 18 19:49:08.138517471 (296h49m5.735288494s) [upload=332459]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Mar 18 19:49:16.284286759 (296h49m13.881057773s) [seg_pos={0 0}, upload=332459]: Begin upload segment...
Mar 18 19:49:16.284352223 (296h49m13.88112325s) [seg_pos={0 0}, upload=332459]: Done begin upload segment.
Mar 18 19:49:16.284395789 (296h49m13.881166818s) [noise=true , address=193.149.164.68:4051 , piece_id=FCHPX2FRYUVNH4JAAPKI2Y7DSJCUJ5UG3EASXQYB7VEQPNERYUGQ, seg_pos={0 0}, upload=332459]: Uploading piece...
Mar 18 19:49:16.284449689 (296h49m13.881220728s) [noise=true , address=145.79.227.13:4822 , piece_id=Y5GSC3KFFGLK4B3NCXEMOZTAC3ASNGPJJOPOKCUBUJIL75E6T6FA, seg_pos={0 0}, upload=332459]: Uploading piece...
Mar 18 19:49:16.284549974 (296h49m13.881320988s) [noise=true , address=45.91.116.5:4308 , piece_id=XRXYA347E22AHAJ4XDENA5EHCV2F43HXL67MTUGRYSU6B3WQBDNQ, seg_pos={0 0}, upload=332459]: Uploading piece...
Mar 18 19:49:16.285144837 (296h49m13.88191587s) [noise=true , address=193.149.164.70:4117 , piece_id=PL6FWCO6XQRSVU26FH2KESA26D4Z2PCQVBPBDKBWLBPXHBV5LQVQ, seg_pos={0 0}, upload=332459]: Uploading piece...
Mar 18 19:49:16.285821284 (296h49m13.882592317s) [noise=true , address=145.79.227.52:4451 , piece_id=SBMEAZS5RDQW3RCJP2UU5T7BHF6P7VOICLP5P46YJ5LE2GB65CGA, seg_pos={0 0}, upload=332459]: Uploading piece...
Mar 18 19:49:16.286556627 (296h49m13.883327669s) [noise=true , address=79.127.205.235:4259 , piece_id=N7BGDCO2CR6HLPUWYF35T5OQRWQFEWFG7FPZ6Y33J7FGR5H2ZURQ, seg_pos={0 0}, upload=332459]: Uploading piece...
Mar 18 19:49:16.287228779 (296h49m13.883999813s) [noise=true , address=79.127.219.45:4864 , piece_id=UXHLRN5HUN3ZRXOZEYAJOS6QUN3PVIXXXED3TS6RPWSHXJ4WKS6Q, seg_pos={0 0}, upload=332459]: Uploading piece...
Mar 18 19:49:16.287269957 (296h49m13.884040985s) [noise=true , address=77.90.31.21:4894 , piece_id=IQ6775I4HFC5JQVYUCQPXYWPY3X6QRHVXM6IF4JEZRQMMRRWQDUQ, seg_pos={0 0}, upload=332459]: Uploading piece...
Mar 18 19:49:16.287437026 (296h49m13.884208049s) [noise=true , address=5.231.73.4:4010 , piece_id=LTONWX7UJVP3BPHHKKJUN6PTAR5TVPVKF3CD6NBV2MZWTLUOVQZQ, seg_pos={0 0}, upload=332459]: Uploading piece...
Mar 18 19:49:16.287832890 (296h49m13.884603914s) [noise=true , address=167.17.72.68:4657 , piece_id=UUIV3ERJSVOJT4TISEHCMR6EQCG6PMZUR6OWGK3ZVV33B33CUZCA, seg_pos={0 0}, upload=332459]: Uploading piece...
Mar 18 19:49:16.287952922 (296h49m13.884723957s) [noise=true , address=79.127.219.41:4939 , piece_id=WK6HMSTVM7LD2P5MPCKML4MBFZJX2QJWZS6SYPITDKNOLPMRPO6Q, seg_pos={0 0}, upload=332459]: Uploading piece...
Mar 18 19:49:16.288418812 (296h49m13.885189842s) [noise=true , address=145.79.227.44:4213 , piece_id=MK3BJSFCDOG2MAJYO2BAE2JG62EVEZMCCUATBBBT4QU44GLKK3QA, seg_pos={0 0}, upload=332459]: Uploading piece...
Mar 18 19:49:16.288922438 (296h49m13.885693461s) [noise=true , address=145.79.227.3:4163 , piece_id=3R3QBIJ2XDQZQXJ66RGNZBWZB6V4ERS5RJEZFBTS2LFMQ25G4FLA, seg_pos={0 0}, upload=332459]: Uploading piece...
Mar 18 19:49:16.289343580 (296h49m13.886114604s) [noise=true , address=193.149.164.71:4310 , piece_id=BKUA4WC73DSXJMEPCMTXTUD2PVQAAYD44MAGEB3SS3FEDJO7377A, seg_pos={0 0}, upload=332459]: Uploading piece...
Mar 18 19:49:16.289780590 (296h49m13.88655162s) [noise=true , address=145.79.227.49:4358 , piece_id=QSE44UR5YBYTJHWGOCBFVQPAUZ3SJV6P5WKBEF3MTFAKVRH2QSHQ, seg_pos={0 0}, upload=332459]: Uploading piece...
Mar 18 19:49:16.290237004 (296h49m13.887008027s) [noise=true , address=77.90.31.14:4685 , piece_id=3KAIZVB2LH6R57CFO55R7BJV7G4MGHFN6M5VTKT3J6LMI5F7PVAA, seg_pos={0 0}, upload=332459]: Uploading piece...
Mar 18 19:49:16.292879809 (296h49m13.889650843s) [noise=true , address=77.90.31.18:4817 , piece_id=FMVKZ4Z3YDPWZHTXU3ZTSWXMVUWLKQX7TUWXPQ4LOKBQK64PHF5Q, seg_pos={0 0}, upload=332459]: Uploading piece...
Mar 18 19:49:16.293664670 (296h49m13.890435693s) [noise=true , address=145.79.227.15:4908 , piece_id=LG24OMYBUVPVRBFZL4UUJLPBVNDTVVHQ4HAYTW2PGKHUKZPF2WCA, seg_pos={0 0}, upload=332459]: Uploading piece...
Mar 18 19:49:16.351440712 (296h49m13.948211724s) [upload=332459]: Next returned nil segment
Mar 18 19:49:16.351460639 (296h49m13.948231661s) [upload=332459]: Waiting for error group managing segments...
Mar 18 19:49:16.355941640 (296h49m13.952712664s) [noise=true , address=77.90.31.21:4894 , piece_id=IQ6775I4HFC5JQVYUCQPXYWPY3X6QRHVXM6IF4JEZRQMMRRWQDUQ, seg_pos={0 0}, upload=332459]: Done uploading piece. err: <nil>
Mar 18 19:49:16.356991929 (296h49m13.953762955s) [noise=true , address=79.127.219.41:4939 , piece_id=WK6HMSTVM7LD2P5MPCKML4MBFZJX2QJWZS6SYPITDKNOLPMRPO6Q, seg_pos={0 0}, upload=332459]: Done uploading piece. err: <nil>
Mar 18 19:49:16.361809763 (296h49m13.958580786s) [noise=true , address=45.91.116.5:4308 , piece_id=XRXYA347E22AHAJ4XDENA5EHCV2F43HXL67MTUGRYSU6B3WQBDNQ, seg_pos={0 0}, upload=332459]: Done uploading piece. err: <nil>
Mar 18 19:49:16.367502365 (296h49m13.964273396s) [noise=true , address=77.90.31.14:4685 , piece_id=3KAIZVB2LH6R57CFO55R7BJV7G4MGHFN6M5VTKT3J6LMI5F7PVAA, seg_pos={0 0}, upload=332459]: Done uploading piece. err: <nil>
Mar 18 19:49:16.381549888 (296h49m13.97832091s) [noise=true , address=79.127.219.45:4864 , piece_id=UXHLRN5HUN3ZRXOZEYAJOS6QUN3PVIXXXED3TS6RPWSHXJ4WKS6Q, seg_pos={0 0}, upload=332459]: Done uploading piece. err: <nil>
Mar 18 19:49:16.387503503 (296h49m13.984274527s) [noise=true , address=77.90.31.18:4817 , piece_id=FMVKZ4Z3YDPWZHTXU3ZTSWXMVUWLKQX7TUWXPQ4LOKBQK64PHF5Q, seg_pos={0 0}, upload=332459]: Done uploading piece. err: <nil>
Mar 18 19:49:16.498139242 (296h49m14.094910266s) [noise=true , address=5.231.73.4:4010 , piece_id=LTONWX7UJVP3BPHHKKJUN6PTAR5TVPVKF3CD6NBV2MZWTLUOVQZQ, seg_pos={0 0}, upload=332459]: Done uploading piece. err: <nil>
Mar 18 19:49:16.852944491 (296h49m14.449715518s) [noise=true , address=167.17.72.68:4657 , piece_id=UUIV3ERJSVOJT4TISEHCMR6EQCG6PMZUR6OWGK3ZVV33B33CUZCA, seg_pos={0 0}, upload=332459]: Done uploading piece. err: <nil>
Mar 18 19:49:16.878068834 (296h49m14.474839867s) [noise=true , address=79.127.205.235:4259 , piece_id=N7BGDCO2CR6HLPUWYF35T5OQRWQFEWFG7FPZ6Y33J7FGR5H2ZURQ, seg_pos={0 0}, upload=332459]: Done uploading piece. err: <nil>
Mar 18 19:49:16.892446253 (296h49m14.489217282s) [noise=true , address=193.149.164.68:4051 , piece_id=FCHPX2FRYUVNH4JAAPKI2Y7DSJCUJ5UG3EASXQYB7VEQPNERYUGQ, seg_pos={0 0}, upload=332459]: Done uploading piece. err: <nil>
Mar 18 19:49:16.893099142 (296h49m14.48987017s) [noise=true , address=193.149.164.71:4310 , piece_id=BKUA4WC73DSXJMEPCMTXTUD2PVQAAYD44MAGEB3SS3FEDJO7377A, seg_pos={0 0}, upload=332459]: Done uploading piece. err: <nil>
Mar 18 19:49:17.031110208 (296h49m14.627881232s) [noise=true , address=193.149.164.70:4117 , piece_id=PL6FWCO6XQRSVU26FH2KESA26D4Z2PCQVBPBDKBWLBPXHBV5LQVQ, seg_pos={0 0}, upload=332459]: Done uploading piece. err: <nil>
Mar 18 19:50:08.096628364 (296h50m5.693399398s) [noise=true , address=145.79.227.52:4451 , piece_id=SBMEAZS5RDQW3RCJP2UU5T7BHF6P7VOICLP5P46YJ5LE2GB65CGA, seg_pos={0 0}, upload=332459]: Done uploading piece. err: ecclient: upload failed (node:12bC4P3xBiPFkxLvavGFVbWKnbprCStL6UJgfKSA2Wd2fX7Mbfw, address:145.79.227.52:4451): protocol: rpc: dial tcp 145.79.227.52:4451: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 19:50:08.096712565 (296h50m5.693483599s) [noise=true , address=145.79.227.44:4213 , piece_id=MK3BJSFCDOG2MAJYO2BAE2JG62EVEZMCCUATBBBT4QU44GLKK3QA, seg_pos={0 0}, upload=332459]: Done uploading piece. err: ecclient: upload failed (node:129LRnDm7b4Fo5db7pto8GXYEtxHz9qdPAtQvaYH6gqdkMatQ6g, address:145.79.227.44:4213): protocol: rpc: dial tcp 145.79.227.44:4213: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 19:50:08.096943717 (296h50m5.693714752s) [noise=true , address=145.79.227.49:4358 , piece_id=QSE44UR5YBYTJHWGOCBFVQPAUZ3SJV6P5WKBEF3MTFAKVRH2QSHQ, seg_pos={0 0}, upload=332459]: Done uploading piece. err: ecclient: upload failed (node:1dwCNEcA9o51ZN3UaN3Frxdawwf8DXBj86iQMywjW8XP4ESmw1, address:145.79.227.49:4358): protocol: rpc: dial tcp 145.79.227.49:4358: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 19:50:08.097022451 (296h50m5.693793484s) [noise=true , address=145.79.227.3:4163 , piece_id=3R3QBIJ2XDQZQXJ66RGNZBWZB6V4ERS5RJEZFBTS2LFMQ25G4FLA, seg_pos={0 0}, upload=332459]: Done uploading piece. err: ecclient: upload failed (node:12w88UkGu5duYivJ6L6C9iJmTbnB63fDKiUUQprhwM3JshpXTdC, address:145.79.227.3:4163): protocol: rpc: dial tcp 145.79.227.3:4163: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 19:50:08.097155433 (296h50m5.693926468s) [noise=true , address=145.79.227.13:4822 , piece_id=Y5GSC3KFFGLK4B3NCXEMOZTAC3ASNGPJJOPOKCUBUJIL75E6T6FA, seg_pos={0 0}, upload=332459]: Done uploading piece. err: ecclient: upload failed (node:12fgLN9DGBGxWfBEePSsk84FTrnKoKWwNgoET3FpotfJxJ28f75, address:145.79.227.13:4822): protocol: rpc: dial tcp 145.79.227.13:4822: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 19:50:08.097360007 (296h50m5.694131039s) [noise=true , address=145.79.227.15:4908 , piece_id=LG24OMYBUVPVRBFZL4UUJLPBVNDTVVHQ4HAYTW2PGKHUKZPF2WCA, seg_pos={0 0}, upload=332459]: Done uploading piece. err: ecclient: upload failed (node:1pfQ5v1GwHhFTvh5Y7Jpb9qpWhZhFYqWX9PJpLEUzTkqKWcy4x, address:145.79.227.15:4908): protocol: rpc: dial tcp 145.79.227.15:4908: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 19:50:08.097462766 (296h50m5.69423378s) [seg_pos={0 0}, upload=332459]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil>
Mar 18 19:50:08.117913456 (296h50m5.714684468s) [upload=332459]: Done uploading segments...
Mar 18 19:50:08.117999321 (296h50m5.714770348s) [upload=332459]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12)
Duration: 19.299853115s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
33 out of 60 is failed (threshold=6) [.......................XXXXXXXXX.XXX...XXXXXXXXXXXXXXXXXXXXX] duration:
5.00230415s trace:
https://jaeger.dev.storj.io/trace/6cc41d13fdeed07a
FAIL (2026-03-18 19:55:32 UTC)
operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.002421156s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)antiblip:
5 out of 60 is failed (threshold=3) [.....................................X..........XX..X....X..] duration:
15.049910935s trace:
https://jaeger.dev.storj.io/trace/6b5565e31dae52f1
FAIL (2026-03-18 19:54:58 UTC)
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 13)
======LOGS======
Mar 18 19:53:19.380863725 (296h53m16.977634736s) [upload=332531]: Starting upload
Mar 18 19:53:19.380944319 (296h53m16.977715346s) [upload=332531]: Done starting upload
Mar 18 19:53:19.380976457 (296h53m16.977747482s) [upload=332531]: Uploading segments...
Mar 18 19:53:19.381333196 (296h53m16.978104222s) [upload=332531]: Got next segment. Inline: false
Mar 18 19:53:19.381379068 (296h53m16.978150099s) [upload=332531]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Mar 18 19:53:19.381386543 (296h53m16.978157579s) [upload=332531]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Mar 18 19:53:23.490845546 (296h53m21.087616603s) [seg_pos={0 0}, upload=332531]: Begin upload segment...
Mar 18 19:53:23.490917951 (296h53m21.087688988s) [seg_pos={0 0}, upload=332531]: Done begin upload segment.
Mar 18 19:53:23.490955926 (296h53m21.087726957s) [noise=true , address=45.91.116.5:4312 , piece_id=BDB6LYZZH2HXC3WWPDVMDQLCAAGRXBM5OCZU2JQVUJ7E6REVJ5SQ, seg_pos={0 0}, upload=332531]: Uploading piece...
Mar 18 19:53:23.492093139 (296h53m21.088864171s) [noise=true , address=193.149.164.67:4003 , piece_id=SOPPFFG22JZWQA5NHYGA2OFHC2KMALZOR3ATBQ6JYWJVXATWLSRQ, seg_pos={0 0}, upload=332531]: Uploading piece...
Mar 18 19:53:23.492835295 (296h53m21.089606329s) [noise=true , address=77.90.31.18:4833 , piece_id=ON3SKTN2H26XEXC43C44MAJIPY32NZLNWRU4PBSRJS6HLMXVNBMA, seg_pos={0 0}, upload=332531]: Uploading piece...
Mar 18 19:53:23.492958664 (296h53m21.089729699s) [noise=true , address=45.91.116.13:4167 , piece_id=3JWBJU4M3WSUM3D2TVZDGJKBSRXZY5ZQHV3GFSBKOJMLORY5W7IQ, seg_pos={0 0}, upload=332531]: Uploading piece...
Mar 18 19:53:23.493624498 (296h53m21.090395533s) [noise=true , address=145.79.227.13:4814 , piece_id=3KU6BHQSH4U4XYBSV4I77FCLQKR5RKWFY3QKZ26D5QRCJDQPKTKQ, seg_pos={0 0}, upload=332531]: Uploading piece...
Mar 18 19:53:23.493650442 (296h53m21.090421482s) [noise=true , address=79.127.205.237:4230 , piece_id=Y6AOKOD27V5RL6QWUY2ZAJMONWPTYU4TTU5VIAYJ4RVMNQMCRUZQ, seg_pos={0 0}, upload=332531]: Uploading piece...
Mar 18 19:53:23.494253839 (296h53m21.091024872s) [noise=true , address=45.91.116.14:4223 , piece_id=X3NSJBVWEZ3F3DUHQOSQSUCECIN7FARCFMRPJR6N7FHIIL3MS3NQ, seg_pos={0 0}, upload=332531]: Uploading piece...
Mar 18 19:53:23.494267618 (296h53m21.091038645s) [noise=true , address=145.79.227.22:4516 , piece_id=OES5UD4FMNBSPC4ZSDS45SRXLTAQT3QNGR6JBI5D5Y363JFGJ73Q, seg_pos={0 0}, upload=332531]: Uploading piece...
Mar 18 19:53:23.494879425 (296h53m21.091650453s) [noise=true , address=193.149.164.71:4287 , piece_id=ZE2Q4NHL3KUKRB36IDRMIZ7DIKDYZIBCM2L3PTLXALGL3PINBGAQ, seg_pos={0 0}, upload=332531]: Uploading piece...
Mar 18 19:53:23.494911912 (296h53m21.091682944s) [noise=true , address=193.149.164.69:4074 , piece_id=FUGRHBCQHA4YJ2EVCMKVQVITADSVLM3PWZHV2XM3QLRDGU4G2GPA, seg_pos={0 0}, upload=332531]: Uploading piece...
Mar 18 19:53:23.495497539 (296h53m21.09226857s) [noise=true , address=193.149.164.68:4066 , piece_id=FY5RBQUD2JCNXJCY2KIK744F4QZFHTW45PBDMASIIP27RWWB7SSA, seg_pos={0 0}, upload=332531]: Uploading piece...
Mar 18 19:53:23.496168841 (296h53m21.092939872s) [noise=true , address=145.79.227.41:4105 , piece_id=RTYHA5YP42CFAYVM6PHY5RTB47GRMWSZEBBPNLC532CXKPBT2VXA, seg_pos={0 0}, upload=332531]: Uploading piece...
Mar 18 19:53:23.497008065 (296h53m21.093779105s) [noise=true , address=79.127.205.235:4271 , piece_id=5XULCXAVIG3CQFTYA6Q3TFPRVICQFHQI2MSI2VIRIAGSK5UKKFXQ, seg_pos={0 0}, upload=332531]: Uploading piece...
Mar 18 19:53:23.497605357 (296h53m21.094376396s) [noise=true , address=145.79.227.37:4590 , piece_id=ZTQLQ2IZHJFZWBHMRT74V7JDZYXF5BG4656VBZDJ4J6NGKS2TMOA, seg_pos={0 0}, upload=332531]: Uploading piece...
Mar 18 19:53:23.498200357 (296h53m21.094971388s) [noise=true , address=193.149.164.66:4989 , piece_id=DVAJPRB7ABJJGUP4TYMLOGJVDLUARAMSPCG6FE2PVQXIDP2QBPGA, seg_pos={0 0}, upload=332531]: Uploading piece...
Mar 18 19:53:23.498757906 (296h53m21.095528939s) [noise=true , address=79.127.219.45:4868 , piece_id=6NDAYV5PFDBOMZKAXKO6LZUODM46MQPQOQIDKBHER3LG4CBEDHKQ, seg_pos={0 0}, upload=332531]: Uploading piece...
Mar 18 19:53:23.499215136 (296h53m21.095986159s) [noise=true , address=145.79.227.52:4453 , piece_id=YRAA65GHFFR45DX7MKR5BWBXID3YVSA4DMQDRKWKRDTJCHKHGMOA, seg_pos={0 0}, upload=332531]: Uploading piece...
Mar 18 19:53:23.499687663 (296h53m21.096458697s) [noise=true , address=77.90.31.14:4678 , piece_id=NWBLAAUF3GACR3Y6JRTHDEUJ24GMOTZIPSRGJIPTPLIVWBCSYAHA, seg_pos={0 0}, upload=332531]: Uploading piece...
Mar 18 19:53:23.600142173 (296h53m21.196913186s) [upload=332531]: Next returned nil segment
Mar 18 19:53:23.600161212 (296h53m21.196932236s) [upload=332531]: Waiting for error group managing segments...
Mar 18 19:53:23.620885875 (296h53m21.217656899s) [noise=true , address=45.91.116.5:4312 , piece_id=BDB6LYZZH2HXC3WWPDVMDQLCAAGRXBM5OCZU2JQVUJ7E6REVJ5SQ, seg_pos={0 0}, upload=332531]: Done uploading piece. err: <nil>
Mar 18 19:53:23.621154396 (296h53m21.217925412s) [noise=true , address=45.91.116.13:4167 , piece_id=3JWBJU4M3WSUM3D2TVZDGJKBSRXZY5ZQHV3GFSBKOJMLORY5W7IQ, seg_pos={0 0}, upload=332531]: Done uploading piece. err: <nil>
Mar 18 19:53:23.628883995 (296h53m21.225655021s) [noise=true , address=45.91.116.14:4223 , piece_id=X3NSJBVWEZ3F3DUHQOSQSUCECIN7FARCFMRPJR6N7FHIIL3MS3NQ, seg_pos={0 0}, upload=332531]: Done uploading piece. err: <nil>
Mar 18 19:53:23.630970813 (296h53m21.227741838s) [noise=true , address=79.127.219.45:4868 , piece_id=6NDAYV5PFDBOMZKAXKO6LZUODM46MQPQOQIDKBHER3LG4CBEDHKQ, seg_pos={0 0}, upload=332531]: Done uploading piece. err: <nil>
Mar 18 19:53:23.639825090 (296h53m21.236596104s) [noise=true , address=77.90.31.14:4678 , piece_id=NWBLAAUF3GACR3Y6JRTHDEUJ24GMOTZIPSRGJIPTPLIVWBCSYAHA, seg_pos={0 0}, upload=332531]: Done uploading piece. err: <nil>
Mar 18 19:53:23.642754677 (296h53m21.239525692s) [noise=true , address=77.90.31.18:4833 , piece_id=ON3SKTN2H26XEXC43C44MAJIPY32NZLNWRU4PBSRJS6HLMXVNBMA, seg_pos={0 0}, upload=332531]: Done uploading piece. err: <nil>
Mar 18 19:53:24.057422272 (296h53m21.654193306s) [noise=true , address=79.127.205.235:4271 , piece_id=5XULCXAVIG3CQFTYA6Q3TFPRVICQFHQI2MSI2VIRIAGSK5UKKFXQ, seg_pos={0 0}, upload=332531]: Done uploading piece. err: <nil>
Mar 18 19:53:24.074244577 (296h53m21.671015604s) [noise=true , address=193.149.164.67:4003 , piece_id=SOPPFFG22JZWQA5NHYGA2OFHC2KMALZOR3ATBQ6JYWJVXATWLSRQ, seg_pos={0 0}, upload=332531]: Done uploading piece. err: <nil>
Mar 18 19:53:24.097678734 (296h53m21.694449747s) [noise=true , address=79.127.205.237:4230 , piece_id=Y6AOKOD27V5RL6QWUY2ZAJMONWPTYU4TTU5VIAYJ4RVMNQMCRUZQ, seg_pos={0 0}, upload=332531]: Done uploading piece. err: <nil>
Mar 18 19:53:24.109685982 (296h53m21.706457009s) [noise=true , address=193.149.164.68:4066 , piece_id=FY5RBQUD2JCNXJCY2KIK744F4QZFHTW45PBDMASIIP27RWWB7SSA, seg_pos={0 0}, upload=332531]: Done uploading piece. err: <nil>
Mar 18 19:53:24.110659716 (296h53m21.707430744s) [noise=true , address=193.149.164.69:4074 , piece_id=FUGRHBCQHA4YJ2EVCMKVQVITADSVLM3PWZHV2XM3QLRDGU4G2GPA, seg_pos={0 0}, upload=332531]: Done uploading piece. err: <nil>
Mar 18 19:53:24.114541274 (296h53m21.7113123s) [noise=true , address=193.149.164.71:4287 , piece_id=ZE2Q4NHL3KUKRB36IDRMIZ7DIKDYZIBCM2L3PTLXALGL3PINBGAQ, seg_pos={0 0}, upload=332531]: Done uploading piece. err: <nil>
Mar 18 19:53:24.140894470 (296h53m21.737665488s) [noise=true , address=193.149.164.66:4989 , piece_id=DVAJPRB7ABJJGUP4TYMLOGJVDLUARAMSPCG6FE2PVQXIDP2QBPGA, seg_pos={0 0}, upload=332531]: Done uploading piece. err: <nil>
Mar 18 19:54:19.329327996 (296h54m16.926099018s) [noise=true , address=145.79.227.13:4814 , piece_id=3KU6BHQSH4U4XYBSV4I77FCLQKR5RKWFY3QKZ26D5QRCJDQPKTKQ, seg_pos={0 0}, upload=332531]: Done uploading piece. err: ecclient: upload failed (node:12ERtrzMuh8rZPsgebtyScmYvNVwz3UKJr5Nu2hLdu6sn8X3qLa, address:145.79.227.13:4814): protocol: rpc: dial tcp 145.79.227.13:4814: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 19:54:19.329567681 (296h54m16.926338706s) [noise=true , address=145.79.227.52:4453 , piece_id=YRAA65GHFFR45DX7MKR5BWBXID3YVSA4DMQDRKWKRDTJCHKHGMOA, seg_pos={0 0}, upload=332531]: Done uploading piece. err: ecclient: upload failed (node:1BmpsQ9VfhGzmkZHvSeMptcGhzGpdoN4oYvi83qZcC1Vh9anhP, address:145.79.227.52:4453): protocol: rpc: dial tcp 145.79.227.52:4453: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 19:54:19.329755822 (296h54m16.926526848s) [noise=true , address=145.79.227.41:4105 , piece_id=RTYHA5YP42CFAYVM6PHY5RTB47GRMWSZEBBPNLC532CXKPBT2VXA, seg_pos={0 0}, upload=332531]: Done uploading piece. err: ecclient: upload failed (node:1TVtMQNVV2kERp2aZwmS8E9JUiua1RCyFpZgFACRqH1Nfscw4C, address:145.79.227.41:4105): protocol: rpc: dial tcp 145.79.227.41:4105: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 19:54:19.330021146 (296h54m16.926792175s) [noise=true , address=145.79.227.22:4516 , piece_id=OES5UD4FMNBSPC4ZSDS45SRXLTAQT3QNGR6JBI5D5Y363JFGJ73Q, seg_pos={0 0}, upload=332531]: Done uploading piece. err: ecclient: upload failed (node:12Lhy3AxzYoF25AxWdvjrrV5NNCwP1cDqkEvv4tz9t9KZmF8WPK, address:145.79.227.22:4516): protocol: rpc: dial tcp 145.79.227.22:4516: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 19:54:19.330213680 (296h54m16.926984707s) [noise=true , address=145.79.227.37:4590 , piece_id=ZTQLQ2IZHJFZWBHMRT74V7JDZYXF5BG4656VBZDJ4J6NGKS2TMOA, seg_pos={0 0}, upload=332531]: Done uploading piece. err: ecclient: upload failed (node:1KnNqY1WNwMSFXFrQsWPAhivUTgSn7dKTx1mTTzYG2UxGCkasj, address:145.79.227.37:4590): protocol: rpc: dial tcp 145.79.227.37:4590: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 19:54:19.330250832 (296h54m16.927021848s) [seg_pos={0 0}, upload=332531]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil>
Mar 18 19:54:19.351979131 (296h54m16.948750155s) [upload=332531]: Done uploading segments...
Mar 18 19:54:19.352025297 (296h54m16.948796327s) [upload=332531]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13)
Duration: 15.049946989s
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
26 out of 60 is failed (threshold=3) [..................................XXXXXXXXXXXXXXXXXXXXXXXXXX] duration:
30.034938262s trace:
https://jaeger.dev.storj.io/trace/4f5c2a5a9c4af103
FAIL (2026-03-18 19:54:55 UTC)
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 6)
======LOGS======
em: *pb.BatchRequestItem_ObjectBegin
Mar 18 19:54:25.443057333 (296h54m23.039828357s) [upload=332551]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Mar 18 19:54:29.440428309 (296h54m27.037199333s) [seg_pos={0 0}, upload=332551]: Begin upload segment...
Mar 18 19:54:29.440516685 (296h54m27.037287709s) [seg_pos={0 0}, upload=332551]: Done begin upload segment.
Mar 18 19:54:29.440550144 (296h54m27.037321173s) [noise=true , address=77.90.31.15:4697 , piece_id=UORURE73FBN53Q6G32MPVFH3XUD43QCZFNQS75UVCN3VQP4PMG3A, seg_pos={0 0}, upload=332551]: Uploading piece...
Mar 18 19:54:29.440646952 (296h54m27.037417978s) [noise=true , address=77.90.31.21:4878 , piece_id=JK5D4ARVAT4IHSIHIGAI77ZTOOXRRAHJGELE6C7DTQCIGKYHVZEA, seg_pos={0 0}, upload=332551]: Uploading piece...
Mar 18 19:54:29.441188969 (296h54m27.037959993s) [noise=true , address=145.79.227.10:4405 , piece_id=EQDCZ425Z6FZZTXDLYC4L7PR6REL6PVDUUMOTTMYYXSA5E6Y2UEA, seg_pos={0 0}, upload=332551]: Uploading piece...
Mar 18 19:54:29.441521359 (296h54m27.038292385s) [noise=true , address=145.79.227.48:4302 , piece_id=KAQTP52Q4V5FQYRPBDNK4NO22NAG2SN4A6PHA7TVSZCL7STVYLVQ, seg_pos={0 0}, upload=332551]: Uploading piece...
Mar 18 19:54:29.441667959 (296h54m27.038438983s) [noise=true , address=77.90.31.12:4590 , piece_id=DHT6KLWEWC5SVRKNFGZNAZY7IRTP5IRYTIUECSSRZEIPL2EFUHEQ, seg_pos={0 0}, upload=332551]: Uploading piece...
Mar 18 19:54:29.442115683 (296h54m27.038886706s) [noise=true , address=145.79.227.20:4439 , piece_id=J3CGQ4JYEQZEXIH7B44CNG55MUAZDG7BEGL5PDV7LFQZKU7J6YIA, seg_pos={0 0}, upload=332551]: Uploading piece...
Mar 18 19:54:29.442581665 (296h54m27.039352688s) [noise=true , address=145.79.227.19:4083 , piece_id=QEVPVIKO7P5PQJSJTHSQMFVQO7BHL7X2IDRXYHD54QIVWYIGHQJA, seg_pos={0 0}, upload=332551]: Uploading piece...
Mar 18 19:54:29.443095524 (296h54m27.039866547s) [noise=true , address=145.79.227.41:4087 , piece_id=RM4CUWGK3KCAKDT35EJHYVG53FZRSRVIIVR4FQ6ALZJ2STTEFGVA, seg_pos={0 0}, upload=332551]: Uploading piece...
Mar 18 19:54:29.443837443 (296h54m27.04060847s) [noise=true , address=145.79.227.17:4973 , piece_id=KVFJBIBKU3PBP7VH43WGREYM4EBMHJSWFYV4AEF4G22II5IW4CFA, seg_pos={0 0}, upload=332551]: Uploading piece...
Mar 18 19:54:29.444118516 (296h54m27.040889549s) [noise=true , address=77.90.31.18:4805 , piece_id=PAVH34ZQB6QDJ6JKMQMJSYRZDMPOTK6ZYM6RFWEVNOBDOC2IP2TQ, seg_pos={0 0}, upload=332551]: Uploading piece...
Mar 18 19:54:29.445544842 (296h54m27.042315868s) [noise=true , address=145.79.227.34:4905 , piece_id=YXHAJC7RD3QLEP2FTTFMAULIFAA6V4AU6QYFNZFAGVQF6GU7S6UQ, seg_pos={0 0}, upload=332551]: Uploading piece...
Mar 18 19:54:29.446560646 (296h54m27.043331678s) [noise=true , address=45.91.116.9:4262 , piece_id=HQHL5GJHPOG2FCILEFMZUPEBQZHK5NDZZ2PRQ3KOJBYUYF44FKZQ, seg_pos={0 0}, upload=332551]: Uploading piece...
Mar 18 19:54:29.448613184 (296h54m27.04538425s) [noise=true , address=145.79.227.38:4979 , piece_id=62R3JX2NH33HJRCV764RRIGLR2344H4QGQUPFOQZTLAQ3NPI7LWQ, seg_pos={0 0}, upload=332551]: Uploading piece...
Mar 18 19:54:29.449994164 (296h54m27.046765178s) [noise=true , address=79.127.219.41:4930 , piece_id=YFE46LZEEQTYZUHYXD5I45VAX4T2P7UUBT3E5WRC22RWTM5TWV2Q, seg_pos={0 0}, upload=332551]: Uploading piece...
Mar 18 19:54:29.451185291 (296h54m27.047956334s) [noise=true , address=145.79.227.49:4338 , piece_id=7NOYL3CDIBGNTKFP2XGLK6R5QVO4E3I4CNIIEV2IOH36SEL3K7MA, seg_pos={0 0}, upload=332551]: Uploading piece...
Mar 18 19:54:29.451963049 (296h54m27.048734071s) [noise=true , address=145.79.227.37:4566 , piece_id=NCYVYYKOUIJNSLM6UKOHWKTJCOZ53NPFXRXHYQWTM7WK5PJ27URA, seg_pos={0 0}, upload=332551]: Uploading piece...
Mar 18 19:54:29.514898992 (296h54m27.111670005s) [upload=332551]: Next returned nil segment
Mar 18 19:54:29.514912847 (296h54m27.11168387s) [upload=332551]: Waiting for error group managing segments...
Mar 18 19:54:29.520158968 (296h54m27.116929991s) [noise=true , address=45.91.116.9:4262 , piece_id=HQHL5GJHPOG2FCILEFMZUPEBQZHK5NDZZ2PRQ3KOJBYUYF44FKZQ, seg_pos={0 0}, upload=332551]: Done uploading piece. err: <nil>
Mar 18 19:54:29.520848838 (296h54m27.117619865s) [noise=true , address=77.90.31.21:4878 , piece_id=JK5D4ARVAT4IHSIHIGAI77ZTOOXRRAHJGELE6C7DTQCIGKYHVZEA, seg_pos={0 0}, upload=332551]: Done uploading piece. err: <nil>
Mar 18 19:54:29.521504833 (296h54m27.118275857s) [noise=true , address=79.127.219.41:4930 , piece_id=YFE46LZEEQTYZUHYXD5I45VAX4T2P7UUBT3E5WRC22RWTM5TWV2Q, seg_pos={0 0}, upload=332551]: Done uploading piece. err: <nil>
Mar 18 19:54:29.521913153 (296h54m27.118684167s) [noise=true , address=77.90.31.12:4590 , piece_id=DHT6KLWEWC5SVRKNFGZNAZY7IRTP5IRYTIUECSSRZEIPL2EFUHEQ, seg_pos={0 0}, upload=332551]: Done uploading piece. err: <nil>
Mar 18 19:54:29.523780796 (296h54m27.120551819s) [noise=true , address=77.90.31.18:4805 , piece_id=PAVH34ZQB6QDJ6JKMQMJSYRZDMPOTK6ZYM6RFWEVNOBDOC2IP2TQ, seg_pos={0 0}, upload=332551]: Done uploading piece. err: <nil>
Mar 18 19:54:29.533252641 (296h54m27.130023664s) [noise=true , address=77.90.31.15:4697 , piece_id=UORURE73FBN53Q6G32MPVFH3XUD43QCZFNQS75UVCN3VQP4PMG3A, seg_pos={0 0}, upload=332551]: Done uploading piece. err: <nil>
Mar 18 19:54:55.405117061 (296h54m53.001888094s) [noise=true , address=145.79.227.37:4566 , piece_id=NCYVYYKOUIJNSLM6UKOHWKTJCOZ53NPFXRXHYQWTM7WK5PJ27URA, seg_pos={0 0}, upload=332551]: Done uploading piece. err: ecclient: upload failed (node:1SaNJZaVix7BSbEWhu2Vmt3XjDDWCsQ8wTSiFR7MQ8oGwD9N2F, address:145.79.227.37:4566): protocol: rpc: dial tcp 145.79.227.37:4566: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 19:54:55.405395781 (296h54m53.002166791s) [noise=true , address=145.79.227.48:4302 , piece_id=KAQTP52Q4V5FQYRPBDNK4NO22NAG2SN4A6PHA7TVSZCL7STVYLVQ, seg_pos={0 0}, upload=332551]: Done uploading piece. err: ecclient: upload failed (node:1tGpuYdPLiy4ncDjxrZbQfp7Bq79NVBHDmr4vjaL4qMpudrvgR, address:145.79.227.48:4302): protocol: rpc: dial tcp 145.79.227.48:4302: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 19:54:55.406029711 (296h54m53.002800739s) [noise=true , address=145.79.227.41:4087 , piece_id=RM4CUWGK3KCAKDT35EJHYVG53FZRSRVIIVR4FQ6ALZJ2STTEFGVA, seg_pos={0 0}, upload=332551]: Done uploading piece. err: ecclient: upload failed (node:1BfeGD2MGRPqsBRGLioTiTAcPykpSTUUx2CBVSPQcz4LQqFdWR, address:145.79.227.41:4087): protocol: rpc: dial tcp 145.79.227.41:4087: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 19:54:55.406145296 (296h54m53.002916339s) [noise=true , address=145.79.227.10:4405 , piece_id=EQDCZ425Z6FZZTXDLYC4L7PR6REL6PVDUUMOTTMYYXSA5E6Y2UEA, seg_pos={0 0}, upload=332551]: Done uploading piece. err: ecclient: upload failed (node:1MCHYW29ZQjE2fHN8P98hDfNRLsjiVqabF86TXz5nZRBPhbGy4, address:145.79.227.10:4405): protocol: rpc: dial tcp 145.79.227.10:4405: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 19:54:55.406543045 (296h54m53.00331408s) [noise=true , address=145.79.227.49:4338 , piece_id=7NOYL3CDIBGNTKFP2XGLK6R5QVO4E3I4CNIIEV2IOH36SEL3K7MA, seg_pos={0 0}, upload=332551]: Done uploading piece. err: ecclient: upload failed (node:1cTadUV1Zu2Z8MoobXvyZ965fv6jPFhYGsy5qgRZHgWeSwyitN, address:145.79.227.49:4338): protocol: rpc: dial tcp 145.79.227.49:4338: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 19:54:55.406824760 (296h54m53.003595793s) [noise=true , address=145.79.227.38:4979 , piece_id=62R3JX2NH33HJRCV764RRIGLR2344H4QGQUPFOQZTLAQ3NPI7LWQ, seg_pos={0 0}, upload=332551]: Done uploading piece. err: ecclient: upload failed (node:12B518TvbbRhj6LiXxV4GwUgAoUYmWMWeTpyYerKnKuZYBjXoma, address:145.79.227.38:4979): protocol: rpc: dial tcp 145.79.227.38:4979: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 19:54:55.406961859 (296h54m53.003732897s) [noise=true , address=145.79.227.17:4973 , piece_id=KVFJBIBKU3PBP7VH43WGREYM4EBMHJSWFYV4AEF4G22II5IW4CFA, seg_pos={0 0}, upload=332551]: Done uploading piece. err: ecclient: upload failed (node:12afPNc97E17nBKr8sz5ujCrrVry16s6e28MEduErC2CRshtv9d, address:145.79.227.17:4973): protocol: rpc: dial tcp 145.79.227.17:4973: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 19:54:55.407240289 (296h54m53.004011322s) [noise=true , address=145.79.227.19:4083 , piece_id=QEVPVIKO7P5PQJSJTHSQMFVQO7BHL7X2IDRXYHD54QIVWYIGHQJA, seg_pos={0 0}, upload=332551]: Done uploading piece. err: ecclient: upload failed (node:12KZ5a6CQ9tUTrC6EYE7M8yg6p9fK95CN8zQLQmP4tdyz4rBfGh, address:145.79.227.19:4083): protocol: rpc: dial tcp 145.79.227.19:4083: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 19:54:55.407680868 (296h54m53.004451902s) [noise=true , address=145.79.227.34:4905 , piece_id=YXHAJC7RD3QLEP2FTTFMAULIFAA6V4AU6QYFNZFAGVQF6GU7S6UQ, seg_pos={0 0}, upload=332551]: Done uploading piece. err: ecclient: upload failed (node:12ffKESNSVpXQzbwJinP6vvgE6zoF3RFz2NRUuuJHvd7bvk1gc9, address:145.79.227.34:4905): protocol: rpc: dial tcp 145.79.227.34:4905: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 19:54:55.408069427 (296h54m53.004840456s) [noise=true , address=145.79.227.20:4439 , piece_id=J3CGQ4JYEQZEXIH7B44CNG55MUAZDG7BEGL5PDV7LFQZKU7J6YIA, seg_pos={0 0}, upload=332551]: Done uploading piece. err: ecclient: upload failed (node:1wgkbc3Mkej7supxhYMtUdBvn9t45i6kmAokAUkqFxEpehMjAn, address:145.79.227.20:4439): protocol: context deadline exceeded; context deadline exceeded
Mar 18 19:54:55.408141542 (296h54m53.004912563s) [seg_pos={0 0}, upload=332551]: Done waiting for segment. successful: 6 optimal: 13 errs: <nil>
Mar 18 19:54:55.437389681 (296h54m53.034160712s) [upload=332551]: Done uploading segments...
Mar 18 19:54:55.437444479 (296h54m53.034215512s) [upload=332551]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 6)
Duration: 30.034977753s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
8 out of 60 is failed (threshold=3) [...................................XXXXXX.XX................] duration:
18.655915923s trace:
https://jaeger.dev.storj.io/trace/4b4d10a8051ad01b
FAIL (2026-03-18 19:54:55 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 18.655980613s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
5 out of 60 is failed (threshold=3) [....................................XXX..........X...X......] duration:
20.68490093s trace:
https://jaeger.dev.storj.io/trace/6a01088813e3d8b7
FAIL (2026-03-18 19:54:54 UTC)
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 12)
======LOGS======
m: *pb.BatchRequestItem_ObjectBegin
Mar 18 19:49:08.138517471 (296h49m5.735288494s) [upload=332459]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Mar 18 19:49:16.284286759 (296h49m13.881057773s) [seg_pos={0 0}, upload=332459]: Begin upload segment...
Mar 18 19:49:16.284352223 (296h49m13.88112325s) [seg_pos={0 0}, upload=332459]: Done begin upload segment.
Mar 18 19:49:16.284395789 (296h49m13.881166818s) [noise=true , address=193.149.164.68:4051 , piece_id=FCHPX2FRYUVNH4JAAPKI2Y7DSJCUJ5UG3EASXQYB7VEQPNERYUGQ, seg_pos={0 0}, upload=332459]: Uploading piece...
Mar 18 19:49:16.284449689 (296h49m13.881220728s) [noise=true , address=145.79.227.13:4822 , piece_id=Y5GSC3KFFGLK4B3NCXEMOZTAC3ASNGPJJOPOKCUBUJIL75E6T6FA, seg_pos={0 0}, upload=332459]: Uploading piece...
Mar 18 19:49:16.284549974 (296h49m13.881320988s) [noise=true , address=45.91.116.5:4308 , piece_id=XRXYA347E22AHAJ4XDENA5EHCV2F43HXL67MTUGRYSU6B3WQBDNQ, seg_pos={0 0}, upload=332459]: Uploading piece...
Mar 18 19:49:16.285144837 (296h49m13.88191587s) [noise=true , address=193.149.164.70:4117 , piece_id=PL6FWCO6XQRSVU26FH2KESA26D4Z2PCQVBPBDKBWLBPXHBV5LQVQ, seg_pos={0 0}, upload=332459]: Uploading piece...
Mar 18 19:49:16.285821284 (296h49m13.882592317s) [noise=true , address=145.79.227.52:4451 , piece_id=SBMEAZS5RDQW3RCJP2UU5T7BHF6P7VOICLP5P46YJ5LE2GB65CGA, seg_pos={0 0}, upload=332459]: Uploading piece...
Mar 18 19:49:16.286556627 (296h49m13.883327669s) [noise=true , address=79.127.205.235:4259 , piece_id=N7BGDCO2CR6HLPUWYF35T5OQRWQFEWFG7FPZ6Y33J7FGR5H2ZURQ, seg_pos={0 0}, upload=332459]: Uploading piece...
Mar 18 19:49:16.287228779 (296h49m13.883999813s) [noise=true , address=79.127.219.45:4864 , piece_id=UXHLRN5HUN3ZRXOZEYAJOS6QUN3PVIXXXED3TS6RPWSHXJ4WKS6Q, seg_pos={0 0}, upload=332459]: Uploading piece...
Mar 18 19:49:16.287269957 (296h49m13.884040985s) [noise=true , address=77.90.31.21:4894 , piece_id=IQ6775I4HFC5JQVYUCQPXYWPY3X6QRHVXM6IF4JEZRQMMRRWQDUQ, seg_pos={0 0}, upload=332459]: Uploading piece...
Mar 18 19:49:16.287437026 (296h49m13.884208049s) [noise=true , address=5.231.73.4:4010 , piece_id=LTONWX7UJVP3BPHHKKJUN6PTAR5TVPVKF3CD6NBV2MZWTLUOVQZQ, seg_pos={0 0}, upload=332459]: Uploading piece...
Mar 18 19:49:16.287832890 (296h49m13.884603914s) [noise=true , address=167.17.72.68:4657 , piece_id=UUIV3ERJSVOJT4TISEHCMR6EQCG6PMZUR6OWGK3ZVV33B33CUZCA, seg_pos={0 0}, upload=332459]: Uploading piece...
Mar 18 19:49:16.287952922 (296h49m13.884723957s) [noise=true , address=79.127.219.41:4939 , piece_id=WK6HMSTVM7LD2P5MPCKML4MBFZJX2QJWZS6SYPITDKNOLPMRPO6Q, seg_pos={0 0}, upload=332459]: Uploading piece...
Mar 18 19:49:16.288418812 (296h49m13.885189842s) [noise=true , address=145.79.227.44:4213 , piece_id=MK3BJSFCDOG2MAJYO2BAE2JG62EVEZMCCUATBBBT4QU44GLKK3QA, seg_pos={0 0}, upload=332459]: Uploading piece...
Mar 18 19:49:16.288922438 (296h49m13.885693461s) [noise=true , address=145.79.227.3:4163 , piece_id=3R3QBIJ2XDQZQXJ66RGNZBWZB6V4ERS5RJEZFBTS2LFMQ25G4FLA, seg_pos={0 0}, upload=332459]: Uploading piece...
Mar 18 19:49:16.289343580 (296h49m13.886114604s) [noise=true , address=193.149.164.71:4310 , piece_id=BKUA4WC73DSXJMEPCMTXTUD2PVQAAYD44MAGEB3SS3FEDJO7377A, seg_pos={0 0}, upload=332459]: Uploading piece...
Mar 18 19:49:16.289780590 (296h49m13.88655162s) [noise=true , address=145.79.227.49:4358 , piece_id=QSE44UR5YBYTJHWGOCBFVQPAUZ3SJV6P5WKBEF3MTFAKVRH2QSHQ, seg_pos={0 0}, upload=332459]: Uploading piece...
Mar 18 19:49:16.290237004 (296h49m13.887008027s) [noise=true , address=77.90.31.14:4685 , piece_id=3KAIZVB2LH6R57CFO55R7BJV7G4MGHFN6M5VTKT3J6LMI5F7PVAA, seg_pos={0 0}, upload=332459]: Uploading piece...
Mar 18 19:49:16.292879809 (296h49m13.889650843s) [noise=true , address=77.90.31.18:4817 , piece_id=FMVKZ4Z3YDPWZHTXU3ZTSWXMVUWLKQX7TUWXPQ4LOKBQK64PHF5Q, seg_pos={0 0}, upload=332459]: Uploading piece...
Mar 18 19:49:16.293664670 (296h49m13.890435693s) [noise=true , address=145.79.227.15:4908 , piece_id=LG24OMYBUVPVRBFZL4UUJLPBVNDTVVHQ4HAYTW2PGKHUKZPF2WCA, seg_pos={0 0}, upload=332459]: Uploading piece...
Mar 18 19:49:16.351440712 (296h49m13.948211724s) [upload=332459]: Next returned nil segment
Mar 18 19:49:16.351460639 (296h49m13.948231661s) [upload=332459]: Waiting for error group managing segments...
Mar 18 19:49:16.355941640 (296h49m13.952712664s) [noise=true , address=77.90.31.21:4894 , piece_id=IQ6775I4HFC5JQVYUCQPXYWPY3X6QRHVXM6IF4JEZRQMMRRWQDUQ, seg_pos={0 0}, upload=332459]: Done uploading piece. err: <nil>
Mar 18 19:49:16.356991929 (296h49m13.953762955s) [noise=true , address=79.127.219.41:4939 , piece_id=WK6HMSTVM7LD2P5MPCKML4MBFZJX2QJWZS6SYPITDKNOLPMRPO6Q, seg_pos={0 0}, upload=332459]: Done uploading piece. err: <nil>
Mar 18 19:49:16.361809763 (296h49m13.958580786s) [noise=true , address=45.91.116.5:4308 , piece_id=XRXYA347E22AHAJ4XDENA5EHCV2F43HXL67MTUGRYSU6B3WQBDNQ, seg_pos={0 0}, upload=332459]: Done uploading piece. err: <nil>
Mar 18 19:49:16.367502365 (296h49m13.964273396s) [noise=true , address=77.90.31.14:4685 , piece_id=3KAIZVB2LH6R57CFO55R7BJV7G4MGHFN6M5VTKT3J6LMI5F7PVAA, seg_pos={0 0}, upload=332459]: Done uploading piece. err: <nil>
Mar 18 19:49:16.381549888 (296h49m13.97832091s) [noise=true , address=79.127.219.45:4864 , piece_id=UXHLRN5HUN3ZRXOZEYAJOS6QUN3PVIXXXED3TS6RPWSHXJ4WKS6Q, seg_pos={0 0}, upload=332459]: Done uploading piece. err: <nil>
Mar 18 19:49:16.387503503 (296h49m13.984274527s) [noise=true , address=77.90.31.18:4817 , piece_id=FMVKZ4Z3YDPWZHTXU3ZTSWXMVUWLKQX7TUWXPQ4LOKBQK64PHF5Q, seg_pos={0 0}, upload=332459]: Done uploading piece. err: <nil>
Mar 18 19:49:16.498139242 (296h49m14.094910266s) [noise=true , address=5.231.73.4:4010 , piece_id=LTONWX7UJVP3BPHHKKJUN6PTAR5TVPVKF3CD6NBV2MZWTLUOVQZQ, seg_pos={0 0}, upload=332459]: Done uploading piece. err: <nil>
Mar 18 19:49:16.852944491 (296h49m14.449715518s) [noise=true , address=167.17.72.68:4657 , piece_id=UUIV3ERJSVOJT4TISEHCMR6EQCG6PMZUR6OWGK3ZVV33B33CUZCA, seg_pos={0 0}, upload=332459]: Done uploading piece. err: <nil>
Mar 18 19:49:16.878068834 (296h49m14.474839867s) [noise=true , address=79.127.205.235:4259 , piece_id=N7BGDCO2CR6HLPUWYF35T5OQRWQFEWFG7FPZ6Y33J7FGR5H2ZURQ, seg_pos={0 0}, upload=332459]: Done uploading piece. err: <nil>
Mar 18 19:49:16.892446253 (296h49m14.489217282s) [noise=true , address=193.149.164.68:4051 , piece_id=FCHPX2FRYUVNH4JAAPKI2Y7DSJCUJ5UG3EASXQYB7VEQPNERYUGQ, seg_pos={0 0}, upload=332459]: Done uploading piece. err: <nil>
Mar 18 19:49:16.893099142 (296h49m14.48987017s) [noise=true , address=193.149.164.71:4310 , piece_id=BKUA4WC73DSXJMEPCMTXTUD2PVQAAYD44MAGEB3SS3FEDJO7377A, seg_pos={0 0}, upload=332459]: Done uploading piece. err: <nil>
Mar 18 19:49:17.031110208 (296h49m14.627881232s) [noise=true , address=193.149.164.70:4117 , piece_id=PL6FWCO6XQRSVU26FH2KESA26D4Z2PCQVBPBDKBWLBPXHBV5LQVQ, seg_pos={0 0}, upload=332459]: Done uploading piece. err: <nil>
Mar 18 19:50:08.096628364 (296h50m5.693399398s) [noise=true , address=145.79.227.52:4451 , piece_id=SBMEAZS5RDQW3RCJP2UU5T7BHF6P7VOICLP5P46YJ5LE2GB65CGA, seg_pos={0 0}, upload=332459]: Done uploading piece. err: ecclient: upload failed (node:12bC4P3xBiPFkxLvavGFVbWKnbprCStL6UJgfKSA2Wd2fX7Mbfw, address:145.79.227.52:4451): protocol: rpc: dial tcp 145.79.227.52:4451: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 19:50:08.096712565 (296h50m5.693483599s) [noise=true , address=145.79.227.44:4213 , piece_id=MK3BJSFCDOG2MAJYO2BAE2JG62EVEZMCCUATBBBT4QU44GLKK3QA, seg_pos={0 0}, upload=332459]: Done uploading piece. err: ecclient: upload failed (node:129LRnDm7b4Fo5db7pto8GXYEtxHz9qdPAtQvaYH6gqdkMatQ6g, address:145.79.227.44:4213): protocol: rpc: dial tcp 145.79.227.44:4213: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 19:50:08.096943717 (296h50m5.693714752s) [noise=true , address=145.79.227.49:4358 , piece_id=QSE44UR5YBYTJHWGOCBFVQPAUZ3SJV6P5WKBEF3MTFAKVRH2QSHQ, seg_pos={0 0}, upload=332459]: Done uploading piece. err: ecclient: upload failed (node:1dwCNEcA9o51ZN3UaN3Frxdawwf8DXBj86iQMywjW8XP4ESmw1, address:145.79.227.49:4358): protocol: rpc: dial tcp 145.79.227.49:4358: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 19:50:08.097022451 (296h50m5.693793484s) [noise=true , address=145.79.227.3:4163 , piece_id=3R3QBIJ2XDQZQXJ66RGNZBWZB6V4ERS5RJEZFBTS2LFMQ25G4FLA, seg_pos={0 0}, upload=332459]: Done uploading piece. err: ecclient: upload failed (node:12w88UkGu5duYivJ6L6C9iJmTbnB63fDKiUUQprhwM3JshpXTdC, address:145.79.227.3:4163): protocol: rpc: dial tcp 145.79.227.3:4163: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 19:50:08.097155433 (296h50m5.693926468s) [noise=true , address=145.79.227.13:4822 , piece_id=Y5GSC3KFFGLK4B3NCXEMOZTAC3ASNGPJJOPOKCUBUJIL75E6T6FA, seg_pos={0 0}, upload=332459]: Done uploading piece. err: ecclient: upload failed (node:12fgLN9DGBGxWfBEePSsk84FTrnKoKWwNgoET3FpotfJxJ28f75, address:145.79.227.13:4822): protocol: rpc: dial tcp 145.79.227.13:4822: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 19:50:08.097360007 (296h50m5.694131039s) [noise=true , address=145.79.227.15:4908 , piece_id=LG24OMYBUVPVRBFZL4UUJLPBVNDTVVHQ4HAYTW2PGKHUKZPF2WCA, seg_pos={0 0}, upload=332459]: Done uploading piece. err: ecclient: upload failed (node:1pfQ5v1GwHhFTvh5Y7Jpb9qpWhZhFYqWX9PJpLEUzTkqKWcy4x, address:145.79.227.15:4908): protocol: rpc: dial tcp 145.79.227.15:4908: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 19:50:08.097462766 (296h50m5.69423378s) [seg_pos={0 0}, upload=332459]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil>
Mar 18 19:50:08.117913456 (296h50m5.714684468s) [upload=332459]: Done uploading segments...
Mar 18 19:50:08.117999321 (296h50m5.714770348s) [upload=332459]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12)
Duration: 20.684933292s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
32 out of 60 is failed (threshold=6) [........................XXXXXXXXX.XXX...XXXXXXXXXXXXXXXXXXXX] duration:
5.000896739s trace:
https://jaeger.dev.storj.io/trace/2db44789e38b6a56
FAIL (2026-03-18 19:54:47 UTC)
operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.000937897s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)antiblip:
5 out of 60 is failed (threshold=3) [......................................X..........XX..X....X.] duration:
20.110411462s trace:
https://jaeger.dev.storj.io/trace/6b5565e31dae52f1
FAIL (2026-03-18 19:54:39 UTC)
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 13)
======LOGS======
Mar 18 19:53:19.380863725 (296h53m16.977634736s) [upload=332531]: Starting upload
Mar 18 19:53:19.380944319 (296h53m16.977715346s) [upload=332531]: Done starting upload
Mar 18 19:53:19.380976457 (296h53m16.977747482s) [upload=332531]: Uploading segments...
Mar 18 19:53:19.381333196 (296h53m16.978104222s) [upload=332531]: Got next segment. Inline: false
Mar 18 19:53:19.381379068 (296h53m16.978150099s) [upload=332531]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Mar 18 19:53:19.381386543 (296h53m16.978157579s) [upload=332531]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Mar 18 19:53:23.490845546 (296h53m21.087616603s) [seg_pos={0 0}, upload=332531]: Begin upload segment...
Mar 18 19:53:23.490917951 (296h53m21.087688988s) [seg_pos={0 0}, upload=332531]: Done begin upload segment.
Mar 18 19:53:23.490955926 (296h53m21.087726957s) [noise=true , address=45.91.116.5:4312 , piece_id=BDB6LYZZH2HXC3WWPDVMDQLCAAGRXBM5OCZU2JQVUJ7E6REVJ5SQ, seg_pos={0 0}, upload=332531]: Uploading piece...
Mar 18 19:53:23.492093139 (296h53m21.088864171s) [noise=true , address=193.149.164.67:4003 , piece_id=SOPPFFG22JZWQA5NHYGA2OFHC2KMALZOR3ATBQ6JYWJVXATWLSRQ, seg_pos={0 0}, upload=332531]: Uploading piece...
Mar 18 19:53:23.492835295 (296h53m21.089606329s) [noise=true , address=77.90.31.18:4833 , piece_id=ON3SKTN2H26XEXC43C44MAJIPY32NZLNWRU4PBSRJS6HLMXVNBMA, seg_pos={0 0}, upload=332531]: Uploading piece...
Mar 18 19:53:23.492958664 (296h53m21.089729699s) [noise=true , address=45.91.116.13:4167 , piece_id=3JWBJU4M3WSUM3D2TVZDGJKBSRXZY5ZQHV3GFSBKOJMLORY5W7IQ, seg_pos={0 0}, upload=332531]: Uploading piece...
Mar 18 19:53:23.493624498 (296h53m21.090395533s) [noise=true , address=145.79.227.13:4814 , piece_id=3KU6BHQSH4U4XYBSV4I77FCLQKR5RKWFY3QKZ26D5QRCJDQPKTKQ, seg_pos={0 0}, upload=332531]: Uploading piece...
Mar 18 19:53:23.493650442 (296h53m21.090421482s) [noise=true , address=79.127.205.237:4230 , piece_id=Y6AOKOD27V5RL6QWUY2ZAJMONWPTYU4TTU5VIAYJ4RVMNQMCRUZQ, seg_pos={0 0}, upload=332531]: Uploading piece...
Mar 18 19:53:23.494253839 (296h53m21.091024872s) [noise=true , address=45.91.116.14:4223 , piece_id=X3NSJBVWEZ3F3DUHQOSQSUCECIN7FARCFMRPJR6N7FHIIL3MS3NQ, seg_pos={0 0}, upload=332531]: Uploading piece...
Mar 18 19:53:23.494267618 (296h53m21.091038645s) [noise=true , address=145.79.227.22:4516 , piece_id=OES5UD4FMNBSPC4ZSDS45SRXLTAQT3QNGR6JBI5D5Y363JFGJ73Q, seg_pos={0 0}, upload=332531]: Uploading piece...
Mar 18 19:53:23.494879425 (296h53m21.091650453s) [noise=true , address=193.149.164.71:4287 , piece_id=ZE2Q4NHL3KUKRB36IDRMIZ7DIKDYZIBCM2L3PTLXALGL3PINBGAQ, seg_pos={0 0}, upload=332531]: Uploading piece...
Mar 18 19:53:23.494911912 (296h53m21.091682944s) [noise=true , address=193.149.164.69:4074 , piece_id=FUGRHBCQHA4YJ2EVCMKVQVITADSVLM3PWZHV2XM3QLRDGU4G2GPA, seg_pos={0 0}, upload=332531]: Uploading piece...
Mar 18 19:53:23.495497539 (296h53m21.09226857s) [noise=true , address=193.149.164.68:4066 , piece_id=FY5RBQUD2JCNXJCY2KIK744F4QZFHTW45PBDMASIIP27RWWB7SSA, seg_pos={0 0}, upload=332531]: Uploading piece...
Mar 18 19:53:23.496168841 (296h53m21.092939872s) [noise=true , address=145.79.227.41:4105 , piece_id=RTYHA5YP42CFAYVM6PHY5RTB47GRMWSZEBBPNLC532CXKPBT2VXA, seg_pos={0 0}, upload=332531]: Uploading piece...
Mar 18 19:53:23.497008065 (296h53m21.093779105s) [noise=true , address=79.127.205.235:4271 , piece_id=5XULCXAVIG3CQFTYA6Q3TFPRVICQFHQI2MSI2VIRIAGSK5UKKFXQ, seg_pos={0 0}, upload=332531]: Uploading piece...
Mar 18 19:53:23.497605357 (296h53m21.094376396s) [noise=true , address=145.79.227.37:4590 , piece_id=ZTQLQ2IZHJFZWBHMRT74V7JDZYXF5BG4656VBZDJ4J6NGKS2TMOA, seg_pos={0 0}, upload=332531]: Uploading piece...
Mar 18 19:53:23.498200357 (296h53m21.094971388s) [noise=true , address=193.149.164.66:4989 , piece_id=DVAJPRB7ABJJGUP4TYMLOGJVDLUARAMSPCG6FE2PVQXIDP2QBPGA, seg_pos={0 0}, upload=332531]: Uploading piece...
Mar 18 19:53:23.498757906 (296h53m21.095528939s) [noise=true , address=79.127.219.45:4868 , piece_id=6NDAYV5PFDBOMZKAXKO6LZUODM46MQPQOQIDKBHER3LG4CBEDHKQ, seg_pos={0 0}, upload=332531]: Uploading piece...
Mar 18 19:53:23.499215136 (296h53m21.095986159s) [noise=true , address=145.79.227.52:4453 , piece_id=YRAA65GHFFR45DX7MKR5BWBXID3YVSA4DMQDRKWKRDTJCHKHGMOA, seg_pos={0 0}, upload=332531]: Uploading piece...
Mar 18 19:53:23.499687663 (296h53m21.096458697s) [noise=true , address=77.90.31.14:4678 , piece_id=NWBLAAUF3GACR3Y6JRTHDEUJ24GMOTZIPSRGJIPTPLIVWBCSYAHA, seg_pos={0 0}, upload=332531]: Uploading piece...
Mar 18 19:53:23.600142173 (296h53m21.196913186s) [upload=332531]: Next returned nil segment
Mar 18 19:53:23.600161212 (296h53m21.196932236s) [upload=332531]: Waiting for error group managing segments...
Mar 18 19:53:23.620885875 (296h53m21.217656899s) [noise=true , address=45.91.116.5:4312 , piece_id=BDB6LYZZH2HXC3WWPDVMDQLCAAGRXBM5OCZU2JQVUJ7E6REVJ5SQ, seg_pos={0 0}, upload=332531]: Done uploading piece. err: <nil>
Mar 18 19:53:23.621154396 (296h53m21.217925412s) [noise=true , address=45.91.116.13:4167 , piece_id=3JWBJU4M3WSUM3D2TVZDGJKBSRXZY5ZQHV3GFSBKOJMLORY5W7IQ, seg_pos={0 0}, upload=332531]: Done uploading piece. err: <nil>
Mar 18 19:53:23.628883995 (296h53m21.225655021s) [noise=true , address=45.91.116.14:4223 , piece_id=X3NSJBVWEZ3F3DUHQOSQSUCECIN7FARCFMRPJR6N7FHIIL3MS3NQ, seg_pos={0 0}, upload=332531]: Done uploading piece. err: <nil>
Mar 18 19:53:23.630970813 (296h53m21.227741838s) [noise=true , address=79.127.219.45:4868 , piece_id=6NDAYV5PFDBOMZKAXKO6LZUODM46MQPQOQIDKBHER3LG4CBEDHKQ, seg_pos={0 0}, upload=332531]: Done uploading piece. err: <nil>
Mar 18 19:53:23.639825090 (296h53m21.236596104s) [noise=true , address=77.90.31.14:4678 , piece_id=NWBLAAUF3GACR3Y6JRTHDEUJ24GMOTZIPSRGJIPTPLIVWBCSYAHA, seg_pos={0 0}, upload=332531]: Done uploading piece. err: <nil>
Mar 18 19:53:23.642754677 (296h53m21.239525692s) [noise=true , address=77.90.31.18:4833 , piece_id=ON3SKTN2H26XEXC43C44MAJIPY32NZLNWRU4PBSRJS6HLMXVNBMA, seg_pos={0 0}, upload=332531]: Done uploading piece. err: <nil>
Mar 18 19:53:24.057422272 (296h53m21.654193306s) [noise=true , address=79.127.205.235:4271 , piece_id=5XULCXAVIG3CQFTYA6Q3TFPRVICQFHQI2MSI2VIRIAGSK5UKKFXQ, seg_pos={0 0}, upload=332531]: Done uploading piece. err: <nil>
Mar 18 19:53:24.074244577 (296h53m21.671015604s) [noise=true , address=193.149.164.67:4003 , piece_id=SOPPFFG22JZWQA5NHYGA2OFHC2KMALZOR3ATBQ6JYWJVXATWLSRQ, seg_pos={0 0}, upload=332531]: Done uploading piece. err: <nil>
Mar 18 19:53:24.097678734 (296h53m21.694449747s) [noise=true , address=79.127.205.237:4230 , piece_id=Y6AOKOD27V5RL6QWUY2ZAJMONWPTYU4TTU5VIAYJ4RVMNQMCRUZQ, seg_pos={0 0}, upload=332531]: Done uploading piece. err: <nil>
Mar 18 19:53:24.109685982 (296h53m21.706457009s) [noise=true , address=193.149.164.68:4066 , piece_id=FY5RBQUD2JCNXJCY2KIK744F4QZFHTW45PBDMASIIP27RWWB7SSA, seg_pos={0 0}, upload=332531]: Done uploading piece. err: <nil>
Mar 18 19:53:24.110659716 (296h53m21.707430744s) [noise=true , address=193.149.164.69:4074 , piece_id=FUGRHBCQHA4YJ2EVCMKVQVITADSVLM3PWZHV2XM3QLRDGU4G2GPA, seg_pos={0 0}, upload=332531]: Done uploading piece. err: <nil>
Mar 18 19:53:24.114541274 (296h53m21.7113123s) [noise=true , address=193.149.164.71:4287 , piece_id=ZE2Q4NHL3KUKRB36IDRMIZ7DIKDYZIBCM2L3PTLXALGL3PINBGAQ, seg_pos={0 0}, upload=332531]: Done uploading piece. err: <nil>
Mar 18 19:53:24.140894470 (296h53m21.737665488s) [noise=true , address=193.149.164.66:4989 , piece_id=DVAJPRB7ABJJGUP4TYMLOGJVDLUARAMSPCG6FE2PVQXIDP2QBPGA, seg_pos={0 0}, upload=332531]: Done uploading piece. err: <nil>
Mar 18 19:54:19.329327996 (296h54m16.926099018s) [noise=true , address=145.79.227.13:4814 , piece_id=3KU6BHQSH4U4XYBSV4I77FCLQKR5RKWFY3QKZ26D5QRCJDQPKTKQ, seg_pos={0 0}, upload=332531]: Done uploading piece. err: ecclient: upload failed (node:12ERtrzMuh8rZPsgebtyScmYvNVwz3UKJr5Nu2hLdu6sn8X3qLa, address:145.79.227.13:4814): protocol: rpc: dial tcp 145.79.227.13:4814: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 19:54:19.329567681 (296h54m16.926338706s) [noise=true , address=145.79.227.52:4453 , piece_id=YRAA65GHFFR45DX7MKR5BWBXID3YVSA4DMQDRKWKRDTJCHKHGMOA, seg_pos={0 0}, upload=332531]: Done uploading piece. err: ecclient: upload failed (node:1BmpsQ9VfhGzmkZHvSeMptcGhzGpdoN4oYvi83qZcC1Vh9anhP, address:145.79.227.52:4453): protocol: rpc: dial tcp 145.79.227.52:4453: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 19:54:19.329755822 (296h54m16.926526848s) [noise=true , address=145.79.227.41:4105 , piece_id=RTYHA5YP42CFAYVM6PHY5RTB47GRMWSZEBBPNLC532CXKPBT2VXA, seg_pos={0 0}, upload=332531]: Done uploading piece. err: ecclient: upload failed (node:1TVtMQNVV2kERp2aZwmS8E9JUiua1RCyFpZgFACRqH1Nfscw4C, address:145.79.227.41:4105): protocol: rpc: dial tcp 145.79.227.41:4105: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 19:54:19.330021146 (296h54m16.926792175s) [noise=true , address=145.79.227.22:4516 , piece_id=OES5UD4FMNBSPC4ZSDS45SRXLTAQT3QNGR6JBI5D5Y363JFGJ73Q, seg_pos={0 0}, upload=332531]: Done uploading piece. err: ecclient: upload failed (node:12Lhy3AxzYoF25AxWdvjrrV5NNCwP1cDqkEvv4tz9t9KZmF8WPK, address:145.79.227.22:4516): protocol: rpc: dial tcp 145.79.227.22:4516: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 19:54:19.330213680 (296h54m16.926984707s) [noise=true , address=145.79.227.37:4590 , piece_id=ZTQLQ2IZHJFZWBHMRT74V7JDZYXF5BG4656VBZDJ4J6NGKS2TMOA, seg_pos={0 0}, upload=332531]: Done uploading piece. err: ecclient: upload failed (node:1KnNqY1WNwMSFXFrQsWPAhivUTgSn7dKTx1mTTzYG2UxGCkasj, address:145.79.227.37:4590): protocol: rpc: dial tcp 145.79.227.37:4590: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 19:54:19.330250832 (296h54m16.927021848s) [seg_pos={0 0}, upload=332531]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil>
Mar 18 19:54:19.351979131 (296h54m16.948750155s) [upload=332531]: Done uploading segments...
Mar 18 19:54:19.352025297 (296h54m16.948796327s) [upload=332531]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13)
Duration: 20.110436042s
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
31 out of 60 is failed (threshold=6) [.........................XXXXXXXXX.XXX...XXXXXXXXXXXXXXXXXXX] duration:
5.001025095s trace:
https://jaeger.dev.storj.io/trace/43a97b56e5475cdf
FAIL (2026-03-18 19:54:25 UTC)
operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.001051201s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)antiblip:
5 out of 60 is failed (threshold=3) [.......................................X..........XX..X....X] duration:
1m0.024928908s trace:
https://jaeger.dev.storj.io/trace/6b5565e31dae52f1
FAIL (2026-03-18 19:54:19 UTC)
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 13)
======LOGS======
Mar 18 19:53:19.380863725 (296h53m16.977634736s) [upload=332531]: Starting upload
Mar 18 19:53:19.380944319 (296h53m16.977715346s) [upload=332531]: Done starting upload
Mar 18 19:53:19.380976457 (296h53m16.977747482s) [upload=332531]: Uploading segments...
Mar 18 19:53:19.381333196 (296h53m16.978104222s) [upload=332531]: Got next segment. Inline: false
Mar 18 19:53:19.381379068 (296h53m16.978150099s) [upload=332531]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Mar 18 19:53:19.381386543 (296h53m16.978157579s) [upload=332531]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Mar 18 19:53:23.490845546 (296h53m21.087616603s) [seg_pos={0 0}, upload=332531]: Begin upload segment...
Mar 18 19:53:23.490917951 (296h53m21.087688988s) [seg_pos={0 0}, upload=332531]: Done begin upload segment.
Mar 18 19:53:23.490955926 (296h53m21.087726957s) [noise=true , address=45.91.116.5:4312 , piece_id=BDB6LYZZH2HXC3WWPDVMDQLCAAGRXBM5OCZU2JQVUJ7E6REVJ5SQ, seg_pos={0 0}, upload=332531]: Uploading piece...
Mar 18 19:53:23.492093139 (296h53m21.088864171s) [noise=true , address=193.149.164.67:4003 , piece_id=SOPPFFG22JZWQA5NHYGA2OFHC2KMALZOR3ATBQ6JYWJVXATWLSRQ, seg_pos={0 0}, upload=332531]: Uploading piece...
Mar 18 19:53:23.492835295 (296h53m21.089606329s) [noise=true , address=77.90.31.18:4833 , piece_id=ON3SKTN2H26XEXC43C44MAJIPY32NZLNWRU4PBSRJS6HLMXVNBMA, seg_pos={0 0}, upload=332531]: Uploading piece...
Mar 18 19:53:23.492958664 (296h53m21.089729699s) [noise=true , address=45.91.116.13:4167 , piece_id=3JWBJU4M3WSUM3D2TVZDGJKBSRXZY5ZQHV3GFSBKOJMLORY5W7IQ, seg_pos={0 0}, upload=332531]: Uploading piece...
Mar 18 19:53:23.493624498 (296h53m21.090395533s) [noise=true , address=145.79.227.13:4814 , piece_id=3KU6BHQSH4U4XYBSV4I77FCLQKR5RKWFY3QKZ26D5QRCJDQPKTKQ, seg_pos={0 0}, upload=332531]: Uploading piece...
Mar 18 19:53:23.493650442 (296h53m21.090421482s) [noise=true , address=79.127.205.237:4230 , piece_id=Y6AOKOD27V5RL6QWUY2ZAJMONWPTYU4TTU5VIAYJ4RVMNQMCRUZQ, seg_pos={0 0}, upload=332531]: Uploading piece...
Mar 18 19:53:23.494253839 (296h53m21.091024872s) [noise=true , address=45.91.116.14:4223 , piece_id=X3NSJBVWEZ3F3DUHQOSQSUCECIN7FARCFMRPJR6N7FHIIL3MS3NQ, seg_pos={0 0}, upload=332531]: Uploading piece...
Mar 18 19:53:23.494267618 (296h53m21.091038645s) [noise=true , address=145.79.227.22:4516 , piece_id=OES5UD4FMNBSPC4ZSDS45SRXLTAQT3QNGR6JBI5D5Y363JFGJ73Q, seg_pos={0 0}, upload=332531]: Uploading piece...
Mar 18 19:53:23.494879425 (296h53m21.091650453s) [noise=true , address=193.149.164.71:4287 , piece_id=ZE2Q4NHL3KUKRB36IDRMIZ7DIKDYZIBCM2L3PTLXALGL3PINBGAQ, seg_pos={0 0}, upload=332531]: Uploading piece...
Mar 18 19:53:23.494911912 (296h53m21.091682944s) [noise=true , address=193.149.164.69:4074 , piece_id=FUGRHBCQHA4YJ2EVCMKVQVITADSVLM3PWZHV2XM3QLRDGU4G2GPA, seg_pos={0 0}, upload=332531]: Uploading piece...
Mar 18 19:53:23.495497539 (296h53m21.09226857s) [noise=true , address=193.149.164.68:4066 , piece_id=FY5RBQUD2JCNXJCY2KIK744F4QZFHTW45PBDMASIIP27RWWB7SSA, seg_pos={0 0}, upload=332531]: Uploading piece...
Mar 18 19:53:23.496168841 (296h53m21.092939872s) [noise=true , address=145.79.227.41:4105 , piece_id=RTYHA5YP42CFAYVM6PHY5RTB47GRMWSZEBBPNLC532CXKPBT2VXA, seg_pos={0 0}, upload=332531]: Uploading piece...
Mar 18 19:53:23.497008065 (296h53m21.093779105s) [noise=true , address=79.127.205.235:4271 , piece_id=5XULCXAVIG3CQFTYA6Q3TFPRVICQFHQI2MSI2VIRIAGSK5UKKFXQ, seg_pos={0 0}, upload=332531]: Uploading piece...
Mar 18 19:53:23.497605357 (296h53m21.094376396s) [noise=true , address=145.79.227.37:4590 , piece_id=ZTQLQ2IZHJFZWBHMRT74V7JDZYXF5BG4656VBZDJ4J6NGKS2TMOA, seg_pos={0 0}, upload=332531]: Uploading piece...
Mar 18 19:53:23.498200357 (296h53m21.094971388s) [noise=true , address=193.149.164.66:4989 , piece_id=DVAJPRB7ABJJGUP4TYMLOGJVDLUARAMSPCG6FE2PVQXIDP2QBPGA, seg_pos={0 0}, upload=332531]: Uploading piece...
Mar 18 19:53:23.498757906 (296h53m21.095528939s) [noise=true , address=79.127.219.45:4868 , piece_id=6NDAYV5PFDBOMZKAXKO6LZUODM46MQPQOQIDKBHER3LG4CBEDHKQ, seg_pos={0 0}, upload=332531]: Uploading piece...
Mar 18 19:53:23.499215136 (296h53m21.095986159s) [noise=true , address=145.79.227.52:4453 , piece_id=YRAA65GHFFR45DX7MKR5BWBXID3YVSA4DMQDRKWKRDTJCHKHGMOA, seg_pos={0 0}, upload=332531]: Uploading piece...
Mar 18 19:53:23.499687663 (296h53m21.096458697s) [noise=true , address=77.90.31.14:4678 , piece_id=NWBLAAUF3GACR3Y6JRTHDEUJ24GMOTZIPSRGJIPTPLIVWBCSYAHA, seg_pos={0 0}, upload=332531]: Uploading piece...
Mar 18 19:53:23.600142173 (296h53m21.196913186s) [upload=332531]: Next returned nil segment
Mar 18 19:53:23.600161212 (296h53m21.196932236s) [upload=332531]: Waiting for error group managing segments...
Mar 18 19:53:23.620885875 (296h53m21.217656899s) [noise=true , address=45.91.116.5:4312 , piece_id=BDB6LYZZH2HXC3WWPDVMDQLCAAGRXBM5OCZU2JQVUJ7E6REVJ5SQ, seg_pos={0 0}, upload=332531]: Done uploading piece. err: <nil>
Mar 18 19:53:23.621154396 (296h53m21.217925412s) [noise=true , address=45.91.116.13:4167 , piece_id=3JWBJU4M3WSUM3D2TVZDGJKBSRXZY5ZQHV3GFSBKOJMLORY5W7IQ, seg_pos={0 0}, upload=332531]: Done uploading piece. err: <nil>
Mar 18 19:53:23.628883995 (296h53m21.225655021s) [noise=true , address=45.91.116.14:4223 , piece_id=X3NSJBVWEZ3F3DUHQOSQSUCECIN7FARCFMRPJR6N7FHIIL3MS3NQ, seg_pos={0 0}, upload=332531]: Done uploading piece. err: <nil>
Mar 18 19:53:23.630970813 (296h53m21.227741838s) [noise=true , address=79.127.219.45:4868 , piece_id=6NDAYV5PFDBOMZKAXKO6LZUODM46MQPQOQIDKBHER3LG4CBEDHKQ, seg_pos={0 0}, upload=332531]: Done uploading piece. err: <nil>
Mar 18 19:53:23.639825090 (296h53m21.236596104s) [noise=true , address=77.90.31.14:4678 , piece_id=NWBLAAUF3GACR3Y6JRTHDEUJ24GMOTZIPSRGJIPTPLIVWBCSYAHA, seg_pos={0 0}, upload=332531]: Done uploading piece. err: <nil>
Mar 18 19:53:23.642754677 (296h53m21.239525692s) [noise=true , address=77.90.31.18:4833 , piece_id=ON3SKTN2H26XEXC43C44MAJIPY32NZLNWRU4PBSRJS6HLMXVNBMA, seg_pos={0 0}, upload=332531]: Done uploading piece. err: <nil>
Mar 18 19:53:24.057422272 (296h53m21.654193306s) [noise=true , address=79.127.205.235:4271 , piece_id=5XULCXAVIG3CQFTYA6Q3TFPRVICQFHQI2MSI2VIRIAGSK5UKKFXQ, seg_pos={0 0}, upload=332531]: Done uploading piece. err: <nil>
Mar 18 19:53:24.074244577 (296h53m21.671015604s) [noise=true , address=193.149.164.67:4003 , piece_id=SOPPFFG22JZWQA5NHYGA2OFHC2KMALZOR3ATBQ6JYWJVXATWLSRQ, seg_pos={0 0}, upload=332531]: Done uploading piece. err: <nil>
Mar 18 19:53:24.097678734 (296h53m21.694449747s) [noise=true , address=79.127.205.237:4230 , piece_id=Y6AOKOD27V5RL6QWUY2ZAJMONWPTYU4TTU5VIAYJ4RVMNQMCRUZQ, seg_pos={0 0}, upload=332531]: Done uploading piece. err: <nil>
Mar 18 19:53:24.109685982 (296h53m21.706457009s) [noise=true , address=193.149.164.68:4066 , piece_id=FY5RBQUD2JCNXJCY2KIK744F4QZFHTW45PBDMASIIP27RWWB7SSA, seg_pos={0 0}, upload=332531]: Done uploading piece. err: <nil>
Mar 18 19:53:24.110659716 (296h53m21.707430744s) [noise=true , address=193.149.164.69:4074 , piece_id=FUGRHBCQHA4YJ2EVCMKVQVITADSVLM3PWZHV2XM3QLRDGU4G2GPA, seg_pos={0 0}, upload=332531]: Done uploading piece. err: <nil>
Mar 18 19:53:24.114541274 (296h53m21.7113123s) [noise=true , address=193.149.164.71:4287 , piece_id=ZE2Q4NHL3KUKRB36IDRMIZ7DIKDYZIBCM2L3PTLXALGL3PINBGAQ, seg_pos={0 0}, upload=332531]: Done uploading piece. err: <nil>
Mar 18 19:53:24.140894470 (296h53m21.737665488s) [noise=true , address=193.149.164.66:4989 , piece_id=DVAJPRB7ABJJGUP4TYMLOGJVDLUARAMSPCG6FE2PVQXIDP2QBPGA, seg_pos={0 0}, upload=332531]: Done uploading piece. err: <nil>
Mar 18 19:54:19.329327996 (296h54m16.926099018s) [noise=true , address=145.79.227.13:4814 , piece_id=3KU6BHQSH4U4XYBSV4I77FCLQKR5RKWFY3QKZ26D5QRCJDQPKTKQ, seg_pos={0 0}, upload=332531]: Done uploading piece. err: ecclient: upload failed (node:12ERtrzMuh8rZPsgebtyScmYvNVwz3UKJr5Nu2hLdu6sn8X3qLa, address:145.79.227.13:4814): protocol: rpc: dial tcp 145.79.227.13:4814: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 19:54:19.329567681 (296h54m16.926338706s) [noise=true , address=145.79.227.52:4453 , piece_id=YRAA65GHFFR45DX7MKR5BWBXID3YVSA4DMQDRKWKRDTJCHKHGMOA, seg_pos={0 0}, upload=332531]: Done uploading piece. err: ecclient: upload failed (node:1BmpsQ9VfhGzmkZHvSeMptcGhzGpdoN4oYvi83qZcC1Vh9anhP, address:145.79.227.52:4453): protocol: rpc: dial tcp 145.79.227.52:4453: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 19:54:19.329755822 (296h54m16.926526848s) [noise=true , address=145.79.227.41:4105 , piece_id=RTYHA5YP42CFAYVM6PHY5RTB47GRMWSZEBBPNLC532CXKPBT2VXA, seg_pos={0 0}, upload=332531]: Done uploading piece. err: ecclient: upload failed (node:1TVtMQNVV2kERp2aZwmS8E9JUiua1RCyFpZgFACRqH1Nfscw4C, address:145.79.227.41:4105): protocol: rpc: dial tcp 145.79.227.41:4105: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 19:54:19.330021146 (296h54m16.926792175s) [noise=true , address=145.79.227.22:4516 , piece_id=OES5UD4FMNBSPC4ZSDS45SRXLTAQT3QNGR6JBI5D5Y363JFGJ73Q, seg_pos={0 0}, upload=332531]: Done uploading piece. err: ecclient: upload failed (node:12Lhy3AxzYoF25AxWdvjrrV5NNCwP1cDqkEvv4tz9t9KZmF8WPK, address:145.79.227.22:4516): protocol: rpc: dial tcp 145.79.227.22:4516: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 19:54:19.330213680 (296h54m16.926984707s) [noise=true , address=145.79.227.37:4590 , piece_id=ZTQLQ2IZHJFZWBHMRT74V7JDZYXF5BG4656VBZDJ4J6NGKS2TMOA, seg_pos={0 0}, upload=332531]: Done uploading piece. err: ecclient: upload failed (node:1KnNqY1WNwMSFXFrQsWPAhivUTgSn7dKTx1mTTzYG2UxGCkasj, address:145.79.227.37:4590): protocol: rpc: dial tcp 145.79.227.37:4590: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 19:54:19.330250832 (296h54m16.927021848s) [seg_pos={0 0}, upload=332531]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil>
Mar 18 19:54:19.351979131 (296h54m16.948750155s) [upload=332531]: Done uploading segments...
Mar 18 19:54:19.352025297 (296h54m16.948796327s) [upload=332531]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13)
Duration: 1m0.02497161s
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
25 out of 60 is failed (threshold=3) [...................................XXXXXXXXXXXXXXXXXXXXXXXXX] duration:
30.02493244s trace:
https://jaeger.dev.storj.io/trace/467795f623f6be20
FAIL (2026-03-18 19:53:56 UTC)
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 8)
======LOGS======
Mar 18 19:53:26.618698679 (296h53m24.215469714s) [upload=332534]: Starting upload
Mar 18 19:53:26.618786764 (296h53m24.215557792s) [upload=332534]: Done starting upload
Mar 18 19:53:26.619059590 (296h53m24.215830612s) [upload=332534]: Uploading segments...
Mar 18 19:53:26.619323023 (296h53m24.216094035s) [upload=332534]: Got next segment. Inline: false
Mar 18 19:53:26.619460574 (296h53m24.2162316s) [upload=332534]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Mar 18 19:53:26.619467531 (296h53m24.216238556s) [upload=332534]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Mar 18 19:53:30.490385751 (296h53m28.087156764s) [seg_pos={0 0}, upload=332534]: Begin upload segment...
Mar 18 19:53:30.490485378 (296h53m28.087256408s) [seg_pos={0 0}, upload=332534]: Done begin upload segment.
Mar 18 19:53:30.490565739 (296h53m28.087336764s) [noise=true , address=145.79.227.29:4946 , piece_id=V5IUDDZL2VGSOMY3KGLIIHFO7JTU4PU6SKLPRB4AWBBPWSGJ6O6A, seg_pos={0 0}, upload=332534]: Uploading piece...
Mar 18 19:53:30.491285695 (296h53m28.088056724s) [noise=true , address=145.79.227.12:4771 , piece_id=DEBQZRXNNDQA2U3KAWKHV36IFXSLKLXW42LHZ4EKKBIPZKIXIS6A, seg_pos={0 0}, upload=332534]: Uploading piece...
Mar 18 19:53:30.491995341 (296h53m28.088766376s) [noise=true , address=145.79.227.22:4512 , piece_id=NPKDW5W4CQZ4DCPREKOPSUDD2YFIHBMG2JGJDIUBQN3XGDI7HZZQ, seg_pos={0 0}, upload=332534]: Uploading piece...
Mar 18 19:53:30.492716682 (296h53m28.089487706s) [noise=true , address=145.79.227.45:4233 , piece_id=PSA5TSYHZDW4YP7KCL4VW3TNUZPMP6H3HN3IFOPECZOXMNQ5VDZQ, seg_pos={0 0}, upload=332534]: Uploading piece...
Mar 18 19:53:30.493226560 (296h53m28.089997598s) [noise=true , address=45.91.116.9:4295 , piece_id=JDCBDJIHC7VNIW2LIR7VDJ7J7F2ZNB6N6URFNN4RVMF3AAF5CXEA, seg_pos={0 0}, upload=332534]: Uploading piece...
Mar 18 19:53:30.493311749 (296h53m28.090082781s) [noise=true , address=145.79.227.4:4190 , piece_id=CTHKDHBXEWGNX4ZP7MRTFDB2UAHPHDDI4QY7H4K4RKVR2ZH77Z7Q, seg_pos={0 0}, upload=332534]: Uploading piece...
Mar 18 19:53:30.493896307 (296h53m28.09066734s) [noise=true , address=77.90.31.14:4666 , piece_id=CWHEQCVBM7MCZVBZOEHMC7TO3VOHT6ELNPFHE2T5NXGWFIVMAZQQ, seg_pos={0 0}, upload=332534]: Uploading piece...
Mar 18 19:53:30.493952466 (296h53m28.090723499s) [noise=true , address=77.90.31.19:4841 , piece_id=M34S7ECE4WHKWBCGLTPEZZMYOXIM3CRVKWHJ7YVCYTMTL2BC5TLQ, seg_pos={0 0}, upload=332534]: Uploading piece...
Mar 18 19:53:30.494570922 (296h53m28.091341956s) [noise=true , address=45.91.116.12:4227 , piece_id=JMFZIFQUQFBA5CP2KWFWPZE5JQTKUHTXCBMOAX4FBFGCZYNP3B4Q, seg_pos={0 0}, upload=332534]: Uploading piece...
Mar 18 19:53:30.494733964 (296h53m28.09150501s) [noise=true , address=145.79.227.42:4126 , piece_id=KTVGUIRA23WVDR5CVAR3BYZU2ROR45QXXYMD66J4SEAQG3XMBUZA, seg_pos={0 0}, upload=332534]: Uploading piece...
Mar 18 19:53:30.495449022 (296h53m28.092220061s) [noise=true , address=145.79.227.52:4445 , piece_id=TZRAATIZCO7MOZ6XEUZLY7RG5F6TZMLLQEXDFU54ARJBCGS4ILLQ, seg_pos={0 0}, upload=332534]: Uploading piece...
Mar 18 19:53:30.496103532 (296h53m28.092874574s) [noise=true , address=77.90.31.21:4881 , piece_id=HEJ7YBYRXPKVJITM7KH6OESRM2C4ZIOLLXLKJ6CW3ZNILS3K3M2Q, seg_pos={0 0}, upload=332534]: Uploading piece...
Mar 18 19:53:30.496160431 (296h53m28.092931464s) [noise=true , address=145.79.227.21:4474 , piece_id=3TCZOENVDA7KNX2LLLWH7LHTZKSWHEDNYPBZGBBM2TKSGXXUIK3A, seg_pos={0 0}, upload=332534]: Uploading piece...
Mar 18 19:53:30.496703168 (296h53m28.0934742s) [noise=true , address=77.90.31.17:4770 , piece_id=IULCBQZSOXEUGUK62OB2OVGS7EYDL53GADZT32UUWPDE5FQ7XJ3A, seg_pos={0 0}, upload=332534]: Uploading piece...
Mar 18 19:53:30.496776611 (296h53m28.093547653s) [noise=true , address=45.91.116.4:4955 , piece_id=OU7FTVU6SCGLAOFV4YG5TO2K3MQGELPT6RBQZ4RRN7NMSXOHRSCA, seg_pos={0 0}, upload=332534]: Uploading piece...
Mar 18 19:53:30.497350782 (296h53m28.094121808s) [noise=true , address=77.90.31.10:4521 , piece_id=YSTOKWPPJUSULE36P5OZJODOXEGZ3IUIWAN6A6DXDUH4SRFB7AWQ, seg_pos={0 0}, upload=332534]: Uploading piece...
Mar 18 19:53:30.602353353 (296h53m28.199124389s) [upload=332534]: Next returned nil segment
Mar 18 19:53:30.602372353 (296h53m28.199143388s) [upload=332534]: Waiting for error group managing segments...
Mar 18 19:53:30.613081239 (296h53m28.20985226s) [noise=true , address=45.91.116.4:4955 , piece_id=OU7FTVU6SCGLAOFV4YG5TO2K3MQGELPT6RBQZ4RRN7NMSXOHRSCA, seg_pos={0 0}, upload=332534]: Done uploading piece. err: <nil>
Mar 18 19:53:30.616635484 (296h53m28.213406513s) [noise=true , address=77.90.31.19:4841 , piece_id=M34S7ECE4WHKWBCGLTPEZZMYOXIM3CRVKWHJ7YVCYTMTL2BC5TLQ, seg_pos={0 0}, upload=332534]: Done uploading piece. err: <nil>
Mar 18 19:53:30.620678205 (296h53m28.217449239s) [noise=true , address=45.91.116.12:4227 , piece_id=JMFZIFQUQFBA5CP2KWFWPZE5JQTKUHTXCBMOAX4FBFGCZYNP3B4Q, seg_pos={0 0}, upload=332534]: Done uploading piece. err: <nil>
Mar 18 19:53:30.625035238 (296h53m28.221806273s) [noise=true , address=45.91.116.9:4295 , piece_id=JDCBDJIHC7VNIW2LIR7VDJ7J7F2ZNB6N6URFNN4RVMF3AAF5CXEA, seg_pos={0 0}, upload=332534]: Done uploading piece. err: <nil>
Mar 18 19:53:30.628880388 (296h53m28.225651424s) [noise=true , address=77.90.31.21:4881 , piece_id=HEJ7YBYRXPKVJITM7KH6OESRM2C4ZIOLLXLKJ6CW3ZNILS3K3M2Q, seg_pos={0 0}, upload=332534]: Done uploading piece. err: <nil>
Mar 18 19:53:30.633045911 (296h53m28.229816945s) [noise=true , address=77.90.31.10:4521 , piece_id=YSTOKWPPJUSULE36P5OZJODOXEGZ3IUIWAN6A6DXDUH4SRFB7AWQ, seg_pos={0 0}, upload=332534]: Done uploading piece. err: <nil>
Mar 18 19:53:30.633161840 (296h53m28.229932867s) [noise=true , address=77.90.31.14:4666 , piece_id=CWHEQCVBM7MCZVBZOEHMC7TO3VOHT6ELNPFHE2T5NXGWFIVMAZQQ, seg_pos={0 0}, upload=332534]: Done uploading piece. err: <nil>
Mar 18 19:53:30.637272868 (296h53m28.234043882s) [noise=true , address=77.90.31.17:4770 , piece_id=IULCBQZSOXEUGUK62OB2OVGS7EYDL53GADZT32UUWPDE5FQ7XJ3A, seg_pos={0 0}, upload=332534]: Done uploading piece. err: <nil>
Mar 18 19:53:56.589075805 (296h53m54.185846836s) [noise=true , address=145.79.227.4:4190 , piece_id=CTHKDHBXEWGNX4ZP7MRTFDB2UAHPHDDI4QY7H4K4RKVR2ZH77Z7Q, seg_pos={0 0}, upload=332534]: Done uploading piece. err: ecclient: upload failed (node:1eNJzs5Dc7Nv7imoQXZDp2mEVWjD8o4viGVkFvuAeZ1arynxu4, address:145.79.227.4:4190): protocol: rpc: dial tcp 145.79.227.4:4190: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 19:53:56.589145813 (296h53m54.185916846s) [noise=true , address=145.79.227.29:4946 , piece_id=V5IUDDZL2VGSOMY3KGLIIHFO7JTU4PU6SKLPRB4AWBBPWSGJ6O6A, seg_pos={0 0}, upload=332534]: Done uploading piece. err: ecclient: upload failed (node:12v9Q7aigjWPNXyshHmQt9YKrDu5af1vd1TxkDeHeKixSrWgrLX, address:145.79.227.29:4946): protocol: rpc: dial tcp 145.79.227.29:4946: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 19:53:56.589548703 (296h53m54.186319735s) [noise=true , address=145.79.227.52:4445 , piece_id=TZRAATIZCO7MOZ6XEUZLY7RG5F6TZMLLQEXDFU54ARJBCGS4ILLQ, seg_pos={0 0}, upload=332534]: Done uploading piece. err: ecclient: upload failed (node:12vXrkhLqCofhR4Fmfqb98gxnqWQF6vMV4q3ATbRvPfcqi82WSo, address:145.79.227.52:4445): protocol: rpc: dial tcp 145.79.227.52:4445: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 19:53:56.589649005 (296h53m54.186420037s) [noise=true , address=145.79.227.21:4474 , piece_id=3TCZOENVDA7KNX2LLLWH7LHTZKSWHEDNYPBZGBBM2TKSGXXUIK3A, seg_pos={0 0}, upload=332534]: Done uploading piece. err: ecclient: upload failed (node:1rL2DnjSgHP36Md3bCgy5padqTAhv3iQCeURbLMuuuMBxtUNZH, address:145.79.227.21:4474): protocol: rpc: dial tcp 145.79.227.21:4474: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 19:53:56.589840589 (296h53m54.186611621s) [noise=true , address=145.79.227.42:4126 , piece_id=KTVGUIRA23WVDR5CVAR3BYZU2ROR45QXXYMD66J4SEAQG3XMBUZA, seg_pos={0 0}, upload=332534]: Done uploading piece. err: ecclient: upload failed (node:12dMWFrq2NMpnMzuPAphWerntC5kkrcnzRJZBA116uzJzyvk6fH, address:145.79.227.42:4126): protocol: rpc: dial tcp 145.79.227.42:4126: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 19:53:56.590019749 (296h53m54.18679078s) [noise=true , address=145.79.227.12:4771 , piece_id=DEBQZRXNNDQA2U3KAWKHV36IFXSLKLXW42LHZ4EKKBIPZKIXIS6A, seg_pos={0 0}, upload=332534]: Done uploading piece. err: ecclient: upload failed (node:1rcgrMxg2fRfvh7EA1eUehzD5rSH1krUd5W5eC8KTo7vjYWFLL, address:145.79.227.12:4771): protocol: context deadline exceeded; context deadline exceeded
Mar 18 19:53:56.590275955 (296h53m54.187046987s) [noise=true , address=145.79.227.45:4233 , piece_id=PSA5TSYHZDW4YP7KCL4VW3TNUZPMP6H3HN3IFOPECZOXMNQ5VDZQ, seg_pos={0 0}, upload=332534]: Done uploading piece. err: ecclient: upload failed (node:1DmQeoQWTnFhQeJFseJoNYzFyo7vZjYKwDtQVJ6pomksWExsTb, address:145.79.227.45:4233): protocol: rpc: dial tcp 145.79.227.45:4233: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 19:53:56.590466677 (296h53m54.187237692s) [noise=true , address=145.79.227.22:4512 , piece_id=NPKDW5W4CQZ4DCPREKOPSUDD2YFIHBMG2JGJDIUBQN3XGDI7HZZQ, seg_pos={0 0}, upload=332534]: Done uploading piece. err: ecclient: upload failed (node:12cnVqYYfo8b6RvRtMykzh8mU7UAqxcqYvWe7H8G7k8DqtNLAR4, address:145.79.227.22:4512): protocol: rpc: dial tcp 145.79.227.22:4512: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 19:53:56.590494703 (296h53m54.187265721s) [seg_pos={0 0}, upload=332534]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil>
Mar 18 19:53:56.608845785 (296h53m54.205616811s) [upload=332534]: Done uploading segments...
Mar 18 19:53:56.608894284 (296h53m54.205665317s) [upload=332534]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8)
Duration: 30.024973622s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
8 out of 60 is failed (threshold=3) [....................................XXXXXX.XX...............] duration:
18.12587328s trace:
https://jaeger.dev.storj.io/trace/4b4d10a8051ad01b
FAIL (2026-03-18 19:53:50 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 18.125959915s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
5 out of 60 is failed (threshold=3) [.....................................XXX..........X...X.....] duration:
19.543771518s trace:
https://jaeger.dev.storj.io/trace/6a01088813e3d8b7
FAIL (2026-03-18 19:53:39 UTC)
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 12)
======LOGS======
m: *pb.BatchRequestItem_ObjectBegin
Mar 18 19:49:08.138517471 (296h49m5.735288494s) [upload=332459]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Mar 18 19:49:16.284286759 (296h49m13.881057773s) [seg_pos={0 0}, upload=332459]: Begin upload segment...
Mar 18 19:49:16.284352223 (296h49m13.88112325s) [seg_pos={0 0}, upload=332459]: Done begin upload segment.
Mar 18 19:49:16.284395789 (296h49m13.881166818s) [noise=true , address=193.149.164.68:4051 , piece_id=FCHPX2FRYUVNH4JAAPKI2Y7DSJCUJ5UG3EASXQYB7VEQPNERYUGQ, seg_pos={0 0}, upload=332459]: Uploading piece...
Mar 18 19:49:16.284449689 (296h49m13.881220728s) [noise=true , address=145.79.227.13:4822 , piece_id=Y5GSC3KFFGLK4B3NCXEMOZTAC3ASNGPJJOPOKCUBUJIL75E6T6FA, seg_pos={0 0}, upload=332459]: Uploading piece...
Mar 18 19:49:16.284549974 (296h49m13.881320988s) [noise=true , address=45.91.116.5:4308 , piece_id=XRXYA347E22AHAJ4XDENA5EHCV2F43HXL67MTUGRYSU6B3WQBDNQ, seg_pos={0 0}, upload=332459]: Uploading piece...
Mar 18 19:49:16.285144837 (296h49m13.88191587s) [noise=true , address=193.149.164.70:4117 , piece_id=PL6FWCO6XQRSVU26FH2KESA26D4Z2PCQVBPBDKBWLBPXHBV5LQVQ, seg_pos={0 0}, upload=332459]: Uploading piece...
Mar 18 19:49:16.285821284 (296h49m13.882592317s) [noise=true , address=145.79.227.52:4451 , piece_id=SBMEAZS5RDQW3RCJP2UU5T7BHF6P7VOICLP5P46YJ5LE2GB65CGA, seg_pos={0 0}, upload=332459]: Uploading piece...
Mar 18 19:49:16.286556627 (296h49m13.883327669s) [noise=true , address=79.127.205.235:4259 , piece_id=N7BGDCO2CR6HLPUWYF35T5OQRWQFEWFG7FPZ6Y33J7FGR5H2ZURQ, seg_pos={0 0}, upload=332459]: Uploading piece...
Mar 18 19:49:16.287228779 (296h49m13.883999813s) [noise=true , address=79.127.219.45:4864 , piece_id=UXHLRN5HUN3ZRXOZEYAJOS6QUN3PVIXXXED3TS6RPWSHXJ4WKS6Q, seg_pos={0 0}, upload=332459]: Uploading piece...
Mar 18 19:49:16.287269957 (296h49m13.884040985s) [noise=true , address=77.90.31.21:4894 , piece_id=IQ6775I4HFC5JQVYUCQPXYWPY3X6QRHVXM6IF4JEZRQMMRRWQDUQ, seg_pos={0 0}, upload=332459]: Uploading piece...
Mar 18 19:49:16.287437026 (296h49m13.884208049s) [noise=true , address=5.231.73.4:4010 , piece_id=LTONWX7UJVP3BPHHKKJUN6PTAR5TVPVKF3CD6NBV2MZWTLUOVQZQ, seg_pos={0 0}, upload=332459]: Uploading piece...
Mar 18 19:49:16.287832890 (296h49m13.884603914s) [noise=true , address=167.17.72.68:4657 , piece_id=UUIV3ERJSVOJT4TISEHCMR6EQCG6PMZUR6OWGK3ZVV33B33CUZCA, seg_pos={0 0}, upload=332459]: Uploading piece...
Mar 18 19:49:16.287952922 (296h49m13.884723957s) [noise=true , address=79.127.219.41:4939 , piece_id=WK6HMSTVM7LD2P5MPCKML4MBFZJX2QJWZS6SYPITDKNOLPMRPO6Q, seg_pos={0 0}, upload=332459]: Uploading piece...
Mar 18 19:49:16.288418812 (296h49m13.885189842s) [noise=true , address=145.79.227.44:4213 , piece_id=MK3BJSFCDOG2MAJYO2BAE2JG62EVEZMCCUATBBBT4QU44GLKK3QA, seg_pos={0 0}, upload=332459]: Uploading piece...
Mar 18 19:49:16.288922438 (296h49m13.885693461s) [noise=true , address=145.79.227.3:4163 , piece_id=3R3QBIJ2XDQZQXJ66RGNZBWZB6V4ERS5RJEZFBTS2LFMQ25G4FLA, seg_pos={0 0}, upload=332459]: Uploading piece...
Mar 18 19:49:16.289343580 (296h49m13.886114604s) [noise=true , address=193.149.164.71:4310 , piece_id=BKUA4WC73DSXJMEPCMTXTUD2PVQAAYD44MAGEB3SS3FEDJO7377A, seg_pos={0 0}, upload=332459]: Uploading piece...
Mar 18 19:49:16.289780590 (296h49m13.88655162s) [noise=true , address=145.79.227.49:4358 , piece_id=QSE44UR5YBYTJHWGOCBFVQPAUZ3SJV6P5WKBEF3MTFAKVRH2QSHQ, seg_pos={0 0}, upload=332459]: Uploading piece...
Mar 18 19:49:16.290237004 (296h49m13.887008027s) [noise=true , address=77.90.31.14:4685 , piece_id=3KAIZVB2LH6R57CFO55R7BJV7G4MGHFN6M5VTKT3J6LMI5F7PVAA, seg_pos={0 0}, upload=332459]: Uploading piece...
Mar 18 19:49:16.292879809 (296h49m13.889650843s) [noise=true , address=77.90.31.18:4817 , piece_id=FMVKZ4Z3YDPWZHTXU3ZTSWXMVUWLKQX7TUWXPQ4LOKBQK64PHF5Q, seg_pos={0 0}, upload=332459]: Uploading piece...
Mar 18 19:49:16.293664670 (296h49m13.890435693s) [noise=true , address=145.79.227.15:4908 , piece_id=LG24OMYBUVPVRBFZL4UUJLPBVNDTVVHQ4HAYTW2PGKHUKZPF2WCA, seg_pos={0 0}, upload=332459]: Uploading piece...
Mar 18 19:49:16.351440712 (296h49m13.948211724s) [upload=332459]: Next returned nil segment
Mar 18 19:49:16.351460639 (296h49m13.948231661s) [upload=332459]: Waiting for error group managing segments...
Mar 18 19:49:16.355941640 (296h49m13.952712664s) [noise=true , address=77.90.31.21:4894 , piece_id=IQ6775I4HFC5JQVYUCQPXYWPY3X6QRHVXM6IF4JEZRQMMRRWQDUQ, seg_pos={0 0}, upload=332459]: Done uploading piece. err: <nil>
Mar 18 19:49:16.356991929 (296h49m13.953762955s) [noise=true , address=79.127.219.41:4939 , piece_id=WK6HMSTVM7LD2P5MPCKML4MBFZJX2QJWZS6SYPITDKNOLPMRPO6Q, seg_pos={0 0}, upload=332459]: Done uploading piece. err: <nil>
Mar 18 19:49:16.361809763 (296h49m13.958580786s) [noise=true , address=45.91.116.5:4308 , piece_id=XRXYA347E22AHAJ4XDENA5EHCV2F43HXL67MTUGRYSU6B3WQBDNQ, seg_pos={0 0}, upload=332459]: Done uploading piece. err: <nil>
Mar 18 19:49:16.367502365 (296h49m13.964273396s) [noise=true , address=77.90.31.14:4685 , piece_id=3KAIZVB2LH6R57CFO55R7BJV7G4MGHFN6M5VTKT3J6LMI5F7PVAA, seg_pos={0 0}, upload=332459]: Done uploading piece. err: <nil>
Mar 18 19:49:16.381549888 (296h49m13.97832091s) [noise=true , address=79.127.219.45:4864 , piece_id=UXHLRN5HUN3ZRXOZEYAJOS6QUN3PVIXXXED3TS6RPWSHXJ4WKS6Q, seg_pos={0 0}, upload=332459]: Done uploading piece. err: <nil>
Mar 18 19:49:16.387503503 (296h49m13.984274527s) [noise=true , address=77.90.31.18:4817 , piece_id=FMVKZ4Z3YDPWZHTXU3ZTSWXMVUWLKQX7TUWXPQ4LOKBQK64PHF5Q, seg_pos={0 0}, upload=332459]: Done uploading piece. err: <nil>
Mar 18 19:49:16.498139242 (296h49m14.094910266s) [noise=true , address=5.231.73.4:4010 , piece_id=LTONWX7UJVP3BPHHKKJUN6PTAR5TVPVKF3CD6NBV2MZWTLUOVQZQ, seg_pos={0 0}, upload=332459]: Done uploading piece. err: <nil>
Mar 18 19:49:16.852944491 (296h49m14.449715518s) [noise=true , address=167.17.72.68:4657 , piece_id=UUIV3ERJSVOJT4TISEHCMR6EQCG6PMZUR6OWGK3ZVV33B33CUZCA, seg_pos={0 0}, upload=332459]: Done uploading piece. err: <nil>
Mar 18 19:49:16.878068834 (296h49m14.474839867s) [noise=true , address=79.127.205.235:4259 , piece_id=N7BGDCO2CR6HLPUWYF35T5OQRWQFEWFG7FPZ6Y33J7FGR5H2ZURQ, seg_pos={0 0}, upload=332459]: Done uploading piece. err: <nil>
Mar 18 19:49:16.892446253 (296h49m14.489217282s) [noise=true , address=193.149.164.68:4051 , piece_id=FCHPX2FRYUVNH4JAAPKI2Y7DSJCUJ5UG3EASXQYB7VEQPNERYUGQ, seg_pos={0 0}, upload=332459]: Done uploading piece. err: <nil>
Mar 18 19:49:16.893099142 (296h49m14.48987017s) [noise=true , address=193.149.164.71:4310 , piece_id=BKUA4WC73DSXJMEPCMTXTUD2PVQAAYD44MAGEB3SS3FEDJO7377A, seg_pos={0 0}, upload=332459]: Done uploading piece. err: <nil>
Mar 18 19:49:17.031110208 (296h49m14.627881232s) [noise=true , address=193.149.164.70:4117 , piece_id=PL6FWCO6XQRSVU26FH2KESA26D4Z2PCQVBPBDKBWLBPXHBV5LQVQ, seg_pos={0 0}, upload=332459]: Done uploading piece. err: <nil>
Mar 18 19:50:08.096628364 (296h50m5.693399398s) [noise=true , address=145.79.227.52:4451 , piece_id=SBMEAZS5RDQW3RCJP2UU5T7BHF6P7VOICLP5P46YJ5LE2GB65CGA, seg_pos={0 0}, upload=332459]: Done uploading piece. err: ecclient: upload failed (node:12bC4P3xBiPFkxLvavGFVbWKnbprCStL6UJgfKSA2Wd2fX7Mbfw, address:145.79.227.52:4451): protocol: rpc: dial tcp 145.79.227.52:4451: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 19:50:08.096712565 (296h50m5.693483599s) [noise=true , address=145.79.227.44:4213 , piece_id=MK3BJSFCDOG2MAJYO2BAE2JG62EVEZMCCUATBBBT4QU44GLKK3QA, seg_pos={0 0}, upload=332459]: Done uploading piece. err: ecclient: upload failed (node:129LRnDm7b4Fo5db7pto8GXYEtxHz9qdPAtQvaYH6gqdkMatQ6g, address:145.79.227.44:4213): protocol: rpc: dial tcp 145.79.227.44:4213: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 19:50:08.096943717 (296h50m5.693714752s) [noise=true , address=145.79.227.49:4358 , piece_id=QSE44UR5YBYTJHWGOCBFVQPAUZ3SJV6P5WKBEF3MTFAKVRH2QSHQ, seg_pos={0 0}, upload=332459]: Done uploading piece. err: ecclient: upload failed (node:1dwCNEcA9o51ZN3UaN3Frxdawwf8DXBj86iQMywjW8XP4ESmw1, address:145.79.227.49:4358): protocol: rpc: dial tcp 145.79.227.49:4358: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 19:50:08.097022451 (296h50m5.693793484s) [noise=true , address=145.79.227.3:4163 , piece_id=3R3QBIJ2XDQZQXJ66RGNZBWZB6V4ERS5RJEZFBTS2LFMQ25G4FLA, seg_pos={0 0}, upload=332459]: Done uploading piece. err: ecclient: upload failed (node:12w88UkGu5duYivJ6L6C9iJmTbnB63fDKiUUQprhwM3JshpXTdC, address:145.79.227.3:4163): protocol: rpc: dial tcp 145.79.227.3:4163: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 19:50:08.097155433 (296h50m5.693926468s) [noise=true , address=145.79.227.13:4822 , piece_id=Y5GSC3KFFGLK4B3NCXEMOZTAC3ASNGPJJOPOKCUBUJIL75E6T6FA, seg_pos={0 0}, upload=332459]: Done uploading piece. err: ecclient: upload failed (node:12fgLN9DGBGxWfBEePSsk84FTrnKoKWwNgoET3FpotfJxJ28f75, address:145.79.227.13:4822): protocol: rpc: dial tcp 145.79.227.13:4822: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 19:50:08.097360007 (296h50m5.694131039s) [noise=true , address=145.79.227.15:4908 , piece_id=LG24OMYBUVPVRBFZL4UUJLPBVNDTVVHQ4HAYTW2PGKHUKZPF2WCA, seg_pos={0 0}, upload=332459]: Done uploading piece. err: ecclient: upload failed (node:1pfQ5v1GwHhFTvh5Y7Jpb9qpWhZhFYqWX9PJpLEUzTkqKWcy4x, address:145.79.227.15:4908): protocol: rpc: dial tcp 145.79.227.15:4908: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 19:50:08.097462766 (296h50m5.69423378s) [seg_pos={0 0}, upload=332459]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil>
Mar 18 19:50:08.117913456 (296h50m5.714684468s) [upload=332459]: Done uploading segments...
Mar 18 19:50:08.117999321 (296h50m5.714770348s) [upload=332459]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12)
Duration: 19.543804228s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
30 out of 60 is failed (threshold=6) [..........................XXXXXXXXX.XXX...XXXXXXXXXXXXXXXXXX] duration:
5.001720647s trace:
https://jaeger.dev.storj.io/trace/7ecafa59586b95fd
FAIL (2026-03-18 19:53:24 UTC)
operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.001744831s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
24 out of 60 is failed (threshold=3) [....................................XXXXXXXXXXXXXXXXXXXXXXXX] duration:
30.030457228s trace:
https://jaeger.dev.storj.io/trace/793c8f707c42053d
FAIL (2026-03-18 19:53:16 UTC)
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 6)
======LOGS======
97217 (296h52m44.198968239s) [upload=332523]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Mar 18 19:52:50.685984409 (296h52m48.282755446s) [seg_pos={0 0}, upload=332523]: Begin upload segment...
Mar 18 19:52:50.686052370 (296h52m48.282823403s) [seg_pos={0 0}, upload=332523]: Done begin upload segment.
Mar 18 19:52:50.686094388 (296h52m48.282865424s) [noise=true , address=145.79.227.23:4551 , piece_id=OIJSN73HORIFMDHQNZJSD7GJY6R4TC3DUUICIJMDHNHR4QADPWJA, seg_pos={0 0}, upload=332523]: Uploading piece...
Mar 18 19:52:50.686141795 (296h52m48.282912827s) [noise=true , address=145.79.227.10:4405 , piece_id=SKCT3FGRCXW7QHNZRKBCEYAATJ22QJA4A235434GB5SK6RTPCQYQ, seg_pos={0 0}, upload=332523]: Uploading piece...
Mar 18 19:52:50.686780983 (296h52m48.283552015s) [noise=true , address=145.79.227.34:4905 , piece_id=SSSDFRJONSSMGDYZANLIKTHQ6HBWT6IFYJDAJSSYIHE5TASNFRGQ, seg_pos={0 0}, upload=332523]: Uploading piece...
Mar 18 19:52:50.687586861 (296h52m48.284357895s) [noise=true , address=145.79.227.51:4409 , piece_id=76FS44U7FERWPRO72CQ3EIGPN6RQI4QOCRJH4IQ4M57EPWXBQW2Q, seg_pos={0 0}, upload=332523]: Uploading piece...
Mar 18 19:52:50.688171226 (296h52m48.284942253s) [noise=true , address=145.79.227.22:4512 , piece_id=ES4PBUL6SUOZQ26TOHVGLWOWEQVNUSPXCJPDRMEA6TXPJ2RKIPIQ, seg_pos={0 0}, upload=332523]: Uploading piece...
Mar 18 19:52:50.688733304 (296h52m48.285504335s) [noise=true , address=145.79.227.35:4486 , piece_id=KSSOZCWN5K7QLOGR7ZWQC4MQ5XXPOWTRTMDNJ5TSPL5L7HFIV5JA, seg_pos={0 0}, upload=332523]: Uploading piece...
Mar 18 19:52:50.689899106 (296h52m48.286670143s) [noise=true , address=77.90.31.17:4769 , piece_id=3AOOY4K55WRDB76UWX77SASXANARF7PRVZ3O22SLPWV254FDD3GQ, seg_pos={0 0}, upload=332523]: Uploading piece...
Mar 18 19:52:50.690441439 (296h52m48.287212471s) [noise=true , address=145.79.227.2:4119 , piece_id=INNE55CDRSMDRPWIQ4P53X7J2YUICGJ5QKQ7MAWC5WFXI2226GNA, seg_pos={0 0}, upload=332523]: Uploading piece...
Mar 18 19:52:50.690726022 (296h52m48.287497061s) [noise=true , address=77.90.31.18:4805 , piece_id=EXNTVDMKW2PALZMLLUXRUNZYE5BHTHTQVPJUSTJ367HDAI45E3MQ, seg_pos={0 0}, upload=332523]: Uploading piece...
Mar 18 19:52:50.692044654 (296h52m48.288815793s) [noise=true , address=145.79.227.49:4337 , piece_id=NSSI5XWZMTJ6VUZNU4JTZRK4ZUWAVCTC6F5KCXYORWZNZ2DA52XQ, seg_pos={0 0}, upload=332523]: Uploading piece...
Mar 18 19:52:50.692640365 (296h52m48.289411402s) [noise=true , address=77.90.31.13:4629 , piece_id=ZBWI6MHPNXI7UQ7Q3L3VMGBX4YZK5F56MTO337EZ77LGPLQ3OSGQ, seg_pos={0 0}, upload=332523]: Uploading piece...
Mar 18 19:52:50.693938530 (296h52m48.290709563s) [noise=true , address=79.127.219.41:4926 , piece_id=XZUTTB2A2K4ZWZLND6OJNG3UUFLTITH7CHWOKULUS4RYMF4SI7VQ, seg_pos={0 0}, upload=332523]: Uploading piece...
Mar 18 19:52:50.694697941 (296h52m48.291468974s) [noise=true , address=145.79.227.26:4658 , piece_id=KV2Q7TQPYETS2P5EK3AYCBWXO75IHPGLMYS7T3EY3DKOQ6HO7QVQ, seg_pos={0 0}, upload=332523]: Uploading piece...
Mar 18 19:52:50.696419031 (296h52m48.293190063s) [noise=true , address=145.79.227.7:4296 , piece_id=M7NXQQ4U4EHSE3637Y2OLERXE7WN5YONSCFIF72TWI5VDGPHJMMQ, seg_pos={0 0}, upload=332523]: Uploading piece...
Mar 18 19:52:50.697182528 (296h52m48.293953558s) [noise=true , address=45.91.116.5:4301 , piece_id=XMAWGDY5NVLXLA3GJVXXCCBNEA3BAT24XUP3GUAS4QCSP3YR4AOA, seg_pos={0 0}, upload=332523]: Uploading piece...
Mar 18 19:52:50.699322787 (296h52m48.296093822s) [noise=true , address=77.90.31.14:4663 , piece_id=K7P3ERBAXZAF2VWQRGHXKDOMEPWP4GSBRO4YNSWGPZ2XKM7TDUEA, seg_pos={0 0}, upload=332523]: Uploading piece...
Mar 18 19:52:50.784539936 (296h52m48.381310966s) [upload=332523]: Next returned nil segment
Mar 18 19:52:50.784558647 (296h52m48.381329677s) [upload=332523]: Waiting for error group managing segments...
Mar 18 19:52:50.789045308 (296h52m48.385816331s) [noise=true , address=45.91.116.5:4301 , piece_id=XMAWGDY5NVLXLA3GJVXXCCBNEA3BAT24XUP3GUAS4QCSP3YR4AOA, seg_pos={0 0}, upload=332523]: Done uploading piece. err: <nil>
Mar 18 19:52:50.789094040 (296h52m48.385865065s) [noise=true , address=77.90.31.17:4769 , piece_id=3AOOY4K55WRDB76UWX77SASXANARF7PRVZ3O22SLPWV254FDD3GQ, seg_pos={0 0}, upload=332523]: Done uploading piece. err: <nil>
Mar 18 19:52:50.798203404 (296h52m48.394974431s) [noise=true , address=77.90.31.18:4805 , piece_id=EXNTVDMKW2PALZMLLUXRUNZYE5BHTHTQVPJUSTJ367HDAI45E3MQ, seg_pos={0 0}, upload=332523]: Done uploading piece. err: <nil>
Mar 18 19:52:50.799473878 (296h52m48.396244912s) [noise=true , address=77.90.31.13:4629 , piece_id=ZBWI6MHPNXI7UQ7Q3L3VMGBX4YZK5F56MTO337EZ77LGPLQ3OSGQ, seg_pos={0 0}, upload=332523]: Done uploading piece. err: <nil>
Mar 18 19:52:50.799659215 (296h52m48.39643024s) [noise=true , address=77.90.31.14:4663 , piece_id=K7P3ERBAXZAF2VWQRGHXKDOMEPWP4GSBRO4YNSWGPZ2XKM7TDUEA, seg_pos={0 0}, upload=332523]: Done uploading piece. err: <nil>
Mar 18 19:52:50.813523156 (296h52m48.410294187s) [noise=true , address=79.127.219.41:4926 , piece_id=XZUTTB2A2K4ZWZLND6OJNG3UUFLTITH7CHWOKULUS4RYMF4SI7VQ, seg_pos={0 0}, upload=332523]: Done uploading piece. err: <nil>
Mar 18 19:53:16.570865535 (296h53m14.167636568s) [noise=true , address=145.79.227.34:4905 , piece_id=SSSDFRJONSSMGDYZANLIKTHQ6HBWT6IFYJDAJSSYIHE5TASNFRGQ, seg_pos={0 0}, upload=332523]: Done uploading piece. err: ecclient: upload failed (node:12ffKESNSVpXQzbwJinP6vvgE6zoF3RFz2NRUuuJHvd7bvk1gc9, address:145.79.227.34:4905): protocol: rpc: dial tcp 145.79.227.34:4905: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 19:53:16.570954939 (296h53m14.167725975s) [noise=true , address=145.79.227.23:4551 , piece_id=OIJSN73HORIFMDHQNZJSD7GJY6R4TC3DUUICIJMDHNHR4QADPWJA, seg_pos={0 0}, upload=332523]: Done uploading piece. err: ecclient: upload failed (node:12vR5WRMGoYyzYDZzwELFw6kLG6vtKsUeFM9kUhgZDLGKB6vcXs, address:145.79.227.23:4551): protocol: rpc: dial tcp 145.79.227.23:4551: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 19:53:16.571490071 (296h53m14.168261104s) [noise=true , address=145.79.227.22:4512 , piece_id=ES4PBUL6SUOZQ26TOHVGLWOWEQVNUSPXCJPDRMEA6TXPJ2RKIPIQ, seg_pos={0 0}, upload=332523]: Done uploading piece. err: ecclient: upload failed (node:12cnVqYYfo8b6RvRtMykzh8mU7UAqxcqYvWe7H8G7k8DqtNLAR4, address:145.79.227.22:4512): protocol: rpc: dial tcp 145.79.227.22:4512: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 19:53:16.571614124 (296h53m14.168385158s) [noise=true , address=145.79.227.2:4119 , piece_id=INNE55CDRSMDRPWIQ4P53X7J2YUICGJ5QKQ7MAWC5WFXI2226GNA, seg_pos={0 0}, upload=332523]: Done uploading piece. err: ecclient: upload failed (node:121xNAnkm9HZXUszp26UmWTsuhp3igyeDVea6gSCdtrQUQDVHpW, address:145.79.227.2:4119): protocol: rpc: dial tcp 145.79.227.2:4119: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 19:53:16.571901972 (296h53m14.168673006s) [noise=true , address=145.79.227.26:4658 , piece_id=KV2Q7TQPYETS2P5EK3AYCBWXO75IHPGLMYS7T3EY3DKOQ6HO7QVQ, seg_pos={0 0}, upload=332523]: Done uploading piece. err: ecclient: upload failed (node:1gcSHRechDzTDwivRoAwNFfNj5Zjqvi6oJwVqzbCQinwgWtpPr, address:145.79.227.26:4658): protocol: rpc: dial tcp 145.79.227.26:4658: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 19:53:16.572092048 (296h53m14.168863082s) [noise=true , address=145.79.227.51:4409 , piece_id=76FS44U7FERWPRO72CQ3EIGPN6RQI4QOCRJH4IQ4M57EPWXBQW2Q, seg_pos={0 0}, upload=332523]: Done uploading piece. err: ecclient: upload failed (node:12UoCGA4FjE4yADPXL1oifSPJssPgSGh81REZHwGv2eu1Y1kEaj, address:145.79.227.51:4409): protocol: rpc: dial tcp 145.79.227.51:4409: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 19:53:16.572198851 (296h53m14.168969882s) [noise=true , address=145.79.227.10:4405 , piece_id=SKCT3FGRCXW7QHNZRKBCEYAATJ22QJA4A235434GB5SK6RTPCQYQ, seg_pos={0 0}, upload=332523]: Done uploading piece. err: ecclient: upload failed (node:1MCHYW29ZQjE2fHN8P98hDfNRLsjiVqabF86TXz5nZRBPhbGy4, address:145.79.227.10:4405): protocol: rpc: dial tcp 145.79.227.10:4405: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 19:53:16.572455355 (296h53m14.169226389s) [noise=true , address=145.79.227.35:4486 , piece_id=KSSOZCWN5K7QLOGR7ZWQC4MQ5XXPOWTRTMDNJ5TSPL5L7HFIV5JA, seg_pos={0 0}, upload=332523]: Done uploading piece. err: ecclient: upload failed (node:1fbs2btcbiu2yYGrvxNZe8sNSqDnNPG1Bmfyhcew86FCFLKAic, address:145.79.227.35:4486): protocol: rpc: dial tcp 145.79.227.35:4486: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 19:53:16.572509808 (296h53m14.16928084s) [noise=true , address=145.79.227.49:4337 , piece_id=NSSI5XWZMTJ6VUZNU4JTZRK4ZUWAVCTC6F5KCXYORWZNZ2DA52XQ, seg_pos={0 0}, upload=332523]: Done uploading piece. err: ecclient: upload failed (node:1nTZFfzNCcVAjut7NXf9seifysJDfvXkS8aZSnXKfw6gh1TrjE, address:145.79.227.49:4337): protocol: rpc: dial tcp 145.79.227.49:4337: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 19:53:16.572948872 (296h53m14.169719905s) [noise=true , address=145.79.227.7:4296 , piece_id=M7NXQQ4U4EHSE3637Y2OLERXE7WN5YONSCFIF72TWI5VDGPHJMMQ, seg_pos={0 0}, upload=332523]: Done uploading piece. err: ecclient: upload failed (node:12cKhDbA4ptwux2LYNPNAJ7Qh8jECsGWd1KWaK6T6pD38bJWdt1, address:145.79.227.7:4296): protocol: rpc: dial tcp 145.79.227.7:4296: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 19:53:16.573027890 (296h53m14.169798915s) [seg_pos={0 0}, upload=332523]: Done waiting for segment. successful: 6 optimal: 13 errs: <nil>
Mar 18 19:53:16.599059755 (296h53m14.195830769s) [upload=332523]: Done uploading segments...
Mar 18 19:53:16.599099776 (296h53m14.195870803s) [upload=332523]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 6)
Duration: 30.030508444s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
8 out of 60 is failed (threshold=3) [.....................................XXXXXX.XX..............] duration:
20.812394242s trace:
https://jaeger.dev.storj.io/trace/4b4d10a8051ad01b
FAIL (2026-03-18 19:53:08 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 20.812430477s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)antiblip:
4 out of 60 is failed (threshold=3) [........................................X..........XX..X....] duration:
25.068542499s trace:
https://jaeger.dev.storj.io/trace/40b499326665cee7
FAIL (2026-03-18 19:53:03 UTC)
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 13)
======LOGS======
Mar 18 19:49:22.100496252 (296h49m19.697267275s) [upload=332463]: Starting upload
Mar 18 19:49:22.100562062 (296h49m19.697333097s) [upload=332463]: Done starting upload
Mar 18 19:49:22.100920922 (296h49m19.697691942s) [upload=332463]: Uploading segments...
Mar 18 19:49:22.101052565 (296h49m19.6978236s) [upload=332463]: Got next segment. Inline: false
Mar 18 19:49:22.101122083 (296h49m19.697893113s) [upload=332463]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Mar 18 19:49:22.101129030 (296h49m19.697900059s) [upload=332463]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Mar 18 19:49:25.902635706 (296h49m23.499406738s) [seg_pos={0 0}, upload=332463]: Begin upload segment...
Mar 18 19:49:25.902699156 (296h49m23.49947018s) [seg_pos={0 0}, upload=332463]: Done begin upload segment.
Mar 18 19:49:25.902751503 (296h49m23.499522526s) [noise=true , address=193.149.164.71:4315 , piece_id=DMR24J3LY6VU7BSXKZ5GGB7PTR2AR7JRIYODUBGB44CUV6XEIJIA, seg_pos={0 0}, upload=332463]: Uploading piece...
Mar 18 19:49:25.902763524 (296h49m23.499534558s) [noise=true , address=145.79.227.29:4951 , piece_id=QOAXIBCXYVHEAKTH6QTZ5DNEUX2FJ6RQNLPNILPIUGW6LR6KBRWA, seg_pos={0 0}, upload=332463]: Uploading piece...
Mar 18 19:49:25.903772480 (296h49m23.500543514s) [noise=true , address=79.127.205.234:4185 , piece_id=3OZE6YVHU3SRSWJSXMWLCKWIALRSKJ4UL4PUJMVC6DXS4EPFJRIA, seg_pos={0 0}, upload=332463]: Uploading piece...
Mar 18 19:49:25.904336133 (296h49m23.501107164s) [noise=true , address=193.149.164.67:4005 , piece_id=TI7LY4ZLLDYHK4OIK63TFHTONJWTYMDGLLWIQMAOBZCLZQXR4QLA, seg_pos={0 0}, upload=332463]: Uploading piece...
Mar 18 19:49:25.904896440 (296h49m23.501667477s) [noise=true , address=145.79.227.49:4365 , piece_id=JN3LS4VKZI2J6DKXNCY34F3M4DKQM5RTNCVMPQSQFLU3HSORWAOA, seg_pos={0 0}, upload=332463]: Uploading piece...
Mar 18 19:49:25.905706107 (296h49m23.502477153s) [noise=true , address=79.127.205.233:4168 , piece_id=3LWUDIIZPVPDGVQFL76P4NLMIR34QK3VAD7DTZZ2D7GO4AIBBKTQ, seg_pos={0 0}, upload=332463]: Uploading piece...
Mar 18 19:49:25.906306716 (296h49m23.503077751s) [noise=true , address=64.49.13.137:4498 , piece_id=CIBHYGP7RZRWSMOU5NVD2FILCMNHCOOPTJHWUVXC6M4O2PA5LQLA, seg_pos={0 0}, upload=332463]: Uploading piece...
Mar 18 19:49:25.906845104 (296h49m23.503616137s) [noise=true , address=167.17.72.69:4723 , piece_id=HOI6UDYIQ25N2MGLWEY4DS5EQ5BCP5C3GK3RJ656CJZ4VB2ETSTQ, seg_pos={0 0}, upload=332463]: Uploading piece...
Mar 18 19:49:25.907488195 (296h49m23.504259226s) [noise=true , address=77.90.31.11:4581 , piece_id=P2JDFEL2V2UKADRJUFRKYXGQAW2ZQ65VGQCE3ICIZQUZX565IH5A, seg_pos={0 0}, upload=332463]: Uploading piece...
Mar 18 19:49:25.908477653 (296h49m23.505248679s) [noise=true , address=64.49.13.138:4489 , piece_id=HDC4HAR3SZL4BL6QZ25PDMKOWLWED75ETK7VUW75OTJGT3GJES2Q, seg_pos={0 0}, upload=332463]: Uploading piece...
Mar 18 19:49:25.910737637 (296h49m23.507508664s) [noise=true , address=145.79.227.12:4772 , piece_id=UAP6J7R5XTMHAQJAHSCCGKBXG73XCMX7NUQ34OMBIHFDDD73GHDQ, seg_pos={0 0}, upload=332463]: Uploading piece...
Mar 18 19:49:25.912589587 (296h49m23.509360619s) [noise=true , address=193.149.164.70:4110 , piece_id=FIFZZLBHFFMLAIZUT5JCAHIJMLVLVGUNC4DF67XSE3XPABIGD7YA, seg_pos={0 0}, upload=332463]: Uploading piece...
Mar 18 19:49:25.914513704 (296h49m23.511284737s) [noise=true , address=64.49.13.170:4723 , piece_id=BQVT2JI5CDAWQLUZQQD76DXJRUNM7EVSVDYEKFNINLQTZXRVOTJQ, seg_pos={0 0}, upload=332463]: Uploading piece...
Mar 18 19:49:25.916789871 (296h49m23.513560899s) [noise=true , address=5.231.73.5:4016 , piece_id=NA6FQDLVM6DMD4JT6ZR2UPSK6SS3THCBU6BK5P6ZOEUFMBGHYFHA, seg_pos={0 0}, upload=332463]: Uploading piece...
Mar 18 19:49:25.918547957 (296h49m23.515318984s) [noise=true , address=145.79.227.33:4904 , piece_id=SFWHDAELZM7YKZMRCLWY3WL5JWO33URLNJQJNWXYJ3UYZ3B3QGYQ, seg_pos={0 0}, upload=332463]: Uploading piece...
Mar 18 19:49:25.920615295 (296h49m23.517386321s) [noise=true , address=77.90.31.17:4775 , piece_id=KHVRKVINWCHZ4ZFQYYW3YFYQHYUSM4LIPQP7AB7L5ERA6KAPQR7Q, seg_pos={0 0}, upload=332463]: Uploading piece...
Mar 18 19:49:25.922559454 (296h49m23.519330488s) [noise=true , address=145.79.227.34:4939 , piece_id=724NW2MLRW5LLICXVGYQNGZWSOCT34UEVYJAJOG3FJ2SIEXQ5SUQ, seg_pos={0 0}, upload=332463]: Uploading piece...
Mar 18 19:49:25.925302234 (296h49m23.522073257s) [noise=true , address=167.17.72.70:4778 , piece_id=WNA2YVAVKVLLTFYUAEZEYAXCHMDNRKYNRKWVISTCGGAUTCMCPEJQ, seg_pos={0 0}, upload=332463]: Uploading piece...
Mar 18 19:49:25.973039411 (296h49m23.569810434s) [upload=332463]: Next returned nil segment
Mar 18 19:49:25.973052806 (296h49m23.569823829s) [upload=332463]: Waiting for error group managing segments...
Mar 18 19:49:25.995559764 (296h49m23.592330781s) [noise=true , address=77.90.31.11:4581 , piece_id=P2JDFEL2V2UKADRJUFRKYXGQAW2ZQ65VGQCE3ICIZQUZX565IH5A, seg_pos={0 0}, upload=332463]: Done uploading piece. err: <nil>
Mar 18 19:49:26.001337339 (296h49m23.598108384s) [noise=true , address=77.90.31.17:4775 , piece_id=KHVRKVINWCHZ4ZFQYYW3YFYQHYUSM4LIPQP7AB7L5ERA6KAPQR7Q, seg_pos={0 0}, upload=332463]: Done uploading piece. err: <nil>
Mar 18 19:49:26.154115776 (296h49m23.750886809s) [noise=true , address=5.231.73.5:4016 , piece_id=NA6FQDLVM6DMD4JT6ZR2UPSK6SS3THCBU6BK5P6ZOEUFMBGHYFHA, seg_pos={0 0}, upload=332463]: Done uploading piece. err: <nil>
Mar 18 19:49:26.253235656 (296h49m23.850006679s) [noise=true , address=167.17.72.69:4723 , piece_id=HOI6UDYIQ25N2MGLWEY4DS5EQ5BCP5C3GK3RJ656CJZ4VB2ETSTQ, seg_pos={0 0}, upload=332463]: Done uploading piece. err: <nil>
Mar 18 19:49:26.493998588 (296h49m24.090769614s) [noise=true , address=79.127.205.234:4185 , piece_id=3OZE6YVHU3SRSWJSXMWLCKWIALRSKJ4UL4PUJMVC6DXS4EPFJRIA, seg_pos={0 0}, upload=332463]: Done uploading piece. err: <nil>
Mar 18 19:49:26.497324258 (296h49m24.094095282s) [noise=true , address=64.49.13.137:4498 , piece_id=CIBHYGP7RZRWSMOU5NVD2FILCMNHCOOPTJHWUVXC6M4O2PA5LQLA, seg_pos={0 0}, upload=332463]: Done uploading piece. err: <nil>
Mar 18 19:49:26.504309786 (296h49m24.101080813s) [noise=true , address=79.127.205.233:4168 , piece_id=3LWUDIIZPVPDGVQFL76P4NLMIR34QK3VAD7DTZZ2D7GO4AIBBKTQ, seg_pos={0 0}, upload=332463]: Done uploading piece. err: <nil>
Mar 18 19:49:26.509531694 (296h49m24.106302708s) [noise=true , address=193.149.164.71:4315 , piece_id=DMR24J3LY6VU7BSXKZ5GGB7PTR2AR7JRIYODUBGB44CUV6XEIJIA, seg_pos={0 0}, upload=332463]: Done uploading piece. err: <nil>
Mar 18 19:49:26.516397335 (296h49m24.113168361s) [noise=true , address=193.149.164.70:4110 , piece_id=FIFZZLBHFFMLAIZUT5JCAHIJMLVLVGUNC4DF67XSE3XPABIGD7YA, seg_pos={0 0}, upload=332463]: Done uploading piece. err: <nil>
Mar 18 19:49:26.533102812 (296h49m24.129873852s) [noise=true , address=64.49.13.170:4723 , piece_id=BQVT2JI5CDAWQLUZQQD76DXJRUNM7EVSVDYEKFNINLQTZXRVOTJQ, seg_pos={0 0}, upload=332463]: Done uploading piece. err: <nil>
Mar 18 19:49:26.546130691 (296h49m24.142901714s) [noise=true , address=193.149.164.67:4005 , piece_id=TI7LY4ZLLDYHK4OIK63TFHTONJWTYMDGLLWIQMAOBZCLZQXR4QLA, seg_pos={0 0}, upload=332463]: Done uploading piece. err: <nil>
Mar 18 19:49:26.552325489 (296h49m24.149096513s) [noise=true , address=64.49.13.138:4489 , piece_id=HDC4HAR3SZL4BL6QZ25PDMKOWLWED75ETK7VUW75OTJGT3GJES2Q, seg_pos={0 0}, upload=332463]: Done uploading piece. err: <nil>
Mar 18 19:49:28.497314925 (296h49m26.09408595s) [noise=true , address=167.17.72.70:4778 , piece_id=WNA2YVAVKVLLTFYUAEZEYAXCHMDNRKYNRKWVISTCGGAUTCMCPEJQ, seg_pos={0 0}, upload=332463]: Done uploading piece. err: <nil>
Mar 18 19:50:22.063196037 (296h50m19.659967069s) [noise=true , address=145.79.227.33:4904 , piece_id=SFWHDAELZM7YKZMRCLWY3WL5JWO33URLNJQJNWXYJ3UYZ3B3QGYQ, seg_pos={0 0}, upload=332463]: Done uploading piece. err: ecclient: upload failed (node:1dC7mC2Cbxmys7ukEkELyF9Us7F1qtptSP6yDJh1EfjVJ6uqsC, address:145.79.227.33:4904): protocol: rpc: dial tcp 145.79.227.33:4904: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 19:50:22.063515331 (296h50m19.660286364s) [noise=true , address=145.79.227.29:4951 , piece_id=QOAXIBCXYVHEAKTH6QTZ5DNEUX2FJ6RQNLPNILPIUGW6LR6KBRWA, seg_pos={0 0}, upload=332463]: Done uploading piece. err: ecclient: upload failed (node:12DH5AoVzQ92aiKUuVJGXecG4yuGZjPJVY6V6cfwQzSYCZAAUNg, address:145.79.227.29:4951): protocol: rpc: dial tcp 145.79.227.29:4951: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 19:50:22.063750837 (296h50m19.66052187s) [noise=true , address=145.79.227.12:4772 , piece_id=UAP6J7R5XTMHAQJAHSCCGKBXG73XCMX7NUQ34OMBIHFDDD73GHDQ, seg_pos={0 0}, upload=332463]: Done uploading piece. err: ecclient: upload failed (node:1BP2yptVJkJzCKenuKzMZ5sBbVbqXcaoVviwKwrnbxodo1pR5w, address:145.79.227.12:4772): protocol: rpc: dial tcp 145.79.227.12:4772: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 19:50:22.063872856 (296h50m19.660643889s) [noise=true , address=145.79.227.49:4365 , piece_id=JN3LS4VKZI2J6DKXNCY34F3M4DKQM5RTNCVMPQSQFLU3HSORWAOA, seg_pos={0 0}, upload=332463]: Done uploading piece. err: ecclient: upload failed (node:122EX4PNuiRTCG2gbgbaL5dAtD8s5SN7ZesShk7NqUTGCZ3gA2a, address:145.79.227.49:4365): protocol: rpc: dial tcp 145.79.227.49:4365: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 19:50:22.064095334 (296h50m19.660866366s) [noise=true , address=145.79.227.34:4939 , piece_id=724NW2MLRW5LLICXVGYQNGZWSOCT34UEVYJAJOG3FJ2SIEXQ5SUQ, seg_pos={0 0}, upload=332463]: Done uploading piece. err: ecclient: upload failed (node:1h7xrPMs5kHtjm7hiB4hfDaZUHFgtJNoMGqdy3HFif2Aned8eu, address:145.79.227.34:4939): protocol: rpc: dial tcp 145.79.227.34:4939: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 19:50:22.064168331 (296h50m19.660939355s) [seg_pos={0 0}, upload=332463]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil>
Mar 18 19:50:22.091389605 (296h50m19.688160629s) [upload=332463]: Done uploading segments...
Mar 18 19:50:22.091441055 (296h50m19.688212078s) [upload=332463]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13)
Duration: 25.068608487s
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
5 out of 60 is failed (threshold=3) [......................................XXX..........X...X....] duration:
25.745933344s trace:
https://jaeger.dev.storj.io/trace/6a01088813e3d8b7
FAIL (2026-03-18 19:52:56 UTC)
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 12)
======LOGS======
m: *pb.BatchRequestItem_ObjectBegin
Mar 18 19:49:08.138517471 (296h49m5.735288494s) [upload=332459]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Mar 18 19:49:16.284286759 (296h49m13.881057773s) [seg_pos={0 0}, upload=332459]: Begin upload segment...
Mar 18 19:49:16.284352223 (296h49m13.88112325s) [seg_pos={0 0}, upload=332459]: Done begin upload segment.
Mar 18 19:49:16.284395789 (296h49m13.881166818s) [noise=true , address=193.149.164.68:4051 , piece_id=FCHPX2FRYUVNH4JAAPKI2Y7DSJCUJ5UG3EASXQYB7VEQPNERYUGQ, seg_pos={0 0}, upload=332459]: Uploading piece...
Mar 18 19:49:16.284449689 (296h49m13.881220728s) [noise=true , address=145.79.227.13:4822 , piece_id=Y5GSC3KFFGLK4B3NCXEMOZTAC3ASNGPJJOPOKCUBUJIL75E6T6FA, seg_pos={0 0}, upload=332459]: Uploading piece...
Mar 18 19:49:16.284549974 (296h49m13.881320988s) [noise=true , address=45.91.116.5:4308 , piece_id=XRXYA347E22AHAJ4XDENA5EHCV2F43HXL67MTUGRYSU6B3WQBDNQ, seg_pos={0 0}, upload=332459]: Uploading piece...
Mar 18 19:49:16.285144837 (296h49m13.88191587s) [noise=true , address=193.149.164.70:4117 , piece_id=PL6FWCO6XQRSVU26FH2KESA26D4Z2PCQVBPBDKBWLBPXHBV5LQVQ, seg_pos={0 0}, upload=332459]: Uploading piece...
Mar 18 19:49:16.285821284 (296h49m13.882592317s) [noise=true , address=145.79.227.52:4451 , piece_id=SBMEAZS5RDQW3RCJP2UU5T7BHF6P7VOICLP5P46YJ5LE2GB65CGA, seg_pos={0 0}, upload=332459]: Uploading piece...
Mar 18 19:49:16.286556627 (296h49m13.883327669s) [noise=true , address=79.127.205.235:4259 , piece_id=N7BGDCO2CR6HLPUWYF35T5OQRWQFEWFG7FPZ6Y33J7FGR5H2ZURQ, seg_pos={0 0}, upload=332459]: Uploading piece...
Mar 18 19:49:16.287228779 (296h49m13.883999813s) [noise=true , address=79.127.219.45:4864 , piece_id=UXHLRN5HUN3ZRXOZEYAJOS6QUN3PVIXXXED3TS6RPWSHXJ4WKS6Q, seg_pos={0 0}, upload=332459]: Uploading piece...
Mar 18 19:49:16.287269957 (296h49m13.884040985s) [noise=true , address=77.90.31.21:4894 , piece_id=IQ6775I4HFC5JQVYUCQPXYWPY3X6QRHVXM6IF4JEZRQMMRRWQDUQ, seg_pos={0 0}, upload=332459]: Uploading piece...
Mar 18 19:49:16.287437026 (296h49m13.884208049s) [noise=true , address=5.231.73.4:4010 , piece_id=LTONWX7UJVP3BPHHKKJUN6PTAR5TVPVKF3CD6NBV2MZWTLUOVQZQ, seg_pos={0 0}, upload=332459]: Uploading piece...
Mar 18 19:49:16.287832890 (296h49m13.884603914s) [noise=true , address=167.17.72.68:4657 , piece_id=UUIV3ERJSVOJT4TISEHCMR6EQCG6PMZUR6OWGK3ZVV33B33CUZCA, seg_pos={0 0}, upload=332459]: Uploading piece...
Mar 18 19:49:16.287952922 (296h49m13.884723957s) [noise=true , address=79.127.219.41:4939 , piece_id=WK6HMSTVM7LD2P5MPCKML4MBFZJX2QJWZS6SYPITDKNOLPMRPO6Q, seg_pos={0 0}, upload=332459]: Uploading piece...
Mar 18 19:49:16.288418812 (296h49m13.885189842s) [noise=true , address=145.79.227.44:4213 , piece_id=MK3BJSFCDOG2MAJYO2BAE2JG62EVEZMCCUATBBBT4QU44GLKK3QA, seg_pos={0 0}, upload=332459]: Uploading piece...
Mar 18 19:49:16.288922438 (296h49m13.885693461s) [noise=true , address=145.79.227.3:4163 , piece_id=3R3QBIJ2XDQZQXJ66RGNZBWZB6V4ERS5RJEZFBTS2LFMQ25G4FLA, seg_pos={0 0}, upload=332459]: Uploading piece...
Mar 18 19:49:16.289343580 (296h49m13.886114604s) [noise=true , address=193.149.164.71:4310 , piece_id=BKUA4WC73DSXJMEPCMTXTUD2PVQAAYD44MAGEB3SS3FEDJO7377A, seg_pos={0 0}, upload=332459]: Uploading piece...
Mar 18 19:49:16.289780590 (296h49m13.88655162s) [noise=true , address=145.79.227.49:4358 , piece_id=QSE44UR5YBYTJHWGOCBFVQPAUZ3SJV6P5WKBEF3MTFAKVRH2QSHQ, seg_pos={0 0}, upload=332459]: Uploading piece...
Mar 18 19:49:16.290237004 (296h49m13.887008027s) [noise=true , address=77.90.31.14:4685 , piece_id=3KAIZVB2LH6R57CFO55R7BJV7G4MGHFN6M5VTKT3J6LMI5F7PVAA, seg_pos={0 0}, upload=332459]: Uploading piece...
Mar 18 19:49:16.292879809 (296h49m13.889650843s) [noise=true , address=77.90.31.18:4817 , piece_id=FMVKZ4Z3YDPWZHTXU3ZTSWXMVUWLKQX7TUWXPQ4LOKBQK64PHF5Q, seg_pos={0 0}, upload=332459]: Uploading piece...
Mar 18 19:49:16.293664670 (296h49m13.890435693s) [noise=true , address=145.79.227.15:4908 , piece_id=LG24OMYBUVPVRBFZL4UUJLPBVNDTVVHQ4HAYTW2PGKHUKZPF2WCA, seg_pos={0 0}, upload=332459]: Uploading piece...
Mar 18 19:49:16.351440712 (296h49m13.948211724s) [upload=332459]: Next returned nil segment
Mar 18 19:49:16.351460639 (296h49m13.948231661s) [upload=332459]: Waiting for error group managing segments...
Mar 18 19:49:16.355941640 (296h49m13.952712664s) [noise=true , address=77.90.31.21:4894 , piece_id=IQ6775I4HFC5JQVYUCQPXYWPY3X6QRHVXM6IF4JEZRQMMRRWQDUQ, seg_pos={0 0}, upload=332459]: Done uploading piece. err: <nil>
Mar 18 19:49:16.356991929 (296h49m13.953762955s) [noise=true , address=79.127.219.41:4939 , piece_id=WK6HMSTVM7LD2P5MPCKML4MBFZJX2QJWZS6SYPITDKNOLPMRPO6Q, seg_pos={0 0}, upload=332459]: Done uploading piece. err: <nil>
Mar 18 19:49:16.361809763 (296h49m13.958580786s) [noise=true , address=45.91.116.5:4308 , piece_id=XRXYA347E22AHAJ4XDENA5EHCV2F43HXL67MTUGRYSU6B3WQBDNQ, seg_pos={0 0}, upload=332459]: Done uploading piece. err: <nil>
Mar 18 19:49:16.367502365 (296h49m13.964273396s) [noise=true , address=77.90.31.14:4685 , piece_id=3KAIZVB2LH6R57CFO55R7BJV7G4MGHFN6M5VTKT3J6LMI5F7PVAA, seg_pos={0 0}, upload=332459]: Done uploading piece. err: <nil>
Mar 18 19:49:16.381549888 (296h49m13.97832091s) [noise=true , address=79.127.219.45:4864 , piece_id=UXHLRN5HUN3ZRXOZEYAJOS6QUN3PVIXXXED3TS6RPWSHXJ4WKS6Q, seg_pos={0 0}, upload=332459]: Done uploading piece. err: <nil>
Mar 18 19:49:16.387503503 (296h49m13.984274527s) [noise=true , address=77.90.31.18:4817 , piece_id=FMVKZ4Z3YDPWZHTXU3ZTSWXMVUWLKQX7TUWXPQ4LOKBQK64PHF5Q, seg_pos={0 0}, upload=332459]: Done uploading piece. err: <nil>
Mar 18 19:49:16.498139242 (296h49m14.094910266s) [noise=true , address=5.231.73.4:4010 , piece_id=LTONWX7UJVP3BPHHKKJUN6PTAR5TVPVKF3CD6NBV2MZWTLUOVQZQ, seg_pos={0 0}, upload=332459]: Done uploading piece. err: <nil>
Mar 18 19:49:16.852944491 (296h49m14.449715518s) [noise=true , address=167.17.72.68:4657 , piece_id=UUIV3ERJSVOJT4TISEHCMR6EQCG6PMZUR6OWGK3ZVV33B33CUZCA, seg_pos={0 0}, upload=332459]: Done uploading piece. err: <nil>
Mar 18 19:49:16.878068834 (296h49m14.474839867s) [noise=true , address=79.127.205.235:4259 , piece_id=N7BGDCO2CR6HLPUWYF35T5OQRWQFEWFG7FPZ6Y33J7FGR5H2ZURQ, seg_pos={0 0}, upload=332459]: Done uploading piece. err: <nil>
Mar 18 19:49:16.892446253 (296h49m14.489217282s) [noise=true , address=193.149.164.68:4051 , piece_id=FCHPX2FRYUVNH4JAAPKI2Y7DSJCUJ5UG3EASXQYB7VEQPNERYUGQ, seg_pos={0 0}, upload=332459]: Done uploading piece. err: <nil>
Mar 18 19:49:16.893099142 (296h49m14.48987017s) [noise=true , address=193.149.164.71:4310 , piece_id=BKUA4WC73DSXJMEPCMTXTUD2PVQAAYD44MAGEB3SS3FEDJO7377A, seg_pos={0 0}, upload=332459]: Done uploading piece. err: <nil>
Mar 18 19:49:17.031110208 (296h49m14.627881232s) [noise=true , address=193.149.164.70:4117 , piece_id=PL6FWCO6XQRSVU26FH2KESA26D4Z2PCQVBPBDKBWLBPXHBV5LQVQ, seg_pos={0 0}, upload=332459]: Done uploading piece. err: <nil>
Mar 18 19:50:08.096628364 (296h50m5.693399398s) [noise=true , address=145.79.227.52:4451 , piece_id=SBMEAZS5RDQW3RCJP2UU5T7BHF6P7VOICLP5P46YJ5LE2GB65CGA, seg_pos={0 0}, upload=332459]: Done uploading piece. err: ecclient: upload failed (node:12bC4P3xBiPFkxLvavGFVbWKnbprCStL6UJgfKSA2Wd2fX7Mbfw, address:145.79.227.52:4451): protocol: rpc: dial tcp 145.79.227.52:4451: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 19:50:08.096712565 (296h50m5.693483599s) [noise=true , address=145.79.227.44:4213 , piece_id=MK3BJSFCDOG2MAJYO2BAE2JG62EVEZMCCUATBBBT4QU44GLKK3QA, seg_pos={0 0}, upload=332459]: Done uploading piece. err: ecclient: upload failed (node:129LRnDm7b4Fo5db7pto8GXYEtxHz9qdPAtQvaYH6gqdkMatQ6g, address:145.79.227.44:4213): protocol: rpc: dial tcp 145.79.227.44:4213: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 19:50:08.096943717 (296h50m5.693714752s) [noise=true , address=145.79.227.49:4358 , piece_id=QSE44UR5YBYTJHWGOCBFVQPAUZ3SJV6P5WKBEF3MTFAKVRH2QSHQ, seg_pos={0 0}, upload=332459]: Done uploading piece. err: ecclient: upload failed (node:1dwCNEcA9o51ZN3UaN3Frxdawwf8DXBj86iQMywjW8XP4ESmw1, address:145.79.227.49:4358): protocol: rpc: dial tcp 145.79.227.49:4358: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 19:50:08.097022451 (296h50m5.693793484s) [noise=true , address=145.79.227.3:4163 , piece_id=3R3QBIJ2XDQZQXJ66RGNZBWZB6V4ERS5RJEZFBTS2LFMQ25G4FLA, seg_pos={0 0}, upload=332459]: Done uploading piece. err: ecclient: upload failed (node:12w88UkGu5duYivJ6L6C9iJmTbnB63fDKiUUQprhwM3JshpXTdC, address:145.79.227.3:4163): protocol: rpc: dial tcp 145.79.227.3:4163: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 19:50:08.097155433 (296h50m5.693926468s) [noise=true , address=145.79.227.13:4822 , piece_id=Y5GSC3KFFGLK4B3NCXEMOZTAC3ASNGPJJOPOKCUBUJIL75E6T6FA, seg_pos={0 0}, upload=332459]: Done uploading piece. err: ecclient: upload failed (node:12fgLN9DGBGxWfBEePSsk84FTrnKoKWwNgoET3FpotfJxJ28f75, address:145.79.227.13:4822): protocol: rpc: dial tcp 145.79.227.13:4822: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 19:50:08.097360007 (296h50m5.694131039s) [noise=true , address=145.79.227.15:4908 , piece_id=LG24OMYBUVPVRBFZL4UUJLPBVNDTVVHQ4HAYTW2PGKHUKZPF2WCA, seg_pos={0 0}, upload=332459]: Done uploading piece. err: ecclient: upload failed (node:1pfQ5v1GwHhFTvh5Y7Jpb9qpWhZhFYqWX9PJpLEUzTkqKWcy4x, address:145.79.227.15:4908): protocol: rpc: dial tcp 145.79.227.15:4908: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 19:50:08.097462766 (296h50m5.69423378s) [seg_pos={0 0}, upload=332459]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil>
Mar 18 19:50:08.117913456 (296h50m5.714684468s) [upload=332459]: Done uploading segments...
Mar 18 19:50:08.117999321 (296h50m5.714770348s) [upload=332459]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12)
Duration: 25.745961249s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
29 out of 60 is failed (threshold=6) [...........................XXXXXXXXX.XXX...XXXXXXXXXXXXXXXXX] duration:
5.00186506s trace:
https://jaeger.dev.storj.io/trace/e206aaf571315ce
FAIL (2026-03-18 19:52:49 UTC)
operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.001889852s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
23 out of 60 is failed (threshold=3) [.....................................XXXXXXXXXXXXXXXXXXXXXXX] duration:
30.028076107s trace:
https://jaeger.dev.storj.io/trace/17f971bdf361b719
FAIL (2026-03-18 19:52:37 UTC)
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 8)
======LOGS======
Mar 18 19:52:07.729963332 (296h52m5.326734355s) [upload=332509]: Starting upload
Mar 18 19:52:07.730068903 (296h52m5.326839935s) [upload=332509]: Done starting upload
Mar 18 19:52:07.730212806 (296h52m5.326983832s) [upload=332509]: Uploading segments...
Mar 18 19:52:07.730454549 (296h52m5.327225563s) [upload=332509]: Got next segment. Inline: false
Mar 18 19:52:07.730518957 (296h52m5.32728998s) [upload=332509]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Mar 18 19:52:07.730524593 (296h52m5.327295616s) [upload=332509]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Mar 18 19:52:11.472693791 (296h52m9.069464824s) [seg_pos={0 0}, upload=332509]: Begin upload segment...
Mar 18 19:52:11.472805213 (296h52m9.069576231s) [seg_pos={0 0}, upload=332509]: Done begin upload segment.
Mar 18 19:52:11.472879844 (296h52m9.069650878s) [noise=true , address=145.79.227.19:4077 , piece_id=LDI6UC3ITCALQX3RZZGHFAHRUKIN32LLZ2G36PREFQXTJHU5RXUA, seg_pos={0 0}, upload=332509]: Uploading piece...
Mar 18 19:52:11.474057098 (296h52m9.070828123s) [noise=true , address=77.90.31.15:4701 , piece_id=A7AIWKG2KWH3WQ6JQ7TXMUTR2QBB5BY2VF5XB5NXFC4IYOVABJEQ, seg_pos={0 0}, upload=332509]: Uploading piece...
Mar 18 19:52:11.474842686 (296h52m9.071613715s) [noise=true , address=77.90.31.21:4878 , piece_id=ABY35TMVBRMNRDQT3VJ6NIDS7K7KSFBTGDGUMNC67VUSQF3V7ZQA, seg_pos={0 0}, upload=332509]: Uploading piece...
Mar 18 19:52:11.476316040 (296h52m9.073087074s) [noise=true , address=145.79.227.21:4473 , piece_id=FKYTPUUV25REQKAD2IZ5BJDRDR2UKANFF4NXEUSSS5DJUFKFZWCQ, seg_pos={0 0}, upload=332509]: Uploading piece...
Mar 18 19:52:11.477446167 (296h52m9.074217194s) [noise=true , address=145.79.227.45:4233 , piece_id=RGTZHXWUSVDDYTEZMFHJ6XD74DEO2ZPTBXR2XWOJGANT7EQ2REBA, seg_pos={0 0}, upload=332509]: Uploading piece...
Mar 18 19:52:11.477933991 (296h52m9.074705014s) [noise=true , address=45.91.116.8:4048 , piece_id=LVHJQU5RNO7D2ULPFETEMAGRLIZYF4LWZTEO3VLWHAFE2G7TGF6Q, seg_pos={0 0}, upload=332509]: Uploading piece...
Mar 18 19:52:11.478433143 (296h52m9.075204172s) [noise=true , address=145.79.227.30:4762 , piece_id=SSH5CNJESZBHE27DVTEG7GDDOMSGGPH6CEWPTCYZN3HJ6BYD6EWA, seg_pos={0 0}, upload=332509]: Uploading piece...
Mar 18 19:52:11.478972899 (296h52m9.075743932s) [noise=true , address=145.79.227.8:4333 , piece_id=42AVHUFNGOGTYC2K5OGYSX7YH6SUJ32B34L5MJED7IPBUOPNVJ6A, seg_pos={0 0}, upload=332509]: Uploading piece...
Mar 18 19:52:11.479669671 (296h52m9.076440705s) [noise=true , address=145.79.227.2:4115 , piece_id=XYD3JLWRSW2WJCHSPXIWUTHN4SMF4G6XN4GWAVSKDSBYNSSYVNMA, seg_pos={0 0}, upload=332509]: Uploading piece...
Mar 18 19:52:11.482273835 (296h52m9.079044864s) [noise=true , address=77.90.31.16:4736 , piece_id=S4WKIO2J2EYQHHQF5IGEMLAXDRWFQZCBQY3YL4MPPPFFXN6JUQLQ, seg_pos={0 0}, upload=332509]: Uploading piece...
Mar 18 19:52:11.483265669 (296h52m9.080036695s) [noise=true , address=79.127.219.38:4817 , piece_id=G5U5SSPMITR4DGZV3OTBOKL64BQYWOOODICC7AUQLOPLHIURCQZQ, seg_pos={0 0}, upload=332509]: Uploading piece...
Mar 18 19:52:11.484028180 (296h52m9.080799212s) [noise=true , address=45.91.116.9:4261 , piece_id=ETLNJKJHGMZZDJGSKEBGOZMMH6RJUHK4VOOUUMCUP7DA3EKNET2Q, seg_pos={0 0}, upload=332509]: Uploading piece...
Mar 18 19:52:11.484680216 (296h52m9.081451246s) [noise=true , address=145.79.227.51:4412 , piece_id=RPODB7ICP6MCDWLVVE6MNCE3HR3WNFXTXSDQSMCSTNXPHGL6VH2Q, seg_pos={0 0}, upload=332509]: Uploading piece...
Mar 18 19:52:11.485295288 (296h52m9.082066315s) [noise=true , address=145.79.227.44:4196 , piece_id=23NC2CQMQBLH4XIGJLYGTBRW4LCRWVM6VEMDMLPKO5HC7CHRR72Q, seg_pos={0 0}, upload=332509]: Uploading piece...
Mar 18 19:52:11.487584122 (296h52m9.08435515s) [noise=true , address=45.91.116.5:4332 , piece_id=YBVLQPZIJEWKUX3AAML5EOEZFAPD5VBZF7G6PZNADFWODV2QGKKA, seg_pos={0 0}, upload=332509]: Uploading piece...
Mar 18 19:52:11.488461440 (296h52m9.08523247s) [noise=true , address=45.91.116.7:4009 , piece_id=YENOAT4ZH7I3V5NUHIHRZJIGHWJGUVKZ3GIXSMPNDKYN7WHMP7UQ, seg_pos={0 0}, upload=332509]: Uploading piece...
Mar 18 19:52:11.564766031 (296h52m9.161537046s) [upload=332509]: Next returned nil segment
Mar 18 19:52:11.564781840 (296h52m9.161552857s) [upload=332509]: Waiting for error group managing segments...
Mar 18 19:52:11.572581793 (296h52m9.169352806s) [noise=true , address=45.91.116.7:4009 , piece_id=YENOAT4ZH7I3V5NUHIHRZJIGHWJGUVKZ3GIXSMPNDKYN7WHMP7UQ, seg_pos={0 0}, upload=332509]: Done uploading piece. err: <nil>
Mar 18 19:52:11.573205826 (296h52m9.169976841s) [noise=true , address=77.90.31.21:4878 , piece_id=ABY35TMVBRMNRDQT3VJ6NIDS7K7KSFBTGDGUMNC67VUSQF3V7ZQA, seg_pos={0 0}, upload=332509]: Done uploading piece. err: <nil>
Mar 18 19:52:11.574960749 (296h52m9.171731773s) [noise=true , address=77.90.31.16:4736 , piece_id=S4WKIO2J2EYQHHQF5IGEMLAXDRWFQZCBQY3YL4MPPPFFXN6JUQLQ, seg_pos={0 0}, upload=332509]: Done uploading piece. err: <nil>
Mar 18 19:52:11.576012176 (296h52m9.172783188s) [noise=true , address=45.91.116.5:4332 , piece_id=YBVLQPZIJEWKUX3AAML5EOEZFAPD5VBZF7G6PZNADFWODV2QGKKA, seg_pos={0 0}, upload=332509]: Done uploading piece. err: <nil>
Mar 18 19:52:11.578125797 (296h52m9.17489683s) [noise=true , address=77.90.31.15:4701 , piece_id=A7AIWKG2KWH3WQ6JQ7TXMUTR2QBB5BY2VF5XB5NXFC4IYOVABJEQ, seg_pos={0 0}, upload=332509]: Done uploading piece. err: <nil>
Mar 18 19:52:11.578814479 (296h52m9.175585503s) [noise=true , address=45.91.116.8:4048 , piece_id=LVHJQU5RNO7D2ULPFETEMAGRLIZYF4LWZTEO3VLWHAFE2G7TGF6Q, seg_pos={0 0}, upload=332509]: Done uploading piece. err: <nil>
Mar 18 19:52:11.590681633 (296h52m9.187452659s) [noise=true , address=79.127.219.38:4817 , piece_id=G5U5SSPMITR4DGZV3OTBOKL64BQYWOOODICC7AUQLOPLHIURCQZQ, seg_pos={0 0}, upload=332509]: Done uploading piece. err: <nil>
Mar 18 19:52:11.594541666 (296h52m9.191312696s) [noise=true , address=45.91.116.9:4261 , piece_id=ETLNJKJHGMZZDJGSKEBGOZMMH6RJUHK4VOOUUMCUP7DA3EKNET2Q, seg_pos={0 0}, upload=332509]: Done uploading piece. err: <nil>
Mar 18 19:52:37.696699646 (296h52m35.293470676s) [noise=true , address=145.79.227.51:4412 , piece_id=RPODB7ICP6MCDWLVVE6MNCE3HR3WNFXTXSDQSMCSTNXPHGL6VH2Q, seg_pos={0 0}, upload=332509]: Done uploading piece. err: ecclient: upload failed (node:12vBvEfCYZxsv17cVsQTbCQ4YjZfiD4RXjWyz7Ka9ZZrtvzC7SS, address:145.79.227.51:4412): protocol: context deadline exceeded; context deadline exceeded
Mar 18 19:52:37.696820644 (296h52m35.293591677s) [noise=true , address=145.79.227.2:4115 , piece_id=XYD3JLWRSW2WJCHSPXIWUTHN4SMF4G6XN4GWAVSKDSBYNSSYVNMA, seg_pos={0 0}, upload=332509]: Done uploading piece. err: ecclient: upload failed (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR, address:145.79.227.2:4115): protocol: context deadline exceeded; context deadline exceeded
Mar 18 19:52:37.697027155 (296h52m35.293798188s) [noise=true , address=145.79.227.45:4233 , piece_id=RGTZHXWUSVDDYTEZMFHJ6XD74DEO2ZPTBXR2XWOJGANT7EQ2REBA, seg_pos={0 0}, upload=332509]: Done uploading piece. err: ecclient: upload failed (node:1DmQeoQWTnFhQeJFseJoNYzFyo7vZjYKwDtQVJ6pomksWExsTb, address:145.79.227.45:4233): protocol: context deadline exceeded; context deadline exceeded
Mar 18 19:52:37.697224526 (296h52m35.293995543s) [noise=true , address=145.79.227.8:4333 , piece_id=42AVHUFNGOGTYC2K5OGYSX7YH6SUJ32B34L5MJED7IPBUOPNVJ6A, seg_pos={0 0}, upload=332509]: Done uploading piece. err: ecclient: upload failed (node:1xRdTZoEk3NsnS445GcdEvwAdDzNLwUYSfgjB7bBSGRx9L9mrU, address:145.79.227.8:4333): protocol: context deadline exceeded; context deadline exceeded
Mar 18 19:52:37.697264105 (296h52m35.294035141s) [noise=true , address=145.79.227.44:4196 , piece_id=23NC2CQMQBLH4XIGJLYGTBRW4LCRWVM6VEMDMLPKO5HC7CHRR72Q, seg_pos={0 0}, upload=332509]: Done uploading piece. err: ecclient: upload failed (node:12Vs4grpszw7MhLJSMgCn7ikaL4jgwSB6XULfVgAt1Msy8hMcaL, address:145.79.227.44:4196): protocol: context deadline exceeded; context deadline exceeded
Mar 18 19:52:37.697380442 (296h52m35.294151474s) [noise=true , address=145.79.227.19:4077 , piece_id=LDI6UC3ITCALQX3RZZGHFAHRUKIN32LLZ2G36PREFQXTJHU5RXUA, seg_pos={0 0}, upload=332509]: Done uploading piece. err: ecclient: upload failed (node:12kDJuQP8iLUjB1rXpcNX5TtHzsqa5HvYcUD9Ef64HYTnTXcwLk, address:145.79.227.19:4077): protocol: context deadline exceeded; context deadline exceeded
Mar 18 19:52:37.697434916 (296h52m35.294205949s) [noise=true , address=145.79.227.30:4762 , piece_id=SSH5CNJESZBHE27DVTEG7GDDOMSGGPH6CEWPTCYZN3HJ6BYD6EWA, seg_pos={0 0}, upload=332509]: Done uploading piece. err: ecclient: upload failed (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b, address:145.79.227.30:4762): protocol: context deadline exceeded; context deadline exceeded
Mar 18 19:52:37.697513762 (296h52m35.294284801s) [noise=true , address=145.79.227.21:4473 , piece_id=FKYTPUUV25REQKAD2IZ5BJDRDR2UKANFF4NXEUSSS5DJUFKFZWCQ, seg_pos={0 0}, upload=332509]: Done uploading piece. err: ecclient: upload failed (node:12DDe4jT3YeUH8qMQgapcvzMnFs8UFB8fTmubho4ZQf5tYVoxuk, address:145.79.227.21:4473): protocol: context deadline exceeded; context deadline exceeded
Mar 18 19:52:37.697541470 (296h52m35.294312489s) [seg_pos={0 0}, upload=332509]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil>
Mar 18 19:52:37.722950488 (296h52m35.319721512s) [upload=332509]: Done uploading segments...
Mar 18 19:52:37.722988940 (296h52m35.319759964s) [upload=332509]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8)
Duration: 30.028101179s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)antiblip:
4 out of 60 is failed (threshold=3) [.........................................X..........XX..X...] duration:
16.175428606s trace:
https://jaeger.dev.storj.io/trace/40b499326665cee7
FAIL (2026-03-18 19:52:29 UTC)
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 13)
======LOGS======
Mar 18 19:49:22.100496252 (296h49m19.697267275s) [upload=332463]: Starting upload
Mar 18 19:49:22.100562062 (296h49m19.697333097s) [upload=332463]: Done starting upload
Mar 18 19:49:22.100920922 (296h49m19.697691942s) [upload=332463]: Uploading segments...
Mar 18 19:49:22.101052565 (296h49m19.6978236s) [upload=332463]: Got next segment. Inline: false
Mar 18 19:49:22.101122083 (296h49m19.697893113s) [upload=332463]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Mar 18 19:49:22.101129030 (296h49m19.697900059s) [upload=332463]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Mar 18 19:49:25.902635706 (296h49m23.499406738s) [seg_pos={0 0}, upload=332463]: Begin upload segment...
Mar 18 19:49:25.902699156 (296h49m23.49947018s) [seg_pos={0 0}, upload=332463]: Done begin upload segment.
Mar 18 19:49:25.902751503 (296h49m23.499522526s) [noise=true , address=193.149.164.71:4315 , piece_id=DMR24J3LY6VU7BSXKZ5GGB7PTR2AR7JRIYODUBGB44CUV6XEIJIA, seg_pos={0 0}, upload=332463]: Uploading piece...
Mar 18 19:49:25.902763524 (296h49m23.499534558s) [noise=true , address=145.79.227.29:4951 , piece_id=QOAXIBCXYVHEAKTH6QTZ5DNEUX2FJ6RQNLPNILPIUGW6LR6KBRWA, seg_pos={0 0}, upload=332463]: Uploading piece...
Mar 18 19:49:25.903772480 (296h49m23.500543514s) [noise=true , address=79.127.205.234:4185 , piece_id=3OZE6YVHU3SRSWJSXMWLCKWIALRSKJ4UL4PUJMVC6DXS4EPFJRIA, seg_pos={0 0}, upload=332463]: Uploading piece...
Mar 18 19:49:25.904336133 (296h49m23.501107164s) [noise=true , address=193.149.164.67:4005 , piece_id=TI7LY4ZLLDYHK4OIK63TFHTONJWTYMDGLLWIQMAOBZCLZQXR4QLA, seg_pos={0 0}, upload=332463]: Uploading piece...
Mar 18 19:49:25.904896440 (296h49m23.501667477s) [noise=true , address=145.79.227.49:4365 , piece_id=JN3LS4VKZI2J6DKXNCY34F3M4DKQM5RTNCVMPQSQFLU3HSORWAOA, seg_pos={0 0}, upload=332463]: Uploading piece...
Mar 18 19:49:25.905706107 (296h49m23.502477153s) [noise=true , address=79.127.205.233:4168 , piece_id=3LWUDIIZPVPDGVQFL76P4NLMIR34QK3VAD7DTZZ2D7GO4AIBBKTQ, seg_pos={0 0}, upload=332463]: Uploading piece...
Mar 18 19:49:25.906306716 (296h49m23.503077751s) [noise=true , address=64.49.13.137:4498 , piece_id=CIBHYGP7RZRWSMOU5NVD2FILCMNHCOOPTJHWUVXC6M4O2PA5LQLA, seg_pos={0 0}, upload=332463]: Uploading piece...
Mar 18 19:49:25.906845104 (296h49m23.503616137s) [noise=true , address=167.17.72.69:4723 , piece_id=HOI6UDYIQ25N2MGLWEY4DS5EQ5BCP5C3GK3RJ656CJZ4VB2ETSTQ, seg_pos={0 0}, upload=332463]: Uploading piece...
Mar 18 19:49:25.907488195 (296h49m23.504259226s) [noise=true , address=77.90.31.11:4581 , piece_id=P2JDFEL2V2UKADRJUFRKYXGQAW2ZQ65VGQCE3ICIZQUZX565IH5A, seg_pos={0 0}, upload=332463]: Uploading piece...
Mar 18 19:49:25.908477653 (296h49m23.505248679s) [noise=true , address=64.49.13.138:4489 , piece_id=HDC4HAR3SZL4BL6QZ25PDMKOWLWED75ETK7VUW75OTJGT3GJES2Q, seg_pos={0 0}, upload=332463]: Uploading piece...
Mar 18 19:49:25.910737637 (296h49m23.507508664s) [noise=true , address=145.79.227.12:4772 , piece_id=UAP6J7R5XTMHAQJAHSCCGKBXG73XCMX7NUQ34OMBIHFDDD73GHDQ, seg_pos={0 0}, upload=332463]: Uploading piece...
Mar 18 19:49:25.912589587 (296h49m23.509360619s) [noise=true , address=193.149.164.70:4110 , piece_id=FIFZZLBHFFMLAIZUT5JCAHIJMLVLVGUNC4DF67XSE3XPABIGD7YA, seg_pos={0 0}, upload=332463]: Uploading piece...
Mar 18 19:49:25.914513704 (296h49m23.511284737s) [noise=true , address=64.49.13.170:4723 , piece_id=BQVT2JI5CDAWQLUZQQD76DXJRUNM7EVSVDYEKFNINLQTZXRVOTJQ, seg_pos={0 0}, upload=332463]: Uploading piece...
Mar 18 19:49:25.916789871 (296h49m23.513560899s) [noise=true , address=5.231.73.5:4016 , piece_id=NA6FQDLVM6DMD4JT6ZR2UPSK6SS3THCBU6BK5P6ZOEUFMBGHYFHA, seg_pos={0 0}, upload=332463]: Uploading piece...
Mar 18 19:49:25.918547957 (296h49m23.515318984s) [noise=true , address=145.79.227.33:4904 , piece_id=SFWHDAELZM7YKZMRCLWY3WL5JWO33URLNJQJNWXYJ3UYZ3B3QGYQ, seg_pos={0 0}, upload=332463]: Uploading piece...
Mar 18 19:49:25.920615295 (296h49m23.517386321s) [noise=true , address=77.90.31.17:4775 , piece_id=KHVRKVINWCHZ4ZFQYYW3YFYQHYUSM4LIPQP7AB7L5ERA6KAPQR7Q, seg_pos={0 0}, upload=332463]: Uploading piece...
Mar 18 19:49:25.922559454 (296h49m23.519330488s) [noise=true , address=145.79.227.34:4939 , piece_id=724NW2MLRW5LLICXVGYQNGZWSOCT34UEVYJAJOG3FJ2SIEXQ5SUQ, seg_pos={0 0}, upload=332463]: Uploading piece...
Mar 18 19:49:25.925302234 (296h49m23.522073257s) [noise=true , address=167.17.72.70:4778 , piece_id=WNA2YVAVKVLLTFYUAEZEYAXCHMDNRKYNRKWVISTCGGAUTCMCPEJQ, seg_pos={0 0}, upload=332463]: Uploading piece...
Mar 18 19:49:25.973039411 (296h49m23.569810434s) [upload=332463]: Next returned nil segment
Mar 18 19:49:25.973052806 (296h49m23.569823829s) [upload=332463]: Waiting for error group managing segments...
Mar 18 19:49:25.995559764 (296h49m23.592330781s) [noise=true , address=77.90.31.11:4581 , piece_id=P2JDFEL2V2UKADRJUFRKYXGQAW2ZQ65VGQCE3ICIZQUZX565IH5A, seg_pos={0 0}, upload=332463]: Done uploading piece. err: <nil>
Mar 18 19:49:26.001337339 (296h49m23.598108384s) [noise=true , address=77.90.31.17:4775 , piece_id=KHVRKVINWCHZ4ZFQYYW3YFYQHYUSM4LIPQP7AB7L5ERA6KAPQR7Q, seg_pos={0 0}, upload=332463]: Done uploading piece. err: <nil>
Mar 18 19:49:26.154115776 (296h49m23.750886809s) [noise=true , address=5.231.73.5:4016 , piece_id=NA6FQDLVM6DMD4JT6ZR2UPSK6SS3THCBU6BK5P6ZOEUFMBGHYFHA, seg_pos={0 0}, upload=332463]: Done uploading piece. err: <nil>
Mar 18 19:49:26.253235656 (296h49m23.850006679s) [noise=true , address=167.17.72.69:4723 , piece_id=HOI6UDYIQ25N2MGLWEY4DS5EQ5BCP5C3GK3RJ656CJZ4VB2ETSTQ, seg_pos={0 0}, upload=332463]: Done uploading piece. err: <nil>
Mar 18 19:49:26.493998588 (296h49m24.090769614s) [noise=true , address=79.127.205.234:4185 , piece_id=3OZE6YVHU3SRSWJSXMWLCKWIALRSKJ4UL4PUJMVC6DXS4EPFJRIA, seg_pos={0 0}, upload=332463]: Done uploading piece. err: <nil>
Mar 18 19:49:26.497324258 (296h49m24.094095282s) [noise=true , address=64.49.13.137:4498 , piece_id=CIBHYGP7RZRWSMOU5NVD2FILCMNHCOOPTJHWUVXC6M4O2PA5LQLA, seg_pos={0 0}, upload=332463]: Done uploading piece. err: <nil>
Mar 18 19:49:26.504309786 (296h49m24.101080813s) [noise=true , address=79.127.205.233:4168 , piece_id=3LWUDIIZPVPDGVQFL76P4NLMIR34QK3VAD7DTZZ2D7GO4AIBBKTQ, seg_pos={0 0}, upload=332463]: Done uploading piece. err: <nil>
Mar 18 19:49:26.509531694 (296h49m24.106302708s) [noise=true , address=193.149.164.71:4315 , piece_id=DMR24J3LY6VU7BSXKZ5GGB7PTR2AR7JRIYODUBGB44CUV6XEIJIA, seg_pos={0 0}, upload=332463]: Done uploading piece. err: <nil>
Mar 18 19:49:26.516397335 (296h49m24.113168361s) [noise=true , address=193.149.164.70:4110 , piece_id=FIFZZLBHFFMLAIZUT5JCAHIJMLVLVGUNC4DF67XSE3XPABIGD7YA, seg_pos={0 0}, upload=332463]: Done uploading piece. err: <nil>
Mar 18 19:49:26.533102812 (296h49m24.129873852s) [noise=true , address=64.49.13.170:4723 , piece_id=BQVT2JI5CDAWQLUZQQD76DXJRUNM7EVSVDYEKFNINLQTZXRVOTJQ, seg_pos={0 0}, upload=332463]: Done uploading piece. err: <nil>
Mar 18 19:49:26.546130691 (296h49m24.142901714s) [noise=true , address=193.149.164.67:4005 , piece_id=TI7LY4ZLLDYHK4OIK63TFHTONJWTYMDGLLWIQMAOBZCLZQXR4QLA, seg_pos={0 0}, upload=332463]: Done uploading piece. err: <nil>
Mar 18 19:49:26.552325489 (296h49m24.149096513s) [noise=true , address=64.49.13.138:4489 , piece_id=HDC4HAR3SZL4BL6QZ25PDMKOWLWED75ETK7VUW75OTJGT3GJES2Q, seg_pos={0 0}, upload=332463]: Done uploading piece. err: <nil>
Mar 18 19:49:28.497314925 (296h49m26.09408595s) [noise=true , address=167.17.72.70:4778 , piece_id=WNA2YVAVKVLLTFYUAEZEYAXCHMDNRKYNRKWVISTCGGAUTCMCPEJQ, seg_pos={0 0}, upload=332463]: Done uploading piece. err: <nil>
Mar 18 19:50:22.063196037 (296h50m19.659967069s) [noise=true , address=145.79.227.33:4904 , piece_id=SFWHDAELZM7YKZMRCLWY3WL5JWO33URLNJQJNWXYJ3UYZ3B3QGYQ, seg_pos={0 0}, upload=332463]: Done uploading piece. err: ecclient: upload failed (node:1dC7mC2Cbxmys7ukEkELyF9Us7F1qtptSP6yDJh1EfjVJ6uqsC, address:145.79.227.33:4904): protocol: rpc: dial tcp 145.79.227.33:4904: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 19:50:22.063515331 (296h50m19.660286364s) [noise=true , address=145.79.227.29:4951 , piece_id=QOAXIBCXYVHEAKTH6QTZ5DNEUX2FJ6RQNLPNILPIUGW6LR6KBRWA, seg_pos={0 0}, upload=332463]: Done uploading piece. err: ecclient: upload failed (node:12DH5AoVzQ92aiKUuVJGXecG4yuGZjPJVY6V6cfwQzSYCZAAUNg, address:145.79.227.29:4951): protocol: rpc: dial tcp 145.79.227.29:4951: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 19:50:22.063750837 (296h50m19.66052187s) [noise=true , address=145.79.227.12:4772 , piece_id=UAP6J7R5XTMHAQJAHSCCGKBXG73XCMX7NUQ34OMBIHFDDD73GHDQ, seg_pos={0 0}, upload=332463]: Done uploading piece. err: ecclient: upload failed (node:1BP2yptVJkJzCKenuKzMZ5sBbVbqXcaoVviwKwrnbxodo1pR5w, address:145.79.227.12:4772): protocol: rpc: dial tcp 145.79.227.12:4772: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 19:50:22.063872856 (296h50m19.660643889s) [noise=true , address=145.79.227.49:4365 , piece_id=JN3LS4VKZI2J6DKXNCY34F3M4DKQM5RTNCVMPQSQFLU3HSORWAOA, seg_pos={0 0}, upload=332463]: Done uploading piece. err: ecclient: upload failed (node:122EX4PNuiRTCG2gbgbaL5dAtD8s5SN7ZesShk7NqUTGCZ3gA2a, address:145.79.227.49:4365): protocol: rpc: dial tcp 145.79.227.49:4365: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 19:50:22.064095334 (296h50m19.660866366s) [noise=true , address=145.79.227.34:4939 , piece_id=724NW2MLRW5LLICXVGYQNGZWSOCT34UEVYJAJOG3FJ2SIEXQ5SUQ, seg_pos={0 0}, upload=332463]: Done uploading piece. err: ecclient: upload failed (node:1h7xrPMs5kHtjm7hiB4hfDaZUHFgtJNoMGqdy3HFif2Aned8eu, address:145.79.227.34:4939): protocol: rpc: dial tcp 145.79.227.34:4939: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 19:50:22.064168331 (296h50m19.660939355s) [seg_pos={0 0}, upload=332463]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil>
Mar 18 19:50:22.091389605 (296h50m19.688160629s) [upload=332463]: Done uploading segments...
Mar 18 19:50:22.091441055 (296h50m19.688212078s) [upload=332463]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13)
Duration: 16.175456226s
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|