Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 7 out of 60 is failed (threshold=3) [.......................XXX..........X...X.........X....X....] duration: 16.390937818s trace: https://jaeger.dev.storj.io/trace/698fb9649d932a68 FAIL (2026-03-18 20:05:32 UTC) operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 20:02:41.019616017 (297h2m38.616387042s) [upload=332700]: Uploading segments... Mar 18 20:02:41.019826191 (297h2m38.616597205s) [upload=332700]: Got next segment. Inline: false Mar 18 20:02:41.019879916 (297h2m38.61665094s) [upload=332700]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 20:02:41.019885584 (297h2m38.616656611s) [upload=332700]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 18 20:02:43.834160391 (297h2m41.430931414s) [seg_pos={0 0}, upload=332700]: Begin upload segment... Mar 18 20:02:43.834230381 (297h2m41.431001396s) [seg_pos={0 0}, upload=332700]: Done begin upload segment. Mar 18 20:02:43.834262428 (297h2m41.431033451s) [noise=true , address=145.79.227.52:4451 , piece_id=LR6KVRYMDO2HAESG5BFI5RYPNGVARP6O5HEY2UP77YSTVRKBNUXQ, seg_pos={0 0}, upload=332700]: Uploading piece... Mar 18 20:02:43.834329472 (297h2m41.431100497s) [noise=true , address=45.91.116.13:4164 , piece_id=MNQLVANO3LFKBCJMADD5RULARMRO4VUQY7LT63TU2CCNO765GW7Q, seg_pos={0 0}, upload=332700]: Uploading piece... Mar 18 20:02:43.834938832 (297h2m41.431709858s) [noise=true , address=77.90.31.15:4716 , piece_id=IDQP2MIPCKVS2FGVVGKAIDYTNUIPE56EER5IYZUSOTRD5LZBSXHA, seg_pos={0 0}, upload=332700]: Uploading piece... Mar 18 20:02:43.835898822 (297h2m41.432669851s) [noise=true , address=79.127.205.237:4227 , piece_id=NO25WIYZTEPBOMGETTLSDOJSPKARMKNLVUWDKNXUKHYGDGF2XC5Q, seg_pos={0 0}, upload=332700]: Uploading piece... Mar 18 20:02:43.836893726 (297h2m41.433664759s) [noise=true , address=193.149.164.68:4064 , piece_id=HAA3NDB24MJLAI2HK2UDZQZRLDUXEF76ZJF2VBE6FLKRWSB27Y3A, seg_pos={0 0}, upload=332700]: Uploading piece... Mar 18 20:02:43.837257949 (297h2m41.434028981s) [noise=true , address=193.149.164.66:4974 , piece_id=XWMBTFEWXVVFIR6IYNQYU35QKGVSQJ7YBVSWZ57QSUZ2VSDCJPRA, seg_pos={0 0}, upload=332700]: Uploading piece... Mar 18 20:02:43.838266356 (297h2m41.435037391s) [noise=true , address=145.79.227.34:4924 , piece_id=BNUESPWRZQBX23K7UGNSMF362AWHILNZ43UITBHVGFXIJ4T6V5HQ, seg_pos={0 0}, upload=332700]: Uploading piece... Mar 18 20:02:43.838688582 (297h2m41.435459605s) [noise=true , address=79.127.205.233:4171 , piece_id=YO42WSHYRB2JJX4RVX653N7U6PYTGIW3KQTNSF4445OTN3UW2TIA, seg_pos={0 0}, upload=332700]: Uploading piece... Mar 18 20:02:43.839263235 (297h2m41.436034258s) [noise=true , address=193.149.164.70:4123 , piece_id=VROUAQKRTKZT42ZDDSCIMKFMRODPKFQ5U5DWDJLVI462BVA5YWXA, seg_pos={0 0}, upload=332700]: Uploading piece... Mar 18 20:02:43.839835928 (297h2m41.436606963s) [noise=true , address=45.91.116.4:4948 , piece_id=ULVH4YAJIYY5RWWAGH5RTSZ3BMAT6WCYYAJPSMSP25LCLZQIPG2A, seg_pos={0 0}, upload=332700]: Uploading piece... Mar 18 20:02:43.840390028 (297h2m41.437161055s) [noise=true , address=45.91.116.5:4319 , piece_id=BY6BRNZNKKQKZNAVJTSWUYASX57P5X7OCKQ433T2ARRBXUFNBRCA, seg_pos={0 0}, upload=332700]: Uploading piece... Mar 18 20:02:43.840790648 (297h2m41.437561682s) [noise=true , address=45.91.116.12:4259 , piece_id=4LBHNF3CIQG5LYEY7RYJGNK5Q62X2TSOAXOXSNHO727OXL5OMA4Q, seg_pos={0 0}, upload=332700]: Uploading piece... Mar 18 20:02:43.841319562 (297h2m41.438090584s) [noise=true , address=145.79.227.29:4966 , piece_id=ZTSKMRBESX4OIZMMIPXRW752W5KQYX57RPDQ22XXPURQTSCAI2YA, seg_pos={0 0}, upload=332700]: Uploading piece... Mar 18 20:02:43.841816046 (297h2m41.438587078s) [noise=true , address=193.149.164.71:4287 , piece_id=FYNZ3QGUB7ZFMUJAMYPMUHXJDS66CJ2VLXPV7256DFCABPJMEOYQ, seg_pos={0 0}, upload=332700]: Uploading piece... Mar 18 20:02:43.842288870 (297h2m41.439059903s) [noise=true , address=79.127.219.41:4941 , piece_id=CQXOXMCJHWJ7FSIXDVYUJZMBFW5FUUOJNKJJOVLZRSK5D6XZXX5Q, seg_pos={0 0}, upload=332700]: Uploading piece... Mar 18 20:02:43.842956832 (297h2m41.439727861s) [noise=true , address=145.79.227.49:4369 , piece_id=CBYHFRBNKSPUNJHAE6S72YOFACWLZKGZIFV6SXWDYMVSA4FBPKQQ, seg_pos={0 0}, upload=332700]: Uploading piece... Mar 18 20:02:43.844386533 (297h2m41.441157559s) [noise=true , address=77.90.31.10:4533 , piece_id=VB5BKAWDUZFVII4ZY3JRVK67VUULPSAHTJVZMXXVATP3SXAR2XHQ, seg_pos={0 0}, upload=332700]: Uploading piece... Mar 18 20:02:43.844868553 (297h2m41.441639576s) [noise=true , address=145.79.227.37:4582 , piece_id=DLOFMRZRG3BPFN7JF73OM5HUV56FLEH2IDDC47UQWWWOTRVRR4HQ, seg_pos={0 0}, upload=332700]: Uploading piece... Mar 18 20:02:43.903559633 (297h2m41.500330666s) [upload=332700]: Next returned nil segment Mar 18 20:02:43.903578892 (297h2m41.500349925s) [upload=332700]: Waiting for error group managing segments... Mar 18 20:02:43.908218461 (297h2m41.504989474s) [noise=true , address=45.91.116.4:4948 , piece_id=ULVH4YAJIYY5RWWAGH5RTSZ3BMAT6WCYYAJPSMSP25LCLZQIPG2A, seg_pos={0 0}, upload=332700]: Done uploading piece. err: <nil> Mar 18 20:02:43.917679744 (297h2m41.514450753s) [noise=true , address=45.91.116.12:4259 , piece_id=4LBHNF3CIQG5LYEY7RYJGNK5Q62X2TSOAXOXSNHO727OXL5OMA4Q, seg_pos={0 0}, upload=332700]: Done uploading piece. err: <nil> Mar 18 20:02:43.924357408 (297h2m41.521128422s) [noise=true , address=77.90.31.10:4533 , piece_id=VB5BKAWDUZFVII4ZY3JRVK67VUULPSAHTJVZMXXVATP3SXAR2XHQ, seg_pos={0 0}, upload=332700]: Done uploading piece. err: <nil> Mar 18 20:02:43.926248407 (297h2m41.52301943s) [noise=true , address=77.90.31.15:4716 , piece_id=IDQP2MIPCKVS2FGVVGKAIDYTNUIPE56EER5IYZUSOTRD5LZBSXHA, seg_pos={0 0}, upload=332700]: Done uploading piece. err: <nil> Mar 18 20:02:43.927212300 (297h2m41.523983323s) [noise=true , address=45.91.116.5:4319 , piece_id=BY6BRNZNKKQKZNAVJTSWUYASX57P5X7OCKQ433T2ARRBXUFNBRCA, seg_pos={0 0}, upload=332700]: Done uploading piece. err: <nil> Mar 18 20:02:43.927791610 (297h2m41.524562634s) [noise=true , address=45.91.116.13:4164 , piece_id=MNQLVANO3LFKBCJMADD5RULARMRO4VUQY7LT63TU2CCNO765GW7Q, seg_pos={0 0}, upload=332700]: Done uploading piece. err: <nil> Mar 18 20:02:44.028504592 (297h2m41.625275611s) [noise=true , address=79.127.219.41:4941 , piece_id=CQXOXMCJHWJ7FSIXDVYUJZMBFW5FUUOJNKJJOVLZRSK5D6XZXX5Q, seg_pos={0 0}, upload=332700]: Done uploading piece. err: <nil> Mar 18 20:02:44.392266647 (297h2m41.989037671s) [noise=true , address=79.127.205.233:4171 , piece_id=YO42WSHYRB2JJX4RVX653N7U6PYTGIW3KQTNSF4445OTN3UW2TIA, seg_pos={0 0}, upload=332700]: Done uploading piece. err: <nil> Mar 18 20:02:44.429403642 (297h2m42.026174669s) [noise=true , address=193.149.164.66:4974 , piece_id=XWMBTFEWXVVFIR6IYNQYU35QKGVSQJ7YBVSWZ57QSUZ2VSDCJPRA, seg_pos={0 0}, upload=332700]: Done uploading piece. err: <nil> Mar 18 20:02:44.433374828 (297h2m42.030145853s) [noise=true , address=193.149.164.68:4064 , piece_id=HAA3NDB24MJLAI2HK2UDZQZRLDUXEF76ZJF2VBE6FLKRWSB27Y3A, seg_pos={0 0}, upload=332700]: Done uploading piece. err: <nil> Mar 18 20:02:44.444389014 (297h2m42.041160042s) [noise=true , address=193.149.164.71:4287 , piece_id=FYNZ3QGUB7ZFMUJAMYPMUHXJDS66CJ2VLXPV7256DFCABPJMEOYQ, seg_pos={0 0}, upload=332700]: Done uploading piece. err: <nil> Mar 18 20:02:44.578333498 (297h2m42.175104529s) [noise=true , address=79.127.205.237:4227 , piece_id=NO25WIYZTEPBOMGETTLSDOJSPKARMKNLVUWDKNXUKHYGDGF2XC5Q, seg_pos={0 0}, upload=332700]: Done uploading piece. err: <nil> Mar 18 20:02:44.665665140 (297h2m42.262436168s) [noise=true , address=193.149.164.70:4123 , piece_id=VROUAQKRTKZT42ZDDSCIMKFMRODPKFQ5U5DWDJLVI462BVA5YWXA, seg_pos={0 0}, upload=332700]: Done uploading piece. err: <nil> Mar 18 20:03:40.987108256 (297h3m38.58387928s) [noise=true , address=145.79.227.49:4369 , piece_id=CBYHFRBNKSPUNJHAE6S72YOFACWLZKGZIFV6SXWDYMVSA4FBPKQQ, seg_pos={0 0}, upload=332700]: Done uploading piece. err: ecclient: upload failed (node:12nLksGXGvDwuTWfnmkceuwZNDmrF17iNepHPdYkkvZG2QQUwG7, address:145.79.227.49:4369): protocol: rpc: dial tcp 145.79.227.49:4369: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:03:40.987479807 (297h3m38.58425084s) [noise=true , address=145.79.227.29:4966 , piece_id=ZTSKMRBESX4OIZMMIPXRW752W5KQYX57RPDQ22XXPURQTSCAI2YA, seg_pos={0 0}, upload=332700]: Done uploading piece. err: ecclient: upload failed (node:1DLnsYVdudE4mY9dNAQKhBRppUEAHvxbf84KX69451pwxSxzZK, address:145.79.227.29:4966): protocol: rpc: dial tcp 145.79.227.29:4966: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:03:40.987517864 (297h3m38.584288896s) [noise=true , address=145.79.227.34:4924 , piece_id=BNUESPWRZQBX23K7UGNSMF362AWHILNZ43UITBHVGFXIJ4T6V5HQ, seg_pos={0 0}, upload=332700]: Done uploading piece. err: ecclient: upload failed (node:19TbrQjuN1EhBNVtha9EcKPSCAWyttfmHpec6QeF1S5zW6DRZs, address:145.79.227.34:4924): protocol: rpc: dial tcp 145.79.227.34:4924: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:03:40.987933296 (297h3m38.584704329s) [noise=true , address=145.79.227.37:4582 , piece_id=DLOFMRZRG3BPFN7JF73OM5HUV56FLEH2IDDC47UQWWWOTRVRR4HQ, seg_pos={0 0}, upload=332700]: Done uploading piece. err: ecclient: upload failed (node:129o91j97t4V5AjaouCCy2wDuHgwF94iQiJh2h9ux1aFo7AVCSn, address:145.79.227.37:4582): protocol: rpc: dial tcp 145.79.227.37:4582: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:03:40.988167350 (297h3m38.584938376s) [noise=true , address=145.79.227.52:4451 , piece_id=LR6KVRYMDO2HAESG5BFI5RYPNGVARP6O5HEY2UP77YSTVRKBNUXQ, seg_pos={0 0}, upload=332700]: Done uploading piece. err: ecclient: upload failed (node:12bC4P3xBiPFkxLvavGFVbWKnbprCStL6UJgfKSA2Wd2fX7Mbfw, address:145.79.227.52:4451): protocol: rpc: dial tcp 145.79.227.52:4451: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:03:40.988197047 (297h3m38.58496806s) [seg_pos={0 0}, upload=332700]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 18 20:03:41.007727578 (297h3m38.604498608s) [upload=332700]: Done uploading segments... Mar 18 20:03:41.007779207 (297h3m38.60455025s) [upload=332700]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 16.390964179s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 7 out of 60 is failed (threshold=3) [........................XXX..........X...X.........X....X...] duration: 16.389954633s trace: https://jaeger.dev.storj.io/trace/698fb9649d932a68 FAIL (2026-03-18 20:05:15 UTC) operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 20:02:41.019616017 (297h2m38.616387042s) [upload=332700]: Uploading segments... Mar 18 20:02:41.019826191 (297h2m38.616597205s) [upload=332700]: Got next segment. Inline: false Mar 18 20:02:41.019879916 (297h2m38.61665094s) [upload=332700]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 20:02:41.019885584 (297h2m38.616656611s) [upload=332700]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 18 20:02:43.834160391 (297h2m41.430931414s) [seg_pos={0 0}, upload=332700]: Begin upload segment... Mar 18 20:02:43.834230381 (297h2m41.431001396s) [seg_pos={0 0}, upload=332700]: Done begin upload segment. Mar 18 20:02:43.834262428 (297h2m41.431033451s) [noise=true , address=145.79.227.52:4451 , piece_id=LR6KVRYMDO2HAESG5BFI5RYPNGVARP6O5HEY2UP77YSTVRKBNUXQ, seg_pos={0 0}, upload=332700]: Uploading piece... Mar 18 20:02:43.834329472 (297h2m41.431100497s) [noise=true , address=45.91.116.13:4164 , piece_id=MNQLVANO3LFKBCJMADD5RULARMRO4VUQY7LT63TU2CCNO765GW7Q, seg_pos={0 0}, upload=332700]: Uploading piece... Mar 18 20:02:43.834938832 (297h2m41.431709858s) [noise=true , address=77.90.31.15:4716 , piece_id=IDQP2MIPCKVS2FGVVGKAIDYTNUIPE56EER5IYZUSOTRD5LZBSXHA, seg_pos={0 0}, upload=332700]: Uploading piece... Mar 18 20:02:43.835898822 (297h2m41.432669851s) [noise=true , address=79.127.205.237:4227 , piece_id=NO25WIYZTEPBOMGETTLSDOJSPKARMKNLVUWDKNXUKHYGDGF2XC5Q, seg_pos={0 0}, upload=332700]: Uploading piece... Mar 18 20:02:43.836893726 (297h2m41.433664759s) [noise=true , address=193.149.164.68:4064 , piece_id=HAA3NDB24MJLAI2HK2UDZQZRLDUXEF76ZJF2VBE6FLKRWSB27Y3A, seg_pos={0 0}, upload=332700]: Uploading piece... Mar 18 20:02:43.837257949 (297h2m41.434028981s) [noise=true , address=193.149.164.66:4974 , piece_id=XWMBTFEWXVVFIR6IYNQYU35QKGVSQJ7YBVSWZ57QSUZ2VSDCJPRA, seg_pos={0 0}, upload=332700]: Uploading piece... Mar 18 20:02:43.838266356 (297h2m41.435037391s) [noise=true , address=145.79.227.34:4924 , piece_id=BNUESPWRZQBX23K7UGNSMF362AWHILNZ43UITBHVGFXIJ4T6V5HQ, seg_pos={0 0}, upload=332700]: Uploading piece... Mar 18 20:02:43.838688582 (297h2m41.435459605s) [noise=true , address=79.127.205.233:4171 , piece_id=YO42WSHYRB2JJX4RVX653N7U6PYTGIW3KQTNSF4445OTN3UW2TIA, seg_pos={0 0}, upload=332700]: Uploading piece... Mar 18 20:02:43.839263235 (297h2m41.436034258s) [noise=true , address=193.149.164.70:4123 , piece_id=VROUAQKRTKZT42ZDDSCIMKFMRODPKFQ5U5DWDJLVI462BVA5YWXA, seg_pos={0 0}, upload=332700]: Uploading piece... Mar 18 20:02:43.839835928 (297h2m41.436606963s) [noise=true , address=45.91.116.4:4948 , piece_id=ULVH4YAJIYY5RWWAGH5RTSZ3BMAT6WCYYAJPSMSP25LCLZQIPG2A, seg_pos={0 0}, upload=332700]: Uploading piece... Mar 18 20:02:43.840390028 (297h2m41.437161055s) [noise=true , address=45.91.116.5:4319 , piece_id=BY6BRNZNKKQKZNAVJTSWUYASX57P5X7OCKQ433T2ARRBXUFNBRCA, seg_pos={0 0}, upload=332700]: Uploading piece... Mar 18 20:02:43.840790648 (297h2m41.437561682s) [noise=true , address=45.91.116.12:4259 , piece_id=4LBHNF3CIQG5LYEY7RYJGNK5Q62X2TSOAXOXSNHO727OXL5OMA4Q, seg_pos={0 0}, upload=332700]: Uploading piece... Mar 18 20:02:43.841319562 (297h2m41.438090584s) [noise=true , address=145.79.227.29:4966 , piece_id=ZTSKMRBESX4OIZMMIPXRW752W5KQYX57RPDQ22XXPURQTSCAI2YA, seg_pos={0 0}, upload=332700]: Uploading piece... Mar 18 20:02:43.841816046 (297h2m41.438587078s) [noise=true , address=193.149.164.71:4287 , piece_id=FYNZ3QGUB7ZFMUJAMYPMUHXJDS66CJ2VLXPV7256DFCABPJMEOYQ, seg_pos={0 0}, upload=332700]: Uploading piece... Mar 18 20:02:43.842288870 (297h2m41.439059903s) [noise=true , address=79.127.219.41:4941 , piece_id=CQXOXMCJHWJ7FSIXDVYUJZMBFW5FUUOJNKJJOVLZRSK5D6XZXX5Q, seg_pos={0 0}, upload=332700]: Uploading piece... Mar 18 20:02:43.842956832 (297h2m41.439727861s) [noise=true , address=145.79.227.49:4369 , piece_id=CBYHFRBNKSPUNJHAE6S72YOFACWLZKGZIFV6SXWDYMVSA4FBPKQQ, seg_pos={0 0}, upload=332700]: Uploading piece... Mar 18 20:02:43.844386533 (297h2m41.441157559s) [noise=true , address=77.90.31.10:4533 , piece_id=VB5BKAWDUZFVII4ZY3JRVK67VUULPSAHTJVZMXXVATP3SXAR2XHQ, seg_pos={0 0}, upload=332700]: Uploading piece... Mar 18 20:02:43.844868553 (297h2m41.441639576s) [noise=true , address=145.79.227.37:4582 , piece_id=DLOFMRZRG3BPFN7JF73OM5HUV56FLEH2IDDC47UQWWWOTRVRR4HQ, seg_pos={0 0}, upload=332700]: Uploading piece... Mar 18 20:02:43.903559633 (297h2m41.500330666s) [upload=332700]: Next returned nil segment Mar 18 20:02:43.903578892 (297h2m41.500349925s) [upload=332700]: Waiting for error group managing segments... Mar 18 20:02:43.908218461 (297h2m41.504989474s) [noise=true , address=45.91.116.4:4948 , piece_id=ULVH4YAJIYY5RWWAGH5RTSZ3BMAT6WCYYAJPSMSP25LCLZQIPG2A, seg_pos={0 0}, upload=332700]: Done uploading piece. err: <nil> Mar 18 20:02:43.917679744 (297h2m41.514450753s) [noise=true , address=45.91.116.12:4259 , piece_id=4LBHNF3CIQG5LYEY7RYJGNK5Q62X2TSOAXOXSNHO727OXL5OMA4Q, seg_pos={0 0}, upload=332700]: Done uploading piece. err: <nil> Mar 18 20:02:43.924357408 (297h2m41.521128422s) [noise=true , address=77.90.31.10:4533 , piece_id=VB5BKAWDUZFVII4ZY3JRVK67VUULPSAHTJVZMXXVATP3SXAR2XHQ, seg_pos={0 0}, upload=332700]: Done uploading piece. err: <nil> Mar 18 20:02:43.926248407 (297h2m41.52301943s) [noise=true , address=77.90.31.15:4716 , piece_id=IDQP2MIPCKVS2FGVVGKAIDYTNUIPE56EER5IYZUSOTRD5LZBSXHA, seg_pos={0 0}, upload=332700]: Done uploading piece. err: <nil> Mar 18 20:02:43.927212300 (297h2m41.523983323s) [noise=true , address=45.91.116.5:4319 , piece_id=BY6BRNZNKKQKZNAVJTSWUYASX57P5X7OCKQ433T2ARRBXUFNBRCA, seg_pos={0 0}, upload=332700]: Done uploading piece. err: <nil> Mar 18 20:02:43.927791610 (297h2m41.524562634s) [noise=true , address=45.91.116.13:4164 , piece_id=MNQLVANO3LFKBCJMADD5RULARMRO4VUQY7LT63TU2CCNO765GW7Q, seg_pos={0 0}, upload=332700]: Done uploading piece. err: <nil> Mar 18 20:02:44.028504592 (297h2m41.625275611s) [noise=true , address=79.127.219.41:4941 , piece_id=CQXOXMCJHWJ7FSIXDVYUJZMBFW5FUUOJNKJJOVLZRSK5D6XZXX5Q, seg_pos={0 0}, upload=332700]: Done uploading piece. err: <nil> Mar 18 20:02:44.392266647 (297h2m41.989037671s) [noise=true , address=79.127.205.233:4171 , piece_id=YO42WSHYRB2JJX4RVX653N7U6PYTGIW3KQTNSF4445OTN3UW2TIA, seg_pos={0 0}, upload=332700]: Done uploading piece. err: <nil> Mar 18 20:02:44.429403642 (297h2m42.026174669s) [noise=true , address=193.149.164.66:4974 , piece_id=XWMBTFEWXVVFIR6IYNQYU35QKGVSQJ7YBVSWZ57QSUZ2VSDCJPRA, seg_pos={0 0}, upload=332700]: Done uploading piece. err: <nil> Mar 18 20:02:44.433374828 (297h2m42.030145853s) [noise=true , address=193.149.164.68:4064 , piece_id=HAA3NDB24MJLAI2HK2UDZQZRLDUXEF76ZJF2VBE6FLKRWSB27Y3A, seg_pos={0 0}, upload=332700]: Done uploading piece. err: <nil> Mar 18 20:02:44.444389014 (297h2m42.041160042s) [noise=true , address=193.149.164.71:4287 , piece_id=FYNZ3QGUB7ZFMUJAMYPMUHXJDS66CJ2VLXPV7256DFCABPJMEOYQ, seg_pos={0 0}, upload=332700]: Done uploading piece. err: <nil> Mar 18 20:02:44.578333498 (297h2m42.175104529s) [noise=true , address=79.127.205.237:4227 , piece_id=NO25WIYZTEPBOMGETTLSDOJSPKARMKNLVUWDKNXUKHYGDGF2XC5Q, seg_pos={0 0}, upload=332700]: Done uploading piece. err: <nil> Mar 18 20:02:44.665665140 (297h2m42.262436168s) [noise=true , address=193.149.164.70:4123 , piece_id=VROUAQKRTKZT42ZDDSCIMKFMRODPKFQ5U5DWDJLVI462BVA5YWXA, seg_pos={0 0}, upload=332700]: Done uploading piece. err: <nil> Mar 18 20:03:40.987108256 (297h3m38.58387928s) [noise=true , address=145.79.227.49:4369 , piece_id=CBYHFRBNKSPUNJHAE6S72YOFACWLZKGZIFV6SXWDYMVSA4FBPKQQ, seg_pos={0 0}, upload=332700]: Done uploading piece. err: ecclient: upload failed (node:12nLksGXGvDwuTWfnmkceuwZNDmrF17iNepHPdYkkvZG2QQUwG7, address:145.79.227.49:4369): protocol: rpc: dial tcp 145.79.227.49:4369: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:03:40.987479807 (297h3m38.58425084s) [noise=true , address=145.79.227.29:4966 , piece_id=ZTSKMRBESX4OIZMMIPXRW752W5KQYX57RPDQ22XXPURQTSCAI2YA, seg_pos={0 0}, upload=332700]: Done uploading piece. err: ecclient: upload failed (node:1DLnsYVdudE4mY9dNAQKhBRppUEAHvxbf84KX69451pwxSxzZK, address:145.79.227.29:4966): protocol: rpc: dial tcp 145.79.227.29:4966: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:03:40.987517864 (297h3m38.584288896s) [noise=true , address=145.79.227.34:4924 , piece_id=BNUESPWRZQBX23K7UGNSMF362AWHILNZ43UITBHVGFXIJ4T6V5HQ, seg_pos={0 0}, upload=332700]: Done uploading piece. err: ecclient: upload failed (node:19TbrQjuN1EhBNVtha9EcKPSCAWyttfmHpec6QeF1S5zW6DRZs, address:145.79.227.34:4924): protocol: rpc: dial tcp 145.79.227.34:4924: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:03:40.987933296 (297h3m38.584704329s) [noise=true , address=145.79.227.37:4582 , piece_id=DLOFMRZRG3BPFN7JF73OM5HUV56FLEH2IDDC47UQWWWOTRVRR4HQ, seg_pos={0 0}, upload=332700]: Done uploading piece. err: ecclient: upload failed (node:129o91j97t4V5AjaouCCy2wDuHgwF94iQiJh2h9ux1aFo7AVCSn, address:145.79.227.37:4582): protocol: rpc: dial tcp 145.79.227.37:4582: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:03:40.988167350 (297h3m38.584938376s) [noise=true , address=145.79.227.52:4451 , piece_id=LR6KVRYMDO2HAESG5BFI5RYPNGVARP6O5HEY2UP77YSTVRKBNUXQ, seg_pos={0 0}, upload=332700]: Done uploading piece. err: ecclient: upload failed (node:12bC4P3xBiPFkxLvavGFVbWKnbprCStL6UJgfKSA2Wd2fX7Mbfw, address:145.79.227.52:4451): protocol: rpc: dial tcp 145.79.227.52:4451: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:03:40.988197047 (297h3m38.58496806s) [seg_pos={0 0}, upload=332700]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 18 20:03:41.007727578 (297h3m38.604498608s) [upload=332700]: Done uploading segments... Mar 18 20:03:41.007779207 (297h3m38.60455025s) [upload=332700]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 16.390067637s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 40 out of 60 is failed (threshold=3) [...................XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.XXX] duration: 30.038512763s trace: https://jaeger.dev.storj.io/trace/5b0f747cb1d1940 FAIL (2026-03-18 20:05:13 UTC) operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 9) ======LOGS====== Mar 18 20:04:43.234379658 (297h4m40.831150682s) [upload=332739]: Starting upload Mar 18 20:04:43.234514357 (297h4m40.831285372s) [upload=332739]: Done starting upload Mar 18 20:04:43.234770926 (297h4m40.831541951s) [upload=332739]: Uploading segments... Mar 18 20:04:43.234845492 (297h4m40.831616527s) [upload=332739]: Got next segment. Inline: false Mar 18 20:04:43.234894018 (297h4m40.831665048s) [upload=332739]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 20:04:43.234900244 (297h4m40.831671275s) [upload=332739]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 18 20:04:46.305008694 (297h4m43.901779721s) [seg_pos={0 0}, upload=332739]: Begin upload segment... Mar 18 20:04:46.305120280 (297h4m43.901891293s) [seg_pos={0 0}, upload=332739]: Done begin upload segment. Mar 18 20:04:46.305158251 (297h4m43.901929264s) [noise=true , address=79.127.219.45:4855 , piece_id=5IWJJQGJUSZ3CPA3BTRZO7KQCPWRUPAE3TSWUMJJIM677IGO3XHQ, seg_pos={0 0}, upload=332739]: Uploading piece... Mar 18 20:04:46.305177128 (297h4m43.901948157s) [noise=true , address=79.127.219.46:4894 , piece_id=YQFFTQXF7J324ZJNXHQQAGMHHOVH4MPZJJICQLQNSF3IAQIYOTSA, seg_pos={0 0}, upload=332739]: Uploading piece... Mar 18 20:04:46.306133788 (297h4m43.902904804s) [noise=true , address=145.79.227.4:4187 , piece_id=Q7G7UMVTSVRUPDUQRLAMUWAJZQYRLV5VYPHGVBR5ZYCYFEDAONNA, seg_pos={0 0}, upload=332739]: Uploading piece... Mar 18 20:04:46.306564638 (297h4m43.90333567s) [noise=true , address=145.79.227.15:4878 , piece_id=C3KOXZFMFPP57V3VN2UAIBFSFIAH7KE75AS7PCABGXRFEIGV7NSA, seg_pos={0 0}, upload=332739]: Uploading piece... Mar 18 20:04:46.307193485 (297h4m43.903964513s) [noise=true , address=145.79.227.43:4161 , piece_id=CAAZUUUGJT2WCGE5O2MOSATKOEOLCCQ5BFY5T34BD2CO2HHEX3RA, seg_pos={0 0}, upload=332739]: Uploading piece... Mar 18 20:04:46.307902859 (297h4m43.904673886s) [noise=true , address=45.91.116.14:4193 , piece_id=GBW5Q6W2PZFND5YHOZXVND7LC2ZX6VVRWY7ER323FSLACTLCNJ3A, seg_pos={0 0}, upload=332739]: Uploading piece... Mar 18 20:04:46.308536118 (297h4m43.905307152s) [noise=true , address=45.91.116.8:4080 , piece_id=REQE5APRO3HJBZH6YJPEBRZKIQXH66QMRQPDTYNKIFBBC5HB67JQ, seg_pos={0 0}, upload=332739]: Uploading piece... Mar 18 20:04:46.308697053 (297h4m43.90546808s) [noise=true , address=45.91.116.13:4188 , piece_id=GIARN34QZRUYXVDSOTBRHDKRUBVINI2NR6P4OVOKMYKL6OSC45HA, seg_pos={0 0}, upload=332739]: Uploading piece... Mar 18 20:04:46.309074759 (297h4m43.90584579s) [noise=true , address=145.79.227.34:4909 , piece_id=3BEBSOA7O3VDPZ6NIWBMC2GFN3YRENXIZAKB767XNDDURY65PP6A, seg_pos={0 0}, upload=332739]: Uploading piece... Mar 18 20:04:46.315117785 (297h4m43.911888802s) [noise=true , address=145.79.227.41:4088 , piece_id=44URON2V56SHZQJ7ZFI32IRK23I4KUW4A5CAR3SXD4VLQWFOMISQ, seg_pos={0 0}, upload=332739]: Uploading piece... Mar 18 20:04:46.317084390 (297h4m43.913855406s) [noise=true , address=145.79.227.44:4197 , piece_id=3BS5H7VQB2R7SE5BUUG6KJSHB4ODLTA2R4FDGFR5ANV4L2UP3DOA, seg_pos={0 0}, upload=332739]: Uploading piece... Mar 18 20:04:46.317246758 (297h4m43.914017774s) [noise=true , address=45.91.116.4:4941 , piece_id=UBPMA2YWYT4FE67ABONUQLODRKAJ5N2HYHADCZ2HQWJ7HPFVVIKA, seg_pos={0 0}, upload=332739]: Uploading piece... Mar 18 20:04:46.317917550 (297h4m43.914688566s) [noise=true , address=45.91.116.5:4298 , piece_id=ECUBRTF37AWUWRWRNNDN4FJH5BHO7YSESAASFCJTCSWWMSCRS7EA, seg_pos={0 0}, upload=332739]: Uploading piece... Mar 18 20:04:46.319528805 (297h4m43.916299814s) [noise=true , address=45.91.116.10:4081 , piece_id=BQ4PUN5OGWGFFOIWNUES5JQJGRAK3FMWXSZ4VRQOFMVDLTM2U3QA, seg_pos={0 0}, upload=332739]: Uploading piece... Mar 18 20:04:46.320610417 (297h4m43.917381435s) [noise=true , address=77.90.31.11:4558 , piece_id=KAKDGD2BSMFOJBSZJTFG2V6Y36MDO2ZK4RMOEJUL4T6GQTU3JEOQ, seg_pos={0 0}, upload=332739]: Uploading piece... Mar 18 20:04:46.322784861 (297h4m43.919555886s) [noise=true , address=145.79.227.2:4118 , piece_id=47OWZUWJNBWDYM2SZJZDRPCYKL5ISSBOMU33DGW4BNMN6B7RLFEA, seg_pos={0 0}, upload=332739]: Uploading piece... Mar 18 20:04:46.390839310 (297h4m43.987610334s) [upload=332739]: Next returned nil segment Mar 18 20:04:46.390854010 (297h4m43.987625034s) [upload=332739]: Waiting for error group managing segments... Mar 18 20:04:46.395927923 (297h4m43.992698936s) [noise=true , address=45.91.116.8:4080 , piece_id=REQE5APRO3HJBZH6YJPEBRZKIQXH66QMRQPDTYNKIFBBC5HB67JQ, seg_pos={0 0}, upload=332739]: Done uploading piece. err: <nil> Mar 18 20:04:46.396043608 (297h4m43.992814635s) [noise=true , address=77.90.31.11:4558 , piece_id=KAKDGD2BSMFOJBSZJTFG2V6Y36MDO2ZK4RMOEJUL4T6GQTU3JEOQ, seg_pos={0 0}, upload=332739]: Done uploading piece. err: <nil> Mar 18 20:04:46.396543561 (297h4m43.993314584s) [noise=true , address=79.127.219.46:4894 , piece_id=YQFFTQXF7J324ZJNXHQQAGMHHOVH4MPZJJICQLQNSF3IAQIYOTSA, seg_pos={0 0}, upload=332739]: Done uploading piece. err: <nil> Mar 18 20:04:46.396590475 (297h4m43.993361488s) [noise=true , address=45.91.116.14:4193 , piece_id=GBW5Q6W2PZFND5YHOZXVND7LC2ZX6VVRWY7ER323FSLACTLCNJ3A, seg_pos={0 0}, upload=332739]: Done uploading piece. err: <nil> Mar 18 20:04:46.398006800 (297h4m43.994777814s) [noise=true , address=45.91.116.4:4941 , piece_id=UBPMA2YWYT4FE67ABONUQLODRKAJ5N2HYHADCZ2HQWJ7HPFVVIKA, seg_pos={0 0}, upload=332739]: Done uploading piece. err: <nil> Mar 18 20:04:46.398376069 (297h4m43.995147081s) [noise=true , address=45.91.116.13:4188 , piece_id=GIARN34QZRUYXVDSOTBRHDKRUBVINI2NR6P4OVOKMYKL6OSC45HA, seg_pos={0 0}, upload=332739]: Done uploading piece. err: <nil> Mar 18 20:04:46.411441727 (297h4m44.008212756s) [noise=true , address=79.127.219.45:4855 , piece_id=5IWJJQGJUSZ3CPA3BTRZO7KQCPWRUPAE3TSWUMJJIM677IGO3XHQ, seg_pos={0 0}, upload=332739]: Done uploading piece. err: <nil> Mar 18 20:04:46.413072093 (297h4m44.009843105s) [noise=true , address=45.91.116.5:4298 , piece_id=ECUBRTF37AWUWRWRNNDN4FJH5BHO7YSESAASFCJTCSWWMSCRS7EA, seg_pos={0 0}, upload=332739]: Done uploading piece. err: <nil> Mar 18 20:04:46.415190026 (297h4m44.011961036s) [noise=true , address=45.91.116.10:4081 , piece_id=BQ4PUN5OGWGFFOIWNUES5JQJGRAK3FMWXSZ4VRQOFMVDLTM2U3QA, seg_pos={0 0}, upload=332739]: Done uploading piece. err: <nil> Mar 18 20:05:13.201917265 (297h5m10.798688298s) [noise=true , address=145.79.227.15:4878 , piece_id=C3KOXZFMFPP57V3VN2UAIBFSFIAH7KE75AS7PCABGXRFEIGV7NSA, seg_pos={0 0}, upload=332739]: Done uploading piece. err: ecclient: upload failed (node:126ZeAKnpjpxa3MgCBU3KZKBDdn8MfZgiixE4zR5fZpPwaQj9NC, address:145.79.227.15:4878): protocol: rpc: dial tcp 145.79.227.15:4878: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:05:13.202094819 (297h5m10.798865855s) [noise=true , address=145.79.227.2:4118 , piece_id=47OWZUWJNBWDYM2SZJZDRPCYKL5ISSBOMU33DGW4BNMN6B7RLFEA, seg_pos={0 0}, upload=332739]: Done uploading piece. err: ecclient: upload failed (node:12ty5LHXwLmHC7nRPmVfj2wVEif6JQAQQ5Z5218vv1fKK5sD3zE, address:145.79.227.2:4118): protocol: rpc: dial tcp 145.79.227.2:4118: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:05:13.202179825 (297h5m10.798950855s) [noise=true , address=145.79.227.41:4088 , piece_id=44URON2V56SHZQJ7ZFI32IRK23I4KUW4A5CAR3SXD4VLQWFOMISQ, seg_pos={0 0}, upload=332739]: Done uploading piece. err: ecclient: upload failed (node:1284h6cPNVXJYMkSekGQVw8b7p89tkhpmgYLxTTYUDu2J7Wadoj, address:145.79.227.41:4088): protocol: rpc: dial tcp 145.79.227.41:4088: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:05:13.202393357 (297h5m10.799164391s) [noise=true , address=145.79.227.4:4187 , piece_id=Q7G7UMVTSVRUPDUQRLAMUWAJZQYRLV5VYPHGVBR5ZYCYFEDAONNA, seg_pos={0 0}, upload=332739]: Done uploading piece. err: ecclient: upload failed (node:12cQxBB2L2CyymYx3xWp1ALj6p8QWr4A34sqvtG5zyeT7sVWv7x, address:145.79.227.4:4187): protocol: rpc: dial tcp 145.79.227.4:4187: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:05:13.202457005 (297h5m10.79922803s) [noise=true , address=145.79.227.44:4197 , piece_id=3BS5H7VQB2R7SE5BUUG6KJSHB4ODLTA2R4FDGFR5ANV4L2UP3DOA, seg_pos={0 0}, upload=332739]: Done uploading piece. err: ecclient: upload failed (node:12niHLkT5pbmYSbarHZGeeCRtwjZRXtvEtHMGAb7ja4a7Xuuvfi, address:145.79.227.44:4197): protocol: rpc: dial tcp 145.79.227.44:4197: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:05:13.202577343 (297h5m10.799348409s) [noise=true , address=145.79.227.43:4161 , piece_id=CAAZUUUGJT2WCGE5O2MOSATKOEOLCCQ5BFY5T34BD2CO2HHEX3RA, seg_pos={0 0}, upload=332739]: Done uploading piece. err: ecclient: upload failed (node:12pyhMcYjL5b9posZurgKXBnCHaMhaEjL3q3YWyc3aWkcYwbvTD, address:145.79.227.43:4161): protocol: context deadline exceeded; context deadline exceeded Mar 18 20:05:13.202883680 (297h5m10.799654714s) [noise=true , address=145.79.227.34:4909 , piece_id=3BEBSOA7O3VDPZ6NIWBMC2GFN3YRENXIZAKB767XNDDURY65PP6A, seg_pos={0 0}, upload=332739]: Done uploading piece. err: ecclient: upload failed (node:14Pqp5cbhmd44QnikAuD5TgsKFDR3gB2qtLuVSroaWVv4U2cDh, address:145.79.227.34:4909): protocol: context deadline exceeded; context deadline exceeded Mar 18 20:05:13.202915567 (297h5m10.799686582s) [seg_pos={0 0}, upload=332739]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Mar 18 20:05:13.236347420 (297h5m10.833118444s) [upload=332739]: Done uploading segments... Mar 18 20:05:13.236382958 (297h5m10.833153982s) [upload=332739]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Duration: 30.038541666s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 8 out of 60 is failed (threshold=3) [......................XXXXXX.XX.............................] duration: 16.269942278s trace: https://jaeger.dev.storj.io/trace/4b4d10a8051ad01b FAIL (2026-03-18 20:05:11 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 16.26997891s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 48 out of 60 is failed (threshold=6) [........XXXXXXXXX.XXX...XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 5.001058325s trace: https://jaeger.dev.storj.io/trace/9a0c0d8fe66fe7c FAIL (2026-03-18 20:05:01 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.00108084s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 6 out of 60 is failed (threshold=3) [..........................X..........XX..X....X........X....] duration: 17.612594111s trace: https://jaeger.dev.storj.io/trace/589419efc52e5e2b FAIL (2026-03-18 20:05:00 UTC) operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== Mar 18 20:00:59.343084966 (297h0m56.939855989s) [upload=332666]: Starting upload Mar 18 20:00:59.343175518 (297h0m56.939946548s) [upload=332666]: Done starting upload Mar 18 20:00:59.343565585 (297h0m56.940336612s) [upload=332666]: Uploading segments... Mar 18 20:00:59.344256647 (297h0m56.941027676s) [upload=332666]: Got next segment. Inline: false Mar 18 20:00:59.344352215 (297h0m56.941123241s) [upload=332666]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 20:00:59.344358706 (297h0m56.941129732s) [upload=332666]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 18 20:01:02.340259299 (297h0m59.937030329s) [seg_pos={0 0}, upload=332666]: Begin upload segment... Mar 18 20:01:02.340368743 (297h0m59.937139767s) [seg_pos={0 0}, upload=332666]: Done begin upload segment. Mar 18 20:01:02.340479563 (297h0m59.93725059s) [noise=true , address=45.91.116.8:4061 , piece_id=6XFVKT4AAO7QN457OQMRKPI6R24TJTAN47UXLEVOEFQVOS6M6C3Q, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.340523837 (297h0m59.93729486s) [noise=true , address=79.127.205.233:4166 , piece_id=SFGK5YCWHFRVXF6QNHYP2CYJCZLSTQ76OYRVJPWWXQTF2KFSFJKA, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.341468438 (297h0m59.938239447s) [noise=true , address=45.91.116.4:4961 , piece_id=ZMUGQECUVPVHDNKN75XPBET5M2HFXXK4WMTP4V5KEK63JVZCX5EA, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.341520025 (297h0m59.938291058s) [noise=true , address=167.17.72.71:4815 , piece_id=CXGTYNNBILN3TT7I6Z7UI75F2ZCS4QUTLS6KC4FJ5IHS3XOYWDJQ, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.342232992 (297h0m59.939004023s) [noise=true , address=145.79.227.36:4524 , piece_id=3TAV3P55ITTHBAZEUUUTOAHJRGQXJXXVGZJP6AVR2ANVNOR3S54Q, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.342917368 (297h0m59.939688398s) [noise=true , address=193.149.164.71:4297 , piece_id=4IJPXLPCX3LKFOBOSRHK2DTNRKH67K7FXJUKQ3JPRWHS47ZFEIDA, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.343244957 (297h0m59.940015982s) [noise=true , address=145.79.227.33:4892 , piece_id=DQ6ALC3VOAGFIJI3IKR6I7QFYM22C66QWAAXMH6NXI2FIQEZNQYA, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.344270079 (297h0m59.941041117s) [noise=true , address=145.79.227.21:4497 , piece_id=SP6EKJB7W4QXF7OIBVKEFWCPOZFL5IRIEDVAGS2M4QJSO34CUVRQ, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.345025057 (297h0m59.941796084s) [noise=true , address=77.90.31.17:4793 , piece_id=2U4IBUA2GGUGDBXQGIGQKUK4ZR3L7CDKC6MTGIKAUME4F7MFZRLA, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.345115802 (297h0m59.941886836s) [noise=true , address=167.17.72.69:4731 , piece_id=VOEVIKKFJMC2Y3TPQIDMYUP37NLFU7KPRH4HRRSMQLWUKTSLOH2Q, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.345690600 (297h0m59.942461634s) [noise=true , address=77.90.31.13:4648 , piece_id=BPMB5CFOYHFT3BS4ORR6JF6BNFC2AOAJZOSPMG67NAMLGGP5XVEQ, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.346311592 (297h0m59.943082623s) [noise=true , address=79.127.205.234:4202 , piece_id=YY4LYWX26O6RIOBONESNKTUZEFZCRL4UUX3NEBQ53AJV4IEDG5MQ, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.346791142 (297h0m59.943562166s) [noise=true , address=45.91.116.10:4110 , piece_id=UYLMDCWJEJOHGICA37KNH7G5AQXNIPRH5NRYLBUUMGDRVSLTEYHQ, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.347094999 (297h0m59.94386607s) [noise=true , address=145.79.227.17:4987 , piece_id=PT3TWVB65JZFEWVH4IUZEU2N53FQ57QBRWVLWAUFP4UDLY3LWNLQ, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.347781281 (297h0m59.944552305s) [noise=true , address=79.127.219.46:4919 , piece_id=KK6HRIVBMYLIHRHGPQH3AWMNSVHTTIBESHDMJZA72T446RX6QKRQ, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.348263123 (297h0m59.945034137s) [noise=true , address=193.149.164.68:4033 , piece_id=JZPQKPYGO3W3RZZBITQ73X2XEPIBBSLGRMWIVOM4Z3GTV5UFKRTQ, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.348915505 (297h0m59.945686536s) [noise=true , address=145.79.227.7:4312 , piece_id=2FAL6E3RMBTBHZBXAJJVH2N5VGF5VYI3KNZXGNWCMWGBBPAUO2NQ, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.349366298 (297h0m59.946137337s) [noise=true , address=193.149.164.67:4007 , piece_id=HURQGH67I5B6V4YUV3PRZL3KSKN3XTSK3UMCHZRNT6L3W3PXXHPQ, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.429834098 (297h1m0.026605126s) [upload=332666]: Next returned nil segment Mar 18 20:01:02.429853411 (297h1m0.026624442s) [upload=332666]: Waiting for error group managing segments... Mar 18 20:01:02.446867680 (297h1m0.043638704s) [noise=true , address=45.91.116.10:4110 , piece_id=UYLMDCWJEJOHGICA37KNH7G5AQXNIPRH5NRYLBUUMGDRVSLTEYHQ, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil> Mar 18 20:01:02.451665165 (297h1m0.048436196s) [noise=true , address=45.91.116.4:4961 , piece_id=ZMUGQECUVPVHDNKN75XPBET5M2HFXXK4WMTP4V5KEK63JVZCX5EA, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil> Mar 18 20:01:02.452538886 (297h1m0.049309913s) [noise=true , address=77.90.31.17:4793 , piece_id=2U4IBUA2GGUGDBXQGIGQKUK4ZR3L7CDKC6MTGIKAUME4F7MFZRLA, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil> Mar 18 20:01:02.455597040 (297h1m0.052368067s) [noise=true , address=45.91.116.8:4061 , piece_id=6XFVKT4AAO7QN457OQMRKPI6R24TJTAN47UXLEVOEFQVOS6M6C3Q, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil> Mar 18 20:01:02.457848739 (297h1m0.054619766s) [noise=true , address=77.90.31.13:4648 , piece_id=BPMB5CFOYHFT3BS4ORR6JF6BNFC2AOAJZOSPMG67NAMLGGP5XVEQ, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil> Mar 18 20:01:02.461479709 (297h1m0.058250746s) [noise=true , address=79.127.219.46:4919 , piece_id=KK6HRIVBMYLIHRHGPQH3AWMNSVHTTIBESHDMJZA72T446RX6QKRQ, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil> Mar 18 20:01:02.676859466 (297h1m0.27363049s) [noise=true , address=167.17.72.69:4731 , piece_id=VOEVIKKFJMC2Y3TPQIDMYUP37NLFU7KPRH4HRRSMQLWUKTSLOH2Q, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil> Mar 18 20:01:02.909860743 (297h1m0.506631766s) [noise=true , address=79.127.205.234:4202 , piece_id=YY4LYWX26O6RIOBONESNKTUZEFZCRL4UUX3NEBQ53AJV4IEDG5MQ, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil> Mar 18 20:01:02.930427742 (297h1m0.527198754s) [noise=true , address=193.149.164.71:4297 , piece_id=4IJPXLPCX3LKFOBOSRHK2DTNRKH67K7FXJUKQ3JPRWHS47ZFEIDA, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil> Mar 18 20:01:02.944696889 (297h1m0.541467914s) [noise=true , address=193.149.164.67:4007 , piece_id=HURQGH67I5B6V4YUV3PRZL3KSKN3XTSK3UMCHZRNT6L3W3PXXHPQ, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil> Mar 18 20:01:02.951319183 (297h1m0.548090212s) [noise=true , address=193.149.164.68:4033 , piece_id=JZPQKPYGO3W3RZZBITQ73X2XEPIBBSLGRMWIVOM4Z3GTV5UFKRTQ, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil> Mar 18 20:01:02.975619436 (297h1m0.572390467s) [noise=true , address=79.127.205.233:4166 , piece_id=SFGK5YCWHFRVXF6QNHYP2CYJCZLSTQ76OYRVJPWWXQTF2KFSFJKA, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil> Mar 18 20:01:03.819918528 (297h1m1.416689564s) [noise=true , address=167.17.72.71:4815 , piece_id=CXGTYNNBILN3TT7I6Z7UI75F2ZCS4QUTLS6KC4FJ5IHS3XOYWDJQ, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil> Mar 18 20:01:59.307873420 (297h1m56.904644452s) [noise=true , address=145.79.227.7:4312 , piece_id=2FAL6E3RMBTBHZBXAJJVH2N5VGF5VYI3KNZXGNWCMWGBBPAUO2NQ, seg_pos={0 0}, upload=332666]: Done uploading piece. err: ecclient: upload failed (node:12AVp3qn5uwLPqYaZEwzcgoGMJhNga1BUh6bK5k2bxjWgZLQiiC, address:145.79.227.7:4312): protocol: rpc: dial tcp 145.79.227.7:4312: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:01:59.308026092 (297h1m56.904797124s) [noise=true , address=145.79.227.17:4987 , piece_id=PT3TWVB65JZFEWVH4IUZEU2N53FQ57QBRWVLWAUFP4UDLY3LWNLQ, seg_pos={0 0}, upload=332666]: Done uploading piece. err: ecclient: upload failed (node:1XsPVNGYUhKnXkQRfVfjyBRzyvhShSdsrnzXtwe4Xtf5MaHMvH, address:145.79.227.17:4987): protocol: rpc: dial tcp 145.79.227.17:4987: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:01:59.308195864 (297h1m56.904966894s) [noise=true , address=145.79.227.36:4524 , piece_id=3TAV3P55ITTHBAZEUUUTOAHJRGQXJXXVGZJP6AVR2ANVNOR3S54Q, seg_pos={0 0}, upload=332666]: Done uploading piece. err: ecclient: upload failed (node:1xfbCzjSwFDRrR1gPggeN8tNCLbRRWDVzCJ6mqhZsbVSe8d4s4, address:145.79.227.36:4524): protocol: rpc: dial tcp 145.79.227.36:4524: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:01:59.308316037 (297h1m56.90508706s) [noise=true , address=145.79.227.33:4892 , piece_id=DQ6ALC3VOAGFIJI3IKR6I7QFYM22C66QWAAXMH6NXI2FIQEZNQYA, seg_pos={0 0}, upload=332666]: Done uploading piece. err: ecclient: upload failed (node:12RHwFxcUjPsn12uHEAH3fBDJKv1v1tM8mRJXoFrZrG5djdswF5, address:145.79.227.33:4892): protocol: rpc: dial tcp 145.79.227.33:4892: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:01:59.308404403 (297h1m56.905175433s) [noise=true , address=145.79.227.21:4497 , piece_id=SP6EKJB7W4QXF7OIBVKEFWCPOZFL5IRIEDVAGS2M4QJSO34CUVRQ, seg_pos={0 0}, upload=332666]: Done uploading piece. err: ecclient: upload failed (node:12ZBe4we7VeFdthqRxyy1V9uQrA4r8ouNX68xqsk5bD26dkf2Gd, address:145.79.227.21:4497): protocol: context deadline exceeded; context deadline exceeded Mar 18 20:01:59.308437457 (297h1m56.90520847s) [seg_pos={0 0}, upload=332666]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 18 20:01:59.338288467 (297h1m56.93505949s) [upload=332666]: Done uploading segments... Mar 18 20:01:59.338335497 (297h1m56.935106523s) [upload=332666]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 17.612624533s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 39 out of 60 is failed (threshold=3) [....................XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.XX] duration: 30.02853497s trace: https://jaeger.dev.storj.io/trace/5bf3205ac7efb06 FAIL (2026-03-18 20:04:42 UTC) operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== Mar 18 20:04:12.773762542 (297h4m10.370533565s) [upload=332730]: Starting upload Mar 18 20:04:12.773839186 (297h4m10.370610202s) [upload=332730]: Done starting upload Mar 18 20:04:12.774141773 (297h4m10.370912789s) [upload=332730]: Uploading segments... Mar 18 20:04:12.774266255 (297h4m10.371037279s) [upload=332730]: Got next segment. Inline: false Mar 18 20:04:12.774342243 (297h4m10.371113275s) [upload=332730]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 20:04:12.774349964 (297h4m10.371120988s) [upload=332730]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 18 20:04:15.793968129 (297h4m13.390739155s) [seg_pos={0 0}, upload=332730]: Begin upload segment... Mar 18 20:04:15.794089989 (297h4m13.390861002s) [seg_pos={0 0}, upload=332730]: Done begin upload segment. Mar 18 20:04:15.794165504 (297h4m13.390936535s) [noise=true , address=145.79.227.4:4189 , piece_id=FFEAVSH2EK7YT3ET5C6Y56XAMRITYHQRFYLOC3WTGWI74OMOUT4Q, seg_pos={0 0}, upload=332730]: Uploading piece... Mar 18 20:04:15.794803278 (297h4m13.391574301s) [noise=true , address=77.90.31.18:4805 , piece_id=5L7MKDVCJWCG4KQ52CKJV5SQS2JZI5BGQRW4QXBJQV6QVXEB6BZA, seg_pos={0 0}, upload=332730]: Uploading piece... Mar 18 20:04:15.800788638 (297h4m13.397559671s) [noise=true , address=145.79.227.9:4369 , piece_id=GC6OCZ7CXOKBSQYFRGM7C3BX5WMIQ5MDIALFAPN2GNKYF4YIN2MQ, seg_pos={0 0}, upload=332730]: Uploading piece... Mar 18 20:04:15.802852907 (297h4m13.399623935s) [noise=true , address=77.90.31.11:4555 , piece_id=23M5BFJQKODZYDFM7SEYR5R4GBVDKLRUUBFM6CD7WHOZMLHTVPNQ, seg_pos={0 0}, upload=332730]: Uploading piece... Mar 18 20:04:15.803161757 (297h4m13.399932783s) [noise=true , address=45.91.116.13:4188 , piece_id=3DGAL7BTW2LROY5GX6WR2FTZEWGJNJ7UOVXOZBHWMLXTVBUTEJZA, seg_pos={0 0}, upload=332730]: Uploading piece... Mar 18 20:04:15.804603133 (297h4m13.401374166s) [noise=true , address=45.91.116.5:4297 , piece_id=2XGIQ7VHVOS6LZ3SSIK5XL2I4ACDPJ57BXKQCFPLZO6HI3ADTHLA, seg_pos={0 0}, upload=332730]: Uploading piece... Mar 18 20:04:15.805736701 (297h4m13.40250774s) [noise=true , address=145.79.227.51:4412 , piece_id=2RAPHKERYTBMLWB4IUWAT7NUSWR7EGOYEEOARZIQZIBQEYRJSYGA, seg_pos={0 0}, upload=332730]: Uploading piece... Mar 18 20:04:15.807683619 (297h4m13.404454648s) [noise=true , address=145.79.227.35:4482 , piece_id=X3UQEIIBUQ5263YMPQLGIRW6EHOOIB476XNJDCCSC4HBYEGKUYTQ, seg_pos={0 0}, upload=332730]: Uploading piece... Mar 18 20:04:15.809131179 (297h4m13.405902211s) [noise=true , address=145.79.227.43:4158 , piece_id=HQLX5SJEGF4IQ6GJIWZUOYFP7TL7IIB7MRU4IDYOA2S2JFS22VFA, seg_pos={0 0}, upload=332730]: Uploading piece... Mar 18 20:04:15.811051943 (297h4m13.407822974s) [noise=true , address=77.90.31.17:4769 , piece_id=L3UU2YI7SO5GJR2YXXLC2HW356VLG2GB4SOWBTSY27AO7EAMD62Q, seg_pos={0 0}, upload=332730]: Uploading piece... Mar 18 20:04:15.811766505 (297h4m13.408537537s) [noise=true , address=45.91.116.11:4121 , piece_id=EGOJ7I4GYLTLYZTLVOXK2YFUJGPB4L2T2EJ3XFNQQYNSGVXLGXEA, seg_pos={0 0}, upload=332730]: Uploading piece... Mar 18 20:04:15.812375245 (297h4m13.409146271s) [noise=true , address=45.91.116.12:4260 , piece_id=JCZ5KUJ7QBW5KIRZ7UHOCOTLTVQP4TLG4IYHAGQDHWMGDFO6D5GQ, seg_pos={0 0}, upload=332730]: Uploading piece... Mar 18 20:04:15.813116066 (297h4m13.409887102s) [noise=true , address=145.79.227.2:4119 , piece_id=3IGMZRN7TOTUQR6BCL6Y4UQCR4KIEFLHHGW37QXQRLZ2JTWOKAGQ, seg_pos={0 0}, upload=332730]: Uploading piece... Mar 18 20:04:15.813250180 (297h4m13.410021212s) [noise=true , address=45.91.116.9:4265 , piece_id=ODQG6YJNSJDMEGZR3DIDE2LV2VG4AZXR74OUROP4YURJ7CLGAJDA, seg_pos={0 0}, upload=332730]: Uploading piece... Mar 18 20:04:15.816392283 (297h4m13.413163315s) [noise=true , address=45.91.116.14:4191 , piece_id=ZF7KWZPHRJDVABY2OMX5H5G6CATYHPD2T32AKEWQ5653HMGC77FA, seg_pos={0 0}, upload=332730]: Uploading piece... Mar 18 20:04:15.818487545 (297h4m13.415258569s) [noise=true , address=79.127.219.46:4894 , piece_id=U33HRDMD5CQSBFGIEAQJTLQDNVYWUHIPDSTH4RVDGT4IB27CSBYA, seg_pos={0 0}, upload=332730]: Uploading piece... Mar 18 20:04:15.888499509 (297h4m13.485270531s) [upload=332730]: Next returned nil segment Mar 18 20:04:15.888523854 (297h4m13.485294882s) [upload=332730]: Waiting for error group managing segments... Mar 18 20:04:15.892357218 (297h4m13.48912823s) [noise=true , address=79.127.219.46:4894 , piece_id=U33HRDMD5CQSBFGIEAQJTLQDNVYWUHIPDSTH4RVDGT4IB27CSBYA, seg_pos={0 0}, upload=332730]: Done uploading piece. err: <nil> Mar 18 20:04:15.892802795 (297h4m13.489573828s) [noise=true , address=45.91.116.12:4260 , piece_id=JCZ5KUJ7QBW5KIRZ7UHOCOTLTVQP4TLG4IYHAGQDHWMGDFO6D5GQ, seg_pos={0 0}, upload=332730]: Done uploading piece. err: <nil> Mar 18 20:04:15.893652700 (297h4m13.49042373s) [noise=true , address=45.91.116.11:4121 , piece_id=EGOJ7I4GYLTLYZTLVOXK2YFUJGPB4L2T2EJ3XFNQQYNSGVXLGXEA, seg_pos={0 0}, upload=332730]: Done uploading piece. err: <nil> Mar 18 20:04:15.893771054 (297h4m13.490542099s) [noise=true , address=45.91.116.13:4188 , piece_id=3DGAL7BTW2LROY5GX6WR2FTZEWGJNJ7UOVXOZBHWMLXTVBUTEJZA, seg_pos={0 0}, upload=332730]: Done uploading piece. err: <nil> Mar 18 20:04:15.894278015 (297h4m13.491049048s) [noise=true , address=45.91.116.14:4191 , piece_id=ZF7KWZPHRJDVABY2OMX5H5G6CATYHPD2T32AKEWQ5653HMGC77FA, seg_pos={0 0}, upload=332730]: Done uploading piece. err: <nil> Mar 18 20:04:15.898058490 (297h4m13.494829504s) [noise=true , address=77.90.31.18:4805 , piece_id=5L7MKDVCJWCG4KQ52CKJV5SQS2JZI5BGQRW4QXBJQV6QVXEB6BZA, seg_pos={0 0}, upload=332730]: Done uploading piece. err: <nil> Mar 18 20:04:15.908841216 (297h4m13.50561224s) [noise=true , address=77.90.31.17:4769 , piece_id=L3UU2YI7SO5GJR2YXXLC2HW356VLG2GB4SOWBTSY27AO7EAMD62Q, seg_pos={0 0}, upload=332730]: Done uploading piece. err: <nil> Mar 18 20:04:15.911211299 (297h4m13.507982313s) [noise=true , address=45.91.116.5:4297 , piece_id=2XGIQ7VHVOS6LZ3SSIK5XL2I4ACDPJ57BXKQCFPLZO6HI3ADTHLA, seg_pos={0 0}, upload=332730]: Done uploading piece. err: <nil> Mar 18 20:04:15.914996180 (297h4m13.511767207s) [noise=true , address=77.90.31.11:4555 , piece_id=23M5BFJQKODZYDFM7SEYR5R4GBVDKLRUUBFM6CD7WHOZMLHTVPNQ, seg_pos={0 0}, upload=332730]: Done uploading piece. err: <nil> Mar 18 20:04:15.960374088 (297h4m13.557145116s) [noise=true , address=45.91.116.9:4265 , piece_id=ODQG6YJNSJDMEGZR3DIDE2LV2VG4AZXR74OUROP4YURJ7CLGAJDA, seg_pos={0 0}, upload=332730]: Done uploading piece. err: <nil> Mar 18 20:04:42.733811317 (297h4m40.330582349s) [noise=true , address=145.79.227.2:4119 , piece_id=3IGMZRN7TOTUQR6BCL6Y4UQCR4KIEFLHHGW37QXQRLZ2JTWOKAGQ, seg_pos={0 0}, upload=332730]: Done uploading piece. err: ecclient: upload failed (node:121xNAnkm9HZXUszp26UmWTsuhp3igyeDVea6gSCdtrQUQDVHpW, address:145.79.227.2:4119): protocol: rpc: dial tcp 145.79.227.2:4119: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:04:42.733841283 (297h4m40.330612309s) [noise=true , address=145.79.227.4:4189 , piece_id=FFEAVSH2EK7YT3ET5C6Y56XAMRITYHQRFYLOC3WTGWI74OMOUT4Q, seg_pos={0 0}, upload=332730]: Done uploading piece. err: ecclient: upload failed (node:1jH3757i2F3q18XSpoXznsysZPaXmEnqynpLPDupgRfxJrTgKv, address:145.79.227.4:4189): protocol: rpc: dial tcp 145.79.227.4:4189: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:04:42.734141754 (297h4m40.330912783s) [noise=true , address=145.79.227.9:4369 , piece_id=GC6OCZ7CXOKBSQYFRGM7C3BX5WMIQ5MDIALFAPN2GNKYF4YIN2MQ, seg_pos={0 0}, upload=332730]: Done uploading piece. err: ecclient: upload failed (node:1nwygPHyRJEZkjCoSZJTVhXDKLHc7AonVMzKMht6ppwZwhcGCT, address:145.79.227.9:4369): protocol: rpc: dial tcp 145.79.227.9:4369: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:04:42.734487144 (297h4m40.331258172s) [noise=true , address=145.79.227.35:4482 , piece_id=X3UQEIIBUQ5263YMPQLGIRW6EHOOIB476XNJDCCSC4HBYEGKUYTQ, seg_pos={0 0}, upload=332730]: Done uploading piece. err: ecclient: upload failed (node:12mNCKAdMzWu87fmyrcwDKZvscMcu2CogjXicwAi5wLSFJa7oVV, address:145.79.227.35:4482): protocol: rpc: dial tcp 145.79.227.35:4482: i/o timeout; conn closed: op; piecestore: piecestore close: context deadline exceeded Mar 18 20:04:42.734614154 (297h4m40.331385186s) [noise=true , address=145.79.227.51:4412 , piece_id=2RAPHKERYTBMLWB4IUWAT7NUSWR7EGOYEEOARZIQZIBQEYRJSYGA, seg_pos={0 0}, upload=332730]: Done uploading piece. err: ecclient: upload failed (node:12vBvEfCYZxsv17cVsQTbCQ4YjZfiD4RXjWyz7Ka9ZZrtvzC7SS, address:145.79.227.51:4412): protocol: rpc: dial tcp 145.79.227.51:4412: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:04:42.734856648 (297h4m40.33162768s) [noise=true , address=145.79.227.43:4158 , piece_id=HQLX5SJEGF4IQ6GJIWZUOYFP7TL7IIB7MRU4IDYOA2S2JFS22VFA, seg_pos={0 0}, upload=332730]: Done uploading piece. err: ecclient: upload failed (node:1vJZrSvPvwebw9V2tTbvkPEmLeLjryBUnq9TL6NwdPsHU9xcq2, address:145.79.227.43:4158): protocol: rpc: dial tcp 145.79.227.43:4158: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:04:42.734890164 (297h4m40.33166118s) [seg_pos={0 0}, upload=332730]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Mar 18 20:04:42.760664021 (297h4m40.357435035s) [upload=332730]: Done uploading segments... Mar 18 20:04:42.760710453 (297h4m40.35748148s) [upload=332730]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration: 30.028552024s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 47 out of 60 is failed (threshold=6) [.........XXXXXXXXX.XXX...XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 5.000743146s trace: https://jaeger.dev.storj.io/trace/2572dad753fd3389 FAIL (2026-03-18 20:04:25 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.000767024s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 8 out of 60 is failed (threshold=3) [.......................XXXXXX.XX............................] duration: 15.262016389s trace: https://jaeger.dev.storj.io/trace/4b4d10a8051ad01b FAIL (2026-03-18 20:04:17 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 15.262076556s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 7 out of 60 is failed (threshold=3) [.........................XXX..........X...X.........X....X..] duration: 16.133494784s trace: https://jaeger.dev.storj.io/trace/698fb9649d932a68 FAIL (2026-03-18 20:04:14 UTC) operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 20:02:41.019616017 (297h2m38.616387042s) [upload=332700]: Uploading segments... Mar 18 20:02:41.019826191 (297h2m38.616597205s) [upload=332700]: Got next segment. Inline: false Mar 18 20:02:41.019879916 (297h2m38.61665094s) [upload=332700]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 20:02:41.019885584 (297h2m38.616656611s) [upload=332700]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 18 20:02:43.834160391 (297h2m41.430931414s) [seg_pos={0 0}, upload=332700]: Begin upload segment... Mar 18 20:02:43.834230381 (297h2m41.431001396s) [seg_pos={0 0}, upload=332700]: Done begin upload segment. Mar 18 20:02:43.834262428 (297h2m41.431033451s) [noise=true , address=145.79.227.52:4451 , piece_id=LR6KVRYMDO2HAESG5BFI5RYPNGVARP6O5HEY2UP77YSTVRKBNUXQ, seg_pos={0 0}, upload=332700]: Uploading piece... Mar 18 20:02:43.834329472 (297h2m41.431100497s) [noise=true , address=45.91.116.13:4164 , piece_id=MNQLVANO3LFKBCJMADD5RULARMRO4VUQY7LT63TU2CCNO765GW7Q, seg_pos={0 0}, upload=332700]: Uploading piece... Mar 18 20:02:43.834938832 (297h2m41.431709858s) [noise=true , address=77.90.31.15:4716 , piece_id=IDQP2MIPCKVS2FGVVGKAIDYTNUIPE56EER5IYZUSOTRD5LZBSXHA, seg_pos={0 0}, upload=332700]: Uploading piece... Mar 18 20:02:43.835898822 (297h2m41.432669851s) [noise=true , address=79.127.205.237:4227 , piece_id=NO25WIYZTEPBOMGETTLSDOJSPKARMKNLVUWDKNXUKHYGDGF2XC5Q, seg_pos={0 0}, upload=332700]: Uploading piece... Mar 18 20:02:43.836893726 (297h2m41.433664759s) [noise=true , address=193.149.164.68:4064 , piece_id=HAA3NDB24MJLAI2HK2UDZQZRLDUXEF76ZJF2VBE6FLKRWSB27Y3A, seg_pos={0 0}, upload=332700]: Uploading piece... Mar 18 20:02:43.837257949 (297h2m41.434028981s) [noise=true , address=193.149.164.66:4974 , piece_id=XWMBTFEWXVVFIR6IYNQYU35QKGVSQJ7YBVSWZ57QSUZ2VSDCJPRA, seg_pos={0 0}, upload=332700]: Uploading piece... Mar 18 20:02:43.838266356 (297h2m41.435037391s) [noise=true , address=145.79.227.34:4924 , piece_id=BNUESPWRZQBX23K7UGNSMF362AWHILNZ43UITBHVGFXIJ4T6V5HQ, seg_pos={0 0}, upload=332700]: Uploading piece... Mar 18 20:02:43.838688582 (297h2m41.435459605s) [noise=true , address=79.127.205.233:4171 , piece_id=YO42WSHYRB2JJX4RVX653N7U6PYTGIW3KQTNSF4445OTN3UW2TIA, seg_pos={0 0}, upload=332700]: Uploading piece... Mar 18 20:02:43.839263235 (297h2m41.436034258s) [noise=true , address=193.149.164.70:4123 , piece_id=VROUAQKRTKZT42ZDDSCIMKFMRODPKFQ5U5DWDJLVI462BVA5YWXA, seg_pos={0 0}, upload=332700]: Uploading piece... Mar 18 20:02:43.839835928 (297h2m41.436606963s) [noise=true , address=45.91.116.4:4948 , piece_id=ULVH4YAJIYY5RWWAGH5RTSZ3BMAT6WCYYAJPSMSP25LCLZQIPG2A, seg_pos={0 0}, upload=332700]: Uploading piece... Mar 18 20:02:43.840390028 (297h2m41.437161055s) [noise=true , address=45.91.116.5:4319 , piece_id=BY6BRNZNKKQKZNAVJTSWUYASX57P5X7OCKQ433T2ARRBXUFNBRCA, seg_pos={0 0}, upload=332700]: Uploading piece... Mar 18 20:02:43.840790648 (297h2m41.437561682s) [noise=true , address=45.91.116.12:4259 , piece_id=4LBHNF3CIQG5LYEY7RYJGNK5Q62X2TSOAXOXSNHO727OXL5OMA4Q, seg_pos={0 0}, upload=332700]: Uploading piece... Mar 18 20:02:43.841319562 (297h2m41.438090584s) [noise=true , address=145.79.227.29:4966 , piece_id=ZTSKMRBESX4OIZMMIPXRW752W5KQYX57RPDQ22XXPURQTSCAI2YA, seg_pos={0 0}, upload=332700]: Uploading piece... Mar 18 20:02:43.841816046 (297h2m41.438587078s) [noise=true , address=193.149.164.71:4287 , piece_id=FYNZ3QGUB7ZFMUJAMYPMUHXJDS66CJ2VLXPV7256DFCABPJMEOYQ, seg_pos={0 0}, upload=332700]: Uploading piece... Mar 18 20:02:43.842288870 (297h2m41.439059903s) [noise=true , address=79.127.219.41:4941 , piece_id=CQXOXMCJHWJ7FSIXDVYUJZMBFW5FUUOJNKJJOVLZRSK5D6XZXX5Q, seg_pos={0 0}, upload=332700]: Uploading piece... Mar 18 20:02:43.842956832 (297h2m41.439727861s) [noise=true , address=145.79.227.49:4369 , piece_id=CBYHFRBNKSPUNJHAE6S72YOFACWLZKGZIFV6SXWDYMVSA4FBPKQQ, seg_pos={0 0}, upload=332700]: Uploading piece... Mar 18 20:02:43.844386533 (297h2m41.441157559s) [noise=true , address=77.90.31.10:4533 , piece_id=VB5BKAWDUZFVII4ZY3JRVK67VUULPSAHTJVZMXXVATP3SXAR2XHQ, seg_pos={0 0}, upload=332700]: Uploading piece... Mar 18 20:02:43.844868553 (297h2m41.441639576s) [noise=true , address=145.79.227.37:4582 , piece_id=DLOFMRZRG3BPFN7JF73OM5HUV56FLEH2IDDC47UQWWWOTRVRR4HQ, seg_pos={0 0}, upload=332700]: Uploading piece... Mar 18 20:02:43.903559633 (297h2m41.500330666s) [upload=332700]: Next returned nil segment Mar 18 20:02:43.903578892 (297h2m41.500349925s) [upload=332700]: Waiting for error group managing segments... Mar 18 20:02:43.908218461 (297h2m41.504989474s) [noise=true , address=45.91.116.4:4948 , piece_id=ULVH4YAJIYY5RWWAGH5RTSZ3BMAT6WCYYAJPSMSP25LCLZQIPG2A, seg_pos={0 0}, upload=332700]: Done uploading piece. err: <nil> Mar 18 20:02:43.917679744 (297h2m41.514450753s) [noise=true , address=45.91.116.12:4259 , piece_id=4LBHNF3CIQG5LYEY7RYJGNK5Q62X2TSOAXOXSNHO727OXL5OMA4Q, seg_pos={0 0}, upload=332700]: Done uploading piece. err: <nil> Mar 18 20:02:43.924357408 (297h2m41.521128422s) [noise=true , address=77.90.31.10:4533 , piece_id=VB5BKAWDUZFVII4ZY3JRVK67VUULPSAHTJVZMXXVATP3SXAR2XHQ, seg_pos={0 0}, upload=332700]: Done uploading piece. err: <nil> Mar 18 20:02:43.926248407 (297h2m41.52301943s) [noise=true , address=77.90.31.15:4716 , piece_id=IDQP2MIPCKVS2FGVVGKAIDYTNUIPE56EER5IYZUSOTRD5LZBSXHA, seg_pos={0 0}, upload=332700]: Done uploading piece. err: <nil> Mar 18 20:02:43.927212300 (297h2m41.523983323s) [noise=true , address=45.91.116.5:4319 , piece_id=BY6BRNZNKKQKZNAVJTSWUYASX57P5X7OCKQ433T2ARRBXUFNBRCA, seg_pos={0 0}, upload=332700]: Done uploading piece. err: <nil> Mar 18 20:02:43.927791610 (297h2m41.524562634s) [noise=true , address=45.91.116.13:4164 , piece_id=MNQLVANO3LFKBCJMADD5RULARMRO4VUQY7LT63TU2CCNO765GW7Q, seg_pos={0 0}, upload=332700]: Done uploading piece. err: <nil> Mar 18 20:02:44.028504592 (297h2m41.625275611s) [noise=true , address=79.127.219.41:4941 , piece_id=CQXOXMCJHWJ7FSIXDVYUJZMBFW5FUUOJNKJJOVLZRSK5D6XZXX5Q, seg_pos={0 0}, upload=332700]: Done uploading piece. err: <nil> Mar 18 20:02:44.392266647 (297h2m41.989037671s) [noise=true , address=79.127.205.233:4171 , piece_id=YO42WSHYRB2JJX4RVX653N7U6PYTGIW3KQTNSF4445OTN3UW2TIA, seg_pos={0 0}, upload=332700]: Done uploading piece. err: <nil> Mar 18 20:02:44.429403642 (297h2m42.026174669s) [noise=true , address=193.149.164.66:4974 , piece_id=XWMBTFEWXVVFIR6IYNQYU35QKGVSQJ7YBVSWZ57QSUZ2VSDCJPRA, seg_pos={0 0}, upload=332700]: Done uploading piece. err: <nil> Mar 18 20:02:44.433374828 (297h2m42.030145853s) [noise=true , address=193.149.164.68:4064 , piece_id=HAA3NDB24MJLAI2HK2UDZQZRLDUXEF76ZJF2VBE6FLKRWSB27Y3A, seg_pos={0 0}, upload=332700]: Done uploading piece. err: <nil> Mar 18 20:02:44.444389014 (297h2m42.041160042s) [noise=true , address=193.149.164.71:4287 , piece_id=FYNZ3QGUB7ZFMUJAMYPMUHXJDS66CJ2VLXPV7256DFCABPJMEOYQ, seg_pos={0 0}, upload=332700]: Done uploading piece. err: <nil> Mar 18 20:02:44.578333498 (297h2m42.175104529s) [noise=true , address=79.127.205.237:4227 , piece_id=NO25WIYZTEPBOMGETTLSDOJSPKARMKNLVUWDKNXUKHYGDGF2XC5Q, seg_pos={0 0}, upload=332700]: Done uploading piece. err: <nil> Mar 18 20:02:44.665665140 (297h2m42.262436168s) [noise=true , address=193.149.164.70:4123 , piece_id=VROUAQKRTKZT42ZDDSCIMKFMRODPKFQ5U5DWDJLVI462BVA5YWXA, seg_pos={0 0}, upload=332700]: Done uploading piece. err: <nil> Mar 18 20:03:40.987108256 (297h3m38.58387928s) [noise=true , address=145.79.227.49:4369 , piece_id=CBYHFRBNKSPUNJHAE6S72YOFACWLZKGZIFV6SXWDYMVSA4FBPKQQ, seg_pos={0 0}, upload=332700]: Done uploading piece. err: ecclient: upload failed (node:12nLksGXGvDwuTWfnmkceuwZNDmrF17iNepHPdYkkvZG2QQUwG7, address:145.79.227.49:4369): protocol: rpc: dial tcp 145.79.227.49:4369: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:03:40.987479807 (297h3m38.58425084s) [noise=true , address=145.79.227.29:4966 , piece_id=ZTSKMRBESX4OIZMMIPXRW752W5KQYX57RPDQ22XXPURQTSCAI2YA, seg_pos={0 0}, upload=332700]: Done uploading piece. err: ecclient: upload failed (node:1DLnsYVdudE4mY9dNAQKhBRppUEAHvxbf84KX69451pwxSxzZK, address:145.79.227.29:4966): protocol: rpc: dial tcp 145.79.227.29:4966: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:03:40.987517864 (297h3m38.584288896s) [noise=true , address=145.79.227.34:4924 , piece_id=BNUESPWRZQBX23K7UGNSMF362AWHILNZ43UITBHVGFXIJ4T6V5HQ, seg_pos={0 0}, upload=332700]: Done uploading piece. err: ecclient: upload failed (node:19TbrQjuN1EhBNVtha9EcKPSCAWyttfmHpec6QeF1S5zW6DRZs, address:145.79.227.34:4924): protocol: rpc: dial tcp 145.79.227.34:4924: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:03:40.987933296 (297h3m38.584704329s) [noise=true , address=145.79.227.37:4582 , piece_id=DLOFMRZRG3BPFN7JF73OM5HUV56FLEH2IDDC47UQWWWOTRVRR4HQ, seg_pos={0 0}, upload=332700]: Done uploading piece. err: ecclient: upload failed (node:129o91j97t4V5AjaouCCy2wDuHgwF94iQiJh2h9ux1aFo7AVCSn, address:145.79.227.37:4582): protocol: rpc: dial tcp 145.79.227.37:4582: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:03:40.988167350 (297h3m38.584938376s) [noise=true , address=145.79.227.52:4451 , piece_id=LR6KVRYMDO2HAESG5BFI5RYPNGVARP6O5HEY2UP77YSTVRKBNUXQ, seg_pos={0 0}, upload=332700]: Done uploading piece. err: ecclient: upload failed (node:12bC4P3xBiPFkxLvavGFVbWKnbprCStL6UJgfKSA2Wd2fX7Mbfw, address:145.79.227.52:4451): protocol: rpc: dial tcp 145.79.227.52:4451: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:03:40.988197047 (297h3m38.58496806s) [seg_pos={0 0}, upload=332700]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 18 20:03:41.007727578 (297h3m38.604498608s) [upload=332700]: Done uploading segments... Mar 18 20:03:41.007779207 (297h3m38.60455025s) [upload=332700]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 16.133527662s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 38 out of 60 is failed (threshold=3) [.....................XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.X] duration: 30.030050123s trace: https://jaeger.dev.storj.io/trace/57ac6ef9f68b1954 FAIL (2026-03-18 20:04:12 UTC) operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 7) ======LOGS====== d=332722]: Uploading segments... Mar 18 20:03:42.682054243 (297h3m40.278825266s) [upload=332722]: Got next segment. Inline: false Mar 18 20:03:42.682092092 (297h3m40.278863115s) [upload=332722]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 20:03:42.682186798 (297h3m40.27895783s) [upload=332722]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 18 20:03:45.793181227 (297h3m43.38995225s) [seg_pos={0 0}, upload=332722]: Begin upload segment... Mar 18 20:03:45.793243492 (297h3m43.390014515s) [seg_pos={0 0}, upload=332722]: Done begin upload segment. Mar 18 20:03:45.793285202 (297h3m43.390056226s) [noise=true , address=145.79.227.19:4077 , piece_id=HBGD3N2Y5Q6PFXWYUIARZNRRHUI7FDF5UNZKH5W6DLM4SG6D2PKA, seg_pos={0 0}, upload=332722]: Uploading piece... Mar 18 20:03:45.793336989 (297h3m43.390108012s) [noise=true , address=45.91.116.7:4012 , piece_id=BP236K3GXCY7EQ7TBQ2LDFXTXQLHHRM36GPUTD7XT7RYWYC2AD7Q, seg_pos={0 0}, upload=332722]: Uploading piece... Mar 18 20:03:45.793852919 (297h3m43.390623946s) [noise=true , address=145.79.227.35:4481 , piece_id=GLA22Z5IAQIBPQ5D2LKB6HMKOZCFRIECSTL3JLCYUD7O6RCAAGPA, seg_pos={0 0}, upload=332722]: Uploading piece... Mar 18 20:03:45.794017856 (297h3m43.390788891s) [noise=true , address=145.79.227.50:4374 , piece_id=O3I3JVAHA3ZTP5WHWUJE6IRI5TYO2E3HHPGY3IZHI3YU6SDLUANA, seg_pos={0 0}, upload=332722]: Uploading piece... Mar 18 20:03:45.794640345 (297h3m43.391411378s) [noise=true , address=145.79.227.25:4622 , piece_id=YFBS5SBLN4RGNHFRNGHVIS6HE7D3R6WM32LW4KJAPVPUHYX22TXA, seg_pos={0 0}, upload=332722]: Uploading piece... Mar 18 20:03:45.795075010 (297h3m43.391846062s) [noise=true , address=145.79.227.17:4975 , piece_id=CLJISVBS45YQFZFWJK2ZPRKUYRJMLUNBLYGQSOM6OYDZ7ZYN7JYQ, seg_pos={0 0}, upload=332722]: Uploading piece... Mar 18 20:03:45.795600020 (297h3m43.392371054s) [noise=true , address=45.91.116.4:4941 , piece_id=KLQQRLVCUITFEFXDHYDAOI7IOW4A2STN3ZG3MN42CBE7MA3ZPMUQ, seg_pos={0 0}, upload=332722]: Uploading piece... Mar 18 20:03:45.796098498 (297h3m43.392869529s) [noise=true , address=77.90.31.14:4663 , piece_id=4JEOFVDMLR6ZKHX64WGCNNAKL2ZVAWTRJHJASNK5BURBNVGNU7WA, seg_pos={0 0}, upload=332722]: Uploading piece... Mar 18 20:03:45.796854422 (297h3m43.393625452s) [noise=true , address=45.91.116.6:4989 , piece_id=ZGKKKAYRV6YS2PRE3B25LA4GP5NRXRMML3B3HUUXJ2OL6LM3KGQQ, seg_pos={0 0}, upload=332722]: Uploading piece... Mar 18 20:03:45.797409061 (297h3m43.394180096s) [noise=true , address=45.91.116.11:4117 , piece_id=ZO6QQSZJ476M7U4ECTWIOO6AHPAXGNVNTMLOSYK3NJYXX5PFC4SA, seg_pos={0 0}, upload=332722]: Uploading piece... Mar 18 20:03:45.797913649 (297h3m43.394684685s) [noise=true , address=145.79.227.45:4233 , piece_id=TVTJKKCNQMLHQCMTFMA44552PUJKHBJKTQIOGHGWLFKF6H4GVM4Q, seg_pos={0 0}, upload=332722]: Uploading piece... Mar 18 20:03:45.798496199 (297h3m43.395267231s) [noise=true , address=145.79.227.21:4473 , piece_id=G7HVAZ2OI5SLBVFGC4JZ7Q3PY33H45IBZMSFBKEJJTIH4QCZTHUA, seg_pos={0 0}, upload=332722]: Uploading piece... Mar 18 20:03:45.799088489 (297h3m43.395859526s) [noise=true , address=145.79.227.7:4297 , piece_id=ZKYGADJJ5MESHKKQR7W3OQYQWTL7ZGBNVS7K6CWIGWAY4DP5IGHA, seg_pos={0 0}, upload=332722]: Uploading piece... Mar 18 20:03:45.799687725 (297h3m43.396458752s) [noise=true , address=145.79.227.46:4604 , piece_id=JNLR4TZMVH2JYBXRQ3D5D5QNFJEWS5GTATXV3KO6IVWUAZKKFGKA, seg_pos={0 0}, upload=332722]: Uploading piece... Mar 18 20:03:45.801186418 (297h3m43.397957461s) [noise=true , address=77.90.31.21:4878 , piece_id=BFCXTZVWLIDZ3P2376LQY7XIYF6GLIYJTXOSVJNYSU4TWPN46LCA, seg_pos={0 0}, upload=332722]: Uploading piece... Mar 18 20:03:45.802871699 (297h3m43.399642731s) [noise=true , address=79.127.219.45:4854 , piece_id=BT5GJEZBDRNVOOAPAYBY4RC2BRB6BCXSOQMWIKAHZ756QTITOPDQ, seg_pos={0 0}, upload=332722]: Uploading piece... Mar 18 20:03:45.854622430 (297h3m43.451393464s) [upload=332722]: Next returned nil segment Mar 18 20:03:45.854639563 (297h3m43.451410605s) [upload=332722]: Waiting for error group managing segments... Mar 18 20:03:45.858552233 (297h3m43.455323257s) [noise=true , address=45.91.116.6:4989 , piece_id=ZGKKKAYRV6YS2PRE3B25LA4GP5NRXRMML3B3HUUXJ2OL6LM3KGQQ, seg_pos={0 0}, upload=332722]: Done uploading piece. err: <nil> Mar 18 20:03:45.858874779 (297h3m43.455645825s) [noise=true , address=45.91.116.11:4117 , piece_id=ZO6QQSZJ476M7U4ECTWIOO6AHPAXGNVNTMLOSYK3NJYXX5PFC4SA, seg_pos={0 0}, upload=332722]: Done uploading piece. err: <nil> Mar 18 20:03:45.859946528 (297h3m43.456717558s) [noise=true , address=45.91.116.4:4941 , piece_id=KLQQRLVCUITFEFXDHYDAOI7IOW4A2STN3ZG3MN42CBE7MA3ZPMUQ, seg_pos={0 0}, upload=332722]: Done uploading piece. err: <nil> Mar 18 20:03:45.860597071 (297h3m43.457368098s) [noise=true , address=77.90.31.14:4663 , piece_id=4JEOFVDMLR6ZKHX64WGCNNAKL2ZVAWTRJHJASNK5BURBNVGNU7WA, seg_pos={0 0}, upload=332722]: Done uploading piece. err: <nil> Mar 18 20:03:45.861234361 (297h3m43.458005388s) [noise=true , address=45.91.116.7:4012 , piece_id=BP236K3GXCY7EQ7TBQ2LDFXTXQLHHRM36GPUTD7XT7RYWYC2AD7Q, seg_pos={0 0}, upload=332722]: Done uploading piece. err: <nil> Mar 18 20:03:45.882215044 (297h3m43.478986074s) [noise=true , address=77.90.31.21:4878 , piece_id=BFCXTZVWLIDZ3P2376LQY7XIYF6GLIYJTXOSVJNYSU4TWPN46LCA, seg_pos={0 0}, upload=332722]: Done uploading piece. err: <nil> Mar 18 20:03:45.886253662 (297h3m43.483024686s) [noise=true , address=79.127.219.45:4854 , piece_id=BT5GJEZBDRNVOOAPAYBY4RC2BRB6BCXSOQMWIKAHZ756QTITOPDQ, seg_pos={0 0}, upload=332722]: Done uploading piece. err: <nil> Mar 18 20:04:12.643217190 (297h4m10.239988219s) [noise=true , address=145.79.227.25:4622 , piece_id=YFBS5SBLN4RGNHFRNGHVIS6HE7D3R6WM32LW4KJAPVPUHYX22TXA, seg_pos={0 0}, upload=332722]: Done uploading piece. err: ecclient: upload failed (node:1xEK5XL3sYcip7PsQcZBQYihgnEZykdMmsKkCr2TUmbTFYncfs, address:145.79.227.25:4622): protocol: rpc: dial tcp 145.79.227.25:4622: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:04:12.643284920 (297h4m10.24005595s) [noise=true , address=145.79.227.17:4975 , piece_id=CLJISVBS45YQFZFWJK2ZPRKUYRJMLUNBLYGQSOM6OYDZ7ZYN7JYQ, seg_pos={0 0}, upload=332722]: Done uploading piece. err: ecclient: upload failed (node:12WiUZeNCJk4rzjtyWuKJwGcZATaYLU2QxN27MZE7i45zVpu9dM, address:145.79.227.17:4975): protocol: rpc: dial tcp 145.79.227.17:4975: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:04:12.643312931 (297h4m10.240083963s) [noise=true , address=145.79.227.21:4473 , piece_id=G7HVAZ2OI5SLBVFGC4JZ7Q3PY33H45IBZMSFBKEJJTIH4QCZTHUA, seg_pos={0 0}, upload=332722]: Done uploading piece. err: ecclient: upload failed (node:12DDe4jT3YeUH8qMQgapcvzMnFs8UFB8fTmubho4ZQf5tYVoxuk, address:145.79.227.21:4473): protocol: rpc: dial tcp 145.79.227.21:4473: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:04:12.643457660 (297h4m10.240228685s) [noise=true , address=145.79.227.46:4604 , piece_id=JNLR4TZMVH2JYBXRQ3D5D5QNFJEWS5GTATXV3KO6IVWUAZKKFGKA, seg_pos={0 0}, upload=332722]: Done uploading piece. err: ecclient: upload failed (node:129d1v3mjqB3FHC2VmbY9Sih6C5fRDrV3kGLiv9NfkDc7JovXjy, address:145.79.227.46:4604): protocol: rpc: dial tcp 145.79.227.46:4604: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:04:12.643515778 (297h4m10.24028681s) [noise=true , address=145.79.227.35:4481 , piece_id=GLA22Z5IAQIBPQ5D2LKB6HMKOZCFRIECSTL3JLCYUD7O6RCAAGPA, seg_pos={0 0}, upload=332722]: Done uploading piece. err: ecclient: upload failed (node:12bRK8WWmAnWUwzCfHu8JtZgrruYSJCvhnsYTKcaLccJcv5RQjp, address:145.79.227.35:4481): protocol: rpc: dial tcp 145.79.227.35:4481: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:04:12.643560804 (297h4m10.240331833s) [noise=true , address=145.79.227.45:4233 , piece_id=TVTJKKCNQMLHQCMTFMA44552PUJKHBJKTQIOGHGWLFKF6H4GVM4Q, seg_pos={0 0}, upload=332722]: Done uploading piece. err: ecclient: upload failed (node:1DmQeoQWTnFhQeJFseJoNYzFyo7vZjYKwDtQVJ6pomksWExsTb, address:145.79.227.45:4233): protocol: rpc: dial tcp 145.79.227.45:4233: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:04:12.643717091 (297h4m10.240488123s) [noise=true , address=145.79.227.7:4297 , piece_id=ZKYGADJJ5MESHKKQR7W3OQYQWTL7ZGBNVS7K6CWIGWAY4DP5IGHA, seg_pos={0 0}, upload=332722]: Done uploading piece. err: ecclient: upload failed (node:1g2pJxnoFW2TAzcAzNhjzDRJ3yYDK2mibG8tdX2Qy2YabNTq86, address:145.79.227.7:4297): protocol: rpc: dial tcp 145.79.227.7:4297: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:04:12.643873094 (297h4m10.240644126s) [noise=true , address=145.79.227.19:4077 , piece_id=HBGD3N2Y5Q6PFXWYUIARZNRRHUI7FDF5UNZKH5W6DLM4SG6D2PKA, seg_pos={0 0}, upload=332722]: Done uploading piece. err: ecclient: upload failed (node:12kDJuQP8iLUjB1rXpcNX5TtHzsqa5HvYcUD9Ef64HYTnTXcwLk, address:145.79.227.19:4077): protocol: rpc: dial tcp 145.79.227.19:4077: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:04:12.644129833 (297h4m10.240900847s) [noise=true , address=145.79.227.50:4374 , piece_id=O3I3JVAHA3ZTP5WHWUJE6IRI5TYO2E3HHPGY3IZHI3YU6SDLUANA, seg_pos={0 0}, upload=332722]: Done uploading piece. err: ecclient: upload failed (node:12QRixhWaUwXPdjcWKYHbiknME5JVomWYABY6egDD6VKwaTPB8K, address:145.79.227.50:4374): protocol: rpc: dial tcp 145.79.227.50:4374: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:04:12.644160114 (297h4m10.24093113s) [seg_pos={0 0}, upload=332722]: Done waiting for segment. successful: 7 optimal: 13 errs: <nil> Mar 18 20:04:12.670772828 (297h4m10.267543862s) [upload=332722]: Done uploading segments... Mar 18 20:04:12.670860004 (297h4m10.267631028s) [upload=332722]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 7) Duration: 30.030140606s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 7 out of 60 is failed (threshold=3) [..........................XXX..........X...X.........X....X.] duration: 15.826202395s trace: https://jaeger.dev.storj.io/trace/698fb9649d932a68 FAIL (2026-03-18 20:03:56 UTC) operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 20:02:41.019616017 (297h2m38.616387042s) [upload=332700]: Uploading segments... Mar 18 20:02:41.019826191 (297h2m38.616597205s) [upload=332700]: Got next segment. Inline: false Mar 18 20:02:41.019879916 (297h2m38.61665094s) [upload=332700]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 20:02:41.019885584 (297h2m38.616656611s) [upload=332700]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 18 20:02:43.834160391 (297h2m41.430931414s) [seg_pos={0 0}, upload=332700]: Begin upload segment... Mar 18 20:02:43.834230381 (297h2m41.431001396s) [seg_pos={0 0}, upload=332700]: Done begin upload segment. Mar 18 20:02:43.834262428 (297h2m41.431033451s) [noise=true , address=145.79.227.52:4451 , piece_id=LR6KVRYMDO2HAESG5BFI5RYPNGVARP6O5HEY2UP77YSTVRKBNUXQ, seg_pos={0 0}, upload=332700]: Uploading piece... Mar 18 20:02:43.834329472 (297h2m41.431100497s) [noise=true , address=45.91.116.13:4164 , piece_id=MNQLVANO3LFKBCJMADD5RULARMRO4VUQY7LT63TU2CCNO765GW7Q, seg_pos={0 0}, upload=332700]: Uploading piece... Mar 18 20:02:43.834938832 (297h2m41.431709858s) [noise=true , address=77.90.31.15:4716 , piece_id=IDQP2MIPCKVS2FGVVGKAIDYTNUIPE56EER5IYZUSOTRD5LZBSXHA, seg_pos={0 0}, upload=332700]: Uploading piece... Mar 18 20:02:43.835898822 (297h2m41.432669851s) [noise=true , address=79.127.205.237:4227 , piece_id=NO25WIYZTEPBOMGETTLSDOJSPKARMKNLVUWDKNXUKHYGDGF2XC5Q, seg_pos={0 0}, upload=332700]: Uploading piece... Mar 18 20:02:43.836893726 (297h2m41.433664759s) [noise=true , address=193.149.164.68:4064 , piece_id=HAA3NDB24MJLAI2HK2UDZQZRLDUXEF76ZJF2VBE6FLKRWSB27Y3A, seg_pos={0 0}, upload=332700]: Uploading piece... Mar 18 20:02:43.837257949 (297h2m41.434028981s) [noise=true , address=193.149.164.66:4974 , piece_id=XWMBTFEWXVVFIR6IYNQYU35QKGVSQJ7YBVSWZ57QSUZ2VSDCJPRA, seg_pos={0 0}, upload=332700]: Uploading piece... Mar 18 20:02:43.838266356 (297h2m41.435037391s) [noise=true , address=145.79.227.34:4924 , piece_id=BNUESPWRZQBX23K7UGNSMF362AWHILNZ43UITBHVGFXIJ4T6V5HQ, seg_pos={0 0}, upload=332700]: Uploading piece... Mar 18 20:02:43.838688582 (297h2m41.435459605s) [noise=true , address=79.127.205.233:4171 , piece_id=YO42WSHYRB2JJX4RVX653N7U6PYTGIW3KQTNSF4445OTN3UW2TIA, seg_pos={0 0}, upload=332700]: Uploading piece... Mar 18 20:02:43.839263235 (297h2m41.436034258s) [noise=true , address=193.149.164.70:4123 , piece_id=VROUAQKRTKZT42ZDDSCIMKFMRODPKFQ5U5DWDJLVI462BVA5YWXA, seg_pos={0 0}, upload=332700]: Uploading piece... Mar 18 20:02:43.839835928 (297h2m41.436606963s) [noise=true , address=45.91.116.4:4948 , piece_id=ULVH4YAJIYY5RWWAGH5RTSZ3BMAT6WCYYAJPSMSP25LCLZQIPG2A, seg_pos={0 0}, upload=332700]: Uploading piece... Mar 18 20:02:43.840390028 (297h2m41.437161055s) [noise=true , address=45.91.116.5:4319 , piece_id=BY6BRNZNKKQKZNAVJTSWUYASX57P5X7OCKQ433T2ARRBXUFNBRCA, seg_pos={0 0}, upload=332700]: Uploading piece... Mar 18 20:02:43.840790648 (297h2m41.437561682s) [noise=true , address=45.91.116.12:4259 , piece_id=4LBHNF3CIQG5LYEY7RYJGNK5Q62X2TSOAXOXSNHO727OXL5OMA4Q, seg_pos={0 0}, upload=332700]: Uploading piece... Mar 18 20:02:43.841319562 (297h2m41.438090584s) [noise=true , address=145.79.227.29:4966 , piece_id=ZTSKMRBESX4OIZMMIPXRW752W5KQYX57RPDQ22XXPURQTSCAI2YA, seg_pos={0 0}, upload=332700]: Uploading piece... Mar 18 20:02:43.841816046 (297h2m41.438587078s) [noise=true , address=193.149.164.71:4287 , piece_id=FYNZ3QGUB7ZFMUJAMYPMUHXJDS66CJ2VLXPV7256DFCABPJMEOYQ, seg_pos={0 0}, upload=332700]: Uploading piece... Mar 18 20:02:43.842288870 (297h2m41.439059903s) [noise=true , address=79.127.219.41:4941 , piece_id=CQXOXMCJHWJ7FSIXDVYUJZMBFW5FUUOJNKJJOVLZRSK5D6XZXX5Q, seg_pos={0 0}, upload=332700]: Uploading piece... Mar 18 20:02:43.842956832 (297h2m41.439727861s) [noise=true , address=145.79.227.49:4369 , piece_id=CBYHFRBNKSPUNJHAE6S72YOFACWLZKGZIFV6SXWDYMVSA4FBPKQQ, seg_pos={0 0}, upload=332700]: Uploading piece... Mar 18 20:02:43.844386533 (297h2m41.441157559s) [noise=true , address=77.90.31.10:4533 , piece_id=VB5BKAWDUZFVII4ZY3JRVK67VUULPSAHTJVZMXXVATP3SXAR2XHQ, seg_pos={0 0}, upload=332700]: Uploading piece... Mar 18 20:02:43.844868553 (297h2m41.441639576s) [noise=true , address=145.79.227.37:4582 , piece_id=DLOFMRZRG3BPFN7JF73OM5HUV56FLEH2IDDC47UQWWWOTRVRR4HQ, seg_pos={0 0}, upload=332700]: Uploading piece... Mar 18 20:02:43.903559633 (297h2m41.500330666s) [upload=332700]: Next returned nil segment Mar 18 20:02:43.903578892 (297h2m41.500349925s) [upload=332700]: Waiting for error group managing segments... Mar 18 20:02:43.908218461 (297h2m41.504989474s) [noise=true , address=45.91.116.4:4948 , piece_id=ULVH4YAJIYY5RWWAGH5RTSZ3BMAT6WCYYAJPSMSP25LCLZQIPG2A, seg_pos={0 0}, upload=332700]: Done uploading piece. err: <nil> Mar 18 20:02:43.917679744 (297h2m41.514450753s) [noise=true , address=45.91.116.12:4259 , piece_id=4LBHNF3CIQG5LYEY7RYJGNK5Q62X2TSOAXOXSNHO727OXL5OMA4Q, seg_pos={0 0}, upload=332700]: Done uploading piece. err: <nil> Mar 18 20:02:43.924357408 (297h2m41.521128422s) [noise=true , address=77.90.31.10:4533 , piece_id=VB5BKAWDUZFVII4ZY3JRVK67VUULPSAHTJVZMXXVATP3SXAR2XHQ, seg_pos={0 0}, upload=332700]: Done uploading piece. err: <nil> Mar 18 20:02:43.926248407 (297h2m41.52301943s) [noise=true , address=77.90.31.15:4716 , piece_id=IDQP2MIPCKVS2FGVVGKAIDYTNUIPE56EER5IYZUSOTRD5LZBSXHA, seg_pos={0 0}, upload=332700]: Done uploading piece. err: <nil> Mar 18 20:02:43.927212300 (297h2m41.523983323s) [noise=true , address=45.91.116.5:4319 , piece_id=BY6BRNZNKKQKZNAVJTSWUYASX57P5X7OCKQ433T2ARRBXUFNBRCA, seg_pos={0 0}, upload=332700]: Done uploading piece. err: <nil> Mar 18 20:02:43.927791610 (297h2m41.524562634s) [noise=true , address=45.91.116.13:4164 , piece_id=MNQLVANO3LFKBCJMADD5RULARMRO4VUQY7LT63TU2CCNO765GW7Q, seg_pos={0 0}, upload=332700]: Done uploading piece. err: <nil> Mar 18 20:02:44.028504592 (297h2m41.625275611s) [noise=true , address=79.127.219.41:4941 , piece_id=CQXOXMCJHWJ7FSIXDVYUJZMBFW5FUUOJNKJJOVLZRSK5D6XZXX5Q, seg_pos={0 0}, upload=332700]: Done uploading piece. err: <nil> Mar 18 20:02:44.392266647 (297h2m41.989037671s) [noise=true , address=79.127.205.233:4171 , piece_id=YO42WSHYRB2JJX4RVX653N7U6PYTGIW3KQTNSF4445OTN3UW2TIA, seg_pos={0 0}, upload=332700]: Done uploading piece. err: <nil> Mar 18 20:02:44.429403642 (297h2m42.026174669s) [noise=true , address=193.149.164.66:4974 , piece_id=XWMBTFEWXVVFIR6IYNQYU35QKGVSQJ7YBVSWZ57QSUZ2VSDCJPRA, seg_pos={0 0}, upload=332700]: Done uploading piece. err: <nil> Mar 18 20:02:44.433374828 (297h2m42.030145853s) [noise=true , address=193.149.164.68:4064 , piece_id=HAA3NDB24MJLAI2HK2UDZQZRLDUXEF76ZJF2VBE6FLKRWSB27Y3A, seg_pos={0 0}, upload=332700]: Done uploading piece. err: <nil> Mar 18 20:02:44.444389014 (297h2m42.041160042s) [noise=true , address=193.149.164.71:4287 , piece_id=FYNZ3QGUB7ZFMUJAMYPMUHXJDS66CJ2VLXPV7256DFCABPJMEOYQ, seg_pos={0 0}, upload=332700]: Done uploading piece. err: <nil> Mar 18 20:02:44.578333498 (297h2m42.175104529s) [noise=true , address=79.127.205.237:4227 , piece_id=NO25WIYZTEPBOMGETTLSDOJSPKARMKNLVUWDKNXUKHYGDGF2XC5Q, seg_pos={0 0}, upload=332700]: Done uploading piece. err: <nil> Mar 18 20:02:44.665665140 (297h2m42.262436168s) [noise=true , address=193.149.164.70:4123 , piece_id=VROUAQKRTKZT42ZDDSCIMKFMRODPKFQ5U5DWDJLVI462BVA5YWXA, seg_pos={0 0}, upload=332700]: Done uploading piece. err: <nil> Mar 18 20:03:40.987108256 (297h3m38.58387928s) [noise=true , address=145.79.227.49:4369 , piece_id=CBYHFRBNKSPUNJHAE6S72YOFACWLZKGZIFV6SXWDYMVSA4FBPKQQ, seg_pos={0 0}, upload=332700]: Done uploading piece. err: ecclient: upload failed (node:12nLksGXGvDwuTWfnmkceuwZNDmrF17iNepHPdYkkvZG2QQUwG7, address:145.79.227.49:4369): protocol: rpc: dial tcp 145.79.227.49:4369: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:03:40.987479807 (297h3m38.58425084s) [noise=true , address=145.79.227.29:4966 , piece_id=ZTSKMRBESX4OIZMMIPXRW752W5KQYX57RPDQ22XXPURQTSCAI2YA, seg_pos={0 0}, upload=332700]: Done uploading piece. err: ecclient: upload failed (node:1DLnsYVdudE4mY9dNAQKhBRppUEAHvxbf84KX69451pwxSxzZK, address:145.79.227.29:4966): protocol: rpc: dial tcp 145.79.227.29:4966: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:03:40.987517864 (297h3m38.584288896s) [noise=true , address=145.79.227.34:4924 , piece_id=BNUESPWRZQBX23K7UGNSMF362AWHILNZ43UITBHVGFXIJ4T6V5HQ, seg_pos={0 0}, upload=332700]: Done uploading piece. err: ecclient: upload failed (node:19TbrQjuN1EhBNVtha9EcKPSCAWyttfmHpec6QeF1S5zW6DRZs, address:145.79.227.34:4924): protocol: rpc: dial tcp 145.79.227.34:4924: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:03:40.987933296 (297h3m38.584704329s) [noise=true , address=145.79.227.37:4582 , piece_id=DLOFMRZRG3BPFN7JF73OM5HUV56FLEH2IDDC47UQWWWOTRVRR4HQ, seg_pos={0 0}, upload=332700]: Done uploading piece. err: ecclient: upload failed (node:129o91j97t4V5AjaouCCy2wDuHgwF94iQiJh2h9ux1aFo7AVCSn, address:145.79.227.37:4582): protocol: rpc: dial tcp 145.79.227.37:4582: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:03:40.988167350 (297h3m38.584938376s) [noise=true , address=145.79.227.52:4451 , piece_id=LR6KVRYMDO2HAESG5BFI5RYPNGVARP6O5HEY2UP77YSTVRKBNUXQ, seg_pos={0 0}, upload=332700]: Done uploading piece. err: ecclient: upload failed (node:12bC4P3xBiPFkxLvavGFVbWKnbprCStL6UJgfKSA2Wd2fX7Mbfw, address:145.79.227.52:4451): protocol: rpc: dial tcp 145.79.227.52:4451: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:03:40.988197047 (297h3m38.58496806s) [seg_pos={0 0}, upload=332700]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 18 20:03:41.007727578 (297h3m38.604498608s) [upload=332700]: Done uploading segments... Mar 18 20:03:41.007779207 (297h3m38.60455025s) [upload=332700]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 15.826222382s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 6 out of 60 is failed (threshold=3) [...........................X..........XX..X....X........X...] duration: 14.973210456s trace: https://jaeger.dev.storj.io/trace/589419efc52e5e2b FAIL (2026-03-18 20:03:49 UTC) operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== Mar 18 20:00:59.343084966 (297h0m56.939855989s) [upload=332666]: Starting upload Mar 18 20:00:59.343175518 (297h0m56.939946548s) [upload=332666]: Done starting upload Mar 18 20:00:59.343565585 (297h0m56.940336612s) [upload=332666]: Uploading segments... Mar 18 20:00:59.344256647 (297h0m56.941027676s) [upload=332666]: Got next segment. Inline: false Mar 18 20:00:59.344352215 (297h0m56.941123241s) [upload=332666]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 20:00:59.344358706 (297h0m56.941129732s) [upload=332666]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 18 20:01:02.340259299 (297h0m59.937030329s) [seg_pos={0 0}, upload=332666]: Begin upload segment... Mar 18 20:01:02.340368743 (297h0m59.937139767s) [seg_pos={0 0}, upload=332666]: Done begin upload segment. Mar 18 20:01:02.340479563 (297h0m59.93725059s) [noise=true , address=45.91.116.8:4061 , piece_id=6XFVKT4AAO7QN457OQMRKPI6R24TJTAN47UXLEVOEFQVOS6M6C3Q, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.340523837 (297h0m59.93729486s) [noise=true , address=79.127.205.233:4166 , piece_id=SFGK5YCWHFRVXF6QNHYP2CYJCZLSTQ76OYRVJPWWXQTF2KFSFJKA, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.341468438 (297h0m59.938239447s) [noise=true , address=45.91.116.4:4961 , piece_id=ZMUGQECUVPVHDNKN75XPBET5M2HFXXK4WMTP4V5KEK63JVZCX5EA, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.341520025 (297h0m59.938291058s) [noise=true , address=167.17.72.71:4815 , piece_id=CXGTYNNBILN3TT7I6Z7UI75F2ZCS4QUTLS6KC4FJ5IHS3XOYWDJQ, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.342232992 (297h0m59.939004023s) [noise=true , address=145.79.227.36:4524 , piece_id=3TAV3P55ITTHBAZEUUUTOAHJRGQXJXXVGZJP6AVR2ANVNOR3S54Q, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.342917368 (297h0m59.939688398s) [noise=true , address=193.149.164.71:4297 , piece_id=4IJPXLPCX3LKFOBOSRHK2DTNRKH67K7FXJUKQ3JPRWHS47ZFEIDA, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.343244957 (297h0m59.940015982s) [noise=true , address=145.79.227.33:4892 , piece_id=DQ6ALC3VOAGFIJI3IKR6I7QFYM22C66QWAAXMH6NXI2FIQEZNQYA, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.344270079 (297h0m59.941041117s) [noise=true , address=145.79.227.21:4497 , piece_id=SP6EKJB7W4QXF7OIBVKEFWCPOZFL5IRIEDVAGS2M4QJSO34CUVRQ, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.345025057 (297h0m59.941796084s) [noise=true , address=77.90.31.17:4793 , piece_id=2U4IBUA2GGUGDBXQGIGQKUK4ZR3L7CDKC6MTGIKAUME4F7MFZRLA, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.345115802 (297h0m59.941886836s) [noise=true , address=167.17.72.69:4731 , piece_id=VOEVIKKFJMC2Y3TPQIDMYUP37NLFU7KPRH4HRRSMQLWUKTSLOH2Q, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.345690600 (297h0m59.942461634s) [noise=true , address=77.90.31.13:4648 , piece_id=BPMB5CFOYHFT3BS4ORR6JF6BNFC2AOAJZOSPMG67NAMLGGP5XVEQ, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.346311592 (297h0m59.943082623s) [noise=true , address=79.127.205.234:4202 , piece_id=YY4LYWX26O6RIOBONESNKTUZEFZCRL4UUX3NEBQ53AJV4IEDG5MQ, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.346791142 (297h0m59.943562166s) [noise=true , address=45.91.116.10:4110 , piece_id=UYLMDCWJEJOHGICA37KNH7G5AQXNIPRH5NRYLBUUMGDRVSLTEYHQ, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.347094999 (297h0m59.94386607s) [noise=true , address=145.79.227.17:4987 , piece_id=PT3TWVB65JZFEWVH4IUZEU2N53FQ57QBRWVLWAUFP4UDLY3LWNLQ, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.347781281 (297h0m59.944552305s) [noise=true , address=79.127.219.46:4919 , piece_id=KK6HRIVBMYLIHRHGPQH3AWMNSVHTTIBESHDMJZA72T446RX6QKRQ, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.348263123 (297h0m59.945034137s) [noise=true , address=193.149.164.68:4033 , piece_id=JZPQKPYGO3W3RZZBITQ73X2XEPIBBSLGRMWIVOM4Z3GTV5UFKRTQ, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.348915505 (297h0m59.945686536s) [noise=true , address=145.79.227.7:4312 , piece_id=2FAL6E3RMBTBHZBXAJJVH2N5VGF5VYI3KNZXGNWCMWGBBPAUO2NQ, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.349366298 (297h0m59.946137337s) [noise=true , address=193.149.164.67:4007 , piece_id=HURQGH67I5B6V4YUV3PRZL3KSKN3XTSK3UMCHZRNT6L3W3PXXHPQ, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.429834098 (297h1m0.026605126s) [upload=332666]: Next returned nil segment Mar 18 20:01:02.429853411 (297h1m0.026624442s) [upload=332666]: Waiting for error group managing segments... Mar 18 20:01:02.446867680 (297h1m0.043638704s) [noise=true , address=45.91.116.10:4110 , piece_id=UYLMDCWJEJOHGICA37KNH7G5AQXNIPRH5NRYLBUUMGDRVSLTEYHQ, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil> Mar 18 20:01:02.451665165 (297h1m0.048436196s) [noise=true , address=45.91.116.4:4961 , piece_id=ZMUGQECUVPVHDNKN75XPBET5M2HFXXK4WMTP4V5KEK63JVZCX5EA, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil> Mar 18 20:01:02.452538886 (297h1m0.049309913s) [noise=true , address=77.90.31.17:4793 , piece_id=2U4IBUA2GGUGDBXQGIGQKUK4ZR3L7CDKC6MTGIKAUME4F7MFZRLA, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil> Mar 18 20:01:02.455597040 (297h1m0.052368067s) [noise=true , address=45.91.116.8:4061 , piece_id=6XFVKT4AAO7QN457OQMRKPI6R24TJTAN47UXLEVOEFQVOS6M6C3Q, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil> Mar 18 20:01:02.457848739 (297h1m0.054619766s) [noise=true , address=77.90.31.13:4648 , piece_id=BPMB5CFOYHFT3BS4ORR6JF6BNFC2AOAJZOSPMG67NAMLGGP5XVEQ, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil> Mar 18 20:01:02.461479709 (297h1m0.058250746s) [noise=true , address=79.127.219.46:4919 , piece_id=KK6HRIVBMYLIHRHGPQH3AWMNSVHTTIBESHDMJZA72T446RX6QKRQ, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil> Mar 18 20:01:02.676859466 (297h1m0.27363049s) [noise=true , address=167.17.72.69:4731 , piece_id=VOEVIKKFJMC2Y3TPQIDMYUP37NLFU7KPRH4HRRSMQLWUKTSLOH2Q, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil> Mar 18 20:01:02.909860743 (297h1m0.506631766s) [noise=true , address=79.127.205.234:4202 , piece_id=YY4LYWX26O6RIOBONESNKTUZEFZCRL4UUX3NEBQ53AJV4IEDG5MQ, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil> Mar 18 20:01:02.930427742 (297h1m0.527198754s) [noise=true , address=193.149.164.71:4297 , piece_id=4IJPXLPCX3LKFOBOSRHK2DTNRKH67K7FXJUKQ3JPRWHS47ZFEIDA, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil> Mar 18 20:01:02.944696889 (297h1m0.541467914s) [noise=true , address=193.149.164.67:4007 , piece_id=HURQGH67I5B6V4YUV3PRZL3KSKN3XTSK3UMCHZRNT6L3W3PXXHPQ, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil> Mar 18 20:01:02.951319183 (297h1m0.548090212s) [noise=true , address=193.149.164.68:4033 , piece_id=JZPQKPYGO3W3RZZBITQ73X2XEPIBBSLGRMWIVOM4Z3GTV5UFKRTQ, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil> Mar 18 20:01:02.975619436 (297h1m0.572390467s) [noise=true , address=79.127.205.233:4166 , piece_id=SFGK5YCWHFRVXF6QNHYP2CYJCZLSTQ76OYRVJPWWXQTF2KFSFJKA, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil> Mar 18 20:01:03.819918528 (297h1m1.416689564s) [noise=true , address=167.17.72.71:4815 , piece_id=CXGTYNNBILN3TT7I6Z7UI75F2ZCS4QUTLS6KC4FJ5IHS3XOYWDJQ, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil> Mar 18 20:01:59.307873420 (297h1m56.904644452s) [noise=true , address=145.79.227.7:4312 , piece_id=2FAL6E3RMBTBHZBXAJJVH2N5VGF5VYI3KNZXGNWCMWGBBPAUO2NQ, seg_pos={0 0}, upload=332666]: Done uploading piece. err: ecclient: upload failed (node:12AVp3qn5uwLPqYaZEwzcgoGMJhNga1BUh6bK5k2bxjWgZLQiiC, address:145.79.227.7:4312): protocol: rpc: dial tcp 145.79.227.7:4312: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:01:59.308026092 (297h1m56.904797124s) [noise=true , address=145.79.227.17:4987 , piece_id=PT3TWVB65JZFEWVH4IUZEU2N53FQ57QBRWVLWAUFP4UDLY3LWNLQ, seg_pos={0 0}, upload=332666]: Done uploading piece. err: ecclient: upload failed (node:1XsPVNGYUhKnXkQRfVfjyBRzyvhShSdsrnzXtwe4Xtf5MaHMvH, address:145.79.227.17:4987): protocol: rpc: dial tcp 145.79.227.17:4987: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:01:59.308195864 (297h1m56.904966894s) [noise=true , address=145.79.227.36:4524 , piece_id=3TAV3P55ITTHBAZEUUUTOAHJRGQXJXXVGZJP6AVR2ANVNOR3S54Q, seg_pos={0 0}, upload=332666]: Done uploading piece. err: ecclient: upload failed (node:1xfbCzjSwFDRrR1gPggeN8tNCLbRRWDVzCJ6mqhZsbVSe8d4s4, address:145.79.227.36:4524): protocol: rpc: dial tcp 145.79.227.36:4524: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:01:59.308316037 (297h1m56.90508706s) [noise=true , address=145.79.227.33:4892 , piece_id=DQ6ALC3VOAGFIJI3IKR6I7QFYM22C66QWAAXMH6NXI2FIQEZNQYA, seg_pos={0 0}, upload=332666]: Done uploading piece. err: ecclient: upload failed (node:12RHwFxcUjPsn12uHEAH3fBDJKv1v1tM8mRJXoFrZrG5djdswF5, address:145.79.227.33:4892): protocol: rpc: dial tcp 145.79.227.33:4892: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:01:59.308404403 (297h1m56.905175433s) [noise=true , address=145.79.227.21:4497 , piece_id=SP6EKJB7W4QXF7OIBVKEFWCPOZFL5IRIEDVAGS2M4QJSO34CUVRQ, seg_pos={0 0}, upload=332666]: Done uploading piece. err: ecclient: upload failed (node:12ZBe4we7VeFdthqRxyy1V9uQrA4r8ouNX68xqsk5bD26dkf2Gd, address:145.79.227.21:4497): protocol: context deadline exceeded; context deadline exceeded Mar 18 20:01:59.308437457 (297h1m56.90520847s) [seg_pos={0 0}, upload=332666]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 18 20:01:59.338288467 (297h1m56.93505949s) [upload=332666]: Done uploading segments... Mar 18 20:01:59.338335497 (297h1m56.935106523s) [upload=332666]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 14.97326428s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 7 out of 60 is failed (threshold=3) [...........................XXX..........X...X.........X....X] duration: 1m0.022816258s trace: https://jaeger.dev.storj.io/trace/698fb9649d932a68 FAIL (2026-03-18 20:03:41 UTC) operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 20:02:41.019616017 (297h2m38.616387042s) [upload=332700]: Uploading segments... Mar 18 20:02:41.019826191 (297h2m38.616597205s) [upload=332700]: Got next segment. Inline: false Mar 18 20:02:41.019879916 (297h2m38.61665094s) [upload=332700]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 20:02:41.019885584 (297h2m38.616656611s) [upload=332700]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 18 20:02:43.834160391 (297h2m41.430931414s) [seg_pos={0 0}, upload=332700]: Begin upload segment... Mar 18 20:02:43.834230381 (297h2m41.431001396s) [seg_pos={0 0}, upload=332700]: Done begin upload segment. Mar 18 20:02:43.834262428 (297h2m41.431033451s) [noise=true , address=145.79.227.52:4451 , piece_id=LR6KVRYMDO2HAESG5BFI5RYPNGVARP6O5HEY2UP77YSTVRKBNUXQ, seg_pos={0 0}, upload=332700]: Uploading piece... Mar 18 20:02:43.834329472 (297h2m41.431100497s) [noise=true , address=45.91.116.13:4164 , piece_id=MNQLVANO3LFKBCJMADD5RULARMRO4VUQY7LT63TU2CCNO765GW7Q, seg_pos={0 0}, upload=332700]: Uploading piece... Mar 18 20:02:43.834938832 (297h2m41.431709858s) [noise=true , address=77.90.31.15:4716 , piece_id=IDQP2MIPCKVS2FGVVGKAIDYTNUIPE56EER5IYZUSOTRD5LZBSXHA, seg_pos={0 0}, upload=332700]: Uploading piece... Mar 18 20:02:43.835898822 (297h2m41.432669851s) [noise=true , address=79.127.205.237:4227 , piece_id=NO25WIYZTEPBOMGETTLSDOJSPKARMKNLVUWDKNXUKHYGDGF2XC5Q, seg_pos={0 0}, upload=332700]: Uploading piece... Mar 18 20:02:43.836893726 (297h2m41.433664759s) [noise=true , address=193.149.164.68:4064 , piece_id=HAA3NDB24MJLAI2HK2UDZQZRLDUXEF76ZJF2VBE6FLKRWSB27Y3A, seg_pos={0 0}, upload=332700]: Uploading piece... Mar 18 20:02:43.837257949 (297h2m41.434028981s) [noise=true , address=193.149.164.66:4974 , piece_id=XWMBTFEWXVVFIR6IYNQYU35QKGVSQJ7YBVSWZ57QSUZ2VSDCJPRA, seg_pos={0 0}, upload=332700]: Uploading piece... Mar 18 20:02:43.838266356 (297h2m41.435037391s) [noise=true , address=145.79.227.34:4924 , piece_id=BNUESPWRZQBX23K7UGNSMF362AWHILNZ43UITBHVGFXIJ4T6V5HQ, seg_pos={0 0}, upload=332700]: Uploading piece... Mar 18 20:02:43.838688582 (297h2m41.435459605s) [noise=true , address=79.127.205.233:4171 , piece_id=YO42WSHYRB2JJX4RVX653N7U6PYTGIW3KQTNSF4445OTN3UW2TIA, seg_pos={0 0}, upload=332700]: Uploading piece... Mar 18 20:02:43.839263235 (297h2m41.436034258s) [noise=true , address=193.149.164.70:4123 , piece_id=VROUAQKRTKZT42ZDDSCIMKFMRODPKFQ5U5DWDJLVI462BVA5YWXA, seg_pos={0 0}, upload=332700]: Uploading piece... Mar 18 20:02:43.839835928 (297h2m41.436606963s) [noise=true , address=45.91.116.4:4948 , piece_id=ULVH4YAJIYY5RWWAGH5RTSZ3BMAT6WCYYAJPSMSP25LCLZQIPG2A, seg_pos={0 0}, upload=332700]: Uploading piece... Mar 18 20:02:43.840390028 (297h2m41.437161055s) [noise=true , address=45.91.116.5:4319 , piece_id=BY6BRNZNKKQKZNAVJTSWUYASX57P5X7OCKQ433T2ARRBXUFNBRCA, seg_pos={0 0}, upload=332700]: Uploading piece... Mar 18 20:02:43.840790648 (297h2m41.437561682s) [noise=true , address=45.91.116.12:4259 , piece_id=4LBHNF3CIQG5LYEY7RYJGNK5Q62X2TSOAXOXSNHO727OXL5OMA4Q, seg_pos={0 0}, upload=332700]: Uploading piece... Mar 18 20:02:43.841319562 (297h2m41.438090584s) [noise=true , address=145.79.227.29:4966 , piece_id=ZTSKMRBESX4OIZMMIPXRW752W5KQYX57RPDQ22XXPURQTSCAI2YA, seg_pos={0 0}, upload=332700]: Uploading piece... Mar 18 20:02:43.841816046 (297h2m41.438587078s) [noise=true , address=193.149.164.71:4287 , piece_id=FYNZ3QGUB7ZFMUJAMYPMUHXJDS66CJ2VLXPV7256DFCABPJMEOYQ, seg_pos={0 0}, upload=332700]: Uploading piece... Mar 18 20:02:43.842288870 (297h2m41.439059903s) [noise=true , address=79.127.219.41:4941 , piece_id=CQXOXMCJHWJ7FSIXDVYUJZMBFW5FUUOJNKJJOVLZRSK5D6XZXX5Q, seg_pos={0 0}, upload=332700]: Uploading piece... Mar 18 20:02:43.842956832 (297h2m41.439727861s) [noise=true , address=145.79.227.49:4369 , piece_id=CBYHFRBNKSPUNJHAE6S72YOFACWLZKGZIFV6SXWDYMVSA4FBPKQQ, seg_pos={0 0}, upload=332700]: Uploading piece... Mar 18 20:02:43.844386533 (297h2m41.441157559s) [noise=true , address=77.90.31.10:4533 , piece_id=VB5BKAWDUZFVII4ZY3JRVK67VUULPSAHTJVZMXXVATP3SXAR2XHQ, seg_pos={0 0}, upload=332700]: Uploading piece... Mar 18 20:02:43.844868553 (297h2m41.441639576s) [noise=true , address=145.79.227.37:4582 , piece_id=DLOFMRZRG3BPFN7JF73OM5HUV56FLEH2IDDC47UQWWWOTRVRR4HQ, seg_pos={0 0}, upload=332700]: Uploading piece... Mar 18 20:02:43.903559633 (297h2m41.500330666s) [upload=332700]: Next returned nil segment Mar 18 20:02:43.903578892 (297h2m41.500349925s) [upload=332700]: Waiting for error group managing segments... Mar 18 20:02:43.908218461 (297h2m41.504989474s) [noise=true , address=45.91.116.4:4948 , piece_id=ULVH4YAJIYY5RWWAGH5RTSZ3BMAT6WCYYAJPSMSP25LCLZQIPG2A, seg_pos={0 0}, upload=332700]: Done uploading piece. err: <nil> Mar 18 20:02:43.917679744 (297h2m41.514450753s) [noise=true , address=45.91.116.12:4259 , piece_id=4LBHNF3CIQG5LYEY7RYJGNK5Q62X2TSOAXOXSNHO727OXL5OMA4Q, seg_pos={0 0}, upload=332700]: Done uploading piece. err: <nil> Mar 18 20:02:43.924357408 (297h2m41.521128422s) [noise=true , address=77.90.31.10:4533 , piece_id=VB5BKAWDUZFVII4ZY3JRVK67VUULPSAHTJVZMXXVATP3SXAR2XHQ, seg_pos={0 0}, upload=332700]: Done uploading piece. err: <nil> Mar 18 20:02:43.926248407 (297h2m41.52301943s) [noise=true , address=77.90.31.15:4716 , piece_id=IDQP2MIPCKVS2FGVVGKAIDYTNUIPE56EER5IYZUSOTRD5LZBSXHA, seg_pos={0 0}, upload=332700]: Done uploading piece. err: <nil> Mar 18 20:02:43.927212300 (297h2m41.523983323s) [noise=true , address=45.91.116.5:4319 , piece_id=BY6BRNZNKKQKZNAVJTSWUYASX57P5X7OCKQ433T2ARRBXUFNBRCA, seg_pos={0 0}, upload=332700]: Done uploading piece. err: <nil> Mar 18 20:02:43.927791610 (297h2m41.524562634s) [noise=true , address=45.91.116.13:4164 , piece_id=MNQLVANO3LFKBCJMADD5RULARMRO4VUQY7LT63TU2CCNO765GW7Q, seg_pos={0 0}, upload=332700]: Done uploading piece. err: <nil> Mar 18 20:02:44.028504592 (297h2m41.625275611s) [noise=true , address=79.127.219.41:4941 , piece_id=CQXOXMCJHWJ7FSIXDVYUJZMBFW5FUUOJNKJJOVLZRSK5D6XZXX5Q, seg_pos={0 0}, upload=332700]: Done uploading piece. err: <nil> Mar 18 20:02:44.392266647 (297h2m41.989037671s) [noise=true , address=79.127.205.233:4171 , piece_id=YO42WSHYRB2JJX4RVX653N7U6PYTGIW3KQTNSF4445OTN3UW2TIA, seg_pos={0 0}, upload=332700]: Done uploading piece. err: <nil> Mar 18 20:02:44.429403642 (297h2m42.026174669s) [noise=true , address=193.149.164.66:4974 , piece_id=XWMBTFEWXVVFIR6IYNQYU35QKGVSQJ7YBVSWZ57QSUZ2VSDCJPRA, seg_pos={0 0}, upload=332700]: Done uploading piece. err: <nil> Mar 18 20:02:44.433374828 (297h2m42.030145853s) [noise=true , address=193.149.164.68:4064 , piece_id=HAA3NDB24MJLAI2HK2UDZQZRLDUXEF76ZJF2VBE6FLKRWSB27Y3A, seg_pos={0 0}, upload=332700]: Done uploading piece. err: <nil> Mar 18 20:02:44.444389014 (297h2m42.041160042s) [noise=true , address=193.149.164.71:4287 , piece_id=FYNZ3QGUB7ZFMUJAMYPMUHXJDS66CJ2VLXPV7256DFCABPJMEOYQ, seg_pos={0 0}, upload=332700]: Done uploading piece. err: <nil> Mar 18 20:02:44.578333498 (297h2m42.175104529s) [noise=true , address=79.127.205.237:4227 , piece_id=NO25WIYZTEPBOMGETTLSDOJSPKARMKNLVUWDKNXUKHYGDGF2XC5Q, seg_pos={0 0}, upload=332700]: Done uploading piece. err: <nil> Mar 18 20:02:44.665665140 (297h2m42.262436168s) [noise=true , address=193.149.164.70:4123 , piece_id=VROUAQKRTKZT42ZDDSCIMKFMRODPKFQ5U5DWDJLVI462BVA5YWXA, seg_pos={0 0}, upload=332700]: Done uploading piece. err: <nil> Mar 18 20:03:40.987108256 (297h3m38.58387928s) [noise=true , address=145.79.227.49:4369 , piece_id=CBYHFRBNKSPUNJHAE6S72YOFACWLZKGZIFV6SXWDYMVSA4FBPKQQ, seg_pos={0 0}, upload=332700]: Done uploading piece. err: ecclient: upload failed (node:12nLksGXGvDwuTWfnmkceuwZNDmrF17iNepHPdYkkvZG2QQUwG7, address:145.79.227.49:4369): protocol: rpc: dial tcp 145.79.227.49:4369: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:03:40.987479807 (297h3m38.58425084s) [noise=true , address=145.79.227.29:4966 , piece_id=ZTSKMRBESX4OIZMMIPXRW752W5KQYX57RPDQ22XXPURQTSCAI2YA, seg_pos={0 0}, upload=332700]: Done uploading piece. err: ecclient: upload failed (node:1DLnsYVdudE4mY9dNAQKhBRppUEAHvxbf84KX69451pwxSxzZK, address:145.79.227.29:4966): protocol: rpc: dial tcp 145.79.227.29:4966: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:03:40.987517864 (297h3m38.584288896s) [noise=true , address=145.79.227.34:4924 , piece_id=BNUESPWRZQBX23K7UGNSMF362AWHILNZ43UITBHVGFXIJ4T6V5HQ, seg_pos={0 0}, upload=332700]: Done uploading piece. err: ecclient: upload failed (node:19TbrQjuN1EhBNVtha9EcKPSCAWyttfmHpec6QeF1S5zW6DRZs, address:145.79.227.34:4924): protocol: rpc: dial tcp 145.79.227.34:4924: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:03:40.987933296 (297h3m38.584704329s) [noise=true , address=145.79.227.37:4582 , piece_id=DLOFMRZRG3BPFN7JF73OM5HUV56FLEH2IDDC47UQWWWOTRVRR4HQ, seg_pos={0 0}, upload=332700]: Done uploading piece. err: ecclient: upload failed (node:129o91j97t4V5AjaouCCy2wDuHgwF94iQiJh2h9ux1aFo7AVCSn, address:145.79.227.37:4582): protocol: rpc: dial tcp 145.79.227.37:4582: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:03:40.988167350 (297h3m38.584938376s) [noise=true , address=145.79.227.52:4451 , piece_id=LR6KVRYMDO2HAESG5BFI5RYPNGVARP6O5HEY2UP77YSTVRKBNUXQ, seg_pos={0 0}, upload=332700]: Done uploading piece. err: ecclient: upload failed (node:12bC4P3xBiPFkxLvavGFVbWKnbprCStL6UJgfKSA2Wd2fX7Mbfw, address:145.79.227.52:4451): protocol: rpc: dial tcp 145.79.227.52:4451: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:03:40.988197047 (297h3m38.58496806s) [seg_pos={0 0}, upload=332700]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 18 20:03:41.007727578 (297h3m38.604498608s) [upload=332700]: Done uploading segments... Mar 18 20:03:41.007779207 (297h3m38.60455025s) [upload=332700]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 1m0.022841994s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 8 out of 60 is failed (threshold=3) [........................XXXXXX.XX...........................] duration: 14.16977593s trace: https://jaeger.dev.storj.io/trace/4b4d10a8051ad01b FAIL (2026-03-18 20:03:40 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 14.169831562s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 37 out of 60 is failed (threshold=3) [......................XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.] duration: 12.315805434s trace: https://jaeger.dev.storj.io/trace/5687315889981ce8 FAIL (2026-03-18 20:03:37 UTC) operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== Mar 18 20:02:51.644702372 (297h2m49.241473391s) [upload=332705]: Starting upload Mar 18 20:02:51.644798111 (297h2m49.241569134s) [upload=332705]: Done starting upload Mar 18 20:02:51.645113854 (297h2m49.241884869s) [upload=332705]: Uploading segments... Mar 18 20:02:51.645238072 (297h2m49.242009104s) [upload=332705]: Got next segment. Inline: false Mar 18 20:02:51.645300317 (297h2m49.242071356s) [upload=332705]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 20:02:51.645306598 (297h2m49.242077631s) [upload=332705]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 18 20:02:54.396505197 (297h2m51.993276213s) [seg_pos={0 0}, upload=332705]: Begin upload segment... Mar 18 20:02:54.396561710 (297h2m51.993332739s) [seg_pos={0 0}, upload=332705]: Done begin upload segment. Mar 18 20:02:54.396603203 (297h2m51.99337423s) [noise=true , address=145.79.227.40:4049 , piece_id=MRAZ3BUALGGG5QDWD4TTEWVNURS4GERXWMJVAIEFJW3RTT2MBIBQ, seg_pos={0 0}, upload=332705]: Uploading piece... Mar 18 20:02:54.396637779 (297h2m51.993408804s) [noise=true , address=45.91.116.9:4264 , piece_id=LDED6USPHNBRH4SXEUL7MISQBR35E6HEN3K73EHNP2PNC7TEI76Q, seg_pos={0 0}, upload=332705]: Uploading piece... Mar 18 20:02:54.397161330 (297h2m51.993932353s) [noise=true , address=77.90.31.14:4665 , piece_id=ASPRJFR6Q25NNYXJZCUB7INVOD3IR7CLRQ4ISYDYQITU4GLFLP6A, seg_pos={0 0}, upload=332705]: Uploading piece... Mar 18 20:02:54.397612650 (297h2m51.994383674s) [noise=true , address=77.90.31.12:4594 , piece_id=6ANB6ACDYWLL6YYGZ47KZF5CZGR3O23O6HU3O5TRRZVQSNFDQPWA, seg_pos={0 0}, upload=332705]: Uploading piece... Mar 18 20:02:54.397642123 (297h2m51.994413145s) [noise=true , address=145.79.227.38:4979 , piece_id=2LNXWHUPDSOD6ENIJYBSKN7KKZQJ7HS4QFFVU7Q55R4LZI3IUHYA, seg_pos={0 0}, upload=332705]: Uploading piece... Mar 18 20:02:54.398038863 (297h2m51.994809877s) [noise=true , address=45.91.116.11:4119 , piece_id=URGCQRIFRSINDMXM33TUPTK5D6PYHP25TWNNOGIEALLAIUQJV6ZA, seg_pos={0 0}, upload=332705]: Uploading piece... Mar 18 20:02:54.398442157 (297h2m51.99521318s) [noise=true , address=77.90.31.18:4810 , piece_id=VIAHWKJYQVIKBYVVPBYHLF4MFYKT4ZXNZ7XGH7N32IMCPICTGXMA, seg_pos={0 0}, upload=332705]: Uploading piece... Mar 18 20:02:54.398978421 (297h2m51.995749435s) [noise=true , address=145.79.227.33:4872 , piece_id=XV4YPUQBLEIUAD3YPRKGMJ3JD6TBPPXRWLWNSFRKOAZVG26SLPBA, seg_pos={0 0}, upload=332705]: Uploading piece... Mar 18 20:02:54.399583059 (297h2m51.996354068s) [noise=true , address=45.91.116.4:4972 , piece_id=GZTYHE32SSKTNG64RB3YGKD76GXHL673MC7DZZDQXMH2RISJIRIQ, seg_pos={0 0}, upload=332705]: Uploading piece... Mar 18 20:02:54.399955533 (297h2m51.99672656s) [noise=true , address=145.79.227.4:4189 , piece_id=EHV52NGZJW2CPO54YNI5I4LW2V32KOZ3SMSDO527KG4CILVS7OWA, seg_pos={0 0}, upload=332705]: Uploading piece... Mar 18 20:02:54.400590874 (297h2m51.99736191s) [noise=true , address=145.79.227.9:4371 , piece_id=CGM5XJXOUDOLOVGFKYODLQXGISBYUUSR3HPIV6V7RS224PBLZDSQ, seg_pos={0 0}, upload=332705]: Uploading piece... Mar 18 20:02:54.400877860 (297h2m51.997648907s) [noise=true , address=45.91.116.8:4048 , piece_id=4VOXISGTUBIQDHO7WAUBXLK3KWATM6J6TORPU3UTIH6LN62EQDPQ, seg_pos={0 0}, upload=332705]: Uploading piece... Mar 18 20:02:54.401283278 (297h2m51.99805431s) [noise=true , address=145.79.227.2:4119 , piece_id=7A5ADO2MBTEHDNL2UAZBP2VXISGWJG4ET5PLPDCUSQZBIKGIBSMA, seg_pos={0 0}, upload=332705]: Uploading piece... Mar 18 20:02:54.401363826 (297h2m51.99813486s) [noise=true , address=79.127.219.41:4927 , piece_id=SBYDPU37E5X2FNQFM3OR7TF7XITJY6DQ7GEDBMXRN4S5QRNRLMKA, seg_pos={0 0}, upload=332705]: Uploading piece... Mar 18 20:02:54.402476102 (297h2m51.999247128s) [noise=true , address=45.91.116.14:4216 , piece_id=2VZGTL6QHH23TMKFTEL3AYC7DYYHYBXKVBSJ6OUEYO352QO43MLA, seg_pos={0 0}, upload=332705]: Uploading piece... Mar 18 20:02:54.403600726 (297h2m52.00037174s) [noise=true , address=45.91.116.7:4026 , piece_id=LJH7X5LXGW33N54YGGRC25D4A2WJCMM3E47BGWEH5RFHF44SH27Q, seg_pos={0 0}, upload=332705]: Uploading piece... Mar 18 20:02:54.477768122 (297h2m52.074539132s) [upload=332705]: Next returned nil segment Mar 18 20:02:54.477782088 (297h2m52.074553111s) [upload=332705]: Waiting for error group managing segments... Mar 18 20:02:54.480685814 (297h2m52.077456826s) [noise=true , address=45.91.116.4:4972 , piece_id=GZTYHE32SSKTNG64RB3YGKD76GXHL673MC7DZZDQXMH2RISJIRIQ, seg_pos={0 0}, upload=332705]: Done uploading piece. err: <nil> Mar 18 20:02:54.481180454 (297h2m52.077951467s) [noise=true , address=45.91.116.9:4264 , piece_id=LDED6USPHNBRH4SXEUL7MISQBR35E6HEN3K73EHNP2PNC7TEI76Q, seg_pos={0 0}, upload=332705]: Done uploading piece. err: <nil> Mar 18 20:02:54.481763004 (297h2m52.078534027s) [noise=true , address=45.91.116.11:4119 , piece_id=URGCQRIFRSINDMXM33TUPTK5D6PYHP25TWNNOGIEALLAIUQJV6ZA, seg_pos={0 0}, upload=332705]: Done uploading piece. err: <nil> Mar 18 20:02:54.482185984 (297h2m52.078957007s) [noise=true , address=77.90.31.18:4810 , piece_id=VIAHWKJYQVIKBYVVPBYHLF4MFYKT4ZXNZ7XGH7N32IMCPICTGXMA, seg_pos={0 0}, upload=332705]: Done uploading piece. err: <nil> Mar 18 20:02:54.482565109 (297h2m52.079336122s) [noise=true , address=45.91.116.7:4026 , piece_id=LJH7X5LXGW33N54YGGRC25D4A2WJCMM3E47BGWEH5RFHF44SH27Q, seg_pos={0 0}, upload=332705]: Done uploading piece. err: <nil> Mar 18 20:02:54.483141108 (297h2m52.079912132s) [noise=true , address=79.127.219.41:4927 , piece_id=SBYDPU37E5X2FNQFM3OR7TF7XITJY6DQ7GEDBMXRN4S5QRNRLMKA, seg_pos={0 0}, upload=332705]: Done uploading piece. err: <nil> Mar 18 20:02:54.483203323 (297h2m52.079974347s) [noise=true , address=45.91.116.8:4048 , piece_id=4VOXISGTUBIQDHO7WAUBXLK3KWATM6J6TORPU3UTIH6LN62EQDPQ, seg_pos={0 0}, upload=332705]: Done uploading piece. err: <nil> Mar 18 20:02:54.483613521 (297h2m52.080384545s) [noise=true , address=77.90.31.14:4665 , piece_id=ASPRJFR6Q25NNYXJZCUB7INVOD3IR7CLRQ4ISYDYQITU4GLFLP6A, seg_pos={0 0}, upload=332705]: Done uploading piece. err: <nil> Mar 18 20:02:54.483950756 (297h2m52.08072178s) [noise=true , address=45.91.116.14:4216 , piece_id=2VZGTL6QHH23TMKFTEL3AYC7DYYHYBXKVBSJ6OUEYO352QO43MLA, seg_pos={0 0}, upload=332705]: Done uploading piece. err: <nil> Mar 18 20:02:54.484098547 (297h2m52.080869561s) [noise=true , address=77.90.31.12:4594 , piece_id=6ANB6ACDYWLL6YYGZ47KZF5CZGR3O23O6HU3O5TRRZVQSNFDQPWA, seg_pos={0 0}, upload=332705]: Done uploading piece. err: <nil> Mar 18 20:03:21.614050439 (297h3m19.210821472s) [noise=true , address=145.79.227.4:4189 , piece_id=EHV52NGZJW2CPO54YNI5I4LW2V32KOZ3SMSDO527KG4CILVS7OWA, seg_pos={0 0}, upload=332705]: Done uploading piece. err: ecclient: upload failed (node:1jH3757i2F3q18XSpoXznsysZPaXmEnqynpLPDupgRfxJrTgKv, address:145.79.227.4:4189): protocol: rpc: dial tcp 145.79.227.4:4189: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:03:21.614446336 (297h3m19.211217371s) [noise=true , address=145.79.227.38:4979 , piece_id=2LNXWHUPDSOD6ENIJYBSKN7KKZQJ7HS4QFFVU7Q55R4LZI3IUHYA, seg_pos={0 0}, upload=332705]: Done uploading piece. err: ecclient: upload failed (node:12B518TvbbRhj6LiXxV4GwUgAoUYmWMWeTpyYerKnKuZYBjXoma, address:145.79.227.38:4979): protocol: rpc: dial tcp 145.79.227.38:4979: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:03:21.614793963 (297h3m19.211564998s) [noise=true , address=145.79.227.33:4872 , piece_id=XV4YPUQBLEIUAD3YPRKGMJ3JD6TBPPXRWLWNSFRKOAZVG26SLPBA, seg_pos={0 0}, upload=332705]: Done uploading piece. err: ecclient: upload failed (node:12dNdm2kAok8jFgbYE24oXUL9KCJVDDiinLUJgDSYEY3fTEBJec, address:145.79.227.33:4872): protocol: rpc: dial tcp 145.79.227.33:4872: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:03:21.615486238 (297h3m19.212257273s) [noise=true , address=145.79.227.40:4049 , piece_id=MRAZ3BUALGGG5QDWD4TTEWVNURS4GERXWMJVAIEFJW3RTT2MBIBQ, seg_pos={0 0}, upload=332705]: Done uploading piece. err: ecclient: upload failed (node:1n91vwhc11FPE57QAUCfgDpffrmPrJxWxh7R9ecgt41JEPux6J, address:145.79.227.40:4049): protocol: rpc: dial tcp 145.79.227.40:4049: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:03:21.615753593 (297h3m19.212524626s) [noise=true , address=145.79.227.9:4371 , piece_id=CGM5XJXOUDOLOVGFKYODLQXGISBYUUSR3HPIV6V7RS224PBLZDSQ, seg_pos={0 0}, upload=332705]: Done uploading piece. err: ecclient: upload failed (node:12cSqcGZqvZqs1QKBi6WK4fUDKgvQcmTupR1C9ZkWz879XeWPEL, address:145.79.227.9:4371): protocol: rpc: dial tcp 145.79.227.9:4371: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:03:21.616646748 (297h3m19.213417781s) [noise=true , address=145.79.227.2:4119 , piece_id=7A5ADO2MBTEHDNL2UAZBP2VXISGWJG4ET5PLPDCUSQZBIKGIBSMA, seg_pos={0 0}, upload=332705]: Done uploading piece. err: ecclient: upload failed (node:121xNAnkm9HZXUszp26UmWTsuhp3igyeDVea6gSCdtrQUQDVHpW, address:145.79.227.2:4119): protocol: rpc: dial tcp 145.79.227.2:4119: i/o timeout; conn closed: op Mar 18 20:03:21.616687674 (297h3m19.213458692s) [seg_pos={0 0}, upload=332705]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Mar 18 20:03:21.636036019 (297h3m19.232807043s) [upload=332705]: Done uploading segments... Mar 18 20:03:21.636087341 (297h3m19.232858366s) [upload=332705]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration: 12.315839413s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 46 out of 60 is failed (threshold=6) [..........XXXXXXXXX.XXX...XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 5.001338876s trace: https://jaeger.dev.storj.io/trace/396e2a4fdfd19f39 FAIL (2026-03-18 20:03:32 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001418723s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 37 out of 60 is failed (threshold=3) [.......................XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 30.025177692s trace: https://jaeger.dev.storj.io/trace/5687315889981ce8 FAIL (2026-03-18 20:03:21 UTC) operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== Mar 18 20:02:51.644702372 (297h2m49.241473391s) [upload=332705]: Starting upload Mar 18 20:02:51.644798111 (297h2m49.241569134s) [upload=332705]: Done starting upload Mar 18 20:02:51.645113854 (297h2m49.241884869s) [upload=332705]: Uploading segments... Mar 18 20:02:51.645238072 (297h2m49.242009104s) [upload=332705]: Got next segment. Inline: false Mar 18 20:02:51.645300317 (297h2m49.242071356s) [upload=332705]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 20:02:51.645306598 (297h2m49.242077631s) [upload=332705]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 18 20:02:54.396505197 (297h2m51.993276213s) [seg_pos={0 0}, upload=332705]: Begin upload segment... Mar 18 20:02:54.396561710 (297h2m51.993332739s) [seg_pos={0 0}, upload=332705]: Done begin upload segment. Mar 18 20:02:54.396603203 (297h2m51.99337423s) [noise=true , address=145.79.227.40:4049 , piece_id=MRAZ3BUALGGG5QDWD4TTEWVNURS4GERXWMJVAIEFJW3RTT2MBIBQ, seg_pos={0 0}, upload=332705]: Uploading piece... Mar 18 20:02:54.396637779 (297h2m51.993408804s) [noise=true , address=45.91.116.9:4264 , piece_id=LDED6USPHNBRH4SXEUL7MISQBR35E6HEN3K73EHNP2PNC7TEI76Q, seg_pos={0 0}, upload=332705]: Uploading piece... Mar 18 20:02:54.397161330 (297h2m51.993932353s) [noise=true , address=77.90.31.14:4665 , piece_id=ASPRJFR6Q25NNYXJZCUB7INVOD3IR7CLRQ4ISYDYQITU4GLFLP6A, seg_pos={0 0}, upload=332705]: Uploading piece... Mar 18 20:02:54.397612650 (297h2m51.994383674s) [noise=true , address=77.90.31.12:4594 , piece_id=6ANB6ACDYWLL6YYGZ47KZF5CZGR3O23O6HU3O5TRRZVQSNFDQPWA, seg_pos={0 0}, upload=332705]: Uploading piece... Mar 18 20:02:54.397642123 (297h2m51.994413145s) [noise=true , address=145.79.227.38:4979 , piece_id=2LNXWHUPDSOD6ENIJYBSKN7KKZQJ7HS4QFFVU7Q55R4LZI3IUHYA, seg_pos={0 0}, upload=332705]: Uploading piece... Mar 18 20:02:54.398038863 (297h2m51.994809877s) [noise=true , address=45.91.116.11:4119 , piece_id=URGCQRIFRSINDMXM33TUPTK5D6PYHP25TWNNOGIEALLAIUQJV6ZA, seg_pos={0 0}, upload=332705]: Uploading piece... Mar 18 20:02:54.398442157 (297h2m51.99521318s) [noise=true , address=77.90.31.18:4810 , piece_id=VIAHWKJYQVIKBYVVPBYHLF4MFYKT4ZXNZ7XGH7N32IMCPICTGXMA, seg_pos={0 0}, upload=332705]: Uploading piece... Mar 18 20:02:54.398978421 (297h2m51.995749435s) [noise=true , address=145.79.227.33:4872 , piece_id=XV4YPUQBLEIUAD3YPRKGMJ3JD6TBPPXRWLWNSFRKOAZVG26SLPBA, seg_pos={0 0}, upload=332705]: Uploading piece... Mar 18 20:02:54.399583059 (297h2m51.996354068s) [noise=true , address=45.91.116.4:4972 , piece_id=GZTYHE32SSKTNG64RB3YGKD76GXHL673MC7DZZDQXMH2RISJIRIQ, seg_pos={0 0}, upload=332705]: Uploading piece... Mar 18 20:02:54.399955533 (297h2m51.99672656s) [noise=true , address=145.79.227.4:4189 , piece_id=EHV52NGZJW2CPO54YNI5I4LW2V32KOZ3SMSDO527KG4CILVS7OWA, seg_pos={0 0}, upload=332705]: Uploading piece... Mar 18 20:02:54.400590874 (297h2m51.99736191s) [noise=true , address=145.79.227.9:4371 , piece_id=CGM5XJXOUDOLOVGFKYODLQXGISBYUUSR3HPIV6V7RS224PBLZDSQ, seg_pos={0 0}, upload=332705]: Uploading piece... Mar 18 20:02:54.400877860 (297h2m51.997648907s) [noise=true , address=45.91.116.8:4048 , piece_id=4VOXISGTUBIQDHO7WAUBXLK3KWATM6J6TORPU3UTIH6LN62EQDPQ, seg_pos={0 0}, upload=332705]: Uploading piece... Mar 18 20:02:54.401283278 (297h2m51.99805431s) [noise=true , address=145.79.227.2:4119 , piece_id=7A5ADO2MBTEHDNL2UAZBP2VXISGWJG4ET5PLPDCUSQZBIKGIBSMA, seg_pos={0 0}, upload=332705]: Uploading piece... Mar 18 20:02:54.401363826 (297h2m51.99813486s) [noise=true , address=79.127.219.41:4927 , piece_id=SBYDPU37E5X2FNQFM3OR7TF7XITJY6DQ7GEDBMXRN4S5QRNRLMKA, seg_pos={0 0}, upload=332705]: Uploading piece... Mar 18 20:02:54.402476102 (297h2m51.999247128s) [noise=true , address=45.91.116.14:4216 , piece_id=2VZGTL6QHH23TMKFTEL3AYC7DYYHYBXKVBSJ6OUEYO352QO43MLA, seg_pos={0 0}, upload=332705]: Uploading piece... Mar 18 20:02:54.403600726 (297h2m52.00037174s) [noise=true , address=45.91.116.7:4026 , piece_id=LJH7X5LXGW33N54YGGRC25D4A2WJCMM3E47BGWEH5RFHF44SH27Q, seg_pos={0 0}, upload=332705]: Uploading piece... Mar 18 20:02:54.477768122 (297h2m52.074539132s) [upload=332705]: Next returned nil segment Mar 18 20:02:54.477782088 (297h2m52.074553111s) [upload=332705]: Waiting for error group managing segments... Mar 18 20:02:54.480685814 (297h2m52.077456826s) [noise=true , address=45.91.116.4:4972 , piece_id=GZTYHE32SSKTNG64RB3YGKD76GXHL673MC7DZZDQXMH2RISJIRIQ, seg_pos={0 0}, upload=332705]: Done uploading piece. err: <nil> Mar 18 20:02:54.481180454 (297h2m52.077951467s) [noise=true , address=45.91.116.9:4264 , piece_id=LDED6USPHNBRH4SXEUL7MISQBR35E6HEN3K73EHNP2PNC7TEI76Q, seg_pos={0 0}, upload=332705]: Done uploading piece. err: <nil> Mar 18 20:02:54.481763004 (297h2m52.078534027s) [noise=true , address=45.91.116.11:4119 , piece_id=URGCQRIFRSINDMXM33TUPTK5D6PYHP25TWNNOGIEALLAIUQJV6ZA, seg_pos={0 0}, upload=332705]: Done uploading piece. err: <nil> Mar 18 20:02:54.482185984 (297h2m52.078957007s) [noise=true , address=77.90.31.18:4810 , piece_id=VIAHWKJYQVIKBYVVPBYHLF4MFYKT4ZXNZ7XGH7N32IMCPICTGXMA, seg_pos={0 0}, upload=332705]: Done uploading piece. err: <nil> Mar 18 20:02:54.482565109 (297h2m52.079336122s) [noise=true , address=45.91.116.7:4026 , piece_id=LJH7X5LXGW33N54YGGRC25D4A2WJCMM3E47BGWEH5RFHF44SH27Q, seg_pos={0 0}, upload=332705]: Done uploading piece. err: <nil> Mar 18 20:02:54.483141108 (297h2m52.079912132s) [noise=true , address=79.127.219.41:4927 , piece_id=SBYDPU37E5X2FNQFM3OR7TF7XITJY6DQ7GEDBMXRN4S5QRNRLMKA, seg_pos={0 0}, upload=332705]: Done uploading piece. err: <nil> Mar 18 20:02:54.483203323 (297h2m52.079974347s) [noise=true , address=45.91.116.8:4048 , piece_id=4VOXISGTUBIQDHO7WAUBXLK3KWATM6J6TORPU3UTIH6LN62EQDPQ, seg_pos={0 0}, upload=332705]: Done uploading piece. err: <nil> Mar 18 20:02:54.483613521 (297h2m52.080384545s) [noise=true , address=77.90.31.14:4665 , piece_id=ASPRJFR6Q25NNYXJZCUB7INVOD3IR7CLRQ4ISYDYQITU4GLFLP6A, seg_pos={0 0}, upload=332705]: Done uploading piece. err: <nil> Mar 18 20:02:54.483950756 (297h2m52.08072178s) [noise=true , address=45.91.116.14:4216 , piece_id=2VZGTL6QHH23TMKFTEL3AYC7DYYHYBXKVBSJ6OUEYO352QO43MLA, seg_pos={0 0}, upload=332705]: Done uploading piece. err: <nil> Mar 18 20:02:54.484098547 (297h2m52.080869561s) [noise=true , address=77.90.31.12:4594 , piece_id=6ANB6ACDYWLL6YYGZ47KZF5CZGR3O23O6HU3O5TRRZVQSNFDQPWA, seg_pos={0 0}, upload=332705]: Done uploading piece. err: <nil> Mar 18 20:03:21.614050439 (297h3m19.210821472s) [noise=true , address=145.79.227.4:4189 , piece_id=EHV52NGZJW2CPO54YNI5I4LW2V32KOZ3SMSDO527KG4CILVS7OWA, seg_pos={0 0}, upload=332705]: Done uploading piece. err: ecclient: upload failed (node:1jH3757i2F3q18XSpoXznsysZPaXmEnqynpLPDupgRfxJrTgKv, address:145.79.227.4:4189): protocol: rpc: dial tcp 145.79.227.4:4189: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:03:21.614446336 (297h3m19.211217371s) [noise=true , address=145.79.227.38:4979 , piece_id=2LNXWHUPDSOD6ENIJYBSKN7KKZQJ7HS4QFFVU7Q55R4LZI3IUHYA, seg_pos={0 0}, upload=332705]: Done uploading piece. err: ecclient: upload failed (node:12B518TvbbRhj6LiXxV4GwUgAoUYmWMWeTpyYerKnKuZYBjXoma, address:145.79.227.38:4979): protocol: rpc: dial tcp 145.79.227.38:4979: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:03:21.614793963 (297h3m19.211564998s) [noise=true , address=145.79.227.33:4872 , piece_id=XV4YPUQBLEIUAD3YPRKGMJ3JD6TBPPXRWLWNSFRKOAZVG26SLPBA, seg_pos={0 0}, upload=332705]: Done uploading piece. err: ecclient: upload failed (node:12dNdm2kAok8jFgbYE24oXUL9KCJVDDiinLUJgDSYEY3fTEBJec, address:145.79.227.33:4872): protocol: rpc: dial tcp 145.79.227.33:4872: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:03:21.615486238 (297h3m19.212257273s) [noise=true , address=145.79.227.40:4049 , piece_id=MRAZ3BUALGGG5QDWD4TTEWVNURS4GERXWMJVAIEFJW3RTT2MBIBQ, seg_pos={0 0}, upload=332705]: Done uploading piece. err: ecclient: upload failed (node:1n91vwhc11FPE57QAUCfgDpffrmPrJxWxh7R9ecgt41JEPux6J, address:145.79.227.40:4049): protocol: rpc: dial tcp 145.79.227.40:4049: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:03:21.615753593 (297h3m19.212524626s) [noise=true , address=145.79.227.9:4371 , piece_id=CGM5XJXOUDOLOVGFKYODLQXGISBYUUSR3HPIV6V7RS224PBLZDSQ, seg_pos={0 0}, upload=332705]: Done uploading piece. err: ecclient: upload failed (node:12cSqcGZqvZqs1QKBi6WK4fUDKgvQcmTupR1C9ZkWz879XeWPEL, address:145.79.227.9:4371): protocol: rpc: dial tcp 145.79.227.9:4371: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:03:21.616646748 (297h3m19.213417781s) [noise=true , address=145.79.227.2:4119 , piece_id=7A5ADO2MBTEHDNL2UAZBP2VXISGWJG4ET5PLPDCUSQZBIKGIBSMA, seg_pos={0 0}, upload=332705]: Done uploading piece. err: ecclient: upload failed (node:121xNAnkm9HZXUszp26UmWTsuhp3igyeDVea6gSCdtrQUQDVHpW, address:145.79.227.2:4119): protocol: rpc: dial tcp 145.79.227.2:4119: i/o timeout; conn closed: op Mar 18 20:03:21.616687674 (297h3m19.213458692s) [seg_pos={0 0}, upload=332705]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Mar 18 20:03:21.636036019 (297h3m19.232807043s) [upload=332705]: Done uploading segments... Mar 18 20:03:21.636087341 (297h3m19.232858366s) [upload=332705]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration: 30.025193035s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 6 out of 60 is failed (threshold=3) [............................X..........XX..X....X........X..] duration: 15.821947039s trace: https://jaeger.dev.storj.io/trace/589419efc52e5e2b FAIL (2026-03-18 20:03:11 UTC) operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== Mar 18 20:00:59.343084966 (297h0m56.939855989s) [upload=332666]: Starting upload Mar 18 20:00:59.343175518 (297h0m56.939946548s) [upload=332666]: Done starting upload Mar 18 20:00:59.343565585 (297h0m56.940336612s) [upload=332666]: Uploading segments... Mar 18 20:00:59.344256647 (297h0m56.941027676s) [upload=332666]: Got next segment. Inline: false Mar 18 20:00:59.344352215 (297h0m56.941123241s) [upload=332666]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 20:00:59.344358706 (297h0m56.941129732s) [upload=332666]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 18 20:01:02.340259299 (297h0m59.937030329s) [seg_pos={0 0}, upload=332666]: Begin upload segment... Mar 18 20:01:02.340368743 (297h0m59.937139767s) [seg_pos={0 0}, upload=332666]: Done begin upload segment. Mar 18 20:01:02.340479563 (297h0m59.93725059s) [noise=true , address=45.91.116.8:4061 , piece_id=6XFVKT4AAO7QN457OQMRKPI6R24TJTAN47UXLEVOEFQVOS6M6C3Q, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.340523837 (297h0m59.93729486s) [noise=true , address=79.127.205.233:4166 , piece_id=SFGK5YCWHFRVXF6QNHYP2CYJCZLSTQ76OYRVJPWWXQTF2KFSFJKA, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.341468438 (297h0m59.938239447s) [noise=true , address=45.91.116.4:4961 , piece_id=ZMUGQECUVPVHDNKN75XPBET5M2HFXXK4WMTP4V5KEK63JVZCX5EA, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.341520025 (297h0m59.938291058s) [noise=true , address=167.17.72.71:4815 , piece_id=CXGTYNNBILN3TT7I6Z7UI75F2ZCS4QUTLS6KC4FJ5IHS3XOYWDJQ, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.342232992 (297h0m59.939004023s) [noise=true , address=145.79.227.36:4524 , piece_id=3TAV3P55ITTHBAZEUUUTOAHJRGQXJXXVGZJP6AVR2ANVNOR3S54Q, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.342917368 (297h0m59.939688398s) [noise=true , address=193.149.164.71:4297 , piece_id=4IJPXLPCX3LKFOBOSRHK2DTNRKH67K7FXJUKQ3JPRWHS47ZFEIDA, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.343244957 (297h0m59.940015982s) [noise=true , address=145.79.227.33:4892 , piece_id=DQ6ALC3VOAGFIJI3IKR6I7QFYM22C66QWAAXMH6NXI2FIQEZNQYA, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.344270079 (297h0m59.941041117s) [noise=true , address=145.79.227.21:4497 , piece_id=SP6EKJB7W4QXF7OIBVKEFWCPOZFL5IRIEDVAGS2M4QJSO34CUVRQ, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.345025057 (297h0m59.941796084s) [noise=true , address=77.90.31.17:4793 , piece_id=2U4IBUA2GGUGDBXQGIGQKUK4ZR3L7CDKC6MTGIKAUME4F7MFZRLA, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.345115802 (297h0m59.941886836s) [noise=true , address=167.17.72.69:4731 , piece_id=VOEVIKKFJMC2Y3TPQIDMYUP37NLFU7KPRH4HRRSMQLWUKTSLOH2Q, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.345690600 (297h0m59.942461634s) [noise=true , address=77.90.31.13:4648 , piece_id=BPMB5CFOYHFT3BS4ORR6JF6BNFC2AOAJZOSPMG67NAMLGGP5XVEQ, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.346311592 (297h0m59.943082623s) [noise=true , address=79.127.205.234:4202 , piece_id=YY4LYWX26O6RIOBONESNKTUZEFZCRL4UUX3NEBQ53AJV4IEDG5MQ, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.346791142 (297h0m59.943562166s) [noise=true , address=45.91.116.10:4110 , piece_id=UYLMDCWJEJOHGICA37KNH7G5AQXNIPRH5NRYLBUUMGDRVSLTEYHQ, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.347094999 (297h0m59.94386607s) [noise=true , address=145.79.227.17:4987 , piece_id=PT3TWVB65JZFEWVH4IUZEU2N53FQ57QBRWVLWAUFP4UDLY3LWNLQ, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.347781281 (297h0m59.944552305s) [noise=true , address=79.127.219.46:4919 , piece_id=KK6HRIVBMYLIHRHGPQH3AWMNSVHTTIBESHDMJZA72T446RX6QKRQ, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.348263123 (297h0m59.945034137s) [noise=true , address=193.149.164.68:4033 , piece_id=JZPQKPYGO3W3RZZBITQ73X2XEPIBBSLGRMWIVOM4Z3GTV5UFKRTQ, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.348915505 (297h0m59.945686536s) [noise=true , address=145.79.227.7:4312 , piece_id=2FAL6E3RMBTBHZBXAJJVH2N5VGF5VYI3KNZXGNWCMWGBBPAUO2NQ, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.349366298 (297h0m59.946137337s) [noise=true , address=193.149.164.67:4007 , piece_id=HURQGH67I5B6V4YUV3PRZL3KSKN3XTSK3UMCHZRNT6L3W3PXXHPQ, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.429834098 (297h1m0.026605126s) [upload=332666]: Next returned nil segment Mar 18 20:01:02.429853411 (297h1m0.026624442s) [upload=332666]: Waiting for error group managing segments... Mar 18 20:01:02.446867680 (297h1m0.043638704s) [noise=true , address=45.91.116.10:4110 , piece_id=UYLMDCWJEJOHGICA37KNH7G5AQXNIPRH5NRYLBUUMGDRVSLTEYHQ, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil> Mar 18 20:01:02.451665165 (297h1m0.048436196s) [noise=true , address=45.91.116.4:4961 , piece_id=ZMUGQECUVPVHDNKN75XPBET5M2HFXXK4WMTP4V5KEK63JVZCX5EA, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil> Mar 18 20:01:02.452538886 (297h1m0.049309913s) [noise=true , address=77.90.31.17:4793 , piece_id=2U4IBUA2GGUGDBXQGIGQKUK4ZR3L7CDKC6MTGIKAUME4F7MFZRLA, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil> Mar 18 20:01:02.455597040 (297h1m0.052368067s) [noise=true , address=45.91.116.8:4061 , piece_id=6XFVKT4AAO7QN457OQMRKPI6R24TJTAN47UXLEVOEFQVOS6M6C3Q, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil> Mar 18 20:01:02.457848739 (297h1m0.054619766s) [noise=true , address=77.90.31.13:4648 , piece_id=BPMB5CFOYHFT3BS4ORR6JF6BNFC2AOAJZOSPMG67NAMLGGP5XVEQ, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil> Mar 18 20:01:02.461479709 (297h1m0.058250746s) [noise=true , address=79.127.219.46:4919 , piece_id=KK6HRIVBMYLIHRHGPQH3AWMNSVHTTIBESHDMJZA72T446RX6QKRQ, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil> Mar 18 20:01:02.676859466 (297h1m0.27363049s) [noise=true , address=167.17.72.69:4731 , piece_id=VOEVIKKFJMC2Y3TPQIDMYUP37NLFU7KPRH4HRRSMQLWUKTSLOH2Q, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil> Mar 18 20:01:02.909860743 (297h1m0.506631766s) [noise=true , address=79.127.205.234:4202 , piece_id=YY4LYWX26O6RIOBONESNKTUZEFZCRL4UUX3NEBQ53AJV4IEDG5MQ, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil> Mar 18 20:01:02.930427742 (297h1m0.527198754s) [noise=true , address=193.149.164.71:4297 , piece_id=4IJPXLPCX3LKFOBOSRHK2DTNRKH67K7FXJUKQ3JPRWHS47ZFEIDA, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil> Mar 18 20:01:02.944696889 (297h1m0.541467914s) [noise=true , address=193.149.164.67:4007 , piece_id=HURQGH67I5B6V4YUV3PRZL3KSKN3XTSK3UMCHZRNT6L3W3PXXHPQ, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil> Mar 18 20:01:02.951319183 (297h1m0.548090212s) [noise=true , address=193.149.164.68:4033 , piece_id=JZPQKPYGO3W3RZZBITQ73X2XEPIBBSLGRMWIVOM4Z3GTV5UFKRTQ, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil> Mar 18 20:01:02.975619436 (297h1m0.572390467s) [noise=true , address=79.127.205.233:4166 , piece_id=SFGK5YCWHFRVXF6QNHYP2CYJCZLSTQ76OYRVJPWWXQTF2KFSFJKA, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil> Mar 18 20:01:03.819918528 (297h1m1.416689564s) [noise=true , address=167.17.72.71:4815 , piece_id=CXGTYNNBILN3TT7I6Z7UI75F2ZCS4QUTLS6KC4FJ5IHS3XOYWDJQ, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil> Mar 18 20:01:59.307873420 (297h1m56.904644452s) [noise=true , address=145.79.227.7:4312 , piece_id=2FAL6E3RMBTBHZBXAJJVH2N5VGF5VYI3KNZXGNWCMWGBBPAUO2NQ, seg_pos={0 0}, upload=332666]: Done uploading piece. err: ecclient: upload failed (node:12AVp3qn5uwLPqYaZEwzcgoGMJhNga1BUh6bK5k2bxjWgZLQiiC, address:145.79.227.7:4312): protocol: rpc: dial tcp 145.79.227.7:4312: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:01:59.308026092 (297h1m56.904797124s) [noise=true , address=145.79.227.17:4987 , piece_id=PT3TWVB65JZFEWVH4IUZEU2N53FQ57QBRWVLWAUFP4UDLY3LWNLQ, seg_pos={0 0}, upload=332666]: Done uploading piece. err: ecclient: upload failed (node:1XsPVNGYUhKnXkQRfVfjyBRzyvhShSdsrnzXtwe4Xtf5MaHMvH, address:145.79.227.17:4987): protocol: rpc: dial tcp 145.79.227.17:4987: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:01:59.308195864 (297h1m56.904966894s) [noise=true , address=145.79.227.36:4524 , piece_id=3TAV3P55ITTHBAZEUUUTOAHJRGQXJXXVGZJP6AVR2ANVNOR3S54Q, seg_pos={0 0}, upload=332666]: Done uploading piece. err: ecclient: upload failed (node:1xfbCzjSwFDRrR1gPggeN8tNCLbRRWDVzCJ6mqhZsbVSe8d4s4, address:145.79.227.36:4524): protocol: rpc: dial tcp 145.79.227.36:4524: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:01:59.308316037 (297h1m56.90508706s) [noise=true , address=145.79.227.33:4892 , piece_id=DQ6ALC3VOAGFIJI3IKR6I7QFYM22C66QWAAXMH6NXI2FIQEZNQYA, seg_pos={0 0}, upload=332666]: Done uploading piece. err: ecclient: upload failed (node:12RHwFxcUjPsn12uHEAH3fBDJKv1v1tM8mRJXoFrZrG5djdswF5, address:145.79.227.33:4892): protocol: rpc: dial tcp 145.79.227.33:4892: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:01:59.308404403 (297h1m56.905175433s) [noise=true , address=145.79.227.21:4497 , piece_id=SP6EKJB7W4QXF7OIBVKEFWCPOZFL5IRIEDVAGS2M4QJSO34CUVRQ, seg_pos={0 0}, upload=332666]: Done uploading piece. err: ecclient: upload failed (node:12ZBe4we7VeFdthqRxyy1V9uQrA4r8ouNX68xqsk5bD26dkf2Gd, address:145.79.227.21:4497): protocol: context deadline exceeded; context deadline exceeded Mar 18 20:01:59.308437457 (297h1m56.90520847s) [seg_pos={0 0}, upload=332666]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 18 20:01:59.338288467 (297h1m56.93505949s) [upload=332666]: Done uploading segments... Mar 18 20:01:59.338335497 (297h1m56.935106523s) [upload=332666]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 15.82197849s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 8 out of 60 is failed (threshold=3) [.........................XXXXXX.XX..........................] duration: 14.87605267s trace: https://jaeger.dev.storj.io/trace/4b4d10a8051ad01b FAIL (2026-03-18 20:02:52 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 14.876082177s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 36 out of 60 is failed (threshold=3) [........................XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 30.025575265s trace: https://jaeger.dev.storj.io/trace/3161cb4de8ca6f34 FAIL (2026-03-18 20:02:51 UTC) operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== Mar 18 20:02:21.560928779 (297h2m19.157699795s) [upload=332695]: Starting upload Mar 18 20:02:21.560987579 (297h2m19.157758611s) [upload=332695]: Done starting upload Mar 18 20:02:21.561028601 (297h2m19.157799615s) [upload=332695]: Uploading segments... Mar 18 20:02:21.561235919 (297h2m19.158006934s) [upload=332695]: Got next segment. Inline: false Mar 18 20:02:21.561276655 (297h2m19.15804768s) [upload=332695]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 20:02:21.561282134 (297h2m19.158053157s) [upload=332695]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 18 20:02:24.383881482 (297h2m21.980652512s) [seg_pos={0 0}, upload=332695]: Begin upload segment... Mar 18 20:02:24.383943027 (297h2m21.980714041s) [seg_pos={0 0}, upload=332695]: Done begin upload segment. Mar 18 20:02:24.383996366 (297h2m21.980767389s) [noise=true , address=77.90.31.14:4666 , piece_id=KQJQAQAV7CCMYAZ2VIIS2XVNTGVE4CEN26DECIBZN76D5XH3UTMQ, seg_pos={0 0}, upload=332695]: Uploading piece... Mar 18 20:02:24.384016629 (297h2m21.980787662s) [noise=true , address=145.79.227.10:4407 , piece_id=HCIQ7MQE5INROVA4JWXXBI6NBTH3GZVOUNW4XOWXBC2OXIPGSGKQ, seg_pos={0 0}, upload=332695]: Uploading piece... Mar 18 20:02:24.384755171 (297h2m21.981526205s) [noise=true , address=145.79.227.37:4570 , piece_id=XNYV7FYPAAZ47BQMTCEZZ63U32IRPHEHYXL52Z4COICZFLX4G7NQ, seg_pos={0 0}, upload=332695]: Uploading piece... Mar 18 20:02:24.385473766 (297h2m21.982244795s) [noise=true , address=45.91.116.5:4298 , piece_id=U4LOFGYW46FX7BTJQCO7KBSDUXENWD7ZZFY6GSDJJF2JZVCHX3IQ, seg_pos={0 0}, upload=332695]: Uploading piece... Mar 18 20:02:24.385608981 (297h2m21.982380015s) [noise=true , address=77.90.31.16:4738 , piece_id=OEA4W3UPM3ECAZQHSITOHEHOHQNLUKPR35A3EPZJP2QF2TJMBGJA, seg_pos={0 0}, upload=332695]: Uploading piece... Mar 18 20:02:24.385941521 (297h2m21.982712555s) [noise=true , address=145.79.227.4:4188 , piece_id=2ZNGXKSLIOL7CTK6BSHIYWSYUDMXI6CP5QAOI5QNWRZLDPCIL2AA, seg_pos={0 0}, upload=332695]: Uploading piece... Mar 18 20:02:24.386256609 (297h2m21.983027642s) [noise=true , address=145.79.227.27:4692 , piece_id=G5NROI32QZ7TNNIQPUFOUSHFYVTNBX7AAEQL52EH3EWFE6YFXH5Q, seg_pos={0 0}, upload=332695]: Uploading piece... Mar 18 20:02:24.386388747 (297h2m21.98315978s) [noise=true , address=77.90.31.13:4629 , piece_id=LO5A3KL4MKLHNBQAIPYLFY6XPKEVRNSLBWBD26EVZUDCK7667WEA, seg_pos={0 0}, upload=332695]: Uploading piece... Mar 18 20:02:24.386877177 (297h2m21.983648205s) [noise=true , address=79.127.219.38:4820 , piece_id=6M4DXDMJE5R7EXV5UOKLFDGX2W77NP7IOXEGFHNOBQTAYEU2LDCQ, seg_pos={0 0}, upload=332695]: Uploading piece... Mar 18 20:02:24.386926693 (297h2m21.983697728s) [noise=true , address=45.91.116.6:4974 , piece_id=FPLZI4645MDQXRMURV5WKJK72OEBMRHAW5INVH3KYVEZDYCBG3AA, seg_pos={0 0}, upload=332695]: Uploading piece... Mar 18 20:02:24.387336730 (297h2m21.984107762s) [noise=true , address=145.79.227.52:4449 , piece_id=UDPFME34QFR33WAXUNJPM7MGQMKNNJSCRPEORMHQOYBYWOHELMVQ, seg_pos={0 0}, upload=332695]: Uploading piece... Mar 18 20:02:24.387700331 (297h2m21.984471365s) [noise=true , address=45.91.116.7:4010 , piece_id=M45XHS6ZQI25PZBGKLFHE32QFDQFLXZCXTI2KHUFS4FESXL4KXWQ, seg_pos={0 0}, upload=332695]: Uploading piece... Mar 18 20:02:24.388464733 (297h2m21.985235766s) [noise=true , address=145.79.227.51:4412 , piece_id=SJTPBG4J5HSG4S2MQOQAWBV6I6P4UHULIBMB65PUGXG765SNNBIQ, seg_pos={0 0}, upload=332695]: Uploading piece... Mar 18 20:02:24.389136574 (297h2m21.985907615s) [noise=true , address=79.127.219.41:4925 , piece_id=ADKWF5GQQ5QFBI4YZ37Z6DHLBIOLKQHNU2JDTF6NVKYRRJUIUC6A, seg_pos={0 0}, upload=332695]: Uploading piece... Mar 18 20:02:24.391737543 (297h2m21.988508575s) [noise=true , address=45.91.116.13:4153 , piece_id=OHTYPSLXBVOSXA7ZW2CPJARQPEEUXYIHRUB4E6RYBMJGOMWCRPQA, seg_pos={0 0}, upload=332695]: Uploading piece... Mar 18 20:02:24.392647606 (297h2m21.989418638s) [noise=true , address=77.90.31.15:4697 , piece_id=UKKDQ4L7IQF5EWCJIOEOZNOURZPQZ6H4QFNCMJSW4OFQT5HCJNEQ, seg_pos={0 0}, upload=332695]: Uploading piece... Mar 18 20:02:24.453558219 (297h2m22.050329232s) [upload=332695]: Next returned nil segment Mar 18 20:02:24.453572009 (297h2m22.050343033s) [upload=332695]: Waiting for error group managing segments... Mar 18 20:02:24.457921598 (297h2m22.054692622s) [noise=true , address=45.91.116.13:4153 , piece_id=OHTYPSLXBVOSXA7ZW2CPJARQPEEUXYIHRUB4E6RYBMJGOMWCRPQA, seg_pos={0 0}, upload=332695]: Done uploading piece. err: <nil> Mar 18 20:02:24.458289093 (297h2m22.05506012s) [noise=true , address=45.91.116.7:4010 , piece_id=M45XHS6ZQI25PZBGKLFHE32QFDQFLXZCXTI2KHUFS4FESXL4KXWQ, seg_pos={0 0}, upload=332695]: Done uploading piece. err: <nil> Mar 18 20:02:24.458574349 (297h2m22.055345362s) [noise=true , address=45.91.116.5:4298 , piece_id=U4LOFGYW46FX7BTJQCO7KBSDUXENWD7ZZFY6GSDJJF2JZVCHX3IQ, seg_pos={0 0}, upload=332695]: Done uploading piece. err: <nil> Mar 18 20:02:24.459608079 (297h2m22.056379102s) [noise=true , address=77.90.31.14:4666 , piece_id=KQJQAQAV7CCMYAZ2VIIS2XVNTGVE4CEN26DECIBZN76D5XH3UTMQ, seg_pos={0 0}, upload=332695]: Done uploading piece. err: <nil> Mar 18 20:02:24.460060467 (297h2m22.056831491s) [noise=true , address=79.127.219.41:4925 , piece_id=ADKWF5GQQ5QFBI4YZ37Z6DHLBIOLKQHNU2JDTF6NVKYRRJUIUC6A, seg_pos={0 0}, upload=332695]: Done uploading piece. err: <nil> Mar 18 20:02:24.460827541 (297h2m22.057598569s) [noise=true , address=77.90.31.16:4738 , piece_id=OEA4W3UPM3ECAZQHSITOHEHOHQNLUKPR35A3EPZJP2QF2TJMBGJA, seg_pos={0 0}, upload=332695]: Done uploading piece. err: <nil> Mar 18 20:02:24.472731792 (297h2m22.069502814s) [noise=true , address=45.91.116.6:4974 , piece_id=FPLZI4645MDQXRMURV5WKJK72OEBMRHAW5INVH3KYVEZDYCBG3AA, seg_pos={0 0}, upload=332695]: Done uploading piece. err: <nil> Mar 18 20:02:24.473622230 (297h2m22.070393254s) [noise=true , address=77.90.31.13:4629 , piece_id=LO5A3KL4MKLHNBQAIPYLFY6XPKEVRNSLBWBD26EVZUDCK7667WEA, seg_pos={0 0}, upload=332695]: Done uploading piece. err: <nil> Mar 18 20:02:24.477080683 (297h2m22.073851706s) [noise=true , address=79.127.219.38:4820 , piece_id=6M4DXDMJE5R7EXV5UOKLFDGX2W77NP7IOXEGFHNOBQTAYEU2LDCQ, seg_pos={0 0}, upload=332695]: Done uploading piece. err: <nil> Mar 18 20:02:24.488478138 (297h2m22.085249169s) [noise=true , address=77.90.31.15:4697 , piece_id=UKKDQ4L7IQF5EWCJIOEOZNOURZPQZ6H4QFNCMJSW4OFQT5HCJNEQ, seg_pos={0 0}, upload=332695]: Done uploading piece. err: <nil> Mar 18 20:02:51.519528481 (297h2m49.11629951s) [noise=true , address=145.79.227.52:4449 , piece_id=UDPFME34QFR33WAXUNJPM7MGQMKNNJSCRPEORMHQOYBYWOHELMVQ, seg_pos={0 0}, upload=332695]: Done uploading piece. err: ecclient: upload failed (node:12f14pHDEEamEnHmeZCe38rjaE6EYWYkGMPrZMsbSDmApN5rckq, address:145.79.227.52:4449): protocol: rpc: dial tcp 145.79.227.52:4449: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:02:51.519712853 (297h2m49.116483887s) [noise=true , address=145.79.227.51:4412 , piece_id=SJTPBG4J5HSG4S2MQOQAWBV6I6P4UHULIBMB65PUGXG765SNNBIQ, seg_pos={0 0}, upload=332695]: Done uploading piece. err: ecclient: upload failed (node:12vBvEfCYZxsv17cVsQTbCQ4YjZfiD4RXjWyz7Ka9ZZrtvzC7SS, address:145.79.227.51:4412): protocol: rpc: dial tcp 145.79.227.51:4412: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:02:51.519812355 (297h2m49.116583388s) [noise=true , address=145.79.227.37:4570 , piece_id=XNYV7FYPAAZ47BQMTCEZZ63U32IRPHEHYXL52Z4COICZFLX4G7NQ, seg_pos={0 0}, upload=332695]: Done uploading piece. err: ecclient: upload failed (node:1q5evYAfULsn19ddsAvSzJwxiDd8wsWbjP6gpcRkuU1b1q57eJ, address:145.79.227.37:4570): protocol: rpc: dial tcp 145.79.227.37:4570: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:02:51.520056957 (297h2m49.116827986s) [noise=true , address=145.79.227.27:4692 , piece_id=G5NROI32QZ7TNNIQPUFOUSHFYVTNBX7AAEQL52EH3EWFE6YFXH5Q, seg_pos={0 0}, upload=332695]: Done uploading piece. err: ecclient: upload failed (node:1n3SDzXGgXbvYBGtLdND9oy93AHNFokRnF6Yv3EvoMC9UgN2H9, address:145.79.227.27:4692): protocol: rpc: dial tcp 145.79.227.27:4692: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:02:51.520328436 (297h2m49.117099466s) [noise=true , address=145.79.227.10:4407 , piece_id=HCIQ7MQE5INROVA4JWXXBI6NBTH3GZVOUNW4XOWXBC2OXIPGSGKQ, seg_pos={0 0}, upload=332695]: Done uploading piece. err: ecclient: upload failed (node:1kJKjEfzwg1wuBUP2Bw4oaSoy6mQZdvDED6PpkbrqQwbkh3bUW, address:145.79.227.10:4407): protocol: rpc: dial tcp 145.79.227.10:4407: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:02:51.520608763 (297h2m49.117379796s) [noise=true , address=145.79.227.4:4188 , piece_id=2ZNGXKSLIOL7CTK6BSHIYWSYUDMXI6CP5QAOI5QNWRZLDPCIL2AA, seg_pos={0 0}, upload=332695]: Done uploading piece. err: ecclient: upload failed (node:12pLFHQ2Xj6n53w9eLQtMB4A29k3Xh6w716z5s2CGxwPnBoZcgZ, address:145.79.227.4:4188): protocol: rpc: dial tcp 145.79.227.4:4188: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:02:51.520646132 (297h2m49.117417141s) [seg_pos={0 0}, upload=332695]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Mar 18 20:02:51.542788720 (297h2m49.139559733s) [upload=332695]: Done uploading segments... Mar 18 20:02:51.542851589 (297h2m49.139622614s) [upload=332695]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration: 30.025621161s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 8 out of 60 is failed (threshold=3) [..........................XXXXXX.XX.........................] duration: 14.107306223s trace: https://jaeger.dev.storj.io/trace/4b4d10a8051ad01b FAIL (2026-03-18 20:02:29 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 14.107332799s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 35 out of 60 is failed (threshold=3) [.........................XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 30.026449939s trace: https://jaeger.dev.storj.io/trace/170dde13c0caf9a9 FAIL (2026-03-18 20:02:21 UTC) operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 7) ======LOGS====== load Mar 18 20:01:51.074918509 (297h1m48.671689533s) [upload=332684]: Uploading segments... Mar 18 20:01:51.075222443 (297h1m48.671993469s) [upload=332684]: Got next segment. Inline: false Mar 18 20:01:51.075304417 (297h1m48.67207544s) [upload=332684]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 20:01:51.075310948 (297h1m48.67208197s) [upload=332684]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 18 20:01:53.933952170 (297h1m51.530723193s) [seg_pos={0 0}, upload=332684]: Begin upload segment... Mar 18 20:01:53.934066695 (297h1m51.530837706s) [seg_pos={0 0}, upload=332684]: Done begin upload segment. Mar 18 20:01:53.934132132 (297h1m51.530903162s) [noise=true , address=45.91.116.12:4226 , piece_id=D7RKYGMQ26EOQ6HHJE7MUT4BNDRNJ26BPV6MFWRFZDR433IFLVZQ, seg_pos={0 0}, upload=332684]: Uploading piece... Mar 18 20:01:53.934244207 (297h1m51.531015241s) [noise=true , address=145.79.227.41:4088 , piece_id=RJHH66QUR7TCA3X54CL6U7DPCD6Y4DBSQZ3CIJKDUFHHVR6YGEGA, seg_pos={0 0}, upload=332684]: Uploading piece... Mar 18 20:01:53.934894525 (297h1m51.531665558s) [noise=true , address=145.79.227.32:4838 , piece_id=NI7ROSVHIFBN33KR6MZDCEUPIBEAW6HYSGXL5SAIAZDAPYIYMPYQ, seg_pos={0 0}, upload=332684]: Uploading piece... Mar 18 20:01:53.935123881 (297h1m51.531894915s) [noise=true , address=77.90.31.11:4558 , piece_id=3RR5RQYF3XYOSFFPZDWJWZA4I46TRAU4662EYPOID23RQF4GNULA, seg_pos={0 0}, upload=332684]: Uploading piece... Mar 18 20:01:53.935887727 (297h1m51.532658754s) [noise=true , address=145.79.227.8:4334 , piece_id=SCOHQEWMAGZXRCHC52U46N4Q47Q54KCUSX7V3FBX3SFDOPGOF4RA, seg_pos={0 0}, upload=332684]: Uploading piece... Mar 18 20:01:53.936399405 (297h1m51.533170431s) [noise=true , address=45.91.116.9:4262 , piece_id=I6MTHUHDDKWBN3NFFUCMACUFTJ4G73AANZCE2GADSJGCI7OTUEJQ, seg_pos={0 0}, upload=332684]: Uploading piece... Mar 18 20:01:53.937971750 (297h1m51.534742779s) [noise=true , address=145.79.227.30:4767 , piece_id=WB3DBNKTC4UIJGQEVUPARWHBXGS3EGURVND745TWKH5A5QMKSS6Q, seg_pos={0 0}, upload=332684]: Uploading piece... Mar 18 20:01:53.938276525 (297h1m51.535047549s) [noise=true , address=45.91.116.4:4952 , piece_id=KPDHHEJ6R42TWLEQXDEV7ZC7G4R2UIUATVPBBGFJGMZGPX32UBWA, seg_pos={0 0}, upload=332684]: Uploading piece... Mar 18 20:01:53.938466427 (297h1m51.535237451s) [noise=true , address=45.91.116.8:4048 , piece_id=BCQNSS2VAOAR23NQ26UVKGYISNHO3FEBNPMUY3MX7GNA54JF556Q, seg_pos={0 0}, upload=332684]: Uploading piece... Mar 18 20:01:53.938781448 (297h1m51.535552477s) [noise=true , address=77.90.31.13:4627 , piece_id=J7SZFF2QGK4VVME24KO3SNQIBPTYHKFGVTTROLT7IYAK6MD6KFHA, seg_pos={0 0}, upload=332684]: Uploading piece... Mar 18 20:01:53.938952229 (297h1m51.535723253s) [noise=true , address=145.79.227.15:4877 , piece_id=PUE4GXHV536SDG73GTRTQLEBRN3KCSBYO2BUTPH27AOHOZ54HWZA, seg_pos={0 0}, upload=332684]: Uploading piece... Mar 18 20:01:53.939252443 (297h1m51.536023471s) [noise=true , address=145.79.227.18:4010 , piece_id=ASGI6MJ7PYVHMRYKBNECSIYEXLEWHYO5CHA5TIMWWBVANM2QQGKQ, seg_pos={0 0}, upload=332684]: Uploading piece... Mar 18 20:01:53.939601963 (297h1m51.536372986s) [noise=true , address=145.79.227.14:4840 , piece_id=7TUQZYOFPGOS4M7GJBZ2NO6WZJNOU4WENVPLDYPDKVJOWNB7KLFQ, seg_pos={0 0}, upload=332684]: Uploading piece... Mar 18 20:01:53.939784097 (297h1m51.536555124s) [noise=true , address=145.79.227.42:4125 , piece_id=CLBBXYAO47WWUCKODZANGQ3AGSFQJ46BB2DIOIL7RHBCAFTP6QPQ, seg_pos={0 0}, upload=332684]: Uploading piece... Mar 18 20:01:53.940370326 (297h1m51.537141349s) [noise=true , address=77.90.31.14:4662 , piece_id=BIZSJ77Y7S5E2FFN2Q44UOZC4PAR7GPJ76S7I5RREPNRMBONKLMQ, seg_pos={0 0}, upload=332684]: Uploading piece... Mar 18 20:01:53.940853610 (297h1m51.537624642s) [noise=true , address=145.79.227.33:4869 , piece_id=IG7AZCFWN3S6XSONQ6WZYJXWWUL2COOXUEYGXJQR7QXD3LO5FENQ, seg_pos={0 0}, upload=332684]: Uploading piece... Mar 18 20:01:53.999102398 (297h1m51.595873414s) [upload=332684]: Next returned nil segment Mar 18 20:01:53.999116524 (297h1m51.59588755s) [upload=332684]: Waiting for error group managing segments... Mar 18 20:01:54.006456679 (297h1m51.603227705s) [noise=true , address=77.90.31.14:4662 , piece_id=BIZSJ77Y7S5E2FFN2Q44UOZC4PAR7GPJ76S7I5RREPNRMBONKLMQ, seg_pos={0 0}, upload=332684]: Done uploading piece. err: <nil> Mar 18 20:01:54.007706339 (297h1m51.604477363s) [noise=true , address=45.91.116.12:4226 , piece_id=D7RKYGMQ26EOQ6HHJE7MUT4BNDRNJ26BPV6MFWRFZDR433IFLVZQ, seg_pos={0 0}, upload=332684]: Done uploading piece. err: <nil> Mar 18 20:01:54.008293909 (297h1m51.605064921s) [noise=true , address=45.91.116.4:4952 , piece_id=KPDHHEJ6R42TWLEQXDEV7ZC7G4R2UIUATVPBBGFJGMZGPX32UBWA, seg_pos={0 0}, upload=332684]: Done uploading piece. err: <nil> Mar 18 20:01:54.008722129 (297h1m51.605493157s) [noise=true , address=45.91.116.9:4262 , piece_id=I6MTHUHDDKWBN3NFFUCMACUFTJ4G73AANZCE2GADSJGCI7OTUEJQ, seg_pos={0 0}, upload=332684]: Done uploading piece. err: <nil> Mar 18 20:01:54.009341223 (297h1m51.606112245s) [noise=true , address=77.90.31.11:4558 , piece_id=3RR5RQYF3XYOSFFPZDWJWZA4I46TRAU4662EYPOID23RQF4GNULA, seg_pos={0 0}, upload=332684]: Done uploading piece. err: <nil> Mar 18 20:01:54.023495053 (297h1m51.620266089s) [noise=true , address=77.90.31.13:4627 , piece_id=J7SZFF2QGK4VVME24KO3SNQIBPTYHKFGVTTROLT7IYAK6MD6KFHA, seg_pos={0 0}, upload=332684]: Done uploading piece. err: <nil> Mar 18 20:01:54.024050289 (297h1m51.620821302s) [noise=true , address=45.91.116.8:4048 , piece_id=BCQNSS2VAOAR23NQ26UVKGYISNHO3FEBNPMUY3MX7GNA54JF556Q, seg_pos={0 0}, upload=332684]: Done uploading piece. err: <nil> Mar 18 20:02:21.040229821 (297h2m18.637000847s) [noise=true , address=145.79.227.8:4334 , piece_id=SCOHQEWMAGZXRCHC52U46N4Q47Q54KCUSX7V3FBX3SFDOPGOF4RA, seg_pos={0 0}, upload=332684]: Done uploading piece. err: ecclient: upload failed (node:12KYMZAFsJbhWC4CdKRod2Z6qsC6ANcpqmpWSWx7UYajp6JvKH1, address:145.79.227.8:4334): protocol: rpc: dial tcp 145.79.227.8:4334: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:02:21.040441707 (297h2m18.63721274s) [noise=true , address=145.79.227.14:4840 , piece_id=7TUQZYOFPGOS4M7GJBZ2NO6WZJNOU4WENVPLDYPDKVJOWNB7KLFQ, seg_pos={0 0}, upload=332684]: Done uploading piece. err: ecclient: upload failed (node:12crcXpNe9NgeXTsUpooLSBgrSUFbRxbSbanLa39w2Hr5mGFww, address:145.79.227.14:4840): protocol: rpc: dial tcp 145.79.227.14:4840: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:02:21.040582017 (297h2m18.637353051s) [noise=true , address=145.79.227.30:4767 , piece_id=WB3DBNKTC4UIJGQEVUPARWHBXGS3EGURVND745TWKH5A5QMKSS6Q, seg_pos={0 0}, upload=332684]: Done uploading piece. err: ecclient: upload failed (node:12sfaYCUhNCAyQAPu1jac5dRWR35Qqc4ZcKg75mp7nVQak295ug, address:145.79.227.30:4767): protocol: rpc: dial tcp 145.79.227.30:4767: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:02:21.040808886 (297h2m18.637579923s) [noise=true , address=145.79.227.32:4838 , piece_id=NI7ROSVHIFBN33KR6MZDCEUPIBEAW6HYSGXL5SAIAZDAPYIYMPYQ, seg_pos={0 0}, upload=332684]: Done uploading piece. err: ecclient: upload failed (node:1eorQLNmgEaXv8myr6t3Q8rysGwsvtdZzu5b7sndhoANb6Waj6, address:145.79.227.32:4838): protocol: context deadline exceeded; context deadline exceeded Mar 18 20:02:21.040879973 (297h2m18.63765103s) [noise=true , address=145.79.227.18:4010 , piece_id=ASGI6MJ7PYVHMRYKBNECSIYEXLEWHYO5CHA5TIMWWBVANM2QQGKQ, seg_pos={0 0}, upload=332684]: Done uploading piece. err: ecclient: upload failed (node:19XkukzF8ZCLdHLTYQTYpX5iF4tvT1iM6Zxxs1ZNQ4r5X4ousp, address:145.79.227.18:4010): protocol: rpc: dial tcp 145.79.227.18:4010: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:02:21.041215410 (297h2m18.637986444s) [noise=true , address=145.79.227.33:4869 , piece_id=IG7AZCFWN3S6XSONQ6WZYJXWWUL2COOXUEYGXJQR7QXD3LO5FENQ, seg_pos={0 0}, upload=332684]: Done uploading piece. err: ecclient: upload failed (node:18CQxjq65r6xVTmNRurSQPzKxj4BoR6cJ8Uq29sTrdh6uXm5sq, address:145.79.227.33:4869): protocol: rpc: dial tcp 145.79.227.33:4869: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:02:21.041518692 (297h2m18.638289728s) [noise=true , address=145.79.227.41:4088 , piece_id=RJHH66QUR7TCA3X54CL6U7DPCD6Y4DBSQZ3CIJKDUFHHVR6YGEGA, seg_pos={0 0}, upload=332684]: Done uploading piece. err: ecclient: upload failed (node:1284h6cPNVXJYMkSekGQVw8b7p89tkhpmgYLxTTYUDu2J7Wadoj, address:145.79.227.41:4088): protocol: rpc: dial tcp 145.79.227.41:4088: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:02:21.041593287 (297h2m18.638364318s) [noise=true , address=145.79.227.15:4877 , piece_id=PUE4GXHV536SDG73GTRTQLEBRN3KCSBYO2BUTPH27AOHOZ54HWZA, seg_pos={0 0}, upload=332684]: Done uploading piece. err: ecclient: upload failed (node:12akiEFhbb3wqcG7uJVZHwj4rHEQpX2AXKXrovAsaUPCdhj2PXz, address:145.79.227.15:4877): protocol: rpc: dial tcp 145.79.227.15:4877: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:02:21.041748925 (297h2m18.638519959s) [noise=true , address=145.79.227.42:4125 , piece_id=CLBBXYAO47WWUCKODZANGQ3AGSFQJ46BB2DIOIL7RHBCAFTP6QPQ, seg_pos={0 0}, upload=332684]: Done uploading piece. err: ecclient: upload failed (node:1MrdHwiDVe8m6wcaQJrYetza9qckLruJXiU9UYbMxGCSWWMV1S, address:145.79.227.42:4125): protocol: rpc: dial tcp 145.79.227.42:4125: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:02:21.041787281 (297h2m18.638558301s) [seg_pos={0 0}, upload=332684]: Done waiting for segment. successful: 7 optimal: 13 errs: <nil> Mar 18 20:02:21.063983330 (297h2m18.660754363s) [upload=332684]: Done uploading segments... Mar 18 20:02:21.064049163 (297h2m18.660820197s) [upload=332684]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 7) Duration: 30.026476083s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 45 out of 60 is failed (threshold=6) [...........XXXXXXXXX.XXX...XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 5.001484078s trace: https://jaeger.dev.storj.io/trace/18f5c0c6bce1ce48 FAIL (2026-03-18 20:02:18 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001608677s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 6 out of 60 is failed (threshold=3) [.............................X..........XX..X....X........X.] duration: 15.615997591s trace: https://jaeger.dev.storj.io/trace/589419efc52e5e2b FAIL (2026-03-18 20:02:15 UTC) operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== Mar 18 20:00:59.343084966 (297h0m56.939855989s) [upload=332666]: Starting upload Mar 18 20:00:59.343175518 (297h0m56.939946548s) [upload=332666]: Done starting upload Mar 18 20:00:59.343565585 (297h0m56.940336612s) [upload=332666]: Uploading segments... Mar 18 20:00:59.344256647 (297h0m56.941027676s) [upload=332666]: Got next segment. Inline: false Mar 18 20:00:59.344352215 (297h0m56.941123241s) [upload=332666]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 20:00:59.344358706 (297h0m56.941129732s) [upload=332666]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 18 20:01:02.340259299 (297h0m59.937030329s) [seg_pos={0 0}, upload=332666]: Begin upload segment... Mar 18 20:01:02.340368743 (297h0m59.937139767s) [seg_pos={0 0}, upload=332666]: Done begin upload segment. Mar 18 20:01:02.340479563 (297h0m59.93725059s) [noise=true , address=45.91.116.8:4061 , piece_id=6XFVKT4AAO7QN457OQMRKPI6R24TJTAN47UXLEVOEFQVOS6M6C3Q, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.340523837 (297h0m59.93729486s) [noise=true , address=79.127.205.233:4166 , piece_id=SFGK5YCWHFRVXF6QNHYP2CYJCZLSTQ76OYRVJPWWXQTF2KFSFJKA, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.341468438 (297h0m59.938239447s) [noise=true , address=45.91.116.4:4961 , piece_id=ZMUGQECUVPVHDNKN75XPBET5M2HFXXK4WMTP4V5KEK63JVZCX5EA, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.341520025 (297h0m59.938291058s) [noise=true , address=167.17.72.71:4815 , piece_id=CXGTYNNBILN3TT7I6Z7UI75F2ZCS4QUTLS6KC4FJ5IHS3XOYWDJQ, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.342232992 (297h0m59.939004023s) [noise=true , address=145.79.227.36:4524 , piece_id=3TAV3P55ITTHBAZEUUUTOAHJRGQXJXXVGZJP6AVR2ANVNOR3S54Q, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.342917368 (297h0m59.939688398s) [noise=true , address=193.149.164.71:4297 , piece_id=4IJPXLPCX3LKFOBOSRHK2DTNRKH67K7FXJUKQ3JPRWHS47ZFEIDA, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.343244957 (297h0m59.940015982s) [noise=true , address=145.79.227.33:4892 , piece_id=DQ6ALC3VOAGFIJI3IKR6I7QFYM22C66QWAAXMH6NXI2FIQEZNQYA, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.344270079 (297h0m59.941041117s) [noise=true , address=145.79.227.21:4497 , piece_id=SP6EKJB7W4QXF7OIBVKEFWCPOZFL5IRIEDVAGS2M4QJSO34CUVRQ, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.345025057 (297h0m59.941796084s) [noise=true , address=77.90.31.17:4793 , piece_id=2U4IBUA2GGUGDBXQGIGQKUK4ZR3L7CDKC6MTGIKAUME4F7MFZRLA, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.345115802 (297h0m59.941886836s) [noise=true , address=167.17.72.69:4731 , piece_id=VOEVIKKFJMC2Y3TPQIDMYUP37NLFU7KPRH4HRRSMQLWUKTSLOH2Q, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.345690600 (297h0m59.942461634s) [noise=true , address=77.90.31.13:4648 , piece_id=BPMB5CFOYHFT3BS4ORR6JF6BNFC2AOAJZOSPMG67NAMLGGP5XVEQ, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.346311592 (297h0m59.943082623s) [noise=true , address=79.127.205.234:4202 , piece_id=YY4LYWX26O6RIOBONESNKTUZEFZCRL4UUX3NEBQ53AJV4IEDG5MQ, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.346791142 (297h0m59.943562166s) [noise=true , address=45.91.116.10:4110 , piece_id=UYLMDCWJEJOHGICA37KNH7G5AQXNIPRH5NRYLBUUMGDRVSLTEYHQ, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.347094999 (297h0m59.94386607s) [noise=true , address=145.79.227.17:4987 , piece_id=PT3TWVB65JZFEWVH4IUZEU2N53FQ57QBRWVLWAUFP4UDLY3LWNLQ, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.347781281 (297h0m59.944552305s) [noise=true , address=79.127.219.46:4919 , piece_id=KK6HRIVBMYLIHRHGPQH3AWMNSVHTTIBESHDMJZA72T446RX6QKRQ, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.348263123 (297h0m59.945034137s) [noise=true , address=193.149.164.68:4033 , piece_id=JZPQKPYGO3W3RZZBITQ73X2XEPIBBSLGRMWIVOM4Z3GTV5UFKRTQ, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.348915505 (297h0m59.945686536s) [noise=true , address=145.79.227.7:4312 , piece_id=2FAL6E3RMBTBHZBXAJJVH2N5VGF5VYI3KNZXGNWCMWGBBPAUO2NQ, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.349366298 (297h0m59.946137337s) [noise=true , address=193.149.164.67:4007 , piece_id=HURQGH67I5B6V4YUV3PRZL3KSKN3XTSK3UMCHZRNT6L3W3PXXHPQ, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.429834098 (297h1m0.026605126s) [upload=332666]: Next returned nil segment Mar 18 20:01:02.429853411 (297h1m0.026624442s) [upload=332666]: Waiting for error group managing segments... Mar 18 20:01:02.446867680 (297h1m0.043638704s) [noise=true , address=45.91.116.10:4110 , piece_id=UYLMDCWJEJOHGICA37KNH7G5AQXNIPRH5NRYLBUUMGDRVSLTEYHQ, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil> Mar 18 20:01:02.451665165 (297h1m0.048436196s) [noise=true , address=45.91.116.4:4961 , piece_id=ZMUGQECUVPVHDNKN75XPBET5M2HFXXK4WMTP4V5KEK63JVZCX5EA, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil> Mar 18 20:01:02.452538886 (297h1m0.049309913s) [noise=true , address=77.90.31.17:4793 , piece_id=2U4IBUA2GGUGDBXQGIGQKUK4ZR3L7CDKC6MTGIKAUME4F7MFZRLA, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil> Mar 18 20:01:02.455597040 (297h1m0.052368067s) [noise=true , address=45.91.116.8:4061 , piece_id=6XFVKT4AAO7QN457OQMRKPI6R24TJTAN47UXLEVOEFQVOS6M6C3Q, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil> Mar 18 20:01:02.457848739 (297h1m0.054619766s) [noise=true , address=77.90.31.13:4648 , piece_id=BPMB5CFOYHFT3BS4ORR6JF6BNFC2AOAJZOSPMG67NAMLGGP5XVEQ, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil> Mar 18 20:01:02.461479709 (297h1m0.058250746s) [noise=true , address=79.127.219.46:4919 , piece_id=KK6HRIVBMYLIHRHGPQH3AWMNSVHTTIBESHDMJZA72T446RX6QKRQ, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil> Mar 18 20:01:02.676859466 (297h1m0.27363049s) [noise=true , address=167.17.72.69:4731 , piece_id=VOEVIKKFJMC2Y3TPQIDMYUP37NLFU7KPRH4HRRSMQLWUKTSLOH2Q, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil> Mar 18 20:01:02.909860743 (297h1m0.506631766s) [noise=true , address=79.127.205.234:4202 , piece_id=YY4LYWX26O6RIOBONESNKTUZEFZCRL4UUX3NEBQ53AJV4IEDG5MQ, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil> Mar 18 20:01:02.930427742 (297h1m0.527198754s) [noise=true , address=193.149.164.71:4297 , piece_id=4IJPXLPCX3LKFOBOSRHK2DTNRKH67K7FXJUKQ3JPRWHS47ZFEIDA, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil> Mar 18 20:01:02.944696889 (297h1m0.541467914s) [noise=true , address=193.149.164.67:4007 , piece_id=HURQGH67I5B6V4YUV3PRZL3KSKN3XTSK3UMCHZRNT6L3W3PXXHPQ, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil> Mar 18 20:01:02.951319183 (297h1m0.548090212s) [noise=true , address=193.149.164.68:4033 , piece_id=JZPQKPYGO3W3RZZBITQ73X2XEPIBBSLGRMWIVOM4Z3GTV5UFKRTQ, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil> Mar 18 20:01:02.975619436 (297h1m0.572390467s) [noise=true , address=79.127.205.233:4166 , piece_id=SFGK5YCWHFRVXF6QNHYP2CYJCZLSTQ76OYRVJPWWXQTF2KFSFJKA, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil> Mar 18 20:01:03.819918528 (297h1m1.416689564s) [noise=true , address=167.17.72.71:4815 , piece_id=CXGTYNNBILN3TT7I6Z7UI75F2ZCS4QUTLS6KC4FJ5IHS3XOYWDJQ, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil> Mar 18 20:01:59.307873420 (297h1m56.904644452s) [noise=true , address=145.79.227.7:4312 , piece_id=2FAL6E3RMBTBHZBXAJJVH2N5VGF5VYI3KNZXGNWCMWGBBPAUO2NQ, seg_pos={0 0}, upload=332666]: Done uploading piece. err: ecclient: upload failed (node:12AVp3qn5uwLPqYaZEwzcgoGMJhNga1BUh6bK5k2bxjWgZLQiiC, address:145.79.227.7:4312): protocol: rpc: dial tcp 145.79.227.7:4312: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:01:59.308026092 (297h1m56.904797124s) [noise=true , address=145.79.227.17:4987 , piece_id=PT3TWVB65JZFEWVH4IUZEU2N53FQ57QBRWVLWAUFP4UDLY3LWNLQ, seg_pos={0 0}, upload=332666]: Done uploading piece. err: ecclient: upload failed (node:1XsPVNGYUhKnXkQRfVfjyBRzyvhShSdsrnzXtwe4Xtf5MaHMvH, address:145.79.227.17:4987): protocol: rpc: dial tcp 145.79.227.17:4987: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:01:59.308195864 (297h1m56.904966894s) [noise=true , address=145.79.227.36:4524 , piece_id=3TAV3P55ITTHBAZEUUUTOAHJRGQXJXXVGZJP6AVR2ANVNOR3S54Q, seg_pos={0 0}, upload=332666]: Done uploading piece. err: ecclient: upload failed (node:1xfbCzjSwFDRrR1gPggeN8tNCLbRRWDVzCJ6mqhZsbVSe8d4s4, address:145.79.227.36:4524): protocol: rpc: dial tcp 145.79.227.36:4524: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:01:59.308316037 (297h1m56.90508706s) [noise=true , address=145.79.227.33:4892 , piece_id=DQ6ALC3VOAGFIJI3IKR6I7QFYM22C66QWAAXMH6NXI2FIQEZNQYA, seg_pos={0 0}, upload=332666]: Done uploading piece. err: ecclient: upload failed (node:12RHwFxcUjPsn12uHEAH3fBDJKv1v1tM8mRJXoFrZrG5djdswF5, address:145.79.227.33:4892): protocol: rpc: dial tcp 145.79.227.33:4892: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:01:59.308404403 (297h1m56.905175433s) [noise=true , address=145.79.227.21:4497 , piece_id=SP6EKJB7W4QXF7OIBVKEFWCPOZFL5IRIEDVAGS2M4QJSO34CUVRQ, seg_pos={0 0}, upload=332666]: Done uploading piece. err: ecclient: upload failed (node:12ZBe4we7VeFdthqRxyy1V9uQrA4r8ouNX68xqsk5bD26dkf2Gd, address:145.79.227.21:4497): protocol: context deadline exceeded; context deadline exceeded Mar 18 20:01:59.308437457 (297h1m56.90520847s) [seg_pos={0 0}, upload=332666]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 18 20:01:59.338288467 (297h1m56.93505949s) [upload=332666]: Done uploading segments... Mar 18 20:01:59.338335497 (297h1m56.935106523s) [upload=332666]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 15.61603191s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 6 out of 60 is failed (threshold=3) [............................XXX..........X...X.........X....] duration: 13.815347464s trace: https://jaeger.dev.storj.io/trace/7c859732d77a7da7 FAIL (2026-03-18 20:01:59 UTC) operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== d=332621]: Uploading segments... Mar 18 19:58:34.964515697 (296h58m32.561286706s) [upload=332621]: Got next segment. Inline: false Mar 18 19:58:34.964555310 (296h58m32.561326334s) [upload=332621]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 19:58:34.964560227 (296h58m32.561331251s) [upload=332621]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 18 19:58:38.382666205 (296h58m35.979437219s) [seg_pos={0 0}, upload=332621]: Begin upload segment... Mar 18 19:58:38.382729299 (296h58m35.979500329s) [seg_pos={0 0}, upload=332621]: Done begin upload segment. Mar 18 19:58:38.382760799 (296h58m35.979531826s) [noise=true , address=145.79.227.46:4607 , piece_id=LXBA2IGR4U6DLXE4NXL2CN23X5HXP7KFN5UCJZH6IBRYAZCQ7N3Q, seg_pos={0 0}, upload=332621]: Uploading piece... Mar 18 19:58:38.382881088 (296h58m35.979652115s) [noise=true , address=145.79.227.22:4538 , piece_id=ALPGNVFU5TWXZ7D7I5XFQBZ2CAN74VYUBNYDFSPG3HIHO7Z6M3AA, seg_pos={0 0}, upload=332621]: Uploading piece... Mar 18 19:58:38.383461716 (296h58m35.980232746s) [noise=true , address=167.17.72.70:4770 , piece_id=5H5RNTJDXOLXRNVU4XG5FP4O5YN5PW2XQH4O2UIX5XBKNMO35WKQ, seg_pos={0 0}, upload=332621]: Uploading piece... Mar 18 19:58:38.384266580 (296h58m35.981037606s) [noise=true , address=45.91.116.10:4092 , piece_id=Y7E2ZWR5D6M7QKRY3I7SL3ZJN32EK36ZWPP3YRW5WCGQFZCNU64Q, seg_pos={0 0}, upload=332621]: Uploading piece... Mar 18 19:58:38.384321708 (296h58m35.981092731s) [noise=true , address=79.127.205.235:4273 , piece_id=CQEOZ6GWLY5MEVRQZHYH6AP6DXQY5WKIAHCHDBJMAI77STRNINYA, seg_pos={0 0}, upload=332621]: Uploading piece... Mar 18 19:58:38.385197892 (296h58m35.981968918s) [noise=true , address=193.149.164.66:4989 , piece_id=EDE7OF4G36VJ7GXFMF47ZI73D6I7OWZSC5AXESS6W74ZDWTBDXWA, seg_pos={0 0}, upload=332621]: Uploading piece... Mar 18 19:58:38.385232613 (296h58m35.982003642s) [noise=true , address=45.91.116.13:4171 , piece_id=74JRIA5AS6SMECS3FFCCVLH3AY5N4EPNWOEDZGITBPCQZEKDGEAQ, seg_pos={0 0}, upload=332621]: Uploading piece... Mar 18 19:58:38.385334543 (296h58m35.982105567s) [noise=true , address=193.149.164.70:4120 , piece_id=AO6H5CLNNY2CQ27JO57WEKF7OTFQA7IXAZMVG6A4EWJLZT7NIRSQ, seg_pos={0 0}, upload=332621]: Uploading piece... Mar 18 19:58:38.386120289 (296h58m35.982891315s) [noise=true , address=79.127.219.46:4908 , piece_id=L2EOXVK7S2ZSQNK47543XTNRWJP2PU57DH2XHZ2BO5OZUTDXDQFA, seg_pos={0 0}, upload=332621]: Uploading piece... Mar 18 19:58:38.386244430 (296h58m35.983015464s) [noise=true , address=79.127.205.237:4236 , piece_id=FKQ3NUK5AUFD35L25PJFPVDLZUVNRURK5RLSPG3GGC75QYTEWP4A, seg_pos={0 0}, upload=332621]: Uploading piece... Mar 18 19:58:38.387158310 (296h58m35.983929342s) [noise=true , address=64.49.13.140:4516 , piece_id=D2V5WAW6D46VSSNBFGNO5ETQNCOTDHQYWM62BSSVF53X2MWU3TYQ, seg_pos={0 0}, upload=332621]: Uploading piece... Mar 18 19:58:38.387233109 (296h58m35.984004132s) [noise=true , address=193.149.164.67:4020 , piece_id=NZQGIA6BVAK3Z6SQZA7WNAOAY25VQKBDTWCRV34KNSYTGVNDSWUA, seg_pos={0 0}, upload=332621]: Uploading piece... Mar 18 19:58:38.388077887 (296h58m35.984848921s) [noise=true , address=145.79.227.49:4343 , piece_id=NCJH6BOJHPW5OTSCNGKTZCZYC3F3LTRJKGVOKCJODWSMHUAATJQQ, seg_pos={0 0}, upload=332621]: Uploading piece... Mar 18 19:58:38.388338947 (296h58m35.985109978s) [noise=true , address=79.127.219.41:4940 , piece_id=C2ODTVQIR73WTRBM7EDKEC54DOBK67L2RV4UXPHHTRY6KL3EQIKQ, seg_pos={0 0}, upload=332621]: Uploading piece... Mar 18 19:58:38.388792743 (296h58m35.985563766s) [noise=true , address=145.79.227.39:4043 , piece_id=D757XSQF27CMMI5UPBRB7LLGISWOYLW7OS3UICYF3U6AI62X73PQ, seg_pos={0 0}, upload=332621]: Uploading piece... Mar 18 19:58:38.389305068 (296h58m35.986076097s) [noise=true , address=45.91.116.9:4280 , piece_id=QECGXDF7C5YUJNURQCU76PMJM5MZ3DCEGPGAJHCZWI3H3OVGERGQ, seg_pos={0 0}, upload=332621]: Uploading piece... Mar 18 19:58:38.389771675 (296h58m35.986542707s) [noise=true , address=145.79.227.36:4528 , piece_id=FJ5LKOZJYVNWLZ5I6OUCZMPFDOBVZXQRVPHREHPQX6J3YWAI2KFA, seg_pos={0 0}, upload=332621]: Uploading piece... Mar 18 19:58:38.390707637 (296h58m35.98747867s) [noise=true , address=77.90.31.18:4827 , piece_id=SGKGAEAHEHU2H4YRLYQLQMTS45U6ZNSAYJ6CYJUGGKEDRMQT6BHQ, seg_pos={0 0}, upload=332621]: Uploading piece... Mar 18 19:58:38.466570556 (296h58m36.063341572s) [upload=332621]: Next returned nil segment Mar 18 19:58:38.466587006 (296h58m36.063358039s) [upload=332621]: Waiting for error group managing segments... Mar 18 19:58:38.471637588 (296h58m36.068408605s) [noise=true , address=45.91.116.9:4280 , piece_id=QECGXDF7C5YUJNURQCU76PMJM5MZ3DCEGPGAJHCZWI3H3OVGERGQ, seg_pos={0 0}, upload=332621]: Done uploading piece. err: <nil> Mar 18 19:58:38.474621081 (296h58m36.071392109s) [noise=true , address=79.127.219.41:4940 , piece_id=C2ODTVQIR73WTRBM7EDKEC54DOBK67L2RV4UXPHHTRY6KL3EQIKQ, seg_pos={0 0}, upload=332621]: Done uploading piece. err: <nil> Mar 18 19:58:38.479487991 (296h58m36.076259007s) [noise=true , address=77.90.31.18:4827 , piece_id=SGKGAEAHEHU2H4YRLYQLQMTS45U6ZNSAYJ6CYJUGGKEDRMQT6BHQ, seg_pos={0 0}, upload=332621]: Done uploading piece. err: <nil> Mar 18 19:58:38.479685133 (296h58m36.076456159s) [noise=true , address=45.91.116.10:4092 , piece_id=Y7E2ZWR5D6M7QKRY3I7SL3ZJN32EK36ZWPP3YRW5WCGQFZCNU64Q, seg_pos={0 0}, upload=332621]: Done uploading piece. err: <nil> Mar 18 19:58:38.485438319 (296h58m36.082209345s) [noise=true , address=45.91.116.13:4171 , piece_id=74JRIA5AS6SMECS3FFCCVLH3AY5N4EPNWOEDZGITBPCQZEKDGEAQ, seg_pos={0 0}, upload=332621]: Done uploading piece. err: <nil> Mar 18 19:58:38.620281221 (296h58m36.217052236s) [noise=true , address=79.127.219.46:4908 , piece_id=L2EOXVK7S2ZSQNK47543XTNRWJP2PU57DH2XHZ2BO5OZUTDXDQFA, seg_pos={0 0}, upload=332621]: Done uploading piece. err: <nil> Mar 18 19:58:38.772264604 (296h58m36.369035618s) [noise=true , address=167.17.72.70:4770 , piece_id=5H5RNTJDXOLXRNVU4XG5FP4O5YN5PW2XQH4O2UIX5XBKNMO35WKQ, seg_pos={0 0}, upload=332621]: Done uploading piece. err: <nil> Mar 18 19:58:38.967982420 (296h58m36.564753435s) [noise=true , address=79.127.205.235:4273 , piece_id=CQEOZ6GWLY5MEVRQZHYH6AP6DXQY5WKIAHCHDBJMAI77STRNINYA, seg_pos={0 0}, upload=332621]: Done uploading piece. err: <nil> Mar 18 19:58:38.977759060 (296h58m36.57453009s) [noise=true , address=79.127.205.237:4236 , piece_id=FKQ3NUK5AUFD35L25PJFPVDLZUVNRURK5RLSPG3GGC75QYTEWP4A, seg_pos={0 0}, upload=332621]: Done uploading piece. err: <nil> Mar 18 19:58:39.005320695 (296h58m36.602091708s) [noise=true , address=193.149.164.70:4120 , piece_id=AO6H5CLNNY2CQ27JO57WEKF7OTFQA7IXAZMVG6A4EWJLZT7NIRSQ, seg_pos={0 0}, upload=332621]: Done uploading piece. err: <nil> Mar 18 19:58:39.025848813 (296h58m36.622619842s) [noise=true , address=193.149.164.67:4020 , piece_id=NZQGIA6BVAK3Z6SQZA7WNAOAY25VQKBDTWCRV34KNSYTGVNDSWUA, seg_pos={0 0}, upload=332621]: Done uploading piece. err: <nil> Mar 18 19:58:39.065107032 (296h58m36.661878045s) [noise=true , address=64.49.13.140:4516 , piece_id=D2V5WAW6D46VSSNBFGNO5ETQNCOTDHQYWM62BSSVF53X2MWU3TYQ, seg_pos={0 0}, upload=332621]: Done uploading piece. err: <nil> Mar 18 19:58:39.201750353 (296h58m36.798521377s) [noise=true , address=193.149.164.66:4989 , piece_id=EDE7OF4G36VJ7GXFMF47ZI73D6I7OWZSC5AXESS6W74ZDWTBDXWA, seg_pos={0 0}, upload=332621]: Done uploading piece. err: <nil> Mar 18 19:59:34.932832250 (296h59m32.529603284s) [noise=true , address=145.79.227.46:4607 , piece_id=LXBA2IGR4U6DLXE4NXL2CN23X5HXP7KFN5UCJZH6IBRYAZCQ7N3Q, seg_pos={0 0}, upload=332621]: Done uploading piece. err: ecclient: upload failed (node:1zDoLfA5bQWM7Uf479gfFVjG3yYvxG5EZS5yZSFGTvoFoPYEAo, address:145.79.227.46:4607): protocol: rpc: dial tcp 145.79.227.46:4607: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 19:59:34.932941758 (296h59m32.529712814s) [noise=true , address=145.79.227.22:4538 , piece_id=ALPGNVFU5TWXZ7D7I5XFQBZ2CAN74VYUBNYDFSPG3HIHO7Z6M3AA, seg_pos={0 0}, upload=332621]: Done uploading piece. err: ecclient: upload failed (node:1kNFzPpkDtwy4VFyzeFQReorJPEVsmdDAd8ULT8ZKoGde57P3v, address:145.79.227.22:4538): protocol: rpc: dial tcp 145.79.227.22:4538: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 19:59:34.933267818 (296h59m32.53003887s) [noise=true , address=145.79.227.49:4343 , piece_id=NCJH6BOJHPW5OTSCNGKTZCZYC3F3LTRJKGVOKCJODWSMHUAATJQQ, seg_pos={0 0}, upload=332621]: Done uploading piece. err: ecclient: upload failed (node:16EsEMRW53sjKVag2WpSDGLhELyE2bSXyW4aAurCvDFKRNkXuf, address:145.79.227.49:4343): protocol: rpc: dial tcp 145.79.227.49:4343: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 19:59:34.933359674 (296h59m32.530130694s) [noise=true , address=145.79.227.36:4528 , piece_id=FJ5LKOZJYVNWLZ5I6OUCZMPFDOBVZXQRVPHREHPQX6J3YWAI2KFA, seg_pos={0 0}, upload=332621]: Done uploading piece. err: ecclient: upload failed (node:12ArcCrcHdpNgBMfTcPdtTXb65RgnTh8N32ihHAx2RrddxnRYgT, address:145.79.227.36:4528): protocol: rpc: dial tcp 145.79.227.36:4528: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 19:59:34.933622410 (296h59m32.530393477s) [noise=true , address=145.79.227.39:4043 , piece_id=D757XSQF27CMMI5UPBRB7LLGISWOYLW7OS3UICYF3U6AI62X73PQ, seg_pos={0 0}, upload=332621]: Done uploading piece. err: ecclient: upload failed (node:12f97TAyWcE4YrxDQCeycC8tnuetQiagu2MV4U5o8ARLq8jGKTX, address:145.79.227.39:4043): protocol: rpc: dial tcp 145.79.227.39:4043: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 19:59:34.933677971 (296h59m32.530448984s) [seg_pos={0 0}, upload=332621]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 18 19:59:34.955309631 (296h59m32.552080664s) [upload=332621]: Done uploading segments... Mar 18 19:59:34.955416416 (296h59m32.552187448s) [upload=332621]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 13.815384785s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 6 out of 60 is failed (threshold=3) [..............................X..........XX..X....X........X] duration: 1m0.03281677s trace: https://jaeger.dev.storj.io/trace/589419efc52e5e2b FAIL (2026-03-18 20:01:59 UTC) operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== Mar 18 20:00:59.343084966 (297h0m56.939855989s) [upload=332666]: Starting upload Mar 18 20:00:59.343175518 (297h0m56.939946548s) [upload=332666]: Done starting upload Mar 18 20:00:59.343565585 (297h0m56.940336612s) [upload=332666]: Uploading segments... Mar 18 20:00:59.344256647 (297h0m56.941027676s) [upload=332666]: Got next segment. Inline: false Mar 18 20:00:59.344352215 (297h0m56.941123241s) [upload=332666]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 20:00:59.344358706 (297h0m56.941129732s) [upload=332666]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 18 20:01:02.340259299 (297h0m59.937030329s) [seg_pos={0 0}, upload=332666]: Begin upload segment... Mar 18 20:01:02.340368743 (297h0m59.937139767s) [seg_pos={0 0}, upload=332666]: Done begin upload segment. Mar 18 20:01:02.340479563 (297h0m59.93725059s) [noise=true , address=45.91.116.8:4061 , piece_id=6XFVKT4AAO7QN457OQMRKPI6R24TJTAN47UXLEVOEFQVOS6M6C3Q, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.340523837 (297h0m59.93729486s) [noise=true , address=79.127.205.233:4166 , piece_id=SFGK5YCWHFRVXF6QNHYP2CYJCZLSTQ76OYRVJPWWXQTF2KFSFJKA, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.341468438 (297h0m59.938239447s) [noise=true , address=45.91.116.4:4961 , piece_id=ZMUGQECUVPVHDNKN75XPBET5M2HFXXK4WMTP4V5KEK63JVZCX5EA, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.341520025 (297h0m59.938291058s) [noise=true , address=167.17.72.71:4815 , piece_id=CXGTYNNBILN3TT7I6Z7UI75F2ZCS4QUTLS6KC4FJ5IHS3XOYWDJQ, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.342232992 (297h0m59.939004023s) [noise=true , address=145.79.227.36:4524 , piece_id=3TAV3P55ITTHBAZEUUUTOAHJRGQXJXXVGZJP6AVR2ANVNOR3S54Q, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.342917368 (297h0m59.939688398s) [noise=true , address=193.149.164.71:4297 , piece_id=4IJPXLPCX3LKFOBOSRHK2DTNRKH67K7FXJUKQ3JPRWHS47ZFEIDA, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.343244957 (297h0m59.940015982s) [noise=true , address=145.79.227.33:4892 , piece_id=DQ6ALC3VOAGFIJI3IKR6I7QFYM22C66QWAAXMH6NXI2FIQEZNQYA, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.344270079 (297h0m59.941041117s) [noise=true , address=145.79.227.21:4497 , piece_id=SP6EKJB7W4QXF7OIBVKEFWCPOZFL5IRIEDVAGS2M4QJSO34CUVRQ, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.345025057 (297h0m59.941796084s) [noise=true , address=77.90.31.17:4793 , piece_id=2U4IBUA2GGUGDBXQGIGQKUK4ZR3L7CDKC6MTGIKAUME4F7MFZRLA, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.345115802 (297h0m59.941886836s) [noise=true , address=167.17.72.69:4731 , piece_id=VOEVIKKFJMC2Y3TPQIDMYUP37NLFU7KPRH4HRRSMQLWUKTSLOH2Q, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.345690600 (297h0m59.942461634s) [noise=true , address=77.90.31.13:4648 , piece_id=BPMB5CFOYHFT3BS4ORR6JF6BNFC2AOAJZOSPMG67NAMLGGP5XVEQ, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.346311592 (297h0m59.943082623s) [noise=true , address=79.127.205.234:4202 , piece_id=YY4LYWX26O6RIOBONESNKTUZEFZCRL4UUX3NEBQ53AJV4IEDG5MQ, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.346791142 (297h0m59.943562166s) [noise=true , address=45.91.116.10:4110 , piece_id=UYLMDCWJEJOHGICA37KNH7G5AQXNIPRH5NRYLBUUMGDRVSLTEYHQ, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.347094999 (297h0m59.94386607s) [noise=true , address=145.79.227.17:4987 , piece_id=PT3TWVB65JZFEWVH4IUZEU2N53FQ57QBRWVLWAUFP4UDLY3LWNLQ, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.347781281 (297h0m59.944552305s) [noise=true , address=79.127.219.46:4919 , piece_id=KK6HRIVBMYLIHRHGPQH3AWMNSVHTTIBESHDMJZA72T446RX6QKRQ, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.348263123 (297h0m59.945034137s) [noise=true , address=193.149.164.68:4033 , piece_id=JZPQKPYGO3W3RZZBITQ73X2XEPIBBSLGRMWIVOM4Z3GTV5UFKRTQ, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.348915505 (297h0m59.945686536s) [noise=true , address=145.79.227.7:4312 , piece_id=2FAL6E3RMBTBHZBXAJJVH2N5VGF5VYI3KNZXGNWCMWGBBPAUO2NQ, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.349366298 (297h0m59.946137337s) [noise=true , address=193.149.164.67:4007 , piece_id=HURQGH67I5B6V4YUV3PRZL3KSKN3XTSK3UMCHZRNT6L3W3PXXHPQ, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.429834098 (297h1m0.026605126s) [upload=332666]: Next returned nil segment Mar 18 20:01:02.429853411 (297h1m0.026624442s) [upload=332666]: Waiting for error group managing segments... Mar 18 20:01:02.446867680 (297h1m0.043638704s) [noise=true , address=45.91.116.10:4110 , piece_id=UYLMDCWJEJOHGICA37KNH7G5AQXNIPRH5NRYLBUUMGDRVSLTEYHQ, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil> Mar 18 20:01:02.451665165 (297h1m0.048436196s) [noise=true , address=45.91.116.4:4961 , piece_id=ZMUGQECUVPVHDNKN75XPBET5M2HFXXK4WMTP4V5KEK63JVZCX5EA, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil> Mar 18 20:01:02.452538886 (297h1m0.049309913s) [noise=true , address=77.90.31.17:4793 , piece_id=2U4IBUA2GGUGDBXQGIGQKUK4ZR3L7CDKC6MTGIKAUME4F7MFZRLA, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil> Mar 18 20:01:02.455597040 (297h1m0.052368067s) [noise=true , address=45.91.116.8:4061 , piece_id=6XFVKT4AAO7QN457OQMRKPI6R24TJTAN47UXLEVOEFQVOS6M6C3Q, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil> Mar 18 20:01:02.457848739 (297h1m0.054619766s) [noise=true , address=77.90.31.13:4648 , piece_id=BPMB5CFOYHFT3BS4ORR6JF6BNFC2AOAJZOSPMG67NAMLGGP5XVEQ, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil> Mar 18 20:01:02.461479709 (297h1m0.058250746s) [noise=true , address=79.127.219.46:4919 , piece_id=KK6HRIVBMYLIHRHGPQH3AWMNSVHTTIBESHDMJZA72T446RX6QKRQ, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil> Mar 18 20:01:02.676859466 (297h1m0.27363049s) [noise=true , address=167.17.72.69:4731 , piece_id=VOEVIKKFJMC2Y3TPQIDMYUP37NLFU7KPRH4HRRSMQLWUKTSLOH2Q, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil> Mar 18 20:01:02.909860743 (297h1m0.506631766s) [noise=true , address=79.127.205.234:4202 , piece_id=YY4LYWX26O6RIOBONESNKTUZEFZCRL4UUX3NEBQ53AJV4IEDG5MQ, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil> Mar 18 20:01:02.930427742 (297h1m0.527198754s) [noise=true , address=193.149.164.71:4297 , piece_id=4IJPXLPCX3LKFOBOSRHK2DTNRKH67K7FXJUKQ3JPRWHS47ZFEIDA, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil> Mar 18 20:01:02.944696889 (297h1m0.541467914s) [noise=true , address=193.149.164.67:4007 , piece_id=HURQGH67I5B6V4YUV3PRZL3KSKN3XTSK3UMCHZRNT6L3W3PXXHPQ, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil> Mar 18 20:01:02.951319183 (297h1m0.548090212s) [noise=true , address=193.149.164.68:4033 , piece_id=JZPQKPYGO3W3RZZBITQ73X2XEPIBBSLGRMWIVOM4Z3GTV5UFKRTQ, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil> Mar 18 20:01:02.975619436 (297h1m0.572390467s) [noise=true , address=79.127.205.233:4166 , piece_id=SFGK5YCWHFRVXF6QNHYP2CYJCZLSTQ76OYRVJPWWXQTF2KFSFJKA, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil> Mar 18 20:01:03.819918528 (297h1m1.416689564s) [noise=true , address=167.17.72.71:4815 , piece_id=CXGTYNNBILN3TT7I6Z7UI75F2ZCS4QUTLS6KC4FJ5IHS3XOYWDJQ, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil> Mar 18 20:01:59.307873420 (297h1m56.904644452s) [noise=true , address=145.79.227.7:4312 , piece_id=2FAL6E3RMBTBHZBXAJJVH2N5VGF5VYI3KNZXGNWCMWGBBPAUO2NQ, seg_pos={0 0}, upload=332666]: Done uploading piece. err: ecclient: upload failed (node:12AVp3qn5uwLPqYaZEwzcgoGMJhNga1BUh6bK5k2bxjWgZLQiiC, address:145.79.227.7:4312): protocol: rpc: dial tcp 145.79.227.7:4312: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:01:59.308026092 (297h1m56.904797124s) [noise=true , address=145.79.227.17:4987 , piece_id=PT3TWVB65JZFEWVH4IUZEU2N53FQ57QBRWVLWAUFP4UDLY3LWNLQ, seg_pos={0 0}, upload=332666]: Done uploading piece. err: ecclient: upload failed (node:1XsPVNGYUhKnXkQRfVfjyBRzyvhShSdsrnzXtwe4Xtf5MaHMvH, address:145.79.227.17:4987): protocol: rpc: dial tcp 145.79.227.17:4987: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:01:59.308195864 (297h1m56.904966894s) [noise=true , address=145.79.227.36:4524 , piece_id=3TAV3P55ITTHBAZEUUUTOAHJRGQXJXXVGZJP6AVR2ANVNOR3S54Q, seg_pos={0 0}, upload=332666]: Done uploading piece. err: ecclient: upload failed (node:1xfbCzjSwFDRrR1gPggeN8tNCLbRRWDVzCJ6mqhZsbVSe8d4s4, address:145.79.227.36:4524): protocol: rpc: dial tcp 145.79.227.36:4524: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:01:59.308316037 (297h1m56.90508706s) [noise=true , address=145.79.227.33:4892 , piece_id=DQ6ALC3VOAGFIJI3IKR6I7QFYM22C66QWAAXMH6NXI2FIQEZNQYA, seg_pos={0 0}, upload=332666]: Done uploading piece. err: ecclient: upload failed (node:12RHwFxcUjPsn12uHEAH3fBDJKv1v1tM8mRJXoFrZrG5djdswF5, address:145.79.227.33:4892): protocol: rpc: dial tcp 145.79.227.33:4892: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:01:59.308404403 (297h1m56.905175433s) [noise=true , address=145.79.227.21:4497 , piece_id=SP6EKJB7W4QXF7OIBVKEFWCPOZFL5IRIEDVAGS2M4QJSO34CUVRQ, seg_pos={0 0}, upload=332666]: Done uploading piece. err: ecclient: upload failed (node:12ZBe4we7VeFdthqRxyy1V9uQrA4r8ouNX68xqsk5bD26dkf2Gd, address:145.79.227.21:4497): protocol: context deadline exceeded; context deadline exceeded Mar 18 20:01:59.308437457 (297h1m56.90520847s) [seg_pos={0 0}, upload=332666]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 18 20:01:59.338288467 (297h1m56.93505949s) [upload=332666]: Done uploading segments... Mar 18 20:01:59.338335497 (297h1m56.935106523s) [upload=332666]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 1m0.03284927s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 44 out of 60 is failed (threshold=6) [............XXXXXXXXX.XXX...XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 5.001378969s trace: https://jaeger.dev.storj.io/trace/7ce32e09b0e6713b FAIL (2026-03-18 20:01:54 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001404583s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 6 out of 60 is failed (threshold=3) [.............................XXX..........X...X.........X...] duration: 15.912762781s trace: https://jaeger.dev.storj.io/trace/7c859732d77a7da7 FAIL (2026-03-18 20:01:34 UTC) operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== d=332621]: Uploading segments... Mar 18 19:58:34.964515697 (296h58m32.561286706s) [upload=332621]: Got next segment. Inline: false Mar 18 19:58:34.964555310 (296h58m32.561326334s) [upload=332621]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 19:58:34.964560227 (296h58m32.561331251s) [upload=332621]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 18 19:58:38.382666205 (296h58m35.979437219s) [seg_pos={0 0}, upload=332621]: Begin upload segment... Mar 18 19:58:38.382729299 (296h58m35.979500329s) [seg_pos={0 0}, upload=332621]: Done begin upload segment. Mar 18 19:58:38.382760799 (296h58m35.979531826s) [noise=true , address=145.79.227.46:4607 , piece_id=LXBA2IGR4U6DLXE4NXL2CN23X5HXP7KFN5UCJZH6IBRYAZCQ7N3Q, seg_pos={0 0}, upload=332621]: Uploading piece... Mar 18 19:58:38.382881088 (296h58m35.979652115s) [noise=true , address=145.79.227.22:4538 , piece_id=ALPGNVFU5TWXZ7D7I5XFQBZ2CAN74VYUBNYDFSPG3HIHO7Z6M3AA, seg_pos={0 0}, upload=332621]: Uploading piece... Mar 18 19:58:38.383461716 (296h58m35.980232746s) [noise=true , address=167.17.72.70:4770 , piece_id=5H5RNTJDXOLXRNVU4XG5FP4O5YN5PW2XQH4O2UIX5XBKNMO35WKQ, seg_pos={0 0}, upload=332621]: Uploading piece... Mar 18 19:58:38.384266580 (296h58m35.981037606s) [noise=true , address=45.91.116.10:4092 , piece_id=Y7E2ZWR5D6M7QKRY3I7SL3ZJN32EK36ZWPP3YRW5WCGQFZCNU64Q, seg_pos={0 0}, upload=332621]: Uploading piece... Mar 18 19:58:38.384321708 (296h58m35.981092731s) [noise=true , address=79.127.205.235:4273 , piece_id=CQEOZ6GWLY5MEVRQZHYH6AP6DXQY5WKIAHCHDBJMAI77STRNINYA, seg_pos={0 0}, upload=332621]: Uploading piece... Mar 18 19:58:38.385197892 (296h58m35.981968918s) [noise=true , address=193.149.164.66:4989 , piece_id=EDE7OF4G36VJ7GXFMF47ZI73D6I7OWZSC5AXESS6W74ZDWTBDXWA, seg_pos={0 0}, upload=332621]: Uploading piece... Mar 18 19:58:38.385232613 (296h58m35.982003642s) [noise=true , address=45.91.116.13:4171 , piece_id=74JRIA5AS6SMECS3FFCCVLH3AY5N4EPNWOEDZGITBPCQZEKDGEAQ, seg_pos={0 0}, upload=332621]: Uploading piece... Mar 18 19:58:38.385334543 (296h58m35.982105567s) [noise=true , address=193.149.164.70:4120 , piece_id=AO6H5CLNNY2CQ27JO57WEKF7OTFQA7IXAZMVG6A4EWJLZT7NIRSQ, seg_pos={0 0}, upload=332621]: Uploading piece... Mar 18 19:58:38.386120289 (296h58m35.982891315s) [noise=true , address=79.127.219.46:4908 , piece_id=L2EOXVK7S2ZSQNK47543XTNRWJP2PU57DH2XHZ2BO5OZUTDXDQFA, seg_pos={0 0}, upload=332621]: Uploading piece... Mar 18 19:58:38.386244430 (296h58m35.983015464s) [noise=true , address=79.127.205.237:4236 , piece_id=FKQ3NUK5AUFD35L25PJFPVDLZUVNRURK5RLSPG3GGC75QYTEWP4A, seg_pos={0 0}, upload=332621]: Uploading piece... Mar 18 19:58:38.387158310 (296h58m35.983929342s) [noise=true , address=64.49.13.140:4516 , piece_id=D2V5WAW6D46VSSNBFGNO5ETQNCOTDHQYWM62BSSVF53X2MWU3TYQ, seg_pos={0 0}, upload=332621]: Uploading piece... Mar 18 19:58:38.387233109 (296h58m35.984004132s) [noise=true , address=193.149.164.67:4020 , piece_id=NZQGIA6BVAK3Z6SQZA7WNAOAY25VQKBDTWCRV34KNSYTGVNDSWUA, seg_pos={0 0}, upload=332621]: Uploading piece... Mar 18 19:58:38.388077887 (296h58m35.984848921s) [noise=true , address=145.79.227.49:4343 , piece_id=NCJH6BOJHPW5OTSCNGKTZCZYC3F3LTRJKGVOKCJODWSMHUAATJQQ, seg_pos={0 0}, upload=332621]: Uploading piece... Mar 18 19:58:38.388338947 (296h58m35.985109978s) [noise=true , address=79.127.219.41:4940 , piece_id=C2ODTVQIR73WTRBM7EDKEC54DOBK67L2RV4UXPHHTRY6KL3EQIKQ, seg_pos={0 0}, upload=332621]: Uploading piece... Mar 18 19:58:38.388792743 (296h58m35.985563766s) [noise=true , address=145.79.227.39:4043 , piece_id=D757XSQF27CMMI5UPBRB7LLGISWOYLW7OS3UICYF3U6AI62X73PQ, seg_pos={0 0}, upload=332621]: Uploading piece... Mar 18 19:58:38.389305068 (296h58m35.986076097s) [noise=true , address=45.91.116.9:4280 , piece_id=QECGXDF7C5YUJNURQCU76PMJM5MZ3DCEGPGAJHCZWI3H3OVGERGQ, seg_pos={0 0}, upload=332621]: Uploading piece... Mar 18 19:58:38.389771675 (296h58m35.986542707s) [noise=true , address=145.79.227.36:4528 , piece_id=FJ5LKOZJYVNWLZ5I6OUCZMPFDOBVZXQRVPHREHPQX6J3YWAI2KFA, seg_pos={0 0}, upload=332621]: Uploading piece... Mar 18 19:58:38.390707637 (296h58m35.98747867s) [noise=true , address=77.90.31.18:4827 , piece_id=SGKGAEAHEHU2H4YRLYQLQMTS45U6ZNSAYJ6CYJUGGKEDRMQT6BHQ, seg_pos={0 0}, upload=332621]: Uploading piece... Mar 18 19:58:38.466570556 (296h58m36.063341572s) [upload=332621]: Next returned nil segment Mar 18 19:58:38.466587006 (296h58m36.063358039s) [upload=332621]: Waiting for error group managing segments... Mar 18 19:58:38.471637588 (296h58m36.068408605s) [noise=true , address=45.91.116.9:4280 , piece_id=QECGXDF7C5YUJNURQCU76PMJM5MZ3DCEGPGAJHCZWI3H3OVGERGQ, seg_pos={0 0}, upload=332621]: Done uploading piece. err: <nil> Mar 18 19:58:38.474621081 (296h58m36.071392109s) [noise=true , address=79.127.219.41:4940 , piece_id=C2ODTVQIR73WTRBM7EDKEC54DOBK67L2RV4UXPHHTRY6KL3EQIKQ, seg_pos={0 0}, upload=332621]: Done uploading piece. err: <nil> Mar 18 19:58:38.479487991 (296h58m36.076259007s) [noise=true , address=77.90.31.18:4827 , piece_id=SGKGAEAHEHU2H4YRLYQLQMTS45U6ZNSAYJ6CYJUGGKEDRMQT6BHQ, seg_pos={0 0}, upload=332621]: Done uploading piece. err: <nil> Mar 18 19:58:38.479685133 (296h58m36.076456159s) [noise=true , address=45.91.116.10:4092 , piece_id=Y7E2ZWR5D6M7QKRY3I7SL3ZJN32EK36ZWPP3YRW5WCGQFZCNU64Q, seg_pos={0 0}, upload=332621]: Done uploading piece. err: <nil> Mar 18 19:58:38.485438319 (296h58m36.082209345s) [noise=true , address=45.91.116.13:4171 , piece_id=74JRIA5AS6SMECS3FFCCVLH3AY5N4EPNWOEDZGITBPCQZEKDGEAQ, seg_pos={0 0}, upload=332621]: Done uploading piece. err: <nil> Mar 18 19:58:38.620281221 (296h58m36.217052236s) [noise=true , address=79.127.219.46:4908 , piece_id=L2EOXVK7S2ZSQNK47543XTNRWJP2PU57DH2XHZ2BO5OZUTDXDQFA, seg_pos={0 0}, upload=332621]: Done uploading piece. err: <nil> Mar 18 19:58:38.772264604 (296h58m36.369035618s) [noise=true , address=167.17.72.70:4770 , piece_id=5H5RNTJDXOLXRNVU4XG5FP4O5YN5PW2XQH4O2UIX5XBKNMO35WKQ, seg_pos={0 0}, upload=332621]: Done uploading piece. err: <nil> Mar 18 19:58:38.967982420 (296h58m36.564753435s) [noise=true , address=79.127.205.235:4273 , piece_id=CQEOZ6GWLY5MEVRQZHYH6AP6DXQY5WKIAHCHDBJMAI77STRNINYA, seg_pos={0 0}, upload=332621]: Done uploading piece. err: <nil> Mar 18 19:58:38.977759060 (296h58m36.57453009s) [noise=true , address=79.127.205.237:4236 , piece_id=FKQ3NUK5AUFD35L25PJFPVDLZUVNRURK5RLSPG3GGC75QYTEWP4A, seg_pos={0 0}, upload=332621]: Done uploading piece. err: <nil> Mar 18 19:58:39.005320695 (296h58m36.602091708s) [noise=true , address=193.149.164.70:4120 , piece_id=AO6H5CLNNY2CQ27JO57WEKF7OTFQA7IXAZMVG6A4EWJLZT7NIRSQ, seg_pos={0 0}, upload=332621]: Done uploading piece. err: <nil> Mar 18 19:58:39.025848813 (296h58m36.622619842s) [noise=true , address=193.149.164.67:4020 , piece_id=NZQGIA6BVAK3Z6SQZA7WNAOAY25VQKBDTWCRV34KNSYTGVNDSWUA, seg_pos={0 0}, upload=332621]: Done uploading piece. err: <nil> Mar 18 19:58:39.065107032 (296h58m36.661878045s) [noise=true , address=64.49.13.140:4516 , piece_id=D2V5WAW6D46VSSNBFGNO5ETQNCOTDHQYWM62BSSVF53X2MWU3TYQ, seg_pos={0 0}, upload=332621]: Done uploading piece. err: <nil> Mar 18 19:58:39.201750353 (296h58m36.798521377s) [noise=true , address=193.149.164.66:4989 , piece_id=EDE7OF4G36VJ7GXFMF47ZI73D6I7OWZSC5AXESS6W74ZDWTBDXWA, seg_pos={0 0}, upload=332621]: Done uploading piece. err: <nil> Mar 18 19:59:34.932832250 (296h59m32.529603284s) [noise=true , address=145.79.227.46:4607 , piece_id=LXBA2IGR4U6DLXE4NXL2CN23X5HXP7KFN5UCJZH6IBRYAZCQ7N3Q, seg_pos={0 0}, upload=332621]: Done uploading piece. err: ecclient: upload failed (node:1zDoLfA5bQWM7Uf479gfFVjG3yYvxG5EZS5yZSFGTvoFoPYEAo, address:145.79.227.46:4607): protocol: rpc: dial tcp 145.79.227.46:4607: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 19:59:34.932941758 (296h59m32.529712814s) [noise=true , address=145.79.227.22:4538 , piece_id=ALPGNVFU5TWXZ7D7I5XFQBZ2CAN74VYUBNYDFSPG3HIHO7Z6M3AA, seg_pos={0 0}, upload=332621]: Done uploading piece. err: ecclient: upload failed (node:1kNFzPpkDtwy4VFyzeFQReorJPEVsmdDAd8ULT8ZKoGde57P3v, address:145.79.227.22:4538): protocol: rpc: dial tcp 145.79.227.22:4538: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 19:59:34.933267818 (296h59m32.53003887s) [noise=true , address=145.79.227.49:4343 , piece_id=NCJH6BOJHPW5OTSCNGKTZCZYC3F3LTRJKGVOKCJODWSMHUAATJQQ, seg_pos={0 0}, upload=332621]: Done uploading piece. err: ecclient: upload failed (node:16EsEMRW53sjKVag2WpSDGLhELyE2bSXyW4aAurCvDFKRNkXuf, address:145.79.227.49:4343): protocol: rpc: dial tcp 145.79.227.49:4343: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 19:59:34.933359674 (296h59m32.530130694s) [noise=true , address=145.79.227.36:4528 , piece_id=FJ5LKOZJYVNWLZ5I6OUCZMPFDOBVZXQRVPHREHPQX6J3YWAI2KFA, seg_pos={0 0}, upload=332621]: Done uploading piece. err: ecclient: upload failed (node:12ArcCrcHdpNgBMfTcPdtTXb65RgnTh8N32ihHAx2RrddxnRYgT, address:145.79.227.36:4528): protocol: rpc: dial tcp 145.79.227.36:4528: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 19:59:34.933622410 (296h59m32.530393477s) [noise=true , address=145.79.227.39:4043 , piece_id=D757XSQF27CMMI5UPBRB7LLGISWOYLW7OS3UICYF3U6AI62X73PQ, seg_pos={0 0}, upload=332621]: Done uploading piece. err: ecclient: upload failed (node:12f97TAyWcE4YrxDQCeycC8tnuetQiagu2MV4U5o8ARLq8jGKTX, address:145.79.227.39:4043): protocol: rpc: dial tcp 145.79.227.39:4043: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 19:59:34.933677971 (296h59m32.530448984s) [seg_pos={0 0}, upload=332621]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 18 19:59:34.955309631 (296h59m32.552080664s) [upload=332621]: Done uploading segments... Mar 18 19:59:34.955416416 (296h59m32.552187448s) [upload=332621]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 15.912793321s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 43 out of 60 is failed (threshold=6) [.............XXXXXXXXX.XXX...XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 5.001275085s trace: https://jaeger.dev.storj.io/trace/31aad4751beb404c FAIL (2026-03-18 20:01:32 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001305976s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 8 out of 60 is failed (threshold=3) [...........................XXXXXX.XX........................] duration: 15.644146979s trace: https://jaeger.dev.storj.io/trace/4b4d10a8051ad01b FAIL (2026-03-18 20:01:28 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 15.644182913s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 34 out of 60 is failed (threshold=3) [..........................XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 30.039082775s trace: https://jaeger.dev.storj.io/trace/301a6ddb546d768d FAIL (2026-03-18 20:01:16 UTC) operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Mar 18 20:00:46.047986689 (297h0m43.644757717s) [upload=332661]: Starting upload Mar 18 20:00:46.048059152 (297h0m43.644830176s) [upload=332661]: Done starting upload Mar 18 20:00:46.048260573 (297h0m43.6450316s) [upload=332661]: Uploading segments... Mar 18 20:00:46.048403656 (297h0m43.645174679s) [upload=332661]: Got next segment. Inline: false Mar 18 20:00:46.048483242 (297h0m43.645254271s) [upload=332661]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 20:00:46.048490376 (297h0m43.645261401s) [upload=332661]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 18 20:00:48.624063646 (297h0m46.220834662s) [seg_pos={0 0}, upload=332661]: Begin upload segment... Mar 18 20:00:48.624188805 (297h0m46.220959836s) [noise=true , address=79.127.219.46:4890 , piece_id=V5PPVJO5G5M6KWUUYVO7SBRLPAHLN7XPMI2UNDBMZEB2JUC2QFZA, seg_pos={0 0}, upload=332661]: Uploading piece... Mar 18 20:00:48.625298493 (297h0m46.22206951s) [seg_pos={0 0}, upload=332661]: Done begin upload segment. Mar 18 20:00:48.625348480 (297h0m46.222119509s) [noise=true , address=45.91.116.7:4012 , piece_id=BN42KPDCRRIG3AUENUBZPO2X2BTN6IO2LKG5LIKAZC2CE5BM5OIA, seg_pos={0 0}, upload=332661]: Uploading piece... Mar 18 20:00:48.625930978 (297h0m46.222702011s) [noise=true , address=45.91.116.14:4224 , piece_id=E6VIKOHYKWAIDHUGXCDGNAE6TQIMVAVPE5HDK3SEPFNZELHSHDCA, seg_pos={0 0}, upload=332661]: Uploading piece... Mar 18 20:00:48.627016185 (297h0m46.223787219s) [noise=true , address=145.79.227.23:4550 , piece_id=2ZULFFECVGQLTDKZODAJTHKD23FQI4JRPZYTG6O74QNU5SJ2GJGQ, seg_pos={0 0}, upload=332661]: Uploading piece... Mar 18 20:00:48.627120057 (297h0m46.223891098s) [noise=true , address=145.79.227.17:4975 , piece_id=X3VKLDYQ56MRO3TYDPJFGXNJQRG5VL3BRDOKUCUDZXXVWIID5F5A, seg_pos={0 0}, upload=332661]: Uploading piece... Mar 18 20:00:48.628244149 (297h0m46.225015181s) [noise=true , address=145.79.227.19:4082 , piece_id=OI4WBO3T6DSGMVS26TXKSFNZQZJ2VS7YW4A7J7KKWIHSQ3RSVMBA, seg_pos={0 0}, upload=332661]: Uploading piece... Mar 18 20:00:48.629044852 (297h0m46.225815885s) [noise=true , address=77.90.31.11:4558 , piece_id=7CRUTD77BW5EATCKAHBZDOAR7NM4QEHPTPQ4DXP3HY7N3ZAE2HVQ, seg_pos={0 0}, upload=332661]: Uploading piece... Mar 18 20:00:48.629779202 (297h0m46.226550235s) [noise=true , address=45.91.116.12:4255 , piece_id=7YNEAIAADRPSJ77SLEE2VW7XTE2Z4LBR266PK4UTMTNMPHNADOGQ, seg_pos={0 0}, upload=332661]: Uploading piece... Mar 18 20:00:48.629860782 (297h0m46.226631817s) [noise=true , address=145.79.227.28:4730 , piece_id=B7MO4XZZNSVN5ADXGH3VWSJ4CGEOOWTL6ZZEEPI3SRLGRWOZPLFQ, seg_pos={0 0}, upload=332661]: Uploading piece... Mar 18 20:00:48.630416915 (297h0m46.227187946s) [noise=true , address=79.127.219.38:4819 , piece_id=HGNLNNX3W7YWN3Y5KYTMWHALHHRVDAZ6HMH4R6HAPUWFSNPEFX2A, seg_pos={0 0}, upload=332661]: Uploading piece... Mar 18 20:00:48.630994468 (297h0m46.227765502s) [noise=true , address=145.79.227.21:4473 , piece_id=VMMJ5REXIGASCRONHHXQK3TJUONIJU72THFUQVD6RL6KKT7EDAGA, seg_pos={0 0}, upload=332661]: Uploading piece... Mar 18 20:00:48.631764171 (297h0m46.228535204s) [noise=true , address=77.90.31.17:4769 , piece_id=ACYWFDCD5EDIHJXT2SXG65NIJCDAXK45VXQJGT7M76B7TXDUDOUQ, seg_pos={0 0}, upload=332661]: Uploading piece... Mar 18 20:00:48.632374853 (297h0m46.229145885s) [noise=true , address=45.91.116.9:4262 , piece_id=IHA2IT23YJWBDLHZ5JVMBEQONZ4HJCENJN35TQTYCT7BKLX4DWNQ, seg_pos={0 0}, upload=332661]: Uploading piece... Mar 18 20:00:48.632980065 (297h0m46.229751099s) [noise=true , address=77.90.31.18:4809 , piece_id=KN3DKOJ43DK3T5VIBIOYT4EJPI7ZOHEZ6JPCQZDCJHWGKJW7JV7Q, seg_pos={0 0}, upload=332661]: Uploading piece... Mar 18 20:00:48.633564750 (297h0m46.230335776s) [noise=true , address=45.91.116.4:4955 , piece_id=JCOPN5T4US7PDS427UMWEGZD5JMBS4D2NYSVMMVRIDDRUA7IRQLQ, seg_pos={0 0}, upload=332661]: Uploading piece... Mar 18 20:00:48.634249814 (297h0m46.231020844s) [noise=true , address=77.90.31.19:4845 , piece_id=PLPC4BDDIHIBJJLOIWVFNM52U6M5YW7TV2CKRPOC5AGRWYFSS6LQ, seg_pos={0 0}, upload=332661]: Uploading piece... Mar 18 20:00:48.731820274 (297h0m46.328591303s) [upload=332661]: Next returned nil segment Mar 18 20:00:48.731837832 (297h0m46.328608865s) [upload=332661]: Waiting for error group managing segments... Mar 18 20:00:48.737543299 (297h0m46.334314324s) [noise=true , address=77.90.31.19:4845 , piece_id=PLPC4BDDIHIBJJLOIWVFNM52U6M5YW7TV2CKRPOC5AGRWYFSS6LQ, seg_pos={0 0}, upload=332661]: Done uploading piece. err: <nil> Mar 18 20:00:48.737623865 (297h0m46.334394884s) [noise=true , address=45.91.116.12:4255 , piece_id=7YNEAIAADRPSJ77SLEE2VW7XTE2Z4LBR266PK4UTMTNMPHNADOGQ, seg_pos={0 0}, upload=332661]: Done uploading piece. err: <nil> Mar 18 20:00:48.738151723 (297h0m46.33492275s) [noise=true , address=45.91.116.7:4012 , piece_id=BN42KPDCRRIG3AUENUBZPO2X2BTN6IO2LKG5LIKAZC2CE5BM5OIA, seg_pos={0 0}, upload=332661]: Done uploading piece. err: <nil> Mar 18 20:00:48.738279093 (297h0m46.335050117s) [noise=true , address=77.90.31.11:4558 , piece_id=7CRUTD77BW5EATCKAHBZDOAR7NM4QEHPTPQ4DXP3HY7N3ZAE2HVQ, seg_pos={0 0}, upload=332661]: Done uploading piece. err: <nil> Mar 18 20:00:48.738789403 (297h0m46.335560428s) [noise=true , address=45.91.116.4:4955 , piece_id=JCOPN5T4US7PDS427UMWEGZD5JMBS4D2NYSVMMVRIDDRUA7IRQLQ, seg_pos={0 0}, upload=332661]: Done uploading piece. err: <nil> Mar 18 20:00:48.739794559 (297h0m46.336565583s) [noise=true , address=79.127.219.38:4819 , piece_id=HGNLNNX3W7YWN3Y5KYTMWHALHHRVDAZ6HMH4R6HAPUWFSNPEFX2A, seg_pos={0 0}, upload=332661]: Done uploading piece. err: <nil> Mar 18 20:00:48.740830748 (297h0m46.337601761s) [noise=true , address=79.127.219.46:4890 , piece_id=V5PPVJO5G5M6KWUUYVO7SBRLPAHLN7XPMI2UNDBMZEB2JUC2QFZA, seg_pos={0 0}, upload=332661]: Done uploading piece. err: <nil> Mar 18 20:00:48.741102770 (297h0m46.337873794s) [noise=true , address=77.90.31.18:4809 , piece_id=KN3DKOJ43DK3T5VIBIOYT4EJPI7ZOHEZ6JPCQZDCJHWGKJW7JV7Q, seg_pos={0 0}, upload=332661]: Done uploading piece. err: <nil> Mar 18 20:00:48.751963944 (297h0m46.348734972s) [noise=true , address=45.91.116.14:4224 , piece_id=E6VIKOHYKWAIDHUGXCDGNAE6TQIMVAVPE5HDK3SEPFNZELHSHDCA, seg_pos={0 0}, upload=332661]: Done uploading piece. err: <nil> Mar 18 20:00:48.752345486 (297h0m46.349116518s) [noise=true , address=45.91.116.9:4262 , piece_id=IHA2IT23YJWBDLHZ5JVMBEQONZ4HJCENJN35TQTYCT7BKLX4DWNQ, seg_pos={0 0}, upload=332661]: Done uploading piece. err: <nil> Mar 18 20:00:48.756397618 (297h0m46.353168634s) [noise=true , address=77.90.31.17:4769 , piece_id=ACYWFDCD5EDIHJXT2SXG65NIJCDAXK45VXQJGT7M76B7TXDUDOUQ, seg_pos={0 0}, upload=332661]: Done uploading piece. err: <nil> Mar 18 20:01:16.010463968 (297h1m13.607235017s) [noise=true , address=145.79.227.19:4082 , piece_id=OI4WBO3T6DSGMVS26TXKSFNZQZJ2VS7YW4A7J7KKWIHSQ3RSVMBA, seg_pos={0 0}, upload=332661]: Done uploading piece. err: ecclient: upload failed (node:12DKsNCtqxLvpJwLszdLs74V3eb8NH2VAV9V3bngzUczawmNoLK, address:145.79.227.19:4082): protocol: context deadline exceeded; context deadline exceeded Mar 18 20:01:16.010637578 (297h1m13.607408636s) [noise=true , address=145.79.227.23:4550 , piece_id=2ZULFFECVGQLTDKZODAJTHKD23FQI4JRPZYTG6O74QNU5SJ2GJGQ, seg_pos={0 0}, upload=332661]: Done uploading piece. err: ecclient: upload failed (node:12kn3uD4MCE56bN4MYYsGjDJd2qebgZ6VB6mavL711M8hoU8fbE, address:145.79.227.23:4550): protocol: context deadline exceeded; context deadline exceeded Mar 18 20:01:16.010760448 (297h1m13.607531505s) [noise=true , address=145.79.227.28:4730 , piece_id=B7MO4XZZNSVN5ADXGH3VWSJ4CGEOOWTL6ZZEEPI3SRLGRWOZPLFQ, seg_pos={0 0}, upload=332661]: Done uploading piece. err: ecclient: upload failed (node:1SXQt5FugmWj1YdhsvaaR7bDoxTBkqHKZpyXSmQqGdX5ZRhKxV, address:145.79.227.28:4730): protocol: context deadline exceeded; context deadline exceeded Mar 18 20:01:16.010822650 (297h1m13.607593684s) [noise=true , address=145.79.227.21:4473 , piece_id=VMMJ5REXIGASCRONHHXQK3TJUONIJU72THFUQVD6RL6KKT7EDAGA, seg_pos={0 0}, upload=332661]: Done uploading piece. err: ecclient: upload failed (node:12DDe4jT3YeUH8qMQgapcvzMnFs8UFB8fTmubho4ZQf5tYVoxuk, address:145.79.227.21:4473): protocol: context deadline exceeded; context deadline exceeded Mar 18 20:01:16.010992880 (297h1m13.607763942s) [noise=true , address=145.79.227.17:4975 , piece_id=X3VKLDYQ56MRO3TYDPJFGXNJQRG5VL3BRDOKUCUDZXXVWIID5F5A, seg_pos={0 0}, upload=332661]: Done uploading piece. err: ecclient: upload failed (node:12WiUZeNCJk4rzjtyWuKJwGcZATaYLU2QxN27MZE7i45zVpu9dM, address:145.79.227.17:4975): protocol: context deadline exceeded; context deadline exceeded Mar 18 20:01:16.011084338 (297h1m13.607855354s) [seg_pos={0 0}, upload=332661]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Mar 18 20:01:16.047423079 (297h1m13.644194101s) [upload=332661]: Done uploading segments... Mar 18 20:01:16.047462919 (297h1m13.644233942s) [upload=332661]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 30.039101022s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 6 out of 60 is failed (threshold=3) [..............................XXX..........X...X.........X..] duration: 16.472635169s trace: https://jaeger.dev.storj.io/trace/7c859732d77a7da7 FAIL (2026-03-18 20:00:50 UTC) operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== d=332621]: Uploading segments... Mar 18 19:58:34.964515697 (296h58m32.561286706s) [upload=332621]: Got next segment. Inline: false Mar 18 19:58:34.964555310 (296h58m32.561326334s) [upload=332621]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 19:58:34.964560227 (296h58m32.561331251s) [upload=332621]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 18 19:58:38.382666205 (296h58m35.979437219s) [seg_pos={0 0}, upload=332621]: Begin upload segment... Mar 18 19:58:38.382729299 (296h58m35.979500329s) [seg_pos={0 0}, upload=332621]: Done begin upload segment. Mar 18 19:58:38.382760799 (296h58m35.979531826s) [noise=true , address=145.79.227.46:4607 , piece_id=LXBA2IGR4U6DLXE4NXL2CN23X5HXP7KFN5UCJZH6IBRYAZCQ7N3Q, seg_pos={0 0}, upload=332621]: Uploading piece... Mar 18 19:58:38.382881088 (296h58m35.979652115s) [noise=true , address=145.79.227.22:4538 , piece_id=ALPGNVFU5TWXZ7D7I5XFQBZ2CAN74VYUBNYDFSPG3HIHO7Z6M3AA, seg_pos={0 0}, upload=332621]: Uploading piece... Mar 18 19:58:38.383461716 (296h58m35.980232746s) [noise=true , address=167.17.72.70:4770 , piece_id=5H5RNTJDXOLXRNVU4XG5FP4O5YN5PW2XQH4O2UIX5XBKNMO35WKQ, seg_pos={0 0}, upload=332621]: Uploading piece... Mar 18 19:58:38.384266580 (296h58m35.981037606s) [noise=true , address=45.91.116.10:4092 , piece_id=Y7E2ZWR5D6M7QKRY3I7SL3ZJN32EK36ZWPP3YRW5WCGQFZCNU64Q, seg_pos={0 0}, upload=332621]: Uploading piece... Mar 18 19:58:38.384321708 (296h58m35.981092731s) [noise=true , address=79.127.205.235:4273 , piece_id=CQEOZ6GWLY5MEVRQZHYH6AP6DXQY5WKIAHCHDBJMAI77STRNINYA, seg_pos={0 0}, upload=332621]: Uploading piece... Mar 18 19:58:38.385197892 (296h58m35.981968918s) [noise=true , address=193.149.164.66:4989 , piece_id=EDE7OF4G36VJ7GXFMF47ZI73D6I7OWZSC5AXESS6W74ZDWTBDXWA, seg_pos={0 0}, upload=332621]: Uploading piece... Mar 18 19:58:38.385232613 (296h58m35.982003642s) [noise=true , address=45.91.116.13:4171 , piece_id=74JRIA5AS6SMECS3FFCCVLH3AY5N4EPNWOEDZGITBPCQZEKDGEAQ, seg_pos={0 0}, upload=332621]: Uploading piece... Mar 18 19:58:38.385334543 (296h58m35.982105567s) [noise=true , address=193.149.164.70:4120 , piece_id=AO6H5CLNNY2CQ27JO57WEKF7OTFQA7IXAZMVG6A4EWJLZT7NIRSQ, seg_pos={0 0}, upload=332621]: Uploading piece... Mar 18 19:58:38.386120289 (296h58m35.982891315s) [noise=true , address=79.127.219.46:4908 , piece_id=L2EOXVK7S2ZSQNK47543XTNRWJP2PU57DH2XHZ2BO5OZUTDXDQFA, seg_pos={0 0}, upload=332621]: Uploading piece... Mar 18 19:58:38.386244430 (296h58m35.983015464s) [noise=true , address=79.127.205.237:4236 , piece_id=FKQ3NUK5AUFD35L25PJFPVDLZUVNRURK5RLSPG3GGC75QYTEWP4A, seg_pos={0 0}, upload=332621]: Uploading piece... Mar 18 19:58:38.387158310 (296h58m35.983929342s) [noise=true , address=64.49.13.140:4516 , piece_id=D2V5WAW6D46VSSNBFGNO5ETQNCOTDHQYWM62BSSVF53X2MWU3TYQ, seg_pos={0 0}, upload=332621]: Uploading piece... Mar 18 19:58:38.387233109 (296h58m35.984004132s) [noise=true , address=193.149.164.67:4020 , piece_id=NZQGIA6BVAK3Z6SQZA7WNAOAY25VQKBDTWCRV34KNSYTGVNDSWUA, seg_pos={0 0}, upload=332621]: Uploading piece... Mar 18 19:58:38.388077887 (296h58m35.984848921s) [noise=true , address=145.79.227.49:4343 , piece_id=NCJH6BOJHPW5OTSCNGKTZCZYC3F3LTRJKGVOKCJODWSMHUAATJQQ, seg_pos={0 0}, upload=332621]: Uploading piece... Mar 18 19:58:38.388338947 (296h58m35.985109978s) [noise=true , address=79.127.219.41:4940 , piece_id=C2ODTVQIR73WTRBM7EDKEC54DOBK67L2RV4UXPHHTRY6KL3EQIKQ, seg_pos={0 0}, upload=332621]: Uploading piece... Mar 18 19:58:38.388792743 (296h58m35.985563766s) [noise=true , address=145.79.227.39:4043 , piece_id=D757XSQF27CMMI5UPBRB7LLGISWOYLW7OS3UICYF3U6AI62X73PQ, seg_pos={0 0}, upload=332621]: Uploading piece... Mar 18 19:58:38.389305068 (296h58m35.986076097s) [noise=true , address=45.91.116.9:4280 , piece_id=QECGXDF7C5YUJNURQCU76PMJM5MZ3DCEGPGAJHCZWI3H3OVGERGQ, seg_pos={0 0}, upload=332621]: Uploading piece... Mar 18 19:58:38.389771675 (296h58m35.986542707s) [noise=true , address=145.79.227.36:4528 , piece_id=FJ5LKOZJYVNWLZ5I6OUCZMPFDOBVZXQRVPHREHPQX6J3YWAI2KFA, seg_pos={0 0}, upload=332621]: Uploading piece... Mar 18 19:58:38.390707637 (296h58m35.98747867s) [noise=true , address=77.90.31.18:4827 , piece_id=SGKGAEAHEHU2H4YRLYQLQMTS45U6ZNSAYJ6CYJUGGKEDRMQT6BHQ, seg_pos={0 0}, upload=332621]: Uploading piece... Mar 18 19:58:38.466570556 (296h58m36.063341572s) [upload=332621]: Next returned nil segment Mar 18 19:58:38.466587006 (296h58m36.063358039s) [upload=332621]: Waiting for error group managing segments... Mar 18 19:58:38.471637588 (296h58m36.068408605s) [noise=true , address=45.91.116.9:4280 , piece_id=QECGXDF7C5YUJNURQCU76PMJM5MZ3DCEGPGAJHCZWI3H3OVGERGQ, seg_pos={0 0}, upload=332621]: Done uploading piece. err: <nil> Mar 18 19:58:38.474621081 (296h58m36.071392109s) [noise=true , address=79.127.219.41:4940 , piece_id=C2ODTVQIR73WTRBM7EDKEC54DOBK67L2RV4UXPHHTRY6KL3EQIKQ, seg_pos={0 0}, upload=332621]: Done uploading piece. err: <nil> Mar 18 19:58:38.479487991 (296h58m36.076259007s) [noise=true , address=77.90.31.18:4827 , piece_id=SGKGAEAHEHU2H4YRLYQLQMTS45U6ZNSAYJ6CYJUGGKEDRMQT6BHQ, seg_pos={0 0}, upload=332621]: Done uploading piece. err: <nil> Mar 18 19:58:38.479685133 (296h58m36.076456159s) [noise=true , address=45.91.116.10:4092 , piece_id=Y7E2ZWR5D6M7QKRY3I7SL3ZJN32EK36ZWPP3YRW5WCGQFZCNU64Q, seg_pos={0 0}, upload=332621]: Done uploading piece. err: <nil> Mar 18 19:58:38.485438319 (296h58m36.082209345s) [noise=true , address=45.91.116.13:4171 , piece_id=74JRIA5AS6SMECS3FFCCVLH3AY5N4EPNWOEDZGITBPCQZEKDGEAQ, seg_pos={0 0}, upload=332621]: Done uploading piece. err: <nil> Mar 18 19:58:38.620281221 (296h58m36.217052236s) [noise=true , address=79.127.219.46:4908 , piece_id=L2EOXVK7S2ZSQNK47543XTNRWJP2PU57DH2XHZ2BO5OZUTDXDQFA, seg_pos={0 0}, upload=332621]: Done uploading piece. err: <nil> Mar 18 19:58:38.772264604 (296h58m36.369035618s) [noise=true , address=167.17.72.70:4770 , piece_id=5H5RNTJDXOLXRNVU4XG5FP4O5YN5PW2XQH4O2UIX5XBKNMO35WKQ, seg_pos={0 0}, upload=332621]: Done uploading piece. err: <nil> Mar 18 19:58:38.967982420 (296h58m36.564753435s) [noise=true , address=79.127.205.235:4273 , piece_id=CQEOZ6GWLY5MEVRQZHYH6AP6DXQY5WKIAHCHDBJMAI77STRNINYA, seg_pos={0 0}, upload=332621]: Done uploading piece. err: <nil> Mar 18 19:58:38.977759060 (296h58m36.57453009s) [noise=true , address=79.127.205.237:4236 , piece_id=FKQ3NUK5AUFD35L25PJFPVDLZUVNRURK5RLSPG3GGC75QYTEWP4A, seg_pos={0 0}, upload=332621]: Done uploading piece. err: <nil> Mar 18 19:58:39.005320695 (296h58m36.602091708s) [noise=true , address=193.149.164.70:4120 , piece_id=AO6H5CLNNY2CQ27JO57WEKF7OTFQA7IXAZMVG6A4EWJLZT7NIRSQ, seg_pos={0 0}, upload=332621]: Done uploading piece. err: <nil> Mar 18 19:58:39.025848813 (296h58m36.622619842s) [noise=true , address=193.149.164.67:4020 , piece_id=NZQGIA6BVAK3Z6SQZA7WNAOAY25VQKBDTWCRV34KNSYTGVNDSWUA, seg_pos={0 0}, upload=332621]: Done uploading piece. err: <nil> Mar 18 19:58:39.065107032 (296h58m36.661878045s) [noise=true , address=64.49.13.140:4516 , piece_id=D2V5WAW6D46VSSNBFGNO5ETQNCOTDHQYWM62BSSVF53X2MWU3TYQ, seg_pos={0 0}, upload=332621]: Done uploading piece. err: <nil> Mar 18 19:58:39.201750353 (296h58m36.798521377s) [noise=true , address=193.149.164.66:4989 , piece_id=EDE7OF4G36VJ7GXFMF47ZI73D6I7OWZSC5AXESS6W74ZDWTBDXWA, seg_pos={0 0}, upload=332621]: Done uploading piece. err: <nil> Mar 18 19:59:34.932832250 (296h59m32.529603284s) [noise=true , address=145.79.227.46:4607 , piece_id=LXBA2IGR4U6DLXE4NXL2CN23X5HXP7KFN5UCJZH6IBRYAZCQ7N3Q, seg_pos={0 0}, upload=332621]: Done uploading piece. err: ecclient: upload failed (node:1zDoLfA5bQWM7Uf479gfFVjG3yYvxG5EZS5yZSFGTvoFoPYEAo, address:145.79.227.46:4607): protocol: rpc: dial tcp 145.79.227.46:4607: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 19:59:34.932941758 (296h59m32.529712814s) [noise=true , address=145.79.227.22:4538 , piece_id=ALPGNVFU5TWXZ7D7I5XFQBZ2CAN74VYUBNYDFSPG3HIHO7Z6M3AA, seg_pos={0 0}, upload=332621]: Done uploading piece. err: ecclient: upload failed (node:1kNFzPpkDtwy4VFyzeFQReorJPEVsmdDAd8ULT8ZKoGde57P3v, address:145.79.227.22:4538): protocol: rpc: dial tcp 145.79.227.22:4538: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 19:59:34.933267818 (296h59m32.53003887s) [noise=true , address=145.79.227.49:4343 , piece_id=NCJH6BOJHPW5OTSCNGKTZCZYC3F3LTRJKGVOKCJODWSMHUAATJQQ, seg_pos={0 0}, upload=332621]: Done uploading piece. err: ecclient: upload failed (node:16EsEMRW53sjKVag2WpSDGLhELyE2bSXyW4aAurCvDFKRNkXuf, address:145.79.227.49:4343): protocol: rpc: dial tcp 145.79.227.49:4343: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 19:59:34.933359674 (296h59m32.530130694s) [noise=true , address=145.79.227.36:4528 , piece_id=FJ5LKOZJYVNWLZ5I6OUCZMPFDOBVZXQRVPHREHPQX6J3YWAI2KFA, seg_pos={0 0}, upload=332621]: Done uploading piece. err: ecclient: upload failed (node:12ArcCrcHdpNgBMfTcPdtTXb65RgnTh8N32ihHAx2RrddxnRYgT, address:145.79.227.36:4528): protocol: rpc: dial tcp 145.79.227.36:4528: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 19:59:34.933622410 (296h59m32.530393477s) [noise=true , address=145.79.227.39:4043 , piece_id=D757XSQF27CMMI5UPBRB7LLGISWOYLW7OS3UICYF3U6AI62X73PQ, seg_pos={0 0}, upload=332621]: Done uploading piece. err: ecclient: upload failed (node:12f97TAyWcE4YrxDQCeycC8tnuetQiagu2MV4U5o8ARLq8jGKTX, address:145.79.227.39:4043): protocol: rpc: dial tcp 145.79.227.39:4043: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 19:59:34.933677971 (296h59m32.530448984s) [seg_pos={0 0}, upload=332621]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 18 19:59:34.955309631 (296h59m32.552080664s) [upload=332621]: Done uploading segments... Mar 18 19:59:34.955416416 (296h59m32.552187448s) [upload=332621]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 16.472736566s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 33 out of 60 is failed (threshold=3) [...........................XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 30.040800674s trace: https://jaeger.dev.storj.io/trace/1bde6d6ff21d8ada FAIL (2026-03-18 20:00:45 UTC) operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 7) ======LOGS====== 8 20:00:15.925245285 (297h0m13.522016302s) [upload=332655]: Uploading segments... Mar 18 20:00:15.925576063 (297h0m13.522347086s) [upload=332655]: Got next segment. Inline: false Mar 18 20:00:15.925633998 (297h0m13.522405023s) [upload=332655]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 20:00:15.925650847 (297h0m13.522421871s) [upload=332655]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 18 20:00:18.825182085 (297h0m16.421953128s) [seg_pos={0 0}, upload=332655]: Begin upload segment... Mar 18 20:00:18.825268000 (297h0m16.422039032s) [seg_pos={0 0}, upload=332655]: Done begin upload segment. Mar 18 20:00:18.825322836 (297h0m16.422093871s) [noise=true , address=77.90.31.18:4808 , piece_id=QX2UP2MVKIVBMJTNELBSSJLNHTXJSQLEBFZJWW2RRJP5AQRRD75Q, seg_pos={0 0}, upload=332655]: Uploading piece... Mar 18 20:00:18.826037678 (297h0m16.422808712s) [noise=true , address=45.91.116.13:4154 , piece_id=KOPEY5KQJKHNL4P4VJGHLRX6ZHF56RPOLQMW5DHBZCNZ7M5NJUTQ, seg_pos={0 0}, upload=332655]: Uploading piece... Mar 18 20:00:18.826353531 (297h0m16.423124566s) [noise=true , address=145.79.227.3:4155 , piece_id=ZXYSRPEF7BGVOWQYPMVMO5VEIGGDQJVNAIOUGZ7EZAKPXL37BOKA, seg_pos={0 0}, upload=332655]: Uploading piece... Mar 18 20:00:18.826910052 (297h0m16.423681085s) [noise=true , address=145.79.227.18:4008 , piece_id=OEGXMD5Z6ML6XFPLYWFHLJ2XYANTFBFOEEV52VE4YYZGE2OLDPBA, seg_pos={0 0}, upload=332655]: Uploading piece... Mar 18 20:00:18.827603347 (297h0m16.42437438s) [noise=true , address=145.79.227.32:4838 , piece_id=G5LSSI6ZAGCL2EALBCVBJIRLY25BHTGZ7U7Q4L22Y2SUED4BSESA, seg_pos={0 0}, upload=332655]: Uploading piece... Mar 18 20:00:18.827936354 (297h0m16.424707387s) [noise=true , address=145.79.227.37:4565 , piece_id=5MZCRBQXNVJHHEN7RHSQU6LEEP4MO4USO5R6AOC3P6NH2FFXSCJQ, seg_pos={0 0}, upload=332655]: Uploading piece... Mar 18 20:00:18.828128486 (297h0m16.424899521s) [noise=true , address=145.79.227.42:4124 , piece_id=SYOA3WBVPJNRGYP3KYOWCJ6XVN3Y4OOH2ZY5CKRSEYOMB7L4MDHA, seg_pos={0 0}, upload=332655]: Uploading piece... Mar 18 20:00:18.828414300 (297h0m16.425185333s) [noise=true , address=145.79.227.35:4482 , piece_id=NV22VBLNJFDF2MQIUHGF55THXH2R3SRU7TNQF5T6ZWHNRJIDB7TA, seg_pos={0 0}, upload=332655]: Uploading piece... Mar 18 20:00:18.828602835 (297h0m16.425373868s) [noise=true , address=77.90.31.11:4554 , piece_id=7KA4OF2HEVY3SDKJQ6JXU34NDGEXLIHXMWOXWXPNTGXQME5YHMDA, seg_pos={0 0}, upload=332655]: Uploading piece... Mar 18 20:00:18.829353782 (297h0m16.426124816s) [noise=true , address=145.79.227.14:4840 , piece_id=HEUBDI22DQAQMR757UB37GXN4A6XFCUNROY3YCX7UHR66EYVTSHA, seg_pos={0 0}, upload=332655]: Uploading piece... Mar 18 20:00:18.829493549 (297h0m16.426264577s) [noise=true , address=77.90.31.14:4662 , piece_id=QGJCQHJKA2YAKG7DVBM7IN7NYOMTBLOR2MXJLG5VAJTX7CBORWKQ, seg_pos={0 0}, upload=332655]: Uploading piece... Mar 18 20:00:18.830061638 (297h0m16.426832674s) [noise=true , address=145.79.227.15:4879 , piece_id=OQTKQSIB4SXYKJK4S74Z5B6WJ5BZ5IS5AEWSV4JUULNXXBW2PCBA, seg_pos={0 0}, upload=332655]: Uploading piece... Mar 18 20:00:18.831873338 (297h0m16.428644369s) [noise=true , address=145.79.227.21:4474 , piece_id=6GBJKNQMXY4HKZKIQRWYDYMDEKR6F3XDBPOZN5QEYQT5W35IJBQA, seg_pos={0 0}, upload=332655]: Uploading piece... Mar 18 20:00:18.832431808 (297h0m16.429202841s) [noise=true , address=79.127.219.46:4891 , piece_id=HQ6FXVOXYAAHURXMZ5AGEIN4CLJV25Z7GQYLRUKBNXDJTYUXFB2A, seg_pos={0 0}, upload=332655]: Uploading piece... Mar 18 20:00:18.833064848 (297h0m16.429835889s) [noise=true , address=45.91.116.12:4228 , piece_id=4MLCRASJJFP7ERLR72YOKE6WWE6ZRATS3TIPMWCTYOISR2QYFXTQ, seg_pos={0 0}, upload=332655]: Uploading piece... Mar 18 20:00:18.833616579 (297h0m16.43038761s) [noise=true , address=77.90.31.21:4878 , piece_id=WTGW6WALVS274OPYYVA7PCFVFKAFX574EUVGZWKRIH7M5QORY2KQ, seg_pos={0 0}, upload=332655]: Uploading piece... Mar 18 20:00:18.920921017 (297h0m16.51769203s) [upload=332655]: Next returned nil segment Mar 18 20:00:18.920936524 (297h0m16.517707549s) [upload=332655]: Waiting for error group managing segments... Mar 18 20:00:18.952579300 (297h0m16.549350316s) [noise=true , address=77.90.31.21:4878 , piece_id=WTGW6WALVS274OPYYVA7PCFVFKAFX574EUVGZWKRIH7M5QORY2KQ, seg_pos={0 0}, upload=332655]: Done uploading piece. err: <nil> Mar 18 20:00:18.958060258 (297h0m16.554831271s) [noise=true , address=77.90.31.18:4808 , piece_id=QX2UP2MVKIVBMJTNELBSSJLNHTXJSQLEBFZJWW2RRJP5AQRRD75Q, seg_pos={0 0}, upload=332655]: Done uploading piece. err: <nil> Mar 18 20:00:18.965083979 (297h0m16.561855012s) [noise=true , address=45.91.116.13:4154 , piece_id=KOPEY5KQJKHNL4P4VJGHLRX6ZHF56RPOLQMW5DHBZCNZ7M5NJUTQ, seg_pos={0 0}, upload=332655]: Done uploading piece. err: <nil> Mar 18 20:00:18.965652051 (297h0m16.562423079s) [noise=true , address=77.90.31.11:4554 , piece_id=7KA4OF2HEVY3SDKJQ6JXU34NDGEXLIHXMWOXWXPNTGXQME5YHMDA, seg_pos={0 0}, upload=332655]: Done uploading piece. err: <nil> Mar 18 20:00:18.967429973 (297h0m16.564200999s) [noise=true , address=79.127.219.46:4891 , piece_id=HQ6FXVOXYAAHURXMZ5AGEIN4CLJV25Z7GQYLRUKBNXDJTYUXFB2A, seg_pos={0 0}, upload=332655]: Done uploading piece. err: <nil> Mar 18 20:00:18.968142726 (297h0m16.564913775s) [noise=true , address=77.90.31.14:4662 , piece_id=QGJCQHJKA2YAKG7DVBM7IN7NYOMTBLOR2MXJLG5VAJTX7CBORWKQ, seg_pos={0 0}, upload=332655]: Done uploading piece. err: <nil> Mar 18 20:00:18.968636332 (297h0m16.565407362s) [noise=true , address=45.91.116.12:4228 , piece_id=4MLCRASJJFP7ERLR72YOKE6WWE6ZRATS3TIPMWCTYOISR2QYFXTQ, seg_pos={0 0}, upload=332655]: Done uploading piece. err: <nil> Mar 18 20:00:45.889380111 (297h0m43.486151144s) [noise=true , address=145.79.227.21:4474 , piece_id=6GBJKNQMXY4HKZKIQRWYDYMDEKR6F3XDBPOZN5QEYQT5W35IJBQA, seg_pos={0 0}, upload=332655]: Done uploading piece. err: ecclient: upload failed (node:1rL2DnjSgHP36Md3bCgy5padqTAhv3iQCeURbLMuuuMBxtUNZH, address:145.79.227.21:4474): protocol: rpc: dial tcp 145.79.227.21:4474: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:00:45.889423790 (297h0m43.486194823s) [noise=true , address=145.79.227.42:4124 , piece_id=SYOA3WBVPJNRGYP3KYOWCJ6XVN3Y4OOH2ZY5CKRSEYOMB7L4MDHA, seg_pos={0 0}, upload=332655]: Done uploading piece. err: ecclient: upload failed (node:12kz79BpzkFLTxbgi2Tjvz4dStT7MtrQ4GmX6bFnYw8v45jBaSy, address:145.79.227.42:4124): protocol: rpc: dial tcp 145.79.227.42:4124: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:00:45.889723675 (297h0m43.486494709s) [noise=true , address=145.79.227.18:4008 , piece_id=OEGXMD5Z6ML6XFPLYWFHLJ2XYANTFBFOEEV52VE4YYZGE2OLDPBA, seg_pos={0 0}, upload=332655]: Done uploading piece. err: ecclient: upload failed (node:148CqQmrsJSfoGByHYRPoZmvP1VgQQpaMVLb7RgRQvNBuo7air, address:145.79.227.18:4008): protocol: rpc: dial tcp 145.79.227.18:4008: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:00:45.889881046 (297h0m43.48665208s) [noise=true , address=145.79.227.15:4879 , piece_id=OQTKQSIB4SXYKJK4S74Z5B6WJ5BZ5IS5AEWSV4JUULNXXBW2PCBA, seg_pos={0 0}, upload=332655]: Done uploading piece. err: ecclient: upload failed (node:12jH7W99nQ77T67b8WVVqmg9Jqi5QoB1qM5Wcj8WexywAZ7fRNT, address:145.79.227.15:4879): protocol: rpc: dial tcp 145.79.227.15:4879: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:00:45.889955257 (297h0m43.486726288s) [noise=true , address=145.79.227.14:4840 , piece_id=HEUBDI22DQAQMR757UB37GXN4A6XFCUNROY3YCX7UHR66EYVTSHA, seg_pos={0 0}, upload=332655]: Done uploading piece. err: ecclient: upload failed (node:12crcXpNe9NgeXTsUpooLSBgrSUFbRxbSbanLa39w2Hr5mGFww, address:145.79.227.14:4840): protocol: rpc: dial tcp 145.79.227.14:4840: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:00:45.890433135 (297h0m43.487204166s) [noise=true , address=145.79.227.32:4838 , piece_id=G5LSSI6ZAGCL2EALBCVBJIRLY25BHTGZ7U7Q4L22Y2SUED4BSESA, seg_pos={0 0}, upload=332655]: Done uploading piece. err: ecclient: upload failed (node:1eorQLNmgEaXv8myr6t3Q8rysGwsvtdZzu5b7sndhoANb6Waj6, address:145.79.227.32:4838): protocol: rpc: dial tcp 145.79.227.32:4838: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:00:45.890673659 (297h0m43.487444693s) [noise=true , address=145.79.227.37:4565 , piece_id=5MZCRBQXNVJHHEN7RHSQU6LEEP4MO4USO5R6AOC3P6NH2FFXSCJQ, seg_pos={0 0}, upload=332655]: Done uploading piece. err: ecclient: upload failed (node:1nvvNczHtQrSEJuJbErnjiAWa4BRDzBrQmBNbB7RrrqN1JWNHT, address:145.79.227.37:4565): protocol: rpc: dial tcp 145.79.227.37:4565: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:00:45.890971358 (297h0m43.487742581s) [noise=true , address=145.79.227.35:4482 , piece_id=NV22VBLNJFDF2MQIUHGF55THXH2R3SRU7TNQF5T6ZWHNRJIDB7TA, seg_pos={0 0}, upload=332655]: Done uploading piece. err: ecclient: upload failed (node:12mNCKAdMzWu87fmyrcwDKZvscMcu2CogjXicwAi5wLSFJa7oVV, address:145.79.227.35:4482): protocol: rpc: dial tcp 145.79.227.35:4482: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:00:45.891479194 (297h0m43.488250228s) [noise=true , address=145.79.227.3:4155 , piece_id=ZXYSRPEF7BGVOWQYPMVMO5VEIGGDQJVNAIOUGZ7EZAKPXL37BOKA, seg_pos={0 0}, upload=332655]: Done uploading piece. err: ecclient: upload failed (node:1yiUBiCEpAs5yDmgVfFGuxgW4R53jzMJugo8Piff3p79EEZdWr, address:145.79.227.3:4155): protocol: rpc: dial tcp 145.79.227.3:4155: i/o timeout; conn closed: op Mar 18 20:00:45.891522024 (297h0m43.488293048s) [seg_pos={0 0}, upload=332655]: Done waiting for segment. successful: 7 optimal: 13 errs: <nil> Mar 18 20:00:45.928738582 (297h0m43.525509606s) [upload=332655]: Done uploading segments... Mar 18 20:00:45.928776244 (297h0m43.525547267s) [upload=332655]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 7) Duration: 30.040823623s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 42 out of 60 is failed (threshold=6) [..............XXXXXXXXX.XXX...XXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 5.001613448s trace: https://jaeger.dev.storj.io/trace/3b48d4e646649a42 FAIL (2026-03-18 20:00:32 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001637234s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 8 out of 60 is failed (threshold=3) [............................XXXXXX.XX.......................] duration: 16.816186294s trace: https://jaeger.dev.storj.io/trace/4b4d10a8051ad01b FAIL (2026-03-18 20:00:19 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 16.816218326s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 5 out of 60 is failed (threshold=3) [...............................X..........XX..X....X........] duration: 17.029159537s trace: https://jaeger.dev.storj.io/trace/6b5565e31dae52f1 FAIL (2026-03-18 20:00:18 UTC) operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== Mar 18 19:53:19.380863725 (296h53m16.977634736s) [upload=332531]: Starting upload Mar 18 19:53:19.380944319 (296h53m16.977715346s) [upload=332531]: Done starting upload Mar 18 19:53:19.380976457 (296h53m16.977747482s) [upload=332531]: Uploading segments... Mar 18 19:53:19.381333196 (296h53m16.978104222s) [upload=332531]: Got next segment. Inline: false Mar 18 19:53:19.381379068 (296h53m16.978150099s) [upload=332531]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 19:53:19.381386543 (296h53m16.978157579s) [upload=332531]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 18 19:53:23.490845546 (296h53m21.087616603s) [seg_pos={0 0}, upload=332531]: Begin upload segment... Mar 18 19:53:23.490917951 (296h53m21.087688988s) [seg_pos={0 0}, upload=332531]: Done begin upload segment. Mar 18 19:53:23.490955926 (296h53m21.087726957s) [noise=true , address=45.91.116.5:4312 , piece_id=BDB6LYZZH2HXC3WWPDVMDQLCAAGRXBM5OCZU2JQVUJ7E6REVJ5SQ, seg_pos={0 0}, upload=332531]: Uploading piece... Mar 18 19:53:23.492093139 (296h53m21.088864171s) [noise=true , address=193.149.164.67:4003 , piece_id=SOPPFFG22JZWQA5NHYGA2OFHC2KMALZOR3ATBQ6JYWJVXATWLSRQ, seg_pos={0 0}, upload=332531]: Uploading piece... Mar 18 19:53:23.492835295 (296h53m21.089606329s) [noise=true , address=77.90.31.18:4833 , piece_id=ON3SKTN2H26XEXC43C44MAJIPY32NZLNWRU4PBSRJS6HLMXVNBMA, seg_pos={0 0}, upload=332531]: Uploading piece... Mar 18 19:53:23.492958664 (296h53m21.089729699s) [noise=true , address=45.91.116.13:4167 , piece_id=3JWBJU4M3WSUM3D2TVZDGJKBSRXZY5ZQHV3GFSBKOJMLORY5W7IQ, seg_pos={0 0}, upload=332531]: Uploading piece... Mar 18 19:53:23.493624498 (296h53m21.090395533s) [noise=true , address=145.79.227.13:4814 , piece_id=3KU6BHQSH4U4XYBSV4I77FCLQKR5RKWFY3QKZ26D5QRCJDQPKTKQ, seg_pos={0 0}, upload=332531]: Uploading piece... Mar 18 19:53:23.493650442 (296h53m21.090421482s) [noise=true , address=79.127.205.237:4230 , piece_id=Y6AOKOD27V5RL6QWUY2ZAJMONWPTYU4TTU5VIAYJ4RVMNQMCRUZQ, seg_pos={0 0}, upload=332531]: Uploading piece... Mar 18 19:53:23.494253839 (296h53m21.091024872s) [noise=true , address=45.91.116.14:4223 , piece_id=X3NSJBVWEZ3F3DUHQOSQSUCECIN7FARCFMRPJR6N7FHIIL3MS3NQ, seg_pos={0 0}, upload=332531]: Uploading piece... Mar 18 19:53:23.494267618 (296h53m21.091038645s) [noise=true , address=145.79.227.22:4516 , piece_id=OES5UD4FMNBSPC4ZSDS45SRXLTAQT3QNGR6JBI5D5Y363JFGJ73Q, seg_pos={0 0}, upload=332531]: Uploading piece... Mar 18 19:53:23.494879425 (296h53m21.091650453s) [noise=true , address=193.149.164.71:4287 , piece_id=ZE2Q4NHL3KUKRB36IDRMIZ7DIKDYZIBCM2L3PTLXALGL3PINBGAQ, seg_pos={0 0}, upload=332531]: Uploading piece... Mar 18 19:53:23.494911912 (296h53m21.091682944s) [noise=true , address=193.149.164.69:4074 , piece_id=FUGRHBCQHA4YJ2EVCMKVQVITADSVLM3PWZHV2XM3QLRDGU4G2GPA, seg_pos={0 0}, upload=332531]: Uploading piece... Mar 18 19:53:23.495497539 (296h53m21.09226857s) [noise=true , address=193.149.164.68:4066 , piece_id=FY5RBQUD2JCNXJCY2KIK744F4QZFHTW45PBDMASIIP27RWWB7SSA, seg_pos={0 0}, upload=332531]: Uploading piece... Mar 18 19:53:23.496168841 (296h53m21.092939872s) [noise=true , address=145.79.227.41:4105 , piece_id=RTYHA5YP42CFAYVM6PHY5RTB47GRMWSZEBBPNLC532CXKPBT2VXA, seg_pos={0 0}, upload=332531]: Uploading piece... Mar 18 19:53:23.497008065 (296h53m21.093779105s) [noise=true , address=79.127.205.235:4271 , piece_id=5XULCXAVIG3CQFTYA6Q3TFPRVICQFHQI2MSI2VIRIAGSK5UKKFXQ, seg_pos={0 0}, upload=332531]: Uploading piece... Mar 18 19:53:23.497605357 (296h53m21.094376396s) [noise=true , address=145.79.227.37:4590 , piece_id=ZTQLQ2IZHJFZWBHMRT74V7JDZYXF5BG4656VBZDJ4J6NGKS2TMOA, seg_pos={0 0}, upload=332531]: Uploading piece... Mar 18 19:53:23.498200357 (296h53m21.094971388s) [noise=true , address=193.149.164.66:4989 , piece_id=DVAJPRB7ABJJGUP4TYMLOGJVDLUARAMSPCG6FE2PVQXIDP2QBPGA, seg_pos={0 0}, upload=332531]: Uploading piece... Mar 18 19:53:23.498757906 (296h53m21.095528939s) [noise=true , address=79.127.219.45:4868 , piece_id=6NDAYV5PFDBOMZKAXKO6LZUODM46MQPQOQIDKBHER3LG4CBEDHKQ, seg_pos={0 0}, upload=332531]: Uploading piece... Mar 18 19:53:23.499215136 (296h53m21.095986159s) [noise=true , address=145.79.227.52:4453 , piece_id=YRAA65GHFFR45DX7MKR5BWBXID3YVSA4DMQDRKWKRDTJCHKHGMOA, seg_pos={0 0}, upload=332531]: Uploading piece... Mar 18 19:53:23.499687663 (296h53m21.096458697s) [noise=true , address=77.90.31.14:4678 , piece_id=NWBLAAUF3GACR3Y6JRTHDEUJ24GMOTZIPSRGJIPTPLIVWBCSYAHA, seg_pos={0 0}, upload=332531]: Uploading piece... Mar 18 19:53:23.600142173 (296h53m21.196913186s) [upload=332531]: Next returned nil segment Mar 18 19:53:23.600161212 (296h53m21.196932236s) [upload=332531]: Waiting for error group managing segments... Mar 18 19:53:23.620885875 (296h53m21.217656899s) [noise=true , address=45.91.116.5:4312 , piece_id=BDB6LYZZH2HXC3WWPDVMDQLCAAGRXBM5OCZU2JQVUJ7E6REVJ5SQ, seg_pos={0 0}, upload=332531]: Done uploading piece. err: <nil> Mar 18 19:53:23.621154396 (296h53m21.217925412s) [noise=true , address=45.91.116.13:4167 , piece_id=3JWBJU4M3WSUM3D2TVZDGJKBSRXZY5ZQHV3GFSBKOJMLORY5W7IQ, seg_pos={0 0}, upload=332531]: Done uploading piece. err: <nil> Mar 18 19:53:23.628883995 (296h53m21.225655021s) [noise=true , address=45.91.116.14:4223 , piece_id=X3NSJBVWEZ3F3DUHQOSQSUCECIN7FARCFMRPJR6N7FHIIL3MS3NQ, seg_pos={0 0}, upload=332531]: Done uploading piece. err: <nil> Mar 18 19:53:23.630970813 (296h53m21.227741838s) [noise=true , address=79.127.219.45:4868 , piece_id=6NDAYV5PFDBOMZKAXKO6LZUODM46MQPQOQIDKBHER3LG4CBEDHKQ, seg_pos={0 0}, upload=332531]: Done uploading piece. err: <nil> Mar 18 19:53:23.639825090 (296h53m21.236596104s) [noise=true , address=77.90.31.14:4678 , piece_id=NWBLAAUF3GACR3Y6JRTHDEUJ24GMOTZIPSRGJIPTPLIVWBCSYAHA, seg_pos={0 0}, upload=332531]: Done uploading piece. err: <nil> Mar 18 19:53:23.642754677 (296h53m21.239525692s) [noise=true , address=77.90.31.18:4833 , piece_id=ON3SKTN2H26XEXC43C44MAJIPY32NZLNWRU4PBSRJS6HLMXVNBMA, seg_pos={0 0}, upload=332531]: Done uploading piece. err: <nil> Mar 18 19:53:24.057422272 (296h53m21.654193306s) [noise=true , address=79.127.205.235:4271 , piece_id=5XULCXAVIG3CQFTYA6Q3TFPRVICQFHQI2MSI2VIRIAGSK5UKKFXQ, seg_pos={0 0}, upload=332531]: Done uploading piece. err: <nil> Mar 18 19:53:24.074244577 (296h53m21.671015604s) [noise=true , address=193.149.164.67:4003 , piece_id=SOPPFFG22JZWQA5NHYGA2OFHC2KMALZOR3ATBQ6JYWJVXATWLSRQ, seg_pos={0 0}, upload=332531]: Done uploading piece. err: <nil> Mar 18 19:53:24.097678734 (296h53m21.694449747s) [noise=true , address=79.127.205.237:4230 , piece_id=Y6AOKOD27V5RL6QWUY2ZAJMONWPTYU4TTU5VIAYJ4RVMNQMCRUZQ, seg_pos={0 0}, upload=332531]: Done uploading piece. err: <nil> Mar 18 19:53:24.109685982 (296h53m21.706457009s) [noise=true , address=193.149.164.68:4066 , piece_id=FY5RBQUD2JCNXJCY2KIK744F4QZFHTW45PBDMASIIP27RWWB7SSA, seg_pos={0 0}, upload=332531]: Done uploading piece. err: <nil> Mar 18 19:53:24.110659716 (296h53m21.707430744s) [noise=true , address=193.149.164.69:4074 , piece_id=FUGRHBCQHA4YJ2EVCMKVQVITADSVLM3PWZHV2XM3QLRDGU4G2GPA, seg_pos={0 0}, upload=332531]: Done uploading piece. err: <nil> Mar 18 19:53:24.114541274 (296h53m21.7113123s) [noise=true , address=193.149.164.71:4287 , piece_id=ZE2Q4NHL3KUKRB36IDRMIZ7DIKDYZIBCM2L3PTLXALGL3PINBGAQ, seg_pos={0 0}, upload=332531]: Done uploading piece. err: <nil> Mar 18 19:53:24.140894470 (296h53m21.737665488s) [noise=true , address=193.149.164.66:4989 , piece_id=DVAJPRB7ABJJGUP4TYMLOGJVDLUARAMSPCG6FE2PVQXIDP2QBPGA, seg_pos={0 0}, upload=332531]: Done uploading piece. err: <nil> Mar 18 19:54:19.329327996 (296h54m16.926099018s) [noise=true , address=145.79.227.13:4814 , piece_id=3KU6BHQSH4U4XYBSV4I77FCLQKR5RKWFY3QKZ26D5QRCJDQPKTKQ, seg_pos={0 0}, upload=332531]: Done uploading piece. err: ecclient: upload failed (node:12ERtrzMuh8rZPsgebtyScmYvNVwz3UKJr5Nu2hLdu6sn8X3qLa, address:145.79.227.13:4814): protocol: rpc: dial tcp 145.79.227.13:4814: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 19:54:19.329567681 (296h54m16.926338706s) [noise=true , address=145.79.227.52:4453 , piece_id=YRAA65GHFFR45DX7MKR5BWBXID3YVSA4DMQDRKWKRDTJCHKHGMOA, seg_pos={0 0}, upload=332531]: Done uploading piece. err: ecclient: upload failed (node:1BmpsQ9VfhGzmkZHvSeMptcGhzGpdoN4oYvi83qZcC1Vh9anhP, address:145.79.227.52:4453): protocol: rpc: dial tcp 145.79.227.52:4453: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 19:54:19.329755822 (296h54m16.926526848s) [noise=true , address=145.79.227.41:4105 , piece_id=RTYHA5YP42CFAYVM6PHY5RTB47GRMWSZEBBPNLC532CXKPBT2VXA, seg_pos={0 0}, upload=332531]: Done uploading piece. err: ecclient: upload failed (node:1TVtMQNVV2kERp2aZwmS8E9JUiua1RCyFpZgFACRqH1Nfscw4C, address:145.79.227.41:4105): protocol: rpc: dial tcp 145.79.227.41:4105: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 19:54:19.330021146 (296h54m16.926792175s) [noise=true , address=145.79.227.22:4516 , piece_id=OES5UD4FMNBSPC4ZSDS45SRXLTAQT3QNGR6JBI5D5Y363JFGJ73Q, seg_pos={0 0}, upload=332531]: Done uploading piece. err: ecclient: upload failed (node:12Lhy3AxzYoF25AxWdvjrrV5NNCwP1cDqkEvv4tz9t9KZmF8WPK, address:145.79.227.22:4516): protocol: rpc: dial tcp 145.79.227.22:4516: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 19:54:19.330213680 (296h54m16.926984707s) [noise=true , address=145.79.227.37:4590 , piece_id=ZTQLQ2IZHJFZWBHMRT74V7JDZYXF5BG4656VBZDJ4J6NGKS2TMOA, seg_pos={0 0}, upload=332531]: Done uploading piece. err: ecclient: upload failed (node:1KnNqY1WNwMSFXFrQsWPAhivUTgSn7dKTx1mTTzYG2UxGCkasj, address:145.79.227.37:4590): protocol: rpc: dial tcp 145.79.227.37:4590: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 19:54:19.330250832 (296h54m16.927021848s) [seg_pos={0 0}, upload=332531]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 18 19:54:19.351979131 (296h54m16.948750155s) [upload=332531]: Done uploading segments... Mar 18 19:54:19.352025297 (296h54m16.948796327s) [upload=332531]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 17.029261005s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 41 out of 60 is failed (threshold=6) [...............XXXXXXXXX.XXX...XXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 5.001749513s trace: https://jaeger.dev.storj.io/trace/1c4d2517c500d09e FAIL (2026-03-18 20:00:14 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001870307s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 6 out of 60 is failed (threshold=3) [...............................XXX..........X...X.........X.] duration: 17.061517597s trace: https://jaeger.dev.storj.io/trace/7c859732d77a7da7 FAIL (2026-03-18 19:59:52 UTC) operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== d=332621]: Uploading segments... Mar 18 19:58:34.964515697 (296h58m32.561286706s) [upload=332621]: Got next segment. Inline: false Mar 18 19:58:34.964555310 (296h58m32.561326334s) [upload=332621]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 19:58:34.964560227 (296h58m32.561331251s) [upload=332621]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 18 19:58:38.382666205 (296h58m35.979437219s) [seg_pos={0 0}, upload=332621]: Begin upload segment... Mar 18 19:58:38.382729299 (296h58m35.979500329s) [seg_pos={0 0}, upload=332621]: Done begin upload segment. Mar 18 19:58:38.382760799 (296h58m35.979531826s) [noise=true , address=145.79.227.46:4607 , piece_id=LXBA2IGR4U6DLXE4NXL2CN23X5HXP7KFN5UCJZH6IBRYAZCQ7N3Q, seg_pos={0 0}, upload=332621]: Uploading piece... Mar 18 19:58:38.382881088 (296h58m35.979652115s) [noise=true , address=145.79.227.22:4538 , piece_id=ALPGNVFU5TWXZ7D7I5XFQBZ2CAN74VYUBNYDFSPG3HIHO7Z6M3AA, seg_pos={0 0}, upload=332621]: Uploading piece... Mar 18 19:58:38.383461716 (296h58m35.980232746s) [noise=true , address=167.17.72.70:4770 , piece_id=5H5RNTJDXOLXRNVU4XG5FP4O5YN5PW2XQH4O2UIX5XBKNMO35WKQ, seg_pos={0 0}, upload=332621]: Uploading piece... Mar 18 19:58:38.384266580 (296h58m35.981037606s) [noise=true , address=45.91.116.10:4092 , piece_id=Y7E2ZWR5D6M7QKRY3I7SL3ZJN32EK36ZWPP3YRW5WCGQFZCNU64Q, seg_pos={0 0}, upload=332621]: Uploading piece... Mar 18 19:58:38.384321708 (296h58m35.981092731s) [noise=true , address=79.127.205.235:4273 , piece_id=CQEOZ6GWLY5MEVRQZHYH6AP6DXQY5WKIAHCHDBJMAI77STRNINYA, seg_pos={0 0}, upload=332621]: Uploading piece... Mar 18 19:58:38.385197892 (296h58m35.981968918s) [noise=true , address=193.149.164.66:4989 , piece_id=EDE7OF4G36VJ7GXFMF47ZI73D6I7OWZSC5AXESS6W74ZDWTBDXWA, seg_pos={0 0}, upload=332621]: Uploading piece... Mar 18 19:58:38.385232613 (296h58m35.982003642s) [noise=true , address=45.91.116.13:4171 , piece_id=74JRIA5AS6SMECS3FFCCVLH3AY5N4EPNWOEDZGITBPCQZEKDGEAQ, seg_pos={0 0}, upload=332621]: Uploading piece... Mar 18 19:58:38.385334543 (296h58m35.982105567s) [noise=true , address=193.149.164.70:4120 , piece_id=AO6H5CLNNY2CQ27JO57WEKF7OTFQA7IXAZMVG6A4EWJLZT7NIRSQ, seg_pos={0 0}, upload=332621]: Uploading piece... Mar 18 19:58:38.386120289 (296h58m35.982891315s) [noise=true , address=79.127.219.46:4908 , piece_id=L2EOXVK7S2ZSQNK47543XTNRWJP2PU57DH2XHZ2BO5OZUTDXDQFA, seg_pos={0 0}, upload=332621]: Uploading piece... Mar 18 19:58:38.386244430 (296h58m35.983015464s) [noise=true , address=79.127.205.237:4236 , piece_id=FKQ3NUK5AUFD35L25PJFPVDLZUVNRURK5RLSPG3GGC75QYTEWP4A, seg_pos={0 0}, upload=332621]: Uploading piece... Mar 18 19:58:38.387158310 (296h58m35.983929342s) [noise=true , address=64.49.13.140:4516 , piece_id=D2V5WAW6D46VSSNBFGNO5ETQNCOTDHQYWM62BSSVF53X2MWU3TYQ, seg_pos={0 0}, upload=332621]: Uploading piece... Mar 18 19:58:38.387233109 (296h58m35.984004132s) [noise=true , address=193.149.164.67:4020 , piece_id=NZQGIA6BVAK3Z6SQZA7WNAOAY25VQKBDTWCRV34KNSYTGVNDSWUA, seg_pos={0 0}, upload=332621]: Uploading piece... Mar 18 19:58:38.388077887 (296h58m35.984848921s) [noise=true , address=145.79.227.49:4343 , piece_id=NCJH6BOJHPW5OTSCNGKTZCZYC3F3LTRJKGVOKCJODWSMHUAATJQQ, seg_pos={0 0}, upload=332621]: Uploading piece... Mar 18 19:58:38.388338947 (296h58m35.985109978s) [noise=true , address=79.127.219.41:4940 , piece_id=C2ODTVQIR73WTRBM7EDKEC54DOBK67L2RV4UXPHHTRY6KL3EQIKQ, seg_pos={0 0}, upload=332621]: Uploading piece... Mar 18 19:58:38.388792743 (296h58m35.985563766s) [noise=true , address=145.79.227.39:4043 , piece_id=D757XSQF27CMMI5UPBRB7LLGISWOYLW7OS3UICYF3U6AI62X73PQ, seg_pos={0 0}, upload=332621]: Uploading piece... Mar 18 19:58:38.389305068 (296h58m35.986076097s) [noise=true , address=45.91.116.9:4280 , piece_id=QECGXDF7C5YUJNURQCU76PMJM5MZ3DCEGPGAJHCZWI3H3OVGERGQ, seg_pos={0 0}, upload=332621]: Uploading piece... Mar 18 19:58:38.389771675 (296h58m35.986542707s) [noise=true , address=145.79.227.36:4528 , piece_id=FJ5LKOZJYVNWLZ5I6OUCZMPFDOBVZXQRVPHREHPQX6J3YWAI2KFA, seg_pos={0 0}, upload=332621]: Uploading piece... Mar 18 19:58:38.390707637 (296h58m35.98747867s) [noise=true , address=77.90.31.18:4827 , piece_id=SGKGAEAHEHU2H4YRLYQLQMTS45U6ZNSAYJ6CYJUGGKEDRMQT6BHQ, seg_pos={0 0}, upload=332621]: Uploading piece... Mar 18 19:58:38.466570556 (296h58m36.063341572s) [upload=332621]: Next returned nil segment Mar 18 19:58:38.466587006 (296h58m36.063358039s) [upload=332621]: Waiting for error group managing segments... Mar 18 19:58:38.471637588 (296h58m36.068408605s) [noise=true , address=45.91.116.9:4280 , piece_id=QECGXDF7C5YUJNURQCU76PMJM5MZ3DCEGPGAJHCZWI3H3OVGERGQ, seg_pos={0 0}, upload=332621]: Done uploading piece. err: <nil> Mar 18 19:58:38.474621081 (296h58m36.071392109s) [noise=true , address=79.127.219.41:4940 , piece_id=C2ODTVQIR73WTRBM7EDKEC54DOBK67L2RV4UXPHHTRY6KL3EQIKQ, seg_pos={0 0}, upload=332621]: Done uploading piece. err: <nil> Mar 18 19:58:38.479487991 (296h58m36.076259007s) [noise=true , address=77.90.31.18:4827 , piece_id=SGKGAEAHEHU2H4YRLYQLQMTS45U6ZNSAYJ6CYJUGGKEDRMQT6BHQ, seg_pos={0 0}, upload=332621]: Done uploading piece. err: <nil> Mar 18 19:58:38.479685133 (296h58m36.076456159s) [noise=true , address=45.91.116.10:4092 , piece_id=Y7E2ZWR5D6M7QKRY3I7SL3ZJN32EK36ZWPP3YRW5WCGQFZCNU64Q, seg_pos={0 0}, upload=332621]: Done uploading piece. err: <nil> Mar 18 19:58:38.485438319 (296h58m36.082209345s) [noise=true , address=45.91.116.13:4171 , piece_id=74JRIA5AS6SMECS3FFCCVLH3AY5N4EPNWOEDZGITBPCQZEKDGEAQ, seg_pos={0 0}, upload=332621]: Done uploading piece. err: <nil> Mar 18 19:58:38.620281221 (296h58m36.217052236s) [noise=true , address=79.127.219.46:4908 , piece_id=L2EOXVK7S2ZSQNK47543XTNRWJP2PU57DH2XHZ2BO5OZUTDXDQFA, seg_pos={0 0}, upload=332621]: Done uploading piece. err: <nil> Mar 18 19:58:38.772264604 (296h58m36.369035618s) [noise=true , address=167.17.72.70:4770 , piece_id=5H5RNTJDXOLXRNVU4XG5FP4O5YN5PW2XQH4O2UIX5XBKNMO35WKQ, seg_pos={0 0}, upload=332621]: Done uploading piece. err: <nil> Mar 18 19:58:38.967982420 (296h58m36.564753435s) [noise=true , address=79.127.205.235:4273 , piece_id=CQEOZ6GWLY5MEVRQZHYH6AP6DXQY5WKIAHCHDBJMAI77STRNINYA, seg_pos={0 0}, upload=332621]: Done uploading piece. err: <nil> Mar 18 19:58:38.977759060 (296h58m36.57453009s) [noise=true , address=79.127.205.237:4236 , piece_id=FKQ3NUK5AUFD35L25PJFPVDLZUVNRURK5RLSPG3GGC75QYTEWP4A, seg_pos={0 0}, upload=332621]: Done uploading piece. err: <nil> Mar 18 19:58:39.005320695 (296h58m36.602091708s) [noise=true , address=193.149.164.70:4120 , piece_id=AO6H5CLNNY2CQ27JO57WEKF7OTFQA7IXAZMVG6A4EWJLZT7NIRSQ, seg_pos={0 0}, upload=332621]: Done uploading piece. err: <nil> Mar 18 19:58:39.025848813 (296h58m36.622619842s) [noise=true , address=193.149.164.67:4020 , piece_id=NZQGIA6BVAK3Z6SQZA7WNAOAY25VQKBDTWCRV34KNSYTGVNDSWUA, seg_pos={0 0}, upload=332621]: Done uploading piece. err: <nil> Mar 18 19:58:39.065107032 (296h58m36.661878045s) [noise=true , address=64.49.13.140:4516 , piece_id=D2V5WAW6D46VSSNBFGNO5ETQNCOTDHQYWM62BSSVF53X2MWU3TYQ, seg_pos={0 0}, upload=332621]: Done uploading piece. err: <nil> Mar 18 19:58:39.201750353 (296h58m36.798521377s) [noise=true , address=193.149.164.66:4989 , piece_id=EDE7OF4G36VJ7GXFMF47ZI73D6I7OWZSC5AXESS6W74ZDWTBDXWA, seg_pos={0 0}, upload=332621]: Done uploading piece. err: <nil> Mar 18 19:59:34.932832250 (296h59m32.529603284s) [noise=true , address=145.79.227.46:4607 , piece_id=LXBA2IGR4U6DLXE4NXL2CN23X5HXP7KFN5UCJZH6IBRYAZCQ7N3Q, seg_pos={0 0}, upload=332621]: Done uploading piece. err: ecclient: upload failed (node:1zDoLfA5bQWM7Uf479gfFVjG3yYvxG5EZS5yZSFGTvoFoPYEAo, address:145.79.227.46:4607): protocol: rpc: dial tcp 145.79.227.46:4607: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 19:59:34.932941758 (296h59m32.529712814s) [noise=true , address=145.79.227.22:4538 , piece_id=ALPGNVFU5TWXZ7D7I5XFQBZ2CAN74VYUBNYDFSPG3HIHO7Z6M3AA, seg_pos={0 0}, upload=332621]: Done uploading piece. err: ecclient: upload failed (node:1kNFzPpkDtwy4VFyzeFQReorJPEVsmdDAd8ULT8ZKoGde57P3v, address:145.79.227.22:4538): protocol: rpc: dial tcp 145.79.227.22:4538: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 19:59:34.933267818 (296h59m32.53003887s) [noise=true , address=145.79.227.49:4343 , piece_id=NCJH6BOJHPW5OTSCNGKTZCZYC3F3LTRJKGVOKCJODWSMHUAATJQQ, seg_pos={0 0}, upload=332621]: Done uploading piece. err: ecclient: upload failed (node:16EsEMRW53sjKVag2WpSDGLhELyE2bSXyW4aAurCvDFKRNkXuf, address:145.79.227.49:4343): protocol: rpc: dial tcp 145.79.227.49:4343: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 19:59:34.933359674 (296h59m32.530130694s) [noise=true , address=145.79.227.36:4528 , piece_id=FJ5LKOZJYVNWLZ5I6OUCZMPFDOBVZXQRVPHREHPQX6J3YWAI2KFA, seg_pos={0 0}, upload=332621]: Done uploading piece. err: ecclient: upload failed (node:12ArcCrcHdpNgBMfTcPdtTXb65RgnTh8N32ihHAx2RrddxnRYgT, address:145.79.227.36:4528): protocol: rpc: dial tcp 145.79.227.36:4528: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 19:59:34.933622410 (296h59m32.530393477s) [noise=true , address=145.79.227.39:4043 , piece_id=D757XSQF27CMMI5UPBRB7LLGISWOYLW7OS3UICYF3U6AI62X73PQ, seg_pos={0 0}, upload=332621]: Done uploading piece. err: ecclient: upload failed (node:12f97TAyWcE4YrxDQCeycC8tnuetQiagu2MV4U5o8ARLq8jGKTX, address:145.79.227.39:4043): protocol: rpc: dial tcp 145.79.227.39:4043: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 19:59:34.933677971 (296h59m32.530448984s) [seg_pos={0 0}, upload=332621]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 18 19:59:34.955309631 (296h59m32.552080664s) [upload=332621]: Done uploading segments... Mar 18 19:59:34.955416416 (296h59m32.552187448s) [upload=332621]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 17.061536855s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 5 out of 60 is failed (threshold=3) [................................X..........XX..X....X.......] duration: 16.956630951s trace: https://jaeger.dev.storj.io/trace/6b5565e31dae52f1 FAIL (2026-03-18 19:59:49 UTC) operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== Mar 18 19:53:19.380863725 (296h53m16.977634736s) [upload=332531]: Starting upload Mar 18 19:53:19.380944319 (296h53m16.977715346s) [upload=332531]: Done starting upload Mar 18 19:53:19.380976457 (296h53m16.977747482s) [upload=332531]: Uploading segments... Mar 18 19:53:19.381333196 (296h53m16.978104222s) [upload=332531]: Got next segment. Inline: false Mar 18 19:53:19.381379068 (296h53m16.978150099s) [upload=332531]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 19:53:19.381386543 (296h53m16.978157579s) [upload=332531]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 18 19:53:23.490845546 (296h53m21.087616603s) [seg_pos={0 0}, upload=332531]: Begin upload segment... Mar 18 19:53:23.490917951 (296h53m21.087688988s) [seg_pos={0 0}, upload=332531]: Done begin upload segment. Mar 18 19:53:23.490955926 (296h53m21.087726957s) [noise=true , address=45.91.116.5:4312 , piece_id=BDB6LYZZH2HXC3WWPDVMDQLCAAGRXBM5OCZU2JQVUJ7E6REVJ5SQ, seg_pos={0 0}, upload=332531]: Uploading piece... Mar 18 19:53:23.492093139 (296h53m21.088864171s) [noise=true , address=193.149.164.67:4003 , piece_id=SOPPFFG22JZWQA5NHYGA2OFHC2KMALZOR3ATBQ6JYWJVXATWLSRQ, seg_pos={0 0}, upload=332531]: Uploading piece... Mar 18 19:53:23.492835295 (296h53m21.089606329s) [noise=true , address=77.90.31.18:4833 , piece_id=ON3SKTN2H26XEXC43C44MAJIPY32NZLNWRU4PBSRJS6HLMXVNBMA, seg_pos={0 0}, upload=332531]: Uploading piece... Mar 18 19:53:23.492958664 (296h53m21.089729699s) [noise=true , address=45.91.116.13:4167 , piece_id=3JWBJU4M3WSUM3D2TVZDGJKBSRXZY5ZQHV3GFSBKOJMLORY5W7IQ, seg_pos={0 0}, upload=332531]: Uploading piece... Mar 18 19:53:23.493624498 (296h53m21.090395533s) [noise=true , address=145.79.227.13:4814 , piece_id=3KU6BHQSH4U4XYBSV4I77FCLQKR5RKWFY3QKZ26D5QRCJDQPKTKQ, seg_pos={0 0}, upload=332531]: Uploading piece... Mar 18 19:53:23.493650442 (296h53m21.090421482s) [noise=true , address=79.127.205.237:4230 , piece_id=Y6AOKOD27V5RL6QWUY2ZAJMONWPTYU4TTU5VIAYJ4RVMNQMCRUZQ, seg_pos={0 0}, upload=332531]: Uploading piece... Mar 18 19:53:23.494253839 (296h53m21.091024872s) [noise=true , address=45.91.116.14:4223 , piece_id=X3NSJBVWEZ3F3DUHQOSQSUCECIN7FARCFMRPJR6N7FHIIL3MS3NQ, seg_pos={0 0}, upload=332531]: Uploading piece... Mar 18 19:53:23.494267618 (296h53m21.091038645s) [noise=true , address=145.79.227.22:4516 , piece_id=OES5UD4FMNBSPC4ZSDS45SRXLTAQT3QNGR6JBI5D5Y363JFGJ73Q, seg_pos={0 0}, upload=332531]: Uploading piece... Mar 18 19:53:23.494879425 (296h53m21.091650453s) [noise=true , address=193.149.164.71:4287 , piece_id=ZE2Q4NHL3KUKRB36IDRMIZ7DIKDYZIBCM2L3PTLXALGL3PINBGAQ, seg_pos={0 0}, upload=332531]: Uploading piece... Mar 18 19:53:23.494911912 (296h53m21.091682944s) [noise=true , address=193.149.164.69:4074 , piece_id=FUGRHBCQHA4YJ2EVCMKVQVITADSVLM3PWZHV2XM3QLRDGU4G2GPA, seg_pos={0 0}, upload=332531]: Uploading piece... Mar 18 19:53:23.495497539 (296h53m21.09226857s) [noise=true , address=193.149.164.68:4066 , piece_id=FY5RBQUD2JCNXJCY2KIK744F4QZFHTW45PBDMASIIP27RWWB7SSA, seg_pos={0 0}, upload=332531]: Uploading piece... Mar 18 19:53:23.496168841 (296h53m21.092939872s) [noise=true , address=145.79.227.41:4105 , piece_id=RTYHA5YP42CFAYVM6PHY5RTB47GRMWSZEBBPNLC532CXKPBT2VXA, seg_pos={0 0}, upload=332531]: Uploading piece... Mar 18 19:53:23.497008065 (296h53m21.093779105s) [noise=true , address=79.127.205.235:4271 , piece_id=5XULCXAVIG3CQFTYA6Q3TFPRVICQFHQI2MSI2VIRIAGSK5UKKFXQ, seg_pos={0 0}, upload=332531]: Uploading piece... Mar 18 19:53:23.497605357 (296h53m21.094376396s) [noise=true , address=145.79.227.37:4590 , piece_id=ZTQLQ2IZHJFZWBHMRT74V7JDZYXF5BG4656VBZDJ4J6NGKS2TMOA, seg_pos={0 0}, upload=332531]: Uploading piece... Mar 18 19:53:23.498200357 (296h53m21.094971388s) [noise=true , address=193.149.164.66:4989 , piece_id=DVAJPRB7ABJJGUP4TYMLOGJVDLUARAMSPCG6FE2PVQXIDP2QBPGA, seg_pos={0 0}, upload=332531]: Uploading piece... Mar 18 19:53:23.498757906 (296h53m21.095528939s) [noise=true , address=79.127.219.45:4868 , piece_id=6NDAYV5PFDBOMZKAXKO6LZUODM46MQPQOQIDKBHER3LG4CBEDHKQ, seg_pos={0 0}, upload=332531]: Uploading piece... Mar 18 19:53:23.499215136 (296h53m21.095986159s) [noise=true , address=145.79.227.52:4453 , piece_id=YRAA65GHFFR45DX7MKR5BWBXID3YVSA4DMQDRKWKRDTJCHKHGMOA, seg_pos={0 0}, upload=332531]: Uploading piece... Mar 18 19:53:23.499687663 (296h53m21.096458697s) [noise=true , address=77.90.31.14:4678 , piece_id=NWBLAAUF3GACR3Y6JRTHDEUJ24GMOTZIPSRGJIPTPLIVWBCSYAHA, seg_pos={0 0}, upload=332531]: Uploading piece... Mar 18 19:53:23.600142173 (296h53m21.196913186s) [upload=332531]: Next returned nil segment Mar 18 19:53:23.600161212 (296h53m21.196932236s) [upload=332531]: Waiting for error group managing segments... Mar 18 19:53:23.620885875 (296h53m21.217656899s) [noise=true , address=45.91.116.5:4312 , piece_id=BDB6LYZZH2HXC3WWPDVMDQLCAAGRXBM5OCZU2JQVUJ7E6REVJ5SQ, seg_pos={0 0}, upload=332531]: Done uploading piece. err: <nil> Mar 18 19:53:23.621154396 (296h53m21.217925412s) [noise=true , address=45.91.116.13:4167 , piece_id=3JWBJU4M3WSUM3D2TVZDGJKBSRXZY5ZQHV3GFSBKOJMLORY5W7IQ, seg_pos={0 0}, upload=332531]: Done uploading piece. err: <nil> Mar 18 19:53:23.628883995 (296h53m21.225655021s) [noise=true , address=45.91.116.14:4223 , piece_id=X3NSJBVWEZ3F3DUHQOSQSUCECIN7FARCFMRPJR6N7FHIIL3MS3NQ, seg_pos={0 0}, upload=332531]: Done uploading piece. err: <nil> Mar 18 19:53:23.630970813 (296h53m21.227741838s) [noise=true , address=79.127.219.45:4868 , piece_id=6NDAYV5PFDBOMZKAXKO6LZUODM46MQPQOQIDKBHER3LG4CBEDHKQ, seg_pos={0 0}, upload=332531]: Done uploading piece. err: <nil> Mar 18 19:53:23.639825090 (296h53m21.236596104s) [noise=true , address=77.90.31.14:4678 , piece_id=NWBLAAUF3GACR3Y6JRTHDEUJ24GMOTZIPSRGJIPTPLIVWBCSYAHA, seg_pos={0 0}, upload=332531]: Done uploading piece. err: <nil> Mar 18 19:53:23.642754677 (296h53m21.239525692s) [noise=true , address=77.90.31.18:4833 , piece_id=ON3SKTN2H26XEXC43C44MAJIPY32NZLNWRU4PBSRJS6HLMXVNBMA, seg_pos={0 0}, upload=332531]: Done uploading piece. err: <nil> Mar 18 19:53:24.057422272 (296h53m21.654193306s) [noise=true , address=79.127.205.235:4271 , piece_id=5XULCXAVIG3CQFTYA6Q3TFPRVICQFHQI2MSI2VIRIAGSK5UKKFXQ, seg_pos={0 0}, upload=332531]: Done uploading piece. err: <nil> Mar 18 19:53:24.074244577 (296h53m21.671015604s) [noise=true , address=193.149.164.67:4003 , piece_id=SOPPFFG22JZWQA5NHYGA2OFHC2KMALZOR3ATBQ6JYWJVXATWLSRQ, seg_pos={0 0}, upload=332531]: Done uploading piece. err: <nil> Mar 18 19:53:24.097678734 (296h53m21.694449747s) [noise=true , address=79.127.205.237:4230 , piece_id=Y6AOKOD27V5RL6QWUY2ZAJMONWPTYU4TTU5VIAYJ4RVMNQMCRUZQ, seg_pos={0 0}, upload=332531]: Done uploading piece. err: <nil> Mar 18 19:53:24.109685982 (296h53m21.706457009s) [noise=true , address=193.149.164.68:4066 , piece_id=FY5RBQUD2JCNXJCY2KIK744F4QZFHTW45PBDMASIIP27RWWB7SSA, seg_pos={0 0}, upload=332531]: Done uploading piece. err: <nil> Mar 18 19:53:24.110659716 (296h53m21.707430744s) [noise=true , address=193.149.164.69:4074 , piece_id=FUGRHBCQHA4YJ2EVCMKVQVITADSVLM3PWZHV2XM3QLRDGU4G2GPA, seg_pos={0 0}, upload=332531]: Done uploading piece. err: <nil> Mar 18 19:53:24.114541274 (296h53m21.7113123s) [noise=true , address=193.149.164.71:4287 , piece_id=ZE2Q4NHL3KUKRB36IDRMIZ7DIKDYZIBCM2L3PTLXALGL3PINBGAQ, seg_pos={0 0}, upload=332531]: Done uploading piece. err: <nil> Mar 18 19:53:24.140894470 (296h53m21.737665488s) [noise=true , address=193.149.164.66:4989 , piece_id=DVAJPRB7ABJJGUP4TYMLOGJVDLUARAMSPCG6FE2PVQXIDP2QBPGA, seg_pos={0 0}, upload=332531]: Done uploading piece. err: <nil> Mar 18 19:54:19.329327996 (296h54m16.926099018s) [noise=true , address=145.79.227.13:4814 , piece_id=3KU6BHQSH4U4XYBSV4I77FCLQKR5RKWFY3QKZ26D5QRCJDQPKTKQ, seg_pos={0 0}, upload=332531]: Done uploading piece. err: ecclient: upload failed (node:12ERtrzMuh8rZPsgebtyScmYvNVwz3UKJr5Nu2hLdu6sn8X3qLa, address:145.79.227.13:4814): protocol: rpc: dial tcp 145.79.227.13:4814: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 19:54:19.329567681 (296h54m16.926338706s) [noise=true , address=145.79.227.52:4453 , piece_id=YRAA65GHFFR45DX7MKR5BWBXID3YVSA4DMQDRKWKRDTJCHKHGMOA, seg_pos={0 0}, upload=332531]: Done uploading piece. err: ecclient: upload failed (node:1BmpsQ9VfhGzmkZHvSeMptcGhzGpdoN4oYvi83qZcC1Vh9anhP, address:145.79.227.52:4453): protocol: rpc: dial tcp 145.79.227.52:4453: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 19:54:19.329755822 (296h54m16.926526848s) [noise=true , address=145.79.227.41:4105 , piece_id=RTYHA5YP42CFAYVM6PHY5RTB47GRMWSZEBBPNLC532CXKPBT2VXA, seg_pos={0 0}, upload=332531]: Done uploading piece. err: ecclient: upload failed (node:1TVtMQNVV2kERp2aZwmS8E9JUiua1RCyFpZgFACRqH1Nfscw4C, address:145.79.227.41:4105): protocol: rpc: dial tcp 145.79.227.41:4105: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 19:54:19.330021146 (296h54m16.926792175s) [noise=true , address=145.79.227.22:4516 , piece_id=OES5UD4FMNBSPC4ZSDS45SRXLTAQT3QNGR6JBI5D5Y363JFGJ73Q, seg_pos={0 0}, upload=332531]: Done uploading piece. err: ecclient: upload failed (node:12Lhy3AxzYoF25AxWdvjrrV5NNCwP1cDqkEvv4tz9t9KZmF8WPK, address:145.79.227.22:4516): protocol: rpc: dial tcp 145.79.227.22:4516: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 19:54:19.330213680 (296h54m16.926984707s) [noise=true , address=145.79.227.37:4590 , piece_id=ZTQLQ2IZHJFZWBHMRT74V7JDZYXF5BG4656VBZDJ4J6NGKS2TMOA, seg_pos={0 0}, upload=332531]: Done uploading piece. err: ecclient: upload failed (node:1KnNqY1WNwMSFXFrQsWPAhivUTgSn7dKTx1mTTzYG2UxGCkasj, address:145.79.227.37:4590): protocol: rpc: dial tcp 145.79.227.37:4590: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 19:54:19.330250832 (296h54m16.927021848s) [seg_pos={0 0}, upload=332531]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 18 19:54:19.351979131 (296h54m16.948750155s) [upload=332531]: Done uploading segments... Mar 18 19:54:19.352025297 (296h54m16.948796327s) [upload=332531]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 16.956659168s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 32 out of 60 is failed (threshold=3) [............................XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 30.025311856s trace: https://jaeger.dev.storj.io/trace/57fa0971ede26b0e FAIL (2026-03-18 19:59:42 UTC) operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 9) ======LOGS====== Mar 18 19:59:12.944142306 (296h59m10.540913329s) [upload=332633]: Starting upload Mar 18 19:59:12.944240508 (296h59m10.541011532s) [upload=332633]: Done starting upload Mar 18 19:59:12.944288725 (296h59m10.541059742s) [upload=332633]: Uploading segments... Mar 18 19:59:12.944551886 (296h59m10.541322902s) [upload=332633]: Got next segment. Inline: false Mar 18 19:59:12.944617978 (296h59m10.541389002s) [upload=332633]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 19:59:12.944626253 (296h59m10.541397276s) [upload=332633]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 18 19:59:16.095841171 (296h59m13.692612194s) [seg_pos={0 0}, upload=332633]: Begin upload segment... Mar 18 19:59:16.095939621 (296h59m13.692710634s) [seg_pos={0 0}, upload=332633]: Done begin upload segment. Mar 18 19:59:16.096152839 (296h59m13.692923873s) [noise=true , address=145.79.227.12:4770 , piece_id=XVXGRV4NOP3GOYM344EEA3AWHIRLRIBUST6GPDHJH32XIUX6JY7A, seg_pos={0 0}, upload=332633]: Uploading piece... Mar 18 19:59:16.096187990 (296h59m13.692959004s) [noise=true , address=79.127.219.38:4822 , piece_id=3Q74DGUVMF5SH5EN3UWMQTHJ63ZSMD4BQBZYCJ6LG7LAVCREECUQ, seg_pos={0 0}, upload=332633]: Uploading piece... Mar 18 19:59:16.097218514 (296h59m13.693989527s) [noise=true , address=145.79.227.47:4268 , piece_id=IEXOBIRQUCTG5VPEPKP5DCCN3GOERW3ZHRS2WU5TAHLFMXRBG3XQ, seg_pos={0 0}, upload=332633]: Uploading piece... Mar 18 19:59:16.098285629 (296h59m13.695056646s) [noise=true , address=79.127.219.46:4892 , piece_id=D6VHKSNID26SHO7R7G3BQYX3DIKNSBE3FG3SYF32XCXINOU6CXPQ, seg_pos={0 0}, upload=332633]: Uploading piece... Mar 18 19:59:16.099115265 (296h59m13.695886282s) [noise=true , address=77.90.31.10:4517 , piece_id=QRXKORPHUXX2JRV2M4UGXSUMPRUHJ7FKYPMHIO3HW72JMOEEJAXA, seg_pos={0 0}, upload=332633]: Uploading piece... Mar 18 19:59:16.100072781 (296h59m13.696843797s) [noise=true , address=145.79.227.52:4447 , piece_id=LTY2QVUDAVYX5ZR3LTIGQUF3JGMOI25EAIGB43HVLAZGZRO3XMRA, seg_pos={0 0}, upload=332633]: Uploading piece... Mar 18 19:59:16.100434721 (296h59m13.697205737s) [noise=true , address=45.91.116.14:4189 , piece_id=EEZVINBVRM55FKO6LCJVPWVIL4LBYAQNA5CW3VYA54LFCU5XTM4Q, seg_pos={0 0}, upload=332633]: Uploading piece... Mar 18 19:59:16.100924433 (296h59m13.69769545s) [noise=true , address=145.79.227.36:4520 , piece_id=RED5UFSKYB23KUIS6NER43ZW3XQKXT2MBG7MMHMD5CMUD54HCALA, seg_pos={0 0}, upload=332633]: Uploading piece... Mar 18 19:59:16.101589544 (296h59m13.69836057s) [noise=true , address=145.79.227.2:4119 , piece_id=FDJQVYEWERQWPQEOVZ6F4ULKUS4UDGXXMU435EES5TM6HUZ5PNAQ, seg_pos={0 0}, upload=332633]: Uploading piece... Mar 18 19:59:16.101799070 (296h59m13.698570098s) [noise=true , address=77.90.31.11:4554 , piece_id=QXQWNC6KPKT3RLWN3B5PEVLMW5XV4SI6LPZUMGZWK2JXVUZHMYLQ, seg_pos={0 0}, upload=332633]: Uploading piece... Mar 18 19:59:16.102708292 (296h59m13.699479302s) [noise=true , address=45.91.116.7:4011 , piece_id=AJAKY6NQ2UUKJAO4MB5JMHVHPCHBJYTR4MZYHXEJL6S7F77OB47Q, seg_pos={0 0}, upload=332633]: Uploading piece... Mar 18 19:59:16.102931983 (296h59m13.699703s) [noise=true , address=77.90.31.18:4809 , piece_id=VEHSQO2JFVES25MCTRS7G37Z7C6FQ5N7F3YKAKPW6MUZ6RIGNCOA, seg_pos={0 0}, upload=332633]: Uploading piece... Mar 18 19:59:16.103735099 (296h59m13.700506115s) [noise=true , address=45.91.116.12:4226 , piece_id=FJQHXY73TKW5DJDVGJINPQP5XEXA37JP2JQ4O2GOXB3FKJU4QFZQ, seg_pos={0 0}, upload=332633]: Uploading piece... Mar 18 19:59:16.104913539 (296h59m13.701684556s) [noise=true , address=77.90.31.19:4845 , piece_id=FODJWUO232AORBN7Z5IB7GXCFHPDWC4F4A6CJHVXTASIJXDIOD5Q, seg_pos={0 0}, upload=332633]: Uploading piece... Mar 18 19:59:16.105924389 (296h59m13.702695406s) [noise=true , address=145.79.227.44:4193 , piece_id=GYV4UHB7O6MIQ5NQFPLVNGWVPUOWNML6VGE4HHO3OYS6Q2YX5URA, seg_pos={0 0}, upload=332633]: Uploading piece... Mar 18 19:59:16.106611801 (296h59m13.703382818s) [noise=true , address=145.79.227.5:4224 , piece_id=HH2LU4HQ3MRMZR64UPNMBG7CXGNFFBWY6YLAK4OVIRT4ET3BD3CQ, seg_pos={0 0}, upload=332633]: Uploading piece... Mar 18 19:59:16.172191860 (296h59m13.768962893s) [upload=332633]: Next returned nil segment Mar 18 19:59:16.172212743 (296h59m13.768983776s) [upload=332633]: Waiting for error group managing segments... Mar 18 19:59:16.175682438 (296h59m13.77245346s) [noise=true , address=45.91.116.12:4226 , piece_id=FJQHXY73TKW5DJDVGJINPQP5XEXA37JP2JQ4O2GOXB3FKJU4QFZQ, seg_pos={0 0}, upload=332633]: Done uploading piece. err: <nil> Mar 18 19:59:16.176108817 (296h59m13.772879843s) [noise=true , address=45.91.116.14:4189 , piece_id=EEZVINBVRM55FKO6LCJVPWVIL4LBYAQNA5CW3VYA54LFCU5XTM4Q, seg_pos={0 0}, upload=332633]: Done uploading piece. err: <nil> Mar 18 19:59:16.178055413 (296h59m13.774826445s) [noise=true , address=77.90.31.11:4554 , piece_id=QXQWNC6KPKT3RLWN3B5PEVLMW5XV4SI6LPZUMGZWK2JXVUZHMYLQ, seg_pos={0 0}, upload=332633]: Done uploading piece. err: <nil> Mar 18 19:59:16.178257555 (296h59m13.775028582s) [noise=true , address=77.90.31.18:4809 , piece_id=VEHSQO2JFVES25MCTRS7G37Z7C6FQ5N7F3YKAKPW6MUZ6RIGNCOA, seg_pos={0 0}, upload=332633]: Done uploading piece. err: <nil> Mar 18 19:59:16.179238886 (296h59m13.77600992s) [noise=true , address=77.90.31.19:4845 , piece_id=FODJWUO232AORBN7Z5IB7GXCFHPDWC4F4A6CJHVXTASIJXDIOD5Q, seg_pos={0 0}, upload=332633]: Done uploading piece. err: <nil> Mar 18 19:59:16.179293136 (296h59m13.776064151s) [noise=true , address=77.90.31.10:4517 , piece_id=QRXKORPHUXX2JRV2M4UGXSUMPRUHJ7FKYPMHIO3HW72JMOEEJAXA, seg_pos={0 0}, upload=332633]: Done uploading piece. err: <nil> Mar 18 19:59:16.179959283 (296h59m13.776730313s) [noise=true , address=79.127.219.46:4892 , piece_id=D6VHKSNID26SHO7R7G3BQYX3DIKNSBE3FG3SYF32XCXINOU6CXPQ, seg_pos={0 0}, upload=332633]: Done uploading piece. err: <nil> Mar 18 19:59:16.191720734 (296h59m13.788491758s) [noise=true , address=45.91.116.7:4011 , piece_id=AJAKY6NQ2UUKJAO4MB5JMHVHPCHBJYTR4MZYHXEJL6S7F77OB47Q, seg_pos={0 0}, upload=332633]: Done uploading piece. err: <nil> Mar 18 19:59:16.211735979 (296h59m13.808507035s) [noise=true , address=79.127.219.38:4822 , piece_id=3Q74DGUVMF5SH5EN3UWMQTHJ63ZSMD4BQBZYCJ6LG7LAVCREECUQ, seg_pos={0 0}, upload=332633]: Done uploading piece. err: <nil> Mar 18 19:59:42.912072381 (296h59m40.508843415s) [noise=true , address=145.79.227.12:4770 , piece_id=XVXGRV4NOP3GOYM344EEA3AWHIRLRIBUST6GPDHJH32XIUX6JY7A, seg_pos={0 0}, upload=332633]: Done uploading piece. err: ecclient: upload failed (node:1Ug6GpYToYDXiHmiJP4GoE7eJV3kqRveix2S5SLDcKQJtEbNPz, address:145.79.227.12:4770): protocol: rpc: dial tcp 145.79.227.12:4770: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 19:59:42.912096113 (296h59m40.508867145s) [noise=true , address=145.79.227.47:4268 , piece_id=IEXOBIRQUCTG5VPEPKP5DCCN3GOERW3ZHRS2WU5TAHLFMXRBG3XQ, seg_pos={0 0}, upload=332633]: Done uploading piece. err: ecclient: upload failed (node:165gcpfWaKHcMWugoiB5cTK8hpFYp1iymG8xnBdLgPuA2o1wkj, address:145.79.227.47:4268): protocol: rpc: dial tcp 145.79.227.47:4268: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 19:59:42.912463296 (296h59m40.50923433s) [noise=true , address=145.79.227.2:4119 , piece_id=FDJQVYEWERQWPQEOVZ6F4ULKUS4UDGXXMU435EES5TM6HUZ5PNAQ, seg_pos={0 0}, upload=332633]: Done uploading piece. err: ecclient: upload failed (node:121xNAnkm9HZXUszp26UmWTsuhp3igyeDVea6gSCdtrQUQDVHpW, address:145.79.227.2:4119): protocol: rpc: dial tcp 145.79.227.2:4119: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 19:59:42.912742760 (296h59m40.509513794s) [noise=true , address=145.79.227.52:4447 , piece_id=LTY2QVUDAVYX5ZR3LTIGQUF3JGMOI25EAIGB43HVLAZGZRO3XMRA, seg_pos={0 0}, upload=332633]: Done uploading piece. err: ecclient: upload failed (node:12fqT9xA7kXTG8K9XwTUxL6FXibV35AkPHDK6grKGHg17fk91fP, address:145.79.227.52:4447): protocol: rpc: dial tcp 145.79.227.52:4447: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 19:59:42.913065012 (296h59m40.509836043s) [noise=true , address=145.79.227.36:4520 , piece_id=RED5UFSKYB23KUIS6NER43ZW3XQKXT2MBG7MMHMD5CMUD54HCALA, seg_pos={0 0}, upload=332633]: Done uploading piece. err: ecclient: upload failed (node:12eNRsboM5zFiyBVPLuQfgb5QSAc5bZFizK9KJHudNd9nstVGyy, address:145.79.227.36:4520): protocol: rpc: dial tcp 145.79.227.36:4520: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 19:59:42.913327892 (296h59m40.510098924s) [noise=true , address=145.79.227.5:4224 , piece_id=HH2LU4HQ3MRMZR64UPNMBG7CXGNFFBWY6YLAK4OVIRT4ET3BD3CQ, seg_pos={0 0}, upload=332633]: Done uploading piece. err: ecclient: upload failed (node:128nLffK44ioiTpjWjAm2vHFTLouLZdw7BijqBAbsdjo99kPYNo, address:145.79.227.5:4224): protocol: rpc: dial tcp 145.79.227.5:4224: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 19:59:42.913415538 (296h59m40.510186572s) [noise=true , address=145.79.227.44:4193 , piece_id=GYV4UHB7O6MIQ5NQFPLVNGWVPUOWNML6VGE4HHO3OYS6Q2YX5URA, seg_pos={0 0}, upload=332633]: Done uploading piece. err: ecclient: upload failed (node:1EXja3EXj5TJmk2j46Bnc6TXT7N3wgxCLjzAEjbawqvtQU7BF8, address:145.79.227.44:4193): protocol: rpc: dial tcp 145.79.227.44:4193: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 19:59:42.913451959 (296h59m40.510222978s) [seg_pos={0 0}, upload=332633]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Mar 18 19:59:42.935231354 (296h59m40.532002389s) [upload=332633]: Done uploading segments... Mar 18 19:59:42.935273834 (296h59m40.532044866s) [upload=332633]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Duration: 30.025331775s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 6 out of 60 is failed (threshold=3) [................................XXX..........X...X.........X] duration: 1m0.025074876s trace: https://jaeger.dev.storj.io/trace/7c859732d77a7da7 FAIL (2026-03-18 19:59:34 UTC) operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== d=332621]: Uploading segments... Mar 18 19:58:34.964515697 (296h58m32.561286706s) [upload=332621]: Got next segment. Inline: false Mar 18 19:58:34.964555310 (296h58m32.561326334s) [upload=332621]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 19:58:34.964560227 (296h58m32.561331251s) [upload=332621]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 18 19:58:38.382666205 (296h58m35.979437219s) [seg_pos={0 0}, upload=332621]: Begin upload segment... Mar 18 19:58:38.382729299 (296h58m35.979500329s) [seg_pos={0 0}, upload=332621]: Done begin upload segment. Mar 18 19:58:38.382760799 (296h58m35.979531826s) [noise=true , address=145.79.227.46:4607 , piece_id=LXBA2IGR4U6DLXE4NXL2CN23X5HXP7KFN5UCJZH6IBRYAZCQ7N3Q, seg_pos={0 0}, upload=332621]: Uploading piece... Mar 18 19:58:38.382881088 (296h58m35.979652115s) [noise=true , address=145.79.227.22:4538 , piece_id=ALPGNVFU5TWXZ7D7I5XFQBZ2CAN74VYUBNYDFSPG3HIHO7Z6M3AA, seg_pos={0 0}, upload=332621]: Uploading piece... Mar 18 19:58:38.383461716 (296h58m35.980232746s) [noise=true , address=167.17.72.70:4770 , piece_id=5H5RNTJDXOLXRNVU4XG5FP4O5YN5PW2XQH4O2UIX5XBKNMO35WKQ, seg_pos={0 0}, upload=332621]: Uploading piece... Mar 18 19:58:38.384266580 (296h58m35.981037606s) [noise=true , address=45.91.116.10:4092 , piece_id=Y7E2ZWR5D6M7QKRY3I7SL3ZJN32EK36ZWPP3YRW5WCGQFZCNU64Q, seg_pos={0 0}, upload=332621]: Uploading piece... Mar 18 19:58:38.384321708 (296h58m35.981092731s) [noise=true , address=79.127.205.235:4273 , piece_id=CQEOZ6GWLY5MEVRQZHYH6AP6DXQY5WKIAHCHDBJMAI77STRNINYA, seg_pos={0 0}, upload=332621]: Uploading piece... Mar 18 19:58:38.385197892 (296h58m35.981968918s) [noise=true , address=193.149.164.66:4989 , piece_id=EDE7OF4G36VJ7GXFMF47ZI73D6I7OWZSC5AXESS6W74ZDWTBDXWA, seg_pos={0 0}, upload=332621]: Uploading piece... Mar 18 19:58:38.385232613 (296h58m35.982003642s) [noise=true , address=45.91.116.13:4171 , piece_id=74JRIA5AS6SMECS3FFCCVLH3AY5N4EPNWOEDZGITBPCQZEKDGEAQ, seg_pos={0 0}, upload=332621]: Uploading piece... Mar 18 19:58:38.385334543 (296h58m35.982105567s) [noise=true , address=193.149.164.70:4120 , piece_id=AO6H5CLNNY2CQ27JO57WEKF7OTFQA7IXAZMVG6A4EWJLZT7NIRSQ, seg_pos={0 0}, upload=332621]: Uploading piece... Mar 18 19:58:38.386120289 (296h58m35.982891315s) [noise=true , address=79.127.219.46:4908 , piece_id=L2EOXVK7S2ZSQNK47543XTNRWJP2PU57DH2XHZ2BO5OZUTDXDQFA, seg_pos={0 0}, upload=332621]: Uploading piece... Mar 18 19:58:38.386244430 (296h58m35.983015464s) [noise=true , address=79.127.205.237:4236 , piece_id=FKQ3NUK5AUFD35L25PJFPVDLZUVNRURK5RLSPG3GGC75QYTEWP4A, seg_pos={0 0}, upload=332621]: Uploading piece... Mar 18 19:58:38.387158310 (296h58m35.983929342s) [noise=true , address=64.49.13.140:4516 , piece_id=D2V5WAW6D46VSSNBFGNO5ETQNCOTDHQYWM62BSSVF53X2MWU3TYQ, seg_pos={0 0}, upload=332621]: Uploading piece... Mar 18 19:58:38.387233109 (296h58m35.984004132s) [noise=true , address=193.149.164.67:4020 , piece_id=NZQGIA6BVAK3Z6SQZA7WNAOAY25VQKBDTWCRV34KNSYTGVNDSWUA, seg_pos={0 0}, upload=332621]: Uploading piece... Mar 18 19:58:38.388077887 (296h58m35.984848921s) [noise=true , address=145.79.227.49:4343 , piece_id=NCJH6BOJHPW5OTSCNGKTZCZYC3F3LTRJKGVOKCJODWSMHUAATJQQ, seg_pos={0 0}, upload=332621]: Uploading piece... Mar 18 19:58:38.388338947 (296h58m35.985109978s) [noise=true , address=79.127.219.41:4940 , piece_id=C2ODTVQIR73WTRBM7EDKEC54DOBK67L2RV4UXPHHTRY6KL3EQIKQ, seg_pos={0 0}, upload=332621]: Uploading piece... Mar 18 19:58:38.388792743 (296h58m35.985563766s) [noise=true , address=145.79.227.39:4043 , piece_id=D757XSQF27CMMI5UPBRB7LLGISWOYLW7OS3UICYF3U6AI62X73PQ, seg_pos={0 0}, upload=332621]: Uploading piece... Mar 18 19:58:38.389305068 (296h58m35.986076097s) [noise=true , address=45.91.116.9:4280 , piece_id=QECGXDF7C5YUJNURQCU76PMJM5MZ3DCEGPGAJHCZWI3H3OVGERGQ, seg_pos={0 0}, upload=332621]: Uploading piece... Mar 18 19:58:38.389771675 (296h58m35.986542707s) [noise=true , address=145.79.227.36:4528 , piece_id=FJ5LKOZJYVNWLZ5I6OUCZMPFDOBVZXQRVPHREHPQX6J3YWAI2KFA, seg_pos={0 0}, upload=332621]: Uploading piece... Mar 18 19:58:38.390707637 (296h58m35.98747867s) [noise=true , address=77.90.31.18:4827 , piece_id=SGKGAEAHEHU2H4YRLYQLQMTS45U6ZNSAYJ6CYJUGGKEDRMQT6BHQ, seg_pos={0 0}, upload=332621]: Uploading piece... Mar 18 19:58:38.466570556 (296h58m36.063341572s) [upload=332621]: Next returned nil segment Mar 18 19:58:38.466587006 (296h58m36.063358039s) [upload=332621]: Waiting for error group managing segments... Mar 18 19:58:38.471637588 (296h58m36.068408605s) [noise=true , address=45.91.116.9:4280 , piece_id=QECGXDF7C5YUJNURQCU76PMJM5MZ3DCEGPGAJHCZWI3H3OVGERGQ, seg_pos={0 0}, upload=332621]: Done uploading piece. err: <nil> Mar 18 19:58:38.474621081 (296h58m36.071392109s) [noise=true , address=79.127.219.41:4940 , piece_id=C2ODTVQIR73WTRBM7EDKEC54DOBK67L2RV4UXPHHTRY6KL3EQIKQ, seg_pos={0 0}, upload=332621]: Done uploading piece. err: <nil> Mar 18 19:58:38.479487991 (296h58m36.076259007s) [noise=true , address=77.90.31.18:4827 , piece_id=SGKGAEAHEHU2H4YRLYQLQMTS45U6ZNSAYJ6CYJUGGKEDRMQT6BHQ, seg_pos={0 0}, upload=332621]: Done uploading piece. err: <nil> Mar 18 19:58:38.479685133 (296h58m36.076456159s) [noise=true , address=45.91.116.10:4092 , piece_id=Y7E2ZWR5D6M7QKRY3I7SL3ZJN32EK36ZWPP3YRW5WCGQFZCNU64Q, seg_pos={0 0}, upload=332621]: Done uploading piece. err: <nil> Mar 18 19:58:38.485438319 (296h58m36.082209345s) [noise=true , address=45.91.116.13:4171 , piece_id=74JRIA5AS6SMECS3FFCCVLH3AY5N4EPNWOEDZGITBPCQZEKDGEAQ, seg_pos={0 0}, upload=332621]: Done uploading piece. err: <nil> Mar 18 19:58:38.620281221 (296h58m36.217052236s) [noise=true , address=79.127.219.46:4908 , piece_id=L2EOXVK7S2ZSQNK47543XTNRWJP2PU57DH2XHZ2BO5OZUTDXDQFA, seg_pos={0 0}, upload=332621]: Done uploading piece. err: <nil> Mar 18 19:58:38.772264604 (296h58m36.369035618s) [noise=true , address=167.17.72.70:4770 , piece_id=5H5RNTJDXOLXRNVU4XG5FP4O5YN5PW2XQH4O2UIX5XBKNMO35WKQ, seg_pos={0 0}, upload=332621]: Done uploading piece. err: <nil> Mar 18 19:58:38.967982420 (296h58m36.564753435s) [noise=true , address=79.127.205.235:4273 , piece_id=CQEOZ6GWLY5MEVRQZHYH6AP6DXQY5WKIAHCHDBJMAI77STRNINYA, seg_pos={0 0}, upload=332621]: Done uploading piece. err: <nil> Mar 18 19:58:38.977759060 (296h58m36.57453009s) [noise=true , address=79.127.205.237:4236 , piece_id=FKQ3NUK5AUFD35L25PJFPVDLZUVNRURK5RLSPG3GGC75QYTEWP4A, seg_pos={0 0}, upload=332621]: Done uploading piece. err: <nil> Mar 18 19:58:39.005320695 (296h58m36.602091708s) [noise=true , address=193.149.164.70:4120 , piece_id=AO6H5CLNNY2CQ27JO57WEKF7OTFQA7IXAZMVG6A4EWJLZT7NIRSQ, seg_pos={0 0}, upload=332621]: Done uploading piece. err: <nil> Mar 18 19:58:39.025848813 (296h58m36.622619842s) [noise=true , address=193.149.164.67:4020 , piece_id=NZQGIA6BVAK3Z6SQZA7WNAOAY25VQKBDTWCRV34KNSYTGVNDSWUA, seg_pos={0 0}, upload=332621]: Done uploading piece. err: <nil> Mar 18 19:58:39.065107032 (296h58m36.661878045s) [noise=true , address=64.49.13.140:4516 , piece_id=D2V5WAW6D46VSSNBFGNO5ETQNCOTDHQYWM62BSSVF53X2MWU3TYQ, seg_pos={0 0}, upload=332621]: Done uploading piece. err: <nil> Mar 18 19:58:39.201750353 (296h58m36.798521377s) [noise=true , address=193.149.164.66:4989 , piece_id=EDE7OF4G36VJ7GXFMF47ZI73D6I7OWZSC5AXESS6W74ZDWTBDXWA, seg_pos={0 0}, upload=332621]: Done uploading piece. err: <nil> Mar 18 19:59:34.932832250 (296h59m32.529603284s) [noise=true , address=145.79.227.46:4607 , piece_id=LXBA2IGR4U6DLXE4NXL2CN23X5HXP7KFN5UCJZH6IBRYAZCQ7N3Q, seg_pos={0 0}, upload=332621]: Done uploading piece. err: ecclient: upload failed (node:1zDoLfA5bQWM7Uf479gfFVjG3yYvxG5EZS5yZSFGTvoFoPYEAo, address:145.79.227.46:4607): protocol: rpc: dial tcp 145.79.227.46:4607: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 19:59:34.932941758 (296h59m32.529712814s) [noise=true , address=145.79.227.22:4538 , piece_id=ALPGNVFU5TWXZ7D7I5XFQBZ2CAN74VYUBNYDFSPG3HIHO7Z6M3AA, seg_pos={0 0}, upload=332621]: Done uploading piece. err: ecclient: upload failed (node:1kNFzPpkDtwy4VFyzeFQReorJPEVsmdDAd8ULT8ZKoGde57P3v, address:145.79.227.22:4538): protocol: rpc: dial tcp 145.79.227.22:4538: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 19:59:34.933267818 (296h59m32.53003887s) [noise=true , address=145.79.227.49:4343 , piece_id=NCJH6BOJHPW5OTSCNGKTZCZYC3F3LTRJKGVOKCJODWSMHUAATJQQ, seg_pos={0 0}, upload=332621]: Done uploading piece. err: ecclient: upload failed (node:16EsEMRW53sjKVag2WpSDGLhELyE2bSXyW4aAurCvDFKRNkXuf, address:145.79.227.49:4343): protocol: rpc: dial tcp 145.79.227.49:4343: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 19:59:34.933359674 (296h59m32.530130694s) [noise=true , address=145.79.227.36:4528 , piece_id=FJ5LKOZJYVNWLZ5I6OUCZMPFDOBVZXQRVPHREHPQX6J3YWAI2KFA, seg_pos={0 0}, upload=332621]: Done uploading piece. err: ecclient: upload failed (node:12ArcCrcHdpNgBMfTcPdtTXb65RgnTh8N32ihHAx2RrddxnRYgT, address:145.79.227.36:4528): protocol: rpc: dial tcp 145.79.227.36:4528: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 19:59:34.933622410 (296h59m32.530393477s) [noise=true , address=145.79.227.39:4043 , piece_id=D757XSQF27CMMI5UPBRB7LLGISWOYLW7OS3UICYF3U6AI62X73PQ, seg_pos={0 0}, upload=332621]: Done uploading piece. err: ecclient: upload failed (node:12f97TAyWcE4YrxDQCeycC8tnuetQiagu2MV4U5o8ARLq8jGKTX, address:145.79.227.39:4043): protocol: rpc: dial tcp 145.79.227.39:4043: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 19:59:34.933677971 (296h59m32.530448984s) [seg_pos={0 0}, upload=332621]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 18 19:59:34.955309631 (296h59m32.552080664s) [upload=332621]: Done uploading segments... Mar 18 19:59:34.955416416 (296h59m32.552187448s) [upload=332621]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 1m0.025212286s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 40 out of 60 is failed (threshold=6) [................XXXXXXXXX.XXX...XXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 5.001434453s trace: https://jaeger.dev.storj.io/trace/7ca099a50e3388fe FAIL (2026-03-18 19:59:34 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001475732s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 8 out of 60 is failed (threshold=3) [.............................XXXXXX.XX......................] duration: 14.357763547s trace: https://jaeger.dev.storj.io/trace/4b4d10a8051ad01b FAIL (2026-03-18 19:59:28 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 14.358618009s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 31 out of 60 is failed (threshold=3) [.............................XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 30.025017191s trace: https://jaeger.dev.storj.io/trace/81d7bf877db5e77 FAIL (2026-03-18 19:59:12 UTC) operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 7) ======LOGS====== 19:58:42.853808005 (296h58m40.45057902s) [upload=332623]: Got next segment. Inline: false Mar 18 19:58:42.853849621 (296h58m40.450620645s) [upload=332623]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 19:58:42.853862849 (296h58m40.450633862s) [upload=332623]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 18 19:58:46.229066847 (296h58m43.825837872s) [seg_pos={0 0}, upload=332623]: Begin upload segment... Mar 18 19:58:46.229232743 (296h58m43.826003766s) [seg_pos={0 0}, upload=332623]: Done begin upload segment. Mar 18 19:58:46.229281375 (296h58m43.826052401s) [noise=true , address=145.79.227.22:4512 , piece_id=YRJBQTCXG5X6G5V3FQCT2POZHCUMXNCVOMDNJJRBACVJAWPY5CAA, seg_pos={0 0}, upload=332623]: Uploading piece... Mar 18 19:58:46.230089393 (296h58m43.826860419s) [noise=true , address=145.79.227.45:4234 , piece_id=HF335WOWCKMG45FPC4HRFSWLWT6XXZF6SQ73ZXAVJNB22ZTK33RA, seg_pos={0 0}, upload=332623]: Uploading piece... Mar 18 19:58:46.230194080 (296h58m43.826965112s) [noise=true , address=145.79.227.24:4585 , piece_id=YHW3ZWDOKCWVKO7JXL4XHYTTI2SFDYDYFJMP26RRG27P75CLIQ6Q, seg_pos={0 0}, upload=332623]: Uploading piece... Mar 18 19:58:46.230581374 (296h58m43.827352398s) [noise=true , address=45.91.116.10:4081 , piece_id=EF76JRIGJA4FD4AMV24UKMM5XMGCZNHLGBB73BHKCKXMF4TSP7VA, seg_pos={0 0}, upload=332623]: Uploading piece... Mar 18 19:58:46.230805983 (296h58m43.827577005s) [noise=true , address=145.79.227.46:4603 , piece_id=YDKENB3HNHHRQH6XGTL24AAXI7GV2JW42CHG45QEJ3EL7XTTKFZA, seg_pos={0 0}, upload=332623]: Uploading piece... Mar 18 19:58:46.230922602 (296h58m43.827693628s) [noise=true , address=45.91.116.11:4152 , piece_id=KDOS4DU3GRCBJGRN4KSNKDAWLQMWKMALPB6NIVXNFHB2E2Y2KXOA, seg_pos={0 0}, upload=332623]: Uploading piece... Mar 18 19:58:46.231408843 (296h58m43.828179869s) [noise=true , address=77.90.31.17:4774 , piece_id=OI6Y5HIOZ2ZXIYAWKP3SGVS6EP7QBAAH2YTYNRGRXYG55KRBUIOQ, seg_pos={0 0}, upload=332623]: Uploading piece... Mar 18 19:58:46.231544376 (296h58m43.828315407s) [noise=true , address=145.79.227.4:4191 , piece_id=C7NPTV4OCU7DSK6JWCXSP7CMBL5ABGJYK5HIRNJYIZ2FNIQYLIPA, seg_pos={0 0}, upload=332623]: Uploading piece... Mar 18 19:58:46.231768858 (296h58m43.828539881s) [noise=true , address=45.91.116.7:4010 , piece_id=SCVINHLW6JSLU5MZL72IAYMXURYVQ2GKI7JVI2TJQETLUG2EWZLQ, seg_pos={0 0}, upload=332623]: Uploading piece... Mar 18 19:58:46.232024483 (296h58m43.828795512s) [noise=true , address=145.79.227.33:4874 , piece_id=73BH73YP5SJWPJI45TXHDUCSH77PKMVZWIGVDQHRKFMPACOAMB7Q, seg_pos={0 0}, upload=332623]: Uploading piece... Mar 18 19:58:46.232134808 (296h58m43.828905831s) [noise=true , address=145.79.227.20:4440 , piece_id=WZIZCLSZSK7PAXAWNT74YUVZNUQ5OSSO5ETPB7D57QZPDOLGAVDA, seg_pos={0 0}, upload=332623]: Uploading piece... Mar 18 19:58:46.232474396 (296h58m43.829245428s) [noise=true , address=79.127.219.41:4929 , piece_id=2L4IZ2TK4B434LWPF74TMUJOE6MJFPH53DUO5MR6TBOJITNYHFVA, seg_pos={0 0}, upload=332623]: Uploading piece... Mar 18 19:58:46.232512479 (296h58m43.829283502s) [noise=true , address=45.91.116.9:4261 , piece_id=HNSLU5HXPP57HJAWSRS6TV4FXKI7IFZWEHUE5MELPXZCKSHOMTHA, seg_pos={0 0}, upload=332623]: Uploading piece... Mar 18 19:58:46.232975431 (296h58m43.829746454s) [noise=true , address=77.90.31.18:4809 , piece_id=A3GJ5PAIK3XRYVLI3NVPXBBGRJXBKNP2YTSEWTO7ACRZJNDKFRWQ, seg_pos={0 0}, upload=332623]: Uploading piece... Mar 18 19:58:46.235286445 (296h58m43.832057489s) [noise=true , address=145.79.227.23:4550 , piece_id=36QDBITILXOENDGIMDXMCETUC6ZUJ7HLMSYRC7ES5NMMKRLZMIXQ, seg_pos={0 0}, upload=332623]: Uploading piece... Mar 18 19:58:46.235804879 (296h58m43.832575902s) [noise=true , address=145.79.227.30:4762 , piece_id=FSYZBAVZTDS363COI2VXSFQHNAW7EBPIER6RRCN4SC4MUJ3QDUXA, seg_pos={0 0}, upload=332623]: Uploading piece... Mar 18 19:58:46.298317784 (296h58m43.895088808s) [upload=332623]: Next returned nil segment Mar 18 19:58:46.298331971 (296h58m43.895102993s) [upload=332623]: Waiting for error group managing segments... Mar 18 19:58:46.302951696 (296h58m43.89972272s) [noise=true , address=45.91.116.11:4152 , piece_id=KDOS4DU3GRCBJGRN4KSNKDAWLQMWKMALPB6NIVXNFHB2E2Y2KXOA, seg_pos={0 0}, upload=332623]: Done uploading piece. err: <nil> Mar 18 19:58:46.302972803 (296h58m43.899743827s) [noise=true , address=45.91.116.9:4261 , piece_id=HNSLU5HXPP57HJAWSRS6TV4FXKI7IFZWEHUE5MELPXZCKSHOMTHA, seg_pos={0 0}, upload=332623]: Done uploading piece. err: <nil> Mar 18 19:58:46.303866368 (296h58m43.900637391s) [noise=true , address=77.90.31.18:4809 , piece_id=A3GJ5PAIK3XRYVLI3NVPXBBGRJXBKNP2YTSEWTO7ACRZJNDKFRWQ, seg_pos={0 0}, upload=332623]: Done uploading piece. err: <nil> Mar 18 19:58:46.305626773 (296h58m43.902397796s) [noise=true , address=77.90.31.17:4774 , piece_id=OI6Y5HIOZ2ZXIYAWKP3SGVS6EP7QBAAH2YTYNRGRXYG55KRBUIOQ, seg_pos={0 0}, upload=332623]: Done uploading piece. err: <nil> Mar 18 19:58:46.316831596 (296h58m43.91360262s) [noise=true , address=45.91.116.10:4081 , piece_id=EF76JRIGJA4FD4AMV24UKMM5XMGCZNHLGBB73BHKCKXMF4TSP7VA, seg_pos={0 0}, upload=332623]: Done uploading piece. err: <nil> Mar 18 19:58:46.322635277 (296h58m43.919406291s) [noise=true , address=45.91.116.7:4010 , piece_id=SCVINHLW6JSLU5MZL72IAYMXURYVQ2GKI7JVI2TJQETLUG2EWZLQ, seg_pos={0 0}, upload=332623]: Done uploading piece. err: <nil> Mar 18 19:58:46.322663286 (296h58m43.919434315s) [noise=true , address=79.127.219.41:4929 , piece_id=2L4IZ2TK4B434LWPF74TMUJOE6MJFPH53DUO5MR6TBOJITNYHFVA, seg_pos={0 0}, upload=332623]: Done uploading piece. err: <nil> Mar 18 19:59:12.822935404 (296h59m10.419706455s) [noise=true , address=145.79.227.20:4440 , piece_id=WZIZCLSZSK7PAXAWNT74YUVZNUQ5OSSO5ETPB7D57QZPDOLGAVDA, seg_pos={0 0}, upload=332623]: Done uploading piece. err: ecclient: upload failed (node:12GfuKp6kJEy1NTHpo9i21UDm62hrAtKq7124Z6FUehNyyLQ33i, address:145.79.227.20:4440): protocol: rpc: dial tcp 145.79.227.20:4440: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 19:59:12.823588047 (296h59m10.42035908s) [noise=true , address=145.79.227.23:4550 , piece_id=36QDBITILXOENDGIMDXMCETUC6ZUJ7HLMSYRC7ES5NMMKRLZMIXQ, seg_pos={0 0}, upload=332623]: Done uploading piece. err: ecclient: upload failed (node:12kn3uD4MCE56bN4MYYsGjDJd2qebgZ6VB6mavL711M8hoU8fbE, address:145.79.227.23:4550): protocol: rpc: dial tcp 145.79.227.23:4550: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 19:59:12.823982710 (296h59m10.420753733s) [noise=true , address=145.79.227.24:4585 , piece_id=YHW3ZWDOKCWVKO7JXL4XHYTTI2SFDYDYFJMP26RRG27P75CLIQ6Q, seg_pos={0 0}, upload=332623]: Done uploading piece. err: ecclient: upload failed (node:1TyK6onzVds1kucaD6Y4aPssPDM5FzzQ91dTrez5B7ttfwvXm, address:145.79.227.24:4585): protocol: rpc: dial tcp 145.79.227.24:4585: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 19:59:12.824226833 (296h59m10.420997864s) [noise=true , address=145.79.227.33:4874 , piece_id=73BH73YP5SJWPJI45TXHDUCSH77PKMVZWIGVDQHRKFMPACOAMB7Q, seg_pos={0 0}, upload=332623]: Done uploading piece. err: ecclient: upload failed (node:12YApHMARaC7Wxeqopt72gpnyJS1xYBwhV7wbU9NJdR6W7hukyq, address:145.79.227.33:4874): protocol: rpc: dial tcp 145.79.227.33:4874: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 19:59:12.824240012 (296h59m10.421011041s) [noise=true , address=145.79.227.4:4191 , piece_id=C7NPTV4OCU7DSK6JWCXSP7CMBL5ABGJYK5HIRNJYIZ2FNIQYLIPA, seg_pos={0 0}, upload=332623]: Done uploading piece. err: ecclient: upload failed (node:1sVWXyKPnLTobAhk6GXZTTP67FbyY3XJ1e2svTiUEa3rrghd2q, address:145.79.227.4:4191): protocol: rpc: dial tcp 145.79.227.4:4191: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 19:59:12.824535581 (296h59m10.421306594s) [noise=true , address=145.79.227.30:4762 , piece_id=FSYZBAVZTDS363COI2VXSFQHNAW7EBPIER6RRCN4SC4MUJ3QDUXA, seg_pos={0 0}, upload=332623]: Done uploading piece. err: ecclient: upload failed (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b, address:145.79.227.30:4762): protocol: rpc: dial tcp 145.79.227.30:4762: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 19:59:12.824732220 (296h59m10.421503255s) [noise=true , address=145.79.227.22:4512 , piece_id=YRJBQTCXG5X6G5V3FQCT2POZHCUMXNCVOMDNJJRBACVJAWPY5CAA, seg_pos={0 0}, upload=332623]: Done uploading piece. err: ecclient: upload failed (node:12cnVqYYfo8b6RvRtMykzh8mU7UAqxcqYvWe7H8G7k8DqtNLAR4, address:145.79.227.22:4512): protocol: rpc: dial tcp 145.79.227.22:4512: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 19:59:12.825191902 (296h59m10.421962935s) [noise=true , address=145.79.227.45:4234 , piece_id=HF335WOWCKMG45FPC4HRFSWLWT6XXZF6SQ73ZXAVJNB22ZTK33RA, seg_pos={0 0}, upload=332623]: Done uploading piece. err: ecclient: upload failed (node:12uthP31WaBmn37h54Q6z9ZMtYRY6K8917Ms4kwyyADqu4EXPJy, address:145.79.227.45:4234): protocol: rpc: dial tcp 145.79.227.45:4234: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 19:59:12.825411179 (296h59m10.422182211s) [noise=true , address=145.79.227.46:4603 , piece_id=YDKENB3HNHHRQH6XGTL24AAXI7GV2JW42CHG45QEJ3EL7XTTKFZA, seg_pos={0 0}, upload=332623]: Done uploading piece. err: ecclient: upload failed (node:1ssbeqxAWdq216V5ukNXZ3zvRroDf1x2hSs8cLtepUpefugtkX, address:145.79.227.46:4603): protocol: rpc: dial tcp 145.79.227.46:4603: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 19:59:12.825458453 (296h59m10.422229462s) [seg_pos={0 0}, upload=332623]: Done waiting for segment. successful: 7 optimal: 13 errs: <nil> Mar 18 19:59:12.845276225 (296h59m10.44204726s) [upload=332623]: Done uploading segments... Mar 18 19:59:12.845355413 (296h59m10.442126425s) [upload=332623]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 7) Duration: 30.025029665s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 5 out of 60 is failed (threshold=3) [.................................X..........XX..X....X......] duration: 17.844205027s trace: https://jaeger.dev.storj.io/trace/6b5565e31dae52f1 FAIL (2026-03-18 19:58:51 UTC) operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== Mar 18 19:53:19.380863725 (296h53m16.977634736s) [upload=332531]: Starting upload Mar 18 19:53:19.380944319 (296h53m16.977715346s) [upload=332531]: Done starting upload Mar 18 19:53:19.380976457 (296h53m16.977747482s) [upload=332531]: Uploading segments... Mar 18 19:53:19.381333196 (296h53m16.978104222s) [upload=332531]: Got next segment. Inline: false Mar 18 19:53:19.381379068 (296h53m16.978150099s) [upload=332531]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 19:53:19.381386543 (296h53m16.978157579s) [upload=332531]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 18 19:53:23.490845546 (296h53m21.087616603s) [seg_pos={0 0}, upload=332531]: Begin upload segment... Mar 18 19:53:23.490917951 (296h53m21.087688988s) [seg_pos={0 0}, upload=332531]: Done begin upload segment. Mar 18 19:53:23.490955926 (296h53m21.087726957s) [noise=true , address=45.91.116.5:4312 , piece_id=BDB6LYZZH2HXC3WWPDVMDQLCAAGRXBM5OCZU2JQVUJ7E6REVJ5SQ, seg_pos={0 0}, upload=332531]: Uploading piece... Mar 18 19:53:23.492093139 (296h53m21.088864171s) [noise=true , address=193.149.164.67:4003 , piece_id=SOPPFFG22JZWQA5NHYGA2OFHC2KMALZOR3ATBQ6JYWJVXATWLSRQ, seg_pos={0 0}, upload=332531]: Uploading piece... Mar 18 19:53:23.492835295 (296h53m21.089606329s) [noise=true , address=77.90.31.18:4833 , piece_id=ON3SKTN2H26XEXC43C44MAJIPY32NZLNWRU4PBSRJS6HLMXVNBMA, seg_pos={0 0}, upload=332531]: Uploading piece... Mar 18 19:53:23.492958664 (296h53m21.089729699s) [noise=true , address=45.91.116.13:4167 , piece_id=3JWBJU4M3WSUM3D2TVZDGJKBSRXZY5ZQHV3GFSBKOJMLORY5W7IQ, seg_pos={0 0}, upload=332531]: Uploading piece... Mar 18 19:53:23.493624498 (296h53m21.090395533s) [noise=true , address=145.79.227.13:4814 , piece_id=3KU6BHQSH4U4XYBSV4I77FCLQKR5RKWFY3QKZ26D5QRCJDQPKTKQ, seg_pos={0 0}, upload=332531]: Uploading piece... Mar 18 19:53:23.493650442 (296h53m21.090421482s) [noise=true , address=79.127.205.237:4230 , piece_id=Y6AOKOD27V5RL6QWUY2ZAJMONWPTYU4TTU5VIAYJ4RVMNQMCRUZQ, seg_pos={0 0}, upload=332531]: Uploading piece... Mar 18 19:53:23.494253839 (296h53m21.091024872s) [noise=true , address=45.91.116.14:4223 , piece_id=X3NSJBVWEZ3F3DUHQOSQSUCECIN7FARCFMRPJR6N7FHIIL3MS3NQ, seg_pos={0 0}, upload=332531]: Uploading piece... Mar 18 19:53:23.494267618 (296h53m21.091038645s) [noise=true , address=145.79.227.22:4516 , piece_id=OES5UD4FMNBSPC4ZSDS45SRXLTAQT3QNGR6JBI5D5Y363JFGJ73Q, seg_pos={0 0}, upload=332531]: Uploading piece... Mar 18 19:53:23.494879425 (296h53m21.091650453s) [noise=true , address=193.149.164.71:4287 , piece_id=ZE2Q4NHL3KUKRB36IDRMIZ7DIKDYZIBCM2L3PTLXALGL3PINBGAQ, seg_pos={0 0}, upload=332531]: Uploading piece... Mar 18 19:53:23.494911912 (296h53m21.091682944s) [noise=true , address=193.149.164.69:4074 , piece_id=FUGRHBCQHA4YJ2EVCMKVQVITADSVLM3PWZHV2XM3QLRDGU4G2GPA, seg_pos={0 0}, upload=332531]: Uploading piece... Mar 18 19:53:23.495497539 (296h53m21.09226857s) [noise=true , address=193.149.164.68:4066 , piece_id=FY5RBQUD2JCNXJCY2KIK744F4QZFHTW45PBDMASIIP27RWWB7SSA, seg_pos={0 0}, upload=332531]: Uploading piece... Mar 18 19:53:23.496168841 (296h53m21.092939872s) [noise=true , address=145.79.227.41:4105 , piece_id=RTYHA5YP42CFAYVM6PHY5RTB47GRMWSZEBBPNLC532CXKPBT2VXA, seg_pos={0 0}, upload=332531]: Uploading piece... Mar 18 19:53:23.497008065 (296h53m21.093779105s) [noise=true , address=79.127.205.235:4271 , piece_id=5XULCXAVIG3CQFTYA6Q3TFPRVICQFHQI2MSI2VIRIAGSK5UKKFXQ, seg_pos={0 0}, upload=332531]: Uploading piece... Mar 18 19:53:23.497605357 (296h53m21.094376396s) [noise=true , address=145.79.227.37:4590 , piece_id=ZTQLQ2IZHJFZWBHMRT74V7JDZYXF5BG4656VBZDJ4J6NGKS2TMOA, seg_pos={0 0}, upload=332531]: Uploading piece... Mar 18 19:53:23.498200357 (296h53m21.094971388s) [noise=true , address=193.149.164.66:4989 , piece_id=DVAJPRB7ABJJGUP4TYMLOGJVDLUARAMSPCG6FE2PVQXIDP2QBPGA, seg_pos={0 0}, upload=332531]: Uploading piece... Mar 18 19:53:23.498757906 (296h53m21.095528939s) [noise=true , address=79.127.219.45:4868 , piece_id=6NDAYV5PFDBOMZKAXKO6LZUODM46MQPQOQIDKBHER3LG4CBEDHKQ, seg_pos={0 0}, upload=332531]: Uploading piece... Mar 18 19:53:23.499215136 (296h53m21.095986159s) [noise=true , address=145.79.227.52:4453 , piece_id=YRAA65GHFFR45DX7MKR5BWBXID3YVSA4DMQDRKWKRDTJCHKHGMOA, seg_pos={0 0}, upload=332531]: Uploading piece... Mar 18 19:53:23.499687663 (296h53m21.096458697s) [noise=true , address=77.90.31.14:4678 , piece_id=NWBLAAUF3GACR3Y6JRTHDEUJ24GMOTZIPSRGJIPTPLIVWBCSYAHA, seg_pos={0 0}, upload=332531]: Uploading piece... Mar 18 19:53:23.600142173 (296h53m21.196913186s) [upload=332531]: Next returned nil segment Mar 18 19:53:23.600161212 (296h53m21.196932236s) [upload=332531]: Waiting for error group managing segments... Mar 18 19:53:23.620885875 (296h53m21.217656899s) [noise=true , address=45.91.116.5:4312 , piece_id=BDB6LYZZH2HXC3WWPDVMDQLCAAGRXBM5OCZU2JQVUJ7E6REVJ5SQ, seg_pos={0 0}, upload=332531]: Done uploading piece. err: <nil> Mar 18 19:53:23.621154396 (296h53m21.217925412s) [noise=true , address=45.91.116.13:4167 , piece_id=3JWBJU4M3WSUM3D2TVZDGJKBSRXZY5ZQHV3GFSBKOJMLORY5W7IQ, seg_pos={0 0}, upload=332531]: Done uploading piece. err: <nil> Mar 18 19:53:23.628883995 (296h53m21.225655021s) [noise=true , address=45.91.116.14:4223 , piece_id=X3NSJBVWEZ3F3DUHQOSQSUCECIN7FARCFMRPJR6N7FHIIL3MS3NQ, seg_pos={0 0}, upload=332531]: Done uploading piece. err: <nil> Mar 18 19:53:23.630970813 (296h53m21.227741838s) [noise=true , address=79.127.219.45:4868 , piece_id=6NDAYV5PFDBOMZKAXKO6LZUODM46MQPQOQIDKBHER3LG4CBEDHKQ, seg_pos={0 0}, upload=332531]: Done uploading piece. err: <nil> Mar 18 19:53:23.639825090 (296h53m21.236596104s) [noise=true , address=77.90.31.14:4678 , piece_id=NWBLAAUF3GACR3Y6JRTHDEUJ24GMOTZIPSRGJIPTPLIVWBCSYAHA, seg_pos={0 0}, upload=332531]: Done uploading piece. err: <nil> Mar 18 19:53:23.642754677 (296h53m21.239525692s) [noise=true , address=77.90.31.18:4833 , piece_id=ON3SKTN2H26XEXC43C44MAJIPY32NZLNWRU4PBSRJS6HLMXVNBMA, seg_pos={0 0}, upload=332531]: Done uploading piece. err: <nil> Mar 18 19:53:24.057422272 (296h53m21.654193306s) [noise=true , address=79.127.205.235:4271 , piece_id=5XULCXAVIG3CQFTYA6Q3TFPRVICQFHQI2MSI2VIRIAGSK5UKKFXQ, seg_pos={0 0}, upload=332531]: Done uploading piece. err: <nil> Mar 18 19:53:24.074244577 (296h53m21.671015604s) [noise=true , address=193.149.164.67:4003 , piece_id=SOPPFFG22JZWQA5NHYGA2OFHC2KMALZOR3ATBQ6JYWJVXATWLSRQ, seg_pos={0 0}, upload=332531]: Done uploading piece. err: <nil> Mar 18 19:53:24.097678734 (296h53m21.694449747s) [noise=true , address=79.127.205.237:4230 , piece_id=Y6AOKOD27V5RL6QWUY2ZAJMONWPTYU4TTU5VIAYJ4RVMNQMCRUZQ, seg_pos={0 0}, upload=332531]: Done uploading piece. err: <nil> Mar 18 19:53:24.109685982 (296h53m21.706457009s) [noise=true , address=193.149.164.68:4066 , piece_id=FY5RBQUD2JCNXJCY2KIK744F4QZFHTW45PBDMASIIP27RWWB7SSA, seg_pos={0 0}, upload=332531]: Done uploading piece. err: <nil> Mar 18 19:53:24.110659716 (296h53m21.707430744s) [noise=true , address=193.149.164.69:4074 , piece_id=FUGRHBCQHA4YJ2EVCMKVQVITADSVLM3PWZHV2XM3QLRDGU4G2GPA, seg_pos={0 0}, upload=332531]: Done uploading piece. err: <nil> Mar 18 19:53:24.114541274 (296h53m21.7113123s) [noise=true , address=193.149.164.71:4287 , piece_id=ZE2Q4NHL3KUKRB36IDRMIZ7DIKDYZIBCM2L3PTLXALGL3PINBGAQ, seg_pos={0 0}, upload=332531]: Done uploading piece. err: <nil> Mar 18 19:53:24.140894470 (296h53m21.737665488s) [noise=true , address=193.149.164.66:4989 , piece_id=DVAJPRB7ABJJGUP4TYMLOGJVDLUARAMSPCG6FE2PVQXIDP2QBPGA, seg_pos={0 0}, upload=332531]: Done uploading piece. err: <nil> Mar 18 19:54:19.329327996 (296h54m16.926099018s) [noise=true , address=145.79.227.13:4814 , piece_id=3KU6BHQSH4U4XYBSV4I77FCLQKR5RKWFY3QKZ26D5QRCJDQPKTKQ, seg_pos={0 0}, upload=332531]: Done uploading piece. err: ecclient: upload failed (node:12ERtrzMuh8rZPsgebtyScmYvNVwz3UKJr5Nu2hLdu6sn8X3qLa, address:145.79.227.13:4814): protocol: rpc: dial tcp 145.79.227.13:4814: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 19:54:19.329567681 (296h54m16.926338706s) [noise=true , address=145.79.227.52:4453 , piece_id=YRAA65GHFFR45DX7MKR5BWBXID3YVSA4DMQDRKWKRDTJCHKHGMOA, seg_pos={0 0}, upload=332531]: Done uploading piece. err: ecclient: upload failed (node:1BmpsQ9VfhGzmkZHvSeMptcGhzGpdoN4oYvi83qZcC1Vh9anhP, address:145.79.227.52:4453): protocol: rpc: dial tcp 145.79.227.52:4453: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 19:54:19.329755822 (296h54m16.926526848s) [noise=true , address=145.79.227.41:4105 , piece_id=RTYHA5YP42CFAYVM6PHY5RTB47GRMWSZEBBPNLC532CXKPBT2VXA, seg_pos={0 0}, upload=332531]: Done uploading piece. err: ecclient: upload failed (node:1TVtMQNVV2kERp2aZwmS8E9JUiua1RCyFpZgFACRqH1Nfscw4C, address:145.79.227.41:4105): protocol: rpc: dial tcp 145.79.227.41:4105: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 19:54:19.330021146 (296h54m16.926792175s) [noise=true , address=145.79.227.22:4516 , piece_id=OES5UD4FMNBSPC4ZSDS45SRXLTAQT3QNGR6JBI5D5Y363JFGJ73Q, seg_pos={0 0}, upload=332531]: Done uploading piece. err: ecclient: upload failed (node:12Lhy3AxzYoF25AxWdvjrrV5NNCwP1cDqkEvv4tz9t9KZmF8WPK, address:145.79.227.22:4516): protocol: rpc: dial tcp 145.79.227.22:4516: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 19:54:19.330213680 (296h54m16.926984707s) [noise=true , address=145.79.227.37:4590 , piece_id=ZTQLQ2IZHJFZWBHMRT74V7JDZYXF5BG4656VBZDJ4J6NGKS2TMOA, seg_pos={0 0}, upload=332531]: Done uploading piece. err: ecclient: upload failed (node:1KnNqY1WNwMSFXFrQsWPAhivUTgSn7dKTx1mTTzYG2UxGCkasj, address:145.79.227.37:4590): protocol: rpc: dial tcp 145.79.227.37:4590: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 19:54:19.330250832 (296h54m16.927021848s) [seg_pos={0 0}, upload=332531]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 18 19:54:19.351979131 (296h54m16.948750155s) [upload=332531]: Done uploading segments... Mar 18 19:54:19.352025297 (296h54m16.948796327s) [upload=332531]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 17.844238703s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 30 out of 60 is failed (threshold=3) [..............................XXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 30.025705026s trace: https://jaeger.dev.storj.io/trace/78b4268f332bc432 FAIL (2026-03-18 19:58:42 UTC) operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 5) ======LOGS====== (296h58m13.001339841s) [seg_pos={0 0}, upload=332615]: Done begin upload segment. Mar 18 19:58:15.404596858 (296h58m13.001367882s) [noise=true , address=145.79.227.52:4449 , piece_id=P3IHQHOZD753TYIHMRGTOPDYQVMUUWQYCNUAUT6IDXVNA7YGNVNQ, seg_pos={0 0}, upload=332615]: Uploading piece... Mar 18 19:58:15.404943486 (296h58m13.001714519s) [noise=true , address=145.79.227.43:4158 , piece_id=DJ76CT74L5R2L3ZVKL2VU7DGB53MQDWFHIEQJ6SBXKBFGCRQCATQ, seg_pos={0 0}, upload=332615]: Uploading piece... Mar 18 19:58:15.405238214 (296h58m13.002009266s) [noise=true , address=45.91.116.10:4085 , piece_id=2BF7BRBHJDZX6UO3FQYBZ542HRSOM24SU4UJEXDCNNLTRYGTKDEQ, seg_pos={0 0}, upload=332615]: Uploading piece... Mar 18 19:58:15.405673582 (296h58m13.002444608s) [noise=true , address=145.79.227.50:4374 , piece_id=64DSSJA6JEZLBQ5THCECZ5L47R72N3UMNY5ALOLIJWQ7PRP675YA, seg_pos={0 0}, upload=332615]: Uploading piece... Mar 18 19:58:15.406078897 (296h58m13.002849927s) [noise=true , address=145.79.227.16:4915 , piece_id=U2HZ26TBMQYPS667U7N2AZ3BNB4B6G3D3HCERWRXUGDCCYR4MHVA, seg_pos={0 0}, upload=332615]: Uploading piece... Mar 18 19:58:15.406277732 (296h58m13.003048764s) [noise=true , address=45.91.116.9:4261 , piece_id=7O6O7VTCFQCASFUYOGCP5LIQYRYDBW6PL3GDDHXDM75TURPNWPRA, seg_pos={0 0}, upload=332615]: Uploading piece... Mar 18 19:58:15.406475622 (296h58m13.003246652s) [noise=true , address=45.91.116.12:4225 , piece_id=SF5QISA3Q4SLRV7GXVVVRAUWKBDA6T6RFZFAGVJEWZMB2UTJZTCQ, seg_pos={0 0}, upload=332615]: Uploading piece... Mar 18 19:58:15.406783206 (296h58m13.003554232s) [noise=true , address=145.79.227.34:4905 , piece_id=LRRLKOSNOK2GYT2YVSHXHEMGX62HWCZKIHL2OWHLW2MRIORMQFLA, seg_pos={0 0}, upload=332615]: Uploading piece... Mar 18 19:58:15.406909092 (296h58m13.003680123s) [noise=true , address=145.79.227.25:4619 , piece_id=NGHMQ3ETO4G53SMS7QUXVFP3MCC3TBDWKVTAVFGLZHULBVNLWBEA, seg_pos={0 0}, upload=332615]: Uploading piece... Mar 18 19:58:15.407261771 (296h58m13.004032794s) [noise=true , address=145.79.227.14:4842 , piece_id=LQOTZJTDB2SXER7Q3GNJCJPQJSJPH4OKU27V3LQ3COJ3HHEFGORQ, seg_pos={0 0}, upload=332615]: Uploading piece... Mar 18 19:58:15.407460750 (296h58m13.004231776s) [noise=true , address=145.79.227.2:4118 , piece_id=OROYDJWDJMPCNMBNQNISU5EHWTGAWL34MXBJ4WY3PRR3HPSPPANA, seg_pos={0 0}, upload=332615]: Uploading piece... Mar 18 19:58:15.407869158 (296h58m13.004640187s) [noise=true , address=145.79.227.11:4734 , piece_id=MYB4ZZNFGG56662IH3BBK453DGA4H7XOA543AN5UDRFRVZL2O4HQ, seg_pos={0 0}, upload=332615]: Uploading piece... Mar 18 19:58:15.407900752 (296h58m13.004671793s) [noise=true , address=145.79.227.18:4008 , piece_id=ZCAXZW54UYYFB5DWGLTRO2Y2KQRNX4RA2HEU323SRI35KQAF3Y5A, seg_pos={0 0}, upload=332615]: Uploading piece... Mar 18 19:58:15.408205363 (296h58m13.004976389s) [noise=true , address=77.90.31.21:4880 , piece_id=H4PBO2R3HNFXDVQXDRXZ3GBV46DUCXZJNN6JK5WQVG5NT6LTFIIQ, seg_pos={0 0}, upload=332615]: Uploading piece... Mar 18 19:58:15.408499977 (296h58m13.005271009s) [noise=true , address=145.79.227.20:4441 , piece_id=IARKU7DCVMVRB7VPFZLLLCWNKCZNRGSZZWLIC45KZLHO7ATYD75Q, seg_pos={0 0}, upload=332615]: Uploading piece... Mar 18 19:58:15.409185391 (296h58m13.005956425s) [noise=true , address=45.91.116.11:4152 , piece_id=4JJ4I3PJUWTDGS7ZYRJT3YMWWP7JB4KKZTGLDC7GRYIY32X6HWRA, seg_pos={0 0}, upload=332615]: Uploading piece... Mar 18 19:58:15.473218271 (296h58m13.069989304s) [upload=332615]: Next returned nil segment Mar 18 19:58:15.473254275 (296h58m13.070025298s) [upload=332615]: Waiting for error group managing segments... Mar 18 19:58:15.477069711 (296h58m13.073840724s) [noise=true , address=45.91.116.12:4225 , piece_id=SF5QISA3Q4SLRV7GXVVVRAUWKBDA6T6RFZFAGVJEWZMB2UTJZTCQ, seg_pos={0 0}, upload=332615]: Done uploading piece. err: <nil> Mar 18 19:58:15.477584542 (296h58m13.07435556s) [noise=true , address=45.91.116.9:4261 , piece_id=7O6O7VTCFQCASFUYOGCP5LIQYRYDBW6PL3GDDHXDM75TURPNWPRA, seg_pos={0 0}, upload=332615]: Done uploading piece. err: <nil> Mar 18 19:58:15.477654632 (296h58m13.074425657s) [noise=true , address=45.91.116.11:4152 , piece_id=4JJ4I3PJUWTDGS7ZYRJT3YMWWP7JB4KKZTGLDC7GRYIY32X6HWRA, seg_pos={0 0}, upload=332615]: Done uploading piece. err: <nil> Mar 18 19:58:15.477995764 (296h58m13.074766787s) [noise=true , address=45.91.116.10:4085 , piece_id=2BF7BRBHJDZX6UO3FQYBZ542HRSOM24SU4UJEXDCNNLTRYGTKDEQ, seg_pos={0 0}, upload=332615]: Done uploading piece. err: <nil> Mar 18 19:58:15.478168327 (296h58m13.074939341s) [noise=true , address=77.90.31.21:4880 , piece_id=H4PBO2R3HNFXDVQXDRXZ3GBV46DUCXZJNN6JK5WQVG5NT6LTFIIQ, seg_pos={0 0}, upload=332615]: Done uploading piece. err: <nil> Mar 18 19:58:42.720035565 (296h58m40.3168066s) [noise=true , address=145.79.227.14:4842 , piece_id=LQOTZJTDB2SXER7Q3GNJCJPQJSJPH4OKU27V3LQ3COJ3HHEFGORQ, seg_pos={0 0}, upload=332615]: Done uploading piece. err: ecclient: upload failed (node:12q4VdT5BcjUb16PBYHK6zrnW8dqENKrcGx9zdLtNefDVB1MUSn, address:145.79.227.14:4842): protocol: rpc: dial tcp 145.79.227.14:4842: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 19:58:42.720306450 (296h58m40.317077478s) [noise=true , address=145.79.227.43:4158 , piece_id=DJ76CT74L5R2L3ZVKL2VU7DGB53MQDWFHIEQJ6SBXKBFGCRQCATQ, seg_pos={0 0}, upload=332615]: Done uploading piece. err: ecclient: upload failed (node:1vJZrSvPvwebw9V2tTbvkPEmLeLjryBUnq9TL6NwdPsHU9xcq2, address:145.79.227.43:4158): protocol: rpc: dial tcp 145.79.227.43:4158: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 19:58:42.720849104 (296h58m40.317620139s) [noise=true , address=145.79.227.2:4118 , piece_id=OROYDJWDJMPCNMBNQNISU5EHWTGAWL34MXBJ4WY3PRR3HPSPPANA, seg_pos={0 0}, upload=332615]: Done uploading piece. err: ecclient: upload failed (node:12ty5LHXwLmHC7nRPmVfj2wVEif6JQAQQ5Z5218vv1fKK5sD3zE, address:145.79.227.2:4118): protocol: rpc: dial tcp 145.79.227.2:4118: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 19:58:42.720985814 (296h58m40.317756845s) [noise=true , address=145.79.227.11:4734 , piece_id=MYB4ZZNFGG56662IH3BBK453DGA4H7XOA543AN5UDRFRVZL2O4HQ, seg_pos={0 0}, upload=332615]: Done uploading piece. err: ecclient: upload failed (node:12GLb5LQje9wwivnj8Wj1tXcnS6mFb2Bco5saNwgmFJBVDujrv7, address:145.79.227.11:4734): protocol: rpc: dial tcp 145.79.227.11:4734: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 19:58:42.721247244 (296h58m40.318018276s) [noise=true , address=145.79.227.16:4915 , piece_id=U2HZ26TBMQYPS667U7N2AZ3BNB4B6G3D3HCERWRXUGDCCYR4MHVA, seg_pos={0 0}, upload=332615]: Done uploading piece. err: ecclient: upload failed (node:12fqZipCJaNd9u5TvdWZ9nJFs26MiGtYX3Bbh41eQWP1Z4eJRBt, address:145.79.227.16:4915): protocol: rpc: dial tcp 145.79.227.16:4915: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 19:58:42.721274509 (296h58m40.318045549s) [noise=true , address=145.79.227.18:4008 , piece_id=ZCAXZW54UYYFB5DWGLTRO2Y2KQRNX4RA2HEU323SRI35KQAF3Y5A, seg_pos={0 0}, upload=332615]: Done uploading piece. err: ecclient: upload failed (node:148CqQmrsJSfoGByHYRPoZmvP1VgQQpaMVLb7RgRQvNBuo7air, address:145.79.227.18:4008): protocol: rpc: dial tcp 145.79.227.18:4008: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 19:58:42.721571050 (296h58m40.318342076s) [noise=true , address=145.79.227.34:4905 , piece_id=LRRLKOSNOK2GYT2YVSHXHEMGX62HWCZKIHL2OWHLW2MRIORMQFLA, seg_pos={0 0}, upload=332615]: Done uploading piece. err: ecclient: upload failed (node:12ffKESNSVpXQzbwJinP6vvgE6zoF3RFz2NRUuuJHvd7bvk1gc9, address:145.79.227.34:4905): protocol: rpc: dial tcp 145.79.227.34:4905: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 19:58:42.721588311 (296h58m40.318359346s) [noise=true , address=145.79.227.52:4449 , piece_id=P3IHQHOZD753TYIHMRGTOPDYQVMUUWQYCNUAUT6IDXVNA7YGNVNQ, seg_pos={0 0}, upload=332615]: Done uploading piece. err: ecclient: upload failed (node:12f14pHDEEamEnHmeZCe38rjaE6EYWYkGMPrZMsbSDmApN5rckq, address:145.79.227.52:4449): protocol: rpc: dial tcp 145.79.227.52:4449: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 19:58:42.721919791 (296h58m40.318690824s) [noise=true , address=145.79.227.50:4374 , piece_id=64DSSJA6JEZLBQ5THCECZ5L47R72N3UMNY5ALOLIJWQ7PRP675YA, seg_pos={0 0}, upload=332615]: Done uploading piece. err: ecclient: upload failed (node:12QRixhWaUwXPdjcWKYHbiknME5JVomWYABY6egDD6VKwaTPB8K, address:145.79.227.50:4374): protocol: rpc: dial tcp 145.79.227.50:4374: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 19:58:42.721963930 (296h58m40.318734962s) [noise=true , address=145.79.227.25:4619 , piece_id=NGHMQ3ETO4G53SMS7QUXVFP3MCC3TBDWKVTAVFGLZHULBVNLWBEA, seg_pos={0 0}, upload=332615]: Done uploading piece. err: ecclient: upload failed (node:1zuLJznAUFHkanuomswSd7JN6CkqPXso8Ak8e5E57eaEqvjodU, address:145.79.227.25:4619): protocol: rpc: dial tcp 145.79.227.25:4619: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 19:58:42.722312844 (296h58m40.319083921s) [noise=true , address=145.79.227.20:4441 , piece_id=IARKU7DCVMVRB7VPFZLLLCWNKCZNRGSZZWLIC45KZLHO7ATYD75Q, seg_pos={0 0}, upload=332615]: Done uploading piece. err: ecclient: upload failed (node:12LjGecRNJE1kq67iMtxZFhWeaUZpUogsGxbz4VXwnHdQtTnbBr, address:145.79.227.20:4441): protocol: rpc: dial tcp 145.79.227.20:4441: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 19:58:42.722352341 (296h58m40.319123356s) [seg_pos={0 0}, upload=332615]: Done waiting for segment. successful: 5 optimal: 13 errs: <nil> Mar 18 19:58:42.743755481 (296h58m40.340526519s) [upload=332615]: Done uploading segments... Mar 18 19:58:42.743840731 (296h58m40.340611769s) [upload=332615]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 5) Duration: 30.025720244s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 39 out of 60 is failed (threshold=6) [.................XXXXXXXXX.XXX...XXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 5.002439039s trace: https://jaeger.dev.storj.io/trace/4557383cefe24a34 FAIL (2026-03-18 19:58:42 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.002468784s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 8 out of 60 is failed (threshold=3) [..............................XXXXXX.XX.....................] duration: 15.435152996s trace: https://jaeger.dev.storj.io/trace/4b4d10a8051ad01b FAIL (2026-03-18 19:58:30 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 15.435176243s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 38 out of 60 is failed (threshold=6) [..................XXXXXXXXX.XXX...XXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 5.001255244s trace: https://jaeger.dev.storj.io/trace/65d4a37f17354bcc FAIL (2026-03-18 19:58:13 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001287691s core uplink us-select-1 us1-placement-20

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)