Orbiter History

checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 8 out of 60 is failed (threshold=3) [....XXXXXX.XX...............................................] duration: 734.122505ms trace: https://jaeger.dev.storj.io/trace/4b4d10a8051ad01b FAIL (2026-03-18 20:18:01 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 734.242098ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 6 out of 60 is failed (threshold=3) [.........X..........XX..X....X........X.....................] duration: 1.288624221s trace: https://jaeger.dev.storj.io/trace/589419efc52e5e2b FAIL (2026-03-18 20:17:52 UTC) operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== Mar 18 20:00:59.343084966 (297h0m56.939855989s) [upload=332666]: Starting upload Mar 18 20:00:59.343175518 (297h0m56.939946548s) [upload=332666]: Done starting upload Mar 18 20:00:59.343565585 (297h0m56.940336612s) [upload=332666]: Uploading segments... Mar 18 20:00:59.344256647 (297h0m56.941027676s) [upload=332666]: Got next segment. Inline: false Mar 18 20:00:59.344352215 (297h0m56.941123241s) [upload=332666]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 20:00:59.344358706 (297h0m56.941129732s) [upload=332666]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 18 20:01:02.340259299 (297h0m59.937030329s) [seg_pos={0 0}, upload=332666]: Begin upload segment... Mar 18 20:01:02.340368743 (297h0m59.937139767s) [seg_pos={0 0}, upload=332666]: Done begin upload segment. Mar 18 20:01:02.340479563 (297h0m59.93725059s) [noise=true , address=45.91.116.8:4061 , piece_id=6XFVKT4AAO7QN457OQMRKPI6R24TJTAN47UXLEVOEFQVOS6M6C3Q, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.340523837 (297h0m59.93729486s) [noise=true , address=79.127.205.233:4166 , piece_id=SFGK5YCWHFRVXF6QNHYP2CYJCZLSTQ76OYRVJPWWXQTF2KFSFJKA, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.341468438 (297h0m59.938239447s) [noise=true , address=45.91.116.4:4961 , piece_id=ZMUGQECUVPVHDNKN75XPBET5M2HFXXK4WMTP4V5KEK63JVZCX5EA, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.341520025 (297h0m59.938291058s) [noise=true , address=167.17.72.71:4815 , piece_id=CXGTYNNBILN3TT7I6Z7UI75F2ZCS4QUTLS6KC4FJ5IHS3XOYWDJQ, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.342232992 (297h0m59.939004023s) [noise=true , address=145.79.227.36:4524 , piece_id=3TAV3P55ITTHBAZEUUUTOAHJRGQXJXXVGZJP6AVR2ANVNOR3S54Q, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.342917368 (297h0m59.939688398s) [noise=true , address=193.149.164.71:4297 , piece_id=4IJPXLPCX3LKFOBOSRHK2DTNRKH67K7FXJUKQ3JPRWHS47ZFEIDA, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.343244957 (297h0m59.940015982s) [noise=true , address=145.79.227.33:4892 , piece_id=DQ6ALC3VOAGFIJI3IKR6I7QFYM22C66QWAAXMH6NXI2FIQEZNQYA, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.344270079 (297h0m59.941041117s) [noise=true , address=145.79.227.21:4497 , piece_id=SP6EKJB7W4QXF7OIBVKEFWCPOZFL5IRIEDVAGS2M4QJSO34CUVRQ, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.345025057 (297h0m59.941796084s) [noise=true , address=77.90.31.17:4793 , piece_id=2U4IBUA2GGUGDBXQGIGQKUK4ZR3L7CDKC6MTGIKAUME4F7MFZRLA, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.345115802 (297h0m59.941886836s) [noise=true , address=167.17.72.69:4731 , piece_id=VOEVIKKFJMC2Y3TPQIDMYUP37NLFU7KPRH4HRRSMQLWUKTSLOH2Q, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.345690600 (297h0m59.942461634s) [noise=true , address=77.90.31.13:4648 , piece_id=BPMB5CFOYHFT3BS4ORR6JF6BNFC2AOAJZOSPMG67NAMLGGP5XVEQ, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.346311592 (297h0m59.943082623s) [noise=true , address=79.127.205.234:4202 , piece_id=YY4LYWX26O6RIOBONESNKTUZEFZCRL4UUX3NEBQ53AJV4IEDG5MQ, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.346791142 (297h0m59.943562166s) [noise=true , address=45.91.116.10:4110 , piece_id=UYLMDCWJEJOHGICA37KNH7G5AQXNIPRH5NRYLBUUMGDRVSLTEYHQ, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.347094999 (297h0m59.94386607s) [noise=true , address=145.79.227.17:4987 , piece_id=PT3TWVB65JZFEWVH4IUZEU2N53FQ57QBRWVLWAUFP4UDLY3LWNLQ, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.347781281 (297h0m59.944552305s) [noise=true , address=79.127.219.46:4919 , piece_id=KK6HRIVBMYLIHRHGPQH3AWMNSVHTTIBESHDMJZA72T446RX6QKRQ, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.348263123 (297h0m59.945034137s) [noise=true , address=193.149.164.68:4033 , piece_id=JZPQKPYGO3W3RZZBITQ73X2XEPIBBSLGRMWIVOM4Z3GTV5UFKRTQ, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.348915505 (297h0m59.945686536s) [noise=true , address=145.79.227.7:4312 , piece_id=2FAL6E3RMBTBHZBXAJJVH2N5VGF5VYI3KNZXGNWCMWGBBPAUO2NQ, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.349366298 (297h0m59.946137337s) [noise=true , address=193.149.164.67:4007 , piece_id=HURQGH67I5B6V4YUV3PRZL3KSKN3XTSK3UMCHZRNT6L3W3PXXHPQ, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.429834098 (297h1m0.026605126s) [upload=332666]: Next returned nil segment Mar 18 20:01:02.429853411 (297h1m0.026624442s) [upload=332666]: Waiting for error group managing segments... Mar 18 20:01:02.446867680 (297h1m0.043638704s) [noise=true , address=45.91.116.10:4110 , piece_id=UYLMDCWJEJOHGICA37KNH7G5AQXNIPRH5NRYLBUUMGDRVSLTEYHQ, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil> Mar 18 20:01:02.451665165 (297h1m0.048436196s) [noise=true , address=45.91.116.4:4961 , piece_id=ZMUGQECUVPVHDNKN75XPBET5M2HFXXK4WMTP4V5KEK63JVZCX5EA, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil> Mar 18 20:01:02.452538886 (297h1m0.049309913s) [noise=true , address=77.90.31.17:4793 , piece_id=2U4IBUA2GGUGDBXQGIGQKUK4ZR3L7CDKC6MTGIKAUME4F7MFZRLA, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil> Mar 18 20:01:02.455597040 (297h1m0.052368067s) [noise=true , address=45.91.116.8:4061 , piece_id=6XFVKT4AAO7QN457OQMRKPI6R24TJTAN47UXLEVOEFQVOS6M6C3Q, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil> Mar 18 20:01:02.457848739 (297h1m0.054619766s) [noise=true , address=77.90.31.13:4648 , piece_id=BPMB5CFOYHFT3BS4ORR6JF6BNFC2AOAJZOSPMG67NAMLGGP5XVEQ, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil> Mar 18 20:01:02.461479709 (297h1m0.058250746s) [noise=true , address=79.127.219.46:4919 , piece_id=KK6HRIVBMYLIHRHGPQH3AWMNSVHTTIBESHDMJZA72T446RX6QKRQ, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil> Mar 18 20:01:02.676859466 (297h1m0.27363049s) [noise=true , address=167.17.72.69:4731 , piece_id=VOEVIKKFJMC2Y3TPQIDMYUP37NLFU7KPRH4HRRSMQLWUKTSLOH2Q, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil> Mar 18 20:01:02.909860743 (297h1m0.506631766s) [noise=true , address=79.127.205.234:4202 , piece_id=YY4LYWX26O6RIOBONESNKTUZEFZCRL4UUX3NEBQ53AJV4IEDG5MQ, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil> Mar 18 20:01:02.930427742 (297h1m0.527198754s) [noise=true , address=193.149.164.71:4297 , piece_id=4IJPXLPCX3LKFOBOSRHK2DTNRKH67K7FXJUKQ3JPRWHS47ZFEIDA, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil> Mar 18 20:01:02.944696889 (297h1m0.541467914s) [noise=true , address=193.149.164.67:4007 , piece_id=HURQGH67I5B6V4YUV3PRZL3KSKN3XTSK3UMCHZRNT6L3W3PXXHPQ, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil> Mar 18 20:01:02.951319183 (297h1m0.548090212s) [noise=true , address=193.149.164.68:4033 , piece_id=JZPQKPYGO3W3RZZBITQ73X2XEPIBBSLGRMWIVOM4Z3GTV5UFKRTQ, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil> Mar 18 20:01:02.975619436 (297h1m0.572390467s) [noise=true , address=79.127.205.233:4166 , piece_id=SFGK5YCWHFRVXF6QNHYP2CYJCZLSTQ76OYRVJPWWXQTF2KFSFJKA, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil> Mar 18 20:01:03.819918528 (297h1m1.416689564s) [noise=true , address=167.17.72.71:4815 , piece_id=CXGTYNNBILN3TT7I6Z7UI75F2ZCS4QUTLS6KC4FJ5IHS3XOYWDJQ, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil> Mar 18 20:01:59.307873420 (297h1m56.904644452s) [noise=true , address=145.79.227.7:4312 , piece_id=2FAL6E3RMBTBHZBXAJJVH2N5VGF5VYI3KNZXGNWCMWGBBPAUO2NQ, seg_pos={0 0}, upload=332666]: Done uploading piece. err: ecclient: upload failed (node:12AVp3qn5uwLPqYaZEwzcgoGMJhNga1BUh6bK5k2bxjWgZLQiiC, address:145.79.227.7:4312): protocol: rpc: dial tcp 145.79.227.7:4312: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:01:59.308026092 (297h1m56.904797124s) [noise=true , address=145.79.227.17:4987 , piece_id=PT3TWVB65JZFEWVH4IUZEU2N53FQ57QBRWVLWAUFP4UDLY3LWNLQ, seg_pos={0 0}, upload=332666]: Done uploading piece. err: ecclient: upload failed (node:1XsPVNGYUhKnXkQRfVfjyBRzyvhShSdsrnzXtwe4Xtf5MaHMvH, address:145.79.227.17:4987): protocol: rpc: dial tcp 145.79.227.17:4987: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:01:59.308195864 (297h1m56.904966894s) [noise=true , address=145.79.227.36:4524 , piece_id=3TAV3P55ITTHBAZEUUUTOAHJRGQXJXXVGZJP6AVR2ANVNOR3S54Q, seg_pos={0 0}, upload=332666]: Done uploading piece. err: ecclient: upload failed (node:1xfbCzjSwFDRrR1gPggeN8tNCLbRRWDVzCJ6mqhZsbVSe8d4s4, address:145.79.227.36:4524): protocol: rpc: dial tcp 145.79.227.36:4524: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:01:59.308316037 (297h1m56.90508706s) [noise=true , address=145.79.227.33:4892 , piece_id=DQ6ALC3VOAGFIJI3IKR6I7QFYM22C66QWAAXMH6NXI2FIQEZNQYA, seg_pos={0 0}, upload=332666]: Done uploading piece. err: ecclient: upload failed (node:12RHwFxcUjPsn12uHEAH3fBDJKv1v1tM8mRJXoFrZrG5djdswF5, address:145.79.227.33:4892): protocol: rpc: dial tcp 145.79.227.33:4892: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:01:59.308404403 (297h1m56.905175433s) [noise=true , address=145.79.227.21:4497 , piece_id=SP6EKJB7W4QXF7OIBVKEFWCPOZFL5IRIEDVAGS2M4QJSO34CUVRQ, seg_pos={0 0}, upload=332666]: Done uploading piece. err: ecclient: upload failed (node:12ZBe4we7VeFdthqRxyy1V9uQrA4r8ouNX68xqsk5bD26dkf2Gd, address:145.79.227.21:4497): protocol: context deadline exceeded; context deadline exceeded Mar 18 20:01:59.308437457 (297h1m56.90520847s) [seg_pos={0 0}, upload=332666]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 18 20:01:59.338288467 (297h1m56.93505949s) [upload=332666]: Done uploading segments... Mar 18 20:01:59.338335497 (297h1m56.935106523s) [upload=332666]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 1.288659398s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 9 out of 60 is failed (threshold=3) [........XXX..........X...X.........X....X....X.......X......] duration: 1.264642295s trace: https://jaeger.dev.storj.io/trace/24517a7cd40ab76d FAIL (2026-03-18 20:17:33 UTC) operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== oad Mar 18 20:12:28.252047067 (297h12m25.848818092s) [upload=332883]: Uploading segments... Mar 18 20:12:28.252323178 (297h12m25.84909421s) [upload=332883]: Got next segment. Inline: false Mar 18 20:12:28.252380536 (297h12m25.849151569s) [upload=332883]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 20:12:28.252388587 (297h12m25.849159622s) [upload=332883]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 18 20:12:31.413020979 (297h12m29.009792004s) [seg_pos={0 0}, upload=332883]: Begin upload segment... Mar 18 20:12:31.413109194 (297h12m29.009880217s) [seg_pos={0 0}, upload=332883]: Done begin upload segment. Mar 18 20:12:31.413182895 (297h12m29.009953924s) [noise=true , address=167.17.72.68:4653 , piece_id=6TO53KV4DBJ5HTWIQUZY2L6WVMB4PBPRWSZD6WHBLVFI5FUULYUQ, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.413918965 (297h12m29.010689999s) [noise=true , address=145.79.227.20:4454 , piece_id=F2WUAPOH2J3GQ7DMWLBJA44UGY5B3MMDS2U5ITXTOIPR4ELY5QYA, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.414450842 (297h12m29.011221869s) [noise=true , address=193.149.164.70:4111 , piece_id=H6GX5VVLRFNUAF2R7FRZ5GFQCZ2F57OL4GXIHWQJGZDOM3ST5DKQ, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.414901508 (297h12m29.011672532s) [noise=true , address=5.231.73.4:4002 , piece_id=3DWXH2Y6ZZNDHRFMJFL6EEMJN7WNDRKJ3ARFKN6UAGAHICNCBSDA, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.415342842 (297h12m29.012113865s) [noise=true , address=145.79.227.28:4750 , piece_id=KJVCFR2SJ5HAQKJUJP4CYPVCZQC6PBAQII6UGNYVKRODWRKENFQA, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.415582477 (297h12m29.01235351s) [noise=true , address=145.79.227.24:4615 , piece_id=ZSEUUMSYRSXH4AJ3CDZNFKXBZFT63AKVDFBFJ2RLAZK344GQ5AFA, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.415917210 (297h12m29.012688235s) [noise=true , address=193.149.164.68:4040 , piece_id=4NT224HIHI45JJNYH3ZCXQKDAMQHHYWY7SZPYQK5KN7JKNYFGI2A, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.416035983 (297h12m29.012807012s) [noise=true , address=193.149.164.67:4030 , piece_id=PYGFDMQLZ2SYL5NTWUGQGHWW52T2YYEGDKP5SUWQQUTVJSJZOO5A, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.416379117 (297h12m29.013150149s) [noise=true , address=45.91.116.5:4322 , piece_id=FN763EWEYGTBEXNHOT3FMKLW52DENMGPRBPBJJHNWQUNQ5JIN3XA, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.416442412 (297h12m29.013213436s) [noise=true , address=145.79.227.23:4564 , piece_id=3S5IXK7TKV2W3I635W2LQDH3PC6JRPHIVBK347CE7EYE4QQ5AGBQ, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.416786035 (297h12m29.013557058s) [noise=true , address=167.17.72.71:4808 , piece_id=JL5TJCWL454VA73PMWKZYBAUT6ZMRPEH55KJDSHEELC37HRVCO7Q, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.416946153 (297h12m29.013717177s) [noise=true , address=64.49.13.169:4705 , piece_id=FW7ICJDZJVOQR77HCGAWU7NKSBTWASW3BXH72WBGZHOFVPXA7XQQ, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.417084028 (297h12m29.013855051s) [noise=true , address=145.79.227.44:4202 , piece_id=UI7I4UPDOZCJ2MRPKV6FP6YPYNVSR5JWYDVFBO2TRFYQJWVWIUNQ, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.424729560 (297h12m29.021500589s) [noise=true , address=79.127.205.233:4156 , piece_id=5OZYLGXB7R6VMYGATW7GJD2CNAVO45PATTCFGNPWYZ3ZURYLZSQA, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.425355484 (297h12m29.022126513s) [noise=true , address=193.149.164.69:4077 , piece_id=GYZMBIAEBL4KBCSFA4H5JLWEB36NBN2BGVQ67A5G4ECUWY5BACXA, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.429910441 (297h12m29.026681471s) [noise=true , address=79.127.219.46:4918 , piece_id=WC2RL2PMO4EVATTOPZBWFHOZE6CBGQ2ROGJ44PDQUPQKYXBFKY6Q, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.431104280 (297h12m29.027875311s) [noise=true , address=79.127.219.45:4883 , piece_id=7DXMZGFUW2QAI2OID7WSSMJG6B3WW52HCMRWX2LS6YFEWAFMZZNQ, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.432173523 (297h12m29.02894455s) [noise=true , address=193.149.164.71:4300 , piece_id=4FI5PVS627OOB2M45U4QTOXGBJHNOZMUDDCFAW57W7PBAAQSJC4A, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.478900188 (297h12m29.075671211s) [upload=332883]: Next returned nil segment Mar 18 20:12:31.478914031 (297h12m29.075685054s) [upload=332883]: Waiting for error group managing segments... Mar 18 20:12:31.512891809 (297h12m29.109662832s) [noise=true , address=79.127.219.46:4918 , piece_id=WC2RL2PMO4EVATTOPZBWFHOZE6CBGQ2ROGJ44PDQUPQKYXBFKY6Q, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil> Mar 18 20:12:31.529013012 (297h12m29.125784038s) [noise=true , address=45.91.116.5:4322 , piece_id=FN763EWEYGTBEXNHOT3FMKLW52DENMGPRBPBJJHNWQUNQ5JIN3XA, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil> Mar 18 20:12:31.564707355 (297h12m29.161478379s) [noise=true , address=79.127.219.45:4883 , piece_id=7DXMZGFUW2QAI2OID7WSSMJG6B3WW52HCMRWX2LS6YFEWAFMZZNQ, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil> Mar 18 20:12:31.611013762 (297h12m29.207784793s) [noise=true , address=5.231.73.4:4002 , piece_id=3DWXH2Y6ZZNDHRFMJFL6EEMJN7WNDRKJ3ARFKN6UAGAHICNCBSDA, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil> Mar 18 20:12:31.713823345 (297h12m29.310594381s) [noise=true , address=167.17.72.68:4653 , piece_id=6TO53KV4DBJ5HTWIQUZY2L6WVMB4PBPRWSZD6WHBLVFI5FUULYUQ, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil> Mar 18 20:12:31.997071787 (297h12m29.59384282s) [noise=true , address=79.127.205.233:4156 , piece_id=5OZYLGXB7R6VMYGATW7GJD2CNAVO45PATTCFGNPWYZ3ZURYLZSQA, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil> Mar 18 20:12:31.999264198 (297h12m29.59603524s) [noise=true , address=193.149.164.70:4111 , piece_id=H6GX5VVLRFNUAF2R7FRZ5GFQCZ2F57OL4GXIHWQJGZDOM3ST5DKQ, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil> Mar 18 20:12:32.008593285 (297h12m29.605364328s) [noise=true , address=193.149.164.71:4300 , piece_id=4FI5PVS627OOB2M45U4QTOXGBJHNOZMUDDCFAW57W7PBAAQSJC4A, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil> Mar 18 20:12:32.011203033 (297h12m29.607974056s) [noise=true , address=193.149.164.68:4040 , piece_id=4NT224HIHI45JJNYH3ZCXQKDAMQHHYWY7SZPYQK5KN7JKNYFGI2A, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil> Mar 18 20:12:32.038310240 (297h12m29.635081269s) [noise=true , address=193.149.164.67:4030 , piece_id=PYGFDMQLZ2SYL5NTWUGQGHWW52T2YYEGDKP5SUWQQUTVJSJZOO5A, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil> Mar 18 20:12:32.091831027 (297h12m29.68860206s) [noise=true , address=64.49.13.169:4705 , piece_id=FW7ICJDZJVOQR77HCGAWU7NKSBTWASW3BXH72WBGZHOFVPXA7XQQ, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil> Mar 18 20:12:32.408908738 (297h12m30.005679764s) [noise=true , address=193.149.164.69:4077 , piece_id=GYZMBIAEBL4KBCSFA4H5JLWEB36NBN2BGVQ67A5G4ECUWY5BACXA, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil> Mar 18 20:12:32.489564296 (297h12m30.086335307s) [noise=true , address=167.17.72.71:4808 , piece_id=JL5TJCWL454VA73PMWKZYBAUT6ZMRPEH55KJDSHEELC37HRVCO7Q, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil> Mar 18 20:13:28.204470030 (297h13m25.801241064s) [noise=true , address=145.79.227.23:4564 , piece_id=3S5IXK7TKV2W3I635W2LQDH3PC6JRPHIVBK347CE7EYE4QQ5AGBQ, seg_pos={0 0}, upload=332883]: Done uploading piece. err: ecclient: upload failed (node:121WW4uBQe8bGQKkasQQKT3k4EAdzncsG1ebNvgKf8S5SyNttCc, address:145.79.227.23:4564): protocol: rpc: dial tcp 145.79.227.23:4564: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:13:28.205066456 (297h13m25.801837482s) [noise=true , address=145.79.227.28:4750 , piece_id=KJVCFR2SJ5HAQKJUJP4CYPVCZQC6PBAQII6UGNYVKRODWRKENFQA, seg_pos={0 0}, upload=332883]: Done uploading piece. err: ecclient: upload failed (node:12UP8PxqeK7t24PXjopH1nziHYQFnsxhkamSHz8dmwcbuDxkAF6, address:145.79.227.28:4750): protocol: rpc: dial tcp 145.79.227.28:4750: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:13:28.205099998 (297h13m25.80187103s) [noise=true , address=145.79.227.44:4202 , piece_id=UI7I4UPDOZCJ2MRPKV6FP6YPYNVSR5JWYDVFBO2TRFYQJWVWIUNQ, seg_pos={0 0}, upload=332883]: Done uploading piece. err: ecclient: upload failed (node:121ogdaiVGybXEVsS8LDJuJRYXGxb1LCNnGPuwbde96eSEzkTW4, address:145.79.227.44:4202): protocol: rpc: dial tcp 145.79.227.44:4202: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:13:28.205340404 (297h13m25.802111436s) [noise=true , address=145.79.227.24:4615 , piece_id=ZSEUUMSYRSXH4AJ3CDZNFKXBZFT63AKVDFBFJ2RLAZK344GQ5AFA, seg_pos={0 0}, upload=332883]: Done uploading piece. err: ecclient: upload failed (node:12NJ5w8pAWCudDA9Xs4ES7b2evnbscesd682wJDh2gMMZ3Jf2zc, address:145.79.227.24:4615): protocol: rpc: dial tcp 145.79.227.24:4615: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:13:28.205593210 (297h13m25.802364235s) [noise=true , address=145.79.227.20:4454 , piece_id=F2WUAPOH2J3GQ7DMWLBJA44UGY5B3MMDS2U5ITXTOIPR4ELY5QYA, seg_pos={0 0}, upload=332883]: Done uploading piece. err: ecclient: upload failed (node:1SafkJPqKHLwMaqyZdLq3yJ1xdjqDtWHFrUwynmegUVwEC7y5J, address:145.79.227.20:4454): protocol: context deadline exceeded; context deadline exceeded Mar 18 20:13:28.205633541 (297h13m25.802404553s) [seg_pos={0 0}, upload=332883]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 18 20:13:28.228774364 (297h13m25.825545388s) [upload=332883]: Done uploading segments... Mar 18 20:13:28.228821883 (297h13m25.825592906s) [upload=332883]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 1.264694321s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 55 out of 60 is failed (threshold=6) [XXX.XXX...XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.] duration: 515.070584ms trace: https://jaeger.dev.storj.io/trace/ee31ab6c3d5a594 FAIL (2026-03-18 20:17:18 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 515.315481ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 50 out of 60 is failed (threshold=3) [..XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.XXXXXX...XX.X.XXX.X.] duration: 516.227609ms trace: https://jaeger.dev.storj.io/trace/4655811ada2b7979 FAIL (2026-03-18 20:17:15 UTC) operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Mar 18 20:16:45.168516210 (297h16m42.765287245s) [upload=332961]: Starting upload Mar 18 20:16:45.168577895 (297h16m42.765348927s) [upload=332961]: Done starting upload Mar 18 20:16:45.168711660 (297h16m42.765482684s) [upload=332961]: Uploading segments... Mar 18 20:16:45.168840198 (297h16m42.765611213s) [upload=332961]: Got next segment. Inline: false Mar 18 20:16:45.168881241 (297h16m42.765652266s) [upload=332961]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 20:16:45.168886171 (297h16m42.765657197s) [upload=332961]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 18 20:16:47.196356553 (297h16m44.793127577s) [seg_pos={0 0}, upload=332961]: Begin upload segment... Mar 18 20:16:47.196451859 (297h16m44.793222892s) [seg_pos={0 0}, upload=332961]: Done begin upload segment. Mar 18 20:16:47.196741750 (297h16m44.793512778s) [noise=true , address=45.91.116.13:4153 , piece_id=EKOL7CYAY63SJRBKYKOV34L4CP4NX7S3SVQLL7RISQPK6ME5GYJA, seg_pos={0 0}, upload=332961]: Uploading piece... Mar 18 20:16:47.199338232 (297h16m44.796109248s) [noise=true , address=145.79.227.26:4658 , piece_id=6GSGUCXKMEKLDBD52JYYFVBUD4FW7NTF5BHWYDF2QK75O43HV2OQ, seg_pos={0 0}, upload=332961]: Uploading piece... Mar 18 20:16:47.200556587 (297h16m44.797327603s) [noise=true , address=77.90.31.17:4770 , piece_id=ECAEHMROWH75SSG35XXVQX3QJWG3Q2YPJEFBH5VAOY5ZJXSOQ24Q, seg_pos={0 0}, upload=332961]: Uploading piece... Mar 18 20:16:47.201992032 (297h16m44.798763048s) [noise=true , address=77.90.31.12:4589 , piece_id=EWOGPPYQ65XQS4JCOQJEO35WPQMDYRBTJHVAO6GVQXZVIY7BFWEQ, seg_pos={0 0}, upload=332961]: Uploading piece... Mar 18 20:16:47.207224672 (297h16m44.803995689s) [noise=true , address=79.127.219.45:4854 , piece_id=S7ZO66UBY4PDELP5O44R3MAK66JYFJLJWEYWGA5H7G2PKGWT7YAA, seg_pos={0 0}, upload=332961]: Uploading piece... Mar 18 20:16:47.208081435 (297h16m44.804852452s) [noise=true , address=77.90.31.15:4697 , piece_id=GQWUXVCTNAMW3IGMNDP2GKOITAKY6BZLBKMZTBKNWCVK7SPOK6TA, seg_pos={0 0}, upload=332961]: Uploading piece... Mar 18 20:16:47.210256535 (297h16m44.807027558s) [noise=true , address=145.79.227.43:4159 , piece_id=F3DV62MNUMKWLCHSOUW5RTDTEIISLAYH5BK3O4OHPRL53CYA25EA, seg_pos={0 0}, upload=332961]: Uploading piece... Mar 18 20:16:47.212078721 (297h16m44.808849737s) [noise=true , address=77.90.31.11:4553 , piece_id=GVGIPFEDZVR5VKPQLM6EL2F6GHZRQMCJBPWBAK6BUR2H6BSKOZWA, seg_pos={0 0}, upload=332961]: Uploading piece... Mar 18 20:16:47.214581476 (297h16m44.811352485s) [noise=true , address=77.90.31.18:4805 , piece_id=YTV5TPM7TBRDTQOGODV345Y2EQVV7O6H2A4GRB7UW5H4IO2ZBKYQ, seg_pos={0 0}, upload=332961]: Uploading piece... Mar 18 20:16:47.217434080 (297h16m44.814205104s) [noise=true , address=145.79.227.28:4728 , piece_id=M5TQNH7BTNVQOHGYJOAOCSYULZYMNJUEL3DM3MMWB2GOXYCKSPFA, seg_pos={0 0}, upload=332961]: Uploading piece... Mar 18 20:16:47.220446152 (297h16m44.817217167s) [noise=true , address=45.91.116.10:4088 , piece_id=3VYOU6U7UMCSDNSIXPLXXOUXU4F7K5G6FMB3HMC4QTDX6C6S2QPA, seg_pos={0 0}, upload=332961]: Uploading piece... Mar 18 20:16:47.222777337 (297h16m44.819548354s) [noise=true , address=145.79.227.41:4087 , piece_id=YXLENLZ7JQLRT566LJPW7SFVI3XAKHI6FZ3D6K6DSRAC6GU577HQ, seg_pos={0 0}, upload=332961]: Uploading piece... Mar 18 20:16:47.225963836 (297h16m44.822734955s) [noise=true , address=145.79.227.42:4121 , piece_id=N7HRLH4ONBTX5DFXFOJKAUNLYDC4W7Z55LBEFPJNXN76JO4JU7MA, seg_pos={0 0}, upload=332961]: Uploading piece... Mar 18 20:16:47.228972751 (297h16m44.825743767s) [noise=true , address=45.91.116.4:4940 , piece_id=JCBPX5UELTV45DQSWPKZUS5PPRAUUN2C6KLVQN5R7ATKKUYCGJRA, seg_pos={0 0}, upload=332961]: Uploading piece... Mar 18 20:16:47.232619020 (297h16m44.829390037s) [noise=true , address=45.91.116.7:4019 , piece_id=7HFQHEC4EXQ6VSHY2KO2BCTUUH4SCSHZEEQEL2UFL56HNQNBMZGA, seg_pos={0 0}, upload=332961]: Uploading piece... Mar 18 20:16:47.235321732 (297h16m44.832092748s) [noise=true , address=45.91.116.5:4298 , piece_id=UB4T5U5EU5ZOJLNSDA5SVEKOL33R5V3TPOOBHY53ZZBFCCKEN45Q, seg_pos={0 0}, upload=332961]: Uploading piece... Mar 18 20:16:47.280083394 (297h16m44.876854417s) [upload=332961]: Next returned nil segment Mar 18 20:16:47.280097207 (297h16m44.876868231s) [upload=332961]: Waiting for error group managing segments... Mar 18 20:16:47.284956192 (297h16m44.881727226s) [noise=true , address=77.90.31.18:4805 , piece_id=YTV5TPM7TBRDTQOGODV345Y2EQVV7O6H2A4GRB7UW5H4IO2ZBKYQ, seg_pos={0 0}, upload=332961]: Done uploading piece. err: <nil> Mar 18 20:16:47.285026311 (297h16m44.881797335s) [noise=true , address=45.91.116.10:4088 , piece_id=3VYOU6U7UMCSDNSIXPLXXOUXU4F7K5G6FMB3HMC4QTDX6C6S2QPA, seg_pos={0 0}, upload=332961]: Done uploading piece. err: <nil> Mar 18 20:16:47.285594755 (297h16m44.882365791s) [noise=true , address=45.91.116.7:4019 , piece_id=7HFQHEC4EXQ6VSHY2KO2BCTUUH4SCSHZEEQEL2UFL56HNQNBMZGA, seg_pos={0 0}, upload=332961]: Done uploading piece. err: <nil> Mar 18 20:16:47.286555006 (297h16m44.88332603s) [noise=true , address=45.91.116.13:4153 , piece_id=EKOL7CYAY63SJRBKYKOV34L4CP4NX7S3SVQLL7RISQPK6ME5GYJA, seg_pos={0 0}, upload=332961]: Done uploading piece. err: <nil> Mar 18 20:16:47.287363557 (297h16m44.884134591s) [noise=true , address=77.90.31.12:4589 , piece_id=EWOGPPYQ65XQS4JCOQJEO35WPQMDYRBTJHVAO6GVQXZVIY7BFWEQ, seg_pos={0 0}, upload=332961]: Done uploading piece. err: <nil> Mar 18 20:16:47.294896048 (297h16m44.891667081s) [noise=true , address=77.90.31.17:4770 , piece_id=ECAEHMROWH75SSG35XXVQX3QJWG3Q2YPJEFBH5VAOY5ZJXSOQ24Q, seg_pos={0 0}, upload=332961]: Done uploading piece. err: <nil> Mar 18 20:16:47.296893527 (297h16m44.893664551s) [noise=true , address=45.91.116.5:4298 , piece_id=UB4T5U5EU5ZOJLNSDA5SVEKOL33R5V3TPOOBHY53ZZBFCCKEN45Q, seg_pos={0 0}, upload=332961]: Done uploading piece. err: <nil> Mar 18 20:16:47.297430780 (297h16m44.894201818s) [noise=true , address=45.91.116.4:4940 , piece_id=JCBPX5UELTV45DQSWPKZUS5PPRAUUN2C6KLVQN5R7ATKKUYCGJRA, seg_pos={0 0}, upload=332961]: Done uploading piece. err: <nil> Mar 18 20:16:47.299517292 (297h16m44.896288305s) [noise=true , address=77.90.31.11:4553 , piece_id=GVGIPFEDZVR5VKPQLM6EL2F6GHZRQMCJBPWBAK6BUR2H6BSKOZWA, seg_pos={0 0}, upload=332961]: Done uploading piece. err: <nil> Mar 18 20:16:47.305052080 (297h16m44.901823106s) [noise=true , address=77.90.31.15:4697 , piece_id=GQWUXVCTNAMW3IGMNDP2GKOITAKY6BZLBKMZTBKNWCVK7SPOK6TA, seg_pos={0 0}, upload=332961]: Done uploading piece. err: <nil> Mar 18 20:16:47.305526292 (297h16m44.902297317s) [noise=true , address=79.127.219.45:4854 , piece_id=S7ZO66UBY4PDELP5O44R3MAK66JYFJLJWEYWGA5H7G2PKGWT7YAA, seg_pos={0 0}, upload=332961]: Done uploading piece. err: <nil> Mar 18 20:17:15.132499335 (297h17m12.72927036s) [noise=true , address=145.79.227.42:4121 , piece_id=N7HRLH4ONBTX5DFXFOJKAUNLYDC4W7Z55LBEFPJNXN76JO4JU7MA, seg_pos={0 0}, upload=332961]: Done uploading piece. err: ecclient: upload failed (node:1QJBWV6xLXS7DcX8d26be79z1DH3r27oi4EcuiwVwR5Dn8JEGh, address:145.79.227.42:4121): protocol: rpc: dial tcp 145.79.227.42:4121: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:17:15.133503333 (297h17m12.730274376s) [noise=true , address=145.79.227.28:4728 , piece_id=M5TQNH7BTNVQOHGYJOAOCSYULZYMNJUEL3DM3MMWB2GOXYCKSPFA, seg_pos={0 0}, upload=332961]: Done uploading piece. err: ecclient: upload failed (node:1YoLzpb4mMYZ8TsdxnMocekZZ1gLfTrdCM8mxTDj4tY6khZGvi, address:145.79.227.28:4728): protocol: rpc: dial tcp 145.79.227.28:4728: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:17:15.133561110 (297h17m12.730332141s) [noise=true , address=145.79.227.26:4658 , piece_id=6GSGUCXKMEKLDBD52JYYFVBUD4FW7NTF5BHWYDF2QK75O43HV2OQ, seg_pos={0 0}, upload=332961]: Done uploading piece. err: ecclient: upload failed (node:1gcSHRechDzTDwivRoAwNFfNj5Zjqvi6oJwVqzbCQinwgWtpPr, address:145.79.227.26:4658): protocol: rpc: dial tcp 145.79.227.26:4658: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:17:15.134239018 (297h17m12.731010051s) [noise=true , address=145.79.227.43:4159 , piece_id=F3DV62MNUMKWLCHSOUW5RTDTEIISLAYH5BK3O4OHPRL53CYA25EA, seg_pos={0 0}, upload=332961]: Done uploading piece. err: ecclient: upload failed (node:12ULUPKjJwZocjV1R6Dy7vmWSrZ4hKx3YAHLAihgtMKf8Uw2UgA, address:145.79.227.43:4159): protocol: rpc: dial tcp 145.79.227.43:4159: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:17:15.134625412 (297h17m12.731396448s) [noise=true , address=145.79.227.41:4087 , piece_id=YXLENLZ7JQLRT566LJPW7SFVI3XAKHI6FZ3D6K6DSRAC6GU577HQ, seg_pos={0 0}, upload=332961]: Done uploading piece. err: ecclient: upload failed (node:1BfeGD2MGRPqsBRGLioTiTAcPykpSTUUx2CBVSPQcz4LQqFdWR, address:145.79.227.41:4087): protocol: rpc: dial tcp 145.79.227.41:4087: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:17:15.135255447 (297h17m12.73202646s) [seg_pos={0 0}, upload=332961]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Mar 18 20:17:15.156515485 (297h17m12.753286522s) [upload=332961]: Done uploading segments... Mar 18 20:17:15.156591674 (297h17m12.753362706s) [upload=332961]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 516.244475ms core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 8 out of 60 is failed (threshold=3) [.....XXXXXX.XX..............................................] duration: 4.658750631s trace: https://jaeger.dev.storj.io/trace/4b4d10a8051ad01b FAIL (2026-03-18 20:17:02 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.658770743s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 6 out of 60 is failed (threshold=3) [..........X..........XX..X....X........X....................] duration: 10.194377922s trace: https://jaeger.dev.storj.io/trace/589419efc52e5e2b FAIL (2026-03-18 20:16:51 UTC) operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== Mar 18 20:00:59.343084966 (297h0m56.939855989s) [upload=332666]: Starting upload Mar 18 20:00:59.343175518 (297h0m56.939946548s) [upload=332666]: Done starting upload Mar 18 20:00:59.343565585 (297h0m56.940336612s) [upload=332666]: Uploading segments... Mar 18 20:00:59.344256647 (297h0m56.941027676s) [upload=332666]: Got next segment. Inline: false Mar 18 20:00:59.344352215 (297h0m56.941123241s) [upload=332666]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 20:00:59.344358706 (297h0m56.941129732s) [upload=332666]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 18 20:01:02.340259299 (297h0m59.937030329s) [seg_pos={0 0}, upload=332666]: Begin upload segment... Mar 18 20:01:02.340368743 (297h0m59.937139767s) [seg_pos={0 0}, upload=332666]: Done begin upload segment. Mar 18 20:01:02.340479563 (297h0m59.93725059s) [noise=true , address=45.91.116.8:4061 , piece_id=6XFVKT4AAO7QN457OQMRKPI6R24TJTAN47UXLEVOEFQVOS6M6C3Q, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.340523837 (297h0m59.93729486s) [noise=true , address=79.127.205.233:4166 , piece_id=SFGK5YCWHFRVXF6QNHYP2CYJCZLSTQ76OYRVJPWWXQTF2KFSFJKA, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.341468438 (297h0m59.938239447s) [noise=true , address=45.91.116.4:4961 , piece_id=ZMUGQECUVPVHDNKN75XPBET5M2HFXXK4WMTP4V5KEK63JVZCX5EA, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.341520025 (297h0m59.938291058s) [noise=true , address=167.17.72.71:4815 , piece_id=CXGTYNNBILN3TT7I6Z7UI75F2ZCS4QUTLS6KC4FJ5IHS3XOYWDJQ, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.342232992 (297h0m59.939004023s) [noise=true , address=145.79.227.36:4524 , piece_id=3TAV3P55ITTHBAZEUUUTOAHJRGQXJXXVGZJP6AVR2ANVNOR3S54Q, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.342917368 (297h0m59.939688398s) [noise=true , address=193.149.164.71:4297 , piece_id=4IJPXLPCX3LKFOBOSRHK2DTNRKH67K7FXJUKQ3JPRWHS47ZFEIDA, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.343244957 (297h0m59.940015982s) [noise=true , address=145.79.227.33:4892 , piece_id=DQ6ALC3VOAGFIJI3IKR6I7QFYM22C66QWAAXMH6NXI2FIQEZNQYA, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.344270079 (297h0m59.941041117s) [noise=true , address=145.79.227.21:4497 , piece_id=SP6EKJB7W4QXF7OIBVKEFWCPOZFL5IRIEDVAGS2M4QJSO34CUVRQ, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.345025057 (297h0m59.941796084s) [noise=true , address=77.90.31.17:4793 , piece_id=2U4IBUA2GGUGDBXQGIGQKUK4ZR3L7CDKC6MTGIKAUME4F7MFZRLA, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.345115802 (297h0m59.941886836s) [noise=true , address=167.17.72.69:4731 , piece_id=VOEVIKKFJMC2Y3TPQIDMYUP37NLFU7KPRH4HRRSMQLWUKTSLOH2Q, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.345690600 (297h0m59.942461634s) [noise=true , address=77.90.31.13:4648 , piece_id=BPMB5CFOYHFT3BS4ORR6JF6BNFC2AOAJZOSPMG67NAMLGGP5XVEQ, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.346311592 (297h0m59.943082623s) [noise=true , address=79.127.205.234:4202 , piece_id=YY4LYWX26O6RIOBONESNKTUZEFZCRL4UUX3NEBQ53AJV4IEDG5MQ, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.346791142 (297h0m59.943562166s) [noise=true , address=45.91.116.10:4110 , piece_id=UYLMDCWJEJOHGICA37KNH7G5AQXNIPRH5NRYLBUUMGDRVSLTEYHQ, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.347094999 (297h0m59.94386607s) [noise=true , address=145.79.227.17:4987 , piece_id=PT3TWVB65JZFEWVH4IUZEU2N53FQ57QBRWVLWAUFP4UDLY3LWNLQ, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.347781281 (297h0m59.944552305s) [noise=true , address=79.127.219.46:4919 , piece_id=KK6HRIVBMYLIHRHGPQH3AWMNSVHTTIBESHDMJZA72T446RX6QKRQ, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.348263123 (297h0m59.945034137s) [noise=true , address=193.149.164.68:4033 , piece_id=JZPQKPYGO3W3RZZBITQ73X2XEPIBBSLGRMWIVOM4Z3GTV5UFKRTQ, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.348915505 (297h0m59.945686536s) [noise=true , address=145.79.227.7:4312 , piece_id=2FAL6E3RMBTBHZBXAJJVH2N5VGF5VYI3KNZXGNWCMWGBBPAUO2NQ, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.349366298 (297h0m59.946137337s) [noise=true , address=193.149.164.67:4007 , piece_id=HURQGH67I5B6V4YUV3PRZL3KSKN3XTSK3UMCHZRNT6L3W3PXXHPQ, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.429834098 (297h1m0.026605126s) [upload=332666]: Next returned nil segment Mar 18 20:01:02.429853411 (297h1m0.026624442s) [upload=332666]: Waiting for error group managing segments... Mar 18 20:01:02.446867680 (297h1m0.043638704s) [noise=true , address=45.91.116.10:4110 , piece_id=UYLMDCWJEJOHGICA37KNH7G5AQXNIPRH5NRYLBUUMGDRVSLTEYHQ, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil> Mar 18 20:01:02.451665165 (297h1m0.048436196s) [noise=true , address=45.91.116.4:4961 , piece_id=ZMUGQECUVPVHDNKN75XPBET5M2HFXXK4WMTP4V5KEK63JVZCX5EA, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil> Mar 18 20:01:02.452538886 (297h1m0.049309913s) [noise=true , address=77.90.31.17:4793 , piece_id=2U4IBUA2GGUGDBXQGIGQKUK4ZR3L7CDKC6MTGIKAUME4F7MFZRLA, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil> Mar 18 20:01:02.455597040 (297h1m0.052368067s) [noise=true , address=45.91.116.8:4061 , piece_id=6XFVKT4AAO7QN457OQMRKPI6R24TJTAN47UXLEVOEFQVOS6M6C3Q, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil> Mar 18 20:01:02.457848739 (297h1m0.054619766s) [noise=true , address=77.90.31.13:4648 , piece_id=BPMB5CFOYHFT3BS4ORR6JF6BNFC2AOAJZOSPMG67NAMLGGP5XVEQ, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil> Mar 18 20:01:02.461479709 (297h1m0.058250746s) [noise=true , address=79.127.219.46:4919 , piece_id=KK6HRIVBMYLIHRHGPQH3AWMNSVHTTIBESHDMJZA72T446RX6QKRQ, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil> Mar 18 20:01:02.676859466 (297h1m0.27363049s) [noise=true , address=167.17.72.69:4731 , piece_id=VOEVIKKFJMC2Y3TPQIDMYUP37NLFU7KPRH4HRRSMQLWUKTSLOH2Q, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil> Mar 18 20:01:02.909860743 (297h1m0.506631766s) [noise=true , address=79.127.205.234:4202 , piece_id=YY4LYWX26O6RIOBONESNKTUZEFZCRL4UUX3NEBQ53AJV4IEDG5MQ, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil> Mar 18 20:01:02.930427742 (297h1m0.527198754s) [noise=true , address=193.149.164.71:4297 , piece_id=4IJPXLPCX3LKFOBOSRHK2DTNRKH67K7FXJUKQ3JPRWHS47ZFEIDA, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil> Mar 18 20:01:02.944696889 (297h1m0.541467914s) [noise=true , address=193.149.164.67:4007 , piece_id=HURQGH67I5B6V4YUV3PRZL3KSKN3XTSK3UMCHZRNT6L3W3PXXHPQ, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil> Mar 18 20:01:02.951319183 (297h1m0.548090212s) [noise=true , address=193.149.164.68:4033 , piece_id=JZPQKPYGO3W3RZZBITQ73X2XEPIBBSLGRMWIVOM4Z3GTV5UFKRTQ, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil> Mar 18 20:01:02.975619436 (297h1m0.572390467s) [noise=true , address=79.127.205.233:4166 , piece_id=SFGK5YCWHFRVXF6QNHYP2CYJCZLSTQ76OYRVJPWWXQTF2KFSFJKA, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil> Mar 18 20:01:03.819918528 (297h1m1.416689564s) [noise=true , address=167.17.72.71:4815 , piece_id=CXGTYNNBILN3TT7I6Z7UI75F2ZCS4QUTLS6KC4FJ5IHS3XOYWDJQ, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil> Mar 18 20:01:59.307873420 (297h1m56.904644452s) [noise=true , address=145.79.227.7:4312 , piece_id=2FAL6E3RMBTBHZBXAJJVH2N5VGF5VYI3KNZXGNWCMWGBBPAUO2NQ, seg_pos={0 0}, upload=332666]: Done uploading piece. err: ecclient: upload failed (node:12AVp3qn5uwLPqYaZEwzcgoGMJhNga1BUh6bK5k2bxjWgZLQiiC, address:145.79.227.7:4312): protocol: rpc: dial tcp 145.79.227.7:4312: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:01:59.308026092 (297h1m56.904797124s) [noise=true , address=145.79.227.17:4987 , piece_id=PT3TWVB65JZFEWVH4IUZEU2N53FQ57QBRWVLWAUFP4UDLY3LWNLQ, seg_pos={0 0}, upload=332666]: Done uploading piece. err: ecclient: upload failed (node:1XsPVNGYUhKnXkQRfVfjyBRzyvhShSdsrnzXtwe4Xtf5MaHMvH, address:145.79.227.17:4987): protocol: rpc: dial tcp 145.79.227.17:4987: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:01:59.308195864 (297h1m56.904966894s) [noise=true , address=145.79.227.36:4524 , piece_id=3TAV3P55ITTHBAZEUUUTOAHJRGQXJXXVGZJP6AVR2ANVNOR3S54Q, seg_pos={0 0}, upload=332666]: Done uploading piece. err: ecclient: upload failed (node:1xfbCzjSwFDRrR1gPggeN8tNCLbRRWDVzCJ6mqhZsbVSe8d4s4, address:145.79.227.36:4524): protocol: rpc: dial tcp 145.79.227.36:4524: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:01:59.308316037 (297h1m56.90508706s) [noise=true , address=145.79.227.33:4892 , piece_id=DQ6ALC3VOAGFIJI3IKR6I7QFYM22C66QWAAXMH6NXI2FIQEZNQYA, seg_pos={0 0}, upload=332666]: Done uploading piece. err: ecclient: upload failed (node:12RHwFxcUjPsn12uHEAH3fBDJKv1v1tM8mRJXoFrZrG5djdswF5, address:145.79.227.33:4892): protocol: rpc: dial tcp 145.79.227.33:4892: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:01:59.308404403 (297h1m56.905175433s) [noise=true , address=145.79.227.21:4497 , piece_id=SP6EKJB7W4QXF7OIBVKEFWCPOZFL5IRIEDVAGS2M4QJSO34CUVRQ, seg_pos={0 0}, upload=332666]: Done uploading piece. err: ecclient: upload failed (node:12ZBe4we7VeFdthqRxyy1V9uQrA4r8ouNX68xqsk5bD26dkf2Gd, address:145.79.227.21:4497): protocol: context deadline exceeded; context deadline exceeded Mar 18 20:01:59.308437457 (297h1m56.90520847s) [seg_pos={0 0}, upload=332666]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 18 20:01:59.338288467 (297h1m56.93505949s) [upload=332666]: Done uploading segments... Mar 18 20:01:59.338335497 (297h1m56.935106523s) [upload=332666]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 10.194408717s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 8 out of 60 is failed (threshold=3) [......XXXXXX.XX.............................................] duration: 9.299387716s trace: https://jaeger.dev.storj.io/trace/4b4d10a8051ad01b FAIL (2026-03-18 20:16:49 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 9.299472908s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 56 out of 60 is failed (threshold=6) [XXXX.XXX...XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 5.001240204s trace: https://jaeger.dev.storj.io/trace/ee31ab6c3d5a594 FAIL (2026-03-18 20:16:31 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001258147s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 8 out of 60 is failed (threshold=3) [.......XXXXXX.XX............................................] duration: 18.368346829s trace: https://jaeger.dev.storj.io/trace/4b4d10a8051ad01b FAIL (2026-03-18 20:16:24 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 18.368373376s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 9 out of 60 is failed (threshold=3) [.........XXX..........X...X.........X....X....X.......X.....] duration: 1.763927428s trace: https://jaeger.dev.storj.io/trace/24517a7cd40ab76d FAIL (2026-03-18 20:16:20 UTC) operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== oad Mar 18 20:12:28.252047067 (297h12m25.848818092s) [upload=332883]: Uploading segments... Mar 18 20:12:28.252323178 (297h12m25.84909421s) [upload=332883]: Got next segment. Inline: false Mar 18 20:12:28.252380536 (297h12m25.849151569s) [upload=332883]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 20:12:28.252388587 (297h12m25.849159622s) [upload=332883]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 18 20:12:31.413020979 (297h12m29.009792004s) [seg_pos={0 0}, upload=332883]: Begin upload segment... Mar 18 20:12:31.413109194 (297h12m29.009880217s) [seg_pos={0 0}, upload=332883]: Done begin upload segment. Mar 18 20:12:31.413182895 (297h12m29.009953924s) [noise=true , address=167.17.72.68:4653 , piece_id=6TO53KV4DBJ5HTWIQUZY2L6WVMB4PBPRWSZD6WHBLVFI5FUULYUQ, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.413918965 (297h12m29.010689999s) [noise=true , address=145.79.227.20:4454 , piece_id=F2WUAPOH2J3GQ7DMWLBJA44UGY5B3MMDS2U5ITXTOIPR4ELY5QYA, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.414450842 (297h12m29.011221869s) [noise=true , address=193.149.164.70:4111 , piece_id=H6GX5VVLRFNUAF2R7FRZ5GFQCZ2F57OL4GXIHWQJGZDOM3ST5DKQ, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.414901508 (297h12m29.011672532s) [noise=true , address=5.231.73.4:4002 , piece_id=3DWXH2Y6ZZNDHRFMJFL6EEMJN7WNDRKJ3ARFKN6UAGAHICNCBSDA, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.415342842 (297h12m29.012113865s) [noise=true , address=145.79.227.28:4750 , piece_id=KJVCFR2SJ5HAQKJUJP4CYPVCZQC6PBAQII6UGNYVKRODWRKENFQA, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.415582477 (297h12m29.01235351s) [noise=true , address=145.79.227.24:4615 , piece_id=ZSEUUMSYRSXH4AJ3CDZNFKXBZFT63AKVDFBFJ2RLAZK344GQ5AFA, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.415917210 (297h12m29.012688235s) [noise=true , address=193.149.164.68:4040 , piece_id=4NT224HIHI45JJNYH3ZCXQKDAMQHHYWY7SZPYQK5KN7JKNYFGI2A, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.416035983 (297h12m29.012807012s) [noise=true , address=193.149.164.67:4030 , piece_id=PYGFDMQLZ2SYL5NTWUGQGHWW52T2YYEGDKP5SUWQQUTVJSJZOO5A, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.416379117 (297h12m29.013150149s) [noise=true , address=45.91.116.5:4322 , piece_id=FN763EWEYGTBEXNHOT3FMKLW52DENMGPRBPBJJHNWQUNQ5JIN3XA, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.416442412 (297h12m29.013213436s) [noise=true , address=145.79.227.23:4564 , piece_id=3S5IXK7TKV2W3I635W2LQDH3PC6JRPHIVBK347CE7EYE4QQ5AGBQ, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.416786035 (297h12m29.013557058s) [noise=true , address=167.17.72.71:4808 , piece_id=JL5TJCWL454VA73PMWKZYBAUT6ZMRPEH55KJDSHEELC37HRVCO7Q, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.416946153 (297h12m29.013717177s) [noise=true , address=64.49.13.169:4705 , piece_id=FW7ICJDZJVOQR77HCGAWU7NKSBTWASW3BXH72WBGZHOFVPXA7XQQ, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.417084028 (297h12m29.013855051s) [noise=true , address=145.79.227.44:4202 , piece_id=UI7I4UPDOZCJ2MRPKV6FP6YPYNVSR5JWYDVFBO2TRFYQJWVWIUNQ, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.424729560 (297h12m29.021500589s) [noise=true , address=79.127.205.233:4156 , piece_id=5OZYLGXB7R6VMYGATW7GJD2CNAVO45PATTCFGNPWYZ3ZURYLZSQA, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.425355484 (297h12m29.022126513s) [noise=true , address=193.149.164.69:4077 , piece_id=GYZMBIAEBL4KBCSFA4H5JLWEB36NBN2BGVQ67A5G4ECUWY5BACXA, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.429910441 (297h12m29.026681471s) [noise=true , address=79.127.219.46:4918 , piece_id=WC2RL2PMO4EVATTOPZBWFHOZE6CBGQ2ROGJ44PDQUPQKYXBFKY6Q, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.431104280 (297h12m29.027875311s) [noise=true , address=79.127.219.45:4883 , piece_id=7DXMZGFUW2QAI2OID7WSSMJG6B3WW52HCMRWX2LS6YFEWAFMZZNQ, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.432173523 (297h12m29.02894455s) [noise=true , address=193.149.164.71:4300 , piece_id=4FI5PVS627OOB2M45U4QTOXGBJHNOZMUDDCFAW57W7PBAAQSJC4A, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.478900188 (297h12m29.075671211s) [upload=332883]: Next returned nil segment Mar 18 20:12:31.478914031 (297h12m29.075685054s) [upload=332883]: Waiting for error group managing segments... Mar 18 20:12:31.512891809 (297h12m29.109662832s) [noise=true , address=79.127.219.46:4918 , piece_id=WC2RL2PMO4EVATTOPZBWFHOZE6CBGQ2ROGJ44PDQUPQKYXBFKY6Q, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil> Mar 18 20:12:31.529013012 (297h12m29.125784038s) [noise=true , address=45.91.116.5:4322 , piece_id=FN763EWEYGTBEXNHOT3FMKLW52DENMGPRBPBJJHNWQUNQ5JIN3XA, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil> Mar 18 20:12:31.564707355 (297h12m29.161478379s) [noise=true , address=79.127.219.45:4883 , piece_id=7DXMZGFUW2QAI2OID7WSSMJG6B3WW52HCMRWX2LS6YFEWAFMZZNQ, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil> Mar 18 20:12:31.611013762 (297h12m29.207784793s) [noise=true , address=5.231.73.4:4002 , piece_id=3DWXH2Y6ZZNDHRFMJFL6EEMJN7WNDRKJ3ARFKN6UAGAHICNCBSDA, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil> Mar 18 20:12:31.713823345 (297h12m29.310594381s) [noise=true , address=167.17.72.68:4653 , piece_id=6TO53KV4DBJ5HTWIQUZY2L6WVMB4PBPRWSZD6WHBLVFI5FUULYUQ, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil> Mar 18 20:12:31.997071787 (297h12m29.59384282s) [noise=true , address=79.127.205.233:4156 , piece_id=5OZYLGXB7R6VMYGATW7GJD2CNAVO45PATTCFGNPWYZ3ZURYLZSQA, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil> Mar 18 20:12:31.999264198 (297h12m29.59603524s) [noise=true , address=193.149.164.70:4111 , piece_id=H6GX5VVLRFNUAF2R7FRZ5GFQCZ2F57OL4GXIHWQJGZDOM3ST5DKQ, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil> Mar 18 20:12:32.008593285 (297h12m29.605364328s) [noise=true , address=193.149.164.71:4300 , piece_id=4FI5PVS627OOB2M45U4QTOXGBJHNOZMUDDCFAW57W7PBAAQSJC4A, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil> Mar 18 20:12:32.011203033 (297h12m29.607974056s) [noise=true , address=193.149.164.68:4040 , piece_id=4NT224HIHI45JJNYH3ZCXQKDAMQHHYWY7SZPYQK5KN7JKNYFGI2A, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil> Mar 18 20:12:32.038310240 (297h12m29.635081269s) [noise=true , address=193.149.164.67:4030 , piece_id=PYGFDMQLZ2SYL5NTWUGQGHWW52T2YYEGDKP5SUWQQUTVJSJZOO5A, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil> Mar 18 20:12:32.091831027 (297h12m29.68860206s) [noise=true , address=64.49.13.169:4705 , piece_id=FW7ICJDZJVOQR77HCGAWU7NKSBTWASW3BXH72WBGZHOFVPXA7XQQ, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil> Mar 18 20:12:32.408908738 (297h12m30.005679764s) [noise=true , address=193.149.164.69:4077 , piece_id=GYZMBIAEBL4KBCSFA4H5JLWEB36NBN2BGVQ67A5G4ECUWY5BACXA, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil> Mar 18 20:12:32.489564296 (297h12m30.086335307s) [noise=true , address=167.17.72.71:4808 , piece_id=JL5TJCWL454VA73PMWKZYBAUT6ZMRPEH55KJDSHEELC37HRVCO7Q, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil> Mar 18 20:13:28.204470030 (297h13m25.801241064s) [noise=true , address=145.79.227.23:4564 , piece_id=3S5IXK7TKV2W3I635W2LQDH3PC6JRPHIVBK347CE7EYE4QQ5AGBQ, seg_pos={0 0}, upload=332883]: Done uploading piece. err: ecclient: upload failed (node:121WW4uBQe8bGQKkasQQKT3k4EAdzncsG1ebNvgKf8S5SyNttCc, address:145.79.227.23:4564): protocol: rpc: dial tcp 145.79.227.23:4564: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:13:28.205066456 (297h13m25.801837482s) [noise=true , address=145.79.227.28:4750 , piece_id=KJVCFR2SJ5HAQKJUJP4CYPVCZQC6PBAQII6UGNYVKRODWRKENFQA, seg_pos={0 0}, upload=332883]: Done uploading piece. err: ecclient: upload failed (node:12UP8PxqeK7t24PXjopH1nziHYQFnsxhkamSHz8dmwcbuDxkAF6, address:145.79.227.28:4750): protocol: rpc: dial tcp 145.79.227.28:4750: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:13:28.205099998 (297h13m25.80187103s) [noise=true , address=145.79.227.44:4202 , piece_id=UI7I4UPDOZCJ2MRPKV6FP6YPYNVSR5JWYDVFBO2TRFYQJWVWIUNQ, seg_pos={0 0}, upload=332883]: Done uploading piece. err: ecclient: upload failed (node:121ogdaiVGybXEVsS8LDJuJRYXGxb1LCNnGPuwbde96eSEzkTW4, address:145.79.227.44:4202): protocol: rpc: dial tcp 145.79.227.44:4202: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:13:28.205340404 (297h13m25.802111436s) [noise=true , address=145.79.227.24:4615 , piece_id=ZSEUUMSYRSXH4AJ3CDZNFKXBZFT63AKVDFBFJ2RLAZK344GQ5AFA, seg_pos={0 0}, upload=332883]: Done uploading piece. err: ecclient: upload failed (node:12NJ5w8pAWCudDA9Xs4ES7b2evnbscesd682wJDh2gMMZ3Jf2zc, address:145.79.227.24:4615): protocol: rpc: dial tcp 145.79.227.24:4615: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:13:28.205593210 (297h13m25.802364235s) [noise=true , address=145.79.227.20:4454 , piece_id=F2WUAPOH2J3GQ7DMWLBJA44UGY5B3MMDS2U5ITXTOIPR4ELY5QYA, seg_pos={0 0}, upload=332883]: Done uploading piece. err: ecclient: upload failed (node:1SafkJPqKHLwMaqyZdLq3yJ1xdjqDtWHFrUwynmegUVwEC7y5J, address:145.79.227.20:4454): protocol: context deadline exceeded; context deadline exceeded Mar 18 20:13:28.205633541 (297h13m25.802404553s) [seg_pos={0 0}, upload=332883]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 18 20:13:28.228774364 (297h13m25.825545388s) [upload=332883]: Done uploading segments... Mar 18 20:13:28.228821883 (297h13m25.825592906s) [upload=332883]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 1.763956547s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 6 out of 60 is failed (threshold=3) [...........X..........XX..X....X........X...................] duration: 13.312327104s trace: https://jaeger.dev.storj.io/trace/589419efc52e5e2b FAIL (2026-03-18 20:16:15 UTC) operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== Mar 18 20:00:59.343084966 (297h0m56.939855989s) [upload=332666]: Starting upload Mar 18 20:00:59.343175518 (297h0m56.939946548s) [upload=332666]: Done starting upload Mar 18 20:00:59.343565585 (297h0m56.940336612s) [upload=332666]: Uploading segments... Mar 18 20:00:59.344256647 (297h0m56.941027676s) [upload=332666]: Got next segment. Inline: false Mar 18 20:00:59.344352215 (297h0m56.941123241s) [upload=332666]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 20:00:59.344358706 (297h0m56.941129732s) [upload=332666]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 18 20:01:02.340259299 (297h0m59.937030329s) [seg_pos={0 0}, upload=332666]: Begin upload segment... Mar 18 20:01:02.340368743 (297h0m59.937139767s) [seg_pos={0 0}, upload=332666]: Done begin upload segment. Mar 18 20:01:02.340479563 (297h0m59.93725059s) [noise=true , address=45.91.116.8:4061 , piece_id=6XFVKT4AAO7QN457OQMRKPI6R24TJTAN47UXLEVOEFQVOS6M6C3Q, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.340523837 (297h0m59.93729486s) [noise=true , address=79.127.205.233:4166 , piece_id=SFGK5YCWHFRVXF6QNHYP2CYJCZLSTQ76OYRVJPWWXQTF2KFSFJKA, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.341468438 (297h0m59.938239447s) [noise=true , address=45.91.116.4:4961 , piece_id=ZMUGQECUVPVHDNKN75XPBET5M2HFXXK4WMTP4V5KEK63JVZCX5EA, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.341520025 (297h0m59.938291058s) [noise=true , address=167.17.72.71:4815 , piece_id=CXGTYNNBILN3TT7I6Z7UI75F2ZCS4QUTLS6KC4FJ5IHS3XOYWDJQ, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.342232992 (297h0m59.939004023s) [noise=true , address=145.79.227.36:4524 , piece_id=3TAV3P55ITTHBAZEUUUTOAHJRGQXJXXVGZJP6AVR2ANVNOR3S54Q, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.342917368 (297h0m59.939688398s) [noise=true , address=193.149.164.71:4297 , piece_id=4IJPXLPCX3LKFOBOSRHK2DTNRKH67K7FXJUKQ3JPRWHS47ZFEIDA, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.343244957 (297h0m59.940015982s) [noise=true , address=145.79.227.33:4892 , piece_id=DQ6ALC3VOAGFIJI3IKR6I7QFYM22C66QWAAXMH6NXI2FIQEZNQYA, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.344270079 (297h0m59.941041117s) [noise=true , address=145.79.227.21:4497 , piece_id=SP6EKJB7W4QXF7OIBVKEFWCPOZFL5IRIEDVAGS2M4QJSO34CUVRQ, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.345025057 (297h0m59.941796084s) [noise=true , address=77.90.31.17:4793 , piece_id=2U4IBUA2GGUGDBXQGIGQKUK4ZR3L7CDKC6MTGIKAUME4F7MFZRLA, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.345115802 (297h0m59.941886836s) [noise=true , address=167.17.72.69:4731 , piece_id=VOEVIKKFJMC2Y3TPQIDMYUP37NLFU7KPRH4HRRSMQLWUKTSLOH2Q, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.345690600 (297h0m59.942461634s) [noise=true , address=77.90.31.13:4648 , piece_id=BPMB5CFOYHFT3BS4ORR6JF6BNFC2AOAJZOSPMG67NAMLGGP5XVEQ, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.346311592 (297h0m59.943082623s) [noise=true , address=79.127.205.234:4202 , piece_id=YY4LYWX26O6RIOBONESNKTUZEFZCRL4UUX3NEBQ53AJV4IEDG5MQ, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.346791142 (297h0m59.943562166s) [noise=true , address=45.91.116.10:4110 , piece_id=UYLMDCWJEJOHGICA37KNH7G5AQXNIPRH5NRYLBUUMGDRVSLTEYHQ, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.347094999 (297h0m59.94386607s) [noise=true , address=145.79.227.17:4987 , piece_id=PT3TWVB65JZFEWVH4IUZEU2N53FQ57QBRWVLWAUFP4UDLY3LWNLQ, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.347781281 (297h0m59.944552305s) [noise=true , address=79.127.219.46:4919 , piece_id=KK6HRIVBMYLIHRHGPQH3AWMNSVHTTIBESHDMJZA72T446RX6QKRQ, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.348263123 (297h0m59.945034137s) [noise=true , address=193.149.164.68:4033 , piece_id=JZPQKPYGO3W3RZZBITQ73X2XEPIBBSLGRMWIVOM4Z3GTV5UFKRTQ, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.348915505 (297h0m59.945686536s) [noise=true , address=145.79.227.7:4312 , piece_id=2FAL6E3RMBTBHZBXAJJVH2N5VGF5VYI3KNZXGNWCMWGBBPAUO2NQ, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.349366298 (297h0m59.946137337s) [noise=true , address=193.149.164.67:4007 , piece_id=HURQGH67I5B6V4YUV3PRZL3KSKN3XTSK3UMCHZRNT6L3W3PXXHPQ, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.429834098 (297h1m0.026605126s) [upload=332666]: Next returned nil segment Mar 18 20:01:02.429853411 (297h1m0.026624442s) [upload=332666]: Waiting for error group managing segments... Mar 18 20:01:02.446867680 (297h1m0.043638704s) [noise=true , address=45.91.116.10:4110 , piece_id=UYLMDCWJEJOHGICA37KNH7G5AQXNIPRH5NRYLBUUMGDRVSLTEYHQ, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil> Mar 18 20:01:02.451665165 (297h1m0.048436196s) [noise=true , address=45.91.116.4:4961 , piece_id=ZMUGQECUVPVHDNKN75XPBET5M2HFXXK4WMTP4V5KEK63JVZCX5EA, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil> Mar 18 20:01:02.452538886 (297h1m0.049309913s) [noise=true , address=77.90.31.17:4793 , piece_id=2U4IBUA2GGUGDBXQGIGQKUK4ZR3L7CDKC6MTGIKAUME4F7MFZRLA, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil> Mar 18 20:01:02.455597040 (297h1m0.052368067s) [noise=true , address=45.91.116.8:4061 , piece_id=6XFVKT4AAO7QN457OQMRKPI6R24TJTAN47UXLEVOEFQVOS6M6C3Q, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil> Mar 18 20:01:02.457848739 (297h1m0.054619766s) [noise=true , address=77.90.31.13:4648 , piece_id=BPMB5CFOYHFT3BS4ORR6JF6BNFC2AOAJZOSPMG67NAMLGGP5XVEQ, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil> Mar 18 20:01:02.461479709 (297h1m0.058250746s) [noise=true , address=79.127.219.46:4919 , piece_id=KK6HRIVBMYLIHRHGPQH3AWMNSVHTTIBESHDMJZA72T446RX6QKRQ, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil> Mar 18 20:01:02.676859466 (297h1m0.27363049s) [noise=true , address=167.17.72.69:4731 , piece_id=VOEVIKKFJMC2Y3TPQIDMYUP37NLFU7KPRH4HRRSMQLWUKTSLOH2Q, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil> Mar 18 20:01:02.909860743 (297h1m0.506631766s) [noise=true , address=79.127.205.234:4202 , piece_id=YY4LYWX26O6RIOBONESNKTUZEFZCRL4UUX3NEBQ53AJV4IEDG5MQ, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil> Mar 18 20:01:02.930427742 (297h1m0.527198754s) [noise=true , address=193.149.164.71:4297 , piece_id=4IJPXLPCX3LKFOBOSRHK2DTNRKH67K7FXJUKQ3JPRWHS47ZFEIDA, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil> Mar 18 20:01:02.944696889 (297h1m0.541467914s) [noise=true , address=193.149.164.67:4007 , piece_id=HURQGH67I5B6V4YUV3PRZL3KSKN3XTSK3UMCHZRNT6L3W3PXXHPQ, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil> Mar 18 20:01:02.951319183 (297h1m0.548090212s) [noise=true , address=193.149.164.68:4033 , piece_id=JZPQKPYGO3W3RZZBITQ73X2XEPIBBSLGRMWIVOM4Z3GTV5UFKRTQ, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil> Mar 18 20:01:02.975619436 (297h1m0.572390467s) [noise=true , address=79.127.205.233:4166 , piece_id=SFGK5YCWHFRVXF6QNHYP2CYJCZLSTQ76OYRVJPWWXQTF2KFSFJKA, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil> Mar 18 20:01:03.819918528 (297h1m1.416689564s) [noise=true , address=167.17.72.71:4815 , piece_id=CXGTYNNBILN3TT7I6Z7UI75F2ZCS4QUTLS6KC4FJ5IHS3XOYWDJQ, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil> Mar 18 20:01:59.307873420 (297h1m56.904644452s) [noise=true , address=145.79.227.7:4312 , piece_id=2FAL6E3RMBTBHZBXAJJVH2N5VGF5VYI3KNZXGNWCMWGBBPAUO2NQ, seg_pos={0 0}, upload=332666]: Done uploading piece. err: ecclient: upload failed (node:12AVp3qn5uwLPqYaZEwzcgoGMJhNga1BUh6bK5k2bxjWgZLQiiC, address:145.79.227.7:4312): protocol: rpc: dial tcp 145.79.227.7:4312: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:01:59.308026092 (297h1m56.904797124s) [noise=true , address=145.79.227.17:4987 , piece_id=PT3TWVB65JZFEWVH4IUZEU2N53FQ57QBRWVLWAUFP4UDLY3LWNLQ, seg_pos={0 0}, upload=332666]: Done uploading piece. err: ecclient: upload failed (node:1XsPVNGYUhKnXkQRfVfjyBRzyvhShSdsrnzXtwe4Xtf5MaHMvH, address:145.79.227.17:4987): protocol: rpc: dial tcp 145.79.227.17:4987: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:01:59.308195864 (297h1m56.904966894s) [noise=true , address=145.79.227.36:4524 , piece_id=3TAV3P55ITTHBAZEUUUTOAHJRGQXJXXVGZJP6AVR2ANVNOR3S54Q, seg_pos={0 0}, upload=332666]: Done uploading piece. err: ecclient: upload failed (node:1xfbCzjSwFDRrR1gPggeN8tNCLbRRWDVzCJ6mqhZsbVSe8d4s4, address:145.79.227.36:4524): protocol: rpc: dial tcp 145.79.227.36:4524: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:01:59.308316037 (297h1m56.90508706s) [noise=true , address=145.79.227.33:4892 , piece_id=DQ6ALC3VOAGFIJI3IKR6I7QFYM22C66QWAAXMH6NXI2FIQEZNQYA, seg_pos={0 0}, upload=332666]: Done uploading piece. err: ecclient: upload failed (node:12RHwFxcUjPsn12uHEAH3fBDJKv1v1tM8mRJXoFrZrG5djdswF5, address:145.79.227.33:4892): protocol: rpc: dial tcp 145.79.227.33:4892: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:01:59.308404403 (297h1m56.905175433s) [noise=true , address=145.79.227.21:4497 , piece_id=SP6EKJB7W4QXF7OIBVKEFWCPOZFL5IRIEDVAGS2M4QJSO34CUVRQ, seg_pos={0 0}, upload=332666]: Done uploading piece. err: ecclient: upload failed (node:12ZBe4we7VeFdthqRxyy1V9uQrA4r8ouNX68xqsk5bD26dkf2Gd, address:145.79.227.21:4497): protocol: context deadline exceeded; context deadline exceeded Mar 18 20:01:59.308437457 (297h1m56.90520847s) [seg_pos={0 0}, upload=332666]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 18 20:01:59.338288467 (297h1m56.93505949s) [upload=332666]: Done uploading segments... Mar 18 20:01:59.338335497 (297h1m56.935106523s) [upload=332666]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 13.312421492s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 49 out of 60 is failed (threshold=3) [....XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.XXXXXX...XX.X.XXX.] duration: 15.793078848s trace: https://jaeger.dev.storj.io/trace/196cc53575776cd7 FAIL (2026-03-18 20:16:05 UTC) operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Mar 18 20:14:37.682141670 (297h14m35.278912684s) [upload=332922]: Starting upload Mar 18 20:14:37.682232663 (297h14m35.279003686s) [upload=332922]: Done starting upload Mar 18 20:14:37.682326485 (297h14m35.279097498s) [upload=332922]: Uploading segments... Mar 18 20:14:37.682539898 (297h14m35.279310921s) [upload=332922]: Got next segment. Inline: false Mar 18 20:14:37.682578355 (297h14m35.279349378s) [upload=332922]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 20:14:37.682586290 (297h14m35.279357314s) [upload=332922]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 18 20:14:40.810771947 (297h14m38.40754298s) [seg_pos={0 0}, upload=332922]: Begin upload segment... Mar 18 20:14:40.810839156 (297h14m38.4076102s) [seg_pos={0 0}, upload=332922]: Done begin upload segment. Mar 18 20:14:40.810872567 (297h14m38.407643598s) [noise=true , address=77.90.31.16:4738 , piece_id=I3V7ZDX6CUB7RKHT3YABX3JSZLNGPE4NQGR2WEO6GCBVWDKMPG6A, seg_pos={0 0}, upload=332922]: Uploading piece... Mar 18 20:14:40.811051185 (297h14m38.40782221s) [noise=true , address=77.90.31.11:4557 , piece_id=IC2XFF76ULCQNPFSZGR52FTUSKG2EPFGSFAQM5MJ45SF6HQ5SAUQ, seg_pos={0 0}, upload=332922]: Uploading piece... Mar 18 20:14:40.811659639 (297h14m38.408430663s) [noise=true , address=45.91.116.5:4298 , piece_id=P4LWCFWYS7UJ2ECSGTM2FIGRMRFGQ2XP3CK3NRR6ZCTCQ4EGAZLA, seg_pos={0 0}, upload=332922]: Uploading piece... Mar 18 20:14:40.812561990 (297h14m38.409333019s) [noise=true , address=145.79.227.30:4767 , piece_id=HAE7DXSSFY4ORPUMW7M54N6YDYW4IK7O345LVO6YEBWCYEHCPTEQ, seg_pos={0 0}, upload=332922]: Uploading piece... Mar 18 20:14:40.812984335 (297h14m38.409755367s) [noise=true , address=45.91.116.14:4189 , piece_id=UDEZIVP3LWCJRLODJQBS652T6MGCK7GGTJ5YOHX6YOZFZA2TC7AQ, seg_pos={0 0}, upload=332922]: Uploading piece... Mar 18 20:14:40.814255674 (297h14m38.411026707s) [noise=true , address=77.90.31.13:4625 , piece_id=7NXAH5JA23PHLCL3MKIYSDCK324QWOEIYJTLL6BV7AKNPUKCPU6Q, seg_pos={0 0}, upload=332922]: Uploading piece... Mar 18 20:14:40.815750654 (297h14m38.412521679s) [noise=true , address=45.91.116.6:4973 , piece_id=M4IKW6YVKVOBHD5HZLV3OWZHD22E4VUA6YBZ6LMFJ5ME5X7UG4TQ, seg_pos={0 0}, upload=332922]: Uploading piece... Mar 18 20:14:40.816806505 (297h14m38.413577531s) [noise=true , address=45.91.116.9:4295 , piece_id=5VSBPT3IOZJWXMLFEH5NFLM27ZDGCOGZQ2ETWO3PLJS2KB52I6LQ, seg_pos={0 0}, upload=332922]: Uploading piece... Mar 18 20:14:40.818259221 (297h14m38.415030254s) [noise=true , address=145.79.227.28:4731 , piece_id=AR4DTQSZ442VSQK6B7UHU2P7EIK2C3HD432V3DZDE7PQPVWMP2ZA, seg_pos={0 0}, upload=332922]: Uploading piece... Mar 18 20:14:40.818995861 (297h14m38.415766887s) [noise=true , address=77.90.31.21:4878 , piece_id=5VCEI4AMXXANPGI3PB5YYSNB2Z72T2KOYYZ7I5ISZJOKO6AED3YQ, seg_pos={0 0}, upload=332922]: Uploading piece... Mar 18 20:14:40.820987229 (297h14m38.417758258s) [noise=true , address=145.79.227.34:4905 , piece_id=AOB3KDKH3YNAHL46SF4U2S53WVXK2B2B2UHUMNQICO65MS5WR3DA, seg_pos={0 0}, upload=332922]: Uploading piece... Mar 18 20:14:40.824407995 (297h14m38.421179027s) [noise=true , address=77.90.31.18:4810 , piece_id=ZKWBR5FNF7ZLVQSXX24VZ3JRHIBJUCHWIND4NE5MB2Y5WNAGPJJA, seg_pos={0 0}, upload=332922]: Uploading piece... Mar 18 20:14:40.826539622 (297h14m38.423310651s) [noise=true , address=79.127.219.45:4855 , piece_id=INCHPU33BNXQFCLGDKBOTDMXXMI5VQVSEQYVPEIDECOSULQ2ZFHQ, seg_pos={0 0}, upload=332922]: Uploading piece... Mar 18 20:14:40.826822640 (297h14m38.42359367s) [noise=true , address=79.127.219.38:4821 , piece_id=H4NDRDOCW4SJXA6UKYGBY55FVBHGGP2C33CWFMLTFB4BDL2AYQPQ, seg_pos={0 0}, upload=332922]: Uploading piece... Mar 18 20:14:40.827521030 (297h14m38.424292047s) [noise=true , address=145.79.227.25:4622 , piece_id=5NLGX2LP2MN4PWT7GM43IMTN244QBBB63PNAPHF3FL5FPBQVMEXA, seg_pos={0 0}, upload=332922]: Uploading piece... Mar 18 20:14:40.837307059 (297h14m38.434078076s) [noise=true , address=77.90.31.15:4700 , piece_id=OOJTAUY7Q7DZDRCV5S5I6MLBMGTW4MW7MYWNCWN5BYUZ7S4JDEJA, seg_pos={0 0}, upload=332922]: Uploading piece... Mar 18 20:14:40.881565358 (297h14m38.478336375s) [upload=332922]: Next returned nil segment Mar 18 20:14:40.881579130 (297h14m38.478350156s) [upload=332922]: Waiting for error group managing segments... Mar 18 20:14:40.886308132 (297h14m38.48307916s) [noise=true , address=45.91.116.9:4295 , piece_id=5VSBPT3IOZJWXMLFEH5NFLM27ZDGCOGZQ2ETWO3PLJS2KB52I6LQ, seg_pos={0 0}, upload=332922]: Done uploading piece. err: <nil> Mar 18 20:14:40.887752042 (297h14m38.484523059s) [noise=true , address=77.90.31.11:4557 , piece_id=IC2XFF76ULCQNPFSZGR52FTUSKG2EPFGSFAQM5MJ45SF6HQ5SAUQ, seg_pos={0 0}, upload=332922]: Done uploading piece. err: <nil> Mar 18 20:14:40.888000047 (297h14m38.484771079s) [noise=true , address=77.90.31.16:4738 , piece_id=I3V7ZDX6CUB7RKHT3YABX3JSZLNGPE4NQGR2WEO6GCBVWDKMPG6A, seg_pos={0 0}, upload=332922]: Done uploading piece. err: <nil> Mar 18 20:14:40.888323598 (297h14m38.48509461s) [noise=true , address=77.90.31.18:4810 , piece_id=ZKWBR5FNF7ZLVQSXX24VZ3JRHIBJUCHWIND4NE5MB2Y5WNAGPJJA, seg_pos={0 0}, upload=332922]: Done uploading piece. err: <nil> Mar 18 20:14:40.888656423 (297h14m38.485427447s) [noise=true , address=79.127.219.45:4855 , piece_id=INCHPU33BNXQFCLGDKBOTDMXXMI5VQVSEQYVPEIDECOSULQ2ZFHQ, seg_pos={0 0}, upload=332922]: Done uploading piece. err: <nil> Mar 18 20:14:40.888787778 (297h14m38.485558791s) [noise=true , address=77.90.31.13:4625 , piece_id=7NXAH5JA23PHLCL3MKIYSDCK324QWOEIYJTLL6BV7AKNPUKCPU6Q, seg_pos={0 0}, upload=332922]: Done uploading piece. err: <nil> Mar 18 20:14:40.895250726 (297h14m38.492021754s) [noise=true , address=45.91.116.6:4973 , piece_id=M4IKW6YVKVOBHD5HZLV3OWZHD22E4VUA6YBZ6LMFJ5ME5X7UG4TQ, seg_pos={0 0}, upload=332922]: Done uploading piece. err: <nil> Mar 18 20:14:40.899563413 (297h14m38.496334489s) [noise=true , address=77.90.31.21:4878 , piece_id=5VCEI4AMXXANPGI3PB5YYSNB2Z72T2KOYYZ7I5ISZJOKO6AED3YQ, seg_pos={0 0}, upload=332922]: Done uploading piece. err: <nil> Mar 18 20:14:40.902386354 (297h14m38.499157367s) [noise=true , address=45.91.116.5:4298 , piece_id=P4LWCFWYS7UJ2ECSGTM2FIGRMRFGQ2XP3CK3NRR6ZCTCQ4EGAZLA, seg_pos={0 0}, upload=332922]: Done uploading piece. err: <nil> Mar 18 20:14:40.906240108 (297h14m38.503011124s) [noise=true , address=79.127.219.38:4821 , piece_id=H4NDRDOCW4SJXA6UKYGBY55FVBHGGP2C33CWFMLTFB4BDL2AYQPQ, seg_pos={0 0}, upload=332922]: Done uploading piece. err: <nil> Mar 18 20:14:40.906346440 (297h14m38.503117456s) [noise=true , address=45.91.116.14:4189 , piece_id=UDEZIVP3LWCJRLODJQBS652T6MGCK7GGTJ5YOHX6YOZFZA2TC7AQ, seg_pos={0 0}, upload=332922]: Done uploading piece. err: <nil> Mar 18 20:14:40.909342371 (297h14m38.506113384s) [noise=true , address=77.90.31.15:4700 , piece_id=OOJTAUY7Q7DZDRCV5S5I6MLBMGTW4MW7MYWNCWN5BYUZ7S4JDEJA, seg_pos={0 0}, upload=332922]: Done uploading piece. err: <nil> Mar 18 20:15:07.652259020 (297h15m5.249030055s) [noise=true , address=145.79.227.34:4905 , piece_id=AOB3KDKH3YNAHL46SF4U2S53WVXK2B2B2UHUMNQICO65MS5WR3DA, seg_pos={0 0}, upload=332922]: Done uploading piece. err: ecclient: upload failed (node:12ffKESNSVpXQzbwJinP6vvgE6zoF3RFz2NRUuuJHvd7bvk1gc9, address:145.79.227.34:4905): protocol: rpc: dial tcp 145.79.227.34:4905: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:15:07.653011591 (297h15m5.249782626s) [noise=true , address=145.79.227.30:4767 , piece_id=HAE7DXSSFY4ORPUMW7M54N6YDYW4IK7O345LVO6YEBWCYEHCPTEQ, seg_pos={0 0}, upload=332922]: Done uploading piece. err: ecclient: upload failed (node:12sfaYCUhNCAyQAPu1jac5dRWR35Qqc4ZcKg75mp7nVQak295ug, address:145.79.227.30:4767): protocol: rpc: dial tcp 145.79.227.30:4767: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:15:07.654192872 (297h15m5.250963907s) [noise=true , address=145.79.227.28:4731 , piece_id=AR4DTQSZ442VSQK6B7UHU2P7EIK2C3HD432V3DZDE7PQPVWMP2ZA, seg_pos={0 0}, upload=332922]: Done uploading piece. err: ecclient: upload failed (node:12NQzTwVYQa6ZzxAHQm1apkwWnqJstTnAJQyHKhBzC2jTN9kEVK, address:145.79.227.28:4731): protocol: rpc: dial tcp 145.79.227.28:4731: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:15:07.654423705 (297h15m5.251194739s) [noise=true , address=145.79.227.25:4622 , piece_id=5NLGX2LP2MN4PWT7GM43IMTN244QBBB63PNAPHF3FL5FPBQVMEXA, seg_pos={0 0}, upload=332922]: Done uploading piece. err: ecclient: upload failed (node:1xEK5XL3sYcip7PsQcZBQYihgnEZykdMmsKkCr2TUmbTFYncfs, address:145.79.227.25:4622): protocol: rpc: dial tcp 145.79.227.25:4622: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:15:07.654712416 (297h15m5.251483452s) [seg_pos={0 0}, upload=332922]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Mar 18 20:15:07.677696504 (297h15m5.274467527s) [upload=332922]: Done uploading segments... Mar 18 20:15:07.677737744 (297h15m5.27450877s) [upload=332922]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 15.793105016s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 56 out of 60 is failed (threshold=6) [XXXXX.XXX...XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 5.002286827s trace: https://jaeger.dev.storj.io/trace/4bd1b22e1c6dfffc FAIL (2026-03-18 20:15:52 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.002313575s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 6 out of 60 is failed (threshold=3) [............X..........XX..X....X........X..................] duration: 16.723322591s trace: https://jaeger.dev.storj.io/trace/589419efc52e5e2b FAIL (2026-03-18 20:15:39 UTC) operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== Mar 18 20:00:59.343084966 (297h0m56.939855989s) [upload=332666]: Starting upload Mar 18 20:00:59.343175518 (297h0m56.939946548s) [upload=332666]: Done starting upload Mar 18 20:00:59.343565585 (297h0m56.940336612s) [upload=332666]: Uploading segments... Mar 18 20:00:59.344256647 (297h0m56.941027676s) [upload=332666]: Got next segment. Inline: false Mar 18 20:00:59.344352215 (297h0m56.941123241s) [upload=332666]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 20:00:59.344358706 (297h0m56.941129732s) [upload=332666]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 18 20:01:02.340259299 (297h0m59.937030329s) [seg_pos={0 0}, upload=332666]: Begin upload segment... Mar 18 20:01:02.340368743 (297h0m59.937139767s) [seg_pos={0 0}, upload=332666]: Done begin upload segment. Mar 18 20:01:02.340479563 (297h0m59.93725059s) [noise=true , address=45.91.116.8:4061 , piece_id=6XFVKT4AAO7QN457OQMRKPI6R24TJTAN47UXLEVOEFQVOS6M6C3Q, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.340523837 (297h0m59.93729486s) [noise=true , address=79.127.205.233:4166 , piece_id=SFGK5YCWHFRVXF6QNHYP2CYJCZLSTQ76OYRVJPWWXQTF2KFSFJKA, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.341468438 (297h0m59.938239447s) [noise=true , address=45.91.116.4:4961 , piece_id=ZMUGQECUVPVHDNKN75XPBET5M2HFXXK4WMTP4V5KEK63JVZCX5EA, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.341520025 (297h0m59.938291058s) [noise=true , address=167.17.72.71:4815 , piece_id=CXGTYNNBILN3TT7I6Z7UI75F2ZCS4QUTLS6KC4FJ5IHS3XOYWDJQ, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.342232992 (297h0m59.939004023s) [noise=true , address=145.79.227.36:4524 , piece_id=3TAV3P55ITTHBAZEUUUTOAHJRGQXJXXVGZJP6AVR2ANVNOR3S54Q, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.342917368 (297h0m59.939688398s) [noise=true , address=193.149.164.71:4297 , piece_id=4IJPXLPCX3LKFOBOSRHK2DTNRKH67K7FXJUKQ3JPRWHS47ZFEIDA, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.343244957 (297h0m59.940015982s) [noise=true , address=145.79.227.33:4892 , piece_id=DQ6ALC3VOAGFIJI3IKR6I7QFYM22C66QWAAXMH6NXI2FIQEZNQYA, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.344270079 (297h0m59.941041117s) [noise=true , address=145.79.227.21:4497 , piece_id=SP6EKJB7W4QXF7OIBVKEFWCPOZFL5IRIEDVAGS2M4QJSO34CUVRQ, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.345025057 (297h0m59.941796084s) [noise=true , address=77.90.31.17:4793 , piece_id=2U4IBUA2GGUGDBXQGIGQKUK4ZR3L7CDKC6MTGIKAUME4F7MFZRLA, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.345115802 (297h0m59.941886836s) [noise=true , address=167.17.72.69:4731 , piece_id=VOEVIKKFJMC2Y3TPQIDMYUP37NLFU7KPRH4HRRSMQLWUKTSLOH2Q, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.345690600 (297h0m59.942461634s) [noise=true , address=77.90.31.13:4648 , piece_id=BPMB5CFOYHFT3BS4ORR6JF6BNFC2AOAJZOSPMG67NAMLGGP5XVEQ, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.346311592 (297h0m59.943082623s) [noise=true , address=79.127.205.234:4202 , piece_id=YY4LYWX26O6RIOBONESNKTUZEFZCRL4UUX3NEBQ53AJV4IEDG5MQ, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.346791142 (297h0m59.943562166s) [noise=true , address=45.91.116.10:4110 , piece_id=UYLMDCWJEJOHGICA37KNH7G5AQXNIPRH5NRYLBUUMGDRVSLTEYHQ, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.347094999 (297h0m59.94386607s) [noise=true , address=145.79.227.17:4987 , piece_id=PT3TWVB65JZFEWVH4IUZEU2N53FQ57QBRWVLWAUFP4UDLY3LWNLQ, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.347781281 (297h0m59.944552305s) [noise=true , address=79.127.219.46:4919 , piece_id=KK6HRIVBMYLIHRHGPQH3AWMNSVHTTIBESHDMJZA72T446RX6QKRQ, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.348263123 (297h0m59.945034137s) [noise=true , address=193.149.164.68:4033 , piece_id=JZPQKPYGO3W3RZZBITQ73X2XEPIBBSLGRMWIVOM4Z3GTV5UFKRTQ, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.348915505 (297h0m59.945686536s) [noise=true , address=145.79.227.7:4312 , piece_id=2FAL6E3RMBTBHZBXAJJVH2N5VGF5VYI3KNZXGNWCMWGBBPAUO2NQ, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.349366298 (297h0m59.946137337s) [noise=true , address=193.149.164.67:4007 , piece_id=HURQGH67I5B6V4YUV3PRZL3KSKN3XTSK3UMCHZRNT6L3W3PXXHPQ, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.429834098 (297h1m0.026605126s) [upload=332666]: Next returned nil segment Mar 18 20:01:02.429853411 (297h1m0.026624442s) [upload=332666]: Waiting for error group managing segments... Mar 18 20:01:02.446867680 (297h1m0.043638704s) [noise=true , address=45.91.116.10:4110 , piece_id=UYLMDCWJEJOHGICA37KNH7G5AQXNIPRH5NRYLBUUMGDRVSLTEYHQ, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil> Mar 18 20:01:02.451665165 (297h1m0.048436196s) [noise=true , address=45.91.116.4:4961 , piece_id=ZMUGQECUVPVHDNKN75XPBET5M2HFXXK4WMTP4V5KEK63JVZCX5EA, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil> Mar 18 20:01:02.452538886 (297h1m0.049309913s) [noise=true , address=77.90.31.17:4793 , piece_id=2U4IBUA2GGUGDBXQGIGQKUK4ZR3L7CDKC6MTGIKAUME4F7MFZRLA, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil> Mar 18 20:01:02.455597040 (297h1m0.052368067s) [noise=true , address=45.91.116.8:4061 , piece_id=6XFVKT4AAO7QN457OQMRKPI6R24TJTAN47UXLEVOEFQVOS6M6C3Q, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil> Mar 18 20:01:02.457848739 (297h1m0.054619766s) [noise=true , address=77.90.31.13:4648 , piece_id=BPMB5CFOYHFT3BS4ORR6JF6BNFC2AOAJZOSPMG67NAMLGGP5XVEQ, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil> Mar 18 20:01:02.461479709 (297h1m0.058250746s) [noise=true , address=79.127.219.46:4919 , piece_id=KK6HRIVBMYLIHRHGPQH3AWMNSVHTTIBESHDMJZA72T446RX6QKRQ, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil> Mar 18 20:01:02.676859466 (297h1m0.27363049s) [noise=true , address=167.17.72.69:4731 , piece_id=VOEVIKKFJMC2Y3TPQIDMYUP37NLFU7KPRH4HRRSMQLWUKTSLOH2Q, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil> Mar 18 20:01:02.909860743 (297h1m0.506631766s) [noise=true , address=79.127.205.234:4202 , piece_id=YY4LYWX26O6RIOBONESNKTUZEFZCRL4UUX3NEBQ53AJV4IEDG5MQ, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil> Mar 18 20:01:02.930427742 (297h1m0.527198754s) [noise=true , address=193.149.164.71:4297 , piece_id=4IJPXLPCX3LKFOBOSRHK2DTNRKH67K7FXJUKQ3JPRWHS47ZFEIDA, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil> Mar 18 20:01:02.944696889 (297h1m0.541467914s) [noise=true , address=193.149.164.67:4007 , piece_id=HURQGH67I5B6V4YUV3PRZL3KSKN3XTSK3UMCHZRNT6L3W3PXXHPQ, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil> Mar 18 20:01:02.951319183 (297h1m0.548090212s) [noise=true , address=193.149.164.68:4033 , piece_id=JZPQKPYGO3W3RZZBITQ73X2XEPIBBSLGRMWIVOM4Z3GTV5UFKRTQ, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil> Mar 18 20:01:02.975619436 (297h1m0.572390467s) [noise=true , address=79.127.205.233:4166 , piece_id=SFGK5YCWHFRVXF6QNHYP2CYJCZLSTQ76OYRVJPWWXQTF2KFSFJKA, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil> Mar 18 20:01:03.819918528 (297h1m1.416689564s) [noise=true , address=167.17.72.71:4815 , piece_id=CXGTYNNBILN3TT7I6Z7UI75F2ZCS4QUTLS6KC4FJ5IHS3XOYWDJQ, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil> Mar 18 20:01:59.307873420 (297h1m56.904644452s) [noise=true , address=145.79.227.7:4312 , piece_id=2FAL6E3RMBTBHZBXAJJVH2N5VGF5VYI3KNZXGNWCMWGBBPAUO2NQ, seg_pos={0 0}, upload=332666]: Done uploading piece. err: ecclient: upload failed (node:12AVp3qn5uwLPqYaZEwzcgoGMJhNga1BUh6bK5k2bxjWgZLQiiC, address:145.79.227.7:4312): protocol: rpc: dial tcp 145.79.227.7:4312: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:01:59.308026092 (297h1m56.904797124s) [noise=true , address=145.79.227.17:4987 , piece_id=PT3TWVB65JZFEWVH4IUZEU2N53FQ57QBRWVLWAUFP4UDLY3LWNLQ, seg_pos={0 0}, upload=332666]: Done uploading piece. err: ecclient: upload failed (node:1XsPVNGYUhKnXkQRfVfjyBRzyvhShSdsrnzXtwe4Xtf5MaHMvH, address:145.79.227.17:4987): protocol: rpc: dial tcp 145.79.227.17:4987: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:01:59.308195864 (297h1m56.904966894s) [noise=true , address=145.79.227.36:4524 , piece_id=3TAV3P55ITTHBAZEUUUTOAHJRGQXJXXVGZJP6AVR2ANVNOR3S54Q, seg_pos={0 0}, upload=332666]: Done uploading piece. err: ecclient: upload failed (node:1xfbCzjSwFDRrR1gPggeN8tNCLbRRWDVzCJ6mqhZsbVSe8d4s4, address:145.79.227.36:4524): protocol: rpc: dial tcp 145.79.227.36:4524: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:01:59.308316037 (297h1m56.90508706s) [noise=true , address=145.79.227.33:4892 , piece_id=DQ6ALC3VOAGFIJI3IKR6I7QFYM22C66QWAAXMH6NXI2FIQEZNQYA, seg_pos={0 0}, upload=332666]: Done uploading piece. err: ecclient: upload failed (node:12RHwFxcUjPsn12uHEAH3fBDJKv1v1tM8mRJXoFrZrG5djdswF5, address:145.79.227.33:4892): protocol: rpc: dial tcp 145.79.227.33:4892: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:01:59.308404403 (297h1m56.905175433s) [noise=true , address=145.79.227.21:4497 , piece_id=SP6EKJB7W4QXF7OIBVKEFWCPOZFL5IRIEDVAGS2M4QJSO34CUVRQ, seg_pos={0 0}, upload=332666]: Done uploading piece. err: ecclient: upload failed (node:12ZBe4we7VeFdthqRxyy1V9uQrA4r8ouNX68xqsk5bD26dkf2Gd, address:145.79.227.21:4497): protocol: context deadline exceeded; context deadline exceeded Mar 18 20:01:59.308437457 (297h1m56.90520847s) [seg_pos={0 0}, upload=332666]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 18 20:01:59.338288467 (297h1m56.93505949s) [upload=332666]: Done uploading segments... Mar 18 20:01:59.338335497 (297h1m56.935106523s) [upload=332666]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 16.723350101s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 9 out of 60 is failed (threshold=3) [..........XXX..........X...X.........X....X....X.......X....] duration: 17.284222024s trace: https://jaeger.dev.storj.io/trace/24517a7cd40ab76d FAIL (2026-03-18 20:15:36 UTC) operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== oad Mar 18 20:12:28.252047067 (297h12m25.848818092s) [upload=332883]: Uploading segments... Mar 18 20:12:28.252323178 (297h12m25.84909421s) [upload=332883]: Got next segment. Inline: false Mar 18 20:12:28.252380536 (297h12m25.849151569s) [upload=332883]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 20:12:28.252388587 (297h12m25.849159622s) [upload=332883]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 18 20:12:31.413020979 (297h12m29.009792004s) [seg_pos={0 0}, upload=332883]: Begin upload segment... Mar 18 20:12:31.413109194 (297h12m29.009880217s) [seg_pos={0 0}, upload=332883]: Done begin upload segment. Mar 18 20:12:31.413182895 (297h12m29.009953924s) [noise=true , address=167.17.72.68:4653 , piece_id=6TO53KV4DBJ5HTWIQUZY2L6WVMB4PBPRWSZD6WHBLVFI5FUULYUQ, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.413918965 (297h12m29.010689999s) [noise=true , address=145.79.227.20:4454 , piece_id=F2WUAPOH2J3GQ7DMWLBJA44UGY5B3MMDS2U5ITXTOIPR4ELY5QYA, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.414450842 (297h12m29.011221869s) [noise=true , address=193.149.164.70:4111 , piece_id=H6GX5VVLRFNUAF2R7FRZ5GFQCZ2F57OL4GXIHWQJGZDOM3ST5DKQ, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.414901508 (297h12m29.011672532s) [noise=true , address=5.231.73.4:4002 , piece_id=3DWXH2Y6ZZNDHRFMJFL6EEMJN7WNDRKJ3ARFKN6UAGAHICNCBSDA, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.415342842 (297h12m29.012113865s) [noise=true , address=145.79.227.28:4750 , piece_id=KJVCFR2SJ5HAQKJUJP4CYPVCZQC6PBAQII6UGNYVKRODWRKENFQA, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.415582477 (297h12m29.01235351s) [noise=true , address=145.79.227.24:4615 , piece_id=ZSEUUMSYRSXH4AJ3CDZNFKXBZFT63AKVDFBFJ2RLAZK344GQ5AFA, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.415917210 (297h12m29.012688235s) [noise=true , address=193.149.164.68:4040 , piece_id=4NT224HIHI45JJNYH3ZCXQKDAMQHHYWY7SZPYQK5KN7JKNYFGI2A, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.416035983 (297h12m29.012807012s) [noise=true , address=193.149.164.67:4030 , piece_id=PYGFDMQLZ2SYL5NTWUGQGHWW52T2YYEGDKP5SUWQQUTVJSJZOO5A, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.416379117 (297h12m29.013150149s) [noise=true , address=45.91.116.5:4322 , piece_id=FN763EWEYGTBEXNHOT3FMKLW52DENMGPRBPBJJHNWQUNQ5JIN3XA, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.416442412 (297h12m29.013213436s) [noise=true , address=145.79.227.23:4564 , piece_id=3S5IXK7TKV2W3I635W2LQDH3PC6JRPHIVBK347CE7EYE4QQ5AGBQ, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.416786035 (297h12m29.013557058s) [noise=true , address=167.17.72.71:4808 , piece_id=JL5TJCWL454VA73PMWKZYBAUT6ZMRPEH55KJDSHEELC37HRVCO7Q, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.416946153 (297h12m29.013717177s) [noise=true , address=64.49.13.169:4705 , piece_id=FW7ICJDZJVOQR77HCGAWU7NKSBTWASW3BXH72WBGZHOFVPXA7XQQ, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.417084028 (297h12m29.013855051s) [noise=true , address=145.79.227.44:4202 , piece_id=UI7I4UPDOZCJ2MRPKV6FP6YPYNVSR5JWYDVFBO2TRFYQJWVWIUNQ, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.424729560 (297h12m29.021500589s) [noise=true , address=79.127.205.233:4156 , piece_id=5OZYLGXB7R6VMYGATW7GJD2CNAVO45PATTCFGNPWYZ3ZURYLZSQA, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.425355484 (297h12m29.022126513s) [noise=true , address=193.149.164.69:4077 , piece_id=GYZMBIAEBL4KBCSFA4H5JLWEB36NBN2BGVQ67A5G4ECUWY5BACXA, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.429910441 (297h12m29.026681471s) [noise=true , address=79.127.219.46:4918 , piece_id=WC2RL2PMO4EVATTOPZBWFHOZE6CBGQ2ROGJ44PDQUPQKYXBFKY6Q, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.431104280 (297h12m29.027875311s) [noise=true , address=79.127.219.45:4883 , piece_id=7DXMZGFUW2QAI2OID7WSSMJG6B3WW52HCMRWX2LS6YFEWAFMZZNQ, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.432173523 (297h12m29.02894455s) [noise=true , address=193.149.164.71:4300 , piece_id=4FI5PVS627OOB2M45U4QTOXGBJHNOZMUDDCFAW57W7PBAAQSJC4A, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.478900188 (297h12m29.075671211s) [upload=332883]: Next returned nil segment Mar 18 20:12:31.478914031 (297h12m29.075685054s) [upload=332883]: Waiting for error group managing segments... Mar 18 20:12:31.512891809 (297h12m29.109662832s) [noise=true , address=79.127.219.46:4918 , piece_id=WC2RL2PMO4EVATTOPZBWFHOZE6CBGQ2ROGJ44PDQUPQKYXBFKY6Q, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil> Mar 18 20:12:31.529013012 (297h12m29.125784038s) [noise=true , address=45.91.116.5:4322 , piece_id=FN763EWEYGTBEXNHOT3FMKLW52DENMGPRBPBJJHNWQUNQ5JIN3XA, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil> Mar 18 20:12:31.564707355 (297h12m29.161478379s) [noise=true , address=79.127.219.45:4883 , piece_id=7DXMZGFUW2QAI2OID7WSSMJG6B3WW52HCMRWX2LS6YFEWAFMZZNQ, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil> Mar 18 20:12:31.611013762 (297h12m29.207784793s) [noise=true , address=5.231.73.4:4002 , piece_id=3DWXH2Y6ZZNDHRFMJFL6EEMJN7WNDRKJ3ARFKN6UAGAHICNCBSDA, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil> Mar 18 20:12:31.713823345 (297h12m29.310594381s) [noise=true , address=167.17.72.68:4653 , piece_id=6TO53KV4DBJ5HTWIQUZY2L6WVMB4PBPRWSZD6WHBLVFI5FUULYUQ, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil> Mar 18 20:12:31.997071787 (297h12m29.59384282s) [noise=true , address=79.127.205.233:4156 , piece_id=5OZYLGXB7R6VMYGATW7GJD2CNAVO45PATTCFGNPWYZ3ZURYLZSQA, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil> Mar 18 20:12:31.999264198 (297h12m29.59603524s) [noise=true , address=193.149.164.70:4111 , piece_id=H6GX5VVLRFNUAF2R7FRZ5GFQCZ2F57OL4GXIHWQJGZDOM3ST5DKQ, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil> Mar 18 20:12:32.008593285 (297h12m29.605364328s) [noise=true , address=193.149.164.71:4300 , piece_id=4FI5PVS627OOB2M45U4QTOXGBJHNOZMUDDCFAW57W7PBAAQSJC4A, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil> Mar 18 20:12:32.011203033 (297h12m29.607974056s) [noise=true , address=193.149.164.68:4040 , piece_id=4NT224HIHI45JJNYH3ZCXQKDAMQHHYWY7SZPYQK5KN7JKNYFGI2A, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil> Mar 18 20:12:32.038310240 (297h12m29.635081269s) [noise=true , address=193.149.164.67:4030 , piece_id=PYGFDMQLZ2SYL5NTWUGQGHWW52T2YYEGDKP5SUWQQUTVJSJZOO5A, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil> Mar 18 20:12:32.091831027 (297h12m29.68860206s) [noise=true , address=64.49.13.169:4705 , piece_id=FW7ICJDZJVOQR77HCGAWU7NKSBTWASW3BXH72WBGZHOFVPXA7XQQ, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil> Mar 18 20:12:32.408908738 (297h12m30.005679764s) [noise=true , address=193.149.164.69:4077 , piece_id=GYZMBIAEBL4KBCSFA4H5JLWEB36NBN2BGVQ67A5G4ECUWY5BACXA, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil> Mar 18 20:12:32.489564296 (297h12m30.086335307s) [noise=true , address=167.17.72.71:4808 , piece_id=JL5TJCWL454VA73PMWKZYBAUT6ZMRPEH55KJDSHEELC37HRVCO7Q, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil> Mar 18 20:13:28.204470030 (297h13m25.801241064s) [noise=true , address=145.79.227.23:4564 , piece_id=3S5IXK7TKV2W3I635W2LQDH3PC6JRPHIVBK347CE7EYE4QQ5AGBQ, seg_pos={0 0}, upload=332883]: Done uploading piece. err: ecclient: upload failed (node:121WW4uBQe8bGQKkasQQKT3k4EAdzncsG1ebNvgKf8S5SyNttCc, address:145.79.227.23:4564): protocol: rpc: dial tcp 145.79.227.23:4564: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:13:28.205066456 (297h13m25.801837482s) [noise=true , address=145.79.227.28:4750 , piece_id=KJVCFR2SJ5HAQKJUJP4CYPVCZQC6PBAQII6UGNYVKRODWRKENFQA, seg_pos={0 0}, upload=332883]: Done uploading piece. err: ecclient: upload failed (node:12UP8PxqeK7t24PXjopH1nziHYQFnsxhkamSHz8dmwcbuDxkAF6, address:145.79.227.28:4750): protocol: rpc: dial tcp 145.79.227.28:4750: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:13:28.205099998 (297h13m25.80187103s) [noise=true , address=145.79.227.44:4202 , piece_id=UI7I4UPDOZCJ2MRPKV6FP6YPYNVSR5JWYDVFBO2TRFYQJWVWIUNQ, seg_pos={0 0}, upload=332883]: Done uploading piece. err: ecclient: upload failed (node:121ogdaiVGybXEVsS8LDJuJRYXGxb1LCNnGPuwbde96eSEzkTW4, address:145.79.227.44:4202): protocol: rpc: dial tcp 145.79.227.44:4202: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:13:28.205340404 (297h13m25.802111436s) [noise=true , address=145.79.227.24:4615 , piece_id=ZSEUUMSYRSXH4AJ3CDZNFKXBZFT63AKVDFBFJ2RLAZK344GQ5AFA, seg_pos={0 0}, upload=332883]: Done uploading piece. err: ecclient: upload failed (node:12NJ5w8pAWCudDA9Xs4ES7b2evnbscesd682wJDh2gMMZ3Jf2zc, address:145.79.227.24:4615): protocol: rpc: dial tcp 145.79.227.24:4615: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:13:28.205593210 (297h13m25.802364235s) [noise=true , address=145.79.227.20:4454 , piece_id=F2WUAPOH2J3GQ7DMWLBJA44UGY5B3MMDS2U5ITXTOIPR4ELY5QYA, seg_pos={0 0}, upload=332883]: Done uploading piece. err: ecclient: upload failed (node:1SafkJPqKHLwMaqyZdLq3yJ1xdjqDtWHFrUwynmegUVwEC7y5J, address:145.79.227.20:4454): protocol: context deadline exceeded; context deadline exceeded Mar 18 20:13:28.205633541 (297h13m25.802404553s) [seg_pos={0 0}, upload=332883]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 18 20:13:28.228774364 (297h13m25.825545388s) [upload=332883]: Done uploading segments... Mar 18 20:13:28.228821883 (297h13m25.825592906s) [upload=332883]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 17.284258595s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 8 out of 60 is failed (threshold=3) [........XXXXXX.XX...........................................] duration: 13.891322054s trace: https://jaeger.dev.storj.io/trace/4b4d10a8051ad01b FAIL (2026-03-18 20:15:26 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 13.891348065s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 6 out of 60 is failed (threshold=3) [.............X..........XX..X....X........X.................] duration: 16.083471396s trace: https://jaeger.dev.storj.io/trace/589419efc52e5e2b FAIL (2026-03-18 20:15:17 UTC) operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== Mar 18 20:00:59.343084966 (297h0m56.939855989s) [upload=332666]: Starting upload Mar 18 20:00:59.343175518 (297h0m56.939946548s) [upload=332666]: Done starting upload Mar 18 20:00:59.343565585 (297h0m56.940336612s) [upload=332666]: Uploading segments... Mar 18 20:00:59.344256647 (297h0m56.941027676s) [upload=332666]: Got next segment. Inline: false Mar 18 20:00:59.344352215 (297h0m56.941123241s) [upload=332666]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 20:00:59.344358706 (297h0m56.941129732s) [upload=332666]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 18 20:01:02.340259299 (297h0m59.937030329s) [seg_pos={0 0}, upload=332666]: Begin upload segment... Mar 18 20:01:02.340368743 (297h0m59.937139767s) [seg_pos={0 0}, upload=332666]: Done begin upload segment. Mar 18 20:01:02.340479563 (297h0m59.93725059s) [noise=true , address=45.91.116.8:4061 , piece_id=6XFVKT4AAO7QN457OQMRKPI6R24TJTAN47UXLEVOEFQVOS6M6C3Q, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.340523837 (297h0m59.93729486s) [noise=true , address=79.127.205.233:4166 , piece_id=SFGK5YCWHFRVXF6QNHYP2CYJCZLSTQ76OYRVJPWWXQTF2KFSFJKA, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.341468438 (297h0m59.938239447s) [noise=true , address=45.91.116.4:4961 , piece_id=ZMUGQECUVPVHDNKN75XPBET5M2HFXXK4WMTP4V5KEK63JVZCX5EA, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.341520025 (297h0m59.938291058s) [noise=true , address=167.17.72.71:4815 , piece_id=CXGTYNNBILN3TT7I6Z7UI75F2ZCS4QUTLS6KC4FJ5IHS3XOYWDJQ, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.342232992 (297h0m59.939004023s) [noise=true , address=145.79.227.36:4524 , piece_id=3TAV3P55ITTHBAZEUUUTOAHJRGQXJXXVGZJP6AVR2ANVNOR3S54Q, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.342917368 (297h0m59.939688398s) [noise=true , address=193.149.164.71:4297 , piece_id=4IJPXLPCX3LKFOBOSRHK2DTNRKH67K7FXJUKQ3JPRWHS47ZFEIDA, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.343244957 (297h0m59.940015982s) [noise=true , address=145.79.227.33:4892 , piece_id=DQ6ALC3VOAGFIJI3IKR6I7QFYM22C66QWAAXMH6NXI2FIQEZNQYA, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.344270079 (297h0m59.941041117s) [noise=true , address=145.79.227.21:4497 , piece_id=SP6EKJB7W4QXF7OIBVKEFWCPOZFL5IRIEDVAGS2M4QJSO34CUVRQ, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.345025057 (297h0m59.941796084s) [noise=true , address=77.90.31.17:4793 , piece_id=2U4IBUA2GGUGDBXQGIGQKUK4ZR3L7CDKC6MTGIKAUME4F7MFZRLA, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.345115802 (297h0m59.941886836s) [noise=true , address=167.17.72.69:4731 , piece_id=VOEVIKKFJMC2Y3TPQIDMYUP37NLFU7KPRH4HRRSMQLWUKTSLOH2Q, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.345690600 (297h0m59.942461634s) [noise=true , address=77.90.31.13:4648 , piece_id=BPMB5CFOYHFT3BS4ORR6JF6BNFC2AOAJZOSPMG67NAMLGGP5XVEQ, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.346311592 (297h0m59.943082623s) [noise=true , address=79.127.205.234:4202 , piece_id=YY4LYWX26O6RIOBONESNKTUZEFZCRL4UUX3NEBQ53AJV4IEDG5MQ, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.346791142 (297h0m59.943562166s) [noise=true , address=45.91.116.10:4110 , piece_id=UYLMDCWJEJOHGICA37KNH7G5AQXNIPRH5NRYLBUUMGDRVSLTEYHQ, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.347094999 (297h0m59.94386607s) [noise=true , address=145.79.227.17:4987 , piece_id=PT3TWVB65JZFEWVH4IUZEU2N53FQ57QBRWVLWAUFP4UDLY3LWNLQ, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.347781281 (297h0m59.944552305s) [noise=true , address=79.127.219.46:4919 , piece_id=KK6HRIVBMYLIHRHGPQH3AWMNSVHTTIBESHDMJZA72T446RX6QKRQ, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.348263123 (297h0m59.945034137s) [noise=true , address=193.149.164.68:4033 , piece_id=JZPQKPYGO3W3RZZBITQ73X2XEPIBBSLGRMWIVOM4Z3GTV5UFKRTQ, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.348915505 (297h0m59.945686536s) [noise=true , address=145.79.227.7:4312 , piece_id=2FAL6E3RMBTBHZBXAJJVH2N5VGF5VYI3KNZXGNWCMWGBBPAUO2NQ, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.349366298 (297h0m59.946137337s) [noise=true , address=193.149.164.67:4007 , piece_id=HURQGH67I5B6V4YUV3PRZL3KSKN3XTSK3UMCHZRNT6L3W3PXXHPQ, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.429834098 (297h1m0.026605126s) [upload=332666]: Next returned nil segment Mar 18 20:01:02.429853411 (297h1m0.026624442s) [upload=332666]: Waiting for error group managing segments... Mar 18 20:01:02.446867680 (297h1m0.043638704s) [noise=true , address=45.91.116.10:4110 , piece_id=UYLMDCWJEJOHGICA37KNH7G5AQXNIPRH5NRYLBUUMGDRVSLTEYHQ, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil> Mar 18 20:01:02.451665165 (297h1m0.048436196s) [noise=true , address=45.91.116.4:4961 , piece_id=ZMUGQECUVPVHDNKN75XPBET5M2HFXXK4WMTP4V5KEK63JVZCX5EA, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil> Mar 18 20:01:02.452538886 (297h1m0.049309913s) [noise=true , address=77.90.31.17:4793 , piece_id=2U4IBUA2GGUGDBXQGIGQKUK4ZR3L7CDKC6MTGIKAUME4F7MFZRLA, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil> Mar 18 20:01:02.455597040 (297h1m0.052368067s) [noise=true , address=45.91.116.8:4061 , piece_id=6XFVKT4AAO7QN457OQMRKPI6R24TJTAN47UXLEVOEFQVOS6M6C3Q, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil> Mar 18 20:01:02.457848739 (297h1m0.054619766s) [noise=true , address=77.90.31.13:4648 , piece_id=BPMB5CFOYHFT3BS4ORR6JF6BNFC2AOAJZOSPMG67NAMLGGP5XVEQ, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil> Mar 18 20:01:02.461479709 (297h1m0.058250746s) [noise=true , address=79.127.219.46:4919 , piece_id=KK6HRIVBMYLIHRHGPQH3AWMNSVHTTIBESHDMJZA72T446RX6QKRQ, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil> Mar 18 20:01:02.676859466 (297h1m0.27363049s) [noise=true , address=167.17.72.69:4731 , piece_id=VOEVIKKFJMC2Y3TPQIDMYUP37NLFU7KPRH4HRRSMQLWUKTSLOH2Q, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil> Mar 18 20:01:02.909860743 (297h1m0.506631766s) [noise=true , address=79.127.205.234:4202 , piece_id=YY4LYWX26O6RIOBONESNKTUZEFZCRL4UUX3NEBQ53AJV4IEDG5MQ, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil> Mar 18 20:01:02.930427742 (297h1m0.527198754s) [noise=true , address=193.149.164.71:4297 , piece_id=4IJPXLPCX3LKFOBOSRHK2DTNRKH67K7FXJUKQ3JPRWHS47ZFEIDA, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil> Mar 18 20:01:02.944696889 (297h1m0.541467914s) [noise=true , address=193.149.164.67:4007 , piece_id=HURQGH67I5B6V4YUV3PRZL3KSKN3XTSK3UMCHZRNT6L3W3PXXHPQ, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil> Mar 18 20:01:02.951319183 (297h1m0.548090212s) [noise=true , address=193.149.164.68:4033 , piece_id=JZPQKPYGO3W3RZZBITQ73X2XEPIBBSLGRMWIVOM4Z3GTV5UFKRTQ, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil> Mar 18 20:01:02.975619436 (297h1m0.572390467s) [noise=true , address=79.127.205.233:4166 , piece_id=SFGK5YCWHFRVXF6QNHYP2CYJCZLSTQ76OYRVJPWWXQTF2KFSFJKA, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil> Mar 18 20:01:03.819918528 (297h1m1.416689564s) [noise=true , address=167.17.72.71:4815 , piece_id=CXGTYNNBILN3TT7I6Z7UI75F2ZCS4QUTLS6KC4FJ5IHS3XOYWDJQ, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil> Mar 18 20:01:59.307873420 (297h1m56.904644452s) [noise=true , address=145.79.227.7:4312 , piece_id=2FAL6E3RMBTBHZBXAJJVH2N5VGF5VYI3KNZXGNWCMWGBBPAUO2NQ, seg_pos={0 0}, upload=332666]: Done uploading piece. err: ecclient: upload failed (node:12AVp3qn5uwLPqYaZEwzcgoGMJhNga1BUh6bK5k2bxjWgZLQiiC, address:145.79.227.7:4312): protocol: rpc: dial tcp 145.79.227.7:4312: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:01:59.308026092 (297h1m56.904797124s) [noise=true , address=145.79.227.17:4987 , piece_id=PT3TWVB65JZFEWVH4IUZEU2N53FQ57QBRWVLWAUFP4UDLY3LWNLQ, seg_pos={0 0}, upload=332666]: Done uploading piece. err: ecclient: upload failed (node:1XsPVNGYUhKnXkQRfVfjyBRzyvhShSdsrnzXtwe4Xtf5MaHMvH, address:145.79.227.17:4987): protocol: rpc: dial tcp 145.79.227.17:4987: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:01:59.308195864 (297h1m56.904966894s) [noise=true , address=145.79.227.36:4524 , piece_id=3TAV3P55ITTHBAZEUUUTOAHJRGQXJXXVGZJP6AVR2ANVNOR3S54Q, seg_pos={0 0}, upload=332666]: Done uploading piece. err: ecclient: upload failed (node:1xfbCzjSwFDRrR1gPggeN8tNCLbRRWDVzCJ6mqhZsbVSe8d4s4, address:145.79.227.36:4524): protocol: rpc: dial tcp 145.79.227.36:4524: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:01:59.308316037 (297h1m56.90508706s) [noise=true , address=145.79.227.33:4892 , piece_id=DQ6ALC3VOAGFIJI3IKR6I7QFYM22C66QWAAXMH6NXI2FIQEZNQYA, seg_pos={0 0}, upload=332666]: Done uploading piece. err: ecclient: upload failed (node:12RHwFxcUjPsn12uHEAH3fBDJKv1v1tM8mRJXoFrZrG5djdswF5, address:145.79.227.33:4892): protocol: rpc: dial tcp 145.79.227.33:4892: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:01:59.308404403 (297h1m56.905175433s) [noise=true , address=145.79.227.21:4497 , piece_id=SP6EKJB7W4QXF7OIBVKEFWCPOZFL5IRIEDVAGS2M4QJSO34CUVRQ, seg_pos={0 0}, upload=332666]: Done uploading piece. err: ecclient: upload failed (node:12ZBe4we7VeFdthqRxyy1V9uQrA4r8ouNX68xqsk5bD26dkf2Gd, address:145.79.227.21:4497): protocol: context deadline exceeded; context deadline exceeded Mar 18 20:01:59.308437457 (297h1m56.90520847s) [seg_pos={0 0}, upload=332666]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 18 20:01:59.338288467 (297h1m56.93505949s) [upload=332666]: Done uploading segments... Mar 18 20:01:59.338335497 (297h1m56.935106523s) [upload=332666]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 16.0834986s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 56 out of 60 is failed (threshold=6) [XXXXXX.XXX...XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 5.000884517s trace: https://jaeger.dev.storj.io/trace/2d5866e693df3e8d FAIL (2026-03-18 20:15:12 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.000912884s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 49 out of 60 is failed (threshold=3) [.....XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.XXXXXX...XX.X.XXX] duration: 30.028451268s trace: https://jaeger.dev.storj.io/trace/196cc53575776cd7 FAIL (2026-03-18 20:15:07 UTC) operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Mar 18 20:14:37.682141670 (297h14m35.278912684s) [upload=332922]: Starting upload Mar 18 20:14:37.682232663 (297h14m35.279003686s) [upload=332922]: Done starting upload Mar 18 20:14:37.682326485 (297h14m35.279097498s) [upload=332922]: Uploading segments... Mar 18 20:14:37.682539898 (297h14m35.279310921s) [upload=332922]: Got next segment. Inline: false Mar 18 20:14:37.682578355 (297h14m35.279349378s) [upload=332922]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 20:14:37.682586290 (297h14m35.279357314s) [upload=332922]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 18 20:14:40.810771947 (297h14m38.40754298s) [seg_pos={0 0}, upload=332922]: Begin upload segment... Mar 18 20:14:40.810839156 (297h14m38.4076102s) [seg_pos={0 0}, upload=332922]: Done begin upload segment. Mar 18 20:14:40.810872567 (297h14m38.407643598s) [noise=true , address=77.90.31.16:4738 , piece_id=I3V7ZDX6CUB7RKHT3YABX3JSZLNGPE4NQGR2WEO6GCBVWDKMPG6A, seg_pos={0 0}, upload=332922]: Uploading piece... Mar 18 20:14:40.811051185 (297h14m38.40782221s) [noise=true , address=77.90.31.11:4557 , piece_id=IC2XFF76ULCQNPFSZGR52FTUSKG2EPFGSFAQM5MJ45SF6HQ5SAUQ, seg_pos={0 0}, upload=332922]: Uploading piece... Mar 18 20:14:40.811659639 (297h14m38.408430663s) [noise=true , address=45.91.116.5:4298 , piece_id=P4LWCFWYS7UJ2ECSGTM2FIGRMRFGQ2XP3CK3NRR6ZCTCQ4EGAZLA, seg_pos={0 0}, upload=332922]: Uploading piece... Mar 18 20:14:40.812561990 (297h14m38.409333019s) [noise=true , address=145.79.227.30:4767 , piece_id=HAE7DXSSFY4ORPUMW7M54N6YDYW4IK7O345LVO6YEBWCYEHCPTEQ, seg_pos={0 0}, upload=332922]: Uploading piece... Mar 18 20:14:40.812984335 (297h14m38.409755367s) [noise=true , address=45.91.116.14:4189 , piece_id=UDEZIVP3LWCJRLODJQBS652T6MGCK7GGTJ5YOHX6YOZFZA2TC7AQ, seg_pos={0 0}, upload=332922]: Uploading piece... Mar 18 20:14:40.814255674 (297h14m38.411026707s) [noise=true , address=77.90.31.13:4625 , piece_id=7NXAH5JA23PHLCL3MKIYSDCK324QWOEIYJTLL6BV7AKNPUKCPU6Q, seg_pos={0 0}, upload=332922]: Uploading piece... Mar 18 20:14:40.815750654 (297h14m38.412521679s) [noise=true , address=45.91.116.6:4973 , piece_id=M4IKW6YVKVOBHD5HZLV3OWZHD22E4VUA6YBZ6LMFJ5ME5X7UG4TQ, seg_pos={0 0}, upload=332922]: Uploading piece... Mar 18 20:14:40.816806505 (297h14m38.413577531s) [noise=true , address=45.91.116.9:4295 , piece_id=5VSBPT3IOZJWXMLFEH5NFLM27ZDGCOGZQ2ETWO3PLJS2KB52I6LQ, seg_pos={0 0}, upload=332922]: Uploading piece... Mar 18 20:14:40.818259221 (297h14m38.415030254s) [noise=true , address=145.79.227.28:4731 , piece_id=AR4DTQSZ442VSQK6B7UHU2P7EIK2C3HD432V3DZDE7PQPVWMP2ZA, seg_pos={0 0}, upload=332922]: Uploading piece... Mar 18 20:14:40.818995861 (297h14m38.415766887s) [noise=true , address=77.90.31.21:4878 , piece_id=5VCEI4AMXXANPGI3PB5YYSNB2Z72T2KOYYZ7I5ISZJOKO6AED3YQ, seg_pos={0 0}, upload=332922]: Uploading piece... Mar 18 20:14:40.820987229 (297h14m38.417758258s) [noise=true , address=145.79.227.34:4905 , piece_id=AOB3KDKH3YNAHL46SF4U2S53WVXK2B2B2UHUMNQICO65MS5WR3DA, seg_pos={0 0}, upload=332922]: Uploading piece... Mar 18 20:14:40.824407995 (297h14m38.421179027s) [noise=true , address=77.90.31.18:4810 , piece_id=ZKWBR5FNF7ZLVQSXX24VZ3JRHIBJUCHWIND4NE5MB2Y5WNAGPJJA, seg_pos={0 0}, upload=332922]: Uploading piece... Mar 18 20:14:40.826539622 (297h14m38.423310651s) [noise=true , address=79.127.219.45:4855 , piece_id=INCHPU33BNXQFCLGDKBOTDMXXMI5VQVSEQYVPEIDECOSULQ2ZFHQ, seg_pos={0 0}, upload=332922]: Uploading piece... Mar 18 20:14:40.826822640 (297h14m38.42359367s) [noise=true , address=79.127.219.38:4821 , piece_id=H4NDRDOCW4SJXA6UKYGBY55FVBHGGP2C33CWFMLTFB4BDL2AYQPQ, seg_pos={0 0}, upload=332922]: Uploading piece... Mar 18 20:14:40.827521030 (297h14m38.424292047s) [noise=true , address=145.79.227.25:4622 , piece_id=5NLGX2LP2MN4PWT7GM43IMTN244QBBB63PNAPHF3FL5FPBQVMEXA, seg_pos={0 0}, upload=332922]: Uploading piece... Mar 18 20:14:40.837307059 (297h14m38.434078076s) [noise=true , address=77.90.31.15:4700 , piece_id=OOJTAUY7Q7DZDRCV5S5I6MLBMGTW4MW7MYWNCWN5BYUZ7S4JDEJA, seg_pos={0 0}, upload=332922]: Uploading piece... Mar 18 20:14:40.881565358 (297h14m38.478336375s) [upload=332922]: Next returned nil segment Mar 18 20:14:40.881579130 (297h14m38.478350156s) [upload=332922]: Waiting for error group managing segments... Mar 18 20:14:40.886308132 (297h14m38.48307916s) [noise=true , address=45.91.116.9:4295 , piece_id=5VSBPT3IOZJWXMLFEH5NFLM27ZDGCOGZQ2ETWO3PLJS2KB52I6LQ, seg_pos={0 0}, upload=332922]: Done uploading piece. err: <nil> Mar 18 20:14:40.887752042 (297h14m38.484523059s) [noise=true , address=77.90.31.11:4557 , piece_id=IC2XFF76ULCQNPFSZGR52FTUSKG2EPFGSFAQM5MJ45SF6HQ5SAUQ, seg_pos={0 0}, upload=332922]: Done uploading piece. err: <nil> Mar 18 20:14:40.888000047 (297h14m38.484771079s) [noise=true , address=77.90.31.16:4738 , piece_id=I3V7ZDX6CUB7RKHT3YABX3JSZLNGPE4NQGR2WEO6GCBVWDKMPG6A, seg_pos={0 0}, upload=332922]: Done uploading piece. err: <nil> Mar 18 20:14:40.888323598 (297h14m38.48509461s) [noise=true , address=77.90.31.18:4810 , piece_id=ZKWBR5FNF7ZLVQSXX24VZ3JRHIBJUCHWIND4NE5MB2Y5WNAGPJJA, seg_pos={0 0}, upload=332922]: Done uploading piece. err: <nil> Mar 18 20:14:40.888656423 (297h14m38.485427447s) [noise=true , address=79.127.219.45:4855 , piece_id=INCHPU33BNXQFCLGDKBOTDMXXMI5VQVSEQYVPEIDECOSULQ2ZFHQ, seg_pos={0 0}, upload=332922]: Done uploading piece. err: <nil> Mar 18 20:14:40.888787778 (297h14m38.485558791s) [noise=true , address=77.90.31.13:4625 , piece_id=7NXAH5JA23PHLCL3MKIYSDCK324QWOEIYJTLL6BV7AKNPUKCPU6Q, seg_pos={0 0}, upload=332922]: Done uploading piece. err: <nil> Mar 18 20:14:40.895250726 (297h14m38.492021754s) [noise=true , address=45.91.116.6:4973 , piece_id=M4IKW6YVKVOBHD5HZLV3OWZHD22E4VUA6YBZ6LMFJ5ME5X7UG4TQ, seg_pos={0 0}, upload=332922]: Done uploading piece. err: <nil> Mar 18 20:14:40.899563413 (297h14m38.496334489s) [noise=true , address=77.90.31.21:4878 , piece_id=5VCEI4AMXXANPGI3PB5YYSNB2Z72T2KOYYZ7I5ISZJOKO6AED3YQ, seg_pos={0 0}, upload=332922]: Done uploading piece. err: <nil> Mar 18 20:14:40.902386354 (297h14m38.499157367s) [noise=true , address=45.91.116.5:4298 , piece_id=P4LWCFWYS7UJ2ECSGTM2FIGRMRFGQ2XP3CK3NRR6ZCTCQ4EGAZLA, seg_pos={0 0}, upload=332922]: Done uploading piece. err: <nil> Mar 18 20:14:40.906240108 (297h14m38.503011124s) [noise=true , address=79.127.219.38:4821 , piece_id=H4NDRDOCW4SJXA6UKYGBY55FVBHGGP2C33CWFMLTFB4BDL2AYQPQ, seg_pos={0 0}, upload=332922]: Done uploading piece. err: <nil> Mar 18 20:14:40.906346440 (297h14m38.503117456s) [noise=true , address=45.91.116.14:4189 , piece_id=UDEZIVP3LWCJRLODJQBS652T6MGCK7GGTJ5YOHX6YOZFZA2TC7AQ, seg_pos={0 0}, upload=332922]: Done uploading piece. err: <nil> Mar 18 20:14:40.909342371 (297h14m38.506113384s) [noise=true , address=77.90.31.15:4700 , piece_id=OOJTAUY7Q7DZDRCV5S5I6MLBMGTW4MW7MYWNCWN5BYUZ7S4JDEJA, seg_pos={0 0}, upload=332922]: Done uploading piece. err: <nil> Mar 18 20:15:07.652259020 (297h15m5.249030055s) [noise=true , address=145.79.227.34:4905 , piece_id=AOB3KDKH3YNAHL46SF4U2S53WVXK2B2B2UHUMNQICO65MS5WR3DA, seg_pos={0 0}, upload=332922]: Done uploading piece. err: ecclient: upload failed (node:12ffKESNSVpXQzbwJinP6vvgE6zoF3RFz2NRUuuJHvd7bvk1gc9, address:145.79.227.34:4905): protocol: rpc: dial tcp 145.79.227.34:4905: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:15:07.653011591 (297h15m5.249782626s) [noise=true , address=145.79.227.30:4767 , piece_id=HAE7DXSSFY4ORPUMW7M54N6YDYW4IK7O345LVO6YEBWCYEHCPTEQ, seg_pos={0 0}, upload=332922]: Done uploading piece. err: ecclient: upload failed (node:12sfaYCUhNCAyQAPu1jac5dRWR35Qqc4ZcKg75mp7nVQak295ug, address:145.79.227.30:4767): protocol: rpc: dial tcp 145.79.227.30:4767: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:15:07.654192872 (297h15m5.250963907s) [noise=true , address=145.79.227.28:4731 , piece_id=AR4DTQSZ442VSQK6B7UHU2P7EIK2C3HD432V3DZDE7PQPVWMP2ZA, seg_pos={0 0}, upload=332922]: Done uploading piece. err: ecclient: upload failed (node:12NQzTwVYQa6ZzxAHQm1apkwWnqJstTnAJQyHKhBzC2jTN9kEVK, address:145.79.227.28:4731): protocol: rpc: dial tcp 145.79.227.28:4731: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:15:07.654423705 (297h15m5.251194739s) [noise=true , address=145.79.227.25:4622 , piece_id=5NLGX2LP2MN4PWT7GM43IMTN244QBBB63PNAPHF3FL5FPBQVMEXA, seg_pos={0 0}, upload=332922]: Done uploading piece. err: ecclient: upload failed (node:1xEK5XL3sYcip7PsQcZBQYihgnEZykdMmsKkCr2TUmbTFYncfs, address:145.79.227.25:4622): protocol: rpc: dial tcp 145.79.227.25:4622: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:15:07.654712416 (297h15m5.251483452s) [seg_pos={0 0}, upload=332922]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Mar 18 20:15:07.677696504 (297h15m5.274467527s) [upload=332922]: Done uploading segments... Mar 18 20:15:07.677737744 (297h15m5.27450877s) [upload=332922]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 30.02846967s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 9 out of 60 is failed (threshold=3) [...........XXX..........X...X.........X....X....X.......X...] duration: 17.737886875s trace: https://jaeger.dev.storj.io/trace/24517a7cd40ab76d FAIL (2026-03-18 20:14:55 UTC) operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== oad Mar 18 20:12:28.252047067 (297h12m25.848818092s) [upload=332883]: Uploading segments... Mar 18 20:12:28.252323178 (297h12m25.84909421s) [upload=332883]: Got next segment. Inline: false Mar 18 20:12:28.252380536 (297h12m25.849151569s) [upload=332883]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 20:12:28.252388587 (297h12m25.849159622s) [upload=332883]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 18 20:12:31.413020979 (297h12m29.009792004s) [seg_pos={0 0}, upload=332883]: Begin upload segment... Mar 18 20:12:31.413109194 (297h12m29.009880217s) [seg_pos={0 0}, upload=332883]: Done begin upload segment. Mar 18 20:12:31.413182895 (297h12m29.009953924s) [noise=true , address=167.17.72.68:4653 , piece_id=6TO53KV4DBJ5HTWIQUZY2L6WVMB4PBPRWSZD6WHBLVFI5FUULYUQ, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.413918965 (297h12m29.010689999s) [noise=true , address=145.79.227.20:4454 , piece_id=F2WUAPOH2J3GQ7DMWLBJA44UGY5B3MMDS2U5ITXTOIPR4ELY5QYA, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.414450842 (297h12m29.011221869s) [noise=true , address=193.149.164.70:4111 , piece_id=H6GX5VVLRFNUAF2R7FRZ5GFQCZ2F57OL4GXIHWQJGZDOM3ST5DKQ, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.414901508 (297h12m29.011672532s) [noise=true , address=5.231.73.4:4002 , piece_id=3DWXH2Y6ZZNDHRFMJFL6EEMJN7WNDRKJ3ARFKN6UAGAHICNCBSDA, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.415342842 (297h12m29.012113865s) [noise=true , address=145.79.227.28:4750 , piece_id=KJVCFR2SJ5HAQKJUJP4CYPVCZQC6PBAQII6UGNYVKRODWRKENFQA, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.415582477 (297h12m29.01235351s) [noise=true , address=145.79.227.24:4615 , piece_id=ZSEUUMSYRSXH4AJ3CDZNFKXBZFT63AKVDFBFJ2RLAZK344GQ5AFA, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.415917210 (297h12m29.012688235s) [noise=true , address=193.149.164.68:4040 , piece_id=4NT224HIHI45JJNYH3ZCXQKDAMQHHYWY7SZPYQK5KN7JKNYFGI2A, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.416035983 (297h12m29.012807012s) [noise=true , address=193.149.164.67:4030 , piece_id=PYGFDMQLZ2SYL5NTWUGQGHWW52T2YYEGDKP5SUWQQUTVJSJZOO5A, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.416379117 (297h12m29.013150149s) [noise=true , address=45.91.116.5:4322 , piece_id=FN763EWEYGTBEXNHOT3FMKLW52DENMGPRBPBJJHNWQUNQ5JIN3XA, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.416442412 (297h12m29.013213436s) [noise=true , address=145.79.227.23:4564 , piece_id=3S5IXK7TKV2W3I635W2LQDH3PC6JRPHIVBK347CE7EYE4QQ5AGBQ, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.416786035 (297h12m29.013557058s) [noise=true , address=167.17.72.71:4808 , piece_id=JL5TJCWL454VA73PMWKZYBAUT6ZMRPEH55KJDSHEELC37HRVCO7Q, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.416946153 (297h12m29.013717177s) [noise=true , address=64.49.13.169:4705 , piece_id=FW7ICJDZJVOQR77HCGAWU7NKSBTWASW3BXH72WBGZHOFVPXA7XQQ, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.417084028 (297h12m29.013855051s) [noise=true , address=145.79.227.44:4202 , piece_id=UI7I4UPDOZCJ2MRPKV6FP6YPYNVSR5JWYDVFBO2TRFYQJWVWIUNQ, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.424729560 (297h12m29.021500589s) [noise=true , address=79.127.205.233:4156 , piece_id=5OZYLGXB7R6VMYGATW7GJD2CNAVO45PATTCFGNPWYZ3ZURYLZSQA, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.425355484 (297h12m29.022126513s) [noise=true , address=193.149.164.69:4077 , piece_id=GYZMBIAEBL4KBCSFA4H5JLWEB36NBN2BGVQ67A5G4ECUWY5BACXA, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.429910441 (297h12m29.026681471s) [noise=true , address=79.127.219.46:4918 , piece_id=WC2RL2PMO4EVATTOPZBWFHOZE6CBGQ2ROGJ44PDQUPQKYXBFKY6Q, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.431104280 (297h12m29.027875311s) [noise=true , address=79.127.219.45:4883 , piece_id=7DXMZGFUW2QAI2OID7WSSMJG6B3WW52HCMRWX2LS6YFEWAFMZZNQ, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.432173523 (297h12m29.02894455s) [noise=true , address=193.149.164.71:4300 , piece_id=4FI5PVS627OOB2M45U4QTOXGBJHNOZMUDDCFAW57W7PBAAQSJC4A, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.478900188 (297h12m29.075671211s) [upload=332883]: Next returned nil segment Mar 18 20:12:31.478914031 (297h12m29.075685054s) [upload=332883]: Waiting for error group managing segments... Mar 18 20:12:31.512891809 (297h12m29.109662832s) [noise=true , address=79.127.219.46:4918 , piece_id=WC2RL2PMO4EVATTOPZBWFHOZE6CBGQ2ROGJ44PDQUPQKYXBFKY6Q, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil> Mar 18 20:12:31.529013012 (297h12m29.125784038s) [noise=true , address=45.91.116.5:4322 , piece_id=FN763EWEYGTBEXNHOT3FMKLW52DENMGPRBPBJJHNWQUNQ5JIN3XA, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil> Mar 18 20:12:31.564707355 (297h12m29.161478379s) [noise=true , address=79.127.219.45:4883 , piece_id=7DXMZGFUW2QAI2OID7WSSMJG6B3WW52HCMRWX2LS6YFEWAFMZZNQ, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil> Mar 18 20:12:31.611013762 (297h12m29.207784793s) [noise=true , address=5.231.73.4:4002 , piece_id=3DWXH2Y6ZZNDHRFMJFL6EEMJN7WNDRKJ3ARFKN6UAGAHICNCBSDA, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil> Mar 18 20:12:31.713823345 (297h12m29.310594381s) [noise=true , address=167.17.72.68:4653 , piece_id=6TO53KV4DBJ5HTWIQUZY2L6WVMB4PBPRWSZD6WHBLVFI5FUULYUQ, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil> Mar 18 20:12:31.997071787 (297h12m29.59384282s) [noise=true , address=79.127.205.233:4156 , piece_id=5OZYLGXB7R6VMYGATW7GJD2CNAVO45PATTCFGNPWYZ3ZURYLZSQA, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil> Mar 18 20:12:31.999264198 (297h12m29.59603524s) [noise=true , address=193.149.164.70:4111 , piece_id=H6GX5VVLRFNUAF2R7FRZ5GFQCZ2F57OL4GXIHWQJGZDOM3ST5DKQ, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil> Mar 18 20:12:32.008593285 (297h12m29.605364328s) [noise=true , address=193.149.164.71:4300 , piece_id=4FI5PVS627OOB2M45U4QTOXGBJHNOZMUDDCFAW57W7PBAAQSJC4A, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil> Mar 18 20:12:32.011203033 (297h12m29.607974056s) [noise=true , address=193.149.164.68:4040 , piece_id=4NT224HIHI45JJNYH3ZCXQKDAMQHHYWY7SZPYQK5KN7JKNYFGI2A, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil> Mar 18 20:12:32.038310240 (297h12m29.635081269s) [noise=true , address=193.149.164.67:4030 , piece_id=PYGFDMQLZ2SYL5NTWUGQGHWW52T2YYEGDKP5SUWQQUTVJSJZOO5A, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil> Mar 18 20:12:32.091831027 (297h12m29.68860206s) [noise=true , address=64.49.13.169:4705 , piece_id=FW7ICJDZJVOQR77HCGAWU7NKSBTWASW3BXH72WBGZHOFVPXA7XQQ, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil> Mar 18 20:12:32.408908738 (297h12m30.005679764s) [noise=true , address=193.149.164.69:4077 , piece_id=GYZMBIAEBL4KBCSFA4H5JLWEB36NBN2BGVQ67A5G4ECUWY5BACXA, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil> Mar 18 20:12:32.489564296 (297h12m30.086335307s) [noise=true , address=167.17.72.71:4808 , piece_id=JL5TJCWL454VA73PMWKZYBAUT6ZMRPEH55KJDSHEELC37HRVCO7Q, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil> Mar 18 20:13:28.204470030 (297h13m25.801241064s) [noise=true , address=145.79.227.23:4564 , piece_id=3S5IXK7TKV2W3I635W2LQDH3PC6JRPHIVBK347CE7EYE4QQ5AGBQ, seg_pos={0 0}, upload=332883]: Done uploading piece. err: ecclient: upload failed (node:121WW4uBQe8bGQKkasQQKT3k4EAdzncsG1ebNvgKf8S5SyNttCc, address:145.79.227.23:4564): protocol: rpc: dial tcp 145.79.227.23:4564: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:13:28.205066456 (297h13m25.801837482s) [noise=true , address=145.79.227.28:4750 , piece_id=KJVCFR2SJ5HAQKJUJP4CYPVCZQC6PBAQII6UGNYVKRODWRKENFQA, seg_pos={0 0}, upload=332883]: Done uploading piece. err: ecclient: upload failed (node:12UP8PxqeK7t24PXjopH1nziHYQFnsxhkamSHz8dmwcbuDxkAF6, address:145.79.227.28:4750): protocol: rpc: dial tcp 145.79.227.28:4750: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:13:28.205099998 (297h13m25.80187103s) [noise=true , address=145.79.227.44:4202 , piece_id=UI7I4UPDOZCJ2MRPKV6FP6YPYNVSR5JWYDVFBO2TRFYQJWVWIUNQ, seg_pos={0 0}, upload=332883]: Done uploading piece. err: ecclient: upload failed (node:121ogdaiVGybXEVsS8LDJuJRYXGxb1LCNnGPuwbde96eSEzkTW4, address:145.79.227.44:4202): protocol: rpc: dial tcp 145.79.227.44:4202: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:13:28.205340404 (297h13m25.802111436s) [noise=true , address=145.79.227.24:4615 , piece_id=ZSEUUMSYRSXH4AJ3CDZNFKXBZFT63AKVDFBFJ2RLAZK344GQ5AFA, seg_pos={0 0}, upload=332883]: Done uploading piece. err: ecclient: upload failed (node:12NJ5w8pAWCudDA9Xs4ES7b2evnbscesd682wJDh2gMMZ3Jf2zc, address:145.79.227.24:4615): protocol: rpc: dial tcp 145.79.227.24:4615: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:13:28.205593210 (297h13m25.802364235s) [noise=true , address=145.79.227.20:4454 , piece_id=F2WUAPOH2J3GQ7DMWLBJA44UGY5B3MMDS2U5ITXTOIPR4ELY5QYA, seg_pos={0 0}, upload=332883]: Done uploading piece. err: ecclient: upload failed (node:1SafkJPqKHLwMaqyZdLq3yJ1xdjqDtWHFrUwynmegUVwEC7y5J, address:145.79.227.20:4454): protocol: context deadline exceeded; context deadline exceeded Mar 18 20:13:28.205633541 (297h13m25.802404553s) [seg_pos={0 0}, upload=332883]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 18 20:13:28.228774364 (297h13m25.825545388s) [upload=332883]: Done uploading segments... Mar 18 20:13:28.228821883 (297h13m25.825592906s) [upload=332883]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 17.737977186s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 8 out of 60 is failed (threshold=3) [.........XXXXXX.XX..........................................] duration: 13.273811171s trace: https://jaeger.dev.storj.io/trace/4b4d10a8051ad01b FAIL (2026-03-18 20:14:46 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 13.273929576s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 48 out of 60 is failed (threshold=3) [......XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.XXXXXX...XX.X.XX] duration: 30.068651654s trace: https://jaeger.dev.storj.io/trace/6d0dfb9e3a760953 FAIL (2026-03-18 20:14:37 UTC) operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Mar 18 20:14:07.544731353 (297h14m5.141502384s) [upload=332913]: Starting upload Mar 18 20:14:07.545033735 (297h14m5.141804772s) [upload=332913]: Done starting upload Mar 18 20:14:07.545389875 (297h14m5.142160889s) [upload=332913]: Uploading segments... Mar 18 20:14:07.545655410 (297h14m5.14242644s) [upload=332913]: Got next segment. Inline: false Mar 18 20:14:07.545751880 (297h14m5.14252291s) [upload=332913]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 20:14:07.545761826 (297h14m5.142532854s) [upload=332913]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 18 20:14:10.021199333 (297h14m7.617970367s) [seg_pos={0 0}, upload=332913]: Begin upload segment... Mar 18 20:14:10.021286806 (297h14m7.618057833s) [seg_pos={0 0}, upload=332913]: Done begin upload segment. Mar 18 20:14:10.021326437 (297h14m7.618097468s) [noise=true , address=45.91.116.14:4224 , piece_id=TZOE2IFBW2ZB5ZBF22K2AZS6IBBBKFD3YWA4MLECNAZ3ERH4TW6A, seg_pos={0 0}, upload=332913]: Uploading piece... Mar 18 20:14:10.022432153 (297h14m7.619203193s) [noise=true , address=145.79.227.26:4657 , piece_id=7NRICKDAL6YEX4PCV4ZA5WN4NIAINMPRJ72JDAN5E2BV5Q7TTIUQ, seg_pos={0 0}, upload=332913]: Uploading piece... Mar 18 20:14:10.023265360 (297h14m7.620036389s) [noise=true , address=77.90.31.15:4702 , piece_id=CDGVVWVNN5NPFFBB2LOKC6DHY76OZTUWF4SXVTDXGLIOVK2TTSRQ, seg_pos={0 0}, upload=332913]: Uploading piece... Mar 18 20:14:10.024063783 (297h14m7.620834816s) [noise=true , address=45.91.116.7:4026 , piece_id=XP2LYXRDKSBJNJHYUQORJ6PT37CLB5JBKFICQ767M3OWTTFC2XBQ, seg_pos={0 0}, upload=332913]: Uploading piece... Mar 18 20:14:10.024673749 (297h14m7.621444782s) [noise=true , address=45.91.116.9:4264 , piece_id=BLQSMXAMRMQ6DBD5UKFYSN4Q5HV73MQQAOZEDUMG3LYQW7FXEKWA, seg_pos={0 0}, upload=332913]: Uploading piece... Mar 18 20:14:10.025297800 (297h14m7.622068832s) [noise=true , address=45.91.116.11:4119 , piece_id=JEKBI3JW7CN5QPZLNKHFAKXYQU7MQUMBVE7NBX4P2LTUVDTK6XYQ, seg_pos={0 0}, upload=332913]: Uploading piece... Mar 18 20:14:10.025949418 (297h14m7.622720452s) [noise=true , address=45.91.116.13:4157 , piece_id=OVXCADBZ6QZWNCCMYRFAZDYR5GN7UI7GUV76PKEXEIIBUEUERRVA, seg_pos={0 0}, upload=332913]: Uploading piece... Mar 18 20:14:10.026642018 (297h14m7.623413051s) [noise=true , address=145.79.227.10:4407 , piece_id=THEJ5SCOYSV5HGLLMCDK5P564ZFNF72YY2LFJGWVVHCTMJI5A26A, seg_pos={0 0}, upload=332913]: Uploading piece... Mar 18 20:14:10.027163213 (297h14m7.623934247s) [noise=true , address=45.91.116.4:4947 , piece_id=Y4GSQEZ3EDHGZ2TDOHY4K4BHCXUOCHX3OBBRX4ZBGWTEHC6HUYEA, seg_pos={0 0}, upload=332913]: Uploading piece... Mar 18 20:14:10.030365658 (297h14m7.627136691s) [noise=true , address=79.127.219.45:4858 , piece_id=SUGRIIUR2IUQFKOFXERG2JWEQJHSWEAR476RITTH37OBLKBAEUHA, seg_pos={0 0}, upload=332913]: Uploading piece... Mar 18 20:14:10.031223506 (297h14m7.627994537s) [noise=true , address=45.91.116.5:4301 , piece_id=AJK3XBGHQE2MO5ZGGLGEG4D6GJPLAH56FFGSP74CV5WQDPTM7ZXA, seg_pos={0 0}, upload=332913]: Uploading piece... Mar 18 20:14:10.031833160 (297h14m7.628604184s) [noise=true , address=145.79.227.33:4869 , piece_id=IMMLUHPS6JF44D4TG4UHQKTRY2JEMYLUIYPEXD2YW66RVFSFCUIQ, seg_pos={0 0}, upload=332913]: Uploading piece... Mar 18 20:14:10.032440163 (297h14m7.629211193s) [noise=true , address=145.79.227.38:4982 , piece_id=X5V7X5PEZMZG6LKZ3N2L4USORRL3MZ52ZZWRL4TIKWSM2IYG32HA, seg_pos={0 0}, upload=332913]: Uploading piece... Mar 18 20:14:10.033224854 (297h14m7.629995889s) [noise=true , address=77.90.31.11:4554 , piece_id=OGVWI4TA4LK2QLKSXI5H4NSLAI6NHFMZRJSSRAHFKHVSK67UFWAA, seg_pos={0 0}, upload=332913]: Uploading piece... Mar 18 20:14:10.034996080 (297h14m7.631767094s) [noise=true , address=79.127.219.41:4926 , piece_id=HOE326OG27GBH6PTDCJYDNNPTEHRGGI5MAIWSP6NVJHTXDHNFXUQ, seg_pos={0 0}, upload=332913]: Uploading piece... Mar 18 20:14:10.037013049 (297h14m7.633784082s) [noise=true , address=77.90.31.18:4810 , piece_id=JAT4XJ5EQJJDYMS6Y2C34WHNCT2C7YSMXHBVXNBNXAH7OBRBKG6Q, seg_pos={0 0}, upload=332913]: Uploading piece... Mar 18 20:14:10.135564693 (297h14m7.732335721s) [upload=332913]: Next returned nil segment Mar 18 20:14:10.135583583 (297h14m7.732354613s) [upload=332913]: Waiting for error group managing segments... Mar 18 20:14:10.140452014 (297h14m7.737223066s) [noise=true , address=77.90.31.18:4810 , piece_id=JAT4XJ5EQJJDYMS6Y2C34WHNCT2C7YSMXHBVXNBNXAH7OBRBKG6Q, seg_pos={0 0}, upload=332913]: Done uploading piece. err: <nil> Mar 18 20:14:10.141407468 (297h14m7.738178524s) [noise=true , address=45.91.116.14:4224 , piece_id=TZOE2IFBW2ZB5ZBF22K2AZS6IBBBKFD3YWA4MLECNAZ3ERH4TW6A, seg_pos={0 0}, upload=332913]: Done uploading piece. err: <nil> Mar 18 20:14:10.141470151 (297h14m7.738241176s) [noise=true , address=45.91.116.9:4264 , piece_id=BLQSMXAMRMQ6DBD5UKFYSN4Q5HV73MQQAOZEDUMG3LYQW7FXEKWA, seg_pos={0 0}, upload=332913]: Done uploading piece. err: <nil> Mar 18 20:14:10.141771569 (297h14m7.738542596s) [noise=true , address=45.91.116.11:4119 , piece_id=JEKBI3JW7CN5QPZLNKHFAKXYQU7MQUMBVE7NBX4P2LTUVDTK6XYQ, seg_pos={0 0}, upload=332913]: Done uploading piece. err: <nil> Mar 18 20:14:10.144580255 (297h14m7.74135129s) [noise=true , address=77.90.31.15:4702 , piece_id=CDGVVWVNN5NPFFBB2LOKC6DHY76OZTUWF4SXVTDXGLIOVK2TTSRQ, seg_pos={0 0}, upload=332913]: Done uploading piece. err: <nil> Mar 18 20:14:10.153355626 (297h14m7.750126654s) [noise=true , address=77.90.31.11:4554 , piece_id=OGVWI4TA4LK2QLKSXI5H4NSLAI6NHFMZRJSSRAHFKHVSK67UFWAA, seg_pos={0 0}, upload=332913]: Done uploading piece. err: <nil> Mar 18 20:14:10.154053075 (297h14m7.750824087s) [noise=true , address=45.91.116.7:4026 , piece_id=XP2LYXRDKSBJNJHYUQORJ6PT37CLB5JBKFICQ767M3OWTTFC2XBQ, seg_pos={0 0}, upload=332913]: Done uploading piece. err: <nil> Mar 18 20:14:10.154600519 (297h14m7.751371542s) [noise=true , address=79.127.219.41:4926 , piece_id=HOE326OG27GBH6PTDCJYDNNPTEHRGGI5MAIWSP6NVJHTXDHNFXUQ, seg_pos={0 0}, upload=332913]: Done uploading piece. err: <nil> Mar 18 20:14:10.155015479 (297h14m7.751786493s) [noise=true , address=45.91.116.4:4947 , piece_id=Y4GSQEZ3EDHGZ2TDOHY4K4BHCXUOCHX3OBBRX4ZBGWTEHC6HUYEA, seg_pos={0 0}, upload=332913]: Done uploading piece. err: <nil> Mar 18 20:14:10.155799383 (297h14m7.752570396s) [noise=true , address=79.127.219.45:4858 , piece_id=SUGRIIUR2IUQFKOFXERG2JWEQJHSWEAR476RITTH37OBLKBAEUHA, seg_pos={0 0}, upload=332913]: Done uploading piece. err: <nil> Mar 18 20:14:10.160897525 (297h14m7.757668555s) [noise=true , address=45.91.116.5:4301 , piece_id=AJK3XBGHQE2MO5ZGGLGEG4D6GJPLAH56FFGSP74CV5WQDPTM7ZXA, seg_pos={0 0}, upload=332913]: Done uploading piece. err: <nil> Mar 18 20:14:10.161613614 (297h14m7.758384648s) [noise=true , address=45.91.116.13:4157 , piece_id=OVXCADBZ6QZWNCCMYRFAZDYR5GN7UI7GUV76PKEXEIIBUEUERRVA, seg_pos={0 0}, upload=332913]: Done uploading piece. err: <nil> Mar 18 20:14:37.512386791 (297h14m35.109157821s) [noise=true , address=145.79.227.10:4407 , piece_id=THEJ5SCOYSV5HGLLMCDK5P564ZFNF72YY2LFJGWVVHCTMJI5A26A, seg_pos={0 0}, upload=332913]: Done uploading piece. err: ecclient: upload failed (node:1kJKjEfzwg1wuBUP2Bw4oaSoy6mQZdvDED6PpkbrqQwbkh3bUW, address:145.79.227.10:4407): protocol: rpc: dial tcp 145.79.227.10:4407: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:14:37.512567974 (297h14m35.109339003s) [noise=true , address=145.79.227.38:4982 , piece_id=X5V7X5PEZMZG6LKZ3N2L4USORRL3MZ52ZZWRL4TIKWSM2IYG32HA, seg_pos={0 0}, upload=332913]: Done uploading piece. err: ecclient: upload failed (node:1NxoYJXCXmTvQncbkXt6yH7mhEBBrzVdzhgL5J6W5TD4N3pkaG, address:145.79.227.38:4982): protocol: rpc: dial tcp 145.79.227.38:4982: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:14:37.512804275 (297h14m35.109575308s) [noise=true , address=145.79.227.26:4657 , piece_id=7NRICKDAL6YEX4PCV4ZA5WN4NIAINMPRJ72JDAN5E2BV5Q7TTIUQ, seg_pos={0 0}, upload=332913]: Done uploading piece. err: ecclient: upload failed (node:129MyUjh5Nw3NhQxefnBzxYUK3mSXqwpW3nLa3KerYrjpuJiUfq, address:145.79.227.26:4657): protocol: rpc: dial tcp 145.79.227.26:4657: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:14:37.514090552 (297h14m35.110861581s) [noise=true , address=145.79.227.33:4869 , piece_id=IMMLUHPS6JF44D4TG4UHQKTRY2JEMYLUIYPEXD2YW66RVFSFCUIQ, seg_pos={0 0}, upload=332913]: Done uploading piece. err: ecclient: upload failed (node:18CQxjq65r6xVTmNRurSQPzKxj4BoR6cJ8Uq29sTrdh6uXm5sq, address:145.79.227.33:4869): protocol: context deadline exceeded; context deadline exceeded Mar 18 20:14:37.514132583 (297h14m35.1109036s) [seg_pos={0 0}, upload=332913]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Mar 18 20:14:37.578192678 (297h14m35.174963704s) [upload=332913]: Done uploading segments... Mar 18 20:14:37.578242214 (297h14m35.175013244s) [upload=332913]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 30.068792459s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 6 out of 60 is failed (threshold=3) [..............X..........XX..X....X........X................] duration: 16.491023929s trace: https://jaeger.dev.storj.io/trace/589419efc52e5e2b FAIL (2026-03-18 20:14:34 UTC) operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== Mar 18 20:00:59.343084966 (297h0m56.939855989s) [upload=332666]: Starting upload Mar 18 20:00:59.343175518 (297h0m56.939946548s) [upload=332666]: Done starting upload Mar 18 20:00:59.343565585 (297h0m56.940336612s) [upload=332666]: Uploading segments... Mar 18 20:00:59.344256647 (297h0m56.941027676s) [upload=332666]: Got next segment. Inline: false Mar 18 20:00:59.344352215 (297h0m56.941123241s) [upload=332666]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 20:00:59.344358706 (297h0m56.941129732s) [upload=332666]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 18 20:01:02.340259299 (297h0m59.937030329s) [seg_pos={0 0}, upload=332666]: Begin upload segment... Mar 18 20:01:02.340368743 (297h0m59.937139767s) [seg_pos={0 0}, upload=332666]: Done begin upload segment. Mar 18 20:01:02.340479563 (297h0m59.93725059s) [noise=true , address=45.91.116.8:4061 , piece_id=6XFVKT4AAO7QN457OQMRKPI6R24TJTAN47UXLEVOEFQVOS6M6C3Q, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.340523837 (297h0m59.93729486s) [noise=true , address=79.127.205.233:4166 , piece_id=SFGK5YCWHFRVXF6QNHYP2CYJCZLSTQ76OYRVJPWWXQTF2KFSFJKA, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.341468438 (297h0m59.938239447s) [noise=true , address=45.91.116.4:4961 , piece_id=ZMUGQECUVPVHDNKN75XPBET5M2HFXXK4WMTP4V5KEK63JVZCX5EA, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.341520025 (297h0m59.938291058s) [noise=true , address=167.17.72.71:4815 , piece_id=CXGTYNNBILN3TT7I6Z7UI75F2ZCS4QUTLS6KC4FJ5IHS3XOYWDJQ, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.342232992 (297h0m59.939004023s) [noise=true , address=145.79.227.36:4524 , piece_id=3TAV3P55ITTHBAZEUUUTOAHJRGQXJXXVGZJP6AVR2ANVNOR3S54Q, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.342917368 (297h0m59.939688398s) [noise=true , address=193.149.164.71:4297 , piece_id=4IJPXLPCX3LKFOBOSRHK2DTNRKH67K7FXJUKQ3JPRWHS47ZFEIDA, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.343244957 (297h0m59.940015982s) [noise=true , address=145.79.227.33:4892 , piece_id=DQ6ALC3VOAGFIJI3IKR6I7QFYM22C66QWAAXMH6NXI2FIQEZNQYA, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.344270079 (297h0m59.941041117s) [noise=true , address=145.79.227.21:4497 , piece_id=SP6EKJB7W4QXF7OIBVKEFWCPOZFL5IRIEDVAGS2M4QJSO34CUVRQ, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.345025057 (297h0m59.941796084s) [noise=true , address=77.90.31.17:4793 , piece_id=2U4IBUA2GGUGDBXQGIGQKUK4ZR3L7CDKC6MTGIKAUME4F7MFZRLA, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.345115802 (297h0m59.941886836s) [noise=true , address=167.17.72.69:4731 , piece_id=VOEVIKKFJMC2Y3TPQIDMYUP37NLFU7KPRH4HRRSMQLWUKTSLOH2Q, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.345690600 (297h0m59.942461634s) [noise=true , address=77.90.31.13:4648 , piece_id=BPMB5CFOYHFT3BS4ORR6JF6BNFC2AOAJZOSPMG67NAMLGGP5XVEQ, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.346311592 (297h0m59.943082623s) [noise=true , address=79.127.205.234:4202 , piece_id=YY4LYWX26O6RIOBONESNKTUZEFZCRL4UUX3NEBQ53AJV4IEDG5MQ, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.346791142 (297h0m59.943562166s) [noise=true , address=45.91.116.10:4110 , piece_id=UYLMDCWJEJOHGICA37KNH7G5AQXNIPRH5NRYLBUUMGDRVSLTEYHQ, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.347094999 (297h0m59.94386607s) [noise=true , address=145.79.227.17:4987 , piece_id=PT3TWVB65JZFEWVH4IUZEU2N53FQ57QBRWVLWAUFP4UDLY3LWNLQ, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.347781281 (297h0m59.944552305s) [noise=true , address=79.127.219.46:4919 , piece_id=KK6HRIVBMYLIHRHGPQH3AWMNSVHTTIBESHDMJZA72T446RX6QKRQ, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.348263123 (297h0m59.945034137s) [noise=true , address=193.149.164.68:4033 , piece_id=JZPQKPYGO3W3RZZBITQ73X2XEPIBBSLGRMWIVOM4Z3GTV5UFKRTQ, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.348915505 (297h0m59.945686536s) [noise=true , address=145.79.227.7:4312 , piece_id=2FAL6E3RMBTBHZBXAJJVH2N5VGF5VYI3KNZXGNWCMWGBBPAUO2NQ, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.349366298 (297h0m59.946137337s) [noise=true , address=193.149.164.67:4007 , piece_id=HURQGH67I5B6V4YUV3PRZL3KSKN3XTSK3UMCHZRNT6L3W3PXXHPQ, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.429834098 (297h1m0.026605126s) [upload=332666]: Next returned nil segment Mar 18 20:01:02.429853411 (297h1m0.026624442s) [upload=332666]: Waiting for error group managing segments... Mar 18 20:01:02.446867680 (297h1m0.043638704s) [noise=true , address=45.91.116.10:4110 , piece_id=UYLMDCWJEJOHGICA37KNH7G5AQXNIPRH5NRYLBUUMGDRVSLTEYHQ, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil> Mar 18 20:01:02.451665165 (297h1m0.048436196s) [noise=true , address=45.91.116.4:4961 , piece_id=ZMUGQECUVPVHDNKN75XPBET5M2HFXXK4WMTP4V5KEK63JVZCX5EA, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil> Mar 18 20:01:02.452538886 (297h1m0.049309913s) [noise=true , address=77.90.31.17:4793 , piece_id=2U4IBUA2GGUGDBXQGIGQKUK4ZR3L7CDKC6MTGIKAUME4F7MFZRLA, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil> Mar 18 20:01:02.455597040 (297h1m0.052368067s) [noise=true , address=45.91.116.8:4061 , piece_id=6XFVKT4AAO7QN457OQMRKPI6R24TJTAN47UXLEVOEFQVOS6M6C3Q, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil> Mar 18 20:01:02.457848739 (297h1m0.054619766s) [noise=true , address=77.90.31.13:4648 , piece_id=BPMB5CFOYHFT3BS4ORR6JF6BNFC2AOAJZOSPMG67NAMLGGP5XVEQ, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil> Mar 18 20:01:02.461479709 (297h1m0.058250746s) [noise=true , address=79.127.219.46:4919 , piece_id=KK6HRIVBMYLIHRHGPQH3AWMNSVHTTIBESHDMJZA72T446RX6QKRQ, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil> Mar 18 20:01:02.676859466 (297h1m0.27363049s) [noise=true , address=167.17.72.69:4731 , piece_id=VOEVIKKFJMC2Y3TPQIDMYUP37NLFU7KPRH4HRRSMQLWUKTSLOH2Q, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil> Mar 18 20:01:02.909860743 (297h1m0.506631766s) [noise=true , address=79.127.205.234:4202 , piece_id=YY4LYWX26O6RIOBONESNKTUZEFZCRL4UUX3NEBQ53AJV4IEDG5MQ, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil> Mar 18 20:01:02.930427742 (297h1m0.527198754s) [noise=true , address=193.149.164.71:4297 , piece_id=4IJPXLPCX3LKFOBOSRHK2DTNRKH67K7FXJUKQ3JPRWHS47ZFEIDA, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil> Mar 18 20:01:02.944696889 (297h1m0.541467914s) [noise=true , address=193.149.164.67:4007 , piece_id=HURQGH67I5B6V4YUV3PRZL3KSKN3XTSK3UMCHZRNT6L3W3PXXHPQ, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil> Mar 18 20:01:02.951319183 (297h1m0.548090212s) [noise=true , address=193.149.164.68:4033 , piece_id=JZPQKPYGO3W3RZZBITQ73X2XEPIBBSLGRMWIVOM4Z3GTV5UFKRTQ, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil> Mar 18 20:01:02.975619436 (297h1m0.572390467s) [noise=true , address=79.127.205.233:4166 , piece_id=SFGK5YCWHFRVXF6QNHYP2CYJCZLSTQ76OYRVJPWWXQTF2KFSFJKA, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil> Mar 18 20:01:03.819918528 (297h1m1.416689564s) [noise=true , address=167.17.72.71:4815 , piece_id=CXGTYNNBILN3TT7I6Z7UI75F2ZCS4QUTLS6KC4FJ5IHS3XOYWDJQ, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil> Mar 18 20:01:59.307873420 (297h1m56.904644452s) [noise=true , address=145.79.227.7:4312 , piece_id=2FAL6E3RMBTBHZBXAJJVH2N5VGF5VYI3KNZXGNWCMWGBBPAUO2NQ, seg_pos={0 0}, upload=332666]: Done uploading piece. err: ecclient: upload failed (node:12AVp3qn5uwLPqYaZEwzcgoGMJhNga1BUh6bK5k2bxjWgZLQiiC, address:145.79.227.7:4312): protocol: rpc: dial tcp 145.79.227.7:4312: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:01:59.308026092 (297h1m56.904797124s) [noise=true , address=145.79.227.17:4987 , piece_id=PT3TWVB65JZFEWVH4IUZEU2N53FQ57QBRWVLWAUFP4UDLY3LWNLQ, seg_pos={0 0}, upload=332666]: Done uploading piece. err: ecclient: upload failed (node:1XsPVNGYUhKnXkQRfVfjyBRzyvhShSdsrnzXtwe4Xtf5MaHMvH, address:145.79.227.17:4987): protocol: rpc: dial tcp 145.79.227.17:4987: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:01:59.308195864 (297h1m56.904966894s) [noise=true , address=145.79.227.36:4524 , piece_id=3TAV3P55ITTHBAZEUUUTOAHJRGQXJXXVGZJP6AVR2ANVNOR3S54Q, seg_pos={0 0}, upload=332666]: Done uploading piece. err: ecclient: upload failed (node:1xfbCzjSwFDRrR1gPggeN8tNCLbRRWDVzCJ6mqhZsbVSe8d4s4, address:145.79.227.36:4524): protocol: rpc: dial tcp 145.79.227.36:4524: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:01:59.308316037 (297h1m56.90508706s) [noise=true , address=145.79.227.33:4892 , piece_id=DQ6ALC3VOAGFIJI3IKR6I7QFYM22C66QWAAXMH6NXI2FIQEZNQYA, seg_pos={0 0}, upload=332666]: Done uploading piece. err: ecclient: upload failed (node:12RHwFxcUjPsn12uHEAH3fBDJKv1v1tM8mRJXoFrZrG5djdswF5, address:145.79.227.33:4892): protocol: rpc: dial tcp 145.79.227.33:4892: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:01:59.308404403 (297h1m56.905175433s) [noise=true , address=145.79.227.21:4497 , piece_id=SP6EKJB7W4QXF7OIBVKEFWCPOZFL5IRIEDVAGS2M4QJSO34CUVRQ, seg_pos={0 0}, upload=332666]: Done uploading piece. err: ecclient: upload failed (node:12ZBe4we7VeFdthqRxyy1V9uQrA4r8ouNX68xqsk5bD26dkf2Gd, address:145.79.227.21:4497): protocol: context deadline exceeded; context deadline exceeded Mar 18 20:01:59.308437457 (297h1m56.90520847s) [seg_pos={0 0}, upload=332666]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 18 20:01:59.338288467 (297h1m56.93505949s) [upload=332666]: Done uploading segments... Mar 18 20:01:59.338335497 (297h1m56.935106523s) [upload=332666]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 16.491196333s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 8 out of 60 is failed (threshold=3) [..........XXXXXX.XX.........................................] duration: 16.547436892s trace: https://jaeger.dev.storj.io/trace/4b4d10a8051ad01b FAIL (2026-03-18 20:14:28 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 16.54746541s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 6 out of 60 is failed (threshold=3) [...............X..........XX..X....X........X...............] duration: 17.320585102s trace: https://jaeger.dev.storj.io/trace/589419efc52e5e2b FAIL (2026-03-18 20:14:14 UTC) operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== Mar 18 20:00:59.343084966 (297h0m56.939855989s) [upload=332666]: Starting upload Mar 18 20:00:59.343175518 (297h0m56.939946548s) [upload=332666]: Done starting upload Mar 18 20:00:59.343565585 (297h0m56.940336612s) [upload=332666]: Uploading segments... Mar 18 20:00:59.344256647 (297h0m56.941027676s) [upload=332666]: Got next segment. Inline: false Mar 18 20:00:59.344352215 (297h0m56.941123241s) [upload=332666]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 20:00:59.344358706 (297h0m56.941129732s) [upload=332666]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 18 20:01:02.340259299 (297h0m59.937030329s) [seg_pos={0 0}, upload=332666]: Begin upload segment... Mar 18 20:01:02.340368743 (297h0m59.937139767s) [seg_pos={0 0}, upload=332666]: Done begin upload segment. Mar 18 20:01:02.340479563 (297h0m59.93725059s) [noise=true , address=45.91.116.8:4061 , piece_id=6XFVKT4AAO7QN457OQMRKPI6R24TJTAN47UXLEVOEFQVOS6M6C3Q, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.340523837 (297h0m59.93729486s) [noise=true , address=79.127.205.233:4166 , piece_id=SFGK5YCWHFRVXF6QNHYP2CYJCZLSTQ76OYRVJPWWXQTF2KFSFJKA, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.341468438 (297h0m59.938239447s) [noise=true , address=45.91.116.4:4961 , piece_id=ZMUGQECUVPVHDNKN75XPBET5M2HFXXK4WMTP4V5KEK63JVZCX5EA, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.341520025 (297h0m59.938291058s) [noise=true , address=167.17.72.71:4815 , piece_id=CXGTYNNBILN3TT7I6Z7UI75F2ZCS4QUTLS6KC4FJ5IHS3XOYWDJQ, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.342232992 (297h0m59.939004023s) [noise=true , address=145.79.227.36:4524 , piece_id=3TAV3P55ITTHBAZEUUUTOAHJRGQXJXXVGZJP6AVR2ANVNOR3S54Q, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.342917368 (297h0m59.939688398s) [noise=true , address=193.149.164.71:4297 , piece_id=4IJPXLPCX3LKFOBOSRHK2DTNRKH67K7FXJUKQ3JPRWHS47ZFEIDA, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.343244957 (297h0m59.940015982s) [noise=true , address=145.79.227.33:4892 , piece_id=DQ6ALC3VOAGFIJI3IKR6I7QFYM22C66QWAAXMH6NXI2FIQEZNQYA, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.344270079 (297h0m59.941041117s) [noise=true , address=145.79.227.21:4497 , piece_id=SP6EKJB7W4QXF7OIBVKEFWCPOZFL5IRIEDVAGS2M4QJSO34CUVRQ, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.345025057 (297h0m59.941796084s) [noise=true , address=77.90.31.17:4793 , piece_id=2U4IBUA2GGUGDBXQGIGQKUK4ZR3L7CDKC6MTGIKAUME4F7MFZRLA, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.345115802 (297h0m59.941886836s) [noise=true , address=167.17.72.69:4731 , piece_id=VOEVIKKFJMC2Y3TPQIDMYUP37NLFU7KPRH4HRRSMQLWUKTSLOH2Q, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.345690600 (297h0m59.942461634s) [noise=true , address=77.90.31.13:4648 , piece_id=BPMB5CFOYHFT3BS4ORR6JF6BNFC2AOAJZOSPMG67NAMLGGP5XVEQ, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.346311592 (297h0m59.943082623s) [noise=true , address=79.127.205.234:4202 , piece_id=YY4LYWX26O6RIOBONESNKTUZEFZCRL4UUX3NEBQ53AJV4IEDG5MQ, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.346791142 (297h0m59.943562166s) [noise=true , address=45.91.116.10:4110 , piece_id=UYLMDCWJEJOHGICA37KNH7G5AQXNIPRH5NRYLBUUMGDRVSLTEYHQ, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.347094999 (297h0m59.94386607s) [noise=true , address=145.79.227.17:4987 , piece_id=PT3TWVB65JZFEWVH4IUZEU2N53FQ57QBRWVLWAUFP4UDLY3LWNLQ, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.347781281 (297h0m59.944552305s) [noise=true , address=79.127.219.46:4919 , piece_id=KK6HRIVBMYLIHRHGPQH3AWMNSVHTTIBESHDMJZA72T446RX6QKRQ, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.348263123 (297h0m59.945034137s) [noise=true , address=193.149.164.68:4033 , piece_id=JZPQKPYGO3W3RZZBITQ73X2XEPIBBSLGRMWIVOM4Z3GTV5UFKRTQ, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.348915505 (297h0m59.945686536s) [noise=true , address=145.79.227.7:4312 , piece_id=2FAL6E3RMBTBHZBXAJJVH2N5VGF5VYI3KNZXGNWCMWGBBPAUO2NQ, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.349366298 (297h0m59.946137337s) [noise=true , address=193.149.164.67:4007 , piece_id=HURQGH67I5B6V4YUV3PRZL3KSKN3XTSK3UMCHZRNT6L3W3PXXHPQ, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.429834098 (297h1m0.026605126s) [upload=332666]: Next returned nil segment Mar 18 20:01:02.429853411 (297h1m0.026624442s) [upload=332666]: Waiting for error group managing segments... Mar 18 20:01:02.446867680 (297h1m0.043638704s) [noise=true , address=45.91.116.10:4110 , piece_id=UYLMDCWJEJOHGICA37KNH7G5AQXNIPRH5NRYLBUUMGDRVSLTEYHQ, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil> Mar 18 20:01:02.451665165 (297h1m0.048436196s) [noise=true , address=45.91.116.4:4961 , piece_id=ZMUGQECUVPVHDNKN75XPBET5M2HFXXK4WMTP4V5KEK63JVZCX5EA, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil> Mar 18 20:01:02.452538886 (297h1m0.049309913s) [noise=true , address=77.90.31.17:4793 , piece_id=2U4IBUA2GGUGDBXQGIGQKUK4ZR3L7CDKC6MTGIKAUME4F7MFZRLA, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil> Mar 18 20:01:02.455597040 (297h1m0.052368067s) [noise=true , address=45.91.116.8:4061 , piece_id=6XFVKT4AAO7QN457OQMRKPI6R24TJTAN47UXLEVOEFQVOS6M6C3Q, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil> Mar 18 20:01:02.457848739 (297h1m0.054619766s) [noise=true , address=77.90.31.13:4648 , piece_id=BPMB5CFOYHFT3BS4ORR6JF6BNFC2AOAJZOSPMG67NAMLGGP5XVEQ, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil> Mar 18 20:01:02.461479709 (297h1m0.058250746s) [noise=true , address=79.127.219.46:4919 , piece_id=KK6HRIVBMYLIHRHGPQH3AWMNSVHTTIBESHDMJZA72T446RX6QKRQ, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil> Mar 18 20:01:02.676859466 (297h1m0.27363049s) [noise=true , address=167.17.72.69:4731 , piece_id=VOEVIKKFJMC2Y3TPQIDMYUP37NLFU7KPRH4HRRSMQLWUKTSLOH2Q, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil> Mar 18 20:01:02.909860743 (297h1m0.506631766s) [noise=true , address=79.127.205.234:4202 , piece_id=YY4LYWX26O6RIOBONESNKTUZEFZCRL4UUX3NEBQ53AJV4IEDG5MQ, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil> Mar 18 20:01:02.930427742 (297h1m0.527198754s) [noise=true , address=193.149.164.71:4297 , piece_id=4IJPXLPCX3LKFOBOSRHK2DTNRKH67K7FXJUKQ3JPRWHS47ZFEIDA, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil> Mar 18 20:01:02.944696889 (297h1m0.541467914s) [noise=true , address=193.149.164.67:4007 , piece_id=HURQGH67I5B6V4YUV3PRZL3KSKN3XTSK3UMCHZRNT6L3W3PXXHPQ, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil> Mar 18 20:01:02.951319183 (297h1m0.548090212s) [noise=true , address=193.149.164.68:4033 , piece_id=JZPQKPYGO3W3RZZBITQ73X2XEPIBBSLGRMWIVOM4Z3GTV5UFKRTQ, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil> Mar 18 20:01:02.975619436 (297h1m0.572390467s) [noise=true , address=79.127.205.233:4166 , piece_id=SFGK5YCWHFRVXF6QNHYP2CYJCZLSTQ76OYRVJPWWXQTF2KFSFJKA, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil> Mar 18 20:01:03.819918528 (297h1m1.416689564s) [noise=true , address=167.17.72.71:4815 , piece_id=CXGTYNNBILN3TT7I6Z7UI75F2ZCS4QUTLS6KC4FJ5IHS3XOYWDJQ, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil> Mar 18 20:01:59.307873420 (297h1m56.904644452s) [noise=true , address=145.79.227.7:4312 , piece_id=2FAL6E3RMBTBHZBXAJJVH2N5VGF5VYI3KNZXGNWCMWGBBPAUO2NQ, seg_pos={0 0}, upload=332666]: Done uploading piece. err: ecclient: upload failed (node:12AVp3qn5uwLPqYaZEwzcgoGMJhNga1BUh6bK5k2bxjWgZLQiiC, address:145.79.227.7:4312): protocol: rpc: dial tcp 145.79.227.7:4312: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:01:59.308026092 (297h1m56.904797124s) [noise=true , address=145.79.227.17:4987 , piece_id=PT3TWVB65JZFEWVH4IUZEU2N53FQ57QBRWVLWAUFP4UDLY3LWNLQ, seg_pos={0 0}, upload=332666]: Done uploading piece. err: ecclient: upload failed (node:1XsPVNGYUhKnXkQRfVfjyBRzyvhShSdsrnzXtwe4Xtf5MaHMvH, address:145.79.227.17:4987): protocol: rpc: dial tcp 145.79.227.17:4987: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:01:59.308195864 (297h1m56.904966894s) [noise=true , address=145.79.227.36:4524 , piece_id=3TAV3P55ITTHBAZEUUUTOAHJRGQXJXXVGZJP6AVR2ANVNOR3S54Q, seg_pos={0 0}, upload=332666]: Done uploading piece. err: ecclient: upload failed (node:1xfbCzjSwFDRrR1gPggeN8tNCLbRRWDVzCJ6mqhZsbVSe8d4s4, address:145.79.227.36:4524): protocol: rpc: dial tcp 145.79.227.36:4524: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:01:59.308316037 (297h1m56.90508706s) [noise=true , address=145.79.227.33:4892 , piece_id=DQ6ALC3VOAGFIJI3IKR6I7QFYM22C66QWAAXMH6NXI2FIQEZNQYA, seg_pos={0 0}, upload=332666]: Done uploading piece. err: ecclient: upload failed (node:12RHwFxcUjPsn12uHEAH3fBDJKv1v1tM8mRJXoFrZrG5djdswF5, address:145.79.227.33:4892): protocol: rpc: dial tcp 145.79.227.33:4892: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:01:59.308404403 (297h1m56.905175433s) [noise=true , address=145.79.227.21:4497 , piece_id=SP6EKJB7W4QXF7OIBVKEFWCPOZFL5IRIEDVAGS2M4QJSO34CUVRQ, seg_pos={0 0}, upload=332666]: Done uploading piece. err: ecclient: upload failed (node:12ZBe4we7VeFdthqRxyy1V9uQrA4r8ouNX68xqsk5bD26dkf2Gd, address:145.79.227.21:4497): protocol: context deadline exceeded; context deadline exceeded Mar 18 20:01:59.308437457 (297h1m56.90520847s) [seg_pos={0 0}, upload=332666]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 18 20:01:59.338288467 (297h1m56.93505949s) [upload=332666]: Done uploading segments... Mar 18 20:01:59.338335497 (297h1m56.935106523s) [upload=332666]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 17.320679415s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 56 out of 60 is failed (threshold=6) [XXXXXXX.XXX...XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 5.001703986s trace: https://jaeger.dev.storj.io/trace/989d3e1d4c4cb78 FAIL (2026-03-18 20:13:58 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001819843s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 9 out of 60 is failed (threshold=3) [............XXX..........X...X.........X....X....X.......X..] duration: 2.120383796s trace: https://jaeger.dev.storj.io/trace/24517a7cd40ab76d FAIL (2026-03-18 20:13:56 UTC) operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== oad Mar 18 20:12:28.252047067 (297h12m25.848818092s) [upload=332883]: Uploading segments... Mar 18 20:12:28.252323178 (297h12m25.84909421s) [upload=332883]: Got next segment. Inline: false Mar 18 20:12:28.252380536 (297h12m25.849151569s) [upload=332883]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 20:12:28.252388587 (297h12m25.849159622s) [upload=332883]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 18 20:12:31.413020979 (297h12m29.009792004s) [seg_pos={0 0}, upload=332883]: Begin upload segment... Mar 18 20:12:31.413109194 (297h12m29.009880217s) [seg_pos={0 0}, upload=332883]: Done begin upload segment. Mar 18 20:12:31.413182895 (297h12m29.009953924s) [noise=true , address=167.17.72.68:4653 , piece_id=6TO53KV4DBJ5HTWIQUZY2L6WVMB4PBPRWSZD6WHBLVFI5FUULYUQ, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.413918965 (297h12m29.010689999s) [noise=true , address=145.79.227.20:4454 , piece_id=F2WUAPOH2J3GQ7DMWLBJA44UGY5B3MMDS2U5ITXTOIPR4ELY5QYA, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.414450842 (297h12m29.011221869s) [noise=true , address=193.149.164.70:4111 , piece_id=H6GX5VVLRFNUAF2R7FRZ5GFQCZ2F57OL4GXIHWQJGZDOM3ST5DKQ, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.414901508 (297h12m29.011672532s) [noise=true , address=5.231.73.4:4002 , piece_id=3DWXH2Y6ZZNDHRFMJFL6EEMJN7WNDRKJ3ARFKN6UAGAHICNCBSDA, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.415342842 (297h12m29.012113865s) [noise=true , address=145.79.227.28:4750 , piece_id=KJVCFR2SJ5HAQKJUJP4CYPVCZQC6PBAQII6UGNYVKRODWRKENFQA, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.415582477 (297h12m29.01235351s) [noise=true , address=145.79.227.24:4615 , piece_id=ZSEUUMSYRSXH4AJ3CDZNFKXBZFT63AKVDFBFJ2RLAZK344GQ5AFA, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.415917210 (297h12m29.012688235s) [noise=true , address=193.149.164.68:4040 , piece_id=4NT224HIHI45JJNYH3ZCXQKDAMQHHYWY7SZPYQK5KN7JKNYFGI2A, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.416035983 (297h12m29.012807012s) [noise=true , address=193.149.164.67:4030 , piece_id=PYGFDMQLZ2SYL5NTWUGQGHWW52T2YYEGDKP5SUWQQUTVJSJZOO5A, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.416379117 (297h12m29.013150149s) [noise=true , address=45.91.116.5:4322 , piece_id=FN763EWEYGTBEXNHOT3FMKLW52DENMGPRBPBJJHNWQUNQ5JIN3XA, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.416442412 (297h12m29.013213436s) [noise=true , address=145.79.227.23:4564 , piece_id=3S5IXK7TKV2W3I635W2LQDH3PC6JRPHIVBK347CE7EYE4QQ5AGBQ, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.416786035 (297h12m29.013557058s) [noise=true , address=167.17.72.71:4808 , piece_id=JL5TJCWL454VA73PMWKZYBAUT6ZMRPEH55KJDSHEELC37HRVCO7Q, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.416946153 (297h12m29.013717177s) [noise=true , address=64.49.13.169:4705 , piece_id=FW7ICJDZJVOQR77HCGAWU7NKSBTWASW3BXH72WBGZHOFVPXA7XQQ, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.417084028 (297h12m29.013855051s) [noise=true , address=145.79.227.44:4202 , piece_id=UI7I4UPDOZCJ2MRPKV6FP6YPYNVSR5JWYDVFBO2TRFYQJWVWIUNQ, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.424729560 (297h12m29.021500589s) [noise=true , address=79.127.205.233:4156 , piece_id=5OZYLGXB7R6VMYGATW7GJD2CNAVO45PATTCFGNPWYZ3ZURYLZSQA, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.425355484 (297h12m29.022126513s) [noise=true , address=193.149.164.69:4077 , piece_id=GYZMBIAEBL4KBCSFA4H5JLWEB36NBN2BGVQ67A5G4ECUWY5BACXA, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.429910441 (297h12m29.026681471s) [noise=true , address=79.127.219.46:4918 , piece_id=WC2RL2PMO4EVATTOPZBWFHOZE6CBGQ2ROGJ44PDQUPQKYXBFKY6Q, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.431104280 (297h12m29.027875311s) [noise=true , address=79.127.219.45:4883 , piece_id=7DXMZGFUW2QAI2OID7WSSMJG6B3WW52HCMRWX2LS6YFEWAFMZZNQ, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.432173523 (297h12m29.02894455s) [noise=true , address=193.149.164.71:4300 , piece_id=4FI5PVS627OOB2M45U4QTOXGBJHNOZMUDDCFAW57W7PBAAQSJC4A, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.478900188 (297h12m29.075671211s) [upload=332883]: Next returned nil segment Mar 18 20:12:31.478914031 (297h12m29.075685054s) [upload=332883]: Waiting for error group managing segments... Mar 18 20:12:31.512891809 (297h12m29.109662832s) [noise=true , address=79.127.219.46:4918 , piece_id=WC2RL2PMO4EVATTOPZBWFHOZE6CBGQ2ROGJ44PDQUPQKYXBFKY6Q, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil> Mar 18 20:12:31.529013012 (297h12m29.125784038s) [noise=true , address=45.91.116.5:4322 , piece_id=FN763EWEYGTBEXNHOT3FMKLW52DENMGPRBPBJJHNWQUNQ5JIN3XA, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil> Mar 18 20:12:31.564707355 (297h12m29.161478379s) [noise=true , address=79.127.219.45:4883 , piece_id=7DXMZGFUW2QAI2OID7WSSMJG6B3WW52HCMRWX2LS6YFEWAFMZZNQ, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil> Mar 18 20:12:31.611013762 (297h12m29.207784793s) [noise=true , address=5.231.73.4:4002 , piece_id=3DWXH2Y6ZZNDHRFMJFL6EEMJN7WNDRKJ3ARFKN6UAGAHICNCBSDA, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil> Mar 18 20:12:31.713823345 (297h12m29.310594381s) [noise=true , address=167.17.72.68:4653 , piece_id=6TO53KV4DBJ5HTWIQUZY2L6WVMB4PBPRWSZD6WHBLVFI5FUULYUQ, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil> Mar 18 20:12:31.997071787 (297h12m29.59384282s) [noise=true , address=79.127.205.233:4156 , piece_id=5OZYLGXB7R6VMYGATW7GJD2CNAVO45PATTCFGNPWYZ3ZURYLZSQA, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil> Mar 18 20:12:31.999264198 (297h12m29.59603524s) [noise=true , address=193.149.164.70:4111 , piece_id=H6GX5VVLRFNUAF2R7FRZ5GFQCZ2F57OL4GXIHWQJGZDOM3ST5DKQ, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil> Mar 18 20:12:32.008593285 (297h12m29.605364328s) [noise=true , address=193.149.164.71:4300 , piece_id=4FI5PVS627OOB2M45U4QTOXGBJHNOZMUDDCFAW57W7PBAAQSJC4A, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil> Mar 18 20:12:32.011203033 (297h12m29.607974056s) [noise=true , address=193.149.164.68:4040 , piece_id=4NT224HIHI45JJNYH3ZCXQKDAMQHHYWY7SZPYQK5KN7JKNYFGI2A, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil> Mar 18 20:12:32.038310240 (297h12m29.635081269s) [noise=true , address=193.149.164.67:4030 , piece_id=PYGFDMQLZ2SYL5NTWUGQGHWW52T2YYEGDKP5SUWQQUTVJSJZOO5A, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil> Mar 18 20:12:32.091831027 (297h12m29.68860206s) [noise=true , address=64.49.13.169:4705 , piece_id=FW7ICJDZJVOQR77HCGAWU7NKSBTWASW3BXH72WBGZHOFVPXA7XQQ, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil> Mar 18 20:12:32.408908738 (297h12m30.005679764s) [noise=true , address=193.149.164.69:4077 , piece_id=GYZMBIAEBL4KBCSFA4H5JLWEB36NBN2BGVQ67A5G4ECUWY5BACXA, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil> Mar 18 20:12:32.489564296 (297h12m30.086335307s) [noise=true , address=167.17.72.71:4808 , piece_id=JL5TJCWL454VA73PMWKZYBAUT6ZMRPEH55KJDSHEELC37HRVCO7Q, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil> Mar 18 20:13:28.204470030 (297h13m25.801241064s) [noise=true , address=145.79.227.23:4564 , piece_id=3S5IXK7TKV2W3I635W2LQDH3PC6JRPHIVBK347CE7EYE4QQ5AGBQ, seg_pos={0 0}, upload=332883]: Done uploading piece. err: ecclient: upload failed (node:121WW4uBQe8bGQKkasQQKT3k4EAdzncsG1ebNvgKf8S5SyNttCc, address:145.79.227.23:4564): protocol: rpc: dial tcp 145.79.227.23:4564: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:13:28.205066456 (297h13m25.801837482s) [noise=true , address=145.79.227.28:4750 , piece_id=KJVCFR2SJ5HAQKJUJP4CYPVCZQC6PBAQII6UGNYVKRODWRKENFQA, seg_pos={0 0}, upload=332883]: Done uploading piece. err: ecclient: upload failed (node:12UP8PxqeK7t24PXjopH1nziHYQFnsxhkamSHz8dmwcbuDxkAF6, address:145.79.227.28:4750): protocol: rpc: dial tcp 145.79.227.28:4750: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:13:28.205099998 (297h13m25.80187103s) [noise=true , address=145.79.227.44:4202 , piece_id=UI7I4UPDOZCJ2MRPKV6FP6YPYNVSR5JWYDVFBO2TRFYQJWVWIUNQ, seg_pos={0 0}, upload=332883]: Done uploading piece. err: ecclient: upload failed (node:121ogdaiVGybXEVsS8LDJuJRYXGxb1LCNnGPuwbde96eSEzkTW4, address:145.79.227.44:4202): protocol: rpc: dial tcp 145.79.227.44:4202: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:13:28.205340404 (297h13m25.802111436s) [noise=true , address=145.79.227.24:4615 , piece_id=ZSEUUMSYRSXH4AJ3CDZNFKXBZFT63AKVDFBFJ2RLAZK344GQ5AFA, seg_pos={0 0}, upload=332883]: Done uploading piece. err: ecclient: upload failed (node:12NJ5w8pAWCudDA9Xs4ES7b2evnbscesd682wJDh2gMMZ3Jf2zc, address:145.79.227.24:4615): protocol: rpc: dial tcp 145.79.227.24:4615: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:13:28.205593210 (297h13m25.802364235s) [noise=true , address=145.79.227.20:4454 , piece_id=F2WUAPOH2J3GQ7DMWLBJA44UGY5B3MMDS2U5ITXTOIPR4ELY5QYA, seg_pos={0 0}, upload=332883]: Done uploading piece. err: ecclient: upload failed (node:1SafkJPqKHLwMaqyZdLq3yJ1xdjqDtWHFrUwynmegUVwEC7y5J, address:145.79.227.20:4454): protocol: context deadline exceeded; context deadline exceeded Mar 18 20:13:28.205633541 (297h13m25.802404553s) [seg_pos={0 0}, upload=332883]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 18 20:13:28.228774364 (297h13m25.825545388s) [upload=332883]: Done uploading segments... Mar 18 20:13:28.228821883 (297h13m25.825592906s) [upload=332883]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 2.120412101s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 9 out of 60 is failed (threshold=3) [.............XXX..........X...X.........X....X....X.......X.] duration: 13.932292111s trace: https://jaeger.dev.storj.io/trace/24517a7cd40ab76d FAIL (2026-03-18 20:13:42 UTC) operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== oad Mar 18 20:12:28.252047067 (297h12m25.848818092s) [upload=332883]: Uploading segments... Mar 18 20:12:28.252323178 (297h12m25.84909421s) [upload=332883]: Got next segment. Inline: false Mar 18 20:12:28.252380536 (297h12m25.849151569s) [upload=332883]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 20:12:28.252388587 (297h12m25.849159622s) [upload=332883]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 18 20:12:31.413020979 (297h12m29.009792004s) [seg_pos={0 0}, upload=332883]: Begin upload segment... Mar 18 20:12:31.413109194 (297h12m29.009880217s) [seg_pos={0 0}, upload=332883]: Done begin upload segment. Mar 18 20:12:31.413182895 (297h12m29.009953924s) [noise=true , address=167.17.72.68:4653 , piece_id=6TO53KV4DBJ5HTWIQUZY2L6WVMB4PBPRWSZD6WHBLVFI5FUULYUQ, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.413918965 (297h12m29.010689999s) [noise=true , address=145.79.227.20:4454 , piece_id=F2WUAPOH2J3GQ7DMWLBJA44UGY5B3MMDS2U5ITXTOIPR4ELY5QYA, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.414450842 (297h12m29.011221869s) [noise=true , address=193.149.164.70:4111 , piece_id=H6GX5VVLRFNUAF2R7FRZ5GFQCZ2F57OL4GXIHWQJGZDOM3ST5DKQ, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.414901508 (297h12m29.011672532s) [noise=true , address=5.231.73.4:4002 , piece_id=3DWXH2Y6ZZNDHRFMJFL6EEMJN7WNDRKJ3ARFKN6UAGAHICNCBSDA, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.415342842 (297h12m29.012113865s) [noise=true , address=145.79.227.28:4750 , piece_id=KJVCFR2SJ5HAQKJUJP4CYPVCZQC6PBAQII6UGNYVKRODWRKENFQA, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.415582477 (297h12m29.01235351s) [noise=true , address=145.79.227.24:4615 , piece_id=ZSEUUMSYRSXH4AJ3CDZNFKXBZFT63AKVDFBFJ2RLAZK344GQ5AFA, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.415917210 (297h12m29.012688235s) [noise=true , address=193.149.164.68:4040 , piece_id=4NT224HIHI45JJNYH3ZCXQKDAMQHHYWY7SZPYQK5KN7JKNYFGI2A, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.416035983 (297h12m29.012807012s) [noise=true , address=193.149.164.67:4030 , piece_id=PYGFDMQLZ2SYL5NTWUGQGHWW52T2YYEGDKP5SUWQQUTVJSJZOO5A, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.416379117 (297h12m29.013150149s) [noise=true , address=45.91.116.5:4322 , piece_id=FN763EWEYGTBEXNHOT3FMKLW52DENMGPRBPBJJHNWQUNQ5JIN3XA, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.416442412 (297h12m29.013213436s) [noise=true , address=145.79.227.23:4564 , piece_id=3S5IXK7TKV2W3I635W2LQDH3PC6JRPHIVBK347CE7EYE4QQ5AGBQ, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.416786035 (297h12m29.013557058s) [noise=true , address=167.17.72.71:4808 , piece_id=JL5TJCWL454VA73PMWKZYBAUT6ZMRPEH55KJDSHEELC37HRVCO7Q, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.416946153 (297h12m29.013717177s) [noise=true , address=64.49.13.169:4705 , piece_id=FW7ICJDZJVOQR77HCGAWU7NKSBTWASW3BXH72WBGZHOFVPXA7XQQ, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.417084028 (297h12m29.013855051s) [noise=true , address=145.79.227.44:4202 , piece_id=UI7I4UPDOZCJ2MRPKV6FP6YPYNVSR5JWYDVFBO2TRFYQJWVWIUNQ, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.424729560 (297h12m29.021500589s) [noise=true , address=79.127.205.233:4156 , piece_id=5OZYLGXB7R6VMYGATW7GJD2CNAVO45PATTCFGNPWYZ3ZURYLZSQA, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.425355484 (297h12m29.022126513s) [noise=true , address=193.149.164.69:4077 , piece_id=GYZMBIAEBL4KBCSFA4H5JLWEB36NBN2BGVQ67A5G4ECUWY5BACXA, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.429910441 (297h12m29.026681471s) [noise=true , address=79.127.219.46:4918 , piece_id=WC2RL2PMO4EVATTOPZBWFHOZE6CBGQ2ROGJ44PDQUPQKYXBFKY6Q, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.431104280 (297h12m29.027875311s) [noise=true , address=79.127.219.45:4883 , piece_id=7DXMZGFUW2QAI2OID7WSSMJG6B3WW52HCMRWX2LS6YFEWAFMZZNQ, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.432173523 (297h12m29.02894455s) [noise=true , address=193.149.164.71:4300 , piece_id=4FI5PVS627OOB2M45U4QTOXGBJHNOZMUDDCFAW57W7PBAAQSJC4A, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.478900188 (297h12m29.075671211s) [upload=332883]: Next returned nil segment Mar 18 20:12:31.478914031 (297h12m29.075685054s) [upload=332883]: Waiting for error group managing segments... Mar 18 20:12:31.512891809 (297h12m29.109662832s) [noise=true , address=79.127.219.46:4918 , piece_id=WC2RL2PMO4EVATTOPZBWFHOZE6CBGQ2ROGJ44PDQUPQKYXBFKY6Q, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil> Mar 18 20:12:31.529013012 (297h12m29.125784038s) [noise=true , address=45.91.116.5:4322 , piece_id=FN763EWEYGTBEXNHOT3FMKLW52DENMGPRBPBJJHNWQUNQ5JIN3XA, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil> Mar 18 20:12:31.564707355 (297h12m29.161478379s) [noise=true , address=79.127.219.45:4883 , piece_id=7DXMZGFUW2QAI2OID7WSSMJG6B3WW52HCMRWX2LS6YFEWAFMZZNQ, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil> Mar 18 20:12:31.611013762 (297h12m29.207784793s) [noise=true , address=5.231.73.4:4002 , piece_id=3DWXH2Y6ZZNDHRFMJFL6EEMJN7WNDRKJ3ARFKN6UAGAHICNCBSDA, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil> Mar 18 20:12:31.713823345 (297h12m29.310594381s) [noise=true , address=167.17.72.68:4653 , piece_id=6TO53KV4DBJ5HTWIQUZY2L6WVMB4PBPRWSZD6WHBLVFI5FUULYUQ, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil> Mar 18 20:12:31.997071787 (297h12m29.59384282s) [noise=true , address=79.127.205.233:4156 , piece_id=5OZYLGXB7R6VMYGATW7GJD2CNAVO45PATTCFGNPWYZ3ZURYLZSQA, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil> Mar 18 20:12:31.999264198 (297h12m29.59603524s) [noise=true , address=193.149.164.70:4111 , piece_id=H6GX5VVLRFNUAF2R7FRZ5GFQCZ2F57OL4GXIHWQJGZDOM3ST5DKQ, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil> Mar 18 20:12:32.008593285 (297h12m29.605364328s) [noise=true , address=193.149.164.71:4300 , piece_id=4FI5PVS627OOB2M45U4QTOXGBJHNOZMUDDCFAW57W7PBAAQSJC4A, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil> Mar 18 20:12:32.011203033 (297h12m29.607974056s) [noise=true , address=193.149.164.68:4040 , piece_id=4NT224HIHI45JJNYH3ZCXQKDAMQHHYWY7SZPYQK5KN7JKNYFGI2A, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil> Mar 18 20:12:32.038310240 (297h12m29.635081269s) [noise=true , address=193.149.164.67:4030 , piece_id=PYGFDMQLZ2SYL5NTWUGQGHWW52T2YYEGDKP5SUWQQUTVJSJZOO5A, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil> Mar 18 20:12:32.091831027 (297h12m29.68860206s) [noise=true , address=64.49.13.169:4705 , piece_id=FW7ICJDZJVOQR77HCGAWU7NKSBTWASW3BXH72WBGZHOFVPXA7XQQ, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil> Mar 18 20:12:32.408908738 (297h12m30.005679764s) [noise=true , address=193.149.164.69:4077 , piece_id=GYZMBIAEBL4KBCSFA4H5JLWEB36NBN2BGVQ67A5G4ECUWY5BACXA, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil> Mar 18 20:12:32.489564296 (297h12m30.086335307s) [noise=true , address=167.17.72.71:4808 , piece_id=JL5TJCWL454VA73PMWKZYBAUT6ZMRPEH55KJDSHEELC37HRVCO7Q, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil> Mar 18 20:13:28.204470030 (297h13m25.801241064s) [noise=true , address=145.79.227.23:4564 , piece_id=3S5IXK7TKV2W3I635W2LQDH3PC6JRPHIVBK347CE7EYE4QQ5AGBQ, seg_pos={0 0}, upload=332883]: Done uploading piece. err: ecclient: upload failed (node:121WW4uBQe8bGQKkasQQKT3k4EAdzncsG1ebNvgKf8S5SyNttCc, address:145.79.227.23:4564): protocol: rpc: dial tcp 145.79.227.23:4564: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:13:28.205066456 (297h13m25.801837482s) [noise=true , address=145.79.227.28:4750 , piece_id=KJVCFR2SJ5HAQKJUJP4CYPVCZQC6PBAQII6UGNYVKRODWRKENFQA, seg_pos={0 0}, upload=332883]: Done uploading piece. err: ecclient: upload failed (node:12UP8PxqeK7t24PXjopH1nziHYQFnsxhkamSHz8dmwcbuDxkAF6, address:145.79.227.28:4750): protocol: rpc: dial tcp 145.79.227.28:4750: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:13:28.205099998 (297h13m25.80187103s) [noise=true , address=145.79.227.44:4202 , piece_id=UI7I4UPDOZCJ2MRPKV6FP6YPYNVSR5JWYDVFBO2TRFYQJWVWIUNQ, seg_pos={0 0}, upload=332883]: Done uploading piece. err: ecclient: upload failed (node:121ogdaiVGybXEVsS8LDJuJRYXGxb1LCNnGPuwbde96eSEzkTW4, address:145.79.227.44:4202): protocol: rpc: dial tcp 145.79.227.44:4202: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:13:28.205340404 (297h13m25.802111436s) [noise=true , address=145.79.227.24:4615 , piece_id=ZSEUUMSYRSXH4AJ3CDZNFKXBZFT63AKVDFBFJ2RLAZK344GQ5AFA, seg_pos={0 0}, upload=332883]: Done uploading piece. err: ecclient: upload failed (node:12NJ5w8pAWCudDA9Xs4ES7b2evnbscesd682wJDh2gMMZ3Jf2zc, address:145.79.227.24:4615): protocol: rpc: dial tcp 145.79.227.24:4615: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:13:28.205593210 (297h13m25.802364235s) [noise=true , address=145.79.227.20:4454 , piece_id=F2WUAPOH2J3GQ7DMWLBJA44UGY5B3MMDS2U5ITXTOIPR4ELY5QYA, seg_pos={0 0}, upload=332883]: Done uploading piece. err: ecclient: upload failed (node:1SafkJPqKHLwMaqyZdLq3yJ1xdjqDtWHFrUwynmegUVwEC7y5J, address:145.79.227.20:4454): protocol: context deadline exceeded; context deadline exceeded Mar 18 20:13:28.205633541 (297h13m25.802404553s) [seg_pos={0 0}, upload=332883]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 18 20:13:28.228774364 (297h13m25.825545388s) [upload=332883]: Done uploading segments... Mar 18 20:13:28.228821883 (297h13m25.825592906s) [upload=332883]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 13.932310646s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 47 out of 60 is failed (threshold=3) [.......XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.XXXXXX...XX.X.X] duration: 30.022680731s trace: https://jaeger.dev.storj.io/trace/1af74945b838a5e5 FAIL (2026-03-18 20:13:35 UTC) operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Mar 18 20:13:05.456861574 (297h13m3.053632597s) [upload=332894]: Starting upload Mar 18 20:13:05.456950013 (297h13m3.053721045s) [upload=332894]: Done starting upload Mar 18 20:13:05.457043990 (297h13m3.053815021s) [upload=332894]: Uploading segments... Mar 18 20:13:05.457238283 (297h13m3.054009309s) [upload=332894]: Got next segment. Inline: false Mar 18 20:13:05.457299354 (297h13m3.054070377s) [upload=332894]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 20:13:05.457305455 (297h13m3.054076482s) [upload=332894]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 18 20:13:08.990260816 (297h13m6.587031848s) [seg_pos={0 0}, upload=332894]: Begin upload segment... Mar 18 20:13:08.990347124 (297h13m6.587118148s) [seg_pos={0 0}, upload=332894]: Done begin upload segment. Mar 18 20:13:08.990426032 (297h13m6.587197065s) [noise=true , address=145.79.227.24:4586 , piece_id=DIKCGGNBXTAADHUSX2TG2EW3Q4VHRIMJZNYFEQNILFAGJGEBCUFQ, seg_pos={0 0}, upload=332894]: Uploading piece... Mar 18 20:13:08.991153679 (297h13m6.587924709s) [noise=true , address=45.91.116.9:4264 , piece_id=3UZ4HBPA7Z4SVNJDFLYVDIAANP7BONTN3GDKLBNOBUDOPZBDW7VA, seg_pos={0 0}, upload=332894]: Uploading piece... Mar 18 20:13:08.991640593 (297h13m6.588411671s) [noise=true , address=77.90.31.14:4665 , piece_id=6QZDQPXMP4JEGIKGWOTS33HW5XGWCWCYYUKTXD6FVOQOJ6EIPBMA, seg_pos={0 0}, upload=332894]: Uploading piece... Mar 18 20:13:08.992051467 (297h13m6.58882251s) [noise=true , address=45.91.116.7:4012 , piece_id=3EWHSNNDSWVYTLS2VWNL27WBFSIBARZYFGB5MWVTWNTGIFZHX3RQ, seg_pos={0 0}, upload=332894]: Uploading piece... Mar 18 20:13:08.992232663 (297h13m6.589003698s) [noise=true , address=79.127.219.41:4929 , piece_id=NV7BGUGRNJDN53PVRENITEAH6DSSAUKGUINVRKAJ362UWB6AANVA, seg_pos={0 0}, upload=332894]: Uploading piece... Mar 18 20:13:08.992465615 (297h13m6.589236641s) [noise=true , address=79.127.219.45:4853 , piece_id=4QKELBNT7JAJTCKGTW24QF7WVTDX554DMBGUQV6GLCB2EDU6YIHA, seg_pos={0 0}, upload=332894]: Uploading piece... Mar 18 20:13:08.992751973 (297h13m6.589523009s) [noise=true , address=145.79.227.10:4406 , piece_id=KZ6GLRUODD3MOHFH4LTAGQCN6SIU66ODKDHK2G7BC6XCC4QWZHAQ, seg_pos={0 0}, upload=332894]: Uploading piece... Mar 18 20:13:08.992824046 (297h13m6.589595075s) [noise=true , address=77.90.31.12:4589 , piece_id=LDRH74PYHQSSNFUUTKRMZT4BB2FJG5LEJODQFIVSACK4PKR226TQ, seg_pos={0 0}, upload=332894]: Uploading piece... Mar 18 20:13:08.993231933 (297h13m6.590002966s) [noise=true , address=145.79.227.37:4570 , piece_id=QLO2E2XN24LL6GFQSFWXIDYD4FX3N5N4DOCSJDB4YKQ7R4VHW3MQ, seg_pos={0 0}, upload=332894]: Uploading piece... Mar 18 20:13:08.993251639 (297h13m6.590022672s) [noise=true , address=77.90.31.17:4770 , piece_id=6T57PGMTVZSGIXDZT5TXD76T3EG5YZL6F4WPQTG4AOWLFHS3G7FA, seg_pos={0 0}, upload=332894]: Uploading piece... Mar 18 20:13:08.993717339 (297h13m6.590488365s) [noise=true , address=45.91.116.12:4228 , piece_id=XC5JRWALBRFT54DFT3H476KL24A3EK47TNY2JWOFQHRFJVGL5KRQ, seg_pos={0 0}, upload=332894]: Uploading piece... Mar 18 20:13:08.993752369 (297h13m6.590523402s) [noise=true , address=145.79.227.42:4124 , piece_id=FFFT3UFNI7VSND34XR5ASPVFYC3TKCH6HQ6N27DWLE42NPXYC72A, seg_pos={0 0}, upload=332894]: Uploading piece... Mar 18 20:13:08.994108809 (297h13m6.590879832s) [noise=true , address=77.90.31.21:4880 , piece_id=GBTHEFLFZ26LHRSMXC5YOETI3QFHYTOFG26Q3XHNMWVLMC327NAA, seg_pos={0 0}, upload=332894]: Uploading piece... Mar 18 20:13:08.994262483 (297h13m6.591033513s) [noise=true , address=45.91.116.8:4049 , piece_id=47QSQSEYYLOMQRIDNGXRJTVZHIRBI3QABSINLNRPGP3CVLN3OKWA, seg_pos={0 0}, upload=332894]: Uploading piece... Mar 18 20:13:08.994503826 (297h13m6.591274851s) [noise=true , address=145.79.227.46:4606 , piece_id=EZ5LBWUNO54QLXDFW2DK3WYWCCA33RL5MV4B54PB45K7AZH4RPBQ, seg_pos={0 0}, upload=332894]: Uploading piece... Mar 18 20:13:08.995347041 (297h13m6.592118086s) [noise=true , address=45.91.116.14:4191 , piece_id=44ZW6CW72XINCJRR37R5T2PZKMET7EF76UB3G4BFWVURGUBA6J2A, seg_pos={0 0}, upload=332894]: Uploading piece... Mar 18 20:13:09.063915716 (297h13m6.660686749s) [upload=332894]: Next returned nil segment Mar 18 20:13:09.063934321 (297h13m6.660705354s) [upload=332894]: Waiting for error group managing segments... Mar 18 20:13:09.069407892 (297h13m6.666178938s) [noise=true , address=45.91.116.9:4264 , piece_id=3UZ4HBPA7Z4SVNJDFLYVDIAANP7BONTN3GDKLBNOBUDOPZBDW7VA, seg_pos={0 0}, upload=332894]: Done uploading piece. err: <nil> Mar 18 20:13:09.070987229 (297h13m6.667758261s) [noise=true , address=77.90.31.12:4589 , piece_id=LDRH74PYHQSSNFUUTKRMZT4BB2FJG5LEJODQFIVSACK4PKR226TQ, seg_pos={0 0}, upload=332894]: Done uploading piece. err: <nil> Mar 18 20:13:09.071184641 (297h13m6.667955716s) [noise=true , address=77.90.31.21:4880 , piece_id=GBTHEFLFZ26LHRSMXC5YOETI3QFHYTOFG26Q3XHNMWVLMC327NAA, seg_pos={0 0}, upload=332894]: Done uploading piece. err: <nil> Mar 18 20:13:09.071749303 (297h13m6.668520327s) [noise=true , address=77.90.31.17:4770 , piece_id=6T57PGMTVZSGIXDZT5TXD76T3EG5YZL6F4WPQTG4AOWLFHS3G7FA, seg_pos={0 0}, upload=332894]: Done uploading piece. err: <nil> Mar 18 20:13:09.072027035 (297h13m6.668798058s) [noise=true , address=45.91.116.14:4191 , piece_id=44ZW6CW72XINCJRR37R5T2PZKMET7EF76UB3G4BFWVURGUBA6J2A, seg_pos={0 0}, upload=332894]: Done uploading piece. err: <nil> Mar 18 20:13:09.072479034 (297h13m6.669250064s) [noise=true , address=79.127.219.41:4929 , piece_id=NV7BGUGRNJDN53PVRENITEAH6DSSAUKGUINVRKAJ362UWB6AANVA, seg_pos={0 0}, upload=332894]: Done uploading piece. err: <nil> Mar 18 20:13:09.072573023 (297h13m6.669344046s) [noise=true , address=77.90.31.14:4665 , piece_id=6QZDQPXMP4JEGIKGWOTS33HW5XGWCWCYYUKTXD6FVOQOJ6EIPBMA, seg_pos={0 0}, upload=332894]: Done uploading piece. err: <nil> Mar 18 20:13:09.072984382 (297h13m6.669755405s) [noise=true , address=45.91.116.8:4049 , piece_id=47QSQSEYYLOMQRIDNGXRJTVZHIRBI3QABSINLNRPGP3CVLN3OKWA, seg_pos={0 0}, upload=332894]: Done uploading piece. err: <nil> Mar 18 20:13:09.073172528 (297h13m6.669943558s) [noise=true , address=79.127.219.45:4853 , piece_id=4QKELBNT7JAJTCKGTW24QF7WVTDX554DMBGUQV6GLCB2EDU6YIHA, seg_pos={0 0}, upload=332894]: Done uploading piece. err: <nil> Mar 18 20:13:09.073417780 (297h13m6.670188805s) [noise=true , address=45.91.116.12:4228 , piece_id=XC5JRWALBRFT54DFT3H476KL24A3EK47TNY2JWOFQHRFJVGL5KRQ, seg_pos={0 0}, upload=332894]: Done uploading piece. err: <nil> Mar 18 20:13:09.089295035 (297h13m6.686066062s) [noise=true , address=45.91.116.7:4012 , piece_id=3EWHSNNDSWVYTLS2VWNL27WBFSIBARZYFGB5MWVTWNTGIFZHX3RQ, seg_pos={0 0}, upload=332894]: Done uploading piece. err: <nil> Mar 18 20:13:35.420342938 (297h13m33.01711397s) [noise=true , address=145.79.227.37:4570 , piece_id=QLO2E2XN24LL6GFQSFWXIDYD4FX3N5N4DOCSJDB4YKQ7R4VHW3MQ, seg_pos={0 0}, upload=332894]: Done uploading piece. err: ecclient: upload failed (node:1q5evYAfULsn19ddsAvSzJwxiDd8wsWbjP6gpcRkuU1b1q57eJ, address:145.79.227.37:4570): protocol: rpc: dial tcp 145.79.227.37:4570: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:13:35.420363122 (297h13m33.017134152s) [noise=true , address=145.79.227.24:4586 , piece_id=DIKCGGNBXTAADHUSX2TG2EW3Q4VHRIMJZNYFEQNILFAGJGEBCUFQ, seg_pos={0 0}, upload=332894]: Done uploading piece. err: ecclient: upload failed (node:1mkxgR2oLXcTdyCsxmR9Q1an9ZeLgkTSHRCtR4ZJqKbsStNEMh, address:145.79.227.24:4586): protocol: rpc: dial tcp 145.79.227.24:4586: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:13:35.420528457 (297h13m33.01729949s) [noise=true , address=145.79.227.10:4406 , piece_id=KZ6GLRUODD3MOHFH4LTAGQCN6SIU66ODKDHK2G7BC6XCC4QWZHAQ, seg_pos={0 0}, upload=332894]: Done uploading piece. err: ecclient: upload failed (node:1KrucHjbKjhBZEUGd2p8cmfyCwm6rx4BVwwVhkpgnEv9EQiJRB, address:145.79.227.10:4406): protocol: context deadline exceeded; context deadline exceeded Mar 18 20:13:35.420620171 (297h13m33.017391205s) [noise=true , address=145.79.227.42:4124 , piece_id=FFFT3UFNI7VSND34XR5ASPVFYC3TKCH6HQ6N27DWLE42NPXYC72A, seg_pos={0 0}, upload=332894]: Done uploading piece. err: ecclient: upload failed (node:12kz79BpzkFLTxbgi2Tjvz4dStT7MtrQ4GmX6bFnYw8v45jBaSy, address:145.79.227.42:4124): protocol: context deadline exceeded; context deadline exceeded Mar 18 20:13:35.420674996 (297h13m33.017446025s) [noise=true , address=145.79.227.46:4606 , piece_id=EZ5LBWUNO54QLXDFW2DK3WYWCCA33RL5MV4B54PB45K7AZH4RPBQ, seg_pos={0 0}, upload=332894]: Done uploading piece. err: ecclient: upload failed (node:12SMRAQn7BaX1Fzpvou2JHaiLko9fcQLdYggaVcjRihU516CbRh, address:145.79.227.46:4606): protocol: context deadline exceeded; context deadline exceeded Mar 18 20:13:35.420701486 (297h13m33.017472501s) [seg_pos={0 0}, upload=332894]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Mar 18 20:13:35.440412621 (297h13m33.037183673s) [upload=332894]: Done uploading segments... Mar 18 20:13:35.440460420 (297h13m33.037231451s) [upload=332894]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 30.022720936s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 9 out of 60 is failed (threshold=3) [..............XXX..........X...X.........X....X....X.......X] duration: 1m0.027335955s trace: https://jaeger.dev.storj.io/trace/24517a7cd40ab76d FAIL (2026-03-18 20:13:28 UTC) operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== oad Mar 18 20:12:28.252047067 (297h12m25.848818092s) [upload=332883]: Uploading segments... Mar 18 20:12:28.252323178 (297h12m25.84909421s) [upload=332883]: Got next segment. Inline: false Mar 18 20:12:28.252380536 (297h12m25.849151569s) [upload=332883]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 20:12:28.252388587 (297h12m25.849159622s) [upload=332883]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 18 20:12:31.413020979 (297h12m29.009792004s) [seg_pos={0 0}, upload=332883]: Begin upload segment... Mar 18 20:12:31.413109194 (297h12m29.009880217s) [seg_pos={0 0}, upload=332883]: Done begin upload segment. Mar 18 20:12:31.413182895 (297h12m29.009953924s) [noise=true , address=167.17.72.68:4653 , piece_id=6TO53KV4DBJ5HTWIQUZY2L6WVMB4PBPRWSZD6WHBLVFI5FUULYUQ, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.413918965 (297h12m29.010689999s) [noise=true , address=145.79.227.20:4454 , piece_id=F2WUAPOH2J3GQ7DMWLBJA44UGY5B3MMDS2U5ITXTOIPR4ELY5QYA, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.414450842 (297h12m29.011221869s) [noise=true , address=193.149.164.70:4111 , piece_id=H6GX5VVLRFNUAF2R7FRZ5GFQCZ2F57OL4GXIHWQJGZDOM3ST5DKQ, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.414901508 (297h12m29.011672532s) [noise=true , address=5.231.73.4:4002 , piece_id=3DWXH2Y6ZZNDHRFMJFL6EEMJN7WNDRKJ3ARFKN6UAGAHICNCBSDA, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.415342842 (297h12m29.012113865s) [noise=true , address=145.79.227.28:4750 , piece_id=KJVCFR2SJ5HAQKJUJP4CYPVCZQC6PBAQII6UGNYVKRODWRKENFQA, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.415582477 (297h12m29.01235351s) [noise=true , address=145.79.227.24:4615 , piece_id=ZSEUUMSYRSXH4AJ3CDZNFKXBZFT63AKVDFBFJ2RLAZK344GQ5AFA, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.415917210 (297h12m29.012688235s) [noise=true , address=193.149.164.68:4040 , piece_id=4NT224HIHI45JJNYH3ZCXQKDAMQHHYWY7SZPYQK5KN7JKNYFGI2A, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.416035983 (297h12m29.012807012s) [noise=true , address=193.149.164.67:4030 , piece_id=PYGFDMQLZ2SYL5NTWUGQGHWW52T2YYEGDKP5SUWQQUTVJSJZOO5A, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.416379117 (297h12m29.013150149s) [noise=true , address=45.91.116.5:4322 , piece_id=FN763EWEYGTBEXNHOT3FMKLW52DENMGPRBPBJJHNWQUNQ5JIN3XA, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.416442412 (297h12m29.013213436s) [noise=true , address=145.79.227.23:4564 , piece_id=3S5IXK7TKV2W3I635W2LQDH3PC6JRPHIVBK347CE7EYE4QQ5AGBQ, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.416786035 (297h12m29.013557058s) [noise=true , address=167.17.72.71:4808 , piece_id=JL5TJCWL454VA73PMWKZYBAUT6ZMRPEH55KJDSHEELC37HRVCO7Q, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.416946153 (297h12m29.013717177s) [noise=true , address=64.49.13.169:4705 , piece_id=FW7ICJDZJVOQR77HCGAWU7NKSBTWASW3BXH72WBGZHOFVPXA7XQQ, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.417084028 (297h12m29.013855051s) [noise=true , address=145.79.227.44:4202 , piece_id=UI7I4UPDOZCJ2MRPKV6FP6YPYNVSR5JWYDVFBO2TRFYQJWVWIUNQ, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.424729560 (297h12m29.021500589s) [noise=true , address=79.127.205.233:4156 , piece_id=5OZYLGXB7R6VMYGATW7GJD2CNAVO45PATTCFGNPWYZ3ZURYLZSQA, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.425355484 (297h12m29.022126513s) [noise=true , address=193.149.164.69:4077 , piece_id=GYZMBIAEBL4KBCSFA4H5JLWEB36NBN2BGVQ67A5G4ECUWY5BACXA, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.429910441 (297h12m29.026681471s) [noise=true , address=79.127.219.46:4918 , piece_id=WC2RL2PMO4EVATTOPZBWFHOZE6CBGQ2ROGJ44PDQUPQKYXBFKY6Q, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.431104280 (297h12m29.027875311s) [noise=true , address=79.127.219.45:4883 , piece_id=7DXMZGFUW2QAI2OID7WSSMJG6B3WW52HCMRWX2LS6YFEWAFMZZNQ, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.432173523 (297h12m29.02894455s) [noise=true , address=193.149.164.71:4300 , piece_id=4FI5PVS627OOB2M45U4QTOXGBJHNOZMUDDCFAW57W7PBAAQSJC4A, seg_pos={0 0}, upload=332883]: Uploading piece... Mar 18 20:12:31.478900188 (297h12m29.075671211s) [upload=332883]: Next returned nil segment Mar 18 20:12:31.478914031 (297h12m29.075685054s) [upload=332883]: Waiting for error group managing segments... Mar 18 20:12:31.512891809 (297h12m29.109662832s) [noise=true , address=79.127.219.46:4918 , piece_id=WC2RL2PMO4EVATTOPZBWFHOZE6CBGQ2ROGJ44PDQUPQKYXBFKY6Q, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil> Mar 18 20:12:31.529013012 (297h12m29.125784038s) [noise=true , address=45.91.116.5:4322 , piece_id=FN763EWEYGTBEXNHOT3FMKLW52DENMGPRBPBJJHNWQUNQ5JIN3XA, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil> Mar 18 20:12:31.564707355 (297h12m29.161478379s) [noise=true , address=79.127.219.45:4883 , piece_id=7DXMZGFUW2QAI2OID7WSSMJG6B3WW52HCMRWX2LS6YFEWAFMZZNQ, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil> Mar 18 20:12:31.611013762 (297h12m29.207784793s) [noise=true , address=5.231.73.4:4002 , piece_id=3DWXH2Y6ZZNDHRFMJFL6EEMJN7WNDRKJ3ARFKN6UAGAHICNCBSDA, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil> Mar 18 20:12:31.713823345 (297h12m29.310594381s) [noise=true , address=167.17.72.68:4653 , piece_id=6TO53KV4DBJ5HTWIQUZY2L6WVMB4PBPRWSZD6WHBLVFI5FUULYUQ, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil> Mar 18 20:12:31.997071787 (297h12m29.59384282s) [noise=true , address=79.127.205.233:4156 , piece_id=5OZYLGXB7R6VMYGATW7GJD2CNAVO45PATTCFGNPWYZ3ZURYLZSQA, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil> Mar 18 20:12:31.999264198 (297h12m29.59603524s) [noise=true , address=193.149.164.70:4111 , piece_id=H6GX5VVLRFNUAF2R7FRZ5GFQCZ2F57OL4GXIHWQJGZDOM3ST5DKQ, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil> Mar 18 20:12:32.008593285 (297h12m29.605364328s) [noise=true , address=193.149.164.71:4300 , piece_id=4FI5PVS627OOB2M45U4QTOXGBJHNOZMUDDCFAW57W7PBAAQSJC4A, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil> Mar 18 20:12:32.011203033 (297h12m29.607974056s) [noise=true , address=193.149.164.68:4040 , piece_id=4NT224HIHI45JJNYH3ZCXQKDAMQHHYWY7SZPYQK5KN7JKNYFGI2A, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil> Mar 18 20:12:32.038310240 (297h12m29.635081269s) [noise=true , address=193.149.164.67:4030 , piece_id=PYGFDMQLZ2SYL5NTWUGQGHWW52T2YYEGDKP5SUWQQUTVJSJZOO5A, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil> Mar 18 20:12:32.091831027 (297h12m29.68860206s) [noise=true , address=64.49.13.169:4705 , piece_id=FW7ICJDZJVOQR77HCGAWU7NKSBTWASW3BXH72WBGZHOFVPXA7XQQ, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil> Mar 18 20:12:32.408908738 (297h12m30.005679764s) [noise=true , address=193.149.164.69:4077 , piece_id=GYZMBIAEBL4KBCSFA4H5JLWEB36NBN2BGVQ67A5G4ECUWY5BACXA, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil> Mar 18 20:12:32.489564296 (297h12m30.086335307s) [noise=true , address=167.17.72.71:4808 , piece_id=JL5TJCWL454VA73PMWKZYBAUT6ZMRPEH55KJDSHEELC37HRVCO7Q, seg_pos={0 0}, upload=332883]: Done uploading piece. err: <nil> Mar 18 20:13:28.204470030 (297h13m25.801241064s) [noise=true , address=145.79.227.23:4564 , piece_id=3S5IXK7TKV2W3I635W2LQDH3PC6JRPHIVBK347CE7EYE4QQ5AGBQ, seg_pos={0 0}, upload=332883]: Done uploading piece. err: ecclient: upload failed (node:121WW4uBQe8bGQKkasQQKT3k4EAdzncsG1ebNvgKf8S5SyNttCc, address:145.79.227.23:4564): protocol: rpc: dial tcp 145.79.227.23:4564: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:13:28.205066456 (297h13m25.801837482s) [noise=true , address=145.79.227.28:4750 , piece_id=KJVCFR2SJ5HAQKJUJP4CYPVCZQC6PBAQII6UGNYVKRODWRKENFQA, seg_pos={0 0}, upload=332883]: Done uploading piece. err: ecclient: upload failed (node:12UP8PxqeK7t24PXjopH1nziHYQFnsxhkamSHz8dmwcbuDxkAF6, address:145.79.227.28:4750): protocol: rpc: dial tcp 145.79.227.28:4750: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:13:28.205099998 (297h13m25.80187103s) [noise=true , address=145.79.227.44:4202 , piece_id=UI7I4UPDOZCJ2MRPKV6FP6YPYNVSR5JWYDVFBO2TRFYQJWVWIUNQ, seg_pos={0 0}, upload=332883]: Done uploading piece. err: ecclient: upload failed (node:121ogdaiVGybXEVsS8LDJuJRYXGxb1LCNnGPuwbde96eSEzkTW4, address:145.79.227.44:4202): protocol: rpc: dial tcp 145.79.227.44:4202: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:13:28.205340404 (297h13m25.802111436s) [noise=true , address=145.79.227.24:4615 , piece_id=ZSEUUMSYRSXH4AJ3CDZNFKXBZFT63AKVDFBFJ2RLAZK344GQ5AFA, seg_pos={0 0}, upload=332883]: Done uploading piece. err: ecclient: upload failed (node:12NJ5w8pAWCudDA9Xs4ES7b2evnbscesd682wJDh2gMMZ3Jf2zc, address:145.79.227.24:4615): protocol: rpc: dial tcp 145.79.227.24:4615: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:13:28.205593210 (297h13m25.802364235s) [noise=true , address=145.79.227.20:4454 , piece_id=F2WUAPOH2J3GQ7DMWLBJA44UGY5B3MMDS2U5ITXTOIPR4ELY5QYA, seg_pos={0 0}, upload=332883]: Done uploading piece. err: ecclient: upload failed (node:1SafkJPqKHLwMaqyZdLq3yJ1xdjqDtWHFrUwynmegUVwEC7y5J, address:145.79.227.20:4454): protocol: context deadline exceeded; context deadline exceeded Mar 18 20:13:28.205633541 (297h13m25.802404553s) [seg_pos={0 0}, upload=332883]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 18 20:13:28.228774364 (297h13m25.825545388s) [upload=332883]: Done uploading segments... Mar 18 20:13:28.228821883 (297h13m25.825592906s) [upload=332883]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 1m0.027507389s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 8 out of 60 is failed (threshold=3) [...........XXXXXX.XX........................................] duration: 17.400143201s trace: https://jaeger.dev.storj.io/trace/4b4d10a8051ad01b FAIL (2026-03-18 20:13:28 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 17.400166138s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 6 out of 60 is failed (threshold=3) [................X..........XX..X....X........X..............] duration: 17.493110418s trace: https://jaeger.dev.storj.io/trace/589419efc52e5e2b FAIL (2026-03-18 20:13:21 UTC) operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== Mar 18 20:00:59.343084966 (297h0m56.939855989s) [upload=332666]: Starting upload Mar 18 20:00:59.343175518 (297h0m56.939946548s) [upload=332666]: Done starting upload Mar 18 20:00:59.343565585 (297h0m56.940336612s) [upload=332666]: Uploading segments... Mar 18 20:00:59.344256647 (297h0m56.941027676s) [upload=332666]: Got next segment. Inline: false Mar 18 20:00:59.344352215 (297h0m56.941123241s) [upload=332666]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 20:00:59.344358706 (297h0m56.941129732s) [upload=332666]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 18 20:01:02.340259299 (297h0m59.937030329s) [seg_pos={0 0}, upload=332666]: Begin upload segment... Mar 18 20:01:02.340368743 (297h0m59.937139767s) [seg_pos={0 0}, upload=332666]: Done begin upload segment. Mar 18 20:01:02.340479563 (297h0m59.93725059s) [noise=true , address=45.91.116.8:4061 , piece_id=6XFVKT4AAO7QN457OQMRKPI6R24TJTAN47UXLEVOEFQVOS6M6C3Q, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.340523837 (297h0m59.93729486s) [noise=true , address=79.127.205.233:4166 , piece_id=SFGK5YCWHFRVXF6QNHYP2CYJCZLSTQ76OYRVJPWWXQTF2KFSFJKA, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.341468438 (297h0m59.938239447s) [noise=true , address=45.91.116.4:4961 , piece_id=ZMUGQECUVPVHDNKN75XPBET5M2HFXXK4WMTP4V5KEK63JVZCX5EA, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.341520025 (297h0m59.938291058s) [noise=true , address=167.17.72.71:4815 , piece_id=CXGTYNNBILN3TT7I6Z7UI75F2ZCS4QUTLS6KC4FJ5IHS3XOYWDJQ, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.342232992 (297h0m59.939004023s) [noise=true , address=145.79.227.36:4524 , piece_id=3TAV3P55ITTHBAZEUUUTOAHJRGQXJXXVGZJP6AVR2ANVNOR3S54Q, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.342917368 (297h0m59.939688398s) [noise=true , address=193.149.164.71:4297 , piece_id=4IJPXLPCX3LKFOBOSRHK2DTNRKH67K7FXJUKQ3JPRWHS47ZFEIDA, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.343244957 (297h0m59.940015982s) [noise=true , address=145.79.227.33:4892 , piece_id=DQ6ALC3VOAGFIJI3IKR6I7QFYM22C66QWAAXMH6NXI2FIQEZNQYA, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.344270079 (297h0m59.941041117s) [noise=true , address=145.79.227.21:4497 , piece_id=SP6EKJB7W4QXF7OIBVKEFWCPOZFL5IRIEDVAGS2M4QJSO34CUVRQ, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.345025057 (297h0m59.941796084s) [noise=true , address=77.90.31.17:4793 , piece_id=2U4IBUA2GGUGDBXQGIGQKUK4ZR3L7CDKC6MTGIKAUME4F7MFZRLA, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.345115802 (297h0m59.941886836s) [noise=true , address=167.17.72.69:4731 , piece_id=VOEVIKKFJMC2Y3TPQIDMYUP37NLFU7KPRH4HRRSMQLWUKTSLOH2Q, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.345690600 (297h0m59.942461634s) [noise=true , address=77.90.31.13:4648 , piece_id=BPMB5CFOYHFT3BS4ORR6JF6BNFC2AOAJZOSPMG67NAMLGGP5XVEQ, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.346311592 (297h0m59.943082623s) [noise=true , address=79.127.205.234:4202 , piece_id=YY4LYWX26O6RIOBONESNKTUZEFZCRL4UUX3NEBQ53AJV4IEDG5MQ, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.346791142 (297h0m59.943562166s) [noise=true , address=45.91.116.10:4110 , piece_id=UYLMDCWJEJOHGICA37KNH7G5AQXNIPRH5NRYLBUUMGDRVSLTEYHQ, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.347094999 (297h0m59.94386607s) [noise=true , address=145.79.227.17:4987 , piece_id=PT3TWVB65JZFEWVH4IUZEU2N53FQ57QBRWVLWAUFP4UDLY3LWNLQ, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.347781281 (297h0m59.944552305s) [noise=true , address=79.127.219.46:4919 , piece_id=KK6HRIVBMYLIHRHGPQH3AWMNSVHTTIBESHDMJZA72T446RX6QKRQ, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.348263123 (297h0m59.945034137s) [noise=true , address=193.149.164.68:4033 , piece_id=JZPQKPYGO3W3RZZBITQ73X2XEPIBBSLGRMWIVOM4Z3GTV5UFKRTQ, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.348915505 (297h0m59.945686536s) [noise=true , address=145.79.227.7:4312 , piece_id=2FAL6E3RMBTBHZBXAJJVH2N5VGF5VYI3KNZXGNWCMWGBBPAUO2NQ, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.349366298 (297h0m59.946137337s) [noise=true , address=193.149.164.67:4007 , piece_id=HURQGH67I5B6V4YUV3PRZL3KSKN3XTSK3UMCHZRNT6L3W3PXXHPQ, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.429834098 (297h1m0.026605126s) [upload=332666]: Next returned nil segment Mar 18 20:01:02.429853411 (297h1m0.026624442s) [upload=332666]: Waiting for error group managing segments... Mar 18 20:01:02.446867680 (297h1m0.043638704s) [noise=true , address=45.91.116.10:4110 , piece_id=UYLMDCWJEJOHGICA37KNH7G5AQXNIPRH5NRYLBUUMGDRVSLTEYHQ, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil> Mar 18 20:01:02.451665165 (297h1m0.048436196s) [noise=true , address=45.91.116.4:4961 , piece_id=ZMUGQECUVPVHDNKN75XPBET5M2HFXXK4WMTP4V5KEK63JVZCX5EA, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil> Mar 18 20:01:02.452538886 (297h1m0.049309913s) [noise=true , address=77.90.31.17:4793 , piece_id=2U4IBUA2GGUGDBXQGIGQKUK4ZR3L7CDKC6MTGIKAUME4F7MFZRLA, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil> Mar 18 20:01:02.455597040 (297h1m0.052368067s) [noise=true , address=45.91.116.8:4061 , piece_id=6XFVKT4AAO7QN457OQMRKPI6R24TJTAN47UXLEVOEFQVOS6M6C3Q, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil> Mar 18 20:01:02.457848739 (297h1m0.054619766s) [noise=true , address=77.90.31.13:4648 , piece_id=BPMB5CFOYHFT3BS4ORR6JF6BNFC2AOAJZOSPMG67NAMLGGP5XVEQ, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil> Mar 18 20:01:02.461479709 (297h1m0.058250746s) [noise=true , address=79.127.219.46:4919 , piece_id=KK6HRIVBMYLIHRHGPQH3AWMNSVHTTIBESHDMJZA72T446RX6QKRQ, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil> Mar 18 20:01:02.676859466 (297h1m0.27363049s) [noise=true , address=167.17.72.69:4731 , piece_id=VOEVIKKFJMC2Y3TPQIDMYUP37NLFU7KPRH4HRRSMQLWUKTSLOH2Q, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil> Mar 18 20:01:02.909860743 (297h1m0.506631766s) [noise=true , address=79.127.205.234:4202 , piece_id=YY4LYWX26O6RIOBONESNKTUZEFZCRL4UUX3NEBQ53AJV4IEDG5MQ, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil> Mar 18 20:01:02.930427742 (297h1m0.527198754s) [noise=true , address=193.149.164.71:4297 , piece_id=4IJPXLPCX3LKFOBOSRHK2DTNRKH67K7FXJUKQ3JPRWHS47ZFEIDA, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil> Mar 18 20:01:02.944696889 (297h1m0.541467914s) [noise=true , address=193.149.164.67:4007 , piece_id=HURQGH67I5B6V4YUV3PRZL3KSKN3XTSK3UMCHZRNT6L3W3PXXHPQ, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil> Mar 18 20:01:02.951319183 (297h1m0.548090212s) [noise=true , address=193.149.164.68:4033 , piece_id=JZPQKPYGO3W3RZZBITQ73X2XEPIBBSLGRMWIVOM4Z3GTV5UFKRTQ, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil> Mar 18 20:01:02.975619436 (297h1m0.572390467s) [noise=true , address=79.127.205.233:4166 , piece_id=SFGK5YCWHFRVXF6QNHYP2CYJCZLSTQ76OYRVJPWWXQTF2KFSFJKA, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil> Mar 18 20:01:03.819918528 (297h1m1.416689564s) [noise=true , address=167.17.72.71:4815 , piece_id=CXGTYNNBILN3TT7I6Z7UI75F2ZCS4QUTLS6KC4FJ5IHS3XOYWDJQ, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil> Mar 18 20:01:59.307873420 (297h1m56.904644452s) [noise=true , address=145.79.227.7:4312 , piece_id=2FAL6E3RMBTBHZBXAJJVH2N5VGF5VYI3KNZXGNWCMWGBBPAUO2NQ, seg_pos={0 0}, upload=332666]: Done uploading piece. err: ecclient: upload failed (node:12AVp3qn5uwLPqYaZEwzcgoGMJhNga1BUh6bK5k2bxjWgZLQiiC, address:145.79.227.7:4312): protocol: rpc: dial tcp 145.79.227.7:4312: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:01:59.308026092 (297h1m56.904797124s) [noise=true , address=145.79.227.17:4987 , piece_id=PT3TWVB65JZFEWVH4IUZEU2N53FQ57QBRWVLWAUFP4UDLY3LWNLQ, seg_pos={0 0}, upload=332666]: Done uploading piece. err: ecclient: upload failed (node:1XsPVNGYUhKnXkQRfVfjyBRzyvhShSdsrnzXtwe4Xtf5MaHMvH, address:145.79.227.17:4987): protocol: rpc: dial tcp 145.79.227.17:4987: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:01:59.308195864 (297h1m56.904966894s) [noise=true , address=145.79.227.36:4524 , piece_id=3TAV3P55ITTHBAZEUUUTOAHJRGQXJXXVGZJP6AVR2ANVNOR3S54Q, seg_pos={0 0}, upload=332666]: Done uploading piece. err: ecclient: upload failed (node:1xfbCzjSwFDRrR1gPggeN8tNCLbRRWDVzCJ6mqhZsbVSe8d4s4, address:145.79.227.36:4524): protocol: rpc: dial tcp 145.79.227.36:4524: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:01:59.308316037 (297h1m56.90508706s) [noise=true , address=145.79.227.33:4892 , piece_id=DQ6ALC3VOAGFIJI3IKR6I7QFYM22C66QWAAXMH6NXI2FIQEZNQYA, seg_pos={0 0}, upload=332666]: Done uploading piece. err: ecclient: upload failed (node:12RHwFxcUjPsn12uHEAH3fBDJKv1v1tM8mRJXoFrZrG5djdswF5, address:145.79.227.33:4892): protocol: rpc: dial tcp 145.79.227.33:4892: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:01:59.308404403 (297h1m56.905175433s) [noise=true , address=145.79.227.21:4497 , piece_id=SP6EKJB7W4QXF7OIBVKEFWCPOZFL5IRIEDVAGS2M4QJSO34CUVRQ, seg_pos={0 0}, upload=332666]: Done uploading piece. err: ecclient: upload failed (node:12ZBe4we7VeFdthqRxyy1V9uQrA4r8ouNX68xqsk5bD26dkf2Gd, address:145.79.227.21:4497): protocol: context deadline exceeded; context deadline exceeded Mar 18 20:01:59.308437457 (297h1m56.90520847s) [seg_pos={0 0}, upload=332666]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 18 20:01:59.338288467 (297h1m56.93505949s) [upload=332666]: Done uploading segments... Mar 18 20:01:59.338335497 (297h1m56.935106523s) [upload=332666]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 17.493139668s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 56 out of 60 is failed (threshold=6) [XXXXXXXX.XXX...XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 5.001542999s trace: https://jaeger.dev.storj.io/trace/6f3d04069d7646d0 FAIL (2026-03-18 20:13:06 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001565451s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 46 out of 60 is failed (threshold=3) [........XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.XXXXXX...XX.X.] duration: 19.051583329s trace: https://jaeger.dev.storj.io/trace/601343143808b18 FAIL (2026-03-18 20:12:35 UTC) operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Mar 18 20:11:17.942948177 (297h11m15.539719208s) [upload=332864]: Starting upload Mar 18 20:11:17.943091638 (297h11m15.539862664s) [upload=332864]: Done starting upload Mar 18 20:11:17.943155714 (297h11m15.53992673s) [upload=332864]: Uploading segments... Mar 18 20:11:17.943462205 (297h11m15.540233222s) [upload=332864]: Got next segment. Inline: false Mar 18 20:11:17.943529204 (297h11m15.540300231s) [upload=332864]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 20:11:17.943536141 (297h11m15.540307167s) [upload=332864]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 18 20:11:21.580830031 (297h11m19.17760108s) [seg_pos={0 0}, upload=332864]: Begin upload segment... Mar 18 20:11:21.580927854 (297h11m19.177698881s) [seg_pos={0 0}, upload=332864]: Done begin upload segment. Mar 18 20:11:21.580991732 (297h11m19.177762763s) [noise=true , address=45.91.116.11:4121 , piece_id=23NVBAV75T2EG6HDNGGB2E224LW2TNUUZVS2Q6AZXBEV56533DCQ, seg_pos={0 0}, upload=332864]: Uploading piece... Mar 18 20:11:21.581000032 (297h11m19.177771061s) [noise=true , address=145.79.227.20:4441 , piece_id=67TR4L5T33FKQQYFZCQX4JBQOCF75OQMX4BESZSUCVGHRF23A4ZA, seg_pos={0 0}, upload=332864]: Uploading piece... Mar 18 20:11:21.581521572 (297h11m19.178292602s) [noise=true , address=77.90.31.18:4808 , piece_id=3EGDWOJ6SMMS7ZAZIAQXWDTPIER2XMIDBOB24ZW3ZF7OXVXUGGOQ, seg_pos={0 0}, upload=332864]: Uploading piece... Mar 18 20:11:21.581999141 (297h11m19.178770178s) [noise=true , address=77.90.31.15:4700 , piece_id=6IAKOFJIJOO2QE3222C4YUEAB22CMYNSNZ6L6ZQZFUEPNOZKYHEA, seg_pos={0 0}, upload=332864]: Uploading piece... Mar 18 20:11:21.582431519 (297h11m19.179202564s) [noise=true , address=145.79.227.35:4481 , piece_id=M5KBTQZCLU3VF5WCQFB3XHDXNAR5XKCRBB4KRC2JYDZOPQ2ECLGA, seg_pos={0 0}, upload=332864]: Uploading piece... Mar 18 20:11:21.582886090 (297h11m19.179657099s) [noise=true , address=79.127.219.41:4930 , piece_id=FMHODO5CFRIBKO4GRJXEI3W5RURLWOQGSOSIQONOKKX5JFYTYEWQ, seg_pos={0 0}, upload=332864]: Uploading piece... Mar 18 20:11:21.583436640 (297h11m19.180207673s) [noise=true , address=45.91.116.9:4265 , piece_id=FKZZKPFHBXRJJ2BWL2GHW5KVSZ4Y6WUQ3YXY6GG2VHD66FDS5IBQ, seg_pos={0 0}, upload=332864]: Uploading piece... Mar 18 20:11:21.583983698 (297h11m19.18075473s) [noise=true , address=45.91.116.8:4047 , piece_id=3SHZ6I45QULZ3YYIGFCQTAFQ23LSXMKY7T2BSGBJDUM2YMKUQNRQ, seg_pos={0 0}, upload=332864]: Uploading piece... Mar 18 20:11:21.584563907 (297h11m19.181334942s) [noise=true , address=145.79.227.34:4907 , piece_id=F2FVEGDWGE467HNTIVPKGCNPGY3CDO74BYGKM5U66Y256NMCUFIA, seg_pos={0 0}, upload=332864]: Uploading piece... Mar 18 20:11:21.586097233 (297h11m19.182868282s) [noise=true , address=77.90.31.21:4880 , piece_id=JJPRYSNNUFWEIUK6BBKLQLS2OHDI7GES74A6UD7OFZDVRW4N7BNA, seg_pos={0 0}, upload=332864]: Uploading piece... Mar 18 20:11:21.586229485 (297h11m19.183000519s) [noise=true , address=45.91.116.7:4009 , piece_id=LRZIZOUYVZUM37PSYFSR5CRCYOAHRPEXWFULVVJDGQEVYDQTRW2Q, seg_pos={0 0}, upload=332864]: Uploading piece... Mar 18 20:11:21.587876633 (297h11m19.184647659s) [noise=true , address=79.127.219.46:4889 , piece_id=L5FCXPYQ2ESKQJQEYWUDSTI7CUAK4WGOD3XE2KKUURXRI2YKEXEA, seg_pos={0 0}, upload=332864]: Uploading piece... Mar 18 20:11:21.589859838 (297h11m19.186630873s) [noise=true , address=45.91.116.12:4227 , piece_id=PCPBXUU4FYD7SKOHU7RGZ2CC2C37Z4XLVET6B5W3MGXTPQABBPDQ, seg_pos={0 0}, upload=332864]: Uploading piece... Mar 18 20:11:21.591680944 (297h11m19.188451994s) [noise=true , address=45.91.116.6:4973 , piece_id=7ANPNX262Y2YIGHMJ5U7MQ3MYPRB2B77KLSCZCIU5MG2CXQ4ID5A, seg_pos={0 0}, upload=332864]: Uploading piece... Mar 18 20:11:21.593792892 (297h11m19.190563928s) [noise=true , address=145.79.227.21:4473 , piece_id=ER4TPXHFHM6X3MWWN2PADSH7PRPV6MMBLSD5WJXB2OAOHRG5OTOQ, seg_pos={0 0}, upload=332864]: Uploading piece... Mar 18 20:11:21.596130805 (297h11m19.192901838s) [noise=true , address=145.79.227.6:4262 , piece_id=ONUXRTAKZOHQASQSWR5YEQIDR7B5TX4YCTV44ISBEEZPSTDGITFQ, seg_pos={0 0}, upload=332864]: Uploading piece... Mar 18 20:11:21.657091067 (297h11m19.253862083s) [upload=332864]: Next returned nil segment Mar 18 20:11:21.657105214 (297h11m19.253876237s) [upload=332864]: Waiting for error group managing segments... Mar 18 20:11:21.662628450 (297h11m19.259399483s) [noise=true , address=45.91.116.12:4227 , piece_id=PCPBXUU4FYD7SKOHU7RGZ2CC2C37Z4XLVET6B5W3MGXTPQABBPDQ, seg_pos={0 0}, upload=332864]: Done uploading piece. err: <nil> Mar 18 20:11:21.662667493 (297h11m19.259438519s) [noise=true , address=45.91.116.11:4121 , piece_id=23NVBAV75T2EG6HDNGGB2E224LW2TNUUZVS2Q6AZXBEV56533DCQ, seg_pos={0 0}, upload=332864]: Done uploading piece. err: <nil> Mar 18 20:11:21.663690046 (297h11m19.260461087s) [noise=true , address=45.91.116.7:4009 , piece_id=LRZIZOUYVZUM37PSYFSR5CRCYOAHRPEXWFULVVJDGQEVYDQTRW2Q, seg_pos={0 0}, upload=332864]: Done uploading piece. err: <nil> Mar 18 20:11:21.664634495 (297h11m19.26140552s) [noise=true , address=77.90.31.15:4700 , piece_id=6IAKOFJIJOO2QE3222C4YUEAB22CMYNSNZ6L6ZQZFUEPNOZKYHEA, seg_pos={0 0}, upload=332864]: Done uploading piece. err: <nil> Mar 18 20:11:21.665900844 (297h11m19.26267186s) [noise=true , address=79.127.219.41:4930 , piece_id=FMHODO5CFRIBKO4GRJXEI3W5RURLWOQGSOSIQONOKKX5JFYTYEWQ, seg_pos={0 0}, upload=332864]: Done uploading piece. err: <nil> Mar 18 20:11:21.666002999 (297h11m19.262774023s) [noise=true , address=79.127.219.46:4889 , piece_id=L5FCXPYQ2ESKQJQEYWUDSTI7CUAK4WGOD3XE2KKUURXRI2YKEXEA, seg_pos={0 0}, upload=332864]: Done uploading piece. err: <nil> Mar 18 20:11:21.668088253 (297h11m19.264859266s) [noise=true , address=45.91.116.9:4265 , piece_id=FKZZKPFHBXRJJ2BWL2GHW5KVSZ4Y6WUQ3YXY6GG2VHD66FDS5IBQ, seg_pos={0 0}, upload=332864]: Done uploading piece. err: <nil> Mar 18 20:11:21.673482814 (297h11m19.270253826s) [noise=true , address=45.91.116.6:4973 , piece_id=7ANPNX262Y2YIGHMJ5U7MQ3MYPRB2B77KLSCZCIU5MG2CXQ4ID5A, seg_pos={0 0}, upload=332864]: Done uploading piece. err: <nil> Mar 18 20:11:21.686416503 (297h11m19.283187526s) [noise=true , address=77.90.31.21:4880 , piece_id=JJPRYSNNUFWEIUK6BBKLQLS2OHDI7GES74A6UD7OFZDVRW4N7BNA, seg_pos={0 0}, upload=332864]: Done uploading piece. err: <nil> Mar 18 20:11:21.693971801 (297h11m19.290742824s) [noise=true , address=45.91.116.8:4047 , piece_id=3SHZ6I45QULZ3YYIGFCQTAFQ23LSXMKY7T2BSGBJDUM2YMKUQNRQ, seg_pos={0 0}, upload=332864]: Done uploading piece. err: <nil> Mar 18 20:11:21.727700946 (297h11m19.324471973s) [noise=true , address=77.90.31.18:4808 , piece_id=3EGDWOJ6SMMS7ZAZIAQXWDTPIER2XMIDBOB24ZW3ZF7OXVXUGGOQ, seg_pos={0 0}, upload=332864]: Done uploading piece. err: <nil> Mar 18 20:11:47.913205594 (297h11m45.509976618s) [noise=true , address=145.79.227.21:4473 , piece_id=ER4TPXHFHM6X3MWWN2PADSH7PRPV6MMBLSD5WJXB2OAOHRG5OTOQ, seg_pos={0 0}, upload=332864]: Done uploading piece. err: ecclient: upload failed (node:12DDe4jT3YeUH8qMQgapcvzMnFs8UFB8fTmubho4ZQf5tYVoxuk, address:145.79.227.21:4473): protocol: rpc: dial tcp 145.79.227.21:4473: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:11:47.913229665 (297h11m45.510000697s) [noise=true , address=145.79.227.6:4262 , piece_id=ONUXRTAKZOHQASQSWR5YEQIDR7B5TX4YCTV44ISBEEZPSTDGITFQ, seg_pos={0 0}, upload=332864]: Done uploading piece. err: ecclient: upload failed (node:12nPw17QDTyrgDBBNKuunFWuDqPV5Rd2GpM7Loc6Ws9vAAZz22s, address:145.79.227.6:4262): protocol: rpc: dial tcp 145.79.227.6:4262: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:11:47.913527933 (297h11m45.510298964s) [noise=true , address=145.79.227.34:4907 , piece_id=F2FVEGDWGE467HNTIVPKGCNPGY3CDO74BYGKM5U66Y256NMCUFIA, seg_pos={0 0}, upload=332864]: Done uploading piece. err: ecclient: upload failed (node:12URx18Ckh7VaAKJ1hdQQucMRu5qiDrAraqaycjYqxETgwZ4uWS, address:145.79.227.34:4907): protocol: rpc: dial tcp 145.79.227.34:4907: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:11:47.913654758 (297h11m45.510425788s) [noise=true , address=145.79.227.35:4481 , piece_id=M5KBTQZCLU3VF5WCQFB3XHDXNAR5XKCRBB4KRC2JYDZOPQ2ECLGA, seg_pos={0 0}, upload=332864]: Done uploading piece. err: ecclient: upload failed (node:12bRK8WWmAnWUwzCfHu8JtZgrruYSJCvhnsYTKcaLccJcv5RQjp, address:145.79.227.35:4481): protocol: rpc: dial tcp 145.79.227.35:4481: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:11:47.914034150 (297h11m45.510805182s) [noise=true , address=145.79.227.20:4441 , piece_id=67TR4L5T33FKQQYFZCQX4JBQOCF75OQMX4BESZSUCVGHRF23A4ZA, seg_pos={0 0}, upload=332864]: Done uploading piece. err: ecclient: upload failed (node:12LjGecRNJE1kq67iMtxZFhWeaUZpUogsGxbz4VXwnHdQtTnbBr, address:145.79.227.20:4441): protocol: context deadline exceeded; context deadline exceeded Mar 18 20:11:47.914082052 (297h11m45.510853068s) [seg_pos={0 0}, upload=332864]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Mar 18 20:11:47.947299775 (297h11m45.5440708s) [upload=332864]: Done uploading segments... Mar 18 20:11:47.947420462 (297h11m45.544191496s) [upload=332864]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 19.051703224s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 8 out of 60 is failed (threshold=3) [............XXXXXX.XX.......................................] duration: 18.023667688s trace: https://jaeger.dev.storj.io/trace/4b4d10a8051ad01b FAIL (2026-03-18 20:12:29 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 18.023766775s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 6 out of 60 is failed (threshold=3) [.................X..........XX..X....X........X.............] duration: 15.767544233s trace: https://jaeger.dev.storj.io/trace/589419efc52e5e2b FAIL (2026-03-18 20:12:26 UTC) operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== Mar 18 20:00:59.343084966 (297h0m56.939855989s) [upload=332666]: Starting upload Mar 18 20:00:59.343175518 (297h0m56.939946548s) [upload=332666]: Done starting upload Mar 18 20:00:59.343565585 (297h0m56.940336612s) [upload=332666]: Uploading segments... Mar 18 20:00:59.344256647 (297h0m56.941027676s) [upload=332666]: Got next segment. Inline: false Mar 18 20:00:59.344352215 (297h0m56.941123241s) [upload=332666]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 20:00:59.344358706 (297h0m56.941129732s) [upload=332666]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 18 20:01:02.340259299 (297h0m59.937030329s) [seg_pos={0 0}, upload=332666]: Begin upload segment... Mar 18 20:01:02.340368743 (297h0m59.937139767s) [seg_pos={0 0}, upload=332666]: Done begin upload segment. Mar 18 20:01:02.340479563 (297h0m59.93725059s) [noise=true , address=45.91.116.8:4061 , piece_id=6XFVKT4AAO7QN457OQMRKPI6R24TJTAN47UXLEVOEFQVOS6M6C3Q, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.340523837 (297h0m59.93729486s) [noise=true , address=79.127.205.233:4166 , piece_id=SFGK5YCWHFRVXF6QNHYP2CYJCZLSTQ76OYRVJPWWXQTF2KFSFJKA, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.341468438 (297h0m59.938239447s) [noise=true , address=45.91.116.4:4961 , piece_id=ZMUGQECUVPVHDNKN75XPBET5M2HFXXK4WMTP4V5KEK63JVZCX5EA, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.341520025 (297h0m59.938291058s) [noise=true , address=167.17.72.71:4815 , piece_id=CXGTYNNBILN3TT7I6Z7UI75F2ZCS4QUTLS6KC4FJ5IHS3XOYWDJQ, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.342232992 (297h0m59.939004023s) [noise=true , address=145.79.227.36:4524 , piece_id=3TAV3P55ITTHBAZEUUUTOAHJRGQXJXXVGZJP6AVR2ANVNOR3S54Q, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.342917368 (297h0m59.939688398s) [noise=true , address=193.149.164.71:4297 , piece_id=4IJPXLPCX3LKFOBOSRHK2DTNRKH67K7FXJUKQ3JPRWHS47ZFEIDA, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.343244957 (297h0m59.940015982s) [noise=true , address=145.79.227.33:4892 , piece_id=DQ6ALC3VOAGFIJI3IKR6I7QFYM22C66QWAAXMH6NXI2FIQEZNQYA, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.344270079 (297h0m59.941041117s) [noise=true , address=145.79.227.21:4497 , piece_id=SP6EKJB7W4QXF7OIBVKEFWCPOZFL5IRIEDVAGS2M4QJSO34CUVRQ, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.345025057 (297h0m59.941796084s) [noise=true , address=77.90.31.17:4793 , piece_id=2U4IBUA2GGUGDBXQGIGQKUK4ZR3L7CDKC6MTGIKAUME4F7MFZRLA, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.345115802 (297h0m59.941886836s) [noise=true , address=167.17.72.69:4731 , piece_id=VOEVIKKFJMC2Y3TPQIDMYUP37NLFU7KPRH4HRRSMQLWUKTSLOH2Q, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.345690600 (297h0m59.942461634s) [noise=true , address=77.90.31.13:4648 , piece_id=BPMB5CFOYHFT3BS4ORR6JF6BNFC2AOAJZOSPMG67NAMLGGP5XVEQ, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.346311592 (297h0m59.943082623s) [noise=true , address=79.127.205.234:4202 , piece_id=YY4LYWX26O6RIOBONESNKTUZEFZCRL4UUX3NEBQ53AJV4IEDG5MQ, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.346791142 (297h0m59.943562166s) [noise=true , address=45.91.116.10:4110 , piece_id=UYLMDCWJEJOHGICA37KNH7G5AQXNIPRH5NRYLBUUMGDRVSLTEYHQ, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.347094999 (297h0m59.94386607s) [noise=true , address=145.79.227.17:4987 , piece_id=PT3TWVB65JZFEWVH4IUZEU2N53FQ57QBRWVLWAUFP4UDLY3LWNLQ, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.347781281 (297h0m59.944552305s) [noise=true , address=79.127.219.46:4919 , piece_id=KK6HRIVBMYLIHRHGPQH3AWMNSVHTTIBESHDMJZA72T446RX6QKRQ, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.348263123 (297h0m59.945034137s) [noise=true , address=193.149.164.68:4033 , piece_id=JZPQKPYGO3W3RZZBITQ73X2XEPIBBSLGRMWIVOM4Z3GTV5UFKRTQ, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.348915505 (297h0m59.945686536s) [noise=true , address=145.79.227.7:4312 , piece_id=2FAL6E3RMBTBHZBXAJJVH2N5VGF5VYI3KNZXGNWCMWGBBPAUO2NQ, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.349366298 (297h0m59.946137337s) [noise=true , address=193.149.164.67:4007 , piece_id=HURQGH67I5B6V4YUV3PRZL3KSKN3XTSK3UMCHZRNT6L3W3PXXHPQ, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.429834098 (297h1m0.026605126s) [upload=332666]: Next returned nil segment Mar 18 20:01:02.429853411 (297h1m0.026624442s) [upload=332666]: Waiting for error group managing segments... Mar 18 20:01:02.446867680 (297h1m0.043638704s) [noise=true , address=45.91.116.10:4110 , piece_id=UYLMDCWJEJOHGICA37KNH7G5AQXNIPRH5NRYLBUUMGDRVSLTEYHQ, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil> Mar 18 20:01:02.451665165 (297h1m0.048436196s) [noise=true , address=45.91.116.4:4961 , piece_id=ZMUGQECUVPVHDNKN75XPBET5M2HFXXK4WMTP4V5KEK63JVZCX5EA, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil> Mar 18 20:01:02.452538886 (297h1m0.049309913s) [noise=true , address=77.90.31.17:4793 , piece_id=2U4IBUA2GGUGDBXQGIGQKUK4ZR3L7CDKC6MTGIKAUME4F7MFZRLA, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil> Mar 18 20:01:02.455597040 (297h1m0.052368067s) [noise=true , address=45.91.116.8:4061 , piece_id=6XFVKT4AAO7QN457OQMRKPI6R24TJTAN47UXLEVOEFQVOS6M6C3Q, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil> Mar 18 20:01:02.457848739 (297h1m0.054619766s) [noise=true , address=77.90.31.13:4648 , piece_id=BPMB5CFOYHFT3BS4ORR6JF6BNFC2AOAJZOSPMG67NAMLGGP5XVEQ, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil> Mar 18 20:01:02.461479709 (297h1m0.058250746s) [noise=true , address=79.127.219.46:4919 , piece_id=KK6HRIVBMYLIHRHGPQH3AWMNSVHTTIBESHDMJZA72T446RX6QKRQ, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil> Mar 18 20:01:02.676859466 (297h1m0.27363049s) [noise=true , address=167.17.72.69:4731 , piece_id=VOEVIKKFJMC2Y3TPQIDMYUP37NLFU7KPRH4HRRSMQLWUKTSLOH2Q, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil> Mar 18 20:01:02.909860743 (297h1m0.506631766s) [noise=true , address=79.127.205.234:4202 , piece_id=YY4LYWX26O6RIOBONESNKTUZEFZCRL4UUX3NEBQ53AJV4IEDG5MQ, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil> Mar 18 20:01:02.930427742 (297h1m0.527198754s) [noise=true , address=193.149.164.71:4297 , piece_id=4IJPXLPCX3LKFOBOSRHK2DTNRKH67K7FXJUKQ3JPRWHS47ZFEIDA, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil> Mar 18 20:01:02.944696889 (297h1m0.541467914s) [noise=true , address=193.149.164.67:4007 , piece_id=HURQGH67I5B6V4YUV3PRZL3KSKN3XTSK3UMCHZRNT6L3W3PXXHPQ, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil> Mar 18 20:01:02.951319183 (297h1m0.548090212s) [noise=true , address=193.149.164.68:4033 , piece_id=JZPQKPYGO3W3RZZBITQ73X2XEPIBBSLGRMWIVOM4Z3GTV5UFKRTQ, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil> Mar 18 20:01:02.975619436 (297h1m0.572390467s) [noise=true , address=79.127.205.233:4166 , piece_id=SFGK5YCWHFRVXF6QNHYP2CYJCZLSTQ76OYRVJPWWXQTF2KFSFJKA, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil> Mar 18 20:01:03.819918528 (297h1m1.416689564s) [noise=true , address=167.17.72.71:4815 , piece_id=CXGTYNNBILN3TT7I6Z7UI75F2ZCS4QUTLS6KC4FJ5IHS3XOYWDJQ, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil> Mar 18 20:01:59.307873420 (297h1m56.904644452s) [noise=true , address=145.79.227.7:4312 , piece_id=2FAL6E3RMBTBHZBXAJJVH2N5VGF5VYI3KNZXGNWCMWGBBPAUO2NQ, seg_pos={0 0}, upload=332666]: Done uploading piece. err: ecclient: upload failed (node:12AVp3qn5uwLPqYaZEwzcgoGMJhNga1BUh6bK5k2bxjWgZLQiiC, address:145.79.227.7:4312): protocol: rpc: dial tcp 145.79.227.7:4312: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:01:59.308026092 (297h1m56.904797124s) [noise=true , address=145.79.227.17:4987 , piece_id=PT3TWVB65JZFEWVH4IUZEU2N53FQ57QBRWVLWAUFP4UDLY3LWNLQ, seg_pos={0 0}, upload=332666]: Done uploading piece. err: ecclient: upload failed (node:1XsPVNGYUhKnXkQRfVfjyBRzyvhShSdsrnzXtwe4Xtf5MaHMvH, address:145.79.227.17:4987): protocol: rpc: dial tcp 145.79.227.17:4987: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:01:59.308195864 (297h1m56.904966894s) [noise=true , address=145.79.227.36:4524 , piece_id=3TAV3P55ITTHBAZEUUUTOAHJRGQXJXXVGZJP6AVR2ANVNOR3S54Q, seg_pos={0 0}, upload=332666]: Done uploading piece. err: ecclient: upload failed (node:1xfbCzjSwFDRrR1gPggeN8tNCLbRRWDVzCJ6mqhZsbVSe8d4s4, address:145.79.227.36:4524): protocol: rpc: dial tcp 145.79.227.36:4524: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:01:59.308316037 (297h1m56.90508706s) [noise=true , address=145.79.227.33:4892 , piece_id=DQ6ALC3VOAGFIJI3IKR6I7QFYM22C66QWAAXMH6NXI2FIQEZNQYA, seg_pos={0 0}, upload=332666]: Done uploading piece. err: ecclient: upload failed (node:12RHwFxcUjPsn12uHEAH3fBDJKv1v1tM8mRJXoFrZrG5djdswF5, address:145.79.227.33:4892): protocol: rpc: dial tcp 145.79.227.33:4892: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:01:59.308404403 (297h1m56.905175433s) [noise=true , address=145.79.227.21:4497 , piece_id=SP6EKJB7W4QXF7OIBVKEFWCPOZFL5IRIEDVAGS2M4QJSO34CUVRQ, seg_pos={0 0}, upload=332666]: Done uploading piece. err: ecclient: upload failed (node:12ZBe4we7VeFdthqRxyy1V9uQrA4r8ouNX68xqsk5bD26dkf2Gd, address:145.79.227.21:4497): protocol: context deadline exceeded; context deadline exceeded Mar 18 20:01:59.308437457 (297h1m56.90520847s) [seg_pos={0 0}, upload=332666]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 18 20:01:59.338288467 (297h1m56.93505949s) [upload=332666]: Done uploading segments... Mar 18 20:01:59.338335497 (297h1m56.935106523s) [upload=332666]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 15.767593049s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 56 out of 60 is failed (threshold=6) [XXXXXXXXX.XXX...XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 5.00105595s trace: https://jaeger.dev.storj.io/trace/4a4d327829edffc5 FAIL (2026-03-18 20:12:25 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001075283s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 8 out of 60 is failed (threshold=3) [...............XXX..........X...X.........X....X....X.......] duration: 13.193095177s trace: https://jaeger.dev.storj.io/trace/75a9000feaeb0b3d FAIL (2026-03-18 20:12:07 UTC) operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== ar 18 20:06:04.693955863 (297h6m2.290726886s) [upload=332765]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 20:06:04.693961555 (297h6m2.290732583s) [upload=332765]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 18 20:06:07.394748117 (297h6m4.99151915s) [seg_pos={0 0}, upload=332765]: Begin upload segment... Mar 18 20:06:07.394875884 (297h6m4.991646919s) [seg_pos={0 0}, upload=332765]: Done begin upload segment. Mar 18 20:06:07.394887373 (297h6m4.991658385s) [noise=true , address=45.91.116.9:4276 , piece_id=J356KNX74ADIKN7PPKCRNGACHPVBJYZA3O5OAGVYSURLB4T6KSZA, seg_pos={0 0}, upload=332765]: Uploading piece... Mar 18 20:06:07.394990884 (297h6m4.991761926s) [noise=true , address=145.79.227.2:4129 , piece_id=2Y7VDPW5SS3D4J45EAMSLOEQAHNKYCZJEPP2FTP7GKVCYKMDQTLA, seg_pos={0 0}, upload=332765]: Uploading piece... Mar 18 20:06:07.396053654 (297h6m4.992824681s) [noise=true , address=45.91.116.11:4150 , piece_id=L2ZK2GKN4VY7W6SJ25MR65RL52UH3M66FGQLSLXI2N6LEZXJQVMA, seg_pos={0 0}, upload=332765]: Uploading piece... Mar 18 20:06:07.396081719 (297h6m4.992852749s) [noise=true , address=45.91.116.12:4246 , piece_id=T5BGURHRLDTEZZ4TORP6F5WNBR2TY3ZDWPWO4PGSXQ575BWTUMVA, seg_pos={0 0}, upload=332765]: Uploading piece... Mar 18 20:06:07.396842874 (297h6m4.993613899s) [noise=true , address=145.79.227.33:4875 , piece_id=5LQOGDTQD5CCBMKIZCSLBVQUHQWU6W5ELX6XKLFSFQLU3NVDLKPQ, seg_pos={0 0}, upload=332765]: Uploading piece... Mar 18 20:06:07.397161075 (297h6m4.993932121s) [noise=true , address=77.90.31.12:4615 , piece_id=HLRHRANL5IHJOB5QHLW2OIN5ZP56RKX3INJPWPU5YYHBHRYVJM6A, seg_pos={0 0}, upload=332765]: Uploading piece... Mar 18 20:06:07.397369139 (297h6m4.994140166s) [noise=true , address=77.90.31.11:4567 , piece_id=UN6S6RWYUVLUA5NPFFJ5TASQDZ26OU2IIWP3ZNFUFVJ3WJAA2PSQ, seg_pos={0 0}, upload=332765]: Uploading piece... Mar 18 20:06:07.397732013 (297h6m4.994503037s) [noise=true , address=45.91.116.14:4206 , piece_id=IDCMPAW3BEN5OQ7HUX3QDGFGKN76DM4E6CDN3M45SJQ4VU2OQGCQ, seg_pos={0 0}, upload=332765]: Uploading piece... Mar 18 20:06:07.397757704 (297h6m4.994528729s) [noise=true , address=145.79.227.3:4177 , piece_id=A7HSMKKEMCEJZENJ3CM7N3GZI3NX4BA2WIQZ27MTRONPXBZKZWHA, seg_pos={0 0}, upload=332765]: Uploading piece... Mar 18 20:06:07.398274955 (297h6m4.995045981s) [noise=true , address=193.149.164.71:4307 , piece_id=TZ5I3SSLCLL2AOKJTEFTMEE7DLXHXDKCJJLNP4WHDSGRCJLLBM6A, seg_pos={0 0}, upload=332765]: Uploading piece... Mar 18 20:06:07.398328768 (297h6m4.995099802s) [noise=true , address=79.127.205.233:4173 , piece_id=I64CQ5XLN6DUP2E465ZQLGOVE5NEIHLNWT33QB7WKJF5Q6QZRPLQ, seg_pos={0 0}, upload=332765]: Uploading piece... Mar 18 20:06:07.398815526 (297h6m4.995586656s) [noise=true , address=145.79.227.51:4424 , piece_id=5XCD27Q3GOCFGSHDUNX7QIGUQ6URBU2UREPVX5ZF7CFICF4X4UNQ, seg_pos={0 0}, upload=332765]: Uploading piece... Mar 18 20:06:07.399136293 (297h6m4.995907322s) [noise=true , address=64.49.13.137:4499 , piece_id=KTFNLPG67BVZMYZMJJO4BDRTASHDANE2FBYO3YUV2E4XBCU5NLCA, seg_pos={0 0}, upload=332765]: Uploading piece... Mar 18 20:06:07.399360723 (297h6m4.996131762s) [noise=true , address=45.91.116.4:4963 , piece_id=ZGY2X7JG2MPOJWKMFWP3VATYO3QTEOXPQ4KYZSOW4QOPST6OV2VQ, seg_pos={0 0}, upload=332765]: Uploading piece... Mar 18 20:06:07.399800937 (297h6m4.996571962s) [noise=true , address=145.79.227.24:4592 , piece_id=NGOLKV5BQKREBAYQHBSXHN2UKBHB52WPVBBO7NIFCCYTBF2MXPBA, seg_pos={0 0}, upload=332765]: Uploading piece... Mar 18 20:06:07.399957240 (297h6m4.996728268s) [noise=true , address=193.149.164.66:4993 , piece_id=CRRZAIXMIV673K5JLX743KZ3HFL5MBGUE5ZE4T4Q2WPCSUQ32JQA, seg_pos={0 0}, upload=332765]: Uploading piece... Mar 18 20:06:07.400205078 (297h6m4.996976103s) [noise=true , address=167.17.72.68:4668 , piece_id=KUTKCCK6Y54CMAIIWGINQERXEV6CGVPCOYEJAQUNPBWTHJ755KJQ, seg_pos={0 0}, upload=332765]: Uploading piece... Mar 18 20:06:07.400506584 (297h6m4.997277612s) [noise=true , address=145.79.227.49:4360 , piece_id=OP5BBV7CUZBSMBO6HTTHQCVXDQGS6XO5LGAZ4I2SKUSPGUAKBXKA, seg_pos={0 0}, upload=332765]: Uploading piece... Mar 18 20:06:07.471265449 (297h6m5.068036462s) [upload=332765]: Next returned nil segment Mar 18 20:06:07.471278058 (297h6m5.068049082s) [upload=332765]: Waiting for error group managing segments... Mar 18 20:06:07.475837847 (297h6m5.07260886s) [noise=true , address=77.90.31.11:4567 , piece_id=UN6S6RWYUVLUA5NPFFJ5TASQDZ26OU2IIWP3ZNFUFVJ3WJAA2PSQ, seg_pos={0 0}, upload=332765]: Done uploading piece. err: <nil> Mar 18 20:06:07.476715720 (297h6m5.073486747s) [noise=true , address=45.91.116.4:4963 , piece_id=ZGY2X7JG2MPOJWKMFWP3VATYO3QTEOXPQ4KYZSOW4QOPST6OV2VQ, seg_pos={0 0}, upload=332765]: Done uploading piece. err: <nil> Mar 18 20:06:07.488555609 (297h6m5.085326636s) [noise=true , address=45.91.116.12:4246 , piece_id=T5BGURHRLDTEZZ4TORP6F5WNBR2TY3ZDWPWO4PGSXQ575BWTUMVA, seg_pos={0 0}, upload=332765]: Done uploading piece. err: <nil> Mar 18 20:06:07.489803434 (297h6m5.086574468s) [noise=true , address=45.91.116.14:4206 , piece_id=IDCMPAW3BEN5OQ7HUX3QDGFGKN76DM4E6CDN3M45SJQ4VU2OQGCQ, seg_pos={0 0}, upload=332765]: Done uploading piece. err: <nil> Mar 18 20:06:07.490598252 (297h6m5.087369276s) [noise=true , address=45.91.116.11:4150 , piece_id=L2ZK2GKN4VY7W6SJ25MR65RL52UH3M66FGQLSLXI2N6LEZXJQVMA, seg_pos={0 0}, upload=332765]: Done uploading piece. err: <nil> Mar 18 20:06:07.492135441 (297h6m5.088906454s) [noise=true , address=45.91.116.9:4276 , piece_id=J356KNX74ADIKN7PPKCRNGACHPVBJYZA3O5OAGVYSURLB4T6KSZA, seg_pos={0 0}, upload=332765]: Done uploading piece. err: <nil> Mar 18 20:06:07.508501889 (297h6m5.105272902s) [noise=true , address=77.90.31.12:4615 , piece_id=HLRHRANL5IHJOB5QHLW2OIN5ZP56RKX3INJPWPU5YYHBHRYVJM6A, seg_pos={0 0}, upload=332765]: Done uploading piece. err: <nil> Mar 18 20:06:07.770562428 (297h6m5.367333451s) [noise=true , address=167.17.72.68:4668 , piece_id=KUTKCCK6Y54CMAIIWGINQERXEV6CGVPCOYEJAQUNPBWTHJ755KJQ, seg_pos={0 0}, upload=332765]: Done uploading piece. err: <nil> Mar 18 20:06:08.006027504 (297h6m5.602798531s) [noise=true , address=193.149.164.66:4993 , piece_id=CRRZAIXMIV673K5JLX743KZ3HFL5MBGUE5ZE4T4Q2WPCSUQ32JQA, seg_pos={0 0}, upload=332765]: Done uploading piece. err: <nil> Mar 18 20:06:08.016743696 (297h6m5.613514734s) [noise=true , address=193.149.164.71:4307 , piece_id=TZ5I3SSLCLL2AOKJTEFTMEE7DLXHXDKCJJLNP4WHDSGRCJLLBM6A, seg_pos={0 0}, upload=332765]: Done uploading piece. err: <nil> Mar 18 20:06:08.043257501 (297h6m5.640028528s) [noise=true , address=64.49.13.137:4499 , piece_id=KTFNLPG67BVZMYZMJJO4BDRTASHDANE2FBYO3YUV2E4XBCU5NLCA, seg_pos={0 0}, upload=332765]: Done uploading piece. err: <nil> Mar 18 20:06:08.323806978 (297h6m5.920577991s) [noise=true , address=79.127.205.233:4173 , piece_id=I64CQ5XLN6DUP2E465ZQLGOVE5NEIHLNWT33QB7WKJF5Q6QZRPLQ, seg_pos={0 0}, upload=332765]: Done uploading piece. err: <nil> Mar 18 20:07:04.645185715 (297h7m2.241956747s) [noise=true , address=145.79.227.3:4177 , piece_id=A7HSMKKEMCEJZENJ3CM7N3GZI3NX4BA2WIQZ27MTRONPXBZKZWHA, seg_pos={0 0}, upload=332765]: Done uploading piece. err: ecclient: upload failed (node:12rFuJ2JyBGWtaASJNnTdHoeKUY1mCfDYfJ74BkxK64U9it8FKF, address:145.79.227.3:4177): protocol: rpc: dial tcp 145.79.227.3:4177: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:07:04.645320026 (297h7m2.24209106s) [noise=true , address=145.79.227.24:4592 , piece_id=NGOLKV5BQKREBAYQHBSXHN2UKBHB52WPVBBO7NIFCCYTBF2MXPBA, seg_pos={0 0}, upload=332765]: Done uploading piece. err: ecclient: upload failed (node:1fB7rqiDhMNQ4j9smkctbKdnGCXXdfFPWTgTxRrwQVZ9fKEFeZ, address:145.79.227.24:4592): protocol: rpc: dial tcp 145.79.227.24:4592: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:07:04.645426518 (297h7m2.242197551s) [noise=true , address=145.79.227.2:4129 , piece_id=2Y7VDPW5SS3D4J45EAMSLOEQAHNKYCZJEPP2FTP7GKVCYKMDQTLA, seg_pos={0 0}, upload=332765]: Done uploading piece. err: ecclient: upload failed (node:12GJof7GVfEN9u2yVyWf6j24hnaTdD3ygoPRoicmSqSfcNJFzs6, address:145.79.227.2:4129): protocol: rpc: dial tcp 145.79.227.2:4129: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:07:04.645453468 (297h7m2.242224501s) [noise=true , address=145.79.227.49:4360 , piece_id=OP5BBV7CUZBSMBO6HTTHQCVXDQGS6XO5LGAZ4I2SKUSPGUAKBXKA, seg_pos={0 0}, upload=332765]: Done uploading piece. err: ecclient: upload failed (node:12T24rtYBJ47E6Ef1ZZKMWcpBrWBpCg5QV2ZHniCZZkzJF2yUNk, address:145.79.227.49:4360): protocol: rpc: dial tcp 145.79.227.49:4360: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:07:04.645552758 (297h7m2.242323785s) [noise=true , address=145.79.227.33:4875 , piece_id=5LQOGDTQD5CCBMKIZCSLBVQUHQWU6W5ELX6XKLFSFQLU3NVDLKPQ, seg_pos={0 0}, upload=332765]: Done uploading piece. err: ecclient: upload failed (node:1ijEJgYouyMkUqpLWWz7eH2NGvvs9XpN2ujDjupPykDe5eWcB1, address:145.79.227.33:4875): protocol: rpc: dial tcp 145.79.227.33:4875: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:07:04.645682576 (297h7m2.242453607s) [noise=true , address=145.79.227.51:4424 , piece_id=5XCD27Q3GOCFGSHDUNX7QIGUQ6URBU2UREPVX5ZF7CFICF4X4UNQ, seg_pos={0 0}, upload=332765]: Done uploading piece. err: ecclient: upload failed (node:12LwZixdSPnCm3Wm27PBY79jj52sjkNh2VuQexagieTGBp4Ps2k, address:145.79.227.51:4424): protocol: rpc: dial tcp 145.79.227.51:4424: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:07:04.645716594 (297h7m2.24248761s) [seg_pos={0 0}, upload=332765]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Mar 18 20:07:04.666238325 (297h7m2.263009349s) [upload=332765]: Done uploading segments... Mar 18 20:07:04.666276723 (297h7m2.263047746s) [upload=332765]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 13.193143801s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 8 out of 60 is failed (threshold=3) [.............XXXXXX.XX......................................] duration: 16.11100289s trace: https://jaeger.dev.storj.io/trace/4b4d10a8051ad01b FAIL (2026-03-18 20:12:01 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 16.111034908s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 55 out of 60 is failed (threshold=6) [.XXXXXXXXX.XXX...XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 5.001020726s trace: https://jaeger.dev.storj.io/trace/21026e602891062c FAIL (2026-03-18 20:11:49 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001071601s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 46 out of 60 is failed (threshold=3) [.........XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.XXXXXX...XX.X] duration: 30.037062285s trace: https://jaeger.dev.storj.io/trace/601343143808b18 FAIL (2026-03-18 20:11:47 UTC) operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Mar 18 20:11:17.942948177 (297h11m15.539719208s) [upload=332864]: Starting upload Mar 18 20:11:17.943091638 (297h11m15.539862664s) [upload=332864]: Done starting upload Mar 18 20:11:17.943155714 (297h11m15.53992673s) [upload=332864]: Uploading segments... Mar 18 20:11:17.943462205 (297h11m15.540233222s) [upload=332864]: Got next segment. Inline: false Mar 18 20:11:17.943529204 (297h11m15.540300231s) [upload=332864]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 20:11:17.943536141 (297h11m15.540307167s) [upload=332864]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 18 20:11:21.580830031 (297h11m19.17760108s) [seg_pos={0 0}, upload=332864]: Begin upload segment... Mar 18 20:11:21.580927854 (297h11m19.177698881s) [seg_pos={0 0}, upload=332864]: Done begin upload segment. Mar 18 20:11:21.580991732 (297h11m19.177762763s) [noise=true , address=45.91.116.11:4121 , piece_id=23NVBAV75T2EG6HDNGGB2E224LW2TNUUZVS2Q6AZXBEV56533DCQ, seg_pos={0 0}, upload=332864]: Uploading piece... Mar 18 20:11:21.581000032 (297h11m19.177771061s) [noise=true , address=145.79.227.20:4441 , piece_id=67TR4L5T33FKQQYFZCQX4JBQOCF75OQMX4BESZSUCVGHRF23A4ZA, seg_pos={0 0}, upload=332864]: Uploading piece... Mar 18 20:11:21.581521572 (297h11m19.178292602s) [noise=true , address=77.90.31.18:4808 , piece_id=3EGDWOJ6SMMS7ZAZIAQXWDTPIER2XMIDBOB24ZW3ZF7OXVXUGGOQ, seg_pos={0 0}, upload=332864]: Uploading piece... Mar 18 20:11:21.581999141 (297h11m19.178770178s) [noise=true , address=77.90.31.15:4700 , piece_id=6IAKOFJIJOO2QE3222C4YUEAB22CMYNSNZ6L6ZQZFUEPNOZKYHEA, seg_pos={0 0}, upload=332864]: Uploading piece... Mar 18 20:11:21.582431519 (297h11m19.179202564s) [noise=true , address=145.79.227.35:4481 , piece_id=M5KBTQZCLU3VF5WCQFB3XHDXNAR5XKCRBB4KRC2JYDZOPQ2ECLGA, seg_pos={0 0}, upload=332864]: Uploading piece... Mar 18 20:11:21.582886090 (297h11m19.179657099s) [noise=true , address=79.127.219.41:4930 , piece_id=FMHODO5CFRIBKO4GRJXEI3W5RURLWOQGSOSIQONOKKX5JFYTYEWQ, seg_pos={0 0}, upload=332864]: Uploading piece... Mar 18 20:11:21.583436640 (297h11m19.180207673s) [noise=true , address=45.91.116.9:4265 , piece_id=FKZZKPFHBXRJJ2BWL2GHW5KVSZ4Y6WUQ3YXY6GG2VHD66FDS5IBQ, seg_pos={0 0}, upload=332864]: Uploading piece... Mar 18 20:11:21.583983698 (297h11m19.18075473s) [noise=true , address=45.91.116.8:4047 , piece_id=3SHZ6I45QULZ3YYIGFCQTAFQ23LSXMKY7T2BSGBJDUM2YMKUQNRQ, seg_pos={0 0}, upload=332864]: Uploading piece... Mar 18 20:11:21.584563907 (297h11m19.181334942s) [noise=true , address=145.79.227.34:4907 , piece_id=F2FVEGDWGE467HNTIVPKGCNPGY3CDO74BYGKM5U66Y256NMCUFIA, seg_pos={0 0}, upload=332864]: Uploading piece... Mar 18 20:11:21.586097233 (297h11m19.182868282s) [noise=true , address=77.90.31.21:4880 , piece_id=JJPRYSNNUFWEIUK6BBKLQLS2OHDI7GES74A6UD7OFZDVRW4N7BNA, seg_pos={0 0}, upload=332864]: Uploading piece... Mar 18 20:11:21.586229485 (297h11m19.183000519s) [noise=true , address=45.91.116.7:4009 , piece_id=LRZIZOUYVZUM37PSYFSR5CRCYOAHRPEXWFULVVJDGQEVYDQTRW2Q, seg_pos={0 0}, upload=332864]: Uploading piece... Mar 18 20:11:21.587876633 (297h11m19.184647659s) [noise=true , address=79.127.219.46:4889 , piece_id=L5FCXPYQ2ESKQJQEYWUDSTI7CUAK4WGOD3XE2KKUURXRI2YKEXEA, seg_pos={0 0}, upload=332864]: Uploading piece... Mar 18 20:11:21.589859838 (297h11m19.186630873s) [noise=true , address=45.91.116.12:4227 , piece_id=PCPBXUU4FYD7SKOHU7RGZ2CC2C37Z4XLVET6B5W3MGXTPQABBPDQ, seg_pos={0 0}, upload=332864]: Uploading piece... Mar 18 20:11:21.591680944 (297h11m19.188451994s) [noise=true , address=45.91.116.6:4973 , piece_id=7ANPNX262Y2YIGHMJ5U7MQ3MYPRB2B77KLSCZCIU5MG2CXQ4ID5A, seg_pos={0 0}, upload=332864]: Uploading piece... Mar 18 20:11:21.593792892 (297h11m19.190563928s) [noise=true , address=145.79.227.21:4473 , piece_id=ER4TPXHFHM6X3MWWN2PADSH7PRPV6MMBLSD5WJXB2OAOHRG5OTOQ, seg_pos={0 0}, upload=332864]: Uploading piece... Mar 18 20:11:21.596130805 (297h11m19.192901838s) [noise=true , address=145.79.227.6:4262 , piece_id=ONUXRTAKZOHQASQSWR5YEQIDR7B5TX4YCTV44ISBEEZPSTDGITFQ, seg_pos={0 0}, upload=332864]: Uploading piece... Mar 18 20:11:21.657091067 (297h11m19.253862083s) [upload=332864]: Next returned nil segment Mar 18 20:11:21.657105214 (297h11m19.253876237s) [upload=332864]: Waiting for error group managing segments... Mar 18 20:11:21.662628450 (297h11m19.259399483s) [noise=true , address=45.91.116.12:4227 , piece_id=PCPBXUU4FYD7SKOHU7RGZ2CC2C37Z4XLVET6B5W3MGXTPQABBPDQ, seg_pos={0 0}, upload=332864]: Done uploading piece. err: <nil> Mar 18 20:11:21.662667493 (297h11m19.259438519s) [noise=true , address=45.91.116.11:4121 , piece_id=23NVBAV75T2EG6HDNGGB2E224LW2TNUUZVS2Q6AZXBEV56533DCQ, seg_pos={0 0}, upload=332864]: Done uploading piece. err: <nil> Mar 18 20:11:21.663690046 (297h11m19.260461087s) [noise=true , address=45.91.116.7:4009 , piece_id=LRZIZOUYVZUM37PSYFSR5CRCYOAHRPEXWFULVVJDGQEVYDQTRW2Q, seg_pos={0 0}, upload=332864]: Done uploading piece. err: <nil> Mar 18 20:11:21.664634495 (297h11m19.26140552s) [noise=true , address=77.90.31.15:4700 , piece_id=6IAKOFJIJOO2QE3222C4YUEAB22CMYNSNZ6L6ZQZFUEPNOZKYHEA, seg_pos={0 0}, upload=332864]: Done uploading piece. err: <nil> Mar 18 20:11:21.665900844 (297h11m19.26267186s) [noise=true , address=79.127.219.41:4930 , piece_id=FMHODO5CFRIBKO4GRJXEI3W5RURLWOQGSOSIQONOKKX5JFYTYEWQ, seg_pos={0 0}, upload=332864]: Done uploading piece. err: <nil> Mar 18 20:11:21.666002999 (297h11m19.262774023s) [noise=true , address=79.127.219.46:4889 , piece_id=L5FCXPYQ2ESKQJQEYWUDSTI7CUAK4WGOD3XE2KKUURXRI2YKEXEA, seg_pos={0 0}, upload=332864]: Done uploading piece. err: <nil> Mar 18 20:11:21.668088253 (297h11m19.264859266s) [noise=true , address=45.91.116.9:4265 , piece_id=FKZZKPFHBXRJJ2BWL2GHW5KVSZ4Y6WUQ3YXY6GG2VHD66FDS5IBQ, seg_pos={0 0}, upload=332864]: Done uploading piece. err: <nil> Mar 18 20:11:21.673482814 (297h11m19.270253826s) [noise=true , address=45.91.116.6:4973 , piece_id=7ANPNX262Y2YIGHMJ5U7MQ3MYPRB2B77KLSCZCIU5MG2CXQ4ID5A, seg_pos={0 0}, upload=332864]: Done uploading piece. err: <nil> Mar 18 20:11:21.686416503 (297h11m19.283187526s) [noise=true , address=77.90.31.21:4880 , piece_id=JJPRYSNNUFWEIUK6BBKLQLS2OHDI7GES74A6UD7OFZDVRW4N7BNA, seg_pos={0 0}, upload=332864]: Done uploading piece. err: <nil> Mar 18 20:11:21.693971801 (297h11m19.290742824s) [noise=true , address=45.91.116.8:4047 , piece_id=3SHZ6I45QULZ3YYIGFCQTAFQ23LSXMKY7T2BSGBJDUM2YMKUQNRQ, seg_pos={0 0}, upload=332864]: Done uploading piece. err: <nil> Mar 18 20:11:21.727700946 (297h11m19.324471973s) [noise=true , address=77.90.31.18:4808 , piece_id=3EGDWOJ6SMMS7ZAZIAQXWDTPIER2XMIDBOB24ZW3ZF7OXVXUGGOQ, seg_pos={0 0}, upload=332864]: Done uploading piece. err: <nil> Mar 18 20:11:47.913205594 (297h11m45.509976618s) [noise=true , address=145.79.227.21:4473 , piece_id=ER4TPXHFHM6X3MWWN2PADSH7PRPV6MMBLSD5WJXB2OAOHRG5OTOQ, seg_pos={0 0}, upload=332864]: Done uploading piece. err: ecclient: upload failed (node:12DDe4jT3YeUH8qMQgapcvzMnFs8UFB8fTmubho4ZQf5tYVoxuk, address:145.79.227.21:4473): protocol: rpc: dial tcp 145.79.227.21:4473: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:11:47.913229665 (297h11m45.510000697s) [noise=true , address=145.79.227.6:4262 , piece_id=ONUXRTAKZOHQASQSWR5YEQIDR7B5TX4YCTV44ISBEEZPSTDGITFQ, seg_pos={0 0}, upload=332864]: Done uploading piece. err: ecclient: upload failed (node:12nPw17QDTyrgDBBNKuunFWuDqPV5Rd2GpM7Loc6Ws9vAAZz22s, address:145.79.227.6:4262): protocol: rpc: dial tcp 145.79.227.6:4262: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:11:47.913527933 (297h11m45.510298964s) [noise=true , address=145.79.227.34:4907 , piece_id=F2FVEGDWGE467HNTIVPKGCNPGY3CDO74BYGKM5U66Y256NMCUFIA, seg_pos={0 0}, upload=332864]: Done uploading piece. err: ecclient: upload failed (node:12URx18Ckh7VaAKJ1hdQQucMRu5qiDrAraqaycjYqxETgwZ4uWS, address:145.79.227.34:4907): protocol: rpc: dial tcp 145.79.227.34:4907: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:11:47.913654758 (297h11m45.510425788s) [noise=true , address=145.79.227.35:4481 , piece_id=M5KBTQZCLU3VF5WCQFB3XHDXNAR5XKCRBB4KRC2JYDZOPQ2ECLGA, seg_pos={0 0}, upload=332864]: Done uploading piece. err: ecclient: upload failed (node:12bRK8WWmAnWUwzCfHu8JtZgrruYSJCvhnsYTKcaLccJcv5RQjp, address:145.79.227.35:4481): protocol: rpc: dial tcp 145.79.227.35:4481: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:11:47.914034150 (297h11m45.510805182s) [noise=true , address=145.79.227.20:4441 , piece_id=67TR4L5T33FKQQYFZCQX4JBQOCF75OQMX4BESZSUCVGHRF23A4ZA, seg_pos={0 0}, upload=332864]: Done uploading piece. err: ecclient: upload failed (node:12LjGecRNJE1kq67iMtxZFhWeaUZpUogsGxbz4VXwnHdQtTnbBr, address:145.79.227.20:4441): protocol: context deadline exceeded; context deadline exceeded Mar 18 20:11:47.914082052 (297h11m45.510853068s) [seg_pos={0 0}, upload=332864]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Mar 18 20:11:47.947299775 (297h11m45.5440708s) [upload=332864]: Done uploading segments... Mar 18 20:11:47.947420462 (297h11m45.544191496s) [upload=332864]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 30.037091745s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 8 out of 60 is failed (threshold=3) [................XXX..........X...X.........X....X....X......] duration: 19.786325558s trace: https://jaeger.dev.storj.io/trace/75a9000feaeb0b3d FAIL (2026-03-18 20:11:37 UTC) operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== ar 18 20:06:04.693955863 (297h6m2.290726886s) [upload=332765]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 20:06:04.693961555 (297h6m2.290732583s) [upload=332765]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 18 20:06:07.394748117 (297h6m4.99151915s) [seg_pos={0 0}, upload=332765]: Begin upload segment... Mar 18 20:06:07.394875884 (297h6m4.991646919s) [seg_pos={0 0}, upload=332765]: Done begin upload segment. Mar 18 20:06:07.394887373 (297h6m4.991658385s) [noise=true , address=45.91.116.9:4276 , piece_id=J356KNX74ADIKN7PPKCRNGACHPVBJYZA3O5OAGVYSURLB4T6KSZA, seg_pos={0 0}, upload=332765]: Uploading piece... Mar 18 20:06:07.394990884 (297h6m4.991761926s) [noise=true , address=145.79.227.2:4129 , piece_id=2Y7VDPW5SS3D4J45EAMSLOEQAHNKYCZJEPP2FTP7GKVCYKMDQTLA, seg_pos={0 0}, upload=332765]: Uploading piece... Mar 18 20:06:07.396053654 (297h6m4.992824681s) [noise=true , address=45.91.116.11:4150 , piece_id=L2ZK2GKN4VY7W6SJ25MR65RL52UH3M66FGQLSLXI2N6LEZXJQVMA, seg_pos={0 0}, upload=332765]: Uploading piece... Mar 18 20:06:07.396081719 (297h6m4.992852749s) [noise=true , address=45.91.116.12:4246 , piece_id=T5BGURHRLDTEZZ4TORP6F5WNBR2TY3ZDWPWO4PGSXQ575BWTUMVA, seg_pos={0 0}, upload=332765]: Uploading piece... Mar 18 20:06:07.396842874 (297h6m4.993613899s) [noise=true , address=145.79.227.33:4875 , piece_id=5LQOGDTQD5CCBMKIZCSLBVQUHQWU6W5ELX6XKLFSFQLU3NVDLKPQ, seg_pos={0 0}, upload=332765]: Uploading piece... Mar 18 20:06:07.397161075 (297h6m4.993932121s) [noise=true , address=77.90.31.12:4615 , piece_id=HLRHRANL5IHJOB5QHLW2OIN5ZP56RKX3INJPWPU5YYHBHRYVJM6A, seg_pos={0 0}, upload=332765]: Uploading piece... Mar 18 20:06:07.397369139 (297h6m4.994140166s) [noise=true , address=77.90.31.11:4567 , piece_id=UN6S6RWYUVLUA5NPFFJ5TASQDZ26OU2IIWP3ZNFUFVJ3WJAA2PSQ, seg_pos={0 0}, upload=332765]: Uploading piece... Mar 18 20:06:07.397732013 (297h6m4.994503037s) [noise=true , address=45.91.116.14:4206 , piece_id=IDCMPAW3BEN5OQ7HUX3QDGFGKN76DM4E6CDN3M45SJQ4VU2OQGCQ, seg_pos={0 0}, upload=332765]: Uploading piece... Mar 18 20:06:07.397757704 (297h6m4.994528729s) [noise=true , address=145.79.227.3:4177 , piece_id=A7HSMKKEMCEJZENJ3CM7N3GZI3NX4BA2WIQZ27MTRONPXBZKZWHA, seg_pos={0 0}, upload=332765]: Uploading piece... Mar 18 20:06:07.398274955 (297h6m4.995045981s) [noise=true , address=193.149.164.71:4307 , piece_id=TZ5I3SSLCLL2AOKJTEFTMEE7DLXHXDKCJJLNP4WHDSGRCJLLBM6A, seg_pos={0 0}, upload=332765]: Uploading piece... Mar 18 20:06:07.398328768 (297h6m4.995099802s) [noise=true , address=79.127.205.233:4173 , piece_id=I64CQ5XLN6DUP2E465ZQLGOVE5NEIHLNWT33QB7WKJF5Q6QZRPLQ, seg_pos={0 0}, upload=332765]: Uploading piece... Mar 18 20:06:07.398815526 (297h6m4.995586656s) [noise=true , address=145.79.227.51:4424 , piece_id=5XCD27Q3GOCFGSHDUNX7QIGUQ6URBU2UREPVX5ZF7CFICF4X4UNQ, seg_pos={0 0}, upload=332765]: Uploading piece... Mar 18 20:06:07.399136293 (297h6m4.995907322s) [noise=true , address=64.49.13.137:4499 , piece_id=KTFNLPG67BVZMYZMJJO4BDRTASHDANE2FBYO3YUV2E4XBCU5NLCA, seg_pos={0 0}, upload=332765]: Uploading piece... Mar 18 20:06:07.399360723 (297h6m4.996131762s) [noise=true , address=45.91.116.4:4963 , piece_id=ZGY2X7JG2MPOJWKMFWP3VATYO3QTEOXPQ4KYZSOW4QOPST6OV2VQ, seg_pos={0 0}, upload=332765]: Uploading piece... Mar 18 20:06:07.399800937 (297h6m4.996571962s) [noise=true , address=145.79.227.24:4592 , piece_id=NGOLKV5BQKREBAYQHBSXHN2UKBHB52WPVBBO7NIFCCYTBF2MXPBA, seg_pos={0 0}, upload=332765]: Uploading piece... Mar 18 20:06:07.399957240 (297h6m4.996728268s) [noise=true , address=193.149.164.66:4993 , piece_id=CRRZAIXMIV673K5JLX743KZ3HFL5MBGUE5ZE4T4Q2WPCSUQ32JQA, seg_pos={0 0}, upload=332765]: Uploading piece... Mar 18 20:06:07.400205078 (297h6m4.996976103s) [noise=true , address=167.17.72.68:4668 , piece_id=KUTKCCK6Y54CMAIIWGINQERXEV6CGVPCOYEJAQUNPBWTHJ755KJQ, seg_pos={0 0}, upload=332765]: Uploading piece... Mar 18 20:06:07.400506584 (297h6m4.997277612s) [noise=true , address=145.79.227.49:4360 , piece_id=OP5BBV7CUZBSMBO6HTTHQCVXDQGS6XO5LGAZ4I2SKUSPGUAKBXKA, seg_pos={0 0}, upload=332765]: Uploading piece... Mar 18 20:06:07.471265449 (297h6m5.068036462s) [upload=332765]: Next returned nil segment Mar 18 20:06:07.471278058 (297h6m5.068049082s) [upload=332765]: Waiting for error group managing segments... Mar 18 20:06:07.475837847 (297h6m5.07260886s) [noise=true , address=77.90.31.11:4567 , piece_id=UN6S6RWYUVLUA5NPFFJ5TASQDZ26OU2IIWP3ZNFUFVJ3WJAA2PSQ, seg_pos={0 0}, upload=332765]: Done uploading piece. err: <nil> Mar 18 20:06:07.476715720 (297h6m5.073486747s) [noise=true , address=45.91.116.4:4963 , piece_id=ZGY2X7JG2MPOJWKMFWP3VATYO3QTEOXPQ4KYZSOW4QOPST6OV2VQ, seg_pos={0 0}, upload=332765]: Done uploading piece. err: <nil> Mar 18 20:06:07.488555609 (297h6m5.085326636s) [noise=true , address=45.91.116.12:4246 , piece_id=T5BGURHRLDTEZZ4TORP6F5WNBR2TY3ZDWPWO4PGSXQ575BWTUMVA, seg_pos={0 0}, upload=332765]: Done uploading piece. err: <nil> Mar 18 20:06:07.489803434 (297h6m5.086574468s) [noise=true , address=45.91.116.14:4206 , piece_id=IDCMPAW3BEN5OQ7HUX3QDGFGKN76DM4E6CDN3M45SJQ4VU2OQGCQ, seg_pos={0 0}, upload=332765]: Done uploading piece. err: <nil> Mar 18 20:06:07.490598252 (297h6m5.087369276s) [noise=true , address=45.91.116.11:4150 , piece_id=L2ZK2GKN4VY7W6SJ25MR65RL52UH3M66FGQLSLXI2N6LEZXJQVMA, seg_pos={0 0}, upload=332765]: Done uploading piece. err: <nil> Mar 18 20:06:07.492135441 (297h6m5.088906454s) [noise=true , address=45.91.116.9:4276 , piece_id=J356KNX74ADIKN7PPKCRNGACHPVBJYZA3O5OAGVYSURLB4T6KSZA, seg_pos={0 0}, upload=332765]: Done uploading piece. err: <nil> Mar 18 20:06:07.508501889 (297h6m5.105272902s) [noise=true , address=77.90.31.12:4615 , piece_id=HLRHRANL5IHJOB5QHLW2OIN5ZP56RKX3INJPWPU5YYHBHRYVJM6A, seg_pos={0 0}, upload=332765]: Done uploading piece. err: <nil> Mar 18 20:06:07.770562428 (297h6m5.367333451s) [noise=true , address=167.17.72.68:4668 , piece_id=KUTKCCK6Y54CMAIIWGINQERXEV6CGVPCOYEJAQUNPBWTHJ755KJQ, seg_pos={0 0}, upload=332765]: Done uploading piece. err: <nil> Mar 18 20:06:08.006027504 (297h6m5.602798531s) [noise=true , address=193.149.164.66:4993 , piece_id=CRRZAIXMIV673K5JLX743KZ3HFL5MBGUE5ZE4T4Q2WPCSUQ32JQA, seg_pos={0 0}, upload=332765]: Done uploading piece. err: <nil> Mar 18 20:06:08.016743696 (297h6m5.613514734s) [noise=true , address=193.149.164.71:4307 , piece_id=TZ5I3SSLCLL2AOKJTEFTMEE7DLXHXDKCJJLNP4WHDSGRCJLLBM6A, seg_pos={0 0}, upload=332765]: Done uploading piece. err: <nil> Mar 18 20:06:08.043257501 (297h6m5.640028528s) [noise=true , address=64.49.13.137:4499 , piece_id=KTFNLPG67BVZMYZMJJO4BDRTASHDANE2FBYO3YUV2E4XBCU5NLCA, seg_pos={0 0}, upload=332765]: Done uploading piece. err: <nil> Mar 18 20:06:08.323806978 (297h6m5.920577991s) [noise=true , address=79.127.205.233:4173 , piece_id=I64CQ5XLN6DUP2E465ZQLGOVE5NEIHLNWT33QB7WKJF5Q6QZRPLQ, seg_pos={0 0}, upload=332765]: Done uploading piece. err: <nil> Mar 18 20:07:04.645185715 (297h7m2.241956747s) [noise=true , address=145.79.227.3:4177 , piece_id=A7HSMKKEMCEJZENJ3CM7N3GZI3NX4BA2WIQZ27MTRONPXBZKZWHA, seg_pos={0 0}, upload=332765]: Done uploading piece. err: ecclient: upload failed (node:12rFuJ2JyBGWtaASJNnTdHoeKUY1mCfDYfJ74BkxK64U9it8FKF, address:145.79.227.3:4177): protocol: rpc: dial tcp 145.79.227.3:4177: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:07:04.645320026 (297h7m2.24209106s) [noise=true , address=145.79.227.24:4592 , piece_id=NGOLKV5BQKREBAYQHBSXHN2UKBHB52WPVBBO7NIFCCYTBF2MXPBA, seg_pos={0 0}, upload=332765]: Done uploading piece. err: ecclient: upload failed (node:1fB7rqiDhMNQ4j9smkctbKdnGCXXdfFPWTgTxRrwQVZ9fKEFeZ, address:145.79.227.24:4592): protocol: rpc: dial tcp 145.79.227.24:4592: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:07:04.645426518 (297h7m2.242197551s) [noise=true , address=145.79.227.2:4129 , piece_id=2Y7VDPW5SS3D4J45EAMSLOEQAHNKYCZJEPP2FTP7GKVCYKMDQTLA, seg_pos={0 0}, upload=332765]: Done uploading piece. err: ecclient: upload failed (node:12GJof7GVfEN9u2yVyWf6j24hnaTdD3ygoPRoicmSqSfcNJFzs6, address:145.79.227.2:4129): protocol: rpc: dial tcp 145.79.227.2:4129: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:07:04.645453468 (297h7m2.242224501s) [noise=true , address=145.79.227.49:4360 , piece_id=OP5BBV7CUZBSMBO6HTTHQCVXDQGS6XO5LGAZ4I2SKUSPGUAKBXKA, seg_pos={0 0}, upload=332765]: Done uploading piece. err: ecclient: upload failed (node:12T24rtYBJ47E6Ef1ZZKMWcpBrWBpCg5QV2ZHniCZZkzJF2yUNk, address:145.79.227.49:4360): protocol: rpc: dial tcp 145.79.227.49:4360: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:07:04.645552758 (297h7m2.242323785s) [noise=true , address=145.79.227.33:4875 , piece_id=5LQOGDTQD5CCBMKIZCSLBVQUHQWU6W5ELX6XKLFSFQLU3NVDLKPQ, seg_pos={0 0}, upload=332765]: Done uploading piece. err: ecclient: upload failed (node:1ijEJgYouyMkUqpLWWz7eH2NGvvs9XpN2ujDjupPykDe5eWcB1, address:145.79.227.33:4875): protocol: rpc: dial tcp 145.79.227.33:4875: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:07:04.645682576 (297h7m2.242453607s) [noise=true , address=145.79.227.51:4424 , piece_id=5XCD27Q3GOCFGSHDUNX7QIGUQ6URBU2UREPVX5ZF7CFICF4X4UNQ, seg_pos={0 0}, upload=332765]: Done uploading piece. err: ecclient: upload failed (node:12LwZixdSPnCm3Wm27PBY79jj52sjkNh2VuQexagieTGBp4Ps2k, address:145.79.227.51:4424): protocol: rpc: dial tcp 145.79.227.51:4424: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:07:04.645716594 (297h7m2.24248761s) [seg_pos={0 0}, upload=332765]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Mar 18 20:07:04.666238325 (297h7m2.263009349s) [upload=332765]: Done uploading segments... Mar 18 20:07:04.666276723 (297h7m2.263047746s) [upload=332765]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 19.786355704s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 8 out of 60 is failed (threshold=3) [..............XXXXXX.XX.....................................] duration: 15.979902321s trace: https://jaeger.dev.storj.io/trace/4b4d10a8051ad01b FAIL (2026-03-18 20:11:32 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 15.97992624s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 6 out of 60 is failed (threshold=3) [..................X..........XX..X....X........X............] duration: 18.576427328s trace: https://jaeger.dev.storj.io/trace/589419efc52e5e2b FAIL (2026-03-18 20:11:30 UTC) operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== Mar 18 20:00:59.343084966 (297h0m56.939855989s) [upload=332666]: Starting upload Mar 18 20:00:59.343175518 (297h0m56.939946548s) [upload=332666]: Done starting upload Mar 18 20:00:59.343565585 (297h0m56.940336612s) [upload=332666]: Uploading segments... Mar 18 20:00:59.344256647 (297h0m56.941027676s) [upload=332666]: Got next segment. Inline: false Mar 18 20:00:59.344352215 (297h0m56.941123241s) [upload=332666]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 20:00:59.344358706 (297h0m56.941129732s) [upload=332666]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 18 20:01:02.340259299 (297h0m59.937030329s) [seg_pos={0 0}, upload=332666]: Begin upload segment... Mar 18 20:01:02.340368743 (297h0m59.937139767s) [seg_pos={0 0}, upload=332666]: Done begin upload segment. Mar 18 20:01:02.340479563 (297h0m59.93725059s) [noise=true , address=45.91.116.8:4061 , piece_id=6XFVKT4AAO7QN457OQMRKPI6R24TJTAN47UXLEVOEFQVOS6M6C3Q, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.340523837 (297h0m59.93729486s) [noise=true , address=79.127.205.233:4166 , piece_id=SFGK5YCWHFRVXF6QNHYP2CYJCZLSTQ76OYRVJPWWXQTF2KFSFJKA, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.341468438 (297h0m59.938239447s) [noise=true , address=45.91.116.4:4961 , piece_id=ZMUGQECUVPVHDNKN75XPBET5M2HFXXK4WMTP4V5KEK63JVZCX5EA, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.341520025 (297h0m59.938291058s) [noise=true , address=167.17.72.71:4815 , piece_id=CXGTYNNBILN3TT7I6Z7UI75F2ZCS4QUTLS6KC4FJ5IHS3XOYWDJQ, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.342232992 (297h0m59.939004023s) [noise=true , address=145.79.227.36:4524 , piece_id=3TAV3P55ITTHBAZEUUUTOAHJRGQXJXXVGZJP6AVR2ANVNOR3S54Q, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.342917368 (297h0m59.939688398s) [noise=true , address=193.149.164.71:4297 , piece_id=4IJPXLPCX3LKFOBOSRHK2DTNRKH67K7FXJUKQ3JPRWHS47ZFEIDA, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.343244957 (297h0m59.940015982s) [noise=true , address=145.79.227.33:4892 , piece_id=DQ6ALC3VOAGFIJI3IKR6I7QFYM22C66QWAAXMH6NXI2FIQEZNQYA, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.344270079 (297h0m59.941041117s) [noise=true , address=145.79.227.21:4497 , piece_id=SP6EKJB7W4QXF7OIBVKEFWCPOZFL5IRIEDVAGS2M4QJSO34CUVRQ, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.345025057 (297h0m59.941796084s) [noise=true , address=77.90.31.17:4793 , piece_id=2U4IBUA2GGUGDBXQGIGQKUK4ZR3L7CDKC6MTGIKAUME4F7MFZRLA, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.345115802 (297h0m59.941886836s) [noise=true , address=167.17.72.69:4731 , piece_id=VOEVIKKFJMC2Y3TPQIDMYUP37NLFU7KPRH4HRRSMQLWUKTSLOH2Q, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.345690600 (297h0m59.942461634s) [noise=true , address=77.90.31.13:4648 , piece_id=BPMB5CFOYHFT3BS4ORR6JF6BNFC2AOAJZOSPMG67NAMLGGP5XVEQ, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.346311592 (297h0m59.943082623s) [noise=true , address=79.127.205.234:4202 , piece_id=YY4LYWX26O6RIOBONESNKTUZEFZCRL4UUX3NEBQ53AJV4IEDG5MQ, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.346791142 (297h0m59.943562166s) [noise=true , address=45.91.116.10:4110 , piece_id=UYLMDCWJEJOHGICA37KNH7G5AQXNIPRH5NRYLBUUMGDRVSLTEYHQ, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.347094999 (297h0m59.94386607s) [noise=true , address=145.79.227.17:4987 , piece_id=PT3TWVB65JZFEWVH4IUZEU2N53FQ57QBRWVLWAUFP4UDLY3LWNLQ, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.347781281 (297h0m59.944552305s) [noise=true , address=79.127.219.46:4919 , piece_id=KK6HRIVBMYLIHRHGPQH3AWMNSVHTTIBESHDMJZA72T446RX6QKRQ, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.348263123 (297h0m59.945034137s) [noise=true , address=193.149.164.68:4033 , piece_id=JZPQKPYGO3W3RZZBITQ73X2XEPIBBSLGRMWIVOM4Z3GTV5UFKRTQ, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.348915505 (297h0m59.945686536s) [noise=true , address=145.79.227.7:4312 , piece_id=2FAL6E3RMBTBHZBXAJJVH2N5VGF5VYI3KNZXGNWCMWGBBPAUO2NQ, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.349366298 (297h0m59.946137337s) [noise=true , address=193.149.164.67:4007 , piece_id=HURQGH67I5B6V4YUV3PRZL3KSKN3XTSK3UMCHZRNT6L3W3PXXHPQ, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.429834098 (297h1m0.026605126s) [upload=332666]: Next returned nil segment Mar 18 20:01:02.429853411 (297h1m0.026624442s) [upload=332666]: Waiting for error group managing segments... Mar 18 20:01:02.446867680 (297h1m0.043638704s) [noise=true , address=45.91.116.10:4110 , piece_id=UYLMDCWJEJOHGICA37KNH7G5AQXNIPRH5NRYLBUUMGDRVSLTEYHQ, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil> Mar 18 20:01:02.451665165 (297h1m0.048436196s) [noise=true , address=45.91.116.4:4961 , piece_id=ZMUGQECUVPVHDNKN75XPBET5M2HFXXK4WMTP4V5KEK63JVZCX5EA, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil> Mar 18 20:01:02.452538886 (297h1m0.049309913s) [noise=true , address=77.90.31.17:4793 , piece_id=2U4IBUA2GGUGDBXQGIGQKUK4ZR3L7CDKC6MTGIKAUME4F7MFZRLA, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil> Mar 18 20:01:02.455597040 (297h1m0.052368067s) [noise=true , address=45.91.116.8:4061 , piece_id=6XFVKT4AAO7QN457OQMRKPI6R24TJTAN47UXLEVOEFQVOS6M6C3Q, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil> Mar 18 20:01:02.457848739 (297h1m0.054619766s) [noise=true , address=77.90.31.13:4648 , piece_id=BPMB5CFOYHFT3BS4ORR6JF6BNFC2AOAJZOSPMG67NAMLGGP5XVEQ, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil> Mar 18 20:01:02.461479709 (297h1m0.058250746s) [noise=true , address=79.127.219.46:4919 , piece_id=KK6HRIVBMYLIHRHGPQH3AWMNSVHTTIBESHDMJZA72T446RX6QKRQ, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil> Mar 18 20:01:02.676859466 (297h1m0.27363049s) [noise=true , address=167.17.72.69:4731 , piece_id=VOEVIKKFJMC2Y3TPQIDMYUP37NLFU7KPRH4HRRSMQLWUKTSLOH2Q, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil> Mar 18 20:01:02.909860743 (297h1m0.506631766s) [noise=true , address=79.127.205.234:4202 , piece_id=YY4LYWX26O6RIOBONESNKTUZEFZCRL4UUX3NEBQ53AJV4IEDG5MQ, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil> Mar 18 20:01:02.930427742 (297h1m0.527198754s) [noise=true , address=193.149.164.71:4297 , piece_id=4IJPXLPCX3LKFOBOSRHK2DTNRKH67K7FXJUKQ3JPRWHS47ZFEIDA, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil> Mar 18 20:01:02.944696889 (297h1m0.541467914s) [noise=true , address=193.149.164.67:4007 , piece_id=HURQGH67I5B6V4YUV3PRZL3KSKN3XTSK3UMCHZRNT6L3W3PXXHPQ, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil> Mar 18 20:01:02.951319183 (297h1m0.548090212s) [noise=true , address=193.149.164.68:4033 , piece_id=JZPQKPYGO3W3RZZBITQ73X2XEPIBBSLGRMWIVOM4Z3GTV5UFKRTQ, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil> Mar 18 20:01:02.975619436 (297h1m0.572390467s) [noise=true , address=79.127.205.233:4166 , piece_id=SFGK5YCWHFRVXF6QNHYP2CYJCZLSTQ76OYRVJPWWXQTF2KFSFJKA, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil> Mar 18 20:01:03.819918528 (297h1m1.416689564s) [noise=true , address=167.17.72.71:4815 , piece_id=CXGTYNNBILN3TT7I6Z7UI75F2ZCS4QUTLS6KC4FJ5IHS3XOYWDJQ, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil> Mar 18 20:01:59.307873420 (297h1m56.904644452s) [noise=true , address=145.79.227.7:4312 , piece_id=2FAL6E3RMBTBHZBXAJJVH2N5VGF5VYI3KNZXGNWCMWGBBPAUO2NQ, seg_pos={0 0}, upload=332666]: Done uploading piece. err: ecclient: upload failed (node:12AVp3qn5uwLPqYaZEwzcgoGMJhNga1BUh6bK5k2bxjWgZLQiiC, address:145.79.227.7:4312): protocol: rpc: dial tcp 145.79.227.7:4312: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:01:59.308026092 (297h1m56.904797124s) [noise=true , address=145.79.227.17:4987 , piece_id=PT3TWVB65JZFEWVH4IUZEU2N53FQ57QBRWVLWAUFP4UDLY3LWNLQ, seg_pos={0 0}, upload=332666]: Done uploading piece. err: ecclient: upload failed (node:1XsPVNGYUhKnXkQRfVfjyBRzyvhShSdsrnzXtwe4Xtf5MaHMvH, address:145.79.227.17:4987): protocol: rpc: dial tcp 145.79.227.17:4987: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:01:59.308195864 (297h1m56.904966894s) [noise=true , address=145.79.227.36:4524 , piece_id=3TAV3P55ITTHBAZEUUUTOAHJRGQXJXXVGZJP6AVR2ANVNOR3S54Q, seg_pos={0 0}, upload=332666]: Done uploading piece. err: ecclient: upload failed (node:1xfbCzjSwFDRrR1gPggeN8tNCLbRRWDVzCJ6mqhZsbVSe8d4s4, address:145.79.227.36:4524): protocol: rpc: dial tcp 145.79.227.36:4524: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:01:59.308316037 (297h1m56.90508706s) [noise=true , address=145.79.227.33:4892 , piece_id=DQ6ALC3VOAGFIJI3IKR6I7QFYM22C66QWAAXMH6NXI2FIQEZNQYA, seg_pos={0 0}, upload=332666]: Done uploading piece. err: ecclient: upload failed (node:12RHwFxcUjPsn12uHEAH3fBDJKv1v1tM8mRJXoFrZrG5djdswF5, address:145.79.227.33:4892): protocol: rpc: dial tcp 145.79.227.33:4892: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:01:59.308404403 (297h1m56.905175433s) [noise=true , address=145.79.227.21:4497 , piece_id=SP6EKJB7W4QXF7OIBVKEFWCPOZFL5IRIEDVAGS2M4QJSO34CUVRQ, seg_pos={0 0}, upload=332666]: Done uploading piece. err: ecclient: upload failed (node:12ZBe4we7VeFdthqRxyy1V9uQrA4r8ouNX68xqsk5bD26dkf2Gd, address:145.79.227.21:4497): protocol: context deadline exceeded; context deadline exceeded Mar 18 20:01:59.308437457 (297h1m56.90520847s) [seg_pos={0 0}, upload=332666]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 18 20:01:59.338288467 (297h1m56.93505949s) [upload=332666]: Done uploading segments... Mar 18 20:01:59.338335497 (297h1m56.935106523s) [upload=332666]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 18.576458275s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 8 out of 60 is failed (threshold=3) [.................XXX..........X...X.........X....X....X.....] duration: 20.185666516s trace: https://jaeger.dev.storj.io/trace/75a9000feaeb0b3d FAIL (2026-03-18 20:11:01 UTC) operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== ar 18 20:06:04.693955863 (297h6m2.290726886s) [upload=332765]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 20:06:04.693961555 (297h6m2.290732583s) [upload=332765]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 18 20:06:07.394748117 (297h6m4.99151915s) [seg_pos={0 0}, upload=332765]: Begin upload segment... Mar 18 20:06:07.394875884 (297h6m4.991646919s) [seg_pos={0 0}, upload=332765]: Done begin upload segment. Mar 18 20:06:07.394887373 (297h6m4.991658385s) [noise=true , address=45.91.116.9:4276 , piece_id=J356KNX74ADIKN7PPKCRNGACHPVBJYZA3O5OAGVYSURLB4T6KSZA, seg_pos={0 0}, upload=332765]: Uploading piece... Mar 18 20:06:07.394990884 (297h6m4.991761926s) [noise=true , address=145.79.227.2:4129 , piece_id=2Y7VDPW5SS3D4J45EAMSLOEQAHNKYCZJEPP2FTP7GKVCYKMDQTLA, seg_pos={0 0}, upload=332765]: Uploading piece... Mar 18 20:06:07.396053654 (297h6m4.992824681s) [noise=true , address=45.91.116.11:4150 , piece_id=L2ZK2GKN4VY7W6SJ25MR65RL52UH3M66FGQLSLXI2N6LEZXJQVMA, seg_pos={0 0}, upload=332765]: Uploading piece... Mar 18 20:06:07.396081719 (297h6m4.992852749s) [noise=true , address=45.91.116.12:4246 , piece_id=T5BGURHRLDTEZZ4TORP6F5WNBR2TY3ZDWPWO4PGSXQ575BWTUMVA, seg_pos={0 0}, upload=332765]: Uploading piece... Mar 18 20:06:07.396842874 (297h6m4.993613899s) [noise=true , address=145.79.227.33:4875 , piece_id=5LQOGDTQD5CCBMKIZCSLBVQUHQWU6W5ELX6XKLFSFQLU3NVDLKPQ, seg_pos={0 0}, upload=332765]: Uploading piece... Mar 18 20:06:07.397161075 (297h6m4.993932121s) [noise=true , address=77.90.31.12:4615 , piece_id=HLRHRANL5IHJOB5QHLW2OIN5ZP56RKX3INJPWPU5YYHBHRYVJM6A, seg_pos={0 0}, upload=332765]: Uploading piece... Mar 18 20:06:07.397369139 (297h6m4.994140166s) [noise=true , address=77.90.31.11:4567 , piece_id=UN6S6RWYUVLUA5NPFFJ5TASQDZ26OU2IIWP3ZNFUFVJ3WJAA2PSQ, seg_pos={0 0}, upload=332765]: Uploading piece... Mar 18 20:06:07.397732013 (297h6m4.994503037s) [noise=true , address=45.91.116.14:4206 , piece_id=IDCMPAW3BEN5OQ7HUX3QDGFGKN76DM4E6CDN3M45SJQ4VU2OQGCQ, seg_pos={0 0}, upload=332765]: Uploading piece... Mar 18 20:06:07.397757704 (297h6m4.994528729s) [noise=true , address=145.79.227.3:4177 , piece_id=A7HSMKKEMCEJZENJ3CM7N3GZI3NX4BA2WIQZ27MTRONPXBZKZWHA, seg_pos={0 0}, upload=332765]: Uploading piece... Mar 18 20:06:07.398274955 (297h6m4.995045981s) [noise=true , address=193.149.164.71:4307 , piece_id=TZ5I3SSLCLL2AOKJTEFTMEE7DLXHXDKCJJLNP4WHDSGRCJLLBM6A, seg_pos={0 0}, upload=332765]: Uploading piece... Mar 18 20:06:07.398328768 (297h6m4.995099802s) [noise=true , address=79.127.205.233:4173 , piece_id=I64CQ5XLN6DUP2E465ZQLGOVE5NEIHLNWT33QB7WKJF5Q6QZRPLQ, seg_pos={0 0}, upload=332765]: Uploading piece... Mar 18 20:06:07.398815526 (297h6m4.995586656s) [noise=true , address=145.79.227.51:4424 , piece_id=5XCD27Q3GOCFGSHDUNX7QIGUQ6URBU2UREPVX5ZF7CFICF4X4UNQ, seg_pos={0 0}, upload=332765]: Uploading piece... Mar 18 20:06:07.399136293 (297h6m4.995907322s) [noise=true , address=64.49.13.137:4499 , piece_id=KTFNLPG67BVZMYZMJJO4BDRTASHDANE2FBYO3YUV2E4XBCU5NLCA, seg_pos={0 0}, upload=332765]: Uploading piece... Mar 18 20:06:07.399360723 (297h6m4.996131762s) [noise=true , address=45.91.116.4:4963 , piece_id=ZGY2X7JG2MPOJWKMFWP3VATYO3QTEOXPQ4KYZSOW4QOPST6OV2VQ, seg_pos={0 0}, upload=332765]: Uploading piece... Mar 18 20:06:07.399800937 (297h6m4.996571962s) [noise=true , address=145.79.227.24:4592 , piece_id=NGOLKV5BQKREBAYQHBSXHN2UKBHB52WPVBBO7NIFCCYTBF2MXPBA, seg_pos={0 0}, upload=332765]: Uploading piece... Mar 18 20:06:07.399957240 (297h6m4.996728268s) [noise=true , address=193.149.164.66:4993 , piece_id=CRRZAIXMIV673K5JLX743KZ3HFL5MBGUE5ZE4T4Q2WPCSUQ32JQA, seg_pos={0 0}, upload=332765]: Uploading piece... Mar 18 20:06:07.400205078 (297h6m4.996976103s) [noise=true , address=167.17.72.68:4668 , piece_id=KUTKCCK6Y54CMAIIWGINQERXEV6CGVPCOYEJAQUNPBWTHJ755KJQ, seg_pos={0 0}, upload=332765]: Uploading piece... Mar 18 20:06:07.400506584 (297h6m4.997277612s) [noise=true , address=145.79.227.49:4360 , piece_id=OP5BBV7CUZBSMBO6HTTHQCVXDQGS6XO5LGAZ4I2SKUSPGUAKBXKA, seg_pos={0 0}, upload=332765]: Uploading piece... Mar 18 20:06:07.471265449 (297h6m5.068036462s) [upload=332765]: Next returned nil segment Mar 18 20:06:07.471278058 (297h6m5.068049082s) [upload=332765]: Waiting for error group managing segments... Mar 18 20:06:07.475837847 (297h6m5.07260886s) [noise=true , address=77.90.31.11:4567 , piece_id=UN6S6RWYUVLUA5NPFFJ5TASQDZ26OU2IIWP3ZNFUFVJ3WJAA2PSQ, seg_pos={0 0}, upload=332765]: Done uploading piece. err: <nil> Mar 18 20:06:07.476715720 (297h6m5.073486747s) [noise=true , address=45.91.116.4:4963 , piece_id=ZGY2X7JG2MPOJWKMFWP3VATYO3QTEOXPQ4KYZSOW4QOPST6OV2VQ, seg_pos={0 0}, upload=332765]: Done uploading piece. err: <nil> Mar 18 20:06:07.488555609 (297h6m5.085326636s) [noise=true , address=45.91.116.12:4246 , piece_id=T5BGURHRLDTEZZ4TORP6F5WNBR2TY3ZDWPWO4PGSXQ575BWTUMVA, seg_pos={0 0}, upload=332765]: Done uploading piece. err: <nil> Mar 18 20:06:07.489803434 (297h6m5.086574468s) [noise=true , address=45.91.116.14:4206 , piece_id=IDCMPAW3BEN5OQ7HUX3QDGFGKN76DM4E6CDN3M45SJQ4VU2OQGCQ, seg_pos={0 0}, upload=332765]: Done uploading piece. err: <nil> Mar 18 20:06:07.490598252 (297h6m5.087369276s) [noise=true , address=45.91.116.11:4150 , piece_id=L2ZK2GKN4VY7W6SJ25MR65RL52UH3M66FGQLSLXI2N6LEZXJQVMA, seg_pos={0 0}, upload=332765]: Done uploading piece. err: <nil> Mar 18 20:06:07.492135441 (297h6m5.088906454s) [noise=true , address=45.91.116.9:4276 , piece_id=J356KNX74ADIKN7PPKCRNGACHPVBJYZA3O5OAGVYSURLB4T6KSZA, seg_pos={0 0}, upload=332765]: Done uploading piece. err: <nil> Mar 18 20:06:07.508501889 (297h6m5.105272902s) [noise=true , address=77.90.31.12:4615 , piece_id=HLRHRANL5IHJOB5QHLW2OIN5ZP56RKX3INJPWPU5YYHBHRYVJM6A, seg_pos={0 0}, upload=332765]: Done uploading piece. err: <nil> Mar 18 20:06:07.770562428 (297h6m5.367333451s) [noise=true , address=167.17.72.68:4668 , piece_id=KUTKCCK6Y54CMAIIWGINQERXEV6CGVPCOYEJAQUNPBWTHJ755KJQ, seg_pos={0 0}, upload=332765]: Done uploading piece. err: <nil> Mar 18 20:06:08.006027504 (297h6m5.602798531s) [noise=true , address=193.149.164.66:4993 , piece_id=CRRZAIXMIV673K5JLX743KZ3HFL5MBGUE5ZE4T4Q2WPCSUQ32JQA, seg_pos={0 0}, upload=332765]: Done uploading piece. err: <nil> Mar 18 20:06:08.016743696 (297h6m5.613514734s) [noise=true , address=193.149.164.71:4307 , piece_id=TZ5I3SSLCLL2AOKJTEFTMEE7DLXHXDKCJJLNP4WHDSGRCJLLBM6A, seg_pos={0 0}, upload=332765]: Done uploading piece. err: <nil> Mar 18 20:06:08.043257501 (297h6m5.640028528s) [noise=true , address=64.49.13.137:4499 , piece_id=KTFNLPG67BVZMYZMJJO4BDRTASHDANE2FBYO3YUV2E4XBCU5NLCA, seg_pos={0 0}, upload=332765]: Done uploading piece. err: <nil> Mar 18 20:06:08.323806978 (297h6m5.920577991s) [noise=true , address=79.127.205.233:4173 , piece_id=I64CQ5XLN6DUP2E465ZQLGOVE5NEIHLNWT33QB7WKJF5Q6QZRPLQ, seg_pos={0 0}, upload=332765]: Done uploading piece. err: <nil> Mar 18 20:07:04.645185715 (297h7m2.241956747s) [noise=true , address=145.79.227.3:4177 , piece_id=A7HSMKKEMCEJZENJ3CM7N3GZI3NX4BA2WIQZ27MTRONPXBZKZWHA, seg_pos={0 0}, upload=332765]: Done uploading piece. err: ecclient: upload failed (node:12rFuJ2JyBGWtaASJNnTdHoeKUY1mCfDYfJ74BkxK64U9it8FKF, address:145.79.227.3:4177): protocol: rpc: dial tcp 145.79.227.3:4177: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:07:04.645320026 (297h7m2.24209106s) [noise=true , address=145.79.227.24:4592 , piece_id=NGOLKV5BQKREBAYQHBSXHN2UKBHB52WPVBBO7NIFCCYTBF2MXPBA, seg_pos={0 0}, upload=332765]: Done uploading piece. err: ecclient: upload failed (node:1fB7rqiDhMNQ4j9smkctbKdnGCXXdfFPWTgTxRrwQVZ9fKEFeZ, address:145.79.227.24:4592): protocol: rpc: dial tcp 145.79.227.24:4592: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:07:04.645426518 (297h7m2.242197551s) [noise=true , address=145.79.227.2:4129 , piece_id=2Y7VDPW5SS3D4J45EAMSLOEQAHNKYCZJEPP2FTP7GKVCYKMDQTLA, seg_pos={0 0}, upload=332765]: Done uploading piece. err: ecclient: upload failed (node:12GJof7GVfEN9u2yVyWf6j24hnaTdD3ygoPRoicmSqSfcNJFzs6, address:145.79.227.2:4129): protocol: rpc: dial tcp 145.79.227.2:4129: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:07:04.645453468 (297h7m2.242224501s) [noise=true , address=145.79.227.49:4360 , piece_id=OP5BBV7CUZBSMBO6HTTHQCVXDQGS6XO5LGAZ4I2SKUSPGUAKBXKA, seg_pos={0 0}, upload=332765]: Done uploading piece. err: ecclient: upload failed (node:12T24rtYBJ47E6Ef1ZZKMWcpBrWBpCg5QV2ZHniCZZkzJF2yUNk, address:145.79.227.49:4360): protocol: rpc: dial tcp 145.79.227.49:4360: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:07:04.645552758 (297h7m2.242323785s) [noise=true , address=145.79.227.33:4875 , piece_id=5LQOGDTQD5CCBMKIZCSLBVQUHQWU6W5ELX6XKLFSFQLU3NVDLKPQ, seg_pos={0 0}, upload=332765]: Done uploading piece. err: ecclient: upload failed (node:1ijEJgYouyMkUqpLWWz7eH2NGvvs9XpN2ujDjupPykDe5eWcB1, address:145.79.227.33:4875): protocol: rpc: dial tcp 145.79.227.33:4875: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:07:04.645682576 (297h7m2.242453607s) [noise=true , address=145.79.227.51:4424 , piece_id=5XCD27Q3GOCFGSHDUNX7QIGUQ6URBU2UREPVX5ZF7CFICF4X4UNQ, seg_pos={0 0}, upload=332765]: Done uploading piece. err: ecclient: upload failed (node:12LwZixdSPnCm3Wm27PBY79jj52sjkNh2VuQexagieTGBp4Ps2k, address:145.79.227.51:4424): protocol: rpc: dial tcp 145.79.227.51:4424: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:07:04.645716594 (297h7m2.24248761s) [seg_pos={0 0}, upload=332765]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Mar 18 20:07:04.666238325 (297h7m2.263009349s) [upload=332765]: Done uploading segments... Mar 18 20:07:04.666276723 (297h7m2.263047746s) [upload=332765]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 20.185768353s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 45 out of 60 is failed (threshold=3) [..........XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.XXXXXX...XX.] duration: 18.024543294s trace: https://jaeger.dev.storj.io/trace/7f244e76e24910ce FAIL (2026-03-18 20:10:46 UTC) operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== Mar 18 20:09:43.681401093 (297h9m41.27817212s) [upload=332827]: Starting upload Mar 18 20:09:43.681562540 (297h9m41.278333571s) [upload=332827]: Done starting upload Mar 18 20:09:43.681680258 (297h9m41.278451271s) [upload=332827]: Uploading segments... Mar 18 20:09:43.682198491 (297h9m41.278969515s) [upload=332827]: Got next segment. Inline: false Mar 18 20:09:43.682243333 (297h9m41.279014355s) [upload=332827]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 20:09:43.682249541 (297h9m41.279020564s) [upload=332827]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 18 20:09:47.295813290 (297h9m44.892584324s) [seg_pos={0 0}, upload=332827]: Begin upload segment... Mar 18 20:09:47.295968626 (297h9m44.892739684s) [seg_pos={0 0}, upload=332827]: Done begin upload segment. Mar 18 20:09:47.296064738 (297h9m44.892835774s) [noise=true , address=45.91.116.4:4972 , piece_id=CLNKBFUCQEL6RJXPGLXICUMLQI52X4FBTZBMMAMSPD6RQ2GBAVKQ, seg_pos={0 0}, upload=332827]: Uploading piece... Mar 18 20:09:47.297458255 (297h9m44.894229283s) [noise=true , address=45.91.116.14:4189 , piece_id=G737HWSEFTVEAJG5D2SIDTHDRGALR5V2ZALEIPQDLITDCWFXIGEQ, seg_pos={0 0}, upload=332827]: Uploading piece... Mar 18 20:09:47.297639191 (297h9m44.894410224s) [noise=true , address=145.79.227.37:4569 , piece_id=HOQQHR2SWV6QQLLX5KNAG5WMZMO3ILP7GIJYZLV4HAS7ZN2XGZ6A, seg_pos={0 0}, upload=332827]: Uploading piece... Mar 18 20:09:47.297993984 (297h9m44.894765013s) [noise=true , address=77.90.31.21:4881 , piece_id=B5EUVXDPLERF3HC2XT4C57AHA73Y7M6DO6R6XEOUR7LQ7GR5Z5TA, seg_pos={0 0}, upload=332827]: Uploading piece... Mar 18 20:09:47.298442620 (297h9m44.895213653s) [noise=true , address=145.79.227.8:4334 , piece_id=BGL2YKM4OW5U4TRLCW7KSUZO2KO5YHAWGU3LIDXSP7VXP547IEBQ, seg_pos={0 0}, upload=332827]: Uploading piece... Mar 18 20:09:47.298505878 (297h9m44.895276907s) [noise=true , address=77.90.31.13:4625 , piece_id=L5CSO5PQURCNWBZDK7G2ZL7V7OK72PGXRQCHNQQMT4BKYLBMVRBA, seg_pos={0 0}, upload=332827]: Uploading piece... Mar 18 20:09:47.298911925 (297h9m44.89568295s) [noise=true , address=145.79.227.40:4053 , piece_id=734XXEVHT7MV7MFLQJTO3F3GGVWZNFQVFJM2NXIBYCU2Z3HULBPQ, seg_pos={0 0}, upload=332827]: Uploading piece... Mar 18 20:09:47.299180159 (297h9m44.895951191s) [noise=true , address=79.127.219.38:4821 , piece_id=X73NW3GPQFROBTV7QQSNQH5NK5TQ3WRHN34FFY5HC2MAXHTXI2WA, seg_pos={0 0}, upload=332827]: Uploading piece... Mar 18 20:09:47.299727913 (297h9m44.896498947s) [noise=true , address=145.79.227.50:4374 , piece_id=P57J5KBOQASJIQNTO5IOR2FFXLVB3LS4VJ6UP7HFCUN7FMAYSIUA, seg_pos={0 0}, upload=332827]: Uploading piece... Mar 18 20:09:47.299990817 (297h9m44.896761851s) [noise=true , address=45.91.116.8:4049 , piece_id=G2OYMGCY46MLQ6CHIXQBMT6FRFWUKH3VDWO6YDEASUJVCSZKUYJQ, seg_pos={0 0}, upload=332827]: Uploading piece... Mar 18 20:09:47.300385098 (297h9m44.897156131s) [noise=true , address=145.79.227.3:4149 , piece_id=LBY7NHK6SZAQI2R5XXX4XPJUPJAZHDDDZIVGVQIZL7YLHPFEBF7A, seg_pos={0 0}, upload=332827]: Uploading piece... Mar 18 20:09:47.300626853 (297h9m44.897397886s) [noise=true , address=45.91.116.9:4264 , piece_id=5EX3SEHGAHFAVXSKCUT32CXX2IBYULE5IBZ3D4HJERZMTSGF4DTQ, seg_pos={0 0}, upload=332827]: Uploading piece... Mar 18 20:09:47.301407094 (297h9m44.898178127s) [noise=true , address=45.91.116.12:4226 , piece_id=UIZLJTMMB2JGKHA2QVWUH3KPNIF7EHJB5P6SSFGBAMGGV2SJQYRQ, seg_pos={0 0}, upload=332827]: Uploading piece... Mar 18 20:09:47.302044730 (297h9m44.898815763s) [noise=true , address=77.90.31.11:4558 , piece_id=NI5DW6LPAPEJ2AQJJBABKEQLHCCPR7HAWKQNJSBF2YYMSTS4SWWA, seg_pos={0 0}, upload=332827]: Uploading piece... Mar 18 20:09:47.302736034 (297h9m44.899507067s) [noise=true , address=145.79.227.20:4442 , piece_id=GD2YCA44T25AOZH6AMA6CCQ3MYTIPW7WGBU33YBGTXFBSDI457RA, seg_pos={0 0}, upload=332827]: Uploading piece... Mar 18 20:09:47.303348473 (297h9m44.900119503s) [noise=true , address=77.90.31.10:4518 , piece_id=J43RLA6QYI6U5UBOWJLQ3Q6QINQB2R3QA7PZVYECL5KM3J3MJAPQ, seg_pos={0 0}, upload=332827]: Uploading piece... Mar 18 20:09:47.381907017 (297h9m44.97867803s) [upload=332827]: Next returned nil segment Mar 18 20:09:47.381939055 (297h9m44.978710079s) [upload=332827]: Waiting for error group managing segments... Mar 18 20:09:47.386556032 (297h9m44.98332706s) [noise=true , address=79.127.219.38:4821 , piece_id=X73NW3GPQFROBTV7QQSNQH5NK5TQ3WRHN34FFY5HC2MAXHTXI2WA, seg_pos={0 0}, upload=332827]: Done uploading piece. err: <nil> Mar 18 20:09:47.386803054 (297h9m44.983574083s) [noise=true , address=77.90.31.11:4558 , piece_id=NI5DW6LPAPEJ2AQJJBABKEQLHCCPR7HAWKQNJSBF2YYMSTS4SWWA, seg_pos={0 0}, upload=332827]: Done uploading piece. err: <nil> Mar 18 20:09:47.387514219 (297h9m44.984285257s) [noise=true , address=45.91.116.9:4264 , piece_id=5EX3SEHGAHFAVXSKCUT32CXX2IBYULE5IBZ3D4HJERZMTSGF4DTQ, seg_pos={0 0}, upload=332827]: Done uploading piece. err: <nil> Mar 18 20:09:47.388649266 (297h9m44.985420297s) [noise=true , address=45.91.116.4:4972 , piece_id=CLNKBFUCQEL6RJXPGLXICUMLQI52X4FBTZBMMAMSPD6RQ2GBAVKQ, seg_pos={0 0}, upload=332827]: Done uploading piece. err: <nil> Mar 18 20:09:47.388887757 (297h9m44.985658801s) [noise=true , address=77.90.31.21:4881 , piece_id=B5EUVXDPLERF3HC2XT4C57AHA73Y7M6DO6R6XEOUR7LQ7GR5Z5TA, seg_pos={0 0}, upload=332827]: Done uploading piece. err: <nil> Mar 18 20:09:47.389319888 (297h9m44.986090912s) [noise=true , address=45.91.116.12:4226 , piece_id=UIZLJTMMB2JGKHA2QVWUH3KPNIF7EHJB5P6SSFGBAMGGV2SJQYRQ, seg_pos={0 0}, upload=332827]: Done uploading piece. err: <nil> Mar 18 20:09:47.402313222 (297h9m44.999084248s) [noise=true , address=77.90.31.10:4518 , piece_id=J43RLA6QYI6U5UBOWJLQ3Q6QINQB2R3QA7PZVYECL5KM3J3MJAPQ, seg_pos={0 0}, upload=332827]: Done uploading piece. err: <nil> Mar 18 20:09:47.406132065 (297h9m45.002903094s) [noise=true , address=45.91.116.14:4189 , piece_id=G737HWSEFTVEAJG5D2SIDTHDRGALR5V2ZALEIPQDLITDCWFXIGEQ, seg_pos={0 0}, upload=332827]: Done uploading piece. err: <nil> Mar 18 20:09:47.407062118 (297h9m45.003833147s) [noise=true , address=45.91.116.8:4049 , piece_id=G2OYMGCY46MLQ6CHIXQBMT6FRFWUKH3VDWO6YDEASUJVCSZKUYJQ, seg_pos={0 0}, upload=332827]: Done uploading piece. err: <nil> Mar 18 20:09:47.527410112 (297h9m45.124181141s) [noise=true , address=77.90.31.13:4625 , piece_id=L5CSO5PQURCNWBZDK7G2ZL7V7OK72PGXRQCHNQQMT4BKYLBMVRBA, seg_pos={0 0}, upload=332827]: Done uploading piece. err: <nil> Mar 18 20:10:13.648195696 (297h10m11.244966727s) [noise=true , address=145.79.227.3:4149 , piece_id=LBY7NHK6SZAQI2R5XXX4XPJUPJAZHDDDZIVGVQIZL7YLHPFEBF7A, seg_pos={0 0}, upload=332827]: Done uploading piece. err: ecclient: upload failed (node:12qexRLJM6TY935UaEWsj1X9ViyBvk2BM1XGjnq8kapnXXXmgbm, address:145.79.227.3:4149): protocol: rpc: dial tcp 145.79.227.3:4149: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:10:13.648389072 (297h10m11.245160107s) [noise=true , address=145.79.227.40:4053 , piece_id=734XXEVHT7MV7MFLQJTO3F3GGVWZNFQVFJM2NXIBYCU2Z3HULBPQ, seg_pos={0 0}, upload=332827]: Done uploading piece. err: ecclient: upload failed (node:12DqyTc4MHvxutFXs2WWRCk8MSuxF3Y1HayyQY4Y5sy8cqDLWuG, address:145.79.227.40:4053): protocol: rpc: dial tcp 145.79.227.40:4053: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:10:13.648516554 (297h10m11.245287583s) [noise=true , address=145.79.227.20:4442 , piece_id=GD2YCA44T25AOZH6AMA6CCQ3MYTIPW7WGBU33YBGTXFBSDI457RA, seg_pos={0 0}, upload=332827]: Done uploading piece. err: ecclient: upload failed (node:125KyoqeSzbjP8XPhXqw2AenTDiLtVmt5KpLBD7ZW7YC9e9zJu9, address:145.79.227.20:4442): protocol: rpc: dial tcp 145.79.227.20:4442: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:10:13.648732866 (297h10m11.245503895s) [noise=true , address=145.79.227.37:4569 , piece_id=HOQQHR2SWV6QQLLX5KNAG5WMZMO3ILP7GIJYZLV4HAS7ZN2XGZ6A, seg_pos={0 0}, upload=332827]: Done uploading piece. err: ecclient: upload failed (node:18TqoejkuHLqZWhEPe3QPH1weVXnWJhaeoi8BGkvjCTLW6t4qT, address:145.79.227.37:4569): protocol: rpc: dial tcp 145.79.227.37:4569: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:10:13.648894806 (297h10m11.245665836s) [noise=true , address=145.79.227.8:4334 , piece_id=BGL2YKM4OW5U4TRLCW7KSUZO2KO5YHAWGU3LIDXSP7VXP547IEBQ, seg_pos={0 0}, upload=332827]: Done uploading piece. err: ecclient: upload failed (node:12KYMZAFsJbhWC4CdKRod2Z6qsC6ANcpqmpWSWx7UYajp6JvKH1, address:145.79.227.8:4334): protocol: rpc: dial tcp 145.79.227.8:4334: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:10:13.649026715 (297h10m11.245797744s) [noise=true , address=145.79.227.50:4374 , piece_id=P57J5KBOQASJIQNTO5IOR2FFXLVB3LS4VJ6UP7HFCUN7FMAYSIUA, seg_pos={0 0}, upload=332827]: Done uploading piece. err: ecclient: upload failed (node:12QRixhWaUwXPdjcWKYHbiknME5JVomWYABY6egDD6VKwaTPB8K, address:145.79.227.50:4374): protocol: rpc: dial tcp 145.79.227.50:4374: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:10:13.649064131 (297h10m11.245835144s) [seg_pos={0 0}, upload=332827]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Mar 18 20:10:13.676763535 (297h10m11.273534576s) [upload=332827]: Done uploading segments... Mar 18 20:10:13.676808312 (297h10m11.273579347s) [upload=332827]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration: 18.024650502s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 6 out of 60 is failed (threshold=3) [...................X..........XX..X....X........X...........] duration: 18.864052781s trace: https://jaeger.dev.storj.io/trace/589419efc52e5e2b FAIL (2026-03-18 20:10:42 UTC) operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== Mar 18 20:00:59.343084966 (297h0m56.939855989s) [upload=332666]: Starting upload Mar 18 20:00:59.343175518 (297h0m56.939946548s) [upload=332666]: Done starting upload Mar 18 20:00:59.343565585 (297h0m56.940336612s) [upload=332666]: Uploading segments... Mar 18 20:00:59.344256647 (297h0m56.941027676s) [upload=332666]: Got next segment. Inline: false Mar 18 20:00:59.344352215 (297h0m56.941123241s) [upload=332666]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 20:00:59.344358706 (297h0m56.941129732s) [upload=332666]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 18 20:01:02.340259299 (297h0m59.937030329s) [seg_pos={0 0}, upload=332666]: Begin upload segment... Mar 18 20:01:02.340368743 (297h0m59.937139767s) [seg_pos={0 0}, upload=332666]: Done begin upload segment. Mar 18 20:01:02.340479563 (297h0m59.93725059s) [noise=true , address=45.91.116.8:4061 , piece_id=6XFVKT4AAO7QN457OQMRKPI6R24TJTAN47UXLEVOEFQVOS6M6C3Q, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.340523837 (297h0m59.93729486s) [noise=true , address=79.127.205.233:4166 , piece_id=SFGK5YCWHFRVXF6QNHYP2CYJCZLSTQ76OYRVJPWWXQTF2KFSFJKA, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.341468438 (297h0m59.938239447s) [noise=true , address=45.91.116.4:4961 , piece_id=ZMUGQECUVPVHDNKN75XPBET5M2HFXXK4WMTP4V5KEK63JVZCX5EA, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.341520025 (297h0m59.938291058s) [noise=true , address=167.17.72.71:4815 , piece_id=CXGTYNNBILN3TT7I6Z7UI75F2ZCS4QUTLS6KC4FJ5IHS3XOYWDJQ, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.342232992 (297h0m59.939004023s) [noise=true , address=145.79.227.36:4524 , piece_id=3TAV3P55ITTHBAZEUUUTOAHJRGQXJXXVGZJP6AVR2ANVNOR3S54Q, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.342917368 (297h0m59.939688398s) [noise=true , address=193.149.164.71:4297 , piece_id=4IJPXLPCX3LKFOBOSRHK2DTNRKH67K7FXJUKQ3JPRWHS47ZFEIDA, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.343244957 (297h0m59.940015982s) [noise=true , address=145.79.227.33:4892 , piece_id=DQ6ALC3VOAGFIJI3IKR6I7QFYM22C66QWAAXMH6NXI2FIQEZNQYA, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.344270079 (297h0m59.941041117s) [noise=true , address=145.79.227.21:4497 , piece_id=SP6EKJB7W4QXF7OIBVKEFWCPOZFL5IRIEDVAGS2M4QJSO34CUVRQ, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.345025057 (297h0m59.941796084s) [noise=true , address=77.90.31.17:4793 , piece_id=2U4IBUA2GGUGDBXQGIGQKUK4ZR3L7CDKC6MTGIKAUME4F7MFZRLA, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.345115802 (297h0m59.941886836s) [noise=true , address=167.17.72.69:4731 , piece_id=VOEVIKKFJMC2Y3TPQIDMYUP37NLFU7KPRH4HRRSMQLWUKTSLOH2Q, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.345690600 (297h0m59.942461634s) [noise=true , address=77.90.31.13:4648 , piece_id=BPMB5CFOYHFT3BS4ORR6JF6BNFC2AOAJZOSPMG67NAMLGGP5XVEQ, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.346311592 (297h0m59.943082623s) [noise=true , address=79.127.205.234:4202 , piece_id=YY4LYWX26O6RIOBONESNKTUZEFZCRL4UUX3NEBQ53AJV4IEDG5MQ, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.346791142 (297h0m59.943562166s) [noise=true , address=45.91.116.10:4110 , piece_id=UYLMDCWJEJOHGICA37KNH7G5AQXNIPRH5NRYLBUUMGDRVSLTEYHQ, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.347094999 (297h0m59.94386607s) [noise=true , address=145.79.227.17:4987 , piece_id=PT3TWVB65JZFEWVH4IUZEU2N53FQ57QBRWVLWAUFP4UDLY3LWNLQ, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.347781281 (297h0m59.944552305s) [noise=true , address=79.127.219.46:4919 , piece_id=KK6HRIVBMYLIHRHGPQH3AWMNSVHTTIBESHDMJZA72T446RX6QKRQ, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.348263123 (297h0m59.945034137s) [noise=true , address=193.149.164.68:4033 , piece_id=JZPQKPYGO3W3RZZBITQ73X2XEPIBBSLGRMWIVOM4Z3GTV5UFKRTQ, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.348915505 (297h0m59.945686536s) [noise=true , address=145.79.227.7:4312 , piece_id=2FAL6E3RMBTBHZBXAJJVH2N5VGF5VYI3KNZXGNWCMWGBBPAUO2NQ, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.349366298 (297h0m59.946137337s) [noise=true , address=193.149.164.67:4007 , piece_id=HURQGH67I5B6V4YUV3PRZL3KSKN3XTSK3UMCHZRNT6L3W3PXXHPQ, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.429834098 (297h1m0.026605126s) [upload=332666]: Next returned nil segment Mar 18 20:01:02.429853411 (297h1m0.026624442s) [upload=332666]: Waiting for error group managing segments... Mar 18 20:01:02.446867680 (297h1m0.043638704s) [noise=true , address=45.91.116.10:4110 , piece_id=UYLMDCWJEJOHGICA37KNH7G5AQXNIPRH5NRYLBUUMGDRVSLTEYHQ, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil> Mar 18 20:01:02.451665165 (297h1m0.048436196s) [noise=true , address=45.91.116.4:4961 , piece_id=ZMUGQECUVPVHDNKN75XPBET5M2HFXXK4WMTP4V5KEK63JVZCX5EA, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil> Mar 18 20:01:02.452538886 (297h1m0.049309913s) [noise=true , address=77.90.31.17:4793 , piece_id=2U4IBUA2GGUGDBXQGIGQKUK4ZR3L7CDKC6MTGIKAUME4F7MFZRLA, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil> Mar 18 20:01:02.455597040 (297h1m0.052368067s) [noise=true , address=45.91.116.8:4061 , piece_id=6XFVKT4AAO7QN457OQMRKPI6R24TJTAN47UXLEVOEFQVOS6M6C3Q, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil> Mar 18 20:01:02.457848739 (297h1m0.054619766s) [noise=true , address=77.90.31.13:4648 , piece_id=BPMB5CFOYHFT3BS4ORR6JF6BNFC2AOAJZOSPMG67NAMLGGP5XVEQ, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil> Mar 18 20:01:02.461479709 (297h1m0.058250746s) [noise=true , address=79.127.219.46:4919 , piece_id=KK6HRIVBMYLIHRHGPQH3AWMNSVHTTIBESHDMJZA72T446RX6QKRQ, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil> Mar 18 20:01:02.676859466 (297h1m0.27363049s) [noise=true , address=167.17.72.69:4731 , piece_id=VOEVIKKFJMC2Y3TPQIDMYUP37NLFU7KPRH4HRRSMQLWUKTSLOH2Q, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil> Mar 18 20:01:02.909860743 (297h1m0.506631766s) [noise=true , address=79.127.205.234:4202 , piece_id=YY4LYWX26O6RIOBONESNKTUZEFZCRL4UUX3NEBQ53AJV4IEDG5MQ, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil> Mar 18 20:01:02.930427742 (297h1m0.527198754s) [noise=true , address=193.149.164.71:4297 , piece_id=4IJPXLPCX3LKFOBOSRHK2DTNRKH67K7FXJUKQ3JPRWHS47ZFEIDA, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil> Mar 18 20:01:02.944696889 (297h1m0.541467914s) [noise=true , address=193.149.164.67:4007 , piece_id=HURQGH67I5B6V4YUV3PRZL3KSKN3XTSK3UMCHZRNT6L3W3PXXHPQ, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil> Mar 18 20:01:02.951319183 (297h1m0.548090212s) [noise=true , address=193.149.164.68:4033 , piece_id=JZPQKPYGO3W3RZZBITQ73X2XEPIBBSLGRMWIVOM4Z3GTV5UFKRTQ, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil> Mar 18 20:01:02.975619436 (297h1m0.572390467s) [noise=true , address=79.127.205.233:4166 , piece_id=SFGK5YCWHFRVXF6QNHYP2CYJCZLSTQ76OYRVJPWWXQTF2KFSFJKA, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil> Mar 18 20:01:03.819918528 (297h1m1.416689564s) [noise=true , address=167.17.72.71:4815 , piece_id=CXGTYNNBILN3TT7I6Z7UI75F2ZCS4QUTLS6KC4FJ5IHS3XOYWDJQ, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil> Mar 18 20:01:59.307873420 (297h1m56.904644452s) [noise=true , address=145.79.227.7:4312 , piece_id=2FAL6E3RMBTBHZBXAJJVH2N5VGF5VYI3KNZXGNWCMWGBBPAUO2NQ, seg_pos={0 0}, upload=332666]: Done uploading piece. err: ecclient: upload failed (node:12AVp3qn5uwLPqYaZEwzcgoGMJhNga1BUh6bK5k2bxjWgZLQiiC, address:145.79.227.7:4312): protocol: rpc: dial tcp 145.79.227.7:4312: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:01:59.308026092 (297h1m56.904797124s) [noise=true , address=145.79.227.17:4987 , piece_id=PT3TWVB65JZFEWVH4IUZEU2N53FQ57QBRWVLWAUFP4UDLY3LWNLQ, seg_pos={0 0}, upload=332666]: Done uploading piece. err: ecclient: upload failed (node:1XsPVNGYUhKnXkQRfVfjyBRzyvhShSdsrnzXtwe4Xtf5MaHMvH, address:145.79.227.17:4987): protocol: rpc: dial tcp 145.79.227.17:4987: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:01:59.308195864 (297h1m56.904966894s) [noise=true , address=145.79.227.36:4524 , piece_id=3TAV3P55ITTHBAZEUUUTOAHJRGQXJXXVGZJP6AVR2ANVNOR3S54Q, seg_pos={0 0}, upload=332666]: Done uploading piece. err: ecclient: upload failed (node:1xfbCzjSwFDRrR1gPggeN8tNCLbRRWDVzCJ6mqhZsbVSe8d4s4, address:145.79.227.36:4524): protocol: rpc: dial tcp 145.79.227.36:4524: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:01:59.308316037 (297h1m56.90508706s) [noise=true , address=145.79.227.33:4892 , piece_id=DQ6ALC3VOAGFIJI3IKR6I7QFYM22C66QWAAXMH6NXI2FIQEZNQYA, seg_pos={0 0}, upload=332666]: Done uploading piece. err: ecclient: upload failed (node:12RHwFxcUjPsn12uHEAH3fBDJKv1v1tM8mRJXoFrZrG5djdswF5, address:145.79.227.33:4892): protocol: rpc: dial tcp 145.79.227.33:4892: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:01:59.308404403 (297h1m56.905175433s) [noise=true , address=145.79.227.21:4497 , piece_id=SP6EKJB7W4QXF7OIBVKEFWCPOZFL5IRIEDVAGS2M4QJSO34CUVRQ, seg_pos={0 0}, upload=332666]: Done uploading piece. err: ecclient: upload failed (node:12ZBe4we7VeFdthqRxyy1V9uQrA4r8ouNX68xqsk5bD26dkf2Gd, address:145.79.227.21:4497): protocol: context deadline exceeded; context deadline exceeded Mar 18 20:01:59.308437457 (297h1m56.90520847s) [seg_pos={0 0}, upload=332666]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 18 20:01:59.338288467 (297h1m56.93505949s) [upload=332666]: Done uploading segments... Mar 18 20:01:59.338335497 (297h1m56.935106523s) [upload=332666]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 18.864085603s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 54 out of 60 is failed (threshold=6) [..XXXXXXXXX.XXX...XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 5.001983646s trace: https://jaeger.dev.storj.io/trace/4cb70949dead2c94 FAIL (2026-03-18 20:10:42 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.002010435s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 8 out of 60 is failed (threshold=3) [...............XXXXXX.XX....................................] duration: 17.104471419s trace: https://jaeger.dev.storj.io/trace/4b4d10a8051ad01b FAIL (2026-03-18 20:10:23 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 17.104510263s core uplink us-select-1 us1-placement-20

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)