Orbiter History

checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 52 out of 60 is failed (threshold=6) [....XXXXXXXXX.XXX...XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 5.00090042s trace: https://jaeger.dev.storj.io/trace/12da2377b35523d8 FAIL (2026-03-18 20:09:25 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.000927319s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 8 out of 60 is failed (threshold=3) [...................XXX..........X...X.........X....X....X...] duration: 19.39886869s trace: https://jaeger.dev.storj.io/trace/75a9000feaeb0b3d FAIL (2026-03-18 20:09:24 UTC) operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== ar 18 20:06:04.693955863 (297h6m2.290726886s) [upload=332765]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 20:06:04.693961555 (297h6m2.290732583s) [upload=332765]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 18 20:06:07.394748117 (297h6m4.99151915s) [seg_pos={0 0}, upload=332765]: Begin upload segment... Mar 18 20:06:07.394875884 (297h6m4.991646919s) [seg_pos={0 0}, upload=332765]: Done begin upload segment. Mar 18 20:06:07.394887373 (297h6m4.991658385s) [noise=true , address=45.91.116.9:4276 , piece_id=J356KNX74ADIKN7PPKCRNGACHPVBJYZA3O5OAGVYSURLB4T6KSZA, seg_pos={0 0}, upload=332765]: Uploading piece... Mar 18 20:06:07.394990884 (297h6m4.991761926s) [noise=true , address=145.79.227.2:4129 , piece_id=2Y7VDPW5SS3D4J45EAMSLOEQAHNKYCZJEPP2FTP7GKVCYKMDQTLA, seg_pos={0 0}, upload=332765]: Uploading piece... Mar 18 20:06:07.396053654 (297h6m4.992824681s) [noise=true , address=45.91.116.11:4150 , piece_id=L2ZK2GKN4VY7W6SJ25MR65RL52UH3M66FGQLSLXI2N6LEZXJQVMA, seg_pos={0 0}, upload=332765]: Uploading piece... Mar 18 20:06:07.396081719 (297h6m4.992852749s) [noise=true , address=45.91.116.12:4246 , piece_id=T5BGURHRLDTEZZ4TORP6F5WNBR2TY3ZDWPWO4PGSXQ575BWTUMVA, seg_pos={0 0}, upload=332765]: Uploading piece... Mar 18 20:06:07.396842874 (297h6m4.993613899s) [noise=true , address=145.79.227.33:4875 , piece_id=5LQOGDTQD5CCBMKIZCSLBVQUHQWU6W5ELX6XKLFSFQLU3NVDLKPQ, seg_pos={0 0}, upload=332765]: Uploading piece... Mar 18 20:06:07.397161075 (297h6m4.993932121s) [noise=true , address=77.90.31.12:4615 , piece_id=HLRHRANL5IHJOB5QHLW2OIN5ZP56RKX3INJPWPU5YYHBHRYVJM6A, seg_pos={0 0}, upload=332765]: Uploading piece... Mar 18 20:06:07.397369139 (297h6m4.994140166s) [noise=true , address=77.90.31.11:4567 , piece_id=UN6S6RWYUVLUA5NPFFJ5TASQDZ26OU2IIWP3ZNFUFVJ3WJAA2PSQ, seg_pos={0 0}, upload=332765]: Uploading piece... Mar 18 20:06:07.397732013 (297h6m4.994503037s) [noise=true , address=45.91.116.14:4206 , piece_id=IDCMPAW3BEN5OQ7HUX3QDGFGKN76DM4E6CDN3M45SJQ4VU2OQGCQ, seg_pos={0 0}, upload=332765]: Uploading piece... Mar 18 20:06:07.397757704 (297h6m4.994528729s) [noise=true , address=145.79.227.3:4177 , piece_id=A7HSMKKEMCEJZENJ3CM7N3GZI3NX4BA2WIQZ27MTRONPXBZKZWHA, seg_pos={0 0}, upload=332765]: Uploading piece... Mar 18 20:06:07.398274955 (297h6m4.995045981s) [noise=true , address=193.149.164.71:4307 , piece_id=TZ5I3SSLCLL2AOKJTEFTMEE7DLXHXDKCJJLNP4WHDSGRCJLLBM6A, seg_pos={0 0}, upload=332765]: Uploading piece... Mar 18 20:06:07.398328768 (297h6m4.995099802s) [noise=true , address=79.127.205.233:4173 , piece_id=I64CQ5XLN6DUP2E465ZQLGOVE5NEIHLNWT33QB7WKJF5Q6QZRPLQ, seg_pos={0 0}, upload=332765]: Uploading piece... Mar 18 20:06:07.398815526 (297h6m4.995586656s) [noise=true , address=145.79.227.51:4424 , piece_id=5XCD27Q3GOCFGSHDUNX7QIGUQ6URBU2UREPVX5ZF7CFICF4X4UNQ, seg_pos={0 0}, upload=332765]: Uploading piece... Mar 18 20:06:07.399136293 (297h6m4.995907322s) [noise=true , address=64.49.13.137:4499 , piece_id=KTFNLPG67BVZMYZMJJO4BDRTASHDANE2FBYO3YUV2E4XBCU5NLCA, seg_pos={0 0}, upload=332765]: Uploading piece... Mar 18 20:06:07.399360723 (297h6m4.996131762s) [noise=true , address=45.91.116.4:4963 , piece_id=ZGY2X7JG2MPOJWKMFWP3VATYO3QTEOXPQ4KYZSOW4QOPST6OV2VQ, seg_pos={0 0}, upload=332765]: Uploading piece... Mar 18 20:06:07.399800937 (297h6m4.996571962s) [noise=true , address=145.79.227.24:4592 , piece_id=NGOLKV5BQKREBAYQHBSXHN2UKBHB52WPVBBO7NIFCCYTBF2MXPBA, seg_pos={0 0}, upload=332765]: Uploading piece... Mar 18 20:06:07.399957240 (297h6m4.996728268s) [noise=true , address=193.149.164.66:4993 , piece_id=CRRZAIXMIV673K5JLX743KZ3HFL5MBGUE5ZE4T4Q2WPCSUQ32JQA, seg_pos={0 0}, upload=332765]: Uploading piece... Mar 18 20:06:07.400205078 (297h6m4.996976103s) [noise=true , address=167.17.72.68:4668 , piece_id=KUTKCCK6Y54CMAIIWGINQERXEV6CGVPCOYEJAQUNPBWTHJ755KJQ, seg_pos={0 0}, upload=332765]: Uploading piece... Mar 18 20:06:07.400506584 (297h6m4.997277612s) [noise=true , address=145.79.227.49:4360 , piece_id=OP5BBV7CUZBSMBO6HTTHQCVXDQGS6XO5LGAZ4I2SKUSPGUAKBXKA, seg_pos={0 0}, upload=332765]: Uploading piece... Mar 18 20:06:07.471265449 (297h6m5.068036462s) [upload=332765]: Next returned nil segment Mar 18 20:06:07.471278058 (297h6m5.068049082s) [upload=332765]: Waiting for error group managing segments... Mar 18 20:06:07.475837847 (297h6m5.07260886s) [noise=true , address=77.90.31.11:4567 , piece_id=UN6S6RWYUVLUA5NPFFJ5TASQDZ26OU2IIWP3ZNFUFVJ3WJAA2PSQ, seg_pos={0 0}, upload=332765]: Done uploading piece. err: <nil> Mar 18 20:06:07.476715720 (297h6m5.073486747s) [noise=true , address=45.91.116.4:4963 , piece_id=ZGY2X7JG2MPOJWKMFWP3VATYO3QTEOXPQ4KYZSOW4QOPST6OV2VQ, seg_pos={0 0}, upload=332765]: Done uploading piece. err: <nil> Mar 18 20:06:07.488555609 (297h6m5.085326636s) [noise=true , address=45.91.116.12:4246 , piece_id=T5BGURHRLDTEZZ4TORP6F5WNBR2TY3ZDWPWO4PGSXQ575BWTUMVA, seg_pos={0 0}, upload=332765]: Done uploading piece. err: <nil> Mar 18 20:06:07.489803434 (297h6m5.086574468s) [noise=true , address=45.91.116.14:4206 , piece_id=IDCMPAW3BEN5OQ7HUX3QDGFGKN76DM4E6CDN3M45SJQ4VU2OQGCQ, seg_pos={0 0}, upload=332765]: Done uploading piece. err: <nil> Mar 18 20:06:07.490598252 (297h6m5.087369276s) [noise=true , address=45.91.116.11:4150 , piece_id=L2ZK2GKN4VY7W6SJ25MR65RL52UH3M66FGQLSLXI2N6LEZXJQVMA, seg_pos={0 0}, upload=332765]: Done uploading piece. err: <nil> Mar 18 20:06:07.492135441 (297h6m5.088906454s) [noise=true , address=45.91.116.9:4276 , piece_id=J356KNX74ADIKN7PPKCRNGACHPVBJYZA3O5OAGVYSURLB4T6KSZA, seg_pos={0 0}, upload=332765]: Done uploading piece. err: <nil> Mar 18 20:06:07.508501889 (297h6m5.105272902s) [noise=true , address=77.90.31.12:4615 , piece_id=HLRHRANL5IHJOB5QHLW2OIN5ZP56RKX3INJPWPU5YYHBHRYVJM6A, seg_pos={0 0}, upload=332765]: Done uploading piece. err: <nil> Mar 18 20:06:07.770562428 (297h6m5.367333451s) [noise=true , address=167.17.72.68:4668 , piece_id=KUTKCCK6Y54CMAIIWGINQERXEV6CGVPCOYEJAQUNPBWTHJ755KJQ, seg_pos={0 0}, upload=332765]: Done uploading piece. err: <nil> Mar 18 20:06:08.006027504 (297h6m5.602798531s) [noise=true , address=193.149.164.66:4993 , piece_id=CRRZAIXMIV673K5JLX743KZ3HFL5MBGUE5ZE4T4Q2WPCSUQ32JQA, seg_pos={0 0}, upload=332765]: Done uploading piece. err: <nil> Mar 18 20:06:08.016743696 (297h6m5.613514734s) [noise=true , address=193.149.164.71:4307 , piece_id=TZ5I3SSLCLL2AOKJTEFTMEE7DLXHXDKCJJLNP4WHDSGRCJLLBM6A, seg_pos={0 0}, upload=332765]: Done uploading piece. err: <nil> Mar 18 20:06:08.043257501 (297h6m5.640028528s) [noise=true , address=64.49.13.137:4499 , piece_id=KTFNLPG67BVZMYZMJJO4BDRTASHDANE2FBYO3YUV2E4XBCU5NLCA, seg_pos={0 0}, upload=332765]: Done uploading piece. err: <nil> Mar 18 20:06:08.323806978 (297h6m5.920577991s) [noise=true , address=79.127.205.233:4173 , piece_id=I64CQ5XLN6DUP2E465ZQLGOVE5NEIHLNWT33QB7WKJF5Q6QZRPLQ, seg_pos={0 0}, upload=332765]: Done uploading piece. err: <nil> Mar 18 20:07:04.645185715 (297h7m2.241956747s) [noise=true , address=145.79.227.3:4177 , piece_id=A7HSMKKEMCEJZENJ3CM7N3GZI3NX4BA2WIQZ27MTRONPXBZKZWHA, seg_pos={0 0}, upload=332765]: Done uploading piece. err: ecclient: upload failed (node:12rFuJ2JyBGWtaASJNnTdHoeKUY1mCfDYfJ74BkxK64U9it8FKF, address:145.79.227.3:4177): protocol: rpc: dial tcp 145.79.227.3:4177: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:07:04.645320026 (297h7m2.24209106s) [noise=true , address=145.79.227.24:4592 , piece_id=NGOLKV5BQKREBAYQHBSXHN2UKBHB52WPVBBO7NIFCCYTBF2MXPBA, seg_pos={0 0}, upload=332765]: Done uploading piece. err: ecclient: upload failed (node:1fB7rqiDhMNQ4j9smkctbKdnGCXXdfFPWTgTxRrwQVZ9fKEFeZ, address:145.79.227.24:4592): protocol: rpc: dial tcp 145.79.227.24:4592: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:07:04.645426518 (297h7m2.242197551s) [noise=true , address=145.79.227.2:4129 , piece_id=2Y7VDPW5SS3D4J45EAMSLOEQAHNKYCZJEPP2FTP7GKVCYKMDQTLA, seg_pos={0 0}, upload=332765]: Done uploading piece. err: ecclient: upload failed (node:12GJof7GVfEN9u2yVyWf6j24hnaTdD3ygoPRoicmSqSfcNJFzs6, address:145.79.227.2:4129): protocol: rpc: dial tcp 145.79.227.2:4129: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:07:04.645453468 (297h7m2.242224501s) [noise=true , address=145.79.227.49:4360 , piece_id=OP5BBV7CUZBSMBO6HTTHQCVXDQGS6XO5LGAZ4I2SKUSPGUAKBXKA, seg_pos={0 0}, upload=332765]: Done uploading piece. err: ecclient: upload failed (node:12T24rtYBJ47E6Ef1ZZKMWcpBrWBpCg5QV2ZHniCZZkzJF2yUNk, address:145.79.227.49:4360): protocol: rpc: dial tcp 145.79.227.49:4360: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:07:04.645552758 (297h7m2.242323785s) [noise=true , address=145.79.227.33:4875 , piece_id=5LQOGDTQD5CCBMKIZCSLBVQUHQWU6W5ELX6XKLFSFQLU3NVDLKPQ, seg_pos={0 0}, upload=332765]: Done uploading piece. err: ecclient: upload failed (node:1ijEJgYouyMkUqpLWWz7eH2NGvvs9XpN2ujDjupPykDe5eWcB1, address:145.79.227.33:4875): protocol: rpc: dial tcp 145.79.227.33:4875: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:07:04.645682576 (297h7m2.242453607s) [noise=true , address=145.79.227.51:4424 , piece_id=5XCD27Q3GOCFGSHDUNX7QIGUQ6URBU2UREPVX5ZF7CFICF4X4UNQ, seg_pos={0 0}, upload=332765]: Done uploading piece. err: ecclient: upload failed (node:12LwZixdSPnCm3Wm27PBY79jj52sjkNh2VuQexagieTGBp4Ps2k, address:145.79.227.51:4424): protocol: rpc: dial tcp 145.79.227.51:4424: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:07:04.645716594 (297h7m2.24248761s) [seg_pos={0 0}, upload=332765]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Mar 18 20:07:04.666238325 (297h7m2.263009349s) [upload=332765]: Done uploading segments... Mar 18 20:07:04.666276723 (297h7m2.263047746s) [upload=332765]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 19.398916908s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 6 out of 60 is failed (threshold=3) [.....................X..........XX..X....X........X.........] duration: 20.703682523s trace: https://jaeger.dev.storj.io/trace/589419efc52e5e2b FAIL (2026-03-18 20:09:07 UTC) operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== Mar 18 20:00:59.343084966 (297h0m56.939855989s) [upload=332666]: Starting upload Mar 18 20:00:59.343175518 (297h0m56.939946548s) [upload=332666]: Done starting upload Mar 18 20:00:59.343565585 (297h0m56.940336612s) [upload=332666]: Uploading segments... Mar 18 20:00:59.344256647 (297h0m56.941027676s) [upload=332666]: Got next segment. Inline: false Mar 18 20:00:59.344352215 (297h0m56.941123241s) [upload=332666]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 20:00:59.344358706 (297h0m56.941129732s) [upload=332666]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 18 20:01:02.340259299 (297h0m59.937030329s) [seg_pos={0 0}, upload=332666]: Begin upload segment... Mar 18 20:01:02.340368743 (297h0m59.937139767s) [seg_pos={0 0}, upload=332666]: Done begin upload segment. Mar 18 20:01:02.340479563 (297h0m59.93725059s) [noise=true , address=45.91.116.8:4061 , piece_id=6XFVKT4AAO7QN457OQMRKPI6R24TJTAN47UXLEVOEFQVOS6M6C3Q, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.340523837 (297h0m59.93729486s) [noise=true , address=79.127.205.233:4166 , piece_id=SFGK5YCWHFRVXF6QNHYP2CYJCZLSTQ76OYRVJPWWXQTF2KFSFJKA, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.341468438 (297h0m59.938239447s) [noise=true , address=45.91.116.4:4961 , piece_id=ZMUGQECUVPVHDNKN75XPBET5M2HFXXK4WMTP4V5KEK63JVZCX5EA, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.341520025 (297h0m59.938291058s) [noise=true , address=167.17.72.71:4815 , piece_id=CXGTYNNBILN3TT7I6Z7UI75F2ZCS4QUTLS6KC4FJ5IHS3XOYWDJQ, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.342232992 (297h0m59.939004023s) [noise=true , address=145.79.227.36:4524 , piece_id=3TAV3P55ITTHBAZEUUUTOAHJRGQXJXXVGZJP6AVR2ANVNOR3S54Q, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.342917368 (297h0m59.939688398s) [noise=true , address=193.149.164.71:4297 , piece_id=4IJPXLPCX3LKFOBOSRHK2DTNRKH67K7FXJUKQ3JPRWHS47ZFEIDA, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.343244957 (297h0m59.940015982s) [noise=true , address=145.79.227.33:4892 , piece_id=DQ6ALC3VOAGFIJI3IKR6I7QFYM22C66QWAAXMH6NXI2FIQEZNQYA, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.344270079 (297h0m59.941041117s) [noise=true , address=145.79.227.21:4497 , piece_id=SP6EKJB7W4QXF7OIBVKEFWCPOZFL5IRIEDVAGS2M4QJSO34CUVRQ, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.345025057 (297h0m59.941796084s) [noise=true , address=77.90.31.17:4793 , piece_id=2U4IBUA2GGUGDBXQGIGQKUK4ZR3L7CDKC6MTGIKAUME4F7MFZRLA, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.345115802 (297h0m59.941886836s) [noise=true , address=167.17.72.69:4731 , piece_id=VOEVIKKFJMC2Y3TPQIDMYUP37NLFU7KPRH4HRRSMQLWUKTSLOH2Q, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.345690600 (297h0m59.942461634s) [noise=true , address=77.90.31.13:4648 , piece_id=BPMB5CFOYHFT3BS4ORR6JF6BNFC2AOAJZOSPMG67NAMLGGP5XVEQ, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.346311592 (297h0m59.943082623s) [noise=true , address=79.127.205.234:4202 , piece_id=YY4LYWX26O6RIOBONESNKTUZEFZCRL4UUX3NEBQ53AJV4IEDG5MQ, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.346791142 (297h0m59.943562166s) [noise=true , address=45.91.116.10:4110 , piece_id=UYLMDCWJEJOHGICA37KNH7G5AQXNIPRH5NRYLBUUMGDRVSLTEYHQ, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.347094999 (297h0m59.94386607s) [noise=true , address=145.79.227.17:4987 , piece_id=PT3TWVB65JZFEWVH4IUZEU2N53FQ57QBRWVLWAUFP4UDLY3LWNLQ, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.347781281 (297h0m59.944552305s) [noise=true , address=79.127.219.46:4919 , piece_id=KK6HRIVBMYLIHRHGPQH3AWMNSVHTTIBESHDMJZA72T446RX6QKRQ, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.348263123 (297h0m59.945034137s) [noise=true , address=193.149.164.68:4033 , piece_id=JZPQKPYGO3W3RZZBITQ73X2XEPIBBSLGRMWIVOM4Z3GTV5UFKRTQ, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.348915505 (297h0m59.945686536s) [noise=true , address=145.79.227.7:4312 , piece_id=2FAL6E3RMBTBHZBXAJJVH2N5VGF5VYI3KNZXGNWCMWGBBPAUO2NQ, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.349366298 (297h0m59.946137337s) [noise=true , address=193.149.164.67:4007 , piece_id=HURQGH67I5B6V4YUV3PRZL3KSKN3XTSK3UMCHZRNT6L3W3PXXHPQ, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.429834098 (297h1m0.026605126s) [upload=332666]: Next returned nil segment Mar 18 20:01:02.429853411 (297h1m0.026624442s) [upload=332666]: Waiting for error group managing segments... Mar 18 20:01:02.446867680 (297h1m0.043638704s) [noise=true , address=45.91.116.10:4110 , piece_id=UYLMDCWJEJOHGICA37KNH7G5AQXNIPRH5NRYLBUUMGDRVSLTEYHQ, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil> Mar 18 20:01:02.451665165 (297h1m0.048436196s) [noise=true , address=45.91.116.4:4961 , piece_id=ZMUGQECUVPVHDNKN75XPBET5M2HFXXK4WMTP4V5KEK63JVZCX5EA, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil> Mar 18 20:01:02.452538886 (297h1m0.049309913s) [noise=true , address=77.90.31.17:4793 , piece_id=2U4IBUA2GGUGDBXQGIGQKUK4ZR3L7CDKC6MTGIKAUME4F7MFZRLA, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil> Mar 18 20:01:02.455597040 (297h1m0.052368067s) [noise=true , address=45.91.116.8:4061 , piece_id=6XFVKT4AAO7QN457OQMRKPI6R24TJTAN47UXLEVOEFQVOS6M6C3Q, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil> Mar 18 20:01:02.457848739 (297h1m0.054619766s) [noise=true , address=77.90.31.13:4648 , piece_id=BPMB5CFOYHFT3BS4ORR6JF6BNFC2AOAJZOSPMG67NAMLGGP5XVEQ, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil> Mar 18 20:01:02.461479709 (297h1m0.058250746s) [noise=true , address=79.127.219.46:4919 , piece_id=KK6HRIVBMYLIHRHGPQH3AWMNSVHTTIBESHDMJZA72T446RX6QKRQ, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil> Mar 18 20:01:02.676859466 (297h1m0.27363049s) [noise=true , address=167.17.72.69:4731 , piece_id=VOEVIKKFJMC2Y3TPQIDMYUP37NLFU7KPRH4HRRSMQLWUKTSLOH2Q, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil> Mar 18 20:01:02.909860743 (297h1m0.506631766s) [noise=true , address=79.127.205.234:4202 , piece_id=YY4LYWX26O6RIOBONESNKTUZEFZCRL4UUX3NEBQ53AJV4IEDG5MQ, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil> Mar 18 20:01:02.930427742 (297h1m0.527198754s) [noise=true , address=193.149.164.71:4297 , piece_id=4IJPXLPCX3LKFOBOSRHK2DTNRKH67K7FXJUKQ3JPRWHS47ZFEIDA, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil> Mar 18 20:01:02.944696889 (297h1m0.541467914s) [noise=true , address=193.149.164.67:4007 , piece_id=HURQGH67I5B6V4YUV3PRZL3KSKN3XTSK3UMCHZRNT6L3W3PXXHPQ, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil> Mar 18 20:01:02.951319183 (297h1m0.548090212s) [noise=true , address=193.149.164.68:4033 , piece_id=JZPQKPYGO3W3RZZBITQ73X2XEPIBBSLGRMWIVOM4Z3GTV5UFKRTQ, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil> Mar 18 20:01:02.975619436 (297h1m0.572390467s) [noise=true , address=79.127.205.233:4166 , piece_id=SFGK5YCWHFRVXF6QNHYP2CYJCZLSTQ76OYRVJPWWXQTF2KFSFJKA, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil> Mar 18 20:01:03.819918528 (297h1m1.416689564s) [noise=true , address=167.17.72.71:4815 , piece_id=CXGTYNNBILN3TT7I6Z7UI75F2ZCS4QUTLS6KC4FJ5IHS3XOYWDJQ, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil> Mar 18 20:01:59.307873420 (297h1m56.904644452s) [noise=true , address=145.79.227.7:4312 , piece_id=2FAL6E3RMBTBHZBXAJJVH2N5VGF5VYI3KNZXGNWCMWGBBPAUO2NQ, seg_pos={0 0}, upload=332666]: Done uploading piece. err: ecclient: upload failed (node:12AVp3qn5uwLPqYaZEwzcgoGMJhNga1BUh6bK5k2bxjWgZLQiiC, address:145.79.227.7:4312): protocol: rpc: dial tcp 145.79.227.7:4312: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:01:59.308026092 (297h1m56.904797124s) [noise=true , address=145.79.227.17:4987 , piece_id=PT3TWVB65JZFEWVH4IUZEU2N53FQ57QBRWVLWAUFP4UDLY3LWNLQ, seg_pos={0 0}, upload=332666]: Done uploading piece. err: ecclient: upload failed (node:1XsPVNGYUhKnXkQRfVfjyBRzyvhShSdsrnzXtwe4Xtf5MaHMvH, address:145.79.227.17:4987): protocol: rpc: dial tcp 145.79.227.17:4987: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:01:59.308195864 (297h1m56.904966894s) [noise=true , address=145.79.227.36:4524 , piece_id=3TAV3P55ITTHBAZEUUUTOAHJRGQXJXXVGZJP6AVR2ANVNOR3S54Q, seg_pos={0 0}, upload=332666]: Done uploading piece. err: ecclient: upload failed (node:1xfbCzjSwFDRrR1gPggeN8tNCLbRRWDVzCJ6mqhZsbVSe8d4s4, address:145.79.227.36:4524): protocol: rpc: dial tcp 145.79.227.36:4524: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:01:59.308316037 (297h1m56.90508706s) [noise=true , address=145.79.227.33:4892 , piece_id=DQ6ALC3VOAGFIJI3IKR6I7QFYM22C66QWAAXMH6NXI2FIQEZNQYA, seg_pos={0 0}, upload=332666]: Done uploading piece. err: ecclient: upload failed (node:12RHwFxcUjPsn12uHEAH3fBDJKv1v1tM8mRJXoFrZrG5djdswF5, address:145.79.227.33:4892): protocol: rpc: dial tcp 145.79.227.33:4892: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:01:59.308404403 (297h1m56.905175433s) [noise=true , address=145.79.227.21:4497 , piece_id=SP6EKJB7W4QXF7OIBVKEFWCPOZFL5IRIEDVAGS2M4QJSO34CUVRQ, seg_pos={0 0}, upload=332666]: Done uploading piece. err: ecclient: upload failed (node:12ZBe4we7VeFdthqRxyy1V9uQrA4r8ouNX68xqsk5bD26dkf2Gd, address:145.79.227.21:4497): protocol: context deadline exceeded; context deadline exceeded Mar 18 20:01:59.308437457 (297h1m56.90520847s) [seg_pos={0 0}, upload=332666]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 18 20:01:59.338288467 (297h1m56.93505949s) [upload=332666]: Done uploading segments... Mar 18 20:01:59.338335497 (297h1m56.935106523s) [upload=332666]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 20.703710464s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 8 out of 60 is failed (threshold=3) [.................XXXXXX.XX..................................] duration: 17.483943258s trace: https://jaeger.dev.storj.io/trace/4b4d10a8051ad01b FAIL (2026-03-18 20:08:56 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 17.484025805s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 43 out of 60 is failed (threshold=3) [.............XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.XXXXXX...] duration: 12.91264107s trace: https://jaeger.dev.storj.io/trace/3da6329b1a74a55f FAIL (2026-03-18 20:08:51 UTC) operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 9) ======LOGS====== Mar 18 20:06:58.715947398 (297h6m56.312718432s) [upload=332778]: Starting upload Mar 18 20:06:58.716027398 (297h6m56.312798434s) [upload=332778]: Done starting upload Mar 18 20:06:58.716264342 (297h6m56.313035366s) [upload=332778]: Uploading segments... Mar 18 20:06:58.716526836 (297h6m56.313297865s) [upload=332778]: Got next segment. Inline: false Mar 18 20:06:58.716595354 (297h6m56.313366377s) [upload=332778]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 20:06:58.716604668 (297h6m56.313375688s) [upload=332778]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 18 20:07:02.118470802 (297h6m59.715241839s) [seg_pos={0 0}, upload=332778]: Begin upload segment... Mar 18 20:07:02.118558314 (297h6m59.715329349s) [seg_pos={0 0}, upload=332778]: Done begin upload segment. Mar 18 20:07:02.118630613 (297h6m59.715401646s) [noise=true , address=145.79.227.22:4509 , piece_id=WUNQ6CLYVMZ4SXAH4ITENGEQ47QSMHBPJIFBZ2KSCOILGYZ5TPUA, seg_pos={0 0}, upload=332778]: Uploading piece... Mar 18 20:07:02.118962399 (297h6m59.715733445s) [noise=true , address=77.90.31.18:4810 , piece_id=SJLQVL5NHWP3KRPYZSD7VTFDM6TGSYZGIXMF4TBRJ3TT4HQFSY5Q, seg_pos={0 0}, upload=332778]: Uploading piece... Mar 18 20:07:02.119740375 (297h6m59.716511409s) [noise=true , address=145.79.227.19:4082 , piece_id=EXK44YLLVLD2JSX5HWVCAB5HUFKI52HVDBTTAXJME7PCHGANPKZA, seg_pos={0 0}, upload=332778]: Uploading piece... Mar 18 20:07:02.119929226 (297h6m59.716700249s) [noise=true , address=145.79.227.9:4369 , piece_id=ZFOPTZ5Y7ZUMA4QITC76KRV2XDD74NQ4FXHKSEYBYGREPA2V3UPQ, seg_pos={0 0}, upload=332778]: Uploading piece... Mar 18 20:07:02.120326891 (297h6m59.717097925s) [noise=true , address=145.79.227.42:4125 , piece_id=5WBU454RPQGE3XD4CTAMZ2DZNAWQOZY6P2LX4YUYENWX3KRJUKJA, seg_pos={0 0}, upload=332778]: Uploading piece... Mar 18 20:07:02.120633870 (297h6m59.717404893s) [noise=true , address=145.79.227.34:4905 , piece_id=4IVTVQ6X3VSFHMTFYCKDAGZ72FBZRDFBHVGJLYLGYWZLG6QFEMRA, seg_pos={0 0}, upload=332778]: Uploading piece... Mar 18 20:07:02.120895233 (297h6m59.717666267s) [noise=true , address=145.79.227.24:4585 , piece_id=SID3EEAMTC53ZSQYKTER5XPHCJGBC3XJIJNO3L2PIZT34ZZG4BHA, seg_pos={0 0}, upload=332778]: Uploading piece... Mar 18 20:07:02.121459406 (297h6m59.718230442s) [noise=true , address=45.91.116.11:4118 , piece_id=KDHBVJNAQXJDSNGDUHQEEAYFZIJNKZYE3S4NLYTWK5PLGX5JGXGA, seg_pos={0 0}, upload=332778]: Uploading piece... Mar 18 20:07:02.121612143 (297h6m59.718383158s) [noise=true , address=45.91.116.5:4298 , piece_id=QIFBVGI4BU3AM2QPX5J7TPY23GMXHTCAJQEFG7MHSOCT4L6WIHLQ, seg_pos={0 0}, upload=332778]: Uploading piece... Mar 18 20:07:02.121963977 (297h6m59.718735013s) [noise=true , address=77.90.31.14:4665 , piece_id=5ZFRAWV7VCAXFQRUPBXT4FXVACUZNMJWO52MZU32TYNXBLZ2ZXEA, seg_pos={0 0}, upload=332778]: Uploading piece... Mar 18 20:07:02.122501166 (297h6m59.719272192s) [noise=true , address=77.90.31.13:4629 , piece_id=O5QMXWEKJTDVUKQ4I6NJLSRWIG3KPYFW5JJJUIXI6UJQPRLF7FTA, seg_pos={0 0}, upload=332778]: Uploading piece... Mar 18 20:07:02.122524377 (297h6m59.719295411s) [noise=true , address=145.79.227.36:4520 , piece_id=DB6ZRXJWAADTPXMEZVW7J36OWYFFIJMRJVVJVDGA6CPYSPZIFWTA, seg_pos={0 0}, upload=332778]: Uploading piece... Mar 18 20:07:02.123108388 (297h6m59.719879423s) [noise=true , address=79.127.219.46:4891 , piece_id=LUL3WAM3XJR3HTDBUJHCS2IY2PZY3MQXVVM6IEJPRRYEC7IKHTWA, seg_pos={0 0}, upload=332778]: Uploading piece... Mar 18 20:07:02.123402779 (297h6m59.720173806s) [noise=true , address=77.90.31.16:4736 , piece_id=GMOVGQUX3JOSJEO2HOAI52Z2IT6ATSDMZLDWNDKNFGSJHOS5QYBQ, seg_pos={0 0}, upload=332778]: Uploading piece... Mar 18 20:07:02.124558739 (297h6m59.721329776s) [noise=true , address=45.91.116.8:4079 , piece_id=DG75AURQXLXGGOIMZTBQO6AXDM64G76QDCHWRAG3DY733TCB2AAQ, seg_pos={0 0}, upload=332778]: Uploading piece... Mar 18 20:07:02.125596207 (297h6m59.722367238s) [noise=true , address=45.91.116.13:4188 , piece_id=BTXACUXJ6UA6LDAG6SBQYYCGCQHAV7JZUFNHV2YXFHIS3ALMDUXQ, seg_pos={0 0}, upload=332778]: Uploading piece... Mar 18 20:07:02.218924261 (297h6m59.815695275s) [upload=332778]: Next returned nil segment Mar 18 20:07:02.218938702 (297h6m59.815709725s) [upload=332778]: Waiting for error group managing segments... Mar 18 20:07:02.223558869 (297h6m59.820329931s) [noise=true , address=45.91.116.13:4188 , piece_id=BTXACUXJ6UA6LDAG6SBQYYCGCQHAV7JZUFNHV2YXFHIS3ALMDUXQ, seg_pos={0 0}, upload=332778]: Done uploading piece. err: <nil> Mar 18 20:07:02.227338598 (297h6m59.824109622s) [noise=true , address=79.127.219.46:4891 , piece_id=LUL3WAM3XJR3HTDBUJHCS2IY2PZY3MQXVVM6IEJPRRYEC7IKHTWA, seg_pos={0 0}, upload=332778]: Done uploading piece. err: <nil> Mar 18 20:07:02.227526619 (297h6m59.824297642s) [noise=true , address=45.91.116.11:4118 , piece_id=KDHBVJNAQXJDSNGDUHQEEAYFZIJNKZYE3S4NLYTWK5PLGX5JGXGA, seg_pos={0 0}, upload=332778]: Done uploading piece. err: <nil> Mar 18 20:07:02.228386926 (297h6m59.82515796s) [noise=true , address=77.90.31.14:4665 , piece_id=5ZFRAWV7VCAXFQRUPBXT4FXVACUZNMJWO52MZU32TYNXBLZ2ZXEA, seg_pos={0 0}, upload=332778]: Done uploading piece. err: <nil> Mar 18 20:07:02.233526771 (297h6m59.830297806s) [noise=true , address=45.91.116.8:4079 , piece_id=DG75AURQXLXGGOIMZTBQO6AXDM64G76QDCHWRAG3DY733TCB2AAQ, seg_pos={0 0}, upload=332778]: Done uploading piece. err: <nil> Mar 18 20:07:02.240960557 (297h6m59.837731587s) [noise=true , address=45.91.116.5:4298 , piece_id=QIFBVGI4BU3AM2QPX5J7TPY23GMXHTCAJQEFG7MHSOCT4L6WIHLQ, seg_pos={0 0}, upload=332778]: Done uploading piece. err: <nil> Mar 18 20:07:02.241684427 (297h6m59.838455454s) [noise=true , address=77.90.31.13:4629 , piece_id=O5QMXWEKJTDVUKQ4I6NJLSRWIG3KPYFW5JJJUIXI6UJQPRLF7FTA, seg_pos={0 0}, upload=332778]: Done uploading piece. err: <nil> Mar 18 20:07:02.242068222 (297h6m59.838839246s) [noise=true , address=77.90.31.16:4736 , piece_id=GMOVGQUX3JOSJEO2HOAI52Z2IT6ATSDMZLDWNDKNFGSJHOS5QYBQ, seg_pos={0 0}, upload=332778]: Done uploading piece. err: <nil> Mar 18 20:07:02.245716670 (297h6m59.842487683s) [noise=true , address=77.90.31.18:4810 , piece_id=SJLQVL5NHWP3KRPYZSD7VTFDM6TGSYZGIXMF4TBRJ3TT4HQFSY5Q, seg_pos={0 0}, upload=332778]: Done uploading piece. err: <nil> Mar 18 20:07:28.683135142 (297h7m26.279906345s) [noise=true , address=145.79.227.42:4125 , piece_id=5WBU454RPQGE3XD4CTAMZ2DZNAWQOZY6P2LX4YUYENWX3KRJUKJA, seg_pos={0 0}, upload=332778]: Done uploading piece. err: ecclient: upload failed (node:1MrdHwiDVe8m6wcaQJrYetza9qckLruJXiU9UYbMxGCSWWMV1S, address:145.79.227.42:4125): protocol: rpc: dial tcp 145.79.227.42:4125: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:07:28.683603562 (297h7m26.280374594s) [noise=true , address=145.79.227.22:4509 , piece_id=WUNQ6CLYVMZ4SXAH4ITENGEQ47QSMHBPJIFBZ2KSCOILGYZ5TPUA, seg_pos={0 0}, upload=332778]: Done uploading piece. err: ecclient: upload failed (node:1229E58YirF37sZU6hfZGrDNFSxQDYsWSo4x3mG9BoYsNsLNtjy, address:145.79.227.22:4509): protocol: rpc: dial tcp 145.79.227.22:4509: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:07:28.683656022 (297h7m26.280427056s) [noise=true , address=145.79.227.34:4905 , piece_id=4IVTVQ6X3VSFHMTFYCKDAGZ72FBZRDFBHVGJLYLGYWZLG6QFEMRA, seg_pos={0 0}, upload=332778]: Done uploading piece. err: ecclient: upload failed (node:12ffKESNSVpXQzbwJinP6vvgE6zoF3RFz2NRUuuJHvd7bvk1gc9, address:145.79.227.34:4905): protocol: rpc: dial tcp 145.79.227.34:4905: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:07:28.683906298 (297h7m26.280677333s) [noise=true , address=145.79.227.36:4520 , piece_id=DB6ZRXJWAADTPXMEZVW7J36OWYFFIJMRJVVJVDGA6CPYSPZIFWTA, seg_pos={0 0}, upload=332778]: Done uploading piece. err: ecclient: upload failed (node:12eNRsboM5zFiyBVPLuQfgb5QSAc5bZFizK9KJHudNd9nstVGyy, address:145.79.227.36:4520): protocol: rpc: dial tcp 145.79.227.36:4520: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:07:28.684046796 (297h7m26.280817825s) [noise=true , address=145.79.227.24:4585 , piece_id=SID3EEAMTC53ZSQYKTER5XPHCJGBC3XJIJNO3L2PIZT34ZZG4BHA, seg_pos={0 0}, upload=332778]: Done uploading piece. err: ecclient: upload failed (node:1TyK6onzVds1kucaD6Y4aPssPDM5FzzQ91dTrez5B7ttfwvXm, address:145.79.227.24:4585): protocol: context deadline exceeded; context deadline exceeded Mar 18 20:07:28.684231613 (297h7m26.281002648s) [noise=true , address=145.79.227.19:4082 , piece_id=EXK44YLLVLD2JSX5HWVCAB5HUFKI52HVDBTTAXJME7PCHGANPKZA, seg_pos={0 0}, upload=332778]: Done uploading piece. err: ecclient: upload failed (node:12DKsNCtqxLvpJwLszdLs74V3eb8NH2VAV9V3bngzUczawmNoLK, address:145.79.227.19:4082): protocol: context deadline exceeded; context deadline exceeded Mar 18 20:07:28.684445387 (297h7m26.281216417s) [noise=true , address=145.79.227.9:4369 , piece_id=ZFOPTZ5Y7ZUMA4QITC76KRV2XDD74NQ4FXHKSEYBYGREPA2V3UPQ, seg_pos={0 0}, upload=332778]: Done uploading piece. err: ecclient: upload failed (node:1nwygPHyRJEZkjCoSZJTVhXDKLHc7AonVMzKMht6ppwZwhcGCT, address:145.79.227.9:4369): protocol: context deadline exceeded; context deadline exceeded Mar 18 20:07:28.684479347 (297h7m26.281250361s) [seg_pos={0 0}, upload=332778]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Mar 18 20:07:28.707270187 (297h7m26.304041202s) [upload=332778]: Done uploading segments... Mar 18 20:07:28.707312813 (297h7m26.304083843s) [upload=332778]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Duration: 12.912701774s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 43 out of 60 is failed (threshold=3) [..............XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.XXXXXX..] duration: 18.071013493s trace: https://jaeger.dev.storj.io/trace/3da6329b1a74a55f FAIL (2026-03-18 20:08:38 UTC) operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 9) ======LOGS====== Mar 18 20:06:58.715947398 (297h6m56.312718432s) [upload=332778]: Starting upload Mar 18 20:06:58.716027398 (297h6m56.312798434s) [upload=332778]: Done starting upload Mar 18 20:06:58.716264342 (297h6m56.313035366s) [upload=332778]: Uploading segments... Mar 18 20:06:58.716526836 (297h6m56.313297865s) [upload=332778]: Got next segment. Inline: false Mar 18 20:06:58.716595354 (297h6m56.313366377s) [upload=332778]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 20:06:58.716604668 (297h6m56.313375688s) [upload=332778]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 18 20:07:02.118470802 (297h6m59.715241839s) [seg_pos={0 0}, upload=332778]: Begin upload segment... Mar 18 20:07:02.118558314 (297h6m59.715329349s) [seg_pos={0 0}, upload=332778]: Done begin upload segment. Mar 18 20:07:02.118630613 (297h6m59.715401646s) [noise=true , address=145.79.227.22:4509 , piece_id=WUNQ6CLYVMZ4SXAH4ITENGEQ47QSMHBPJIFBZ2KSCOILGYZ5TPUA, seg_pos={0 0}, upload=332778]: Uploading piece... Mar 18 20:07:02.118962399 (297h6m59.715733445s) [noise=true , address=77.90.31.18:4810 , piece_id=SJLQVL5NHWP3KRPYZSD7VTFDM6TGSYZGIXMF4TBRJ3TT4HQFSY5Q, seg_pos={0 0}, upload=332778]: Uploading piece... Mar 18 20:07:02.119740375 (297h6m59.716511409s) [noise=true , address=145.79.227.19:4082 , piece_id=EXK44YLLVLD2JSX5HWVCAB5HUFKI52HVDBTTAXJME7PCHGANPKZA, seg_pos={0 0}, upload=332778]: Uploading piece... Mar 18 20:07:02.119929226 (297h6m59.716700249s) [noise=true , address=145.79.227.9:4369 , piece_id=ZFOPTZ5Y7ZUMA4QITC76KRV2XDD74NQ4FXHKSEYBYGREPA2V3UPQ, seg_pos={0 0}, upload=332778]: Uploading piece... Mar 18 20:07:02.120326891 (297h6m59.717097925s) [noise=true , address=145.79.227.42:4125 , piece_id=5WBU454RPQGE3XD4CTAMZ2DZNAWQOZY6P2LX4YUYENWX3KRJUKJA, seg_pos={0 0}, upload=332778]: Uploading piece... Mar 18 20:07:02.120633870 (297h6m59.717404893s) [noise=true , address=145.79.227.34:4905 , piece_id=4IVTVQ6X3VSFHMTFYCKDAGZ72FBZRDFBHVGJLYLGYWZLG6QFEMRA, seg_pos={0 0}, upload=332778]: Uploading piece... Mar 18 20:07:02.120895233 (297h6m59.717666267s) [noise=true , address=145.79.227.24:4585 , piece_id=SID3EEAMTC53ZSQYKTER5XPHCJGBC3XJIJNO3L2PIZT34ZZG4BHA, seg_pos={0 0}, upload=332778]: Uploading piece... Mar 18 20:07:02.121459406 (297h6m59.718230442s) [noise=true , address=45.91.116.11:4118 , piece_id=KDHBVJNAQXJDSNGDUHQEEAYFZIJNKZYE3S4NLYTWK5PLGX5JGXGA, seg_pos={0 0}, upload=332778]: Uploading piece... Mar 18 20:07:02.121612143 (297h6m59.718383158s) [noise=true , address=45.91.116.5:4298 , piece_id=QIFBVGI4BU3AM2QPX5J7TPY23GMXHTCAJQEFG7MHSOCT4L6WIHLQ, seg_pos={0 0}, upload=332778]: Uploading piece... Mar 18 20:07:02.121963977 (297h6m59.718735013s) [noise=true , address=77.90.31.14:4665 , piece_id=5ZFRAWV7VCAXFQRUPBXT4FXVACUZNMJWO52MZU32TYNXBLZ2ZXEA, seg_pos={0 0}, upload=332778]: Uploading piece... Mar 18 20:07:02.122501166 (297h6m59.719272192s) [noise=true , address=77.90.31.13:4629 , piece_id=O5QMXWEKJTDVUKQ4I6NJLSRWIG3KPYFW5JJJUIXI6UJQPRLF7FTA, seg_pos={0 0}, upload=332778]: Uploading piece... Mar 18 20:07:02.122524377 (297h6m59.719295411s) [noise=true , address=145.79.227.36:4520 , piece_id=DB6ZRXJWAADTPXMEZVW7J36OWYFFIJMRJVVJVDGA6CPYSPZIFWTA, seg_pos={0 0}, upload=332778]: Uploading piece... Mar 18 20:07:02.123108388 (297h6m59.719879423s) [noise=true , address=79.127.219.46:4891 , piece_id=LUL3WAM3XJR3HTDBUJHCS2IY2PZY3MQXVVM6IEJPRRYEC7IKHTWA, seg_pos={0 0}, upload=332778]: Uploading piece... Mar 18 20:07:02.123402779 (297h6m59.720173806s) [noise=true , address=77.90.31.16:4736 , piece_id=GMOVGQUX3JOSJEO2HOAI52Z2IT6ATSDMZLDWNDKNFGSJHOS5QYBQ, seg_pos={0 0}, upload=332778]: Uploading piece... Mar 18 20:07:02.124558739 (297h6m59.721329776s) [noise=true , address=45.91.116.8:4079 , piece_id=DG75AURQXLXGGOIMZTBQO6AXDM64G76QDCHWRAG3DY733TCB2AAQ, seg_pos={0 0}, upload=332778]: Uploading piece... Mar 18 20:07:02.125596207 (297h6m59.722367238s) [noise=true , address=45.91.116.13:4188 , piece_id=BTXACUXJ6UA6LDAG6SBQYYCGCQHAV7JZUFNHV2YXFHIS3ALMDUXQ, seg_pos={0 0}, upload=332778]: Uploading piece... Mar 18 20:07:02.218924261 (297h6m59.815695275s) [upload=332778]: Next returned nil segment Mar 18 20:07:02.218938702 (297h6m59.815709725s) [upload=332778]: Waiting for error group managing segments... Mar 18 20:07:02.223558869 (297h6m59.820329931s) [noise=true , address=45.91.116.13:4188 , piece_id=BTXACUXJ6UA6LDAG6SBQYYCGCQHAV7JZUFNHV2YXFHIS3ALMDUXQ, seg_pos={0 0}, upload=332778]: Done uploading piece. err: <nil> Mar 18 20:07:02.227338598 (297h6m59.824109622s) [noise=true , address=79.127.219.46:4891 , piece_id=LUL3WAM3XJR3HTDBUJHCS2IY2PZY3MQXVVM6IEJPRRYEC7IKHTWA, seg_pos={0 0}, upload=332778]: Done uploading piece. err: <nil> Mar 18 20:07:02.227526619 (297h6m59.824297642s) [noise=true , address=45.91.116.11:4118 , piece_id=KDHBVJNAQXJDSNGDUHQEEAYFZIJNKZYE3S4NLYTWK5PLGX5JGXGA, seg_pos={0 0}, upload=332778]: Done uploading piece. err: <nil> Mar 18 20:07:02.228386926 (297h6m59.82515796s) [noise=true , address=77.90.31.14:4665 , piece_id=5ZFRAWV7VCAXFQRUPBXT4FXVACUZNMJWO52MZU32TYNXBLZ2ZXEA, seg_pos={0 0}, upload=332778]: Done uploading piece. err: <nil> Mar 18 20:07:02.233526771 (297h6m59.830297806s) [noise=true , address=45.91.116.8:4079 , piece_id=DG75AURQXLXGGOIMZTBQO6AXDM64G76QDCHWRAG3DY733TCB2AAQ, seg_pos={0 0}, upload=332778]: Done uploading piece. err: <nil> Mar 18 20:07:02.240960557 (297h6m59.837731587s) [noise=true , address=45.91.116.5:4298 , piece_id=QIFBVGI4BU3AM2QPX5J7TPY23GMXHTCAJQEFG7MHSOCT4L6WIHLQ, seg_pos={0 0}, upload=332778]: Done uploading piece. err: <nil> Mar 18 20:07:02.241684427 (297h6m59.838455454s) [noise=true , address=77.90.31.13:4629 , piece_id=O5QMXWEKJTDVUKQ4I6NJLSRWIG3KPYFW5JJJUIXI6UJQPRLF7FTA, seg_pos={0 0}, upload=332778]: Done uploading piece. err: <nil> Mar 18 20:07:02.242068222 (297h6m59.838839246s) [noise=true , address=77.90.31.16:4736 , piece_id=GMOVGQUX3JOSJEO2HOAI52Z2IT6ATSDMZLDWNDKNFGSJHOS5QYBQ, seg_pos={0 0}, upload=332778]: Done uploading piece. err: <nil> Mar 18 20:07:02.245716670 (297h6m59.842487683s) [noise=true , address=77.90.31.18:4810 , piece_id=SJLQVL5NHWP3KRPYZSD7VTFDM6TGSYZGIXMF4TBRJ3TT4HQFSY5Q, seg_pos={0 0}, upload=332778]: Done uploading piece. err: <nil> Mar 18 20:07:28.683135142 (297h7m26.279906345s) [noise=true , address=145.79.227.42:4125 , piece_id=5WBU454RPQGE3XD4CTAMZ2DZNAWQOZY6P2LX4YUYENWX3KRJUKJA, seg_pos={0 0}, upload=332778]: Done uploading piece. err: ecclient: upload failed (node:1MrdHwiDVe8m6wcaQJrYetza9qckLruJXiU9UYbMxGCSWWMV1S, address:145.79.227.42:4125): protocol: rpc: dial tcp 145.79.227.42:4125: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:07:28.683603562 (297h7m26.280374594s) [noise=true , address=145.79.227.22:4509 , piece_id=WUNQ6CLYVMZ4SXAH4ITENGEQ47QSMHBPJIFBZ2KSCOILGYZ5TPUA, seg_pos={0 0}, upload=332778]: Done uploading piece. err: ecclient: upload failed (node:1229E58YirF37sZU6hfZGrDNFSxQDYsWSo4x3mG9BoYsNsLNtjy, address:145.79.227.22:4509): protocol: rpc: dial tcp 145.79.227.22:4509: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:07:28.683656022 (297h7m26.280427056s) [noise=true , address=145.79.227.34:4905 , piece_id=4IVTVQ6X3VSFHMTFYCKDAGZ72FBZRDFBHVGJLYLGYWZLG6QFEMRA, seg_pos={0 0}, upload=332778]: Done uploading piece. err: ecclient: upload failed (node:12ffKESNSVpXQzbwJinP6vvgE6zoF3RFz2NRUuuJHvd7bvk1gc9, address:145.79.227.34:4905): protocol: rpc: dial tcp 145.79.227.34:4905: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:07:28.683906298 (297h7m26.280677333s) [noise=true , address=145.79.227.36:4520 , piece_id=DB6ZRXJWAADTPXMEZVW7J36OWYFFIJMRJVVJVDGA6CPYSPZIFWTA, seg_pos={0 0}, upload=332778]: Done uploading piece. err: ecclient: upload failed (node:12eNRsboM5zFiyBVPLuQfgb5QSAc5bZFizK9KJHudNd9nstVGyy, address:145.79.227.36:4520): protocol: rpc: dial tcp 145.79.227.36:4520: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:07:28.684046796 (297h7m26.280817825s) [noise=true , address=145.79.227.24:4585 , piece_id=SID3EEAMTC53ZSQYKTER5XPHCJGBC3XJIJNO3L2PIZT34ZZG4BHA, seg_pos={0 0}, upload=332778]: Done uploading piece. err: ecclient: upload failed (node:1TyK6onzVds1kucaD6Y4aPssPDM5FzzQ91dTrez5B7ttfwvXm, address:145.79.227.24:4585): protocol: context deadline exceeded; context deadline exceeded Mar 18 20:07:28.684231613 (297h7m26.281002648s) [noise=true , address=145.79.227.19:4082 , piece_id=EXK44YLLVLD2JSX5HWVCAB5HUFKI52HVDBTTAXJME7PCHGANPKZA, seg_pos={0 0}, upload=332778]: Done uploading piece. err: ecclient: upload failed (node:12DKsNCtqxLvpJwLszdLs74V3eb8NH2VAV9V3bngzUczawmNoLK, address:145.79.227.19:4082): protocol: context deadline exceeded; context deadline exceeded Mar 18 20:07:28.684445387 (297h7m26.281216417s) [noise=true , address=145.79.227.9:4369 , piece_id=ZFOPTZ5Y7ZUMA4QITC76KRV2XDD74NQ4FXHKSEYBYGREPA2V3UPQ, seg_pos={0 0}, upload=332778]: Done uploading piece. err: ecclient: upload failed (node:1nwygPHyRJEZkjCoSZJTVhXDKLHc7AonVMzKMht6ppwZwhcGCT, address:145.79.227.9:4369): protocol: context deadline exceeded; context deadline exceeded Mar 18 20:07:28.684479347 (297h7m26.281250361s) [seg_pos={0 0}, upload=332778]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Mar 18 20:07:28.707270187 (297h7m26.304041202s) [upload=332778]: Done uploading segments... Mar 18 20:07:28.707312813 (297h7m26.304083843s) [upload=332778]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Duration: 18.071063494s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 6 out of 60 is failed (threshold=3) [......................X..........XX..X....X........X........] duration: 20.43078596s trace: https://jaeger.dev.storj.io/trace/589419efc52e5e2b FAIL (2026-03-18 20:08:33 UTC) operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== Mar 18 20:00:59.343084966 (297h0m56.939855989s) [upload=332666]: Starting upload Mar 18 20:00:59.343175518 (297h0m56.939946548s) [upload=332666]: Done starting upload Mar 18 20:00:59.343565585 (297h0m56.940336612s) [upload=332666]: Uploading segments... Mar 18 20:00:59.344256647 (297h0m56.941027676s) [upload=332666]: Got next segment. Inline: false Mar 18 20:00:59.344352215 (297h0m56.941123241s) [upload=332666]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 20:00:59.344358706 (297h0m56.941129732s) [upload=332666]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 18 20:01:02.340259299 (297h0m59.937030329s) [seg_pos={0 0}, upload=332666]: Begin upload segment... Mar 18 20:01:02.340368743 (297h0m59.937139767s) [seg_pos={0 0}, upload=332666]: Done begin upload segment. Mar 18 20:01:02.340479563 (297h0m59.93725059s) [noise=true , address=45.91.116.8:4061 , piece_id=6XFVKT4AAO7QN457OQMRKPI6R24TJTAN47UXLEVOEFQVOS6M6C3Q, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.340523837 (297h0m59.93729486s) [noise=true , address=79.127.205.233:4166 , piece_id=SFGK5YCWHFRVXF6QNHYP2CYJCZLSTQ76OYRVJPWWXQTF2KFSFJKA, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.341468438 (297h0m59.938239447s) [noise=true , address=45.91.116.4:4961 , piece_id=ZMUGQECUVPVHDNKN75XPBET5M2HFXXK4WMTP4V5KEK63JVZCX5EA, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.341520025 (297h0m59.938291058s) [noise=true , address=167.17.72.71:4815 , piece_id=CXGTYNNBILN3TT7I6Z7UI75F2ZCS4QUTLS6KC4FJ5IHS3XOYWDJQ, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.342232992 (297h0m59.939004023s) [noise=true , address=145.79.227.36:4524 , piece_id=3TAV3P55ITTHBAZEUUUTOAHJRGQXJXXVGZJP6AVR2ANVNOR3S54Q, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.342917368 (297h0m59.939688398s) [noise=true , address=193.149.164.71:4297 , piece_id=4IJPXLPCX3LKFOBOSRHK2DTNRKH67K7FXJUKQ3JPRWHS47ZFEIDA, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.343244957 (297h0m59.940015982s) [noise=true , address=145.79.227.33:4892 , piece_id=DQ6ALC3VOAGFIJI3IKR6I7QFYM22C66QWAAXMH6NXI2FIQEZNQYA, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.344270079 (297h0m59.941041117s) [noise=true , address=145.79.227.21:4497 , piece_id=SP6EKJB7W4QXF7OIBVKEFWCPOZFL5IRIEDVAGS2M4QJSO34CUVRQ, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.345025057 (297h0m59.941796084s) [noise=true , address=77.90.31.17:4793 , piece_id=2U4IBUA2GGUGDBXQGIGQKUK4ZR3L7CDKC6MTGIKAUME4F7MFZRLA, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.345115802 (297h0m59.941886836s) [noise=true , address=167.17.72.69:4731 , piece_id=VOEVIKKFJMC2Y3TPQIDMYUP37NLFU7KPRH4HRRSMQLWUKTSLOH2Q, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.345690600 (297h0m59.942461634s) [noise=true , address=77.90.31.13:4648 , piece_id=BPMB5CFOYHFT3BS4ORR6JF6BNFC2AOAJZOSPMG67NAMLGGP5XVEQ, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.346311592 (297h0m59.943082623s) [noise=true , address=79.127.205.234:4202 , piece_id=YY4LYWX26O6RIOBONESNKTUZEFZCRL4UUX3NEBQ53AJV4IEDG5MQ, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.346791142 (297h0m59.943562166s) [noise=true , address=45.91.116.10:4110 , piece_id=UYLMDCWJEJOHGICA37KNH7G5AQXNIPRH5NRYLBUUMGDRVSLTEYHQ, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.347094999 (297h0m59.94386607s) [noise=true , address=145.79.227.17:4987 , piece_id=PT3TWVB65JZFEWVH4IUZEU2N53FQ57QBRWVLWAUFP4UDLY3LWNLQ, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.347781281 (297h0m59.944552305s) [noise=true , address=79.127.219.46:4919 , piece_id=KK6HRIVBMYLIHRHGPQH3AWMNSVHTTIBESHDMJZA72T446RX6QKRQ, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.348263123 (297h0m59.945034137s) [noise=true , address=193.149.164.68:4033 , piece_id=JZPQKPYGO3W3RZZBITQ73X2XEPIBBSLGRMWIVOM4Z3GTV5UFKRTQ, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.348915505 (297h0m59.945686536s) [noise=true , address=145.79.227.7:4312 , piece_id=2FAL6E3RMBTBHZBXAJJVH2N5VGF5VYI3KNZXGNWCMWGBBPAUO2NQ, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.349366298 (297h0m59.946137337s) [noise=true , address=193.149.164.67:4007 , piece_id=HURQGH67I5B6V4YUV3PRZL3KSKN3XTSK3UMCHZRNT6L3W3PXXHPQ, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.429834098 (297h1m0.026605126s) [upload=332666]: Next returned nil segment Mar 18 20:01:02.429853411 (297h1m0.026624442s) [upload=332666]: Waiting for error group managing segments... Mar 18 20:01:02.446867680 (297h1m0.043638704s) [noise=true , address=45.91.116.10:4110 , piece_id=UYLMDCWJEJOHGICA37KNH7G5AQXNIPRH5NRYLBUUMGDRVSLTEYHQ, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil> Mar 18 20:01:02.451665165 (297h1m0.048436196s) [noise=true , address=45.91.116.4:4961 , piece_id=ZMUGQECUVPVHDNKN75XPBET5M2HFXXK4WMTP4V5KEK63JVZCX5EA, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil> Mar 18 20:01:02.452538886 (297h1m0.049309913s) [noise=true , address=77.90.31.17:4793 , piece_id=2U4IBUA2GGUGDBXQGIGQKUK4ZR3L7CDKC6MTGIKAUME4F7MFZRLA, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil> Mar 18 20:01:02.455597040 (297h1m0.052368067s) [noise=true , address=45.91.116.8:4061 , piece_id=6XFVKT4AAO7QN457OQMRKPI6R24TJTAN47UXLEVOEFQVOS6M6C3Q, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil> Mar 18 20:01:02.457848739 (297h1m0.054619766s) [noise=true , address=77.90.31.13:4648 , piece_id=BPMB5CFOYHFT3BS4ORR6JF6BNFC2AOAJZOSPMG67NAMLGGP5XVEQ, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil> Mar 18 20:01:02.461479709 (297h1m0.058250746s) [noise=true , address=79.127.219.46:4919 , piece_id=KK6HRIVBMYLIHRHGPQH3AWMNSVHTTIBESHDMJZA72T446RX6QKRQ, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil> Mar 18 20:01:02.676859466 (297h1m0.27363049s) [noise=true , address=167.17.72.69:4731 , piece_id=VOEVIKKFJMC2Y3TPQIDMYUP37NLFU7KPRH4HRRSMQLWUKTSLOH2Q, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil> Mar 18 20:01:02.909860743 (297h1m0.506631766s) [noise=true , address=79.127.205.234:4202 , piece_id=YY4LYWX26O6RIOBONESNKTUZEFZCRL4UUX3NEBQ53AJV4IEDG5MQ, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil> Mar 18 20:01:02.930427742 (297h1m0.527198754s) [noise=true , address=193.149.164.71:4297 , piece_id=4IJPXLPCX3LKFOBOSRHK2DTNRKH67K7FXJUKQ3JPRWHS47ZFEIDA, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil> Mar 18 20:01:02.944696889 (297h1m0.541467914s) [noise=true , address=193.149.164.67:4007 , piece_id=HURQGH67I5B6V4YUV3PRZL3KSKN3XTSK3UMCHZRNT6L3W3PXXHPQ, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil> Mar 18 20:01:02.951319183 (297h1m0.548090212s) [noise=true , address=193.149.164.68:4033 , piece_id=JZPQKPYGO3W3RZZBITQ73X2XEPIBBSLGRMWIVOM4Z3GTV5UFKRTQ, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil> Mar 18 20:01:02.975619436 (297h1m0.572390467s) [noise=true , address=79.127.205.233:4166 , piece_id=SFGK5YCWHFRVXF6QNHYP2CYJCZLSTQ76OYRVJPWWXQTF2KFSFJKA, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil> Mar 18 20:01:03.819918528 (297h1m1.416689564s) [noise=true , address=167.17.72.71:4815 , piece_id=CXGTYNNBILN3TT7I6Z7UI75F2ZCS4QUTLS6KC4FJ5IHS3XOYWDJQ, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil> Mar 18 20:01:59.307873420 (297h1m56.904644452s) [noise=true , address=145.79.227.7:4312 , piece_id=2FAL6E3RMBTBHZBXAJJVH2N5VGF5VYI3KNZXGNWCMWGBBPAUO2NQ, seg_pos={0 0}, upload=332666]: Done uploading piece. err: ecclient: upload failed (node:12AVp3qn5uwLPqYaZEwzcgoGMJhNga1BUh6bK5k2bxjWgZLQiiC, address:145.79.227.7:4312): protocol: rpc: dial tcp 145.79.227.7:4312: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:01:59.308026092 (297h1m56.904797124s) [noise=true , address=145.79.227.17:4987 , piece_id=PT3TWVB65JZFEWVH4IUZEU2N53FQ57QBRWVLWAUFP4UDLY3LWNLQ, seg_pos={0 0}, upload=332666]: Done uploading piece. err: ecclient: upload failed (node:1XsPVNGYUhKnXkQRfVfjyBRzyvhShSdsrnzXtwe4Xtf5MaHMvH, address:145.79.227.17:4987): protocol: rpc: dial tcp 145.79.227.17:4987: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:01:59.308195864 (297h1m56.904966894s) [noise=true , address=145.79.227.36:4524 , piece_id=3TAV3P55ITTHBAZEUUUTOAHJRGQXJXXVGZJP6AVR2ANVNOR3S54Q, seg_pos={0 0}, upload=332666]: Done uploading piece. err: ecclient: upload failed (node:1xfbCzjSwFDRrR1gPggeN8tNCLbRRWDVzCJ6mqhZsbVSe8d4s4, address:145.79.227.36:4524): protocol: rpc: dial tcp 145.79.227.36:4524: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:01:59.308316037 (297h1m56.90508706s) [noise=true , address=145.79.227.33:4892 , piece_id=DQ6ALC3VOAGFIJI3IKR6I7QFYM22C66QWAAXMH6NXI2FIQEZNQYA, seg_pos={0 0}, upload=332666]: Done uploading piece. err: ecclient: upload failed (node:12RHwFxcUjPsn12uHEAH3fBDJKv1v1tM8mRJXoFrZrG5djdswF5, address:145.79.227.33:4892): protocol: rpc: dial tcp 145.79.227.33:4892: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:01:59.308404403 (297h1m56.905175433s) [noise=true , address=145.79.227.21:4497 , piece_id=SP6EKJB7W4QXF7OIBVKEFWCPOZFL5IRIEDVAGS2M4QJSO34CUVRQ, seg_pos={0 0}, upload=332666]: Done uploading piece. err: ecclient: upload failed (node:12ZBe4we7VeFdthqRxyy1V9uQrA4r8ouNX68xqsk5bD26dkf2Gd, address:145.79.227.21:4497): protocol: context deadline exceeded; context deadline exceeded Mar 18 20:01:59.308437457 (297h1m56.90520847s) [seg_pos={0 0}, upload=332666]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 18 20:01:59.338288467 (297h1m56.93505949s) [upload=332666]: Done uploading segments... Mar 18 20:01:59.338335497 (297h1m56.935106523s) [upload=332666]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 20.430805044s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 8 out of 60 is failed (threshold=3) [....................XXX..........X...X.........X....X....X..] duration: 15.709755974s trace: https://jaeger.dev.storj.io/trace/75a9000feaeb0b3d FAIL (2026-03-18 20:08:29 UTC) operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== ar 18 20:06:04.693955863 (297h6m2.290726886s) [upload=332765]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 20:06:04.693961555 (297h6m2.290732583s) [upload=332765]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 18 20:06:07.394748117 (297h6m4.99151915s) [seg_pos={0 0}, upload=332765]: Begin upload segment... Mar 18 20:06:07.394875884 (297h6m4.991646919s) [seg_pos={0 0}, upload=332765]: Done begin upload segment. Mar 18 20:06:07.394887373 (297h6m4.991658385s) [noise=true , address=45.91.116.9:4276 , piece_id=J356KNX74ADIKN7PPKCRNGACHPVBJYZA3O5OAGVYSURLB4T6KSZA, seg_pos={0 0}, upload=332765]: Uploading piece... Mar 18 20:06:07.394990884 (297h6m4.991761926s) [noise=true , address=145.79.227.2:4129 , piece_id=2Y7VDPW5SS3D4J45EAMSLOEQAHNKYCZJEPP2FTP7GKVCYKMDQTLA, seg_pos={0 0}, upload=332765]: Uploading piece... Mar 18 20:06:07.396053654 (297h6m4.992824681s) [noise=true , address=45.91.116.11:4150 , piece_id=L2ZK2GKN4VY7W6SJ25MR65RL52UH3M66FGQLSLXI2N6LEZXJQVMA, seg_pos={0 0}, upload=332765]: Uploading piece... Mar 18 20:06:07.396081719 (297h6m4.992852749s) [noise=true , address=45.91.116.12:4246 , piece_id=T5BGURHRLDTEZZ4TORP6F5WNBR2TY3ZDWPWO4PGSXQ575BWTUMVA, seg_pos={0 0}, upload=332765]: Uploading piece... Mar 18 20:06:07.396842874 (297h6m4.993613899s) [noise=true , address=145.79.227.33:4875 , piece_id=5LQOGDTQD5CCBMKIZCSLBVQUHQWU6W5ELX6XKLFSFQLU3NVDLKPQ, seg_pos={0 0}, upload=332765]: Uploading piece... Mar 18 20:06:07.397161075 (297h6m4.993932121s) [noise=true , address=77.90.31.12:4615 , piece_id=HLRHRANL5IHJOB5QHLW2OIN5ZP56RKX3INJPWPU5YYHBHRYVJM6A, seg_pos={0 0}, upload=332765]: Uploading piece... Mar 18 20:06:07.397369139 (297h6m4.994140166s) [noise=true , address=77.90.31.11:4567 , piece_id=UN6S6RWYUVLUA5NPFFJ5TASQDZ26OU2IIWP3ZNFUFVJ3WJAA2PSQ, seg_pos={0 0}, upload=332765]: Uploading piece... Mar 18 20:06:07.397732013 (297h6m4.994503037s) [noise=true , address=45.91.116.14:4206 , piece_id=IDCMPAW3BEN5OQ7HUX3QDGFGKN76DM4E6CDN3M45SJQ4VU2OQGCQ, seg_pos={0 0}, upload=332765]: Uploading piece... Mar 18 20:06:07.397757704 (297h6m4.994528729s) [noise=true , address=145.79.227.3:4177 , piece_id=A7HSMKKEMCEJZENJ3CM7N3GZI3NX4BA2WIQZ27MTRONPXBZKZWHA, seg_pos={0 0}, upload=332765]: Uploading piece... Mar 18 20:06:07.398274955 (297h6m4.995045981s) [noise=true , address=193.149.164.71:4307 , piece_id=TZ5I3SSLCLL2AOKJTEFTMEE7DLXHXDKCJJLNP4WHDSGRCJLLBM6A, seg_pos={0 0}, upload=332765]: Uploading piece... Mar 18 20:06:07.398328768 (297h6m4.995099802s) [noise=true , address=79.127.205.233:4173 , piece_id=I64CQ5XLN6DUP2E465ZQLGOVE5NEIHLNWT33QB7WKJF5Q6QZRPLQ, seg_pos={0 0}, upload=332765]: Uploading piece... Mar 18 20:06:07.398815526 (297h6m4.995586656s) [noise=true , address=145.79.227.51:4424 , piece_id=5XCD27Q3GOCFGSHDUNX7QIGUQ6URBU2UREPVX5ZF7CFICF4X4UNQ, seg_pos={0 0}, upload=332765]: Uploading piece... Mar 18 20:06:07.399136293 (297h6m4.995907322s) [noise=true , address=64.49.13.137:4499 , piece_id=KTFNLPG67BVZMYZMJJO4BDRTASHDANE2FBYO3YUV2E4XBCU5NLCA, seg_pos={0 0}, upload=332765]: Uploading piece... Mar 18 20:06:07.399360723 (297h6m4.996131762s) [noise=true , address=45.91.116.4:4963 , piece_id=ZGY2X7JG2MPOJWKMFWP3VATYO3QTEOXPQ4KYZSOW4QOPST6OV2VQ, seg_pos={0 0}, upload=332765]: Uploading piece... Mar 18 20:06:07.399800937 (297h6m4.996571962s) [noise=true , address=145.79.227.24:4592 , piece_id=NGOLKV5BQKREBAYQHBSXHN2UKBHB52WPVBBO7NIFCCYTBF2MXPBA, seg_pos={0 0}, upload=332765]: Uploading piece... Mar 18 20:06:07.399957240 (297h6m4.996728268s) [noise=true , address=193.149.164.66:4993 , piece_id=CRRZAIXMIV673K5JLX743KZ3HFL5MBGUE5ZE4T4Q2WPCSUQ32JQA, seg_pos={0 0}, upload=332765]: Uploading piece... Mar 18 20:06:07.400205078 (297h6m4.996976103s) [noise=true , address=167.17.72.68:4668 , piece_id=KUTKCCK6Y54CMAIIWGINQERXEV6CGVPCOYEJAQUNPBWTHJ755KJQ, seg_pos={0 0}, upload=332765]: Uploading piece... Mar 18 20:06:07.400506584 (297h6m4.997277612s) [noise=true , address=145.79.227.49:4360 , piece_id=OP5BBV7CUZBSMBO6HTTHQCVXDQGS6XO5LGAZ4I2SKUSPGUAKBXKA, seg_pos={0 0}, upload=332765]: Uploading piece... Mar 18 20:06:07.471265449 (297h6m5.068036462s) [upload=332765]: Next returned nil segment Mar 18 20:06:07.471278058 (297h6m5.068049082s) [upload=332765]: Waiting for error group managing segments... Mar 18 20:06:07.475837847 (297h6m5.07260886s) [noise=true , address=77.90.31.11:4567 , piece_id=UN6S6RWYUVLUA5NPFFJ5TASQDZ26OU2IIWP3ZNFUFVJ3WJAA2PSQ, seg_pos={0 0}, upload=332765]: Done uploading piece. err: <nil> Mar 18 20:06:07.476715720 (297h6m5.073486747s) [noise=true , address=45.91.116.4:4963 , piece_id=ZGY2X7JG2MPOJWKMFWP3VATYO3QTEOXPQ4KYZSOW4QOPST6OV2VQ, seg_pos={0 0}, upload=332765]: Done uploading piece. err: <nil> Mar 18 20:06:07.488555609 (297h6m5.085326636s) [noise=true , address=45.91.116.12:4246 , piece_id=T5BGURHRLDTEZZ4TORP6F5WNBR2TY3ZDWPWO4PGSXQ575BWTUMVA, seg_pos={0 0}, upload=332765]: Done uploading piece. err: <nil> Mar 18 20:06:07.489803434 (297h6m5.086574468s) [noise=true , address=45.91.116.14:4206 , piece_id=IDCMPAW3BEN5OQ7HUX3QDGFGKN76DM4E6CDN3M45SJQ4VU2OQGCQ, seg_pos={0 0}, upload=332765]: Done uploading piece. err: <nil> Mar 18 20:06:07.490598252 (297h6m5.087369276s) [noise=true , address=45.91.116.11:4150 , piece_id=L2ZK2GKN4VY7W6SJ25MR65RL52UH3M66FGQLSLXI2N6LEZXJQVMA, seg_pos={0 0}, upload=332765]: Done uploading piece. err: <nil> Mar 18 20:06:07.492135441 (297h6m5.088906454s) [noise=true , address=45.91.116.9:4276 , piece_id=J356KNX74ADIKN7PPKCRNGACHPVBJYZA3O5OAGVYSURLB4T6KSZA, seg_pos={0 0}, upload=332765]: Done uploading piece. err: <nil> Mar 18 20:06:07.508501889 (297h6m5.105272902s) [noise=true , address=77.90.31.12:4615 , piece_id=HLRHRANL5IHJOB5QHLW2OIN5ZP56RKX3INJPWPU5YYHBHRYVJM6A, seg_pos={0 0}, upload=332765]: Done uploading piece. err: <nil> Mar 18 20:06:07.770562428 (297h6m5.367333451s) [noise=true , address=167.17.72.68:4668 , piece_id=KUTKCCK6Y54CMAIIWGINQERXEV6CGVPCOYEJAQUNPBWTHJ755KJQ, seg_pos={0 0}, upload=332765]: Done uploading piece. err: <nil> Mar 18 20:06:08.006027504 (297h6m5.602798531s) [noise=true , address=193.149.164.66:4993 , piece_id=CRRZAIXMIV673K5JLX743KZ3HFL5MBGUE5ZE4T4Q2WPCSUQ32JQA, seg_pos={0 0}, upload=332765]: Done uploading piece. err: <nil> Mar 18 20:06:08.016743696 (297h6m5.613514734s) [noise=true , address=193.149.164.71:4307 , piece_id=TZ5I3SSLCLL2AOKJTEFTMEE7DLXHXDKCJJLNP4WHDSGRCJLLBM6A, seg_pos={0 0}, upload=332765]: Done uploading piece. err: <nil> Mar 18 20:06:08.043257501 (297h6m5.640028528s) [noise=true , address=64.49.13.137:4499 , piece_id=KTFNLPG67BVZMYZMJJO4BDRTASHDANE2FBYO3YUV2E4XBCU5NLCA, seg_pos={0 0}, upload=332765]: Done uploading piece. err: <nil> Mar 18 20:06:08.323806978 (297h6m5.920577991s) [noise=true , address=79.127.205.233:4173 , piece_id=I64CQ5XLN6DUP2E465ZQLGOVE5NEIHLNWT33QB7WKJF5Q6QZRPLQ, seg_pos={0 0}, upload=332765]: Done uploading piece. err: <nil> Mar 18 20:07:04.645185715 (297h7m2.241956747s) [noise=true , address=145.79.227.3:4177 , piece_id=A7HSMKKEMCEJZENJ3CM7N3GZI3NX4BA2WIQZ27MTRONPXBZKZWHA, seg_pos={0 0}, upload=332765]: Done uploading piece. err: ecclient: upload failed (node:12rFuJ2JyBGWtaASJNnTdHoeKUY1mCfDYfJ74BkxK64U9it8FKF, address:145.79.227.3:4177): protocol: rpc: dial tcp 145.79.227.3:4177: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:07:04.645320026 (297h7m2.24209106s) [noise=true , address=145.79.227.24:4592 , piece_id=NGOLKV5BQKREBAYQHBSXHN2UKBHB52WPVBBO7NIFCCYTBF2MXPBA, seg_pos={0 0}, upload=332765]: Done uploading piece. err: ecclient: upload failed (node:1fB7rqiDhMNQ4j9smkctbKdnGCXXdfFPWTgTxRrwQVZ9fKEFeZ, address:145.79.227.24:4592): protocol: rpc: dial tcp 145.79.227.24:4592: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:07:04.645426518 (297h7m2.242197551s) [noise=true , address=145.79.227.2:4129 , piece_id=2Y7VDPW5SS3D4J45EAMSLOEQAHNKYCZJEPP2FTP7GKVCYKMDQTLA, seg_pos={0 0}, upload=332765]: Done uploading piece. err: ecclient: upload failed (node:12GJof7GVfEN9u2yVyWf6j24hnaTdD3ygoPRoicmSqSfcNJFzs6, address:145.79.227.2:4129): protocol: rpc: dial tcp 145.79.227.2:4129: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:07:04.645453468 (297h7m2.242224501s) [noise=true , address=145.79.227.49:4360 , piece_id=OP5BBV7CUZBSMBO6HTTHQCVXDQGS6XO5LGAZ4I2SKUSPGUAKBXKA, seg_pos={0 0}, upload=332765]: Done uploading piece. err: ecclient: upload failed (node:12T24rtYBJ47E6Ef1ZZKMWcpBrWBpCg5QV2ZHniCZZkzJF2yUNk, address:145.79.227.49:4360): protocol: rpc: dial tcp 145.79.227.49:4360: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:07:04.645552758 (297h7m2.242323785s) [noise=true , address=145.79.227.33:4875 , piece_id=5LQOGDTQD5CCBMKIZCSLBVQUHQWU6W5ELX6XKLFSFQLU3NVDLKPQ, seg_pos={0 0}, upload=332765]: Done uploading piece. err: ecclient: upload failed (node:1ijEJgYouyMkUqpLWWz7eH2NGvvs9XpN2ujDjupPykDe5eWcB1, address:145.79.227.33:4875): protocol: rpc: dial tcp 145.79.227.33:4875: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:07:04.645682576 (297h7m2.242453607s) [noise=true , address=145.79.227.51:4424 , piece_id=5XCD27Q3GOCFGSHDUNX7QIGUQ6URBU2UREPVX5ZF7CFICF4X4UNQ, seg_pos={0 0}, upload=332765]: Done uploading piece. err: ecclient: upload failed (node:12LwZixdSPnCm3Wm27PBY79jj52sjkNh2VuQexagieTGBp4Ps2k, address:145.79.227.51:4424): protocol: rpc: dial tcp 145.79.227.51:4424: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:07:04.645716594 (297h7m2.24248761s) [seg_pos={0 0}, upload=332765]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Mar 18 20:07:04.666238325 (297h7m2.263009349s) [upload=332765]: Done uploading segments... Mar 18 20:07:04.666276723 (297h7m2.263047746s) [upload=332765]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 15.709787401s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 51 out of 60 is failed (threshold=6) [.....XXXXXXXXX.XXX...XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 5.001057452s trace: https://jaeger.dev.storj.io/trace/20a82e04a7673308 FAIL (2026-03-18 20:08:22 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001077391s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 43 out of 60 is failed (threshold=3) [...............XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.XXXXXX.] duration: 1.604652065s trace: https://jaeger.dev.storj.io/trace/3da6329b1a74a55f FAIL (2026-03-18 20:08:02 UTC) operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 9) ======LOGS====== Mar 18 20:06:58.715947398 (297h6m56.312718432s) [upload=332778]: Starting upload Mar 18 20:06:58.716027398 (297h6m56.312798434s) [upload=332778]: Done starting upload Mar 18 20:06:58.716264342 (297h6m56.313035366s) [upload=332778]: Uploading segments... Mar 18 20:06:58.716526836 (297h6m56.313297865s) [upload=332778]: Got next segment. Inline: false Mar 18 20:06:58.716595354 (297h6m56.313366377s) [upload=332778]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 20:06:58.716604668 (297h6m56.313375688s) [upload=332778]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 18 20:07:02.118470802 (297h6m59.715241839s) [seg_pos={0 0}, upload=332778]: Begin upload segment... Mar 18 20:07:02.118558314 (297h6m59.715329349s) [seg_pos={0 0}, upload=332778]: Done begin upload segment. Mar 18 20:07:02.118630613 (297h6m59.715401646s) [noise=true , address=145.79.227.22:4509 , piece_id=WUNQ6CLYVMZ4SXAH4ITENGEQ47QSMHBPJIFBZ2KSCOILGYZ5TPUA, seg_pos={0 0}, upload=332778]: Uploading piece... Mar 18 20:07:02.118962399 (297h6m59.715733445s) [noise=true , address=77.90.31.18:4810 , piece_id=SJLQVL5NHWP3KRPYZSD7VTFDM6TGSYZGIXMF4TBRJ3TT4HQFSY5Q, seg_pos={0 0}, upload=332778]: Uploading piece... Mar 18 20:07:02.119740375 (297h6m59.716511409s) [noise=true , address=145.79.227.19:4082 , piece_id=EXK44YLLVLD2JSX5HWVCAB5HUFKI52HVDBTTAXJME7PCHGANPKZA, seg_pos={0 0}, upload=332778]: Uploading piece... Mar 18 20:07:02.119929226 (297h6m59.716700249s) [noise=true , address=145.79.227.9:4369 , piece_id=ZFOPTZ5Y7ZUMA4QITC76KRV2XDD74NQ4FXHKSEYBYGREPA2V3UPQ, seg_pos={0 0}, upload=332778]: Uploading piece... Mar 18 20:07:02.120326891 (297h6m59.717097925s) [noise=true , address=145.79.227.42:4125 , piece_id=5WBU454RPQGE3XD4CTAMZ2DZNAWQOZY6P2LX4YUYENWX3KRJUKJA, seg_pos={0 0}, upload=332778]: Uploading piece... Mar 18 20:07:02.120633870 (297h6m59.717404893s) [noise=true , address=145.79.227.34:4905 , piece_id=4IVTVQ6X3VSFHMTFYCKDAGZ72FBZRDFBHVGJLYLGYWZLG6QFEMRA, seg_pos={0 0}, upload=332778]: Uploading piece... Mar 18 20:07:02.120895233 (297h6m59.717666267s) [noise=true , address=145.79.227.24:4585 , piece_id=SID3EEAMTC53ZSQYKTER5XPHCJGBC3XJIJNO3L2PIZT34ZZG4BHA, seg_pos={0 0}, upload=332778]: Uploading piece... Mar 18 20:07:02.121459406 (297h6m59.718230442s) [noise=true , address=45.91.116.11:4118 , piece_id=KDHBVJNAQXJDSNGDUHQEEAYFZIJNKZYE3S4NLYTWK5PLGX5JGXGA, seg_pos={0 0}, upload=332778]: Uploading piece... Mar 18 20:07:02.121612143 (297h6m59.718383158s) [noise=true , address=45.91.116.5:4298 , piece_id=QIFBVGI4BU3AM2QPX5J7TPY23GMXHTCAJQEFG7MHSOCT4L6WIHLQ, seg_pos={0 0}, upload=332778]: Uploading piece... Mar 18 20:07:02.121963977 (297h6m59.718735013s) [noise=true , address=77.90.31.14:4665 , piece_id=5ZFRAWV7VCAXFQRUPBXT4FXVACUZNMJWO52MZU32TYNXBLZ2ZXEA, seg_pos={0 0}, upload=332778]: Uploading piece... Mar 18 20:07:02.122501166 (297h6m59.719272192s) [noise=true , address=77.90.31.13:4629 , piece_id=O5QMXWEKJTDVUKQ4I6NJLSRWIG3KPYFW5JJJUIXI6UJQPRLF7FTA, seg_pos={0 0}, upload=332778]: Uploading piece... Mar 18 20:07:02.122524377 (297h6m59.719295411s) [noise=true , address=145.79.227.36:4520 , piece_id=DB6ZRXJWAADTPXMEZVW7J36OWYFFIJMRJVVJVDGA6CPYSPZIFWTA, seg_pos={0 0}, upload=332778]: Uploading piece... Mar 18 20:07:02.123108388 (297h6m59.719879423s) [noise=true , address=79.127.219.46:4891 , piece_id=LUL3WAM3XJR3HTDBUJHCS2IY2PZY3MQXVVM6IEJPRRYEC7IKHTWA, seg_pos={0 0}, upload=332778]: Uploading piece... Mar 18 20:07:02.123402779 (297h6m59.720173806s) [noise=true , address=77.90.31.16:4736 , piece_id=GMOVGQUX3JOSJEO2HOAI52Z2IT6ATSDMZLDWNDKNFGSJHOS5QYBQ, seg_pos={0 0}, upload=332778]: Uploading piece... Mar 18 20:07:02.124558739 (297h6m59.721329776s) [noise=true , address=45.91.116.8:4079 , piece_id=DG75AURQXLXGGOIMZTBQO6AXDM64G76QDCHWRAG3DY733TCB2AAQ, seg_pos={0 0}, upload=332778]: Uploading piece... Mar 18 20:07:02.125596207 (297h6m59.722367238s) [noise=true , address=45.91.116.13:4188 , piece_id=BTXACUXJ6UA6LDAG6SBQYYCGCQHAV7JZUFNHV2YXFHIS3ALMDUXQ, seg_pos={0 0}, upload=332778]: Uploading piece... Mar 18 20:07:02.218924261 (297h6m59.815695275s) [upload=332778]: Next returned nil segment Mar 18 20:07:02.218938702 (297h6m59.815709725s) [upload=332778]: Waiting for error group managing segments... Mar 18 20:07:02.223558869 (297h6m59.820329931s) [noise=true , address=45.91.116.13:4188 , piece_id=BTXACUXJ6UA6LDAG6SBQYYCGCQHAV7JZUFNHV2YXFHIS3ALMDUXQ, seg_pos={0 0}, upload=332778]: Done uploading piece. err: <nil> Mar 18 20:07:02.227338598 (297h6m59.824109622s) [noise=true , address=79.127.219.46:4891 , piece_id=LUL3WAM3XJR3HTDBUJHCS2IY2PZY3MQXVVM6IEJPRRYEC7IKHTWA, seg_pos={0 0}, upload=332778]: Done uploading piece. err: <nil> Mar 18 20:07:02.227526619 (297h6m59.824297642s) [noise=true , address=45.91.116.11:4118 , piece_id=KDHBVJNAQXJDSNGDUHQEEAYFZIJNKZYE3S4NLYTWK5PLGX5JGXGA, seg_pos={0 0}, upload=332778]: Done uploading piece. err: <nil> Mar 18 20:07:02.228386926 (297h6m59.82515796s) [noise=true , address=77.90.31.14:4665 , piece_id=5ZFRAWV7VCAXFQRUPBXT4FXVACUZNMJWO52MZU32TYNXBLZ2ZXEA, seg_pos={0 0}, upload=332778]: Done uploading piece. err: <nil> Mar 18 20:07:02.233526771 (297h6m59.830297806s) [noise=true , address=45.91.116.8:4079 , piece_id=DG75AURQXLXGGOIMZTBQO6AXDM64G76QDCHWRAG3DY733TCB2AAQ, seg_pos={0 0}, upload=332778]: Done uploading piece. err: <nil> Mar 18 20:07:02.240960557 (297h6m59.837731587s) [noise=true , address=45.91.116.5:4298 , piece_id=QIFBVGI4BU3AM2QPX5J7TPY23GMXHTCAJQEFG7MHSOCT4L6WIHLQ, seg_pos={0 0}, upload=332778]: Done uploading piece. err: <nil> Mar 18 20:07:02.241684427 (297h6m59.838455454s) [noise=true , address=77.90.31.13:4629 , piece_id=O5QMXWEKJTDVUKQ4I6NJLSRWIG3KPYFW5JJJUIXI6UJQPRLF7FTA, seg_pos={0 0}, upload=332778]: Done uploading piece. err: <nil> Mar 18 20:07:02.242068222 (297h6m59.838839246s) [noise=true , address=77.90.31.16:4736 , piece_id=GMOVGQUX3JOSJEO2HOAI52Z2IT6ATSDMZLDWNDKNFGSJHOS5QYBQ, seg_pos={0 0}, upload=332778]: Done uploading piece. err: <nil> Mar 18 20:07:02.245716670 (297h6m59.842487683s) [noise=true , address=77.90.31.18:4810 , piece_id=SJLQVL5NHWP3KRPYZSD7VTFDM6TGSYZGIXMF4TBRJ3TT4HQFSY5Q, seg_pos={0 0}, upload=332778]: Done uploading piece. err: <nil> Mar 18 20:07:28.683135142 (297h7m26.279906345s) [noise=true , address=145.79.227.42:4125 , piece_id=5WBU454RPQGE3XD4CTAMZ2DZNAWQOZY6P2LX4YUYENWX3KRJUKJA, seg_pos={0 0}, upload=332778]: Done uploading piece. err: ecclient: upload failed (node:1MrdHwiDVe8m6wcaQJrYetza9qckLruJXiU9UYbMxGCSWWMV1S, address:145.79.227.42:4125): protocol: rpc: dial tcp 145.79.227.42:4125: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:07:28.683603562 (297h7m26.280374594s) [noise=true , address=145.79.227.22:4509 , piece_id=WUNQ6CLYVMZ4SXAH4ITENGEQ47QSMHBPJIFBZ2KSCOILGYZ5TPUA, seg_pos={0 0}, upload=332778]: Done uploading piece. err: ecclient: upload failed (node:1229E58YirF37sZU6hfZGrDNFSxQDYsWSo4x3mG9BoYsNsLNtjy, address:145.79.227.22:4509): protocol: rpc: dial tcp 145.79.227.22:4509: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:07:28.683656022 (297h7m26.280427056s) [noise=true , address=145.79.227.34:4905 , piece_id=4IVTVQ6X3VSFHMTFYCKDAGZ72FBZRDFBHVGJLYLGYWZLG6QFEMRA, seg_pos={0 0}, upload=332778]: Done uploading piece. err: ecclient: upload failed (node:12ffKESNSVpXQzbwJinP6vvgE6zoF3RFz2NRUuuJHvd7bvk1gc9, address:145.79.227.34:4905): protocol: rpc: dial tcp 145.79.227.34:4905: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:07:28.683906298 (297h7m26.280677333s) [noise=true , address=145.79.227.36:4520 , piece_id=DB6ZRXJWAADTPXMEZVW7J36OWYFFIJMRJVVJVDGA6CPYSPZIFWTA, seg_pos={0 0}, upload=332778]: Done uploading piece. err: ecclient: upload failed (node:12eNRsboM5zFiyBVPLuQfgb5QSAc5bZFizK9KJHudNd9nstVGyy, address:145.79.227.36:4520): protocol: rpc: dial tcp 145.79.227.36:4520: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:07:28.684046796 (297h7m26.280817825s) [noise=true , address=145.79.227.24:4585 , piece_id=SID3EEAMTC53ZSQYKTER5XPHCJGBC3XJIJNO3L2PIZT34ZZG4BHA, seg_pos={0 0}, upload=332778]: Done uploading piece. err: ecclient: upload failed (node:1TyK6onzVds1kucaD6Y4aPssPDM5FzzQ91dTrez5B7ttfwvXm, address:145.79.227.24:4585): protocol: context deadline exceeded; context deadline exceeded Mar 18 20:07:28.684231613 (297h7m26.281002648s) [noise=true , address=145.79.227.19:4082 , piece_id=EXK44YLLVLD2JSX5HWVCAB5HUFKI52HVDBTTAXJME7PCHGANPKZA, seg_pos={0 0}, upload=332778]: Done uploading piece. err: ecclient: upload failed (node:12DKsNCtqxLvpJwLszdLs74V3eb8NH2VAV9V3bngzUczawmNoLK, address:145.79.227.19:4082): protocol: context deadline exceeded; context deadline exceeded Mar 18 20:07:28.684445387 (297h7m26.281216417s) [noise=true , address=145.79.227.9:4369 , piece_id=ZFOPTZ5Y7ZUMA4QITC76KRV2XDD74NQ4FXHKSEYBYGREPA2V3UPQ, seg_pos={0 0}, upload=332778]: Done uploading piece. err: ecclient: upload failed (node:1nwygPHyRJEZkjCoSZJTVhXDKLHc7AonVMzKMht6ppwZwhcGCT, address:145.79.227.9:4369): protocol: context deadline exceeded; context deadline exceeded Mar 18 20:07:28.684479347 (297h7m26.281250361s) [seg_pos={0 0}, upload=332778]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Mar 18 20:07:28.707270187 (297h7m26.304041202s) [upload=332778]: Done uploading segments... Mar 18 20:07:28.707312813 (297h7m26.304083843s) [upload=332778]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Duration: 1.604670974s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 8 out of 60 is failed (threshold=3) [..................XXXXXX.XX.................................] duration: 20.32171957s trace: https://jaeger.dev.storj.io/trace/4b4d10a8051ad01b FAIL (2026-03-18 20:07:59 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 20.321750949s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 6 out of 60 is failed (threshold=3) [.......................X..........XX..X....X........X.......] duration: 14.329684612s trace: https://jaeger.dev.storj.io/trace/589419efc52e5e2b FAIL (2026-03-18 20:07:43 UTC) operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== Mar 18 20:00:59.343084966 (297h0m56.939855989s) [upload=332666]: Starting upload Mar 18 20:00:59.343175518 (297h0m56.939946548s) [upload=332666]: Done starting upload Mar 18 20:00:59.343565585 (297h0m56.940336612s) [upload=332666]: Uploading segments... Mar 18 20:00:59.344256647 (297h0m56.941027676s) [upload=332666]: Got next segment. Inline: false Mar 18 20:00:59.344352215 (297h0m56.941123241s) [upload=332666]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 20:00:59.344358706 (297h0m56.941129732s) [upload=332666]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 18 20:01:02.340259299 (297h0m59.937030329s) [seg_pos={0 0}, upload=332666]: Begin upload segment... Mar 18 20:01:02.340368743 (297h0m59.937139767s) [seg_pos={0 0}, upload=332666]: Done begin upload segment. Mar 18 20:01:02.340479563 (297h0m59.93725059s) [noise=true , address=45.91.116.8:4061 , piece_id=6XFVKT4AAO7QN457OQMRKPI6R24TJTAN47UXLEVOEFQVOS6M6C3Q, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.340523837 (297h0m59.93729486s) [noise=true , address=79.127.205.233:4166 , piece_id=SFGK5YCWHFRVXF6QNHYP2CYJCZLSTQ76OYRVJPWWXQTF2KFSFJKA, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.341468438 (297h0m59.938239447s) [noise=true , address=45.91.116.4:4961 , piece_id=ZMUGQECUVPVHDNKN75XPBET5M2HFXXK4WMTP4V5KEK63JVZCX5EA, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.341520025 (297h0m59.938291058s) [noise=true , address=167.17.72.71:4815 , piece_id=CXGTYNNBILN3TT7I6Z7UI75F2ZCS4QUTLS6KC4FJ5IHS3XOYWDJQ, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.342232992 (297h0m59.939004023s) [noise=true , address=145.79.227.36:4524 , piece_id=3TAV3P55ITTHBAZEUUUTOAHJRGQXJXXVGZJP6AVR2ANVNOR3S54Q, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.342917368 (297h0m59.939688398s) [noise=true , address=193.149.164.71:4297 , piece_id=4IJPXLPCX3LKFOBOSRHK2DTNRKH67K7FXJUKQ3JPRWHS47ZFEIDA, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.343244957 (297h0m59.940015982s) [noise=true , address=145.79.227.33:4892 , piece_id=DQ6ALC3VOAGFIJI3IKR6I7QFYM22C66QWAAXMH6NXI2FIQEZNQYA, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.344270079 (297h0m59.941041117s) [noise=true , address=145.79.227.21:4497 , piece_id=SP6EKJB7W4QXF7OIBVKEFWCPOZFL5IRIEDVAGS2M4QJSO34CUVRQ, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.345025057 (297h0m59.941796084s) [noise=true , address=77.90.31.17:4793 , piece_id=2U4IBUA2GGUGDBXQGIGQKUK4ZR3L7CDKC6MTGIKAUME4F7MFZRLA, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.345115802 (297h0m59.941886836s) [noise=true , address=167.17.72.69:4731 , piece_id=VOEVIKKFJMC2Y3TPQIDMYUP37NLFU7KPRH4HRRSMQLWUKTSLOH2Q, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.345690600 (297h0m59.942461634s) [noise=true , address=77.90.31.13:4648 , piece_id=BPMB5CFOYHFT3BS4ORR6JF6BNFC2AOAJZOSPMG67NAMLGGP5XVEQ, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.346311592 (297h0m59.943082623s) [noise=true , address=79.127.205.234:4202 , piece_id=YY4LYWX26O6RIOBONESNKTUZEFZCRL4UUX3NEBQ53AJV4IEDG5MQ, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.346791142 (297h0m59.943562166s) [noise=true , address=45.91.116.10:4110 , piece_id=UYLMDCWJEJOHGICA37KNH7G5AQXNIPRH5NRYLBUUMGDRVSLTEYHQ, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.347094999 (297h0m59.94386607s) [noise=true , address=145.79.227.17:4987 , piece_id=PT3TWVB65JZFEWVH4IUZEU2N53FQ57QBRWVLWAUFP4UDLY3LWNLQ, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.347781281 (297h0m59.944552305s) [noise=true , address=79.127.219.46:4919 , piece_id=KK6HRIVBMYLIHRHGPQH3AWMNSVHTTIBESHDMJZA72T446RX6QKRQ, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.348263123 (297h0m59.945034137s) [noise=true , address=193.149.164.68:4033 , piece_id=JZPQKPYGO3W3RZZBITQ73X2XEPIBBSLGRMWIVOM4Z3GTV5UFKRTQ, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.348915505 (297h0m59.945686536s) [noise=true , address=145.79.227.7:4312 , piece_id=2FAL6E3RMBTBHZBXAJJVH2N5VGF5VYI3KNZXGNWCMWGBBPAUO2NQ, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.349366298 (297h0m59.946137337s) [noise=true , address=193.149.164.67:4007 , piece_id=HURQGH67I5B6V4YUV3PRZL3KSKN3XTSK3UMCHZRNT6L3W3PXXHPQ, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.429834098 (297h1m0.026605126s) [upload=332666]: Next returned nil segment Mar 18 20:01:02.429853411 (297h1m0.026624442s) [upload=332666]: Waiting for error group managing segments... Mar 18 20:01:02.446867680 (297h1m0.043638704s) [noise=true , address=45.91.116.10:4110 , piece_id=UYLMDCWJEJOHGICA37KNH7G5AQXNIPRH5NRYLBUUMGDRVSLTEYHQ, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil> Mar 18 20:01:02.451665165 (297h1m0.048436196s) [noise=true , address=45.91.116.4:4961 , piece_id=ZMUGQECUVPVHDNKN75XPBET5M2HFXXK4WMTP4V5KEK63JVZCX5EA, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil> Mar 18 20:01:02.452538886 (297h1m0.049309913s) [noise=true , address=77.90.31.17:4793 , piece_id=2U4IBUA2GGUGDBXQGIGQKUK4ZR3L7CDKC6MTGIKAUME4F7MFZRLA, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil> Mar 18 20:01:02.455597040 (297h1m0.052368067s) [noise=true , address=45.91.116.8:4061 , piece_id=6XFVKT4AAO7QN457OQMRKPI6R24TJTAN47UXLEVOEFQVOS6M6C3Q, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil> Mar 18 20:01:02.457848739 (297h1m0.054619766s) [noise=true , address=77.90.31.13:4648 , piece_id=BPMB5CFOYHFT3BS4ORR6JF6BNFC2AOAJZOSPMG67NAMLGGP5XVEQ, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil> Mar 18 20:01:02.461479709 (297h1m0.058250746s) [noise=true , address=79.127.219.46:4919 , piece_id=KK6HRIVBMYLIHRHGPQH3AWMNSVHTTIBESHDMJZA72T446RX6QKRQ, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil> Mar 18 20:01:02.676859466 (297h1m0.27363049s) [noise=true , address=167.17.72.69:4731 , piece_id=VOEVIKKFJMC2Y3TPQIDMYUP37NLFU7KPRH4HRRSMQLWUKTSLOH2Q, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil> Mar 18 20:01:02.909860743 (297h1m0.506631766s) [noise=true , address=79.127.205.234:4202 , piece_id=YY4LYWX26O6RIOBONESNKTUZEFZCRL4UUX3NEBQ53AJV4IEDG5MQ, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil> Mar 18 20:01:02.930427742 (297h1m0.527198754s) [noise=true , address=193.149.164.71:4297 , piece_id=4IJPXLPCX3LKFOBOSRHK2DTNRKH67K7FXJUKQ3JPRWHS47ZFEIDA, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil> Mar 18 20:01:02.944696889 (297h1m0.541467914s) [noise=true , address=193.149.164.67:4007 , piece_id=HURQGH67I5B6V4YUV3PRZL3KSKN3XTSK3UMCHZRNT6L3W3PXXHPQ, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil> Mar 18 20:01:02.951319183 (297h1m0.548090212s) [noise=true , address=193.149.164.68:4033 , piece_id=JZPQKPYGO3W3RZZBITQ73X2XEPIBBSLGRMWIVOM4Z3GTV5UFKRTQ, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil> Mar 18 20:01:02.975619436 (297h1m0.572390467s) [noise=true , address=79.127.205.233:4166 , piece_id=SFGK5YCWHFRVXF6QNHYP2CYJCZLSTQ76OYRVJPWWXQTF2KFSFJKA, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil> Mar 18 20:01:03.819918528 (297h1m1.416689564s) [noise=true , address=167.17.72.71:4815 , piece_id=CXGTYNNBILN3TT7I6Z7UI75F2ZCS4QUTLS6KC4FJ5IHS3XOYWDJQ, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil> Mar 18 20:01:59.307873420 (297h1m56.904644452s) [noise=true , address=145.79.227.7:4312 , piece_id=2FAL6E3RMBTBHZBXAJJVH2N5VGF5VYI3KNZXGNWCMWGBBPAUO2NQ, seg_pos={0 0}, upload=332666]: Done uploading piece. err: ecclient: upload failed (node:12AVp3qn5uwLPqYaZEwzcgoGMJhNga1BUh6bK5k2bxjWgZLQiiC, address:145.79.227.7:4312): protocol: rpc: dial tcp 145.79.227.7:4312: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:01:59.308026092 (297h1m56.904797124s) [noise=true , address=145.79.227.17:4987 , piece_id=PT3TWVB65JZFEWVH4IUZEU2N53FQ57QBRWVLWAUFP4UDLY3LWNLQ, seg_pos={0 0}, upload=332666]: Done uploading piece. err: ecclient: upload failed (node:1XsPVNGYUhKnXkQRfVfjyBRzyvhShSdsrnzXtwe4Xtf5MaHMvH, address:145.79.227.17:4987): protocol: rpc: dial tcp 145.79.227.17:4987: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:01:59.308195864 (297h1m56.904966894s) [noise=true , address=145.79.227.36:4524 , piece_id=3TAV3P55ITTHBAZEUUUTOAHJRGQXJXXVGZJP6AVR2ANVNOR3S54Q, seg_pos={0 0}, upload=332666]: Done uploading piece. err: ecclient: upload failed (node:1xfbCzjSwFDRrR1gPggeN8tNCLbRRWDVzCJ6mqhZsbVSe8d4s4, address:145.79.227.36:4524): protocol: rpc: dial tcp 145.79.227.36:4524: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:01:59.308316037 (297h1m56.90508706s) [noise=true , address=145.79.227.33:4892 , piece_id=DQ6ALC3VOAGFIJI3IKR6I7QFYM22C66QWAAXMH6NXI2FIQEZNQYA, seg_pos={0 0}, upload=332666]: Done uploading piece. err: ecclient: upload failed (node:12RHwFxcUjPsn12uHEAH3fBDJKv1v1tM8mRJXoFrZrG5djdswF5, address:145.79.227.33:4892): protocol: rpc: dial tcp 145.79.227.33:4892: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:01:59.308404403 (297h1m56.905175433s) [noise=true , address=145.79.227.21:4497 , piece_id=SP6EKJB7W4QXF7OIBVKEFWCPOZFL5IRIEDVAGS2M4QJSO34CUVRQ, seg_pos={0 0}, upload=332666]: Done uploading piece. err: ecclient: upload failed (node:12ZBe4we7VeFdthqRxyy1V9uQrA4r8ouNX68xqsk5bD26dkf2Gd, address:145.79.227.21:4497): protocol: context deadline exceeded; context deadline exceeded Mar 18 20:01:59.308437457 (297h1m56.90520847s) [seg_pos={0 0}, upload=332666]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 18 20:01:59.338288467 (297h1m56.93505949s) [upload=332666]: Done uploading segments... Mar 18 20:01:59.338335497 (297h1m56.935106523s) [upload=332666]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 14.329709858s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 43 out of 60 is failed (threshold=3) [................XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.XXXXXX] duration: 30.026747664s trace: https://jaeger.dev.storj.io/trace/3da6329b1a74a55f FAIL (2026-03-18 20:07:28 UTC) operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 9) ======LOGS====== Mar 18 20:06:58.715947398 (297h6m56.312718432s) [upload=332778]: Starting upload Mar 18 20:06:58.716027398 (297h6m56.312798434s) [upload=332778]: Done starting upload Mar 18 20:06:58.716264342 (297h6m56.313035366s) [upload=332778]: Uploading segments... Mar 18 20:06:58.716526836 (297h6m56.313297865s) [upload=332778]: Got next segment. Inline: false Mar 18 20:06:58.716595354 (297h6m56.313366377s) [upload=332778]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 20:06:58.716604668 (297h6m56.313375688s) [upload=332778]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 18 20:07:02.118470802 (297h6m59.715241839s) [seg_pos={0 0}, upload=332778]: Begin upload segment... Mar 18 20:07:02.118558314 (297h6m59.715329349s) [seg_pos={0 0}, upload=332778]: Done begin upload segment. Mar 18 20:07:02.118630613 (297h6m59.715401646s) [noise=true , address=145.79.227.22:4509 , piece_id=WUNQ6CLYVMZ4SXAH4ITENGEQ47QSMHBPJIFBZ2KSCOILGYZ5TPUA, seg_pos={0 0}, upload=332778]: Uploading piece... Mar 18 20:07:02.118962399 (297h6m59.715733445s) [noise=true , address=77.90.31.18:4810 , piece_id=SJLQVL5NHWP3KRPYZSD7VTFDM6TGSYZGIXMF4TBRJ3TT4HQFSY5Q, seg_pos={0 0}, upload=332778]: Uploading piece... Mar 18 20:07:02.119740375 (297h6m59.716511409s) [noise=true , address=145.79.227.19:4082 , piece_id=EXK44YLLVLD2JSX5HWVCAB5HUFKI52HVDBTTAXJME7PCHGANPKZA, seg_pos={0 0}, upload=332778]: Uploading piece... Mar 18 20:07:02.119929226 (297h6m59.716700249s) [noise=true , address=145.79.227.9:4369 , piece_id=ZFOPTZ5Y7ZUMA4QITC76KRV2XDD74NQ4FXHKSEYBYGREPA2V3UPQ, seg_pos={0 0}, upload=332778]: Uploading piece... Mar 18 20:07:02.120326891 (297h6m59.717097925s) [noise=true , address=145.79.227.42:4125 , piece_id=5WBU454RPQGE3XD4CTAMZ2DZNAWQOZY6P2LX4YUYENWX3KRJUKJA, seg_pos={0 0}, upload=332778]: Uploading piece... Mar 18 20:07:02.120633870 (297h6m59.717404893s) [noise=true , address=145.79.227.34:4905 , piece_id=4IVTVQ6X3VSFHMTFYCKDAGZ72FBZRDFBHVGJLYLGYWZLG6QFEMRA, seg_pos={0 0}, upload=332778]: Uploading piece... Mar 18 20:07:02.120895233 (297h6m59.717666267s) [noise=true , address=145.79.227.24:4585 , piece_id=SID3EEAMTC53ZSQYKTER5XPHCJGBC3XJIJNO3L2PIZT34ZZG4BHA, seg_pos={0 0}, upload=332778]: Uploading piece... Mar 18 20:07:02.121459406 (297h6m59.718230442s) [noise=true , address=45.91.116.11:4118 , piece_id=KDHBVJNAQXJDSNGDUHQEEAYFZIJNKZYE3S4NLYTWK5PLGX5JGXGA, seg_pos={0 0}, upload=332778]: Uploading piece... Mar 18 20:07:02.121612143 (297h6m59.718383158s) [noise=true , address=45.91.116.5:4298 , piece_id=QIFBVGI4BU3AM2QPX5J7TPY23GMXHTCAJQEFG7MHSOCT4L6WIHLQ, seg_pos={0 0}, upload=332778]: Uploading piece... Mar 18 20:07:02.121963977 (297h6m59.718735013s) [noise=true , address=77.90.31.14:4665 , piece_id=5ZFRAWV7VCAXFQRUPBXT4FXVACUZNMJWO52MZU32TYNXBLZ2ZXEA, seg_pos={0 0}, upload=332778]: Uploading piece... Mar 18 20:07:02.122501166 (297h6m59.719272192s) [noise=true , address=77.90.31.13:4629 , piece_id=O5QMXWEKJTDVUKQ4I6NJLSRWIG3KPYFW5JJJUIXI6UJQPRLF7FTA, seg_pos={0 0}, upload=332778]: Uploading piece... Mar 18 20:07:02.122524377 (297h6m59.719295411s) [noise=true , address=145.79.227.36:4520 , piece_id=DB6ZRXJWAADTPXMEZVW7J36OWYFFIJMRJVVJVDGA6CPYSPZIFWTA, seg_pos={0 0}, upload=332778]: Uploading piece... Mar 18 20:07:02.123108388 (297h6m59.719879423s) [noise=true , address=79.127.219.46:4891 , piece_id=LUL3WAM3XJR3HTDBUJHCS2IY2PZY3MQXVVM6IEJPRRYEC7IKHTWA, seg_pos={0 0}, upload=332778]: Uploading piece... Mar 18 20:07:02.123402779 (297h6m59.720173806s) [noise=true , address=77.90.31.16:4736 , piece_id=GMOVGQUX3JOSJEO2HOAI52Z2IT6ATSDMZLDWNDKNFGSJHOS5QYBQ, seg_pos={0 0}, upload=332778]: Uploading piece... Mar 18 20:07:02.124558739 (297h6m59.721329776s) [noise=true , address=45.91.116.8:4079 , piece_id=DG75AURQXLXGGOIMZTBQO6AXDM64G76QDCHWRAG3DY733TCB2AAQ, seg_pos={0 0}, upload=332778]: Uploading piece... Mar 18 20:07:02.125596207 (297h6m59.722367238s) [noise=true , address=45.91.116.13:4188 , piece_id=BTXACUXJ6UA6LDAG6SBQYYCGCQHAV7JZUFNHV2YXFHIS3ALMDUXQ, seg_pos={0 0}, upload=332778]: Uploading piece... Mar 18 20:07:02.218924261 (297h6m59.815695275s) [upload=332778]: Next returned nil segment Mar 18 20:07:02.218938702 (297h6m59.815709725s) [upload=332778]: Waiting for error group managing segments... Mar 18 20:07:02.223558869 (297h6m59.820329931s) [noise=true , address=45.91.116.13:4188 , piece_id=BTXACUXJ6UA6LDAG6SBQYYCGCQHAV7JZUFNHV2YXFHIS3ALMDUXQ, seg_pos={0 0}, upload=332778]: Done uploading piece. err: <nil> Mar 18 20:07:02.227338598 (297h6m59.824109622s) [noise=true , address=79.127.219.46:4891 , piece_id=LUL3WAM3XJR3HTDBUJHCS2IY2PZY3MQXVVM6IEJPRRYEC7IKHTWA, seg_pos={0 0}, upload=332778]: Done uploading piece. err: <nil> Mar 18 20:07:02.227526619 (297h6m59.824297642s) [noise=true , address=45.91.116.11:4118 , piece_id=KDHBVJNAQXJDSNGDUHQEEAYFZIJNKZYE3S4NLYTWK5PLGX5JGXGA, seg_pos={0 0}, upload=332778]: Done uploading piece. err: <nil> Mar 18 20:07:02.228386926 (297h6m59.82515796s) [noise=true , address=77.90.31.14:4665 , piece_id=5ZFRAWV7VCAXFQRUPBXT4FXVACUZNMJWO52MZU32TYNXBLZ2ZXEA, seg_pos={0 0}, upload=332778]: Done uploading piece. err: <nil> Mar 18 20:07:02.233526771 (297h6m59.830297806s) [noise=true , address=45.91.116.8:4079 , piece_id=DG75AURQXLXGGOIMZTBQO6AXDM64G76QDCHWRAG3DY733TCB2AAQ, seg_pos={0 0}, upload=332778]: Done uploading piece. err: <nil> Mar 18 20:07:02.240960557 (297h6m59.837731587s) [noise=true , address=45.91.116.5:4298 , piece_id=QIFBVGI4BU3AM2QPX5J7TPY23GMXHTCAJQEFG7MHSOCT4L6WIHLQ, seg_pos={0 0}, upload=332778]: Done uploading piece. err: <nil> Mar 18 20:07:02.241684427 (297h6m59.838455454s) [noise=true , address=77.90.31.13:4629 , piece_id=O5QMXWEKJTDVUKQ4I6NJLSRWIG3KPYFW5JJJUIXI6UJQPRLF7FTA, seg_pos={0 0}, upload=332778]: Done uploading piece. err: <nil> Mar 18 20:07:02.242068222 (297h6m59.838839246s) [noise=true , address=77.90.31.16:4736 , piece_id=GMOVGQUX3JOSJEO2HOAI52Z2IT6ATSDMZLDWNDKNFGSJHOS5QYBQ, seg_pos={0 0}, upload=332778]: Done uploading piece. err: <nil> Mar 18 20:07:02.245716670 (297h6m59.842487683s) [noise=true , address=77.90.31.18:4810 , piece_id=SJLQVL5NHWP3KRPYZSD7VTFDM6TGSYZGIXMF4TBRJ3TT4HQFSY5Q, seg_pos={0 0}, upload=332778]: Done uploading piece. err: <nil> Mar 18 20:07:28.683135142 (297h7m26.279906345s) [noise=true , address=145.79.227.42:4125 , piece_id=5WBU454RPQGE3XD4CTAMZ2DZNAWQOZY6P2LX4YUYENWX3KRJUKJA, seg_pos={0 0}, upload=332778]: Done uploading piece. err: ecclient: upload failed (node:1MrdHwiDVe8m6wcaQJrYetza9qckLruJXiU9UYbMxGCSWWMV1S, address:145.79.227.42:4125): protocol: rpc: dial tcp 145.79.227.42:4125: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:07:28.683603562 (297h7m26.280374594s) [noise=true , address=145.79.227.22:4509 , piece_id=WUNQ6CLYVMZ4SXAH4ITENGEQ47QSMHBPJIFBZ2KSCOILGYZ5TPUA, seg_pos={0 0}, upload=332778]: Done uploading piece. err: ecclient: upload failed (node:1229E58YirF37sZU6hfZGrDNFSxQDYsWSo4x3mG9BoYsNsLNtjy, address:145.79.227.22:4509): protocol: rpc: dial tcp 145.79.227.22:4509: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:07:28.683656022 (297h7m26.280427056s) [noise=true , address=145.79.227.34:4905 , piece_id=4IVTVQ6X3VSFHMTFYCKDAGZ72FBZRDFBHVGJLYLGYWZLG6QFEMRA, seg_pos={0 0}, upload=332778]: Done uploading piece. err: ecclient: upload failed (node:12ffKESNSVpXQzbwJinP6vvgE6zoF3RFz2NRUuuJHvd7bvk1gc9, address:145.79.227.34:4905): protocol: rpc: dial tcp 145.79.227.34:4905: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:07:28.683906298 (297h7m26.280677333s) [noise=true , address=145.79.227.36:4520 , piece_id=DB6ZRXJWAADTPXMEZVW7J36OWYFFIJMRJVVJVDGA6CPYSPZIFWTA, seg_pos={0 0}, upload=332778]: Done uploading piece. err: ecclient: upload failed (node:12eNRsboM5zFiyBVPLuQfgb5QSAc5bZFizK9KJHudNd9nstVGyy, address:145.79.227.36:4520): protocol: rpc: dial tcp 145.79.227.36:4520: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:07:28.684046796 (297h7m26.280817825s) [noise=true , address=145.79.227.24:4585 , piece_id=SID3EEAMTC53ZSQYKTER5XPHCJGBC3XJIJNO3L2PIZT34ZZG4BHA, seg_pos={0 0}, upload=332778]: Done uploading piece. err: ecclient: upload failed (node:1TyK6onzVds1kucaD6Y4aPssPDM5FzzQ91dTrez5B7ttfwvXm, address:145.79.227.24:4585): protocol: context deadline exceeded; context deadline exceeded Mar 18 20:07:28.684231613 (297h7m26.281002648s) [noise=true , address=145.79.227.19:4082 , piece_id=EXK44YLLVLD2JSX5HWVCAB5HUFKI52HVDBTTAXJME7PCHGANPKZA, seg_pos={0 0}, upload=332778]: Done uploading piece. err: ecclient: upload failed (node:12DKsNCtqxLvpJwLszdLs74V3eb8NH2VAV9V3bngzUczawmNoLK, address:145.79.227.19:4082): protocol: context deadline exceeded; context deadline exceeded Mar 18 20:07:28.684445387 (297h7m26.281216417s) [noise=true , address=145.79.227.9:4369 , piece_id=ZFOPTZ5Y7ZUMA4QITC76KRV2XDD74NQ4FXHKSEYBYGREPA2V3UPQ, seg_pos={0 0}, upload=332778]: Done uploading piece. err: ecclient: upload failed (node:1nwygPHyRJEZkjCoSZJTVhXDKLHc7AonVMzKMht6ppwZwhcGCT, address:145.79.227.9:4369): protocol: context deadline exceeded; context deadline exceeded Mar 18 20:07:28.684479347 (297h7m26.281250361s) [seg_pos={0 0}, upload=332778]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Mar 18 20:07:28.707270187 (297h7m26.304041202s) [upload=332778]: Done uploading segments... Mar 18 20:07:28.707312813 (297h7m26.304083843s) [upload=332778]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Duration: 30.026774221s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 8 out of 60 is failed (threshold=3) [.....................XXX..........X...X.........X....X....X.] duration: 17.503733876s trace: https://jaeger.dev.storj.io/trace/75a9000feaeb0b3d FAIL (2026-03-18 20:07:22 UTC) operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== ar 18 20:06:04.693955863 (297h6m2.290726886s) [upload=332765]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 20:06:04.693961555 (297h6m2.290732583s) [upload=332765]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 18 20:06:07.394748117 (297h6m4.99151915s) [seg_pos={0 0}, upload=332765]: Begin upload segment... Mar 18 20:06:07.394875884 (297h6m4.991646919s) [seg_pos={0 0}, upload=332765]: Done begin upload segment. Mar 18 20:06:07.394887373 (297h6m4.991658385s) [noise=true , address=45.91.116.9:4276 , piece_id=J356KNX74ADIKN7PPKCRNGACHPVBJYZA3O5OAGVYSURLB4T6KSZA, seg_pos={0 0}, upload=332765]: Uploading piece... Mar 18 20:06:07.394990884 (297h6m4.991761926s) [noise=true , address=145.79.227.2:4129 , piece_id=2Y7VDPW5SS3D4J45EAMSLOEQAHNKYCZJEPP2FTP7GKVCYKMDQTLA, seg_pos={0 0}, upload=332765]: Uploading piece... Mar 18 20:06:07.396053654 (297h6m4.992824681s) [noise=true , address=45.91.116.11:4150 , piece_id=L2ZK2GKN4VY7W6SJ25MR65RL52UH3M66FGQLSLXI2N6LEZXJQVMA, seg_pos={0 0}, upload=332765]: Uploading piece... Mar 18 20:06:07.396081719 (297h6m4.992852749s) [noise=true , address=45.91.116.12:4246 , piece_id=T5BGURHRLDTEZZ4TORP6F5WNBR2TY3ZDWPWO4PGSXQ575BWTUMVA, seg_pos={0 0}, upload=332765]: Uploading piece... Mar 18 20:06:07.396842874 (297h6m4.993613899s) [noise=true , address=145.79.227.33:4875 , piece_id=5LQOGDTQD5CCBMKIZCSLBVQUHQWU6W5ELX6XKLFSFQLU3NVDLKPQ, seg_pos={0 0}, upload=332765]: Uploading piece... Mar 18 20:06:07.397161075 (297h6m4.993932121s) [noise=true , address=77.90.31.12:4615 , piece_id=HLRHRANL5IHJOB5QHLW2OIN5ZP56RKX3INJPWPU5YYHBHRYVJM6A, seg_pos={0 0}, upload=332765]: Uploading piece... Mar 18 20:06:07.397369139 (297h6m4.994140166s) [noise=true , address=77.90.31.11:4567 , piece_id=UN6S6RWYUVLUA5NPFFJ5TASQDZ26OU2IIWP3ZNFUFVJ3WJAA2PSQ, seg_pos={0 0}, upload=332765]: Uploading piece... Mar 18 20:06:07.397732013 (297h6m4.994503037s) [noise=true , address=45.91.116.14:4206 , piece_id=IDCMPAW3BEN5OQ7HUX3QDGFGKN76DM4E6CDN3M45SJQ4VU2OQGCQ, seg_pos={0 0}, upload=332765]: Uploading piece... Mar 18 20:06:07.397757704 (297h6m4.994528729s) [noise=true , address=145.79.227.3:4177 , piece_id=A7HSMKKEMCEJZENJ3CM7N3GZI3NX4BA2WIQZ27MTRONPXBZKZWHA, seg_pos={0 0}, upload=332765]: Uploading piece... Mar 18 20:06:07.398274955 (297h6m4.995045981s) [noise=true , address=193.149.164.71:4307 , piece_id=TZ5I3SSLCLL2AOKJTEFTMEE7DLXHXDKCJJLNP4WHDSGRCJLLBM6A, seg_pos={0 0}, upload=332765]: Uploading piece... Mar 18 20:06:07.398328768 (297h6m4.995099802s) [noise=true , address=79.127.205.233:4173 , piece_id=I64CQ5XLN6DUP2E465ZQLGOVE5NEIHLNWT33QB7WKJF5Q6QZRPLQ, seg_pos={0 0}, upload=332765]: Uploading piece... Mar 18 20:06:07.398815526 (297h6m4.995586656s) [noise=true , address=145.79.227.51:4424 , piece_id=5XCD27Q3GOCFGSHDUNX7QIGUQ6URBU2UREPVX5ZF7CFICF4X4UNQ, seg_pos={0 0}, upload=332765]: Uploading piece... Mar 18 20:06:07.399136293 (297h6m4.995907322s) [noise=true , address=64.49.13.137:4499 , piece_id=KTFNLPG67BVZMYZMJJO4BDRTASHDANE2FBYO3YUV2E4XBCU5NLCA, seg_pos={0 0}, upload=332765]: Uploading piece... Mar 18 20:06:07.399360723 (297h6m4.996131762s) [noise=true , address=45.91.116.4:4963 , piece_id=ZGY2X7JG2MPOJWKMFWP3VATYO3QTEOXPQ4KYZSOW4QOPST6OV2VQ, seg_pos={0 0}, upload=332765]: Uploading piece... Mar 18 20:06:07.399800937 (297h6m4.996571962s) [noise=true , address=145.79.227.24:4592 , piece_id=NGOLKV5BQKREBAYQHBSXHN2UKBHB52WPVBBO7NIFCCYTBF2MXPBA, seg_pos={0 0}, upload=332765]: Uploading piece... Mar 18 20:06:07.399957240 (297h6m4.996728268s) [noise=true , address=193.149.164.66:4993 , piece_id=CRRZAIXMIV673K5JLX743KZ3HFL5MBGUE5ZE4T4Q2WPCSUQ32JQA, seg_pos={0 0}, upload=332765]: Uploading piece... Mar 18 20:06:07.400205078 (297h6m4.996976103s) [noise=true , address=167.17.72.68:4668 , piece_id=KUTKCCK6Y54CMAIIWGINQERXEV6CGVPCOYEJAQUNPBWTHJ755KJQ, seg_pos={0 0}, upload=332765]: Uploading piece... Mar 18 20:06:07.400506584 (297h6m4.997277612s) [noise=true , address=145.79.227.49:4360 , piece_id=OP5BBV7CUZBSMBO6HTTHQCVXDQGS6XO5LGAZ4I2SKUSPGUAKBXKA, seg_pos={0 0}, upload=332765]: Uploading piece... Mar 18 20:06:07.471265449 (297h6m5.068036462s) [upload=332765]: Next returned nil segment Mar 18 20:06:07.471278058 (297h6m5.068049082s) [upload=332765]: Waiting for error group managing segments... Mar 18 20:06:07.475837847 (297h6m5.07260886s) [noise=true , address=77.90.31.11:4567 , piece_id=UN6S6RWYUVLUA5NPFFJ5TASQDZ26OU2IIWP3ZNFUFVJ3WJAA2PSQ, seg_pos={0 0}, upload=332765]: Done uploading piece. err: <nil> Mar 18 20:06:07.476715720 (297h6m5.073486747s) [noise=true , address=45.91.116.4:4963 , piece_id=ZGY2X7JG2MPOJWKMFWP3VATYO3QTEOXPQ4KYZSOW4QOPST6OV2VQ, seg_pos={0 0}, upload=332765]: Done uploading piece. err: <nil> Mar 18 20:06:07.488555609 (297h6m5.085326636s) [noise=true , address=45.91.116.12:4246 , piece_id=T5BGURHRLDTEZZ4TORP6F5WNBR2TY3ZDWPWO4PGSXQ575BWTUMVA, seg_pos={0 0}, upload=332765]: Done uploading piece. err: <nil> Mar 18 20:06:07.489803434 (297h6m5.086574468s) [noise=true , address=45.91.116.14:4206 , piece_id=IDCMPAW3BEN5OQ7HUX3QDGFGKN76DM4E6CDN3M45SJQ4VU2OQGCQ, seg_pos={0 0}, upload=332765]: Done uploading piece. err: <nil> Mar 18 20:06:07.490598252 (297h6m5.087369276s) [noise=true , address=45.91.116.11:4150 , piece_id=L2ZK2GKN4VY7W6SJ25MR65RL52UH3M66FGQLSLXI2N6LEZXJQVMA, seg_pos={0 0}, upload=332765]: Done uploading piece. err: <nil> Mar 18 20:06:07.492135441 (297h6m5.088906454s) [noise=true , address=45.91.116.9:4276 , piece_id=J356KNX74ADIKN7PPKCRNGACHPVBJYZA3O5OAGVYSURLB4T6KSZA, seg_pos={0 0}, upload=332765]: Done uploading piece. err: <nil> Mar 18 20:06:07.508501889 (297h6m5.105272902s) [noise=true , address=77.90.31.12:4615 , piece_id=HLRHRANL5IHJOB5QHLW2OIN5ZP56RKX3INJPWPU5YYHBHRYVJM6A, seg_pos={0 0}, upload=332765]: Done uploading piece. err: <nil> Mar 18 20:06:07.770562428 (297h6m5.367333451s) [noise=true , address=167.17.72.68:4668 , piece_id=KUTKCCK6Y54CMAIIWGINQERXEV6CGVPCOYEJAQUNPBWTHJ755KJQ, seg_pos={0 0}, upload=332765]: Done uploading piece. err: <nil> Mar 18 20:06:08.006027504 (297h6m5.602798531s) [noise=true , address=193.149.164.66:4993 , piece_id=CRRZAIXMIV673K5JLX743KZ3HFL5MBGUE5ZE4T4Q2WPCSUQ32JQA, seg_pos={0 0}, upload=332765]: Done uploading piece. err: <nil> Mar 18 20:06:08.016743696 (297h6m5.613514734s) [noise=true , address=193.149.164.71:4307 , piece_id=TZ5I3SSLCLL2AOKJTEFTMEE7DLXHXDKCJJLNP4WHDSGRCJLLBM6A, seg_pos={0 0}, upload=332765]: Done uploading piece. err: <nil> Mar 18 20:06:08.043257501 (297h6m5.640028528s) [noise=true , address=64.49.13.137:4499 , piece_id=KTFNLPG67BVZMYZMJJO4BDRTASHDANE2FBYO3YUV2E4XBCU5NLCA, seg_pos={0 0}, upload=332765]: Done uploading piece. err: <nil> Mar 18 20:06:08.323806978 (297h6m5.920577991s) [noise=true , address=79.127.205.233:4173 , piece_id=I64CQ5XLN6DUP2E465ZQLGOVE5NEIHLNWT33QB7WKJF5Q6QZRPLQ, seg_pos={0 0}, upload=332765]: Done uploading piece. err: <nil> Mar 18 20:07:04.645185715 (297h7m2.241956747s) [noise=true , address=145.79.227.3:4177 , piece_id=A7HSMKKEMCEJZENJ3CM7N3GZI3NX4BA2WIQZ27MTRONPXBZKZWHA, seg_pos={0 0}, upload=332765]: Done uploading piece. err: ecclient: upload failed (node:12rFuJ2JyBGWtaASJNnTdHoeKUY1mCfDYfJ74BkxK64U9it8FKF, address:145.79.227.3:4177): protocol: rpc: dial tcp 145.79.227.3:4177: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:07:04.645320026 (297h7m2.24209106s) [noise=true , address=145.79.227.24:4592 , piece_id=NGOLKV5BQKREBAYQHBSXHN2UKBHB52WPVBBO7NIFCCYTBF2MXPBA, seg_pos={0 0}, upload=332765]: Done uploading piece. err: ecclient: upload failed (node:1fB7rqiDhMNQ4j9smkctbKdnGCXXdfFPWTgTxRrwQVZ9fKEFeZ, address:145.79.227.24:4592): protocol: rpc: dial tcp 145.79.227.24:4592: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:07:04.645426518 (297h7m2.242197551s) [noise=true , address=145.79.227.2:4129 , piece_id=2Y7VDPW5SS3D4J45EAMSLOEQAHNKYCZJEPP2FTP7GKVCYKMDQTLA, seg_pos={0 0}, upload=332765]: Done uploading piece. err: ecclient: upload failed (node:12GJof7GVfEN9u2yVyWf6j24hnaTdD3ygoPRoicmSqSfcNJFzs6, address:145.79.227.2:4129): protocol: rpc: dial tcp 145.79.227.2:4129: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:07:04.645453468 (297h7m2.242224501s) [noise=true , address=145.79.227.49:4360 , piece_id=OP5BBV7CUZBSMBO6HTTHQCVXDQGS6XO5LGAZ4I2SKUSPGUAKBXKA, seg_pos={0 0}, upload=332765]: Done uploading piece. err: ecclient: upload failed (node:12T24rtYBJ47E6Ef1ZZKMWcpBrWBpCg5QV2ZHniCZZkzJF2yUNk, address:145.79.227.49:4360): protocol: rpc: dial tcp 145.79.227.49:4360: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:07:04.645552758 (297h7m2.242323785s) [noise=true , address=145.79.227.33:4875 , piece_id=5LQOGDTQD5CCBMKIZCSLBVQUHQWU6W5ELX6XKLFSFQLU3NVDLKPQ, seg_pos={0 0}, upload=332765]: Done uploading piece. err: ecclient: upload failed (node:1ijEJgYouyMkUqpLWWz7eH2NGvvs9XpN2ujDjupPykDe5eWcB1, address:145.79.227.33:4875): protocol: rpc: dial tcp 145.79.227.33:4875: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:07:04.645682576 (297h7m2.242453607s) [noise=true , address=145.79.227.51:4424 , piece_id=5XCD27Q3GOCFGSHDUNX7QIGUQ6URBU2UREPVX5ZF7CFICF4X4UNQ, seg_pos={0 0}, upload=332765]: Done uploading piece. err: ecclient: upload failed (node:12LwZixdSPnCm3Wm27PBY79jj52sjkNh2VuQexagieTGBp4Ps2k, address:145.79.227.51:4424): protocol: rpc: dial tcp 145.79.227.51:4424: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:07:04.645716594 (297h7m2.24248761s) [seg_pos={0 0}, upload=332765]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Mar 18 20:07:04.666238325 (297h7m2.263009349s) [upload=332765]: Done uploading segments... Mar 18 20:07:04.666276723 (297h7m2.263047746s) [upload=332765]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 17.503830234s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 50 out of 60 is failed (threshold=6) [......XXXXXXXXX.XXX...XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 5.000810524s trace: https://jaeger.dev.storj.io/trace/ee65acec7861626 FAIL (2026-03-18 20:07:14 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.000891765s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 8 out of 60 is failed (threshold=3) [...................XXXXXX.XX................................] duration: 19.616230383s trace: https://jaeger.dev.storj.io/trace/4b4d10a8051ad01b FAIL (2026-03-18 20:07:09 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 19.616262667s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 8 out of 60 is failed (threshold=3) [......................XXX..........X...X.........X....X....X] duration: 1m0.024272327s trace: https://jaeger.dev.storj.io/trace/75a9000feaeb0b3d FAIL (2026-03-18 20:07:04 UTC) operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== ar 18 20:06:04.693955863 (297h6m2.290726886s) [upload=332765]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 20:06:04.693961555 (297h6m2.290732583s) [upload=332765]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 18 20:06:07.394748117 (297h6m4.99151915s) [seg_pos={0 0}, upload=332765]: Begin upload segment... Mar 18 20:06:07.394875884 (297h6m4.991646919s) [seg_pos={0 0}, upload=332765]: Done begin upload segment. Mar 18 20:06:07.394887373 (297h6m4.991658385s) [noise=true , address=45.91.116.9:4276 , piece_id=J356KNX74ADIKN7PPKCRNGACHPVBJYZA3O5OAGVYSURLB4T6KSZA, seg_pos={0 0}, upload=332765]: Uploading piece... Mar 18 20:06:07.394990884 (297h6m4.991761926s) [noise=true , address=145.79.227.2:4129 , piece_id=2Y7VDPW5SS3D4J45EAMSLOEQAHNKYCZJEPP2FTP7GKVCYKMDQTLA, seg_pos={0 0}, upload=332765]: Uploading piece... Mar 18 20:06:07.396053654 (297h6m4.992824681s) [noise=true , address=45.91.116.11:4150 , piece_id=L2ZK2GKN4VY7W6SJ25MR65RL52UH3M66FGQLSLXI2N6LEZXJQVMA, seg_pos={0 0}, upload=332765]: Uploading piece... Mar 18 20:06:07.396081719 (297h6m4.992852749s) [noise=true , address=45.91.116.12:4246 , piece_id=T5BGURHRLDTEZZ4TORP6F5WNBR2TY3ZDWPWO4PGSXQ575BWTUMVA, seg_pos={0 0}, upload=332765]: Uploading piece... Mar 18 20:06:07.396842874 (297h6m4.993613899s) [noise=true , address=145.79.227.33:4875 , piece_id=5LQOGDTQD5CCBMKIZCSLBVQUHQWU6W5ELX6XKLFSFQLU3NVDLKPQ, seg_pos={0 0}, upload=332765]: Uploading piece... Mar 18 20:06:07.397161075 (297h6m4.993932121s) [noise=true , address=77.90.31.12:4615 , piece_id=HLRHRANL5IHJOB5QHLW2OIN5ZP56RKX3INJPWPU5YYHBHRYVJM6A, seg_pos={0 0}, upload=332765]: Uploading piece... Mar 18 20:06:07.397369139 (297h6m4.994140166s) [noise=true , address=77.90.31.11:4567 , piece_id=UN6S6RWYUVLUA5NPFFJ5TASQDZ26OU2IIWP3ZNFUFVJ3WJAA2PSQ, seg_pos={0 0}, upload=332765]: Uploading piece... Mar 18 20:06:07.397732013 (297h6m4.994503037s) [noise=true , address=45.91.116.14:4206 , piece_id=IDCMPAW3BEN5OQ7HUX3QDGFGKN76DM4E6CDN3M45SJQ4VU2OQGCQ, seg_pos={0 0}, upload=332765]: Uploading piece... Mar 18 20:06:07.397757704 (297h6m4.994528729s) [noise=true , address=145.79.227.3:4177 , piece_id=A7HSMKKEMCEJZENJ3CM7N3GZI3NX4BA2WIQZ27MTRONPXBZKZWHA, seg_pos={0 0}, upload=332765]: Uploading piece... Mar 18 20:06:07.398274955 (297h6m4.995045981s) [noise=true , address=193.149.164.71:4307 , piece_id=TZ5I3SSLCLL2AOKJTEFTMEE7DLXHXDKCJJLNP4WHDSGRCJLLBM6A, seg_pos={0 0}, upload=332765]: Uploading piece... Mar 18 20:06:07.398328768 (297h6m4.995099802s) [noise=true , address=79.127.205.233:4173 , piece_id=I64CQ5XLN6DUP2E465ZQLGOVE5NEIHLNWT33QB7WKJF5Q6QZRPLQ, seg_pos={0 0}, upload=332765]: Uploading piece... Mar 18 20:06:07.398815526 (297h6m4.995586656s) [noise=true , address=145.79.227.51:4424 , piece_id=5XCD27Q3GOCFGSHDUNX7QIGUQ6URBU2UREPVX5ZF7CFICF4X4UNQ, seg_pos={0 0}, upload=332765]: Uploading piece... Mar 18 20:06:07.399136293 (297h6m4.995907322s) [noise=true , address=64.49.13.137:4499 , piece_id=KTFNLPG67BVZMYZMJJO4BDRTASHDANE2FBYO3YUV2E4XBCU5NLCA, seg_pos={0 0}, upload=332765]: Uploading piece... Mar 18 20:06:07.399360723 (297h6m4.996131762s) [noise=true , address=45.91.116.4:4963 , piece_id=ZGY2X7JG2MPOJWKMFWP3VATYO3QTEOXPQ4KYZSOW4QOPST6OV2VQ, seg_pos={0 0}, upload=332765]: Uploading piece... Mar 18 20:06:07.399800937 (297h6m4.996571962s) [noise=true , address=145.79.227.24:4592 , piece_id=NGOLKV5BQKREBAYQHBSXHN2UKBHB52WPVBBO7NIFCCYTBF2MXPBA, seg_pos={0 0}, upload=332765]: Uploading piece... Mar 18 20:06:07.399957240 (297h6m4.996728268s) [noise=true , address=193.149.164.66:4993 , piece_id=CRRZAIXMIV673K5JLX743KZ3HFL5MBGUE5ZE4T4Q2WPCSUQ32JQA, seg_pos={0 0}, upload=332765]: Uploading piece... Mar 18 20:06:07.400205078 (297h6m4.996976103s) [noise=true , address=167.17.72.68:4668 , piece_id=KUTKCCK6Y54CMAIIWGINQERXEV6CGVPCOYEJAQUNPBWTHJ755KJQ, seg_pos={0 0}, upload=332765]: Uploading piece... Mar 18 20:06:07.400506584 (297h6m4.997277612s) [noise=true , address=145.79.227.49:4360 , piece_id=OP5BBV7CUZBSMBO6HTTHQCVXDQGS6XO5LGAZ4I2SKUSPGUAKBXKA, seg_pos={0 0}, upload=332765]: Uploading piece... Mar 18 20:06:07.471265449 (297h6m5.068036462s) [upload=332765]: Next returned nil segment Mar 18 20:06:07.471278058 (297h6m5.068049082s) [upload=332765]: Waiting for error group managing segments... Mar 18 20:06:07.475837847 (297h6m5.07260886s) [noise=true , address=77.90.31.11:4567 , piece_id=UN6S6RWYUVLUA5NPFFJ5TASQDZ26OU2IIWP3ZNFUFVJ3WJAA2PSQ, seg_pos={0 0}, upload=332765]: Done uploading piece. err: <nil> Mar 18 20:06:07.476715720 (297h6m5.073486747s) [noise=true , address=45.91.116.4:4963 , piece_id=ZGY2X7JG2MPOJWKMFWP3VATYO3QTEOXPQ4KYZSOW4QOPST6OV2VQ, seg_pos={0 0}, upload=332765]: Done uploading piece. err: <nil> Mar 18 20:06:07.488555609 (297h6m5.085326636s) [noise=true , address=45.91.116.12:4246 , piece_id=T5BGURHRLDTEZZ4TORP6F5WNBR2TY3ZDWPWO4PGSXQ575BWTUMVA, seg_pos={0 0}, upload=332765]: Done uploading piece. err: <nil> Mar 18 20:06:07.489803434 (297h6m5.086574468s) [noise=true , address=45.91.116.14:4206 , piece_id=IDCMPAW3BEN5OQ7HUX3QDGFGKN76DM4E6CDN3M45SJQ4VU2OQGCQ, seg_pos={0 0}, upload=332765]: Done uploading piece. err: <nil> Mar 18 20:06:07.490598252 (297h6m5.087369276s) [noise=true , address=45.91.116.11:4150 , piece_id=L2ZK2GKN4VY7W6SJ25MR65RL52UH3M66FGQLSLXI2N6LEZXJQVMA, seg_pos={0 0}, upload=332765]: Done uploading piece. err: <nil> Mar 18 20:06:07.492135441 (297h6m5.088906454s) [noise=true , address=45.91.116.9:4276 , piece_id=J356KNX74ADIKN7PPKCRNGACHPVBJYZA3O5OAGVYSURLB4T6KSZA, seg_pos={0 0}, upload=332765]: Done uploading piece. err: <nil> Mar 18 20:06:07.508501889 (297h6m5.105272902s) [noise=true , address=77.90.31.12:4615 , piece_id=HLRHRANL5IHJOB5QHLW2OIN5ZP56RKX3INJPWPU5YYHBHRYVJM6A, seg_pos={0 0}, upload=332765]: Done uploading piece. err: <nil> Mar 18 20:06:07.770562428 (297h6m5.367333451s) [noise=true , address=167.17.72.68:4668 , piece_id=KUTKCCK6Y54CMAIIWGINQERXEV6CGVPCOYEJAQUNPBWTHJ755KJQ, seg_pos={0 0}, upload=332765]: Done uploading piece. err: <nil> Mar 18 20:06:08.006027504 (297h6m5.602798531s) [noise=true , address=193.149.164.66:4993 , piece_id=CRRZAIXMIV673K5JLX743KZ3HFL5MBGUE5ZE4T4Q2WPCSUQ32JQA, seg_pos={0 0}, upload=332765]: Done uploading piece. err: <nil> Mar 18 20:06:08.016743696 (297h6m5.613514734s) [noise=true , address=193.149.164.71:4307 , piece_id=TZ5I3SSLCLL2AOKJTEFTMEE7DLXHXDKCJJLNP4WHDSGRCJLLBM6A, seg_pos={0 0}, upload=332765]: Done uploading piece. err: <nil> Mar 18 20:06:08.043257501 (297h6m5.640028528s) [noise=true , address=64.49.13.137:4499 , piece_id=KTFNLPG67BVZMYZMJJO4BDRTASHDANE2FBYO3YUV2E4XBCU5NLCA, seg_pos={0 0}, upload=332765]: Done uploading piece. err: <nil> Mar 18 20:06:08.323806978 (297h6m5.920577991s) [noise=true , address=79.127.205.233:4173 , piece_id=I64CQ5XLN6DUP2E465ZQLGOVE5NEIHLNWT33QB7WKJF5Q6QZRPLQ, seg_pos={0 0}, upload=332765]: Done uploading piece. err: <nil> Mar 18 20:07:04.645185715 (297h7m2.241956747s) [noise=true , address=145.79.227.3:4177 , piece_id=A7HSMKKEMCEJZENJ3CM7N3GZI3NX4BA2WIQZ27MTRONPXBZKZWHA, seg_pos={0 0}, upload=332765]: Done uploading piece. err: ecclient: upload failed (node:12rFuJ2JyBGWtaASJNnTdHoeKUY1mCfDYfJ74BkxK64U9it8FKF, address:145.79.227.3:4177): protocol: rpc: dial tcp 145.79.227.3:4177: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:07:04.645320026 (297h7m2.24209106s) [noise=true , address=145.79.227.24:4592 , piece_id=NGOLKV5BQKREBAYQHBSXHN2UKBHB52WPVBBO7NIFCCYTBF2MXPBA, seg_pos={0 0}, upload=332765]: Done uploading piece. err: ecclient: upload failed (node:1fB7rqiDhMNQ4j9smkctbKdnGCXXdfFPWTgTxRrwQVZ9fKEFeZ, address:145.79.227.24:4592): protocol: rpc: dial tcp 145.79.227.24:4592: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:07:04.645426518 (297h7m2.242197551s) [noise=true , address=145.79.227.2:4129 , piece_id=2Y7VDPW5SS3D4J45EAMSLOEQAHNKYCZJEPP2FTP7GKVCYKMDQTLA, seg_pos={0 0}, upload=332765]: Done uploading piece. err: ecclient: upload failed (node:12GJof7GVfEN9u2yVyWf6j24hnaTdD3ygoPRoicmSqSfcNJFzs6, address:145.79.227.2:4129): protocol: rpc: dial tcp 145.79.227.2:4129: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:07:04.645453468 (297h7m2.242224501s) [noise=true , address=145.79.227.49:4360 , piece_id=OP5BBV7CUZBSMBO6HTTHQCVXDQGS6XO5LGAZ4I2SKUSPGUAKBXKA, seg_pos={0 0}, upload=332765]: Done uploading piece. err: ecclient: upload failed (node:12T24rtYBJ47E6Ef1ZZKMWcpBrWBpCg5QV2ZHniCZZkzJF2yUNk, address:145.79.227.49:4360): protocol: rpc: dial tcp 145.79.227.49:4360: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:07:04.645552758 (297h7m2.242323785s) [noise=true , address=145.79.227.33:4875 , piece_id=5LQOGDTQD5CCBMKIZCSLBVQUHQWU6W5ELX6XKLFSFQLU3NVDLKPQ, seg_pos={0 0}, upload=332765]: Done uploading piece. err: ecclient: upload failed (node:1ijEJgYouyMkUqpLWWz7eH2NGvvs9XpN2ujDjupPykDe5eWcB1, address:145.79.227.33:4875): protocol: rpc: dial tcp 145.79.227.33:4875: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:07:04.645682576 (297h7m2.242453607s) [noise=true , address=145.79.227.51:4424 , piece_id=5XCD27Q3GOCFGSHDUNX7QIGUQ6URBU2UREPVX5ZF7CFICF4X4UNQ, seg_pos={0 0}, upload=332765]: Done uploading piece. err: ecclient: upload failed (node:12LwZixdSPnCm3Wm27PBY79jj52sjkNh2VuQexagieTGBp4Ps2k, address:145.79.227.51:4424): protocol: rpc: dial tcp 145.79.227.51:4424: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:07:04.645716594 (297h7m2.24248761s) [seg_pos={0 0}, upload=332765]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Mar 18 20:07:04.666238325 (297h7m2.263009349s) [upload=332765]: Done uploading segments... Mar 18 20:07:04.666276723 (297h7m2.263047746s) [upload=332765]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 1m0.024298418s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 6 out of 60 is failed (threshold=3) [........................X..........XX..X....X........X......] duration: 19.413184936s trace: https://jaeger.dev.storj.io/trace/589419efc52e5e2b FAIL (2026-03-18 20:06:37 UTC) operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== Mar 18 20:00:59.343084966 (297h0m56.939855989s) [upload=332666]: Starting upload Mar 18 20:00:59.343175518 (297h0m56.939946548s) [upload=332666]: Done starting upload Mar 18 20:00:59.343565585 (297h0m56.940336612s) [upload=332666]: Uploading segments... Mar 18 20:00:59.344256647 (297h0m56.941027676s) [upload=332666]: Got next segment. Inline: false Mar 18 20:00:59.344352215 (297h0m56.941123241s) [upload=332666]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 20:00:59.344358706 (297h0m56.941129732s) [upload=332666]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 18 20:01:02.340259299 (297h0m59.937030329s) [seg_pos={0 0}, upload=332666]: Begin upload segment... Mar 18 20:01:02.340368743 (297h0m59.937139767s) [seg_pos={0 0}, upload=332666]: Done begin upload segment. Mar 18 20:01:02.340479563 (297h0m59.93725059s) [noise=true , address=45.91.116.8:4061 , piece_id=6XFVKT4AAO7QN457OQMRKPI6R24TJTAN47UXLEVOEFQVOS6M6C3Q, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.340523837 (297h0m59.93729486s) [noise=true , address=79.127.205.233:4166 , piece_id=SFGK5YCWHFRVXF6QNHYP2CYJCZLSTQ76OYRVJPWWXQTF2KFSFJKA, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.341468438 (297h0m59.938239447s) [noise=true , address=45.91.116.4:4961 , piece_id=ZMUGQECUVPVHDNKN75XPBET5M2HFXXK4WMTP4V5KEK63JVZCX5EA, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.341520025 (297h0m59.938291058s) [noise=true , address=167.17.72.71:4815 , piece_id=CXGTYNNBILN3TT7I6Z7UI75F2ZCS4QUTLS6KC4FJ5IHS3XOYWDJQ, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.342232992 (297h0m59.939004023s) [noise=true , address=145.79.227.36:4524 , piece_id=3TAV3P55ITTHBAZEUUUTOAHJRGQXJXXVGZJP6AVR2ANVNOR3S54Q, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.342917368 (297h0m59.939688398s) [noise=true , address=193.149.164.71:4297 , piece_id=4IJPXLPCX3LKFOBOSRHK2DTNRKH67K7FXJUKQ3JPRWHS47ZFEIDA, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.343244957 (297h0m59.940015982s) [noise=true , address=145.79.227.33:4892 , piece_id=DQ6ALC3VOAGFIJI3IKR6I7QFYM22C66QWAAXMH6NXI2FIQEZNQYA, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.344270079 (297h0m59.941041117s) [noise=true , address=145.79.227.21:4497 , piece_id=SP6EKJB7W4QXF7OIBVKEFWCPOZFL5IRIEDVAGS2M4QJSO34CUVRQ, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.345025057 (297h0m59.941796084s) [noise=true , address=77.90.31.17:4793 , piece_id=2U4IBUA2GGUGDBXQGIGQKUK4ZR3L7CDKC6MTGIKAUME4F7MFZRLA, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.345115802 (297h0m59.941886836s) [noise=true , address=167.17.72.69:4731 , piece_id=VOEVIKKFJMC2Y3TPQIDMYUP37NLFU7KPRH4HRRSMQLWUKTSLOH2Q, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.345690600 (297h0m59.942461634s) [noise=true , address=77.90.31.13:4648 , piece_id=BPMB5CFOYHFT3BS4ORR6JF6BNFC2AOAJZOSPMG67NAMLGGP5XVEQ, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.346311592 (297h0m59.943082623s) [noise=true , address=79.127.205.234:4202 , piece_id=YY4LYWX26O6RIOBONESNKTUZEFZCRL4UUX3NEBQ53AJV4IEDG5MQ, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.346791142 (297h0m59.943562166s) [noise=true , address=45.91.116.10:4110 , piece_id=UYLMDCWJEJOHGICA37KNH7G5AQXNIPRH5NRYLBUUMGDRVSLTEYHQ, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.347094999 (297h0m59.94386607s) [noise=true , address=145.79.227.17:4987 , piece_id=PT3TWVB65JZFEWVH4IUZEU2N53FQ57QBRWVLWAUFP4UDLY3LWNLQ, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.347781281 (297h0m59.944552305s) [noise=true , address=79.127.219.46:4919 , piece_id=KK6HRIVBMYLIHRHGPQH3AWMNSVHTTIBESHDMJZA72T446RX6QKRQ, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.348263123 (297h0m59.945034137s) [noise=true , address=193.149.164.68:4033 , piece_id=JZPQKPYGO3W3RZZBITQ73X2XEPIBBSLGRMWIVOM4Z3GTV5UFKRTQ, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.348915505 (297h0m59.945686536s) [noise=true , address=145.79.227.7:4312 , piece_id=2FAL6E3RMBTBHZBXAJJVH2N5VGF5VYI3KNZXGNWCMWGBBPAUO2NQ, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.349366298 (297h0m59.946137337s) [noise=true , address=193.149.164.67:4007 , piece_id=HURQGH67I5B6V4YUV3PRZL3KSKN3XTSK3UMCHZRNT6L3W3PXXHPQ, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.429834098 (297h1m0.026605126s) [upload=332666]: Next returned nil segment Mar 18 20:01:02.429853411 (297h1m0.026624442s) [upload=332666]: Waiting for error group managing segments... Mar 18 20:01:02.446867680 (297h1m0.043638704s) [noise=true , address=45.91.116.10:4110 , piece_id=UYLMDCWJEJOHGICA37KNH7G5AQXNIPRH5NRYLBUUMGDRVSLTEYHQ, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil> Mar 18 20:01:02.451665165 (297h1m0.048436196s) [noise=true , address=45.91.116.4:4961 , piece_id=ZMUGQECUVPVHDNKN75XPBET5M2HFXXK4WMTP4V5KEK63JVZCX5EA, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil> Mar 18 20:01:02.452538886 (297h1m0.049309913s) [noise=true , address=77.90.31.17:4793 , piece_id=2U4IBUA2GGUGDBXQGIGQKUK4ZR3L7CDKC6MTGIKAUME4F7MFZRLA, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil> Mar 18 20:01:02.455597040 (297h1m0.052368067s) [noise=true , address=45.91.116.8:4061 , piece_id=6XFVKT4AAO7QN457OQMRKPI6R24TJTAN47UXLEVOEFQVOS6M6C3Q, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil> Mar 18 20:01:02.457848739 (297h1m0.054619766s) [noise=true , address=77.90.31.13:4648 , piece_id=BPMB5CFOYHFT3BS4ORR6JF6BNFC2AOAJZOSPMG67NAMLGGP5XVEQ, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil> Mar 18 20:01:02.461479709 (297h1m0.058250746s) [noise=true , address=79.127.219.46:4919 , piece_id=KK6HRIVBMYLIHRHGPQH3AWMNSVHTTIBESHDMJZA72T446RX6QKRQ, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil> Mar 18 20:01:02.676859466 (297h1m0.27363049s) [noise=true , address=167.17.72.69:4731 , piece_id=VOEVIKKFJMC2Y3TPQIDMYUP37NLFU7KPRH4HRRSMQLWUKTSLOH2Q, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil> Mar 18 20:01:02.909860743 (297h1m0.506631766s) [noise=true , address=79.127.205.234:4202 , piece_id=YY4LYWX26O6RIOBONESNKTUZEFZCRL4UUX3NEBQ53AJV4IEDG5MQ, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil> Mar 18 20:01:02.930427742 (297h1m0.527198754s) [noise=true , address=193.149.164.71:4297 , piece_id=4IJPXLPCX3LKFOBOSRHK2DTNRKH67K7FXJUKQ3JPRWHS47ZFEIDA, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil> Mar 18 20:01:02.944696889 (297h1m0.541467914s) [noise=true , address=193.149.164.67:4007 , piece_id=HURQGH67I5B6V4YUV3PRZL3KSKN3XTSK3UMCHZRNT6L3W3PXXHPQ, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil> Mar 18 20:01:02.951319183 (297h1m0.548090212s) [noise=true , address=193.149.164.68:4033 , piece_id=JZPQKPYGO3W3RZZBITQ73X2XEPIBBSLGRMWIVOM4Z3GTV5UFKRTQ, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil> Mar 18 20:01:02.975619436 (297h1m0.572390467s) [noise=true , address=79.127.205.233:4166 , piece_id=SFGK5YCWHFRVXF6QNHYP2CYJCZLSTQ76OYRVJPWWXQTF2KFSFJKA, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil> Mar 18 20:01:03.819918528 (297h1m1.416689564s) [noise=true , address=167.17.72.71:4815 , piece_id=CXGTYNNBILN3TT7I6Z7UI75F2ZCS4QUTLS6KC4FJ5IHS3XOYWDJQ, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil> Mar 18 20:01:59.307873420 (297h1m56.904644452s) [noise=true , address=145.79.227.7:4312 , piece_id=2FAL6E3RMBTBHZBXAJJVH2N5VGF5VYI3KNZXGNWCMWGBBPAUO2NQ, seg_pos={0 0}, upload=332666]: Done uploading piece. err: ecclient: upload failed (node:12AVp3qn5uwLPqYaZEwzcgoGMJhNga1BUh6bK5k2bxjWgZLQiiC, address:145.79.227.7:4312): protocol: rpc: dial tcp 145.79.227.7:4312: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:01:59.308026092 (297h1m56.904797124s) [noise=true , address=145.79.227.17:4987 , piece_id=PT3TWVB65JZFEWVH4IUZEU2N53FQ57QBRWVLWAUFP4UDLY3LWNLQ, seg_pos={0 0}, upload=332666]: Done uploading piece. err: ecclient: upload failed (node:1XsPVNGYUhKnXkQRfVfjyBRzyvhShSdsrnzXtwe4Xtf5MaHMvH, address:145.79.227.17:4987): protocol: rpc: dial tcp 145.79.227.17:4987: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:01:59.308195864 (297h1m56.904966894s) [noise=true , address=145.79.227.36:4524 , piece_id=3TAV3P55ITTHBAZEUUUTOAHJRGQXJXXVGZJP6AVR2ANVNOR3S54Q, seg_pos={0 0}, upload=332666]: Done uploading piece. err: ecclient: upload failed (node:1xfbCzjSwFDRrR1gPggeN8tNCLbRRWDVzCJ6mqhZsbVSe8d4s4, address:145.79.227.36:4524): protocol: rpc: dial tcp 145.79.227.36:4524: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:01:59.308316037 (297h1m56.90508706s) [noise=true , address=145.79.227.33:4892 , piece_id=DQ6ALC3VOAGFIJI3IKR6I7QFYM22C66QWAAXMH6NXI2FIQEZNQYA, seg_pos={0 0}, upload=332666]: Done uploading piece. err: ecclient: upload failed (node:12RHwFxcUjPsn12uHEAH3fBDJKv1v1tM8mRJXoFrZrG5djdswF5, address:145.79.227.33:4892): protocol: rpc: dial tcp 145.79.227.33:4892: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:01:59.308404403 (297h1m56.905175433s) [noise=true , address=145.79.227.21:4497 , piece_id=SP6EKJB7W4QXF7OIBVKEFWCPOZFL5IRIEDVAGS2M4QJSO34CUVRQ, seg_pos={0 0}, upload=332666]: Done uploading piece. err: ecclient: upload failed (node:12ZBe4we7VeFdthqRxyy1V9uQrA4r8ouNX68xqsk5bD26dkf2Gd, address:145.79.227.21:4497): protocol: context deadline exceeded; context deadline exceeded Mar 18 20:01:59.308437457 (297h1m56.90520847s) [seg_pos={0 0}, upload=332666]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 18 20:01:59.338288467 (297h1m56.93505949s) [upload=332666]: Done uploading segments... Mar 18 20:01:59.338335497 (297h1m56.935106523s) [upload=332666]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 19.413232746s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 42 out of 60 is failed (threshold=3) [.................XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.XXXXX] duration: 30.030247022s trace: https://jaeger.dev.storj.io/trace/23494490358cd6b7 FAIL (2026-03-18 20:06:35 UTC) operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Mar 18 20:06:05.282628304 (297h6m2.879399334s) [upload=332766]: Starting upload Mar 18 20:06:05.282795681 (297h6m2.879566704s) [upload=332766]: Done starting upload Mar 18 20:06:05.282842377 (297h6m2.8796134s) [upload=332766]: Uploading segments... Mar 18 20:06:05.283238719 (297h6m2.880009743s) [upload=332766]: Got next segment. Inline: false Mar 18 20:06:05.283293147 (297h6m2.88006417s) [upload=332766]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 20:06:05.283298441 (297h6m2.880069464s) [upload=332766]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 18 20:06:08.632159628 (297h6m6.228930653s) [seg_pos={0 0}, upload=332766]: Begin upload segment... Mar 18 20:06:08.632244590 (297h6m6.229015629s) [seg_pos={0 0}, upload=332766]: Done begin upload segment. Mar 18 20:06:08.632298654 (297h6m6.229069683s) [noise=true , address=145.79.227.46:4602 , piece_id=MZJEPPRGLNNPJILG4BDSS6UPTS6YUQU2MF3NYRSJHDHZWJRAHVQQ, seg_pos={0 0}, upload=332766]: Uploading piece... Mar 18 20:06:08.632372815 (297h6m6.229143859s) [noise=true , address=45.91.116.6:4973 , piece_id=Q7F7BWO7GLLVQ2JD5J4IXVQRBTKSM2XQ2THGRUNTIZVPQGIBTHXA, seg_pos={0 0}, upload=332766]: Uploading piece... Mar 18 20:06:08.632853150 (297h6m6.229624179s) [noise=true , address=79.127.219.38:4817 , piece_id=3ITGVZ6D3UEZM6EBT74PVGDKEPDREROCSABVOI2DJCUG3MXBGZXA, seg_pos={0 0}, upload=332766]: Uploading piece... Mar 18 20:06:08.633037964 (297h6m6.229808991s) [noise=true , address=45.91.116.14:4190 , piece_id=Q2DINKOGKXMXHK2FWIUJ7UJPJOK7YXTT3J5URCRAPE33QE3CVKQQ, seg_pos={0 0}, upload=332766]: Uploading piece... Mar 18 20:06:08.633600584 (297h6m6.230371615s) [noise=true , address=145.79.227.30:4765 , piece_id=C755IGFDESRWBLB4HJUCCU6AT2OQ527LJ45TXJGP3JDDBTDU3SYA, seg_pos={0 0}, upload=332766]: Uploading piece... Mar 18 20:06:08.634088772 (297h6m6.2308598s) [noise=true , address=45.91.116.8:4049 , piece_id=BZ3SWDXEQOUYG63JWNJSHVWYUT7CZII3XOZFOM42FJYEU7TVBQCA, seg_pos={0 0}, upload=332766]: Uploading piece... Mar 18 20:06:08.634552252 (297h6m6.231323281s) [noise=true , address=45.91.116.9:4295 , piece_id=D2SKIJ2U2TBN77JAACFJ4VPM5DOSWUJCDM7XAGW7A5XFDOTFCGJQ, seg_pos={0 0}, upload=332766]: Uploading piece... Mar 18 20:06:08.634902655 (297h6m6.231673681s) [noise=true , address=145.79.227.18:4011 , piece_id=GUAK2YGKWHTFT6ILBEBTW3WDUVDSPRMJMK4QKFKQAMP5Z4T6DN5Q, seg_pos={0 0}, upload=332766]: Uploading piece... Mar 18 20:06:08.635482314 (297h6m6.232253348s) [noise=true , address=45.91.116.12:4227 , piece_id=L7ET2KRZ466GPRMPOHMXH3B6QOLBCJ3IBKZCUPHKDBGOZ6KJJYKQ, seg_pos={0 0}, upload=332766]: Uploading piece... Mar 18 20:06:08.635895918 (297h6m6.232666945s) [noise=true , address=77.90.31.19:4842 , piece_id=PVL4WQXTUPD7BQKBIDLMGEXT7SMQNDM3RNPXQOOAJCVOPBB3VR3Q, seg_pos={0 0}, upload=332766]: Uploading piece... Mar 18 20:06:08.636347717 (297h6m6.233118746s) [noise=true , address=79.127.219.46:4891 , piece_id=A2ST6ST3S3JIEJMYVK7H6JV7DVZH4ZG5M4DDSFGUL7D72C4WN2NA, seg_pos={0 0}, upload=332766]: Uploading piece... Mar 18 20:06:08.636839781 (297h6m6.233610807s) [noise=true , address=77.90.31.17:4774 , piece_id=OMPZMSWC67NOCJ2ORYHWPTIETQJ4PR2QYRXMTPX3GVHTUQWACHYQ, seg_pos={0 0}, upload=332766]: Uploading piece... Mar 18 20:06:08.637322012 (297h6m6.234093045s) [noise=true , address=145.79.227.22:4509 , piece_id=ILF5RNHD2SBPYIPIKPBQCA2ECLKHJVZTRK7E4ZOYTHRHVSYIEW2Q, seg_pos={0 0}, upload=332766]: Uploading piece... Mar 18 20:06:08.637771901 (297h6m6.234542933s) [noise=true , address=79.127.219.45:4853 , piece_id=THWASPJHYKFJEUTNSNG3NXABTGMTB3YKAVN5VS4VNHSYVSTCPRAA, seg_pos={0 0}, upload=332766]: Uploading piece... Mar 18 20:06:08.638337171 (297h6m6.23510821s) [noise=true , address=45.91.116.4:4937 , piece_id=PI3L4NXYSFOSZS7GNFMHR2R5XUPZ4DQITRW6OKUDEKP62BERTPKA, seg_pos={0 0}, upload=332766]: Uploading piece... Mar 18 20:06:08.639651650 (297h6m6.23642268s) [noise=true , address=45.91.116.10:4084 , piece_id=ARZ3H4U6YV22DQNONIN4RWYBVLHXPMBR7E2QQS243WRJUTIMMKWA, seg_pos={0 0}, upload=332766]: Uploading piece... Mar 18 20:06:08.710281830 (297h6m6.307052853s) [upload=332766]: Next returned nil segment Mar 18 20:06:08.710295449 (297h6m6.307066478s) [upload=332766]: Waiting for error group managing segments... Mar 18 20:06:08.714842804 (297h6m6.31161383s) [noise=true , address=77.90.31.19:4842 , piece_id=PVL4WQXTUPD7BQKBIDLMGEXT7SMQNDM3RNPXQOOAJCVOPBB3VR3Q, seg_pos={0 0}, upload=332766]: Done uploading piece. err: <nil> Mar 18 20:06:08.715287762 (297h6m6.312058792s) [noise=true , address=45.91.116.10:4084 , piece_id=ARZ3H4U6YV22DQNONIN4RWYBVLHXPMBR7E2QQS243WRJUTIMMKWA, seg_pos={0 0}, upload=332766]: Done uploading piece. err: <nil> Mar 18 20:06:08.715890901 (297h6m6.312661936s) [noise=true , address=45.91.116.12:4227 , piece_id=L7ET2KRZ466GPRMPOHMXH3B6QOLBCJ3IBKZCUPHKDBGOZ6KJJYKQ, seg_pos={0 0}, upload=332766]: Done uploading piece. err: <nil> Mar 18 20:06:08.716046394 (297h6m6.312817428s) [noise=true , address=79.127.219.45:4853 , piece_id=THWASPJHYKFJEUTNSNG3NXABTGMTB3YKAVN5VS4VNHSYVSTCPRAA, seg_pos={0 0}, upload=332766]: Done uploading piece. err: <nil> Mar 18 20:06:08.716572280 (297h6m6.313343304s) [noise=true , address=45.91.116.9:4295 , piece_id=D2SKIJ2U2TBN77JAACFJ4VPM5DOSWUJCDM7XAGW7A5XFDOTFCGJQ, seg_pos={0 0}, upload=332766]: Done uploading piece. err: <nil> Mar 18 20:06:08.716710311 (297h6m6.313481337s) [noise=true , address=45.91.116.8:4049 , piece_id=BZ3SWDXEQOUYG63JWNJSHVWYUT7CZII3XOZFOM42FJYEU7TVBQCA, seg_pos={0 0}, upload=332766]: Done uploading piece. err: <nil> Mar 18 20:06:08.717219445 (297h6m6.313990468s) [noise=true , address=79.127.219.38:4817 , piece_id=3ITGVZ6D3UEZM6EBT74PVGDKEPDREROCSABVOI2DJCUG3MXBGZXA, seg_pos={0 0}, upload=332766]: Done uploading piece. err: <nil> Mar 18 20:06:08.718925552 (297h6m6.315696584s) [noise=true , address=79.127.219.46:4891 , piece_id=A2ST6ST3S3JIEJMYVK7H6JV7DVZH4ZG5M4DDSFGUL7D72C4WN2NA, seg_pos={0 0}, upload=332766]: Done uploading piece. err: <nil> Mar 18 20:06:08.727125237 (297h6m6.323896271s) [noise=true , address=45.91.116.6:4973 , piece_id=Q7F7BWO7GLLVQ2JD5J4IXVQRBTKSM2XQ2THGRUNTIZVPQGIBTHXA, seg_pos={0 0}, upload=332766]: Done uploading piece. err: <nil> Mar 18 20:06:08.730082278 (297h6m6.32685331s) [noise=true , address=45.91.116.4:4937 , piece_id=PI3L4NXYSFOSZS7GNFMHR2R5XUPZ4DQITRW6OKUDEKP62BERTPKA, seg_pos={0 0}, upload=332766]: Done uploading piece. err: <nil> Mar 18 20:06:08.730260498 (297h6m6.327031522s) [noise=true , address=77.90.31.17:4774 , piece_id=OMPZMSWC67NOCJ2ORYHWPTIETQJ4PR2QYRXMTPX3GVHTUQWACHYQ, seg_pos={0 0}, upload=332766]: Done uploading piece. err: <nil> Mar 18 20:06:08.733242108 (297h6m6.330013134s) [noise=true , address=45.91.116.14:4190 , piece_id=Q2DINKOGKXMXHK2FWIUJ7UJPJOK7YXTT3J5URCRAPE33QE3CVKQQ, seg_pos={0 0}, upload=332766]: Done uploading piece. err: <nil> Mar 18 20:06:35.252034720 (297h6m32.848805753s) [noise=true , address=145.79.227.18:4011 , piece_id=GUAK2YGKWHTFT6ILBEBTW3WDUVDSPRMJMK4QKFKQAMP5Z4T6DN5Q, seg_pos={0 0}, upload=332766]: Done uploading piece. err: ecclient: upload failed (node:12J8J47emDh4p115m69EqUYroi9GgruC3y8gmv2xVULVJSvFYYP, address:145.79.227.18:4011): protocol: context deadline exceeded; context deadline exceeded Mar 18 20:06:35.252396325 (297h6m32.849167367s) [noise=true , address=145.79.227.46:4602 , piece_id=MZJEPPRGLNNPJILG4BDSS6UPTS6YUQU2MF3NYRSJHDHZWJRAHVQQ, seg_pos={0 0}, upload=332766]: Done uploading piece. err: ecclient: upload failed (node:1kgUqvw1r3gSKw1bNjzys3AiAJXdkAnSafmvcPgNVPBbN9JS1d, address:145.79.227.46:4602): protocol: context deadline exceeded; context deadline exceeded Mar 18 20:06:35.252724550 (297h6m32.849495582s) [noise=true , address=145.79.227.22:4509 , piece_id=ILF5RNHD2SBPYIPIKPBQCA2ECLKHJVZTRK7E4ZOYTHRHVSYIEW2Q, seg_pos={0 0}, upload=332766]: Done uploading piece. err: ecclient: upload failed (node:1229E58YirF37sZU6hfZGrDNFSxQDYsWSo4x3mG9BoYsNsLNtjy, address:145.79.227.22:4509): protocol: rpc: dial tcp 145.79.227.22:4509: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:06:35.252737446 (297h6m32.849508456s) [noise=true , address=145.79.227.30:4765 , piece_id=C755IGFDESRWBLB4HJUCCU6AT2OQ527LJ45TXJGP3JDDBTDU3SYA, seg_pos={0 0}, upload=332766]: Done uploading piece. err: ecclient: upload failed (node:12SRRQBu9efFwqpKMvLBQhaZ9s3XQXVekdmefCBXuQPgRdik2QT, address:145.79.227.30:4765): protocol: context deadline exceeded; context deadline exceeded Mar 18 20:06:35.252789145 (297h6m32.849560155s) [seg_pos={0 0}, upload=332766]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Mar 18 20:06:35.279003450 (297h6m32.875774491s) [upload=332766]: Done uploading segments... Mar 18 20:06:35.279077816 (297h6m32.875848848s) [upload=332766]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 30.030273403s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 6 out of 60 is failed (threshold=3) [.........................X..........XX..X....X........X.....] duration: 18.571785438s trace: https://jaeger.dev.storj.io/trace/589419efc52e5e2b FAIL (2026-03-18 20:06:07 UTC) operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== Mar 18 20:00:59.343084966 (297h0m56.939855989s) [upload=332666]: Starting upload Mar 18 20:00:59.343175518 (297h0m56.939946548s) [upload=332666]: Done starting upload Mar 18 20:00:59.343565585 (297h0m56.940336612s) [upload=332666]: Uploading segments... Mar 18 20:00:59.344256647 (297h0m56.941027676s) [upload=332666]: Got next segment. Inline: false Mar 18 20:00:59.344352215 (297h0m56.941123241s) [upload=332666]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 20:00:59.344358706 (297h0m56.941129732s) [upload=332666]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 18 20:01:02.340259299 (297h0m59.937030329s) [seg_pos={0 0}, upload=332666]: Begin upload segment... Mar 18 20:01:02.340368743 (297h0m59.937139767s) [seg_pos={0 0}, upload=332666]: Done begin upload segment. Mar 18 20:01:02.340479563 (297h0m59.93725059s) [noise=true , address=45.91.116.8:4061 , piece_id=6XFVKT4AAO7QN457OQMRKPI6R24TJTAN47UXLEVOEFQVOS6M6C3Q, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.340523837 (297h0m59.93729486s) [noise=true , address=79.127.205.233:4166 , piece_id=SFGK5YCWHFRVXF6QNHYP2CYJCZLSTQ76OYRVJPWWXQTF2KFSFJKA, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.341468438 (297h0m59.938239447s) [noise=true , address=45.91.116.4:4961 , piece_id=ZMUGQECUVPVHDNKN75XPBET5M2HFXXK4WMTP4V5KEK63JVZCX5EA, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.341520025 (297h0m59.938291058s) [noise=true , address=167.17.72.71:4815 , piece_id=CXGTYNNBILN3TT7I6Z7UI75F2ZCS4QUTLS6KC4FJ5IHS3XOYWDJQ, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.342232992 (297h0m59.939004023s) [noise=true , address=145.79.227.36:4524 , piece_id=3TAV3P55ITTHBAZEUUUTOAHJRGQXJXXVGZJP6AVR2ANVNOR3S54Q, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.342917368 (297h0m59.939688398s) [noise=true , address=193.149.164.71:4297 , piece_id=4IJPXLPCX3LKFOBOSRHK2DTNRKH67K7FXJUKQ3JPRWHS47ZFEIDA, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.343244957 (297h0m59.940015982s) [noise=true , address=145.79.227.33:4892 , piece_id=DQ6ALC3VOAGFIJI3IKR6I7QFYM22C66QWAAXMH6NXI2FIQEZNQYA, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.344270079 (297h0m59.941041117s) [noise=true , address=145.79.227.21:4497 , piece_id=SP6EKJB7W4QXF7OIBVKEFWCPOZFL5IRIEDVAGS2M4QJSO34CUVRQ, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.345025057 (297h0m59.941796084s) [noise=true , address=77.90.31.17:4793 , piece_id=2U4IBUA2GGUGDBXQGIGQKUK4ZR3L7CDKC6MTGIKAUME4F7MFZRLA, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.345115802 (297h0m59.941886836s) [noise=true , address=167.17.72.69:4731 , piece_id=VOEVIKKFJMC2Y3TPQIDMYUP37NLFU7KPRH4HRRSMQLWUKTSLOH2Q, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.345690600 (297h0m59.942461634s) [noise=true , address=77.90.31.13:4648 , piece_id=BPMB5CFOYHFT3BS4ORR6JF6BNFC2AOAJZOSPMG67NAMLGGP5XVEQ, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.346311592 (297h0m59.943082623s) [noise=true , address=79.127.205.234:4202 , piece_id=YY4LYWX26O6RIOBONESNKTUZEFZCRL4UUX3NEBQ53AJV4IEDG5MQ, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.346791142 (297h0m59.943562166s) [noise=true , address=45.91.116.10:4110 , piece_id=UYLMDCWJEJOHGICA37KNH7G5AQXNIPRH5NRYLBUUMGDRVSLTEYHQ, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.347094999 (297h0m59.94386607s) [noise=true , address=145.79.227.17:4987 , piece_id=PT3TWVB65JZFEWVH4IUZEU2N53FQ57QBRWVLWAUFP4UDLY3LWNLQ, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.347781281 (297h0m59.944552305s) [noise=true , address=79.127.219.46:4919 , piece_id=KK6HRIVBMYLIHRHGPQH3AWMNSVHTTIBESHDMJZA72T446RX6QKRQ, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.348263123 (297h0m59.945034137s) [noise=true , address=193.149.164.68:4033 , piece_id=JZPQKPYGO3W3RZZBITQ73X2XEPIBBSLGRMWIVOM4Z3GTV5UFKRTQ, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.348915505 (297h0m59.945686536s) [noise=true , address=145.79.227.7:4312 , piece_id=2FAL6E3RMBTBHZBXAJJVH2N5VGF5VYI3KNZXGNWCMWGBBPAUO2NQ, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.349366298 (297h0m59.946137337s) [noise=true , address=193.149.164.67:4007 , piece_id=HURQGH67I5B6V4YUV3PRZL3KSKN3XTSK3UMCHZRNT6L3W3PXXHPQ, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.429834098 (297h1m0.026605126s) [upload=332666]: Next returned nil segment Mar 18 20:01:02.429853411 (297h1m0.026624442s) [upload=332666]: Waiting for error group managing segments... Mar 18 20:01:02.446867680 (297h1m0.043638704s) [noise=true , address=45.91.116.10:4110 , piece_id=UYLMDCWJEJOHGICA37KNH7G5AQXNIPRH5NRYLBUUMGDRVSLTEYHQ, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil> Mar 18 20:01:02.451665165 (297h1m0.048436196s) [noise=true , address=45.91.116.4:4961 , piece_id=ZMUGQECUVPVHDNKN75XPBET5M2HFXXK4WMTP4V5KEK63JVZCX5EA, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil> Mar 18 20:01:02.452538886 (297h1m0.049309913s) [noise=true , address=77.90.31.17:4793 , piece_id=2U4IBUA2GGUGDBXQGIGQKUK4ZR3L7CDKC6MTGIKAUME4F7MFZRLA, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil> Mar 18 20:01:02.455597040 (297h1m0.052368067s) [noise=true , address=45.91.116.8:4061 , piece_id=6XFVKT4AAO7QN457OQMRKPI6R24TJTAN47UXLEVOEFQVOS6M6C3Q, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil> Mar 18 20:01:02.457848739 (297h1m0.054619766s) [noise=true , address=77.90.31.13:4648 , piece_id=BPMB5CFOYHFT3BS4ORR6JF6BNFC2AOAJZOSPMG67NAMLGGP5XVEQ, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil> Mar 18 20:01:02.461479709 (297h1m0.058250746s) [noise=true , address=79.127.219.46:4919 , piece_id=KK6HRIVBMYLIHRHGPQH3AWMNSVHTTIBESHDMJZA72T446RX6QKRQ, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil> Mar 18 20:01:02.676859466 (297h1m0.27363049s) [noise=true , address=167.17.72.69:4731 , piece_id=VOEVIKKFJMC2Y3TPQIDMYUP37NLFU7KPRH4HRRSMQLWUKTSLOH2Q, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil> Mar 18 20:01:02.909860743 (297h1m0.506631766s) [noise=true , address=79.127.205.234:4202 , piece_id=YY4LYWX26O6RIOBONESNKTUZEFZCRL4UUX3NEBQ53AJV4IEDG5MQ, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil> Mar 18 20:01:02.930427742 (297h1m0.527198754s) [noise=true , address=193.149.164.71:4297 , piece_id=4IJPXLPCX3LKFOBOSRHK2DTNRKH67K7FXJUKQ3JPRWHS47ZFEIDA, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil> Mar 18 20:01:02.944696889 (297h1m0.541467914s) [noise=true , address=193.149.164.67:4007 , piece_id=HURQGH67I5B6V4YUV3PRZL3KSKN3XTSK3UMCHZRNT6L3W3PXXHPQ, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil> Mar 18 20:01:02.951319183 (297h1m0.548090212s) [noise=true , address=193.149.164.68:4033 , piece_id=JZPQKPYGO3W3RZZBITQ73X2XEPIBBSLGRMWIVOM4Z3GTV5UFKRTQ, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil> Mar 18 20:01:02.975619436 (297h1m0.572390467s) [noise=true , address=79.127.205.233:4166 , piece_id=SFGK5YCWHFRVXF6QNHYP2CYJCZLSTQ76OYRVJPWWXQTF2KFSFJKA, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil> Mar 18 20:01:03.819918528 (297h1m1.416689564s) [noise=true , address=167.17.72.71:4815 , piece_id=CXGTYNNBILN3TT7I6Z7UI75F2ZCS4QUTLS6KC4FJ5IHS3XOYWDJQ, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil> Mar 18 20:01:59.307873420 (297h1m56.904644452s) [noise=true , address=145.79.227.7:4312 , piece_id=2FAL6E3RMBTBHZBXAJJVH2N5VGF5VYI3KNZXGNWCMWGBBPAUO2NQ, seg_pos={0 0}, upload=332666]: Done uploading piece. err: ecclient: upload failed (node:12AVp3qn5uwLPqYaZEwzcgoGMJhNga1BUh6bK5k2bxjWgZLQiiC, address:145.79.227.7:4312): protocol: rpc: dial tcp 145.79.227.7:4312: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:01:59.308026092 (297h1m56.904797124s) [noise=true , address=145.79.227.17:4987 , piece_id=PT3TWVB65JZFEWVH4IUZEU2N53FQ57QBRWVLWAUFP4UDLY3LWNLQ, seg_pos={0 0}, upload=332666]: Done uploading piece. err: ecclient: upload failed (node:1XsPVNGYUhKnXkQRfVfjyBRzyvhShSdsrnzXtwe4Xtf5MaHMvH, address:145.79.227.17:4987): protocol: rpc: dial tcp 145.79.227.17:4987: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:01:59.308195864 (297h1m56.904966894s) [noise=true , address=145.79.227.36:4524 , piece_id=3TAV3P55ITTHBAZEUUUTOAHJRGQXJXXVGZJP6AVR2ANVNOR3S54Q, seg_pos={0 0}, upload=332666]: Done uploading piece. err: ecclient: upload failed (node:1xfbCzjSwFDRrR1gPggeN8tNCLbRRWDVzCJ6mqhZsbVSe8d4s4, address:145.79.227.36:4524): protocol: rpc: dial tcp 145.79.227.36:4524: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:01:59.308316037 (297h1m56.90508706s) [noise=true , address=145.79.227.33:4892 , piece_id=DQ6ALC3VOAGFIJI3IKR6I7QFYM22C66QWAAXMH6NXI2FIQEZNQYA, seg_pos={0 0}, upload=332666]: Done uploading piece. err: ecclient: upload failed (node:12RHwFxcUjPsn12uHEAH3fBDJKv1v1tM8mRJXoFrZrG5djdswF5, address:145.79.227.33:4892): protocol: rpc: dial tcp 145.79.227.33:4892: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:01:59.308404403 (297h1m56.905175433s) [noise=true , address=145.79.227.21:4497 , piece_id=SP6EKJB7W4QXF7OIBVKEFWCPOZFL5IRIEDVAGS2M4QJSO34CUVRQ, seg_pos={0 0}, upload=332666]: Done uploading piece. err: ecclient: upload failed (node:12ZBe4we7VeFdthqRxyy1V9uQrA4r8ouNX68xqsk5bD26dkf2Gd, address:145.79.227.21:4497): protocol: context deadline exceeded; context deadline exceeded Mar 18 20:01:59.308437457 (297h1m56.90520847s) [seg_pos={0 0}, upload=332666]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 18 20:01:59.338288467 (297h1m56.93505949s) [upload=332666]: Done uploading segments... Mar 18 20:01:59.338335497 (297h1m56.935106523s) [upload=332666]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 18.571847763s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 8 out of 60 is failed (threshold=3) [....................XXXXXX.XX...............................] duration: 18.030044887s trace: https://jaeger.dev.storj.io/trace/4b4d10a8051ad01b FAIL (2026-03-18 20:06:01 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 18.030077196s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 49 out of 60 is failed (threshold=6) [.......XXXXXXXXX.XXX...XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 5.001898184s trace: https://jaeger.dev.storj.io/trace/4f0aa72e410026b0 FAIL (2026-03-18 20:06:00 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001948228s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 41 out of 60 is failed (threshold=3) [..................XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.XXXX] duration: 30.023636333s trace: https://jaeger.dev.storj.io/trace/6ffa58c7b38ee67d FAIL (2026-03-18 20:05:43 UTC) operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 9) ======LOGS====== Mar 18 20:05:13.344937208 (297h5m10.941708232s) [upload=332747]: Starting upload Mar 18 20:05:13.345037845 (297h5m10.941808859s) [upload=332747]: Done starting upload Mar 18 20:05:13.345147372 (297h5m10.941918385s) [upload=332747]: Uploading segments... Mar 18 20:05:13.345560083 (297h5m10.942331115s) [upload=332747]: Got next segment. Inline: false Mar 18 20:05:13.345638058 (297h5m10.942409081s) [upload=332747]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 20:05:13.345643663 (297h5m10.942414676s) [upload=332747]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 18 20:05:16.704565606 (297h5m14.301336636s) [seg_pos={0 0}, upload=332747]: Begin upload segment... Mar 18 20:05:16.704663720 (297h5m14.301434744s) [seg_pos={0 0}, upload=332747]: Done begin upload segment. Mar 18 20:05:16.704714791 (297h5m14.301485823s) [noise=true , address=145.79.227.27:4693 , piece_id=OP73NTHTRIJBLTUGBKUVT7XNKHSSU44G3J6TCWTRLG7ZMK44YUZQ, seg_pos={0 0}, upload=332747]: Uploading piece... Mar 18 20:05:16.705177102 (297h5m14.301948125s) [noise=true , address=45.91.116.7:4011 , piece_id=2ZNTV6534LGCRE7NEZ4HYNB3QAUNSXT7EY5LXPQ4F4BOFMWF44CA, seg_pos={0 0}, upload=332747]: Uploading piece... Mar 18 20:05:16.705985696 (297h5m14.302756731s) [noise=true , address=145.79.227.52:4449 , piece_id=TG6JNDK4R74JOUN2OBQZ5YMTSFJKZUYWNWDS2SFRVEJBZFHBUUGA, seg_pos={0 0}, upload=332747]: Uploading piece... Mar 18 20:05:16.706160338 (297h5m14.302931362s) [noise=true , address=145.79.227.19:4081 , piece_id=W2SSYYSG4KHKIOLCWJIPLIY7SU6IM7SFQNAKBVYPQQRT2MQY7VIQ, seg_pos={0 0}, upload=332747]: Uploading piece... Mar 18 20:05:16.706734630 (297h5m14.303505663s) [noise=true , address=145.79.227.33:4869 , piece_id=OCXMBM2JTQGL6UZQO6KZTKTSHDQPETKHNP52UKUPAQUMCXH4N6IA, seg_pos={0 0}, upload=332747]: Uploading piece... Mar 18 20:05:16.706801964 (297h5m14.303572993s) [noise=true , address=77.90.31.18:4809 , piece_id=XLKDBACUY3XLMV7PA7T7NJWHEW7JH6ECFXHSDEUSKX5AQYRLTSIA, seg_pos={0 0}, upload=332747]: Uploading piece... Mar 18 20:05:16.707645514 (297h5m14.304416543s) [noise=true , address=77.90.31.13:4630 , piece_id=6J7JYSKVQZRKJYKC5AFRG7P7GKIERXDQK775567QDHWFYBZZTLAA, seg_pos={0 0}, upload=332747]: Uploading piece... Mar 18 20:05:16.707735787 (297h5m14.304506814s) [noise=true , address=45.91.116.10:4081 , piece_id=SV5TOYTRTIAP2RPNYLLYLAVJRQ2RWXPDAJGKUUPAGM7CBCE3C65Q, seg_pos={0 0}, upload=332747]: Uploading piece... Mar 18 20:05:16.708373568 (297h5m14.305144601s) [noise=true , address=145.79.227.44:4198 , piece_id=LBZ5CSZEDGZ2HYKAEEQAD6ULE4KR7QNTJTMQKHLTEHFY352GNRDA, seg_pos={0 0}, upload=332747]: Uploading piece... Mar 18 20:05:16.708422621 (297h5m14.30519365s) [noise=true , address=79.127.219.41:4927 , piece_id=NK2HFTTCTNPEMJKKB3XCD6MPOOQBRS73AOGVGBGNG4YIZEP4GV5Q, seg_pos={0 0}, upload=332747]: Uploading piece... Mar 18 20:05:16.708967154 (297h5m14.30573818s) [noise=true , address=145.79.227.17:4973 , piece_id=QFYTYQBMDV3Y3ZHCAE4Q7ZQBUP6MH7TNQTAGZS7N6LJWFX2LMQCQ, seg_pos={0 0}, upload=332747]: Uploading piece... Mar 18 20:05:16.709394623 (297h5m14.306165678s) [noise=true , address=45.91.116.14:4196 , piece_id=6JNYL3T3W3BZTNOLU6XNBU3I4KQRGFB6SC3VAMY7FJDXDDJLKORQ, seg_pos={0 0}, upload=332747]: Uploading piece... Mar 18 20:05:16.710113535 (297h5m14.306884583s) [noise=true , address=45.91.116.5:4332 , piece_id=6WIZHQ6L6HZJWHKDCISGFTR2S7VDJKSYFNAYPGGULD6NSYW43TSQ, seg_pos={0 0}, upload=332747]: Uploading piece... Mar 18 20:05:16.710838296 (297h5m14.307609333s) [noise=true , address=45.91.116.12:4226 , piece_id=US372HTBBAOX2FVUBUJKA2DWI5RKFLGJE2FYSX45YVPLBXLYJNVA, seg_pos={0 0}, upload=332747]: Uploading piece... Mar 18 20:05:16.711661010 (297h5m14.308432045s) [noise=true , address=145.79.227.3:4149 , piece_id=PB5KZRXWUWHVGQD24AHI7LLY7O4LELHAXSGQVF75UMSIBEVXJNQA, seg_pos={0 0}, upload=332747]: Uploading piece... Mar 18 20:05:16.712416678 (297h5m14.309187712s) [noise=true , address=45.91.116.4:4953 , piece_id=E7PW2NOTQFP3LLGNQEM7AFJUNW2ALDIPVAFU6MOHA43NPTXTR6RA, seg_pos={0 0}, upload=332747]: Uploading piece... Mar 18 20:05:16.814917880 (297h5m14.411688893s) [upload=332747]: Next returned nil segment Mar 18 20:05:16.814931862 (297h5m14.411702875s) [upload=332747]: Waiting for error group managing segments... Mar 18 20:05:16.832178751 (297h5m14.428949764s) [noise=true , address=79.127.219.41:4927 , piece_id=NK2HFTTCTNPEMJKKB3XCD6MPOOQBRS73AOGVGBGNG4YIZEP4GV5Q, seg_pos={0 0}, upload=332747]: Done uploading piece. err: <nil> Mar 18 20:05:16.848662444 (297h5m14.445433461s) [noise=true , address=45.91.116.12:4226 , piece_id=US372HTBBAOX2FVUBUJKA2DWI5RKFLGJE2FYSX45YVPLBXLYJNVA, seg_pos={0 0}, upload=332747]: Done uploading piece. err: <nil> Mar 18 20:05:16.849100628 (297h5m14.445871651s) [noise=true , address=77.90.31.18:4809 , piece_id=XLKDBACUY3XLMV7PA7T7NJWHEW7JH6ECFXHSDEUSKX5AQYRLTSIA, seg_pos={0 0}, upload=332747]: Done uploading piece. err: <nil> Mar 18 20:05:16.849646177 (297h5m14.446417195s) [noise=true , address=45.91.116.10:4081 , piece_id=SV5TOYTRTIAP2RPNYLLYLAVJRQ2RWXPDAJGKUUPAGM7CBCE3C65Q, seg_pos={0 0}, upload=332747]: Done uploading piece. err: <nil> Mar 18 20:05:16.852780530 (297h5m14.449551554s) [noise=true , address=45.91.116.14:4196 , piece_id=6JNYL3T3W3BZTNOLU6XNBU3I4KQRGFB6SC3VAMY7FJDXDDJLKORQ, seg_pos={0 0}, upload=332747]: Done uploading piece. err: <nil> Mar 18 20:05:16.854852024 (297h5m14.45162305s) [noise=true , address=45.91.116.5:4332 , piece_id=6WIZHQ6L6HZJWHKDCISGFTR2S7VDJKSYFNAYPGGULD6NSYW43TSQ, seg_pos={0 0}, upload=332747]: Done uploading piece. err: <nil> Mar 18 20:05:16.859407025 (297h5m14.456178058s) [noise=true , address=77.90.31.13:4630 , piece_id=6J7JYSKVQZRKJYKC5AFRG7P7GKIERXDQK775567QDHWFYBZZTLAA, seg_pos={0 0}, upload=332747]: Done uploading piece. err: <nil> Mar 18 20:05:16.878768350 (297h5m14.475539377s) [noise=true , address=45.91.116.4:4953 , piece_id=E7PW2NOTQFP3LLGNQEM7AFJUNW2ALDIPVAFU6MOHA43NPTXTR6RA, seg_pos={0 0}, upload=332747]: Done uploading piece. err: <nil> Mar 18 20:05:16.880876087 (297h5m14.477647121s) [noise=true , address=45.91.116.7:4011 , piece_id=2ZNTV6534LGCRE7NEZ4HYNB3QAUNSXT7EY5LXPQ4F4BOFMWF44CA, seg_pos={0 0}, upload=332747]: Done uploading piece. err: <nil> Mar 18 20:05:43.314317110 (297h5m40.911088144s) [noise=true , address=145.79.227.19:4081 , piece_id=W2SSYYSG4KHKIOLCWJIPLIY7SU6IM7SFQNAKBVYPQQRT2MQY7VIQ, seg_pos={0 0}, upload=332747]: Done uploading piece. err: ecclient: upload failed (node:12t9rGfCQ7bLkNnZzaPVgkE2Y9rXsiFp1XGdeECzv9M1jk8Qqmp, address:145.79.227.19:4081): protocol: rpc: dial tcp 145.79.227.19:4081: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:05:43.314379650 (297h5m40.911150696s) [noise=true , address=145.79.227.44:4198 , piece_id=LBZ5CSZEDGZ2HYKAEEQAD6ULE4KR7QNTJTMQKHLTEHFY352GNRDA, seg_pos={0 0}, upload=332747]: Done uploading piece. err: ecclient: upload failed (node:12YxL4YR6mNE536w1SnBrMUZHiaRTuU68Zjg9LBvRURVx5s5HWL, address:145.79.227.44:4198): protocol: context deadline exceeded; context deadline exceeded Mar 18 20:05:43.314396681 (297h5m40.911167714s) [noise=true , address=145.79.227.33:4869 , piece_id=OCXMBM2JTQGL6UZQO6KZTKTSHDQPETKHNP52UKUPAQUMCXH4N6IA, seg_pos={0 0}, upload=332747]: Done uploading piece. err: ecclient: upload failed (node:18CQxjq65r6xVTmNRurSQPzKxj4BoR6cJ8Uq29sTrdh6uXm5sq, address:145.79.227.33:4869): protocol: rpc: dial tcp 145.79.227.33:4869: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:05:43.314491172 (297h5m40.911262203s) [noise=true , address=145.79.227.17:4973 , piece_id=QFYTYQBMDV3Y3ZHCAE4Q7ZQBUP6MH7TNQTAGZS7N6LJWFX2LMQCQ, seg_pos={0 0}, upload=332747]: Done uploading piece. err: ecclient: upload failed (node:12afPNc97E17nBKr8sz5ujCrrVry16s6e28MEduErC2CRshtv9d, address:145.79.227.17:4973): protocol: rpc: dial tcp 145.79.227.17:4973: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:05:43.314540416 (297h5m40.911311443s) [noise=true , address=145.79.227.27:4693 , piece_id=OP73NTHTRIJBLTUGBKUVT7XNKHSSU44G3J6TCWTRLG7ZMK44YUZQ, seg_pos={0 0}, upload=332747]: Done uploading piece. err: ecclient: upload failed (node:11E39ZZcpnJR43M2RU3Ggb8neyXj6bbSJkANxdJNGuZJiH8c9d, address:145.79.227.27:4693): protocol: rpc: dial tcp 145.79.227.27:4693: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:05:43.314590775 (297h5m40.911361801s) [noise=true , address=145.79.227.3:4149 , piece_id=PB5KZRXWUWHVGQD24AHI7LLY7O4LELHAXSGQVF75UMSIBEVXJNQA, seg_pos={0 0}, upload=332747]: Done uploading piece. err: ecclient: upload failed (node:12qexRLJM6TY935UaEWsj1X9ViyBvk2BM1XGjnq8kapnXXXmgbm, address:145.79.227.3:4149): protocol: rpc: dial tcp 145.79.227.3:4149: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:05:43.314793982 (297h5m40.911565018s) [noise=true , address=145.79.227.52:4449 , piece_id=TG6JNDK4R74JOUN2OBQZ5YMTSFJKZUYWNWDS2SFRVEJBZFHBUUGA, seg_pos={0 0}, upload=332747]: Done uploading piece. err: ecclient: upload failed (node:12f14pHDEEamEnHmeZCe38rjaE6EYWYkGMPrZMsbSDmApN5rckq, address:145.79.227.52:4449): protocol: context deadline exceeded; context deadline exceeded Mar 18 20:05:43.314829156 (297h5m40.911600174s) [seg_pos={0 0}, upload=332747]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Mar 18 20:05:43.335589889 (297h5m40.932360902s) [upload=332747]: Done uploading segments... Mar 18 20:05:43.335626278 (297h5m40.932397304s) [upload=332747]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Duration: 30.023652117s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 8 out of 60 is failed (threshold=3) [.....................XXXXXX.XX..............................] duration: 16.853365089s trace: https://jaeger.dev.storj.io/trace/4b4d10a8051ad01b FAIL (2026-03-18 20:05:42 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 16.853392188s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 7 out of 60 is failed (threshold=3) [.......................XXX..........X...X.........X....X....] duration: 16.390937818s trace: https://jaeger.dev.storj.io/trace/698fb9649d932a68 FAIL (2026-03-18 20:05:32 UTC) operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 20:02:41.019616017 (297h2m38.616387042s) [upload=332700]: Uploading segments... Mar 18 20:02:41.019826191 (297h2m38.616597205s) [upload=332700]: Got next segment. Inline: false Mar 18 20:02:41.019879916 (297h2m38.61665094s) [upload=332700]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 20:02:41.019885584 (297h2m38.616656611s) [upload=332700]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 18 20:02:43.834160391 (297h2m41.430931414s) [seg_pos={0 0}, upload=332700]: Begin upload segment... Mar 18 20:02:43.834230381 (297h2m41.431001396s) [seg_pos={0 0}, upload=332700]: Done begin upload segment. Mar 18 20:02:43.834262428 (297h2m41.431033451s) [noise=true , address=145.79.227.52:4451 , piece_id=LR6KVRYMDO2HAESG5BFI5RYPNGVARP6O5HEY2UP77YSTVRKBNUXQ, seg_pos={0 0}, upload=332700]: Uploading piece... Mar 18 20:02:43.834329472 (297h2m41.431100497s) [noise=true , address=45.91.116.13:4164 , piece_id=MNQLVANO3LFKBCJMADD5RULARMRO4VUQY7LT63TU2CCNO765GW7Q, seg_pos={0 0}, upload=332700]: Uploading piece... Mar 18 20:02:43.834938832 (297h2m41.431709858s) [noise=true , address=77.90.31.15:4716 , piece_id=IDQP2MIPCKVS2FGVVGKAIDYTNUIPE56EER5IYZUSOTRD5LZBSXHA, seg_pos={0 0}, upload=332700]: Uploading piece... Mar 18 20:02:43.835898822 (297h2m41.432669851s) [noise=true , address=79.127.205.237:4227 , piece_id=NO25WIYZTEPBOMGETTLSDOJSPKARMKNLVUWDKNXUKHYGDGF2XC5Q, seg_pos={0 0}, upload=332700]: Uploading piece... Mar 18 20:02:43.836893726 (297h2m41.433664759s) [noise=true , address=193.149.164.68:4064 , piece_id=HAA3NDB24MJLAI2HK2UDZQZRLDUXEF76ZJF2VBE6FLKRWSB27Y3A, seg_pos={0 0}, upload=332700]: Uploading piece... Mar 18 20:02:43.837257949 (297h2m41.434028981s) [noise=true , address=193.149.164.66:4974 , piece_id=XWMBTFEWXVVFIR6IYNQYU35QKGVSQJ7YBVSWZ57QSUZ2VSDCJPRA, seg_pos={0 0}, upload=332700]: Uploading piece... Mar 18 20:02:43.838266356 (297h2m41.435037391s) [noise=true , address=145.79.227.34:4924 , piece_id=BNUESPWRZQBX23K7UGNSMF362AWHILNZ43UITBHVGFXIJ4T6V5HQ, seg_pos={0 0}, upload=332700]: Uploading piece... Mar 18 20:02:43.838688582 (297h2m41.435459605s) [noise=true , address=79.127.205.233:4171 , piece_id=YO42WSHYRB2JJX4RVX653N7U6PYTGIW3KQTNSF4445OTN3UW2TIA, seg_pos={0 0}, upload=332700]: Uploading piece... Mar 18 20:02:43.839263235 (297h2m41.436034258s) [noise=true , address=193.149.164.70:4123 , piece_id=VROUAQKRTKZT42ZDDSCIMKFMRODPKFQ5U5DWDJLVI462BVA5YWXA, seg_pos={0 0}, upload=332700]: Uploading piece... Mar 18 20:02:43.839835928 (297h2m41.436606963s) [noise=true , address=45.91.116.4:4948 , piece_id=ULVH4YAJIYY5RWWAGH5RTSZ3BMAT6WCYYAJPSMSP25LCLZQIPG2A, seg_pos={0 0}, upload=332700]: Uploading piece... Mar 18 20:02:43.840390028 (297h2m41.437161055s) [noise=true , address=45.91.116.5:4319 , piece_id=BY6BRNZNKKQKZNAVJTSWUYASX57P5X7OCKQ433T2ARRBXUFNBRCA, seg_pos={0 0}, upload=332700]: Uploading piece... Mar 18 20:02:43.840790648 (297h2m41.437561682s) [noise=true , address=45.91.116.12:4259 , piece_id=4LBHNF3CIQG5LYEY7RYJGNK5Q62X2TSOAXOXSNHO727OXL5OMA4Q, seg_pos={0 0}, upload=332700]: Uploading piece... Mar 18 20:02:43.841319562 (297h2m41.438090584s) [noise=true , address=145.79.227.29:4966 , piece_id=ZTSKMRBESX4OIZMMIPXRW752W5KQYX57RPDQ22XXPURQTSCAI2YA, seg_pos={0 0}, upload=332700]: Uploading piece... Mar 18 20:02:43.841816046 (297h2m41.438587078s) [noise=true , address=193.149.164.71:4287 , piece_id=FYNZ3QGUB7ZFMUJAMYPMUHXJDS66CJ2VLXPV7256DFCABPJMEOYQ, seg_pos={0 0}, upload=332700]: Uploading piece... Mar 18 20:02:43.842288870 (297h2m41.439059903s) [noise=true , address=79.127.219.41:4941 , piece_id=CQXOXMCJHWJ7FSIXDVYUJZMBFW5FUUOJNKJJOVLZRSK5D6XZXX5Q, seg_pos={0 0}, upload=332700]: Uploading piece... Mar 18 20:02:43.842956832 (297h2m41.439727861s) [noise=true , address=145.79.227.49:4369 , piece_id=CBYHFRBNKSPUNJHAE6S72YOFACWLZKGZIFV6SXWDYMVSA4FBPKQQ, seg_pos={0 0}, upload=332700]: Uploading piece... Mar 18 20:02:43.844386533 (297h2m41.441157559s) [noise=true , address=77.90.31.10:4533 , piece_id=VB5BKAWDUZFVII4ZY3JRVK67VUULPSAHTJVZMXXVATP3SXAR2XHQ, seg_pos={0 0}, upload=332700]: Uploading piece... Mar 18 20:02:43.844868553 (297h2m41.441639576s) [noise=true , address=145.79.227.37:4582 , piece_id=DLOFMRZRG3BPFN7JF73OM5HUV56FLEH2IDDC47UQWWWOTRVRR4HQ, seg_pos={0 0}, upload=332700]: Uploading piece... Mar 18 20:02:43.903559633 (297h2m41.500330666s) [upload=332700]: Next returned nil segment Mar 18 20:02:43.903578892 (297h2m41.500349925s) [upload=332700]: Waiting for error group managing segments... Mar 18 20:02:43.908218461 (297h2m41.504989474s) [noise=true , address=45.91.116.4:4948 , piece_id=ULVH4YAJIYY5RWWAGH5RTSZ3BMAT6WCYYAJPSMSP25LCLZQIPG2A, seg_pos={0 0}, upload=332700]: Done uploading piece. err: <nil> Mar 18 20:02:43.917679744 (297h2m41.514450753s) [noise=true , address=45.91.116.12:4259 , piece_id=4LBHNF3CIQG5LYEY7RYJGNK5Q62X2TSOAXOXSNHO727OXL5OMA4Q, seg_pos={0 0}, upload=332700]: Done uploading piece. err: <nil> Mar 18 20:02:43.924357408 (297h2m41.521128422s) [noise=true , address=77.90.31.10:4533 , piece_id=VB5BKAWDUZFVII4ZY3JRVK67VUULPSAHTJVZMXXVATP3SXAR2XHQ, seg_pos={0 0}, upload=332700]: Done uploading piece. err: <nil> Mar 18 20:02:43.926248407 (297h2m41.52301943s) [noise=true , address=77.90.31.15:4716 , piece_id=IDQP2MIPCKVS2FGVVGKAIDYTNUIPE56EER5IYZUSOTRD5LZBSXHA, seg_pos={0 0}, upload=332700]: Done uploading piece. err: <nil> Mar 18 20:02:43.927212300 (297h2m41.523983323s) [noise=true , address=45.91.116.5:4319 , piece_id=BY6BRNZNKKQKZNAVJTSWUYASX57P5X7OCKQ433T2ARRBXUFNBRCA, seg_pos={0 0}, upload=332700]: Done uploading piece. err: <nil> Mar 18 20:02:43.927791610 (297h2m41.524562634s) [noise=true , address=45.91.116.13:4164 , piece_id=MNQLVANO3LFKBCJMADD5RULARMRO4VUQY7LT63TU2CCNO765GW7Q, seg_pos={0 0}, upload=332700]: Done uploading piece. err: <nil> Mar 18 20:02:44.028504592 (297h2m41.625275611s) [noise=true , address=79.127.219.41:4941 , piece_id=CQXOXMCJHWJ7FSIXDVYUJZMBFW5FUUOJNKJJOVLZRSK5D6XZXX5Q, seg_pos={0 0}, upload=332700]: Done uploading piece. err: <nil> Mar 18 20:02:44.392266647 (297h2m41.989037671s) [noise=true , address=79.127.205.233:4171 , piece_id=YO42WSHYRB2JJX4RVX653N7U6PYTGIW3KQTNSF4445OTN3UW2TIA, seg_pos={0 0}, upload=332700]: Done uploading piece. err: <nil> Mar 18 20:02:44.429403642 (297h2m42.026174669s) [noise=true , address=193.149.164.66:4974 , piece_id=XWMBTFEWXVVFIR6IYNQYU35QKGVSQJ7YBVSWZ57QSUZ2VSDCJPRA, seg_pos={0 0}, upload=332700]: Done uploading piece. err: <nil> Mar 18 20:02:44.433374828 (297h2m42.030145853s) [noise=true , address=193.149.164.68:4064 , piece_id=HAA3NDB24MJLAI2HK2UDZQZRLDUXEF76ZJF2VBE6FLKRWSB27Y3A, seg_pos={0 0}, upload=332700]: Done uploading piece. err: <nil> Mar 18 20:02:44.444389014 (297h2m42.041160042s) [noise=true , address=193.149.164.71:4287 , piece_id=FYNZ3QGUB7ZFMUJAMYPMUHXJDS66CJ2VLXPV7256DFCABPJMEOYQ, seg_pos={0 0}, upload=332700]: Done uploading piece. err: <nil> Mar 18 20:02:44.578333498 (297h2m42.175104529s) [noise=true , address=79.127.205.237:4227 , piece_id=NO25WIYZTEPBOMGETTLSDOJSPKARMKNLVUWDKNXUKHYGDGF2XC5Q, seg_pos={0 0}, upload=332700]: Done uploading piece. err: <nil> Mar 18 20:02:44.665665140 (297h2m42.262436168s) [noise=true , address=193.149.164.70:4123 , piece_id=VROUAQKRTKZT42ZDDSCIMKFMRODPKFQ5U5DWDJLVI462BVA5YWXA, seg_pos={0 0}, upload=332700]: Done uploading piece. err: <nil> Mar 18 20:03:40.987108256 (297h3m38.58387928s) [noise=true , address=145.79.227.49:4369 , piece_id=CBYHFRBNKSPUNJHAE6S72YOFACWLZKGZIFV6SXWDYMVSA4FBPKQQ, seg_pos={0 0}, upload=332700]: Done uploading piece. err: ecclient: upload failed (node:12nLksGXGvDwuTWfnmkceuwZNDmrF17iNepHPdYkkvZG2QQUwG7, address:145.79.227.49:4369): protocol: rpc: dial tcp 145.79.227.49:4369: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:03:40.987479807 (297h3m38.58425084s) [noise=true , address=145.79.227.29:4966 , piece_id=ZTSKMRBESX4OIZMMIPXRW752W5KQYX57RPDQ22XXPURQTSCAI2YA, seg_pos={0 0}, upload=332700]: Done uploading piece. err: ecclient: upload failed (node:1DLnsYVdudE4mY9dNAQKhBRppUEAHvxbf84KX69451pwxSxzZK, address:145.79.227.29:4966): protocol: rpc: dial tcp 145.79.227.29:4966: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:03:40.987517864 (297h3m38.584288896s) [noise=true , address=145.79.227.34:4924 , piece_id=BNUESPWRZQBX23K7UGNSMF362AWHILNZ43UITBHVGFXIJ4T6V5HQ, seg_pos={0 0}, upload=332700]: Done uploading piece. err: ecclient: upload failed (node:19TbrQjuN1EhBNVtha9EcKPSCAWyttfmHpec6QeF1S5zW6DRZs, address:145.79.227.34:4924): protocol: rpc: dial tcp 145.79.227.34:4924: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:03:40.987933296 (297h3m38.584704329s) [noise=true , address=145.79.227.37:4582 , piece_id=DLOFMRZRG3BPFN7JF73OM5HUV56FLEH2IDDC47UQWWWOTRVRR4HQ, seg_pos={0 0}, upload=332700]: Done uploading piece. err: ecclient: upload failed (node:129o91j97t4V5AjaouCCy2wDuHgwF94iQiJh2h9ux1aFo7AVCSn, address:145.79.227.37:4582): protocol: rpc: dial tcp 145.79.227.37:4582: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:03:40.988167350 (297h3m38.584938376s) [noise=true , address=145.79.227.52:4451 , piece_id=LR6KVRYMDO2HAESG5BFI5RYPNGVARP6O5HEY2UP77YSTVRKBNUXQ, seg_pos={0 0}, upload=332700]: Done uploading piece. err: ecclient: upload failed (node:12bC4P3xBiPFkxLvavGFVbWKnbprCStL6UJgfKSA2Wd2fX7Mbfw, address:145.79.227.52:4451): protocol: rpc: dial tcp 145.79.227.52:4451: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:03:40.988197047 (297h3m38.58496806s) [seg_pos={0 0}, upload=332700]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 18 20:03:41.007727578 (297h3m38.604498608s) [upload=332700]: Done uploading segments... Mar 18 20:03:41.007779207 (297h3m38.60455025s) [upload=332700]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 16.390964179s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 7 out of 60 is failed (threshold=3) [........................XXX..........X...X.........X....X...] duration: 16.389954633s trace: https://jaeger.dev.storj.io/trace/698fb9649d932a68 FAIL (2026-03-18 20:05:15 UTC) operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 20:02:41.019616017 (297h2m38.616387042s) [upload=332700]: Uploading segments... Mar 18 20:02:41.019826191 (297h2m38.616597205s) [upload=332700]: Got next segment. Inline: false Mar 18 20:02:41.019879916 (297h2m38.61665094s) [upload=332700]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 20:02:41.019885584 (297h2m38.616656611s) [upload=332700]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 18 20:02:43.834160391 (297h2m41.430931414s) [seg_pos={0 0}, upload=332700]: Begin upload segment... Mar 18 20:02:43.834230381 (297h2m41.431001396s) [seg_pos={0 0}, upload=332700]: Done begin upload segment. Mar 18 20:02:43.834262428 (297h2m41.431033451s) [noise=true , address=145.79.227.52:4451 , piece_id=LR6KVRYMDO2HAESG5BFI5RYPNGVARP6O5HEY2UP77YSTVRKBNUXQ, seg_pos={0 0}, upload=332700]: Uploading piece... Mar 18 20:02:43.834329472 (297h2m41.431100497s) [noise=true , address=45.91.116.13:4164 , piece_id=MNQLVANO3LFKBCJMADD5RULARMRO4VUQY7LT63TU2CCNO765GW7Q, seg_pos={0 0}, upload=332700]: Uploading piece... Mar 18 20:02:43.834938832 (297h2m41.431709858s) [noise=true , address=77.90.31.15:4716 , piece_id=IDQP2MIPCKVS2FGVVGKAIDYTNUIPE56EER5IYZUSOTRD5LZBSXHA, seg_pos={0 0}, upload=332700]: Uploading piece... Mar 18 20:02:43.835898822 (297h2m41.432669851s) [noise=true , address=79.127.205.237:4227 , piece_id=NO25WIYZTEPBOMGETTLSDOJSPKARMKNLVUWDKNXUKHYGDGF2XC5Q, seg_pos={0 0}, upload=332700]: Uploading piece... Mar 18 20:02:43.836893726 (297h2m41.433664759s) [noise=true , address=193.149.164.68:4064 , piece_id=HAA3NDB24MJLAI2HK2UDZQZRLDUXEF76ZJF2VBE6FLKRWSB27Y3A, seg_pos={0 0}, upload=332700]: Uploading piece... Mar 18 20:02:43.837257949 (297h2m41.434028981s) [noise=true , address=193.149.164.66:4974 , piece_id=XWMBTFEWXVVFIR6IYNQYU35QKGVSQJ7YBVSWZ57QSUZ2VSDCJPRA, seg_pos={0 0}, upload=332700]: Uploading piece... Mar 18 20:02:43.838266356 (297h2m41.435037391s) [noise=true , address=145.79.227.34:4924 , piece_id=BNUESPWRZQBX23K7UGNSMF362AWHILNZ43UITBHVGFXIJ4T6V5HQ, seg_pos={0 0}, upload=332700]: Uploading piece... Mar 18 20:02:43.838688582 (297h2m41.435459605s) [noise=true , address=79.127.205.233:4171 , piece_id=YO42WSHYRB2JJX4RVX653N7U6PYTGIW3KQTNSF4445OTN3UW2TIA, seg_pos={0 0}, upload=332700]: Uploading piece... Mar 18 20:02:43.839263235 (297h2m41.436034258s) [noise=true , address=193.149.164.70:4123 , piece_id=VROUAQKRTKZT42ZDDSCIMKFMRODPKFQ5U5DWDJLVI462BVA5YWXA, seg_pos={0 0}, upload=332700]: Uploading piece... Mar 18 20:02:43.839835928 (297h2m41.436606963s) [noise=true , address=45.91.116.4:4948 , piece_id=ULVH4YAJIYY5RWWAGH5RTSZ3BMAT6WCYYAJPSMSP25LCLZQIPG2A, seg_pos={0 0}, upload=332700]: Uploading piece... Mar 18 20:02:43.840390028 (297h2m41.437161055s) [noise=true , address=45.91.116.5:4319 , piece_id=BY6BRNZNKKQKZNAVJTSWUYASX57P5X7OCKQ433T2ARRBXUFNBRCA, seg_pos={0 0}, upload=332700]: Uploading piece... Mar 18 20:02:43.840790648 (297h2m41.437561682s) [noise=true , address=45.91.116.12:4259 , piece_id=4LBHNF3CIQG5LYEY7RYJGNK5Q62X2TSOAXOXSNHO727OXL5OMA4Q, seg_pos={0 0}, upload=332700]: Uploading piece... Mar 18 20:02:43.841319562 (297h2m41.438090584s) [noise=true , address=145.79.227.29:4966 , piece_id=ZTSKMRBESX4OIZMMIPXRW752W5KQYX57RPDQ22XXPURQTSCAI2YA, seg_pos={0 0}, upload=332700]: Uploading piece... Mar 18 20:02:43.841816046 (297h2m41.438587078s) [noise=true , address=193.149.164.71:4287 , piece_id=FYNZ3QGUB7ZFMUJAMYPMUHXJDS66CJ2VLXPV7256DFCABPJMEOYQ, seg_pos={0 0}, upload=332700]: Uploading piece... Mar 18 20:02:43.842288870 (297h2m41.439059903s) [noise=true , address=79.127.219.41:4941 , piece_id=CQXOXMCJHWJ7FSIXDVYUJZMBFW5FUUOJNKJJOVLZRSK5D6XZXX5Q, seg_pos={0 0}, upload=332700]: Uploading piece... Mar 18 20:02:43.842956832 (297h2m41.439727861s) [noise=true , address=145.79.227.49:4369 , piece_id=CBYHFRBNKSPUNJHAE6S72YOFACWLZKGZIFV6SXWDYMVSA4FBPKQQ, seg_pos={0 0}, upload=332700]: Uploading piece... Mar 18 20:02:43.844386533 (297h2m41.441157559s) [noise=true , address=77.90.31.10:4533 , piece_id=VB5BKAWDUZFVII4ZY3JRVK67VUULPSAHTJVZMXXVATP3SXAR2XHQ, seg_pos={0 0}, upload=332700]: Uploading piece... Mar 18 20:02:43.844868553 (297h2m41.441639576s) [noise=true , address=145.79.227.37:4582 , piece_id=DLOFMRZRG3BPFN7JF73OM5HUV56FLEH2IDDC47UQWWWOTRVRR4HQ, seg_pos={0 0}, upload=332700]: Uploading piece... Mar 18 20:02:43.903559633 (297h2m41.500330666s) [upload=332700]: Next returned nil segment Mar 18 20:02:43.903578892 (297h2m41.500349925s) [upload=332700]: Waiting for error group managing segments... Mar 18 20:02:43.908218461 (297h2m41.504989474s) [noise=true , address=45.91.116.4:4948 , piece_id=ULVH4YAJIYY5RWWAGH5RTSZ3BMAT6WCYYAJPSMSP25LCLZQIPG2A, seg_pos={0 0}, upload=332700]: Done uploading piece. err: <nil> Mar 18 20:02:43.917679744 (297h2m41.514450753s) [noise=true , address=45.91.116.12:4259 , piece_id=4LBHNF3CIQG5LYEY7RYJGNK5Q62X2TSOAXOXSNHO727OXL5OMA4Q, seg_pos={0 0}, upload=332700]: Done uploading piece. err: <nil> Mar 18 20:02:43.924357408 (297h2m41.521128422s) [noise=true , address=77.90.31.10:4533 , piece_id=VB5BKAWDUZFVII4ZY3JRVK67VUULPSAHTJVZMXXVATP3SXAR2XHQ, seg_pos={0 0}, upload=332700]: Done uploading piece. err: <nil> Mar 18 20:02:43.926248407 (297h2m41.52301943s) [noise=true , address=77.90.31.15:4716 , piece_id=IDQP2MIPCKVS2FGVVGKAIDYTNUIPE56EER5IYZUSOTRD5LZBSXHA, seg_pos={0 0}, upload=332700]: Done uploading piece. err: <nil> Mar 18 20:02:43.927212300 (297h2m41.523983323s) [noise=true , address=45.91.116.5:4319 , piece_id=BY6BRNZNKKQKZNAVJTSWUYASX57P5X7OCKQ433T2ARRBXUFNBRCA, seg_pos={0 0}, upload=332700]: Done uploading piece. err: <nil> Mar 18 20:02:43.927791610 (297h2m41.524562634s) [noise=true , address=45.91.116.13:4164 , piece_id=MNQLVANO3LFKBCJMADD5RULARMRO4VUQY7LT63TU2CCNO765GW7Q, seg_pos={0 0}, upload=332700]: Done uploading piece. err: <nil> Mar 18 20:02:44.028504592 (297h2m41.625275611s) [noise=true , address=79.127.219.41:4941 , piece_id=CQXOXMCJHWJ7FSIXDVYUJZMBFW5FUUOJNKJJOVLZRSK5D6XZXX5Q, seg_pos={0 0}, upload=332700]: Done uploading piece. err: <nil> Mar 18 20:02:44.392266647 (297h2m41.989037671s) [noise=true , address=79.127.205.233:4171 , piece_id=YO42WSHYRB2JJX4RVX653N7U6PYTGIW3KQTNSF4445OTN3UW2TIA, seg_pos={0 0}, upload=332700]: Done uploading piece. err: <nil> Mar 18 20:02:44.429403642 (297h2m42.026174669s) [noise=true , address=193.149.164.66:4974 , piece_id=XWMBTFEWXVVFIR6IYNQYU35QKGVSQJ7YBVSWZ57QSUZ2VSDCJPRA, seg_pos={0 0}, upload=332700]: Done uploading piece. err: <nil> Mar 18 20:02:44.433374828 (297h2m42.030145853s) [noise=true , address=193.149.164.68:4064 , piece_id=HAA3NDB24MJLAI2HK2UDZQZRLDUXEF76ZJF2VBE6FLKRWSB27Y3A, seg_pos={0 0}, upload=332700]: Done uploading piece. err: <nil> Mar 18 20:02:44.444389014 (297h2m42.041160042s) [noise=true , address=193.149.164.71:4287 , piece_id=FYNZ3QGUB7ZFMUJAMYPMUHXJDS66CJ2VLXPV7256DFCABPJMEOYQ, seg_pos={0 0}, upload=332700]: Done uploading piece. err: <nil> Mar 18 20:02:44.578333498 (297h2m42.175104529s) [noise=true , address=79.127.205.237:4227 , piece_id=NO25WIYZTEPBOMGETTLSDOJSPKARMKNLVUWDKNXUKHYGDGF2XC5Q, seg_pos={0 0}, upload=332700]: Done uploading piece. err: <nil> Mar 18 20:02:44.665665140 (297h2m42.262436168s) [noise=true , address=193.149.164.70:4123 , piece_id=VROUAQKRTKZT42ZDDSCIMKFMRODPKFQ5U5DWDJLVI462BVA5YWXA, seg_pos={0 0}, upload=332700]: Done uploading piece. err: <nil> Mar 18 20:03:40.987108256 (297h3m38.58387928s) [noise=true , address=145.79.227.49:4369 , piece_id=CBYHFRBNKSPUNJHAE6S72YOFACWLZKGZIFV6SXWDYMVSA4FBPKQQ, seg_pos={0 0}, upload=332700]: Done uploading piece. err: ecclient: upload failed (node:12nLksGXGvDwuTWfnmkceuwZNDmrF17iNepHPdYkkvZG2QQUwG7, address:145.79.227.49:4369): protocol: rpc: dial tcp 145.79.227.49:4369: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:03:40.987479807 (297h3m38.58425084s) [noise=true , address=145.79.227.29:4966 , piece_id=ZTSKMRBESX4OIZMMIPXRW752W5KQYX57RPDQ22XXPURQTSCAI2YA, seg_pos={0 0}, upload=332700]: Done uploading piece. err: ecclient: upload failed (node:1DLnsYVdudE4mY9dNAQKhBRppUEAHvxbf84KX69451pwxSxzZK, address:145.79.227.29:4966): protocol: rpc: dial tcp 145.79.227.29:4966: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:03:40.987517864 (297h3m38.584288896s) [noise=true , address=145.79.227.34:4924 , piece_id=BNUESPWRZQBX23K7UGNSMF362AWHILNZ43UITBHVGFXIJ4T6V5HQ, seg_pos={0 0}, upload=332700]: Done uploading piece. err: ecclient: upload failed (node:19TbrQjuN1EhBNVtha9EcKPSCAWyttfmHpec6QeF1S5zW6DRZs, address:145.79.227.34:4924): protocol: rpc: dial tcp 145.79.227.34:4924: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:03:40.987933296 (297h3m38.584704329s) [noise=true , address=145.79.227.37:4582 , piece_id=DLOFMRZRG3BPFN7JF73OM5HUV56FLEH2IDDC47UQWWWOTRVRR4HQ, seg_pos={0 0}, upload=332700]: Done uploading piece. err: ecclient: upload failed (node:129o91j97t4V5AjaouCCy2wDuHgwF94iQiJh2h9ux1aFo7AVCSn, address:145.79.227.37:4582): protocol: rpc: dial tcp 145.79.227.37:4582: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:03:40.988167350 (297h3m38.584938376s) [noise=true , address=145.79.227.52:4451 , piece_id=LR6KVRYMDO2HAESG5BFI5RYPNGVARP6O5HEY2UP77YSTVRKBNUXQ, seg_pos={0 0}, upload=332700]: Done uploading piece. err: ecclient: upload failed (node:12bC4P3xBiPFkxLvavGFVbWKnbprCStL6UJgfKSA2Wd2fX7Mbfw, address:145.79.227.52:4451): protocol: rpc: dial tcp 145.79.227.52:4451: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:03:40.988197047 (297h3m38.58496806s) [seg_pos={0 0}, upload=332700]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 18 20:03:41.007727578 (297h3m38.604498608s) [upload=332700]: Done uploading segments... Mar 18 20:03:41.007779207 (297h3m38.60455025s) [upload=332700]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 16.390067637s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 40 out of 60 is failed (threshold=3) [...................XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.XXX] duration: 30.038512763s trace: https://jaeger.dev.storj.io/trace/5b0f747cb1d1940 FAIL (2026-03-18 20:05:13 UTC) operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 9) ======LOGS====== Mar 18 20:04:43.234379658 (297h4m40.831150682s) [upload=332739]: Starting upload Mar 18 20:04:43.234514357 (297h4m40.831285372s) [upload=332739]: Done starting upload Mar 18 20:04:43.234770926 (297h4m40.831541951s) [upload=332739]: Uploading segments... Mar 18 20:04:43.234845492 (297h4m40.831616527s) [upload=332739]: Got next segment. Inline: false Mar 18 20:04:43.234894018 (297h4m40.831665048s) [upload=332739]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 20:04:43.234900244 (297h4m40.831671275s) [upload=332739]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 18 20:04:46.305008694 (297h4m43.901779721s) [seg_pos={0 0}, upload=332739]: Begin upload segment... Mar 18 20:04:46.305120280 (297h4m43.901891293s) [seg_pos={0 0}, upload=332739]: Done begin upload segment. Mar 18 20:04:46.305158251 (297h4m43.901929264s) [noise=true , address=79.127.219.45:4855 , piece_id=5IWJJQGJUSZ3CPA3BTRZO7KQCPWRUPAE3TSWUMJJIM677IGO3XHQ, seg_pos={0 0}, upload=332739]: Uploading piece... Mar 18 20:04:46.305177128 (297h4m43.901948157s) [noise=true , address=79.127.219.46:4894 , piece_id=YQFFTQXF7J324ZJNXHQQAGMHHOVH4MPZJJICQLQNSF3IAQIYOTSA, seg_pos={0 0}, upload=332739]: Uploading piece... Mar 18 20:04:46.306133788 (297h4m43.902904804s) [noise=true , address=145.79.227.4:4187 , piece_id=Q7G7UMVTSVRUPDUQRLAMUWAJZQYRLV5VYPHGVBR5ZYCYFEDAONNA, seg_pos={0 0}, upload=332739]: Uploading piece... Mar 18 20:04:46.306564638 (297h4m43.90333567s) [noise=true , address=145.79.227.15:4878 , piece_id=C3KOXZFMFPP57V3VN2UAIBFSFIAH7KE75AS7PCABGXRFEIGV7NSA, seg_pos={0 0}, upload=332739]: Uploading piece... Mar 18 20:04:46.307193485 (297h4m43.903964513s) [noise=true , address=145.79.227.43:4161 , piece_id=CAAZUUUGJT2WCGE5O2MOSATKOEOLCCQ5BFY5T34BD2CO2HHEX3RA, seg_pos={0 0}, upload=332739]: Uploading piece... Mar 18 20:04:46.307902859 (297h4m43.904673886s) [noise=true , address=45.91.116.14:4193 , piece_id=GBW5Q6W2PZFND5YHOZXVND7LC2ZX6VVRWY7ER323FSLACTLCNJ3A, seg_pos={0 0}, upload=332739]: Uploading piece... Mar 18 20:04:46.308536118 (297h4m43.905307152s) [noise=true , address=45.91.116.8:4080 , piece_id=REQE5APRO3HJBZH6YJPEBRZKIQXH66QMRQPDTYNKIFBBC5HB67JQ, seg_pos={0 0}, upload=332739]: Uploading piece... Mar 18 20:04:46.308697053 (297h4m43.90546808s) [noise=true , address=45.91.116.13:4188 , piece_id=GIARN34QZRUYXVDSOTBRHDKRUBVINI2NR6P4OVOKMYKL6OSC45HA, seg_pos={0 0}, upload=332739]: Uploading piece... Mar 18 20:04:46.309074759 (297h4m43.90584579s) [noise=true , address=145.79.227.34:4909 , piece_id=3BEBSOA7O3VDPZ6NIWBMC2GFN3YRENXIZAKB767XNDDURY65PP6A, seg_pos={0 0}, upload=332739]: Uploading piece... Mar 18 20:04:46.315117785 (297h4m43.911888802s) [noise=true , address=145.79.227.41:4088 , piece_id=44URON2V56SHZQJ7ZFI32IRK23I4KUW4A5CAR3SXD4VLQWFOMISQ, seg_pos={0 0}, upload=332739]: Uploading piece... Mar 18 20:04:46.317084390 (297h4m43.913855406s) [noise=true , address=145.79.227.44:4197 , piece_id=3BS5H7VQB2R7SE5BUUG6KJSHB4ODLTA2R4FDGFR5ANV4L2UP3DOA, seg_pos={0 0}, upload=332739]: Uploading piece... Mar 18 20:04:46.317246758 (297h4m43.914017774s) [noise=true , address=45.91.116.4:4941 , piece_id=UBPMA2YWYT4FE67ABONUQLODRKAJ5N2HYHADCZ2HQWJ7HPFVVIKA, seg_pos={0 0}, upload=332739]: Uploading piece... Mar 18 20:04:46.317917550 (297h4m43.914688566s) [noise=true , address=45.91.116.5:4298 , piece_id=ECUBRTF37AWUWRWRNNDN4FJH5BHO7YSESAASFCJTCSWWMSCRS7EA, seg_pos={0 0}, upload=332739]: Uploading piece... Mar 18 20:04:46.319528805 (297h4m43.916299814s) [noise=true , address=45.91.116.10:4081 , piece_id=BQ4PUN5OGWGFFOIWNUES5JQJGRAK3FMWXSZ4VRQOFMVDLTM2U3QA, seg_pos={0 0}, upload=332739]: Uploading piece... Mar 18 20:04:46.320610417 (297h4m43.917381435s) [noise=true , address=77.90.31.11:4558 , piece_id=KAKDGD2BSMFOJBSZJTFG2V6Y36MDO2ZK4RMOEJUL4T6GQTU3JEOQ, seg_pos={0 0}, upload=332739]: Uploading piece... Mar 18 20:04:46.322784861 (297h4m43.919555886s) [noise=true , address=145.79.227.2:4118 , piece_id=47OWZUWJNBWDYM2SZJZDRPCYKL5ISSBOMU33DGW4BNMN6B7RLFEA, seg_pos={0 0}, upload=332739]: Uploading piece... Mar 18 20:04:46.390839310 (297h4m43.987610334s) [upload=332739]: Next returned nil segment Mar 18 20:04:46.390854010 (297h4m43.987625034s) [upload=332739]: Waiting for error group managing segments... Mar 18 20:04:46.395927923 (297h4m43.992698936s) [noise=true , address=45.91.116.8:4080 , piece_id=REQE5APRO3HJBZH6YJPEBRZKIQXH66QMRQPDTYNKIFBBC5HB67JQ, seg_pos={0 0}, upload=332739]: Done uploading piece. err: <nil> Mar 18 20:04:46.396043608 (297h4m43.992814635s) [noise=true , address=77.90.31.11:4558 , piece_id=KAKDGD2BSMFOJBSZJTFG2V6Y36MDO2ZK4RMOEJUL4T6GQTU3JEOQ, seg_pos={0 0}, upload=332739]: Done uploading piece. err: <nil> Mar 18 20:04:46.396543561 (297h4m43.993314584s) [noise=true , address=79.127.219.46:4894 , piece_id=YQFFTQXF7J324ZJNXHQQAGMHHOVH4MPZJJICQLQNSF3IAQIYOTSA, seg_pos={0 0}, upload=332739]: Done uploading piece. err: <nil> Mar 18 20:04:46.396590475 (297h4m43.993361488s) [noise=true , address=45.91.116.14:4193 , piece_id=GBW5Q6W2PZFND5YHOZXVND7LC2ZX6VVRWY7ER323FSLACTLCNJ3A, seg_pos={0 0}, upload=332739]: Done uploading piece. err: <nil> Mar 18 20:04:46.398006800 (297h4m43.994777814s) [noise=true , address=45.91.116.4:4941 , piece_id=UBPMA2YWYT4FE67ABONUQLODRKAJ5N2HYHADCZ2HQWJ7HPFVVIKA, seg_pos={0 0}, upload=332739]: Done uploading piece. err: <nil> Mar 18 20:04:46.398376069 (297h4m43.995147081s) [noise=true , address=45.91.116.13:4188 , piece_id=GIARN34QZRUYXVDSOTBRHDKRUBVINI2NR6P4OVOKMYKL6OSC45HA, seg_pos={0 0}, upload=332739]: Done uploading piece. err: <nil> Mar 18 20:04:46.411441727 (297h4m44.008212756s) [noise=true , address=79.127.219.45:4855 , piece_id=5IWJJQGJUSZ3CPA3BTRZO7KQCPWRUPAE3TSWUMJJIM677IGO3XHQ, seg_pos={0 0}, upload=332739]: Done uploading piece. err: <nil> Mar 18 20:04:46.413072093 (297h4m44.009843105s) [noise=true , address=45.91.116.5:4298 , piece_id=ECUBRTF37AWUWRWRNNDN4FJH5BHO7YSESAASFCJTCSWWMSCRS7EA, seg_pos={0 0}, upload=332739]: Done uploading piece. err: <nil> Mar 18 20:04:46.415190026 (297h4m44.011961036s) [noise=true , address=45.91.116.10:4081 , piece_id=BQ4PUN5OGWGFFOIWNUES5JQJGRAK3FMWXSZ4VRQOFMVDLTM2U3QA, seg_pos={0 0}, upload=332739]: Done uploading piece. err: <nil> Mar 18 20:05:13.201917265 (297h5m10.798688298s) [noise=true , address=145.79.227.15:4878 , piece_id=C3KOXZFMFPP57V3VN2UAIBFSFIAH7KE75AS7PCABGXRFEIGV7NSA, seg_pos={0 0}, upload=332739]: Done uploading piece. err: ecclient: upload failed (node:126ZeAKnpjpxa3MgCBU3KZKBDdn8MfZgiixE4zR5fZpPwaQj9NC, address:145.79.227.15:4878): protocol: rpc: dial tcp 145.79.227.15:4878: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:05:13.202094819 (297h5m10.798865855s) [noise=true , address=145.79.227.2:4118 , piece_id=47OWZUWJNBWDYM2SZJZDRPCYKL5ISSBOMU33DGW4BNMN6B7RLFEA, seg_pos={0 0}, upload=332739]: Done uploading piece. err: ecclient: upload failed (node:12ty5LHXwLmHC7nRPmVfj2wVEif6JQAQQ5Z5218vv1fKK5sD3zE, address:145.79.227.2:4118): protocol: rpc: dial tcp 145.79.227.2:4118: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:05:13.202179825 (297h5m10.798950855s) [noise=true , address=145.79.227.41:4088 , piece_id=44URON2V56SHZQJ7ZFI32IRK23I4KUW4A5CAR3SXD4VLQWFOMISQ, seg_pos={0 0}, upload=332739]: Done uploading piece. err: ecclient: upload failed (node:1284h6cPNVXJYMkSekGQVw8b7p89tkhpmgYLxTTYUDu2J7Wadoj, address:145.79.227.41:4088): protocol: rpc: dial tcp 145.79.227.41:4088: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:05:13.202393357 (297h5m10.799164391s) [noise=true , address=145.79.227.4:4187 , piece_id=Q7G7UMVTSVRUPDUQRLAMUWAJZQYRLV5VYPHGVBR5ZYCYFEDAONNA, seg_pos={0 0}, upload=332739]: Done uploading piece. err: ecclient: upload failed (node:12cQxBB2L2CyymYx3xWp1ALj6p8QWr4A34sqvtG5zyeT7sVWv7x, address:145.79.227.4:4187): protocol: rpc: dial tcp 145.79.227.4:4187: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:05:13.202457005 (297h5m10.79922803s) [noise=true , address=145.79.227.44:4197 , piece_id=3BS5H7VQB2R7SE5BUUG6KJSHB4ODLTA2R4FDGFR5ANV4L2UP3DOA, seg_pos={0 0}, upload=332739]: Done uploading piece. err: ecclient: upload failed (node:12niHLkT5pbmYSbarHZGeeCRtwjZRXtvEtHMGAb7ja4a7Xuuvfi, address:145.79.227.44:4197): protocol: rpc: dial tcp 145.79.227.44:4197: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:05:13.202577343 (297h5m10.799348409s) [noise=true , address=145.79.227.43:4161 , piece_id=CAAZUUUGJT2WCGE5O2MOSATKOEOLCCQ5BFY5T34BD2CO2HHEX3RA, seg_pos={0 0}, upload=332739]: Done uploading piece. err: ecclient: upload failed (node:12pyhMcYjL5b9posZurgKXBnCHaMhaEjL3q3YWyc3aWkcYwbvTD, address:145.79.227.43:4161): protocol: context deadline exceeded; context deadline exceeded Mar 18 20:05:13.202883680 (297h5m10.799654714s) [noise=true , address=145.79.227.34:4909 , piece_id=3BEBSOA7O3VDPZ6NIWBMC2GFN3YRENXIZAKB767XNDDURY65PP6A, seg_pos={0 0}, upload=332739]: Done uploading piece. err: ecclient: upload failed (node:14Pqp5cbhmd44QnikAuD5TgsKFDR3gB2qtLuVSroaWVv4U2cDh, address:145.79.227.34:4909): protocol: context deadline exceeded; context deadline exceeded Mar 18 20:05:13.202915567 (297h5m10.799686582s) [seg_pos={0 0}, upload=332739]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Mar 18 20:05:13.236347420 (297h5m10.833118444s) [upload=332739]: Done uploading segments... Mar 18 20:05:13.236382958 (297h5m10.833153982s) [upload=332739]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Duration: 30.038541666s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 8 out of 60 is failed (threshold=3) [......................XXXXXX.XX.............................] duration: 16.269942278s trace: https://jaeger.dev.storj.io/trace/4b4d10a8051ad01b FAIL (2026-03-18 20:05:11 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 16.26997891s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 48 out of 60 is failed (threshold=6) [........XXXXXXXXX.XXX...XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 5.001058325s trace: https://jaeger.dev.storj.io/trace/9a0c0d8fe66fe7c FAIL (2026-03-18 20:05:01 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.00108084s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 6 out of 60 is failed (threshold=3) [..........................X..........XX..X....X........X....] duration: 17.612594111s trace: https://jaeger.dev.storj.io/trace/589419efc52e5e2b FAIL (2026-03-18 20:05:00 UTC) operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== Mar 18 20:00:59.343084966 (297h0m56.939855989s) [upload=332666]: Starting upload Mar 18 20:00:59.343175518 (297h0m56.939946548s) [upload=332666]: Done starting upload Mar 18 20:00:59.343565585 (297h0m56.940336612s) [upload=332666]: Uploading segments... Mar 18 20:00:59.344256647 (297h0m56.941027676s) [upload=332666]: Got next segment. Inline: false Mar 18 20:00:59.344352215 (297h0m56.941123241s) [upload=332666]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 20:00:59.344358706 (297h0m56.941129732s) [upload=332666]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 18 20:01:02.340259299 (297h0m59.937030329s) [seg_pos={0 0}, upload=332666]: Begin upload segment... Mar 18 20:01:02.340368743 (297h0m59.937139767s) [seg_pos={0 0}, upload=332666]: Done begin upload segment. Mar 18 20:01:02.340479563 (297h0m59.93725059s) [noise=true , address=45.91.116.8:4061 , piece_id=6XFVKT4AAO7QN457OQMRKPI6R24TJTAN47UXLEVOEFQVOS6M6C3Q, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.340523837 (297h0m59.93729486s) [noise=true , address=79.127.205.233:4166 , piece_id=SFGK5YCWHFRVXF6QNHYP2CYJCZLSTQ76OYRVJPWWXQTF2KFSFJKA, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.341468438 (297h0m59.938239447s) [noise=true , address=45.91.116.4:4961 , piece_id=ZMUGQECUVPVHDNKN75XPBET5M2HFXXK4WMTP4V5KEK63JVZCX5EA, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.341520025 (297h0m59.938291058s) [noise=true , address=167.17.72.71:4815 , piece_id=CXGTYNNBILN3TT7I6Z7UI75F2ZCS4QUTLS6KC4FJ5IHS3XOYWDJQ, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.342232992 (297h0m59.939004023s) [noise=true , address=145.79.227.36:4524 , piece_id=3TAV3P55ITTHBAZEUUUTOAHJRGQXJXXVGZJP6AVR2ANVNOR3S54Q, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.342917368 (297h0m59.939688398s) [noise=true , address=193.149.164.71:4297 , piece_id=4IJPXLPCX3LKFOBOSRHK2DTNRKH67K7FXJUKQ3JPRWHS47ZFEIDA, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.343244957 (297h0m59.940015982s) [noise=true , address=145.79.227.33:4892 , piece_id=DQ6ALC3VOAGFIJI3IKR6I7QFYM22C66QWAAXMH6NXI2FIQEZNQYA, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.344270079 (297h0m59.941041117s) [noise=true , address=145.79.227.21:4497 , piece_id=SP6EKJB7W4QXF7OIBVKEFWCPOZFL5IRIEDVAGS2M4QJSO34CUVRQ, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.345025057 (297h0m59.941796084s) [noise=true , address=77.90.31.17:4793 , piece_id=2U4IBUA2GGUGDBXQGIGQKUK4ZR3L7CDKC6MTGIKAUME4F7MFZRLA, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.345115802 (297h0m59.941886836s) [noise=true , address=167.17.72.69:4731 , piece_id=VOEVIKKFJMC2Y3TPQIDMYUP37NLFU7KPRH4HRRSMQLWUKTSLOH2Q, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.345690600 (297h0m59.942461634s) [noise=true , address=77.90.31.13:4648 , piece_id=BPMB5CFOYHFT3BS4ORR6JF6BNFC2AOAJZOSPMG67NAMLGGP5XVEQ, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.346311592 (297h0m59.943082623s) [noise=true , address=79.127.205.234:4202 , piece_id=YY4LYWX26O6RIOBONESNKTUZEFZCRL4UUX3NEBQ53AJV4IEDG5MQ, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.346791142 (297h0m59.943562166s) [noise=true , address=45.91.116.10:4110 , piece_id=UYLMDCWJEJOHGICA37KNH7G5AQXNIPRH5NRYLBUUMGDRVSLTEYHQ, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.347094999 (297h0m59.94386607s) [noise=true , address=145.79.227.17:4987 , piece_id=PT3TWVB65JZFEWVH4IUZEU2N53FQ57QBRWVLWAUFP4UDLY3LWNLQ, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.347781281 (297h0m59.944552305s) [noise=true , address=79.127.219.46:4919 , piece_id=KK6HRIVBMYLIHRHGPQH3AWMNSVHTTIBESHDMJZA72T446RX6QKRQ, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.348263123 (297h0m59.945034137s) [noise=true , address=193.149.164.68:4033 , piece_id=JZPQKPYGO3W3RZZBITQ73X2XEPIBBSLGRMWIVOM4Z3GTV5UFKRTQ, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.348915505 (297h0m59.945686536s) [noise=true , address=145.79.227.7:4312 , piece_id=2FAL6E3RMBTBHZBXAJJVH2N5VGF5VYI3KNZXGNWCMWGBBPAUO2NQ, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.349366298 (297h0m59.946137337s) [noise=true , address=193.149.164.67:4007 , piece_id=HURQGH67I5B6V4YUV3PRZL3KSKN3XTSK3UMCHZRNT6L3W3PXXHPQ, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.429834098 (297h1m0.026605126s) [upload=332666]: Next returned nil segment Mar 18 20:01:02.429853411 (297h1m0.026624442s) [upload=332666]: Waiting for error group managing segments... Mar 18 20:01:02.446867680 (297h1m0.043638704s) [noise=true , address=45.91.116.10:4110 , piece_id=UYLMDCWJEJOHGICA37KNH7G5AQXNIPRH5NRYLBUUMGDRVSLTEYHQ, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil> Mar 18 20:01:02.451665165 (297h1m0.048436196s) [noise=true , address=45.91.116.4:4961 , piece_id=ZMUGQECUVPVHDNKN75XPBET5M2HFXXK4WMTP4V5KEK63JVZCX5EA, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil> Mar 18 20:01:02.452538886 (297h1m0.049309913s) [noise=true , address=77.90.31.17:4793 , piece_id=2U4IBUA2GGUGDBXQGIGQKUK4ZR3L7CDKC6MTGIKAUME4F7MFZRLA, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil> Mar 18 20:01:02.455597040 (297h1m0.052368067s) [noise=true , address=45.91.116.8:4061 , piece_id=6XFVKT4AAO7QN457OQMRKPI6R24TJTAN47UXLEVOEFQVOS6M6C3Q, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil> Mar 18 20:01:02.457848739 (297h1m0.054619766s) [noise=true , address=77.90.31.13:4648 , piece_id=BPMB5CFOYHFT3BS4ORR6JF6BNFC2AOAJZOSPMG67NAMLGGP5XVEQ, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil> Mar 18 20:01:02.461479709 (297h1m0.058250746s) [noise=true , address=79.127.219.46:4919 , piece_id=KK6HRIVBMYLIHRHGPQH3AWMNSVHTTIBESHDMJZA72T446RX6QKRQ, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil> Mar 18 20:01:02.676859466 (297h1m0.27363049s) [noise=true , address=167.17.72.69:4731 , piece_id=VOEVIKKFJMC2Y3TPQIDMYUP37NLFU7KPRH4HRRSMQLWUKTSLOH2Q, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil> Mar 18 20:01:02.909860743 (297h1m0.506631766s) [noise=true , address=79.127.205.234:4202 , piece_id=YY4LYWX26O6RIOBONESNKTUZEFZCRL4UUX3NEBQ53AJV4IEDG5MQ, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil> Mar 18 20:01:02.930427742 (297h1m0.527198754s) [noise=true , address=193.149.164.71:4297 , piece_id=4IJPXLPCX3LKFOBOSRHK2DTNRKH67K7FXJUKQ3JPRWHS47ZFEIDA, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil> Mar 18 20:01:02.944696889 (297h1m0.541467914s) [noise=true , address=193.149.164.67:4007 , piece_id=HURQGH67I5B6V4YUV3PRZL3KSKN3XTSK3UMCHZRNT6L3W3PXXHPQ, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil> Mar 18 20:01:02.951319183 (297h1m0.548090212s) [noise=true , address=193.149.164.68:4033 , piece_id=JZPQKPYGO3W3RZZBITQ73X2XEPIBBSLGRMWIVOM4Z3GTV5UFKRTQ, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil> Mar 18 20:01:02.975619436 (297h1m0.572390467s) [noise=true , address=79.127.205.233:4166 , piece_id=SFGK5YCWHFRVXF6QNHYP2CYJCZLSTQ76OYRVJPWWXQTF2KFSFJKA, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil> Mar 18 20:01:03.819918528 (297h1m1.416689564s) [noise=true , address=167.17.72.71:4815 , piece_id=CXGTYNNBILN3TT7I6Z7UI75F2ZCS4QUTLS6KC4FJ5IHS3XOYWDJQ, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil> Mar 18 20:01:59.307873420 (297h1m56.904644452s) [noise=true , address=145.79.227.7:4312 , piece_id=2FAL6E3RMBTBHZBXAJJVH2N5VGF5VYI3KNZXGNWCMWGBBPAUO2NQ, seg_pos={0 0}, upload=332666]: Done uploading piece. err: ecclient: upload failed (node:12AVp3qn5uwLPqYaZEwzcgoGMJhNga1BUh6bK5k2bxjWgZLQiiC, address:145.79.227.7:4312): protocol: rpc: dial tcp 145.79.227.7:4312: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:01:59.308026092 (297h1m56.904797124s) [noise=true , address=145.79.227.17:4987 , piece_id=PT3TWVB65JZFEWVH4IUZEU2N53FQ57QBRWVLWAUFP4UDLY3LWNLQ, seg_pos={0 0}, upload=332666]: Done uploading piece. err: ecclient: upload failed (node:1XsPVNGYUhKnXkQRfVfjyBRzyvhShSdsrnzXtwe4Xtf5MaHMvH, address:145.79.227.17:4987): protocol: rpc: dial tcp 145.79.227.17:4987: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:01:59.308195864 (297h1m56.904966894s) [noise=true , address=145.79.227.36:4524 , piece_id=3TAV3P55ITTHBAZEUUUTOAHJRGQXJXXVGZJP6AVR2ANVNOR3S54Q, seg_pos={0 0}, upload=332666]: Done uploading piece. err: ecclient: upload failed (node:1xfbCzjSwFDRrR1gPggeN8tNCLbRRWDVzCJ6mqhZsbVSe8d4s4, address:145.79.227.36:4524): protocol: rpc: dial tcp 145.79.227.36:4524: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:01:59.308316037 (297h1m56.90508706s) [noise=true , address=145.79.227.33:4892 , piece_id=DQ6ALC3VOAGFIJI3IKR6I7QFYM22C66QWAAXMH6NXI2FIQEZNQYA, seg_pos={0 0}, upload=332666]: Done uploading piece. err: ecclient: upload failed (node:12RHwFxcUjPsn12uHEAH3fBDJKv1v1tM8mRJXoFrZrG5djdswF5, address:145.79.227.33:4892): protocol: rpc: dial tcp 145.79.227.33:4892: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:01:59.308404403 (297h1m56.905175433s) [noise=true , address=145.79.227.21:4497 , piece_id=SP6EKJB7W4QXF7OIBVKEFWCPOZFL5IRIEDVAGS2M4QJSO34CUVRQ, seg_pos={0 0}, upload=332666]: Done uploading piece. err: ecclient: upload failed (node:12ZBe4we7VeFdthqRxyy1V9uQrA4r8ouNX68xqsk5bD26dkf2Gd, address:145.79.227.21:4497): protocol: context deadline exceeded; context deadline exceeded Mar 18 20:01:59.308437457 (297h1m56.90520847s) [seg_pos={0 0}, upload=332666]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 18 20:01:59.338288467 (297h1m56.93505949s) [upload=332666]: Done uploading segments... Mar 18 20:01:59.338335497 (297h1m56.935106523s) [upload=332666]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 17.612624533s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 39 out of 60 is failed (threshold=3) [....................XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.XX] duration: 30.02853497s trace: https://jaeger.dev.storj.io/trace/5bf3205ac7efb06 FAIL (2026-03-18 20:04:42 UTC) operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== Mar 18 20:04:12.773762542 (297h4m10.370533565s) [upload=332730]: Starting upload Mar 18 20:04:12.773839186 (297h4m10.370610202s) [upload=332730]: Done starting upload Mar 18 20:04:12.774141773 (297h4m10.370912789s) [upload=332730]: Uploading segments... Mar 18 20:04:12.774266255 (297h4m10.371037279s) [upload=332730]: Got next segment. Inline: false Mar 18 20:04:12.774342243 (297h4m10.371113275s) [upload=332730]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 20:04:12.774349964 (297h4m10.371120988s) [upload=332730]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 18 20:04:15.793968129 (297h4m13.390739155s) [seg_pos={0 0}, upload=332730]: Begin upload segment... Mar 18 20:04:15.794089989 (297h4m13.390861002s) [seg_pos={0 0}, upload=332730]: Done begin upload segment. Mar 18 20:04:15.794165504 (297h4m13.390936535s) [noise=true , address=145.79.227.4:4189 , piece_id=FFEAVSH2EK7YT3ET5C6Y56XAMRITYHQRFYLOC3WTGWI74OMOUT4Q, seg_pos={0 0}, upload=332730]: Uploading piece... Mar 18 20:04:15.794803278 (297h4m13.391574301s) [noise=true , address=77.90.31.18:4805 , piece_id=5L7MKDVCJWCG4KQ52CKJV5SQS2JZI5BGQRW4QXBJQV6QVXEB6BZA, seg_pos={0 0}, upload=332730]: Uploading piece... Mar 18 20:04:15.800788638 (297h4m13.397559671s) [noise=true , address=145.79.227.9:4369 , piece_id=GC6OCZ7CXOKBSQYFRGM7C3BX5WMIQ5MDIALFAPN2GNKYF4YIN2MQ, seg_pos={0 0}, upload=332730]: Uploading piece... Mar 18 20:04:15.802852907 (297h4m13.399623935s) [noise=true , address=77.90.31.11:4555 , piece_id=23M5BFJQKODZYDFM7SEYR5R4GBVDKLRUUBFM6CD7WHOZMLHTVPNQ, seg_pos={0 0}, upload=332730]: Uploading piece... Mar 18 20:04:15.803161757 (297h4m13.399932783s) [noise=true , address=45.91.116.13:4188 , piece_id=3DGAL7BTW2LROY5GX6WR2FTZEWGJNJ7UOVXOZBHWMLXTVBUTEJZA, seg_pos={0 0}, upload=332730]: Uploading piece... Mar 18 20:04:15.804603133 (297h4m13.401374166s) [noise=true , address=45.91.116.5:4297 , piece_id=2XGIQ7VHVOS6LZ3SSIK5XL2I4ACDPJ57BXKQCFPLZO6HI3ADTHLA, seg_pos={0 0}, upload=332730]: Uploading piece... Mar 18 20:04:15.805736701 (297h4m13.40250774s) [noise=true , address=145.79.227.51:4412 , piece_id=2RAPHKERYTBMLWB4IUWAT7NUSWR7EGOYEEOARZIQZIBQEYRJSYGA, seg_pos={0 0}, upload=332730]: Uploading piece... Mar 18 20:04:15.807683619 (297h4m13.404454648s) [noise=true , address=145.79.227.35:4482 , piece_id=X3UQEIIBUQ5263YMPQLGIRW6EHOOIB476XNJDCCSC4HBYEGKUYTQ, seg_pos={0 0}, upload=332730]: Uploading piece... Mar 18 20:04:15.809131179 (297h4m13.405902211s) [noise=true , address=145.79.227.43:4158 , piece_id=HQLX5SJEGF4IQ6GJIWZUOYFP7TL7IIB7MRU4IDYOA2S2JFS22VFA, seg_pos={0 0}, upload=332730]: Uploading piece... Mar 18 20:04:15.811051943 (297h4m13.407822974s) [noise=true , address=77.90.31.17:4769 , piece_id=L3UU2YI7SO5GJR2YXXLC2HW356VLG2GB4SOWBTSY27AO7EAMD62Q, seg_pos={0 0}, upload=332730]: Uploading piece... Mar 18 20:04:15.811766505 (297h4m13.408537537s) [noise=true , address=45.91.116.11:4121 , piece_id=EGOJ7I4GYLTLYZTLVOXK2YFUJGPB4L2T2EJ3XFNQQYNSGVXLGXEA, seg_pos={0 0}, upload=332730]: Uploading piece... Mar 18 20:04:15.812375245 (297h4m13.409146271s) [noise=true , address=45.91.116.12:4260 , piece_id=JCZ5KUJ7QBW5KIRZ7UHOCOTLTVQP4TLG4IYHAGQDHWMGDFO6D5GQ, seg_pos={0 0}, upload=332730]: Uploading piece... Mar 18 20:04:15.813116066 (297h4m13.409887102s) [noise=true , address=145.79.227.2:4119 , piece_id=3IGMZRN7TOTUQR6BCL6Y4UQCR4KIEFLHHGW37QXQRLZ2JTWOKAGQ, seg_pos={0 0}, upload=332730]: Uploading piece... Mar 18 20:04:15.813250180 (297h4m13.410021212s) [noise=true , address=45.91.116.9:4265 , piece_id=ODQG6YJNSJDMEGZR3DIDE2LV2VG4AZXR74OUROP4YURJ7CLGAJDA, seg_pos={0 0}, upload=332730]: Uploading piece... Mar 18 20:04:15.816392283 (297h4m13.413163315s) [noise=true , address=45.91.116.14:4191 , piece_id=ZF7KWZPHRJDVABY2OMX5H5G6CATYHPD2T32AKEWQ5653HMGC77FA, seg_pos={0 0}, upload=332730]: Uploading piece... Mar 18 20:04:15.818487545 (297h4m13.415258569s) [noise=true , address=79.127.219.46:4894 , piece_id=U33HRDMD5CQSBFGIEAQJTLQDNVYWUHIPDSTH4RVDGT4IB27CSBYA, seg_pos={0 0}, upload=332730]: Uploading piece... Mar 18 20:04:15.888499509 (297h4m13.485270531s) [upload=332730]: Next returned nil segment Mar 18 20:04:15.888523854 (297h4m13.485294882s) [upload=332730]: Waiting for error group managing segments... Mar 18 20:04:15.892357218 (297h4m13.48912823s) [noise=true , address=79.127.219.46:4894 , piece_id=U33HRDMD5CQSBFGIEAQJTLQDNVYWUHIPDSTH4RVDGT4IB27CSBYA, seg_pos={0 0}, upload=332730]: Done uploading piece. err: <nil> Mar 18 20:04:15.892802795 (297h4m13.489573828s) [noise=true , address=45.91.116.12:4260 , piece_id=JCZ5KUJ7QBW5KIRZ7UHOCOTLTVQP4TLG4IYHAGQDHWMGDFO6D5GQ, seg_pos={0 0}, upload=332730]: Done uploading piece. err: <nil> Mar 18 20:04:15.893652700 (297h4m13.49042373s) [noise=true , address=45.91.116.11:4121 , piece_id=EGOJ7I4GYLTLYZTLVOXK2YFUJGPB4L2T2EJ3XFNQQYNSGVXLGXEA, seg_pos={0 0}, upload=332730]: Done uploading piece. err: <nil> Mar 18 20:04:15.893771054 (297h4m13.490542099s) [noise=true , address=45.91.116.13:4188 , piece_id=3DGAL7BTW2LROY5GX6WR2FTZEWGJNJ7UOVXOZBHWMLXTVBUTEJZA, seg_pos={0 0}, upload=332730]: Done uploading piece. err: <nil> Mar 18 20:04:15.894278015 (297h4m13.491049048s) [noise=true , address=45.91.116.14:4191 , piece_id=ZF7KWZPHRJDVABY2OMX5H5G6CATYHPD2T32AKEWQ5653HMGC77FA, seg_pos={0 0}, upload=332730]: Done uploading piece. err: <nil> Mar 18 20:04:15.898058490 (297h4m13.494829504s) [noise=true , address=77.90.31.18:4805 , piece_id=5L7MKDVCJWCG4KQ52CKJV5SQS2JZI5BGQRW4QXBJQV6QVXEB6BZA, seg_pos={0 0}, upload=332730]: Done uploading piece. err: <nil> Mar 18 20:04:15.908841216 (297h4m13.50561224s) [noise=true , address=77.90.31.17:4769 , piece_id=L3UU2YI7SO5GJR2YXXLC2HW356VLG2GB4SOWBTSY27AO7EAMD62Q, seg_pos={0 0}, upload=332730]: Done uploading piece. err: <nil> Mar 18 20:04:15.911211299 (297h4m13.507982313s) [noise=true , address=45.91.116.5:4297 , piece_id=2XGIQ7VHVOS6LZ3SSIK5XL2I4ACDPJ57BXKQCFPLZO6HI3ADTHLA, seg_pos={0 0}, upload=332730]: Done uploading piece. err: <nil> Mar 18 20:04:15.914996180 (297h4m13.511767207s) [noise=true , address=77.90.31.11:4555 , piece_id=23M5BFJQKODZYDFM7SEYR5R4GBVDKLRUUBFM6CD7WHOZMLHTVPNQ, seg_pos={0 0}, upload=332730]: Done uploading piece. err: <nil> Mar 18 20:04:15.960374088 (297h4m13.557145116s) [noise=true , address=45.91.116.9:4265 , piece_id=ODQG6YJNSJDMEGZR3DIDE2LV2VG4AZXR74OUROP4YURJ7CLGAJDA, seg_pos={0 0}, upload=332730]: Done uploading piece. err: <nil> Mar 18 20:04:42.733811317 (297h4m40.330582349s) [noise=true , address=145.79.227.2:4119 , piece_id=3IGMZRN7TOTUQR6BCL6Y4UQCR4KIEFLHHGW37QXQRLZ2JTWOKAGQ, seg_pos={0 0}, upload=332730]: Done uploading piece. err: ecclient: upload failed (node:121xNAnkm9HZXUszp26UmWTsuhp3igyeDVea6gSCdtrQUQDVHpW, address:145.79.227.2:4119): protocol: rpc: dial tcp 145.79.227.2:4119: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:04:42.733841283 (297h4m40.330612309s) [noise=true , address=145.79.227.4:4189 , piece_id=FFEAVSH2EK7YT3ET5C6Y56XAMRITYHQRFYLOC3WTGWI74OMOUT4Q, seg_pos={0 0}, upload=332730]: Done uploading piece. err: ecclient: upload failed (node:1jH3757i2F3q18XSpoXznsysZPaXmEnqynpLPDupgRfxJrTgKv, address:145.79.227.4:4189): protocol: rpc: dial tcp 145.79.227.4:4189: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:04:42.734141754 (297h4m40.330912783s) [noise=true , address=145.79.227.9:4369 , piece_id=GC6OCZ7CXOKBSQYFRGM7C3BX5WMIQ5MDIALFAPN2GNKYF4YIN2MQ, seg_pos={0 0}, upload=332730]: Done uploading piece. err: ecclient: upload failed (node:1nwygPHyRJEZkjCoSZJTVhXDKLHc7AonVMzKMht6ppwZwhcGCT, address:145.79.227.9:4369): protocol: rpc: dial tcp 145.79.227.9:4369: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:04:42.734487144 (297h4m40.331258172s) [noise=true , address=145.79.227.35:4482 , piece_id=X3UQEIIBUQ5263YMPQLGIRW6EHOOIB476XNJDCCSC4HBYEGKUYTQ, seg_pos={0 0}, upload=332730]: Done uploading piece. err: ecclient: upload failed (node:12mNCKAdMzWu87fmyrcwDKZvscMcu2CogjXicwAi5wLSFJa7oVV, address:145.79.227.35:4482): protocol: rpc: dial tcp 145.79.227.35:4482: i/o timeout; conn closed: op; piecestore: piecestore close: context deadline exceeded Mar 18 20:04:42.734614154 (297h4m40.331385186s) [noise=true , address=145.79.227.51:4412 , piece_id=2RAPHKERYTBMLWB4IUWAT7NUSWR7EGOYEEOARZIQZIBQEYRJSYGA, seg_pos={0 0}, upload=332730]: Done uploading piece. err: ecclient: upload failed (node:12vBvEfCYZxsv17cVsQTbCQ4YjZfiD4RXjWyz7Ka9ZZrtvzC7SS, address:145.79.227.51:4412): protocol: rpc: dial tcp 145.79.227.51:4412: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:04:42.734856648 (297h4m40.33162768s) [noise=true , address=145.79.227.43:4158 , piece_id=HQLX5SJEGF4IQ6GJIWZUOYFP7TL7IIB7MRU4IDYOA2S2JFS22VFA, seg_pos={0 0}, upload=332730]: Done uploading piece. err: ecclient: upload failed (node:1vJZrSvPvwebw9V2tTbvkPEmLeLjryBUnq9TL6NwdPsHU9xcq2, address:145.79.227.43:4158): protocol: rpc: dial tcp 145.79.227.43:4158: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:04:42.734890164 (297h4m40.33166118s) [seg_pos={0 0}, upload=332730]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Mar 18 20:04:42.760664021 (297h4m40.357435035s) [upload=332730]: Done uploading segments... Mar 18 20:04:42.760710453 (297h4m40.35748148s) [upload=332730]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration: 30.028552024s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 47 out of 60 is failed (threshold=6) [.........XXXXXXXXX.XXX...XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 5.000743146s trace: https://jaeger.dev.storj.io/trace/2572dad753fd3389 FAIL (2026-03-18 20:04:25 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.000767024s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 8 out of 60 is failed (threshold=3) [.......................XXXXXX.XX............................] duration: 15.262016389s trace: https://jaeger.dev.storj.io/trace/4b4d10a8051ad01b FAIL (2026-03-18 20:04:17 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 15.262076556s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 7 out of 60 is failed (threshold=3) [.........................XXX..........X...X.........X....X..] duration: 16.133494784s trace: https://jaeger.dev.storj.io/trace/698fb9649d932a68 FAIL (2026-03-18 20:04:14 UTC) operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 20:02:41.019616017 (297h2m38.616387042s) [upload=332700]: Uploading segments... Mar 18 20:02:41.019826191 (297h2m38.616597205s) [upload=332700]: Got next segment. Inline: false Mar 18 20:02:41.019879916 (297h2m38.61665094s) [upload=332700]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 20:02:41.019885584 (297h2m38.616656611s) [upload=332700]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 18 20:02:43.834160391 (297h2m41.430931414s) [seg_pos={0 0}, upload=332700]: Begin upload segment... Mar 18 20:02:43.834230381 (297h2m41.431001396s) [seg_pos={0 0}, upload=332700]: Done begin upload segment. Mar 18 20:02:43.834262428 (297h2m41.431033451s) [noise=true , address=145.79.227.52:4451 , piece_id=LR6KVRYMDO2HAESG5BFI5RYPNGVARP6O5HEY2UP77YSTVRKBNUXQ, seg_pos={0 0}, upload=332700]: Uploading piece... Mar 18 20:02:43.834329472 (297h2m41.431100497s) [noise=true , address=45.91.116.13:4164 , piece_id=MNQLVANO3LFKBCJMADD5RULARMRO4VUQY7LT63TU2CCNO765GW7Q, seg_pos={0 0}, upload=332700]: Uploading piece... Mar 18 20:02:43.834938832 (297h2m41.431709858s) [noise=true , address=77.90.31.15:4716 , piece_id=IDQP2MIPCKVS2FGVVGKAIDYTNUIPE56EER5IYZUSOTRD5LZBSXHA, seg_pos={0 0}, upload=332700]: Uploading piece... Mar 18 20:02:43.835898822 (297h2m41.432669851s) [noise=true , address=79.127.205.237:4227 , piece_id=NO25WIYZTEPBOMGETTLSDOJSPKARMKNLVUWDKNXUKHYGDGF2XC5Q, seg_pos={0 0}, upload=332700]: Uploading piece... Mar 18 20:02:43.836893726 (297h2m41.433664759s) [noise=true , address=193.149.164.68:4064 , piece_id=HAA3NDB24MJLAI2HK2UDZQZRLDUXEF76ZJF2VBE6FLKRWSB27Y3A, seg_pos={0 0}, upload=332700]: Uploading piece... Mar 18 20:02:43.837257949 (297h2m41.434028981s) [noise=true , address=193.149.164.66:4974 , piece_id=XWMBTFEWXVVFIR6IYNQYU35QKGVSQJ7YBVSWZ57QSUZ2VSDCJPRA, seg_pos={0 0}, upload=332700]: Uploading piece... Mar 18 20:02:43.838266356 (297h2m41.435037391s) [noise=true , address=145.79.227.34:4924 , piece_id=BNUESPWRZQBX23K7UGNSMF362AWHILNZ43UITBHVGFXIJ4T6V5HQ, seg_pos={0 0}, upload=332700]: Uploading piece... Mar 18 20:02:43.838688582 (297h2m41.435459605s) [noise=true , address=79.127.205.233:4171 , piece_id=YO42WSHYRB2JJX4RVX653N7U6PYTGIW3KQTNSF4445OTN3UW2TIA, seg_pos={0 0}, upload=332700]: Uploading piece... Mar 18 20:02:43.839263235 (297h2m41.436034258s) [noise=true , address=193.149.164.70:4123 , piece_id=VROUAQKRTKZT42ZDDSCIMKFMRODPKFQ5U5DWDJLVI462BVA5YWXA, seg_pos={0 0}, upload=332700]: Uploading piece... Mar 18 20:02:43.839835928 (297h2m41.436606963s) [noise=true , address=45.91.116.4:4948 , piece_id=ULVH4YAJIYY5RWWAGH5RTSZ3BMAT6WCYYAJPSMSP25LCLZQIPG2A, seg_pos={0 0}, upload=332700]: Uploading piece... Mar 18 20:02:43.840390028 (297h2m41.437161055s) [noise=true , address=45.91.116.5:4319 , piece_id=BY6BRNZNKKQKZNAVJTSWUYASX57P5X7OCKQ433T2ARRBXUFNBRCA, seg_pos={0 0}, upload=332700]: Uploading piece... Mar 18 20:02:43.840790648 (297h2m41.437561682s) [noise=true , address=45.91.116.12:4259 , piece_id=4LBHNF3CIQG5LYEY7RYJGNK5Q62X2TSOAXOXSNHO727OXL5OMA4Q, seg_pos={0 0}, upload=332700]: Uploading piece... Mar 18 20:02:43.841319562 (297h2m41.438090584s) [noise=true , address=145.79.227.29:4966 , piece_id=ZTSKMRBESX4OIZMMIPXRW752W5KQYX57RPDQ22XXPURQTSCAI2YA, seg_pos={0 0}, upload=332700]: Uploading piece... Mar 18 20:02:43.841816046 (297h2m41.438587078s) [noise=true , address=193.149.164.71:4287 , piece_id=FYNZ3QGUB7ZFMUJAMYPMUHXJDS66CJ2VLXPV7256DFCABPJMEOYQ, seg_pos={0 0}, upload=332700]: Uploading piece... Mar 18 20:02:43.842288870 (297h2m41.439059903s) [noise=true , address=79.127.219.41:4941 , piece_id=CQXOXMCJHWJ7FSIXDVYUJZMBFW5FUUOJNKJJOVLZRSK5D6XZXX5Q, seg_pos={0 0}, upload=332700]: Uploading piece... Mar 18 20:02:43.842956832 (297h2m41.439727861s) [noise=true , address=145.79.227.49:4369 , piece_id=CBYHFRBNKSPUNJHAE6S72YOFACWLZKGZIFV6SXWDYMVSA4FBPKQQ, seg_pos={0 0}, upload=332700]: Uploading piece... Mar 18 20:02:43.844386533 (297h2m41.441157559s) [noise=true , address=77.90.31.10:4533 , piece_id=VB5BKAWDUZFVII4ZY3JRVK67VUULPSAHTJVZMXXVATP3SXAR2XHQ, seg_pos={0 0}, upload=332700]: Uploading piece... Mar 18 20:02:43.844868553 (297h2m41.441639576s) [noise=true , address=145.79.227.37:4582 , piece_id=DLOFMRZRG3BPFN7JF73OM5HUV56FLEH2IDDC47UQWWWOTRVRR4HQ, seg_pos={0 0}, upload=332700]: Uploading piece... Mar 18 20:02:43.903559633 (297h2m41.500330666s) [upload=332700]: Next returned nil segment Mar 18 20:02:43.903578892 (297h2m41.500349925s) [upload=332700]: Waiting for error group managing segments... Mar 18 20:02:43.908218461 (297h2m41.504989474s) [noise=true , address=45.91.116.4:4948 , piece_id=ULVH4YAJIYY5RWWAGH5RTSZ3BMAT6WCYYAJPSMSP25LCLZQIPG2A, seg_pos={0 0}, upload=332700]: Done uploading piece. err: <nil> Mar 18 20:02:43.917679744 (297h2m41.514450753s) [noise=true , address=45.91.116.12:4259 , piece_id=4LBHNF3CIQG5LYEY7RYJGNK5Q62X2TSOAXOXSNHO727OXL5OMA4Q, seg_pos={0 0}, upload=332700]: Done uploading piece. err: <nil> Mar 18 20:02:43.924357408 (297h2m41.521128422s) [noise=true , address=77.90.31.10:4533 , piece_id=VB5BKAWDUZFVII4ZY3JRVK67VUULPSAHTJVZMXXVATP3SXAR2XHQ, seg_pos={0 0}, upload=332700]: Done uploading piece. err: <nil> Mar 18 20:02:43.926248407 (297h2m41.52301943s) [noise=true , address=77.90.31.15:4716 , piece_id=IDQP2MIPCKVS2FGVVGKAIDYTNUIPE56EER5IYZUSOTRD5LZBSXHA, seg_pos={0 0}, upload=332700]: Done uploading piece. err: <nil> Mar 18 20:02:43.927212300 (297h2m41.523983323s) [noise=true , address=45.91.116.5:4319 , piece_id=BY6BRNZNKKQKZNAVJTSWUYASX57P5X7OCKQ433T2ARRBXUFNBRCA, seg_pos={0 0}, upload=332700]: Done uploading piece. err: <nil> Mar 18 20:02:43.927791610 (297h2m41.524562634s) [noise=true , address=45.91.116.13:4164 , piece_id=MNQLVANO3LFKBCJMADD5RULARMRO4VUQY7LT63TU2CCNO765GW7Q, seg_pos={0 0}, upload=332700]: Done uploading piece. err: <nil> Mar 18 20:02:44.028504592 (297h2m41.625275611s) [noise=true , address=79.127.219.41:4941 , piece_id=CQXOXMCJHWJ7FSIXDVYUJZMBFW5FUUOJNKJJOVLZRSK5D6XZXX5Q, seg_pos={0 0}, upload=332700]: Done uploading piece. err: <nil> Mar 18 20:02:44.392266647 (297h2m41.989037671s) [noise=true , address=79.127.205.233:4171 , piece_id=YO42WSHYRB2JJX4RVX653N7U6PYTGIW3KQTNSF4445OTN3UW2TIA, seg_pos={0 0}, upload=332700]: Done uploading piece. err: <nil> Mar 18 20:02:44.429403642 (297h2m42.026174669s) [noise=true , address=193.149.164.66:4974 , piece_id=XWMBTFEWXVVFIR6IYNQYU35QKGVSQJ7YBVSWZ57QSUZ2VSDCJPRA, seg_pos={0 0}, upload=332700]: Done uploading piece. err: <nil> Mar 18 20:02:44.433374828 (297h2m42.030145853s) [noise=true , address=193.149.164.68:4064 , piece_id=HAA3NDB24MJLAI2HK2UDZQZRLDUXEF76ZJF2VBE6FLKRWSB27Y3A, seg_pos={0 0}, upload=332700]: Done uploading piece. err: <nil> Mar 18 20:02:44.444389014 (297h2m42.041160042s) [noise=true , address=193.149.164.71:4287 , piece_id=FYNZ3QGUB7ZFMUJAMYPMUHXJDS66CJ2VLXPV7256DFCABPJMEOYQ, seg_pos={0 0}, upload=332700]: Done uploading piece. err: <nil> Mar 18 20:02:44.578333498 (297h2m42.175104529s) [noise=true , address=79.127.205.237:4227 , piece_id=NO25WIYZTEPBOMGETTLSDOJSPKARMKNLVUWDKNXUKHYGDGF2XC5Q, seg_pos={0 0}, upload=332700]: Done uploading piece. err: <nil> Mar 18 20:02:44.665665140 (297h2m42.262436168s) [noise=true , address=193.149.164.70:4123 , piece_id=VROUAQKRTKZT42ZDDSCIMKFMRODPKFQ5U5DWDJLVI462BVA5YWXA, seg_pos={0 0}, upload=332700]: Done uploading piece. err: <nil> Mar 18 20:03:40.987108256 (297h3m38.58387928s) [noise=true , address=145.79.227.49:4369 , piece_id=CBYHFRBNKSPUNJHAE6S72YOFACWLZKGZIFV6SXWDYMVSA4FBPKQQ, seg_pos={0 0}, upload=332700]: Done uploading piece. err: ecclient: upload failed (node:12nLksGXGvDwuTWfnmkceuwZNDmrF17iNepHPdYkkvZG2QQUwG7, address:145.79.227.49:4369): protocol: rpc: dial tcp 145.79.227.49:4369: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:03:40.987479807 (297h3m38.58425084s) [noise=true , address=145.79.227.29:4966 , piece_id=ZTSKMRBESX4OIZMMIPXRW752W5KQYX57RPDQ22XXPURQTSCAI2YA, seg_pos={0 0}, upload=332700]: Done uploading piece. err: ecclient: upload failed (node:1DLnsYVdudE4mY9dNAQKhBRppUEAHvxbf84KX69451pwxSxzZK, address:145.79.227.29:4966): protocol: rpc: dial tcp 145.79.227.29:4966: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:03:40.987517864 (297h3m38.584288896s) [noise=true , address=145.79.227.34:4924 , piece_id=BNUESPWRZQBX23K7UGNSMF362AWHILNZ43UITBHVGFXIJ4T6V5HQ, seg_pos={0 0}, upload=332700]: Done uploading piece. err: ecclient: upload failed (node:19TbrQjuN1EhBNVtha9EcKPSCAWyttfmHpec6QeF1S5zW6DRZs, address:145.79.227.34:4924): protocol: rpc: dial tcp 145.79.227.34:4924: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:03:40.987933296 (297h3m38.584704329s) [noise=true , address=145.79.227.37:4582 , piece_id=DLOFMRZRG3BPFN7JF73OM5HUV56FLEH2IDDC47UQWWWOTRVRR4HQ, seg_pos={0 0}, upload=332700]: Done uploading piece. err: ecclient: upload failed (node:129o91j97t4V5AjaouCCy2wDuHgwF94iQiJh2h9ux1aFo7AVCSn, address:145.79.227.37:4582): protocol: rpc: dial tcp 145.79.227.37:4582: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:03:40.988167350 (297h3m38.584938376s) [noise=true , address=145.79.227.52:4451 , piece_id=LR6KVRYMDO2HAESG5BFI5RYPNGVARP6O5HEY2UP77YSTVRKBNUXQ, seg_pos={0 0}, upload=332700]: Done uploading piece. err: ecclient: upload failed (node:12bC4P3xBiPFkxLvavGFVbWKnbprCStL6UJgfKSA2Wd2fX7Mbfw, address:145.79.227.52:4451): protocol: rpc: dial tcp 145.79.227.52:4451: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:03:40.988197047 (297h3m38.58496806s) [seg_pos={0 0}, upload=332700]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 18 20:03:41.007727578 (297h3m38.604498608s) [upload=332700]: Done uploading segments... Mar 18 20:03:41.007779207 (297h3m38.60455025s) [upload=332700]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 16.133527662s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 38 out of 60 is failed (threshold=3) [.....................XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.X] duration: 30.030050123s trace: https://jaeger.dev.storj.io/trace/57ac6ef9f68b1954 FAIL (2026-03-18 20:04:12 UTC) operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 7) ======LOGS====== d=332722]: Uploading segments... Mar 18 20:03:42.682054243 (297h3m40.278825266s) [upload=332722]: Got next segment. Inline: false Mar 18 20:03:42.682092092 (297h3m40.278863115s) [upload=332722]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 20:03:42.682186798 (297h3m40.27895783s) [upload=332722]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 18 20:03:45.793181227 (297h3m43.38995225s) [seg_pos={0 0}, upload=332722]: Begin upload segment... Mar 18 20:03:45.793243492 (297h3m43.390014515s) [seg_pos={0 0}, upload=332722]: Done begin upload segment. Mar 18 20:03:45.793285202 (297h3m43.390056226s) [noise=true , address=145.79.227.19:4077 , piece_id=HBGD3N2Y5Q6PFXWYUIARZNRRHUI7FDF5UNZKH5W6DLM4SG6D2PKA, seg_pos={0 0}, upload=332722]: Uploading piece... Mar 18 20:03:45.793336989 (297h3m43.390108012s) [noise=true , address=45.91.116.7:4012 , piece_id=BP236K3GXCY7EQ7TBQ2LDFXTXQLHHRM36GPUTD7XT7RYWYC2AD7Q, seg_pos={0 0}, upload=332722]: Uploading piece... Mar 18 20:03:45.793852919 (297h3m43.390623946s) [noise=true , address=145.79.227.35:4481 , piece_id=GLA22Z5IAQIBPQ5D2LKB6HMKOZCFRIECSTL3JLCYUD7O6RCAAGPA, seg_pos={0 0}, upload=332722]: Uploading piece... Mar 18 20:03:45.794017856 (297h3m43.390788891s) [noise=true , address=145.79.227.50:4374 , piece_id=O3I3JVAHA3ZTP5WHWUJE6IRI5TYO2E3HHPGY3IZHI3YU6SDLUANA, seg_pos={0 0}, upload=332722]: Uploading piece... Mar 18 20:03:45.794640345 (297h3m43.391411378s) [noise=true , address=145.79.227.25:4622 , piece_id=YFBS5SBLN4RGNHFRNGHVIS6HE7D3R6WM32LW4KJAPVPUHYX22TXA, seg_pos={0 0}, upload=332722]: Uploading piece... Mar 18 20:03:45.795075010 (297h3m43.391846062s) [noise=true , address=145.79.227.17:4975 , piece_id=CLJISVBS45YQFZFWJK2ZPRKUYRJMLUNBLYGQSOM6OYDZ7ZYN7JYQ, seg_pos={0 0}, upload=332722]: Uploading piece... Mar 18 20:03:45.795600020 (297h3m43.392371054s) [noise=true , address=45.91.116.4:4941 , piece_id=KLQQRLVCUITFEFXDHYDAOI7IOW4A2STN3ZG3MN42CBE7MA3ZPMUQ, seg_pos={0 0}, upload=332722]: Uploading piece... Mar 18 20:03:45.796098498 (297h3m43.392869529s) [noise=true , address=77.90.31.14:4663 , piece_id=4JEOFVDMLR6ZKHX64WGCNNAKL2ZVAWTRJHJASNK5BURBNVGNU7WA, seg_pos={0 0}, upload=332722]: Uploading piece... Mar 18 20:03:45.796854422 (297h3m43.393625452s) [noise=true , address=45.91.116.6:4989 , piece_id=ZGKKKAYRV6YS2PRE3B25LA4GP5NRXRMML3B3HUUXJ2OL6LM3KGQQ, seg_pos={0 0}, upload=332722]: Uploading piece... Mar 18 20:03:45.797409061 (297h3m43.394180096s) [noise=true , address=45.91.116.11:4117 , piece_id=ZO6QQSZJ476M7U4ECTWIOO6AHPAXGNVNTMLOSYK3NJYXX5PFC4SA, seg_pos={0 0}, upload=332722]: Uploading piece... Mar 18 20:03:45.797913649 (297h3m43.394684685s) [noise=true , address=145.79.227.45:4233 , piece_id=TVTJKKCNQMLHQCMTFMA44552PUJKHBJKTQIOGHGWLFKF6H4GVM4Q, seg_pos={0 0}, upload=332722]: Uploading piece... Mar 18 20:03:45.798496199 (297h3m43.395267231s) [noise=true , address=145.79.227.21:4473 , piece_id=G7HVAZ2OI5SLBVFGC4JZ7Q3PY33H45IBZMSFBKEJJTIH4QCZTHUA, seg_pos={0 0}, upload=332722]: Uploading piece... Mar 18 20:03:45.799088489 (297h3m43.395859526s) [noise=true , address=145.79.227.7:4297 , piece_id=ZKYGADJJ5MESHKKQR7W3OQYQWTL7ZGBNVS7K6CWIGWAY4DP5IGHA, seg_pos={0 0}, upload=332722]: Uploading piece... Mar 18 20:03:45.799687725 (297h3m43.396458752s) [noise=true , address=145.79.227.46:4604 , piece_id=JNLR4TZMVH2JYBXRQ3D5D5QNFJEWS5GTATXV3KO6IVWUAZKKFGKA, seg_pos={0 0}, upload=332722]: Uploading piece... Mar 18 20:03:45.801186418 (297h3m43.397957461s) [noise=true , address=77.90.31.21:4878 , piece_id=BFCXTZVWLIDZ3P2376LQY7XIYF6GLIYJTXOSVJNYSU4TWPN46LCA, seg_pos={0 0}, upload=332722]: Uploading piece... Mar 18 20:03:45.802871699 (297h3m43.399642731s) [noise=true , address=79.127.219.45:4854 , piece_id=BT5GJEZBDRNVOOAPAYBY4RC2BRB6BCXSOQMWIKAHZ756QTITOPDQ, seg_pos={0 0}, upload=332722]: Uploading piece... Mar 18 20:03:45.854622430 (297h3m43.451393464s) [upload=332722]: Next returned nil segment Mar 18 20:03:45.854639563 (297h3m43.451410605s) [upload=332722]: Waiting for error group managing segments... Mar 18 20:03:45.858552233 (297h3m43.455323257s) [noise=true , address=45.91.116.6:4989 , piece_id=ZGKKKAYRV6YS2PRE3B25LA4GP5NRXRMML3B3HUUXJ2OL6LM3KGQQ, seg_pos={0 0}, upload=332722]: Done uploading piece. err: <nil> Mar 18 20:03:45.858874779 (297h3m43.455645825s) [noise=true , address=45.91.116.11:4117 , piece_id=ZO6QQSZJ476M7U4ECTWIOO6AHPAXGNVNTMLOSYK3NJYXX5PFC4SA, seg_pos={0 0}, upload=332722]: Done uploading piece. err: <nil> Mar 18 20:03:45.859946528 (297h3m43.456717558s) [noise=true , address=45.91.116.4:4941 , piece_id=KLQQRLVCUITFEFXDHYDAOI7IOW4A2STN3ZG3MN42CBE7MA3ZPMUQ, seg_pos={0 0}, upload=332722]: Done uploading piece. err: <nil> Mar 18 20:03:45.860597071 (297h3m43.457368098s) [noise=true , address=77.90.31.14:4663 , piece_id=4JEOFVDMLR6ZKHX64WGCNNAKL2ZVAWTRJHJASNK5BURBNVGNU7WA, seg_pos={0 0}, upload=332722]: Done uploading piece. err: <nil> Mar 18 20:03:45.861234361 (297h3m43.458005388s) [noise=true , address=45.91.116.7:4012 , piece_id=BP236K3GXCY7EQ7TBQ2LDFXTXQLHHRM36GPUTD7XT7RYWYC2AD7Q, seg_pos={0 0}, upload=332722]: Done uploading piece. err: <nil> Mar 18 20:03:45.882215044 (297h3m43.478986074s) [noise=true , address=77.90.31.21:4878 , piece_id=BFCXTZVWLIDZ3P2376LQY7XIYF6GLIYJTXOSVJNYSU4TWPN46LCA, seg_pos={0 0}, upload=332722]: Done uploading piece. err: <nil> Mar 18 20:03:45.886253662 (297h3m43.483024686s) [noise=true , address=79.127.219.45:4854 , piece_id=BT5GJEZBDRNVOOAPAYBY4RC2BRB6BCXSOQMWIKAHZ756QTITOPDQ, seg_pos={0 0}, upload=332722]: Done uploading piece. err: <nil> Mar 18 20:04:12.643217190 (297h4m10.239988219s) [noise=true , address=145.79.227.25:4622 , piece_id=YFBS5SBLN4RGNHFRNGHVIS6HE7D3R6WM32LW4KJAPVPUHYX22TXA, seg_pos={0 0}, upload=332722]: Done uploading piece. err: ecclient: upload failed (node:1xEK5XL3sYcip7PsQcZBQYihgnEZykdMmsKkCr2TUmbTFYncfs, address:145.79.227.25:4622): protocol: rpc: dial tcp 145.79.227.25:4622: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:04:12.643284920 (297h4m10.24005595s) [noise=true , address=145.79.227.17:4975 , piece_id=CLJISVBS45YQFZFWJK2ZPRKUYRJMLUNBLYGQSOM6OYDZ7ZYN7JYQ, seg_pos={0 0}, upload=332722]: Done uploading piece. err: ecclient: upload failed (node:12WiUZeNCJk4rzjtyWuKJwGcZATaYLU2QxN27MZE7i45zVpu9dM, address:145.79.227.17:4975): protocol: rpc: dial tcp 145.79.227.17:4975: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:04:12.643312931 (297h4m10.240083963s) [noise=true , address=145.79.227.21:4473 , piece_id=G7HVAZ2OI5SLBVFGC4JZ7Q3PY33H45IBZMSFBKEJJTIH4QCZTHUA, seg_pos={0 0}, upload=332722]: Done uploading piece. err: ecclient: upload failed (node:12DDe4jT3YeUH8qMQgapcvzMnFs8UFB8fTmubho4ZQf5tYVoxuk, address:145.79.227.21:4473): protocol: rpc: dial tcp 145.79.227.21:4473: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:04:12.643457660 (297h4m10.240228685s) [noise=true , address=145.79.227.46:4604 , piece_id=JNLR4TZMVH2JYBXRQ3D5D5QNFJEWS5GTATXV3KO6IVWUAZKKFGKA, seg_pos={0 0}, upload=332722]: Done uploading piece. err: ecclient: upload failed (node:129d1v3mjqB3FHC2VmbY9Sih6C5fRDrV3kGLiv9NfkDc7JovXjy, address:145.79.227.46:4604): protocol: rpc: dial tcp 145.79.227.46:4604: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:04:12.643515778 (297h4m10.24028681s) [noise=true , address=145.79.227.35:4481 , piece_id=GLA22Z5IAQIBPQ5D2LKB6HMKOZCFRIECSTL3JLCYUD7O6RCAAGPA, seg_pos={0 0}, upload=332722]: Done uploading piece. err: ecclient: upload failed (node:12bRK8WWmAnWUwzCfHu8JtZgrruYSJCvhnsYTKcaLccJcv5RQjp, address:145.79.227.35:4481): protocol: rpc: dial tcp 145.79.227.35:4481: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:04:12.643560804 (297h4m10.240331833s) [noise=true , address=145.79.227.45:4233 , piece_id=TVTJKKCNQMLHQCMTFMA44552PUJKHBJKTQIOGHGWLFKF6H4GVM4Q, seg_pos={0 0}, upload=332722]: Done uploading piece. err: ecclient: upload failed (node:1DmQeoQWTnFhQeJFseJoNYzFyo7vZjYKwDtQVJ6pomksWExsTb, address:145.79.227.45:4233): protocol: rpc: dial tcp 145.79.227.45:4233: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:04:12.643717091 (297h4m10.240488123s) [noise=true , address=145.79.227.7:4297 , piece_id=ZKYGADJJ5MESHKKQR7W3OQYQWTL7ZGBNVS7K6CWIGWAY4DP5IGHA, seg_pos={0 0}, upload=332722]: Done uploading piece. err: ecclient: upload failed (node:1g2pJxnoFW2TAzcAzNhjzDRJ3yYDK2mibG8tdX2Qy2YabNTq86, address:145.79.227.7:4297): protocol: rpc: dial tcp 145.79.227.7:4297: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:04:12.643873094 (297h4m10.240644126s) [noise=true , address=145.79.227.19:4077 , piece_id=HBGD3N2Y5Q6PFXWYUIARZNRRHUI7FDF5UNZKH5W6DLM4SG6D2PKA, seg_pos={0 0}, upload=332722]: Done uploading piece. err: ecclient: upload failed (node:12kDJuQP8iLUjB1rXpcNX5TtHzsqa5HvYcUD9Ef64HYTnTXcwLk, address:145.79.227.19:4077): protocol: rpc: dial tcp 145.79.227.19:4077: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:04:12.644129833 (297h4m10.240900847s) [noise=true , address=145.79.227.50:4374 , piece_id=O3I3JVAHA3ZTP5WHWUJE6IRI5TYO2E3HHPGY3IZHI3YU6SDLUANA, seg_pos={0 0}, upload=332722]: Done uploading piece. err: ecclient: upload failed (node:12QRixhWaUwXPdjcWKYHbiknME5JVomWYABY6egDD6VKwaTPB8K, address:145.79.227.50:4374): protocol: rpc: dial tcp 145.79.227.50:4374: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:04:12.644160114 (297h4m10.24093113s) [seg_pos={0 0}, upload=332722]: Done waiting for segment. successful: 7 optimal: 13 errs: <nil> Mar 18 20:04:12.670772828 (297h4m10.267543862s) [upload=332722]: Done uploading segments... Mar 18 20:04:12.670860004 (297h4m10.267631028s) [upload=332722]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 7) Duration: 30.030140606s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 7 out of 60 is failed (threshold=3) [..........................XXX..........X...X.........X....X.] duration: 15.826202395s trace: https://jaeger.dev.storj.io/trace/698fb9649d932a68 FAIL (2026-03-18 20:03:56 UTC) operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 20:02:41.019616017 (297h2m38.616387042s) [upload=332700]: Uploading segments... Mar 18 20:02:41.019826191 (297h2m38.616597205s) [upload=332700]: Got next segment. Inline: false Mar 18 20:02:41.019879916 (297h2m38.61665094s) [upload=332700]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 20:02:41.019885584 (297h2m38.616656611s) [upload=332700]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 18 20:02:43.834160391 (297h2m41.430931414s) [seg_pos={0 0}, upload=332700]: Begin upload segment... Mar 18 20:02:43.834230381 (297h2m41.431001396s) [seg_pos={0 0}, upload=332700]: Done begin upload segment. Mar 18 20:02:43.834262428 (297h2m41.431033451s) [noise=true , address=145.79.227.52:4451 , piece_id=LR6KVRYMDO2HAESG5BFI5RYPNGVARP6O5HEY2UP77YSTVRKBNUXQ, seg_pos={0 0}, upload=332700]: Uploading piece... Mar 18 20:02:43.834329472 (297h2m41.431100497s) [noise=true , address=45.91.116.13:4164 , piece_id=MNQLVANO3LFKBCJMADD5RULARMRO4VUQY7LT63TU2CCNO765GW7Q, seg_pos={0 0}, upload=332700]: Uploading piece... Mar 18 20:02:43.834938832 (297h2m41.431709858s) [noise=true , address=77.90.31.15:4716 , piece_id=IDQP2MIPCKVS2FGVVGKAIDYTNUIPE56EER5IYZUSOTRD5LZBSXHA, seg_pos={0 0}, upload=332700]: Uploading piece... Mar 18 20:02:43.835898822 (297h2m41.432669851s) [noise=true , address=79.127.205.237:4227 , piece_id=NO25WIYZTEPBOMGETTLSDOJSPKARMKNLVUWDKNXUKHYGDGF2XC5Q, seg_pos={0 0}, upload=332700]: Uploading piece... Mar 18 20:02:43.836893726 (297h2m41.433664759s) [noise=true , address=193.149.164.68:4064 , piece_id=HAA3NDB24MJLAI2HK2UDZQZRLDUXEF76ZJF2VBE6FLKRWSB27Y3A, seg_pos={0 0}, upload=332700]: Uploading piece... Mar 18 20:02:43.837257949 (297h2m41.434028981s) [noise=true , address=193.149.164.66:4974 , piece_id=XWMBTFEWXVVFIR6IYNQYU35QKGVSQJ7YBVSWZ57QSUZ2VSDCJPRA, seg_pos={0 0}, upload=332700]: Uploading piece... Mar 18 20:02:43.838266356 (297h2m41.435037391s) [noise=true , address=145.79.227.34:4924 , piece_id=BNUESPWRZQBX23K7UGNSMF362AWHILNZ43UITBHVGFXIJ4T6V5HQ, seg_pos={0 0}, upload=332700]: Uploading piece... Mar 18 20:02:43.838688582 (297h2m41.435459605s) [noise=true , address=79.127.205.233:4171 , piece_id=YO42WSHYRB2JJX4RVX653N7U6PYTGIW3KQTNSF4445OTN3UW2TIA, seg_pos={0 0}, upload=332700]: Uploading piece... Mar 18 20:02:43.839263235 (297h2m41.436034258s) [noise=true , address=193.149.164.70:4123 , piece_id=VROUAQKRTKZT42ZDDSCIMKFMRODPKFQ5U5DWDJLVI462BVA5YWXA, seg_pos={0 0}, upload=332700]: Uploading piece... Mar 18 20:02:43.839835928 (297h2m41.436606963s) [noise=true , address=45.91.116.4:4948 , piece_id=ULVH4YAJIYY5RWWAGH5RTSZ3BMAT6WCYYAJPSMSP25LCLZQIPG2A, seg_pos={0 0}, upload=332700]: Uploading piece... Mar 18 20:02:43.840390028 (297h2m41.437161055s) [noise=true , address=45.91.116.5:4319 , piece_id=BY6BRNZNKKQKZNAVJTSWUYASX57P5X7OCKQ433T2ARRBXUFNBRCA, seg_pos={0 0}, upload=332700]: Uploading piece... Mar 18 20:02:43.840790648 (297h2m41.437561682s) [noise=true , address=45.91.116.12:4259 , piece_id=4LBHNF3CIQG5LYEY7RYJGNK5Q62X2TSOAXOXSNHO727OXL5OMA4Q, seg_pos={0 0}, upload=332700]: Uploading piece... Mar 18 20:02:43.841319562 (297h2m41.438090584s) [noise=true , address=145.79.227.29:4966 , piece_id=ZTSKMRBESX4OIZMMIPXRW752W5KQYX57RPDQ22XXPURQTSCAI2YA, seg_pos={0 0}, upload=332700]: Uploading piece... Mar 18 20:02:43.841816046 (297h2m41.438587078s) [noise=true , address=193.149.164.71:4287 , piece_id=FYNZ3QGUB7ZFMUJAMYPMUHXJDS66CJ2VLXPV7256DFCABPJMEOYQ, seg_pos={0 0}, upload=332700]: Uploading piece... Mar 18 20:02:43.842288870 (297h2m41.439059903s) [noise=true , address=79.127.219.41:4941 , piece_id=CQXOXMCJHWJ7FSIXDVYUJZMBFW5FUUOJNKJJOVLZRSK5D6XZXX5Q, seg_pos={0 0}, upload=332700]: Uploading piece... Mar 18 20:02:43.842956832 (297h2m41.439727861s) [noise=true , address=145.79.227.49:4369 , piece_id=CBYHFRBNKSPUNJHAE6S72YOFACWLZKGZIFV6SXWDYMVSA4FBPKQQ, seg_pos={0 0}, upload=332700]: Uploading piece... Mar 18 20:02:43.844386533 (297h2m41.441157559s) [noise=true , address=77.90.31.10:4533 , piece_id=VB5BKAWDUZFVII4ZY3JRVK67VUULPSAHTJVZMXXVATP3SXAR2XHQ, seg_pos={0 0}, upload=332700]: Uploading piece... Mar 18 20:02:43.844868553 (297h2m41.441639576s) [noise=true , address=145.79.227.37:4582 , piece_id=DLOFMRZRG3BPFN7JF73OM5HUV56FLEH2IDDC47UQWWWOTRVRR4HQ, seg_pos={0 0}, upload=332700]: Uploading piece... Mar 18 20:02:43.903559633 (297h2m41.500330666s) [upload=332700]: Next returned nil segment Mar 18 20:02:43.903578892 (297h2m41.500349925s) [upload=332700]: Waiting for error group managing segments... Mar 18 20:02:43.908218461 (297h2m41.504989474s) [noise=true , address=45.91.116.4:4948 , piece_id=ULVH4YAJIYY5RWWAGH5RTSZ3BMAT6WCYYAJPSMSP25LCLZQIPG2A, seg_pos={0 0}, upload=332700]: Done uploading piece. err: <nil> Mar 18 20:02:43.917679744 (297h2m41.514450753s) [noise=true , address=45.91.116.12:4259 , piece_id=4LBHNF3CIQG5LYEY7RYJGNK5Q62X2TSOAXOXSNHO727OXL5OMA4Q, seg_pos={0 0}, upload=332700]: Done uploading piece. err: <nil> Mar 18 20:02:43.924357408 (297h2m41.521128422s) [noise=true , address=77.90.31.10:4533 , piece_id=VB5BKAWDUZFVII4ZY3JRVK67VUULPSAHTJVZMXXVATP3SXAR2XHQ, seg_pos={0 0}, upload=332700]: Done uploading piece. err: <nil> Mar 18 20:02:43.926248407 (297h2m41.52301943s) [noise=true , address=77.90.31.15:4716 , piece_id=IDQP2MIPCKVS2FGVVGKAIDYTNUIPE56EER5IYZUSOTRD5LZBSXHA, seg_pos={0 0}, upload=332700]: Done uploading piece. err: <nil> Mar 18 20:02:43.927212300 (297h2m41.523983323s) [noise=true , address=45.91.116.5:4319 , piece_id=BY6BRNZNKKQKZNAVJTSWUYASX57P5X7OCKQ433T2ARRBXUFNBRCA, seg_pos={0 0}, upload=332700]: Done uploading piece. err: <nil> Mar 18 20:02:43.927791610 (297h2m41.524562634s) [noise=true , address=45.91.116.13:4164 , piece_id=MNQLVANO3LFKBCJMADD5RULARMRO4VUQY7LT63TU2CCNO765GW7Q, seg_pos={0 0}, upload=332700]: Done uploading piece. err: <nil> Mar 18 20:02:44.028504592 (297h2m41.625275611s) [noise=true , address=79.127.219.41:4941 , piece_id=CQXOXMCJHWJ7FSIXDVYUJZMBFW5FUUOJNKJJOVLZRSK5D6XZXX5Q, seg_pos={0 0}, upload=332700]: Done uploading piece. err: <nil> Mar 18 20:02:44.392266647 (297h2m41.989037671s) [noise=true , address=79.127.205.233:4171 , piece_id=YO42WSHYRB2JJX4RVX653N7U6PYTGIW3KQTNSF4445OTN3UW2TIA, seg_pos={0 0}, upload=332700]: Done uploading piece. err: <nil> Mar 18 20:02:44.429403642 (297h2m42.026174669s) [noise=true , address=193.149.164.66:4974 , piece_id=XWMBTFEWXVVFIR6IYNQYU35QKGVSQJ7YBVSWZ57QSUZ2VSDCJPRA, seg_pos={0 0}, upload=332700]: Done uploading piece. err: <nil> Mar 18 20:02:44.433374828 (297h2m42.030145853s) [noise=true , address=193.149.164.68:4064 , piece_id=HAA3NDB24MJLAI2HK2UDZQZRLDUXEF76ZJF2VBE6FLKRWSB27Y3A, seg_pos={0 0}, upload=332700]: Done uploading piece. err: <nil> Mar 18 20:02:44.444389014 (297h2m42.041160042s) [noise=true , address=193.149.164.71:4287 , piece_id=FYNZ3QGUB7ZFMUJAMYPMUHXJDS66CJ2VLXPV7256DFCABPJMEOYQ, seg_pos={0 0}, upload=332700]: Done uploading piece. err: <nil> Mar 18 20:02:44.578333498 (297h2m42.175104529s) [noise=true , address=79.127.205.237:4227 , piece_id=NO25WIYZTEPBOMGETTLSDOJSPKARMKNLVUWDKNXUKHYGDGF2XC5Q, seg_pos={0 0}, upload=332700]: Done uploading piece. err: <nil> Mar 18 20:02:44.665665140 (297h2m42.262436168s) [noise=true , address=193.149.164.70:4123 , piece_id=VROUAQKRTKZT42ZDDSCIMKFMRODPKFQ5U5DWDJLVI462BVA5YWXA, seg_pos={0 0}, upload=332700]: Done uploading piece. err: <nil> Mar 18 20:03:40.987108256 (297h3m38.58387928s) [noise=true , address=145.79.227.49:4369 , piece_id=CBYHFRBNKSPUNJHAE6S72YOFACWLZKGZIFV6SXWDYMVSA4FBPKQQ, seg_pos={0 0}, upload=332700]: Done uploading piece. err: ecclient: upload failed (node:12nLksGXGvDwuTWfnmkceuwZNDmrF17iNepHPdYkkvZG2QQUwG7, address:145.79.227.49:4369): protocol: rpc: dial tcp 145.79.227.49:4369: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:03:40.987479807 (297h3m38.58425084s) [noise=true , address=145.79.227.29:4966 , piece_id=ZTSKMRBESX4OIZMMIPXRW752W5KQYX57RPDQ22XXPURQTSCAI2YA, seg_pos={0 0}, upload=332700]: Done uploading piece. err: ecclient: upload failed (node:1DLnsYVdudE4mY9dNAQKhBRppUEAHvxbf84KX69451pwxSxzZK, address:145.79.227.29:4966): protocol: rpc: dial tcp 145.79.227.29:4966: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:03:40.987517864 (297h3m38.584288896s) [noise=true , address=145.79.227.34:4924 , piece_id=BNUESPWRZQBX23K7UGNSMF362AWHILNZ43UITBHVGFXIJ4T6V5HQ, seg_pos={0 0}, upload=332700]: Done uploading piece. err: ecclient: upload failed (node:19TbrQjuN1EhBNVtha9EcKPSCAWyttfmHpec6QeF1S5zW6DRZs, address:145.79.227.34:4924): protocol: rpc: dial tcp 145.79.227.34:4924: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:03:40.987933296 (297h3m38.584704329s) [noise=true , address=145.79.227.37:4582 , piece_id=DLOFMRZRG3BPFN7JF73OM5HUV56FLEH2IDDC47UQWWWOTRVRR4HQ, seg_pos={0 0}, upload=332700]: Done uploading piece. err: ecclient: upload failed (node:129o91j97t4V5AjaouCCy2wDuHgwF94iQiJh2h9ux1aFo7AVCSn, address:145.79.227.37:4582): protocol: rpc: dial tcp 145.79.227.37:4582: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:03:40.988167350 (297h3m38.584938376s) [noise=true , address=145.79.227.52:4451 , piece_id=LR6KVRYMDO2HAESG5BFI5RYPNGVARP6O5HEY2UP77YSTVRKBNUXQ, seg_pos={0 0}, upload=332700]: Done uploading piece. err: ecclient: upload failed (node:12bC4P3xBiPFkxLvavGFVbWKnbprCStL6UJgfKSA2Wd2fX7Mbfw, address:145.79.227.52:4451): protocol: rpc: dial tcp 145.79.227.52:4451: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:03:40.988197047 (297h3m38.58496806s) [seg_pos={0 0}, upload=332700]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 18 20:03:41.007727578 (297h3m38.604498608s) [upload=332700]: Done uploading segments... Mar 18 20:03:41.007779207 (297h3m38.60455025s) [upload=332700]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 15.826222382s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 6 out of 60 is failed (threshold=3) [...........................X..........XX..X....X........X...] duration: 14.973210456s trace: https://jaeger.dev.storj.io/trace/589419efc52e5e2b FAIL (2026-03-18 20:03:49 UTC) operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== Mar 18 20:00:59.343084966 (297h0m56.939855989s) [upload=332666]: Starting upload Mar 18 20:00:59.343175518 (297h0m56.939946548s) [upload=332666]: Done starting upload Mar 18 20:00:59.343565585 (297h0m56.940336612s) [upload=332666]: Uploading segments... Mar 18 20:00:59.344256647 (297h0m56.941027676s) [upload=332666]: Got next segment. Inline: false Mar 18 20:00:59.344352215 (297h0m56.941123241s) [upload=332666]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 20:00:59.344358706 (297h0m56.941129732s) [upload=332666]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 18 20:01:02.340259299 (297h0m59.937030329s) [seg_pos={0 0}, upload=332666]: Begin upload segment... Mar 18 20:01:02.340368743 (297h0m59.937139767s) [seg_pos={0 0}, upload=332666]: Done begin upload segment. Mar 18 20:01:02.340479563 (297h0m59.93725059s) [noise=true , address=45.91.116.8:4061 , piece_id=6XFVKT4AAO7QN457OQMRKPI6R24TJTAN47UXLEVOEFQVOS6M6C3Q, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.340523837 (297h0m59.93729486s) [noise=true , address=79.127.205.233:4166 , piece_id=SFGK5YCWHFRVXF6QNHYP2CYJCZLSTQ76OYRVJPWWXQTF2KFSFJKA, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.341468438 (297h0m59.938239447s) [noise=true , address=45.91.116.4:4961 , piece_id=ZMUGQECUVPVHDNKN75XPBET5M2HFXXK4WMTP4V5KEK63JVZCX5EA, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.341520025 (297h0m59.938291058s) [noise=true , address=167.17.72.71:4815 , piece_id=CXGTYNNBILN3TT7I6Z7UI75F2ZCS4QUTLS6KC4FJ5IHS3XOYWDJQ, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.342232992 (297h0m59.939004023s) [noise=true , address=145.79.227.36:4524 , piece_id=3TAV3P55ITTHBAZEUUUTOAHJRGQXJXXVGZJP6AVR2ANVNOR3S54Q, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.342917368 (297h0m59.939688398s) [noise=true , address=193.149.164.71:4297 , piece_id=4IJPXLPCX3LKFOBOSRHK2DTNRKH67K7FXJUKQ3JPRWHS47ZFEIDA, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.343244957 (297h0m59.940015982s) [noise=true , address=145.79.227.33:4892 , piece_id=DQ6ALC3VOAGFIJI3IKR6I7QFYM22C66QWAAXMH6NXI2FIQEZNQYA, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.344270079 (297h0m59.941041117s) [noise=true , address=145.79.227.21:4497 , piece_id=SP6EKJB7W4QXF7OIBVKEFWCPOZFL5IRIEDVAGS2M4QJSO34CUVRQ, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.345025057 (297h0m59.941796084s) [noise=true , address=77.90.31.17:4793 , piece_id=2U4IBUA2GGUGDBXQGIGQKUK4ZR3L7CDKC6MTGIKAUME4F7MFZRLA, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.345115802 (297h0m59.941886836s) [noise=true , address=167.17.72.69:4731 , piece_id=VOEVIKKFJMC2Y3TPQIDMYUP37NLFU7KPRH4HRRSMQLWUKTSLOH2Q, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.345690600 (297h0m59.942461634s) [noise=true , address=77.90.31.13:4648 , piece_id=BPMB5CFOYHFT3BS4ORR6JF6BNFC2AOAJZOSPMG67NAMLGGP5XVEQ, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.346311592 (297h0m59.943082623s) [noise=true , address=79.127.205.234:4202 , piece_id=YY4LYWX26O6RIOBONESNKTUZEFZCRL4UUX3NEBQ53AJV4IEDG5MQ, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.346791142 (297h0m59.943562166s) [noise=true , address=45.91.116.10:4110 , piece_id=UYLMDCWJEJOHGICA37KNH7G5AQXNIPRH5NRYLBUUMGDRVSLTEYHQ, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.347094999 (297h0m59.94386607s) [noise=true , address=145.79.227.17:4987 , piece_id=PT3TWVB65JZFEWVH4IUZEU2N53FQ57QBRWVLWAUFP4UDLY3LWNLQ, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.347781281 (297h0m59.944552305s) [noise=true , address=79.127.219.46:4919 , piece_id=KK6HRIVBMYLIHRHGPQH3AWMNSVHTTIBESHDMJZA72T446RX6QKRQ, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.348263123 (297h0m59.945034137s) [noise=true , address=193.149.164.68:4033 , piece_id=JZPQKPYGO3W3RZZBITQ73X2XEPIBBSLGRMWIVOM4Z3GTV5UFKRTQ, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.348915505 (297h0m59.945686536s) [noise=true , address=145.79.227.7:4312 , piece_id=2FAL6E3RMBTBHZBXAJJVH2N5VGF5VYI3KNZXGNWCMWGBBPAUO2NQ, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.349366298 (297h0m59.946137337s) [noise=true , address=193.149.164.67:4007 , piece_id=HURQGH67I5B6V4YUV3PRZL3KSKN3XTSK3UMCHZRNT6L3W3PXXHPQ, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.429834098 (297h1m0.026605126s) [upload=332666]: Next returned nil segment Mar 18 20:01:02.429853411 (297h1m0.026624442s) [upload=332666]: Waiting for error group managing segments... Mar 18 20:01:02.446867680 (297h1m0.043638704s) [noise=true , address=45.91.116.10:4110 , piece_id=UYLMDCWJEJOHGICA37KNH7G5AQXNIPRH5NRYLBUUMGDRVSLTEYHQ, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil> Mar 18 20:01:02.451665165 (297h1m0.048436196s) [noise=true , address=45.91.116.4:4961 , piece_id=ZMUGQECUVPVHDNKN75XPBET5M2HFXXK4WMTP4V5KEK63JVZCX5EA, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil> Mar 18 20:01:02.452538886 (297h1m0.049309913s) [noise=true , address=77.90.31.17:4793 , piece_id=2U4IBUA2GGUGDBXQGIGQKUK4ZR3L7CDKC6MTGIKAUME4F7MFZRLA, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil> Mar 18 20:01:02.455597040 (297h1m0.052368067s) [noise=true , address=45.91.116.8:4061 , piece_id=6XFVKT4AAO7QN457OQMRKPI6R24TJTAN47UXLEVOEFQVOS6M6C3Q, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil> Mar 18 20:01:02.457848739 (297h1m0.054619766s) [noise=true , address=77.90.31.13:4648 , piece_id=BPMB5CFOYHFT3BS4ORR6JF6BNFC2AOAJZOSPMG67NAMLGGP5XVEQ, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil> Mar 18 20:01:02.461479709 (297h1m0.058250746s) [noise=true , address=79.127.219.46:4919 , piece_id=KK6HRIVBMYLIHRHGPQH3AWMNSVHTTIBESHDMJZA72T446RX6QKRQ, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil> Mar 18 20:01:02.676859466 (297h1m0.27363049s) [noise=true , address=167.17.72.69:4731 , piece_id=VOEVIKKFJMC2Y3TPQIDMYUP37NLFU7KPRH4HRRSMQLWUKTSLOH2Q, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil> Mar 18 20:01:02.909860743 (297h1m0.506631766s) [noise=true , address=79.127.205.234:4202 , piece_id=YY4LYWX26O6RIOBONESNKTUZEFZCRL4UUX3NEBQ53AJV4IEDG5MQ, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil> Mar 18 20:01:02.930427742 (297h1m0.527198754s) [noise=true , address=193.149.164.71:4297 , piece_id=4IJPXLPCX3LKFOBOSRHK2DTNRKH67K7FXJUKQ3JPRWHS47ZFEIDA, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil> Mar 18 20:01:02.944696889 (297h1m0.541467914s) [noise=true , address=193.149.164.67:4007 , piece_id=HURQGH67I5B6V4YUV3PRZL3KSKN3XTSK3UMCHZRNT6L3W3PXXHPQ, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil> Mar 18 20:01:02.951319183 (297h1m0.548090212s) [noise=true , address=193.149.164.68:4033 , piece_id=JZPQKPYGO3W3RZZBITQ73X2XEPIBBSLGRMWIVOM4Z3GTV5UFKRTQ, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil> Mar 18 20:01:02.975619436 (297h1m0.572390467s) [noise=true , address=79.127.205.233:4166 , piece_id=SFGK5YCWHFRVXF6QNHYP2CYJCZLSTQ76OYRVJPWWXQTF2KFSFJKA, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil> Mar 18 20:01:03.819918528 (297h1m1.416689564s) [noise=true , address=167.17.72.71:4815 , piece_id=CXGTYNNBILN3TT7I6Z7UI75F2ZCS4QUTLS6KC4FJ5IHS3XOYWDJQ, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil> Mar 18 20:01:59.307873420 (297h1m56.904644452s) [noise=true , address=145.79.227.7:4312 , piece_id=2FAL6E3RMBTBHZBXAJJVH2N5VGF5VYI3KNZXGNWCMWGBBPAUO2NQ, seg_pos={0 0}, upload=332666]: Done uploading piece. err: ecclient: upload failed (node:12AVp3qn5uwLPqYaZEwzcgoGMJhNga1BUh6bK5k2bxjWgZLQiiC, address:145.79.227.7:4312): protocol: rpc: dial tcp 145.79.227.7:4312: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:01:59.308026092 (297h1m56.904797124s) [noise=true , address=145.79.227.17:4987 , piece_id=PT3TWVB65JZFEWVH4IUZEU2N53FQ57QBRWVLWAUFP4UDLY3LWNLQ, seg_pos={0 0}, upload=332666]: Done uploading piece. err: ecclient: upload failed (node:1XsPVNGYUhKnXkQRfVfjyBRzyvhShSdsrnzXtwe4Xtf5MaHMvH, address:145.79.227.17:4987): protocol: rpc: dial tcp 145.79.227.17:4987: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:01:59.308195864 (297h1m56.904966894s) [noise=true , address=145.79.227.36:4524 , piece_id=3TAV3P55ITTHBAZEUUUTOAHJRGQXJXXVGZJP6AVR2ANVNOR3S54Q, seg_pos={0 0}, upload=332666]: Done uploading piece. err: ecclient: upload failed (node:1xfbCzjSwFDRrR1gPggeN8tNCLbRRWDVzCJ6mqhZsbVSe8d4s4, address:145.79.227.36:4524): protocol: rpc: dial tcp 145.79.227.36:4524: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:01:59.308316037 (297h1m56.90508706s) [noise=true , address=145.79.227.33:4892 , piece_id=DQ6ALC3VOAGFIJI3IKR6I7QFYM22C66QWAAXMH6NXI2FIQEZNQYA, seg_pos={0 0}, upload=332666]: Done uploading piece. err: ecclient: upload failed (node:12RHwFxcUjPsn12uHEAH3fBDJKv1v1tM8mRJXoFrZrG5djdswF5, address:145.79.227.33:4892): protocol: rpc: dial tcp 145.79.227.33:4892: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:01:59.308404403 (297h1m56.905175433s) [noise=true , address=145.79.227.21:4497 , piece_id=SP6EKJB7W4QXF7OIBVKEFWCPOZFL5IRIEDVAGS2M4QJSO34CUVRQ, seg_pos={0 0}, upload=332666]: Done uploading piece. err: ecclient: upload failed (node:12ZBe4we7VeFdthqRxyy1V9uQrA4r8ouNX68xqsk5bD26dkf2Gd, address:145.79.227.21:4497): protocol: context deadline exceeded; context deadline exceeded Mar 18 20:01:59.308437457 (297h1m56.90520847s) [seg_pos={0 0}, upload=332666]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 18 20:01:59.338288467 (297h1m56.93505949s) [upload=332666]: Done uploading segments... Mar 18 20:01:59.338335497 (297h1m56.935106523s) [upload=332666]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 14.97326428s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 7 out of 60 is failed (threshold=3) [...........................XXX..........X...X.........X....X] duration: 1m0.022816258s trace: https://jaeger.dev.storj.io/trace/698fb9649d932a68 FAIL (2026-03-18 20:03:41 UTC) operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 20:02:41.019616017 (297h2m38.616387042s) [upload=332700]: Uploading segments... Mar 18 20:02:41.019826191 (297h2m38.616597205s) [upload=332700]: Got next segment. Inline: false Mar 18 20:02:41.019879916 (297h2m38.61665094s) [upload=332700]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 20:02:41.019885584 (297h2m38.616656611s) [upload=332700]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 18 20:02:43.834160391 (297h2m41.430931414s) [seg_pos={0 0}, upload=332700]: Begin upload segment... Mar 18 20:02:43.834230381 (297h2m41.431001396s) [seg_pos={0 0}, upload=332700]: Done begin upload segment. Mar 18 20:02:43.834262428 (297h2m41.431033451s) [noise=true , address=145.79.227.52:4451 , piece_id=LR6KVRYMDO2HAESG5BFI5RYPNGVARP6O5HEY2UP77YSTVRKBNUXQ, seg_pos={0 0}, upload=332700]: Uploading piece... Mar 18 20:02:43.834329472 (297h2m41.431100497s) [noise=true , address=45.91.116.13:4164 , piece_id=MNQLVANO3LFKBCJMADD5RULARMRO4VUQY7LT63TU2CCNO765GW7Q, seg_pos={0 0}, upload=332700]: Uploading piece... Mar 18 20:02:43.834938832 (297h2m41.431709858s) [noise=true , address=77.90.31.15:4716 , piece_id=IDQP2MIPCKVS2FGVVGKAIDYTNUIPE56EER5IYZUSOTRD5LZBSXHA, seg_pos={0 0}, upload=332700]: Uploading piece... Mar 18 20:02:43.835898822 (297h2m41.432669851s) [noise=true , address=79.127.205.237:4227 , piece_id=NO25WIYZTEPBOMGETTLSDOJSPKARMKNLVUWDKNXUKHYGDGF2XC5Q, seg_pos={0 0}, upload=332700]: Uploading piece... Mar 18 20:02:43.836893726 (297h2m41.433664759s) [noise=true , address=193.149.164.68:4064 , piece_id=HAA3NDB24MJLAI2HK2UDZQZRLDUXEF76ZJF2VBE6FLKRWSB27Y3A, seg_pos={0 0}, upload=332700]: Uploading piece... Mar 18 20:02:43.837257949 (297h2m41.434028981s) [noise=true , address=193.149.164.66:4974 , piece_id=XWMBTFEWXVVFIR6IYNQYU35QKGVSQJ7YBVSWZ57QSUZ2VSDCJPRA, seg_pos={0 0}, upload=332700]: Uploading piece... Mar 18 20:02:43.838266356 (297h2m41.435037391s) [noise=true , address=145.79.227.34:4924 , piece_id=BNUESPWRZQBX23K7UGNSMF362AWHILNZ43UITBHVGFXIJ4T6V5HQ, seg_pos={0 0}, upload=332700]: Uploading piece... Mar 18 20:02:43.838688582 (297h2m41.435459605s) [noise=true , address=79.127.205.233:4171 , piece_id=YO42WSHYRB2JJX4RVX653N7U6PYTGIW3KQTNSF4445OTN3UW2TIA, seg_pos={0 0}, upload=332700]: Uploading piece... Mar 18 20:02:43.839263235 (297h2m41.436034258s) [noise=true , address=193.149.164.70:4123 , piece_id=VROUAQKRTKZT42ZDDSCIMKFMRODPKFQ5U5DWDJLVI462BVA5YWXA, seg_pos={0 0}, upload=332700]: Uploading piece... Mar 18 20:02:43.839835928 (297h2m41.436606963s) [noise=true , address=45.91.116.4:4948 , piece_id=ULVH4YAJIYY5RWWAGH5RTSZ3BMAT6WCYYAJPSMSP25LCLZQIPG2A, seg_pos={0 0}, upload=332700]: Uploading piece... Mar 18 20:02:43.840390028 (297h2m41.437161055s) [noise=true , address=45.91.116.5:4319 , piece_id=BY6BRNZNKKQKZNAVJTSWUYASX57P5X7OCKQ433T2ARRBXUFNBRCA, seg_pos={0 0}, upload=332700]: Uploading piece... Mar 18 20:02:43.840790648 (297h2m41.437561682s) [noise=true , address=45.91.116.12:4259 , piece_id=4LBHNF3CIQG5LYEY7RYJGNK5Q62X2TSOAXOXSNHO727OXL5OMA4Q, seg_pos={0 0}, upload=332700]: Uploading piece... Mar 18 20:02:43.841319562 (297h2m41.438090584s) [noise=true , address=145.79.227.29:4966 , piece_id=ZTSKMRBESX4OIZMMIPXRW752W5KQYX57RPDQ22XXPURQTSCAI2YA, seg_pos={0 0}, upload=332700]: Uploading piece... Mar 18 20:02:43.841816046 (297h2m41.438587078s) [noise=true , address=193.149.164.71:4287 , piece_id=FYNZ3QGUB7ZFMUJAMYPMUHXJDS66CJ2VLXPV7256DFCABPJMEOYQ, seg_pos={0 0}, upload=332700]: Uploading piece... Mar 18 20:02:43.842288870 (297h2m41.439059903s) [noise=true , address=79.127.219.41:4941 , piece_id=CQXOXMCJHWJ7FSIXDVYUJZMBFW5FUUOJNKJJOVLZRSK5D6XZXX5Q, seg_pos={0 0}, upload=332700]: Uploading piece... Mar 18 20:02:43.842956832 (297h2m41.439727861s) [noise=true , address=145.79.227.49:4369 , piece_id=CBYHFRBNKSPUNJHAE6S72YOFACWLZKGZIFV6SXWDYMVSA4FBPKQQ, seg_pos={0 0}, upload=332700]: Uploading piece... Mar 18 20:02:43.844386533 (297h2m41.441157559s) [noise=true , address=77.90.31.10:4533 , piece_id=VB5BKAWDUZFVII4ZY3JRVK67VUULPSAHTJVZMXXVATP3SXAR2XHQ, seg_pos={0 0}, upload=332700]: Uploading piece... Mar 18 20:02:43.844868553 (297h2m41.441639576s) [noise=true , address=145.79.227.37:4582 , piece_id=DLOFMRZRG3BPFN7JF73OM5HUV56FLEH2IDDC47UQWWWOTRVRR4HQ, seg_pos={0 0}, upload=332700]: Uploading piece... Mar 18 20:02:43.903559633 (297h2m41.500330666s) [upload=332700]: Next returned nil segment Mar 18 20:02:43.903578892 (297h2m41.500349925s) [upload=332700]: Waiting for error group managing segments... Mar 18 20:02:43.908218461 (297h2m41.504989474s) [noise=true , address=45.91.116.4:4948 , piece_id=ULVH4YAJIYY5RWWAGH5RTSZ3BMAT6WCYYAJPSMSP25LCLZQIPG2A, seg_pos={0 0}, upload=332700]: Done uploading piece. err: <nil> Mar 18 20:02:43.917679744 (297h2m41.514450753s) [noise=true , address=45.91.116.12:4259 , piece_id=4LBHNF3CIQG5LYEY7RYJGNK5Q62X2TSOAXOXSNHO727OXL5OMA4Q, seg_pos={0 0}, upload=332700]: Done uploading piece. err: <nil> Mar 18 20:02:43.924357408 (297h2m41.521128422s) [noise=true , address=77.90.31.10:4533 , piece_id=VB5BKAWDUZFVII4ZY3JRVK67VUULPSAHTJVZMXXVATP3SXAR2XHQ, seg_pos={0 0}, upload=332700]: Done uploading piece. err: <nil> Mar 18 20:02:43.926248407 (297h2m41.52301943s) [noise=true , address=77.90.31.15:4716 , piece_id=IDQP2MIPCKVS2FGVVGKAIDYTNUIPE56EER5IYZUSOTRD5LZBSXHA, seg_pos={0 0}, upload=332700]: Done uploading piece. err: <nil> Mar 18 20:02:43.927212300 (297h2m41.523983323s) [noise=true , address=45.91.116.5:4319 , piece_id=BY6BRNZNKKQKZNAVJTSWUYASX57P5X7OCKQ433T2ARRBXUFNBRCA, seg_pos={0 0}, upload=332700]: Done uploading piece. err: <nil> Mar 18 20:02:43.927791610 (297h2m41.524562634s) [noise=true , address=45.91.116.13:4164 , piece_id=MNQLVANO3LFKBCJMADD5RULARMRO4VUQY7LT63TU2CCNO765GW7Q, seg_pos={0 0}, upload=332700]: Done uploading piece. err: <nil> Mar 18 20:02:44.028504592 (297h2m41.625275611s) [noise=true , address=79.127.219.41:4941 , piece_id=CQXOXMCJHWJ7FSIXDVYUJZMBFW5FUUOJNKJJOVLZRSK5D6XZXX5Q, seg_pos={0 0}, upload=332700]: Done uploading piece. err: <nil> Mar 18 20:02:44.392266647 (297h2m41.989037671s) [noise=true , address=79.127.205.233:4171 , piece_id=YO42WSHYRB2JJX4RVX653N7U6PYTGIW3KQTNSF4445OTN3UW2TIA, seg_pos={0 0}, upload=332700]: Done uploading piece. err: <nil> Mar 18 20:02:44.429403642 (297h2m42.026174669s) [noise=true , address=193.149.164.66:4974 , piece_id=XWMBTFEWXVVFIR6IYNQYU35QKGVSQJ7YBVSWZ57QSUZ2VSDCJPRA, seg_pos={0 0}, upload=332700]: Done uploading piece. err: <nil> Mar 18 20:02:44.433374828 (297h2m42.030145853s) [noise=true , address=193.149.164.68:4064 , piece_id=HAA3NDB24MJLAI2HK2UDZQZRLDUXEF76ZJF2VBE6FLKRWSB27Y3A, seg_pos={0 0}, upload=332700]: Done uploading piece. err: <nil> Mar 18 20:02:44.444389014 (297h2m42.041160042s) [noise=true , address=193.149.164.71:4287 , piece_id=FYNZ3QGUB7ZFMUJAMYPMUHXJDS66CJ2VLXPV7256DFCABPJMEOYQ, seg_pos={0 0}, upload=332700]: Done uploading piece. err: <nil> Mar 18 20:02:44.578333498 (297h2m42.175104529s) [noise=true , address=79.127.205.237:4227 , piece_id=NO25WIYZTEPBOMGETTLSDOJSPKARMKNLVUWDKNXUKHYGDGF2XC5Q, seg_pos={0 0}, upload=332700]: Done uploading piece. err: <nil> Mar 18 20:02:44.665665140 (297h2m42.262436168s) [noise=true , address=193.149.164.70:4123 , piece_id=VROUAQKRTKZT42ZDDSCIMKFMRODPKFQ5U5DWDJLVI462BVA5YWXA, seg_pos={0 0}, upload=332700]: Done uploading piece. err: <nil> Mar 18 20:03:40.987108256 (297h3m38.58387928s) [noise=true , address=145.79.227.49:4369 , piece_id=CBYHFRBNKSPUNJHAE6S72YOFACWLZKGZIFV6SXWDYMVSA4FBPKQQ, seg_pos={0 0}, upload=332700]: Done uploading piece. err: ecclient: upload failed (node:12nLksGXGvDwuTWfnmkceuwZNDmrF17iNepHPdYkkvZG2QQUwG7, address:145.79.227.49:4369): protocol: rpc: dial tcp 145.79.227.49:4369: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:03:40.987479807 (297h3m38.58425084s) [noise=true , address=145.79.227.29:4966 , piece_id=ZTSKMRBESX4OIZMMIPXRW752W5KQYX57RPDQ22XXPURQTSCAI2YA, seg_pos={0 0}, upload=332700]: Done uploading piece. err: ecclient: upload failed (node:1DLnsYVdudE4mY9dNAQKhBRppUEAHvxbf84KX69451pwxSxzZK, address:145.79.227.29:4966): protocol: rpc: dial tcp 145.79.227.29:4966: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:03:40.987517864 (297h3m38.584288896s) [noise=true , address=145.79.227.34:4924 , piece_id=BNUESPWRZQBX23K7UGNSMF362AWHILNZ43UITBHVGFXIJ4T6V5HQ, seg_pos={0 0}, upload=332700]: Done uploading piece. err: ecclient: upload failed (node:19TbrQjuN1EhBNVtha9EcKPSCAWyttfmHpec6QeF1S5zW6DRZs, address:145.79.227.34:4924): protocol: rpc: dial tcp 145.79.227.34:4924: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:03:40.987933296 (297h3m38.584704329s) [noise=true , address=145.79.227.37:4582 , piece_id=DLOFMRZRG3BPFN7JF73OM5HUV56FLEH2IDDC47UQWWWOTRVRR4HQ, seg_pos={0 0}, upload=332700]: Done uploading piece. err: ecclient: upload failed (node:129o91j97t4V5AjaouCCy2wDuHgwF94iQiJh2h9ux1aFo7AVCSn, address:145.79.227.37:4582): protocol: rpc: dial tcp 145.79.227.37:4582: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:03:40.988167350 (297h3m38.584938376s) [noise=true , address=145.79.227.52:4451 , piece_id=LR6KVRYMDO2HAESG5BFI5RYPNGVARP6O5HEY2UP77YSTVRKBNUXQ, seg_pos={0 0}, upload=332700]: Done uploading piece. err: ecclient: upload failed (node:12bC4P3xBiPFkxLvavGFVbWKnbprCStL6UJgfKSA2Wd2fX7Mbfw, address:145.79.227.52:4451): protocol: rpc: dial tcp 145.79.227.52:4451: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:03:40.988197047 (297h3m38.58496806s) [seg_pos={0 0}, upload=332700]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 18 20:03:41.007727578 (297h3m38.604498608s) [upload=332700]: Done uploading segments... Mar 18 20:03:41.007779207 (297h3m38.60455025s) [upload=332700]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 1m0.022841994s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 8 out of 60 is failed (threshold=3) [........................XXXXXX.XX...........................] duration: 14.16977593s trace: https://jaeger.dev.storj.io/trace/4b4d10a8051ad01b FAIL (2026-03-18 20:03:40 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 14.169831562s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 37 out of 60 is failed (threshold=3) [......................XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.] duration: 12.315805434s trace: https://jaeger.dev.storj.io/trace/5687315889981ce8 FAIL (2026-03-18 20:03:37 UTC) operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== Mar 18 20:02:51.644702372 (297h2m49.241473391s) [upload=332705]: Starting upload Mar 18 20:02:51.644798111 (297h2m49.241569134s) [upload=332705]: Done starting upload Mar 18 20:02:51.645113854 (297h2m49.241884869s) [upload=332705]: Uploading segments... Mar 18 20:02:51.645238072 (297h2m49.242009104s) [upload=332705]: Got next segment. Inline: false Mar 18 20:02:51.645300317 (297h2m49.242071356s) [upload=332705]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 20:02:51.645306598 (297h2m49.242077631s) [upload=332705]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 18 20:02:54.396505197 (297h2m51.993276213s) [seg_pos={0 0}, upload=332705]: Begin upload segment... Mar 18 20:02:54.396561710 (297h2m51.993332739s) [seg_pos={0 0}, upload=332705]: Done begin upload segment. Mar 18 20:02:54.396603203 (297h2m51.99337423s) [noise=true , address=145.79.227.40:4049 , piece_id=MRAZ3BUALGGG5QDWD4TTEWVNURS4GERXWMJVAIEFJW3RTT2MBIBQ, seg_pos={0 0}, upload=332705]: Uploading piece... Mar 18 20:02:54.396637779 (297h2m51.993408804s) [noise=true , address=45.91.116.9:4264 , piece_id=LDED6USPHNBRH4SXEUL7MISQBR35E6HEN3K73EHNP2PNC7TEI76Q, seg_pos={0 0}, upload=332705]: Uploading piece... Mar 18 20:02:54.397161330 (297h2m51.993932353s) [noise=true , address=77.90.31.14:4665 , piece_id=ASPRJFR6Q25NNYXJZCUB7INVOD3IR7CLRQ4ISYDYQITU4GLFLP6A, seg_pos={0 0}, upload=332705]: Uploading piece... Mar 18 20:02:54.397612650 (297h2m51.994383674s) [noise=true , address=77.90.31.12:4594 , piece_id=6ANB6ACDYWLL6YYGZ47KZF5CZGR3O23O6HU3O5TRRZVQSNFDQPWA, seg_pos={0 0}, upload=332705]: Uploading piece... Mar 18 20:02:54.397642123 (297h2m51.994413145s) [noise=true , address=145.79.227.38:4979 , piece_id=2LNXWHUPDSOD6ENIJYBSKN7KKZQJ7HS4QFFVU7Q55R4LZI3IUHYA, seg_pos={0 0}, upload=332705]: Uploading piece... Mar 18 20:02:54.398038863 (297h2m51.994809877s) [noise=true , address=45.91.116.11:4119 , piece_id=URGCQRIFRSINDMXM33TUPTK5D6PYHP25TWNNOGIEALLAIUQJV6ZA, seg_pos={0 0}, upload=332705]: Uploading piece... Mar 18 20:02:54.398442157 (297h2m51.99521318s) [noise=true , address=77.90.31.18:4810 , piece_id=VIAHWKJYQVIKBYVVPBYHLF4MFYKT4ZXNZ7XGH7N32IMCPICTGXMA, seg_pos={0 0}, upload=332705]: Uploading piece... Mar 18 20:02:54.398978421 (297h2m51.995749435s) [noise=true , address=145.79.227.33:4872 , piece_id=XV4YPUQBLEIUAD3YPRKGMJ3JD6TBPPXRWLWNSFRKOAZVG26SLPBA, seg_pos={0 0}, upload=332705]: Uploading piece... Mar 18 20:02:54.399583059 (297h2m51.996354068s) [noise=true , address=45.91.116.4:4972 , piece_id=GZTYHE32SSKTNG64RB3YGKD76GXHL673MC7DZZDQXMH2RISJIRIQ, seg_pos={0 0}, upload=332705]: Uploading piece... Mar 18 20:02:54.399955533 (297h2m51.99672656s) [noise=true , address=145.79.227.4:4189 , piece_id=EHV52NGZJW2CPO54YNI5I4LW2V32KOZ3SMSDO527KG4CILVS7OWA, seg_pos={0 0}, upload=332705]: Uploading piece... Mar 18 20:02:54.400590874 (297h2m51.99736191s) [noise=true , address=145.79.227.9:4371 , piece_id=CGM5XJXOUDOLOVGFKYODLQXGISBYUUSR3HPIV6V7RS224PBLZDSQ, seg_pos={0 0}, upload=332705]: Uploading piece... Mar 18 20:02:54.400877860 (297h2m51.997648907s) [noise=true , address=45.91.116.8:4048 , piece_id=4VOXISGTUBIQDHO7WAUBXLK3KWATM6J6TORPU3UTIH6LN62EQDPQ, seg_pos={0 0}, upload=332705]: Uploading piece... Mar 18 20:02:54.401283278 (297h2m51.99805431s) [noise=true , address=145.79.227.2:4119 , piece_id=7A5ADO2MBTEHDNL2UAZBP2VXISGWJG4ET5PLPDCUSQZBIKGIBSMA, seg_pos={0 0}, upload=332705]: Uploading piece... Mar 18 20:02:54.401363826 (297h2m51.99813486s) [noise=true , address=79.127.219.41:4927 , piece_id=SBYDPU37E5X2FNQFM3OR7TF7XITJY6DQ7GEDBMXRN4S5QRNRLMKA, seg_pos={0 0}, upload=332705]: Uploading piece... Mar 18 20:02:54.402476102 (297h2m51.999247128s) [noise=true , address=45.91.116.14:4216 , piece_id=2VZGTL6QHH23TMKFTEL3AYC7DYYHYBXKVBSJ6OUEYO352QO43MLA, seg_pos={0 0}, upload=332705]: Uploading piece... Mar 18 20:02:54.403600726 (297h2m52.00037174s) [noise=true , address=45.91.116.7:4026 , piece_id=LJH7X5LXGW33N54YGGRC25D4A2WJCMM3E47BGWEH5RFHF44SH27Q, seg_pos={0 0}, upload=332705]: Uploading piece... Mar 18 20:02:54.477768122 (297h2m52.074539132s) [upload=332705]: Next returned nil segment Mar 18 20:02:54.477782088 (297h2m52.074553111s) [upload=332705]: Waiting for error group managing segments... Mar 18 20:02:54.480685814 (297h2m52.077456826s) [noise=true , address=45.91.116.4:4972 , piece_id=GZTYHE32SSKTNG64RB3YGKD76GXHL673MC7DZZDQXMH2RISJIRIQ, seg_pos={0 0}, upload=332705]: Done uploading piece. err: <nil> Mar 18 20:02:54.481180454 (297h2m52.077951467s) [noise=true , address=45.91.116.9:4264 , piece_id=LDED6USPHNBRH4SXEUL7MISQBR35E6HEN3K73EHNP2PNC7TEI76Q, seg_pos={0 0}, upload=332705]: Done uploading piece. err: <nil> Mar 18 20:02:54.481763004 (297h2m52.078534027s) [noise=true , address=45.91.116.11:4119 , piece_id=URGCQRIFRSINDMXM33TUPTK5D6PYHP25TWNNOGIEALLAIUQJV6ZA, seg_pos={0 0}, upload=332705]: Done uploading piece. err: <nil> Mar 18 20:02:54.482185984 (297h2m52.078957007s) [noise=true , address=77.90.31.18:4810 , piece_id=VIAHWKJYQVIKBYVVPBYHLF4MFYKT4ZXNZ7XGH7N32IMCPICTGXMA, seg_pos={0 0}, upload=332705]: Done uploading piece. err: <nil> Mar 18 20:02:54.482565109 (297h2m52.079336122s) [noise=true , address=45.91.116.7:4026 , piece_id=LJH7X5LXGW33N54YGGRC25D4A2WJCMM3E47BGWEH5RFHF44SH27Q, seg_pos={0 0}, upload=332705]: Done uploading piece. err: <nil> Mar 18 20:02:54.483141108 (297h2m52.079912132s) [noise=true , address=79.127.219.41:4927 , piece_id=SBYDPU37E5X2FNQFM3OR7TF7XITJY6DQ7GEDBMXRN4S5QRNRLMKA, seg_pos={0 0}, upload=332705]: Done uploading piece. err: <nil> Mar 18 20:02:54.483203323 (297h2m52.079974347s) [noise=true , address=45.91.116.8:4048 , piece_id=4VOXISGTUBIQDHO7WAUBXLK3KWATM6J6TORPU3UTIH6LN62EQDPQ, seg_pos={0 0}, upload=332705]: Done uploading piece. err: <nil> Mar 18 20:02:54.483613521 (297h2m52.080384545s) [noise=true , address=77.90.31.14:4665 , piece_id=ASPRJFR6Q25NNYXJZCUB7INVOD3IR7CLRQ4ISYDYQITU4GLFLP6A, seg_pos={0 0}, upload=332705]: Done uploading piece. err: <nil> Mar 18 20:02:54.483950756 (297h2m52.08072178s) [noise=true , address=45.91.116.14:4216 , piece_id=2VZGTL6QHH23TMKFTEL3AYC7DYYHYBXKVBSJ6OUEYO352QO43MLA, seg_pos={0 0}, upload=332705]: Done uploading piece. err: <nil> Mar 18 20:02:54.484098547 (297h2m52.080869561s) [noise=true , address=77.90.31.12:4594 , piece_id=6ANB6ACDYWLL6YYGZ47KZF5CZGR3O23O6HU3O5TRRZVQSNFDQPWA, seg_pos={0 0}, upload=332705]: Done uploading piece. err: <nil> Mar 18 20:03:21.614050439 (297h3m19.210821472s) [noise=true , address=145.79.227.4:4189 , piece_id=EHV52NGZJW2CPO54YNI5I4LW2V32KOZ3SMSDO527KG4CILVS7OWA, seg_pos={0 0}, upload=332705]: Done uploading piece. err: ecclient: upload failed (node:1jH3757i2F3q18XSpoXznsysZPaXmEnqynpLPDupgRfxJrTgKv, address:145.79.227.4:4189): protocol: rpc: dial tcp 145.79.227.4:4189: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:03:21.614446336 (297h3m19.211217371s) [noise=true , address=145.79.227.38:4979 , piece_id=2LNXWHUPDSOD6ENIJYBSKN7KKZQJ7HS4QFFVU7Q55R4LZI3IUHYA, seg_pos={0 0}, upload=332705]: Done uploading piece. err: ecclient: upload failed (node:12B518TvbbRhj6LiXxV4GwUgAoUYmWMWeTpyYerKnKuZYBjXoma, address:145.79.227.38:4979): protocol: rpc: dial tcp 145.79.227.38:4979: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:03:21.614793963 (297h3m19.211564998s) [noise=true , address=145.79.227.33:4872 , piece_id=XV4YPUQBLEIUAD3YPRKGMJ3JD6TBPPXRWLWNSFRKOAZVG26SLPBA, seg_pos={0 0}, upload=332705]: Done uploading piece. err: ecclient: upload failed (node:12dNdm2kAok8jFgbYE24oXUL9KCJVDDiinLUJgDSYEY3fTEBJec, address:145.79.227.33:4872): protocol: rpc: dial tcp 145.79.227.33:4872: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:03:21.615486238 (297h3m19.212257273s) [noise=true , address=145.79.227.40:4049 , piece_id=MRAZ3BUALGGG5QDWD4TTEWVNURS4GERXWMJVAIEFJW3RTT2MBIBQ, seg_pos={0 0}, upload=332705]: Done uploading piece. err: ecclient: upload failed (node:1n91vwhc11FPE57QAUCfgDpffrmPrJxWxh7R9ecgt41JEPux6J, address:145.79.227.40:4049): protocol: rpc: dial tcp 145.79.227.40:4049: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:03:21.615753593 (297h3m19.212524626s) [noise=true , address=145.79.227.9:4371 , piece_id=CGM5XJXOUDOLOVGFKYODLQXGISBYUUSR3HPIV6V7RS224PBLZDSQ, seg_pos={0 0}, upload=332705]: Done uploading piece. err: ecclient: upload failed (node:12cSqcGZqvZqs1QKBi6WK4fUDKgvQcmTupR1C9ZkWz879XeWPEL, address:145.79.227.9:4371): protocol: rpc: dial tcp 145.79.227.9:4371: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:03:21.616646748 (297h3m19.213417781s) [noise=true , address=145.79.227.2:4119 , piece_id=7A5ADO2MBTEHDNL2UAZBP2VXISGWJG4ET5PLPDCUSQZBIKGIBSMA, seg_pos={0 0}, upload=332705]: Done uploading piece. err: ecclient: upload failed (node:121xNAnkm9HZXUszp26UmWTsuhp3igyeDVea6gSCdtrQUQDVHpW, address:145.79.227.2:4119): protocol: rpc: dial tcp 145.79.227.2:4119: i/o timeout; conn closed: op Mar 18 20:03:21.616687674 (297h3m19.213458692s) [seg_pos={0 0}, upload=332705]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Mar 18 20:03:21.636036019 (297h3m19.232807043s) [upload=332705]: Done uploading segments... Mar 18 20:03:21.636087341 (297h3m19.232858366s) [upload=332705]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration: 12.315839413s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 46 out of 60 is failed (threshold=6) [..........XXXXXXXXX.XXX...XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 5.001338876s trace: https://jaeger.dev.storj.io/trace/396e2a4fdfd19f39 FAIL (2026-03-18 20:03:32 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001418723s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 37 out of 60 is failed (threshold=3) [.......................XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 30.025177692s trace: https://jaeger.dev.storj.io/trace/5687315889981ce8 FAIL (2026-03-18 20:03:21 UTC) operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== Mar 18 20:02:51.644702372 (297h2m49.241473391s) [upload=332705]: Starting upload Mar 18 20:02:51.644798111 (297h2m49.241569134s) [upload=332705]: Done starting upload Mar 18 20:02:51.645113854 (297h2m49.241884869s) [upload=332705]: Uploading segments... Mar 18 20:02:51.645238072 (297h2m49.242009104s) [upload=332705]: Got next segment. Inline: false Mar 18 20:02:51.645300317 (297h2m49.242071356s) [upload=332705]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 20:02:51.645306598 (297h2m49.242077631s) [upload=332705]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 18 20:02:54.396505197 (297h2m51.993276213s) [seg_pos={0 0}, upload=332705]: Begin upload segment... Mar 18 20:02:54.396561710 (297h2m51.993332739s) [seg_pos={0 0}, upload=332705]: Done begin upload segment. Mar 18 20:02:54.396603203 (297h2m51.99337423s) [noise=true , address=145.79.227.40:4049 , piece_id=MRAZ3BUALGGG5QDWD4TTEWVNURS4GERXWMJVAIEFJW3RTT2MBIBQ, seg_pos={0 0}, upload=332705]: Uploading piece... Mar 18 20:02:54.396637779 (297h2m51.993408804s) [noise=true , address=45.91.116.9:4264 , piece_id=LDED6USPHNBRH4SXEUL7MISQBR35E6HEN3K73EHNP2PNC7TEI76Q, seg_pos={0 0}, upload=332705]: Uploading piece... Mar 18 20:02:54.397161330 (297h2m51.993932353s) [noise=true , address=77.90.31.14:4665 , piece_id=ASPRJFR6Q25NNYXJZCUB7INVOD3IR7CLRQ4ISYDYQITU4GLFLP6A, seg_pos={0 0}, upload=332705]: Uploading piece... Mar 18 20:02:54.397612650 (297h2m51.994383674s) [noise=true , address=77.90.31.12:4594 , piece_id=6ANB6ACDYWLL6YYGZ47KZF5CZGR3O23O6HU3O5TRRZVQSNFDQPWA, seg_pos={0 0}, upload=332705]: Uploading piece... Mar 18 20:02:54.397642123 (297h2m51.994413145s) [noise=true , address=145.79.227.38:4979 , piece_id=2LNXWHUPDSOD6ENIJYBSKN7KKZQJ7HS4QFFVU7Q55R4LZI3IUHYA, seg_pos={0 0}, upload=332705]: Uploading piece... Mar 18 20:02:54.398038863 (297h2m51.994809877s) [noise=true , address=45.91.116.11:4119 , piece_id=URGCQRIFRSINDMXM33TUPTK5D6PYHP25TWNNOGIEALLAIUQJV6ZA, seg_pos={0 0}, upload=332705]: Uploading piece... Mar 18 20:02:54.398442157 (297h2m51.99521318s) [noise=true , address=77.90.31.18:4810 , piece_id=VIAHWKJYQVIKBYVVPBYHLF4MFYKT4ZXNZ7XGH7N32IMCPICTGXMA, seg_pos={0 0}, upload=332705]: Uploading piece... Mar 18 20:02:54.398978421 (297h2m51.995749435s) [noise=true , address=145.79.227.33:4872 , piece_id=XV4YPUQBLEIUAD3YPRKGMJ3JD6TBPPXRWLWNSFRKOAZVG26SLPBA, seg_pos={0 0}, upload=332705]: Uploading piece... Mar 18 20:02:54.399583059 (297h2m51.996354068s) [noise=true , address=45.91.116.4:4972 , piece_id=GZTYHE32SSKTNG64RB3YGKD76GXHL673MC7DZZDQXMH2RISJIRIQ, seg_pos={0 0}, upload=332705]: Uploading piece... Mar 18 20:02:54.399955533 (297h2m51.99672656s) [noise=true , address=145.79.227.4:4189 , piece_id=EHV52NGZJW2CPO54YNI5I4LW2V32KOZ3SMSDO527KG4CILVS7OWA, seg_pos={0 0}, upload=332705]: Uploading piece... Mar 18 20:02:54.400590874 (297h2m51.99736191s) [noise=true , address=145.79.227.9:4371 , piece_id=CGM5XJXOUDOLOVGFKYODLQXGISBYUUSR3HPIV6V7RS224PBLZDSQ, seg_pos={0 0}, upload=332705]: Uploading piece... Mar 18 20:02:54.400877860 (297h2m51.997648907s) [noise=true , address=45.91.116.8:4048 , piece_id=4VOXISGTUBIQDHO7WAUBXLK3KWATM6J6TORPU3UTIH6LN62EQDPQ, seg_pos={0 0}, upload=332705]: Uploading piece... Mar 18 20:02:54.401283278 (297h2m51.99805431s) [noise=true , address=145.79.227.2:4119 , piece_id=7A5ADO2MBTEHDNL2UAZBP2VXISGWJG4ET5PLPDCUSQZBIKGIBSMA, seg_pos={0 0}, upload=332705]: Uploading piece... Mar 18 20:02:54.401363826 (297h2m51.99813486s) [noise=true , address=79.127.219.41:4927 , piece_id=SBYDPU37E5X2FNQFM3OR7TF7XITJY6DQ7GEDBMXRN4S5QRNRLMKA, seg_pos={0 0}, upload=332705]: Uploading piece... Mar 18 20:02:54.402476102 (297h2m51.999247128s) [noise=true , address=45.91.116.14:4216 , piece_id=2VZGTL6QHH23TMKFTEL3AYC7DYYHYBXKVBSJ6OUEYO352QO43MLA, seg_pos={0 0}, upload=332705]: Uploading piece... Mar 18 20:02:54.403600726 (297h2m52.00037174s) [noise=true , address=45.91.116.7:4026 , piece_id=LJH7X5LXGW33N54YGGRC25D4A2WJCMM3E47BGWEH5RFHF44SH27Q, seg_pos={0 0}, upload=332705]: Uploading piece... Mar 18 20:02:54.477768122 (297h2m52.074539132s) [upload=332705]: Next returned nil segment Mar 18 20:02:54.477782088 (297h2m52.074553111s) [upload=332705]: Waiting for error group managing segments... Mar 18 20:02:54.480685814 (297h2m52.077456826s) [noise=true , address=45.91.116.4:4972 , piece_id=GZTYHE32SSKTNG64RB3YGKD76GXHL673MC7DZZDQXMH2RISJIRIQ, seg_pos={0 0}, upload=332705]: Done uploading piece. err: <nil> Mar 18 20:02:54.481180454 (297h2m52.077951467s) [noise=true , address=45.91.116.9:4264 , piece_id=LDED6USPHNBRH4SXEUL7MISQBR35E6HEN3K73EHNP2PNC7TEI76Q, seg_pos={0 0}, upload=332705]: Done uploading piece. err: <nil> Mar 18 20:02:54.481763004 (297h2m52.078534027s) [noise=true , address=45.91.116.11:4119 , piece_id=URGCQRIFRSINDMXM33TUPTK5D6PYHP25TWNNOGIEALLAIUQJV6ZA, seg_pos={0 0}, upload=332705]: Done uploading piece. err: <nil> Mar 18 20:02:54.482185984 (297h2m52.078957007s) [noise=true , address=77.90.31.18:4810 , piece_id=VIAHWKJYQVIKBYVVPBYHLF4MFYKT4ZXNZ7XGH7N32IMCPICTGXMA, seg_pos={0 0}, upload=332705]: Done uploading piece. err: <nil> Mar 18 20:02:54.482565109 (297h2m52.079336122s) [noise=true , address=45.91.116.7:4026 , piece_id=LJH7X5LXGW33N54YGGRC25D4A2WJCMM3E47BGWEH5RFHF44SH27Q, seg_pos={0 0}, upload=332705]: Done uploading piece. err: <nil> Mar 18 20:02:54.483141108 (297h2m52.079912132s) [noise=true , address=79.127.219.41:4927 , piece_id=SBYDPU37E5X2FNQFM3OR7TF7XITJY6DQ7GEDBMXRN4S5QRNRLMKA, seg_pos={0 0}, upload=332705]: Done uploading piece. err: <nil> Mar 18 20:02:54.483203323 (297h2m52.079974347s) [noise=true , address=45.91.116.8:4048 , piece_id=4VOXISGTUBIQDHO7WAUBXLK3KWATM6J6TORPU3UTIH6LN62EQDPQ, seg_pos={0 0}, upload=332705]: Done uploading piece. err: <nil> Mar 18 20:02:54.483613521 (297h2m52.080384545s) [noise=true , address=77.90.31.14:4665 , piece_id=ASPRJFR6Q25NNYXJZCUB7INVOD3IR7CLRQ4ISYDYQITU4GLFLP6A, seg_pos={0 0}, upload=332705]: Done uploading piece. err: <nil> Mar 18 20:02:54.483950756 (297h2m52.08072178s) [noise=true , address=45.91.116.14:4216 , piece_id=2VZGTL6QHH23TMKFTEL3AYC7DYYHYBXKVBSJ6OUEYO352QO43MLA, seg_pos={0 0}, upload=332705]: Done uploading piece. err: <nil> Mar 18 20:02:54.484098547 (297h2m52.080869561s) [noise=true , address=77.90.31.12:4594 , piece_id=6ANB6ACDYWLL6YYGZ47KZF5CZGR3O23O6HU3O5TRRZVQSNFDQPWA, seg_pos={0 0}, upload=332705]: Done uploading piece. err: <nil> Mar 18 20:03:21.614050439 (297h3m19.210821472s) [noise=true , address=145.79.227.4:4189 , piece_id=EHV52NGZJW2CPO54YNI5I4LW2V32KOZ3SMSDO527KG4CILVS7OWA, seg_pos={0 0}, upload=332705]: Done uploading piece. err: ecclient: upload failed (node:1jH3757i2F3q18XSpoXznsysZPaXmEnqynpLPDupgRfxJrTgKv, address:145.79.227.4:4189): protocol: rpc: dial tcp 145.79.227.4:4189: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:03:21.614446336 (297h3m19.211217371s) [noise=true , address=145.79.227.38:4979 , piece_id=2LNXWHUPDSOD6ENIJYBSKN7KKZQJ7HS4QFFVU7Q55R4LZI3IUHYA, seg_pos={0 0}, upload=332705]: Done uploading piece. err: ecclient: upload failed (node:12B518TvbbRhj6LiXxV4GwUgAoUYmWMWeTpyYerKnKuZYBjXoma, address:145.79.227.38:4979): protocol: rpc: dial tcp 145.79.227.38:4979: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:03:21.614793963 (297h3m19.211564998s) [noise=true , address=145.79.227.33:4872 , piece_id=XV4YPUQBLEIUAD3YPRKGMJ3JD6TBPPXRWLWNSFRKOAZVG26SLPBA, seg_pos={0 0}, upload=332705]: Done uploading piece. err: ecclient: upload failed (node:12dNdm2kAok8jFgbYE24oXUL9KCJVDDiinLUJgDSYEY3fTEBJec, address:145.79.227.33:4872): protocol: rpc: dial tcp 145.79.227.33:4872: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:03:21.615486238 (297h3m19.212257273s) [noise=true , address=145.79.227.40:4049 , piece_id=MRAZ3BUALGGG5QDWD4TTEWVNURS4GERXWMJVAIEFJW3RTT2MBIBQ, seg_pos={0 0}, upload=332705]: Done uploading piece. err: ecclient: upload failed (node:1n91vwhc11FPE57QAUCfgDpffrmPrJxWxh7R9ecgt41JEPux6J, address:145.79.227.40:4049): protocol: rpc: dial tcp 145.79.227.40:4049: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:03:21.615753593 (297h3m19.212524626s) [noise=true , address=145.79.227.9:4371 , piece_id=CGM5XJXOUDOLOVGFKYODLQXGISBYUUSR3HPIV6V7RS224PBLZDSQ, seg_pos={0 0}, upload=332705]: Done uploading piece. err: ecclient: upload failed (node:12cSqcGZqvZqs1QKBi6WK4fUDKgvQcmTupR1C9ZkWz879XeWPEL, address:145.79.227.9:4371): protocol: rpc: dial tcp 145.79.227.9:4371: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:03:21.616646748 (297h3m19.213417781s) [noise=true , address=145.79.227.2:4119 , piece_id=7A5ADO2MBTEHDNL2UAZBP2VXISGWJG4ET5PLPDCUSQZBIKGIBSMA, seg_pos={0 0}, upload=332705]: Done uploading piece. err: ecclient: upload failed (node:121xNAnkm9HZXUszp26UmWTsuhp3igyeDVea6gSCdtrQUQDVHpW, address:145.79.227.2:4119): protocol: rpc: dial tcp 145.79.227.2:4119: i/o timeout; conn closed: op Mar 18 20:03:21.616687674 (297h3m19.213458692s) [seg_pos={0 0}, upload=332705]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Mar 18 20:03:21.636036019 (297h3m19.232807043s) [upload=332705]: Done uploading segments... Mar 18 20:03:21.636087341 (297h3m19.232858366s) [upload=332705]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration: 30.025193035s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 6 out of 60 is failed (threshold=3) [............................X..........XX..X....X........X..] duration: 15.821947039s trace: https://jaeger.dev.storj.io/trace/589419efc52e5e2b FAIL (2026-03-18 20:03:11 UTC) operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== Mar 18 20:00:59.343084966 (297h0m56.939855989s) [upload=332666]: Starting upload Mar 18 20:00:59.343175518 (297h0m56.939946548s) [upload=332666]: Done starting upload Mar 18 20:00:59.343565585 (297h0m56.940336612s) [upload=332666]: Uploading segments... Mar 18 20:00:59.344256647 (297h0m56.941027676s) [upload=332666]: Got next segment. Inline: false Mar 18 20:00:59.344352215 (297h0m56.941123241s) [upload=332666]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 20:00:59.344358706 (297h0m56.941129732s) [upload=332666]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 18 20:01:02.340259299 (297h0m59.937030329s) [seg_pos={0 0}, upload=332666]: Begin upload segment... Mar 18 20:01:02.340368743 (297h0m59.937139767s) [seg_pos={0 0}, upload=332666]: Done begin upload segment. Mar 18 20:01:02.340479563 (297h0m59.93725059s) [noise=true , address=45.91.116.8:4061 , piece_id=6XFVKT4AAO7QN457OQMRKPI6R24TJTAN47UXLEVOEFQVOS6M6C3Q, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.340523837 (297h0m59.93729486s) [noise=true , address=79.127.205.233:4166 , piece_id=SFGK5YCWHFRVXF6QNHYP2CYJCZLSTQ76OYRVJPWWXQTF2KFSFJKA, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.341468438 (297h0m59.938239447s) [noise=true , address=45.91.116.4:4961 , piece_id=ZMUGQECUVPVHDNKN75XPBET5M2HFXXK4WMTP4V5KEK63JVZCX5EA, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.341520025 (297h0m59.938291058s) [noise=true , address=167.17.72.71:4815 , piece_id=CXGTYNNBILN3TT7I6Z7UI75F2ZCS4QUTLS6KC4FJ5IHS3XOYWDJQ, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.342232992 (297h0m59.939004023s) [noise=true , address=145.79.227.36:4524 , piece_id=3TAV3P55ITTHBAZEUUUTOAHJRGQXJXXVGZJP6AVR2ANVNOR3S54Q, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.342917368 (297h0m59.939688398s) [noise=true , address=193.149.164.71:4297 , piece_id=4IJPXLPCX3LKFOBOSRHK2DTNRKH67K7FXJUKQ3JPRWHS47ZFEIDA, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.343244957 (297h0m59.940015982s) [noise=true , address=145.79.227.33:4892 , piece_id=DQ6ALC3VOAGFIJI3IKR6I7QFYM22C66QWAAXMH6NXI2FIQEZNQYA, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.344270079 (297h0m59.941041117s) [noise=true , address=145.79.227.21:4497 , piece_id=SP6EKJB7W4QXF7OIBVKEFWCPOZFL5IRIEDVAGS2M4QJSO34CUVRQ, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.345025057 (297h0m59.941796084s) [noise=true , address=77.90.31.17:4793 , piece_id=2U4IBUA2GGUGDBXQGIGQKUK4ZR3L7CDKC6MTGIKAUME4F7MFZRLA, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.345115802 (297h0m59.941886836s) [noise=true , address=167.17.72.69:4731 , piece_id=VOEVIKKFJMC2Y3TPQIDMYUP37NLFU7KPRH4HRRSMQLWUKTSLOH2Q, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.345690600 (297h0m59.942461634s) [noise=true , address=77.90.31.13:4648 , piece_id=BPMB5CFOYHFT3BS4ORR6JF6BNFC2AOAJZOSPMG67NAMLGGP5XVEQ, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.346311592 (297h0m59.943082623s) [noise=true , address=79.127.205.234:4202 , piece_id=YY4LYWX26O6RIOBONESNKTUZEFZCRL4UUX3NEBQ53AJV4IEDG5MQ, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.346791142 (297h0m59.943562166s) [noise=true , address=45.91.116.10:4110 , piece_id=UYLMDCWJEJOHGICA37KNH7G5AQXNIPRH5NRYLBUUMGDRVSLTEYHQ, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.347094999 (297h0m59.94386607s) [noise=true , address=145.79.227.17:4987 , piece_id=PT3TWVB65JZFEWVH4IUZEU2N53FQ57QBRWVLWAUFP4UDLY3LWNLQ, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.347781281 (297h0m59.944552305s) [noise=true , address=79.127.219.46:4919 , piece_id=KK6HRIVBMYLIHRHGPQH3AWMNSVHTTIBESHDMJZA72T446RX6QKRQ, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.348263123 (297h0m59.945034137s) [noise=true , address=193.149.164.68:4033 , piece_id=JZPQKPYGO3W3RZZBITQ73X2XEPIBBSLGRMWIVOM4Z3GTV5UFKRTQ, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.348915505 (297h0m59.945686536s) [noise=true , address=145.79.227.7:4312 , piece_id=2FAL6E3RMBTBHZBXAJJVH2N5VGF5VYI3KNZXGNWCMWGBBPAUO2NQ, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.349366298 (297h0m59.946137337s) [noise=true , address=193.149.164.67:4007 , piece_id=HURQGH67I5B6V4YUV3PRZL3KSKN3XTSK3UMCHZRNT6L3W3PXXHPQ, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.429834098 (297h1m0.026605126s) [upload=332666]: Next returned nil segment Mar 18 20:01:02.429853411 (297h1m0.026624442s) [upload=332666]: Waiting for error group managing segments... Mar 18 20:01:02.446867680 (297h1m0.043638704s) [noise=true , address=45.91.116.10:4110 , piece_id=UYLMDCWJEJOHGICA37KNH7G5AQXNIPRH5NRYLBUUMGDRVSLTEYHQ, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil> Mar 18 20:01:02.451665165 (297h1m0.048436196s) [noise=true , address=45.91.116.4:4961 , piece_id=ZMUGQECUVPVHDNKN75XPBET5M2HFXXK4WMTP4V5KEK63JVZCX5EA, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil> Mar 18 20:01:02.452538886 (297h1m0.049309913s) [noise=true , address=77.90.31.17:4793 , piece_id=2U4IBUA2GGUGDBXQGIGQKUK4ZR3L7CDKC6MTGIKAUME4F7MFZRLA, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil> Mar 18 20:01:02.455597040 (297h1m0.052368067s) [noise=true , address=45.91.116.8:4061 , piece_id=6XFVKT4AAO7QN457OQMRKPI6R24TJTAN47UXLEVOEFQVOS6M6C3Q, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil> Mar 18 20:01:02.457848739 (297h1m0.054619766s) [noise=true , address=77.90.31.13:4648 , piece_id=BPMB5CFOYHFT3BS4ORR6JF6BNFC2AOAJZOSPMG67NAMLGGP5XVEQ, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil> Mar 18 20:01:02.461479709 (297h1m0.058250746s) [noise=true , address=79.127.219.46:4919 , piece_id=KK6HRIVBMYLIHRHGPQH3AWMNSVHTTIBESHDMJZA72T446RX6QKRQ, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil> Mar 18 20:01:02.676859466 (297h1m0.27363049s) [noise=true , address=167.17.72.69:4731 , piece_id=VOEVIKKFJMC2Y3TPQIDMYUP37NLFU7KPRH4HRRSMQLWUKTSLOH2Q, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil> Mar 18 20:01:02.909860743 (297h1m0.506631766s) [noise=true , address=79.127.205.234:4202 , piece_id=YY4LYWX26O6RIOBONESNKTUZEFZCRL4UUX3NEBQ53AJV4IEDG5MQ, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil> Mar 18 20:01:02.930427742 (297h1m0.527198754s) [noise=true , address=193.149.164.71:4297 , piece_id=4IJPXLPCX3LKFOBOSRHK2DTNRKH67K7FXJUKQ3JPRWHS47ZFEIDA, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil> Mar 18 20:01:02.944696889 (297h1m0.541467914s) [noise=true , address=193.149.164.67:4007 , piece_id=HURQGH67I5B6V4YUV3PRZL3KSKN3XTSK3UMCHZRNT6L3W3PXXHPQ, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil> Mar 18 20:01:02.951319183 (297h1m0.548090212s) [noise=true , address=193.149.164.68:4033 , piece_id=JZPQKPYGO3W3RZZBITQ73X2XEPIBBSLGRMWIVOM4Z3GTV5UFKRTQ, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil> Mar 18 20:01:02.975619436 (297h1m0.572390467s) [noise=true , address=79.127.205.233:4166 , piece_id=SFGK5YCWHFRVXF6QNHYP2CYJCZLSTQ76OYRVJPWWXQTF2KFSFJKA, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil> Mar 18 20:01:03.819918528 (297h1m1.416689564s) [noise=true , address=167.17.72.71:4815 , piece_id=CXGTYNNBILN3TT7I6Z7UI75F2ZCS4QUTLS6KC4FJ5IHS3XOYWDJQ, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil> Mar 18 20:01:59.307873420 (297h1m56.904644452s) [noise=true , address=145.79.227.7:4312 , piece_id=2FAL6E3RMBTBHZBXAJJVH2N5VGF5VYI3KNZXGNWCMWGBBPAUO2NQ, seg_pos={0 0}, upload=332666]: Done uploading piece. err: ecclient: upload failed (node:12AVp3qn5uwLPqYaZEwzcgoGMJhNga1BUh6bK5k2bxjWgZLQiiC, address:145.79.227.7:4312): protocol: rpc: dial tcp 145.79.227.7:4312: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:01:59.308026092 (297h1m56.904797124s) [noise=true , address=145.79.227.17:4987 , piece_id=PT3TWVB65JZFEWVH4IUZEU2N53FQ57QBRWVLWAUFP4UDLY3LWNLQ, seg_pos={0 0}, upload=332666]: Done uploading piece. err: ecclient: upload failed (node:1XsPVNGYUhKnXkQRfVfjyBRzyvhShSdsrnzXtwe4Xtf5MaHMvH, address:145.79.227.17:4987): protocol: rpc: dial tcp 145.79.227.17:4987: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:01:59.308195864 (297h1m56.904966894s) [noise=true , address=145.79.227.36:4524 , piece_id=3TAV3P55ITTHBAZEUUUTOAHJRGQXJXXVGZJP6AVR2ANVNOR3S54Q, seg_pos={0 0}, upload=332666]: Done uploading piece. err: ecclient: upload failed (node:1xfbCzjSwFDRrR1gPggeN8tNCLbRRWDVzCJ6mqhZsbVSe8d4s4, address:145.79.227.36:4524): protocol: rpc: dial tcp 145.79.227.36:4524: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:01:59.308316037 (297h1m56.90508706s) [noise=true , address=145.79.227.33:4892 , piece_id=DQ6ALC3VOAGFIJI3IKR6I7QFYM22C66QWAAXMH6NXI2FIQEZNQYA, seg_pos={0 0}, upload=332666]: Done uploading piece. err: ecclient: upload failed (node:12RHwFxcUjPsn12uHEAH3fBDJKv1v1tM8mRJXoFrZrG5djdswF5, address:145.79.227.33:4892): protocol: rpc: dial tcp 145.79.227.33:4892: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:01:59.308404403 (297h1m56.905175433s) [noise=true , address=145.79.227.21:4497 , piece_id=SP6EKJB7W4QXF7OIBVKEFWCPOZFL5IRIEDVAGS2M4QJSO34CUVRQ, seg_pos={0 0}, upload=332666]: Done uploading piece. err: ecclient: upload failed (node:12ZBe4we7VeFdthqRxyy1V9uQrA4r8ouNX68xqsk5bD26dkf2Gd, address:145.79.227.21:4497): protocol: context deadline exceeded; context deadline exceeded Mar 18 20:01:59.308437457 (297h1m56.90520847s) [seg_pos={0 0}, upload=332666]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 18 20:01:59.338288467 (297h1m56.93505949s) [upload=332666]: Done uploading segments... Mar 18 20:01:59.338335497 (297h1m56.935106523s) [upload=332666]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 15.82197849s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 8 out of 60 is failed (threshold=3) [.........................XXXXXX.XX..........................] duration: 14.87605267s trace: https://jaeger.dev.storj.io/trace/4b4d10a8051ad01b FAIL (2026-03-18 20:02:52 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 14.876082177s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 36 out of 60 is failed (threshold=3) [........................XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 30.025575265s trace: https://jaeger.dev.storj.io/trace/3161cb4de8ca6f34 FAIL (2026-03-18 20:02:51 UTC) operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== Mar 18 20:02:21.560928779 (297h2m19.157699795s) [upload=332695]: Starting upload Mar 18 20:02:21.560987579 (297h2m19.157758611s) [upload=332695]: Done starting upload Mar 18 20:02:21.561028601 (297h2m19.157799615s) [upload=332695]: Uploading segments... Mar 18 20:02:21.561235919 (297h2m19.158006934s) [upload=332695]: Got next segment. Inline: false Mar 18 20:02:21.561276655 (297h2m19.15804768s) [upload=332695]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 20:02:21.561282134 (297h2m19.158053157s) [upload=332695]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 18 20:02:24.383881482 (297h2m21.980652512s) [seg_pos={0 0}, upload=332695]: Begin upload segment... Mar 18 20:02:24.383943027 (297h2m21.980714041s) [seg_pos={0 0}, upload=332695]: Done begin upload segment. Mar 18 20:02:24.383996366 (297h2m21.980767389s) [noise=true , address=77.90.31.14:4666 , piece_id=KQJQAQAV7CCMYAZ2VIIS2XVNTGVE4CEN26DECIBZN76D5XH3UTMQ, seg_pos={0 0}, upload=332695]: Uploading piece... Mar 18 20:02:24.384016629 (297h2m21.980787662s) [noise=true , address=145.79.227.10:4407 , piece_id=HCIQ7MQE5INROVA4JWXXBI6NBTH3GZVOUNW4XOWXBC2OXIPGSGKQ, seg_pos={0 0}, upload=332695]: Uploading piece... Mar 18 20:02:24.384755171 (297h2m21.981526205s) [noise=true , address=145.79.227.37:4570 , piece_id=XNYV7FYPAAZ47BQMTCEZZ63U32IRPHEHYXL52Z4COICZFLX4G7NQ, seg_pos={0 0}, upload=332695]: Uploading piece... Mar 18 20:02:24.385473766 (297h2m21.982244795s) [noise=true , address=45.91.116.5:4298 , piece_id=U4LOFGYW46FX7BTJQCO7KBSDUXENWD7ZZFY6GSDJJF2JZVCHX3IQ, seg_pos={0 0}, upload=332695]: Uploading piece... Mar 18 20:02:24.385608981 (297h2m21.982380015s) [noise=true , address=77.90.31.16:4738 , piece_id=OEA4W3UPM3ECAZQHSITOHEHOHQNLUKPR35A3EPZJP2QF2TJMBGJA, seg_pos={0 0}, upload=332695]: Uploading piece... Mar 18 20:02:24.385941521 (297h2m21.982712555s) [noise=true , address=145.79.227.4:4188 , piece_id=2ZNGXKSLIOL7CTK6BSHIYWSYUDMXI6CP5QAOI5QNWRZLDPCIL2AA, seg_pos={0 0}, upload=332695]: Uploading piece... Mar 18 20:02:24.386256609 (297h2m21.983027642s) [noise=true , address=145.79.227.27:4692 , piece_id=G5NROI32QZ7TNNIQPUFOUSHFYVTNBX7AAEQL52EH3EWFE6YFXH5Q, seg_pos={0 0}, upload=332695]: Uploading piece... Mar 18 20:02:24.386388747 (297h2m21.98315978s) [noise=true , address=77.90.31.13:4629 , piece_id=LO5A3KL4MKLHNBQAIPYLFY6XPKEVRNSLBWBD26EVZUDCK7667WEA, seg_pos={0 0}, upload=332695]: Uploading piece... Mar 18 20:02:24.386877177 (297h2m21.983648205s) [noise=true , address=79.127.219.38:4820 , piece_id=6M4DXDMJE5R7EXV5UOKLFDGX2W77NP7IOXEGFHNOBQTAYEU2LDCQ, seg_pos={0 0}, upload=332695]: Uploading piece... Mar 18 20:02:24.386926693 (297h2m21.983697728s) [noise=true , address=45.91.116.6:4974 , piece_id=FPLZI4645MDQXRMURV5WKJK72OEBMRHAW5INVH3KYVEZDYCBG3AA, seg_pos={0 0}, upload=332695]: Uploading piece... Mar 18 20:02:24.387336730 (297h2m21.984107762s) [noise=true , address=145.79.227.52:4449 , piece_id=UDPFME34QFR33WAXUNJPM7MGQMKNNJSCRPEORMHQOYBYWOHELMVQ, seg_pos={0 0}, upload=332695]: Uploading piece... Mar 18 20:02:24.387700331 (297h2m21.984471365s) [noise=true , address=45.91.116.7:4010 , piece_id=M45XHS6ZQI25PZBGKLFHE32QFDQFLXZCXTI2KHUFS4FESXL4KXWQ, seg_pos={0 0}, upload=332695]: Uploading piece... Mar 18 20:02:24.388464733 (297h2m21.985235766s) [noise=true , address=145.79.227.51:4412 , piece_id=SJTPBG4J5HSG4S2MQOQAWBV6I6P4UHULIBMB65PUGXG765SNNBIQ, seg_pos={0 0}, upload=332695]: Uploading piece... Mar 18 20:02:24.389136574 (297h2m21.985907615s) [noise=true , address=79.127.219.41:4925 , piece_id=ADKWF5GQQ5QFBI4YZ37Z6DHLBIOLKQHNU2JDTF6NVKYRRJUIUC6A, seg_pos={0 0}, upload=332695]: Uploading piece... Mar 18 20:02:24.391737543 (297h2m21.988508575s) [noise=true , address=45.91.116.13:4153 , piece_id=OHTYPSLXBVOSXA7ZW2CPJARQPEEUXYIHRUB4E6RYBMJGOMWCRPQA, seg_pos={0 0}, upload=332695]: Uploading piece... Mar 18 20:02:24.392647606 (297h2m21.989418638s) [noise=true , address=77.90.31.15:4697 , piece_id=UKKDQ4L7IQF5EWCJIOEOZNOURZPQZ6H4QFNCMJSW4OFQT5HCJNEQ, seg_pos={0 0}, upload=332695]: Uploading piece... Mar 18 20:02:24.453558219 (297h2m22.050329232s) [upload=332695]: Next returned nil segment Mar 18 20:02:24.453572009 (297h2m22.050343033s) [upload=332695]: Waiting for error group managing segments... Mar 18 20:02:24.457921598 (297h2m22.054692622s) [noise=true , address=45.91.116.13:4153 , piece_id=OHTYPSLXBVOSXA7ZW2CPJARQPEEUXYIHRUB4E6RYBMJGOMWCRPQA, seg_pos={0 0}, upload=332695]: Done uploading piece. err: <nil> Mar 18 20:02:24.458289093 (297h2m22.05506012s) [noise=true , address=45.91.116.7:4010 , piece_id=M45XHS6ZQI25PZBGKLFHE32QFDQFLXZCXTI2KHUFS4FESXL4KXWQ, seg_pos={0 0}, upload=332695]: Done uploading piece. err: <nil> Mar 18 20:02:24.458574349 (297h2m22.055345362s) [noise=true , address=45.91.116.5:4298 , piece_id=U4LOFGYW46FX7BTJQCO7KBSDUXENWD7ZZFY6GSDJJF2JZVCHX3IQ, seg_pos={0 0}, upload=332695]: Done uploading piece. err: <nil> Mar 18 20:02:24.459608079 (297h2m22.056379102s) [noise=true , address=77.90.31.14:4666 , piece_id=KQJQAQAV7CCMYAZ2VIIS2XVNTGVE4CEN26DECIBZN76D5XH3UTMQ, seg_pos={0 0}, upload=332695]: Done uploading piece. err: <nil> Mar 18 20:02:24.460060467 (297h2m22.056831491s) [noise=true , address=79.127.219.41:4925 , piece_id=ADKWF5GQQ5QFBI4YZ37Z6DHLBIOLKQHNU2JDTF6NVKYRRJUIUC6A, seg_pos={0 0}, upload=332695]: Done uploading piece. err: <nil> Mar 18 20:02:24.460827541 (297h2m22.057598569s) [noise=true , address=77.90.31.16:4738 , piece_id=OEA4W3UPM3ECAZQHSITOHEHOHQNLUKPR35A3EPZJP2QF2TJMBGJA, seg_pos={0 0}, upload=332695]: Done uploading piece. err: <nil> Mar 18 20:02:24.472731792 (297h2m22.069502814s) [noise=true , address=45.91.116.6:4974 , piece_id=FPLZI4645MDQXRMURV5WKJK72OEBMRHAW5INVH3KYVEZDYCBG3AA, seg_pos={0 0}, upload=332695]: Done uploading piece. err: <nil> Mar 18 20:02:24.473622230 (297h2m22.070393254s) [noise=true , address=77.90.31.13:4629 , piece_id=LO5A3KL4MKLHNBQAIPYLFY6XPKEVRNSLBWBD26EVZUDCK7667WEA, seg_pos={0 0}, upload=332695]: Done uploading piece. err: <nil> Mar 18 20:02:24.477080683 (297h2m22.073851706s) [noise=true , address=79.127.219.38:4820 , piece_id=6M4DXDMJE5R7EXV5UOKLFDGX2W77NP7IOXEGFHNOBQTAYEU2LDCQ, seg_pos={0 0}, upload=332695]: Done uploading piece. err: <nil> Mar 18 20:02:24.488478138 (297h2m22.085249169s) [noise=true , address=77.90.31.15:4697 , piece_id=UKKDQ4L7IQF5EWCJIOEOZNOURZPQZ6H4QFNCMJSW4OFQT5HCJNEQ, seg_pos={0 0}, upload=332695]: Done uploading piece. err: <nil> Mar 18 20:02:51.519528481 (297h2m49.11629951s) [noise=true , address=145.79.227.52:4449 , piece_id=UDPFME34QFR33WAXUNJPM7MGQMKNNJSCRPEORMHQOYBYWOHELMVQ, seg_pos={0 0}, upload=332695]: Done uploading piece. err: ecclient: upload failed (node:12f14pHDEEamEnHmeZCe38rjaE6EYWYkGMPrZMsbSDmApN5rckq, address:145.79.227.52:4449): protocol: rpc: dial tcp 145.79.227.52:4449: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:02:51.519712853 (297h2m49.116483887s) [noise=true , address=145.79.227.51:4412 , piece_id=SJTPBG4J5HSG4S2MQOQAWBV6I6P4UHULIBMB65PUGXG765SNNBIQ, seg_pos={0 0}, upload=332695]: Done uploading piece. err: ecclient: upload failed (node:12vBvEfCYZxsv17cVsQTbCQ4YjZfiD4RXjWyz7Ka9ZZrtvzC7SS, address:145.79.227.51:4412): protocol: rpc: dial tcp 145.79.227.51:4412: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:02:51.519812355 (297h2m49.116583388s) [noise=true , address=145.79.227.37:4570 , piece_id=XNYV7FYPAAZ47BQMTCEZZ63U32IRPHEHYXL52Z4COICZFLX4G7NQ, seg_pos={0 0}, upload=332695]: Done uploading piece. err: ecclient: upload failed (node:1q5evYAfULsn19ddsAvSzJwxiDd8wsWbjP6gpcRkuU1b1q57eJ, address:145.79.227.37:4570): protocol: rpc: dial tcp 145.79.227.37:4570: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:02:51.520056957 (297h2m49.116827986s) [noise=true , address=145.79.227.27:4692 , piece_id=G5NROI32QZ7TNNIQPUFOUSHFYVTNBX7AAEQL52EH3EWFE6YFXH5Q, seg_pos={0 0}, upload=332695]: Done uploading piece. err: ecclient: upload failed (node:1n3SDzXGgXbvYBGtLdND9oy93AHNFokRnF6Yv3EvoMC9UgN2H9, address:145.79.227.27:4692): protocol: rpc: dial tcp 145.79.227.27:4692: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:02:51.520328436 (297h2m49.117099466s) [noise=true , address=145.79.227.10:4407 , piece_id=HCIQ7MQE5INROVA4JWXXBI6NBTH3GZVOUNW4XOWXBC2OXIPGSGKQ, seg_pos={0 0}, upload=332695]: Done uploading piece. err: ecclient: upload failed (node:1kJKjEfzwg1wuBUP2Bw4oaSoy6mQZdvDED6PpkbrqQwbkh3bUW, address:145.79.227.10:4407): protocol: rpc: dial tcp 145.79.227.10:4407: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:02:51.520608763 (297h2m49.117379796s) [noise=true , address=145.79.227.4:4188 , piece_id=2ZNGXKSLIOL7CTK6BSHIYWSYUDMXI6CP5QAOI5QNWRZLDPCIL2AA, seg_pos={0 0}, upload=332695]: Done uploading piece. err: ecclient: upload failed (node:12pLFHQ2Xj6n53w9eLQtMB4A29k3Xh6w716z5s2CGxwPnBoZcgZ, address:145.79.227.4:4188): protocol: rpc: dial tcp 145.79.227.4:4188: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:02:51.520646132 (297h2m49.117417141s) [seg_pos={0 0}, upload=332695]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Mar 18 20:02:51.542788720 (297h2m49.139559733s) [upload=332695]: Done uploading segments... Mar 18 20:02:51.542851589 (297h2m49.139622614s) [upload=332695]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration: 30.025621161s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 8 out of 60 is failed (threshold=3) [..........................XXXXXX.XX.........................] duration: 14.107306223s trace: https://jaeger.dev.storj.io/trace/4b4d10a8051ad01b FAIL (2026-03-18 20:02:29 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 14.107332799s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 35 out of 60 is failed (threshold=3) [.........................XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 30.026449939s trace: https://jaeger.dev.storj.io/trace/170dde13c0caf9a9 FAIL (2026-03-18 20:02:21 UTC) operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 7) ======LOGS====== load Mar 18 20:01:51.074918509 (297h1m48.671689533s) [upload=332684]: Uploading segments... Mar 18 20:01:51.075222443 (297h1m48.671993469s) [upload=332684]: Got next segment. Inline: false Mar 18 20:01:51.075304417 (297h1m48.67207544s) [upload=332684]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 20:01:51.075310948 (297h1m48.67208197s) [upload=332684]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 18 20:01:53.933952170 (297h1m51.530723193s) [seg_pos={0 0}, upload=332684]: Begin upload segment... Mar 18 20:01:53.934066695 (297h1m51.530837706s) [seg_pos={0 0}, upload=332684]: Done begin upload segment. Mar 18 20:01:53.934132132 (297h1m51.530903162s) [noise=true , address=45.91.116.12:4226 , piece_id=D7RKYGMQ26EOQ6HHJE7MUT4BNDRNJ26BPV6MFWRFZDR433IFLVZQ, seg_pos={0 0}, upload=332684]: Uploading piece... Mar 18 20:01:53.934244207 (297h1m51.531015241s) [noise=true , address=145.79.227.41:4088 , piece_id=RJHH66QUR7TCA3X54CL6U7DPCD6Y4DBSQZ3CIJKDUFHHVR6YGEGA, seg_pos={0 0}, upload=332684]: Uploading piece... Mar 18 20:01:53.934894525 (297h1m51.531665558s) [noise=true , address=145.79.227.32:4838 , piece_id=NI7ROSVHIFBN33KR6MZDCEUPIBEAW6HYSGXL5SAIAZDAPYIYMPYQ, seg_pos={0 0}, upload=332684]: Uploading piece... Mar 18 20:01:53.935123881 (297h1m51.531894915s) [noise=true , address=77.90.31.11:4558 , piece_id=3RR5RQYF3XYOSFFPZDWJWZA4I46TRAU4662EYPOID23RQF4GNULA, seg_pos={0 0}, upload=332684]: Uploading piece... Mar 18 20:01:53.935887727 (297h1m51.532658754s) [noise=true , address=145.79.227.8:4334 , piece_id=SCOHQEWMAGZXRCHC52U46N4Q47Q54KCUSX7V3FBX3SFDOPGOF4RA, seg_pos={0 0}, upload=332684]: Uploading piece... Mar 18 20:01:53.936399405 (297h1m51.533170431s) [noise=true , address=45.91.116.9:4262 , piece_id=I6MTHUHDDKWBN3NFFUCMACUFTJ4G73AANZCE2GADSJGCI7OTUEJQ, seg_pos={0 0}, upload=332684]: Uploading piece... Mar 18 20:01:53.937971750 (297h1m51.534742779s) [noise=true , address=145.79.227.30:4767 , piece_id=WB3DBNKTC4UIJGQEVUPARWHBXGS3EGURVND745TWKH5A5QMKSS6Q, seg_pos={0 0}, upload=332684]: Uploading piece... Mar 18 20:01:53.938276525 (297h1m51.535047549s) [noise=true , address=45.91.116.4:4952 , piece_id=KPDHHEJ6R42TWLEQXDEV7ZC7G4R2UIUATVPBBGFJGMZGPX32UBWA, seg_pos={0 0}, upload=332684]: Uploading piece... Mar 18 20:01:53.938466427 (297h1m51.535237451s) [noise=true , address=45.91.116.8:4048 , piece_id=BCQNSS2VAOAR23NQ26UVKGYISNHO3FEBNPMUY3MX7GNA54JF556Q, seg_pos={0 0}, upload=332684]: Uploading piece... Mar 18 20:01:53.938781448 (297h1m51.535552477s) [noise=true , address=77.90.31.13:4627 , piece_id=J7SZFF2QGK4VVME24KO3SNQIBPTYHKFGVTTROLT7IYAK6MD6KFHA, seg_pos={0 0}, upload=332684]: Uploading piece... Mar 18 20:01:53.938952229 (297h1m51.535723253s) [noise=true , address=145.79.227.15:4877 , piece_id=PUE4GXHV536SDG73GTRTQLEBRN3KCSBYO2BUTPH27AOHOZ54HWZA, seg_pos={0 0}, upload=332684]: Uploading piece... Mar 18 20:01:53.939252443 (297h1m51.536023471s) [noise=true , address=145.79.227.18:4010 , piece_id=ASGI6MJ7PYVHMRYKBNECSIYEXLEWHYO5CHA5TIMWWBVANM2QQGKQ, seg_pos={0 0}, upload=332684]: Uploading piece... Mar 18 20:01:53.939601963 (297h1m51.536372986s) [noise=true , address=145.79.227.14:4840 , piece_id=7TUQZYOFPGOS4M7GJBZ2NO6WZJNOU4WENVPLDYPDKVJOWNB7KLFQ, seg_pos={0 0}, upload=332684]: Uploading piece... Mar 18 20:01:53.939784097 (297h1m51.536555124s) [noise=true , address=145.79.227.42:4125 , piece_id=CLBBXYAO47WWUCKODZANGQ3AGSFQJ46BB2DIOIL7RHBCAFTP6QPQ, seg_pos={0 0}, upload=332684]: Uploading piece... Mar 18 20:01:53.940370326 (297h1m51.537141349s) [noise=true , address=77.90.31.14:4662 , piece_id=BIZSJ77Y7S5E2FFN2Q44UOZC4PAR7GPJ76S7I5RREPNRMBONKLMQ, seg_pos={0 0}, upload=332684]: Uploading piece... Mar 18 20:01:53.940853610 (297h1m51.537624642s) [noise=true , address=145.79.227.33:4869 , piece_id=IG7AZCFWN3S6XSONQ6WZYJXWWUL2COOXUEYGXJQR7QXD3LO5FENQ, seg_pos={0 0}, upload=332684]: Uploading piece... Mar 18 20:01:53.999102398 (297h1m51.595873414s) [upload=332684]: Next returned nil segment Mar 18 20:01:53.999116524 (297h1m51.59588755s) [upload=332684]: Waiting for error group managing segments... Mar 18 20:01:54.006456679 (297h1m51.603227705s) [noise=true , address=77.90.31.14:4662 , piece_id=BIZSJ77Y7S5E2FFN2Q44UOZC4PAR7GPJ76S7I5RREPNRMBONKLMQ, seg_pos={0 0}, upload=332684]: Done uploading piece. err: <nil> Mar 18 20:01:54.007706339 (297h1m51.604477363s) [noise=true , address=45.91.116.12:4226 , piece_id=D7RKYGMQ26EOQ6HHJE7MUT4BNDRNJ26BPV6MFWRFZDR433IFLVZQ, seg_pos={0 0}, upload=332684]: Done uploading piece. err: <nil> Mar 18 20:01:54.008293909 (297h1m51.605064921s) [noise=true , address=45.91.116.4:4952 , piece_id=KPDHHEJ6R42TWLEQXDEV7ZC7G4R2UIUATVPBBGFJGMZGPX32UBWA, seg_pos={0 0}, upload=332684]: Done uploading piece. err: <nil> Mar 18 20:01:54.008722129 (297h1m51.605493157s) [noise=true , address=45.91.116.9:4262 , piece_id=I6MTHUHDDKWBN3NFFUCMACUFTJ4G73AANZCE2GADSJGCI7OTUEJQ, seg_pos={0 0}, upload=332684]: Done uploading piece. err: <nil> Mar 18 20:01:54.009341223 (297h1m51.606112245s) [noise=true , address=77.90.31.11:4558 , piece_id=3RR5RQYF3XYOSFFPZDWJWZA4I46TRAU4662EYPOID23RQF4GNULA, seg_pos={0 0}, upload=332684]: Done uploading piece. err: <nil> Mar 18 20:01:54.023495053 (297h1m51.620266089s) [noise=true , address=77.90.31.13:4627 , piece_id=J7SZFF2QGK4VVME24KO3SNQIBPTYHKFGVTTROLT7IYAK6MD6KFHA, seg_pos={0 0}, upload=332684]: Done uploading piece. err: <nil> Mar 18 20:01:54.024050289 (297h1m51.620821302s) [noise=true , address=45.91.116.8:4048 , piece_id=BCQNSS2VAOAR23NQ26UVKGYISNHO3FEBNPMUY3MX7GNA54JF556Q, seg_pos={0 0}, upload=332684]: Done uploading piece. err: <nil> Mar 18 20:02:21.040229821 (297h2m18.637000847s) [noise=true , address=145.79.227.8:4334 , piece_id=SCOHQEWMAGZXRCHC52U46N4Q47Q54KCUSX7V3FBX3SFDOPGOF4RA, seg_pos={0 0}, upload=332684]: Done uploading piece. err: ecclient: upload failed (node:12KYMZAFsJbhWC4CdKRod2Z6qsC6ANcpqmpWSWx7UYajp6JvKH1, address:145.79.227.8:4334): protocol: rpc: dial tcp 145.79.227.8:4334: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:02:21.040441707 (297h2m18.63721274s) [noise=true , address=145.79.227.14:4840 , piece_id=7TUQZYOFPGOS4M7GJBZ2NO6WZJNOU4WENVPLDYPDKVJOWNB7KLFQ, seg_pos={0 0}, upload=332684]: Done uploading piece. err: ecclient: upload failed (node:12crcXpNe9NgeXTsUpooLSBgrSUFbRxbSbanLa39w2Hr5mGFww, address:145.79.227.14:4840): protocol: rpc: dial tcp 145.79.227.14:4840: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:02:21.040582017 (297h2m18.637353051s) [noise=true , address=145.79.227.30:4767 , piece_id=WB3DBNKTC4UIJGQEVUPARWHBXGS3EGURVND745TWKH5A5QMKSS6Q, seg_pos={0 0}, upload=332684]: Done uploading piece. err: ecclient: upload failed (node:12sfaYCUhNCAyQAPu1jac5dRWR35Qqc4ZcKg75mp7nVQak295ug, address:145.79.227.30:4767): protocol: rpc: dial tcp 145.79.227.30:4767: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:02:21.040808886 (297h2m18.637579923s) [noise=true , address=145.79.227.32:4838 , piece_id=NI7ROSVHIFBN33KR6MZDCEUPIBEAW6HYSGXL5SAIAZDAPYIYMPYQ, seg_pos={0 0}, upload=332684]: Done uploading piece. err: ecclient: upload failed (node:1eorQLNmgEaXv8myr6t3Q8rysGwsvtdZzu5b7sndhoANb6Waj6, address:145.79.227.32:4838): protocol: context deadline exceeded; context deadline exceeded Mar 18 20:02:21.040879973 (297h2m18.63765103s) [noise=true , address=145.79.227.18:4010 , piece_id=ASGI6MJ7PYVHMRYKBNECSIYEXLEWHYO5CHA5TIMWWBVANM2QQGKQ, seg_pos={0 0}, upload=332684]: Done uploading piece. err: ecclient: upload failed (node:19XkukzF8ZCLdHLTYQTYpX5iF4tvT1iM6Zxxs1ZNQ4r5X4ousp, address:145.79.227.18:4010): protocol: rpc: dial tcp 145.79.227.18:4010: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:02:21.041215410 (297h2m18.637986444s) [noise=true , address=145.79.227.33:4869 , piece_id=IG7AZCFWN3S6XSONQ6WZYJXWWUL2COOXUEYGXJQR7QXD3LO5FENQ, seg_pos={0 0}, upload=332684]: Done uploading piece. err: ecclient: upload failed (node:18CQxjq65r6xVTmNRurSQPzKxj4BoR6cJ8Uq29sTrdh6uXm5sq, address:145.79.227.33:4869): protocol: rpc: dial tcp 145.79.227.33:4869: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:02:21.041518692 (297h2m18.638289728s) [noise=true , address=145.79.227.41:4088 , piece_id=RJHH66QUR7TCA3X54CL6U7DPCD6Y4DBSQZ3CIJKDUFHHVR6YGEGA, seg_pos={0 0}, upload=332684]: Done uploading piece. err: ecclient: upload failed (node:1284h6cPNVXJYMkSekGQVw8b7p89tkhpmgYLxTTYUDu2J7Wadoj, address:145.79.227.41:4088): protocol: rpc: dial tcp 145.79.227.41:4088: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:02:21.041593287 (297h2m18.638364318s) [noise=true , address=145.79.227.15:4877 , piece_id=PUE4GXHV536SDG73GTRTQLEBRN3KCSBYO2BUTPH27AOHOZ54HWZA, seg_pos={0 0}, upload=332684]: Done uploading piece. err: ecclient: upload failed (node:12akiEFhbb3wqcG7uJVZHwj4rHEQpX2AXKXrovAsaUPCdhj2PXz, address:145.79.227.15:4877): protocol: rpc: dial tcp 145.79.227.15:4877: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:02:21.041748925 (297h2m18.638519959s) [noise=true , address=145.79.227.42:4125 , piece_id=CLBBXYAO47WWUCKODZANGQ3AGSFQJ46BB2DIOIL7RHBCAFTP6QPQ, seg_pos={0 0}, upload=332684]: Done uploading piece. err: ecclient: upload failed (node:1MrdHwiDVe8m6wcaQJrYetza9qckLruJXiU9UYbMxGCSWWMV1S, address:145.79.227.42:4125): protocol: rpc: dial tcp 145.79.227.42:4125: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:02:21.041787281 (297h2m18.638558301s) [seg_pos={0 0}, upload=332684]: Done waiting for segment. successful: 7 optimal: 13 errs: <nil> Mar 18 20:02:21.063983330 (297h2m18.660754363s) [upload=332684]: Done uploading segments... Mar 18 20:02:21.064049163 (297h2m18.660820197s) [upload=332684]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 7) Duration: 30.026476083s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 45 out of 60 is failed (threshold=6) [...........XXXXXXXXX.XXX...XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 5.001484078s trace: https://jaeger.dev.storj.io/trace/18f5c0c6bce1ce48 FAIL (2026-03-18 20:02:18 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001608677s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 6 out of 60 is failed (threshold=3) [.............................X..........XX..X....X........X.] duration: 15.615997591s trace: https://jaeger.dev.storj.io/trace/589419efc52e5e2b FAIL (2026-03-18 20:02:15 UTC) operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== Mar 18 20:00:59.343084966 (297h0m56.939855989s) [upload=332666]: Starting upload Mar 18 20:00:59.343175518 (297h0m56.939946548s) [upload=332666]: Done starting upload Mar 18 20:00:59.343565585 (297h0m56.940336612s) [upload=332666]: Uploading segments... Mar 18 20:00:59.344256647 (297h0m56.941027676s) [upload=332666]: Got next segment. Inline: false Mar 18 20:00:59.344352215 (297h0m56.941123241s) [upload=332666]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 20:00:59.344358706 (297h0m56.941129732s) [upload=332666]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 18 20:01:02.340259299 (297h0m59.937030329s) [seg_pos={0 0}, upload=332666]: Begin upload segment... Mar 18 20:01:02.340368743 (297h0m59.937139767s) [seg_pos={0 0}, upload=332666]: Done begin upload segment. Mar 18 20:01:02.340479563 (297h0m59.93725059s) [noise=true , address=45.91.116.8:4061 , piece_id=6XFVKT4AAO7QN457OQMRKPI6R24TJTAN47UXLEVOEFQVOS6M6C3Q, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.340523837 (297h0m59.93729486s) [noise=true , address=79.127.205.233:4166 , piece_id=SFGK5YCWHFRVXF6QNHYP2CYJCZLSTQ76OYRVJPWWXQTF2KFSFJKA, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.341468438 (297h0m59.938239447s) [noise=true , address=45.91.116.4:4961 , piece_id=ZMUGQECUVPVHDNKN75XPBET5M2HFXXK4WMTP4V5KEK63JVZCX5EA, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.341520025 (297h0m59.938291058s) [noise=true , address=167.17.72.71:4815 , piece_id=CXGTYNNBILN3TT7I6Z7UI75F2ZCS4QUTLS6KC4FJ5IHS3XOYWDJQ, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.342232992 (297h0m59.939004023s) [noise=true , address=145.79.227.36:4524 , piece_id=3TAV3P55ITTHBAZEUUUTOAHJRGQXJXXVGZJP6AVR2ANVNOR3S54Q, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.342917368 (297h0m59.939688398s) [noise=true , address=193.149.164.71:4297 , piece_id=4IJPXLPCX3LKFOBOSRHK2DTNRKH67K7FXJUKQ3JPRWHS47ZFEIDA, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.343244957 (297h0m59.940015982s) [noise=true , address=145.79.227.33:4892 , piece_id=DQ6ALC3VOAGFIJI3IKR6I7QFYM22C66QWAAXMH6NXI2FIQEZNQYA, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.344270079 (297h0m59.941041117s) [noise=true , address=145.79.227.21:4497 , piece_id=SP6EKJB7W4QXF7OIBVKEFWCPOZFL5IRIEDVAGS2M4QJSO34CUVRQ, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.345025057 (297h0m59.941796084s) [noise=true , address=77.90.31.17:4793 , piece_id=2U4IBUA2GGUGDBXQGIGQKUK4ZR3L7CDKC6MTGIKAUME4F7MFZRLA, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.345115802 (297h0m59.941886836s) [noise=true , address=167.17.72.69:4731 , piece_id=VOEVIKKFJMC2Y3TPQIDMYUP37NLFU7KPRH4HRRSMQLWUKTSLOH2Q, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.345690600 (297h0m59.942461634s) [noise=true , address=77.90.31.13:4648 , piece_id=BPMB5CFOYHFT3BS4ORR6JF6BNFC2AOAJZOSPMG67NAMLGGP5XVEQ, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.346311592 (297h0m59.943082623s) [noise=true , address=79.127.205.234:4202 , piece_id=YY4LYWX26O6RIOBONESNKTUZEFZCRL4UUX3NEBQ53AJV4IEDG5MQ, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.346791142 (297h0m59.943562166s) [noise=true , address=45.91.116.10:4110 , piece_id=UYLMDCWJEJOHGICA37KNH7G5AQXNIPRH5NRYLBUUMGDRVSLTEYHQ, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.347094999 (297h0m59.94386607s) [noise=true , address=145.79.227.17:4987 , piece_id=PT3TWVB65JZFEWVH4IUZEU2N53FQ57QBRWVLWAUFP4UDLY3LWNLQ, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.347781281 (297h0m59.944552305s) [noise=true , address=79.127.219.46:4919 , piece_id=KK6HRIVBMYLIHRHGPQH3AWMNSVHTTIBESHDMJZA72T446RX6QKRQ, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.348263123 (297h0m59.945034137s) [noise=true , address=193.149.164.68:4033 , piece_id=JZPQKPYGO3W3RZZBITQ73X2XEPIBBSLGRMWIVOM4Z3GTV5UFKRTQ, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.348915505 (297h0m59.945686536s) [noise=true , address=145.79.227.7:4312 , piece_id=2FAL6E3RMBTBHZBXAJJVH2N5VGF5VYI3KNZXGNWCMWGBBPAUO2NQ, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.349366298 (297h0m59.946137337s) [noise=true , address=193.149.164.67:4007 , piece_id=HURQGH67I5B6V4YUV3PRZL3KSKN3XTSK3UMCHZRNT6L3W3PXXHPQ, seg_pos={0 0}, upload=332666]: Uploading piece... Mar 18 20:01:02.429834098 (297h1m0.026605126s) [upload=332666]: Next returned nil segment Mar 18 20:01:02.429853411 (297h1m0.026624442s) [upload=332666]: Waiting for error group managing segments... Mar 18 20:01:02.446867680 (297h1m0.043638704s) [noise=true , address=45.91.116.10:4110 , piece_id=UYLMDCWJEJOHGICA37KNH7G5AQXNIPRH5NRYLBUUMGDRVSLTEYHQ, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil> Mar 18 20:01:02.451665165 (297h1m0.048436196s) [noise=true , address=45.91.116.4:4961 , piece_id=ZMUGQECUVPVHDNKN75XPBET5M2HFXXK4WMTP4V5KEK63JVZCX5EA, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil> Mar 18 20:01:02.452538886 (297h1m0.049309913s) [noise=true , address=77.90.31.17:4793 , piece_id=2U4IBUA2GGUGDBXQGIGQKUK4ZR3L7CDKC6MTGIKAUME4F7MFZRLA, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil> Mar 18 20:01:02.455597040 (297h1m0.052368067s) [noise=true , address=45.91.116.8:4061 , piece_id=6XFVKT4AAO7QN457OQMRKPI6R24TJTAN47UXLEVOEFQVOS6M6C3Q, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil> Mar 18 20:01:02.457848739 (297h1m0.054619766s) [noise=true , address=77.90.31.13:4648 , piece_id=BPMB5CFOYHFT3BS4ORR6JF6BNFC2AOAJZOSPMG67NAMLGGP5XVEQ, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil> Mar 18 20:01:02.461479709 (297h1m0.058250746s) [noise=true , address=79.127.219.46:4919 , piece_id=KK6HRIVBMYLIHRHGPQH3AWMNSVHTTIBESHDMJZA72T446RX6QKRQ, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil> Mar 18 20:01:02.676859466 (297h1m0.27363049s) [noise=true , address=167.17.72.69:4731 , piece_id=VOEVIKKFJMC2Y3TPQIDMYUP37NLFU7KPRH4HRRSMQLWUKTSLOH2Q, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil> Mar 18 20:01:02.909860743 (297h1m0.506631766s) [noise=true , address=79.127.205.234:4202 , piece_id=YY4LYWX26O6RIOBONESNKTUZEFZCRL4UUX3NEBQ53AJV4IEDG5MQ, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil> Mar 18 20:01:02.930427742 (297h1m0.527198754s) [noise=true , address=193.149.164.71:4297 , piece_id=4IJPXLPCX3LKFOBOSRHK2DTNRKH67K7FXJUKQ3JPRWHS47ZFEIDA, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil> Mar 18 20:01:02.944696889 (297h1m0.541467914s) [noise=true , address=193.149.164.67:4007 , piece_id=HURQGH67I5B6V4YUV3PRZL3KSKN3XTSK3UMCHZRNT6L3W3PXXHPQ, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil> Mar 18 20:01:02.951319183 (297h1m0.548090212s) [noise=true , address=193.149.164.68:4033 , piece_id=JZPQKPYGO3W3RZZBITQ73X2XEPIBBSLGRMWIVOM4Z3GTV5UFKRTQ, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil> Mar 18 20:01:02.975619436 (297h1m0.572390467s) [noise=true , address=79.127.205.233:4166 , piece_id=SFGK5YCWHFRVXF6QNHYP2CYJCZLSTQ76OYRVJPWWXQTF2KFSFJKA, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil> Mar 18 20:01:03.819918528 (297h1m1.416689564s) [noise=true , address=167.17.72.71:4815 , piece_id=CXGTYNNBILN3TT7I6Z7UI75F2ZCS4QUTLS6KC4FJ5IHS3XOYWDJQ, seg_pos={0 0}, upload=332666]: Done uploading piece. err: <nil> Mar 18 20:01:59.307873420 (297h1m56.904644452s) [noise=true , address=145.79.227.7:4312 , piece_id=2FAL6E3RMBTBHZBXAJJVH2N5VGF5VYI3KNZXGNWCMWGBBPAUO2NQ, seg_pos={0 0}, upload=332666]: Done uploading piece. err: ecclient: upload failed (node:12AVp3qn5uwLPqYaZEwzcgoGMJhNga1BUh6bK5k2bxjWgZLQiiC, address:145.79.227.7:4312): protocol: rpc: dial tcp 145.79.227.7:4312: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:01:59.308026092 (297h1m56.904797124s) [noise=true , address=145.79.227.17:4987 , piece_id=PT3TWVB65JZFEWVH4IUZEU2N53FQ57QBRWVLWAUFP4UDLY3LWNLQ, seg_pos={0 0}, upload=332666]: Done uploading piece. err: ecclient: upload failed (node:1XsPVNGYUhKnXkQRfVfjyBRzyvhShSdsrnzXtwe4Xtf5MaHMvH, address:145.79.227.17:4987): protocol: rpc: dial tcp 145.79.227.17:4987: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:01:59.308195864 (297h1m56.904966894s) [noise=true , address=145.79.227.36:4524 , piece_id=3TAV3P55ITTHBAZEUUUTOAHJRGQXJXXVGZJP6AVR2ANVNOR3S54Q, seg_pos={0 0}, upload=332666]: Done uploading piece. err: ecclient: upload failed (node:1xfbCzjSwFDRrR1gPggeN8tNCLbRRWDVzCJ6mqhZsbVSe8d4s4, address:145.79.227.36:4524): protocol: rpc: dial tcp 145.79.227.36:4524: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:01:59.308316037 (297h1m56.90508706s) [noise=true , address=145.79.227.33:4892 , piece_id=DQ6ALC3VOAGFIJI3IKR6I7QFYM22C66QWAAXMH6NXI2FIQEZNQYA, seg_pos={0 0}, upload=332666]: Done uploading piece. err: ecclient: upload failed (node:12RHwFxcUjPsn12uHEAH3fBDJKv1v1tM8mRJXoFrZrG5djdswF5, address:145.79.227.33:4892): protocol: rpc: dial tcp 145.79.227.33:4892: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:01:59.308404403 (297h1m56.905175433s) [noise=true , address=145.79.227.21:4497 , piece_id=SP6EKJB7W4QXF7OIBVKEFWCPOZFL5IRIEDVAGS2M4QJSO34CUVRQ, seg_pos={0 0}, upload=332666]: Done uploading piece. err: ecclient: upload failed (node:12ZBe4we7VeFdthqRxyy1V9uQrA4r8ouNX68xqsk5bD26dkf2Gd, address:145.79.227.21:4497): protocol: context deadline exceeded; context deadline exceeded Mar 18 20:01:59.308437457 (297h1m56.90520847s) [seg_pos={0 0}, upload=332666]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 18 20:01:59.338288467 (297h1m56.93505949s) [upload=332666]: Done uploading segments... Mar 18 20:01:59.338335497 (297h1m56.935106523s) [upload=332666]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 15.61603191s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 6 out of 60 is failed (threshold=3) [............................XXX..........X...X.........X....] duration: 13.815347464s trace: https://jaeger.dev.storj.io/trace/7c859732d77a7da7 FAIL (2026-03-18 20:01:59 UTC) operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== d=332621]: Uploading segments... Mar 18 19:58:34.964515697 (296h58m32.561286706s) [upload=332621]: Got next segment. Inline: false Mar 18 19:58:34.964555310 (296h58m32.561326334s) [upload=332621]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 19:58:34.964560227 (296h58m32.561331251s) [upload=332621]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 18 19:58:38.382666205 (296h58m35.979437219s) [seg_pos={0 0}, upload=332621]: Begin upload segment... Mar 18 19:58:38.382729299 (296h58m35.979500329s) [seg_pos={0 0}, upload=332621]: Done begin upload segment. Mar 18 19:58:38.382760799 (296h58m35.979531826s) [noise=true , address=145.79.227.46:4607 , piece_id=LXBA2IGR4U6DLXE4NXL2CN23X5HXP7KFN5UCJZH6IBRYAZCQ7N3Q, seg_pos={0 0}, upload=332621]: Uploading piece... Mar 18 19:58:38.382881088 (296h58m35.979652115s) [noise=true , address=145.79.227.22:4538 , piece_id=ALPGNVFU5TWXZ7D7I5XFQBZ2CAN74VYUBNYDFSPG3HIHO7Z6M3AA, seg_pos={0 0}, upload=332621]: Uploading piece... Mar 18 19:58:38.383461716 (296h58m35.980232746s) [noise=true , address=167.17.72.70:4770 , piece_id=5H5RNTJDXOLXRNVU4XG5FP4O5YN5PW2XQH4O2UIX5XBKNMO35WKQ, seg_pos={0 0}, upload=332621]: Uploading piece... Mar 18 19:58:38.384266580 (296h58m35.981037606s) [noise=true , address=45.91.116.10:4092 , piece_id=Y7E2ZWR5D6M7QKRY3I7SL3ZJN32EK36ZWPP3YRW5WCGQFZCNU64Q, seg_pos={0 0}, upload=332621]: Uploading piece... Mar 18 19:58:38.384321708 (296h58m35.981092731s) [noise=true , address=79.127.205.235:4273 , piece_id=CQEOZ6GWLY5MEVRQZHYH6AP6DXQY5WKIAHCHDBJMAI77STRNINYA, seg_pos={0 0}, upload=332621]: Uploading piece... Mar 18 19:58:38.385197892 (296h58m35.981968918s) [noise=true , address=193.149.164.66:4989 , piece_id=EDE7OF4G36VJ7GXFMF47ZI73D6I7OWZSC5AXESS6W74ZDWTBDXWA, seg_pos={0 0}, upload=332621]: Uploading piece... Mar 18 19:58:38.385232613 (296h58m35.982003642s) [noise=true , address=45.91.116.13:4171 , piece_id=74JRIA5AS6SMECS3FFCCVLH3AY5N4EPNWOEDZGITBPCQZEKDGEAQ, seg_pos={0 0}, upload=332621]: Uploading piece... Mar 18 19:58:38.385334543 (296h58m35.982105567s) [noise=true , address=193.149.164.70:4120 , piece_id=AO6H5CLNNY2CQ27JO57WEKF7OTFQA7IXAZMVG6A4EWJLZT7NIRSQ, seg_pos={0 0}, upload=332621]: Uploading piece... Mar 18 19:58:38.386120289 (296h58m35.982891315s) [noise=true , address=79.127.219.46:4908 , piece_id=L2EOXVK7S2ZSQNK47543XTNRWJP2PU57DH2XHZ2BO5OZUTDXDQFA, seg_pos={0 0}, upload=332621]: Uploading piece... Mar 18 19:58:38.386244430 (296h58m35.983015464s) [noise=true , address=79.127.205.237:4236 , piece_id=FKQ3NUK5AUFD35L25PJFPVDLZUVNRURK5RLSPG3GGC75QYTEWP4A, seg_pos={0 0}, upload=332621]: Uploading piece... Mar 18 19:58:38.387158310 (296h58m35.983929342s) [noise=true , address=64.49.13.140:4516 , piece_id=D2V5WAW6D46VSSNBFGNO5ETQNCOTDHQYWM62BSSVF53X2MWU3TYQ, seg_pos={0 0}, upload=332621]: Uploading piece... Mar 18 19:58:38.387233109 (296h58m35.984004132s) [noise=true , address=193.149.164.67:4020 , piece_id=NZQGIA6BVAK3Z6SQZA7WNAOAY25VQKBDTWCRV34KNSYTGVNDSWUA, seg_pos={0 0}, upload=332621]: Uploading piece... Mar 18 19:58:38.388077887 (296h58m35.984848921s) [noise=true , address=145.79.227.49:4343 , piece_id=NCJH6BOJHPW5OTSCNGKTZCZYC3F3LTRJKGVOKCJODWSMHUAATJQQ, seg_pos={0 0}, upload=332621]: Uploading piece... Mar 18 19:58:38.388338947 (296h58m35.985109978s) [noise=true , address=79.127.219.41:4940 , piece_id=C2ODTVQIR73WTRBM7EDKEC54DOBK67L2RV4UXPHHTRY6KL3EQIKQ, seg_pos={0 0}, upload=332621]: Uploading piece... Mar 18 19:58:38.388792743 (296h58m35.985563766s) [noise=true , address=145.79.227.39:4043 , piece_id=D757XSQF27CMMI5UPBRB7LLGISWOYLW7OS3UICYF3U6AI62X73PQ, seg_pos={0 0}, upload=332621]: Uploading piece... Mar 18 19:58:38.389305068 (296h58m35.986076097s) [noise=true , address=45.91.116.9:4280 , piece_id=QECGXDF7C5YUJNURQCU76PMJM5MZ3DCEGPGAJHCZWI3H3OVGERGQ, seg_pos={0 0}, upload=332621]: Uploading piece... Mar 18 19:58:38.389771675 (296h58m35.986542707s) [noise=true , address=145.79.227.36:4528 , piece_id=FJ5LKOZJYVNWLZ5I6OUCZMPFDOBVZXQRVPHREHPQX6J3YWAI2KFA, seg_pos={0 0}, upload=332621]: Uploading piece... Mar 18 19:58:38.390707637 (296h58m35.98747867s) [noise=true , address=77.90.31.18:4827 , piece_id=SGKGAEAHEHU2H4YRLYQLQMTS45U6ZNSAYJ6CYJUGGKEDRMQT6BHQ, seg_pos={0 0}, upload=332621]: Uploading piece... Mar 18 19:58:38.466570556 (296h58m36.063341572s) [upload=332621]: Next returned nil segment Mar 18 19:58:38.466587006 (296h58m36.063358039s) [upload=332621]: Waiting for error group managing segments... Mar 18 19:58:38.471637588 (296h58m36.068408605s) [noise=true , address=45.91.116.9:4280 , piece_id=QECGXDF7C5YUJNURQCU76PMJM5MZ3DCEGPGAJHCZWI3H3OVGERGQ, seg_pos={0 0}, upload=332621]: Done uploading piece. err: <nil> Mar 18 19:58:38.474621081 (296h58m36.071392109s) [noise=true , address=79.127.219.41:4940 , piece_id=C2ODTVQIR73WTRBM7EDKEC54DOBK67L2RV4UXPHHTRY6KL3EQIKQ, seg_pos={0 0}, upload=332621]: Done uploading piece. err: <nil> Mar 18 19:58:38.479487991 (296h58m36.076259007s) [noise=true , address=77.90.31.18:4827 , piece_id=SGKGAEAHEHU2H4YRLYQLQMTS45U6ZNSAYJ6CYJUGGKEDRMQT6BHQ, seg_pos={0 0}, upload=332621]: Done uploading piece. err: <nil> Mar 18 19:58:38.479685133 (296h58m36.076456159s) [noise=true , address=45.91.116.10:4092 , piece_id=Y7E2ZWR5D6M7QKRY3I7SL3ZJN32EK36ZWPP3YRW5WCGQFZCNU64Q, seg_pos={0 0}, upload=332621]: Done uploading piece. err: <nil> Mar 18 19:58:38.485438319 (296h58m36.082209345s) [noise=true , address=45.91.116.13:4171 , piece_id=74JRIA5AS6SMECS3FFCCVLH3AY5N4EPNWOEDZGITBPCQZEKDGEAQ, seg_pos={0 0}, upload=332621]: Done uploading piece. err: <nil> Mar 18 19:58:38.620281221 (296h58m36.217052236s) [noise=true , address=79.127.219.46:4908 , piece_id=L2EOXVK7S2ZSQNK47543XTNRWJP2PU57DH2XHZ2BO5OZUTDXDQFA, seg_pos={0 0}, upload=332621]: Done uploading piece. err: <nil> Mar 18 19:58:38.772264604 (296h58m36.369035618s) [noise=true , address=167.17.72.70:4770 , piece_id=5H5RNTJDXOLXRNVU4XG5FP4O5YN5PW2XQH4O2UIX5XBKNMO35WKQ, seg_pos={0 0}, upload=332621]: Done uploading piece. err: <nil> Mar 18 19:58:38.967982420 (296h58m36.564753435s) [noise=true , address=79.127.205.235:4273 , piece_id=CQEOZ6GWLY5MEVRQZHYH6AP6DXQY5WKIAHCHDBJMAI77STRNINYA, seg_pos={0 0}, upload=332621]: Done uploading piece. err: <nil> Mar 18 19:58:38.977759060 (296h58m36.57453009s) [noise=true , address=79.127.205.237:4236 , piece_id=FKQ3NUK5AUFD35L25PJFPVDLZUVNRURK5RLSPG3GGC75QYTEWP4A, seg_pos={0 0}, upload=332621]: Done uploading piece. err: <nil> Mar 18 19:58:39.005320695 (296h58m36.602091708s) [noise=true , address=193.149.164.70:4120 , piece_id=AO6H5CLNNY2CQ27JO57WEKF7OTFQA7IXAZMVG6A4EWJLZT7NIRSQ, seg_pos={0 0}, upload=332621]: Done uploading piece. err: <nil> Mar 18 19:58:39.025848813 (296h58m36.622619842s) [noise=true , address=193.149.164.67:4020 , piece_id=NZQGIA6BVAK3Z6SQZA7WNAOAY25VQKBDTWCRV34KNSYTGVNDSWUA, seg_pos={0 0}, upload=332621]: Done uploading piece. err: <nil> Mar 18 19:58:39.065107032 (296h58m36.661878045s) [noise=true , address=64.49.13.140:4516 , piece_id=D2V5WAW6D46VSSNBFGNO5ETQNCOTDHQYWM62BSSVF53X2MWU3TYQ, seg_pos={0 0}, upload=332621]: Done uploading piece. err: <nil> Mar 18 19:58:39.201750353 (296h58m36.798521377s) [noise=true , address=193.149.164.66:4989 , piece_id=EDE7OF4G36VJ7GXFMF47ZI73D6I7OWZSC5AXESS6W74ZDWTBDXWA, seg_pos={0 0}, upload=332621]: Done uploading piece. err: <nil> Mar 18 19:59:34.932832250 (296h59m32.529603284s) [noise=true , address=145.79.227.46:4607 , piece_id=LXBA2IGR4U6DLXE4NXL2CN23X5HXP7KFN5UCJZH6IBRYAZCQ7N3Q, seg_pos={0 0}, upload=332621]: Done uploading piece. err: ecclient: upload failed (node:1zDoLfA5bQWM7Uf479gfFVjG3yYvxG5EZS5yZSFGTvoFoPYEAo, address:145.79.227.46:4607): protocol: rpc: dial tcp 145.79.227.46:4607: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 19:59:34.932941758 (296h59m32.529712814s) [noise=true , address=145.79.227.22:4538 , piece_id=ALPGNVFU5TWXZ7D7I5XFQBZ2CAN74VYUBNYDFSPG3HIHO7Z6M3AA, seg_pos={0 0}, upload=332621]: Done uploading piece. err: ecclient: upload failed (node:1kNFzPpkDtwy4VFyzeFQReorJPEVsmdDAd8ULT8ZKoGde57P3v, address:145.79.227.22:4538): protocol: rpc: dial tcp 145.79.227.22:4538: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 19:59:34.933267818 (296h59m32.53003887s) [noise=true , address=145.79.227.49:4343 , piece_id=NCJH6BOJHPW5OTSCNGKTZCZYC3F3LTRJKGVOKCJODWSMHUAATJQQ, seg_pos={0 0}, upload=332621]: Done uploading piece. err: ecclient: upload failed (node:16EsEMRW53sjKVag2WpSDGLhELyE2bSXyW4aAurCvDFKRNkXuf, address:145.79.227.49:4343): protocol: rpc: dial tcp 145.79.227.49:4343: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 19:59:34.933359674 (296h59m32.530130694s) [noise=true , address=145.79.227.36:4528 , piece_id=FJ5LKOZJYVNWLZ5I6OUCZMPFDOBVZXQRVPHREHPQX6J3YWAI2KFA, seg_pos={0 0}, upload=332621]: Done uploading piece. err: ecclient: upload failed (node:12ArcCrcHdpNgBMfTcPdtTXb65RgnTh8N32ihHAx2RrddxnRYgT, address:145.79.227.36:4528): protocol: rpc: dial tcp 145.79.227.36:4528: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 19:59:34.933622410 (296h59m32.530393477s) [noise=true , address=145.79.227.39:4043 , piece_id=D757XSQF27CMMI5UPBRB7LLGISWOYLW7OS3UICYF3U6AI62X73PQ, seg_pos={0 0}, upload=332621]: Done uploading piece. err: ecclient: upload failed (node:12f97TAyWcE4YrxDQCeycC8tnuetQiagu2MV4U5o8ARLq8jGKTX, address:145.79.227.39:4043): protocol: rpc: dial tcp 145.79.227.39:4043: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 19:59:34.933677971 (296h59m32.530448984s) [seg_pos={0 0}, upload=332621]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 18 19:59:34.955309631 (296h59m32.552080664s) [upload=332621]: Done uploading segments... Mar 18 19:59:34.955416416 (296h59m32.552187448s) [upload=332621]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 13.815384785s core uplink us-select-1 us1-placement-20

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)