Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.X..........................................................] duration: 552.340386ms FAIL (2025-07-27 04:48:44 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== {0 0}, upload=5911]: Begin upload segment... Jul 27 04:07:17.745307658 (6h15m33.526241043s) [seg_pos={0 0}, upload=5911]: Done begin upload segment. Jul 27 04:07:17.745349241 (6h15m33.526282625s) [noise=true , address=104.36.82.52:4098 , piece_id=M2ETF25QQ53RRGFPUKXJRINIWV7AWXSECEDR2AIM645IED72NURQ, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.745439694 (6h15m33.526373073s) [noise=true , address=104.36.82.56:4244 , piece_id=LQCZ5W54NBHWAVBVYK7HYSSJDC73E6FV4CP4YIM4L5H2N3OLOZXA, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.745972271 (6h15m33.526905654s) [noise=true , address=104.36.82.36:4650 , piece_id=BKJULWGPESBINNTSUYP6E4KZRXKPACVU5I7DOSZBO4YMK2V26HEQ, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.745980243 (6h15m33.526913621s) [noise=true , address=104.36.82.38:4722 , piece_id=B2MBQY5CTMEI6WSE7FTLOV5RUJ22SCNG7N2EEJILEATGJ4TIALXA, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.746403818 (6h15m33.527337198s) [noise=true , address=104.36.82.22:4534 , piece_id=CW4AQT3XUGA4A5OC6NUY4NYOOAMLN5FFJJN5PQBYEON7HPD5LCYQ, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.746455493 (6h15m33.527388875s) [noise=true , address=104.36.82.14:4248 , piece_id=LDSGCUM7Q2VX5CDFI4JRGEEG7D5H5Q5Z5WUWKU277VN2JFRX6UGA, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.746872127 (6h15m33.527805505s) [noise=true , address=104.36.82.40:4954 , piece_id=YLOHTJLRAED66OMEQYPZHNNRI77V6WOR4EHTEZOOEHMGZHJ2O3QQ, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.747123162 (6h15m33.52805654s) [noise=true , address=104.36.82.46:4496 , piece_id=GWNQR5WRGRRQBLSGWJFD3O3UNCMNE62JQNPIV5SYKBUIL3BQKA5A, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.747280678 (6h15m33.528214053s) [noise=true , address=104.36.82.47:4524 , piece_id=DONJFTKUQ2XCW6BOU7AVOKJ7XMDMBPGQVUPRANH7X2YS7PJXUKIA, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.747736400 (6h15m33.528669779s) [noise=true , address=104.36.82.44:4893 , piece_id=HUJICRJ6W54O3AZZR6H3KG6E4TGBTNS46O6XVMQM3PFAZE7A2HZA, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.747755734 (6h15m33.528689109s) [noise=true , address=104.36.82.55:4204 , piece_id=72WISGRHMOAZDV5EDGTBOMLBL4HDOF3CBEMQAAIRGILPC7J5JKAQ, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.748313124 (6h15m33.5292465s) [noise=true , address=104.36.82.58:4287 , piece_id=BVDYUEWU7TVEQDMZXX5SJNS4QICVKMTGFLCUV6P2YHOE7TI25VXA, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.748485217 (6h15m33.529418594s) [noise=true , address=104.36.82.53:4123 , piece_id=NXJRZ5BQSLXSOIOSUJ4MM4AGKHP4UIDUGEJKLUUX7YOGQJA6WWZA, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.748831329 (6h15m33.529764704s) [noise=true , address=104.36.82.42:4829 , piece_id=ZLWJFP3LGX3YLXBJYIR66DXTYEVDBKXKVXCQSF2PCYCYJBIEJGMA, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.749299293 (6h15m33.530232669s) [noise=true , address=104.36.82.8:4002 , piece_id=6XSPWQC3US355PS6IQ4K3IYNQZH5MAT4IVIPGSYRUM3XYPSCWE6Q, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.749790974 (6h15m33.53072436s) [noise=true , address=104.36.82.44:4901 , piece_id=N4YLCLR2QZSULUL2XWONEKBFUMNOU7KUX2QEOK4SSDFSCI7IL6LA, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.750478221 (6h15m33.531411597s) [noise=true , address=104.36.82.54:4172 , piece_id=ZNA53MWDRWPUH6FUR4DJERBDT2IOYGA3IYUADKXXIOLQJLQOXE7Q, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.752884255 (6h15m33.533817625s) [noise=true , address=64.49.13.140:4508 , piece_id=MZ52U75VOIK6Q7QFXMMYRRG7JXFWDFU2JWQFYPERUCMH3BUX6XPQ, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.879839555 (6h15m33.660772917s) [upload=5911]: Next returned nil segment Jul 27 04:07:17.879861389 (6h15m33.660794765s) [upload=5911]: Waiting for error group managing segments... Jul 27 04:07:17.888652438 (6h15m33.669585811s) [noise=true , address=104.36.82.58:4287 , piece_id=BVDYUEWU7TVEQDMZXX5SJNS4QICVKMTGFLCUV6P2YHOE7TI25VXA, seg_pos={0 0}, upload=5911]: Done uploading piece. err: <nil> Jul 27 04:07:17.952192045 (6h15m33.733125429s) [noise=true , address=104.36.82.53:4123 , piece_id=NXJRZ5BQSLXSOIOSUJ4MM4AGKHP4UIDUGEJKLUUX7YOGQJA6WWZA, seg_pos={0 0}, upload=5911]: Done uploading piece. err: <nil> Jul 27 04:07:17.981907539 (6h15m33.762840911s) [noise=true , address=104.36.82.56:4244 , piece_id=LQCZ5W54NBHWAVBVYK7HYSSJDC73E6FV4CP4YIM4L5H2N3OLOZXA, seg_pos={0 0}, upload=5911]: Done uploading piece. err: <nil> Jul 27 04:07:17.982370496 (6h15m33.7633039s) [noise=true , address=104.36.82.55:4204 , piece_id=72WISGRHMOAZDV5EDGTBOMLBL4HDOF3CBEMQAAIRGILPC7J5JKAQ, seg_pos={0 0}, upload=5911]: Done uploading piece. err: <nil> Jul 27 04:07:18.009548506 (6h15m33.790481886s) [noise=true , address=104.36.82.52:4098 , piece_id=M2ETF25QQ53RRGFPUKXJRINIWV7AWXSECEDR2AIM645IED72NURQ, seg_pos={0 0}, upload=5911]: Done uploading piece. err: <nil> Jul 27 04:07:18.023770598 (6h15m33.804703983s) [noise=true , address=104.36.82.40:4954 , piece_id=YLOHTJLRAED66OMEQYPZHNNRI77V6WOR4EHTEZOOEHMGZHJ2O3QQ, seg_pos={0 0}, upload=5911]: Done uploading piece. err: <nil> Jul 27 04:07:18.139745581 (6h15m33.92067895s) [noise=true , address=104.36.82.22:4534 , piece_id=CW4AQT3XUGA4A5OC6NUY4NYOOAMLN5FFJJN5PQBYEON7HPD5LCYQ, seg_pos={0 0}, upload=5911]: Done uploading piece. err: <nil> Jul 27 04:07:18.423748140 (6h15m34.204681511s) [noise=true , address=64.49.13.140:4508 , piece_id=MZ52U75VOIK6Q7QFXMMYRRG7JXFWDFU2JWQFYPERUCMH3BUX6XPQ, seg_pos={0 0}, upload=5911]: Done uploading piece. err: <nil> Jul 27 04:08:16.739940381 (6h16m32.52087375s) [noise=true , address=104.36.82.54:4172 , piece_id=ZNA53MWDRWPUH6FUR4DJERBDT2IOYGA3IYUADKXXIOLQJLQOXE7Q, seg_pos={0 0}, upload=5911]: Done uploading piece. err: <nil> Jul 27 04:08:17.177295397 (6h16m32.958228766s) [noise=true , address=104.36.82.8:4002 , piece_id=6XSPWQC3US355PS6IQ4K3IYNQZH5MAT4IVIPGSYRUM3XYPSCWE6Q, seg_pos={0 0}, upload=5911]: Done uploading piece. err: <nil> Jul 27 04:08:17.534296584 (6h16m33.315229953s) [noise=true , address=104.36.82.36:4650 , piece_id=BKJULWGPESBINNTSUYP6E4KZRXKPACVU5I7DOSZBO4YMK2V26HEQ, seg_pos={0 0}, upload=5911]: Done uploading piece. err: <nil> Jul 27 04:08:17.677851566 (6h16m33.458784943s) [noise=true , address=104.36.82.14:4248 , piece_id=LDSGCUM7Q2VX5CDFI4JRGEEG7D5H5Q5Z5WUWKU277VN2JFRX6UGA, seg_pos={0 0}, upload=5911]: Done uploading piece. err: ecclient: upload failed (node:1u6ir2PLqDa9xGm64cuqq1RqY4qeBa1fqCo9VghspBf54XEXqn, address:104.36.82.14:4248): protocol: rpc: dial tcp 104.36.82.14:4248: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 04:08:17.678115170 (6h16m33.459048549s) [noise=true , address=104.36.82.46:4496 , piece_id=GWNQR5WRGRRQBLSGWJFD3O3UNCMNE62JQNPIV5SYKBUIL3BQKA5A, seg_pos={0 0}, upload=5911]: Done uploading piece. err: ecclient: upload failed (node:12V8XhSWkyGUVYFR8MykifFAP1YAfGCwzqucDcMMfzw3xsVW77Z, address:104.36.82.46:4496): protocol: read tcp 159.223.143.39:52800->104.36.82.46:4496: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 04:08:17.678144409 (6h16m33.459077789s) [noise=true , address=104.36.82.44:4901 , piece_id=N4YLCLR2QZSULUL2XWONEKBFUMNOU7KUX2QEOK4SSDFSCI7IL6LA, seg_pos={0 0}, upload=5911]: Done uploading piece. err: ecclient: upload failed (node:1FHE84zvBb1MKwB1Ht7puTafgQxDEvPt4mYqkYVSSufAMpWQA1, address:104.36.82.44:4901): protocol: rpc: dial tcp 104.36.82.44:4901: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 04:08:17.678229622 (6h16m33.459162988s) [noise=true , address=104.36.82.44:4893 , piece_id=HUJICRJ6W54O3AZZR6H3KG6E4TGBTNS46O6XVMQM3PFAZE7A2HZA, seg_pos={0 0}, upload=5911]: Done uploading piece. err: ecclient: upload failed (node:17VMFZ2T8jrSJVtchciJ9SnyrR6JstvcMoXrN5Q3zTD1aQU1oJ, address:104.36.82.44:4893): protocol: write tcp 159.223.143.39:35302->104.36.82.44:4893: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 04:08:17.678310218 (6h16m33.459243583s) [noise=true , address=104.36.82.42:4829 , piece_id=ZLWJFP3LGX3YLXBJYIR66DXTYEVDBKXKVXCQSF2PCYCYJBIEJGMA, seg_pos={0 0}, upload=5911]: Done uploading piece. err: ecclient: upload failed (node:1F5T8Vt1x4nXFaSenvGBCYjxhRL1H1sxRvzLjW3CqzwDVhq39F, address:104.36.82.42:4829): protocol: rpc: dial tcp 104.36.82.42:4829: i/o timeout; conn closed: op; piecestore: piecestore close: context deadline exceeded Jul 27 04:08:17.678412989 (6h16m33.459346365s) [noise=true , address=104.36.82.38:4722 , piece_id=B2MBQY5CTMEI6WSE7FTLOV5RUJ22SCNG7N2EEJILEATGJ4TIALXA, seg_pos={0 0}, upload=5911]: Done uploading piece. err: ecclient: upload failed (node:12Zw3dtUg9gARz52KhwDLTjcqC3CNm7dpPzpdx7P7PbL1jwe8uc, address:104.36.82.38:4722): protocol: read tcp 159.223.143.39:56048->104.36.82.38:4722: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 04:08:17.678455247 (6h16m33.459388624s) [noise=true , address=104.36.82.47:4524 , piece_id=DONJFTKUQ2XCW6BOU7AVOKJ7XMDMBPGQVUPRANH7X2YS7PJXUKIA, seg_pos={0 0}, upload=5911]: Done uploading piece. err: ecclient: upload failed (node:1xfbCzjSwFDRrR1gPggeN8tNCLbRRWDVzCJ6mqhZsbVSe8d4s4, address:104.36.82.47:4524): protocol: read tcp 159.223.143.39:46694->104.36.82.47:4524: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 04:08:17.678510668 (6h16m33.459444052s) [seg_pos={0 0}, upload=5911]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Jul 27 04:08:17.706442864 (6h16m33.487376232s) [upload=5911]: Done uploading segments... Jul 27 04:08:17.706495329 (6h16m33.487428705s) [upload=5911]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 552.372452ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [..X.........................................................] duration: 656.023719ms FAIL (2025-07-27 04:48:17 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== {0 0}, upload=5911]: Begin upload segment... Jul 27 04:07:17.745307658 (6h15m33.526241043s) [seg_pos={0 0}, upload=5911]: Done begin upload segment. Jul 27 04:07:17.745349241 (6h15m33.526282625s) [noise=true , address=104.36.82.52:4098 , piece_id=M2ETF25QQ53RRGFPUKXJRINIWV7AWXSECEDR2AIM645IED72NURQ, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.745439694 (6h15m33.526373073s) [noise=true , address=104.36.82.56:4244 , piece_id=LQCZ5W54NBHWAVBVYK7HYSSJDC73E6FV4CP4YIM4L5H2N3OLOZXA, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.745972271 (6h15m33.526905654s) [noise=true , address=104.36.82.36:4650 , piece_id=BKJULWGPESBINNTSUYP6E4KZRXKPACVU5I7DOSZBO4YMK2V26HEQ, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.745980243 (6h15m33.526913621s) [noise=true , address=104.36.82.38:4722 , piece_id=B2MBQY5CTMEI6WSE7FTLOV5RUJ22SCNG7N2EEJILEATGJ4TIALXA, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.746403818 (6h15m33.527337198s) [noise=true , address=104.36.82.22:4534 , piece_id=CW4AQT3XUGA4A5OC6NUY4NYOOAMLN5FFJJN5PQBYEON7HPD5LCYQ, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.746455493 (6h15m33.527388875s) [noise=true , address=104.36.82.14:4248 , piece_id=LDSGCUM7Q2VX5CDFI4JRGEEG7D5H5Q5Z5WUWKU277VN2JFRX6UGA, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.746872127 (6h15m33.527805505s) [noise=true , address=104.36.82.40:4954 , piece_id=YLOHTJLRAED66OMEQYPZHNNRI77V6WOR4EHTEZOOEHMGZHJ2O3QQ, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.747123162 (6h15m33.52805654s) [noise=true , address=104.36.82.46:4496 , piece_id=GWNQR5WRGRRQBLSGWJFD3O3UNCMNE62JQNPIV5SYKBUIL3BQKA5A, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.747280678 (6h15m33.528214053s) [noise=true , address=104.36.82.47:4524 , piece_id=DONJFTKUQ2XCW6BOU7AVOKJ7XMDMBPGQVUPRANH7X2YS7PJXUKIA, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.747736400 (6h15m33.528669779s) [noise=true , address=104.36.82.44:4893 , piece_id=HUJICRJ6W54O3AZZR6H3KG6E4TGBTNS46O6XVMQM3PFAZE7A2HZA, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.747755734 (6h15m33.528689109s) [noise=true , address=104.36.82.55:4204 , piece_id=72WISGRHMOAZDV5EDGTBOMLBL4HDOF3CBEMQAAIRGILPC7J5JKAQ, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.748313124 (6h15m33.5292465s) [noise=true , address=104.36.82.58:4287 , piece_id=BVDYUEWU7TVEQDMZXX5SJNS4QICVKMTGFLCUV6P2YHOE7TI25VXA, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.748485217 (6h15m33.529418594s) [noise=true , address=104.36.82.53:4123 , piece_id=NXJRZ5BQSLXSOIOSUJ4MM4AGKHP4UIDUGEJKLUUX7YOGQJA6WWZA, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.748831329 (6h15m33.529764704s) [noise=true , address=104.36.82.42:4829 , piece_id=ZLWJFP3LGX3YLXBJYIR66DXTYEVDBKXKVXCQSF2PCYCYJBIEJGMA, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.749299293 (6h15m33.530232669s) [noise=true , address=104.36.82.8:4002 , piece_id=6XSPWQC3US355PS6IQ4K3IYNQZH5MAT4IVIPGSYRUM3XYPSCWE6Q, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.749790974 (6h15m33.53072436s) [noise=true , address=104.36.82.44:4901 , piece_id=N4YLCLR2QZSULUL2XWONEKBFUMNOU7KUX2QEOK4SSDFSCI7IL6LA, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.750478221 (6h15m33.531411597s) [noise=true , address=104.36.82.54:4172 , piece_id=ZNA53MWDRWPUH6FUR4DJERBDT2IOYGA3IYUADKXXIOLQJLQOXE7Q, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.752884255 (6h15m33.533817625s) [noise=true , address=64.49.13.140:4508 , piece_id=MZ52U75VOIK6Q7QFXMMYRRG7JXFWDFU2JWQFYPERUCMH3BUX6XPQ, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.879839555 (6h15m33.660772917s) [upload=5911]: Next returned nil segment Jul 27 04:07:17.879861389 (6h15m33.660794765s) [upload=5911]: Waiting for error group managing segments... Jul 27 04:07:17.888652438 (6h15m33.669585811s) [noise=true , address=104.36.82.58:4287 , piece_id=BVDYUEWU7TVEQDMZXX5SJNS4QICVKMTGFLCUV6P2YHOE7TI25VXA, seg_pos={0 0}, upload=5911]: Done uploading piece. err: <nil> Jul 27 04:07:17.952192045 (6h15m33.733125429s) [noise=true , address=104.36.82.53:4123 , piece_id=NXJRZ5BQSLXSOIOSUJ4MM4AGKHP4UIDUGEJKLUUX7YOGQJA6WWZA, seg_pos={0 0}, upload=5911]: Done uploading piece. err: <nil> Jul 27 04:07:17.981907539 (6h15m33.762840911s) [noise=true , address=104.36.82.56:4244 , piece_id=LQCZ5W54NBHWAVBVYK7HYSSJDC73E6FV4CP4YIM4L5H2N3OLOZXA, seg_pos={0 0}, upload=5911]: Done uploading piece. err: <nil> Jul 27 04:07:17.982370496 (6h15m33.7633039s) [noise=true , address=104.36.82.55:4204 , piece_id=72WISGRHMOAZDV5EDGTBOMLBL4HDOF3CBEMQAAIRGILPC7J5JKAQ, seg_pos={0 0}, upload=5911]: Done uploading piece. err: <nil> Jul 27 04:07:18.009548506 (6h15m33.790481886s) [noise=true , address=104.36.82.52:4098 , piece_id=M2ETF25QQ53RRGFPUKXJRINIWV7AWXSECEDR2AIM645IED72NURQ, seg_pos={0 0}, upload=5911]: Done uploading piece. err: <nil> Jul 27 04:07:18.023770598 (6h15m33.804703983s) [noise=true , address=104.36.82.40:4954 , piece_id=YLOHTJLRAED66OMEQYPZHNNRI77V6WOR4EHTEZOOEHMGZHJ2O3QQ, seg_pos={0 0}, upload=5911]: Done uploading piece. err: <nil> Jul 27 04:07:18.139745581 (6h15m33.92067895s) [noise=true , address=104.36.82.22:4534 , piece_id=CW4AQT3XUGA4A5OC6NUY4NYOOAMLN5FFJJN5PQBYEON7HPD5LCYQ, seg_pos={0 0}, upload=5911]: Done uploading piece. err: <nil> Jul 27 04:07:18.423748140 (6h15m34.204681511s) [noise=true , address=64.49.13.140:4508 , piece_id=MZ52U75VOIK6Q7QFXMMYRRG7JXFWDFU2JWQFYPERUCMH3BUX6XPQ, seg_pos={0 0}, upload=5911]: Done uploading piece. err: <nil> Jul 27 04:08:16.739940381 (6h16m32.52087375s) [noise=true , address=104.36.82.54:4172 , piece_id=ZNA53MWDRWPUH6FUR4DJERBDT2IOYGA3IYUADKXXIOLQJLQOXE7Q, seg_pos={0 0}, upload=5911]: Done uploading piece. err: <nil> Jul 27 04:08:17.177295397 (6h16m32.958228766s) [noise=true , address=104.36.82.8:4002 , piece_id=6XSPWQC3US355PS6IQ4K3IYNQZH5MAT4IVIPGSYRUM3XYPSCWE6Q, seg_pos={0 0}, upload=5911]: Done uploading piece. err: <nil> Jul 27 04:08:17.534296584 (6h16m33.315229953s) [noise=true , address=104.36.82.36:4650 , piece_id=BKJULWGPESBINNTSUYP6E4KZRXKPACVU5I7DOSZBO4YMK2V26HEQ, seg_pos={0 0}, upload=5911]: Done uploading piece. err: <nil> Jul 27 04:08:17.677851566 (6h16m33.458784943s) [noise=true , address=104.36.82.14:4248 , piece_id=LDSGCUM7Q2VX5CDFI4JRGEEG7D5H5Q5Z5WUWKU277VN2JFRX6UGA, seg_pos={0 0}, upload=5911]: Done uploading piece. err: ecclient: upload failed (node:1u6ir2PLqDa9xGm64cuqq1RqY4qeBa1fqCo9VghspBf54XEXqn, address:104.36.82.14:4248): protocol: rpc: dial tcp 104.36.82.14:4248: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 04:08:17.678115170 (6h16m33.459048549s) [noise=true , address=104.36.82.46:4496 , piece_id=GWNQR5WRGRRQBLSGWJFD3O3UNCMNE62JQNPIV5SYKBUIL3BQKA5A, seg_pos={0 0}, upload=5911]: Done uploading piece. err: ecclient: upload failed (node:12V8XhSWkyGUVYFR8MykifFAP1YAfGCwzqucDcMMfzw3xsVW77Z, address:104.36.82.46:4496): protocol: read tcp 159.223.143.39:52800->104.36.82.46:4496: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 04:08:17.678144409 (6h16m33.459077789s) [noise=true , address=104.36.82.44:4901 , piece_id=N4YLCLR2QZSULUL2XWONEKBFUMNOU7KUX2QEOK4SSDFSCI7IL6LA, seg_pos={0 0}, upload=5911]: Done uploading piece. err: ecclient: upload failed (node:1FHE84zvBb1MKwB1Ht7puTafgQxDEvPt4mYqkYVSSufAMpWQA1, address:104.36.82.44:4901): protocol: rpc: dial tcp 104.36.82.44:4901: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 04:08:17.678229622 (6h16m33.459162988s) [noise=true , address=104.36.82.44:4893 , piece_id=HUJICRJ6W54O3AZZR6H3KG6E4TGBTNS46O6XVMQM3PFAZE7A2HZA, seg_pos={0 0}, upload=5911]: Done uploading piece. err: ecclient: upload failed (node:17VMFZ2T8jrSJVtchciJ9SnyrR6JstvcMoXrN5Q3zTD1aQU1oJ, address:104.36.82.44:4893): protocol: write tcp 159.223.143.39:35302->104.36.82.44:4893: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 04:08:17.678310218 (6h16m33.459243583s) [noise=true , address=104.36.82.42:4829 , piece_id=ZLWJFP3LGX3YLXBJYIR66DXTYEVDBKXKVXCQSF2PCYCYJBIEJGMA, seg_pos={0 0}, upload=5911]: Done uploading piece. err: ecclient: upload failed (node:1F5T8Vt1x4nXFaSenvGBCYjxhRL1H1sxRvzLjW3CqzwDVhq39F, address:104.36.82.42:4829): protocol: rpc: dial tcp 104.36.82.42:4829: i/o timeout; conn closed: op; piecestore: piecestore close: context deadline exceeded Jul 27 04:08:17.678412989 (6h16m33.459346365s) [noise=true , address=104.36.82.38:4722 , piece_id=B2MBQY5CTMEI6WSE7FTLOV5RUJ22SCNG7N2EEJILEATGJ4TIALXA, seg_pos={0 0}, upload=5911]: Done uploading piece. err: ecclient: upload failed (node:12Zw3dtUg9gARz52KhwDLTjcqC3CNm7dpPzpdx7P7PbL1jwe8uc, address:104.36.82.38:4722): protocol: read tcp 159.223.143.39:56048->104.36.82.38:4722: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 04:08:17.678455247 (6h16m33.459388624s) [noise=true , address=104.36.82.47:4524 , piece_id=DONJFTKUQ2XCW6BOU7AVOKJ7XMDMBPGQVUPRANH7X2YS7PJXUKIA, seg_pos={0 0}, upload=5911]: Done uploading piece. err: ecclient: upload failed (node:1xfbCzjSwFDRrR1gPggeN8tNCLbRRWDVzCJ6mqhZsbVSe8d4s4, address:104.36.82.47:4524): protocol: read tcp 159.223.143.39:46694->104.36.82.47:4524: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 04:08:17.678510668 (6h16m33.459444052s) [seg_pos={0 0}, upload=5911]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Jul 27 04:08:17.706442864 (6h16m33.487376232s) [upload=5911]: Done uploading segments... Jul 27 04:08:17.706495329 (6h16m33.487428705s) [upload=5911]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 656.066048ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............X............................................] duration: 342.474594ms FAIL (2025-07-27 04:47:53 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 12) Duration: 342.505386ms core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...X........................................................] duration: 492.402639ms FAIL (2025-07-27 04:47:44 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== ece_id=NYOFDEZ7CV4PHGQ6JZMEEVAYGJRPYG6IBW7MYJO42HMWOJM7PHZA, seg_pos={0 0}, upload=5879]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12crcXpNe9NgeXTsUpooLSBgrSUFbRxbSbanLa39w2Hr5mGFww): protocol: read tcp 159.223.143.39:53630->104.36.82.5:4840: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 04:05:32.508904485 (6h13m48.289837866s) [seg_pos={0 0}, upload=5879]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 27 04:05:32.509028070 (6h13m48.289961457s) [upload=5879]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 27 04:05:32.509034645 (6h13m48.289968129s) [upload=5879]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 27 04:05:32.581834839 (6h13m48.362768225s) [upload=5879]: Done uploading segments... Jul 27 04:05:32.581875919 (6h13m48.362809298s) [upload=5879]: Upload finished. err: <nil> Jul 27 04:06:23.829155331 (6h14m39.610088696s) [upload=5895]: Starting upload Jul 27 04:06:23.829225021 (6h14m39.610158398s) [upload=5895]: Done starting upload Jul 27 04:06:23.829294606 (6h14m39.61022799s) [upload=5895]: Uploading segments... Jul 27 04:06:23.829577656 (6h14m39.610511053s) [upload=5895]: Got next segment. Inline: false Jul 27 04:06:23.829649664 (6h14m39.610583027s) [upload=5895]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 27 04:06:23.829657065 (6h14m39.61059043s) [upload=5895]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 27 04:06:23.861405530 (6h14m39.642338913s) [seg_pos={0 0}, upload=5895]: Begin upload segment... Jul 27 04:06:23.861476351 (6h14m39.642409713s) [seg_pos={0 0}, upload=5895]: Done begin upload segment. Jul 27 04:06:23.861509120 (6h14m39.642442506s) [noise=true , address=77.90.31.19:4843 , piece_id=YML5RDQIK432Z4FSE4DKXIV5U47LQBEZRFMAVLANUSGFUQR23UWQ, seg_pos={0 0}, upload=5895]: Uploading piece... Jul 27 04:06:23.861953723 (6h14m39.6428871s) [noise=true , address=45.91.116.10:4081 , piece_id=XGDTSEZOUNRTEVWHTO3DHTZMH2VZZOPJPEKLCFAHTXESLLYABHSA, seg_pos={0 0}, upload=5895]: Uploading piece... Jul 27 04:06:23.862060983 (6h14m39.642994358s) [noise=true , address=45.91.116.9:4262 , piece_id=M5ETD45HH2XZD6HDLTJ3JO2BOOIROXFWKVPTN42PSQEWG4F7KAAQ, seg_pos={0 0}, upload=5895]: Uploading piece... Jul 27 04:06:23.862759464 (6h14m39.64369284s) [noise=true , address=104.36.82.4:4804 , piece_id=IU6FDNVB5AUPA7MJ6RSXUTLBETPHOFGNJ7YGWBFIZSQFSO6W6BBA, seg_pos={0 0}, upload=5895]: Uploading piece... Jul 27 04:06:23.862835086 (6h14m39.643768461s) [noise=true , address=104.36.82.6:4873 , piece_id=YP4VJORZZUL5EVCH3P7LD3LNVMHB7CWRTKHGIA7UROXE44N6MMZA, seg_pos={0 0}, upload=5895]: Uploading piece... Jul 27 04:06:23.863593880 (6h14m39.644527258s) [noise=true , address=77.90.31.15:4697 , piece_id=Q76TJDSZPSYZSJQNFVOI462U6DFILMH2RV3LQISODXSOCXGRKW6Q, seg_pos={0 0}, upload=5895]: Uploading piece... Jul 27 04:06:23.863656891 (6h14m39.644590266s) [noise=true , address=45.91.116.7:4011 , piece_id=XMH6MCCHZBBRQWHJFWLRGYHLQD4KUGSGL7MMZSKA5TZJCND3DUDA, seg_pos={0 0}, upload=5895]: Uploading piece... Jul 27 04:06:23.864408612 (6h14m39.645341987s) [noise=true , address=45.91.116.6:4973 , piece_id=J3VULPU24CIM64VEVE3UHO62HPN6TJXGZU6BCJIS5JQJKHJK2LIQ, seg_pos={0 0}, upload=5895]: Uploading piece... Jul 27 04:06:23.864521549 (6h14m39.64545491s) [noise=true , address=104.36.82.10:4079 , piece_id=2SH5D6JOMLFR7VSN7ZY5CLMR5RHIRFYQ7Z5I2JIYTJQM43G4O6OQ, seg_pos={0 0}, upload=5895]: Uploading piece... Jul 27 04:06:23.865518491 (6h14m39.646451872s) [noise=true , address=77.90.31.17:4771 , piece_id=U2LF22GKCMKRZN6EWCF26ITDZQHBFZXCGSQXT33MCX6P57IGI35Q, seg_pos={0 0}, upload=5895]: Uploading piece... Jul 27 04:06:23.865654950 (6h14m39.646588325s) [noise=true , address=45.91.116.8:4048 , piece_id=5REWMVVTBVLDQXQ5KAEHLY4VJL5SGF7ONQWDQQ3N2FDU3OAO6AZA, seg_pos={0 0}, upload=5895]: Uploading piece... Jul 27 04:06:23.866190892 (6h14m39.647124271s) [noise=true , address=104.36.82.12:4151 , piece_id=E2TEUTQBD4Y4AUHSCPMCIFEXBSY7S5HJWBKNKWWTUWEW2BLAAE6Q, seg_pos={0 0}, upload=5895]: Uploading piece... Jul 27 04:06:23.866412590 (6h14m39.647345968s) [noise=true , address=104.36.82.11:4115 , piece_id=YXMNVT6ZHGESGQ2BXKZUVR7U2CNNAI2PO7ZTMDG5UIOV7HTDDUTA, seg_pos={0 0}, upload=5895]: Uploading piece... Jul 27 04:06:23.866715972 (6h14m39.647649359s) [noise=true , address=45.91.116.13:4153 , piece_id=H4535KOCG3F4RKA6AKKMTJSOM6Q6PGRC6M7FR3UIS3BORTSP5UQA, seg_pos={0 0}, upload=5895]: Uploading piece... Jul 27 04:06:23.866998965 (6h14m39.64793234s) [noise=true , address=104.36.82.13:4186 , piece_id=TBLDKALGJYBGR5NLCOCKJ4QKEFNIJXCKAFUBD7VR3UMNWEMKMKHQ, seg_pos={0 0}, upload=5895]: Uploading piece... Jul 27 04:06:23.867230593 (6h14m39.648163974s) [noise=true , address=104.36.82.16:4293 , piece_id=AHH6PRFUSGTDGB7H52DRITARA4IF6GFLVSRYS2Y6EPJCZSKUOYWQ, seg_pos={0 0}, upload=5895]: Uploading piece... Jul 27 04:06:23.949422808 (6h14m39.730356187s) [upload=5895]: Next returned nil segment Jul 27 04:06:23.949442576 (6h14m39.73037596s) [upload=5895]: Waiting for error group managing segments... Jul 27 04:06:23.953497426 (6h14m39.734430802s) [noise=true , address=45.91.116.10:4081 , piece_id=XGDTSEZOUNRTEVWHTO3DHTZMH2VZZOPJPEKLCFAHTXESLLYABHSA, seg_pos={0 0}, upload=5895]: Done uploading piece. err: <nil> Jul 27 04:06:23.953531796 (6h14m39.734465159s) [noise=true , address=45.91.116.6:4973 , piece_id=J3VULPU24CIM64VEVE3UHO62HPN6TJXGZU6BCJIS5JQJKHJK2LIQ, seg_pos={0 0}, upload=5895]: Done uploading piece. err: <nil> Jul 27 04:06:23.954068660 (6h14m39.735002029s) [noise=true , address=45.91.116.9:4262 , piece_id=M5ETD45HH2XZD6HDLTJ3JO2BOOIROXFWKVPTN42PSQEWG4F7KAAQ, seg_pos={0 0}, upload=5895]: Done uploading piece. err: <nil> Jul 27 04:06:23.954194339 (6h14m39.735127707s) [noise=true , address=45.91.116.8:4048 , piece_id=5REWMVVTBVLDQXQ5KAEHLY4VJL5SGF7ONQWDQQ3N2FDU3OAO6AZA, seg_pos={0 0}, upload=5895]: Done uploading piece. err: <nil> Jul 27 04:06:23.954601730 (6h14m39.735535112s) [noise=true , address=45.91.116.7:4011 , piece_id=XMH6MCCHZBBRQWHJFWLRGYHLQD4KUGSGL7MMZSKA5TZJCND3DUDA, seg_pos={0 0}, upload=5895]: Done uploading piece. err: <nil> Jul 27 04:06:23.954755872 (6h14m39.735689237s) [noise=true , address=45.91.116.13:4153 , piece_id=H4535KOCG3F4RKA6AKKMTJSOM6Q6PGRC6M7FR3UIS3BORTSP5UQA, seg_pos={0 0}, upload=5895]: Done uploading piece. err: <nil> Jul 27 04:06:23.955611883 (6h14m39.73654526s) [noise=true , address=77.90.31.19:4843 , piece_id=YML5RDQIK432Z4FSE4DKXIV5U47LQBEZRFMAVLANUSGFUQR23UWQ, seg_pos={0 0}, upload=5895]: Done uploading piece. err: <nil> Jul 27 04:06:23.956373391 (6h14m39.737306756s) [noise=true , address=77.90.31.15:4697 , piece_id=Q76TJDSZPSYZSJQNFVOI462U6DFILMH2RV3LQISODXSOCXGRKW6Q, seg_pos={0 0}, upload=5895]: Done uploading piece. err: <nil> Jul 27 04:06:23.958659122 (6h14m39.73959249s) [noise=true , address=77.90.31.17:4771 , piece_id=U2LF22GKCMKRZN6EWCF26ITDZQHBFZXCGSQXT33MCX6P57IGI35Q, seg_pos={0 0}, upload=5895]: Done uploading piece. err: <nil> Jul 27 04:06:35.058944013 (6h14m50.83987739s) [noise=true , address=104.36.82.16:4293 , piece_id=AHH6PRFUSGTDGB7H52DRITARA4IF6GFLVSRYS2Y6EPJCZSKUOYWQ, seg_pos={0 0}, upload=5895]: Done uploading piece. err: <nil> Jul 27 04:06:41.103204002 (6h14m56.884137386s) [noise=true , address=104.36.82.4:4804 , piece_id=IU6FDNVB5AUPA7MJ6RSXUTLBETPHOFGNJ7YGWBFIZSQFSO6W6BBA, seg_pos={0 0}, upload=5895]: Done uploading piece. err: <nil> Jul 27 04:06:43.191702285 (6h14m58.972635651s) [noise=true , address=104.36.82.6:4873 , piece_id=YP4VJORZZUL5EVCH3P7LD3LNVMHB7CWRTKHGIA7UROXE44N6MMZA, seg_pos={0 0}, upload=5895]: Done uploading piece. err: <nil> Jul 27 04:06:53.777973493 (6h15m9.558906861s) [noise=true , address=104.36.82.11:4115 , piece_id=YXMNVT6ZHGESGQ2BXKZUVR7U2CNNAI2PO7ZTMDG5UIOV7HTDDUTA, seg_pos={0 0}, upload=5895]: Done uploading piece. err: ecclient: upload failed (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR, address:104.36.82.11:4115): protocol: write tcp 159.223.143.39:34210->104.36.82.11:4115: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 04:06:53.778019532 (6h15m9.558952909s) [noise=true , address=104.36.82.12:4151 , piece_id=E2TEUTQBD4Y4AUHSCPMCIFEXBSY7S5HJWBKNKWWTUWEW2BLAAE6Q, seg_pos={0 0}, upload=5895]: Done uploading piece. err: ecclient: upload failed (node:1w6pwZMdeQKjt92qx6WbTWU5VPYJto5gS87f6iKvCBZH6LfyeB, address:104.36.82.12:4151): protocol: rpc: dial tcp 104.36.82.12:4151: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 04:06:53.778043653 (6h15m9.558977021s) [noise=true , address=104.36.82.13:4186 , piece_id=TBLDKALGJYBGR5NLCOCKJ4QKEFNIJXCKAFUBD7VR3UMNWEMKMKHQ, seg_pos={0 0}, upload=5895]: Done uploading piece. err: ecclient: upload failed (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht, address:104.36.82.13:4186): protocol: read tcp 159.223.143.39:34340->104.36.82.13:4186: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 04:06:53.778095228 (6h15m9.559028593s) [noise=true , address=104.36.82.10:4079 , piece_id=2SH5D6JOMLFR7VSN7ZY5CLMR5RHIRFYQ7Z5I2JIYTJQM43G4O6OQ, seg_pos={0 0}, upload=5895]: Done uploading piece. err: ecclient: upload failed (node:12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud, address:104.36.82.10:4079): protocol: write tcp 159.223.143.39:53840->104.36.82.10:4079: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 04:06:53.778142167 (6h15m9.559075532s) [seg_pos={0 0}, upload=5895]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 27 04:06:53.837032161 (6h15m9.617965543s) [upload=5895]: Done uploading segments... Jul 27 04:06:53.837055996 (6h15m9.617989366s) [upload=5895]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 492.429554ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [................X...........................................] duration: 417.101307ms FAIL (2025-07-27 04:47:18 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 12) Duration: 417.141817ms core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [...X........................................................] duration: 2.784369533s FAIL (2025-07-27 04:47:07 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== {0 0}, upload=5911]: Begin upload segment... Jul 27 04:07:17.745307658 (6h15m33.526241043s) [seg_pos={0 0}, upload=5911]: Done begin upload segment. Jul 27 04:07:17.745349241 (6h15m33.526282625s) [noise=true , address=104.36.82.52:4098 , piece_id=M2ETF25QQ53RRGFPUKXJRINIWV7AWXSECEDR2AIM645IED72NURQ, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.745439694 (6h15m33.526373073s) [noise=true , address=104.36.82.56:4244 , piece_id=LQCZ5W54NBHWAVBVYK7HYSSJDC73E6FV4CP4YIM4L5H2N3OLOZXA, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.745972271 (6h15m33.526905654s) [noise=true , address=104.36.82.36:4650 , piece_id=BKJULWGPESBINNTSUYP6E4KZRXKPACVU5I7DOSZBO4YMK2V26HEQ, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.745980243 (6h15m33.526913621s) [noise=true , address=104.36.82.38:4722 , piece_id=B2MBQY5CTMEI6WSE7FTLOV5RUJ22SCNG7N2EEJILEATGJ4TIALXA, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.746403818 (6h15m33.527337198s) [noise=true , address=104.36.82.22:4534 , piece_id=CW4AQT3XUGA4A5OC6NUY4NYOOAMLN5FFJJN5PQBYEON7HPD5LCYQ, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.746455493 (6h15m33.527388875s) [noise=true , address=104.36.82.14:4248 , piece_id=LDSGCUM7Q2VX5CDFI4JRGEEG7D5H5Q5Z5WUWKU277VN2JFRX6UGA, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.746872127 (6h15m33.527805505s) [noise=true , address=104.36.82.40:4954 , piece_id=YLOHTJLRAED66OMEQYPZHNNRI77V6WOR4EHTEZOOEHMGZHJ2O3QQ, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.747123162 (6h15m33.52805654s) [noise=true , address=104.36.82.46:4496 , piece_id=GWNQR5WRGRRQBLSGWJFD3O3UNCMNE62JQNPIV5SYKBUIL3BQKA5A, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.747280678 (6h15m33.528214053s) [noise=true , address=104.36.82.47:4524 , piece_id=DONJFTKUQ2XCW6BOU7AVOKJ7XMDMBPGQVUPRANH7X2YS7PJXUKIA, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.747736400 (6h15m33.528669779s) [noise=true , address=104.36.82.44:4893 , piece_id=HUJICRJ6W54O3AZZR6H3KG6E4TGBTNS46O6XVMQM3PFAZE7A2HZA, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.747755734 (6h15m33.528689109s) [noise=true , address=104.36.82.55:4204 , piece_id=72WISGRHMOAZDV5EDGTBOMLBL4HDOF3CBEMQAAIRGILPC7J5JKAQ, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.748313124 (6h15m33.5292465s) [noise=true , address=104.36.82.58:4287 , piece_id=BVDYUEWU7TVEQDMZXX5SJNS4QICVKMTGFLCUV6P2YHOE7TI25VXA, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.748485217 (6h15m33.529418594s) [noise=true , address=104.36.82.53:4123 , piece_id=NXJRZ5BQSLXSOIOSUJ4MM4AGKHP4UIDUGEJKLUUX7YOGQJA6WWZA, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.748831329 (6h15m33.529764704s) [noise=true , address=104.36.82.42:4829 , piece_id=ZLWJFP3LGX3YLXBJYIR66DXTYEVDBKXKVXCQSF2PCYCYJBIEJGMA, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.749299293 (6h15m33.530232669s) [noise=true , address=104.36.82.8:4002 , piece_id=6XSPWQC3US355PS6IQ4K3IYNQZH5MAT4IVIPGSYRUM3XYPSCWE6Q, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.749790974 (6h15m33.53072436s) [noise=true , address=104.36.82.44:4901 , piece_id=N4YLCLR2QZSULUL2XWONEKBFUMNOU7KUX2QEOK4SSDFSCI7IL6LA, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.750478221 (6h15m33.531411597s) [noise=true , address=104.36.82.54:4172 , piece_id=ZNA53MWDRWPUH6FUR4DJERBDT2IOYGA3IYUADKXXIOLQJLQOXE7Q, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.752884255 (6h15m33.533817625s) [noise=true , address=64.49.13.140:4508 , piece_id=MZ52U75VOIK6Q7QFXMMYRRG7JXFWDFU2JWQFYPERUCMH3BUX6XPQ, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.879839555 (6h15m33.660772917s) [upload=5911]: Next returned nil segment Jul 27 04:07:17.879861389 (6h15m33.660794765s) [upload=5911]: Waiting for error group managing segments... Jul 27 04:07:17.888652438 (6h15m33.669585811s) [noise=true , address=104.36.82.58:4287 , piece_id=BVDYUEWU7TVEQDMZXX5SJNS4QICVKMTGFLCUV6P2YHOE7TI25VXA, seg_pos={0 0}, upload=5911]: Done uploading piece. err: <nil> Jul 27 04:07:17.952192045 (6h15m33.733125429s) [noise=true , address=104.36.82.53:4123 , piece_id=NXJRZ5BQSLXSOIOSUJ4MM4AGKHP4UIDUGEJKLUUX7YOGQJA6WWZA, seg_pos={0 0}, upload=5911]: Done uploading piece. err: <nil> Jul 27 04:07:17.981907539 (6h15m33.762840911s) [noise=true , address=104.36.82.56:4244 , piece_id=LQCZ5W54NBHWAVBVYK7HYSSJDC73E6FV4CP4YIM4L5H2N3OLOZXA, seg_pos={0 0}, upload=5911]: Done uploading piece. err: <nil> Jul 27 04:07:17.982370496 (6h15m33.7633039s) [noise=true , address=104.36.82.55:4204 , piece_id=72WISGRHMOAZDV5EDGTBOMLBL4HDOF3CBEMQAAIRGILPC7J5JKAQ, seg_pos={0 0}, upload=5911]: Done uploading piece. err: <nil> Jul 27 04:07:18.009548506 (6h15m33.790481886s) [noise=true , address=104.36.82.52:4098 , piece_id=M2ETF25QQ53RRGFPUKXJRINIWV7AWXSECEDR2AIM645IED72NURQ, seg_pos={0 0}, upload=5911]: Done uploading piece. err: <nil> Jul 27 04:07:18.023770598 (6h15m33.804703983s) [noise=true , address=104.36.82.40:4954 , piece_id=YLOHTJLRAED66OMEQYPZHNNRI77V6WOR4EHTEZOOEHMGZHJ2O3QQ, seg_pos={0 0}, upload=5911]: Done uploading piece. err: <nil> Jul 27 04:07:18.139745581 (6h15m33.92067895s) [noise=true , address=104.36.82.22:4534 , piece_id=CW4AQT3XUGA4A5OC6NUY4NYOOAMLN5FFJJN5PQBYEON7HPD5LCYQ, seg_pos={0 0}, upload=5911]: Done uploading piece. err: <nil> Jul 27 04:07:18.423748140 (6h15m34.204681511s) [noise=true , address=64.49.13.140:4508 , piece_id=MZ52U75VOIK6Q7QFXMMYRRG7JXFWDFU2JWQFYPERUCMH3BUX6XPQ, seg_pos={0 0}, upload=5911]: Done uploading piece. err: <nil> Jul 27 04:08:16.739940381 (6h16m32.52087375s) [noise=true , address=104.36.82.54:4172 , piece_id=ZNA53MWDRWPUH6FUR4DJERBDT2IOYGA3IYUADKXXIOLQJLQOXE7Q, seg_pos={0 0}, upload=5911]: Done uploading piece. err: <nil> Jul 27 04:08:17.177295397 (6h16m32.958228766s) [noise=true , address=104.36.82.8:4002 , piece_id=6XSPWQC3US355PS6IQ4K3IYNQZH5MAT4IVIPGSYRUM3XYPSCWE6Q, seg_pos={0 0}, upload=5911]: Done uploading piece. err: <nil> Jul 27 04:08:17.534296584 (6h16m33.315229953s) [noise=true , address=104.36.82.36:4650 , piece_id=BKJULWGPESBINNTSUYP6E4KZRXKPACVU5I7DOSZBO4YMK2V26HEQ, seg_pos={0 0}, upload=5911]: Done uploading piece. err: <nil> Jul 27 04:08:17.677851566 (6h16m33.458784943s) [noise=true , address=104.36.82.14:4248 , piece_id=LDSGCUM7Q2VX5CDFI4JRGEEG7D5H5Q5Z5WUWKU277VN2JFRX6UGA, seg_pos={0 0}, upload=5911]: Done uploading piece. err: ecclient: upload failed (node:1u6ir2PLqDa9xGm64cuqq1RqY4qeBa1fqCo9VghspBf54XEXqn, address:104.36.82.14:4248): protocol: rpc: dial tcp 104.36.82.14:4248: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 04:08:17.678115170 (6h16m33.459048549s) [noise=true , address=104.36.82.46:4496 , piece_id=GWNQR5WRGRRQBLSGWJFD3O3UNCMNE62JQNPIV5SYKBUIL3BQKA5A, seg_pos={0 0}, upload=5911]: Done uploading piece. err: ecclient: upload failed (node:12V8XhSWkyGUVYFR8MykifFAP1YAfGCwzqucDcMMfzw3xsVW77Z, address:104.36.82.46:4496): protocol: read tcp 159.223.143.39:52800->104.36.82.46:4496: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 04:08:17.678144409 (6h16m33.459077789s) [noise=true , address=104.36.82.44:4901 , piece_id=N4YLCLR2QZSULUL2XWONEKBFUMNOU7KUX2QEOK4SSDFSCI7IL6LA, seg_pos={0 0}, upload=5911]: Done uploading piece. err: ecclient: upload failed (node:1FHE84zvBb1MKwB1Ht7puTafgQxDEvPt4mYqkYVSSufAMpWQA1, address:104.36.82.44:4901): protocol: rpc: dial tcp 104.36.82.44:4901: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 04:08:17.678229622 (6h16m33.459162988s) [noise=true , address=104.36.82.44:4893 , piece_id=HUJICRJ6W54O3AZZR6H3KG6E4TGBTNS46O6XVMQM3PFAZE7A2HZA, seg_pos={0 0}, upload=5911]: Done uploading piece. err: ecclient: upload failed (node:17VMFZ2T8jrSJVtchciJ9SnyrR6JstvcMoXrN5Q3zTD1aQU1oJ, address:104.36.82.44:4893): protocol: write tcp 159.223.143.39:35302->104.36.82.44:4893: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 04:08:17.678310218 (6h16m33.459243583s) [noise=true , address=104.36.82.42:4829 , piece_id=ZLWJFP3LGX3YLXBJYIR66DXTYEVDBKXKVXCQSF2PCYCYJBIEJGMA, seg_pos={0 0}, upload=5911]: Done uploading piece. err: ecclient: upload failed (node:1F5T8Vt1x4nXFaSenvGBCYjxhRL1H1sxRvzLjW3CqzwDVhq39F, address:104.36.82.42:4829): protocol: rpc: dial tcp 104.36.82.42:4829: i/o timeout; conn closed: op; piecestore: piecestore close: context deadline exceeded Jul 27 04:08:17.678412989 (6h16m33.459346365s) [noise=true , address=104.36.82.38:4722 , piece_id=B2MBQY5CTMEI6WSE7FTLOV5RUJ22SCNG7N2EEJILEATGJ4TIALXA, seg_pos={0 0}, upload=5911]: Done uploading piece. err: ecclient: upload failed (node:12Zw3dtUg9gARz52KhwDLTjcqC3CNm7dpPzpdx7P7PbL1jwe8uc, address:104.36.82.38:4722): protocol: read tcp 159.223.143.39:56048->104.36.82.38:4722: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 04:08:17.678455247 (6h16m33.459388624s) [noise=true , address=104.36.82.47:4524 , piece_id=DONJFTKUQ2XCW6BOU7AVOKJ7XMDMBPGQVUPRANH7X2YS7PJXUKIA, seg_pos={0 0}, upload=5911]: Done uploading piece. err: ecclient: upload failed (node:1xfbCzjSwFDRrR1gPggeN8tNCLbRRWDVzCJ6mqhZsbVSe8d4s4, address:104.36.82.47:4524): protocol: read tcp 159.223.143.39:46694->104.36.82.47:4524: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 04:08:17.678510668 (6h16m33.459444052s) [seg_pos={0 0}, upload=5911]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Jul 27 04:08:17.706442864 (6h16m33.487376232s) [upload=5911]: Done uploading segments... Jul 27 04:08:17.706495329 (6h16m33.487428705s) [upload=5911]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 2.78447669s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................X..........................................] duration: 605.653164ms FAIL (2025-07-27 04:46:54 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 12) Duration: 605.680004ms core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [....X.......................................................] duration: 786.540531ms FAIL (2025-07-27 04:46:45 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== {0 0}, upload=5911]: Begin upload segment... Jul 27 04:07:17.745307658 (6h15m33.526241043s) [seg_pos={0 0}, upload=5911]: Done begin upload segment. Jul 27 04:07:17.745349241 (6h15m33.526282625s) [noise=true , address=104.36.82.52:4098 , piece_id=M2ETF25QQ53RRGFPUKXJRINIWV7AWXSECEDR2AIM645IED72NURQ, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.745439694 (6h15m33.526373073s) [noise=true , address=104.36.82.56:4244 , piece_id=LQCZ5W54NBHWAVBVYK7HYSSJDC73E6FV4CP4YIM4L5H2N3OLOZXA, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.745972271 (6h15m33.526905654s) [noise=true , address=104.36.82.36:4650 , piece_id=BKJULWGPESBINNTSUYP6E4KZRXKPACVU5I7DOSZBO4YMK2V26HEQ, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.745980243 (6h15m33.526913621s) [noise=true , address=104.36.82.38:4722 , piece_id=B2MBQY5CTMEI6WSE7FTLOV5RUJ22SCNG7N2EEJILEATGJ4TIALXA, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.746403818 (6h15m33.527337198s) [noise=true , address=104.36.82.22:4534 , piece_id=CW4AQT3XUGA4A5OC6NUY4NYOOAMLN5FFJJN5PQBYEON7HPD5LCYQ, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.746455493 (6h15m33.527388875s) [noise=true , address=104.36.82.14:4248 , piece_id=LDSGCUM7Q2VX5CDFI4JRGEEG7D5H5Q5Z5WUWKU277VN2JFRX6UGA, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.746872127 (6h15m33.527805505s) [noise=true , address=104.36.82.40:4954 , piece_id=YLOHTJLRAED66OMEQYPZHNNRI77V6WOR4EHTEZOOEHMGZHJ2O3QQ, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.747123162 (6h15m33.52805654s) [noise=true , address=104.36.82.46:4496 , piece_id=GWNQR5WRGRRQBLSGWJFD3O3UNCMNE62JQNPIV5SYKBUIL3BQKA5A, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.747280678 (6h15m33.528214053s) [noise=true , address=104.36.82.47:4524 , piece_id=DONJFTKUQ2XCW6BOU7AVOKJ7XMDMBPGQVUPRANH7X2YS7PJXUKIA, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.747736400 (6h15m33.528669779s) [noise=true , address=104.36.82.44:4893 , piece_id=HUJICRJ6W54O3AZZR6H3KG6E4TGBTNS46O6XVMQM3PFAZE7A2HZA, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.747755734 (6h15m33.528689109s) [noise=true , address=104.36.82.55:4204 , piece_id=72WISGRHMOAZDV5EDGTBOMLBL4HDOF3CBEMQAAIRGILPC7J5JKAQ, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.748313124 (6h15m33.5292465s) [noise=true , address=104.36.82.58:4287 , piece_id=BVDYUEWU7TVEQDMZXX5SJNS4QICVKMTGFLCUV6P2YHOE7TI25VXA, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.748485217 (6h15m33.529418594s) [noise=true , address=104.36.82.53:4123 , piece_id=NXJRZ5BQSLXSOIOSUJ4MM4AGKHP4UIDUGEJKLUUX7YOGQJA6WWZA, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.748831329 (6h15m33.529764704s) [noise=true , address=104.36.82.42:4829 , piece_id=ZLWJFP3LGX3YLXBJYIR66DXTYEVDBKXKVXCQSF2PCYCYJBIEJGMA, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.749299293 (6h15m33.530232669s) [noise=true , address=104.36.82.8:4002 , piece_id=6XSPWQC3US355PS6IQ4K3IYNQZH5MAT4IVIPGSYRUM3XYPSCWE6Q, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.749790974 (6h15m33.53072436s) [noise=true , address=104.36.82.44:4901 , piece_id=N4YLCLR2QZSULUL2XWONEKBFUMNOU7KUX2QEOK4SSDFSCI7IL6LA, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.750478221 (6h15m33.531411597s) [noise=true , address=104.36.82.54:4172 , piece_id=ZNA53MWDRWPUH6FUR4DJERBDT2IOYGA3IYUADKXXIOLQJLQOXE7Q, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.752884255 (6h15m33.533817625s) [noise=true , address=64.49.13.140:4508 , piece_id=MZ52U75VOIK6Q7QFXMMYRRG7JXFWDFU2JWQFYPERUCMH3BUX6XPQ, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.879839555 (6h15m33.660772917s) [upload=5911]: Next returned nil segment Jul 27 04:07:17.879861389 (6h15m33.660794765s) [upload=5911]: Waiting for error group managing segments... Jul 27 04:07:17.888652438 (6h15m33.669585811s) [noise=true , address=104.36.82.58:4287 , piece_id=BVDYUEWU7TVEQDMZXX5SJNS4QICVKMTGFLCUV6P2YHOE7TI25VXA, seg_pos={0 0}, upload=5911]: Done uploading piece. err: <nil> Jul 27 04:07:17.952192045 (6h15m33.733125429s) [noise=true , address=104.36.82.53:4123 , piece_id=NXJRZ5BQSLXSOIOSUJ4MM4AGKHP4UIDUGEJKLUUX7YOGQJA6WWZA, seg_pos={0 0}, upload=5911]: Done uploading piece. err: <nil> Jul 27 04:07:17.981907539 (6h15m33.762840911s) [noise=true , address=104.36.82.56:4244 , piece_id=LQCZ5W54NBHWAVBVYK7HYSSJDC73E6FV4CP4YIM4L5H2N3OLOZXA, seg_pos={0 0}, upload=5911]: Done uploading piece. err: <nil> Jul 27 04:07:17.982370496 (6h15m33.7633039s) [noise=true , address=104.36.82.55:4204 , piece_id=72WISGRHMOAZDV5EDGTBOMLBL4HDOF3CBEMQAAIRGILPC7J5JKAQ, seg_pos={0 0}, upload=5911]: Done uploading piece. err: <nil> Jul 27 04:07:18.009548506 (6h15m33.790481886s) [noise=true , address=104.36.82.52:4098 , piece_id=M2ETF25QQ53RRGFPUKXJRINIWV7AWXSECEDR2AIM645IED72NURQ, seg_pos={0 0}, upload=5911]: Done uploading piece. err: <nil> Jul 27 04:07:18.023770598 (6h15m33.804703983s) [noise=true , address=104.36.82.40:4954 , piece_id=YLOHTJLRAED66OMEQYPZHNNRI77V6WOR4EHTEZOOEHMGZHJ2O3QQ, seg_pos={0 0}, upload=5911]: Done uploading piece. err: <nil> Jul 27 04:07:18.139745581 (6h15m33.92067895s) [noise=true , address=104.36.82.22:4534 , piece_id=CW4AQT3XUGA4A5OC6NUY4NYOOAMLN5FFJJN5PQBYEON7HPD5LCYQ, seg_pos={0 0}, upload=5911]: Done uploading piece. err: <nil> Jul 27 04:07:18.423748140 (6h15m34.204681511s) [noise=true , address=64.49.13.140:4508 , piece_id=MZ52U75VOIK6Q7QFXMMYRRG7JXFWDFU2JWQFYPERUCMH3BUX6XPQ, seg_pos={0 0}, upload=5911]: Done uploading piece. err: <nil> Jul 27 04:08:16.739940381 (6h16m32.52087375s) [noise=true , address=104.36.82.54:4172 , piece_id=ZNA53MWDRWPUH6FUR4DJERBDT2IOYGA3IYUADKXXIOLQJLQOXE7Q, seg_pos={0 0}, upload=5911]: Done uploading piece. err: <nil> Jul 27 04:08:17.177295397 (6h16m32.958228766s) [noise=true , address=104.36.82.8:4002 , piece_id=6XSPWQC3US355PS6IQ4K3IYNQZH5MAT4IVIPGSYRUM3XYPSCWE6Q, seg_pos={0 0}, upload=5911]: Done uploading piece. err: <nil> Jul 27 04:08:17.534296584 (6h16m33.315229953s) [noise=true , address=104.36.82.36:4650 , piece_id=BKJULWGPESBINNTSUYP6E4KZRXKPACVU5I7DOSZBO4YMK2V26HEQ, seg_pos={0 0}, upload=5911]: Done uploading piece. err: <nil> Jul 27 04:08:17.677851566 (6h16m33.458784943s) [noise=true , address=104.36.82.14:4248 , piece_id=LDSGCUM7Q2VX5CDFI4JRGEEG7D5H5Q5Z5WUWKU277VN2JFRX6UGA, seg_pos={0 0}, upload=5911]: Done uploading piece. err: ecclient: upload failed (node:1u6ir2PLqDa9xGm64cuqq1RqY4qeBa1fqCo9VghspBf54XEXqn, address:104.36.82.14:4248): protocol: rpc: dial tcp 104.36.82.14:4248: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 04:08:17.678115170 (6h16m33.459048549s) [noise=true , address=104.36.82.46:4496 , piece_id=GWNQR5WRGRRQBLSGWJFD3O3UNCMNE62JQNPIV5SYKBUIL3BQKA5A, seg_pos={0 0}, upload=5911]: Done uploading piece. err: ecclient: upload failed (node:12V8XhSWkyGUVYFR8MykifFAP1YAfGCwzqucDcMMfzw3xsVW77Z, address:104.36.82.46:4496): protocol: read tcp 159.223.143.39:52800->104.36.82.46:4496: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 04:08:17.678144409 (6h16m33.459077789s) [noise=true , address=104.36.82.44:4901 , piece_id=N4YLCLR2QZSULUL2XWONEKBFUMNOU7KUX2QEOK4SSDFSCI7IL6LA, seg_pos={0 0}, upload=5911]: Done uploading piece. err: ecclient: upload failed (node:1FHE84zvBb1MKwB1Ht7puTafgQxDEvPt4mYqkYVSSufAMpWQA1, address:104.36.82.44:4901): protocol: rpc: dial tcp 104.36.82.44:4901: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 04:08:17.678229622 (6h16m33.459162988s) [noise=true , address=104.36.82.44:4893 , piece_id=HUJICRJ6W54O3AZZR6H3KG6E4TGBTNS46O6XVMQM3PFAZE7A2HZA, seg_pos={0 0}, upload=5911]: Done uploading piece. err: ecclient: upload failed (node:17VMFZ2T8jrSJVtchciJ9SnyrR6JstvcMoXrN5Q3zTD1aQU1oJ, address:104.36.82.44:4893): protocol: write tcp 159.223.143.39:35302->104.36.82.44:4893: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 04:08:17.678310218 (6h16m33.459243583s) [noise=true , address=104.36.82.42:4829 , piece_id=ZLWJFP3LGX3YLXBJYIR66DXTYEVDBKXKVXCQSF2PCYCYJBIEJGMA, seg_pos={0 0}, upload=5911]: Done uploading piece. err: ecclient: upload failed (node:1F5T8Vt1x4nXFaSenvGBCYjxhRL1H1sxRvzLjW3CqzwDVhq39F, address:104.36.82.42:4829): protocol: rpc: dial tcp 104.36.82.42:4829: i/o timeout; conn closed: op; piecestore: piecestore close: context deadline exceeded Jul 27 04:08:17.678412989 (6h16m33.459346365s) [noise=true , address=104.36.82.38:4722 , piece_id=B2MBQY5CTMEI6WSE7FTLOV5RUJ22SCNG7N2EEJILEATGJ4TIALXA, seg_pos={0 0}, upload=5911]: Done uploading piece. err: ecclient: upload failed (node:12Zw3dtUg9gARz52KhwDLTjcqC3CNm7dpPzpdx7P7PbL1jwe8uc, address:104.36.82.38:4722): protocol: read tcp 159.223.143.39:56048->104.36.82.38:4722: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 04:08:17.678455247 (6h16m33.459388624s) [noise=true , address=104.36.82.47:4524 , piece_id=DONJFTKUQ2XCW6BOU7AVOKJ7XMDMBPGQVUPRANH7X2YS7PJXUKIA, seg_pos={0 0}, upload=5911]: Done uploading piece. err: ecclient: upload failed (node:1xfbCzjSwFDRrR1gPggeN8tNCLbRRWDVzCJ6mqhZsbVSe8d4s4, address:104.36.82.47:4524): protocol: read tcp 159.223.143.39:46694->104.36.82.47:4524: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 04:08:17.678510668 (6h16m33.459444052s) [seg_pos={0 0}, upload=5911]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Jul 27 04:08:17.706442864 (6h16m33.487376232s) [upload=5911]: Done uploading segments... Jul 27 04:08:17.706495329 (6h16m33.487428705s) [upload=5911]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 786.574734ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....X.......................................................] duration: 439.753965ms FAIL (2025-07-27 04:46:36 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== ece_id=NYOFDEZ7CV4PHGQ6JZMEEVAYGJRPYG6IBW7MYJO42HMWOJM7PHZA, seg_pos={0 0}, upload=5879]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12crcXpNe9NgeXTsUpooLSBgrSUFbRxbSbanLa39w2Hr5mGFww): protocol: read tcp 159.223.143.39:53630->104.36.82.5:4840: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 04:05:32.508904485 (6h13m48.289837866s) [seg_pos={0 0}, upload=5879]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 27 04:05:32.509028070 (6h13m48.289961457s) [upload=5879]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 27 04:05:32.509034645 (6h13m48.289968129s) [upload=5879]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 27 04:05:32.581834839 (6h13m48.362768225s) [upload=5879]: Done uploading segments... Jul 27 04:05:32.581875919 (6h13m48.362809298s) [upload=5879]: Upload finished. err: <nil> Jul 27 04:06:23.829155331 (6h14m39.610088696s) [upload=5895]: Starting upload Jul 27 04:06:23.829225021 (6h14m39.610158398s) [upload=5895]: Done starting upload Jul 27 04:06:23.829294606 (6h14m39.61022799s) [upload=5895]: Uploading segments... Jul 27 04:06:23.829577656 (6h14m39.610511053s) [upload=5895]: Got next segment. Inline: false Jul 27 04:06:23.829649664 (6h14m39.610583027s) [upload=5895]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 27 04:06:23.829657065 (6h14m39.61059043s) [upload=5895]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 27 04:06:23.861405530 (6h14m39.642338913s) [seg_pos={0 0}, upload=5895]: Begin upload segment... Jul 27 04:06:23.861476351 (6h14m39.642409713s) [seg_pos={0 0}, upload=5895]: Done begin upload segment. Jul 27 04:06:23.861509120 (6h14m39.642442506s) [noise=true , address=77.90.31.19:4843 , piece_id=YML5RDQIK432Z4FSE4DKXIV5U47LQBEZRFMAVLANUSGFUQR23UWQ, seg_pos={0 0}, upload=5895]: Uploading piece... Jul 27 04:06:23.861953723 (6h14m39.6428871s) [noise=true , address=45.91.116.10:4081 , piece_id=XGDTSEZOUNRTEVWHTO3DHTZMH2VZZOPJPEKLCFAHTXESLLYABHSA, seg_pos={0 0}, upload=5895]: Uploading piece... Jul 27 04:06:23.862060983 (6h14m39.642994358s) [noise=true , address=45.91.116.9:4262 , piece_id=M5ETD45HH2XZD6HDLTJ3JO2BOOIROXFWKVPTN42PSQEWG4F7KAAQ, seg_pos={0 0}, upload=5895]: Uploading piece... Jul 27 04:06:23.862759464 (6h14m39.64369284s) [noise=true , address=104.36.82.4:4804 , piece_id=IU6FDNVB5AUPA7MJ6RSXUTLBETPHOFGNJ7YGWBFIZSQFSO6W6BBA, seg_pos={0 0}, upload=5895]: Uploading piece... Jul 27 04:06:23.862835086 (6h14m39.643768461s) [noise=true , address=104.36.82.6:4873 , piece_id=YP4VJORZZUL5EVCH3P7LD3LNVMHB7CWRTKHGIA7UROXE44N6MMZA, seg_pos={0 0}, upload=5895]: Uploading piece... Jul 27 04:06:23.863593880 (6h14m39.644527258s) [noise=true , address=77.90.31.15:4697 , piece_id=Q76TJDSZPSYZSJQNFVOI462U6DFILMH2RV3LQISODXSOCXGRKW6Q, seg_pos={0 0}, upload=5895]: Uploading piece... Jul 27 04:06:23.863656891 (6h14m39.644590266s) [noise=true , address=45.91.116.7:4011 , piece_id=XMH6MCCHZBBRQWHJFWLRGYHLQD4KUGSGL7MMZSKA5TZJCND3DUDA, seg_pos={0 0}, upload=5895]: Uploading piece... Jul 27 04:06:23.864408612 (6h14m39.645341987s) [noise=true , address=45.91.116.6:4973 , piece_id=J3VULPU24CIM64VEVE3UHO62HPN6TJXGZU6BCJIS5JQJKHJK2LIQ, seg_pos={0 0}, upload=5895]: Uploading piece... Jul 27 04:06:23.864521549 (6h14m39.64545491s) [noise=true , address=104.36.82.10:4079 , piece_id=2SH5D6JOMLFR7VSN7ZY5CLMR5RHIRFYQ7Z5I2JIYTJQM43G4O6OQ, seg_pos={0 0}, upload=5895]: Uploading piece... Jul 27 04:06:23.865518491 (6h14m39.646451872s) [noise=true , address=77.90.31.17:4771 , piece_id=U2LF22GKCMKRZN6EWCF26ITDZQHBFZXCGSQXT33MCX6P57IGI35Q, seg_pos={0 0}, upload=5895]: Uploading piece... Jul 27 04:06:23.865654950 (6h14m39.646588325s) [noise=true , address=45.91.116.8:4048 , piece_id=5REWMVVTBVLDQXQ5KAEHLY4VJL5SGF7ONQWDQQ3N2FDU3OAO6AZA, seg_pos={0 0}, upload=5895]: Uploading piece... Jul 27 04:06:23.866190892 (6h14m39.647124271s) [noise=true , address=104.36.82.12:4151 , piece_id=E2TEUTQBD4Y4AUHSCPMCIFEXBSY7S5HJWBKNKWWTUWEW2BLAAE6Q, seg_pos={0 0}, upload=5895]: Uploading piece... Jul 27 04:06:23.866412590 (6h14m39.647345968s) [noise=true , address=104.36.82.11:4115 , piece_id=YXMNVT6ZHGESGQ2BXKZUVR7U2CNNAI2PO7ZTMDG5UIOV7HTDDUTA, seg_pos={0 0}, upload=5895]: Uploading piece... Jul 27 04:06:23.866715972 (6h14m39.647649359s) [noise=true , address=45.91.116.13:4153 , piece_id=H4535KOCG3F4RKA6AKKMTJSOM6Q6PGRC6M7FR3UIS3BORTSP5UQA, seg_pos={0 0}, upload=5895]: Uploading piece... Jul 27 04:06:23.866998965 (6h14m39.64793234s) [noise=true , address=104.36.82.13:4186 , piece_id=TBLDKALGJYBGR5NLCOCKJ4QKEFNIJXCKAFUBD7VR3UMNWEMKMKHQ, seg_pos={0 0}, upload=5895]: Uploading piece... Jul 27 04:06:23.867230593 (6h14m39.648163974s) [noise=true , address=104.36.82.16:4293 , piece_id=AHH6PRFUSGTDGB7H52DRITARA4IF6GFLVSRYS2Y6EPJCZSKUOYWQ, seg_pos={0 0}, upload=5895]: Uploading piece... Jul 27 04:06:23.949422808 (6h14m39.730356187s) [upload=5895]: Next returned nil segment Jul 27 04:06:23.949442576 (6h14m39.73037596s) [upload=5895]: Waiting for error group managing segments... Jul 27 04:06:23.953497426 (6h14m39.734430802s) [noise=true , address=45.91.116.10:4081 , piece_id=XGDTSEZOUNRTEVWHTO3DHTZMH2VZZOPJPEKLCFAHTXESLLYABHSA, seg_pos={0 0}, upload=5895]: Done uploading piece. err: <nil> Jul 27 04:06:23.953531796 (6h14m39.734465159s) [noise=true , address=45.91.116.6:4973 , piece_id=J3VULPU24CIM64VEVE3UHO62HPN6TJXGZU6BCJIS5JQJKHJK2LIQ, seg_pos={0 0}, upload=5895]: Done uploading piece. err: <nil> Jul 27 04:06:23.954068660 (6h14m39.735002029s) [noise=true , address=45.91.116.9:4262 , piece_id=M5ETD45HH2XZD6HDLTJ3JO2BOOIROXFWKVPTN42PSQEWG4F7KAAQ, seg_pos={0 0}, upload=5895]: Done uploading piece. err: <nil> Jul 27 04:06:23.954194339 (6h14m39.735127707s) [noise=true , address=45.91.116.8:4048 , piece_id=5REWMVVTBVLDQXQ5KAEHLY4VJL5SGF7ONQWDQQ3N2FDU3OAO6AZA, seg_pos={0 0}, upload=5895]: Done uploading piece. err: <nil> Jul 27 04:06:23.954601730 (6h14m39.735535112s) [noise=true , address=45.91.116.7:4011 , piece_id=XMH6MCCHZBBRQWHJFWLRGYHLQD4KUGSGL7MMZSKA5TZJCND3DUDA, seg_pos={0 0}, upload=5895]: Done uploading piece. err: <nil> Jul 27 04:06:23.954755872 (6h14m39.735689237s) [noise=true , address=45.91.116.13:4153 , piece_id=H4535KOCG3F4RKA6AKKMTJSOM6Q6PGRC6M7FR3UIS3BORTSP5UQA, seg_pos={0 0}, upload=5895]: Done uploading piece. err: <nil> Jul 27 04:06:23.955611883 (6h14m39.73654526s) [noise=true , address=77.90.31.19:4843 , piece_id=YML5RDQIK432Z4FSE4DKXIV5U47LQBEZRFMAVLANUSGFUQR23UWQ, seg_pos={0 0}, upload=5895]: Done uploading piece. err: <nil> Jul 27 04:06:23.956373391 (6h14m39.737306756s) [noise=true , address=77.90.31.15:4697 , piece_id=Q76TJDSZPSYZSJQNFVOI462U6DFILMH2RV3LQISODXSOCXGRKW6Q, seg_pos={0 0}, upload=5895]: Done uploading piece. err: <nil> Jul 27 04:06:23.958659122 (6h14m39.73959249s) [noise=true , address=77.90.31.17:4771 , piece_id=U2LF22GKCMKRZN6EWCF26ITDZQHBFZXCGSQXT33MCX6P57IGI35Q, seg_pos={0 0}, upload=5895]: Done uploading piece. err: <nil> Jul 27 04:06:35.058944013 (6h14m50.83987739s) [noise=true , address=104.36.82.16:4293 , piece_id=AHH6PRFUSGTDGB7H52DRITARA4IF6GFLVSRYS2Y6EPJCZSKUOYWQ, seg_pos={0 0}, upload=5895]: Done uploading piece. err: <nil> Jul 27 04:06:41.103204002 (6h14m56.884137386s) [noise=true , address=104.36.82.4:4804 , piece_id=IU6FDNVB5AUPA7MJ6RSXUTLBETPHOFGNJ7YGWBFIZSQFSO6W6BBA, seg_pos={0 0}, upload=5895]: Done uploading piece. err: <nil> Jul 27 04:06:43.191702285 (6h14m58.972635651s) [noise=true , address=104.36.82.6:4873 , piece_id=YP4VJORZZUL5EVCH3P7LD3LNVMHB7CWRTKHGIA7UROXE44N6MMZA, seg_pos={0 0}, upload=5895]: Done uploading piece. err: <nil> Jul 27 04:06:53.777973493 (6h15m9.558906861s) [noise=true , address=104.36.82.11:4115 , piece_id=YXMNVT6ZHGESGQ2BXKZUVR7U2CNNAI2PO7ZTMDG5UIOV7HTDDUTA, seg_pos={0 0}, upload=5895]: Done uploading piece. err: ecclient: upload failed (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR, address:104.36.82.11:4115): protocol: write tcp 159.223.143.39:34210->104.36.82.11:4115: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 04:06:53.778019532 (6h15m9.558952909s) [noise=true , address=104.36.82.12:4151 , piece_id=E2TEUTQBD4Y4AUHSCPMCIFEXBSY7S5HJWBKNKWWTUWEW2BLAAE6Q, seg_pos={0 0}, upload=5895]: Done uploading piece. err: ecclient: upload failed (node:1w6pwZMdeQKjt92qx6WbTWU5VPYJto5gS87f6iKvCBZH6LfyeB, address:104.36.82.12:4151): protocol: rpc: dial tcp 104.36.82.12:4151: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 04:06:53.778043653 (6h15m9.558977021s) [noise=true , address=104.36.82.13:4186 , piece_id=TBLDKALGJYBGR5NLCOCKJ4QKEFNIJXCKAFUBD7VR3UMNWEMKMKHQ, seg_pos={0 0}, upload=5895]: Done uploading piece. err: ecclient: upload failed (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht, address:104.36.82.13:4186): protocol: read tcp 159.223.143.39:34340->104.36.82.13:4186: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 04:06:53.778095228 (6h15m9.559028593s) [noise=true , address=104.36.82.10:4079 , piece_id=2SH5D6JOMLFR7VSN7ZY5CLMR5RHIRFYQ7Z5I2JIYTJQM43G4O6OQ, seg_pos={0 0}, upload=5895]: Done uploading piece. err: ecclient: upload failed (node:12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud, address:104.36.82.10:4079): protocol: write tcp 159.223.143.39:53840->104.36.82.10:4079: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 04:06:53.778142167 (6h15m9.559075532s) [seg_pos={0 0}, upload=5895]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 27 04:06:53.837032161 (6h15m9.617965543s) [upload=5895]: Done uploading segments... Jul 27 04:06:53.837055996 (6h15m9.617989366s) [upload=5895]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 439.784062ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.....X......................................................] duration: 1.050796715s FAIL (2025-07-27 04:46:06 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== {0 0}, upload=5911]: Begin upload segment... Jul 27 04:07:17.745307658 (6h15m33.526241043s) [seg_pos={0 0}, upload=5911]: Done begin upload segment. Jul 27 04:07:17.745349241 (6h15m33.526282625s) [noise=true , address=104.36.82.52:4098 , piece_id=M2ETF25QQ53RRGFPUKXJRINIWV7AWXSECEDR2AIM645IED72NURQ, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.745439694 (6h15m33.526373073s) [noise=true , address=104.36.82.56:4244 , piece_id=LQCZ5W54NBHWAVBVYK7HYSSJDC73E6FV4CP4YIM4L5H2N3OLOZXA, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.745972271 (6h15m33.526905654s) [noise=true , address=104.36.82.36:4650 , piece_id=BKJULWGPESBINNTSUYP6E4KZRXKPACVU5I7DOSZBO4YMK2V26HEQ, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.745980243 (6h15m33.526913621s) [noise=true , address=104.36.82.38:4722 , piece_id=B2MBQY5CTMEI6WSE7FTLOV5RUJ22SCNG7N2EEJILEATGJ4TIALXA, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.746403818 (6h15m33.527337198s) [noise=true , address=104.36.82.22:4534 , piece_id=CW4AQT3XUGA4A5OC6NUY4NYOOAMLN5FFJJN5PQBYEON7HPD5LCYQ, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.746455493 (6h15m33.527388875s) [noise=true , address=104.36.82.14:4248 , piece_id=LDSGCUM7Q2VX5CDFI4JRGEEG7D5H5Q5Z5WUWKU277VN2JFRX6UGA, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.746872127 (6h15m33.527805505s) [noise=true , address=104.36.82.40:4954 , piece_id=YLOHTJLRAED66OMEQYPZHNNRI77V6WOR4EHTEZOOEHMGZHJ2O3QQ, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.747123162 (6h15m33.52805654s) [noise=true , address=104.36.82.46:4496 , piece_id=GWNQR5WRGRRQBLSGWJFD3O3UNCMNE62JQNPIV5SYKBUIL3BQKA5A, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.747280678 (6h15m33.528214053s) [noise=true , address=104.36.82.47:4524 , piece_id=DONJFTKUQ2XCW6BOU7AVOKJ7XMDMBPGQVUPRANH7X2YS7PJXUKIA, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.747736400 (6h15m33.528669779s) [noise=true , address=104.36.82.44:4893 , piece_id=HUJICRJ6W54O3AZZR6H3KG6E4TGBTNS46O6XVMQM3PFAZE7A2HZA, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.747755734 (6h15m33.528689109s) [noise=true , address=104.36.82.55:4204 , piece_id=72WISGRHMOAZDV5EDGTBOMLBL4HDOF3CBEMQAAIRGILPC7J5JKAQ, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.748313124 (6h15m33.5292465s) [noise=true , address=104.36.82.58:4287 , piece_id=BVDYUEWU7TVEQDMZXX5SJNS4QICVKMTGFLCUV6P2YHOE7TI25VXA, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.748485217 (6h15m33.529418594s) [noise=true , address=104.36.82.53:4123 , piece_id=NXJRZ5BQSLXSOIOSUJ4MM4AGKHP4UIDUGEJKLUUX7YOGQJA6WWZA, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.748831329 (6h15m33.529764704s) [noise=true , address=104.36.82.42:4829 , piece_id=ZLWJFP3LGX3YLXBJYIR66DXTYEVDBKXKVXCQSF2PCYCYJBIEJGMA, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.749299293 (6h15m33.530232669s) [noise=true , address=104.36.82.8:4002 , piece_id=6XSPWQC3US355PS6IQ4K3IYNQZH5MAT4IVIPGSYRUM3XYPSCWE6Q, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.749790974 (6h15m33.53072436s) [noise=true , address=104.36.82.44:4901 , piece_id=N4YLCLR2QZSULUL2XWONEKBFUMNOU7KUX2QEOK4SSDFSCI7IL6LA, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.750478221 (6h15m33.531411597s) [noise=true , address=104.36.82.54:4172 , piece_id=ZNA53MWDRWPUH6FUR4DJERBDT2IOYGA3IYUADKXXIOLQJLQOXE7Q, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.752884255 (6h15m33.533817625s) [noise=true , address=64.49.13.140:4508 , piece_id=MZ52U75VOIK6Q7QFXMMYRRG7JXFWDFU2JWQFYPERUCMH3BUX6XPQ, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.879839555 (6h15m33.660772917s) [upload=5911]: Next returned nil segment Jul 27 04:07:17.879861389 (6h15m33.660794765s) [upload=5911]: Waiting for error group managing segments... Jul 27 04:07:17.888652438 (6h15m33.669585811s) [noise=true , address=104.36.82.58:4287 , piece_id=BVDYUEWU7TVEQDMZXX5SJNS4QICVKMTGFLCUV6P2YHOE7TI25VXA, seg_pos={0 0}, upload=5911]: Done uploading piece. err: <nil> Jul 27 04:07:17.952192045 (6h15m33.733125429s) [noise=true , address=104.36.82.53:4123 , piece_id=NXJRZ5BQSLXSOIOSUJ4MM4AGKHP4UIDUGEJKLUUX7YOGQJA6WWZA, seg_pos={0 0}, upload=5911]: Done uploading piece. err: <nil> Jul 27 04:07:17.981907539 (6h15m33.762840911s) [noise=true , address=104.36.82.56:4244 , piece_id=LQCZ5W54NBHWAVBVYK7HYSSJDC73E6FV4CP4YIM4L5H2N3OLOZXA, seg_pos={0 0}, upload=5911]: Done uploading piece. err: <nil> Jul 27 04:07:17.982370496 (6h15m33.7633039s) [noise=true , address=104.36.82.55:4204 , piece_id=72WISGRHMOAZDV5EDGTBOMLBL4HDOF3CBEMQAAIRGILPC7J5JKAQ, seg_pos={0 0}, upload=5911]: Done uploading piece. err: <nil> Jul 27 04:07:18.009548506 (6h15m33.790481886s) [noise=true , address=104.36.82.52:4098 , piece_id=M2ETF25QQ53RRGFPUKXJRINIWV7AWXSECEDR2AIM645IED72NURQ, seg_pos={0 0}, upload=5911]: Done uploading piece. err: <nil> Jul 27 04:07:18.023770598 (6h15m33.804703983s) [noise=true , address=104.36.82.40:4954 , piece_id=YLOHTJLRAED66OMEQYPZHNNRI77V6WOR4EHTEZOOEHMGZHJ2O3QQ, seg_pos={0 0}, upload=5911]: Done uploading piece. err: <nil> Jul 27 04:07:18.139745581 (6h15m33.92067895s) [noise=true , address=104.36.82.22:4534 , piece_id=CW4AQT3XUGA4A5OC6NUY4NYOOAMLN5FFJJN5PQBYEON7HPD5LCYQ, seg_pos={0 0}, upload=5911]: Done uploading piece. err: <nil> Jul 27 04:07:18.423748140 (6h15m34.204681511s) [noise=true , address=64.49.13.140:4508 , piece_id=MZ52U75VOIK6Q7QFXMMYRRG7JXFWDFU2JWQFYPERUCMH3BUX6XPQ, seg_pos={0 0}, upload=5911]: Done uploading piece. err: <nil> Jul 27 04:08:16.739940381 (6h16m32.52087375s) [noise=true , address=104.36.82.54:4172 , piece_id=ZNA53MWDRWPUH6FUR4DJERBDT2IOYGA3IYUADKXXIOLQJLQOXE7Q, seg_pos={0 0}, upload=5911]: Done uploading piece. err: <nil> Jul 27 04:08:17.177295397 (6h16m32.958228766s) [noise=true , address=104.36.82.8:4002 , piece_id=6XSPWQC3US355PS6IQ4K3IYNQZH5MAT4IVIPGSYRUM3XYPSCWE6Q, seg_pos={0 0}, upload=5911]: Done uploading piece. err: <nil> Jul 27 04:08:17.534296584 (6h16m33.315229953s) [noise=true , address=104.36.82.36:4650 , piece_id=BKJULWGPESBINNTSUYP6E4KZRXKPACVU5I7DOSZBO4YMK2V26HEQ, seg_pos={0 0}, upload=5911]: Done uploading piece. err: <nil> Jul 27 04:08:17.677851566 (6h16m33.458784943s) [noise=true , address=104.36.82.14:4248 , piece_id=LDSGCUM7Q2VX5CDFI4JRGEEG7D5H5Q5Z5WUWKU277VN2JFRX6UGA, seg_pos={0 0}, upload=5911]: Done uploading piece. err: ecclient: upload failed (node:1u6ir2PLqDa9xGm64cuqq1RqY4qeBa1fqCo9VghspBf54XEXqn, address:104.36.82.14:4248): protocol: rpc: dial tcp 104.36.82.14:4248: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 04:08:17.678115170 (6h16m33.459048549s) [noise=true , address=104.36.82.46:4496 , piece_id=GWNQR5WRGRRQBLSGWJFD3O3UNCMNE62JQNPIV5SYKBUIL3BQKA5A, seg_pos={0 0}, upload=5911]: Done uploading piece. err: ecclient: upload failed (node:12V8XhSWkyGUVYFR8MykifFAP1YAfGCwzqucDcMMfzw3xsVW77Z, address:104.36.82.46:4496): protocol: read tcp 159.223.143.39:52800->104.36.82.46:4496: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 04:08:17.678144409 (6h16m33.459077789s) [noise=true , address=104.36.82.44:4901 , piece_id=N4YLCLR2QZSULUL2XWONEKBFUMNOU7KUX2QEOK4SSDFSCI7IL6LA, seg_pos={0 0}, upload=5911]: Done uploading piece. err: ecclient: upload failed (node:1FHE84zvBb1MKwB1Ht7puTafgQxDEvPt4mYqkYVSSufAMpWQA1, address:104.36.82.44:4901): protocol: rpc: dial tcp 104.36.82.44:4901: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 04:08:17.678229622 (6h16m33.459162988s) [noise=true , address=104.36.82.44:4893 , piece_id=HUJICRJ6W54O3AZZR6H3KG6E4TGBTNS46O6XVMQM3PFAZE7A2HZA, seg_pos={0 0}, upload=5911]: Done uploading piece. err: ecclient: upload failed (node:17VMFZ2T8jrSJVtchciJ9SnyrR6JstvcMoXrN5Q3zTD1aQU1oJ, address:104.36.82.44:4893): protocol: write tcp 159.223.143.39:35302->104.36.82.44:4893: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 04:08:17.678310218 (6h16m33.459243583s) [noise=true , address=104.36.82.42:4829 , piece_id=ZLWJFP3LGX3YLXBJYIR66DXTYEVDBKXKVXCQSF2PCYCYJBIEJGMA, seg_pos={0 0}, upload=5911]: Done uploading piece. err: ecclient: upload failed (node:1F5T8Vt1x4nXFaSenvGBCYjxhRL1H1sxRvzLjW3CqzwDVhq39F, address:104.36.82.42:4829): protocol: rpc: dial tcp 104.36.82.42:4829: i/o timeout; conn closed: op; piecestore: piecestore close: context deadline exceeded Jul 27 04:08:17.678412989 (6h16m33.459346365s) [noise=true , address=104.36.82.38:4722 , piece_id=B2MBQY5CTMEI6WSE7FTLOV5RUJ22SCNG7N2EEJILEATGJ4TIALXA, seg_pos={0 0}, upload=5911]: Done uploading piece. err: ecclient: upload failed (node:12Zw3dtUg9gARz52KhwDLTjcqC3CNm7dpPzpdx7P7PbL1jwe8uc, address:104.36.82.38:4722): protocol: read tcp 159.223.143.39:56048->104.36.82.38:4722: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 04:08:17.678455247 (6h16m33.459388624s) [noise=true , address=104.36.82.47:4524 , piece_id=DONJFTKUQ2XCW6BOU7AVOKJ7XMDMBPGQVUPRANH7X2YS7PJXUKIA, seg_pos={0 0}, upload=5911]: Done uploading piece. err: ecclient: upload failed (node:1xfbCzjSwFDRrR1gPggeN8tNCLbRRWDVzCJ6mqhZsbVSe8d4s4, address:104.36.82.47:4524): protocol: read tcp 159.223.143.39:46694->104.36.82.47:4524: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 04:08:17.678510668 (6h16m33.459444052s) [seg_pos={0 0}, upload=5911]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Jul 27 04:08:17.706442864 (6h16m33.487376232s) [upload=5911]: Done uploading segments... Jul 27 04:08:17.706495329 (6h16m33.487428705s) [upload=5911]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 1.05082754s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................X.........................................] duration: 594.763447ms FAIL (2025-07-27 04:45:59 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 12) Duration: 594.819528ms core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....X......................................................] duration: 357.219454ms FAIL (2025-07-27 04:45:46 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== ece_id=NYOFDEZ7CV4PHGQ6JZMEEVAYGJRPYG6IBW7MYJO42HMWOJM7PHZA, seg_pos={0 0}, upload=5879]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12crcXpNe9NgeXTsUpooLSBgrSUFbRxbSbanLa39w2Hr5mGFww): protocol: read tcp 159.223.143.39:53630->104.36.82.5:4840: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 04:05:32.508904485 (6h13m48.289837866s) [seg_pos={0 0}, upload=5879]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 27 04:05:32.509028070 (6h13m48.289961457s) [upload=5879]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 27 04:05:32.509034645 (6h13m48.289968129s) [upload=5879]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 27 04:05:32.581834839 (6h13m48.362768225s) [upload=5879]: Done uploading segments... Jul 27 04:05:32.581875919 (6h13m48.362809298s) [upload=5879]: Upload finished. err: <nil> Jul 27 04:06:23.829155331 (6h14m39.610088696s) [upload=5895]: Starting upload Jul 27 04:06:23.829225021 (6h14m39.610158398s) [upload=5895]: Done starting upload Jul 27 04:06:23.829294606 (6h14m39.61022799s) [upload=5895]: Uploading segments... Jul 27 04:06:23.829577656 (6h14m39.610511053s) [upload=5895]: Got next segment. Inline: false Jul 27 04:06:23.829649664 (6h14m39.610583027s) [upload=5895]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 27 04:06:23.829657065 (6h14m39.61059043s) [upload=5895]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 27 04:06:23.861405530 (6h14m39.642338913s) [seg_pos={0 0}, upload=5895]: Begin upload segment... Jul 27 04:06:23.861476351 (6h14m39.642409713s) [seg_pos={0 0}, upload=5895]: Done begin upload segment. Jul 27 04:06:23.861509120 (6h14m39.642442506s) [noise=true , address=77.90.31.19:4843 , piece_id=YML5RDQIK432Z4FSE4DKXIV5U47LQBEZRFMAVLANUSGFUQR23UWQ, seg_pos={0 0}, upload=5895]: Uploading piece... Jul 27 04:06:23.861953723 (6h14m39.6428871s) [noise=true , address=45.91.116.10:4081 , piece_id=XGDTSEZOUNRTEVWHTO3DHTZMH2VZZOPJPEKLCFAHTXESLLYABHSA, seg_pos={0 0}, upload=5895]: Uploading piece... Jul 27 04:06:23.862060983 (6h14m39.642994358s) [noise=true , address=45.91.116.9:4262 , piece_id=M5ETD45HH2XZD6HDLTJ3JO2BOOIROXFWKVPTN42PSQEWG4F7KAAQ, seg_pos={0 0}, upload=5895]: Uploading piece... Jul 27 04:06:23.862759464 (6h14m39.64369284s) [noise=true , address=104.36.82.4:4804 , piece_id=IU6FDNVB5AUPA7MJ6RSXUTLBETPHOFGNJ7YGWBFIZSQFSO6W6BBA, seg_pos={0 0}, upload=5895]: Uploading piece... Jul 27 04:06:23.862835086 (6h14m39.643768461s) [noise=true , address=104.36.82.6:4873 , piece_id=YP4VJORZZUL5EVCH3P7LD3LNVMHB7CWRTKHGIA7UROXE44N6MMZA, seg_pos={0 0}, upload=5895]: Uploading piece... Jul 27 04:06:23.863593880 (6h14m39.644527258s) [noise=true , address=77.90.31.15:4697 , piece_id=Q76TJDSZPSYZSJQNFVOI462U6DFILMH2RV3LQISODXSOCXGRKW6Q, seg_pos={0 0}, upload=5895]: Uploading piece... Jul 27 04:06:23.863656891 (6h14m39.644590266s) [noise=true , address=45.91.116.7:4011 , piece_id=XMH6MCCHZBBRQWHJFWLRGYHLQD4KUGSGL7MMZSKA5TZJCND3DUDA, seg_pos={0 0}, upload=5895]: Uploading piece... Jul 27 04:06:23.864408612 (6h14m39.645341987s) [noise=true , address=45.91.116.6:4973 , piece_id=J3VULPU24CIM64VEVE3UHO62HPN6TJXGZU6BCJIS5JQJKHJK2LIQ, seg_pos={0 0}, upload=5895]: Uploading piece... Jul 27 04:06:23.864521549 (6h14m39.64545491s) [noise=true , address=104.36.82.10:4079 , piece_id=2SH5D6JOMLFR7VSN7ZY5CLMR5RHIRFYQ7Z5I2JIYTJQM43G4O6OQ, seg_pos={0 0}, upload=5895]: Uploading piece... Jul 27 04:06:23.865518491 (6h14m39.646451872s) [noise=true , address=77.90.31.17:4771 , piece_id=U2LF22GKCMKRZN6EWCF26ITDZQHBFZXCGSQXT33MCX6P57IGI35Q, seg_pos={0 0}, upload=5895]: Uploading piece... Jul 27 04:06:23.865654950 (6h14m39.646588325s) [noise=true , address=45.91.116.8:4048 , piece_id=5REWMVVTBVLDQXQ5KAEHLY4VJL5SGF7ONQWDQQ3N2FDU3OAO6AZA, seg_pos={0 0}, upload=5895]: Uploading piece... Jul 27 04:06:23.866190892 (6h14m39.647124271s) [noise=true , address=104.36.82.12:4151 , piece_id=E2TEUTQBD4Y4AUHSCPMCIFEXBSY7S5HJWBKNKWWTUWEW2BLAAE6Q, seg_pos={0 0}, upload=5895]: Uploading piece... Jul 27 04:06:23.866412590 (6h14m39.647345968s) [noise=true , address=104.36.82.11:4115 , piece_id=YXMNVT6ZHGESGQ2BXKZUVR7U2CNNAI2PO7ZTMDG5UIOV7HTDDUTA, seg_pos={0 0}, upload=5895]: Uploading piece... Jul 27 04:06:23.866715972 (6h14m39.647649359s) [noise=true , address=45.91.116.13:4153 , piece_id=H4535KOCG3F4RKA6AKKMTJSOM6Q6PGRC6M7FR3UIS3BORTSP5UQA, seg_pos={0 0}, upload=5895]: Uploading piece... Jul 27 04:06:23.866998965 (6h14m39.64793234s) [noise=true , address=104.36.82.13:4186 , piece_id=TBLDKALGJYBGR5NLCOCKJ4QKEFNIJXCKAFUBD7VR3UMNWEMKMKHQ, seg_pos={0 0}, upload=5895]: Uploading piece... Jul 27 04:06:23.867230593 (6h14m39.648163974s) [noise=true , address=104.36.82.16:4293 , piece_id=AHH6PRFUSGTDGB7H52DRITARA4IF6GFLVSRYS2Y6EPJCZSKUOYWQ, seg_pos={0 0}, upload=5895]: Uploading piece... Jul 27 04:06:23.949422808 (6h14m39.730356187s) [upload=5895]: Next returned nil segment Jul 27 04:06:23.949442576 (6h14m39.73037596s) [upload=5895]: Waiting for error group managing segments... Jul 27 04:06:23.953497426 (6h14m39.734430802s) [noise=true , address=45.91.116.10:4081 , piece_id=XGDTSEZOUNRTEVWHTO3DHTZMH2VZZOPJPEKLCFAHTXESLLYABHSA, seg_pos={0 0}, upload=5895]: Done uploading piece. err: <nil> Jul 27 04:06:23.953531796 (6h14m39.734465159s) [noise=true , address=45.91.116.6:4973 , piece_id=J3VULPU24CIM64VEVE3UHO62HPN6TJXGZU6BCJIS5JQJKHJK2LIQ, seg_pos={0 0}, upload=5895]: Done uploading piece. err: <nil> Jul 27 04:06:23.954068660 (6h14m39.735002029s) [noise=true , address=45.91.116.9:4262 , piece_id=M5ETD45HH2XZD6HDLTJ3JO2BOOIROXFWKVPTN42PSQEWG4F7KAAQ, seg_pos={0 0}, upload=5895]: Done uploading piece. err: <nil> Jul 27 04:06:23.954194339 (6h14m39.735127707s) [noise=true , address=45.91.116.8:4048 , piece_id=5REWMVVTBVLDQXQ5KAEHLY4VJL5SGF7ONQWDQQ3N2FDU3OAO6AZA, seg_pos={0 0}, upload=5895]: Done uploading piece. err: <nil> Jul 27 04:06:23.954601730 (6h14m39.735535112s) [noise=true , address=45.91.116.7:4011 , piece_id=XMH6MCCHZBBRQWHJFWLRGYHLQD4KUGSGL7MMZSKA5TZJCND3DUDA, seg_pos={0 0}, upload=5895]: Done uploading piece. err: <nil> Jul 27 04:06:23.954755872 (6h14m39.735689237s) [noise=true , address=45.91.116.13:4153 , piece_id=H4535KOCG3F4RKA6AKKMTJSOM6Q6PGRC6M7FR3UIS3BORTSP5UQA, seg_pos={0 0}, upload=5895]: Done uploading piece. err: <nil> Jul 27 04:06:23.955611883 (6h14m39.73654526s) [noise=true , address=77.90.31.19:4843 , piece_id=YML5RDQIK432Z4FSE4DKXIV5U47LQBEZRFMAVLANUSGFUQR23UWQ, seg_pos={0 0}, upload=5895]: Done uploading piece. err: <nil> Jul 27 04:06:23.956373391 (6h14m39.737306756s) [noise=true , address=77.90.31.15:4697 , piece_id=Q76TJDSZPSYZSJQNFVOI462U6DFILMH2RV3LQISODXSOCXGRKW6Q, seg_pos={0 0}, upload=5895]: Done uploading piece. err: <nil> Jul 27 04:06:23.958659122 (6h14m39.73959249s) [noise=true , address=77.90.31.17:4771 , piece_id=U2LF22GKCMKRZN6EWCF26ITDZQHBFZXCGSQXT33MCX6P57IGI35Q, seg_pos={0 0}, upload=5895]: Done uploading piece. err: <nil> Jul 27 04:06:35.058944013 (6h14m50.83987739s) [noise=true , address=104.36.82.16:4293 , piece_id=AHH6PRFUSGTDGB7H52DRITARA4IF6GFLVSRYS2Y6EPJCZSKUOYWQ, seg_pos={0 0}, upload=5895]: Done uploading piece. err: <nil> Jul 27 04:06:41.103204002 (6h14m56.884137386s) [noise=true , address=104.36.82.4:4804 , piece_id=IU6FDNVB5AUPA7MJ6RSXUTLBETPHOFGNJ7YGWBFIZSQFSO6W6BBA, seg_pos={0 0}, upload=5895]: Done uploading piece. err: <nil> Jul 27 04:06:43.191702285 (6h14m58.972635651s) [noise=true , address=104.36.82.6:4873 , piece_id=YP4VJORZZUL5EVCH3P7LD3LNVMHB7CWRTKHGIA7UROXE44N6MMZA, seg_pos={0 0}, upload=5895]: Done uploading piece. err: <nil> Jul 27 04:06:53.777973493 (6h15m9.558906861s) [noise=true , address=104.36.82.11:4115 , piece_id=YXMNVT6ZHGESGQ2BXKZUVR7U2CNNAI2PO7ZTMDG5UIOV7HTDDUTA, seg_pos={0 0}, upload=5895]: Done uploading piece. err: ecclient: upload failed (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR, address:104.36.82.11:4115): protocol: write tcp 159.223.143.39:34210->104.36.82.11:4115: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 04:06:53.778019532 (6h15m9.558952909s) [noise=true , address=104.36.82.12:4151 , piece_id=E2TEUTQBD4Y4AUHSCPMCIFEXBSY7S5HJWBKNKWWTUWEW2BLAAE6Q, seg_pos={0 0}, upload=5895]: Done uploading piece. err: ecclient: upload failed (node:1w6pwZMdeQKjt92qx6WbTWU5VPYJto5gS87f6iKvCBZH6LfyeB, address:104.36.82.12:4151): protocol: rpc: dial tcp 104.36.82.12:4151: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 04:06:53.778043653 (6h15m9.558977021s) [noise=true , address=104.36.82.13:4186 , piece_id=TBLDKALGJYBGR5NLCOCKJ4QKEFNIJXCKAFUBD7VR3UMNWEMKMKHQ, seg_pos={0 0}, upload=5895]: Done uploading piece. err: ecclient: upload failed (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht, address:104.36.82.13:4186): protocol: read tcp 159.223.143.39:34340->104.36.82.13:4186: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 04:06:53.778095228 (6h15m9.559028593s) [noise=true , address=104.36.82.10:4079 , piece_id=2SH5D6JOMLFR7VSN7ZY5CLMR5RHIRFYQ7Z5I2JIYTJQM43G4O6OQ, seg_pos={0 0}, upload=5895]: Done uploading piece. err: ecclient: upload failed (node:12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud, address:104.36.82.10:4079): protocol: write tcp 159.223.143.39:53840->104.36.82.10:4079: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 04:06:53.778142167 (6h15m9.559075532s) [seg_pos={0 0}, upload=5895]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 27 04:06:53.837032161 (6h15m9.617965543s) [upload=5895]: Done uploading segments... Jul 27 04:06:53.837055996 (6h15m9.617989366s) [upload=5895]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 357.249079ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................X........................................] duration: 417.237683ms FAIL (2025-07-27 04:45:41 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 12) Duration: 417.266591ms core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [......X.....................................................] duration: 1.350581036s FAIL (2025-07-27 04:45:34 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== {0 0}, upload=5911]: Begin upload segment... Jul 27 04:07:17.745307658 (6h15m33.526241043s) [seg_pos={0 0}, upload=5911]: Done begin upload segment. Jul 27 04:07:17.745349241 (6h15m33.526282625s) [noise=true , address=104.36.82.52:4098 , piece_id=M2ETF25QQ53RRGFPUKXJRINIWV7AWXSECEDR2AIM645IED72NURQ, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.745439694 (6h15m33.526373073s) [noise=true , address=104.36.82.56:4244 , piece_id=LQCZ5W54NBHWAVBVYK7HYSSJDC73E6FV4CP4YIM4L5H2N3OLOZXA, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.745972271 (6h15m33.526905654s) [noise=true , address=104.36.82.36:4650 , piece_id=BKJULWGPESBINNTSUYP6E4KZRXKPACVU5I7DOSZBO4YMK2V26HEQ, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.745980243 (6h15m33.526913621s) [noise=true , address=104.36.82.38:4722 , piece_id=B2MBQY5CTMEI6WSE7FTLOV5RUJ22SCNG7N2EEJILEATGJ4TIALXA, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.746403818 (6h15m33.527337198s) [noise=true , address=104.36.82.22:4534 , piece_id=CW4AQT3XUGA4A5OC6NUY4NYOOAMLN5FFJJN5PQBYEON7HPD5LCYQ, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.746455493 (6h15m33.527388875s) [noise=true , address=104.36.82.14:4248 , piece_id=LDSGCUM7Q2VX5CDFI4JRGEEG7D5H5Q5Z5WUWKU277VN2JFRX6UGA, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.746872127 (6h15m33.527805505s) [noise=true , address=104.36.82.40:4954 , piece_id=YLOHTJLRAED66OMEQYPZHNNRI77V6WOR4EHTEZOOEHMGZHJ2O3QQ, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.747123162 (6h15m33.52805654s) [noise=true , address=104.36.82.46:4496 , piece_id=GWNQR5WRGRRQBLSGWJFD3O3UNCMNE62JQNPIV5SYKBUIL3BQKA5A, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.747280678 (6h15m33.528214053s) [noise=true , address=104.36.82.47:4524 , piece_id=DONJFTKUQ2XCW6BOU7AVOKJ7XMDMBPGQVUPRANH7X2YS7PJXUKIA, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.747736400 (6h15m33.528669779s) [noise=true , address=104.36.82.44:4893 , piece_id=HUJICRJ6W54O3AZZR6H3KG6E4TGBTNS46O6XVMQM3PFAZE7A2HZA, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.747755734 (6h15m33.528689109s) [noise=true , address=104.36.82.55:4204 , piece_id=72WISGRHMOAZDV5EDGTBOMLBL4HDOF3CBEMQAAIRGILPC7J5JKAQ, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.748313124 (6h15m33.5292465s) [noise=true , address=104.36.82.58:4287 , piece_id=BVDYUEWU7TVEQDMZXX5SJNS4QICVKMTGFLCUV6P2YHOE7TI25VXA, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.748485217 (6h15m33.529418594s) [noise=true , address=104.36.82.53:4123 , piece_id=NXJRZ5BQSLXSOIOSUJ4MM4AGKHP4UIDUGEJKLUUX7YOGQJA6WWZA, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.748831329 (6h15m33.529764704s) [noise=true , address=104.36.82.42:4829 , piece_id=ZLWJFP3LGX3YLXBJYIR66DXTYEVDBKXKVXCQSF2PCYCYJBIEJGMA, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.749299293 (6h15m33.530232669s) [noise=true , address=104.36.82.8:4002 , piece_id=6XSPWQC3US355PS6IQ4K3IYNQZH5MAT4IVIPGSYRUM3XYPSCWE6Q, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.749790974 (6h15m33.53072436s) [noise=true , address=104.36.82.44:4901 , piece_id=N4YLCLR2QZSULUL2XWONEKBFUMNOU7KUX2QEOK4SSDFSCI7IL6LA, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.750478221 (6h15m33.531411597s) [noise=true , address=104.36.82.54:4172 , piece_id=ZNA53MWDRWPUH6FUR4DJERBDT2IOYGA3IYUADKXXIOLQJLQOXE7Q, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.752884255 (6h15m33.533817625s) [noise=true , address=64.49.13.140:4508 , piece_id=MZ52U75VOIK6Q7QFXMMYRRG7JXFWDFU2JWQFYPERUCMH3BUX6XPQ, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.879839555 (6h15m33.660772917s) [upload=5911]: Next returned nil segment Jul 27 04:07:17.879861389 (6h15m33.660794765s) [upload=5911]: Waiting for error group managing segments... Jul 27 04:07:17.888652438 (6h15m33.669585811s) [noise=true , address=104.36.82.58:4287 , piece_id=BVDYUEWU7TVEQDMZXX5SJNS4QICVKMTGFLCUV6P2YHOE7TI25VXA, seg_pos={0 0}, upload=5911]: Done uploading piece. err: <nil> Jul 27 04:07:17.952192045 (6h15m33.733125429s) [noise=true , address=104.36.82.53:4123 , piece_id=NXJRZ5BQSLXSOIOSUJ4MM4AGKHP4UIDUGEJKLUUX7YOGQJA6WWZA, seg_pos={0 0}, upload=5911]: Done uploading piece. err: <nil> Jul 27 04:07:17.981907539 (6h15m33.762840911s) [noise=true , address=104.36.82.56:4244 , piece_id=LQCZ5W54NBHWAVBVYK7HYSSJDC73E6FV4CP4YIM4L5H2N3OLOZXA, seg_pos={0 0}, upload=5911]: Done uploading piece. err: <nil> Jul 27 04:07:17.982370496 (6h15m33.7633039s) [noise=true , address=104.36.82.55:4204 , piece_id=72WISGRHMOAZDV5EDGTBOMLBL4HDOF3CBEMQAAIRGILPC7J5JKAQ, seg_pos={0 0}, upload=5911]: Done uploading piece. err: <nil> Jul 27 04:07:18.009548506 (6h15m33.790481886s) [noise=true , address=104.36.82.52:4098 , piece_id=M2ETF25QQ53RRGFPUKXJRINIWV7AWXSECEDR2AIM645IED72NURQ, seg_pos={0 0}, upload=5911]: Done uploading piece. err: <nil> Jul 27 04:07:18.023770598 (6h15m33.804703983s) [noise=true , address=104.36.82.40:4954 , piece_id=YLOHTJLRAED66OMEQYPZHNNRI77V6WOR4EHTEZOOEHMGZHJ2O3QQ, seg_pos={0 0}, upload=5911]: Done uploading piece. err: <nil> Jul 27 04:07:18.139745581 (6h15m33.92067895s) [noise=true , address=104.36.82.22:4534 , piece_id=CW4AQT3XUGA4A5OC6NUY4NYOOAMLN5FFJJN5PQBYEON7HPD5LCYQ, seg_pos={0 0}, upload=5911]: Done uploading piece. err: <nil> Jul 27 04:07:18.423748140 (6h15m34.204681511s) [noise=true , address=64.49.13.140:4508 , piece_id=MZ52U75VOIK6Q7QFXMMYRRG7JXFWDFU2JWQFYPERUCMH3BUX6XPQ, seg_pos={0 0}, upload=5911]: Done uploading piece. err: <nil> Jul 27 04:08:16.739940381 (6h16m32.52087375s) [noise=true , address=104.36.82.54:4172 , piece_id=ZNA53MWDRWPUH6FUR4DJERBDT2IOYGA3IYUADKXXIOLQJLQOXE7Q, seg_pos={0 0}, upload=5911]: Done uploading piece. err: <nil> Jul 27 04:08:17.177295397 (6h16m32.958228766s) [noise=true , address=104.36.82.8:4002 , piece_id=6XSPWQC3US355PS6IQ4K3IYNQZH5MAT4IVIPGSYRUM3XYPSCWE6Q, seg_pos={0 0}, upload=5911]: Done uploading piece. err: <nil> Jul 27 04:08:17.534296584 (6h16m33.315229953s) [noise=true , address=104.36.82.36:4650 , piece_id=BKJULWGPESBINNTSUYP6E4KZRXKPACVU5I7DOSZBO4YMK2V26HEQ, seg_pos={0 0}, upload=5911]: Done uploading piece. err: <nil> Jul 27 04:08:17.677851566 (6h16m33.458784943s) [noise=true , address=104.36.82.14:4248 , piece_id=LDSGCUM7Q2VX5CDFI4JRGEEG7D5H5Q5Z5WUWKU277VN2JFRX6UGA, seg_pos={0 0}, upload=5911]: Done uploading piece. err: ecclient: upload failed (node:1u6ir2PLqDa9xGm64cuqq1RqY4qeBa1fqCo9VghspBf54XEXqn, address:104.36.82.14:4248): protocol: rpc: dial tcp 104.36.82.14:4248: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 04:08:17.678115170 (6h16m33.459048549s) [noise=true , address=104.36.82.46:4496 , piece_id=GWNQR5WRGRRQBLSGWJFD3O3UNCMNE62JQNPIV5SYKBUIL3BQKA5A, seg_pos={0 0}, upload=5911]: Done uploading piece. err: ecclient: upload failed (node:12V8XhSWkyGUVYFR8MykifFAP1YAfGCwzqucDcMMfzw3xsVW77Z, address:104.36.82.46:4496): protocol: read tcp 159.223.143.39:52800->104.36.82.46:4496: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 04:08:17.678144409 (6h16m33.459077789s) [noise=true , address=104.36.82.44:4901 , piece_id=N4YLCLR2QZSULUL2XWONEKBFUMNOU7KUX2QEOK4SSDFSCI7IL6LA, seg_pos={0 0}, upload=5911]: Done uploading piece. err: ecclient: upload failed (node:1FHE84zvBb1MKwB1Ht7puTafgQxDEvPt4mYqkYVSSufAMpWQA1, address:104.36.82.44:4901): protocol: rpc: dial tcp 104.36.82.44:4901: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 04:08:17.678229622 (6h16m33.459162988s) [noise=true , address=104.36.82.44:4893 , piece_id=HUJICRJ6W54O3AZZR6H3KG6E4TGBTNS46O6XVMQM3PFAZE7A2HZA, seg_pos={0 0}, upload=5911]: Done uploading piece. err: ecclient: upload failed (node:17VMFZ2T8jrSJVtchciJ9SnyrR6JstvcMoXrN5Q3zTD1aQU1oJ, address:104.36.82.44:4893): protocol: write tcp 159.223.143.39:35302->104.36.82.44:4893: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 04:08:17.678310218 (6h16m33.459243583s) [noise=true , address=104.36.82.42:4829 , piece_id=ZLWJFP3LGX3YLXBJYIR66DXTYEVDBKXKVXCQSF2PCYCYJBIEJGMA, seg_pos={0 0}, upload=5911]: Done uploading piece. err: ecclient: upload failed (node:1F5T8Vt1x4nXFaSenvGBCYjxhRL1H1sxRvzLjW3CqzwDVhq39F, address:104.36.82.42:4829): protocol: rpc: dial tcp 104.36.82.42:4829: i/o timeout; conn closed: op; piecestore: piecestore close: context deadline exceeded Jul 27 04:08:17.678412989 (6h16m33.459346365s) [noise=true , address=104.36.82.38:4722 , piece_id=B2MBQY5CTMEI6WSE7FTLOV5RUJ22SCNG7N2EEJILEATGJ4TIALXA, seg_pos={0 0}, upload=5911]: Done uploading piece. err: ecclient: upload failed (node:12Zw3dtUg9gARz52KhwDLTjcqC3CNm7dpPzpdx7P7PbL1jwe8uc, address:104.36.82.38:4722): protocol: read tcp 159.223.143.39:56048->104.36.82.38:4722: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 04:08:17.678455247 (6h16m33.459388624s) [noise=true , address=104.36.82.47:4524 , piece_id=DONJFTKUQ2XCW6BOU7AVOKJ7XMDMBPGQVUPRANH7X2YS7PJXUKIA, seg_pos={0 0}, upload=5911]: Done uploading piece. err: ecclient: upload failed (node:1xfbCzjSwFDRrR1gPggeN8tNCLbRRWDVzCJ6mqhZsbVSe8d4s4, address:104.36.82.47:4524): protocol: read tcp 159.223.143.39:46694->104.36.82.47:4524: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 04:08:17.678510668 (6h16m33.459444052s) [seg_pos={0 0}, upload=5911]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Jul 27 04:08:17.706442864 (6h16m33.487376232s) [upload=5911]: Done uploading segments... Jul 27 04:08:17.706495329 (6h16m33.487428705s) [upload=5911]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 1.350612966s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.......X....................................................] duration: 3.276492438s FAIL (2025-07-27 04:45:04 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== {0 0}, upload=5911]: Begin upload segment... Jul 27 04:07:17.745307658 (6h15m33.526241043s) [seg_pos={0 0}, upload=5911]: Done begin upload segment. Jul 27 04:07:17.745349241 (6h15m33.526282625s) [noise=true , address=104.36.82.52:4098 , piece_id=M2ETF25QQ53RRGFPUKXJRINIWV7AWXSECEDR2AIM645IED72NURQ, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.745439694 (6h15m33.526373073s) [noise=true , address=104.36.82.56:4244 , piece_id=LQCZ5W54NBHWAVBVYK7HYSSJDC73E6FV4CP4YIM4L5H2N3OLOZXA, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.745972271 (6h15m33.526905654s) [noise=true , address=104.36.82.36:4650 , piece_id=BKJULWGPESBINNTSUYP6E4KZRXKPACVU5I7DOSZBO4YMK2V26HEQ, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.745980243 (6h15m33.526913621s) [noise=true , address=104.36.82.38:4722 , piece_id=B2MBQY5CTMEI6WSE7FTLOV5RUJ22SCNG7N2EEJILEATGJ4TIALXA, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.746403818 (6h15m33.527337198s) [noise=true , address=104.36.82.22:4534 , piece_id=CW4AQT3XUGA4A5OC6NUY4NYOOAMLN5FFJJN5PQBYEON7HPD5LCYQ, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.746455493 (6h15m33.527388875s) [noise=true , address=104.36.82.14:4248 , piece_id=LDSGCUM7Q2VX5CDFI4JRGEEG7D5H5Q5Z5WUWKU277VN2JFRX6UGA, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.746872127 (6h15m33.527805505s) [noise=true , address=104.36.82.40:4954 , piece_id=YLOHTJLRAED66OMEQYPZHNNRI77V6WOR4EHTEZOOEHMGZHJ2O3QQ, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.747123162 (6h15m33.52805654s) [noise=true , address=104.36.82.46:4496 , piece_id=GWNQR5WRGRRQBLSGWJFD3O3UNCMNE62JQNPIV5SYKBUIL3BQKA5A, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.747280678 (6h15m33.528214053s) [noise=true , address=104.36.82.47:4524 , piece_id=DONJFTKUQ2XCW6BOU7AVOKJ7XMDMBPGQVUPRANH7X2YS7PJXUKIA, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.747736400 (6h15m33.528669779s) [noise=true , address=104.36.82.44:4893 , piece_id=HUJICRJ6W54O3AZZR6H3KG6E4TGBTNS46O6XVMQM3PFAZE7A2HZA, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.747755734 (6h15m33.528689109s) [noise=true , address=104.36.82.55:4204 , piece_id=72WISGRHMOAZDV5EDGTBOMLBL4HDOF3CBEMQAAIRGILPC7J5JKAQ, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.748313124 (6h15m33.5292465s) [noise=true , address=104.36.82.58:4287 , piece_id=BVDYUEWU7TVEQDMZXX5SJNS4QICVKMTGFLCUV6P2YHOE7TI25VXA, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.748485217 (6h15m33.529418594s) [noise=true , address=104.36.82.53:4123 , piece_id=NXJRZ5BQSLXSOIOSUJ4MM4AGKHP4UIDUGEJKLUUX7YOGQJA6WWZA, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.748831329 (6h15m33.529764704s) [noise=true , address=104.36.82.42:4829 , piece_id=ZLWJFP3LGX3YLXBJYIR66DXTYEVDBKXKVXCQSF2PCYCYJBIEJGMA, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.749299293 (6h15m33.530232669s) [noise=true , address=104.36.82.8:4002 , piece_id=6XSPWQC3US355PS6IQ4K3IYNQZH5MAT4IVIPGSYRUM3XYPSCWE6Q, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.749790974 (6h15m33.53072436s) [noise=true , address=104.36.82.44:4901 , piece_id=N4YLCLR2QZSULUL2XWONEKBFUMNOU7KUX2QEOK4SSDFSCI7IL6LA, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.750478221 (6h15m33.531411597s) [noise=true , address=104.36.82.54:4172 , piece_id=ZNA53MWDRWPUH6FUR4DJERBDT2IOYGA3IYUADKXXIOLQJLQOXE7Q, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.752884255 (6h15m33.533817625s) [noise=true , address=64.49.13.140:4508 , piece_id=MZ52U75VOIK6Q7QFXMMYRRG7JXFWDFU2JWQFYPERUCMH3BUX6XPQ, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.879839555 (6h15m33.660772917s) [upload=5911]: Next returned nil segment Jul 27 04:07:17.879861389 (6h15m33.660794765s) [upload=5911]: Waiting for error group managing segments... Jul 27 04:07:17.888652438 (6h15m33.669585811s) [noise=true , address=104.36.82.58:4287 , piece_id=BVDYUEWU7TVEQDMZXX5SJNS4QICVKMTGFLCUV6P2YHOE7TI25VXA, seg_pos={0 0}, upload=5911]: Done uploading piece. err: <nil> Jul 27 04:07:17.952192045 (6h15m33.733125429s) [noise=true , address=104.36.82.53:4123 , piece_id=NXJRZ5BQSLXSOIOSUJ4MM4AGKHP4UIDUGEJKLUUX7YOGQJA6WWZA, seg_pos={0 0}, upload=5911]: Done uploading piece. err: <nil> Jul 27 04:07:17.981907539 (6h15m33.762840911s) [noise=true , address=104.36.82.56:4244 , piece_id=LQCZ5W54NBHWAVBVYK7HYSSJDC73E6FV4CP4YIM4L5H2N3OLOZXA, seg_pos={0 0}, upload=5911]: Done uploading piece. err: <nil> Jul 27 04:07:17.982370496 (6h15m33.7633039s) [noise=true , address=104.36.82.55:4204 , piece_id=72WISGRHMOAZDV5EDGTBOMLBL4HDOF3CBEMQAAIRGILPC7J5JKAQ, seg_pos={0 0}, upload=5911]: Done uploading piece. err: <nil> Jul 27 04:07:18.009548506 (6h15m33.790481886s) [noise=true , address=104.36.82.52:4098 , piece_id=M2ETF25QQ53RRGFPUKXJRINIWV7AWXSECEDR2AIM645IED72NURQ, seg_pos={0 0}, upload=5911]: Done uploading piece. err: <nil> Jul 27 04:07:18.023770598 (6h15m33.804703983s) [noise=true , address=104.36.82.40:4954 , piece_id=YLOHTJLRAED66OMEQYPZHNNRI77V6WOR4EHTEZOOEHMGZHJ2O3QQ, seg_pos={0 0}, upload=5911]: Done uploading piece. err: <nil> Jul 27 04:07:18.139745581 (6h15m33.92067895s) [noise=true , address=104.36.82.22:4534 , piece_id=CW4AQT3XUGA4A5OC6NUY4NYOOAMLN5FFJJN5PQBYEON7HPD5LCYQ, seg_pos={0 0}, upload=5911]: Done uploading piece. err: <nil> Jul 27 04:07:18.423748140 (6h15m34.204681511s) [noise=true , address=64.49.13.140:4508 , piece_id=MZ52U75VOIK6Q7QFXMMYRRG7JXFWDFU2JWQFYPERUCMH3BUX6XPQ, seg_pos={0 0}, upload=5911]: Done uploading piece. err: <nil> Jul 27 04:08:16.739940381 (6h16m32.52087375s) [noise=true , address=104.36.82.54:4172 , piece_id=ZNA53MWDRWPUH6FUR4DJERBDT2IOYGA3IYUADKXXIOLQJLQOXE7Q, seg_pos={0 0}, upload=5911]: Done uploading piece. err: <nil> Jul 27 04:08:17.177295397 (6h16m32.958228766s) [noise=true , address=104.36.82.8:4002 , piece_id=6XSPWQC3US355PS6IQ4K3IYNQZH5MAT4IVIPGSYRUM3XYPSCWE6Q, seg_pos={0 0}, upload=5911]: Done uploading piece. err: <nil> Jul 27 04:08:17.534296584 (6h16m33.315229953s) [noise=true , address=104.36.82.36:4650 , piece_id=BKJULWGPESBINNTSUYP6E4KZRXKPACVU5I7DOSZBO4YMK2V26HEQ, seg_pos={0 0}, upload=5911]: Done uploading piece. err: <nil> Jul 27 04:08:17.677851566 (6h16m33.458784943s) [noise=true , address=104.36.82.14:4248 , piece_id=LDSGCUM7Q2VX5CDFI4JRGEEG7D5H5Q5Z5WUWKU277VN2JFRX6UGA, seg_pos={0 0}, upload=5911]: Done uploading piece. err: ecclient: upload failed (node:1u6ir2PLqDa9xGm64cuqq1RqY4qeBa1fqCo9VghspBf54XEXqn, address:104.36.82.14:4248): protocol: rpc: dial tcp 104.36.82.14:4248: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 04:08:17.678115170 (6h16m33.459048549s) [noise=true , address=104.36.82.46:4496 , piece_id=GWNQR5WRGRRQBLSGWJFD3O3UNCMNE62JQNPIV5SYKBUIL3BQKA5A, seg_pos={0 0}, upload=5911]: Done uploading piece. err: ecclient: upload failed (node:12V8XhSWkyGUVYFR8MykifFAP1YAfGCwzqucDcMMfzw3xsVW77Z, address:104.36.82.46:4496): protocol: read tcp 159.223.143.39:52800->104.36.82.46:4496: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 04:08:17.678144409 (6h16m33.459077789s) [noise=true , address=104.36.82.44:4901 , piece_id=N4YLCLR2QZSULUL2XWONEKBFUMNOU7KUX2QEOK4SSDFSCI7IL6LA, seg_pos={0 0}, upload=5911]: Done uploading piece. err: ecclient: upload failed (node:1FHE84zvBb1MKwB1Ht7puTafgQxDEvPt4mYqkYVSSufAMpWQA1, address:104.36.82.44:4901): protocol: rpc: dial tcp 104.36.82.44:4901: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 04:08:17.678229622 (6h16m33.459162988s) [noise=true , address=104.36.82.44:4893 , piece_id=HUJICRJ6W54O3AZZR6H3KG6E4TGBTNS46O6XVMQM3PFAZE7A2HZA, seg_pos={0 0}, upload=5911]: Done uploading piece. err: ecclient: upload failed (node:17VMFZ2T8jrSJVtchciJ9SnyrR6JstvcMoXrN5Q3zTD1aQU1oJ, address:104.36.82.44:4893): protocol: write tcp 159.223.143.39:35302->104.36.82.44:4893: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 04:08:17.678310218 (6h16m33.459243583s) [noise=true , address=104.36.82.42:4829 , piece_id=ZLWJFP3LGX3YLXBJYIR66DXTYEVDBKXKVXCQSF2PCYCYJBIEJGMA, seg_pos={0 0}, upload=5911]: Done uploading piece. err: ecclient: upload failed (node:1F5T8Vt1x4nXFaSenvGBCYjxhRL1H1sxRvzLjW3CqzwDVhq39F, address:104.36.82.42:4829): protocol: rpc: dial tcp 104.36.82.42:4829: i/o timeout; conn closed: op; piecestore: piecestore close: context deadline exceeded Jul 27 04:08:17.678412989 (6h16m33.459346365s) [noise=true , address=104.36.82.38:4722 , piece_id=B2MBQY5CTMEI6WSE7FTLOV5RUJ22SCNG7N2EEJILEATGJ4TIALXA, seg_pos={0 0}, upload=5911]: Done uploading piece. err: ecclient: upload failed (node:12Zw3dtUg9gARz52KhwDLTjcqC3CNm7dpPzpdx7P7PbL1jwe8uc, address:104.36.82.38:4722): protocol: read tcp 159.223.143.39:56048->104.36.82.38:4722: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 04:08:17.678455247 (6h16m33.459388624s) [noise=true , address=104.36.82.47:4524 , piece_id=DONJFTKUQ2XCW6BOU7AVOKJ7XMDMBPGQVUPRANH7X2YS7PJXUKIA, seg_pos={0 0}, upload=5911]: Done uploading piece. err: ecclient: upload failed (node:1xfbCzjSwFDRrR1gPggeN8tNCLbRRWDVzCJ6mqhZsbVSe8d4s4, address:104.36.82.47:4524): protocol: read tcp 159.223.143.39:46694->104.36.82.47:4524: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 04:08:17.678510668 (6h16m33.459444052s) [seg_pos={0 0}, upload=5911]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Jul 27 04:08:17.706442864 (6h16m33.487376232s) [upload=5911]: Done uploading segments... Jul 27 04:08:17.706495329 (6h16m33.487428705s) [upload=5911]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 3.276564095s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......X.....................................................] duration: 429.054014ms FAIL (2025-07-27 04:44:55 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== ece_id=NYOFDEZ7CV4PHGQ6JZMEEVAYGJRPYG6IBW7MYJO42HMWOJM7PHZA, seg_pos={0 0}, upload=5879]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12crcXpNe9NgeXTsUpooLSBgrSUFbRxbSbanLa39w2Hr5mGFww): protocol: read tcp 159.223.143.39:53630->104.36.82.5:4840: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 04:05:32.508904485 (6h13m48.289837866s) [seg_pos={0 0}, upload=5879]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 27 04:05:32.509028070 (6h13m48.289961457s) [upload=5879]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 27 04:05:32.509034645 (6h13m48.289968129s) [upload=5879]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 27 04:05:32.581834839 (6h13m48.362768225s) [upload=5879]: Done uploading segments... Jul 27 04:05:32.581875919 (6h13m48.362809298s) [upload=5879]: Upload finished. err: <nil> Jul 27 04:06:23.829155331 (6h14m39.610088696s) [upload=5895]: Starting upload Jul 27 04:06:23.829225021 (6h14m39.610158398s) [upload=5895]: Done starting upload Jul 27 04:06:23.829294606 (6h14m39.61022799s) [upload=5895]: Uploading segments... Jul 27 04:06:23.829577656 (6h14m39.610511053s) [upload=5895]: Got next segment. Inline: false Jul 27 04:06:23.829649664 (6h14m39.610583027s) [upload=5895]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 27 04:06:23.829657065 (6h14m39.61059043s) [upload=5895]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 27 04:06:23.861405530 (6h14m39.642338913s) [seg_pos={0 0}, upload=5895]: Begin upload segment... Jul 27 04:06:23.861476351 (6h14m39.642409713s) [seg_pos={0 0}, upload=5895]: Done begin upload segment. Jul 27 04:06:23.861509120 (6h14m39.642442506s) [noise=true , address=77.90.31.19:4843 , piece_id=YML5RDQIK432Z4FSE4DKXIV5U47LQBEZRFMAVLANUSGFUQR23UWQ, seg_pos={0 0}, upload=5895]: Uploading piece... Jul 27 04:06:23.861953723 (6h14m39.6428871s) [noise=true , address=45.91.116.10:4081 , piece_id=XGDTSEZOUNRTEVWHTO3DHTZMH2VZZOPJPEKLCFAHTXESLLYABHSA, seg_pos={0 0}, upload=5895]: Uploading piece... Jul 27 04:06:23.862060983 (6h14m39.642994358s) [noise=true , address=45.91.116.9:4262 , piece_id=M5ETD45HH2XZD6HDLTJ3JO2BOOIROXFWKVPTN42PSQEWG4F7KAAQ, seg_pos={0 0}, upload=5895]: Uploading piece... Jul 27 04:06:23.862759464 (6h14m39.64369284s) [noise=true , address=104.36.82.4:4804 , piece_id=IU6FDNVB5AUPA7MJ6RSXUTLBETPHOFGNJ7YGWBFIZSQFSO6W6BBA, seg_pos={0 0}, upload=5895]: Uploading piece... Jul 27 04:06:23.862835086 (6h14m39.643768461s) [noise=true , address=104.36.82.6:4873 , piece_id=YP4VJORZZUL5EVCH3P7LD3LNVMHB7CWRTKHGIA7UROXE44N6MMZA, seg_pos={0 0}, upload=5895]: Uploading piece... Jul 27 04:06:23.863593880 (6h14m39.644527258s) [noise=true , address=77.90.31.15:4697 , piece_id=Q76TJDSZPSYZSJQNFVOI462U6DFILMH2RV3LQISODXSOCXGRKW6Q, seg_pos={0 0}, upload=5895]: Uploading piece... Jul 27 04:06:23.863656891 (6h14m39.644590266s) [noise=true , address=45.91.116.7:4011 , piece_id=XMH6MCCHZBBRQWHJFWLRGYHLQD4KUGSGL7MMZSKA5TZJCND3DUDA, seg_pos={0 0}, upload=5895]: Uploading piece... Jul 27 04:06:23.864408612 (6h14m39.645341987s) [noise=true , address=45.91.116.6:4973 , piece_id=J3VULPU24CIM64VEVE3UHO62HPN6TJXGZU6BCJIS5JQJKHJK2LIQ, seg_pos={0 0}, upload=5895]: Uploading piece... Jul 27 04:06:23.864521549 (6h14m39.64545491s) [noise=true , address=104.36.82.10:4079 , piece_id=2SH5D6JOMLFR7VSN7ZY5CLMR5RHIRFYQ7Z5I2JIYTJQM43G4O6OQ, seg_pos={0 0}, upload=5895]: Uploading piece... Jul 27 04:06:23.865518491 (6h14m39.646451872s) [noise=true , address=77.90.31.17:4771 , piece_id=U2LF22GKCMKRZN6EWCF26ITDZQHBFZXCGSQXT33MCX6P57IGI35Q, seg_pos={0 0}, upload=5895]: Uploading piece... Jul 27 04:06:23.865654950 (6h14m39.646588325s) [noise=true , address=45.91.116.8:4048 , piece_id=5REWMVVTBVLDQXQ5KAEHLY4VJL5SGF7ONQWDQQ3N2FDU3OAO6AZA, seg_pos={0 0}, upload=5895]: Uploading piece... Jul 27 04:06:23.866190892 (6h14m39.647124271s) [noise=true , address=104.36.82.12:4151 , piece_id=E2TEUTQBD4Y4AUHSCPMCIFEXBSY7S5HJWBKNKWWTUWEW2BLAAE6Q, seg_pos={0 0}, upload=5895]: Uploading piece... Jul 27 04:06:23.866412590 (6h14m39.647345968s) [noise=true , address=104.36.82.11:4115 , piece_id=YXMNVT6ZHGESGQ2BXKZUVR7U2CNNAI2PO7ZTMDG5UIOV7HTDDUTA, seg_pos={0 0}, upload=5895]: Uploading piece... Jul 27 04:06:23.866715972 (6h14m39.647649359s) [noise=true , address=45.91.116.13:4153 , piece_id=H4535KOCG3F4RKA6AKKMTJSOM6Q6PGRC6M7FR3UIS3BORTSP5UQA, seg_pos={0 0}, upload=5895]: Uploading piece... Jul 27 04:06:23.866998965 (6h14m39.64793234s) [noise=true , address=104.36.82.13:4186 , piece_id=TBLDKALGJYBGR5NLCOCKJ4QKEFNIJXCKAFUBD7VR3UMNWEMKMKHQ, seg_pos={0 0}, upload=5895]: Uploading piece... Jul 27 04:06:23.867230593 (6h14m39.648163974s) [noise=true , address=104.36.82.16:4293 , piece_id=AHH6PRFUSGTDGB7H52DRITARA4IF6GFLVSRYS2Y6EPJCZSKUOYWQ, seg_pos={0 0}, upload=5895]: Uploading piece... Jul 27 04:06:23.949422808 (6h14m39.730356187s) [upload=5895]: Next returned nil segment Jul 27 04:06:23.949442576 (6h14m39.73037596s) [upload=5895]: Waiting for error group managing segments... Jul 27 04:06:23.953497426 (6h14m39.734430802s) [noise=true , address=45.91.116.10:4081 , piece_id=XGDTSEZOUNRTEVWHTO3DHTZMH2VZZOPJPEKLCFAHTXESLLYABHSA, seg_pos={0 0}, upload=5895]: Done uploading piece. err: <nil> Jul 27 04:06:23.953531796 (6h14m39.734465159s) [noise=true , address=45.91.116.6:4973 , piece_id=J3VULPU24CIM64VEVE3UHO62HPN6TJXGZU6BCJIS5JQJKHJK2LIQ, seg_pos={0 0}, upload=5895]: Done uploading piece. err: <nil> Jul 27 04:06:23.954068660 (6h14m39.735002029s) [noise=true , address=45.91.116.9:4262 , piece_id=M5ETD45HH2XZD6HDLTJ3JO2BOOIROXFWKVPTN42PSQEWG4F7KAAQ, seg_pos={0 0}, upload=5895]: Done uploading piece. err: <nil> Jul 27 04:06:23.954194339 (6h14m39.735127707s) [noise=true , address=45.91.116.8:4048 , piece_id=5REWMVVTBVLDQXQ5KAEHLY4VJL5SGF7ONQWDQQ3N2FDU3OAO6AZA, seg_pos={0 0}, upload=5895]: Done uploading piece. err: <nil> Jul 27 04:06:23.954601730 (6h14m39.735535112s) [noise=true , address=45.91.116.7:4011 , piece_id=XMH6MCCHZBBRQWHJFWLRGYHLQD4KUGSGL7MMZSKA5TZJCND3DUDA, seg_pos={0 0}, upload=5895]: Done uploading piece. err: <nil> Jul 27 04:06:23.954755872 (6h14m39.735689237s) [noise=true , address=45.91.116.13:4153 , piece_id=H4535KOCG3F4RKA6AKKMTJSOM6Q6PGRC6M7FR3UIS3BORTSP5UQA, seg_pos={0 0}, upload=5895]: Done uploading piece. err: <nil> Jul 27 04:06:23.955611883 (6h14m39.73654526s) [noise=true , address=77.90.31.19:4843 , piece_id=YML5RDQIK432Z4FSE4DKXIV5U47LQBEZRFMAVLANUSGFUQR23UWQ, seg_pos={0 0}, upload=5895]: Done uploading piece. err: <nil> Jul 27 04:06:23.956373391 (6h14m39.737306756s) [noise=true , address=77.90.31.15:4697 , piece_id=Q76TJDSZPSYZSJQNFVOI462U6DFILMH2RV3LQISODXSOCXGRKW6Q, seg_pos={0 0}, upload=5895]: Done uploading piece. err: <nil> Jul 27 04:06:23.958659122 (6h14m39.73959249s) [noise=true , address=77.90.31.17:4771 , piece_id=U2LF22GKCMKRZN6EWCF26ITDZQHBFZXCGSQXT33MCX6P57IGI35Q, seg_pos={0 0}, upload=5895]: Done uploading piece. err: <nil> Jul 27 04:06:35.058944013 (6h14m50.83987739s) [noise=true , address=104.36.82.16:4293 , piece_id=AHH6PRFUSGTDGB7H52DRITARA4IF6GFLVSRYS2Y6EPJCZSKUOYWQ, seg_pos={0 0}, upload=5895]: Done uploading piece. err: <nil> Jul 27 04:06:41.103204002 (6h14m56.884137386s) [noise=true , address=104.36.82.4:4804 , piece_id=IU6FDNVB5AUPA7MJ6RSXUTLBETPHOFGNJ7YGWBFIZSQFSO6W6BBA, seg_pos={0 0}, upload=5895]: Done uploading piece. err: <nil> Jul 27 04:06:43.191702285 (6h14m58.972635651s) [noise=true , address=104.36.82.6:4873 , piece_id=YP4VJORZZUL5EVCH3P7LD3LNVMHB7CWRTKHGIA7UROXE44N6MMZA, seg_pos={0 0}, upload=5895]: Done uploading piece. err: <nil> Jul 27 04:06:53.777973493 (6h15m9.558906861s) [noise=true , address=104.36.82.11:4115 , piece_id=YXMNVT6ZHGESGQ2BXKZUVR7U2CNNAI2PO7ZTMDG5UIOV7HTDDUTA, seg_pos={0 0}, upload=5895]: Done uploading piece. err: ecclient: upload failed (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR, address:104.36.82.11:4115): protocol: write tcp 159.223.143.39:34210->104.36.82.11:4115: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 04:06:53.778019532 (6h15m9.558952909s) [noise=true , address=104.36.82.12:4151 , piece_id=E2TEUTQBD4Y4AUHSCPMCIFEXBSY7S5HJWBKNKWWTUWEW2BLAAE6Q, seg_pos={0 0}, upload=5895]: Done uploading piece. err: ecclient: upload failed (node:1w6pwZMdeQKjt92qx6WbTWU5VPYJto5gS87f6iKvCBZH6LfyeB, address:104.36.82.12:4151): protocol: rpc: dial tcp 104.36.82.12:4151: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 04:06:53.778043653 (6h15m9.558977021s) [noise=true , address=104.36.82.13:4186 , piece_id=TBLDKALGJYBGR5NLCOCKJ4QKEFNIJXCKAFUBD7VR3UMNWEMKMKHQ, seg_pos={0 0}, upload=5895]: Done uploading piece. err: ecclient: upload failed (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht, address:104.36.82.13:4186): protocol: read tcp 159.223.143.39:34340->104.36.82.13:4186: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 04:06:53.778095228 (6h15m9.559028593s) [noise=true , address=104.36.82.10:4079 , piece_id=2SH5D6JOMLFR7VSN7ZY5CLMR5RHIRFYQ7Z5I2JIYTJQM43G4O6OQ, seg_pos={0 0}, upload=5895]: Done uploading piece. err: ecclient: upload failed (node:12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud, address:104.36.82.10:4079): protocol: write tcp 159.223.143.39:53840->104.36.82.10:4079: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 04:06:53.778142167 (6h15m9.559075532s) [seg_pos={0 0}, upload=5895]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 27 04:06:53.837032161 (6h15m9.617965543s) [upload=5895]: Done uploading segments... Jul 27 04:06:53.837055996 (6h15m9.617989366s) [upload=5895]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 429.096745ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [........X...................................................] duration: 29.164646913s FAIL (2025-07-27 04:44:32 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== {0 0}, upload=5911]: Begin upload segment... Jul 27 04:07:17.745307658 (6h15m33.526241043s) [seg_pos={0 0}, upload=5911]: Done begin upload segment. Jul 27 04:07:17.745349241 (6h15m33.526282625s) [noise=true , address=104.36.82.52:4098 , piece_id=M2ETF25QQ53RRGFPUKXJRINIWV7AWXSECEDR2AIM645IED72NURQ, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.745439694 (6h15m33.526373073s) [noise=true , address=104.36.82.56:4244 , piece_id=LQCZ5W54NBHWAVBVYK7HYSSJDC73E6FV4CP4YIM4L5H2N3OLOZXA, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.745972271 (6h15m33.526905654s) [noise=true , address=104.36.82.36:4650 , piece_id=BKJULWGPESBINNTSUYP6E4KZRXKPACVU5I7DOSZBO4YMK2V26HEQ, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.745980243 (6h15m33.526913621s) [noise=true , address=104.36.82.38:4722 , piece_id=B2MBQY5CTMEI6WSE7FTLOV5RUJ22SCNG7N2EEJILEATGJ4TIALXA, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.746403818 (6h15m33.527337198s) [noise=true , address=104.36.82.22:4534 , piece_id=CW4AQT3XUGA4A5OC6NUY4NYOOAMLN5FFJJN5PQBYEON7HPD5LCYQ, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.746455493 (6h15m33.527388875s) [noise=true , address=104.36.82.14:4248 , piece_id=LDSGCUM7Q2VX5CDFI4JRGEEG7D5H5Q5Z5WUWKU277VN2JFRX6UGA, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.746872127 (6h15m33.527805505s) [noise=true , address=104.36.82.40:4954 , piece_id=YLOHTJLRAED66OMEQYPZHNNRI77V6WOR4EHTEZOOEHMGZHJ2O3QQ, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.747123162 (6h15m33.52805654s) [noise=true , address=104.36.82.46:4496 , piece_id=GWNQR5WRGRRQBLSGWJFD3O3UNCMNE62JQNPIV5SYKBUIL3BQKA5A, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.747280678 (6h15m33.528214053s) [noise=true , address=104.36.82.47:4524 , piece_id=DONJFTKUQ2XCW6BOU7AVOKJ7XMDMBPGQVUPRANH7X2YS7PJXUKIA, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.747736400 (6h15m33.528669779s) [noise=true , address=104.36.82.44:4893 , piece_id=HUJICRJ6W54O3AZZR6H3KG6E4TGBTNS46O6XVMQM3PFAZE7A2HZA, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.747755734 (6h15m33.528689109s) [noise=true , address=104.36.82.55:4204 , piece_id=72WISGRHMOAZDV5EDGTBOMLBL4HDOF3CBEMQAAIRGILPC7J5JKAQ, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.748313124 (6h15m33.5292465s) [noise=true , address=104.36.82.58:4287 , piece_id=BVDYUEWU7TVEQDMZXX5SJNS4QICVKMTGFLCUV6P2YHOE7TI25VXA, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.748485217 (6h15m33.529418594s) [noise=true , address=104.36.82.53:4123 , piece_id=NXJRZ5BQSLXSOIOSUJ4MM4AGKHP4UIDUGEJKLUUX7YOGQJA6WWZA, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.748831329 (6h15m33.529764704s) [noise=true , address=104.36.82.42:4829 , piece_id=ZLWJFP3LGX3YLXBJYIR66DXTYEVDBKXKVXCQSF2PCYCYJBIEJGMA, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.749299293 (6h15m33.530232669s) [noise=true , address=104.36.82.8:4002 , piece_id=6XSPWQC3US355PS6IQ4K3IYNQZH5MAT4IVIPGSYRUM3XYPSCWE6Q, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.749790974 (6h15m33.53072436s) [noise=true , address=104.36.82.44:4901 , piece_id=N4YLCLR2QZSULUL2XWONEKBFUMNOU7KUX2QEOK4SSDFSCI7IL6LA, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.750478221 (6h15m33.531411597s) [noise=true , address=104.36.82.54:4172 , piece_id=ZNA53MWDRWPUH6FUR4DJERBDT2IOYGA3IYUADKXXIOLQJLQOXE7Q, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.752884255 (6h15m33.533817625s) [noise=true , address=64.49.13.140:4508 , piece_id=MZ52U75VOIK6Q7QFXMMYRRG7JXFWDFU2JWQFYPERUCMH3BUX6XPQ, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.879839555 (6h15m33.660772917s) [upload=5911]: Next returned nil segment Jul 27 04:07:17.879861389 (6h15m33.660794765s) [upload=5911]: Waiting for error group managing segments... Jul 27 04:07:17.888652438 (6h15m33.669585811s) [noise=true , address=104.36.82.58:4287 , piece_id=BVDYUEWU7TVEQDMZXX5SJNS4QICVKMTGFLCUV6P2YHOE7TI25VXA, seg_pos={0 0}, upload=5911]: Done uploading piece. err: <nil> Jul 27 04:07:17.952192045 (6h15m33.733125429s) [noise=true , address=104.36.82.53:4123 , piece_id=NXJRZ5BQSLXSOIOSUJ4MM4AGKHP4UIDUGEJKLUUX7YOGQJA6WWZA, seg_pos={0 0}, upload=5911]: Done uploading piece. err: <nil> Jul 27 04:07:17.981907539 (6h15m33.762840911s) [noise=true , address=104.36.82.56:4244 , piece_id=LQCZ5W54NBHWAVBVYK7HYSSJDC73E6FV4CP4YIM4L5H2N3OLOZXA, seg_pos={0 0}, upload=5911]: Done uploading piece. err: <nil> Jul 27 04:07:17.982370496 (6h15m33.7633039s) [noise=true , address=104.36.82.55:4204 , piece_id=72WISGRHMOAZDV5EDGTBOMLBL4HDOF3CBEMQAAIRGILPC7J5JKAQ, seg_pos={0 0}, upload=5911]: Done uploading piece. err: <nil> Jul 27 04:07:18.009548506 (6h15m33.790481886s) [noise=true , address=104.36.82.52:4098 , piece_id=M2ETF25QQ53RRGFPUKXJRINIWV7AWXSECEDR2AIM645IED72NURQ, seg_pos={0 0}, upload=5911]: Done uploading piece. err: <nil> Jul 27 04:07:18.023770598 (6h15m33.804703983s) [noise=true , address=104.36.82.40:4954 , piece_id=YLOHTJLRAED66OMEQYPZHNNRI77V6WOR4EHTEZOOEHMGZHJ2O3QQ, seg_pos={0 0}, upload=5911]: Done uploading piece. err: <nil> Jul 27 04:07:18.139745581 (6h15m33.92067895s) [noise=true , address=104.36.82.22:4534 , piece_id=CW4AQT3XUGA4A5OC6NUY4NYOOAMLN5FFJJN5PQBYEON7HPD5LCYQ, seg_pos={0 0}, upload=5911]: Done uploading piece. err: <nil> Jul 27 04:07:18.423748140 (6h15m34.204681511s) [noise=true , address=64.49.13.140:4508 , piece_id=MZ52U75VOIK6Q7QFXMMYRRG7JXFWDFU2JWQFYPERUCMH3BUX6XPQ, seg_pos={0 0}, upload=5911]: Done uploading piece. err: <nil> Jul 27 04:08:16.739940381 (6h16m32.52087375s) [noise=true , address=104.36.82.54:4172 , piece_id=ZNA53MWDRWPUH6FUR4DJERBDT2IOYGA3IYUADKXXIOLQJLQOXE7Q, seg_pos={0 0}, upload=5911]: Done uploading piece. err: <nil> Jul 27 04:08:17.177295397 (6h16m32.958228766s) [noise=true , address=104.36.82.8:4002 , piece_id=6XSPWQC3US355PS6IQ4K3IYNQZH5MAT4IVIPGSYRUM3XYPSCWE6Q, seg_pos={0 0}, upload=5911]: Done uploading piece. err: <nil> Jul 27 04:08:17.534296584 (6h16m33.315229953s) [noise=true , address=104.36.82.36:4650 , piece_id=BKJULWGPESBINNTSUYP6E4KZRXKPACVU5I7DOSZBO4YMK2V26HEQ, seg_pos={0 0}, upload=5911]: Done uploading piece. err: <nil> Jul 27 04:08:17.677851566 (6h16m33.458784943s) [noise=true , address=104.36.82.14:4248 , piece_id=LDSGCUM7Q2VX5CDFI4JRGEEG7D5H5Q5Z5WUWKU277VN2JFRX6UGA, seg_pos={0 0}, upload=5911]: Done uploading piece. err: ecclient: upload failed (node:1u6ir2PLqDa9xGm64cuqq1RqY4qeBa1fqCo9VghspBf54XEXqn, address:104.36.82.14:4248): protocol: rpc: dial tcp 104.36.82.14:4248: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 04:08:17.678115170 (6h16m33.459048549s) [noise=true , address=104.36.82.46:4496 , piece_id=GWNQR5WRGRRQBLSGWJFD3O3UNCMNE62JQNPIV5SYKBUIL3BQKA5A, seg_pos={0 0}, upload=5911]: Done uploading piece. err: ecclient: upload failed (node:12V8XhSWkyGUVYFR8MykifFAP1YAfGCwzqucDcMMfzw3xsVW77Z, address:104.36.82.46:4496): protocol: read tcp 159.223.143.39:52800->104.36.82.46:4496: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 04:08:17.678144409 (6h16m33.459077789s) [noise=true , address=104.36.82.44:4901 , piece_id=N4YLCLR2QZSULUL2XWONEKBFUMNOU7KUX2QEOK4SSDFSCI7IL6LA, seg_pos={0 0}, upload=5911]: Done uploading piece. err: ecclient: upload failed (node:1FHE84zvBb1MKwB1Ht7puTafgQxDEvPt4mYqkYVSSufAMpWQA1, address:104.36.82.44:4901): protocol: rpc: dial tcp 104.36.82.44:4901: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 04:08:17.678229622 (6h16m33.459162988s) [noise=true , address=104.36.82.44:4893 , piece_id=HUJICRJ6W54O3AZZR6H3KG6E4TGBTNS46O6XVMQM3PFAZE7A2HZA, seg_pos={0 0}, upload=5911]: Done uploading piece. err: ecclient: upload failed (node:17VMFZ2T8jrSJVtchciJ9SnyrR6JstvcMoXrN5Q3zTD1aQU1oJ, address:104.36.82.44:4893): protocol: write tcp 159.223.143.39:35302->104.36.82.44:4893: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 04:08:17.678310218 (6h16m33.459243583s) [noise=true , address=104.36.82.42:4829 , piece_id=ZLWJFP3LGX3YLXBJYIR66DXTYEVDBKXKVXCQSF2PCYCYJBIEJGMA, seg_pos={0 0}, upload=5911]: Done uploading piece. err: ecclient: upload failed (node:1F5T8Vt1x4nXFaSenvGBCYjxhRL1H1sxRvzLjW3CqzwDVhq39F, address:104.36.82.42:4829): protocol: rpc: dial tcp 104.36.82.42:4829: i/o timeout; conn closed: op; piecestore: piecestore close: context deadline exceeded Jul 27 04:08:17.678412989 (6h16m33.459346365s) [noise=true , address=104.36.82.38:4722 , piece_id=B2MBQY5CTMEI6WSE7FTLOV5RUJ22SCNG7N2EEJILEATGJ4TIALXA, seg_pos={0 0}, upload=5911]: Done uploading piece. err: ecclient: upload failed (node:12Zw3dtUg9gARz52KhwDLTjcqC3CNm7dpPzpdx7P7PbL1jwe8uc, address:104.36.82.38:4722): protocol: read tcp 159.223.143.39:56048->104.36.82.38:4722: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 04:08:17.678455247 (6h16m33.459388624s) [noise=true , address=104.36.82.47:4524 , piece_id=DONJFTKUQ2XCW6BOU7AVOKJ7XMDMBPGQVUPRANH7X2YS7PJXUKIA, seg_pos={0 0}, upload=5911]: Done uploading piece. err: ecclient: upload failed (node:1xfbCzjSwFDRrR1gPggeN8tNCLbRRWDVzCJ6mqhZsbVSe8d4s4, address:104.36.82.47:4524): protocol: read tcp 159.223.143.39:46694->104.36.82.47:4524: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 04:08:17.678510668 (6h16m33.459444052s) [seg_pos={0 0}, upload=5911]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Jul 27 04:08:17.706442864 (6h16m33.487376232s) [upload=5911]: Done uploading segments... Jul 27 04:08:17.706495329 (6h16m33.487428705s) [upload=5911]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 29.164680818s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................X.......................................] duration: 384.972119ms FAIL (2025-07-27 04:44:31 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 12) Duration: 384.999578ms core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................X......................................] duration: 693.759569ms FAIL (2025-07-27 04:43:54 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 12) Duration: 693.785891ms core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......X....................................................] duration: 510.367129ms FAIL (2025-07-27 04:43:53 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== ece_id=NYOFDEZ7CV4PHGQ6JZMEEVAYGJRPYG6IBW7MYJO42HMWOJM7PHZA, seg_pos={0 0}, upload=5879]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12crcXpNe9NgeXTsUpooLSBgrSUFbRxbSbanLa39w2Hr5mGFww): protocol: read tcp 159.223.143.39:53630->104.36.82.5:4840: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 04:05:32.508904485 (6h13m48.289837866s) [seg_pos={0 0}, upload=5879]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 27 04:05:32.509028070 (6h13m48.289961457s) [upload=5879]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 27 04:05:32.509034645 (6h13m48.289968129s) [upload=5879]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 27 04:05:32.581834839 (6h13m48.362768225s) [upload=5879]: Done uploading segments... Jul 27 04:05:32.581875919 (6h13m48.362809298s) [upload=5879]: Upload finished. err: <nil> Jul 27 04:06:23.829155331 (6h14m39.610088696s) [upload=5895]: Starting upload Jul 27 04:06:23.829225021 (6h14m39.610158398s) [upload=5895]: Done starting upload Jul 27 04:06:23.829294606 (6h14m39.61022799s) [upload=5895]: Uploading segments... Jul 27 04:06:23.829577656 (6h14m39.610511053s) [upload=5895]: Got next segment. Inline: false Jul 27 04:06:23.829649664 (6h14m39.610583027s) [upload=5895]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 27 04:06:23.829657065 (6h14m39.61059043s) [upload=5895]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 27 04:06:23.861405530 (6h14m39.642338913s) [seg_pos={0 0}, upload=5895]: Begin upload segment... Jul 27 04:06:23.861476351 (6h14m39.642409713s) [seg_pos={0 0}, upload=5895]: Done begin upload segment. Jul 27 04:06:23.861509120 (6h14m39.642442506s) [noise=true , address=77.90.31.19:4843 , piece_id=YML5RDQIK432Z4FSE4DKXIV5U47LQBEZRFMAVLANUSGFUQR23UWQ, seg_pos={0 0}, upload=5895]: Uploading piece... Jul 27 04:06:23.861953723 (6h14m39.6428871s) [noise=true , address=45.91.116.10:4081 , piece_id=XGDTSEZOUNRTEVWHTO3DHTZMH2VZZOPJPEKLCFAHTXESLLYABHSA, seg_pos={0 0}, upload=5895]: Uploading piece... Jul 27 04:06:23.862060983 (6h14m39.642994358s) [noise=true , address=45.91.116.9:4262 , piece_id=M5ETD45HH2XZD6HDLTJ3JO2BOOIROXFWKVPTN42PSQEWG4F7KAAQ, seg_pos={0 0}, upload=5895]: Uploading piece... Jul 27 04:06:23.862759464 (6h14m39.64369284s) [noise=true , address=104.36.82.4:4804 , piece_id=IU6FDNVB5AUPA7MJ6RSXUTLBETPHOFGNJ7YGWBFIZSQFSO6W6BBA, seg_pos={0 0}, upload=5895]: Uploading piece... Jul 27 04:06:23.862835086 (6h14m39.643768461s) [noise=true , address=104.36.82.6:4873 , piece_id=YP4VJORZZUL5EVCH3P7LD3LNVMHB7CWRTKHGIA7UROXE44N6MMZA, seg_pos={0 0}, upload=5895]: Uploading piece... Jul 27 04:06:23.863593880 (6h14m39.644527258s) [noise=true , address=77.90.31.15:4697 , piece_id=Q76TJDSZPSYZSJQNFVOI462U6DFILMH2RV3LQISODXSOCXGRKW6Q, seg_pos={0 0}, upload=5895]: Uploading piece... Jul 27 04:06:23.863656891 (6h14m39.644590266s) [noise=true , address=45.91.116.7:4011 , piece_id=XMH6MCCHZBBRQWHJFWLRGYHLQD4KUGSGL7MMZSKA5TZJCND3DUDA, seg_pos={0 0}, upload=5895]: Uploading piece... Jul 27 04:06:23.864408612 (6h14m39.645341987s) [noise=true , address=45.91.116.6:4973 , piece_id=J3VULPU24CIM64VEVE3UHO62HPN6TJXGZU6BCJIS5JQJKHJK2LIQ, seg_pos={0 0}, upload=5895]: Uploading piece... Jul 27 04:06:23.864521549 (6h14m39.64545491s) [noise=true , address=104.36.82.10:4079 , piece_id=2SH5D6JOMLFR7VSN7ZY5CLMR5RHIRFYQ7Z5I2JIYTJQM43G4O6OQ, seg_pos={0 0}, upload=5895]: Uploading piece... Jul 27 04:06:23.865518491 (6h14m39.646451872s) [noise=true , address=77.90.31.17:4771 , piece_id=U2LF22GKCMKRZN6EWCF26ITDZQHBFZXCGSQXT33MCX6P57IGI35Q, seg_pos={0 0}, upload=5895]: Uploading piece... Jul 27 04:06:23.865654950 (6h14m39.646588325s) [noise=true , address=45.91.116.8:4048 , piece_id=5REWMVVTBVLDQXQ5KAEHLY4VJL5SGF7ONQWDQQ3N2FDU3OAO6AZA, seg_pos={0 0}, upload=5895]: Uploading piece... Jul 27 04:06:23.866190892 (6h14m39.647124271s) [noise=true , address=104.36.82.12:4151 , piece_id=E2TEUTQBD4Y4AUHSCPMCIFEXBSY7S5HJWBKNKWWTUWEW2BLAAE6Q, seg_pos={0 0}, upload=5895]: Uploading piece... Jul 27 04:06:23.866412590 (6h14m39.647345968s) [noise=true , address=104.36.82.11:4115 , piece_id=YXMNVT6ZHGESGQ2BXKZUVR7U2CNNAI2PO7ZTMDG5UIOV7HTDDUTA, seg_pos={0 0}, upload=5895]: Uploading piece... Jul 27 04:06:23.866715972 (6h14m39.647649359s) [noise=true , address=45.91.116.13:4153 , piece_id=H4535KOCG3F4RKA6AKKMTJSOM6Q6PGRC6M7FR3UIS3BORTSP5UQA, seg_pos={0 0}, upload=5895]: Uploading piece... Jul 27 04:06:23.866998965 (6h14m39.64793234s) [noise=true , address=104.36.82.13:4186 , piece_id=TBLDKALGJYBGR5NLCOCKJ4QKEFNIJXCKAFUBD7VR3UMNWEMKMKHQ, seg_pos={0 0}, upload=5895]: Uploading piece... Jul 27 04:06:23.867230593 (6h14m39.648163974s) [noise=true , address=104.36.82.16:4293 , piece_id=AHH6PRFUSGTDGB7H52DRITARA4IF6GFLVSRYS2Y6EPJCZSKUOYWQ, seg_pos={0 0}, upload=5895]: Uploading piece... Jul 27 04:06:23.949422808 (6h14m39.730356187s) [upload=5895]: Next returned nil segment Jul 27 04:06:23.949442576 (6h14m39.73037596s) [upload=5895]: Waiting for error group managing segments... Jul 27 04:06:23.953497426 (6h14m39.734430802s) [noise=true , address=45.91.116.10:4081 , piece_id=XGDTSEZOUNRTEVWHTO3DHTZMH2VZZOPJPEKLCFAHTXESLLYABHSA, seg_pos={0 0}, upload=5895]: Done uploading piece. err: <nil> Jul 27 04:06:23.953531796 (6h14m39.734465159s) [noise=true , address=45.91.116.6:4973 , piece_id=J3VULPU24CIM64VEVE3UHO62HPN6TJXGZU6BCJIS5JQJKHJK2LIQ, seg_pos={0 0}, upload=5895]: Done uploading piece. err: <nil> Jul 27 04:06:23.954068660 (6h14m39.735002029s) [noise=true , address=45.91.116.9:4262 , piece_id=M5ETD45HH2XZD6HDLTJ3JO2BOOIROXFWKVPTN42PSQEWG4F7KAAQ, seg_pos={0 0}, upload=5895]: Done uploading piece. err: <nil> Jul 27 04:06:23.954194339 (6h14m39.735127707s) [noise=true , address=45.91.116.8:4048 , piece_id=5REWMVVTBVLDQXQ5KAEHLY4VJL5SGF7ONQWDQQ3N2FDU3OAO6AZA, seg_pos={0 0}, upload=5895]: Done uploading piece. err: <nil> Jul 27 04:06:23.954601730 (6h14m39.735535112s) [noise=true , address=45.91.116.7:4011 , piece_id=XMH6MCCHZBBRQWHJFWLRGYHLQD4KUGSGL7MMZSKA5TZJCND3DUDA, seg_pos={0 0}, upload=5895]: Done uploading piece. err: <nil> Jul 27 04:06:23.954755872 (6h14m39.735689237s) [noise=true , address=45.91.116.13:4153 , piece_id=H4535KOCG3F4RKA6AKKMTJSOM6Q6PGRC6M7FR3UIS3BORTSP5UQA, seg_pos={0 0}, upload=5895]: Done uploading piece. err: <nil> Jul 27 04:06:23.955611883 (6h14m39.73654526s) [noise=true , address=77.90.31.19:4843 , piece_id=YML5RDQIK432Z4FSE4DKXIV5U47LQBEZRFMAVLANUSGFUQR23UWQ, seg_pos={0 0}, upload=5895]: Done uploading piece. err: <nil> Jul 27 04:06:23.956373391 (6h14m39.737306756s) [noise=true , address=77.90.31.15:4697 , piece_id=Q76TJDSZPSYZSJQNFVOI462U6DFILMH2RV3LQISODXSOCXGRKW6Q, seg_pos={0 0}, upload=5895]: Done uploading piece. err: <nil> Jul 27 04:06:23.958659122 (6h14m39.73959249s) [noise=true , address=77.90.31.17:4771 , piece_id=U2LF22GKCMKRZN6EWCF26ITDZQHBFZXCGSQXT33MCX6P57IGI35Q, seg_pos={0 0}, upload=5895]: Done uploading piece. err: <nil> Jul 27 04:06:35.058944013 (6h14m50.83987739s) [noise=true , address=104.36.82.16:4293 , piece_id=AHH6PRFUSGTDGB7H52DRITARA4IF6GFLVSRYS2Y6EPJCZSKUOYWQ, seg_pos={0 0}, upload=5895]: Done uploading piece. err: <nil> Jul 27 04:06:41.103204002 (6h14m56.884137386s) [noise=true , address=104.36.82.4:4804 , piece_id=IU6FDNVB5AUPA7MJ6RSXUTLBETPHOFGNJ7YGWBFIZSQFSO6W6BBA, seg_pos={0 0}, upload=5895]: Done uploading piece. err: <nil> Jul 27 04:06:43.191702285 (6h14m58.972635651s) [noise=true , address=104.36.82.6:4873 , piece_id=YP4VJORZZUL5EVCH3P7LD3LNVMHB7CWRTKHGIA7UROXE44N6MMZA, seg_pos={0 0}, upload=5895]: Done uploading piece. err: <nil> Jul 27 04:06:53.777973493 (6h15m9.558906861s) [noise=true , address=104.36.82.11:4115 , piece_id=YXMNVT6ZHGESGQ2BXKZUVR7U2CNNAI2PO7ZTMDG5UIOV7HTDDUTA, seg_pos={0 0}, upload=5895]: Done uploading piece. err: ecclient: upload failed (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR, address:104.36.82.11:4115): protocol: write tcp 159.223.143.39:34210->104.36.82.11:4115: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 04:06:53.778019532 (6h15m9.558952909s) [noise=true , address=104.36.82.12:4151 , piece_id=E2TEUTQBD4Y4AUHSCPMCIFEXBSY7S5HJWBKNKWWTUWEW2BLAAE6Q, seg_pos={0 0}, upload=5895]: Done uploading piece. err: ecclient: upload failed (node:1w6pwZMdeQKjt92qx6WbTWU5VPYJto5gS87f6iKvCBZH6LfyeB, address:104.36.82.12:4151): protocol: rpc: dial tcp 104.36.82.12:4151: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 04:06:53.778043653 (6h15m9.558977021s) [noise=true , address=104.36.82.13:4186 , piece_id=TBLDKALGJYBGR5NLCOCKJ4QKEFNIJXCKAFUBD7VR3UMNWEMKMKHQ, seg_pos={0 0}, upload=5895]: Done uploading piece. err: ecclient: upload failed (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht, address:104.36.82.13:4186): protocol: read tcp 159.223.143.39:34340->104.36.82.13:4186: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 04:06:53.778095228 (6h15m9.559028593s) [noise=true , address=104.36.82.10:4079 , piece_id=2SH5D6JOMLFR7VSN7ZY5CLMR5RHIRFYQ7Z5I2JIYTJQM43G4O6OQ, seg_pos={0 0}, upload=5895]: Done uploading piece. err: ecclient: upload failed (node:12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud, address:104.36.82.10:4079): protocol: write tcp 159.223.143.39:53840->104.36.82.10:4079: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 04:06:53.778142167 (6h15m9.559075532s) [seg_pos={0 0}, upload=5895]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 27 04:06:53.837032161 (6h15m9.617965543s) [upload=5895]: Done uploading segments... Jul 27 04:06:53.837055996 (6h15m9.617989366s) [upload=5895]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 510.404056ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.........X..................................................] duration: 697.438278ms FAIL (2025-07-27 04:43:29 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== {0 0}, upload=5911]: Begin upload segment... Jul 27 04:07:17.745307658 (6h15m33.526241043s) [seg_pos={0 0}, upload=5911]: Done begin upload segment. Jul 27 04:07:17.745349241 (6h15m33.526282625s) [noise=true , address=104.36.82.52:4098 , piece_id=M2ETF25QQ53RRGFPUKXJRINIWV7AWXSECEDR2AIM645IED72NURQ, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.745439694 (6h15m33.526373073s) [noise=true , address=104.36.82.56:4244 , piece_id=LQCZ5W54NBHWAVBVYK7HYSSJDC73E6FV4CP4YIM4L5H2N3OLOZXA, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.745972271 (6h15m33.526905654s) [noise=true , address=104.36.82.36:4650 , piece_id=BKJULWGPESBINNTSUYP6E4KZRXKPACVU5I7DOSZBO4YMK2V26HEQ, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.745980243 (6h15m33.526913621s) [noise=true , address=104.36.82.38:4722 , piece_id=B2MBQY5CTMEI6WSE7FTLOV5RUJ22SCNG7N2EEJILEATGJ4TIALXA, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.746403818 (6h15m33.527337198s) [noise=true , address=104.36.82.22:4534 , piece_id=CW4AQT3XUGA4A5OC6NUY4NYOOAMLN5FFJJN5PQBYEON7HPD5LCYQ, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.746455493 (6h15m33.527388875s) [noise=true , address=104.36.82.14:4248 , piece_id=LDSGCUM7Q2VX5CDFI4JRGEEG7D5H5Q5Z5WUWKU277VN2JFRX6UGA, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.746872127 (6h15m33.527805505s) [noise=true , address=104.36.82.40:4954 , piece_id=YLOHTJLRAED66OMEQYPZHNNRI77V6WOR4EHTEZOOEHMGZHJ2O3QQ, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.747123162 (6h15m33.52805654s) [noise=true , address=104.36.82.46:4496 , piece_id=GWNQR5WRGRRQBLSGWJFD3O3UNCMNE62JQNPIV5SYKBUIL3BQKA5A, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.747280678 (6h15m33.528214053s) [noise=true , address=104.36.82.47:4524 , piece_id=DONJFTKUQ2XCW6BOU7AVOKJ7XMDMBPGQVUPRANH7X2YS7PJXUKIA, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.747736400 (6h15m33.528669779s) [noise=true , address=104.36.82.44:4893 , piece_id=HUJICRJ6W54O3AZZR6H3KG6E4TGBTNS46O6XVMQM3PFAZE7A2HZA, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.747755734 (6h15m33.528689109s) [noise=true , address=104.36.82.55:4204 , piece_id=72WISGRHMOAZDV5EDGTBOMLBL4HDOF3CBEMQAAIRGILPC7J5JKAQ, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.748313124 (6h15m33.5292465s) [noise=true , address=104.36.82.58:4287 , piece_id=BVDYUEWU7TVEQDMZXX5SJNS4QICVKMTGFLCUV6P2YHOE7TI25VXA, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.748485217 (6h15m33.529418594s) [noise=true , address=104.36.82.53:4123 , piece_id=NXJRZ5BQSLXSOIOSUJ4MM4AGKHP4UIDUGEJKLUUX7YOGQJA6WWZA, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.748831329 (6h15m33.529764704s) [noise=true , address=104.36.82.42:4829 , piece_id=ZLWJFP3LGX3YLXBJYIR66DXTYEVDBKXKVXCQSF2PCYCYJBIEJGMA, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.749299293 (6h15m33.530232669s) [noise=true , address=104.36.82.8:4002 , piece_id=6XSPWQC3US355PS6IQ4K3IYNQZH5MAT4IVIPGSYRUM3XYPSCWE6Q, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.749790974 (6h15m33.53072436s) [noise=true , address=104.36.82.44:4901 , piece_id=N4YLCLR2QZSULUL2XWONEKBFUMNOU7KUX2QEOK4SSDFSCI7IL6LA, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.750478221 (6h15m33.531411597s) [noise=true , address=104.36.82.54:4172 , piece_id=ZNA53MWDRWPUH6FUR4DJERBDT2IOYGA3IYUADKXXIOLQJLQOXE7Q, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.752884255 (6h15m33.533817625s) [noise=true , address=64.49.13.140:4508 , piece_id=MZ52U75VOIK6Q7QFXMMYRRG7JXFWDFU2JWQFYPERUCMH3BUX6XPQ, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.879839555 (6h15m33.660772917s) [upload=5911]: Next returned nil segment Jul 27 04:07:17.879861389 (6h15m33.660794765s) [upload=5911]: Waiting for error group managing segments... Jul 27 04:07:17.888652438 (6h15m33.669585811s) [noise=true , address=104.36.82.58:4287 , piece_id=BVDYUEWU7TVEQDMZXX5SJNS4QICVKMTGFLCUV6P2YHOE7TI25VXA, seg_pos={0 0}, upload=5911]: Done uploading piece. err: <nil> Jul 27 04:07:17.952192045 (6h15m33.733125429s) [noise=true , address=104.36.82.53:4123 , piece_id=NXJRZ5BQSLXSOIOSUJ4MM4AGKHP4UIDUGEJKLUUX7YOGQJA6WWZA, seg_pos={0 0}, upload=5911]: Done uploading piece. err: <nil> Jul 27 04:07:17.981907539 (6h15m33.762840911s) [noise=true , address=104.36.82.56:4244 , piece_id=LQCZ5W54NBHWAVBVYK7HYSSJDC73E6FV4CP4YIM4L5H2N3OLOZXA, seg_pos={0 0}, upload=5911]: Done uploading piece. err: <nil> Jul 27 04:07:17.982370496 (6h15m33.7633039s) [noise=true , address=104.36.82.55:4204 , piece_id=72WISGRHMOAZDV5EDGTBOMLBL4HDOF3CBEMQAAIRGILPC7J5JKAQ, seg_pos={0 0}, upload=5911]: Done uploading piece. err: <nil> Jul 27 04:07:18.009548506 (6h15m33.790481886s) [noise=true , address=104.36.82.52:4098 , piece_id=M2ETF25QQ53RRGFPUKXJRINIWV7AWXSECEDR2AIM645IED72NURQ, seg_pos={0 0}, upload=5911]: Done uploading piece. err: <nil> Jul 27 04:07:18.023770598 (6h15m33.804703983s) [noise=true , address=104.36.82.40:4954 , piece_id=YLOHTJLRAED66OMEQYPZHNNRI77V6WOR4EHTEZOOEHMGZHJ2O3QQ, seg_pos={0 0}, upload=5911]: Done uploading piece. err: <nil> Jul 27 04:07:18.139745581 (6h15m33.92067895s) [noise=true , address=104.36.82.22:4534 , piece_id=CW4AQT3XUGA4A5OC6NUY4NYOOAMLN5FFJJN5PQBYEON7HPD5LCYQ, seg_pos={0 0}, upload=5911]: Done uploading piece. err: <nil> Jul 27 04:07:18.423748140 (6h15m34.204681511s) [noise=true , address=64.49.13.140:4508 , piece_id=MZ52U75VOIK6Q7QFXMMYRRG7JXFWDFU2JWQFYPERUCMH3BUX6XPQ, seg_pos={0 0}, upload=5911]: Done uploading piece. err: <nil> Jul 27 04:08:16.739940381 (6h16m32.52087375s) [noise=true , address=104.36.82.54:4172 , piece_id=ZNA53MWDRWPUH6FUR4DJERBDT2IOYGA3IYUADKXXIOLQJLQOXE7Q, seg_pos={0 0}, upload=5911]: Done uploading piece. err: <nil> Jul 27 04:08:17.177295397 (6h16m32.958228766s) [noise=true , address=104.36.82.8:4002 , piece_id=6XSPWQC3US355PS6IQ4K3IYNQZH5MAT4IVIPGSYRUM3XYPSCWE6Q, seg_pos={0 0}, upload=5911]: Done uploading piece. err: <nil> Jul 27 04:08:17.534296584 (6h16m33.315229953s) [noise=true , address=104.36.82.36:4650 , piece_id=BKJULWGPESBINNTSUYP6E4KZRXKPACVU5I7DOSZBO4YMK2V26HEQ, seg_pos={0 0}, upload=5911]: Done uploading piece. err: <nil> Jul 27 04:08:17.677851566 (6h16m33.458784943s) [noise=true , address=104.36.82.14:4248 , piece_id=LDSGCUM7Q2VX5CDFI4JRGEEG7D5H5Q5Z5WUWKU277VN2JFRX6UGA, seg_pos={0 0}, upload=5911]: Done uploading piece. err: ecclient: upload failed (node:1u6ir2PLqDa9xGm64cuqq1RqY4qeBa1fqCo9VghspBf54XEXqn, address:104.36.82.14:4248): protocol: rpc: dial tcp 104.36.82.14:4248: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 04:08:17.678115170 (6h16m33.459048549s) [noise=true , address=104.36.82.46:4496 , piece_id=GWNQR5WRGRRQBLSGWJFD3O3UNCMNE62JQNPIV5SYKBUIL3BQKA5A, seg_pos={0 0}, upload=5911]: Done uploading piece. err: ecclient: upload failed (node:12V8XhSWkyGUVYFR8MykifFAP1YAfGCwzqucDcMMfzw3xsVW77Z, address:104.36.82.46:4496): protocol: read tcp 159.223.143.39:52800->104.36.82.46:4496: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 04:08:17.678144409 (6h16m33.459077789s) [noise=true , address=104.36.82.44:4901 , piece_id=N4YLCLR2QZSULUL2XWONEKBFUMNOU7KUX2QEOK4SSDFSCI7IL6LA, seg_pos={0 0}, upload=5911]: Done uploading piece. err: ecclient: upload failed (node:1FHE84zvBb1MKwB1Ht7puTafgQxDEvPt4mYqkYVSSufAMpWQA1, address:104.36.82.44:4901): protocol: rpc: dial tcp 104.36.82.44:4901: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 04:08:17.678229622 (6h16m33.459162988s) [noise=true , address=104.36.82.44:4893 , piece_id=HUJICRJ6W54O3AZZR6H3KG6E4TGBTNS46O6XVMQM3PFAZE7A2HZA, seg_pos={0 0}, upload=5911]: Done uploading piece. err: ecclient: upload failed (node:17VMFZ2T8jrSJVtchciJ9SnyrR6JstvcMoXrN5Q3zTD1aQU1oJ, address:104.36.82.44:4893): protocol: write tcp 159.223.143.39:35302->104.36.82.44:4893: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 04:08:17.678310218 (6h16m33.459243583s) [noise=true , address=104.36.82.42:4829 , piece_id=ZLWJFP3LGX3YLXBJYIR66DXTYEVDBKXKVXCQSF2PCYCYJBIEJGMA, seg_pos={0 0}, upload=5911]: Done uploading piece. err: ecclient: upload failed (node:1F5T8Vt1x4nXFaSenvGBCYjxhRL1H1sxRvzLjW3CqzwDVhq39F, address:104.36.82.42:4829): protocol: rpc: dial tcp 104.36.82.42:4829: i/o timeout; conn closed: op; piecestore: piecestore close: context deadline exceeded Jul 27 04:08:17.678412989 (6h16m33.459346365s) [noise=true , address=104.36.82.38:4722 , piece_id=B2MBQY5CTMEI6WSE7FTLOV5RUJ22SCNG7N2EEJILEATGJ4TIALXA, seg_pos={0 0}, upload=5911]: Done uploading piece. err: ecclient: upload failed (node:12Zw3dtUg9gARz52KhwDLTjcqC3CNm7dpPzpdx7P7PbL1jwe8uc, address:104.36.82.38:4722): protocol: read tcp 159.223.143.39:56048->104.36.82.38:4722: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 04:08:17.678455247 (6h16m33.459388624s) [noise=true , address=104.36.82.47:4524 , piece_id=DONJFTKUQ2XCW6BOU7AVOKJ7XMDMBPGQVUPRANH7X2YS7PJXUKIA, seg_pos={0 0}, upload=5911]: Done uploading piece. err: ecclient: upload failed (node:1xfbCzjSwFDRrR1gPggeN8tNCLbRRWDVzCJ6mqhZsbVSe8d4s4, address:104.36.82.47:4524): protocol: read tcp 159.223.143.39:46694->104.36.82.47:4524: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 04:08:17.678510668 (6h16m33.459444052s) [seg_pos={0 0}, upload=5911]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Jul 27 04:08:17.706442864 (6h16m33.487376232s) [upload=5911]: Done uploading segments... Jul 27 04:08:17.706495329 (6h16m33.487428705s) [upload=5911]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 697.460541ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........X...................................................] duration: 529.483743ms FAIL (2025-07-27 04:42:43 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== ece_id=NYOFDEZ7CV4PHGQ6JZMEEVAYGJRPYG6IBW7MYJO42HMWOJM7PHZA, seg_pos={0 0}, upload=5879]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12crcXpNe9NgeXTsUpooLSBgrSUFbRxbSbanLa39w2Hr5mGFww): protocol: read tcp 159.223.143.39:53630->104.36.82.5:4840: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 04:05:32.508904485 (6h13m48.289837866s) [seg_pos={0 0}, upload=5879]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 27 04:05:32.509028070 (6h13m48.289961457s) [upload=5879]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 27 04:05:32.509034645 (6h13m48.289968129s) [upload=5879]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 27 04:05:32.581834839 (6h13m48.362768225s) [upload=5879]: Done uploading segments... Jul 27 04:05:32.581875919 (6h13m48.362809298s) [upload=5879]: Upload finished. err: <nil> Jul 27 04:06:23.829155331 (6h14m39.610088696s) [upload=5895]: Starting upload Jul 27 04:06:23.829225021 (6h14m39.610158398s) [upload=5895]: Done starting upload Jul 27 04:06:23.829294606 (6h14m39.61022799s) [upload=5895]: Uploading segments... Jul 27 04:06:23.829577656 (6h14m39.610511053s) [upload=5895]: Got next segment. Inline: false Jul 27 04:06:23.829649664 (6h14m39.610583027s) [upload=5895]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 27 04:06:23.829657065 (6h14m39.61059043s) [upload=5895]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 27 04:06:23.861405530 (6h14m39.642338913s) [seg_pos={0 0}, upload=5895]: Begin upload segment... Jul 27 04:06:23.861476351 (6h14m39.642409713s) [seg_pos={0 0}, upload=5895]: Done begin upload segment. Jul 27 04:06:23.861509120 (6h14m39.642442506s) [noise=true , address=77.90.31.19:4843 , piece_id=YML5RDQIK432Z4FSE4DKXIV5U47LQBEZRFMAVLANUSGFUQR23UWQ, seg_pos={0 0}, upload=5895]: Uploading piece... Jul 27 04:06:23.861953723 (6h14m39.6428871s) [noise=true , address=45.91.116.10:4081 , piece_id=XGDTSEZOUNRTEVWHTO3DHTZMH2VZZOPJPEKLCFAHTXESLLYABHSA, seg_pos={0 0}, upload=5895]: Uploading piece... Jul 27 04:06:23.862060983 (6h14m39.642994358s) [noise=true , address=45.91.116.9:4262 , piece_id=M5ETD45HH2XZD6HDLTJ3JO2BOOIROXFWKVPTN42PSQEWG4F7KAAQ, seg_pos={0 0}, upload=5895]: Uploading piece... Jul 27 04:06:23.862759464 (6h14m39.64369284s) [noise=true , address=104.36.82.4:4804 , piece_id=IU6FDNVB5AUPA7MJ6RSXUTLBETPHOFGNJ7YGWBFIZSQFSO6W6BBA, seg_pos={0 0}, upload=5895]: Uploading piece... Jul 27 04:06:23.862835086 (6h14m39.643768461s) [noise=true , address=104.36.82.6:4873 , piece_id=YP4VJORZZUL5EVCH3P7LD3LNVMHB7CWRTKHGIA7UROXE44N6MMZA, seg_pos={0 0}, upload=5895]: Uploading piece... Jul 27 04:06:23.863593880 (6h14m39.644527258s) [noise=true , address=77.90.31.15:4697 , piece_id=Q76TJDSZPSYZSJQNFVOI462U6DFILMH2RV3LQISODXSOCXGRKW6Q, seg_pos={0 0}, upload=5895]: Uploading piece... Jul 27 04:06:23.863656891 (6h14m39.644590266s) [noise=true , address=45.91.116.7:4011 , piece_id=XMH6MCCHZBBRQWHJFWLRGYHLQD4KUGSGL7MMZSKA5TZJCND3DUDA, seg_pos={0 0}, upload=5895]: Uploading piece... Jul 27 04:06:23.864408612 (6h14m39.645341987s) [noise=true , address=45.91.116.6:4973 , piece_id=J3VULPU24CIM64VEVE3UHO62HPN6TJXGZU6BCJIS5JQJKHJK2LIQ, seg_pos={0 0}, upload=5895]: Uploading piece... Jul 27 04:06:23.864521549 (6h14m39.64545491s) [noise=true , address=104.36.82.10:4079 , piece_id=2SH5D6JOMLFR7VSN7ZY5CLMR5RHIRFYQ7Z5I2JIYTJQM43G4O6OQ, seg_pos={0 0}, upload=5895]: Uploading piece... Jul 27 04:06:23.865518491 (6h14m39.646451872s) [noise=true , address=77.90.31.17:4771 , piece_id=U2LF22GKCMKRZN6EWCF26ITDZQHBFZXCGSQXT33MCX6P57IGI35Q, seg_pos={0 0}, upload=5895]: Uploading piece... Jul 27 04:06:23.865654950 (6h14m39.646588325s) [noise=true , address=45.91.116.8:4048 , piece_id=5REWMVVTBVLDQXQ5KAEHLY4VJL5SGF7ONQWDQQ3N2FDU3OAO6AZA, seg_pos={0 0}, upload=5895]: Uploading piece... Jul 27 04:06:23.866190892 (6h14m39.647124271s) [noise=true , address=104.36.82.12:4151 , piece_id=E2TEUTQBD4Y4AUHSCPMCIFEXBSY7S5HJWBKNKWWTUWEW2BLAAE6Q, seg_pos={0 0}, upload=5895]: Uploading piece... Jul 27 04:06:23.866412590 (6h14m39.647345968s) [noise=true , address=104.36.82.11:4115 , piece_id=YXMNVT6ZHGESGQ2BXKZUVR7U2CNNAI2PO7ZTMDG5UIOV7HTDDUTA, seg_pos={0 0}, upload=5895]: Uploading piece... Jul 27 04:06:23.866715972 (6h14m39.647649359s) [noise=true , address=45.91.116.13:4153 , piece_id=H4535KOCG3F4RKA6AKKMTJSOM6Q6PGRC6M7FR3UIS3BORTSP5UQA, seg_pos={0 0}, upload=5895]: Uploading piece... Jul 27 04:06:23.866998965 (6h14m39.64793234s) [noise=true , address=104.36.82.13:4186 , piece_id=TBLDKALGJYBGR5NLCOCKJ4QKEFNIJXCKAFUBD7VR3UMNWEMKMKHQ, seg_pos={0 0}, upload=5895]: Uploading piece... Jul 27 04:06:23.867230593 (6h14m39.648163974s) [noise=true , address=104.36.82.16:4293 , piece_id=AHH6PRFUSGTDGB7H52DRITARA4IF6GFLVSRYS2Y6EPJCZSKUOYWQ, seg_pos={0 0}, upload=5895]: Uploading piece... Jul 27 04:06:23.949422808 (6h14m39.730356187s) [upload=5895]: Next returned nil segment Jul 27 04:06:23.949442576 (6h14m39.73037596s) [upload=5895]: Waiting for error group managing segments... Jul 27 04:06:23.953497426 (6h14m39.734430802s) [noise=true , address=45.91.116.10:4081 , piece_id=XGDTSEZOUNRTEVWHTO3DHTZMH2VZZOPJPEKLCFAHTXESLLYABHSA, seg_pos={0 0}, upload=5895]: Done uploading piece. err: <nil> Jul 27 04:06:23.953531796 (6h14m39.734465159s) [noise=true , address=45.91.116.6:4973 , piece_id=J3VULPU24CIM64VEVE3UHO62HPN6TJXGZU6BCJIS5JQJKHJK2LIQ, seg_pos={0 0}, upload=5895]: Done uploading piece. err: <nil> Jul 27 04:06:23.954068660 (6h14m39.735002029s) [noise=true , address=45.91.116.9:4262 , piece_id=M5ETD45HH2XZD6HDLTJ3JO2BOOIROXFWKVPTN42PSQEWG4F7KAAQ, seg_pos={0 0}, upload=5895]: Done uploading piece. err: <nil> Jul 27 04:06:23.954194339 (6h14m39.735127707s) [noise=true , address=45.91.116.8:4048 , piece_id=5REWMVVTBVLDQXQ5KAEHLY4VJL5SGF7ONQWDQQ3N2FDU3OAO6AZA, seg_pos={0 0}, upload=5895]: Done uploading piece. err: <nil> Jul 27 04:06:23.954601730 (6h14m39.735535112s) [noise=true , address=45.91.116.7:4011 , piece_id=XMH6MCCHZBBRQWHJFWLRGYHLQD4KUGSGL7MMZSKA5TZJCND3DUDA, seg_pos={0 0}, upload=5895]: Done uploading piece. err: <nil> Jul 27 04:06:23.954755872 (6h14m39.735689237s) [noise=true , address=45.91.116.13:4153 , piece_id=H4535KOCG3F4RKA6AKKMTJSOM6Q6PGRC6M7FR3UIS3BORTSP5UQA, seg_pos={0 0}, upload=5895]: Done uploading piece. err: <nil> Jul 27 04:06:23.955611883 (6h14m39.73654526s) [noise=true , address=77.90.31.19:4843 , piece_id=YML5RDQIK432Z4FSE4DKXIV5U47LQBEZRFMAVLANUSGFUQR23UWQ, seg_pos={0 0}, upload=5895]: Done uploading piece. err: <nil> Jul 27 04:06:23.956373391 (6h14m39.737306756s) [noise=true , address=77.90.31.15:4697 , piece_id=Q76TJDSZPSYZSJQNFVOI462U6DFILMH2RV3LQISODXSOCXGRKW6Q, seg_pos={0 0}, upload=5895]: Done uploading piece. err: <nil> Jul 27 04:06:23.958659122 (6h14m39.73959249s) [noise=true , address=77.90.31.17:4771 , piece_id=U2LF22GKCMKRZN6EWCF26ITDZQHBFZXCGSQXT33MCX6P57IGI35Q, seg_pos={0 0}, upload=5895]: Done uploading piece. err: <nil> Jul 27 04:06:35.058944013 (6h14m50.83987739s) [noise=true , address=104.36.82.16:4293 , piece_id=AHH6PRFUSGTDGB7H52DRITARA4IF6GFLVSRYS2Y6EPJCZSKUOYWQ, seg_pos={0 0}, upload=5895]: Done uploading piece. err: <nil> Jul 27 04:06:41.103204002 (6h14m56.884137386s) [noise=true , address=104.36.82.4:4804 , piece_id=IU6FDNVB5AUPA7MJ6RSXUTLBETPHOFGNJ7YGWBFIZSQFSO6W6BBA, seg_pos={0 0}, upload=5895]: Done uploading piece. err: <nil> Jul 27 04:06:43.191702285 (6h14m58.972635651s) [noise=true , address=104.36.82.6:4873 , piece_id=YP4VJORZZUL5EVCH3P7LD3LNVMHB7CWRTKHGIA7UROXE44N6MMZA, seg_pos={0 0}, upload=5895]: Done uploading piece. err: <nil> Jul 27 04:06:53.777973493 (6h15m9.558906861s) [noise=true , address=104.36.82.11:4115 , piece_id=YXMNVT6ZHGESGQ2BXKZUVR7U2CNNAI2PO7ZTMDG5UIOV7HTDDUTA, seg_pos={0 0}, upload=5895]: Done uploading piece. err: ecclient: upload failed (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR, address:104.36.82.11:4115): protocol: write tcp 159.223.143.39:34210->104.36.82.11:4115: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 04:06:53.778019532 (6h15m9.558952909s) [noise=true , address=104.36.82.12:4151 , piece_id=E2TEUTQBD4Y4AUHSCPMCIFEXBSY7S5HJWBKNKWWTUWEW2BLAAE6Q, seg_pos={0 0}, upload=5895]: Done uploading piece. err: ecclient: upload failed (node:1w6pwZMdeQKjt92qx6WbTWU5VPYJto5gS87f6iKvCBZH6LfyeB, address:104.36.82.12:4151): protocol: rpc: dial tcp 104.36.82.12:4151: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 04:06:53.778043653 (6h15m9.558977021s) [noise=true , address=104.36.82.13:4186 , piece_id=TBLDKALGJYBGR5NLCOCKJ4QKEFNIJXCKAFUBD7VR3UMNWEMKMKHQ, seg_pos={0 0}, upload=5895]: Done uploading piece. err: ecclient: upload failed (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht, address:104.36.82.13:4186): protocol: read tcp 159.223.143.39:34340->104.36.82.13:4186: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 04:06:53.778095228 (6h15m9.559028593s) [noise=true , address=104.36.82.10:4079 , piece_id=2SH5D6JOMLFR7VSN7ZY5CLMR5RHIRFYQ7Z5I2JIYTJQM43G4O6OQ, seg_pos={0 0}, upload=5895]: Done uploading piece. err: ecclient: upload failed (node:12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud, address:104.36.82.10:4079): protocol: write tcp 159.223.143.39:53840->104.36.82.10:4079: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 04:06:53.778142167 (6h15m9.559075532s) [seg_pos={0 0}, upload=5895]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 27 04:06:53.837032161 (6h15m9.617965543s) [upload=5895]: Done uploading segments... Jul 27 04:06:53.837055996 (6h15m9.617989366s) [upload=5895]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 529.509223ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................X.....................................] duration: 413.978649ms FAIL (2025-07-27 04:42:43 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 12) Duration: 414.044899ms core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [..........X.................................................] duration: 640.789976ms FAIL (2025-07-27 04:42:29 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== {0 0}, upload=5911]: Begin upload segment... Jul 27 04:07:17.745307658 (6h15m33.526241043s) [seg_pos={0 0}, upload=5911]: Done begin upload segment. Jul 27 04:07:17.745349241 (6h15m33.526282625s) [noise=true , address=104.36.82.52:4098 , piece_id=M2ETF25QQ53RRGFPUKXJRINIWV7AWXSECEDR2AIM645IED72NURQ, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.745439694 (6h15m33.526373073s) [noise=true , address=104.36.82.56:4244 , piece_id=LQCZ5W54NBHWAVBVYK7HYSSJDC73E6FV4CP4YIM4L5H2N3OLOZXA, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.745972271 (6h15m33.526905654s) [noise=true , address=104.36.82.36:4650 , piece_id=BKJULWGPESBINNTSUYP6E4KZRXKPACVU5I7DOSZBO4YMK2V26HEQ, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.745980243 (6h15m33.526913621s) [noise=true , address=104.36.82.38:4722 , piece_id=B2MBQY5CTMEI6WSE7FTLOV5RUJ22SCNG7N2EEJILEATGJ4TIALXA, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.746403818 (6h15m33.527337198s) [noise=true , address=104.36.82.22:4534 , piece_id=CW4AQT3XUGA4A5OC6NUY4NYOOAMLN5FFJJN5PQBYEON7HPD5LCYQ, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.746455493 (6h15m33.527388875s) [noise=true , address=104.36.82.14:4248 , piece_id=LDSGCUM7Q2VX5CDFI4JRGEEG7D5H5Q5Z5WUWKU277VN2JFRX6UGA, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.746872127 (6h15m33.527805505s) [noise=true , address=104.36.82.40:4954 , piece_id=YLOHTJLRAED66OMEQYPZHNNRI77V6WOR4EHTEZOOEHMGZHJ2O3QQ, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.747123162 (6h15m33.52805654s) [noise=true , address=104.36.82.46:4496 , piece_id=GWNQR5WRGRRQBLSGWJFD3O3UNCMNE62JQNPIV5SYKBUIL3BQKA5A, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.747280678 (6h15m33.528214053s) [noise=true , address=104.36.82.47:4524 , piece_id=DONJFTKUQ2XCW6BOU7AVOKJ7XMDMBPGQVUPRANH7X2YS7PJXUKIA, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.747736400 (6h15m33.528669779s) [noise=true , address=104.36.82.44:4893 , piece_id=HUJICRJ6W54O3AZZR6H3KG6E4TGBTNS46O6XVMQM3PFAZE7A2HZA, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.747755734 (6h15m33.528689109s) [noise=true , address=104.36.82.55:4204 , piece_id=72WISGRHMOAZDV5EDGTBOMLBL4HDOF3CBEMQAAIRGILPC7J5JKAQ, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.748313124 (6h15m33.5292465s) [noise=true , address=104.36.82.58:4287 , piece_id=BVDYUEWU7TVEQDMZXX5SJNS4QICVKMTGFLCUV6P2YHOE7TI25VXA, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.748485217 (6h15m33.529418594s) [noise=true , address=104.36.82.53:4123 , piece_id=NXJRZ5BQSLXSOIOSUJ4MM4AGKHP4UIDUGEJKLUUX7YOGQJA6WWZA, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.748831329 (6h15m33.529764704s) [noise=true , address=104.36.82.42:4829 , piece_id=ZLWJFP3LGX3YLXBJYIR66DXTYEVDBKXKVXCQSF2PCYCYJBIEJGMA, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.749299293 (6h15m33.530232669s) [noise=true , address=104.36.82.8:4002 , piece_id=6XSPWQC3US355PS6IQ4K3IYNQZH5MAT4IVIPGSYRUM3XYPSCWE6Q, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.749790974 (6h15m33.53072436s) [noise=true , address=104.36.82.44:4901 , piece_id=N4YLCLR2QZSULUL2XWONEKBFUMNOU7KUX2QEOK4SSDFSCI7IL6LA, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.750478221 (6h15m33.531411597s) [noise=true , address=104.36.82.54:4172 , piece_id=ZNA53MWDRWPUH6FUR4DJERBDT2IOYGA3IYUADKXXIOLQJLQOXE7Q, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.752884255 (6h15m33.533817625s) [noise=true , address=64.49.13.140:4508 , piece_id=MZ52U75VOIK6Q7QFXMMYRRG7JXFWDFU2JWQFYPERUCMH3BUX6XPQ, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.879839555 (6h15m33.660772917s) [upload=5911]: Next returned nil segment Jul 27 04:07:17.879861389 (6h15m33.660794765s) [upload=5911]: Waiting for error group managing segments... Jul 27 04:07:17.888652438 (6h15m33.669585811s) [noise=true , address=104.36.82.58:4287 , piece_id=BVDYUEWU7TVEQDMZXX5SJNS4QICVKMTGFLCUV6P2YHOE7TI25VXA, seg_pos={0 0}, upload=5911]: Done uploading piece. err: <nil> Jul 27 04:07:17.952192045 (6h15m33.733125429s) [noise=true , address=104.36.82.53:4123 , piece_id=NXJRZ5BQSLXSOIOSUJ4MM4AGKHP4UIDUGEJKLUUX7YOGQJA6WWZA, seg_pos={0 0}, upload=5911]: Done uploading piece. err: <nil> Jul 27 04:07:17.981907539 (6h15m33.762840911s) [noise=true , address=104.36.82.56:4244 , piece_id=LQCZ5W54NBHWAVBVYK7HYSSJDC73E6FV4CP4YIM4L5H2N3OLOZXA, seg_pos={0 0}, upload=5911]: Done uploading piece. err: <nil> Jul 27 04:07:17.982370496 (6h15m33.7633039s) [noise=true , address=104.36.82.55:4204 , piece_id=72WISGRHMOAZDV5EDGTBOMLBL4HDOF3CBEMQAAIRGILPC7J5JKAQ, seg_pos={0 0}, upload=5911]: Done uploading piece. err: <nil> Jul 27 04:07:18.009548506 (6h15m33.790481886s) [noise=true , address=104.36.82.52:4098 , piece_id=M2ETF25QQ53RRGFPUKXJRINIWV7AWXSECEDR2AIM645IED72NURQ, seg_pos={0 0}, upload=5911]: Done uploading piece. err: <nil> Jul 27 04:07:18.023770598 (6h15m33.804703983s) [noise=true , address=104.36.82.40:4954 , piece_id=YLOHTJLRAED66OMEQYPZHNNRI77V6WOR4EHTEZOOEHMGZHJ2O3QQ, seg_pos={0 0}, upload=5911]: Done uploading piece. err: <nil> Jul 27 04:07:18.139745581 (6h15m33.92067895s) [noise=true , address=104.36.82.22:4534 , piece_id=CW4AQT3XUGA4A5OC6NUY4NYOOAMLN5FFJJN5PQBYEON7HPD5LCYQ, seg_pos={0 0}, upload=5911]: Done uploading piece. err: <nil> Jul 27 04:07:18.423748140 (6h15m34.204681511s) [noise=true , address=64.49.13.140:4508 , piece_id=MZ52U75VOIK6Q7QFXMMYRRG7JXFWDFU2JWQFYPERUCMH3BUX6XPQ, seg_pos={0 0}, upload=5911]: Done uploading piece. err: <nil> Jul 27 04:08:16.739940381 (6h16m32.52087375s) [noise=true , address=104.36.82.54:4172 , piece_id=ZNA53MWDRWPUH6FUR4DJERBDT2IOYGA3IYUADKXXIOLQJLQOXE7Q, seg_pos={0 0}, upload=5911]: Done uploading piece. err: <nil> Jul 27 04:08:17.177295397 (6h16m32.958228766s) [noise=true , address=104.36.82.8:4002 , piece_id=6XSPWQC3US355PS6IQ4K3IYNQZH5MAT4IVIPGSYRUM3XYPSCWE6Q, seg_pos={0 0}, upload=5911]: Done uploading piece. err: <nil> Jul 27 04:08:17.534296584 (6h16m33.315229953s) [noise=true , address=104.36.82.36:4650 , piece_id=BKJULWGPESBINNTSUYP6E4KZRXKPACVU5I7DOSZBO4YMK2V26HEQ, seg_pos={0 0}, upload=5911]: Done uploading piece. err: <nil> Jul 27 04:08:17.677851566 (6h16m33.458784943s) [noise=true , address=104.36.82.14:4248 , piece_id=LDSGCUM7Q2VX5CDFI4JRGEEG7D5H5Q5Z5WUWKU277VN2JFRX6UGA, seg_pos={0 0}, upload=5911]: Done uploading piece. err: ecclient: upload failed (node:1u6ir2PLqDa9xGm64cuqq1RqY4qeBa1fqCo9VghspBf54XEXqn, address:104.36.82.14:4248): protocol: rpc: dial tcp 104.36.82.14:4248: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 04:08:17.678115170 (6h16m33.459048549s) [noise=true , address=104.36.82.46:4496 , piece_id=GWNQR5WRGRRQBLSGWJFD3O3UNCMNE62JQNPIV5SYKBUIL3BQKA5A, seg_pos={0 0}, upload=5911]: Done uploading piece. err: ecclient: upload failed (node:12V8XhSWkyGUVYFR8MykifFAP1YAfGCwzqucDcMMfzw3xsVW77Z, address:104.36.82.46:4496): protocol: read tcp 159.223.143.39:52800->104.36.82.46:4496: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 04:08:17.678144409 (6h16m33.459077789s) [noise=true , address=104.36.82.44:4901 , piece_id=N4YLCLR2QZSULUL2XWONEKBFUMNOU7KUX2QEOK4SSDFSCI7IL6LA, seg_pos={0 0}, upload=5911]: Done uploading piece. err: ecclient: upload failed (node:1FHE84zvBb1MKwB1Ht7puTafgQxDEvPt4mYqkYVSSufAMpWQA1, address:104.36.82.44:4901): protocol: rpc: dial tcp 104.36.82.44:4901: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 04:08:17.678229622 (6h16m33.459162988s) [noise=true , address=104.36.82.44:4893 , piece_id=HUJICRJ6W54O3AZZR6H3KG6E4TGBTNS46O6XVMQM3PFAZE7A2HZA, seg_pos={0 0}, upload=5911]: Done uploading piece. err: ecclient: upload failed (node:17VMFZ2T8jrSJVtchciJ9SnyrR6JstvcMoXrN5Q3zTD1aQU1oJ, address:104.36.82.44:4893): protocol: write tcp 159.223.143.39:35302->104.36.82.44:4893: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 04:08:17.678310218 (6h16m33.459243583s) [noise=true , address=104.36.82.42:4829 , piece_id=ZLWJFP3LGX3YLXBJYIR66DXTYEVDBKXKVXCQSF2PCYCYJBIEJGMA, seg_pos={0 0}, upload=5911]: Done uploading piece. err: ecclient: upload failed (node:1F5T8Vt1x4nXFaSenvGBCYjxhRL1H1sxRvzLjW3CqzwDVhq39F, address:104.36.82.42:4829): protocol: rpc: dial tcp 104.36.82.42:4829: i/o timeout; conn closed: op; piecestore: piecestore close: context deadline exceeded Jul 27 04:08:17.678412989 (6h16m33.459346365s) [noise=true , address=104.36.82.38:4722 , piece_id=B2MBQY5CTMEI6WSE7FTLOV5RUJ22SCNG7N2EEJILEATGJ4TIALXA, seg_pos={0 0}, upload=5911]: Done uploading piece. err: ecclient: upload failed (node:12Zw3dtUg9gARz52KhwDLTjcqC3CNm7dpPzpdx7P7PbL1jwe8uc, address:104.36.82.38:4722): protocol: read tcp 159.223.143.39:56048->104.36.82.38:4722: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 04:08:17.678455247 (6h16m33.459388624s) [noise=true , address=104.36.82.47:4524 , piece_id=DONJFTKUQ2XCW6BOU7AVOKJ7XMDMBPGQVUPRANH7X2YS7PJXUKIA, seg_pos={0 0}, upload=5911]: Done uploading piece. err: ecclient: upload failed (node:1xfbCzjSwFDRrR1gPggeN8tNCLbRRWDVzCJ6mqhZsbVSe8d4s4, address:104.36.82.47:4524): protocol: read tcp 159.223.143.39:46694->104.36.82.47:4524: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 04:08:17.678510668 (6h16m33.459444052s) [seg_pos={0 0}, upload=5911]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Jul 27 04:08:17.706442864 (6h16m33.487376232s) [upload=5911]: Done uploading segments... Jul 27 04:08:17.706495329 (6h16m33.487428705s) [upload=5911]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 640.831524ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [...........X................................................] duration: 7.333813262s FAIL (2025-07-27 04:41:54 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== {0 0}, upload=5911]: Begin upload segment... Jul 27 04:07:17.745307658 (6h15m33.526241043s) [seg_pos={0 0}, upload=5911]: Done begin upload segment. Jul 27 04:07:17.745349241 (6h15m33.526282625s) [noise=true , address=104.36.82.52:4098 , piece_id=M2ETF25QQ53RRGFPUKXJRINIWV7AWXSECEDR2AIM645IED72NURQ, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.745439694 (6h15m33.526373073s) [noise=true , address=104.36.82.56:4244 , piece_id=LQCZ5W54NBHWAVBVYK7HYSSJDC73E6FV4CP4YIM4L5H2N3OLOZXA, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.745972271 (6h15m33.526905654s) [noise=true , address=104.36.82.36:4650 , piece_id=BKJULWGPESBINNTSUYP6E4KZRXKPACVU5I7DOSZBO4YMK2V26HEQ, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.745980243 (6h15m33.526913621s) [noise=true , address=104.36.82.38:4722 , piece_id=B2MBQY5CTMEI6WSE7FTLOV5RUJ22SCNG7N2EEJILEATGJ4TIALXA, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.746403818 (6h15m33.527337198s) [noise=true , address=104.36.82.22:4534 , piece_id=CW4AQT3XUGA4A5OC6NUY4NYOOAMLN5FFJJN5PQBYEON7HPD5LCYQ, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.746455493 (6h15m33.527388875s) [noise=true , address=104.36.82.14:4248 , piece_id=LDSGCUM7Q2VX5CDFI4JRGEEG7D5H5Q5Z5WUWKU277VN2JFRX6UGA, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.746872127 (6h15m33.527805505s) [noise=true , address=104.36.82.40:4954 , piece_id=YLOHTJLRAED66OMEQYPZHNNRI77V6WOR4EHTEZOOEHMGZHJ2O3QQ, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.747123162 (6h15m33.52805654s) [noise=true , address=104.36.82.46:4496 , piece_id=GWNQR5WRGRRQBLSGWJFD3O3UNCMNE62JQNPIV5SYKBUIL3BQKA5A, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.747280678 (6h15m33.528214053s) [noise=true , address=104.36.82.47:4524 , piece_id=DONJFTKUQ2XCW6BOU7AVOKJ7XMDMBPGQVUPRANH7X2YS7PJXUKIA, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.747736400 (6h15m33.528669779s) [noise=true , address=104.36.82.44:4893 , piece_id=HUJICRJ6W54O3AZZR6H3KG6E4TGBTNS46O6XVMQM3PFAZE7A2HZA, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.747755734 (6h15m33.528689109s) [noise=true , address=104.36.82.55:4204 , piece_id=72WISGRHMOAZDV5EDGTBOMLBL4HDOF3CBEMQAAIRGILPC7J5JKAQ, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.748313124 (6h15m33.5292465s) [noise=true , address=104.36.82.58:4287 , piece_id=BVDYUEWU7TVEQDMZXX5SJNS4QICVKMTGFLCUV6P2YHOE7TI25VXA, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.748485217 (6h15m33.529418594s) [noise=true , address=104.36.82.53:4123 , piece_id=NXJRZ5BQSLXSOIOSUJ4MM4AGKHP4UIDUGEJKLUUX7YOGQJA6WWZA, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.748831329 (6h15m33.529764704s) [noise=true , address=104.36.82.42:4829 , piece_id=ZLWJFP3LGX3YLXBJYIR66DXTYEVDBKXKVXCQSF2PCYCYJBIEJGMA, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.749299293 (6h15m33.530232669s) [noise=true , address=104.36.82.8:4002 , piece_id=6XSPWQC3US355PS6IQ4K3IYNQZH5MAT4IVIPGSYRUM3XYPSCWE6Q, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.749790974 (6h15m33.53072436s) [noise=true , address=104.36.82.44:4901 , piece_id=N4YLCLR2QZSULUL2XWONEKBFUMNOU7KUX2QEOK4SSDFSCI7IL6LA, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.750478221 (6h15m33.531411597s) [noise=true , address=104.36.82.54:4172 , piece_id=ZNA53MWDRWPUH6FUR4DJERBDT2IOYGA3IYUADKXXIOLQJLQOXE7Q, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.752884255 (6h15m33.533817625s) [noise=true , address=64.49.13.140:4508 , piece_id=MZ52U75VOIK6Q7QFXMMYRRG7JXFWDFU2JWQFYPERUCMH3BUX6XPQ, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.879839555 (6h15m33.660772917s) [upload=5911]: Next returned nil segment Jul 27 04:07:17.879861389 (6h15m33.660794765s) [upload=5911]: Waiting for error group managing segments... Jul 27 04:07:17.888652438 (6h15m33.669585811s) [noise=true , address=104.36.82.58:4287 , piece_id=BVDYUEWU7TVEQDMZXX5SJNS4QICVKMTGFLCUV6P2YHOE7TI25VXA, seg_pos={0 0}, upload=5911]: Done uploading piece. err: <nil> Jul 27 04:07:17.952192045 (6h15m33.733125429s) [noise=true , address=104.36.82.53:4123 , piece_id=NXJRZ5BQSLXSOIOSUJ4MM4AGKHP4UIDUGEJKLUUX7YOGQJA6WWZA, seg_pos={0 0}, upload=5911]: Done uploading piece. err: <nil> Jul 27 04:07:17.981907539 (6h15m33.762840911s) [noise=true , address=104.36.82.56:4244 , piece_id=LQCZ5W54NBHWAVBVYK7HYSSJDC73E6FV4CP4YIM4L5H2N3OLOZXA, seg_pos={0 0}, upload=5911]: Done uploading piece. err: <nil> Jul 27 04:07:17.982370496 (6h15m33.7633039s) [noise=true , address=104.36.82.55:4204 , piece_id=72WISGRHMOAZDV5EDGTBOMLBL4HDOF3CBEMQAAIRGILPC7J5JKAQ, seg_pos={0 0}, upload=5911]: Done uploading piece. err: <nil> Jul 27 04:07:18.009548506 (6h15m33.790481886s) [noise=true , address=104.36.82.52:4098 , piece_id=M2ETF25QQ53RRGFPUKXJRINIWV7AWXSECEDR2AIM645IED72NURQ, seg_pos={0 0}, upload=5911]: Done uploading piece. err: <nil> Jul 27 04:07:18.023770598 (6h15m33.804703983s) [noise=true , address=104.36.82.40:4954 , piece_id=YLOHTJLRAED66OMEQYPZHNNRI77V6WOR4EHTEZOOEHMGZHJ2O3QQ, seg_pos={0 0}, upload=5911]: Done uploading piece. err: <nil> Jul 27 04:07:18.139745581 (6h15m33.92067895s) [noise=true , address=104.36.82.22:4534 , piece_id=CW4AQT3XUGA4A5OC6NUY4NYOOAMLN5FFJJN5PQBYEON7HPD5LCYQ, seg_pos={0 0}, upload=5911]: Done uploading piece. err: <nil> Jul 27 04:07:18.423748140 (6h15m34.204681511s) [noise=true , address=64.49.13.140:4508 , piece_id=MZ52U75VOIK6Q7QFXMMYRRG7JXFWDFU2JWQFYPERUCMH3BUX6XPQ, seg_pos={0 0}, upload=5911]: Done uploading piece. err: <nil> Jul 27 04:08:16.739940381 (6h16m32.52087375s) [noise=true , address=104.36.82.54:4172 , piece_id=ZNA53MWDRWPUH6FUR4DJERBDT2IOYGA3IYUADKXXIOLQJLQOXE7Q, seg_pos={0 0}, upload=5911]: Done uploading piece. err: <nil> Jul 27 04:08:17.177295397 (6h16m32.958228766s) [noise=true , address=104.36.82.8:4002 , piece_id=6XSPWQC3US355PS6IQ4K3IYNQZH5MAT4IVIPGSYRUM3XYPSCWE6Q, seg_pos={0 0}, upload=5911]: Done uploading piece. err: <nil> Jul 27 04:08:17.534296584 (6h16m33.315229953s) [noise=true , address=104.36.82.36:4650 , piece_id=BKJULWGPESBINNTSUYP6E4KZRXKPACVU5I7DOSZBO4YMK2V26HEQ, seg_pos={0 0}, upload=5911]: Done uploading piece. err: <nil> Jul 27 04:08:17.677851566 (6h16m33.458784943s) [noise=true , address=104.36.82.14:4248 , piece_id=LDSGCUM7Q2VX5CDFI4JRGEEG7D5H5Q5Z5WUWKU277VN2JFRX6UGA, seg_pos={0 0}, upload=5911]: Done uploading piece. err: ecclient: upload failed (node:1u6ir2PLqDa9xGm64cuqq1RqY4qeBa1fqCo9VghspBf54XEXqn, address:104.36.82.14:4248): protocol: rpc: dial tcp 104.36.82.14:4248: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 04:08:17.678115170 (6h16m33.459048549s) [noise=true , address=104.36.82.46:4496 , piece_id=GWNQR5WRGRRQBLSGWJFD3O3UNCMNE62JQNPIV5SYKBUIL3BQKA5A, seg_pos={0 0}, upload=5911]: Done uploading piece. err: ecclient: upload failed (node:12V8XhSWkyGUVYFR8MykifFAP1YAfGCwzqucDcMMfzw3xsVW77Z, address:104.36.82.46:4496): protocol: read tcp 159.223.143.39:52800->104.36.82.46:4496: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 04:08:17.678144409 (6h16m33.459077789s) [noise=true , address=104.36.82.44:4901 , piece_id=N4YLCLR2QZSULUL2XWONEKBFUMNOU7KUX2QEOK4SSDFSCI7IL6LA, seg_pos={0 0}, upload=5911]: Done uploading piece. err: ecclient: upload failed (node:1FHE84zvBb1MKwB1Ht7puTafgQxDEvPt4mYqkYVSSufAMpWQA1, address:104.36.82.44:4901): protocol: rpc: dial tcp 104.36.82.44:4901: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 04:08:17.678229622 (6h16m33.459162988s) [noise=true , address=104.36.82.44:4893 , piece_id=HUJICRJ6W54O3AZZR6H3KG6E4TGBTNS46O6XVMQM3PFAZE7A2HZA, seg_pos={0 0}, upload=5911]: Done uploading piece. err: ecclient: upload failed (node:17VMFZ2T8jrSJVtchciJ9SnyrR6JstvcMoXrN5Q3zTD1aQU1oJ, address:104.36.82.44:4893): protocol: write tcp 159.223.143.39:35302->104.36.82.44:4893: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 04:08:17.678310218 (6h16m33.459243583s) [noise=true , address=104.36.82.42:4829 , piece_id=ZLWJFP3LGX3YLXBJYIR66DXTYEVDBKXKVXCQSF2PCYCYJBIEJGMA, seg_pos={0 0}, upload=5911]: Done uploading piece. err: ecclient: upload failed (node:1F5T8Vt1x4nXFaSenvGBCYjxhRL1H1sxRvzLjW3CqzwDVhq39F, address:104.36.82.42:4829): protocol: rpc: dial tcp 104.36.82.42:4829: i/o timeout; conn closed: op; piecestore: piecestore close: context deadline exceeded Jul 27 04:08:17.678412989 (6h16m33.459346365s) [noise=true , address=104.36.82.38:4722 , piece_id=B2MBQY5CTMEI6WSE7FTLOV5RUJ22SCNG7N2EEJILEATGJ4TIALXA, seg_pos={0 0}, upload=5911]: Done uploading piece. err: ecclient: upload failed (node:12Zw3dtUg9gARz52KhwDLTjcqC3CNm7dpPzpdx7P7PbL1jwe8uc, address:104.36.82.38:4722): protocol: read tcp 159.223.143.39:56048->104.36.82.38:4722: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 04:08:17.678455247 (6h16m33.459388624s) [noise=true , address=104.36.82.47:4524 , piece_id=DONJFTKUQ2XCW6BOU7AVOKJ7XMDMBPGQVUPRANH7X2YS7PJXUKIA, seg_pos={0 0}, upload=5911]: Done uploading piece. err: ecclient: upload failed (node:1xfbCzjSwFDRrR1gPggeN8tNCLbRRWDVzCJ6mqhZsbVSe8d4s4, address:104.36.82.47:4524): protocol: read tcp 159.223.143.39:46694->104.36.82.47:4524: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 04:08:17.678510668 (6h16m33.459444052s) [seg_pos={0 0}, upload=5911]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Jul 27 04:08:17.706442864 (6h16m33.487376232s) [upload=5911]: Done uploading segments... Jul 27 04:08:17.706495329 (6h16m33.487428705s) [upload=5911]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 7.333853641s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................X....................................] duration: 478.170781ms FAIL (2025-07-27 04:41:33 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 12) Duration: 478.203179ms core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [............X...............................................] duration: 1.151277489s FAIL (2025-07-27 04:41:30 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== {0 0}, upload=5911]: Begin upload segment... Jul 27 04:07:17.745307658 (6h15m33.526241043s) [seg_pos={0 0}, upload=5911]: Done begin upload segment. Jul 27 04:07:17.745349241 (6h15m33.526282625s) [noise=true , address=104.36.82.52:4098 , piece_id=M2ETF25QQ53RRGFPUKXJRINIWV7AWXSECEDR2AIM645IED72NURQ, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.745439694 (6h15m33.526373073s) [noise=true , address=104.36.82.56:4244 , piece_id=LQCZ5W54NBHWAVBVYK7HYSSJDC73E6FV4CP4YIM4L5H2N3OLOZXA, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.745972271 (6h15m33.526905654s) [noise=true , address=104.36.82.36:4650 , piece_id=BKJULWGPESBINNTSUYP6E4KZRXKPACVU5I7DOSZBO4YMK2V26HEQ, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.745980243 (6h15m33.526913621s) [noise=true , address=104.36.82.38:4722 , piece_id=B2MBQY5CTMEI6WSE7FTLOV5RUJ22SCNG7N2EEJILEATGJ4TIALXA, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.746403818 (6h15m33.527337198s) [noise=true , address=104.36.82.22:4534 , piece_id=CW4AQT3XUGA4A5OC6NUY4NYOOAMLN5FFJJN5PQBYEON7HPD5LCYQ, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.746455493 (6h15m33.527388875s) [noise=true , address=104.36.82.14:4248 , piece_id=LDSGCUM7Q2VX5CDFI4JRGEEG7D5H5Q5Z5WUWKU277VN2JFRX6UGA, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.746872127 (6h15m33.527805505s) [noise=true , address=104.36.82.40:4954 , piece_id=YLOHTJLRAED66OMEQYPZHNNRI77V6WOR4EHTEZOOEHMGZHJ2O3QQ, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.747123162 (6h15m33.52805654s) [noise=true , address=104.36.82.46:4496 , piece_id=GWNQR5WRGRRQBLSGWJFD3O3UNCMNE62JQNPIV5SYKBUIL3BQKA5A, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.747280678 (6h15m33.528214053s) [noise=true , address=104.36.82.47:4524 , piece_id=DONJFTKUQ2XCW6BOU7AVOKJ7XMDMBPGQVUPRANH7X2YS7PJXUKIA, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.747736400 (6h15m33.528669779s) [noise=true , address=104.36.82.44:4893 , piece_id=HUJICRJ6W54O3AZZR6H3KG6E4TGBTNS46O6XVMQM3PFAZE7A2HZA, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.747755734 (6h15m33.528689109s) [noise=true , address=104.36.82.55:4204 , piece_id=72WISGRHMOAZDV5EDGTBOMLBL4HDOF3CBEMQAAIRGILPC7J5JKAQ, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.748313124 (6h15m33.5292465s) [noise=true , address=104.36.82.58:4287 , piece_id=BVDYUEWU7TVEQDMZXX5SJNS4QICVKMTGFLCUV6P2YHOE7TI25VXA, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.748485217 (6h15m33.529418594s) [noise=true , address=104.36.82.53:4123 , piece_id=NXJRZ5BQSLXSOIOSUJ4MM4AGKHP4UIDUGEJKLUUX7YOGQJA6WWZA, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.748831329 (6h15m33.529764704s) [noise=true , address=104.36.82.42:4829 , piece_id=ZLWJFP3LGX3YLXBJYIR66DXTYEVDBKXKVXCQSF2PCYCYJBIEJGMA, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.749299293 (6h15m33.530232669s) [noise=true , address=104.36.82.8:4002 , piece_id=6XSPWQC3US355PS6IQ4K3IYNQZH5MAT4IVIPGSYRUM3XYPSCWE6Q, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.749790974 (6h15m33.53072436s) [noise=true , address=104.36.82.44:4901 , piece_id=N4YLCLR2QZSULUL2XWONEKBFUMNOU7KUX2QEOK4SSDFSCI7IL6LA, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.750478221 (6h15m33.531411597s) [noise=true , address=104.36.82.54:4172 , piece_id=ZNA53MWDRWPUH6FUR4DJERBDT2IOYGA3IYUADKXXIOLQJLQOXE7Q, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.752884255 (6h15m33.533817625s) [noise=true , address=64.49.13.140:4508 , piece_id=MZ52U75VOIK6Q7QFXMMYRRG7JXFWDFU2JWQFYPERUCMH3BUX6XPQ, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.879839555 (6h15m33.660772917s) [upload=5911]: Next returned nil segment Jul 27 04:07:17.879861389 (6h15m33.660794765s) [upload=5911]: Waiting for error group managing segments... Jul 27 04:07:17.888652438 (6h15m33.669585811s) [noise=true , address=104.36.82.58:4287 , piece_id=BVDYUEWU7TVEQDMZXX5SJNS4QICVKMTGFLCUV6P2YHOE7TI25VXA, seg_pos={0 0}, upload=5911]: Done uploading piece. err: <nil> Jul 27 04:07:17.952192045 (6h15m33.733125429s) [noise=true , address=104.36.82.53:4123 , piece_id=NXJRZ5BQSLXSOIOSUJ4MM4AGKHP4UIDUGEJKLUUX7YOGQJA6WWZA, seg_pos={0 0}, upload=5911]: Done uploading piece. err: <nil> Jul 27 04:07:17.981907539 (6h15m33.762840911s) [noise=true , address=104.36.82.56:4244 , piece_id=LQCZ5W54NBHWAVBVYK7HYSSJDC73E6FV4CP4YIM4L5H2N3OLOZXA, seg_pos={0 0}, upload=5911]: Done uploading piece. err: <nil> Jul 27 04:07:17.982370496 (6h15m33.7633039s) [noise=true , address=104.36.82.55:4204 , piece_id=72WISGRHMOAZDV5EDGTBOMLBL4HDOF3CBEMQAAIRGILPC7J5JKAQ, seg_pos={0 0}, upload=5911]: Done uploading piece. err: <nil> Jul 27 04:07:18.009548506 (6h15m33.790481886s) [noise=true , address=104.36.82.52:4098 , piece_id=M2ETF25QQ53RRGFPUKXJRINIWV7AWXSECEDR2AIM645IED72NURQ, seg_pos={0 0}, upload=5911]: Done uploading piece. err: <nil> Jul 27 04:07:18.023770598 (6h15m33.804703983s) [noise=true , address=104.36.82.40:4954 , piece_id=YLOHTJLRAED66OMEQYPZHNNRI77V6WOR4EHTEZOOEHMGZHJ2O3QQ, seg_pos={0 0}, upload=5911]: Done uploading piece. err: <nil> Jul 27 04:07:18.139745581 (6h15m33.92067895s) [noise=true , address=104.36.82.22:4534 , piece_id=CW4AQT3XUGA4A5OC6NUY4NYOOAMLN5FFJJN5PQBYEON7HPD5LCYQ, seg_pos={0 0}, upload=5911]: Done uploading piece. err: <nil> Jul 27 04:07:18.423748140 (6h15m34.204681511s) [noise=true , address=64.49.13.140:4508 , piece_id=MZ52U75VOIK6Q7QFXMMYRRG7JXFWDFU2JWQFYPERUCMH3BUX6XPQ, seg_pos={0 0}, upload=5911]: Done uploading piece. err: <nil> Jul 27 04:08:16.739940381 (6h16m32.52087375s) [noise=true , address=104.36.82.54:4172 , piece_id=ZNA53MWDRWPUH6FUR4DJERBDT2IOYGA3IYUADKXXIOLQJLQOXE7Q, seg_pos={0 0}, upload=5911]: Done uploading piece. err: <nil> Jul 27 04:08:17.177295397 (6h16m32.958228766s) [noise=true , address=104.36.82.8:4002 , piece_id=6XSPWQC3US355PS6IQ4K3IYNQZH5MAT4IVIPGSYRUM3XYPSCWE6Q, seg_pos={0 0}, upload=5911]: Done uploading piece. err: <nil> Jul 27 04:08:17.534296584 (6h16m33.315229953s) [noise=true , address=104.36.82.36:4650 , piece_id=BKJULWGPESBINNTSUYP6E4KZRXKPACVU5I7DOSZBO4YMK2V26HEQ, seg_pos={0 0}, upload=5911]: Done uploading piece. err: <nil> Jul 27 04:08:17.677851566 (6h16m33.458784943s) [noise=true , address=104.36.82.14:4248 , piece_id=LDSGCUM7Q2VX5CDFI4JRGEEG7D5H5Q5Z5WUWKU277VN2JFRX6UGA, seg_pos={0 0}, upload=5911]: Done uploading piece. err: ecclient: upload failed (node:1u6ir2PLqDa9xGm64cuqq1RqY4qeBa1fqCo9VghspBf54XEXqn, address:104.36.82.14:4248): protocol: rpc: dial tcp 104.36.82.14:4248: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 04:08:17.678115170 (6h16m33.459048549s) [noise=true , address=104.36.82.46:4496 , piece_id=GWNQR5WRGRRQBLSGWJFD3O3UNCMNE62JQNPIV5SYKBUIL3BQKA5A, seg_pos={0 0}, upload=5911]: Done uploading piece. err: ecclient: upload failed (node:12V8XhSWkyGUVYFR8MykifFAP1YAfGCwzqucDcMMfzw3xsVW77Z, address:104.36.82.46:4496): protocol: read tcp 159.223.143.39:52800->104.36.82.46:4496: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 04:08:17.678144409 (6h16m33.459077789s) [noise=true , address=104.36.82.44:4901 , piece_id=N4YLCLR2QZSULUL2XWONEKBFUMNOU7KUX2QEOK4SSDFSCI7IL6LA, seg_pos={0 0}, upload=5911]: Done uploading piece. err: ecclient: upload failed (node:1FHE84zvBb1MKwB1Ht7puTafgQxDEvPt4mYqkYVSSufAMpWQA1, address:104.36.82.44:4901): protocol: rpc: dial tcp 104.36.82.44:4901: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 04:08:17.678229622 (6h16m33.459162988s) [noise=true , address=104.36.82.44:4893 , piece_id=HUJICRJ6W54O3AZZR6H3KG6E4TGBTNS46O6XVMQM3PFAZE7A2HZA, seg_pos={0 0}, upload=5911]: Done uploading piece. err: ecclient: upload failed (node:17VMFZ2T8jrSJVtchciJ9SnyrR6JstvcMoXrN5Q3zTD1aQU1oJ, address:104.36.82.44:4893): protocol: write tcp 159.223.143.39:35302->104.36.82.44:4893: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 04:08:17.678310218 (6h16m33.459243583s) [noise=true , address=104.36.82.42:4829 , piece_id=ZLWJFP3LGX3YLXBJYIR66DXTYEVDBKXKVXCQSF2PCYCYJBIEJGMA, seg_pos={0 0}, upload=5911]: Done uploading piece. err: ecclient: upload failed (node:1F5T8Vt1x4nXFaSenvGBCYjxhRL1H1sxRvzLjW3CqzwDVhq39F, address:104.36.82.42:4829): protocol: rpc: dial tcp 104.36.82.42:4829: i/o timeout; conn closed: op; piecestore: piecestore close: context deadline exceeded Jul 27 04:08:17.678412989 (6h16m33.459346365s) [noise=true , address=104.36.82.38:4722 , piece_id=B2MBQY5CTMEI6WSE7FTLOV5RUJ22SCNG7N2EEJILEATGJ4TIALXA, seg_pos={0 0}, upload=5911]: Done uploading piece. err: ecclient: upload failed (node:12Zw3dtUg9gARz52KhwDLTjcqC3CNm7dpPzpdx7P7PbL1jwe8uc, address:104.36.82.38:4722): protocol: read tcp 159.223.143.39:56048->104.36.82.38:4722: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 04:08:17.678455247 (6h16m33.459388624s) [noise=true , address=104.36.82.47:4524 , piece_id=DONJFTKUQ2XCW6BOU7AVOKJ7XMDMBPGQVUPRANH7X2YS7PJXUKIA, seg_pos={0 0}, upload=5911]: Done uploading piece. err: ecclient: upload failed (node:1xfbCzjSwFDRrR1gPggeN8tNCLbRRWDVzCJ6mqhZsbVSe8d4s4, address:104.36.82.47:4524): protocol: read tcp 159.223.143.39:46694->104.36.82.47:4524: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 04:08:17.678510668 (6h16m33.459444052s) [seg_pos={0 0}, upload=5911]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Jul 27 04:08:17.706442864 (6h16m33.487376232s) [upload=5911]: Done uploading segments... Jul 27 04:08:17.706495329 (6h16m33.487428705s) [upload=5911]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 1.151311167s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........X..................................................] duration: 404.050766ms FAIL (2025-07-27 04:41:29 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== ece_id=NYOFDEZ7CV4PHGQ6JZMEEVAYGJRPYG6IBW7MYJO42HMWOJM7PHZA, seg_pos={0 0}, upload=5879]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12crcXpNe9NgeXTsUpooLSBgrSUFbRxbSbanLa39w2Hr5mGFww): protocol: read tcp 159.223.143.39:53630->104.36.82.5:4840: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 04:05:32.508904485 (6h13m48.289837866s) [seg_pos={0 0}, upload=5879]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 27 04:05:32.509028070 (6h13m48.289961457s) [upload=5879]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 27 04:05:32.509034645 (6h13m48.289968129s) [upload=5879]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 27 04:05:32.581834839 (6h13m48.362768225s) [upload=5879]: Done uploading segments... Jul 27 04:05:32.581875919 (6h13m48.362809298s) [upload=5879]: Upload finished. err: <nil> Jul 27 04:06:23.829155331 (6h14m39.610088696s) [upload=5895]: Starting upload Jul 27 04:06:23.829225021 (6h14m39.610158398s) [upload=5895]: Done starting upload Jul 27 04:06:23.829294606 (6h14m39.61022799s) [upload=5895]: Uploading segments... Jul 27 04:06:23.829577656 (6h14m39.610511053s) [upload=5895]: Got next segment. Inline: false Jul 27 04:06:23.829649664 (6h14m39.610583027s) [upload=5895]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 27 04:06:23.829657065 (6h14m39.61059043s) [upload=5895]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 27 04:06:23.861405530 (6h14m39.642338913s) [seg_pos={0 0}, upload=5895]: Begin upload segment... Jul 27 04:06:23.861476351 (6h14m39.642409713s) [seg_pos={0 0}, upload=5895]: Done begin upload segment. Jul 27 04:06:23.861509120 (6h14m39.642442506s) [noise=true , address=77.90.31.19:4843 , piece_id=YML5RDQIK432Z4FSE4DKXIV5U47LQBEZRFMAVLANUSGFUQR23UWQ, seg_pos={0 0}, upload=5895]: Uploading piece... Jul 27 04:06:23.861953723 (6h14m39.6428871s) [noise=true , address=45.91.116.10:4081 , piece_id=XGDTSEZOUNRTEVWHTO3DHTZMH2VZZOPJPEKLCFAHTXESLLYABHSA, seg_pos={0 0}, upload=5895]: Uploading piece... Jul 27 04:06:23.862060983 (6h14m39.642994358s) [noise=true , address=45.91.116.9:4262 , piece_id=M5ETD45HH2XZD6HDLTJ3JO2BOOIROXFWKVPTN42PSQEWG4F7KAAQ, seg_pos={0 0}, upload=5895]: Uploading piece... Jul 27 04:06:23.862759464 (6h14m39.64369284s) [noise=true , address=104.36.82.4:4804 , piece_id=IU6FDNVB5AUPA7MJ6RSXUTLBETPHOFGNJ7YGWBFIZSQFSO6W6BBA, seg_pos={0 0}, upload=5895]: Uploading piece... Jul 27 04:06:23.862835086 (6h14m39.643768461s) [noise=true , address=104.36.82.6:4873 , piece_id=YP4VJORZZUL5EVCH3P7LD3LNVMHB7CWRTKHGIA7UROXE44N6MMZA, seg_pos={0 0}, upload=5895]: Uploading piece... Jul 27 04:06:23.863593880 (6h14m39.644527258s) [noise=true , address=77.90.31.15:4697 , piece_id=Q76TJDSZPSYZSJQNFVOI462U6DFILMH2RV3LQISODXSOCXGRKW6Q, seg_pos={0 0}, upload=5895]: Uploading piece... Jul 27 04:06:23.863656891 (6h14m39.644590266s) [noise=true , address=45.91.116.7:4011 , piece_id=XMH6MCCHZBBRQWHJFWLRGYHLQD4KUGSGL7MMZSKA5TZJCND3DUDA, seg_pos={0 0}, upload=5895]: Uploading piece... Jul 27 04:06:23.864408612 (6h14m39.645341987s) [noise=true , address=45.91.116.6:4973 , piece_id=J3VULPU24CIM64VEVE3UHO62HPN6TJXGZU6BCJIS5JQJKHJK2LIQ, seg_pos={0 0}, upload=5895]: Uploading piece... Jul 27 04:06:23.864521549 (6h14m39.64545491s) [noise=true , address=104.36.82.10:4079 , piece_id=2SH5D6JOMLFR7VSN7ZY5CLMR5RHIRFYQ7Z5I2JIYTJQM43G4O6OQ, seg_pos={0 0}, upload=5895]: Uploading piece... Jul 27 04:06:23.865518491 (6h14m39.646451872s) [noise=true , address=77.90.31.17:4771 , piece_id=U2LF22GKCMKRZN6EWCF26ITDZQHBFZXCGSQXT33MCX6P57IGI35Q, seg_pos={0 0}, upload=5895]: Uploading piece... Jul 27 04:06:23.865654950 (6h14m39.646588325s) [noise=true , address=45.91.116.8:4048 , piece_id=5REWMVVTBVLDQXQ5KAEHLY4VJL5SGF7ONQWDQQ3N2FDU3OAO6AZA, seg_pos={0 0}, upload=5895]: Uploading piece... Jul 27 04:06:23.866190892 (6h14m39.647124271s) [noise=true , address=104.36.82.12:4151 , piece_id=E2TEUTQBD4Y4AUHSCPMCIFEXBSY7S5HJWBKNKWWTUWEW2BLAAE6Q, seg_pos={0 0}, upload=5895]: Uploading piece... Jul 27 04:06:23.866412590 (6h14m39.647345968s) [noise=true , address=104.36.82.11:4115 , piece_id=YXMNVT6ZHGESGQ2BXKZUVR7U2CNNAI2PO7ZTMDG5UIOV7HTDDUTA, seg_pos={0 0}, upload=5895]: Uploading piece... Jul 27 04:06:23.866715972 (6h14m39.647649359s) [noise=true , address=45.91.116.13:4153 , piece_id=H4535KOCG3F4RKA6AKKMTJSOM6Q6PGRC6M7FR3UIS3BORTSP5UQA, seg_pos={0 0}, upload=5895]: Uploading piece... Jul 27 04:06:23.866998965 (6h14m39.64793234s) [noise=true , address=104.36.82.13:4186 , piece_id=TBLDKALGJYBGR5NLCOCKJ4QKEFNIJXCKAFUBD7VR3UMNWEMKMKHQ, seg_pos={0 0}, upload=5895]: Uploading piece... Jul 27 04:06:23.867230593 (6h14m39.648163974s) [noise=true , address=104.36.82.16:4293 , piece_id=AHH6PRFUSGTDGB7H52DRITARA4IF6GFLVSRYS2Y6EPJCZSKUOYWQ, seg_pos={0 0}, upload=5895]: Uploading piece... Jul 27 04:06:23.949422808 (6h14m39.730356187s) [upload=5895]: Next returned nil segment Jul 27 04:06:23.949442576 (6h14m39.73037596s) [upload=5895]: Waiting for error group managing segments... Jul 27 04:06:23.953497426 (6h14m39.734430802s) [noise=true , address=45.91.116.10:4081 , piece_id=XGDTSEZOUNRTEVWHTO3DHTZMH2VZZOPJPEKLCFAHTXESLLYABHSA, seg_pos={0 0}, upload=5895]: Done uploading piece. err: <nil> Jul 27 04:06:23.953531796 (6h14m39.734465159s) [noise=true , address=45.91.116.6:4973 , piece_id=J3VULPU24CIM64VEVE3UHO62HPN6TJXGZU6BCJIS5JQJKHJK2LIQ, seg_pos={0 0}, upload=5895]: Done uploading piece. err: <nil> Jul 27 04:06:23.954068660 (6h14m39.735002029s) [noise=true , address=45.91.116.9:4262 , piece_id=M5ETD45HH2XZD6HDLTJ3JO2BOOIROXFWKVPTN42PSQEWG4F7KAAQ, seg_pos={0 0}, upload=5895]: Done uploading piece. err: <nil> Jul 27 04:06:23.954194339 (6h14m39.735127707s) [noise=true , address=45.91.116.8:4048 , piece_id=5REWMVVTBVLDQXQ5KAEHLY4VJL5SGF7ONQWDQQ3N2FDU3OAO6AZA, seg_pos={0 0}, upload=5895]: Done uploading piece. err: <nil> Jul 27 04:06:23.954601730 (6h14m39.735535112s) [noise=true , address=45.91.116.7:4011 , piece_id=XMH6MCCHZBBRQWHJFWLRGYHLQD4KUGSGL7MMZSKA5TZJCND3DUDA, seg_pos={0 0}, upload=5895]: Done uploading piece. err: <nil> Jul 27 04:06:23.954755872 (6h14m39.735689237s) [noise=true , address=45.91.116.13:4153 , piece_id=H4535KOCG3F4RKA6AKKMTJSOM6Q6PGRC6M7FR3UIS3BORTSP5UQA, seg_pos={0 0}, upload=5895]: Done uploading piece. err: <nil> Jul 27 04:06:23.955611883 (6h14m39.73654526s) [noise=true , address=77.90.31.19:4843 , piece_id=YML5RDQIK432Z4FSE4DKXIV5U47LQBEZRFMAVLANUSGFUQR23UWQ, seg_pos={0 0}, upload=5895]: Done uploading piece. err: <nil> Jul 27 04:06:23.956373391 (6h14m39.737306756s) [noise=true , address=77.90.31.15:4697 , piece_id=Q76TJDSZPSYZSJQNFVOI462U6DFILMH2RV3LQISODXSOCXGRKW6Q, seg_pos={0 0}, upload=5895]: Done uploading piece. err: <nil> Jul 27 04:06:23.958659122 (6h14m39.73959249s) [noise=true , address=77.90.31.17:4771 , piece_id=U2LF22GKCMKRZN6EWCF26ITDZQHBFZXCGSQXT33MCX6P57IGI35Q, seg_pos={0 0}, upload=5895]: Done uploading piece. err: <nil> Jul 27 04:06:35.058944013 (6h14m50.83987739s) [noise=true , address=104.36.82.16:4293 , piece_id=AHH6PRFUSGTDGB7H52DRITARA4IF6GFLVSRYS2Y6EPJCZSKUOYWQ, seg_pos={0 0}, upload=5895]: Done uploading piece. err: <nil> Jul 27 04:06:41.103204002 (6h14m56.884137386s) [noise=true , address=104.36.82.4:4804 , piece_id=IU6FDNVB5AUPA7MJ6RSXUTLBETPHOFGNJ7YGWBFIZSQFSO6W6BBA, seg_pos={0 0}, upload=5895]: Done uploading piece. err: <nil> Jul 27 04:06:43.191702285 (6h14m58.972635651s) [noise=true , address=104.36.82.6:4873 , piece_id=YP4VJORZZUL5EVCH3P7LD3LNVMHB7CWRTKHGIA7UROXE44N6MMZA, seg_pos={0 0}, upload=5895]: Done uploading piece. err: <nil> Jul 27 04:06:53.777973493 (6h15m9.558906861s) [noise=true , address=104.36.82.11:4115 , piece_id=YXMNVT6ZHGESGQ2BXKZUVR7U2CNNAI2PO7ZTMDG5UIOV7HTDDUTA, seg_pos={0 0}, upload=5895]: Done uploading piece. err: ecclient: upload failed (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR, address:104.36.82.11:4115): protocol: write tcp 159.223.143.39:34210->104.36.82.11:4115: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 04:06:53.778019532 (6h15m9.558952909s) [noise=true , address=104.36.82.12:4151 , piece_id=E2TEUTQBD4Y4AUHSCPMCIFEXBSY7S5HJWBKNKWWTUWEW2BLAAE6Q, seg_pos={0 0}, upload=5895]: Done uploading piece. err: ecclient: upload failed (node:1w6pwZMdeQKjt92qx6WbTWU5VPYJto5gS87f6iKvCBZH6LfyeB, address:104.36.82.12:4151): protocol: rpc: dial tcp 104.36.82.12:4151: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 04:06:53.778043653 (6h15m9.558977021s) [noise=true , address=104.36.82.13:4186 , piece_id=TBLDKALGJYBGR5NLCOCKJ4QKEFNIJXCKAFUBD7VR3UMNWEMKMKHQ, seg_pos={0 0}, upload=5895]: Done uploading piece. err: ecclient: upload failed (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht, address:104.36.82.13:4186): protocol: read tcp 159.223.143.39:34340->104.36.82.13:4186: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 04:06:53.778095228 (6h15m9.559028593s) [noise=true , address=104.36.82.10:4079 , piece_id=2SH5D6JOMLFR7VSN7ZY5CLMR5RHIRFYQ7Z5I2JIYTJQM43G4O6OQ, seg_pos={0 0}, upload=5895]: Done uploading piece. err: ecclient: upload failed (node:12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud, address:104.36.82.10:4079): protocol: write tcp 159.223.143.39:53840->104.36.82.10:4079: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 04:06:53.778142167 (6h15m9.559075532s) [seg_pos={0 0}, upload=5895]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 27 04:06:53.837032161 (6h15m9.617965543s) [upload=5895]: Done uploading segments... Jul 27 04:06:53.837055996 (6h15m9.617989366s) [upload=5895]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 404.080108ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........X.................................................] duration: 714.301908ms FAIL (2025-07-27 04:41:14 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== ece_id=NYOFDEZ7CV4PHGQ6JZMEEVAYGJRPYG6IBW7MYJO42HMWOJM7PHZA, seg_pos={0 0}, upload=5879]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12crcXpNe9NgeXTsUpooLSBgrSUFbRxbSbanLa39w2Hr5mGFww): protocol: read tcp 159.223.143.39:53630->104.36.82.5:4840: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 04:05:32.508904485 (6h13m48.289837866s) [seg_pos={0 0}, upload=5879]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 27 04:05:32.509028070 (6h13m48.289961457s) [upload=5879]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 27 04:05:32.509034645 (6h13m48.289968129s) [upload=5879]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 27 04:05:32.581834839 (6h13m48.362768225s) [upload=5879]: Done uploading segments... Jul 27 04:05:32.581875919 (6h13m48.362809298s) [upload=5879]: Upload finished. err: <nil> Jul 27 04:06:23.829155331 (6h14m39.610088696s) [upload=5895]: Starting upload Jul 27 04:06:23.829225021 (6h14m39.610158398s) [upload=5895]: Done starting upload Jul 27 04:06:23.829294606 (6h14m39.61022799s) [upload=5895]: Uploading segments... Jul 27 04:06:23.829577656 (6h14m39.610511053s) [upload=5895]: Got next segment. Inline: false Jul 27 04:06:23.829649664 (6h14m39.610583027s) [upload=5895]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 27 04:06:23.829657065 (6h14m39.61059043s) [upload=5895]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 27 04:06:23.861405530 (6h14m39.642338913s) [seg_pos={0 0}, upload=5895]: Begin upload segment... Jul 27 04:06:23.861476351 (6h14m39.642409713s) [seg_pos={0 0}, upload=5895]: Done begin upload segment. Jul 27 04:06:23.861509120 (6h14m39.642442506s) [noise=true , address=77.90.31.19:4843 , piece_id=YML5RDQIK432Z4FSE4DKXIV5U47LQBEZRFMAVLANUSGFUQR23UWQ, seg_pos={0 0}, upload=5895]: Uploading piece... Jul 27 04:06:23.861953723 (6h14m39.6428871s) [noise=true , address=45.91.116.10:4081 , piece_id=XGDTSEZOUNRTEVWHTO3DHTZMH2VZZOPJPEKLCFAHTXESLLYABHSA, seg_pos={0 0}, upload=5895]: Uploading piece... Jul 27 04:06:23.862060983 (6h14m39.642994358s) [noise=true , address=45.91.116.9:4262 , piece_id=M5ETD45HH2XZD6HDLTJ3JO2BOOIROXFWKVPTN42PSQEWG4F7KAAQ, seg_pos={0 0}, upload=5895]: Uploading piece... Jul 27 04:06:23.862759464 (6h14m39.64369284s) [noise=true , address=104.36.82.4:4804 , piece_id=IU6FDNVB5AUPA7MJ6RSXUTLBETPHOFGNJ7YGWBFIZSQFSO6W6BBA, seg_pos={0 0}, upload=5895]: Uploading piece... Jul 27 04:06:23.862835086 (6h14m39.643768461s) [noise=true , address=104.36.82.6:4873 , piece_id=YP4VJORZZUL5EVCH3P7LD3LNVMHB7CWRTKHGIA7UROXE44N6MMZA, seg_pos={0 0}, upload=5895]: Uploading piece... Jul 27 04:06:23.863593880 (6h14m39.644527258s) [noise=true , address=77.90.31.15:4697 , piece_id=Q76TJDSZPSYZSJQNFVOI462U6DFILMH2RV3LQISODXSOCXGRKW6Q, seg_pos={0 0}, upload=5895]: Uploading piece... Jul 27 04:06:23.863656891 (6h14m39.644590266s) [noise=true , address=45.91.116.7:4011 , piece_id=XMH6MCCHZBBRQWHJFWLRGYHLQD4KUGSGL7MMZSKA5TZJCND3DUDA, seg_pos={0 0}, upload=5895]: Uploading piece... Jul 27 04:06:23.864408612 (6h14m39.645341987s) [noise=true , address=45.91.116.6:4973 , piece_id=J3VULPU24CIM64VEVE3UHO62HPN6TJXGZU6BCJIS5JQJKHJK2LIQ, seg_pos={0 0}, upload=5895]: Uploading piece... Jul 27 04:06:23.864521549 (6h14m39.64545491s) [noise=true , address=104.36.82.10:4079 , piece_id=2SH5D6JOMLFR7VSN7ZY5CLMR5RHIRFYQ7Z5I2JIYTJQM43G4O6OQ, seg_pos={0 0}, upload=5895]: Uploading piece... Jul 27 04:06:23.865518491 (6h14m39.646451872s) [noise=true , address=77.90.31.17:4771 , piece_id=U2LF22GKCMKRZN6EWCF26ITDZQHBFZXCGSQXT33MCX6P57IGI35Q, seg_pos={0 0}, upload=5895]: Uploading piece... Jul 27 04:06:23.865654950 (6h14m39.646588325s) [noise=true , address=45.91.116.8:4048 , piece_id=5REWMVVTBVLDQXQ5KAEHLY4VJL5SGF7ONQWDQQ3N2FDU3OAO6AZA, seg_pos={0 0}, upload=5895]: Uploading piece... Jul 27 04:06:23.866190892 (6h14m39.647124271s) [noise=true , address=104.36.82.12:4151 , piece_id=E2TEUTQBD4Y4AUHSCPMCIFEXBSY7S5HJWBKNKWWTUWEW2BLAAE6Q, seg_pos={0 0}, upload=5895]: Uploading piece... Jul 27 04:06:23.866412590 (6h14m39.647345968s) [noise=true , address=104.36.82.11:4115 , piece_id=YXMNVT6ZHGESGQ2BXKZUVR7U2CNNAI2PO7ZTMDG5UIOV7HTDDUTA, seg_pos={0 0}, upload=5895]: Uploading piece... Jul 27 04:06:23.866715972 (6h14m39.647649359s) [noise=true , address=45.91.116.13:4153 , piece_id=H4535KOCG3F4RKA6AKKMTJSOM6Q6PGRC6M7FR3UIS3BORTSP5UQA, seg_pos={0 0}, upload=5895]: Uploading piece... Jul 27 04:06:23.866998965 (6h14m39.64793234s) [noise=true , address=104.36.82.13:4186 , piece_id=TBLDKALGJYBGR5NLCOCKJ4QKEFNIJXCKAFUBD7VR3UMNWEMKMKHQ, seg_pos={0 0}, upload=5895]: Uploading piece... Jul 27 04:06:23.867230593 (6h14m39.648163974s) [noise=true , address=104.36.82.16:4293 , piece_id=AHH6PRFUSGTDGB7H52DRITARA4IF6GFLVSRYS2Y6EPJCZSKUOYWQ, seg_pos={0 0}, upload=5895]: Uploading piece... Jul 27 04:06:23.949422808 (6h14m39.730356187s) [upload=5895]: Next returned nil segment Jul 27 04:06:23.949442576 (6h14m39.73037596s) [upload=5895]: Waiting for error group managing segments... Jul 27 04:06:23.953497426 (6h14m39.734430802s) [noise=true , address=45.91.116.10:4081 , piece_id=XGDTSEZOUNRTEVWHTO3DHTZMH2VZZOPJPEKLCFAHTXESLLYABHSA, seg_pos={0 0}, upload=5895]: Done uploading piece. err: <nil> Jul 27 04:06:23.953531796 (6h14m39.734465159s) [noise=true , address=45.91.116.6:4973 , piece_id=J3VULPU24CIM64VEVE3UHO62HPN6TJXGZU6BCJIS5JQJKHJK2LIQ, seg_pos={0 0}, upload=5895]: Done uploading piece. err: <nil> Jul 27 04:06:23.954068660 (6h14m39.735002029s) [noise=true , address=45.91.116.9:4262 , piece_id=M5ETD45HH2XZD6HDLTJ3JO2BOOIROXFWKVPTN42PSQEWG4F7KAAQ, seg_pos={0 0}, upload=5895]: Done uploading piece. err: <nil> Jul 27 04:06:23.954194339 (6h14m39.735127707s) [noise=true , address=45.91.116.8:4048 , piece_id=5REWMVVTBVLDQXQ5KAEHLY4VJL5SGF7ONQWDQQ3N2FDU3OAO6AZA, seg_pos={0 0}, upload=5895]: Done uploading piece. err: <nil> Jul 27 04:06:23.954601730 (6h14m39.735535112s) [noise=true , address=45.91.116.7:4011 , piece_id=XMH6MCCHZBBRQWHJFWLRGYHLQD4KUGSGL7MMZSKA5TZJCND3DUDA, seg_pos={0 0}, upload=5895]: Done uploading piece. err: <nil> Jul 27 04:06:23.954755872 (6h14m39.735689237s) [noise=true , address=45.91.116.13:4153 , piece_id=H4535KOCG3F4RKA6AKKMTJSOM6Q6PGRC6M7FR3UIS3BORTSP5UQA, seg_pos={0 0}, upload=5895]: Done uploading piece. err: <nil> Jul 27 04:06:23.955611883 (6h14m39.73654526s) [noise=true , address=77.90.31.19:4843 , piece_id=YML5RDQIK432Z4FSE4DKXIV5U47LQBEZRFMAVLANUSGFUQR23UWQ, seg_pos={0 0}, upload=5895]: Done uploading piece. err: <nil> Jul 27 04:06:23.956373391 (6h14m39.737306756s) [noise=true , address=77.90.31.15:4697 , piece_id=Q76TJDSZPSYZSJQNFVOI462U6DFILMH2RV3LQISODXSOCXGRKW6Q, seg_pos={0 0}, upload=5895]: Done uploading piece. err: <nil> Jul 27 04:06:23.958659122 (6h14m39.73959249s) [noise=true , address=77.90.31.17:4771 , piece_id=U2LF22GKCMKRZN6EWCF26ITDZQHBFZXCGSQXT33MCX6P57IGI35Q, seg_pos={0 0}, upload=5895]: Done uploading piece. err: <nil> Jul 27 04:06:35.058944013 (6h14m50.83987739s) [noise=true , address=104.36.82.16:4293 , piece_id=AHH6PRFUSGTDGB7H52DRITARA4IF6GFLVSRYS2Y6EPJCZSKUOYWQ, seg_pos={0 0}, upload=5895]: Done uploading piece. err: <nil> Jul 27 04:06:41.103204002 (6h14m56.884137386s) [noise=true , address=104.36.82.4:4804 , piece_id=IU6FDNVB5AUPA7MJ6RSXUTLBETPHOFGNJ7YGWBFIZSQFSO6W6BBA, seg_pos={0 0}, upload=5895]: Done uploading piece. err: <nil> Jul 27 04:06:43.191702285 (6h14m58.972635651s) [noise=true , address=104.36.82.6:4873 , piece_id=YP4VJORZZUL5EVCH3P7LD3LNVMHB7CWRTKHGIA7UROXE44N6MMZA, seg_pos={0 0}, upload=5895]: Done uploading piece. err: <nil> Jul 27 04:06:53.777973493 (6h15m9.558906861s) [noise=true , address=104.36.82.11:4115 , piece_id=YXMNVT6ZHGESGQ2BXKZUVR7U2CNNAI2PO7ZTMDG5UIOV7HTDDUTA, seg_pos={0 0}, upload=5895]: Done uploading piece. err: ecclient: upload failed (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR, address:104.36.82.11:4115): protocol: write tcp 159.223.143.39:34210->104.36.82.11:4115: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 04:06:53.778019532 (6h15m9.558952909s) [noise=true , address=104.36.82.12:4151 , piece_id=E2TEUTQBD4Y4AUHSCPMCIFEXBSY7S5HJWBKNKWWTUWEW2BLAAE6Q, seg_pos={0 0}, upload=5895]: Done uploading piece. err: ecclient: upload failed (node:1w6pwZMdeQKjt92qx6WbTWU5VPYJto5gS87f6iKvCBZH6LfyeB, address:104.36.82.12:4151): protocol: rpc: dial tcp 104.36.82.12:4151: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 04:06:53.778043653 (6h15m9.558977021s) [noise=true , address=104.36.82.13:4186 , piece_id=TBLDKALGJYBGR5NLCOCKJ4QKEFNIJXCKAFUBD7VR3UMNWEMKMKHQ, seg_pos={0 0}, upload=5895]: Done uploading piece. err: ecclient: upload failed (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht, address:104.36.82.13:4186): protocol: read tcp 159.223.143.39:34340->104.36.82.13:4186: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 04:06:53.778095228 (6h15m9.559028593s) [noise=true , address=104.36.82.10:4079 , piece_id=2SH5D6JOMLFR7VSN7ZY5CLMR5RHIRFYQ7Z5I2JIYTJQM43G4O6OQ, seg_pos={0 0}, upload=5895]: Done uploading piece. err: ecclient: upload failed (node:12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud, address:104.36.82.10:4079): protocol: write tcp 159.223.143.39:53840->104.36.82.10:4079: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 04:06:53.778142167 (6h15m9.559075532s) [seg_pos={0 0}, upload=5895]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 27 04:06:53.837032161 (6h15m9.617965543s) [upload=5895]: Done uploading segments... Jul 27 04:06:53.837055996 (6h15m9.617989366s) [upload=5895]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 714.379734ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.............X..............................................] duration: 1.153828503s FAIL (2025-07-27 04:41:11 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== {0 0}, upload=5911]: Begin upload segment... Jul 27 04:07:17.745307658 (6h15m33.526241043s) [seg_pos={0 0}, upload=5911]: Done begin upload segment. Jul 27 04:07:17.745349241 (6h15m33.526282625s) [noise=true , address=104.36.82.52:4098 , piece_id=M2ETF25QQ53RRGFPUKXJRINIWV7AWXSECEDR2AIM645IED72NURQ, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.745439694 (6h15m33.526373073s) [noise=true , address=104.36.82.56:4244 , piece_id=LQCZ5W54NBHWAVBVYK7HYSSJDC73E6FV4CP4YIM4L5H2N3OLOZXA, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.745972271 (6h15m33.526905654s) [noise=true , address=104.36.82.36:4650 , piece_id=BKJULWGPESBINNTSUYP6E4KZRXKPACVU5I7DOSZBO4YMK2V26HEQ, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.745980243 (6h15m33.526913621s) [noise=true , address=104.36.82.38:4722 , piece_id=B2MBQY5CTMEI6WSE7FTLOV5RUJ22SCNG7N2EEJILEATGJ4TIALXA, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.746403818 (6h15m33.527337198s) [noise=true , address=104.36.82.22:4534 , piece_id=CW4AQT3XUGA4A5OC6NUY4NYOOAMLN5FFJJN5PQBYEON7HPD5LCYQ, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.746455493 (6h15m33.527388875s) [noise=true , address=104.36.82.14:4248 , piece_id=LDSGCUM7Q2VX5CDFI4JRGEEG7D5H5Q5Z5WUWKU277VN2JFRX6UGA, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.746872127 (6h15m33.527805505s) [noise=true , address=104.36.82.40:4954 , piece_id=YLOHTJLRAED66OMEQYPZHNNRI77V6WOR4EHTEZOOEHMGZHJ2O3QQ, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.747123162 (6h15m33.52805654s) [noise=true , address=104.36.82.46:4496 , piece_id=GWNQR5WRGRRQBLSGWJFD3O3UNCMNE62JQNPIV5SYKBUIL3BQKA5A, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.747280678 (6h15m33.528214053s) [noise=true , address=104.36.82.47:4524 , piece_id=DONJFTKUQ2XCW6BOU7AVOKJ7XMDMBPGQVUPRANH7X2YS7PJXUKIA, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.747736400 (6h15m33.528669779s) [noise=true , address=104.36.82.44:4893 , piece_id=HUJICRJ6W54O3AZZR6H3KG6E4TGBTNS46O6XVMQM3PFAZE7A2HZA, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.747755734 (6h15m33.528689109s) [noise=true , address=104.36.82.55:4204 , piece_id=72WISGRHMOAZDV5EDGTBOMLBL4HDOF3CBEMQAAIRGILPC7J5JKAQ, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.748313124 (6h15m33.5292465s) [noise=true , address=104.36.82.58:4287 , piece_id=BVDYUEWU7TVEQDMZXX5SJNS4QICVKMTGFLCUV6P2YHOE7TI25VXA, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.748485217 (6h15m33.529418594s) [noise=true , address=104.36.82.53:4123 , piece_id=NXJRZ5BQSLXSOIOSUJ4MM4AGKHP4UIDUGEJKLUUX7YOGQJA6WWZA, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.748831329 (6h15m33.529764704s) [noise=true , address=104.36.82.42:4829 , piece_id=ZLWJFP3LGX3YLXBJYIR66DXTYEVDBKXKVXCQSF2PCYCYJBIEJGMA, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.749299293 (6h15m33.530232669s) [noise=true , address=104.36.82.8:4002 , piece_id=6XSPWQC3US355PS6IQ4K3IYNQZH5MAT4IVIPGSYRUM3XYPSCWE6Q, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.749790974 (6h15m33.53072436s) [noise=true , address=104.36.82.44:4901 , piece_id=N4YLCLR2QZSULUL2XWONEKBFUMNOU7KUX2QEOK4SSDFSCI7IL6LA, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.750478221 (6h15m33.531411597s) [noise=true , address=104.36.82.54:4172 , piece_id=ZNA53MWDRWPUH6FUR4DJERBDT2IOYGA3IYUADKXXIOLQJLQOXE7Q, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.752884255 (6h15m33.533817625s) [noise=true , address=64.49.13.140:4508 , piece_id=MZ52U75VOIK6Q7QFXMMYRRG7JXFWDFU2JWQFYPERUCMH3BUX6XPQ, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.879839555 (6h15m33.660772917s) [upload=5911]: Next returned nil segment Jul 27 04:07:17.879861389 (6h15m33.660794765s) [upload=5911]: Waiting for error group managing segments... Jul 27 04:07:17.888652438 (6h15m33.669585811s) [noise=true , address=104.36.82.58:4287 , piece_id=BVDYUEWU7TVEQDMZXX5SJNS4QICVKMTGFLCUV6P2YHOE7TI25VXA, seg_pos={0 0}, upload=5911]: Done uploading piece. err: <nil> Jul 27 04:07:17.952192045 (6h15m33.733125429s) [noise=true , address=104.36.82.53:4123 , piece_id=NXJRZ5BQSLXSOIOSUJ4MM4AGKHP4UIDUGEJKLUUX7YOGQJA6WWZA, seg_pos={0 0}, upload=5911]: Done uploading piece. err: <nil> Jul 27 04:07:17.981907539 (6h15m33.762840911s) [noise=true , address=104.36.82.56:4244 , piece_id=LQCZ5W54NBHWAVBVYK7HYSSJDC73E6FV4CP4YIM4L5H2N3OLOZXA, seg_pos={0 0}, upload=5911]: Done uploading piece. err: <nil> Jul 27 04:07:17.982370496 (6h15m33.7633039s) [noise=true , address=104.36.82.55:4204 , piece_id=72WISGRHMOAZDV5EDGTBOMLBL4HDOF3CBEMQAAIRGILPC7J5JKAQ, seg_pos={0 0}, upload=5911]: Done uploading piece. err: <nil> Jul 27 04:07:18.009548506 (6h15m33.790481886s) [noise=true , address=104.36.82.52:4098 , piece_id=M2ETF25QQ53RRGFPUKXJRINIWV7AWXSECEDR2AIM645IED72NURQ, seg_pos={0 0}, upload=5911]: Done uploading piece. err: <nil> Jul 27 04:07:18.023770598 (6h15m33.804703983s) [noise=true , address=104.36.82.40:4954 , piece_id=YLOHTJLRAED66OMEQYPZHNNRI77V6WOR4EHTEZOOEHMGZHJ2O3QQ, seg_pos={0 0}, upload=5911]: Done uploading piece. err: <nil> Jul 27 04:07:18.139745581 (6h15m33.92067895s) [noise=true , address=104.36.82.22:4534 , piece_id=CW4AQT3XUGA4A5OC6NUY4NYOOAMLN5FFJJN5PQBYEON7HPD5LCYQ, seg_pos={0 0}, upload=5911]: Done uploading piece. err: <nil> Jul 27 04:07:18.423748140 (6h15m34.204681511s) [noise=true , address=64.49.13.140:4508 , piece_id=MZ52U75VOIK6Q7QFXMMYRRG7JXFWDFU2JWQFYPERUCMH3BUX6XPQ, seg_pos={0 0}, upload=5911]: Done uploading piece. err: <nil> Jul 27 04:08:16.739940381 (6h16m32.52087375s) [noise=true , address=104.36.82.54:4172 , piece_id=ZNA53MWDRWPUH6FUR4DJERBDT2IOYGA3IYUADKXXIOLQJLQOXE7Q, seg_pos={0 0}, upload=5911]: Done uploading piece. err: <nil> Jul 27 04:08:17.177295397 (6h16m32.958228766s) [noise=true , address=104.36.82.8:4002 , piece_id=6XSPWQC3US355PS6IQ4K3IYNQZH5MAT4IVIPGSYRUM3XYPSCWE6Q, seg_pos={0 0}, upload=5911]: Done uploading piece. err: <nil> Jul 27 04:08:17.534296584 (6h16m33.315229953s) [noise=true , address=104.36.82.36:4650 , piece_id=BKJULWGPESBINNTSUYP6E4KZRXKPACVU5I7DOSZBO4YMK2V26HEQ, seg_pos={0 0}, upload=5911]: Done uploading piece. err: <nil> Jul 27 04:08:17.677851566 (6h16m33.458784943s) [noise=true , address=104.36.82.14:4248 , piece_id=LDSGCUM7Q2VX5CDFI4JRGEEG7D5H5Q5Z5WUWKU277VN2JFRX6UGA, seg_pos={0 0}, upload=5911]: Done uploading piece. err: ecclient: upload failed (node:1u6ir2PLqDa9xGm64cuqq1RqY4qeBa1fqCo9VghspBf54XEXqn, address:104.36.82.14:4248): protocol: rpc: dial tcp 104.36.82.14:4248: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 04:08:17.678115170 (6h16m33.459048549s) [noise=true , address=104.36.82.46:4496 , piece_id=GWNQR5WRGRRQBLSGWJFD3O3UNCMNE62JQNPIV5SYKBUIL3BQKA5A, seg_pos={0 0}, upload=5911]: Done uploading piece. err: ecclient: upload failed (node:12V8XhSWkyGUVYFR8MykifFAP1YAfGCwzqucDcMMfzw3xsVW77Z, address:104.36.82.46:4496): protocol: read tcp 159.223.143.39:52800->104.36.82.46:4496: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 04:08:17.678144409 (6h16m33.459077789s) [noise=true , address=104.36.82.44:4901 , piece_id=N4YLCLR2QZSULUL2XWONEKBFUMNOU7KUX2QEOK4SSDFSCI7IL6LA, seg_pos={0 0}, upload=5911]: Done uploading piece. err: ecclient: upload failed (node:1FHE84zvBb1MKwB1Ht7puTafgQxDEvPt4mYqkYVSSufAMpWQA1, address:104.36.82.44:4901): protocol: rpc: dial tcp 104.36.82.44:4901: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 04:08:17.678229622 (6h16m33.459162988s) [noise=true , address=104.36.82.44:4893 , piece_id=HUJICRJ6W54O3AZZR6H3KG6E4TGBTNS46O6XVMQM3PFAZE7A2HZA, seg_pos={0 0}, upload=5911]: Done uploading piece. err: ecclient: upload failed (node:17VMFZ2T8jrSJVtchciJ9SnyrR6JstvcMoXrN5Q3zTD1aQU1oJ, address:104.36.82.44:4893): protocol: write tcp 159.223.143.39:35302->104.36.82.44:4893: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 04:08:17.678310218 (6h16m33.459243583s) [noise=true , address=104.36.82.42:4829 , piece_id=ZLWJFP3LGX3YLXBJYIR66DXTYEVDBKXKVXCQSF2PCYCYJBIEJGMA, seg_pos={0 0}, upload=5911]: Done uploading piece. err: ecclient: upload failed (node:1F5T8Vt1x4nXFaSenvGBCYjxhRL1H1sxRvzLjW3CqzwDVhq39F, address:104.36.82.42:4829): protocol: rpc: dial tcp 104.36.82.42:4829: i/o timeout; conn closed: op; piecestore: piecestore close: context deadline exceeded Jul 27 04:08:17.678412989 (6h16m33.459346365s) [noise=true , address=104.36.82.38:4722 , piece_id=B2MBQY5CTMEI6WSE7FTLOV5RUJ22SCNG7N2EEJILEATGJ4TIALXA, seg_pos={0 0}, upload=5911]: Done uploading piece. err: ecclient: upload failed (node:12Zw3dtUg9gARz52KhwDLTjcqC3CNm7dpPzpdx7P7PbL1jwe8uc, address:104.36.82.38:4722): protocol: read tcp 159.223.143.39:56048->104.36.82.38:4722: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 04:08:17.678455247 (6h16m33.459388624s) [noise=true , address=104.36.82.47:4524 , piece_id=DONJFTKUQ2XCW6BOU7AVOKJ7XMDMBPGQVUPRANH7X2YS7PJXUKIA, seg_pos={0 0}, upload=5911]: Done uploading piece. err: ecclient: upload failed (node:1xfbCzjSwFDRrR1gPggeN8tNCLbRRWDVzCJ6mqhZsbVSe8d4s4, address:104.36.82.47:4524): protocol: read tcp 159.223.143.39:46694->104.36.82.47:4524: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 04:08:17.678510668 (6h16m33.459444052s) [seg_pos={0 0}, upload=5911]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Jul 27 04:08:17.706442864 (6h16m33.487376232s) [upload=5911]: Done uploading segments... Jul 27 04:08:17.706495329 (6h16m33.487428705s) [upload=5911]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 1.153859043s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [..............X.............................................] duration: 6.607513061s FAIL (2025-07-27 04:40:56 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== {0 0}, upload=5911]: Begin upload segment... Jul 27 04:07:17.745307658 (6h15m33.526241043s) [seg_pos={0 0}, upload=5911]: Done begin upload segment. Jul 27 04:07:17.745349241 (6h15m33.526282625s) [noise=true , address=104.36.82.52:4098 , piece_id=M2ETF25QQ53RRGFPUKXJRINIWV7AWXSECEDR2AIM645IED72NURQ, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.745439694 (6h15m33.526373073s) [noise=true , address=104.36.82.56:4244 , piece_id=LQCZ5W54NBHWAVBVYK7HYSSJDC73E6FV4CP4YIM4L5H2N3OLOZXA, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.745972271 (6h15m33.526905654s) [noise=true , address=104.36.82.36:4650 , piece_id=BKJULWGPESBINNTSUYP6E4KZRXKPACVU5I7DOSZBO4YMK2V26HEQ, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.745980243 (6h15m33.526913621s) [noise=true , address=104.36.82.38:4722 , piece_id=B2MBQY5CTMEI6WSE7FTLOV5RUJ22SCNG7N2EEJILEATGJ4TIALXA, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.746403818 (6h15m33.527337198s) [noise=true , address=104.36.82.22:4534 , piece_id=CW4AQT3XUGA4A5OC6NUY4NYOOAMLN5FFJJN5PQBYEON7HPD5LCYQ, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.746455493 (6h15m33.527388875s) [noise=true , address=104.36.82.14:4248 , piece_id=LDSGCUM7Q2VX5CDFI4JRGEEG7D5H5Q5Z5WUWKU277VN2JFRX6UGA, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.746872127 (6h15m33.527805505s) [noise=true , address=104.36.82.40:4954 , piece_id=YLOHTJLRAED66OMEQYPZHNNRI77V6WOR4EHTEZOOEHMGZHJ2O3QQ, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.747123162 (6h15m33.52805654s) [noise=true , address=104.36.82.46:4496 , piece_id=GWNQR5WRGRRQBLSGWJFD3O3UNCMNE62JQNPIV5SYKBUIL3BQKA5A, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.747280678 (6h15m33.528214053s) [noise=true , address=104.36.82.47:4524 , piece_id=DONJFTKUQ2XCW6BOU7AVOKJ7XMDMBPGQVUPRANH7X2YS7PJXUKIA, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.747736400 (6h15m33.528669779s) [noise=true , address=104.36.82.44:4893 , piece_id=HUJICRJ6W54O3AZZR6H3KG6E4TGBTNS46O6XVMQM3PFAZE7A2HZA, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.747755734 (6h15m33.528689109s) [noise=true , address=104.36.82.55:4204 , piece_id=72WISGRHMOAZDV5EDGTBOMLBL4HDOF3CBEMQAAIRGILPC7J5JKAQ, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.748313124 (6h15m33.5292465s) [noise=true , address=104.36.82.58:4287 , piece_id=BVDYUEWU7TVEQDMZXX5SJNS4QICVKMTGFLCUV6P2YHOE7TI25VXA, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.748485217 (6h15m33.529418594s) [noise=true , address=104.36.82.53:4123 , piece_id=NXJRZ5BQSLXSOIOSUJ4MM4AGKHP4UIDUGEJKLUUX7YOGQJA6WWZA, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.748831329 (6h15m33.529764704s) [noise=true , address=104.36.82.42:4829 , piece_id=ZLWJFP3LGX3YLXBJYIR66DXTYEVDBKXKVXCQSF2PCYCYJBIEJGMA, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.749299293 (6h15m33.530232669s) [noise=true , address=104.36.82.8:4002 , piece_id=6XSPWQC3US355PS6IQ4K3IYNQZH5MAT4IVIPGSYRUM3XYPSCWE6Q, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.749790974 (6h15m33.53072436s) [noise=true , address=104.36.82.44:4901 , piece_id=N4YLCLR2QZSULUL2XWONEKBFUMNOU7KUX2QEOK4SSDFSCI7IL6LA, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.750478221 (6h15m33.531411597s) [noise=true , address=104.36.82.54:4172 , piece_id=ZNA53MWDRWPUH6FUR4DJERBDT2IOYGA3IYUADKXXIOLQJLQOXE7Q, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.752884255 (6h15m33.533817625s) [noise=true , address=64.49.13.140:4508 , piece_id=MZ52U75VOIK6Q7QFXMMYRRG7JXFWDFU2JWQFYPERUCMH3BUX6XPQ, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.879839555 (6h15m33.660772917s) [upload=5911]: Next returned nil segment Jul 27 04:07:17.879861389 (6h15m33.660794765s) [upload=5911]: Waiting for error group managing segments... Jul 27 04:07:17.888652438 (6h15m33.669585811s) [noise=true , address=104.36.82.58:4287 , piece_id=BVDYUEWU7TVEQDMZXX5SJNS4QICVKMTGFLCUV6P2YHOE7TI25VXA, seg_pos={0 0}, upload=5911]: Done uploading piece. err: <nil> Jul 27 04:07:17.952192045 (6h15m33.733125429s) [noise=true , address=104.36.82.53:4123 , piece_id=NXJRZ5BQSLXSOIOSUJ4MM4AGKHP4UIDUGEJKLUUX7YOGQJA6WWZA, seg_pos={0 0}, upload=5911]: Done uploading piece. err: <nil> Jul 27 04:07:17.981907539 (6h15m33.762840911s) [noise=true , address=104.36.82.56:4244 , piece_id=LQCZ5W54NBHWAVBVYK7HYSSJDC73E6FV4CP4YIM4L5H2N3OLOZXA, seg_pos={0 0}, upload=5911]: Done uploading piece. err: <nil> Jul 27 04:07:17.982370496 (6h15m33.7633039s) [noise=true , address=104.36.82.55:4204 , piece_id=72WISGRHMOAZDV5EDGTBOMLBL4HDOF3CBEMQAAIRGILPC7J5JKAQ, seg_pos={0 0}, upload=5911]: Done uploading piece. err: <nil> Jul 27 04:07:18.009548506 (6h15m33.790481886s) [noise=true , address=104.36.82.52:4098 , piece_id=M2ETF25QQ53RRGFPUKXJRINIWV7AWXSECEDR2AIM645IED72NURQ, seg_pos={0 0}, upload=5911]: Done uploading piece. err: <nil> Jul 27 04:07:18.023770598 (6h15m33.804703983s) [noise=true , address=104.36.82.40:4954 , piece_id=YLOHTJLRAED66OMEQYPZHNNRI77V6WOR4EHTEZOOEHMGZHJ2O3QQ, seg_pos={0 0}, upload=5911]: Done uploading piece. err: <nil> Jul 27 04:07:18.139745581 (6h15m33.92067895s) [noise=true , address=104.36.82.22:4534 , piece_id=CW4AQT3XUGA4A5OC6NUY4NYOOAMLN5FFJJN5PQBYEON7HPD5LCYQ, seg_pos={0 0}, upload=5911]: Done uploading piece. err: <nil> Jul 27 04:07:18.423748140 (6h15m34.204681511s) [noise=true , address=64.49.13.140:4508 , piece_id=MZ52U75VOIK6Q7QFXMMYRRG7JXFWDFU2JWQFYPERUCMH3BUX6XPQ, seg_pos={0 0}, upload=5911]: Done uploading piece. err: <nil> Jul 27 04:08:16.739940381 (6h16m32.52087375s) [noise=true , address=104.36.82.54:4172 , piece_id=ZNA53MWDRWPUH6FUR4DJERBDT2IOYGA3IYUADKXXIOLQJLQOXE7Q, seg_pos={0 0}, upload=5911]: Done uploading piece. err: <nil> Jul 27 04:08:17.177295397 (6h16m32.958228766s) [noise=true , address=104.36.82.8:4002 , piece_id=6XSPWQC3US355PS6IQ4K3IYNQZH5MAT4IVIPGSYRUM3XYPSCWE6Q, seg_pos={0 0}, upload=5911]: Done uploading piece. err: <nil> Jul 27 04:08:17.534296584 (6h16m33.315229953s) [noise=true , address=104.36.82.36:4650 , piece_id=BKJULWGPESBINNTSUYP6E4KZRXKPACVU5I7DOSZBO4YMK2V26HEQ, seg_pos={0 0}, upload=5911]: Done uploading piece. err: <nil> Jul 27 04:08:17.677851566 (6h16m33.458784943s) [noise=true , address=104.36.82.14:4248 , piece_id=LDSGCUM7Q2VX5CDFI4JRGEEG7D5H5Q5Z5WUWKU277VN2JFRX6UGA, seg_pos={0 0}, upload=5911]: Done uploading piece. err: ecclient: upload failed (node:1u6ir2PLqDa9xGm64cuqq1RqY4qeBa1fqCo9VghspBf54XEXqn, address:104.36.82.14:4248): protocol: rpc: dial tcp 104.36.82.14:4248: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 04:08:17.678115170 (6h16m33.459048549s) [noise=true , address=104.36.82.46:4496 , piece_id=GWNQR5WRGRRQBLSGWJFD3O3UNCMNE62JQNPIV5SYKBUIL3BQKA5A, seg_pos={0 0}, upload=5911]: Done uploading piece. err: ecclient: upload failed (node:12V8XhSWkyGUVYFR8MykifFAP1YAfGCwzqucDcMMfzw3xsVW77Z, address:104.36.82.46:4496): protocol: read tcp 159.223.143.39:52800->104.36.82.46:4496: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 04:08:17.678144409 (6h16m33.459077789s) [noise=true , address=104.36.82.44:4901 , piece_id=N4YLCLR2QZSULUL2XWONEKBFUMNOU7KUX2QEOK4SSDFSCI7IL6LA, seg_pos={0 0}, upload=5911]: Done uploading piece. err: ecclient: upload failed (node:1FHE84zvBb1MKwB1Ht7puTafgQxDEvPt4mYqkYVSSufAMpWQA1, address:104.36.82.44:4901): protocol: rpc: dial tcp 104.36.82.44:4901: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 04:08:17.678229622 (6h16m33.459162988s) [noise=true , address=104.36.82.44:4893 , piece_id=HUJICRJ6W54O3AZZR6H3KG6E4TGBTNS46O6XVMQM3PFAZE7A2HZA, seg_pos={0 0}, upload=5911]: Done uploading piece. err: ecclient: upload failed (node:17VMFZ2T8jrSJVtchciJ9SnyrR6JstvcMoXrN5Q3zTD1aQU1oJ, address:104.36.82.44:4893): protocol: write tcp 159.223.143.39:35302->104.36.82.44:4893: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 04:08:17.678310218 (6h16m33.459243583s) [noise=true , address=104.36.82.42:4829 , piece_id=ZLWJFP3LGX3YLXBJYIR66DXTYEVDBKXKVXCQSF2PCYCYJBIEJGMA, seg_pos={0 0}, upload=5911]: Done uploading piece. err: ecclient: upload failed (node:1F5T8Vt1x4nXFaSenvGBCYjxhRL1H1sxRvzLjW3CqzwDVhq39F, address:104.36.82.42:4829): protocol: rpc: dial tcp 104.36.82.42:4829: i/o timeout; conn closed: op; piecestore: piecestore close: context deadline exceeded Jul 27 04:08:17.678412989 (6h16m33.459346365s) [noise=true , address=104.36.82.38:4722 , piece_id=B2MBQY5CTMEI6WSE7FTLOV5RUJ22SCNG7N2EEJILEATGJ4TIALXA, seg_pos={0 0}, upload=5911]: Done uploading piece. err: ecclient: upload failed (node:12Zw3dtUg9gARz52KhwDLTjcqC3CNm7dpPzpdx7P7PbL1jwe8uc, address:104.36.82.38:4722): protocol: read tcp 159.223.143.39:56048->104.36.82.38:4722: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 04:08:17.678455247 (6h16m33.459388624s) [noise=true , address=104.36.82.47:4524 , piece_id=DONJFTKUQ2XCW6BOU7AVOKJ7XMDMBPGQVUPRANH7X2YS7PJXUKIA, seg_pos={0 0}, upload=5911]: Done uploading piece. err: ecclient: upload failed (node:1xfbCzjSwFDRrR1gPggeN8tNCLbRRWDVzCJ6mqhZsbVSe8d4s4, address:104.36.82.47:4524): protocol: read tcp 159.223.143.39:46694->104.36.82.47:4524: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 04:08:17.678510668 (6h16m33.459444052s) [seg_pos={0 0}, upload=5911]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Jul 27 04:08:17.706442864 (6h16m33.487376232s) [upload=5911]: Done uploading segments... Jul 27 04:08:17.706495329 (6h16m33.487428705s) [upload=5911]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 6.607542263s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........X................................................] duration: 430.960371ms FAIL (2025-07-27 04:40:25 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== ece_id=NYOFDEZ7CV4PHGQ6JZMEEVAYGJRPYG6IBW7MYJO42HMWOJM7PHZA, seg_pos={0 0}, upload=5879]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12crcXpNe9NgeXTsUpooLSBgrSUFbRxbSbanLa39w2Hr5mGFww): protocol: read tcp 159.223.143.39:53630->104.36.82.5:4840: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 04:05:32.508904485 (6h13m48.289837866s) [seg_pos={0 0}, upload=5879]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 27 04:05:32.509028070 (6h13m48.289961457s) [upload=5879]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 27 04:05:32.509034645 (6h13m48.289968129s) [upload=5879]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 27 04:05:32.581834839 (6h13m48.362768225s) [upload=5879]: Done uploading segments... Jul 27 04:05:32.581875919 (6h13m48.362809298s) [upload=5879]: Upload finished. err: <nil> Jul 27 04:06:23.829155331 (6h14m39.610088696s) [upload=5895]: Starting upload Jul 27 04:06:23.829225021 (6h14m39.610158398s) [upload=5895]: Done starting upload Jul 27 04:06:23.829294606 (6h14m39.61022799s) [upload=5895]: Uploading segments... Jul 27 04:06:23.829577656 (6h14m39.610511053s) [upload=5895]: Got next segment. Inline: false Jul 27 04:06:23.829649664 (6h14m39.610583027s) [upload=5895]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 27 04:06:23.829657065 (6h14m39.61059043s) [upload=5895]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 27 04:06:23.861405530 (6h14m39.642338913s) [seg_pos={0 0}, upload=5895]: Begin upload segment... Jul 27 04:06:23.861476351 (6h14m39.642409713s) [seg_pos={0 0}, upload=5895]: Done begin upload segment. Jul 27 04:06:23.861509120 (6h14m39.642442506s) [noise=true , address=77.90.31.19:4843 , piece_id=YML5RDQIK432Z4FSE4DKXIV5U47LQBEZRFMAVLANUSGFUQR23UWQ, seg_pos={0 0}, upload=5895]: Uploading piece... Jul 27 04:06:23.861953723 (6h14m39.6428871s) [noise=true , address=45.91.116.10:4081 , piece_id=XGDTSEZOUNRTEVWHTO3DHTZMH2VZZOPJPEKLCFAHTXESLLYABHSA, seg_pos={0 0}, upload=5895]: Uploading piece... Jul 27 04:06:23.862060983 (6h14m39.642994358s) [noise=true , address=45.91.116.9:4262 , piece_id=M5ETD45HH2XZD6HDLTJ3JO2BOOIROXFWKVPTN42PSQEWG4F7KAAQ, seg_pos={0 0}, upload=5895]: Uploading piece... Jul 27 04:06:23.862759464 (6h14m39.64369284s) [noise=true , address=104.36.82.4:4804 , piece_id=IU6FDNVB5AUPA7MJ6RSXUTLBETPHOFGNJ7YGWBFIZSQFSO6W6BBA, seg_pos={0 0}, upload=5895]: Uploading piece... Jul 27 04:06:23.862835086 (6h14m39.643768461s) [noise=true , address=104.36.82.6:4873 , piece_id=YP4VJORZZUL5EVCH3P7LD3LNVMHB7CWRTKHGIA7UROXE44N6MMZA, seg_pos={0 0}, upload=5895]: Uploading piece... Jul 27 04:06:23.863593880 (6h14m39.644527258s) [noise=true , address=77.90.31.15:4697 , piece_id=Q76TJDSZPSYZSJQNFVOI462U6DFILMH2RV3LQISODXSOCXGRKW6Q, seg_pos={0 0}, upload=5895]: Uploading piece... Jul 27 04:06:23.863656891 (6h14m39.644590266s) [noise=true , address=45.91.116.7:4011 , piece_id=XMH6MCCHZBBRQWHJFWLRGYHLQD4KUGSGL7MMZSKA5TZJCND3DUDA, seg_pos={0 0}, upload=5895]: Uploading piece... Jul 27 04:06:23.864408612 (6h14m39.645341987s) [noise=true , address=45.91.116.6:4973 , piece_id=J3VULPU24CIM64VEVE3UHO62HPN6TJXGZU6BCJIS5JQJKHJK2LIQ, seg_pos={0 0}, upload=5895]: Uploading piece... Jul 27 04:06:23.864521549 (6h14m39.64545491s) [noise=true , address=104.36.82.10:4079 , piece_id=2SH5D6JOMLFR7VSN7ZY5CLMR5RHIRFYQ7Z5I2JIYTJQM43G4O6OQ, seg_pos={0 0}, upload=5895]: Uploading piece... Jul 27 04:06:23.865518491 (6h14m39.646451872s) [noise=true , address=77.90.31.17:4771 , piece_id=U2LF22GKCMKRZN6EWCF26ITDZQHBFZXCGSQXT33MCX6P57IGI35Q, seg_pos={0 0}, upload=5895]: Uploading piece... Jul 27 04:06:23.865654950 (6h14m39.646588325s) [noise=true , address=45.91.116.8:4048 , piece_id=5REWMVVTBVLDQXQ5KAEHLY4VJL5SGF7ONQWDQQ3N2FDU3OAO6AZA, seg_pos={0 0}, upload=5895]: Uploading piece... Jul 27 04:06:23.866190892 (6h14m39.647124271s) [noise=true , address=104.36.82.12:4151 , piece_id=E2TEUTQBD4Y4AUHSCPMCIFEXBSY7S5HJWBKNKWWTUWEW2BLAAE6Q, seg_pos={0 0}, upload=5895]: Uploading piece... Jul 27 04:06:23.866412590 (6h14m39.647345968s) [noise=true , address=104.36.82.11:4115 , piece_id=YXMNVT6ZHGESGQ2BXKZUVR7U2CNNAI2PO7ZTMDG5UIOV7HTDDUTA, seg_pos={0 0}, upload=5895]: Uploading piece... Jul 27 04:06:23.866715972 (6h14m39.647649359s) [noise=true , address=45.91.116.13:4153 , piece_id=H4535KOCG3F4RKA6AKKMTJSOM6Q6PGRC6M7FR3UIS3BORTSP5UQA, seg_pos={0 0}, upload=5895]: Uploading piece... Jul 27 04:06:23.866998965 (6h14m39.64793234s) [noise=true , address=104.36.82.13:4186 , piece_id=TBLDKALGJYBGR5NLCOCKJ4QKEFNIJXCKAFUBD7VR3UMNWEMKMKHQ, seg_pos={0 0}, upload=5895]: Uploading piece... Jul 27 04:06:23.867230593 (6h14m39.648163974s) [noise=true , address=104.36.82.16:4293 , piece_id=AHH6PRFUSGTDGB7H52DRITARA4IF6GFLVSRYS2Y6EPJCZSKUOYWQ, seg_pos={0 0}, upload=5895]: Uploading piece... Jul 27 04:06:23.949422808 (6h14m39.730356187s) [upload=5895]: Next returned nil segment Jul 27 04:06:23.949442576 (6h14m39.73037596s) [upload=5895]: Waiting for error group managing segments... Jul 27 04:06:23.953497426 (6h14m39.734430802s) [noise=true , address=45.91.116.10:4081 , piece_id=XGDTSEZOUNRTEVWHTO3DHTZMH2VZZOPJPEKLCFAHTXESLLYABHSA, seg_pos={0 0}, upload=5895]: Done uploading piece. err: <nil> Jul 27 04:06:23.953531796 (6h14m39.734465159s) [noise=true , address=45.91.116.6:4973 , piece_id=J3VULPU24CIM64VEVE3UHO62HPN6TJXGZU6BCJIS5JQJKHJK2LIQ, seg_pos={0 0}, upload=5895]: Done uploading piece. err: <nil> Jul 27 04:06:23.954068660 (6h14m39.735002029s) [noise=true , address=45.91.116.9:4262 , piece_id=M5ETD45HH2XZD6HDLTJ3JO2BOOIROXFWKVPTN42PSQEWG4F7KAAQ, seg_pos={0 0}, upload=5895]: Done uploading piece. err: <nil> Jul 27 04:06:23.954194339 (6h14m39.735127707s) [noise=true , address=45.91.116.8:4048 , piece_id=5REWMVVTBVLDQXQ5KAEHLY4VJL5SGF7ONQWDQQ3N2FDU3OAO6AZA, seg_pos={0 0}, upload=5895]: Done uploading piece. err: <nil> Jul 27 04:06:23.954601730 (6h14m39.735535112s) [noise=true , address=45.91.116.7:4011 , piece_id=XMH6MCCHZBBRQWHJFWLRGYHLQD4KUGSGL7MMZSKA5TZJCND3DUDA, seg_pos={0 0}, upload=5895]: Done uploading piece. err: <nil> Jul 27 04:06:23.954755872 (6h14m39.735689237s) [noise=true , address=45.91.116.13:4153 , piece_id=H4535KOCG3F4RKA6AKKMTJSOM6Q6PGRC6M7FR3UIS3BORTSP5UQA, seg_pos={0 0}, upload=5895]: Done uploading piece. err: <nil> Jul 27 04:06:23.955611883 (6h14m39.73654526s) [noise=true , address=77.90.31.19:4843 , piece_id=YML5RDQIK432Z4FSE4DKXIV5U47LQBEZRFMAVLANUSGFUQR23UWQ, seg_pos={0 0}, upload=5895]: Done uploading piece. err: <nil> Jul 27 04:06:23.956373391 (6h14m39.737306756s) [noise=true , address=77.90.31.15:4697 , piece_id=Q76TJDSZPSYZSJQNFVOI462U6DFILMH2RV3LQISODXSOCXGRKW6Q, seg_pos={0 0}, upload=5895]: Done uploading piece. err: <nil> Jul 27 04:06:23.958659122 (6h14m39.73959249s) [noise=true , address=77.90.31.17:4771 , piece_id=U2LF22GKCMKRZN6EWCF26ITDZQHBFZXCGSQXT33MCX6P57IGI35Q, seg_pos={0 0}, upload=5895]: Done uploading piece. err: <nil> Jul 27 04:06:35.058944013 (6h14m50.83987739s) [noise=true , address=104.36.82.16:4293 , piece_id=AHH6PRFUSGTDGB7H52DRITARA4IF6GFLVSRYS2Y6EPJCZSKUOYWQ, seg_pos={0 0}, upload=5895]: Done uploading piece. err: <nil> Jul 27 04:06:41.103204002 (6h14m56.884137386s) [noise=true , address=104.36.82.4:4804 , piece_id=IU6FDNVB5AUPA7MJ6RSXUTLBETPHOFGNJ7YGWBFIZSQFSO6W6BBA, seg_pos={0 0}, upload=5895]: Done uploading piece. err: <nil> Jul 27 04:06:43.191702285 (6h14m58.972635651s) [noise=true , address=104.36.82.6:4873 , piece_id=YP4VJORZZUL5EVCH3P7LD3LNVMHB7CWRTKHGIA7UROXE44N6MMZA, seg_pos={0 0}, upload=5895]: Done uploading piece. err: <nil> Jul 27 04:06:53.777973493 (6h15m9.558906861s) [noise=true , address=104.36.82.11:4115 , piece_id=YXMNVT6ZHGESGQ2BXKZUVR7U2CNNAI2PO7ZTMDG5UIOV7HTDDUTA, seg_pos={0 0}, upload=5895]: Done uploading piece. err: ecclient: upload failed (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR, address:104.36.82.11:4115): protocol: write tcp 159.223.143.39:34210->104.36.82.11:4115: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 04:06:53.778019532 (6h15m9.558952909s) [noise=true , address=104.36.82.12:4151 , piece_id=E2TEUTQBD4Y4AUHSCPMCIFEXBSY7S5HJWBKNKWWTUWEW2BLAAE6Q, seg_pos={0 0}, upload=5895]: Done uploading piece. err: ecclient: upload failed (node:1w6pwZMdeQKjt92qx6WbTWU5VPYJto5gS87f6iKvCBZH6LfyeB, address:104.36.82.12:4151): protocol: rpc: dial tcp 104.36.82.12:4151: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 04:06:53.778043653 (6h15m9.558977021s) [noise=true , address=104.36.82.13:4186 , piece_id=TBLDKALGJYBGR5NLCOCKJ4QKEFNIJXCKAFUBD7VR3UMNWEMKMKHQ, seg_pos={0 0}, upload=5895]: Done uploading piece. err: ecclient: upload failed (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht, address:104.36.82.13:4186): protocol: read tcp 159.223.143.39:34340->104.36.82.13:4186: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 04:06:53.778095228 (6h15m9.559028593s) [noise=true , address=104.36.82.10:4079 , piece_id=2SH5D6JOMLFR7VSN7ZY5CLMR5RHIRFYQ7Z5I2JIYTJQM43G4O6OQ, seg_pos={0 0}, upload=5895]: Done uploading piece. err: ecclient: upload failed (node:12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud, address:104.36.82.10:4079): protocol: write tcp 159.223.143.39:53840->104.36.82.10:4079: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 04:06:53.778142167 (6h15m9.559075532s) [seg_pos={0 0}, upload=5895]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 27 04:06:53.837032161 (6h15m9.617965543s) [upload=5895]: Done uploading segments... Jul 27 04:06:53.837055996 (6h15m9.617989366s) [upload=5895]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 430.987587ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................X...................................] duration: 412.50291ms FAIL (2025-07-27 04:40:24 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 12) Duration: 412.566447ms core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............X...............................................] duration: 381.117943ms FAIL (2025-07-27 04:39:58 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== ece_id=NYOFDEZ7CV4PHGQ6JZMEEVAYGJRPYG6IBW7MYJO42HMWOJM7PHZA, seg_pos={0 0}, upload=5879]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12crcXpNe9NgeXTsUpooLSBgrSUFbRxbSbanLa39w2Hr5mGFww): protocol: read tcp 159.223.143.39:53630->104.36.82.5:4840: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 04:05:32.508904485 (6h13m48.289837866s) [seg_pos={0 0}, upload=5879]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 27 04:05:32.509028070 (6h13m48.289961457s) [upload=5879]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 27 04:05:32.509034645 (6h13m48.289968129s) [upload=5879]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 27 04:05:32.581834839 (6h13m48.362768225s) [upload=5879]: Done uploading segments... Jul 27 04:05:32.581875919 (6h13m48.362809298s) [upload=5879]: Upload finished. err: <nil> Jul 27 04:06:23.829155331 (6h14m39.610088696s) [upload=5895]: Starting upload Jul 27 04:06:23.829225021 (6h14m39.610158398s) [upload=5895]: Done starting upload Jul 27 04:06:23.829294606 (6h14m39.61022799s) [upload=5895]: Uploading segments... Jul 27 04:06:23.829577656 (6h14m39.610511053s) [upload=5895]: Got next segment. Inline: false Jul 27 04:06:23.829649664 (6h14m39.610583027s) [upload=5895]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 27 04:06:23.829657065 (6h14m39.61059043s) [upload=5895]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 27 04:06:23.861405530 (6h14m39.642338913s) [seg_pos={0 0}, upload=5895]: Begin upload segment... Jul 27 04:06:23.861476351 (6h14m39.642409713s) [seg_pos={0 0}, upload=5895]: Done begin upload segment. Jul 27 04:06:23.861509120 (6h14m39.642442506s) [noise=true , address=77.90.31.19:4843 , piece_id=YML5RDQIK432Z4FSE4DKXIV5U47LQBEZRFMAVLANUSGFUQR23UWQ, seg_pos={0 0}, upload=5895]: Uploading piece... Jul 27 04:06:23.861953723 (6h14m39.6428871s) [noise=true , address=45.91.116.10:4081 , piece_id=XGDTSEZOUNRTEVWHTO3DHTZMH2VZZOPJPEKLCFAHTXESLLYABHSA, seg_pos={0 0}, upload=5895]: Uploading piece... Jul 27 04:06:23.862060983 (6h14m39.642994358s) [noise=true , address=45.91.116.9:4262 , piece_id=M5ETD45HH2XZD6HDLTJ3JO2BOOIROXFWKVPTN42PSQEWG4F7KAAQ, seg_pos={0 0}, upload=5895]: Uploading piece... Jul 27 04:06:23.862759464 (6h14m39.64369284s) [noise=true , address=104.36.82.4:4804 , piece_id=IU6FDNVB5AUPA7MJ6RSXUTLBETPHOFGNJ7YGWBFIZSQFSO6W6BBA, seg_pos={0 0}, upload=5895]: Uploading piece... Jul 27 04:06:23.862835086 (6h14m39.643768461s) [noise=true , address=104.36.82.6:4873 , piece_id=YP4VJORZZUL5EVCH3P7LD3LNVMHB7CWRTKHGIA7UROXE44N6MMZA, seg_pos={0 0}, upload=5895]: Uploading piece... Jul 27 04:06:23.863593880 (6h14m39.644527258s) [noise=true , address=77.90.31.15:4697 , piece_id=Q76TJDSZPSYZSJQNFVOI462U6DFILMH2RV3LQISODXSOCXGRKW6Q, seg_pos={0 0}, upload=5895]: Uploading piece... Jul 27 04:06:23.863656891 (6h14m39.644590266s) [noise=true , address=45.91.116.7:4011 , piece_id=XMH6MCCHZBBRQWHJFWLRGYHLQD4KUGSGL7MMZSKA5TZJCND3DUDA, seg_pos={0 0}, upload=5895]: Uploading piece... Jul 27 04:06:23.864408612 (6h14m39.645341987s) [noise=true , address=45.91.116.6:4973 , piece_id=J3VULPU24CIM64VEVE3UHO62HPN6TJXGZU6BCJIS5JQJKHJK2LIQ, seg_pos={0 0}, upload=5895]: Uploading piece... Jul 27 04:06:23.864521549 (6h14m39.64545491s) [noise=true , address=104.36.82.10:4079 , piece_id=2SH5D6JOMLFR7VSN7ZY5CLMR5RHIRFYQ7Z5I2JIYTJQM43G4O6OQ, seg_pos={0 0}, upload=5895]: Uploading piece... Jul 27 04:06:23.865518491 (6h14m39.646451872s) [noise=true , address=77.90.31.17:4771 , piece_id=U2LF22GKCMKRZN6EWCF26ITDZQHBFZXCGSQXT33MCX6P57IGI35Q, seg_pos={0 0}, upload=5895]: Uploading piece... Jul 27 04:06:23.865654950 (6h14m39.646588325s) [noise=true , address=45.91.116.8:4048 , piece_id=5REWMVVTBVLDQXQ5KAEHLY4VJL5SGF7ONQWDQQ3N2FDU3OAO6AZA, seg_pos={0 0}, upload=5895]: Uploading piece... Jul 27 04:06:23.866190892 (6h14m39.647124271s) [noise=true , address=104.36.82.12:4151 , piece_id=E2TEUTQBD4Y4AUHSCPMCIFEXBSY7S5HJWBKNKWWTUWEW2BLAAE6Q, seg_pos={0 0}, upload=5895]: Uploading piece... Jul 27 04:06:23.866412590 (6h14m39.647345968s) [noise=true , address=104.36.82.11:4115 , piece_id=YXMNVT6ZHGESGQ2BXKZUVR7U2CNNAI2PO7ZTMDG5UIOV7HTDDUTA, seg_pos={0 0}, upload=5895]: Uploading piece... Jul 27 04:06:23.866715972 (6h14m39.647649359s) [noise=true , address=45.91.116.13:4153 , piece_id=H4535KOCG3F4RKA6AKKMTJSOM6Q6PGRC6M7FR3UIS3BORTSP5UQA, seg_pos={0 0}, upload=5895]: Uploading piece... Jul 27 04:06:23.866998965 (6h14m39.64793234s) [noise=true , address=104.36.82.13:4186 , piece_id=TBLDKALGJYBGR5NLCOCKJ4QKEFNIJXCKAFUBD7VR3UMNWEMKMKHQ, seg_pos={0 0}, upload=5895]: Uploading piece... Jul 27 04:06:23.867230593 (6h14m39.648163974s) [noise=true , address=104.36.82.16:4293 , piece_id=AHH6PRFUSGTDGB7H52DRITARA4IF6GFLVSRYS2Y6EPJCZSKUOYWQ, seg_pos={0 0}, upload=5895]: Uploading piece... Jul 27 04:06:23.949422808 (6h14m39.730356187s) [upload=5895]: Next returned nil segment Jul 27 04:06:23.949442576 (6h14m39.73037596s) [upload=5895]: Waiting for error group managing segments... Jul 27 04:06:23.953497426 (6h14m39.734430802s) [noise=true , address=45.91.116.10:4081 , piece_id=XGDTSEZOUNRTEVWHTO3DHTZMH2VZZOPJPEKLCFAHTXESLLYABHSA, seg_pos={0 0}, upload=5895]: Done uploading piece. err: <nil> Jul 27 04:06:23.953531796 (6h14m39.734465159s) [noise=true , address=45.91.116.6:4973 , piece_id=J3VULPU24CIM64VEVE3UHO62HPN6TJXGZU6BCJIS5JQJKHJK2LIQ, seg_pos={0 0}, upload=5895]: Done uploading piece. err: <nil> Jul 27 04:06:23.954068660 (6h14m39.735002029s) [noise=true , address=45.91.116.9:4262 , piece_id=M5ETD45HH2XZD6HDLTJ3JO2BOOIROXFWKVPTN42PSQEWG4F7KAAQ, seg_pos={0 0}, upload=5895]: Done uploading piece. err: <nil> Jul 27 04:06:23.954194339 (6h14m39.735127707s) [noise=true , address=45.91.116.8:4048 , piece_id=5REWMVVTBVLDQXQ5KAEHLY4VJL5SGF7ONQWDQQ3N2FDU3OAO6AZA, seg_pos={0 0}, upload=5895]: Done uploading piece. err: <nil> Jul 27 04:06:23.954601730 (6h14m39.735535112s) [noise=true , address=45.91.116.7:4011 , piece_id=XMH6MCCHZBBRQWHJFWLRGYHLQD4KUGSGL7MMZSKA5TZJCND3DUDA, seg_pos={0 0}, upload=5895]: Done uploading piece. err: <nil> Jul 27 04:06:23.954755872 (6h14m39.735689237s) [noise=true , address=45.91.116.13:4153 , piece_id=H4535KOCG3F4RKA6AKKMTJSOM6Q6PGRC6M7FR3UIS3BORTSP5UQA, seg_pos={0 0}, upload=5895]: Done uploading piece. err: <nil> Jul 27 04:06:23.955611883 (6h14m39.73654526s) [noise=true , address=77.90.31.19:4843 , piece_id=YML5RDQIK432Z4FSE4DKXIV5U47LQBEZRFMAVLANUSGFUQR23UWQ, seg_pos={0 0}, upload=5895]: Done uploading piece. err: <nil> Jul 27 04:06:23.956373391 (6h14m39.737306756s) [noise=true , address=77.90.31.15:4697 , piece_id=Q76TJDSZPSYZSJQNFVOI462U6DFILMH2RV3LQISODXSOCXGRKW6Q, seg_pos={0 0}, upload=5895]: Done uploading piece. err: <nil> Jul 27 04:06:23.958659122 (6h14m39.73959249s) [noise=true , address=77.90.31.17:4771 , piece_id=U2LF22GKCMKRZN6EWCF26ITDZQHBFZXCGSQXT33MCX6P57IGI35Q, seg_pos={0 0}, upload=5895]: Done uploading piece. err: <nil> Jul 27 04:06:35.058944013 (6h14m50.83987739s) [noise=true , address=104.36.82.16:4293 , piece_id=AHH6PRFUSGTDGB7H52DRITARA4IF6GFLVSRYS2Y6EPJCZSKUOYWQ, seg_pos={0 0}, upload=5895]: Done uploading piece. err: <nil> Jul 27 04:06:41.103204002 (6h14m56.884137386s) [noise=true , address=104.36.82.4:4804 , piece_id=IU6FDNVB5AUPA7MJ6RSXUTLBETPHOFGNJ7YGWBFIZSQFSO6W6BBA, seg_pos={0 0}, upload=5895]: Done uploading piece. err: <nil> Jul 27 04:06:43.191702285 (6h14m58.972635651s) [noise=true , address=104.36.82.6:4873 , piece_id=YP4VJORZZUL5EVCH3P7LD3LNVMHB7CWRTKHGIA7UROXE44N6MMZA, seg_pos={0 0}, upload=5895]: Done uploading piece. err: <nil> Jul 27 04:06:53.777973493 (6h15m9.558906861s) [noise=true , address=104.36.82.11:4115 , piece_id=YXMNVT6ZHGESGQ2BXKZUVR7U2CNNAI2PO7ZTMDG5UIOV7HTDDUTA, seg_pos={0 0}, upload=5895]: Done uploading piece. err: ecclient: upload failed (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR, address:104.36.82.11:4115): protocol: write tcp 159.223.143.39:34210->104.36.82.11:4115: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 04:06:53.778019532 (6h15m9.558952909s) [noise=true , address=104.36.82.12:4151 , piece_id=E2TEUTQBD4Y4AUHSCPMCIFEXBSY7S5HJWBKNKWWTUWEW2BLAAE6Q, seg_pos={0 0}, upload=5895]: Done uploading piece. err: ecclient: upload failed (node:1w6pwZMdeQKjt92qx6WbTWU5VPYJto5gS87f6iKvCBZH6LfyeB, address:104.36.82.12:4151): protocol: rpc: dial tcp 104.36.82.12:4151: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 04:06:53.778043653 (6h15m9.558977021s) [noise=true , address=104.36.82.13:4186 , piece_id=TBLDKALGJYBGR5NLCOCKJ4QKEFNIJXCKAFUBD7VR3UMNWEMKMKHQ, seg_pos={0 0}, upload=5895]: Done uploading piece. err: ecclient: upload failed (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht, address:104.36.82.13:4186): protocol: read tcp 159.223.143.39:34340->104.36.82.13:4186: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 04:06:53.778095228 (6h15m9.559028593s) [noise=true , address=104.36.82.10:4079 , piece_id=2SH5D6JOMLFR7VSN7ZY5CLMR5RHIRFYQ7Z5I2JIYTJQM43G4O6OQ, seg_pos={0 0}, upload=5895]: Done uploading piece. err: ecclient: upload failed (node:12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud, address:104.36.82.10:4079): protocol: write tcp 159.223.143.39:53840->104.36.82.10:4079: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 04:06:53.778142167 (6h15m9.559075532s) [seg_pos={0 0}, upload=5895]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 27 04:06:53.837032161 (6h15m9.617965543s) [upload=5895]: Done uploading segments... Jul 27 04:06:53.837055996 (6h15m9.617989366s) [upload=5895]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 381.157849ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................X..................................] duration: 410.247365ms FAIL (2025-07-27 04:39:57 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 12) Duration: 410.27553ms core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [...............X............................................] duration: 496.324466ms FAIL (2025-07-27 04:39:52 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== {0 0}, upload=5911]: Begin upload segment... Jul 27 04:07:17.745307658 (6h15m33.526241043s) [seg_pos={0 0}, upload=5911]: Done begin upload segment. Jul 27 04:07:17.745349241 (6h15m33.526282625s) [noise=true , address=104.36.82.52:4098 , piece_id=M2ETF25QQ53RRGFPUKXJRINIWV7AWXSECEDR2AIM645IED72NURQ, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.745439694 (6h15m33.526373073s) [noise=true , address=104.36.82.56:4244 , piece_id=LQCZ5W54NBHWAVBVYK7HYSSJDC73E6FV4CP4YIM4L5H2N3OLOZXA, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.745972271 (6h15m33.526905654s) [noise=true , address=104.36.82.36:4650 , piece_id=BKJULWGPESBINNTSUYP6E4KZRXKPACVU5I7DOSZBO4YMK2V26HEQ, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.745980243 (6h15m33.526913621s) [noise=true , address=104.36.82.38:4722 , piece_id=B2MBQY5CTMEI6WSE7FTLOV5RUJ22SCNG7N2EEJILEATGJ4TIALXA, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.746403818 (6h15m33.527337198s) [noise=true , address=104.36.82.22:4534 , piece_id=CW4AQT3XUGA4A5OC6NUY4NYOOAMLN5FFJJN5PQBYEON7HPD5LCYQ, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.746455493 (6h15m33.527388875s) [noise=true , address=104.36.82.14:4248 , piece_id=LDSGCUM7Q2VX5CDFI4JRGEEG7D5H5Q5Z5WUWKU277VN2JFRX6UGA, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.746872127 (6h15m33.527805505s) [noise=true , address=104.36.82.40:4954 , piece_id=YLOHTJLRAED66OMEQYPZHNNRI77V6WOR4EHTEZOOEHMGZHJ2O3QQ, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.747123162 (6h15m33.52805654s) [noise=true , address=104.36.82.46:4496 , piece_id=GWNQR5WRGRRQBLSGWJFD3O3UNCMNE62JQNPIV5SYKBUIL3BQKA5A, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.747280678 (6h15m33.528214053s) [noise=true , address=104.36.82.47:4524 , piece_id=DONJFTKUQ2XCW6BOU7AVOKJ7XMDMBPGQVUPRANH7X2YS7PJXUKIA, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.747736400 (6h15m33.528669779s) [noise=true , address=104.36.82.44:4893 , piece_id=HUJICRJ6W54O3AZZR6H3KG6E4TGBTNS46O6XVMQM3PFAZE7A2HZA, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.747755734 (6h15m33.528689109s) [noise=true , address=104.36.82.55:4204 , piece_id=72WISGRHMOAZDV5EDGTBOMLBL4HDOF3CBEMQAAIRGILPC7J5JKAQ, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.748313124 (6h15m33.5292465s) [noise=true , address=104.36.82.58:4287 , piece_id=BVDYUEWU7TVEQDMZXX5SJNS4QICVKMTGFLCUV6P2YHOE7TI25VXA, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.748485217 (6h15m33.529418594s) [noise=true , address=104.36.82.53:4123 , piece_id=NXJRZ5BQSLXSOIOSUJ4MM4AGKHP4UIDUGEJKLUUX7YOGQJA6WWZA, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.748831329 (6h15m33.529764704s) [noise=true , address=104.36.82.42:4829 , piece_id=ZLWJFP3LGX3YLXBJYIR66DXTYEVDBKXKVXCQSF2PCYCYJBIEJGMA, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.749299293 (6h15m33.530232669s) [noise=true , address=104.36.82.8:4002 , piece_id=6XSPWQC3US355PS6IQ4K3IYNQZH5MAT4IVIPGSYRUM3XYPSCWE6Q, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.749790974 (6h15m33.53072436s) [noise=true , address=104.36.82.44:4901 , piece_id=N4YLCLR2QZSULUL2XWONEKBFUMNOU7KUX2QEOK4SSDFSCI7IL6LA, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.750478221 (6h15m33.531411597s) [noise=true , address=104.36.82.54:4172 , piece_id=ZNA53MWDRWPUH6FUR4DJERBDT2IOYGA3IYUADKXXIOLQJLQOXE7Q, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.752884255 (6h15m33.533817625s) [noise=true , address=64.49.13.140:4508 , piece_id=MZ52U75VOIK6Q7QFXMMYRRG7JXFWDFU2JWQFYPERUCMH3BUX6XPQ, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.879839555 (6h15m33.660772917s) [upload=5911]: Next returned nil segment Jul 27 04:07:17.879861389 (6h15m33.660794765s) [upload=5911]: Waiting for error group managing segments... Jul 27 04:07:17.888652438 (6h15m33.669585811s) [noise=true , address=104.36.82.58:4287 , piece_id=BVDYUEWU7TVEQDMZXX5SJNS4QICVKMTGFLCUV6P2YHOE7TI25VXA, seg_pos={0 0}, upload=5911]: Done uploading piece. err: <nil> Jul 27 04:07:17.952192045 (6h15m33.733125429s) [noise=true , address=104.36.82.53:4123 , piece_id=NXJRZ5BQSLXSOIOSUJ4MM4AGKHP4UIDUGEJKLUUX7YOGQJA6WWZA, seg_pos={0 0}, upload=5911]: Done uploading piece. err: <nil> Jul 27 04:07:17.981907539 (6h15m33.762840911s) [noise=true , address=104.36.82.56:4244 , piece_id=LQCZ5W54NBHWAVBVYK7HYSSJDC73E6FV4CP4YIM4L5H2N3OLOZXA, seg_pos={0 0}, upload=5911]: Done uploading piece. err: <nil> Jul 27 04:07:17.982370496 (6h15m33.7633039s) [noise=true , address=104.36.82.55:4204 , piece_id=72WISGRHMOAZDV5EDGTBOMLBL4HDOF3CBEMQAAIRGILPC7J5JKAQ, seg_pos={0 0}, upload=5911]: Done uploading piece. err: <nil> Jul 27 04:07:18.009548506 (6h15m33.790481886s) [noise=true , address=104.36.82.52:4098 , piece_id=M2ETF25QQ53RRGFPUKXJRINIWV7AWXSECEDR2AIM645IED72NURQ, seg_pos={0 0}, upload=5911]: Done uploading piece. err: <nil> Jul 27 04:07:18.023770598 (6h15m33.804703983s) [noise=true , address=104.36.82.40:4954 , piece_id=YLOHTJLRAED66OMEQYPZHNNRI77V6WOR4EHTEZOOEHMGZHJ2O3QQ, seg_pos={0 0}, upload=5911]: Done uploading piece. err: <nil> Jul 27 04:07:18.139745581 (6h15m33.92067895s) [noise=true , address=104.36.82.22:4534 , piece_id=CW4AQT3XUGA4A5OC6NUY4NYOOAMLN5FFJJN5PQBYEON7HPD5LCYQ, seg_pos={0 0}, upload=5911]: Done uploading piece. err: <nil> Jul 27 04:07:18.423748140 (6h15m34.204681511s) [noise=true , address=64.49.13.140:4508 , piece_id=MZ52U75VOIK6Q7QFXMMYRRG7JXFWDFU2JWQFYPERUCMH3BUX6XPQ, seg_pos={0 0}, upload=5911]: Done uploading piece. err: <nil> Jul 27 04:08:16.739940381 (6h16m32.52087375s) [noise=true , address=104.36.82.54:4172 , piece_id=ZNA53MWDRWPUH6FUR4DJERBDT2IOYGA3IYUADKXXIOLQJLQOXE7Q, seg_pos={0 0}, upload=5911]: Done uploading piece. err: <nil> Jul 27 04:08:17.177295397 (6h16m32.958228766s) [noise=true , address=104.36.82.8:4002 , piece_id=6XSPWQC3US355PS6IQ4K3IYNQZH5MAT4IVIPGSYRUM3XYPSCWE6Q, seg_pos={0 0}, upload=5911]: Done uploading piece. err: <nil> Jul 27 04:08:17.534296584 (6h16m33.315229953s) [noise=true , address=104.36.82.36:4650 , piece_id=BKJULWGPESBINNTSUYP6E4KZRXKPACVU5I7DOSZBO4YMK2V26HEQ, seg_pos={0 0}, upload=5911]: Done uploading piece. err: <nil> Jul 27 04:08:17.677851566 (6h16m33.458784943s) [noise=true , address=104.36.82.14:4248 , piece_id=LDSGCUM7Q2VX5CDFI4JRGEEG7D5H5Q5Z5WUWKU277VN2JFRX6UGA, seg_pos={0 0}, upload=5911]: Done uploading piece. err: ecclient: upload failed (node:1u6ir2PLqDa9xGm64cuqq1RqY4qeBa1fqCo9VghspBf54XEXqn, address:104.36.82.14:4248): protocol: rpc: dial tcp 104.36.82.14:4248: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 04:08:17.678115170 (6h16m33.459048549s) [noise=true , address=104.36.82.46:4496 , piece_id=GWNQR5WRGRRQBLSGWJFD3O3UNCMNE62JQNPIV5SYKBUIL3BQKA5A, seg_pos={0 0}, upload=5911]: Done uploading piece. err: ecclient: upload failed (node:12V8XhSWkyGUVYFR8MykifFAP1YAfGCwzqucDcMMfzw3xsVW77Z, address:104.36.82.46:4496): protocol: read tcp 159.223.143.39:52800->104.36.82.46:4496: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 04:08:17.678144409 (6h16m33.459077789s) [noise=true , address=104.36.82.44:4901 , piece_id=N4YLCLR2QZSULUL2XWONEKBFUMNOU7KUX2QEOK4SSDFSCI7IL6LA, seg_pos={0 0}, upload=5911]: Done uploading piece. err: ecclient: upload failed (node:1FHE84zvBb1MKwB1Ht7puTafgQxDEvPt4mYqkYVSSufAMpWQA1, address:104.36.82.44:4901): protocol: rpc: dial tcp 104.36.82.44:4901: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 04:08:17.678229622 (6h16m33.459162988s) [noise=true , address=104.36.82.44:4893 , piece_id=HUJICRJ6W54O3AZZR6H3KG6E4TGBTNS46O6XVMQM3PFAZE7A2HZA, seg_pos={0 0}, upload=5911]: Done uploading piece. err: ecclient: upload failed (node:17VMFZ2T8jrSJVtchciJ9SnyrR6JstvcMoXrN5Q3zTD1aQU1oJ, address:104.36.82.44:4893): protocol: write tcp 159.223.143.39:35302->104.36.82.44:4893: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 04:08:17.678310218 (6h16m33.459243583s) [noise=true , address=104.36.82.42:4829 , piece_id=ZLWJFP3LGX3YLXBJYIR66DXTYEVDBKXKVXCQSF2PCYCYJBIEJGMA, seg_pos={0 0}, upload=5911]: Done uploading piece. err: ecclient: upload failed (node:1F5T8Vt1x4nXFaSenvGBCYjxhRL1H1sxRvzLjW3CqzwDVhq39F, address:104.36.82.42:4829): protocol: rpc: dial tcp 104.36.82.42:4829: i/o timeout; conn closed: op; piecestore: piecestore close: context deadline exceeded Jul 27 04:08:17.678412989 (6h16m33.459346365s) [noise=true , address=104.36.82.38:4722 , piece_id=B2MBQY5CTMEI6WSE7FTLOV5RUJ22SCNG7N2EEJILEATGJ4TIALXA, seg_pos={0 0}, upload=5911]: Done uploading piece. err: ecclient: upload failed (node:12Zw3dtUg9gARz52KhwDLTjcqC3CNm7dpPzpdx7P7PbL1jwe8uc, address:104.36.82.38:4722): protocol: read tcp 159.223.143.39:56048->104.36.82.38:4722: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 04:08:17.678455247 (6h16m33.459388624s) [noise=true , address=104.36.82.47:4524 , piece_id=DONJFTKUQ2XCW6BOU7AVOKJ7XMDMBPGQVUPRANH7X2YS7PJXUKIA, seg_pos={0 0}, upload=5911]: Done uploading piece. err: ecclient: upload failed (node:1xfbCzjSwFDRrR1gPggeN8tNCLbRRWDVzCJ6mqhZsbVSe8d4s4, address:104.36.82.47:4524): protocol: read tcp 159.223.143.39:46694->104.36.82.47:4524: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 04:08:17.678510668 (6h16m33.459444052s) [seg_pos={0 0}, upload=5911]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Jul 27 04:08:17.706442864 (6h16m33.487376232s) [upload=5911]: Done uploading segments... Jul 27 04:08:17.706495329 (6h16m33.487428705s) [upload=5911]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 496.35948ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................X.................................] duration: 8.033057557s FAIL (2025-07-27 04:39:46 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 12) Duration: 8.033089462s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [................X...........................................] duration: 649.714236ms FAIL (2025-07-27 04:39:26 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== {0 0}, upload=5911]: Begin upload segment... Jul 27 04:07:17.745307658 (6h15m33.526241043s) [seg_pos={0 0}, upload=5911]: Done begin upload segment. Jul 27 04:07:17.745349241 (6h15m33.526282625s) [noise=true , address=104.36.82.52:4098 , piece_id=M2ETF25QQ53RRGFPUKXJRINIWV7AWXSECEDR2AIM645IED72NURQ, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.745439694 (6h15m33.526373073s) [noise=true , address=104.36.82.56:4244 , piece_id=LQCZ5W54NBHWAVBVYK7HYSSJDC73E6FV4CP4YIM4L5H2N3OLOZXA, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.745972271 (6h15m33.526905654s) [noise=true , address=104.36.82.36:4650 , piece_id=BKJULWGPESBINNTSUYP6E4KZRXKPACVU5I7DOSZBO4YMK2V26HEQ, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.745980243 (6h15m33.526913621s) [noise=true , address=104.36.82.38:4722 , piece_id=B2MBQY5CTMEI6WSE7FTLOV5RUJ22SCNG7N2EEJILEATGJ4TIALXA, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.746403818 (6h15m33.527337198s) [noise=true , address=104.36.82.22:4534 , piece_id=CW4AQT3XUGA4A5OC6NUY4NYOOAMLN5FFJJN5PQBYEON7HPD5LCYQ, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.746455493 (6h15m33.527388875s) [noise=true , address=104.36.82.14:4248 , piece_id=LDSGCUM7Q2VX5CDFI4JRGEEG7D5H5Q5Z5WUWKU277VN2JFRX6UGA, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.746872127 (6h15m33.527805505s) [noise=true , address=104.36.82.40:4954 , piece_id=YLOHTJLRAED66OMEQYPZHNNRI77V6WOR4EHTEZOOEHMGZHJ2O3QQ, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.747123162 (6h15m33.52805654s) [noise=true , address=104.36.82.46:4496 , piece_id=GWNQR5WRGRRQBLSGWJFD3O3UNCMNE62JQNPIV5SYKBUIL3BQKA5A, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.747280678 (6h15m33.528214053s) [noise=true , address=104.36.82.47:4524 , piece_id=DONJFTKUQ2XCW6BOU7AVOKJ7XMDMBPGQVUPRANH7X2YS7PJXUKIA, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.747736400 (6h15m33.528669779s) [noise=true , address=104.36.82.44:4893 , piece_id=HUJICRJ6W54O3AZZR6H3KG6E4TGBTNS46O6XVMQM3PFAZE7A2HZA, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.747755734 (6h15m33.528689109s) [noise=true , address=104.36.82.55:4204 , piece_id=72WISGRHMOAZDV5EDGTBOMLBL4HDOF3CBEMQAAIRGILPC7J5JKAQ, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.748313124 (6h15m33.5292465s) [noise=true , address=104.36.82.58:4287 , piece_id=BVDYUEWU7TVEQDMZXX5SJNS4QICVKMTGFLCUV6P2YHOE7TI25VXA, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.748485217 (6h15m33.529418594s) [noise=true , address=104.36.82.53:4123 , piece_id=NXJRZ5BQSLXSOIOSUJ4MM4AGKHP4UIDUGEJKLUUX7YOGQJA6WWZA, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.748831329 (6h15m33.529764704s) [noise=true , address=104.36.82.42:4829 , piece_id=ZLWJFP3LGX3YLXBJYIR66DXTYEVDBKXKVXCQSF2PCYCYJBIEJGMA, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.749299293 (6h15m33.530232669s) [noise=true , address=104.36.82.8:4002 , piece_id=6XSPWQC3US355PS6IQ4K3IYNQZH5MAT4IVIPGSYRUM3XYPSCWE6Q, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.749790974 (6h15m33.53072436s) [noise=true , address=104.36.82.44:4901 , piece_id=N4YLCLR2QZSULUL2XWONEKBFUMNOU7KUX2QEOK4SSDFSCI7IL6LA, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.750478221 (6h15m33.531411597s) [noise=true , address=104.36.82.54:4172 , piece_id=ZNA53MWDRWPUH6FUR4DJERBDT2IOYGA3IYUADKXXIOLQJLQOXE7Q, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.752884255 (6h15m33.533817625s) [noise=true , address=64.49.13.140:4508 , piece_id=MZ52U75VOIK6Q7QFXMMYRRG7JXFWDFU2JWQFYPERUCMH3BUX6XPQ, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.879839555 (6h15m33.660772917s) [upload=5911]: Next returned nil segment Jul 27 04:07:17.879861389 (6h15m33.660794765s) [upload=5911]: Waiting for error group managing segments... Jul 27 04:07:17.888652438 (6h15m33.669585811s) [noise=true , address=104.36.82.58:4287 , piece_id=BVDYUEWU7TVEQDMZXX5SJNS4QICVKMTGFLCUV6P2YHOE7TI25VXA, seg_pos={0 0}, upload=5911]: Done uploading piece. err: <nil> Jul 27 04:07:17.952192045 (6h15m33.733125429s) [noise=true , address=104.36.82.53:4123 , piece_id=NXJRZ5BQSLXSOIOSUJ4MM4AGKHP4UIDUGEJKLUUX7YOGQJA6WWZA, seg_pos={0 0}, upload=5911]: Done uploading piece. err: <nil> Jul 27 04:07:17.981907539 (6h15m33.762840911s) [noise=true , address=104.36.82.56:4244 , piece_id=LQCZ5W54NBHWAVBVYK7HYSSJDC73E6FV4CP4YIM4L5H2N3OLOZXA, seg_pos={0 0}, upload=5911]: Done uploading piece. err: <nil> Jul 27 04:07:17.982370496 (6h15m33.7633039s) [noise=true , address=104.36.82.55:4204 , piece_id=72WISGRHMOAZDV5EDGTBOMLBL4HDOF3CBEMQAAIRGILPC7J5JKAQ, seg_pos={0 0}, upload=5911]: Done uploading piece. err: <nil> Jul 27 04:07:18.009548506 (6h15m33.790481886s) [noise=true , address=104.36.82.52:4098 , piece_id=M2ETF25QQ53RRGFPUKXJRINIWV7AWXSECEDR2AIM645IED72NURQ, seg_pos={0 0}, upload=5911]: Done uploading piece. err: <nil> Jul 27 04:07:18.023770598 (6h15m33.804703983s) [noise=true , address=104.36.82.40:4954 , piece_id=YLOHTJLRAED66OMEQYPZHNNRI77V6WOR4EHTEZOOEHMGZHJ2O3QQ, seg_pos={0 0}, upload=5911]: Done uploading piece. err: <nil> Jul 27 04:07:18.139745581 (6h15m33.92067895s) [noise=true , address=104.36.82.22:4534 , piece_id=CW4AQT3XUGA4A5OC6NUY4NYOOAMLN5FFJJN5PQBYEON7HPD5LCYQ, seg_pos={0 0}, upload=5911]: Done uploading piece. err: <nil> Jul 27 04:07:18.423748140 (6h15m34.204681511s) [noise=true , address=64.49.13.140:4508 , piece_id=MZ52U75VOIK6Q7QFXMMYRRG7JXFWDFU2JWQFYPERUCMH3BUX6XPQ, seg_pos={0 0}, upload=5911]: Done uploading piece. err: <nil> Jul 27 04:08:16.739940381 (6h16m32.52087375s) [noise=true , address=104.36.82.54:4172 , piece_id=ZNA53MWDRWPUH6FUR4DJERBDT2IOYGA3IYUADKXXIOLQJLQOXE7Q, seg_pos={0 0}, upload=5911]: Done uploading piece. err: <nil> Jul 27 04:08:17.177295397 (6h16m32.958228766s) [noise=true , address=104.36.82.8:4002 , piece_id=6XSPWQC3US355PS6IQ4K3IYNQZH5MAT4IVIPGSYRUM3XYPSCWE6Q, seg_pos={0 0}, upload=5911]: Done uploading piece. err: <nil> Jul 27 04:08:17.534296584 (6h16m33.315229953s) [noise=true , address=104.36.82.36:4650 , piece_id=BKJULWGPESBINNTSUYP6E4KZRXKPACVU5I7DOSZBO4YMK2V26HEQ, seg_pos={0 0}, upload=5911]: Done uploading piece. err: <nil> Jul 27 04:08:17.677851566 (6h16m33.458784943s) [noise=true , address=104.36.82.14:4248 , piece_id=LDSGCUM7Q2VX5CDFI4JRGEEG7D5H5Q5Z5WUWKU277VN2JFRX6UGA, seg_pos={0 0}, upload=5911]: Done uploading piece. err: ecclient: upload failed (node:1u6ir2PLqDa9xGm64cuqq1RqY4qeBa1fqCo9VghspBf54XEXqn, address:104.36.82.14:4248): protocol: rpc: dial tcp 104.36.82.14:4248: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 04:08:17.678115170 (6h16m33.459048549s) [noise=true , address=104.36.82.46:4496 , piece_id=GWNQR5WRGRRQBLSGWJFD3O3UNCMNE62JQNPIV5SYKBUIL3BQKA5A, seg_pos={0 0}, upload=5911]: Done uploading piece. err: ecclient: upload failed (node:12V8XhSWkyGUVYFR8MykifFAP1YAfGCwzqucDcMMfzw3xsVW77Z, address:104.36.82.46:4496): protocol: read tcp 159.223.143.39:52800->104.36.82.46:4496: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 04:08:17.678144409 (6h16m33.459077789s) [noise=true , address=104.36.82.44:4901 , piece_id=N4YLCLR2QZSULUL2XWONEKBFUMNOU7KUX2QEOK4SSDFSCI7IL6LA, seg_pos={0 0}, upload=5911]: Done uploading piece. err: ecclient: upload failed (node:1FHE84zvBb1MKwB1Ht7puTafgQxDEvPt4mYqkYVSSufAMpWQA1, address:104.36.82.44:4901): protocol: rpc: dial tcp 104.36.82.44:4901: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 04:08:17.678229622 (6h16m33.459162988s) [noise=true , address=104.36.82.44:4893 , piece_id=HUJICRJ6W54O3AZZR6H3KG6E4TGBTNS46O6XVMQM3PFAZE7A2HZA, seg_pos={0 0}, upload=5911]: Done uploading piece. err: ecclient: upload failed (node:17VMFZ2T8jrSJVtchciJ9SnyrR6JstvcMoXrN5Q3zTD1aQU1oJ, address:104.36.82.44:4893): protocol: write tcp 159.223.143.39:35302->104.36.82.44:4893: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 04:08:17.678310218 (6h16m33.459243583s) [noise=true , address=104.36.82.42:4829 , piece_id=ZLWJFP3LGX3YLXBJYIR66DXTYEVDBKXKVXCQSF2PCYCYJBIEJGMA, seg_pos={0 0}, upload=5911]: Done uploading piece. err: ecclient: upload failed (node:1F5T8Vt1x4nXFaSenvGBCYjxhRL1H1sxRvzLjW3CqzwDVhq39F, address:104.36.82.42:4829): protocol: rpc: dial tcp 104.36.82.42:4829: i/o timeout; conn closed: op; piecestore: piecestore close: context deadline exceeded Jul 27 04:08:17.678412989 (6h16m33.459346365s) [noise=true , address=104.36.82.38:4722 , piece_id=B2MBQY5CTMEI6WSE7FTLOV5RUJ22SCNG7N2EEJILEATGJ4TIALXA, seg_pos={0 0}, upload=5911]: Done uploading piece. err: ecclient: upload failed (node:12Zw3dtUg9gARz52KhwDLTjcqC3CNm7dpPzpdx7P7PbL1jwe8uc, address:104.36.82.38:4722): protocol: read tcp 159.223.143.39:56048->104.36.82.38:4722: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 04:08:17.678455247 (6h16m33.459388624s) [noise=true , address=104.36.82.47:4524 , piece_id=DONJFTKUQ2XCW6BOU7AVOKJ7XMDMBPGQVUPRANH7X2YS7PJXUKIA, seg_pos={0 0}, upload=5911]: Done uploading piece. err: ecclient: upload failed (node:1xfbCzjSwFDRrR1gPggeN8tNCLbRRWDVzCJ6mqhZsbVSe8d4s4, address:104.36.82.47:4524): protocol: read tcp 159.223.143.39:46694->104.36.82.47:4524: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 04:08:17.678510668 (6h16m33.459444052s) [seg_pos={0 0}, upload=5911]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Jul 27 04:08:17.706442864 (6h16m33.487376232s) [upload=5911]: Done uploading segments... Jul 27 04:08:17.706495329 (6h16m33.487428705s) [upload=5911]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 649.764062ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............X..............................................] duration: 19.498547646s FAIL (2025-07-27 04:39:19 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== ece_id=NYOFDEZ7CV4PHGQ6JZMEEVAYGJRPYG6IBW7MYJO42HMWOJM7PHZA, seg_pos={0 0}, upload=5879]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12crcXpNe9NgeXTsUpooLSBgrSUFbRxbSbanLa39w2Hr5mGFww): protocol: read tcp 159.223.143.39:53630->104.36.82.5:4840: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 04:05:32.508904485 (6h13m48.289837866s) [seg_pos={0 0}, upload=5879]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 27 04:05:32.509028070 (6h13m48.289961457s) [upload=5879]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 27 04:05:32.509034645 (6h13m48.289968129s) [upload=5879]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 27 04:05:32.581834839 (6h13m48.362768225s) [upload=5879]: Done uploading segments... Jul 27 04:05:32.581875919 (6h13m48.362809298s) [upload=5879]: Upload finished. err: <nil> Jul 27 04:06:23.829155331 (6h14m39.610088696s) [upload=5895]: Starting upload Jul 27 04:06:23.829225021 (6h14m39.610158398s) [upload=5895]: Done starting upload Jul 27 04:06:23.829294606 (6h14m39.61022799s) [upload=5895]: Uploading segments... Jul 27 04:06:23.829577656 (6h14m39.610511053s) [upload=5895]: Got next segment. Inline: false Jul 27 04:06:23.829649664 (6h14m39.610583027s) [upload=5895]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 27 04:06:23.829657065 (6h14m39.61059043s) [upload=5895]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 27 04:06:23.861405530 (6h14m39.642338913s) [seg_pos={0 0}, upload=5895]: Begin upload segment... Jul 27 04:06:23.861476351 (6h14m39.642409713s) [seg_pos={0 0}, upload=5895]: Done begin upload segment. Jul 27 04:06:23.861509120 (6h14m39.642442506s) [noise=true , address=77.90.31.19:4843 , piece_id=YML5RDQIK432Z4FSE4DKXIV5U47LQBEZRFMAVLANUSGFUQR23UWQ, seg_pos={0 0}, upload=5895]: Uploading piece... Jul 27 04:06:23.861953723 (6h14m39.6428871s) [noise=true , address=45.91.116.10:4081 , piece_id=XGDTSEZOUNRTEVWHTO3DHTZMH2VZZOPJPEKLCFAHTXESLLYABHSA, seg_pos={0 0}, upload=5895]: Uploading piece... Jul 27 04:06:23.862060983 (6h14m39.642994358s) [noise=true , address=45.91.116.9:4262 , piece_id=M5ETD45HH2XZD6HDLTJ3JO2BOOIROXFWKVPTN42PSQEWG4F7KAAQ, seg_pos={0 0}, upload=5895]: Uploading piece... Jul 27 04:06:23.862759464 (6h14m39.64369284s) [noise=true , address=104.36.82.4:4804 , piece_id=IU6FDNVB5AUPA7MJ6RSXUTLBETPHOFGNJ7YGWBFIZSQFSO6W6BBA, seg_pos={0 0}, upload=5895]: Uploading piece... Jul 27 04:06:23.862835086 (6h14m39.643768461s) [noise=true , address=104.36.82.6:4873 , piece_id=YP4VJORZZUL5EVCH3P7LD3LNVMHB7CWRTKHGIA7UROXE44N6MMZA, seg_pos={0 0}, upload=5895]: Uploading piece... Jul 27 04:06:23.863593880 (6h14m39.644527258s) [noise=true , address=77.90.31.15:4697 , piece_id=Q76TJDSZPSYZSJQNFVOI462U6DFILMH2RV3LQISODXSOCXGRKW6Q, seg_pos={0 0}, upload=5895]: Uploading piece... Jul 27 04:06:23.863656891 (6h14m39.644590266s) [noise=true , address=45.91.116.7:4011 , piece_id=XMH6MCCHZBBRQWHJFWLRGYHLQD4KUGSGL7MMZSKA5TZJCND3DUDA, seg_pos={0 0}, upload=5895]: Uploading piece... Jul 27 04:06:23.864408612 (6h14m39.645341987s) [noise=true , address=45.91.116.6:4973 , piece_id=J3VULPU24CIM64VEVE3UHO62HPN6TJXGZU6BCJIS5JQJKHJK2LIQ, seg_pos={0 0}, upload=5895]: Uploading piece... Jul 27 04:06:23.864521549 (6h14m39.64545491s) [noise=true , address=104.36.82.10:4079 , piece_id=2SH5D6JOMLFR7VSN7ZY5CLMR5RHIRFYQ7Z5I2JIYTJQM43G4O6OQ, seg_pos={0 0}, upload=5895]: Uploading piece... Jul 27 04:06:23.865518491 (6h14m39.646451872s) [noise=true , address=77.90.31.17:4771 , piece_id=U2LF22GKCMKRZN6EWCF26ITDZQHBFZXCGSQXT33MCX6P57IGI35Q, seg_pos={0 0}, upload=5895]: Uploading piece... Jul 27 04:06:23.865654950 (6h14m39.646588325s) [noise=true , address=45.91.116.8:4048 , piece_id=5REWMVVTBVLDQXQ5KAEHLY4VJL5SGF7ONQWDQQ3N2FDU3OAO6AZA, seg_pos={0 0}, upload=5895]: Uploading piece... Jul 27 04:06:23.866190892 (6h14m39.647124271s) [noise=true , address=104.36.82.12:4151 , piece_id=E2TEUTQBD4Y4AUHSCPMCIFEXBSY7S5HJWBKNKWWTUWEW2BLAAE6Q, seg_pos={0 0}, upload=5895]: Uploading piece... Jul 27 04:06:23.866412590 (6h14m39.647345968s) [noise=true , address=104.36.82.11:4115 , piece_id=YXMNVT6ZHGESGQ2BXKZUVR7U2CNNAI2PO7ZTMDG5UIOV7HTDDUTA, seg_pos={0 0}, upload=5895]: Uploading piece... Jul 27 04:06:23.866715972 (6h14m39.647649359s) [noise=true , address=45.91.116.13:4153 , piece_id=H4535KOCG3F4RKA6AKKMTJSOM6Q6PGRC6M7FR3UIS3BORTSP5UQA, seg_pos={0 0}, upload=5895]: Uploading piece... Jul 27 04:06:23.866998965 (6h14m39.64793234s) [noise=true , address=104.36.82.13:4186 , piece_id=TBLDKALGJYBGR5NLCOCKJ4QKEFNIJXCKAFUBD7VR3UMNWEMKMKHQ, seg_pos={0 0}, upload=5895]: Uploading piece... Jul 27 04:06:23.867230593 (6h14m39.648163974s) [noise=true , address=104.36.82.16:4293 , piece_id=AHH6PRFUSGTDGB7H52DRITARA4IF6GFLVSRYS2Y6EPJCZSKUOYWQ, seg_pos={0 0}, upload=5895]: Uploading piece... Jul 27 04:06:23.949422808 (6h14m39.730356187s) [upload=5895]: Next returned nil segment Jul 27 04:06:23.949442576 (6h14m39.73037596s) [upload=5895]: Waiting for error group managing segments... Jul 27 04:06:23.953497426 (6h14m39.734430802s) [noise=true , address=45.91.116.10:4081 , piece_id=XGDTSEZOUNRTEVWHTO3DHTZMH2VZZOPJPEKLCFAHTXESLLYABHSA, seg_pos={0 0}, upload=5895]: Done uploading piece. err: <nil> Jul 27 04:06:23.953531796 (6h14m39.734465159s) [noise=true , address=45.91.116.6:4973 , piece_id=J3VULPU24CIM64VEVE3UHO62HPN6TJXGZU6BCJIS5JQJKHJK2LIQ, seg_pos={0 0}, upload=5895]: Done uploading piece. err: <nil> Jul 27 04:06:23.954068660 (6h14m39.735002029s) [noise=true , address=45.91.116.9:4262 , piece_id=M5ETD45HH2XZD6HDLTJ3JO2BOOIROXFWKVPTN42PSQEWG4F7KAAQ, seg_pos={0 0}, upload=5895]: Done uploading piece. err: <nil> Jul 27 04:06:23.954194339 (6h14m39.735127707s) [noise=true , address=45.91.116.8:4048 , piece_id=5REWMVVTBVLDQXQ5KAEHLY4VJL5SGF7ONQWDQQ3N2FDU3OAO6AZA, seg_pos={0 0}, upload=5895]: Done uploading piece. err: <nil> Jul 27 04:06:23.954601730 (6h14m39.735535112s) [noise=true , address=45.91.116.7:4011 , piece_id=XMH6MCCHZBBRQWHJFWLRGYHLQD4KUGSGL7MMZSKA5TZJCND3DUDA, seg_pos={0 0}, upload=5895]: Done uploading piece. err: <nil> Jul 27 04:06:23.954755872 (6h14m39.735689237s) [noise=true , address=45.91.116.13:4153 , piece_id=H4535KOCG3F4RKA6AKKMTJSOM6Q6PGRC6M7FR3UIS3BORTSP5UQA, seg_pos={0 0}, upload=5895]: Done uploading piece. err: <nil> Jul 27 04:06:23.955611883 (6h14m39.73654526s) [noise=true , address=77.90.31.19:4843 , piece_id=YML5RDQIK432Z4FSE4DKXIV5U47LQBEZRFMAVLANUSGFUQR23UWQ, seg_pos={0 0}, upload=5895]: Done uploading piece. err: <nil> Jul 27 04:06:23.956373391 (6h14m39.737306756s) [noise=true , address=77.90.31.15:4697 , piece_id=Q76TJDSZPSYZSJQNFVOI462U6DFILMH2RV3LQISODXSOCXGRKW6Q, seg_pos={0 0}, upload=5895]: Done uploading piece. err: <nil> Jul 27 04:06:23.958659122 (6h14m39.73959249s) [noise=true , address=77.90.31.17:4771 , piece_id=U2LF22GKCMKRZN6EWCF26ITDZQHBFZXCGSQXT33MCX6P57IGI35Q, seg_pos={0 0}, upload=5895]: Done uploading piece. err: <nil> Jul 27 04:06:35.058944013 (6h14m50.83987739s) [noise=true , address=104.36.82.16:4293 , piece_id=AHH6PRFUSGTDGB7H52DRITARA4IF6GFLVSRYS2Y6EPJCZSKUOYWQ, seg_pos={0 0}, upload=5895]: Done uploading piece. err: <nil> Jul 27 04:06:41.103204002 (6h14m56.884137386s) [noise=true , address=104.36.82.4:4804 , piece_id=IU6FDNVB5AUPA7MJ6RSXUTLBETPHOFGNJ7YGWBFIZSQFSO6W6BBA, seg_pos={0 0}, upload=5895]: Done uploading piece. err: <nil> Jul 27 04:06:43.191702285 (6h14m58.972635651s) [noise=true , address=104.36.82.6:4873 , piece_id=YP4VJORZZUL5EVCH3P7LD3LNVMHB7CWRTKHGIA7UROXE44N6MMZA, seg_pos={0 0}, upload=5895]: Done uploading piece. err: <nil> Jul 27 04:06:53.777973493 (6h15m9.558906861s) [noise=true , address=104.36.82.11:4115 , piece_id=YXMNVT6ZHGESGQ2BXKZUVR7U2CNNAI2PO7ZTMDG5UIOV7HTDDUTA, seg_pos={0 0}, upload=5895]: Done uploading piece. err: ecclient: upload failed (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR, address:104.36.82.11:4115): protocol: write tcp 159.223.143.39:34210->104.36.82.11:4115: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 04:06:53.778019532 (6h15m9.558952909s) [noise=true , address=104.36.82.12:4151 , piece_id=E2TEUTQBD4Y4AUHSCPMCIFEXBSY7S5HJWBKNKWWTUWEW2BLAAE6Q, seg_pos={0 0}, upload=5895]: Done uploading piece. err: ecclient: upload failed (node:1w6pwZMdeQKjt92qx6WbTWU5VPYJto5gS87f6iKvCBZH6LfyeB, address:104.36.82.12:4151): protocol: rpc: dial tcp 104.36.82.12:4151: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 04:06:53.778043653 (6h15m9.558977021s) [noise=true , address=104.36.82.13:4186 , piece_id=TBLDKALGJYBGR5NLCOCKJ4QKEFNIJXCKAFUBD7VR3UMNWEMKMKHQ, seg_pos={0 0}, upload=5895]: Done uploading piece. err: ecclient: upload failed (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht, address:104.36.82.13:4186): protocol: read tcp 159.223.143.39:34340->104.36.82.13:4186: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 04:06:53.778095228 (6h15m9.559028593s) [noise=true , address=104.36.82.10:4079 , piece_id=2SH5D6JOMLFR7VSN7ZY5CLMR5RHIRFYQ7Z5I2JIYTJQM43G4O6OQ, seg_pos={0 0}, upload=5895]: Done uploading piece. err: ecclient: upload failed (node:12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud, address:104.36.82.10:4079): protocol: write tcp 159.223.143.39:53840->104.36.82.10:4079: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 04:06:53.778142167 (6h15m9.559075532s) [seg_pos={0 0}, upload=5895]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 27 04:06:53.837032161 (6h15m9.617965543s) [upload=5895]: Done uploading segments... Jul 27 04:06:53.837055996 (6h15m9.617989366s) [upload=5895]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 19.498578136s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.................X..........................................] duration: 571.488229ms FAIL (2025-07-27 04:38:52 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== {0 0}, upload=5911]: Begin upload segment... Jul 27 04:07:17.745307658 (6h15m33.526241043s) [seg_pos={0 0}, upload=5911]: Done begin upload segment. Jul 27 04:07:17.745349241 (6h15m33.526282625s) [noise=true , address=104.36.82.52:4098 , piece_id=M2ETF25QQ53RRGFPUKXJRINIWV7AWXSECEDR2AIM645IED72NURQ, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.745439694 (6h15m33.526373073s) [noise=true , address=104.36.82.56:4244 , piece_id=LQCZ5W54NBHWAVBVYK7HYSSJDC73E6FV4CP4YIM4L5H2N3OLOZXA, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.745972271 (6h15m33.526905654s) [noise=true , address=104.36.82.36:4650 , piece_id=BKJULWGPESBINNTSUYP6E4KZRXKPACVU5I7DOSZBO4YMK2V26HEQ, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.745980243 (6h15m33.526913621s) [noise=true , address=104.36.82.38:4722 , piece_id=B2MBQY5CTMEI6WSE7FTLOV5RUJ22SCNG7N2EEJILEATGJ4TIALXA, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.746403818 (6h15m33.527337198s) [noise=true , address=104.36.82.22:4534 , piece_id=CW4AQT3XUGA4A5OC6NUY4NYOOAMLN5FFJJN5PQBYEON7HPD5LCYQ, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.746455493 (6h15m33.527388875s) [noise=true , address=104.36.82.14:4248 , piece_id=LDSGCUM7Q2VX5CDFI4JRGEEG7D5H5Q5Z5WUWKU277VN2JFRX6UGA, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.746872127 (6h15m33.527805505s) [noise=true , address=104.36.82.40:4954 , piece_id=YLOHTJLRAED66OMEQYPZHNNRI77V6WOR4EHTEZOOEHMGZHJ2O3QQ, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.747123162 (6h15m33.52805654s) [noise=true , address=104.36.82.46:4496 , piece_id=GWNQR5WRGRRQBLSGWJFD3O3UNCMNE62JQNPIV5SYKBUIL3BQKA5A, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.747280678 (6h15m33.528214053s) [noise=true , address=104.36.82.47:4524 , piece_id=DONJFTKUQ2XCW6BOU7AVOKJ7XMDMBPGQVUPRANH7X2YS7PJXUKIA, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.747736400 (6h15m33.528669779s) [noise=true , address=104.36.82.44:4893 , piece_id=HUJICRJ6W54O3AZZR6H3KG6E4TGBTNS46O6XVMQM3PFAZE7A2HZA, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.747755734 (6h15m33.528689109s) [noise=true , address=104.36.82.55:4204 , piece_id=72WISGRHMOAZDV5EDGTBOMLBL4HDOF3CBEMQAAIRGILPC7J5JKAQ, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.748313124 (6h15m33.5292465s) [noise=true , address=104.36.82.58:4287 , piece_id=BVDYUEWU7TVEQDMZXX5SJNS4QICVKMTGFLCUV6P2YHOE7TI25VXA, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.748485217 (6h15m33.529418594s) [noise=true , address=104.36.82.53:4123 , piece_id=NXJRZ5BQSLXSOIOSUJ4MM4AGKHP4UIDUGEJKLUUX7YOGQJA6WWZA, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.748831329 (6h15m33.529764704s) [noise=true , address=104.36.82.42:4829 , piece_id=ZLWJFP3LGX3YLXBJYIR66DXTYEVDBKXKVXCQSF2PCYCYJBIEJGMA, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.749299293 (6h15m33.530232669s) [noise=true , address=104.36.82.8:4002 , piece_id=6XSPWQC3US355PS6IQ4K3IYNQZH5MAT4IVIPGSYRUM3XYPSCWE6Q, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.749790974 (6h15m33.53072436s) [noise=true , address=104.36.82.44:4901 , piece_id=N4YLCLR2QZSULUL2XWONEKBFUMNOU7KUX2QEOK4SSDFSCI7IL6LA, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.750478221 (6h15m33.531411597s) [noise=true , address=104.36.82.54:4172 , piece_id=ZNA53MWDRWPUH6FUR4DJERBDT2IOYGA3IYUADKXXIOLQJLQOXE7Q, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.752884255 (6h15m33.533817625s) [noise=true , address=64.49.13.140:4508 , piece_id=MZ52U75VOIK6Q7QFXMMYRRG7JXFWDFU2JWQFYPERUCMH3BUX6XPQ, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.879839555 (6h15m33.660772917s) [upload=5911]: Next returned nil segment Jul 27 04:07:17.879861389 (6h15m33.660794765s) [upload=5911]: Waiting for error group managing segments... Jul 27 04:07:17.888652438 (6h15m33.669585811s) [noise=true , address=104.36.82.58:4287 , piece_id=BVDYUEWU7TVEQDMZXX5SJNS4QICVKMTGFLCUV6P2YHOE7TI25VXA, seg_pos={0 0}, upload=5911]: Done uploading piece. err: <nil> Jul 27 04:07:17.952192045 (6h15m33.733125429s) [noise=true , address=104.36.82.53:4123 , piece_id=NXJRZ5BQSLXSOIOSUJ4MM4AGKHP4UIDUGEJKLUUX7YOGQJA6WWZA, seg_pos={0 0}, upload=5911]: Done uploading piece. err: <nil> Jul 27 04:07:17.981907539 (6h15m33.762840911s) [noise=true , address=104.36.82.56:4244 , piece_id=LQCZ5W54NBHWAVBVYK7HYSSJDC73E6FV4CP4YIM4L5H2N3OLOZXA, seg_pos={0 0}, upload=5911]: Done uploading piece. err: <nil> Jul 27 04:07:17.982370496 (6h15m33.7633039s) [noise=true , address=104.36.82.55:4204 , piece_id=72WISGRHMOAZDV5EDGTBOMLBL4HDOF3CBEMQAAIRGILPC7J5JKAQ, seg_pos={0 0}, upload=5911]: Done uploading piece. err: <nil> Jul 27 04:07:18.009548506 (6h15m33.790481886s) [noise=true , address=104.36.82.52:4098 , piece_id=M2ETF25QQ53RRGFPUKXJRINIWV7AWXSECEDR2AIM645IED72NURQ, seg_pos={0 0}, upload=5911]: Done uploading piece. err: <nil> Jul 27 04:07:18.023770598 (6h15m33.804703983s) [noise=true , address=104.36.82.40:4954 , piece_id=YLOHTJLRAED66OMEQYPZHNNRI77V6WOR4EHTEZOOEHMGZHJ2O3QQ, seg_pos={0 0}, upload=5911]: Done uploading piece. err: <nil> Jul 27 04:07:18.139745581 (6h15m33.92067895s) [noise=true , address=104.36.82.22:4534 , piece_id=CW4AQT3XUGA4A5OC6NUY4NYOOAMLN5FFJJN5PQBYEON7HPD5LCYQ, seg_pos={0 0}, upload=5911]: Done uploading piece. err: <nil> Jul 27 04:07:18.423748140 (6h15m34.204681511s) [noise=true , address=64.49.13.140:4508 , piece_id=MZ52U75VOIK6Q7QFXMMYRRG7JXFWDFU2JWQFYPERUCMH3BUX6XPQ, seg_pos={0 0}, upload=5911]: Done uploading piece. err: <nil> Jul 27 04:08:16.739940381 (6h16m32.52087375s) [noise=true , address=104.36.82.54:4172 , piece_id=ZNA53MWDRWPUH6FUR4DJERBDT2IOYGA3IYUADKXXIOLQJLQOXE7Q, seg_pos={0 0}, upload=5911]: Done uploading piece. err: <nil> Jul 27 04:08:17.177295397 (6h16m32.958228766s) [noise=true , address=104.36.82.8:4002 , piece_id=6XSPWQC3US355PS6IQ4K3IYNQZH5MAT4IVIPGSYRUM3XYPSCWE6Q, seg_pos={0 0}, upload=5911]: Done uploading piece. err: <nil> Jul 27 04:08:17.534296584 (6h16m33.315229953s) [noise=true , address=104.36.82.36:4650 , piece_id=BKJULWGPESBINNTSUYP6E4KZRXKPACVU5I7DOSZBO4YMK2V26HEQ, seg_pos={0 0}, upload=5911]: Done uploading piece. err: <nil> Jul 27 04:08:17.677851566 (6h16m33.458784943s) [noise=true , address=104.36.82.14:4248 , piece_id=LDSGCUM7Q2VX5CDFI4JRGEEG7D5H5Q5Z5WUWKU277VN2JFRX6UGA, seg_pos={0 0}, upload=5911]: Done uploading piece. err: ecclient: upload failed (node:1u6ir2PLqDa9xGm64cuqq1RqY4qeBa1fqCo9VghspBf54XEXqn, address:104.36.82.14:4248): protocol: rpc: dial tcp 104.36.82.14:4248: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 04:08:17.678115170 (6h16m33.459048549s) [noise=true , address=104.36.82.46:4496 , piece_id=GWNQR5WRGRRQBLSGWJFD3O3UNCMNE62JQNPIV5SYKBUIL3BQKA5A, seg_pos={0 0}, upload=5911]: Done uploading piece. err: ecclient: upload failed (node:12V8XhSWkyGUVYFR8MykifFAP1YAfGCwzqucDcMMfzw3xsVW77Z, address:104.36.82.46:4496): protocol: read tcp 159.223.143.39:52800->104.36.82.46:4496: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 04:08:17.678144409 (6h16m33.459077789s) [noise=true , address=104.36.82.44:4901 , piece_id=N4YLCLR2QZSULUL2XWONEKBFUMNOU7KUX2QEOK4SSDFSCI7IL6LA, seg_pos={0 0}, upload=5911]: Done uploading piece. err: ecclient: upload failed (node:1FHE84zvBb1MKwB1Ht7puTafgQxDEvPt4mYqkYVSSufAMpWQA1, address:104.36.82.44:4901): protocol: rpc: dial tcp 104.36.82.44:4901: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 04:08:17.678229622 (6h16m33.459162988s) [noise=true , address=104.36.82.44:4893 , piece_id=HUJICRJ6W54O3AZZR6H3KG6E4TGBTNS46O6XVMQM3PFAZE7A2HZA, seg_pos={0 0}, upload=5911]: Done uploading piece. err: ecclient: upload failed (node:17VMFZ2T8jrSJVtchciJ9SnyrR6JstvcMoXrN5Q3zTD1aQU1oJ, address:104.36.82.44:4893): protocol: write tcp 159.223.143.39:35302->104.36.82.44:4893: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 04:08:17.678310218 (6h16m33.459243583s) [noise=true , address=104.36.82.42:4829 , piece_id=ZLWJFP3LGX3YLXBJYIR66DXTYEVDBKXKVXCQSF2PCYCYJBIEJGMA, seg_pos={0 0}, upload=5911]: Done uploading piece. err: ecclient: upload failed (node:1F5T8Vt1x4nXFaSenvGBCYjxhRL1H1sxRvzLjW3CqzwDVhq39F, address:104.36.82.42:4829): protocol: rpc: dial tcp 104.36.82.42:4829: i/o timeout; conn closed: op; piecestore: piecestore close: context deadline exceeded Jul 27 04:08:17.678412989 (6h16m33.459346365s) [noise=true , address=104.36.82.38:4722 , piece_id=B2MBQY5CTMEI6WSE7FTLOV5RUJ22SCNG7N2EEJILEATGJ4TIALXA, seg_pos={0 0}, upload=5911]: Done uploading piece. err: ecclient: upload failed (node:12Zw3dtUg9gARz52KhwDLTjcqC3CNm7dpPzpdx7P7PbL1jwe8uc, address:104.36.82.38:4722): protocol: read tcp 159.223.143.39:56048->104.36.82.38:4722: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 04:08:17.678455247 (6h16m33.459388624s) [noise=true , address=104.36.82.47:4524 , piece_id=DONJFTKUQ2XCW6BOU7AVOKJ7XMDMBPGQVUPRANH7X2YS7PJXUKIA, seg_pos={0 0}, upload=5911]: Done uploading piece. err: ecclient: upload failed (node:1xfbCzjSwFDRrR1gPggeN8tNCLbRRWDVzCJ6mqhZsbVSe8d4s4, address:104.36.82.47:4524): protocol: read tcp 159.223.143.39:46694->104.36.82.47:4524: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 04:08:17.678510668 (6h16m33.459444052s) [seg_pos={0 0}, upload=5911]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Jul 27 04:08:17.706442864 (6h16m33.487376232s) [upload=5911]: Done uploading segments... Jul 27 04:08:17.706495329 (6h16m33.487428705s) [upload=5911]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 571.524964ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................X................................] duration: 485.659826ms FAIL (2025-07-27 04:38:43 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 12) Duration: 485.6932ms core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [............................X...............................] duration: 434.436612ms FAIL (2025-07-27 04:38:21 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 12) Duration: 434.46298ms core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [..................X.........................................] duration: 10.851021139s FAIL (2025-07-27 04:38:14 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== {0 0}, upload=5911]: Begin upload segment... Jul 27 04:07:17.745307658 (6h15m33.526241043s) [seg_pos={0 0}, upload=5911]: Done begin upload segment. Jul 27 04:07:17.745349241 (6h15m33.526282625s) [noise=true , address=104.36.82.52:4098 , piece_id=M2ETF25QQ53RRGFPUKXJRINIWV7AWXSECEDR2AIM645IED72NURQ, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.745439694 (6h15m33.526373073s) [noise=true , address=104.36.82.56:4244 , piece_id=LQCZ5W54NBHWAVBVYK7HYSSJDC73E6FV4CP4YIM4L5H2N3OLOZXA, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.745972271 (6h15m33.526905654s) [noise=true , address=104.36.82.36:4650 , piece_id=BKJULWGPESBINNTSUYP6E4KZRXKPACVU5I7DOSZBO4YMK2V26HEQ, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.745980243 (6h15m33.526913621s) [noise=true , address=104.36.82.38:4722 , piece_id=B2MBQY5CTMEI6WSE7FTLOV5RUJ22SCNG7N2EEJILEATGJ4TIALXA, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.746403818 (6h15m33.527337198s) [noise=true , address=104.36.82.22:4534 , piece_id=CW4AQT3XUGA4A5OC6NUY4NYOOAMLN5FFJJN5PQBYEON7HPD5LCYQ, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.746455493 (6h15m33.527388875s) [noise=true , address=104.36.82.14:4248 , piece_id=LDSGCUM7Q2VX5CDFI4JRGEEG7D5H5Q5Z5WUWKU277VN2JFRX6UGA, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.746872127 (6h15m33.527805505s) [noise=true , address=104.36.82.40:4954 , piece_id=YLOHTJLRAED66OMEQYPZHNNRI77V6WOR4EHTEZOOEHMGZHJ2O3QQ, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.747123162 (6h15m33.52805654s) [noise=true , address=104.36.82.46:4496 , piece_id=GWNQR5WRGRRQBLSGWJFD3O3UNCMNE62JQNPIV5SYKBUIL3BQKA5A, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.747280678 (6h15m33.528214053s) [noise=true , address=104.36.82.47:4524 , piece_id=DONJFTKUQ2XCW6BOU7AVOKJ7XMDMBPGQVUPRANH7X2YS7PJXUKIA, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.747736400 (6h15m33.528669779s) [noise=true , address=104.36.82.44:4893 , piece_id=HUJICRJ6W54O3AZZR6H3KG6E4TGBTNS46O6XVMQM3PFAZE7A2HZA, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.747755734 (6h15m33.528689109s) [noise=true , address=104.36.82.55:4204 , piece_id=72WISGRHMOAZDV5EDGTBOMLBL4HDOF3CBEMQAAIRGILPC7J5JKAQ, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.748313124 (6h15m33.5292465s) [noise=true , address=104.36.82.58:4287 , piece_id=BVDYUEWU7TVEQDMZXX5SJNS4QICVKMTGFLCUV6P2YHOE7TI25VXA, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.748485217 (6h15m33.529418594s) [noise=true , address=104.36.82.53:4123 , piece_id=NXJRZ5BQSLXSOIOSUJ4MM4AGKHP4UIDUGEJKLUUX7YOGQJA6WWZA, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.748831329 (6h15m33.529764704s) [noise=true , address=104.36.82.42:4829 , piece_id=ZLWJFP3LGX3YLXBJYIR66DXTYEVDBKXKVXCQSF2PCYCYJBIEJGMA, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.749299293 (6h15m33.530232669s) [noise=true , address=104.36.82.8:4002 , piece_id=6XSPWQC3US355PS6IQ4K3IYNQZH5MAT4IVIPGSYRUM3XYPSCWE6Q, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.749790974 (6h15m33.53072436s) [noise=true , address=104.36.82.44:4901 , piece_id=N4YLCLR2QZSULUL2XWONEKBFUMNOU7KUX2QEOK4SSDFSCI7IL6LA, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.750478221 (6h15m33.531411597s) [noise=true , address=104.36.82.54:4172 , piece_id=ZNA53MWDRWPUH6FUR4DJERBDT2IOYGA3IYUADKXXIOLQJLQOXE7Q, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.752884255 (6h15m33.533817625s) [noise=true , address=64.49.13.140:4508 , piece_id=MZ52U75VOIK6Q7QFXMMYRRG7JXFWDFU2JWQFYPERUCMH3BUX6XPQ, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.879839555 (6h15m33.660772917s) [upload=5911]: Next returned nil segment Jul 27 04:07:17.879861389 (6h15m33.660794765s) [upload=5911]: Waiting for error group managing segments... Jul 27 04:07:17.888652438 (6h15m33.669585811s) [noise=true , address=104.36.82.58:4287 , piece_id=BVDYUEWU7TVEQDMZXX5SJNS4QICVKMTGFLCUV6P2YHOE7TI25VXA, seg_pos={0 0}, upload=5911]: Done uploading piece. err: <nil> Jul 27 04:07:17.952192045 (6h15m33.733125429s) [noise=true , address=104.36.82.53:4123 , piece_id=NXJRZ5BQSLXSOIOSUJ4MM4AGKHP4UIDUGEJKLUUX7YOGQJA6WWZA, seg_pos={0 0}, upload=5911]: Done uploading piece. err: <nil> Jul 27 04:07:17.981907539 (6h15m33.762840911s) [noise=true , address=104.36.82.56:4244 , piece_id=LQCZ5W54NBHWAVBVYK7HYSSJDC73E6FV4CP4YIM4L5H2N3OLOZXA, seg_pos={0 0}, upload=5911]: Done uploading piece. err: <nil> Jul 27 04:07:17.982370496 (6h15m33.7633039s) [noise=true , address=104.36.82.55:4204 , piece_id=72WISGRHMOAZDV5EDGTBOMLBL4HDOF3CBEMQAAIRGILPC7J5JKAQ, seg_pos={0 0}, upload=5911]: Done uploading piece. err: <nil> Jul 27 04:07:18.009548506 (6h15m33.790481886s) [noise=true , address=104.36.82.52:4098 , piece_id=M2ETF25QQ53RRGFPUKXJRINIWV7AWXSECEDR2AIM645IED72NURQ, seg_pos={0 0}, upload=5911]: Done uploading piece. err: <nil> Jul 27 04:07:18.023770598 (6h15m33.804703983s) [noise=true , address=104.36.82.40:4954 , piece_id=YLOHTJLRAED66OMEQYPZHNNRI77V6WOR4EHTEZOOEHMGZHJ2O3QQ, seg_pos={0 0}, upload=5911]: Done uploading piece. err: <nil> Jul 27 04:07:18.139745581 (6h15m33.92067895s) [noise=true , address=104.36.82.22:4534 , piece_id=CW4AQT3XUGA4A5OC6NUY4NYOOAMLN5FFJJN5PQBYEON7HPD5LCYQ, seg_pos={0 0}, upload=5911]: Done uploading piece. err: <nil> Jul 27 04:07:18.423748140 (6h15m34.204681511s) [noise=true , address=64.49.13.140:4508 , piece_id=MZ52U75VOIK6Q7QFXMMYRRG7JXFWDFU2JWQFYPERUCMH3BUX6XPQ, seg_pos={0 0}, upload=5911]: Done uploading piece. err: <nil> Jul 27 04:08:16.739940381 (6h16m32.52087375s) [noise=true , address=104.36.82.54:4172 , piece_id=ZNA53MWDRWPUH6FUR4DJERBDT2IOYGA3IYUADKXXIOLQJLQOXE7Q, seg_pos={0 0}, upload=5911]: Done uploading piece. err: <nil> Jul 27 04:08:17.177295397 (6h16m32.958228766s) [noise=true , address=104.36.82.8:4002 , piece_id=6XSPWQC3US355PS6IQ4K3IYNQZH5MAT4IVIPGSYRUM3XYPSCWE6Q, seg_pos={0 0}, upload=5911]: Done uploading piece. err: <nil> Jul 27 04:08:17.534296584 (6h16m33.315229953s) [noise=true , address=104.36.82.36:4650 , piece_id=BKJULWGPESBINNTSUYP6E4KZRXKPACVU5I7DOSZBO4YMK2V26HEQ, seg_pos={0 0}, upload=5911]: Done uploading piece. err: <nil> Jul 27 04:08:17.677851566 (6h16m33.458784943s) [noise=true , address=104.36.82.14:4248 , piece_id=LDSGCUM7Q2VX5CDFI4JRGEEG7D5H5Q5Z5WUWKU277VN2JFRX6UGA, seg_pos={0 0}, upload=5911]: Done uploading piece. err: ecclient: upload failed (node:1u6ir2PLqDa9xGm64cuqq1RqY4qeBa1fqCo9VghspBf54XEXqn, address:104.36.82.14:4248): protocol: rpc: dial tcp 104.36.82.14:4248: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 04:08:17.678115170 (6h16m33.459048549s) [noise=true , address=104.36.82.46:4496 , piece_id=GWNQR5WRGRRQBLSGWJFD3O3UNCMNE62JQNPIV5SYKBUIL3BQKA5A, seg_pos={0 0}, upload=5911]: Done uploading piece. err: ecclient: upload failed (node:12V8XhSWkyGUVYFR8MykifFAP1YAfGCwzqucDcMMfzw3xsVW77Z, address:104.36.82.46:4496): protocol: read tcp 159.223.143.39:52800->104.36.82.46:4496: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 04:08:17.678144409 (6h16m33.459077789s) [noise=true , address=104.36.82.44:4901 , piece_id=N4YLCLR2QZSULUL2XWONEKBFUMNOU7KUX2QEOK4SSDFSCI7IL6LA, seg_pos={0 0}, upload=5911]: Done uploading piece. err: ecclient: upload failed (node:1FHE84zvBb1MKwB1Ht7puTafgQxDEvPt4mYqkYVSSufAMpWQA1, address:104.36.82.44:4901): protocol: rpc: dial tcp 104.36.82.44:4901: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 04:08:17.678229622 (6h16m33.459162988s) [noise=true , address=104.36.82.44:4893 , piece_id=HUJICRJ6W54O3AZZR6H3KG6E4TGBTNS46O6XVMQM3PFAZE7A2HZA, seg_pos={0 0}, upload=5911]: Done uploading piece. err: ecclient: upload failed (node:17VMFZ2T8jrSJVtchciJ9SnyrR6JstvcMoXrN5Q3zTD1aQU1oJ, address:104.36.82.44:4893): protocol: write tcp 159.223.143.39:35302->104.36.82.44:4893: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 04:08:17.678310218 (6h16m33.459243583s) [noise=true , address=104.36.82.42:4829 , piece_id=ZLWJFP3LGX3YLXBJYIR66DXTYEVDBKXKVXCQSF2PCYCYJBIEJGMA, seg_pos={0 0}, upload=5911]: Done uploading piece. err: ecclient: upload failed (node:1F5T8Vt1x4nXFaSenvGBCYjxhRL1H1sxRvzLjW3CqzwDVhq39F, address:104.36.82.42:4829): protocol: rpc: dial tcp 104.36.82.42:4829: i/o timeout; conn closed: op; piecestore: piecestore close: context deadline exceeded Jul 27 04:08:17.678412989 (6h16m33.459346365s) [noise=true , address=104.36.82.38:4722 , piece_id=B2MBQY5CTMEI6WSE7FTLOV5RUJ22SCNG7N2EEJILEATGJ4TIALXA, seg_pos={0 0}, upload=5911]: Done uploading piece. err: ecclient: upload failed (node:12Zw3dtUg9gARz52KhwDLTjcqC3CNm7dpPzpdx7P7PbL1jwe8uc, address:104.36.82.38:4722): protocol: read tcp 159.223.143.39:56048->104.36.82.38:4722: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 04:08:17.678455247 (6h16m33.459388624s) [noise=true , address=104.36.82.47:4524 , piece_id=DONJFTKUQ2XCW6BOU7AVOKJ7XMDMBPGQVUPRANH7X2YS7PJXUKIA, seg_pos={0 0}, upload=5911]: Done uploading piece. err: ecclient: upload failed (node:1xfbCzjSwFDRrR1gPggeN8tNCLbRRWDVzCJ6mqhZsbVSe8d4s4, address:104.36.82.47:4524): protocol: read tcp 159.223.143.39:46694->104.36.82.47:4524: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 04:08:17.678510668 (6h16m33.459444052s) [seg_pos={0 0}, upload=5911]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Jul 27 04:08:17.706442864 (6h16m33.487376232s) [upload=5911]: Done uploading segments... Jul 27 04:08:17.706495329 (6h16m33.487428705s) [upload=5911]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 10.851055636s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [...................X........................................] duration: 16.463309465s FAIL (2025-07-27 04:37:55 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== {0 0}, upload=5911]: Begin upload segment... Jul 27 04:07:17.745307658 (6h15m33.526241043s) [seg_pos={0 0}, upload=5911]: Done begin upload segment. Jul 27 04:07:17.745349241 (6h15m33.526282625s) [noise=true , address=104.36.82.52:4098 , piece_id=M2ETF25QQ53RRGFPUKXJRINIWV7AWXSECEDR2AIM645IED72NURQ, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.745439694 (6h15m33.526373073s) [noise=true , address=104.36.82.56:4244 , piece_id=LQCZ5W54NBHWAVBVYK7HYSSJDC73E6FV4CP4YIM4L5H2N3OLOZXA, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.745972271 (6h15m33.526905654s) [noise=true , address=104.36.82.36:4650 , piece_id=BKJULWGPESBINNTSUYP6E4KZRXKPACVU5I7DOSZBO4YMK2V26HEQ, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.745980243 (6h15m33.526913621s) [noise=true , address=104.36.82.38:4722 , piece_id=B2MBQY5CTMEI6WSE7FTLOV5RUJ22SCNG7N2EEJILEATGJ4TIALXA, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.746403818 (6h15m33.527337198s) [noise=true , address=104.36.82.22:4534 , piece_id=CW4AQT3XUGA4A5OC6NUY4NYOOAMLN5FFJJN5PQBYEON7HPD5LCYQ, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.746455493 (6h15m33.527388875s) [noise=true , address=104.36.82.14:4248 , piece_id=LDSGCUM7Q2VX5CDFI4JRGEEG7D5H5Q5Z5WUWKU277VN2JFRX6UGA, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.746872127 (6h15m33.527805505s) [noise=true , address=104.36.82.40:4954 , piece_id=YLOHTJLRAED66OMEQYPZHNNRI77V6WOR4EHTEZOOEHMGZHJ2O3QQ, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.747123162 (6h15m33.52805654s) [noise=true , address=104.36.82.46:4496 , piece_id=GWNQR5WRGRRQBLSGWJFD3O3UNCMNE62JQNPIV5SYKBUIL3BQKA5A, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.747280678 (6h15m33.528214053s) [noise=true , address=104.36.82.47:4524 , piece_id=DONJFTKUQ2XCW6BOU7AVOKJ7XMDMBPGQVUPRANH7X2YS7PJXUKIA, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.747736400 (6h15m33.528669779s) [noise=true , address=104.36.82.44:4893 , piece_id=HUJICRJ6W54O3AZZR6H3KG6E4TGBTNS46O6XVMQM3PFAZE7A2HZA, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.747755734 (6h15m33.528689109s) [noise=true , address=104.36.82.55:4204 , piece_id=72WISGRHMOAZDV5EDGTBOMLBL4HDOF3CBEMQAAIRGILPC7J5JKAQ, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.748313124 (6h15m33.5292465s) [noise=true , address=104.36.82.58:4287 , piece_id=BVDYUEWU7TVEQDMZXX5SJNS4QICVKMTGFLCUV6P2YHOE7TI25VXA, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.748485217 (6h15m33.529418594s) [noise=true , address=104.36.82.53:4123 , piece_id=NXJRZ5BQSLXSOIOSUJ4MM4AGKHP4UIDUGEJKLUUX7YOGQJA6WWZA, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.748831329 (6h15m33.529764704s) [noise=true , address=104.36.82.42:4829 , piece_id=ZLWJFP3LGX3YLXBJYIR66DXTYEVDBKXKVXCQSF2PCYCYJBIEJGMA, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.749299293 (6h15m33.530232669s) [noise=true , address=104.36.82.8:4002 , piece_id=6XSPWQC3US355PS6IQ4K3IYNQZH5MAT4IVIPGSYRUM3XYPSCWE6Q, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.749790974 (6h15m33.53072436s) [noise=true , address=104.36.82.44:4901 , piece_id=N4YLCLR2QZSULUL2XWONEKBFUMNOU7KUX2QEOK4SSDFSCI7IL6LA, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.750478221 (6h15m33.531411597s) [noise=true , address=104.36.82.54:4172 , piece_id=ZNA53MWDRWPUH6FUR4DJERBDT2IOYGA3IYUADKXXIOLQJLQOXE7Q, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.752884255 (6h15m33.533817625s) [noise=true , address=64.49.13.140:4508 , piece_id=MZ52U75VOIK6Q7QFXMMYRRG7JXFWDFU2JWQFYPERUCMH3BUX6XPQ, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.879839555 (6h15m33.660772917s) [upload=5911]: Next returned nil segment Jul 27 04:07:17.879861389 (6h15m33.660794765s) [upload=5911]: Waiting for error group managing segments... Jul 27 04:07:17.888652438 (6h15m33.669585811s) [noise=true , address=104.36.82.58:4287 , piece_id=BVDYUEWU7TVEQDMZXX5SJNS4QICVKMTGFLCUV6P2YHOE7TI25VXA, seg_pos={0 0}, upload=5911]: Done uploading piece. err: <nil> Jul 27 04:07:17.952192045 (6h15m33.733125429s) [noise=true , address=104.36.82.53:4123 , piece_id=NXJRZ5BQSLXSOIOSUJ4MM4AGKHP4UIDUGEJKLUUX7YOGQJA6WWZA, seg_pos={0 0}, upload=5911]: Done uploading piece. err: <nil> Jul 27 04:07:17.981907539 (6h15m33.762840911s) [noise=true , address=104.36.82.56:4244 , piece_id=LQCZ5W54NBHWAVBVYK7HYSSJDC73E6FV4CP4YIM4L5H2N3OLOZXA, seg_pos={0 0}, upload=5911]: Done uploading piece. err: <nil> Jul 27 04:07:17.982370496 (6h15m33.7633039s) [noise=true , address=104.36.82.55:4204 , piece_id=72WISGRHMOAZDV5EDGTBOMLBL4HDOF3CBEMQAAIRGILPC7J5JKAQ, seg_pos={0 0}, upload=5911]: Done uploading piece. err: <nil> Jul 27 04:07:18.009548506 (6h15m33.790481886s) [noise=true , address=104.36.82.52:4098 , piece_id=M2ETF25QQ53RRGFPUKXJRINIWV7AWXSECEDR2AIM645IED72NURQ, seg_pos={0 0}, upload=5911]: Done uploading piece. err: <nil> Jul 27 04:07:18.023770598 (6h15m33.804703983s) [noise=true , address=104.36.82.40:4954 , piece_id=YLOHTJLRAED66OMEQYPZHNNRI77V6WOR4EHTEZOOEHMGZHJ2O3QQ, seg_pos={0 0}, upload=5911]: Done uploading piece. err: <nil> Jul 27 04:07:18.139745581 (6h15m33.92067895s) [noise=true , address=104.36.82.22:4534 , piece_id=CW4AQT3XUGA4A5OC6NUY4NYOOAMLN5FFJJN5PQBYEON7HPD5LCYQ, seg_pos={0 0}, upload=5911]: Done uploading piece. err: <nil> Jul 27 04:07:18.423748140 (6h15m34.204681511s) [noise=true , address=64.49.13.140:4508 , piece_id=MZ52U75VOIK6Q7QFXMMYRRG7JXFWDFU2JWQFYPERUCMH3BUX6XPQ, seg_pos={0 0}, upload=5911]: Done uploading piece. err: <nil> Jul 27 04:08:16.739940381 (6h16m32.52087375s) [noise=true , address=104.36.82.54:4172 , piece_id=ZNA53MWDRWPUH6FUR4DJERBDT2IOYGA3IYUADKXXIOLQJLQOXE7Q, seg_pos={0 0}, upload=5911]: Done uploading piece. err: <nil> Jul 27 04:08:17.177295397 (6h16m32.958228766s) [noise=true , address=104.36.82.8:4002 , piece_id=6XSPWQC3US355PS6IQ4K3IYNQZH5MAT4IVIPGSYRUM3XYPSCWE6Q, seg_pos={0 0}, upload=5911]: Done uploading piece. err: <nil> Jul 27 04:08:17.534296584 (6h16m33.315229953s) [noise=true , address=104.36.82.36:4650 , piece_id=BKJULWGPESBINNTSUYP6E4KZRXKPACVU5I7DOSZBO4YMK2V26HEQ, seg_pos={0 0}, upload=5911]: Done uploading piece. err: <nil> Jul 27 04:08:17.677851566 (6h16m33.458784943s) [noise=true , address=104.36.82.14:4248 , piece_id=LDSGCUM7Q2VX5CDFI4JRGEEG7D5H5Q5Z5WUWKU277VN2JFRX6UGA, seg_pos={0 0}, upload=5911]: Done uploading piece. err: ecclient: upload failed (node:1u6ir2PLqDa9xGm64cuqq1RqY4qeBa1fqCo9VghspBf54XEXqn, address:104.36.82.14:4248): protocol: rpc: dial tcp 104.36.82.14:4248: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 04:08:17.678115170 (6h16m33.459048549s) [noise=true , address=104.36.82.46:4496 , piece_id=GWNQR5WRGRRQBLSGWJFD3O3UNCMNE62JQNPIV5SYKBUIL3BQKA5A, seg_pos={0 0}, upload=5911]: Done uploading piece. err: ecclient: upload failed (node:12V8XhSWkyGUVYFR8MykifFAP1YAfGCwzqucDcMMfzw3xsVW77Z, address:104.36.82.46:4496): protocol: read tcp 159.223.143.39:52800->104.36.82.46:4496: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 04:08:17.678144409 (6h16m33.459077789s) [noise=true , address=104.36.82.44:4901 , piece_id=N4YLCLR2QZSULUL2XWONEKBFUMNOU7KUX2QEOK4SSDFSCI7IL6LA, seg_pos={0 0}, upload=5911]: Done uploading piece. err: ecclient: upload failed (node:1FHE84zvBb1MKwB1Ht7puTafgQxDEvPt4mYqkYVSSufAMpWQA1, address:104.36.82.44:4901): protocol: rpc: dial tcp 104.36.82.44:4901: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 04:08:17.678229622 (6h16m33.459162988s) [noise=true , address=104.36.82.44:4893 , piece_id=HUJICRJ6W54O3AZZR6H3KG6E4TGBTNS46O6XVMQM3PFAZE7A2HZA, seg_pos={0 0}, upload=5911]: Done uploading piece. err: ecclient: upload failed (node:17VMFZ2T8jrSJVtchciJ9SnyrR6JstvcMoXrN5Q3zTD1aQU1oJ, address:104.36.82.44:4893): protocol: write tcp 159.223.143.39:35302->104.36.82.44:4893: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 04:08:17.678310218 (6h16m33.459243583s) [noise=true , address=104.36.82.42:4829 , piece_id=ZLWJFP3LGX3YLXBJYIR66DXTYEVDBKXKVXCQSF2PCYCYJBIEJGMA, seg_pos={0 0}, upload=5911]: Done uploading piece. err: ecclient: upload failed (node:1F5T8Vt1x4nXFaSenvGBCYjxhRL1H1sxRvzLjW3CqzwDVhq39F, address:104.36.82.42:4829): protocol: rpc: dial tcp 104.36.82.42:4829: i/o timeout; conn closed: op; piecestore: piecestore close: context deadline exceeded Jul 27 04:08:17.678412989 (6h16m33.459346365s) [noise=true , address=104.36.82.38:4722 , piece_id=B2MBQY5CTMEI6WSE7FTLOV5RUJ22SCNG7N2EEJILEATGJ4TIALXA, seg_pos={0 0}, upload=5911]: Done uploading piece. err: ecclient: upload failed (node:12Zw3dtUg9gARz52KhwDLTjcqC3CNm7dpPzpdx7P7PbL1jwe8uc, address:104.36.82.38:4722): protocol: read tcp 159.223.143.39:56048->104.36.82.38:4722: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 04:08:17.678455247 (6h16m33.459388624s) [noise=true , address=104.36.82.47:4524 , piece_id=DONJFTKUQ2XCW6BOU7AVOKJ7XMDMBPGQVUPRANH7X2YS7PJXUKIA, seg_pos={0 0}, upload=5911]: Done uploading piece. err: ecclient: upload failed (node:1xfbCzjSwFDRrR1gPggeN8tNCLbRRWDVzCJ6mqhZsbVSe8d4s4, address:104.36.82.47:4524): protocol: read tcp 159.223.143.39:46694->104.36.82.47:4524: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 04:08:17.678510668 (6h16m33.459444052s) [seg_pos={0 0}, upload=5911]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Jul 27 04:08:17.706442864 (6h16m33.487376232s) [upload=5911]: Done uploading segments... Jul 27 04:08:17.706495329 (6h16m33.487428705s) [upload=5911]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 16.463350252s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............X.............................................] duration: 657.672414ms FAIL (2025-07-27 04:37:54 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== ece_id=NYOFDEZ7CV4PHGQ6JZMEEVAYGJRPYG6IBW7MYJO42HMWOJM7PHZA, seg_pos={0 0}, upload=5879]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12crcXpNe9NgeXTsUpooLSBgrSUFbRxbSbanLa39w2Hr5mGFww): protocol: read tcp 159.223.143.39:53630->104.36.82.5:4840: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 04:05:32.508904485 (6h13m48.289837866s) [seg_pos={0 0}, upload=5879]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 27 04:05:32.509028070 (6h13m48.289961457s) [upload=5879]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 27 04:05:32.509034645 (6h13m48.289968129s) [upload=5879]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 27 04:05:32.581834839 (6h13m48.362768225s) [upload=5879]: Done uploading segments... Jul 27 04:05:32.581875919 (6h13m48.362809298s) [upload=5879]: Upload finished. err: <nil> Jul 27 04:06:23.829155331 (6h14m39.610088696s) [upload=5895]: Starting upload Jul 27 04:06:23.829225021 (6h14m39.610158398s) [upload=5895]: Done starting upload Jul 27 04:06:23.829294606 (6h14m39.61022799s) [upload=5895]: Uploading segments... Jul 27 04:06:23.829577656 (6h14m39.610511053s) [upload=5895]: Got next segment. Inline: false Jul 27 04:06:23.829649664 (6h14m39.610583027s) [upload=5895]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 27 04:06:23.829657065 (6h14m39.61059043s) [upload=5895]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 27 04:06:23.861405530 (6h14m39.642338913s) [seg_pos={0 0}, upload=5895]: Begin upload segment... Jul 27 04:06:23.861476351 (6h14m39.642409713s) [seg_pos={0 0}, upload=5895]: Done begin upload segment. Jul 27 04:06:23.861509120 (6h14m39.642442506s) [noise=true , address=77.90.31.19:4843 , piece_id=YML5RDQIK432Z4FSE4DKXIV5U47LQBEZRFMAVLANUSGFUQR23UWQ, seg_pos={0 0}, upload=5895]: Uploading piece... Jul 27 04:06:23.861953723 (6h14m39.6428871s) [noise=true , address=45.91.116.10:4081 , piece_id=XGDTSEZOUNRTEVWHTO3DHTZMH2VZZOPJPEKLCFAHTXESLLYABHSA, seg_pos={0 0}, upload=5895]: Uploading piece... Jul 27 04:06:23.862060983 (6h14m39.642994358s) [noise=true , address=45.91.116.9:4262 , piece_id=M5ETD45HH2XZD6HDLTJ3JO2BOOIROXFWKVPTN42PSQEWG4F7KAAQ, seg_pos={0 0}, upload=5895]: Uploading piece... Jul 27 04:06:23.862759464 (6h14m39.64369284s) [noise=true , address=104.36.82.4:4804 , piece_id=IU6FDNVB5AUPA7MJ6RSXUTLBETPHOFGNJ7YGWBFIZSQFSO6W6BBA, seg_pos={0 0}, upload=5895]: Uploading piece... Jul 27 04:06:23.862835086 (6h14m39.643768461s) [noise=true , address=104.36.82.6:4873 , piece_id=YP4VJORZZUL5EVCH3P7LD3LNVMHB7CWRTKHGIA7UROXE44N6MMZA, seg_pos={0 0}, upload=5895]: Uploading piece... Jul 27 04:06:23.863593880 (6h14m39.644527258s) [noise=true , address=77.90.31.15:4697 , piece_id=Q76TJDSZPSYZSJQNFVOI462U6DFILMH2RV3LQISODXSOCXGRKW6Q, seg_pos={0 0}, upload=5895]: Uploading piece... Jul 27 04:06:23.863656891 (6h14m39.644590266s) [noise=true , address=45.91.116.7:4011 , piece_id=XMH6MCCHZBBRQWHJFWLRGYHLQD4KUGSGL7MMZSKA5TZJCND3DUDA, seg_pos={0 0}, upload=5895]: Uploading piece... Jul 27 04:06:23.864408612 (6h14m39.645341987s) [noise=true , address=45.91.116.6:4973 , piece_id=J3VULPU24CIM64VEVE3UHO62HPN6TJXGZU6BCJIS5JQJKHJK2LIQ, seg_pos={0 0}, upload=5895]: Uploading piece... Jul 27 04:06:23.864521549 (6h14m39.64545491s) [noise=true , address=104.36.82.10:4079 , piece_id=2SH5D6JOMLFR7VSN7ZY5CLMR5RHIRFYQ7Z5I2JIYTJQM43G4O6OQ, seg_pos={0 0}, upload=5895]: Uploading piece... Jul 27 04:06:23.865518491 (6h14m39.646451872s) [noise=true , address=77.90.31.17:4771 , piece_id=U2LF22GKCMKRZN6EWCF26ITDZQHBFZXCGSQXT33MCX6P57IGI35Q, seg_pos={0 0}, upload=5895]: Uploading piece... Jul 27 04:06:23.865654950 (6h14m39.646588325s) [noise=true , address=45.91.116.8:4048 , piece_id=5REWMVVTBVLDQXQ5KAEHLY4VJL5SGF7ONQWDQQ3N2FDU3OAO6AZA, seg_pos={0 0}, upload=5895]: Uploading piece... Jul 27 04:06:23.866190892 (6h14m39.647124271s) [noise=true , address=104.36.82.12:4151 , piece_id=E2TEUTQBD4Y4AUHSCPMCIFEXBSY7S5HJWBKNKWWTUWEW2BLAAE6Q, seg_pos={0 0}, upload=5895]: Uploading piece... Jul 27 04:06:23.866412590 (6h14m39.647345968s) [noise=true , address=104.36.82.11:4115 , piece_id=YXMNVT6ZHGESGQ2BXKZUVR7U2CNNAI2PO7ZTMDG5UIOV7HTDDUTA, seg_pos={0 0}, upload=5895]: Uploading piece... Jul 27 04:06:23.866715972 (6h14m39.647649359s) [noise=true , address=45.91.116.13:4153 , piece_id=H4535KOCG3F4RKA6AKKMTJSOM6Q6PGRC6M7FR3UIS3BORTSP5UQA, seg_pos={0 0}, upload=5895]: Uploading piece... Jul 27 04:06:23.866998965 (6h14m39.64793234s) [noise=true , address=104.36.82.13:4186 , piece_id=TBLDKALGJYBGR5NLCOCKJ4QKEFNIJXCKAFUBD7VR3UMNWEMKMKHQ, seg_pos={0 0}, upload=5895]: Uploading piece... Jul 27 04:06:23.867230593 (6h14m39.648163974s) [noise=true , address=104.36.82.16:4293 , piece_id=AHH6PRFUSGTDGB7H52DRITARA4IF6GFLVSRYS2Y6EPJCZSKUOYWQ, seg_pos={0 0}, upload=5895]: Uploading piece... Jul 27 04:06:23.949422808 (6h14m39.730356187s) [upload=5895]: Next returned nil segment Jul 27 04:06:23.949442576 (6h14m39.73037596s) [upload=5895]: Waiting for error group managing segments... Jul 27 04:06:23.953497426 (6h14m39.734430802s) [noise=true , address=45.91.116.10:4081 , piece_id=XGDTSEZOUNRTEVWHTO3DHTZMH2VZZOPJPEKLCFAHTXESLLYABHSA, seg_pos={0 0}, upload=5895]: Done uploading piece. err: <nil> Jul 27 04:06:23.953531796 (6h14m39.734465159s) [noise=true , address=45.91.116.6:4973 , piece_id=J3VULPU24CIM64VEVE3UHO62HPN6TJXGZU6BCJIS5JQJKHJK2LIQ, seg_pos={0 0}, upload=5895]: Done uploading piece. err: <nil> Jul 27 04:06:23.954068660 (6h14m39.735002029s) [noise=true , address=45.91.116.9:4262 , piece_id=M5ETD45HH2XZD6HDLTJ3JO2BOOIROXFWKVPTN42PSQEWG4F7KAAQ, seg_pos={0 0}, upload=5895]: Done uploading piece. err: <nil> Jul 27 04:06:23.954194339 (6h14m39.735127707s) [noise=true , address=45.91.116.8:4048 , piece_id=5REWMVVTBVLDQXQ5KAEHLY4VJL5SGF7ONQWDQQ3N2FDU3OAO6AZA, seg_pos={0 0}, upload=5895]: Done uploading piece. err: <nil> Jul 27 04:06:23.954601730 (6h14m39.735535112s) [noise=true , address=45.91.116.7:4011 , piece_id=XMH6MCCHZBBRQWHJFWLRGYHLQD4KUGSGL7MMZSKA5TZJCND3DUDA, seg_pos={0 0}, upload=5895]: Done uploading piece. err: <nil> Jul 27 04:06:23.954755872 (6h14m39.735689237s) [noise=true , address=45.91.116.13:4153 , piece_id=H4535KOCG3F4RKA6AKKMTJSOM6Q6PGRC6M7FR3UIS3BORTSP5UQA, seg_pos={0 0}, upload=5895]: Done uploading piece. err: <nil> Jul 27 04:06:23.955611883 (6h14m39.73654526s) [noise=true , address=77.90.31.19:4843 , piece_id=YML5RDQIK432Z4FSE4DKXIV5U47LQBEZRFMAVLANUSGFUQR23UWQ, seg_pos={0 0}, upload=5895]: Done uploading piece. err: <nil> Jul 27 04:06:23.956373391 (6h14m39.737306756s) [noise=true , address=77.90.31.15:4697 , piece_id=Q76TJDSZPSYZSJQNFVOI462U6DFILMH2RV3LQISODXSOCXGRKW6Q, seg_pos={0 0}, upload=5895]: Done uploading piece. err: <nil> Jul 27 04:06:23.958659122 (6h14m39.73959249s) [noise=true , address=77.90.31.17:4771 , piece_id=U2LF22GKCMKRZN6EWCF26ITDZQHBFZXCGSQXT33MCX6P57IGI35Q, seg_pos={0 0}, upload=5895]: Done uploading piece. err: <nil> Jul 27 04:06:35.058944013 (6h14m50.83987739s) [noise=true , address=104.36.82.16:4293 , piece_id=AHH6PRFUSGTDGB7H52DRITARA4IF6GFLVSRYS2Y6EPJCZSKUOYWQ, seg_pos={0 0}, upload=5895]: Done uploading piece. err: <nil> Jul 27 04:06:41.103204002 (6h14m56.884137386s) [noise=true , address=104.36.82.4:4804 , piece_id=IU6FDNVB5AUPA7MJ6RSXUTLBETPHOFGNJ7YGWBFIZSQFSO6W6BBA, seg_pos={0 0}, upload=5895]: Done uploading piece. err: <nil> Jul 27 04:06:43.191702285 (6h14m58.972635651s) [noise=true , address=104.36.82.6:4873 , piece_id=YP4VJORZZUL5EVCH3P7LD3LNVMHB7CWRTKHGIA7UROXE44N6MMZA, seg_pos={0 0}, upload=5895]: Done uploading piece. err: <nil> Jul 27 04:06:53.777973493 (6h15m9.558906861s) [noise=true , address=104.36.82.11:4115 , piece_id=YXMNVT6ZHGESGQ2BXKZUVR7U2CNNAI2PO7ZTMDG5UIOV7HTDDUTA, seg_pos={0 0}, upload=5895]: Done uploading piece. err: ecclient: upload failed (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR, address:104.36.82.11:4115): protocol: write tcp 159.223.143.39:34210->104.36.82.11:4115: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 04:06:53.778019532 (6h15m9.558952909s) [noise=true , address=104.36.82.12:4151 , piece_id=E2TEUTQBD4Y4AUHSCPMCIFEXBSY7S5HJWBKNKWWTUWEW2BLAAE6Q, seg_pos={0 0}, upload=5895]: Done uploading piece. err: ecclient: upload failed (node:1w6pwZMdeQKjt92qx6WbTWU5VPYJto5gS87f6iKvCBZH6LfyeB, address:104.36.82.12:4151): protocol: rpc: dial tcp 104.36.82.12:4151: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 04:06:53.778043653 (6h15m9.558977021s) [noise=true , address=104.36.82.13:4186 , piece_id=TBLDKALGJYBGR5NLCOCKJ4QKEFNIJXCKAFUBD7VR3UMNWEMKMKHQ, seg_pos={0 0}, upload=5895]: Done uploading piece. err: ecclient: upload failed (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht, address:104.36.82.13:4186): protocol: read tcp 159.223.143.39:34340->104.36.82.13:4186: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 04:06:53.778095228 (6h15m9.559028593s) [noise=true , address=104.36.82.10:4079 , piece_id=2SH5D6JOMLFR7VSN7ZY5CLMR5RHIRFYQ7Z5I2JIYTJQM43G4O6OQ, seg_pos={0 0}, upload=5895]: Done uploading piece. err: ecclient: upload failed (node:12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud, address:104.36.82.10:4079): protocol: write tcp 159.223.143.39:53840->104.36.82.10:4079: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 04:06:53.778142167 (6h15m9.559075532s) [seg_pos={0 0}, upload=5895]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 27 04:06:53.837032161 (6h15m9.617965543s) [upload=5895]: Done uploading segments... Jul 27 04:06:53.837055996 (6h15m9.617989366s) [upload=5895]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 657.703119ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [....................X.......................................] duration: 599.642676ms FAIL (2025-07-27 04:37:15 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== {0 0}, upload=5911]: Begin upload segment... Jul 27 04:07:17.745307658 (6h15m33.526241043s) [seg_pos={0 0}, upload=5911]: Done begin upload segment. Jul 27 04:07:17.745349241 (6h15m33.526282625s) [noise=true , address=104.36.82.52:4098 , piece_id=M2ETF25QQ53RRGFPUKXJRINIWV7AWXSECEDR2AIM645IED72NURQ, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.745439694 (6h15m33.526373073s) [noise=true , address=104.36.82.56:4244 , piece_id=LQCZ5W54NBHWAVBVYK7HYSSJDC73E6FV4CP4YIM4L5H2N3OLOZXA, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.745972271 (6h15m33.526905654s) [noise=true , address=104.36.82.36:4650 , piece_id=BKJULWGPESBINNTSUYP6E4KZRXKPACVU5I7DOSZBO4YMK2V26HEQ, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.745980243 (6h15m33.526913621s) [noise=true , address=104.36.82.38:4722 , piece_id=B2MBQY5CTMEI6WSE7FTLOV5RUJ22SCNG7N2EEJILEATGJ4TIALXA, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.746403818 (6h15m33.527337198s) [noise=true , address=104.36.82.22:4534 , piece_id=CW4AQT3XUGA4A5OC6NUY4NYOOAMLN5FFJJN5PQBYEON7HPD5LCYQ, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.746455493 (6h15m33.527388875s) [noise=true , address=104.36.82.14:4248 , piece_id=LDSGCUM7Q2VX5CDFI4JRGEEG7D5H5Q5Z5WUWKU277VN2JFRX6UGA, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.746872127 (6h15m33.527805505s) [noise=true , address=104.36.82.40:4954 , piece_id=YLOHTJLRAED66OMEQYPZHNNRI77V6WOR4EHTEZOOEHMGZHJ2O3QQ, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.747123162 (6h15m33.52805654s) [noise=true , address=104.36.82.46:4496 , piece_id=GWNQR5WRGRRQBLSGWJFD3O3UNCMNE62JQNPIV5SYKBUIL3BQKA5A, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.747280678 (6h15m33.528214053s) [noise=true , address=104.36.82.47:4524 , piece_id=DONJFTKUQ2XCW6BOU7AVOKJ7XMDMBPGQVUPRANH7X2YS7PJXUKIA, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.747736400 (6h15m33.528669779s) [noise=true , address=104.36.82.44:4893 , piece_id=HUJICRJ6W54O3AZZR6H3KG6E4TGBTNS46O6XVMQM3PFAZE7A2HZA, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.747755734 (6h15m33.528689109s) [noise=true , address=104.36.82.55:4204 , piece_id=72WISGRHMOAZDV5EDGTBOMLBL4HDOF3CBEMQAAIRGILPC7J5JKAQ, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.748313124 (6h15m33.5292465s) [noise=true , address=104.36.82.58:4287 , piece_id=BVDYUEWU7TVEQDMZXX5SJNS4QICVKMTGFLCUV6P2YHOE7TI25VXA, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.748485217 (6h15m33.529418594s) [noise=true , address=104.36.82.53:4123 , piece_id=NXJRZ5BQSLXSOIOSUJ4MM4AGKHP4UIDUGEJKLUUX7YOGQJA6WWZA, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.748831329 (6h15m33.529764704s) [noise=true , address=104.36.82.42:4829 , piece_id=ZLWJFP3LGX3YLXBJYIR66DXTYEVDBKXKVXCQSF2PCYCYJBIEJGMA, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.749299293 (6h15m33.530232669s) [noise=true , address=104.36.82.8:4002 , piece_id=6XSPWQC3US355PS6IQ4K3IYNQZH5MAT4IVIPGSYRUM3XYPSCWE6Q, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.749790974 (6h15m33.53072436s) [noise=true , address=104.36.82.44:4901 , piece_id=N4YLCLR2QZSULUL2XWONEKBFUMNOU7KUX2QEOK4SSDFSCI7IL6LA, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.750478221 (6h15m33.531411597s) [noise=true , address=104.36.82.54:4172 , piece_id=ZNA53MWDRWPUH6FUR4DJERBDT2IOYGA3IYUADKXXIOLQJLQOXE7Q, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.752884255 (6h15m33.533817625s) [noise=true , address=64.49.13.140:4508 , piece_id=MZ52U75VOIK6Q7QFXMMYRRG7JXFWDFU2JWQFYPERUCMH3BUX6XPQ, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.879839555 (6h15m33.660772917s) [upload=5911]: Next returned nil segment Jul 27 04:07:17.879861389 (6h15m33.660794765s) [upload=5911]: Waiting for error group managing segments... Jul 27 04:07:17.888652438 (6h15m33.669585811s) [noise=true , address=104.36.82.58:4287 , piece_id=BVDYUEWU7TVEQDMZXX5SJNS4QICVKMTGFLCUV6P2YHOE7TI25VXA, seg_pos={0 0}, upload=5911]: Done uploading piece. err: <nil> Jul 27 04:07:17.952192045 (6h15m33.733125429s) [noise=true , address=104.36.82.53:4123 , piece_id=NXJRZ5BQSLXSOIOSUJ4MM4AGKHP4UIDUGEJKLUUX7YOGQJA6WWZA, seg_pos={0 0}, upload=5911]: Done uploading piece. err: <nil> Jul 27 04:07:17.981907539 (6h15m33.762840911s) [noise=true , address=104.36.82.56:4244 , piece_id=LQCZ5W54NBHWAVBVYK7HYSSJDC73E6FV4CP4YIM4L5H2N3OLOZXA, seg_pos={0 0}, upload=5911]: Done uploading piece. err: <nil> Jul 27 04:07:17.982370496 (6h15m33.7633039s) [noise=true , address=104.36.82.55:4204 , piece_id=72WISGRHMOAZDV5EDGTBOMLBL4HDOF3CBEMQAAIRGILPC7J5JKAQ, seg_pos={0 0}, upload=5911]: Done uploading piece. err: <nil> Jul 27 04:07:18.009548506 (6h15m33.790481886s) [noise=true , address=104.36.82.52:4098 , piece_id=M2ETF25QQ53RRGFPUKXJRINIWV7AWXSECEDR2AIM645IED72NURQ, seg_pos={0 0}, upload=5911]: Done uploading piece. err: <nil> Jul 27 04:07:18.023770598 (6h15m33.804703983s) [noise=true , address=104.36.82.40:4954 , piece_id=YLOHTJLRAED66OMEQYPZHNNRI77V6WOR4EHTEZOOEHMGZHJ2O3QQ, seg_pos={0 0}, upload=5911]: Done uploading piece. err: <nil> Jul 27 04:07:18.139745581 (6h15m33.92067895s) [noise=true , address=104.36.82.22:4534 , piece_id=CW4AQT3XUGA4A5OC6NUY4NYOOAMLN5FFJJN5PQBYEON7HPD5LCYQ, seg_pos={0 0}, upload=5911]: Done uploading piece. err: <nil> Jul 27 04:07:18.423748140 (6h15m34.204681511s) [noise=true , address=64.49.13.140:4508 , piece_id=MZ52U75VOIK6Q7QFXMMYRRG7JXFWDFU2JWQFYPERUCMH3BUX6XPQ, seg_pos={0 0}, upload=5911]: Done uploading piece. err: <nil> Jul 27 04:08:16.739940381 (6h16m32.52087375s) [noise=true , address=104.36.82.54:4172 , piece_id=ZNA53MWDRWPUH6FUR4DJERBDT2IOYGA3IYUADKXXIOLQJLQOXE7Q, seg_pos={0 0}, upload=5911]: Done uploading piece. err: <nil> Jul 27 04:08:17.177295397 (6h16m32.958228766s) [noise=true , address=104.36.82.8:4002 , piece_id=6XSPWQC3US355PS6IQ4K3IYNQZH5MAT4IVIPGSYRUM3XYPSCWE6Q, seg_pos={0 0}, upload=5911]: Done uploading piece. err: <nil> Jul 27 04:08:17.534296584 (6h16m33.315229953s) [noise=true , address=104.36.82.36:4650 , piece_id=BKJULWGPESBINNTSUYP6E4KZRXKPACVU5I7DOSZBO4YMK2V26HEQ, seg_pos={0 0}, upload=5911]: Done uploading piece. err: <nil> Jul 27 04:08:17.677851566 (6h16m33.458784943s) [noise=true , address=104.36.82.14:4248 , piece_id=LDSGCUM7Q2VX5CDFI4JRGEEG7D5H5Q5Z5WUWKU277VN2JFRX6UGA, seg_pos={0 0}, upload=5911]: Done uploading piece. err: ecclient: upload failed (node:1u6ir2PLqDa9xGm64cuqq1RqY4qeBa1fqCo9VghspBf54XEXqn, address:104.36.82.14:4248): protocol: rpc: dial tcp 104.36.82.14:4248: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 04:08:17.678115170 (6h16m33.459048549s) [noise=true , address=104.36.82.46:4496 , piece_id=GWNQR5WRGRRQBLSGWJFD3O3UNCMNE62JQNPIV5SYKBUIL3BQKA5A, seg_pos={0 0}, upload=5911]: Done uploading piece. err: ecclient: upload failed (node:12V8XhSWkyGUVYFR8MykifFAP1YAfGCwzqucDcMMfzw3xsVW77Z, address:104.36.82.46:4496): protocol: read tcp 159.223.143.39:52800->104.36.82.46:4496: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 04:08:17.678144409 (6h16m33.459077789s) [noise=true , address=104.36.82.44:4901 , piece_id=N4YLCLR2QZSULUL2XWONEKBFUMNOU7KUX2QEOK4SSDFSCI7IL6LA, seg_pos={0 0}, upload=5911]: Done uploading piece. err: ecclient: upload failed (node:1FHE84zvBb1MKwB1Ht7puTafgQxDEvPt4mYqkYVSSufAMpWQA1, address:104.36.82.44:4901): protocol: rpc: dial tcp 104.36.82.44:4901: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 04:08:17.678229622 (6h16m33.459162988s) [noise=true , address=104.36.82.44:4893 , piece_id=HUJICRJ6W54O3AZZR6H3KG6E4TGBTNS46O6XVMQM3PFAZE7A2HZA, seg_pos={0 0}, upload=5911]: Done uploading piece. err: ecclient: upload failed (node:17VMFZ2T8jrSJVtchciJ9SnyrR6JstvcMoXrN5Q3zTD1aQU1oJ, address:104.36.82.44:4893): protocol: write tcp 159.223.143.39:35302->104.36.82.44:4893: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 04:08:17.678310218 (6h16m33.459243583s) [noise=true , address=104.36.82.42:4829 , piece_id=ZLWJFP3LGX3YLXBJYIR66DXTYEVDBKXKVXCQSF2PCYCYJBIEJGMA, seg_pos={0 0}, upload=5911]: Done uploading piece. err: ecclient: upload failed (node:1F5T8Vt1x4nXFaSenvGBCYjxhRL1H1sxRvzLjW3CqzwDVhq39F, address:104.36.82.42:4829): protocol: rpc: dial tcp 104.36.82.42:4829: i/o timeout; conn closed: op; piecestore: piecestore close: context deadline exceeded Jul 27 04:08:17.678412989 (6h16m33.459346365s) [noise=true , address=104.36.82.38:4722 , piece_id=B2MBQY5CTMEI6WSE7FTLOV5RUJ22SCNG7N2EEJILEATGJ4TIALXA, seg_pos={0 0}, upload=5911]: Done uploading piece. err: ecclient: upload failed (node:12Zw3dtUg9gARz52KhwDLTjcqC3CNm7dpPzpdx7P7PbL1jwe8uc, address:104.36.82.38:4722): protocol: read tcp 159.223.143.39:56048->104.36.82.38:4722: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 04:08:17.678455247 (6h16m33.459388624s) [noise=true , address=104.36.82.47:4524 , piece_id=DONJFTKUQ2XCW6BOU7AVOKJ7XMDMBPGQVUPRANH7X2YS7PJXUKIA, seg_pos={0 0}, upload=5911]: Done uploading piece. err: ecclient: upload failed (node:1xfbCzjSwFDRrR1gPggeN8tNCLbRRWDVzCJ6mqhZsbVSe8d4s4, address:104.36.82.47:4524): protocol: read tcp 159.223.143.39:46694->104.36.82.47:4524: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 04:08:17.678510668 (6h16m33.459444052s) [seg_pos={0 0}, upload=5911]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Jul 27 04:08:17.706442864 (6h16m33.487376232s) [upload=5911]: Done uploading segments... Jul 27 04:08:17.706495329 (6h16m33.487428705s) [upload=5911]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 599.671979ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............................X..............................] duration: 640.240152ms FAIL (2025-07-27 04:37:09 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 12) Duration: 640.309177ms core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.....................X......................................] duration: 484.124007ms FAIL (2025-07-27 04:36:42 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== {0 0}, upload=5911]: Begin upload segment... Jul 27 04:07:17.745307658 (6h15m33.526241043s) [seg_pos={0 0}, upload=5911]: Done begin upload segment. Jul 27 04:07:17.745349241 (6h15m33.526282625s) [noise=true , address=104.36.82.52:4098 , piece_id=M2ETF25QQ53RRGFPUKXJRINIWV7AWXSECEDR2AIM645IED72NURQ, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.745439694 (6h15m33.526373073s) [noise=true , address=104.36.82.56:4244 , piece_id=LQCZ5W54NBHWAVBVYK7HYSSJDC73E6FV4CP4YIM4L5H2N3OLOZXA, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.745972271 (6h15m33.526905654s) [noise=true , address=104.36.82.36:4650 , piece_id=BKJULWGPESBINNTSUYP6E4KZRXKPACVU5I7DOSZBO4YMK2V26HEQ, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.745980243 (6h15m33.526913621s) [noise=true , address=104.36.82.38:4722 , piece_id=B2MBQY5CTMEI6WSE7FTLOV5RUJ22SCNG7N2EEJILEATGJ4TIALXA, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.746403818 (6h15m33.527337198s) [noise=true , address=104.36.82.22:4534 , piece_id=CW4AQT3XUGA4A5OC6NUY4NYOOAMLN5FFJJN5PQBYEON7HPD5LCYQ, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.746455493 (6h15m33.527388875s) [noise=true , address=104.36.82.14:4248 , piece_id=LDSGCUM7Q2VX5CDFI4JRGEEG7D5H5Q5Z5WUWKU277VN2JFRX6UGA, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.746872127 (6h15m33.527805505s) [noise=true , address=104.36.82.40:4954 , piece_id=YLOHTJLRAED66OMEQYPZHNNRI77V6WOR4EHTEZOOEHMGZHJ2O3QQ, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.747123162 (6h15m33.52805654s) [noise=true , address=104.36.82.46:4496 , piece_id=GWNQR5WRGRRQBLSGWJFD3O3UNCMNE62JQNPIV5SYKBUIL3BQKA5A, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.747280678 (6h15m33.528214053s) [noise=true , address=104.36.82.47:4524 , piece_id=DONJFTKUQ2XCW6BOU7AVOKJ7XMDMBPGQVUPRANH7X2YS7PJXUKIA, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.747736400 (6h15m33.528669779s) [noise=true , address=104.36.82.44:4893 , piece_id=HUJICRJ6W54O3AZZR6H3KG6E4TGBTNS46O6XVMQM3PFAZE7A2HZA, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.747755734 (6h15m33.528689109s) [noise=true , address=104.36.82.55:4204 , piece_id=72WISGRHMOAZDV5EDGTBOMLBL4HDOF3CBEMQAAIRGILPC7J5JKAQ, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.748313124 (6h15m33.5292465s) [noise=true , address=104.36.82.58:4287 , piece_id=BVDYUEWU7TVEQDMZXX5SJNS4QICVKMTGFLCUV6P2YHOE7TI25VXA, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.748485217 (6h15m33.529418594s) [noise=true , address=104.36.82.53:4123 , piece_id=NXJRZ5BQSLXSOIOSUJ4MM4AGKHP4UIDUGEJKLUUX7YOGQJA6WWZA, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.748831329 (6h15m33.529764704s) [noise=true , address=104.36.82.42:4829 , piece_id=ZLWJFP3LGX3YLXBJYIR66DXTYEVDBKXKVXCQSF2PCYCYJBIEJGMA, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.749299293 (6h15m33.530232669s) [noise=true , address=104.36.82.8:4002 , piece_id=6XSPWQC3US355PS6IQ4K3IYNQZH5MAT4IVIPGSYRUM3XYPSCWE6Q, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.749790974 (6h15m33.53072436s) [noise=true , address=104.36.82.44:4901 , piece_id=N4YLCLR2QZSULUL2XWONEKBFUMNOU7KUX2QEOK4SSDFSCI7IL6LA, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.750478221 (6h15m33.531411597s) [noise=true , address=104.36.82.54:4172 , piece_id=ZNA53MWDRWPUH6FUR4DJERBDT2IOYGA3IYUADKXXIOLQJLQOXE7Q, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.752884255 (6h15m33.533817625s) [noise=true , address=64.49.13.140:4508 , piece_id=MZ52U75VOIK6Q7QFXMMYRRG7JXFWDFU2JWQFYPERUCMH3BUX6XPQ, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.879839555 (6h15m33.660772917s) [upload=5911]: Next returned nil segment Jul 27 04:07:17.879861389 (6h15m33.660794765s) [upload=5911]: Waiting for error group managing segments... Jul 27 04:07:17.888652438 (6h15m33.669585811s) [noise=true , address=104.36.82.58:4287 , piece_id=BVDYUEWU7TVEQDMZXX5SJNS4QICVKMTGFLCUV6P2YHOE7TI25VXA, seg_pos={0 0}, upload=5911]: Done uploading piece. err: <nil> Jul 27 04:07:17.952192045 (6h15m33.733125429s) [noise=true , address=104.36.82.53:4123 , piece_id=NXJRZ5BQSLXSOIOSUJ4MM4AGKHP4UIDUGEJKLUUX7YOGQJA6WWZA, seg_pos={0 0}, upload=5911]: Done uploading piece. err: <nil> Jul 27 04:07:17.981907539 (6h15m33.762840911s) [noise=true , address=104.36.82.56:4244 , piece_id=LQCZ5W54NBHWAVBVYK7HYSSJDC73E6FV4CP4YIM4L5H2N3OLOZXA, seg_pos={0 0}, upload=5911]: Done uploading piece. err: <nil> Jul 27 04:07:17.982370496 (6h15m33.7633039s) [noise=true , address=104.36.82.55:4204 , piece_id=72WISGRHMOAZDV5EDGTBOMLBL4HDOF3CBEMQAAIRGILPC7J5JKAQ, seg_pos={0 0}, upload=5911]: Done uploading piece. err: <nil> Jul 27 04:07:18.009548506 (6h15m33.790481886s) [noise=true , address=104.36.82.52:4098 , piece_id=M2ETF25QQ53RRGFPUKXJRINIWV7AWXSECEDR2AIM645IED72NURQ, seg_pos={0 0}, upload=5911]: Done uploading piece. err: <nil> Jul 27 04:07:18.023770598 (6h15m33.804703983s) [noise=true , address=104.36.82.40:4954 , piece_id=YLOHTJLRAED66OMEQYPZHNNRI77V6WOR4EHTEZOOEHMGZHJ2O3QQ, seg_pos={0 0}, upload=5911]: Done uploading piece. err: <nil> Jul 27 04:07:18.139745581 (6h15m33.92067895s) [noise=true , address=104.36.82.22:4534 , piece_id=CW4AQT3XUGA4A5OC6NUY4NYOOAMLN5FFJJN5PQBYEON7HPD5LCYQ, seg_pos={0 0}, upload=5911]: Done uploading piece. err: <nil> Jul 27 04:07:18.423748140 (6h15m34.204681511s) [noise=true , address=64.49.13.140:4508 , piece_id=MZ52U75VOIK6Q7QFXMMYRRG7JXFWDFU2JWQFYPERUCMH3BUX6XPQ, seg_pos={0 0}, upload=5911]: Done uploading piece. err: <nil> Jul 27 04:08:16.739940381 (6h16m32.52087375s) [noise=true , address=104.36.82.54:4172 , piece_id=ZNA53MWDRWPUH6FUR4DJERBDT2IOYGA3IYUADKXXIOLQJLQOXE7Q, seg_pos={0 0}, upload=5911]: Done uploading piece. err: <nil> Jul 27 04:08:17.177295397 (6h16m32.958228766s) [noise=true , address=104.36.82.8:4002 , piece_id=6XSPWQC3US355PS6IQ4K3IYNQZH5MAT4IVIPGSYRUM3XYPSCWE6Q, seg_pos={0 0}, upload=5911]: Done uploading piece. err: <nil> Jul 27 04:08:17.534296584 (6h16m33.315229953s) [noise=true , address=104.36.82.36:4650 , piece_id=BKJULWGPESBINNTSUYP6E4KZRXKPACVU5I7DOSZBO4YMK2V26HEQ, seg_pos={0 0}, upload=5911]: Done uploading piece. err: <nil> Jul 27 04:08:17.677851566 (6h16m33.458784943s) [noise=true , address=104.36.82.14:4248 , piece_id=LDSGCUM7Q2VX5CDFI4JRGEEG7D5H5Q5Z5WUWKU277VN2JFRX6UGA, seg_pos={0 0}, upload=5911]: Done uploading piece. err: ecclient: upload failed (node:1u6ir2PLqDa9xGm64cuqq1RqY4qeBa1fqCo9VghspBf54XEXqn, address:104.36.82.14:4248): protocol: rpc: dial tcp 104.36.82.14:4248: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 04:08:17.678115170 (6h16m33.459048549s) [noise=true , address=104.36.82.46:4496 , piece_id=GWNQR5WRGRRQBLSGWJFD3O3UNCMNE62JQNPIV5SYKBUIL3BQKA5A, seg_pos={0 0}, upload=5911]: Done uploading piece. err: ecclient: upload failed (node:12V8XhSWkyGUVYFR8MykifFAP1YAfGCwzqucDcMMfzw3xsVW77Z, address:104.36.82.46:4496): protocol: read tcp 159.223.143.39:52800->104.36.82.46:4496: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 04:08:17.678144409 (6h16m33.459077789s) [noise=true , address=104.36.82.44:4901 , piece_id=N4YLCLR2QZSULUL2XWONEKBFUMNOU7KUX2QEOK4SSDFSCI7IL6LA, seg_pos={0 0}, upload=5911]: Done uploading piece. err: ecclient: upload failed (node:1FHE84zvBb1MKwB1Ht7puTafgQxDEvPt4mYqkYVSSufAMpWQA1, address:104.36.82.44:4901): protocol: rpc: dial tcp 104.36.82.44:4901: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 04:08:17.678229622 (6h16m33.459162988s) [noise=true , address=104.36.82.44:4893 , piece_id=HUJICRJ6W54O3AZZR6H3KG6E4TGBTNS46O6XVMQM3PFAZE7A2HZA, seg_pos={0 0}, upload=5911]: Done uploading piece. err: ecclient: upload failed (node:17VMFZ2T8jrSJVtchciJ9SnyrR6JstvcMoXrN5Q3zTD1aQU1oJ, address:104.36.82.44:4893): protocol: write tcp 159.223.143.39:35302->104.36.82.44:4893: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 04:08:17.678310218 (6h16m33.459243583s) [noise=true , address=104.36.82.42:4829 , piece_id=ZLWJFP3LGX3YLXBJYIR66DXTYEVDBKXKVXCQSF2PCYCYJBIEJGMA, seg_pos={0 0}, upload=5911]: Done uploading piece. err: ecclient: upload failed (node:1F5T8Vt1x4nXFaSenvGBCYjxhRL1H1sxRvzLjW3CqzwDVhq39F, address:104.36.82.42:4829): protocol: rpc: dial tcp 104.36.82.42:4829: i/o timeout; conn closed: op; piecestore: piecestore close: context deadline exceeded Jul 27 04:08:17.678412989 (6h16m33.459346365s) [noise=true , address=104.36.82.38:4722 , piece_id=B2MBQY5CTMEI6WSE7FTLOV5RUJ22SCNG7N2EEJILEATGJ4TIALXA, seg_pos={0 0}, upload=5911]: Done uploading piece. err: ecclient: upload failed (node:12Zw3dtUg9gARz52KhwDLTjcqC3CNm7dpPzpdx7P7PbL1jwe8uc, address:104.36.82.38:4722): protocol: read tcp 159.223.143.39:56048->104.36.82.38:4722: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 04:08:17.678455247 (6h16m33.459388624s) [noise=true , address=104.36.82.47:4524 , piece_id=DONJFTKUQ2XCW6BOU7AVOKJ7XMDMBPGQVUPRANH7X2YS7PJXUKIA, seg_pos={0 0}, upload=5911]: Done uploading piece. err: ecclient: upload failed (node:1xfbCzjSwFDRrR1gPggeN8tNCLbRRWDVzCJ6mqhZsbVSe8d4s4, address:104.36.82.47:4524): protocol: read tcp 159.223.143.39:46694->104.36.82.47:4524: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 04:08:17.678510668 (6h16m33.459444052s) [seg_pos={0 0}, upload=5911]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Jul 27 04:08:17.706442864 (6h16m33.487376232s) [upload=5911]: Done uploading segments... Jul 27 04:08:17.706495329 (6h16m33.487428705s) [upload=5911]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 484.160396ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............X............................................] duration: 432.754365ms FAIL (2025-07-27 04:36:41 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== ece_id=NYOFDEZ7CV4PHGQ6JZMEEVAYGJRPYG6IBW7MYJO42HMWOJM7PHZA, seg_pos={0 0}, upload=5879]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12crcXpNe9NgeXTsUpooLSBgrSUFbRxbSbanLa39w2Hr5mGFww): protocol: read tcp 159.223.143.39:53630->104.36.82.5:4840: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 04:05:32.508904485 (6h13m48.289837866s) [seg_pos={0 0}, upload=5879]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 27 04:05:32.509028070 (6h13m48.289961457s) [upload=5879]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 27 04:05:32.509034645 (6h13m48.289968129s) [upload=5879]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 27 04:05:32.581834839 (6h13m48.362768225s) [upload=5879]: Done uploading segments... Jul 27 04:05:32.581875919 (6h13m48.362809298s) [upload=5879]: Upload finished. err: <nil> Jul 27 04:06:23.829155331 (6h14m39.610088696s) [upload=5895]: Starting upload Jul 27 04:06:23.829225021 (6h14m39.610158398s) [upload=5895]: Done starting upload Jul 27 04:06:23.829294606 (6h14m39.61022799s) [upload=5895]: Uploading segments... Jul 27 04:06:23.829577656 (6h14m39.610511053s) [upload=5895]: Got next segment. Inline: false Jul 27 04:06:23.829649664 (6h14m39.610583027s) [upload=5895]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 27 04:06:23.829657065 (6h14m39.61059043s) [upload=5895]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 27 04:06:23.861405530 (6h14m39.642338913s) [seg_pos={0 0}, upload=5895]: Begin upload segment... Jul 27 04:06:23.861476351 (6h14m39.642409713s) [seg_pos={0 0}, upload=5895]: Done begin upload segment. Jul 27 04:06:23.861509120 (6h14m39.642442506s) [noise=true , address=77.90.31.19:4843 , piece_id=YML5RDQIK432Z4FSE4DKXIV5U47LQBEZRFMAVLANUSGFUQR23UWQ, seg_pos={0 0}, upload=5895]: Uploading piece... Jul 27 04:06:23.861953723 (6h14m39.6428871s) [noise=true , address=45.91.116.10:4081 , piece_id=XGDTSEZOUNRTEVWHTO3DHTZMH2VZZOPJPEKLCFAHTXESLLYABHSA, seg_pos={0 0}, upload=5895]: Uploading piece... Jul 27 04:06:23.862060983 (6h14m39.642994358s) [noise=true , address=45.91.116.9:4262 , piece_id=M5ETD45HH2XZD6HDLTJ3JO2BOOIROXFWKVPTN42PSQEWG4F7KAAQ, seg_pos={0 0}, upload=5895]: Uploading piece... Jul 27 04:06:23.862759464 (6h14m39.64369284s) [noise=true , address=104.36.82.4:4804 , piece_id=IU6FDNVB5AUPA7MJ6RSXUTLBETPHOFGNJ7YGWBFIZSQFSO6W6BBA, seg_pos={0 0}, upload=5895]: Uploading piece... Jul 27 04:06:23.862835086 (6h14m39.643768461s) [noise=true , address=104.36.82.6:4873 , piece_id=YP4VJORZZUL5EVCH3P7LD3LNVMHB7CWRTKHGIA7UROXE44N6MMZA, seg_pos={0 0}, upload=5895]: Uploading piece... Jul 27 04:06:23.863593880 (6h14m39.644527258s) [noise=true , address=77.90.31.15:4697 , piece_id=Q76TJDSZPSYZSJQNFVOI462U6DFILMH2RV3LQISODXSOCXGRKW6Q, seg_pos={0 0}, upload=5895]: Uploading piece... Jul 27 04:06:23.863656891 (6h14m39.644590266s) [noise=true , address=45.91.116.7:4011 , piece_id=XMH6MCCHZBBRQWHJFWLRGYHLQD4KUGSGL7MMZSKA5TZJCND3DUDA, seg_pos={0 0}, upload=5895]: Uploading piece... Jul 27 04:06:23.864408612 (6h14m39.645341987s) [noise=true , address=45.91.116.6:4973 , piece_id=J3VULPU24CIM64VEVE3UHO62HPN6TJXGZU6BCJIS5JQJKHJK2LIQ, seg_pos={0 0}, upload=5895]: Uploading piece... Jul 27 04:06:23.864521549 (6h14m39.64545491s) [noise=true , address=104.36.82.10:4079 , piece_id=2SH5D6JOMLFR7VSN7ZY5CLMR5RHIRFYQ7Z5I2JIYTJQM43G4O6OQ, seg_pos={0 0}, upload=5895]: Uploading piece... Jul 27 04:06:23.865518491 (6h14m39.646451872s) [noise=true , address=77.90.31.17:4771 , piece_id=U2LF22GKCMKRZN6EWCF26ITDZQHBFZXCGSQXT33MCX6P57IGI35Q, seg_pos={0 0}, upload=5895]: Uploading piece... Jul 27 04:06:23.865654950 (6h14m39.646588325s) [noise=true , address=45.91.116.8:4048 , piece_id=5REWMVVTBVLDQXQ5KAEHLY4VJL5SGF7ONQWDQQ3N2FDU3OAO6AZA, seg_pos={0 0}, upload=5895]: Uploading piece... Jul 27 04:06:23.866190892 (6h14m39.647124271s) [noise=true , address=104.36.82.12:4151 , piece_id=E2TEUTQBD4Y4AUHSCPMCIFEXBSY7S5HJWBKNKWWTUWEW2BLAAE6Q, seg_pos={0 0}, upload=5895]: Uploading piece... Jul 27 04:06:23.866412590 (6h14m39.647345968s) [noise=true , address=104.36.82.11:4115 , piece_id=YXMNVT6ZHGESGQ2BXKZUVR7U2CNNAI2PO7ZTMDG5UIOV7HTDDUTA, seg_pos={0 0}, upload=5895]: Uploading piece... Jul 27 04:06:23.866715972 (6h14m39.647649359s) [noise=true , address=45.91.116.13:4153 , piece_id=H4535KOCG3F4RKA6AKKMTJSOM6Q6PGRC6M7FR3UIS3BORTSP5UQA, seg_pos={0 0}, upload=5895]: Uploading piece... Jul 27 04:06:23.866998965 (6h14m39.64793234s) [noise=true , address=104.36.82.13:4186 , piece_id=TBLDKALGJYBGR5NLCOCKJ4QKEFNIJXCKAFUBD7VR3UMNWEMKMKHQ, seg_pos={0 0}, upload=5895]: Uploading piece... Jul 27 04:06:23.867230593 (6h14m39.648163974s) [noise=true , address=104.36.82.16:4293 , piece_id=AHH6PRFUSGTDGB7H52DRITARA4IF6GFLVSRYS2Y6EPJCZSKUOYWQ, seg_pos={0 0}, upload=5895]: Uploading piece... Jul 27 04:06:23.949422808 (6h14m39.730356187s) [upload=5895]: Next returned nil segment Jul 27 04:06:23.949442576 (6h14m39.73037596s) [upload=5895]: Waiting for error group managing segments... Jul 27 04:06:23.953497426 (6h14m39.734430802s) [noise=true , address=45.91.116.10:4081 , piece_id=XGDTSEZOUNRTEVWHTO3DHTZMH2VZZOPJPEKLCFAHTXESLLYABHSA, seg_pos={0 0}, upload=5895]: Done uploading piece. err: <nil> Jul 27 04:06:23.953531796 (6h14m39.734465159s) [noise=true , address=45.91.116.6:4973 , piece_id=J3VULPU24CIM64VEVE3UHO62HPN6TJXGZU6BCJIS5JQJKHJK2LIQ, seg_pos={0 0}, upload=5895]: Done uploading piece. err: <nil> Jul 27 04:06:23.954068660 (6h14m39.735002029s) [noise=true , address=45.91.116.9:4262 , piece_id=M5ETD45HH2XZD6HDLTJ3JO2BOOIROXFWKVPTN42PSQEWG4F7KAAQ, seg_pos={0 0}, upload=5895]: Done uploading piece. err: <nil> Jul 27 04:06:23.954194339 (6h14m39.735127707s) [noise=true , address=45.91.116.8:4048 , piece_id=5REWMVVTBVLDQXQ5KAEHLY4VJL5SGF7ONQWDQQ3N2FDU3OAO6AZA, seg_pos={0 0}, upload=5895]: Done uploading piece. err: <nil> Jul 27 04:06:23.954601730 (6h14m39.735535112s) [noise=true , address=45.91.116.7:4011 , piece_id=XMH6MCCHZBBRQWHJFWLRGYHLQD4KUGSGL7MMZSKA5TZJCND3DUDA, seg_pos={0 0}, upload=5895]: Done uploading piece. err: <nil> Jul 27 04:06:23.954755872 (6h14m39.735689237s) [noise=true , address=45.91.116.13:4153 , piece_id=H4535KOCG3F4RKA6AKKMTJSOM6Q6PGRC6M7FR3UIS3BORTSP5UQA, seg_pos={0 0}, upload=5895]: Done uploading piece. err: <nil> Jul 27 04:06:23.955611883 (6h14m39.73654526s) [noise=true , address=77.90.31.19:4843 , piece_id=YML5RDQIK432Z4FSE4DKXIV5U47LQBEZRFMAVLANUSGFUQR23UWQ, seg_pos={0 0}, upload=5895]: Done uploading piece. err: <nil> Jul 27 04:06:23.956373391 (6h14m39.737306756s) [noise=true , address=77.90.31.15:4697 , piece_id=Q76TJDSZPSYZSJQNFVOI462U6DFILMH2RV3LQISODXSOCXGRKW6Q, seg_pos={0 0}, upload=5895]: Done uploading piece. err: <nil> Jul 27 04:06:23.958659122 (6h14m39.73959249s) [noise=true , address=77.90.31.17:4771 , piece_id=U2LF22GKCMKRZN6EWCF26ITDZQHBFZXCGSQXT33MCX6P57IGI35Q, seg_pos={0 0}, upload=5895]: Done uploading piece. err: <nil> Jul 27 04:06:35.058944013 (6h14m50.83987739s) [noise=true , address=104.36.82.16:4293 , piece_id=AHH6PRFUSGTDGB7H52DRITARA4IF6GFLVSRYS2Y6EPJCZSKUOYWQ, seg_pos={0 0}, upload=5895]: Done uploading piece. err: <nil> Jul 27 04:06:41.103204002 (6h14m56.884137386s) [noise=true , address=104.36.82.4:4804 , piece_id=IU6FDNVB5AUPA7MJ6RSXUTLBETPHOFGNJ7YGWBFIZSQFSO6W6BBA, seg_pos={0 0}, upload=5895]: Done uploading piece. err: <nil> Jul 27 04:06:43.191702285 (6h14m58.972635651s) [noise=true , address=104.36.82.6:4873 , piece_id=YP4VJORZZUL5EVCH3P7LD3LNVMHB7CWRTKHGIA7UROXE44N6MMZA, seg_pos={0 0}, upload=5895]: Done uploading piece. err: <nil> Jul 27 04:06:53.777973493 (6h15m9.558906861s) [noise=true , address=104.36.82.11:4115 , piece_id=YXMNVT6ZHGESGQ2BXKZUVR7U2CNNAI2PO7ZTMDG5UIOV7HTDDUTA, seg_pos={0 0}, upload=5895]: Done uploading piece. err: ecclient: upload failed (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR, address:104.36.82.11:4115): protocol: write tcp 159.223.143.39:34210->104.36.82.11:4115: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 04:06:53.778019532 (6h15m9.558952909s) [noise=true , address=104.36.82.12:4151 , piece_id=E2TEUTQBD4Y4AUHSCPMCIFEXBSY7S5HJWBKNKWWTUWEW2BLAAE6Q, seg_pos={0 0}, upload=5895]: Done uploading piece. err: ecclient: upload failed (node:1w6pwZMdeQKjt92qx6WbTWU5VPYJto5gS87f6iKvCBZH6LfyeB, address:104.36.82.12:4151): protocol: rpc: dial tcp 104.36.82.12:4151: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 04:06:53.778043653 (6h15m9.558977021s) [noise=true , address=104.36.82.13:4186 , piece_id=TBLDKALGJYBGR5NLCOCKJ4QKEFNIJXCKAFUBD7VR3UMNWEMKMKHQ, seg_pos={0 0}, upload=5895]: Done uploading piece. err: ecclient: upload failed (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht, address:104.36.82.13:4186): protocol: read tcp 159.223.143.39:34340->104.36.82.13:4186: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 04:06:53.778095228 (6h15m9.559028593s) [noise=true , address=104.36.82.10:4079 , piece_id=2SH5D6JOMLFR7VSN7ZY5CLMR5RHIRFYQ7Z5I2JIYTJQM43G4O6OQ, seg_pos={0 0}, upload=5895]: Done uploading piece. err: ecclient: upload failed (node:12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud, address:104.36.82.10:4079): protocol: write tcp 159.223.143.39:53840->104.36.82.10:4079: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 04:06:53.778142167 (6h15m9.559075532s) [seg_pos={0 0}, upload=5895]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 27 04:06:53.837032161 (6h15m9.617965543s) [upload=5895]: Done uploading segments... Jul 27 04:06:53.837055996 (6h15m9.617989366s) [upload=5895]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 432.79049ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............................X.............................] duration: 373.999929ms FAIL (2025-07-27 04:36:25 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 12) Duration: 374.055024ms core uplink us1 vivint

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)