Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................X..................................] duration: 901.869088ms FAIL (2025-07-27 04:30:10 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== ece_id=NYOFDEZ7CV4PHGQ6JZMEEVAYGJRPYG6IBW7MYJO42HMWOJM7PHZA, seg_pos={0 0}, upload=5879]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12crcXpNe9NgeXTsUpooLSBgrSUFbRxbSbanLa39w2Hr5mGFww): protocol: read tcp 159.223.143.39:53630->104.36.82.5:4840: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 04:05:32.508904485 (6h13m48.289837866s) [seg_pos={0 0}, upload=5879]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 27 04:05:32.509028070 (6h13m48.289961457s) [upload=5879]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 27 04:05:32.509034645 (6h13m48.289968129s) [upload=5879]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 27 04:05:32.581834839 (6h13m48.362768225s) [upload=5879]: Done uploading segments... Jul 27 04:05:32.581875919 (6h13m48.362809298s) [upload=5879]: Upload finished. err: <nil> Jul 27 04:06:23.829155331 (6h14m39.610088696s) [upload=5895]: Starting upload Jul 27 04:06:23.829225021 (6h14m39.610158398s) [upload=5895]: Done starting upload Jul 27 04:06:23.829294606 (6h14m39.61022799s) [upload=5895]: Uploading segments... Jul 27 04:06:23.829577656 (6h14m39.610511053s) [upload=5895]: Got next segment. Inline: false Jul 27 04:06:23.829649664 (6h14m39.610583027s) [upload=5895]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 27 04:06:23.829657065 (6h14m39.61059043s) [upload=5895]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 27 04:06:23.861405530 (6h14m39.642338913s) [seg_pos={0 0}, upload=5895]: Begin upload segment... Jul 27 04:06:23.861476351 (6h14m39.642409713s) [seg_pos={0 0}, upload=5895]: Done begin upload segment. Jul 27 04:06:23.861509120 (6h14m39.642442506s) [noise=true , address=77.90.31.19:4843 , piece_id=YML5RDQIK432Z4FSE4DKXIV5U47LQBEZRFMAVLANUSGFUQR23UWQ, seg_pos={0 0}, upload=5895]: Uploading piece... Jul 27 04:06:23.861953723 (6h14m39.6428871s) [noise=true , address=45.91.116.10:4081 , piece_id=XGDTSEZOUNRTEVWHTO3DHTZMH2VZZOPJPEKLCFAHTXESLLYABHSA, seg_pos={0 0}, upload=5895]: Uploading piece... Jul 27 04:06:23.862060983 (6h14m39.642994358s) [noise=true , address=45.91.116.9:4262 , piece_id=M5ETD45HH2XZD6HDLTJ3JO2BOOIROXFWKVPTN42PSQEWG4F7KAAQ, seg_pos={0 0}, upload=5895]: Uploading piece... Jul 27 04:06:23.862759464 (6h14m39.64369284s) [noise=true , address=104.36.82.4:4804 , piece_id=IU6FDNVB5AUPA7MJ6RSXUTLBETPHOFGNJ7YGWBFIZSQFSO6W6BBA, seg_pos={0 0}, upload=5895]: Uploading piece... Jul 27 04:06:23.862835086 (6h14m39.643768461s) [noise=true , address=104.36.82.6:4873 , piece_id=YP4VJORZZUL5EVCH3P7LD3LNVMHB7CWRTKHGIA7UROXE44N6MMZA, seg_pos={0 0}, upload=5895]: Uploading piece... Jul 27 04:06:23.863593880 (6h14m39.644527258s) [noise=true , address=77.90.31.15:4697 , piece_id=Q76TJDSZPSYZSJQNFVOI462U6DFILMH2RV3LQISODXSOCXGRKW6Q, seg_pos={0 0}, upload=5895]: Uploading piece... Jul 27 04:06:23.863656891 (6h14m39.644590266s) [noise=true , address=45.91.116.7:4011 , piece_id=XMH6MCCHZBBRQWHJFWLRGYHLQD4KUGSGL7MMZSKA5TZJCND3DUDA, seg_pos={0 0}, upload=5895]: Uploading piece... Jul 27 04:06:23.864408612 (6h14m39.645341987s) [noise=true , address=45.91.116.6:4973 , piece_id=J3VULPU24CIM64VEVE3UHO62HPN6TJXGZU6BCJIS5JQJKHJK2LIQ, seg_pos={0 0}, upload=5895]: Uploading piece... Jul 27 04:06:23.864521549 (6h14m39.64545491s) [noise=true , address=104.36.82.10:4079 , piece_id=2SH5D6JOMLFR7VSN7ZY5CLMR5RHIRFYQ7Z5I2JIYTJQM43G4O6OQ, seg_pos={0 0}, upload=5895]: Uploading piece... Jul 27 04:06:23.865518491 (6h14m39.646451872s) [noise=true , address=77.90.31.17:4771 , piece_id=U2LF22GKCMKRZN6EWCF26ITDZQHBFZXCGSQXT33MCX6P57IGI35Q, seg_pos={0 0}, upload=5895]: Uploading piece... Jul 27 04:06:23.865654950 (6h14m39.646588325s) [noise=true , address=45.91.116.8:4048 , piece_id=5REWMVVTBVLDQXQ5KAEHLY4VJL5SGF7ONQWDQQ3N2FDU3OAO6AZA, seg_pos={0 0}, upload=5895]: Uploading piece... Jul 27 04:06:23.866190892 (6h14m39.647124271s) [noise=true , address=104.36.82.12:4151 , piece_id=E2TEUTQBD4Y4AUHSCPMCIFEXBSY7S5HJWBKNKWWTUWEW2BLAAE6Q, seg_pos={0 0}, upload=5895]: Uploading piece... Jul 27 04:06:23.866412590 (6h14m39.647345968s) [noise=true , address=104.36.82.11:4115 , piece_id=YXMNVT6ZHGESGQ2BXKZUVR7U2CNNAI2PO7ZTMDG5UIOV7HTDDUTA, seg_pos={0 0}, upload=5895]: Uploading piece... Jul 27 04:06:23.866715972 (6h14m39.647649359s) [noise=true , address=45.91.116.13:4153 , piece_id=H4535KOCG3F4RKA6AKKMTJSOM6Q6PGRC6M7FR3UIS3BORTSP5UQA, seg_pos={0 0}, upload=5895]: Uploading piece... Jul 27 04:06:23.866998965 (6h14m39.64793234s) [noise=true , address=104.36.82.13:4186 , piece_id=TBLDKALGJYBGR5NLCOCKJ4QKEFNIJXCKAFUBD7VR3UMNWEMKMKHQ, seg_pos={0 0}, upload=5895]: Uploading piece... Jul 27 04:06:23.867230593 (6h14m39.648163974s) [noise=true , address=104.36.82.16:4293 , piece_id=AHH6PRFUSGTDGB7H52DRITARA4IF6GFLVSRYS2Y6EPJCZSKUOYWQ, seg_pos={0 0}, upload=5895]: Uploading piece... Jul 27 04:06:23.949422808 (6h14m39.730356187s) [upload=5895]: Next returned nil segment Jul 27 04:06:23.949442576 (6h14m39.73037596s) [upload=5895]: Waiting for error group managing segments... Jul 27 04:06:23.953497426 (6h14m39.734430802s) [noise=true , address=45.91.116.10:4081 , piece_id=XGDTSEZOUNRTEVWHTO3DHTZMH2VZZOPJPEKLCFAHTXESLLYABHSA, seg_pos={0 0}, upload=5895]: Done uploading piece. err: <nil> Jul 27 04:06:23.953531796 (6h14m39.734465159s) [noise=true , address=45.91.116.6:4973 , piece_id=J3VULPU24CIM64VEVE3UHO62HPN6TJXGZU6BCJIS5JQJKHJK2LIQ, seg_pos={0 0}, upload=5895]: Done uploading piece. err: <nil> Jul 27 04:06:23.954068660 (6h14m39.735002029s) [noise=true , address=45.91.116.9:4262 , piece_id=M5ETD45HH2XZD6HDLTJ3JO2BOOIROXFWKVPTN42PSQEWG4F7KAAQ, seg_pos={0 0}, upload=5895]: Done uploading piece. err: <nil> Jul 27 04:06:23.954194339 (6h14m39.735127707s) [noise=true , address=45.91.116.8:4048 , piece_id=5REWMVVTBVLDQXQ5KAEHLY4VJL5SGF7ONQWDQQ3N2FDU3OAO6AZA, seg_pos={0 0}, upload=5895]: Done uploading piece. err: <nil> Jul 27 04:06:23.954601730 (6h14m39.735535112s) [noise=true , address=45.91.116.7:4011 , piece_id=XMH6MCCHZBBRQWHJFWLRGYHLQD4KUGSGL7MMZSKA5TZJCND3DUDA, seg_pos={0 0}, upload=5895]: Done uploading piece. err: <nil> Jul 27 04:06:23.954755872 (6h14m39.735689237s) [noise=true , address=45.91.116.13:4153 , piece_id=H4535KOCG3F4RKA6AKKMTJSOM6Q6PGRC6M7FR3UIS3BORTSP5UQA, seg_pos={0 0}, upload=5895]: Done uploading piece. err: <nil> Jul 27 04:06:23.955611883 (6h14m39.73654526s) [noise=true , address=77.90.31.19:4843 , piece_id=YML5RDQIK432Z4FSE4DKXIV5U47LQBEZRFMAVLANUSGFUQR23UWQ, seg_pos={0 0}, upload=5895]: Done uploading piece. err: <nil> Jul 27 04:06:23.956373391 (6h14m39.737306756s) [noise=true , address=77.90.31.15:4697 , piece_id=Q76TJDSZPSYZSJQNFVOI462U6DFILMH2RV3LQISODXSOCXGRKW6Q, seg_pos={0 0}, upload=5895]: Done uploading piece. err: <nil> Jul 27 04:06:23.958659122 (6h14m39.73959249s) [noise=true , address=77.90.31.17:4771 , piece_id=U2LF22GKCMKRZN6EWCF26ITDZQHBFZXCGSQXT33MCX6P57IGI35Q, seg_pos={0 0}, upload=5895]: Done uploading piece. err: <nil> Jul 27 04:06:35.058944013 (6h14m50.83987739s) [noise=true , address=104.36.82.16:4293 , piece_id=AHH6PRFUSGTDGB7H52DRITARA4IF6GFLVSRYS2Y6EPJCZSKUOYWQ, seg_pos={0 0}, upload=5895]: Done uploading piece. err: <nil> Jul 27 04:06:41.103204002 (6h14m56.884137386s) [noise=true , address=104.36.82.4:4804 , piece_id=IU6FDNVB5AUPA7MJ6RSXUTLBETPHOFGNJ7YGWBFIZSQFSO6W6BBA, seg_pos={0 0}, upload=5895]: Done uploading piece. err: <nil> Jul 27 04:06:43.191702285 (6h14m58.972635651s) [noise=true , address=104.36.82.6:4873 , piece_id=YP4VJORZZUL5EVCH3P7LD3LNVMHB7CWRTKHGIA7UROXE44N6MMZA, seg_pos={0 0}, upload=5895]: Done uploading piece. err: <nil> Jul 27 04:06:53.777973493 (6h15m9.558906861s) [noise=true , address=104.36.82.11:4115 , piece_id=YXMNVT6ZHGESGQ2BXKZUVR7U2CNNAI2PO7ZTMDG5UIOV7HTDDUTA, seg_pos={0 0}, upload=5895]: Done uploading piece. err: ecclient: upload failed (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR, address:104.36.82.11:4115): protocol: write tcp 159.223.143.39:34210->104.36.82.11:4115: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 04:06:53.778019532 (6h15m9.558952909s) [noise=true , address=104.36.82.12:4151 , piece_id=E2TEUTQBD4Y4AUHSCPMCIFEXBSY7S5HJWBKNKWWTUWEW2BLAAE6Q, seg_pos={0 0}, upload=5895]: Done uploading piece. err: ecclient: upload failed (node:1w6pwZMdeQKjt92qx6WbTWU5VPYJto5gS87f6iKvCBZH6LfyeB, address:104.36.82.12:4151): protocol: rpc: dial tcp 104.36.82.12:4151: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 04:06:53.778043653 (6h15m9.558977021s) [noise=true , address=104.36.82.13:4186 , piece_id=TBLDKALGJYBGR5NLCOCKJ4QKEFNIJXCKAFUBD7VR3UMNWEMKMKHQ, seg_pos={0 0}, upload=5895]: Done uploading piece. err: ecclient: upload failed (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht, address:104.36.82.13:4186): protocol: read tcp 159.223.143.39:34340->104.36.82.13:4186: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 04:06:53.778095228 (6h15m9.559028593s) [noise=true , address=104.36.82.10:4079 , piece_id=2SH5D6JOMLFR7VSN7ZY5CLMR5RHIRFYQ7Z5I2JIYTJQM43G4O6OQ, seg_pos={0 0}, upload=5895]: Done uploading piece. err: ecclient: upload failed (node:12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud, address:104.36.82.10:4079): protocol: write tcp 159.223.143.39:53840->104.36.82.10:4079: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 04:06:53.778142167 (6h15m9.559075532s) [seg_pos={0 0}, upload=5895]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 27 04:06:53.837032161 (6h15m9.617965543s) [upload=5895]: Done uploading segments... Jul 27 04:06:53.837055996 (6h15m9.617989366s) [upload=5895]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 901.895226ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.............................X..............................] duration: 1.043541745s FAIL (2025-07-27 04:30:04 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== {0 0}, upload=5911]: Begin upload segment... Jul 27 04:07:17.745307658 (6h15m33.526241043s) [seg_pos={0 0}, upload=5911]: Done begin upload segment. Jul 27 04:07:17.745349241 (6h15m33.526282625s) [noise=true , address=104.36.82.52:4098 , piece_id=M2ETF25QQ53RRGFPUKXJRINIWV7AWXSECEDR2AIM645IED72NURQ, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.745439694 (6h15m33.526373073s) [noise=true , address=104.36.82.56:4244 , piece_id=LQCZ5W54NBHWAVBVYK7HYSSJDC73E6FV4CP4YIM4L5H2N3OLOZXA, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.745972271 (6h15m33.526905654s) [noise=true , address=104.36.82.36:4650 , piece_id=BKJULWGPESBINNTSUYP6E4KZRXKPACVU5I7DOSZBO4YMK2V26HEQ, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.745980243 (6h15m33.526913621s) [noise=true , address=104.36.82.38:4722 , piece_id=B2MBQY5CTMEI6WSE7FTLOV5RUJ22SCNG7N2EEJILEATGJ4TIALXA, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.746403818 (6h15m33.527337198s) [noise=true , address=104.36.82.22:4534 , piece_id=CW4AQT3XUGA4A5OC6NUY4NYOOAMLN5FFJJN5PQBYEON7HPD5LCYQ, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.746455493 (6h15m33.527388875s) [noise=true , address=104.36.82.14:4248 , piece_id=LDSGCUM7Q2VX5CDFI4JRGEEG7D5H5Q5Z5WUWKU277VN2JFRX6UGA, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.746872127 (6h15m33.527805505s) [noise=true , address=104.36.82.40:4954 , piece_id=YLOHTJLRAED66OMEQYPZHNNRI77V6WOR4EHTEZOOEHMGZHJ2O3QQ, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.747123162 (6h15m33.52805654s) [noise=true , address=104.36.82.46:4496 , piece_id=GWNQR5WRGRRQBLSGWJFD3O3UNCMNE62JQNPIV5SYKBUIL3BQKA5A, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.747280678 (6h15m33.528214053s) [noise=true , address=104.36.82.47:4524 , piece_id=DONJFTKUQ2XCW6BOU7AVOKJ7XMDMBPGQVUPRANH7X2YS7PJXUKIA, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.747736400 (6h15m33.528669779s) [noise=true , address=104.36.82.44:4893 , piece_id=HUJICRJ6W54O3AZZR6H3KG6E4TGBTNS46O6XVMQM3PFAZE7A2HZA, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.747755734 (6h15m33.528689109s) [noise=true , address=104.36.82.55:4204 , piece_id=72WISGRHMOAZDV5EDGTBOMLBL4HDOF3CBEMQAAIRGILPC7J5JKAQ, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.748313124 (6h15m33.5292465s) [noise=true , address=104.36.82.58:4287 , piece_id=BVDYUEWU7TVEQDMZXX5SJNS4QICVKMTGFLCUV6P2YHOE7TI25VXA, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.748485217 (6h15m33.529418594s) [noise=true , address=104.36.82.53:4123 , piece_id=NXJRZ5BQSLXSOIOSUJ4MM4AGKHP4UIDUGEJKLUUX7YOGQJA6WWZA, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.748831329 (6h15m33.529764704s) [noise=true , address=104.36.82.42:4829 , piece_id=ZLWJFP3LGX3YLXBJYIR66DXTYEVDBKXKVXCQSF2PCYCYJBIEJGMA, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.749299293 (6h15m33.530232669s) [noise=true , address=104.36.82.8:4002 , piece_id=6XSPWQC3US355PS6IQ4K3IYNQZH5MAT4IVIPGSYRUM3XYPSCWE6Q, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.749790974 (6h15m33.53072436s) [noise=true , address=104.36.82.44:4901 , piece_id=N4YLCLR2QZSULUL2XWONEKBFUMNOU7KUX2QEOK4SSDFSCI7IL6LA, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.750478221 (6h15m33.531411597s) [noise=true , address=104.36.82.54:4172 , piece_id=ZNA53MWDRWPUH6FUR4DJERBDT2IOYGA3IYUADKXXIOLQJLQOXE7Q, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.752884255 (6h15m33.533817625s) [noise=true , address=64.49.13.140:4508 , piece_id=MZ52U75VOIK6Q7QFXMMYRRG7JXFWDFU2JWQFYPERUCMH3BUX6XPQ, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.879839555 (6h15m33.660772917s) [upload=5911]: Next returned nil segment Jul 27 04:07:17.879861389 (6h15m33.660794765s) [upload=5911]: Waiting for error group managing segments... Jul 27 04:07:17.888652438 (6h15m33.669585811s) [noise=true , address=104.36.82.58:4287 , piece_id=BVDYUEWU7TVEQDMZXX5SJNS4QICVKMTGFLCUV6P2YHOE7TI25VXA, seg_pos={0 0}, upload=5911]: Done uploading piece. err: <nil> Jul 27 04:07:17.952192045 (6h15m33.733125429s) [noise=true , address=104.36.82.53:4123 , piece_id=NXJRZ5BQSLXSOIOSUJ4MM4AGKHP4UIDUGEJKLUUX7YOGQJA6WWZA, seg_pos={0 0}, upload=5911]: Done uploading piece. err: <nil> Jul 27 04:07:17.981907539 (6h15m33.762840911s) [noise=true , address=104.36.82.56:4244 , piece_id=LQCZ5W54NBHWAVBVYK7HYSSJDC73E6FV4CP4YIM4L5H2N3OLOZXA, seg_pos={0 0}, upload=5911]: Done uploading piece. err: <nil> Jul 27 04:07:17.982370496 (6h15m33.7633039s) [noise=true , address=104.36.82.55:4204 , piece_id=72WISGRHMOAZDV5EDGTBOMLBL4HDOF3CBEMQAAIRGILPC7J5JKAQ, seg_pos={0 0}, upload=5911]: Done uploading piece. err: <nil> Jul 27 04:07:18.009548506 (6h15m33.790481886s) [noise=true , address=104.36.82.52:4098 , piece_id=M2ETF25QQ53RRGFPUKXJRINIWV7AWXSECEDR2AIM645IED72NURQ, seg_pos={0 0}, upload=5911]: Done uploading piece. err: <nil> Jul 27 04:07:18.023770598 (6h15m33.804703983s) [noise=true , address=104.36.82.40:4954 , piece_id=YLOHTJLRAED66OMEQYPZHNNRI77V6WOR4EHTEZOOEHMGZHJ2O3QQ, seg_pos={0 0}, upload=5911]: Done uploading piece. err: <nil> Jul 27 04:07:18.139745581 (6h15m33.92067895s) [noise=true , address=104.36.82.22:4534 , piece_id=CW4AQT3XUGA4A5OC6NUY4NYOOAMLN5FFJJN5PQBYEON7HPD5LCYQ, seg_pos={0 0}, upload=5911]: Done uploading piece. err: <nil> Jul 27 04:07:18.423748140 (6h15m34.204681511s) [noise=true , address=64.49.13.140:4508 , piece_id=MZ52U75VOIK6Q7QFXMMYRRG7JXFWDFU2JWQFYPERUCMH3BUX6XPQ, seg_pos={0 0}, upload=5911]: Done uploading piece. err: <nil> Jul 27 04:08:16.739940381 (6h16m32.52087375s) [noise=true , address=104.36.82.54:4172 , piece_id=ZNA53MWDRWPUH6FUR4DJERBDT2IOYGA3IYUADKXXIOLQJLQOXE7Q, seg_pos={0 0}, upload=5911]: Done uploading piece. err: <nil> Jul 27 04:08:17.177295397 (6h16m32.958228766s) [noise=true , address=104.36.82.8:4002 , piece_id=6XSPWQC3US355PS6IQ4K3IYNQZH5MAT4IVIPGSYRUM3XYPSCWE6Q, seg_pos={0 0}, upload=5911]: Done uploading piece. err: <nil> Jul 27 04:08:17.534296584 (6h16m33.315229953s) [noise=true , address=104.36.82.36:4650 , piece_id=BKJULWGPESBINNTSUYP6E4KZRXKPACVU5I7DOSZBO4YMK2V26HEQ, seg_pos={0 0}, upload=5911]: Done uploading piece. err: <nil> Jul 27 04:08:17.677851566 (6h16m33.458784943s) [noise=true , address=104.36.82.14:4248 , piece_id=LDSGCUM7Q2VX5CDFI4JRGEEG7D5H5Q5Z5WUWKU277VN2JFRX6UGA, seg_pos={0 0}, upload=5911]: Done uploading piece. err: ecclient: upload failed (node:1u6ir2PLqDa9xGm64cuqq1RqY4qeBa1fqCo9VghspBf54XEXqn, address:104.36.82.14:4248): protocol: rpc: dial tcp 104.36.82.14:4248: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 04:08:17.678115170 (6h16m33.459048549s) [noise=true , address=104.36.82.46:4496 , piece_id=GWNQR5WRGRRQBLSGWJFD3O3UNCMNE62JQNPIV5SYKBUIL3BQKA5A, seg_pos={0 0}, upload=5911]: Done uploading piece. err: ecclient: upload failed (node:12V8XhSWkyGUVYFR8MykifFAP1YAfGCwzqucDcMMfzw3xsVW77Z, address:104.36.82.46:4496): protocol: read tcp 159.223.143.39:52800->104.36.82.46:4496: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 04:08:17.678144409 (6h16m33.459077789s) [noise=true , address=104.36.82.44:4901 , piece_id=N4YLCLR2QZSULUL2XWONEKBFUMNOU7KUX2QEOK4SSDFSCI7IL6LA, seg_pos={0 0}, upload=5911]: Done uploading piece. err: ecclient: upload failed (node:1FHE84zvBb1MKwB1Ht7puTafgQxDEvPt4mYqkYVSSufAMpWQA1, address:104.36.82.44:4901): protocol: rpc: dial tcp 104.36.82.44:4901: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 04:08:17.678229622 (6h16m33.459162988s) [noise=true , address=104.36.82.44:4893 , piece_id=HUJICRJ6W54O3AZZR6H3KG6E4TGBTNS46O6XVMQM3PFAZE7A2HZA, seg_pos={0 0}, upload=5911]: Done uploading piece. err: ecclient: upload failed (node:17VMFZ2T8jrSJVtchciJ9SnyrR6JstvcMoXrN5Q3zTD1aQU1oJ, address:104.36.82.44:4893): protocol: write tcp 159.223.143.39:35302->104.36.82.44:4893: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 04:08:17.678310218 (6h16m33.459243583s) [noise=true , address=104.36.82.42:4829 , piece_id=ZLWJFP3LGX3YLXBJYIR66DXTYEVDBKXKVXCQSF2PCYCYJBIEJGMA, seg_pos={0 0}, upload=5911]: Done uploading piece. err: ecclient: upload failed (node:1F5T8Vt1x4nXFaSenvGBCYjxhRL1H1sxRvzLjW3CqzwDVhq39F, address:104.36.82.42:4829): protocol: rpc: dial tcp 104.36.82.42:4829: i/o timeout; conn closed: op; piecestore: piecestore close: context deadline exceeded Jul 27 04:08:17.678412989 (6h16m33.459346365s) [noise=true , address=104.36.82.38:4722 , piece_id=B2MBQY5CTMEI6WSE7FTLOV5RUJ22SCNG7N2EEJILEATGJ4TIALXA, seg_pos={0 0}, upload=5911]: Done uploading piece. err: ecclient: upload failed (node:12Zw3dtUg9gARz52KhwDLTjcqC3CNm7dpPzpdx7P7PbL1jwe8uc, address:104.36.82.38:4722): protocol: read tcp 159.223.143.39:56048->104.36.82.38:4722: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 04:08:17.678455247 (6h16m33.459388624s) [noise=true , address=104.36.82.47:4524 , piece_id=DONJFTKUQ2XCW6BOU7AVOKJ7XMDMBPGQVUPRANH7X2YS7PJXUKIA, seg_pos={0 0}, upload=5911]: Done uploading piece. err: ecclient: upload failed (node:1xfbCzjSwFDRrR1gPggeN8tNCLbRRWDVzCJ6mqhZsbVSe8d4s4, address:104.36.82.47:4524): protocol: read tcp 159.223.143.39:46694->104.36.82.47:4524: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 04:08:17.678510668 (6h16m33.459444052s) [seg_pos={0 0}, upload=5911]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Jul 27 04:08:17.706442864 (6h16m33.487376232s) [upload=5911]: Done uploading segments... Jul 27 04:08:17.706495329 (6h16m33.487428705s) [upload=5911]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 1.043577658s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................................X.....................] duration: 11.793905353s FAIL (2025-07-27 04:29:58 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 12) Duration: 11.793938954s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [..............................X.............................] duration: 16.605672279s FAIL (2025-07-27 04:29:24 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== {0 0}, upload=5911]: Begin upload segment... Jul 27 04:07:17.745307658 (6h15m33.526241043s) [seg_pos={0 0}, upload=5911]: Done begin upload segment. Jul 27 04:07:17.745349241 (6h15m33.526282625s) [noise=true , address=104.36.82.52:4098 , piece_id=M2ETF25QQ53RRGFPUKXJRINIWV7AWXSECEDR2AIM645IED72NURQ, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.745439694 (6h15m33.526373073s) [noise=true , address=104.36.82.56:4244 , piece_id=LQCZ5W54NBHWAVBVYK7HYSSJDC73E6FV4CP4YIM4L5H2N3OLOZXA, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.745972271 (6h15m33.526905654s) [noise=true , address=104.36.82.36:4650 , piece_id=BKJULWGPESBINNTSUYP6E4KZRXKPACVU5I7DOSZBO4YMK2V26HEQ, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.745980243 (6h15m33.526913621s) [noise=true , address=104.36.82.38:4722 , piece_id=B2MBQY5CTMEI6WSE7FTLOV5RUJ22SCNG7N2EEJILEATGJ4TIALXA, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.746403818 (6h15m33.527337198s) [noise=true , address=104.36.82.22:4534 , piece_id=CW4AQT3XUGA4A5OC6NUY4NYOOAMLN5FFJJN5PQBYEON7HPD5LCYQ, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.746455493 (6h15m33.527388875s) [noise=true , address=104.36.82.14:4248 , piece_id=LDSGCUM7Q2VX5CDFI4JRGEEG7D5H5Q5Z5WUWKU277VN2JFRX6UGA, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.746872127 (6h15m33.527805505s) [noise=true , address=104.36.82.40:4954 , piece_id=YLOHTJLRAED66OMEQYPZHNNRI77V6WOR4EHTEZOOEHMGZHJ2O3QQ, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.747123162 (6h15m33.52805654s) [noise=true , address=104.36.82.46:4496 , piece_id=GWNQR5WRGRRQBLSGWJFD3O3UNCMNE62JQNPIV5SYKBUIL3BQKA5A, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.747280678 (6h15m33.528214053s) [noise=true , address=104.36.82.47:4524 , piece_id=DONJFTKUQ2XCW6BOU7AVOKJ7XMDMBPGQVUPRANH7X2YS7PJXUKIA, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.747736400 (6h15m33.528669779s) [noise=true , address=104.36.82.44:4893 , piece_id=HUJICRJ6W54O3AZZR6H3KG6E4TGBTNS46O6XVMQM3PFAZE7A2HZA, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.747755734 (6h15m33.528689109s) [noise=true , address=104.36.82.55:4204 , piece_id=72WISGRHMOAZDV5EDGTBOMLBL4HDOF3CBEMQAAIRGILPC7J5JKAQ, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.748313124 (6h15m33.5292465s) [noise=true , address=104.36.82.58:4287 , piece_id=BVDYUEWU7TVEQDMZXX5SJNS4QICVKMTGFLCUV6P2YHOE7TI25VXA, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.748485217 (6h15m33.529418594s) [noise=true , address=104.36.82.53:4123 , piece_id=NXJRZ5BQSLXSOIOSUJ4MM4AGKHP4UIDUGEJKLUUX7YOGQJA6WWZA, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.748831329 (6h15m33.529764704s) [noise=true , address=104.36.82.42:4829 , piece_id=ZLWJFP3LGX3YLXBJYIR66DXTYEVDBKXKVXCQSF2PCYCYJBIEJGMA, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.749299293 (6h15m33.530232669s) [noise=true , address=104.36.82.8:4002 , piece_id=6XSPWQC3US355PS6IQ4K3IYNQZH5MAT4IVIPGSYRUM3XYPSCWE6Q, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.749790974 (6h15m33.53072436s) [noise=true , address=104.36.82.44:4901 , piece_id=N4YLCLR2QZSULUL2XWONEKBFUMNOU7KUX2QEOK4SSDFSCI7IL6LA, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.750478221 (6h15m33.531411597s) [noise=true , address=104.36.82.54:4172 , piece_id=ZNA53MWDRWPUH6FUR4DJERBDT2IOYGA3IYUADKXXIOLQJLQOXE7Q, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.752884255 (6h15m33.533817625s) [noise=true , address=64.49.13.140:4508 , piece_id=MZ52U75VOIK6Q7QFXMMYRRG7JXFWDFU2JWQFYPERUCMH3BUX6XPQ, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.879839555 (6h15m33.660772917s) [upload=5911]: Next returned nil segment Jul 27 04:07:17.879861389 (6h15m33.660794765s) [upload=5911]: Waiting for error group managing segments... Jul 27 04:07:17.888652438 (6h15m33.669585811s) [noise=true , address=104.36.82.58:4287 , piece_id=BVDYUEWU7TVEQDMZXX5SJNS4QICVKMTGFLCUV6P2YHOE7TI25VXA, seg_pos={0 0}, upload=5911]: Done uploading piece. err: <nil> Jul 27 04:07:17.952192045 (6h15m33.733125429s) [noise=true , address=104.36.82.53:4123 , piece_id=NXJRZ5BQSLXSOIOSUJ4MM4AGKHP4UIDUGEJKLUUX7YOGQJA6WWZA, seg_pos={0 0}, upload=5911]: Done uploading piece. err: <nil> Jul 27 04:07:17.981907539 (6h15m33.762840911s) [noise=true , address=104.36.82.56:4244 , piece_id=LQCZ5W54NBHWAVBVYK7HYSSJDC73E6FV4CP4YIM4L5H2N3OLOZXA, seg_pos={0 0}, upload=5911]: Done uploading piece. err: <nil> Jul 27 04:07:17.982370496 (6h15m33.7633039s) [noise=true , address=104.36.82.55:4204 , piece_id=72WISGRHMOAZDV5EDGTBOMLBL4HDOF3CBEMQAAIRGILPC7J5JKAQ, seg_pos={0 0}, upload=5911]: Done uploading piece. err: <nil> Jul 27 04:07:18.009548506 (6h15m33.790481886s) [noise=true , address=104.36.82.52:4098 , piece_id=M2ETF25QQ53RRGFPUKXJRINIWV7AWXSECEDR2AIM645IED72NURQ, seg_pos={0 0}, upload=5911]: Done uploading piece. err: <nil> Jul 27 04:07:18.023770598 (6h15m33.804703983s) [noise=true , address=104.36.82.40:4954 , piece_id=YLOHTJLRAED66OMEQYPZHNNRI77V6WOR4EHTEZOOEHMGZHJ2O3QQ, seg_pos={0 0}, upload=5911]: Done uploading piece. err: <nil> Jul 27 04:07:18.139745581 (6h15m33.92067895s) [noise=true , address=104.36.82.22:4534 , piece_id=CW4AQT3XUGA4A5OC6NUY4NYOOAMLN5FFJJN5PQBYEON7HPD5LCYQ, seg_pos={0 0}, upload=5911]: Done uploading piece. err: <nil> Jul 27 04:07:18.423748140 (6h15m34.204681511s) [noise=true , address=64.49.13.140:4508 , piece_id=MZ52U75VOIK6Q7QFXMMYRRG7JXFWDFU2JWQFYPERUCMH3BUX6XPQ, seg_pos={0 0}, upload=5911]: Done uploading piece. err: <nil> Jul 27 04:08:16.739940381 (6h16m32.52087375s) [noise=true , address=104.36.82.54:4172 , piece_id=ZNA53MWDRWPUH6FUR4DJERBDT2IOYGA3IYUADKXXIOLQJLQOXE7Q, seg_pos={0 0}, upload=5911]: Done uploading piece. err: <nil> Jul 27 04:08:17.177295397 (6h16m32.958228766s) [noise=true , address=104.36.82.8:4002 , piece_id=6XSPWQC3US355PS6IQ4K3IYNQZH5MAT4IVIPGSYRUM3XYPSCWE6Q, seg_pos={0 0}, upload=5911]: Done uploading piece. err: <nil> Jul 27 04:08:17.534296584 (6h16m33.315229953s) [noise=true , address=104.36.82.36:4650 , piece_id=BKJULWGPESBINNTSUYP6E4KZRXKPACVU5I7DOSZBO4YMK2V26HEQ, seg_pos={0 0}, upload=5911]: Done uploading piece. err: <nil> Jul 27 04:08:17.677851566 (6h16m33.458784943s) [noise=true , address=104.36.82.14:4248 , piece_id=LDSGCUM7Q2VX5CDFI4JRGEEG7D5H5Q5Z5WUWKU277VN2JFRX6UGA, seg_pos={0 0}, upload=5911]: Done uploading piece. err: ecclient: upload failed (node:1u6ir2PLqDa9xGm64cuqq1RqY4qeBa1fqCo9VghspBf54XEXqn, address:104.36.82.14:4248): protocol: rpc: dial tcp 104.36.82.14:4248: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 04:08:17.678115170 (6h16m33.459048549s) [noise=true , address=104.36.82.46:4496 , piece_id=GWNQR5WRGRRQBLSGWJFD3O3UNCMNE62JQNPIV5SYKBUIL3BQKA5A, seg_pos={0 0}, upload=5911]: Done uploading piece. err: ecclient: upload failed (node:12V8XhSWkyGUVYFR8MykifFAP1YAfGCwzqucDcMMfzw3xsVW77Z, address:104.36.82.46:4496): protocol: read tcp 159.223.143.39:52800->104.36.82.46:4496: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 04:08:17.678144409 (6h16m33.459077789s) [noise=true , address=104.36.82.44:4901 , piece_id=N4YLCLR2QZSULUL2XWONEKBFUMNOU7KUX2QEOK4SSDFSCI7IL6LA, seg_pos={0 0}, upload=5911]: Done uploading piece. err: ecclient: upload failed (node:1FHE84zvBb1MKwB1Ht7puTafgQxDEvPt4mYqkYVSSufAMpWQA1, address:104.36.82.44:4901): protocol: rpc: dial tcp 104.36.82.44:4901: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 04:08:17.678229622 (6h16m33.459162988s) [noise=true , address=104.36.82.44:4893 , piece_id=HUJICRJ6W54O3AZZR6H3KG6E4TGBTNS46O6XVMQM3PFAZE7A2HZA, seg_pos={0 0}, upload=5911]: Done uploading piece. err: ecclient: upload failed (node:17VMFZ2T8jrSJVtchciJ9SnyrR6JstvcMoXrN5Q3zTD1aQU1oJ, address:104.36.82.44:4893): protocol: write tcp 159.223.143.39:35302->104.36.82.44:4893: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 04:08:17.678310218 (6h16m33.459243583s) [noise=true , address=104.36.82.42:4829 , piece_id=ZLWJFP3LGX3YLXBJYIR66DXTYEVDBKXKVXCQSF2PCYCYJBIEJGMA, seg_pos={0 0}, upload=5911]: Done uploading piece. err: ecclient: upload failed (node:1F5T8Vt1x4nXFaSenvGBCYjxhRL1H1sxRvzLjW3CqzwDVhq39F, address:104.36.82.42:4829): protocol: rpc: dial tcp 104.36.82.42:4829: i/o timeout; conn closed: op; piecestore: piecestore close: context deadline exceeded Jul 27 04:08:17.678412989 (6h16m33.459346365s) [noise=true , address=104.36.82.38:4722 , piece_id=B2MBQY5CTMEI6WSE7FTLOV5RUJ22SCNG7N2EEJILEATGJ4TIALXA, seg_pos={0 0}, upload=5911]: Done uploading piece. err: ecclient: upload failed (node:12Zw3dtUg9gARz52KhwDLTjcqC3CNm7dpPzpdx7P7PbL1jwe8uc, address:104.36.82.38:4722): protocol: read tcp 159.223.143.39:56048->104.36.82.38:4722: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 04:08:17.678455247 (6h16m33.459388624s) [noise=true , address=104.36.82.47:4524 , piece_id=DONJFTKUQ2XCW6BOU7AVOKJ7XMDMBPGQVUPRANH7X2YS7PJXUKIA, seg_pos={0 0}, upload=5911]: Done uploading piece. err: ecclient: upload failed (node:1xfbCzjSwFDRrR1gPggeN8tNCLbRRWDVzCJ6mqhZsbVSe8d4s4, address:104.36.82.47:4524): protocol: read tcp 159.223.143.39:46694->104.36.82.47:4524: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 04:08:17.678510668 (6h16m33.459444052s) [seg_pos={0 0}, upload=5911]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Jul 27 04:08:17.706442864 (6h16m33.487376232s) [upload=5911]: Done uploading segments... Jul 27 04:08:17.706495329 (6h16m33.487428705s) [upload=5911]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 16.605760602s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................X.................................] duration: 497.847249ms FAIL (2025-07-27 04:29:05 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== ece_id=NYOFDEZ7CV4PHGQ6JZMEEVAYGJRPYG6IBW7MYJO42HMWOJM7PHZA, seg_pos={0 0}, upload=5879]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12crcXpNe9NgeXTsUpooLSBgrSUFbRxbSbanLa39w2Hr5mGFww): protocol: read tcp 159.223.143.39:53630->104.36.82.5:4840: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 04:05:32.508904485 (6h13m48.289837866s) [seg_pos={0 0}, upload=5879]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 27 04:05:32.509028070 (6h13m48.289961457s) [upload=5879]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 27 04:05:32.509034645 (6h13m48.289968129s) [upload=5879]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 27 04:05:32.581834839 (6h13m48.362768225s) [upload=5879]: Done uploading segments... Jul 27 04:05:32.581875919 (6h13m48.362809298s) [upload=5879]: Upload finished. err: <nil> Jul 27 04:06:23.829155331 (6h14m39.610088696s) [upload=5895]: Starting upload Jul 27 04:06:23.829225021 (6h14m39.610158398s) [upload=5895]: Done starting upload Jul 27 04:06:23.829294606 (6h14m39.61022799s) [upload=5895]: Uploading segments... Jul 27 04:06:23.829577656 (6h14m39.610511053s) [upload=5895]: Got next segment. Inline: false Jul 27 04:06:23.829649664 (6h14m39.610583027s) [upload=5895]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 27 04:06:23.829657065 (6h14m39.61059043s) [upload=5895]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 27 04:06:23.861405530 (6h14m39.642338913s) [seg_pos={0 0}, upload=5895]: Begin upload segment... Jul 27 04:06:23.861476351 (6h14m39.642409713s) [seg_pos={0 0}, upload=5895]: Done begin upload segment. Jul 27 04:06:23.861509120 (6h14m39.642442506s) [noise=true , address=77.90.31.19:4843 , piece_id=YML5RDQIK432Z4FSE4DKXIV5U47LQBEZRFMAVLANUSGFUQR23UWQ, seg_pos={0 0}, upload=5895]: Uploading piece... Jul 27 04:06:23.861953723 (6h14m39.6428871s) [noise=true , address=45.91.116.10:4081 , piece_id=XGDTSEZOUNRTEVWHTO3DHTZMH2VZZOPJPEKLCFAHTXESLLYABHSA, seg_pos={0 0}, upload=5895]: Uploading piece... Jul 27 04:06:23.862060983 (6h14m39.642994358s) [noise=true , address=45.91.116.9:4262 , piece_id=M5ETD45HH2XZD6HDLTJ3JO2BOOIROXFWKVPTN42PSQEWG4F7KAAQ, seg_pos={0 0}, upload=5895]: Uploading piece... Jul 27 04:06:23.862759464 (6h14m39.64369284s) [noise=true , address=104.36.82.4:4804 , piece_id=IU6FDNVB5AUPA7MJ6RSXUTLBETPHOFGNJ7YGWBFIZSQFSO6W6BBA, seg_pos={0 0}, upload=5895]: Uploading piece... Jul 27 04:06:23.862835086 (6h14m39.643768461s) [noise=true , address=104.36.82.6:4873 , piece_id=YP4VJORZZUL5EVCH3P7LD3LNVMHB7CWRTKHGIA7UROXE44N6MMZA, seg_pos={0 0}, upload=5895]: Uploading piece... Jul 27 04:06:23.863593880 (6h14m39.644527258s) [noise=true , address=77.90.31.15:4697 , piece_id=Q76TJDSZPSYZSJQNFVOI462U6DFILMH2RV3LQISODXSOCXGRKW6Q, seg_pos={0 0}, upload=5895]: Uploading piece... Jul 27 04:06:23.863656891 (6h14m39.644590266s) [noise=true , address=45.91.116.7:4011 , piece_id=XMH6MCCHZBBRQWHJFWLRGYHLQD4KUGSGL7MMZSKA5TZJCND3DUDA, seg_pos={0 0}, upload=5895]: Uploading piece... Jul 27 04:06:23.864408612 (6h14m39.645341987s) [noise=true , address=45.91.116.6:4973 , piece_id=J3VULPU24CIM64VEVE3UHO62HPN6TJXGZU6BCJIS5JQJKHJK2LIQ, seg_pos={0 0}, upload=5895]: Uploading piece... Jul 27 04:06:23.864521549 (6h14m39.64545491s) [noise=true , address=104.36.82.10:4079 , piece_id=2SH5D6JOMLFR7VSN7ZY5CLMR5RHIRFYQ7Z5I2JIYTJQM43G4O6OQ, seg_pos={0 0}, upload=5895]: Uploading piece... Jul 27 04:06:23.865518491 (6h14m39.646451872s) [noise=true , address=77.90.31.17:4771 , piece_id=U2LF22GKCMKRZN6EWCF26ITDZQHBFZXCGSQXT33MCX6P57IGI35Q, seg_pos={0 0}, upload=5895]: Uploading piece... Jul 27 04:06:23.865654950 (6h14m39.646588325s) [noise=true , address=45.91.116.8:4048 , piece_id=5REWMVVTBVLDQXQ5KAEHLY4VJL5SGF7ONQWDQQ3N2FDU3OAO6AZA, seg_pos={0 0}, upload=5895]: Uploading piece... Jul 27 04:06:23.866190892 (6h14m39.647124271s) [noise=true , address=104.36.82.12:4151 , piece_id=E2TEUTQBD4Y4AUHSCPMCIFEXBSY7S5HJWBKNKWWTUWEW2BLAAE6Q, seg_pos={0 0}, upload=5895]: Uploading piece... Jul 27 04:06:23.866412590 (6h14m39.647345968s) [noise=true , address=104.36.82.11:4115 , piece_id=YXMNVT6ZHGESGQ2BXKZUVR7U2CNNAI2PO7ZTMDG5UIOV7HTDDUTA, seg_pos={0 0}, upload=5895]: Uploading piece... Jul 27 04:06:23.866715972 (6h14m39.647649359s) [noise=true , address=45.91.116.13:4153 , piece_id=H4535KOCG3F4RKA6AKKMTJSOM6Q6PGRC6M7FR3UIS3BORTSP5UQA, seg_pos={0 0}, upload=5895]: Uploading piece... Jul 27 04:06:23.866998965 (6h14m39.64793234s) [noise=true , address=104.36.82.13:4186 , piece_id=TBLDKALGJYBGR5NLCOCKJ4QKEFNIJXCKAFUBD7VR3UMNWEMKMKHQ, seg_pos={0 0}, upload=5895]: Uploading piece... Jul 27 04:06:23.867230593 (6h14m39.648163974s) [noise=true , address=104.36.82.16:4293 , piece_id=AHH6PRFUSGTDGB7H52DRITARA4IF6GFLVSRYS2Y6EPJCZSKUOYWQ, seg_pos={0 0}, upload=5895]: Uploading piece... Jul 27 04:06:23.949422808 (6h14m39.730356187s) [upload=5895]: Next returned nil segment Jul 27 04:06:23.949442576 (6h14m39.73037596s) [upload=5895]: Waiting for error group managing segments... Jul 27 04:06:23.953497426 (6h14m39.734430802s) [noise=true , address=45.91.116.10:4081 , piece_id=XGDTSEZOUNRTEVWHTO3DHTZMH2VZZOPJPEKLCFAHTXESLLYABHSA, seg_pos={0 0}, upload=5895]: Done uploading piece. err: <nil> Jul 27 04:06:23.953531796 (6h14m39.734465159s) [noise=true , address=45.91.116.6:4973 , piece_id=J3VULPU24CIM64VEVE3UHO62HPN6TJXGZU6BCJIS5JQJKHJK2LIQ, seg_pos={0 0}, upload=5895]: Done uploading piece. err: <nil> Jul 27 04:06:23.954068660 (6h14m39.735002029s) [noise=true , address=45.91.116.9:4262 , piece_id=M5ETD45HH2XZD6HDLTJ3JO2BOOIROXFWKVPTN42PSQEWG4F7KAAQ, seg_pos={0 0}, upload=5895]: Done uploading piece. err: <nil> Jul 27 04:06:23.954194339 (6h14m39.735127707s) [noise=true , address=45.91.116.8:4048 , piece_id=5REWMVVTBVLDQXQ5KAEHLY4VJL5SGF7ONQWDQQ3N2FDU3OAO6AZA, seg_pos={0 0}, upload=5895]: Done uploading piece. err: <nil> Jul 27 04:06:23.954601730 (6h14m39.735535112s) [noise=true , address=45.91.116.7:4011 , piece_id=XMH6MCCHZBBRQWHJFWLRGYHLQD4KUGSGL7MMZSKA5TZJCND3DUDA, seg_pos={0 0}, upload=5895]: Done uploading piece. err: <nil> Jul 27 04:06:23.954755872 (6h14m39.735689237s) [noise=true , address=45.91.116.13:4153 , piece_id=H4535KOCG3F4RKA6AKKMTJSOM6Q6PGRC6M7FR3UIS3BORTSP5UQA, seg_pos={0 0}, upload=5895]: Done uploading piece. err: <nil> Jul 27 04:06:23.955611883 (6h14m39.73654526s) [noise=true , address=77.90.31.19:4843 , piece_id=YML5RDQIK432Z4FSE4DKXIV5U47LQBEZRFMAVLANUSGFUQR23UWQ, seg_pos={0 0}, upload=5895]: Done uploading piece. err: <nil> Jul 27 04:06:23.956373391 (6h14m39.737306756s) [noise=true , address=77.90.31.15:4697 , piece_id=Q76TJDSZPSYZSJQNFVOI462U6DFILMH2RV3LQISODXSOCXGRKW6Q, seg_pos={0 0}, upload=5895]: Done uploading piece. err: <nil> Jul 27 04:06:23.958659122 (6h14m39.73959249s) [noise=true , address=77.90.31.17:4771 , piece_id=U2LF22GKCMKRZN6EWCF26ITDZQHBFZXCGSQXT33MCX6P57IGI35Q, seg_pos={0 0}, upload=5895]: Done uploading piece. err: <nil> Jul 27 04:06:35.058944013 (6h14m50.83987739s) [noise=true , address=104.36.82.16:4293 , piece_id=AHH6PRFUSGTDGB7H52DRITARA4IF6GFLVSRYS2Y6EPJCZSKUOYWQ, seg_pos={0 0}, upload=5895]: Done uploading piece. err: <nil> Jul 27 04:06:41.103204002 (6h14m56.884137386s) [noise=true , address=104.36.82.4:4804 , piece_id=IU6FDNVB5AUPA7MJ6RSXUTLBETPHOFGNJ7YGWBFIZSQFSO6W6BBA, seg_pos={0 0}, upload=5895]: Done uploading piece. err: <nil> Jul 27 04:06:43.191702285 (6h14m58.972635651s) [noise=true , address=104.36.82.6:4873 , piece_id=YP4VJORZZUL5EVCH3P7LD3LNVMHB7CWRTKHGIA7UROXE44N6MMZA, seg_pos={0 0}, upload=5895]: Done uploading piece. err: <nil> Jul 27 04:06:53.777973493 (6h15m9.558906861s) [noise=true , address=104.36.82.11:4115 , piece_id=YXMNVT6ZHGESGQ2BXKZUVR7U2CNNAI2PO7ZTMDG5UIOV7HTDDUTA, seg_pos={0 0}, upload=5895]: Done uploading piece. err: ecclient: upload failed (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR, address:104.36.82.11:4115): protocol: write tcp 159.223.143.39:34210->104.36.82.11:4115: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 04:06:53.778019532 (6h15m9.558952909s) [noise=true , address=104.36.82.12:4151 , piece_id=E2TEUTQBD4Y4AUHSCPMCIFEXBSY7S5HJWBKNKWWTUWEW2BLAAE6Q, seg_pos={0 0}, upload=5895]: Done uploading piece. err: ecclient: upload failed (node:1w6pwZMdeQKjt92qx6WbTWU5VPYJto5gS87f6iKvCBZH6LfyeB, address:104.36.82.12:4151): protocol: rpc: dial tcp 104.36.82.12:4151: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 04:06:53.778043653 (6h15m9.558977021s) [noise=true , address=104.36.82.13:4186 , piece_id=TBLDKALGJYBGR5NLCOCKJ4QKEFNIJXCKAFUBD7VR3UMNWEMKMKHQ, seg_pos={0 0}, upload=5895]: Done uploading piece. err: ecclient: upload failed (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht, address:104.36.82.13:4186): protocol: read tcp 159.223.143.39:34340->104.36.82.13:4186: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 04:06:53.778095228 (6h15m9.559028593s) [noise=true , address=104.36.82.10:4079 , piece_id=2SH5D6JOMLFR7VSN7ZY5CLMR5RHIRFYQ7Z5I2JIYTJQM43G4O6OQ, seg_pos={0 0}, upload=5895]: Done uploading piece. err: ecclient: upload failed (node:12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud, address:104.36.82.10:4079): protocol: write tcp 159.223.143.39:53840->104.36.82.10:4079: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 04:06:53.778142167 (6h15m9.559075532s) [seg_pos={0 0}, upload=5895]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 27 04:06:53.837032161 (6h15m9.617965543s) [upload=5895]: Done uploading segments... Jul 27 04:06:53.837055996 (6h15m9.617989366s) [upload=5895]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 497.87939ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................................X....................] duration: 505.758528ms FAIL (2025-07-27 04:29:05 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 12) Duration: 506.50678ms core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................................X...................] duration: 410.420429ms FAIL (2025-07-27 04:28:28 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 12) Duration: 410.478126ms core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [...............................X............................] duration: 4.276112506s FAIL (2025-07-27 04:28:21 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== {0 0}, upload=5911]: Begin upload segment... Jul 27 04:07:17.745307658 (6h15m33.526241043s) [seg_pos={0 0}, upload=5911]: Done begin upload segment. Jul 27 04:07:17.745349241 (6h15m33.526282625s) [noise=true , address=104.36.82.52:4098 , piece_id=M2ETF25QQ53RRGFPUKXJRINIWV7AWXSECEDR2AIM645IED72NURQ, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.745439694 (6h15m33.526373073s) [noise=true , address=104.36.82.56:4244 , piece_id=LQCZ5W54NBHWAVBVYK7HYSSJDC73E6FV4CP4YIM4L5H2N3OLOZXA, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.745972271 (6h15m33.526905654s) [noise=true , address=104.36.82.36:4650 , piece_id=BKJULWGPESBINNTSUYP6E4KZRXKPACVU5I7DOSZBO4YMK2V26HEQ, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.745980243 (6h15m33.526913621s) [noise=true , address=104.36.82.38:4722 , piece_id=B2MBQY5CTMEI6WSE7FTLOV5RUJ22SCNG7N2EEJILEATGJ4TIALXA, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.746403818 (6h15m33.527337198s) [noise=true , address=104.36.82.22:4534 , piece_id=CW4AQT3XUGA4A5OC6NUY4NYOOAMLN5FFJJN5PQBYEON7HPD5LCYQ, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.746455493 (6h15m33.527388875s) [noise=true , address=104.36.82.14:4248 , piece_id=LDSGCUM7Q2VX5CDFI4JRGEEG7D5H5Q5Z5WUWKU277VN2JFRX6UGA, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.746872127 (6h15m33.527805505s) [noise=true , address=104.36.82.40:4954 , piece_id=YLOHTJLRAED66OMEQYPZHNNRI77V6WOR4EHTEZOOEHMGZHJ2O3QQ, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.747123162 (6h15m33.52805654s) [noise=true , address=104.36.82.46:4496 , piece_id=GWNQR5WRGRRQBLSGWJFD3O3UNCMNE62JQNPIV5SYKBUIL3BQKA5A, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.747280678 (6h15m33.528214053s) [noise=true , address=104.36.82.47:4524 , piece_id=DONJFTKUQ2XCW6BOU7AVOKJ7XMDMBPGQVUPRANH7X2YS7PJXUKIA, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.747736400 (6h15m33.528669779s) [noise=true , address=104.36.82.44:4893 , piece_id=HUJICRJ6W54O3AZZR6H3KG6E4TGBTNS46O6XVMQM3PFAZE7A2HZA, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.747755734 (6h15m33.528689109s) [noise=true , address=104.36.82.55:4204 , piece_id=72WISGRHMOAZDV5EDGTBOMLBL4HDOF3CBEMQAAIRGILPC7J5JKAQ, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.748313124 (6h15m33.5292465s) [noise=true , address=104.36.82.58:4287 , piece_id=BVDYUEWU7TVEQDMZXX5SJNS4QICVKMTGFLCUV6P2YHOE7TI25VXA, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.748485217 (6h15m33.529418594s) [noise=true , address=104.36.82.53:4123 , piece_id=NXJRZ5BQSLXSOIOSUJ4MM4AGKHP4UIDUGEJKLUUX7YOGQJA6WWZA, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.748831329 (6h15m33.529764704s) [noise=true , address=104.36.82.42:4829 , piece_id=ZLWJFP3LGX3YLXBJYIR66DXTYEVDBKXKVXCQSF2PCYCYJBIEJGMA, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.749299293 (6h15m33.530232669s) [noise=true , address=104.36.82.8:4002 , piece_id=6XSPWQC3US355PS6IQ4K3IYNQZH5MAT4IVIPGSYRUM3XYPSCWE6Q, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.749790974 (6h15m33.53072436s) [noise=true , address=104.36.82.44:4901 , piece_id=N4YLCLR2QZSULUL2XWONEKBFUMNOU7KUX2QEOK4SSDFSCI7IL6LA, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.750478221 (6h15m33.531411597s) [noise=true , address=104.36.82.54:4172 , piece_id=ZNA53MWDRWPUH6FUR4DJERBDT2IOYGA3IYUADKXXIOLQJLQOXE7Q, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.752884255 (6h15m33.533817625s) [noise=true , address=64.49.13.140:4508 , piece_id=MZ52U75VOIK6Q7QFXMMYRRG7JXFWDFU2JWQFYPERUCMH3BUX6XPQ, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.879839555 (6h15m33.660772917s) [upload=5911]: Next returned nil segment Jul 27 04:07:17.879861389 (6h15m33.660794765s) [upload=5911]: Waiting for error group managing segments... Jul 27 04:07:17.888652438 (6h15m33.669585811s) [noise=true , address=104.36.82.58:4287 , piece_id=BVDYUEWU7TVEQDMZXX5SJNS4QICVKMTGFLCUV6P2YHOE7TI25VXA, seg_pos={0 0}, upload=5911]: Done uploading piece. err: <nil> Jul 27 04:07:17.952192045 (6h15m33.733125429s) [noise=true , address=104.36.82.53:4123 , piece_id=NXJRZ5BQSLXSOIOSUJ4MM4AGKHP4UIDUGEJKLUUX7YOGQJA6WWZA, seg_pos={0 0}, upload=5911]: Done uploading piece. err: <nil> Jul 27 04:07:17.981907539 (6h15m33.762840911s) [noise=true , address=104.36.82.56:4244 , piece_id=LQCZ5W54NBHWAVBVYK7HYSSJDC73E6FV4CP4YIM4L5H2N3OLOZXA, seg_pos={0 0}, upload=5911]: Done uploading piece. err: <nil> Jul 27 04:07:17.982370496 (6h15m33.7633039s) [noise=true , address=104.36.82.55:4204 , piece_id=72WISGRHMOAZDV5EDGTBOMLBL4HDOF3CBEMQAAIRGILPC7J5JKAQ, seg_pos={0 0}, upload=5911]: Done uploading piece. err: <nil> Jul 27 04:07:18.009548506 (6h15m33.790481886s) [noise=true , address=104.36.82.52:4098 , piece_id=M2ETF25QQ53RRGFPUKXJRINIWV7AWXSECEDR2AIM645IED72NURQ, seg_pos={0 0}, upload=5911]: Done uploading piece. err: <nil> Jul 27 04:07:18.023770598 (6h15m33.804703983s) [noise=true , address=104.36.82.40:4954 , piece_id=YLOHTJLRAED66OMEQYPZHNNRI77V6WOR4EHTEZOOEHMGZHJ2O3QQ, seg_pos={0 0}, upload=5911]: Done uploading piece. err: <nil> Jul 27 04:07:18.139745581 (6h15m33.92067895s) [noise=true , address=104.36.82.22:4534 , piece_id=CW4AQT3XUGA4A5OC6NUY4NYOOAMLN5FFJJN5PQBYEON7HPD5LCYQ, seg_pos={0 0}, upload=5911]: Done uploading piece. err: <nil> Jul 27 04:07:18.423748140 (6h15m34.204681511s) [noise=true , address=64.49.13.140:4508 , piece_id=MZ52U75VOIK6Q7QFXMMYRRG7JXFWDFU2JWQFYPERUCMH3BUX6XPQ, seg_pos={0 0}, upload=5911]: Done uploading piece. err: <nil> Jul 27 04:08:16.739940381 (6h16m32.52087375s) [noise=true , address=104.36.82.54:4172 , piece_id=ZNA53MWDRWPUH6FUR4DJERBDT2IOYGA3IYUADKXXIOLQJLQOXE7Q, seg_pos={0 0}, upload=5911]: Done uploading piece. err: <nil> Jul 27 04:08:17.177295397 (6h16m32.958228766s) [noise=true , address=104.36.82.8:4002 , piece_id=6XSPWQC3US355PS6IQ4K3IYNQZH5MAT4IVIPGSYRUM3XYPSCWE6Q, seg_pos={0 0}, upload=5911]: Done uploading piece. err: <nil> Jul 27 04:08:17.534296584 (6h16m33.315229953s) [noise=true , address=104.36.82.36:4650 , piece_id=BKJULWGPESBINNTSUYP6E4KZRXKPACVU5I7DOSZBO4YMK2V26HEQ, seg_pos={0 0}, upload=5911]: Done uploading piece. err: <nil> Jul 27 04:08:17.677851566 (6h16m33.458784943s) [noise=true , address=104.36.82.14:4248 , piece_id=LDSGCUM7Q2VX5CDFI4JRGEEG7D5H5Q5Z5WUWKU277VN2JFRX6UGA, seg_pos={0 0}, upload=5911]: Done uploading piece. err: ecclient: upload failed (node:1u6ir2PLqDa9xGm64cuqq1RqY4qeBa1fqCo9VghspBf54XEXqn, address:104.36.82.14:4248): protocol: rpc: dial tcp 104.36.82.14:4248: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 04:08:17.678115170 (6h16m33.459048549s) [noise=true , address=104.36.82.46:4496 , piece_id=GWNQR5WRGRRQBLSGWJFD3O3UNCMNE62JQNPIV5SYKBUIL3BQKA5A, seg_pos={0 0}, upload=5911]: Done uploading piece. err: ecclient: upload failed (node:12V8XhSWkyGUVYFR8MykifFAP1YAfGCwzqucDcMMfzw3xsVW77Z, address:104.36.82.46:4496): protocol: read tcp 159.223.143.39:52800->104.36.82.46:4496: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 04:08:17.678144409 (6h16m33.459077789s) [noise=true , address=104.36.82.44:4901 , piece_id=N4YLCLR2QZSULUL2XWONEKBFUMNOU7KUX2QEOK4SSDFSCI7IL6LA, seg_pos={0 0}, upload=5911]: Done uploading piece. err: ecclient: upload failed (node:1FHE84zvBb1MKwB1Ht7puTafgQxDEvPt4mYqkYVSSufAMpWQA1, address:104.36.82.44:4901): protocol: rpc: dial tcp 104.36.82.44:4901: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 04:08:17.678229622 (6h16m33.459162988s) [noise=true , address=104.36.82.44:4893 , piece_id=HUJICRJ6W54O3AZZR6H3KG6E4TGBTNS46O6XVMQM3PFAZE7A2HZA, seg_pos={0 0}, upload=5911]: Done uploading piece. err: ecclient: upload failed (node:17VMFZ2T8jrSJVtchciJ9SnyrR6JstvcMoXrN5Q3zTD1aQU1oJ, address:104.36.82.44:4893): protocol: write tcp 159.223.143.39:35302->104.36.82.44:4893: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 04:08:17.678310218 (6h16m33.459243583s) [noise=true , address=104.36.82.42:4829 , piece_id=ZLWJFP3LGX3YLXBJYIR66DXTYEVDBKXKVXCQSF2PCYCYJBIEJGMA, seg_pos={0 0}, upload=5911]: Done uploading piece. err: ecclient: upload failed (node:1F5T8Vt1x4nXFaSenvGBCYjxhRL1H1sxRvzLjW3CqzwDVhq39F, address:104.36.82.42:4829): protocol: rpc: dial tcp 104.36.82.42:4829: i/o timeout; conn closed: op; piecestore: piecestore close: context deadline exceeded Jul 27 04:08:17.678412989 (6h16m33.459346365s) [noise=true , address=104.36.82.38:4722 , piece_id=B2MBQY5CTMEI6WSE7FTLOV5RUJ22SCNG7N2EEJILEATGJ4TIALXA, seg_pos={0 0}, upload=5911]: Done uploading piece. err: ecclient: upload failed (node:12Zw3dtUg9gARz52KhwDLTjcqC3CNm7dpPzpdx7P7PbL1jwe8uc, address:104.36.82.38:4722): protocol: read tcp 159.223.143.39:56048->104.36.82.38:4722: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 04:08:17.678455247 (6h16m33.459388624s) [noise=true , address=104.36.82.47:4524 , piece_id=DONJFTKUQ2XCW6BOU7AVOKJ7XMDMBPGQVUPRANH7X2YS7PJXUKIA, seg_pos={0 0}, upload=5911]: Done uploading piece. err: ecclient: upload failed (node:1xfbCzjSwFDRrR1gPggeN8tNCLbRRWDVzCJ6mqhZsbVSe8d4s4, address:104.36.82.47:4524): protocol: read tcp 159.223.143.39:46694->104.36.82.47:4524: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 04:08:17.678510668 (6h16m33.459444052s) [seg_pos={0 0}, upload=5911]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Jul 27 04:08:17.706442864 (6h16m33.487376232s) [upload=5911]: Done uploading segments... Jul 27 04:08:17.706495329 (6h16m33.487428705s) [upload=5911]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 4.276144549s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................X................................] duration: 427.679443ms FAIL (2025-07-27 04:28:12 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== ece_id=NYOFDEZ7CV4PHGQ6JZMEEVAYGJRPYG6IBW7MYJO42HMWOJM7PHZA, seg_pos={0 0}, upload=5879]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12crcXpNe9NgeXTsUpooLSBgrSUFbRxbSbanLa39w2Hr5mGFww): protocol: read tcp 159.223.143.39:53630->104.36.82.5:4840: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 04:05:32.508904485 (6h13m48.289837866s) [seg_pos={0 0}, upload=5879]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 27 04:05:32.509028070 (6h13m48.289961457s) [upload=5879]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 27 04:05:32.509034645 (6h13m48.289968129s) [upload=5879]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 27 04:05:32.581834839 (6h13m48.362768225s) [upload=5879]: Done uploading segments... Jul 27 04:05:32.581875919 (6h13m48.362809298s) [upload=5879]: Upload finished. err: <nil> Jul 27 04:06:23.829155331 (6h14m39.610088696s) [upload=5895]: Starting upload Jul 27 04:06:23.829225021 (6h14m39.610158398s) [upload=5895]: Done starting upload Jul 27 04:06:23.829294606 (6h14m39.61022799s) [upload=5895]: Uploading segments... Jul 27 04:06:23.829577656 (6h14m39.610511053s) [upload=5895]: Got next segment. Inline: false Jul 27 04:06:23.829649664 (6h14m39.610583027s) [upload=5895]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 27 04:06:23.829657065 (6h14m39.61059043s) [upload=5895]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 27 04:06:23.861405530 (6h14m39.642338913s) [seg_pos={0 0}, upload=5895]: Begin upload segment... Jul 27 04:06:23.861476351 (6h14m39.642409713s) [seg_pos={0 0}, upload=5895]: Done begin upload segment. Jul 27 04:06:23.861509120 (6h14m39.642442506s) [noise=true , address=77.90.31.19:4843 , piece_id=YML5RDQIK432Z4FSE4DKXIV5U47LQBEZRFMAVLANUSGFUQR23UWQ, seg_pos={0 0}, upload=5895]: Uploading piece... Jul 27 04:06:23.861953723 (6h14m39.6428871s) [noise=true , address=45.91.116.10:4081 , piece_id=XGDTSEZOUNRTEVWHTO3DHTZMH2VZZOPJPEKLCFAHTXESLLYABHSA, seg_pos={0 0}, upload=5895]: Uploading piece... Jul 27 04:06:23.862060983 (6h14m39.642994358s) [noise=true , address=45.91.116.9:4262 , piece_id=M5ETD45HH2XZD6HDLTJ3JO2BOOIROXFWKVPTN42PSQEWG4F7KAAQ, seg_pos={0 0}, upload=5895]: Uploading piece... Jul 27 04:06:23.862759464 (6h14m39.64369284s) [noise=true , address=104.36.82.4:4804 , piece_id=IU6FDNVB5AUPA7MJ6RSXUTLBETPHOFGNJ7YGWBFIZSQFSO6W6BBA, seg_pos={0 0}, upload=5895]: Uploading piece... Jul 27 04:06:23.862835086 (6h14m39.643768461s) [noise=true , address=104.36.82.6:4873 , piece_id=YP4VJORZZUL5EVCH3P7LD3LNVMHB7CWRTKHGIA7UROXE44N6MMZA, seg_pos={0 0}, upload=5895]: Uploading piece... Jul 27 04:06:23.863593880 (6h14m39.644527258s) [noise=true , address=77.90.31.15:4697 , piece_id=Q76TJDSZPSYZSJQNFVOI462U6DFILMH2RV3LQISODXSOCXGRKW6Q, seg_pos={0 0}, upload=5895]: Uploading piece... Jul 27 04:06:23.863656891 (6h14m39.644590266s) [noise=true , address=45.91.116.7:4011 , piece_id=XMH6MCCHZBBRQWHJFWLRGYHLQD4KUGSGL7MMZSKA5TZJCND3DUDA, seg_pos={0 0}, upload=5895]: Uploading piece... Jul 27 04:06:23.864408612 (6h14m39.645341987s) [noise=true , address=45.91.116.6:4973 , piece_id=J3VULPU24CIM64VEVE3UHO62HPN6TJXGZU6BCJIS5JQJKHJK2LIQ, seg_pos={0 0}, upload=5895]: Uploading piece... Jul 27 04:06:23.864521549 (6h14m39.64545491s) [noise=true , address=104.36.82.10:4079 , piece_id=2SH5D6JOMLFR7VSN7ZY5CLMR5RHIRFYQ7Z5I2JIYTJQM43G4O6OQ, seg_pos={0 0}, upload=5895]: Uploading piece... Jul 27 04:06:23.865518491 (6h14m39.646451872s) [noise=true , address=77.90.31.17:4771 , piece_id=U2LF22GKCMKRZN6EWCF26ITDZQHBFZXCGSQXT33MCX6P57IGI35Q, seg_pos={0 0}, upload=5895]: Uploading piece... Jul 27 04:06:23.865654950 (6h14m39.646588325s) [noise=true , address=45.91.116.8:4048 , piece_id=5REWMVVTBVLDQXQ5KAEHLY4VJL5SGF7ONQWDQQ3N2FDU3OAO6AZA, seg_pos={0 0}, upload=5895]: Uploading piece... Jul 27 04:06:23.866190892 (6h14m39.647124271s) [noise=true , address=104.36.82.12:4151 , piece_id=E2TEUTQBD4Y4AUHSCPMCIFEXBSY7S5HJWBKNKWWTUWEW2BLAAE6Q, seg_pos={0 0}, upload=5895]: Uploading piece... Jul 27 04:06:23.866412590 (6h14m39.647345968s) [noise=true , address=104.36.82.11:4115 , piece_id=YXMNVT6ZHGESGQ2BXKZUVR7U2CNNAI2PO7ZTMDG5UIOV7HTDDUTA, seg_pos={0 0}, upload=5895]: Uploading piece... Jul 27 04:06:23.866715972 (6h14m39.647649359s) [noise=true , address=45.91.116.13:4153 , piece_id=H4535KOCG3F4RKA6AKKMTJSOM6Q6PGRC6M7FR3UIS3BORTSP5UQA, seg_pos={0 0}, upload=5895]: Uploading piece... Jul 27 04:06:23.866998965 (6h14m39.64793234s) [noise=true , address=104.36.82.13:4186 , piece_id=TBLDKALGJYBGR5NLCOCKJ4QKEFNIJXCKAFUBD7VR3UMNWEMKMKHQ, seg_pos={0 0}, upload=5895]: Uploading piece... Jul 27 04:06:23.867230593 (6h14m39.648163974s) [noise=true , address=104.36.82.16:4293 , piece_id=AHH6PRFUSGTDGB7H52DRITARA4IF6GFLVSRYS2Y6EPJCZSKUOYWQ, seg_pos={0 0}, upload=5895]: Uploading piece... Jul 27 04:06:23.949422808 (6h14m39.730356187s) [upload=5895]: Next returned nil segment Jul 27 04:06:23.949442576 (6h14m39.73037596s) [upload=5895]: Waiting for error group managing segments... Jul 27 04:06:23.953497426 (6h14m39.734430802s) [noise=true , address=45.91.116.10:4081 , piece_id=XGDTSEZOUNRTEVWHTO3DHTZMH2VZZOPJPEKLCFAHTXESLLYABHSA, seg_pos={0 0}, upload=5895]: Done uploading piece. err: <nil> Jul 27 04:06:23.953531796 (6h14m39.734465159s) [noise=true , address=45.91.116.6:4973 , piece_id=J3VULPU24CIM64VEVE3UHO62HPN6TJXGZU6BCJIS5JQJKHJK2LIQ, seg_pos={0 0}, upload=5895]: Done uploading piece. err: <nil> Jul 27 04:06:23.954068660 (6h14m39.735002029s) [noise=true , address=45.91.116.9:4262 , piece_id=M5ETD45HH2XZD6HDLTJ3JO2BOOIROXFWKVPTN42PSQEWG4F7KAAQ, seg_pos={0 0}, upload=5895]: Done uploading piece. err: <nil> Jul 27 04:06:23.954194339 (6h14m39.735127707s) [noise=true , address=45.91.116.8:4048 , piece_id=5REWMVVTBVLDQXQ5KAEHLY4VJL5SGF7ONQWDQQ3N2FDU3OAO6AZA, seg_pos={0 0}, upload=5895]: Done uploading piece. err: <nil> Jul 27 04:06:23.954601730 (6h14m39.735535112s) [noise=true , address=45.91.116.7:4011 , piece_id=XMH6MCCHZBBRQWHJFWLRGYHLQD4KUGSGL7MMZSKA5TZJCND3DUDA, seg_pos={0 0}, upload=5895]: Done uploading piece. err: <nil> Jul 27 04:06:23.954755872 (6h14m39.735689237s) [noise=true , address=45.91.116.13:4153 , piece_id=H4535KOCG3F4RKA6AKKMTJSOM6Q6PGRC6M7FR3UIS3BORTSP5UQA, seg_pos={0 0}, upload=5895]: Done uploading piece. err: <nil> Jul 27 04:06:23.955611883 (6h14m39.73654526s) [noise=true , address=77.90.31.19:4843 , piece_id=YML5RDQIK432Z4FSE4DKXIV5U47LQBEZRFMAVLANUSGFUQR23UWQ, seg_pos={0 0}, upload=5895]: Done uploading piece. err: <nil> Jul 27 04:06:23.956373391 (6h14m39.737306756s) [noise=true , address=77.90.31.15:4697 , piece_id=Q76TJDSZPSYZSJQNFVOI462U6DFILMH2RV3LQISODXSOCXGRKW6Q, seg_pos={0 0}, upload=5895]: Done uploading piece. err: <nil> Jul 27 04:06:23.958659122 (6h14m39.73959249s) [noise=true , address=77.90.31.17:4771 , piece_id=U2LF22GKCMKRZN6EWCF26ITDZQHBFZXCGSQXT33MCX6P57IGI35Q, seg_pos={0 0}, upload=5895]: Done uploading piece. err: <nil> Jul 27 04:06:35.058944013 (6h14m50.83987739s) [noise=true , address=104.36.82.16:4293 , piece_id=AHH6PRFUSGTDGB7H52DRITARA4IF6GFLVSRYS2Y6EPJCZSKUOYWQ, seg_pos={0 0}, upload=5895]: Done uploading piece. err: <nil> Jul 27 04:06:41.103204002 (6h14m56.884137386s) [noise=true , address=104.36.82.4:4804 , piece_id=IU6FDNVB5AUPA7MJ6RSXUTLBETPHOFGNJ7YGWBFIZSQFSO6W6BBA, seg_pos={0 0}, upload=5895]: Done uploading piece. err: <nil> Jul 27 04:06:43.191702285 (6h14m58.972635651s) [noise=true , address=104.36.82.6:4873 , piece_id=YP4VJORZZUL5EVCH3P7LD3LNVMHB7CWRTKHGIA7UROXE44N6MMZA, seg_pos={0 0}, upload=5895]: Done uploading piece. err: <nil> Jul 27 04:06:53.777973493 (6h15m9.558906861s) [noise=true , address=104.36.82.11:4115 , piece_id=YXMNVT6ZHGESGQ2BXKZUVR7U2CNNAI2PO7ZTMDG5UIOV7HTDDUTA, seg_pos={0 0}, upload=5895]: Done uploading piece. err: ecclient: upload failed (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR, address:104.36.82.11:4115): protocol: write tcp 159.223.143.39:34210->104.36.82.11:4115: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 04:06:53.778019532 (6h15m9.558952909s) [noise=true , address=104.36.82.12:4151 , piece_id=E2TEUTQBD4Y4AUHSCPMCIFEXBSY7S5HJWBKNKWWTUWEW2BLAAE6Q, seg_pos={0 0}, upload=5895]: Done uploading piece. err: ecclient: upload failed (node:1w6pwZMdeQKjt92qx6WbTWU5VPYJto5gS87f6iKvCBZH6LfyeB, address:104.36.82.12:4151): protocol: rpc: dial tcp 104.36.82.12:4151: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 04:06:53.778043653 (6h15m9.558977021s) [noise=true , address=104.36.82.13:4186 , piece_id=TBLDKALGJYBGR5NLCOCKJ4QKEFNIJXCKAFUBD7VR3UMNWEMKMKHQ, seg_pos={0 0}, upload=5895]: Done uploading piece. err: ecclient: upload failed (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht, address:104.36.82.13:4186): protocol: read tcp 159.223.143.39:34340->104.36.82.13:4186: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 04:06:53.778095228 (6h15m9.559028593s) [noise=true , address=104.36.82.10:4079 , piece_id=2SH5D6JOMLFR7VSN7ZY5CLMR5RHIRFYQ7Z5I2JIYTJQM43G4O6OQ, seg_pos={0 0}, upload=5895]: Done uploading piece. err: ecclient: upload failed (node:12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud, address:104.36.82.10:4079): protocol: write tcp 159.223.143.39:53840->104.36.82.10:4079: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 04:06:53.778142167 (6h15m9.559075532s) [seg_pos={0 0}, upload=5895]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 27 04:06:53.837032161 (6h15m9.617965543s) [upload=5895]: Done uploading segments... Jul 27 04:06:53.837055996 (6h15m9.617989366s) [upload=5895]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 427.710323ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [................................X...........................] duration: 527.152574ms FAIL (2025-07-27 04:27:47 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== {0 0}, upload=5911]: Begin upload segment... Jul 27 04:07:17.745307658 (6h15m33.526241043s) [seg_pos={0 0}, upload=5911]: Done begin upload segment. Jul 27 04:07:17.745349241 (6h15m33.526282625s) [noise=true , address=104.36.82.52:4098 , piece_id=M2ETF25QQ53RRGFPUKXJRINIWV7AWXSECEDR2AIM645IED72NURQ, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.745439694 (6h15m33.526373073s) [noise=true , address=104.36.82.56:4244 , piece_id=LQCZ5W54NBHWAVBVYK7HYSSJDC73E6FV4CP4YIM4L5H2N3OLOZXA, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.745972271 (6h15m33.526905654s) [noise=true , address=104.36.82.36:4650 , piece_id=BKJULWGPESBINNTSUYP6E4KZRXKPACVU5I7DOSZBO4YMK2V26HEQ, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.745980243 (6h15m33.526913621s) [noise=true , address=104.36.82.38:4722 , piece_id=B2MBQY5CTMEI6WSE7FTLOV5RUJ22SCNG7N2EEJILEATGJ4TIALXA, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.746403818 (6h15m33.527337198s) [noise=true , address=104.36.82.22:4534 , piece_id=CW4AQT3XUGA4A5OC6NUY4NYOOAMLN5FFJJN5PQBYEON7HPD5LCYQ, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.746455493 (6h15m33.527388875s) [noise=true , address=104.36.82.14:4248 , piece_id=LDSGCUM7Q2VX5CDFI4JRGEEG7D5H5Q5Z5WUWKU277VN2JFRX6UGA, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.746872127 (6h15m33.527805505s) [noise=true , address=104.36.82.40:4954 , piece_id=YLOHTJLRAED66OMEQYPZHNNRI77V6WOR4EHTEZOOEHMGZHJ2O3QQ, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.747123162 (6h15m33.52805654s) [noise=true , address=104.36.82.46:4496 , piece_id=GWNQR5WRGRRQBLSGWJFD3O3UNCMNE62JQNPIV5SYKBUIL3BQKA5A, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.747280678 (6h15m33.528214053s) [noise=true , address=104.36.82.47:4524 , piece_id=DONJFTKUQ2XCW6BOU7AVOKJ7XMDMBPGQVUPRANH7X2YS7PJXUKIA, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.747736400 (6h15m33.528669779s) [noise=true , address=104.36.82.44:4893 , piece_id=HUJICRJ6W54O3AZZR6H3KG6E4TGBTNS46O6XVMQM3PFAZE7A2HZA, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.747755734 (6h15m33.528689109s) [noise=true , address=104.36.82.55:4204 , piece_id=72WISGRHMOAZDV5EDGTBOMLBL4HDOF3CBEMQAAIRGILPC7J5JKAQ, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.748313124 (6h15m33.5292465s) [noise=true , address=104.36.82.58:4287 , piece_id=BVDYUEWU7TVEQDMZXX5SJNS4QICVKMTGFLCUV6P2YHOE7TI25VXA, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.748485217 (6h15m33.529418594s) [noise=true , address=104.36.82.53:4123 , piece_id=NXJRZ5BQSLXSOIOSUJ4MM4AGKHP4UIDUGEJKLUUX7YOGQJA6WWZA, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.748831329 (6h15m33.529764704s) [noise=true , address=104.36.82.42:4829 , piece_id=ZLWJFP3LGX3YLXBJYIR66DXTYEVDBKXKVXCQSF2PCYCYJBIEJGMA, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.749299293 (6h15m33.530232669s) [noise=true , address=104.36.82.8:4002 , piece_id=6XSPWQC3US355PS6IQ4K3IYNQZH5MAT4IVIPGSYRUM3XYPSCWE6Q, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.749790974 (6h15m33.53072436s) [noise=true , address=104.36.82.44:4901 , piece_id=N4YLCLR2QZSULUL2XWONEKBFUMNOU7KUX2QEOK4SSDFSCI7IL6LA, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.750478221 (6h15m33.531411597s) [noise=true , address=104.36.82.54:4172 , piece_id=ZNA53MWDRWPUH6FUR4DJERBDT2IOYGA3IYUADKXXIOLQJLQOXE7Q, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.752884255 (6h15m33.533817625s) [noise=true , address=64.49.13.140:4508 , piece_id=MZ52U75VOIK6Q7QFXMMYRRG7JXFWDFU2JWQFYPERUCMH3BUX6XPQ, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.879839555 (6h15m33.660772917s) [upload=5911]: Next returned nil segment Jul 27 04:07:17.879861389 (6h15m33.660794765s) [upload=5911]: Waiting for error group managing segments... Jul 27 04:07:17.888652438 (6h15m33.669585811s) [noise=true , address=104.36.82.58:4287 , piece_id=BVDYUEWU7TVEQDMZXX5SJNS4QICVKMTGFLCUV6P2YHOE7TI25VXA, seg_pos={0 0}, upload=5911]: Done uploading piece. err: <nil> Jul 27 04:07:17.952192045 (6h15m33.733125429s) [noise=true , address=104.36.82.53:4123 , piece_id=NXJRZ5BQSLXSOIOSUJ4MM4AGKHP4UIDUGEJKLUUX7YOGQJA6WWZA, seg_pos={0 0}, upload=5911]: Done uploading piece. err: <nil> Jul 27 04:07:17.981907539 (6h15m33.762840911s) [noise=true , address=104.36.82.56:4244 , piece_id=LQCZ5W54NBHWAVBVYK7HYSSJDC73E6FV4CP4YIM4L5H2N3OLOZXA, seg_pos={0 0}, upload=5911]: Done uploading piece. err: <nil> Jul 27 04:07:17.982370496 (6h15m33.7633039s) [noise=true , address=104.36.82.55:4204 , piece_id=72WISGRHMOAZDV5EDGTBOMLBL4HDOF3CBEMQAAIRGILPC7J5JKAQ, seg_pos={0 0}, upload=5911]: Done uploading piece. err: <nil> Jul 27 04:07:18.009548506 (6h15m33.790481886s) [noise=true , address=104.36.82.52:4098 , piece_id=M2ETF25QQ53RRGFPUKXJRINIWV7AWXSECEDR2AIM645IED72NURQ, seg_pos={0 0}, upload=5911]: Done uploading piece. err: <nil> Jul 27 04:07:18.023770598 (6h15m33.804703983s) [noise=true , address=104.36.82.40:4954 , piece_id=YLOHTJLRAED66OMEQYPZHNNRI77V6WOR4EHTEZOOEHMGZHJ2O3QQ, seg_pos={0 0}, upload=5911]: Done uploading piece. err: <nil> Jul 27 04:07:18.139745581 (6h15m33.92067895s) [noise=true , address=104.36.82.22:4534 , piece_id=CW4AQT3XUGA4A5OC6NUY4NYOOAMLN5FFJJN5PQBYEON7HPD5LCYQ, seg_pos={0 0}, upload=5911]: Done uploading piece. err: <nil> Jul 27 04:07:18.423748140 (6h15m34.204681511s) [noise=true , address=64.49.13.140:4508 , piece_id=MZ52U75VOIK6Q7QFXMMYRRG7JXFWDFU2JWQFYPERUCMH3BUX6XPQ, seg_pos={0 0}, upload=5911]: Done uploading piece. err: <nil> Jul 27 04:08:16.739940381 (6h16m32.52087375s) [noise=true , address=104.36.82.54:4172 , piece_id=ZNA53MWDRWPUH6FUR4DJERBDT2IOYGA3IYUADKXXIOLQJLQOXE7Q, seg_pos={0 0}, upload=5911]: Done uploading piece. err: <nil> Jul 27 04:08:17.177295397 (6h16m32.958228766s) [noise=true , address=104.36.82.8:4002 , piece_id=6XSPWQC3US355PS6IQ4K3IYNQZH5MAT4IVIPGSYRUM3XYPSCWE6Q, seg_pos={0 0}, upload=5911]: Done uploading piece. err: <nil> Jul 27 04:08:17.534296584 (6h16m33.315229953s) [noise=true , address=104.36.82.36:4650 , piece_id=BKJULWGPESBINNTSUYP6E4KZRXKPACVU5I7DOSZBO4YMK2V26HEQ, seg_pos={0 0}, upload=5911]: Done uploading piece. err: <nil> Jul 27 04:08:17.677851566 (6h16m33.458784943s) [noise=true , address=104.36.82.14:4248 , piece_id=LDSGCUM7Q2VX5CDFI4JRGEEG7D5H5Q5Z5WUWKU277VN2JFRX6UGA, seg_pos={0 0}, upload=5911]: Done uploading piece. err: ecclient: upload failed (node:1u6ir2PLqDa9xGm64cuqq1RqY4qeBa1fqCo9VghspBf54XEXqn, address:104.36.82.14:4248): protocol: rpc: dial tcp 104.36.82.14:4248: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 04:08:17.678115170 (6h16m33.459048549s) [noise=true , address=104.36.82.46:4496 , piece_id=GWNQR5WRGRRQBLSGWJFD3O3UNCMNE62JQNPIV5SYKBUIL3BQKA5A, seg_pos={0 0}, upload=5911]: Done uploading piece. err: ecclient: upload failed (node:12V8XhSWkyGUVYFR8MykifFAP1YAfGCwzqucDcMMfzw3xsVW77Z, address:104.36.82.46:4496): protocol: read tcp 159.223.143.39:52800->104.36.82.46:4496: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 04:08:17.678144409 (6h16m33.459077789s) [noise=true , address=104.36.82.44:4901 , piece_id=N4YLCLR2QZSULUL2XWONEKBFUMNOU7KUX2QEOK4SSDFSCI7IL6LA, seg_pos={0 0}, upload=5911]: Done uploading piece. err: ecclient: upload failed (node:1FHE84zvBb1MKwB1Ht7puTafgQxDEvPt4mYqkYVSSufAMpWQA1, address:104.36.82.44:4901): protocol: rpc: dial tcp 104.36.82.44:4901: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 04:08:17.678229622 (6h16m33.459162988s) [noise=true , address=104.36.82.44:4893 , piece_id=HUJICRJ6W54O3AZZR6H3KG6E4TGBTNS46O6XVMQM3PFAZE7A2HZA, seg_pos={0 0}, upload=5911]: Done uploading piece. err: ecclient: upload failed (node:17VMFZ2T8jrSJVtchciJ9SnyrR6JstvcMoXrN5Q3zTD1aQU1oJ, address:104.36.82.44:4893): protocol: write tcp 159.223.143.39:35302->104.36.82.44:4893: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 04:08:17.678310218 (6h16m33.459243583s) [noise=true , address=104.36.82.42:4829 , piece_id=ZLWJFP3LGX3YLXBJYIR66DXTYEVDBKXKVXCQSF2PCYCYJBIEJGMA, seg_pos={0 0}, upload=5911]: Done uploading piece. err: ecclient: upload failed (node:1F5T8Vt1x4nXFaSenvGBCYjxhRL1H1sxRvzLjW3CqzwDVhq39F, address:104.36.82.42:4829): protocol: rpc: dial tcp 104.36.82.42:4829: i/o timeout; conn closed: op; piecestore: piecestore close: context deadline exceeded Jul 27 04:08:17.678412989 (6h16m33.459346365s) [noise=true , address=104.36.82.38:4722 , piece_id=B2MBQY5CTMEI6WSE7FTLOV5RUJ22SCNG7N2EEJILEATGJ4TIALXA, seg_pos={0 0}, upload=5911]: Done uploading piece. err: ecclient: upload failed (node:12Zw3dtUg9gARz52KhwDLTjcqC3CNm7dpPzpdx7P7PbL1jwe8uc, address:104.36.82.38:4722): protocol: read tcp 159.223.143.39:56048->104.36.82.38:4722: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 04:08:17.678455247 (6h16m33.459388624s) [noise=true , address=104.36.82.47:4524 , piece_id=DONJFTKUQ2XCW6BOU7AVOKJ7XMDMBPGQVUPRANH7X2YS7PJXUKIA, seg_pos={0 0}, upload=5911]: Done uploading piece. err: ecclient: upload failed (node:1xfbCzjSwFDRrR1gPggeN8tNCLbRRWDVzCJ6mqhZsbVSe8d4s4, address:104.36.82.47:4524): protocol: read tcp 159.223.143.39:46694->104.36.82.47:4524: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 04:08:17.678510668 (6h16m33.459444052s) [seg_pos={0 0}, upload=5911]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Jul 27 04:08:17.706442864 (6h16m33.487376232s) [upload=5911]: Done uploading segments... Jul 27 04:08:17.706495329 (6h16m33.487428705s) [upload=5911]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 527.180069ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.................................X..........................] duration: 17.829261838s FAIL (2025-07-27 04:27:46 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== {0 0}, upload=5911]: Begin upload segment... Jul 27 04:07:17.745307658 (6h15m33.526241043s) [seg_pos={0 0}, upload=5911]: Done begin upload segment. Jul 27 04:07:17.745349241 (6h15m33.526282625s) [noise=true , address=104.36.82.52:4098 , piece_id=M2ETF25QQ53RRGFPUKXJRINIWV7AWXSECEDR2AIM645IED72NURQ, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.745439694 (6h15m33.526373073s) [noise=true , address=104.36.82.56:4244 , piece_id=LQCZ5W54NBHWAVBVYK7HYSSJDC73E6FV4CP4YIM4L5H2N3OLOZXA, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.745972271 (6h15m33.526905654s) [noise=true , address=104.36.82.36:4650 , piece_id=BKJULWGPESBINNTSUYP6E4KZRXKPACVU5I7DOSZBO4YMK2V26HEQ, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.745980243 (6h15m33.526913621s) [noise=true , address=104.36.82.38:4722 , piece_id=B2MBQY5CTMEI6WSE7FTLOV5RUJ22SCNG7N2EEJILEATGJ4TIALXA, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.746403818 (6h15m33.527337198s) [noise=true , address=104.36.82.22:4534 , piece_id=CW4AQT3XUGA4A5OC6NUY4NYOOAMLN5FFJJN5PQBYEON7HPD5LCYQ, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.746455493 (6h15m33.527388875s) [noise=true , address=104.36.82.14:4248 , piece_id=LDSGCUM7Q2VX5CDFI4JRGEEG7D5H5Q5Z5WUWKU277VN2JFRX6UGA, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.746872127 (6h15m33.527805505s) [noise=true , address=104.36.82.40:4954 , piece_id=YLOHTJLRAED66OMEQYPZHNNRI77V6WOR4EHTEZOOEHMGZHJ2O3QQ, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.747123162 (6h15m33.52805654s) [noise=true , address=104.36.82.46:4496 , piece_id=GWNQR5WRGRRQBLSGWJFD3O3UNCMNE62JQNPIV5SYKBUIL3BQKA5A, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.747280678 (6h15m33.528214053s) [noise=true , address=104.36.82.47:4524 , piece_id=DONJFTKUQ2XCW6BOU7AVOKJ7XMDMBPGQVUPRANH7X2YS7PJXUKIA, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.747736400 (6h15m33.528669779s) [noise=true , address=104.36.82.44:4893 , piece_id=HUJICRJ6W54O3AZZR6H3KG6E4TGBTNS46O6XVMQM3PFAZE7A2HZA, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.747755734 (6h15m33.528689109s) [noise=true , address=104.36.82.55:4204 , piece_id=72WISGRHMOAZDV5EDGTBOMLBL4HDOF3CBEMQAAIRGILPC7J5JKAQ, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.748313124 (6h15m33.5292465s) [noise=true , address=104.36.82.58:4287 , piece_id=BVDYUEWU7TVEQDMZXX5SJNS4QICVKMTGFLCUV6P2YHOE7TI25VXA, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.748485217 (6h15m33.529418594s) [noise=true , address=104.36.82.53:4123 , piece_id=NXJRZ5BQSLXSOIOSUJ4MM4AGKHP4UIDUGEJKLUUX7YOGQJA6WWZA, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.748831329 (6h15m33.529764704s) [noise=true , address=104.36.82.42:4829 , piece_id=ZLWJFP3LGX3YLXBJYIR66DXTYEVDBKXKVXCQSF2PCYCYJBIEJGMA, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.749299293 (6h15m33.530232669s) [noise=true , address=104.36.82.8:4002 , piece_id=6XSPWQC3US355PS6IQ4K3IYNQZH5MAT4IVIPGSYRUM3XYPSCWE6Q, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.749790974 (6h15m33.53072436s) [noise=true , address=104.36.82.44:4901 , piece_id=N4YLCLR2QZSULUL2XWONEKBFUMNOU7KUX2QEOK4SSDFSCI7IL6LA, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.750478221 (6h15m33.531411597s) [noise=true , address=104.36.82.54:4172 , piece_id=ZNA53MWDRWPUH6FUR4DJERBDT2IOYGA3IYUADKXXIOLQJLQOXE7Q, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.752884255 (6h15m33.533817625s) [noise=true , address=64.49.13.140:4508 , piece_id=MZ52U75VOIK6Q7QFXMMYRRG7JXFWDFU2JWQFYPERUCMH3BUX6XPQ, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.879839555 (6h15m33.660772917s) [upload=5911]: Next returned nil segment Jul 27 04:07:17.879861389 (6h15m33.660794765s) [upload=5911]: Waiting for error group managing segments... Jul 27 04:07:17.888652438 (6h15m33.669585811s) [noise=true , address=104.36.82.58:4287 , piece_id=BVDYUEWU7TVEQDMZXX5SJNS4QICVKMTGFLCUV6P2YHOE7TI25VXA, seg_pos={0 0}, upload=5911]: Done uploading piece. err: <nil> Jul 27 04:07:17.952192045 (6h15m33.733125429s) [noise=true , address=104.36.82.53:4123 , piece_id=NXJRZ5BQSLXSOIOSUJ4MM4AGKHP4UIDUGEJKLUUX7YOGQJA6WWZA, seg_pos={0 0}, upload=5911]: Done uploading piece. err: <nil> Jul 27 04:07:17.981907539 (6h15m33.762840911s) [noise=true , address=104.36.82.56:4244 , piece_id=LQCZ5W54NBHWAVBVYK7HYSSJDC73E6FV4CP4YIM4L5H2N3OLOZXA, seg_pos={0 0}, upload=5911]: Done uploading piece. err: <nil> Jul 27 04:07:17.982370496 (6h15m33.7633039s) [noise=true , address=104.36.82.55:4204 , piece_id=72WISGRHMOAZDV5EDGTBOMLBL4HDOF3CBEMQAAIRGILPC7J5JKAQ, seg_pos={0 0}, upload=5911]: Done uploading piece. err: <nil> Jul 27 04:07:18.009548506 (6h15m33.790481886s) [noise=true , address=104.36.82.52:4098 , piece_id=M2ETF25QQ53RRGFPUKXJRINIWV7AWXSECEDR2AIM645IED72NURQ, seg_pos={0 0}, upload=5911]: Done uploading piece. err: <nil> Jul 27 04:07:18.023770598 (6h15m33.804703983s) [noise=true , address=104.36.82.40:4954 , piece_id=YLOHTJLRAED66OMEQYPZHNNRI77V6WOR4EHTEZOOEHMGZHJ2O3QQ, seg_pos={0 0}, upload=5911]: Done uploading piece. err: <nil> Jul 27 04:07:18.139745581 (6h15m33.92067895s) [noise=true , address=104.36.82.22:4534 , piece_id=CW4AQT3XUGA4A5OC6NUY4NYOOAMLN5FFJJN5PQBYEON7HPD5LCYQ, seg_pos={0 0}, upload=5911]: Done uploading piece. err: <nil> Jul 27 04:07:18.423748140 (6h15m34.204681511s) [noise=true , address=64.49.13.140:4508 , piece_id=MZ52U75VOIK6Q7QFXMMYRRG7JXFWDFU2JWQFYPERUCMH3BUX6XPQ, seg_pos={0 0}, upload=5911]: Done uploading piece. err: <nil> Jul 27 04:08:16.739940381 (6h16m32.52087375s) [noise=true , address=104.36.82.54:4172 , piece_id=ZNA53MWDRWPUH6FUR4DJERBDT2IOYGA3IYUADKXXIOLQJLQOXE7Q, seg_pos={0 0}, upload=5911]: Done uploading piece. err: <nil> Jul 27 04:08:17.177295397 (6h16m32.958228766s) [noise=true , address=104.36.82.8:4002 , piece_id=6XSPWQC3US355PS6IQ4K3IYNQZH5MAT4IVIPGSYRUM3XYPSCWE6Q, seg_pos={0 0}, upload=5911]: Done uploading piece. err: <nil> Jul 27 04:08:17.534296584 (6h16m33.315229953s) [noise=true , address=104.36.82.36:4650 , piece_id=BKJULWGPESBINNTSUYP6E4KZRXKPACVU5I7DOSZBO4YMK2V26HEQ, seg_pos={0 0}, upload=5911]: Done uploading piece. err: <nil> Jul 27 04:08:17.677851566 (6h16m33.458784943s) [noise=true , address=104.36.82.14:4248 , piece_id=LDSGCUM7Q2VX5CDFI4JRGEEG7D5H5Q5Z5WUWKU277VN2JFRX6UGA, seg_pos={0 0}, upload=5911]: Done uploading piece. err: ecclient: upload failed (node:1u6ir2PLqDa9xGm64cuqq1RqY4qeBa1fqCo9VghspBf54XEXqn, address:104.36.82.14:4248): protocol: rpc: dial tcp 104.36.82.14:4248: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 04:08:17.678115170 (6h16m33.459048549s) [noise=true , address=104.36.82.46:4496 , piece_id=GWNQR5WRGRRQBLSGWJFD3O3UNCMNE62JQNPIV5SYKBUIL3BQKA5A, seg_pos={0 0}, upload=5911]: Done uploading piece. err: ecclient: upload failed (node:12V8XhSWkyGUVYFR8MykifFAP1YAfGCwzqucDcMMfzw3xsVW77Z, address:104.36.82.46:4496): protocol: read tcp 159.223.143.39:52800->104.36.82.46:4496: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 04:08:17.678144409 (6h16m33.459077789s) [noise=true , address=104.36.82.44:4901 , piece_id=N4YLCLR2QZSULUL2XWONEKBFUMNOU7KUX2QEOK4SSDFSCI7IL6LA, seg_pos={0 0}, upload=5911]: Done uploading piece. err: ecclient: upload failed (node:1FHE84zvBb1MKwB1Ht7puTafgQxDEvPt4mYqkYVSSufAMpWQA1, address:104.36.82.44:4901): protocol: rpc: dial tcp 104.36.82.44:4901: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 04:08:17.678229622 (6h16m33.459162988s) [noise=true , address=104.36.82.44:4893 , piece_id=HUJICRJ6W54O3AZZR6H3KG6E4TGBTNS46O6XVMQM3PFAZE7A2HZA, seg_pos={0 0}, upload=5911]: Done uploading piece. err: ecclient: upload failed (node:17VMFZ2T8jrSJVtchciJ9SnyrR6JstvcMoXrN5Q3zTD1aQU1oJ, address:104.36.82.44:4893): protocol: write tcp 159.223.143.39:35302->104.36.82.44:4893: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 04:08:17.678310218 (6h16m33.459243583s) [noise=true , address=104.36.82.42:4829 , piece_id=ZLWJFP3LGX3YLXBJYIR66DXTYEVDBKXKVXCQSF2PCYCYJBIEJGMA, seg_pos={0 0}, upload=5911]: Done uploading piece. err: ecclient: upload failed (node:1F5T8Vt1x4nXFaSenvGBCYjxhRL1H1sxRvzLjW3CqzwDVhq39F, address:104.36.82.42:4829): protocol: rpc: dial tcp 104.36.82.42:4829: i/o timeout; conn closed: op; piecestore: piecestore close: context deadline exceeded Jul 27 04:08:17.678412989 (6h16m33.459346365s) [noise=true , address=104.36.82.38:4722 , piece_id=B2MBQY5CTMEI6WSE7FTLOV5RUJ22SCNG7N2EEJILEATGJ4TIALXA, seg_pos={0 0}, upload=5911]: Done uploading piece. err: ecclient: upload failed (node:12Zw3dtUg9gARz52KhwDLTjcqC3CNm7dpPzpdx7P7PbL1jwe8uc, address:104.36.82.38:4722): protocol: read tcp 159.223.143.39:56048->104.36.82.38:4722: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 04:08:17.678455247 (6h16m33.459388624s) [noise=true , address=104.36.82.47:4524 , piece_id=DONJFTKUQ2XCW6BOU7AVOKJ7XMDMBPGQVUPRANH7X2YS7PJXUKIA, seg_pos={0 0}, upload=5911]: Done uploading piece. err: ecclient: upload failed (node:1xfbCzjSwFDRrR1gPggeN8tNCLbRRWDVzCJ6mqhZsbVSe8d4s4, address:104.36.82.47:4524): protocol: read tcp 159.223.143.39:46694->104.36.82.47:4524: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 04:08:17.678510668 (6h16m33.459444052s) [seg_pos={0 0}, upload=5911]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Jul 27 04:08:17.706442864 (6h16m33.487376232s) [upload=5911]: Done uploading segments... Jul 27 04:08:17.706495329 (6h16m33.487428705s) [upload=5911]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 17.829292042s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................X..................] duration: 454.092621ms FAIL (2025-07-27 04:27:46 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 12) Duration: 454.136776ms core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................X.................] duration: 454.262732ms FAIL (2025-07-27 04:27:24 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 12) Duration: 454.325782ms core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................X................] duration: 510.243049ms FAIL (2025-07-27 04:26:59 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 12) Duration: 510.273959ms core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............................X...............................] duration: 454.806363ms FAIL (2025-07-27 04:26:58 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== ece_id=NYOFDEZ7CV4PHGQ6JZMEEVAYGJRPYG6IBW7MYJO42HMWOJM7PHZA, seg_pos={0 0}, upload=5879]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12crcXpNe9NgeXTsUpooLSBgrSUFbRxbSbanLa39w2Hr5mGFww): protocol: read tcp 159.223.143.39:53630->104.36.82.5:4840: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 04:05:32.508904485 (6h13m48.289837866s) [seg_pos={0 0}, upload=5879]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 27 04:05:32.509028070 (6h13m48.289961457s) [upload=5879]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 27 04:05:32.509034645 (6h13m48.289968129s) [upload=5879]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 27 04:05:32.581834839 (6h13m48.362768225s) [upload=5879]: Done uploading segments... Jul 27 04:05:32.581875919 (6h13m48.362809298s) [upload=5879]: Upload finished. err: <nil> Jul 27 04:06:23.829155331 (6h14m39.610088696s) [upload=5895]: Starting upload Jul 27 04:06:23.829225021 (6h14m39.610158398s) [upload=5895]: Done starting upload Jul 27 04:06:23.829294606 (6h14m39.61022799s) [upload=5895]: Uploading segments... Jul 27 04:06:23.829577656 (6h14m39.610511053s) [upload=5895]: Got next segment. Inline: false Jul 27 04:06:23.829649664 (6h14m39.610583027s) [upload=5895]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 27 04:06:23.829657065 (6h14m39.61059043s) [upload=5895]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 27 04:06:23.861405530 (6h14m39.642338913s) [seg_pos={0 0}, upload=5895]: Begin upload segment... Jul 27 04:06:23.861476351 (6h14m39.642409713s) [seg_pos={0 0}, upload=5895]: Done begin upload segment. Jul 27 04:06:23.861509120 (6h14m39.642442506s) [noise=true , address=77.90.31.19:4843 , piece_id=YML5RDQIK432Z4FSE4DKXIV5U47LQBEZRFMAVLANUSGFUQR23UWQ, seg_pos={0 0}, upload=5895]: Uploading piece... Jul 27 04:06:23.861953723 (6h14m39.6428871s) [noise=true , address=45.91.116.10:4081 , piece_id=XGDTSEZOUNRTEVWHTO3DHTZMH2VZZOPJPEKLCFAHTXESLLYABHSA, seg_pos={0 0}, upload=5895]: Uploading piece... Jul 27 04:06:23.862060983 (6h14m39.642994358s) [noise=true , address=45.91.116.9:4262 , piece_id=M5ETD45HH2XZD6HDLTJ3JO2BOOIROXFWKVPTN42PSQEWG4F7KAAQ, seg_pos={0 0}, upload=5895]: Uploading piece... Jul 27 04:06:23.862759464 (6h14m39.64369284s) [noise=true , address=104.36.82.4:4804 , piece_id=IU6FDNVB5AUPA7MJ6RSXUTLBETPHOFGNJ7YGWBFIZSQFSO6W6BBA, seg_pos={0 0}, upload=5895]: Uploading piece... Jul 27 04:06:23.862835086 (6h14m39.643768461s) [noise=true , address=104.36.82.6:4873 , piece_id=YP4VJORZZUL5EVCH3P7LD3LNVMHB7CWRTKHGIA7UROXE44N6MMZA, seg_pos={0 0}, upload=5895]: Uploading piece... Jul 27 04:06:23.863593880 (6h14m39.644527258s) [noise=true , address=77.90.31.15:4697 , piece_id=Q76TJDSZPSYZSJQNFVOI462U6DFILMH2RV3LQISODXSOCXGRKW6Q, seg_pos={0 0}, upload=5895]: Uploading piece... Jul 27 04:06:23.863656891 (6h14m39.644590266s) [noise=true , address=45.91.116.7:4011 , piece_id=XMH6MCCHZBBRQWHJFWLRGYHLQD4KUGSGL7MMZSKA5TZJCND3DUDA, seg_pos={0 0}, upload=5895]: Uploading piece... Jul 27 04:06:23.864408612 (6h14m39.645341987s) [noise=true , address=45.91.116.6:4973 , piece_id=J3VULPU24CIM64VEVE3UHO62HPN6TJXGZU6BCJIS5JQJKHJK2LIQ, seg_pos={0 0}, upload=5895]: Uploading piece... Jul 27 04:06:23.864521549 (6h14m39.64545491s) [noise=true , address=104.36.82.10:4079 , piece_id=2SH5D6JOMLFR7VSN7ZY5CLMR5RHIRFYQ7Z5I2JIYTJQM43G4O6OQ, seg_pos={0 0}, upload=5895]: Uploading piece... Jul 27 04:06:23.865518491 (6h14m39.646451872s) [noise=true , address=77.90.31.17:4771 , piece_id=U2LF22GKCMKRZN6EWCF26ITDZQHBFZXCGSQXT33MCX6P57IGI35Q, seg_pos={0 0}, upload=5895]: Uploading piece... Jul 27 04:06:23.865654950 (6h14m39.646588325s) [noise=true , address=45.91.116.8:4048 , piece_id=5REWMVVTBVLDQXQ5KAEHLY4VJL5SGF7ONQWDQQ3N2FDU3OAO6AZA, seg_pos={0 0}, upload=5895]: Uploading piece... Jul 27 04:06:23.866190892 (6h14m39.647124271s) [noise=true , address=104.36.82.12:4151 , piece_id=E2TEUTQBD4Y4AUHSCPMCIFEXBSY7S5HJWBKNKWWTUWEW2BLAAE6Q, seg_pos={0 0}, upload=5895]: Uploading piece... Jul 27 04:06:23.866412590 (6h14m39.647345968s) [noise=true , address=104.36.82.11:4115 , piece_id=YXMNVT6ZHGESGQ2BXKZUVR7U2CNNAI2PO7ZTMDG5UIOV7HTDDUTA, seg_pos={0 0}, upload=5895]: Uploading piece... Jul 27 04:06:23.866715972 (6h14m39.647649359s) [noise=true , address=45.91.116.13:4153 , piece_id=H4535KOCG3F4RKA6AKKMTJSOM6Q6PGRC6M7FR3UIS3BORTSP5UQA, seg_pos={0 0}, upload=5895]: Uploading piece... Jul 27 04:06:23.866998965 (6h14m39.64793234s) [noise=true , address=104.36.82.13:4186 , piece_id=TBLDKALGJYBGR5NLCOCKJ4QKEFNIJXCKAFUBD7VR3UMNWEMKMKHQ, seg_pos={0 0}, upload=5895]: Uploading piece... Jul 27 04:06:23.867230593 (6h14m39.648163974s) [noise=true , address=104.36.82.16:4293 , piece_id=AHH6PRFUSGTDGB7H52DRITARA4IF6GFLVSRYS2Y6EPJCZSKUOYWQ, seg_pos={0 0}, upload=5895]: Uploading piece... Jul 27 04:06:23.949422808 (6h14m39.730356187s) [upload=5895]: Next returned nil segment Jul 27 04:06:23.949442576 (6h14m39.73037596s) [upload=5895]: Waiting for error group managing segments... Jul 27 04:06:23.953497426 (6h14m39.734430802s) [noise=true , address=45.91.116.10:4081 , piece_id=XGDTSEZOUNRTEVWHTO3DHTZMH2VZZOPJPEKLCFAHTXESLLYABHSA, seg_pos={0 0}, upload=5895]: Done uploading piece. err: <nil> Jul 27 04:06:23.953531796 (6h14m39.734465159s) [noise=true , address=45.91.116.6:4973 , piece_id=J3VULPU24CIM64VEVE3UHO62HPN6TJXGZU6BCJIS5JQJKHJK2LIQ, seg_pos={0 0}, upload=5895]: Done uploading piece. err: <nil> Jul 27 04:06:23.954068660 (6h14m39.735002029s) [noise=true , address=45.91.116.9:4262 , piece_id=M5ETD45HH2XZD6HDLTJ3JO2BOOIROXFWKVPTN42PSQEWG4F7KAAQ, seg_pos={0 0}, upload=5895]: Done uploading piece. err: <nil> Jul 27 04:06:23.954194339 (6h14m39.735127707s) [noise=true , address=45.91.116.8:4048 , piece_id=5REWMVVTBVLDQXQ5KAEHLY4VJL5SGF7ONQWDQQ3N2FDU3OAO6AZA, seg_pos={0 0}, upload=5895]: Done uploading piece. err: <nil> Jul 27 04:06:23.954601730 (6h14m39.735535112s) [noise=true , address=45.91.116.7:4011 , piece_id=XMH6MCCHZBBRQWHJFWLRGYHLQD4KUGSGL7MMZSKA5TZJCND3DUDA, seg_pos={0 0}, upload=5895]: Done uploading piece. err: <nil> Jul 27 04:06:23.954755872 (6h14m39.735689237s) [noise=true , address=45.91.116.13:4153 , piece_id=H4535KOCG3F4RKA6AKKMTJSOM6Q6PGRC6M7FR3UIS3BORTSP5UQA, seg_pos={0 0}, upload=5895]: Done uploading piece. err: <nil> Jul 27 04:06:23.955611883 (6h14m39.73654526s) [noise=true , address=77.90.31.19:4843 , piece_id=YML5RDQIK432Z4FSE4DKXIV5U47LQBEZRFMAVLANUSGFUQR23UWQ, seg_pos={0 0}, upload=5895]: Done uploading piece. err: <nil> Jul 27 04:06:23.956373391 (6h14m39.737306756s) [noise=true , address=77.90.31.15:4697 , piece_id=Q76TJDSZPSYZSJQNFVOI462U6DFILMH2RV3LQISODXSOCXGRKW6Q, seg_pos={0 0}, upload=5895]: Done uploading piece. err: <nil> Jul 27 04:06:23.958659122 (6h14m39.73959249s) [noise=true , address=77.90.31.17:4771 , piece_id=U2LF22GKCMKRZN6EWCF26ITDZQHBFZXCGSQXT33MCX6P57IGI35Q, seg_pos={0 0}, upload=5895]: Done uploading piece. err: <nil> Jul 27 04:06:35.058944013 (6h14m50.83987739s) [noise=true , address=104.36.82.16:4293 , piece_id=AHH6PRFUSGTDGB7H52DRITARA4IF6GFLVSRYS2Y6EPJCZSKUOYWQ, seg_pos={0 0}, upload=5895]: Done uploading piece. err: <nil> Jul 27 04:06:41.103204002 (6h14m56.884137386s) [noise=true , address=104.36.82.4:4804 , piece_id=IU6FDNVB5AUPA7MJ6RSXUTLBETPHOFGNJ7YGWBFIZSQFSO6W6BBA, seg_pos={0 0}, upload=5895]: Done uploading piece. err: <nil> Jul 27 04:06:43.191702285 (6h14m58.972635651s) [noise=true , address=104.36.82.6:4873 , piece_id=YP4VJORZZUL5EVCH3P7LD3LNVMHB7CWRTKHGIA7UROXE44N6MMZA, seg_pos={0 0}, upload=5895]: Done uploading piece. err: <nil> Jul 27 04:06:53.777973493 (6h15m9.558906861s) [noise=true , address=104.36.82.11:4115 , piece_id=YXMNVT6ZHGESGQ2BXKZUVR7U2CNNAI2PO7ZTMDG5UIOV7HTDDUTA, seg_pos={0 0}, upload=5895]: Done uploading piece. err: ecclient: upload failed (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR, address:104.36.82.11:4115): protocol: write tcp 159.223.143.39:34210->104.36.82.11:4115: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 04:06:53.778019532 (6h15m9.558952909s) [noise=true , address=104.36.82.12:4151 , piece_id=E2TEUTQBD4Y4AUHSCPMCIFEXBSY7S5HJWBKNKWWTUWEW2BLAAE6Q, seg_pos={0 0}, upload=5895]: Done uploading piece. err: ecclient: upload failed (node:1w6pwZMdeQKjt92qx6WbTWU5VPYJto5gS87f6iKvCBZH6LfyeB, address:104.36.82.12:4151): protocol: rpc: dial tcp 104.36.82.12:4151: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 04:06:53.778043653 (6h15m9.558977021s) [noise=true , address=104.36.82.13:4186 , piece_id=TBLDKALGJYBGR5NLCOCKJ4QKEFNIJXCKAFUBD7VR3UMNWEMKMKHQ, seg_pos={0 0}, upload=5895]: Done uploading piece. err: ecclient: upload failed (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht, address:104.36.82.13:4186): protocol: read tcp 159.223.143.39:34340->104.36.82.13:4186: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 04:06:53.778095228 (6h15m9.559028593s) [noise=true , address=104.36.82.10:4079 , piece_id=2SH5D6JOMLFR7VSN7ZY5CLMR5RHIRFYQ7Z5I2JIYTJQM43G4O6OQ, seg_pos={0 0}, upload=5895]: Done uploading piece. err: ecclient: upload failed (node:12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud, address:104.36.82.10:4079): protocol: write tcp 159.223.143.39:53840->104.36.82.10:4079: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 04:06:53.778142167 (6h15m9.559075532s) [seg_pos={0 0}, upload=5895]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 27 04:06:53.837032161 (6h15m9.617965543s) [upload=5895]: Done uploading segments... Jul 27 04:06:53.837055996 (6h15m9.617989366s) [upload=5895]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 454.840605ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [..................................X.........................] duration: 2.539507977s FAIL (2025-07-27 04:26:45 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== {0 0}, upload=5911]: Begin upload segment... Jul 27 04:07:17.745307658 (6h15m33.526241043s) [seg_pos={0 0}, upload=5911]: Done begin upload segment. Jul 27 04:07:17.745349241 (6h15m33.526282625s) [noise=true , address=104.36.82.52:4098 , piece_id=M2ETF25QQ53RRGFPUKXJRINIWV7AWXSECEDR2AIM645IED72NURQ, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.745439694 (6h15m33.526373073s) [noise=true , address=104.36.82.56:4244 , piece_id=LQCZ5W54NBHWAVBVYK7HYSSJDC73E6FV4CP4YIM4L5H2N3OLOZXA, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.745972271 (6h15m33.526905654s) [noise=true , address=104.36.82.36:4650 , piece_id=BKJULWGPESBINNTSUYP6E4KZRXKPACVU5I7DOSZBO4YMK2V26HEQ, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.745980243 (6h15m33.526913621s) [noise=true , address=104.36.82.38:4722 , piece_id=B2MBQY5CTMEI6WSE7FTLOV5RUJ22SCNG7N2EEJILEATGJ4TIALXA, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.746403818 (6h15m33.527337198s) [noise=true , address=104.36.82.22:4534 , piece_id=CW4AQT3XUGA4A5OC6NUY4NYOOAMLN5FFJJN5PQBYEON7HPD5LCYQ, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.746455493 (6h15m33.527388875s) [noise=true , address=104.36.82.14:4248 , piece_id=LDSGCUM7Q2VX5CDFI4JRGEEG7D5H5Q5Z5WUWKU277VN2JFRX6UGA, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.746872127 (6h15m33.527805505s) [noise=true , address=104.36.82.40:4954 , piece_id=YLOHTJLRAED66OMEQYPZHNNRI77V6WOR4EHTEZOOEHMGZHJ2O3QQ, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.747123162 (6h15m33.52805654s) [noise=true , address=104.36.82.46:4496 , piece_id=GWNQR5WRGRRQBLSGWJFD3O3UNCMNE62JQNPIV5SYKBUIL3BQKA5A, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.747280678 (6h15m33.528214053s) [noise=true , address=104.36.82.47:4524 , piece_id=DONJFTKUQ2XCW6BOU7AVOKJ7XMDMBPGQVUPRANH7X2YS7PJXUKIA, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.747736400 (6h15m33.528669779s) [noise=true , address=104.36.82.44:4893 , piece_id=HUJICRJ6W54O3AZZR6H3KG6E4TGBTNS46O6XVMQM3PFAZE7A2HZA, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.747755734 (6h15m33.528689109s) [noise=true , address=104.36.82.55:4204 , piece_id=72WISGRHMOAZDV5EDGTBOMLBL4HDOF3CBEMQAAIRGILPC7J5JKAQ, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.748313124 (6h15m33.5292465s) [noise=true , address=104.36.82.58:4287 , piece_id=BVDYUEWU7TVEQDMZXX5SJNS4QICVKMTGFLCUV6P2YHOE7TI25VXA, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.748485217 (6h15m33.529418594s) [noise=true , address=104.36.82.53:4123 , piece_id=NXJRZ5BQSLXSOIOSUJ4MM4AGKHP4UIDUGEJKLUUX7YOGQJA6WWZA, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.748831329 (6h15m33.529764704s) [noise=true , address=104.36.82.42:4829 , piece_id=ZLWJFP3LGX3YLXBJYIR66DXTYEVDBKXKVXCQSF2PCYCYJBIEJGMA, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.749299293 (6h15m33.530232669s) [noise=true , address=104.36.82.8:4002 , piece_id=6XSPWQC3US355PS6IQ4K3IYNQZH5MAT4IVIPGSYRUM3XYPSCWE6Q, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.749790974 (6h15m33.53072436s) [noise=true , address=104.36.82.44:4901 , piece_id=N4YLCLR2QZSULUL2XWONEKBFUMNOU7KUX2QEOK4SSDFSCI7IL6LA, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.750478221 (6h15m33.531411597s) [noise=true , address=104.36.82.54:4172 , piece_id=ZNA53MWDRWPUH6FUR4DJERBDT2IOYGA3IYUADKXXIOLQJLQOXE7Q, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.752884255 (6h15m33.533817625s) [noise=true , address=64.49.13.140:4508 , piece_id=MZ52U75VOIK6Q7QFXMMYRRG7JXFWDFU2JWQFYPERUCMH3BUX6XPQ, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.879839555 (6h15m33.660772917s) [upload=5911]: Next returned nil segment Jul 27 04:07:17.879861389 (6h15m33.660794765s) [upload=5911]: Waiting for error group managing segments... Jul 27 04:07:17.888652438 (6h15m33.669585811s) [noise=true , address=104.36.82.58:4287 , piece_id=BVDYUEWU7TVEQDMZXX5SJNS4QICVKMTGFLCUV6P2YHOE7TI25VXA, seg_pos={0 0}, upload=5911]: Done uploading piece. err: <nil> Jul 27 04:07:17.952192045 (6h15m33.733125429s) [noise=true , address=104.36.82.53:4123 , piece_id=NXJRZ5BQSLXSOIOSUJ4MM4AGKHP4UIDUGEJKLUUX7YOGQJA6WWZA, seg_pos={0 0}, upload=5911]: Done uploading piece. err: <nil> Jul 27 04:07:17.981907539 (6h15m33.762840911s) [noise=true , address=104.36.82.56:4244 , piece_id=LQCZ5W54NBHWAVBVYK7HYSSJDC73E6FV4CP4YIM4L5H2N3OLOZXA, seg_pos={0 0}, upload=5911]: Done uploading piece. err: <nil> Jul 27 04:07:17.982370496 (6h15m33.7633039s) [noise=true , address=104.36.82.55:4204 , piece_id=72WISGRHMOAZDV5EDGTBOMLBL4HDOF3CBEMQAAIRGILPC7J5JKAQ, seg_pos={0 0}, upload=5911]: Done uploading piece. err: <nil> Jul 27 04:07:18.009548506 (6h15m33.790481886s) [noise=true , address=104.36.82.52:4098 , piece_id=M2ETF25QQ53RRGFPUKXJRINIWV7AWXSECEDR2AIM645IED72NURQ, seg_pos={0 0}, upload=5911]: Done uploading piece. err: <nil> Jul 27 04:07:18.023770598 (6h15m33.804703983s) [noise=true , address=104.36.82.40:4954 , piece_id=YLOHTJLRAED66OMEQYPZHNNRI77V6WOR4EHTEZOOEHMGZHJ2O3QQ, seg_pos={0 0}, upload=5911]: Done uploading piece. err: <nil> Jul 27 04:07:18.139745581 (6h15m33.92067895s) [noise=true , address=104.36.82.22:4534 , piece_id=CW4AQT3XUGA4A5OC6NUY4NYOOAMLN5FFJJN5PQBYEON7HPD5LCYQ, seg_pos={0 0}, upload=5911]: Done uploading piece. err: <nil> Jul 27 04:07:18.423748140 (6h15m34.204681511s) [noise=true , address=64.49.13.140:4508 , piece_id=MZ52U75VOIK6Q7QFXMMYRRG7JXFWDFU2JWQFYPERUCMH3BUX6XPQ, seg_pos={0 0}, upload=5911]: Done uploading piece. err: <nil> Jul 27 04:08:16.739940381 (6h16m32.52087375s) [noise=true , address=104.36.82.54:4172 , piece_id=ZNA53MWDRWPUH6FUR4DJERBDT2IOYGA3IYUADKXXIOLQJLQOXE7Q, seg_pos={0 0}, upload=5911]: Done uploading piece. err: <nil> Jul 27 04:08:17.177295397 (6h16m32.958228766s) [noise=true , address=104.36.82.8:4002 , piece_id=6XSPWQC3US355PS6IQ4K3IYNQZH5MAT4IVIPGSYRUM3XYPSCWE6Q, seg_pos={0 0}, upload=5911]: Done uploading piece. err: <nil> Jul 27 04:08:17.534296584 (6h16m33.315229953s) [noise=true , address=104.36.82.36:4650 , piece_id=BKJULWGPESBINNTSUYP6E4KZRXKPACVU5I7DOSZBO4YMK2V26HEQ, seg_pos={0 0}, upload=5911]: Done uploading piece. err: <nil> Jul 27 04:08:17.677851566 (6h16m33.458784943s) [noise=true , address=104.36.82.14:4248 , piece_id=LDSGCUM7Q2VX5CDFI4JRGEEG7D5H5Q5Z5WUWKU277VN2JFRX6UGA, seg_pos={0 0}, upload=5911]: Done uploading piece. err: ecclient: upload failed (node:1u6ir2PLqDa9xGm64cuqq1RqY4qeBa1fqCo9VghspBf54XEXqn, address:104.36.82.14:4248): protocol: rpc: dial tcp 104.36.82.14:4248: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 04:08:17.678115170 (6h16m33.459048549s) [noise=true , address=104.36.82.46:4496 , piece_id=GWNQR5WRGRRQBLSGWJFD3O3UNCMNE62JQNPIV5SYKBUIL3BQKA5A, seg_pos={0 0}, upload=5911]: Done uploading piece. err: ecclient: upload failed (node:12V8XhSWkyGUVYFR8MykifFAP1YAfGCwzqucDcMMfzw3xsVW77Z, address:104.36.82.46:4496): protocol: read tcp 159.223.143.39:52800->104.36.82.46:4496: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 04:08:17.678144409 (6h16m33.459077789s) [noise=true , address=104.36.82.44:4901 , piece_id=N4YLCLR2QZSULUL2XWONEKBFUMNOU7KUX2QEOK4SSDFSCI7IL6LA, seg_pos={0 0}, upload=5911]: Done uploading piece. err: ecclient: upload failed (node:1FHE84zvBb1MKwB1Ht7puTafgQxDEvPt4mYqkYVSSufAMpWQA1, address:104.36.82.44:4901): protocol: rpc: dial tcp 104.36.82.44:4901: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 04:08:17.678229622 (6h16m33.459162988s) [noise=true , address=104.36.82.44:4893 , piece_id=HUJICRJ6W54O3AZZR6H3KG6E4TGBTNS46O6XVMQM3PFAZE7A2HZA, seg_pos={0 0}, upload=5911]: Done uploading piece. err: ecclient: upload failed (node:17VMFZ2T8jrSJVtchciJ9SnyrR6JstvcMoXrN5Q3zTD1aQU1oJ, address:104.36.82.44:4893): protocol: write tcp 159.223.143.39:35302->104.36.82.44:4893: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 04:08:17.678310218 (6h16m33.459243583s) [noise=true , address=104.36.82.42:4829 , piece_id=ZLWJFP3LGX3YLXBJYIR66DXTYEVDBKXKVXCQSF2PCYCYJBIEJGMA, seg_pos={0 0}, upload=5911]: Done uploading piece. err: ecclient: upload failed (node:1F5T8Vt1x4nXFaSenvGBCYjxhRL1H1sxRvzLjW3CqzwDVhq39F, address:104.36.82.42:4829): protocol: rpc: dial tcp 104.36.82.42:4829: i/o timeout; conn closed: op; piecestore: piecestore close: context deadline exceeded Jul 27 04:08:17.678412989 (6h16m33.459346365s) [noise=true , address=104.36.82.38:4722 , piece_id=B2MBQY5CTMEI6WSE7FTLOV5RUJ22SCNG7N2EEJILEATGJ4TIALXA, seg_pos={0 0}, upload=5911]: Done uploading piece. err: ecclient: upload failed (node:12Zw3dtUg9gARz52KhwDLTjcqC3CNm7dpPzpdx7P7PbL1jwe8uc, address:104.36.82.38:4722): protocol: read tcp 159.223.143.39:56048->104.36.82.38:4722: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 04:08:17.678455247 (6h16m33.459388624s) [noise=true , address=104.36.82.47:4524 , piece_id=DONJFTKUQ2XCW6BOU7AVOKJ7XMDMBPGQVUPRANH7X2YS7PJXUKIA, seg_pos={0 0}, upload=5911]: Done uploading piece. err: ecclient: upload failed (node:1xfbCzjSwFDRrR1gPggeN8tNCLbRRWDVzCJ6mqhZsbVSe8d4s4, address:104.36.82.47:4524): protocol: read tcp 159.223.143.39:46694->104.36.82.47:4524: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 04:08:17.678510668 (6h16m33.459444052s) [seg_pos={0 0}, upload=5911]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Jul 27 04:08:17.706442864 (6h16m33.487376232s) [upload=5911]: Done uploading segments... Jul 27 04:08:17.706495329 (6h16m33.487428705s) [upload=5911]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 2.539670018s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [............................................X...............] duration: 542.090505ms FAIL (2025-07-27 04:26:43 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 12) Duration: 542.120388ms core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............................X..............................] duration: 474.105271ms FAIL (2025-07-27 04:26:26 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== ece_id=NYOFDEZ7CV4PHGQ6JZMEEVAYGJRPYG6IBW7MYJO42HMWOJM7PHZA, seg_pos={0 0}, upload=5879]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12crcXpNe9NgeXTsUpooLSBgrSUFbRxbSbanLa39w2Hr5mGFww): protocol: read tcp 159.223.143.39:53630->104.36.82.5:4840: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 04:05:32.508904485 (6h13m48.289837866s) [seg_pos={0 0}, upload=5879]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 27 04:05:32.509028070 (6h13m48.289961457s) [upload=5879]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 27 04:05:32.509034645 (6h13m48.289968129s) [upload=5879]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 27 04:05:32.581834839 (6h13m48.362768225s) [upload=5879]: Done uploading segments... Jul 27 04:05:32.581875919 (6h13m48.362809298s) [upload=5879]: Upload finished. err: <nil> Jul 27 04:06:23.829155331 (6h14m39.610088696s) [upload=5895]: Starting upload Jul 27 04:06:23.829225021 (6h14m39.610158398s) [upload=5895]: Done starting upload Jul 27 04:06:23.829294606 (6h14m39.61022799s) [upload=5895]: Uploading segments... Jul 27 04:06:23.829577656 (6h14m39.610511053s) [upload=5895]: Got next segment. Inline: false Jul 27 04:06:23.829649664 (6h14m39.610583027s) [upload=5895]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 27 04:06:23.829657065 (6h14m39.61059043s) [upload=5895]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 27 04:06:23.861405530 (6h14m39.642338913s) [seg_pos={0 0}, upload=5895]: Begin upload segment... Jul 27 04:06:23.861476351 (6h14m39.642409713s) [seg_pos={0 0}, upload=5895]: Done begin upload segment. Jul 27 04:06:23.861509120 (6h14m39.642442506s) [noise=true , address=77.90.31.19:4843 , piece_id=YML5RDQIK432Z4FSE4DKXIV5U47LQBEZRFMAVLANUSGFUQR23UWQ, seg_pos={0 0}, upload=5895]: Uploading piece... Jul 27 04:06:23.861953723 (6h14m39.6428871s) [noise=true , address=45.91.116.10:4081 , piece_id=XGDTSEZOUNRTEVWHTO3DHTZMH2VZZOPJPEKLCFAHTXESLLYABHSA, seg_pos={0 0}, upload=5895]: Uploading piece... Jul 27 04:06:23.862060983 (6h14m39.642994358s) [noise=true , address=45.91.116.9:4262 , piece_id=M5ETD45HH2XZD6HDLTJ3JO2BOOIROXFWKVPTN42PSQEWG4F7KAAQ, seg_pos={0 0}, upload=5895]: Uploading piece... Jul 27 04:06:23.862759464 (6h14m39.64369284s) [noise=true , address=104.36.82.4:4804 , piece_id=IU6FDNVB5AUPA7MJ6RSXUTLBETPHOFGNJ7YGWBFIZSQFSO6W6BBA, seg_pos={0 0}, upload=5895]: Uploading piece... Jul 27 04:06:23.862835086 (6h14m39.643768461s) [noise=true , address=104.36.82.6:4873 , piece_id=YP4VJORZZUL5EVCH3P7LD3LNVMHB7CWRTKHGIA7UROXE44N6MMZA, seg_pos={0 0}, upload=5895]: Uploading piece... Jul 27 04:06:23.863593880 (6h14m39.644527258s) [noise=true , address=77.90.31.15:4697 , piece_id=Q76TJDSZPSYZSJQNFVOI462U6DFILMH2RV3LQISODXSOCXGRKW6Q, seg_pos={0 0}, upload=5895]: Uploading piece... Jul 27 04:06:23.863656891 (6h14m39.644590266s) [noise=true , address=45.91.116.7:4011 , piece_id=XMH6MCCHZBBRQWHJFWLRGYHLQD4KUGSGL7MMZSKA5TZJCND3DUDA, seg_pos={0 0}, upload=5895]: Uploading piece... Jul 27 04:06:23.864408612 (6h14m39.645341987s) [noise=true , address=45.91.116.6:4973 , piece_id=J3VULPU24CIM64VEVE3UHO62HPN6TJXGZU6BCJIS5JQJKHJK2LIQ, seg_pos={0 0}, upload=5895]: Uploading piece... Jul 27 04:06:23.864521549 (6h14m39.64545491s) [noise=true , address=104.36.82.10:4079 , piece_id=2SH5D6JOMLFR7VSN7ZY5CLMR5RHIRFYQ7Z5I2JIYTJQM43G4O6OQ, seg_pos={0 0}, upload=5895]: Uploading piece... Jul 27 04:06:23.865518491 (6h14m39.646451872s) [noise=true , address=77.90.31.17:4771 , piece_id=U2LF22GKCMKRZN6EWCF26ITDZQHBFZXCGSQXT33MCX6P57IGI35Q, seg_pos={0 0}, upload=5895]: Uploading piece... Jul 27 04:06:23.865654950 (6h14m39.646588325s) [noise=true , address=45.91.116.8:4048 , piece_id=5REWMVVTBVLDQXQ5KAEHLY4VJL5SGF7ONQWDQQ3N2FDU3OAO6AZA, seg_pos={0 0}, upload=5895]: Uploading piece... Jul 27 04:06:23.866190892 (6h14m39.647124271s) [noise=true , address=104.36.82.12:4151 , piece_id=E2TEUTQBD4Y4AUHSCPMCIFEXBSY7S5HJWBKNKWWTUWEW2BLAAE6Q, seg_pos={0 0}, upload=5895]: Uploading piece... Jul 27 04:06:23.866412590 (6h14m39.647345968s) [noise=true , address=104.36.82.11:4115 , piece_id=YXMNVT6ZHGESGQ2BXKZUVR7U2CNNAI2PO7ZTMDG5UIOV7HTDDUTA, seg_pos={0 0}, upload=5895]: Uploading piece... Jul 27 04:06:23.866715972 (6h14m39.647649359s) [noise=true , address=45.91.116.13:4153 , piece_id=H4535KOCG3F4RKA6AKKMTJSOM6Q6PGRC6M7FR3UIS3BORTSP5UQA, seg_pos={0 0}, upload=5895]: Uploading piece... Jul 27 04:06:23.866998965 (6h14m39.64793234s) [noise=true , address=104.36.82.13:4186 , piece_id=TBLDKALGJYBGR5NLCOCKJ4QKEFNIJXCKAFUBD7VR3UMNWEMKMKHQ, seg_pos={0 0}, upload=5895]: Uploading piece... Jul 27 04:06:23.867230593 (6h14m39.648163974s) [noise=true , address=104.36.82.16:4293 , piece_id=AHH6PRFUSGTDGB7H52DRITARA4IF6GFLVSRYS2Y6EPJCZSKUOYWQ, seg_pos={0 0}, upload=5895]: Uploading piece... Jul 27 04:06:23.949422808 (6h14m39.730356187s) [upload=5895]: Next returned nil segment Jul 27 04:06:23.949442576 (6h14m39.73037596s) [upload=5895]: Waiting for error group managing segments... Jul 27 04:06:23.953497426 (6h14m39.734430802s) [noise=true , address=45.91.116.10:4081 , piece_id=XGDTSEZOUNRTEVWHTO3DHTZMH2VZZOPJPEKLCFAHTXESLLYABHSA, seg_pos={0 0}, upload=5895]: Done uploading piece. err: <nil> Jul 27 04:06:23.953531796 (6h14m39.734465159s) [noise=true , address=45.91.116.6:4973 , piece_id=J3VULPU24CIM64VEVE3UHO62HPN6TJXGZU6BCJIS5JQJKHJK2LIQ, seg_pos={0 0}, upload=5895]: Done uploading piece. err: <nil> Jul 27 04:06:23.954068660 (6h14m39.735002029s) [noise=true , address=45.91.116.9:4262 , piece_id=M5ETD45HH2XZD6HDLTJ3JO2BOOIROXFWKVPTN42PSQEWG4F7KAAQ, seg_pos={0 0}, upload=5895]: Done uploading piece. err: <nil> Jul 27 04:06:23.954194339 (6h14m39.735127707s) [noise=true , address=45.91.116.8:4048 , piece_id=5REWMVVTBVLDQXQ5KAEHLY4VJL5SGF7ONQWDQQ3N2FDU3OAO6AZA, seg_pos={0 0}, upload=5895]: Done uploading piece. err: <nil> Jul 27 04:06:23.954601730 (6h14m39.735535112s) [noise=true , address=45.91.116.7:4011 , piece_id=XMH6MCCHZBBRQWHJFWLRGYHLQD4KUGSGL7MMZSKA5TZJCND3DUDA, seg_pos={0 0}, upload=5895]: Done uploading piece. err: <nil> Jul 27 04:06:23.954755872 (6h14m39.735689237s) [noise=true , address=45.91.116.13:4153 , piece_id=H4535KOCG3F4RKA6AKKMTJSOM6Q6PGRC6M7FR3UIS3BORTSP5UQA, seg_pos={0 0}, upload=5895]: Done uploading piece. err: <nil> Jul 27 04:06:23.955611883 (6h14m39.73654526s) [noise=true , address=77.90.31.19:4843 , piece_id=YML5RDQIK432Z4FSE4DKXIV5U47LQBEZRFMAVLANUSGFUQR23UWQ, seg_pos={0 0}, upload=5895]: Done uploading piece. err: <nil> Jul 27 04:06:23.956373391 (6h14m39.737306756s) [noise=true , address=77.90.31.15:4697 , piece_id=Q76TJDSZPSYZSJQNFVOI462U6DFILMH2RV3LQISODXSOCXGRKW6Q, seg_pos={0 0}, upload=5895]: Done uploading piece. err: <nil> Jul 27 04:06:23.958659122 (6h14m39.73959249s) [noise=true , address=77.90.31.17:4771 , piece_id=U2LF22GKCMKRZN6EWCF26ITDZQHBFZXCGSQXT33MCX6P57IGI35Q, seg_pos={0 0}, upload=5895]: Done uploading piece. err: <nil> Jul 27 04:06:35.058944013 (6h14m50.83987739s) [noise=true , address=104.36.82.16:4293 , piece_id=AHH6PRFUSGTDGB7H52DRITARA4IF6GFLVSRYS2Y6EPJCZSKUOYWQ, seg_pos={0 0}, upload=5895]: Done uploading piece. err: <nil> Jul 27 04:06:41.103204002 (6h14m56.884137386s) [noise=true , address=104.36.82.4:4804 , piece_id=IU6FDNVB5AUPA7MJ6RSXUTLBETPHOFGNJ7YGWBFIZSQFSO6W6BBA, seg_pos={0 0}, upload=5895]: Done uploading piece. err: <nil> Jul 27 04:06:43.191702285 (6h14m58.972635651s) [noise=true , address=104.36.82.6:4873 , piece_id=YP4VJORZZUL5EVCH3P7LD3LNVMHB7CWRTKHGIA7UROXE44N6MMZA, seg_pos={0 0}, upload=5895]: Done uploading piece. err: <nil> Jul 27 04:06:53.777973493 (6h15m9.558906861s) [noise=true , address=104.36.82.11:4115 , piece_id=YXMNVT6ZHGESGQ2BXKZUVR7U2CNNAI2PO7ZTMDG5UIOV7HTDDUTA, seg_pos={0 0}, upload=5895]: Done uploading piece. err: ecclient: upload failed (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR, address:104.36.82.11:4115): protocol: write tcp 159.223.143.39:34210->104.36.82.11:4115: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 04:06:53.778019532 (6h15m9.558952909s) [noise=true , address=104.36.82.12:4151 , piece_id=E2TEUTQBD4Y4AUHSCPMCIFEXBSY7S5HJWBKNKWWTUWEW2BLAAE6Q, seg_pos={0 0}, upload=5895]: Done uploading piece. err: ecclient: upload failed (node:1w6pwZMdeQKjt92qx6WbTWU5VPYJto5gS87f6iKvCBZH6LfyeB, address:104.36.82.12:4151): protocol: rpc: dial tcp 104.36.82.12:4151: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 04:06:53.778043653 (6h15m9.558977021s) [noise=true , address=104.36.82.13:4186 , piece_id=TBLDKALGJYBGR5NLCOCKJ4QKEFNIJXCKAFUBD7VR3UMNWEMKMKHQ, seg_pos={0 0}, upload=5895]: Done uploading piece. err: ecclient: upload failed (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht, address:104.36.82.13:4186): protocol: read tcp 159.223.143.39:34340->104.36.82.13:4186: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 04:06:53.778095228 (6h15m9.559028593s) [noise=true , address=104.36.82.10:4079 , piece_id=2SH5D6JOMLFR7VSN7ZY5CLMR5RHIRFYQ7Z5I2JIYTJQM43G4O6OQ, seg_pos={0 0}, upload=5895]: Done uploading piece. err: ecclient: upload failed (node:12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud, address:104.36.82.10:4079): protocol: write tcp 159.223.143.39:53840->104.36.82.10:4079: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 04:06:53.778142167 (6h15m9.559075532s) [seg_pos={0 0}, upload=5895]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 27 04:06:53.837032161 (6h15m9.617965543s) [upload=5895]: Done uploading segments... Jul 27 04:06:53.837055996 (6h15m9.617989366s) [upload=5895]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 474.148518ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [...................................X........................] duration: 4.557647375s FAIL (2025-07-27 04:26:15 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== {0 0}, upload=5911]: Begin upload segment... Jul 27 04:07:17.745307658 (6h15m33.526241043s) [seg_pos={0 0}, upload=5911]: Done begin upload segment. Jul 27 04:07:17.745349241 (6h15m33.526282625s) [noise=true , address=104.36.82.52:4098 , piece_id=M2ETF25QQ53RRGFPUKXJRINIWV7AWXSECEDR2AIM645IED72NURQ, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.745439694 (6h15m33.526373073s) [noise=true , address=104.36.82.56:4244 , piece_id=LQCZ5W54NBHWAVBVYK7HYSSJDC73E6FV4CP4YIM4L5H2N3OLOZXA, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.745972271 (6h15m33.526905654s) [noise=true , address=104.36.82.36:4650 , piece_id=BKJULWGPESBINNTSUYP6E4KZRXKPACVU5I7DOSZBO4YMK2V26HEQ, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.745980243 (6h15m33.526913621s) [noise=true , address=104.36.82.38:4722 , piece_id=B2MBQY5CTMEI6WSE7FTLOV5RUJ22SCNG7N2EEJILEATGJ4TIALXA, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.746403818 (6h15m33.527337198s) [noise=true , address=104.36.82.22:4534 , piece_id=CW4AQT3XUGA4A5OC6NUY4NYOOAMLN5FFJJN5PQBYEON7HPD5LCYQ, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.746455493 (6h15m33.527388875s) [noise=true , address=104.36.82.14:4248 , piece_id=LDSGCUM7Q2VX5CDFI4JRGEEG7D5H5Q5Z5WUWKU277VN2JFRX6UGA, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.746872127 (6h15m33.527805505s) [noise=true , address=104.36.82.40:4954 , piece_id=YLOHTJLRAED66OMEQYPZHNNRI77V6WOR4EHTEZOOEHMGZHJ2O3QQ, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.747123162 (6h15m33.52805654s) [noise=true , address=104.36.82.46:4496 , piece_id=GWNQR5WRGRRQBLSGWJFD3O3UNCMNE62JQNPIV5SYKBUIL3BQKA5A, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.747280678 (6h15m33.528214053s) [noise=true , address=104.36.82.47:4524 , piece_id=DONJFTKUQ2XCW6BOU7AVOKJ7XMDMBPGQVUPRANH7X2YS7PJXUKIA, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.747736400 (6h15m33.528669779s) [noise=true , address=104.36.82.44:4893 , piece_id=HUJICRJ6W54O3AZZR6H3KG6E4TGBTNS46O6XVMQM3PFAZE7A2HZA, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.747755734 (6h15m33.528689109s) [noise=true , address=104.36.82.55:4204 , piece_id=72WISGRHMOAZDV5EDGTBOMLBL4HDOF3CBEMQAAIRGILPC7J5JKAQ, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.748313124 (6h15m33.5292465s) [noise=true , address=104.36.82.58:4287 , piece_id=BVDYUEWU7TVEQDMZXX5SJNS4QICVKMTGFLCUV6P2YHOE7TI25VXA, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.748485217 (6h15m33.529418594s) [noise=true , address=104.36.82.53:4123 , piece_id=NXJRZ5BQSLXSOIOSUJ4MM4AGKHP4UIDUGEJKLUUX7YOGQJA6WWZA, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.748831329 (6h15m33.529764704s) [noise=true , address=104.36.82.42:4829 , piece_id=ZLWJFP3LGX3YLXBJYIR66DXTYEVDBKXKVXCQSF2PCYCYJBIEJGMA, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.749299293 (6h15m33.530232669s) [noise=true , address=104.36.82.8:4002 , piece_id=6XSPWQC3US355PS6IQ4K3IYNQZH5MAT4IVIPGSYRUM3XYPSCWE6Q, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.749790974 (6h15m33.53072436s) [noise=true , address=104.36.82.44:4901 , piece_id=N4YLCLR2QZSULUL2XWONEKBFUMNOU7KUX2QEOK4SSDFSCI7IL6LA, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.750478221 (6h15m33.531411597s) [noise=true , address=104.36.82.54:4172 , piece_id=ZNA53MWDRWPUH6FUR4DJERBDT2IOYGA3IYUADKXXIOLQJLQOXE7Q, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.752884255 (6h15m33.533817625s) [noise=true , address=64.49.13.140:4508 , piece_id=MZ52U75VOIK6Q7QFXMMYRRG7JXFWDFU2JWQFYPERUCMH3BUX6XPQ, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.879839555 (6h15m33.660772917s) [upload=5911]: Next returned nil segment Jul 27 04:07:17.879861389 (6h15m33.660794765s) [upload=5911]: Waiting for error group managing segments... Jul 27 04:07:17.888652438 (6h15m33.669585811s) [noise=true , address=104.36.82.58:4287 , piece_id=BVDYUEWU7TVEQDMZXX5SJNS4QICVKMTGFLCUV6P2YHOE7TI25VXA, seg_pos={0 0}, upload=5911]: Done uploading piece. err: <nil> Jul 27 04:07:17.952192045 (6h15m33.733125429s) [noise=true , address=104.36.82.53:4123 , piece_id=NXJRZ5BQSLXSOIOSUJ4MM4AGKHP4UIDUGEJKLUUX7YOGQJA6WWZA, seg_pos={0 0}, upload=5911]: Done uploading piece. err: <nil> Jul 27 04:07:17.981907539 (6h15m33.762840911s) [noise=true , address=104.36.82.56:4244 , piece_id=LQCZ5W54NBHWAVBVYK7HYSSJDC73E6FV4CP4YIM4L5H2N3OLOZXA, seg_pos={0 0}, upload=5911]: Done uploading piece. err: <nil> Jul 27 04:07:17.982370496 (6h15m33.7633039s) [noise=true , address=104.36.82.55:4204 , piece_id=72WISGRHMOAZDV5EDGTBOMLBL4HDOF3CBEMQAAIRGILPC7J5JKAQ, seg_pos={0 0}, upload=5911]: Done uploading piece. err: <nil> Jul 27 04:07:18.009548506 (6h15m33.790481886s) [noise=true , address=104.36.82.52:4098 , piece_id=M2ETF25QQ53RRGFPUKXJRINIWV7AWXSECEDR2AIM645IED72NURQ, seg_pos={0 0}, upload=5911]: Done uploading piece. err: <nil> Jul 27 04:07:18.023770598 (6h15m33.804703983s) [noise=true , address=104.36.82.40:4954 , piece_id=YLOHTJLRAED66OMEQYPZHNNRI77V6WOR4EHTEZOOEHMGZHJ2O3QQ, seg_pos={0 0}, upload=5911]: Done uploading piece. err: <nil> Jul 27 04:07:18.139745581 (6h15m33.92067895s) [noise=true , address=104.36.82.22:4534 , piece_id=CW4AQT3XUGA4A5OC6NUY4NYOOAMLN5FFJJN5PQBYEON7HPD5LCYQ, seg_pos={0 0}, upload=5911]: Done uploading piece. err: <nil> Jul 27 04:07:18.423748140 (6h15m34.204681511s) [noise=true , address=64.49.13.140:4508 , piece_id=MZ52U75VOIK6Q7QFXMMYRRG7JXFWDFU2JWQFYPERUCMH3BUX6XPQ, seg_pos={0 0}, upload=5911]: Done uploading piece. err: <nil> Jul 27 04:08:16.739940381 (6h16m32.52087375s) [noise=true , address=104.36.82.54:4172 , piece_id=ZNA53MWDRWPUH6FUR4DJERBDT2IOYGA3IYUADKXXIOLQJLQOXE7Q, seg_pos={0 0}, upload=5911]: Done uploading piece. err: <nil> Jul 27 04:08:17.177295397 (6h16m32.958228766s) [noise=true , address=104.36.82.8:4002 , piece_id=6XSPWQC3US355PS6IQ4K3IYNQZH5MAT4IVIPGSYRUM3XYPSCWE6Q, seg_pos={0 0}, upload=5911]: Done uploading piece. err: <nil> Jul 27 04:08:17.534296584 (6h16m33.315229953s) [noise=true , address=104.36.82.36:4650 , piece_id=BKJULWGPESBINNTSUYP6E4KZRXKPACVU5I7DOSZBO4YMK2V26HEQ, seg_pos={0 0}, upload=5911]: Done uploading piece. err: <nil> Jul 27 04:08:17.677851566 (6h16m33.458784943s) [noise=true , address=104.36.82.14:4248 , piece_id=LDSGCUM7Q2VX5CDFI4JRGEEG7D5H5Q5Z5WUWKU277VN2JFRX6UGA, seg_pos={0 0}, upload=5911]: Done uploading piece. err: ecclient: upload failed (node:1u6ir2PLqDa9xGm64cuqq1RqY4qeBa1fqCo9VghspBf54XEXqn, address:104.36.82.14:4248): protocol: rpc: dial tcp 104.36.82.14:4248: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 04:08:17.678115170 (6h16m33.459048549s) [noise=true , address=104.36.82.46:4496 , piece_id=GWNQR5WRGRRQBLSGWJFD3O3UNCMNE62JQNPIV5SYKBUIL3BQKA5A, seg_pos={0 0}, upload=5911]: Done uploading piece. err: ecclient: upload failed (node:12V8XhSWkyGUVYFR8MykifFAP1YAfGCwzqucDcMMfzw3xsVW77Z, address:104.36.82.46:4496): protocol: read tcp 159.223.143.39:52800->104.36.82.46:4496: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 04:08:17.678144409 (6h16m33.459077789s) [noise=true , address=104.36.82.44:4901 , piece_id=N4YLCLR2QZSULUL2XWONEKBFUMNOU7KUX2QEOK4SSDFSCI7IL6LA, seg_pos={0 0}, upload=5911]: Done uploading piece. err: ecclient: upload failed (node:1FHE84zvBb1MKwB1Ht7puTafgQxDEvPt4mYqkYVSSufAMpWQA1, address:104.36.82.44:4901): protocol: rpc: dial tcp 104.36.82.44:4901: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 04:08:17.678229622 (6h16m33.459162988s) [noise=true , address=104.36.82.44:4893 , piece_id=HUJICRJ6W54O3AZZR6H3KG6E4TGBTNS46O6XVMQM3PFAZE7A2HZA, seg_pos={0 0}, upload=5911]: Done uploading piece. err: ecclient: upload failed (node:17VMFZ2T8jrSJVtchciJ9SnyrR6JstvcMoXrN5Q3zTD1aQU1oJ, address:104.36.82.44:4893): protocol: write tcp 159.223.143.39:35302->104.36.82.44:4893: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 04:08:17.678310218 (6h16m33.459243583s) [noise=true , address=104.36.82.42:4829 , piece_id=ZLWJFP3LGX3YLXBJYIR66DXTYEVDBKXKVXCQSF2PCYCYJBIEJGMA, seg_pos={0 0}, upload=5911]: Done uploading piece. err: ecclient: upload failed (node:1F5T8Vt1x4nXFaSenvGBCYjxhRL1H1sxRvzLjW3CqzwDVhq39F, address:104.36.82.42:4829): protocol: rpc: dial tcp 104.36.82.42:4829: i/o timeout; conn closed: op; piecestore: piecestore close: context deadline exceeded Jul 27 04:08:17.678412989 (6h16m33.459346365s) [noise=true , address=104.36.82.38:4722 , piece_id=B2MBQY5CTMEI6WSE7FTLOV5RUJ22SCNG7N2EEJILEATGJ4TIALXA, seg_pos={0 0}, upload=5911]: Done uploading piece. err: ecclient: upload failed (node:12Zw3dtUg9gARz52KhwDLTjcqC3CNm7dpPzpdx7P7PbL1jwe8uc, address:104.36.82.38:4722): protocol: read tcp 159.223.143.39:56048->104.36.82.38:4722: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 04:08:17.678455247 (6h16m33.459388624s) [noise=true , address=104.36.82.47:4524 , piece_id=DONJFTKUQ2XCW6BOU7AVOKJ7XMDMBPGQVUPRANH7X2YS7PJXUKIA, seg_pos={0 0}, upload=5911]: Done uploading piece. err: ecclient: upload failed (node:1xfbCzjSwFDRrR1gPggeN8tNCLbRRWDVzCJ6mqhZsbVSe8d4s4, address:104.36.82.47:4524): protocol: read tcp 159.223.143.39:46694->104.36.82.47:4524: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 04:08:17.678510668 (6h16m33.459444052s) [seg_pos={0 0}, upload=5911]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Jul 27 04:08:17.706442864 (6h16m33.487376232s) [upload=5911]: Done uploading segments... Jul 27 04:08:17.706495329 (6h16m33.487428705s) [upload=5911]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 4.557744051s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............................................X..............] duration: 407.19187ms FAIL (2025-07-27 04:25:56 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 12) Duration: 407.221758ms core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [....................................X.......................] duration: 7.492338556s FAIL (2025-07-27 04:25:35 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== {0 0}, upload=5911]: Begin upload segment... Jul 27 04:07:17.745307658 (6h15m33.526241043s) [seg_pos={0 0}, upload=5911]: Done begin upload segment. Jul 27 04:07:17.745349241 (6h15m33.526282625s) [noise=true , address=104.36.82.52:4098 , piece_id=M2ETF25QQ53RRGFPUKXJRINIWV7AWXSECEDR2AIM645IED72NURQ, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.745439694 (6h15m33.526373073s) [noise=true , address=104.36.82.56:4244 , piece_id=LQCZ5W54NBHWAVBVYK7HYSSJDC73E6FV4CP4YIM4L5H2N3OLOZXA, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.745972271 (6h15m33.526905654s) [noise=true , address=104.36.82.36:4650 , piece_id=BKJULWGPESBINNTSUYP6E4KZRXKPACVU5I7DOSZBO4YMK2V26HEQ, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.745980243 (6h15m33.526913621s) [noise=true , address=104.36.82.38:4722 , piece_id=B2MBQY5CTMEI6WSE7FTLOV5RUJ22SCNG7N2EEJILEATGJ4TIALXA, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.746403818 (6h15m33.527337198s) [noise=true , address=104.36.82.22:4534 , piece_id=CW4AQT3XUGA4A5OC6NUY4NYOOAMLN5FFJJN5PQBYEON7HPD5LCYQ, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.746455493 (6h15m33.527388875s) [noise=true , address=104.36.82.14:4248 , piece_id=LDSGCUM7Q2VX5CDFI4JRGEEG7D5H5Q5Z5WUWKU277VN2JFRX6UGA, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.746872127 (6h15m33.527805505s) [noise=true , address=104.36.82.40:4954 , piece_id=YLOHTJLRAED66OMEQYPZHNNRI77V6WOR4EHTEZOOEHMGZHJ2O3QQ, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.747123162 (6h15m33.52805654s) [noise=true , address=104.36.82.46:4496 , piece_id=GWNQR5WRGRRQBLSGWJFD3O3UNCMNE62JQNPIV5SYKBUIL3BQKA5A, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.747280678 (6h15m33.528214053s) [noise=true , address=104.36.82.47:4524 , piece_id=DONJFTKUQ2XCW6BOU7AVOKJ7XMDMBPGQVUPRANH7X2YS7PJXUKIA, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.747736400 (6h15m33.528669779s) [noise=true , address=104.36.82.44:4893 , piece_id=HUJICRJ6W54O3AZZR6H3KG6E4TGBTNS46O6XVMQM3PFAZE7A2HZA, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.747755734 (6h15m33.528689109s) [noise=true , address=104.36.82.55:4204 , piece_id=72WISGRHMOAZDV5EDGTBOMLBL4HDOF3CBEMQAAIRGILPC7J5JKAQ, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.748313124 (6h15m33.5292465s) [noise=true , address=104.36.82.58:4287 , piece_id=BVDYUEWU7TVEQDMZXX5SJNS4QICVKMTGFLCUV6P2YHOE7TI25VXA, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.748485217 (6h15m33.529418594s) [noise=true , address=104.36.82.53:4123 , piece_id=NXJRZ5BQSLXSOIOSUJ4MM4AGKHP4UIDUGEJKLUUX7YOGQJA6WWZA, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.748831329 (6h15m33.529764704s) [noise=true , address=104.36.82.42:4829 , piece_id=ZLWJFP3LGX3YLXBJYIR66DXTYEVDBKXKVXCQSF2PCYCYJBIEJGMA, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.749299293 (6h15m33.530232669s) [noise=true , address=104.36.82.8:4002 , piece_id=6XSPWQC3US355PS6IQ4K3IYNQZH5MAT4IVIPGSYRUM3XYPSCWE6Q, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.749790974 (6h15m33.53072436s) [noise=true , address=104.36.82.44:4901 , piece_id=N4YLCLR2QZSULUL2XWONEKBFUMNOU7KUX2QEOK4SSDFSCI7IL6LA, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.750478221 (6h15m33.531411597s) [noise=true , address=104.36.82.54:4172 , piece_id=ZNA53MWDRWPUH6FUR4DJERBDT2IOYGA3IYUADKXXIOLQJLQOXE7Q, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.752884255 (6h15m33.533817625s) [noise=true , address=64.49.13.140:4508 , piece_id=MZ52U75VOIK6Q7QFXMMYRRG7JXFWDFU2JWQFYPERUCMH3BUX6XPQ, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.879839555 (6h15m33.660772917s) [upload=5911]: Next returned nil segment Jul 27 04:07:17.879861389 (6h15m33.660794765s) [upload=5911]: Waiting for error group managing segments... Jul 27 04:07:17.888652438 (6h15m33.669585811s) [noise=true , address=104.36.82.58:4287 , piece_id=BVDYUEWU7TVEQDMZXX5SJNS4QICVKMTGFLCUV6P2YHOE7TI25VXA, seg_pos={0 0}, upload=5911]: Done uploading piece. err: <nil> Jul 27 04:07:17.952192045 (6h15m33.733125429s) [noise=true , address=104.36.82.53:4123 , piece_id=NXJRZ5BQSLXSOIOSUJ4MM4AGKHP4UIDUGEJKLUUX7YOGQJA6WWZA, seg_pos={0 0}, upload=5911]: Done uploading piece. err: <nil> Jul 27 04:07:17.981907539 (6h15m33.762840911s) [noise=true , address=104.36.82.56:4244 , piece_id=LQCZ5W54NBHWAVBVYK7HYSSJDC73E6FV4CP4YIM4L5H2N3OLOZXA, seg_pos={0 0}, upload=5911]: Done uploading piece. err: <nil> Jul 27 04:07:17.982370496 (6h15m33.7633039s) [noise=true , address=104.36.82.55:4204 , piece_id=72WISGRHMOAZDV5EDGTBOMLBL4HDOF3CBEMQAAIRGILPC7J5JKAQ, seg_pos={0 0}, upload=5911]: Done uploading piece. err: <nil> Jul 27 04:07:18.009548506 (6h15m33.790481886s) [noise=true , address=104.36.82.52:4098 , piece_id=M2ETF25QQ53RRGFPUKXJRINIWV7AWXSECEDR2AIM645IED72NURQ, seg_pos={0 0}, upload=5911]: Done uploading piece. err: <nil> Jul 27 04:07:18.023770598 (6h15m33.804703983s) [noise=true , address=104.36.82.40:4954 , piece_id=YLOHTJLRAED66OMEQYPZHNNRI77V6WOR4EHTEZOOEHMGZHJ2O3QQ, seg_pos={0 0}, upload=5911]: Done uploading piece. err: <nil> Jul 27 04:07:18.139745581 (6h15m33.92067895s) [noise=true , address=104.36.82.22:4534 , piece_id=CW4AQT3XUGA4A5OC6NUY4NYOOAMLN5FFJJN5PQBYEON7HPD5LCYQ, seg_pos={0 0}, upload=5911]: Done uploading piece. err: <nil> Jul 27 04:07:18.423748140 (6h15m34.204681511s) [noise=true , address=64.49.13.140:4508 , piece_id=MZ52U75VOIK6Q7QFXMMYRRG7JXFWDFU2JWQFYPERUCMH3BUX6XPQ, seg_pos={0 0}, upload=5911]: Done uploading piece. err: <nil> Jul 27 04:08:16.739940381 (6h16m32.52087375s) [noise=true , address=104.36.82.54:4172 , piece_id=ZNA53MWDRWPUH6FUR4DJERBDT2IOYGA3IYUADKXXIOLQJLQOXE7Q, seg_pos={0 0}, upload=5911]: Done uploading piece. err: <nil> Jul 27 04:08:17.177295397 (6h16m32.958228766s) [noise=true , address=104.36.82.8:4002 , piece_id=6XSPWQC3US355PS6IQ4K3IYNQZH5MAT4IVIPGSYRUM3XYPSCWE6Q, seg_pos={0 0}, upload=5911]: Done uploading piece. err: <nil> Jul 27 04:08:17.534296584 (6h16m33.315229953s) [noise=true , address=104.36.82.36:4650 , piece_id=BKJULWGPESBINNTSUYP6E4KZRXKPACVU5I7DOSZBO4YMK2V26HEQ, seg_pos={0 0}, upload=5911]: Done uploading piece. err: <nil> Jul 27 04:08:17.677851566 (6h16m33.458784943s) [noise=true , address=104.36.82.14:4248 , piece_id=LDSGCUM7Q2VX5CDFI4JRGEEG7D5H5Q5Z5WUWKU277VN2JFRX6UGA, seg_pos={0 0}, upload=5911]: Done uploading piece. err: ecclient: upload failed (node:1u6ir2PLqDa9xGm64cuqq1RqY4qeBa1fqCo9VghspBf54XEXqn, address:104.36.82.14:4248): protocol: rpc: dial tcp 104.36.82.14:4248: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 04:08:17.678115170 (6h16m33.459048549s) [noise=true , address=104.36.82.46:4496 , piece_id=GWNQR5WRGRRQBLSGWJFD3O3UNCMNE62JQNPIV5SYKBUIL3BQKA5A, seg_pos={0 0}, upload=5911]: Done uploading piece. err: ecclient: upload failed (node:12V8XhSWkyGUVYFR8MykifFAP1YAfGCwzqucDcMMfzw3xsVW77Z, address:104.36.82.46:4496): protocol: read tcp 159.223.143.39:52800->104.36.82.46:4496: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 04:08:17.678144409 (6h16m33.459077789s) [noise=true , address=104.36.82.44:4901 , piece_id=N4YLCLR2QZSULUL2XWONEKBFUMNOU7KUX2QEOK4SSDFSCI7IL6LA, seg_pos={0 0}, upload=5911]: Done uploading piece. err: ecclient: upload failed (node:1FHE84zvBb1MKwB1Ht7puTafgQxDEvPt4mYqkYVSSufAMpWQA1, address:104.36.82.44:4901): protocol: rpc: dial tcp 104.36.82.44:4901: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 04:08:17.678229622 (6h16m33.459162988s) [noise=true , address=104.36.82.44:4893 , piece_id=HUJICRJ6W54O3AZZR6H3KG6E4TGBTNS46O6XVMQM3PFAZE7A2HZA, seg_pos={0 0}, upload=5911]: Done uploading piece. err: ecclient: upload failed (node:17VMFZ2T8jrSJVtchciJ9SnyrR6JstvcMoXrN5Q3zTD1aQU1oJ, address:104.36.82.44:4893): protocol: write tcp 159.223.143.39:35302->104.36.82.44:4893: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 04:08:17.678310218 (6h16m33.459243583s) [noise=true , address=104.36.82.42:4829 , piece_id=ZLWJFP3LGX3YLXBJYIR66DXTYEVDBKXKVXCQSF2PCYCYJBIEJGMA, seg_pos={0 0}, upload=5911]: Done uploading piece. err: ecclient: upload failed (node:1F5T8Vt1x4nXFaSenvGBCYjxhRL1H1sxRvzLjW3CqzwDVhq39F, address:104.36.82.42:4829): protocol: rpc: dial tcp 104.36.82.42:4829: i/o timeout; conn closed: op; piecestore: piecestore close: context deadline exceeded Jul 27 04:08:17.678412989 (6h16m33.459346365s) [noise=true , address=104.36.82.38:4722 , piece_id=B2MBQY5CTMEI6WSE7FTLOV5RUJ22SCNG7N2EEJILEATGJ4TIALXA, seg_pos={0 0}, upload=5911]: Done uploading piece. err: ecclient: upload failed (node:12Zw3dtUg9gARz52KhwDLTjcqC3CNm7dpPzpdx7P7PbL1jwe8uc, address:104.36.82.38:4722): protocol: read tcp 159.223.143.39:56048->104.36.82.38:4722: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 04:08:17.678455247 (6h16m33.459388624s) [noise=true , address=104.36.82.47:4524 , piece_id=DONJFTKUQ2XCW6BOU7AVOKJ7XMDMBPGQVUPRANH7X2YS7PJXUKIA, seg_pos={0 0}, upload=5911]: Done uploading piece. err: ecclient: upload failed (node:1xfbCzjSwFDRrR1gPggeN8tNCLbRRWDVzCJ6mqhZsbVSe8d4s4, address:104.36.82.47:4524): protocol: read tcp 159.223.143.39:46694->104.36.82.47:4524: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 04:08:17.678510668 (6h16m33.459444052s) [seg_pos={0 0}, upload=5911]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Jul 27 04:08:17.706442864 (6h16m33.487376232s) [upload=5911]: Done uploading segments... Jul 27 04:08:17.706495329 (6h16m33.487428705s) [upload=5911]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 7.492375575s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............................................X.............] duration: 412.106368ms FAIL (2025-07-27 04:25:31 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 12) Duration: 412.152365ms core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............................X.............................] duration: 434.571776ms FAIL (2025-07-27 04:25:15 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== ece_id=NYOFDEZ7CV4PHGQ6JZMEEVAYGJRPYG6IBW7MYJO42HMWOJM7PHZA, seg_pos={0 0}, upload=5879]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12crcXpNe9NgeXTsUpooLSBgrSUFbRxbSbanLa39w2Hr5mGFww): protocol: read tcp 159.223.143.39:53630->104.36.82.5:4840: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 04:05:32.508904485 (6h13m48.289837866s) [seg_pos={0 0}, upload=5879]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 27 04:05:32.509028070 (6h13m48.289961457s) [upload=5879]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 27 04:05:32.509034645 (6h13m48.289968129s) [upload=5879]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 27 04:05:32.581834839 (6h13m48.362768225s) [upload=5879]: Done uploading segments... Jul 27 04:05:32.581875919 (6h13m48.362809298s) [upload=5879]: Upload finished. err: <nil> Jul 27 04:06:23.829155331 (6h14m39.610088696s) [upload=5895]: Starting upload Jul 27 04:06:23.829225021 (6h14m39.610158398s) [upload=5895]: Done starting upload Jul 27 04:06:23.829294606 (6h14m39.61022799s) [upload=5895]: Uploading segments... Jul 27 04:06:23.829577656 (6h14m39.610511053s) [upload=5895]: Got next segment. Inline: false Jul 27 04:06:23.829649664 (6h14m39.610583027s) [upload=5895]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 27 04:06:23.829657065 (6h14m39.61059043s) [upload=5895]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 27 04:06:23.861405530 (6h14m39.642338913s) [seg_pos={0 0}, upload=5895]: Begin upload segment... Jul 27 04:06:23.861476351 (6h14m39.642409713s) [seg_pos={0 0}, upload=5895]: Done begin upload segment. Jul 27 04:06:23.861509120 (6h14m39.642442506s) [noise=true , address=77.90.31.19:4843 , piece_id=YML5RDQIK432Z4FSE4DKXIV5U47LQBEZRFMAVLANUSGFUQR23UWQ, seg_pos={0 0}, upload=5895]: Uploading piece... Jul 27 04:06:23.861953723 (6h14m39.6428871s) [noise=true , address=45.91.116.10:4081 , piece_id=XGDTSEZOUNRTEVWHTO3DHTZMH2VZZOPJPEKLCFAHTXESLLYABHSA, seg_pos={0 0}, upload=5895]: Uploading piece... Jul 27 04:06:23.862060983 (6h14m39.642994358s) [noise=true , address=45.91.116.9:4262 , piece_id=M5ETD45HH2XZD6HDLTJ3JO2BOOIROXFWKVPTN42PSQEWG4F7KAAQ, seg_pos={0 0}, upload=5895]: Uploading piece... Jul 27 04:06:23.862759464 (6h14m39.64369284s) [noise=true , address=104.36.82.4:4804 , piece_id=IU6FDNVB5AUPA7MJ6RSXUTLBETPHOFGNJ7YGWBFIZSQFSO6W6BBA, seg_pos={0 0}, upload=5895]: Uploading piece... Jul 27 04:06:23.862835086 (6h14m39.643768461s) [noise=true , address=104.36.82.6:4873 , piece_id=YP4VJORZZUL5EVCH3P7LD3LNVMHB7CWRTKHGIA7UROXE44N6MMZA, seg_pos={0 0}, upload=5895]: Uploading piece... Jul 27 04:06:23.863593880 (6h14m39.644527258s) [noise=true , address=77.90.31.15:4697 , piece_id=Q76TJDSZPSYZSJQNFVOI462U6DFILMH2RV3LQISODXSOCXGRKW6Q, seg_pos={0 0}, upload=5895]: Uploading piece... Jul 27 04:06:23.863656891 (6h14m39.644590266s) [noise=true , address=45.91.116.7:4011 , piece_id=XMH6MCCHZBBRQWHJFWLRGYHLQD4KUGSGL7MMZSKA5TZJCND3DUDA, seg_pos={0 0}, upload=5895]: Uploading piece... Jul 27 04:06:23.864408612 (6h14m39.645341987s) [noise=true , address=45.91.116.6:4973 , piece_id=J3VULPU24CIM64VEVE3UHO62HPN6TJXGZU6BCJIS5JQJKHJK2LIQ, seg_pos={0 0}, upload=5895]: Uploading piece... Jul 27 04:06:23.864521549 (6h14m39.64545491s) [noise=true , address=104.36.82.10:4079 , piece_id=2SH5D6JOMLFR7VSN7ZY5CLMR5RHIRFYQ7Z5I2JIYTJQM43G4O6OQ, seg_pos={0 0}, upload=5895]: Uploading piece... Jul 27 04:06:23.865518491 (6h14m39.646451872s) [noise=true , address=77.90.31.17:4771 , piece_id=U2LF22GKCMKRZN6EWCF26ITDZQHBFZXCGSQXT33MCX6P57IGI35Q, seg_pos={0 0}, upload=5895]: Uploading piece... Jul 27 04:06:23.865654950 (6h14m39.646588325s) [noise=true , address=45.91.116.8:4048 , piece_id=5REWMVVTBVLDQXQ5KAEHLY4VJL5SGF7ONQWDQQ3N2FDU3OAO6AZA, seg_pos={0 0}, upload=5895]: Uploading piece... Jul 27 04:06:23.866190892 (6h14m39.647124271s) [noise=true , address=104.36.82.12:4151 , piece_id=E2TEUTQBD4Y4AUHSCPMCIFEXBSY7S5HJWBKNKWWTUWEW2BLAAE6Q, seg_pos={0 0}, upload=5895]: Uploading piece... Jul 27 04:06:23.866412590 (6h14m39.647345968s) [noise=true , address=104.36.82.11:4115 , piece_id=YXMNVT6ZHGESGQ2BXKZUVR7U2CNNAI2PO7ZTMDG5UIOV7HTDDUTA, seg_pos={0 0}, upload=5895]: Uploading piece... Jul 27 04:06:23.866715972 (6h14m39.647649359s) [noise=true , address=45.91.116.13:4153 , piece_id=H4535KOCG3F4RKA6AKKMTJSOM6Q6PGRC6M7FR3UIS3BORTSP5UQA, seg_pos={0 0}, upload=5895]: Uploading piece... Jul 27 04:06:23.866998965 (6h14m39.64793234s) [noise=true , address=104.36.82.13:4186 , piece_id=TBLDKALGJYBGR5NLCOCKJ4QKEFNIJXCKAFUBD7VR3UMNWEMKMKHQ, seg_pos={0 0}, upload=5895]: Uploading piece... Jul 27 04:06:23.867230593 (6h14m39.648163974s) [noise=true , address=104.36.82.16:4293 , piece_id=AHH6PRFUSGTDGB7H52DRITARA4IF6GFLVSRYS2Y6EPJCZSKUOYWQ, seg_pos={0 0}, upload=5895]: Uploading piece... Jul 27 04:06:23.949422808 (6h14m39.730356187s) [upload=5895]: Next returned nil segment Jul 27 04:06:23.949442576 (6h14m39.73037596s) [upload=5895]: Waiting for error group managing segments... Jul 27 04:06:23.953497426 (6h14m39.734430802s) [noise=true , address=45.91.116.10:4081 , piece_id=XGDTSEZOUNRTEVWHTO3DHTZMH2VZZOPJPEKLCFAHTXESLLYABHSA, seg_pos={0 0}, upload=5895]: Done uploading piece. err: <nil> Jul 27 04:06:23.953531796 (6h14m39.734465159s) [noise=true , address=45.91.116.6:4973 , piece_id=J3VULPU24CIM64VEVE3UHO62HPN6TJXGZU6BCJIS5JQJKHJK2LIQ, seg_pos={0 0}, upload=5895]: Done uploading piece. err: <nil> Jul 27 04:06:23.954068660 (6h14m39.735002029s) [noise=true , address=45.91.116.9:4262 , piece_id=M5ETD45HH2XZD6HDLTJ3JO2BOOIROXFWKVPTN42PSQEWG4F7KAAQ, seg_pos={0 0}, upload=5895]: Done uploading piece. err: <nil> Jul 27 04:06:23.954194339 (6h14m39.735127707s) [noise=true , address=45.91.116.8:4048 , piece_id=5REWMVVTBVLDQXQ5KAEHLY4VJL5SGF7ONQWDQQ3N2FDU3OAO6AZA, seg_pos={0 0}, upload=5895]: Done uploading piece. err: <nil> Jul 27 04:06:23.954601730 (6h14m39.735535112s) [noise=true , address=45.91.116.7:4011 , piece_id=XMH6MCCHZBBRQWHJFWLRGYHLQD4KUGSGL7MMZSKA5TZJCND3DUDA, seg_pos={0 0}, upload=5895]: Done uploading piece. err: <nil> Jul 27 04:06:23.954755872 (6h14m39.735689237s) [noise=true , address=45.91.116.13:4153 , piece_id=H4535KOCG3F4RKA6AKKMTJSOM6Q6PGRC6M7FR3UIS3BORTSP5UQA, seg_pos={0 0}, upload=5895]: Done uploading piece. err: <nil> Jul 27 04:06:23.955611883 (6h14m39.73654526s) [noise=true , address=77.90.31.19:4843 , piece_id=YML5RDQIK432Z4FSE4DKXIV5U47LQBEZRFMAVLANUSGFUQR23UWQ, seg_pos={0 0}, upload=5895]: Done uploading piece. err: <nil> Jul 27 04:06:23.956373391 (6h14m39.737306756s) [noise=true , address=77.90.31.15:4697 , piece_id=Q76TJDSZPSYZSJQNFVOI462U6DFILMH2RV3LQISODXSOCXGRKW6Q, seg_pos={0 0}, upload=5895]: Done uploading piece. err: <nil> Jul 27 04:06:23.958659122 (6h14m39.73959249s) [noise=true , address=77.90.31.17:4771 , piece_id=U2LF22GKCMKRZN6EWCF26ITDZQHBFZXCGSQXT33MCX6P57IGI35Q, seg_pos={0 0}, upload=5895]: Done uploading piece. err: <nil> Jul 27 04:06:35.058944013 (6h14m50.83987739s) [noise=true , address=104.36.82.16:4293 , piece_id=AHH6PRFUSGTDGB7H52DRITARA4IF6GFLVSRYS2Y6EPJCZSKUOYWQ, seg_pos={0 0}, upload=5895]: Done uploading piece. err: <nil> Jul 27 04:06:41.103204002 (6h14m56.884137386s) [noise=true , address=104.36.82.4:4804 , piece_id=IU6FDNVB5AUPA7MJ6RSXUTLBETPHOFGNJ7YGWBFIZSQFSO6W6BBA, seg_pos={0 0}, upload=5895]: Done uploading piece. err: <nil> Jul 27 04:06:43.191702285 (6h14m58.972635651s) [noise=true , address=104.36.82.6:4873 , piece_id=YP4VJORZZUL5EVCH3P7LD3LNVMHB7CWRTKHGIA7UROXE44N6MMZA, seg_pos={0 0}, upload=5895]: Done uploading piece. err: <nil> Jul 27 04:06:53.777973493 (6h15m9.558906861s) [noise=true , address=104.36.82.11:4115 , piece_id=YXMNVT6ZHGESGQ2BXKZUVR7U2CNNAI2PO7ZTMDG5UIOV7HTDDUTA, seg_pos={0 0}, upload=5895]: Done uploading piece. err: ecclient: upload failed (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR, address:104.36.82.11:4115): protocol: write tcp 159.223.143.39:34210->104.36.82.11:4115: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 04:06:53.778019532 (6h15m9.558952909s) [noise=true , address=104.36.82.12:4151 , piece_id=E2TEUTQBD4Y4AUHSCPMCIFEXBSY7S5HJWBKNKWWTUWEW2BLAAE6Q, seg_pos={0 0}, upload=5895]: Done uploading piece. err: ecclient: upload failed (node:1w6pwZMdeQKjt92qx6WbTWU5VPYJto5gS87f6iKvCBZH6LfyeB, address:104.36.82.12:4151): protocol: rpc: dial tcp 104.36.82.12:4151: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 04:06:53.778043653 (6h15m9.558977021s) [noise=true , address=104.36.82.13:4186 , piece_id=TBLDKALGJYBGR5NLCOCKJ4QKEFNIJXCKAFUBD7VR3UMNWEMKMKHQ, seg_pos={0 0}, upload=5895]: Done uploading piece. err: ecclient: upload failed (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht, address:104.36.82.13:4186): protocol: read tcp 159.223.143.39:34340->104.36.82.13:4186: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 04:06:53.778095228 (6h15m9.559028593s) [noise=true , address=104.36.82.10:4079 , piece_id=2SH5D6JOMLFR7VSN7ZY5CLMR5RHIRFYQ7Z5I2JIYTJQM43G4O6OQ, seg_pos={0 0}, upload=5895]: Done uploading piece. err: ecclient: upload failed (node:12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud, address:104.36.82.10:4079): protocol: write tcp 159.223.143.39:53840->104.36.82.10:4079: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 04:06:53.778142167 (6h15m9.559075532s) [seg_pos={0 0}, upload=5895]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 27 04:06:53.837032161 (6h15m9.617965543s) [upload=5895]: Done uploading segments... Jul 27 04:06:53.837055996 (6h15m9.617989366s) [upload=5895]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 434.604699ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.....................................X......................] duration: 28.425697141s FAIL (2025-07-27 04:25:12 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== {0 0}, upload=5911]: Begin upload segment... Jul 27 04:07:17.745307658 (6h15m33.526241043s) [seg_pos={0 0}, upload=5911]: Done begin upload segment. Jul 27 04:07:17.745349241 (6h15m33.526282625s) [noise=true , address=104.36.82.52:4098 , piece_id=M2ETF25QQ53RRGFPUKXJRINIWV7AWXSECEDR2AIM645IED72NURQ, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.745439694 (6h15m33.526373073s) [noise=true , address=104.36.82.56:4244 , piece_id=LQCZ5W54NBHWAVBVYK7HYSSJDC73E6FV4CP4YIM4L5H2N3OLOZXA, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.745972271 (6h15m33.526905654s) [noise=true , address=104.36.82.36:4650 , piece_id=BKJULWGPESBINNTSUYP6E4KZRXKPACVU5I7DOSZBO4YMK2V26HEQ, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.745980243 (6h15m33.526913621s) [noise=true , address=104.36.82.38:4722 , piece_id=B2MBQY5CTMEI6WSE7FTLOV5RUJ22SCNG7N2EEJILEATGJ4TIALXA, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.746403818 (6h15m33.527337198s) [noise=true , address=104.36.82.22:4534 , piece_id=CW4AQT3XUGA4A5OC6NUY4NYOOAMLN5FFJJN5PQBYEON7HPD5LCYQ, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.746455493 (6h15m33.527388875s) [noise=true , address=104.36.82.14:4248 , piece_id=LDSGCUM7Q2VX5CDFI4JRGEEG7D5H5Q5Z5WUWKU277VN2JFRX6UGA, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.746872127 (6h15m33.527805505s) [noise=true , address=104.36.82.40:4954 , piece_id=YLOHTJLRAED66OMEQYPZHNNRI77V6WOR4EHTEZOOEHMGZHJ2O3QQ, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.747123162 (6h15m33.52805654s) [noise=true , address=104.36.82.46:4496 , piece_id=GWNQR5WRGRRQBLSGWJFD3O3UNCMNE62JQNPIV5SYKBUIL3BQKA5A, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.747280678 (6h15m33.528214053s) [noise=true , address=104.36.82.47:4524 , piece_id=DONJFTKUQ2XCW6BOU7AVOKJ7XMDMBPGQVUPRANH7X2YS7PJXUKIA, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.747736400 (6h15m33.528669779s) [noise=true , address=104.36.82.44:4893 , piece_id=HUJICRJ6W54O3AZZR6H3KG6E4TGBTNS46O6XVMQM3PFAZE7A2HZA, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.747755734 (6h15m33.528689109s) [noise=true , address=104.36.82.55:4204 , piece_id=72WISGRHMOAZDV5EDGTBOMLBL4HDOF3CBEMQAAIRGILPC7J5JKAQ, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.748313124 (6h15m33.5292465s) [noise=true , address=104.36.82.58:4287 , piece_id=BVDYUEWU7TVEQDMZXX5SJNS4QICVKMTGFLCUV6P2YHOE7TI25VXA, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.748485217 (6h15m33.529418594s) [noise=true , address=104.36.82.53:4123 , piece_id=NXJRZ5BQSLXSOIOSUJ4MM4AGKHP4UIDUGEJKLUUX7YOGQJA6WWZA, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.748831329 (6h15m33.529764704s) [noise=true , address=104.36.82.42:4829 , piece_id=ZLWJFP3LGX3YLXBJYIR66DXTYEVDBKXKVXCQSF2PCYCYJBIEJGMA, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.749299293 (6h15m33.530232669s) [noise=true , address=104.36.82.8:4002 , piece_id=6XSPWQC3US355PS6IQ4K3IYNQZH5MAT4IVIPGSYRUM3XYPSCWE6Q, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.749790974 (6h15m33.53072436s) [noise=true , address=104.36.82.44:4901 , piece_id=N4YLCLR2QZSULUL2XWONEKBFUMNOU7KUX2QEOK4SSDFSCI7IL6LA, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.750478221 (6h15m33.531411597s) [noise=true , address=104.36.82.54:4172 , piece_id=ZNA53MWDRWPUH6FUR4DJERBDT2IOYGA3IYUADKXXIOLQJLQOXE7Q, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.752884255 (6h15m33.533817625s) [noise=true , address=64.49.13.140:4508 , piece_id=MZ52U75VOIK6Q7QFXMMYRRG7JXFWDFU2JWQFYPERUCMH3BUX6XPQ, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.879839555 (6h15m33.660772917s) [upload=5911]: Next returned nil segment Jul 27 04:07:17.879861389 (6h15m33.660794765s) [upload=5911]: Waiting for error group managing segments... Jul 27 04:07:17.888652438 (6h15m33.669585811s) [noise=true , address=104.36.82.58:4287 , piece_id=BVDYUEWU7TVEQDMZXX5SJNS4QICVKMTGFLCUV6P2YHOE7TI25VXA, seg_pos={0 0}, upload=5911]: Done uploading piece. err: <nil> Jul 27 04:07:17.952192045 (6h15m33.733125429s) [noise=true , address=104.36.82.53:4123 , piece_id=NXJRZ5BQSLXSOIOSUJ4MM4AGKHP4UIDUGEJKLUUX7YOGQJA6WWZA, seg_pos={0 0}, upload=5911]: Done uploading piece. err: <nil> Jul 27 04:07:17.981907539 (6h15m33.762840911s) [noise=true , address=104.36.82.56:4244 , piece_id=LQCZ5W54NBHWAVBVYK7HYSSJDC73E6FV4CP4YIM4L5H2N3OLOZXA, seg_pos={0 0}, upload=5911]: Done uploading piece. err: <nil> Jul 27 04:07:17.982370496 (6h15m33.7633039s) [noise=true , address=104.36.82.55:4204 , piece_id=72WISGRHMOAZDV5EDGTBOMLBL4HDOF3CBEMQAAIRGILPC7J5JKAQ, seg_pos={0 0}, upload=5911]: Done uploading piece. err: <nil> Jul 27 04:07:18.009548506 (6h15m33.790481886s) [noise=true , address=104.36.82.52:4098 , piece_id=M2ETF25QQ53RRGFPUKXJRINIWV7AWXSECEDR2AIM645IED72NURQ, seg_pos={0 0}, upload=5911]: Done uploading piece. err: <nil> Jul 27 04:07:18.023770598 (6h15m33.804703983s) [noise=true , address=104.36.82.40:4954 , piece_id=YLOHTJLRAED66OMEQYPZHNNRI77V6WOR4EHTEZOOEHMGZHJ2O3QQ, seg_pos={0 0}, upload=5911]: Done uploading piece. err: <nil> Jul 27 04:07:18.139745581 (6h15m33.92067895s) [noise=true , address=104.36.82.22:4534 , piece_id=CW4AQT3XUGA4A5OC6NUY4NYOOAMLN5FFJJN5PQBYEON7HPD5LCYQ, seg_pos={0 0}, upload=5911]: Done uploading piece. err: <nil> Jul 27 04:07:18.423748140 (6h15m34.204681511s) [noise=true , address=64.49.13.140:4508 , piece_id=MZ52U75VOIK6Q7QFXMMYRRG7JXFWDFU2JWQFYPERUCMH3BUX6XPQ, seg_pos={0 0}, upload=5911]: Done uploading piece. err: <nil> Jul 27 04:08:16.739940381 (6h16m32.52087375s) [noise=true , address=104.36.82.54:4172 , piece_id=ZNA53MWDRWPUH6FUR4DJERBDT2IOYGA3IYUADKXXIOLQJLQOXE7Q, seg_pos={0 0}, upload=5911]: Done uploading piece. err: <nil> Jul 27 04:08:17.177295397 (6h16m32.958228766s) [noise=true , address=104.36.82.8:4002 , piece_id=6XSPWQC3US355PS6IQ4K3IYNQZH5MAT4IVIPGSYRUM3XYPSCWE6Q, seg_pos={0 0}, upload=5911]: Done uploading piece. err: <nil> Jul 27 04:08:17.534296584 (6h16m33.315229953s) [noise=true , address=104.36.82.36:4650 , piece_id=BKJULWGPESBINNTSUYP6E4KZRXKPACVU5I7DOSZBO4YMK2V26HEQ, seg_pos={0 0}, upload=5911]: Done uploading piece. err: <nil> Jul 27 04:08:17.677851566 (6h16m33.458784943s) [noise=true , address=104.36.82.14:4248 , piece_id=LDSGCUM7Q2VX5CDFI4JRGEEG7D5H5Q5Z5WUWKU277VN2JFRX6UGA, seg_pos={0 0}, upload=5911]: Done uploading piece. err: ecclient: upload failed (node:1u6ir2PLqDa9xGm64cuqq1RqY4qeBa1fqCo9VghspBf54XEXqn, address:104.36.82.14:4248): protocol: rpc: dial tcp 104.36.82.14:4248: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 04:08:17.678115170 (6h16m33.459048549s) [noise=true , address=104.36.82.46:4496 , piece_id=GWNQR5WRGRRQBLSGWJFD3O3UNCMNE62JQNPIV5SYKBUIL3BQKA5A, seg_pos={0 0}, upload=5911]: Done uploading piece. err: ecclient: upload failed (node:12V8XhSWkyGUVYFR8MykifFAP1YAfGCwzqucDcMMfzw3xsVW77Z, address:104.36.82.46:4496): protocol: read tcp 159.223.143.39:52800->104.36.82.46:4496: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 04:08:17.678144409 (6h16m33.459077789s) [noise=true , address=104.36.82.44:4901 , piece_id=N4YLCLR2QZSULUL2XWONEKBFUMNOU7KUX2QEOK4SSDFSCI7IL6LA, seg_pos={0 0}, upload=5911]: Done uploading piece. err: ecclient: upload failed (node:1FHE84zvBb1MKwB1Ht7puTafgQxDEvPt4mYqkYVSSufAMpWQA1, address:104.36.82.44:4901): protocol: rpc: dial tcp 104.36.82.44:4901: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 04:08:17.678229622 (6h16m33.459162988s) [noise=true , address=104.36.82.44:4893 , piece_id=HUJICRJ6W54O3AZZR6H3KG6E4TGBTNS46O6XVMQM3PFAZE7A2HZA, seg_pos={0 0}, upload=5911]: Done uploading piece. err: ecclient: upload failed (node:17VMFZ2T8jrSJVtchciJ9SnyrR6JstvcMoXrN5Q3zTD1aQU1oJ, address:104.36.82.44:4893): protocol: write tcp 159.223.143.39:35302->104.36.82.44:4893: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 04:08:17.678310218 (6h16m33.459243583s) [noise=true , address=104.36.82.42:4829 , piece_id=ZLWJFP3LGX3YLXBJYIR66DXTYEVDBKXKVXCQSF2PCYCYJBIEJGMA, seg_pos={0 0}, upload=5911]: Done uploading piece. err: ecclient: upload failed (node:1F5T8Vt1x4nXFaSenvGBCYjxhRL1H1sxRvzLjW3CqzwDVhq39F, address:104.36.82.42:4829): protocol: rpc: dial tcp 104.36.82.42:4829: i/o timeout; conn closed: op; piecestore: piecestore close: context deadline exceeded Jul 27 04:08:17.678412989 (6h16m33.459346365s) [noise=true , address=104.36.82.38:4722 , piece_id=B2MBQY5CTMEI6WSE7FTLOV5RUJ22SCNG7N2EEJILEATGJ4TIALXA, seg_pos={0 0}, upload=5911]: Done uploading piece. err: ecclient: upload failed (node:12Zw3dtUg9gARz52KhwDLTjcqC3CNm7dpPzpdx7P7PbL1jwe8uc, address:104.36.82.38:4722): protocol: read tcp 159.223.143.39:56048->104.36.82.38:4722: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 04:08:17.678455247 (6h16m33.459388624s) [noise=true , address=104.36.82.47:4524 , piece_id=DONJFTKUQ2XCW6BOU7AVOKJ7XMDMBPGQVUPRANH7X2YS7PJXUKIA, seg_pos={0 0}, upload=5911]: Done uploading piece. err: ecclient: upload failed (node:1xfbCzjSwFDRrR1gPggeN8tNCLbRRWDVzCJ6mqhZsbVSe8d4s4, address:104.36.82.47:4524): protocol: read tcp 159.223.143.39:46694->104.36.82.47:4524: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 04:08:17.678510668 (6h16m33.459444052s) [seg_pos={0 0}, upload=5911]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Jul 27 04:08:17.706442864 (6h16m33.487376232s) [upload=5911]: Done uploading segments... Jul 27 04:08:17.706495329 (6h16m33.487428705s) [upload=5911]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 28.425734868s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............................X............................] duration: 399.700225ms FAIL (2025-07-27 04:24:47 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== ece_id=NYOFDEZ7CV4PHGQ6JZMEEVAYGJRPYG6IBW7MYJO42HMWOJM7PHZA, seg_pos={0 0}, upload=5879]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12crcXpNe9NgeXTsUpooLSBgrSUFbRxbSbanLa39w2Hr5mGFww): protocol: read tcp 159.223.143.39:53630->104.36.82.5:4840: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 04:05:32.508904485 (6h13m48.289837866s) [seg_pos={0 0}, upload=5879]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 27 04:05:32.509028070 (6h13m48.289961457s) [upload=5879]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 27 04:05:32.509034645 (6h13m48.289968129s) [upload=5879]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 27 04:05:32.581834839 (6h13m48.362768225s) [upload=5879]: Done uploading segments... Jul 27 04:05:32.581875919 (6h13m48.362809298s) [upload=5879]: Upload finished. err: <nil> Jul 27 04:06:23.829155331 (6h14m39.610088696s) [upload=5895]: Starting upload Jul 27 04:06:23.829225021 (6h14m39.610158398s) [upload=5895]: Done starting upload Jul 27 04:06:23.829294606 (6h14m39.61022799s) [upload=5895]: Uploading segments... Jul 27 04:06:23.829577656 (6h14m39.610511053s) [upload=5895]: Got next segment. Inline: false Jul 27 04:06:23.829649664 (6h14m39.610583027s) [upload=5895]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 27 04:06:23.829657065 (6h14m39.61059043s) [upload=5895]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 27 04:06:23.861405530 (6h14m39.642338913s) [seg_pos={0 0}, upload=5895]: Begin upload segment... Jul 27 04:06:23.861476351 (6h14m39.642409713s) [seg_pos={0 0}, upload=5895]: Done begin upload segment. Jul 27 04:06:23.861509120 (6h14m39.642442506s) [noise=true , address=77.90.31.19:4843 , piece_id=YML5RDQIK432Z4FSE4DKXIV5U47LQBEZRFMAVLANUSGFUQR23UWQ, seg_pos={0 0}, upload=5895]: Uploading piece... Jul 27 04:06:23.861953723 (6h14m39.6428871s) [noise=true , address=45.91.116.10:4081 , piece_id=XGDTSEZOUNRTEVWHTO3DHTZMH2VZZOPJPEKLCFAHTXESLLYABHSA, seg_pos={0 0}, upload=5895]: Uploading piece... Jul 27 04:06:23.862060983 (6h14m39.642994358s) [noise=true , address=45.91.116.9:4262 , piece_id=M5ETD45HH2XZD6HDLTJ3JO2BOOIROXFWKVPTN42PSQEWG4F7KAAQ, seg_pos={0 0}, upload=5895]: Uploading piece... Jul 27 04:06:23.862759464 (6h14m39.64369284s) [noise=true , address=104.36.82.4:4804 , piece_id=IU6FDNVB5AUPA7MJ6RSXUTLBETPHOFGNJ7YGWBFIZSQFSO6W6BBA, seg_pos={0 0}, upload=5895]: Uploading piece... Jul 27 04:06:23.862835086 (6h14m39.643768461s) [noise=true , address=104.36.82.6:4873 , piece_id=YP4VJORZZUL5EVCH3P7LD3LNVMHB7CWRTKHGIA7UROXE44N6MMZA, seg_pos={0 0}, upload=5895]: Uploading piece... Jul 27 04:06:23.863593880 (6h14m39.644527258s) [noise=true , address=77.90.31.15:4697 , piece_id=Q76TJDSZPSYZSJQNFVOI462U6DFILMH2RV3LQISODXSOCXGRKW6Q, seg_pos={0 0}, upload=5895]: Uploading piece... Jul 27 04:06:23.863656891 (6h14m39.644590266s) [noise=true , address=45.91.116.7:4011 , piece_id=XMH6MCCHZBBRQWHJFWLRGYHLQD4KUGSGL7MMZSKA5TZJCND3DUDA, seg_pos={0 0}, upload=5895]: Uploading piece... Jul 27 04:06:23.864408612 (6h14m39.645341987s) [noise=true , address=45.91.116.6:4973 , piece_id=J3VULPU24CIM64VEVE3UHO62HPN6TJXGZU6BCJIS5JQJKHJK2LIQ, seg_pos={0 0}, upload=5895]: Uploading piece... Jul 27 04:06:23.864521549 (6h14m39.64545491s) [noise=true , address=104.36.82.10:4079 , piece_id=2SH5D6JOMLFR7VSN7ZY5CLMR5RHIRFYQ7Z5I2JIYTJQM43G4O6OQ, seg_pos={0 0}, upload=5895]: Uploading piece... Jul 27 04:06:23.865518491 (6h14m39.646451872s) [noise=true , address=77.90.31.17:4771 , piece_id=U2LF22GKCMKRZN6EWCF26ITDZQHBFZXCGSQXT33MCX6P57IGI35Q, seg_pos={0 0}, upload=5895]: Uploading piece... Jul 27 04:06:23.865654950 (6h14m39.646588325s) [noise=true , address=45.91.116.8:4048 , piece_id=5REWMVVTBVLDQXQ5KAEHLY4VJL5SGF7ONQWDQQ3N2FDU3OAO6AZA, seg_pos={0 0}, upload=5895]: Uploading piece... Jul 27 04:06:23.866190892 (6h14m39.647124271s) [noise=true , address=104.36.82.12:4151 , piece_id=E2TEUTQBD4Y4AUHSCPMCIFEXBSY7S5HJWBKNKWWTUWEW2BLAAE6Q, seg_pos={0 0}, upload=5895]: Uploading piece... Jul 27 04:06:23.866412590 (6h14m39.647345968s) [noise=true , address=104.36.82.11:4115 , piece_id=YXMNVT6ZHGESGQ2BXKZUVR7U2CNNAI2PO7ZTMDG5UIOV7HTDDUTA, seg_pos={0 0}, upload=5895]: Uploading piece... Jul 27 04:06:23.866715972 (6h14m39.647649359s) [noise=true , address=45.91.116.13:4153 , piece_id=H4535KOCG3F4RKA6AKKMTJSOM6Q6PGRC6M7FR3UIS3BORTSP5UQA, seg_pos={0 0}, upload=5895]: Uploading piece... Jul 27 04:06:23.866998965 (6h14m39.64793234s) [noise=true , address=104.36.82.13:4186 , piece_id=TBLDKALGJYBGR5NLCOCKJ4QKEFNIJXCKAFUBD7VR3UMNWEMKMKHQ, seg_pos={0 0}, upload=5895]: Uploading piece... Jul 27 04:06:23.867230593 (6h14m39.648163974s) [noise=true , address=104.36.82.16:4293 , piece_id=AHH6PRFUSGTDGB7H52DRITARA4IF6GFLVSRYS2Y6EPJCZSKUOYWQ, seg_pos={0 0}, upload=5895]: Uploading piece... Jul 27 04:06:23.949422808 (6h14m39.730356187s) [upload=5895]: Next returned nil segment Jul 27 04:06:23.949442576 (6h14m39.73037596s) [upload=5895]: Waiting for error group managing segments... Jul 27 04:06:23.953497426 (6h14m39.734430802s) [noise=true , address=45.91.116.10:4081 , piece_id=XGDTSEZOUNRTEVWHTO3DHTZMH2VZZOPJPEKLCFAHTXESLLYABHSA, seg_pos={0 0}, upload=5895]: Done uploading piece. err: <nil> Jul 27 04:06:23.953531796 (6h14m39.734465159s) [noise=true , address=45.91.116.6:4973 , piece_id=J3VULPU24CIM64VEVE3UHO62HPN6TJXGZU6BCJIS5JQJKHJK2LIQ, seg_pos={0 0}, upload=5895]: Done uploading piece. err: <nil> Jul 27 04:06:23.954068660 (6h14m39.735002029s) [noise=true , address=45.91.116.9:4262 , piece_id=M5ETD45HH2XZD6HDLTJ3JO2BOOIROXFWKVPTN42PSQEWG4F7KAAQ, seg_pos={0 0}, upload=5895]: Done uploading piece. err: <nil> Jul 27 04:06:23.954194339 (6h14m39.735127707s) [noise=true , address=45.91.116.8:4048 , piece_id=5REWMVVTBVLDQXQ5KAEHLY4VJL5SGF7ONQWDQQ3N2FDU3OAO6AZA, seg_pos={0 0}, upload=5895]: Done uploading piece. err: <nil> Jul 27 04:06:23.954601730 (6h14m39.735535112s) [noise=true , address=45.91.116.7:4011 , piece_id=XMH6MCCHZBBRQWHJFWLRGYHLQD4KUGSGL7MMZSKA5TZJCND3DUDA, seg_pos={0 0}, upload=5895]: Done uploading piece. err: <nil> Jul 27 04:06:23.954755872 (6h14m39.735689237s) [noise=true , address=45.91.116.13:4153 , piece_id=H4535KOCG3F4RKA6AKKMTJSOM6Q6PGRC6M7FR3UIS3BORTSP5UQA, seg_pos={0 0}, upload=5895]: Done uploading piece. err: <nil> Jul 27 04:06:23.955611883 (6h14m39.73654526s) [noise=true , address=77.90.31.19:4843 , piece_id=YML5RDQIK432Z4FSE4DKXIV5U47LQBEZRFMAVLANUSGFUQR23UWQ, seg_pos={0 0}, upload=5895]: Done uploading piece. err: <nil> Jul 27 04:06:23.956373391 (6h14m39.737306756s) [noise=true , address=77.90.31.15:4697 , piece_id=Q76TJDSZPSYZSJQNFVOI462U6DFILMH2RV3LQISODXSOCXGRKW6Q, seg_pos={0 0}, upload=5895]: Done uploading piece. err: <nil> Jul 27 04:06:23.958659122 (6h14m39.73959249s) [noise=true , address=77.90.31.17:4771 , piece_id=U2LF22GKCMKRZN6EWCF26ITDZQHBFZXCGSQXT33MCX6P57IGI35Q, seg_pos={0 0}, upload=5895]: Done uploading piece. err: <nil> Jul 27 04:06:35.058944013 (6h14m50.83987739s) [noise=true , address=104.36.82.16:4293 , piece_id=AHH6PRFUSGTDGB7H52DRITARA4IF6GFLVSRYS2Y6EPJCZSKUOYWQ, seg_pos={0 0}, upload=5895]: Done uploading piece. err: <nil> Jul 27 04:06:41.103204002 (6h14m56.884137386s) [noise=true , address=104.36.82.4:4804 , piece_id=IU6FDNVB5AUPA7MJ6RSXUTLBETPHOFGNJ7YGWBFIZSQFSO6W6BBA, seg_pos={0 0}, upload=5895]: Done uploading piece. err: <nil> Jul 27 04:06:43.191702285 (6h14m58.972635651s) [noise=true , address=104.36.82.6:4873 , piece_id=YP4VJORZZUL5EVCH3P7LD3LNVMHB7CWRTKHGIA7UROXE44N6MMZA, seg_pos={0 0}, upload=5895]: Done uploading piece. err: <nil> Jul 27 04:06:53.777973493 (6h15m9.558906861s) [noise=true , address=104.36.82.11:4115 , piece_id=YXMNVT6ZHGESGQ2BXKZUVR7U2CNNAI2PO7ZTMDG5UIOV7HTDDUTA, seg_pos={0 0}, upload=5895]: Done uploading piece. err: ecclient: upload failed (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR, address:104.36.82.11:4115): protocol: write tcp 159.223.143.39:34210->104.36.82.11:4115: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 04:06:53.778019532 (6h15m9.558952909s) [noise=true , address=104.36.82.12:4151 , piece_id=E2TEUTQBD4Y4AUHSCPMCIFEXBSY7S5HJWBKNKWWTUWEW2BLAAE6Q, seg_pos={0 0}, upload=5895]: Done uploading piece. err: ecclient: upload failed (node:1w6pwZMdeQKjt92qx6WbTWU5VPYJto5gS87f6iKvCBZH6LfyeB, address:104.36.82.12:4151): protocol: rpc: dial tcp 104.36.82.12:4151: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 04:06:53.778043653 (6h15m9.558977021s) [noise=true , address=104.36.82.13:4186 , piece_id=TBLDKALGJYBGR5NLCOCKJ4QKEFNIJXCKAFUBD7VR3UMNWEMKMKHQ, seg_pos={0 0}, upload=5895]: Done uploading piece. err: ecclient: upload failed (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht, address:104.36.82.13:4186): protocol: read tcp 159.223.143.39:34340->104.36.82.13:4186: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 04:06:53.778095228 (6h15m9.559028593s) [noise=true , address=104.36.82.10:4079 , piece_id=2SH5D6JOMLFR7VSN7ZY5CLMR5RHIRFYQ7Z5I2JIYTJQM43G4O6OQ, seg_pos={0 0}, upload=5895]: Done uploading piece. err: ecclient: upload failed (node:12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud, address:104.36.82.10:4079): protocol: write tcp 159.223.143.39:53840->104.36.82.10:4079: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 04:06:53.778142167 (6h15m9.559075532s) [seg_pos={0 0}, upload=5895]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 27 04:06:53.837032161 (6h15m9.617965543s) [upload=5895]: Done uploading segments... Jul 27 04:06:53.837055996 (6h15m9.617989366s) [upload=5895]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 399.737722ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................................X...........................] duration: 650.502133ms FAIL (2025-07-27 04:24:24 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== ece_id=NYOFDEZ7CV4PHGQ6JZMEEVAYGJRPYG6IBW7MYJO42HMWOJM7PHZA, seg_pos={0 0}, upload=5879]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12crcXpNe9NgeXTsUpooLSBgrSUFbRxbSbanLa39w2Hr5mGFww): protocol: read tcp 159.223.143.39:53630->104.36.82.5:4840: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 04:05:32.508904485 (6h13m48.289837866s) [seg_pos={0 0}, upload=5879]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 27 04:05:32.509028070 (6h13m48.289961457s) [upload=5879]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 27 04:05:32.509034645 (6h13m48.289968129s) [upload=5879]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 27 04:05:32.581834839 (6h13m48.362768225s) [upload=5879]: Done uploading segments... Jul 27 04:05:32.581875919 (6h13m48.362809298s) [upload=5879]: Upload finished. err: <nil> Jul 27 04:06:23.829155331 (6h14m39.610088696s) [upload=5895]: Starting upload Jul 27 04:06:23.829225021 (6h14m39.610158398s) [upload=5895]: Done starting upload Jul 27 04:06:23.829294606 (6h14m39.61022799s) [upload=5895]: Uploading segments... Jul 27 04:06:23.829577656 (6h14m39.610511053s) [upload=5895]: Got next segment. Inline: false Jul 27 04:06:23.829649664 (6h14m39.610583027s) [upload=5895]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 27 04:06:23.829657065 (6h14m39.61059043s) [upload=5895]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 27 04:06:23.861405530 (6h14m39.642338913s) [seg_pos={0 0}, upload=5895]: Begin upload segment... Jul 27 04:06:23.861476351 (6h14m39.642409713s) [seg_pos={0 0}, upload=5895]: Done begin upload segment. Jul 27 04:06:23.861509120 (6h14m39.642442506s) [noise=true , address=77.90.31.19:4843 , piece_id=YML5RDQIK432Z4FSE4DKXIV5U47LQBEZRFMAVLANUSGFUQR23UWQ, seg_pos={0 0}, upload=5895]: Uploading piece... Jul 27 04:06:23.861953723 (6h14m39.6428871s) [noise=true , address=45.91.116.10:4081 , piece_id=XGDTSEZOUNRTEVWHTO3DHTZMH2VZZOPJPEKLCFAHTXESLLYABHSA, seg_pos={0 0}, upload=5895]: Uploading piece... Jul 27 04:06:23.862060983 (6h14m39.642994358s) [noise=true , address=45.91.116.9:4262 , piece_id=M5ETD45HH2XZD6HDLTJ3JO2BOOIROXFWKVPTN42PSQEWG4F7KAAQ, seg_pos={0 0}, upload=5895]: Uploading piece... Jul 27 04:06:23.862759464 (6h14m39.64369284s) [noise=true , address=104.36.82.4:4804 , piece_id=IU6FDNVB5AUPA7MJ6RSXUTLBETPHOFGNJ7YGWBFIZSQFSO6W6BBA, seg_pos={0 0}, upload=5895]: Uploading piece... Jul 27 04:06:23.862835086 (6h14m39.643768461s) [noise=true , address=104.36.82.6:4873 , piece_id=YP4VJORZZUL5EVCH3P7LD3LNVMHB7CWRTKHGIA7UROXE44N6MMZA, seg_pos={0 0}, upload=5895]: Uploading piece... Jul 27 04:06:23.863593880 (6h14m39.644527258s) [noise=true , address=77.90.31.15:4697 , piece_id=Q76TJDSZPSYZSJQNFVOI462U6DFILMH2RV3LQISODXSOCXGRKW6Q, seg_pos={0 0}, upload=5895]: Uploading piece... Jul 27 04:06:23.863656891 (6h14m39.644590266s) [noise=true , address=45.91.116.7:4011 , piece_id=XMH6MCCHZBBRQWHJFWLRGYHLQD4KUGSGL7MMZSKA5TZJCND3DUDA, seg_pos={0 0}, upload=5895]: Uploading piece... Jul 27 04:06:23.864408612 (6h14m39.645341987s) [noise=true , address=45.91.116.6:4973 , piece_id=J3VULPU24CIM64VEVE3UHO62HPN6TJXGZU6BCJIS5JQJKHJK2LIQ, seg_pos={0 0}, upload=5895]: Uploading piece... Jul 27 04:06:23.864521549 (6h14m39.64545491s) [noise=true , address=104.36.82.10:4079 , piece_id=2SH5D6JOMLFR7VSN7ZY5CLMR5RHIRFYQ7Z5I2JIYTJQM43G4O6OQ, seg_pos={0 0}, upload=5895]: Uploading piece... Jul 27 04:06:23.865518491 (6h14m39.646451872s) [noise=true , address=77.90.31.17:4771 , piece_id=U2LF22GKCMKRZN6EWCF26ITDZQHBFZXCGSQXT33MCX6P57IGI35Q, seg_pos={0 0}, upload=5895]: Uploading piece... Jul 27 04:06:23.865654950 (6h14m39.646588325s) [noise=true , address=45.91.116.8:4048 , piece_id=5REWMVVTBVLDQXQ5KAEHLY4VJL5SGF7ONQWDQQ3N2FDU3OAO6AZA, seg_pos={0 0}, upload=5895]: Uploading piece... Jul 27 04:06:23.866190892 (6h14m39.647124271s) [noise=true , address=104.36.82.12:4151 , piece_id=E2TEUTQBD4Y4AUHSCPMCIFEXBSY7S5HJWBKNKWWTUWEW2BLAAE6Q, seg_pos={0 0}, upload=5895]: Uploading piece... Jul 27 04:06:23.866412590 (6h14m39.647345968s) [noise=true , address=104.36.82.11:4115 , piece_id=YXMNVT6ZHGESGQ2BXKZUVR7U2CNNAI2PO7ZTMDG5UIOV7HTDDUTA, seg_pos={0 0}, upload=5895]: Uploading piece... Jul 27 04:06:23.866715972 (6h14m39.647649359s) [noise=true , address=45.91.116.13:4153 , piece_id=H4535KOCG3F4RKA6AKKMTJSOM6Q6PGRC6M7FR3UIS3BORTSP5UQA, seg_pos={0 0}, upload=5895]: Uploading piece... Jul 27 04:06:23.866998965 (6h14m39.64793234s) [noise=true , address=104.36.82.13:4186 , piece_id=TBLDKALGJYBGR5NLCOCKJ4QKEFNIJXCKAFUBD7VR3UMNWEMKMKHQ, seg_pos={0 0}, upload=5895]: Uploading piece... Jul 27 04:06:23.867230593 (6h14m39.648163974s) [noise=true , address=104.36.82.16:4293 , piece_id=AHH6PRFUSGTDGB7H52DRITARA4IF6GFLVSRYS2Y6EPJCZSKUOYWQ, seg_pos={0 0}, upload=5895]: Uploading piece... Jul 27 04:06:23.949422808 (6h14m39.730356187s) [upload=5895]: Next returned nil segment Jul 27 04:06:23.949442576 (6h14m39.73037596s) [upload=5895]: Waiting for error group managing segments... Jul 27 04:06:23.953497426 (6h14m39.734430802s) [noise=true , address=45.91.116.10:4081 , piece_id=XGDTSEZOUNRTEVWHTO3DHTZMH2VZZOPJPEKLCFAHTXESLLYABHSA, seg_pos={0 0}, upload=5895]: Done uploading piece. err: <nil> Jul 27 04:06:23.953531796 (6h14m39.734465159s) [noise=true , address=45.91.116.6:4973 , piece_id=J3VULPU24CIM64VEVE3UHO62HPN6TJXGZU6BCJIS5JQJKHJK2LIQ, seg_pos={0 0}, upload=5895]: Done uploading piece. err: <nil> Jul 27 04:06:23.954068660 (6h14m39.735002029s) [noise=true , address=45.91.116.9:4262 , piece_id=M5ETD45HH2XZD6HDLTJ3JO2BOOIROXFWKVPTN42PSQEWG4F7KAAQ, seg_pos={0 0}, upload=5895]: Done uploading piece. err: <nil> Jul 27 04:06:23.954194339 (6h14m39.735127707s) [noise=true , address=45.91.116.8:4048 , piece_id=5REWMVVTBVLDQXQ5KAEHLY4VJL5SGF7ONQWDQQ3N2FDU3OAO6AZA, seg_pos={0 0}, upload=5895]: Done uploading piece. err: <nil> Jul 27 04:06:23.954601730 (6h14m39.735535112s) [noise=true , address=45.91.116.7:4011 , piece_id=XMH6MCCHZBBRQWHJFWLRGYHLQD4KUGSGL7MMZSKA5TZJCND3DUDA, seg_pos={0 0}, upload=5895]: Done uploading piece. err: <nil> Jul 27 04:06:23.954755872 (6h14m39.735689237s) [noise=true , address=45.91.116.13:4153 , piece_id=H4535KOCG3F4RKA6AKKMTJSOM6Q6PGRC6M7FR3UIS3BORTSP5UQA, seg_pos={0 0}, upload=5895]: Done uploading piece. err: <nil> Jul 27 04:06:23.955611883 (6h14m39.73654526s) [noise=true , address=77.90.31.19:4843 , piece_id=YML5RDQIK432Z4FSE4DKXIV5U47LQBEZRFMAVLANUSGFUQR23UWQ, seg_pos={0 0}, upload=5895]: Done uploading piece. err: <nil> Jul 27 04:06:23.956373391 (6h14m39.737306756s) [noise=true , address=77.90.31.15:4697 , piece_id=Q76TJDSZPSYZSJQNFVOI462U6DFILMH2RV3LQISODXSOCXGRKW6Q, seg_pos={0 0}, upload=5895]: Done uploading piece. err: <nil> Jul 27 04:06:23.958659122 (6h14m39.73959249s) [noise=true , address=77.90.31.17:4771 , piece_id=U2LF22GKCMKRZN6EWCF26ITDZQHBFZXCGSQXT33MCX6P57IGI35Q, seg_pos={0 0}, upload=5895]: Done uploading piece. err: <nil> Jul 27 04:06:35.058944013 (6h14m50.83987739s) [noise=true , address=104.36.82.16:4293 , piece_id=AHH6PRFUSGTDGB7H52DRITARA4IF6GFLVSRYS2Y6EPJCZSKUOYWQ, seg_pos={0 0}, upload=5895]: Done uploading piece. err: <nil> Jul 27 04:06:41.103204002 (6h14m56.884137386s) [noise=true , address=104.36.82.4:4804 , piece_id=IU6FDNVB5AUPA7MJ6RSXUTLBETPHOFGNJ7YGWBFIZSQFSO6W6BBA, seg_pos={0 0}, upload=5895]: Done uploading piece. err: <nil> Jul 27 04:06:43.191702285 (6h14m58.972635651s) [noise=true , address=104.36.82.6:4873 , piece_id=YP4VJORZZUL5EVCH3P7LD3LNVMHB7CWRTKHGIA7UROXE44N6MMZA, seg_pos={0 0}, upload=5895]: Done uploading piece. err: <nil> Jul 27 04:06:53.777973493 (6h15m9.558906861s) [noise=true , address=104.36.82.11:4115 , piece_id=YXMNVT6ZHGESGQ2BXKZUVR7U2CNNAI2PO7ZTMDG5UIOV7HTDDUTA, seg_pos={0 0}, upload=5895]: Done uploading piece. err: ecclient: upload failed (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR, address:104.36.82.11:4115): protocol: write tcp 159.223.143.39:34210->104.36.82.11:4115: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 04:06:53.778019532 (6h15m9.558952909s) [noise=true , address=104.36.82.12:4151 , piece_id=E2TEUTQBD4Y4AUHSCPMCIFEXBSY7S5HJWBKNKWWTUWEW2BLAAE6Q, seg_pos={0 0}, upload=5895]: Done uploading piece. err: ecclient: upload failed (node:1w6pwZMdeQKjt92qx6WbTWU5VPYJto5gS87f6iKvCBZH6LfyeB, address:104.36.82.12:4151): protocol: rpc: dial tcp 104.36.82.12:4151: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 04:06:53.778043653 (6h15m9.558977021s) [noise=true , address=104.36.82.13:4186 , piece_id=TBLDKALGJYBGR5NLCOCKJ4QKEFNIJXCKAFUBD7VR3UMNWEMKMKHQ, seg_pos={0 0}, upload=5895]: Done uploading piece. err: ecclient: upload failed (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht, address:104.36.82.13:4186): protocol: read tcp 159.223.143.39:34340->104.36.82.13:4186: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 04:06:53.778095228 (6h15m9.559028593s) [noise=true , address=104.36.82.10:4079 , piece_id=2SH5D6JOMLFR7VSN7ZY5CLMR5RHIRFYQ7Z5I2JIYTJQM43G4O6OQ, seg_pos={0 0}, upload=5895]: Done uploading piece. err: ecclient: upload failed (node:12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud, address:104.36.82.10:4079): protocol: write tcp 159.223.143.39:53840->104.36.82.10:4079: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 04:06:53.778142167 (6h15m9.559075532s) [seg_pos={0 0}, upload=5895]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 27 04:06:53.837032161 (6h15m9.617965543s) [upload=5895]: Done uploading segments... Jul 27 04:06:53.837055996 (6h15m9.617989366s) [upload=5895]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 650.549318ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............................................X............] duration: 373.021169ms FAIL (2025-07-27 04:24:21 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 12) Duration: 373.051956ms core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [......................................X.....................] duration: 547.561159ms FAIL (2025-07-27 04:24:00 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== {0 0}, upload=5911]: Begin upload segment... Jul 27 04:07:17.745307658 (6h15m33.526241043s) [seg_pos={0 0}, upload=5911]: Done begin upload segment. Jul 27 04:07:17.745349241 (6h15m33.526282625s) [noise=true , address=104.36.82.52:4098 , piece_id=M2ETF25QQ53RRGFPUKXJRINIWV7AWXSECEDR2AIM645IED72NURQ, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.745439694 (6h15m33.526373073s) [noise=true , address=104.36.82.56:4244 , piece_id=LQCZ5W54NBHWAVBVYK7HYSSJDC73E6FV4CP4YIM4L5H2N3OLOZXA, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.745972271 (6h15m33.526905654s) [noise=true , address=104.36.82.36:4650 , piece_id=BKJULWGPESBINNTSUYP6E4KZRXKPACVU5I7DOSZBO4YMK2V26HEQ, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.745980243 (6h15m33.526913621s) [noise=true , address=104.36.82.38:4722 , piece_id=B2MBQY5CTMEI6WSE7FTLOV5RUJ22SCNG7N2EEJILEATGJ4TIALXA, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.746403818 (6h15m33.527337198s) [noise=true , address=104.36.82.22:4534 , piece_id=CW4AQT3XUGA4A5OC6NUY4NYOOAMLN5FFJJN5PQBYEON7HPD5LCYQ, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.746455493 (6h15m33.527388875s) [noise=true , address=104.36.82.14:4248 , piece_id=LDSGCUM7Q2VX5CDFI4JRGEEG7D5H5Q5Z5WUWKU277VN2JFRX6UGA, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.746872127 (6h15m33.527805505s) [noise=true , address=104.36.82.40:4954 , piece_id=YLOHTJLRAED66OMEQYPZHNNRI77V6WOR4EHTEZOOEHMGZHJ2O3QQ, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.747123162 (6h15m33.52805654s) [noise=true , address=104.36.82.46:4496 , piece_id=GWNQR5WRGRRQBLSGWJFD3O3UNCMNE62JQNPIV5SYKBUIL3BQKA5A, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.747280678 (6h15m33.528214053s) [noise=true , address=104.36.82.47:4524 , piece_id=DONJFTKUQ2XCW6BOU7AVOKJ7XMDMBPGQVUPRANH7X2YS7PJXUKIA, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.747736400 (6h15m33.528669779s) [noise=true , address=104.36.82.44:4893 , piece_id=HUJICRJ6W54O3AZZR6H3KG6E4TGBTNS46O6XVMQM3PFAZE7A2HZA, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.747755734 (6h15m33.528689109s) [noise=true , address=104.36.82.55:4204 , piece_id=72WISGRHMOAZDV5EDGTBOMLBL4HDOF3CBEMQAAIRGILPC7J5JKAQ, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.748313124 (6h15m33.5292465s) [noise=true , address=104.36.82.58:4287 , piece_id=BVDYUEWU7TVEQDMZXX5SJNS4QICVKMTGFLCUV6P2YHOE7TI25VXA, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.748485217 (6h15m33.529418594s) [noise=true , address=104.36.82.53:4123 , piece_id=NXJRZ5BQSLXSOIOSUJ4MM4AGKHP4UIDUGEJKLUUX7YOGQJA6WWZA, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.748831329 (6h15m33.529764704s) [noise=true , address=104.36.82.42:4829 , piece_id=ZLWJFP3LGX3YLXBJYIR66DXTYEVDBKXKVXCQSF2PCYCYJBIEJGMA, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.749299293 (6h15m33.530232669s) [noise=true , address=104.36.82.8:4002 , piece_id=6XSPWQC3US355PS6IQ4K3IYNQZH5MAT4IVIPGSYRUM3XYPSCWE6Q, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.749790974 (6h15m33.53072436s) [noise=true , address=104.36.82.44:4901 , piece_id=N4YLCLR2QZSULUL2XWONEKBFUMNOU7KUX2QEOK4SSDFSCI7IL6LA, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.750478221 (6h15m33.531411597s) [noise=true , address=104.36.82.54:4172 , piece_id=ZNA53MWDRWPUH6FUR4DJERBDT2IOYGA3IYUADKXXIOLQJLQOXE7Q, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.752884255 (6h15m33.533817625s) [noise=true , address=64.49.13.140:4508 , piece_id=MZ52U75VOIK6Q7QFXMMYRRG7JXFWDFU2JWQFYPERUCMH3BUX6XPQ, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.879839555 (6h15m33.660772917s) [upload=5911]: Next returned nil segment Jul 27 04:07:17.879861389 (6h15m33.660794765s) [upload=5911]: Waiting for error group managing segments... Jul 27 04:07:17.888652438 (6h15m33.669585811s) [noise=true , address=104.36.82.58:4287 , piece_id=BVDYUEWU7TVEQDMZXX5SJNS4QICVKMTGFLCUV6P2YHOE7TI25VXA, seg_pos={0 0}, upload=5911]: Done uploading piece. err: <nil> Jul 27 04:07:17.952192045 (6h15m33.733125429s) [noise=true , address=104.36.82.53:4123 , piece_id=NXJRZ5BQSLXSOIOSUJ4MM4AGKHP4UIDUGEJKLUUX7YOGQJA6WWZA, seg_pos={0 0}, upload=5911]: Done uploading piece. err: <nil> Jul 27 04:07:17.981907539 (6h15m33.762840911s) [noise=true , address=104.36.82.56:4244 , piece_id=LQCZ5W54NBHWAVBVYK7HYSSJDC73E6FV4CP4YIM4L5H2N3OLOZXA, seg_pos={0 0}, upload=5911]: Done uploading piece. err: <nil> Jul 27 04:07:17.982370496 (6h15m33.7633039s) [noise=true , address=104.36.82.55:4204 , piece_id=72WISGRHMOAZDV5EDGTBOMLBL4HDOF3CBEMQAAIRGILPC7J5JKAQ, seg_pos={0 0}, upload=5911]: Done uploading piece. err: <nil> Jul 27 04:07:18.009548506 (6h15m33.790481886s) [noise=true , address=104.36.82.52:4098 , piece_id=M2ETF25QQ53RRGFPUKXJRINIWV7AWXSECEDR2AIM645IED72NURQ, seg_pos={0 0}, upload=5911]: Done uploading piece. err: <nil> Jul 27 04:07:18.023770598 (6h15m33.804703983s) [noise=true , address=104.36.82.40:4954 , piece_id=YLOHTJLRAED66OMEQYPZHNNRI77V6WOR4EHTEZOOEHMGZHJ2O3QQ, seg_pos={0 0}, upload=5911]: Done uploading piece. err: <nil> Jul 27 04:07:18.139745581 (6h15m33.92067895s) [noise=true , address=104.36.82.22:4534 , piece_id=CW4AQT3XUGA4A5OC6NUY4NYOOAMLN5FFJJN5PQBYEON7HPD5LCYQ, seg_pos={0 0}, upload=5911]: Done uploading piece. err: <nil> Jul 27 04:07:18.423748140 (6h15m34.204681511s) [noise=true , address=64.49.13.140:4508 , piece_id=MZ52U75VOIK6Q7QFXMMYRRG7JXFWDFU2JWQFYPERUCMH3BUX6XPQ, seg_pos={0 0}, upload=5911]: Done uploading piece. err: <nil> Jul 27 04:08:16.739940381 (6h16m32.52087375s) [noise=true , address=104.36.82.54:4172 , piece_id=ZNA53MWDRWPUH6FUR4DJERBDT2IOYGA3IYUADKXXIOLQJLQOXE7Q, seg_pos={0 0}, upload=5911]: Done uploading piece. err: <nil> Jul 27 04:08:17.177295397 (6h16m32.958228766s) [noise=true , address=104.36.82.8:4002 , piece_id=6XSPWQC3US355PS6IQ4K3IYNQZH5MAT4IVIPGSYRUM3XYPSCWE6Q, seg_pos={0 0}, upload=5911]: Done uploading piece. err: <nil> Jul 27 04:08:17.534296584 (6h16m33.315229953s) [noise=true , address=104.36.82.36:4650 , piece_id=BKJULWGPESBINNTSUYP6E4KZRXKPACVU5I7DOSZBO4YMK2V26HEQ, seg_pos={0 0}, upload=5911]: Done uploading piece. err: <nil> Jul 27 04:08:17.677851566 (6h16m33.458784943s) [noise=true , address=104.36.82.14:4248 , piece_id=LDSGCUM7Q2VX5CDFI4JRGEEG7D5H5Q5Z5WUWKU277VN2JFRX6UGA, seg_pos={0 0}, upload=5911]: Done uploading piece. err: ecclient: upload failed (node:1u6ir2PLqDa9xGm64cuqq1RqY4qeBa1fqCo9VghspBf54XEXqn, address:104.36.82.14:4248): protocol: rpc: dial tcp 104.36.82.14:4248: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 04:08:17.678115170 (6h16m33.459048549s) [noise=true , address=104.36.82.46:4496 , piece_id=GWNQR5WRGRRQBLSGWJFD3O3UNCMNE62JQNPIV5SYKBUIL3BQKA5A, seg_pos={0 0}, upload=5911]: Done uploading piece. err: ecclient: upload failed (node:12V8XhSWkyGUVYFR8MykifFAP1YAfGCwzqucDcMMfzw3xsVW77Z, address:104.36.82.46:4496): protocol: read tcp 159.223.143.39:52800->104.36.82.46:4496: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 04:08:17.678144409 (6h16m33.459077789s) [noise=true , address=104.36.82.44:4901 , piece_id=N4YLCLR2QZSULUL2XWONEKBFUMNOU7KUX2QEOK4SSDFSCI7IL6LA, seg_pos={0 0}, upload=5911]: Done uploading piece. err: ecclient: upload failed (node:1FHE84zvBb1MKwB1Ht7puTafgQxDEvPt4mYqkYVSSufAMpWQA1, address:104.36.82.44:4901): protocol: rpc: dial tcp 104.36.82.44:4901: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 04:08:17.678229622 (6h16m33.459162988s) [noise=true , address=104.36.82.44:4893 , piece_id=HUJICRJ6W54O3AZZR6H3KG6E4TGBTNS46O6XVMQM3PFAZE7A2HZA, seg_pos={0 0}, upload=5911]: Done uploading piece. err: ecclient: upload failed (node:17VMFZ2T8jrSJVtchciJ9SnyrR6JstvcMoXrN5Q3zTD1aQU1oJ, address:104.36.82.44:4893): protocol: write tcp 159.223.143.39:35302->104.36.82.44:4893: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 04:08:17.678310218 (6h16m33.459243583s) [noise=true , address=104.36.82.42:4829 , piece_id=ZLWJFP3LGX3YLXBJYIR66DXTYEVDBKXKVXCQSF2PCYCYJBIEJGMA, seg_pos={0 0}, upload=5911]: Done uploading piece. err: ecclient: upload failed (node:1F5T8Vt1x4nXFaSenvGBCYjxhRL1H1sxRvzLjW3CqzwDVhq39F, address:104.36.82.42:4829): protocol: rpc: dial tcp 104.36.82.42:4829: i/o timeout; conn closed: op; piecestore: piecestore close: context deadline exceeded Jul 27 04:08:17.678412989 (6h16m33.459346365s) [noise=true , address=104.36.82.38:4722 , piece_id=B2MBQY5CTMEI6WSE7FTLOV5RUJ22SCNG7N2EEJILEATGJ4TIALXA, seg_pos={0 0}, upload=5911]: Done uploading piece. err: ecclient: upload failed (node:12Zw3dtUg9gARz52KhwDLTjcqC3CNm7dpPzpdx7P7PbL1jwe8uc, address:104.36.82.38:4722): protocol: read tcp 159.223.143.39:56048->104.36.82.38:4722: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 04:08:17.678455247 (6h16m33.459388624s) [noise=true , address=104.36.82.47:4524 , piece_id=DONJFTKUQ2XCW6BOU7AVOKJ7XMDMBPGQVUPRANH7X2YS7PJXUKIA, seg_pos={0 0}, upload=5911]: Done uploading piece. err: ecclient: upload failed (node:1xfbCzjSwFDRrR1gPggeN8tNCLbRRWDVzCJ6mqhZsbVSe8d4s4, address:104.36.82.47:4524): protocol: read tcp 159.223.143.39:46694->104.36.82.47:4524: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 04:08:17.678510668 (6h16m33.459444052s) [seg_pos={0 0}, upload=5911]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Jul 27 04:08:17.706442864 (6h16m33.487376232s) [upload=5911]: Done uploading segments... Jul 27 04:08:17.706495329 (6h16m33.487428705s) [upload=5911]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 547.592763ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [................................................X...........] duration: 14.382829305s FAIL (2025-07-27 04:23:57 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 12) Duration: 14.382933567s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................................X..........................] duration: 3.013683504s FAIL (2025-07-27 04:23:56 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== ece_id=NYOFDEZ7CV4PHGQ6JZMEEVAYGJRPYG6IBW7MYJO42HMWOJM7PHZA, seg_pos={0 0}, upload=5879]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12crcXpNe9NgeXTsUpooLSBgrSUFbRxbSbanLa39w2Hr5mGFww): protocol: read tcp 159.223.143.39:53630->104.36.82.5:4840: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 04:05:32.508904485 (6h13m48.289837866s) [seg_pos={0 0}, upload=5879]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 27 04:05:32.509028070 (6h13m48.289961457s) [upload=5879]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 27 04:05:32.509034645 (6h13m48.289968129s) [upload=5879]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 27 04:05:32.581834839 (6h13m48.362768225s) [upload=5879]: Done uploading segments... Jul 27 04:05:32.581875919 (6h13m48.362809298s) [upload=5879]: Upload finished. err: <nil> Jul 27 04:06:23.829155331 (6h14m39.610088696s) [upload=5895]: Starting upload Jul 27 04:06:23.829225021 (6h14m39.610158398s) [upload=5895]: Done starting upload Jul 27 04:06:23.829294606 (6h14m39.61022799s) [upload=5895]: Uploading segments... Jul 27 04:06:23.829577656 (6h14m39.610511053s) [upload=5895]: Got next segment. Inline: false Jul 27 04:06:23.829649664 (6h14m39.610583027s) [upload=5895]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 27 04:06:23.829657065 (6h14m39.61059043s) [upload=5895]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 27 04:06:23.861405530 (6h14m39.642338913s) [seg_pos={0 0}, upload=5895]: Begin upload segment... Jul 27 04:06:23.861476351 (6h14m39.642409713s) [seg_pos={0 0}, upload=5895]: Done begin upload segment. Jul 27 04:06:23.861509120 (6h14m39.642442506s) [noise=true , address=77.90.31.19:4843 , piece_id=YML5RDQIK432Z4FSE4DKXIV5U47LQBEZRFMAVLANUSGFUQR23UWQ, seg_pos={0 0}, upload=5895]: Uploading piece... Jul 27 04:06:23.861953723 (6h14m39.6428871s) [noise=true , address=45.91.116.10:4081 , piece_id=XGDTSEZOUNRTEVWHTO3DHTZMH2VZZOPJPEKLCFAHTXESLLYABHSA, seg_pos={0 0}, upload=5895]: Uploading piece... Jul 27 04:06:23.862060983 (6h14m39.642994358s) [noise=true , address=45.91.116.9:4262 , piece_id=M5ETD45HH2XZD6HDLTJ3JO2BOOIROXFWKVPTN42PSQEWG4F7KAAQ, seg_pos={0 0}, upload=5895]: Uploading piece... Jul 27 04:06:23.862759464 (6h14m39.64369284s) [noise=true , address=104.36.82.4:4804 , piece_id=IU6FDNVB5AUPA7MJ6RSXUTLBETPHOFGNJ7YGWBFIZSQFSO6W6BBA, seg_pos={0 0}, upload=5895]: Uploading piece... Jul 27 04:06:23.862835086 (6h14m39.643768461s) [noise=true , address=104.36.82.6:4873 , piece_id=YP4VJORZZUL5EVCH3P7LD3LNVMHB7CWRTKHGIA7UROXE44N6MMZA, seg_pos={0 0}, upload=5895]: Uploading piece... Jul 27 04:06:23.863593880 (6h14m39.644527258s) [noise=true , address=77.90.31.15:4697 , piece_id=Q76TJDSZPSYZSJQNFVOI462U6DFILMH2RV3LQISODXSOCXGRKW6Q, seg_pos={0 0}, upload=5895]: Uploading piece... Jul 27 04:06:23.863656891 (6h14m39.644590266s) [noise=true , address=45.91.116.7:4011 , piece_id=XMH6MCCHZBBRQWHJFWLRGYHLQD4KUGSGL7MMZSKA5TZJCND3DUDA, seg_pos={0 0}, upload=5895]: Uploading piece... Jul 27 04:06:23.864408612 (6h14m39.645341987s) [noise=true , address=45.91.116.6:4973 , piece_id=J3VULPU24CIM64VEVE3UHO62HPN6TJXGZU6BCJIS5JQJKHJK2LIQ, seg_pos={0 0}, upload=5895]: Uploading piece... Jul 27 04:06:23.864521549 (6h14m39.64545491s) [noise=true , address=104.36.82.10:4079 , piece_id=2SH5D6JOMLFR7VSN7ZY5CLMR5RHIRFYQ7Z5I2JIYTJQM43G4O6OQ, seg_pos={0 0}, upload=5895]: Uploading piece... Jul 27 04:06:23.865518491 (6h14m39.646451872s) [noise=true , address=77.90.31.17:4771 , piece_id=U2LF22GKCMKRZN6EWCF26ITDZQHBFZXCGSQXT33MCX6P57IGI35Q, seg_pos={0 0}, upload=5895]: Uploading piece... Jul 27 04:06:23.865654950 (6h14m39.646588325s) [noise=true , address=45.91.116.8:4048 , piece_id=5REWMVVTBVLDQXQ5KAEHLY4VJL5SGF7ONQWDQQ3N2FDU3OAO6AZA, seg_pos={0 0}, upload=5895]: Uploading piece... Jul 27 04:06:23.866190892 (6h14m39.647124271s) [noise=true , address=104.36.82.12:4151 , piece_id=E2TEUTQBD4Y4AUHSCPMCIFEXBSY7S5HJWBKNKWWTUWEW2BLAAE6Q, seg_pos={0 0}, upload=5895]: Uploading piece... Jul 27 04:06:23.866412590 (6h14m39.647345968s) [noise=true , address=104.36.82.11:4115 , piece_id=YXMNVT6ZHGESGQ2BXKZUVR7U2CNNAI2PO7ZTMDG5UIOV7HTDDUTA, seg_pos={0 0}, upload=5895]: Uploading piece... Jul 27 04:06:23.866715972 (6h14m39.647649359s) [noise=true , address=45.91.116.13:4153 , piece_id=H4535KOCG3F4RKA6AKKMTJSOM6Q6PGRC6M7FR3UIS3BORTSP5UQA, seg_pos={0 0}, upload=5895]: Uploading piece... Jul 27 04:06:23.866998965 (6h14m39.64793234s) [noise=true , address=104.36.82.13:4186 , piece_id=TBLDKALGJYBGR5NLCOCKJ4QKEFNIJXCKAFUBD7VR3UMNWEMKMKHQ, seg_pos={0 0}, upload=5895]: Uploading piece... Jul 27 04:06:23.867230593 (6h14m39.648163974s) [noise=true , address=104.36.82.16:4293 , piece_id=AHH6PRFUSGTDGB7H52DRITARA4IF6GFLVSRYS2Y6EPJCZSKUOYWQ, seg_pos={0 0}, upload=5895]: Uploading piece... Jul 27 04:06:23.949422808 (6h14m39.730356187s) [upload=5895]: Next returned nil segment Jul 27 04:06:23.949442576 (6h14m39.73037596s) [upload=5895]: Waiting for error group managing segments... Jul 27 04:06:23.953497426 (6h14m39.734430802s) [noise=true , address=45.91.116.10:4081 , piece_id=XGDTSEZOUNRTEVWHTO3DHTZMH2VZZOPJPEKLCFAHTXESLLYABHSA, seg_pos={0 0}, upload=5895]: Done uploading piece. err: <nil> Jul 27 04:06:23.953531796 (6h14m39.734465159s) [noise=true , address=45.91.116.6:4973 , piece_id=J3VULPU24CIM64VEVE3UHO62HPN6TJXGZU6BCJIS5JQJKHJK2LIQ, seg_pos={0 0}, upload=5895]: Done uploading piece. err: <nil> Jul 27 04:06:23.954068660 (6h14m39.735002029s) [noise=true , address=45.91.116.9:4262 , piece_id=M5ETD45HH2XZD6HDLTJ3JO2BOOIROXFWKVPTN42PSQEWG4F7KAAQ, seg_pos={0 0}, upload=5895]: Done uploading piece. err: <nil> Jul 27 04:06:23.954194339 (6h14m39.735127707s) [noise=true , address=45.91.116.8:4048 , piece_id=5REWMVVTBVLDQXQ5KAEHLY4VJL5SGF7ONQWDQQ3N2FDU3OAO6AZA, seg_pos={0 0}, upload=5895]: Done uploading piece. err: <nil> Jul 27 04:06:23.954601730 (6h14m39.735535112s) [noise=true , address=45.91.116.7:4011 , piece_id=XMH6MCCHZBBRQWHJFWLRGYHLQD4KUGSGL7MMZSKA5TZJCND3DUDA, seg_pos={0 0}, upload=5895]: Done uploading piece. err: <nil> Jul 27 04:06:23.954755872 (6h14m39.735689237s) [noise=true , address=45.91.116.13:4153 , piece_id=H4535KOCG3F4RKA6AKKMTJSOM6Q6PGRC6M7FR3UIS3BORTSP5UQA, seg_pos={0 0}, upload=5895]: Done uploading piece. err: <nil> Jul 27 04:06:23.955611883 (6h14m39.73654526s) [noise=true , address=77.90.31.19:4843 , piece_id=YML5RDQIK432Z4FSE4DKXIV5U47LQBEZRFMAVLANUSGFUQR23UWQ, seg_pos={0 0}, upload=5895]: Done uploading piece. err: <nil> Jul 27 04:06:23.956373391 (6h14m39.737306756s) [noise=true , address=77.90.31.15:4697 , piece_id=Q76TJDSZPSYZSJQNFVOI462U6DFILMH2RV3LQISODXSOCXGRKW6Q, seg_pos={0 0}, upload=5895]: Done uploading piece. err: <nil> Jul 27 04:06:23.958659122 (6h14m39.73959249s) [noise=true , address=77.90.31.17:4771 , piece_id=U2LF22GKCMKRZN6EWCF26ITDZQHBFZXCGSQXT33MCX6P57IGI35Q, seg_pos={0 0}, upload=5895]: Done uploading piece. err: <nil> Jul 27 04:06:35.058944013 (6h14m50.83987739s) [noise=true , address=104.36.82.16:4293 , piece_id=AHH6PRFUSGTDGB7H52DRITARA4IF6GFLVSRYS2Y6EPJCZSKUOYWQ, seg_pos={0 0}, upload=5895]: Done uploading piece. err: <nil> Jul 27 04:06:41.103204002 (6h14m56.884137386s) [noise=true , address=104.36.82.4:4804 , piece_id=IU6FDNVB5AUPA7MJ6RSXUTLBETPHOFGNJ7YGWBFIZSQFSO6W6BBA, seg_pos={0 0}, upload=5895]: Done uploading piece. err: <nil> Jul 27 04:06:43.191702285 (6h14m58.972635651s) [noise=true , address=104.36.82.6:4873 , piece_id=YP4VJORZZUL5EVCH3P7LD3LNVMHB7CWRTKHGIA7UROXE44N6MMZA, seg_pos={0 0}, upload=5895]: Done uploading piece. err: <nil> Jul 27 04:06:53.777973493 (6h15m9.558906861s) [noise=true , address=104.36.82.11:4115 , piece_id=YXMNVT6ZHGESGQ2BXKZUVR7U2CNNAI2PO7ZTMDG5UIOV7HTDDUTA, seg_pos={0 0}, upload=5895]: Done uploading piece. err: ecclient: upload failed (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR, address:104.36.82.11:4115): protocol: write tcp 159.223.143.39:34210->104.36.82.11:4115: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 04:06:53.778019532 (6h15m9.558952909s) [noise=true , address=104.36.82.12:4151 , piece_id=E2TEUTQBD4Y4AUHSCPMCIFEXBSY7S5HJWBKNKWWTUWEW2BLAAE6Q, seg_pos={0 0}, upload=5895]: Done uploading piece. err: ecclient: upload failed (node:1w6pwZMdeQKjt92qx6WbTWU5VPYJto5gS87f6iKvCBZH6LfyeB, address:104.36.82.12:4151): protocol: rpc: dial tcp 104.36.82.12:4151: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 04:06:53.778043653 (6h15m9.558977021s) [noise=true , address=104.36.82.13:4186 , piece_id=TBLDKALGJYBGR5NLCOCKJ4QKEFNIJXCKAFUBD7VR3UMNWEMKMKHQ, seg_pos={0 0}, upload=5895]: Done uploading piece. err: ecclient: upload failed (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht, address:104.36.82.13:4186): protocol: read tcp 159.223.143.39:34340->104.36.82.13:4186: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 04:06:53.778095228 (6h15m9.559028593s) [noise=true , address=104.36.82.10:4079 , piece_id=2SH5D6JOMLFR7VSN7ZY5CLMR5RHIRFYQ7Z5I2JIYTJQM43G4O6OQ, seg_pos={0 0}, upload=5895]: Done uploading piece. err: ecclient: upload failed (node:12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud, address:104.36.82.10:4079): protocol: write tcp 159.223.143.39:53840->104.36.82.10:4079: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 04:06:53.778142167 (6h15m9.559075532s) [seg_pos={0 0}, upload=5895]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 27 04:06:53.837032161 (6h15m9.617965543s) [upload=5895]: Done uploading segments... Jul 27 04:06:53.837055996 (6h15m9.617989366s) [upload=5895]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 3.013716934s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................................X.........................] duration: 422.883666ms FAIL (2025-07-27 04:23:27 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== ece_id=NYOFDEZ7CV4PHGQ6JZMEEVAYGJRPYG6IBW7MYJO42HMWOJM7PHZA, seg_pos={0 0}, upload=5879]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12crcXpNe9NgeXTsUpooLSBgrSUFbRxbSbanLa39w2Hr5mGFww): protocol: read tcp 159.223.143.39:53630->104.36.82.5:4840: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 04:05:32.508904485 (6h13m48.289837866s) [seg_pos={0 0}, upload=5879]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 27 04:05:32.509028070 (6h13m48.289961457s) [upload=5879]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 27 04:05:32.509034645 (6h13m48.289968129s) [upload=5879]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 27 04:05:32.581834839 (6h13m48.362768225s) [upload=5879]: Done uploading segments... Jul 27 04:05:32.581875919 (6h13m48.362809298s) [upload=5879]: Upload finished. err: <nil> Jul 27 04:06:23.829155331 (6h14m39.610088696s) [upload=5895]: Starting upload Jul 27 04:06:23.829225021 (6h14m39.610158398s) [upload=5895]: Done starting upload Jul 27 04:06:23.829294606 (6h14m39.61022799s) [upload=5895]: Uploading segments... Jul 27 04:06:23.829577656 (6h14m39.610511053s) [upload=5895]: Got next segment. Inline: false Jul 27 04:06:23.829649664 (6h14m39.610583027s) [upload=5895]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 27 04:06:23.829657065 (6h14m39.61059043s) [upload=5895]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 27 04:06:23.861405530 (6h14m39.642338913s) [seg_pos={0 0}, upload=5895]: Begin upload segment... Jul 27 04:06:23.861476351 (6h14m39.642409713s) [seg_pos={0 0}, upload=5895]: Done begin upload segment. Jul 27 04:06:23.861509120 (6h14m39.642442506s) [noise=true , address=77.90.31.19:4843 , piece_id=YML5RDQIK432Z4FSE4DKXIV5U47LQBEZRFMAVLANUSGFUQR23UWQ, seg_pos={0 0}, upload=5895]: Uploading piece... Jul 27 04:06:23.861953723 (6h14m39.6428871s) [noise=true , address=45.91.116.10:4081 , piece_id=XGDTSEZOUNRTEVWHTO3DHTZMH2VZZOPJPEKLCFAHTXESLLYABHSA, seg_pos={0 0}, upload=5895]: Uploading piece... Jul 27 04:06:23.862060983 (6h14m39.642994358s) [noise=true , address=45.91.116.9:4262 , piece_id=M5ETD45HH2XZD6HDLTJ3JO2BOOIROXFWKVPTN42PSQEWG4F7KAAQ, seg_pos={0 0}, upload=5895]: Uploading piece... Jul 27 04:06:23.862759464 (6h14m39.64369284s) [noise=true , address=104.36.82.4:4804 , piece_id=IU6FDNVB5AUPA7MJ6RSXUTLBETPHOFGNJ7YGWBFIZSQFSO6W6BBA, seg_pos={0 0}, upload=5895]: Uploading piece... Jul 27 04:06:23.862835086 (6h14m39.643768461s) [noise=true , address=104.36.82.6:4873 , piece_id=YP4VJORZZUL5EVCH3P7LD3LNVMHB7CWRTKHGIA7UROXE44N6MMZA, seg_pos={0 0}, upload=5895]: Uploading piece... Jul 27 04:06:23.863593880 (6h14m39.644527258s) [noise=true , address=77.90.31.15:4697 , piece_id=Q76TJDSZPSYZSJQNFVOI462U6DFILMH2RV3LQISODXSOCXGRKW6Q, seg_pos={0 0}, upload=5895]: Uploading piece... Jul 27 04:06:23.863656891 (6h14m39.644590266s) [noise=true , address=45.91.116.7:4011 , piece_id=XMH6MCCHZBBRQWHJFWLRGYHLQD4KUGSGL7MMZSKA5TZJCND3DUDA, seg_pos={0 0}, upload=5895]: Uploading piece... Jul 27 04:06:23.864408612 (6h14m39.645341987s) [noise=true , address=45.91.116.6:4973 , piece_id=J3VULPU24CIM64VEVE3UHO62HPN6TJXGZU6BCJIS5JQJKHJK2LIQ, seg_pos={0 0}, upload=5895]: Uploading piece... Jul 27 04:06:23.864521549 (6h14m39.64545491s) [noise=true , address=104.36.82.10:4079 , piece_id=2SH5D6JOMLFR7VSN7ZY5CLMR5RHIRFYQ7Z5I2JIYTJQM43G4O6OQ, seg_pos={0 0}, upload=5895]: Uploading piece... Jul 27 04:06:23.865518491 (6h14m39.646451872s) [noise=true , address=77.90.31.17:4771 , piece_id=U2LF22GKCMKRZN6EWCF26ITDZQHBFZXCGSQXT33MCX6P57IGI35Q, seg_pos={0 0}, upload=5895]: Uploading piece... Jul 27 04:06:23.865654950 (6h14m39.646588325s) [noise=true , address=45.91.116.8:4048 , piece_id=5REWMVVTBVLDQXQ5KAEHLY4VJL5SGF7ONQWDQQ3N2FDU3OAO6AZA, seg_pos={0 0}, upload=5895]: Uploading piece... Jul 27 04:06:23.866190892 (6h14m39.647124271s) [noise=true , address=104.36.82.12:4151 , piece_id=E2TEUTQBD4Y4AUHSCPMCIFEXBSY7S5HJWBKNKWWTUWEW2BLAAE6Q, seg_pos={0 0}, upload=5895]: Uploading piece... Jul 27 04:06:23.866412590 (6h14m39.647345968s) [noise=true , address=104.36.82.11:4115 , piece_id=YXMNVT6ZHGESGQ2BXKZUVR7U2CNNAI2PO7ZTMDG5UIOV7HTDDUTA, seg_pos={0 0}, upload=5895]: Uploading piece... Jul 27 04:06:23.866715972 (6h14m39.647649359s) [noise=true , address=45.91.116.13:4153 , piece_id=H4535KOCG3F4RKA6AKKMTJSOM6Q6PGRC6M7FR3UIS3BORTSP5UQA, seg_pos={0 0}, upload=5895]: Uploading piece... Jul 27 04:06:23.866998965 (6h14m39.64793234s) [noise=true , address=104.36.82.13:4186 , piece_id=TBLDKALGJYBGR5NLCOCKJ4QKEFNIJXCKAFUBD7VR3UMNWEMKMKHQ, seg_pos={0 0}, upload=5895]: Uploading piece... Jul 27 04:06:23.867230593 (6h14m39.648163974s) [noise=true , address=104.36.82.16:4293 , piece_id=AHH6PRFUSGTDGB7H52DRITARA4IF6GFLVSRYS2Y6EPJCZSKUOYWQ, seg_pos={0 0}, upload=5895]: Uploading piece... Jul 27 04:06:23.949422808 (6h14m39.730356187s) [upload=5895]: Next returned nil segment Jul 27 04:06:23.949442576 (6h14m39.73037596s) [upload=5895]: Waiting for error group managing segments... Jul 27 04:06:23.953497426 (6h14m39.734430802s) [noise=true , address=45.91.116.10:4081 , piece_id=XGDTSEZOUNRTEVWHTO3DHTZMH2VZZOPJPEKLCFAHTXESLLYABHSA, seg_pos={0 0}, upload=5895]: Done uploading piece. err: <nil> Jul 27 04:06:23.953531796 (6h14m39.734465159s) [noise=true , address=45.91.116.6:4973 , piece_id=J3VULPU24CIM64VEVE3UHO62HPN6TJXGZU6BCJIS5JQJKHJK2LIQ, seg_pos={0 0}, upload=5895]: Done uploading piece. err: <nil> Jul 27 04:06:23.954068660 (6h14m39.735002029s) [noise=true , address=45.91.116.9:4262 , piece_id=M5ETD45HH2XZD6HDLTJ3JO2BOOIROXFWKVPTN42PSQEWG4F7KAAQ, seg_pos={0 0}, upload=5895]: Done uploading piece. err: <nil> Jul 27 04:06:23.954194339 (6h14m39.735127707s) [noise=true , address=45.91.116.8:4048 , piece_id=5REWMVVTBVLDQXQ5KAEHLY4VJL5SGF7ONQWDQQ3N2FDU3OAO6AZA, seg_pos={0 0}, upload=5895]: Done uploading piece. err: <nil> Jul 27 04:06:23.954601730 (6h14m39.735535112s) [noise=true , address=45.91.116.7:4011 , piece_id=XMH6MCCHZBBRQWHJFWLRGYHLQD4KUGSGL7MMZSKA5TZJCND3DUDA, seg_pos={0 0}, upload=5895]: Done uploading piece. err: <nil> Jul 27 04:06:23.954755872 (6h14m39.735689237s) [noise=true , address=45.91.116.13:4153 , piece_id=H4535KOCG3F4RKA6AKKMTJSOM6Q6PGRC6M7FR3UIS3BORTSP5UQA, seg_pos={0 0}, upload=5895]: Done uploading piece. err: <nil> Jul 27 04:06:23.955611883 (6h14m39.73654526s) [noise=true , address=77.90.31.19:4843 , piece_id=YML5RDQIK432Z4FSE4DKXIV5U47LQBEZRFMAVLANUSGFUQR23UWQ, seg_pos={0 0}, upload=5895]: Done uploading piece. err: <nil> Jul 27 04:06:23.956373391 (6h14m39.737306756s) [noise=true , address=77.90.31.15:4697 , piece_id=Q76TJDSZPSYZSJQNFVOI462U6DFILMH2RV3LQISODXSOCXGRKW6Q, seg_pos={0 0}, upload=5895]: Done uploading piece. err: <nil> Jul 27 04:06:23.958659122 (6h14m39.73959249s) [noise=true , address=77.90.31.17:4771 , piece_id=U2LF22GKCMKRZN6EWCF26ITDZQHBFZXCGSQXT33MCX6P57IGI35Q, seg_pos={0 0}, upload=5895]: Done uploading piece. err: <nil> Jul 27 04:06:35.058944013 (6h14m50.83987739s) [noise=true , address=104.36.82.16:4293 , piece_id=AHH6PRFUSGTDGB7H52DRITARA4IF6GFLVSRYS2Y6EPJCZSKUOYWQ, seg_pos={0 0}, upload=5895]: Done uploading piece. err: <nil> Jul 27 04:06:41.103204002 (6h14m56.884137386s) [noise=true , address=104.36.82.4:4804 , piece_id=IU6FDNVB5AUPA7MJ6RSXUTLBETPHOFGNJ7YGWBFIZSQFSO6W6BBA, seg_pos={0 0}, upload=5895]: Done uploading piece. err: <nil> Jul 27 04:06:43.191702285 (6h14m58.972635651s) [noise=true , address=104.36.82.6:4873 , piece_id=YP4VJORZZUL5EVCH3P7LD3LNVMHB7CWRTKHGIA7UROXE44N6MMZA, seg_pos={0 0}, upload=5895]: Done uploading piece. err: <nil> Jul 27 04:06:53.777973493 (6h15m9.558906861s) [noise=true , address=104.36.82.11:4115 , piece_id=YXMNVT6ZHGESGQ2BXKZUVR7U2CNNAI2PO7ZTMDG5UIOV7HTDDUTA, seg_pos={0 0}, upload=5895]: Done uploading piece. err: ecclient: upload failed (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR, address:104.36.82.11:4115): protocol: write tcp 159.223.143.39:34210->104.36.82.11:4115: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 04:06:53.778019532 (6h15m9.558952909s) [noise=true , address=104.36.82.12:4151 , piece_id=E2TEUTQBD4Y4AUHSCPMCIFEXBSY7S5HJWBKNKWWTUWEW2BLAAE6Q, seg_pos={0 0}, upload=5895]: Done uploading piece. err: ecclient: upload failed (node:1w6pwZMdeQKjt92qx6WbTWU5VPYJto5gS87f6iKvCBZH6LfyeB, address:104.36.82.12:4151): protocol: rpc: dial tcp 104.36.82.12:4151: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 04:06:53.778043653 (6h15m9.558977021s) [noise=true , address=104.36.82.13:4186 , piece_id=TBLDKALGJYBGR5NLCOCKJ4QKEFNIJXCKAFUBD7VR3UMNWEMKMKHQ, seg_pos={0 0}, upload=5895]: Done uploading piece. err: ecclient: upload failed (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht, address:104.36.82.13:4186): protocol: read tcp 159.223.143.39:34340->104.36.82.13:4186: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 04:06:53.778095228 (6h15m9.559028593s) [noise=true , address=104.36.82.10:4079 , piece_id=2SH5D6JOMLFR7VSN7ZY5CLMR5RHIRFYQ7Z5I2JIYTJQM43G4O6OQ, seg_pos={0 0}, upload=5895]: Done uploading piece. err: ecclient: upload failed (node:12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud, address:104.36.82.10:4079): protocol: write tcp 159.223.143.39:53840->104.36.82.10:4079: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 04:06:53.778142167 (6h15m9.559075532s) [seg_pos={0 0}, upload=5895]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 27 04:06:53.837032161 (6h15m9.617965543s) [upload=5895]: Done uploading segments... Jul 27 04:06:53.837055996 (6h15m9.617989366s) [upload=5895]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 422.917036ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................................................X..........] duration: 445.317896ms FAIL (2025-07-27 04:23:06 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 12) Duration: 445.351135ms core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.......................................X....................] duration: 704.072934ms FAIL (2025-07-27 04:22:49 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== {0 0}, upload=5911]: Begin upload segment... Jul 27 04:07:17.745307658 (6h15m33.526241043s) [seg_pos={0 0}, upload=5911]: Done begin upload segment. Jul 27 04:07:17.745349241 (6h15m33.526282625s) [noise=true , address=104.36.82.52:4098 , piece_id=M2ETF25QQ53RRGFPUKXJRINIWV7AWXSECEDR2AIM645IED72NURQ, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.745439694 (6h15m33.526373073s) [noise=true , address=104.36.82.56:4244 , piece_id=LQCZ5W54NBHWAVBVYK7HYSSJDC73E6FV4CP4YIM4L5H2N3OLOZXA, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.745972271 (6h15m33.526905654s) [noise=true , address=104.36.82.36:4650 , piece_id=BKJULWGPESBINNTSUYP6E4KZRXKPACVU5I7DOSZBO4YMK2V26HEQ, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.745980243 (6h15m33.526913621s) [noise=true , address=104.36.82.38:4722 , piece_id=B2MBQY5CTMEI6WSE7FTLOV5RUJ22SCNG7N2EEJILEATGJ4TIALXA, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.746403818 (6h15m33.527337198s) [noise=true , address=104.36.82.22:4534 , piece_id=CW4AQT3XUGA4A5OC6NUY4NYOOAMLN5FFJJN5PQBYEON7HPD5LCYQ, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.746455493 (6h15m33.527388875s) [noise=true , address=104.36.82.14:4248 , piece_id=LDSGCUM7Q2VX5CDFI4JRGEEG7D5H5Q5Z5WUWKU277VN2JFRX6UGA, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.746872127 (6h15m33.527805505s) [noise=true , address=104.36.82.40:4954 , piece_id=YLOHTJLRAED66OMEQYPZHNNRI77V6WOR4EHTEZOOEHMGZHJ2O3QQ, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.747123162 (6h15m33.52805654s) [noise=true , address=104.36.82.46:4496 , piece_id=GWNQR5WRGRRQBLSGWJFD3O3UNCMNE62JQNPIV5SYKBUIL3BQKA5A, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.747280678 (6h15m33.528214053s) [noise=true , address=104.36.82.47:4524 , piece_id=DONJFTKUQ2XCW6BOU7AVOKJ7XMDMBPGQVUPRANH7X2YS7PJXUKIA, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.747736400 (6h15m33.528669779s) [noise=true , address=104.36.82.44:4893 , piece_id=HUJICRJ6W54O3AZZR6H3KG6E4TGBTNS46O6XVMQM3PFAZE7A2HZA, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.747755734 (6h15m33.528689109s) [noise=true , address=104.36.82.55:4204 , piece_id=72WISGRHMOAZDV5EDGTBOMLBL4HDOF3CBEMQAAIRGILPC7J5JKAQ, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.748313124 (6h15m33.5292465s) [noise=true , address=104.36.82.58:4287 , piece_id=BVDYUEWU7TVEQDMZXX5SJNS4QICVKMTGFLCUV6P2YHOE7TI25VXA, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.748485217 (6h15m33.529418594s) [noise=true , address=104.36.82.53:4123 , piece_id=NXJRZ5BQSLXSOIOSUJ4MM4AGKHP4UIDUGEJKLUUX7YOGQJA6WWZA, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.748831329 (6h15m33.529764704s) [noise=true , address=104.36.82.42:4829 , piece_id=ZLWJFP3LGX3YLXBJYIR66DXTYEVDBKXKVXCQSF2PCYCYJBIEJGMA, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.749299293 (6h15m33.530232669s) [noise=true , address=104.36.82.8:4002 , piece_id=6XSPWQC3US355PS6IQ4K3IYNQZH5MAT4IVIPGSYRUM3XYPSCWE6Q, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.749790974 (6h15m33.53072436s) [noise=true , address=104.36.82.44:4901 , piece_id=N4YLCLR2QZSULUL2XWONEKBFUMNOU7KUX2QEOK4SSDFSCI7IL6LA, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.750478221 (6h15m33.531411597s) [noise=true , address=104.36.82.54:4172 , piece_id=ZNA53MWDRWPUH6FUR4DJERBDT2IOYGA3IYUADKXXIOLQJLQOXE7Q, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.752884255 (6h15m33.533817625s) [noise=true , address=64.49.13.140:4508 , piece_id=MZ52U75VOIK6Q7QFXMMYRRG7JXFWDFU2JWQFYPERUCMH3BUX6XPQ, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.879839555 (6h15m33.660772917s) [upload=5911]: Next returned nil segment Jul 27 04:07:17.879861389 (6h15m33.660794765s) [upload=5911]: Waiting for error group managing segments... Jul 27 04:07:17.888652438 (6h15m33.669585811s) [noise=true , address=104.36.82.58:4287 , piece_id=BVDYUEWU7TVEQDMZXX5SJNS4QICVKMTGFLCUV6P2YHOE7TI25VXA, seg_pos={0 0}, upload=5911]: Done uploading piece. err: <nil> Jul 27 04:07:17.952192045 (6h15m33.733125429s) [noise=true , address=104.36.82.53:4123 , piece_id=NXJRZ5BQSLXSOIOSUJ4MM4AGKHP4UIDUGEJKLUUX7YOGQJA6WWZA, seg_pos={0 0}, upload=5911]: Done uploading piece. err: <nil> Jul 27 04:07:17.981907539 (6h15m33.762840911s) [noise=true , address=104.36.82.56:4244 , piece_id=LQCZ5W54NBHWAVBVYK7HYSSJDC73E6FV4CP4YIM4L5H2N3OLOZXA, seg_pos={0 0}, upload=5911]: Done uploading piece. err: <nil> Jul 27 04:07:17.982370496 (6h15m33.7633039s) [noise=true , address=104.36.82.55:4204 , piece_id=72WISGRHMOAZDV5EDGTBOMLBL4HDOF3CBEMQAAIRGILPC7J5JKAQ, seg_pos={0 0}, upload=5911]: Done uploading piece. err: <nil> Jul 27 04:07:18.009548506 (6h15m33.790481886s) [noise=true , address=104.36.82.52:4098 , piece_id=M2ETF25QQ53RRGFPUKXJRINIWV7AWXSECEDR2AIM645IED72NURQ, seg_pos={0 0}, upload=5911]: Done uploading piece. err: <nil> Jul 27 04:07:18.023770598 (6h15m33.804703983s) [noise=true , address=104.36.82.40:4954 , piece_id=YLOHTJLRAED66OMEQYPZHNNRI77V6WOR4EHTEZOOEHMGZHJ2O3QQ, seg_pos={0 0}, upload=5911]: Done uploading piece. err: <nil> Jul 27 04:07:18.139745581 (6h15m33.92067895s) [noise=true , address=104.36.82.22:4534 , piece_id=CW4AQT3XUGA4A5OC6NUY4NYOOAMLN5FFJJN5PQBYEON7HPD5LCYQ, seg_pos={0 0}, upload=5911]: Done uploading piece. err: <nil> Jul 27 04:07:18.423748140 (6h15m34.204681511s) [noise=true , address=64.49.13.140:4508 , piece_id=MZ52U75VOIK6Q7QFXMMYRRG7JXFWDFU2JWQFYPERUCMH3BUX6XPQ, seg_pos={0 0}, upload=5911]: Done uploading piece. err: <nil> Jul 27 04:08:16.739940381 (6h16m32.52087375s) [noise=true , address=104.36.82.54:4172 , piece_id=ZNA53MWDRWPUH6FUR4DJERBDT2IOYGA3IYUADKXXIOLQJLQOXE7Q, seg_pos={0 0}, upload=5911]: Done uploading piece. err: <nil> Jul 27 04:08:17.177295397 (6h16m32.958228766s) [noise=true , address=104.36.82.8:4002 , piece_id=6XSPWQC3US355PS6IQ4K3IYNQZH5MAT4IVIPGSYRUM3XYPSCWE6Q, seg_pos={0 0}, upload=5911]: Done uploading piece. err: <nil> Jul 27 04:08:17.534296584 (6h16m33.315229953s) [noise=true , address=104.36.82.36:4650 , piece_id=BKJULWGPESBINNTSUYP6E4KZRXKPACVU5I7DOSZBO4YMK2V26HEQ, seg_pos={0 0}, upload=5911]: Done uploading piece. err: <nil> Jul 27 04:08:17.677851566 (6h16m33.458784943s) [noise=true , address=104.36.82.14:4248 , piece_id=LDSGCUM7Q2VX5CDFI4JRGEEG7D5H5Q5Z5WUWKU277VN2JFRX6UGA, seg_pos={0 0}, upload=5911]: Done uploading piece. err: ecclient: upload failed (node:1u6ir2PLqDa9xGm64cuqq1RqY4qeBa1fqCo9VghspBf54XEXqn, address:104.36.82.14:4248): protocol: rpc: dial tcp 104.36.82.14:4248: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 04:08:17.678115170 (6h16m33.459048549s) [noise=true , address=104.36.82.46:4496 , piece_id=GWNQR5WRGRRQBLSGWJFD3O3UNCMNE62JQNPIV5SYKBUIL3BQKA5A, seg_pos={0 0}, upload=5911]: Done uploading piece. err: ecclient: upload failed (node:12V8XhSWkyGUVYFR8MykifFAP1YAfGCwzqucDcMMfzw3xsVW77Z, address:104.36.82.46:4496): protocol: read tcp 159.223.143.39:52800->104.36.82.46:4496: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 04:08:17.678144409 (6h16m33.459077789s) [noise=true , address=104.36.82.44:4901 , piece_id=N4YLCLR2QZSULUL2XWONEKBFUMNOU7KUX2QEOK4SSDFSCI7IL6LA, seg_pos={0 0}, upload=5911]: Done uploading piece. err: ecclient: upload failed (node:1FHE84zvBb1MKwB1Ht7puTafgQxDEvPt4mYqkYVSSufAMpWQA1, address:104.36.82.44:4901): protocol: rpc: dial tcp 104.36.82.44:4901: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 04:08:17.678229622 (6h16m33.459162988s) [noise=true , address=104.36.82.44:4893 , piece_id=HUJICRJ6W54O3AZZR6H3KG6E4TGBTNS46O6XVMQM3PFAZE7A2HZA, seg_pos={0 0}, upload=5911]: Done uploading piece. err: ecclient: upload failed (node:17VMFZ2T8jrSJVtchciJ9SnyrR6JstvcMoXrN5Q3zTD1aQU1oJ, address:104.36.82.44:4893): protocol: write tcp 159.223.143.39:35302->104.36.82.44:4893: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 04:08:17.678310218 (6h16m33.459243583s) [noise=true , address=104.36.82.42:4829 , piece_id=ZLWJFP3LGX3YLXBJYIR66DXTYEVDBKXKVXCQSF2PCYCYJBIEJGMA, seg_pos={0 0}, upload=5911]: Done uploading piece. err: ecclient: upload failed (node:1F5T8Vt1x4nXFaSenvGBCYjxhRL1H1sxRvzLjW3CqzwDVhq39F, address:104.36.82.42:4829): protocol: rpc: dial tcp 104.36.82.42:4829: i/o timeout; conn closed: op; piecestore: piecestore close: context deadline exceeded Jul 27 04:08:17.678412989 (6h16m33.459346365s) [noise=true , address=104.36.82.38:4722 , piece_id=B2MBQY5CTMEI6WSE7FTLOV5RUJ22SCNG7N2EEJILEATGJ4TIALXA, seg_pos={0 0}, upload=5911]: Done uploading piece. err: ecclient: upload failed (node:12Zw3dtUg9gARz52KhwDLTjcqC3CNm7dpPzpdx7P7PbL1jwe8uc, address:104.36.82.38:4722): protocol: read tcp 159.223.143.39:56048->104.36.82.38:4722: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 04:08:17.678455247 (6h16m33.459388624s) [noise=true , address=104.36.82.47:4524 , piece_id=DONJFTKUQ2XCW6BOU7AVOKJ7XMDMBPGQVUPRANH7X2YS7PJXUKIA, seg_pos={0 0}, upload=5911]: Done uploading piece. err: ecclient: upload failed (node:1xfbCzjSwFDRrR1gPggeN8tNCLbRRWDVzCJ6mqhZsbVSe8d4s4, address:104.36.82.47:4524): protocol: read tcp 159.223.143.39:46694->104.36.82.47:4524: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 04:08:17.678510668 (6h16m33.459444052s) [seg_pos={0 0}, upload=5911]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Jul 27 04:08:17.706442864 (6h16m33.487376232s) [upload=5911]: Done uploading segments... Jul 27 04:08:17.706495329 (6h16m33.487428705s) [upload=5911]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 704.104831ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................................................X.........] duration: 5.302915355s FAIL (2025-07-27 04:22:40 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.302948524s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................................X........................] duration: 399.3981ms FAIL (2025-07-27 04:22:37 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== ece_id=NYOFDEZ7CV4PHGQ6JZMEEVAYGJRPYG6IBW7MYJO42HMWOJM7PHZA, seg_pos={0 0}, upload=5879]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12crcXpNe9NgeXTsUpooLSBgrSUFbRxbSbanLa39w2Hr5mGFww): protocol: read tcp 159.223.143.39:53630->104.36.82.5:4840: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 04:05:32.508904485 (6h13m48.289837866s) [seg_pos={0 0}, upload=5879]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 27 04:05:32.509028070 (6h13m48.289961457s) [upload=5879]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 27 04:05:32.509034645 (6h13m48.289968129s) [upload=5879]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 27 04:05:32.581834839 (6h13m48.362768225s) [upload=5879]: Done uploading segments... Jul 27 04:05:32.581875919 (6h13m48.362809298s) [upload=5879]: Upload finished. err: <nil> Jul 27 04:06:23.829155331 (6h14m39.610088696s) [upload=5895]: Starting upload Jul 27 04:06:23.829225021 (6h14m39.610158398s) [upload=5895]: Done starting upload Jul 27 04:06:23.829294606 (6h14m39.61022799s) [upload=5895]: Uploading segments... Jul 27 04:06:23.829577656 (6h14m39.610511053s) [upload=5895]: Got next segment. Inline: false Jul 27 04:06:23.829649664 (6h14m39.610583027s) [upload=5895]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 27 04:06:23.829657065 (6h14m39.61059043s) [upload=5895]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 27 04:06:23.861405530 (6h14m39.642338913s) [seg_pos={0 0}, upload=5895]: Begin upload segment... Jul 27 04:06:23.861476351 (6h14m39.642409713s) [seg_pos={0 0}, upload=5895]: Done begin upload segment. Jul 27 04:06:23.861509120 (6h14m39.642442506s) [noise=true , address=77.90.31.19:4843 , piece_id=YML5RDQIK432Z4FSE4DKXIV5U47LQBEZRFMAVLANUSGFUQR23UWQ, seg_pos={0 0}, upload=5895]: Uploading piece... Jul 27 04:06:23.861953723 (6h14m39.6428871s) [noise=true , address=45.91.116.10:4081 , piece_id=XGDTSEZOUNRTEVWHTO3DHTZMH2VZZOPJPEKLCFAHTXESLLYABHSA, seg_pos={0 0}, upload=5895]: Uploading piece... Jul 27 04:06:23.862060983 (6h14m39.642994358s) [noise=true , address=45.91.116.9:4262 , piece_id=M5ETD45HH2XZD6HDLTJ3JO2BOOIROXFWKVPTN42PSQEWG4F7KAAQ, seg_pos={0 0}, upload=5895]: Uploading piece... Jul 27 04:06:23.862759464 (6h14m39.64369284s) [noise=true , address=104.36.82.4:4804 , piece_id=IU6FDNVB5AUPA7MJ6RSXUTLBETPHOFGNJ7YGWBFIZSQFSO6W6BBA, seg_pos={0 0}, upload=5895]: Uploading piece... Jul 27 04:06:23.862835086 (6h14m39.643768461s) [noise=true , address=104.36.82.6:4873 , piece_id=YP4VJORZZUL5EVCH3P7LD3LNVMHB7CWRTKHGIA7UROXE44N6MMZA, seg_pos={0 0}, upload=5895]: Uploading piece... Jul 27 04:06:23.863593880 (6h14m39.644527258s) [noise=true , address=77.90.31.15:4697 , piece_id=Q76TJDSZPSYZSJQNFVOI462U6DFILMH2RV3LQISODXSOCXGRKW6Q, seg_pos={0 0}, upload=5895]: Uploading piece... Jul 27 04:06:23.863656891 (6h14m39.644590266s) [noise=true , address=45.91.116.7:4011 , piece_id=XMH6MCCHZBBRQWHJFWLRGYHLQD4KUGSGL7MMZSKA5TZJCND3DUDA, seg_pos={0 0}, upload=5895]: Uploading piece... Jul 27 04:06:23.864408612 (6h14m39.645341987s) [noise=true , address=45.91.116.6:4973 , piece_id=J3VULPU24CIM64VEVE3UHO62HPN6TJXGZU6BCJIS5JQJKHJK2LIQ, seg_pos={0 0}, upload=5895]: Uploading piece... Jul 27 04:06:23.864521549 (6h14m39.64545491s) [noise=true , address=104.36.82.10:4079 , piece_id=2SH5D6JOMLFR7VSN7ZY5CLMR5RHIRFYQ7Z5I2JIYTJQM43G4O6OQ, seg_pos={0 0}, upload=5895]: Uploading piece... Jul 27 04:06:23.865518491 (6h14m39.646451872s) [noise=true , address=77.90.31.17:4771 , piece_id=U2LF22GKCMKRZN6EWCF26ITDZQHBFZXCGSQXT33MCX6P57IGI35Q, seg_pos={0 0}, upload=5895]: Uploading piece... Jul 27 04:06:23.865654950 (6h14m39.646588325s) [noise=true , address=45.91.116.8:4048 , piece_id=5REWMVVTBVLDQXQ5KAEHLY4VJL5SGF7ONQWDQQ3N2FDU3OAO6AZA, seg_pos={0 0}, upload=5895]: Uploading piece... Jul 27 04:06:23.866190892 (6h14m39.647124271s) [noise=true , address=104.36.82.12:4151 , piece_id=E2TEUTQBD4Y4AUHSCPMCIFEXBSY7S5HJWBKNKWWTUWEW2BLAAE6Q, seg_pos={0 0}, upload=5895]: Uploading piece... Jul 27 04:06:23.866412590 (6h14m39.647345968s) [noise=true , address=104.36.82.11:4115 , piece_id=YXMNVT6ZHGESGQ2BXKZUVR7U2CNNAI2PO7ZTMDG5UIOV7HTDDUTA, seg_pos={0 0}, upload=5895]: Uploading piece... Jul 27 04:06:23.866715972 (6h14m39.647649359s) [noise=true , address=45.91.116.13:4153 , piece_id=H4535KOCG3F4RKA6AKKMTJSOM6Q6PGRC6M7FR3UIS3BORTSP5UQA, seg_pos={0 0}, upload=5895]: Uploading piece... Jul 27 04:06:23.866998965 (6h14m39.64793234s) [noise=true , address=104.36.82.13:4186 , piece_id=TBLDKALGJYBGR5NLCOCKJ4QKEFNIJXCKAFUBD7VR3UMNWEMKMKHQ, seg_pos={0 0}, upload=5895]: Uploading piece... Jul 27 04:06:23.867230593 (6h14m39.648163974s) [noise=true , address=104.36.82.16:4293 , piece_id=AHH6PRFUSGTDGB7H52DRITARA4IF6GFLVSRYS2Y6EPJCZSKUOYWQ, seg_pos={0 0}, upload=5895]: Uploading piece... Jul 27 04:06:23.949422808 (6h14m39.730356187s) [upload=5895]: Next returned nil segment Jul 27 04:06:23.949442576 (6h14m39.73037596s) [upload=5895]: Waiting for error group managing segments... Jul 27 04:06:23.953497426 (6h14m39.734430802s) [noise=true , address=45.91.116.10:4081 , piece_id=XGDTSEZOUNRTEVWHTO3DHTZMH2VZZOPJPEKLCFAHTXESLLYABHSA, seg_pos={0 0}, upload=5895]: Done uploading piece. err: <nil> Jul 27 04:06:23.953531796 (6h14m39.734465159s) [noise=true , address=45.91.116.6:4973 , piece_id=J3VULPU24CIM64VEVE3UHO62HPN6TJXGZU6BCJIS5JQJKHJK2LIQ, seg_pos={0 0}, upload=5895]: Done uploading piece. err: <nil> Jul 27 04:06:23.954068660 (6h14m39.735002029s) [noise=true , address=45.91.116.9:4262 , piece_id=M5ETD45HH2XZD6HDLTJ3JO2BOOIROXFWKVPTN42PSQEWG4F7KAAQ, seg_pos={0 0}, upload=5895]: Done uploading piece. err: <nil> Jul 27 04:06:23.954194339 (6h14m39.735127707s) [noise=true , address=45.91.116.8:4048 , piece_id=5REWMVVTBVLDQXQ5KAEHLY4VJL5SGF7ONQWDQQ3N2FDU3OAO6AZA, seg_pos={0 0}, upload=5895]: Done uploading piece. err: <nil> Jul 27 04:06:23.954601730 (6h14m39.735535112s) [noise=true , address=45.91.116.7:4011 , piece_id=XMH6MCCHZBBRQWHJFWLRGYHLQD4KUGSGL7MMZSKA5TZJCND3DUDA, seg_pos={0 0}, upload=5895]: Done uploading piece. err: <nil> Jul 27 04:06:23.954755872 (6h14m39.735689237s) [noise=true , address=45.91.116.13:4153 , piece_id=H4535KOCG3F4RKA6AKKMTJSOM6Q6PGRC6M7FR3UIS3BORTSP5UQA, seg_pos={0 0}, upload=5895]: Done uploading piece. err: <nil> Jul 27 04:06:23.955611883 (6h14m39.73654526s) [noise=true , address=77.90.31.19:4843 , piece_id=YML5RDQIK432Z4FSE4DKXIV5U47LQBEZRFMAVLANUSGFUQR23UWQ, seg_pos={0 0}, upload=5895]: Done uploading piece. err: <nil> Jul 27 04:06:23.956373391 (6h14m39.737306756s) [noise=true , address=77.90.31.15:4697 , piece_id=Q76TJDSZPSYZSJQNFVOI462U6DFILMH2RV3LQISODXSOCXGRKW6Q, seg_pos={0 0}, upload=5895]: Done uploading piece. err: <nil> Jul 27 04:06:23.958659122 (6h14m39.73959249s) [noise=true , address=77.90.31.17:4771 , piece_id=U2LF22GKCMKRZN6EWCF26ITDZQHBFZXCGSQXT33MCX6P57IGI35Q, seg_pos={0 0}, upload=5895]: Done uploading piece. err: <nil> Jul 27 04:06:35.058944013 (6h14m50.83987739s) [noise=true , address=104.36.82.16:4293 , piece_id=AHH6PRFUSGTDGB7H52DRITARA4IF6GFLVSRYS2Y6EPJCZSKUOYWQ, seg_pos={0 0}, upload=5895]: Done uploading piece. err: <nil> Jul 27 04:06:41.103204002 (6h14m56.884137386s) [noise=true , address=104.36.82.4:4804 , piece_id=IU6FDNVB5AUPA7MJ6RSXUTLBETPHOFGNJ7YGWBFIZSQFSO6W6BBA, seg_pos={0 0}, upload=5895]: Done uploading piece. err: <nil> Jul 27 04:06:43.191702285 (6h14m58.972635651s) [noise=true , address=104.36.82.6:4873 , piece_id=YP4VJORZZUL5EVCH3P7LD3LNVMHB7CWRTKHGIA7UROXE44N6MMZA, seg_pos={0 0}, upload=5895]: Done uploading piece. err: <nil> Jul 27 04:06:53.777973493 (6h15m9.558906861s) [noise=true , address=104.36.82.11:4115 , piece_id=YXMNVT6ZHGESGQ2BXKZUVR7U2CNNAI2PO7ZTMDG5UIOV7HTDDUTA, seg_pos={0 0}, upload=5895]: Done uploading piece. err: ecclient: upload failed (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR, address:104.36.82.11:4115): protocol: write tcp 159.223.143.39:34210->104.36.82.11:4115: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 04:06:53.778019532 (6h15m9.558952909s) [noise=true , address=104.36.82.12:4151 , piece_id=E2TEUTQBD4Y4AUHSCPMCIFEXBSY7S5HJWBKNKWWTUWEW2BLAAE6Q, seg_pos={0 0}, upload=5895]: Done uploading piece. err: ecclient: upload failed (node:1w6pwZMdeQKjt92qx6WbTWU5VPYJto5gS87f6iKvCBZH6LfyeB, address:104.36.82.12:4151): protocol: rpc: dial tcp 104.36.82.12:4151: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 04:06:53.778043653 (6h15m9.558977021s) [noise=true , address=104.36.82.13:4186 , piece_id=TBLDKALGJYBGR5NLCOCKJ4QKEFNIJXCKAFUBD7VR3UMNWEMKMKHQ, seg_pos={0 0}, upload=5895]: Done uploading piece. err: ecclient: upload failed (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht, address:104.36.82.13:4186): protocol: read tcp 159.223.143.39:34340->104.36.82.13:4186: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 04:06:53.778095228 (6h15m9.559028593s) [noise=true , address=104.36.82.10:4079 , piece_id=2SH5D6JOMLFR7VSN7ZY5CLMR5RHIRFYQ7Z5I2JIYTJQM43G4O6OQ, seg_pos={0 0}, upload=5895]: Done uploading piece. err: ecclient: upload failed (node:12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud, address:104.36.82.10:4079): protocol: write tcp 159.223.143.39:53840->104.36.82.10:4079: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 04:06:53.778142167 (6h15m9.559075532s) [seg_pos={0 0}, upload=5895]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 27 04:06:53.837032161 (6h15m9.617965543s) [upload=5895]: Done uploading segments... Jul 27 04:06:53.837055996 (6h15m9.617989366s) [upload=5895]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 399.428295ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................................X.......................] duration: 25.85169731s FAIL (2025-07-27 04:22:36 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== ece_id=NYOFDEZ7CV4PHGQ6JZMEEVAYGJRPYG6IBW7MYJO42HMWOJM7PHZA, seg_pos={0 0}, upload=5879]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12crcXpNe9NgeXTsUpooLSBgrSUFbRxbSbanLa39w2Hr5mGFww): protocol: read tcp 159.223.143.39:53630->104.36.82.5:4840: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 04:05:32.508904485 (6h13m48.289837866s) [seg_pos={0 0}, upload=5879]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 27 04:05:32.509028070 (6h13m48.289961457s) [upload=5879]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 27 04:05:32.509034645 (6h13m48.289968129s) [upload=5879]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 27 04:05:32.581834839 (6h13m48.362768225s) [upload=5879]: Done uploading segments... Jul 27 04:05:32.581875919 (6h13m48.362809298s) [upload=5879]: Upload finished. err: <nil> Jul 27 04:06:23.829155331 (6h14m39.610088696s) [upload=5895]: Starting upload Jul 27 04:06:23.829225021 (6h14m39.610158398s) [upload=5895]: Done starting upload Jul 27 04:06:23.829294606 (6h14m39.61022799s) [upload=5895]: Uploading segments... Jul 27 04:06:23.829577656 (6h14m39.610511053s) [upload=5895]: Got next segment. Inline: false Jul 27 04:06:23.829649664 (6h14m39.610583027s) [upload=5895]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 27 04:06:23.829657065 (6h14m39.61059043s) [upload=5895]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 27 04:06:23.861405530 (6h14m39.642338913s) [seg_pos={0 0}, upload=5895]: Begin upload segment... Jul 27 04:06:23.861476351 (6h14m39.642409713s) [seg_pos={0 0}, upload=5895]: Done begin upload segment. Jul 27 04:06:23.861509120 (6h14m39.642442506s) [noise=true , address=77.90.31.19:4843 , piece_id=YML5RDQIK432Z4FSE4DKXIV5U47LQBEZRFMAVLANUSGFUQR23UWQ, seg_pos={0 0}, upload=5895]: Uploading piece... Jul 27 04:06:23.861953723 (6h14m39.6428871s) [noise=true , address=45.91.116.10:4081 , piece_id=XGDTSEZOUNRTEVWHTO3DHTZMH2VZZOPJPEKLCFAHTXESLLYABHSA, seg_pos={0 0}, upload=5895]: Uploading piece... Jul 27 04:06:23.862060983 (6h14m39.642994358s) [noise=true , address=45.91.116.9:4262 , piece_id=M5ETD45HH2XZD6HDLTJ3JO2BOOIROXFWKVPTN42PSQEWG4F7KAAQ, seg_pos={0 0}, upload=5895]: Uploading piece... Jul 27 04:06:23.862759464 (6h14m39.64369284s) [noise=true , address=104.36.82.4:4804 , piece_id=IU6FDNVB5AUPA7MJ6RSXUTLBETPHOFGNJ7YGWBFIZSQFSO6W6BBA, seg_pos={0 0}, upload=5895]: Uploading piece... Jul 27 04:06:23.862835086 (6h14m39.643768461s) [noise=true , address=104.36.82.6:4873 , piece_id=YP4VJORZZUL5EVCH3P7LD3LNVMHB7CWRTKHGIA7UROXE44N6MMZA, seg_pos={0 0}, upload=5895]: Uploading piece... Jul 27 04:06:23.863593880 (6h14m39.644527258s) [noise=true , address=77.90.31.15:4697 , piece_id=Q76TJDSZPSYZSJQNFVOI462U6DFILMH2RV3LQISODXSOCXGRKW6Q, seg_pos={0 0}, upload=5895]: Uploading piece... Jul 27 04:06:23.863656891 (6h14m39.644590266s) [noise=true , address=45.91.116.7:4011 , piece_id=XMH6MCCHZBBRQWHJFWLRGYHLQD4KUGSGL7MMZSKA5TZJCND3DUDA, seg_pos={0 0}, upload=5895]: Uploading piece... Jul 27 04:06:23.864408612 (6h14m39.645341987s) [noise=true , address=45.91.116.6:4973 , piece_id=J3VULPU24CIM64VEVE3UHO62HPN6TJXGZU6BCJIS5JQJKHJK2LIQ, seg_pos={0 0}, upload=5895]: Uploading piece... Jul 27 04:06:23.864521549 (6h14m39.64545491s) [noise=true , address=104.36.82.10:4079 , piece_id=2SH5D6JOMLFR7VSN7ZY5CLMR5RHIRFYQ7Z5I2JIYTJQM43G4O6OQ, seg_pos={0 0}, upload=5895]: Uploading piece... Jul 27 04:06:23.865518491 (6h14m39.646451872s) [noise=true , address=77.90.31.17:4771 , piece_id=U2LF22GKCMKRZN6EWCF26ITDZQHBFZXCGSQXT33MCX6P57IGI35Q, seg_pos={0 0}, upload=5895]: Uploading piece... Jul 27 04:06:23.865654950 (6h14m39.646588325s) [noise=true , address=45.91.116.8:4048 , piece_id=5REWMVVTBVLDQXQ5KAEHLY4VJL5SGF7ONQWDQQ3N2FDU3OAO6AZA, seg_pos={0 0}, upload=5895]: Uploading piece... Jul 27 04:06:23.866190892 (6h14m39.647124271s) [noise=true , address=104.36.82.12:4151 , piece_id=E2TEUTQBD4Y4AUHSCPMCIFEXBSY7S5HJWBKNKWWTUWEW2BLAAE6Q, seg_pos={0 0}, upload=5895]: Uploading piece... Jul 27 04:06:23.866412590 (6h14m39.647345968s) [noise=true , address=104.36.82.11:4115 , piece_id=YXMNVT6ZHGESGQ2BXKZUVR7U2CNNAI2PO7ZTMDG5UIOV7HTDDUTA, seg_pos={0 0}, upload=5895]: Uploading piece... Jul 27 04:06:23.866715972 (6h14m39.647649359s) [noise=true , address=45.91.116.13:4153 , piece_id=H4535KOCG3F4RKA6AKKMTJSOM6Q6PGRC6M7FR3UIS3BORTSP5UQA, seg_pos={0 0}, upload=5895]: Uploading piece... Jul 27 04:06:23.866998965 (6h14m39.64793234s) [noise=true , address=104.36.82.13:4186 , piece_id=TBLDKALGJYBGR5NLCOCKJ4QKEFNIJXCKAFUBD7VR3UMNWEMKMKHQ, seg_pos={0 0}, upload=5895]: Uploading piece... Jul 27 04:06:23.867230593 (6h14m39.648163974s) [noise=true , address=104.36.82.16:4293 , piece_id=AHH6PRFUSGTDGB7H52DRITARA4IF6GFLVSRYS2Y6EPJCZSKUOYWQ, seg_pos={0 0}, upload=5895]: Uploading piece... Jul 27 04:06:23.949422808 (6h14m39.730356187s) [upload=5895]: Next returned nil segment Jul 27 04:06:23.949442576 (6h14m39.73037596s) [upload=5895]: Waiting for error group managing segments... Jul 27 04:06:23.953497426 (6h14m39.734430802s) [noise=true , address=45.91.116.10:4081 , piece_id=XGDTSEZOUNRTEVWHTO3DHTZMH2VZZOPJPEKLCFAHTXESLLYABHSA, seg_pos={0 0}, upload=5895]: Done uploading piece. err: <nil> Jul 27 04:06:23.953531796 (6h14m39.734465159s) [noise=true , address=45.91.116.6:4973 , piece_id=J3VULPU24CIM64VEVE3UHO62HPN6TJXGZU6BCJIS5JQJKHJK2LIQ, seg_pos={0 0}, upload=5895]: Done uploading piece. err: <nil> Jul 27 04:06:23.954068660 (6h14m39.735002029s) [noise=true , address=45.91.116.9:4262 , piece_id=M5ETD45HH2XZD6HDLTJ3JO2BOOIROXFWKVPTN42PSQEWG4F7KAAQ, seg_pos={0 0}, upload=5895]: Done uploading piece. err: <nil> Jul 27 04:06:23.954194339 (6h14m39.735127707s) [noise=true , address=45.91.116.8:4048 , piece_id=5REWMVVTBVLDQXQ5KAEHLY4VJL5SGF7ONQWDQQ3N2FDU3OAO6AZA, seg_pos={0 0}, upload=5895]: Done uploading piece. err: <nil> Jul 27 04:06:23.954601730 (6h14m39.735535112s) [noise=true , address=45.91.116.7:4011 , piece_id=XMH6MCCHZBBRQWHJFWLRGYHLQD4KUGSGL7MMZSKA5TZJCND3DUDA, seg_pos={0 0}, upload=5895]: Done uploading piece. err: <nil> Jul 27 04:06:23.954755872 (6h14m39.735689237s) [noise=true , address=45.91.116.13:4153 , piece_id=H4535KOCG3F4RKA6AKKMTJSOM6Q6PGRC6M7FR3UIS3BORTSP5UQA, seg_pos={0 0}, upload=5895]: Done uploading piece. err: <nil> Jul 27 04:06:23.955611883 (6h14m39.73654526s) [noise=true , address=77.90.31.19:4843 , piece_id=YML5RDQIK432Z4FSE4DKXIV5U47LQBEZRFMAVLANUSGFUQR23UWQ, seg_pos={0 0}, upload=5895]: Done uploading piece. err: <nil> Jul 27 04:06:23.956373391 (6h14m39.737306756s) [noise=true , address=77.90.31.15:4697 , piece_id=Q76TJDSZPSYZSJQNFVOI462U6DFILMH2RV3LQISODXSOCXGRKW6Q, seg_pos={0 0}, upload=5895]: Done uploading piece. err: <nil> Jul 27 04:06:23.958659122 (6h14m39.73959249s) [noise=true , address=77.90.31.17:4771 , piece_id=U2LF22GKCMKRZN6EWCF26ITDZQHBFZXCGSQXT33MCX6P57IGI35Q, seg_pos={0 0}, upload=5895]: Done uploading piece. err: <nil> Jul 27 04:06:35.058944013 (6h14m50.83987739s) [noise=true , address=104.36.82.16:4293 , piece_id=AHH6PRFUSGTDGB7H52DRITARA4IF6GFLVSRYS2Y6EPJCZSKUOYWQ, seg_pos={0 0}, upload=5895]: Done uploading piece. err: <nil> Jul 27 04:06:41.103204002 (6h14m56.884137386s) [noise=true , address=104.36.82.4:4804 , piece_id=IU6FDNVB5AUPA7MJ6RSXUTLBETPHOFGNJ7YGWBFIZSQFSO6W6BBA, seg_pos={0 0}, upload=5895]: Done uploading piece. err: <nil> Jul 27 04:06:43.191702285 (6h14m58.972635651s) [noise=true , address=104.36.82.6:4873 , piece_id=YP4VJORZZUL5EVCH3P7LD3LNVMHB7CWRTKHGIA7UROXE44N6MMZA, seg_pos={0 0}, upload=5895]: Done uploading piece. err: <nil> Jul 27 04:06:53.777973493 (6h15m9.558906861s) [noise=true , address=104.36.82.11:4115 , piece_id=YXMNVT6ZHGESGQ2BXKZUVR7U2CNNAI2PO7ZTMDG5UIOV7HTDDUTA, seg_pos={0 0}, upload=5895]: Done uploading piece. err: ecclient: upload failed (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR, address:104.36.82.11:4115): protocol: write tcp 159.223.143.39:34210->104.36.82.11:4115: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 04:06:53.778019532 (6h15m9.558952909s) [noise=true , address=104.36.82.12:4151 , piece_id=E2TEUTQBD4Y4AUHSCPMCIFEXBSY7S5HJWBKNKWWTUWEW2BLAAE6Q, seg_pos={0 0}, upload=5895]: Done uploading piece. err: ecclient: upload failed (node:1w6pwZMdeQKjt92qx6WbTWU5VPYJto5gS87f6iKvCBZH6LfyeB, address:104.36.82.12:4151): protocol: rpc: dial tcp 104.36.82.12:4151: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 04:06:53.778043653 (6h15m9.558977021s) [noise=true , address=104.36.82.13:4186 , piece_id=TBLDKALGJYBGR5NLCOCKJ4QKEFNIJXCKAFUBD7VR3UMNWEMKMKHQ, seg_pos={0 0}, upload=5895]: Done uploading piece. err: ecclient: upload failed (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht, address:104.36.82.13:4186): protocol: read tcp 159.223.143.39:34340->104.36.82.13:4186: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 04:06:53.778095228 (6h15m9.559028593s) [noise=true , address=104.36.82.10:4079 , piece_id=2SH5D6JOMLFR7VSN7ZY5CLMR5RHIRFYQ7Z5I2JIYTJQM43G4O6OQ, seg_pos={0 0}, upload=5895]: Done uploading piece. err: ecclient: upload failed (node:12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud, address:104.36.82.10:4079): protocol: write tcp 159.223.143.39:53840->104.36.82.10:4079: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 04:06:53.778142167 (6h15m9.559075532s) [seg_pos={0 0}, upload=5895]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 27 04:06:53.837032161 (6h15m9.617965543s) [upload=5895]: Done uploading segments... Jul 27 04:06:53.837055996 (6h15m9.617989366s) [upload=5895]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 25.851722471s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................................................X........] duration: 380.564062ms FAIL (2025-07-27 04:22:07 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 12) Duration: 380.70376ms core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [........................................X...................] duration: 633.711369ms FAIL (2025-07-27 04:21:49 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== {0 0}, upload=5911]: Begin upload segment... Jul 27 04:07:17.745307658 (6h15m33.526241043s) [seg_pos={0 0}, upload=5911]: Done begin upload segment. Jul 27 04:07:17.745349241 (6h15m33.526282625s) [noise=true , address=104.36.82.52:4098 , piece_id=M2ETF25QQ53RRGFPUKXJRINIWV7AWXSECEDR2AIM645IED72NURQ, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.745439694 (6h15m33.526373073s) [noise=true , address=104.36.82.56:4244 , piece_id=LQCZ5W54NBHWAVBVYK7HYSSJDC73E6FV4CP4YIM4L5H2N3OLOZXA, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.745972271 (6h15m33.526905654s) [noise=true , address=104.36.82.36:4650 , piece_id=BKJULWGPESBINNTSUYP6E4KZRXKPACVU5I7DOSZBO4YMK2V26HEQ, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.745980243 (6h15m33.526913621s) [noise=true , address=104.36.82.38:4722 , piece_id=B2MBQY5CTMEI6WSE7FTLOV5RUJ22SCNG7N2EEJILEATGJ4TIALXA, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.746403818 (6h15m33.527337198s) [noise=true , address=104.36.82.22:4534 , piece_id=CW4AQT3XUGA4A5OC6NUY4NYOOAMLN5FFJJN5PQBYEON7HPD5LCYQ, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.746455493 (6h15m33.527388875s) [noise=true , address=104.36.82.14:4248 , piece_id=LDSGCUM7Q2VX5CDFI4JRGEEG7D5H5Q5Z5WUWKU277VN2JFRX6UGA, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.746872127 (6h15m33.527805505s) [noise=true , address=104.36.82.40:4954 , piece_id=YLOHTJLRAED66OMEQYPZHNNRI77V6WOR4EHTEZOOEHMGZHJ2O3QQ, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.747123162 (6h15m33.52805654s) [noise=true , address=104.36.82.46:4496 , piece_id=GWNQR5WRGRRQBLSGWJFD3O3UNCMNE62JQNPIV5SYKBUIL3BQKA5A, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.747280678 (6h15m33.528214053s) [noise=true , address=104.36.82.47:4524 , piece_id=DONJFTKUQ2XCW6BOU7AVOKJ7XMDMBPGQVUPRANH7X2YS7PJXUKIA, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.747736400 (6h15m33.528669779s) [noise=true , address=104.36.82.44:4893 , piece_id=HUJICRJ6W54O3AZZR6H3KG6E4TGBTNS46O6XVMQM3PFAZE7A2HZA, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.747755734 (6h15m33.528689109s) [noise=true , address=104.36.82.55:4204 , piece_id=72WISGRHMOAZDV5EDGTBOMLBL4HDOF3CBEMQAAIRGILPC7J5JKAQ, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.748313124 (6h15m33.5292465s) [noise=true , address=104.36.82.58:4287 , piece_id=BVDYUEWU7TVEQDMZXX5SJNS4QICVKMTGFLCUV6P2YHOE7TI25VXA, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.748485217 (6h15m33.529418594s) [noise=true , address=104.36.82.53:4123 , piece_id=NXJRZ5BQSLXSOIOSUJ4MM4AGKHP4UIDUGEJKLUUX7YOGQJA6WWZA, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.748831329 (6h15m33.529764704s) [noise=true , address=104.36.82.42:4829 , piece_id=ZLWJFP3LGX3YLXBJYIR66DXTYEVDBKXKVXCQSF2PCYCYJBIEJGMA, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.749299293 (6h15m33.530232669s) [noise=true , address=104.36.82.8:4002 , piece_id=6XSPWQC3US355PS6IQ4K3IYNQZH5MAT4IVIPGSYRUM3XYPSCWE6Q, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.749790974 (6h15m33.53072436s) [noise=true , address=104.36.82.44:4901 , piece_id=N4YLCLR2QZSULUL2XWONEKBFUMNOU7KUX2QEOK4SSDFSCI7IL6LA, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.750478221 (6h15m33.531411597s) [noise=true , address=104.36.82.54:4172 , piece_id=ZNA53MWDRWPUH6FUR4DJERBDT2IOYGA3IYUADKXXIOLQJLQOXE7Q, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.752884255 (6h15m33.533817625s) [noise=true , address=64.49.13.140:4508 , piece_id=MZ52U75VOIK6Q7QFXMMYRRG7JXFWDFU2JWQFYPERUCMH3BUX6XPQ, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.879839555 (6h15m33.660772917s) [upload=5911]: Next returned nil segment Jul 27 04:07:17.879861389 (6h15m33.660794765s) [upload=5911]: Waiting for error group managing segments... Jul 27 04:07:17.888652438 (6h15m33.669585811s) [noise=true , address=104.36.82.58:4287 , piece_id=BVDYUEWU7TVEQDMZXX5SJNS4QICVKMTGFLCUV6P2YHOE7TI25VXA, seg_pos={0 0}, upload=5911]: Done uploading piece. err: <nil> Jul 27 04:07:17.952192045 (6h15m33.733125429s) [noise=true , address=104.36.82.53:4123 , piece_id=NXJRZ5BQSLXSOIOSUJ4MM4AGKHP4UIDUGEJKLUUX7YOGQJA6WWZA, seg_pos={0 0}, upload=5911]: Done uploading piece. err: <nil> Jul 27 04:07:17.981907539 (6h15m33.762840911s) [noise=true , address=104.36.82.56:4244 , piece_id=LQCZ5W54NBHWAVBVYK7HYSSJDC73E6FV4CP4YIM4L5H2N3OLOZXA, seg_pos={0 0}, upload=5911]: Done uploading piece. err: <nil> Jul 27 04:07:17.982370496 (6h15m33.7633039s) [noise=true , address=104.36.82.55:4204 , piece_id=72WISGRHMOAZDV5EDGTBOMLBL4HDOF3CBEMQAAIRGILPC7J5JKAQ, seg_pos={0 0}, upload=5911]: Done uploading piece. err: <nil> Jul 27 04:07:18.009548506 (6h15m33.790481886s) [noise=true , address=104.36.82.52:4098 , piece_id=M2ETF25QQ53RRGFPUKXJRINIWV7AWXSECEDR2AIM645IED72NURQ, seg_pos={0 0}, upload=5911]: Done uploading piece. err: <nil> Jul 27 04:07:18.023770598 (6h15m33.804703983s) [noise=true , address=104.36.82.40:4954 , piece_id=YLOHTJLRAED66OMEQYPZHNNRI77V6WOR4EHTEZOOEHMGZHJ2O3QQ, seg_pos={0 0}, upload=5911]: Done uploading piece. err: <nil> Jul 27 04:07:18.139745581 (6h15m33.92067895s) [noise=true , address=104.36.82.22:4534 , piece_id=CW4AQT3XUGA4A5OC6NUY4NYOOAMLN5FFJJN5PQBYEON7HPD5LCYQ, seg_pos={0 0}, upload=5911]: Done uploading piece. err: <nil> Jul 27 04:07:18.423748140 (6h15m34.204681511s) [noise=true , address=64.49.13.140:4508 , piece_id=MZ52U75VOIK6Q7QFXMMYRRG7JXFWDFU2JWQFYPERUCMH3BUX6XPQ, seg_pos={0 0}, upload=5911]: Done uploading piece. err: <nil> Jul 27 04:08:16.739940381 (6h16m32.52087375s) [noise=true , address=104.36.82.54:4172 , piece_id=ZNA53MWDRWPUH6FUR4DJERBDT2IOYGA3IYUADKXXIOLQJLQOXE7Q, seg_pos={0 0}, upload=5911]: Done uploading piece. err: <nil> Jul 27 04:08:17.177295397 (6h16m32.958228766s) [noise=true , address=104.36.82.8:4002 , piece_id=6XSPWQC3US355PS6IQ4K3IYNQZH5MAT4IVIPGSYRUM3XYPSCWE6Q, seg_pos={0 0}, upload=5911]: Done uploading piece. err: <nil> Jul 27 04:08:17.534296584 (6h16m33.315229953s) [noise=true , address=104.36.82.36:4650 , piece_id=BKJULWGPESBINNTSUYP6E4KZRXKPACVU5I7DOSZBO4YMK2V26HEQ, seg_pos={0 0}, upload=5911]: Done uploading piece. err: <nil> Jul 27 04:08:17.677851566 (6h16m33.458784943s) [noise=true , address=104.36.82.14:4248 , piece_id=LDSGCUM7Q2VX5CDFI4JRGEEG7D5H5Q5Z5WUWKU277VN2JFRX6UGA, seg_pos={0 0}, upload=5911]: Done uploading piece. err: ecclient: upload failed (node:1u6ir2PLqDa9xGm64cuqq1RqY4qeBa1fqCo9VghspBf54XEXqn, address:104.36.82.14:4248): protocol: rpc: dial tcp 104.36.82.14:4248: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 04:08:17.678115170 (6h16m33.459048549s) [noise=true , address=104.36.82.46:4496 , piece_id=GWNQR5WRGRRQBLSGWJFD3O3UNCMNE62JQNPIV5SYKBUIL3BQKA5A, seg_pos={0 0}, upload=5911]: Done uploading piece. err: ecclient: upload failed (node:12V8XhSWkyGUVYFR8MykifFAP1YAfGCwzqucDcMMfzw3xsVW77Z, address:104.36.82.46:4496): protocol: read tcp 159.223.143.39:52800->104.36.82.46:4496: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 04:08:17.678144409 (6h16m33.459077789s) [noise=true , address=104.36.82.44:4901 , piece_id=N4YLCLR2QZSULUL2XWONEKBFUMNOU7KUX2QEOK4SSDFSCI7IL6LA, seg_pos={0 0}, upload=5911]: Done uploading piece. err: ecclient: upload failed (node:1FHE84zvBb1MKwB1Ht7puTafgQxDEvPt4mYqkYVSSufAMpWQA1, address:104.36.82.44:4901): protocol: rpc: dial tcp 104.36.82.44:4901: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 04:08:17.678229622 (6h16m33.459162988s) [noise=true , address=104.36.82.44:4893 , piece_id=HUJICRJ6W54O3AZZR6H3KG6E4TGBTNS46O6XVMQM3PFAZE7A2HZA, seg_pos={0 0}, upload=5911]: Done uploading piece. err: ecclient: upload failed (node:17VMFZ2T8jrSJVtchciJ9SnyrR6JstvcMoXrN5Q3zTD1aQU1oJ, address:104.36.82.44:4893): protocol: write tcp 159.223.143.39:35302->104.36.82.44:4893: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 04:08:17.678310218 (6h16m33.459243583s) [noise=true , address=104.36.82.42:4829 , piece_id=ZLWJFP3LGX3YLXBJYIR66DXTYEVDBKXKVXCQSF2PCYCYJBIEJGMA, seg_pos={0 0}, upload=5911]: Done uploading piece. err: ecclient: upload failed (node:1F5T8Vt1x4nXFaSenvGBCYjxhRL1H1sxRvzLjW3CqzwDVhq39F, address:104.36.82.42:4829): protocol: rpc: dial tcp 104.36.82.42:4829: i/o timeout; conn closed: op; piecestore: piecestore close: context deadline exceeded Jul 27 04:08:17.678412989 (6h16m33.459346365s) [noise=true , address=104.36.82.38:4722 , piece_id=B2MBQY5CTMEI6WSE7FTLOV5RUJ22SCNG7N2EEJILEATGJ4TIALXA, seg_pos={0 0}, upload=5911]: Done uploading piece. err: ecclient: upload failed (node:12Zw3dtUg9gARz52KhwDLTjcqC3CNm7dpPzpdx7P7PbL1jwe8uc, address:104.36.82.38:4722): protocol: read tcp 159.223.143.39:56048->104.36.82.38:4722: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 04:08:17.678455247 (6h16m33.459388624s) [noise=true , address=104.36.82.47:4524 , piece_id=DONJFTKUQ2XCW6BOU7AVOKJ7XMDMBPGQVUPRANH7X2YS7PJXUKIA, seg_pos={0 0}, upload=5911]: Done uploading piece. err: ecclient: upload failed (node:1xfbCzjSwFDRrR1gPggeN8tNCLbRRWDVzCJ6mqhZsbVSe8d4s4, address:104.36.82.47:4524): protocol: read tcp 159.223.143.39:46694->104.36.82.47:4524: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 04:08:17.678510668 (6h16m33.459444052s) [seg_pos={0 0}, upload=5911]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Jul 27 04:08:17.706442864 (6h16m33.487376232s) [upload=5911]: Done uploading segments... Jul 27 04:08:17.706495329 (6h16m33.487428705s) [upload=5911]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 633.746796ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................................X......................] duration: 445.226548ms FAIL (2025-07-27 04:21:39 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== ece_id=NYOFDEZ7CV4PHGQ6JZMEEVAYGJRPYG6IBW7MYJO42HMWOJM7PHZA, seg_pos={0 0}, upload=5879]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12crcXpNe9NgeXTsUpooLSBgrSUFbRxbSbanLa39w2Hr5mGFww): protocol: read tcp 159.223.143.39:53630->104.36.82.5:4840: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 04:05:32.508904485 (6h13m48.289837866s) [seg_pos={0 0}, upload=5879]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 27 04:05:32.509028070 (6h13m48.289961457s) [upload=5879]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 27 04:05:32.509034645 (6h13m48.289968129s) [upload=5879]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 27 04:05:32.581834839 (6h13m48.362768225s) [upload=5879]: Done uploading segments... Jul 27 04:05:32.581875919 (6h13m48.362809298s) [upload=5879]: Upload finished. err: <nil> Jul 27 04:06:23.829155331 (6h14m39.610088696s) [upload=5895]: Starting upload Jul 27 04:06:23.829225021 (6h14m39.610158398s) [upload=5895]: Done starting upload Jul 27 04:06:23.829294606 (6h14m39.61022799s) [upload=5895]: Uploading segments... Jul 27 04:06:23.829577656 (6h14m39.610511053s) [upload=5895]: Got next segment. Inline: false Jul 27 04:06:23.829649664 (6h14m39.610583027s) [upload=5895]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 27 04:06:23.829657065 (6h14m39.61059043s) [upload=5895]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 27 04:06:23.861405530 (6h14m39.642338913s) [seg_pos={0 0}, upload=5895]: Begin upload segment... Jul 27 04:06:23.861476351 (6h14m39.642409713s) [seg_pos={0 0}, upload=5895]: Done begin upload segment. Jul 27 04:06:23.861509120 (6h14m39.642442506s) [noise=true , address=77.90.31.19:4843 , piece_id=YML5RDQIK432Z4FSE4DKXIV5U47LQBEZRFMAVLANUSGFUQR23UWQ, seg_pos={0 0}, upload=5895]: Uploading piece... Jul 27 04:06:23.861953723 (6h14m39.6428871s) [noise=true , address=45.91.116.10:4081 , piece_id=XGDTSEZOUNRTEVWHTO3DHTZMH2VZZOPJPEKLCFAHTXESLLYABHSA, seg_pos={0 0}, upload=5895]: Uploading piece... Jul 27 04:06:23.862060983 (6h14m39.642994358s) [noise=true , address=45.91.116.9:4262 , piece_id=M5ETD45HH2XZD6HDLTJ3JO2BOOIROXFWKVPTN42PSQEWG4F7KAAQ, seg_pos={0 0}, upload=5895]: Uploading piece... Jul 27 04:06:23.862759464 (6h14m39.64369284s) [noise=true , address=104.36.82.4:4804 , piece_id=IU6FDNVB5AUPA7MJ6RSXUTLBETPHOFGNJ7YGWBFIZSQFSO6W6BBA, seg_pos={0 0}, upload=5895]: Uploading piece... Jul 27 04:06:23.862835086 (6h14m39.643768461s) [noise=true , address=104.36.82.6:4873 , piece_id=YP4VJORZZUL5EVCH3P7LD3LNVMHB7CWRTKHGIA7UROXE44N6MMZA, seg_pos={0 0}, upload=5895]: Uploading piece... Jul 27 04:06:23.863593880 (6h14m39.644527258s) [noise=true , address=77.90.31.15:4697 , piece_id=Q76TJDSZPSYZSJQNFVOI462U6DFILMH2RV3LQISODXSOCXGRKW6Q, seg_pos={0 0}, upload=5895]: Uploading piece... Jul 27 04:06:23.863656891 (6h14m39.644590266s) [noise=true , address=45.91.116.7:4011 , piece_id=XMH6MCCHZBBRQWHJFWLRGYHLQD4KUGSGL7MMZSKA5TZJCND3DUDA, seg_pos={0 0}, upload=5895]: Uploading piece... Jul 27 04:06:23.864408612 (6h14m39.645341987s) [noise=true , address=45.91.116.6:4973 , piece_id=J3VULPU24CIM64VEVE3UHO62HPN6TJXGZU6BCJIS5JQJKHJK2LIQ, seg_pos={0 0}, upload=5895]: Uploading piece... Jul 27 04:06:23.864521549 (6h14m39.64545491s) [noise=true , address=104.36.82.10:4079 , piece_id=2SH5D6JOMLFR7VSN7ZY5CLMR5RHIRFYQ7Z5I2JIYTJQM43G4O6OQ, seg_pos={0 0}, upload=5895]: Uploading piece... Jul 27 04:06:23.865518491 (6h14m39.646451872s) [noise=true , address=77.90.31.17:4771 , piece_id=U2LF22GKCMKRZN6EWCF26ITDZQHBFZXCGSQXT33MCX6P57IGI35Q, seg_pos={0 0}, upload=5895]: Uploading piece... Jul 27 04:06:23.865654950 (6h14m39.646588325s) [noise=true , address=45.91.116.8:4048 , piece_id=5REWMVVTBVLDQXQ5KAEHLY4VJL5SGF7ONQWDQQ3N2FDU3OAO6AZA, seg_pos={0 0}, upload=5895]: Uploading piece... Jul 27 04:06:23.866190892 (6h14m39.647124271s) [noise=true , address=104.36.82.12:4151 , piece_id=E2TEUTQBD4Y4AUHSCPMCIFEXBSY7S5HJWBKNKWWTUWEW2BLAAE6Q, seg_pos={0 0}, upload=5895]: Uploading piece... Jul 27 04:06:23.866412590 (6h14m39.647345968s) [noise=true , address=104.36.82.11:4115 , piece_id=YXMNVT6ZHGESGQ2BXKZUVR7U2CNNAI2PO7ZTMDG5UIOV7HTDDUTA, seg_pos={0 0}, upload=5895]: Uploading piece... Jul 27 04:06:23.866715972 (6h14m39.647649359s) [noise=true , address=45.91.116.13:4153 , piece_id=H4535KOCG3F4RKA6AKKMTJSOM6Q6PGRC6M7FR3UIS3BORTSP5UQA, seg_pos={0 0}, upload=5895]: Uploading piece... Jul 27 04:06:23.866998965 (6h14m39.64793234s) [noise=true , address=104.36.82.13:4186 , piece_id=TBLDKALGJYBGR5NLCOCKJ4QKEFNIJXCKAFUBD7VR3UMNWEMKMKHQ, seg_pos={0 0}, upload=5895]: Uploading piece... Jul 27 04:06:23.867230593 (6h14m39.648163974s) [noise=true , address=104.36.82.16:4293 , piece_id=AHH6PRFUSGTDGB7H52DRITARA4IF6GFLVSRYS2Y6EPJCZSKUOYWQ, seg_pos={0 0}, upload=5895]: Uploading piece... Jul 27 04:06:23.949422808 (6h14m39.730356187s) [upload=5895]: Next returned nil segment Jul 27 04:06:23.949442576 (6h14m39.73037596s) [upload=5895]: Waiting for error group managing segments... Jul 27 04:06:23.953497426 (6h14m39.734430802s) [noise=true , address=45.91.116.10:4081 , piece_id=XGDTSEZOUNRTEVWHTO3DHTZMH2VZZOPJPEKLCFAHTXESLLYABHSA, seg_pos={0 0}, upload=5895]: Done uploading piece. err: <nil> Jul 27 04:06:23.953531796 (6h14m39.734465159s) [noise=true , address=45.91.116.6:4973 , piece_id=J3VULPU24CIM64VEVE3UHO62HPN6TJXGZU6BCJIS5JQJKHJK2LIQ, seg_pos={0 0}, upload=5895]: Done uploading piece. err: <nil> Jul 27 04:06:23.954068660 (6h14m39.735002029s) [noise=true , address=45.91.116.9:4262 , piece_id=M5ETD45HH2XZD6HDLTJ3JO2BOOIROXFWKVPTN42PSQEWG4F7KAAQ, seg_pos={0 0}, upload=5895]: Done uploading piece. err: <nil> Jul 27 04:06:23.954194339 (6h14m39.735127707s) [noise=true , address=45.91.116.8:4048 , piece_id=5REWMVVTBVLDQXQ5KAEHLY4VJL5SGF7ONQWDQQ3N2FDU3OAO6AZA, seg_pos={0 0}, upload=5895]: Done uploading piece. err: <nil> Jul 27 04:06:23.954601730 (6h14m39.735535112s) [noise=true , address=45.91.116.7:4011 , piece_id=XMH6MCCHZBBRQWHJFWLRGYHLQD4KUGSGL7MMZSKA5TZJCND3DUDA, seg_pos={0 0}, upload=5895]: Done uploading piece. err: <nil> Jul 27 04:06:23.954755872 (6h14m39.735689237s) [noise=true , address=45.91.116.13:4153 , piece_id=H4535KOCG3F4RKA6AKKMTJSOM6Q6PGRC6M7FR3UIS3BORTSP5UQA, seg_pos={0 0}, upload=5895]: Done uploading piece. err: <nil> Jul 27 04:06:23.955611883 (6h14m39.73654526s) [noise=true , address=77.90.31.19:4843 , piece_id=YML5RDQIK432Z4FSE4DKXIV5U47LQBEZRFMAVLANUSGFUQR23UWQ, seg_pos={0 0}, upload=5895]: Done uploading piece. err: <nil> Jul 27 04:06:23.956373391 (6h14m39.737306756s) [noise=true , address=77.90.31.15:4697 , piece_id=Q76TJDSZPSYZSJQNFVOI462U6DFILMH2RV3LQISODXSOCXGRKW6Q, seg_pos={0 0}, upload=5895]: Done uploading piece. err: <nil> Jul 27 04:06:23.958659122 (6h14m39.73959249s) [noise=true , address=77.90.31.17:4771 , piece_id=U2LF22GKCMKRZN6EWCF26ITDZQHBFZXCGSQXT33MCX6P57IGI35Q, seg_pos={0 0}, upload=5895]: Done uploading piece. err: <nil> Jul 27 04:06:35.058944013 (6h14m50.83987739s) [noise=true , address=104.36.82.16:4293 , piece_id=AHH6PRFUSGTDGB7H52DRITARA4IF6GFLVSRYS2Y6EPJCZSKUOYWQ, seg_pos={0 0}, upload=5895]: Done uploading piece. err: <nil> Jul 27 04:06:41.103204002 (6h14m56.884137386s) [noise=true , address=104.36.82.4:4804 , piece_id=IU6FDNVB5AUPA7MJ6RSXUTLBETPHOFGNJ7YGWBFIZSQFSO6W6BBA, seg_pos={0 0}, upload=5895]: Done uploading piece. err: <nil> Jul 27 04:06:43.191702285 (6h14m58.972635651s) [noise=true , address=104.36.82.6:4873 , piece_id=YP4VJORZZUL5EVCH3P7LD3LNVMHB7CWRTKHGIA7UROXE44N6MMZA, seg_pos={0 0}, upload=5895]: Done uploading piece. err: <nil> Jul 27 04:06:53.777973493 (6h15m9.558906861s) [noise=true , address=104.36.82.11:4115 , piece_id=YXMNVT6ZHGESGQ2BXKZUVR7U2CNNAI2PO7ZTMDG5UIOV7HTDDUTA, seg_pos={0 0}, upload=5895]: Done uploading piece. err: ecclient: upload failed (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR, address:104.36.82.11:4115): protocol: write tcp 159.223.143.39:34210->104.36.82.11:4115: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 04:06:53.778019532 (6h15m9.558952909s) [noise=true , address=104.36.82.12:4151 , piece_id=E2TEUTQBD4Y4AUHSCPMCIFEXBSY7S5HJWBKNKWWTUWEW2BLAAE6Q, seg_pos={0 0}, upload=5895]: Done uploading piece. err: ecclient: upload failed (node:1w6pwZMdeQKjt92qx6WbTWU5VPYJto5gS87f6iKvCBZH6LfyeB, address:104.36.82.12:4151): protocol: rpc: dial tcp 104.36.82.12:4151: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 04:06:53.778043653 (6h15m9.558977021s) [noise=true , address=104.36.82.13:4186 , piece_id=TBLDKALGJYBGR5NLCOCKJ4QKEFNIJXCKAFUBD7VR3UMNWEMKMKHQ, seg_pos={0 0}, upload=5895]: Done uploading piece. err: ecclient: upload failed (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht, address:104.36.82.13:4186): protocol: read tcp 159.223.143.39:34340->104.36.82.13:4186: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 04:06:53.778095228 (6h15m9.559028593s) [noise=true , address=104.36.82.10:4079 , piece_id=2SH5D6JOMLFR7VSN7ZY5CLMR5RHIRFYQ7Z5I2JIYTJQM43G4O6OQ, seg_pos={0 0}, upload=5895]: Done uploading piece. err: ecclient: upload failed (node:12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud, address:104.36.82.10:4079): protocol: write tcp 159.223.143.39:53840->104.36.82.10:4079: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 04:06:53.778142167 (6h15m9.559075532s) [seg_pos={0 0}, upload=5895]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 27 04:06:53.837032161 (6h15m9.617965543s) [upload=5895]: Done uploading segments... Jul 27 04:06:53.837055996 (6h15m9.617989366s) [upload=5895]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 445.258209ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................................................X.......] duration: 473.679497ms FAIL (2025-07-27 04:21:39 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 12) Duration: 473.716793ms core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.........................................X..................] duration: 579.871617ms FAIL (2025-07-27 04:21:30 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== {0 0}, upload=5911]: Begin upload segment... Jul 27 04:07:17.745307658 (6h15m33.526241043s) [seg_pos={0 0}, upload=5911]: Done begin upload segment. Jul 27 04:07:17.745349241 (6h15m33.526282625s) [noise=true , address=104.36.82.52:4098 , piece_id=M2ETF25QQ53RRGFPUKXJRINIWV7AWXSECEDR2AIM645IED72NURQ, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.745439694 (6h15m33.526373073s) [noise=true , address=104.36.82.56:4244 , piece_id=LQCZ5W54NBHWAVBVYK7HYSSJDC73E6FV4CP4YIM4L5H2N3OLOZXA, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.745972271 (6h15m33.526905654s) [noise=true , address=104.36.82.36:4650 , piece_id=BKJULWGPESBINNTSUYP6E4KZRXKPACVU5I7DOSZBO4YMK2V26HEQ, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.745980243 (6h15m33.526913621s) [noise=true , address=104.36.82.38:4722 , piece_id=B2MBQY5CTMEI6WSE7FTLOV5RUJ22SCNG7N2EEJILEATGJ4TIALXA, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.746403818 (6h15m33.527337198s) [noise=true , address=104.36.82.22:4534 , piece_id=CW4AQT3XUGA4A5OC6NUY4NYOOAMLN5FFJJN5PQBYEON7HPD5LCYQ, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.746455493 (6h15m33.527388875s) [noise=true , address=104.36.82.14:4248 , piece_id=LDSGCUM7Q2VX5CDFI4JRGEEG7D5H5Q5Z5WUWKU277VN2JFRX6UGA, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.746872127 (6h15m33.527805505s) [noise=true , address=104.36.82.40:4954 , piece_id=YLOHTJLRAED66OMEQYPZHNNRI77V6WOR4EHTEZOOEHMGZHJ2O3QQ, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.747123162 (6h15m33.52805654s) [noise=true , address=104.36.82.46:4496 , piece_id=GWNQR5WRGRRQBLSGWJFD3O3UNCMNE62JQNPIV5SYKBUIL3BQKA5A, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.747280678 (6h15m33.528214053s) [noise=true , address=104.36.82.47:4524 , piece_id=DONJFTKUQ2XCW6BOU7AVOKJ7XMDMBPGQVUPRANH7X2YS7PJXUKIA, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.747736400 (6h15m33.528669779s) [noise=true , address=104.36.82.44:4893 , piece_id=HUJICRJ6W54O3AZZR6H3KG6E4TGBTNS46O6XVMQM3PFAZE7A2HZA, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.747755734 (6h15m33.528689109s) [noise=true , address=104.36.82.55:4204 , piece_id=72WISGRHMOAZDV5EDGTBOMLBL4HDOF3CBEMQAAIRGILPC7J5JKAQ, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.748313124 (6h15m33.5292465s) [noise=true , address=104.36.82.58:4287 , piece_id=BVDYUEWU7TVEQDMZXX5SJNS4QICVKMTGFLCUV6P2YHOE7TI25VXA, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.748485217 (6h15m33.529418594s) [noise=true , address=104.36.82.53:4123 , piece_id=NXJRZ5BQSLXSOIOSUJ4MM4AGKHP4UIDUGEJKLUUX7YOGQJA6WWZA, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.748831329 (6h15m33.529764704s) [noise=true , address=104.36.82.42:4829 , piece_id=ZLWJFP3LGX3YLXBJYIR66DXTYEVDBKXKVXCQSF2PCYCYJBIEJGMA, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.749299293 (6h15m33.530232669s) [noise=true , address=104.36.82.8:4002 , piece_id=6XSPWQC3US355PS6IQ4K3IYNQZH5MAT4IVIPGSYRUM3XYPSCWE6Q, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.749790974 (6h15m33.53072436s) [noise=true , address=104.36.82.44:4901 , piece_id=N4YLCLR2QZSULUL2XWONEKBFUMNOU7KUX2QEOK4SSDFSCI7IL6LA, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.750478221 (6h15m33.531411597s) [noise=true , address=104.36.82.54:4172 , piece_id=ZNA53MWDRWPUH6FUR4DJERBDT2IOYGA3IYUADKXXIOLQJLQOXE7Q, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.752884255 (6h15m33.533817625s) [noise=true , address=64.49.13.140:4508 , piece_id=MZ52U75VOIK6Q7QFXMMYRRG7JXFWDFU2JWQFYPERUCMH3BUX6XPQ, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.879839555 (6h15m33.660772917s) [upload=5911]: Next returned nil segment Jul 27 04:07:17.879861389 (6h15m33.660794765s) [upload=5911]: Waiting for error group managing segments... Jul 27 04:07:17.888652438 (6h15m33.669585811s) [noise=true , address=104.36.82.58:4287 , piece_id=BVDYUEWU7TVEQDMZXX5SJNS4QICVKMTGFLCUV6P2YHOE7TI25VXA, seg_pos={0 0}, upload=5911]: Done uploading piece. err: <nil> Jul 27 04:07:17.952192045 (6h15m33.733125429s) [noise=true , address=104.36.82.53:4123 , piece_id=NXJRZ5BQSLXSOIOSUJ4MM4AGKHP4UIDUGEJKLUUX7YOGQJA6WWZA, seg_pos={0 0}, upload=5911]: Done uploading piece. err: <nil> Jul 27 04:07:17.981907539 (6h15m33.762840911s) [noise=true , address=104.36.82.56:4244 , piece_id=LQCZ5W54NBHWAVBVYK7HYSSJDC73E6FV4CP4YIM4L5H2N3OLOZXA, seg_pos={0 0}, upload=5911]: Done uploading piece. err: <nil> Jul 27 04:07:17.982370496 (6h15m33.7633039s) [noise=true , address=104.36.82.55:4204 , piece_id=72WISGRHMOAZDV5EDGTBOMLBL4HDOF3CBEMQAAIRGILPC7J5JKAQ, seg_pos={0 0}, upload=5911]: Done uploading piece. err: <nil> Jul 27 04:07:18.009548506 (6h15m33.790481886s) [noise=true , address=104.36.82.52:4098 , piece_id=M2ETF25QQ53RRGFPUKXJRINIWV7AWXSECEDR2AIM645IED72NURQ, seg_pos={0 0}, upload=5911]: Done uploading piece. err: <nil> Jul 27 04:07:18.023770598 (6h15m33.804703983s) [noise=true , address=104.36.82.40:4954 , piece_id=YLOHTJLRAED66OMEQYPZHNNRI77V6WOR4EHTEZOOEHMGZHJ2O3QQ, seg_pos={0 0}, upload=5911]: Done uploading piece. err: <nil> Jul 27 04:07:18.139745581 (6h15m33.92067895s) [noise=true , address=104.36.82.22:4534 , piece_id=CW4AQT3XUGA4A5OC6NUY4NYOOAMLN5FFJJN5PQBYEON7HPD5LCYQ, seg_pos={0 0}, upload=5911]: Done uploading piece. err: <nil> Jul 27 04:07:18.423748140 (6h15m34.204681511s) [noise=true , address=64.49.13.140:4508 , piece_id=MZ52U75VOIK6Q7QFXMMYRRG7JXFWDFU2JWQFYPERUCMH3BUX6XPQ, seg_pos={0 0}, upload=5911]: Done uploading piece. err: <nil> Jul 27 04:08:16.739940381 (6h16m32.52087375s) [noise=true , address=104.36.82.54:4172 , piece_id=ZNA53MWDRWPUH6FUR4DJERBDT2IOYGA3IYUADKXXIOLQJLQOXE7Q, seg_pos={0 0}, upload=5911]: Done uploading piece. err: <nil> Jul 27 04:08:17.177295397 (6h16m32.958228766s) [noise=true , address=104.36.82.8:4002 , piece_id=6XSPWQC3US355PS6IQ4K3IYNQZH5MAT4IVIPGSYRUM3XYPSCWE6Q, seg_pos={0 0}, upload=5911]: Done uploading piece. err: <nil> Jul 27 04:08:17.534296584 (6h16m33.315229953s) [noise=true , address=104.36.82.36:4650 , piece_id=BKJULWGPESBINNTSUYP6E4KZRXKPACVU5I7DOSZBO4YMK2V26HEQ, seg_pos={0 0}, upload=5911]: Done uploading piece. err: <nil> Jul 27 04:08:17.677851566 (6h16m33.458784943s) [noise=true , address=104.36.82.14:4248 , piece_id=LDSGCUM7Q2VX5CDFI4JRGEEG7D5H5Q5Z5WUWKU277VN2JFRX6UGA, seg_pos={0 0}, upload=5911]: Done uploading piece. err: ecclient: upload failed (node:1u6ir2PLqDa9xGm64cuqq1RqY4qeBa1fqCo9VghspBf54XEXqn, address:104.36.82.14:4248): protocol: rpc: dial tcp 104.36.82.14:4248: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 04:08:17.678115170 (6h16m33.459048549s) [noise=true , address=104.36.82.46:4496 , piece_id=GWNQR5WRGRRQBLSGWJFD3O3UNCMNE62JQNPIV5SYKBUIL3BQKA5A, seg_pos={0 0}, upload=5911]: Done uploading piece. err: ecclient: upload failed (node:12V8XhSWkyGUVYFR8MykifFAP1YAfGCwzqucDcMMfzw3xsVW77Z, address:104.36.82.46:4496): protocol: read tcp 159.223.143.39:52800->104.36.82.46:4496: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 04:08:17.678144409 (6h16m33.459077789s) [noise=true , address=104.36.82.44:4901 , piece_id=N4YLCLR2QZSULUL2XWONEKBFUMNOU7KUX2QEOK4SSDFSCI7IL6LA, seg_pos={0 0}, upload=5911]: Done uploading piece. err: ecclient: upload failed (node:1FHE84zvBb1MKwB1Ht7puTafgQxDEvPt4mYqkYVSSufAMpWQA1, address:104.36.82.44:4901): protocol: rpc: dial tcp 104.36.82.44:4901: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 04:08:17.678229622 (6h16m33.459162988s) [noise=true , address=104.36.82.44:4893 , piece_id=HUJICRJ6W54O3AZZR6H3KG6E4TGBTNS46O6XVMQM3PFAZE7A2HZA, seg_pos={0 0}, upload=5911]: Done uploading piece. err: ecclient: upload failed (node:17VMFZ2T8jrSJVtchciJ9SnyrR6JstvcMoXrN5Q3zTD1aQU1oJ, address:104.36.82.44:4893): protocol: write tcp 159.223.143.39:35302->104.36.82.44:4893: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 04:08:17.678310218 (6h16m33.459243583s) [noise=true , address=104.36.82.42:4829 , piece_id=ZLWJFP3LGX3YLXBJYIR66DXTYEVDBKXKVXCQSF2PCYCYJBIEJGMA, seg_pos={0 0}, upload=5911]: Done uploading piece. err: ecclient: upload failed (node:1F5T8Vt1x4nXFaSenvGBCYjxhRL1H1sxRvzLjW3CqzwDVhq39F, address:104.36.82.42:4829): protocol: rpc: dial tcp 104.36.82.42:4829: i/o timeout; conn closed: op; piecestore: piecestore close: context deadline exceeded Jul 27 04:08:17.678412989 (6h16m33.459346365s) [noise=true , address=104.36.82.38:4722 , piece_id=B2MBQY5CTMEI6WSE7FTLOV5RUJ22SCNG7N2EEJILEATGJ4TIALXA, seg_pos={0 0}, upload=5911]: Done uploading piece. err: ecclient: upload failed (node:12Zw3dtUg9gARz52KhwDLTjcqC3CNm7dpPzpdx7P7PbL1jwe8uc, address:104.36.82.38:4722): protocol: read tcp 159.223.143.39:56048->104.36.82.38:4722: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 04:08:17.678455247 (6h16m33.459388624s) [noise=true , address=104.36.82.47:4524 , piece_id=DONJFTKUQ2XCW6BOU7AVOKJ7XMDMBPGQVUPRANH7X2YS7PJXUKIA, seg_pos={0 0}, upload=5911]: Done uploading piece. err: ecclient: upload failed (node:1xfbCzjSwFDRrR1gPggeN8tNCLbRRWDVzCJ6mqhZsbVSe8d4s4, address:104.36.82.47:4524): protocol: read tcp 159.223.143.39:46694->104.36.82.47:4524: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 04:08:17.678510668 (6h16m33.459444052s) [seg_pos={0 0}, upload=5911]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Jul 27 04:08:17.706442864 (6h16m33.487376232s) [upload=5911]: Done uploading segments... Jul 27 04:08:17.706495329 (6h16m33.487428705s) [upload=5911]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 579.929421ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [..........................................X.................] duration: 45.454709262s FAIL (2025-07-27 04:21:29 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== {0 0}, upload=5911]: Begin upload segment... Jul 27 04:07:17.745307658 (6h15m33.526241043s) [seg_pos={0 0}, upload=5911]: Done begin upload segment. Jul 27 04:07:17.745349241 (6h15m33.526282625s) [noise=true , address=104.36.82.52:4098 , piece_id=M2ETF25QQ53RRGFPUKXJRINIWV7AWXSECEDR2AIM645IED72NURQ, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.745439694 (6h15m33.526373073s) [noise=true , address=104.36.82.56:4244 , piece_id=LQCZ5W54NBHWAVBVYK7HYSSJDC73E6FV4CP4YIM4L5H2N3OLOZXA, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.745972271 (6h15m33.526905654s) [noise=true , address=104.36.82.36:4650 , piece_id=BKJULWGPESBINNTSUYP6E4KZRXKPACVU5I7DOSZBO4YMK2V26HEQ, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.745980243 (6h15m33.526913621s) [noise=true , address=104.36.82.38:4722 , piece_id=B2MBQY5CTMEI6WSE7FTLOV5RUJ22SCNG7N2EEJILEATGJ4TIALXA, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.746403818 (6h15m33.527337198s) [noise=true , address=104.36.82.22:4534 , piece_id=CW4AQT3XUGA4A5OC6NUY4NYOOAMLN5FFJJN5PQBYEON7HPD5LCYQ, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.746455493 (6h15m33.527388875s) [noise=true , address=104.36.82.14:4248 , piece_id=LDSGCUM7Q2VX5CDFI4JRGEEG7D5H5Q5Z5WUWKU277VN2JFRX6UGA, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.746872127 (6h15m33.527805505s) [noise=true , address=104.36.82.40:4954 , piece_id=YLOHTJLRAED66OMEQYPZHNNRI77V6WOR4EHTEZOOEHMGZHJ2O3QQ, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.747123162 (6h15m33.52805654s) [noise=true , address=104.36.82.46:4496 , piece_id=GWNQR5WRGRRQBLSGWJFD3O3UNCMNE62JQNPIV5SYKBUIL3BQKA5A, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.747280678 (6h15m33.528214053s) [noise=true , address=104.36.82.47:4524 , piece_id=DONJFTKUQ2XCW6BOU7AVOKJ7XMDMBPGQVUPRANH7X2YS7PJXUKIA, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.747736400 (6h15m33.528669779s) [noise=true , address=104.36.82.44:4893 , piece_id=HUJICRJ6W54O3AZZR6H3KG6E4TGBTNS46O6XVMQM3PFAZE7A2HZA, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.747755734 (6h15m33.528689109s) [noise=true , address=104.36.82.55:4204 , piece_id=72WISGRHMOAZDV5EDGTBOMLBL4HDOF3CBEMQAAIRGILPC7J5JKAQ, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.748313124 (6h15m33.5292465s) [noise=true , address=104.36.82.58:4287 , piece_id=BVDYUEWU7TVEQDMZXX5SJNS4QICVKMTGFLCUV6P2YHOE7TI25VXA, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.748485217 (6h15m33.529418594s) [noise=true , address=104.36.82.53:4123 , piece_id=NXJRZ5BQSLXSOIOSUJ4MM4AGKHP4UIDUGEJKLUUX7YOGQJA6WWZA, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.748831329 (6h15m33.529764704s) [noise=true , address=104.36.82.42:4829 , piece_id=ZLWJFP3LGX3YLXBJYIR66DXTYEVDBKXKVXCQSF2PCYCYJBIEJGMA, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.749299293 (6h15m33.530232669s) [noise=true , address=104.36.82.8:4002 , piece_id=6XSPWQC3US355PS6IQ4K3IYNQZH5MAT4IVIPGSYRUM3XYPSCWE6Q, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.749790974 (6h15m33.53072436s) [noise=true , address=104.36.82.44:4901 , piece_id=N4YLCLR2QZSULUL2XWONEKBFUMNOU7KUX2QEOK4SSDFSCI7IL6LA, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.750478221 (6h15m33.531411597s) [noise=true , address=104.36.82.54:4172 , piece_id=ZNA53MWDRWPUH6FUR4DJERBDT2IOYGA3IYUADKXXIOLQJLQOXE7Q, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.752884255 (6h15m33.533817625s) [noise=true , address=64.49.13.140:4508 , piece_id=MZ52U75VOIK6Q7QFXMMYRRG7JXFWDFU2JWQFYPERUCMH3BUX6XPQ, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.879839555 (6h15m33.660772917s) [upload=5911]: Next returned nil segment Jul 27 04:07:17.879861389 (6h15m33.660794765s) [upload=5911]: Waiting for error group managing segments... Jul 27 04:07:17.888652438 (6h15m33.669585811s) [noise=true , address=104.36.82.58:4287 , piece_id=BVDYUEWU7TVEQDMZXX5SJNS4QICVKMTGFLCUV6P2YHOE7TI25VXA, seg_pos={0 0}, upload=5911]: Done uploading piece. err: <nil> Jul 27 04:07:17.952192045 (6h15m33.733125429s) [noise=true , address=104.36.82.53:4123 , piece_id=NXJRZ5BQSLXSOIOSUJ4MM4AGKHP4UIDUGEJKLUUX7YOGQJA6WWZA, seg_pos={0 0}, upload=5911]: Done uploading piece. err: <nil> Jul 27 04:07:17.981907539 (6h15m33.762840911s) [noise=true , address=104.36.82.56:4244 , piece_id=LQCZ5W54NBHWAVBVYK7HYSSJDC73E6FV4CP4YIM4L5H2N3OLOZXA, seg_pos={0 0}, upload=5911]: Done uploading piece. err: <nil> Jul 27 04:07:17.982370496 (6h15m33.7633039s) [noise=true , address=104.36.82.55:4204 , piece_id=72WISGRHMOAZDV5EDGTBOMLBL4HDOF3CBEMQAAIRGILPC7J5JKAQ, seg_pos={0 0}, upload=5911]: Done uploading piece. err: <nil> Jul 27 04:07:18.009548506 (6h15m33.790481886s) [noise=true , address=104.36.82.52:4098 , piece_id=M2ETF25QQ53RRGFPUKXJRINIWV7AWXSECEDR2AIM645IED72NURQ, seg_pos={0 0}, upload=5911]: Done uploading piece. err: <nil> Jul 27 04:07:18.023770598 (6h15m33.804703983s) [noise=true , address=104.36.82.40:4954 , piece_id=YLOHTJLRAED66OMEQYPZHNNRI77V6WOR4EHTEZOOEHMGZHJ2O3QQ, seg_pos={0 0}, upload=5911]: Done uploading piece. err: <nil> Jul 27 04:07:18.139745581 (6h15m33.92067895s) [noise=true , address=104.36.82.22:4534 , piece_id=CW4AQT3XUGA4A5OC6NUY4NYOOAMLN5FFJJN5PQBYEON7HPD5LCYQ, seg_pos={0 0}, upload=5911]: Done uploading piece. err: <nil> Jul 27 04:07:18.423748140 (6h15m34.204681511s) [noise=true , address=64.49.13.140:4508 , piece_id=MZ52U75VOIK6Q7QFXMMYRRG7JXFWDFU2JWQFYPERUCMH3BUX6XPQ, seg_pos={0 0}, upload=5911]: Done uploading piece. err: <nil> Jul 27 04:08:16.739940381 (6h16m32.52087375s) [noise=true , address=104.36.82.54:4172 , piece_id=ZNA53MWDRWPUH6FUR4DJERBDT2IOYGA3IYUADKXXIOLQJLQOXE7Q, seg_pos={0 0}, upload=5911]: Done uploading piece. err: <nil> Jul 27 04:08:17.177295397 (6h16m32.958228766s) [noise=true , address=104.36.82.8:4002 , piece_id=6XSPWQC3US355PS6IQ4K3IYNQZH5MAT4IVIPGSYRUM3XYPSCWE6Q, seg_pos={0 0}, upload=5911]: Done uploading piece. err: <nil> Jul 27 04:08:17.534296584 (6h16m33.315229953s) [noise=true , address=104.36.82.36:4650 , piece_id=BKJULWGPESBINNTSUYP6E4KZRXKPACVU5I7DOSZBO4YMK2V26HEQ, seg_pos={0 0}, upload=5911]: Done uploading piece. err: <nil> Jul 27 04:08:17.677851566 (6h16m33.458784943s) [noise=true , address=104.36.82.14:4248 , piece_id=LDSGCUM7Q2VX5CDFI4JRGEEG7D5H5Q5Z5WUWKU277VN2JFRX6UGA, seg_pos={0 0}, upload=5911]: Done uploading piece. err: ecclient: upload failed (node:1u6ir2PLqDa9xGm64cuqq1RqY4qeBa1fqCo9VghspBf54XEXqn, address:104.36.82.14:4248): protocol: rpc: dial tcp 104.36.82.14:4248: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 04:08:17.678115170 (6h16m33.459048549s) [noise=true , address=104.36.82.46:4496 , piece_id=GWNQR5WRGRRQBLSGWJFD3O3UNCMNE62JQNPIV5SYKBUIL3BQKA5A, seg_pos={0 0}, upload=5911]: Done uploading piece. err: ecclient: upload failed (node:12V8XhSWkyGUVYFR8MykifFAP1YAfGCwzqucDcMMfzw3xsVW77Z, address:104.36.82.46:4496): protocol: read tcp 159.223.143.39:52800->104.36.82.46:4496: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 04:08:17.678144409 (6h16m33.459077789s) [noise=true , address=104.36.82.44:4901 , piece_id=N4YLCLR2QZSULUL2XWONEKBFUMNOU7KUX2QEOK4SSDFSCI7IL6LA, seg_pos={0 0}, upload=5911]: Done uploading piece. err: ecclient: upload failed (node:1FHE84zvBb1MKwB1Ht7puTafgQxDEvPt4mYqkYVSSufAMpWQA1, address:104.36.82.44:4901): protocol: rpc: dial tcp 104.36.82.44:4901: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 04:08:17.678229622 (6h16m33.459162988s) [noise=true , address=104.36.82.44:4893 , piece_id=HUJICRJ6W54O3AZZR6H3KG6E4TGBTNS46O6XVMQM3PFAZE7A2HZA, seg_pos={0 0}, upload=5911]: Done uploading piece. err: ecclient: upload failed (node:17VMFZ2T8jrSJVtchciJ9SnyrR6JstvcMoXrN5Q3zTD1aQU1oJ, address:104.36.82.44:4893): protocol: write tcp 159.223.143.39:35302->104.36.82.44:4893: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 04:08:17.678310218 (6h16m33.459243583s) [noise=true , address=104.36.82.42:4829 , piece_id=ZLWJFP3LGX3YLXBJYIR66DXTYEVDBKXKVXCQSF2PCYCYJBIEJGMA, seg_pos={0 0}, upload=5911]: Done uploading piece. err: ecclient: upload failed (node:1F5T8Vt1x4nXFaSenvGBCYjxhRL1H1sxRvzLjW3CqzwDVhq39F, address:104.36.82.42:4829): protocol: rpc: dial tcp 104.36.82.42:4829: i/o timeout; conn closed: op; piecestore: piecestore close: context deadline exceeded Jul 27 04:08:17.678412989 (6h16m33.459346365s) [noise=true , address=104.36.82.38:4722 , piece_id=B2MBQY5CTMEI6WSE7FTLOV5RUJ22SCNG7N2EEJILEATGJ4TIALXA, seg_pos={0 0}, upload=5911]: Done uploading piece. err: ecclient: upload failed (node:12Zw3dtUg9gARz52KhwDLTjcqC3CNm7dpPzpdx7P7PbL1jwe8uc, address:104.36.82.38:4722): protocol: read tcp 159.223.143.39:56048->104.36.82.38:4722: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 04:08:17.678455247 (6h16m33.459388624s) [noise=true , address=104.36.82.47:4524 , piece_id=DONJFTKUQ2XCW6BOU7AVOKJ7XMDMBPGQVUPRANH7X2YS7PJXUKIA, seg_pos={0 0}, upload=5911]: Done uploading piece. err: ecclient: upload failed (node:1xfbCzjSwFDRrR1gPggeN8tNCLbRRWDVzCJ6mqhZsbVSe8d4s4, address:104.36.82.47:4524): protocol: read tcp 159.223.143.39:46694->104.36.82.47:4524: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 04:08:17.678510668 (6h16m33.459444052s) [seg_pos={0 0}, upload=5911]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Jul 27 04:08:17.706442864 (6h16m33.487376232s) [upload=5911]: Done uploading segments... Jul 27 04:08:17.706495329 (6h16m33.487428705s) [upload=5911]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 45.454740928s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................................X.....................] duration: 420.379802ms FAIL (2025-07-27 04:21:17 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== ece_id=NYOFDEZ7CV4PHGQ6JZMEEVAYGJRPYG6IBW7MYJO42HMWOJM7PHZA, seg_pos={0 0}, upload=5879]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12crcXpNe9NgeXTsUpooLSBgrSUFbRxbSbanLa39w2Hr5mGFww): protocol: read tcp 159.223.143.39:53630->104.36.82.5:4840: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 04:05:32.508904485 (6h13m48.289837866s) [seg_pos={0 0}, upload=5879]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 27 04:05:32.509028070 (6h13m48.289961457s) [upload=5879]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 27 04:05:32.509034645 (6h13m48.289968129s) [upload=5879]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 27 04:05:32.581834839 (6h13m48.362768225s) [upload=5879]: Done uploading segments... Jul 27 04:05:32.581875919 (6h13m48.362809298s) [upload=5879]: Upload finished. err: <nil> Jul 27 04:06:23.829155331 (6h14m39.610088696s) [upload=5895]: Starting upload Jul 27 04:06:23.829225021 (6h14m39.610158398s) [upload=5895]: Done starting upload Jul 27 04:06:23.829294606 (6h14m39.61022799s) [upload=5895]: Uploading segments... Jul 27 04:06:23.829577656 (6h14m39.610511053s) [upload=5895]: Got next segment. Inline: false Jul 27 04:06:23.829649664 (6h14m39.610583027s) [upload=5895]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 27 04:06:23.829657065 (6h14m39.61059043s) [upload=5895]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 27 04:06:23.861405530 (6h14m39.642338913s) [seg_pos={0 0}, upload=5895]: Begin upload segment... Jul 27 04:06:23.861476351 (6h14m39.642409713s) [seg_pos={0 0}, upload=5895]: Done begin upload segment. Jul 27 04:06:23.861509120 (6h14m39.642442506s) [noise=true , address=77.90.31.19:4843 , piece_id=YML5RDQIK432Z4FSE4DKXIV5U47LQBEZRFMAVLANUSGFUQR23UWQ, seg_pos={0 0}, upload=5895]: Uploading piece... Jul 27 04:06:23.861953723 (6h14m39.6428871s) [noise=true , address=45.91.116.10:4081 , piece_id=XGDTSEZOUNRTEVWHTO3DHTZMH2VZZOPJPEKLCFAHTXESLLYABHSA, seg_pos={0 0}, upload=5895]: Uploading piece... Jul 27 04:06:23.862060983 (6h14m39.642994358s) [noise=true , address=45.91.116.9:4262 , piece_id=M5ETD45HH2XZD6HDLTJ3JO2BOOIROXFWKVPTN42PSQEWG4F7KAAQ, seg_pos={0 0}, upload=5895]: Uploading piece... Jul 27 04:06:23.862759464 (6h14m39.64369284s) [noise=true , address=104.36.82.4:4804 , piece_id=IU6FDNVB5AUPA7MJ6RSXUTLBETPHOFGNJ7YGWBFIZSQFSO6W6BBA, seg_pos={0 0}, upload=5895]: Uploading piece... Jul 27 04:06:23.862835086 (6h14m39.643768461s) [noise=true , address=104.36.82.6:4873 , piece_id=YP4VJORZZUL5EVCH3P7LD3LNVMHB7CWRTKHGIA7UROXE44N6MMZA, seg_pos={0 0}, upload=5895]: Uploading piece... Jul 27 04:06:23.863593880 (6h14m39.644527258s) [noise=true , address=77.90.31.15:4697 , piece_id=Q76TJDSZPSYZSJQNFVOI462U6DFILMH2RV3LQISODXSOCXGRKW6Q, seg_pos={0 0}, upload=5895]: Uploading piece... Jul 27 04:06:23.863656891 (6h14m39.644590266s) [noise=true , address=45.91.116.7:4011 , piece_id=XMH6MCCHZBBRQWHJFWLRGYHLQD4KUGSGL7MMZSKA5TZJCND3DUDA, seg_pos={0 0}, upload=5895]: Uploading piece... Jul 27 04:06:23.864408612 (6h14m39.645341987s) [noise=true , address=45.91.116.6:4973 , piece_id=J3VULPU24CIM64VEVE3UHO62HPN6TJXGZU6BCJIS5JQJKHJK2LIQ, seg_pos={0 0}, upload=5895]: Uploading piece... Jul 27 04:06:23.864521549 (6h14m39.64545491s) [noise=true , address=104.36.82.10:4079 , piece_id=2SH5D6JOMLFR7VSN7ZY5CLMR5RHIRFYQ7Z5I2JIYTJQM43G4O6OQ, seg_pos={0 0}, upload=5895]: Uploading piece... Jul 27 04:06:23.865518491 (6h14m39.646451872s) [noise=true , address=77.90.31.17:4771 , piece_id=U2LF22GKCMKRZN6EWCF26ITDZQHBFZXCGSQXT33MCX6P57IGI35Q, seg_pos={0 0}, upload=5895]: Uploading piece... Jul 27 04:06:23.865654950 (6h14m39.646588325s) [noise=true , address=45.91.116.8:4048 , piece_id=5REWMVVTBVLDQXQ5KAEHLY4VJL5SGF7ONQWDQQ3N2FDU3OAO6AZA, seg_pos={0 0}, upload=5895]: Uploading piece... Jul 27 04:06:23.866190892 (6h14m39.647124271s) [noise=true , address=104.36.82.12:4151 , piece_id=E2TEUTQBD4Y4AUHSCPMCIFEXBSY7S5HJWBKNKWWTUWEW2BLAAE6Q, seg_pos={0 0}, upload=5895]: Uploading piece... Jul 27 04:06:23.866412590 (6h14m39.647345968s) [noise=true , address=104.36.82.11:4115 , piece_id=YXMNVT6ZHGESGQ2BXKZUVR7U2CNNAI2PO7ZTMDG5UIOV7HTDDUTA, seg_pos={0 0}, upload=5895]: Uploading piece... Jul 27 04:06:23.866715972 (6h14m39.647649359s) [noise=true , address=45.91.116.13:4153 , piece_id=H4535KOCG3F4RKA6AKKMTJSOM6Q6PGRC6M7FR3UIS3BORTSP5UQA, seg_pos={0 0}, upload=5895]: Uploading piece... Jul 27 04:06:23.866998965 (6h14m39.64793234s) [noise=true , address=104.36.82.13:4186 , piece_id=TBLDKALGJYBGR5NLCOCKJ4QKEFNIJXCKAFUBD7VR3UMNWEMKMKHQ, seg_pos={0 0}, upload=5895]: Uploading piece... Jul 27 04:06:23.867230593 (6h14m39.648163974s) [noise=true , address=104.36.82.16:4293 , piece_id=AHH6PRFUSGTDGB7H52DRITARA4IF6GFLVSRYS2Y6EPJCZSKUOYWQ, seg_pos={0 0}, upload=5895]: Uploading piece... Jul 27 04:06:23.949422808 (6h14m39.730356187s) [upload=5895]: Next returned nil segment Jul 27 04:06:23.949442576 (6h14m39.73037596s) [upload=5895]: Waiting for error group managing segments... Jul 27 04:06:23.953497426 (6h14m39.734430802s) [noise=true , address=45.91.116.10:4081 , piece_id=XGDTSEZOUNRTEVWHTO3DHTZMH2VZZOPJPEKLCFAHTXESLLYABHSA, seg_pos={0 0}, upload=5895]: Done uploading piece. err: <nil> Jul 27 04:06:23.953531796 (6h14m39.734465159s) [noise=true , address=45.91.116.6:4973 , piece_id=J3VULPU24CIM64VEVE3UHO62HPN6TJXGZU6BCJIS5JQJKHJK2LIQ, seg_pos={0 0}, upload=5895]: Done uploading piece. err: <nil> Jul 27 04:06:23.954068660 (6h14m39.735002029s) [noise=true , address=45.91.116.9:4262 , piece_id=M5ETD45HH2XZD6HDLTJ3JO2BOOIROXFWKVPTN42PSQEWG4F7KAAQ, seg_pos={0 0}, upload=5895]: Done uploading piece. err: <nil> Jul 27 04:06:23.954194339 (6h14m39.735127707s) [noise=true , address=45.91.116.8:4048 , piece_id=5REWMVVTBVLDQXQ5KAEHLY4VJL5SGF7ONQWDQQ3N2FDU3OAO6AZA, seg_pos={0 0}, upload=5895]: Done uploading piece. err: <nil> Jul 27 04:06:23.954601730 (6h14m39.735535112s) [noise=true , address=45.91.116.7:4011 , piece_id=XMH6MCCHZBBRQWHJFWLRGYHLQD4KUGSGL7MMZSKA5TZJCND3DUDA, seg_pos={0 0}, upload=5895]: Done uploading piece. err: <nil> Jul 27 04:06:23.954755872 (6h14m39.735689237s) [noise=true , address=45.91.116.13:4153 , piece_id=H4535KOCG3F4RKA6AKKMTJSOM6Q6PGRC6M7FR3UIS3BORTSP5UQA, seg_pos={0 0}, upload=5895]: Done uploading piece. err: <nil> Jul 27 04:06:23.955611883 (6h14m39.73654526s) [noise=true , address=77.90.31.19:4843 , piece_id=YML5RDQIK432Z4FSE4DKXIV5U47LQBEZRFMAVLANUSGFUQR23UWQ, seg_pos={0 0}, upload=5895]: Done uploading piece. err: <nil> Jul 27 04:06:23.956373391 (6h14m39.737306756s) [noise=true , address=77.90.31.15:4697 , piece_id=Q76TJDSZPSYZSJQNFVOI462U6DFILMH2RV3LQISODXSOCXGRKW6Q, seg_pos={0 0}, upload=5895]: Done uploading piece. err: <nil> Jul 27 04:06:23.958659122 (6h14m39.73959249s) [noise=true , address=77.90.31.17:4771 , piece_id=U2LF22GKCMKRZN6EWCF26ITDZQHBFZXCGSQXT33MCX6P57IGI35Q, seg_pos={0 0}, upload=5895]: Done uploading piece. err: <nil> Jul 27 04:06:35.058944013 (6h14m50.83987739s) [noise=true , address=104.36.82.16:4293 , piece_id=AHH6PRFUSGTDGB7H52DRITARA4IF6GFLVSRYS2Y6EPJCZSKUOYWQ, seg_pos={0 0}, upload=5895]: Done uploading piece. err: <nil> Jul 27 04:06:41.103204002 (6h14m56.884137386s) [noise=true , address=104.36.82.4:4804 , piece_id=IU6FDNVB5AUPA7MJ6RSXUTLBETPHOFGNJ7YGWBFIZSQFSO6W6BBA, seg_pos={0 0}, upload=5895]: Done uploading piece. err: <nil> Jul 27 04:06:43.191702285 (6h14m58.972635651s) [noise=true , address=104.36.82.6:4873 , piece_id=YP4VJORZZUL5EVCH3P7LD3LNVMHB7CWRTKHGIA7UROXE44N6MMZA, seg_pos={0 0}, upload=5895]: Done uploading piece. err: <nil> Jul 27 04:06:53.777973493 (6h15m9.558906861s) [noise=true , address=104.36.82.11:4115 , piece_id=YXMNVT6ZHGESGQ2BXKZUVR7U2CNNAI2PO7ZTMDG5UIOV7HTDDUTA, seg_pos={0 0}, upload=5895]: Done uploading piece. err: ecclient: upload failed (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR, address:104.36.82.11:4115): protocol: write tcp 159.223.143.39:34210->104.36.82.11:4115: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 04:06:53.778019532 (6h15m9.558952909s) [noise=true , address=104.36.82.12:4151 , piece_id=E2TEUTQBD4Y4AUHSCPMCIFEXBSY7S5HJWBKNKWWTUWEW2BLAAE6Q, seg_pos={0 0}, upload=5895]: Done uploading piece. err: ecclient: upload failed (node:1w6pwZMdeQKjt92qx6WbTWU5VPYJto5gS87f6iKvCBZH6LfyeB, address:104.36.82.12:4151): protocol: rpc: dial tcp 104.36.82.12:4151: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 04:06:53.778043653 (6h15m9.558977021s) [noise=true , address=104.36.82.13:4186 , piece_id=TBLDKALGJYBGR5NLCOCKJ4QKEFNIJXCKAFUBD7VR3UMNWEMKMKHQ, seg_pos={0 0}, upload=5895]: Done uploading piece. err: ecclient: upload failed (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht, address:104.36.82.13:4186): protocol: read tcp 159.223.143.39:34340->104.36.82.13:4186: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 04:06:53.778095228 (6h15m9.559028593s) [noise=true , address=104.36.82.10:4079 , piece_id=2SH5D6JOMLFR7VSN7ZY5CLMR5RHIRFYQ7Z5I2JIYTJQM43G4O6OQ, seg_pos={0 0}, upload=5895]: Done uploading piece. err: ecclient: upload failed (node:12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud, address:104.36.82.10:4079): protocol: write tcp 159.223.143.39:53840->104.36.82.10:4079: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 04:06:53.778142167 (6h15m9.559075532s) [seg_pos={0 0}, upload=5895]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 27 04:06:53.837032161 (6h15m9.617965543s) [upload=5895]: Done uploading segments... Jul 27 04:06:53.837055996 (6h15m9.617989366s) [upload=5895]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 420.409941ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................................................X......] duration: 507.419469ms FAIL (2025-07-27 04:21:00 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 12) Duration: 507.456167ms core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................................X....................] duration: 429.277146ms FAIL (2025-07-27 04:20:22 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== ece_id=NYOFDEZ7CV4PHGQ6JZMEEVAYGJRPYG6IBW7MYJO42HMWOJM7PHZA, seg_pos={0 0}, upload=5879]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12crcXpNe9NgeXTsUpooLSBgrSUFbRxbSbanLa39w2Hr5mGFww): protocol: read tcp 159.223.143.39:53630->104.36.82.5:4840: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 04:05:32.508904485 (6h13m48.289837866s) [seg_pos={0 0}, upload=5879]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 27 04:05:32.509028070 (6h13m48.289961457s) [upload=5879]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 27 04:05:32.509034645 (6h13m48.289968129s) [upload=5879]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 27 04:05:32.581834839 (6h13m48.362768225s) [upload=5879]: Done uploading segments... Jul 27 04:05:32.581875919 (6h13m48.362809298s) [upload=5879]: Upload finished. err: <nil> Jul 27 04:06:23.829155331 (6h14m39.610088696s) [upload=5895]: Starting upload Jul 27 04:06:23.829225021 (6h14m39.610158398s) [upload=5895]: Done starting upload Jul 27 04:06:23.829294606 (6h14m39.61022799s) [upload=5895]: Uploading segments... Jul 27 04:06:23.829577656 (6h14m39.610511053s) [upload=5895]: Got next segment. Inline: false Jul 27 04:06:23.829649664 (6h14m39.610583027s) [upload=5895]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 27 04:06:23.829657065 (6h14m39.61059043s) [upload=5895]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 27 04:06:23.861405530 (6h14m39.642338913s) [seg_pos={0 0}, upload=5895]: Begin upload segment... Jul 27 04:06:23.861476351 (6h14m39.642409713s) [seg_pos={0 0}, upload=5895]: Done begin upload segment. Jul 27 04:06:23.861509120 (6h14m39.642442506s) [noise=true , address=77.90.31.19:4843 , piece_id=YML5RDQIK432Z4FSE4DKXIV5U47LQBEZRFMAVLANUSGFUQR23UWQ, seg_pos={0 0}, upload=5895]: Uploading piece... Jul 27 04:06:23.861953723 (6h14m39.6428871s) [noise=true , address=45.91.116.10:4081 , piece_id=XGDTSEZOUNRTEVWHTO3DHTZMH2VZZOPJPEKLCFAHTXESLLYABHSA, seg_pos={0 0}, upload=5895]: Uploading piece... Jul 27 04:06:23.862060983 (6h14m39.642994358s) [noise=true , address=45.91.116.9:4262 , piece_id=M5ETD45HH2XZD6HDLTJ3JO2BOOIROXFWKVPTN42PSQEWG4F7KAAQ, seg_pos={0 0}, upload=5895]: Uploading piece... Jul 27 04:06:23.862759464 (6h14m39.64369284s) [noise=true , address=104.36.82.4:4804 , piece_id=IU6FDNVB5AUPA7MJ6RSXUTLBETPHOFGNJ7YGWBFIZSQFSO6W6BBA, seg_pos={0 0}, upload=5895]: Uploading piece... Jul 27 04:06:23.862835086 (6h14m39.643768461s) [noise=true , address=104.36.82.6:4873 , piece_id=YP4VJORZZUL5EVCH3P7LD3LNVMHB7CWRTKHGIA7UROXE44N6MMZA, seg_pos={0 0}, upload=5895]: Uploading piece... Jul 27 04:06:23.863593880 (6h14m39.644527258s) [noise=true , address=77.90.31.15:4697 , piece_id=Q76TJDSZPSYZSJQNFVOI462U6DFILMH2RV3LQISODXSOCXGRKW6Q, seg_pos={0 0}, upload=5895]: Uploading piece... Jul 27 04:06:23.863656891 (6h14m39.644590266s) [noise=true , address=45.91.116.7:4011 , piece_id=XMH6MCCHZBBRQWHJFWLRGYHLQD4KUGSGL7MMZSKA5TZJCND3DUDA, seg_pos={0 0}, upload=5895]: Uploading piece... Jul 27 04:06:23.864408612 (6h14m39.645341987s) [noise=true , address=45.91.116.6:4973 , piece_id=J3VULPU24CIM64VEVE3UHO62HPN6TJXGZU6BCJIS5JQJKHJK2LIQ, seg_pos={0 0}, upload=5895]: Uploading piece... Jul 27 04:06:23.864521549 (6h14m39.64545491s) [noise=true , address=104.36.82.10:4079 , piece_id=2SH5D6JOMLFR7VSN7ZY5CLMR5RHIRFYQ7Z5I2JIYTJQM43G4O6OQ, seg_pos={0 0}, upload=5895]: Uploading piece... Jul 27 04:06:23.865518491 (6h14m39.646451872s) [noise=true , address=77.90.31.17:4771 , piece_id=U2LF22GKCMKRZN6EWCF26ITDZQHBFZXCGSQXT33MCX6P57IGI35Q, seg_pos={0 0}, upload=5895]: Uploading piece... Jul 27 04:06:23.865654950 (6h14m39.646588325s) [noise=true , address=45.91.116.8:4048 , piece_id=5REWMVVTBVLDQXQ5KAEHLY4VJL5SGF7ONQWDQQ3N2FDU3OAO6AZA, seg_pos={0 0}, upload=5895]: Uploading piece... Jul 27 04:06:23.866190892 (6h14m39.647124271s) [noise=true , address=104.36.82.12:4151 , piece_id=E2TEUTQBD4Y4AUHSCPMCIFEXBSY7S5HJWBKNKWWTUWEW2BLAAE6Q, seg_pos={0 0}, upload=5895]: Uploading piece... Jul 27 04:06:23.866412590 (6h14m39.647345968s) [noise=true , address=104.36.82.11:4115 , piece_id=YXMNVT6ZHGESGQ2BXKZUVR7U2CNNAI2PO7ZTMDG5UIOV7HTDDUTA, seg_pos={0 0}, upload=5895]: Uploading piece... Jul 27 04:06:23.866715972 (6h14m39.647649359s) [noise=true , address=45.91.116.13:4153 , piece_id=H4535KOCG3F4RKA6AKKMTJSOM6Q6PGRC6M7FR3UIS3BORTSP5UQA, seg_pos={0 0}, upload=5895]: Uploading piece... Jul 27 04:06:23.866998965 (6h14m39.64793234s) [noise=true , address=104.36.82.13:4186 , piece_id=TBLDKALGJYBGR5NLCOCKJ4QKEFNIJXCKAFUBD7VR3UMNWEMKMKHQ, seg_pos={0 0}, upload=5895]: Uploading piece... Jul 27 04:06:23.867230593 (6h14m39.648163974s) [noise=true , address=104.36.82.16:4293 , piece_id=AHH6PRFUSGTDGB7H52DRITARA4IF6GFLVSRYS2Y6EPJCZSKUOYWQ, seg_pos={0 0}, upload=5895]: Uploading piece... Jul 27 04:06:23.949422808 (6h14m39.730356187s) [upload=5895]: Next returned nil segment Jul 27 04:06:23.949442576 (6h14m39.73037596s) [upload=5895]: Waiting for error group managing segments... Jul 27 04:06:23.953497426 (6h14m39.734430802s) [noise=true , address=45.91.116.10:4081 , piece_id=XGDTSEZOUNRTEVWHTO3DHTZMH2VZZOPJPEKLCFAHTXESLLYABHSA, seg_pos={0 0}, upload=5895]: Done uploading piece. err: <nil> Jul 27 04:06:23.953531796 (6h14m39.734465159s) [noise=true , address=45.91.116.6:4973 , piece_id=J3VULPU24CIM64VEVE3UHO62HPN6TJXGZU6BCJIS5JQJKHJK2LIQ, seg_pos={0 0}, upload=5895]: Done uploading piece. err: <nil> Jul 27 04:06:23.954068660 (6h14m39.735002029s) [noise=true , address=45.91.116.9:4262 , piece_id=M5ETD45HH2XZD6HDLTJ3JO2BOOIROXFWKVPTN42PSQEWG4F7KAAQ, seg_pos={0 0}, upload=5895]: Done uploading piece. err: <nil> Jul 27 04:06:23.954194339 (6h14m39.735127707s) [noise=true , address=45.91.116.8:4048 , piece_id=5REWMVVTBVLDQXQ5KAEHLY4VJL5SGF7ONQWDQQ3N2FDU3OAO6AZA, seg_pos={0 0}, upload=5895]: Done uploading piece. err: <nil> Jul 27 04:06:23.954601730 (6h14m39.735535112s) [noise=true , address=45.91.116.7:4011 , piece_id=XMH6MCCHZBBRQWHJFWLRGYHLQD4KUGSGL7MMZSKA5TZJCND3DUDA, seg_pos={0 0}, upload=5895]: Done uploading piece. err: <nil> Jul 27 04:06:23.954755872 (6h14m39.735689237s) [noise=true , address=45.91.116.13:4153 , piece_id=H4535KOCG3F4RKA6AKKMTJSOM6Q6PGRC6M7FR3UIS3BORTSP5UQA, seg_pos={0 0}, upload=5895]: Done uploading piece. err: <nil> Jul 27 04:06:23.955611883 (6h14m39.73654526s) [noise=true , address=77.90.31.19:4843 , piece_id=YML5RDQIK432Z4FSE4DKXIV5U47LQBEZRFMAVLANUSGFUQR23UWQ, seg_pos={0 0}, upload=5895]: Done uploading piece. err: <nil> Jul 27 04:06:23.956373391 (6h14m39.737306756s) [noise=true , address=77.90.31.15:4697 , piece_id=Q76TJDSZPSYZSJQNFVOI462U6DFILMH2RV3LQISODXSOCXGRKW6Q, seg_pos={0 0}, upload=5895]: Done uploading piece. err: <nil> Jul 27 04:06:23.958659122 (6h14m39.73959249s) [noise=true , address=77.90.31.17:4771 , piece_id=U2LF22GKCMKRZN6EWCF26ITDZQHBFZXCGSQXT33MCX6P57IGI35Q, seg_pos={0 0}, upload=5895]: Done uploading piece. err: <nil> Jul 27 04:06:35.058944013 (6h14m50.83987739s) [noise=true , address=104.36.82.16:4293 , piece_id=AHH6PRFUSGTDGB7H52DRITARA4IF6GFLVSRYS2Y6EPJCZSKUOYWQ, seg_pos={0 0}, upload=5895]: Done uploading piece. err: <nil> Jul 27 04:06:41.103204002 (6h14m56.884137386s) [noise=true , address=104.36.82.4:4804 , piece_id=IU6FDNVB5AUPA7MJ6RSXUTLBETPHOFGNJ7YGWBFIZSQFSO6W6BBA, seg_pos={0 0}, upload=5895]: Done uploading piece. err: <nil> Jul 27 04:06:43.191702285 (6h14m58.972635651s) [noise=true , address=104.36.82.6:4873 , piece_id=YP4VJORZZUL5EVCH3P7LD3LNVMHB7CWRTKHGIA7UROXE44N6MMZA, seg_pos={0 0}, upload=5895]: Done uploading piece. err: <nil> Jul 27 04:06:53.777973493 (6h15m9.558906861s) [noise=true , address=104.36.82.11:4115 , piece_id=YXMNVT6ZHGESGQ2BXKZUVR7U2CNNAI2PO7ZTMDG5UIOV7HTDDUTA, seg_pos={0 0}, upload=5895]: Done uploading piece. err: ecclient: upload failed (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR, address:104.36.82.11:4115): protocol: write tcp 159.223.143.39:34210->104.36.82.11:4115: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 04:06:53.778019532 (6h15m9.558952909s) [noise=true , address=104.36.82.12:4151 , piece_id=E2TEUTQBD4Y4AUHSCPMCIFEXBSY7S5HJWBKNKWWTUWEW2BLAAE6Q, seg_pos={0 0}, upload=5895]: Done uploading piece. err: ecclient: upload failed (node:1w6pwZMdeQKjt92qx6WbTWU5VPYJto5gS87f6iKvCBZH6LfyeB, address:104.36.82.12:4151): protocol: rpc: dial tcp 104.36.82.12:4151: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 04:06:53.778043653 (6h15m9.558977021s) [noise=true , address=104.36.82.13:4186 , piece_id=TBLDKALGJYBGR5NLCOCKJ4QKEFNIJXCKAFUBD7VR3UMNWEMKMKHQ, seg_pos={0 0}, upload=5895]: Done uploading piece. err: ecclient: upload failed (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht, address:104.36.82.13:4186): protocol: read tcp 159.223.143.39:34340->104.36.82.13:4186: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 04:06:53.778095228 (6h15m9.559028593s) [noise=true , address=104.36.82.10:4079 , piece_id=2SH5D6JOMLFR7VSN7ZY5CLMR5RHIRFYQ7Z5I2JIYTJQM43G4O6OQ, seg_pos={0 0}, upload=5895]: Done uploading piece. err: ecclient: upload failed (node:12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud, address:104.36.82.10:4079): protocol: write tcp 159.223.143.39:53840->104.36.82.10:4079: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 04:06:53.778142167 (6h15m9.559075532s) [seg_pos={0 0}, upload=5895]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 27 04:06:53.837032161 (6h15m9.617965543s) [upload=5895]: Done uploading segments... Jul 27 04:06:53.837055996 (6h15m9.617989366s) [upload=5895]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 429.306246ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................................................X.....] duration: 557.384058ms FAIL (2025-07-27 04:20:19 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 12) Duration: 557.412723ms core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [...........................................X................] duration: 11.004459641s FAIL (2025-07-27 04:20:15 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== {0 0}, upload=5911]: Begin upload segment... Jul 27 04:07:17.745307658 (6h15m33.526241043s) [seg_pos={0 0}, upload=5911]: Done begin upload segment. Jul 27 04:07:17.745349241 (6h15m33.526282625s) [noise=true , address=104.36.82.52:4098 , piece_id=M2ETF25QQ53RRGFPUKXJRINIWV7AWXSECEDR2AIM645IED72NURQ, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.745439694 (6h15m33.526373073s) [noise=true , address=104.36.82.56:4244 , piece_id=LQCZ5W54NBHWAVBVYK7HYSSJDC73E6FV4CP4YIM4L5H2N3OLOZXA, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.745972271 (6h15m33.526905654s) [noise=true , address=104.36.82.36:4650 , piece_id=BKJULWGPESBINNTSUYP6E4KZRXKPACVU5I7DOSZBO4YMK2V26HEQ, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.745980243 (6h15m33.526913621s) [noise=true , address=104.36.82.38:4722 , piece_id=B2MBQY5CTMEI6WSE7FTLOV5RUJ22SCNG7N2EEJILEATGJ4TIALXA, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.746403818 (6h15m33.527337198s) [noise=true , address=104.36.82.22:4534 , piece_id=CW4AQT3XUGA4A5OC6NUY4NYOOAMLN5FFJJN5PQBYEON7HPD5LCYQ, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.746455493 (6h15m33.527388875s) [noise=true , address=104.36.82.14:4248 , piece_id=LDSGCUM7Q2VX5CDFI4JRGEEG7D5H5Q5Z5WUWKU277VN2JFRX6UGA, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.746872127 (6h15m33.527805505s) [noise=true , address=104.36.82.40:4954 , piece_id=YLOHTJLRAED66OMEQYPZHNNRI77V6WOR4EHTEZOOEHMGZHJ2O3QQ, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.747123162 (6h15m33.52805654s) [noise=true , address=104.36.82.46:4496 , piece_id=GWNQR5WRGRRQBLSGWJFD3O3UNCMNE62JQNPIV5SYKBUIL3BQKA5A, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.747280678 (6h15m33.528214053s) [noise=true , address=104.36.82.47:4524 , piece_id=DONJFTKUQ2XCW6BOU7AVOKJ7XMDMBPGQVUPRANH7X2YS7PJXUKIA, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.747736400 (6h15m33.528669779s) [noise=true , address=104.36.82.44:4893 , piece_id=HUJICRJ6W54O3AZZR6H3KG6E4TGBTNS46O6XVMQM3PFAZE7A2HZA, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.747755734 (6h15m33.528689109s) [noise=true , address=104.36.82.55:4204 , piece_id=72WISGRHMOAZDV5EDGTBOMLBL4HDOF3CBEMQAAIRGILPC7J5JKAQ, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.748313124 (6h15m33.5292465s) [noise=true , address=104.36.82.58:4287 , piece_id=BVDYUEWU7TVEQDMZXX5SJNS4QICVKMTGFLCUV6P2YHOE7TI25VXA, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.748485217 (6h15m33.529418594s) [noise=true , address=104.36.82.53:4123 , piece_id=NXJRZ5BQSLXSOIOSUJ4MM4AGKHP4UIDUGEJKLUUX7YOGQJA6WWZA, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.748831329 (6h15m33.529764704s) [noise=true , address=104.36.82.42:4829 , piece_id=ZLWJFP3LGX3YLXBJYIR66DXTYEVDBKXKVXCQSF2PCYCYJBIEJGMA, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.749299293 (6h15m33.530232669s) [noise=true , address=104.36.82.8:4002 , piece_id=6XSPWQC3US355PS6IQ4K3IYNQZH5MAT4IVIPGSYRUM3XYPSCWE6Q, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.749790974 (6h15m33.53072436s) [noise=true , address=104.36.82.44:4901 , piece_id=N4YLCLR2QZSULUL2XWONEKBFUMNOU7KUX2QEOK4SSDFSCI7IL6LA, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.750478221 (6h15m33.531411597s) [noise=true , address=104.36.82.54:4172 , piece_id=ZNA53MWDRWPUH6FUR4DJERBDT2IOYGA3IYUADKXXIOLQJLQOXE7Q, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.752884255 (6h15m33.533817625s) [noise=true , address=64.49.13.140:4508 , piece_id=MZ52U75VOIK6Q7QFXMMYRRG7JXFWDFU2JWQFYPERUCMH3BUX6XPQ, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.879839555 (6h15m33.660772917s) [upload=5911]: Next returned nil segment Jul 27 04:07:17.879861389 (6h15m33.660794765s) [upload=5911]: Waiting for error group managing segments... Jul 27 04:07:17.888652438 (6h15m33.669585811s) [noise=true , address=104.36.82.58:4287 , piece_id=BVDYUEWU7TVEQDMZXX5SJNS4QICVKMTGFLCUV6P2YHOE7TI25VXA, seg_pos={0 0}, upload=5911]: Done uploading piece. err: <nil> Jul 27 04:07:17.952192045 (6h15m33.733125429s) [noise=true , address=104.36.82.53:4123 , piece_id=NXJRZ5BQSLXSOIOSUJ4MM4AGKHP4UIDUGEJKLUUX7YOGQJA6WWZA, seg_pos={0 0}, upload=5911]: Done uploading piece. err: <nil> Jul 27 04:07:17.981907539 (6h15m33.762840911s) [noise=true , address=104.36.82.56:4244 , piece_id=LQCZ5W54NBHWAVBVYK7HYSSJDC73E6FV4CP4YIM4L5H2N3OLOZXA, seg_pos={0 0}, upload=5911]: Done uploading piece. err: <nil> Jul 27 04:07:17.982370496 (6h15m33.7633039s) [noise=true , address=104.36.82.55:4204 , piece_id=72WISGRHMOAZDV5EDGTBOMLBL4HDOF3CBEMQAAIRGILPC7J5JKAQ, seg_pos={0 0}, upload=5911]: Done uploading piece. err: <nil> Jul 27 04:07:18.009548506 (6h15m33.790481886s) [noise=true , address=104.36.82.52:4098 , piece_id=M2ETF25QQ53RRGFPUKXJRINIWV7AWXSECEDR2AIM645IED72NURQ, seg_pos={0 0}, upload=5911]: Done uploading piece. err: <nil> Jul 27 04:07:18.023770598 (6h15m33.804703983s) [noise=true , address=104.36.82.40:4954 , piece_id=YLOHTJLRAED66OMEQYPZHNNRI77V6WOR4EHTEZOOEHMGZHJ2O3QQ, seg_pos={0 0}, upload=5911]: Done uploading piece. err: <nil> Jul 27 04:07:18.139745581 (6h15m33.92067895s) [noise=true , address=104.36.82.22:4534 , piece_id=CW4AQT3XUGA4A5OC6NUY4NYOOAMLN5FFJJN5PQBYEON7HPD5LCYQ, seg_pos={0 0}, upload=5911]: Done uploading piece. err: <nil> Jul 27 04:07:18.423748140 (6h15m34.204681511s) [noise=true , address=64.49.13.140:4508 , piece_id=MZ52U75VOIK6Q7QFXMMYRRG7JXFWDFU2JWQFYPERUCMH3BUX6XPQ, seg_pos={0 0}, upload=5911]: Done uploading piece. err: <nil> Jul 27 04:08:16.739940381 (6h16m32.52087375s) [noise=true , address=104.36.82.54:4172 , piece_id=ZNA53MWDRWPUH6FUR4DJERBDT2IOYGA3IYUADKXXIOLQJLQOXE7Q, seg_pos={0 0}, upload=5911]: Done uploading piece. err: <nil> Jul 27 04:08:17.177295397 (6h16m32.958228766s) [noise=true , address=104.36.82.8:4002 , piece_id=6XSPWQC3US355PS6IQ4K3IYNQZH5MAT4IVIPGSYRUM3XYPSCWE6Q, seg_pos={0 0}, upload=5911]: Done uploading piece. err: <nil> Jul 27 04:08:17.534296584 (6h16m33.315229953s) [noise=true , address=104.36.82.36:4650 , piece_id=BKJULWGPESBINNTSUYP6E4KZRXKPACVU5I7DOSZBO4YMK2V26HEQ, seg_pos={0 0}, upload=5911]: Done uploading piece. err: <nil> Jul 27 04:08:17.677851566 (6h16m33.458784943s) [noise=true , address=104.36.82.14:4248 , piece_id=LDSGCUM7Q2VX5CDFI4JRGEEG7D5H5Q5Z5WUWKU277VN2JFRX6UGA, seg_pos={0 0}, upload=5911]: Done uploading piece. err: ecclient: upload failed (node:1u6ir2PLqDa9xGm64cuqq1RqY4qeBa1fqCo9VghspBf54XEXqn, address:104.36.82.14:4248): protocol: rpc: dial tcp 104.36.82.14:4248: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 04:08:17.678115170 (6h16m33.459048549s) [noise=true , address=104.36.82.46:4496 , piece_id=GWNQR5WRGRRQBLSGWJFD3O3UNCMNE62JQNPIV5SYKBUIL3BQKA5A, seg_pos={0 0}, upload=5911]: Done uploading piece. err: ecclient: upload failed (node:12V8XhSWkyGUVYFR8MykifFAP1YAfGCwzqucDcMMfzw3xsVW77Z, address:104.36.82.46:4496): protocol: read tcp 159.223.143.39:52800->104.36.82.46:4496: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 04:08:17.678144409 (6h16m33.459077789s) [noise=true , address=104.36.82.44:4901 , piece_id=N4YLCLR2QZSULUL2XWONEKBFUMNOU7KUX2QEOK4SSDFSCI7IL6LA, seg_pos={0 0}, upload=5911]: Done uploading piece. err: ecclient: upload failed (node:1FHE84zvBb1MKwB1Ht7puTafgQxDEvPt4mYqkYVSSufAMpWQA1, address:104.36.82.44:4901): protocol: rpc: dial tcp 104.36.82.44:4901: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 04:08:17.678229622 (6h16m33.459162988s) [noise=true , address=104.36.82.44:4893 , piece_id=HUJICRJ6W54O3AZZR6H3KG6E4TGBTNS46O6XVMQM3PFAZE7A2HZA, seg_pos={0 0}, upload=5911]: Done uploading piece. err: ecclient: upload failed (node:17VMFZ2T8jrSJVtchciJ9SnyrR6JstvcMoXrN5Q3zTD1aQU1oJ, address:104.36.82.44:4893): protocol: write tcp 159.223.143.39:35302->104.36.82.44:4893: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 04:08:17.678310218 (6h16m33.459243583s) [noise=true , address=104.36.82.42:4829 , piece_id=ZLWJFP3LGX3YLXBJYIR66DXTYEVDBKXKVXCQSF2PCYCYJBIEJGMA, seg_pos={0 0}, upload=5911]: Done uploading piece. err: ecclient: upload failed (node:1F5T8Vt1x4nXFaSenvGBCYjxhRL1H1sxRvzLjW3CqzwDVhq39F, address:104.36.82.42:4829): protocol: rpc: dial tcp 104.36.82.42:4829: i/o timeout; conn closed: op; piecestore: piecestore close: context deadline exceeded Jul 27 04:08:17.678412989 (6h16m33.459346365s) [noise=true , address=104.36.82.38:4722 , piece_id=B2MBQY5CTMEI6WSE7FTLOV5RUJ22SCNG7N2EEJILEATGJ4TIALXA, seg_pos={0 0}, upload=5911]: Done uploading piece. err: ecclient: upload failed (node:12Zw3dtUg9gARz52KhwDLTjcqC3CNm7dpPzpdx7P7PbL1jwe8uc, address:104.36.82.38:4722): protocol: read tcp 159.223.143.39:56048->104.36.82.38:4722: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 04:08:17.678455247 (6h16m33.459388624s) [noise=true , address=104.36.82.47:4524 , piece_id=DONJFTKUQ2XCW6BOU7AVOKJ7XMDMBPGQVUPRANH7X2YS7PJXUKIA, seg_pos={0 0}, upload=5911]: Done uploading piece. err: ecclient: upload failed (node:1xfbCzjSwFDRrR1gPggeN8tNCLbRRWDVzCJ6mqhZsbVSe8d4s4, address:104.36.82.47:4524): protocol: read tcp 159.223.143.39:46694->104.36.82.47:4524: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 04:08:17.678510668 (6h16m33.459444052s) [seg_pos={0 0}, upload=5911]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Jul 27 04:08:17.706442864 (6h16m33.487376232s) [upload=5911]: Done uploading segments... Jul 27 04:08:17.706495329 (6h16m33.487428705s) [upload=5911]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 11.004570661s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [............................................X...............] duration: 1.474958171s FAIL (2025-07-27 04:19:18 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== {0 0}, upload=5911]: Begin upload segment... Jul 27 04:07:17.745307658 (6h15m33.526241043s) [seg_pos={0 0}, upload=5911]: Done begin upload segment. Jul 27 04:07:17.745349241 (6h15m33.526282625s) [noise=true , address=104.36.82.52:4098 , piece_id=M2ETF25QQ53RRGFPUKXJRINIWV7AWXSECEDR2AIM645IED72NURQ, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.745439694 (6h15m33.526373073s) [noise=true , address=104.36.82.56:4244 , piece_id=LQCZ5W54NBHWAVBVYK7HYSSJDC73E6FV4CP4YIM4L5H2N3OLOZXA, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.745972271 (6h15m33.526905654s) [noise=true , address=104.36.82.36:4650 , piece_id=BKJULWGPESBINNTSUYP6E4KZRXKPACVU5I7DOSZBO4YMK2V26HEQ, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.745980243 (6h15m33.526913621s) [noise=true , address=104.36.82.38:4722 , piece_id=B2MBQY5CTMEI6WSE7FTLOV5RUJ22SCNG7N2EEJILEATGJ4TIALXA, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.746403818 (6h15m33.527337198s) [noise=true , address=104.36.82.22:4534 , piece_id=CW4AQT3XUGA4A5OC6NUY4NYOOAMLN5FFJJN5PQBYEON7HPD5LCYQ, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.746455493 (6h15m33.527388875s) [noise=true , address=104.36.82.14:4248 , piece_id=LDSGCUM7Q2VX5CDFI4JRGEEG7D5H5Q5Z5WUWKU277VN2JFRX6UGA, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.746872127 (6h15m33.527805505s) [noise=true , address=104.36.82.40:4954 , piece_id=YLOHTJLRAED66OMEQYPZHNNRI77V6WOR4EHTEZOOEHMGZHJ2O3QQ, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.747123162 (6h15m33.52805654s) [noise=true , address=104.36.82.46:4496 , piece_id=GWNQR5WRGRRQBLSGWJFD3O3UNCMNE62JQNPIV5SYKBUIL3BQKA5A, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.747280678 (6h15m33.528214053s) [noise=true , address=104.36.82.47:4524 , piece_id=DONJFTKUQ2XCW6BOU7AVOKJ7XMDMBPGQVUPRANH7X2YS7PJXUKIA, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.747736400 (6h15m33.528669779s) [noise=true , address=104.36.82.44:4893 , piece_id=HUJICRJ6W54O3AZZR6H3KG6E4TGBTNS46O6XVMQM3PFAZE7A2HZA, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.747755734 (6h15m33.528689109s) [noise=true , address=104.36.82.55:4204 , piece_id=72WISGRHMOAZDV5EDGTBOMLBL4HDOF3CBEMQAAIRGILPC7J5JKAQ, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.748313124 (6h15m33.5292465s) [noise=true , address=104.36.82.58:4287 , piece_id=BVDYUEWU7TVEQDMZXX5SJNS4QICVKMTGFLCUV6P2YHOE7TI25VXA, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.748485217 (6h15m33.529418594s) [noise=true , address=104.36.82.53:4123 , piece_id=NXJRZ5BQSLXSOIOSUJ4MM4AGKHP4UIDUGEJKLUUX7YOGQJA6WWZA, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.748831329 (6h15m33.529764704s) [noise=true , address=104.36.82.42:4829 , piece_id=ZLWJFP3LGX3YLXBJYIR66DXTYEVDBKXKVXCQSF2PCYCYJBIEJGMA, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.749299293 (6h15m33.530232669s) [noise=true , address=104.36.82.8:4002 , piece_id=6XSPWQC3US355PS6IQ4K3IYNQZH5MAT4IVIPGSYRUM3XYPSCWE6Q, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.749790974 (6h15m33.53072436s) [noise=true , address=104.36.82.44:4901 , piece_id=N4YLCLR2QZSULUL2XWONEKBFUMNOU7KUX2QEOK4SSDFSCI7IL6LA, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.750478221 (6h15m33.531411597s) [noise=true , address=104.36.82.54:4172 , piece_id=ZNA53MWDRWPUH6FUR4DJERBDT2IOYGA3IYUADKXXIOLQJLQOXE7Q, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.752884255 (6h15m33.533817625s) [noise=true , address=64.49.13.140:4508 , piece_id=MZ52U75VOIK6Q7QFXMMYRRG7JXFWDFU2JWQFYPERUCMH3BUX6XPQ, seg_pos={0 0}, upload=5911]: Uploading piece... Jul 27 04:07:17.879839555 (6h15m33.660772917s) [upload=5911]: Next returned nil segment Jul 27 04:07:17.879861389 (6h15m33.660794765s) [upload=5911]: Waiting for error group managing segments... Jul 27 04:07:17.888652438 (6h15m33.669585811s) [noise=true , address=104.36.82.58:4287 , piece_id=BVDYUEWU7TVEQDMZXX5SJNS4QICVKMTGFLCUV6P2YHOE7TI25VXA, seg_pos={0 0}, upload=5911]: Done uploading piece. err: <nil> Jul 27 04:07:17.952192045 (6h15m33.733125429s) [noise=true , address=104.36.82.53:4123 , piece_id=NXJRZ5BQSLXSOIOSUJ4MM4AGKHP4UIDUGEJKLUUX7YOGQJA6WWZA, seg_pos={0 0}, upload=5911]: Done uploading piece. err: <nil> Jul 27 04:07:17.981907539 (6h15m33.762840911s) [noise=true , address=104.36.82.56:4244 , piece_id=LQCZ5W54NBHWAVBVYK7HYSSJDC73E6FV4CP4YIM4L5H2N3OLOZXA, seg_pos={0 0}, upload=5911]: Done uploading piece. err: <nil> Jul 27 04:07:17.982370496 (6h15m33.7633039s) [noise=true , address=104.36.82.55:4204 , piece_id=72WISGRHMOAZDV5EDGTBOMLBL4HDOF3CBEMQAAIRGILPC7J5JKAQ, seg_pos={0 0}, upload=5911]: Done uploading piece. err: <nil> Jul 27 04:07:18.009548506 (6h15m33.790481886s) [noise=true , address=104.36.82.52:4098 , piece_id=M2ETF25QQ53RRGFPUKXJRINIWV7AWXSECEDR2AIM645IED72NURQ, seg_pos={0 0}, upload=5911]: Done uploading piece. err: <nil> Jul 27 04:07:18.023770598 (6h15m33.804703983s) [noise=true , address=104.36.82.40:4954 , piece_id=YLOHTJLRAED66OMEQYPZHNNRI77V6WOR4EHTEZOOEHMGZHJ2O3QQ, seg_pos={0 0}, upload=5911]: Done uploading piece. err: <nil> Jul 27 04:07:18.139745581 (6h15m33.92067895s) [noise=true , address=104.36.82.22:4534 , piece_id=CW4AQT3XUGA4A5OC6NUY4NYOOAMLN5FFJJN5PQBYEON7HPD5LCYQ, seg_pos={0 0}, upload=5911]: Done uploading piece. err: <nil> Jul 27 04:07:18.423748140 (6h15m34.204681511s) [noise=true , address=64.49.13.140:4508 , piece_id=MZ52U75VOIK6Q7QFXMMYRRG7JXFWDFU2JWQFYPERUCMH3BUX6XPQ, seg_pos={0 0}, upload=5911]: Done uploading piece. err: <nil> Jul 27 04:08:16.739940381 (6h16m32.52087375s) [noise=true , address=104.36.82.54:4172 , piece_id=ZNA53MWDRWPUH6FUR4DJERBDT2IOYGA3IYUADKXXIOLQJLQOXE7Q, seg_pos={0 0}, upload=5911]: Done uploading piece. err: <nil> Jul 27 04:08:17.177295397 (6h16m32.958228766s) [noise=true , address=104.36.82.8:4002 , piece_id=6XSPWQC3US355PS6IQ4K3IYNQZH5MAT4IVIPGSYRUM3XYPSCWE6Q, seg_pos={0 0}, upload=5911]: Done uploading piece. err: <nil> Jul 27 04:08:17.534296584 (6h16m33.315229953s) [noise=true , address=104.36.82.36:4650 , piece_id=BKJULWGPESBINNTSUYP6E4KZRXKPACVU5I7DOSZBO4YMK2V26HEQ, seg_pos={0 0}, upload=5911]: Done uploading piece. err: <nil> Jul 27 04:08:17.677851566 (6h16m33.458784943s) [noise=true , address=104.36.82.14:4248 , piece_id=LDSGCUM7Q2VX5CDFI4JRGEEG7D5H5Q5Z5WUWKU277VN2JFRX6UGA, seg_pos={0 0}, upload=5911]: Done uploading piece. err: ecclient: upload failed (node:1u6ir2PLqDa9xGm64cuqq1RqY4qeBa1fqCo9VghspBf54XEXqn, address:104.36.82.14:4248): protocol: rpc: dial tcp 104.36.82.14:4248: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 04:08:17.678115170 (6h16m33.459048549s) [noise=true , address=104.36.82.46:4496 , piece_id=GWNQR5WRGRRQBLSGWJFD3O3UNCMNE62JQNPIV5SYKBUIL3BQKA5A, seg_pos={0 0}, upload=5911]: Done uploading piece. err: ecclient: upload failed (node:12V8XhSWkyGUVYFR8MykifFAP1YAfGCwzqucDcMMfzw3xsVW77Z, address:104.36.82.46:4496): protocol: read tcp 159.223.143.39:52800->104.36.82.46:4496: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 04:08:17.678144409 (6h16m33.459077789s) [noise=true , address=104.36.82.44:4901 , piece_id=N4YLCLR2QZSULUL2XWONEKBFUMNOU7KUX2QEOK4SSDFSCI7IL6LA, seg_pos={0 0}, upload=5911]: Done uploading piece. err: ecclient: upload failed (node:1FHE84zvBb1MKwB1Ht7puTafgQxDEvPt4mYqkYVSSufAMpWQA1, address:104.36.82.44:4901): protocol: rpc: dial tcp 104.36.82.44:4901: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 04:08:17.678229622 (6h16m33.459162988s) [noise=true , address=104.36.82.44:4893 , piece_id=HUJICRJ6W54O3AZZR6H3KG6E4TGBTNS46O6XVMQM3PFAZE7A2HZA, seg_pos={0 0}, upload=5911]: Done uploading piece. err: ecclient: upload failed (node:17VMFZ2T8jrSJVtchciJ9SnyrR6JstvcMoXrN5Q3zTD1aQU1oJ, address:104.36.82.44:4893): protocol: write tcp 159.223.143.39:35302->104.36.82.44:4893: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 04:08:17.678310218 (6h16m33.459243583s) [noise=true , address=104.36.82.42:4829 , piece_id=ZLWJFP3LGX3YLXBJYIR66DXTYEVDBKXKVXCQSF2PCYCYJBIEJGMA, seg_pos={0 0}, upload=5911]: Done uploading piece. err: ecclient: upload failed (node:1F5T8Vt1x4nXFaSenvGBCYjxhRL1H1sxRvzLjW3CqzwDVhq39F, address:104.36.82.42:4829): protocol: rpc: dial tcp 104.36.82.42:4829: i/o timeout; conn closed: op; piecestore: piecestore close: context deadline exceeded Jul 27 04:08:17.678412989 (6h16m33.459346365s) [noise=true , address=104.36.82.38:4722 , piece_id=B2MBQY5CTMEI6WSE7FTLOV5RUJ22SCNG7N2EEJILEATGJ4TIALXA, seg_pos={0 0}, upload=5911]: Done uploading piece. err: ecclient: upload failed (node:12Zw3dtUg9gARz52KhwDLTjcqC3CNm7dpPzpdx7P7PbL1jwe8uc, address:104.36.82.38:4722): protocol: read tcp 159.223.143.39:56048->104.36.82.38:4722: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 04:08:17.678455247 (6h16m33.459388624s) [noise=true , address=104.36.82.47:4524 , piece_id=DONJFTKUQ2XCW6BOU7AVOKJ7XMDMBPGQVUPRANH7X2YS7PJXUKIA, seg_pos={0 0}, upload=5911]: Done uploading piece. err: ecclient: upload failed (node:1xfbCzjSwFDRrR1gPggeN8tNCLbRRWDVzCJ6mqhZsbVSe8d4s4, address:104.36.82.47:4524): protocol: read tcp 159.223.143.39:46694->104.36.82.47:4524: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 04:08:17.678510668 (6h16m33.459444052s) [seg_pos={0 0}, upload=5911]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Jul 27 04:08:17.706442864 (6h16m33.487376232s) [upload=5911]: Done uploading segments... Jul 27 04:08:17.706495329 (6h16m33.487428705s) [upload=5911]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 1.474996046s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................................X...................] duration: 652.924484ms FAIL (2025-07-27 04:19:13 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== ece_id=NYOFDEZ7CV4PHGQ6JZMEEVAYGJRPYG6IBW7MYJO42HMWOJM7PHZA, seg_pos={0 0}, upload=5879]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12crcXpNe9NgeXTsUpooLSBgrSUFbRxbSbanLa39w2Hr5mGFww): protocol: read tcp 159.223.143.39:53630->104.36.82.5:4840: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 04:05:32.508904485 (6h13m48.289837866s) [seg_pos={0 0}, upload=5879]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 27 04:05:32.509028070 (6h13m48.289961457s) [upload=5879]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 27 04:05:32.509034645 (6h13m48.289968129s) [upload=5879]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 27 04:05:32.581834839 (6h13m48.362768225s) [upload=5879]: Done uploading segments... Jul 27 04:05:32.581875919 (6h13m48.362809298s) [upload=5879]: Upload finished. err: <nil> Jul 27 04:06:23.829155331 (6h14m39.610088696s) [upload=5895]: Starting upload Jul 27 04:06:23.829225021 (6h14m39.610158398s) [upload=5895]: Done starting upload Jul 27 04:06:23.829294606 (6h14m39.61022799s) [upload=5895]: Uploading segments... Jul 27 04:06:23.829577656 (6h14m39.610511053s) [upload=5895]: Got next segment. Inline: false Jul 27 04:06:23.829649664 (6h14m39.610583027s) [upload=5895]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 27 04:06:23.829657065 (6h14m39.61059043s) [upload=5895]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 27 04:06:23.861405530 (6h14m39.642338913s) [seg_pos={0 0}, upload=5895]: Begin upload segment... Jul 27 04:06:23.861476351 (6h14m39.642409713s) [seg_pos={0 0}, upload=5895]: Done begin upload segment. Jul 27 04:06:23.861509120 (6h14m39.642442506s) [noise=true , address=77.90.31.19:4843 , piece_id=YML5RDQIK432Z4FSE4DKXIV5U47LQBEZRFMAVLANUSGFUQR23UWQ, seg_pos={0 0}, upload=5895]: Uploading piece... Jul 27 04:06:23.861953723 (6h14m39.6428871s) [noise=true , address=45.91.116.10:4081 , piece_id=XGDTSEZOUNRTEVWHTO3DHTZMH2VZZOPJPEKLCFAHTXESLLYABHSA, seg_pos={0 0}, upload=5895]: Uploading piece... Jul 27 04:06:23.862060983 (6h14m39.642994358s) [noise=true , address=45.91.116.9:4262 , piece_id=M5ETD45HH2XZD6HDLTJ3JO2BOOIROXFWKVPTN42PSQEWG4F7KAAQ, seg_pos={0 0}, upload=5895]: Uploading piece... Jul 27 04:06:23.862759464 (6h14m39.64369284s) [noise=true , address=104.36.82.4:4804 , piece_id=IU6FDNVB5AUPA7MJ6RSXUTLBETPHOFGNJ7YGWBFIZSQFSO6W6BBA, seg_pos={0 0}, upload=5895]: Uploading piece... Jul 27 04:06:23.862835086 (6h14m39.643768461s) [noise=true , address=104.36.82.6:4873 , piece_id=YP4VJORZZUL5EVCH3P7LD3LNVMHB7CWRTKHGIA7UROXE44N6MMZA, seg_pos={0 0}, upload=5895]: Uploading piece... Jul 27 04:06:23.863593880 (6h14m39.644527258s) [noise=true , address=77.90.31.15:4697 , piece_id=Q76TJDSZPSYZSJQNFVOI462U6DFILMH2RV3LQISODXSOCXGRKW6Q, seg_pos={0 0}, upload=5895]: Uploading piece... Jul 27 04:06:23.863656891 (6h14m39.644590266s) [noise=true , address=45.91.116.7:4011 , piece_id=XMH6MCCHZBBRQWHJFWLRGYHLQD4KUGSGL7MMZSKA5TZJCND3DUDA, seg_pos={0 0}, upload=5895]: Uploading piece... Jul 27 04:06:23.864408612 (6h14m39.645341987s) [noise=true , address=45.91.116.6:4973 , piece_id=J3VULPU24CIM64VEVE3UHO62HPN6TJXGZU6BCJIS5JQJKHJK2LIQ, seg_pos={0 0}, upload=5895]: Uploading piece... Jul 27 04:06:23.864521549 (6h14m39.64545491s) [noise=true , address=104.36.82.10:4079 , piece_id=2SH5D6JOMLFR7VSN7ZY5CLMR5RHIRFYQ7Z5I2JIYTJQM43G4O6OQ, seg_pos={0 0}, upload=5895]: Uploading piece... Jul 27 04:06:23.865518491 (6h14m39.646451872s) [noise=true , address=77.90.31.17:4771 , piece_id=U2LF22GKCMKRZN6EWCF26ITDZQHBFZXCGSQXT33MCX6P57IGI35Q, seg_pos={0 0}, upload=5895]: Uploading piece... Jul 27 04:06:23.865654950 (6h14m39.646588325s) [noise=true , address=45.91.116.8:4048 , piece_id=5REWMVVTBVLDQXQ5KAEHLY4VJL5SGF7ONQWDQQ3N2FDU3OAO6AZA, seg_pos={0 0}, upload=5895]: Uploading piece... Jul 27 04:06:23.866190892 (6h14m39.647124271s) [noise=true , address=104.36.82.12:4151 , piece_id=E2TEUTQBD4Y4AUHSCPMCIFEXBSY7S5HJWBKNKWWTUWEW2BLAAE6Q, seg_pos={0 0}, upload=5895]: Uploading piece... Jul 27 04:06:23.866412590 (6h14m39.647345968s) [noise=true , address=104.36.82.11:4115 , piece_id=YXMNVT6ZHGESGQ2BXKZUVR7U2CNNAI2PO7ZTMDG5UIOV7HTDDUTA, seg_pos={0 0}, upload=5895]: Uploading piece... Jul 27 04:06:23.866715972 (6h14m39.647649359s) [noise=true , address=45.91.116.13:4153 , piece_id=H4535KOCG3F4RKA6AKKMTJSOM6Q6PGRC6M7FR3UIS3BORTSP5UQA, seg_pos={0 0}, upload=5895]: Uploading piece... Jul 27 04:06:23.866998965 (6h14m39.64793234s) [noise=true , address=104.36.82.13:4186 , piece_id=TBLDKALGJYBGR5NLCOCKJ4QKEFNIJXCKAFUBD7VR3UMNWEMKMKHQ, seg_pos={0 0}, upload=5895]: Uploading piece... Jul 27 04:06:23.867230593 (6h14m39.648163974s) [noise=true , address=104.36.82.16:4293 , piece_id=AHH6PRFUSGTDGB7H52DRITARA4IF6GFLVSRYS2Y6EPJCZSKUOYWQ, seg_pos={0 0}, upload=5895]: Uploading piece... Jul 27 04:06:23.949422808 (6h14m39.730356187s) [upload=5895]: Next returned nil segment Jul 27 04:06:23.949442576 (6h14m39.73037596s) [upload=5895]: Waiting for error group managing segments... Jul 27 04:06:23.953497426 (6h14m39.734430802s) [noise=true , address=45.91.116.10:4081 , piece_id=XGDTSEZOUNRTEVWHTO3DHTZMH2VZZOPJPEKLCFAHTXESLLYABHSA, seg_pos={0 0}, upload=5895]: Done uploading piece. err: <nil> Jul 27 04:06:23.953531796 (6h14m39.734465159s) [noise=true , address=45.91.116.6:4973 , piece_id=J3VULPU24CIM64VEVE3UHO62HPN6TJXGZU6BCJIS5JQJKHJK2LIQ, seg_pos={0 0}, upload=5895]: Done uploading piece. err: <nil> Jul 27 04:06:23.954068660 (6h14m39.735002029s) [noise=true , address=45.91.116.9:4262 , piece_id=M5ETD45HH2XZD6HDLTJ3JO2BOOIROXFWKVPTN42PSQEWG4F7KAAQ, seg_pos={0 0}, upload=5895]: Done uploading piece. err: <nil> Jul 27 04:06:23.954194339 (6h14m39.735127707s) [noise=true , address=45.91.116.8:4048 , piece_id=5REWMVVTBVLDQXQ5KAEHLY4VJL5SGF7ONQWDQQ3N2FDU3OAO6AZA, seg_pos={0 0}, upload=5895]: Done uploading piece. err: <nil> Jul 27 04:06:23.954601730 (6h14m39.735535112s) [noise=true , address=45.91.116.7:4011 , piece_id=XMH6MCCHZBBRQWHJFWLRGYHLQD4KUGSGL7MMZSKA5TZJCND3DUDA, seg_pos={0 0}, upload=5895]: Done uploading piece. err: <nil> Jul 27 04:06:23.954755872 (6h14m39.735689237s) [noise=true , address=45.91.116.13:4153 , piece_id=H4535KOCG3F4RKA6AKKMTJSOM6Q6PGRC6M7FR3UIS3BORTSP5UQA, seg_pos={0 0}, upload=5895]: Done uploading piece. err: <nil> Jul 27 04:06:23.955611883 (6h14m39.73654526s) [noise=true , address=77.90.31.19:4843 , piece_id=YML5RDQIK432Z4FSE4DKXIV5U47LQBEZRFMAVLANUSGFUQR23UWQ, seg_pos={0 0}, upload=5895]: Done uploading piece. err: <nil> Jul 27 04:06:23.956373391 (6h14m39.737306756s) [noise=true , address=77.90.31.15:4697 , piece_id=Q76TJDSZPSYZSJQNFVOI462U6DFILMH2RV3LQISODXSOCXGRKW6Q, seg_pos={0 0}, upload=5895]: Done uploading piece. err: <nil> Jul 27 04:06:23.958659122 (6h14m39.73959249s) [noise=true , address=77.90.31.17:4771 , piece_id=U2LF22GKCMKRZN6EWCF26ITDZQHBFZXCGSQXT33MCX6P57IGI35Q, seg_pos={0 0}, upload=5895]: Done uploading piece. err: <nil> Jul 27 04:06:35.058944013 (6h14m50.83987739s) [noise=true , address=104.36.82.16:4293 , piece_id=AHH6PRFUSGTDGB7H52DRITARA4IF6GFLVSRYS2Y6EPJCZSKUOYWQ, seg_pos={0 0}, upload=5895]: Done uploading piece. err: <nil> Jul 27 04:06:41.103204002 (6h14m56.884137386s) [noise=true , address=104.36.82.4:4804 , piece_id=IU6FDNVB5AUPA7MJ6RSXUTLBETPHOFGNJ7YGWBFIZSQFSO6W6BBA, seg_pos={0 0}, upload=5895]: Done uploading piece. err: <nil> Jul 27 04:06:43.191702285 (6h14m58.972635651s) [noise=true , address=104.36.82.6:4873 , piece_id=YP4VJORZZUL5EVCH3P7LD3LNVMHB7CWRTKHGIA7UROXE44N6MMZA, seg_pos={0 0}, upload=5895]: Done uploading piece. err: <nil> Jul 27 04:06:53.777973493 (6h15m9.558906861s) [noise=true , address=104.36.82.11:4115 , piece_id=YXMNVT6ZHGESGQ2BXKZUVR7U2CNNAI2PO7ZTMDG5UIOV7HTDDUTA, seg_pos={0 0}, upload=5895]: Done uploading piece. err: ecclient: upload failed (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR, address:104.36.82.11:4115): protocol: write tcp 159.223.143.39:34210->104.36.82.11:4115: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 04:06:53.778019532 (6h15m9.558952909s) [noise=true , address=104.36.82.12:4151 , piece_id=E2TEUTQBD4Y4AUHSCPMCIFEXBSY7S5HJWBKNKWWTUWEW2BLAAE6Q, seg_pos={0 0}, upload=5895]: Done uploading piece. err: ecclient: upload failed (node:1w6pwZMdeQKjt92qx6WbTWU5VPYJto5gS87f6iKvCBZH6LfyeB, address:104.36.82.12:4151): protocol: rpc: dial tcp 104.36.82.12:4151: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 04:06:53.778043653 (6h15m9.558977021s) [noise=true , address=104.36.82.13:4186 , piece_id=TBLDKALGJYBGR5NLCOCKJ4QKEFNIJXCKAFUBD7VR3UMNWEMKMKHQ, seg_pos={0 0}, upload=5895]: Done uploading piece. err: ecclient: upload failed (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht, address:104.36.82.13:4186): protocol: read tcp 159.223.143.39:34340->104.36.82.13:4186: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 04:06:53.778095228 (6h15m9.559028593s) [noise=true , address=104.36.82.10:4079 , piece_id=2SH5D6JOMLFR7VSN7ZY5CLMR5RHIRFYQ7Z5I2JIYTJQM43G4O6OQ, seg_pos={0 0}, upload=5895]: Done uploading piece. err: ecclient: upload failed (node:12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud, address:104.36.82.10:4079): protocol: write tcp 159.223.143.39:53840->104.36.82.10:4079: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 04:06:53.778142167 (6h15m9.559075532s) [seg_pos={0 0}, upload=5895]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 27 04:06:53.837032161 (6h15m9.617965543s) [upload=5895]: Done uploading segments... Jul 27 04:06:53.837055996 (6h15m9.617989366s) [upload=5895]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 652.963384ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................................................X....] duration: 730.75375ms FAIL (2025-07-27 04:19:13 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 12) Duration: 730.788897ms core uplink us1 vivint

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)