Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [...................................X........................] duration: 16.774098176s FAIL (2025-06-22 01:50:54 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== oad=64949]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jun 22 01:32:13.674692303 (81h8m49.238015311s) [upload=64949]: Done uploading segments... Jun 22 01:32:13.674727219 (81h8m49.238050233s) [upload=64949]: Upload finished. err: <nil> Jun 22 01:32:47.266407995 (81h9m22.829731002s) [upload=64958]: Starting upload Jun 22 01:32:47.266524767 (81h9m22.829847771s) [upload=64958]: Done starting upload Jun 22 01:32:47.266813109 (81h9m22.830136116s) [upload=64958]: Uploading segments... Jun 22 01:32:47.266969526 (81h9m22.830292526s) [upload=64958]: Got next segment. Inline: false Jun 22 01:32:47.267075475 (81h9m22.830398475s) [upload=64958]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 22 01:32:47.267081679 (81h9m22.830404667s) [upload=64958]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 22 01:32:47.301770382 (81h9m22.865093389s) [seg_pos={0 0}, upload=64958]: Begin upload segment... Jun 22 01:32:47.301823796 (81h9m22.865146799s) [seg_pos={0 0}, upload=64958]: Done begin upload segment. Jun 22 01:32:47.301855055 (81h9m22.865178087s) [noise=true , address=104.36.82.43:6841 , piece_id=PIHZMSTO3CII7KGDLGSAILTDRSLMEBVOLBCRCZJULMCFISXIZWXA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.301902600 (81h9m22.865225606s) [noise=true , address=104.36.82.42:6811 , piece_id=A5BFEHZ3WGGK53IZ7MKI6N4CVXHOORF6L47NR3UFDCAX7LBHRHVQ, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.302234027 (81h9m22.865557029s) [noise=true , address=104.36.82.51:4084 , piece_id=D43FA7BFE54KPCWGDGDME2EGSWM7PONOVPD3DN2XY42MJFOKVDIQ, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.302496354 (81h9m22.865819363s) [noise=true , address=79.127.219.38:4023 , piece_id=BWLRCXXHBMBTNCZEVKKIIS5PB35PDQXQXCLIQB7GBLEIOWWUFBFA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.302577370 (81h9m22.865900378s) [noise=true , address=104.36.82.52:4089 , piece_id=JSOEKLNWA3W3LOVKI3EGGKJYJ6J3UZY6KGABKQPQPXTIOOPMKVHA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.302917829 (81h9m22.866240835s) [noise=true , address=104.36.82.53:4122 , piece_id=HZZQBIKHQWLUE5EHK4KSKJGIUEG7D44RQASUJHCYXX6BICCGX6EA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.303087165 (81h9m22.866410168s) [noise=true , address=77.90.31.14:4689 , piece_id=Q3HDS3BCMYDTODFRMU4XFRDCQK2QZE3ZMXVLUX4M54PDEBS7RR6A, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.303250714 (81h9m22.866573717s) [noise=true , address=79.127.205.234:5601 , piece_id=V7PMXBP2VFWBSZDG4GTLBD435T6F5DT7A43L4GQLQ5BDDTJ4WY6A, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.303579249 (81h9m22.866902255s) [noise=true , address=104.36.82.12:6158 , piece_id=LJZEOJVJCJAR2BRILIJI3ZVDGYMPBBTP2G3HCTN3MHNS4KYDPASQ, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.303599776 (81h9m22.866922784s) [noise=true , address=104.36.82.9:6023 , piece_id=SISFAUU4RPF57OLGWM2BWQWE6GRUSCMORBYHWEXETNAVM2EEMO6Q, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.304005346 (81h9m22.867328346s) [noise=true , address=45.91.116.12:5242 , piece_id=6ABT5IPYGSLFFJMBYMBYRU4EYIXABUAUHG3OVILOHBIUDS3BP27A, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.304388850 (81h9m22.867711851s) [noise=true , address=64.49.13.137:4500 , piece_id=FAE2RXD57IRP5FXA2XR6THXFW27U7IWAZ4JDV5M2WRPXKUGZ4VIA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.304881529 (81h9m22.868204536s) [noise=true , address=79.127.220.101:6048 , piece_id=AN7UQUXB42T2YXEH7ZJRZUDIW47M3RWUEOCL2VEBOGH74BZF3YKA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.305249860 (81h9m22.86857287s) [noise=true , address=104.36.82.54:4165 , piece_id=4CVOABPCWC7J7SRHZB6IN2WJJ6KGWUQ66ELUIZIFA67ETLQ7JEUQ, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.305604108 (81h9m22.868927112s) [noise=true , address=104.36.82.15:6289 , piece_id=GWL3P55BIK4IJEL5YFZKI7JCA4LADYOOHUSIQUNGJEKBJOTKMP5A, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.305930423 (81h9m22.869253425s) [noise=true , address=64.49.13.139:4476 , piece_id=P7UKBNNZGI52B3I6WKTV6MVC262FYNKYYPJUI5G47UO5S4MU7PWA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.306355311 (81h9m22.869678319s) [noise=true , address=104.36.82.55:4213 , piece_id=X4IHS2XSPH52LIMNUYT7KGWMAPUH3TEMJOEEVB3JV5N45SEOH7YA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.306788243 (81h9m22.870111252s) [noise=true , address=104.36.82.22:6539 , piece_id=GHQ2MVSTDGVYLUKLV23D6YWRR4PP7UUYHCUEQ4WWSWEMTBFO3P4A, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.392174770 (81h9m22.955497773s) [upload=64958]: Next returned nil segment Jun 22 01:32:47.392195767 (81h9m22.955518775s) [upload=64958]: Waiting for error group managing segments... Jun 22 01:32:47.480767633 (81h9m23.044090623s) [noise=true , address=104.36.82.22:6539 , piece_id=GHQ2MVSTDGVYLUKLV23D6YWRR4PP7UUYHCUEQ4WWSWEMTBFO3P4A, seg_pos={0 0}, upload=64958]: Done uploading piece. err: ecclient: upload failed (node:1QL6w8DSFR6CxDdnJmDuyVZxp6JU55VKiJofBZRfQi7q31i8Jc, address:104.36.82.22:6539): protocol: storage node overloaded, request limit: 2000 Jun 22 01:32:47.585663142 (81h9m23.148986155s) [noise=true , address=79.127.220.101:6048 , piece_id=AN7UQUXB42T2YXEH7ZJRZUDIW47M3RWUEOCL2VEBOGH74BZF3YKA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:47.640263002 (81h9m23.203586013s) [noise=true , address=45.91.116.12:5242 , piece_id=6ABT5IPYGSLFFJMBYMBYRU4EYIXABUAUHG3OVILOHBIUDS3BP27A, seg_pos={0 0}, upload=64958]: Done uploading piece. err: ecclient: upload failed (node:12JsFL62jgXmza8Md8eQ9k8cyw5V9Vz2RzN4y85ZvgHZjwrq4vQ, address:45.91.116.12:5242): protocol: storage node overloaded, request limit: 2000 Jun 22 01:32:47.984127275 (81h9m23.54745028s) [noise=true , address=64.49.13.137:4500 , piece_id=FAE2RXD57IRP5FXA2XR6THXFW27U7IWAZ4JDV5M2WRPXKUGZ4VIA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:48.100255249 (81h9m23.663578241s) [noise=true , address=77.90.31.14:4689 , piece_id=Q3HDS3BCMYDTODFRMU4XFRDCQK2QZE3ZMXVLUX4M54PDEBS7RR6A, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:48.221764035 (81h9m23.785087034s) [noise=true , address=104.36.82.54:4165 , piece_id=4CVOABPCWC7J7SRHZB6IN2WJJ6KGWUQ66ELUIZIFA67ETLQ7JEUQ, seg_pos={0 0}, upload=64958]: Done uploading piece. err: ecclient: upload failed (node:12wGUMXVEubEtki4qGnjGEsbYFB1iUauZcu3oDRKrCQsJMV3bV2, address:104.36.82.54:4165): protocol: storage node overloaded, request limit: 2000 Jun 22 01:32:48.509996527 (81h9m24.073319547s) [noise=true , address=104.36.82.9:6023 , piece_id=SISFAUU4RPF57OLGWM2BWQWE6GRUSCMORBYHWEXETNAVM2EEMO6Q, seg_pos={0 0}, upload=64958]: Done uploading piece. err: ecclient: upload failed (node:1j2S9tgWpLwmyvXZSq2qKKcbaC8r7LkzfDCaMxCejAni4CST1g, address:104.36.82.9:6023): protocol: storage node overloaded, request limit: 2000 Jun 22 01:32:48.510513169 (81h9m24.073836177s) [noise=true , address=104.36.82.42:6811 , piece_id=A5BFEHZ3WGGK53IZ7MKI6N4CVXHOORF6L47NR3UFDCAX7LBHRHVQ, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:48.565245605 (81h9m24.128568606s) [noise=true , address=79.127.205.234:5601 , piece_id=V7PMXBP2VFWBSZDG4GTLBD435T6F5DT7A43L4GQLQ5BDDTJ4WY6A, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:49.979979824 (81h9m25.543302825s) [noise=true , address=104.36.82.15:6289 , piece_id=GWL3P55BIK4IJEL5YFZKI7JCA4LADYOOHUSIQUNGJEKBJOTKMP5A, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:50.463750280 (81h9m26.02707328s) [noise=true , address=104.36.82.51:4084 , piece_id=D43FA7BFE54KPCWGDGDME2EGSWM7PONOVPD3DN2XY42MJFOKVDIQ, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:51.480599795 (81h9m27.043922798s) [noise=true , address=79.127.219.38:4023 , piece_id=BWLRCXXHBMBTNCZEVKKIIS5PB35PDQXQXCLIQB7GBLEIOWWUFBFA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:52.941200625 (81h9m28.504523627s) [noise=true , address=104.36.82.43:6841 , piece_id=PIHZMSTO3CII7KGDLGSAILTDRSLMEBVOLBCRCZJULMCFISXIZWXA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:58.281803598 (81h9m33.845126612s) [noise=true , address=64.49.13.139:4476 , piece_id=P7UKBNNZGI52B3I6WKTV6MVC262FYNKYYPJUI5G47UO5S4MU7PWA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:33:00.277271877 (81h9m35.840594867s) [noise=true , address=104.36.82.55:4213 , piece_id=X4IHS2XSPH52LIMNUYT7KGWMAPUH3TEMJOEEVB3JV5N45SEOH7YA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:33:09.435339213 (81h9m44.998662206s) [noise=true , address=104.36.82.52:4089 , piece_id=JSOEKLNWA3W3LOVKI3EGGKJYJ6J3UZY6KGABKQPQPXTIOOPMKVHA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:33:31.472868445 (81h10m7.03619143s) [noise=true , address=104.36.82.53:4122 , piece_id=HZZQBIKHQWLUE5EHK4KSKJGIUEG7D44RQASUJHCYXX6BICCGX6EA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:33:47.235404653 (81h10m22.798727642s) [noise=true , address=104.36.82.12:6158 , piece_id=LJZEOJVJCJAR2BRILIJI3ZVDGYMPBBTP2G3HCTN3MHNS4KYDPASQ, seg_pos={0 0}, upload=64958]: Done uploading piece. err: ecclient: upload failed (node:1SjM6Utm2YizeUbrBUD9Z7etE1nLadW5Fa1zYP7MioPeeqhdJ7, address:104.36.82.12:6158): protocol: expected piece hash; context deadline exceeded; EOF Jun 22 01:33:47.235448297 (81h10m22.798771306s) [seg_pos={0 0}, upload=64958]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jun 22 01:33:47.265442393 (81h10m22.828765401s) [upload=64958]: Done uploading segments... Jun 22 01:33:47.265476139 (81h10m22.82879914s) [upload=64958]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration:* 16.774211366s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [....................................X.......................] duration: 14.85489583s FAIL (2025-06-22 01:49:53 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== oad=64949]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jun 22 01:32:13.674692303 (81h8m49.238015311s) [upload=64949]: Done uploading segments... Jun 22 01:32:13.674727219 (81h8m49.238050233s) [upload=64949]: Upload finished. err: <nil> Jun 22 01:32:47.266407995 (81h9m22.829731002s) [upload=64958]: Starting upload Jun 22 01:32:47.266524767 (81h9m22.829847771s) [upload=64958]: Done starting upload Jun 22 01:32:47.266813109 (81h9m22.830136116s) [upload=64958]: Uploading segments... Jun 22 01:32:47.266969526 (81h9m22.830292526s) [upload=64958]: Got next segment. Inline: false Jun 22 01:32:47.267075475 (81h9m22.830398475s) [upload=64958]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 22 01:32:47.267081679 (81h9m22.830404667s) [upload=64958]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 22 01:32:47.301770382 (81h9m22.865093389s) [seg_pos={0 0}, upload=64958]: Begin upload segment... Jun 22 01:32:47.301823796 (81h9m22.865146799s) [seg_pos={0 0}, upload=64958]: Done begin upload segment. Jun 22 01:32:47.301855055 (81h9m22.865178087s) [noise=true , address=104.36.82.43:6841 , piece_id=PIHZMSTO3CII7KGDLGSAILTDRSLMEBVOLBCRCZJULMCFISXIZWXA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.301902600 (81h9m22.865225606s) [noise=true , address=104.36.82.42:6811 , piece_id=A5BFEHZ3WGGK53IZ7MKI6N4CVXHOORF6L47NR3UFDCAX7LBHRHVQ, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.302234027 (81h9m22.865557029s) [noise=true , address=104.36.82.51:4084 , piece_id=D43FA7BFE54KPCWGDGDME2EGSWM7PONOVPD3DN2XY42MJFOKVDIQ, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.302496354 (81h9m22.865819363s) [noise=true , address=79.127.219.38:4023 , piece_id=BWLRCXXHBMBTNCZEVKKIIS5PB35PDQXQXCLIQB7GBLEIOWWUFBFA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.302577370 (81h9m22.865900378s) [noise=true , address=104.36.82.52:4089 , piece_id=JSOEKLNWA3W3LOVKI3EGGKJYJ6J3UZY6KGABKQPQPXTIOOPMKVHA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.302917829 (81h9m22.866240835s) [noise=true , address=104.36.82.53:4122 , piece_id=HZZQBIKHQWLUE5EHK4KSKJGIUEG7D44RQASUJHCYXX6BICCGX6EA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.303087165 (81h9m22.866410168s) [noise=true , address=77.90.31.14:4689 , piece_id=Q3HDS3BCMYDTODFRMU4XFRDCQK2QZE3ZMXVLUX4M54PDEBS7RR6A, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.303250714 (81h9m22.866573717s) [noise=true , address=79.127.205.234:5601 , piece_id=V7PMXBP2VFWBSZDG4GTLBD435T6F5DT7A43L4GQLQ5BDDTJ4WY6A, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.303579249 (81h9m22.866902255s) [noise=true , address=104.36.82.12:6158 , piece_id=LJZEOJVJCJAR2BRILIJI3ZVDGYMPBBTP2G3HCTN3MHNS4KYDPASQ, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.303599776 (81h9m22.866922784s) [noise=true , address=104.36.82.9:6023 , piece_id=SISFAUU4RPF57OLGWM2BWQWE6GRUSCMORBYHWEXETNAVM2EEMO6Q, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.304005346 (81h9m22.867328346s) [noise=true , address=45.91.116.12:5242 , piece_id=6ABT5IPYGSLFFJMBYMBYRU4EYIXABUAUHG3OVILOHBIUDS3BP27A, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.304388850 (81h9m22.867711851s) [noise=true , address=64.49.13.137:4500 , piece_id=FAE2RXD57IRP5FXA2XR6THXFW27U7IWAZ4JDV5M2WRPXKUGZ4VIA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.304881529 (81h9m22.868204536s) [noise=true , address=79.127.220.101:6048 , piece_id=AN7UQUXB42T2YXEH7ZJRZUDIW47M3RWUEOCL2VEBOGH74BZF3YKA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.305249860 (81h9m22.86857287s) [noise=true , address=104.36.82.54:4165 , piece_id=4CVOABPCWC7J7SRHZB6IN2WJJ6KGWUQ66ELUIZIFA67ETLQ7JEUQ, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.305604108 (81h9m22.868927112s) [noise=true , address=104.36.82.15:6289 , piece_id=GWL3P55BIK4IJEL5YFZKI7JCA4LADYOOHUSIQUNGJEKBJOTKMP5A, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.305930423 (81h9m22.869253425s) [noise=true , address=64.49.13.139:4476 , piece_id=P7UKBNNZGI52B3I6WKTV6MVC262FYNKYYPJUI5G47UO5S4MU7PWA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.306355311 (81h9m22.869678319s) [noise=true , address=104.36.82.55:4213 , piece_id=X4IHS2XSPH52LIMNUYT7KGWMAPUH3TEMJOEEVB3JV5N45SEOH7YA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.306788243 (81h9m22.870111252s) [noise=true , address=104.36.82.22:6539 , piece_id=GHQ2MVSTDGVYLUKLV23D6YWRR4PP7UUYHCUEQ4WWSWEMTBFO3P4A, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.392174770 (81h9m22.955497773s) [upload=64958]: Next returned nil segment Jun 22 01:32:47.392195767 (81h9m22.955518775s) [upload=64958]: Waiting for error group managing segments... Jun 22 01:32:47.480767633 (81h9m23.044090623s) [noise=true , address=104.36.82.22:6539 , piece_id=GHQ2MVSTDGVYLUKLV23D6YWRR4PP7UUYHCUEQ4WWSWEMTBFO3P4A, seg_pos={0 0}, upload=64958]: Done uploading piece. err: ecclient: upload failed (node:1QL6w8DSFR6CxDdnJmDuyVZxp6JU55VKiJofBZRfQi7q31i8Jc, address:104.36.82.22:6539): protocol: storage node overloaded, request limit: 2000 Jun 22 01:32:47.585663142 (81h9m23.148986155s) [noise=true , address=79.127.220.101:6048 , piece_id=AN7UQUXB42T2YXEH7ZJRZUDIW47M3RWUEOCL2VEBOGH74BZF3YKA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:47.640263002 (81h9m23.203586013s) [noise=true , address=45.91.116.12:5242 , piece_id=6ABT5IPYGSLFFJMBYMBYRU4EYIXABUAUHG3OVILOHBIUDS3BP27A, seg_pos={0 0}, upload=64958]: Done uploading piece. err: ecclient: upload failed (node:12JsFL62jgXmza8Md8eQ9k8cyw5V9Vz2RzN4y85ZvgHZjwrq4vQ, address:45.91.116.12:5242): protocol: storage node overloaded, request limit: 2000 Jun 22 01:32:47.984127275 (81h9m23.54745028s) [noise=true , address=64.49.13.137:4500 , piece_id=FAE2RXD57IRP5FXA2XR6THXFW27U7IWAZ4JDV5M2WRPXKUGZ4VIA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:48.100255249 (81h9m23.663578241s) [noise=true , address=77.90.31.14:4689 , piece_id=Q3HDS3BCMYDTODFRMU4XFRDCQK2QZE3ZMXVLUX4M54PDEBS7RR6A, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:48.221764035 (81h9m23.785087034s) [noise=true , address=104.36.82.54:4165 , piece_id=4CVOABPCWC7J7SRHZB6IN2WJJ6KGWUQ66ELUIZIFA67ETLQ7JEUQ, seg_pos={0 0}, upload=64958]: Done uploading piece. err: ecclient: upload failed (node:12wGUMXVEubEtki4qGnjGEsbYFB1iUauZcu3oDRKrCQsJMV3bV2, address:104.36.82.54:4165): protocol: storage node overloaded, request limit: 2000 Jun 22 01:32:48.509996527 (81h9m24.073319547s) [noise=true , address=104.36.82.9:6023 , piece_id=SISFAUU4RPF57OLGWM2BWQWE6GRUSCMORBYHWEXETNAVM2EEMO6Q, seg_pos={0 0}, upload=64958]: Done uploading piece. err: ecclient: upload failed (node:1j2S9tgWpLwmyvXZSq2qKKcbaC8r7LkzfDCaMxCejAni4CST1g, address:104.36.82.9:6023): protocol: storage node overloaded, request limit: 2000 Jun 22 01:32:48.510513169 (81h9m24.073836177s) [noise=true , address=104.36.82.42:6811 , piece_id=A5BFEHZ3WGGK53IZ7MKI6N4CVXHOORF6L47NR3UFDCAX7LBHRHVQ, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:48.565245605 (81h9m24.128568606s) [noise=true , address=79.127.205.234:5601 , piece_id=V7PMXBP2VFWBSZDG4GTLBD435T6F5DT7A43L4GQLQ5BDDTJ4WY6A, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:49.979979824 (81h9m25.543302825s) [noise=true , address=104.36.82.15:6289 , piece_id=GWL3P55BIK4IJEL5YFZKI7JCA4LADYOOHUSIQUNGJEKBJOTKMP5A, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:50.463750280 (81h9m26.02707328s) [noise=true , address=104.36.82.51:4084 , piece_id=D43FA7BFE54KPCWGDGDME2EGSWM7PONOVPD3DN2XY42MJFOKVDIQ, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:51.480599795 (81h9m27.043922798s) [noise=true , address=79.127.219.38:4023 , piece_id=BWLRCXXHBMBTNCZEVKKIIS5PB35PDQXQXCLIQB7GBLEIOWWUFBFA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:52.941200625 (81h9m28.504523627s) [noise=true , address=104.36.82.43:6841 , piece_id=PIHZMSTO3CII7KGDLGSAILTDRSLMEBVOLBCRCZJULMCFISXIZWXA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:58.281803598 (81h9m33.845126612s) [noise=true , address=64.49.13.139:4476 , piece_id=P7UKBNNZGI52B3I6WKTV6MVC262FYNKYYPJUI5G47UO5S4MU7PWA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:33:00.277271877 (81h9m35.840594867s) [noise=true , address=104.36.82.55:4213 , piece_id=X4IHS2XSPH52LIMNUYT7KGWMAPUH3TEMJOEEVB3JV5N45SEOH7YA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:33:09.435339213 (81h9m44.998662206s) [noise=true , address=104.36.82.52:4089 , piece_id=JSOEKLNWA3W3LOVKI3EGGKJYJ6J3UZY6KGABKQPQPXTIOOPMKVHA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:33:31.472868445 (81h10m7.03619143s) [noise=true , address=104.36.82.53:4122 , piece_id=HZZQBIKHQWLUE5EHK4KSKJGIUEG7D44RQASUJHCYXX6BICCGX6EA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:33:47.235404653 (81h10m22.798727642s) [noise=true , address=104.36.82.12:6158 , piece_id=LJZEOJVJCJAR2BRILIJI3ZVDGYMPBBTP2G3HCTN3MHNS4KYDPASQ, seg_pos={0 0}, upload=64958]: Done uploading piece. err: ecclient: upload failed (node:1SjM6Utm2YizeUbrBUD9Z7etE1nLadW5Fa1zYP7MioPeeqhdJ7, address:104.36.82.12:6158): protocol: expected piece hash; context deadline exceeded; EOF Jun 22 01:33:47.235448297 (81h10m22.798771306s) [seg_pos={0 0}, upload=64958]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jun 22 01:33:47.265442393 (81h10m22.828765401s) [upload=64958]: Done uploading segments... Jun 22 01:33:47.265476139 (81h10m22.82879914s) [upload=64958]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration:* 14.854924503s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [....................................................X.......] duration: 15.838811539s FAIL (2025-06-22 01:49:48 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 15.838831514s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.....................................X......................] duration: 21.921295987s FAIL (2025-06-22 01:49:32 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== oad=64949]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jun 22 01:32:13.674692303 (81h8m49.238015311s) [upload=64949]: Done uploading segments... Jun 22 01:32:13.674727219 (81h8m49.238050233s) [upload=64949]: Upload finished. err: <nil> Jun 22 01:32:47.266407995 (81h9m22.829731002s) [upload=64958]: Starting upload Jun 22 01:32:47.266524767 (81h9m22.829847771s) [upload=64958]: Done starting upload Jun 22 01:32:47.266813109 (81h9m22.830136116s) [upload=64958]: Uploading segments... Jun 22 01:32:47.266969526 (81h9m22.830292526s) [upload=64958]: Got next segment. Inline: false Jun 22 01:32:47.267075475 (81h9m22.830398475s) [upload=64958]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 22 01:32:47.267081679 (81h9m22.830404667s) [upload=64958]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 22 01:32:47.301770382 (81h9m22.865093389s) [seg_pos={0 0}, upload=64958]: Begin upload segment... Jun 22 01:32:47.301823796 (81h9m22.865146799s) [seg_pos={0 0}, upload=64958]: Done begin upload segment. Jun 22 01:32:47.301855055 (81h9m22.865178087s) [noise=true , address=104.36.82.43:6841 , piece_id=PIHZMSTO3CII7KGDLGSAILTDRSLMEBVOLBCRCZJULMCFISXIZWXA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.301902600 (81h9m22.865225606s) [noise=true , address=104.36.82.42:6811 , piece_id=A5BFEHZ3WGGK53IZ7MKI6N4CVXHOORF6L47NR3UFDCAX7LBHRHVQ, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.302234027 (81h9m22.865557029s) [noise=true , address=104.36.82.51:4084 , piece_id=D43FA7BFE54KPCWGDGDME2EGSWM7PONOVPD3DN2XY42MJFOKVDIQ, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.302496354 (81h9m22.865819363s) [noise=true , address=79.127.219.38:4023 , piece_id=BWLRCXXHBMBTNCZEVKKIIS5PB35PDQXQXCLIQB7GBLEIOWWUFBFA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.302577370 (81h9m22.865900378s) [noise=true , address=104.36.82.52:4089 , piece_id=JSOEKLNWA3W3LOVKI3EGGKJYJ6J3UZY6KGABKQPQPXTIOOPMKVHA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.302917829 (81h9m22.866240835s) [noise=true , address=104.36.82.53:4122 , piece_id=HZZQBIKHQWLUE5EHK4KSKJGIUEG7D44RQASUJHCYXX6BICCGX6EA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.303087165 (81h9m22.866410168s) [noise=true , address=77.90.31.14:4689 , piece_id=Q3HDS3BCMYDTODFRMU4XFRDCQK2QZE3ZMXVLUX4M54PDEBS7RR6A, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.303250714 (81h9m22.866573717s) [noise=true , address=79.127.205.234:5601 , piece_id=V7PMXBP2VFWBSZDG4GTLBD435T6F5DT7A43L4GQLQ5BDDTJ4WY6A, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.303579249 (81h9m22.866902255s) [noise=true , address=104.36.82.12:6158 , piece_id=LJZEOJVJCJAR2BRILIJI3ZVDGYMPBBTP2G3HCTN3MHNS4KYDPASQ, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.303599776 (81h9m22.866922784s) [noise=true , address=104.36.82.9:6023 , piece_id=SISFAUU4RPF57OLGWM2BWQWE6GRUSCMORBYHWEXETNAVM2EEMO6Q, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.304005346 (81h9m22.867328346s) [noise=true , address=45.91.116.12:5242 , piece_id=6ABT5IPYGSLFFJMBYMBYRU4EYIXABUAUHG3OVILOHBIUDS3BP27A, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.304388850 (81h9m22.867711851s) [noise=true , address=64.49.13.137:4500 , piece_id=FAE2RXD57IRP5FXA2XR6THXFW27U7IWAZ4JDV5M2WRPXKUGZ4VIA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.304881529 (81h9m22.868204536s) [noise=true , address=79.127.220.101:6048 , piece_id=AN7UQUXB42T2YXEH7ZJRZUDIW47M3RWUEOCL2VEBOGH74BZF3YKA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.305249860 (81h9m22.86857287s) [noise=true , address=104.36.82.54:4165 , piece_id=4CVOABPCWC7J7SRHZB6IN2WJJ6KGWUQ66ELUIZIFA67ETLQ7JEUQ, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.305604108 (81h9m22.868927112s) [noise=true , address=104.36.82.15:6289 , piece_id=GWL3P55BIK4IJEL5YFZKI7JCA4LADYOOHUSIQUNGJEKBJOTKMP5A, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.305930423 (81h9m22.869253425s) [noise=true , address=64.49.13.139:4476 , piece_id=P7UKBNNZGI52B3I6WKTV6MVC262FYNKYYPJUI5G47UO5S4MU7PWA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.306355311 (81h9m22.869678319s) [noise=true , address=104.36.82.55:4213 , piece_id=X4IHS2XSPH52LIMNUYT7KGWMAPUH3TEMJOEEVB3JV5N45SEOH7YA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.306788243 (81h9m22.870111252s) [noise=true , address=104.36.82.22:6539 , piece_id=GHQ2MVSTDGVYLUKLV23D6YWRR4PP7UUYHCUEQ4WWSWEMTBFO3P4A, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.392174770 (81h9m22.955497773s) [upload=64958]: Next returned nil segment Jun 22 01:32:47.392195767 (81h9m22.955518775s) [upload=64958]: Waiting for error group managing segments... Jun 22 01:32:47.480767633 (81h9m23.044090623s) [noise=true , address=104.36.82.22:6539 , piece_id=GHQ2MVSTDGVYLUKLV23D6YWRR4PP7UUYHCUEQ4WWSWEMTBFO3P4A, seg_pos={0 0}, upload=64958]: Done uploading piece. err: ecclient: upload failed (node:1QL6w8DSFR6CxDdnJmDuyVZxp6JU55VKiJofBZRfQi7q31i8Jc, address:104.36.82.22:6539): protocol: storage node overloaded, request limit: 2000 Jun 22 01:32:47.585663142 (81h9m23.148986155s) [noise=true , address=79.127.220.101:6048 , piece_id=AN7UQUXB42T2YXEH7ZJRZUDIW47M3RWUEOCL2VEBOGH74BZF3YKA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:47.640263002 (81h9m23.203586013s) [noise=true , address=45.91.116.12:5242 , piece_id=6ABT5IPYGSLFFJMBYMBYRU4EYIXABUAUHG3OVILOHBIUDS3BP27A, seg_pos={0 0}, upload=64958]: Done uploading piece. err: ecclient: upload failed (node:12JsFL62jgXmza8Md8eQ9k8cyw5V9Vz2RzN4y85ZvgHZjwrq4vQ, address:45.91.116.12:5242): protocol: storage node overloaded, request limit: 2000 Jun 22 01:32:47.984127275 (81h9m23.54745028s) [noise=true , address=64.49.13.137:4500 , piece_id=FAE2RXD57IRP5FXA2XR6THXFW27U7IWAZ4JDV5M2WRPXKUGZ4VIA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:48.100255249 (81h9m23.663578241s) [noise=true , address=77.90.31.14:4689 , piece_id=Q3HDS3BCMYDTODFRMU4XFRDCQK2QZE3ZMXVLUX4M54PDEBS7RR6A, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:48.221764035 (81h9m23.785087034s) [noise=true , address=104.36.82.54:4165 , piece_id=4CVOABPCWC7J7SRHZB6IN2WJJ6KGWUQ66ELUIZIFA67ETLQ7JEUQ, seg_pos={0 0}, upload=64958]: Done uploading piece. err: ecclient: upload failed (node:12wGUMXVEubEtki4qGnjGEsbYFB1iUauZcu3oDRKrCQsJMV3bV2, address:104.36.82.54:4165): protocol: storage node overloaded, request limit: 2000 Jun 22 01:32:48.509996527 (81h9m24.073319547s) [noise=true , address=104.36.82.9:6023 , piece_id=SISFAUU4RPF57OLGWM2BWQWE6GRUSCMORBYHWEXETNAVM2EEMO6Q, seg_pos={0 0}, upload=64958]: Done uploading piece. err: ecclient: upload failed (node:1j2S9tgWpLwmyvXZSq2qKKcbaC8r7LkzfDCaMxCejAni4CST1g, address:104.36.82.9:6023): protocol: storage node overloaded, request limit: 2000 Jun 22 01:32:48.510513169 (81h9m24.073836177s) [noise=true , address=104.36.82.42:6811 , piece_id=A5BFEHZ3WGGK53IZ7MKI6N4CVXHOORF6L47NR3UFDCAX7LBHRHVQ, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:48.565245605 (81h9m24.128568606s) [noise=true , address=79.127.205.234:5601 , piece_id=V7PMXBP2VFWBSZDG4GTLBD435T6F5DT7A43L4GQLQ5BDDTJ4WY6A, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:49.979979824 (81h9m25.543302825s) [noise=true , address=104.36.82.15:6289 , piece_id=GWL3P55BIK4IJEL5YFZKI7JCA4LADYOOHUSIQUNGJEKBJOTKMP5A, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:50.463750280 (81h9m26.02707328s) [noise=true , address=104.36.82.51:4084 , piece_id=D43FA7BFE54KPCWGDGDME2EGSWM7PONOVPD3DN2XY42MJFOKVDIQ, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:51.480599795 (81h9m27.043922798s) [noise=true , address=79.127.219.38:4023 , piece_id=BWLRCXXHBMBTNCZEVKKIIS5PB35PDQXQXCLIQB7GBLEIOWWUFBFA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:52.941200625 (81h9m28.504523627s) [noise=true , address=104.36.82.43:6841 , piece_id=PIHZMSTO3CII7KGDLGSAILTDRSLMEBVOLBCRCZJULMCFISXIZWXA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:58.281803598 (81h9m33.845126612s) [noise=true , address=64.49.13.139:4476 , piece_id=P7UKBNNZGI52B3I6WKTV6MVC262FYNKYYPJUI5G47UO5S4MU7PWA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:33:00.277271877 (81h9m35.840594867s) [noise=true , address=104.36.82.55:4213 , piece_id=X4IHS2XSPH52LIMNUYT7KGWMAPUH3TEMJOEEVB3JV5N45SEOH7YA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:33:09.435339213 (81h9m44.998662206s) [noise=true , address=104.36.82.52:4089 , piece_id=JSOEKLNWA3W3LOVKI3EGGKJYJ6J3UZY6KGABKQPQPXTIOOPMKVHA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:33:31.472868445 (81h10m7.03619143s) [noise=true , address=104.36.82.53:4122 , piece_id=HZZQBIKHQWLUE5EHK4KSKJGIUEG7D44RQASUJHCYXX6BICCGX6EA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:33:47.235404653 (81h10m22.798727642s) [noise=true , address=104.36.82.12:6158 , piece_id=LJZEOJVJCJAR2BRILIJI3ZVDGYMPBBTP2G3HCTN3MHNS4KYDPASQ, seg_pos={0 0}, upload=64958]: Done uploading piece. err: ecclient: upload failed (node:1SjM6Utm2YizeUbrBUD9Z7etE1nLadW5Fa1zYP7MioPeeqhdJ7, address:104.36.82.12:6158): protocol: expected piece hash; context deadline exceeded; EOF Jun 22 01:33:47.235448297 (81h10m22.798771306s) [seg_pos={0 0}, upload=64958]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jun 22 01:33:47.265442393 (81h10m22.828765401s) [upload=64958]: Done uploading segments... Jun 22 01:33:47.265476139 (81h10m22.82879914s) [upload=64958]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration:* 21.921334981s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.....................................................X......] duration: 21.882394132s FAIL (2025-06-22 01:49:25 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 21.882430029s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [......................................X.....................] duration: 18.770519674s FAIL (2025-06-22 01:48:39 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== oad=64949]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jun 22 01:32:13.674692303 (81h8m49.238015311s) [upload=64949]: Done uploading segments... Jun 22 01:32:13.674727219 (81h8m49.238050233s) [upload=64949]: Upload finished. err: <nil> Jun 22 01:32:47.266407995 (81h9m22.829731002s) [upload=64958]: Starting upload Jun 22 01:32:47.266524767 (81h9m22.829847771s) [upload=64958]: Done starting upload Jun 22 01:32:47.266813109 (81h9m22.830136116s) [upload=64958]: Uploading segments... Jun 22 01:32:47.266969526 (81h9m22.830292526s) [upload=64958]: Got next segment. Inline: false Jun 22 01:32:47.267075475 (81h9m22.830398475s) [upload=64958]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 22 01:32:47.267081679 (81h9m22.830404667s) [upload=64958]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 22 01:32:47.301770382 (81h9m22.865093389s) [seg_pos={0 0}, upload=64958]: Begin upload segment... Jun 22 01:32:47.301823796 (81h9m22.865146799s) [seg_pos={0 0}, upload=64958]: Done begin upload segment. Jun 22 01:32:47.301855055 (81h9m22.865178087s) [noise=true , address=104.36.82.43:6841 , piece_id=PIHZMSTO3CII7KGDLGSAILTDRSLMEBVOLBCRCZJULMCFISXIZWXA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.301902600 (81h9m22.865225606s) [noise=true , address=104.36.82.42:6811 , piece_id=A5BFEHZ3WGGK53IZ7MKI6N4CVXHOORF6L47NR3UFDCAX7LBHRHVQ, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.302234027 (81h9m22.865557029s) [noise=true , address=104.36.82.51:4084 , piece_id=D43FA7BFE54KPCWGDGDME2EGSWM7PONOVPD3DN2XY42MJFOKVDIQ, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.302496354 (81h9m22.865819363s) [noise=true , address=79.127.219.38:4023 , piece_id=BWLRCXXHBMBTNCZEVKKIIS5PB35PDQXQXCLIQB7GBLEIOWWUFBFA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.302577370 (81h9m22.865900378s) [noise=true , address=104.36.82.52:4089 , piece_id=JSOEKLNWA3W3LOVKI3EGGKJYJ6J3UZY6KGABKQPQPXTIOOPMKVHA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.302917829 (81h9m22.866240835s) [noise=true , address=104.36.82.53:4122 , piece_id=HZZQBIKHQWLUE5EHK4KSKJGIUEG7D44RQASUJHCYXX6BICCGX6EA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.303087165 (81h9m22.866410168s) [noise=true , address=77.90.31.14:4689 , piece_id=Q3HDS3BCMYDTODFRMU4XFRDCQK2QZE3ZMXVLUX4M54PDEBS7RR6A, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.303250714 (81h9m22.866573717s) [noise=true , address=79.127.205.234:5601 , piece_id=V7PMXBP2VFWBSZDG4GTLBD435T6F5DT7A43L4GQLQ5BDDTJ4WY6A, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.303579249 (81h9m22.866902255s) [noise=true , address=104.36.82.12:6158 , piece_id=LJZEOJVJCJAR2BRILIJI3ZVDGYMPBBTP2G3HCTN3MHNS4KYDPASQ, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.303599776 (81h9m22.866922784s) [noise=true , address=104.36.82.9:6023 , piece_id=SISFAUU4RPF57OLGWM2BWQWE6GRUSCMORBYHWEXETNAVM2EEMO6Q, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.304005346 (81h9m22.867328346s) [noise=true , address=45.91.116.12:5242 , piece_id=6ABT5IPYGSLFFJMBYMBYRU4EYIXABUAUHG3OVILOHBIUDS3BP27A, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.304388850 (81h9m22.867711851s) [noise=true , address=64.49.13.137:4500 , piece_id=FAE2RXD57IRP5FXA2XR6THXFW27U7IWAZ4JDV5M2WRPXKUGZ4VIA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.304881529 (81h9m22.868204536s) [noise=true , address=79.127.220.101:6048 , piece_id=AN7UQUXB42T2YXEH7ZJRZUDIW47M3RWUEOCL2VEBOGH74BZF3YKA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.305249860 (81h9m22.86857287s) [noise=true , address=104.36.82.54:4165 , piece_id=4CVOABPCWC7J7SRHZB6IN2WJJ6KGWUQ66ELUIZIFA67ETLQ7JEUQ, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.305604108 (81h9m22.868927112s) [noise=true , address=104.36.82.15:6289 , piece_id=GWL3P55BIK4IJEL5YFZKI7JCA4LADYOOHUSIQUNGJEKBJOTKMP5A, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.305930423 (81h9m22.869253425s) [noise=true , address=64.49.13.139:4476 , piece_id=P7UKBNNZGI52B3I6WKTV6MVC262FYNKYYPJUI5G47UO5S4MU7PWA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.306355311 (81h9m22.869678319s) [noise=true , address=104.36.82.55:4213 , piece_id=X4IHS2XSPH52LIMNUYT7KGWMAPUH3TEMJOEEVB3JV5N45SEOH7YA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.306788243 (81h9m22.870111252s) [noise=true , address=104.36.82.22:6539 , piece_id=GHQ2MVSTDGVYLUKLV23D6YWRR4PP7UUYHCUEQ4WWSWEMTBFO3P4A, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.392174770 (81h9m22.955497773s) [upload=64958]: Next returned nil segment Jun 22 01:32:47.392195767 (81h9m22.955518775s) [upload=64958]: Waiting for error group managing segments... Jun 22 01:32:47.480767633 (81h9m23.044090623s) [noise=true , address=104.36.82.22:6539 , piece_id=GHQ2MVSTDGVYLUKLV23D6YWRR4PP7UUYHCUEQ4WWSWEMTBFO3P4A, seg_pos={0 0}, upload=64958]: Done uploading piece. err: ecclient: upload failed (node:1QL6w8DSFR6CxDdnJmDuyVZxp6JU55VKiJofBZRfQi7q31i8Jc, address:104.36.82.22:6539): protocol: storage node overloaded, request limit: 2000 Jun 22 01:32:47.585663142 (81h9m23.148986155s) [noise=true , address=79.127.220.101:6048 , piece_id=AN7UQUXB42T2YXEH7ZJRZUDIW47M3RWUEOCL2VEBOGH74BZF3YKA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:47.640263002 (81h9m23.203586013s) [noise=true , address=45.91.116.12:5242 , piece_id=6ABT5IPYGSLFFJMBYMBYRU4EYIXABUAUHG3OVILOHBIUDS3BP27A, seg_pos={0 0}, upload=64958]: Done uploading piece. err: ecclient: upload failed (node:12JsFL62jgXmza8Md8eQ9k8cyw5V9Vz2RzN4y85ZvgHZjwrq4vQ, address:45.91.116.12:5242): protocol: storage node overloaded, request limit: 2000 Jun 22 01:32:47.984127275 (81h9m23.54745028s) [noise=true , address=64.49.13.137:4500 , piece_id=FAE2RXD57IRP5FXA2XR6THXFW27U7IWAZ4JDV5M2WRPXKUGZ4VIA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:48.100255249 (81h9m23.663578241s) [noise=true , address=77.90.31.14:4689 , piece_id=Q3HDS3BCMYDTODFRMU4XFRDCQK2QZE3ZMXVLUX4M54PDEBS7RR6A, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:48.221764035 (81h9m23.785087034s) [noise=true , address=104.36.82.54:4165 , piece_id=4CVOABPCWC7J7SRHZB6IN2WJJ6KGWUQ66ELUIZIFA67ETLQ7JEUQ, seg_pos={0 0}, upload=64958]: Done uploading piece. err: ecclient: upload failed (node:12wGUMXVEubEtki4qGnjGEsbYFB1iUauZcu3oDRKrCQsJMV3bV2, address:104.36.82.54:4165): protocol: storage node overloaded, request limit: 2000 Jun 22 01:32:48.509996527 (81h9m24.073319547s) [noise=true , address=104.36.82.9:6023 , piece_id=SISFAUU4RPF57OLGWM2BWQWE6GRUSCMORBYHWEXETNAVM2EEMO6Q, seg_pos={0 0}, upload=64958]: Done uploading piece. err: ecclient: upload failed (node:1j2S9tgWpLwmyvXZSq2qKKcbaC8r7LkzfDCaMxCejAni4CST1g, address:104.36.82.9:6023): protocol: storage node overloaded, request limit: 2000 Jun 22 01:32:48.510513169 (81h9m24.073836177s) [noise=true , address=104.36.82.42:6811 , piece_id=A5BFEHZ3WGGK53IZ7MKI6N4CVXHOORF6L47NR3UFDCAX7LBHRHVQ, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:48.565245605 (81h9m24.128568606s) [noise=true , address=79.127.205.234:5601 , piece_id=V7PMXBP2VFWBSZDG4GTLBD435T6F5DT7A43L4GQLQ5BDDTJ4WY6A, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:49.979979824 (81h9m25.543302825s) [noise=true , address=104.36.82.15:6289 , piece_id=GWL3P55BIK4IJEL5YFZKI7JCA4LADYOOHUSIQUNGJEKBJOTKMP5A, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:50.463750280 (81h9m26.02707328s) [noise=true , address=104.36.82.51:4084 , piece_id=D43FA7BFE54KPCWGDGDME2EGSWM7PONOVPD3DN2XY42MJFOKVDIQ, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:51.480599795 (81h9m27.043922798s) [noise=true , address=79.127.219.38:4023 , piece_id=BWLRCXXHBMBTNCZEVKKIIS5PB35PDQXQXCLIQB7GBLEIOWWUFBFA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:52.941200625 (81h9m28.504523627s) [noise=true , address=104.36.82.43:6841 , piece_id=PIHZMSTO3CII7KGDLGSAILTDRSLMEBVOLBCRCZJULMCFISXIZWXA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:58.281803598 (81h9m33.845126612s) [noise=true , address=64.49.13.139:4476 , piece_id=P7UKBNNZGI52B3I6WKTV6MVC262FYNKYYPJUI5G47UO5S4MU7PWA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:33:00.277271877 (81h9m35.840594867s) [noise=true , address=104.36.82.55:4213 , piece_id=X4IHS2XSPH52LIMNUYT7KGWMAPUH3TEMJOEEVB3JV5N45SEOH7YA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:33:09.435339213 (81h9m44.998662206s) [noise=true , address=104.36.82.52:4089 , piece_id=JSOEKLNWA3W3LOVKI3EGGKJYJ6J3UZY6KGABKQPQPXTIOOPMKVHA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:33:31.472868445 (81h10m7.03619143s) [noise=true , address=104.36.82.53:4122 , piece_id=HZZQBIKHQWLUE5EHK4KSKJGIUEG7D44RQASUJHCYXX6BICCGX6EA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:33:47.235404653 (81h10m22.798727642s) [noise=true , address=104.36.82.12:6158 , piece_id=LJZEOJVJCJAR2BRILIJI3ZVDGYMPBBTP2G3HCTN3MHNS4KYDPASQ, seg_pos={0 0}, upload=64958]: Done uploading piece. err: ecclient: upload failed (node:1SjM6Utm2YizeUbrBUD9Z7etE1nLadW5Fa1zYP7MioPeeqhdJ7, address:104.36.82.12:6158): protocol: expected piece hash; context deadline exceeded; EOF Jun 22 01:33:47.235448297 (81h10m22.798771306s) [seg_pos={0 0}, upload=64958]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jun 22 01:33:47.265442393 (81h10m22.828765401s) [upload=64958]: Done uploading segments... Jun 22 01:33:47.265476139 (81h10m22.82879914s) [upload=64958]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration:* 18.770564878s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [......................................................X.....] duration: 25.811115944s FAIL (2025-06-22 01:48:19 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 25.811152273s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.......................................X....................] duration: 33.782852375s FAIL (2025-06-22 01:47:43 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== oad=64949]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jun 22 01:32:13.674692303 (81h8m49.238015311s) [upload=64949]: Done uploading segments... Jun 22 01:32:13.674727219 (81h8m49.238050233s) [upload=64949]: Upload finished. err: <nil> Jun 22 01:32:47.266407995 (81h9m22.829731002s) [upload=64958]: Starting upload Jun 22 01:32:47.266524767 (81h9m22.829847771s) [upload=64958]: Done starting upload Jun 22 01:32:47.266813109 (81h9m22.830136116s) [upload=64958]: Uploading segments... Jun 22 01:32:47.266969526 (81h9m22.830292526s) [upload=64958]: Got next segment. Inline: false Jun 22 01:32:47.267075475 (81h9m22.830398475s) [upload=64958]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 22 01:32:47.267081679 (81h9m22.830404667s) [upload=64958]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 22 01:32:47.301770382 (81h9m22.865093389s) [seg_pos={0 0}, upload=64958]: Begin upload segment... Jun 22 01:32:47.301823796 (81h9m22.865146799s) [seg_pos={0 0}, upload=64958]: Done begin upload segment. Jun 22 01:32:47.301855055 (81h9m22.865178087s) [noise=true , address=104.36.82.43:6841 , piece_id=PIHZMSTO3CII7KGDLGSAILTDRSLMEBVOLBCRCZJULMCFISXIZWXA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.301902600 (81h9m22.865225606s) [noise=true , address=104.36.82.42:6811 , piece_id=A5BFEHZ3WGGK53IZ7MKI6N4CVXHOORF6L47NR3UFDCAX7LBHRHVQ, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.302234027 (81h9m22.865557029s) [noise=true , address=104.36.82.51:4084 , piece_id=D43FA7BFE54KPCWGDGDME2EGSWM7PONOVPD3DN2XY42MJFOKVDIQ, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.302496354 (81h9m22.865819363s) [noise=true , address=79.127.219.38:4023 , piece_id=BWLRCXXHBMBTNCZEVKKIIS5PB35PDQXQXCLIQB7GBLEIOWWUFBFA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.302577370 (81h9m22.865900378s) [noise=true , address=104.36.82.52:4089 , piece_id=JSOEKLNWA3W3LOVKI3EGGKJYJ6J3UZY6KGABKQPQPXTIOOPMKVHA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.302917829 (81h9m22.866240835s) [noise=true , address=104.36.82.53:4122 , piece_id=HZZQBIKHQWLUE5EHK4KSKJGIUEG7D44RQASUJHCYXX6BICCGX6EA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.303087165 (81h9m22.866410168s) [noise=true , address=77.90.31.14:4689 , piece_id=Q3HDS3BCMYDTODFRMU4XFRDCQK2QZE3ZMXVLUX4M54PDEBS7RR6A, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.303250714 (81h9m22.866573717s) [noise=true , address=79.127.205.234:5601 , piece_id=V7PMXBP2VFWBSZDG4GTLBD435T6F5DT7A43L4GQLQ5BDDTJ4WY6A, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.303579249 (81h9m22.866902255s) [noise=true , address=104.36.82.12:6158 , piece_id=LJZEOJVJCJAR2BRILIJI3ZVDGYMPBBTP2G3HCTN3MHNS4KYDPASQ, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.303599776 (81h9m22.866922784s) [noise=true , address=104.36.82.9:6023 , piece_id=SISFAUU4RPF57OLGWM2BWQWE6GRUSCMORBYHWEXETNAVM2EEMO6Q, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.304005346 (81h9m22.867328346s) [noise=true , address=45.91.116.12:5242 , piece_id=6ABT5IPYGSLFFJMBYMBYRU4EYIXABUAUHG3OVILOHBIUDS3BP27A, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.304388850 (81h9m22.867711851s) [noise=true , address=64.49.13.137:4500 , piece_id=FAE2RXD57IRP5FXA2XR6THXFW27U7IWAZ4JDV5M2WRPXKUGZ4VIA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.304881529 (81h9m22.868204536s) [noise=true , address=79.127.220.101:6048 , piece_id=AN7UQUXB42T2YXEH7ZJRZUDIW47M3RWUEOCL2VEBOGH74BZF3YKA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.305249860 (81h9m22.86857287s) [noise=true , address=104.36.82.54:4165 , piece_id=4CVOABPCWC7J7SRHZB6IN2WJJ6KGWUQ66ELUIZIFA67ETLQ7JEUQ, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.305604108 (81h9m22.868927112s) [noise=true , address=104.36.82.15:6289 , piece_id=GWL3P55BIK4IJEL5YFZKI7JCA4LADYOOHUSIQUNGJEKBJOTKMP5A, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.305930423 (81h9m22.869253425s) [noise=true , address=64.49.13.139:4476 , piece_id=P7UKBNNZGI52B3I6WKTV6MVC262FYNKYYPJUI5G47UO5S4MU7PWA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.306355311 (81h9m22.869678319s) [noise=true , address=104.36.82.55:4213 , piece_id=X4IHS2XSPH52LIMNUYT7KGWMAPUH3TEMJOEEVB3JV5N45SEOH7YA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.306788243 (81h9m22.870111252s) [noise=true , address=104.36.82.22:6539 , piece_id=GHQ2MVSTDGVYLUKLV23D6YWRR4PP7UUYHCUEQ4WWSWEMTBFO3P4A, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.392174770 (81h9m22.955497773s) [upload=64958]: Next returned nil segment Jun 22 01:32:47.392195767 (81h9m22.955518775s) [upload=64958]: Waiting for error group managing segments... Jun 22 01:32:47.480767633 (81h9m23.044090623s) [noise=true , address=104.36.82.22:6539 , piece_id=GHQ2MVSTDGVYLUKLV23D6YWRR4PP7UUYHCUEQ4WWSWEMTBFO3P4A, seg_pos={0 0}, upload=64958]: Done uploading piece. err: ecclient: upload failed (node:1QL6w8DSFR6CxDdnJmDuyVZxp6JU55VKiJofBZRfQi7q31i8Jc, address:104.36.82.22:6539): protocol: storage node overloaded, request limit: 2000 Jun 22 01:32:47.585663142 (81h9m23.148986155s) [noise=true , address=79.127.220.101:6048 , piece_id=AN7UQUXB42T2YXEH7ZJRZUDIW47M3RWUEOCL2VEBOGH74BZF3YKA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:47.640263002 (81h9m23.203586013s) [noise=true , address=45.91.116.12:5242 , piece_id=6ABT5IPYGSLFFJMBYMBYRU4EYIXABUAUHG3OVILOHBIUDS3BP27A, seg_pos={0 0}, upload=64958]: Done uploading piece. err: ecclient: upload failed (node:12JsFL62jgXmza8Md8eQ9k8cyw5V9Vz2RzN4y85ZvgHZjwrq4vQ, address:45.91.116.12:5242): protocol: storage node overloaded, request limit: 2000 Jun 22 01:32:47.984127275 (81h9m23.54745028s) [noise=true , address=64.49.13.137:4500 , piece_id=FAE2RXD57IRP5FXA2XR6THXFW27U7IWAZ4JDV5M2WRPXKUGZ4VIA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:48.100255249 (81h9m23.663578241s) [noise=true , address=77.90.31.14:4689 , piece_id=Q3HDS3BCMYDTODFRMU4XFRDCQK2QZE3ZMXVLUX4M54PDEBS7RR6A, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:48.221764035 (81h9m23.785087034s) [noise=true , address=104.36.82.54:4165 , piece_id=4CVOABPCWC7J7SRHZB6IN2WJJ6KGWUQ66ELUIZIFA67ETLQ7JEUQ, seg_pos={0 0}, upload=64958]: Done uploading piece. err: ecclient: upload failed (node:12wGUMXVEubEtki4qGnjGEsbYFB1iUauZcu3oDRKrCQsJMV3bV2, address:104.36.82.54:4165): protocol: storage node overloaded, request limit: 2000 Jun 22 01:32:48.509996527 (81h9m24.073319547s) [noise=true , address=104.36.82.9:6023 , piece_id=SISFAUU4RPF57OLGWM2BWQWE6GRUSCMORBYHWEXETNAVM2EEMO6Q, seg_pos={0 0}, upload=64958]: Done uploading piece. err: ecclient: upload failed (node:1j2S9tgWpLwmyvXZSq2qKKcbaC8r7LkzfDCaMxCejAni4CST1g, address:104.36.82.9:6023): protocol: storage node overloaded, request limit: 2000 Jun 22 01:32:48.510513169 (81h9m24.073836177s) [noise=true , address=104.36.82.42:6811 , piece_id=A5BFEHZ3WGGK53IZ7MKI6N4CVXHOORF6L47NR3UFDCAX7LBHRHVQ, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:48.565245605 (81h9m24.128568606s) [noise=true , address=79.127.205.234:5601 , piece_id=V7PMXBP2VFWBSZDG4GTLBD435T6F5DT7A43L4GQLQ5BDDTJ4WY6A, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:49.979979824 (81h9m25.543302825s) [noise=true , address=104.36.82.15:6289 , piece_id=GWL3P55BIK4IJEL5YFZKI7JCA4LADYOOHUSIQUNGJEKBJOTKMP5A, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:50.463750280 (81h9m26.02707328s) [noise=true , address=104.36.82.51:4084 , piece_id=D43FA7BFE54KPCWGDGDME2EGSWM7PONOVPD3DN2XY42MJFOKVDIQ, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:51.480599795 (81h9m27.043922798s) [noise=true , address=79.127.219.38:4023 , piece_id=BWLRCXXHBMBTNCZEVKKIIS5PB35PDQXQXCLIQB7GBLEIOWWUFBFA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:52.941200625 (81h9m28.504523627s) [noise=true , address=104.36.82.43:6841 , piece_id=PIHZMSTO3CII7KGDLGSAILTDRSLMEBVOLBCRCZJULMCFISXIZWXA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:58.281803598 (81h9m33.845126612s) [noise=true , address=64.49.13.139:4476 , piece_id=P7UKBNNZGI52B3I6WKTV6MVC262FYNKYYPJUI5G47UO5S4MU7PWA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:33:00.277271877 (81h9m35.840594867s) [noise=true , address=104.36.82.55:4213 , piece_id=X4IHS2XSPH52LIMNUYT7KGWMAPUH3TEMJOEEVB3JV5N45SEOH7YA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:33:09.435339213 (81h9m44.998662206s) [noise=true , address=104.36.82.52:4089 , piece_id=JSOEKLNWA3W3LOVKI3EGGKJYJ6J3UZY6KGABKQPQPXTIOOPMKVHA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:33:31.472868445 (81h10m7.03619143s) [noise=true , address=104.36.82.53:4122 , piece_id=HZZQBIKHQWLUE5EHK4KSKJGIUEG7D44RQASUJHCYXX6BICCGX6EA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:33:47.235404653 (81h10m22.798727642s) [noise=true , address=104.36.82.12:6158 , piece_id=LJZEOJVJCJAR2BRILIJI3ZVDGYMPBBTP2G3HCTN3MHNS4KYDPASQ, seg_pos={0 0}, upload=64958]: Done uploading piece. err: ecclient: upload failed (node:1SjM6Utm2YizeUbrBUD9Z7etE1nLadW5Fa1zYP7MioPeeqhdJ7, address:104.36.82.12:6158): protocol: expected piece hash; context deadline exceeded; EOF Jun 22 01:33:47.235448297 (81h10m22.798771306s) [seg_pos={0 0}, upload=64958]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jun 22 01:33:47.265442393 (81h10m22.828765401s) [upload=64958]: Done uploading segments... Jun 22 01:33:47.265476139 (81h10m22.82879914s) [upload=64958]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration:* 33.78289126s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.......................................................X....] duration: 25.595753098s FAIL (2025-06-22 01:47:26 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 25.595796813s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [........................................X...................] duration: 25.246085699s FAIL (2025-06-22 01:47:09 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== oad=64949]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jun 22 01:32:13.674692303 (81h8m49.238015311s) [upload=64949]: Done uploading segments... Jun 22 01:32:13.674727219 (81h8m49.238050233s) [upload=64949]: Upload finished. err: <nil> Jun 22 01:32:47.266407995 (81h9m22.829731002s) [upload=64958]: Starting upload Jun 22 01:32:47.266524767 (81h9m22.829847771s) [upload=64958]: Done starting upload Jun 22 01:32:47.266813109 (81h9m22.830136116s) [upload=64958]: Uploading segments... Jun 22 01:32:47.266969526 (81h9m22.830292526s) [upload=64958]: Got next segment. Inline: false Jun 22 01:32:47.267075475 (81h9m22.830398475s) [upload=64958]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 22 01:32:47.267081679 (81h9m22.830404667s) [upload=64958]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 22 01:32:47.301770382 (81h9m22.865093389s) [seg_pos={0 0}, upload=64958]: Begin upload segment... Jun 22 01:32:47.301823796 (81h9m22.865146799s) [seg_pos={0 0}, upload=64958]: Done begin upload segment. Jun 22 01:32:47.301855055 (81h9m22.865178087s) [noise=true , address=104.36.82.43:6841 , piece_id=PIHZMSTO3CII7KGDLGSAILTDRSLMEBVOLBCRCZJULMCFISXIZWXA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.301902600 (81h9m22.865225606s) [noise=true , address=104.36.82.42:6811 , piece_id=A5BFEHZ3WGGK53IZ7MKI6N4CVXHOORF6L47NR3UFDCAX7LBHRHVQ, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.302234027 (81h9m22.865557029s) [noise=true , address=104.36.82.51:4084 , piece_id=D43FA7BFE54KPCWGDGDME2EGSWM7PONOVPD3DN2XY42MJFOKVDIQ, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.302496354 (81h9m22.865819363s) [noise=true , address=79.127.219.38:4023 , piece_id=BWLRCXXHBMBTNCZEVKKIIS5PB35PDQXQXCLIQB7GBLEIOWWUFBFA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.302577370 (81h9m22.865900378s) [noise=true , address=104.36.82.52:4089 , piece_id=JSOEKLNWA3W3LOVKI3EGGKJYJ6J3UZY6KGABKQPQPXTIOOPMKVHA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.302917829 (81h9m22.866240835s) [noise=true , address=104.36.82.53:4122 , piece_id=HZZQBIKHQWLUE5EHK4KSKJGIUEG7D44RQASUJHCYXX6BICCGX6EA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.303087165 (81h9m22.866410168s) [noise=true , address=77.90.31.14:4689 , piece_id=Q3HDS3BCMYDTODFRMU4XFRDCQK2QZE3ZMXVLUX4M54PDEBS7RR6A, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.303250714 (81h9m22.866573717s) [noise=true , address=79.127.205.234:5601 , piece_id=V7PMXBP2VFWBSZDG4GTLBD435T6F5DT7A43L4GQLQ5BDDTJ4WY6A, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.303579249 (81h9m22.866902255s) [noise=true , address=104.36.82.12:6158 , piece_id=LJZEOJVJCJAR2BRILIJI3ZVDGYMPBBTP2G3HCTN3MHNS4KYDPASQ, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.303599776 (81h9m22.866922784s) [noise=true , address=104.36.82.9:6023 , piece_id=SISFAUU4RPF57OLGWM2BWQWE6GRUSCMORBYHWEXETNAVM2EEMO6Q, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.304005346 (81h9m22.867328346s) [noise=true , address=45.91.116.12:5242 , piece_id=6ABT5IPYGSLFFJMBYMBYRU4EYIXABUAUHG3OVILOHBIUDS3BP27A, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.304388850 (81h9m22.867711851s) [noise=true , address=64.49.13.137:4500 , piece_id=FAE2RXD57IRP5FXA2XR6THXFW27U7IWAZ4JDV5M2WRPXKUGZ4VIA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.304881529 (81h9m22.868204536s) [noise=true , address=79.127.220.101:6048 , piece_id=AN7UQUXB42T2YXEH7ZJRZUDIW47M3RWUEOCL2VEBOGH74BZF3YKA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.305249860 (81h9m22.86857287s) [noise=true , address=104.36.82.54:4165 , piece_id=4CVOABPCWC7J7SRHZB6IN2WJJ6KGWUQ66ELUIZIFA67ETLQ7JEUQ, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.305604108 (81h9m22.868927112s) [noise=true , address=104.36.82.15:6289 , piece_id=GWL3P55BIK4IJEL5YFZKI7JCA4LADYOOHUSIQUNGJEKBJOTKMP5A, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.305930423 (81h9m22.869253425s) [noise=true , address=64.49.13.139:4476 , piece_id=P7UKBNNZGI52B3I6WKTV6MVC262FYNKYYPJUI5G47UO5S4MU7PWA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.306355311 (81h9m22.869678319s) [noise=true , address=104.36.82.55:4213 , piece_id=X4IHS2XSPH52LIMNUYT7KGWMAPUH3TEMJOEEVB3JV5N45SEOH7YA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.306788243 (81h9m22.870111252s) [noise=true , address=104.36.82.22:6539 , piece_id=GHQ2MVSTDGVYLUKLV23D6YWRR4PP7UUYHCUEQ4WWSWEMTBFO3P4A, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.392174770 (81h9m22.955497773s) [upload=64958]: Next returned nil segment Jun 22 01:32:47.392195767 (81h9m22.955518775s) [upload=64958]: Waiting for error group managing segments... Jun 22 01:32:47.480767633 (81h9m23.044090623s) [noise=true , address=104.36.82.22:6539 , piece_id=GHQ2MVSTDGVYLUKLV23D6YWRR4PP7UUYHCUEQ4WWSWEMTBFO3P4A, seg_pos={0 0}, upload=64958]: Done uploading piece. err: ecclient: upload failed (node:1QL6w8DSFR6CxDdnJmDuyVZxp6JU55VKiJofBZRfQi7q31i8Jc, address:104.36.82.22:6539): protocol: storage node overloaded, request limit: 2000 Jun 22 01:32:47.585663142 (81h9m23.148986155s) [noise=true , address=79.127.220.101:6048 , piece_id=AN7UQUXB42T2YXEH7ZJRZUDIW47M3RWUEOCL2VEBOGH74BZF3YKA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:47.640263002 (81h9m23.203586013s) [noise=true , address=45.91.116.12:5242 , piece_id=6ABT5IPYGSLFFJMBYMBYRU4EYIXABUAUHG3OVILOHBIUDS3BP27A, seg_pos={0 0}, upload=64958]: Done uploading piece. err: ecclient: upload failed (node:12JsFL62jgXmza8Md8eQ9k8cyw5V9Vz2RzN4y85ZvgHZjwrq4vQ, address:45.91.116.12:5242): protocol: storage node overloaded, request limit: 2000 Jun 22 01:32:47.984127275 (81h9m23.54745028s) [noise=true , address=64.49.13.137:4500 , piece_id=FAE2RXD57IRP5FXA2XR6THXFW27U7IWAZ4JDV5M2WRPXKUGZ4VIA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:48.100255249 (81h9m23.663578241s) [noise=true , address=77.90.31.14:4689 , piece_id=Q3HDS3BCMYDTODFRMU4XFRDCQK2QZE3ZMXVLUX4M54PDEBS7RR6A, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:48.221764035 (81h9m23.785087034s) [noise=true , address=104.36.82.54:4165 , piece_id=4CVOABPCWC7J7SRHZB6IN2WJJ6KGWUQ66ELUIZIFA67ETLQ7JEUQ, seg_pos={0 0}, upload=64958]: Done uploading piece. err: ecclient: upload failed (node:12wGUMXVEubEtki4qGnjGEsbYFB1iUauZcu3oDRKrCQsJMV3bV2, address:104.36.82.54:4165): protocol: storage node overloaded, request limit: 2000 Jun 22 01:32:48.509996527 (81h9m24.073319547s) [noise=true , address=104.36.82.9:6023 , piece_id=SISFAUU4RPF57OLGWM2BWQWE6GRUSCMORBYHWEXETNAVM2EEMO6Q, seg_pos={0 0}, upload=64958]: Done uploading piece. err: ecclient: upload failed (node:1j2S9tgWpLwmyvXZSq2qKKcbaC8r7LkzfDCaMxCejAni4CST1g, address:104.36.82.9:6023): protocol: storage node overloaded, request limit: 2000 Jun 22 01:32:48.510513169 (81h9m24.073836177s) [noise=true , address=104.36.82.42:6811 , piece_id=A5BFEHZ3WGGK53IZ7MKI6N4CVXHOORF6L47NR3UFDCAX7LBHRHVQ, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:48.565245605 (81h9m24.128568606s) [noise=true , address=79.127.205.234:5601 , piece_id=V7PMXBP2VFWBSZDG4GTLBD435T6F5DT7A43L4GQLQ5BDDTJ4WY6A, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:49.979979824 (81h9m25.543302825s) [noise=true , address=104.36.82.15:6289 , piece_id=GWL3P55BIK4IJEL5YFZKI7JCA4LADYOOHUSIQUNGJEKBJOTKMP5A, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:50.463750280 (81h9m26.02707328s) [noise=true , address=104.36.82.51:4084 , piece_id=D43FA7BFE54KPCWGDGDME2EGSWM7PONOVPD3DN2XY42MJFOKVDIQ, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:51.480599795 (81h9m27.043922798s) [noise=true , address=79.127.219.38:4023 , piece_id=BWLRCXXHBMBTNCZEVKKIIS5PB35PDQXQXCLIQB7GBLEIOWWUFBFA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:52.941200625 (81h9m28.504523627s) [noise=true , address=104.36.82.43:6841 , piece_id=PIHZMSTO3CII7KGDLGSAILTDRSLMEBVOLBCRCZJULMCFISXIZWXA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:58.281803598 (81h9m33.845126612s) [noise=true , address=64.49.13.139:4476 , piece_id=P7UKBNNZGI52B3I6WKTV6MVC262FYNKYYPJUI5G47UO5S4MU7PWA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:33:00.277271877 (81h9m35.840594867s) [noise=true , address=104.36.82.55:4213 , piece_id=X4IHS2XSPH52LIMNUYT7KGWMAPUH3TEMJOEEVB3JV5N45SEOH7YA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:33:09.435339213 (81h9m44.998662206s) [noise=true , address=104.36.82.52:4089 , piece_id=JSOEKLNWA3W3LOVKI3EGGKJYJ6J3UZY6KGABKQPQPXTIOOPMKVHA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:33:31.472868445 (81h10m7.03619143s) [noise=true , address=104.36.82.53:4122 , piece_id=HZZQBIKHQWLUE5EHK4KSKJGIUEG7D44RQASUJHCYXX6BICCGX6EA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:33:47.235404653 (81h10m22.798727642s) [noise=true , address=104.36.82.12:6158 , piece_id=LJZEOJVJCJAR2BRILIJI3ZVDGYMPBBTP2G3HCTN3MHNS4KYDPASQ, seg_pos={0 0}, upload=64958]: Done uploading piece. err: ecclient: upload failed (node:1SjM6Utm2YizeUbrBUD9Z7etE1nLadW5Fa1zYP7MioPeeqhdJ7, address:104.36.82.12:6158): protocol: expected piece hash; context deadline exceeded; EOF Jun 22 01:33:47.235448297 (81h10m22.798771306s) [seg_pos={0 0}, upload=64958]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jun 22 01:33:47.265442393 (81h10m22.828765401s) [upload=64958]: Done uploading segments... Jun 22 01:33:47.265476139 (81h10m22.82879914s) [upload=64958]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration:* 25.246117239s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [........................................................X...] duration: 18.751117954s FAIL (2025-06-22 01:46:34 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 18.751169735s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.........................................X..................] duration: 14.352526291s FAIL (2025-06-22 01:46:28 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== oad=64949]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jun 22 01:32:13.674692303 (81h8m49.238015311s) [upload=64949]: Done uploading segments... Jun 22 01:32:13.674727219 (81h8m49.238050233s) [upload=64949]: Upload finished. err: <nil> Jun 22 01:32:47.266407995 (81h9m22.829731002s) [upload=64958]: Starting upload Jun 22 01:32:47.266524767 (81h9m22.829847771s) [upload=64958]: Done starting upload Jun 22 01:32:47.266813109 (81h9m22.830136116s) [upload=64958]: Uploading segments... Jun 22 01:32:47.266969526 (81h9m22.830292526s) [upload=64958]: Got next segment. Inline: false Jun 22 01:32:47.267075475 (81h9m22.830398475s) [upload=64958]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 22 01:32:47.267081679 (81h9m22.830404667s) [upload=64958]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 22 01:32:47.301770382 (81h9m22.865093389s) [seg_pos={0 0}, upload=64958]: Begin upload segment... Jun 22 01:32:47.301823796 (81h9m22.865146799s) [seg_pos={0 0}, upload=64958]: Done begin upload segment. Jun 22 01:32:47.301855055 (81h9m22.865178087s) [noise=true , address=104.36.82.43:6841 , piece_id=PIHZMSTO3CII7KGDLGSAILTDRSLMEBVOLBCRCZJULMCFISXIZWXA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.301902600 (81h9m22.865225606s) [noise=true , address=104.36.82.42:6811 , piece_id=A5BFEHZ3WGGK53IZ7MKI6N4CVXHOORF6L47NR3UFDCAX7LBHRHVQ, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.302234027 (81h9m22.865557029s) [noise=true , address=104.36.82.51:4084 , piece_id=D43FA7BFE54KPCWGDGDME2EGSWM7PONOVPD3DN2XY42MJFOKVDIQ, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.302496354 (81h9m22.865819363s) [noise=true , address=79.127.219.38:4023 , piece_id=BWLRCXXHBMBTNCZEVKKIIS5PB35PDQXQXCLIQB7GBLEIOWWUFBFA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.302577370 (81h9m22.865900378s) [noise=true , address=104.36.82.52:4089 , piece_id=JSOEKLNWA3W3LOVKI3EGGKJYJ6J3UZY6KGABKQPQPXTIOOPMKVHA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.302917829 (81h9m22.866240835s) [noise=true , address=104.36.82.53:4122 , piece_id=HZZQBIKHQWLUE5EHK4KSKJGIUEG7D44RQASUJHCYXX6BICCGX6EA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.303087165 (81h9m22.866410168s) [noise=true , address=77.90.31.14:4689 , piece_id=Q3HDS3BCMYDTODFRMU4XFRDCQK2QZE3ZMXVLUX4M54PDEBS7RR6A, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.303250714 (81h9m22.866573717s) [noise=true , address=79.127.205.234:5601 , piece_id=V7PMXBP2VFWBSZDG4GTLBD435T6F5DT7A43L4GQLQ5BDDTJ4WY6A, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.303579249 (81h9m22.866902255s) [noise=true , address=104.36.82.12:6158 , piece_id=LJZEOJVJCJAR2BRILIJI3ZVDGYMPBBTP2G3HCTN3MHNS4KYDPASQ, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.303599776 (81h9m22.866922784s) [noise=true , address=104.36.82.9:6023 , piece_id=SISFAUU4RPF57OLGWM2BWQWE6GRUSCMORBYHWEXETNAVM2EEMO6Q, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.304005346 (81h9m22.867328346s) [noise=true , address=45.91.116.12:5242 , piece_id=6ABT5IPYGSLFFJMBYMBYRU4EYIXABUAUHG3OVILOHBIUDS3BP27A, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.304388850 (81h9m22.867711851s) [noise=true , address=64.49.13.137:4500 , piece_id=FAE2RXD57IRP5FXA2XR6THXFW27U7IWAZ4JDV5M2WRPXKUGZ4VIA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.304881529 (81h9m22.868204536s) [noise=true , address=79.127.220.101:6048 , piece_id=AN7UQUXB42T2YXEH7ZJRZUDIW47M3RWUEOCL2VEBOGH74BZF3YKA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.305249860 (81h9m22.86857287s) [noise=true , address=104.36.82.54:4165 , piece_id=4CVOABPCWC7J7SRHZB6IN2WJJ6KGWUQ66ELUIZIFA67ETLQ7JEUQ, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.305604108 (81h9m22.868927112s) [noise=true , address=104.36.82.15:6289 , piece_id=GWL3P55BIK4IJEL5YFZKI7JCA4LADYOOHUSIQUNGJEKBJOTKMP5A, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.305930423 (81h9m22.869253425s) [noise=true , address=64.49.13.139:4476 , piece_id=P7UKBNNZGI52B3I6WKTV6MVC262FYNKYYPJUI5G47UO5S4MU7PWA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.306355311 (81h9m22.869678319s) [noise=true , address=104.36.82.55:4213 , piece_id=X4IHS2XSPH52LIMNUYT7KGWMAPUH3TEMJOEEVB3JV5N45SEOH7YA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.306788243 (81h9m22.870111252s) [noise=true , address=104.36.82.22:6539 , piece_id=GHQ2MVSTDGVYLUKLV23D6YWRR4PP7UUYHCUEQ4WWSWEMTBFO3P4A, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.392174770 (81h9m22.955497773s) [upload=64958]: Next returned nil segment Jun 22 01:32:47.392195767 (81h9m22.955518775s) [upload=64958]: Waiting for error group managing segments... Jun 22 01:32:47.480767633 (81h9m23.044090623s) [noise=true , address=104.36.82.22:6539 , piece_id=GHQ2MVSTDGVYLUKLV23D6YWRR4PP7UUYHCUEQ4WWSWEMTBFO3P4A, seg_pos={0 0}, upload=64958]: Done uploading piece. err: ecclient: upload failed (node:1QL6w8DSFR6CxDdnJmDuyVZxp6JU55VKiJofBZRfQi7q31i8Jc, address:104.36.82.22:6539): protocol: storage node overloaded, request limit: 2000 Jun 22 01:32:47.585663142 (81h9m23.148986155s) [noise=true , address=79.127.220.101:6048 , piece_id=AN7UQUXB42T2YXEH7ZJRZUDIW47M3RWUEOCL2VEBOGH74BZF3YKA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:47.640263002 (81h9m23.203586013s) [noise=true , address=45.91.116.12:5242 , piece_id=6ABT5IPYGSLFFJMBYMBYRU4EYIXABUAUHG3OVILOHBIUDS3BP27A, seg_pos={0 0}, upload=64958]: Done uploading piece. err: ecclient: upload failed (node:12JsFL62jgXmza8Md8eQ9k8cyw5V9Vz2RzN4y85ZvgHZjwrq4vQ, address:45.91.116.12:5242): protocol: storage node overloaded, request limit: 2000 Jun 22 01:32:47.984127275 (81h9m23.54745028s) [noise=true , address=64.49.13.137:4500 , piece_id=FAE2RXD57IRP5FXA2XR6THXFW27U7IWAZ4JDV5M2WRPXKUGZ4VIA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:48.100255249 (81h9m23.663578241s) [noise=true , address=77.90.31.14:4689 , piece_id=Q3HDS3BCMYDTODFRMU4XFRDCQK2QZE3ZMXVLUX4M54PDEBS7RR6A, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:48.221764035 (81h9m23.785087034s) [noise=true , address=104.36.82.54:4165 , piece_id=4CVOABPCWC7J7SRHZB6IN2WJJ6KGWUQ66ELUIZIFA67ETLQ7JEUQ, seg_pos={0 0}, upload=64958]: Done uploading piece. err: ecclient: upload failed (node:12wGUMXVEubEtki4qGnjGEsbYFB1iUauZcu3oDRKrCQsJMV3bV2, address:104.36.82.54:4165): protocol: storage node overloaded, request limit: 2000 Jun 22 01:32:48.509996527 (81h9m24.073319547s) [noise=true , address=104.36.82.9:6023 , piece_id=SISFAUU4RPF57OLGWM2BWQWE6GRUSCMORBYHWEXETNAVM2EEMO6Q, seg_pos={0 0}, upload=64958]: Done uploading piece. err: ecclient: upload failed (node:1j2S9tgWpLwmyvXZSq2qKKcbaC8r7LkzfDCaMxCejAni4CST1g, address:104.36.82.9:6023): protocol: storage node overloaded, request limit: 2000 Jun 22 01:32:48.510513169 (81h9m24.073836177s) [noise=true , address=104.36.82.42:6811 , piece_id=A5BFEHZ3WGGK53IZ7MKI6N4CVXHOORF6L47NR3UFDCAX7LBHRHVQ, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:48.565245605 (81h9m24.128568606s) [noise=true , address=79.127.205.234:5601 , piece_id=V7PMXBP2VFWBSZDG4GTLBD435T6F5DT7A43L4GQLQ5BDDTJ4WY6A, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:49.979979824 (81h9m25.543302825s) [noise=true , address=104.36.82.15:6289 , piece_id=GWL3P55BIK4IJEL5YFZKI7JCA4LADYOOHUSIQUNGJEKBJOTKMP5A, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:50.463750280 (81h9m26.02707328s) [noise=true , address=104.36.82.51:4084 , piece_id=D43FA7BFE54KPCWGDGDME2EGSWM7PONOVPD3DN2XY42MJFOKVDIQ, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:51.480599795 (81h9m27.043922798s) [noise=true , address=79.127.219.38:4023 , piece_id=BWLRCXXHBMBTNCZEVKKIIS5PB35PDQXQXCLIQB7GBLEIOWWUFBFA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:52.941200625 (81h9m28.504523627s) [noise=true , address=104.36.82.43:6841 , piece_id=PIHZMSTO3CII7KGDLGSAILTDRSLMEBVOLBCRCZJULMCFISXIZWXA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:58.281803598 (81h9m33.845126612s) [noise=true , address=64.49.13.139:4476 , piece_id=P7UKBNNZGI52B3I6WKTV6MVC262FYNKYYPJUI5G47UO5S4MU7PWA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:33:00.277271877 (81h9m35.840594867s) [noise=true , address=104.36.82.55:4213 , piece_id=X4IHS2XSPH52LIMNUYT7KGWMAPUH3TEMJOEEVB3JV5N45SEOH7YA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:33:09.435339213 (81h9m44.998662206s) [noise=true , address=104.36.82.52:4089 , piece_id=JSOEKLNWA3W3LOVKI3EGGKJYJ6J3UZY6KGABKQPQPXTIOOPMKVHA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:33:31.472868445 (81h10m7.03619143s) [noise=true , address=104.36.82.53:4122 , piece_id=HZZQBIKHQWLUE5EHK4KSKJGIUEG7D44RQASUJHCYXX6BICCGX6EA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:33:47.235404653 (81h10m22.798727642s) [noise=true , address=104.36.82.12:6158 , piece_id=LJZEOJVJCJAR2BRILIJI3ZVDGYMPBBTP2G3HCTN3MHNS4KYDPASQ, seg_pos={0 0}, upload=64958]: Done uploading piece. err: ecclient: upload failed (node:1SjM6Utm2YizeUbrBUD9Z7etE1nLadW5Fa1zYP7MioPeeqhdJ7, address:104.36.82.12:6158): protocol: expected piece hash; context deadline exceeded; EOF Jun 22 01:33:47.235448297 (81h10m22.798771306s) [seg_pos={0 0}, upload=64958]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jun 22 01:33:47.265442393 (81h10m22.828765401s) [upload=64958]: Done uploading segments... Jun 22 01:33:47.265476139 (81h10m22.82879914s) [upload=64958]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration:* 14.352555769s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.........................................................X..] duration: 15.625552002s FAIL (2025-06-22 01:46:04 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 15.625940325s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [..........................................................X.] duration: 27.836739222s FAIL (2025-06-22 01:45:30 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 27.836772716s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [..........................................X.................] duration: 13.766458314s FAIL (2025-06-22 01:45:19 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== oad=64949]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jun 22 01:32:13.674692303 (81h8m49.238015311s) [upload=64949]: Done uploading segments... Jun 22 01:32:13.674727219 (81h8m49.238050233s) [upload=64949]: Upload finished. err: <nil> Jun 22 01:32:47.266407995 (81h9m22.829731002s) [upload=64958]: Starting upload Jun 22 01:32:47.266524767 (81h9m22.829847771s) [upload=64958]: Done starting upload Jun 22 01:32:47.266813109 (81h9m22.830136116s) [upload=64958]: Uploading segments... Jun 22 01:32:47.266969526 (81h9m22.830292526s) [upload=64958]: Got next segment. Inline: false Jun 22 01:32:47.267075475 (81h9m22.830398475s) [upload=64958]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 22 01:32:47.267081679 (81h9m22.830404667s) [upload=64958]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 22 01:32:47.301770382 (81h9m22.865093389s) [seg_pos={0 0}, upload=64958]: Begin upload segment... Jun 22 01:32:47.301823796 (81h9m22.865146799s) [seg_pos={0 0}, upload=64958]: Done begin upload segment. Jun 22 01:32:47.301855055 (81h9m22.865178087s) [noise=true , address=104.36.82.43:6841 , piece_id=PIHZMSTO3CII7KGDLGSAILTDRSLMEBVOLBCRCZJULMCFISXIZWXA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.301902600 (81h9m22.865225606s) [noise=true , address=104.36.82.42:6811 , piece_id=A5BFEHZ3WGGK53IZ7MKI6N4CVXHOORF6L47NR3UFDCAX7LBHRHVQ, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.302234027 (81h9m22.865557029s) [noise=true , address=104.36.82.51:4084 , piece_id=D43FA7BFE54KPCWGDGDME2EGSWM7PONOVPD3DN2XY42MJFOKVDIQ, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.302496354 (81h9m22.865819363s) [noise=true , address=79.127.219.38:4023 , piece_id=BWLRCXXHBMBTNCZEVKKIIS5PB35PDQXQXCLIQB7GBLEIOWWUFBFA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.302577370 (81h9m22.865900378s) [noise=true , address=104.36.82.52:4089 , piece_id=JSOEKLNWA3W3LOVKI3EGGKJYJ6J3UZY6KGABKQPQPXTIOOPMKVHA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.302917829 (81h9m22.866240835s) [noise=true , address=104.36.82.53:4122 , piece_id=HZZQBIKHQWLUE5EHK4KSKJGIUEG7D44RQASUJHCYXX6BICCGX6EA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.303087165 (81h9m22.866410168s) [noise=true , address=77.90.31.14:4689 , piece_id=Q3HDS3BCMYDTODFRMU4XFRDCQK2QZE3ZMXVLUX4M54PDEBS7RR6A, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.303250714 (81h9m22.866573717s) [noise=true , address=79.127.205.234:5601 , piece_id=V7PMXBP2VFWBSZDG4GTLBD435T6F5DT7A43L4GQLQ5BDDTJ4WY6A, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.303579249 (81h9m22.866902255s) [noise=true , address=104.36.82.12:6158 , piece_id=LJZEOJVJCJAR2BRILIJI3ZVDGYMPBBTP2G3HCTN3MHNS4KYDPASQ, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.303599776 (81h9m22.866922784s) [noise=true , address=104.36.82.9:6023 , piece_id=SISFAUU4RPF57OLGWM2BWQWE6GRUSCMORBYHWEXETNAVM2EEMO6Q, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.304005346 (81h9m22.867328346s) [noise=true , address=45.91.116.12:5242 , piece_id=6ABT5IPYGSLFFJMBYMBYRU4EYIXABUAUHG3OVILOHBIUDS3BP27A, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.304388850 (81h9m22.867711851s) [noise=true , address=64.49.13.137:4500 , piece_id=FAE2RXD57IRP5FXA2XR6THXFW27U7IWAZ4JDV5M2WRPXKUGZ4VIA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.304881529 (81h9m22.868204536s) [noise=true , address=79.127.220.101:6048 , piece_id=AN7UQUXB42T2YXEH7ZJRZUDIW47M3RWUEOCL2VEBOGH74BZF3YKA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.305249860 (81h9m22.86857287s) [noise=true , address=104.36.82.54:4165 , piece_id=4CVOABPCWC7J7SRHZB6IN2WJJ6KGWUQ66ELUIZIFA67ETLQ7JEUQ, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.305604108 (81h9m22.868927112s) [noise=true , address=104.36.82.15:6289 , piece_id=GWL3P55BIK4IJEL5YFZKI7JCA4LADYOOHUSIQUNGJEKBJOTKMP5A, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.305930423 (81h9m22.869253425s) [noise=true , address=64.49.13.139:4476 , piece_id=P7UKBNNZGI52B3I6WKTV6MVC262FYNKYYPJUI5G47UO5S4MU7PWA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.306355311 (81h9m22.869678319s) [noise=true , address=104.36.82.55:4213 , piece_id=X4IHS2XSPH52LIMNUYT7KGWMAPUH3TEMJOEEVB3JV5N45SEOH7YA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.306788243 (81h9m22.870111252s) [noise=true , address=104.36.82.22:6539 , piece_id=GHQ2MVSTDGVYLUKLV23D6YWRR4PP7UUYHCUEQ4WWSWEMTBFO3P4A, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.392174770 (81h9m22.955497773s) [upload=64958]: Next returned nil segment Jun 22 01:32:47.392195767 (81h9m22.955518775s) [upload=64958]: Waiting for error group managing segments... Jun 22 01:32:47.480767633 (81h9m23.044090623s) [noise=true , address=104.36.82.22:6539 , piece_id=GHQ2MVSTDGVYLUKLV23D6YWRR4PP7UUYHCUEQ4WWSWEMTBFO3P4A, seg_pos={0 0}, upload=64958]: Done uploading piece. err: ecclient: upload failed (node:1QL6w8DSFR6CxDdnJmDuyVZxp6JU55VKiJofBZRfQi7q31i8Jc, address:104.36.82.22:6539): protocol: storage node overloaded, request limit: 2000 Jun 22 01:32:47.585663142 (81h9m23.148986155s) [noise=true , address=79.127.220.101:6048 , piece_id=AN7UQUXB42T2YXEH7ZJRZUDIW47M3RWUEOCL2VEBOGH74BZF3YKA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:47.640263002 (81h9m23.203586013s) [noise=true , address=45.91.116.12:5242 , piece_id=6ABT5IPYGSLFFJMBYMBYRU4EYIXABUAUHG3OVILOHBIUDS3BP27A, seg_pos={0 0}, upload=64958]: Done uploading piece. err: ecclient: upload failed (node:12JsFL62jgXmza8Md8eQ9k8cyw5V9Vz2RzN4y85ZvgHZjwrq4vQ, address:45.91.116.12:5242): protocol: storage node overloaded, request limit: 2000 Jun 22 01:32:47.984127275 (81h9m23.54745028s) [noise=true , address=64.49.13.137:4500 , piece_id=FAE2RXD57IRP5FXA2XR6THXFW27U7IWAZ4JDV5M2WRPXKUGZ4VIA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:48.100255249 (81h9m23.663578241s) [noise=true , address=77.90.31.14:4689 , piece_id=Q3HDS3BCMYDTODFRMU4XFRDCQK2QZE3ZMXVLUX4M54PDEBS7RR6A, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:48.221764035 (81h9m23.785087034s) [noise=true , address=104.36.82.54:4165 , piece_id=4CVOABPCWC7J7SRHZB6IN2WJJ6KGWUQ66ELUIZIFA67ETLQ7JEUQ, seg_pos={0 0}, upload=64958]: Done uploading piece. err: ecclient: upload failed (node:12wGUMXVEubEtki4qGnjGEsbYFB1iUauZcu3oDRKrCQsJMV3bV2, address:104.36.82.54:4165): protocol: storage node overloaded, request limit: 2000 Jun 22 01:32:48.509996527 (81h9m24.073319547s) [noise=true , address=104.36.82.9:6023 , piece_id=SISFAUU4RPF57OLGWM2BWQWE6GRUSCMORBYHWEXETNAVM2EEMO6Q, seg_pos={0 0}, upload=64958]: Done uploading piece. err: ecclient: upload failed (node:1j2S9tgWpLwmyvXZSq2qKKcbaC8r7LkzfDCaMxCejAni4CST1g, address:104.36.82.9:6023): protocol: storage node overloaded, request limit: 2000 Jun 22 01:32:48.510513169 (81h9m24.073836177s) [noise=true , address=104.36.82.42:6811 , piece_id=A5BFEHZ3WGGK53IZ7MKI6N4CVXHOORF6L47NR3UFDCAX7LBHRHVQ, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:48.565245605 (81h9m24.128568606s) [noise=true , address=79.127.205.234:5601 , piece_id=V7PMXBP2VFWBSZDG4GTLBD435T6F5DT7A43L4GQLQ5BDDTJ4WY6A, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:49.979979824 (81h9m25.543302825s) [noise=true , address=104.36.82.15:6289 , piece_id=GWL3P55BIK4IJEL5YFZKI7JCA4LADYOOHUSIQUNGJEKBJOTKMP5A, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:50.463750280 (81h9m26.02707328s) [noise=true , address=104.36.82.51:4084 , piece_id=D43FA7BFE54KPCWGDGDME2EGSWM7PONOVPD3DN2XY42MJFOKVDIQ, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:51.480599795 (81h9m27.043922798s) [noise=true , address=79.127.219.38:4023 , piece_id=BWLRCXXHBMBTNCZEVKKIIS5PB35PDQXQXCLIQB7GBLEIOWWUFBFA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:52.941200625 (81h9m28.504523627s) [noise=true , address=104.36.82.43:6841 , piece_id=PIHZMSTO3CII7KGDLGSAILTDRSLMEBVOLBCRCZJULMCFISXIZWXA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:58.281803598 (81h9m33.845126612s) [noise=true , address=64.49.13.139:4476 , piece_id=P7UKBNNZGI52B3I6WKTV6MVC262FYNKYYPJUI5G47UO5S4MU7PWA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:33:00.277271877 (81h9m35.840594867s) [noise=true , address=104.36.82.55:4213 , piece_id=X4IHS2XSPH52LIMNUYT7KGWMAPUH3TEMJOEEVB3JV5N45SEOH7YA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:33:09.435339213 (81h9m44.998662206s) [noise=true , address=104.36.82.52:4089 , piece_id=JSOEKLNWA3W3LOVKI3EGGKJYJ6J3UZY6KGABKQPQPXTIOOPMKVHA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:33:31.472868445 (81h10m7.03619143s) [noise=true , address=104.36.82.53:4122 , piece_id=HZZQBIKHQWLUE5EHK4KSKJGIUEG7D44RQASUJHCYXX6BICCGX6EA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:33:47.235404653 (81h10m22.798727642s) [noise=true , address=104.36.82.12:6158 , piece_id=LJZEOJVJCJAR2BRILIJI3ZVDGYMPBBTP2G3HCTN3MHNS4KYDPASQ, seg_pos={0 0}, upload=64958]: Done uploading piece. err: ecclient: upload failed (node:1SjM6Utm2YizeUbrBUD9Z7etE1nLadW5Fa1zYP7MioPeeqhdJ7, address:104.36.82.12:6158): protocol: expected piece hash; context deadline exceeded; EOF Jun 22 01:33:47.235448297 (81h10m22.798771306s) [seg_pos={0 0}, upload=64958]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jun 22 01:33:47.265442393 (81h10m22.828765401s) [upload=64958]: Done uploading segments... Jun 22 01:33:47.265476139 (81h10m22.82879914s) [upload=64958]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration:* 13.766490015s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [...........................................................X] duration: 1m0.001358065s FAIL (2025-06-22 01:45:02 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1m0.001394117s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [...........................................X................] duration: 17.628523942s FAIL (2025-06-22 01:44:29 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== oad=64949]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jun 22 01:32:13.674692303 (81h8m49.238015311s) [upload=64949]: Done uploading segments... Jun 22 01:32:13.674727219 (81h8m49.238050233s) [upload=64949]: Upload finished. err: <nil> Jun 22 01:32:47.266407995 (81h9m22.829731002s) [upload=64958]: Starting upload Jun 22 01:32:47.266524767 (81h9m22.829847771s) [upload=64958]: Done starting upload Jun 22 01:32:47.266813109 (81h9m22.830136116s) [upload=64958]: Uploading segments... Jun 22 01:32:47.266969526 (81h9m22.830292526s) [upload=64958]: Got next segment. Inline: false Jun 22 01:32:47.267075475 (81h9m22.830398475s) [upload=64958]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 22 01:32:47.267081679 (81h9m22.830404667s) [upload=64958]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 22 01:32:47.301770382 (81h9m22.865093389s) [seg_pos={0 0}, upload=64958]: Begin upload segment... Jun 22 01:32:47.301823796 (81h9m22.865146799s) [seg_pos={0 0}, upload=64958]: Done begin upload segment. Jun 22 01:32:47.301855055 (81h9m22.865178087s) [noise=true , address=104.36.82.43:6841 , piece_id=PIHZMSTO3CII7KGDLGSAILTDRSLMEBVOLBCRCZJULMCFISXIZWXA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.301902600 (81h9m22.865225606s) [noise=true , address=104.36.82.42:6811 , piece_id=A5BFEHZ3WGGK53IZ7MKI6N4CVXHOORF6L47NR3UFDCAX7LBHRHVQ, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.302234027 (81h9m22.865557029s) [noise=true , address=104.36.82.51:4084 , piece_id=D43FA7BFE54KPCWGDGDME2EGSWM7PONOVPD3DN2XY42MJFOKVDIQ, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.302496354 (81h9m22.865819363s) [noise=true , address=79.127.219.38:4023 , piece_id=BWLRCXXHBMBTNCZEVKKIIS5PB35PDQXQXCLIQB7GBLEIOWWUFBFA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.302577370 (81h9m22.865900378s) [noise=true , address=104.36.82.52:4089 , piece_id=JSOEKLNWA3W3LOVKI3EGGKJYJ6J3UZY6KGABKQPQPXTIOOPMKVHA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.302917829 (81h9m22.866240835s) [noise=true , address=104.36.82.53:4122 , piece_id=HZZQBIKHQWLUE5EHK4KSKJGIUEG7D44RQASUJHCYXX6BICCGX6EA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.303087165 (81h9m22.866410168s) [noise=true , address=77.90.31.14:4689 , piece_id=Q3HDS3BCMYDTODFRMU4XFRDCQK2QZE3ZMXVLUX4M54PDEBS7RR6A, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.303250714 (81h9m22.866573717s) [noise=true , address=79.127.205.234:5601 , piece_id=V7PMXBP2VFWBSZDG4GTLBD435T6F5DT7A43L4GQLQ5BDDTJ4WY6A, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.303579249 (81h9m22.866902255s) [noise=true , address=104.36.82.12:6158 , piece_id=LJZEOJVJCJAR2BRILIJI3ZVDGYMPBBTP2G3HCTN3MHNS4KYDPASQ, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.303599776 (81h9m22.866922784s) [noise=true , address=104.36.82.9:6023 , piece_id=SISFAUU4RPF57OLGWM2BWQWE6GRUSCMORBYHWEXETNAVM2EEMO6Q, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.304005346 (81h9m22.867328346s) [noise=true , address=45.91.116.12:5242 , piece_id=6ABT5IPYGSLFFJMBYMBYRU4EYIXABUAUHG3OVILOHBIUDS3BP27A, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.304388850 (81h9m22.867711851s) [noise=true , address=64.49.13.137:4500 , piece_id=FAE2RXD57IRP5FXA2XR6THXFW27U7IWAZ4JDV5M2WRPXKUGZ4VIA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.304881529 (81h9m22.868204536s) [noise=true , address=79.127.220.101:6048 , piece_id=AN7UQUXB42T2YXEH7ZJRZUDIW47M3RWUEOCL2VEBOGH74BZF3YKA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.305249860 (81h9m22.86857287s) [noise=true , address=104.36.82.54:4165 , piece_id=4CVOABPCWC7J7SRHZB6IN2WJJ6KGWUQ66ELUIZIFA67ETLQ7JEUQ, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.305604108 (81h9m22.868927112s) [noise=true , address=104.36.82.15:6289 , piece_id=GWL3P55BIK4IJEL5YFZKI7JCA4LADYOOHUSIQUNGJEKBJOTKMP5A, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.305930423 (81h9m22.869253425s) [noise=true , address=64.49.13.139:4476 , piece_id=P7UKBNNZGI52B3I6WKTV6MVC262FYNKYYPJUI5G47UO5S4MU7PWA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.306355311 (81h9m22.869678319s) [noise=true , address=104.36.82.55:4213 , piece_id=X4IHS2XSPH52LIMNUYT7KGWMAPUH3TEMJOEEVB3JV5N45SEOH7YA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.306788243 (81h9m22.870111252s) [noise=true , address=104.36.82.22:6539 , piece_id=GHQ2MVSTDGVYLUKLV23D6YWRR4PP7UUYHCUEQ4WWSWEMTBFO3P4A, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.392174770 (81h9m22.955497773s) [upload=64958]: Next returned nil segment Jun 22 01:32:47.392195767 (81h9m22.955518775s) [upload=64958]: Waiting for error group managing segments... Jun 22 01:32:47.480767633 (81h9m23.044090623s) [noise=true , address=104.36.82.22:6539 , piece_id=GHQ2MVSTDGVYLUKLV23D6YWRR4PP7UUYHCUEQ4WWSWEMTBFO3P4A, seg_pos={0 0}, upload=64958]: Done uploading piece. err: ecclient: upload failed (node:1QL6w8DSFR6CxDdnJmDuyVZxp6JU55VKiJofBZRfQi7q31i8Jc, address:104.36.82.22:6539): protocol: storage node overloaded, request limit: 2000 Jun 22 01:32:47.585663142 (81h9m23.148986155s) [noise=true , address=79.127.220.101:6048 , piece_id=AN7UQUXB42T2YXEH7ZJRZUDIW47M3RWUEOCL2VEBOGH74BZF3YKA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:47.640263002 (81h9m23.203586013s) [noise=true , address=45.91.116.12:5242 , piece_id=6ABT5IPYGSLFFJMBYMBYRU4EYIXABUAUHG3OVILOHBIUDS3BP27A, seg_pos={0 0}, upload=64958]: Done uploading piece. err: ecclient: upload failed (node:12JsFL62jgXmza8Md8eQ9k8cyw5V9Vz2RzN4y85ZvgHZjwrq4vQ, address:45.91.116.12:5242): protocol: storage node overloaded, request limit: 2000 Jun 22 01:32:47.984127275 (81h9m23.54745028s) [noise=true , address=64.49.13.137:4500 , piece_id=FAE2RXD57IRP5FXA2XR6THXFW27U7IWAZ4JDV5M2WRPXKUGZ4VIA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:48.100255249 (81h9m23.663578241s) [noise=true , address=77.90.31.14:4689 , piece_id=Q3HDS3BCMYDTODFRMU4XFRDCQK2QZE3ZMXVLUX4M54PDEBS7RR6A, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:48.221764035 (81h9m23.785087034s) [noise=true , address=104.36.82.54:4165 , piece_id=4CVOABPCWC7J7SRHZB6IN2WJJ6KGWUQ66ELUIZIFA67ETLQ7JEUQ, seg_pos={0 0}, upload=64958]: Done uploading piece. err: ecclient: upload failed (node:12wGUMXVEubEtki4qGnjGEsbYFB1iUauZcu3oDRKrCQsJMV3bV2, address:104.36.82.54:4165): protocol: storage node overloaded, request limit: 2000 Jun 22 01:32:48.509996527 (81h9m24.073319547s) [noise=true , address=104.36.82.9:6023 , piece_id=SISFAUU4RPF57OLGWM2BWQWE6GRUSCMORBYHWEXETNAVM2EEMO6Q, seg_pos={0 0}, upload=64958]: Done uploading piece. err: ecclient: upload failed (node:1j2S9tgWpLwmyvXZSq2qKKcbaC8r7LkzfDCaMxCejAni4CST1g, address:104.36.82.9:6023): protocol: storage node overloaded, request limit: 2000 Jun 22 01:32:48.510513169 (81h9m24.073836177s) [noise=true , address=104.36.82.42:6811 , piece_id=A5BFEHZ3WGGK53IZ7MKI6N4CVXHOORF6L47NR3UFDCAX7LBHRHVQ, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:48.565245605 (81h9m24.128568606s) [noise=true , address=79.127.205.234:5601 , piece_id=V7PMXBP2VFWBSZDG4GTLBD435T6F5DT7A43L4GQLQ5BDDTJ4WY6A, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:49.979979824 (81h9m25.543302825s) [noise=true , address=104.36.82.15:6289 , piece_id=GWL3P55BIK4IJEL5YFZKI7JCA4LADYOOHUSIQUNGJEKBJOTKMP5A, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:50.463750280 (81h9m26.02707328s) [noise=true , address=104.36.82.51:4084 , piece_id=D43FA7BFE54KPCWGDGDME2EGSWM7PONOVPD3DN2XY42MJFOKVDIQ, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:51.480599795 (81h9m27.043922798s) [noise=true , address=79.127.219.38:4023 , piece_id=BWLRCXXHBMBTNCZEVKKIIS5PB35PDQXQXCLIQB7GBLEIOWWUFBFA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:52.941200625 (81h9m28.504523627s) [noise=true , address=104.36.82.43:6841 , piece_id=PIHZMSTO3CII7KGDLGSAILTDRSLMEBVOLBCRCZJULMCFISXIZWXA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:58.281803598 (81h9m33.845126612s) [noise=true , address=64.49.13.139:4476 , piece_id=P7UKBNNZGI52B3I6WKTV6MVC262FYNKYYPJUI5G47UO5S4MU7PWA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:33:00.277271877 (81h9m35.840594867s) [noise=true , address=104.36.82.55:4213 , piece_id=X4IHS2XSPH52LIMNUYT7KGWMAPUH3TEMJOEEVB3JV5N45SEOH7YA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:33:09.435339213 (81h9m44.998662206s) [noise=true , address=104.36.82.52:4089 , piece_id=JSOEKLNWA3W3LOVKI3EGGKJYJ6J3UZY6KGABKQPQPXTIOOPMKVHA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:33:31.472868445 (81h10m7.03619143s) [noise=true , address=104.36.82.53:4122 , piece_id=HZZQBIKHQWLUE5EHK4KSKJGIUEG7D44RQASUJHCYXX6BICCGX6EA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:33:47.235404653 (81h10m22.798727642s) [noise=true , address=104.36.82.12:6158 , piece_id=LJZEOJVJCJAR2BRILIJI3ZVDGYMPBBTP2G3HCTN3MHNS4KYDPASQ, seg_pos={0 0}, upload=64958]: Done uploading piece. err: ecclient: upload failed (node:1SjM6Utm2YizeUbrBUD9Z7etE1nLadW5Fa1zYP7MioPeeqhdJ7, address:104.36.82.12:6158): protocol: expected piece hash; context deadline exceeded; EOF Jun 22 01:33:47.235448297 (81h10m22.798771306s) [seg_pos={0 0}, upload=64958]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jun 22 01:33:47.265442393 (81h10m22.828765401s) [upload=64958]: Done uploading segments... Jun 22 01:33:47.265476139 (81h10m22.82879914s) [upload=64958]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration:* 17.628541937s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [............................................X...............] duration: 35.654373092s FAIL (2025-06-22 01:44:11 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== oad=64949]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jun 22 01:32:13.674692303 (81h8m49.238015311s) [upload=64949]: Done uploading segments... Jun 22 01:32:13.674727219 (81h8m49.238050233s) [upload=64949]: Upload finished. err: <nil> Jun 22 01:32:47.266407995 (81h9m22.829731002s) [upload=64958]: Starting upload Jun 22 01:32:47.266524767 (81h9m22.829847771s) [upload=64958]: Done starting upload Jun 22 01:32:47.266813109 (81h9m22.830136116s) [upload=64958]: Uploading segments... Jun 22 01:32:47.266969526 (81h9m22.830292526s) [upload=64958]: Got next segment. Inline: false Jun 22 01:32:47.267075475 (81h9m22.830398475s) [upload=64958]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 22 01:32:47.267081679 (81h9m22.830404667s) [upload=64958]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 22 01:32:47.301770382 (81h9m22.865093389s) [seg_pos={0 0}, upload=64958]: Begin upload segment... Jun 22 01:32:47.301823796 (81h9m22.865146799s) [seg_pos={0 0}, upload=64958]: Done begin upload segment. Jun 22 01:32:47.301855055 (81h9m22.865178087s) [noise=true , address=104.36.82.43:6841 , piece_id=PIHZMSTO3CII7KGDLGSAILTDRSLMEBVOLBCRCZJULMCFISXIZWXA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.301902600 (81h9m22.865225606s) [noise=true , address=104.36.82.42:6811 , piece_id=A5BFEHZ3WGGK53IZ7MKI6N4CVXHOORF6L47NR3UFDCAX7LBHRHVQ, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.302234027 (81h9m22.865557029s) [noise=true , address=104.36.82.51:4084 , piece_id=D43FA7BFE54KPCWGDGDME2EGSWM7PONOVPD3DN2XY42MJFOKVDIQ, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.302496354 (81h9m22.865819363s) [noise=true , address=79.127.219.38:4023 , piece_id=BWLRCXXHBMBTNCZEVKKIIS5PB35PDQXQXCLIQB7GBLEIOWWUFBFA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.302577370 (81h9m22.865900378s) [noise=true , address=104.36.82.52:4089 , piece_id=JSOEKLNWA3W3LOVKI3EGGKJYJ6J3UZY6KGABKQPQPXTIOOPMKVHA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.302917829 (81h9m22.866240835s) [noise=true , address=104.36.82.53:4122 , piece_id=HZZQBIKHQWLUE5EHK4KSKJGIUEG7D44RQASUJHCYXX6BICCGX6EA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.303087165 (81h9m22.866410168s) [noise=true , address=77.90.31.14:4689 , piece_id=Q3HDS3BCMYDTODFRMU4XFRDCQK2QZE3ZMXVLUX4M54PDEBS7RR6A, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.303250714 (81h9m22.866573717s) [noise=true , address=79.127.205.234:5601 , piece_id=V7PMXBP2VFWBSZDG4GTLBD435T6F5DT7A43L4GQLQ5BDDTJ4WY6A, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.303579249 (81h9m22.866902255s) [noise=true , address=104.36.82.12:6158 , piece_id=LJZEOJVJCJAR2BRILIJI3ZVDGYMPBBTP2G3HCTN3MHNS4KYDPASQ, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.303599776 (81h9m22.866922784s) [noise=true , address=104.36.82.9:6023 , piece_id=SISFAUU4RPF57OLGWM2BWQWE6GRUSCMORBYHWEXETNAVM2EEMO6Q, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.304005346 (81h9m22.867328346s) [noise=true , address=45.91.116.12:5242 , piece_id=6ABT5IPYGSLFFJMBYMBYRU4EYIXABUAUHG3OVILOHBIUDS3BP27A, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.304388850 (81h9m22.867711851s) [noise=true , address=64.49.13.137:4500 , piece_id=FAE2RXD57IRP5FXA2XR6THXFW27U7IWAZ4JDV5M2WRPXKUGZ4VIA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.304881529 (81h9m22.868204536s) [noise=true , address=79.127.220.101:6048 , piece_id=AN7UQUXB42T2YXEH7ZJRZUDIW47M3RWUEOCL2VEBOGH74BZF3YKA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.305249860 (81h9m22.86857287s) [noise=true , address=104.36.82.54:4165 , piece_id=4CVOABPCWC7J7SRHZB6IN2WJJ6KGWUQ66ELUIZIFA67ETLQ7JEUQ, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.305604108 (81h9m22.868927112s) [noise=true , address=104.36.82.15:6289 , piece_id=GWL3P55BIK4IJEL5YFZKI7JCA4LADYOOHUSIQUNGJEKBJOTKMP5A, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.305930423 (81h9m22.869253425s) [noise=true , address=64.49.13.139:4476 , piece_id=P7UKBNNZGI52B3I6WKTV6MVC262FYNKYYPJUI5G47UO5S4MU7PWA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.306355311 (81h9m22.869678319s) [noise=true , address=104.36.82.55:4213 , piece_id=X4IHS2XSPH52LIMNUYT7KGWMAPUH3TEMJOEEVB3JV5N45SEOH7YA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.306788243 (81h9m22.870111252s) [noise=true , address=104.36.82.22:6539 , piece_id=GHQ2MVSTDGVYLUKLV23D6YWRR4PP7UUYHCUEQ4WWSWEMTBFO3P4A, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.392174770 (81h9m22.955497773s) [upload=64958]: Next returned nil segment Jun 22 01:32:47.392195767 (81h9m22.955518775s) [upload=64958]: Waiting for error group managing segments... Jun 22 01:32:47.480767633 (81h9m23.044090623s) [noise=true , address=104.36.82.22:6539 , piece_id=GHQ2MVSTDGVYLUKLV23D6YWRR4PP7UUYHCUEQ4WWSWEMTBFO3P4A, seg_pos={0 0}, upload=64958]: Done uploading piece. err: ecclient: upload failed (node:1QL6w8DSFR6CxDdnJmDuyVZxp6JU55VKiJofBZRfQi7q31i8Jc, address:104.36.82.22:6539): protocol: storage node overloaded, request limit: 2000 Jun 22 01:32:47.585663142 (81h9m23.148986155s) [noise=true , address=79.127.220.101:6048 , piece_id=AN7UQUXB42T2YXEH7ZJRZUDIW47M3RWUEOCL2VEBOGH74BZF3YKA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:47.640263002 (81h9m23.203586013s) [noise=true , address=45.91.116.12:5242 , piece_id=6ABT5IPYGSLFFJMBYMBYRU4EYIXABUAUHG3OVILOHBIUDS3BP27A, seg_pos={0 0}, upload=64958]: Done uploading piece. err: ecclient: upload failed (node:12JsFL62jgXmza8Md8eQ9k8cyw5V9Vz2RzN4y85ZvgHZjwrq4vQ, address:45.91.116.12:5242): protocol: storage node overloaded, request limit: 2000 Jun 22 01:32:47.984127275 (81h9m23.54745028s) [noise=true , address=64.49.13.137:4500 , piece_id=FAE2RXD57IRP5FXA2XR6THXFW27U7IWAZ4JDV5M2WRPXKUGZ4VIA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:48.100255249 (81h9m23.663578241s) [noise=true , address=77.90.31.14:4689 , piece_id=Q3HDS3BCMYDTODFRMU4XFRDCQK2QZE3ZMXVLUX4M54PDEBS7RR6A, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:48.221764035 (81h9m23.785087034s) [noise=true , address=104.36.82.54:4165 , piece_id=4CVOABPCWC7J7SRHZB6IN2WJJ6KGWUQ66ELUIZIFA67ETLQ7JEUQ, seg_pos={0 0}, upload=64958]: Done uploading piece. err: ecclient: upload failed (node:12wGUMXVEubEtki4qGnjGEsbYFB1iUauZcu3oDRKrCQsJMV3bV2, address:104.36.82.54:4165): protocol: storage node overloaded, request limit: 2000 Jun 22 01:32:48.509996527 (81h9m24.073319547s) [noise=true , address=104.36.82.9:6023 , piece_id=SISFAUU4RPF57OLGWM2BWQWE6GRUSCMORBYHWEXETNAVM2EEMO6Q, seg_pos={0 0}, upload=64958]: Done uploading piece. err: ecclient: upload failed (node:1j2S9tgWpLwmyvXZSq2qKKcbaC8r7LkzfDCaMxCejAni4CST1g, address:104.36.82.9:6023): protocol: storage node overloaded, request limit: 2000 Jun 22 01:32:48.510513169 (81h9m24.073836177s) [noise=true , address=104.36.82.42:6811 , piece_id=A5BFEHZ3WGGK53IZ7MKI6N4CVXHOORF6L47NR3UFDCAX7LBHRHVQ, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:48.565245605 (81h9m24.128568606s) [noise=true , address=79.127.205.234:5601 , piece_id=V7PMXBP2VFWBSZDG4GTLBD435T6F5DT7A43L4GQLQ5BDDTJ4WY6A, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:49.979979824 (81h9m25.543302825s) [noise=true , address=104.36.82.15:6289 , piece_id=GWL3P55BIK4IJEL5YFZKI7JCA4LADYOOHUSIQUNGJEKBJOTKMP5A, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:50.463750280 (81h9m26.02707328s) [noise=true , address=104.36.82.51:4084 , piece_id=D43FA7BFE54KPCWGDGDME2EGSWM7PONOVPD3DN2XY42MJFOKVDIQ, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:51.480599795 (81h9m27.043922798s) [noise=true , address=79.127.219.38:4023 , piece_id=BWLRCXXHBMBTNCZEVKKIIS5PB35PDQXQXCLIQB7GBLEIOWWUFBFA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:52.941200625 (81h9m28.504523627s) [noise=true , address=104.36.82.43:6841 , piece_id=PIHZMSTO3CII7KGDLGSAILTDRSLMEBVOLBCRCZJULMCFISXIZWXA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:58.281803598 (81h9m33.845126612s) [noise=true , address=64.49.13.139:4476 , piece_id=P7UKBNNZGI52B3I6WKTV6MVC262FYNKYYPJUI5G47UO5S4MU7PWA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:33:00.277271877 (81h9m35.840594867s) [noise=true , address=104.36.82.55:4213 , piece_id=X4IHS2XSPH52LIMNUYT7KGWMAPUH3TEMJOEEVB3JV5N45SEOH7YA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:33:09.435339213 (81h9m44.998662206s) [noise=true , address=104.36.82.52:4089 , piece_id=JSOEKLNWA3W3LOVKI3EGGKJYJ6J3UZY6KGABKQPQPXTIOOPMKVHA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:33:31.472868445 (81h10m7.03619143s) [noise=true , address=104.36.82.53:4122 , piece_id=HZZQBIKHQWLUE5EHK4KSKJGIUEG7D44RQASUJHCYXX6BICCGX6EA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:33:47.235404653 (81h10m22.798727642s) [noise=true , address=104.36.82.12:6158 , piece_id=LJZEOJVJCJAR2BRILIJI3ZVDGYMPBBTP2G3HCTN3MHNS4KYDPASQ, seg_pos={0 0}, upload=64958]: Done uploading piece. err: ecclient: upload failed (node:1SjM6Utm2YizeUbrBUD9Z7etE1nLadW5Fa1zYP7MioPeeqhdJ7, address:104.36.82.12:6158): protocol: expected piece hash; context deadline exceeded; EOF Jun 22 01:33:47.235448297 (81h10m22.798771306s) [seg_pos={0 0}, upload=64958]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jun 22 01:33:47.265442393 (81h10m22.828765401s) [upload=64958]: Done uploading segments... Jun 22 01:33:47.265476139 (81h10m22.82879914s) [upload=64958]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration:* 35.654427032s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.............................................X..............] duration: 14.325491555s FAIL (2025-06-22 01:43:29 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== oad=64949]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jun 22 01:32:13.674692303 (81h8m49.238015311s) [upload=64949]: Done uploading segments... Jun 22 01:32:13.674727219 (81h8m49.238050233s) [upload=64949]: Upload finished. err: <nil> Jun 22 01:32:47.266407995 (81h9m22.829731002s) [upload=64958]: Starting upload Jun 22 01:32:47.266524767 (81h9m22.829847771s) [upload=64958]: Done starting upload Jun 22 01:32:47.266813109 (81h9m22.830136116s) [upload=64958]: Uploading segments... Jun 22 01:32:47.266969526 (81h9m22.830292526s) [upload=64958]: Got next segment. Inline: false Jun 22 01:32:47.267075475 (81h9m22.830398475s) [upload=64958]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 22 01:32:47.267081679 (81h9m22.830404667s) [upload=64958]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 22 01:32:47.301770382 (81h9m22.865093389s) [seg_pos={0 0}, upload=64958]: Begin upload segment... Jun 22 01:32:47.301823796 (81h9m22.865146799s) [seg_pos={0 0}, upload=64958]: Done begin upload segment. Jun 22 01:32:47.301855055 (81h9m22.865178087s) [noise=true , address=104.36.82.43:6841 , piece_id=PIHZMSTO3CII7KGDLGSAILTDRSLMEBVOLBCRCZJULMCFISXIZWXA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.301902600 (81h9m22.865225606s) [noise=true , address=104.36.82.42:6811 , piece_id=A5BFEHZ3WGGK53IZ7MKI6N4CVXHOORF6L47NR3UFDCAX7LBHRHVQ, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.302234027 (81h9m22.865557029s) [noise=true , address=104.36.82.51:4084 , piece_id=D43FA7BFE54KPCWGDGDME2EGSWM7PONOVPD3DN2XY42MJFOKVDIQ, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.302496354 (81h9m22.865819363s) [noise=true , address=79.127.219.38:4023 , piece_id=BWLRCXXHBMBTNCZEVKKIIS5PB35PDQXQXCLIQB7GBLEIOWWUFBFA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.302577370 (81h9m22.865900378s) [noise=true , address=104.36.82.52:4089 , piece_id=JSOEKLNWA3W3LOVKI3EGGKJYJ6J3UZY6KGABKQPQPXTIOOPMKVHA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.302917829 (81h9m22.866240835s) [noise=true , address=104.36.82.53:4122 , piece_id=HZZQBIKHQWLUE5EHK4KSKJGIUEG7D44RQASUJHCYXX6BICCGX6EA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.303087165 (81h9m22.866410168s) [noise=true , address=77.90.31.14:4689 , piece_id=Q3HDS3BCMYDTODFRMU4XFRDCQK2QZE3ZMXVLUX4M54PDEBS7RR6A, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.303250714 (81h9m22.866573717s) [noise=true , address=79.127.205.234:5601 , piece_id=V7PMXBP2VFWBSZDG4GTLBD435T6F5DT7A43L4GQLQ5BDDTJ4WY6A, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.303579249 (81h9m22.866902255s) [noise=true , address=104.36.82.12:6158 , piece_id=LJZEOJVJCJAR2BRILIJI3ZVDGYMPBBTP2G3HCTN3MHNS4KYDPASQ, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.303599776 (81h9m22.866922784s) [noise=true , address=104.36.82.9:6023 , piece_id=SISFAUU4RPF57OLGWM2BWQWE6GRUSCMORBYHWEXETNAVM2EEMO6Q, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.304005346 (81h9m22.867328346s) [noise=true , address=45.91.116.12:5242 , piece_id=6ABT5IPYGSLFFJMBYMBYRU4EYIXABUAUHG3OVILOHBIUDS3BP27A, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.304388850 (81h9m22.867711851s) [noise=true , address=64.49.13.137:4500 , piece_id=FAE2RXD57IRP5FXA2XR6THXFW27U7IWAZ4JDV5M2WRPXKUGZ4VIA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.304881529 (81h9m22.868204536s) [noise=true , address=79.127.220.101:6048 , piece_id=AN7UQUXB42T2YXEH7ZJRZUDIW47M3RWUEOCL2VEBOGH74BZF3YKA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.305249860 (81h9m22.86857287s) [noise=true , address=104.36.82.54:4165 , piece_id=4CVOABPCWC7J7SRHZB6IN2WJJ6KGWUQ66ELUIZIFA67ETLQ7JEUQ, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.305604108 (81h9m22.868927112s) [noise=true , address=104.36.82.15:6289 , piece_id=GWL3P55BIK4IJEL5YFZKI7JCA4LADYOOHUSIQUNGJEKBJOTKMP5A, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.305930423 (81h9m22.869253425s) [noise=true , address=64.49.13.139:4476 , piece_id=P7UKBNNZGI52B3I6WKTV6MVC262FYNKYYPJUI5G47UO5S4MU7PWA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.306355311 (81h9m22.869678319s) [noise=true , address=104.36.82.55:4213 , piece_id=X4IHS2XSPH52LIMNUYT7KGWMAPUH3TEMJOEEVB3JV5N45SEOH7YA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.306788243 (81h9m22.870111252s) [noise=true , address=104.36.82.22:6539 , piece_id=GHQ2MVSTDGVYLUKLV23D6YWRR4PP7UUYHCUEQ4WWSWEMTBFO3P4A, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.392174770 (81h9m22.955497773s) [upload=64958]: Next returned nil segment Jun 22 01:32:47.392195767 (81h9m22.955518775s) [upload=64958]: Waiting for error group managing segments... Jun 22 01:32:47.480767633 (81h9m23.044090623s) [noise=true , address=104.36.82.22:6539 , piece_id=GHQ2MVSTDGVYLUKLV23D6YWRR4PP7UUYHCUEQ4WWSWEMTBFO3P4A, seg_pos={0 0}, upload=64958]: Done uploading piece. err: ecclient: upload failed (node:1QL6w8DSFR6CxDdnJmDuyVZxp6JU55VKiJofBZRfQi7q31i8Jc, address:104.36.82.22:6539): protocol: storage node overloaded, request limit: 2000 Jun 22 01:32:47.585663142 (81h9m23.148986155s) [noise=true , address=79.127.220.101:6048 , piece_id=AN7UQUXB42T2YXEH7ZJRZUDIW47M3RWUEOCL2VEBOGH74BZF3YKA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:47.640263002 (81h9m23.203586013s) [noise=true , address=45.91.116.12:5242 , piece_id=6ABT5IPYGSLFFJMBYMBYRU4EYIXABUAUHG3OVILOHBIUDS3BP27A, seg_pos={0 0}, upload=64958]: Done uploading piece. err: ecclient: upload failed (node:12JsFL62jgXmza8Md8eQ9k8cyw5V9Vz2RzN4y85ZvgHZjwrq4vQ, address:45.91.116.12:5242): protocol: storage node overloaded, request limit: 2000 Jun 22 01:32:47.984127275 (81h9m23.54745028s) [noise=true , address=64.49.13.137:4500 , piece_id=FAE2RXD57IRP5FXA2XR6THXFW27U7IWAZ4JDV5M2WRPXKUGZ4VIA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:48.100255249 (81h9m23.663578241s) [noise=true , address=77.90.31.14:4689 , piece_id=Q3HDS3BCMYDTODFRMU4XFRDCQK2QZE3ZMXVLUX4M54PDEBS7RR6A, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:48.221764035 (81h9m23.785087034s) [noise=true , address=104.36.82.54:4165 , piece_id=4CVOABPCWC7J7SRHZB6IN2WJJ6KGWUQ66ELUIZIFA67ETLQ7JEUQ, seg_pos={0 0}, upload=64958]: Done uploading piece. err: ecclient: upload failed (node:12wGUMXVEubEtki4qGnjGEsbYFB1iUauZcu3oDRKrCQsJMV3bV2, address:104.36.82.54:4165): protocol: storage node overloaded, request limit: 2000 Jun 22 01:32:48.509996527 (81h9m24.073319547s) [noise=true , address=104.36.82.9:6023 , piece_id=SISFAUU4RPF57OLGWM2BWQWE6GRUSCMORBYHWEXETNAVM2EEMO6Q, seg_pos={0 0}, upload=64958]: Done uploading piece. err: ecclient: upload failed (node:1j2S9tgWpLwmyvXZSq2qKKcbaC8r7LkzfDCaMxCejAni4CST1g, address:104.36.82.9:6023): protocol: storage node overloaded, request limit: 2000 Jun 22 01:32:48.510513169 (81h9m24.073836177s) [noise=true , address=104.36.82.42:6811 , piece_id=A5BFEHZ3WGGK53IZ7MKI6N4CVXHOORF6L47NR3UFDCAX7LBHRHVQ, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:48.565245605 (81h9m24.128568606s) [noise=true , address=79.127.205.234:5601 , piece_id=V7PMXBP2VFWBSZDG4GTLBD435T6F5DT7A43L4GQLQ5BDDTJ4WY6A, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:49.979979824 (81h9m25.543302825s) [noise=true , address=104.36.82.15:6289 , piece_id=GWL3P55BIK4IJEL5YFZKI7JCA4LADYOOHUSIQUNGJEKBJOTKMP5A, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:50.463750280 (81h9m26.02707328s) [noise=true , address=104.36.82.51:4084 , piece_id=D43FA7BFE54KPCWGDGDME2EGSWM7PONOVPD3DN2XY42MJFOKVDIQ, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:51.480599795 (81h9m27.043922798s) [noise=true , address=79.127.219.38:4023 , piece_id=BWLRCXXHBMBTNCZEVKKIIS5PB35PDQXQXCLIQB7GBLEIOWWUFBFA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:52.941200625 (81h9m28.504523627s) [noise=true , address=104.36.82.43:6841 , piece_id=PIHZMSTO3CII7KGDLGSAILTDRSLMEBVOLBCRCZJULMCFISXIZWXA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:58.281803598 (81h9m33.845126612s) [noise=true , address=64.49.13.139:4476 , piece_id=P7UKBNNZGI52B3I6WKTV6MVC262FYNKYYPJUI5G47UO5S4MU7PWA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:33:00.277271877 (81h9m35.840594867s) [noise=true , address=104.36.82.55:4213 , piece_id=X4IHS2XSPH52LIMNUYT7KGWMAPUH3TEMJOEEVB3JV5N45SEOH7YA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:33:09.435339213 (81h9m44.998662206s) [noise=true , address=104.36.82.52:4089 , piece_id=JSOEKLNWA3W3LOVKI3EGGKJYJ6J3UZY6KGABKQPQPXTIOOPMKVHA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:33:31.472868445 (81h10m7.03619143s) [noise=true , address=104.36.82.53:4122 , piece_id=HZZQBIKHQWLUE5EHK4KSKJGIUEG7D44RQASUJHCYXX6BICCGX6EA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:33:47.235404653 (81h10m22.798727642s) [noise=true , address=104.36.82.12:6158 , piece_id=LJZEOJVJCJAR2BRILIJI3ZVDGYMPBBTP2G3HCTN3MHNS4KYDPASQ, seg_pos={0 0}, upload=64958]: Done uploading piece. err: ecclient: upload failed (node:1SjM6Utm2YizeUbrBUD9Z7etE1nLadW5Fa1zYP7MioPeeqhdJ7, address:104.36.82.12:6158): protocol: expected piece hash; context deadline exceeded; EOF Jun 22 01:33:47.235448297 (81h10m22.798771306s) [seg_pos={0 0}, upload=64958]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jun 22 01:33:47.265442393 (81h10m22.828765401s) [upload=64958]: Done uploading segments... Jun 22 01:33:47.265476139 (81h10m22.82879914s) [upload=64958]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration:* 14.325525009s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [..............................................X.............] duration: 13.264913007s FAIL (2025-06-22 01:42:25 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== oad=64949]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jun 22 01:32:13.674692303 (81h8m49.238015311s) [upload=64949]: Done uploading segments... Jun 22 01:32:13.674727219 (81h8m49.238050233s) [upload=64949]: Upload finished. err: <nil> Jun 22 01:32:47.266407995 (81h9m22.829731002s) [upload=64958]: Starting upload Jun 22 01:32:47.266524767 (81h9m22.829847771s) [upload=64958]: Done starting upload Jun 22 01:32:47.266813109 (81h9m22.830136116s) [upload=64958]: Uploading segments... Jun 22 01:32:47.266969526 (81h9m22.830292526s) [upload=64958]: Got next segment. Inline: false Jun 22 01:32:47.267075475 (81h9m22.830398475s) [upload=64958]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 22 01:32:47.267081679 (81h9m22.830404667s) [upload=64958]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 22 01:32:47.301770382 (81h9m22.865093389s) [seg_pos={0 0}, upload=64958]: Begin upload segment... Jun 22 01:32:47.301823796 (81h9m22.865146799s) [seg_pos={0 0}, upload=64958]: Done begin upload segment. Jun 22 01:32:47.301855055 (81h9m22.865178087s) [noise=true , address=104.36.82.43:6841 , piece_id=PIHZMSTO3CII7KGDLGSAILTDRSLMEBVOLBCRCZJULMCFISXIZWXA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.301902600 (81h9m22.865225606s) [noise=true , address=104.36.82.42:6811 , piece_id=A5BFEHZ3WGGK53IZ7MKI6N4CVXHOORF6L47NR3UFDCAX7LBHRHVQ, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.302234027 (81h9m22.865557029s) [noise=true , address=104.36.82.51:4084 , piece_id=D43FA7BFE54KPCWGDGDME2EGSWM7PONOVPD3DN2XY42MJFOKVDIQ, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.302496354 (81h9m22.865819363s) [noise=true , address=79.127.219.38:4023 , piece_id=BWLRCXXHBMBTNCZEVKKIIS5PB35PDQXQXCLIQB7GBLEIOWWUFBFA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.302577370 (81h9m22.865900378s) [noise=true , address=104.36.82.52:4089 , piece_id=JSOEKLNWA3W3LOVKI3EGGKJYJ6J3UZY6KGABKQPQPXTIOOPMKVHA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.302917829 (81h9m22.866240835s) [noise=true , address=104.36.82.53:4122 , piece_id=HZZQBIKHQWLUE5EHK4KSKJGIUEG7D44RQASUJHCYXX6BICCGX6EA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.303087165 (81h9m22.866410168s) [noise=true , address=77.90.31.14:4689 , piece_id=Q3HDS3BCMYDTODFRMU4XFRDCQK2QZE3ZMXVLUX4M54PDEBS7RR6A, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.303250714 (81h9m22.866573717s) [noise=true , address=79.127.205.234:5601 , piece_id=V7PMXBP2VFWBSZDG4GTLBD435T6F5DT7A43L4GQLQ5BDDTJ4WY6A, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.303579249 (81h9m22.866902255s) [noise=true , address=104.36.82.12:6158 , piece_id=LJZEOJVJCJAR2BRILIJI3ZVDGYMPBBTP2G3HCTN3MHNS4KYDPASQ, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.303599776 (81h9m22.866922784s) [noise=true , address=104.36.82.9:6023 , piece_id=SISFAUU4RPF57OLGWM2BWQWE6GRUSCMORBYHWEXETNAVM2EEMO6Q, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.304005346 (81h9m22.867328346s) [noise=true , address=45.91.116.12:5242 , piece_id=6ABT5IPYGSLFFJMBYMBYRU4EYIXABUAUHG3OVILOHBIUDS3BP27A, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.304388850 (81h9m22.867711851s) [noise=true , address=64.49.13.137:4500 , piece_id=FAE2RXD57IRP5FXA2XR6THXFW27U7IWAZ4JDV5M2WRPXKUGZ4VIA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.304881529 (81h9m22.868204536s) [noise=true , address=79.127.220.101:6048 , piece_id=AN7UQUXB42T2YXEH7ZJRZUDIW47M3RWUEOCL2VEBOGH74BZF3YKA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.305249860 (81h9m22.86857287s) [noise=true , address=104.36.82.54:4165 , piece_id=4CVOABPCWC7J7SRHZB6IN2WJJ6KGWUQ66ELUIZIFA67ETLQ7JEUQ, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.305604108 (81h9m22.868927112s) [noise=true , address=104.36.82.15:6289 , piece_id=GWL3P55BIK4IJEL5YFZKI7JCA4LADYOOHUSIQUNGJEKBJOTKMP5A, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.305930423 (81h9m22.869253425s) [noise=true , address=64.49.13.139:4476 , piece_id=P7UKBNNZGI52B3I6WKTV6MVC262FYNKYYPJUI5G47UO5S4MU7PWA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.306355311 (81h9m22.869678319s) [noise=true , address=104.36.82.55:4213 , piece_id=X4IHS2XSPH52LIMNUYT7KGWMAPUH3TEMJOEEVB3JV5N45SEOH7YA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.306788243 (81h9m22.870111252s) [noise=true , address=104.36.82.22:6539 , piece_id=GHQ2MVSTDGVYLUKLV23D6YWRR4PP7UUYHCUEQ4WWSWEMTBFO3P4A, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.392174770 (81h9m22.955497773s) [upload=64958]: Next returned nil segment Jun 22 01:32:47.392195767 (81h9m22.955518775s) [upload=64958]: Waiting for error group managing segments... Jun 22 01:32:47.480767633 (81h9m23.044090623s) [noise=true , address=104.36.82.22:6539 , piece_id=GHQ2MVSTDGVYLUKLV23D6YWRR4PP7UUYHCUEQ4WWSWEMTBFO3P4A, seg_pos={0 0}, upload=64958]: Done uploading piece. err: ecclient: upload failed (node:1QL6w8DSFR6CxDdnJmDuyVZxp6JU55VKiJofBZRfQi7q31i8Jc, address:104.36.82.22:6539): protocol: storage node overloaded, request limit: 2000 Jun 22 01:32:47.585663142 (81h9m23.148986155s) [noise=true , address=79.127.220.101:6048 , piece_id=AN7UQUXB42T2YXEH7ZJRZUDIW47M3RWUEOCL2VEBOGH74BZF3YKA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:47.640263002 (81h9m23.203586013s) [noise=true , address=45.91.116.12:5242 , piece_id=6ABT5IPYGSLFFJMBYMBYRU4EYIXABUAUHG3OVILOHBIUDS3BP27A, seg_pos={0 0}, upload=64958]: Done uploading piece. err: ecclient: upload failed (node:12JsFL62jgXmza8Md8eQ9k8cyw5V9Vz2RzN4y85ZvgHZjwrq4vQ, address:45.91.116.12:5242): protocol: storage node overloaded, request limit: 2000 Jun 22 01:32:47.984127275 (81h9m23.54745028s) [noise=true , address=64.49.13.137:4500 , piece_id=FAE2RXD57IRP5FXA2XR6THXFW27U7IWAZ4JDV5M2WRPXKUGZ4VIA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:48.100255249 (81h9m23.663578241s) [noise=true , address=77.90.31.14:4689 , piece_id=Q3HDS3BCMYDTODFRMU4XFRDCQK2QZE3ZMXVLUX4M54PDEBS7RR6A, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:48.221764035 (81h9m23.785087034s) [noise=true , address=104.36.82.54:4165 , piece_id=4CVOABPCWC7J7SRHZB6IN2WJJ6KGWUQ66ELUIZIFA67ETLQ7JEUQ, seg_pos={0 0}, upload=64958]: Done uploading piece. err: ecclient: upload failed (node:12wGUMXVEubEtki4qGnjGEsbYFB1iUauZcu3oDRKrCQsJMV3bV2, address:104.36.82.54:4165): protocol: storage node overloaded, request limit: 2000 Jun 22 01:32:48.509996527 (81h9m24.073319547s) [noise=true , address=104.36.82.9:6023 , piece_id=SISFAUU4RPF57OLGWM2BWQWE6GRUSCMORBYHWEXETNAVM2EEMO6Q, seg_pos={0 0}, upload=64958]: Done uploading piece. err: ecclient: upload failed (node:1j2S9tgWpLwmyvXZSq2qKKcbaC8r7LkzfDCaMxCejAni4CST1g, address:104.36.82.9:6023): protocol: storage node overloaded, request limit: 2000 Jun 22 01:32:48.510513169 (81h9m24.073836177s) [noise=true , address=104.36.82.42:6811 , piece_id=A5BFEHZ3WGGK53IZ7MKI6N4CVXHOORF6L47NR3UFDCAX7LBHRHVQ, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:48.565245605 (81h9m24.128568606s) [noise=true , address=79.127.205.234:5601 , piece_id=V7PMXBP2VFWBSZDG4GTLBD435T6F5DT7A43L4GQLQ5BDDTJ4WY6A, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:49.979979824 (81h9m25.543302825s) [noise=true , address=104.36.82.15:6289 , piece_id=GWL3P55BIK4IJEL5YFZKI7JCA4LADYOOHUSIQUNGJEKBJOTKMP5A, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:50.463750280 (81h9m26.02707328s) [noise=true , address=104.36.82.51:4084 , piece_id=D43FA7BFE54KPCWGDGDME2EGSWM7PONOVPD3DN2XY42MJFOKVDIQ, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:51.480599795 (81h9m27.043922798s) [noise=true , address=79.127.219.38:4023 , piece_id=BWLRCXXHBMBTNCZEVKKIIS5PB35PDQXQXCLIQB7GBLEIOWWUFBFA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:52.941200625 (81h9m28.504523627s) [noise=true , address=104.36.82.43:6841 , piece_id=PIHZMSTO3CII7KGDLGSAILTDRSLMEBVOLBCRCZJULMCFISXIZWXA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:58.281803598 (81h9m33.845126612s) [noise=true , address=64.49.13.139:4476 , piece_id=P7UKBNNZGI52B3I6WKTV6MVC262FYNKYYPJUI5G47UO5S4MU7PWA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:33:00.277271877 (81h9m35.840594867s) [noise=true , address=104.36.82.55:4213 , piece_id=X4IHS2XSPH52LIMNUYT7KGWMAPUH3TEMJOEEVB3JV5N45SEOH7YA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:33:09.435339213 (81h9m44.998662206s) [noise=true , address=104.36.82.52:4089 , piece_id=JSOEKLNWA3W3LOVKI3EGGKJYJ6J3UZY6KGABKQPQPXTIOOPMKVHA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:33:31.472868445 (81h10m7.03619143s) [noise=true , address=104.36.82.53:4122 , piece_id=HZZQBIKHQWLUE5EHK4KSKJGIUEG7D44RQASUJHCYXX6BICCGX6EA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:33:47.235404653 (81h10m22.798727642s) [noise=true , address=104.36.82.12:6158 , piece_id=LJZEOJVJCJAR2BRILIJI3ZVDGYMPBBTP2G3HCTN3MHNS4KYDPASQ, seg_pos={0 0}, upload=64958]: Done uploading piece. err: ecclient: upload failed (node:1SjM6Utm2YizeUbrBUD9Z7etE1nLadW5Fa1zYP7MioPeeqhdJ7, address:104.36.82.12:6158): protocol: expected piece hash; context deadline exceeded; EOF Jun 22 01:33:47.235448297 (81h10m22.798771306s) [seg_pos={0 0}, upload=64958]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jun 22 01:33:47.265442393 (81h10m22.828765401s) [upload=64958]: Done uploading segments... Jun 22 01:33:47.265476139 (81h10m22.82879914s) [upload=64958]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration:* 13.264942434s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [...............................................X............] duration: 12.396198231s FAIL (2025-06-22 01:41:37 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== oad=64949]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jun 22 01:32:13.674692303 (81h8m49.238015311s) [upload=64949]: Done uploading segments... Jun 22 01:32:13.674727219 (81h8m49.238050233s) [upload=64949]: Upload finished. err: <nil> Jun 22 01:32:47.266407995 (81h9m22.829731002s) [upload=64958]: Starting upload Jun 22 01:32:47.266524767 (81h9m22.829847771s) [upload=64958]: Done starting upload Jun 22 01:32:47.266813109 (81h9m22.830136116s) [upload=64958]: Uploading segments... Jun 22 01:32:47.266969526 (81h9m22.830292526s) [upload=64958]: Got next segment. Inline: false Jun 22 01:32:47.267075475 (81h9m22.830398475s) [upload=64958]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 22 01:32:47.267081679 (81h9m22.830404667s) [upload=64958]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 22 01:32:47.301770382 (81h9m22.865093389s) [seg_pos={0 0}, upload=64958]: Begin upload segment... Jun 22 01:32:47.301823796 (81h9m22.865146799s) [seg_pos={0 0}, upload=64958]: Done begin upload segment. Jun 22 01:32:47.301855055 (81h9m22.865178087s) [noise=true , address=104.36.82.43:6841 , piece_id=PIHZMSTO3CII7KGDLGSAILTDRSLMEBVOLBCRCZJULMCFISXIZWXA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.301902600 (81h9m22.865225606s) [noise=true , address=104.36.82.42:6811 , piece_id=A5BFEHZ3WGGK53IZ7MKI6N4CVXHOORF6L47NR3UFDCAX7LBHRHVQ, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.302234027 (81h9m22.865557029s) [noise=true , address=104.36.82.51:4084 , piece_id=D43FA7BFE54KPCWGDGDME2EGSWM7PONOVPD3DN2XY42MJFOKVDIQ, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.302496354 (81h9m22.865819363s) [noise=true , address=79.127.219.38:4023 , piece_id=BWLRCXXHBMBTNCZEVKKIIS5PB35PDQXQXCLIQB7GBLEIOWWUFBFA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.302577370 (81h9m22.865900378s) [noise=true , address=104.36.82.52:4089 , piece_id=JSOEKLNWA3W3LOVKI3EGGKJYJ6J3UZY6KGABKQPQPXTIOOPMKVHA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.302917829 (81h9m22.866240835s) [noise=true , address=104.36.82.53:4122 , piece_id=HZZQBIKHQWLUE5EHK4KSKJGIUEG7D44RQASUJHCYXX6BICCGX6EA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.303087165 (81h9m22.866410168s) [noise=true , address=77.90.31.14:4689 , piece_id=Q3HDS3BCMYDTODFRMU4XFRDCQK2QZE3ZMXVLUX4M54PDEBS7RR6A, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.303250714 (81h9m22.866573717s) [noise=true , address=79.127.205.234:5601 , piece_id=V7PMXBP2VFWBSZDG4GTLBD435T6F5DT7A43L4GQLQ5BDDTJ4WY6A, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.303579249 (81h9m22.866902255s) [noise=true , address=104.36.82.12:6158 , piece_id=LJZEOJVJCJAR2BRILIJI3ZVDGYMPBBTP2G3HCTN3MHNS4KYDPASQ, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.303599776 (81h9m22.866922784s) [noise=true , address=104.36.82.9:6023 , piece_id=SISFAUU4RPF57OLGWM2BWQWE6GRUSCMORBYHWEXETNAVM2EEMO6Q, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.304005346 (81h9m22.867328346s) [noise=true , address=45.91.116.12:5242 , piece_id=6ABT5IPYGSLFFJMBYMBYRU4EYIXABUAUHG3OVILOHBIUDS3BP27A, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.304388850 (81h9m22.867711851s) [noise=true , address=64.49.13.137:4500 , piece_id=FAE2RXD57IRP5FXA2XR6THXFW27U7IWAZ4JDV5M2WRPXKUGZ4VIA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.304881529 (81h9m22.868204536s) [noise=true , address=79.127.220.101:6048 , piece_id=AN7UQUXB42T2YXEH7ZJRZUDIW47M3RWUEOCL2VEBOGH74BZF3YKA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.305249860 (81h9m22.86857287s) [noise=true , address=104.36.82.54:4165 , piece_id=4CVOABPCWC7J7SRHZB6IN2WJJ6KGWUQ66ELUIZIFA67ETLQ7JEUQ, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.305604108 (81h9m22.868927112s) [noise=true , address=104.36.82.15:6289 , piece_id=GWL3P55BIK4IJEL5YFZKI7JCA4LADYOOHUSIQUNGJEKBJOTKMP5A, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.305930423 (81h9m22.869253425s) [noise=true , address=64.49.13.139:4476 , piece_id=P7UKBNNZGI52B3I6WKTV6MVC262FYNKYYPJUI5G47UO5S4MU7PWA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.306355311 (81h9m22.869678319s) [noise=true , address=104.36.82.55:4213 , piece_id=X4IHS2XSPH52LIMNUYT7KGWMAPUH3TEMJOEEVB3JV5N45SEOH7YA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.306788243 (81h9m22.870111252s) [noise=true , address=104.36.82.22:6539 , piece_id=GHQ2MVSTDGVYLUKLV23D6YWRR4PP7UUYHCUEQ4WWSWEMTBFO3P4A, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.392174770 (81h9m22.955497773s) [upload=64958]: Next returned nil segment Jun 22 01:32:47.392195767 (81h9m22.955518775s) [upload=64958]: Waiting for error group managing segments... Jun 22 01:32:47.480767633 (81h9m23.044090623s) [noise=true , address=104.36.82.22:6539 , piece_id=GHQ2MVSTDGVYLUKLV23D6YWRR4PP7UUYHCUEQ4WWSWEMTBFO3P4A, seg_pos={0 0}, upload=64958]: Done uploading piece. err: ecclient: upload failed (node:1QL6w8DSFR6CxDdnJmDuyVZxp6JU55VKiJofBZRfQi7q31i8Jc, address:104.36.82.22:6539): protocol: storage node overloaded, request limit: 2000 Jun 22 01:32:47.585663142 (81h9m23.148986155s) [noise=true , address=79.127.220.101:6048 , piece_id=AN7UQUXB42T2YXEH7ZJRZUDIW47M3RWUEOCL2VEBOGH74BZF3YKA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:47.640263002 (81h9m23.203586013s) [noise=true , address=45.91.116.12:5242 , piece_id=6ABT5IPYGSLFFJMBYMBYRU4EYIXABUAUHG3OVILOHBIUDS3BP27A, seg_pos={0 0}, upload=64958]: Done uploading piece. err: ecclient: upload failed (node:12JsFL62jgXmza8Md8eQ9k8cyw5V9Vz2RzN4y85ZvgHZjwrq4vQ, address:45.91.116.12:5242): protocol: storage node overloaded, request limit: 2000 Jun 22 01:32:47.984127275 (81h9m23.54745028s) [noise=true , address=64.49.13.137:4500 , piece_id=FAE2RXD57IRP5FXA2XR6THXFW27U7IWAZ4JDV5M2WRPXKUGZ4VIA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:48.100255249 (81h9m23.663578241s) [noise=true , address=77.90.31.14:4689 , piece_id=Q3HDS3BCMYDTODFRMU4XFRDCQK2QZE3ZMXVLUX4M54PDEBS7RR6A, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:48.221764035 (81h9m23.785087034s) [noise=true , address=104.36.82.54:4165 , piece_id=4CVOABPCWC7J7SRHZB6IN2WJJ6KGWUQ66ELUIZIFA67ETLQ7JEUQ, seg_pos={0 0}, upload=64958]: Done uploading piece. err: ecclient: upload failed (node:12wGUMXVEubEtki4qGnjGEsbYFB1iUauZcu3oDRKrCQsJMV3bV2, address:104.36.82.54:4165): protocol: storage node overloaded, request limit: 2000 Jun 22 01:32:48.509996527 (81h9m24.073319547s) [noise=true , address=104.36.82.9:6023 , piece_id=SISFAUU4RPF57OLGWM2BWQWE6GRUSCMORBYHWEXETNAVM2EEMO6Q, seg_pos={0 0}, upload=64958]: Done uploading piece. err: ecclient: upload failed (node:1j2S9tgWpLwmyvXZSq2qKKcbaC8r7LkzfDCaMxCejAni4CST1g, address:104.36.82.9:6023): protocol: storage node overloaded, request limit: 2000 Jun 22 01:32:48.510513169 (81h9m24.073836177s) [noise=true , address=104.36.82.42:6811 , piece_id=A5BFEHZ3WGGK53IZ7MKI6N4CVXHOORF6L47NR3UFDCAX7LBHRHVQ, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:48.565245605 (81h9m24.128568606s) [noise=true , address=79.127.205.234:5601 , piece_id=V7PMXBP2VFWBSZDG4GTLBD435T6F5DT7A43L4GQLQ5BDDTJ4WY6A, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:49.979979824 (81h9m25.543302825s) [noise=true , address=104.36.82.15:6289 , piece_id=GWL3P55BIK4IJEL5YFZKI7JCA4LADYOOHUSIQUNGJEKBJOTKMP5A, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:50.463750280 (81h9m26.02707328s) [noise=true , address=104.36.82.51:4084 , piece_id=D43FA7BFE54KPCWGDGDME2EGSWM7PONOVPD3DN2XY42MJFOKVDIQ, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:51.480599795 (81h9m27.043922798s) [noise=true , address=79.127.219.38:4023 , piece_id=BWLRCXXHBMBTNCZEVKKIIS5PB35PDQXQXCLIQB7GBLEIOWWUFBFA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:52.941200625 (81h9m28.504523627s) [noise=true , address=104.36.82.43:6841 , piece_id=PIHZMSTO3CII7KGDLGSAILTDRSLMEBVOLBCRCZJULMCFISXIZWXA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:58.281803598 (81h9m33.845126612s) [noise=true , address=64.49.13.139:4476 , piece_id=P7UKBNNZGI52B3I6WKTV6MVC262FYNKYYPJUI5G47UO5S4MU7PWA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:33:00.277271877 (81h9m35.840594867s) [noise=true , address=104.36.82.55:4213 , piece_id=X4IHS2XSPH52LIMNUYT7KGWMAPUH3TEMJOEEVB3JV5N45SEOH7YA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:33:09.435339213 (81h9m44.998662206s) [noise=true , address=104.36.82.52:4089 , piece_id=JSOEKLNWA3W3LOVKI3EGGKJYJ6J3UZY6KGABKQPQPXTIOOPMKVHA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:33:31.472868445 (81h10m7.03619143s) [noise=true , address=104.36.82.53:4122 , piece_id=HZZQBIKHQWLUE5EHK4KSKJGIUEG7D44RQASUJHCYXX6BICCGX6EA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:33:47.235404653 (81h10m22.798727642s) [noise=true , address=104.36.82.12:6158 , piece_id=LJZEOJVJCJAR2BRILIJI3ZVDGYMPBBTP2G3HCTN3MHNS4KYDPASQ, seg_pos={0 0}, upload=64958]: Done uploading piece. err: ecclient: upload failed (node:1SjM6Utm2YizeUbrBUD9Z7etE1nLadW5Fa1zYP7MioPeeqhdJ7, address:104.36.82.12:6158): protocol: expected piece hash; context deadline exceeded; EOF Jun 22 01:33:47.235448297 (81h10m22.798771306s) [seg_pos={0 0}, upload=64958]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jun 22 01:33:47.265442393 (81h10m22.828765401s) [upload=64958]: Done uploading segments... Jun 22 01:33:47.265476139 (81h10m22.82879914s) [upload=64958]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration:* 12.39622384s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [................................................X...........] duration: 25.423644721s FAIL (2025-06-22 01:40:41 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== oad=64949]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jun 22 01:32:13.674692303 (81h8m49.238015311s) [upload=64949]: Done uploading segments... Jun 22 01:32:13.674727219 (81h8m49.238050233s) [upload=64949]: Upload finished. err: <nil> Jun 22 01:32:47.266407995 (81h9m22.829731002s) [upload=64958]: Starting upload Jun 22 01:32:47.266524767 (81h9m22.829847771s) [upload=64958]: Done starting upload Jun 22 01:32:47.266813109 (81h9m22.830136116s) [upload=64958]: Uploading segments... Jun 22 01:32:47.266969526 (81h9m22.830292526s) [upload=64958]: Got next segment. Inline: false Jun 22 01:32:47.267075475 (81h9m22.830398475s) [upload=64958]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 22 01:32:47.267081679 (81h9m22.830404667s) [upload=64958]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 22 01:32:47.301770382 (81h9m22.865093389s) [seg_pos={0 0}, upload=64958]: Begin upload segment... Jun 22 01:32:47.301823796 (81h9m22.865146799s) [seg_pos={0 0}, upload=64958]: Done begin upload segment. Jun 22 01:32:47.301855055 (81h9m22.865178087s) [noise=true , address=104.36.82.43:6841 , piece_id=PIHZMSTO3CII7KGDLGSAILTDRSLMEBVOLBCRCZJULMCFISXIZWXA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.301902600 (81h9m22.865225606s) [noise=true , address=104.36.82.42:6811 , piece_id=A5BFEHZ3WGGK53IZ7MKI6N4CVXHOORF6L47NR3UFDCAX7LBHRHVQ, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.302234027 (81h9m22.865557029s) [noise=true , address=104.36.82.51:4084 , piece_id=D43FA7BFE54KPCWGDGDME2EGSWM7PONOVPD3DN2XY42MJFOKVDIQ, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.302496354 (81h9m22.865819363s) [noise=true , address=79.127.219.38:4023 , piece_id=BWLRCXXHBMBTNCZEVKKIIS5PB35PDQXQXCLIQB7GBLEIOWWUFBFA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.302577370 (81h9m22.865900378s) [noise=true , address=104.36.82.52:4089 , piece_id=JSOEKLNWA3W3LOVKI3EGGKJYJ6J3UZY6KGABKQPQPXTIOOPMKVHA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.302917829 (81h9m22.866240835s) [noise=true , address=104.36.82.53:4122 , piece_id=HZZQBIKHQWLUE5EHK4KSKJGIUEG7D44RQASUJHCYXX6BICCGX6EA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.303087165 (81h9m22.866410168s) [noise=true , address=77.90.31.14:4689 , piece_id=Q3HDS3BCMYDTODFRMU4XFRDCQK2QZE3ZMXVLUX4M54PDEBS7RR6A, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.303250714 (81h9m22.866573717s) [noise=true , address=79.127.205.234:5601 , piece_id=V7PMXBP2VFWBSZDG4GTLBD435T6F5DT7A43L4GQLQ5BDDTJ4WY6A, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.303579249 (81h9m22.866902255s) [noise=true , address=104.36.82.12:6158 , piece_id=LJZEOJVJCJAR2BRILIJI3ZVDGYMPBBTP2G3HCTN3MHNS4KYDPASQ, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.303599776 (81h9m22.866922784s) [noise=true , address=104.36.82.9:6023 , piece_id=SISFAUU4RPF57OLGWM2BWQWE6GRUSCMORBYHWEXETNAVM2EEMO6Q, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.304005346 (81h9m22.867328346s) [noise=true , address=45.91.116.12:5242 , piece_id=6ABT5IPYGSLFFJMBYMBYRU4EYIXABUAUHG3OVILOHBIUDS3BP27A, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.304388850 (81h9m22.867711851s) [noise=true , address=64.49.13.137:4500 , piece_id=FAE2RXD57IRP5FXA2XR6THXFW27U7IWAZ4JDV5M2WRPXKUGZ4VIA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.304881529 (81h9m22.868204536s) [noise=true , address=79.127.220.101:6048 , piece_id=AN7UQUXB42T2YXEH7ZJRZUDIW47M3RWUEOCL2VEBOGH74BZF3YKA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.305249860 (81h9m22.86857287s) [noise=true , address=104.36.82.54:4165 , piece_id=4CVOABPCWC7J7SRHZB6IN2WJJ6KGWUQ66ELUIZIFA67ETLQ7JEUQ, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.305604108 (81h9m22.868927112s) [noise=true , address=104.36.82.15:6289 , piece_id=GWL3P55BIK4IJEL5YFZKI7JCA4LADYOOHUSIQUNGJEKBJOTKMP5A, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.305930423 (81h9m22.869253425s) [noise=true , address=64.49.13.139:4476 , piece_id=P7UKBNNZGI52B3I6WKTV6MVC262FYNKYYPJUI5G47UO5S4MU7PWA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.306355311 (81h9m22.869678319s) [noise=true , address=104.36.82.55:4213 , piece_id=X4IHS2XSPH52LIMNUYT7KGWMAPUH3TEMJOEEVB3JV5N45SEOH7YA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.306788243 (81h9m22.870111252s) [noise=true , address=104.36.82.22:6539 , piece_id=GHQ2MVSTDGVYLUKLV23D6YWRR4PP7UUYHCUEQ4WWSWEMTBFO3P4A, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.392174770 (81h9m22.955497773s) [upload=64958]: Next returned nil segment Jun 22 01:32:47.392195767 (81h9m22.955518775s) [upload=64958]: Waiting for error group managing segments... Jun 22 01:32:47.480767633 (81h9m23.044090623s) [noise=true , address=104.36.82.22:6539 , piece_id=GHQ2MVSTDGVYLUKLV23D6YWRR4PP7UUYHCUEQ4WWSWEMTBFO3P4A, seg_pos={0 0}, upload=64958]: Done uploading piece. err: ecclient: upload failed (node:1QL6w8DSFR6CxDdnJmDuyVZxp6JU55VKiJofBZRfQi7q31i8Jc, address:104.36.82.22:6539): protocol: storage node overloaded, request limit: 2000 Jun 22 01:32:47.585663142 (81h9m23.148986155s) [noise=true , address=79.127.220.101:6048 , piece_id=AN7UQUXB42T2YXEH7ZJRZUDIW47M3RWUEOCL2VEBOGH74BZF3YKA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:47.640263002 (81h9m23.203586013s) [noise=true , address=45.91.116.12:5242 , piece_id=6ABT5IPYGSLFFJMBYMBYRU4EYIXABUAUHG3OVILOHBIUDS3BP27A, seg_pos={0 0}, upload=64958]: Done uploading piece. err: ecclient: upload failed (node:12JsFL62jgXmza8Md8eQ9k8cyw5V9Vz2RzN4y85ZvgHZjwrq4vQ, address:45.91.116.12:5242): protocol: storage node overloaded, request limit: 2000 Jun 22 01:32:47.984127275 (81h9m23.54745028s) [noise=true , address=64.49.13.137:4500 , piece_id=FAE2RXD57IRP5FXA2XR6THXFW27U7IWAZ4JDV5M2WRPXKUGZ4VIA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:48.100255249 (81h9m23.663578241s) [noise=true , address=77.90.31.14:4689 , piece_id=Q3HDS3BCMYDTODFRMU4XFRDCQK2QZE3ZMXVLUX4M54PDEBS7RR6A, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:48.221764035 (81h9m23.785087034s) [noise=true , address=104.36.82.54:4165 , piece_id=4CVOABPCWC7J7SRHZB6IN2WJJ6KGWUQ66ELUIZIFA67ETLQ7JEUQ, seg_pos={0 0}, upload=64958]: Done uploading piece. err: ecclient: upload failed (node:12wGUMXVEubEtki4qGnjGEsbYFB1iUauZcu3oDRKrCQsJMV3bV2, address:104.36.82.54:4165): protocol: storage node overloaded, request limit: 2000 Jun 22 01:32:48.509996527 (81h9m24.073319547s) [noise=true , address=104.36.82.9:6023 , piece_id=SISFAUU4RPF57OLGWM2BWQWE6GRUSCMORBYHWEXETNAVM2EEMO6Q, seg_pos={0 0}, upload=64958]: Done uploading piece. err: ecclient: upload failed (node:1j2S9tgWpLwmyvXZSq2qKKcbaC8r7LkzfDCaMxCejAni4CST1g, address:104.36.82.9:6023): protocol: storage node overloaded, request limit: 2000 Jun 22 01:32:48.510513169 (81h9m24.073836177s) [noise=true , address=104.36.82.42:6811 , piece_id=A5BFEHZ3WGGK53IZ7MKI6N4CVXHOORF6L47NR3UFDCAX7LBHRHVQ, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:48.565245605 (81h9m24.128568606s) [noise=true , address=79.127.205.234:5601 , piece_id=V7PMXBP2VFWBSZDG4GTLBD435T6F5DT7A43L4GQLQ5BDDTJ4WY6A, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:49.979979824 (81h9m25.543302825s) [noise=true , address=104.36.82.15:6289 , piece_id=GWL3P55BIK4IJEL5YFZKI7JCA4LADYOOHUSIQUNGJEKBJOTKMP5A, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:50.463750280 (81h9m26.02707328s) [noise=true , address=104.36.82.51:4084 , piece_id=D43FA7BFE54KPCWGDGDME2EGSWM7PONOVPD3DN2XY42MJFOKVDIQ, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:51.480599795 (81h9m27.043922798s) [noise=true , address=79.127.219.38:4023 , piece_id=BWLRCXXHBMBTNCZEVKKIIS5PB35PDQXQXCLIQB7GBLEIOWWUFBFA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:52.941200625 (81h9m28.504523627s) [noise=true , address=104.36.82.43:6841 , piece_id=PIHZMSTO3CII7KGDLGSAILTDRSLMEBVOLBCRCZJULMCFISXIZWXA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:58.281803598 (81h9m33.845126612s) [noise=true , address=64.49.13.139:4476 , piece_id=P7UKBNNZGI52B3I6WKTV6MVC262FYNKYYPJUI5G47UO5S4MU7PWA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:33:00.277271877 (81h9m35.840594867s) [noise=true , address=104.36.82.55:4213 , piece_id=X4IHS2XSPH52LIMNUYT7KGWMAPUH3TEMJOEEVB3JV5N45SEOH7YA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:33:09.435339213 (81h9m44.998662206s) [noise=true , address=104.36.82.52:4089 , piece_id=JSOEKLNWA3W3LOVKI3EGGKJYJ6J3UZY6KGABKQPQPXTIOOPMKVHA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:33:31.472868445 (81h10m7.03619143s) [noise=true , address=104.36.82.53:4122 , piece_id=HZZQBIKHQWLUE5EHK4KSKJGIUEG7D44RQASUJHCYXX6BICCGX6EA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:33:47.235404653 (81h10m22.798727642s) [noise=true , address=104.36.82.12:6158 , piece_id=LJZEOJVJCJAR2BRILIJI3ZVDGYMPBBTP2G3HCTN3MHNS4KYDPASQ, seg_pos={0 0}, upload=64958]: Done uploading piece. err: ecclient: upload failed (node:1SjM6Utm2YizeUbrBUD9Z7etE1nLadW5Fa1zYP7MioPeeqhdJ7, address:104.36.82.12:6158): protocol: expected piece hash; context deadline exceeded; EOF Jun 22 01:33:47.235448297 (81h10m22.798771306s) [seg_pos={0 0}, upload=64958]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jun 22 01:33:47.265442393 (81h10m22.828765401s) [upload=64958]: Done uploading segments... Jun 22 01:33:47.265476139 (81h10m22.82879914s) [upload=64958]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration:* 25.423676446s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.................................................X..........] duration: 24.677962764s FAIL (2025-06-22 01:40:04 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== oad=64949]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jun 22 01:32:13.674692303 (81h8m49.238015311s) [upload=64949]: Done uploading segments... Jun 22 01:32:13.674727219 (81h8m49.238050233s) [upload=64949]: Upload finished. err: <nil> Jun 22 01:32:47.266407995 (81h9m22.829731002s) [upload=64958]: Starting upload Jun 22 01:32:47.266524767 (81h9m22.829847771s) [upload=64958]: Done starting upload Jun 22 01:32:47.266813109 (81h9m22.830136116s) [upload=64958]: Uploading segments... Jun 22 01:32:47.266969526 (81h9m22.830292526s) [upload=64958]: Got next segment. Inline: false Jun 22 01:32:47.267075475 (81h9m22.830398475s) [upload=64958]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 22 01:32:47.267081679 (81h9m22.830404667s) [upload=64958]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 22 01:32:47.301770382 (81h9m22.865093389s) [seg_pos={0 0}, upload=64958]: Begin upload segment... Jun 22 01:32:47.301823796 (81h9m22.865146799s) [seg_pos={0 0}, upload=64958]: Done begin upload segment. Jun 22 01:32:47.301855055 (81h9m22.865178087s) [noise=true , address=104.36.82.43:6841 , piece_id=PIHZMSTO3CII7KGDLGSAILTDRSLMEBVOLBCRCZJULMCFISXIZWXA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.301902600 (81h9m22.865225606s) [noise=true , address=104.36.82.42:6811 , piece_id=A5BFEHZ3WGGK53IZ7MKI6N4CVXHOORF6L47NR3UFDCAX7LBHRHVQ, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.302234027 (81h9m22.865557029s) [noise=true , address=104.36.82.51:4084 , piece_id=D43FA7BFE54KPCWGDGDME2EGSWM7PONOVPD3DN2XY42MJFOKVDIQ, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.302496354 (81h9m22.865819363s) [noise=true , address=79.127.219.38:4023 , piece_id=BWLRCXXHBMBTNCZEVKKIIS5PB35PDQXQXCLIQB7GBLEIOWWUFBFA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.302577370 (81h9m22.865900378s) [noise=true , address=104.36.82.52:4089 , piece_id=JSOEKLNWA3W3LOVKI3EGGKJYJ6J3UZY6KGABKQPQPXTIOOPMKVHA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.302917829 (81h9m22.866240835s) [noise=true , address=104.36.82.53:4122 , piece_id=HZZQBIKHQWLUE5EHK4KSKJGIUEG7D44RQASUJHCYXX6BICCGX6EA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.303087165 (81h9m22.866410168s) [noise=true , address=77.90.31.14:4689 , piece_id=Q3HDS3BCMYDTODFRMU4XFRDCQK2QZE3ZMXVLUX4M54PDEBS7RR6A, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.303250714 (81h9m22.866573717s) [noise=true , address=79.127.205.234:5601 , piece_id=V7PMXBP2VFWBSZDG4GTLBD435T6F5DT7A43L4GQLQ5BDDTJ4WY6A, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.303579249 (81h9m22.866902255s) [noise=true , address=104.36.82.12:6158 , piece_id=LJZEOJVJCJAR2BRILIJI3ZVDGYMPBBTP2G3HCTN3MHNS4KYDPASQ, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.303599776 (81h9m22.866922784s) [noise=true , address=104.36.82.9:6023 , piece_id=SISFAUU4RPF57OLGWM2BWQWE6GRUSCMORBYHWEXETNAVM2EEMO6Q, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.304005346 (81h9m22.867328346s) [noise=true , address=45.91.116.12:5242 , piece_id=6ABT5IPYGSLFFJMBYMBYRU4EYIXABUAUHG3OVILOHBIUDS3BP27A, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.304388850 (81h9m22.867711851s) [noise=true , address=64.49.13.137:4500 , piece_id=FAE2RXD57IRP5FXA2XR6THXFW27U7IWAZ4JDV5M2WRPXKUGZ4VIA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.304881529 (81h9m22.868204536s) [noise=true , address=79.127.220.101:6048 , piece_id=AN7UQUXB42T2YXEH7ZJRZUDIW47M3RWUEOCL2VEBOGH74BZF3YKA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.305249860 (81h9m22.86857287s) [noise=true , address=104.36.82.54:4165 , piece_id=4CVOABPCWC7J7SRHZB6IN2WJJ6KGWUQ66ELUIZIFA67ETLQ7JEUQ, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.305604108 (81h9m22.868927112s) [noise=true , address=104.36.82.15:6289 , piece_id=GWL3P55BIK4IJEL5YFZKI7JCA4LADYOOHUSIQUNGJEKBJOTKMP5A, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.305930423 (81h9m22.869253425s) [noise=true , address=64.49.13.139:4476 , piece_id=P7UKBNNZGI52B3I6WKTV6MVC262FYNKYYPJUI5G47UO5S4MU7PWA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.306355311 (81h9m22.869678319s) [noise=true , address=104.36.82.55:4213 , piece_id=X4IHS2XSPH52LIMNUYT7KGWMAPUH3TEMJOEEVB3JV5N45SEOH7YA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.306788243 (81h9m22.870111252s) [noise=true , address=104.36.82.22:6539 , piece_id=GHQ2MVSTDGVYLUKLV23D6YWRR4PP7UUYHCUEQ4WWSWEMTBFO3P4A, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.392174770 (81h9m22.955497773s) [upload=64958]: Next returned nil segment Jun 22 01:32:47.392195767 (81h9m22.955518775s) [upload=64958]: Waiting for error group managing segments... Jun 22 01:32:47.480767633 (81h9m23.044090623s) [noise=true , address=104.36.82.22:6539 , piece_id=GHQ2MVSTDGVYLUKLV23D6YWRR4PP7UUYHCUEQ4WWSWEMTBFO3P4A, seg_pos={0 0}, upload=64958]: Done uploading piece. err: ecclient: upload failed (node:1QL6w8DSFR6CxDdnJmDuyVZxp6JU55VKiJofBZRfQi7q31i8Jc, address:104.36.82.22:6539): protocol: storage node overloaded, request limit: 2000 Jun 22 01:32:47.585663142 (81h9m23.148986155s) [noise=true , address=79.127.220.101:6048 , piece_id=AN7UQUXB42T2YXEH7ZJRZUDIW47M3RWUEOCL2VEBOGH74BZF3YKA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:47.640263002 (81h9m23.203586013s) [noise=true , address=45.91.116.12:5242 , piece_id=6ABT5IPYGSLFFJMBYMBYRU4EYIXABUAUHG3OVILOHBIUDS3BP27A, seg_pos={0 0}, upload=64958]: Done uploading piece. err: ecclient: upload failed (node:12JsFL62jgXmza8Md8eQ9k8cyw5V9Vz2RzN4y85ZvgHZjwrq4vQ, address:45.91.116.12:5242): protocol: storage node overloaded, request limit: 2000 Jun 22 01:32:47.984127275 (81h9m23.54745028s) [noise=true , address=64.49.13.137:4500 , piece_id=FAE2RXD57IRP5FXA2XR6THXFW27U7IWAZ4JDV5M2WRPXKUGZ4VIA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:48.100255249 (81h9m23.663578241s) [noise=true , address=77.90.31.14:4689 , piece_id=Q3HDS3BCMYDTODFRMU4XFRDCQK2QZE3ZMXVLUX4M54PDEBS7RR6A, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:48.221764035 (81h9m23.785087034s) [noise=true , address=104.36.82.54:4165 , piece_id=4CVOABPCWC7J7SRHZB6IN2WJJ6KGWUQ66ELUIZIFA67ETLQ7JEUQ, seg_pos={0 0}, upload=64958]: Done uploading piece. err: ecclient: upload failed (node:12wGUMXVEubEtki4qGnjGEsbYFB1iUauZcu3oDRKrCQsJMV3bV2, address:104.36.82.54:4165): protocol: storage node overloaded, request limit: 2000 Jun 22 01:32:48.509996527 (81h9m24.073319547s) [noise=true , address=104.36.82.9:6023 , piece_id=SISFAUU4RPF57OLGWM2BWQWE6GRUSCMORBYHWEXETNAVM2EEMO6Q, seg_pos={0 0}, upload=64958]: Done uploading piece. err: ecclient: upload failed (node:1j2S9tgWpLwmyvXZSq2qKKcbaC8r7LkzfDCaMxCejAni4CST1g, address:104.36.82.9:6023): protocol: storage node overloaded, request limit: 2000 Jun 22 01:32:48.510513169 (81h9m24.073836177s) [noise=true , address=104.36.82.42:6811 , piece_id=A5BFEHZ3WGGK53IZ7MKI6N4CVXHOORF6L47NR3UFDCAX7LBHRHVQ, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:48.565245605 (81h9m24.128568606s) [noise=true , address=79.127.205.234:5601 , piece_id=V7PMXBP2VFWBSZDG4GTLBD435T6F5DT7A43L4GQLQ5BDDTJ4WY6A, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:49.979979824 (81h9m25.543302825s) [noise=true , address=104.36.82.15:6289 , piece_id=GWL3P55BIK4IJEL5YFZKI7JCA4LADYOOHUSIQUNGJEKBJOTKMP5A, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:50.463750280 (81h9m26.02707328s) [noise=true , address=104.36.82.51:4084 , piece_id=D43FA7BFE54KPCWGDGDME2EGSWM7PONOVPD3DN2XY42MJFOKVDIQ, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:51.480599795 (81h9m27.043922798s) [noise=true , address=79.127.219.38:4023 , piece_id=BWLRCXXHBMBTNCZEVKKIIS5PB35PDQXQXCLIQB7GBLEIOWWUFBFA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:52.941200625 (81h9m28.504523627s) [noise=true , address=104.36.82.43:6841 , piece_id=PIHZMSTO3CII7KGDLGSAILTDRSLMEBVOLBCRCZJULMCFISXIZWXA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:58.281803598 (81h9m33.845126612s) [noise=true , address=64.49.13.139:4476 , piece_id=P7UKBNNZGI52B3I6WKTV6MVC262FYNKYYPJUI5G47UO5S4MU7PWA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:33:00.277271877 (81h9m35.840594867s) [noise=true , address=104.36.82.55:4213 , piece_id=X4IHS2XSPH52LIMNUYT7KGWMAPUH3TEMJOEEVB3JV5N45SEOH7YA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:33:09.435339213 (81h9m44.998662206s) [noise=true , address=104.36.82.52:4089 , piece_id=JSOEKLNWA3W3LOVKI3EGGKJYJ6J3UZY6KGABKQPQPXTIOOPMKVHA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:33:31.472868445 (81h10m7.03619143s) [noise=true , address=104.36.82.53:4122 , piece_id=HZZQBIKHQWLUE5EHK4KSKJGIUEG7D44RQASUJHCYXX6BICCGX6EA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:33:47.235404653 (81h10m22.798727642s) [noise=true , address=104.36.82.12:6158 , piece_id=LJZEOJVJCJAR2BRILIJI3ZVDGYMPBBTP2G3HCTN3MHNS4KYDPASQ, seg_pos={0 0}, upload=64958]: Done uploading piece. err: ecclient: upload failed (node:1SjM6Utm2YizeUbrBUD9Z7etE1nLadW5Fa1zYP7MioPeeqhdJ7, address:104.36.82.12:6158): protocol: expected piece hash; context deadline exceeded; EOF Jun 22 01:33:47.235448297 (81h10m22.798771306s) [seg_pos={0 0}, upload=64958]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jun 22 01:33:47.265442393 (81h10m22.828765401s) [upload=64958]: Done uploading segments... Jun 22 01:33:47.265476139 (81h10m22.82879914s) [upload=64958]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration:* 24.677995058s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [..................................................X.........] duration: 23.545069395s FAIL (2025-06-22 01:39:39 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== oad=64949]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jun 22 01:32:13.674692303 (81h8m49.238015311s) [upload=64949]: Done uploading segments... Jun 22 01:32:13.674727219 (81h8m49.238050233s) [upload=64949]: Upload finished. err: <nil> Jun 22 01:32:47.266407995 (81h9m22.829731002s) [upload=64958]: Starting upload Jun 22 01:32:47.266524767 (81h9m22.829847771s) [upload=64958]: Done starting upload Jun 22 01:32:47.266813109 (81h9m22.830136116s) [upload=64958]: Uploading segments... Jun 22 01:32:47.266969526 (81h9m22.830292526s) [upload=64958]: Got next segment. Inline: false Jun 22 01:32:47.267075475 (81h9m22.830398475s) [upload=64958]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 22 01:32:47.267081679 (81h9m22.830404667s) [upload=64958]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 22 01:32:47.301770382 (81h9m22.865093389s) [seg_pos={0 0}, upload=64958]: Begin upload segment... Jun 22 01:32:47.301823796 (81h9m22.865146799s) [seg_pos={0 0}, upload=64958]: Done begin upload segment. Jun 22 01:32:47.301855055 (81h9m22.865178087s) [noise=true , address=104.36.82.43:6841 , piece_id=PIHZMSTO3CII7KGDLGSAILTDRSLMEBVOLBCRCZJULMCFISXIZWXA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.301902600 (81h9m22.865225606s) [noise=true , address=104.36.82.42:6811 , piece_id=A5BFEHZ3WGGK53IZ7MKI6N4CVXHOORF6L47NR3UFDCAX7LBHRHVQ, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.302234027 (81h9m22.865557029s) [noise=true , address=104.36.82.51:4084 , piece_id=D43FA7BFE54KPCWGDGDME2EGSWM7PONOVPD3DN2XY42MJFOKVDIQ, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.302496354 (81h9m22.865819363s) [noise=true , address=79.127.219.38:4023 , piece_id=BWLRCXXHBMBTNCZEVKKIIS5PB35PDQXQXCLIQB7GBLEIOWWUFBFA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.302577370 (81h9m22.865900378s) [noise=true , address=104.36.82.52:4089 , piece_id=JSOEKLNWA3W3LOVKI3EGGKJYJ6J3UZY6KGABKQPQPXTIOOPMKVHA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.302917829 (81h9m22.866240835s) [noise=true , address=104.36.82.53:4122 , piece_id=HZZQBIKHQWLUE5EHK4KSKJGIUEG7D44RQASUJHCYXX6BICCGX6EA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.303087165 (81h9m22.866410168s) [noise=true , address=77.90.31.14:4689 , piece_id=Q3HDS3BCMYDTODFRMU4XFRDCQK2QZE3ZMXVLUX4M54PDEBS7RR6A, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.303250714 (81h9m22.866573717s) [noise=true , address=79.127.205.234:5601 , piece_id=V7PMXBP2VFWBSZDG4GTLBD435T6F5DT7A43L4GQLQ5BDDTJ4WY6A, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.303579249 (81h9m22.866902255s) [noise=true , address=104.36.82.12:6158 , piece_id=LJZEOJVJCJAR2BRILIJI3ZVDGYMPBBTP2G3HCTN3MHNS4KYDPASQ, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.303599776 (81h9m22.866922784s) [noise=true , address=104.36.82.9:6023 , piece_id=SISFAUU4RPF57OLGWM2BWQWE6GRUSCMORBYHWEXETNAVM2EEMO6Q, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.304005346 (81h9m22.867328346s) [noise=true , address=45.91.116.12:5242 , piece_id=6ABT5IPYGSLFFJMBYMBYRU4EYIXABUAUHG3OVILOHBIUDS3BP27A, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.304388850 (81h9m22.867711851s) [noise=true , address=64.49.13.137:4500 , piece_id=FAE2RXD57IRP5FXA2XR6THXFW27U7IWAZ4JDV5M2WRPXKUGZ4VIA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.304881529 (81h9m22.868204536s) [noise=true , address=79.127.220.101:6048 , piece_id=AN7UQUXB42T2YXEH7ZJRZUDIW47M3RWUEOCL2VEBOGH74BZF3YKA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.305249860 (81h9m22.86857287s) [noise=true , address=104.36.82.54:4165 , piece_id=4CVOABPCWC7J7SRHZB6IN2WJJ6KGWUQ66ELUIZIFA67ETLQ7JEUQ, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.305604108 (81h9m22.868927112s) [noise=true , address=104.36.82.15:6289 , piece_id=GWL3P55BIK4IJEL5YFZKI7JCA4LADYOOHUSIQUNGJEKBJOTKMP5A, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.305930423 (81h9m22.869253425s) [noise=true , address=64.49.13.139:4476 , piece_id=P7UKBNNZGI52B3I6WKTV6MVC262FYNKYYPJUI5G47UO5S4MU7PWA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.306355311 (81h9m22.869678319s) [noise=true , address=104.36.82.55:4213 , piece_id=X4IHS2XSPH52LIMNUYT7KGWMAPUH3TEMJOEEVB3JV5N45SEOH7YA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.306788243 (81h9m22.870111252s) [noise=true , address=104.36.82.22:6539 , piece_id=GHQ2MVSTDGVYLUKLV23D6YWRR4PP7UUYHCUEQ4WWSWEMTBFO3P4A, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.392174770 (81h9m22.955497773s) [upload=64958]: Next returned nil segment Jun 22 01:32:47.392195767 (81h9m22.955518775s) [upload=64958]: Waiting for error group managing segments... Jun 22 01:32:47.480767633 (81h9m23.044090623s) [noise=true , address=104.36.82.22:6539 , piece_id=GHQ2MVSTDGVYLUKLV23D6YWRR4PP7UUYHCUEQ4WWSWEMTBFO3P4A, seg_pos={0 0}, upload=64958]: Done uploading piece. err: ecclient: upload failed (node:1QL6w8DSFR6CxDdnJmDuyVZxp6JU55VKiJofBZRfQi7q31i8Jc, address:104.36.82.22:6539): protocol: storage node overloaded, request limit: 2000 Jun 22 01:32:47.585663142 (81h9m23.148986155s) [noise=true , address=79.127.220.101:6048 , piece_id=AN7UQUXB42T2YXEH7ZJRZUDIW47M3RWUEOCL2VEBOGH74BZF3YKA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:47.640263002 (81h9m23.203586013s) [noise=true , address=45.91.116.12:5242 , piece_id=6ABT5IPYGSLFFJMBYMBYRU4EYIXABUAUHG3OVILOHBIUDS3BP27A, seg_pos={0 0}, upload=64958]: Done uploading piece. err: ecclient: upload failed (node:12JsFL62jgXmza8Md8eQ9k8cyw5V9Vz2RzN4y85ZvgHZjwrq4vQ, address:45.91.116.12:5242): protocol: storage node overloaded, request limit: 2000 Jun 22 01:32:47.984127275 (81h9m23.54745028s) [noise=true , address=64.49.13.137:4500 , piece_id=FAE2RXD57IRP5FXA2XR6THXFW27U7IWAZ4JDV5M2WRPXKUGZ4VIA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:48.100255249 (81h9m23.663578241s) [noise=true , address=77.90.31.14:4689 , piece_id=Q3HDS3BCMYDTODFRMU4XFRDCQK2QZE3ZMXVLUX4M54PDEBS7RR6A, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:48.221764035 (81h9m23.785087034s) [noise=true , address=104.36.82.54:4165 , piece_id=4CVOABPCWC7J7SRHZB6IN2WJJ6KGWUQ66ELUIZIFA67ETLQ7JEUQ, seg_pos={0 0}, upload=64958]: Done uploading piece. err: ecclient: upload failed (node:12wGUMXVEubEtki4qGnjGEsbYFB1iUauZcu3oDRKrCQsJMV3bV2, address:104.36.82.54:4165): protocol: storage node overloaded, request limit: 2000 Jun 22 01:32:48.509996527 (81h9m24.073319547s) [noise=true , address=104.36.82.9:6023 , piece_id=SISFAUU4RPF57OLGWM2BWQWE6GRUSCMORBYHWEXETNAVM2EEMO6Q, seg_pos={0 0}, upload=64958]: Done uploading piece. err: ecclient: upload failed (node:1j2S9tgWpLwmyvXZSq2qKKcbaC8r7LkzfDCaMxCejAni4CST1g, address:104.36.82.9:6023): protocol: storage node overloaded, request limit: 2000 Jun 22 01:32:48.510513169 (81h9m24.073836177s) [noise=true , address=104.36.82.42:6811 , piece_id=A5BFEHZ3WGGK53IZ7MKI6N4CVXHOORF6L47NR3UFDCAX7LBHRHVQ, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:48.565245605 (81h9m24.128568606s) [noise=true , address=79.127.205.234:5601 , piece_id=V7PMXBP2VFWBSZDG4GTLBD435T6F5DT7A43L4GQLQ5BDDTJ4WY6A, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:49.979979824 (81h9m25.543302825s) [noise=true , address=104.36.82.15:6289 , piece_id=GWL3P55BIK4IJEL5YFZKI7JCA4LADYOOHUSIQUNGJEKBJOTKMP5A, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:50.463750280 (81h9m26.02707328s) [noise=true , address=104.36.82.51:4084 , piece_id=D43FA7BFE54KPCWGDGDME2EGSWM7PONOVPD3DN2XY42MJFOKVDIQ, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:51.480599795 (81h9m27.043922798s) [noise=true , address=79.127.219.38:4023 , piece_id=BWLRCXXHBMBTNCZEVKKIIS5PB35PDQXQXCLIQB7GBLEIOWWUFBFA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:52.941200625 (81h9m28.504523627s) [noise=true , address=104.36.82.43:6841 , piece_id=PIHZMSTO3CII7KGDLGSAILTDRSLMEBVOLBCRCZJULMCFISXIZWXA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:58.281803598 (81h9m33.845126612s) [noise=true , address=64.49.13.139:4476 , piece_id=P7UKBNNZGI52B3I6WKTV6MVC262FYNKYYPJUI5G47UO5S4MU7PWA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:33:00.277271877 (81h9m35.840594867s) [noise=true , address=104.36.82.55:4213 , piece_id=X4IHS2XSPH52LIMNUYT7KGWMAPUH3TEMJOEEVB3JV5N45SEOH7YA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:33:09.435339213 (81h9m44.998662206s) [noise=true , address=104.36.82.52:4089 , piece_id=JSOEKLNWA3W3LOVKI3EGGKJYJ6J3UZY6KGABKQPQPXTIOOPMKVHA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:33:31.472868445 (81h10m7.03619143s) [noise=true , address=104.36.82.53:4122 , piece_id=HZZQBIKHQWLUE5EHK4KSKJGIUEG7D44RQASUJHCYXX6BICCGX6EA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:33:47.235404653 (81h10m22.798727642s) [noise=true , address=104.36.82.12:6158 , piece_id=LJZEOJVJCJAR2BRILIJI3ZVDGYMPBBTP2G3HCTN3MHNS4KYDPASQ, seg_pos={0 0}, upload=64958]: Done uploading piece. err: ecclient: upload failed (node:1SjM6Utm2YizeUbrBUD9Z7etE1nLadW5Fa1zYP7MioPeeqhdJ7, address:104.36.82.12:6158): protocol: expected piece hash; context deadline exceeded; EOF Jun 22 01:33:47.235448297 (81h10m22.798771306s) [seg_pos={0 0}, upload=64958]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jun 22 01:33:47.265442393 (81h10m22.828765401s) [upload=64958]: Done uploading segments... Jun 22 01:33:47.265476139 (81h10m22.82879914s) [upload=64958]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration:* 23.545102473s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [...................................................X........] duration: 7.413740935s FAIL (2025-06-22 01:38:08 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== oad=64949]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jun 22 01:32:13.674692303 (81h8m49.238015311s) [upload=64949]: Done uploading segments... Jun 22 01:32:13.674727219 (81h8m49.238050233s) [upload=64949]: Upload finished. err: <nil> Jun 22 01:32:47.266407995 (81h9m22.829731002s) [upload=64958]: Starting upload Jun 22 01:32:47.266524767 (81h9m22.829847771s) [upload=64958]: Done starting upload Jun 22 01:32:47.266813109 (81h9m22.830136116s) [upload=64958]: Uploading segments... Jun 22 01:32:47.266969526 (81h9m22.830292526s) [upload=64958]: Got next segment. Inline: false Jun 22 01:32:47.267075475 (81h9m22.830398475s) [upload=64958]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 22 01:32:47.267081679 (81h9m22.830404667s) [upload=64958]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 22 01:32:47.301770382 (81h9m22.865093389s) [seg_pos={0 0}, upload=64958]: Begin upload segment... Jun 22 01:32:47.301823796 (81h9m22.865146799s) [seg_pos={0 0}, upload=64958]: Done begin upload segment. Jun 22 01:32:47.301855055 (81h9m22.865178087s) [noise=true , address=104.36.82.43:6841 , piece_id=PIHZMSTO3CII7KGDLGSAILTDRSLMEBVOLBCRCZJULMCFISXIZWXA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.301902600 (81h9m22.865225606s) [noise=true , address=104.36.82.42:6811 , piece_id=A5BFEHZ3WGGK53IZ7MKI6N4CVXHOORF6L47NR3UFDCAX7LBHRHVQ, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.302234027 (81h9m22.865557029s) [noise=true , address=104.36.82.51:4084 , piece_id=D43FA7BFE54KPCWGDGDME2EGSWM7PONOVPD3DN2XY42MJFOKVDIQ, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.302496354 (81h9m22.865819363s) [noise=true , address=79.127.219.38:4023 , piece_id=BWLRCXXHBMBTNCZEVKKIIS5PB35PDQXQXCLIQB7GBLEIOWWUFBFA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.302577370 (81h9m22.865900378s) [noise=true , address=104.36.82.52:4089 , piece_id=JSOEKLNWA3W3LOVKI3EGGKJYJ6J3UZY6KGABKQPQPXTIOOPMKVHA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.302917829 (81h9m22.866240835s) [noise=true , address=104.36.82.53:4122 , piece_id=HZZQBIKHQWLUE5EHK4KSKJGIUEG7D44RQASUJHCYXX6BICCGX6EA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.303087165 (81h9m22.866410168s) [noise=true , address=77.90.31.14:4689 , piece_id=Q3HDS3BCMYDTODFRMU4XFRDCQK2QZE3ZMXVLUX4M54PDEBS7RR6A, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.303250714 (81h9m22.866573717s) [noise=true , address=79.127.205.234:5601 , piece_id=V7PMXBP2VFWBSZDG4GTLBD435T6F5DT7A43L4GQLQ5BDDTJ4WY6A, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.303579249 (81h9m22.866902255s) [noise=true , address=104.36.82.12:6158 , piece_id=LJZEOJVJCJAR2BRILIJI3ZVDGYMPBBTP2G3HCTN3MHNS4KYDPASQ, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.303599776 (81h9m22.866922784s) [noise=true , address=104.36.82.9:6023 , piece_id=SISFAUU4RPF57OLGWM2BWQWE6GRUSCMORBYHWEXETNAVM2EEMO6Q, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.304005346 (81h9m22.867328346s) [noise=true , address=45.91.116.12:5242 , piece_id=6ABT5IPYGSLFFJMBYMBYRU4EYIXABUAUHG3OVILOHBIUDS3BP27A, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.304388850 (81h9m22.867711851s) [noise=true , address=64.49.13.137:4500 , piece_id=FAE2RXD57IRP5FXA2XR6THXFW27U7IWAZ4JDV5M2WRPXKUGZ4VIA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.304881529 (81h9m22.868204536s) [noise=true , address=79.127.220.101:6048 , piece_id=AN7UQUXB42T2YXEH7ZJRZUDIW47M3RWUEOCL2VEBOGH74BZF3YKA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.305249860 (81h9m22.86857287s) [noise=true , address=104.36.82.54:4165 , piece_id=4CVOABPCWC7J7SRHZB6IN2WJJ6KGWUQ66ELUIZIFA67ETLQ7JEUQ, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.305604108 (81h9m22.868927112s) [noise=true , address=104.36.82.15:6289 , piece_id=GWL3P55BIK4IJEL5YFZKI7JCA4LADYOOHUSIQUNGJEKBJOTKMP5A, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.305930423 (81h9m22.869253425s) [noise=true , address=64.49.13.139:4476 , piece_id=P7UKBNNZGI52B3I6WKTV6MVC262FYNKYYPJUI5G47UO5S4MU7PWA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.306355311 (81h9m22.869678319s) [noise=true , address=104.36.82.55:4213 , piece_id=X4IHS2XSPH52LIMNUYT7KGWMAPUH3TEMJOEEVB3JV5N45SEOH7YA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.306788243 (81h9m22.870111252s) [noise=true , address=104.36.82.22:6539 , piece_id=GHQ2MVSTDGVYLUKLV23D6YWRR4PP7UUYHCUEQ4WWSWEMTBFO3P4A, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.392174770 (81h9m22.955497773s) [upload=64958]: Next returned nil segment Jun 22 01:32:47.392195767 (81h9m22.955518775s) [upload=64958]: Waiting for error group managing segments... Jun 22 01:32:47.480767633 (81h9m23.044090623s) [noise=true , address=104.36.82.22:6539 , piece_id=GHQ2MVSTDGVYLUKLV23D6YWRR4PP7UUYHCUEQ4WWSWEMTBFO3P4A, seg_pos={0 0}, upload=64958]: Done uploading piece. err: ecclient: upload failed (node:1QL6w8DSFR6CxDdnJmDuyVZxp6JU55VKiJofBZRfQi7q31i8Jc, address:104.36.82.22:6539): protocol: storage node overloaded, request limit: 2000 Jun 22 01:32:47.585663142 (81h9m23.148986155s) [noise=true , address=79.127.220.101:6048 , piece_id=AN7UQUXB42T2YXEH7ZJRZUDIW47M3RWUEOCL2VEBOGH74BZF3YKA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:47.640263002 (81h9m23.203586013s) [noise=true , address=45.91.116.12:5242 , piece_id=6ABT5IPYGSLFFJMBYMBYRU4EYIXABUAUHG3OVILOHBIUDS3BP27A, seg_pos={0 0}, upload=64958]: Done uploading piece. err: ecclient: upload failed (node:12JsFL62jgXmza8Md8eQ9k8cyw5V9Vz2RzN4y85ZvgHZjwrq4vQ, address:45.91.116.12:5242): protocol: storage node overloaded, request limit: 2000 Jun 22 01:32:47.984127275 (81h9m23.54745028s) [noise=true , address=64.49.13.137:4500 , piece_id=FAE2RXD57IRP5FXA2XR6THXFW27U7IWAZ4JDV5M2WRPXKUGZ4VIA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:48.100255249 (81h9m23.663578241s) [noise=true , address=77.90.31.14:4689 , piece_id=Q3HDS3BCMYDTODFRMU4XFRDCQK2QZE3ZMXVLUX4M54PDEBS7RR6A, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:48.221764035 (81h9m23.785087034s) [noise=true , address=104.36.82.54:4165 , piece_id=4CVOABPCWC7J7SRHZB6IN2WJJ6KGWUQ66ELUIZIFA67ETLQ7JEUQ, seg_pos={0 0}, upload=64958]: Done uploading piece. err: ecclient: upload failed (node:12wGUMXVEubEtki4qGnjGEsbYFB1iUauZcu3oDRKrCQsJMV3bV2, address:104.36.82.54:4165): protocol: storage node overloaded, request limit: 2000 Jun 22 01:32:48.509996527 (81h9m24.073319547s) [noise=true , address=104.36.82.9:6023 , piece_id=SISFAUU4RPF57OLGWM2BWQWE6GRUSCMORBYHWEXETNAVM2EEMO6Q, seg_pos={0 0}, upload=64958]: Done uploading piece. err: ecclient: upload failed (node:1j2S9tgWpLwmyvXZSq2qKKcbaC8r7LkzfDCaMxCejAni4CST1g, address:104.36.82.9:6023): protocol: storage node overloaded, request limit: 2000 Jun 22 01:32:48.510513169 (81h9m24.073836177s) [noise=true , address=104.36.82.42:6811 , piece_id=A5BFEHZ3WGGK53IZ7MKI6N4CVXHOORF6L47NR3UFDCAX7LBHRHVQ, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:48.565245605 (81h9m24.128568606s) [noise=true , address=79.127.205.234:5601 , piece_id=V7PMXBP2VFWBSZDG4GTLBD435T6F5DT7A43L4GQLQ5BDDTJ4WY6A, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:49.979979824 (81h9m25.543302825s) [noise=true , address=104.36.82.15:6289 , piece_id=GWL3P55BIK4IJEL5YFZKI7JCA4LADYOOHUSIQUNGJEKBJOTKMP5A, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:50.463750280 (81h9m26.02707328s) [noise=true , address=104.36.82.51:4084 , piece_id=D43FA7BFE54KPCWGDGDME2EGSWM7PONOVPD3DN2XY42MJFOKVDIQ, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:51.480599795 (81h9m27.043922798s) [noise=true , address=79.127.219.38:4023 , piece_id=BWLRCXXHBMBTNCZEVKKIIS5PB35PDQXQXCLIQB7GBLEIOWWUFBFA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:52.941200625 (81h9m28.504523627s) [noise=true , address=104.36.82.43:6841 , piece_id=PIHZMSTO3CII7KGDLGSAILTDRSLMEBVOLBCRCZJULMCFISXIZWXA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:58.281803598 (81h9m33.845126612s) [noise=true , address=64.49.13.139:4476 , piece_id=P7UKBNNZGI52B3I6WKTV6MVC262FYNKYYPJUI5G47UO5S4MU7PWA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:33:00.277271877 (81h9m35.840594867s) [noise=true , address=104.36.82.55:4213 , piece_id=X4IHS2XSPH52LIMNUYT7KGWMAPUH3TEMJOEEVB3JV5N45SEOH7YA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:33:09.435339213 (81h9m44.998662206s) [noise=true , address=104.36.82.52:4089 , piece_id=JSOEKLNWA3W3LOVKI3EGGKJYJ6J3UZY6KGABKQPQPXTIOOPMKVHA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:33:31.472868445 (81h10m7.03619143s) [noise=true , address=104.36.82.53:4122 , piece_id=HZZQBIKHQWLUE5EHK4KSKJGIUEG7D44RQASUJHCYXX6BICCGX6EA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:33:47.235404653 (81h10m22.798727642s) [noise=true , address=104.36.82.12:6158 , piece_id=LJZEOJVJCJAR2BRILIJI3ZVDGYMPBBTP2G3HCTN3MHNS4KYDPASQ, seg_pos={0 0}, upload=64958]: Done uploading piece. err: ecclient: upload failed (node:1SjM6Utm2YizeUbrBUD9Z7etE1nLadW5Fa1zYP7MioPeeqhdJ7, address:104.36.82.12:6158): protocol: expected piece hash; context deadline exceeded; EOF Jun 22 01:33:47.235448297 (81h10m22.798771306s) [seg_pos={0 0}, upload=64958]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jun 22 01:33:47.265442393 (81h10m22.828765401s) [upload=64958]: Done uploading segments... Jun 22 01:33:47.265476139 (81h10m22.82879914s) [upload=64958]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration:* 7.413859897s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [....................................................X.......] duration: 14.521546748s FAIL (2025-06-22 01:37:25 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== oad=64949]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jun 22 01:32:13.674692303 (81h8m49.238015311s) [upload=64949]: Done uploading segments... Jun 22 01:32:13.674727219 (81h8m49.238050233s) [upload=64949]: Upload finished. err: <nil> Jun 22 01:32:47.266407995 (81h9m22.829731002s) [upload=64958]: Starting upload Jun 22 01:32:47.266524767 (81h9m22.829847771s) [upload=64958]: Done starting upload Jun 22 01:32:47.266813109 (81h9m22.830136116s) [upload=64958]: Uploading segments... Jun 22 01:32:47.266969526 (81h9m22.830292526s) [upload=64958]: Got next segment. Inline: false Jun 22 01:32:47.267075475 (81h9m22.830398475s) [upload=64958]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 22 01:32:47.267081679 (81h9m22.830404667s) [upload=64958]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 22 01:32:47.301770382 (81h9m22.865093389s) [seg_pos={0 0}, upload=64958]: Begin upload segment... Jun 22 01:32:47.301823796 (81h9m22.865146799s) [seg_pos={0 0}, upload=64958]: Done begin upload segment. Jun 22 01:32:47.301855055 (81h9m22.865178087s) [noise=true , address=104.36.82.43:6841 , piece_id=PIHZMSTO3CII7KGDLGSAILTDRSLMEBVOLBCRCZJULMCFISXIZWXA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.301902600 (81h9m22.865225606s) [noise=true , address=104.36.82.42:6811 , piece_id=A5BFEHZ3WGGK53IZ7MKI6N4CVXHOORF6L47NR3UFDCAX7LBHRHVQ, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.302234027 (81h9m22.865557029s) [noise=true , address=104.36.82.51:4084 , piece_id=D43FA7BFE54KPCWGDGDME2EGSWM7PONOVPD3DN2XY42MJFOKVDIQ, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.302496354 (81h9m22.865819363s) [noise=true , address=79.127.219.38:4023 , piece_id=BWLRCXXHBMBTNCZEVKKIIS5PB35PDQXQXCLIQB7GBLEIOWWUFBFA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.302577370 (81h9m22.865900378s) [noise=true , address=104.36.82.52:4089 , piece_id=JSOEKLNWA3W3LOVKI3EGGKJYJ6J3UZY6KGABKQPQPXTIOOPMKVHA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.302917829 (81h9m22.866240835s) [noise=true , address=104.36.82.53:4122 , piece_id=HZZQBIKHQWLUE5EHK4KSKJGIUEG7D44RQASUJHCYXX6BICCGX6EA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.303087165 (81h9m22.866410168s) [noise=true , address=77.90.31.14:4689 , piece_id=Q3HDS3BCMYDTODFRMU4XFRDCQK2QZE3ZMXVLUX4M54PDEBS7RR6A, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.303250714 (81h9m22.866573717s) [noise=true , address=79.127.205.234:5601 , piece_id=V7PMXBP2VFWBSZDG4GTLBD435T6F5DT7A43L4GQLQ5BDDTJ4WY6A, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.303579249 (81h9m22.866902255s) [noise=true , address=104.36.82.12:6158 , piece_id=LJZEOJVJCJAR2BRILIJI3ZVDGYMPBBTP2G3HCTN3MHNS4KYDPASQ, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.303599776 (81h9m22.866922784s) [noise=true , address=104.36.82.9:6023 , piece_id=SISFAUU4RPF57OLGWM2BWQWE6GRUSCMORBYHWEXETNAVM2EEMO6Q, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.304005346 (81h9m22.867328346s) [noise=true , address=45.91.116.12:5242 , piece_id=6ABT5IPYGSLFFJMBYMBYRU4EYIXABUAUHG3OVILOHBIUDS3BP27A, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.304388850 (81h9m22.867711851s) [noise=true , address=64.49.13.137:4500 , piece_id=FAE2RXD57IRP5FXA2XR6THXFW27U7IWAZ4JDV5M2WRPXKUGZ4VIA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.304881529 (81h9m22.868204536s) [noise=true , address=79.127.220.101:6048 , piece_id=AN7UQUXB42T2YXEH7ZJRZUDIW47M3RWUEOCL2VEBOGH74BZF3YKA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.305249860 (81h9m22.86857287s) [noise=true , address=104.36.82.54:4165 , piece_id=4CVOABPCWC7J7SRHZB6IN2WJJ6KGWUQ66ELUIZIFA67ETLQ7JEUQ, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.305604108 (81h9m22.868927112s) [noise=true , address=104.36.82.15:6289 , piece_id=GWL3P55BIK4IJEL5YFZKI7JCA4LADYOOHUSIQUNGJEKBJOTKMP5A, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.305930423 (81h9m22.869253425s) [noise=true , address=64.49.13.139:4476 , piece_id=P7UKBNNZGI52B3I6WKTV6MVC262FYNKYYPJUI5G47UO5S4MU7PWA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.306355311 (81h9m22.869678319s) [noise=true , address=104.36.82.55:4213 , piece_id=X4IHS2XSPH52LIMNUYT7KGWMAPUH3TEMJOEEVB3JV5N45SEOH7YA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.306788243 (81h9m22.870111252s) [noise=true , address=104.36.82.22:6539 , piece_id=GHQ2MVSTDGVYLUKLV23D6YWRR4PP7UUYHCUEQ4WWSWEMTBFO3P4A, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.392174770 (81h9m22.955497773s) [upload=64958]: Next returned nil segment Jun 22 01:32:47.392195767 (81h9m22.955518775s) [upload=64958]: Waiting for error group managing segments... Jun 22 01:32:47.480767633 (81h9m23.044090623s) [noise=true , address=104.36.82.22:6539 , piece_id=GHQ2MVSTDGVYLUKLV23D6YWRR4PP7UUYHCUEQ4WWSWEMTBFO3P4A, seg_pos={0 0}, upload=64958]: Done uploading piece. err: ecclient: upload failed (node:1QL6w8DSFR6CxDdnJmDuyVZxp6JU55VKiJofBZRfQi7q31i8Jc, address:104.36.82.22:6539): protocol: storage node overloaded, request limit: 2000 Jun 22 01:32:47.585663142 (81h9m23.148986155s) [noise=true , address=79.127.220.101:6048 , piece_id=AN7UQUXB42T2YXEH7ZJRZUDIW47M3RWUEOCL2VEBOGH74BZF3YKA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:47.640263002 (81h9m23.203586013s) [noise=true , address=45.91.116.12:5242 , piece_id=6ABT5IPYGSLFFJMBYMBYRU4EYIXABUAUHG3OVILOHBIUDS3BP27A, seg_pos={0 0}, upload=64958]: Done uploading piece. err: ecclient: upload failed (node:12JsFL62jgXmza8Md8eQ9k8cyw5V9Vz2RzN4y85ZvgHZjwrq4vQ, address:45.91.116.12:5242): protocol: storage node overloaded, request limit: 2000 Jun 22 01:32:47.984127275 (81h9m23.54745028s) [noise=true , address=64.49.13.137:4500 , piece_id=FAE2RXD57IRP5FXA2XR6THXFW27U7IWAZ4JDV5M2WRPXKUGZ4VIA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:48.100255249 (81h9m23.663578241s) [noise=true , address=77.90.31.14:4689 , piece_id=Q3HDS3BCMYDTODFRMU4XFRDCQK2QZE3ZMXVLUX4M54PDEBS7RR6A, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:48.221764035 (81h9m23.785087034s) [noise=true , address=104.36.82.54:4165 , piece_id=4CVOABPCWC7J7SRHZB6IN2WJJ6KGWUQ66ELUIZIFA67ETLQ7JEUQ, seg_pos={0 0}, upload=64958]: Done uploading piece. err: ecclient: upload failed (node:12wGUMXVEubEtki4qGnjGEsbYFB1iUauZcu3oDRKrCQsJMV3bV2, address:104.36.82.54:4165): protocol: storage node overloaded, request limit: 2000 Jun 22 01:32:48.509996527 (81h9m24.073319547s) [noise=true , address=104.36.82.9:6023 , piece_id=SISFAUU4RPF57OLGWM2BWQWE6GRUSCMORBYHWEXETNAVM2EEMO6Q, seg_pos={0 0}, upload=64958]: Done uploading piece. err: ecclient: upload failed (node:1j2S9tgWpLwmyvXZSq2qKKcbaC8r7LkzfDCaMxCejAni4CST1g, address:104.36.82.9:6023): protocol: storage node overloaded, request limit: 2000 Jun 22 01:32:48.510513169 (81h9m24.073836177s) [noise=true , address=104.36.82.42:6811 , piece_id=A5BFEHZ3WGGK53IZ7MKI6N4CVXHOORF6L47NR3UFDCAX7LBHRHVQ, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:48.565245605 (81h9m24.128568606s) [noise=true , address=79.127.205.234:5601 , piece_id=V7PMXBP2VFWBSZDG4GTLBD435T6F5DT7A43L4GQLQ5BDDTJ4WY6A, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:49.979979824 (81h9m25.543302825s) [noise=true , address=104.36.82.15:6289 , piece_id=GWL3P55BIK4IJEL5YFZKI7JCA4LADYOOHUSIQUNGJEKBJOTKMP5A, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:50.463750280 (81h9m26.02707328s) [noise=true , address=104.36.82.51:4084 , piece_id=D43FA7BFE54KPCWGDGDME2EGSWM7PONOVPD3DN2XY42MJFOKVDIQ, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:51.480599795 (81h9m27.043922798s) [noise=true , address=79.127.219.38:4023 , piece_id=BWLRCXXHBMBTNCZEVKKIIS5PB35PDQXQXCLIQB7GBLEIOWWUFBFA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:52.941200625 (81h9m28.504523627s) [noise=true , address=104.36.82.43:6841 , piece_id=PIHZMSTO3CII7KGDLGSAILTDRSLMEBVOLBCRCZJULMCFISXIZWXA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:58.281803598 (81h9m33.845126612s) [noise=true , address=64.49.13.139:4476 , piece_id=P7UKBNNZGI52B3I6WKTV6MVC262FYNKYYPJUI5G47UO5S4MU7PWA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:33:00.277271877 (81h9m35.840594867s) [noise=true , address=104.36.82.55:4213 , piece_id=X4IHS2XSPH52LIMNUYT7KGWMAPUH3TEMJOEEVB3JV5N45SEOH7YA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:33:09.435339213 (81h9m44.998662206s) [noise=true , address=104.36.82.52:4089 , piece_id=JSOEKLNWA3W3LOVKI3EGGKJYJ6J3UZY6KGABKQPQPXTIOOPMKVHA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:33:31.472868445 (81h10m7.03619143s) [noise=true , address=104.36.82.53:4122 , piece_id=HZZQBIKHQWLUE5EHK4KSKJGIUEG7D44RQASUJHCYXX6BICCGX6EA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:33:47.235404653 (81h10m22.798727642s) [noise=true , address=104.36.82.12:6158 , piece_id=LJZEOJVJCJAR2BRILIJI3ZVDGYMPBBTP2G3HCTN3MHNS4KYDPASQ, seg_pos={0 0}, upload=64958]: Done uploading piece. err: ecclient: upload failed (node:1SjM6Utm2YizeUbrBUD9Z7etE1nLadW5Fa1zYP7MioPeeqhdJ7, address:104.36.82.12:6158): protocol: expected piece hash; context deadline exceeded; EOF Jun 22 01:33:47.235448297 (81h10m22.798771306s) [seg_pos={0 0}, upload=64958]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jun 22 01:33:47.265442393 (81h10m22.828765401s) [upload=64958]: Done uploading segments... Jun 22 01:33:47.265476139 (81h10m22.82879914s) [upload=64958]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration:* 14.52157431s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.....................................................X......] duration: 14.953593708s FAIL (2025-06-22 01:36:25 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== oad=64949]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jun 22 01:32:13.674692303 (81h8m49.238015311s) [upload=64949]: Done uploading segments... Jun 22 01:32:13.674727219 (81h8m49.238050233s) [upload=64949]: Upload finished. err: <nil> Jun 22 01:32:47.266407995 (81h9m22.829731002s) [upload=64958]: Starting upload Jun 22 01:32:47.266524767 (81h9m22.829847771s) [upload=64958]: Done starting upload Jun 22 01:32:47.266813109 (81h9m22.830136116s) [upload=64958]: Uploading segments... Jun 22 01:32:47.266969526 (81h9m22.830292526s) [upload=64958]: Got next segment. Inline: false Jun 22 01:32:47.267075475 (81h9m22.830398475s) [upload=64958]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 22 01:32:47.267081679 (81h9m22.830404667s) [upload=64958]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 22 01:32:47.301770382 (81h9m22.865093389s) [seg_pos={0 0}, upload=64958]: Begin upload segment... Jun 22 01:32:47.301823796 (81h9m22.865146799s) [seg_pos={0 0}, upload=64958]: Done begin upload segment. Jun 22 01:32:47.301855055 (81h9m22.865178087s) [noise=true , address=104.36.82.43:6841 , piece_id=PIHZMSTO3CII7KGDLGSAILTDRSLMEBVOLBCRCZJULMCFISXIZWXA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.301902600 (81h9m22.865225606s) [noise=true , address=104.36.82.42:6811 , piece_id=A5BFEHZ3WGGK53IZ7MKI6N4CVXHOORF6L47NR3UFDCAX7LBHRHVQ, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.302234027 (81h9m22.865557029s) [noise=true , address=104.36.82.51:4084 , piece_id=D43FA7BFE54KPCWGDGDME2EGSWM7PONOVPD3DN2XY42MJFOKVDIQ, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.302496354 (81h9m22.865819363s) [noise=true , address=79.127.219.38:4023 , piece_id=BWLRCXXHBMBTNCZEVKKIIS5PB35PDQXQXCLIQB7GBLEIOWWUFBFA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.302577370 (81h9m22.865900378s) [noise=true , address=104.36.82.52:4089 , piece_id=JSOEKLNWA3W3LOVKI3EGGKJYJ6J3UZY6KGABKQPQPXTIOOPMKVHA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.302917829 (81h9m22.866240835s) [noise=true , address=104.36.82.53:4122 , piece_id=HZZQBIKHQWLUE5EHK4KSKJGIUEG7D44RQASUJHCYXX6BICCGX6EA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.303087165 (81h9m22.866410168s) [noise=true , address=77.90.31.14:4689 , piece_id=Q3HDS3BCMYDTODFRMU4XFRDCQK2QZE3ZMXVLUX4M54PDEBS7RR6A, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.303250714 (81h9m22.866573717s) [noise=true , address=79.127.205.234:5601 , piece_id=V7PMXBP2VFWBSZDG4GTLBD435T6F5DT7A43L4GQLQ5BDDTJ4WY6A, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.303579249 (81h9m22.866902255s) [noise=true , address=104.36.82.12:6158 , piece_id=LJZEOJVJCJAR2BRILIJI3ZVDGYMPBBTP2G3HCTN3MHNS4KYDPASQ, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.303599776 (81h9m22.866922784s) [noise=true , address=104.36.82.9:6023 , piece_id=SISFAUU4RPF57OLGWM2BWQWE6GRUSCMORBYHWEXETNAVM2EEMO6Q, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.304005346 (81h9m22.867328346s) [noise=true , address=45.91.116.12:5242 , piece_id=6ABT5IPYGSLFFJMBYMBYRU4EYIXABUAUHG3OVILOHBIUDS3BP27A, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.304388850 (81h9m22.867711851s) [noise=true , address=64.49.13.137:4500 , piece_id=FAE2RXD57IRP5FXA2XR6THXFW27U7IWAZ4JDV5M2WRPXKUGZ4VIA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.304881529 (81h9m22.868204536s) [noise=true , address=79.127.220.101:6048 , piece_id=AN7UQUXB42T2YXEH7ZJRZUDIW47M3RWUEOCL2VEBOGH74BZF3YKA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.305249860 (81h9m22.86857287s) [noise=true , address=104.36.82.54:4165 , piece_id=4CVOABPCWC7J7SRHZB6IN2WJJ6KGWUQ66ELUIZIFA67ETLQ7JEUQ, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.305604108 (81h9m22.868927112s) [noise=true , address=104.36.82.15:6289 , piece_id=GWL3P55BIK4IJEL5YFZKI7JCA4LADYOOHUSIQUNGJEKBJOTKMP5A, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.305930423 (81h9m22.869253425s) [noise=true , address=64.49.13.139:4476 , piece_id=P7UKBNNZGI52B3I6WKTV6MVC262FYNKYYPJUI5G47UO5S4MU7PWA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.306355311 (81h9m22.869678319s) [noise=true , address=104.36.82.55:4213 , piece_id=X4IHS2XSPH52LIMNUYT7KGWMAPUH3TEMJOEEVB3JV5N45SEOH7YA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.306788243 (81h9m22.870111252s) [noise=true , address=104.36.82.22:6539 , piece_id=GHQ2MVSTDGVYLUKLV23D6YWRR4PP7UUYHCUEQ4WWSWEMTBFO3P4A, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.392174770 (81h9m22.955497773s) [upload=64958]: Next returned nil segment Jun 22 01:32:47.392195767 (81h9m22.955518775s) [upload=64958]: Waiting for error group managing segments... Jun 22 01:32:47.480767633 (81h9m23.044090623s) [noise=true , address=104.36.82.22:6539 , piece_id=GHQ2MVSTDGVYLUKLV23D6YWRR4PP7UUYHCUEQ4WWSWEMTBFO3P4A, seg_pos={0 0}, upload=64958]: Done uploading piece. err: ecclient: upload failed (node:1QL6w8DSFR6CxDdnJmDuyVZxp6JU55VKiJofBZRfQi7q31i8Jc, address:104.36.82.22:6539): protocol: storage node overloaded, request limit: 2000 Jun 22 01:32:47.585663142 (81h9m23.148986155s) [noise=true , address=79.127.220.101:6048 , piece_id=AN7UQUXB42T2YXEH7ZJRZUDIW47M3RWUEOCL2VEBOGH74BZF3YKA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:47.640263002 (81h9m23.203586013s) [noise=true , address=45.91.116.12:5242 , piece_id=6ABT5IPYGSLFFJMBYMBYRU4EYIXABUAUHG3OVILOHBIUDS3BP27A, seg_pos={0 0}, upload=64958]: Done uploading piece. err: ecclient: upload failed (node:12JsFL62jgXmza8Md8eQ9k8cyw5V9Vz2RzN4y85ZvgHZjwrq4vQ, address:45.91.116.12:5242): protocol: storage node overloaded, request limit: 2000 Jun 22 01:32:47.984127275 (81h9m23.54745028s) [noise=true , address=64.49.13.137:4500 , piece_id=FAE2RXD57IRP5FXA2XR6THXFW27U7IWAZ4JDV5M2WRPXKUGZ4VIA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:48.100255249 (81h9m23.663578241s) [noise=true , address=77.90.31.14:4689 , piece_id=Q3HDS3BCMYDTODFRMU4XFRDCQK2QZE3ZMXVLUX4M54PDEBS7RR6A, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:48.221764035 (81h9m23.785087034s) [noise=true , address=104.36.82.54:4165 , piece_id=4CVOABPCWC7J7SRHZB6IN2WJJ6KGWUQ66ELUIZIFA67ETLQ7JEUQ, seg_pos={0 0}, upload=64958]: Done uploading piece. err: ecclient: upload failed (node:12wGUMXVEubEtki4qGnjGEsbYFB1iUauZcu3oDRKrCQsJMV3bV2, address:104.36.82.54:4165): protocol: storage node overloaded, request limit: 2000 Jun 22 01:32:48.509996527 (81h9m24.073319547s) [noise=true , address=104.36.82.9:6023 , piece_id=SISFAUU4RPF57OLGWM2BWQWE6GRUSCMORBYHWEXETNAVM2EEMO6Q, seg_pos={0 0}, upload=64958]: Done uploading piece. err: ecclient: upload failed (node:1j2S9tgWpLwmyvXZSq2qKKcbaC8r7LkzfDCaMxCejAni4CST1g, address:104.36.82.9:6023): protocol: storage node overloaded, request limit: 2000 Jun 22 01:32:48.510513169 (81h9m24.073836177s) [noise=true , address=104.36.82.42:6811 , piece_id=A5BFEHZ3WGGK53IZ7MKI6N4CVXHOORF6L47NR3UFDCAX7LBHRHVQ, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:48.565245605 (81h9m24.128568606s) [noise=true , address=79.127.205.234:5601 , piece_id=V7PMXBP2VFWBSZDG4GTLBD435T6F5DT7A43L4GQLQ5BDDTJ4WY6A, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:49.979979824 (81h9m25.543302825s) [noise=true , address=104.36.82.15:6289 , piece_id=GWL3P55BIK4IJEL5YFZKI7JCA4LADYOOHUSIQUNGJEKBJOTKMP5A, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:50.463750280 (81h9m26.02707328s) [noise=true , address=104.36.82.51:4084 , piece_id=D43FA7BFE54KPCWGDGDME2EGSWM7PONOVPD3DN2XY42MJFOKVDIQ, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:51.480599795 (81h9m27.043922798s) [noise=true , address=79.127.219.38:4023 , piece_id=BWLRCXXHBMBTNCZEVKKIIS5PB35PDQXQXCLIQB7GBLEIOWWUFBFA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:52.941200625 (81h9m28.504523627s) [noise=true , address=104.36.82.43:6841 , piece_id=PIHZMSTO3CII7KGDLGSAILTDRSLMEBVOLBCRCZJULMCFISXIZWXA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:58.281803598 (81h9m33.845126612s) [noise=true , address=64.49.13.139:4476 , piece_id=P7UKBNNZGI52B3I6WKTV6MVC262FYNKYYPJUI5G47UO5S4MU7PWA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:33:00.277271877 (81h9m35.840594867s) [noise=true , address=104.36.82.55:4213 , piece_id=X4IHS2XSPH52LIMNUYT7KGWMAPUH3TEMJOEEVB3JV5N45SEOH7YA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:33:09.435339213 (81h9m44.998662206s) [noise=true , address=104.36.82.52:4089 , piece_id=JSOEKLNWA3W3LOVKI3EGGKJYJ6J3UZY6KGABKQPQPXTIOOPMKVHA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:33:31.472868445 (81h10m7.03619143s) [noise=true , address=104.36.82.53:4122 , piece_id=HZZQBIKHQWLUE5EHK4KSKJGIUEG7D44RQASUJHCYXX6BICCGX6EA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:33:47.235404653 (81h10m22.798727642s) [noise=true , address=104.36.82.12:6158 , piece_id=LJZEOJVJCJAR2BRILIJI3ZVDGYMPBBTP2G3HCTN3MHNS4KYDPASQ, seg_pos={0 0}, upload=64958]: Done uploading piece. err: ecclient: upload failed (node:1SjM6Utm2YizeUbrBUD9Z7etE1nLadW5Fa1zYP7MioPeeqhdJ7, address:104.36.82.12:6158): protocol: expected piece hash; context deadline exceeded; EOF Jun 22 01:33:47.235448297 (81h10m22.798771306s) [seg_pos={0 0}, upload=64958]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jun 22 01:33:47.265442393 (81h10m22.828765401s) [upload=64958]: Done uploading segments... Jun 22 01:33:47.265476139 (81h10m22.82879914s) [upload=64958]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration:* 14.953619933s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [......................................................X.....] duration: 17.638117891s FAIL (2025-06-22 01:36:08 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== oad=64949]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jun 22 01:32:13.674692303 (81h8m49.238015311s) [upload=64949]: Done uploading segments... Jun 22 01:32:13.674727219 (81h8m49.238050233s) [upload=64949]: Upload finished. err: <nil> Jun 22 01:32:47.266407995 (81h9m22.829731002s) [upload=64958]: Starting upload Jun 22 01:32:47.266524767 (81h9m22.829847771s) [upload=64958]: Done starting upload Jun 22 01:32:47.266813109 (81h9m22.830136116s) [upload=64958]: Uploading segments... Jun 22 01:32:47.266969526 (81h9m22.830292526s) [upload=64958]: Got next segment. Inline: false Jun 22 01:32:47.267075475 (81h9m22.830398475s) [upload=64958]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 22 01:32:47.267081679 (81h9m22.830404667s) [upload=64958]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 22 01:32:47.301770382 (81h9m22.865093389s) [seg_pos={0 0}, upload=64958]: Begin upload segment... Jun 22 01:32:47.301823796 (81h9m22.865146799s) [seg_pos={0 0}, upload=64958]: Done begin upload segment. Jun 22 01:32:47.301855055 (81h9m22.865178087s) [noise=true , address=104.36.82.43:6841 , piece_id=PIHZMSTO3CII7KGDLGSAILTDRSLMEBVOLBCRCZJULMCFISXIZWXA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.301902600 (81h9m22.865225606s) [noise=true , address=104.36.82.42:6811 , piece_id=A5BFEHZ3WGGK53IZ7MKI6N4CVXHOORF6L47NR3UFDCAX7LBHRHVQ, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.302234027 (81h9m22.865557029s) [noise=true , address=104.36.82.51:4084 , piece_id=D43FA7BFE54KPCWGDGDME2EGSWM7PONOVPD3DN2XY42MJFOKVDIQ, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.302496354 (81h9m22.865819363s) [noise=true , address=79.127.219.38:4023 , piece_id=BWLRCXXHBMBTNCZEVKKIIS5PB35PDQXQXCLIQB7GBLEIOWWUFBFA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.302577370 (81h9m22.865900378s) [noise=true , address=104.36.82.52:4089 , piece_id=JSOEKLNWA3W3LOVKI3EGGKJYJ6J3UZY6KGABKQPQPXTIOOPMKVHA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.302917829 (81h9m22.866240835s) [noise=true , address=104.36.82.53:4122 , piece_id=HZZQBIKHQWLUE5EHK4KSKJGIUEG7D44RQASUJHCYXX6BICCGX6EA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.303087165 (81h9m22.866410168s) [noise=true , address=77.90.31.14:4689 , piece_id=Q3HDS3BCMYDTODFRMU4XFRDCQK2QZE3ZMXVLUX4M54PDEBS7RR6A, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.303250714 (81h9m22.866573717s) [noise=true , address=79.127.205.234:5601 , piece_id=V7PMXBP2VFWBSZDG4GTLBD435T6F5DT7A43L4GQLQ5BDDTJ4WY6A, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.303579249 (81h9m22.866902255s) [noise=true , address=104.36.82.12:6158 , piece_id=LJZEOJVJCJAR2BRILIJI3ZVDGYMPBBTP2G3HCTN3MHNS4KYDPASQ, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.303599776 (81h9m22.866922784s) [noise=true , address=104.36.82.9:6023 , piece_id=SISFAUU4RPF57OLGWM2BWQWE6GRUSCMORBYHWEXETNAVM2EEMO6Q, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.304005346 (81h9m22.867328346s) [noise=true , address=45.91.116.12:5242 , piece_id=6ABT5IPYGSLFFJMBYMBYRU4EYIXABUAUHG3OVILOHBIUDS3BP27A, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.304388850 (81h9m22.867711851s) [noise=true , address=64.49.13.137:4500 , piece_id=FAE2RXD57IRP5FXA2XR6THXFW27U7IWAZ4JDV5M2WRPXKUGZ4VIA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.304881529 (81h9m22.868204536s) [noise=true , address=79.127.220.101:6048 , piece_id=AN7UQUXB42T2YXEH7ZJRZUDIW47M3RWUEOCL2VEBOGH74BZF3YKA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.305249860 (81h9m22.86857287s) [noise=true , address=104.36.82.54:4165 , piece_id=4CVOABPCWC7J7SRHZB6IN2WJJ6KGWUQ66ELUIZIFA67ETLQ7JEUQ, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.305604108 (81h9m22.868927112s) [noise=true , address=104.36.82.15:6289 , piece_id=GWL3P55BIK4IJEL5YFZKI7JCA4LADYOOHUSIQUNGJEKBJOTKMP5A, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.305930423 (81h9m22.869253425s) [noise=true , address=64.49.13.139:4476 , piece_id=P7UKBNNZGI52B3I6WKTV6MVC262FYNKYYPJUI5G47UO5S4MU7PWA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.306355311 (81h9m22.869678319s) [noise=true , address=104.36.82.55:4213 , piece_id=X4IHS2XSPH52LIMNUYT7KGWMAPUH3TEMJOEEVB3JV5N45SEOH7YA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.306788243 (81h9m22.870111252s) [noise=true , address=104.36.82.22:6539 , piece_id=GHQ2MVSTDGVYLUKLV23D6YWRR4PP7UUYHCUEQ4WWSWEMTBFO3P4A, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.392174770 (81h9m22.955497773s) [upload=64958]: Next returned nil segment Jun 22 01:32:47.392195767 (81h9m22.955518775s) [upload=64958]: Waiting for error group managing segments... Jun 22 01:32:47.480767633 (81h9m23.044090623s) [noise=true , address=104.36.82.22:6539 , piece_id=GHQ2MVSTDGVYLUKLV23D6YWRR4PP7UUYHCUEQ4WWSWEMTBFO3P4A, seg_pos={0 0}, upload=64958]: Done uploading piece. err: ecclient: upload failed (node:1QL6w8DSFR6CxDdnJmDuyVZxp6JU55VKiJofBZRfQi7q31i8Jc, address:104.36.82.22:6539): protocol: storage node overloaded, request limit: 2000 Jun 22 01:32:47.585663142 (81h9m23.148986155s) [noise=true , address=79.127.220.101:6048 , piece_id=AN7UQUXB42T2YXEH7ZJRZUDIW47M3RWUEOCL2VEBOGH74BZF3YKA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:47.640263002 (81h9m23.203586013s) [noise=true , address=45.91.116.12:5242 , piece_id=6ABT5IPYGSLFFJMBYMBYRU4EYIXABUAUHG3OVILOHBIUDS3BP27A, seg_pos={0 0}, upload=64958]: Done uploading piece. err: ecclient: upload failed (node:12JsFL62jgXmza8Md8eQ9k8cyw5V9Vz2RzN4y85ZvgHZjwrq4vQ, address:45.91.116.12:5242): protocol: storage node overloaded, request limit: 2000 Jun 22 01:32:47.984127275 (81h9m23.54745028s) [noise=true , address=64.49.13.137:4500 , piece_id=FAE2RXD57IRP5FXA2XR6THXFW27U7IWAZ4JDV5M2WRPXKUGZ4VIA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:48.100255249 (81h9m23.663578241s) [noise=true , address=77.90.31.14:4689 , piece_id=Q3HDS3BCMYDTODFRMU4XFRDCQK2QZE3ZMXVLUX4M54PDEBS7RR6A, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:48.221764035 (81h9m23.785087034s) [noise=true , address=104.36.82.54:4165 , piece_id=4CVOABPCWC7J7SRHZB6IN2WJJ6KGWUQ66ELUIZIFA67ETLQ7JEUQ, seg_pos={0 0}, upload=64958]: Done uploading piece. err: ecclient: upload failed (node:12wGUMXVEubEtki4qGnjGEsbYFB1iUauZcu3oDRKrCQsJMV3bV2, address:104.36.82.54:4165): protocol: storage node overloaded, request limit: 2000 Jun 22 01:32:48.509996527 (81h9m24.073319547s) [noise=true , address=104.36.82.9:6023 , piece_id=SISFAUU4RPF57OLGWM2BWQWE6GRUSCMORBYHWEXETNAVM2EEMO6Q, seg_pos={0 0}, upload=64958]: Done uploading piece. err: ecclient: upload failed (node:1j2S9tgWpLwmyvXZSq2qKKcbaC8r7LkzfDCaMxCejAni4CST1g, address:104.36.82.9:6023): protocol: storage node overloaded, request limit: 2000 Jun 22 01:32:48.510513169 (81h9m24.073836177s) [noise=true , address=104.36.82.42:6811 , piece_id=A5BFEHZ3WGGK53IZ7MKI6N4CVXHOORF6L47NR3UFDCAX7LBHRHVQ, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:48.565245605 (81h9m24.128568606s) [noise=true , address=79.127.205.234:5601 , piece_id=V7PMXBP2VFWBSZDG4GTLBD435T6F5DT7A43L4GQLQ5BDDTJ4WY6A, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:49.979979824 (81h9m25.543302825s) [noise=true , address=104.36.82.15:6289 , piece_id=GWL3P55BIK4IJEL5YFZKI7JCA4LADYOOHUSIQUNGJEKBJOTKMP5A, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:50.463750280 (81h9m26.02707328s) [noise=true , address=104.36.82.51:4084 , piece_id=D43FA7BFE54KPCWGDGDME2EGSWM7PONOVPD3DN2XY42MJFOKVDIQ, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:51.480599795 (81h9m27.043922798s) [noise=true , address=79.127.219.38:4023 , piece_id=BWLRCXXHBMBTNCZEVKKIIS5PB35PDQXQXCLIQB7GBLEIOWWUFBFA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:52.941200625 (81h9m28.504523627s) [noise=true , address=104.36.82.43:6841 , piece_id=PIHZMSTO3CII7KGDLGSAILTDRSLMEBVOLBCRCZJULMCFISXIZWXA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:58.281803598 (81h9m33.845126612s) [noise=true , address=64.49.13.139:4476 , piece_id=P7UKBNNZGI52B3I6WKTV6MVC262FYNKYYPJUI5G47UO5S4MU7PWA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:33:00.277271877 (81h9m35.840594867s) [noise=true , address=104.36.82.55:4213 , piece_id=X4IHS2XSPH52LIMNUYT7KGWMAPUH3TEMJOEEVB3JV5N45SEOH7YA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:33:09.435339213 (81h9m44.998662206s) [noise=true , address=104.36.82.52:4089 , piece_id=JSOEKLNWA3W3LOVKI3EGGKJYJ6J3UZY6KGABKQPQPXTIOOPMKVHA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:33:31.472868445 (81h10m7.03619143s) [noise=true , address=104.36.82.53:4122 , piece_id=HZZQBIKHQWLUE5EHK4KSKJGIUEG7D44RQASUJHCYXX6BICCGX6EA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:33:47.235404653 (81h10m22.798727642s) [noise=true , address=104.36.82.12:6158 , piece_id=LJZEOJVJCJAR2BRILIJI3ZVDGYMPBBTP2G3HCTN3MHNS4KYDPASQ, seg_pos={0 0}, upload=64958]: Done uploading piece. err: ecclient: upload failed (node:1SjM6Utm2YizeUbrBUD9Z7etE1nLadW5Fa1zYP7MioPeeqhdJ7, address:104.36.82.12:6158): protocol: expected piece hash; context deadline exceeded; EOF Jun 22 01:33:47.235448297 (81h10m22.798771306s) [seg_pos={0 0}, upload=64958]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jun 22 01:33:47.265442393 (81h10m22.828765401s) [upload=64958]: Done uploading segments... Jun 22 01:33:47.265476139 (81h10m22.82879914s) [upload=64958]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration:* 17.638168973s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.......................................................X....] duration: 13.991277901s FAIL (2025-06-22 01:35:46 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== oad=64949]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jun 22 01:32:13.674692303 (81h8m49.238015311s) [upload=64949]: Done uploading segments... Jun 22 01:32:13.674727219 (81h8m49.238050233s) [upload=64949]: Upload finished. err: <nil> Jun 22 01:32:47.266407995 (81h9m22.829731002s) [upload=64958]: Starting upload Jun 22 01:32:47.266524767 (81h9m22.829847771s) [upload=64958]: Done starting upload Jun 22 01:32:47.266813109 (81h9m22.830136116s) [upload=64958]: Uploading segments... Jun 22 01:32:47.266969526 (81h9m22.830292526s) [upload=64958]: Got next segment. Inline: false Jun 22 01:32:47.267075475 (81h9m22.830398475s) [upload=64958]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 22 01:32:47.267081679 (81h9m22.830404667s) [upload=64958]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 22 01:32:47.301770382 (81h9m22.865093389s) [seg_pos={0 0}, upload=64958]: Begin upload segment... Jun 22 01:32:47.301823796 (81h9m22.865146799s) [seg_pos={0 0}, upload=64958]: Done begin upload segment. Jun 22 01:32:47.301855055 (81h9m22.865178087s) [noise=true , address=104.36.82.43:6841 , piece_id=PIHZMSTO3CII7KGDLGSAILTDRSLMEBVOLBCRCZJULMCFISXIZWXA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.301902600 (81h9m22.865225606s) [noise=true , address=104.36.82.42:6811 , piece_id=A5BFEHZ3WGGK53IZ7MKI6N4CVXHOORF6L47NR3UFDCAX7LBHRHVQ, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.302234027 (81h9m22.865557029s) [noise=true , address=104.36.82.51:4084 , piece_id=D43FA7BFE54KPCWGDGDME2EGSWM7PONOVPD3DN2XY42MJFOKVDIQ, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.302496354 (81h9m22.865819363s) [noise=true , address=79.127.219.38:4023 , piece_id=BWLRCXXHBMBTNCZEVKKIIS5PB35PDQXQXCLIQB7GBLEIOWWUFBFA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.302577370 (81h9m22.865900378s) [noise=true , address=104.36.82.52:4089 , piece_id=JSOEKLNWA3W3LOVKI3EGGKJYJ6J3UZY6KGABKQPQPXTIOOPMKVHA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.302917829 (81h9m22.866240835s) [noise=true , address=104.36.82.53:4122 , piece_id=HZZQBIKHQWLUE5EHK4KSKJGIUEG7D44RQASUJHCYXX6BICCGX6EA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.303087165 (81h9m22.866410168s) [noise=true , address=77.90.31.14:4689 , piece_id=Q3HDS3BCMYDTODFRMU4XFRDCQK2QZE3ZMXVLUX4M54PDEBS7RR6A, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.303250714 (81h9m22.866573717s) [noise=true , address=79.127.205.234:5601 , piece_id=V7PMXBP2VFWBSZDG4GTLBD435T6F5DT7A43L4GQLQ5BDDTJ4WY6A, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.303579249 (81h9m22.866902255s) [noise=true , address=104.36.82.12:6158 , piece_id=LJZEOJVJCJAR2BRILIJI3ZVDGYMPBBTP2G3HCTN3MHNS4KYDPASQ, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.303599776 (81h9m22.866922784s) [noise=true , address=104.36.82.9:6023 , piece_id=SISFAUU4RPF57OLGWM2BWQWE6GRUSCMORBYHWEXETNAVM2EEMO6Q, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.304005346 (81h9m22.867328346s) [noise=true , address=45.91.116.12:5242 , piece_id=6ABT5IPYGSLFFJMBYMBYRU4EYIXABUAUHG3OVILOHBIUDS3BP27A, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.304388850 (81h9m22.867711851s) [noise=true , address=64.49.13.137:4500 , piece_id=FAE2RXD57IRP5FXA2XR6THXFW27U7IWAZ4JDV5M2WRPXKUGZ4VIA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.304881529 (81h9m22.868204536s) [noise=true , address=79.127.220.101:6048 , piece_id=AN7UQUXB42T2YXEH7ZJRZUDIW47M3RWUEOCL2VEBOGH74BZF3YKA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.305249860 (81h9m22.86857287s) [noise=true , address=104.36.82.54:4165 , piece_id=4CVOABPCWC7J7SRHZB6IN2WJJ6KGWUQ66ELUIZIFA67ETLQ7JEUQ, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.305604108 (81h9m22.868927112s) [noise=true , address=104.36.82.15:6289 , piece_id=GWL3P55BIK4IJEL5YFZKI7JCA4LADYOOHUSIQUNGJEKBJOTKMP5A, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.305930423 (81h9m22.869253425s) [noise=true , address=64.49.13.139:4476 , piece_id=P7UKBNNZGI52B3I6WKTV6MVC262FYNKYYPJUI5G47UO5S4MU7PWA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.306355311 (81h9m22.869678319s) [noise=true , address=104.36.82.55:4213 , piece_id=X4IHS2XSPH52LIMNUYT7KGWMAPUH3TEMJOEEVB3JV5N45SEOH7YA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.306788243 (81h9m22.870111252s) [noise=true , address=104.36.82.22:6539 , piece_id=GHQ2MVSTDGVYLUKLV23D6YWRR4PP7UUYHCUEQ4WWSWEMTBFO3P4A, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.392174770 (81h9m22.955497773s) [upload=64958]: Next returned nil segment Jun 22 01:32:47.392195767 (81h9m22.955518775s) [upload=64958]: Waiting for error group managing segments... Jun 22 01:32:47.480767633 (81h9m23.044090623s) [noise=true , address=104.36.82.22:6539 , piece_id=GHQ2MVSTDGVYLUKLV23D6YWRR4PP7UUYHCUEQ4WWSWEMTBFO3P4A, seg_pos={0 0}, upload=64958]: Done uploading piece. err: ecclient: upload failed (node:1QL6w8DSFR6CxDdnJmDuyVZxp6JU55VKiJofBZRfQi7q31i8Jc, address:104.36.82.22:6539): protocol: storage node overloaded, request limit: 2000 Jun 22 01:32:47.585663142 (81h9m23.148986155s) [noise=true , address=79.127.220.101:6048 , piece_id=AN7UQUXB42T2YXEH7ZJRZUDIW47M3RWUEOCL2VEBOGH74BZF3YKA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:47.640263002 (81h9m23.203586013s) [noise=true , address=45.91.116.12:5242 , piece_id=6ABT5IPYGSLFFJMBYMBYRU4EYIXABUAUHG3OVILOHBIUDS3BP27A, seg_pos={0 0}, upload=64958]: Done uploading piece. err: ecclient: upload failed (node:12JsFL62jgXmza8Md8eQ9k8cyw5V9Vz2RzN4y85ZvgHZjwrq4vQ, address:45.91.116.12:5242): protocol: storage node overloaded, request limit: 2000 Jun 22 01:32:47.984127275 (81h9m23.54745028s) [noise=true , address=64.49.13.137:4500 , piece_id=FAE2RXD57IRP5FXA2XR6THXFW27U7IWAZ4JDV5M2WRPXKUGZ4VIA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:48.100255249 (81h9m23.663578241s) [noise=true , address=77.90.31.14:4689 , piece_id=Q3HDS3BCMYDTODFRMU4XFRDCQK2QZE3ZMXVLUX4M54PDEBS7RR6A, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:48.221764035 (81h9m23.785087034s) [noise=true , address=104.36.82.54:4165 , piece_id=4CVOABPCWC7J7SRHZB6IN2WJJ6KGWUQ66ELUIZIFA67ETLQ7JEUQ, seg_pos={0 0}, upload=64958]: Done uploading piece. err: ecclient: upload failed (node:12wGUMXVEubEtki4qGnjGEsbYFB1iUauZcu3oDRKrCQsJMV3bV2, address:104.36.82.54:4165): protocol: storage node overloaded, request limit: 2000 Jun 22 01:32:48.509996527 (81h9m24.073319547s) [noise=true , address=104.36.82.9:6023 , piece_id=SISFAUU4RPF57OLGWM2BWQWE6GRUSCMORBYHWEXETNAVM2EEMO6Q, seg_pos={0 0}, upload=64958]: Done uploading piece. err: ecclient: upload failed (node:1j2S9tgWpLwmyvXZSq2qKKcbaC8r7LkzfDCaMxCejAni4CST1g, address:104.36.82.9:6023): protocol: storage node overloaded, request limit: 2000 Jun 22 01:32:48.510513169 (81h9m24.073836177s) [noise=true , address=104.36.82.42:6811 , piece_id=A5BFEHZ3WGGK53IZ7MKI6N4CVXHOORF6L47NR3UFDCAX7LBHRHVQ, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:48.565245605 (81h9m24.128568606s) [noise=true , address=79.127.205.234:5601 , piece_id=V7PMXBP2VFWBSZDG4GTLBD435T6F5DT7A43L4GQLQ5BDDTJ4WY6A, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:49.979979824 (81h9m25.543302825s) [noise=true , address=104.36.82.15:6289 , piece_id=GWL3P55BIK4IJEL5YFZKI7JCA4LADYOOHUSIQUNGJEKBJOTKMP5A, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:50.463750280 (81h9m26.02707328s) [noise=true , address=104.36.82.51:4084 , piece_id=D43FA7BFE54KPCWGDGDME2EGSWM7PONOVPD3DN2XY42MJFOKVDIQ, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:51.480599795 (81h9m27.043922798s) [noise=true , address=79.127.219.38:4023 , piece_id=BWLRCXXHBMBTNCZEVKKIIS5PB35PDQXQXCLIQB7GBLEIOWWUFBFA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:52.941200625 (81h9m28.504523627s) [noise=true , address=104.36.82.43:6841 , piece_id=PIHZMSTO3CII7KGDLGSAILTDRSLMEBVOLBCRCZJULMCFISXIZWXA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:58.281803598 (81h9m33.845126612s) [noise=true , address=64.49.13.139:4476 , piece_id=P7UKBNNZGI52B3I6WKTV6MVC262FYNKYYPJUI5G47UO5S4MU7PWA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:33:00.277271877 (81h9m35.840594867s) [noise=true , address=104.36.82.55:4213 , piece_id=X4IHS2XSPH52LIMNUYT7KGWMAPUH3TEMJOEEVB3JV5N45SEOH7YA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:33:09.435339213 (81h9m44.998662206s) [noise=true , address=104.36.82.52:4089 , piece_id=JSOEKLNWA3W3LOVKI3EGGKJYJ6J3UZY6KGABKQPQPXTIOOPMKVHA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:33:31.472868445 (81h10m7.03619143s) [noise=true , address=104.36.82.53:4122 , piece_id=HZZQBIKHQWLUE5EHK4KSKJGIUEG7D44RQASUJHCYXX6BICCGX6EA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:33:47.235404653 (81h10m22.798727642s) [noise=true , address=104.36.82.12:6158 , piece_id=LJZEOJVJCJAR2BRILIJI3ZVDGYMPBBTP2G3HCTN3MHNS4KYDPASQ, seg_pos={0 0}, upload=64958]: Done uploading piece. err: ecclient: upload failed (node:1SjM6Utm2YizeUbrBUD9Z7etE1nLadW5Fa1zYP7MioPeeqhdJ7, address:104.36.82.12:6158): protocol: expected piece hash; context deadline exceeded; EOF Jun 22 01:33:47.235448297 (81h10m22.798771306s) [seg_pos={0 0}, upload=64958]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jun 22 01:33:47.265442393 (81h10m22.828765401s) [upload=64958]: Done uploading segments... Jun 22 01:33:47.265476139 (81h10m22.82879914s) [upload=64958]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration:* 13.991312119s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [........................................................X...] duration: 18.244897936s FAIL (2025-06-22 01:35:30 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== oad=64949]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jun 22 01:32:13.674692303 (81h8m49.238015311s) [upload=64949]: Done uploading segments... Jun 22 01:32:13.674727219 (81h8m49.238050233s) [upload=64949]: Upload finished. err: <nil> Jun 22 01:32:47.266407995 (81h9m22.829731002s) [upload=64958]: Starting upload Jun 22 01:32:47.266524767 (81h9m22.829847771s) [upload=64958]: Done starting upload Jun 22 01:32:47.266813109 (81h9m22.830136116s) [upload=64958]: Uploading segments... Jun 22 01:32:47.266969526 (81h9m22.830292526s) [upload=64958]: Got next segment. Inline: false Jun 22 01:32:47.267075475 (81h9m22.830398475s) [upload=64958]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 22 01:32:47.267081679 (81h9m22.830404667s) [upload=64958]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 22 01:32:47.301770382 (81h9m22.865093389s) [seg_pos={0 0}, upload=64958]: Begin upload segment... Jun 22 01:32:47.301823796 (81h9m22.865146799s) [seg_pos={0 0}, upload=64958]: Done begin upload segment. Jun 22 01:32:47.301855055 (81h9m22.865178087s) [noise=true , address=104.36.82.43:6841 , piece_id=PIHZMSTO3CII7KGDLGSAILTDRSLMEBVOLBCRCZJULMCFISXIZWXA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.301902600 (81h9m22.865225606s) [noise=true , address=104.36.82.42:6811 , piece_id=A5BFEHZ3WGGK53IZ7MKI6N4CVXHOORF6L47NR3UFDCAX7LBHRHVQ, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.302234027 (81h9m22.865557029s) [noise=true , address=104.36.82.51:4084 , piece_id=D43FA7BFE54KPCWGDGDME2EGSWM7PONOVPD3DN2XY42MJFOKVDIQ, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.302496354 (81h9m22.865819363s) [noise=true , address=79.127.219.38:4023 , piece_id=BWLRCXXHBMBTNCZEVKKIIS5PB35PDQXQXCLIQB7GBLEIOWWUFBFA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.302577370 (81h9m22.865900378s) [noise=true , address=104.36.82.52:4089 , piece_id=JSOEKLNWA3W3LOVKI3EGGKJYJ6J3UZY6KGABKQPQPXTIOOPMKVHA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.302917829 (81h9m22.866240835s) [noise=true , address=104.36.82.53:4122 , piece_id=HZZQBIKHQWLUE5EHK4KSKJGIUEG7D44RQASUJHCYXX6BICCGX6EA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.303087165 (81h9m22.866410168s) [noise=true , address=77.90.31.14:4689 , piece_id=Q3HDS3BCMYDTODFRMU4XFRDCQK2QZE3ZMXVLUX4M54PDEBS7RR6A, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.303250714 (81h9m22.866573717s) [noise=true , address=79.127.205.234:5601 , piece_id=V7PMXBP2VFWBSZDG4GTLBD435T6F5DT7A43L4GQLQ5BDDTJ4WY6A, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.303579249 (81h9m22.866902255s) [noise=true , address=104.36.82.12:6158 , piece_id=LJZEOJVJCJAR2BRILIJI3ZVDGYMPBBTP2G3HCTN3MHNS4KYDPASQ, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.303599776 (81h9m22.866922784s) [noise=true , address=104.36.82.9:6023 , piece_id=SISFAUU4RPF57OLGWM2BWQWE6GRUSCMORBYHWEXETNAVM2EEMO6Q, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.304005346 (81h9m22.867328346s) [noise=true , address=45.91.116.12:5242 , piece_id=6ABT5IPYGSLFFJMBYMBYRU4EYIXABUAUHG3OVILOHBIUDS3BP27A, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.304388850 (81h9m22.867711851s) [noise=true , address=64.49.13.137:4500 , piece_id=FAE2RXD57IRP5FXA2XR6THXFW27U7IWAZ4JDV5M2WRPXKUGZ4VIA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.304881529 (81h9m22.868204536s) [noise=true , address=79.127.220.101:6048 , piece_id=AN7UQUXB42T2YXEH7ZJRZUDIW47M3RWUEOCL2VEBOGH74BZF3YKA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.305249860 (81h9m22.86857287s) [noise=true , address=104.36.82.54:4165 , piece_id=4CVOABPCWC7J7SRHZB6IN2WJJ6KGWUQ66ELUIZIFA67ETLQ7JEUQ, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.305604108 (81h9m22.868927112s) [noise=true , address=104.36.82.15:6289 , piece_id=GWL3P55BIK4IJEL5YFZKI7JCA4LADYOOHUSIQUNGJEKBJOTKMP5A, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.305930423 (81h9m22.869253425s) [noise=true , address=64.49.13.139:4476 , piece_id=P7UKBNNZGI52B3I6WKTV6MVC262FYNKYYPJUI5G47UO5S4MU7PWA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.306355311 (81h9m22.869678319s) [noise=true , address=104.36.82.55:4213 , piece_id=X4IHS2XSPH52LIMNUYT7KGWMAPUH3TEMJOEEVB3JV5N45SEOH7YA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.306788243 (81h9m22.870111252s) [noise=true , address=104.36.82.22:6539 , piece_id=GHQ2MVSTDGVYLUKLV23D6YWRR4PP7UUYHCUEQ4WWSWEMTBFO3P4A, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.392174770 (81h9m22.955497773s) [upload=64958]: Next returned nil segment Jun 22 01:32:47.392195767 (81h9m22.955518775s) [upload=64958]: Waiting for error group managing segments... Jun 22 01:32:47.480767633 (81h9m23.044090623s) [noise=true , address=104.36.82.22:6539 , piece_id=GHQ2MVSTDGVYLUKLV23D6YWRR4PP7UUYHCUEQ4WWSWEMTBFO3P4A, seg_pos={0 0}, upload=64958]: Done uploading piece. err: ecclient: upload failed (node:1QL6w8DSFR6CxDdnJmDuyVZxp6JU55VKiJofBZRfQi7q31i8Jc, address:104.36.82.22:6539): protocol: storage node overloaded, request limit: 2000 Jun 22 01:32:47.585663142 (81h9m23.148986155s) [noise=true , address=79.127.220.101:6048 , piece_id=AN7UQUXB42T2YXEH7ZJRZUDIW47M3RWUEOCL2VEBOGH74BZF3YKA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:47.640263002 (81h9m23.203586013s) [noise=true , address=45.91.116.12:5242 , piece_id=6ABT5IPYGSLFFJMBYMBYRU4EYIXABUAUHG3OVILOHBIUDS3BP27A, seg_pos={0 0}, upload=64958]: Done uploading piece. err: ecclient: upload failed (node:12JsFL62jgXmza8Md8eQ9k8cyw5V9Vz2RzN4y85ZvgHZjwrq4vQ, address:45.91.116.12:5242): protocol: storage node overloaded, request limit: 2000 Jun 22 01:32:47.984127275 (81h9m23.54745028s) [noise=true , address=64.49.13.137:4500 , piece_id=FAE2RXD57IRP5FXA2XR6THXFW27U7IWAZ4JDV5M2WRPXKUGZ4VIA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:48.100255249 (81h9m23.663578241s) [noise=true , address=77.90.31.14:4689 , piece_id=Q3HDS3BCMYDTODFRMU4XFRDCQK2QZE3ZMXVLUX4M54PDEBS7RR6A, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:48.221764035 (81h9m23.785087034s) [noise=true , address=104.36.82.54:4165 , piece_id=4CVOABPCWC7J7SRHZB6IN2WJJ6KGWUQ66ELUIZIFA67ETLQ7JEUQ, seg_pos={0 0}, upload=64958]: Done uploading piece. err: ecclient: upload failed (node:12wGUMXVEubEtki4qGnjGEsbYFB1iUauZcu3oDRKrCQsJMV3bV2, address:104.36.82.54:4165): protocol: storage node overloaded, request limit: 2000 Jun 22 01:32:48.509996527 (81h9m24.073319547s) [noise=true , address=104.36.82.9:6023 , piece_id=SISFAUU4RPF57OLGWM2BWQWE6GRUSCMORBYHWEXETNAVM2EEMO6Q, seg_pos={0 0}, upload=64958]: Done uploading piece. err: ecclient: upload failed (node:1j2S9tgWpLwmyvXZSq2qKKcbaC8r7LkzfDCaMxCejAni4CST1g, address:104.36.82.9:6023): protocol: storage node overloaded, request limit: 2000 Jun 22 01:32:48.510513169 (81h9m24.073836177s) [noise=true , address=104.36.82.42:6811 , piece_id=A5BFEHZ3WGGK53IZ7MKI6N4CVXHOORF6L47NR3UFDCAX7LBHRHVQ, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:48.565245605 (81h9m24.128568606s) [noise=true , address=79.127.205.234:5601 , piece_id=V7PMXBP2VFWBSZDG4GTLBD435T6F5DT7A43L4GQLQ5BDDTJ4WY6A, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:49.979979824 (81h9m25.543302825s) [noise=true , address=104.36.82.15:6289 , piece_id=GWL3P55BIK4IJEL5YFZKI7JCA4LADYOOHUSIQUNGJEKBJOTKMP5A, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:50.463750280 (81h9m26.02707328s) [noise=true , address=104.36.82.51:4084 , piece_id=D43FA7BFE54KPCWGDGDME2EGSWM7PONOVPD3DN2XY42MJFOKVDIQ, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:51.480599795 (81h9m27.043922798s) [noise=true , address=79.127.219.38:4023 , piece_id=BWLRCXXHBMBTNCZEVKKIIS5PB35PDQXQXCLIQB7GBLEIOWWUFBFA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:52.941200625 (81h9m28.504523627s) [noise=true , address=104.36.82.43:6841 , piece_id=PIHZMSTO3CII7KGDLGSAILTDRSLMEBVOLBCRCZJULMCFISXIZWXA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:58.281803598 (81h9m33.845126612s) [noise=true , address=64.49.13.139:4476 , piece_id=P7UKBNNZGI52B3I6WKTV6MVC262FYNKYYPJUI5G47UO5S4MU7PWA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:33:00.277271877 (81h9m35.840594867s) [noise=true , address=104.36.82.55:4213 , piece_id=X4IHS2XSPH52LIMNUYT7KGWMAPUH3TEMJOEEVB3JV5N45SEOH7YA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:33:09.435339213 (81h9m44.998662206s) [noise=true , address=104.36.82.52:4089 , piece_id=JSOEKLNWA3W3LOVKI3EGGKJYJ6J3UZY6KGABKQPQPXTIOOPMKVHA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:33:31.472868445 (81h10m7.03619143s) [noise=true , address=104.36.82.53:4122 , piece_id=HZZQBIKHQWLUE5EHK4KSKJGIUEG7D44RQASUJHCYXX6BICCGX6EA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:33:47.235404653 (81h10m22.798727642s) [noise=true , address=104.36.82.12:6158 , piece_id=LJZEOJVJCJAR2BRILIJI3ZVDGYMPBBTP2G3HCTN3MHNS4KYDPASQ, seg_pos={0 0}, upload=64958]: Done uploading piece. err: ecclient: upload failed (node:1SjM6Utm2YizeUbrBUD9Z7etE1nLadW5Fa1zYP7MioPeeqhdJ7, address:104.36.82.12:6158): protocol: expected piece hash; context deadline exceeded; EOF Jun 22 01:33:47.235448297 (81h10m22.798771306s) [seg_pos={0 0}, upload=64958]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jun 22 01:33:47.265442393 (81h10m22.828765401s) [upload=64958]: Done uploading segments... Jun 22 01:33:47.265476139 (81h10m22.82879914s) [upload=64958]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration:* 18.244925788s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.........................................................X..] duration: 12.84335489s FAIL (2025-06-22 01:35:02 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== oad=64949]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jun 22 01:32:13.674692303 (81h8m49.238015311s) [upload=64949]: Done uploading segments... Jun 22 01:32:13.674727219 (81h8m49.238050233s) [upload=64949]: Upload finished. err: <nil> Jun 22 01:32:47.266407995 (81h9m22.829731002s) [upload=64958]: Starting upload Jun 22 01:32:47.266524767 (81h9m22.829847771s) [upload=64958]: Done starting upload Jun 22 01:32:47.266813109 (81h9m22.830136116s) [upload=64958]: Uploading segments... Jun 22 01:32:47.266969526 (81h9m22.830292526s) [upload=64958]: Got next segment. Inline: false Jun 22 01:32:47.267075475 (81h9m22.830398475s) [upload=64958]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 22 01:32:47.267081679 (81h9m22.830404667s) [upload=64958]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 22 01:32:47.301770382 (81h9m22.865093389s) [seg_pos={0 0}, upload=64958]: Begin upload segment... Jun 22 01:32:47.301823796 (81h9m22.865146799s) [seg_pos={0 0}, upload=64958]: Done begin upload segment. Jun 22 01:32:47.301855055 (81h9m22.865178087s) [noise=true , address=104.36.82.43:6841 , piece_id=PIHZMSTO3CII7KGDLGSAILTDRSLMEBVOLBCRCZJULMCFISXIZWXA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.301902600 (81h9m22.865225606s) [noise=true , address=104.36.82.42:6811 , piece_id=A5BFEHZ3WGGK53IZ7MKI6N4CVXHOORF6L47NR3UFDCAX7LBHRHVQ, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.302234027 (81h9m22.865557029s) [noise=true , address=104.36.82.51:4084 , piece_id=D43FA7BFE54KPCWGDGDME2EGSWM7PONOVPD3DN2XY42MJFOKVDIQ, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.302496354 (81h9m22.865819363s) [noise=true , address=79.127.219.38:4023 , piece_id=BWLRCXXHBMBTNCZEVKKIIS5PB35PDQXQXCLIQB7GBLEIOWWUFBFA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.302577370 (81h9m22.865900378s) [noise=true , address=104.36.82.52:4089 , piece_id=JSOEKLNWA3W3LOVKI3EGGKJYJ6J3UZY6KGABKQPQPXTIOOPMKVHA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.302917829 (81h9m22.866240835s) [noise=true , address=104.36.82.53:4122 , piece_id=HZZQBIKHQWLUE5EHK4KSKJGIUEG7D44RQASUJHCYXX6BICCGX6EA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.303087165 (81h9m22.866410168s) [noise=true , address=77.90.31.14:4689 , piece_id=Q3HDS3BCMYDTODFRMU4XFRDCQK2QZE3ZMXVLUX4M54PDEBS7RR6A, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.303250714 (81h9m22.866573717s) [noise=true , address=79.127.205.234:5601 , piece_id=V7PMXBP2VFWBSZDG4GTLBD435T6F5DT7A43L4GQLQ5BDDTJ4WY6A, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.303579249 (81h9m22.866902255s) [noise=true , address=104.36.82.12:6158 , piece_id=LJZEOJVJCJAR2BRILIJI3ZVDGYMPBBTP2G3HCTN3MHNS4KYDPASQ, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.303599776 (81h9m22.866922784s) [noise=true , address=104.36.82.9:6023 , piece_id=SISFAUU4RPF57OLGWM2BWQWE6GRUSCMORBYHWEXETNAVM2EEMO6Q, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.304005346 (81h9m22.867328346s) [noise=true , address=45.91.116.12:5242 , piece_id=6ABT5IPYGSLFFJMBYMBYRU4EYIXABUAUHG3OVILOHBIUDS3BP27A, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.304388850 (81h9m22.867711851s) [noise=true , address=64.49.13.137:4500 , piece_id=FAE2RXD57IRP5FXA2XR6THXFW27U7IWAZ4JDV5M2WRPXKUGZ4VIA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.304881529 (81h9m22.868204536s) [noise=true , address=79.127.220.101:6048 , piece_id=AN7UQUXB42T2YXEH7ZJRZUDIW47M3RWUEOCL2VEBOGH74BZF3YKA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.305249860 (81h9m22.86857287s) [noise=true , address=104.36.82.54:4165 , piece_id=4CVOABPCWC7J7SRHZB6IN2WJJ6KGWUQ66ELUIZIFA67ETLQ7JEUQ, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.305604108 (81h9m22.868927112s) [noise=true , address=104.36.82.15:6289 , piece_id=GWL3P55BIK4IJEL5YFZKI7JCA4LADYOOHUSIQUNGJEKBJOTKMP5A, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.305930423 (81h9m22.869253425s) [noise=true , address=64.49.13.139:4476 , piece_id=P7UKBNNZGI52B3I6WKTV6MVC262FYNKYYPJUI5G47UO5S4MU7PWA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.306355311 (81h9m22.869678319s) [noise=true , address=104.36.82.55:4213 , piece_id=X4IHS2XSPH52LIMNUYT7KGWMAPUH3TEMJOEEVB3JV5N45SEOH7YA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.306788243 (81h9m22.870111252s) [noise=true , address=104.36.82.22:6539 , piece_id=GHQ2MVSTDGVYLUKLV23D6YWRR4PP7UUYHCUEQ4WWSWEMTBFO3P4A, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.392174770 (81h9m22.955497773s) [upload=64958]: Next returned nil segment Jun 22 01:32:47.392195767 (81h9m22.955518775s) [upload=64958]: Waiting for error group managing segments... Jun 22 01:32:47.480767633 (81h9m23.044090623s) [noise=true , address=104.36.82.22:6539 , piece_id=GHQ2MVSTDGVYLUKLV23D6YWRR4PP7UUYHCUEQ4WWSWEMTBFO3P4A, seg_pos={0 0}, upload=64958]: Done uploading piece. err: ecclient: upload failed (node:1QL6w8DSFR6CxDdnJmDuyVZxp6JU55VKiJofBZRfQi7q31i8Jc, address:104.36.82.22:6539): protocol: storage node overloaded, request limit: 2000 Jun 22 01:32:47.585663142 (81h9m23.148986155s) [noise=true , address=79.127.220.101:6048 , piece_id=AN7UQUXB42T2YXEH7ZJRZUDIW47M3RWUEOCL2VEBOGH74BZF3YKA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:47.640263002 (81h9m23.203586013s) [noise=true , address=45.91.116.12:5242 , piece_id=6ABT5IPYGSLFFJMBYMBYRU4EYIXABUAUHG3OVILOHBIUDS3BP27A, seg_pos={0 0}, upload=64958]: Done uploading piece. err: ecclient: upload failed (node:12JsFL62jgXmza8Md8eQ9k8cyw5V9Vz2RzN4y85ZvgHZjwrq4vQ, address:45.91.116.12:5242): protocol: storage node overloaded, request limit: 2000 Jun 22 01:32:47.984127275 (81h9m23.54745028s) [noise=true , address=64.49.13.137:4500 , piece_id=FAE2RXD57IRP5FXA2XR6THXFW27U7IWAZ4JDV5M2WRPXKUGZ4VIA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:48.100255249 (81h9m23.663578241s) [noise=true , address=77.90.31.14:4689 , piece_id=Q3HDS3BCMYDTODFRMU4XFRDCQK2QZE3ZMXVLUX4M54PDEBS7RR6A, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:48.221764035 (81h9m23.785087034s) [noise=true , address=104.36.82.54:4165 , piece_id=4CVOABPCWC7J7SRHZB6IN2WJJ6KGWUQ66ELUIZIFA67ETLQ7JEUQ, seg_pos={0 0}, upload=64958]: Done uploading piece. err: ecclient: upload failed (node:12wGUMXVEubEtki4qGnjGEsbYFB1iUauZcu3oDRKrCQsJMV3bV2, address:104.36.82.54:4165): protocol: storage node overloaded, request limit: 2000 Jun 22 01:32:48.509996527 (81h9m24.073319547s) [noise=true , address=104.36.82.9:6023 , piece_id=SISFAUU4RPF57OLGWM2BWQWE6GRUSCMORBYHWEXETNAVM2EEMO6Q, seg_pos={0 0}, upload=64958]: Done uploading piece. err: ecclient: upload failed (node:1j2S9tgWpLwmyvXZSq2qKKcbaC8r7LkzfDCaMxCejAni4CST1g, address:104.36.82.9:6023): protocol: storage node overloaded, request limit: 2000 Jun 22 01:32:48.510513169 (81h9m24.073836177s) [noise=true , address=104.36.82.42:6811 , piece_id=A5BFEHZ3WGGK53IZ7MKI6N4CVXHOORF6L47NR3UFDCAX7LBHRHVQ, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:48.565245605 (81h9m24.128568606s) [noise=true , address=79.127.205.234:5601 , piece_id=V7PMXBP2VFWBSZDG4GTLBD435T6F5DT7A43L4GQLQ5BDDTJ4WY6A, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:49.979979824 (81h9m25.543302825s) [noise=true , address=104.36.82.15:6289 , piece_id=GWL3P55BIK4IJEL5YFZKI7JCA4LADYOOHUSIQUNGJEKBJOTKMP5A, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:50.463750280 (81h9m26.02707328s) [noise=true , address=104.36.82.51:4084 , piece_id=D43FA7BFE54KPCWGDGDME2EGSWM7PONOVPD3DN2XY42MJFOKVDIQ, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:51.480599795 (81h9m27.043922798s) [noise=true , address=79.127.219.38:4023 , piece_id=BWLRCXXHBMBTNCZEVKKIIS5PB35PDQXQXCLIQB7GBLEIOWWUFBFA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:52.941200625 (81h9m28.504523627s) [noise=true , address=104.36.82.43:6841 , piece_id=PIHZMSTO3CII7KGDLGSAILTDRSLMEBVOLBCRCZJULMCFISXIZWXA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:58.281803598 (81h9m33.845126612s) [noise=true , address=64.49.13.139:4476 , piece_id=P7UKBNNZGI52B3I6WKTV6MVC262FYNKYYPJUI5G47UO5S4MU7PWA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:33:00.277271877 (81h9m35.840594867s) [noise=true , address=104.36.82.55:4213 , piece_id=X4IHS2XSPH52LIMNUYT7KGWMAPUH3TEMJOEEVB3JV5N45SEOH7YA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:33:09.435339213 (81h9m44.998662206s) [noise=true , address=104.36.82.52:4089 , piece_id=JSOEKLNWA3W3LOVKI3EGGKJYJ6J3UZY6KGABKQPQPXTIOOPMKVHA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:33:31.472868445 (81h10m7.03619143s) [noise=true , address=104.36.82.53:4122 , piece_id=HZZQBIKHQWLUE5EHK4KSKJGIUEG7D44RQASUJHCYXX6BICCGX6EA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:33:47.235404653 (81h10m22.798727642s) [noise=true , address=104.36.82.12:6158 , piece_id=LJZEOJVJCJAR2BRILIJI3ZVDGYMPBBTP2G3HCTN3MHNS4KYDPASQ, seg_pos={0 0}, upload=64958]: Done uploading piece. err: ecclient: upload failed (node:1SjM6Utm2YizeUbrBUD9Z7etE1nLadW5Fa1zYP7MioPeeqhdJ7, address:104.36.82.12:6158): protocol: expected piece hash; context deadline exceeded; EOF Jun 22 01:33:47.235448297 (81h10m22.798771306s) [seg_pos={0 0}, upload=64958]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jun 22 01:33:47.265442393 (81h10m22.828765401s) [upload=64958]: Done uploading segments... Jun 22 01:33:47.265476139 (81h10m22.82879914s) [upload=64958]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration:* 12.843384121s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [..........................................................X.] duration: 20.619723644s FAIL (2025-06-22 01:34:07 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== oad=64949]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jun 22 01:32:13.674692303 (81h8m49.238015311s) [upload=64949]: Done uploading segments... Jun 22 01:32:13.674727219 (81h8m49.238050233s) [upload=64949]: Upload finished. err: <nil> Jun 22 01:32:47.266407995 (81h9m22.829731002s) [upload=64958]: Starting upload Jun 22 01:32:47.266524767 (81h9m22.829847771s) [upload=64958]: Done starting upload Jun 22 01:32:47.266813109 (81h9m22.830136116s) [upload=64958]: Uploading segments... Jun 22 01:32:47.266969526 (81h9m22.830292526s) [upload=64958]: Got next segment. Inline: false Jun 22 01:32:47.267075475 (81h9m22.830398475s) [upload=64958]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 22 01:32:47.267081679 (81h9m22.830404667s) [upload=64958]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 22 01:32:47.301770382 (81h9m22.865093389s) [seg_pos={0 0}, upload=64958]: Begin upload segment... Jun 22 01:32:47.301823796 (81h9m22.865146799s) [seg_pos={0 0}, upload=64958]: Done begin upload segment. Jun 22 01:32:47.301855055 (81h9m22.865178087s) [noise=true , address=104.36.82.43:6841 , piece_id=PIHZMSTO3CII7KGDLGSAILTDRSLMEBVOLBCRCZJULMCFISXIZWXA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.301902600 (81h9m22.865225606s) [noise=true , address=104.36.82.42:6811 , piece_id=A5BFEHZ3WGGK53IZ7MKI6N4CVXHOORF6L47NR3UFDCAX7LBHRHVQ, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.302234027 (81h9m22.865557029s) [noise=true , address=104.36.82.51:4084 , piece_id=D43FA7BFE54KPCWGDGDME2EGSWM7PONOVPD3DN2XY42MJFOKVDIQ, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.302496354 (81h9m22.865819363s) [noise=true , address=79.127.219.38:4023 , piece_id=BWLRCXXHBMBTNCZEVKKIIS5PB35PDQXQXCLIQB7GBLEIOWWUFBFA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.302577370 (81h9m22.865900378s) [noise=true , address=104.36.82.52:4089 , piece_id=JSOEKLNWA3W3LOVKI3EGGKJYJ6J3UZY6KGABKQPQPXTIOOPMKVHA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.302917829 (81h9m22.866240835s) [noise=true , address=104.36.82.53:4122 , piece_id=HZZQBIKHQWLUE5EHK4KSKJGIUEG7D44RQASUJHCYXX6BICCGX6EA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.303087165 (81h9m22.866410168s) [noise=true , address=77.90.31.14:4689 , piece_id=Q3HDS3BCMYDTODFRMU4XFRDCQK2QZE3ZMXVLUX4M54PDEBS7RR6A, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.303250714 (81h9m22.866573717s) [noise=true , address=79.127.205.234:5601 , piece_id=V7PMXBP2VFWBSZDG4GTLBD435T6F5DT7A43L4GQLQ5BDDTJ4WY6A, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.303579249 (81h9m22.866902255s) [noise=true , address=104.36.82.12:6158 , piece_id=LJZEOJVJCJAR2BRILIJI3ZVDGYMPBBTP2G3HCTN3MHNS4KYDPASQ, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.303599776 (81h9m22.866922784s) [noise=true , address=104.36.82.9:6023 , piece_id=SISFAUU4RPF57OLGWM2BWQWE6GRUSCMORBYHWEXETNAVM2EEMO6Q, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.304005346 (81h9m22.867328346s) [noise=true , address=45.91.116.12:5242 , piece_id=6ABT5IPYGSLFFJMBYMBYRU4EYIXABUAUHG3OVILOHBIUDS3BP27A, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.304388850 (81h9m22.867711851s) [noise=true , address=64.49.13.137:4500 , piece_id=FAE2RXD57IRP5FXA2XR6THXFW27U7IWAZ4JDV5M2WRPXKUGZ4VIA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.304881529 (81h9m22.868204536s) [noise=true , address=79.127.220.101:6048 , piece_id=AN7UQUXB42T2YXEH7ZJRZUDIW47M3RWUEOCL2VEBOGH74BZF3YKA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.305249860 (81h9m22.86857287s) [noise=true , address=104.36.82.54:4165 , piece_id=4CVOABPCWC7J7SRHZB6IN2WJJ6KGWUQ66ELUIZIFA67ETLQ7JEUQ, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.305604108 (81h9m22.868927112s) [noise=true , address=104.36.82.15:6289 , piece_id=GWL3P55BIK4IJEL5YFZKI7JCA4LADYOOHUSIQUNGJEKBJOTKMP5A, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.305930423 (81h9m22.869253425s) [noise=true , address=64.49.13.139:4476 , piece_id=P7UKBNNZGI52B3I6WKTV6MVC262FYNKYYPJUI5G47UO5S4MU7PWA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.306355311 (81h9m22.869678319s) [noise=true , address=104.36.82.55:4213 , piece_id=X4IHS2XSPH52LIMNUYT7KGWMAPUH3TEMJOEEVB3JV5N45SEOH7YA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.306788243 (81h9m22.870111252s) [noise=true , address=104.36.82.22:6539 , piece_id=GHQ2MVSTDGVYLUKLV23D6YWRR4PP7UUYHCUEQ4WWSWEMTBFO3P4A, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.392174770 (81h9m22.955497773s) [upload=64958]: Next returned nil segment Jun 22 01:32:47.392195767 (81h9m22.955518775s) [upload=64958]: Waiting for error group managing segments... Jun 22 01:32:47.480767633 (81h9m23.044090623s) [noise=true , address=104.36.82.22:6539 , piece_id=GHQ2MVSTDGVYLUKLV23D6YWRR4PP7UUYHCUEQ4WWSWEMTBFO3P4A, seg_pos={0 0}, upload=64958]: Done uploading piece. err: ecclient: upload failed (node:1QL6w8DSFR6CxDdnJmDuyVZxp6JU55VKiJofBZRfQi7q31i8Jc, address:104.36.82.22:6539): protocol: storage node overloaded, request limit: 2000 Jun 22 01:32:47.585663142 (81h9m23.148986155s) [noise=true , address=79.127.220.101:6048 , piece_id=AN7UQUXB42T2YXEH7ZJRZUDIW47M3RWUEOCL2VEBOGH74BZF3YKA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:47.640263002 (81h9m23.203586013s) [noise=true , address=45.91.116.12:5242 , piece_id=6ABT5IPYGSLFFJMBYMBYRU4EYIXABUAUHG3OVILOHBIUDS3BP27A, seg_pos={0 0}, upload=64958]: Done uploading piece. err: ecclient: upload failed (node:12JsFL62jgXmza8Md8eQ9k8cyw5V9Vz2RzN4y85ZvgHZjwrq4vQ, address:45.91.116.12:5242): protocol: storage node overloaded, request limit: 2000 Jun 22 01:32:47.984127275 (81h9m23.54745028s) [noise=true , address=64.49.13.137:4500 , piece_id=FAE2RXD57IRP5FXA2XR6THXFW27U7IWAZ4JDV5M2WRPXKUGZ4VIA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:48.100255249 (81h9m23.663578241s) [noise=true , address=77.90.31.14:4689 , piece_id=Q3HDS3BCMYDTODFRMU4XFRDCQK2QZE3ZMXVLUX4M54PDEBS7RR6A, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:48.221764035 (81h9m23.785087034s) [noise=true , address=104.36.82.54:4165 , piece_id=4CVOABPCWC7J7SRHZB6IN2WJJ6KGWUQ66ELUIZIFA67ETLQ7JEUQ, seg_pos={0 0}, upload=64958]: Done uploading piece. err: ecclient: upload failed (node:12wGUMXVEubEtki4qGnjGEsbYFB1iUauZcu3oDRKrCQsJMV3bV2, address:104.36.82.54:4165): protocol: storage node overloaded, request limit: 2000 Jun 22 01:32:48.509996527 (81h9m24.073319547s) [noise=true , address=104.36.82.9:6023 , piece_id=SISFAUU4RPF57OLGWM2BWQWE6GRUSCMORBYHWEXETNAVM2EEMO6Q, seg_pos={0 0}, upload=64958]: Done uploading piece. err: ecclient: upload failed (node:1j2S9tgWpLwmyvXZSq2qKKcbaC8r7LkzfDCaMxCejAni4CST1g, address:104.36.82.9:6023): protocol: storage node overloaded, request limit: 2000 Jun 22 01:32:48.510513169 (81h9m24.073836177s) [noise=true , address=104.36.82.42:6811 , piece_id=A5BFEHZ3WGGK53IZ7MKI6N4CVXHOORF6L47NR3UFDCAX7LBHRHVQ, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:48.565245605 (81h9m24.128568606s) [noise=true , address=79.127.205.234:5601 , piece_id=V7PMXBP2VFWBSZDG4GTLBD435T6F5DT7A43L4GQLQ5BDDTJ4WY6A, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:49.979979824 (81h9m25.543302825s) [noise=true , address=104.36.82.15:6289 , piece_id=GWL3P55BIK4IJEL5YFZKI7JCA4LADYOOHUSIQUNGJEKBJOTKMP5A, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:50.463750280 (81h9m26.02707328s) [noise=true , address=104.36.82.51:4084 , piece_id=D43FA7BFE54KPCWGDGDME2EGSWM7PONOVPD3DN2XY42MJFOKVDIQ, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:51.480599795 (81h9m27.043922798s) [noise=true , address=79.127.219.38:4023 , piece_id=BWLRCXXHBMBTNCZEVKKIIS5PB35PDQXQXCLIQB7GBLEIOWWUFBFA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:52.941200625 (81h9m28.504523627s) [noise=true , address=104.36.82.43:6841 , piece_id=PIHZMSTO3CII7KGDLGSAILTDRSLMEBVOLBCRCZJULMCFISXIZWXA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:58.281803598 (81h9m33.845126612s) [noise=true , address=64.49.13.139:4476 , piece_id=P7UKBNNZGI52B3I6WKTV6MVC262FYNKYYPJUI5G47UO5S4MU7PWA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:33:00.277271877 (81h9m35.840594867s) [noise=true , address=104.36.82.55:4213 , piece_id=X4IHS2XSPH52LIMNUYT7KGWMAPUH3TEMJOEEVB3JV5N45SEOH7YA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:33:09.435339213 (81h9m44.998662206s) [noise=true , address=104.36.82.52:4089 , piece_id=JSOEKLNWA3W3LOVKI3EGGKJYJ6J3UZY6KGABKQPQPXTIOOPMKVHA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:33:31.472868445 (81h10m7.03619143s) [noise=true , address=104.36.82.53:4122 , piece_id=HZZQBIKHQWLUE5EHK4KSKJGIUEG7D44RQASUJHCYXX6BICCGX6EA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:33:47.235404653 (81h10m22.798727642s) [noise=true , address=104.36.82.12:6158 , piece_id=LJZEOJVJCJAR2BRILIJI3ZVDGYMPBBTP2G3HCTN3MHNS4KYDPASQ, seg_pos={0 0}, upload=64958]: Done uploading piece. err: ecclient: upload failed (node:1SjM6Utm2YizeUbrBUD9Z7etE1nLadW5Fa1zYP7MioPeeqhdJ7, address:104.36.82.12:6158): protocol: expected piece hash; context deadline exceeded; EOF Jun 22 01:33:47.235448297 (81h10m22.798771306s) [seg_pos={0 0}, upload=64958]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jun 22 01:33:47.265442393 (81h10m22.828765401s) [upload=64958]: Done uploading segments... Jun 22 01:33:47.265476139 (81h10m22.82879914s) [upload=64958]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration:* 20.619745381s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [...........................................................X] duration: 1m0.032267347s FAIL (2025-06-22 01:33:47 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== oad=64949]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jun 22 01:32:13.674692303 (81h8m49.238015311s) [upload=64949]: Done uploading segments... Jun 22 01:32:13.674727219 (81h8m49.238050233s) [upload=64949]: Upload finished. err: <nil> Jun 22 01:32:47.266407995 (81h9m22.829731002s) [upload=64958]: Starting upload Jun 22 01:32:47.266524767 (81h9m22.829847771s) [upload=64958]: Done starting upload Jun 22 01:32:47.266813109 (81h9m22.830136116s) [upload=64958]: Uploading segments... Jun 22 01:32:47.266969526 (81h9m22.830292526s) [upload=64958]: Got next segment. Inline: false Jun 22 01:32:47.267075475 (81h9m22.830398475s) [upload=64958]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 22 01:32:47.267081679 (81h9m22.830404667s) [upload=64958]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 22 01:32:47.301770382 (81h9m22.865093389s) [seg_pos={0 0}, upload=64958]: Begin upload segment... Jun 22 01:32:47.301823796 (81h9m22.865146799s) [seg_pos={0 0}, upload=64958]: Done begin upload segment. Jun 22 01:32:47.301855055 (81h9m22.865178087s) [noise=true , address=104.36.82.43:6841 , piece_id=PIHZMSTO3CII7KGDLGSAILTDRSLMEBVOLBCRCZJULMCFISXIZWXA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.301902600 (81h9m22.865225606s) [noise=true , address=104.36.82.42:6811 , piece_id=A5BFEHZ3WGGK53IZ7MKI6N4CVXHOORF6L47NR3UFDCAX7LBHRHVQ, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.302234027 (81h9m22.865557029s) [noise=true , address=104.36.82.51:4084 , piece_id=D43FA7BFE54KPCWGDGDME2EGSWM7PONOVPD3DN2XY42MJFOKVDIQ, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.302496354 (81h9m22.865819363s) [noise=true , address=79.127.219.38:4023 , piece_id=BWLRCXXHBMBTNCZEVKKIIS5PB35PDQXQXCLIQB7GBLEIOWWUFBFA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.302577370 (81h9m22.865900378s) [noise=true , address=104.36.82.52:4089 , piece_id=JSOEKLNWA3W3LOVKI3EGGKJYJ6J3UZY6KGABKQPQPXTIOOPMKVHA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.302917829 (81h9m22.866240835s) [noise=true , address=104.36.82.53:4122 , piece_id=HZZQBIKHQWLUE5EHK4KSKJGIUEG7D44RQASUJHCYXX6BICCGX6EA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.303087165 (81h9m22.866410168s) [noise=true , address=77.90.31.14:4689 , piece_id=Q3HDS3BCMYDTODFRMU4XFRDCQK2QZE3ZMXVLUX4M54PDEBS7RR6A, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.303250714 (81h9m22.866573717s) [noise=true , address=79.127.205.234:5601 , piece_id=V7PMXBP2VFWBSZDG4GTLBD435T6F5DT7A43L4GQLQ5BDDTJ4WY6A, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.303579249 (81h9m22.866902255s) [noise=true , address=104.36.82.12:6158 , piece_id=LJZEOJVJCJAR2BRILIJI3ZVDGYMPBBTP2G3HCTN3MHNS4KYDPASQ, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.303599776 (81h9m22.866922784s) [noise=true , address=104.36.82.9:6023 , piece_id=SISFAUU4RPF57OLGWM2BWQWE6GRUSCMORBYHWEXETNAVM2EEMO6Q, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.304005346 (81h9m22.867328346s) [noise=true , address=45.91.116.12:5242 , piece_id=6ABT5IPYGSLFFJMBYMBYRU4EYIXABUAUHG3OVILOHBIUDS3BP27A, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.304388850 (81h9m22.867711851s) [noise=true , address=64.49.13.137:4500 , piece_id=FAE2RXD57IRP5FXA2XR6THXFW27U7IWAZ4JDV5M2WRPXKUGZ4VIA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.304881529 (81h9m22.868204536s) [noise=true , address=79.127.220.101:6048 , piece_id=AN7UQUXB42T2YXEH7ZJRZUDIW47M3RWUEOCL2VEBOGH74BZF3YKA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.305249860 (81h9m22.86857287s) [noise=true , address=104.36.82.54:4165 , piece_id=4CVOABPCWC7J7SRHZB6IN2WJJ6KGWUQ66ELUIZIFA67ETLQ7JEUQ, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.305604108 (81h9m22.868927112s) [noise=true , address=104.36.82.15:6289 , piece_id=GWL3P55BIK4IJEL5YFZKI7JCA4LADYOOHUSIQUNGJEKBJOTKMP5A, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.305930423 (81h9m22.869253425s) [noise=true , address=64.49.13.139:4476 , piece_id=P7UKBNNZGI52B3I6WKTV6MVC262FYNKYYPJUI5G47UO5S4MU7PWA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.306355311 (81h9m22.869678319s) [noise=true , address=104.36.82.55:4213 , piece_id=X4IHS2XSPH52LIMNUYT7KGWMAPUH3TEMJOEEVB3JV5N45SEOH7YA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.306788243 (81h9m22.870111252s) [noise=true , address=104.36.82.22:6539 , piece_id=GHQ2MVSTDGVYLUKLV23D6YWRR4PP7UUYHCUEQ4WWSWEMTBFO3P4A, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.392174770 (81h9m22.955497773s) [upload=64958]: Next returned nil segment Jun 22 01:32:47.392195767 (81h9m22.955518775s) [upload=64958]: Waiting for error group managing segments... Jun 22 01:32:47.480767633 (81h9m23.044090623s) [noise=true , address=104.36.82.22:6539 , piece_id=GHQ2MVSTDGVYLUKLV23D6YWRR4PP7UUYHCUEQ4WWSWEMTBFO3P4A, seg_pos={0 0}, upload=64958]: Done uploading piece. err: ecclient: upload failed (node:1QL6w8DSFR6CxDdnJmDuyVZxp6JU55VKiJofBZRfQi7q31i8Jc, address:104.36.82.22:6539): protocol: storage node overloaded, request limit: 2000 Jun 22 01:32:47.585663142 (81h9m23.148986155s) [noise=true , address=79.127.220.101:6048 , piece_id=AN7UQUXB42T2YXEH7ZJRZUDIW47M3RWUEOCL2VEBOGH74BZF3YKA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:47.640263002 (81h9m23.203586013s) [noise=true , address=45.91.116.12:5242 , piece_id=6ABT5IPYGSLFFJMBYMBYRU4EYIXABUAUHG3OVILOHBIUDS3BP27A, seg_pos={0 0}, upload=64958]: Done uploading piece. err: ecclient: upload failed (node:12JsFL62jgXmza8Md8eQ9k8cyw5V9Vz2RzN4y85ZvgHZjwrq4vQ, address:45.91.116.12:5242): protocol: storage node overloaded, request limit: 2000 Jun 22 01:32:47.984127275 (81h9m23.54745028s) [noise=true , address=64.49.13.137:4500 , piece_id=FAE2RXD57IRP5FXA2XR6THXFW27U7IWAZ4JDV5M2WRPXKUGZ4VIA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:48.100255249 (81h9m23.663578241s) [noise=true , address=77.90.31.14:4689 , piece_id=Q3HDS3BCMYDTODFRMU4XFRDCQK2QZE3ZMXVLUX4M54PDEBS7RR6A, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:48.221764035 (81h9m23.785087034s) [noise=true , address=104.36.82.54:4165 , piece_id=4CVOABPCWC7J7SRHZB6IN2WJJ6KGWUQ66ELUIZIFA67ETLQ7JEUQ, seg_pos={0 0}, upload=64958]: Done uploading piece. err: ecclient: upload failed (node:12wGUMXVEubEtki4qGnjGEsbYFB1iUauZcu3oDRKrCQsJMV3bV2, address:104.36.82.54:4165): protocol: storage node overloaded, request limit: 2000 Jun 22 01:32:48.509996527 (81h9m24.073319547s) [noise=true , address=104.36.82.9:6023 , piece_id=SISFAUU4RPF57OLGWM2BWQWE6GRUSCMORBYHWEXETNAVM2EEMO6Q, seg_pos={0 0}, upload=64958]: Done uploading piece. err: ecclient: upload failed (node:1j2S9tgWpLwmyvXZSq2qKKcbaC8r7LkzfDCaMxCejAni4CST1g, address:104.36.82.9:6023): protocol: storage node overloaded, request limit: 2000 Jun 22 01:32:48.510513169 (81h9m24.073836177s) [noise=true , address=104.36.82.42:6811 , piece_id=A5BFEHZ3WGGK53IZ7MKI6N4CVXHOORF6L47NR3UFDCAX7LBHRHVQ, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:48.565245605 (81h9m24.128568606s) [noise=true , address=79.127.205.234:5601 , piece_id=V7PMXBP2VFWBSZDG4GTLBD435T6F5DT7A43L4GQLQ5BDDTJ4WY6A, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:49.979979824 (81h9m25.543302825s) [noise=true , address=104.36.82.15:6289 , piece_id=GWL3P55BIK4IJEL5YFZKI7JCA4LADYOOHUSIQUNGJEKBJOTKMP5A, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:50.463750280 (81h9m26.02707328s) [noise=true , address=104.36.82.51:4084 , piece_id=D43FA7BFE54KPCWGDGDME2EGSWM7PONOVPD3DN2XY42MJFOKVDIQ, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:51.480599795 (81h9m27.043922798s) [noise=true , address=79.127.219.38:4023 , piece_id=BWLRCXXHBMBTNCZEVKKIIS5PB35PDQXQXCLIQB7GBLEIOWWUFBFA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:52.941200625 (81h9m28.504523627s) [noise=true , address=104.36.82.43:6841 , piece_id=PIHZMSTO3CII7KGDLGSAILTDRSLMEBVOLBCRCZJULMCFISXIZWXA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:58.281803598 (81h9m33.845126612s) [noise=true , address=64.49.13.139:4476 , piece_id=P7UKBNNZGI52B3I6WKTV6MVC262FYNKYYPJUI5G47UO5S4MU7PWA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:33:00.277271877 (81h9m35.840594867s) [noise=true , address=104.36.82.55:4213 , piece_id=X4IHS2XSPH52LIMNUYT7KGWMAPUH3TEMJOEEVB3JV5N45SEOH7YA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:33:09.435339213 (81h9m44.998662206s) [noise=true , address=104.36.82.52:4089 , piece_id=JSOEKLNWA3W3LOVKI3EGGKJYJ6J3UZY6KGABKQPQPXTIOOPMKVHA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:33:31.472868445 (81h10m7.03619143s) [noise=true , address=104.36.82.53:4122 , piece_id=HZZQBIKHQWLUE5EHK4KSKJGIUEG7D44RQASUJHCYXX6BICCGX6EA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:33:47.235404653 (81h10m22.798727642s) [noise=true , address=104.36.82.12:6158 , piece_id=LJZEOJVJCJAR2BRILIJI3ZVDGYMPBBTP2G3HCTN3MHNS4KYDPASQ, seg_pos={0 0}, upload=64958]: Done uploading piece. err: ecclient: upload failed (node:1SjM6Utm2YizeUbrBUD9Z7etE1nLadW5Fa1zYP7MioPeeqhdJ7, address:104.36.82.12:6158): protocol: expected piece hash; context deadline exceeded; EOF Jun 22 01:33:47.235448297 (81h10m22.798771306s) [seg_pos={0 0}, upload=64958]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jun 22 01:33:47.265442393 (81h10m22.828765401s) [upload=64958]: Done uploading segments... Jun 22 01:33:47.265476139 (81h10m22.82879914s) [upload=64958]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration:* 1m0.032314731s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [X...........................................................] duration: 15.816073733s FAIL (2025-06-22 01:19:29 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== ing upload Jun 22 00:30:55.303646184 (80h7m30.866969171s) [upload=64142]: Uploading segments... Jun 22 00:30:55.303706406 (80h7m30.867029416s) [upload=64142]: Got next segment. Inline: false Jun 22 00:30:55.303762479 (80h7m30.867085468s) [upload=64142]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 22 00:30:55.303768627 (80h7m30.867091636s) [upload=64142]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 22 00:30:55.349058140 (80h7m30.912381168s) [seg_pos={0 0}, upload=64142]: Begin upload segment... Jun 22 00:30:55.349113768 (80h7m30.912436772s) [seg_pos={0 0}, upload=64142]: Done begin upload segment. Jun 22 00:30:55.349143121 (80h7m30.912466131s) [noise=true , address=45.91.116.5:5310 , piece_id=AXJOKDZRPVNL7HFT4QKJWRAHFM2UCM3GEVQB2KNYD6CXB6TH4D4Q, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.349660917 (80h7m30.912983928s) [noise=true , address=77.90.31.10:4533 , piece_id=SE6YAKKFDL3BV677CDDUC32UNMPXJHTG23GWWYK5B4YIPKER6GAQ, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.350093077 (80h7m30.913416089s) [noise=true , address=79.127.201.215:4326 , piece_id=USEWZDX6P52U6MCALTBNQG3SV7N2JRKNU6O23TLS6JEWAS4D7ARA, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.350480894 (80h7m30.913803916s) [noise=true , address=104.36.82.13:6199 , piece_id=YP62N2TO446YR3QRG5YLXECD7TN6SJ4UNNLZSK5H4CMV6JIF4AKA, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.350669998 (80h7m30.913993011s) [noise=true , address=104.36.82.54:4184 , piece_id=IZICVGZFMAYLIJTNUJ2OFLZVNZ6IPPYDZO4BSTJJPLHVCVOE5EAA, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.351201352 (80h7m30.914524366s) [noise=true , address=104.36.82.55:4224 , piece_id=XF7H2BYSNNNZOS2T77VIN75N76AYUXAQEI6D4LRXACU2KU62V2TA, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.351668071 (80h7m30.914991083s) [noise=true , address=104.36.82.56:4261 , piece_id=EYOYOYLDACGMPGJYX6ILQT3XHMUTQNXZEDC7ANHLZXIKYVDQFFVQ, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.352193126 (80h7m30.915516138s) [noise=true , address=104.36.82.52:4089 , piece_id=X2VG4S3UHVNWA4AQJS7FLKWNVLTOZ3AKWI5ZDH24QCA3ENQW7BXQ, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.352587096 (80h7m30.915910103s) [noise=true , address=79.127.220.98:4402 , piece_id=TE2WQO5LK45AVMPIAIGVSIBOVZLC4FG4ESQ72X6KFYC7IAEN2DRA, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.353420253 (80h7m30.916743265s) [noise=true , address=104.36.82.11:6125 , piece_id=2CX7RJOKNQA6ODZZZXYPSCYAQY5P35TWX7I56CMYAPFLJE3JEO5Q, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.353791565 (80h7m30.917114578s) [noise=true , address=104.36.82.42:6803 , piece_id=R626UFD326S2UBJEZ2WMHVR7EN4SK427VXPDJZOOZ32MX25JWZWQ, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.354171226 (80h7m30.917494233s) [noise=true , address=104.36.82.14:6245 , piece_id=II36G46OA6TINLR244U6D7Q6LGV754UA5M62K3EX7ZOBESMJUEHA, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.354542282 (80h7m30.917865288s) [noise=true , address=79.127.205.235:5660 , piece_id=UGFPVS7VNB7U2YSFPSGYDIRZV5R2DHCN57PJP6VOXO52WIAYHLXA, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.354891655 (80h7m30.918214678s) [noise=true , address=104.36.82.15:6283 , piece_id=N7TH2EMIOPATVHCSNH2WXUGU7KMMEJ5LCHR22SWEHSA5OV4CUCXQ, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.355499503 (80h7m30.918822519s) [noise=true , address=77.90.31.14:4666 , piece_id=H4DTFDAVCN266PB235BJIQTQ32FBXTIB5AZSX3B4R2VIK554AFSQ, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.356575547 (80h7m30.919898539s) [noise=true , address=104.36.82.39:6756 , piece_id=OKKWMDJPZO6D5VRMGOE3GOCYGHM725JZTNOVYAWKJW3WYZPI4V5Q, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.357039590 (80h7m30.920362596s) [noise=true , address=104.36.82.53:4146 , piece_id=3V3DVGUOPJKDWIGQ2TU7M6VFYFVAS6ODY62ERGSU6LI3LSTWQ4YQ, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.360343566 (80h7m30.923666582s) [noise=true , address=104.36.82.9:6033 , piece_id=T5XA23LZELMSTWOLBSHOX4AHKDACF65OYCKTMZYTXDIBSXNA2T4A, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.452444064 (80h7m31.015767055s) [upload=64142]: Next returned nil segment Jun 22 00:30:55.452495148 (80h7m31.015818149s) [upload=64142]: Waiting for error group managing segments... Jun 22 00:30:55.532958808 (80h7m31.096281883s) [noise=true , address=104.36.82.9:6033 , piece_id=T5XA23LZELMSTWOLBSHOX4AHKDACF65OYCKTMZYTXDIBSXNA2T4A, seg_pos={0 0}, upload=64142]: Done uploading piece. err: ecclient: upload failed (node:1SV3LrcQW16wLewDzfyUWFB6sUgnERNJVdhz72CJmyde95M79C, address:104.36.82.9:6033): protocol: storage node overloaded, request limit: 2000 Jun 22 00:30:55.547802184 (80h7m31.111125188s) [noise=true , address=79.127.220.98:4402 , piece_id=TE2WQO5LK45AVMPIAIGVSIBOVZLC4FG4ESQ72X6KFYC7IAEN2DRA, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:30:55.554223621 (80h7m31.117546616s) [noise=true , address=77.90.31.14:4666 , piece_id=H4DTFDAVCN266PB235BJIQTQ32FBXTIB5AZSX3B4R2VIK554AFSQ, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:30:55.762514255 (80h7m31.325837245s) [noise=true , address=104.36.82.14:6245 , piece_id=II36G46OA6TINLR244U6D7Q6LGV754UA5M62K3EX7ZOBESMJUEHA, seg_pos={0 0}, upload=64142]: Done uploading piece. err: ecclient: upload failed (node:1Zu8teNPTXn638grHKwtJqz4UEY7oFYLx1DHvuVP6RZXcatVqX, address:104.36.82.14:6245): protocol: storage node overloaded, request limit: 2000 Jun 22 00:30:55.773669583 (80h7m31.3369926s) [noise=true , address=79.127.205.235:5660 , piece_id=UGFPVS7VNB7U2YSFPSGYDIRZV5R2DHCN57PJP6VOXO52WIAYHLXA, seg_pos={0 0}, upload=64142]: Done uploading piece. err: ecclient: upload failed (node:12E6uCX9KL4Ud7YpxMXepvMXuqTFJCiSXByDjW94Am3U9H2Nf7f, address:79.127.205.235:5660): protocol: storage node overloaded, request limit: 2000 Jun 22 00:30:56.968366655 (80h7m32.53168964s) [noise=true , address=77.90.31.10:4533 , piece_id=SE6YAKKFDL3BV677CDDUC32UNMPXJHTG23GWWYK5B4YIPKER6GAQ, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:30:57.403572874 (80h7m32.966895883s) [noise=true , address=45.91.116.5:5310 , piece_id=AXJOKDZRPVNL7HFT4QKJWRAHFM2UCM3GEVQB2KNYD6CXB6TH4D4Q, seg_pos={0 0}, upload=64142]: Done uploading piece. err: ecclient: upload failed (node:127RMUc62xRq9RU5z3XfkZsq5sehmv3kP8o53tr9BxVcPE3GQCY, address:45.91.116.5:5310): protocol: storage node overloaded, request limit: 2000 Jun 22 00:30:58.210389994 (80h7m33.773712995s) [noise=true , address=104.36.82.13:6199 , piece_id=YP62N2TO446YR3QRG5YLXECD7TN6SJ4UNNLZSK5H4CMV6JIF4AKA, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:30:58.693037961 (80h7m34.256360966s) [noise=true , address=104.36.82.39:6756 , piece_id=OKKWMDJPZO6D5VRMGOE3GOCYGHM725JZTNOVYAWKJW3WYZPI4V5Q, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:30:59.124834814 (80h7m34.688157819s) [noise=true , address=104.36.82.15:6283 , piece_id=N7TH2EMIOPATVHCSNH2WXUGU7KMMEJ5LCHR22SWEHSA5OV4CUCXQ, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:31:00.629657982 (80h7m36.192980977s) [noise=true , address=104.36.82.42:6803 , piece_id=R626UFD326S2UBJEZ2WMHVR7EN4SK427VXPDJZOOZ32MX25JWZWQ, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:31:00.933110546 (80h7m36.496433536s) [noise=true , address=79.127.201.215:4326 , piece_id=USEWZDX6P52U6MCALTBNQG3SV7N2JRKNU6O23TLS6JEWAS4D7ARA, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:31:05.669167243 (80h7m41.232490237s) [noise=true , address=104.36.82.56:4261 , piece_id=EYOYOYLDACGMPGJYX6ILQT3XHMUTQNXZEDC7ANHLZXIKYVDQFFVQ, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:31:08.443582763 (80h7m44.006905763s) [noise=true , address=104.36.82.54:4184 , piece_id=IZICVGZFMAYLIJTNUJ2OFLZVNZ6IPPYDZO4BSTJJPLHVCVOE5EAA, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:31:26.459138014 (80h8m2.022461013s) [noise=true , address=104.36.82.53:4146 , piece_id=3V3DVGUOPJKDWIGQ2TU7M6VFYFVAS6ODY62ERGSU6LI3LSTWQ4YQ, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:31:27.456050336 (80h8m3.019373336s) [noise=true , address=104.36.82.52:4089 , piece_id=X2VG4S3UHVNWA4AQJS7FLKWNVLTOZ3AKWI5ZDH24QCA3ENQW7BXQ, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:31:55.270234251 (80h8m30.833557256s) [noise=true , address=104.36.82.11:6125 , piece_id=2CX7RJOKNQA6ODZZZXYPSCYAQY5P35TWX7I56CMYAPFLJE3JEO5Q, seg_pos={0 0}, upload=64142]: Done uploading piece. err: ecclient: upload failed (node:12vricofwx67FRaxAGLbmgT1TYyKkqfJbxdsyeEPL4R7kg8pKzT, address:104.36.82.11:6125): protocol: expected piece hash; context deadline exceeded; EOF Jun 22 00:31:55.270319418 (80h8m30.833642407s) [noise=true , address=104.36.82.55:4224 , piece_id=XF7H2BYSNNNZOS2T77VIN75N76AYUXAQEI6D4LRXACU2KU62V2TA, seg_pos={0 0}, upload=64142]: Done uploading piece. err: ecclient: upload failed (node:12EjKmUiv4pxkVCB4WoiXim4uFpxrBPyfiSS4efTY3HKSE3jb5y, address:104.36.82.55:4224): protocol: write tcp 159.223.143.39:39152->104.36.82.55:4224: use of closed network connection; write tcp 159.223.143.39:39152->104.36.82.55:4224: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:39152->104.36.82.55:4224: use of closed network connection Jun 22 00:31:55.270353811 (80h8m30.833676799s) [seg_pos={0 0}, upload=64142]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Jun 22 00:31:55.301099877 (80h8m30.864422884s) [upload=64142]: Done uploading segments... Jun 22 00:31:55.301121567 (80h8m30.864444566s) [upload=64142]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 15.816092851s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.X..........................................................] duration: 23.485291236s FAIL (2025-06-22 01:19:13 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== ing upload Jun 22 00:30:55.303646184 (80h7m30.866969171s) [upload=64142]: Uploading segments... Jun 22 00:30:55.303706406 (80h7m30.867029416s) [upload=64142]: Got next segment. Inline: false Jun 22 00:30:55.303762479 (80h7m30.867085468s) [upload=64142]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 22 00:30:55.303768627 (80h7m30.867091636s) [upload=64142]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 22 00:30:55.349058140 (80h7m30.912381168s) [seg_pos={0 0}, upload=64142]: Begin upload segment... Jun 22 00:30:55.349113768 (80h7m30.912436772s) [seg_pos={0 0}, upload=64142]: Done begin upload segment. Jun 22 00:30:55.349143121 (80h7m30.912466131s) [noise=true , address=45.91.116.5:5310 , piece_id=AXJOKDZRPVNL7HFT4QKJWRAHFM2UCM3GEVQB2KNYD6CXB6TH4D4Q, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.349660917 (80h7m30.912983928s) [noise=true , address=77.90.31.10:4533 , piece_id=SE6YAKKFDL3BV677CDDUC32UNMPXJHTG23GWWYK5B4YIPKER6GAQ, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.350093077 (80h7m30.913416089s) [noise=true , address=79.127.201.215:4326 , piece_id=USEWZDX6P52U6MCALTBNQG3SV7N2JRKNU6O23TLS6JEWAS4D7ARA, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.350480894 (80h7m30.913803916s) [noise=true , address=104.36.82.13:6199 , piece_id=YP62N2TO446YR3QRG5YLXECD7TN6SJ4UNNLZSK5H4CMV6JIF4AKA, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.350669998 (80h7m30.913993011s) [noise=true , address=104.36.82.54:4184 , piece_id=IZICVGZFMAYLIJTNUJ2OFLZVNZ6IPPYDZO4BSTJJPLHVCVOE5EAA, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.351201352 (80h7m30.914524366s) [noise=true , address=104.36.82.55:4224 , piece_id=XF7H2BYSNNNZOS2T77VIN75N76AYUXAQEI6D4LRXACU2KU62V2TA, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.351668071 (80h7m30.914991083s) [noise=true , address=104.36.82.56:4261 , piece_id=EYOYOYLDACGMPGJYX6ILQT3XHMUTQNXZEDC7ANHLZXIKYVDQFFVQ, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.352193126 (80h7m30.915516138s) [noise=true , address=104.36.82.52:4089 , piece_id=X2VG4S3UHVNWA4AQJS7FLKWNVLTOZ3AKWI5ZDH24QCA3ENQW7BXQ, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.352587096 (80h7m30.915910103s) [noise=true , address=79.127.220.98:4402 , piece_id=TE2WQO5LK45AVMPIAIGVSIBOVZLC4FG4ESQ72X6KFYC7IAEN2DRA, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.353420253 (80h7m30.916743265s) [noise=true , address=104.36.82.11:6125 , piece_id=2CX7RJOKNQA6ODZZZXYPSCYAQY5P35TWX7I56CMYAPFLJE3JEO5Q, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.353791565 (80h7m30.917114578s) [noise=true , address=104.36.82.42:6803 , piece_id=R626UFD326S2UBJEZ2WMHVR7EN4SK427VXPDJZOOZ32MX25JWZWQ, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.354171226 (80h7m30.917494233s) [noise=true , address=104.36.82.14:6245 , piece_id=II36G46OA6TINLR244U6D7Q6LGV754UA5M62K3EX7ZOBESMJUEHA, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.354542282 (80h7m30.917865288s) [noise=true , address=79.127.205.235:5660 , piece_id=UGFPVS7VNB7U2YSFPSGYDIRZV5R2DHCN57PJP6VOXO52WIAYHLXA, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.354891655 (80h7m30.918214678s) [noise=true , address=104.36.82.15:6283 , piece_id=N7TH2EMIOPATVHCSNH2WXUGU7KMMEJ5LCHR22SWEHSA5OV4CUCXQ, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.355499503 (80h7m30.918822519s) [noise=true , address=77.90.31.14:4666 , piece_id=H4DTFDAVCN266PB235BJIQTQ32FBXTIB5AZSX3B4R2VIK554AFSQ, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.356575547 (80h7m30.919898539s) [noise=true , address=104.36.82.39:6756 , piece_id=OKKWMDJPZO6D5VRMGOE3GOCYGHM725JZTNOVYAWKJW3WYZPI4V5Q, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.357039590 (80h7m30.920362596s) [noise=true , address=104.36.82.53:4146 , piece_id=3V3DVGUOPJKDWIGQ2TU7M6VFYFVAS6ODY62ERGSU6LI3LSTWQ4YQ, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.360343566 (80h7m30.923666582s) [noise=true , address=104.36.82.9:6033 , piece_id=T5XA23LZELMSTWOLBSHOX4AHKDACF65OYCKTMZYTXDIBSXNA2T4A, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.452444064 (80h7m31.015767055s) [upload=64142]: Next returned nil segment Jun 22 00:30:55.452495148 (80h7m31.015818149s) [upload=64142]: Waiting for error group managing segments... Jun 22 00:30:55.532958808 (80h7m31.096281883s) [noise=true , address=104.36.82.9:6033 , piece_id=T5XA23LZELMSTWOLBSHOX4AHKDACF65OYCKTMZYTXDIBSXNA2T4A, seg_pos={0 0}, upload=64142]: Done uploading piece. err: ecclient: upload failed (node:1SV3LrcQW16wLewDzfyUWFB6sUgnERNJVdhz72CJmyde95M79C, address:104.36.82.9:6033): protocol: storage node overloaded, request limit: 2000 Jun 22 00:30:55.547802184 (80h7m31.111125188s) [noise=true , address=79.127.220.98:4402 , piece_id=TE2WQO5LK45AVMPIAIGVSIBOVZLC4FG4ESQ72X6KFYC7IAEN2DRA, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:30:55.554223621 (80h7m31.117546616s) [noise=true , address=77.90.31.14:4666 , piece_id=H4DTFDAVCN266PB235BJIQTQ32FBXTIB5AZSX3B4R2VIK554AFSQ, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:30:55.762514255 (80h7m31.325837245s) [noise=true , address=104.36.82.14:6245 , piece_id=II36G46OA6TINLR244U6D7Q6LGV754UA5M62K3EX7ZOBESMJUEHA, seg_pos={0 0}, upload=64142]: Done uploading piece. err: ecclient: upload failed (node:1Zu8teNPTXn638grHKwtJqz4UEY7oFYLx1DHvuVP6RZXcatVqX, address:104.36.82.14:6245): protocol: storage node overloaded, request limit: 2000 Jun 22 00:30:55.773669583 (80h7m31.3369926s) [noise=true , address=79.127.205.235:5660 , piece_id=UGFPVS7VNB7U2YSFPSGYDIRZV5R2DHCN57PJP6VOXO52WIAYHLXA, seg_pos={0 0}, upload=64142]: Done uploading piece. err: ecclient: upload failed (node:12E6uCX9KL4Ud7YpxMXepvMXuqTFJCiSXByDjW94Am3U9H2Nf7f, address:79.127.205.235:5660): protocol: storage node overloaded, request limit: 2000 Jun 22 00:30:56.968366655 (80h7m32.53168964s) [noise=true , address=77.90.31.10:4533 , piece_id=SE6YAKKFDL3BV677CDDUC32UNMPXJHTG23GWWYK5B4YIPKER6GAQ, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:30:57.403572874 (80h7m32.966895883s) [noise=true , address=45.91.116.5:5310 , piece_id=AXJOKDZRPVNL7HFT4QKJWRAHFM2UCM3GEVQB2KNYD6CXB6TH4D4Q, seg_pos={0 0}, upload=64142]: Done uploading piece. err: ecclient: upload failed (node:127RMUc62xRq9RU5z3XfkZsq5sehmv3kP8o53tr9BxVcPE3GQCY, address:45.91.116.5:5310): protocol: storage node overloaded, request limit: 2000 Jun 22 00:30:58.210389994 (80h7m33.773712995s) [noise=true , address=104.36.82.13:6199 , piece_id=YP62N2TO446YR3QRG5YLXECD7TN6SJ4UNNLZSK5H4CMV6JIF4AKA, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:30:58.693037961 (80h7m34.256360966s) [noise=true , address=104.36.82.39:6756 , piece_id=OKKWMDJPZO6D5VRMGOE3GOCYGHM725JZTNOVYAWKJW3WYZPI4V5Q, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:30:59.124834814 (80h7m34.688157819s) [noise=true , address=104.36.82.15:6283 , piece_id=N7TH2EMIOPATVHCSNH2WXUGU7KMMEJ5LCHR22SWEHSA5OV4CUCXQ, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:31:00.629657982 (80h7m36.192980977s) [noise=true , address=104.36.82.42:6803 , piece_id=R626UFD326S2UBJEZ2WMHVR7EN4SK427VXPDJZOOZ32MX25JWZWQ, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:31:00.933110546 (80h7m36.496433536s) [noise=true , address=79.127.201.215:4326 , piece_id=USEWZDX6P52U6MCALTBNQG3SV7N2JRKNU6O23TLS6JEWAS4D7ARA, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:31:05.669167243 (80h7m41.232490237s) [noise=true , address=104.36.82.56:4261 , piece_id=EYOYOYLDACGMPGJYX6ILQT3XHMUTQNXZEDC7ANHLZXIKYVDQFFVQ, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:31:08.443582763 (80h7m44.006905763s) [noise=true , address=104.36.82.54:4184 , piece_id=IZICVGZFMAYLIJTNUJ2OFLZVNZ6IPPYDZO4BSTJJPLHVCVOE5EAA, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:31:26.459138014 (80h8m2.022461013s) [noise=true , address=104.36.82.53:4146 , piece_id=3V3DVGUOPJKDWIGQ2TU7M6VFYFVAS6ODY62ERGSU6LI3LSTWQ4YQ, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:31:27.456050336 (80h8m3.019373336s) [noise=true , address=104.36.82.52:4089 , piece_id=X2VG4S3UHVNWA4AQJS7FLKWNVLTOZ3AKWI5ZDH24QCA3ENQW7BXQ, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:31:55.270234251 (80h8m30.833557256s) [noise=true , address=104.36.82.11:6125 , piece_id=2CX7RJOKNQA6ODZZZXYPSCYAQY5P35TWX7I56CMYAPFLJE3JEO5Q, seg_pos={0 0}, upload=64142]: Done uploading piece. err: ecclient: upload failed (node:12vricofwx67FRaxAGLbmgT1TYyKkqfJbxdsyeEPL4R7kg8pKzT, address:104.36.82.11:6125): protocol: expected piece hash; context deadline exceeded; EOF Jun 22 00:31:55.270319418 (80h8m30.833642407s) [noise=true , address=104.36.82.55:4224 , piece_id=XF7H2BYSNNNZOS2T77VIN75N76AYUXAQEI6D4LRXACU2KU62V2TA, seg_pos={0 0}, upload=64142]: Done uploading piece. err: ecclient: upload failed (node:12EjKmUiv4pxkVCB4WoiXim4uFpxrBPyfiSS4efTY3HKSE3jb5y, address:104.36.82.55:4224): protocol: write tcp 159.223.143.39:39152->104.36.82.55:4224: use of closed network connection; write tcp 159.223.143.39:39152->104.36.82.55:4224: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:39152->104.36.82.55:4224: use of closed network connection Jun 22 00:31:55.270353811 (80h8m30.833676799s) [seg_pos={0 0}, upload=64142]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Jun 22 00:31:55.301099877 (80h8m30.864422884s) [upload=64142]: Done uploading segments... Jun 22 00:31:55.301121567 (80h8m30.864444566s) [upload=64142]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 23.485332539s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [..X.........................................................] duration: 16.403803761s FAIL (2025-06-22 01:17:55 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== ing upload Jun 22 00:30:55.303646184 (80h7m30.866969171s) [upload=64142]: Uploading segments... Jun 22 00:30:55.303706406 (80h7m30.867029416s) [upload=64142]: Got next segment. Inline: false Jun 22 00:30:55.303762479 (80h7m30.867085468s) [upload=64142]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 22 00:30:55.303768627 (80h7m30.867091636s) [upload=64142]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 22 00:30:55.349058140 (80h7m30.912381168s) [seg_pos={0 0}, upload=64142]: Begin upload segment... Jun 22 00:30:55.349113768 (80h7m30.912436772s) [seg_pos={0 0}, upload=64142]: Done begin upload segment. Jun 22 00:30:55.349143121 (80h7m30.912466131s) [noise=true , address=45.91.116.5:5310 , piece_id=AXJOKDZRPVNL7HFT4QKJWRAHFM2UCM3GEVQB2KNYD6CXB6TH4D4Q, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.349660917 (80h7m30.912983928s) [noise=true , address=77.90.31.10:4533 , piece_id=SE6YAKKFDL3BV677CDDUC32UNMPXJHTG23GWWYK5B4YIPKER6GAQ, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.350093077 (80h7m30.913416089s) [noise=true , address=79.127.201.215:4326 , piece_id=USEWZDX6P52U6MCALTBNQG3SV7N2JRKNU6O23TLS6JEWAS4D7ARA, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.350480894 (80h7m30.913803916s) [noise=true , address=104.36.82.13:6199 , piece_id=YP62N2TO446YR3QRG5YLXECD7TN6SJ4UNNLZSK5H4CMV6JIF4AKA, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.350669998 (80h7m30.913993011s) [noise=true , address=104.36.82.54:4184 , piece_id=IZICVGZFMAYLIJTNUJ2OFLZVNZ6IPPYDZO4BSTJJPLHVCVOE5EAA, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.351201352 (80h7m30.914524366s) [noise=true , address=104.36.82.55:4224 , piece_id=XF7H2BYSNNNZOS2T77VIN75N76AYUXAQEI6D4LRXACU2KU62V2TA, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.351668071 (80h7m30.914991083s) [noise=true , address=104.36.82.56:4261 , piece_id=EYOYOYLDACGMPGJYX6ILQT3XHMUTQNXZEDC7ANHLZXIKYVDQFFVQ, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.352193126 (80h7m30.915516138s) [noise=true , address=104.36.82.52:4089 , piece_id=X2VG4S3UHVNWA4AQJS7FLKWNVLTOZ3AKWI5ZDH24QCA3ENQW7BXQ, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.352587096 (80h7m30.915910103s) [noise=true , address=79.127.220.98:4402 , piece_id=TE2WQO5LK45AVMPIAIGVSIBOVZLC4FG4ESQ72X6KFYC7IAEN2DRA, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.353420253 (80h7m30.916743265s) [noise=true , address=104.36.82.11:6125 , piece_id=2CX7RJOKNQA6ODZZZXYPSCYAQY5P35TWX7I56CMYAPFLJE3JEO5Q, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.353791565 (80h7m30.917114578s) [noise=true , address=104.36.82.42:6803 , piece_id=R626UFD326S2UBJEZ2WMHVR7EN4SK427VXPDJZOOZ32MX25JWZWQ, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.354171226 (80h7m30.917494233s) [noise=true , address=104.36.82.14:6245 , piece_id=II36G46OA6TINLR244U6D7Q6LGV754UA5M62K3EX7ZOBESMJUEHA, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.354542282 (80h7m30.917865288s) [noise=true , address=79.127.205.235:5660 , piece_id=UGFPVS7VNB7U2YSFPSGYDIRZV5R2DHCN57PJP6VOXO52WIAYHLXA, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.354891655 (80h7m30.918214678s) [noise=true , address=104.36.82.15:6283 , piece_id=N7TH2EMIOPATVHCSNH2WXUGU7KMMEJ5LCHR22SWEHSA5OV4CUCXQ, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.355499503 (80h7m30.918822519s) [noise=true , address=77.90.31.14:4666 , piece_id=H4DTFDAVCN266PB235BJIQTQ32FBXTIB5AZSX3B4R2VIK554AFSQ, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.356575547 (80h7m30.919898539s) [noise=true , address=104.36.82.39:6756 , piece_id=OKKWMDJPZO6D5VRMGOE3GOCYGHM725JZTNOVYAWKJW3WYZPI4V5Q, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.357039590 (80h7m30.920362596s) [noise=true , address=104.36.82.53:4146 , piece_id=3V3DVGUOPJKDWIGQ2TU7M6VFYFVAS6ODY62ERGSU6LI3LSTWQ4YQ, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.360343566 (80h7m30.923666582s) [noise=true , address=104.36.82.9:6033 , piece_id=T5XA23LZELMSTWOLBSHOX4AHKDACF65OYCKTMZYTXDIBSXNA2T4A, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.452444064 (80h7m31.015767055s) [upload=64142]: Next returned nil segment Jun 22 00:30:55.452495148 (80h7m31.015818149s) [upload=64142]: Waiting for error group managing segments... Jun 22 00:30:55.532958808 (80h7m31.096281883s) [noise=true , address=104.36.82.9:6033 , piece_id=T5XA23LZELMSTWOLBSHOX4AHKDACF65OYCKTMZYTXDIBSXNA2T4A, seg_pos={0 0}, upload=64142]: Done uploading piece. err: ecclient: upload failed (node:1SV3LrcQW16wLewDzfyUWFB6sUgnERNJVdhz72CJmyde95M79C, address:104.36.82.9:6033): protocol: storage node overloaded, request limit: 2000 Jun 22 00:30:55.547802184 (80h7m31.111125188s) [noise=true , address=79.127.220.98:4402 , piece_id=TE2WQO5LK45AVMPIAIGVSIBOVZLC4FG4ESQ72X6KFYC7IAEN2DRA, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:30:55.554223621 (80h7m31.117546616s) [noise=true , address=77.90.31.14:4666 , piece_id=H4DTFDAVCN266PB235BJIQTQ32FBXTIB5AZSX3B4R2VIK554AFSQ, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:30:55.762514255 (80h7m31.325837245s) [noise=true , address=104.36.82.14:6245 , piece_id=II36G46OA6TINLR244U6D7Q6LGV754UA5M62K3EX7ZOBESMJUEHA, seg_pos={0 0}, upload=64142]: Done uploading piece. err: ecclient: upload failed (node:1Zu8teNPTXn638grHKwtJqz4UEY7oFYLx1DHvuVP6RZXcatVqX, address:104.36.82.14:6245): protocol: storage node overloaded, request limit: 2000 Jun 22 00:30:55.773669583 (80h7m31.3369926s) [noise=true , address=79.127.205.235:5660 , piece_id=UGFPVS7VNB7U2YSFPSGYDIRZV5R2DHCN57PJP6VOXO52WIAYHLXA, seg_pos={0 0}, upload=64142]: Done uploading piece. err: ecclient: upload failed (node:12E6uCX9KL4Ud7YpxMXepvMXuqTFJCiSXByDjW94Am3U9H2Nf7f, address:79.127.205.235:5660): protocol: storage node overloaded, request limit: 2000 Jun 22 00:30:56.968366655 (80h7m32.53168964s) [noise=true , address=77.90.31.10:4533 , piece_id=SE6YAKKFDL3BV677CDDUC32UNMPXJHTG23GWWYK5B4YIPKER6GAQ, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:30:57.403572874 (80h7m32.966895883s) [noise=true , address=45.91.116.5:5310 , piece_id=AXJOKDZRPVNL7HFT4QKJWRAHFM2UCM3GEVQB2KNYD6CXB6TH4D4Q, seg_pos={0 0}, upload=64142]: Done uploading piece. err: ecclient: upload failed (node:127RMUc62xRq9RU5z3XfkZsq5sehmv3kP8o53tr9BxVcPE3GQCY, address:45.91.116.5:5310): protocol: storage node overloaded, request limit: 2000 Jun 22 00:30:58.210389994 (80h7m33.773712995s) [noise=true , address=104.36.82.13:6199 , piece_id=YP62N2TO446YR3QRG5YLXECD7TN6SJ4UNNLZSK5H4CMV6JIF4AKA, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:30:58.693037961 (80h7m34.256360966s) [noise=true , address=104.36.82.39:6756 , piece_id=OKKWMDJPZO6D5VRMGOE3GOCYGHM725JZTNOVYAWKJW3WYZPI4V5Q, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:30:59.124834814 (80h7m34.688157819s) [noise=true , address=104.36.82.15:6283 , piece_id=N7TH2EMIOPATVHCSNH2WXUGU7KMMEJ5LCHR22SWEHSA5OV4CUCXQ, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:31:00.629657982 (80h7m36.192980977s) [noise=true , address=104.36.82.42:6803 , piece_id=R626UFD326S2UBJEZ2WMHVR7EN4SK427VXPDJZOOZ32MX25JWZWQ, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:31:00.933110546 (80h7m36.496433536s) [noise=true , address=79.127.201.215:4326 , piece_id=USEWZDX6P52U6MCALTBNQG3SV7N2JRKNU6O23TLS6JEWAS4D7ARA, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:31:05.669167243 (80h7m41.232490237s) [noise=true , address=104.36.82.56:4261 , piece_id=EYOYOYLDACGMPGJYX6ILQT3XHMUTQNXZEDC7ANHLZXIKYVDQFFVQ, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:31:08.443582763 (80h7m44.006905763s) [noise=true , address=104.36.82.54:4184 , piece_id=IZICVGZFMAYLIJTNUJ2OFLZVNZ6IPPYDZO4BSTJJPLHVCVOE5EAA, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:31:26.459138014 (80h8m2.022461013s) [noise=true , address=104.36.82.53:4146 , piece_id=3V3DVGUOPJKDWIGQ2TU7M6VFYFVAS6ODY62ERGSU6LI3LSTWQ4YQ, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:31:27.456050336 (80h8m3.019373336s) [noise=true , address=104.36.82.52:4089 , piece_id=X2VG4S3UHVNWA4AQJS7FLKWNVLTOZ3AKWI5ZDH24QCA3ENQW7BXQ, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:31:55.270234251 (80h8m30.833557256s) [noise=true , address=104.36.82.11:6125 , piece_id=2CX7RJOKNQA6ODZZZXYPSCYAQY5P35TWX7I56CMYAPFLJE3JEO5Q, seg_pos={0 0}, upload=64142]: Done uploading piece. err: ecclient: upload failed (node:12vricofwx67FRaxAGLbmgT1TYyKkqfJbxdsyeEPL4R7kg8pKzT, address:104.36.82.11:6125): protocol: expected piece hash; context deadline exceeded; EOF Jun 22 00:31:55.270319418 (80h8m30.833642407s) [noise=true , address=104.36.82.55:4224 , piece_id=XF7H2BYSNNNZOS2T77VIN75N76AYUXAQEI6D4LRXACU2KU62V2TA, seg_pos={0 0}, upload=64142]: Done uploading piece. err: ecclient: upload failed (node:12EjKmUiv4pxkVCB4WoiXim4uFpxrBPyfiSS4efTY3HKSE3jb5y, address:104.36.82.55:4224): protocol: write tcp 159.223.143.39:39152->104.36.82.55:4224: use of closed network connection; write tcp 159.223.143.39:39152->104.36.82.55:4224: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:39152->104.36.82.55:4224: use of closed network connection Jun 22 00:31:55.270353811 (80h8m30.833676799s) [seg_pos={0 0}, upload=64142]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Jun 22 00:31:55.301099877 (80h8m30.864422884s) [upload=64142]: Done uploading segments... Jun 22 00:31:55.301121567 (80h8m30.864444566s) [upload=64142]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 16.403826667s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [X...........................................................] duration: 25.854757279s FAIL (2025-06-22 01:17:49 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 25.854782232s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [...X........................................................] duration: 19.615903909s FAIL (2025-06-22 01:17:38 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== ing upload Jun 22 00:30:55.303646184 (80h7m30.866969171s) [upload=64142]: Uploading segments... Jun 22 00:30:55.303706406 (80h7m30.867029416s) [upload=64142]: Got next segment. Inline: false Jun 22 00:30:55.303762479 (80h7m30.867085468s) [upload=64142]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 22 00:30:55.303768627 (80h7m30.867091636s) [upload=64142]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 22 00:30:55.349058140 (80h7m30.912381168s) [seg_pos={0 0}, upload=64142]: Begin upload segment... Jun 22 00:30:55.349113768 (80h7m30.912436772s) [seg_pos={0 0}, upload=64142]: Done begin upload segment. Jun 22 00:30:55.349143121 (80h7m30.912466131s) [noise=true , address=45.91.116.5:5310 , piece_id=AXJOKDZRPVNL7HFT4QKJWRAHFM2UCM3GEVQB2KNYD6CXB6TH4D4Q, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.349660917 (80h7m30.912983928s) [noise=true , address=77.90.31.10:4533 , piece_id=SE6YAKKFDL3BV677CDDUC32UNMPXJHTG23GWWYK5B4YIPKER6GAQ, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.350093077 (80h7m30.913416089s) [noise=true , address=79.127.201.215:4326 , piece_id=USEWZDX6P52U6MCALTBNQG3SV7N2JRKNU6O23TLS6JEWAS4D7ARA, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.350480894 (80h7m30.913803916s) [noise=true , address=104.36.82.13:6199 , piece_id=YP62N2TO446YR3QRG5YLXECD7TN6SJ4UNNLZSK5H4CMV6JIF4AKA, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.350669998 (80h7m30.913993011s) [noise=true , address=104.36.82.54:4184 , piece_id=IZICVGZFMAYLIJTNUJ2OFLZVNZ6IPPYDZO4BSTJJPLHVCVOE5EAA, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.351201352 (80h7m30.914524366s) [noise=true , address=104.36.82.55:4224 , piece_id=XF7H2BYSNNNZOS2T77VIN75N76AYUXAQEI6D4LRXACU2KU62V2TA, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.351668071 (80h7m30.914991083s) [noise=true , address=104.36.82.56:4261 , piece_id=EYOYOYLDACGMPGJYX6ILQT3XHMUTQNXZEDC7ANHLZXIKYVDQFFVQ, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.352193126 (80h7m30.915516138s) [noise=true , address=104.36.82.52:4089 , piece_id=X2VG4S3UHVNWA4AQJS7FLKWNVLTOZ3AKWI5ZDH24QCA3ENQW7BXQ, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.352587096 (80h7m30.915910103s) [noise=true , address=79.127.220.98:4402 , piece_id=TE2WQO5LK45AVMPIAIGVSIBOVZLC4FG4ESQ72X6KFYC7IAEN2DRA, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.353420253 (80h7m30.916743265s) [noise=true , address=104.36.82.11:6125 , piece_id=2CX7RJOKNQA6ODZZZXYPSCYAQY5P35TWX7I56CMYAPFLJE3JEO5Q, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.353791565 (80h7m30.917114578s) [noise=true , address=104.36.82.42:6803 , piece_id=R626UFD326S2UBJEZ2WMHVR7EN4SK427VXPDJZOOZ32MX25JWZWQ, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.354171226 (80h7m30.917494233s) [noise=true , address=104.36.82.14:6245 , piece_id=II36G46OA6TINLR244U6D7Q6LGV754UA5M62K3EX7ZOBESMJUEHA, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.354542282 (80h7m30.917865288s) [noise=true , address=79.127.205.235:5660 , piece_id=UGFPVS7VNB7U2YSFPSGYDIRZV5R2DHCN57PJP6VOXO52WIAYHLXA, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.354891655 (80h7m30.918214678s) [noise=true , address=104.36.82.15:6283 , piece_id=N7TH2EMIOPATVHCSNH2WXUGU7KMMEJ5LCHR22SWEHSA5OV4CUCXQ, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.355499503 (80h7m30.918822519s) [noise=true , address=77.90.31.14:4666 , piece_id=H4DTFDAVCN266PB235BJIQTQ32FBXTIB5AZSX3B4R2VIK554AFSQ, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.356575547 (80h7m30.919898539s) [noise=true , address=104.36.82.39:6756 , piece_id=OKKWMDJPZO6D5VRMGOE3GOCYGHM725JZTNOVYAWKJW3WYZPI4V5Q, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.357039590 (80h7m30.920362596s) [noise=true , address=104.36.82.53:4146 , piece_id=3V3DVGUOPJKDWIGQ2TU7M6VFYFVAS6ODY62ERGSU6LI3LSTWQ4YQ, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.360343566 (80h7m30.923666582s) [noise=true , address=104.36.82.9:6033 , piece_id=T5XA23LZELMSTWOLBSHOX4AHKDACF65OYCKTMZYTXDIBSXNA2T4A, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.452444064 (80h7m31.015767055s) [upload=64142]: Next returned nil segment Jun 22 00:30:55.452495148 (80h7m31.015818149s) [upload=64142]: Waiting for error group managing segments... Jun 22 00:30:55.532958808 (80h7m31.096281883s) [noise=true , address=104.36.82.9:6033 , piece_id=T5XA23LZELMSTWOLBSHOX4AHKDACF65OYCKTMZYTXDIBSXNA2T4A, seg_pos={0 0}, upload=64142]: Done uploading piece. err: ecclient: upload failed (node:1SV3LrcQW16wLewDzfyUWFB6sUgnERNJVdhz72CJmyde95M79C, address:104.36.82.9:6033): protocol: storage node overloaded, request limit: 2000 Jun 22 00:30:55.547802184 (80h7m31.111125188s) [noise=true , address=79.127.220.98:4402 , piece_id=TE2WQO5LK45AVMPIAIGVSIBOVZLC4FG4ESQ72X6KFYC7IAEN2DRA, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:30:55.554223621 (80h7m31.117546616s) [noise=true , address=77.90.31.14:4666 , piece_id=H4DTFDAVCN266PB235BJIQTQ32FBXTIB5AZSX3B4R2VIK554AFSQ, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:30:55.762514255 (80h7m31.325837245s) [noise=true , address=104.36.82.14:6245 , piece_id=II36G46OA6TINLR244U6D7Q6LGV754UA5M62K3EX7ZOBESMJUEHA, seg_pos={0 0}, upload=64142]: Done uploading piece. err: ecclient: upload failed (node:1Zu8teNPTXn638grHKwtJqz4UEY7oFYLx1DHvuVP6RZXcatVqX, address:104.36.82.14:6245): protocol: storage node overloaded, request limit: 2000 Jun 22 00:30:55.773669583 (80h7m31.3369926s) [noise=true , address=79.127.205.235:5660 , piece_id=UGFPVS7VNB7U2YSFPSGYDIRZV5R2DHCN57PJP6VOXO52WIAYHLXA, seg_pos={0 0}, upload=64142]: Done uploading piece. err: ecclient: upload failed (node:12E6uCX9KL4Ud7YpxMXepvMXuqTFJCiSXByDjW94Am3U9H2Nf7f, address:79.127.205.235:5660): protocol: storage node overloaded, request limit: 2000 Jun 22 00:30:56.968366655 (80h7m32.53168964s) [noise=true , address=77.90.31.10:4533 , piece_id=SE6YAKKFDL3BV677CDDUC32UNMPXJHTG23GWWYK5B4YIPKER6GAQ, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:30:57.403572874 (80h7m32.966895883s) [noise=true , address=45.91.116.5:5310 , piece_id=AXJOKDZRPVNL7HFT4QKJWRAHFM2UCM3GEVQB2KNYD6CXB6TH4D4Q, seg_pos={0 0}, upload=64142]: Done uploading piece. err: ecclient: upload failed (node:127RMUc62xRq9RU5z3XfkZsq5sehmv3kP8o53tr9BxVcPE3GQCY, address:45.91.116.5:5310): protocol: storage node overloaded, request limit: 2000 Jun 22 00:30:58.210389994 (80h7m33.773712995s) [noise=true , address=104.36.82.13:6199 , piece_id=YP62N2TO446YR3QRG5YLXECD7TN6SJ4UNNLZSK5H4CMV6JIF4AKA, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:30:58.693037961 (80h7m34.256360966s) [noise=true , address=104.36.82.39:6756 , piece_id=OKKWMDJPZO6D5VRMGOE3GOCYGHM725JZTNOVYAWKJW3WYZPI4V5Q, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:30:59.124834814 (80h7m34.688157819s) [noise=true , address=104.36.82.15:6283 , piece_id=N7TH2EMIOPATVHCSNH2WXUGU7KMMEJ5LCHR22SWEHSA5OV4CUCXQ, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:31:00.629657982 (80h7m36.192980977s) [noise=true , address=104.36.82.42:6803 , piece_id=R626UFD326S2UBJEZ2WMHVR7EN4SK427VXPDJZOOZ32MX25JWZWQ, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:31:00.933110546 (80h7m36.496433536s) [noise=true , address=79.127.201.215:4326 , piece_id=USEWZDX6P52U6MCALTBNQG3SV7N2JRKNU6O23TLS6JEWAS4D7ARA, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:31:05.669167243 (80h7m41.232490237s) [noise=true , address=104.36.82.56:4261 , piece_id=EYOYOYLDACGMPGJYX6ILQT3XHMUTQNXZEDC7ANHLZXIKYVDQFFVQ, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:31:08.443582763 (80h7m44.006905763s) [noise=true , address=104.36.82.54:4184 , piece_id=IZICVGZFMAYLIJTNUJ2OFLZVNZ6IPPYDZO4BSTJJPLHVCVOE5EAA, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:31:26.459138014 (80h8m2.022461013s) [noise=true , address=104.36.82.53:4146 , piece_id=3V3DVGUOPJKDWIGQ2TU7M6VFYFVAS6ODY62ERGSU6LI3LSTWQ4YQ, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:31:27.456050336 (80h8m3.019373336s) [noise=true , address=104.36.82.52:4089 , piece_id=X2VG4S3UHVNWA4AQJS7FLKWNVLTOZ3AKWI5ZDH24QCA3ENQW7BXQ, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:31:55.270234251 (80h8m30.833557256s) [noise=true , address=104.36.82.11:6125 , piece_id=2CX7RJOKNQA6ODZZZXYPSCYAQY5P35TWX7I56CMYAPFLJE3JEO5Q, seg_pos={0 0}, upload=64142]: Done uploading piece. err: ecclient: upload failed (node:12vricofwx67FRaxAGLbmgT1TYyKkqfJbxdsyeEPL4R7kg8pKzT, address:104.36.82.11:6125): protocol: expected piece hash; context deadline exceeded; EOF Jun 22 00:31:55.270319418 (80h8m30.833642407s) [noise=true , address=104.36.82.55:4224 , piece_id=XF7H2BYSNNNZOS2T77VIN75N76AYUXAQEI6D4LRXACU2KU62V2TA, seg_pos={0 0}, upload=64142]: Done uploading piece. err: ecclient: upload failed (node:12EjKmUiv4pxkVCB4WoiXim4uFpxrBPyfiSS4efTY3HKSE3jb5y, address:104.36.82.55:4224): protocol: write tcp 159.223.143.39:39152->104.36.82.55:4224: use of closed network connection; write tcp 159.223.143.39:39152->104.36.82.55:4224: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:39152->104.36.82.55:4224: use of closed network connection Jun 22 00:31:55.270353811 (80h8m30.833676799s) [seg_pos={0 0}, upload=64142]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Jun 22 00:31:55.301099877 (80h8m30.864422884s) [upload=64142]: Done uploading segments... Jun 22 00:31:55.301121567 (80h8m30.864444566s) [upload=64142]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 19.615933273s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [....X.......................................................] duration: 28.565884154s FAIL (2025-06-22 01:16:58 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== ing upload Jun 22 00:30:55.303646184 (80h7m30.866969171s) [upload=64142]: Uploading segments... Jun 22 00:30:55.303706406 (80h7m30.867029416s) [upload=64142]: Got next segment. Inline: false Jun 22 00:30:55.303762479 (80h7m30.867085468s) [upload=64142]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 22 00:30:55.303768627 (80h7m30.867091636s) [upload=64142]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 22 00:30:55.349058140 (80h7m30.912381168s) [seg_pos={0 0}, upload=64142]: Begin upload segment... Jun 22 00:30:55.349113768 (80h7m30.912436772s) [seg_pos={0 0}, upload=64142]: Done begin upload segment. Jun 22 00:30:55.349143121 (80h7m30.912466131s) [noise=true , address=45.91.116.5:5310 , piece_id=AXJOKDZRPVNL7HFT4QKJWRAHFM2UCM3GEVQB2KNYD6CXB6TH4D4Q, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.349660917 (80h7m30.912983928s) [noise=true , address=77.90.31.10:4533 , piece_id=SE6YAKKFDL3BV677CDDUC32UNMPXJHTG23GWWYK5B4YIPKER6GAQ, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.350093077 (80h7m30.913416089s) [noise=true , address=79.127.201.215:4326 , piece_id=USEWZDX6P52U6MCALTBNQG3SV7N2JRKNU6O23TLS6JEWAS4D7ARA, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.350480894 (80h7m30.913803916s) [noise=true , address=104.36.82.13:6199 , piece_id=YP62N2TO446YR3QRG5YLXECD7TN6SJ4UNNLZSK5H4CMV6JIF4AKA, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.350669998 (80h7m30.913993011s) [noise=true , address=104.36.82.54:4184 , piece_id=IZICVGZFMAYLIJTNUJ2OFLZVNZ6IPPYDZO4BSTJJPLHVCVOE5EAA, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.351201352 (80h7m30.914524366s) [noise=true , address=104.36.82.55:4224 , piece_id=XF7H2BYSNNNZOS2T77VIN75N76AYUXAQEI6D4LRXACU2KU62V2TA, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.351668071 (80h7m30.914991083s) [noise=true , address=104.36.82.56:4261 , piece_id=EYOYOYLDACGMPGJYX6ILQT3XHMUTQNXZEDC7ANHLZXIKYVDQFFVQ, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.352193126 (80h7m30.915516138s) [noise=true , address=104.36.82.52:4089 , piece_id=X2VG4S3UHVNWA4AQJS7FLKWNVLTOZ3AKWI5ZDH24QCA3ENQW7BXQ, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.352587096 (80h7m30.915910103s) [noise=true , address=79.127.220.98:4402 , piece_id=TE2WQO5LK45AVMPIAIGVSIBOVZLC4FG4ESQ72X6KFYC7IAEN2DRA, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.353420253 (80h7m30.916743265s) [noise=true , address=104.36.82.11:6125 , piece_id=2CX7RJOKNQA6ODZZZXYPSCYAQY5P35TWX7I56CMYAPFLJE3JEO5Q, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.353791565 (80h7m30.917114578s) [noise=true , address=104.36.82.42:6803 , piece_id=R626UFD326S2UBJEZ2WMHVR7EN4SK427VXPDJZOOZ32MX25JWZWQ, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.354171226 (80h7m30.917494233s) [noise=true , address=104.36.82.14:6245 , piece_id=II36G46OA6TINLR244U6D7Q6LGV754UA5M62K3EX7ZOBESMJUEHA, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.354542282 (80h7m30.917865288s) [noise=true , address=79.127.205.235:5660 , piece_id=UGFPVS7VNB7U2YSFPSGYDIRZV5R2DHCN57PJP6VOXO52WIAYHLXA, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.354891655 (80h7m30.918214678s) [noise=true , address=104.36.82.15:6283 , piece_id=N7TH2EMIOPATVHCSNH2WXUGU7KMMEJ5LCHR22SWEHSA5OV4CUCXQ, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.355499503 (80h7m30.918822519s) [noise=true , address=77.90.31.14:4666 , piece_id=H4DTFDAVCN266PB235BJIQTQ32FBXTIB5AZSX3B4R2VIK554AFSQ, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.356575547 (80h7m30.919898539s) [noise=true , address=104.36.82.39:6756 , piece_id=OKKWMDJPZO6D5VRMGOE3GOCYGHM725JZTNOVYAWKJW3WYZPI4V5Q, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.357039590 (80h7m30.920362596s) [noise=true , address=104.36.82.53:4146 , piece_id=3V3DVGUOPJKDWIGQ2TU7M6VFYFVAS6ODY62ERGSU6LI3LSTWQ4YQ, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.360343566 (80h7m30.923666582s) [noise=true , address=104.36.82.9:6033 , piece_id=T5XA23LZELMSTWOLBSHOX4AHKDACF65OYCKTMZYTXDIBSXNA2T4A, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.452444064 (80h7m31.015767055s) [upload=64142]: Next returned nil segment Jun 22 00:30:55.452495148 (80h7m31.015818149s) [upload=64142]: Waiting for error group managing segments... Jun 22 00:30:55.532958808 (80h7m31.096281883s) [noise=true , address=104.36.82.9:6033 , piece_id=T5XA23LZELMSTWOLBSHOX4AHKDACF65OYCKTMZYTXDIBSXNA2T4A, seg_pos={0 0}, upload=64142]: Done uploading piece. err: ecclient: upload failed (node:1SV3LrcQW16wLewDzfyUWFB6sUgnERNJVdhz72CJmyde95M79C, address:104.36.82.9:6033): protocol: storage node overloaded, request limit: 2000 Jun 22 00:30:55.547802184 (80h7m31.111125188s) [noise=true , address=79.127.220.98:4402 , piece_id=TE2WQO5LK45AVMPIAIGVSIBOVZLC4FG4ESQ72X6KFYC7IAEN2DRA, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:30:55.554223621 (80h7m31.117546616s) [noise=true , address=77.90.31.14:4666 , piece_id=H4DTFDAVCN266PB235BJIQTQ32FBXTIB5AZSX3B4R2VIK554AFSQ, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:30:55.762514255 (80h7m31.325837245s) [noise=true , address=104.36.82.14:6245 , piece_id=II36G46OA6TINLR244U6D7Q6LGV754UA5M62K3EX7ZOBESMJUEHA, seg_pos={0 0}, upload=64142]: Done uploading piece. err: ecclient: upload failed (node:1Zu8teNPTXn638grHKwtJqz4UEY7oFYLx1DHvuVP6RZXcatVqX, address:104.36.82.14:6245): protocol: storage node overloaded, request limit: 2000 Jun 22 00:30:55.773669583 (80h7m31.3369926s) [noise=true , address=79.127.205.235:5660 , piece_id=UGFPVS7VNB7U2YSFPSGYDIRZV5R2DHCN57PJP6VOXO52WIAYHLXA, seg_pos={0 0}, upload=64142]: Done uploading piece. err: ecclient: upload failed (node:12E6uCX9KL4Ud7YpxMXepvMXuqTFJCiSXByDjW94Am3U9H2Nf7f, address:79.127.205.235:5660): protocol: storage node overloaded, request limit: 2000 Jun 22 00:30:56.968366655 (80h7m32.53168964s) [noise=true , address=77.90.31.10:4533 , piece_id=SE6YAKKFDL3BV677CDDUC32UNMPXJHTG23GWWYK5B4YIPKER6GAQ, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:30:57.403572874 (80h7m32.966895883s) [noise=true , address=45.91.116.5:5310 , piece_id=AXJOKDZRPVNL7HFT4QKJWRAHFM2UCM3GEVQB2KNYD6CXB6TH4D4Q, seg_pos={0 0}, upload=64142]: Done uploading piece. err: ecclient: upload failed (node:127RMUc62xRq9RU5z3XfkZsq5sehmv3kP8o53tr9BxVcPE3GQCY, address:45.91.116.5:5310): protocol: storage node overloaded, request limit: 2000 Jun 22 00:30:58.210389994 (80h7m33.773712995s) [noise=true , address=104.36.82.13:6199 , piece_id=YP62N2TO446YR3QRG5YLXECD7TN6SJ4UNNLZSK5H4CMV6JIF4AKA, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:30:58.693037961 (80h7m34.256360966s) [noise=true , address=104.36.82.39:6756 , piece_id=OKKWMDJPZO6D5VRMGOE3GOCYGHM725JZTNOVYAWKJW3WYZPI4V5Q, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:30:59.124834814 (80h7m34.688157819s) [noise=true , address=104.36.82.15:6283 , piece_id=N7TH2EMIOPATVHCSNH2WXUGU7KMMEJ5LCHR22SWEHSA5OV4CUCXQ, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:31:00.629657982 (80h7m36.192980977s) [noise=true , address=104.36.82.42:6803 , piece_id=R626UFD326S2UBJEZ2WMHVR7EN4SK427VXPDJZOOZ32MX25JWZWQ, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:31:00.933110546 (80h7m36.496433536s) [noise=true , address=79.127.201.215:4326 , piece_id=USEWZDX6P52U6MCALTBNQG3SV7N2JRKNU6O23TLS6JEWAS4D7ARA, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:31:05.669167243 (80h7m41.232490237s) [noise=true , address=104.36.82.56:4261 , piece_id=EYOYOYLDACGMPGJYX6ILQT3XHMUTQNXZEDC7ANHLZXIKYVDQFFVQ, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:31:08.443582763 (80h7m44.006905763s) [noise=true , address=104.36.82.54:4184 , piece_id=IZICVGZFMAYLIJTNUJ2OFLZVNZ6IPPYDZO4BSTJJPLHVCVOE5EAA, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:31:26.459138014 (80h8m2.022461013s) [noise=true , address=104.36.82.53:4146 , piece_id=3V3DVGUOPJKDWIGQ2TU7M6VFYFVAS6ODY62ERGSU6LI3LSTWQ4YQ, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:31:27.456050336 (80h8m3.019373336s) [noise=true , address=104.36.82.52:4089 , piece_id=X2VG4S3UHVNWA4AQJS7FLKWNVLTOZ3AKWI5ZDH24QCA3ENQW7BXQ, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:31:55.270234251 (80h8m30.833557256s) [noise=true , address=104.36.82.11:6125 , piece_id=2CX7RJOKNQA6ODZZZXYPSCYAQY5P35TWX7I56CMYAPFLJE3JEO5Q, seg_pos={0 0}, upload=64142]: Done uploading piece. err: ecclient: upload failed (node:12vricofwx67FRaxAGLbmgT1TYyKkqfJbxdsyeEPL4R7kg8pKzT, address:104.36.82.11:6125): protocol: expected piece hash; context deadline exceeded; EOF Jun 22 00:31:55.270319418 (80h8m30.833642407s) [noise=true , address=104.36.82.55:4224 , piece_id=XF7H2BYSNNNZOS2T77VIN75N76AYUXAQEI6D4LRXACU2KU62V2TA, seg_pos={0 0}, upload=64142]: Done uploading piece. err: ecclient: upload failed (node:12EjKmUiv4pxkVCB4WoiXim4uFpxrBPyfiSS4efTY3HKSE3jb5y, address:104.36.82.55:4224): protocol: write tcp 159.223.143.39:39152->104.36.82.55:4224: use of closed network connection; write tcp 159.223.143.39:39152->104.36.82.55:4224: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:39152->104.36.82.55:4224: use of closed network connection Jun 22 00:31:55.270353811 (80h8m30.833676799s) [seg_pos={0 0}, upload=64142]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Jun 22 00:31:55.301099877 (80h8m30.864422884s) [upload=64142]: Done uploading segments... Jun 22 00:31:55.301121567 (80h8m30.864444566s) [upload=64142]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 28.56591117s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.X..........................................................] duration: 20.700396436s FAIL (2025-06-22 01:16:32 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 20.700446057s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [..X.........................................................] duration: 28.851471141s FAIL (2025-06-22 01:15:35 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 28.85151806s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.....X......................................................] duration: 16.312190906s FAIL (2025-06-22 01:15:33 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== ing upload Jun 22 00:30:55.303646184 (80h7m30.866969171s) [upload=64142]: Uploading segments... Jun 22 00:30:55.303706406 (80h7m30.867029416s) [upload=64142]: Got next segment. Inline: false Jun 22 00:30:55.303762479 (80h7m30.867085468s) [upload=64142]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 22 00:30:55.303768627 (80h7m30.867091636s) [upload=64142]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 22 00:30:55.349058140 (80h7m30.912381168s) [seg_pos={0 0}, upload=64142]: Begin upload segment... Jun 22 00:30:55.349113768 (80h7m30.912436772s) [seg_pos={0 0}, upload=64142]: Done begin upload segment. Jun 22 00:30:55.349143121 (80h7m30.912466131s) [noise=true , address=45.91.116.5:5310 , piece_id=AXJOKDZRPVNL7HFT4QKJWRAHFM2UCM3GEVQB2KNYD6CXB6TH4D4Q, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.349660917 (80h7m30.912983928s) [noise=true , address=77.90.31.10:4533 , piece_id=SE6YAKKFDL3BV677CDDUC32UNMPXJHTG23GWWYK5B4YIPKER6GAQ, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.350093077 (80h7m30.913416089s) [noise=true , address=79.127.201.215:4326 , piece_id=USEWZDX6P52U6MCALTBNQG3SV7N2JRKNU6O23TLS6JEWAS4D7ARA, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.350480894 (80h7m30.913803916s) [noise=true , address=104.36.82.13:6199 , piece_id=YP62N2TO446YR3QRG5YLXECD7TN6SJ4UNNLZSK5H4CMV6JIF4AKA, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.350669998 (80h7m30.913993011s) [noise=true , address=104.36.82.54:4184 , piece_id=IZICVGZFMAYLIJTNUJ2OFLZVNZ6IPPYDZO4BSTJJPLHVCVOE5EAA, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.351201352 (80h7m30.914524366s) [noise=true , address=104.36.82.55:4224 , piece_id=XF7H2BYSNNNZOS2T77VIN75N76AYUXAQEI6D4LRXACU2KU62V2TA, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.351668071 (80h7m30.914991083s) [noise=true , address=104.36.82.56:4261 , piece_id=EYOYOYLDACGMPGJYX6ILQT3XHMUTQNXZEDC7ANHLZXIKYVDQFFVQ, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.352193126 (80h7m30.915516138s) [noise=true , address=104.36.82.52:4089 , piece_id=X2VG4S3UHVNWA4AQJS7FLKWNVLTOZ3AKWI5ZDH24QCA3ENQW7BXQ, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.352587096 (80h7m30.915910103s) [noise=true , address=79.127.220.98:4402 , piece_id=TE2WQO5LK45AVMPIAIGVSIBOVZLC4FG4ESQ72X6KFYC7IAEN2DRA, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.353420253 (80h7m30.916743265s) [noise=true , address=104.36.82.11:6125 , piece_id=2CX7RJOKNQA6ODZZZXYPSCYAQY5P35TWX7I56CMYAPFLJE3JEO5Q, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.353791565 (80h7m30.917114578s) [noise=true , address=104.36.82.42:6803 , piece_id=R626UFD326S2UBJEZ2WMHVR7EN4SK427VXPDJZOOZ32MX25JWZWQ, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.354171226 (80h7m30.917494233s) [noise=true , address=104.36.82.14:6245 , piece_id=II36G46OA6TINLR244U6D7Q6LGV754UA5M62K3EX7ZOBESMJUEHA, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.354542282 (80h7m30.917865288s) [noise=true , address=79.127.205.235:5660 , piece_id=UGFPVS7VNB7U2YSFPSGYDIRZV5R2DHCN57PJP6VOXO52WIAYHLXA, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.354891655 (80h7m30.918214678s) [noise=true , address=104.36.82.15:6283 , piece_id=N7TH2EMIOPATVHCSNH2WXUGU7KMMEJ5LCHR22SWEHSA5OV4CUCXQ, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.355499503 (80h7m30.918822519s) [noise=true , address=77.90.31.14:4666 , piece_id=H4DTFDAVCN266PB235BJIQTQ32FBXTIB5AZSX3B4R2VIK554AFSQ, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.356575547 (80h7m30.919898539s) [noise=true , address=104.36.82.39:6756 , piece_id=OKKWMDJPZO6D5VRMGOE3GOCYGHM725JZTNOVYAWKJW3WYZPI4V5Q, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.357039590 (80h7m30.920362596s) [noise=true , address=104.36.82.53:4146 , piece_id=3V3DVGUOPJKDWIGQ2TU7M6VFYFVAS6ODY62ERGSU6LI3LSTWQ4YQ, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.360343566 (80h7m30.923666582s) [noise=true , address=104.36.82.9:6033 , piece_id=T5XA23LZELMSTWOLBSHOX4AHKDACF65OYCKTMZYTXDIBSXNA2T4A, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.452444064 (80h7m31.015767055s) [upload=64142]: Next returned nil segment Jun 22 00:30:55.452495148 (80h7m31.015818149s) [upload=64142]: Waiting for error group managing segments... Jun 22 00:30:55.532958808 (80h7m31.096281883s) [noise=true , address=104.36.82.9:6033 , piece_id=T5XA23LZELMSTWOLBSHOX4AHKDACF65OYCKTMZYTXDIBSXNA2T4A, seg_pos={0 0}, upload=64142]: Done uploading piece. err: ecclient: upload failed (node:1SV3LrcQW16wLewDzfyUWFB6sUgnERNJVdhz72CJmyde95M79C, address:104.36.82.9:6033): protocol: storage node overloaded, request limit: 2000 Jun 22 00:30:55.547802184 (80h7m31.111125188s) [noise=true , address=79.127.220.98:4402 , piece_id=TE2WQO5LK45AVMPIAIGVSIBOVZLC4FG4ESQ72X6KFYC7IAEN2DRA, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:30:55.554223621 (80h7m31.117546616s) [noise=true , address=77.90.31.14:4666 , piece_id=H4DTFDAVCN266PB235BJIQTQ32FBXTIB5AZSX3B4R2VIK554AFSQ, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:30:55.762514255 (80h7m31.325837245s) [noise=true , address=104.36.82.14:6245 , piece_id=II36G46OA6TINLR244U6D7Q6LGV754UA5M62K3EX7ZOBESMJUEHA, seg_pos={0 0}, upload=64142]: Done uploading piece. err: ecclient: upload failed (node:1Zu8teNPTXn638grHKwtJqz4UEY7oFYLx1DHvuVP6RZXcatVqX, address:104.36.82.14:6245): protocol: storage node overloaded, request limit: 2000 Jun 22 00:30:55.773669583 (80h7m31.3369926s) [noise=true , address=79.127.205.235:5660 , piece_id=UGFPVS7VNB7U2YSFPSGYDIRZV5R2DHCN57PJP6VOXO52WIAYHLXA, seg_pos={0 0}, upload=64142]: Done uploading piece. err: ecclient: upload failed (node:12E6uCX9KL4Ud7YpxMXepvMXuqTFJCiSXByDjW94Am3U9H2Nf7f, address:79.127.205.235:5660): protocol: storage node overloaded, request limit: 2000 Jun 22 00:30:56.968366655 (80h7m32.53168964s) [noise=true , address=77.90.31.10:4533 , piece_id=SE6YAKKFDL3BV677CDDUC32UNMPXJHTG23GWWYK5B4YIPKER6GAQ, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:30:57.403572874 (80h7m32.966895883s) [noise=true , address=45.91.116.5:5310 , piece_id=AXJOKDZRPVNL7HFT4QKJWRAHFM2UCM3GEVQB2KNYD6CXB6TH4D4Q, seg_pos={0 0}, upload=64142]: Done uploading piece. err: ecclient: upload failed (node:127RMUc62xRq9RU5z3XfkZsq5sehmv3kP8o53tr9BxVcPE3GQCY, address:45.91.116.5:5310): protocol: storage node overloaded, request limit: 2000 Jun 22 00:30:58.210389994 (80h7m33.773712995s) [noise=true , address=104.36.82.13:6199 , piece_id=YP62N2TO446YR3QRG5YLXECD7TN6SJ4UNNLZSK5H4CMV6JIF4AKA, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:30:58.693037961 (80h7m34.256360966s) [noise=true , address=104.36.82.39:6756 , piece_id=OKKWMDJPZO6D5VRMGOE3GOCYGHM725JZTNOVYAWKJW3WYZPI4V5Q, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:30:59.124834814 (80h7m34.688157819s) [noise=true , address=104.36.82.15:6283 , piece_id=N7TH2EMIOPATVHCSNH2WXUGU7KMMEJ5LCHR22SWEHSA5OV4CUCXQ, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:31:00.629657982 (80h7m36.192980977s) [noise=true , address=104.36.82.42:6803 , piece_id=R626UFD326S2UBJEZ2WMHVR7EN4SK427VXPDJZOOZ32MX25JWZWQ, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:31:00.933110546 (80h7m36.496433536s) [noise=true , address=79.127.201.215:4326 , piece_id=USEWZDX6P52U6MCALTBNQG3SV7N2JRKNU6O23TLS6JEWAS4D7ARA, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:31:05.669167243 (80h7m41.232490237s) [noise=true , address=104.36.82.56:4261 , piece_id=EYOYOYLDACGMPGJYX6ILQT3XHMUTQNXZEDC7ANHLZXIKYVDQFFVQ, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:31:08.443582763 (80h7m44.006905763s) [noise=true , address=104.36.82.54:4184 , piece_id=IZICVGZFMAYLIJTNUJ2OFLZVNZ6IPPYDZO4BSTJJPLHVCVOE5EAA, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:31:26.459138014 (80h8m2.022461013s) [noise=true , address=104.36.82.53:4146 , piece_id=3V3DVGUOPJKDWIGQ2TU7M6VFYFVAS6ODY62ERGSU6LI3LSTWQ4YQ, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:31:27.456050336 (80h8m3.019373336s) [noise=true , address=104.36.82.52:4089 , piece_id=X2VG4S3UHVNWA4AQJS7FLKWNVLTOZ3AKWI5ZDH24QCA3ENQW7BXQ, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:31:55.270234251 (80h8m30.833557256s) [noise=true , address=104.36.82.11:6125 , piece_id=2CX7RJOKNQA6ODZZZXYPSCYAQY5P35TWX7I56CMYAPFLJE3JEO5Q, seg_pos={0 0}, upload=64142]: Done uploading piece. err: ecclient: upload failed (node:12vricofwx67FRaxAGLbmgT1TYyKkqfJbxdsyeEPL4R7kg8pKzT, address:104.36.82.11:6125): protocol: expected piece hash; context deadline exceeded; EOF Jun 22 00:31:55.270319418 (80h8m30.833642407s) [noise=true , address=104.36.82.55:4224 , piece_id=XF7H2BYSNNNZOS2T77VIN75N76AYUXAQEI6D4LRXACU2KU62V2TA, seg_pos={0 0}, upload=64142]: Done uploading piece. err: ecclient: upload failed (node:12EjKmUiv4pxkVCB4WoiXim4uFpxrBPyfiSS4efTY3HKSE3jb5y, address:104.36.82.55:4224): protocol: write tcp 159.223.143.39:39152->104.36.82.55:4224: use of closed network connection; write tcp 159.223.143.39:39152->104.36.82.55:4224: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:39152->104.36.82.55:4224: use of closed network connection Jun 22 00:31:55.270353811 (80h8m30.833676799s) [seg_pos={0 0}, upload=64142]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Jun 22 00:31:55.301099877 (80h8m30.864422884s) [upload=64142]: Done uploading segments... Jun 22 00:31:55.301121567 (80h8m30.864444566s) [upload=64142]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 16.31221608s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [......X.....................................................] duration: 43.455852546s FAIL (2025-06-22 01:15:16 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== ing upload Jun 22 00:30:55.303646184 (80h7m30.866969171s) [upload=64142]: Uploading segments... Jun 22 00:30:55.303706406 (80h7m30.867029416s) [upload=64142]: Got next segment. Inline: false Jun 22 00:30:55.303762479 (80h7m30.867085468s) [upload=64142]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 22 00:30:55.303768627 (80h7m30.867091636s) [upload=64142]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 22 00:30:55.349058140 (80h7m30.912381168s) [seg_pos={0 0}, upload=64142]: Begin upload segment... Jun 22 00:30:55.349113768 (80h7m30.912436772s) [seg_pos={0 0}, upload=64142]: Done begin upload segment. Jun 22 00:30:55.349143121 (80h7m30.912466131s) [noise=true , address=45.91.116.5:5310 , piece_id=AXJOKDZRPVNL7HFT4QKJWRAHFM2UCM3GEVQB2KNYD6CXB6TH4D4Q, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.349660917 (80h7m30.912983928s) [noise=true , address=77.90.31.10:4533 , piece_id=SE6YAKKFDL3BV677CDDUC32UNMPXJHTG23GWWYK5B4YIPKER6GAQ, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.350093077 (80h7m30.913416089s) [noise=true , address=79.127.201.215:4326 , piece_id=USEWZDX6P52U6MCALTBNQG3SV7N2JRKNU6O23TLS6JEWAS4D7ARA, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.350480894 (80h7m30.913803916s) [noise=true , address=104.36.82.13:6199 , piece_id=YP62N2TO446YR3QRG5YLXECD7TN6SJ4UNNLZSK5H4CMV6JIF4AKA, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.350669998 (80h7m30.913993011s) [noise=true , address=104.36.82.54:4184 , piece_id=IZICVGZFMAYLIJTNUJ2OFLZVNZ6IPPYDZO4BSTJJPLHVCVOE5EAA, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.351201352 (80h7m30.914524366s) [noise=true , address=104.36.82.55:4224 , piece_id=XF7H2BYSNNNZOS2T77VIN75N76AYUXAQEI6D4LRXACU2KU62V2TA, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.351668071 (80h7m30.914991083s) [noise=true , address=104.36.82.56:4261 , piece_id=EYOYOYLDACGMPGJYX6ILQT3XHMUTQNXZEDC7ANHLZXIKYVDQFFVQ, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.352193126 (80h7m30.915516138s) [noise=true , address=104.36.82.52:4089 , piece_id=X2VG4S3UHVNWA4AQJS7FLKWNVLTOZ3AKWI5ZDH24QCA3ENQW7BXQ, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.352587096 (80h7m30.915910103s) [noise=true , address=79.127.220.98:4402 , piece_id=TE2WQO5LK45AVMPIAIGVSIBOVZLC4FG4ESQ72X6KFYC7IAEN2DRA, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.353420253 (80h7m30.916743265s) [noise=true , address=104.36.82.11:6125 , piece_id=2CX7RJOKNQA6ODZZZXYPSCYAQY5P35TWX7I56CMYAPFLJE3JEO5Q, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.353791565 (80h7m30.917114578s) [noise=true , address=104.36.82.42:6803 , piece_id=R626UFD326S2UBJEZ2WMHVR7EN4SK427VXPDJZOOZ32MX25JWZWQ, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.354171226 (80h7m30.917494233s) [noise=true , address=104.36.82.14:6245 , piece_id=II36G46OA6TINLR244U6D7Q6LGV754UA5M62K3EX7ZOBESMJUEHA, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.354542282 (80h7m30.917865288s) [noise=true , address=79.127.205.235:5660 , piece_id=UGFPVS7VNB7U2YSFPSGYDIRZV5R2DHCN57PJP6VOXO52WIAYHLXA, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.354891655 (80h7m30.918214678s) [noise=true , address=104.36.82.15:6283 , piece_id=N7TH2EMIOPATVHCSNH2WXUGU7KMMEJ5LCHR22SWEHSA5OV4CUCXQ, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.355499503 (80h7m30.918822519s) [noise=true , address=77.90.31.14:4666 , piece_id=H4DTFDAVCN266PB235BJIQTQ32FBXTIB5AZSX3B4R2VIK554AFSQ, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.356575547 (80h7m30.919898539s) [noise=true , address=104.36.82.39:6756 , piece_id=OKKWMDJPZO6D5VRMGOE3GOCYGHM725JZTNOVYAWKJW3WYZPI4V5Q, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.357039590 (80h7m30.920362596s) [noise=true , address=104.36.82.53:4146 , piece_id=3V3DVGUOPJKDWIGQ2TU7M6VFYFVAS6ODY62ERGSU6LI3LSTWQ4YQ, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.360343566 (80h7m30.923666582s) [noise=true , address=104.36.82.9:6033 , piece_id=T5XA23LZELMSTWOLBSHOX4AHKDACF65OYCKTMZYTXDIBSXNA2T4A, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.452444064 (80h7m31.015767055s) [upload=64142]: Next returned nil segment Jun 22 00:30:55.452495148 (80h7m31.015818149s) [upload=64142]: Waiting for error group managing segments... Jun 22 00:30:55.532958808 (80h7m31.096281883s) [noise=true , address=104.36.82.9:6033 , piece_id=T5XA23LZELMSTWOLBSHOX4AHKDACF65OYCKTMZYTXDIBSXNA2T4A, seg_pos={0 0}, upload=64142]: Done uploading piece. err: ecclient: upload failed (node:1SV3LrcQW16wLewDzfyUWFB6sUgnERNJVdhz72CJmyde95M79C, address:104.36.82.9:6033): protocol: storage node overloaded, request limit: 2000 Jun 22 00:30:55.547802184 (80h7m31.111125188s) [noise=true , address=79.127.220.98:4402 , piece_id=TE2WQO5LK45AVMPIAIGVSIBOVZLC4FG4ESQ72X6KFYC7IAEN2DRA, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:30:55.554223621 (80h7m31.117546616s) [noise=true , address=77.90.31.14:4666 , piece_id=H4DTFDAVCN266PB235BJIQTQ32FBXTIB5AZSX3B4R2VIK554AFSQ, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:30:55.762514255 (80h7m31.325837245s) [noise=true , address=104.36.82.14:6245 , piece_id=II36G46OA6TINLR244U6D7Q6LGV754UA5M62K3EX7ZOBESMJUEHA, seg_pos={0 0}, upload=64142]: Done uploading piece. err: ecclient: upload failed (node:1Zu8teNPTXn638grHKwtJqz4UEY7oFYLx1DHvuVP6RZXcatVqX, address:104.36.82.14:6245): protocol: storage node overloaded, request limit: 2000 Jun 22 00:30:55.773669583 (80h7m31.3369926s) [noise=true , address=79.127.205.235:5660 , piece_id=UGFPVS7VNB7U2YSFPSGYDIRZV5R2DHCN57PJP6VOXO52WIAYHLXA, seg_pos={0 0}, upload=64142]: Done uploading piece. err: ecclient: upload failed (node:12E6uCX9KL4Ud7YpxMXepvMXuqTFJCiSXByDjW94Am3U9H2Nf7f, address:79.127.205.235:5660): protocol: storage node overloaded, request limit: 2000 Jun 22 00:30:56.968366655 (80h7m32.53168964s) [noise=true , address=77.90.31.10:4533 , piece_id=SE6YAKKFDL3BV677CDDUC32UNMPXJHTG23GWWYK5B4YIPKER6GAQ, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:30:57.403572874 (80h7m32.966895883s) [noise=true , address=45.91.116.5:5310 , piece_id=AXJOKDZRPVNL7HFT4QKJWRAHFM2UCM3GEVQB2KNYD6CXB6TH4D4Q, seg_pos={0 0}, upload=64142]: Done uploading piece. err: ecclient: upload failed (node:127RMUc62xRq9RU5z3XfkZsq5sehmv3kP8o53tr9BxVcPE3GQCY, address:45.91.116.5:5310): protocol: storage node overloaded, request limit: 2000 Jun 22 00:30:58.210389994 (80h7m33.773712995s) [noise=true , address=104.36.82.13:6199 , piece_id=YP62N2TO446YR3QRG5YLXECD7TN6SJ4UNNLZSK5H4CMV6JIF4AKA, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:30:58.693037961 (80h7m34.256360966s) [noise=true , address=104.36.82.39:6756 , piece_id=OKKWMDJPZO6D5VRMGOE3GOCYGHM725JZTNOVYAWKJW3WYZPI4V5Q, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:30:59.124834814 (80h7m34.688157819s) [noise=true , address=104.36.82.15:6283 , piece_id=N7TH2EMIOPATVHCSNH2WXUGU7KMMEJ5LCHR22SWEHSA5OV4CUCXQ, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:31:00.629657982 (80h7m36.192980977s) [noise=true , address=104.36.82.42:6803 , piece_id=R626UFD326S2UBJEZ2WMHVR7EN4SK427VXPDJZOOZ32MX25JWZWQ, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:31:00.933110546 (80h7m36.496433536s) [noise=true , address=79.127.201.215:4326 , piece_id=USEWZDX6P52U6MCALTBNQG3SV7N2JRKNU6O23TLS6JEWAS4D7ARA, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:31:05.669167243 (80h7m41.232490237s) [noise=true , address=104.36.82.56:4261 , piece_id=EYOYOYLDACGMPGJYX6ILQT3XHMUTQNXZEDC7ANHLZXIKYVDQFFVQ, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:31:08.443582763 (80h7m44.006905763s) [noise=true , address=104.36.82.54:4184 , piece_id=IZICVGZFMAYLIJTNUJ2OFLZVNZ6IPPYDZO4BSTJJPLHVCVOE5EAA, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:31:26.459138014 (80h8m2.022461013s) [noise=true , address=104.36.82.53:4146 , piece_id=3V3DVGUOPJKDWIGQ2TU7M6VFYFVAS6ODY62ERGSU6LI3LSTWQ4YQ, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:31:27.456050336 (80h8m3.019373336s) [noise=true , address=104.36.82.52:4089 , piece_id=X2VG4S3UHVNWA4AQJS7FLKWNVLTOZ3AKWI5ZDH24QCA3ENQW7BXQ, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:31:55.270234251 (80h8m30.833557256s) [noise=true , address=104.36.82.11:6125 , piece_id=2CX7RJOKNQA6ODZZZXYPSCYAQY5P35TWX7I56CMYAPFLJE3JEO5Q, seg_pos={0 0}, upload=64142]: Done uploading piece. err: ecclient: upload failed (node:12vricofwx67FRaxAGLbmgT1TYyKkqfJbxdsyeEPL4R7kg8pKzT, address:104.36.82.11:6125): protocol: expected piece hash; context deadline exceeded; EOF Jun 22 00:31:55.270319418 (80h8m30.833642407s) [noise=true , address=104.36.82.55:4224 , piece_id=XF7H2BYSNNNZOS2T77VIN75N76AYUXAQEI6D4LRXACU2KU62V2TA, seg_pos={0 0}, upload=64142]: Done uploading piece. err: ecclient: upload failed (node:12EjKmUiv4pxkVCB4WoiXim4uFpxrBPyfiSS4efTY3HKSE3jb5y, address:104.36.82.55:4224): protocol: write tcp 159.223.143.39:39152->104.36.82.55:4224: use of closed network connection; write tcp 159.223.143.39:39152->104.36.82.55:4224: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:39152->104.36.82.55:4224: use of closed network connection Jun 22 00:31:55.270353811 (80h8m30.833676799s) [seg_pos={0 0}, upload=64142]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Jun 22 00:31:55.301099877 (80h8m30.864422884s) [upload=64142]: Done uploading segments... Jun 22 00:31:55.301121567 (80h8m30.864444566s) [upload=64142]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 43.455885795s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [...X........................................................] duration: 9.788952268s FAIL (2025-06-22 01:14:47 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 9.789026758s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [....X.......................................................] duration: 18.509306094s FAIL (2025-06-22 01:14:21 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 18.509329988s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.....X......................................................] duration: 15.997078948s FAIL (2025-06-22 01:13:57 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 15.997098631s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.......X....................................................] duration: 17.757053968s FAIL (2025-06-22 01:13:37 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== ing upload Jun 22 00:30:55.303646184 (80h7m30.866969171s) [upload=64142]: Uploading segments... Jun 22 00:30:55.303706406 (80h7m30.867029416s) [upload=64142]: Got next segment. Inline: false Jun 22 00:30:55.303762479 (80h7m30.867085468s) [upload=64142]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 22 00:30:55.303768627 (80h7m30.867091636s) [upload=64142]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 22 00:30:55.349058140 (80h7m30.912381168s) [seg_pos={0 0}, upload=64142]: Begin upload segment... Jun 22 00:30:55.349113768 (80h7m30.912436772s) [seg_pos={0 0}, upload=64142]: Done begin upload segment. Jun 22 00:30:55.349143121 (80h7m30.912466131s) [noise=true , address=45.91.116.5:5310 , piece_id=AXJOKDZRPVNL7HFT4QKJWRAHFM2UCM3GEVQB2KNYD6CXB6TH4D4Q, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.349660917 (80h7m30.912983928s) [noise=true , address=77.90.31.10:4533 , piece_id=SE6YAKKFDL3BV677CDDUC32UNMPXJHTG23GWWYK5B4YIPKER6GAQ, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.350093077 (80h7m30.913416089s) [noise=true , address=79.127.201.215:4326 , piece_id=USEWZDX6P52U6MCALTBNQG3SV7N2JRKNU6O23TLS6JEWAS4D7ARA, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.350480894 (80h7m30.913803916s) [noise=true , address=104.36.82.13:6199 , piece_id=YP62N2TO446YR3QRG5YLXECD7TN6SJ4UNNLZSK5H4CMV6JIF4AKA, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.350669998 (80h7m30.913993011s) [noise=true , address=104.36.82.54:4184 , piece_id=IZICVGZFMAYLIJTNUJ2OFLZVNZ6IPPYDZO4BSTJJPLHVCVOE5EAA, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.351201352 (80h7m30.914524366s) [noise=true , address=104.36.82.55:4224 , piece_id=XF7H2BYSNNNZOS2T77VIN75N76AYUXAQEI6D4LRXACU2KU62V2TA, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.351668071 (80h7m30.914991083s) [noise=true , address=104.36.82.56:4261 , piece_id=EYOYOYLDACGMPGJYX6ILQT3XHMUTQNXZEDC7ANHLZXIKYVDQFFVQ, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.352193126 (80h7m30.915516138s) [noise=true , address=104.36.82.52:4089 , piece_id=X2VG4S3UHVNWA4AQJS7FLKWNVLTOZ3AKWI5ZDH24QCA3ENQW7BXQ, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.352587096 (80h7m30.915910103s) [noise=true , address=79.127.220.98:4402 , piece_id=TE2WQO5LK45AVMPIAIGVSIBOVZLC4FG4ESQ72X6KFYC7IAEN2DRA, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.353420253 (80h7m30.916743265s) [noise=true , address=104.36.82.11:6125 , piece_id=2CX7RJOKNQA6ODZZZXYPSCYAQY5P35TWX7I56CMYAPFLJE3JEO5Q, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.353791565 (80h7m30.917114578s) [noise=true , address=104.36.82.42:6803 , piece_id=R626UFD326S2UBJEZ2WMHVR7EN4SK427VXPDJZOOZ32MX25JWZWQ, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.354171226 (80h7m30.917494233s) [noise=true , address=104.36.82.14:6245 , piece_id=II36G46OA6TINLR244U6D7Q6LGV754UA5M62K3EX7ZOBESMJUEHA, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.354542282 (80h7m30.917865288s) [noise=true , address=79.127.205.235:5660 , piece_id=UGFPVS7VNB7U2YSFPSGYDIRZV5R2DHCN57PJP6VOXO52WIAYHLXA, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.354891655 (80h7m30.918214678s) [noise=true , address=104.36.82.15:6283 , piece_id=N7TH2EMIOPATVHCSNH2WXUGU7KMMEJ5LCHR22SWEHSA5OV4CUCXQ, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.355499503 (80h7m30.918822519s) [noise=true , address=77.90.31.14:4666 , piece_id=H4DTFDAVCN266PB235BJIQTQ32FBXTIB5AZSX3B4R2VIK554AFSQ, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.356575547 (80h7m30.919898539s) [noise=true , address=104.36.82.39:6756 , piece_id=OKKWMDJPZO6D5VRMGOE3GOCYGHM725JZTNOVYAWKJW3WYZPI4V5Q, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.357039590 (80h7m30.920362596s) [noise=true , address=104.36.82.53:4146 , piece_id=3V3DVGUOPJKDWIGQ2TU7M6VFYFVAS6ODY62ERGSU6LI3LSTWQ4YQ, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.360343566 (80h7m30.923666582s) [noise=true , address=104.36.82.9:6033 , piece_id=T5XA23LZELMSTWOLBSHOX4AHKDACF65OYCKTMZYTXDIBSXNA2T4A, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.452444064 (80h7m31.015767055s) [upload=64142]: Next returned nil segment Jun 22 00:30:55.452495148 (80h7m31.015818149s) [upload=64142]: Waiting for error group managing segments... Jun 22 00:30:55.532958808 (80h7m31.096281883s) [noise=true , address=104.36.82.9:6033 , piece_id=T5XA23LZELMSTWOLBSHOX4AHKDACF65OYCKTMZYTXDIBSXNA2T4A, seg_pos={0 0}, upload=64142]: Done uploading piece. err: ecclient: upload failed (node:1SV3LrcQW16wLewDzfyUWFB6sUgnERNJVdhz72CJmyde95M79C, address:104.36.82.9:6033): protocol: storage node overloaded, request limit: 2000 Jun 22 00:30:55.547802184 (80h7m31.111125188s) [noise=true , address=79.127.220.98:4402 , piece_id=TE2WQO5LK45AVMPIAIGVSIBOVZLC4FG4ESQ72X6KFYC7IAEN2DRA, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:30:55.554223621 (80h7m31.117546616s) [noise=true , address=77.90.31.14:4666 , piece_id=H4DTFDAVCN266PB235BJIQTQ32FBXTIB5AZSX3B4R2VIK554AFSQ, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:30:55.762514255 (80h7m31.325837245s) [noise=true , address=104.36.82.14:6245 , piece_id=II36G46OA6TINLR244U6D7Q6LGV754UA5M62K3EX7ZOBESMJUEHA, seg_pos={0 0}, upload=64142]: Done uploading piece. err: ecclient: upload failed (node:1Zu8teNPTXn638grHKwtJqz4UEY7oFYLx1DHvuVP6RZXcatVqX, address:104.36.82.14:6245): protocol: storage node overloaded, request limit: 2000 Jun 22 00:30:55.773669583 (80h7m31.3369926s) [noise=true , address=79.127.205.235:5660 , piece_id=UGFPVS7VNB7U2YSFPSGYDIRZV5R2DHCN57PJP6VOXO52WIAYHLXA, seg_pos={0 0}, upload=64142]: Done uploading piece. err: ecclient: upload failed (node:12E6uCX9KL4Ud7YpxMXepvMXuqTFJCiSXByDjW94Am3U9H2Nf7f, address:79.127.205.235:5660): protocol: storage node overloaded, request limit: 2000 Jun 22 00:30:56.968366655 (80h7m32.53168964s) [noise=true , address=77.90.31.10:4533 , piece_id=SE6YAKKFDL3BV677CDDUC32UNMPXJHTG23GWWYK5B4YIPKER6GAQ, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:30:57.403572874 (80h7m32.966895883s) [noise=true , address=45.91.116.5:5310 , piece_id=AXJOKDZRPVNL7HFT4QKJWRAHFM2UCM3GEVQB2KNYD6CXB6TH4D4Q, seg_pos={0 0}, upload=64142]: Done uploading piece. err: ecclient: upload failed (node:127RMUc62xRq9RU5z3XfkZsq5sehmv3kP8o53tr9BxVcPE3GQCY, address:45.91.116.5:5310): protocol: storage node overloaded, request limit: 2000 Jun 22 00:30:58.210389994 (80h7m33.773712995s) [noise=true , address=104.36.82.13:6199 , piece_id=YP62N2TO446YR3QRG5YLXECD7TN6SJ4UNNLZSK5H4CMV6JIF4AKA, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:30:58.693037961 (80h7m34.256360966s) [noise=true , address=104.36.82.39:6756 , piece_id=OKKWMDJPZO6D5VRMGOE3GOCYGHM725JZTNOVYAWKJW3WYZPI4V5Q, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:30:59.124834814 (80h7m34.688157819s) [noise=true , address=104.36.82.15:6283 , piece_id=N7TH2EMIOPATVHCSNH2WXUGU7KMMEJ5LCHR22SWEHSA5OV4CUCXQ, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:31:00.629657982 (80h7m36.192980977s) [noise=true , address=104.36.82.42:6803 , piece_id=R626UFD326S2UBJEZ2WMHVR7EN4SK427VXPDJZOOZ32MX25JWZWQ, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:31:00.933110546 (80h7m36.496433536s) [noise=true , address=79.127.201.215:4326 , piece_id=USEWZDX6P52U6MCALTBNQG3SV7N2JRKNU6O23TLS6JEWAS4D7ARA, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:31:05.669167243 (80h7m41.232490237s) [noise=true , address=104.36.82.56:4261 , piece_id=EYOYOYLDACGMPGJYX6ILQT3XHMUTQNXZEDC7ANHLZXIKYVDQFFVQ, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:31:08.443582763 (80h7m44.006905763s) [noise=true , address=104.36.82.54:4184 , piece_id=IZICVGZFMAYLIJTNUJ2OFLZVNZ6IPPYDZO4BSTJJPLHVCVOE5EAA, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:31:26.459138014 (80h8m2.022461013s) [noise=true , address=104.36.82.53:4146 , piece_id=3V3DVGUOPJKDWIGQ2TU7M6VFYFVAS6ODY62ERGSU6LI3LSTWQ4YQ, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:31:27.456050336 (80h8m3.019373336s) [noise=true , address=104.36.82.52:4089 , piece_id=X2VG4S3UHVNWA4AQJS7FLKWNVLTOZ3AKWI5ZDH24QCA3ENQW7BXQ, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:31:55.270234251 (80h8m30.833557256s) [noise=true , address=104.36.82.11:6125 , piece_id=2CX7RJOKNQA6ODZZZXYPSCYAQY5P35TWX7I56CMYAPFLJE3JEO5Q, seg_pos={0 0}, upload=64142]: Done uploading piece. err: ecclient: upload failed (node:12vricofwx67FRaxAGLbmgT1TYyKkqfJbxdsyeEPL4R7kg8pKzT, address:104.36.82.11:6125): protocol: expected piece hash; context deadline exceeded; EOF Jun 22 00:31:55.270319418 (80h8m30.833642407s) [noise=true , address=104.36.82.55:4224 , piece_id=XF7H2BYSNNNZOS2T77VIN75N76AYUXAQEI6D4LRXACU2KU62V2TA, seg_pos={0 0}, upload=64142]: Done uploading piece. err: ecclient: upload failed (node:12EjKmUiv4pxkVCB4WoiXim4uFpxrBPyfiSS4efTY3HKSE3jb5y, address:104.36.82.55:4224): protocol: write tcp 159.223.143.39:39152->104.36.82.55:4224: use of closed network connection; write tcp 159.223.143.39:39152->104.36.82.55:4224: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:39152->104.36.82.55:4224: use of closed network connection Jun 22 00:31:55.270353811 (80h8m30.833676799s) [seg_pos={0 0}, upload=64142]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Jun 22 00:31:55.301099877 (80h8m30.864422884s) [upload=64142]: Done uploading segments... Jun 22 00:31:55.301121567 (80h8m30.864444566s) [upload=64142]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 17.757081434s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [......X.....................................................] duration: 18.532376076s FAIL (2025-06-22 01:13:17 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 18.532411407s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [........X...................................................] duration: 20.33316048s FAIL (2025-06-22 01:12:55 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== ing upload Jun 22 00:30:55.303646184 (80h7m30.866969171s) [upload=64142]: Uploading segments... Jun 22 00:30:55.303706406 (80h7m30.867029416s) [upload=64142]: Got next segment. Inline: false Jun 22 00:30:55.303762479 (80h7m30.867085468s) [upload=64142]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 22 00:30:55.303768627 (80h7m30.867091636s) [upload=64142]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 22 00:30:55.349058140 (80h7m30.912381168s) [seg_pos={0 0}, upload=64142]: Begin upload segment... Jun 22 00:30:55.349113768 (80h7m30.912436772s) [seg_pos={0 0}, upload=64142]: Done begin upload segment. Jun 22 00:30:55.349143121 (80h7m30.912466131s) [noise=true , address=45.91.116.5:5310 , piece_id=AXJOKDZRPVNL7HFT4QKJWRAHFM2UCM3GEVQB2KNYD6CXB6TH4D4Q, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.349660917 (80h7m30.912983928s) [noise=true , address=77.90.31.10:4533 , piece_id=SE6YAKKFDL3BV677CDDUC32UNMPXJHTG23GWWYK5B4YIPKER6GAQ, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.350093077 (80h7m30.913416089s) [noise=true , address=79.127.201.215:4326 , piece_id=USEWZDX6P52U6MCALTBNQG3SV7N2JRKNU6O23TLS6JEWAS4D7ARA, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.350480894 (80h7m30.913803916s) [noise=true , address=104.36.82.13:6199 , piece_id=YP62N2TO446YR3QRG5YLXECD7TN6SJ4UNNLZSK5H4CMV6JIF4AKA, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.350669998 (80h7m30.913993011s) [noise=true , address=104.36.82.54:4184 , piece_id=IZICVGZFMAYLIJTNUJ2OFLZVNZ6IPPYDZO4BSTJJPLHVCVOE5EAA, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.351201352 (80h7m30.914524366s) [noise=true , address=104.36.82.55:4224 , piece_id=XF7H2BYSNNNZOS2T77VIN75N76AYUXAQEI6D4LRXACU2KU62V2TA, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.351668071 (80h7m30.914991083s) [noise=true , address=104.36.82.56:4261 , piece_id=EYOYOYLDACGMPGJYX6ILQT3XHMUTQNXZEDC7ANHLZXIKYVDQFFVQ, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.352193126 (80h7m30.915516138s) [noise=true , address=104.36.82.52:4089 , piece_id=X2VG4S3UHVNWA4AQJS7FLKWNVLTOZ3AKWI5ZDH24QCA3ENQW7BXQ, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.352587096 (80h7m30.915910103s) [noise=true , address=79.127.220.98:4402 , piece_id=TE2WQO5LK45AVMPIAIGVSIBOVZLC4FG4ESQ72X6KFYC7IAEN2DRA, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.353420253 (80h7m30.916743265s) [noise=true , address=104.36.82.11:6125 , piece_id=2CX7RJOKNQA6ODZZZXYPSCYAQY5P35TWX7I56CMYAPFLJE3JEO5Q, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.353791565 (80h7m30.917114578s) [noise=true , address=104.36.82.42:6803 , piece_id=R626UFD326S2UBJEZ2WMHVR7EN4SK427VXPDJZOOZ32MX25JWZWQ, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.354171226 (80h7m30.917494233s) [noise=true , address=104.36.82.14:6245 , piece_id=II36G46OA6TINLR244U6D7Q6LGV754UA5M62K3EX7ZOBESMJUEHA, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.354542282 (80h7m30.917865288s) [noise=true , address=79.127.205.235:5660 , piece_id=UGFPVS7VNB7U2YSFPSGYDIRZV5R2DHCN57PJP6VOXO52WIAYHLXA, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.354891655 (80h7m30.918214678s) [noise=true , address=104.36.82.15:6283 , piece_id=N7TH2EMIOPATVHCSNH2WXUGU7KMMEJ5LCHR22SWEHSA5OV4CUCXQ, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.355499503 (80h7m30.918822519s) [noise=true , address=77.90.31.14:4666 , piece_id=H4DTFDAVCN266PB235BJIQTQ32FBXTIB5AZSX3B4R2VIK554AFSQ, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.356575547 (80h7m30.919898539s) [noise=true , address=104.36.82.39:6756 , piece_id=OKKWMDJPZO6D5VRMGOE3GOCYGHM725JZTNOVYAWKJW3WYZPI4V5Q, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.357039590 (80h7m30.920362596s) [noise=true , address=104.36.82.53:4146 , piece_id=3V3DVGUOPJKDWIGQ2TU7M6VFYFVAS6ODY62ERGSU6LI3LSTWQ4YQ, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.360343566 (80h7m30.923666582s) [noise=true , address=104.36.82.9:6033 , piece_id=T5XA23LZELMSTWOLBSHOX4AHKDACF65OYCKTMZYTXDIBSXNA2T4A, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.452444064 (80h7m31.015767055s) [upload=64142]: Next returned nil segment Jun 22 00:30:55.452495148 (80h7m31.015818149s) [upload=64142]: Waiting for error group managing segments... Jun 22 00:30:55.532958808 (80h7m31.096281883s) [noise=true , address=104.36.82.9:6033 , piece_id=T5XA23LZELMSTWOLBSHOX4AHKDACF65OYCKTMZYTXDIBSXNA2T4A, seg_pos={0 0}, upload=64142]: Done uploading piece. err: ecclient: upload failed (node:1SV3LrcQW16wLewDzfyUWFB6sUgnERNJVdhz72CJmyde95M79C, address:104.36.82.9:6033): protocol: storage node overloaded, request limit: 2000 Jun 22 00:30:55.547802184 (80h7m31.111125188s) [noise=true , address=79.127.220.98:4402 , piece_id=TE2WQO5LK45AVMPIAIGVSIBOVZLC4FG4ESQ72X6KFYC7IAEN2DRA, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:30:55.554223621 (80h7m31.117546616s) [noise=true , address=77.90.31.14:4666 , piece_id=H4DTFDAVCN266PB235BJIQTQ32FBXTIB5AZSX3B4R2VIK554AFSQ, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:30:55.762514255 (80h7m31.325837245s) [noise=true , address=104.36.82.14:6245 , piece_id=II36G46OA6TINLR244U6D7Q6LGV754UA5M62K3EX7ZOBESMJUEHA, seg_pos={0 0}, upload=64142]: Done uploading piece. err: ecclient: upload failed (node:1Zu8teNPTXn638grHKwtJqz4UEY7oFYLx1DHvuVP6RZXcatVqX, address:104.36.82.14:6245): protocol: storage node overloaded, request limit: 2000 Jun 22 00:30:55.773669583 (80h7m31.3369926s) [noise=true , address=79.127.205.235:5660 , piece_id=UGFPVS7VNB7U2YSFPSGYDIRZV5R2DHCN57PJP6VOXO52WIAYHLXA, seg_pos={0 0}, upload=64142]: Done uploading piece. err: ecclient: upload failed (node:12E6uCX9KL4Ud7YpxMXepvMXuqTFJCiSXByDjW94Am3U9H2Nf7f, address:79.127.205.235:5660): protocol: storage node overloaded, request limit: 2000 Jun 22 00:30:56.968366655 (80h7m32.53168964s) [noise=true , address=77.90.31.10:4533 , piece_id=SE6YAKKFDL3BV677CDDUC32UNMPXJHTG23GWWYK5B4YIPKER6GAQ, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:30:57.403572874 (80h7m32.966895883s) [noise=true , address=45.91.116.5:5310 , piece_id=AXJOKDZRPVNL7HFT4QKJWRAHFM2UCM3GEVQB2KNYD6CXB6TH4D4Q, seg_pos={0 0}, upload=64142]: Done uploading piece. err: ecclient: upload failed (node:127RMUc62xRq9RU5z3XfkZsq5sehmv3kP8o53tr9BxVcPE3GQCY, address:45.91.116.5:5310): protocol: storage node overloaded, request limit: 2000 Jun 22 00:30:58.210389994 (80h7m33.773712995s) [noise=true , address=104.36.82.13:6199 , piece_id=YP62N2TO446YR3QRG5YLXECD7TN6SJ4UNNLZSK5H4CMV6JIF4AKA, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:30:58.693037961 (80h7m34.256360966s) [noise=true , address=104.36.82.39:6756 , piece_id=OKKWMDJPZO6D5VRMGOE3GOCYGHM725JZTNOVYAWKJW3WYZPI4V5Q, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:30:59.124834814 (80h7m34.688157819s) [noise=true , address=104.36.82.15:6283 , piece_id=N7TH2EMIOPATVHCSNH2WXUGU7KMMEJ5LCHR22SWEHSA5OV4CUCXQ, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:31:00.629657982 (80h7m36.192980977s) [noise=true , address=104.36.82.42:6803 , piece_id=R626UFD326S2UBJEZ2WMHVR7EN4SK427VXPDJZOOZ32MX25JWZWQ, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:31:00.933110546 (80h7m36.496433536s) [noise=true , address=79.127.201.215:4326 , piece_id=USEWZDX6P52U6MCALTBNQG3SV7N2JRKNU6O23TLS6JEWAS4D7ARA, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:31:05.669167243 (80h7m41.232490237s) [noise=true , address=104.36.82.56:4261 , piece_id=EYOYOYLDACGMPGJYX6ILQT3XHMUTQNXZEDC7ANHLZXIKYVDQFFVQ, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:31:08.443582763 (80h7m44.006905763s) [noise=true , address=104.36.82.54:4184 , piece_id=IZICVGZFMAYLIJTNUJ2OFLZVNZ6IPPYDZO4BSTJJPLHVCVOE5EAA, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:31:26.459138014 (80h8m2.022461013s) [noise=true , address=104.36.82.53:4146 , piece_id=3V3DVGUOPJKDWIGQ2TU7M6VFYFVAS6ODY62ERGSU6LI3LSTWQ4YQ, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:31:27.456050336 (80h8m3.019373336s) [noise=true , address=104.36.82.52:4089 , piece_id=X2VG4S3UHVNWA4AQJS7FLKWNVLTOZ3AKWI5ZDH24QCA3ENQW7BXQ, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:31:55.270234251 (80h8m30.833557256s) [noise=true , address=104.36.82.11:6125 , piece_id=2CX7RJOKNQA6ODZZZXYPSCYAQY5P35TWX7I56CMYAPFLJE3JEO5Q, seg_pos={0 0}, upload=64142]: Done uploading piece. err: ecclient: upload failed (node:12vricofwx67FRaxAGLbmgT1TYyKkqfJbxdsyeEPL4R7kg8pKzT, address:104.36.82.11:6125): protocol: expected piece hash; context deadline exceeded; EOF Jun 22 00:31:55.270319418 (80h8m30.833642407s) [noise=true , address=104.36.82.55:4224 , piece_id=XF7H2BYSNNNZOS2T77VIN75N76AYUXAQEI6D4LRXACU2KU62V2TA, seg_pos={0 0}, upload=64142]: Done uploading piece. err: ecclient: upload failed (node:12EjKmUiv4pxkVCB4WoiXim4uFpxrBPyfiSS4efTY3HKSE3jb5y, address:104.36.82.55:4224): protocol: write tcp 159.223.143.39:39152->104.36.82.55:4224: use of closed network connection; write tcp 159.223.143.39:39152->104.36.82.55:4224: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:39152->104.36.82.55:4224: use of closed network connection Jun 22 00:31:55.270353811 (80h8m30.833676799s) [seg_pos={0 0}, upload=64142]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Jun 22 00:31:55.301099877 (80h8m30.864422884s) [upload=64142]: Done uploading segments... Jun 22 00:31:55.301121567 (80h8m30.864444566s) [upload=64142]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 20.333195678s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.......X....................................................] duration: 13.712531294s FAIL (2025-06-22 01:12:52 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 13.712555481s core uplink us-select-1 us1-placement-20

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)