Orbiter History

checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [......................................X.................X...] duration: 17.928945476s FAIL (2025-06-22 02:01:23 UTC) orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded Duration: 17.928975783s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [........................X...................................] duration: 20.07768635s FAIL (2025-06-22 02:01:01 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== oad=64949]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jun 22 01:32:13.674692303 (81h8m49.238015311s) [upload=64949]: Done uploading segments... Jun 22 01:32:13.674727219 (81h8m49.238050233s) [upload=64949]: Upload finished. err: <nil> Jun 22 01:32:47.266407995 (81h9m22.829731002s) [upload=64958]: Starting upload Jun 22 01:32:47.266524767 (81h9m22.829847771s) [upload=64958]: Done starting upload Jun 22 01:32:47.266813109 (81h9m22.830136116s) [upload=64958]: Uploading segments... Jun 22 01:32:47.266969526 (81h9m22.830292526s) [upload=64958]: Got next segment. Inline: false Jun 22 01:32:47.267075475 (81h9m22.830398475s) [upload=64958]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 22 01:32:47.267081679 (81h9m22.830404667s) [upload=64958]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 22 01:32:47.301770382 (81h9m22.865093389s) [seg_pos={0 0}, upload=64958]: Begin upload segment... Jun 22 01:32:47.301823796 (81h9m22.865146799s) [seg_pos={0 0}, upload=64958]: Done begin upload segment. Jun 22 01:32:47.301855055 (81h9m22.865178087s) [noise=true , address=104.36.82.43:6841 , piece_id=PIHZMSTO3CII7KGDLGSAILTDRSLMEBVOLBCRCZJULMCFISXIZWXA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.301902600 (81h9m22.865225606s) [noise=true , address=104.36.82.42:6811 , piece_id=A5BFEHZ3WGGK53IZ7MKI6N4CVXHOORF6L47NR3UFDCAX7LBHRHVQ, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.302234027 (81h9m22.865557029s) [noise=true , address=104.36.82.51:4084 , piece_id=D43FA7BFE54KPCWGDGDME2EGSWM7PONOVPD3DN2XY42MJFOKVDIQ, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.302496354 (81h9m22.865819363s) [noise=true , address=79.127.219.38:4023 , piece_id=BWLRCXXHBMBTNCZEVKKIIS5PB35PDQXQXCLIQB7GBLEIOWWUFBFA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.302577370 (81h9m22.865900378s) [noise=true , address=104.36.82.52:4089 , piece_id=JSOEKLNWA3W3LOVKI3EGGKJYJ6J3UZY6KGABKQPQPXTIOOPMKVHA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.302917829 (81h9m22.866240835s) [noise=true , address=104.36.82.53:4122 , piece_id=HZZQBIKHQWLUE5EHK4KSKJGIUEG7D44RQASUJHCYXX6BICCGX6EA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.303087165 (81h9m22.866410168s) [noise=true , address=77.90.31.14:4689 , piece_id=Q3HDS3BCMYDTODFRMU4XFRDCQK2QZE3ZMXVLUX4M54PDEBS7RR6A, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.303250714 (81h9m22.866573717s) [noise=true , address=79.127.205.234:5601 , piece_id=V7PMXBP2VFWBSZDG4GTLBD435T6F5DT7A43L4GQLQ5BDDTJ4WY6A, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.303579249 (81h9m22.866902255s) [noise=true , address=104.36.82.12:6158 , piece_id=LJZEOJVJCJAR2BRILIJI3ZVDGYMPBBTP2G3HCTN3MHNS4KYDPASQ, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.303599776 (81h9m22.866922784s) [noise=true , address=104.36.82.9:6023 , piece_id=SISFAUU4RPF57OLGWM2BWQWE6GRUSCMORBYHWEXETNAVM2EEMO6Q, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.304005346 (81h9m22.867328346s) [noise=true , address=45.91.116.12:5242 , piece_id=6ABT5IPYGSLFFJMBYMBYRU4EYIXABUAUHG3OVILOHBIUDS3BP27A, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.304388850 (81h9m22.867711851s) [noise=true , address=64.49.13.137:4500 , piece_id=FAE2RXD57IRP5FXA2XR6THXFW27U7IWAZ4JDV5M2WRPXKUGZ4VIA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.304881529 (81h9m22.868204536s) [noise=true , address=79.127.220.101:6048 , piece_id=AN7UQUXB42T2YXEH7ZJRZUDIW47M3RWUEOCL2VEBOGH74BZF3YKA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.305249860 (81h9m22.86857287s) [noise=true , address=104.36.82.54:4165 , piece_id=4CVOABPCWC7J7SRHZB6IN2WJJ6KGWUQ66ELUIZIFA67ETLQ7JEUQ, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.305604108 (81h9m22.868927112s) [noise=true , address=104.36.82.15:6289 , piece_id=GWL3P55BIK4IJEL5YFZKI7JCA4LADYOOHUSIQUNGJEKBJOTKMP5A, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.305930423 (81h9m22.869253425s) [noise=true , address=64.49.13.139:4476 , piece_id=P7UKBNNZGI52B3I6WKTV6MVC262FYNKYYPJUI5G47UO5S4MU7PWA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.306355311 (81h9m22.869678319s) [noise=true , address=104.36.82.55:4213 , piece_id=X4IHS2XSPH52LIMNUYT7KGWMAPUH3TEMJOEEVB3JV5N45SEOH7YA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.306788243 (81h9m22.870111252s) [noise=true , address=104.36.82.22:6539 , piece_id=GHQ2MVSTDGVYLUKLV23D6YWRR4PP7UUYHCUEQ4WWSWEMTBFO3P4A, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.392174770 (81h9m22.955497773s) [upload=64958]: Next returned nil segment Jun 22 01:32:47.392195767 (81h9m22.955518775s) [upload=64958]: Waiting for error group managing segments... Jun 22 01:32:47.480767633 (81h9m23.044090623s) [noise=true , address=104.36.82.22:6539 , piece_id=GHQ2MVSTDGVYLUKLV23D6YWRR4PP7UUYHCUEQ4WWSWEMTBFO3P4A, seg_pos={0 0}, upload=64958]: Done uploading piece. err: ecclient: upload failed (node:1QL6w8DSFR6CxDdnJmDuyVZxp6JU55VKiJofBZRfQi7q31i8Jc, address:104.36.82.22:6539): protocol: storage node overloaded, request limit: 2000 Jun 22 01:32:47.585663142 (81h9m23.148986155s) [noise=true , address=79.127.220.101:6048 , piece_id=AN7UQUXB42T2YXEH7ZJRZUDIW47M3RWUEOCL2VEBOGH74BZF3YKA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:47.640263002 (81h9m23.203586013s) [noise=true , address=45.91.116.12:5242 , piece_id=6ABT5IPYGSLFFJMBYMBYRU4EYIXABUAUHG3OVILOHBIUDS3BP27A, seg_pos={0 0}, upload=64958]: Done uploading piece. err: ecclient: upload failed (node:12JsFL62jgXmza8Md8eQ9k8cyw5V9Vz2RzN4y85ZvgHZjwrq4vQ, address:45.91.116.12:5242): protocol: storage node overloaded, request limit: 2000 Jun 22 01:32:47.984127275 (81h9m23.54745028s) [noise=true , address=64.49.13.137:4500 , piece_id=FAE2RXD57IRP5FXA2XR6THXFW27U7IWAZ4JDV5M2WRPXKUGZ4VIA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:48.100255249 (81h9m23.663578241s) [noise=true , address=77.90.31.14:4689 , piece_id=Q3HDS3BCMYDTODFRMU4XFRDCQK2QZE3ZMXVLUX4M54PDEBS7RR6A, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:48.221764035 (81h9m23.785087034s) [noise=true , address=104.36.82.54:4165 , piece_id=4CVOABPCWC7J7SRHZB6IN2WJJ6KGWUQ66ELUIZIFA67ETLQ7JEUQ, seg_pos={0 0}, upload=64958]: Done uploading piece. err: ecclient: upload failed (node:12wGUMXVEubEtki4qGnjGEsbYFB1iUauZcu3oDRKrCQsJMV3bV2, address:104.36.82.54:4165): protocol: storage node overloaded, request limit: 2000 Jun 22 01:32:48.509996527 (81h9m24.073319547s) [noise=true , address=104.36.82.9:6023 , piece_id=SISFAUU4RPF57OLGWM2BWQWE6GRUSCMORBYHWEXETNAVM2EEMO6Q, seg_pos={0 0}, upload=64958]: Done uploading piece. err: ecclient: upload failed (node:1j2S9tgWpLwmyvXZSq2qKKcbaC8r7LkzfDCaMxCejAni4CST1g, address:104.36.82.9:6023): protocol: storage node overloaded, request limit: 2000 Jun 22 01:32:48.510513169 (81h9m24.073836177s) [noise=true , address=104.36.82.42:6811 , piece_id=A5BFEHZ3WGGK53IZ7MKI6N4CVXHOORF6L47NR3UFDCAX7LBHRHVQ, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:48.565245605 (81h9m24.128568606s) [noise=true , address=79.127.205.234:5601 , piece_id=V7PMXBP2VFWBSZDG4GTLBD435T6F5DT7A43L4GQLQ5BDDTJ4WY6A, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:49.979979824 (81h9m25.543302825s) [noise=true , address=104.36.82.15:6289 , piece_id=GWL3P55BIK4IJEL5YFZKI7JCA4LADYOOHUSIQUNGJEKBJOTKMP5A, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:50.463750280 (81h9m26.02707328s) [noise=true , address=104.36.82.51:4084 , piece_id=D43FA7BFE54KPCWGDGDME2EGSWM7PONOVPD3DN2XY42MJFOKVDIQ, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:51.480599795 (81h9m27.043922798s) [noise=true , address=79.127.219.38:4023 , piece_id=BWLRCXXHBMBTNCZEVKKIIS5PB35PDQXQXCLIQB7GBLEIOWWUFBFA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:52.941200625 (81h9m28.504523627s) [noise=true , address=104.36.82.43:6841 , piece_id=PIHZMSTO3CII7KGDLGSAILTDRSLMEBVOLBCRCZJULMCFISXIZWXA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:58.281803598 (81h9m33.845126612s) [noise=true , address=64.49.13.139:4476 , piece_id=P7UKBNNZGI52B3I6WKTV6MVC262FYNKYYPJUI5G47UO5S4MU7PWA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:33:00.277271877 (81h9m35.840594867s) [noise=true , address=104.36.82.55:4213 , piece_id=X4IHS2XSPH52LIMNUYT7KGWMAPUH3TEMJOEEVB3JV5N45SEOH7YA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:33:09.435339213 (81h9m44.998662206s) [noise=true , address=104.36.82.52:4089 , piece_id=JSOEKLNWA3W3LOVKI3EGGKJYJ6J3UZY6KGABKQPQPXTIOOPMKVHA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:33:31.472868445 (81h10m7.03619143s) [noise=true , address=104.36.82.53:4122 , piece_id=HZZQBIKHQWLUE5EHK4KSKJGIUEG7D44RQASUJHCYXX6BICCGX6EA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:33:47.235404653 (81h10m22.798727642s) [noise=true , address=104.36.82.12:6158 , piece_id=LJZEOJVJCJAR2BRILIJI3ZVDGYMPBBTP2G3HCTN3MHNS4KYDPASQ, seg_pos={0 0}, upload=64958]: Done uploading piece. err: ecclient: upload failed (node:1SjM6Utm2YizeUbrBUD9Z7etE1nLadW5Fa1zYP7MioPeeqhdJ7, address:104.36.82.12:6158): protocol: expected piece hash; context deadline exceeded; EOF Jun 22 01:33:47.235448297 (81h10m22.798771306s) [seg_pos={0 0}, upload=64958]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jun 22 01:33:47.265442393 (81h10m22.828765401s) [upload=64958]: Done uploading segments... Jun 22 01:33:47.265476139 (81h10m22.82879914s) [upload=64958]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration:* 20.077840452s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [.......................................X.................X..] duration: 15.192784319s FAIL (2025-06-22 02:00:45 UTC) orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded Duration: 15.192834356s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.........................X..................................] duration: 20.800677498s FAIL (2025-06-22 02:00:41 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== oad=64949]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jun 22 01:32:13.674692303 (81h8m49.238015311s) [upload=64949]: Done uploading segments... Jun 22 01:32:13.674727219 (81h8m49.238050233s) [upload=64949]: Upload finished. err: <nil> Jun 22 01:32:47.266407995 (81h9m22.829731002s) [upload=64958]: Starting upload Jun 22 01:32:47.266524767 (81h9m22.829847771s) [upload=64958]: Done starting upload Jun 22 01:32:47.266813109 (81h9m22.830136116s) [upload=64958]: Uploading segments... Jun 22 01:32:47.266969526 (81h9m22.830292526s) [upload=64958]: Got next segment. Inline: false Jun 22 01:32:47.267075475 (81h9m22.830398475s) [upload=64958]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 22 01:32:47.267081679 (81h9m22.830404667s) [upload=64958]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 22 01:32:47.301770382 (81h9m22.865093389s) [seg_pos={0 0}, upload=64958]: Begin upload segment... Jun 22 01:32:47.301823796 (81h9m22.865146799s) [seg_pos={0 0}, upload=64958]: Done begin upload segment. Jun 22 01:32:47.301855055 (81h9m22.865178087s) [noise=true , address=104.36.82.43:6841 , piece_id=PIHZMSTO3CII7KGDLGSAILTDRSLMEBVOLBCRCZJULMCFISXIZWXA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.301902600 (81h9m22.865225606s) [noise=true , address=104.36.82.42:6811 , piece_id=A5BFEHZ3WGGK53IZ7MKI6N4CVXHOORF6L47NR3UFDCAX7LBHRHVQ, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.302234027 (81h9m22.865557029s) [noise=true , address=104.36.82.51:4084 , piece_id=D43FA7BFE54KPCWGDGDME2EGSWM7PONOVPD3DN2XY42MJFOKVDIQ, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.302496354 (81h9m22.865819363s) [noise=true , address=79.127.219.38:4023 , piece_id=BWLRCXXHBMBTNCZEVKKIIS5PB35PDQXQXCLIQB7GBLEIOWWUFBFA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.302577370 (81h9m22.865900378s) [noise=true , address=104.36.82.52:4089 , piece_id=JSOEKLNWA3W3LOVKI3EGGKJYJ6J3UZY6KGABKQPQPXTIOOPMKVHA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.302917829 (81h9m22.866240835s) [noise=true , address=104.36.82.53:4122 , piece_id=HZZQBIKHQWLUE5EHK4KSKJGIUEG7D44RQASUJHCYXX6BICCGX6EA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.303087165 (81h9m22.866410168s) [noise=true , address=77.90.31.14:4689 , piece_id=Q3HDS3BCMYDTODFRMU4XFRDCQK2QZE3ZMXVLUX4M54PDEBS7RR6A, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.303250714 (81h9m22.866573717s) [noise=true , address=79.127.205.234:5601 , piece_id=V7PMXBP2VFWBSZDG4GTLBD435T6F5DT7A43L4GQLQ5BDDTJ4WY6A, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.303579249 (81h9m22.866902255s) [noise=true , address=104.36.82.12:6158 , piece_id=LJZEOJVJCJAR2BRILIJI3ZVDGYMPBBTP2G3HCTN3MHNS4KYDPASQ, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.303599776 (81h9m22.866922784s) [noise=true , address=104.36.82.9:6023 , piece_id=SISFAUU4RPF57OLGWM2BWQWE6GRUSCMORBYHWEXETNAVM2EEMO6Q, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.304005346 (81h9m22.867328346s) [noise=true , address=45.91.116.12:5242 , piece_id=6ABT5IPYGSLFFJMBYMBYRU4EYIXABUAUHG3OVILOHBIUDS3BP27A, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.304388850 (81h9m22.867711851s) [noise=true , address=64.49.13.137:4500 , piece_id=FAE2RXD57IRP5FXA2XR6THXFW27U7IWAZ4JDV5M2WRPXKUGZ4VIA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.304881529 (81h9m22.868204536s) [noise=true , address=79.127.220.101:6048 , piece_id=AN7UQUXB42T2YXEH7ZJRZUDIW47M3RWUEOCL2VEBOGH74BZF3YKA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.305249860 (81h9m22.86857287s) [noise=true , address=104.36.82.54:4165 , piece_id=4CVOABPCWC7J7SRHZB6IN2WJJ6KGWUQ66ELUIZIFA67ETLQ7JEUQ, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.305604108 (81h9m22.868927112s) [noise=true , address=104.36.82.15:6289 , piece_id=GWL3P55BIK4IJEL5YFZKI7JCA4LADYOOHUSIQUNGJEKBJOTKMP5A, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.305930423 (81h9m22.869253425s) [noise=true , address=64.49.13.139:4476 , piece_id=P7UKBNNZGI52B3I6WKTV6MVC262FYNKYYPJUI5G47UO5S4MU7PWA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.306355311 (81h9m22.869678319s) [noise=true , address=104.36.82.55:4213 , piece_id=X4IHS2XSPH52LIMNUYT7KGWMAPUH3TEMJOEEVB3JV5N45SEOH7YA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.306788243 (81h9m22.870111252s) [noise=true , address=104.36.82.22:6539 , piece_id=GHQ2MVSTDGVYLUKLV23D6YWRR4PP7UUYHCUEQ4WWSWEMTBFO3P4A, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.392174770 (81h9m22.955497773s) [upload=64958]: Next returned nil segment Jun 22 01:32:47.392195767 (81h9m22.955518775s) [upload=64958]: Waiting for error group managing segments... Jun 22 01:32:47.480767633 (81h9m23.044090623s) [noise=true , address=104.36.82.22:6539 , piece_id=GHQ2MVSTDGVYLUKLV23D6YWRR4PP7UUYHCUEQ4WWSWEMTBFO3P4A, seg_pos={0 0}, upload=64958]: Done uploading piece. err: ecclient: upload failed (node:1QL6w8DSFR6CxDdnJmDuyVZxp6JU55VKiJofBZRfQi7q31i8Jc, address:104.36.82.22:6539): protocol: storage node overloaded, request limit: 2000 Jun 22 01:32:47.585663142 (81h9m23.148986155s) [noise=true , address=79.127.220.101:6048 , piece_id=AN7UQUXB42T2YXEH7ZJRZUDIW47M3RWUEOCL2VEBOGH74BZF3YKA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:47.640263002 (81h9m23.203586013s) [noise=true , address=45.91.116.12:5242 , piece_id=6ABT5IPYGSLFFJMBYMBYRU4EYIXABUAUHG3OVILOHBIUDS3BP27A, seg_pos={0 0}, upload=64958]: Done uploading piece. err: ecclient: upload failed (node:12JsFL62jgXmza8Md8eQ9k8cyw5V9Vz2RzN4y85ZvgHZjwrq4vQ, address:45.91.116.12:5242): protocol: storage node overloaded, request limit: 2000 Jun 22 01:32:47.984127275 (81h9m23.54745028s) [noise=true , address=64.49.13.137:4500 , piece_id=FAE2RXD57IRP5FXA2XR6THXFW27U7IWAZ4JDV5M2WRPXKUGZ4VIA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:48.100255249 (81h9m23.663578241s) [noise=true , address=77.90.31.14:4689 , piece_id=Q3HDS3BCMYDTODFRMU4XFRDCQK2QZE3ZMXVLUX4M54PDEBS7RR6A, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:48.221764035 (81h9m23.785087034s) [noise=true , address=104.36.82.54:4165 , piece_id=4CVOABPCWC7J7SRHZB6IN2WJJ6KGWUQ66ELUIZIFA67ETLQ7JEUQ, seg_pos={0 0}, upload=64958]: Done uploading piece. err: ecclient: upload failed (node:12wGUMXVEubEtki4qGnjGEsbYFB1iUauZcu3oDRKrCQsJMV3bV2, address:104.36.82.54:4165): protocol: storage node overloaded, request limit: 2000 Jun 22 01:32:48.509996527 (81h9m24.073319547s) [noise=true , address=104.36.82.9:6023 , piece_id=SISFAUU4RPF57OLGWM2BWQWE6GRUSCMORBYHWEXETNAVM2EEMO6Q, seg_pos={0 0}, upload=64958]: Done uploading piece. err: ecclient: upload failed (node:1j2S9tgWpLwmyvXZSq2qKKcbaC8r7LkzfDCaMxCejAni4CST1g, address:104.36.82.9:6023): protocol: storage node overloaded, request limit: 2000 Jun 22 01:32:48.510513169 (81h9m24.073836177s) [noise=true , address=104.36.82.42:6811 , piece_id=A5BFEHZ3WGGK53IZ7MKI6N4CVXHOORF6L47NR3UFDCAX7LBHRHVQ, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:48.565245605 (81h9m24.128568606s) [noise=true , address=79.127.205.234:5601 , piece_id=V7PMXBP2VFWBSZDG4GTLBD435T6F5DT7A43L4GQLQ5BDDTJ4WY6A, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:49.979979824 (81h9m25.543302825s) [noise=true , address=104.36.82.15:6289 , piece_id=GWL3P55BIK4IJEL5YFZKI7JCA4LADYOOHUSIQUNGJEKBJOTKMP5A, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:50.463750280 (81h9m26.02707328s) [noise=true , address=104.36.82.51:4084 , piece_id=D43FA7BFE54KPCWGDGDME2EGSWM7PONOVPD3DN2XY42MJFOKVDIQ, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:51.480599795 (81h9m27.043922798s) [noise=true , address=79.127.219.38:4023 , piece_id=BWLRCXXHBMBTNCZEVKKIIS5PB35PDQXQXCLIQB7GBLEIOWWUFBFA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:52.941200625 (81h9m28.504523627s) [noise=true , address=104.36.82.43:6841 , piece_id=PIHZMSTO3CII7KGDLGSAILTDRSLMEBVOLBCRCZJULMCFISXIZWXA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:58.281803598 (81h9m33.845126612s) [noise=true , address=64.49.13.139:4476 , piece_id=P7UKBNNZGI52B3I6WKTV6MVC262FYNKYYPJUI5G47UO5S4MU7PWA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:33:00.277271877 (81h9m35.840594867s) [noise=true , address=104.36.82.55:4213 , piece_id=X4IHS2XSPH52LIMNUYT7KGWMAPUH3TEMJOEEVB3JV5N45SEOH7YA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:33:09.435339213 (81h9m44.998662206s) [noise=true , address=104.36.82.52:4089 , piece_id=JSOEKLNWA3W3LOVKI3EGGKJYJ6J3UZY6KGABKQPQPXTIOOPMKVHA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:33:31.472868445 (81h10m7.03619143s) [noise=true , address=104.36.82.53:4122 , piece_id=HZZQBIKHQWLUE5EHK4KSKJGIUEG7D44RQASUJHCYXX6BICCGX6EA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:33:47.235404653 (81h10m22.798727642s) [noise=true , address=104.36.82.12:6158 , piece_id=LJZEOJVJCJAR2BRILIJI3ZVDGYMPBBTP2G3HCTN3MHNS4KYDPASQ, seg_pos={0 0}, upload=64958]: Done uploading piece. err: ecclient: upload failed (node:1SjM6Utm2YizeUbrBUD9Z7etE1nLadW5Fa1zYP7MioPeeqhdJ7, address:104.36.82.12:6158): protocol: expected piece hash; context deadline exceeded; EOF Jun 22 01:33:47.235448297 (81h10m22.798771306s) [seg_pos={0 0}, upload=64958]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jun 22 01:33:47.265442393 (81h10m22.828765401s) [upload=64958]: Done uploading segments... Jun 22 01:33:47.265476139 (81h10m22.82879914s) [upload=64958]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration:* 20.800702614s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [........................................X.................X.] duration: 13.901428185s FAIL (2025-06-22 02:00:04 UTC) orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded Duration: 13.901463762s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [..........................X.................................] duration: 24.637509398s FAIL (2025-06-22 01:59:52 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== oad=64949]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jun 22 01:32:13.674692303 (81h8m49.238015311s) [upload=64949]: Done uploading segments... Jun 22 01:32:13.674727219 (81h8m49.238050233s) [upload=64949]: Upload finished. err: <nil> Jun 22 01:32:47.266407995 (81h9m22.829731002s) [upload=64958]: Starting upload Jun 22 01:32:47.266524767 (81h9m22.829847771s) [upload=64958]: Done starting upload Jun 22 01:32:47.266813109 (81h9m22.830136116s) [upload=64958]: Uploading segments... Jun 22 01:32:47.266969526 (81h9m22.830292526s) [upload=64958]: Got next segment. Inline: false Jun 22 01:32:47.267075475 (81h9m22.830398475s) [upload=64958]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 22 01:32:47.267081679 (81h9m22.830404667s) [upload=64958]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 22 01:32:47.301770382 (81h9m22.865093389s) [seg_pos={0 0}, upload=64958]: Begin upload segment... Jun 22 01:32:47.301823796 (81h9m22.865146799s) [seg_pos={0 0}, upload=64958]: Done begin upload segment. Jun 22 01:32:47.301855055 (81h9m22.865178087s) [noise=true , address=104.36.82.43:6841 , piece_id=PIHZMSTO3CII7KGDLGSAILTDRSLMEBVOLBCRCZJULMCFISXIZWXA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.301902600 (81h9m22.865225606s) [noise=true , address=104.36.82.42:6811 , piece_id=A5BFEHZ3WGGK53IZ7MKI6N4CVXHOORF6L47NR3UFDCAX7LBHRHVQ, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.302234027 (81h9m22.865557029s) [noise=true , address=104.36.82.51:4084 , piece_id=D43FA7BFE54KPCWGDGDME2EGSWM7PONOVPD3DN2XY42MJFOKVDIQ, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.302496354 (81h9m22.865819363s) [noise=true , address=79.127.219.38:4023 , piece_id=BWLRCXXHBMBTNCZEVKKIIS5PB35PDQXQXCLIQB7GBLEIOWWUFBFA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.302577370 (81h9m22.865900378s) [noise=true , address=104.36.82.52:4089 , piece_id=JSOEKLNWA3W3LOVKI3EGGKJYJ6J3UZY6KGABKQPQPXTIOOPMKVHA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.302917829 (81h9m22.866240835s) [noise=true , address=104.36.82.53:4122 , piece_id=HZZQBIKHQWLUE5EHK4KSKJGIUEG7D44RQASUJHCYXX6BICCGX6EA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.303087165 (81h9m22.866410168s) [noise=true , address=77.90.31.14:4689 , piece_id=Q3HDS3BCMYDTODFRMU4XFRDCQK2QZE3ZMXVLUX4M54PDEBS7RR6A, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.303250714 (81h9m22.866573717s) [noise=true , address=79.127.205.234:5601 , piece_id=V7PMXBP2VFWBSZDG4GTLBD435T6F5DT7A43L4GQLQ5BDDTJ4WY6A, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.303579249 (81h9m22.866902255s) [noise=true , address=104.36.82.12:6158 , piece_id=LJZEOJVJCJAR2BRILIJI3ZVDGYMPBBTP2G3HCTN3MHNS4KYDPASQ, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.303599776 (81h9m22.866922784s) [noise=true , address=104.36.82.9:6023 , piece_id=SISFAUU4RPF57OLGWM2BWQWE6GRUSCMORBYHWEXETNAVM2EEMO6Q, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.304005346 (81h9m22.867328346s) [noise=true , address=45.91.116.12:5242 , piece_id=6ABT5IPYGSLFFJMBYMBYRU4EYIXABUAUHG3OVILOHBIUDS3BP27A, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.304388850 (81h9m22.867711851s) [noise=true , address=64.49.13.137:4500 , piece_id=FAE2RXD57IRP5FXA2XR6THXFW27U7IWAZ4JDV5M2WRPXKUGZ4VIA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.304881529 (81h9m22.868204536s) [noise=true , address=79.127.220.101:6048 , piece_id=AN7UQUXB42T2YXEH7ZJRZUDIW47M3RWUEOCL2VEBOGH74BZF3YKA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.305249860 (81h9m22.86857287s) [noise=true , address=104.36.82.54:4165 , piece_id=4CVOABPCWC7J7SRHZB6IN2WJJ6KGWUQ66ELUIZIFA67ETLQ7JEUQ, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.305604108 (81h9m22.868927112s) [noise=true , address=104.36.82.15:6289 , piece_id=GWL3P55BIK4IJEL5YFZKI7JCA4LADYOOHUSIQUNGJEKBJOTKMP5A, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.305930423 (81h9m22.869253425s) [noise=true , address=64.49.13.139:4476 , piece_id=P7UKBNNZGI52B3I6WKTV6MVC262FYNKYYPJUI5G47UO5S4MU7PWA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.306355311 (81h9m22.869678319s) [noise=true , address=104.36.82.55:4213 , piece_id=X4IHS2XSPH52LIMNUYT7KGWMAPUH3TEMJOEEVB3JV5N45SEOH7YA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.306788243 (81h9m22.870111252s) [noise=true , address=104.36.82.22:6539 , piece_id=GHQ2MVSTDGVYLUKLV23D6YWRR4PP7UUYHCUEQ4WWSWEMTBFO3P4A, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.392174770 (81h9m22.955497773s) [upload=64958]: Next returned nil segment Jun 22 01:32:47.392195767 (81h9m22.955518775s) [upload=64958]: Waiting for error group managing segments... Jun 22 01:32:47.480767633 (81h9m23.044090623s) [noise=true , address=104.36.82.22:6539 , piece_id=GHQ2MVSTDGVYLUKLV23D6YWRR4PP7UUYHCUEQ4WWSWEMTBFO3P4A, seg_pos={0 0}, upload=64958]: Done uploading piece. err: ecclient: upload failed (node:1QL6w8DSFR6CxDdnJmDuyVZxp6JU55VKiJofBZRfQi7q31i8Jc, address:104.36.82.22:6539): protocol: storage node overloaded, request limit: 2000 Jun 22 01:32:47.585663142 (81h9m23.148986155s) [noise=true , address=79.127.220.101:6048 , piece_id=AN7UQUXB42T2YXEH7ZJRZUDIW47M3RWUEOCL2VEBOGH74BZF3YKA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:47.640263002 (81h9m23.203586013s) [noise=true , address=45.91.116.12:5242 , piece_id=6ABT5IPYGSLFFJMBYMBYRU4EYIXABUAUHG3OVILOHBIUDS3BP27A, seg_pos={0 0}, upload=64958]: Done uploading piece. err: ecclient: upload failed (node:12JsFL62jgXmza8Md8eQ9k8cyw5V9Vz2RzN4y85ZvgHZjwrq4vQ, address:45.91.116.12:5242): protocol: storage node overloaded, request limit: 2000 Jun 22 01:32:47.984127275 (81h9m23.54745028s) [noise=true , address=64.49.13.137:4500 , piece_id=FAE2RXD57IRP5FXA2XR6THXFW27U7IWAZ4JDV5M2WRPXKUGZ4VIA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:48.100255249 (81h9m23.663578241s) [noise=true , address=77.90.31.14:4689 , piece_id=Q3HDS3BCMYDTODFRMU4XFRDCQK2QZE3ZMXVLUX4M54PDEBS7RR6A, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:48.221764035 (81h9m23.785087034s) [noise=true , address=104.36.82.54:4165 , piece_id=4CVOABPCWC7J7SRHZB6IN2WJJ6KGWUQ66ELUIZIFA67ETLQ7JEUQ, seg_pos={0 0}, upload=64958]: Done uploading piece. err: ecclient: upload failed (node:12wGUMXVEubEtki4qGnjGEsbYFB1iUauZcu3oDRKrCQsJMV3bV2, address:104.36.82.54:4165): protocol: storage node overloaded, request limit: 2000 Jun 22 01:32:48.509996527 (81h9m24.073319547s) [noise=true , address=104.36.82.9:6023 , piece_id=SISFAUU4RPF57OLGWM2BWQWE6GRUSCMORBYHWEXETNAVM2EEMO6Q, seg_pos={0 0}, upload=64958]: Done uploading piece. err: ecclient: upload failed (node:1j2S9tgWpLwmyvXZSq2qKKcbaC8r7LkzfDCaMxCejAni4CST1g, address:104.36.82.9:6023): protocol: storage node overloaded, request limit: 2000 Jun 22 01:32:48.510513169 (81h9m24.073836177s) [noise=true , address=104.36.82.42:6811 , piece_id=A5BFEHZ3WGGK53IZ7MKI6N4CVXHOORF6L47NR3UFDCAX7LBHRHVQ, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:48.565245605 (81h9m24.128568606s) [noise=true , address=79.127.205.234:5601 , piece_id=V7PMXBP2VFWBSZDG4GTLBD435T6F5DT7A43L4GQLQ5BDDTJ4WY6A, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:49.979979824 (81h9m25.543302825s) [noise=true , address=104.36.82.15:6289 , piece_id=GWL3P55BIK4IJEL5YFZKI7JCA4LADYOOHUSIQUNGJEKBJOTKMP5A, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:50.463750280 (81h9m26.02707328s) [noise=true , address=104.36.82.51:4084 , piece_id=D43FA7BFE54KPCWGDGDME2EGSWM7PONOVPD3DN2XY42MJFOKVDIQ, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:51.480599795 (81h9m27.043922798s) [noise=true , address=79.127.219.38:4023 , piece_id=BWLRCXXHBMBTNCZEVKKIIS5PB35PDQXQXCLIQB7GBLEIOWWUFBFA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:52.941200625 (81h9m28.504523627s) [noise=true , address=104.36.82.43:6841 , piece_id=PIHZMSTO3CII7KGDLGSAILTDRSLMEBVOLBCRCZJULMCFISXIZWXA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:58.281803598 (81h9m33.845126612s) [noise=true , address=64.49.13.139:4476 , piece_id=P7UKBNNZGI52B3I6WKTV6MVC262FYNKYYPJUI5G47UO5S4MU7PWA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:33:00.277271877 (81h9m35.840594867s) [noise=true , address=104.36.82.55:4213 , piece_id=X4IHS2XSPH52LIMNUYT7KGWMAPUH3TEMJOEEVB3JV5N45SEOH7YA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:33:09.435339213 (81h9m44.998662206s) [noise=true , address=104.36.82.52:4089 , piece_id=JSOEKLNWA3W3LOVKI3EGGKJYJ6J3UZY6KGABKQPQPXTIOOPMKVHA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:33:31.472868445 (81h10m7.03619143s) [noise=true , address=104.36.82.53:4122 , piece_id=HZZQBIKHQWLUE5EHK4KSKJGIUEG7D44RQASUJHCYXX6BICCGX6EA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:33:47.235404653 (81h10m22.798727642s) [noise=true , address=104.36.82.12:6158 , piece_id=LJZEOJVJCJAR2BRILIJI3ZVDGYMPBBTP2G3HCTN3MHNS4KYDPASQ, seg_pos={0 0}, upload=64958]: Done uploading piece. err: ecclient: upload failed (node:1SjM6Utm2YizeUbrBUD9Z7etE1nLadW5Fa1zYP7MioPeeqhdJ7, address:104.36.82.12:6158): protocol: expected piece hash; context deadline exceeded; EOF Jun 22 01:33:47.235448297 (81h10m22.798771306s) [seg_pos={0 0}, upload=64958]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jun 22 01:33:47.265442393 (81h10m22.828765401s) [upload=64958]: Done uploading segments... Jun 22 01:33:47.265476139 (81h10m22.82879914s) [upload=64958]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration:* 24.637536645s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [.........................................X.................X] duration: 1m0.00038861s FAIL (2025-06-22 01:59:50 UTC) orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded Duration: 1m0.000448676s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [...........................X................................] duration: 11.284093686s FAIL (2025-06-22 01:58:36 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== oad=64949]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jun 22 01:32:13.674692303 (81h8m49.238015311s) [upload=64949]: Done uploading segments... Jun 22 01:32:13.674727219 (81h8m49.238050233s) [upload=64949]: Upload finished. err: <nil> Jun 22 01:32:47.266407995 (81h9m22.829731002s) [upload=64958]: Starting upload Jun 22 01:32:47.266524767 (81h9m22.829847771s) [upload=64958]: Done starting upload Jun 22 01:32:47.266813109 (81h9m22.830136116s) [upload=64958]: Uploading segments... Jun 22 01:32:47.266969526 (81h9m22.830292526s) [upload=64958]: Got next segment. Inline: false Jun 22 01:32:47.267075475 (81h9m22.830398475s) [upload=64958]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 22 01:32:47.267081679 (81h9m22.830404667s) [upload=64958]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 22 01:32:47.301770382 (81h9m22.865093389s) [seg_pos={0 0}, upload=64958]: Begin upload segment... Jun 22 01:32:47.301823796 (81h9m22.865146799s) [seg_pos={0 0}, upload=64958]: Done begin upload segment. Jun 22 01:32:47.301855055 (81h9m22.865178087s) [noise=true , address=104.36.82.43:6841 , piece_id=PIHZMSTO3CII7KGDLGSAILTDRSLMEBVOLBCRCZJULMCFISXIZWXA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.301902600 (81h9m22.865225606s) [noise=true , address=104.36.82.42:6811 , piece_id=A5BFEHZ3WGGK53IZ7MKI6N4CVXHOORF6L47NR3UFDCAX7LBHRHVQ, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.302234027 (81h9m22.865557029s) [noise=true , address=104.36.82.51:4084 , piece_id=D43FA7BFE54KPCWGDGDME2EGSWM7PONOVPD3DN2XY42MJFOKVDIQ, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.302496354 (81h9m22.865819363s) [noise=true , address=79.127.219.38:4023 , piece_id=BWLRCXXHBMBTNCZEVKKIIS5PB35PDQXQXCLIQB7GBLEIOWWUFBFA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.302577370 (81h9m22.865900378s) [noise=true , address=104.36.82.52:4089 , piece_id=JSOEKLNWA3W3LOVKI3EGGKJYJ6J3UZY6KGABKQPQPXTIOOPMKVHA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.302917829 (81h9m22.866240835s) [noise=true , address=104.36.82.53:4122 , piece_id=HZZQBIKHQWLUE5EHK4KSKJGIUEG7D44RQASUJHCYXX6BICCGX6EA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.303087165 (81h9m22.866410168s) [noise=true , address=77.90.31.14:4689 , piece_id=Q3HDS3BCMYDTODFRMU4XFRDCQK2QZE3ZMXVLUX4M54PDEBS7RR6A, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.303250714 (81h9m22.866573717s) [noise=true , address=79.127.205.234:5601 , piece_id=V7PMXBP2VFWBSZDG4GTLBD435T6F5DT7A43L4GQLQ5BDDTJ4WY6A, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.303579249 (81h9m22.866902255s) [noise=true , address=104.36.82.12:6158 , piece_id=LJZEOJVJCJAR2BRILIJI3ZVDGYMPBBTP2G3HCTN3MHNS4KYDPASQ, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.303599776 (81h9m22.866922784s) [noise=true , address=104.36.82.9:6023 , piece_id=SISFAUU4RPF57OLGWM2BWQWE6GRUSCMORBYHWEXETNAVM2EEMO6Q, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.304005346 (81h9m22.867328346s) [noise=true , address=45.91.116.12:5242 , piece_id=6ABT5IPYGSLFFJMBYMBYRU4EYIXABUAUHG3OVILOHBIUDS3BP27A, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.304388850 (81h9m22.867711851s) [noise=true , address=64.49.13.137:4500 , piece_id=FAE2RXD57IRP5FXA2XR6THXFW27U7IWAZ4JDV5M2WRPXKUGZ4VIA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.304881529 (81h9m22.868204536s) [noise=true , address=79.127.220.101:6048 , piece_id=AN7UQUXB42T2YXEH7ZJRZUDIW47M3RWUEOCL2VEBOGH74BZF3YKA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.305249860 (81h9m22.86857287s) [noise=true , address=104.36.82.54:4165 , piece_id=4CVOABPCWC7J7SRHZB6IN2WJJ6KGWUQ66ELUIZIFA67ETLQ7JEUQ, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.305604108 (81h9m22.868927112s) [noise=true , address=104.36.82.15:6289 , piece_id=GWL3P55BIK4IJEL5YFZKI7JCA4LADYOOHUSIQUNGJEKBJOTKMP5A, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.305930423 (81h9m22.869253425s) [noise=true , address=64.49.13.139:4476 , piece_id=P7UKBNNZGI52B3I6WKTV6MVC262FYNKYYPJUI5G47UO5S4MU7PWA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.306355311 (81h9m22.869678319s) [noise=true , address=104.36.82.55:4213 , piece_id=X4IHS2XSPH52LIMNUYT7KGWMAPUH3TEMJOEEVB3JV5N45SEOH7YA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.306788243 (81h9m22.870111252s) [noise=true , address=104.36.82.22:6539 , piece_id=GHQ2MVSTDGVYLUKLV23D6YWRR4PP7UUYHCUEQ4WWSWEMTBFO3P4A, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.392174770 (81h9m22.955497773s) [upload=64958]: Next returned nil segment Jun 22 01:32:47.392195767 (81h9m22.955518775s) [upload=64958]: Waiting for error group managing segments... Jun 22 01:32:47.480767633 (81h9m23.044090623s) [noise=true , address=104.36.82.22:6539 , piece_id=GHQ2MVSTDGVYLUKLV23D6YWRR4PP7UUYHCUEQ4WWSWEMTBFO3P4A, seg_pos={0 0}, upload=64958]: Done uploading piece. err: ecclient: upload failed (node:1QL6w8DSFR6CxDdnJmDuyVZxp6JU55VKiJofBZRfQi7q31i8Jc, address:104.36.82.22:6539): protocol: storage node overloaded, request limit: 2000 Jun 22 01:32:47.585663142 (81h9m23.148986155s) [noise=true , address=79.127.220.101:6048 , piece_id=AN7UQUXB42T2YXEH7ZJRZUDIW47M3RWUEOCL2VEBOGH74BZF3YKA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:47.640263002 (81h9m23.203586013s) [noise=true , address=45.91.116.12:5242 , piece_id=6ABT5IPYGSLFFJMBYMBYRU4EYIXABUAUHG3OVILOHBIUDS3BP27A, seg_pos={0 0}, upload=64958]: Done uploading piece. err: ecclient: upload failed (node:12JsFL62jgXmza8Md8eQ9k8cyw5V9Vz2RzN4y85ZvgHZjwrq4vQ, address:45.91.116.12:5242): protocol: storage node overloaded, request limit: 2000 Jun 22 01:32:47.984127275 (81h9m23.54745028s) [noise=true , address=64.49.13.137:4500 , piece_id=FAE2RXD57IRP5FXA2XR6THXFW27U7IWAZ4JDV5M2WRPXKUGZ4VIA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:48.100255249 (81h9m23.663578241s) [noise=true , address=77.90.31.14:4689 , piece_id=Q3HDS3BCMYDTODFRMU4XFRDCQK2QZE3ZMXVLUX4M54PDEBS7RR6A, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:48.221764035 (81h9m23.785087034s) [noise=true , address=104.36.82.54:4165 , piece_id=4CVOABPCWC7J7SRHZB6IN2WJJ6KGWUQ66ELUIZIFA67ETLQ7JEUQ, seg_pos={0 0}, upload=64958]: Done uploading piece. err: ecclient: upload failed (node:12wGUMXVEubEtki4qGnjGEsbYFB1iUauZcu3oDRKrCQsJMV3bV2, address:104.36.82.54:4165): protocol: storage node overloaded, request limit: 2000 Jun 22 01:32:48.509996527 (81h9m24.073319547s) [noise=true , address=104.36.82.9:6023 , piece_id=SISFAUU4RPF57OLGWM2BWQWE6GRUSCMORBYHWEXETNAVM2EEMO6Q, seg_pos={0 0}, upload=64958]: Done uploading piece. err: ecclient: upload failed (node:1j2S9tgWpLwmyvXZSq2qKKcbaC8r7LkzfDCaMxCejAni4CST1g, address:104.36.82.9:6023): protocol: storage node overloaded, request limit: 2000 Jun 22 01:32:48.510513169 (81h9m24.073836177s) [noise=true , address=104.36.82.42:6811 , piece_id=A5BFEHZ3WGGK53IZ7MKI6N4CVXHOORF6L47NR3UFDCAX7LBHRHVQ, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:48.565245605 (81h9m24.128568606s) [noise=true , address=79.127.205.234:5601 , piece_id=V7PMXBP2VFWBSZDG4GTLBD435T6F5DT7A43L4GQLQ5BDDTJ4WY6A, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:49.979979824 (81h9m25.543302825s) [noise=true , address=104.36.82.15:6289 , piece_id=GWL3P55BIK4IJEL5YFZKI7JCA4LADYOOHUSIQUNGJEKBJOTKMP5A, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:50.463750280 (81h9m26.02707328s) [noise=true , address=104.36.82.51:4084 , piece_id=D43FA7BFE54KPCWGDGDME2EGSWM7PONOVPD3DN2XY42MJFOKVDIQ, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:51.480599795 (81h9m27.043922798s) [noise=true , address=79.127.219.38:4023 , piece_id=BWLRCXXHBMBTNCZEVKKIIS5PB35PDQXQXCLIQB7GBLEIOWWUFBFA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:52.941200625 (81h9m28.504523627s) [noise=true , address=104.36.82.43:6841 , piece_id=PIHZMSTO3CII7KGDLGSAILTDRSLMEBVOLBCRCZJULMCFISXIZWXA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:58.281803598 (81h9m33.845126612s) [noise=true , address=64.49.13.139:4476 , piece_id=P7UKBNNZGI52B3I6WKTV6MVC262FYNKYYPJUI5G47UO5S4MU7PWA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:33:00.277271877 (81h9m35.840594867s) [noise=true , address=104.36.82.55:4213 , piece_id=X4IHS2XSPH52LIMNUYT7KGWMAPUH3TEMJOEEVB3JV5N45SEOH7YA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:33:09.435339213 (81h9m44.998662206s) [noise=true , address=104.36.82.52:4089 , piece_id=JSOEKLNWA3W3LOVKI3EGGKJYJ6J3UZY6KGABKQPQPXTIOOPMKVHA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:33:31.472868445 (81h10m7.03619143s) [noise=true , address=104.36.82.53:4122 , piece_id=HZZQBIKHQWLUE5EHK4KSKJGIUEG7D44RQASUJHCYXX6BICCGX6EA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:33:47.235404653 (81h10m22.798727642s) [noise=true , address=104.36.82.12:6158 , piece_id=LJZEOJVJCJAR2BRILIJI3ZVDGYMPBBTP2G3HCTN3MHNS4KYDPASQ, seg_pos={0 0}, upload=64958]: Done uploading piece. err: ecclient: upload failed (node:1SjM6Utm2YizeUbrBUD9Z7etE1nLadW5Fa1zYP7MioPeeqhdJ7, address:104.36.82.12:6158): protocol: expected piece hash; context deadline exceeded; EOF Jun 22 01:33:47.235448297 (81h10m22.798771306s) [seg_pos={0 0}, upload=64958]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jun 22 01:33:47.265442393 (81h10m22.828765401s) [upload=64958]: Done uploading segments... Jun 22 01:33:47.265476139 (81h10m22.82879914s) [upload=64958]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration:* 11.284117549s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [............................X...............................] duration: 35.447183479s FAIL (2025-06-22 01:58:15 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== oad=64949]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jun 22 01:32:13.674692303 (81h8m49.238015311s) [upload=64949]: Done uploading segments... Jun 22 01:32:13.674727219 (81h8m49.238050233s) [upload=64949]: Upload finished. err: <nil> Jun 22 01:32:47.266407995 (81h9m22.829731002s) [upload=64958]: Starting upload Jun 22 01:32:47.266524767 (81h9m22.829847771s) [upload=64958]: Done starting upload Jun 22 01:32:47.266813109 (81h9m22.830136116s) [upload=64958]: Uploading segments... Jun 22 01:32:47.266969526 (81h9m22.830292526s) [upload=64958]: Got next segment. Inline: false Jun 22 01:32:47.267075475 (81h9m22.830398475s) [upload=64958]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 22 01:32:47.267081679 (81h9m22.830404667s) [upload=64958]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 22 01:32:47.301770382 (81h9m22.865093389s) [seg_pos={0 0}, upload=64958]: Begin upload segment... Jun 22 01:32:47.301823796 (81h9m22.865146799s) [seg_pos={0 0}, upload=64958]: Done begin upload segment. Jun 22 01:32:47.301855055 (81h9m22.865178087s) [noise=true , address=104.36.82.43:6841 , piece_id=PIHZMSTO3CII7KGDLGSAILTDRSLMEBVOLBCRCZJULMCFISXIZWXA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.301902600 (81h9m22.865225606s) [noise=true , address=104.36.82.42:6811 , piece_id=A5BFEHZ3WGGK53IZ7MKI6N4CVXHOORF6L47NR3UFDCAX7LBHRHVQ, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.302234027 (81h9m22.865557029s) [noise=true , address=104.36.82.51:4084 , piece_id=D43FA7BFE54KPCWGDGDME2EGSWM7PONOVPD3DN2XY42MJFOKVDIQ, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.302496354 (81h9m22.865819363s) [noise=true , address=79.127.219.38:4023 , piece_id=BWLRCXXHBMBTNCZEVKKIIS5PB35PDQXQXCLIQB7GBLEIOWWUFBFA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.302577370 (81h9m22.865900378s) [noise=true , address=104.36.82.52:4089 , piece_id=JSOEKLNWA3W3LOVKI3EGGKJYJ6J3UZY6KGABKQPQPXTIOOPMKVHA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.302917829 (81h9m22.866240835s) [noise=true , address=104.36.82.53:4122 , piece_id=HZZQBIKHQWLUE5EHK4KSKJGIUEG7D44RQASUJHCYXX6BICCGX6EA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.303087165 (81h9m22.866410168s) [noise=true , address=77.90.31.14:4689 , piece_id=Q3HDS3BCMYDTODFRMU4XFRDCQK2QZE3ZMXVLUX4M54PDEBS7RR6A, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.303250714 (81h9m22.866573717s) [noise=true , address=79.127.205.234:5601 , piece_id=V7PMXBP2VFWBSZDG4GTLBD435T6F5DT7A43L4GQLQ5BDDTJ4WY6A, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.303579249 (81h9m22.866902255s) [noise=true , address=104.36.82.12:6158 , piece_id=LJZEOJVJCJAR2BRILIJI3ZVDGYMPBBTP2G3HCTN3MHNS4KYDPASQ, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.303599776 (81h9m22.866922784s) [noise=true , address=104.36.82.9:6023 , piece_id=SISFAUU4RPF57OLGWM2BWQWE6GRUSCMORBYHWEXETNAVM2EEMO6Q, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.304005346 (81h9m22.867328346s) [noise=true , address=45.91.116.12:5242 , piece_id=6ABT5IPYGSLFFJMBYMBYRU4EYIXABUAUHG3OVILOHBIUDS3BP27A, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.304388850 (81h9m22.867711851s) [noise=true , address=64.49.13.137:4500 , piece_id=FAE2RXD57IRP5FXA2XR6THXFW27U7IWAZ4JDV5M2WRPXKUGZ4VIA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.304881529 (81h9m22.868204536s) [noise=true , address=79.127.220.101:6048 , piece_id=AN7UQUXB42T2YXEH7ZJRZUDIW47M3RWUEOCL2VEBOGH74BZF3YKA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.305249860 (81h9m22.86857287s) [noise=true , address=104.36.82.54:4165 , piece_id=4CVOABPCWC7J7SRHZB6IN2WJJ6KGWUQ66ELUIZIFA67ETLQ7JEUQ, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.305604108 (81h9m22.868927112s) [noise=true , address=104.36.82.15:6289 , piece_id=GWL3P55BIK4IJEL5YFZKI7JCA4LADYOOHUSIQUNGJEKBJOTKMP5A, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.305930423 (81h9m22.869253425s) [noise=true , address=64.49.13.139:4476 , piece_id=P7UKBNNZGI52B3I6WKTV6MVC262FYNKYYPJUI5G47UO5S4MU7PWA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.306355311 (81h9m22.869678319s) [noise=true , address=104.36.82.55:4213 , piece_id=X4IHS2XSPH52LIMNUYT7KGWMAPUH3TEMJOEEVB3JV5N45SEOH7YA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.306788243 (81h9m22.870111252s) [noise=true , address=104.36.82.22:6539 , piece_id=GHQ2MVSTDGVYLUKLV23D6YWRR4PP7UUYHCUEQ4WWSWEMTBFO3P4A, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.392174770 (81h9m22.955497773s) [upload=64958]: Next returned nil segment Jun 22 01:32:47.392195767 (81h9m22.955518775s) [upload=64958]: Waiting for error group managing segments... Jun 22 01:32:47.480767633 (81h9m23.044090623s) [noise=true , address=104.36.82.22:6539 , piece_id=GHQ2MVSTDGVYLUKLV23D6YWRR4PP7UUYHCUEQ4WWSWEMTBFO3P4A, seg_pos={0 0}, upload=64958]: Done uploading piece. err: ecclient: upload failed (node:1QL6w8DSFR6CxDdnJmDuyVZxp6JU55VKiJofBZRfQi7q31i8Jc, address:104.36.82.22:6539): protocol: storage node overloaded, request limit: 2000 Jun 22 01:32:47.585663142 (81h9m23.148986155s) [noise=true , address=79.127.220.101:6048 , piece_id=AN7UQUXB42T2YXEH7ZJRZUDIW47M3RWUEOCL2VEBOGH74BZF3YKA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:47.640263002 (81h9m23.203586013s) [noise=true , address=45.91.116.12:5242 , piece_id=6ABT5IPYGSLFFJMBYMBYRU4EYIXABUAUHG3OVILOHBIUDS3BP27A, seg_pos={0 0}, upload=64958]: Done uploading piece. err: ecclient: upload failed (node:12JsFL62jgXmza8Md8eQ9k8cyw5V9Vz2RzN4y85ZvgHZjwrq4vQ, address:45.91.116.12:5242): protocol: storage node overloaded, request limit: 2000 Jun 22 01:32:47.984127275 (81h9m23.54745028s) [noise=true , address=64.49.13.137:4500 , piece_id=FAE2RXD57IRP5FXA2XR6THXFW27U7IWAZ4JDV5M2WRPXKUGZ4VIA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:48.100255249 (81h9m23.663578241s) [noise=true , address=77.90.31.14:4689 , piece_id=Q3HDS3BCMYDTODFRMU4XFRDCQK2QZE3ZMXVLUX4M54PDEBS7RR6A, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:48.221764035 (81h9m23.785087034s) [noise=true , address=104.36.82.54:4165 , piece_id=4CVOABPCWC7J7SRHZB6IN2WJJ6KGWUQ66ELUIZIFA67ETLQ7JEUQ, seg_pos={0 0}, upload=64958]: Done uploading piece. err: ecclient: upload failed (node:12wGUMXVEubEtki4qGnjGEsbYFB1iUauZcu3oDRKrCQsJMV3bV2, address:104.36.82.54:4165): protocol: storage node overloaded, request limit: 2000 Jun 22 01:32:48.509996527 (81h9m24.073319547s) [noise=true , address=104.36.82.9:6023 , piece_id=SISFAUU4RPF57OLGWM2BWQWE6GRUSCMORBYHWEXETNAVM2EEMO6Q, seg_pos={0 0}, upload=64958]: Done uploading piece. err: ecclient: upload failed (node:1j2S9tgWpLwmyvXZSq2qKKcbaC8r7LkzfDCaMxCejAni4CST1g, address:104.36.82.9:6023): protocol: storage node overloaded, request limit: 2000 Jun 22 01:32:48.510513169 (81h9m24.073836177s) [noise=true , address=104.36.82.42:6811 , piece_id=A5BFEHZ3WGGK53IZ7MKI6N4CVXHOORF6L47NR3UFDCAX7LBHRHVQ, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:48.565245605 (81h9m24.128568606s) [noise=true , address=79.127.205.234:5601 , piece_id=V7PMXBP2VFWBSZDG4GTLBD435T6F5DT7A43L4GQLQ5BDDTJ4WY6A, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:49.979979824 (81h9m25.543302825s) [noise=true , address=104.36.82.15:6289 , piece_id=GWL3P55BIK4IJEL5YFZKI7JCA4LADYOOHUSIQUNGJEKBJOTKMP5A, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:50.463750280 (81h9m26.02707328s) [noise=true , address=104.36.82.51:4084 , piece_id=D43FA7BFE54KPCWGDGDME2EGSWM7PONOVPD3DN2XY42MJFOKVDIQ, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:51.480599795 (81h9m27.043922798s) [noise=true , address=79.127.219.38:4023 , piece_id=BWLRCXXHBMBTNCZEVKKIIS5PB35PDQXQXCLIQB7GBLEIOWWUFBFA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:52.941200625 (81h9m28.504523627s) [noise=true , address=104.36.82.43:6841 , piece_id=PIHZMSTO3CII7KGDLGSAILTDRSLMEBVOLBCRCZJULMCFISXIZWXA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:58.281803598 (81h9m33.845126612s) [noise=true , address=64.49.13.139:4476 , piece_id=P7UKBNNZGI52B3I6WKTV6MVC262FYNKYYPJUI5G47UO5S4MU7PWA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:33:00.277271877 (81h9m35.840594867s) [noise=true , address=104.36.82.55:4213 , piece_id=X4IHS2XSPH52LIMNUYT7KGWMAPUH3TEMJOEEVB3JV5N45SEOH7YA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:33:09.435339213 (81h9m44.998662206s) [noise=true , address=104.36.82.52:4089 , piece_id=JSOEKLNWA3W3LOVKI3EGGKJYJ6J3UZY6KGABKQPQPXTIOOPMKVHA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:33:31.472868445 (81h10m7.03619143s) [noise=true , address=104.36.82.53:4122 , piece_id=HZZQBIKHQWLUE5EHK4KSKJGIUEG7D44RQASUJHCYXX6BICCGX6EA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:33:47.235404653 (81h10m22.798727642s) [noise=true , address=104.36.82.12:6158 , piece_id=LJZEOJVJCJAR2BRILIJI3ZVDGYMPBBTP2G3HCTN3MHNS4KYDPASQ, seg_pos={0 0}, upload=64958]: Done uploading piece. err: ecclient: upload failed (node:1SjM6Utm2YizeUbrBUD9Z7etE1nLadW5Fa1zYP7MioPeeqhdJ7, address:104.36.82.12:6158): protocol: expected piece hash; context deadline exceeded; EOF Jun 22 01:33:47.235448297 (81h10m22.798771306s) [seg_pos={0 0}, upload=64958]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jun 22 01:33:47.265442393 (81h10m22.828765401s) [upload=64958]: Done uploading segments... Jun 22 01:33:47.265476139 (81h10m22.82879914s) [upload=64958]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration:* 35.447211893s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [..........................................X.................] duration: 15.210048459s FAIL (2025-06-22 01:58:07 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 15.21007496s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.............................X..............................] duration: 51.322929225s FAIL (2025-06-22 01:57:20 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== oad=64949]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jun 22 01:32:13.674692303 (81h8m49.238015311s) [upload=64949]: Done uploading segments... Jun 22 01:32:13.674727219 (81h8m49.238050233s) [upload=64949]: Upload finished. err: <nil> Jun 22 01:32:47.266407995 (81h9m22.829731002s) [upload=64958]: Starting upload Jun 22 01:32:47.266524767 (81h9m22.829847771s) [upload=64958]: Done starting upload Jun 22 01:32:47.266813109 (81h9m22.830136116s) [upload=64958]: Uploading segments... Jun 22 01:32:47.266969526 (81h9m22.830292526s) [upload=64958]: Got next segment. Inline: false Jun 22 01:32:47.267075475 (81h9m22.830398475s) [upload=64958]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 22 01:32:47.267081679 (81h9m22.830404667s) [upload=64958]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 22 01:32:47.301770382 (81h9m22.865093389s) [seg_pos={0 0}, upload=64958]: Begin upload segment... Jun 22 01:32:47.301823796 (81h9m22.865146799s) [seg_pos={0 0}, upload=64958]: Done begin upload segment. Jun 22 01:32:47.301855055 (81h9m22.865178087s) [noise=true , address=104.36.82.43:6841 , piece_id=PIHZMSTO3CII7KGDLGSAILTDRSLMEBVOLBCRCZJULMCFISXIZWXA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.301902600 (81h9m22.865225606s) [noise=true , address=104.36.82.42:6811 , piece_id=A5BFEHZ3WGGK53IZ7MKI6N4CVXHOORF6L47NR3UFDCAX7LBHRHVQ, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.302234027 (81h9m22.865557029s) [noise=true , address=104.36.82.51:4084 , piece_id=D43FA7BFE54KPCWGDGDME2EGSWM7PONOVPD3DN2XY42MJFOKVDIQ, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.302496354 (81h9m22.865819363s) [noise=true , address=79.127.219.38:4023 , piece_id=BWLRCXXHBMBTNCZEVKKIIS5PB35PDQXQXCLIQB7GBLEIOWWUFBFA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.302577370 (81h9m22.865900378s) [noise=true , address=104.36.82.52:4089 , piece_id=JSOEKLNWA3W3LOVKI3EGGKJYJ6J3UZY6KGABKQPQPXTIOOPMKVHA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.302917829 (81h9m22.866240835s) [noise=true , address=104.36.82.53:4122 , piece_id=HZZQBIKHQWLUE5EHK4KSKJGIUEG7D44RQASUJHCYXX6BICCGX6EA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.303087165 (81h9m22.866410168s) [noise=true , address=77.90.31.14:4689 , piece_id=Q3HDS3BCMYDTODFRMU4XFRDCQK2QZE3ZMXVLUX4M54PDEBS7RR6A, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.303250714 (81h9m22.866573717s) [noise=true , address=79.127.205.234:5601 , piece_id=V7PMXBP2VFWBSZDG4GTLBD435T6F5DT7A43L4GQLQ5BDDTJ4WY6A, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.303579249 (81h9m22.866902255s) [noise=true , address=104.36.82.12:6158 , piece_id=LJZEOJVJCJAR2BRILIJI3ZVDGYMPBBTP2G3HCTN3MHNS4KYDPASQ, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.303599776 (81h9m22.866922784s) [noise=true , address=104.36.82.9:6023 , piece_id=SISFAUU4RPF57OLGWM2BWQWE6GRUSCMORBYHWEXETNAVM2EEMO6Q, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.304005346 (81h9m22.867328346s) [noise=true , address=45.91.116.12:5242 , piece_id=6ABT5IPYGSLFFJMBYMBYRU4EYIXABUAUHG3OVILOHBIUDS3BP27A, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.304388850 (81h9m22.867711851s) [noise=true , address=64.49.13.137:4500 , piece_id=FAE2RXD57IRP5FXA2XR6THXFW27U7IWAZ4JDV5M2WRPXKUGZ4VIA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.304881529 (81h9m22.868204536s) [noise=true , address=79.127.220.101:6048 , piece_id=AN7UQUXB42T2YXEH7ZJRZUDIW47M3RWUEOCL2VEBOGH74BZF3YKA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.305249860 (81h9m22.86857287s) [noise=true , address=104.36.82.54:4165 , piece_id=4CVOABPCWC7J7SRHZB6IN2WJJ6KGWUQ66ELUIZIFA67ETLQ7JEUQ, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.305604108 (81h9m22.868927112s) [noise=true , address=104.36.82.15:6289 , piece_id=GWL3P55BIK4IJEL5YFZKI7JCA4LADYOOHUSIQUNGJEKBJOTKMP5A, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.305930423 (81h9m22.869253425s) [noise=true , address=64.49.13.139:4476 , piece_id=P7UKBNNZGI52B3I6WKTV6MVC262FYNKYYPJUI5G47UO5S4MU7PWA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.306355311 (81h9m22.869678319s) [noise=true , address=104.36.82.55:4213 , piece_id=X4IHS2XSPH52LIMNUYT7KGWMAPUH3TEMJOEEVB3JV5N45SEOH7YA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.306788243 (81h9m22.870111252s) [noise=true , address=104.36.82.22:6539 , piece_id=GHQ2MVSTDGVYLUKLV23D6YWRR4PP7UUYHCUEQ4WWSWEMTBFO3P4A, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.392174770 (81h9m22.955497773s) [upload=64958]: Next returned nil segment Jun 22 01:32:47.392195767 (81h9m22.955518775s) [upload=64958]: Waiting for error group managing segments... Jun 22 01:32:47.480767633 (81h9m23.044090623s) [noise=true , address=104.36.82.22:6539 , piece_id=GHQ2MVSTDGVYLUKLV23D6YWRR4PP7UUYHCUEQ4WWSWEMTBFO3P4A, seg_pos={0 0}, upload=64958]: Done uploading piece. err: ecclient: upload failed (node:1QL6w8DSFR6CxDdnJmDuyVZxp6JU55VKiJofBZRfQi7q31i8Jc, address:104.36.82.22:6539): protocol: storage node overloaded, request limit: 2000 Jun 22 01:32:47.585663142 (81h9m23.148986155s) [noise=true , address=79.127.220.101:6048 , piece_id=AN7UQUXB42T2YXEH7ZJRZUDIW47M3RWUEOCL2VEBOGH74BZF3YKA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:47.640263002 (81h9m23.203586013s) [noise=true , address=45.91.116.12:5242 , piece_id=6ABT5IPYGSLFFJMBYMBYRU4EYIXABUAUHG3OVILOHBIUDS3BP27A, seg_pos={0 0}, upload=64958]: Done uploading piece. err: ecclient: upload failed (node:12JsFL62jgXmza8Md8eQ9k8cyw5V9Vz2RzN4y85ZvgHZjwrq4vQ, address:45.91.116.12:5242): protocol: storage node overloaded, request limit: 2000 Jun 22 01:32:47.984127275 (81h9m23.54745028s) [noise=true , address=64.49.13.137:4500 , piece_id=FAE2RXD57IRP5FXA2XR6THXFW27U7IWAZ4JDV5M2WRPXKUGZ4VIA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:48.100255249 (81h9m23.663578241s) [noise=true , address=77.90.31.14:4689 , piece_id=Q3HDS3BCMYDTODFRMU4XFRDCQK2QZE3ZMXVLUX4M54PDEBS7RR6A, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:48.221764035 (81h9m23.785087034s) [noise=true , address=104.36.82.54:4165 , piece_id=4CVOABPCWC7J7SRHZB6IN2WJJ6KGWUQ66ELUIZIFA67ETLQ7JEUQ, seg_pos={0 0}, upload=64958]: Done uploading piece. err: ecclient: upload failed (node:12wGUMXVEubEtki4qGnjGEsbYFB1iUauZcu3oDRKrCQsJMV3bV2, address:104.36.82.54:4165): protocol: storage node overloaded, request limit: 2000 Jun 22 01:32:48.509996527 (81h9m24.073319547s) [noise=true , address=104.36.82.9:6023 , piece_id=SISFAUU4RPF57OLGWM2BWQWE6GRUSCMORBYHWEXETNAVM2EEMO6Q, seg_pos={0 0}, upload=64958]: Done uploading piece. err: ecclient: upload failed (node:1j2S9tgWpLwmyvXZSq2qKKcbaC8r7LkzfDCaMxCejAni4CST1g, address:104.36.82.9:6023): protocol: storage node overloaded, request limit: 2000 Jun 22 01:32:48.510513169 (81h9m24.073836177s) [noise=true , address=104.36.82.42:6811 , piece_id=A5BFEHZ3WGGK53IZ7MKI6N4CVXHOORF6L47NR3UFDCAX7LBHRHVQ, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:48.565245605 (81h9m24.128568606s) [noise=true , address=79.127.205.234:5601 , piece_id=V7PMXBP2VFWBSZDG4GTLBD435T6F5DT7A43L4GQLQ5BDDTJ4WY6A, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:49.979979824 (81h9m25.543302825s) [noise=true , address=104.36.82.15:6289 , piece_id=GWL3P55BIK4IJEL5YFZKI7JCA4LADYOOHUSIQUNGJEKBJOTKMP5A, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:50.463750280 (81h9m26.02707328s) [noise=true , address=104.36.82.51:4084 , piece_id=D43FA7BFE54KPCWGDGDME2EGSWM7PONOVPD3DN2XY42MJFOKVDIQ, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:51.480599795 (81h9m27.043922798s) [noise=true , address=79.127.219.38:4023 , piece_id=BWLRCXXHBMBTNCZEVKKIIS5PB35PDQXQXCLIQB7GBLEIOWWUFBFA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:52.941200625 (81h9m28.504523627s) [noise=true , address=104.36.82.43:6841 , piece_id=PIHZMSTO3CII7KGDLGSAILTDRSLMEBVOLBCRCZJULMCFISXIZWXA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:58.281803598 (81h9m33.845126612s) [noise=true , address=64.49.13.139:4476 , piece_id=P7UKBNNZGI52B3I6WKTV6MVC262FYNKYYPJUI5G47UO5S4MU7PWA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:33:00.277271877 (81h9m35.840594867s) [noise=true , address=104.36.82.55:4213 , piece_id=X4IHS2XSPH52LIMNUYT7KGWMAPUH3TEMJOEEVB3JV5N45SEOH7YA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:33:09.435339213 (81h9m44.998662206s) [noise=true , address=104.36.82.52:4089 , piece_id=JSOEKLNWA3W3LOVKI3EGGKJYJ6J3UZY6KGABKQPQPXTIOOPMKVHA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:33:31.472868445 (81h10m7.03619143s) [noise=true , address=104.36.82.53:4122 , piece_id=HZZQBIKHQWLUE5EHK4KSKJGIUEG7D44RQASUJHCYXX6BICCGX6EA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:33:47.235404653 (81h10m22.798727642s) [noise=true , address=104.36.82.12:6158 , piece_id=LJZEOJVJCJAR2BRILIJI3ZVDGYMPBBTP2G3HCTN3MHNS4KYDPASQ, seg_pos={0 0}, upload=64958]: Done uploading piece. err: ecclient: upload failed (node:1SjM6Utm2YizeUbrBUD9Z7etE1nLadW5Fa1zYP7MioPeeqhdJ7, address:104.36.82.12:6158): protocol: expected piece hash; context deadline exceeded; EOF Jun 22 01:33:47.235448297 (81h10m22.798771306s) [seg_pos={0 0}, upload=64958]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jun 22 01:33:47.265442393 (81h10m22.828765401s) [upload=64958]: Done uploading segments... Jun 22 01:33:47.265476139 (81h10m22.82879914s) [upload=64958]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration:* 51.322949851s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [...........................................X................] duration: 17.967422921s FAIL (2025-06-22 01:57:04 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 17.967441304s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [............................................X...............] duration: 34.462274542s FAIL (2025-06-22 01:56:46 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 34.462356649s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [..............................X.............................] duration: 21.471511957s FAIL (2025-06-22 01:55:50 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== oad=64949]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jun 22 01:32:13.674692303 (81h8m49.238015311s) [upload=64949]: Done uploading segments... Jun 22 01:32:13.674727219 (81h8m49.238050233s) [upload=64949]: Upload finished. err: <nil> Jun 22 01:32:47.266407995 (81h9m22.829731002s) [upload=64958]: Starting upload Jun 22 01:32:47.266524767 (81h9m22.829847771s) [upload=64958]: Done starting upload Jun 22 01:32:47.266813109 (81h9m22.830136116s) [upload=64958]: Uploading segments... Jun 22 01:32:47.266969526 (81h9m22.830292526s) [upload=64958]: Got next segment. Inline: false Jun 22 01:32:47.267075475 (81h9m22.830398475s) [upload=64958]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 22 01:32:47.267081679 (81h9m22.830404667s) [upload=64958]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 22 01:32:47.301770382 (81h9m22.865093389s) [seg_pos={0 0}, upload=64958]: Begin upload segment... Jun 22 01:32:47.301823796 (81h9m22.865146799s) [seg_pos={0 0}, upload=64958]: Done begin upload segment. Jun 22 01:32:47.301855055 (81h9m22.865178087s) [noise=true , address=104.36.82.43:6841 , piece_id=PIHZMSTO3CII7KGDLGSAILTDRSLMEBVOLBCRCZJULMCFISXIZWXA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.301902600 (81h9m22.865225606s) [noise=true , address=104.36.82.42:6811 , piece_id=A5BFEHZ3WGGK53IZ7MKI6N4CVXHOORF6L47NR3UFDCAX7LBHRHVQ, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.302234027 (81h9m22.865557029s) [noise=true , address=104.36.82.51:4084 , piece_id=D43FA7BFE54KPCWGDGDME2EGSWM7PONOVPD3DN2XY42MJFOKVDIQ, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.302496354 (81h9m22.865819363s) [noise=true , address=79.127.219.38:4023 , piece_id=BWLRCXXHBMBTNCZEVKKIIS5PB35PDQXQXCLIQB7GBLEIOWWUFBFA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.302577370 (81h9m22.865900378s) [noise=true , address=104.36.82.52:4089 , piece_id=JSOEKLNWA3W3LOVKI3EGGKJYJ6J3UZY6KGABKQPQPXTIOOPMKVHA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.302917829 (81h9m22.866240835s) [noise=true , address=104.36.82.53:4122 , piece_id=HZZQBIKHQWLUE5EHK4KSKJGIUEG7D44RQASUJHCYXX6BICCGX6EA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.303087165 (81h9m22.866410168s) [noise=true , address=77.90.31.14:4689 , piece_id=Q3HDS3BCMYDTODFRMU4XFRDCQK2QZE3ZMXVLUX4M54PDEBS7RR6A, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.303250714 (81h9m22.866573717s) [noise=true , address=79.127.205.234:5601 , piece_id=V7PMXBP2VFWBSZDG4GTLBD435T6F5DT7A43L4GQLQ5BDDTJ4WY6A, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.303579249 (81h9m22.866902255s) [noise=true , address=104.36.82.12:6158 , piece_id=LJZEOJVJCJAR2BRILIJI3ZVDGYMPBBTP2G3HCTN3MHNS4KYDPASQ, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.303599776 (81h9m22.866922784s) [noise=true , address=104.36.82.9:6023 , piece_id=SISFAUU4RPF57OLGWM2BWQWE6GRUSCMORBYHWEXETNAVM2EEMO6Q, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.304005346 (81h9m22.867328346s) [noise=true , address=45.91.116.12:5242 , piece_id=6ABT5IPYGSLFFJMBYMBYRU4EYIXABUAUHG3OVILOHBIUDS3BP27A, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.304388850 (81h9m22.867711851s) [noise=true , address=64.49.13.137:4500 , piece_id=FAE2RXD57IRP5FXA2XR6THXFW27U7IWAZ4JDV5M2WRPXKUGZ4VIA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.304881529 (81h9m22.868204536s) [noise=true , address=79.127.220.101:6048 , piece_id=AN7UQUXB42T2YXEH7ZJRZUDIW47M3RWUEOCL2VEBOGH74BZF3YKA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.305249860 (81h9m22.86857287s) [noise=true , address=104.36.82.54:4165 , piece_id=4CVOABPCWC7J7SRHZB6IN2WJJ6KGWUQ66ELUIZIFA67ETLQ7JEUQ, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.305604108 (81h9m22.868927112s) [noise=true , address=104.36.82.15:6289 , piece_id=GWL3P55BIK4IJEL5YFZKI7JCA4LADYOOHUSIQUNGJEKBJOTKMP5A, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.305930423 (81h9m22.869253425s) [noise=true , address=64.49.13.139:4476 , piece_id=P7UKBNNZGI52B3I6WKTV6MVC262FYNKYYPJUI5G47UO5S4MU7PWA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.306355311 (81h9m22.869678319s) [noise=true , address=104.36.82.55:4213 , piece_id=X4IHS2XSPH52LIMNUYT7KGWMAPUH3TEMJOEEVB3JV5N45SEOH7YA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.306788243 (81h9m22.870111252s) [noise=true , address=104.36.82.22:6539 , piece_id=GHQ2MVSTDGVYLUKLV23D6YWRR4PP7UUYHCUEQ4WWSWEMTBFO3P4A, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.392174770 (81h9m22.955497773s) [upload=64958]: Next returned nil segment Jun 22 01:32:47.392195767 (81h9m22.955518775s) [upload=64958]: Waiting for error group managing segments... Jun 22 01:32:47.480767633 (81h9m23.044090623s) [noise=true , address=104.36.82.22:6539 , piece_id=GHQ2MVSTDGVYLUKLV23D6YWRR4PP7UUYHCUEQ4WWSWEMTBFO3P4A, seg_pos={0 0}, upload=64958]: Done uploading piece. err: ecclient: upload failed (node:1QL6w8DSFR6CxDdnJmDuyVZxp6JU55VKiJofBZRfQi7q31i8Jc, address:104.36.82.22:6539): protocol: storage node overloaded, request limit: 2000 Jun 22 01:32:47.585663142 (81h9m23.148986155s) [noise=true , address=79.127.220.101:6048 , piece_id=AN7UQUXB42T2YXEH7ZJRZUDIW47M3RWUEOCL2VEBOGH74BZF3YKA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:47.640263002 (81h9m23.203586013s) [noise=true , address=45.91.116.12:5242 , piece_id=6ABT5IPYGSLFFJMBYMBYRU4EYIXABUAUHG3OVILOHBIUDS3BP27A, seg_pos={0 0}, upload=64958]: Done uploading piece. err: ecclient: upload failed (node:12JsFL62jgXmza8Md8eQ9k8cyw5V9Vz2RzN4y85ZvgHZjwrq4vQ, address:45.91.116.12:5242): protocol: storage node overloaded, request limit: 2000 Jun 22 01:32:47.984127275 (81h9m23.54745028s) [noise=true , address=64.49.13.137:4500 , piece_id=FAE2RXD57IRP5FXA2XR6THXFW27U7IWAZ4JDV5M2WRPXKUGZ4VIA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:48.100255249 (81h9m23.663578241s) [noise=true , address=77.90.31.14:4689 , piece_id=Q3HDS3BCMYDTODFRMU4XFRDCQK2QZE3ZMXVLUX4M54PDEBS7RR6A, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:48.221764035 (81h9m23.785087034s) [noise=true , address=104.36.82.54:4165 , piece_id=4CVOABPCWC7J7SRHZB6IN2WJJ6KGWUQ66ELUIZIFA67ETLQ7JEUQ, seg_pos={0 0}, upload=64958]: Done uploading piece. err: ecclient: upload failed (node:12wGUMXVEubEtki4qGnjGEsbYFB1iUauZcu3oDRKrCQsJMV3bV2, address:104.36.82.54:4165): protocol: storage node overloaded, request limit: 2000 Jun 22 01:32:48.509996527 (81h9m24.073319547s) [noise=true , address=104.36.82.9:6023 , piece_id=SISFAUU4RPF57OLGWM2BWQWE6GRUSCMORBYHWEXETNAVM2EEMO6Q, seg_pos={0 0}, upload=64958]: Done uploading piece. err: ecclient: upload failed (node:1j2S9tgWpLwmyvXZSq2qKKcbaC8r7LkzfDCaMxCejAni4CST1g, address:104.36.82.9:6023): protocol: storage node overloaded, request limit: 2000 Jun 22 01:32:48.510513169 (81h9m24.073836177s) [noise=true , address=104.36.82.42:6811 , piece_id=A5BFEHZ3WGGK53IZ7MKI6N4CVXHOORF6L47NR3UFDCAX7LBHRHVQ, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:48.565245605 (81h9m24.128568606s) [noise=true , address=79.127.205.234:5601 , piece_id=V7PMXBP2VFWBSZDG4GTLBD435T6F5DT7A43L4GQLQ5BDDTJ4WY6A, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:49.979979824 (81h9m25.543302825s) [noise=true , address=104.36.82.15:6289 , piece_id=GWL3P55BIK4IJEL5YFZKI7JCA4LADYOOHUSIQUNGJEKBJOTKMP5A, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:50.463750280 (81h9m26.02707328s) [noise=true , address=104.36.82.51:4084 , piece_id=D43FA7BFE54KPCWGDGDME2EGSWM7PONOVPD3DN2XY42MJFOKVDIQ, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:51.480599795 (81h9m27.043922798s) [noise=true , address=79.127.219.38:4023 , piece_id=BWLRCXXHBMBTNCZEVKKIIS5PB35PDQXQXCLIQB7GBLEIOWWUFBFA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:52.941200625 (81h9m28.504523627s) [noise=true , address=104.36.82.43:6841 , piece_id=PIHZMSTO3CII7KGDLGSAILTDRSLMEBVOLBCRCZJULMCFISXIZWXA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:58.281803598 (81h9m33.845126612s) [noise=true , address=64.49.13.139:4476 , piece_id=P7UKBNNZGI52B3I6WKTV6MVC262FYNKYYPJUI5G47UO5S4MU7PWA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:33:00.277271877 (81h9m35.840594867s) [noise=true , address=104.36.82.55:4213 , piece_id=X4IHS2XSPH52LIMNUYT7KGWMAPUH3TEMJOEEVB3JV5N45SEOH7YA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:33:09.435339213 (81h9m44.998662206s) [noise=true , address=104.36.82.52:4089 , piece_id=JSOEKLNWA3W3LOVKI3EGGKJYJ6J3UZY6KGABKQPQPXTIOOPMKVHA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:33:31.472868445 (81h10m7.03619143s) [noise=true , address=104.36.82.53:4122 , piece_id=HZZQBIKHQWLUE5EHK4KSKJGIUEG7D44RQASUJHCYXX6BICCGX6EA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:33:47.235404653 (81h10m22.798727642s) [noise=true , address=104.36.82.12:6158 , piece_id=LJZEOJVJCJAR2BRILIJI3ZVDGYMPBBTP2G3HCTN3MHNS4KYDPASQ, seg_pos={0 0}, upload=64958]: Done uploading piece. err: ecclient: upload failed (node:1SjM6Utm2YizeUbrBUD9Z7etE1nLadW5Fa1zYP7MioPeeqhdJ7, address:104.36.82.12:6158): protocol: expected piece hash; context deadline exceeded; EOF Jun 22 01:33:47.235448297 (81h10m22.798771306s) [seg_pos={0 0}, upload=64958]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jun 22 01:33:47.265442393 (81h10m22.828765401s) [upload=64958]: Done uploading segments... Jun 22 01:33:47.265476139 (81h10m22.82879914s) [upload=64958]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration:* 21.471543598s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.............................................X..............] duration: 16.592939492s FAIL (2025-06-22 01:55:26 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 16.592962863s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [...............................X............................] duration: 20.984687167s FAIL (2025-06-22 01:55:03 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== oad=64949]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jun 22 01:32:13.674692303 (81h8m49.238015311s) [upload=64949]: Done uploading segments... Jun 22 01:32:13.674727219 (81h8m49.238050233s) [upload=64949]: Upload finished. err: <nil> Jun 22 01:32:47.266407995 (81h9m22.829731002s) [upload=64958]: Starting upload Jun 22 01:32:47.266524767 (81h9m22.829847771s) [upload=64958]: Done starting upload Jun 22 01:32:47.266813109 (81h9m22.830136116s) [upload=64958]: Uploading segments... Jun 22 01:32:47.266969526 (81h9m22.830292526s) [upload=64958]: Got next segment. Inline: false Jun 22 01:32:47.267075475 (81h9m22.830398475s) [upload=64958]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 22 01:32:47.267081679 (81h9m22.830404667s) [upload=64958]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 22 01:32:47.301770382 (81h9m22.865093389s) [seg_pos={0 0}, upload=64958]: Begin upload segment... Jun 22 01:32:47.301823796 (81h9m22.865146799s) [seg_pos={0 0}, upload=64958]: Done begin upload segment. Jun 22 01:32:47.301855055 (81h9m22.865178087s) [noise=true , address=104.36.82.43:6841 , piece_id=PIHZMSTO3CII7KGDLGSAILTDRSLMEBVOLBCRCZJULMCFISXIZWXA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.301902600 (81h9m22.865225606s) [noise=true , address=104.36.82.42:6811 , piece_id=A5BFEHZ3WGGK53IZ7MKI6N4CVXHOORF6L47NR3UFDCAX7LBHRHVQ, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.302234027 (81h9m22.865557029s) [noise=true , address=104.36.82.51:4084 , piece_id=D43FA7BFE54KPCWGDGDME2EGSWM7PONOVPD3DN2XY42MJFOKVDIQ, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.302496354 (81h9m22.865819363s) [noise=true , address=79.127.219.38:4023 , piece_id=BWLRCXXHBMBTNCZEVKKIIS5PB35PDQXQXCLIQB7GBLEIOWWUFBFA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.302577370 (81h9m22.865900378s) [noise=true , address=104.36.82.52:4089 , piece_id=JSOEKLNWA3W3LOVKI3EGGKJYJ6J3UZY6KGABKQPQPXTIOOPMKVHA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.302917829 (81h9m22.866240835s) [noise=true , address=104.36.82.53:4122 , piece_id=HZZQBIKHQWLUE5EHK4KSKJGIUEG7D44RQASUJHCYXX6BICCGX6EA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.303087165 (81h9m22.866410168s) [noise=true , address=77.90.31.14:4689 , piece_id=Q3HDS3BCMYDTODFRMU4XFRDCQK2QZE3ZMXVLUX4M54PDEBS7RR6A, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.303250714 (81h9m22.866573717s) [noise=true , address=79.127.205.234:5601 , piece_id=V7PMXBP2VFWBSZDG4GTLBD435T6F5DT7A43L4GQLQ5BDDTJ4WY6A, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.303579249 (81h9m22.866902255s) [noise=true , address=104.36.82.12:6158 , piece_id=LJZEOJVJCJAR2BRILIJI3ZVDGYMPBBTP2G3HCTN3MHNS4KYDPASQ, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.303599776 (81h9m22.866922784s) [noise=true , address=104.36.82.9:6023 , piece_id=SISFAUU4RPF57OLGWM2BWQWE6GRUSCMORBYHWEXETNAVM2EEMO6Q, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.304005346 (81h9m22.867328346s) [noise=true , address=45.91.116.12:5242 , piece_id=6ABT5IPYGSLFFJMBYMBYRU4EYIXABUAUHG3OVILOHBIUDS3BP27A, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.304388850 (81h9m22.867711851s) [noise=true , address=64.49.13.137:4500 , piece_id=FAE2RXD57IRP5FXA2XR6THXFW27U7IWAZ4JDV5M2WRPXKUGZ4VIA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.304881529 (81h9m22.868204536s) [noise=true , address=79.127.220.101:6048 , piece_id=AN7UQUXB42T2YXEH7ZJRZUDIW47M3RWUEOCL2VEBOGH74BZF3YKA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.305249860 (81h9m22.86857287s) [noise=true , address=104.36.82.54:4165 , piece_id=4CVOABPCWC7J7SRHZB6IN2WJJ6KGWUQ66ELUIZIFA67ETLQ7JEUQ, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.305604108 (81h9m22.868927112s) [noise=true , address=104.36.82.15:6289 , piece_id=GWL3P55BIK4IJEL5YFZKI7JCA4LADYOOHUSIQUNGJEKBJOTKMP5A, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.305930423 (81h9m22.869253425s) [noise=true , address=64.49.13.139:4476 , piece_id=P7UKBNNZGI52B3I6WKTV6MVC262FYNKYYPJUI5G47UO5S4MU7PWA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.306355311 (81h9m22.869678319s) [noise=true , address=104.36.82.55:4213 , piece_id=X4IHS2XSPH52LIMNUYT7KGWMAPUH3TEMJOEEVB3JV5N45SEOH7YA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.306788243 (81h9m22.870111252s) [noise=true , address=104.36.82.22:6539 , piece_id=GHQ2MVSTDGVYLUKLV23D6YWRR4PP7UUYHCUEQ4WWSWEMTBFO3P4A, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.392174770 (81h9m22.955497773s) [upload=64958]: Next returned nil segment Jun 22 01:32:47.392195767 (81h9m22.955518775s) [upload=64958]: Waiting for error group managing segments... Jun 22 01:32:47.480767633 (81h9m23.044090623s) [noise=true , address=104.36.82.22:6539 , piece_id=GHQ2MVSTDGVYLUKLV23D6YWRR4PP7UUYHCUEQ4WWSWEMTBFO3P4A, seg_pos={0 0}, upload=64958]: Done uploading piece. err: ecclient: upload failed (node:1QL6w8DSFR6CxDdnJmDuyVZxp6JU55VKiJofBZRfQi7q31i8Jc, address:104.36.82.22:6539): protocol: storage node overloaded, request limit: 2000 Jun 22 01:32:47.585663142 (81h9m23.148986155s) [noise=true , address=79.127.220.101:6048 , piece_id=AN7UQUXB42T2YXEH7ZJRZUDIW47M3RWUEOCL2VEBOGH74BZF3YKA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:47.640263002 (81h9m23.203586013s) [noise=true , address=45.91.116.12:5242 , piece_id=6ABT5IPYGSLFFJMBYMBYRU4EYIXABUAUHG3OVILOHBIUDS3BP27A, seg_pos={0 0}, upload=64958]: Done uploading piece. err: ecclient: upload failed (node:12JsFL62jgXmza8Md8eQ9k8cyw5V9Vz2RzN4y85ZvgHZjwrq4vQ, address:45.91.116.12:5242): protocol: storage node overloaded, request limit: 2000 Jun 22 01:32:47.984127275 (81h9m23.54745028s) [noise=true , address=64.49.13.137:4500 , piece_id=FAE2RXD57IRP5FXA2XR6THXFW27U7IWAZ4JDV5M2WRPXKUGZ4VIA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:48.100255249 (81h9m23.663578241s) [noise=true , address=77.90.31.14:4689 , piece_id=Q3HDS3BCMYDTODFRMU4XFRDCQK2QZE3ZMXVLUX4M54PDEBS7RR6A, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:48.221764035 (81h9m23.785087034s) [noise=true , address=104.36.82.54:4165 , piece_id=4CVOABPCWC7J7SRHZB6IN2WJJ6KGWUQ66ELUIZIFA67ETLQ7JEUQ, seg_pos={0 0}, upload=64958]: Done uploading piece. err: ecclient: upload failed (node:12wGUMXVEubEtki4qGnjGEsbYFB1iUauZcu3oDRKrCQsJMV3bV2, address:104.36.82.54:4165): protocol: storage node overloaded, request limit: 2000 Jun 22 01:32:48.509996527 (81h9m24.073319547s) [noise=true , address=104.36.82.9:6023 , piece_id=SISFAUU4RPF57OLGWM2BWQWE6GRUSCMORBYHWEXETNAVM2EEMO6Q, seg_pos={0 0}, upload=64958]: Done uploading piece. err: ecclient: upload failed (node:1j2S9tgWpLwmyvXZSq2qKKcbaC8r7LkzfDCaMxCejAni4CST1g, address:104.36.82.9:6023): protocol: storage node overloaded, request limit: 2000 Jun 22 01:32:48.510513169 (81h9m24.073836177s) [noise=true , address=104.36.82.42:6811 , piece_id=A5BFEHZ3WGGK53IZ7MKI6N4CVXHOORF6L47NR3UFDCAX7LBHRHVQ, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:48.565245605 (81h9m24.128568606s) [noise=true , address=79.127.205.234:5601 , piece_id=V7PMXBP2VFWBSZDG4GTLBD435T6F5DT7A43L4GQLQ5BDDTJ4WY6A, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:49.979979824 (81h9m25.543302825s) [noise=true , address=104.36.82.15:6289 , piece_id=GWL3P55BIK4IJEL5YFZKI7JCA4LADYOOHUSIQUNGJEKBJOTKMP5A, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:50.463750280 (81h9m26.02707328s) [noise=true , address=104.36.82.51:4084 , piece_id=D43FA7BFE54KPCWGDGDME2EGSWM7PONOVPD3DN2XY42MJFOKVDIQ, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:51.480599795 (81h9m27.043922798s) [noise=true , address=79.127.219.38:4023 , piece_id=BWLRCXXHBMBTNCZEVKKIIS5PB35PDQXQXCLIQB7GBLEIOWWUFBFA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:52.941200625 (81h9m28.504523627s) [noise=true , address=104.36.82.43:6841 , piece_id=PIHZMSTO3CII7KGDLGSAILTDRSLMEBVOLBCRCZJULMCFISXIZWXA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:58.281803598 (81h9m33.845126612s) [noise=true , address=64.49.13.139:4476 , piece_id=P7UKBNNZGI52B3I6WKTV6MVC262FYNKYYPJUI5G47UO5S4MU7PWA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:33:00.277271877 (81h9m35.840594867s) [noise=true , address=104.36.82.55:4213 , piece_id=X4IHS2XSPH52LIMNUYT7KGWMAPUH3TEMJOEEVB3JV5N45SEOH7YA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:33:09.435339213 (81h9m44.998662206s) [noise=true , address=104.36.82.52:4089 , piece_id=JSOEKLNWA3W3LOVKI3EGGKJYJ6J3UZY6KGABKQPQPXTIOOPMKVHA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:33:31.472868445 (81h10m7.03619143s) [noise=true , address=104.36.82.53:4122 , piece_id=HZZQBIKHQWLUE5EHK4KSKJGIUEG7D44RQASUJHCYXX6BICCGX6EA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:33:47.235404653 (81h10m22.798727642s) [noise=true , address=104.36.82.12:6158 , piece_id=LJZEOJVJCJAR2BRILIJI3ZVDGYMPBBTP2G3HCTN3MHNS4KYDPASQ, seg_pos={0 0}, upload=64958]: Done uploading piece. err: ecclient: upload failed (node:1SjM6Utm2YizeUbrBUD9Z7etE1nLadW5Fa1zYP7MioPeeqhdJ7, address:104.36.82.12:6158): protocol: expected piece hash; context deadline exceeded; EOF Jun 22 01:33:47.235448297 (81h10m22.798771306s) [seg_pos={0 0}, upload=64958]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jun 22 01:33:47.265442393 (81h10m22.828765401s) [upload=64958]: Done uploading segments... Jun 22 01:33:47.265476139 (81h10m22.82879914s) [upload=64958]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration:* 20.984722668s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [..............................................X.............] duration: 22.75126827s FAIL (2025-06-22 01:54:39 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 22.751293422s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [................................X...........................] duration: 17.074210711s FAIL (2025-06-22 01:53:57 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== oad=64949]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jun 22 01:32:13.674692303 (81h8m49.238015311s) [upload=64949]: Done uploading segments... Jun 22 01:32:13.674727219 (81h8m49.238050233s) [upload=64949]: Upload finished. err: <nil> Jun 22 01:32:47.266407995 (81h9m22.829731002s) [upload=64958]: Starting upload Jun 22 01:32:47.266524767 (81h9m22.829847771s) [upload=64958]: Done starting upload Jun 22 01:32:47.266813109 (81h9m22.830136116s) [upload=64958]: Uploading segments... Jun 22 01:32:47.266969526 (81h9m22.830292526s) [upload=64958]: Got next segment. Inline: false Jun 22 01:32:47.267075475 (81h9m22.830398475s) [upload=64958]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 22 01:32:47.267081679 (81h9m22.830404667s) [upload=64958]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 22 01:32:47.301770382 (81h9m22.865093389s) [seg_pos={0 0}, upload=64958]: Begin upload segment... Jun 22 01:32:47.301823796 (81h9m22.865146799s) [seg_pos={0 0}, upload=64958]: Done begin upload segment. Jun 22 01:32:47.301855055 (81h9m22.865178087s) [noise=true , address=104.36.82.43:6841 , piece_id=PIHZMSTO3CII7KGDLGSAILTDRSLMEBVOLBCRCZJULMCFISXIZWXA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.301902600 (81h9m22.865225606s) [noise=true , address=104.36.82.42:6811 , piece_id=A5BFEHZ3WGGK53IZ7MKI6N4CVXHOORF6L47NR3UFDCAX7LBHRHVQ, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.302234027 (81h9m22.865557029s) [noise=true , address=104.36.82.51:4084 , piece_id=D43FA7BFE54KPCWGDGDME2EGSWM7PONOVPD3DN2XY42MJFOKVDIQ, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.302496354 (81h9m22.865819363s) [noise=true , address=79.127.219.38:4023 , piece_id=BWLRCXXHBMBTNCZEVKKIIS5PB35PDQXQXCLIQB7GBLEIOWWUFBFA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.302577370 (81h9m22.865900378s) [noise=true , address=104.36.82.52:4089 , piece_id=JSOEKLNWA3W3LOVKI3EGGKJYJ6J3UZY6KGABKQPQPXTIOOPMKVHA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.302917829 (81h9m22.866240835s) [noise=true , address=104.36.82.53:4122 , piece_id=HZZQBIKHQWLUE5EHK4KSKJGIUEG7D44RQASUJHCYXX6BICCGX6EA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.303087165 (81h9m22.866410168s) [noise=true , address=77.90.31.14:4689 , piece_id=Q3HDS3BCMYDTODFRMU4XFRDCQK2QZE3ZMXVLUX4M54PDEBS7RR6A, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.303250714 (81h9m22.866573717s) [noise=true , address=79.127.205.234:5601 , piece_id=V7PMXBP2VFWBSZDG4GTLBD435T6F5DT7A43L4GQLQ5BDDTJ4WY6A, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.303579249 (81h9m22.866902255s) [noise=true , address=104.36.82.12:6158 , piece_id=LJZEOJVJCJAR2BRILIJI3ZVDGYMPBBTP2G3HCTN3MHNS4KYDPASQ, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.303599776 (81h9m22.866922784s) [noise=true , address=104.36.82.9:6023 , piece_id=SISFAUU4RPF57OLGWM2BWQWE6GRUSCMORBYHWEXETNAVM2EEMO6Q, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.304005346 (81h9m22.867328346s) [noise=true , address=45.91.116.12:5242 , piece_id=6ABT5IPYGSLFFJMBYMBYRU4EYIXABUAUHG3OVILOHBIUDS3BP27A, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.304388850 (81h9m22.867711851s) [noise=true , address=64.49.13.137:4500 , piece_id=FAE2RXD57IRP5FXA2XR6THXFW27U7IWAZ4JDV5M2WRPXKUGZ4VIA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.304881529 (81h9m22.868204536s) [noise=true , address=79.127.220.101:6048 , piece_id=AN7UQUXB42T2YXEH7ZJRZUDIW47M3RWUEOCL2VEBOGH74BZF3YKA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.305249860 (81h9m22.86857287s) [noise=true , address=104.36.82.54:4165 , piece_id=4CVOABPCWC7J7SRHZB6IN2WJJ6KGWUQ66ELUIZIFA67ETLQ7JEUQ, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.305604108 (81h9m22.868927112s) [noise=true , address=104.36.82.15:6289 , piece_id=GWL3P55BIK4IJEL5YFZKI7JCA4LADYOOHUSIQUNGJEKBJOTKMP5A, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.305930423 (81h9m22.869253425s) [noise=true , address=64.49.13.139:4476 , piece_id=P7UKBNNZGI52B3I6WKTV6MVC262FYNKYYPJUI5G47UO5S4MU7PWA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.306355311 (81h9m22.869678319s) [noise=true , address=104.36.82.55:4213 , piece_id=X4IHS2XSPH52LIMNUYT7KGWMAPUH3TEMJOEEVB3JV5N45SEOH7YA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.306788243 (81h9m22.870111252s) [noise=true , address=104.36.82.22:6539 , piece_id=GHQ2MVSTDGVYLUKLV23D6YWRR4PP7UUYHCUEQ4WWSWEMTBFO3P4A, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.392174770 (81h9m22.955497773s) [upload=64958]: Next returned nil segment Jun 22 01:32:47.392195767 (81h9m22.955518775s) [upload=64958]: Waiting for error group managing segments... Jun 22 01:32:47.480767633 (81h9m23.044090623s) [noise=true , address=104.36.82.22:6539 , piece_id=GHQ2MVSTDGVYLUKLV23D6YWRR4PP7UUYHCUEQ4WWSWEMTBFO3P4A, seg_pos={0 0}, upload=64958]: Done uploading piece. err: ecclient: upload failed (node:1QL6w8DSFR6CxDdnJmDuyVZxp6JU55VKiJofBZRfQi7q31i8Jc, address:104.36.82.22:6539): protocol: storage node overloaded, request limit: 2000 Jun 22 01:32:47.585663142 (81h9m23.148986155s) [noise=true , address=79.127.220.101:6048 , piece_id=AN7UQUXB42T2YXEH7ZJRZUDIW47M3RWUEOCL2VEBOGH74BZF3YKA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:47.640263002 (81h9m23.203586013s) [noise=true , address=45.91.116.12:5242 , piece_id=6ABT5IPYGSLFFJMBYMBYRU4EYIXABUAUHG3OVILOHBIUDS3BP27A, seg_pos={0 0}, upload=64958]: Done uploading piece. err: ecclient: upload failed (node:12JsFL62jgXmza8Md8eQ9k8cyw5V9Vz2RzN4y85ZvgHZjwrq4vQ, address:45.91.116.12:5242): protocol: storage node overloaded, request limit: 2000 Jun 22 01:32:47.984127275 (81h9m23.54745028s) [noise=true , address=64.49.13.137:4500 , piece_id=FAE2RXD57IRP5FXA2XR6THXFW27U7IWAZ4JDV5M2WRPXKUGZ4VIA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:48.100255249 (81h9m23.663578241s) [noise=true , address=77.90.31.14:4689 , piece_id=Q3HDS3BCMYDTODFRMU4XFRDCQK2QZE3ZMXVLUX4M54PDEBS7RR6A, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:48.221764035 (81h9m23.785087034s) [noise=true , address=104.36.82.54:4165 , piece_id=4CVOABPCWC7J7SRHZB6IN2WJJ6KGWUQ66ELUIZIFA67ETLQ7JEUQ, seg_pos={0 0}, upload=64958]: Done uploading piece. err: ecclient: upload failed (node:12wGUMXVEubEtki4qGnjGEsbYFB1iUauZcu3oDRKrCQsJMV3bV2, address:104.36.82.54:4165): protocol: storage node overloaded, request limit: 2000 Jun 22 01:32:48.509996527 (81h9m24.073319547s) [noise=true , address=104.36.82.9:6023 , piece_id=SISFAUU4RPF57OLGWM2BWQWE6GRUSCMORBYHWEXETNAVM2EEMO6Q, seg_pos={0 0}, upload=64958]: Done uploading piece. err: ecclient: upload failed (node:1j2S9tgWpLwmyvXZSq2qKKcbaC8r7LkzfDCaMxCejAni4CST1g, address:104.36.82.9:6023): protocol: storage node overloaded, request limit: 2000 Jun 22 01:32:48.510513169 (81h9m24.073836177s) [noise=true , address=104.36.82.42:6811 , piece_id=A5BFEHZ3WGGK53IZ7MKI6N4CVXHOORF6L47NR3UFDCAX7LBHRHVQ, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:48.565245605 (81h9m24.128568606s) [noise=true , address=79.127.205.234:5601 , piece_id=V7PMXBP2VFWBSZDG4GTLBD435T6F5DT7A43L4GQLQ5BDDTJ4WY6A, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:49.979979824 (81h9m25.543302825s) [noise=true , address=104.36.82.15:6289 , piece_id=GWL3P55BIK4IJEL5YFZKI7JCA4LADYOOHUSIQUNGJEKBJOTKMP5A, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:50.463750280 (81h9m26.02707328s) [noise=true , address=104.36.82.51:4084 , piece_id=D43FA7BFE54KPCWGDGDME2EGSWM7PONOVPD3DN2XY42MJFOKVDIQ, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:51.480599795 (81h9m27.043922798s) [noise=true , address=79.127.219.38:4023 , piece_id=BWLRCXXHBMBTNCZEVKKIIS5PB35PDQXQXCLIQB7GBLEIOWWUFBFA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:52.941200625 (81h9m28.504523627s) [noise=true , address=104.36.82.43:6841 , piece_id=PIHZMSTO3CII7KGDLGSAILTDRSLMEBVOLBCRCZJULMCFISXIZWXA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:58.281803598 (81h9m33.845126612s) [noise=true , address=64.49.13.139:4476 , piece_id=P7UKBNNZGI52B3I6WKTV6MVC262FYNKYYPJUI5G47UO5S4MU7PWA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:33:00.277271877 (81h9m35.840594867s) [noise=true , address=104.36.82.55:4213 , piece_id=X4IHS2XSPH52LIMNUYT7KGWMAPUH3TEMJOEEVB3JV5N45SEOH7YA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:33:09.435339213 (81h9m44.998662206s) [noise=true , address=104.36.82.52:4089 , piece_id=JSOEKLNWA3W3LOVKI3EGGKJYJ6J3UZY6KGABKQPQPXTIOOPMKVHA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:33:31.472868445 (81h10m7.03619143s) [noise=true , address=104.36.82.53:4122 , piece_id=HZZQBIKHQWLUE5EHK4KSKJGIUEG7D44RQASUJHCYXX6BICCGX6EA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:33:47.235404653 (81h10m22.798727642s) [noise=true , address=104.36.82.12:6158 , piece_id=LJZEOJVJCJAR2BRILIJI3ZVDGYMPBBTP2G3HCTN3MHNS4KYDPASQ, seg_pos={0 0}, upload=64958]: Done uploading piece. err: ecclient: upload failed (node:1SjM6Utm2YizeUbrBUD9Z7etE1nLadW5Fa1zYP7MioPeeqhdJ7, address:104.36.82.12:6158): protocol: expected piece hash; context deadline exceeded; EOF Jun 22 01:33:47.235448297 (81h10m22.798771306s) [seg_pos={0 0}, upload=64958]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jun 22 01:33:47.265442393 (81h10m22.828765401s) [upload=64958]: Done uploading segments... Jun 22 01:33:47.265476139 (81h10m22.82879914s) [upload=64958]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration:* 17.074270958s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [...............................................X............] duration: 22.350940742s FAIL (2025-06-22 01:53:42 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 22.350996597s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [................................................X...........] duration: 22.399929398s FAIL (2025-06-22 01:53:19 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 22.399954527s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.................................................X..........] duration: 55.056514749s FAIL (2025-06-22 01:52:53 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 55.056656361s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.................................X..........................] duration: 22.494709581s FAIL (2025-06-22 01:52:48 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== oad=64949]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jun 22 01:32:13.674692303 (81h8m49.238015311s) [upload=64949]: Done uploading segments... Jun 22 01:32:13.674727219 (81h8m49.238050233s) [upload=64949]: Upload finished. err: <nil> Jun 22 01:32:47.266407995 (81h9m22.829731002s) [upload=64958]: Starting upload Jun 22 01:32:47.266524767 (81h9m22.829847771s) [upload=64958]: Done starting upload Jun 22 01:32:47.266813109 (81h9m22.830136116s) [upload=64958]: Uploading segments... Jun 22 01:32:47.266969526 (81h9m22.830292526s) [upload=64958]: Got next segment. Inline: false Jun 22 01:32:47.267075475 (81h9m22.830398475s) [upload=64958]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 22 01:32:47.267081679 (81h9m22.830404667s) [upload=64958]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 22 01:32:47.301770382 (81h9m22.865093389s) [seg_pos={0 0}, upload=64958]: Begin upload segment... Jun 22 01:32:47.301823796 (81h9m22.865146799s) [seg_pos={0 0}, upload=64958]: Done begin upload segment. Jun 22 01:32:47.301855055 (81h9m22.865178087s) [noise=true , address=104.36.82.43:6841 , piece_id=PIHZMSTO3CII7KGDLGSAILTDRSLMEBVOLBCRCZJULMCFISXIZWXA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.301902600 (81h9m22.865225606s) [noise=true , address=104.36.82.42:6811 , piece_id=A5BFEHZ3WGGK53IZ7MKI6N4CVXHOORF6L47NR3UFDCAX7LBHRHVQ, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.302234027 (81h9m22.865557029s) [noise=true , address=104.36.82.51:4084 , piece_id=D43FA7BFE54KPCWGDGDME2EGSWM7PONOVPD3DN2XY42MJFOKVDIQ, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.302496354 (81h9m22.865819363s) [noise=true , address=79.127.219.38:4023 , piece_id=BWLRCXXHBMBTNCZEVKKIIS5PB35PDQXQXCLIQB7GBLEIOWWUFBFA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.302577370 (81h9m22.865900378s) [noise=true , address=104.36.82.52:4089 , piece_id=JSOEKLNWA3W3LOVKI3EGGKJYJ6J3UZY6KGABKQPQPXTIOOPMKVHA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.302917829 (81h9m22.866240835s) [noise=true , address=104.36.82.53:4122 , piece_id=HZZQBIKHQWLUE5EHK4KSKJGIUEG7D44RQASUJHCYXX6BICCGX6EA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.303087165 (81h9m22.866410168s) [noise=true , address=77.90.31.14:4689 , piece_id=Q3HDS3BCMYDTODFRMU4XFRDCQK2QZE3ZMXVLUX4M54PDEBS7RR6A, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.303250714 (81h9m22.866573717s) [noise=true , address=79.127.205.234:5601 , piece_id=V7PMXBP2VFWBSZDG4GTLBD435T6F5DT7A43L4GQLQ5BDDTJ4WY6A, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.303579249 (81h9m22.866902255s) [noise=true , address=104.36.82.12:6158 , piece_id=LJZEOJVJCJAR2BRILIJI3ZVDGYMPBBTP2G3HCTN3MHNS4KYDPASQ, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.303599776 (81h9m22.866922784s) [noise=true , address=104.36.82.9:6023 , piece_id=SISFAUU4RPF57OLGWM2BWQWE6GRUSCMORBYHWEXETNAVM2EEMO6Q, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.304005346 (81h9m22.867328346s) [noise=true , address=45.91.116.12:5242 , piece_id=6ABT5IPYGSLFFJMBYMBYRU4EYIXABUAUHG3OVILOHBIUDS3BP27A, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.304388850 (81h9m22.867711851s) [noise=true , address=64.49.13.137:4500 , piece_id=FAE2RXD57IRP5FXA2XR6THXFW27U7IWAZ4JDV5M2WRPXKUGZ4VIA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.304881529 (81h9m22.868204536s) [noise=true , address=79.127.220.101:6048 , piece_id=AN7UQUXB42T2YXEH7ZJRZUDIW47M3RWUEOCL2VEBOGH74BZF3YKA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.305249860 (81h9m22.86857287s) [noise=true , address=104.36.82.54:4165 , piece_id=4CVOABPCWC7J7SRHZB6IN2WJJ6KGWUQ66ELUIZIFA67ETLQ7JEUQ, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.305604108 (81h9m22.868927112s) [noise=true , address=104.36.82.15:6289 , piece_id=GWL3P55BIK4IJEL5YFZKI7JCA4LADYOOHUSIQUNGJEKBJOTKMP5A, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.305930423 (81h9m22.869253425s) [noise=true , address=64.49.13.139:4476 , piece_id=P7UKBNNZGI52B3I6WKTV6MVC262FYNKYYPJUI5G47UO5S4MU7PWA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.306355311 (81h9m22.869678319s) [noise=true , address=104.36.82.55:4213 , piece_id=X4IHS2XSPH52LIMNUYT7KGWMAPUH3TEMJOEEVB3JV5N45SEOH7YA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.306788243 (81h9m22.870111252s) [noise=true , address=104.36.82.22:6539 , piece_id=GHQ2MVSTDGVYLUKLV23D6YWRR4PP7UUYHCUEQ4WWSWEMTBFO3P4A, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.392174770 (81h9m22.955497773s) [upload=64958]: Next returned nil segment Jun 22 01:32:47.392195767 (81h9m22.955518775s) [upload=64958]: Waiting for error group managing segments... Jun 22 01:32:47.480767633 (81h9m23.044090623s) [noise=true , address=104.36.82.22:6539 , piece_id=GHQ2MVSTDGVYLUKLV23D6YWRR4PP7UUYHCUEQ4WWSWEMTBFO3P4A, seg_pos={0 0}, upload=64958]: Done uploading piece. err: ecclient: upload failed (node:1QL6w8DSFR6CxDdnJmDuyVZxp6JU55VKiJofBZRfQi7q31i8Jc, address:104.36.82.22:6539): protocol: storage node overloaded, request limit: 2000 Jun 22 01:32:47.585663142 (81h9m23.148986155s) [noise=true , address=79.127.220.101:6048 , piece_id=AN7UQUXB42T2YXEH7ZJRZUDIW47M3RWUEOCL2VEBOGH74BZF3YKA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:47.640263002 (81h9m23.203586013s) [noise=true , address=45.91.116.12:5242 , piece_id=6ABT5IPYGSLFFJMBYMBYRU4EYIXABUAUHG3OVILOHBIUDS3BP27A, seg_pos={0 0}, upload=64958]: Done uploading piece. err: ecclient: upload failed (node:12JsFL62jgXmza8Md8eQ9k8cyw5V9Vz2RzN4y85ZvgHZjwrq4vQ, address:45.91.116.12:5242): protocol: storage node overloaded, request limit: 2000 Jun 22 01:32:47.984127275 (81h9m23.54745028s) [noise=true , address=64.49.13.137:4500 , piece_id=FAE2RXD57IRP5FXA2XR6THXFW27U7IWAZ4JDV5M2WRPXKUGZ4VIA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:48.100255249 (81h9m23.663578241s) [noise=true , address=77.90.31.14:4689 , piece_id=Q3HDS3BCMYDTODFRMU4XFRDCQK2QZE3ZMXVLUX4M54PDEBS7RR6A, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:48.221764035 (81h9m23.785087034s) [noise=true , address=104.36.82.54:4165 , piece_id=4CVOABPCWC7J7SRHZB6IN2WJJ6KGWUQ66ELUIZIFA67ETLQ7JEUQ, seg_pos={0 0}, upload=64958]: Done uploading piece. err: ecclient: upload failed (node:12wGUMXVEubEtki4qGnjGEsbYFB1iUauZcu3oDRKrCQsJMV3bV2, address:104.36.82.54:4165): protocol: storage node overloaded, request limit: 2000 Jun 22 01:32:48.509996527 (81h9m24.073319547s) [noise=true , address=104.36.82.9:6023 , piece_id=SISFAUU4RPF57OLGWM2BWQWE6GRUSCMORBYHWEXETNAVM2EEMO6Q, seg_pos={0 0}, upload=64958]: Done uploading piece. err: ecclient: upload failed (node:1j2S9tgWpLwmyvXZSq2qKKcbaC8r7LkzfDCaMxCejAni4CST1g, address:104.36.82.9:6023): protocol: storage node overloaded, request limit: 2000 Jun 22 01:32:48.510513169 (81h9m24.073836177s) [noise=true , address=104.36.82.42:6811 , piece_id=A5BFEHZ3WGGK53IZ7MKI6N4CVXHOORF6L47NR3UFDCAX7LBHRHVQ, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:48.565245605 (81h9m24.128568606s) [noise=true , address=79.127.205.234:5601 , piece_id=V7PMXBP2VFWBSZDG4GTLBD435T6F5DT7A43L4GQLQ5BDDTJ4WY6A, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:49.979979824 (81h9m25.543302825s) [noise=true , address=104.36.82.15:6289 , piece_id=GWL3P55BIK4IJEL5YFZKI7JCA4LADYOOHUSIQUNGJEKBJOTKMP5A, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:50.463750280 (81h9m26.02707328s) [noise=true , address=104.36.82.51:4084 , piece_id=D43FA7BFE54KPCWGDGDME2EGSWM7PONOVPD3DN2XY42MJFOKVDIQ, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:51.480599795 (81h9m27.043922798s) [noise=true , address=79.127.219.38:4023 , piece_id=BWLRCXXHBMBTNCZEVKKIIS5PB35PDQXQXCLIQB7GBLEIOWWUFBFA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:52.941200625 (81h9m28.504523627s) [noise=true , address=104.36.82.43:6841 , piece_id=PIHZMSTO3CII7KGDLGSAILTDRSLMEBVOLBCRCZJULMCFISXIZWXA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:58.281803598 (81h9m33.845126612s) [noise=true , address=64.49.13.139:4476 , piece_id=P7UKBNNZGI52B3I6WKTV6MVC262FYNKYYPJUI5G47UO5S4MU7PWA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:33:00.277271877 (81h9m35.840594867s) [noise=true , address=104.36.82.55:4213 , piece_id=X4IHS2XSPH52LIMNUYT7KGWMAPUH3TEMJOEEVB3JV5N45SEOH7YA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:33:09.435339213 (81h9m44.998662206s) [noise=true , address=104.36.82.52:4089 , piece_id=JSOEKLNWA3W3LOVKI3EGGKJYJ6J3UZY6KGABKQPQPXTIOOPMKVHA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:33:31.472868445 (81h10m7.03619143s) [noise=true , address=104.36.82.53:4122 , piece_id=HZZQBIKHQWLUE5EHK4KSKJGIUEG7D44RQASUJHCYXX6BICCGX6EA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:33:47.235404653 (81h10m22.798727642s) [noise=true , address=104.36.82.12:6158 , piece_id=LJZEOJVJCJAR2BRILIJI3ZVDGYMPBBTP2G3HCTN3MHNS4KYDPASQ, seg_pos={0 0}, upload=64958]: Done uploading piece. err: ecclient: upload failed (node:1SjM6Utm2YizeUbrBUD9Z7etE1nLadW5Fa1zYP7MioPeeqhdJ7, address:104.36.82.12:6158): protocol: expected piece hash; context deadline exceeded; EOF Jun 22 01:33:47.235448297 (81h10m22.798771306s) [seg_pos={0 0}, upload=64958]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jun 22 01:33:47.265442393 (81h10m22.828765401s) [upload=64958]: Done uploading segments... Jun 22 01:33:47.265476139 (81h10m22.82879914s) [upload=64958]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration:* 22.494753528s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [..................................X.........................] duration: 24.825329s FAIL (2025-06-22 01:51:47 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== oad=64949]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jun 22 01:32:13.674692303 (81h8m49.238015311s) [upload=64949]: Done uploading segments... Jun 22 01:32:13.674727219 (81h8m49.238050233s) [upload=64949]: Upload finished. err: <nil> Jun 22 01:32:47.266407995 (81h9m22.829731002s) [upload=64958]: Starting upload Jun 22 01:32:47.266524767 (81h9m22.829847771s) [upload=64958]: Done starting upload Jun 22 01:32:47.266813109 (81h9m22.830136116s) [upload=64958]: Uploading segments... Jun 22 01:32:47.266969526 (81h9m22.830292526s) [upload=64958]: Got next segment. Inline: false Jun 22 01:32:47.267075475 (81h9m22.830398475s) [upload=64958]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 22 01:32:47.267081679 (81h9m22.830404667s) [upload=64958]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 22 01:32:47.301770382 (81h9m22.865093389s) [seg_pos={0 0}, upload=64958]: Begin upload segment... Jun 22 01:32:47.301823796 (81h9m22.865146799s) [seg_pos={0 0}, upload=64958]: Done begin upload segment. Jun 22 01:32:47.301855055 (81h9m22.865178087s) [noise=true , address=104.36.82.43:6841 , piece_id=PIHZMSTO3CII7KGDLGSAILTDRSLMEBVOLBCRCZJULMCFISXIZWXA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.301902600 (81h9m22.865225606s) [noise=true , address=104.36.82.42:6811 , piece_id=A5BFEHZ3WGGK53IZ7MKI6N4CVXHOORF6L47NR3UFDCAX7LBHRHVQ, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.302234027 (81h9m22.865557029s) [noise=true , address=104.36.82.51:4084 , piece_id=D43FA7BFE54KPCWGDGDME2EGSWM7PONOVPD3DN2XY42MJFOKVDIQ, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.302496354 (81h9m22.865819363s) [noise=true , address=79.127.219.38:4023 , piece_id=BWLRCXXHBMBTNCZEVKKIIS5PB35PDQXQXCLIQB7GBLEIOWWUFBFA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.302577370 (81h9m22.865900378s) [noise=true , address=104.36.82.52:4089 , piece_id=JSOEKLNWA3W3LOVKI3EGGKJYJ6J3UZY6KGABKQPQPXTIOOPMKVHA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.302917829 (81h9m22.866240835s) [noise=true , address=104.36.82.53:4122 , piece_id=HZZQBIKHQWLUE5EHK4KSKJGIUEG7D44RQASUJHCYXX6BICCGX6EA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.303087165 (81h9m22.866410168s) [noise=true , address=77.90.31.14:4689 , piece_id=Q3HDS3BCMYDTODFRMU4XFRDCQK2QZE3ZMXVLUX4M54PDEBS7RR6A, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.303250714 (81h9m22.866573717s) [noise=true , address=79.127.205.234:5601 , piece_id=V7PMXBP2VFWBSZDG4GTLBD435T6F5DT7A43L4GQLQ5BDDTJ4WY6A, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.303579249 (81h9m22.866902255s) [noise=true , address=104.36.82.12:6158 , piece_id=LJZEOJVJCJAR2BRILIJI3ZVDGYMPBBTP2G3HCTN3MHNS4KYDPASQ, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.303599776 (81h9m22.866922784s) [noise=true , address=104.36.82.9:6023 , piece_id=SISFAUU4RPF57OLGWM2BWQWE6GRUSCMORBYHWEXETNAVM2EEMO6Q, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.304005346 (81h9m22.867328346s) [noise=true , address=45.91.116.12:5242 , piece_id=6ABT5IPYGSLFFJMBYMBYRU4EYIXABUAUHG3OVILOHBIUDS3BP27A, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.304388850 (81h9m22.867711851s) [noise=true , address=64.49.13.137:4500 , piece_id=FAE2RXD57IRP5FXA2XR6THXFW27U7IWAZ4JDV5M2WRPXKUGZ4VIA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.304881529 (81h9m22.868204536s) [noise=true , address=79.127.220.101:6048 , piece_id=AN7UQUXB42T2YXEH7ZJRZUDIW47M3RWUEOCL2VEBOGH74BZF3YKA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.305249860 (81h9m22.86857287s) [noise=true , address=104.36.82.54:4165 , piece_id=4CVOABPCWC7J7SRHZB6IN2WJJ6KGWUQ66ELUIZIFA67ETLQ7JEUQ, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.305604108 (81h9m22.868927112s) [noise=true , address=104.36.82.15:6289 , piece_id=GWL3P55BIK4IJEL5YFZKI7JCA4LADYOOHUSIQUNGJEKBJOTKMP5A, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.305930423 (81h9m22.869253425s) [noise=true , address=64.49.13.139:4476 , piece_id=P7UKBNNZGI52B3I6WKTV6MVC262FYNKYYPJUI5G47UO5S4MU7PWA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.306355311 (81h9m22.869678319s) [noise=true , address=104.36.82.55:4213 , piece_id=X4IHS2XSPH52LIMNUYT7KGWMAPUH3TEMJOEEVB3JV5N45SEOH7YA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.306788243 (81h9m22.870111252s) [noise=true , address=104.36.82.22:6539 , piece_id=GHQ2MVSTDGVYLUKLV23D6YWRR4PP7UUYHCUEQ4WWSWEMTBFO3P4A, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.392174770 (81h9m22.955497773s) [upload=64958]: Next returned nil segment Jun 22 01:32:47.392195767 (81h9m22.955518775s) [upload=64958]: Waiting for error group managing segments... Jun 22 01:32:47.480767633 (81h9m23.044090623s) [noise=true , address=104.36.82.22:6539 , piece_id=GHQ2MVSTDGVYLUKLV23D6YWRR4PP7UUYHCUEQ4WWSWEMTBFO3P4A, seg_pos={0 0}, upload=64958]: Done uploading piece. err: ecclient: upload failed (node:1QL6w8DSFR6CxDdnJmDuyVZxp6JU55VKiJofBZRfQi7q31i8Jc, address:104.36.82.22:6539): protocol: storage node overloaded, request limit: 2000 Jun 22 01:32:47.585663142 (81h9m23.148986155s) [noise=true , address=79.127.220.101:6048 , piece_id=AN7UQUXB42T2YXEH7ZJRZUDIW47M3RWUEOCL2VEBOGH74BZF3YKA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:47.640263002 (81h9m23.203586013s) [noise=true , address=45.91.116.12:5242 , piece_id=6ABT5IPYGSLFFJMBYMBYRU4EYIXABUAUHG3OVILOHBIUDS3BP27A, seg_pos={0 0}, upload=64958]: Done uploading piece. err: ecclient: upload failed (node:12JsFL62jgXmza8Md8eQ9k8cyw5V9Vz2RzN4y85ZvgHZjwrq4vQ, address:45.91.116.12:5242): protocol: storage node overloaded, request limit: 2000 Jun 22 01:32:47.984127275 (81h9m23.54745028s) [noise=true , address=64.49.13.137:4500 , piece_id=FAE2RXD57IRP5FXA2XR6THXFW27U7IWAZ4JDV5M2WRPXKUGZ4VIA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:48.100255249 (81h9m23.663578241s) [noise=true , address=77.90.31.14:4689 , piece_id=Q3HDS3BCMYDTODFRMU4XFRDCQK2QZE3ZMXVLUX4M54PDEBS7RR6A, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:48.221764035 (81h9m23.785087034s) [noise=true , address=104.36.82.54:4165 , piece_id=4CVOABPCWC7J7SRHZB6IN2WJJ6KGWUQ66ELUIZIFA67ETLQ7JEUQ, seg_pos={0 0}, upload=64958]: Done uploading piece. err: ecclient: upload failed (node:12wGUMXVEubEtki4qGnjGEsbYFB1iUauZcu3oDRKrCQsJMV3bV2, address:104.36.82.54:4165): protocol: storage node overloaded, request limit: 2000 Jun 22 01:32:48.509996527 (81h9m24.073319547s) [noise=true , address=104.36.82.9:6023 , piece_id=SISFAUU4RPF57OLGWM2BWQWE6GRUSCMORBYHWEXETNAVM2EEMO6Q, seg_pos={0 0}, upload=64958]: Done uploading piece. err: ecclient: upload failed (node:1j2S9tgWpLwmyvXZSq2qKKcbaC8r7LkzfDCaMxCejAni4CST1g, address:104.36.82.9:6023): protocol: storage node overloaded, request limit: 2000 Jun 22 01:32:48.510513169 (81h9m24.073836177s) [noise=true , address=104.36.82.42:6811 , piece_id=A5BFEHZ3WGGK53IZ7MKI6N4CVXHOORF6L47NR3UFDCAX7LBHRHVQ, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:48.565245605 (81h9m24.128568606s) [noise=true , address=79.127.205.234:5601 , piece_id=V7PMXBP2VFWBSZDG4GTLBD435T6F5DT7A43L4GQLQ5BDDTJ4WY6A, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:49.979979824 (81h9m25.543302825s) [noise=true , address=104.36.82.15:6289 , piece_id=GWL3P55BIK4IJEL5YFZKI7JCA4LADYOOHUSIQUNGJEKBJOTKMP5A, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:50.463750280 (81h9m26.02707328s) [noise=true , address=104.36.82.51:4084 , piece_id=D43FA7BFE54KPCWGDGDME2EGSWM7PONOVPD3DN2XY42MJFOKVDIQ, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:51.480599795 (81h9m27.043922798s) [noise=true , address=79.127.219.38:4023 , piece_id=BWLRCXXHBMBTNCZEVKKIIS5PB35PDQXQXCLIQB7GBLEIOWWUFBFA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:52.941200625 (81h9m28.504523627s) [noise=true , address=104.36.82.43:6841 , piece_id=PIHZMSTO3CII7KGDLGSAILTDRSLMEBVOLBCRCZJULMCFISXIZWXA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:58.281803598 (81h9m33.845126612s) [noise=true , address=64.49.13.139:4476 , piece_id=P7UKBNNZGI52B3I6WKTV6MVC262FYNKYYPJUI5G47UO5S4MU7PWA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:33:00.277271877 (81h9m35.840594867s) [noise=true , address=104.36.82.55:4213 , piece_id=X4IHS2XSPH52LIMNUYT7KGWMAPUH3TEMJOEEVB3JV5N45SEOH7YA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:33:09.435339213 (81h9m44.998662206s) [noise=true , address=104.36.82.52:4089 , piece_id=JSOEKLNWA3W3LOVKI3EGGKJYJ6J3UZY6KGABKQPQPXTIOOPMKVHA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:33:31.472868445 (81h10m7.03619143s) [noise=true , address=104.36.82.53:4122 , piece_id=HZZQBIKHQWLUE5EHK4KSKJGIUEG7D44RQASUJHCYXX6BICCGX6EA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:33:47.235404653 (81h10m22.798727642s) [noise=true , address=104.36.82.12:6158 , piece_id=LJZEOJVJCJAR2BRILIJI3ZVDGYMPBBTP2G3HCTN3MHNS4KYDPASQ, seg_pos={0 0}, upload=64958]: Done uploading piece. err: ecclient: upload failed (node:1SjM6Utm2YizeUbrBUD9Z7etE1nLadW5Fa1zYP7MioPeeqhdJ7, address:104.36.82.12:6158): protocol: expected piece hash; context deadline exceeded; EOF Jun 22 01:33:47.235448297 (81h10m22.798771306s) [seg_pos={0 0}, upload=64958]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jun 22 01:33:47.265442393 (81h10m22.828765401s) [upload=64958]: Done uploading segments... Jun 22 01:33:47.265476139 (81h10m22.82879914s) [upload=64958]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration:* 24.825362237s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [..................................................X.........] duration: 37.123619797s FAIL (2025-06-22 01:51:39 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 37.12365105s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [...................................................X........] duration: 18.274712942s FAIL (2025-06-22 01:51:02 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 18.274770727s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [...................................X........................] duration: 16.774098176s FAIL (2025-06-22 01:50:54 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== oad=64949]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jun 22 01:32:13.674692303 (81h8m49.238015311s) [upload=64949]: Done uploading segments... Jun 22 01:32:13.674727219 (81h8m49.238050233s) [upload=64949]: Upload finished. err: <nil> Jun 22 01:32:47.266407995 (81h9m22.829731002s) [upload=64958]: Starting upload Jun 22 01:32:47.266524767 (81h9m22.829847771s) [upload=64958]: Done starting upload Jun 22 01:32:47.266813109 (81h9m22.830136116s) [upload=64958]: Uploading segments... Jun 22 01:32:47.266969526 (81h9m22.830292526s) [upload=64958]: Got next segment. Inline: false Jun 22 01:32:47.267075475 (81h9m22.830398475s) [upload=64958]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 22 01:32:47.267081679 (81h9m22.830404667s) [upload=64958]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 22 01:32:47.301770382 (81h9m22.865093389s) [seg_pos={0 0}, upload=64958]: Begin upload segment... Jun 22 01:32:47.301823796 (81h9m22.865146799s) [seg_pos={0 0}, upload=64958]: Done begin upload segment. Jun 22 01:32:47.301855055 (81h9m22.865178087s) [noise=true , address=104.36.82.43:6841 , piece_id=PIHZMSTO3CII7KGDLGSAILTDRSLMEBVOLBCRCZJULMCFISXIZWXA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.301902600 (81h9m22.865225606s) [noise=true , address=104.36.82.42:6811 , piece_id=A5BFEHZ3WGGK53IZ7MKI6N4CVXHOORF6L47NR3UFDCAX7LBHRHVQ, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.302234027 (81h9m22.865557029s) [noise=true , address=104.36.82.51:4084 , piece_id=D43FA7BFE54KPCWGDGDME2EGSWM7PONOVPD3DN2XY42MJFOKVDIQ, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.302496354 (81h9m22.865819363s) [noise=true , address=79.127.219.38:4023 , piece_id=BWLRCXXHBMBTNCZEVKKIIS5PB35PDQXQXCLIQB7GBLEIOWWUFBFA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.302577370 (81h9m22.865900378s) [noise=true , address=104.36.82.52:4089 , piece_id=JSOEKLNWA3W3LOVKI3EGGKJYJ6J3UZY6KGABKQPQPXTIOOPMKVHA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.302917829 (81h9m22.866240835s) [noise=true , address=104.36.82.53:4122 , piece_id=HZZQBIKHQWLUE5EHK4KSKJGIUEG7D44RQASUJHCYXX6BICCGX6EA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.303087165 (81h9m22.866410168s) [noise=true , address=77.90.31.14:4689 , piece_id=Q3HDS3BCMYDTODFRMU4XFRDCQK2QZE3ZMXVLUX4M54PDEBS7RR6A, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.303250714 (81h9m22.866573717s) [noise=true , address=79.127.205.234:5601 , piece_id=V7PMXBP2VFWBSZDG4GTLBD435T6F5DT7A43L4GQLQ5BDDTJ4WY6A, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.303579249 (81h9m22.866902255s) [noise=true , address=104.36.82.12:6158 , piece_id=LJZEOJVJCJAR2BRILIJI3ZVDGYMPBBTP2G3HCTN3MHNS4KYDPASQ, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.303599776 (81h9m22.866922784s) [noise=true , address=104.36.82.9:6023 , piece_id=SISFAUU4RPF57OLGWM2BWQWE6GRUSCMORBYHWEXETNAVM2EEMO6Q, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.304005346 (81h9m22.867328346s) [noise=true , address=45.91.116.12:5242 , piece_id=6ABT5IPYGSLFFJMBYMBYRU4EYIXABUAUHG3OVILOHBIUDS3BP27A, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.304388850 (81h9m22.867711851s) [noise=true , address=64.49.13.137:4500 , piece_id=FAE2RXD57IRP5FXA2XR6THXFW27U7IWAZ4JDV5M2WRPXKUGZ4VIA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.304881529 (81h9m22.868204536s) [noise=true , address=79.127.220.101:6048 , piece_id=AN7UQUXB42T2YXEH7ZJRZUDIW47M3RWUEOCL2VEBOGH74BZF3YKA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.305249860 (81h9m22.86857287s) [noise=true , address=104.36.82.54:4165 , piece_id=4CVOABPCWC7J7SRHZB6IN2WJJ6KGWUQ66ELUIZIFA67ETLQ7JEUQ, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.305604108 (81h9m22.868927112s) [noise=true , address=104.36.82.15:6289 , piece_id=GWL3P55BIK4IJEL5YFZKI7JCA4LADYOOHUSIQUNGJEKBJOTKMP5A, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.305930423 (81h9m22.869253425s) [noise=true , address=64.49.13.139:4476 , piece_id=P7UKBNNZGI52B3I6WKTV6MVC262FYNKYYPJUI5G47UO5S4MU7PWA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.306355311 (81h9m22.869678319s) [noise=true , address=104.36.82.55:4213 , piece_id=X4IHS2XSPH52LIMNUYT7KGWMAPUH3TEMJOEEVB3JV5N45SEOH7YA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.306788243 (81h9m22.870111252s) [noise=true , address=104.36.82.22:6539 , piece_id=GHQ2MVSTDGVYLUKLV23D6YWRR4PP7UUYHCUEQ4WWSWEMTBFO3P4A, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.392174770 (81h9m22.955497773s) [upload=64958]: Next returned nil segment Jun 22 01:32:47.392195767 (81h9m22.955518775s) [upload=64958]: Waiting for error group managing segments... Jun 22 01:32:47.480767633 (81h9m23.044090623s) [noise=true , address=104.36.82.22:6539 , piece_id=GHQ2MVSTDGVYLUKLV23D6YWRR4PP7UUYHCUEQ4WWSWEMTBFO3P4A, seg_pos={0 0}, upload=64958]: Done uploading piece. err: ecclient: upload failed (node:1QL6w8DSFR6CxDdnJmDuyVZxp6JU55VKiJofBZRfQi7q31i8Jc, address:104.36.82.22:6539): protocol: storage node overloaded, request limit: 2000 Jun 22 01:32:47.585663142 (81h9m23.148986155s) [noise=true , address=79.127.220.101:6048 , piece_id=AN7UQUXB42T2YXEH7ZJRZUDIW47M3RWUEOCL2VEBOGH74BZF3YKA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:47.640263002 (81h9m23.203586013s) [noise=true , address=45.91.116.12:5242 , piece_id=6ABT5IPYGSLFFJMBYMBYRU4EYIXABUAUHG3OVILOHBIUDS3BP27A, seg_pos={0 0}, upload=64958]: Done uploading piece. err: ecclient: upload failed (node:12JsFL62jgXmza8Md8eQ9k8cyw5V9Vz2RzN4y85ZvgHZjwrq4vQ, address:45.91.116.12:5242): protocol: storage node overloaded, request limit: 2000 Jun 22 01:32:47.984127275 (81h9m23.54745028s) [noise=true , address=64.49.13.137:4500 , piece_id=FAE2RXD57IRP5FXA2XR6THXFW27U7IWAZ4JDV5M2WRPXKUGZ4VIA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:48.100255249 (81h9m23.663578241s) [noise=true , address=77.90.31.14:4689 , piece_id=Q3HDS3BCMYDTODFRMU4XFRDCQK2QZE3ZMXVLUX4M54PDEBS7RR6A, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:48.221764035 (81h9m23.785087034s) [noise=true , address=104.36.82.54:4165 , piece_id=4CVOABPCWC7J7SRHZB6IN2WJJ6KGWUQ66ELUIZIFA67ETLQ7JEUQ, seg_pos={0 0}, upload=64958]: Done uploading piece. err: ecclient: upload failed (node:12wGUMXVEubEtki4qGnjGEsbYFB1iUauZcu3oDRKrCQsJMV3bV2, address:104.36.82.54:4165): protocol: storage node overloaded, request limit: 2000 Jun 22 01:32:48.509996527 (81h9m24.073319547s) [noise=true , address=104.36.82.9:6023 , piece_id=SISFAUU4RPF57OLGWM2BWQWE6GRUSCMORBYHWEXETNAVM2EEMO6Q, seg_pos={0 0}, upload=64958]: Done uploading piece. err: ecclient: upload failed (node:1j2S9tgWpLwmyvXZSq2qKKcbaC8r7LkzfDCaMxCejAni4CST1g, address:104.36.82.9:6023): protocol: storage node overloaded, request limit: 2000 Jun 22 01:32:48.510513169 (81h9m24.073836177s) [noise=true , address=104.36.82.42:6811 , piece_id=A5BFEHZ3WGGK53IZ7MKI6N4CVXHOORF6L47NR3UFDCAX7LBHRHVQ, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:48.565245605 (81h9m24.128568606s) [noise=true , address=79.127.205.234:5601 , piece_id=V7PMXBP2VFWBSZDG4GTLBD435T6F5DT7A43L4GQLQ5BDDTJ4WY6A, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:49.979979824 (81h9m25.543302825s) [noise=true , address=104.36.82.15:6289 , piece_id=GWL3P55BIK4IJEL5YFZKI7JCA4LADYOOHUSIQUNGJEKBJOTKMP5A, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:50.463750280 (81h9m26.02707328s) [noise=true , address=104.36.82.51:4084 , piece_id=D43FA7BFE54KPCWGDGDME2EGSWM7PONOVPD3DN2XY42MJFOKVDIQ, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:51.480599795 (81h9m27.043922798s) [noise=true , address=79.127.219.38:4023 , piece_id=BWLRCXXHBMBTNCZEVKKIIS5PB35PDQXQXCLIQB7GBLEIOWWUFBFA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:52.941200625 (81h9m28.504523627s) [noise=true , address=104.36.82.43:6841 , piece_id=PIHZMSTO3CII7KGDLGSAILTDRSLMEBVOLBCRCZJULMCFISXIZWXA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:58.281803598 (81h9m33.845126612s) [noise=true , address=64.49.13.139:4476 , piece_id=P7UKBNNZGI52B3I6WKTV6MVC262FYNKYYPJUI5G47UO5S4MU7PWA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:33:00.277271877 (81h9m35.840594867s) [noise=true , address=104.36.82.55:4213 , piece_id=X4IHS2XSPH52LIMNUYT7KGWMAPUH3TEMJOEEVB3JV5N45SEOH7YA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:33:09.435339213 (81h9m44.998662206s) [noise=true , address=104.36.82.52:4089 , piece_id=JSOEKLNWA3W3LOVKI3EGGKJYJ6J3UZY6KGABKQPQPXTIOOPMKVHA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:33:31.472868445 (81h10m7.03619143s) [noise=true , address=104.36.82.53:4122 , piece_id=HZZQBIKHQWLUE5EHK4KSKJGIUEG7D44RQASUJHCYXX6BICCGX6EA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:33:47.235404653 (81h10m22.798727642s) [noise=true , address=104.36.82.12:6158 , piece_id=LJZEOJVJCJAR2BRILIJI3ZVDGYMPBBTP2G3HCTN3MHNS4KYDPASQ, seg_pos={0 0}, upload=64958]: Done uploading piece. err: ecclient: upload failed (node:1SjM6Utm2YizeUbrBUD9Z7etE1nLadW5Fa1zYP7MioPeeqhdJ7, address:104.36.82.12:6158): protocol: expected piece hash; context deadline exceeded; EOF Jun 22 01:33:47.235448297 (81h10m22.798771306s) [seg_pos={0 0}, upload=64958]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jun 22 01:33:47.265442393 (81h10m22.828765401s) [upload=64958]: Done uploading segments... Jun 22 01:33:47.265476139 (81h10m22.82879914s) [upload=64958]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration:* 16.774211366s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [....................................X.......................] duration: 14.85489583s FAIL (2025-06-22 01:49:53 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== oad=64949]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jun 22 01:32:13.674692303 (81h8m49.238015311s) [upload=64949]: Done uploading segments... Jun 22 01:32:13.674727219 (81h8m49.238050233s) [upload=64949]: Upload finished. err: <nil> Jun 22 01:32:47.266407995 (81h9m22.829731002s) [upload=64958]: Starting upload Jun 22 01:32:47.266524767 (81h9m22.829847771s) [upload=64958]: Done starting upload Jun 22 01:32:47.266813109 (81h9m22.830136116s) [upload=64958]: Uploading segments... Jun 22 01:32:47.266969526 (81h9m22.830292526s) [upload=64958]: Got next segment. Inline: false Jun 22 01:32:47.267075475 (81h9m22.830398475s) [upload=64958]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 22 01:32:47.267081679 (81h9m22.830404667s) [upload=64958]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 22 01:32:47.301770382 (81h9m22.865093389s) [seg_pos={0 0}, upload=64958]: Begin upload segment... Jun 22 01:32:47.301823796 (81h9m22.865146799s) [seg_pos={0 0}, upload=64958]: Done begin upload segment. Jun 22 01:32:47.301855055 (81h9m22.865178087s) [noise=true , address=104.36.82.43:6841 , piece_id=PIHZMSTO3CII7KGDLGSAILTDRSLMEBVOLBCRCZJULMCFISXIZWXA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.301902600 (81h9m22.865225606s) [noise=true , address=104.36.82.42:6811 , piece_id=A5BFEHZ3WGGK53IZ7MKI6N4CVXHOORF6L47NR3UFDCAX7LBHRHVQ, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.302234027 (81h9m22.865557029s) [noise=true , address=104.36.82.51:4084 , piece_id=D43FA7BFE54KPCWGDGDME2EGSWM7PONOVPD3DN2XY42MJFOKVDIQ, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.302496354 (81h9m22.865819363s) [noise=true , address=79.127.219.38:4023 , piece_id=BWLRCXXHBMBTNCZEVKKIIS5PB35PDQXQXCLIQB7GBLEIOWWUFBFA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.302577370 (81h9m22.865900378s) [noise=true , address=104.36.82.52:4089 , piece_id=JSOEKLNWA3W3LOVKI3EGGKJYJ6J3UZY6KGABKQPQPXTIOOPMKVHA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.302917829 (81h9m22.866240835s) [noise=true , address=104.36.82.53:4122 , piece_id=HZZQBIKHQWLUE5EHK4KSKJGIUEG7D44RQASUJHCYXX6BICCGX6EA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.303087165 (81h9m22.866410168s) [noise=true , address=77.90.31.14:4689 , piece_id=Q3HDS3BCMYDTODFRMU4XFRDCQK2QZE3ZMXVLUX4M54PDEBS7RR6A, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.303250714 (81h9m22.866573717s) [noise=true , address=79.127.205.234:5601 , piece_id=V7PMXBP2VFWBSZDG4GTLBD435T6F5DT7A43L4GQLQ5BDDTJ4WY6A, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.303579249 (81h9m22.866902255s) [noise=true , address=104.36.82.12:6158 , piece_id=LJZEOJVJCJAR2BRILIJI3ZVDGYMPBBTP2G3HCTN3MHNS4KYDPASQ, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.303599776 (81h9m22.866922784s) [noise=true , address=104.36.82.9:6023 , piece_id=SISFAUU4RPF57OLGWM2BWQWE6GRUSCMORBYHWEXETNAVM2EEMO6Q, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.304005346 (81h9m22.867328346s) [noise=true , address=45.91.116.12:5242 , piece_id=6ABT5IPYGSLFFJMBYMBYRU4EYIXABUAUHG3OVILOHBIUDS3BP27A, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.304388850 (81h9m22.867711851s) [noise=true , address=64.49.13.137:4500 , piece_id=FAE2RXD57IRP5FXA2XR6THXFW27U7IWAZ4JDV5M2WRPXKUGZ4VIA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.304881529 (81h9m22.868204536s) [noise=true , address=79.127.220.101:6048 , piece_id=AN7UQUXB42T2YXEH7ZJRZUDIW47M3RWUEOCL2VEBOGH74BZF3YKA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.305249860 (81h9m22.86857287s) [noise=true , address=104.36.82.54:4165 , piece_id=4CVOABPCWC7J7SRHZB6IN2WJJ6KGWUQ66ELUIZIFA67ETLQ7JEUQ, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.305604108 (81h9m22.868927112s) [noise=true , address=104.36.82.15:6289 , piece_id=GWL3P55BIK4IJEL5YFZKI7JCA4LADYOOHUSIQUNGJEKBJOTKMP5A, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.305930423 (81h9m22.869253425s) [noise=true , address=64.49.13.139:4476 , piece_id=P7UKBNNZGI52B3I6WKTV6MVC262FYNKYYPJUI5G47UO5S4MU7PWA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.306355311 (81h9m22.869678319s) [noise=true , address=104.36.82.55:4213 , piece_id=X4IHS2XSPH52LIMNUYT7KGWMAPUH3TEMJOEEVB3JV5N45SEOH7YA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.306788243 (81h9m22.870111252s) [noise=true , address=104.36.82.22:6539 , piece_id=GHQ2MVSTDGVYLUKLV23D6YWRR4PP7UUYHCUEQ4WWSWEMTBFO3P4A, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.392174770 (81h9m22.955497773s) [upload=64958]: Next returned nil segment Jun 22 01:32:47.392195767 (81h9m22.955518775s) [upload=64958]: Waiting for error group managing segments... Jun 22 01:32:47.480767633 (81h9m23.044090623s) [noise=true , address=104.36.82.22:6539 , piece_id=GHQ2MVSTDGVYLUKLV23D6YWRR4PP7UUYHCUEQ4WWSWEMTBFO3P4A, seg_pos={0 0}, upload=64958]: Done uploading piece. err: ecclient: upload failed (node:1QL6w8DSFR6CxDdnJmDuyVZxp6JU55VKiJofBZRfQi7q31i8Jc, address:104.36.82.22:6539): protocol: storage node overloaded, request limit: 2000 Jun 22 01:32:47.585663142 (81h9m23.148986155s) [noise=true , address=79.127.220.101:6048 , piece_id=AN7UQUXB42T2YXEH7ZJRZUDIW47M3RWUEOCL2VEBOGH74BZF3YKA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:47.640263002 (81h9m23.203586013s) [noise=true , address=45.91.116.12:5242 , piece_id=6ABT5IPYGSLFFJMBYMBYRU4EYIXABUAUHG3OVILOHBIUDS3BP27A, seg_pos={0 0}, upload=64958]: Done uploading piece. err: ecclient: upload failed (node:12JsFL62jgXmza8Md8eQ9k8cyw5V9Vz2RzN4y85ZvgHZjwrq4vQ, address:45.91.116.12:5242): protocol: storage node overloaded, request limit: 2000 Jun 22 01:32:47.984127275 (81h9m23.54745028s) [noise=true , address=64.49.13.137:4500 , piece_id=FAE2RXD57IRP5FXA2XR6THXFW27U7IWAZ4JDV5M2WRPXKUGZ4VIA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:48.100255249 (81h9m23.663578241s) [noise=true , address=77.90.31.14:4689 , piece_id=Q3HDS3BCMYDTODFRMU4XFRDCQK2QZE3ZMXVLUX4M54PDEBS7RR6A, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:48.221764035 (81h9m23.785087034s) [noise=true , address=104.36.82.54:4165 , piece_id=4CVOABPCWC7J7SRHZB6IN2WJJ6KGWUQ66ELUIZIFA67ETLQ7JEUQ, seg_pos={0 0}, upload=64958]: Done uploading piece. err: ecclient: upload failed (node:12wGUMXVEubEtki4qGnjGEsbYFB1iUauZcu3oDRKrCQsJMV3bV2, address:104.36.82.54:4165): protocol: storage node overloaded, request limit: 2000 Jun 22 01:32:48.509996527 (81h9m24.073319547s) [noise=true , address=104.36.82.9:6023 , piece_id=SISFAUU4RPF57OLGWM2BWQWE6GRUSCMORBYHWEXETNAVM2EEMO6Q, seg_pos={0 0}, upload=64958]: Done uploading piece. err: ecclient: upload failed (node:1j2S9tgWpLwmyvXZSq2qKKcbaC8r7LkzfDCaMxCejAni4CST1g, address:104.36.82.9:6023): protocol: storage node overloaded, request limit: 2000 Jun 22 01:32:48.510513169 (81h9m24.073836177s) [noise=true , address=104.36.82.42:6811 , piece_id=A5BFEHZ3WGGK53IZ7MKI6N4CVXHOORF6L47NR3UFDCAX7LBHRHVQ, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:48.565245605 (81h9m24.128568606s) [noise=true , address=79.127.205.234:5601 , piece_id=V7PMXBP2VFWBSZDG4GTLBD435T6F5DT7A43L4GQLQ5BDDTJ4WY6A, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:49.979979824 (81h9m25.543302825s) [noise=true , address=104.36.82.15:6289 , piece_id=GWL3P55BIK4IJEL5YFZKI7JCA4LADYOOHUSIQUNGJEKBJOTKMP5A, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:50.463750280 (81h9m26.02707328s) [noise=true , address=104.36.82.51:4084 , piece_id=D43FA7BFE54KPCWGDGDME2EGSWM7PONOVPD3DN2XY42MJFOKVDIQ, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:51.480599795 (81h9m27.043922798s) [noise=true , address=79.127.219.38:4023 , piece_id=BWLRCXXHBMBTNCZEVKKIIS5PB35PDQXQXCLIQB7GBLEIOWWUFBFA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:52.941200625 (81h9m28.504523627s) [noise=true , address=104.36.82.43:6841 , piece_id=PIHZMSTO3CII7KGDLGSAILTDRSLMEBVOLBCRCZJULMCFISXIZWXA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:58.281803598 (81h9m33.845126612s) [noise=true , address=64.49.13.139:4476 , piece_id=P7UKBNNZGI52B3I6WKTV6MVC262FYNKYYPJUI5G47UO5S4MU7PWA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:33:00.277271877 (81h9m35.840594867s) [noise=true , address=104.36.82.55:4213 , piece_id=X4IHS2XSPH52LIMNUYT7KGWMAPUH3TEMJOEEVB3JV5N45SEOH7YA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:33:09.435339213 (81h9m44.998662206s) [noise=true , address=104.36.82.52:4089 , piece_id=JSOEKLNWA3W3LOVKI3EGGKJYJ6J3UZY6KGABKQPQPXTIOOPMKVHA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:33:31.472868445 (81h10m7.03619143s) [noise=true , address=104.36.82.53:4122 , piece_id=HZZQBIKHQWLUE5EHK4KSKJGIUEG7D44RQASUJHCYXX6BICCGX6EA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:33:47.235404653 (81h10m22.798727642s) [noise=true , address=104.36.82.12:6158 , piece_id=LJZEOJVJCJAR2BRILIJI3ZVDGYMPBBTP2G3HCTN3MHNS4KYDPASQ, seg_pos={0 0}, upload=64958]: Done uploading piece. err: ecclient: upload failed (node:1SjM6Utm2YizeUbrBUD9Z7etE1nLadW5Fa1zYP7MioPeeqhdJ7, address:104.36.82.12:6158): protocol: expected piece hash; context deadline exceeded; EOF Jun 22 01:33:47.235448297 (81h10m22.798771306s) [seg_pos={0 0}, upload=64958]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jun 22 01:33:47.265442393 (81h10m22.828765401s) [upload=64958]: Done uploading segments... Jun 22 01:33:47.265476139 (81h10m22.82879914s) [upload=64958]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration:* 14.854924503s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [....................................................X.......] duration: 15.838811539s FAIL (2025-06-22 01:49:48 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 15.838831514s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.....................................X......................] duration: 21.921295987s FAIL (2025-06-22 01:49:32 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== oad=64949]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jun 22 01:32:13.674692303 (81h8m49.238015311s) [upload=64949]: Done uploading segments... Jun 22 01:32:13.674727219 (81h8m49.238050233s) [upload=64949]: Upload finished. err: <nil> Jun 22 01:32:47.266407995 (81h9m22.829731002s) [upload=64958]: Starting upload Jun 22 01:32:47.266524767 (81h9m22.829847771s) [upload=64958]: Done starting upload Jun 22 01:32:47.266813109 (81h9m22.830136116s) [upload=64958]: Uploading segments... Jun 22 01:32:47.266969526 (81h9m22.830292526s) [upload=64958]: Got next segment. Inline: false Jun 22 01:32:47.267075475 (81h9m22.830398475s) [upload=64958]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 22 01:32:47.267081679 (81h9m22.830404667s) [upload=64958]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 22 01:32:47.301770382 (81h9m22.865093389s) [seg_pos={0 0}, upload=64958]: Begin upload segment... Jun 22 01:32:47.301823796 (81h9m22.865146799s) [seg_pos={0 0}, upload=64958]: Done begin upload segment. Jun 22 01:32:47.301855055 (81h9m22.865178087s) [noise=true , address=104.36.82.43:6841 , piece_id=PIHZMSTO3CII7KGDLGSAILTDRSLMEBVOLBCRCZJULMCFISXIZWXA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.301902600 (81h9m22.865225606s) [noise=true , address=104.36.82.42:6811 , piece_id=A5BFEHZ3WGGK53IZ7MKI6N4CVXHOORF6L47NR3UFDCAX7LBHRHVQ, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.302234027 (81h9m22.865557029s) [noise=true , address=104.36.82.51:4084 , piece_id=D43FA7BFE54KPCWGDGDME2EGSWM7PONOVPD3DN2XY42MJFOKVDIQ, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.302496354 (81h9m22.865819363s) [noise=true , address=79.127.219.38:4023 , piece_id=BWLRCXXHBMBTNCZEVKKIIS5PB35PDQXQXCLIQB7GBLEIOWWUFBFA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.302577370 (81h9m22.865900378s) [noise=true , address=104.36.82.52:4089 , piece_id=JSOEKLNWA3W3LOVKI3EGGKJYJ6J3UZY6KGABKQPQPXTIOOPMKVHA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.302917829 (81h9m22.866240835s) [noise=true , address=104.36.82.53:4122 , piece_id=HZZQBIKHQWLUE5EHK4KSKJGIUEG7D44RQASUJHCYXX6BICCGX6EA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.303087165 (81h9m22.866410168s) [noise=true , address=77.90.31.14:4689 , piece_id=Q3HDS3BCMYDTODFRMU4XFRDCQK2QZE3ZMXVLUX4M54PDEBS7RR6A, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.303250714 (81h9m22.866573717s) [noise=true , address=79.127.205.234:5601 , piece_id=V7PMXBP2VFWBSZDG4GTLBD435T6F5DT7A43L4GQLQ5BDDTJ4WY6A, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.303579249 (81h9m22.866902255s) [noise=true , address=104.36.82.12:6158 , piece_id=LJZEOJVJCJAR2BRILIJI3ZVDGYMPBBTP2G3HCTN3MHNS4KYDPASQ, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.303599776 (81h9m22.866922784s) [noise=true , address=104.36.82.9:6023 , piece_id=SISFAUU4RPF57OLGWM2BWQWE6GRUSCMORBYHWEXETNAVM2EEMO6Q, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.304005346 (81h9m22.867328346s) [noise=true , address=45.91.116.12:5242 , piece_id=6ABT5IPYGSLFFJMBYMBYRU4EYIXABUAUHG3OVILOHBIUDS3BP27A, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.304388850 (81h9m22.867711851s) [noise=true , address=64.49.13.137:4500 , piece_id=FAE2RXD57IRP5FXA2XR6THXFW27U7IWAZ4JDV5M2WRPXKUGZ4VIA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.304881529 (81h9m22.868204536s) [noise=true , address=79.127.220.101:6048 , piece_id=AN7UQUXB42T2YXEH7ZJRZUDIW47M3RWUEOCL2VEBOGH74BZF3YKA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.305249860 (81h9m22.86857287s) [noise=true , address=104.36.82.54:4165 , piece_id=4CVOABPCWC7J7SRHZB6IN2WJJ6KGWUQ66ELUIZIFA67ETLQ7JEUQ, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.305604108 (81h9m22.868927112s) [noise=true , address=104.36.82.15:6289 , piece_id=GWL3P55BIK4IJEL5YFZKI7JCA4LADYOOHUSIQUNGJEKBJOTKMP5A, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.305930423 (81h9m22.869253425s) [noise=true , address=64.49.13.139:4476 , piece_id=P7UKBNNZGI52B3I6WKTV6MVC262FYNKYYPJUI5G47UO5S4MU7PWA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.306355311 (81h9m22.869678319s) [noise=true , address=104.36.82.55:4213 , piece_id=X4IHS2XSPH52LIMNUYT7KGWMAPUH3TEMJOEEVB3JV5N45SEOH7YA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.306788243 (81h9m22.870111252s) [noise=true , address=104.36.82.22:6539 , piece_id=GHQ2MVSTDGVYLUKLV23D6YWRR4PP7UUYHCUEQ4WWSWEMTBFO3P4A, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.392174770 (81h9m22.955497773s) [upload=64958]: Next returned nil segment Jun 22 01:32:47.392195767 (81h9m22.955518775s) [upload=64958]: Waiting for error group managing segments... Jun 22 01:32:47.480767633 (81h9m23.044090623s) [noise=true , address=104.36.82.22:6539 , piece_id=GHQ2MVSTDGVYLUKLV23D6YWRR4PP7UUYHCUEQ4WWSWEMTBFO3P4A, seg_pos={0 0}, upload=64958]: Done uploading piece. err: ecclient: upload failed (node:1QL6w8DSFR6CxDdnJmDuyVZxp6JU55VKiJofBZRfQi7q31i8Jc, address:104.36.82.22:6539): protocol: storage node overloaded, request limit: 2000 Jun 22 01:32:47.585663142 (81h9m23.148986155s) [noise=true , address=79.127.220.101:6048 , piece_id=AN7UQUXB42T2YXEH7ZJRZUDIW47M3RWUEOCL2VEBOGH74BZF3YKA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:47.640263002 (81h9m23.203586013s) [noise=true , address=45.91.116.12:5242 , piece_id=6ABT5IPYGSLFFJMBYMBYRU4EYIXABUAUHG3OVILOHBIUDS3BP27A, seg_pos={0 0}, upload=64958]: Done uploading piece. err: ecclient: upload failed (node:12JsFL62jgXmza8Md8eQ9k8cyw5V9Vz2RzN4y85ZvgHZjwrq4vQ, address:45.91.116.12:5242): protocol: storage node overloaded, request limit: 2000 Jun 22 01:32:47.984127275 (81h9m23.54745028s) [noise=true , address=64.49.13.137:4500 , piece_id=FAE2RXD57IRP5FXA2XR6THXFW27U7IWAZ4JDV5M2WRPXKUGZ4VIA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:48.100255249 (81h9m23.663578241s) [noise=true , address=77.90.31.14:4689 , piece_id=Q3HDS3BCMYDTODFRMU4XFRDCQK2QZE3ZMXVLUX4M54PDEBS7RR6A, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:48.221764035 (81h9m23.785087034s) [noise=true , address=104.36.82.54:4165 , piece_id=4CVOABPCWC7J7SRHZB6IN2WJJ6KGWUQ66ELUIZIFA67ETLQ7JEUQ, seg_pos={0 0}, upload=64958]: Done uploading piece. err: ecclient: upload failed (node:12wGUMXVEubEtki4qGnjGEsbYFB1iUauZcu3oDRKrCQsJMV3bV2, address:104.36.82.54:4165): protocol: storage node overloaded, request limit: 2000 Jun 22 01:32:48.509996527 (81h9m24.073319547s) [noise=true , address=104.36.82.9:6023 , piece_id=SISFAUU4RPF57OLGWM2BWQWE6GRUSCMORBYHWEXETNAVM2EEMO6Q, seg_pos={0 0}, upload=64958]: Done uploading piece. err: ecclient: upload failed (node:1j2S9tgWpLwmyvXZSq2qKKcbaC8r7LkzfDCaMxCejAni4CST1g, address:104.36.82.9:6023): protocol: storage node overloaded, request limit: 2000 Jun 22 01:32:48.510513169 (81h9m24.073836177s) [noise=true , address=104.36.82.42:6811 , piece_id=A5BFEHZ3WGGK53IZ7MKI6N4CVXHOORF6L47NR3UFDCAX7LBHRHVQ, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:48.565245605 (81h9m24.128568606s) [noise=true , address=79.127.205.234:5601 , piece_id=V7PMXBP2VFWBSZDG4GTLBD435T6F5DT7A43L4GQLQ5BDDTJ4WY6A, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:49.979979824 (81h9m25.543302825s) [noise=true , address=104.36.82.15:6289 , piece_id=GWL3P55BIK4IJEL5YFZKI7JCA4LADYOOHUSIQUNGJEKBJOTKMP5A, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:50.463750280 (81h9m26.02707328s) [noise=true , address=104.36.82.51:4084 , piece_id=D43FA7BFE54KPCWGDGDME2EGSWM7PONOVPD3DN2XY42MJFOKVDIQ, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:51.480599795 (81h9m27.043922798s) [noise=true , address=79.127.219.38:4023 , piece_id=BWLRCXXHBMBTNCZEVKKIIS5PB35PDQXQXCLIQB7GBLEIOWWUFBFA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:52.941200625 (81h9m28.504523627s) [noise=true , address=104.36.82.43:6841 , piece_id=PIHZMSTO3CII7KGDLGSAILTDRSLMEBVOLBCRCZJULMCFISXIZWXA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:58.281803598 (81h9m33.845126612s) [noise=true , address=64.49.13.139:4476 , piece_id=P7UKBNNZGI52B3I6WKTV6MVC262FYNKYYPJUI5G47UO5S4MU7PWA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:33:00.277271877 (81h9m35.840594867s) [noise=true , address=104.36.82.55:4213 , piece_id=X4IHS2XSPH52LIMNUYT7KGWMAPUH3TEMJOEEVB3JV5N45SEOH7YA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:33:09.435339213 (81h9m44.998662206s) [noise=true , address=104.36.82.52:4089 , piece_id=JSOEKLNWA3W3LOVKI3EGGKJYJ6J3UZY6KGABKQPQPXTIOOPMKVHA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:33:31.472868445 (81h10m7.03619143s) [noise=true , address=104.36.82.53:4122 , piece_id=HZZQBIKHQWLUE5EHK4KSKJGIUEG7D44RQASUJHCYXX6BICCGX6EA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:33:47.235404653 (81h10m22.798727642s) [noise=true , address=104.36.82.12:6158 , piece_id=LJZEOJVJCJAR2BRILIJI3ZVDGYMPBBTP2G3HCTN3MHNS4KYDPASQ, seg_pos={0 0}, upload=64958]: Done uploading piece. err: ecclient: upload failed (node:1SjM6Utm2YizeUbrBUD9Z7etE1nLadW5Fa1zYP7MioPeeqhdJ7, address:104.36.82.12:6158): protocol: expected piece hash; context deadline exceeded; EOF Jun 22 01:33:47.235448297 (81h10m22.798771306s) [seg_pos={0 0}, upload=64958]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jun 22 01:33:47.265442393 (81h10m22.828765401s) [upload=64958]: Done uploading segments... Jun 22 01:33:47.265476139 (81h10m22.82879914s) [upload=64958]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration:* 21.921334981s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.....................................................X......] duration: 21.882394132s FAIL (2025-06-22 01:49:25 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 21.882430029s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [......................................X.....................] duration: 18.770519674s FAIL (2025-06-22 01:48:39 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== oad=64949]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jun 22 01:32:13.674692303 (81h8m49.238015311s) [upload=64949]: Done uploading segments... Jun 22 01:32:13.674727219 (81h8m49.238050233s) [upload=64949]: Upload finished. err: <nil> Jun 22 01:32:47.266407995 (81h9m22.829731002s) [upload=64958]: Starting upload Jun 22 01:32:47.266524767 (81h9m22.829847771s) [upload=64958]: Done starting upload Jun 22 01:32:47.266813109 (81h9m22.830136116s) [upload=64958]: Uploading segments... Jun 22 01:32:47.266969526 (81h9m22.830292526s) [upload=64958]: Got next segment. Inline: false Jun 22 01:32:47.267075475 (81h9m22.830398475s) [upload=64958]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 22 01:32:47.267081679 (81h9m22.830404667s) [upload=64958]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 22 01:32:47.301770382 (81h9m22.865093389s) [seg_pos={0 0}, upload=64958]: Begin upload segment... Jun 22 01:32:47.301823796 (81h9m22.865146799s) [seg_pos={0 0}, upload=64958]: Done begin upload segment. Jun 22 01:32:47.301855055 (81h9m22.865178087s) [noise=true , address=104.36.82.43:6841 , piece_id=PIHZMSTO3CII7KGDLGSAILTDRSLMEBVOLBCRCZJULMCFISXIZWXA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.301902600 (81h9m22.865225606s) [noise=true , address=104.36.82.42:6811 , piece_id=A5BFEHZ3WGGK53IZ7MKI6N4CVXHOORF6L47NR3UFDCAX7LBHRHVQ, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.302234027 (81h9m22.865557029s) [noise=true , address=104.36.82.51:4084 , piece_id=D43FA7BFE54KPCWGDGDME2EGSWM7PONOVPD3DN2XY42MJFOKVDIQ, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.302496354 (81h9m22.865819363s) [noise=true , address=79.127.219.38:4023 , piece_id=BWLRCXXHBMBTNCZEVKKIIS5PB35PDQXQXCLIQB7GBLEIOWWUFBFA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.302577370 (81h9m22.865900378s) [noise=true , address=104.36.82.52:4089 , piece_id=JSOEKLNWA3W3LOVKI3EGGKJYJ6J3UZY6KGABKQPQPXTIOOPMKVHA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.302917829 (81h9m22.866240835s) [noise=true , address=104.36.82.53:4122 , piece_id=HZZQBIKHQWLUE5EHK4KSKJGIUEG7D44RQASUJHCYXX6BICCGX6EA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.303087165 (81h9m22.866410168s) [noise=true , address=77.90.31.14:4689 , piece_id=Q3HDS3BCMYDTODFRMU4XFRDCQK2QZE3ZMXVLUX4M54PDEBS7RR6A, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.303250714 (81h9m22.866573717s) [noise=true , address=79.127.205.234:5601 , piece_id=V7PMXBP2VFWBSZDG4GTLBD435T6F5DT7A43L4GQLQ5BDDTJ4WY6A, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.303579249 (81h9m22.866902255s) [noise=true , address=104.36.82.12:6158 , piece_id=LJZEOJVJCJAR2BRILIJI3ZVDGYMPBBTP2G3HCTN3MHNS4KYDPASQ, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.303599776 (81h9m22.866922784s) [noise=true , address=104.36.82.9:6023 , piece_id=SISFAUU4RPF57OLGWM2BWQWE6GRUSCMORBYHWEXETNAVM2EEMO6Q, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.304005346 (81h9m22.867328346s) [noise=true , address=45.91.116.12:5242 , piece_id=6ABT5IPYGSLFFJMBYMBYRU4EYIXABUAUHG3OVILOHBIUDS3BP27A, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.304388850 (81h9m22.867711851s) [noise=true , address=64.49.13.137:4500 , piece_id=FAE2RXD57IRP5FXA2XR6THXFW27U7IWAZ4JDV5M2WRPXKUGZ4VIA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.304881529 (81h9m22.868204536s) [noise=true , address=79.127.220.101:6048 , piece_id=AN7UQUXB42T2YXEH7ZJRZUDIW47M3RWUEOCL2VEBOGH74BZF3YKA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.305249860 (81h9m22.86857287s) [noise=true , address=104.36.82.54:4165 , piece_id=4CVOABPCWC7J7SRHZB6IN2WJJ6KGWUQ66ELUIZIFA67ETLQ7JEUQ, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.305604108 (81h9m22.868927112s) [noise=true , address=104.36.82.15:6289 , piece_id=GWL3P55BIK4IJEL5YFZKI7JCA4LADYOOHUSIQUNGJEKBJOTKMP5A, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.305930423 (81h9m22.869253425s) [noise=true , address=64.49.13.139:4476 , piece_id=P7UKBNNZGI52B3I6WKTV6MVC262FYNKYYPJUI5G47UO5S4MU7PWA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.306355311 (81h9m22.869678319s) [noise=true , address=104.36.82.55:4213 , piece_id=X4IHS2XSPH52LIMNUYT7KGWMAPUH3TEMJOEEVB3JV5N45SEOH7YA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.306788243 (81h9m22.870111252s) [noise=true , address=104.36.82.22:6539 , piece_id=GHQ2MVSTDGVYLUKLV23D6YWRR4PP7UUYHCUEQ4WWSWEMTBFO3P4A, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.392174770 (81h9m22.955497773s) [upload=64958]: Next returned nil segment Jun 22 01:32:47.392195767 (81h9m22.955518775s) [upload=64958]: Waiting for error group managing segments... Jun 22 01:32:47.480767633 (81h9m23.044090623s) [noise=true , address=104.36.82.22:6539 , piece_id=GHQ2MVSTDGVYLUKLV23D6YWRR4PP7UUYHCUEQ4WWSWEMTBFO3P4A, seg_pos={0 0}, upload=64958]: Done uploading piece. err: ecclient: upload failed (node:1QL6w8DSFR6CxDdnJmDuyVZxp6JU55VKiJofBZRfQi7q31i8Jc, address:104.36.82.22:6539): protocol: storage node overloaded, request limit: 2000 Jun 22 01:32:47.585663142 (81h9m23.148986155s) [noise=true , address=79.127.220.101:6048 , piece_id=AN7UQUXB42T2YXEH7ZJRZUDIW47M3RWUEOCL2VEBOGH74BZF3YKA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:47.640263002 (81h9m23.203586013s) [noise=true , address=45.91.116.12:5242 , piece_id=6ABT5IPYGSLFFJMBYMBYRU4EYIXABUAUHG3OVILOHBIUDS3BP27A, seg_pos={0 0}, upload=64958]: Done uploading piece. err: ecclient: upload failed (node:12JsFL62jgXmza8Md8eQ9k8cyw5V9Vz2RzN4y85ZvgHZjwrq4vQ, address:45.91.116.12:5242): protocol: storage node overloaded, request limit: 2000 Jun 22 01:32:47.984127275 (81h9m23.54745028s) [noise=true , address=64.49.13.137:4500 , piece_id=FAE2RXD57IRP5FXA2XR6THXFW27U7IWAZ4JDV5M2WRPXKUGZ4VIA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:48.100255249 (81h9m23.663578241s) [noise=true , address=77.90.31.14:4689 , piece_id=Q3HDS3BCMYDTODFRMU4XFRDCQK2QZE3ZMXVLUX4M54PDEBS7RR6A, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:48.221764035 (81h9m23.785087034s) [noise=true , address=104.36.82.54:4165 , piece_id=4CVOABPCWC7J7SRHZB6IN2WJJ6KGWUQ66ELUIZIFA67ETLQ7JEUQ, seg_pos={0 0}, upload=64958]: Done uploading piece. err: ecclient: upload failed (node:12wGUMXVEubEtki4qGnjGEsbYFB1iUauZcu3oDRKrCQsJMV3bV2, address:104.36.82.54:4165): protocol: storage node overloaded, request limit: 2000 Jun 22 01:32:48.509996527 (81h9m24.073319547s) [noise=true , address=104.36.82.9:6023 , piece_id=SISFAUU4RPF57OLGWM2BWQWE6GRUSCMORBYHWEXETNAVM2EEMO6Q, seg_pos={0 0}, upload=64958]: Done uploading piece. err: ecclient: upload failed (node:1j2S9tgWpLwmyvXZSq2qKKcbaC8r7LkzfDCaMxCejAni4CST1g, address:104.36.82.9:6023): protocol: storage node overloaded, request limit: 2000 Jun 22 01:32:48.510513169 (81h9m24.073836177s) [noise=true , address=104.36.82.42:6811 , piece_id=A5BFEHZ3WGGK53IZ7MKI6N4CVXHOORF6L47NR3UFDCAX7LBHRHVQ, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:48.565245605 (81h9m24.128568606s) [noise=true , address=79.127.205.234:5601 , piece_id=V7PMXBP2VFWBSZDG4GTLBD435T6F5DT7A43L4GQLQ5BDDTJ4WY6A, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:49.979979824 (81h9m25.543302825s) [noise=true , address=104.36.82.15:6289 , piece_id=GWL3P55BIK4IJEL5YFZKI7JCA4LADYOOHUSIQUNGJEKBJOTKMP5A, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:50.463750280 (81h9m26.02707328s) [noise=true , address=104.36.82.51:4084 , piece_id=D43FA7BFE54KPCWGDGDME2EGSWM7PONOVPD3DN2XY42MJFOKVDIQ, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:51.480599795 (81h9m27.043922798s) [noise=true , address=79.127.219.38:4023 , piece_id=BWLRCXXHBMBTNCZEVKKIIS5PB35PDQXQXCLIQB7GBLEIOWWUFBFA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:52.941200625 (81h9m28.504523627s) [noise=true , address=104.36.82.43:6841 , piece_id=PIHZMSTO3CII7KGDLGSAILTDRSLMEBVOLBCRCZJULMCFISXIZWXA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:58.281803598 (81h9m33.845126612s) [noise=true , address=64.49.13.139:4476 , piece_id=P7UKBNNZGI52B3I6WKTV6MVC262FYNKYYPJUI5G47UO5S4MU7PWA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:33:00.277271877 (81h9m35.840594867s) [noise=true , address=104.36.82.55:4213 , piece_id=X4IHS2XSPH52LIMNUYT7KGWMAPUH3TEMJOEEVB3JV5N45SEOH7YA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:33:09.435339213 (81h9m44.998662206s) [noise=true , address=104.36.82.52:4089 , piece_id=JSOEKLNWA3W3LOVKI3EGGKJYJ6J3UZY6KGABKQPQPXTIOOPMKVHA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:33:31.472868445 (81h10m7.03619143s) [noise=true , address=104.36.82.53:4122 , piece_id=HZZQBIKHQWLUE5EHK4KSKJGIUEG7D44RQASUJHCYXX6BICCGX6EA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:33:47.235404653 (81h10m22.798727642s) [noise=true , address=104.36.82.12:6158 , piece_id=LJZEOJVJCJAR2BRILIJI3ZVDGYMPBBTP2G3HCTN3MHNS4KYDPASQ, seg_pos={0 0}, upload=64958]: Done uploading piece. err: ecclient: upload failed (node:1SjM6Utm2YizeUbrBUD9Z7etE1nLadW5Fa1zYP7MioPeeqhdJ7, address:104.36.82.12:6158): protocol: expected piece hash; context deadline exceeded; EOF Jun 22 01:33:47.235448297 (81h10m22.798771306s) [seg_pos={0 0}, upload=64958]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jun 22 01:33:47.265442393 (81h10m22.828765401s) [upload=64958]: Done uploading segments... Jun 22 01:33:47.265476139 (81h10m22.82879914s) [upload=64958]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration:* 18.770564878s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [......................................................X.....] duration: 25.811115944s FAIL (2025-06-22 01:48:19 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 25.811152273s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.......................................X....................] duration: 33.782852375s FAIL (2025-06-22 01:47:43 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== oad=64949]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jun 22 01:32:13.674692303 (81h8m49.238015311s) [upload=64949]: Done uploading segments... Jun 22 01:32:13.674727219 (81h8m49.238050233s) [upload=64949]: Upload finished. err: <nil> Jun 22 01:32:47.266407995 (81h9m22.829731002s) [upload=64958]: Starting upload Jun 22 01:32:47.266524767 (81h9m22.829847771s) [upload=64958]: Done starting upload Jun 22 01:32:47.266813109 (81h9m22.830136116s) [upload=64958]: Uploading segments... Jun 22 01:32:47.266969526 (81h9m22.830292526s) [upload=64958]: Got next segment. Inline: false Jun 22 01:32:47.267075475 (81h9m22.830398475s) [upload=64958]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 22 01:32:47.267081679 (81h9m22.830404667s) [upload=64958]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 22 01:32:47.301770382 (81h9m22.865093389s) [seg_pos={0 0}, upload=64958]: Begin upload segment... Jun 22 01:32:47.301823796 (81h9m22.865146799s) [seg_pos={0 0}, upload=64958]: Done begin upload segment. Jun 22 01:32:47.301855055 (81h9m22.865178087s) [noise=true , address=104.36.82.43:6841 , piece_id=PIHZMSTO3CII7KGDLGSAILTDRSLMEBVOLBCRCZJULMCFISXIZWXA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.301902600 (81h9m22.865225606s) [noise=true , address=104.36.82.42:6811 , piece_id=A5BFEHZ3WGGK53IZ7MKI6N4CVXHOORF6L47NR3UFDCAX7LBHRHVQ, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.302234027 (81h9m22.865557029s) [noise=true , address=104.36.82.51:4084 , piece_id=D43FA7BFE54KPCWGDGDME2EGSWM7PONOVPD3DN2XY42MJFOKVDIQ, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.302496354 (81h9m22.865819363s) [noise=true , address=79.127.219.38:4023 , piece_id=BWLRCXXHBMBTNCZEVKKIIS5PB35PDQXQXCLIQB7GBLEIOWWUFBFA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.302577370 (81h9m22.865900378s) [noise=true , address=104.36.82.52:4089 , piece_id=JSOEKLNWA3W3LOVKI3EGGKJYJ6J3UZY6KGABKQPQPXTIOOPMKVHA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.302917829 (81h9m22.866240835s) [noise=true , address=104.36.82.53:4122 , piece_id=HZZQBIKHQWLUE5EHK4KSKJGIUEG7D44RQASUJHCYXX6BICCGX6EA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.303087165 (81h9m22.866410168s) [noise=true , address=77.90.31.14:4689 , piece_id=Q3HDS3BCMYDTODFRMU4XFRDCQK2QZE3ZMXVLUX4M54PDEBS7RR6A, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.303250714 (81h9m22.866573717s) [noise=true , address=79.127.205.234:5601 , piece_id=V7PMXBP2VFWBSZDG4GTLBD435T6F5DT7A43L4GQLQ5BDDTJ4WY6A, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.303579249 (81h9m22.866902255s) [noise=true , address=104.36.82.12:6158 , piece_id=LJZEOJVJCJAR2BRILIJI3ZVDGYMPBBTP2G3HCTN3MHNS4KYDPASQ, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.303599776 (81h9m22.866922784s) [noise=true , address=104.36.82.9:6023 , piece_id=SISFAUU4RPF57OLGWM2BWQWE6GRUSCMORBYHWEXETNAVM2EEMO6Q, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.304005346 (81h9m22.867328346s) [noise=true , address=45.91.116.12:5242 , piece_id=6ABT5IPYGSLFFJMBYMBYRU4EYIXABUAUHG3OVILOHBIUDS3BP27A, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.304388850 (81h9m22.867711851s) [noise=true , address=64.49.13.137:4500 , piece_id=FAE2RXD57IRP5FXA2XR6THXFW27U7IWAZ4JDV5M2WRPXKUGZ4VIA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.304881529 (81h9m22.868204536s) [noise=true , address=79.127.220.101:6048 , piece_id=AN7UQUXB42T2YXEH7ZJRZUDIW47M3RWUEOCL2VEBOGH74BZF3YKA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.305249860 (81h9m22.86857287s) [noise=true , address=104.36.82.54:4165 , piece_id=4CVOABPCWC7J7SRHZB6IN2WJJ6KGWUQ66ELUIZIFA67ETLQ7JEUQ, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.305604108 (81h9m22.868927112s) [noise=true , address=104.36.82.15:6289 , piece_id=GWL3P55BIK4IJEL5YFZKI7JCA4LADYOOHUSIQUNGJEKBJOTKMP5A, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.305930423 (81h9m22.869253425s) [noise=true , address=64.49.13.139:4476 , piece_id=P7UKBNNZGI52B3I6WKTV6MVC262FYNKYYPJUI5G47UO5S4MU7PWA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.306355311 (81h9m22.869678319s) [noise=true , address=104.36.82.55:4213 , piece_id=X4IHS2XSPH52LIMNUYT7KGWMAPUH3TEMJOEEVB3JV5N45SEOH7YA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.306788243 (81h9m22.870111252s) [noise=true , address=104.36.82.22:6539 , piece_id=GHQ2MVSTDGVYLUKLV23D6YWRR4PP7UUYHCUEQ4WWSWEMTBFO3P4A, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.392174770 (81h9m22.955497773s) [upload=64958]: Next returned nil segment Jun 22 01:32:47.392195767 (81h9m22.955518775s) [upload=64958]: Waiting for error group managing segments... Jun 22 01:32:47.480767633 (81h9m23.044090623s) [noise=true , address=104.36.82.22:6539 , piece_id=GHQ2MVSTDGVYLUKLV23D6YWRR4PP7UUYHCUEQ4WWSWEMTBFO3P4A, seg_pos={0 0}, upload=64958]: Done uploading piece. err: ecclient: upload failed (node:1QL6w8DSFR6CxDdnJmDuyVZxp6JU55VKiJofBZRfQi7q31i8Jc, address:104.36.82.22:6539): protocol: storage node overloaded, request limit: 2000 Jun 22 01:32:47.585663142 (81h9m23.148986155s) [noise=true , address=79.127.220.101:6048 , piece_id=AN7UQUXB42T2YXEH7ZJRZUDIW47M3RWUEOCL2VEBOGH74BZF3YKA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:47.640263002 (81h9m23.203586013s) [noise=true , address=45.91.116.12:5242 , piece_id=6ABT5IPYGSLFFJMBYMBYRU4EYIXABUAUHG3OVILOHBIUDS3BP27A, seg_pos={0 0}, upload=64958]: Done uploading piece. err: ecclient: upload failed (node:12JsFL62jgXmza8Md8eQ9k8cyw5V9Vz2RzN4y85ZvgHZjwrq4vQ, address:45.91.116.12:5242): protocol: storage node overloaded, request limit: 2000 Jun 22 01:32:47.984127275 (81h9m23.54745028s) [noise=true , address=64.49.13.137:4500 , piece_id=FAE2RXD57IRP5FXA2XR6THXFW27U7IWAZ4JDV5M2WRPXKUGZ4VIA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:48.100255249 (81h9m23.663578241s) [noise=true , address=77.90.31.14:4689 , piece_id=Q3HDS3BCMYDTODFRMU4XFRDCQK2QZE3ZMXVLUX4M54PDEBS7RR6A, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:48.221764035 (81h9m23.785087034s) [noise=true , address=104.36.82.54:4165 , piece_id=4CVOABPCWC7J7SRHZB6IN2WJJ6KGWUQ66ELUIZIFA67ETLQ7JEUQ, seg_pos={0 0}, upload=64958]: Done uploading piece. err: ecclient: upload failed (node:12wGUMXVEubEtki4qGnjGEsbYFB1iUauZcu3oDRKrCQsJMV3bV2, address:104.36.82.54:4165): protocol: storage node overloaded, request limit: 2000 Jun 22 01:32:48.509996527 (81h9m24.073319547s) [noise=true , address=104.36.82.9:6023 , piece_id=SISFAUU4RPF57OLGWM2BWQWE6GRUSCMORBYHWEXETNAVM2EEMO6Q, seg_pos={0 0}, upload=64958]: Done uploading piece. err: ecclient: upload failed (node:1j2S9tgWpLwmyvXZSq2qKKcbaC8r7LkzfDCaMxCejAni4CST1g, address:104.36.82.9:6023): protocol: storage node overloaded, request limit: 2000 Jun 22 01:32:48.510513169 (81h9m24.073836177s) [noise=true , address=104.36.82.42:6811 , piece_id=A5BFEHZ3WGGK53IZ7MKI6N4CVXHOORF6L47NR3UFDCAX7LBHRHVQ, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:48.565245605 (81h9m24.128568606s) [noise=true , address=79.127.205.234:5601 , piece_id=V7PMXBP2VFWBSZDG4GTLBD435T6F5DT7A43L4GQLQ5BDDTJ4WY6A, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:49.979979824 (81h9m25.543302825s) [noise=true , address=104.36.82.15:6289 , piece_id=GWL3P55BIK4IJEL5YFZKI7JCA4LADYOOHUSIQUNGJEKBJOTKMP5A, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:50.463750280 (81h9m26.02707328s) [noise=true , address=104.36.82.51:4084 , piece_id=D43FA7BFE54KPCWGDGDME2EGSWM7PONOVPD3DN2XY42MJFOKVDIQ, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:51.480599795 (81h9m27.043922798s) [noise=true , address=79.127.219.38:4023 , piece_id=BWLRCXXHBMBTNCZEVKKIIS5PB35PDQXQXCLIQB7GBLEIOWWUFBFA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:52.941200625 (81h9m28.504523627s) [noise=true , address=104.36.82.43:6841 , piece_id=PIHZMSTO3CII7KGDLGSAILTDRSLMEBVOLBCRCZJULMCFISXIZWXA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:58.281803598 (81h9m33.845126612s) [noise=true , address=64.49.13.139:4476 , piece_id=P7UKBNNZGI52B3I6WKTV6MVC262FYNKYYPJUI5G47UO5S4MU7PWA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:33:00.277271877 (81h9m35.840594867s) [noise=true , address=104.36.82.55:4213 , piece_id=X4IHS2XSPH52LIMNUYT7KGWMAPUH3TEMJOEEVB3JV5N45SEOH7YA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:33:09.435339213 (81h9m44.998662206s) [noise=true , address=104.36.82.52:4089 , piece_id=JSOEKLNWA3W3LOVKI3EGGKJYJ6J3UZY6KGABKQPQPXTIOOPMKVHA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:33:31.472868445 (81h10m7.03619143s) [noise=true , address=104.36.82.53:4122 , piece_id=HZZQBIKHQWLUE5EHK4KSKJGIUEG7D44RQASUJHCYXX6BICCGX6EA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:33:47.235404653 (81h10m22.798727642s) [noise=true , address=104.36.82.12:6158 , piece_id=LJZEOJVJCJAR2BRILIJI3ZVDGYMPBBTP2G3HCTN3MHNS4KYDPASQ, seg_pos={0 0}, upload=64958]: Done uploading piece. err: ecclient: upload failed (node:1SjM6Utm2YizeUbrBUD9Z7etE1nLadW5Fa1zYP7MioPeeqhdJ7, address:104.36.82.12:6158): protocol: expected piece hash; context deadline exceeded; EOF Jun 22 01:33:47.235448297 (81h10m22.798771306s) [seg_pos={0 0}, upload=64958]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jun 22 01:33:47.265442393 (81h10m22.828765401s) [upload=64958]: Done uploading segments... Jun 22 01:33:47.265476139 (81h10m22.82879914s) [upload=64958]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration:* 33.78289126s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.......................................................X....] duration: 25.595753098s FAIL (2025-06-22 01:47:26 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 25.595796813s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [........................................X...................] duration: 25.246085699s FAIL (2025-06-22 01:47:09 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== oad=64949]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jun 22 01:32:13.674692303 (81h8m49.238015311s) [upload=64949]: Done uploading segments... Jun 22 01:32:13.674727219 (81h8m49.238050233s) [upload=64949]: Upload finished. err: <nil> Jun 22 01:32:47.266407995 (81h9m22.829731002s) [upload=64958]: Starting upload Jun 22 01:32:47.266524767 (81h9m22.829847771s) [upload=64958]: Done starting upload Jun 22 01:32:47.266813109 (81h9m22.830136116s) [upload=64958]: Uploading segments... Jun 22 01:32:47.266969526 (81h9m22.830292526s) [upload=64958]: Got next segment. Inline: false Jun 22 01:32:47.267075475 (81h9m22.830398475s) [upload=64958]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 22 01:32:47.267081679 (81h9m22.830404667s) [upload=64958]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 22 01:32:47.301770382 (81h9m22.865093389s) [seg_pos={0 0}, upload=64958]: Begin upload segment... Jun 22 01:32:47.301823796 (81h9m22.865146799s) [seg_pos={0 0}, upload=64958]: Done begin upload segment. Jun 22 01:32:47.301855055 (81h9m22.865178087s) [noise=true , address=104.36.82.43:6841 , piece_id=PIHZMSTO3CII7KGDLGSAILTDRSLMEBVOLBCRCZJULMCFISXIZWXA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.301902600 (81h9m22.865225606s) [noise=true , address=104.36.82.42:6811 , piece_id=A5BFEHZ3WGGK53IZ7MKI6N4CVXHOORF6L47NR3UFDCAX7LBHRHVQ, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.302234027 (81h9m22.865557029s) [noise=true , address=104.36.82.51:4084 , piece_id=D43FA7BFE54KPCWGDGDME2EGSWM7PONOVPD3DN2XY42MJFOKVDIQ, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.302496354 (81h9m22.865819363s) [noise=true , address=79.127.219.38:4023 , piece_id=BWLRCXXHBMBTNCZEVKKIIS5PB35PDQXQXCLIQB7GBLEIOWWUFBFA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.302577370 (81h9m22.865900378s) [noise=true , address=104.36.82.52:4089 , piece_id=JSOEKLNWA3W3LOVKI3EGGKJYJ6J3UZY6KGABKQPQPXTIOOPMKVHA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.302917829 (81h9m22.866240835s) [noise=true , address=104.36.82.53:4122 , piece_id=HZZQBIKHQWLUE5EHK4KSKJGIUEG7D44RQASUJHCYXX6BICCGX6EA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.303087165 (81h9m22.866410168s) [noise=true , address=77.90.31.14:4689 , piece_id=Q3HDS3BCMYDTODFRMU4XFRDCQK2QZE3ZMXVLUX4M54PDEBS7RR6A, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.303250714 (81h9m22.866573717s) [noise=true , address=79.127.205.234:5601 , piece_id=V7PMXBP2VFWBSZDG4GTLBD435T6F5DT7A43L4GQLQ5BDDTJ4WY6A, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.303579249 (81h9m22.866902255s) [noise=true , address=104.36.82.12:6158 , piece_id=LJZEOJVJCJAR2BRILIJI3ZVDGYMPBBTP2G3HCTN3MHNS4KYDPASQ, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.303599776 (81h9m22.866922784s) [noise=true , address=104.36.82.9:6023 , piece_id=SISFAUU4RPF57OLGWM2BWQWE6GRUSCMORBYHWEXETNAVM2EEMO6Q, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.304005346 (81h9m22.867328346s) [noise=true , address=45.91.116.12:5242 , piece_id=6ABT5IPYGSLFFJMBYMBYRU4EYIXABUAUHG3OVILOHBIUDS3BP27A, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.304388850 (81h9m22.867711851s) [noise=true , address=64.49.13.137:4500 , piece_id=FAE2RXD57IRP5FXA2XR6THXFW27U7IWAZ4JDV5M2WRPXKUGZ4VIA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.304881529 (81h9m22.868204536s) [noise=true , address=79.127.220.101:6048 , piece_id=AN7UQUXB42T2YXEH7ZJRZUDIW47M3RWUEOCL2VEBOGH74BZF3YKA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.305249860 (81h9m22.86857287s) [noise=true , address=104.36.82.54:4165 , piece_id=4CVOABPCWC7J7SRHZB6IN2WJJ6KGWUQ66ELUIZIFA67ETLQ7JEUQ, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.305604108 (81h9m22.868927112s) [noise=true , address=104.36.82.15:6289 , piece_id=GWL3P55BIK4IJEL5YFZKI7JCA4LADYOOHUSIQUNGJEKBJOTKMP5A, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.305930423 (81h9m22.869253425s) [noise=true , address=64.49.13.139:4476 , piece_id=P7UKBNNZGI52B3I6WKTV6MVC262FYNKYYPJUI5G47UO5S4MU7PWA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.306355311 (81h9m22.869678319s) [noise=true , address=104.36.82.55:4213 , piece_id=X4IHS2XSPH52LIMNUYT7KGWMAPUH3TEMJOEEVB3JV5N45SEOH7YA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.306788243 (81h9m22.870111252s) [noise=true , address=104.36.82.22:6539 , piece_id=GHQ2MVSTDGVYLUKLV23D6YWRR4PP7UUYHCUEQ4WWSWEMTBFO3P4A, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.392174770 (81h9m22.955497773s) [upload=64958]: Next returned nil segment Jun 22 01:32:47.392195767 (81h9m22.955518775s) [upload=64958]: Waiting for error group managing segments... Jun 22 01:32:47.480767633 (81h9m23.044090623s) [noise=true , address=104.36.82.22:6539 , piece_id=GHQ2MVSTDGVYLUKLV23D6YWRR4PP7UUYHCUEQ4WWSWEMTBFO3P4A, seg_pos={0 0}, upload=64958]: Done uploading piece. err: ecclient: upload failed (node:1QL6w8DSFR6CxDdnJmDuyVZxp6JU55VKiJofBZRfQi7q31i8Jc, address:104.36.82.22:6539): protocol: storage node overloaded, request limit: 2000 Jun 22 01:32:47.585663142 (81h9m23.148986155s) [noise=true , address=79.127.220.101:6048 , piece_id=AN7UQUXB42T2YXEH7ZJRZUDIW47M3RWUEOCL2VEBOGH74BZF3YKA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:47.640263002 (81h9m23.203586013s) [noise=true , address=45.91.116.12:5242 , piece_id=6ABT5IPYGSLFFJMBYMBYRU4EYIXABUAUHG3OVILOHBIUDS3BP27A, seg_pos={0 0}, upload=64958]: Done uploading piece. err: ecclient: upload failed (node:12JsFL62jgXmza8Md8eQ9k8cyw5V9Vz2RzN4y85ZvgHZjwrq4vQ, address:45.91.116.12:5242): protocol: storage node overloaded, request limit: 2000 Jun 22 01:32:47.984127275 (81h9m23.54745028s) [noise=true , address=64.49.13.137:4500 , piece_id=FAE2RXD57IRP5FXA2XR6THXFW27U7IWAZ4JDV5M2WRPXKUGZ4VIA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:48.100255249 (81h9m23.663578241s) [noise=true , address=77.90.31.14:4689 , piece_id=Q3HDS3BCMYDTODFRMU4XFRDCQK2QZE3ZMXVLUX4M54PDEBS7RR6A, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:48.221764035 (81h9m23.785087034s) [noise=true , address=104.36.82.54:4165 , piece_id=4CVOABPCWC7J7SRHZB6IN2WJJ6KGWUQ66ELUIZIFA67ETLQ7JEUQ, seg_pos={0 0}, upload=64958]: Done uploading piece. err: ecclient: upload failed (node:12wGUMXVEubEtki4qGnjGEsbYFB1iUauZcu3oDRKrCQsJMV3bV2, address:104.36.82.54:4165): protocol: storage node overloaded, request limit: 2000 Jun 22 01:32:48.509996527 (81h9m24.073319547s) [noise=true , address=104.36.82.9:6023 , piece_id=SISFAUU4RPF57OLGWM2BWQWE6GRUSCMORBYHWEXETNAVM2EEMO6Q, seg_pos={0 0}, upload=64958]: Done uploading piece. err: ecclient: upload failed (node:1j2S9tgWpLwmyvXZSq2qKKcbaC8r7LkzfDCaMxCejAni4CST1g, address:104.36.82.9:6023): protocol: storage node overloaded, request limit: 2000 Jun 22 01:32:48.510513169 (81h9m24.073836177s) [noise=true , address=104.36.82.42:6811 , piece_id=A5BFEHZ3WGGK53IZ7MKI6N4CVXHOORF6L47NR3UFDCAX7LBHRHVQ, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:48.565245605 (81h9m24.128568606s) [noise=true , address=79.127.205.234:5601 , piece_id=V7PMXBP2VFWBSZDG4GTLBD435T6F5DT7A43L4GQLQ5BDDTJ4WY6A, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:49.979979824 (81h9m25.543302825s) [noise=true , address=104.36.82.15:6289 , piece_id=GWL3P55BIK4IJEL5YFZKI7JCA4LADYOOHUSIQUNGJEKBJOTKMP5A, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:50.463750280 (81h9m26.02707328s) [noise=true , address=104.36.82.51:4084 , piece_id=D43FA7BFE54KPCWGDGDME2EGSWM7PONOVPD3DN2XY42MJFOKVDIQ, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:51.480599795 (81h9m27.043922798s) [noise=true , address=79.127.219.38:4023 , piece_id=BWLRCXXHBMBTNCZEVKKIIS5PB35PDQXQXCLIQB7GBLEIOWWUFBFA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:52.941200625 (81h9m28.504523627s) [noise=true , address=104.36.82.43:6841 , piece_id=PIHZMSTO3CII7KGDLGSAILTDRSLMEBVOLBCRCZJULMCFISXIZWXA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:58.281803598 (81h9m33.845126612s) [noise=true , address=64.49.13.139:4476 , piece_id=P7UKBNNZGI52B3I6WKTV6MVC262FYNKYYPJUI5G47UO5S4MU7PWA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:33:00.277271877 (81h9m35.840594867s) [noise=true , address=104.36.82.55:4213 , piece_id=X4IHS2XSPH52LIMNUYT7KGWMAPUH3TEMJOEEVB3JV5N45SEOH7YA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:33:09.435339213 (81h9m44.998662206s) [noise=true , address=104.36.82.52:4089 , piece_id=JSOEKLNWA3W3LOVKI3EGGKJYJ6J3UZY6KGABKQPQPXTIOOPMKVHA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:33:31.472868445 (81h10m7.03619143s) [noise=true , address=104.36.82.53:4122 , piece_id=HZZQBIKHQWLUE5EHK4KSKJGIUEG7D44RQASUJHCYXX6BICCGX6EA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:33:47.235404653 (81h10m22.798727642s) [noise=true , address=104.36.82.12:6158 , piece_id=LJZEOJVJCJAR2BRILIJI3ZVDGYMPBBTP2G3HCTN3MHNS4KYDPASQ, seg_pos={0 0}, upload=64958]: Done uploading piece. err: ecclient: upload failed (node:1SjM6Utm2YizeUbrBUD9Z7etE1nLadW5Fa1zYP7MioPeeqhdJ7, address:104.36.82.12:6158): protocol: expected piece hash; context deadline exceeded; EOF Jun 22 01:33:47.235448297 (81h10m22.798771306s) [seg_pos={0 0}, upload=64958]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jun 22 01:33:47.265442393 (81h10m22.828765401s) [upload=64958]: Done uploading segments... Jun 22 01:33:47.265476139 (81h10m22.82879914s) [upload=64958]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration:* 25.246117239s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [........................................................X...] duration: 18.751117954s FAIL (2025-06-22 01:46:34 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 18.751169735s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.........................................X..................] duration: 14.352526291s FAIL (2025-06-22 01:46:28 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== oad=64949]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jun 22 01:32:13.674692303 (81h8m49.238015311s) [upload=64949]: Done uploading segments... Jun 22 01:32:13.674727219 (81h8m49.238050233s) [upload=64949]: Upload finished. err: <nil> Jun 22 01:32:47.266407995 (81h9m22.829731002s) [upload=64958]: Starting upload Jun 22 01:32:47.266524767 (81h9m22.829847771s) [upload=64958]: Done starting upload Jun 22 01:32:47.266813109 (81h9m22.830136116s) [upload=64958]: Uploading segments... Jun 22 01:32:47.266969526 (81h9m22.830292526s) [upload=64958]: Got next segment. Inline: false Jun 22 01:32:47.267075475 (81h9m22.830398475s) [upload=64958]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 22 01:32:47.267081679 (81h9m22.830404667s) [upload=64958]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 22 01:32:47.301770382 (81h9m22.865093389s) [seg_pos={0 0}, upload=64958]: Begin upload segment... Jun 22 01:32:47.301823796 (81h9m22.865146799s) [seg_pos={0 0}, upload=64958]: Done begin upload segment. Jun 22 01:32:47.301855055 (81h9m22.865178087s) [noise=true , address=104.36.82.43:6841 , piece_id=PIHZMSTO3CII7KGDLGSAILTDRSLMEBVOLBCRCZJULMCFISXIZWXA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.301902600 (81h9m22.865225606s) [noise=true , address=104.36.82.42:6811 , piece_id=A5BFEHZ3WGGK53IZ7MKI6N4CVXHOORF6L47NR3UFDCAX7LBHRHVQ, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.302234027 (81h9m22.865557029s) [noise=true , address=104.36.82.51:4084 , piece_id=D43FA7BFE54KPCWGDGDME2EGSWM7PONOVPD3DN2XY42MJFOKVDIQ, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.302496354 (81h9m22.865819363s) [noise=true , address=79.127.219.38:4023 , piece_id=BWLRCXXHBMBTNCZEVKKIIS5PB35PDQXQXCLIQB7GBLEIOWWUFBFA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.302577370 (81h9m22.865900378s) [noise=true , address=104.36.82.52:4089 , piece_id=JSOEKLNWA3W3LOVKI3EGGKJYJ6J3UZY6KGABKQPQPXTIOOPMKVHA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.302917829 (81h9m22.866240835s) [noise=true , address=104.36.82.53:4122 , piece_id=HZZQBIKHQWLUE5EHK4KSKJGIUEG7D44RQASUJHCYXX6BICCGX6EA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.303087165 (81h9m22.866410168s) [noise=true , address=77.90.31.14:4689 , piece_id=Q3HDS3BCMYDTODFRMU4XFRDCQK2QZE3ZMXVLUX4M54PDEBS7RR6A, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.303250714 (81h9m22.866573717s) [noise=true , address=79.127.205.234:5601 , piece_id=V7PMXBP2VFWBSZDG4GTLBD435T6F5DT7A43L4GQLQ5BDDTJ4WY6A, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.303579249 (81h9m22.866902255s) [noise=true , address=104.36.82.12:6158 , piece_id=LJZEOJVJCJAR2BRILIJI3ZVDGYMPBBTP2G3HCTN3MHNS4KYDPASQ, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.303599776 (81h9m22.866922784s) [noise=true , address=104.36.82.9:6023 , piece_id=SISFAUU4RPF57OLGWM2BWQWE6GRUSCMORBYHWEXETNAVM2EEMO6Q, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.304005346 (81h9m22.867328346s) [noise=true , address=45.91.116.12:5242 , piece_id=6ABT5IPYGSLFFJMBYMBYRU4EYIXABUAUHG3OVILOHBIUDS3BP27A, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.304388850 (81h9m22.867711851s) [noise=true , address=64.49.13.137:4500 , piece_id=FAE2RXD57IRP5FXA2XR6THXFW27U7IWAZ4JDV5M2WRPXKUGZ4VIA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.304881529 (81h9m22.868204536s) [noise=true , address=79.127.220.101:6048 , piece_id=AN7UQUXB42T2YXEH7ZJRZUDIW47M3RWUEOCL2VEBOGH74BZF3YKA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.305249860 (81h9m22.86857287s) [noise=true , address=104.36.82.54:4165 , piece_id=4CVOABPCWC7J7SRHZB6IN2WJJ6KGWUQ66ELUIZIFA67ETLQ7JEUQ, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.305604108 (81h9m22.868927112s) [noise=true , address=104.36.82.15:6289 , piece_id=GWL3P55BIK4IJEL5YFZKI7JCA4LADYOOHUSIQUNGJEKBJOTKMP5A, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.305930423 (81h9m22.869253425s) [noise=true , address=64.49.13.139:4476 , piece_id=P7UKBNNZGI52B3I6WKTV6MVC262FYNKYYPJUI5G47UO5S4MU7PWA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.306355311 (81h9m22.869678319s) [noise=true , address=104.36.82.55:4213 , piece_id=X4IHS2XSPH52LIMNUYT7KGWMAPUH3TEMJOEEVB3JV5N45SEOH7YA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.306788243 (81h9m22.870111252s) [noise=true , address=104.36.82.22:6539 , piece_id=GHQ2MVSTDGVYLUKLV23D6YWRR4PP7UUYHCUEQ4WWSWEMTBFO3P4A, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.392174770 (81h9m22.955497773s) [upload=64958]: Next returned nil segment Jun 22 01:32:47.392195767 (81h9m22.955518775s) [upload=64958]: Waiting for error group managing segments... Jun 22 01:32:47.480767633 (81h9m23.044090623s) [noise=true , address=104.36.82.22:6539 , piece_id=GHQ2MVSTDGVYLUKLV23D6YWRR4PP7UUYHCUEQ4WWSWEMTBFO3P4A, seg_pos={0 0}, upload=64958]: Done uploading piece. err: ecclient: upload failed (node:1QL6w8DSFR6CxDdnJmDuyVZxp6JU55VKiJofBZRfQi7q31i8Jc, address:104.36.82.22:6539): protocol: storage node overloaded, request limit: 2000 Jun 22 01:32:47.585663142 (81h9m23.148986155s) [noise=true , address=79.127.220.101:6048 , piece_id=AN7UQUXB42T2YXEH7ZJRZUDIW47M3RWUEOCL2VEBOGH74BZF3YKA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:47.640263002 (81h9m23.203586013s) [noise=true , address=45.91.116.12:5242 , piece_id=6ABT5IPYGSLFFJMBYMBYRU4EYIXABUAUHG3OVILOHBIUDS3BP27A, seg_pos={0 0}, upload=64958]: Done uploading piece. err: ecclient: upload failed (node:12JsFL62jgXmza8Md8eQ9k8cyw5V9Vz2RzN4y85ZvgHZjwrq4vQ, address:45.91.116.12:5242): protocol: storage node overloaded, request limit: 2000 Jun 22 01:32:47.984127275 (81h9m23.54745028s) [noise=true , address=64.49.13.137:4500 , piece_id=FAE2RXD57IRP5FXA2XR6THXFW27U7IWAZ4JDV5M2WRPXKUGZ4VIA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:48.100255249 (81h9m23.663578241s) [noise=true , address=77.90.31.14:4689 , piece_id=Q3HDS3BCMYDTODFRMU4XFRDCQK2QZE3ZMXVLUX4M54PDEBS7RR6A, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:48.221764035 (81h9m23.785087034s) [noise=true , address=104.36.82.54:4165 , piece_id=4CVOABPCWC7J7SRHZB6IN2WJJ6KGWUQ66ELUIZIFA67ETLQ7JEUQ, seg_pos={0 0}, upload=64958]: Done uploading piece. err: ecclient: upload failed (node:12wGUMXVEubEtki4qGnjGEsbYFB1iUauZcu3oDRKrCQsJMV3bV2, address:104.36.82.54:4165): protocol: storage node overloaded, request limit: 2000 Jun 22 01:32:48.509996527 (81h9m24.073319547s) [noise=true , address=104.36.82.9:6023 , piece_id=SISFAUU4RPF57OLGWM2BWQWE6GRUSCMORBYHWEXETNAVM2EEMO6Q, seg_pos={0 0}, upload=64958]: Done uploading piece. err: ecclient: upload failed (node:1j2S9tgWpLwmyvXZSq2qKKcbaC8r7LkzfDCaMxCejAni4CST1g, address:104.36.82.9:6023): protocol: storage node overloaded, request limit: 2000 Jun 22 01:32:48.510513169 (81h9m24.073836177s) [noise=true , address=104.36.82.42:6811 , piece_id=A5BFEHZ3WGGK53IZ7MKI6N4CVXHOORF6L47NR3UFDCAX7LBHRHVQ, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:48.565245605 (81h9m24.128568606s) [noise=true , address=79.127.205.234:5601 , piece_id=V7PMXBP2VFWBSZDG4GTLBD435T6F5DT7A43L4GQLQ5BDDTJ4WY6A, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:49.979979824 (81h9m25.543302825s) [noise=true , address=104.36.82.15:6289 , piece_id=GWL3P55BIK4IJEL5YFZKI7JCA4LADYOOHUSIQUNGJEKBJOTKMP5A, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:50.463750280 (81h9m26.02707328s) [noise=true , address=104.36.82.51:4084 , piece_id=D43FA7BFE54KPCWGDGDME2EGSWM7PONOVPD3DN2XY42MJFOKVDIQ, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:51.480599795 (81h9m27.043922798s) [noise=true , address=79.127.219.38:4023 , piece_id=BWLRCXXHBMBTNCZEVKKIIS5PB35PDQXQXCLIQB7GBLEIOWWUFBFA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:52.941200625 (81h9m28.504523627s) [noise=true , address=104.36.82.43:6841 , piece_id=PIHZMSTO3CII7KGDLGSAILTDRSLMEBVOLBCRCZJULMCFISXIZWXA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:58.281803598 (81h9m33.845126612s) [noise=true , address=64.49.13.139:4476 , piece_id=P7UKBNNZGI52B3I6WKTV6MVC262FYNKYYPJUI5G47UO5S4MU7PWA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:33:00.277271877 (81h9m35.840594867s) [noise=true , address=104.36.82.55:4213 , piece_id=X4IHS2XSPH52LIMNUYT7KGWMAPUH3TEMJOEEVB3JV5N45SEOH7YA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:33:09.435339213 (81h9m44.998662206s) [noise=true , address=104.36.82.52:4089 , piece_id=JSOEKLNWA3W3LOVKI3EGGKJYJ6J3UZY6KGABKQPQPXTIOOPMKVHA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:33:31.472868445 (81h10m7.03619143s) [noise=true , address=104.36.82.53:4122 , piece_id=HZZQBIKHQWLUE5EHK4KSKJGIUEG7D44RQASUJHCYXX6BICCGX6EA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:33:47.235404653 (81h10m22.798727642s) [noise=true , address=104.36.82.12:6158 , piece_id=LJZEOJVJCJAR2BRILIJI3ZVDGYMPBBTP2G3HCTN3MHNS4KYDPASQ, seg_pos={0 0}, upload=64958]: Done uploading piece. err: ecclient: upload failed (node:1SjM6Utm2YizeUbrBUD9Z7etE1nLadW5Fa1zYP7MioPeeqhdJ7, address:104.36.82.12:6158): protocol: expected piece hash; context deadline exceeded; EOF Jun 22 01:33:47.235448297 (81h10m22.798771306s) [seg_pos={0 0}, upload=64958]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jun 22 01:33:47.265442393 (81h10m22.828765401s) [upload=64958]: Done uploading segments... Jun 22 01:33:47.265476139 (81h10m22.82879914s) [upload=64958]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration:* 14.352555769s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.........................................................X..] duration: 15.625552002s FAIL (2025-06-22 01:46:04 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 15.625940325s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [..........................................................X.] duration: 27.836739222s FAIL (2025-06-22 01:45:30 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 27.836772716s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [..........................................X.................] duration: 13.766458314s FAIL (2025-06-22 01:45:19 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== oad=64949]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jun 22 01:32:13.674692303 (81h8m49.238015311s) [upload=64949]: Done uploading segments... Jun 22 01:32:13.674727219 (81h8m49.238050233s) [upload=64949]: Upload finished. err: <nil> Jun 22 01:32:47.266407995 (81h9m22.829731002s) [upload=64958]: Starting upload Jun 22 01:32:47.266524767 (81h9m22.829847771s) [upload=64958]: Done starting upload Jun 22 01:32:47.266813109 (81h9m22.830136116s) [upload=64958]: Uploading segments... Jun 22 01:32:47.266969526 (81h9m22.830292526s) [upload=64958]: Got next segment. Inline: false Jun 22 01:32:47.267075475 (81h9m22.830398475s) [upload=64958]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 22 01:32:47.267081679 (81h9m22.830404667s) [upload=64958]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 22 01:32:47.301770382 (81h9m22.865093389s) [seg_pos={0 0}, upload=64958]: Begin upload segment... Jun 22 01:32:47.301823796 (81h9m22.865146799s) [seg_pos={0 0}, upload=64958]: Done begin upload segment. Jun 22 01:32:47.301855055 (81h9m22.865178087s) [noise=true , address=104.36.82.43:6841 , piece_id=PIHZMSTO3CII7KGDLGSAILTDRSLMEBVOLBCRCZJULMCFISXIZWXA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.301902600 (81h9m22.865225606s) [noise=true , address=104.36.82.42:6811 , piece_id=A5BFEHZ3WGGK53IZ7MKI6N4CVXHOORF6L47NR3UFDCAX7LBHRHVQ, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.302234027 (81h9m22.865557029s) [noise=true , address=104.36.82.51:4084 , piece_id=D43FA7BFE54KPCWGDGDME2EGSWM7PONOVPD3DN2XY42MJFOKVDIQ, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.302496354 (81h9m22.865819363s) [noise=true , address=79.127.219.38:4023 , piece_id=BWLRCXXHBMBTNCZEVKKIIS5PB35PDQXQXCLIQB7GBLEIOWWUFBFA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.302577370 (81h9m22.865900378s) [noise=true , address=104.36.82.52:4089 , piece_id=JSOEKLNWA3W3LOVKI3EGGKJYJ6J3UZY6KGABKQPQPXTIOOPMKVHA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.302917829 (81h9m22.866240835s) [noise=true , address=104.36.82.53:4122 , piece_id=HZZQBIKHQWLUE5EHK4KSKJGIUEG7D44RQASUJHCYXX6BICCGX6EA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.303087165 (81h9m22.866410168s) [noise=true , address=77.90.31.14:4689 , piece_id=Q3HDS3BCMYDTODFRMU4XFRDCQK2QZE3ZMXVLUX4M54PDEBS7RR6A, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.303250714 (81h9m22.866573717s) [noise=true , address=79.127.205.234:5601 , piece_id=V7PMXBP2VFWBSZDG4GTLBD435T6F5DT7A43L4GQLQ5BDDTJ4WY6A, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.303579249 (81h9m22.866902255s) [noise=true , address=104.36.82.12:6158 , piece_id=LJZEOJVJCJAR2BRILIJI3ZVDGYMPBBTP2G3HCTN3MHNS4KYDPASQ, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.303599776 (81h9m22.866922784s) [noise=true , address=104.36.82.9:6023 , piece_id=SISFAUU4RPF57OLGWM2BWQWE6GRUSCMORBYHWEXETNAVM2EEMO6Q, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.304005346 (81h9m22.867328346s) [noise=true , address=45.91.116.12:5242 , piece_id=6ABT5IPYGSLFFJMBYMBYRU4EYIXABUAUHG3OVILOHBIUDS3BP27A, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.304388850 (81h9m22.867711851s) [noise=true , address=64.49.13.137:4500 , piece_id=FAE2RXD57IRP5FXA2XR6THXFW27U7IWAZ4JDV5M2WRPXKUGZ4VIA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.304881529 (81h9m22.868204536s) [noise=true , address=79.127.220.101:6048 , piece_id=AN7UQUXB42T2YXEH7ZJRZUDIW47M3RWUEOCL2VEBOGH74BZF3YKA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.305249860 (81h9m22.86857287s) [noise=true , address=104.36.82.54:4165 , piece_id=4CVOABPCWC7J7SRHZB6IN2WJJ6KGWUQ66ELUIZIFA67ETLQ7JEUQ, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.305604108 (81h9m22.868927112s) [noise=true , address=104.36.82.15:6289 , piece_id=GWL3P55BIK4IJEL5YFZKI7JCA4LADYOOHUSIQUNGJEKBJOTKMP5A, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.305930423 (81h9m22.869253425s) [noise=true , address=64.49.13.139:4476 , piece_id=P7UKBNNZGI52B3I6WKTV6MVC262FYNKYYPJUI5G47UO5S4MU7PWA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.306355311 (81h9m22.869678319s) [noise=true , address=104.36.82.55:4213 , piece_id=X4IHS2XSPH52LIMNUYT7KGWMAPUH3TEMJOEEVB3JV5N45SEOH7YA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.306788243 (81h9m22.870111252s) [noise=true , address=104.36.82.22:6539 , piece_id=GHQ2MVSTDGVYLUKLV23D6YWRR4PP7UUYHCUEQ4WWSWEMTBFO3P4A, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.392174770 (81h9m22.955497773s) [upload=64958]: Next returned nil segment Jun 22 01:32:47.392195767 (81h9m22.955518775s) [upload=64958]: Waiting for error group managing segments... Jun 22 01:32:47.480767633 (81h9m23.044090623s) [noise=true , address=104.36.82.22:6539 , piece_id=GHQ2MVSTDGVYLUKLV23D6YWRR4PP7UUYHCUEQ4WWSWEMTBFO3P4A, seg_pos={0 0}, upload=64958]: Done uploading piece. err: ecclient: upload failed (node:1QL6w8DSFR6CxDdnJmDuyVZxp6JU55VKiJofBZRfQi7q31i8Jc, address:104.36.82.22:6539): protocol: storage node overloaded, request limit: 2000 Jun 22 01:32:47.585663142 (81h9m23.148986155s) [noise=true , address=79.127.220.101:6048 , piece_id=AN7UQUXB42T2YXEH7ZJRZUDIW47M3RWUEOCL2VEBOGH74BZF3YKA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:47.640263002 (81h9m23.203586013s) [noise=true , address=45.91.116.12:5242 , piece_id=6ABT5IPYGSLFFJMBYMBYRU4EYIXABUAUHG3OVILOHBIUDS3BP27A, seg_pos={0 0}, upload=64958]: Done uploading piece. err: ecclient: upload failed (node:12JsFL62jgXmza8Md8eQ9k8cyw5V9Vz2RzN4y85ZvgHZjwrq4vQ, address:45.91.116.12:5242): protocol: storage node overloaded, request limit: 2000 Jun 22 01:32:47.984127275 (81h9m23.54745028s) [noise=true , address=64.49.13.137:4500 , piece_id=FAE2RXD57IRP5FXA2XR6THXFW27U7IWAZ4JDV5M2WRPXKUGZ4VIA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:48.100255249 (81h9m23.663578241s) [noise=true , address=77.90.31.14:4689 , piece_id=Q3HDS3BCMYDTODFRMU4XFRDCQK2QZE3ZMXVLUX4M54PDEBS7RR6A, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:48.221764035 (81h9m23.785087034s) [noise=true , address=104.36.82.54:4165 , piece_id=4CVOABPCWC7J7SRHZB6IN2WJJ6KGWUQ66ELUIZIFA67ETLQ7JEUQ, seg_pos={0 0}, upload=64958]: Done uploading piece. err: ecclient: upload failed (node:12wGUMXVEubEtki4qGnjGEsbYFB1iUauZcu3oDRKrCQsJMV3bV2, address:104.36.82.54:4165): protocol: storage node overloaded, request limit: 2000 Jun 22 01:32:48.509996527 (81h9m24.073319547s) [noise=true , address=104.36.82.9:6023 , piece_id=SISFAUU4RPF57OLGWM2BWQWE6GRUSCMORBYHWEXETNAVM2EEMO6Q, seg_pos={0 0}, upload=64958]: Done uploading piece. err: ecclient: upload failed (node:1j2S9tgWpLwmyvXZSq2qKKcbaC8r7LkzfDCaMxCejAni4CST1g, address:104.36.82.9:6023): protocol: storage node overloaded, request limit: 2000 Jun 22 01:32:48.510513169 (81h9m24.073836177s) [noise=true , address=104.36.82.42:6811 , piece_id=A5BFEHZ3WGGK53IZ7MKI6N4CVXHOORF6L47NR3UFDCAX7LBHRHVQ, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:48.565245605 (81h9m24.128568606s) [noise=true , address=79.127.205.234:5601 , piece_id=V7PMXBP2VFWBSZDG4GTLBD435T6F5DT7A43L4GQLQ5BDDTJ4WY6A, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:49.979979824 (81h9m25.543302825s) [noise=true , address=104.36.82.15:6289 , piece_id=GWL3P55BIK4IJEL5YFZKI7JCA4LADYOOHUSIQUNGJEKBJOTKMP5A, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:50.463750280 (81h9m26.02707328s) [noise=true , address=104.36.82.51:4084 , piece_id=D43FA7BFE54KPCWGDGDME2EGSWM7PONOVPD3DN2XY42MJFOKVDIQ, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:51.480599795 (81h9m27.043922798s) [noise=true , address=79.127.219.38:4023 , piece_id=BWLRCXXHBMBTNCZEVKKIIS5PB35PDQXQXCLIQB7GBLEIOWWUFBFA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:52.941200625 (81h9m28.504523627s) [noise=true , address=104.36.82.43:6841 , piece_id=PIHZMSTO3CII7KGDLGSAILTDRSLMEBVOLBCRCZJULMCFISXIZWXA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:58.281803598 (81h9m33.845126612s) [noise=true , address=64.49.13.139:4476 , piece_id=P7UKBNNZGI52B3I6WKTV6MVC262FYNKYYPJUI5G47UO5S4MU7PWA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:33:00.277271877 (81h9m35.840594867s) [noise=true , address=104.36.82.55:4213 , piece_id=X4IHS2XSPH52LIMNUYT7KGWMAPUH3TEMJOEEVB3JV5N45SEOH7YA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:33:09.435339213 (81h9m44.998662206s) [noise=true , address=104.36.82.52:4089 , piece_id=JSOEKLNWA3W3LOVKI3EGGKJYJ6J3UZY6KGABKQPQPXTIOOPMKVHA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:33:31.472868445 (81h10m7.03619143s) [noise=true , address=104.36.82.53:4122 , piece_id=HZZQBIKHQWLUE5EHK4KSKJGIUEG7D44RQASUJHCYXX6BICCGX6EA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:33:47.235404653 (81h10m22.798727642s) [noise=true , address=104.36.82.12:6158 , piece_id=LJZEOJVJCJAR2BRILIJI3ZVDGYMPBBTP2G3HCTN3MHNS4KYDPASQ, seg_pos={0 0}, upload=64958]: Done uploading piece. err: ecclient: upload failed (node:1SjM6Utm2YizeUbrBUD9Z7etE1nLadW5Fa1zYP7MioPeeqhdJ7, address:104.36.82.12:6158): protocol: expected piece hash; context deadline exceeded; EOF Jun 22 01:33:47.235448297 (81h10m22.798771306s) [seg_pos={0 0}, upload=64958]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jun 22 01:33:47.265442393 (81h10m22.828765401s) [upload=64958]: Done uploading segments... Jun 22 01:33:47.265476139 (81h10m22.82879914s) [upload=64958]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration:* 13.766490015s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [...........................................................X] duration: 1m0.001358065s FAIL (2025-06-22 01:45:02 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1m0.001394117s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [...........................................X................] duration: 17.628523942s FAIL (2025-06-22 01:44:29 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== oad=64949]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jun 22 01:32:13.674692303 (81h8m49.238015311s) [upload=64949]: Done uploading segments... Jun 22 01:32:13.674727219 (81h8m49.238050233s) [upload=64949]: Upload finished. err: <nil> Jun 22 01:32:47.266407995 (81h9m22.829731002s) [upload=64958]: Starting upload Jun 22 01:32:47.266524767 (81h9m22.829847771s) [upload=64958]: Done starting upload Jun 22 01:32:47.266813109 (81h9m22.830136116s) [upload=64958]: Uploading segments... Jun 22 01:32:47.266969526 (81h9m22.830292526s) [upload=64958]: Got next segment. Inline: false Jun 22 01:32:47.267075475 (81h9m22.830398475s) [upload=64958]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 22 01:32:47.267081679 (81h9m22.830404667s) [upload=64958]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 22 01:32:47.301770382 (81h9m22.865093389s) [seg_pos={0 0}, upload=64958]: Begin upload segment... Jun 22 01:32:47.301823796 (81h9m22.865146799s) [seg_pos={0 0}, upload=64958]: Done begin upload segment. Jun 22 01:32:47.301855055 (81h9m22.865178087s) [noise=true , address=104.36.82.43:6841 , piece_id=PIHZMSTO3CII7KGDLGSAILTDRSLMEBVOLBCRCZJULMCFISXIZWXA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.301902600 (81h9m22.865225606s) [noise=true , address=104.36.82.42:6811 , piece_id=A5BFEHZ3WGGK53IZ7MKI6N4CVXHOORF6L47NR3UFDCAX7LBHRHVQ, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.302234027 (81h9m22.865557029s) [noise=true , address=104.36.82.51:4084 , piece_id=D43FA7BFE54KPCWGDGDME2EGSWM7PONOVPD3DN2XY42MJFOKVDIQ, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.302496354 (81h9m22.865819363s) [noise=true , address=79.127.219.38:4023 , piece_id=BWLRCXXHBMBTNCZEVKKIIS5PB35PDQXQXCLIQB7GBLEIOWWUFBFA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.302577370 (81h9m22.865900378s) [noise=true , address=104.36.82.52:4089 , piece_id=JSOEKLNWA3W3LOVKI3EGGKJYJ6J3UZY6KGABKQPQPXTIOOPMKVHA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.302917829 (81h9m22.866240835s) [noise=true , address=104.36.82.53:4122 , piece_id=HZZQBIKHQWLUE5EHK4KSKJGIUEG7D44RQASUJHCYXX6BICCGX6EA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.303087165 (81h9m22.866410168s) [noise=true , address=77.90.31.14:4689 , piece_id=Q3HDS3BCMYDTODFRMU4XFRDCQK2QZE3ZMXVLUX4M54PDEBS7RR6A, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.303250714 (81h9m22.866573717s) [noise=true , address=79.127.205.234:5601 , piece_id=V7PMXBP2VFWBSZDG4GTLBD435T6F5DT7A43L4GQLQ5BDDTJ4WY6A, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.303579249 (81h9m22.866902255s) [noise=true , address=104.36.82.12:6158 , piece_id=LJZEOJVJCJAR2BRILIJI3ZVDGYMPBBTP2G3HCTN3MHNS4KYDPASQ, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.303599776 (81h9m22.866922784s) [noise=true , address=104.36.82.9:6023 , piece_id=SISFAUU4RPF57OLGWM2BWQWE6GRUSCMORBYHWEXETNAVM2EEMO6Q, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.304005346 (81h9m22.867328346s) [noise=true , address=45.91.116.12:5242 , piece_id=6ABT5IPYGSLFFJMBYMBYRU4EYIXABUAUHG3OVILOHBIUDS3BP27A, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.304388850 (81h9m22.867711851s) [noise=true , address=64.49.13.137:4500 , piece_id=FAE2RXD57IRP5FXA2XR6THXFW27U7IWAZ4JDV5M2WRPXKUGZ4VIA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.304881529 (81h9m22.868204536s) [noise=true , address=79.127.220.101:6048 , piece_id=AN7UQUXB42T2YXEH7ZJRZUDIW47M3RWUEOCL2VEBOGH74BZF3YKA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.305249860 (81h9m22.86857287s) [noise=true , address=104.36.82.54:4165 , piece_id=4CVOABPCWC7J7SRHZB6IN2WJJ6KGWUQ66ELUIZIFA67ETLQ7JEUQ, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.305604108 (81h9m22.868927112s) [noise=true , address=104.36.82.15:6289 , piece_id=GWL3P55BIK4IJEL5YFZKI7JCA4LADYOOHUSIQUNGJEKBJOTKMP5A, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.305930423 (81h9m22.869253425s) [noise=true , address=64.49.13.139:4476 , piece_id=P7UKBNNZGI52B3I6WKTV6MVC262FYNKYYPJUI5G47UO5S4MU7PWA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.306355311 (81h9m22.869678319s) [noise=true , address=104.36.82.55:4213 , piece_id=X4IHS2XSPH52LIMNUYT7KGWMAPUH3TEMJOEEVB3JV5N45SEOH7YA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.306788243 (81h9m22.870111252s) [noise=true , address=104.36.82.22:6539 , piece_id=GHQ2MVSTDGVYLUKLV23D6YWRR4PP7UUYHCUEQ4WWSWEMTBFO3P4A, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.392174770 (81h9m22.955497773s) [upload=64958]: Next returned nil segment Jun 22 01:32:47.392195767 (81h9m22.955518775s) [upload=64958]: Waiting for error group managing segments... Jun 22 01:32:47.480767633 (81h9m23.044090623s) [noise=true , address=104.36.82.22:6539 , piece_id=GHQ2MVSTDGVYLUKLV23D6YWRR4PP7UUYHCUEQ4WWSWEMTBFO3P4A, seg_pos={0 0}, upload=64958]: Done uploading piece. err: ecclient: upload failed (node:1QL6w8DSFR6CxDdnJmDuyVZxp6JU55VKiJofBZRfQi7q31i8Jc, address:104.36.82.22:6539): protocol: storage node overloaded, request limit: 2000 Jun 22 01:32:47.585663142 (81h9m23.148986155s) [noise=true , address=79.127.220.101:6048 , piece_id=AN7UQUXB42T2YXEH7ZJRZUDIW47M3RWUEOCL2VEBOGH74BZF3YKA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:47.640263002 (81h9m23.203586013s) [noise=true , address=45.91.116.12:5242 , piece_id=6ABT5IPYGSLFFJMBYMBYRU4EYIXABUAUHG3OVILOHBIUDS3BP27A, seg_pos={0 0}, upload=64958]: Done uploading piece. err: ecclient: upload failed (node:12JsFL62jgXmza8Md8eQ9k8cyw5V9Vz2RzN4y85ZvgHZjwrq4vQ, address:45.91.116.12:5242): protocol: storage node overloaded, request limit: 2000 Jun 22 01:32:47.984127275 (81h9m23.54745028s) [noise=true , address=64.49.13.137:4500 , piece_id=FAE2RXD57IRP5FXA2XR6THXFW27U7IWAZ4JDV5M2WRPXKUGZ4VIA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:48.100255249 (81h9m23.663578241s) [noise=true , address=77.90.31.14:4689 , piece_id=Q3HDS3BCMYDTODFRMU4XFRDCQK2QZE3ZMXVLUX4M54PDEBS7RR6A, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:48.221764035 (81h9m23.785087034s) [noise=true , address=104.36.82.54:4165 , piece_id=4CVOABPCWC7J7SRHZB6IN2WJJ6KGWUQ66ELUIZIFA67ETLQ7JEUQ, seg_pos={0 0}, upload=64958]: Done uploading piece. err: ecclient: upload failed (node:12wGUMXVEubEtki4qGnjGEsbYFB1iUauZcu3oDRKrCQsJMV3bV2, address:104.36.82.54:4165): protocol: storage node overloaded, request limit: 2000 Jun 22 01:32:48.509996527 (81h9m24.073319547s) [noise=true , address=104.36.82.9:6023 , piece_id=SISFAUU4RPF57OLGWM2BWQWE6GRUSCMORBYHWEXETNAVM2EEMO6Q, seg_pos={0 0}, upload=64958]: Done uploading piece. err: ecclient: upload failed (node:1j2S9tgWpLwmyvXZSq2qKKcbaC8r7LkzfDCaMxCejAni4CST1g, address:104.36.82.9:6023): protocol: storage node overloaded, request limit: 2000 Jun 22 01:32:48.510513169 (81h9m24.073836177s) [noise=true , address=104.36.82.42:6811 , piece_id=A5BFEHZ3WGGK53IZ7MKI6N4CVXHOORF6L47NR3UFDCAX7LBHRHVQ, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:48.565245605 (81h9m24.128568606s) [noise=true , address=79.127.205.234:5601 , piece_id=V7PMXBP2VFWBSZDG4GTLBD435T6F5DT7A43L4GQLQ5BDDTJ4WY6A, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:49.979979824 (81h9m25.543302825s) [noise=true , address=104.36.82.15:6289 , piece_id=GWL3P55BIK4IJEL5YFZKI7JCA4LADYOOHUSIQUNGJEKBJOTKMP5A, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:50.463750280 (81h9m26.02707328s) [noise=true , address=104.36.82.51:4084 , piece_id=D43FA7BFE54KPCWGDGDME2EGSWM7PONOVPD3DN2XY42MJFOKVDIQ, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:51.480599795 (81h9m27.043922798s) [noise=true , address=79.127.219.38:4023 , piece_id=BWLRCXXHBMBTNCZEVKKIIS5PB35PDQXQXCLIQB7GBLEIOWWUFBFA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:52.941200625 (81h9m28.504523627s) [noise=true , address=104.36.82.43:6841 , piece_id=PIHZMSTO3CII7KGDLGSAILTDRSLMEBVOLBCRCZJULMCFISXIZWXA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:58.281803598 (81h9m33.845126612s) [noise=true , address=64.49.13.139:4476 , piece_id=P7UKBNNZGI52B3I6WKTV6MVC262FYNKYYPJUI5G47UO5S4MU7PWA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:33:00.277271877 (81h9m35.840594867s) [noise=true , address=104.36.82.55:4213 , piece_id=X4IHS2XSPH52LIMNUYT7KGWMAPUH3TEMJOEEVB3JV5N45SEOH7YA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:33:09.435339213 (81h9m44.998662206s) [noise=true , address=104.36.82.52:4089 , piece_id=JSOEKLNWA3W3LOVKI3EGGKJYJ6J3UZY6KGABKQPQPXTIOOPMKVHA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:33:31.472868445 (81h10m7.03619143s) [noise=true , address=104.36.82.53:4122 , piece_id=HZZQBIKHQWLUE5EHK4KSKJGIUEG7D44RQASUJHCYXX6BICCGX6EA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:33:47.235404653 (81h10m22.798727642s) [noise=true , address=104.36.82.12:6158 , piece_id=LJZEOJVJCJAR2BRILIJI3ZVDGYMPBBTP2G3HCTN3MHNS4KYDPASQ, seg_pos={0 0}, upload=64958]: Done uploading piece. err: ecclient: upload failed (node:1SjM6Utm2YizeUbrBUD9Z7etE1nLadW5Fa1zYP7MioPeeqhdJ7, address:104.36.82.12:6158): protocol: expected piece hash; context deadline exceeded; EOF Jun 22 01:33:47.235448297 (81h10m22.798771306s) [seg_pos={0 0}, upload=64958]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jun 22 01:33:47.265442393 (81h10m22.828765401s) [upload=64958]: Done uploading segments... Jun 22 01:33:47.265476139 (81h10m22.82879914s) [upload=64958]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration:* 17.628541937s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [............................................X...............] duration: 35.654373092s FAIL (2025-06-22 01:44:11 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== oad=64949]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jun 22 01:32:13.674692303 (81h8m49.238015311s) [upload=64949]: Done uploading segments... Jun 22 01:32:13.674727219 (81h8m49.238050233s) [upload=64949]: Upload finished. err: <nil> Jun 22 01:32:47.266407995 (81h9m22.829731002s) [upload=64958]: Starting upload Jun 22 01:32:47.266524767 (81h9m22.829847771s) [upload=64958]: Done starting upload Jun 22 01:32:47.266813109 (81h9m22.830136116s) [upload=64958]: Uploading segments... Jun 22 01:32:47.266969526 (81h9m22.830292526s) [upload=64958]: Got next segment. Inline: false Jun 22 01:32:47.267075475 (81h9m22.830398475s) [upload=64958]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 22 01:32:47.267081679 (81h9m22.830404667s) [upload=64958]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 22 01:32:47.301770382 (81h9m22.865093389s) [seg_pos={0 0}, upload=64958]: Begin upload segment... Jun 22 01:32:47.301823796 (81h9m22.865146799s) [seg_pos={0 0}, upload=64958]: Done begin upload segment. Jun 22 01:32:47.301855055 (81h9m22.865178087s) [noise=true , address=104.36.82.43:6841 , piece_id=PIHZMSTO3CII7KGDLGSAILTDRSLMEBVOLBCRCZJULMCFISXIZWXA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.301902600 (81h9m22.865225606s) [noise=true , address=104.36.82.42:6811 , piece_id=A5BFEHZ3WGGK53IZ7MKI6N4CVXHOORF6L47NR3UFDCAX7LBHRHVQ, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.302234027 (81h9m22.865557029s) [noise=true , address=104.36.82.51:4084 , piece_id=D43FA7BFE54KPCWGDGDME2EGSWM7PONOVPD3DN2XY42MJFOKVDIQ, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.302496354 (81h9m22.865819363s) [noise=true , address=79.127.219.38:4023 , piece_id=BWLRCXXHBMBTNCZEVKKIIS5PB35PDQXQXCLIQB7GBLEIOWWUFBFA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.302577370 (81h9m22.865900378s) [noise=true , address=104.36.82.52:4089 , piece_id=JSOEKLNWA3W3LOVKI3EGGKJYJ6J3UZY6KGABKQPQPXTIOOPMKVHA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.302917829 (81h9m22.866240835s) [noise=true , address=104.36.82.53:4122 , piece_id=HZZQBIKHQWLUE5EHK4KSKJGIUEG7D44RQASUJHCYXX6BICCGX6EA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.303087165 (81h9m22.866410168s) [noise=true , address=77.90.31.14:4689 , piece_id=Q3HDS3BCMYDTODFRMU4XFRDCQK2QZE3ZMXVLUX4M54PDEBS7RR6A, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.303250714 (81h9m22.866573717s) [noise=true , address=79.127.205.234:5601 , piece_id=V7PMXBP2VFWBSZDG4GTLBD435T6F5DT7A43L4GQLQ5BDDTJ4WY6A, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.303579249 (81h9m22.866902255s) [noise=true , address=104.36.82.12:6158 , piece_id=LJZEOJVJCJAR2BRILIJI3ZVDGYMPBBTP2G3HCTN3MHNS4KYDPASQ, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.303599776 (81h9m22.866922784s) [noise=true , address=104.36.82.9:6023 , piece_id=SISFAUU4RPF57OLGWM2BWQWE6GRUSCMORBYHWEXETNAVM2EEMO6Q, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.304005346 (81h9m22.867328346s) [noise=true , address=45.91.116.12:5242 , piece_id=6ABT5IPYGSLFFJMBYMBYRU4EYIXABUAUHG3OVILOHBIUDS3BP27A, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.304388850 (81h9m22.867711851s) [noise=true , address=64.49.13.137:4500 , piece_id=FAE2RXD57IRP5FXA2XR6THXFW27U7IWAZ4JDV5M2WRPXKUGZ4VIA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.304881529 (81h9m22.868204536s) [noise=true , address=79.127.220.101:6048 , piece_id=AN7UQUXB42T2YXEH7ZJRZUDIW47M3RWUEOCL2VEBOGH74BZF3YKA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.305249860 (81h9m22.86857287s) [noise=true , address=104.36.82.54:4165 , piece_id=4CVOABPCWC7J7SRHZB6IN2WJJ6KGWUQ66ELUIZIFA67ETLQ7JEUQ, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.305604108 (81h9m22.868927112s) [noise=true , address=104.36.82.15:6289 , piece_id=GWL3P55BIK4IJEL5YFZKI7JCA4LADYOOHUSIQUNGJEKBJOTKMP5A, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.305930423 (81h9m22.869253425s) [noise=true , address=64.49.13.139:4476 , piece_id=P7UKBNNZGI52B3I6WKTV6MVC262FYNKYYPJUI5G47UO5S4MU7PWA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.306355311 (81h9m22.869678319s) [noise=true , address=104.36.82.55:4213 , piece_id=X4IHS2XSPH52LIMNUYT7KGWMAPUH3TEMJOEEVB3JV5N45SEOH7YA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.306788243 (81h9m22.870111252s) [noise=true , address=104.36.82.22:6539 , piece_id=GHQ2MVSTDGVYLUKLV23D6YWRR4PP7UUYHCUEQ4WWSWEMTBFO3P4A, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.392174770 (81h9m22.955497773s) [upload=64958]: Next returned nil segment Jun 22 01:32:47.392195767 (81h9m22.955518775s) [upload=64958]: Waiting for error group managing segments... Jun 22 01:32:47.480767633 (81h9m23.044090623s) [noise=true , address=104.36.82.22:6539 , piece_id=GHQ2MVSTDGVYLUKLV23D6YWRR4PP7UUYHCUEQ4WWSWEMTBFO3P4A, seg_pos={0 0}, upload=64958]: Done uploading piece. err: ecclient: upload failed (node:1QL6w8DSFR6CxDdnJmDuyVZxp6JU55VKiJofBZRfQi7q31i8Jc, address:104.36.82.22:6539): protocol: storage node overloaded, request limit: 2000 Jun 22 01:32:47.585663142 (81h9m23.148986155s) [noise=true , address=79.127.220.101:6048 , piece_id=AN7UQUXB42T2YXEH7ZJRZUDIW47M3RWUEOCL2VEBOGH74BZF3YKA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:47.640263002 (81h9m23.203586013s) [noise=true , address=45.91.116.12:5242 , piece_id=6ABT5IPYGSLFFJMBYMBYRU4EYIXABUAUHG3OVILOHBIUDS3BP27A, seg_pos={0 0}, upload=64958]: Done uploading piece. err: ecclient: upload failed (node:12JsFL62jgXmza8Md8eQ9k8cyw5V9Vz2RzN4y85ZvgHZjwrq4vQ, address:45.91.116.12:5242): protocol: storage node overloaded, request limit: 2000 Jun 22 01:32:47.984127275 (81h9m23.54745028s) [noise=true , address=64.49.13.137:4500 , piece_id=FAE2RXD57IRP5FXA2XR6THXFW27U7IWAZ4JDV5M2WRPXKUGZ4VIA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:48.100255249 (81h9m23.663578241s) [noise=true , address=77.90.31.14:4689 , piece_id=Q3HDS3BCMYDTODFRMU4XFRDCQK2QZE3ZMXVLUX4M54PDEBS7RR6A, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:48.221764035 (81h9m23.785087034s) [noise=true , address=104.36.82.54:4165 , piece_id=4CVOABPCWC7J7SRHZB6IN2WJJ6KGWUQ66ELUIZIFA67ETLQ7JEUQ, seg_pos={0 0}, upload=64958]: Done uploading piece. err: ecclient: upload failed (node:12wGUMXVEubEtki4qGnjGEsbYFB1iUauZcu3oDRKrCQsJMV3bV2, address:104.36.82.54:4165): protocol: storage node overloaded, request limit: 2000 Jun 22 01:32:48.509996527 (81h9m24.073319547s) [noise=true , address=104.36.82.9:6023 , piece_id=SISFAUU4RPF57OLGWM2BWQWE6GRUSCMORBYHWEXETNAVM2EEMO6Q, seg_pos={0 0}, upload=64958]: Done uploading piece. err: ecclient: upload failed (node:1j2S9tgWpLwmyvXZSq2qKKcbaC8r7LkzfDCaMxCejAni4CST1g, address:104.36.82.9:6023): protocol: storage node overloaded, request limit: 2000 Jun 22 01:32:48.510513169 (81h9m24.073836177s) [noise=true , address=104.36.82.42:6811 , piece_id=A5BFEHZ3WGGK53IZ7MKI6N4CVXHOORF6L47NR3UFDCAX7LBHRHVQ, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:48.565245605 (81h9m24.128568606s) [noise=true , address=79.127.205.234:5601 , piece_id=V7PMXBP2VFWBSZDG4GTLBD435T6F5DT7A43L4GQLQ5BDDTJ4WY6A, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:49.979979824 (81h9m25.543302825s) [noise=true , address=104.36.82.15:6289 , piece_id=GWL3P55BIK4IJEL5YFZKI7JCA4LADYOOHUSIQUNGJEKBJOTKMP5A, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:50.463750280 (81h9m26.02707328s) [noise=true , address=104.36.82.51:4084 , piece_id=D43FA7BFE54KPCWGDGDME2EGSWM7PONOVPD3DN2XY42MJFOKVDIQ, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:51.480599795 (81h9m27.043922798s) [noise=true , address=79.127.219.38:4023 , piece_id=BWLRCXXHBMBTNCZEVKKIIS5PB35PDQXQXCLIQB7GBLEIOWWUFBFA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:52.941200625 (81h9m28.504523627s) [noise=true , address=104.36.82.43:6841 , piece_id=PIHZMSTO3CII7KGDLGSAILTDRSLMEBVOLBCRCZJULMCFISXIZWXA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:58.281803598 (81h9m33.845126612s) [noise=true , address=64.49.13.139:4476 , piece_id=P7UKBNNZGI52B3I6WKTV6MVC262FYNKYYPJUI5G47UO5S4MU7PWA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:33:00.277271877 (81h9m35.840594867s) [noise=true , address=104.36.82.55:4213 , piece_id=X4IHS2XSPH52LIMNUYT7KGWMAPUH3TEMJOEEVB3JV5N45SEOH7YA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:33:09.435339213 (81h9m44.998662206s) [noise=true , address=104.36.82.52:4089 , piece_id=JSOEKLNWA3W3LOVKI3EGGKJYJ6J3UZY6KGABKQPQPXTIOOPMKVHA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:33:31.472868445 (81h10m7.03619143s) [noise=true , address=104.36.82.53:4122 , piece_id=HZZQBIKHQWLUE5EHK4KSKJGIUEG7D44RQASUJHCYXX6BICCGX6EA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:33:47.235404653 (81h10m22.798727642s) [noise=true , address=104.36.82.12:6158 , piece_id=LJZEOJVJCJAR2BRILIJI3ZVDGYMPBBTP2G3HCTN3MHNS4KYDPASQ, seg_pos={0 0}, upload=64958]: Done uploading piece. err: ecclient: upload failed (node:1SjM6Utm2YizeUbrBUD9Z7etE1nLadW5Fa1zYP7MioPeeqhdJ7, address:104.36.82.12:6158): protocol: expected piece hash; context deadline exceeded; EOF Jun 22 01:33:47.235448297 (81h10m22.798771306s) [seg_pos={0 0}, upload=64958]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jun 22 01:33:47.265442393 (81h10m22.828765401s) [upload=64958]: Done uploading segments... Jun 22 01:33:47.265476139 (81h10m22.82879914s) [upload=64958]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration:* 35.654427032s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.............................................X..............] duration: 14.325491555s FAIL (2025-06-22 01:43:29 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== oad=64949]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jun 22 01:32:13.674692303 (81h8m49.238015311s) [upload=64949]: Done uploading segments... Jun 22 01:32:13.674727219 (81h8m49.238050233s) [upload=64949]: Upload finished. err: <nil> Jun 22 01:32:47.266407995 (81h9m22.829731002s) [upload=64958]: Starting upload Jun 22 01:32:47.266524767 (81h9m22.829847771s) [upload=64958]: Done starting upload Jun 22 01:32:47.266813109 (81h9m22.830136116s) [upload=64958]: Uploading segments... Jun 22 01:32:47.266969526 (81h9m22.830292526s) [upload=64958]: Got next segment. Inline: false Jun 22 01:32:47.267075475 (81h9m22.830398475s) [upload=64958]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 22 01:32:47.267081679 (81h9m22.830404667s) [upload=64958]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 22 01:32:47.301770382 (81h9m22.865093389s) [seg_pos={0 0}, upload=64958]: Begin upload segment... Jun 22 01:32:47.301823796 (81h9m22.865146799s) [seg_pos={0 0}, upload=64958]: Done begin upload segment. Jun 22 01:32:47.301855055 (81h9m22.865178087s) [noise=true , address=104.36.82.43:6841 , piece_id=PIHZMSTO3CII7KGDLGSAILTDRSLMEBVOLBCRCZJULMCFISXIZWXA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.301902600 (81h9m22.865225606s) [noise=true , address=104.36.82.42:6811 , piece_id=A5BFEHZ3WGGK53IZ7MKI6N4CVXHOORF6L47NR3UFDCAX7LBHRHVQ, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.302234027 (81h9m22.865557029s) [noise=true , address=104.36.82.51:4084 , piece_id=D43FA7BFE54KPCWGDGDME2EGSWM7PONOVPD3DN2XY42MJFOKVDIQ, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.302496354 (81h9m22.865819363s) [noise=true , address=79.127.219.38:4023 , piece_id=BWLRCXXHBMBTNCZEVKKIIS5PB35PDQXQXCLIQB7GBLEIOWWUFBFA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.302577370 (81h9m22.865900378s) [noise=true , address=104.36.82.52:4089 , piece_id=JSOEKLNWA3W3LOVKI3EGGKJYJ6J3UZY6KGABKQPQPXTIOOPMKVHA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.302917829 (81h9m22.866240835s) [noise=true , address=104.36.82.53:4122 , piece_id=HZZQBIKHQWLUE5EHK4KSKJGIUEG7D44RQASUJHCYXX6BICCGX6EA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.303087165 (81h9m22.866410168s) [noise=true , address=77.90.31.14:4689 , piece_id=Q3HDS3BCMYDTODFRMU4XFRDCQK2QZE3ZMXVLUX4M54PDEBS7RR6A, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.303250714 (81h9m22.866573717s) [noise=true , address=79.127.205.234:5601 , piece_id=V7PMXBP2VFWBSZDG4GTLBD435T6F5DT7A43L4GQLQ5BDDTJ4WY6A, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.303579249 (81h9m22.866902255s) [noise=true , address=104.36.82.12:6158 , piece_id=LJZEOJVJCJAR2BRILIJI3ZVDGYMPBBTP2G3HCTN3MHNS4KYDPASQ, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.303599776 (81h9m22.866922784s) [noise=true , address=104.36.82.9:6023 , piece_id=SISFAUU4RPF57OLGWM2BWQWE6GRUSCMORBYHWEXETNAVM2EEMO6Q, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.304005346 (81h9m22.867328346s) [noise=true , address=45.91.116.12:5242 , piece_id=6ABT5IPYGSLFFJMBYMBYRU4EYIXABUAUHG3OVILOHBIUDS3BP27A, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.304388850 (81h9m22.867711851s) [noise=true , address=64.49.13.137:4500 , piece_id=FAE2RXD57IRP5FXA2XR6THXFW27U7IWAZ4JDV5M2WRPXKUGZ4VIA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.304881529 (81h9m22.868204536s) [noise=true , address=79.127.220.101:6048 , piece_id=AN7UQUXB42T2YXEH7ZJRZUDIW47M3RWUEOCL2VEBOGH74BZF3YKA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.305249860 (81h9m22.86857287s) [noise=true , address=104.36.82.54:4165 , piece_id=4CVOABPCWC7J7SRHZB6IN2WJJ6KGWUQ66ELUIZIFA67ETLQ7JEUQ, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.305604108 (81h9m22.868927112s) [noise=true , address=104.36.82.15:6289 , piece_id=GWL3P55BIK4IJEL5YFZKI7JCA4LADYOOHUSIQUNGJEKBJOTKMP5A, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.305930423 (81h9m22.869253425s) [noise=true , address=64.49.13.139:4476 , piece_id=P7UKBNNZGI52B3I6WKTV6MVC262FYNKYYPJUI5G47UO5S4MU7PWA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.306355311 (81h9m22.869678319s) [noise=true , address=104.36.82.55:4213 , piece_id=X4IHS2XSPH52LIMNUYT7KGWMAPUH3TEMJOEEVB3JV5N45SEOH7YA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.306788243 (81h9m22.870111252s) [noise=true , address=104.36.82.22:6539 , piece_id=GHQ2MVSTDGVYLUKLV23D6YWRR4PP7UUYHCUEQ4WWSWEMTBFO3P4A, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.392174770 (81h9m22.955497773s) [upload=64958]: Next returned nil segment Jun 22 01:32:47.392195767 (81h9m22.955518775s) [upload=64958]: Waiting for error group managing segments... Jun 22 01:32:47.480767633 (81h9m23.044090623s) [noise=true , address=104.36.82.22:6539 , piece_id=GHQ2MVSTDGVYLUKLV23D6YWRR4PP7UUYHCUEQ4WWSWEMTBFO3P4A, seg_pos={0 0}, upload=64958]: Done uploading piece. err: ecclient: upload failed (node:1QL6w8DSFR6CxDdnJmDuyVZxp6JU55VKiJofBZRfQi7q31i8Jc, address:104.36.82.22:6539): protocol: storage node overloaded, request limit: 2000 Jun 22 01:32:47.585663142 (81h9m23.148986155s) [noise=true , address=79.127.220.101:6048 , piece_id=AN7UQUXB42T2YXEH7ZJRZUDIW47M3RWUEOCL2VEBOGH74BZF3YKA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:47.640263002 (81h9m23.203586013s) [noise=true , address=45.91.116.12:5242 , piece_id=6ABT5IPYGSLFFJMBYMBYRU4EYIXABUAUHG3OVILOHBIUDS3BP27A, seg_pos={0 0}, upload=64958]: Done uploading piece. err: ecclient: upload failed (node:12JsFL62jgXmza8Md8eQ9k8cyw5V9Vz2RzN4y85ZvgHZjwrq4vQ, address:45.91.116.12:5242): protocol: storage node overloaded, request limit: 2000 Jun 22 01:32:47.984127275 (81h9m23.54745028s) [noise=true , address=64.49.13.137:4500 , piece_id=FAE2RXD57IRP5FXA2XR6THXFW27U7IWAZ4JDV5M2WRPXKUGZ4VIA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:48.100255249 (81h9m23.663578241s) [noise=true , address=77.90.31.14:4689 , piece_id=Q3HDS3BCMYDTODFRMU4XFRDCQK2QZE3ZMXVLUX4M54PDEBS7RR6A, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:48.221764035 (81h9m23.785087034s) [noise=true , address=104.36.82.54:4165 , piece_id=4CVOABPCWC7J7SRHZB6IN2WJJ6KGWUQ66ELUIZIFA67ETLQ7JEUQ, seg_pos={0 0}, upload=64958]: Done uploading piece. err: ecclient: upload failed (node:12wGUMXVEubEtki4qGnjGEsbYFB1iUauZcu3oDRKrCQsJMV3bV2, address:104.36.82.54:4165): protocol: storage node overloaded, request limit: 2000 Jun 22 01:32:48.509996527 (81h9m24.073319547s) [noise=true , address=104.36.82.9:6023 , piece_id=SISFAUU4RPF57OLGWM2BWQWE6GRUSCMORBYHWEXETNAVM2EEMO6Q, seg_pos={0 0}, upload=64958]: Done uploading piece. err: ecclient: upload failed (node:1j2S9tgWpLwmyvXZSq2qKKcbaC8r7LkzfDCaMxCejAni4CST1g, address:104.36.82.9:6023): protocol: storage node overloaded, request limit: 2000 Jun 22 01:32:48.510513169 (81h9m24.073836177s) [noise=true , address=104.36.82.42:6811 , piece_id=A5BFEHZ3WGGK53IZ7MKI6N4CVXHOORF6L47NR3UFDCAX7LBHRHVQ, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:48.565245605 (81h9m24.128568606s) [noise=true , address=79.127.205.234:5601 , piece_id=V7PMXBP2VFWBSZDG4GTLBD435T6F5DT7A43L4GQLQ5BDDTJ4WY6A, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:49.979979824 (81h9m25.543302825s) [noise=true , address=104.36.82.15:6289 , piece_id=GWL3P55BIK4IJEL5YFZKI7JCA4LADYOOHUSIQUNGJEKBJOTKMP5A, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:50.463750280 (81h9m26.02707328s) [noise=true , address=104.36.82.51:4084 , piece_id=D43FA7BFE54KPCWGDGDME2EGSWM7PONOVPD3DN2XY42MJFOKVDIQ, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:51.480599795 (81h9m27.043922798s) [noise=true , address=79.127.219.38:4023 , piece_id=BWLRCXXHBMBTNCZEVKKIIS5PB35PDQXQXCLIQB7GBLEIOWWUFBFA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:52.941200625 (81h9m28.504523627s) [noise=true , address=104.36.82.43:6841 , piece_id=PIHZMSTO3CII7KGDLGSAILTDRSLMEBVOLBCRCZJULMCFISXIZWXA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:58.281803598 (81h9m33.845126612s) [noise=true , address=64.49.13.139:4476 , piece_id=P7UKBNNZGI52B3I6WKTV6MVC262FYNKYYPJUI5G47UO5S4MU7PWA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:33:00.277271877 (81h9m35.840594867s) [noise=true , address=104.36.82.55:4213 , piece_id=X4IHS2XSPH52LIMNUYT7KGWMAPUH3TEMJOEEVB3JV5N45SEOH7YA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:33:09.435339213 (81h9m44.998662206s) [noise=true , address=104.36.82.52:4089 , piece_id=JSOEKLNWA3W3LOVKI3EGGKJYJ6J3UZY6KGABKQPQPXTIOOPMKVHA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:33:31.472868445 (81h10m7.03619143s) [noise=true , address=104.36.82.53:4122 , piece_id=HZZQBIKHQWLUE5EHK4KSKJGIUEG7D44RQASUJHCYXX6BICCGX6EA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:33:47.235404653 (81h10m22.798727642s) [noise=true , address=104.36.82.12:6158 , piece_id=LJZEOJVJCJAR2BRILIJI3ZVDGYMPBBTP2G3HCTN3MHNS4KYDPASQ, seg_pos={0 0}, upload=64958]: Done uploading piece. err: ecclient: upload failed (node:1SjM6Utm2YizeUbrBUD9Z7etE1nLadW5Fa1zYP7MioPeeqhdJ7, address:104.36.82.12:6158): protocol: expected piece hash; context deadline exceeded; EOF Jun 22 01:33:47.235448297 (81h10m22.798771306s) [seg_pos={0 0}, upload=64958]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jun 22 01:33:47.265442393 (81h10m22.828765401s) [upload=64958]: Done uploading segments... Jun 22 01:33:47.265476139 (81h10m22.82879914s) [upload=64958]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration:* 14.325525009s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [..............................................X.............] duration: 13.264913007s FAIL (2025-06-22 01:42:25 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== oad=64949]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jun 22 01:32:13.674692303 (81h8m49.238015311s) [upload=64949]: Done uploading segments... Jun 22 01:32:13.674727219 (81h8m49.238050233s) [upload=64949]: Upload finished. err: <nil> Jun 22 01:32:47.266407995 (81h9m22.829731002s) [upload=64958]: Starting upload Jun 22 01:32:47.266524767 (81h9m22.829847771s) [upload=64958]: Done starting upload Jun 22 01:32:47.266813109 (81h9m22.830136116s) [upload=64958]: Uploading segments... Jun 22 01:32:47.266969526 (81h9m22.830292526s) [upload=64958]: Got next segment. Inline: false Jun 22 01:32:47.267075475 (81h9m22.830398475s) [upload=64958]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 22 01:32:47.267081679 (81h9m22.830404667s) [upload=64958]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 22 01:32:47.301770382 (81h9m22.865093389s) [seg_pos={0 0}, upload=64958]: Begin upload segment... Jun 22 01:32:47.301823796 (81h9m22.865146799s) [seg_pos={0 0}, upload=64958]: Done begin upload segment. Jun 22 01:32:47.301855055 (81h9m22.865178087s) [noise=true , address=104.36.82.43:6841 , piece_id=PIHZMSTO3CII7KGDLGSAILTDRSLMEBVOLBCRCZJULMCFISXIZWXA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.301902600 (81h9m22.865225606s) [noise=true , address=104.36.82.42:6811 , piece_id=A5BFEHZ3WGGK53IZ7MKI6N4CVXHOORF6L47NR3UFDCAX7LBHRHVQ, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.302234027 (81h9m22.865557029s) [noise=true , address=104.36.82.51:4084 , piece_id=D43FA7BFE54KPCWGDGDME2EGSWM7PONOVPD3DN2XY42MJFOKVDIQ, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.302496354 (81h9m22.865819363s) [noise=true , address=79.127.219.38:4023 , piece_id=BWLRCXXHBMBTNCZEVKKIIS5PB35PDQXQXCLIQB7GBLEIOWWUFBFA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.302577370 (81h9m22.865900378s) [noise=true , address=104.36.82.52:4089 , piece_id=JSOEKLNWA3W3LOVKI3EGGKJYJ6J3UZY6KGABKQPQPXTIOOPMKVHA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.302917829 (81h9m22.866240835s) [noise=true , address=104.36.82.53:4122 , piece_id=HZZQBIKHQWLUE5EHK4KSKJGIUEG7D44RQASUJHCYXX6BICCGX6EA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.303087165 (81h9m22.866410168s) [noise=true , address=77.90.31.14:4689 , piece_id=Q3HDS3BCMYDTODFRMU4XFRDCQK2QZE3ZMXVLUX4M54PDEBS7RR6A, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.303250714 (81h9m22.866573717s) [noise=true , address=79.127.205.234:5601 , piece_id=V7PMXBP2VFWBSZDG4GTLBD435T6F5DT7A43L4GQLQ5BDDTJ4WY6A, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.303579249 (81h9m22.866902255s) [noise=true , address=104.36.82.12:6158 , piece_id=LJZEOJVJCJAR2BRILIJI3ZVDGYMPBBTP2G3HCTN3MHNS4KYDPASQ, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.303599776 (81h9m22.866922784s) [noise=true , address=104.36.82.9:6023 , piece_id=SISFAUU4RPF57OLGWM2BWQWE6GRUSCMORBYHWEXETNAVM2EEMO6Q, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.304005346 (81h9m22.867328346s) [noise=true , address=45.91.116.12:5242 , piece_id=6ABT5IPYGSLFFJMBYMBYRU4EYIXABUAUHG3OVILOHBIUDS3BP27A, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.304388850 (81h9m22.867711851s) [noise=true , address=64.49.13.137:4500 , piece_id=FAE2RXD57IRP5FXA2XR6THXFW27U7IWAZ4JDV5M2WRPXKUGZ4VIA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.304881529 (81h9m22.868204536s) [noise=true , address=79.127.220.101:6048 , piece_id=AN7UQUXB42T2YXEH7ZJRZUDIW47M3RWUEOCL2VEBOGH74BZF3YKA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.305249860 (81h9m22.86857287s) [noise=true , address=104.36.82.54:4165 , piece_id=4CVOABPCWC7J7SRHZB6IN2WJJ6KGWUQ66ELUIZIFA67ETLQ7JEUQ, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.305604108 (81h9m22.868927112s) [noise=true , address=104.36.82.15:6289 , piece_id=GWL3P55BIK4IJEL5YFZKI7JCA4LADYOOHUSIQUNGJEKBJOTKMP5A, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.305930423 (81h9m22.869253425s) [noise=true , address=64.49.13.139:4476 , piece_id=P7UKBNNZGI52B3I6WKTV6MVC262FYNKYYPJUI5G47UO5S4MU7PWA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.306355311 (81h9m22.869678319s) [noise=true , address=104.36.82.55:4213 , piece_id=X4IHS2XSPH52LIMNUYT7KGWMAPUH3TEMJOEEVB3JV5N45SEOH7YA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.306788243 (81h9m22.870111252s) [noise=true , address=104.36.82.22:6539 , piece_id=GHQ2MVSTDGVYLUKLV23D6YWRR4PP7UUYHCUEQ4WWSWEMTBFO3P4A, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.392174770 (81h9m22.955497773s) [upload=64958]: Next returned nil segment Jun 22 01:32:47.392195767 (81h9m22.955518775s) [upload=64958]: Waiting for error group managing segments... Jun 22 01:32:47.480767633 (81h9m23.044090623s) [noise=true , address=104.36.82.22:6539 , piece_id=GHQ2MVSTDGVYLUKLV23D6YWRR4PP7UUYHCUEQ4WWSWEMTBFO3P4A, seg_pos={0 0}, upload=64958]: Done uploading piece. err: ecclient: upload failed (node:1QL6w8DSFR6CxDdnJmDuyVZxp6JU55VKiJofBZRfQi7q31i8Jc, address:104.36.82.22:6539): protocol: storage node overloaded, request limit: 2000 Jun 22 01:32:47.585663142 (81h9m23.148986155s) [noise=true , address=79.127.220.101:6048 , piece_id=AN7UQUXB42T2YXEH7ZJRZUDIW47M3RWUEOCL2VEBOGH74BZF3YKA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:47.640263002 (81h9m23.203586013s) [noise=true , address=45.91.116.12:5242 , piece_id=6ABT5IPYGSLFFJMBYMBYRU4EYIXABUAUHG3OVILOHBIUDS3BP27A, seg_pos={0 0}, upload=64958]: Done uploading piece. err: ecclient: upload failed (node:12JsFL62jgXmza8Md8eQ9k8cyw5V9Vz2RzN4y85ZvgHZjwrq4vQ, address:45.91.116.12:5242): protocol: storage node overloaded, request limit: 2000 Jun 22 01:32:47.984127275 (81h9m23.54745028s) [noise=true , address=64.49.13.137:4500 , piece_id=FAE2RXD57IRP5FXA2XR6THXFW27U7IWAZ4JDV5M2WRPXKUGZ4VIA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:48.100255249 (81h9m23.663578241s) [noise=true , address=77.90.31.14:4689 , piece_id=Q3HDS3BCMYDTODFRMU4XFRDCQK2QZE3ZMXVLUX4M54PDEBS7RR6A, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:48.221764035 (81h9m23.785087034s) [noise=true , address=104.36.82.54:4165 , piece_id=4CVOABPCWC7J7SRHZB6IN2WJJ6KGWUQ66ELUIZIFA67ETLQ7JEUQ, seg_pos={0 0}, upload=64958]: Done uploading piece. err: ecclient: upload failed (node:12wGUMXVEubEtki4qGnjGEsbYFB1iUauZcu3oDRKrCQsJMV3bV2, address:104.36.82.54:4165): protocol: storage node overloaded, request limit: 2000 Jun 22 01:32:48.509996527 (81h9m24.073319547s) [noise=true , address=104.36.82.9:6023 , piece_id=SISFAUU4RPF57OLGWM2BWQWE6GRUSCMORBYHWEXETNAVM2EEMO6Q, seg_pos={0 0}, upload=64958]: Done uploading piece. err: ecclient: upload failed (node:1j2S9tgWpLwmyvXZSq2qKKcbaC8r7LkzfDCaMxCejAni4CST1g, address:104.36.82.9:6023): protocol: storage node overloaded, request limit: 2000 Jun 22 01:32:48.510513169 (81h9m24.073836177s) [noise=true , address=104.36.82.42:6811 , piece_id=A5BFEHZ3WGGK53IZ7MKI6N4CVXHOORF6L47NR3UFDCAX7LBHRHVQ, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:48.565245605 (81h9m24.128568606s) [noise=true , address=79.127.205.234:5601 , piece_id=V7PMXBP2VFWBSZDG4GTLBD435T6F5DT7A43L4GQLQ5BDDTJ4WY6A, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:49.979979824 (81h9m25.543302825s) [noise=true , address=104.36.82.15:6289 , piece_id=GWL3P55BIK4IJEL5YFZKI7JCA4LADYOOHUSIQUNGJEKBJOTKMP5A, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:50.463750280 (81h9m26.02707328s) [noise=true , address=104.36.82.51:4084 , piece_id=D43FA7BFE54KPCWGDGDME2EGSWM7PONOVPD3DN2XY42MJFOKVDIQ, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:51.480599795 (81h9m27.043922798s) [noise=true , address=79.127.219.38:4023 , piece_id=BWLRCXXHBMBTNCZEVKKIIS5PB35PDQXQXCLIQB7GBLEIOWWUFBFA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:52.941200625 (81h9m28.504523627s) [noise=true , address=104.36.82.43:6841 , piece_id=PIHZMSTO3CII7KGDLGSAILTDRSLMEBVOLBCRCZJULMCFISXIZWXA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:58.281803598 (81h9m33.845126612s) [noise=true , address=64.49.13.139:4476 , piece_id=P7UKBNNZGI52B3I6WKTV6MVC262FYNKYYPJUI5G47UO5S4MU7PWA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:33:00.277271877 (81h9m35.840594867s) [noise=true , address=104.36.82.55:4213 , piece_id=X4IHS2XSPH52LIMNUYT7KGWMAPUH3TEMJOEEVB3JV5N45SEOH7YA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:33:09.435339213 (81h9m44.998662206s) [noise=true , address=104.36.82.52:4089 , piece_id=JSOEKLNWA3W3LOVKI3EGGKJYJ6J3UZY6KGABKQPQPXTIOOPMKVHA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:33:31.472868445 (81h10m7.03619143s) [noise=true , address=104.36.82.53:4122 , piece_id=HZZQBIKHQWLUE5EHK4KSKJGIUEG7D44RQASUJHCYXX6BICCGX6EA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:33:47.235404653 (81h10m22.798727642s) [noise=true , address=104.36.82.12:6158 , piece_id=LJZEOJVJCJAR2BRILIJI3ZVDGYMPBBTP2G3HCTN3MHNS4KYDPASQ, seg_pos={0 0}, upload=64958]: Done uploading piece. err: ecclient: upload failed (node:1SjM6Utm2YizeUbrBUD9Z7etE1nLadW5Fa1zYP7MioPeeqhdJ7, address:104.36.82.12:6158): protocol: expected piece hash; context deadline exceeded; EOF Jun 22 01:33:47.235448297 (81h10m22.798771306s) [seg_pos={0 0}, upload=64958]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jun 22 01:33:47.265442393 (81h10m22.828765401s) [upload=64958]: Done uploading segments... Jun 22 01:33:47.265476139 (81h10m22.82879914s) [upload=64958]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration:* 13.264942434s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [...............................................X............] duration: 12.396198231s FAIL (2025-06-22 01:41:37 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== oad=64949]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jun 22 01:32:13.674692303 (81h8m49.238015311s) [upload=64949]: Done uploading segments... Jun 22 01:32:13.674727219 (81h8m49.238050233s) [upload=64949]: Upload finished. err: <nil> Jun 22 01:32:47.266407995 (81h9m22.829731002s) [upload=64958]: Starting upload Jun 22 01:32:47.266524767 (81h9m22.829847771s) [upload=64958]: Done starting upload Jun 22 01:32:47.266813109 (81h9m22.830136116s) [upload=64958]: Uploading segments... Jun 22 01:32:47.266969526 (81h9m22.830292526s) [upload=64958]: Got next segment. Inline: false Jun 22 01:32:47.267075475 (81h9m22.830398475s) [upload=64958]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 22 01:32:47.267081679 (81h9m22.830404667s) [upload=64958]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 22 01:32:47.301770382 (81h9m22.865093389s) [seg_pos={0 0}, upload=64958]: Begin upload segment... Jun 22 01:32:47.301823796 (81h9m22.865146799s) [seg_pos={0 0}, upload=64958]: Done begin upload segment. Jun 22 01:32:47.301855055 (81h9m22.865178087s) [noise=true , address=104.36.82.43:6841 , piece_id=PIHZMSTO3CII7KGDLGSAILTDRSLMEBVOLBCRCZJULMCFISXIZWXA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.301902600 (81h9m22.865225606s) [noise=true , address=104.36.82.42:6811 , piece_id=A5BFEHZ3WGGK53IZ7MKI6N4CVXHOORF6L47NR3UFDCAX7LBHRHVQ, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.302234027 (81h9m22.865557029s) [noise=true , address=104.36.82.51:4084 , piece_id=D43FA7BFE54KPCWGDGDME2EGSWM7PONOVPD3DN2XY42MJFOKVDIQ, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.302496354 (81h9m22.865819363s) [noise=true , address=79.127.219.38:4023 , piece_id=BWLRCXXHBMBTNCZEVKKIIS5PB35PDQXQXCLIQB7GBLEIOWWUFBFA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.302577370 (81h9m22.865900378s) [noise=true , address=104.36.82.52:4089 , piece_id=JSOEKLNWA3W3LOVKI3EGGKJYJ6J3UZY6KGABKQPQPXTIOOPMKVHA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.302917829 (81h9m22.866240835s) [noise=true , address=104.36.82.53:4122 , piece_id=HZZQBIKHQWLUE5EHK4KSKJGIUEG7D44RQASUJHCYXX6BICCGX6EA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.303087165 (81h9m22.866410168s) [noise=true , address=77.90.31.14:4689 , piece_id=Q3HDS3BCMYDTODFRMU4XFRDCQK2QZE3ZMXVLUX4M54PDEBS7RR6A, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.303250714 (81h9m22.866573717s) [noise=true , address=79.127.205.234:5601 , piece_id=V7PMXBP2VFWBSZDG4GTLBD435T6F5DT7A43L4GQLQ5BDDTJ4WY6A, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.303579249 (81h9m22.866902255s) [noise=true , address=104.36.82.12:6158 , piece_id=LJZEOJVJCJAR2BRILIJI3ZVDGYMPBBTP2G3HCTN3MHNS4KYDPASQ, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.303599776 (81h9m22.866922784s) [noise=true , address=104.36.82.9:6023 , piece_id=SISFAUU4RPF57OLGWM2BWQWE6GRUSCMORBYHWEXETNAVM2EEMO6Q, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.304005346 (81h9m22.867328346s) [noise=true , address=45.91.116.12:5242 , piece_id=6ABT5IPYGSLFFJMBYMBYRU4EYIXABUAUHG3OVILOHBIUDS3BP27A, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.304388850 (81h9m22.867711851s) [noise=true , address=64.49.13.137:4500 , piece_id=FAE2RXD57IRP5FXA2XR6THXFW27U7IWAZ4JDV5M2WRPXKUGZ4VIA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.304881529 (81h9m22.868204536s) [noise=true , address=79.127.220.101:6048 , piece_id=AN7UQUXB42T2YXEH7ZJRZUDIW47M3RWUEOCL2VEBOGH74BZF3YKA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.305249860 (81h9m22.86857287s) [noise=true , address=104.36.82.54:4165 , piece_id=4CVOABPCWC7J7SRHZB6IN2WJJ6KGWUQ66ELUIZIFA67ETLQ7JEUQ, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.305604108 (81h9m22.868927112s) [noise=true , address=104.36.82.15:6289 , piece_id=GWL3P55BIK4IJEL5YFZKI7JCA4LADYOOHUSIQUNGJEKBJOTKMP5A, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.305930423 (81h9m22.869253425s) [noise=true , address=64.49.13.139:4476 , piece_id=P7UKBNNZGI52B3I6WKTV6MVC262FYNKYYPJUI5G47UO5S4MU7PWA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.306355311 (81h9m22.869678319s) [noise=true , address=104.36.82.55:4213 , piece_id=X4IHS2XSPH52LIMNUYT7KGWMAPUH3TEMJOEEVB3JV5N45SEOH7YA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.306788243 (81h9m22.870111252s) [noise=true , address=104.36.82.22:6539 , piece_id=GHQ2MVSTDGVYLUKLV23D6YWRR4PP7UUYHCUEQ4WWSWEMTBFO3P4A, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.392174770 (81h9m22.955497773s) [upload=64958]: Next returned nil segment Jun 22 01:32:47.392195767 (81h9m22.955518775s) [upload=64958]: Waiting for error group managing segments... Jun 22 01:32:47.480767633 (81h9m23.044090623s) [noise=true , address=104.36.82.22:6539 , piece_id=GHQ2MVSTDGVYLUKLV23D6YWRR4PP7UUYHCUEQ4WWSWEMTBFO3P4A, seg_pos={0 0}, upload=64958]: Done uploading piece. err: ecclient: upload failed (node:1QL6w8DSFR6CxDdnJmDuyVZxp6JU55VKiJofBZRfQi7q31i8Jc, address:104.36.82.22:6539): protocol: storage node overloaded, request limit: 2000 Jun 22 01:32:47.585663142 (81h9m23.148986155s) [noise=true , address=79.127.220.101:6048 , piece_id=AN7UQUXB42T2YXEH7ZJRZUDIW47M3RWUEOCL2VEBOGH74BZF3YKA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:47.640263002 (81h9m23.203586013s) [noise=true , address=45.91.116.12:5242 , piece_id=6ABT5IPYGSLFFJMBYMBYRU4EYIXABUAUHG3OVILOHBIUDS3BP27A, seg_pos={0 0}, upload=64958]: Done uploading piece. err: ecclient: upload failed (node:12JsFL62jgXmza8Md8eQ9k8cyw5V9Vz2RzN4y85ZvgHZjwrq4vQ, address:45.91.116.12:5242): protocol: storage node overloaded, request limit: 2000 Jun 22 01:32:47.984127275 (81h9m23.54745028s) [noise=true , address=64.49.13.137:4500 , piece_id=FAE2RXD57IRP5FXA2XR6THXFW27U7IWAZ4JDV5M2WRPXKUGZ4VIA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:48.100255249 (81h9m23.663578241s) [noise=true , address=77.90.31.14:4689 , piece_id=Q3HDS3BCMYDTODFRMU4XFRDCQK2QZE3ZMXVLUX4M54PDEBS7RR6A, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:48.221764035 (81h9m23.785087034s) [noise=true , address=104.36.82.54:4165 , piece_id=4CVOABPCWC7J7SRHZB6IN2WJJ6KGWUQ66ELUIZIFA67ETLQ7JEUQ, seg_pos={0 0}, upload=64958]: Done uploading piece. err: ecclient: upload failed (node:12wGUMXVEubEtki4qGnjGEsbYFB1iUauZcu3oDRKrCQsJMV3bV2, address:104.36.82.54:4165): protocol: storage node overloaded, request limit: 2000 Jun 22 01:32:48.509996527 (81h9m24.073319547s) [noise=true , address=104.36.82.9:6023 , piece_id=SISFAUU4RPF57OLGWM2BWQWE6GRUSCMORBYHWEXETNAVM2EEMO6Q, seg_pos={0 0}, upload=64958]: Done uploading piece. err: ecclient: upload failed (node:1j2S9tgWpLwmyvXZSq2qKKcbaC8r7LkzfDCaMxCejAni4CST1g, address:104.36.82.9:6023): protocol: storage node overloaded, request limit: 2000 Jun 22 01:32:48.510513169 (81h9m24.073836177s) [noise=true , address=104.36.82.42:6811 , piece_id=A5BFEHZ3WGGK53IZ7MKI6N4CVXHOORF6L47NR3UFDCAX7LBHRHVQ, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:48.565245605 (81h9m24.128568606s) [noise=true , address=79.127.205.234:5601 , piece_id=V7PMXBP2VFWBSZDG4GTLBD435T6F5DT7A43L4GQLQ5BDDTJ4WY6A, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:49.979979824 (81h9m25.543302825s) [noise=true , address=104.36.82.15:6289 , piece_id=GWL3P55BIK4IJEL5YFZKI7JCA4LADYOOHUSIQUNGJEKBJOTKMP5A, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:50.463750280 (81h9m26.02707328s) [noise=true , address=104.36.82.51:4084 , piece_id=D43FA7BFE54KPCWGDGDME2EGSWM7PONOVPD3DN2XY42MJFOKVDIQ, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:51.480599795 (81h9m27.043922798s) [noise=true , address=79.127.219.38:4023 , piece_id=BWLRCXXHBMBTNCZEVKKIIS5PB35PDQXQXCLIQB7GBLEIOWWUFBFA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:52.941200625 (81h9m28.504523627s) [noise=true , address=104.36.82.43:6841 , piece_id=PIHZMSTO3CII7KGDLGSAILTDRSLMEBVOLBCRCZJULMCFISXIZWXA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:58.281803598 (81h9m33.845126612s) [noise=true , address=64.49.13.139:4476 , piece_id=P7UKBNNZGI52B3I6WKTV6MVC262FYNKYYPJUI5G47UO5S4MU7PWA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:33:00.277271877 (81h9m35.840594867s) [noise=true , address=104.36.82.55:4213 , piece_id=X4IHS2XSPH52LIMNUYT7KGWMAPUH3TEMJOEEVB3JV5N45SEOH7YA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:33:09.435339213 (81h9m44.998662206s) [noise=true , address=104.36.82.52:4089 , piece_id=JSOEKLNWA3W3LOVKI3EGGKJYJ6J3UZY6KGABKQPQPXTIOOPMKVHA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:33:31.472868445 (81h10m7.03619143s) [noise=true , address=104.36.82.53:4122 , piece_id=HZZQBIKHQWLUE5EHK4KSKJGIUEG7D44RQASUJHCYXX6BICCGX6EA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:33:47.235404653 (81h10m22.798727642s) [noise=true , address=104.36.82.12:6158 , piece_id=LJZEOJVJCJAR2BRILIJI3ZVDGYMPBBTP2G3HCTN3MHNS4KYDPASQ, seg_pos={0 0}, upload=64958]: Done uploading piece. err: ecclient: upload failed (node:1SjM6Utm2YizeUbrBUD9Z7etE1nLadW5Fa1zYP7MioPeeqhdJ7, address:104.36.82.12:6158): protocol: expected piece hash; context deadline exceeded; EOF Jun 22 01:33:47.235448297 (81h10m22.798771306s) [seg_pos={0 0}, upload=64958]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jun 22 01:33:47.265442393 (81h10m22.828765401s) [upload=64958]: Done uploading segments... Jun 22 01:33:47.265476139 (81h10m22.82879914s) [upload=64958]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration:* 12.39622384s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [................................................X...........] duration: 25.423644721s FAIL (2025-06-22 01:40:41 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== oad=64949]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jun 22 01:32:13.674692303 (81h8m49.238015311s) [upload=64949]: Done uploading segments... Jun 22 01:32:13.674727219 (81h8m49.238050233s) [upload=64949]: Upload finished. err: <nil> Jun 22 01:32:47.266407995 (81h9m22.829731002s) [upload=64958]: Starting upload Jun 22 01:32:47.266524767 (81h9m22.829847771s) [upload=64958]: Done starting upload Jun 22 01:32:47.266813109 (81h9m22.830136116s) [upload=64958]: Uploading segments... Jun 22 01:32:47.266969526 (81h9m22.830292526s) [upload=64958]: Got next segment. Inline: false Jun 22 01:32:47.267075475 (81h9m22.830398475s) [upload=64958]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 22 01:32:47.267081679 (81h9m22.830404667s) [upload=64958]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 22 01:32:47.301770382 (81h9m22.865093389s) [seg_pos={0 0}, upload=64958]: Begin upload segment... Jun 22 01:32:47.301823796 (81h9m22.865146799s) [seg_pos={0 0}, upload=64958]: Done begin upload segment. Jun 22 01:32:47.301855055 (81h9m22.865178087s) [noise=true , address=104.36.82.43:6841 , piece_id=PIHZMSTO3CII7KGDLGSAILTDRSLMEBVOLBCRCZJULMCFISXIZWXA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.301902600 (81h9m22.865225606s) [noise=true , address=104.36.82.42:6811 , piece_id=A5BFEHZ3WGGK53IZ7MKI6N4CVXHOORF6L47NR3UFDCAX7LBHRHVQ, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.302234027 (81h9m22.865557029s) [noise=true , address=104.36.82.51:4084 , piece_id=D43FA7BFE54KPCWGDGDME2EGSWM7PONOVPD3DN2XY42MJFOKVDIQ, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.302496354 (81h9m22.865819363s) [noise=true , address=79.127.219.38:4023 , piece_id=BWLRCXXHBMBTNCZEVKKIIS5PB35PDQXQXCLIQB7GBLEIOWWUFBFA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.302577370 (81h9m22.865900378s) [noise=true , address=104.36.82.52:4089 , piece_id=JSOEKLNWA3W3LOVKI3EGGKJYJ6J3UZY6KGABKQPQPXTIOOPMKVHA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.302917829 (81h9m22.866240835s) [noise=true , address=104.36.82.53:4122 , piece_id=HZZQBIKHQWLUE5EHK4KSKJGIUEG7D44RQASUJHCYXX6BICCGX6EA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.303087165 (81h9m22.866410168s) [noise=true , address=77.90.31.14:4689 , piece_id=Q3HDS3BCMYDTODFRMU4XFRDCQK2QZE3ZMXVLUX4M54PDEBS7RR6A, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.303250714 (81h9m22.866573717s) [noise=true , address=79.127.205.234:5601 , piece_id=V7PMXBP2VFWBSZDG4GTLBD435T6F5DT7A43L4GQLQ5BDDTJ4WY6A, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.303579249 (81h9m22.866902255s) [noise=true , address=104.36.82.12:6158 , piece_id=LJZEOJVJCJAR2BRILIJI3ZVDGYMPBBTP2G3HCTN3MHNS4KYDPASQ, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.303599776 (81h9m22.866922784s) [noise=true , address=104.36.82.9:6023 , piece_id=SISFAUU4RPF57OLGWM2BWQWE6GRUSCMORBYHWEXETNAVM2EEMO6Q, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.304005346 (81h9m22.867328346s) [noise=true , address=45.91.116.12:5242 , piece_id=6ABT5IPYGSLFFJMBYMBYRU4EYIXABUAUHG3OVILOHBIUDS3BP27A, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.304388850 (81h9m22.867711851s) [noise=true , address=64.49.13.137:4500 , piece_id=FAE2RXD57IRP5FXA2XR6THXFW27U7IWAZ4JDV5M2WRPXKUGZ4VIA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.304881529 (81h9m22.868204536s) [noise=true , address=79.127.220.101:6048 , piece_id=AN7UQUXB42T2YXEH7ZJRZUDIW47M3RWUEOCL2VEBOGH74BZF3YKA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.305249860 (81h9m22.86857287s) [noise=true , address=104.36.82.54:4165 , piece_id=4CVOABPCWC7J7SRHZB6IN2WJJ6KGWUQ66ELUIZIFA67ETLQ7JEUQ, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.305604108 (81h9m22.868927112s) [noise=true , address=104.36.82.15:6289 , piece_id=GWL3P55BIK4IJEL5YFZKI7JCA4LADYOOHUSIQUNGJEKBJOTKMP5A, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.305930423 (81h9m22.869253425s) [noise=true , address=64.49.13.139:4476 , piece_id=P7UKBNNZGI52B3I6WKTV6MVC262FYNKYYPJUI5G47UO5S4MU7PWA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.306355311 (81h9m22.869678319s) [noise=true , address=104.36.82.55:4213 , piece_id=X4IHS2XSPH52LIMNUYT7KGWMAPUH3TEMJOEEVB3JV5N45SEOH7YA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.306788243 (81h9m22.870111252s) [noise=true , address=104.36.82.22:6539 , piece_id=GHQ2MVSTDGVYLUKLV23D6YWRR4PP7UUYHCUEQ4WWSWEMTBFO3P4A, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.392174770 (81h9m22.955497773s) [upload=64958]: Next returned nil segment Jun 22 01:32:47.392195767 (81h9m22.955518775s) [upload=64958]: Waiting for error group managing segments... Jun 22 01:32:47.480767633 (81h9m23.044090623s) [noise=true , address=104.36.82.22:6539 , piece_id=GHQ2MVSTDGVYLUKLV23D6YWRR4PP7UUYHCUEQ4WWSWEMTBFO3P4A, seg_pos={0 0}, upload=64958]: Done uploading piece. err: ecclient: upload failed (node:1QL6w8DSFR6CxDdnJmDuyVZxp6JU55VKiJofBZRfQi7q31i8Jc, address:104.36.82.22:6539): protocol: storage node overloaded, request limit: 2000 Jun 22 01:32:47.585663142 (81h9m23.148986155s) [noise=true , address=79.127.220.101:6048 , piece_id=AN7UQUXB42T2YXEH7ZJRZUDIW47M3RWUEOCL2VEBOGH74BZF3YKA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:47.640263002 (81h9m23.203586013s) [noise=true , address=45.91.116.12:5242 , piece_id=6ABT5IPYGSLFFJMBYMBYRU4EYIXABUAUHG3OVILOHBIUDS3BP27A, seg_pos={0 0}, upload=64958]: Done uploading piece. err: ecclient: upload failed (node:12JsFL62jgXmza8Md8eQ9k8cyw5V9Vz2RzN4y85ZvgHZjwrq4vQ, address:45.91.116.12:5242): protocol: storage node overloaded, request limit: 2000 Jun 22 01:32:47.984127275 (81h9m23.54745028s) [noise=true , address=64.49.13.137:4500 , piece_id=FAE2RXD57IRP5FXA2XR6THXFW27U7IWAZ4JDV5M2WRPXKUGZ4VIA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:48.100255249 (81h9m23.663578241s) [noise=true , address=77.90.31.14:4689 , piece_id=Q3HDS3BCMYDTODFRMU4XFRDCQK2QZE3ZMXVLUX4M54PDEBS7RR6A, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:48.221764035 (81h9m23.785087034s) [noise=true , address=104.36.82.54:4165 , piece_id=4CVOABPCWC7J7SRHZB6IN2WJJ6KGWUQ66ELUIZIFA67ETLQ7JEUQ, seg_pos={0 0}, upload=64958]: Done uploading piece. err: ecclient: upload failed (node:12wGUMXVEubEtki4qGnjGEsbYFB1iUauZcu3oDRKrCQsJMV3bV2, address:104.36.82.54:4165): protocol: storage node overloaded, request limit: 2000 Jun 22 01:32:48.509996527 (81h9m24.073319547s) [noise=true , address=104.36.82.9:6023 , piece_id=SISFAUU4RPF57OLGWM2BWQWE6GRUSCMORBYHWEXETNAVM2EEMO6Q, seg_pos={0 0}, upload=64958]: Done uploading piece. err: ecclient: upload failed (node:1j2S9tgWpLwmyvXZSq2qKKcbaC8r7LkzfDCaMxCejAni4CST1g, address:104.36.82.9:6023): protocol: storage node overloaded, request limit: 2000 Jun 22 01:32:48.510513169 (81h9m24.073836177s) [noise=true , address=104.36.82.42:6811 , piece_id=A5BFEHZ3WGGK53IZ7MKI6N4CVXHOORF6L47NR3UFDCAX7LBHRHVQ, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:48.565245605 (81h9m24.128568606s) [noise=true , address=79.127.205.234:5601 , piece_id=V7PMXBP2VFWBSZDG4GTLBD435T6F5DT7A43L4GQLQ5BDDTJ4WY6A, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:49.979979824 (81h9m25.543302825s) [noise=true , address=104.36.82.15:6289 , piece_id=GWL3P55BIK4IJEL5YFZKI7JCA4LADYOOHUSIQUNGJEKBJOTKMP5A, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:50.463750280 (81h9m26.02707328s) [noise=true , address=104.36.82.51:4084 , piece_id=D43FA7BFE54KPCWGDGDME2EGSWM7PONOVPD3DN2XY42MJFOKVDIQ, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:51.480599795 (81h9m27.043922798s) [noise=true , address=79.127.219.38:4023 , piece_id=BWLRCXXHBMBTNCZEVKKIIS5PB35PDQXQXCLIQB7GBLEIOWWUFBFA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:52.941200625 (81h9m28.504523627s) [noise=true , address=104.36.82.43:6841 , piece_id=PIHZMSTO3CII7KGDLGSAILTDRSLMEBVOLBCRCZJULMCFISXIZWXA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:58.281803598 (81h9m33.845126612s) [noise=true , address=64.49.13.139:4476 , piece_id=P7UKBNNZGI52B3I6WKTV6MVC262FYNKYYPJUI5G47UO5S4MU7PWA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:33:00.277271877 (81h9m35.840594867s) [noise=true , address=104.36.82.55:4213 , piece_id=X4IHS2XSPH52LIMNUYT7KGWMAPUH3TEMJOEEVB3JV5N45SEOH7YA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:33:09.435339213 (81h9m44.998662206s) [noise=true , address=104.36.82.52:4089 , piece_id=JSOEKLNWA3W3LOVKI3EGGKJYJ6J3UZY6KGABKQPQPXTIOOPMKVHA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:33:31.472868445 (81h10m7.03619143s) [noise=true , address=104.36.82.53:4122 , piece_id=HZZQBIKHQWLUE5EHK4KSKJGIUEG7D44RQASUJHCYXX6BICCGX6EA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:33:47.235404653 (81h10m22.798727642s) [noise=true , address=104.36.82.12:6158 , piece_id=LJZEOJVJCJAR2BRILIJI3ZVDGYMPBBTP2G3HCTN3MHNS4KYDPASQ, seg_pos={0 0}, upload=64958]: Done uploading piece. err: ecclient: upload failed (node:1SjM6Utm2YizeUbrBUD9Z7etE1nLadW5Fa1zYP7MioPeeqhdJ7, address:104.36.82.12:6158): protocol: expected piece hash; context deadline exceeded; EOF Jun 22 01:33:47.235448297 (81h10m22.798771306s) [seg_pos={0 0}, upload=64958]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jun 22 01:33:47.265442393 (81h10m22.828765401s) [upload=64958]: Done uploading segments... Jun 22 01:33:47.265476139 (81h10m22.82879914s) [upload=64958]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration:* 25.423676446s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.................................................X..........] duration: 24.677962764s FAIL (2025-06-22 01:40:04 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== oad=64949]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jun 22 01:32:13.674692303 (81h8m49.238015311s) [upload=64949]: Done uploading segments... Jun 22 01:32:13.674727219 (81h8m49.238050233s) [upload=64949]: Upload finished. err: <nil> Jun 22 01:32:47.266407995 (81h9m22.829731002s) [upload=64958]: Starting upload Jun 22 01:32:47.266524767 (81h9m22.829847771s) [upload=64958]: Done starting upload Jun 22 01:32:47.266813109 (81h9m22.830136116s) [upload=64958]: Uploading segments... Jun 22 01:32:47.266969526 (81h9m22.830292526s) [upload=64958]: Got next segment. Inline: false Jun 22 01:32:47.267075475 (81h9m22.830398475s) [upload=64958]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 22 01:32:47.267081679 (81h9m22.830404667s) [upload=64958]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 22 01:32:47.301770382 (81h9m22.865093389s) [seg_pos={0 0}, upload=64958]: Begin upload segment... Jun 22 01:32:47.301823796 (81h9m22.865146799s) [seg_pos={0 0}, upload=64958]: Done begin upload segment. Jun 22 01:32:47.301855055 (81h9m22.865178087s) [noise=true , address=104.36.82.43:6841 , piece_id=PIHZMSTO3CII7KGDLGSAILTDRSLMEBVOLBCRCZJULMCFISXIZWXA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.301902600 (81h9m22.865225606s) [noise=true , address=104.36.82.42:6811 , piece_id=A5BFEHZ3WGGK53IZ7MKI6N4CVXHOORF6L47NR3UFDCAX7LBHRHVQ, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.302234027 (81h9m22.865557029s) [noise=true , address=104.36.82.51:4084 , piece_id=D43FA7BFE54KPCWGDGDME2EGSWM7PONOVPD3DN2XY42MJFOKVDIQ, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.302496354 (81h9m22.865819363s) [noise=true , address=79.127.219.38:4023 , piece_id=BWLRCXXHBMBTNCZEVKKIIS5PB35PDQXQXCLIQB7GBLEIOWWUFBFA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.302577370 (81h9m22.865900378s) [noise=true , address=104.36.82.52:4089 , piece_id=JSOEKLNWA3W3LOVKI3EGGKJYJ6J3UZY6KGABKQPQPXTIOOPMKVHA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.302917829 (81h9m22.866240835s) [noise=true , address=104.36.82.53:4122 , piece_id=HZZQBIKHQWLUE5EHK4KSKJGIUEG7D44RQASUJHCYXX6BICCGX6EA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.303087165 (81h9m22.866410168s) [noise=true , address=77.90.31.14:4689 , piece_id=Q3HDS3BCMYDTODFRMU4XFRDCQK2QZE3ZMXVLUX4M54PDEBS7RR6A, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.303250714 (81h9m22.866573717s) [noise=true , address=79.127.205.234:5601 , piece_id=V7PMXBP2VFWBSZDG4GTLBD435T6F5DT7A43L4GQLQ5BDDTJ4WY6A, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.303579249 (81h9m22.866902255s) [noise=true , address=104.36.82.12:6158 , piece_id=LJZEOJVJCJAR2BRILIJI3ZVDGYMPBBTP2G3HCTN3MHNS4KYDPASQ, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.303599776 (81h9m22.866922784s) [noise=true , address=104.36.82.9:6023 , piece_id=SISFAUU4RPF57OLGWM2BWQWE6GRUSCMORBYHWEXETNAVM2EEMO6Q, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.304005346 (81h9m22.867328346s) [noise=true , address=45.91.116.12:5242 , piece_id=6ABT5IPYGSLFFJMBYMBYRU4EYIXABUAUHG3OVILOHBIUDS3BP27A, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.304388850 (81h9m22.867711851s) [noise=true , address=64.49.13.137:4500 , piece_id=FAE2RXD57IRP5FXA2XR6THXFW27U7IWAZ4JDV5M2WRPXKUGZ4VIA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.304881529 (81h9m22.868204536s) [noise=true , address=79.127.220.101:6048 , piece_id=AN7UQUXB42T2YXEH7ZJRZUDIW47M3RWUEOCL2VEBOGH74BZF3YKA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.305249860 (81h9m22.86857287s) [noise=true , address=104.36.82.54:4165 , piece_id=4CVOABPCWC7J7SRHZB6IN2WJJ6KGWUQ66ELUIZIFA67ETLQ7JEUQ, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.305604108 (81h9m22.868927112s) [noise=true , address=104.36.82.15:6289 , piece_id=GWL3P55BIK4IJEL5YFZKI7JCA4LADYOOHUSIQUNGJEKBJOTKMP5A, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.305930423 (81h9m22.869253425s) [noise=true , address=64.49.13.139:4476 , piece_id=P7UKBNNZGI52B3I6WKTV6MVC262FYNKYYPJUI5G47UO5S4MU7PWA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.306355311 (81h9m22.869678319s) [noise=true , address=104.36.82.55:4213 , piece_id=X4IHS2XSPH52LIMNUYT7KGWMAPUH3TEMJOEEVB3JV5N45SEOH7YA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.306788243 (81h9m22.870111252s) [noise=true , address=104.36.82.22:6539 , piece_id=GHQ2MVSTDGVYLUKLV23D6YWRR4PP7UUYHCUEQ4WWSWEMTBFO3P4A, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.392174770 (81h9m22.955497773s) [upload=64958]: Next returned nil segment Jun 22 01:32:47.392195767 (81h9m22.955518775s) [upload=64958]: Waiting for error group managing segments... Jun 22 01:32:47.480767633 (81h9m23.044090623s) [noise=true , address=104.36.82.22:6539 , piece_id=GHQ2MVSTDGVYLUKLV23D6YWRR4PP7UUYHCUEQ4WWSWEMTBFO3P4A, seg_pos={0 0}, upload=64958]: Done uploading piece. err: ecclient: upload failed (node:1QL6w8DSFR6CxDdnJmDuyVZxp6JU55VKiJofBZRfQi7q31i8Jc, address:104.36.82.22:6539): protocol: storage node overloaded, request limit: 2000 Jun 22 01:32:47.585663142 (81h9m23.148986155s) [noise=true , address=79.127.220.101:6048 , piece_id=AN7UQUXB42T2YXEH7ZJRZUDIW47M3RWUEOCL2VEBOGH74BZF3YKA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:47.640263002 (81h9m23.203586013s) [noise=true , address=45.91.116.12:5242 , piece_id=6ABT5IPYGSLFFJMBYMBYRU4EYIXABUAUHG3OVILOHBIUDS3BP27A, seg_pos={0 0}, upload=64958]: Done uploading piece. err: ecclient: upload failed (node:12JsFL62jgXmza8Md8eQ9k8cyw5V9Vz2RzN4y85ZvgHZjwrq4vQ, address:45.91.116.12:5242): protocol: storage node overloaded, request limit: 2000 Jun 22 01:32:47.984127275 (81h9m23.54745028s) [noise=true , address=64.49.13.137:4500 , piece_id=FAE2RXD57IRP5FXA2XR6THXFW27U7IWAZ4JDV5M2WRPXKUGZ4VIA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:48.100255249 (81h9m23.663578241s) [noise=true , address=77.90.31.14:4689 , piece_id=Q3HDS3BCMYDTODFRMU4XFRDCQK2QZE3ZMXVLUX4M54PDEBS7RR6A, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:48.221764035 (81h9m23.785087034s) [noise=true , address=104.36.82.54:4165 , piece_id=4CVOABPCWC7J7SRHZB6IN2WJJ6KGWUQ66ELUIZIFA67ETLQ7JEUQ, seg_pos={0 0}, upload=64958]: Done uploading piece. err: ecclient: upload failed (node:12wGUMXVEubEtki4qGnjGEsbYFB1iUauZcu3oDRKrCQsJMV3bV2, address:104.36.82.54:4165): protocol: storage node overloaded, request limit: 2000 Jun 22 01:32:48.509996527 (81h9m24.073319547s) [noise=true , address=104.36.82.9:6023 , piece_id=SISFAUU4RPF57OLGWM2BWQWE6GRUSCMORBYHWEXETNAVM2EEMO6Q, seg_pos={0 0}, upload=64958]: Done uploading piece. err: ecclient: upload failed (node:1j2S9tgWpLwmyvXZSq2qKKcbaC8r7LkzfDCaMxCejAni4CST1g, address:104.36.82.9:6023): protocol: storage node overloaded, request limit: 2000 Jun 22 01:32:48.510513169 (81h9m24.073836177s) [noise=true , address=104.36.82.42:6811 , piece_id=A5BFEHZ3WGGK53IZ7MKI6N4CVXHOORF6L47NR3UFDCAX7LBHRHVQ, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:48.565245605 (81h9m24.128568606s) [noise=true , address=79.127.205.234:5601 , piece_id=V7PMXBP2VFWBSZDG4GTLBD435T6F5DT7A43L4GQLQ5BDDTJ4WY6A, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:49.979979824 (81h9m25.543302825s) [noise=true , address=104.36.82.15:6289 , piece_id=GWL3P55BIK4IJEL5YFZKI7JCA4LADYOOHUSIQUNGJEKBJOTKMP5A, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:50.463750280 (81h9m26.02707328s) [noise=true , address=104.36.82.51:4084 , piece_id=D43FA7BFE54KPCWGDGDME2EGSWM7PONOVPD3DN2XY42MJFOKVDIQ, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:51.480599795 (81h9m27.043922798s) [noise=true , address=79.127.219.38:4023 , piece_id=BWLRCXXHBMBTNCZEVKKIIS5PB35PDQXQXCLIQB7GBLEIOWWUFBFA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:52.941200625 (81h9m28.504523627s) [noise=true , address=104.36.82.43:6841 , piece_id=PIHZMSTO3CII7KGDLGSAILTDRSLMEBVOLBCRCZJULMCFISXIZWXA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:58.281803598 (81h9m33.845126612s) [noise=true , address=64.49.13.139:4476 , piece_id=P7UKBNNZGI52B3I6WKTV6MVC262FYNKYYPJUI5G47UO5S4MU7PWA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:33:00.277271877 (81h9m35.840594867s) [noise=true , address=104.36.82.55:4213 , piece_id=X4IHS2XSPH52LIMNUYT7KGWMAPUH3TEMJOEEVB3JV5N45SEOH7YA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:33:09.435339213 (81h9m44.998662206s) [noise=true , address=104.36.82.52:4089 , piece_id=JSOEKLNWA3W3LOVKI3EGGKJYJ6J3UZY6KGABKQPQPXTIOOPMKVHA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:33:31.472868445 (81h10m7.03619143s) [noise=true , address=104.36.82.53:4122 , piece_id=HZZQBIKHQWLUE5EHK4KSKJGIUEG7D44RQASUJHCYXX6BICCGX6EA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:33:47.235404653 (81h10m22.798727642s) [noise=true , address=104.36.82.12:6158 , piece_id=LJZEOJVJCJAR2BRILIJI3ZVDGYMPBBTP2G3HCTN3MHNS4KYDPASQ, seg_pos={0 0}, upload=64958]: Done uploading piece. err: ecclient: upload failed (node:1SjM6Utm2YizeUbrBUD9Z7etE1nLadW5Fa1zYP7MioPeeqhdJ7, address:104.36.82.12:6158): protocol: expected piece hash; context deadline exceeded; EOF Jun 22 01:33:47.235448297 (81h10m22.798771306s) [seg_pos={0 0}, upload=64958]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jun 22 01:33:47.265442393 (81h10m22.828765401s) [upload=64958]: Done uploading segments... Jun 22 01:33:47.265476139 (81h10m22.82879914s) [upload=64958]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration:* 24.677995058s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [..................................................X.........] duration: 23.545069395s FAIL (2025-06-22 01:39:39 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== oad=64949]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jun 22 01:32:13.674692303 (81h8m49.238015311s) [upload=64949]: Done uploading segments... Jun 22 01:32:13.674727219 (81h8m49.238050233s) [upload=64949]: Upload finished. err: <nil> Jun 22 01:32:47.266407995 (81h9m22.829731002s) [upload=64958]: Starting upload Jun 22 01:32:47.266524767 (81h9m22.829847771s) [upload=64958]: Done starting upload Jun 22 01:32:47.266813109 (81h9m22.830136116s) [upload=64958]: Uploading segments... Jun 22 01:32:47.266969526 (81h9m22.830292526s) [upload=64958]: Got next segment. Inline: false Jun 22 01:32:47.267075475 (81h9m22.830398475s) [upload=64958]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 22 01:32:47.267081679 (81h9m22.830404667s) [upload=64958]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 22 01:32:47.301770382 (81h9m22.865093389s) [seg_pos={0 0}, upload=64958]: Begin upload segment... Jun 22 01:32:47.301823796 (81h9m22.865146799s) [seg_pos={0 0}, upload=64958]: Done begin upload segment. Jun 22 01:32:47.301855055 (81h9m22.865178087s) [noise=true , address=104.36.82.43:6841 , piece_id=PIHZMSTO3CII7KGDLGSAILTDRSLMEBVOLBCRCZJULMCFISXIZWXA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.301902600 (81h9m22.865225606s) [noise=true , address=104.36.82.42:6811 , piece_id=A5BFEHZ3WGGK53IZ7MKI6N4CVXHOORF6L47NR3UFDCAX7LBHRHVQ, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.302234027 (81h9m22.865557029s) [noise=true , address=104.36.82.51:4084 , piece_id=D43FA7BFE54KPCWGDGDME2EGSWM7PONOVPD3DN2XY42MJFOKVDIQ, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.302496354 (81h9m22.865819363s) [noise=true , address=79.127.219.38:4023 , piece_id=BWLRCXXHBMBTNCZEVKKIIS5PB35PDQXQXCLIQB7GBLEIOWWUFBFA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.302577370 (81h9m22.865900378s) [noise=true , address=104.36.82.52:4089 , piece_id=JSOEKLNWA3W3LOVKI3EGGKJYJ6J3UZY6KGABKQPQPXTIOOPMKVHA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.302917829 (81h9m22.866240835s) [noise=true , address=104.36.82.53:4122 , piece_id=HZZQBIKHQWLUE5EHK4KSKJGIUEG7D44RQASUJHCYXX6BICCGX6EA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.303087165 (81h9m22.866410168s) [noise=true , address=77.90.31.14:4689 , piece_id=Q3HDS3BCMYDTODFRMU4XFRDCQK2QZE3ZMXVLUX4M54PDEBS7RR6A, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.303250714 (81h9m22.866573717s) [noise=true , address=79.127.205.234:5601 , piece_id=V7PMXBP2VFWBSZDG4GTLBD435T6F5DT7A43L4GQLQ5BDDTJ4WY6A, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.303579249 (81h9m22.866902255s) [noise=true , address=104.36.82.12:6158 , piece_id=LJZEOJVJCJAR2BRILIJI3ZVDGYMPBBTP2G3HCTN3MHNS4KYDPASQ, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.303599776 (81h9m22.866922784s) [noise=true , address=104.36.82.9:6023 , piece_id=SISFAUU4RPF57OLGWM2BWQWE6GRUSCMORBYHWEXETNAVM2EEMO6Q, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.304005346 (81h9m22.867328346s) [noise=true , address=45.91.116.12:5242 , piece_id=6ABT5IPYGSLFFJMBYMBYRU4EYIXABUAUHG3OVILOHBIUDS3BP27A, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.304388850 (81h9m22.867711851s) [noise=true , address=64.49.13.137:4500 , piece_id=FAE2RXD57IRP5FXA2XR6THXFW27U7IWAZ4JDV5M2WRPXKUGZ4VIA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.304881529 (81h9m22.868204536s) [noise=true , address=79.127.220.101:6048 , piece_id=AN7UQUXB42T2YXEH7ZJRZUDIW47M3RWUEOCL2VEBOGH74BZF3YKA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.305249860 (81h9m22.86857287s) [noise=true , address=104.36.82.54:4165 , piece_id=4CVOABPCWC7J7SRHZB6IN2WJJ6KGWUQ66ELUIZIFA67ETLQ7JEUQ, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.305604108 (81h9m22.868927112s) [noise=true , address=104.36.82.15:6289 , piece_id=GWL3P55BIK4IJEL5YFZKI7JCA4LADYOOHUSIQUNGJEKBJOTKMP5A, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.305930423 (81h9m22.869253425s) [noise=true , address=64.49.13.139:4476 , piece_id=P7UKBNNZGI52B3I6WKTV6MVC262FYNKYYPJUI5G47UO5S4MU7PWA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.306355311 (81h9m22.869678319s) [noise=true , address=104.36.82.55:4213 , piece_id=X4IHS2XSPH52LIMNUYT7KGWMAPUH3TEMJOEEVB3JV5N45SEOH7YA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.306788243 (81h9m22.870111252s) [noise=true , address=104.36.82.22:6539 , piece_id=GHQ2MVSTDGVYLUKLV23D6YWRR4PP7UUYHCUEQ4WWSWEMTBFO3P4A, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.392174770 (81h9m22.955497773s) [upload=64958]: Next returned nil segment Jun 22 01:32:47.392195767 (81h9m22.955518775s) [upload=64958]: Waiting for error group managing segments... Jun 22 01:32:47.480767633 (81h9m23.044090623s) [noise=true , address=104.36.82.22:6539 , piece_id=GHQ2MVSTDGVYLUKLV23D6YWRR4PP7UUYHCUEQ4WWSWEMTBFO3P4A, seg_pos={0 0}, upload=64958]: Done uploading piece. err: ecclient: upload failed (node:1QL6w8DSFR6CxDdnJmDuyVZxp6JU55VKiJofBZRfQi7q31i8Jc, address:104.36.82.22:6539): protocol: storage node overloaded, request limit: 2000 Jun 22 01:32:47.585663142 (81h9m23.148986155s) [noise=true , address=79.127.220.101:6048 , piece_id=AN7UQUXB42T2YXEH7ZJRZUDIW47M3RWUEOCL2VEBOGH74BZF3YKA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:47.640263002 (81h9m23.203586013s) [noise=true , address=45.91.116.12:5242 , piece_id=6ABT5IPYGSLFFJMBYMBYRU4EYIXABUAUHG3OVILOHBIUDS3BP27A, seg_pos={0 0}, upload=64958]: Done uploading piece. err: ecclient: upload failed (node:12JsFL62jgXmza8Md8eQ9k8cyw5V9Vz2RzN4y85ZvgHZjwrq4vQ, address:45.91.116.12:5242): protocol: storage node overloaded, request limit: 2000 Jun 22 01:32:47.984127275 (81h9m23.54745028s) [noise=true , address=64.49.13.137:4500 , piece_id=FAE2RXD57IRP5FXA2XR6THXFW27U7IWAZ4JDV5M2WRPXKUGZ4VIA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:48.100255249 (81h9m23.663578241s) [noise=true , address=77.90.31.14:4689 , piece_id=Q3HDS3BCMYDTODFRMU4XFRDCQK2QZE3ZMXVLUX4M54PDEBS7RR6A, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:48.221764035 (81h9m23.785087034s) [noise=true , address=104.36.82.54:4165 , piece_id=4CVOABPCWC7J7SRHZB6IN2WJJ6KGWUQ66ELUIZIFA67ETLQ7JEUQ, seg_pos={0 0}, upload=64958]: Done uploading piece. err: ecclient: upload failed (node:12wGUMXVEubEtki4qGnjGEsbYFB1iUauZcu3oDRKrCQsJMV3bV2, address:104.36.82.54:4165): protocol: storage node overloaded, request limit: 2000 Jun 22 01:32:48.509996527 (81h9m24.073319547s) [noise=true , address=104.36.82.9:6023 , piece_id=SISFAUU4RPF57OLGWM2BWQWE6GRUSCMORBYHWEXETNAVM2EEMO6Q, seg_pos={0 0}, upload=64958]: Done uploading piece. err: ecclient: upload failed (node:1j2S9tgWpLwmyvXZSq2qKKcbaC8r7LkzfDCaMxCejAni4CST1g, address:104.36.82.9:6023): protocol: storage node overloaded, request limit: 2000 Jun 22 01:32:48.510513169 (81h9m24.073836177s) [noise=true , address=104.36.82.42:6811 , piece_id=A5BFEHZ3WGGK53IZ7MKI6N4CVXHOORF6L47NR3UFDCAX7LBHRHVQ, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:48.565245605 (81h9m24.128568606s) [noise=true , address=79.127.205.234:5601 , piece_id=V7PMXBP2VFWBSZDG4GTLBD435T6F5DT7A43L4GQLQ5BDDTJ4WY6A, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:49.979979824 (81h9m25.543302825s) [noise=true , address=104.36.82.15:6289 , piece_id=GWL3P55BIK4IJEL5YFZKI7JCA4LADYOOHUSIQUNGJEKBJOTKMP5A, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:50.463750280 (81h9m26.02707328s) [noise=true , address=104.36.82.51:4084 , piece_id=D43FA7BFE54KPCWGDGDME2EGSWM7PONOVPD3DN2XY42MJFOKVDIQ, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:51.480599795 (81h9m27.043922798s) [noise=true , address=79.127.219.38:4023 , piece_id=BWLRCXXHBMBTNCZEVKKIIS5PB35PDQXQXCLIQB7GBLEIOWWUFBFA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:52.941200625 (81h9m28.504523627s) [noise=true , address=104.36.82.43:6841 , piece_id=PIHZMSTO3CII7KGDLGSAILTDRSLMEBVOLBCRCZJULMCFISXIZWXA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:58.281803598 (81h9m33.845126612s) [noise=true , address=64.49.13.139:4476 , piece_id=P7UKBNNZGI52B3I6WKTV6MVC262FYNKYYPJUI5G47UO5S4MU7PWA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:33:00.277271877 (81h9m35.840594867s) [noise=true , address=104.36.82.55:4213 , piece_id=X4IHS2XSPH52LIMNUYT7KGWMAPUH3TEMJOEEVB3JV5N45SEOH7YA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:33:09.435339213 (81h9m44.998662206s) [noise=true , address=104.36.82.52:4089 , piece_id=JSOEKLNWA3W3LOVKI3EGGKJYJ6J3UZY6KGABKQPQPXTIOOPMKVHA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:33:31.472868445 (81h10m7.03619143s) [noise=true , address=104.36.82.53:4122 , piece_id=HZZQBIKHQWLUE5EHK4KSKJGIUEG7D44RQASUJHCYXX6BICCGX6EA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:33:47.235404653 (81h10m22.798727642s) [noise=true , address=104.36.82.12:6158 , piece_id=LJZEOJVJCJAR2BRILIJI3ZVDGYMPBBTP2G3HCTN3MHNS4KYDPASQ, seg_pos={0 0}, upload=64958]: Done uploading piece. err: ecclient: upload failed (node:1SjM6Utm2YizeUbrBUD9Z7etE1nLadW5Fa1zYP7MioPeeqhdJ7, address:104.36.82.12:6158): protocol: expected piece hash; context deadline exceeded; EOF Jun 22 01:33:47.235448297 (81h10m22.798771306s) [seg_pos={0 0}, upload=64958]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jun 22 01:33:47.265442393 (81h10m22.828765401s) [upload=64958]: Done uploading segments... Jun 22 01:33:47.265476139 (81h10m22.82879914s) [upload=64958]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration:* 23.545102473s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [...................................................X........] duration: 7.413740935s FAIL (2025-06-22 01:38:08 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== oad=64949]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jun 22 01:32:13.674692303 (81h8m49.238015311s) [upload=64949]: Done uploading segments... Jun 22 01:32:13.674727219 (81h8m49.238050233s) [upload=64949]: Upload finished. err: <nil> Jun 22 01:32:47.266407995 (81h9m22.829731002s) [upload=64958]: Starting upload Jun 22 01:32:47.266524767 (81h9m22.829847771s) [upload=64958]: Done starting upload Jun 22 01:32:47.266813109 (81h9m22.830136116s) [upload=64958]: Uploading segments... Jun 22 01:32:47.266969526 (81h9m22.830292526s) [upload=64958]: Got next segment. Inline: false Jun 22 01:32:47.267075475 (81h9m22.830398475s) [upload=64958]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 22 01:32:47.267081679 (81h9m22.830404667s) [upload=64958]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 22 01:32:47.301770382 (81h9m22.865093389s) [seg_pos={0 0}, upload=64958]: Begin upload segment... Jun 22 01:32:47.301823796 (81h9m22.865146799s) [seg_pos={0 0}, upload=64958]: Done begin upload segment. Jun 22 01:32:47.301855055 (81h9m22.865178087s) [noise=true , address=104.36.82.43:6841 , piece_id=PIHZMSTO3CII7KGDLGSAILTDRSLMEBVOLBCRCZJULMCFISXIZWXA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.301902600 (81h9m22.865225606s) [noise=true , address=104.36.82.42:6811 , piece_id=A5BFEHZ3WGGK53IZ7MKI6N4CVXHOORF6L47NR3UFDCAX7LBHRHVQ, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.302234027 (81h9m22.865557029s) [noise=true , address=104.36.82.51:4084 , piece_id=D43FA7BFE54KPCWGDGDME2EGSWM7PONOVPD3DN2XY42MJFOKVDIQ, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.302496354 (81h9m22.865819363s) [noise=true , address=79.127.219.38:4023 , piece_id=BWLRCXXHBMBTNCZEVKKIIS5PB35PDQXQXCLIQB7GBLEIOWWUFBFA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.302577370 (81h9m22.865900378s) [noise=true , address=104.36.82.52:4089 , piece_id=JSOEKLNWA3W3LOVKI3EGGKJYJ6J3UZY6KGABKQPQPXTIOOPMKVHA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.302917829 (81h9m22.866240835s) [noise=true , address=104.36.82.53:4122 , piece_id=HZZQBIKHQWLUE5EHK4KSKJGIUEG7D44RQASUJHCYXX6BICCGX6EA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.303087165 (81h9m22.866410168s) [noise=true , address=77.90.31.14:4689 , piece_id=Q3HDS3BCMYDTODFRMU4XFRDCQK2QZE3ZMXVLUX4M54PDEBS7RR6A, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.303250714 (81h9m22.866573717s) [noise=true , address=79.127.205.234:5601 , piece_id=V7PMXBP2VFWBSZDG4GTLBD435T6F5DT7A43L4GQLQ5BDDTJ4WY6A, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.303579249 (81h9m22.866902255s) [noise=true , address=104.36.82.12:6158 , piece_id=LJZEOJVJCJAR2BRILIJI3ZVDGYMPBBTP2G3HCTN3MHNS4KYDPASQ, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.303599776 (81h9m22.866922784s) [noise=true , address=104.36.82.9:6023 , piece_id=SISFAUU4RPF57OLGWM2BWQWE6GRUSCMORBYHWEXETNAVM2EEMO6Q, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.304005346 (81h9m22.867328346s) [noise=true , address=45.91.116.12:5242 , piece_id=6ABT5IPYGSLFFJMBYMBYRU4EYIXABUAUHG3OVILOHBIUDS3BP27A, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.304388850 (81h9m22.867711851s) [noise=true , address=64.49.13.137:4500 , piece_id=FAE2RXD57IRP5FXA2XR6THXFW27U7IWAZ4JDV5M2WRPXKUGZ4VIA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.304881529 (81h9m22.868204536s) [noise=true , address=79.127.220.101:6048 , piece_id=AN7UQUXB42T2YXEH7ZJRZUDIW47M3RWUEOCL2VEBOGH74BZF3YKA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.305249860 (81h9m22.86857287s) [noise=true , address=104.36.82.54:4165 , piece_id=4CVOABPCWC7J7SRHZB6IN2WJJ6KGWUQ66ELUIZIFA67ETLQ7JEUQ, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.305604108 (81h9m22.868927112s) [noise=true , address=104.36.82.15:6289 , piece_id=GWL3P55BIK4IJEL5YFZKI7JCA4LADYOOHUSIQUNGJEKBJOTKMP5A, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.305930423 (81h9m22.869253425s) [noise=true , address=64.49.13.139:4476 , piece_id=P7UKBNNZGI52B3I6WKTV6MVC262FYNKYYPJUI5G47UO5S4MU7PWA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.306355311 (81h9m22.869678319s) [noise=true , address=104.36.82.55:4213 , piece_id=X4IHS2XSPH52LIMNUYT7KGWMAPUH3TEMJOEEVB3JV5N45SEOH7YA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.306788243 (81h9m22.870111252s) [noise=true , address=104.36.82.22:6539 , piece_id=GHQ2MVSTDGVYLUKLV23D6YWRR4PP7UUYHCUEQ4WWSWEMTBFO3P4A, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.392174770 (81h9m22.955497773s) [upload=64958]: Next returned nil segment Jun 22 01:32:47.392195767 (81h9m22.955518775s) [upload=64958]: Waiting for error group managing segments... Jun 22 01:32:47.480767633 (81h9m23.044090623s) [noise=true , address=104.36.82.22:6539 , piece_id=GHQ2MVSTDGVYLUKLV23D6YWRR4PP7UUYHCUEQ4WWSWEMTBFO3P4A, seg_pos={0 0}, upload=64958]: Done uploading piece. err: ecclient: upload failed (node:1QL6w8DSFR6CxDdnJmDuyVZxp6JU55VKiJofBZRfQi7q31i8Jc, address:104.36.82.22:6539): protocol: storage node overloaded, request limit: 2000 Jun 22 01:32:47.585663142 (81h9m23.148986155s) [noise=true , address=79.127.220.101:6048 , piece_id=AN7UQUXB42T2YXEH7ZJRZUDIW47M3RWUEOCL2VEBOGH74BZF3YKA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:47.640263002 (81h9m23.203586013s) [noise=true , address=45.91.116.12:5242 , piece_id=6ABT5IPYGSLFFJMBYMBYRU4EYIXABUAUHG3OVILOHBIUDS3BP27A, seg_pos={0 0}, upload=64958]: Done uploading piece. err: ecclient: upload failed (node:12JsFL62jgXmza8Md8eQ9k8cyw5V9Vz2RzN4y85ZvgHZjwrq4vQ, address:45.91.116.12:5242): protocol: storage node overloaded, request limit: 2000 Jun 22 01:32:47.984127275 (81h9m23.54745028s) [noise=true , address=64.49.13.137:4500 , piece_id=FAE2RXD57IRP5FXA2XR6THXFW27U7IWAZ4JDV5M2WRPXKUGZ4VIA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:48.100255249 (81h9m23.663578241s) [noise=true , address=77.90.31.14:4689 , piece_id=Q3HDS3BCMYDTODFRMU4XFRDCQK2QZE3ZMXVLUX4M54PDEBS7RR6A, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:48.221764035 (81h9m23.785087034s) [noise=true , address=104.36.82.54:4165 , piece_id=4CVOABPCWC7J7SRHZB6IN2WJJ6KGWUQ66ELUIZIFA67ETLQ7JEUQ, seg_pos={0 0}, upload=64958]: Done uploading piece. err: ecclient: upload failed (node:12wGUMXVEubEtki4qGnjGEsbYFB1iUauZcu3oDRKrCQsJMV3bV2, address:104.36.82.54:4165): protocol: storage node overloaded, request limit: 2000 Jun 22 01:32:48.509996527 (81h9m24.073319547s) [noise=true , address=104.36.82.9:6023 , piece_id=SISFAUU4RPF57OLGWM2BWQWE6GRUSCMORBYHWEXETNAVM2EEMO6Q, seg_pos={0 0}, upload=64958]: Done uploading piece. err: ecclient: upload failed (node:1j2S9tgWpLwmyvXZSq2qKKcbaC8r7LkzfDCaMxCejAni4CST1g, address:104.36.82.9:6023): protocol: storage node overloaded, request limit: 2000 Jun 22 01:32:48.510513169 (81h9m24.073836177s) [noise=true , address=104.36.82.42:6811 , piece_id=A5BFEHZ3WGGK53IZ7MKI6N4CVXHOORF6L47NR3UFDCAX7LBHRHVQ, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:48.565245605 (81h9m24.128568606s) [noise=true , address=79.127.205.234:5601 , piece_id=V7PMXBP2VFWBSZDG4GTLBD435T6F5DT7A43L4GQLQ5BDDTJ4WY6A, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:49.979979824 (81h9m25.543302825s) [noise=true , address=104.36.82.15:6289 , piece_id=GWL3P55BIK4IJEL5YFZKI7JCA4LADYOOHUSIQUNGJEKBJOTKMP5A, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:50.463750280 (81h9m26.02707328s) [noise=true , address=104.36.82.51:4084 , piece_id=D43FA7BFE54KPCWGDGDME2EGSWM7PONOVPD3DN2XY42MJFOKVDIQ, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:51.480599795 (81h9m27.043922798s) [noise=true , address=79.127.219.38:4023 , piece_id=BWLRCXXHBMBTNCZEVKKIIS5PB35PDQXQXCLIQB7GBLEIOWWUFBFA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:52.941200625 (81h9m28.504523627s) [noise=true , address=104.36.82.43:6841 , piece_id=PIHZMSTO3CII7KGDLGSAILTDRSLMEBVOLBCRCZJULMCFISXIZWXA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:58.281803598 (81h9m33.845126612s) [noise=true , address=64.49.13.139:4476 , piece_id=P7UKBNNZGI52B3I6WKTV6MVC262FYNKYYPJUI5G47UO5S4MU7PWA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:33:00.277271877 (81h9m35.840594867s) [noise=true , address=104.36.82.55:4213 , piece_id=X4IHS2XSPH52LIMNUYT7KGWMAPUH3TEMJOEEVB3JV5N45SEOH7YA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:33:09.435339213 (81h9m44.998662206s) [noise=true , address=104.36.82.52:4089 , piece_id=JSOEKLNWA3W3LOVKI3EGGKJYJ6J3UZY6KGABKQPQPXTIOOPMKVHA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:33:31.472868445 (81h10m7.03619143s) [noise=true , address=104.36.82.53:4122 , piece_id=HZZQBIKHQWLUE5EHK4KSKJGIUEG7D44RQASUJHCYXX6BICCGX6EA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:33:47.235404653 (81h10m22.798727642s) [noise=true , address=104.36.82.12:6158 , piece_id=LJZEOJVJCJAR2BRILIJI3ZVDGYMPBBTP2G3HCTN3MHNS4KYDPASQ, seg_pos={0 0}, upload=64958]: Done uploading piece. err: ecclient: upload failed (node:1SjM6Utm2YizeUbrBUD9Z7etE1nLadW5Fa1zYP7MioPeeqhdJ7, address:104.36.82.12:6158): protocol: expected piece hash; context deadline exceeded; EOF Jun 22 01:33:47.235448297 (81h10m22.798771306s) [seg_pos={0 0}, upload=64958]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jun 22 01:33:47.265442393 (81h10m22.828765401s) [upload=64958]: Done uploading segments... Jun 22 01:33:47.265476139 (81h10m22.82879914s) [upload=64958]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration:* 7.413859897s core uplink us-select-1 us1-placement-20

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)