Orbiter History

checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [............X.................X..........................X..] duration: 12.95463393s FAIL (2025-06-22 02:22:01 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 12.95465489s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [.............X.................X..........................X.] duration: 11.978917903s FAIL (2025-06-22 02:21:24 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 11.978938953s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [..............X.................X..........................X] duration: 1m0.00044011s FAIL (2025-06-22 02:21:05 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1m0.00046186s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [...............X.................X..........................] duration: 19.942406171s FAIL (2025-06-22 02:19:55 UTC) orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded Duration: 19.942443593s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [................X.................X.........................] duration: 13.766676867s FAIL (2025-06-22 02:18:40 UTC) orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded Duration: 13.766702344s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [X...........................................................] duration: 13.046645337s FAIL (2025-06-22 02:18:34 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== oad=64949]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jun 22 01:32:13.674692303 (81h8m49.238015311s) [upload=64949]: Done uploading segments... Jun 22 01:32:13.674727219 (81h8m49.238050233s) [upload=64949]: Upload finished. err: <nil> Jun 22 01:32:47.266407995 (81h9m22.829731002s) [upload=64958]: Starting upload Jun 22 01:32:47.266524767 (81h9m22.829847771s) [upload=64958]: Done starting upload Jun 22 01:32:47.266813109 (81h9m22.830136116s) [upload=64958]: Uploading segments... Jun 22 01:32:47.266969526 (81h9m22.830292526s) [upload=64958]: Got next segment. Inline: false Jun 22 01:32:47.267075475 (81h9m22.830398475s) [upload=64958]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 22 01:32:47.267081679 (81h9m22.830404667s) [upload=64958]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 22 01:32:47.301770382 (81h9m22.865093389s) [seg_pos={0 0}, upload=64958]: Begin upload segment... Jun 22 01:32:47.301823796 (81h9m22.865146799s) [seg_pos={0 0}, upload=64958]: Done begin upload segment. Jun 22 01:32:47.301855055 (81h9m22.865178087s) [noise=true , address=104.36.82.43:6841 , piece_id=PIHZMSTO3CII7KGDLGSAILTDRSLMEBVOLBCRCZJULMCFISXIZWXA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.301902600 (81h9m22.865225606s) [noise=true , address=104.36.82.42:6811 , piece_id=A5BFEHZ3WGGK53IZ7MKI6N4CVXHOORF6L47NR3UFDCAX7LBHRHVQ, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.302234027 (81h9m22.865557029s) [noise=true , address=104.36.82.51:4084 , piece_id=D43FA7BFE54KPCWGDGDME2EGSWM7PONOVPD3DN2XY42MJFOKVDIQ, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.302496354 (81h9m22.865819363s) [noise=true , address=79.127.219.38:4023 , piece_id=BWLRCXXHBMBTNCZEVKKIIS5PB35PDQXQXCLIQB7GBLEIOWWUFBFA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.302577370 (81h9m22.865900378s) [noise=true , address=104.36.82.52:4089 , piece_id=JSOEKLNWA3W3LOVKI3EGGKJYJ6J3UZY6KGABKQPQPXTIOOPMKVHA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.302917829 (81h9m22.866240835s) [noise=true , address=104.36.82.53:4122 , piece_id=HZZQBIKHQWLUE5EHK4KSKJGIUEG7D44RQASUJHCYXX6BICCGX6EA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.303087165 (81h9m22.866410168s) [noise=true , address=77.90.31.14:4689 , piece_id=Q3HDS3BCMYDTODFRMU4XFRDCQK2QZE3ZMXVLUX4M54PDEBS7RR6A, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.303250714 (81h9m22.866573717s) [noise=true , address=79.127.205.234:5601 , piece_id=V7PMXBP2VFWBSZDG4GTLBD435T6F5DT7A43L4GQLQ5BDDTJ4WY6A, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.303579249 (81h9m22.866902255s) [noise=true , address=104.36.82.12:6158 , piece_id=LJZEOJVJCJAR2BRILIJI3ZVDGYMPBBTP2G3HCTN3MHNS4KYDPASQ, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.303599776 (81h9m22.866922784s) [noise=true , address=104.36.82.9:6023 , piece_id=SISFAUU4RPF57OLGWM2BWQWE6GRUSCMORBYHWEXETNAVM2EEMO6Q, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.304005346 (81h9m22.867328346s) [noise=true , address=45.91.116.12:5242 , piece_id=6ABT5IPYGSLFFJMBYMBYRU4EYIXABUAUHG3OVILOHBIUDS3BP27A, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.304388850 (81h9m22.867711851s) [noise=true , address=64.49.13.137:4500 , piece_id=FAE2RXD57IRP5FXA2XR6THXFW27U7IWAZ4JDV5M2WRPXKUGZ4VIA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.304881529 (81h9m22.868204536s) [noise=true , address=79.127.220.101:6048 , piece_id=AN7UQUXB42T2YXEH7ZJRZUDIW47M3RWUEOCL2VEBOGH74BZF3YKA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.305249860 (81h9m22.86857287s) [noise=true , address=104.36.82.54:4165 , piece_id=4CVOABPCWC7J7SRHZB6IN2WJJ6KGWUQ66ELUIZIFA67ETLQ7JEUQ, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.305604108 (81h9m22.868927112s) [noise=true , address=104.36.82.15:6289 , piece_id=GWL3P55BIK4IJEL5YFZKI7JCA4LADYOOHUSIQUNGJEKBJOTKMP5A, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.305930423 (81h9m22.869253425s) [noise=true , address=64.49.13.139:4476 , piece_id=P7UKBNNZGI52B3I6WKTV6MVC262FYNKYYPJUI5G47UO5S4MU7PWA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.306355311 (81h9m22.869678319s) [noise=true , address=104.36.82.55:4213 , piece_id=X4IHS2XSPH52LIMNUYT7KGWMAPUH3TEMJOEEVB3JV5N45SEOH7YA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.306788243 (81h9m22.870111252s) [noise=true , address=104.36.82.22:6539 , piece_id=GHQ2MVSTDGVYLUKLV23D6YWRR4PP7UUYHCUEQ4WWSWEMTBFO3P4A, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.392174770 (81h9m22.955497773s) [upload=64958]: Next returned nil segment Jun 22 01:32:47.392195767 (81h9m22.955518775s) [upload=64958]: Waiting for error group managing segments... Jun 22 01:32:47.480767633 (81h9m23.044090623s) [noise=true , address=104.36.82.22:6539 , piece_id=GHQ2MVSTDGVYLUKLV23D6YWRR4PP7UUYHCUEQ4WWSWEMTBFO3P4A, seg_pos={0 0}, upload=64958]: Done uploading piece. err: ecclient: upload failed (node:1QL6w8DSFR6CxDdnJmDuyVZxp6JU55VKiJofBZRfQi7q31i8Jc, address:104.36.82.22:6539): protocol: storage node overloaded, request limit: 2000 Jun 22 01:32:47.585663142 (81h9m23.148986155s) [noise=true , address=79.127.220.101:6048 , piece_id=AN7UQUXB42T2YXEH7ZJRZUDIW47M3RWUEOCL2VEBOGH74BZF3YKA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:47.640263002 (81h9m23.203586013s) [noise=true , address=45.91.116.12:5242 , piece_id=6ABT5IPYGSLFFJMBYMBYRU4EYIXABUAUHG3OVILOHBIUDS3BP27A, seg_pos={0 0}, upload=64958]: Done uploading piece. err: ecclient: upload failed (node:12JsFL62jgXmza8Md8eQ9k8cyw5V9Vz2RzN4y85ZvgHZjwrq4vQ, address:45.91.116.12:5242): protocol: storage node overloaded, request limit: 2000 Jun 22 01:32:47.984127275 (81h9m23.54745028s) [noise=true , address=64.49.13.137:4500 , piece_id=FAE2RXD57IRP5FXA2XR6THXFW27U7IWAZ4JDV5M2WRPXKUGZ4VIA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:48.100255249 (81h9m23.663578241s) [noise=true , address=77.90.31.14:4689 , piece_id=Q3HDS3BCMYDTODFRMU4XFRDCQK2QZE3ZMXVLUX4M54PDEBS7RR6A, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:48.221764035 (81h9m23.785087034s) [noise=true , address=104.36.82.54:4165 , piece_id=4CVOABPCWC7J7SRHZB6IN2WJJ6KGWUQ66ELUIZIFA67ETLQ7JEUQ, seg_pos={0 0}, upload=64958]: Done uploading piece. err: ecclient: upload failed (node:12wGUMXVEubEtki4qGnjGEsbYFB1iUauZcu3oDRKrCQsJMV3bV2, address:104.36.82.54:4165): protocol: storage node overloaded, request limit: 2000 Jun 22 01:32:48.509996527 (81h9m24.073319547s) [noise=true , address=104.36.82.9:6023 , piece_id=SISFAUU4RPF57OLGWM2BWQWE6GRUSCMORBYHWEXETNAVM2EEMO6Q, seg_pos={0 0}, upload=64958]: Done uploading piece. err: ecclient: upload failed (node:1j2S9tgWpLwmyvXZSq2qKKcbaC8r7LkzfDCaMxCejAni4CST1g, address:104.36.82.9:6023): protocol: storage node overloaded, request limit: 2000 Jun 22 01:32:48.510513169 (81h9m24.073836177s) [noise=true , address=104.36.82.42:6811 , piece_id=A5BFEHZ3WGGK53IZ7MKI6N4CVXHOORF6L47NR3UFDCAX7LBHRHVQ, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:48.565245605 (81h9m24.128568606s) [noise=true , address=79.127.205.234:5601 , piece_id=V7PMXBP2VFWBSZDG4GTLBD435T6F5DT7A43L4GQLQ5BDDTJ4WY6A, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:49.979979824 (81h9m25.543302825s) [noise=true , address=104.36.82.15:6289 , piece_id=GWL3P55BIK4IJEL5YFZKI7JCA4LADYOOHUSIQUNGJEKBJOTKMP5A, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:50.463750280 (81h9m26.02707328s) [noise=true , address=104.36.82.51:4084 , piece_id=D43FA7BFE54KPCWGDGDME2EGSWM7PONOVPD3DN2XY42MJFOKVDIQ, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:51.480599795 (81h9m27.043922798s) [noise=true , address=79.127.219.38:4023 , piece_id=BWLRCXXHBMBTNCZEVKKIIS5PB35PDQXQXCLIQB7GBLEIOWWUFBFA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:52.941200625 (81h9m28.504523627s) [noise=true , address=104.36.82.43:6841 , piece_id=PIHZMSTO3CII7KGDLGSAILTDRSLMEBVOLBCRCZJULMCFISXIZWXA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:58.281803598 (81h9m33.845126612s) [noise=true , address=64.49.13.139:4476 , piece_id=P7UKBNNZGI52B3I6WKTV6MVC262FYNKYYPJUI5G47UO5S4MU7PWA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:33:00.277271877 (81h9m35.840594867s) [noise=true , address=104.36.82.55:4213 , piece_id=X4IHS2XSPH52LIMNUYT7KGWMAPUH3TEMJOEEVB3JV5N45SEOH7YA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:33:09.435339213 (81h9m44.998662206s) [noise=true , address=104.36.82.52:4089 , piece_id=JSOEKLNWA3W3LOVKI3EGGKJYJ6J3UZY6KGABKQPQPXTIOOPMKVHA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:33:31.472868445 (81h10m7.03619143s) [noise=true , address=104.36.82.53:4122 , piece_id=HZZQBIKHQWLUE5EHK4KSKJGIUEG7D44RQASUJHCYXX6BICCGX6EA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:33:47.235404653 (81h10m22.798727642s) [noise=true , address=104.36.82.12:6158 , piece_id=LJZEOJVJCJAR2BRILIJI3ZVDGYMPBBTP2G3HCTN3MHNS4KYDPASQ, seg_pos={0 0}, upload=64958]: Done uploading piece. err: ecclient: upload failed (node:1SjM6Utm2YizeUbrBUD9Z7etE1nLadW5Fa1zYP7MioPeeqhdJ7, address:104.36.82.12:6158): protocol: expected piece hash; context deadline exceeded; EOF Jun 22 01:33:47.235448297 (81h10m22.798771306s) [seg_pos={0 0}, upload=64958]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jun 22 01:33:47.265442393 (81h10m22.828765401s) [upload=64958]: Done uploading segments... Jun 22 01:33:47.265476139 (81h10m22.82879914s) [upload=64958]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration:* 13.046705452s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.X..........................................................] duration: 20.629215493s FAIL (2025-06-22 02:18:00 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== oad=64949]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jun 22 01:32:13.674692303 (81h8m49.238015311s) [upload=64949]: Done uploading segments... Jun 22 01:32:13.674727219 (81h8m49.238050233s) [upload=64949]: Upload finished. err: <nil> Jun 22 01:32:47.266407995 (81h9m22.829731002s) [upload=64958]: Starting upload Jun 22 01:32:47.266524767 (81h9m22.829847771s) [upload=64958]: Done starting upload Jun 22 01:32:47.266813109 (81h9m22.830136116s) [upload=64958]: Uploading segments... Jun 22 01:32:47.266969526 (81h9m22.830292526s) [upload=64958]: Got next segment. Inline: false Jun 22 01:32:47.267075475 (81h9m22.830398475s) [upload=64958]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 22 01:32:47.267081679 (81h9m22.830404667s) [upload=64958]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 22 01:32:47.301770382 (81h9m22.865093389s) [seg_pos={0 0}, upload=64958]: Begin upload segment... Jun 22 01:32:47.301823796 (81h9m22.865146799s) [seg_pos={0 0}, upload=64958]: Done begin upload segment. Jun 22 01:32:47.301855055 (81h9m22.865178087s) [noise=true , address=104.36.82.43:6841 , piece_id=PIHZMSTO3CII7KGDLGSAILTDRSLMEBVOLBCRCZJULMCFISXIZWXA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.301902600 (81h9m22.865225606s) [noise=true , address=104.36.82.42:6811 , piece_id=A5BFEHZ3WGGK53IZ7MKI6N4CVXHOORF6L47NR3UFDCAX7LBHRHVQ, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.302234027 (81h9m22.865557029s) [noise=true , address=104.36.82.51:4084 , piece_id=D43FA7BFE54KPCWGDGDME2EGSWM7PONOVPD3DN2XY42MJFOKVDIQ, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.302496354 (81h9m22.865819363s) [noise=true , address=79.127.219.38:4023 , piece_id=BWLRCXXHBMBTNCZEVKKIIS5PB35PDQXQXCLIQB7GBLEIOWWUFBFA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.302577370 (81h9m22.865900378s) [noise=true , address=104.36.82.52:4089 , piece_id=JSOEKLNWA3W3LOVKI3EGGKJYJ6J3UZY6KGABKQPQPXTIOOPMKVHA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.302917829 (81h9m22.866240835s) [noise=true , address=104.36.82.53:4122 , piece_id=HZZQBIKHQWLUE5EHK4KSKJGIUEG7D44RQASUJHCYXX6BICCGX6EA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.303087165 (81h9m22.866410168s) [noise=true , address=77.90.31.14:4689 , piece_id=Q3HDS3BCMYDTODFRMU4XFRDCQK2QZE3ZMXVLUX4M54PDEBS7RR6A, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.303250714 (81h9m22.866573717s) [noise=true , address=79.127.205.234:5601 , piece_id=V7PMXBP2VFWBSZDG4GTLBD435T6F5DT7A43L4GQLQ5BDDTJ4WY6A, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.303579249 (81h9m22.866902255s) [noise=true , address=104.36.82.12:6158 , piece_id=LJZEOJVJCJAR2BRILIJI3ZVDGYMPBBTP2G3HCTN3MHNS4KYDPASQ, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.303599776 (81h9m22.866922784s) [noise=true , address=104.36.82.9:6023 , piece_id=SISFAUU4RPF57OLGWM2BWQWE6GRUSCMORBYHWEXETNAVM2EEMO6Q, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.304005346 (81h9m22.867328346s) [noise=true , address=45.91.116.12:5242 , piece_id=6ABT5IPYGSLFFJMBYMBYRU4EYIXABUAUHG3OVILOHBIUDS3BP27A, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.304388850 (81h9m22.867711851s) [noise=true , address=64.49.13.137:4500 , piece_id=FAE2RXD57IRP5FXA2XR6THXFW27U7IWAZ4JDV5M2WRPXKUGZ4VIA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.304881529 (81h9m22.868204536s) [noise=true , address=79.127.220.101:6048 , piece_id=AN7UQUXB42T2YXEH7ZJRZUDIW47M3RWUEOCL2VEBOGH74BZF3YKA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.305249860 (81h9m22.86857287s) [noise=true , address=104.36.82.54:4165 , piece_id=4CVOABPCWC7J7SRHZB6IN2WJJ6KGWUQ66ELUIZIFA67ETLQ7JEUQ, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.305604108 (81h9m22.868927112s) [noise=true , address=104.36.82.15:6289 , piece_id=GWL3P55BIK4IJEL5YFZKI7JCA4LADYOOHUSIQUNGJEKBJOTKMP5A, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.305930423 (81h9m22.869253425s) [noise=true , address=64.49.13.139:4476 , piece_id=P7UKBNNZGI52B3I6WKTV6MVC262FYNKYYPJUI5G47UO5S4MU7PWA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.306355311 (81h9m22.869678319s) [noise=true , address=104.36.82.55:4213 , piece_id=X4IHS2XSPH52LIMNUYT7KGWMAPUH3TEMJOEEVB3JV5N45SEOH7YA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.306788243 (81h9m22.870111252s) [noise=true , address=104.36.82.22:6539 , piece_id=GHQ2MVSTDGVYLUKLV23D6YWRR4PP7UUYHCUEQ4WWSWEMTBFO3P4A, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.392174770 (81h9m22.955497773s) [upload=64958]: Next returned nil segment Jun 22 01:32:47.392195767 (81h9m22.955518775s) [upload=64958]: Waiting for error group managing segments... Jun 22 01:32:47.480767633 (81h9m23.044090623s) [noise=true , address=104.36.82.22:6539 , piece_id=GHQ2MVSTDGVYLUKLV23D6YWRR4PP7UUYHCUEQ4WWSWEMTBFO3P4A, seg_pos={0 0}, upload=64958]: Done uploading piece. err: ecclient: upload failed (node:1QL6w8DSFR6CxDdnJmDuyVZxp6JU55VKiJofBZRfQi7q31i8Jc, address:104.36.82.22:6539): protocol: storage node overloaded, request limit: 2000 Jun 22 01:32:47.585663142 (81h9m23.148986155s) [noise=true , address=79.127.220.101:6048 , piece_id=AN7UQUXB42T2YXEH7ZJRZUDIW47M3RWUEOCL2VEBOGH74BZF3YKA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:47.640263002 (81h9m23.203586013s) [noise=true , address=45.91.116.12:5242 , piece_id=6ABT5IPYGSLFFJMBYMBYRU4EYIXABUAUHG3OVILOHBIUDS3BP27A, seg_pos={0 0}, upload=64958]: Done uploading piece. err: ecclient: upload failed (node:12JsFL62jgXmza8Md8eQ9k8cyw5V9Vz2RzN4y85ZvgHZjwrq4vQ, address:45.91.116.12:5242): protocol: storage node overloaded, request limit: 2000 Jun 22 01:32:47.984127275 (81h9m23.54745028s) [noise=true , address=64.49.13.137:4500 , piece_id=FAE2RXD57IRP5FXA2XR6THXFW27U7IWAZ4JDV5M2WRPXKUGZ4VIA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:48.100255249 (81h9m23.663578241s) [noise=true , address=77.90.31.14:4689 , piece_id=Q3HDS3BCMYDTODFRMU4XFRDCQK2QZE3ZMXVLUX4M54PDEBS7RR6A, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:48.221764035 (81h9m23.785087034s) [noise=true , address=104.36.82.54:4165 , piece_id=4CVOABPCWC7J7SRHZB6IN2WJJ6KGWUQ66ELUIZIFA67ETLQ7JEUQ, seg_pos={0 0}, upload=64958]: Done uploading piece. err: ecclient: upload failed (node:12wGUMXVEubEtki4qGnjGEsbYFB1iUauZcu3oDRKrCQsJMV3bV2, address:104.36.82.54:4165): protocol: storage node overloaded, request limit: 2000 Jun 22 01:32:48.509996527 (81h9m24.073319547s) [noise=true , address=104.36.82.9:6023 , piece_id=SISFAUU4RPF57OLGWM2BWQWE6GRUSCMORBYHWEXETNAVM2EEMO6Q, seg_pos={0 0}, upload=64958]: Done uploading piece. err: ecclient: upload failed (node:1j2S9tgWpLwmyvXZSq2qKKcbaC8r7LkzfDCaMxCejAni4CST1g, address:104.36.82.9:6023): protocol: storage node overloaded, request limit: 2000 Jun 22 01:32:48.510513169 (81h9m24.073836177s) [noise=true , address=104.36.82.42:6811 , piece_id=A5BFEHZ3WGGK53IZ7MKI6N4CVXHOORF6L47NR3UFDCAX7LBHRHVQ, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:48.565245605 (81h9m24.128568606s) [noise=true , address=79.127.205.234:5601 , piece_id=V7PMXBP2VFWBSZDG4GTLBD435T6F5DT7A43L4GQLQ5BDDTJ4WY6A, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:49.979979824 (81h9m25.543302825s) [noise=true , address=104.36.82.15:6289 , piece_id=GWL3P55BIK4IJEL5YFZKI7JCA4LADYOOHUSIQUNGJEKBJOTKMP5A, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:50.463750280 (81h9m26.02707328s) [noise=true , address=104.36.82.51:4084 , piece_id=D43FA7BFE54KPCWGDGDME2EGSWM7PONOVPD3DN2XY42MJFOKVDIQ, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:51.480599795 (81h9m27.043922798s) [noise=true , address=79.127.219.38:4023 , piece_id=BWLRCXXHBMBTNCZEVKKIIS5PB35PDQXQXCLIQB7GBLEIOWWUFBFA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:52.941200625 (81h9m28.504523627s) [noise=true , address=104.36.82.43:6841 , piece_id=PIHZMSTO3CII7KGDLGSAILTDRSLMEBVOLBCRCZJULMCFISXIZWXA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:58.281803598 (81h9m33.845126612s) [noise=true , address=64.49.13.139:4476 , piece_id=P7UKBNNZGI52B3I6WKTV6MVC262FYNKYYPJUI5G47UO5S4MU7PWA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:33:00.277271877 (81h9m35.840594867s) [noise=true , address=104.36.82.55:4213 , piece_id=X4IHS2XSPH52LIMNUYT7KGWMAPUH3TEMJOEEVB3JV5N45SEOH7YA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:33:09.435339213 (81h9m44.998662206s) [noise=true , address=104.36.82.52:4089 , piece_id=JSOEKLNWA3W3LOVKI3EGGKJYJ6J3UZY6KGABKQPQPXTIOOPMKVHA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:33:31.472868445 (81h10m7.03619143s) [noise=true , address=104.36.82.53:4122 , piece_id=HZZQBIKHQWLUE5EHK4KSKJGIUEG7D44RQASUJHCYXX6BICCGX6EA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:33:47.235404653 (81h10m22.798727642s) [noise=true , address=104.36.82.12:6158 , piece_id=LJZEOJVJCJAR2BRILIJI3ZVDGYMPBBTP2G3HCTN3MHNS4KYDPASQ, seg_pos={0 0}, upload=64958]: Done uploading piece. err: ecclient: upload failed (node:1SjM6Utm2YizeUbrBUD9Z7etE1nLadW5Fa1zYP7MioPeeqhdJ7, address:104.36.82.12:6158): protocol: expected piece hash; context deadline exceeded; EOF Jun 22 01:33:47.235448297 (81h10m22.798771306s) [seg_pos={0 0}, upload=64958]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jun 22 01:33:47.265442393 (81h10m22.828765401s) [upload=64958]: Done uploading segments... Jun 22 01:33:47.265476139 (81h10m22.82879914s) [upload=64958]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration:* 20.629247799s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [.................X.................X........................] duration: 15.870037103s FAIL (2025-06-22 02:17:51 UTC) orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded Duration: 15.870065515s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [..X.........................................................] duration: 23.473436888s FAIL (2025-06-22 02:17:06 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== oad=64949]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jun 22 01:32:13.674692303 (81h8m49.238015311s) [upload=64949]: Done uploading segments... Jun 22 01:32:13.674727219 (81h8m49.238050233s) [upload=64949]: Upload finished. err: <nil> Jun 22 01:32:47.266407995 (81h9m22.829731002s) [upload=64958]: Starting upload Jun 22 01:32:47.266524767 (81h9m22.829847771s) [upload=64958]: Done starting upload Jun 22 01:32:47.266813109 (81h9m22.830136116s) [upload=64958]: Uploading segments... Jun 22 01:32:47.266969526 (81h9m22.830292526s) [upload=64958]: Got next segment. Inline: false Jun 22 01:32:47.267075475 (81h9m22.830398475s) [upload=64958]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 22 01:32:47.267081679 (81h9m22.830404667s) [upload=64958]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 22 01:32:47.301770382 (81h9m22.865093389s) [seg_pos={0 0}, upload=64958]: Begin upload segment... Jun 22 01:32:47.301823796 (81h9m22.865146799s) [seg_pos={0 0}, upload=64958]: Done begin upload segment. Jun 22 01:32:47.301855055 (81h9m22.865178087s) [noise=true , address=104.36.82.43:6841 , piece_id=PIHZMSTO3CII7KGDLGSAILTDRSLMEBVOLBCRCZJULMCFISXIZWXA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.301902600 (81h9m22.865225606s) [noise=true , address=104.36.82.42:6811 , piece_id=A5BFEHZ3WGGK53IZ7MKI6N4CVXHOORF6L47NR3UFDCAX7LBHRHVQ, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.302234027 (81h9m22.865557029s) [noise=true , address=104.36.82.51:4084 , piece_id=D43FA7BFE54KPCWGDGDME2EGSWM7PONOVPD3DN2XY42MJFOKVDIQ, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.302496354 (81h9m22.865819363s) [noise=true , address=79.127.219.38:4023 , piece_id=BWLRCXXHBMBTNCZEVKKIIS5PB35PDQXQXCLIQB7GBLEIOWWUFBFA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.302577370 (81h9m22.865900378s) [noise=true , address=104.36.82.52:4089 , piece_id=JSOEKLNWA3W3LOVKI3EGGKJYJ6J3UZY6KGABKQPQPXTIOOPMKVHA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.302917829 (81h9m22.866240835s) [noise=true , address=104.36.82.53:4122 , piece_id=HZZQBIKHQWLUE5EHK4KSKJGIUEG7D44RQASUJHCYXX6BICCGX6EA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.303087165 (81h9m22.866410168s) [noise=true , address=77.90.31.14:4689 , piece_id=Q3HDS3BCMYDTODFRMU4XFRDCQK2QZE3ZMXVLUX4M54PDEBS7RR6A, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.303250714 (81h9m22.866573717s) [noise=true , address=79.127.205.234:5601 , piece_id=V7PMXBP2VFWBSZDG4GTLBD435T6F5DT7A43L4GQLQ5BDDTJ4WY6A, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.303579249 (81h9m22.866902255s) [noise=true , address=104.36.82.12:6158 , piece_id=LJZEOJVJCJAR2BRILIJI3ZVDGYMPBBTP2G3HCTN3MHNS4KYDPASQ, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.303599776 (81h9m22.866922784s) [noise=true , address=104.36.82.9:6023 , piece_id=SISFAUU4RPF57OLGWM2BWQWE6GRUSCMORBYHWEXETNAVM2EEMO6Q, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.304005346 (81h9m22.867328346s) [noise=true , address=45.91.116.12:5242 , piece_id=6ABT5IPYGSLFFJMBYMBYRU4EYIXABUAUHG3OVILOHBIUDS3BP27A, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.304388850 (81h9m22.867711851s) [noise=true , address=64.49.13.137:4500 , piece_id=FAE2RXD57IRP5FXA2XR6THXFW27U7IWAZ4JDV5M2WRPXKUGZ4VIA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.304881529 (81h9m22.868204536s) [noise=true , address=79.127.220.101:6048 , piece_id=AN7UQUXB42T2YXEH7ZJRZUDIW47M3RWUEOCL2VEBOGH74BZF3YKA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.305249860 (81h9m22.86857287s) [noise=true , address=104.36.82.54:4165 , piece_id=4CVOABPCWC7J7SRHZB6IN2WJJ6KGWUQ66ELUIZIFA67ETLQ7JEUQ, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.305604108 (81h9m22.868927112s) [noise=true , address=104.36.82.15:6289 , piece_id=GWL3P55BIK4IJEL5YFZKI7JCA4LADYOOHUSIQUNGJEKBJOTKMP5A, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.305930423 (81h9m22.869253425s) [noise=true , address=64.49.13.139:4476 , piece_id=P7UKBNNZGI52B3I6WKTV6MVC262FYNKYYPJUI5G47UO5S4MU7PWA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.306355311 (81h9m22.869678319s) [noise=true , address=104.36.82.55:4213 , piece_id=X4IHS2XSPH52LIMNUYT7KGWMAPUH3TEMJOEEVB3JV5N45SEOH7YA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.306788243 (81h9m22.870111252s) [noise=true , address=104.36.82.22:6539 , piece_id=GHQ2MVSTDGVYLUKLV23D6YWRR4PP7UUYHCUEQ4WWSWEMTBFO3P4A, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.392174770 (81h9m22.955497773s) [upload=64958]: Next returned nil segment Jun 22 01:32:47.392195767 (81h9m22.955518775s) [upload=64958]: Waiting for error group managing segments... Jun 22 01:32:47.480767633 (81h9m23.044090623s) [noise=true , address=104.36.82.22:6539 , piece_id=GHQ2MVSTDGVYLUKLV23D6YWRR4PP7UUYHCUEQ4WWSWEMTBFO3P4A, seg_pos={0 0}, upload=64958]: Done uploading piece. err: ecclient: upload failed (node:1QL6w8DSFR6CxDdnJmDuyVZxp6JU55VKiJofBZRfQi7q31i8Jc, address:104.36.82.22:6539): protocol: storage node overloaded, request limit: 2000 Jun 22 01:32:47.585663142 (81h9m23.148986155s) [noise=true , address=79.127.220.101:6048 , piece_id=AN7UQUXB42T2YXEH7ZJRZUDIW47M3RWUEOCL2VEBOGH74BZF3YKA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:47.640263002 (81h9m23.203586013s) [noise=true , address=45.91.116.12:5242 , piece_id=6ABT5IPYGSLFFJMBYMBYRU4EYIXABUAUHG3OVILOHBIUDS3BP27A, seg_pos={0 0}, upload=64958]: Done uploading piece. err: ecclient: upload failed (node:12JsFL62jgXmza8Md8eQ9k8cyw5V9Vz2RzN4y85ZvgHZjwrq4vQ, address:45.91.116.12:5242): protocol: storage node overloaded, request limit: 2000 Jun 22 01:32:47.984127275 (81h9m23.54745028s) [noise=true , address=64.49.13.137:4500 , piece_id=FAE2RXD57IRP5FXA2XR6THXFW27U7IWAZ4JDV5M2WRPXKUGZ4VIA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:48.100255249 (81h9m23.663578241s) [noise=true , address=77.90.31.14:4689 , piece_id=Q3HDS3BCMYDTODFRMU4XFRDCQK2QZE3ZMXVLUX4M54PDEBS7RR6A, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:48.221764035 (81h9m23.785087034s) [noise=true , address=104.36.82.54:4165 , piece_id=4CVOABPCWC7J7SRHZB6IN2WJJ6KGWUQ66ELUIZIFA67ETLQ7JEUQ, seg_pos={0 0}, upload=64958]: Done uploading piece. err: ecclient: upload failed (node:12wGUMXVEubEtki4qGnjGEsbYFB1iUauZcu3oDRKrCQsJMV3bV2, address:104.36.82.54:4165): protocol: storage node overloaded, request limit: 2000 Jun 22 01:32:48.509996527 (81h9m24.073319547s) [noise=true , address=104.36.82.9:6023 , piece_id=SISFAUU4RPF57OLGWM2BWQWE6GRUSCMORBYHWEXETNAVM2EEMO6Q, seg_pos={0 0}, upload=64958]: Done uploading piece. err: ecclient: upload failed (node:1j2S9tgWpLwmyvXZSq2qKKcbaC8r7LkzfDCaMxCejAni4CST1g, address:104.36.82.9:6023): protocol: storage node overloaded, request limit: 2000 Jun 22 01:32:48.510513169 (81h9m24.073836177s) [noise=true , address=104.36.82.42:6811 , piece_id=A5BFEHZ3WGGK53IZ7MKI6N4CVXHOORF6L47NR3UFDCAX7LBHRHVQ, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:48.565245605 (81h9m24.128568606s) [noise=true , address=79.127.205.234:5601 , piece_id=V7PMXBP2VFWBSZDG4GTLBD435T6F5DT7A43L4GQLQ5BDDTJ4WY6A, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:49.979979824 (81h9m25.543302825s) [noise=true , address=104.36.82.15:6289 , piece_id=GWL3P55BIK4IJEL5YFZKI7JCA4LADYOOHUSIQUNGJEKBJOTKMP5A, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:50.463750280 (81h9m26.02707328s) [noise=true , address=104.36.82.51:4084 , piece_id=D43FA7BFE54KPCWGDGDME2EGSWM7PONOVPD3DN2XY42MJFOKVDIQ, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:51.480599795 (81h9m27.043922798s) [noise=true , address=79.127.219.38:4023 , piece_id=BWLRCXXHBMBTNCZEVKKIIS5PB35PDQXQXCLIQB7GBLEIOWWUFBFA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:52.941200625 (81h9m28.504523627s) [noise=true , address=104.36.82.43:6841 , piece_id=PIHZMSTO3CII7KGDLGSAILTDRSLMEBVOLBCRCZJULMCFISXIZWXA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:58.281803598 (81h9m33.845126612s) [noise=true , address=64.49.13.139:4476 , piece_id=P7UKBNNZGI52B3I6WKTV6MVC262FYNKYYPJUI5G47UO5S4MU7PWA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:33:00.277271877 (81h9m35.840594867s) [noise=true , address=104.36.82.55:4213 , piece_id=X4IHS2XSPH52LIMNUYT7KGWMAPUH3TEMJOEEVB3JV5N45SEOH7YA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:33:09.435339213 (81h9m44.998662206s) [noise=true , address=104.36.82.52:4089 , piece_id=JSOEKLNWA3W3LOVKI3EGGKJYJ6J3UZY6KGABKQPQPXTIOOPMKVHA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:33:31.472868445 (81h10m7.03619143s) [noise=true , address=104.36.82.53:4122 , piece_id=HZZQBIKHQWLUE5EHK4KSKJGIUEG7D44RQASUJHCYXX6BICCGX6EA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:33:47.235404653 (81h10m22.798727642s) [noise=true , address=104.36.82.12:6158 , piece_id=LJZEOJVJCJAR2BRILIJI3ZVDGYMPBBTP2G3HCTN3MHNS4KYDPASQ, seg_pos={0 0}, upload=64958]: Done uploading piece. err: ecclient: upload failed (node:1SjM6Utm2YizeUbrBUD9Z7etE1nLadW5Fa1zYP7MioPeeqhdJ7, address:104.36.82.12:6158): protocol: expected piece hash; context deadline exceeded; EOF Jun 22 01:33:47.235448297 (81h10m22.798771306s) [seg_pos={0 0}, upload=64958]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jun 22 01:33:47.265442393 (81h10m22.828765401s) [upload=64958]: Done uploading segments... Jun 22 01:33:47.265476139 (81h10m22.82879914s) [upload=64958]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration:* 23.473462805s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [..................X.................X.......................] duration: 14.115057441s FAIL (2025-06-22 02:16:36 UTC) orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded Duration: 14.11547638s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [...X........................................................] duration: 25.908561475s FAIL (2025-06-22 02:16:16 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== oad=64949]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jun 22 01:32:13.674692303 (81h8m49.238015311s) [upload=64949]: Done uploading segments... Jun 22 01:32:13.674727219 (81h8m49.238050233s) [upload=64949]: Upload finished. err: <nil> Jun 22 01:32:47.266407995 (81h9m22.829731002s) [upload=64958]: Starting upload Jun 22 01:32:47.266524767 (81h9m22.829847771s) [upload=64958]: Done starting upload Jun 22 01:32:47.266813109 (81h9m22.830136116s) [upload=64958]: Uploading segments... Jun 22 01:32:47.266969526 (81h9m22.830292526s) [upload=64958]: Got next segment. Inline: false Jun 22 01:32:47.267075475 (81h9m22.830398475s) [upload=64958]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 22 01:32:47.267081679 (81h9m22.830404667s) [upload=64958]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 22 01:32:47.301770382 (81h9m22.865093389s) [seg_pos={0 0}, upload=64958]: Begin upload segment... Jun 22 01:32:47.301823796 (81h9m22.865146799s) [seg_pos={0 0}, upload=64958]: Done begin upload segment. Jun 22 01:32:47.301855055 (81h9m22.865178087s) [noise=true , address=104.36.82.43:6841 , piece_id=PIHZMSTO3CII7KGDLGSAILTDRSLMEBVOLBCRCZJULMCFISXIZWXA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.301902600 (81h9m22.865225606s) [noise=true , address=104.36.82.42:6811 , piece_id=A5BFEHZ3WGGK53IZ7MKI6N4CVXHOORF6L47NR3UFDCAX7LBHRHVQ, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.302234027 (81h9m22.865557029s) [noise=true , address=104.36.82.51:4084 , piece_id=D43FA7BFE54KPCWGDGDME2EGSWM7PONOVPD3DN2XY42MJFOKVDIQ, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.302496354 (81h9m22.865819363s) [noise=true , address=79.127.219.38:4023 , piece_id=BWLRCXXHBMBTNCZEVKKIIS5PB35PDQXQXCLIQB7GBLEIOWWUFBFA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.302577370 (81h9m22.865900378s) [noise=true , address=104.36.82.52:4089 , piece_id=JSOEKLNWA3W3LOVKI3EGGKJYJ6J3UZY6KGABKQPQPXTIOOPMKVHA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.302917829 (81h9m22.866240835s) [noise=true , address=104.36.82.53:4122 , piece_id=HZZQBIKHQWLUE5EHK4KSKJGIUEG7D44RQASUJHCYXX6BICCGX6EA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.303087165 (81h9m22.866410168s) [noise=true , address=77.90.31.14:4689 , piece_id=Q3HDS3BCMYDTODFRMU4XFRDCQK2QZE3ZMXVLUX4M54PDEBS7RR6A, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.303250714 (81h9m22.866573717s) [noise=true , address=79.127.205.234:5601 , piece_id=V7PMXBP2VFWBSZDG4GTLBD435T6F5DT7A43L4GQLQ5BDDTJ4WY6A, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.303579249 (81h9m22.866902255s) [noise=true , address=104.36.82.12:6158 , piece_id=LJZEOJVJCJAR2BRILIJI3ZVDGYMPBBTP2G3HCTN3MHNS4KYDPASQ, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.303599776 (81h9m22.866922784s) [noise=true , address=104.36.82.9:6023 , piece_id=SISFAUU4RPF57OLGWM2BWQWE6GRUSCMORBYHWEXETNAVM2EEMO6Q, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.304005346 (81h9m22.867328346s) [noise=true , address=45.91.116.12:5242 , piece_id=6ABT5IPYGSLFFJMBYMBYRU4EYIXABUAUHG3OVILOHBIUDS3BP27A, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.304388850 (81h9m22.867711851s) [noise=true , address=64.49.13.137:4500 , piece_id=FAE2RXD57IRP5FXA2XR6THXFW27U7IWAZ4JDV5M2WRPXKUGZ4VIA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.304881529 (81h9m22.868204536s) [noise=true , address=79.127.220.101:6048 , piece_id=AN7UQUXB42T2YXEH7ZJRZUDIW47M3RWUEOCL2VEBOGH74BZF3YKA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.305249860 (81h9m22.86857287s) [noise=true , address=104.36.82.54:4165 , piece_id=4CVOABPCWC7J7SRHZB6IN2WJJ6KGWUQ66ELUIZIFA67ETLQ7JEUQ, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.305604108 (81h9m22.868927112s) [noise=true , address=104.36.82.15:6289 , piece_id=GWL3P55BIK4IJEL5YFZKI7JCA4LADYOOHUSIQUNGJEKBJOTKMP5A, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.305930423 (81h9m22.869253425s) [noise=true , address=64.49.13.139:4476 , piece_id=P7UKBNNZGI52B3I6WKTV6MVC262FYNKYYPJUI5G47UO5S4MU7PWA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.306355311 (81h9m22.869678319s) [noise=true , address=104.36.82.55:4213 , piece_id=X4IHS2XSPH52LIMNUYT7KGWMAPUH3TEMJOEEVB3JV5N45SEOH7YA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.306788243 (81h9m22.870111252s) [noise=true , address=104.36.82.22:6539 , piece_id=GHQ2MVSTDGVYLUKLV23D6YWRR4PP7UUYHCUEQ4WWSWEMTBFO3P4A, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.392174770 (81h9m22.955497773s) [upload=64958]: Next returned nil segment Jun 22 01:32:47.392195767 (81h9m22.955518775s) [upload=64958]: Waiting for error group managing segments... Jun 22 01:32:47.480767633 (81h9m23.044090623s) [noise=true , address=104.36.82.22:6539 , piece_id=GHQ2MVSTDGVYLUKLV23D6YWRR4PP7UUYHCUEQ4WWSWEMTBFO3P4A, seg_pos={0 0}, upload=64958]: Done uploading piece. err: ecclient: upload failed (node:1QL6w8DSFR6CxDdnJmDuyVZxp6JU55VKiJofBZRfQi7q31i8Jc, address:104.36.82.22:6539): protocol: storage node overloaded, request limit: 2000 Jun 22 01:32:47.585663142 (81h9m23.148986155s) [noise=true , address=79.127.220.101:6048 , piece_id=AN7UQUXB42T2YXEH7ZJRZUDIW47M3RWUEOCL2VEBOGH74BZF3YKA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:47.640263002 (81h9m23.203586013s) [noise=true , address=45.91.116.12:5242 , piece_id=6ABT5IPYGSLFFJMBYMBYRU4EYIXABUAUHG3OVILOHBIUDS3BP27A, seg_pos={0 0}, upload=64958]: Done uploading piece. err: ecclient: upload failed (node:12JsFL62jgXmza8Md8eQ9k8cyw5V9Vz2RzN4y85ZvgHZjwrq4vQ, address:45.91.116.12:5242): protocol: storage node overloaded, request limit: 2000 Jun 22 01:32:47.984127275 (81h9m23.54745028s) [noise=true , address=64.49.13.137:4500 , piece_id=FAE2RXD57IRP5FXA2XR6THXFW27U7IWAZ4JDV5M2WRPXKUGZ4VIA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:48.100255249 (81h9m23.663578241s) [noise=true , address=77.90.31.14:4689 , piece_id=Q3HDS3BCMYDTODFRMU4XFRDCQK2QZE3ZMXVLUX4M54PDEBS7RR6A, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:48.221764035 (81h9m23.785087034s) [noise=true , address=104.36.82.54:4165 , piece_id=4CVOABPCWC7J7SRHZB6IN2WJJ6KGWUQ66ELUIZIFA67ETLQ7JEUQ, seg_pos={0 0}, upload=64958]: Done uploading piece. err: ecclient: upload failed (node:12wGUMXVEubEtki4qGnjGEsbYFB1iUauZcu3oDRKrCQsJMV3bV2, address:104.36.82.54:4165): protocol: storage node overloaded, request limit: 2000 Jun 22 01:32:48.509996527 (81h9m24.073319547s) [noise=true , address=104.36.82.9:6023 , piece_id=SISFAUU4RPF57OLGWM2BWQWE6GRUSCMORBYHWEXETNAVM2EEMO6Q, seg_pos={0 0}, upload=64958]: Done uploading piece. err: ecclient: upload failed (node:1j2S9tgWpLwmyvXZSq2qKKcbaC8r7LkzfDCaMxCejAni4CST1g, address:104.36.82.9:6023): protocol: storage node overloaded, request limit: 2000 Jun 22 01:32:48.510513169 (81h9m24.073836177s) [noise=true , address=104.36.82.42:6811 , piece_id=A5BFEHZ3WGGK53IZ7MKI6N4CVXHOORF6L47NR3UFDCAX7LBHRHVQ, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:48.565245605 (81h9m24.128568606s) [noise=true , address=79.127.205.234:5601 , piece_id=V7PMXBP2VFWBSZDG4GTLBD435T6F5DT7A43L4GQLQ5BDDTJ4WY6A, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:49.979979824 (81h9m25.543302825s) [noise=true , address=104.36.82.15:6289 , piece_id=GWL3P55BIK4IJEL5YFZKI7JCA4LADYOOHUSIQUNGJEKBJOTKMP5A, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:50.463750280 (81h9m26.02707328s) [noise=true , address=104.36.82.51:4084 , piece_id=D43FA7BFE54KPCWGDGDME2EGSWM7PONOVPD3DN2XY42MJFOKVDIQ, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:51.480599795 (81h9m27.043922798s) [noise=true , address=79.127.219.38:4023 , piece_id=BWLRCXXHBMBTNCZEVKKIIS5PB35PDQXQXCLIQB7GBLEIOWWUFBFA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:52.941200625 (81h9m28.504523627s) [noise=true , address=104.36.82.43:6841 , piece_id=PIHZMSTO3CII7KGDLGSAILTDRSLMEBVOLBCRCZJULMCFISXIZWXA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:58.281803598 (81h9m33.845126612s) [noise=true , address=64.49.13.139:4476 , piece_id=P7UKBNNZGI52B3I6WKTV6MVC262FYNKYYPJUI5G47UO5S4MU7PWA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:33:00.277271877 (81h9m35.840594867s) [noise=true , address=104.36.82.55:4213 , piece_id=X4IHS2XSPH52LIMNUYT7KGWMAPUH3TEMJOEEVB3JV5N45SEOH7YA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:33:09.435339213 (81h9m44.998662206s) [noise=true , address=104.36.82.52:4089 , piece_id=JSOEKLNWA3W3LOVKI3EGGKJYJ6J3UZY6KGABKQPQPXTIOOPMKVHA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:33:31.472868445 (81h10m7.03619143s) [noise=true , address=104.36.82.53:4122 , piece_id=HZZQBIKHQWLUE5EHK4KSKJGIUEG7D44RQASUJHCYXX6BICCGX6EA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:33:47.235404653 (81h10m22.798727642s) [noise=true , address=104.36.82.12:6158 , piece_id=LJZEOJVJCJAR2BRILIJI3ZVDGYMPBBTP2G3HCTN3MHNS4KYDPASQ, seg_pos={0 0}, upload=64958]: Done uploading piece. err: ecclient: upload failed (node:1SjM6Utm2YizeUbrBUD9Z7etE1nLadW5Fa1zYP7MioPeeqhdJ7, address:104.36.82.12:6158): protocol: expected piece hash; context deadline exceeded; EOF Jun 22 01:33:47.235448297 (81h10m22.798771306s) [seg_pos={0 0}, upload=64958]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jun 22 01:33:47.265442393 (81h10m22.828765401s) [upload=64958]: Done uploading segments... Jun 22 01:33:47.265476139 (81h10m22.82879914s) [upload=64958]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration:* 25.908592201s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [...................X.................X......................] duration: 12.711777548s FAIL (2025-06-22 02:15:45 UTC) orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded Duration: 12.711813897s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [....................X.................X.....................] duration: 6.894667431s FAIL (2025-06-22 02:15:11 UTC) orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded Duration: 6.894703325s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [....X.......................................................] duration: 13.88733807s FAIL (2025-06-22 02:14:50 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== oad=64949]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jun 22 01:32:13.674692303 (81h8m49.238015311s) [upload=64949]: Done uploading segments... Jun 22 01:32:13.674727219 (81h8m49.238050233s) [upload=64949]: Upload finished. err: <nil> Jun 22 01:32:47.266407995 (81h9m22.829731002s) [upload=64958]: Starting upload Jun 22 01:32:47.266524767 (81h9m22.829847771s) [upload=64958]: Done starting upload Jun 22 01:32:47.266813109 (81h9m22.830136116s) [upload=64958]: Uploading segments... Jun 22 01:32:47.266969526 (81h9m22.830292526s) [upload=64958]: Got next segment. Inline: false Jun 22 01:32:47.267075475 (81h9m22.830398475s) [upload=64958]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 22 01:32:47.267081679 (81h9m22.830404667s) [upload=64958]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 22 01:32:47.301770382 (81h9m22.865093389s) [seg_pos={0 0}, upload=64958]: Begin upload segment... Jun 22 01:32:47.301823796 (81h9m22.865146799s) [seg_pos={0 0}, upload=64958]: Done begin upload segment. Jun 22 01:32:47.301855055 (81h9m22.865178087s) [noise=true , address=104.36.82.43:6841 , piece_id=PIHZMSTO3CII7KGDLGSAILTDRSLMEBVOLBCRCZJULMCFISXIZWXA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.301902600 (81h9m22.865225606s) [noise=true , address=104.36.82.42:6811 , piece_id=A5BFEHZ3WGGK53IZ7MKI6N4CVXHOORF6L47NR3UFDCAX7LBHRHVQ, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.302234027 (81h9m22.865557029s) [noise=true , address=104.36.82.51:4084 , piece_id=D43FA7BFE54KPCWGDGDME2EGSWM7PONOVPD3DN2XY42MJFOKVDIQ, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.302496354 (81h9m22.865819363s) [noise=true , address=79.127.219.38:4023 , piece_id=BWLRCXXHBMBTNCZEVKKIIS5PB35PDQXQXCLIQB7GBLEIOWWUFBFA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.302577370 (81h9m22.865900378s) [noise=true , address=104.36.82.52:4089 , piece_id=JSOEKLNWA3W3LOVKI3EGGKJYJ6J3UZY6KGABKQPQPXTIOOPMKVHA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.302917829 (81h9m22.866240835s) [noise=true , address=104.36.82.53:4122 , piece_id=HZZQBIKHQWLUE5EHK4KSKJGIUEG7D44RQASUJHCYXX6BICCGX6EA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.303087165 (81h9m22.866410168s) [noise=true , address=77.90.31.14:4689 , piece_id=Q3HDS3BCMYDTODFRMU4XFRDCQK2QZE3ZMXVLUX4M54PDEBS7RR6A, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.303250714 (81h9m22.866573717s) [noise=true , address=79.127.205.234:5601 , piece_id=V7PMXBP2VFWBSZDG4GTLBD435T6F5DT7A43L4GQLQ5BDDTJ4WY6A, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.303579249 (81h9m22.866902255s) [noise=true , address=104.36.82.12:6158 , piece_id=LJZEOJVJCJAR2BRILIJI3ZVDGYMPBBTP2G3HCTN3MHNS4KYDPASQ, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.303599776 (81h9m22.866922784s) [noise=true , address=104.36.82.9:6023 , piece_id=SISFAUU4RPF57OLGWM2BWQWE6GRUSCMORBYHWEXETNAVM2EEMO6Q, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.304005346 (81h9m22.867328346s) [noise=true , address=45.91.116.12:5242 , piece_id=6ABT5IPYGSLFFJMBYMBYRU4EYIXABUAUHG3OVILOHBIUDS3BP27A, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.304388850 (81h9m22.867711851s) [noise=true , address=64.49.13.137:4500 , piece_id=FAE2RXD57IRP5FXA2XR6THXFW27U7IWAZ4JDV5M2WRPXKUGZ4VIA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.304881529 (81h9m22.868204536s) [noise=true , address=79.127.220.101:6048 , piece_id=AN7UQUXB42T2YXEH7ZJRZUDIW47M3RWUEOCL2VEBOGH74BZF3YKA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.305249860 (81h9m22.86857287s) [noise=true , address=104.36.82.54:4165 , piece_id=4CVOABPCWC7J7SRHZB6IN2WJJ6KGWUQ66ELUIZIFA67ETLQ7JEUQ, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.305604108 (81h9m22.868927112s) [noise=true , address=104.36.82.15:6289 , piece_id=GWL3P55BIK4IJEL5YFZKI7JCA4LADYOOHUSIQUNGJEKBJOTKMP5A, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.305930423 (81h9m22.869253425s) [noise=true , address=64.49.13.139:4476 , piece_id=P7UKBNNZGI52B3I6WKTV6MVC262FYNKYYPJUI5G47UO5S4MU7PWA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.306355311 (81h9m22.869678319s) [noise=true , address=104.36.82.55:4213 , piece_id=X4IHS2XSPH52LIMNUYT7KGWMAPUH3TEMJOEEVB3JV5N45SEOH7YA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.306788243 (81h9m22.870111252s) [noise=true , address=104.36.82.22:6539 , piece_id=GHQ2MVSTDGVYLUKLV23D6YWRR4PP7UUYHCUEQ4WWSWEMTBFO3P4A, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.392174770 (81h9m22.955497773s) [upload=64958]: Next returned nil segment Jun 22 01:32:47.392195767 (81h9m22.955518775s) [upload=64958]: Waiting for error group managing segments... Jun 22 01:32:47.480767633 (81h9m23.044090623s) [noise=true , address=104.36.82.22:6539 , piece_id=GHQ2MVSTDGVYLUKLV23D6YWRR4PP7UUYHCUEQ4WWSWEMTBFO3P4A, seg_pos={0 0}, upload=64958]: Done uploading piece. err: ecclient: upload failed (node:1QL6w8DSFR6CxDdnJmDuyVZxp6JU55VKiJofBZRfQi7q31i8Jc, address:104.36.82.22:6539): protocol: storage node overloaded, request limit: 2000 Jun 22 01:32:47.585663142 (81h9m23.148986155s) [noise=true , address=79.127.220.101:6048 , piece_id=AN7UQUXB42T2YXEH7ZJRZUDIW47M3RWUEOCL2VEBOGH74BZF3YKA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:47.640263002 (81h9m23.203586013s) [noise=true , address=45.91.116.12:5242 , piece_id=6ABT5IPYGSLFFJMBYMBYRU4EYIXABUAUHG3OVILOHBIUDS3BP27A, seg_pos={0 0}, upload=64958]: Done uploading piece. err: ecclient: upload failed (node:12JsFL62jgXmza8Md8eQ9k8cyw5V9Vz2RzN4y85ZvgHZjwrq4vQ, address:45.91.116.12:5242): protocol: storage node overloaded, request limit: 2000 Jun 22 01:32:47.984127275 (81h9m23.54745028s) [noise=true , address=64.49.13.137:4500 , piece_id=FAE2RXD57IRP5FXA2XR6THXFW27U7IWAZ4JDV5M2WRPXKUGZ4VIA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:48.100255249 (81h9m23.663578241s) [noise=true , address=77.90.31.14:4689 , piece_id=Q3HDS3BCMYDTODFRMU4XFRDCQK2QZE3ZMXVLUX4M54PDEBS7RR6A, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:48.221764035 (81h9m23.785087034s) [noise=true , address=104.36.82.54:4165 , piece_id=4CVOABPCWC7J7SRHZB6IN2WJJ6KGWUQ66ELUIZIFA67ETLQ7JEUQ, seg_pos={0 0}, upload=64958]: Done uploading piece. err: ecclient: upload failed (node:12wGUMXVEubEtki4qGnjGEsbYFB1iUauZcu3oDRKrCQsJMV3bV2, address:104.36.82.54:4165): protocol: storage node overloaded, request limit: 2000 Jun 22 01:32:48.509996527 (81h9m24.073319547s) [noise=true , address=104.36.82.9:6023 , piece_id=SISFAUU4RPF57OLGWM2BWQWE6GRUSCMORBYHWEXETNAVM2EEMO6Q, seg_pos={0 0}, upload=64958]: Done uploading piece. err: ecclient: upload failed (node:1j2S9tgWpLwmyvXZSq2qKKcbaC8r7LkzfDCaMxCejAni4CST1g, address:104.36.82.9:6023): protocol: storage node overloaded, request limit: 2000 Jun 22 01:32:48.510513169 (81h9m24.073836177s) [noise=true , address=104.36.82.42:6811 , piece_id=A5BFEHZ3WGGK53IZ7MKI6N4CVXHOORF6L47NR3UFDCAX7LBHRHVQ, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:48.565245605 (81h9m24.128568606s) [noise=true , address=79.127.205.234:5601 , piece_id=V7PMXBP2VFWBSZDG4GTLBD435T6F5DT7A43L4GQLQ5BDDTJ4WY6A, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:49.979979824 (81h9m25.543302825s) [noise=true , address=104.36.82.15:6289 , piece_id=GWL3P55BIK4IJEL5YFZKI7JCA4LADYOOHUSIQUNGJEKBJOTKMP5A, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:50.463750280 (81h9m26.02707328s) [noise=true , address=104.36.82.51:4084 , piece_id=D43FA7BFE54KPCWGDGDME2EGSWM7PONOVPD3DN2XY42MJFOKVDIQ, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:51.480599795 (81h9m27.043922798s) [noise=true , address=79.127.219.38:4023 , piece_id=BWLRCXXHBMBTNCZEVKKIIS5PB35PDQXQXCLIQB7GBLEIOWWUFBFA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:52.941200625 (81h9m28.504523627s) [noise=true , address=104.36.82.43:6841 , piece_id=PIHZMSTO3CII7KGDLGSAILTDRSLMEBVOLBCRCZJULMCFISXIZWXA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:58.281803598 (81h9m33.845126612s) [noise=true , address=64.49.13.139:4476 , piece_id=P7UKBNNZGI52B3I6WKTV6MVC262FYNKYYPJUI5G47UO5S4MU7PWA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:33:00.277271877 (81h9m35.840594867s) [noise=true , address=104.36.82.55:4213 , piece_id=X4IHS2XSPH52LIMNUYT7KGWMAPUH3TEMJOEEVB3JV5N45SEOH7YA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:33:09.435339213 (81h9m44.998662206s) [noise=true , address=104.36.82.52:4089 , piece_id=JSOEKLNWA3W3LOVKI3EGGKJYJ6J3UZY6KGABKQPQPXTIOOPMKVHA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:33:31.472868445 (81h10m7.03619143s) [noise=true , address=104.36.82.53:4122 , piece_id=HZZQBIKHQWLUE5EHK4KSKJGIUEG7D44RQASUJHCYXX6BICCGX6EA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:33:47.235404653 (81h10m22.798727642s) [noise=true , address=104.36.82.12:6158 , piece_id=LJZEOJVJCJAR2BRILIJI3ZVDGYMPBBTP2G3HCTN3MHNS4KYDPASQ, seg_pos={0 0}, upload=64958]: Done uploading piece. err: ecclient: upload failed (node:1SjM6Utm2YizeUbrBUD9Z7etE1nLadW5Fa1zYP7MioPeeqhdJ7, address:104.36.82.12:6158): protocol: expected piece hash; context deadline exceeded; EOF Jun 22 01:33:47.235448297 (81h10m22.798771306s) [seg_pos={0 0}, upload=64958]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jun 22 01:33:47.265442393 (81h10m22.828765401s) [upload=64958]: Done uploading segments... Jun 22 01:33:47.265476139 (81h10m22.82879914s) [upload=64958]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration:* 13.887365656s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [.....................X.................X....................] duration: 11.177547856s FAIL (2025-06-22 02:14:46 UTC) orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded Duration: 11.177575538s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.....X......................................................] duration: 21.64294656s FAIL (2025-06-22 02:14:28 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== oad=64949]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jun 22 01:32:13.674692303 (81h8m49.238015311s) [upload=64949]: Done uploading segments... Jun 22 01:32:13.674727219 (81h8m49.238050233s) [upload=64949]: Upload finished. err: <nil> Jun 22 01:32:47.266407995 (81h9m22.829731002s) [upload=64958]: Starting upload Jun 22 01:32:47.266524767 (81h9m22.829847771s) [upload=64958]: Done starting upload Jun 22 01:32:47.266813109 (81h9m22.830136116s) [upload=64958]: Uploading segments... Jun 22 01:32:47.266969526 (81h9m22.830292526s) [upload=64958]: Got next segment. Inline: false Jun 22 01:32:47.267075475 (81h9m22.830398475s) [upload=64958]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 22 01:32:47.267081679 (81h9m22.830404667s) [upload=64958]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 22 01:32:47.301770382 (81h9m22.865093389s) [seg_pos={0 0}, upload=64958]: Begin upload segment... Jun 22 01:32:47.301823796 (81h9m22.865146799s) [seg_pos={0 0}, upload=64958]: Done begin upload segment. Jun 22 01:32:47.301855055 (81h9m22.865178087s) [noise=true , address=104.36.82.43:6841 , piece_id=PIHZMSTO3CII7KGDLGSAILTDRSLMEBVOLBCRCZJULMCFISXIZWXA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.301902600 (81h9m22.865225606s) [noise=true , address=104.36.82.42:6811 , piece_id=A5BFEHZ3WGGK53IZ7MKI6N4CVXHOORF6L47NR3UFDCAX7LBHRHVQ, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.302234027 (81h9m22.865557029s) [noise=true , address=104.36.82.51:4084 , piece_id=D43FA7BFE54KPCWGDGDME2EGSWM7PONOVPD3DN2XY42MJFOKVDIQ, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.302496354 (81h9m22.865819363s) [noise=true , address=79.127.219.38:4023 , piece_id=BWLRCXXHBMBTNCZEVKKIIS5PB35PDQXQXCLIQB7GBLEIOWWUFBFA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.302577370 (81h9m22.865900378s) [noise=true , address=104.36.82.52:4089 , piece_id=JSOEKLNWA3W3LOVKI3EGGKJYJ6J3UZY6KGABKQPQPXTIOOPMKVHA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.302917829 (81h9m22.866240835s) [noise=true , address=104.36.82.53:4122 , piece_id=HZZQBIKHQWLUE5EHK4KSKJGIUEG7D44RQASUJHCYXX6BICCGX6EA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.303087165 (81h9m22.866410168s) [noise=true , address=77.90.31.14:4689 , piece_id=Q3HDS3BCMYDTODFRMU4XFRDCQK2QZE3ZMXVLUX4M54PDEBS7RR6A, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.303250714 (81h9m22.866573717s) [noise=true , address=79.127.205.234:5601 , piece_id=V7PMXBP2VFWBSZDG4GTLBD435T6F5DT7A43L4GQLQ5BDDTJ4WY6A, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.303579249 (81h9m22.866902255s) [noise=true , address=104.36.82.12:6158 , piece_id=LJZEOJVJCJAR2BRILIJI3ZVDGYMPBBTP2G3HCTN3MHNS4KYDPASQ, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.303599776 (81h9m22.866922784s) [noise=true , address=104.36.82.9:6023 , piece_id=SISFAUU4RPF57OLGWM2BWQWE6GRUSCMORBYHWEXETNAVM2EEMO6Q, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.304005346 (81h9m22.867328346s) [noise=true , address=45.91.116.12:5242 , piece_id=6ABT5IPYGSLFFJMBYMBYRU4EYIXABUAUHG3OVILOHBIUDS3BP27A, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.304388850 (81h9m22.867711851s) [noise=true , address=64.49.13.137:4500 , piece_id=FAE2RXD57IRP5FXA2XR6THXFW27U7IWAZ4JDV5M2WRPXKUGZ4VIA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.304881529 (81h9m22.868204536s) [noise=true , address=79.127.220.101:6048 , piece_id=AN7UQUXB42T2YXEH7ZJRZUDIW47M3RWUEOCL2VEBOGH74BZF3YKA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.305249860 (81h9m22.86857287s) [noise=true , address=104.36.82.54:4165 , piece_id=4CVOABPCWC7J7SRHZB6IN2WJJ6KGWUQ66ELUIZIFA67ETLQ7JEUQ, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.305604108 (81h9m22.868927112s) [noise=true , address=104.36.82.15:6289 , piece_id=GWL3P55BIK4IJEL5YFZKI7JCA4LADYOOHUSIQUNGJEKBJOTKMP5A, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.305930423 (81h9m22.869253425s) [noise=true , address=64.49.13.139:4476 , piece_id=P7UKBNNZGI52B3I6WKTV6MVC262FYNKYYPJUI5G47UO5S4MU7PWA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.306355311 (81h9m22.869678319s) [noise=true , address=104.36.82.55:4213 , piece_id=X4IHS2XSPH52LIMNUYT7KGWMAPUH3TEMJOEEVB3JV5N45SEOH7YA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.306788243 (81h9m22.870111252s) [noise=true , address=104.36.82.22:6539 , piece_id=GHQ2MVSTDGVYLUKLV23D6YWRR4PP7UUYHCUEQ4WWSWEMTBFO3P4A, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.392174770 (81h9m22.955497773s) [upload=64958]: Next returned nil segment Jun 22 01:32:47.392195767 (81h9m22.955518775s) [upload=64958]: Waiting for error group managing segments... Jun 22 01:32:47.480767633 (81h9m23.044090623s) [noise=true , address=104.36.82.22:6539 , piece_id=GHQ2MVSTDGVYLUKLV23D6YWRR4PP7UUYHCUEQ4WWSWEMTBFO3P4A, seg_pos={0 0}, upload=64958]: Done uploading piece. err: ecclient: upload failed (node:1QL6w8DSFR6CxDdnJmDuyVZxp6JU55VKiJofBZRfQi7q31i8Jc, address:104.36.82.22:6539): protocol: storage node overloaded, request limit: 2000 Jun 22 01:32:47.585663142 (81h9m23.148986155s) [noise=true , address=79.127.220.101:6048 , piece_id=AN7UQUXB42T2YXEH7ZJRZUDIW47M3RWUEOCL2VEBOGH74BZF3YKA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:47.640263002 (81h9m23.203586013s) [noise=true , address=45.91.116.12:5242 , piece_id=6ABT5IPYGSLFFJMBYMBYRU4EYIXABUAUHG3OVILOHBIUDS3BP27A, seg_pos={0 0}, upload=64958]: Done uploading piece. err: ecclient: upload failed (node:12JsFL62jgXmza8Md8eQ9k8cyw5V9Vz2RzN4y85ZvgHZjwrq4vQ, address:45.91.116.12:5242): protocol: storage node overloaded, request limit: 2000 Jun 22 01:32:47.984127275 (81h9m23.54745028s) [noise=true , address=64.49.13.137:4500 , piece_id=FAE2RXD57IRP5FXA2XR6THXFW27U7IWAZ4JDV5M2WRPXKUGZ4VIA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:48.100255249 (81h9m23.663578241s) [noise=true , address=77.90.31.14:4689 , piece_id=Q3HDS3BCMYDTODFRMU4XFRDCQK2QZE3ZMXVLUX4M54PDEBS7RR6A, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:48.221764035 (81h9m23.785087034s) [noise=true , address=104.36.82.54:4165 , piece_id=4CVOABPCWC7J7SRHZB6IN2WJJ6KGWUQ66ELUIZIFA67ETLQ7JEUQ, seg_pos={0 0}, upload=64958]: Done uploading piece. err: ecclient: upload failed (node:12wGUMXVEubEtki4qGnjGEsbYFB1iUauZcu3oDRKrCQsJMV3bV2, address:104.36.82.54:4165): protocol: storage node overloaded, request limit: 2000 Jun 22 01:32:48.509996527 (81h9m24.073319547s) [noise=true , address=104.36.82.9:6023 , piece_id=SISFAUU4RPF57OLGWM2BWQWE6GRUSCMORBYHWEXETNAVM2EEMO6Q, seg_pos={0 0}, upload=64958]: Done uploading piece. err: ecclient: upload failed (node:1j2S9tgWpLwmyvXZSq2qKKcbaC8r7LkzfDCaMxCejAni4CST1g, address:104.36.82.9:6023): protocol: storage node overloaded, request limit: 2000 Jun 22 01:32:48.510513169 (81h9m24.073836177s) [noise=true , address=104.36.82.42:6811 , piece_id=A5BFEHZ3WGGK53IZ7MKI6N4CVXHOORF6L47NR3UFDCAX7LBHRHVQ, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:48.565245605 (81h9m24.128568606s) [noise=true , address=79.127.205.234:5601 , piece_id=V7PMXBP2VFWBSZDG4GTLBD435T6F5DT7A43L4GQLQ5BDDTJ4WY6A, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:49.979979824 (81h9m25.543302825s) [noise=true , address=104.36.82.15:6289 , piece_id=GWL3P55BIK4IJEL5YFZKI7JCA4LADYOOHUSIQUNGJEKBJOTKMP5A, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:50.463750280 (81h9m26.02707328s) [noise=true , address=104.36.82.51:4084 , piece_id=D43FA7BFE54KPCWGDGDME2EGSWM7PONOVPD3DN2XY42MJFOKVDIQ, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:51.480599795 (81h9m27.043922798s) [noise=true , address=79.127.219.38:4023 , piece_id=BWLRCXXHBMBTNCZEVKKIIS5PB35PDQXQXCLIQB7GBLEIOWWUFBFA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:52.941200625 (81h9m28.504523627s) [noise=true , address=104.36.82.43:6841 , piece_id=PIHZMSTO3CII7KGDLGSAILTDRSLMEBVOLBCRCZJULMCFISXIZWXA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:58.281803598 (81h9m33.845126612s) [noise=true , address=64.49.13.139:4476 , piece_id=P7UKBNNZGI52B3I6WKTV6MVC262FYNKYYPJUI5G47UO5S4MU7PWA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:33:00.277271877 (81h9m35.840594867s) [noise=true , address=104.36.82.55:4213 , piece_id=X4IHS2XSPH52LIMNUYT7KGWMAPUH3TEMJOEEVB3JV5N45SEOH7YA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:33:09.435339213 (81h9m44.998662206s) [noise=true , address=104.36.82.52:4089 , piece_id=JSOEKLNWA3W3LOVKI3EGGKJYJ6J3UZY6KGABKQPQPXTIOOPMKVHA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:33:31.472868445 (81h10m7.03619143s) [noise=true , address=104.36.82.53:4122 , piece_id=HZZQBIKHQWLUE5EHK4KSKJGIUEG7D44RQASUJHCYXX6BICCGX6EA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:33:47.235404653 (81h10m22.798727642s) [noise=true , address=104.36.82.12:6158 , piece_id=LJZEOJVJCJAR2BRILIJI3ZVDGYMPBBTP2G3HCTN3MHNS4KYDPASQ, seg_pos={0 0}, upload=64958]: Done uploading piece. err: ecclient: upload failed (node:1SjM6Utm2YizeUbrBUD9Z7etE1nLadW5Fa1zYP7MioPeeqhdJ7, address:104.36.82.12:6158): protocol: expected piece hash; context deadline exceeded; EOF Jun 22 01:33:47.235448297 (81h10m22.798771306s) [seg_pos={0 0}, upload=64958]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jun 22 01:33:47.265442393 (81h10m22.828765401s) [upload=64958]: Done uploading segments... Jun 22 01:33:47.265476139 (81h10m22.82879914s) [upload=64958]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration:* 21.642981793s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [......................X.................X...................] duration: 17.08573282s FAIL (2025-06-22 02:13:58 UTC) orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded Duration: 17.085758897s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [......X.....................................................] duration: 13.560910504s FAIL (2025-06-22 02:13:48 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== oad=64949]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jun 22 01:32:13.674692303 (81h8m49.238015311s) [upload=64949]: Done uploading segments... Jun 22 01:32:13.674727219 (81h8m49.238050233s) [upload=64949]: Upload finished. err: <nil> Jun 22 01:32:47.266407995 (81h9m22.829731002s) [upload=64958]: Starting upload Jun 22 01:32:47.266524767 (81h9m22.829847771s) [upload=64958]: Done starting upload Jun 22 01:32:47.266813109 (81h9m22.830136116s) [upload=64958]: Uploading segments... Jun 22 01:32:47.266969526 (81h9m22.830292526s) [upload=64958]: Got next segment. Inline: false Jun 22 01:32:47.267075475 (81h9m22.830398475s) [upload=64958]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 22 01:32:47.267081679 (81h9m22.830404667s) [upload=64958]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 22 01:32:47.301770382 (81h9m22.865093389s) [seg_pos={0 0}, upload=64958]: Begin upload segment... Jun 22 01:32:47.301823796 (81h9m22.865146799s) [seg_pos={0 0}, upload=64958]: Done begin upload segment. Jun 22 01:32:47.301855055 (81h9m22.865178087s) [noise=true , address=104.36.82.43:6841 , piece_id=PIHZMSTO3CII7KGDLGSAILTDRSLMEBVOLBCRCZJULMCFISXIZWXA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.301902600 (81h9m22.865225606s) [noise=true , address=104.36.82.42:6811 , piece_id=A5BFEHZ3WGGK53IZ7MKI6N4CVXHOORF6L47NR3UFDCAX7LBHRHVQ, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.302234027 (81h9m22.865557029s) [noise=true , address=104.36.82.51:4084 , piece_id=D43FA7BFE54KPCWGDGDME2EGSWM7PONOVPD3DN2XY42MJFOKVDIQ, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.302496354 (81h9m22.865819363s) [noise=true , address=79.127.219.38:4023 , piece_id=BWLRCXXHBMBTNCZEVKKIIS5PB35PDQXQXCLIQB7GBLEIOWWUFBFA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.302577370 (81h9m22.865900378s) [noise=true , address=104.36.82.52:4089 , piece_id=JSOEKLNWA3W3LOVKI3EGGKJYJ6J3UZY6KGABKQPQPXTIOOPMKVHA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.302917829 (81h9m22.866240835s) [noise=true , address=104.36.82.53:4122 , piece_id=HZZQBIKHQWLUE5EHK4KSKJGIUEG7D44RQASUJHCYXX6BICCGX6EA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.303087165 (81h9m22.866410168s) [noise=true , address=77.90.31.14:4689 , piece_id=Q3HDS3BCMYDTODFRMU4XFRDCQK2QZE3ZMXVLUX4M54PDEBS7RR6A, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.303250714 (81h9m22.866573717s) [noise=true , address=79.127.205.234:5601 , piece_id=V7PMXBP2VFWBSZDG4GTLBD435T6F5DT7A43L4GQLQ5BDDTJ4WY6A, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.303579249 (81h9m22.866902255s) [noise=true , address=104.36.82.12:6158 , piece_id=LJZEOJVJCJAR2BRILIJI3ZVDGYMPBBTP2G3HCTN3MHNS4KYDPASQ, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.303599776 (81h9m22.866922784s) [noise=true , address=104.36.82.9:6023 , piece_id=SISFAUU4RPF57OLGWM2BWQWE6GRUSCMORBYHWEXETNAVM2EEMO6Q, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.304005346 (81h9m22.867328346s) [noise=true , address=45.91.116.12:5242 , piece_id=6ABT5IPYGSLFFJMBYMBYRU4EYIXABUAUHG3OVILOHBIUDS3BP27A, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.304388850 (81h9m22.867711851s) [noise=true , address=64.49.13.137:4500 , piece_id=FAE2RXD57IRP5FXA2XR6THXFW27U7IWAZ4JDV5M2WRPXKUGZ4VIA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.304881529 (81h9m22.868204536s) [noise=true , address=79.127.220.101:6048 , piece_id=AN7UQUXB42T2YXEH7ZJRZUDIW47M3RWUEOCL2VEBOGH74BZF3YKA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.305249860 (81h9m22.86857287s) [noise=true , address=104.36.82.54:4165 , piece_id=4CVOABPCWC7J7SRHZB6IN2WJJ6KGWUQ66ELUIZIFA67ETLQ7JEUQ, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.305604108 (81h9m22.868927112s) [noise=true , address=104.36.82.15:6289 , piece_id=GWL3P55BIK4IJEL5YFZKI7JCA4LADYOOHUSIQUNGJEKBJOTKMP5A, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.305930423 (81h9m22.869253425s) [noise=true , address=64.49.13.139:4476 , piece_id=P7UKBNNZGI52B3I6WKTV6MVC262FYNKYYPJUI5G47UO5S4MU7PWA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.306355311 (81h9m22.869678319s) [noise=true , address=104.36.82.55:4213 , piece_id=X4IHS2XSPH52LIMNUYT7KGWMAPUH3TEMJOEEVB3JV5N45SEOH7YA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.306788243 (81h9m22.870111252s) [noise=true , address=104.36.82.22:6539 , piece_id=GHQ2MVSTDGVYLUKLV23D6YWRR4PP7UUYHCUEQ4WWSWEMTBFO3P4A, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.392174770 (81h9m22.955497773s) [upload=64958]: Next returned nil segment Jun 22 01:32:47.392195767 (81h9m22.955518775s) [upload=64958]: Waiting for error group managing segments... Jun 22 01:32:47.480767633 (81h9m23.044090623s) [noise=true , address=104.36.82.22:6539 , piece_id=GHQ2MVSTDGVYLUKLV23D6YWRR4PP7UUYHCUEQ4WWSWEMTBFO3P4A, seg_pos={0 0}, upload=64958]: Done uploading piece. err: ecclient: upload failed (node:1QL6w8DSFR6CxDdnJmDuyVZxp6JU55VKiJofBZRfQi7q31i8Jc, address:104.36.82.22:6539): protocol: storage node overloaded, request limit: 2000 Jun 22 01:32:47.585663142 (81h9m23.148986155s) [noise=true , address=79.127.220.101:6048 , piece_id=AN7UQUXB42T2YXEH7ZJRZUDIW47M3RWUEOCL2VEBOGH74BZF3YKA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:47.640263002 (81h9m23.203586013s) [noise=true , address=45.91.116.12:5242 , piece_id=6ABT5IPYGSLFFJMBYMBYRU4EYIXABUAUHG3OVILOHBIUDS3BP27A, seg_pos={0 0}, upload=64958]: Done uploading piece. err: ecclient: upload failed (node:12JsFL62jgXmza8Md8eQ9k8cyw5V9Vz2RzN4y85ZvgHZjwrq4vQ, address:45.91.116.12:5242): protocol: storage node overloaded, request limit: 2000 Jun 22 01:32:47.984127275 (81h9m23.54745028s) [noise=true , address=64.49.13.137:4500 , piece_id=FAE2RXD57IRP5FXA2XR6THXFW27U7IWAZ4JDV5M2WRPXKUGZ4VIA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:48.100255249 (81h9m23.663578241s) [noise=true , address=77.90.31.14:4689 , piece_id=Q3HDS3BCMYDTODFRMU4XFRDCQK2QZE3ZMXVLUX4M54PDEBS7RR6A, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:48.221764035 (81h9m23.785087034s) [noise=true , address=104.36.82.54:4165 , piece_id=4CVOABPCWC7J7SRHZB6IN2WJJ6KGWUQ66ELUIZIFA67ETLQ7JEUQ, seg_pos={0 0}, upload=64958]: Done uploading piece. err: ecclient: upload failed (node:12wGUMXVEubEtki4qGnjGEsbYFB1iUauZcu3oDRKrCQsJMV3bV2, address:104.36.82.54:4165): protocol: storage node overloaded, request limit: 2000 Jun 22 01:32:48.509996527 (81h9m24.073319547s) [noise=true , address=104.36.82.9:6023 , piece_id=SISFAUU4RPF57OLGWM2BWQWE6GRUSCMORBYHWEXETNAVM2EEMO6Q, seg_pos={0 0}, upload=64958]: Done uploading piece. err: ecclient: upload failed (node:1j2S9tgWpLwmyvXZSq2qKKcbaC8r7LkzfDCaMxCejAni4CST1g, address:104.36.82.9:6023): protocol: storage node overloaded, request limit: 2000 Jun 22 01:32:48.510513169 (81h9m24.073836177s) [noise=true , address=104.36.82.42:6811 , piece_id=A5BFEHZ3WGGK53IZ7MKI6N4CVXHOORF6L47NR3UFDCAX7LBHRHVQ, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:48.565245605 (81h9m24.128568606s) [noise=true , address=79.127.205.234:5601 , piece_id=V7PMXBP2VFWBSZDG4GTLBD435T6F5DT7A43L4GQLQ5BDDTJ4WY6A, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:49.979979824 (81h9m25.543302825s) [noise=true , address=104.36.82.15:6289 , piece_id=GWL3P55BIK4IJEL5YFZKI7JCA4LADYOOHUSIQUNGJEKBJOTKMP5A, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:50.463750280 (81h9m26.02707328s) [noise=true , address=104.36.82.51:4084 , piece_id=D43FA7BFE54KPCWGDGDME2EGSWM7PONOVPD3DN2XY42MJFOKVDIQ, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:51.480599795 (81h9m27.043922798s) [noise=true , address=79.127.219.38:4023 , piece_id=BWLRCXXHBMBTNCZEVKKIIS5PB35PDQXQXCLIQB7GBLEIOWWUFBFA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:52.941200625 (81h9m28.504523627s) [noise=true , address=104.36.82.43:6841 , piece_id=PIHZMSTO3CII7KGDLGSAILTDRSLMEBVOLBCRCZJULMCFISXIZWXA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:58.281803598 (81h9m33.845126612s) [noise=true , address=64.49.13.139:4476 , piece_id=P7UKBNNZGI52B3I6WKTV6MVC262FYNKYYPJUI5G47UO5S4MU7PWA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:33:00.277271877 (81h9m35.840594867s) [noise=true , address=104.36.82.55:4213 , piece_id=X4IHS2XSPH52LIMNUYT7KGWMAPUH3TEMJOEEVB3JV5N45SEOH7YA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:33:09.435339213 (81h9m44.998662206s) [noise=true , address=104.36.82.52:4089 , piece_id=JSOEKLNWA3W3LOVKI3EGGKJYJ6J3UZY6KGABKQPQPXTIOOPMKVHA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:33:31.472868445 (81h10m7.03619143s) [noise=true , address=104.36.82.53:4122 , piece_id=HZZQBIKHQWLUE5EHK4KSKJGIUEG7D44RQASUJHCYXX6BICCGX6EA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:33:47.235404653 (81h10m22.798727642s) [noise=true , address=104.36.82.12:6158 , piece_id=LJZEOJVJCJAR2BRILIJI3ZVDGYMPBBTP2G3HCTN3MHNS4KYDPASQ, seg_pos={0 0}, upload=64958]: Done uploading piece. err: ecclient: upload failed (node:1SjM6Utm2YizeUbrBUD9Z7etE1nLadW5Fa1zYP7MioPeeqhdJ7, address:104.36.82.12:6158): protocol: expected piece hash; context deadline exceeded; EOF Jun 22 01:33:47.235448297 (81h10m22.798771306s) [seg_pos={0 0}, upload=64958]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jun 22 01:33:47.265442393 (81h10m22.828765401s) [upload=64958]: Done uploading segments... Jun 22 01:33:47.265476139 (81h10m22.82879914s) [upload=64958]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration:* 13.560940438s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.......X....................................................] duration: 30.736011807s FAIL (2025-06-22 02:13:15 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== oad=64949]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jun 22 01:32:13.674692303 (81h8m49.238015311s) [upload=64949]: Done uploading segments... Jun 22 01:32:13.674727219 (81h8m49.238050233s) [upload=64949]: Upload finished. err: <nil> Jun 22 01:32:47.266407995 (81h9m22.829731002s) [upload=64958]: Starting upload Jun 22 01:32:47.266524767 (81h9m22.829847771s) [upload=64958]: Done starting upload Jun 22 01:32:47.266813109 (81h9m22.830136116s) [upload=64958]: Uploading segments... Jun 22 01:32:47.266969526 (81h9m22.830292526s) [upload=64958]: Got next segment. Inline: false Jun 22 01:32:47.267075475 (81h9m22.830398475s) [upload=64958]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 22 01:32:47.267081679 (81h9m22.830404667s) [upload=64958]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 22 01:32:47.301770382 (81h9m22.865093389s) [seg_pos={0 0}, upload=64958]: Begin upload segment... Jun 22 01:32:47.301823796 (81h9m22.865146799s) [seg_pos={0 0}, upload=64958]: Done begin upload segment. Jun 22 01:32:47.301855055 (81h9m22.865178087s) [noise=true , address=104.36.82.43:6841 , piece_id=PIHZMSTO3CII7KGDLGSAILTDRSLMEBVOLBCRCZJULMCFISXIZWXA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.301902600 (81h9m22.865225606s) [noise=true , address=104.36.82.42:6811 , piece_id=A5BFEHZ3WGGK53IZ7MKI6N4CVXHOORF6L47NR3UFDCAX7LBHRHVQ, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.302234027 (81h9m22.865557029s) [noise=true , address=104.36.82.51:4084 , piece_id=D43FA7BFE54KPCWGDGDME2EGSWM7PONOVPD3DN2XY42MJFOKVDIQ, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.302496354 (81h9m22.865819363s) [noise=true , address=79.127.219.38:4023 , piece_id=BWLRCXXHBMBTNCZEVKKIIS5PB35PDQXQXCLIQB7GBLEIOWWUFBFA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.302577370 (81h9m22.865900378s) [noise=true , address=104.36.82.52:4089 , piece_id=JSOEKLNWA3W3LOVKI3EGGKJYJ6J3UZY6KGABKQPQPXTIOOPMKVHA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.302917829 (81h9m22.866240835s) [noise=true , address=104.36.82.53:4122 , piece_id=HZZQBIKHQWLUE5EHK4KSKJGIUEG7D44RQASUJHCYXX6BICCGX6EA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.303087165 (81h9m22.866410168s) [noise=true , address=77.90.31.14:4689 , piece_id=Q3HDS3BCMYDTODFRMU4XFRDCQK2QZE3ZMXVLUX4M54PDEBS7RR6A, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.303250714 (81h9m22.866573717s) [noise=true , address=79.127.205.234:5601 , piece_id=V7PMXBP2VFWBSZDG4GTLBD435T6F5DT7A43L4GQLQ5BDDTJ4WY6A, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.303579249 (81h9m22.866902255s) [noise=true , address=104.36.82.12:6158 , piece_id=LJZEOJVJCJAR2BRILIJI3ZVDGYMPBBTP2G3HCTN3MHNS4KYDPASQ, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.303599776 (81h9m22.866922784s) [noise=true , address=104.36.82.9:6023 , piece_id=SISFAUU4RPF57OLGWM2BWQWE6GRUSCMORBYHWEXETNAVM2EEMO6Q, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.304005346 (81h9m22.867328346s) [noise=true , address=45.91.116.12:5242 , piece_id=6ABT5IPYGSLFFJMBYMBYRU4EYIXABUAUHG3OVILOHBIUDS3BP27A, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.304388850 (81h9m22.867711851s) [noise=true , address=64.49.13.137:4500 , piece_id=FAE2RXD57IRP5FXA2XR6THXFW27U7IWAZ4JDV5M2WRPXKUGZ4VIA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.304881529 (81h9m22.868204536s) [noise=true , address=79.127.220.101:6048 , piece_id=AN7UQUXB42T2YXEH7ZJRZUDIW47M3RWUEOCL2VEBOGH74BZF3YKA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.305249860 (81h9m22.86857287s) [noise=true , address=104.36.82.54:4165 , piece_id=4CVOABPCWC7J7SRHZB6IN2WJJ6KGWUQ66ELUIZIFA67ETLQ7JEUQ, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.305604108 (81h9m22.868927112s) [noise=true , address=104.36.82.15:6289 , piece_id=GWL3P55BIK4IJEL5YFZKI7JCA4LADYOOHUSIQUNGJEKBJOTKMP5A, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.305930423 (81h9m22.869253425s) [noise=true , address=64.49.13.139:4476 , piece_id=P7UKBNNZGI52B3I6WKTV6MVC262FYNKYYPJUI5G47UO5S4MU7PWA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.306355311 (81h9m22.869678319s) [noise=true , address=104.36.82.55:4213 , piece_id=X4IHS2XSPH52LIMNUYT7KGWMAPUH3TEMJOEEVB3JV5N45SEOH7YA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.306788243 (81h9m22.870111252s) [noise=true , address=104.36.82.22:6539 , piece_id=GHQ2MVSTDGVYLUKLV23D6YWRR4PP7UUYHCUEQ4WWSWEMTBFO3P4A, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.392174770 (81h9m22.955497773s) [upload=64958]: Next returned nil segment Jun 22 01:32:47.392195767 (81h9m22.955518775s) [upload=64958]: Waiting for error group managing segments... Jun 22 01:32:47.480767633 (81h9m23.044090623s) [noise=true , address=104.36.82.22:6539 , piece_id=GHQ2MVSTDGVYLUKLV23D6YWRR4PP7UUYHCUEQ4WWSWEMTBFO3P4A, seg_pos={0 0}, upload=64958]: Done uploading piece. err: ecclient: upload failed (node:1QL6w8DSFR6CxDdnJmDuyVZxp6JU55VKiJofBZRfQi7q31i8Jc, address:104.36.82.22:6539): protocol: storage node overloaded, request limit: 2000 Jun 22 01:32:47.585663142 (81h9m23.148986155s) [noise=true , address=79.127.220.101:6048 , piece_id=AN7UQUXB42T2YXEH7ZJRZUDIW47M3RWUEOCL2VEBOGH74BZF3YKA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:47.640263002 (81h9m23.203586013s) [noise=true , address=45.91.116.12:5242 , piece_id=6ABT5IPYGSLFFJMBYMBYRU4EYIXABUAUHG3OVILOHBIUDS3BP27A, seg_pos={0 0}, upload=64958]: Done uploading piece. err: ecclient: upload failed (node:12JsFL62jgXmza8Md8eQ9k8cyw5V9Vz2RzN4y85ZvgHZjwrq4vQ, address:45.91.116.12:5242): protocol: storage node overloaded, request limit: 2000 Jun 22 01:32:47.984127275 (81h9m23.54745028s) [noise=true , address=64.49.13.137:4500 , piece_id=FAE2RXD57IRP5FXA2XR6THXFW27U7IWAZ4JDV5M2WRPXKUGZ4VIA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:48.100255249 (81h9m23.663578241s) [noise=true , address=77.90.31.14:4689 , piece_id=Q3HDS3BCMYDTODFRMU4XFRDCQK2QZE3ZMXVLUX4M54PDEBS7RR6A, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:48.221764035 (81h9m23.785087034s) [noise=true , address=104.36.82.54:4165 , piece_id=4CVOABPCWC7J7SRHZB6IN2WJJ6KGWUQ66ELUIZIFA67ETLQ7JEUQ, seg_pos={0 0}, upload=64958]: Done uploading piece. err: ecclient: upload failed (node:12wGUMXVEubEtki4qGnjGEsbYFB1iUauZcu3oDRKrCQsJMV3bV2, address:104.36.82.54:4165): protocol: storage node overloaded, request limit: 2000 Jun 22 01:32:48.509996527 (81h9m24.073319547s) [noise=true , address=104.36.82.9:6023 , piece_id=SISFAUU4RPF57OLGWM2BWQWE6GRUSCMORBYHWEXETNAVM2EEMO6Q, seg_pos={0 0}, upload=64958]: Done uploading piece. err: ecclient: upload failed (node:1j2S9tgWpLwmyvXZSq2qKKcbaC8r7LkzfDCaMxCejAni4CST1g, address:104.36.82.9:6023): protocol: storage node overloaded, request limit: 2000 Jun 22 01:32:48.510513169 (81h9m24.073836177s) [noise=true , address=104.36.82.42:6811 , piece_id=A5BFEHZ3WGGK53IZ7MKI6N4CVXHOORF6L47NR3UFDCAX7LBHRHVQ, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:48.565245605 (81h9m24.128568606s) [noise=true , address=79.127.205.234:5601 , piece_id=V7PMXBP2VFWBSZDG4GTLBD435T6F5DT7A43L4GQLQ5BDDTJ4WY6A, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:49.979979824 (81h9m25.543302825s) [noise=true , address=104.36.82.15:6289 , piece_id=GWL3P55BIK4IJEL5YFZKI7JCA4LADYOOHUSIQUNGJEKBJOTKMP5A, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:50.463750280 (81h9m26.02707328s) [noise=true , address=104.36.82.51:4084 , piece_id=D43FA7BFE54KPCWGDGDME2EGSWM7PONOVPD3DN2XY42MJFOKVDIQ, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:51.480599795 (81h9m27.043922798s) [noise=true , address=79.127.219.38:4023 , piece_id=BWLRCXXHBMBTNCZEVKKIIS5PB35PDQXQXCLIQB7GBLEIOWWUFBFA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:52.941200625 (81h9m28.504523627s) [noise=true , address=104.36.82.43:6841 , piece_id=PIHZMSTO3CII7KGDLGSAILTDRSLMEBVOLBCRCZJULMCFISXIZWXA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:58.281803598 (81h9m33.845126612s) [noise=true , address=64.49.13.139:4476 , piece_id=P7UKBNNZGI52B3I6WKTV6MVC262FYNKYYPJUI5G47UO5S4MU7PWA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:33:00.277271877 (81h9m35.840594867s) [noise=true , address=104.36.82.55:4213 , piece_id=X4IHS2XSPH52LIMNUYT7KGWMAPUH3TEMJOEEVB3JV5N45SEOH7YA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:33:09.435339213 (81h9m44.998662206s) [noise=true , address=104.36.82.52:4089 , piece_id=JSOEKLNWA3W3LOVKI3EGGKJYJ6J3UZY6KGABKQPQPXTIOOPMKVHA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:33:31.472868445 (81h10m7.03619143s) [noise=true , address=104.36.82.53:4122 , piece_id=HZZQBIKHQWLUE5EHK4KSKJGIUEG7D44RQASUJHCYXX6BICCGX6EA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:33:47.235404653 (81h10m22.798727642s) [noise=true , address=104.36.82.12:6158 , piece_id=LJZEOJVJCJAR2BRILIJI3ZVDGYMPBBTP2G3HCTN3MHNS4KYDPASQ, seg_pos={0 0}, upload=64958]: Done uploading piece. err: ecclient: upload failed (node:1SjM6Utm2YizeUbrBUD9Z7etE1nLadW5Fa1zYP7MioPeeqhdJ7, address:104.36.82.12:6158): protocol: expected piece hash; context deadline exceeded; EOF Jun 22 01:33:47.235448297 (81h10m22.798771306s) [seg_pos={0 0}, upload=64958]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jun 22 01:33:47.265442393 (81h10m22.828765401s) [upload=64958]: Done uploading segments... Jun 22 01:33:47.265476139 (81h10m22.82879914s) [upload=64958]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration:* 30.736091152s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [.......................X.................X..................] duration: 25.0027999s FAIL (2025-06-22 02:13:04 UTC) orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded Duration: 25.002825744s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [........X...................................................] duration: 10.851413039s FAIL (2025-06-22 02:12:23 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== oad=64949]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jun 22 01:32:13.674692303 (81h8m49.238015311s) [upload=64949]: Done uploading segments... Jun 22 01:32:13.674727219 (81h8m49.238050233s) [upload=64949]: Upload finished. err: <nil> Jun 22 01:32:47.266407995 (81h9m22.829731002s) [upload=64958]: Starting upload Jun 22 01:32:47.266524767 (81h9m22.829847771s) [upload=64958]: Done starting upload Jun 22 01:32:47.266813109 (81h9m22.830136116s) [upload=64958]: Uploading segments... Jun 22 01:32:47.266969526 (81h9m22.830292526s) [upload=64958]: Got next segment. Inline: false Jun 22 01:32:47.267075475 (81h9m22.830398475s) [upload=64958]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 22 01:32:47.267081679 (81h9m22.830404667s) [upload=64958]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 22 01:32:47.301770382 (81h9m22.865093389s) [seg_pos={0 0}, upload=64958]: Begin upload segment... Jun 22 01:32:47.301823796 (81h9m22.865146799s) [seg_pos={0 0}, upload=64958]: Done begin upload segment. Jun 22 01:32:47.301855055 (81h9m22.865178087s) [noise=true , address=104.36.82.43:6841 , piece_id=PIHZMSTO3CII7KGDLGSAILTDRSLMEBVOLBCRCZJULMCFISXIZWXA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.301902600 (81h9m22.865225606s) [noise=true , address=104.36.82.42:6811 , piece_id=A5BFEHZ3WGGK53IZ7MKI6N4CVXHOORF6L47NR3UFDCAX7LBHRHVQ, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.302234027 (81h9m22.865557029s) [noise=true , address=104.36.82.51:4084 , piece_id=D43FA7BFE54KPCWGDGDME2EGSWM7PONOVPD3DN2XY42MJFOKVDIQ, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.302496354 (81h9m22.865819363s) [noise=true , address=79.127.219.38:4023 , piece_id=BWLRCXXHBMBTNCZEVKKIIS5PB35PDQXQXCLIQB7GBLEIOWWUFBFA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.302577370 (81h9m22.865900378s) [noise=true , address=104.36.82.52:4089 , piece_id=JSOEKLNWA3W3LOVKI3EGGKJYJ6J3UZY6KGABKQPQPXTIOOPMKVHA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.302917829 (81h9m22.866240835s) [noise=true , address=104.36.82.53:4122 , piece_id=HZZQBIKHQWLUE5EHK4KSKJGIUEG7D44RQASUJHCYXX6BICCGX6EA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.303087165 (81h9m22.866410168s) [noise=true , address=77.90.31.14:4689 , piece_id=Q3HDS3BCMYDTODFRMU4XFRDCQK2QZE3ZMXVLUX4M54PDEBS7RR6A, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.303250714 (81h9m22.866573717s) [noise=true , address=79.127.205.234:5601 , piece_id=V7PMXBP2VFWBSZDG4GTLBD435T6F5DT7A43L4GQLQ5BDDTJ4WY6A, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.303579249 (81h9m22.866902255s) [noise=true , address=104.36.82.12:6158 , piece_id=LJZEOJVJCJAR2BRILIJI3ZVDGYMPBBTP2G3HCTN3MHNS4KYDPASQ, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.303599776 (81h9m22.866922784s) [noise=true , address=104.36.82.9:6023 , piece_id=SISFAUU4RPF57OLGWM2BWQWE6GRUSCMORBYHWEXETNAVM2EEMO6Q, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.304005346 (81h9m22.867328346s) [noise=true , address=45.91.116.12:5242 , piece_id=6ABT5IPYGSLFFJMBYMBYRU4EYIXABUAUHG3OVILOHBIUDS3BP27A, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.304388850 (81h9m22.867711851s) [noise=true , address=64.49.13.137:4500 , piece_id=FAE2RXD57IRP5FXA2XR6THXFW27U7IWAZ4JDV5M2WRPXKUGZ4VIA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.304881529 (81h9m22.868204536s) [noise=true , address=79.127.220.101:6048 , piece_id=AN7UQUXB42T2YXEH7ZJRZUDIW47M3RWUEOCL2VEBOGH74BZF3YKA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.305249860 (81h9m22.86857287s) [noise=true , address=104.36.82.54:4165 , piece_id=4CVOABPCWC7J7SRHZB6IN2WJJ6KGWUQ66ELUIZIFA67ETLQ7JEUQ, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.305604108 (81h9m22.868927112s) [noise=true , address=104.36.82.15:6289 , piece_id=GWL3P55BIK4IJEL5YFZKI7JCA4LADYOOHUSIQUNGJEKBJOTKMP5A, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.305930423 (81h9m22.869253425s) [noise=true , address=64.49.13.139:4476 , piece_id=P7UKBNNZGI52B3I6WKTV6MVC262FYNKYYPJUI5G47UO5S4MU7PWA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.306355311 (81h9m22.869678319s) [noise=true , address=104.36.82.55:4213 , piece_id=X4IHS2XSPH52LIMNUYT7KGWMAPUH3TEMJOEEVB3JV5N45SEOH7YA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.306788243 (81h9m22.870111252s) [noise=true , address=104.36.82.22:6539 , piece_id=GHQ2MVSTDGVYLUKLV23D6YWRR4PP7UUYHCUEQ4WWSWEMTBFO3P4A, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.392174770 (81h9m22.955497773s) [upload=64958]: Next returned nil segment Jun 22 01:32:47.392195767 (81h9m22.955518775s) [upload=64958]: Waiting for error group managing segments... Jun 22 01:32:47.480767633 (81h9m23.044090623s) [noise=true , address=104.36.82.22:6539 , piece_id=GHQ2MVSTDGVYLUKLV23D6YWRR4PP7UUYHCUEQ4WWSWEMTBFO3P4A, seg_pos={0 0}, upload=64958]: Done uploading piece. err: ecclient: upload failed (node:1QL6w8DSFR6CxDdnJmDuyVZxp6JU55VKiJofBZRfQi7q31i8Jc, address:104.36.82.22:6539): protocol: storage node overloaded, request limit: 2000 Jun 22 01:32:47.585663142 (81h9m23.148986155s) [noise=true , address=79.127.220.101:6048 , piece_id=AN7UQUXB42T2YXEH7ZJRZUDIW47M3RWUEOCL2VEBOGH74BZF3YKA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:47.640263002 (81h9m23.203586013s) [noise=true , address=45.91.116.12:5242 , piece_id=6ABT5IPYGSLFFJMBYMBYRU4EYIXABUAUHG3OVILOHBIUDS3BP27A, seg_pos={0 0}, upload=64958]: Done uploading piece. err: ecclient: upload failed (node:12JsFL62jgXmza8Md8eQ9k8cyw5V9Vz2RzN4y85ZvgHZjwrq4vQ, address:45.91.116.12:5242): protocol: storage node overloaded, request limit: 2000 Jun 22 01:32:47.984127275 (81h9m23.54745028s) [noise=true , address=64.49.13.137:4500 , piece_id=FAE2RXD57IRP5FXA2XR6THXFW27U7IWAZ4JDV5M2WRPXKUGZ4VIA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:48.100255249 (81h9m23.663578241s) [noise=true , address=77.90.31.14:4689 , piece_id=Q3HDS3BCMYDTODFRMU4XFRDCQK2QZE3ZMXVLUX4M54PDEBS7RR6A, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:48.221764035 (81h9m23.785087034s) [noise=true , address=104.36.82.54:4165 , piece_id=4CVOABPCWC7J7SRHZB6IN2WJJ6KGWUQ66ELUIZIFA67ETLQ7JEUQ, seg_pos={0 0}, upload=64958]: Done uploading piece. err: ecclient: upload failed (node:12wGUMXVEubEtki4qGnjGEsbYFB1iUauZcu3oDRKrCQsJMV3bV2, address:104.36.82.54:4165): protocol: storage node overloaded, request limit: 2000 Jun 22 01:32:48.509996527 (81h9m24.073319547s) [noise=true , address=104.36.82.9:6023 , piece_id=SISFAUU4RPF57OLGWM2BWQWE6GRUSCMORBYHWEXETNAVM2EEMO6Q, seg_pos={0 0}, upload=64958]: Done uploading piece. err: ecclient: upload failed (node:1j2S9tgWpLwmyvXZSq2qKKcbaC8r7LkzfDCaMxCejAni4CST1g, address:104.36.82.9:6023): protocol: storage node overloaded, request limit: 2000 Jun 22 01:32:48.510513169 (81h9m24.073836177s) [noise=true , address=104.36.82.42:6811 , piece_id=A5BFEHZ3WGGK53IZ7MKI6N4CVXHOORF6L47NR3UFDCAX7LBHRHVQ, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:48.565245605 (81h9m24.128568606s) [noise=true , address=79.127.205.234:5601 , piece_id=V7PMXBP2VFWBSZDG4GTLBD435T6F5DT7A43L4GQLQ5BDDTJ4WY6A, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:49.979979824 (81h9m25.543302825s) [noise=true , address=104.36.82.15:6289 , piece_id=GWL3P55BIK4IJEL5YFZKI7JCA4LADYOOHUSIQUNGJEKBJOTKMP5A, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:50.463750280 (81h9m26.02707328s) [noise=true , address=104.36.82.51:4084 , piece_id=D43FA7BFE54KPCWGDGDME2EGSWM7PONOVPD3DN2XY42MJFOKVDIQ, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:51.480599795 (81h9m27.043922798s) [noise=true , address=79.127.219.38:4023 , piece_id=BWLRCXXHBMBTNCZEVKKIIS5PB35PDQXQXCLIQB7GBLEIOWWUFBFA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:52.941200625 (81h9m28.504523627s) [noise=true , address=104.36.82.43:6841 , piece_id=PIHZMSTO3CII7KGDLGSAILTDRSLMEBVOLBCRCZJULMCFISXIZWXA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:58.281803598 (81h9m33.845126612s) [noise=true , address=64.49.13.139:4476 , piece_id=P7UKBNNZGI52B3I6WKTV6MVC262FYNKYYPJUI5G47UO5S4MU7PWA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:33:00.277271877 (81h9m35.840594867s) [noise=true , address=104.36.82.55:4213 , piece_id=X4IHS2XSPH52LIMNUYT7KGWMAPUH3TEMJOEEVB3JV5N45SEOH7YA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:33:09.435339213 (81h9m44.998662206s) [noise=true , address=104.36.82.52:4089 , piece_id=JSOEKLNWA3W3LOVKI3EGGKJYJ6J3UZY6KGABKQPQPXTIOOPMKVHA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:33:31.472868445 (81h10m7.03619143s) [noise=true , address=104.36.82.53:4122 , piece_id=HZZQBIKHQWLUE5EHK4KSKJGIUEG7D44RQASUJHCYXX6BICCGX6EA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:33:47.235404653 (81h10m22.798727642s) [noise=true , address=104.36.82.12:6158 , piece_id=LJZEOJVJCJAR2BRILIJI3ZVDGYMPBBTP2G3HCTN3MHNS4KYDPASQ, seg_pos={0 0}, upload=64958]: Done uploading piece. err: ecclient: upload failed (node:1SjM6Utm2YizeUbrBUD9Z7etE1nLadW5Fa1zYP7MioPeeqhdJ7, address:104.36.82.12:6158): protocol: expected piece hash; context deadline exceeded; EOF Jun 22 01:33:47.235448297 (81h10m22.798771306s) [seg_pos={0 0}, upload=64958]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jun 22 01:33:47.265442393 (81h10m22.828765401s) [upload=64958]: Done uploading segments... Jun 22 01:33:47.265476139 (81h10m22.82879914s) [upload=64958]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration:* 10.851447634s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [........................X.................X.................] duration: 39.640865737s FAIL (2025-06-22 02:12:14 UTC) orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded Duration: 39.640900147s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.........X..................................................] duration: 12.83560721s FAIL (2025-06-22 02:12:06 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== oad=64949]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jun 22 01:32:13.674692303 (81h8m49.238015311s) [upload=64949]: Done uploading segments... Jun 22 01:32:13.674727219 (81h8m49.238050233s) [upload=64949]: Upload finished. err: <nil> Jun 22 01:32:47.266407995 (81h9m22.829731002s) [upload=64958]: Starting upload Jun 22 01:32:47.266524767 (81h9m22.829847771s) [upload=64958]: Done starting upload Jun 22 01:32:47.266813109 (81h9m22.830136116s) [upload=64958]: Uploading segments... Jun 22 01:32:47.266969526 (81h9m22.830292526s) [upload=64958]: Got next segment. Inline: false Jun 22 01:32:47.267075475 (81h9m22.830398475s) [upload=64958]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 22 01:32:47.267081679 (81h9m22.830404667s) [upload=64958]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 22 01:32:47.301770382 (81h9m22.865093389s) [seg_pos={0 0}, upload=64958]: Begin upload segment... Jun 22 01:32:47.301823796 (81h9m22.865146799s) [seg_pos={0 0}, upload=64958]: Done begin upload segment. Jun 22 01:32:47.301855055 (81h9m22.865178087s) [noise=true , address=104.36.82.43:6841 , piece_id=PIHZMSTO3CII7KGDLGSAILTDRSLMEBVOLBCRCZJULMCFISXIZWXA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.301902600 (81h9m22.865225606s) [noise=true , address=104.36.82.42:6811 , piece_id=A5BFEHZ3WGGK53IZ7MKI6N4CVXHOORF6L47NR3UFDCAX7LBHRHVQ, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.302234027 (81h9m22.865557029s) [noise=true , address=104.36.82.51:4084 , piece_id=D43FA7BFE54KPCWGDGDME2EGSWM7PONOVPD3DN2XY42MJFOKVDIQ, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.302496354 (81h9m22.865819363s) [noise=true , address=79.127.219.38:4023 , piece_id=BWLRCXXHBMBTNCZEVKKIIS5PB35PDQXQXCLIQB7GBLEIOWWUFBFA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.302577370 (81h9m22.865900378s) [noise=true , address=104.36.82.52:4089 , piece_id=JSOEKLNWA3W3LOVKI3EGGKJYJ6J3UZY6KGABKQPQPXTIOOPMKVHA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.302917829 (81h9m22.866240835s) [noise=true , address=104.36.82.53:4122 , piece_id=HZZQBIKHQWLUE5EHK4KSKJGIUEG7D44RQASUJHCYXX6BICCGX6EA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.303087165 (81h9m22.866410168s) [noise=true , address=77.90.31.14:4689 , piece_id=Q3HDS3BCMYDTODFRMU4XFRDCQK2QZE3ZMXVLUX4M54PDEBS7RR6A, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.303250714 (81h9m22.866573717s) [noise=true , address=79.127.205.234:5601 , piece_id=V7PMXBP2VFWBSZDG4GTLBD435T6F5DT7A43L4GQLQ5BDDTJ4WY6A, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.303579249 (81h9m22.866902255s) [noise=true , address=104.36.82.12:6158 , piece_id=LJZEOJVJCJAR2BRILIJI3ZVDGYMPBBTP2G3HCTN3MHNS4KYDPASQ, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.303599776 (81h9m22.866922784s) [noise=true , address=104.36.82.9:6023 , piece_id=SISFAUU4RPF57OLGWM2BWQWE6GRUSCMORBYHWEXETNAVM2EEMO6Q, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.304005346 (81h9m22.867328346s) [noise=true , address=45.91.116.12:5242 , piece_id=6ABT5IPYGSLFFJMBYMBYRU4EYIXABUAUHG3OVILOHBIUDS3BP27A, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.304388850 (81h9m22.867711851s) [noise=true , address=64.49.13.137:4500 , piece_id=FAE2RXD57IRP5FXA2XR6THXFW27U7IWAZ4JDV5M2WRPXKUGZ4VIA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.304881529 (81h9m22.868204536s) [noise=true , address=79.127.220.101:6048 , piece_id=AN7UQUXB42T2YXEH7ZJRZUDIW47M3RWUEOCL2VEBOGH74BZF3YKA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.305249860 (81h9m22.86857287s) [noise=true , address=104.36.82.54:4165 , piece_id=4CVOABPCWC7J7SRHZB6IN2WJJ6KGWUQ66ELUIZIFA67ETLQ7JEUQ, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.305604108 (81h9m22.868927112s) [noise=true , address=104.36.82.15:6289 , piece_id=GWL3P55BIK4IJEL5YFZKI7JCA4LADYOOHUSIQUNGJEKBJOTKMP5A, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.305930423 (81h9m22.869253425s) [noise=true , address=64.49.13.139:4476 , piece_id=P7UKBNNZGI52B3I6WKTV6MVC262FYNKYYPJUI5G47UO5S4MU7PWA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.306355311 (81h9m22.869678319s) [noise=true , address=104.36.82.55:4213 , piece_id=X4IHS2XSPH52LIMNUYT7KGWMAPUH3TEMJOEEVB3JV5N45SEOH7YA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.306788243 (81h9m22.870111252s) [noise=true , address=104.36.82.22:6539 , piece_id=GHQ2MVSTDGVYLUKLV23D6YWRR4PP7UUYHCUEQ4WWSWEMTBFO3P4A, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.392174770 (81h9m22.955497773s) [upload=64958]: Next returned nil segment Jun 22 01:32:47.392195767 (81h9m22.955518775s) [upload=64958]: Waiting for error group managing segments... Jun 22 01:32:47.480767633 (81h9m23.044090623s) [noise=true , address=104.36.82.22:6539 , piece_id=GHQ2MVSTDGVYLUKLV23D6YWRR4PP7UUYHCUEQ4WWSWEMTBFO3P4A, seg_pos={0 0}, upload=64958]: Done uploading piece. err: ecclient: upload failed (node:1QL6w8DSFR6CxDdnJmDuyVZxp6JU55VKiJofBZRfQi7q31i8Jc, address:104.36.82.22:6539): protocol: storage node overloaded, request limit: 2000 Jun 22 01:32:47.585663142 (81h9m23.148986155s) [noise=true , address=79.127.220.101:6048 , piece_id=AN7UQUXB42T2YXEH7ZJRZUDIW47M3RWUEOCL2VEBOGH74BZF3YKA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:47.640263002 (81h9m23.203586013s) [noise=true , address=45.91.116.12:5242 , piece_id=6ABT5IPYGSLFFJMBYMBYRU4EYIXABUAUHG3OVILOHBIUDS3BP27A, seg_pos={0 0}, upload=64958]: Done uploading piece. err: ecclient: upload failed (node:12JsFL62jgXmza8Md8eQ9k8cyw5V9Vz2RzN4y85ZvgHZjwrq4vQ, address:45.91.116.12:5242): protocol: storage node overloaded, request limit: 2000 Jun 22 01:32:47.984127275 (81h9m23.54745028s) [noise=true , address=64.49.13.137:4500 , piece_id=FAE2RXD57IRP5FXA2XR6THXFW27U7IWAZ4JDV5M2WRPXKUGZ4VIA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:48.100255249 (81h9m23.663578241s) [noise=true , address=77.90.31.14:4689 , piece_id=Q3HDS3BCMYDTODFRMU4XFRDCQK2QZE3ZMXVLUX4M54PDEBS7RR6A, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:48.221764035 (81h9m23.785087034s) [noise=true , address=104.36.82.54:4165 , piece_id=4CVOABPCWC7J7SRHZB6IN2WJJ6KGWUQ66ELUIZIFA67ETLQ7JEUQ, seg_pos={0 0}, upload=64958]: Done uploading piece. err: ecclient: upload failed (node:12wGUMXVEubEtki4qGnjGEsbYFB1iUauZcu3oDRKrCQsJMV3bV2, address:104.36.82.54:4165): protocol: storage node overloaded, request limit: 2000 Jun 22 01:32:48.509996527 (81h9m24.073319547s) [noise=true , address=104.36.82.9:6023 , piece_id=SISFAUU4RPF57OLGWM2BWQWE6GRUSCMORBYHWEXETNAVM2EEMO6Q, seg_pos={0 0}, upload=64958]: Done uploading piece. err: ecclient: upload failed (node:1j2S9tgWpLwmyvXZSq2qKKcbaC8r7LkzfDCaMxCejAni4CST1g, address:104.36.82.9:6023): protocol: storage node overloaded, request limit: 2000 Jun 22 01:32:48.510513169 (81h9m24.073836177s) [noise=true , address=104.36.82.42:6811 , piece_id=A5BFEHZ3WGGK53IZ7MKI6N4CVXHOORF6L47NR3UFDCAX7LBHRHVQ, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:48.565245605 (81h9m24.128568606s) [noise=true , address=79.127.205.234:5601 , piece_id=V7PMXBP2VFWBSZDG4GTLBD435T6F5DT7A43L4GQLQ5BDDTJ4WY6A, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:49.979979824 (81h9m25.543302825s) [noise=true , address=104.36.82.15:6289 , piece_id=GWL3P55BIK4IJEL5YFZKI7JCA4LADYOOHUSIQUNGJEKBJOTKMP5A, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:50.463750280 (81h9m26.02707328s) [noise=true , address=104.36.82.51:4084 , piece_id=D43FA7BFE54KPCWGDGDME2EGSWM7PONOVPD3DN2XY42MJFOKVDIQ, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:51.480599795 (81h9m27.043922798s) [noise=true , address=79.127.219.38:4023 , piece_id=BWLRCXXHBMBTNCZEVKKIIS5PB35PDQXQXCLIQB7GBLEIOWWUFBFA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:52.941200625 (81h9m28.504523627s) [noise=true , address=104.36.82.43:6841 , piece_id=PIHZMSTO3CII7KGDLGSAILTDRSLMEBVOLBCRCZJULMCFISXIZWXA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:58.281803598 (81h9m33.845126612s) [noise=true , address=64.49.13.139:4476 , piece_id=P7UKBNNZGI52B3I6WKTV6MVC262FYNKYYPJUI5G47UO5S4MU7PWA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:33:00.277271877 (81h9m35.840594867s) [noise=true , address=104.36.82.55:4213 , piece_id=X4IHS2XSPH52LIMNUYT7KGWMAPUH3TEMJOEEVB3JV5N45SEOH7YA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:33:09.435339213 (81h9m44.998662206s) [noise=true , address=104.36.82.52:4089 , piece_id=JSOEKLNWA3W3LOVKI3EGGKJYJ6J3UZY6KGABKQPQPXTIOOPMKVHA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:33:31.472868445 (81h10m7.03619143s) [noise=true , address=104.36.82.53:4122 , piece_id=HZZQBIKHQWLUE5EHK4KSKJGIUEG7D44RQASUJHCYXX6BICCGX6EA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:33:47.235404653 (81h10m22.798727642s) [noise=true , address=104.36.82.12:6158 , piece_id=LJZEOJVJCJAR2BRILIJI3ZVDGYMPBBTP2G3HCTN3MHNS4KYDPASQ, seg_pos={0 0}, upload=64958]: Done uploading piece. err: ecclient: upload failed (node:1SjM6Utm2YizeUbrBUD9Z7etE1nLadW5Fa1zYP7MioPeeqhdJ7, address:104.36.82.12:6158): protocol: expected piece hash; context deadline exceeded; EOF Jun 22 01:33:47.235448297 (81h10m22.798771306s) [seg_pos={0 0}, upload=64958]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jun 22 01:33:47.265442393 (81h10m22.828765401s) [upload=64958]: Done uploading segments... Jun 22 01:33:47.265476139 (81h10m22.82879914s) [upload=64958]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration:* 12.835643869s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [..........X.................................................] duration: 20.716983002s FAIL (2025-06-22 02:11:36 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== oad=64949]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jun 22 01:32:13.674692303 (81h8m49.238015311s) [upload=64949]: Done uploading segments... Jun 22 01:32:13.674727219 (81h8m49.238050233s) [upload=64949]: Upload finished. err: <nil> Jun 22 01:32:47.266407995 (81h9m22.829731002s) [upload=64958]: Starting upload Jun 22 01:32:47.266524767 (81h9m22.829847771s) [upload=64958]: Done starting upload Jun 22 01:32:47.266813109 (81h9m22.830136116s) [upload=64958]: Uploading segments... Jun 22 01:32:47.266969526 (81h9m22.830292526s) [upload=64958]: Got next segment. Inline: false Jun 22 01:32:47.267075475 (81h9m22.830398475s) [upload=64958]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 22 01:32:47.267081679 (81h9m22.830404667s) [upload=64958]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 22 01:32:47.301770382 (81h9m22.865093389s) [seg_pos={0 0}, upload=64958]: Begin upload segment... Jun 22 01:32:47.301823796 (81h9m22.865146799s) [seg_pos={0 0}, upload=64958]: Done begin upload segment. Jun 22 01:32:47.301855055 (81h9m22.865178087s) [noise=true , address=104.36.82.43:6841 , piece_id=PIHZMSTO3CII7KGDLGSAILTDRSLMEBVOLBCRCZJULMCFISXIZWXA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.301902600 (81h9m22.865225606s) [noise=true , address=104.36.82.42:6811 , piece_id=A5BFEHZ3WGGK53IZ7MKI6N4CVXHOORF6L47NR3UFDCAX7LBHRHVQ, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.302234027 (81h9m22.865557029s) [noise=true , address=104.36.82.51:4084 , piece_id=D43FA7BFE54KPCWGDGDME2EGSWM7PONOVPD3DN2XY42MJFOKVDIQ, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.302496354 (81h9m22.865819363s) [noise=true , address=79.127.219.38:4023 , piece_id=BWLRCXXHBMBTNCZEVKKIIS5PB35PDQXQXCLIQB7GBLEIOWWUFBFA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.302577370 (81h9m22.865900378s) [noise=true , address=104.36.82.52:4089 , piece_id=JSOEKLNWA3W3LOVKI3EGGKJYJ6J3UZY6KGABKQPQPXTIOOPMKVHA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.302917829 (81h9m22.866240835s) [noise=true , address=104.36.82.53:4122 , piece_id=HZZQBIKHQWLUE5EHK4KSKJGIUEG7D44RQASUJHCYXX6BICCGX6EA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.303087165 (81h9m22.866410168s) [noise=true , address=77.90.31.14:4689 , piece_id=Q3HDS3BCMYDTODFRMU4XFRDCQK2QZE3ZMXVLUX4M54PDEBS7RR6A, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.303250714 (81h9m22.866573717s) [noise=true , address=79.127.205.234:5601 , piece_id=V7PMXBP2VFWBSZDG4GTLBD435T6F5DT7A43L4GQLQ5BDDTJ4WY6A, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.303579249 (81h9m22.866902255s) [noise=true , address=104.36.82.12:6158 , piece_id=LJZEOJVJCJAR2BRILIJI3ZVDGYMPBBTP2G3HCTN3MHNS4KYDPASQ, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.303599776 (81h9m22.866922784s) [noise=true , address=104.36.82.9:6023 , piece_id=SISFAUU4RPF57OLGWM2BWQWE6GRUSCMORBYHWEXETNAVM2EEMO6Q, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.304005346 (81h9m22.867328346s) [noise=true , address=45.91.116.12:5242 , piece_id=6ABT5IPYGSLFFJMBYMBYRU4EYIXABUAUHG3OVILOHBIUDS3BP27A, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.304388850 (81h9m22.867711851s) [noise=true , address=64.49.13.137:4500 , piece_id=FAE2RXD57IRP5FXA2XR6THXFW27U7IWAZ4JDV5M2WRPXKUGZ4VIA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.304881529 (81h9m22.868204536s) [noise=true , address=79.127.220.101:6048 , piece_id=AN7UQUXB42T2YXEH7ZJRZUDIW47M3RWUEOCL2VEBOGH74BZF3YKA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.305249860 (81h9m22.86857287s) [noise=true , address=104.36.82.54:4165 , piece_id=4CVOABPCWC7J7SRHZB6IN2WJJ6KGWUQ66ELUIZIFA67ETLQ7JEUQ, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.305604108 (81h9m22.868927112s) [noise=true , address=104.36.82.15:6289 , piece_id=GWL3P55BIK4IJEL5YFZKI7JCA4LADYOOHUSIQUNGJEKBJOTKMP5A, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.305930423 (81h9m22.869253425s) [noise=true , address=64.49.13.139:4476 , piece_id=P7UKBNNZGI52B3I6WKTV6MVC262FYNKYYPJUI5G47UO5S4MU7PWA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.306355311 (81h9m22.869678319s) [noise=true , address=104.36.82.55:4213 , piece_id=X4IHS2XSPH52LIMNUYT7KGWMAPUH3TEMJOEEVB3JV5N45SEOH7YA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.306788243 (81h9m22.870111252s) [noise=true , address=104.36.82.22:6539 , piece_id=GHQ2MVSTDGVYLUKLV23D6YWRR4PP7UUYHCUEQ4WWSWEMTBFO3P4A, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.392174770 (81h9m22.955497773s) [upload=64958]: Next returned nil segment Jun 22 01:32:47.392195767 (81h9m22.955518775s) [upload=64958]: Waiting for error group managing segments... Jun 22 01:32:47.480767633 (81h9m23.044090623s) [noise=true , address=104.36.82.22:6539 , piece_id=GHQ2MVSTDGVYLUKLV23D6YWRR4PP7UUYHCUEQ4WWSWEMTBFO3P4A, seg_pos={0 0}, upload=64958]: Done uploading piece. err: ecclient: upload failed (node:1QL6w8DSFR6CxDdnJmDuyVZxp6JU55VKiJofBZRfQi7q31i8Jc, address:104.36.82.22:6539): protocol: storage node overloaded, request limit: 2000 Jun 22 01:32:47.585663142 (81h9m23.148986155s) [noise=true , address=79.127.220.101:6048 , piece_id=AN7UQUXB42T2YXEH7ZJRZUDIW47M3RWUEOCL2VEBOGH74BZF3YKA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:47.640263002 (81h9m23.203586013s) [noise=true , address=45.91.116.12:5242 , piece_id=6ABT5IPYGSLFFJMBYMBYRU4EYIXABUAUHG3OVILOHBIUDS3BP27A, seg_pos={0 0}, upload=64958]: Done uploading piece. err: ecclient: upload failed (node:12JsFL62jgXmza8Md8eQ9k8cyw5V9Vz2RzN4y85ZvgHZjwrq4vQ, address:45.91.116.12:5242): protocol: storage node overloaded, request limit: 2000 Jun 22 01:32:47.984127275 (81h9m23.54745028s) [noise=true , address=64.49.13.137:4500 , piece_id=FAE2RXD57IRP5FXA2XR6THXFW27U7IWAZ4JDV5M2WRPXKUGZ4VIA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:48.100255249 (81h9m23.663578241s) [noise=true , address=77.90.31.14:4689 , piece_id=Q3HDS3BCMYDTODFRMU4XFRDCQK2QZE3ZMXVLUX4M54PDEBS7RR6A, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:48.221764035 (81h9m23.785087034s) [noise=true , address=104.36.82.54:4165 , piece_id=4CVOABPCWC7J7SRHZB6IN2WJJ6KGWUQ66ELUIZIFA67ETLQ7JEUQ, seg_pos={0 0}, upload=64958]: Done uploading piece. err: ecclient: upload failed (node:12wGUMXVEubEtki4qGnjGEsbYFB1iUauZcu3oDRKrCQsJMV3bV2, address:104.36.82.54:4165): protocol: storage node overloaded, request limit: 2000 Jun 22 01:32:48.509996527 (81h9m24.073319547s) [noise=true , address=104.36.82.9:6023 , piece_id=SISFAUU4RPF57OLGWM2BWQWE6GRUSCMORBYHWEXETNAVM2EEMO6Q, seg_pos={0 0}, upload=64958]: Done uploading piece. err: ecclient: upload failed (node:1j2S9tgWpLwmyvXZSq2qKKcbaC8r7LkzfDCaMxCejAni4CST1g, address:104.36.82.9:6023): protocol: storage node overloaded, request limit: 2000 Jun 22 01:32:48.510513169 (81h9m24.073836177s) [noise=true , address=104.36.82.42:6811 , piece_id=A5BFEHZ3WGGK53IZ7MKI6N4CVXHOORF6L47NR3UFDCAX7LBHRHVQ, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:48.565245605 (81h9m24.128568606s) [noise=true , address=79.127.205.234:5601 , piece_id=V7PMXBP2VFWBSZDG4GTLBD435T6F5DT7A43L4GQLQ5BDDTJ4WY6A, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:49.979979824 (81h9m25.543302825s) [noise=true , address=104.36.82.15:6289 , piece_id=GWL3P55BIK4IJEL5YFZKI7JCA4LADYOOHUSIQUNGJEKBJOTKMP5A, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:50.463750280 (81h9m26.02707328s) [noise=true , address=104.36.82.51:4084 , piece_id=D43FA7BFE54KPCWGDGDME2EGSWM7PONOVPD3DN2XY42MJFOKVDIQ, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:51.480599795 (81h9m27.043922798s) [noise=true , address=79.127.219.38:4023 , piece_id=BWLRCXXHBMBTNCZEVKKIIS5PB35PDQXQXCLIQB7GBLEIOWWUFBFA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:52.941200625 (81h9m28.504523627s) [noise=true , address=104.36.82.43:6841 , piece_id=PIHZMSTO3CII7KGDLGSAILTDRSLMEBVOLBCRCZJULMCFISXIZWXA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:58.281803598 (81h9m33.845126612s) [noise=true , address=64.49.13.139:4476 , piece_id=P7UKBNNZGI52B3I6WKTV6MVC262FYNKYYPJUI5G47UO5S4MU7PWA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:33:00.277271877 (81h9m35.840594867s) [noise=true , address=104.36.82.55:4213 , piece_id=X4IHS2XSPH52LIMNUYT7KGWMAPUH3TEMJOEEVB3JV5N45SEOH7YA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:33:09.435339213 (81h9m44.998662206s) [noise=true , address=104.36.82.52:4089 , piece_id=JSOEKLNWA3W3LOVKI3EGGKJYJ6J3UZY6KGABKQPQPXTIOOPMKVHA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:33:31.472868445 (81h10m7.03619143s) [noise=true , address=104.36.82.53:4122 , piece_id=HZZQBIKHQWLUE5EHK4KSKJGIUEG7D44RQASUJHCYXX6BICCGX6EA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:33:47.235404653 (81h10m22.798727642s) [noise=true , address=104.36.82.12:6158 , piece_id=LJZEOJVJCJAR2BRILIJI3ZVDGYMPBBTP2G3HCTN3MHNS4KYDPASQ, seg_pos={0 0}, upload=64958]: Done uploading piece. err: ecclient: upload failed (node:1SjM6Utm2YizeUbrBUD9Z7etE1nLadW5Fa1zYP7MioPeeqhdJ7, address:104.36.82.12:6158): protocol: expected piece hash; context deadline exceeded; EOF Jun 22 01:33:47.235448297 (81h10m22.798771306s) [seg_pos={0 0}, upload=64958]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jun 22 01:33:47.265442393 (81h10m22.828765401s) [upload=64958]: Done uploading segments... Jun 22 01:33:47.265476139 (81h10m22.82879914s) [upload=64958]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration:* 20.717009757s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [.........................X.................X................] duration: 25.201801345s FAIL (2025-06-22 02:11:25 UTC) orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded Duration: 25.201828747s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [..........................X.................X...............] duration: 18.716996318s FAIL (2025-06-22 02:10:19 UTC) orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded Duration: 18.717022364s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [...........X................................................] duration: 17.31391732s FAIL (2025-06-22 02:10:17 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== oad=64949]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jun 22 01:32:13.674692303 (81h8m49.238015311s) [upload=64949]: Done uploading segments... Jun 22 01:32:13.674727219 (81h8m49.238050233s) [upload=64949]: Upload finished. err: <nil> Jun 22 01:32:47.266407995 (81h9m22.829731002s) [upload=64958]: Starting upload Jun 22 01:32:47.266524767 (81h9m22.829847771s) [upload=64958]: Done starting upload Jun 22 01:32:47.266813109 (81h9m22.830136116s) [upload=64958]: Uploading segments... Jun 22 01:32:47.266969526 (81h9m22.830292526s) [upload=64958]: Got next segment. Inline: false Jun 22 01:32:47.267075475 (81h9m22.830398475s) [upload=64958]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 22 01:32:47.267081679 (81h9m22.830404667s) [upload=64958]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 22 01:32:47.301770382 (81h9m22.865093389s) [seg_pos={0 0}, upload=64958]: Begin upload segment... Jun 22 01:32:47.301823796 (81h9m22.865146799s) [seg_pos={0 0}, upload=64958]: Done begin upload segment. Jun 22 01:32:47.301855055 (81h9m22.865178087s) [noise=true , address=104.36.82.43:6841 , piece_id=PIHZMSTO3CII7KGDLGSAILTDRSLMEBVOLBCRCZJULMCFISXIZWXA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.301902600 (81h9m22.865225606s) [noise=true , address=104.36.82.42:6811 , piece_id=A5BFEHZ3WGGK53IZ7MKI6N4CVXHOORF6L47NR3UFDCAX7LBHRHVQ, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.302234027 (81h9m22.865557029s) [noise=true , address=104.36.82.51:4084 , piece_id=D43FA7BFE54KPCWGDGDME2EGSWM7PONOVPD3DN2XY42MJFOKVDIQ, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.302496354 (81h9m22.865819363s) [noise=true , address=79.127.219.38:4023 , piece_id=BWLRCXXHBMBTNCZEVKKIIS5PB35PDQXQXCLIQB7GBLEIOWWUFBFA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.302577370 (81h9m22.865900378s) [noise=true , address=104.36.82.52:4089 , piece_id=JSOEKLNWA3W3LOVKI3EGGKJYJ6J3UZY6KGABKQPQPXTIOOPMKVHA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.302917829 (81h9m22.866240835s) [noise=true , address=104.36.82.53:4122 , piece_id=HZZQBIKHQWLUE5EHK4KSKJGIUEG7D44RQASUJHCYXX6BICCGX6EA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.303087165 (81h9m22.866410168s) [noise=true , address=77.90.31.14:4689 , piece_id=Q3HDS3BCMYDTODFRMU4XFRDCQK2QZE3ZMXVLUX4M54PDEBS7RR6A, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.303250714 (81h9m22.866573717s) [noise=true , address=79.127.205.234:5601 , piece_id=V7PMXBP2VFWBSZDG4GTLBD435T6F5DT7A43L4GQLQ5BDDTJ4WY6A, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.303579249 (81h9m22.866902255s) [noise=true , address=104.36.82.12:6158 , piece_id=LJZEOJVJCJAR2BRILIJI3ZVDGYMPBBTP2G3HCTN3MHNS4KYDPASQ, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.303599776 (81h9m22.866922784s) [noise=true , address=104.36.82.9:6023 , piece_id=SISFAUU4RPF57OLGWM2BWQWE6GRUSCMORBYHWEXETNAVM2EEMO6Q, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.304005346 (81h9m22.867328346s) [noise=true , address=45.91.116.12:5242 , piece_id=6ABT5IPYGSLFFJMBYMBYRU4EYIXABUAUHG3OVILOHBIUDS3BP27A, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.304388850 (81h9m22.867711851s) [noise=true , address=64.49.13.137:4500 , piece_id=FAE2RXD57IRP5FXA2XR6THXFW27U7IWAZ4JDV5M2WRPXKUGZ4VIA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.304881529 (81h9m22.868204536s) [noise=true , address=79.127.220.101:6048 , piece_id=AN7UQUXB42T2YXEH7ZJRZUDIW47M3RWUEOCL2VEBOGH74BZF3YKA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.305249860 (81h9m22.86857287s) [noise=true , address=104.36.82.54:4165 , piece_id=4CVOABPCWC7J7SRHZB6IN2WJJ6KGWUQ66ELUIZIFA67ETLQ7JEUQ, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.305604108 (81h9m22.868927112s) [noise=true , address=104.36.82.15:6289 , piece_id=GWL3P55BIK4IJEL5YFZKI7JCA4LADYOOHUSIQUNGJEKBJOTKMP5A, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.305930423 (81h9m22.869253425s) [noise=true , address=64.49.13.139:4476 , piece_id=P7UKBNNZGI52B3I6WKTV6MVC262FYNKYYPJUI5G47UO5S4MU7PWA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.306355311 (81h9m22.869678319s) [noise=true , address=104.36.82.55:4213 , piece_id=X4IHS2XSPH52LIMNUYT7KGWMAPUH3TEMJOEEVB3JV5N45SEOH7YA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.306788243 (81h9m22.870111252s) [noise=true , address=104.36.82.22:6539 , piece_id=GHQ2MVSTDGVYLUKLV23D6YWRR4PP7UUYHCUEQ4WWSWEMTBFO3P4A, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.392174770 (81h9m22.955497773s) [upload=64958]: Next returned nil segment Jun 22 01:32:47.392195767 (81h9m22.955518775s) [upload=64958]: Waiting for error group managing segments... Jun 22 01:32:47.480767633 (81h9m23.044090623s) [noise=true , address=104.36.82.22:6539 , piece_id=GHQ2MVSTDGVYLUKLV23D6YWRR4PP7UUYHCUEQ4WWSWEMTBFO3P4A, seg_pos={0 0}, upload=64958]: Done uploading piece. err: ecclient: upload failed (node:1QL6w8DSFR6CxDdnJmDuyVZxp6JU55VKiJofBZRfQi7q31i8Jc, address:104.36.82.22:6539): protocol: storage node overloaded, request limit: 2000 Jun 22 01:32:47.585663142 (81h9m23.148986155s) [noise=true , address=79.127.220.101:6048 , piece_id=AN7UQUXB42T2YXEH7ZJRZUDIW47M3RWUEOCL2VEBOGH74BZF3YKA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:47.640263002 (81h9m23.203586013s) [noise=true , address=45.91.116.12:5242 , piece_id=6ABT5IPYGSLFFJMBYMBYRU4EYIXABUAUHG3OVILOHBIUDS3BP27A, seg_pos={0 0}, upload=64958]: Done uploading piece. err: ecclient: upload failed (node:12JsFL62jgXmza8Md8eQ9k8cyw5V9Vz2RzN4y85ZvgHZjwrq4vQ, address:45.91.116.12:5242): protocol: storage node overloaded, request limit: 2000 Jun 22 01:32:47.984127275 (81h9m23.54745028s) [noise=true , address=64.49.13.137:4500 , piece_id=FAE2RXD57IRP5FXA2XR6THXFW27U7IWAZ4JDV5M2WRPXKUGZ4VIA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:48.100255249 (81h9m23.663578241s) [noise=true , address=77.90.31.14:4689 , piece_id=Q3HDS3BCMYDTODFRMU4XFRDCQK2QZE3ZMXVLUX4M54PDEBS7RR6A, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:48.221764035 (81h9m23.785087034s) [noise=true , address=104.36.82.54:4165 , piece_id=4CVOABPCWC7J7SRHZB6IN2WJJ6KGWUQ66ELUIZIFA67ETLQ7JEUQ, seg_pos={0 0}, upload=64958]: Done uploading piece. err: ecclient: upload failed (node:12wGUMXVEubEtki4qGnjGEsbYFB1iUauZcu3oDRKrCQsJMV3bV2, address:104.36.82.54:4165): protocol: storage node overloaded, request limit: 2000 Jun 22 01:32:48.509996527 (81h9m24.073319547s) [noise=true , address=104.36.82.9:6023 , piece_id=SISFAUU4RPF57OLGWM2BWQWE6GRUSCMORBYHWEXETNAVM2EEMO6Q, seg_pos={0 0}, upload=64958]: Done uploading piece. err: ecclient: upload failed (node:1j2S9tgWpLwmyvXZSq2qKKcbaC8r7LkzfDCaMxCejAni4CST1g, address:104.36.82.9:6023): protocol: storage node overloaded, request limit: 2000 Jun 22 01:32:48.510513169 (81h9m24.073836177s) [noise=true , address=104.36.82.42:6811 , piece_id=A5BFEHZ3WGGK53IZ7MKI6N4CVXHOORF6L47NR3UFDCAX7LBHRHVQ, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:48.565245605 (81h9m24.128568606s) [noise=true , address=79.127.205.234:5601 , piece_id=V7PMXBP2VFWBSZDG4GTLBD435T6F5DT7A43L4GQLQ5BDDTJ4WY6A, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:49.979979824 (81h9m25.543302825s) [noise=true , address=104.36.82.15:6289 , piece_id=GWL3P55BIK4IJEL5YFZKI7JCA4LADYOOHUSIQUNGJEKBJOTKMP5A, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:50.463750280 (81h9m26.02707328s) [noise=true , address=104.36.82.51:4084 , piece_id=D43FA7BFE54KPCWGDGDME2EGSWM7PONOVPD3DN2XY42MJFOKVDIQ, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:51.480599795 (81h9m27.043922798s) [noise=true , address=79.127.219.38:4023 , piece_id=BWLRCXXHBMBTNCZEVKKIIS5PB35PDQXQXCLIQB7GBLEIOWWUFBFA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:52.941200625 (81h9m28.504523627s) [noise=true , address=104.36.82.43:6841 , piece_id=PIHZMSTO3CII7KGDLGSAILTDRSLMEBVOLBCRCZJULMCFISXIZWXA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:58.281803598 (81h9m33.845126612s) [noise=true , address=64.49.13.139:4476 , piece_id=P7UKBNNZGI52B3I6WKTV6MVC262FYNKYYPJUI5G47UO5S4MU7PWA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:33:00.277271877 (81h9m35.840594867s) [noise=true , address=104.36.82.55:4213 , piece_id=X4IHS2XSPH52LIMNUYT7KGWMAPUH3TEMJOEEVB3JV5N45SEOH7YA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:33:09.435339213 (81h9m44.998662206s) [noise=true , address=104.36.82.52:4089 , piece_id=JSOEKLNWA3W3LOVKI3EGGKJYJ6J3UZY6KGABKQPQPXTIOOPMKVHA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:33:31.472868445 (81h10m7.03619143s) [noise=true , address=104.36.82.53:4122 , piece_id=HZZQBIKHQWLUE5EHK4KSKJGIUEG7D44RQASUJHCYXX6BICCGX6EA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:33:47.235404653 (81h10m22.798727642s) [noise=true , address=104.36.82.12:6158 , piece_id=LJZEOJVJCJAR2BRILIJI3ZVDGYMPBBTP2G3HCTN3MHNS4KYDPASQ, seg_pos={0 0}, upload=64958]: Done uploading piece. err: ecclient: upload failed (node:1SjM6Utm2YizeUbrBUD9Z7etE1nLadW5Fa1zYP7MioPeeqhdJ7, address:104.36.82.12:6158): protocol: expected piece hash; context deadline exceeded; EOF Jun 22 01:33:47.235448297 (81h10m22.798771306s) [seg_pos={0 0}, upload=64958]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jun 22 01:33:47.265442393 (81h10m22.828765401s) [upload=64958]: Done uploading segments... Jun 22 01:33:47.265476139 (81h10m22.82879914s) [upload=64958]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration:* 17.313967932s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [............X...............................................] duration: 13.795394827s FAIL (2025-06-22 02:09:11 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== oad=64949]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jun 22 01:32:13.674692303 (81h8m49.238015311s) [upload=64949]: Done uploading segments... Jun 22 01:32:13.674727219 (81h8m49.238050233s) [upload=64949]: Upload finished. err: <nil> Jun 22 01:32:47.266407995 (81h9m22.829731002s) [upload=64958]: Starting upload Jun 22 01:32:47.266524767 (81h9m22.829847771s) [upload=64958]: Done starting upload Jun 22 01:32:47.266813109 (81h9m22.830136116s) [upload=64958]: Uploading segments... Jun 22 01:32:47.266969526 (81h9m22.830292526s) [upload=64958]: Got next segment. Inline: false Jun 22 01:32:47.267075475 (81h9m22.830398475s) [upload=64958]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 22 01:32:47.267081679 (81h9m22.830404667s) [upload=64958]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 22 01:32:47.301770382 (81h9m22.865093389s) [seg_pos={0 0}, upload=64958]: Begin upload segment... Jun 22 01:32:47.301823796 (81h9m22.865146799s) [seg_pos={0 0}, upload=64958]: Done begin upload segment. Jun 22 01:32:47.301855055 (81h9m22.865178087s) [noise=true , address=104.36.82.43:6841 , piece_id=PIHZMSTO3CII7KGDLGSAILTDRSLMEBVOLBCRCZJULMCFISXIZWXA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.301902600 (81h9m22.865225606s) [noise=true , address=104.36.82.42:6811 , piece_id=A5BFEHZ3WGGK53IZ7MKI6N4CVXHOORF6L47NR3UFDCAX7LBHRHVQ, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.302234027 (81h9m22.865557029s) [noise=true , address=104.36.82.51:4084 , piece_id=D43FA7BFE54KPCWGDGDME2EGSWM7PONOVPD3DN2XY42MJFOKVDIQ, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.302496354 (81h9m22.865819363s) [noise=true , address=79.127.219.38:4023 , piece_id=BWLRCXXHBMBTNCZEVKKIIS5PB35PDQXQXCLIQB7GBLEIOWWUFBFA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.302577370 (81h9m22.865900378s) [noise=true , address=104.36.82.52:4089 , piece_id=JSOEKLNWA3W3LOVKI3EGGKJYJ6J3UZY6KGABKQPQPXTIOOPMKVHA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.302917829 (81h9m22.866240835s) [noise=true , address=104.36.82.53:4122 , piece_id=HZZQBIKHQWLUE5EHK4KSKJGIUEG7D44RQASUJHCYXX6BICCGX6EA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.303087165 (81h9m22.866410168s) [noise=true , address=77.90.31.14:4689 , piece_id=Q3HDS3BCMYDTODFRMU4XFRDCQK2QZE3ZMXVLUX4M54PDEBS7RR6A, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.303250714 (81h9m22.866573717s) [noise=true , address=79.127.205.234:5601 , piece_id=V7PMXBP2VFWBSZDG4GTLBD435T6F5DT7A43L4GQLQ5BDDTJ4WY6A, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.303579249 (81h9m22.866902255s) [noise=true , address=104.36.82.12:6158 , piece_id=LJZEOJVJCJAR2BRILIJI3ZVDGYMPBBTP2G3HCTN3MHNS4KYDPASQ, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.303599776 (81h9m22.866922784s) [noise=true , address=104.36.82.9:6023 , piece_id=SISFAUU4RPF57OLGWM2BWQWE6GRUSCMORBYHWEXETNAVM2EEMO6Q, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.304005346 (81h9m22.867328346s) [noise=true , address=45.91.116.12:5242 , piece_id=6ABT5IPYGSLFFJMBYMBYRU4EYIXABUAUHG3OVILOHBIUDS3BP27A, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.304388850 (81h9m22.867711851s) [noise=true , address=64.49.13.137:4500 , piece_id=FAE2RXD57IRP5FXA2XR6THXFW27U7IWAZ4JDV5M2WRPXKUGZ4VIA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.304881529 (81h9m22.868204536s) [noise=true , address=79.127.220.101:6048 , piece_id=AN7UQUXB42T2YXEH7ZJRZUDIW47M3RWUEOCL2VEBOGH74BZF3YKA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.305249860 (81h9m22.86857287s) [noise=true , address=104.36.82.54:4165 , piece_id=4CVOABPCWC7J7SRHZB6IN2WJJ6KGWUQ66ELUIZIFA67ETLQ7JEUQ, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.305604108 (81h9m22.868927112s) [noise=true , address=104.36.82.15:6289 , piece_id=GWL3P55BIK4IJEL5YFZKI7JCA4LADYOOHUSIQUNGJEKBJOTKMP5A, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.305930423 (81h9m22.869253425s) [noise=true , address=64.49.13.139:4476 , piece_id=P7UKBNNZGI52B3I6WKTV6MVC262FYNKYYPJUI5G47UO5S4MU7PWA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.306355311 (81h9m22.869678319s) [noise=true , address=104.36.82.55:4213 , piece_id=X4IHS2XSPH52LIMNUYT7KGWMAPUH3TEMJOEEVB3JV5N45SEOH7YA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.306788243 (81h9m22.870111252s) [noise=true , address=104.36.82.22:6539 , piece_id=GHQ2MVSTDGVYLUKLV23D6YWRR4PP7UUYHCUEQ4WWSWEMTBFO3P4A, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.392174770 (81h9m22.955497773s) [upload=64958]: Next returned nil segment Jun 22 01:32:47.392195767 (81h9m22.955518775s) [upload=64958]: Waiting for error group managing segments... Jun 22 01:32:47.480767633 (81h9m23.044090623s) [noise=true , address=104.36.82.22:6539 , piece_id=GHQ2MVSTDGVYLUKLV23D6YWRR4PP7UUYHCUEQ4WWSWEMTBFO3P4A, seg_pos={0 0}, upload=64958]: Done uploading piece. err: ecclient: upload failed (node:1QL6w8DSFR6CxDdnJmDuyVZxp6JU55VKiJofBZRfQi7q31i8Jc, address:104.36.82.22:6539): protocol: storage node overloaded, request limit: 2000 Jun 22 01:32:47.585663142 (81h9m23.148986155s) [noise=true , address=79.127.220.101:6048 , piece_id=AN7UQUXB42T2YXEH7ZJRZUDIW47M3RWUEOCL2VEBOGH74BZF3YKA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:47.640263002 (81h9m23.203586013s) [noise=true , address=45.91.116.12:5242 , piece_id=6ABT5IPYGSLFFJMBYMBYRU4EYIXABUAUHG3OVILOHBIUDS3BP27A, seg_pos={0 0}, upload=64958]: Done uploading piece. err: ecclient: upload failed (node:12JsFL62jgXmza8Md8eQ9k8cyw5V9Vz2RzN4y85ZvgHZjwrq4vQ, address:45.91.116.12:5242): protocol: storage node overloaded, request limit: 2000 Jun 22 01:32:47.984127275 (81h9m23.54745028s) [noise=true , address=64.49.13.137:4500 , piece_id=FAE2RXD57IRP5FXA2XR6THXFW27U7IWAZ4JDV5M2WRPXKUGZ4VIA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:48.100255249 (81h9m23.663578241s) [noise=true , address=77.90.31.14:4689 , piece_id=Q3HDS3BCMYDTODFRMU4XFRDCQK2QZE3ZMXVLUX4M54PDEBS7RR6A, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:48.221764035 (81h9m23.785087034s) [noise=true , address=104.36.82.54:4165 , piece_id=4CVOABPCWC7J7SRHZB6IN2WJJ6KGWUQ66ELUIZIFA67ETLQ7JEUQ, seg_pos={0 0}, upload=64958]: Done uploading piece. err: ecclient: upload failed (node:12wGUMXVEubEtki4qGnjGEsbYFB1iUauZcu3oDRKrCQsJMV3bV2, address:104.36.82.54:4165): protocol: storage node overloaded, request limit: 2000 Jun 22 01:32:48.509996527 (81h9m24.073319547s) [noise=true , address=104.36.82.9:6023 , piece_id=SISFAUU4RPF57OLGWM2BWQWE6GRUSCMORBYHWEXETNAVM2EEMO6Q, seg_pos={0 0}, upload=64958]: Done uploading piece. err: ecclient: upload failed (node:1j2S9tgWpLwmyvXZSq2qKKcbaC8r7LkzfDCaMxCejAni4CST1g, address:104.36.82.9:6023): protocol: storage node overloaded, request limit: 2000 Jun 22 01:32:48.510513169 (81h9m24.073836177s) [noise=true , address=104.36.82.42:6811 , piece_id=A5BFEHZ3WGGK53IZ7MKI6N4CVXHOORF6L47NR3UFDCAX7LBHRHVQ, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:48.565245605 (81h9m24.128568606s) [noise=true , address=79.127.205.234:5601 , piece_id=V7PMXBP2VFWBSZDG4GTLBD435T6F5DT7A43L4GQLQ5BDDTJ4WY6A, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:49.979979824 (81h9m25.543302825s) [noise=true , address=104.36.82.15:6289 , piece_id=GWL3P55BIK4IJEL5YFZKI7JCA4LADYOOHUSIQUNGJEKBJOTKMP5A, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:50.463750280 (81h9m26.02707328s) [noise=true , address=104.36.82.51:4084 , piece_id=D43FA7BFE54KPCWGDGDME2EGSWM7PONOVPD3DN2XY42MJFOKVDIQ, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:51.480599795 (81h9m27.043922798s) [noise=true , address=79.127.219.38:4023 , piece_id=BWLRCXXHBMBTNCZEVKKIIS5PB35PDQXQXCLIQB7GBLEIOWWUFBFA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:52.941200625 (81h9m28.504523627s) [noise=true , address=104.36.82.43:6841 , piece_id=PIHZMSTO3CII7KGDLGSAILTDRSLMEBVOLBCRCZJULMCFISXIZWXA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:58.281803598 (81h9m33.845126612s) [noise=true , address=64.49.13.139:4476 , piece_id=P7UKBNNZGI52B3I6WKTV6MVC262FYNKYYPJUI5G47UO5S4MU7PWA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:33:00.277271877 (81h9m35.840594867s) [noise=true , address=104.36.82.55:4213 , piece_id=X4IHS2XSPH52LIMNUYT7KGWMAPUH3TEMJOEEVB3JV5N45SEOH7YA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:33:09.435339213 (81h9m44.998662206s) [noise=true , address=104.36.82.52:4089 , piece_id=JSOEKLNWA3W3LOVKI3EGGKJYJ6J3UZY6KGABKQPQPXTIOOPMKVHA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:33:31.472868445 (81h10m7.03619143s) [noise=true , address=104.36.82.53:4122 , piece_id=HZZQBIKHQWLUE5EHK4KSKJGIUEG7D44RQASUJHCYXX6BICCGX6EA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:33:47.235404653 (81h10m22.798727642s) [noise=true , address=104.36.82.12:6158 , piece_id=LJZEOJVJCJAR2BRILIJI3ZVDGYMPBBTP2G3HCTN3MHNS4KYDPASQ, seg_pos={0 0}, upload=64958]: Done uploading piece. err: ecclient: upload failed (node:1SjM6Utm2YizeUbrBUD9Z7etE1nLadW5Fa1zYP7MioPeeqhdJ7, address:104.36.82.12:6158): protocol: expected piece hash; context deadline exceeded; EOF Jun 22 01:33:47.235448297 (81h10m22.798771306s) [seg_pos={0 0}, upload=64958]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jun 22 01:33:47.265442393 (81h10m22.828765401s) [upload=64958]: Done uploading segments... Jun 22 01:33:47.265476139 (81h10m22.82879914s) [upload=64958]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration:* 13.795423025s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [...........................X.................X..............] duration: 15.291732299s FAIL (2025-06-22 02:09:06 UTC) orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded Duration: 15.291752697s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [............................X.................X.............] duration: 17.771609827s FAIL (2025-06-22 02:08:50 UTC) orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded Duration: 17.771645895s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.............X..............................................] duration: 9.490154932s FAIL (2025-06-22 02:08:38 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== oad=64949]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jun 22 01:32:13.674692303 (81h8m49.238015311s) [upload=64949]: Done uploading segments... Jun 22 01:32:13.674727219 (81h8m49.238050233s) [upload=64949]: Upload finished. err: <nil> Jun 22 01:32:47.266407995 (81h9m22.829731002s) [upload=64958]: Starting upload Jun 22 01:32:47.266524767 (81h9m22.829847771s) [upload=64958]: Done starting upload Jun 22 01:32:47.266813109 (81h9m22.830136116s) [upload=64958]: Uploading segments... Jun 22 01:32:47.266969526 (81h9m22.830292526s) [upload=64958]: Got next segment. Inline: false Jun 22 01:32:47.267075475 (81h9m22.830398475s) [upload=64958]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 22 01:32:47.267081679 (81h9m22.830404667s) [upload=64958]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 22 01:32:47.301770382 (81h9m22.865093389s) [seg_pos={0 0}, upload=64958]: Begin upload segment... Jun 22 01:32:47.301823796 (81h9m22.865146799s) [seg_pos={0 0}, upload=64958]: Done begin upload segment. Jun 22 01:32:47.301855055 (81h9m22.865178087s) [noise=true , address=104.36.82.43:6841 , piece_id=PIHZMSTO3CII7KGDLGSAILTDRSLMEBVOLBCRCZJULMCFISXIZWXA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.301902600 (81h9m22.865225606s) [noise=true , address=104.36.82.42:6811 , piece_id=A5BFEHZ3WGGK53IZ7MKI6N4CVXHOORF6L47NR3UFDCAX7LBHRHVQ, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.302234027 (81h9m22.865557029s) [noise=true , address=104.36.82.51:4084 , piece_id=D43FA7BFE54KPCWGDGDME2EGSWM7PONOVPD3DN2XY42MJFOKVDIQ, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.302496354 (81h9m22.865819363s) [noise=true , address=79.127.219.38:4023 , piece_id=BWLRCXXHBMBTNCZEVKKIIS5PB35PDQXQXCLIQB7GBLEIOWWUFBFA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.302577370 (81h9m22.865900378s) [noise=true , address=104.36.82.52:4089 , piece_id=JSOEKLNWA3W3LOVKI3EGGKJYJ6J3UZY6KGABKQPQPXTIOOPMKVHA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.302917829 (81h9m22.866240835s) [noise=true , address=104.36.82.53:4122 , piece_id=HZZQBIKHQWLUE5EHK4KSKJGIUEG7D44RQASUJHCYXX6BICCGX6EA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.303087165 (81h9m22.866410168s) [noise=true , address=77.90.31.14:4689 , piece_id=Q3HDS3BCMYDTODFRMU4XFRDCQK2QZE3ZMXVLUX4M54PDEBS7RR6A, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.303250714 (81h9m22.866573717s) [noise=true , address=79.127.205.234:5601 , piece_id=V7PMXBP2VFWBSZDG4GTLBD435T6F5DT7A43L4GQLQ5BDDTJ4WY6A, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.303579249 (81h9m22.866902255s) [noise=true , address=104.36.82.12:6158 , piece_id=LJZEOJVJCJAR2BRILIJI3ZVDGYMPBBTP2G3HCTN3MHNS4KYDPASQ, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.303599776 (81h9m22.866922784s) [noise=true , address=104.36.82.9:6023 , piece_id=SISFAUU4RPF57OLGWM2BWQWE6GRUSCMORBYHWEXETNAVM2EEMO6Q, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.304005346 (81h9m22.867328346s) [noise=true , address=45.91.116.12:5242 , piece_id=6ABT5IPYGSLFFJMBYMBYRU4EYIXABUAUHG3OVILOHBIUDS3BP27A, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.304388850 (81h9m22.867711851s) [noise=true , address=64.49.13.137:4500 , piece_id=FAE2RXD57IRP5FXA2XR6THXFW27U7IWAZ4JDV5M2WRPXKUGZ4VIA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.304881529 (81h9m22.868204536s) [noise=true , address=79.127.220.101:6048 , piece_id=AN7UQUXB42T2YXEH7ZJRZUDIW47M3RWUEOCL2VEBOGH74BZF3YKA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.305249860 (81h9m22.86857287s) [noise=true , address=104.36.82.54:4165 , piece_id=4CVOABPCWC7J7SRHZB6IN2WJJ6KGWUQ66ELUIZIFA67ETLQ7JEUQ, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.305604108 (81h9m22.868927112s) [noise=true , address=104.36.82.15:6289 , piece_id=GWL3P55BIK4IJEL5YFZKI7JCA4LADYOOHUSIQUNGJEKBJOTKMP5A, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.305930423 (81h9m22.869253425s) [noise=true , address=64.49.13.139:4476 , piece_id=P7UKBNNZGI52B3I6WKTV6MVC262FYNKYYPJUI5G47UO5S4MU7PWA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.306355311 (81h9m22.869678319s) [noise=true , address=104.36.82.55:4213 , piece_id=X4IHS2XSPH52LIMNUYT7KGWMAPUH3TEMJOEEVB3JV5N45SEOH7YA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.306788243 (81h9m22.870111252s) [noise=true , address=104.36.82.22:6539 , piece_id=GHQ2MVSTDGVYLUKLV23D6YWRR4PP7UUYHCUEQ4WWSWEMTBFO3P4A, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.392174770 (81h9m22.955497773s) [upload=64958]: Next returned nil segment Jun 22 01:32:47.392195767 (81h9m22.955518775s) [upload=64958]: Waiting for error group managing segments... Jun 22 01:32:47.480767633 (81h9m23.044090623s) [noise=true , address=104.36.82.22:6539 , piece_id=GHQ2MVSTDGVYLUKLV23D6YWRR4PP7UUYHCUEQ4WWSWEMTBFO3P4A, seg_pos={0 0}, upload=64958]: Done uploading piece. err: ecclient: upload failed (node:1QL6w8DSFR6CxDdnJmDuyVZxp6JU55VKiJofBZRfQi7q31i8Jc, address:104.36.82.22:6539): protocol: storage node overloaded, request limit: 2000 Jun 22 01:32:47.585663142 (81h9m23.148986155s) [noise=true , address=79.127.220.101:6048 , piece_id=AN7UQUXB42T2YXEH7ZJRZUDIW47M3RWUEOCL2VEBOGH74BZF3YKA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:47.640263002 (81h9m23.203586013s) [noise=true , address=45.91.116.12:5242 , piece_id=6ABT5IPYGSLFFJMBYMBYRU4EYIXABUAUHG3OVILOHBIUDS3BP27A, seg_pos={0 0}, upload=64958]: Done uploading piece. err: ecclient: upload failed (node:12JsFL62jgXmza8Md8eQ9k8cyw5V9Vz2RzN4y85ZvgHZjwrq4vQ, address:45.91.116.12:5242): protocol: storage node overloaded, request limit: 2000 Jun 22 01:32:47.984127275 (81h9m23.54745028s) [noise=true , address=64.49.13.137:4500 , piece_id=FAE2RXD57IRP5FXA2XR6THXFW27U7IWAZ4JDV5M2WRPXKUGZ4VIA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:48.100255249 (81h9m23.663578241s) [noise=true , address=77.90.31.14:4689 , piece_id=Q3HDS3BCMYDTODFRMU4XFRDCQK2QZE3ZMXVLUX4M54PDEBS7RR6A, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:48.221764035 (81h9m23.785087034s) [noise=true , address=104.36.82.54:4165 , piece_id=4CVOABPCWC7J7SRHZB6IN2WJJ6KGWUQ66ELUIZIFA67ETLQ7JEUQ, seg_pos={0 0}, upload=64958]: Done uploading piece. err: ecclient: upload failed (node:12wGUMXVEubEtki4qGnjGEsbYFB1iUauZcu3oDRKrCQsJMV3bV2, address:104.36.82.54:4165): protocol: storage node overloaded, request limit: 2000 Jun 22 01:32:48.509996527 (81h9m24.073319547s) [noise=true , address=104.36.82.9:6023 , piece_id=SISFAUU4RPF57OLGWM2BWQWE6GRUSCMORBYHWEXETNAVM2EEMO6Q, seg_pos={0 0}, upload=64958]: Done uploading piece. err: ecclient: upload failed (node:1j2S9tgWpLwmyvXZSq2qKKcbaC8r7LkzfDCaMxCejAni4CST1g, address:104.36.82.9:6023): protocol: storage node overloaded, request limit: 2000 Jun 22 01:32:48.510513169 (81h9m24.073836177s) [noise=true , address=104.36.82.42:6811 , piece_id=A5BFEHZ3WGGK53IZ7MKI6N4CVXHOORF6L47NR3UFDCAX7LBHRHVQ, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:48.565245605 (81h9m24.128568606s) [noise=true , address=79.127.205.234:5601 , piece_id=V7PMXBP2VFWBSZDG4GTLBD435T6F5DT7A43L4GQLQ5BDDTJ4WY6A, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:49.979979824 (81h9m25.543302825s) [noise=true , address=104.36.82.15:6289 , piece_id=GWL3P55BIK4IJEL5YFZKI7JCA4LADYOOHUSIQUNGJEKBJOTKMP5A, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:50.463750280 (81h9m26.02707328s) [noise=true , address=104.36.82.51:4084 , piece_id=D43FA7BFE54KPCWGDGDME2EGSWM7PONOVPD3DN2XY42MJFOKVDIQ, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:51.480599795 (81h9m27.043922798s) [noise=true , address=79.127.219.38:4023 , piece_id=BWLRCXXHBMBTNCZEVKKIIS5PB35PDQXQXCLIQB7GBLEIOWWUFBFA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:52.941200625 (81h9m28.504523627s) [noise=true , address=104.36.82.43:6841 , piece_id=PIHZMSTO3CII7KGDLGSAILTDRSLMEBVOLBCRCZJULMCFISXIZWXA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:58.281803598 (81h9m33.845126612s) [noise=true , address=64.49.13.139:4476 , piece_id=P7UKBNNZGI52B3I6WKTV6MVC262FYNKYYPJUI5G47UO5S4MU7PWA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:33:00.277271877 (81h9m35.840594867s) [noise=true , address=104.36.82.55:4213 , piece_id=X4IHS2XSPH52LIMNUYT7KGWMAPUH3TEMJOEEVB3JV5N45SEOH7YA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:33:09.435339213 (81h9m44.998662206s) [noise=true , address=104.36.82.52:4089 , piece_id=JSOEKLNWA3W3LOVKI3EGGKJYJ6J3UZY6KGABKQPQPXTIOOPMKVHA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:33:31.472868445 (81h10m7.03619143s) [noise=true , address=104.36.82.53:4122 , piece_id=HZZQBIKHQWLUE5EHK4KSKJGIUEG7D44RQASUJHCYXX6BICCGX6EA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:33:47.235404653 (81h10m22.798727642s) [noise=true , address=104.36.82.12:6158 , piece_id=LJZEOJVJCJAR2BRILIJI3ZVDGYMPBBTP2G3HCTN3MHNS4KYDPASQ, seg_pos={0 0}, upload=64958]: Done uploading piece. err: ecclient: upload failed (node:1SjM6Utm2YizeUbrBUD9Z7etE1nLadW5Fa1zYP7MioPeeqhdJ7, address:104.36.82.12:6158): protocol: expected piece hash; context deadline exceeded; EOF Jun 22 01:33:47.235448297 (81h10m22.798771306s) [seg_pos={0 0}, upload=64958]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jun 22 01:33:47.265442393 (81h10m22.828765401s) [upload=64958]: Done uploading segments... Jun 22 01:33:47.265476139 (81h10m22.82879914s) [upload=64958]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration:* 9.490174905s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [..............X.............................................] duration: 34.566834385s FAIL (2025-06-22 02:08:28 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== oad=64949]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jun 22 01:32:13.674692303 (81h8m49.238015311s) [upload=64949]: Done uploading segments... Jun 22 01:32:13.674727219 (81h8m49.238050233s) [upload=64949]: Upload finished. err: <nil> Jun 22 01:32:47.266407995 (81h9m22.829731002s) [upload=64958]: Starting upload Jun 22 01:32:47.266524767 (81h9m22.829847771s) [upload=64958]: Done starting upload Jun 22 01:32:47.266813109 (81h9m22.830136116s) [upload=64958]: Uploading segments... Jun 22 01:32:47.266969526 (81h9m22.830292526s) [upload=64958]: Got next segment. Inline: false Jun 22 01:32:47.267075475 (81h9m22.830398475s) [upload=64958]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 22 01:32:47.267081679 (81h9m22.830404667s) [upload=64958]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 22 01:32:47.301770382 (81h9m22.865093389s) [seg_pos={0 0}, upload=64958]: Begin upload segment... Jun 22 01:32:47.301823796 (81h9m22.865146799s) [seg_pos={0 0}, upload=64958]: Done begin upload segment. Jun 22 01:32:47.301855055 (81h9m22.865178087s) [noise=true , address=104.36.82.43:6841 , piece_id=PIHZMSTO3CII7KGDLGSAILTDRSLMEBVOLBCRCZJULMCFISXIZWXA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.301902600 (81h9m22.865225606s) [noise=true , address=104.36.82.42:6811 , piece_id=A5BFEHZ3WGGK53IZ7MKI6N4CVXHOORF6L47NR3UFDCAX7LBHRHVQ, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.302234027 (81h9m22.865557029s) [noise=true , address=104.36.82.51:4084 , piece_id=D43FA7BFE54KPCWGDGDME2EGSWM7PONOVPD3DN2XY42MJFOKVDIQ, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.302496354 (81h9m22.865819363s) [noise=true , address=79.127.219.38:4023 , piece_id=BWLRCXXHBMBTNCZEVKKIIS5PB35PDQXQXCLIQB7GBLEIOWWUFBFA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.302577370 (81h9m22.865900378s) [noise=true , address=104.36.82.52:4089 , piece_id=JSOEKLNWA3W3LOVKI3EGGKJYJ6J3UZY6KGABKQPQPXTIOOPMKVHA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.302917829 (81h9m22.866240835s) [noise=true , address=104.36.82.53:4122 , piece_id=HZZQBIKHQWLUE5EHK4KSKJGIUEG7D44RQASUJHCYXX6BICCGX6EA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.303087165 (81h9m22.866410168s) [noise=true , address=77.90.31.14:4689 , piece_id=Q3HDS3BCMYDTODFRMU4XFRDCQK2QZE3ZMXVLUX4M54PDEBS7RR6A, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.303250714 (81h9m22.866573717s) [noise=true , address=79.127.205.234:5601 , piece_id=V7PMXBP2VFWBSZDG4GTLBD435T6F5DT7A43L4GQLQ5BDDTJ4WY6A, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.303579249 (81h9m22.866902255s) [noise=true , address=104.36.82.12:6158 , piece_id=LJZEOJVJCJAR2BRILIJI3ZVDGYMPBBTP2G3HCTN3MHNS4KYDPASQ, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.303599776 (81h9m22.866922784s) [noise=true , address=104.36.82.9:6023 , piece_id=SISFAUU4RPF57OLGWM2BWQWE6GRUSCMORBYHWEXETNAVM2EEMO6Q, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.304005346 (81h9m22.867328346s) [noise=true , address=45.91.116.12:5242 , piece_id=6ABT5IPYGSLFFJMBYMBYRU4EYIXABUAUHG3OVILOHBIUDS3BP27A, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.304388850 (81h9m22.867711851s) [noise=true , address=64.49.13.137:4500 , piece_id=FAE2RXD57IRP5FXA2XR6THXFW27U7IWAZ4JDV5M2WRPXKUGZ4VIA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.304881529 (81h9m22.868204536s) [noise=true , address=79.127.220.101:6048 , piece_id=AN7UQUXB42T2YXEH7ZJRZUDIW47M3RWUEOCL2VEBOGH74BZF3YKA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.305249860 (81h9m22.86857287s) [noise=true , address=104.36.82.54:4165 , piece_id=4CVOABPCWC7J7SRHZB6IN2WJJ6KGWUQ66ELUIZIFA67ETLQ7JEUQ, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.305604108 (81h9m22.868927112s) [noise=true , address=104.36.82.15:6289 , piece_id=GWL3P55BIK4IJEL5YFZKI7JCA4LADYOOHUSIQUNGJEKBJOTKMP5A, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.305930423 (81h9m22.869253425s) [noise=true , address=64.49.13.139:4476 , piece_id=P7UKBNNZGI52B3I6WKTV6MVC262FYNKYYPJUI5G47UO5S4MU7PWA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.306355311 (81h9m22.869678319s) [noise=true , address=104.36.82.55:4213 , piece_id=X4IHS2XSPH52LIMNUYT7KGWMAPUH3TEMJOEEVB3JV5N45SEOH7YA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.306788243 (81h9m22.870111252s) [noise=true , address=104.36.82.22:6539 , piece_id=GHQ2MVSTDGVYLUKLV23D6YWRR4PP7UUYHCUEQ4WWSWEMTBFO3P4A, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.392174770 (81h9m22.955497773s) [upload=64958]: Next returned nil segment Jun 22 01:32:47.392195767 (81h9m22.955518775s) [upload=64958]: Waiting for error group managing segments... Jun 22 01:32:47.480767633 (81h9m23.044090623s) [noise=true , address=104.36.82.22:6539 , piece_id=GHQ2MVSTDGVYLUKLV23D6YWRR4PP7UUYHCUEQ4WWSWEMTBFO3P4A, seg_pos={0 0}, upload=64958]: Done uploading piece. err: ecclient: upload failed (node:1QL6w8DSFR6CxDdnJmDuyVZxp6JU55VKiJofBZRfQi7q31i8Jc, address:104.36.82.22:6539): protocol: storage node overloaded, request limit: 2000 Jun 22 01:32:47.585663142 (81h9m23.148986155s) [noise=true , address=79.127.220.101:6048 , piece_id=AN7UQUXB42T2YXEH7ZJRZUDIW47M3RWUEOCL2VEBOGH74BZF3YKA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:47.640263002 (81h9m23.203586013s) [noise=true , address=45.91.116.12:5242 , piece_id=6ABT5IPYGSLFFJMBYMBYRU4EYIXABUAUHG3OVILOHBIUDS3BP27A, seg_pos={0 0}, upload=64958]: Done uploading piece. err: ecclient: upload failed (node:12JsFL62jgXmza8Md8eQ9k8cyw5V9Vz2RzN4y85ZvgHZjwrq4vQ, address:45.91.116.12:5242): protocol: storage node overloaded, request limit: 2000 Jun 22 01:32:47.984127275 (81h9m23.54745028s) [noise=true , address=64.49.13.137:4500 , piece_id=FAE2RXD57IRP5FXA2XR6THXFW27U7IWAZ4JDV5M2WRPXKUGZ4VIA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:48.100255249 (81h9m23.663578241s) [noise=true , address=77.90.31.14:4689 , piece_id=Q3HDS3BCMYDTODFRMU4XFRDCQK2QZE3ZMXVLUX4M54PDEBS7RR6A, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:48.221764035 (81h9m23.785087034s) [noise=true , address=104.36.82.54:4165 , piece_id=4CVOABPCWC7J7SRHZB6IN2WJJ6KGWUQ66ELUIZIFA67ETLQ7JEUQ, seg_pos={0 0}, upload=64958]: Done uploading piece. err: ecclient: upload failed (node:12wGUMXVEubEtki4qGnjGEsbYFB1iUauZcu3oDRKrCQsJMV3bV2, address:104.36.82.54:4165): protocol: storage node overloaded, request limit: 2000 Jun 22 01:32:48.509996527 (81h9m24.073319547s) [noise=true , address=104.36.82.9:6023 , piece_id=SISFAUU4RPF57OLGWM2BWQWE6GRUSCMORBYHWEXETNAVM2EEMO6Q, seg_pos={0 0}, upload=64958]: Done uploading piece. err: ecclient: upload failed (node:1j2S9tgWpLwmyvXZSq2qKKcbaC8r7LkzfDCaMxCejAni4CST1g, address:104.36.82.9:6023): protocol: storage node overloaded, request limit: 2000 Jun 22 01:32:48.510513169 (81h9m24.073836177s) [noise=true , address=104.36.82.42:6811 , piece_id=A5BFEHZ3WGGK53IZ7MKI6N4CVXHOORF6L47NR3UFDCAX7LBHRHVQ, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:48.565245605 (81h9m24.128568606s) [noise=true , address=79.127.205.234:5601 , piece_id=V7PMXBP2VFWBSZDG4GTLBD435T6F5DT7A43L4GQLQ5BDDTJ4WY6A, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:49.979979824 (81h9m25.543302825s) [noise=true , address=104.36.82.15:6289 , piece_id=GWL3P55BIK4IJEL5YFZKI7JCA4LADYOOHUSIQUNGJEKBJOTKMP5A, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:50.463750280 (81h9m26.02707328s) [noise=true , address=104.36.82.51:4084 , piece_id=D43FA7BFE54KPCWGDGDME2EGSWM7PONOVPD3DN2XY42MJFOKVDIQ, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:51.480599795 (81h9m27.043922798s) [noise=true , address=79.127.219.38:4023 , piece_id=BWLRCXXHBMBTNCZEVKKIIS5PB35PDQXQXCLIQB7GBLEIOWWUFBFA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:52.941200625 (81h9m28.504523627s) [noise=true , address=104.36.82.43:6841 , piece_id=PIHZMSTO3CII7KGDLGSAILTDRSLMEBVOLBCRCZJULMCFISXIZWXA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:58.281803598 (81h9m33.845126612s) [noise=true , address=64.49.13.139:4476 , piece_id=P7UKBNNZGI52B3I6WKTV6MVC262FYNKYYPJUI5G47UO5S4MU7PWA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:33:00.277271877 (81h9m35.840594867s) [noise=true , address=104.36.82.55:4213 , piece_id=X4IHS2XSPH52LIMNUYT7KGWMAPUH3TEMJOEEVB3JV5N45SEOH7YA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:33:09.435339213 (81h9m44.998662206s) [noise=true , address=104.36.82.52:4089 , piece_id=JSOEKLNWA3W3LOVKI3EGGKJYJ6J3UZY6KGABKQPQPXTIOOPMKVHA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:33:31.472868445 (81h10m7.03619143s) [noise=true , address=104.36.82.53:4122 , piece_id=HZZQBIKHQWLUE5EHK4KSKJGIUEG7D44RQASUJHCYXX6BICCGX6EA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:33:47.235404653 (81h10m22.798727642s) [noise=true , address=104.36.82.12:6158 , piece_id=LJZEOJVJCJAR2BRILIJI3ZVDGYMPBBTP2G3HCTN3MHNS4KYDPASQ, seg_pos={0 0}, upload=64958]: Done uploading piece. err: ecclient: upload failed (node:1SjM6Utm2YizeUbrBUD9Z7etE1nLadW5Fa1zYP7MioPeeqhdJ7, address:104.36.82.12:6158): protocol: expected piece hash; context deadline exceeded; EOF Jun 22 01:33:47.235448297 (81h10m22.798771306s) [seg_pos={0 0}, upload=64958]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jun 22 01:33:47.265442393 (81h10m22.828765401s) [upload=64958]: Done uploading segments... Jun 22 01:33:47.265476139 (81h10m22.82879914s) [upload=64958]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration:* 34.566863804s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [.............................X.................X............] duration: 18.716559413s FAIL (2025-06-22 02:07:41 UTC) orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded Duration: 18.717046279s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [...............X............................................] duration: 10.653013138s FAIL (2025-06-22 02:07:33 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== oad=64949]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jun 22 01:32:13.674692303 (81h8m49.238015311s) [upload=64949]: Done uploading segments... Jun 22 01:32:13.674727219 (81h8m49.238050233s) [upload=64949]: Upload finished. err: <nil> Jun 22 01:32:47.266407995 (81h9m22.829731002s) [upload=64958]: Starting upload Jun 22 01:32:47.266524767 (81h9m22.829847771s) [upload=64958]: Done starting upload Jun 22 01:32:47.266813109 (81h9m22.830136116s) [upload=64958]: Uploading segments... Jun 22 01:32:47.266969526 (81h9m22.830292526s) [upload=64958]: Got next segment. Inline: false Jun 22 01:32:47.267075475 (81h9m22.830398475s) [upload=64958]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 22 01:32:47.267081679 (81h9m22.830404667s) [upload=64958]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 22 01:32:47.301770382 (81h9m22.865093389s) [seg_pos={0 0}, upload=64958]: Begin upload segment... Jun 22 01:32:47.301823796 (81h9m22.865146799s) [seg_pos={0 0}, upload=64958]: Done begin upload segment. Jun 22 01:32:47.301855055 (81h9m22.865178087s) [noise=true , address=104.36.82.43:6841 , piece_id=PIHZMSTO3CII7KGDLGSAILTDRSLMEBVOLBCRCZJULMCFISXIZWXA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.301902600 (81h9m22.865225606s) [noise=true , address=104.36.82.42:6811 , piece_id=A5BFEHZ3WGGK53IZ7MKI6N4CVXHOORF6L47NR3UFDCAX7LBHRHVQ, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.302234027 (81h9m22.865557029s) [noise=true , address=104.36.82.51:4084 , piece_id=D43FA7BFE54KPCWGDGDME2EGSWM7PONOVPD3DN2XY42MJFOKVDIQ, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.302496354 (81h9m22.865819363s) [noise=true , address=79.127.219.38:4023 , piece_id=BWLRCXXHBMBTNCZEVKKIIS5PB35PDQXQXCLIQB7GBLEIOWWUFBFA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.302577370 (81h9m22.865900378s) [noise=true , address=104.36.82.52:4089 , piece_id=JSOEKLNWA3W3LOVKI3EGGKJYJ6J3UZY6KGABKQPQPXTIOOPMKVHA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.302917829 (81h9m22.866240835s) [noise=true , address=104.36.82.53:4122 , piece_id=HZZQBIKHQWLUE5EHK4KSKJGIUEG7D44RQASUJHCYXX6BICCGX6EA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.303087165 (81h9m22.866410168s) [noise=true , address=77.90.31.14:4689 , piece_id=Q3HDS3BCMYDTODFRMU4XFRDCQK2QZE3ZMXVLUX4M54PDEBS7RR6A, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.303250714 (81h9m22.866573717s) [noise=true , address=79.127.205.234:5601 , piece_id=V7PMXBP2VFWBSZDG4GTLBD435T6F5DT7A43L4GQLQ5BDDTJ4WY6A, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.303579249 (81h9m22.866902255s) [noise=true , address=104.36.82.12:6158 , piece_id=LJZEOJVJCJAR2BRILIJI3ZVDGYMPBBTP2G3HCTN3MHNS4KYDPASQ, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.303599776 (81h9m22.866922784s) [noise=true , address=104.36.82.9:6023 , piece_id=SISFAUU4RPF57OLGWM2BWQWE6GRUSCMORBYHWEXETNAVM2EEMO6Q, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.304005346 (81h9m22.867328346s) [noise=true , address=45.91.116.12:5242 , piece_id=6ABT5IPYGSLFFJMBYMBYRU4EYIXABUAUHG3OVILOHBIUDS3BP27A, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.304388850 (81h9m22.867711851s) [noise=true , address=64.49.13.137:4500 , piece_id=FAE2RXD57IRP5FXA2XR6THXFW27U7IWAZ4JDV5M2WRPXKUGZ4VIA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.304881529 (81h9m22.868204536s) [noise=true , address=79.127.220.101:6048 , piece_id=AN7UQUXB42T2YXEH7ZJRZUDIW47M3RWUEOCL2VEBOGH74BZF3YKA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.305249860 (81h9m22.86857287s) [noise=true , address=104.36.82.54:4165 , piece_id=4CVOABPCWC7J7SRHZB6IN2WJJ6KGWUQ66ELUIZIFA67ETLQ7JEUQ, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.305604108 (81h9m22.868927112s) [noise=true , address=104.36.82.15:6289 , piece_id=GWL3P55BIK4IJEL5YFZKI7JCA4LADYOOHUSIQUNGJEKBJOTKMP5A, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.305930423 (81h9m22.869253425s) [noise=true , address=64.49.13.139:4476 , piece_id=P7UKBNNZGI52B3I6WKTV6MVC262FYNKYYPJUI5G47UO5S4MU7PWA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.306355311 (81h9m22.869678319s) [noise=true , address=104.36.82.55:4213 , piece_id=X4IHS2XSPH52LIMNUYT7KGWMAPUH3TEMJOEEVB3JV5N45SEOH7YA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.306788243 (81h9m22.870111252s) [noise=true , address=104.36.82.22:6539 , piece_id=GHQ2MVSTDGVYLUKLV23D6YWRR4PP7UUYHCUEQ4WWSWEMTBFO3P4A, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.392174770 (81h9m22.955497773s) [upload=64958]: Next returned nil segment Jun 22 01:32:47.392195767 (81h9m22.955518775s) [upload=64958]: Waiting for error group managing segments... Jun 22 01:32:47.480767633 (81h9m23.044090623s) [noise=true , address=104.36.82.22:6539 , piece_id=GHQ2MVSTDGVYLUKLV23D6YWRR4PP7UUYHCUEQ4WWSWEMTBFO3P4A, seg_pos={0 0}, upload=64958]: Done uploading piece. err: ecclient: upload failed (node:1QL6w8DSFR6CxDdnJmDuyVZxp6JU55VKiJofBZRfQi7q31i8Jc, address:104.36.82.22:6539): protocol: storage node overloaded, request limit: 2000 Jun 22 01:32:47.585663142 (81h9m23.148986155s) [noise=true , address=79.127.220.101:6048 , piece_id=AN7UQUXB42T2YXEH7ZJRZUDIW47M3RWUEOCL2VEBOGH74BZF3YKA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:47.640263002 (81h9m23.203586013s) [noise=true , address=45.91.116.12:5242 , piece_id=6ABT5IPYGSLFFJMBYMBYRU4EYIXABUAUHG3OVILOHBIUDS3BP27A, seg_pos={0 0}, upload=64958]: Done uploading piece. err: ecclient: upload failed (node:12JsFL62jgXmza8Md8eQ9k8cyw5V9Vz2RzN4y85ZvgHZjwrq4vQ, address:45.91.116.12:5242): protocol: storage node overloaded, request limit: 2000 Jun 22 01:32:47.984127275 (81h9m23.54745028s) [noise=true , address=64.49.13.137:4500 , piece_id=FAE2RXD57IRP5FXA2XR6THXFW27U7IWAZ4JDV5M2WRPXKUGZ4VIA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:48.100255249 (81h9m23.663578241s) [noise=true , address=77.90.31.14:4689 , piece_id=Q3HDS3BCMYDTODFRMU4XFRDCQK2QZE3ZMXVLUX4M54PDEBS7RR6A, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:48.221764035 (81h9m23.785087034s) [noise=true , address=104.36.82.54:4165 , piece_id=4CVOABPCWC7J7SRHZB6IN2WJJ6KGWUQ66ELUIZIFA67ETLQ7JEUQ, seg_pos={0 0}, upload=64958]: Done uploading piece. err: ecclient: upload failed (node:12wGUMXVEubEtki4qGnjGEsbYFB1iUauZcu3oDRKrCQsJMV3bV2, address:104.36.82.54:4165): protocol: storage node overloaded, request limit: 2000 Jun 22 01:32:48.509996527 (81h9m24.073319547s) [noise=true , address=104.36.82.9:6023 , piece_id=SISFAUU4RPF57OLGWM2BWQWE6GRUSCMORBYHWEXETNAVM2EEMO6Q, seg_pos={0 0}, upload=64958]: Done uploading piece. err: ecclient: upload failed (node:1j2S9tgWpLwmyvXZSq2qKKcbaC8r7LkzfDCaMxCejAni4CST1g, address:104.36.82.9:6023): protocol: storage node overloaded, request limit: 2000 Jun 22 01:32:48.510513169 (81h9m24.073836177s) [noise=true , address=104.36.82.42:6811 , piece_id=A5BFEHZ3WGGK53IZ7MKI6N4CVXHOORF6L47NR3UFDCAX7LBHRHVQ, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:48.565245605 (81h9m24.128568606s) [noise=true , address=79.127.205.234:5601 , piece_id=V7PMXBP2VFWBSZDG4GTLBD435T6F5DT7A43L4GQLQ5BDDTJ4WY6A, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:49.979979824 (81h9m25.543302825s) [noise=true , address=104.36.82.15:6289 , piece_id=GWL3P55BIK4IJEL5YFZKI7JCA4LADYOOHUSIQUNGJEKBJOTKMP5A, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:50.463750280 (81h9m26.02707328s) [noise=true , address=104.36.82.51:4084 , piece_id=D43FA7BFE54KPCWGDGDME2EGSWM7PONOVPD3DN2XY42MJFOKVDIQ, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:51.480599795 (81h9m27.043922798s) [noise=true , address=79.127.219.38:4023 , piece_id=BWLRCXXHBMBTNCZEVKKIIS5PB35PDQXQXCLIQB7GBLEIOWWUFBFA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:52.941200625 (81h9m28.504523627s) [noise=true , address=104.36.82.43:6841 , piece_id=PIHZMSTO3CII7KGDLGSAILTDRSLMEBVOLBCRCZJULMCFISXIZWXA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:58.281803598 (81h9m33.845126612s) [noise=true , address=64.49.13.139:4476 , piece_id=P7UKBNNZGI52B3I6WKTV6MVC262FYNKYYPJUI5G47UO5S4MU7PWA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:33:00.277271877 (81h9m35.840594867s) [noise=true , address=104.36.82.55:4213 , piece_id=X4IHS2XSPH52LIMNUYT7KGWMAPUH3TEMJOEEVB3JV5N45SEOH7YA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:33:09.435339213 (81h9m44.998662206s) [noise=true , address=104.36.82.52:4089 , piece_id=JSOEKLNWA3W3LOVKI3EGGKJYJ6J3UZY6KGABKQPQPXTIOOPMKVHA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:33:31.472868445 (81h10m7.03619143s) [noise=true , address=104.36.82.53:4122 , piece_id=HZZQBIKHQWLUE5EHK4KSKJGIUEG7D44RQASUJHCYXX6BICCGX6EA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:33:47.235404653 (81h10m22.798727642s) [noise=true , address=104.36.82.12:6158 , piece_id=LJZEOJVJCJAR2BRILIJI3ZVDGYMPBBTP2G3HCTN3MHNS4KYDPASQ, seg_pos={0 0}, upload=64958]: Done uploading piece. err: ecclient: upload failed (node:1SjM6Utm2YizeUbrBUD9Z7etE1nLadW5Fa1zYP7MioPeeqhdJ7, address:104.36.82.12:6158): protocol: expected piece hash; context deadline exceeded; EOF Jun 22 01:33:47.235448297 (81h10m22.798771306s) [seg_pos={0 0}, upload=64958]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jun 22 01:33:47.265442393 (81h10m22.828765401s) [upload=64958]: Done uploading segments... Jun 22 01:33:47.265476139 (81h10m22.82879914s) [upload=64958]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration:* 10.653036145s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [..............................X.................X...........] duration: 58.957723358s FAIL (2025-06-22 02:07:22 UTC) orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded Duration: 58.957780874s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [................X...........................................] duration: 13.239417059s FAIL (2025-06-22 02:06:29 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== oad=64949]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jun 22 01:32:13.674692303 (81h8m49.238015311s) [upload=64949]: Done uploading segments... Jun 22 01:32:13.674727219 (81h8m49.238050233s) [upload=64949]: Upload finished. err: <nil> Jun 22 01:32:47.266407995 (81h9m22.829731002s) [upload=64958]: Starting upload Jun 22 01:32:47.266524767 (81h9m22.829847771s) [upload=64958]: Done starting upload Jun 22 01:32:47.266813109 (81h9m22.830136116s) [upload=64958]: Uploading segments... Jun 22 01:32:47.266969526 (81h9m22.830292526s) [upload=64958]: Got next segment. Inline: false Jun 22 01:32:47.267075475 (81h9m22.830398475s) [upload=64958]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 22 01:32:47.267081679 (81h9m22.830404667s) [upload=64958]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 22 01:32:47.301770382 (81h9m22.865093389s) [seg_pos={0 0}, upload=64958]: Begin upload segment... Jun 22 01:32:47.301823796 (81h9m22.865146799s) [seg_pos={0 0}, upload=64958]: Done begin upload segment. Jun 22 01:32:47.301855055 (81h9m22.865178087s) [noise=true , address=104.36.82.43:6841 , piece_id=PIHZMSTO3CII7KGDLGSAILTDRSLMEBVOLBCRCZJULMCFISXIZWXA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.301902600 (81h9m22.865225606s) [noise=true , address=104.36.82.42:6811 , piece_id=A5BFEHZ3WGGK53IZ7MKI6N4CVXHOORF6L47NR3UFDCAX7LBHRHVQ, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.302234027 (81h9m22.865557029s) [noise=true , address=104.36.82.51:4084 , piece_id=D43FA7BFE54KPCWGDGDME2EGSWM7PONOVPD3DN2XY42MJFOKVDIQ, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.302496354 (81h9m22.865819363s) [noise=true , address=79.127.219.38:4023 , piece_id=BWLRCXXHBMBTNCZEVKKIIS5PB35PDQXQXCLIQB7GBLEIOWWUFBFA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.302577370 (81h9m22.865900378s) [noise=true , address=104.36.82.52:4089 , piece_id=JSOEKLNWA3W3LOVKI3EGGKJYJ6J3UZY6KGABKQPQPXTIOOPMKVHA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.302917829 (81h9m22.866240835s) [noise=true , address=104.36.82.53:4122 , piece_id=HZZQBIKHQWLUE5EHK4KSKJGIUEG7D44RQASUJHCYXX6BICCGX6EA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.303087165 (81h9m22.866410168s) [noise=true , address=77.90.31.14:4689 , piece_id=Q3HDS3BCMYDTODFRMU4XFRDCQK2QZE3ZMXVLUX4M54PDEBS7RR6A, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.303250714 (81h9m22.866573717s) [noise=true , address=79.127.205.234:5601 , piece_id=V7PMXBP2VFWBSZDG4GTLBD435T6F5DT7A43L4GQLQ5BDDTJ4WY6A, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.303579249 (81h9m22.866902255s) [noise=true , address=104.36.82.12:6158 , piece_id=LJZEOJVJCJAR2BRILIJI3ZVDGYMPBBTP2G3HCTN3MHNS4KYDPASQ, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.303599776 (81h9m22.866922784s) [noise=true , address=104.36.82.9:6023 , piece_id=SISFAUU4RPF57OLGWM2BWQWE6GRUSCMORBYHWEXETNAVM2EEMO6Q, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.304005346 (81h9m22.867328346s) [noise=true , address=45.91.116.12:5242 , piece_id=6ABT5IPYGSLFFJMBYMBYRU4EYIXABUAUHG3OVILOHBIUDS3BP27A, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.304388850 (81h9m22.867711851s) [noise=true , address=64.49.13.137:4500 , piece_id=FAE2RXD57IRP5FXA2XR6THXFW27U7IWAZ4JDV5M2WRPXKUGZ4VIA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.304881529 (81h9m22.868204536s) [noise=true , address=79.127.220.101:6048 , piece_id=AN7UQUXB42T2YXEH7ZJRZUDIW47M3RWUEOCL2VEBOGH74BZF3YKA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.305249860 (81h9m22.86857287s) [noise=true , address=104.36.82.54:4165 , piece_id=4CVOABPCWC7J7SRHZB6IN2WJJ6KGWUQ66ELUIZIFA67ETLQ7JEUQ, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.305604108 (81h9m22.868927112s) [noise=true , address=104.36.82.15:6289 , piece_id=GWL3P55BIK4IJEL5YFZKI7JCA4LADYOOHUSIQUNGJEKBJOTKMP5A, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.305930423 (81h9m22.869253425s) [noise=true , address=64.49.13.139:4476 , piece_id=P7UKBNNZGI52B3I6WKTV6MVC262FYNKYYPJUI5G47UO5S4MU7PWA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.306355311 (81h9m22.869678319s) [noise=true , address=104.36.82.55:4213 , piece_id=X4IHS2XSPH52LIMNUYT7KGWMAPUH3TEMJOEEVB3JV5N45SEOH7YA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.306788243 (81h9m22.870111252s) [noise=true , address=104.36.82.22:6539 , piece_id=GHQ2MVSTDGVYLUKLV23D6YWRR4PP7UUYHCUEQ4WWSWEMTBFO3P4A, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.392174770 (81h9m22.955497773s) [upload=64958]: Next returned nil segment Jun 22 01:32:47.392195767 (81h9m22.955518775s) [upload=64958]: Waiting for error group managing segments... Jun 22 01:32:47.480767633 (81h9m23.044090623s) [noise=true , address=104.36.82.22:6539 , piece_id=GHQ2MVSTDGVYLUKLV23D6YWRR4PP7UUYHCUEQ4WWSWEMTBFO3P4A, seg_pos={0 0}, upload=64958]: Done uploading piece. err: ecclient: upload failed (node:1QL6w8DSFR6CxDdnJmDuyVZxp6JU55VKiJofBZRfQi7q31i8Jc, address:104.36.82.22:6539): protocol: storage node overloaded, request limit: 2000 Jun 22 01:32:47.585663142 (81h9m23.148986155s) [noise=true , address=79.127.220.101:6048 , piece_id=AN7UQUXB42T2YXEH7ZJRZUDIW47M3RWUEOCL2VEBOGH74BZF3YKA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:47.640263002 (81h9m23.203586013s) [noise=true , address=45.91.116.12:5242 , piece_id=6ABT5IPYGSLFFJMBYMBYRU4EYIXABUAUHG3OVILOHBIUDS3BP27A, seg_pos={0 0}, upload=64958]: Done uploading piece. err: ecclient: upload failed (node:12JsFL62jgXmza8Md8eQ9k8cyw5V9Vz2RzN4y85ZvgHZjwrq4vQ, address:45.91.116.12:5242): protocol: storage node overloaded, request limit: 2000 Jun 22 01:32:47.984127275 (81h9m23.54745028s) [noise=true , address=64.49.13.137:4500 , piece_id=FAE2RXD57IRP5FXA2XR6THXFW27U7IWAZ4JDV5M2WRPXKUGZ4VIA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:48.100255249 (81h9m23.663578241s) [noise=true , address=77.90.31.14:4689 , piece_id=Q3HDS3BCMYDTODFRMU4XFRDCQK2QZE3ZMXVLUX4M54PDEBS7RR6A, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:48.221764035 (81h9m23.785087034s) [noise=true , address=104.36.82.54:4165 , piece_id=4CVOABPCWC7J7SRHZB6IN2WJJ6KGWUQ66ELUIZIFA67ETLQ7JEUQ, seg_pos={0 0}, upload=64958]: Done uploading piece. err: ecclient: upload failed (node:12wGUMXVEubEtki4qGnjGEsbYFB1iUauZcu3oDRKrCQsJMV3bV2, address:104.36.82.54:4165): protocol: storage node overloaded, request limit: 2000 Jun 22 01:32:48.509996527 (81h9m24.073319547s) [noise=true , address=104.36.82.9:6023 , piece_id=SISFAUU4RPF57OLGWM2BWQWE6GRUSCMORBYHWEXETNAVM2EEMO6Q, seg_pos={0 0}, upload=64958]: Done uploading piece. err: ecclient: upload failed (node:1j2S9tgWpLwmyvXZSq2qKKcbaC8r7LkzfDCaMxCejAni4CST1g, address:104.36.82.9:6023): protocol: storage node overloaded, request limit: 2000 Jun 22 01:32:48.510513169 (81h9m24.073836177s) [noise=true , address=104.36.82.42:6811 , piece_id=A5BFEHZ3WGGK53IZ7MKI6N4CVXHOORF6L47NR3UFDCAX7LBHRHVQ, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:48.565245605 (81h9m24.128568606s) [noise=true , address=79.127.205.234:5601 , piece_id=V7PMXBP2VFWBSZDG4GTLBD435T6F5DT7A43L4GQLQ5BDDTJ4WY6A, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:49.979979824 (81h9m25.543302825s) [noise=true , address=104.36.82.15:6289 , piece_id=GWL3P55BIK4IJEL5YFZKI7JCA4LADYOOHUSIQUNGJEKBJOTKMP5A, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:50.463750280 (81h9m26.02707328s) [noise=true , address=104.36.82.51:4084 , piece_id=D43FA7BFE54KPCWGDGDME2EGSWM7PONOVPD3DN2XY42MJFOKVDIQ, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:51.480599795 (81h9m27.043922798s) [noise=true , address=79.127.219.38:4023 , piece_id=BWLRCXXHBMBTNCZEVKKIIS5PB35PDQXQXCLIQB7GBLEIOWWUFBFA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:52.941200625 (81h9m28.504523627s) [noise=true , address=104.36.82.43:6841 , piece_id=PIHZMSTO3CII7KGDLGSAILTDRSLMEBVOLBCRCZJULMCFISXIZWXA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:58.281803598 (81h9m33.845126612s) [noise=true , address=64.49.13.139:4476 , piece_id=P7UKBNNZGI52B3I6WKTV6MVC262FYNKYYPJUI5G47UO5S4MU7PWA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:33:00.277271877 (81h9m35.840594867s) [noise=true , address=104.36.82.55:4213 , piece_id=X4IHS2XSPH52LIMNUYT7KGWMAPUH3TEMJOEEVB3JV5N45SEOH7YA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:33:09.435339213 (81h9m44.998662206s) [noise=true , address=104.36.82.52:4089 , piece_id=JSOEKLNWA3W3LOVKI3EGGKJYJ6J3UZY6KGABKQPQPXTIOOPMKVHA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:33:31.472868445 (81h10m7.03619143s) [noise=true , address=104.36.82.53:4122 , piece_id=HZZQBIKHQWLUE5EHK4KSKJGIUEG7D44RQASUJHCYXX6BICCGX6EA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:33:47.235404653 (81h10m22.798727642s) [noise=true , address=104.36.82.12:6158 , piece_id=LJZEOJVJCJAR2BRILIJI3ZVDGYMPBBTP2G3HCTN3MHNS4KYDPASQ, seg_pos={0 0}, upload=64958]: Done uploading piece. err: ecclient: upload failed (node:1SjM6Utm2YizeUbrBUD9Z7etE1nLadW5Fa1zYP7MioPeeqhdJ7, address:104.36.82.12:6158): protocol: expected piece hash; context deadline exceeded; EOF Jun 22 01:33:47.235448297 (81h10m22.798771306s) [seg_pos={0 0}, upload=64958]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jun 22 01:33:47.265442393 (81h10m22.828765401s) [upload=64958]: Done uploading segments... Jun 22 01:33:47.265476139 (81h10m22.82879914s) [upload=64958]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration:* 13.239443669s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [...............................X.................X..........] duration: 21.42019477s FAIL (2025-06-22 02:05:40 UTC) orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded Duration: 21.420247273s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.................X..........................................] duration: 22.368021126s FAIL (2025-06-22 02:05:23 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== oad=64949]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jun 22 01:32:13.674692303 (81h8m49.238015311s) [upload=64949]: Done uploading segments... Jun 22 01:32:13.674727219 (81h8m49.238050233s) [upload=64949]: Upload finished. err: <nil> Jun 22 01:32:47.266407995 (81h9m22.829731002s) [upload=64958]: Starting upload Jun 22 01:32:47.266524767 (81h9m22.829847771s) [upload=64958]: Done starting upload Jun 22 01:32:47.266813109 (81h9m22.830136116s) [upload=64958]: Uploading segments... Jun 22 01:32:47.266969526 (81h9m22.830292526s) [upload=64958]: Got next segment. Inline: false Jun 22 01:32:47.267075475 (81h9m22.830398475s) [upload=64958]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 22 01:32:47.267081679 (81h9m22.830404667s) [upload=64958]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 22 01:32:47.301770382 (81h9m22.865093389s) [seg_pos={0 0}, upload=64958]: Begin upload segment... Jun 22 01:32:47.301823796 (81h9m22.865146799s) [seg_pos={0 0}, upload=64958]: Done begin upload segment. Jun 22 01:32:47.301855055 (81h9m22.865178087s) [noise=true , address=104.36.82.43:6841 , piece_id=PIHZMSTO3CII7KGDLGSAILTDRSLMEBVOLBCRCZJULMCFISXIZWXA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.301902600 (81h9m22.865225606s) [noise=true , address=104.36.82.42:6811 , piece_id=A5BFEHZ3WGGK53IZ7MKI6N4CVXHOORF6L47NR3UFDCAX7LBHRHVQ, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.302234027 (81h9m22.865557029s) [noise=true , address=104.36.82.51:4084 , piece_id=D43FA7BFE54KPCWGDGDME2EGSWM7PONOVPD3DN2XY42MJFOKVDIQ, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.302496354 (81h9m22.865819363s) [noise=true , address=79.127.219.38:4023 , piece_id=BWLRCXXHBMBTNCZEVKKIIS5PB35PDQXQXCLIQB7GBLEIOWWUFBFA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.302577370 (81h9m22.865900378s) [noise=true , address=104.36.82.52:4089 , piece_id=JSOEKLNWA3W3LOVKI3EGGKJYJ6J3UZY6KGABKQPQPXTIOOPMKVHA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.302917829 (81h9m22.866240835s) [noise=true , address=104.36.82.53:4122 , piece_id=HZZQBIKHQWLUE5EHK4KSKJGIUEG7D44RQASUJHCYXX6BICCGX6EA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.303087165 (81h9m22.866410168s) [noise=true , address=77.90.31.14:4689 , piece_id=Q3HDS3BCMYDTODFRMU4XFRDCQK2QZE3ZMXVLUX4M54PDEBS7RR6A, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.303250714 (81h9m22.866573717s) [noise=true , address=79.127.205.234:5601 , piece_id=V7PMXBP2VFWBSZDG4GTLBD435T6F5DT7A43L4GQLQ5BDDTJ4WY6A, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.303579249 (81h9m22.866902255s) [noise=true , address=104.36.82.12:6158 , piece_id=LJZEOJVJCJAR2BRILIJI3ZVDGYMPBBTP2G3HCTN3MHNS4KYDPASQ, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.303599776 (81h9m22.866922784s) [noise=true , address=104.36.82.9:6023 , piece_id=SISFAUU4RPF57OLGWM2BWQWE6GRUSCMORBYHWEXETNAVM2EEMO6Q, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.304005346 (81h9m22.867328346s) [noise=true , address=45.91.116.12:5242 , piece_id=6ABT5IPYGSLFFJMBYMBYRU4EYIXABUAUHG3OVILOHBIUDS3BP27A, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.304388850 (81h9m22.867711851s) [noise=true , address=64.49.13.137:4500 , piece_id=FAE2RXD57IRP5FXA2XR6THXFW27U7IWAZ4JDV5M2WRPXKUGZ4VIA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.304881529 (81h9m22.868204536s) [noise=true , address=79.127.220.101:6048 , piece_id=AN7UQUXB42T2YXEH7ZJRZUDIW47M3RWUEOCL2VEBOGH74BZF3YKA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.305249860 (81h9m22.86857287s) [noise=true , address=104.36.82.54:4165 , piece_id=4CVOABPCWC7J7SRHZB6IN2WJJ6KGWUQ66ELUIZIFA67ETLQ7JEUQ, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.305604108 (81h9m22.868927112s) [noise=true , address=104.36.82.15:6289 , piece_id=GWL3P55BIK4IJEL5YFZKI7JCA4LADYOOHUSIQUNGJEKBJOTKMP5A, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.305930423 (81h9m22.869253425s) [noise=true , address=64.49.13.139:4476 , piece_id=P7UKBNNZGI52B3I6WKTV6MVC262FYNKYYPJUI5G47UO5S4MU7PWA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.306355311 (81h9m22.869678319s) [noise=true , address=104.36.82.55:4213 , piece_id=X4IHS2XSPH52LIMNUYT7KGWMAPUH3TEMJOEEVB3JV5N45SEOH7YA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.306788243 (81h9m22.870111252s) [noise=true , address=104.36.82.22:6539 , piece_id=GHQ2MVSTDGVYLUKLV23D6YWRR4PP7UUYHCUEQ4WWSWEMTBFO3P4A, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.392174770 (81h9m22.955497773s) [upload=64958]: Next returned nil segment Jun 22 01:32:47.392195767 (81h9m22.955518775s) [upload=64958]: Waiting for error group managing segments... Jun 22 01:32:47.480767633 (81h9m23.044090623s) [noise=true , address=104.36.82.22:6539 , piece_id=GHQ2MVSTDGVYLUKLV23D6YWRR4PP7UUYHCUEQ4WWSWEMTBFO3P4A, seg_pos={0 0}, upload=64958]: Done uploading piece. err: ecclient: upload failed (node:1QL6w8DSFR6CxDdnJmDuyVZxp6JU55VKiJofBZRfQi7q31i8Jc, address:104.36.82.22:6539): protocol: storage node overloaded, request limit: 2000 Jun 22 01:32:47.585663142 (81h9m23.148986155s) [noise=true , address=79.127.220.101:6048 , piece_id=AN7UQUXB42T2YXEH7ZJRZUDIW47M3RWUEOCL2VEBOGH74BZF3YKA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:47.640263002 (81h9m23.203586013s) [noise=true , address=45.91.116.12:5242 , piece_id=6ABT5IPYGSLFFJMBYMBYRU4EYIXABUAUHG3OVILOHBIUDS3BP27A, seg_pos={0 0}, upload=64958]: Done uploading piece. err: ecclient: upload failed (node:12JsFL62jgXmza8Md8eQ9k8cyw5V9Vz2RzN4y85ZvgHZjwrq4vQ, address:45.91.116.12:5242): protocol: storage node overloaded, request limit: 2000 Jun 22 01:32:47.984127275 (81h9m23.54745028s) [noise=true , address=64.49.13.137:4500 , piece_id=FAE2RXD57IRP5FXA2XR6THXFW27U7IWAZ4JDV5M2WRPXKUGZ4VIA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:48.100255249 (81h9m23.663578241s) [noise=true , address=77.90.31.14:4689 , piece_id=Q3HDS3BCMYDTODFRMU4XFRDCQK2QZE3ZMXVLUX4M54PDEBS7RR6A, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:48.221764035 (81h9m23.785087034s) [noise=true , address=104.36.82.54:4165 , piece_id=4CVOABPCWC7J7SRHZB6IN2WJJ6KGWUQ66ELUIZIFA67ETLQ7JEUQ, seg_pos={0 0}, upload=64958]: Done uploading piece. err: ecclient: upload failed (node:12wGUMXVEubEtki4qGnjGEsbYFB1iUauZcu3oDRKrCQsJMV3bV2, address:104.36.82.54:4165): protocol: storage node overloaded, request limit: 2000 Jun 22 01:32:48.509996527 (81h9m24.073319547s) [noise=true , address=104.36.82.9:6023 , piece_id=SISFAUU4RPF57OLGWM2BWQWE6GRUSCMORBYHWEXETNAVM2EEMO6Q, seg_pos={0 0}, upload=64958]: Done uploading piece. err: ecclient: upload failed (node:1j2S9tgWpLwmyvXZSq2qKKcbaC8r7LkzfDCaMxCejAni4CST1g, address:104.36.82.9:6023): protocol: storage node overloaded, request limit: 2000 Jun 22 01:32:48.510513169 (81h9m24.073836177s) [noise=true , address=104.36.82.42:6811 , piece_id=A5BFEHZ3WGGK53IZ7MKI6N4CVXHOORF6L47NR3UFDCAX7LBHRHVQ, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:48.565245605 (81h9m24.128568606s) [noise=true , address=79.127.205.234:5601 , piece_id=V7PMXBP2VFWBSZDG4GTLBD435T6F5DT7A43L4GQLQ5BDDTJ4WY6A, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:49.979979824 (81h9m25.543302825s) [noise=true , address=104.36.82.15:6289 , piece_id=GWL3P55BIK4IJEL5YFZKI7JCA4LADYOOHUSIQUNGJEKBJOTKMP5A, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:50.463750280 (81h9m26.02707328s) [noise=true , address=104.36.82.51:4084 , piece_id=D43FA7BFE54KPCWGDGDME2EGSWM7PONOVPD3DN2XY42MJFOKVDIQ, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:51.480599795 (81h9m27.043922798s) [noise=true , address=79.127.219.38:4023 , piece_id=BWLRCXXHBMBTNCZEVKKIIS5PB35PDQXQXCLIQB7GBLEIOWWUFBFA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:52.941200625 (81h9m28.504523627s) [noise=true , address=104.36.82.43:6841 , piece_id=PIHZMSTO3CII7KGDLGSAILTDRSLMEBVOLBCRCZJULMCFISXIZWXA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:58.281803598 (81h9m33.845126612s) [noise=true , address=64.49.13.139:4476 , piece_id=P7UKBNNZGI52B3I6WKTV6MVC262FYNKYYPJUI5G47UO5S4MU7PWA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:33:00.277271877 (81h9m35.840594867s) [noise=true , address=104.36.82.55:4213 , piece_id=X4IHS2XSPH52LIMNUYT7KGWMAPUH3TEMJOEEVB3JV5N45SEOH7YA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:33:09.435339213 (81h9m44.998662206s) [noise=true , address=104.36.82.52:4089 , piece_id=JSOEKLNWA3W3LOVKI3EGGKJYJ6J3UZY6KGABKQPQPXTIOOPMKVHA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:33:31.472868445 (81h10m7.03619143s) [noise=true , address=104.36.82.53:4122 , piece_id=HZZQBIKHQWLUE5EHK4KSKJGIUEG7D44RQASUJHCYXX6BICCGX6EA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:33:47.235404653 (81h10m22.798727642s) [noise=true , address=104.36.82.12:6158 , piece_id=LJZEOJVJCJAR2BRILIJI3ZVDGYMPBBTP2G3HCTN3MHNS4KYDPASQ, seg_pos={0 0}, upload=64958]: Done uploading piece. err: ecclient: upload failed (node:1SjM6Utm2YizeUbrBUD9Z7etE1nLadW5Fa1zYP7MioPeeqhdJ7, address:104.36.82.12:6158): protocol: expected piece hash; context deadline exceeded; EOF Jun 22 01:33:47.235448297 (81h10m22.798771306s) [seg_pos={0 0}, upload=64958]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jun 22 01:33:47.265442393 (81h10m22.828765401s) [upload=64958]: Done uploading segments... Jun 22 01:33:47.265476139 (81h10m22.82879914s) [upload=64958]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration:* 22.368052543s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [................................X.................X.........] duration: 26.845273523s FAIL (2025-06-22 02:05:17 UTC) orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded Duration: 26.845295191s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [..................X.........................................] duration: 13.813532448s FAIL (2025-06-22 02:05:00 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== oad=64949]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jun 22 01:32:13.674692303 (81h8m49.238015311s) [upload=64949]: Done uploading segments... Jun 22 01:32:13.674727219 (81h8m49.238050233s) [upload=64949]: Upload finished. err: <nil> Jun 22 01:32:47.266407995 (81h9m22.829731002s) [upload=64958]: Starting upload Jun 22 01:32:47.266524767 (81h9m22.829847771s) [upload=64958]: Done starting upload Jun 22 01:32:47.266813109 (81h9m22.830136116s) [upload=64958]: Uploading segments... Jun 22 01:32:47.266969526 (81h9m22.830292526s) [upload=64958]: Got next segment. Inline: false Jun 22 01:32:47.267075475 (81h9m22.830398475s) [upload=64958]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 22 01:32:47.267081679 (81h9m22.830404667s) [upload=64958]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 22 01:32:47.301770382 (81h9m22.865093389s) [seg_pos={0 0}, upload=64958]: Begin upload segment... Jun 22 01:32:47.301823796 (81h9m22.865146799s) [seg_pos={0 0}, upload=64958]: Done begin upload segment. Jun 22 01:32:47.301855055 (81h9m22.865178087s) [noise=true , address=104.36.82.43:6841 , piece_id=PIHZMSTO3CII7KGDLGSAILTDRSLMEBVOLBCRCZJULMCFISXIZWXA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.301902600 (81h9m22.865225606s) [noise=true , address=104.36.82.42:6811 , piece_id=A5BFEHZ3WGGK53IZ7MKI6N4CVXHOORF6L47NR3UFDCAX7LBHRHVQ, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.302234027 (81h9m22.865557029s) [noise=true , address=104.36.82.51:4084 , piece_id=D43FA7BFE54KPCWGDGDME2EGSWM7PONOVPD3DN2XY42MJFOKVDIQ, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.302496354 (81h9m22.865819363s) [noise=true , address=79.127.219.38:4023 , piece_id=BWLRCXXHBMBTNCZEVKKIIS5PB35PDQXQXCLIQB7GBLEIOWWUFBFA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.302577370 (81h9m22.865900378s) [noise=true , address=104.36.82.52:4089 , piece_id=JSOEKLNWA3W3LOVKI3EGGKJYJ6J3UZY6KGABKQPQPXTIOOPMKVHA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.302917829 (81h9m22.866240835s) [noise=true , address=104.36.82.53:4122 , piece_id=HZZQBIKHQWLUE5EHK4KSKJGIUEG7D44RQASUJHCYXX6BICCGX6EA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.303087165 (81h9m22.866410168s) [noise=true , address=77.90.31.14:4689 , piece_id=Q3HDS3BCMYDTODFRMU4XFRDCQK2QZE3ZMXVLUX4M54PDEBS7RR6A, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.303250714 (81h9m22.866573717s) [noise=true , address=79.127.205.234:5601 , piece_id=V7PMXBP2VFWBSZDG4GTLBD435T6F5DT7A43L4GQLQ5BDDTJ4WY6A, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.303579249 (81h9m22.866902255s) [noise=true , address=104.36.82.12:6158 , piece_id=LJZEOJVJCJAR2BRILIJI3ZVDGYMPBBTP2G3HCTN3MHNS4KYDPASQ, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.303599776 (81h9m22.866922784s) [noise=true , address=104.36.82.9:6023 , piece_id=SISFAUU4RPF57OLGWM2BWQWE6GRUSCMORBYHWEXETNAVM2EEMO6Q, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.304005346 (81h9m22.867328346s) [noise=true , address=45.91.116.12:5242 , piece_id=6ABT5IPYGSLFFJMBYMBYRU4EYIXABUAUHG3OVILOHBIUDS3BP27A, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.304388850 (81h9m22.867711851s) [noise=true , address=64.49.13.137:4500 , piece_id=FAE2RXD57IRP5FXA2XR6THXFW27U7IWAZ4JDV5M2WRPXKUGZ4VIA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.304881529 (81h9m22.868204536s) [noise=true , address=79.127.220.101:6048 , piece_id=AN7UQUXB42T2YXEH7ZJRZUDIW47M3RWUEOCL2VEBOGH74BZF3YKA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.305249860 (81h9m22.86857287s) [noise=true , address=104.36.82.54:4165 , piece_id=4CVOABPCWC7J7SRHZB6IN2WJJ6KGWUQ66ELUIZIFA67ETLQ7JEUQ, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.305604108 (81h9m22.868927112s) [noise=true , address=104.36.82.15:6289 , piece_id=GWL3P55BIK4IJEL5YFZKI7JCA4LADYOOHUSIQUNGJEKBJOTKMP5A, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.305930423 (81h9m22.869253425s) [noise=true , address=64.49.13.139:4476 , piece_id=P7UKBNNZGI52B3I6WKTV6MVC262FYNKYYPJUI5G47UO5S4MU7PWA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.306355311 (81h9m22.869678319s) [noise=true , address=104.36.82.55:4213 , piece_id=X4IHS2XSPH52LIMNUYT7KGWMAPUH3TEMJOEEVB3JV5N45SEOH7YA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.306788243 (81h9m22.870111252s) [noise=true , address=104.36.82.22:6539 , piece_id=GHQ2MVSTDGVYLUKLV23D6YWRR4PP7UUYHCUEQ4WWSWEMTBFO3P4A, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.392174770 (81h9m22.955497773s) [upload=64958]: Next returned nil segment Jun 22 01:32:47.392195767 (81h9m22.955518775s) [upload=64958]: Waiting for error group managing segments... Jun 22 01:32:47.480767633 (81h9m23.044090623s) [noise=true , address=104.36.82.22:6539 , piece_id=GHQ2MVSTDGVYLUKLV23D6YWRR4PP7UUYHCUEQ4WWSWEMTBFO3P4A, seg_pos={0 0}, upload=64958]: Done uploading piece. err: ecclient: upload failed (node:1QL6w8DSFR6CxDdnJmDuyVZxp6JU55VKiJofBZRfQi7q31i8Jc, address:104.36.82.22:6539): protocol: storage node overloaded, request limit: 2000 Jun 22 01:32:47.585663142 (81h9m23.148986155s) [noise=true , address=79.127.220.101:6048 , piece_id=AN7UQUXB42T2YXEH7ZJRZUDIW47M3RWUEOCL2VEBOGH74BZF3YKA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:47.640263002 (81h9m23.203586013s) [noise=true , address=45.91.116.12:5242 , piece_id=6ABT5IPYGSLFFJMBYMBYRU4EYIXABUAUHG3OVILOHBIUDS3BP27A, seg_pos={0 0}, upload=64958]: Done uploading piece. err: ecclient: upload failed (node:12JsFL62jgXmza8Md8eQ9k8cyw5V9Vz2RzN4y85ZvgHZjwrq4vQ, address:45.91.116.12:5242): protocol: storage node overloaded, request limit: 2000 Jun 22 01:32:47.984127275 (81h9m23.54745028s) [noise=true , address=64.49.13.137:4500 , piece_id=FAE2RXD57IRP5FXA2XR6THXFW27U7IWAZ4JDV5M2WRPXKUGZ4VIA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:48.100255249 (81h9m23.663578241s) [noise=true , address=77.90.31.14:4689 , piece_id=Q3HDS3BCMYDTODFRMU4XFRDCQK2QZE3ZMXVLUX4M54PDEBS7RR6A, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:48.221764035 (81h9m23.785087034s) [noise=true , address=104.36.82.54:4165 , piece_id=4CVOABPCWC7J7SRHZB6IN2WJJ6KGWUQ66ELUIZIFA67ETLQ7JEUQ, seg_pos={0 0}, upload=64958]: Done uploading piece. err: ecclient: upload failed (node:12wGUMXVEubEtki4qGnjGEsbYFB1iUauZcu3oDRKrCQsJMV3bV2, address:104.36.82.54:4165): protocol: storage node overloaded, request limit: 2000 Jun 22 01:32:48.509996527 (81h9m24.073319547s) [noise=true , address=104.36.82.9:6023 , piece_id=SISFAUU4RPF57OLGWM2BWQWE6GRUSCMORBYHWEXETNAVM2EEMO6Q, seg_pos={0 0}, upload=64958]: Done uploading piece. err: ecclient: upload failed (node:1j2S9tgWpLwmyvXZSq2qKKcbaC8r7LkzfDCaMxCejAni4CST1g, address:104.36.82.9:6023): protocol: storage node overloaded, request limit: 2000 Jun 22 01:32:48.510513169 (81h9m24.073836177s) [noise=true , address=104.36.82.42:6811 , piece_id=A5BFEHZ3WGGK53IZ7MKI6N4CVXHOORF6L47NR3UFDCAX7LBHRHVQ, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:48.565245605 (81h9m24.128568606s) [noise=true , address=79.127.205.234:5601 , piece_id=V7PMXBP2VFWBSZDG4GTLBD435T6F5DT7A43L4GQLQ5BDDTJ4WY6A, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:49.979979824 (81h9m25.543302825s) [noise=true , address=104.36.82.15:6289 , piece_id=GWL3P55BIK4IJEL5YFZKI7JCA4LADYOOHUSIQUNGJEKBJOTKMP5A, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:50.463750280 (81h9m26.02707328s) [noise=true , address=104.36.82.51:4084 , piece_id=D43FA7BFE54KPCWGDGDME2EGSWM7PONOVPD3DN2XY42MJFOKVDIQ, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:51.480599795 (81h9m27.043922798s) [noise=true , address=79.127.219.38:4023 , piece_id=BWLRCXXHBMBTNCZEVKKIIS5PB35PDQXQXCLIQB7GBLEIOWWUFBFA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:52.941200625 (81h9m28.504523627s) [noise=true , address=104.36.82.43:6841 , piece_id=PIHZMSTO3CII7KGDLGSAILTDRSLMEBVOLBCRCZJULMCFISXIZWXA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:58.281803598 (81h9m33.845126612s) [noise=true , address=64.49.13.139:4476 , piece_id=P7UKBNNZGI52B3I6WKTV6MVC262FYNKYYPJUI5G47UO5S4MU7PWA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:33:00.277271877 (81h9m35.840594867s) [noise=true , address=104.36.82.55:4213 , piece_id=X4IHS2XSPH52LIMNUYT7KGWMAPUH3TEMJOEEVB3JV5N45SEOH7YA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:33:09.435339213 (81h9m44.998662206s) [noise=true , address=104.36.82.52:4089 , piece_id=JSOEKLNWA3W3LOVKI3EGGKJYJ6J3UZY6KGABKQPQPXTIOOPMKVHA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:33:31.472868445 (81h10m7.03619143s) [noise=true , address=104.36.82.53:4122 , piece_id=HZZQBIKHQWLUE5EHK4KSKJGIUEG7D44RQASUJHCYXX6BICCGX6EA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:33:47.235404653 (81h10m22.798727642s) [noise=true , address=104.36.82.12:6158 , piece_id=LJZEOJVJCJAR2BRILIJI3ZVDGYMPBBTP2G3HCTN3MHNS4KYDPASQ, seg_pos={0 0}, upload=64958]: Done uploading piece. err: ecclient: upload failed (node:1SjM6Utm2YizeUbrBUD9Z7etE1nLadW5Fa1zYP7MioPeeqhdJ7, address:104.36.82.12:6158): protocol: expected piece hash; context deadline exceeded; EOF Jun 22 01:33:47.235448297 (81h10m22.798771306s) [seg_pos={0 0}, upload=64958]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jun 22 01:33:47.265442393 (81h10m22.828765401s) [upload=64958]: Done uploading segments... Jun 22 01:33:47.265476139 (81h10m22.82879914s) [upload=64958]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration:* 13.81356694s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [...................X........................................] duration: 22.144833836s FAIL (2025-06-22 02:04:46 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== oad=64949]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jun 22 01:32:13.674692303 (81h8m49.238015311s) [upload=64949]: Done uploading segments... Jun 22 01:32:13.674727219 (81h8m49.238050233s) [upload=64949]: Upload finished. err: <nil> Jun 22 01:32:47.266407995 (81h9m22.829731002s) [upload=64958]: Starting upload Jun 22 01:32:47.266524767 (81h9m22.829847771s) [upload=64958]: Done starting upload Jun 22 01:32:47.266813109 (81h9m22.830136116s) [upload=64958]: Uploading segments... Jun 22 01:32:47.266969526 (81h9m22.830292526s) [upload=64958]: Got next segment. Inline: false Jun 22 01:32:47.267075475 (81h9m22.830398475s) [upload=64958]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 22 01:32:47.267081679 (81h9m22.830404667s) [upload=64958]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 22 01:32:47.301770382 (81h9m22.865093389s) [seg_pos={0 0}, upload=64958]: Begin upload segment... Jun 22 01:32:47.301823796 (81h9m22.865146799s) [seg_pos={0 0}, upload=64958]: Done begin upload segment. Jun 22 01:32:47.301855055 (81h9m22.865178087s) [noise=true , address=104.36.82.43:6841 , piece_id=PIHZMSTO3CII7KGDLGSAILTDRSLMEBVOLBCRCZJULMCFISXIZWXA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.301902600 (81h9m22.865225606s) [noise=true , address=104.36.82.42:6811 , piece_id=A5BFEHZ3WGGK53IZ7MKI6N4CVXHOORF6L47NR3UFDCAX7LBHRHVQ, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.302234027 (81h9m22.865557029s) [noise=true , address=104.36.82.51:4084 , piece_id=D43FA7BFE54KPCWGDGDME2EGSWM7PONOVPD3DN2XY42MJFOKVDIQ, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.302496354 (81h9m22.865819363s) [noise=true , address=79.127.219.38:4023 , piece_id=BWLRCXXHBMBTNCZEVKKIIS5PB35PDQXQXCLIQB7GBLEIOWWUFBFA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.302577370 (81h9m22.865900378s) [noise=true , address=104.36.82.52:4089 , piece_id=JSOEKLNWA3W3LOVKI3EGGKJYJ6J3UZY6KGABKQPQPXTIOOPMKVHA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.302917829 (81h9m22.866240835s) [noise=true , address=104.36.82.53:4122 , piece_id=HZZQBIKHQWLUE5EHK4KSKJGIUEG7D44RQASUJHCYXX6BICCGX6EA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.303087165 (81h9m22.866410168s) [noise=true , address=77.90.31.14:4689 , piece_id=Q3HDS3BCMYDTODFRMU4XFRDCQK2QZE3ZMXVLUX4M54PDEBS7RR6A, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.303250714 (81h9m22.866573717s) [noise=true , address=79.127.205.234:5601 , piece_id=V7PMXBP2VFWBSZDG4GTLBD435T6F5DT7A43L4GQLQ5BDDTJ4WY6A, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.303579249 (81h9m22.866902255s) [noise=true , address=104.36.82.12:6158 , piece_id=LJZEOJVJCJAR2BRILIJI3ZVDGYMPBBTP2G3HCTN3MHNS4KYDPASQ, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.303599776 (81h9m22.866922784s) [noise=true , address=104.36.82.9:6023 , piece_id=SISFAUU4RPF57OLGWM2BWQWE6GRUSCMORBYHWEXETNAVM2EEMO6Q, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.304005346 (81h9m22.867328346s) [noise=true , address=45.91.116.12:5242 , piece_id=6ABT5IPYGSLFFJMBYMBYRU4EYIXABUAUHG3OVILOHBIUDS3BP27A, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.304388850 (81h9m22.867711851s) [noise=true , address=64.49.13.137:4500 , piece_id=FAE2RXD57IRP5FXA2XR6THXFW27U7IWAZ4JDV5M2WRPXKUGZ4VIA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.304881529 (81h9m22.868204536s) [noise=true , address=79.127.220.101:6048 , piece_id=AN7UQUXB42T2YXEH7ZJRZUDIW47M3RWUEOCL2VEBOGH74BZF3YKA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.305249860 (81h9m22.86857287s) [noise=true , address=104.36.82.54:4165 , piece_id=4CVOABPCWC7J7SRHZB6IN2WJJ6KGWUQ66ELUIZIFA67ETLQ7JEUQ, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.305604108 (81h9m22.868927112s) [noise=true , address=104.36.82.15:6289 , piece_id=GWL3P55BIK4IJEL5YFZKI7JCA4LADYOOHUSIQUNGJEKBJOTKMP5A, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.305930423 (81h9m22.869253425s) [noise=true , address=64.49.13.139:4476 , piece_id=P7UKBNNZGI52B3I6WKTV6MVC262FYNKYYPJUI5G47UO5S4MU7PWA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.306355311 (81h9m22.869678319s) [noise=true , address=104.36.82.55:4213 , piece_id=X4IHS2XSPH52LIMNUYT7KGWMAPUH3TEMJOEEVB3JV5N45SEOH7YA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.306788243 (81h9m22.870111252s) [noise=true , address=104.36.82.22:6539 , piece_id=GHQ2MVSTDGVYLUKLV23D6YWRR4PP7UUYHCUEQ4WWSWEMTBFO3P4A, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.392174770 (81h9m22.955497773s) [upload=64958]: Next returned nil segment Jun 22 01:32:47.392195767 (81h9m22.955518775s) [upload=64958]: Waiting for error group managing segments... Jun 22 01:32:47.480767633 (81h9m23.044090623s) [noise=true , address=104.36.82.22:6539 , piece_id=GHQ2MVSTDGVYLUKLV23D6YWRR4PP7UUYHCUEQ4WWSWEMTBFO3P4A, seg_pos={0 0}, upload=64958]: Done uploading piece. err: ecclient: upload failed (node:1QL6w8DSFR6CxDdnJmDuyVZxp6JU55VKiJofBZRfQi7q31i8Jc, address:104.36.82.22:6539): protocol: storage node overloaded, request limit: 2000 Jun 22 01:32:47.585663142 (81h9m23.148986155s) [noise=true , address=79.127.220.101:6048 , piece_id=AN7UQUXB42T2YXEH7ZJRZUDIW47M3RWUEOCL2VEBOGH74BZF3YKA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:47.640263002 (81h9m23.203586013s) [noise=true , address=45.91.116.12:5242 , piece_id=6ABT5IPYGSLFFJMBYMBYRU4EYIXABUAUHG3OVILOHBIUDS3BP27A, seg_pos={0 0}, upload=64958]: Done uploading piece. err: ecclient: upload failed (node:12JsFL62jgXmza8Md8eQ9k8cyw5V9Vz2RzN4y85ZvgHZjwrq4vQ, address:45.91.116.12:5242): protocol: storage node overloaded, request limit: 2000 Jun 22 01:32:47.984127275 (81h9m23.54745028s) [noise=true , address=64.49.13.137:4500 , piece_id=FAE2RXD57IRP5FXA2XR6THXFW27U7IWAZ4JDV5M2WRPXKUGZ4VIA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:48.100255249 (81h9m23.663578241s) [noise=true , address=77.90.31.14:4689 , piece_id=Q3HDS3BCMYDTODFRMU4XFRDCQK2QZE3ZMXVLUX4M54PDEBS7RR6A, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:48.221764035 (81h9m23.785087034s) [noise=true , address=104.36.82.54:4165 , piece_id=4CVOABPCWC7J7SRHZB6IN2WJJ6KGWUQ66ELUIZIFA67ETLQ7JEUQ, seg_pos={0 0}, upload=64958]: Done uploading piece. err: ecclient: upload failed (node:12wGUMXVEubEtki4qGnjGEsbYFB1iUauZcu3oDRKrCQsJMV3bV2, address:104.36.82.54:4165): protocol: storage node overloaded, request limit: 2000 Jun 22 01:32:48.509996527 (81h9m24.073319547s) [noise=true , address=104.36.82.9:6023 , piece_id=SISFAUU4RPF57OLGWM2BWQWE6GRUSCMORBYHWEXETNAVM2EEMO6Q, seg_pos={0 0}, upload=64958]: Done uploading piece. err: ecclient: upload failed (node:1j2S9tgWpLwmyvXZSq2qKKcbaC8r7LkzfDCaMxCejAni4CST1g, address:104.36.82.9:6023): protocol: storage node overloaded, request limit: 2000 Jun 22 01:32:48.510513169 (81h9m24.073836177s) [noise=true , address=104.36.82.42:6811 , piece_id=A5BFEHZ3WGGK53IZ7MKI6N4CVXHOORF6L47NR3UFDCAX7LBHRHVQ, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:48.565245605 (81h9m24.128568606s) [noise=true , address=79.127.205.234:5601 , piece_id=V7PMXBP2VFWBSZDG4GTLBD435T6F5DT7A43L4GQLQ5BDDTJ4WY6A, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:49.979979824 (81h9m25.543302825s) [noise=true , address=104.36.82.15:6289 , piece_id=GWL3P55BIK4IJEL5YFZKI7JCA4LADYOOHUSIQUNGJEKBJOTKMP5A, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:50.463750280 (81h9m26.02707328s) [noise=true , address=104.36.82.51:4084 , piece_id=D43FA7BFE54KPCWGDGDME2EGSWM7PONOVPD3DN2XY42MJFOKVDIQ, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:51.480599795 (81h9m27.043922798s) [noise=true , address=79.127.219.38:4023 , piece_id=BWLRCXXHBMBTNCZEVKKIIS5PB35PDQXQXCLIQB7GBLEIOWWUFBFA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:52.941200625 (81h9m28.504523627s) [noise=true , address=104.36.82.43:6841 , piece_id=PIHZMSTO3CII7KGDLGSAILTDRSLMEBVOLBCRCZJULMCFISXIZWXA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:58.281803598 (81h9m33.845126612s) [noise=true , address=64.49.13.139:4476 , piece_id=P7UKBNNZGI52B3I6WKTV6MVC262FYNKYYPJUI5G47UO5S4MU7PWA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:33:00.277271877 (81h9m35.840594867s) [noise=true , address=104.36.82.55:4213 , piece_id=X4IHS2XSPH52LIMNUYT7KGWMAPUH3TEMJOEEVB3JV5N45SEOH7YA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:33:09.435339213 (81h9m44.998662206s) [noise=true , address=104.36.82.52:4089 , piece_id=JSOEKLNWA3W3LOVKI3EGGKJYJ6J3UZY6KGABKQPQPXTIOOPMKVHA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:33:31.472868445 (81h10m7.03619143s) [noise=true , address=104.36.82.53:4122 , piece_id=HZZQBIKHQWLUE5EHK4KSKJGIUEG7D44RQASUJHCYXX6BICCGX6EA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:33:47.235404653 (81h10m22.798727642s) [noise=true , address=104.36.82.12:6158 , piece_id=LJZEOJVJCJAR2BRILIJI3ZVDGYMPBBTP2G3HCTN3MHNS4KYDPASQ, seg_pos={0 0}, upload=64958]: Done uploading piece. err: ecclient: upload failed (node:1SjM6Utm2YizeUbrBUD9Z7etE1nLadW5Fa1zYP7MioPeeqhdJ7, address:104.36.82.12:6158): protocol: expected piece hash; context deadline exceeded; EOF Jun 22 01:33:47.235448297 (81h10m22.798771306s) [seg_pos={0 0}, upload=64958]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jun 22 01:33:47.265442393 (81h10m22.828765401s) [upload=64958]: Done uploading segments... Jun 22 01:33:47.265476139 (81h10m22.82879914s) [upload=64958]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration:* 22.144861552s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [.................................X.................X........] duration: 36.095418934s FAIL (2025-06-22 02:04:28 UTC) orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded Duration: 36.095447464s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [..................................X.................X.......] duration: 23.749782242s FAIL (2025-06-22 02:03:43 UTC) orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded Duration: 23.749809528s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [....................X.......................................] duration: 16.278038531s FAIL (2025-06-22 02:03:41 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== oad=64949]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jun 22 01:32:13.674692303 (81h8m49.238015311s) [upload=64949]: Done uploading segments... Jun 22 01:32:13.674727219 (81h8m49.238050233s) [upload=64949]: Upload finished. err: <nil> Jun 22 01:32:47.266407995 (81h9m22.829731002s) [upload=64958]: Starting upload Jun 22 01:32:47.266524767 (81h9m22.829847771s) [upload=64958]: Done starting upload Jun 22 01:32:47.266813109 (81h9m22.830136116s) [upload=64958]: Uploading segments... Jun 22 01:32:47.266969526 (81h9m22.830292526s) [upload=64958]: Got next segment. Inline: false Jun 22 01:32:47.267075475 (81h9m22.830398475s) [upload=64958]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 22 01:32:47.267081679 (81h9m22.830404667s) [upload=64958]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 22 01:32:47.301770382 (81h9m22.865093389s) [seg_pos={0 0}, upload=64958]: Begin upload segment... Jun 22 01:32:47.301823796 (81h9m22.865146799s) [seg_pos={0 0}, upload=64958]: Done begin upload segment. Jun 22 01:32:47.301855055 (81h9m22.865178087s) [noise=true , address=104.36.82.43:6841 , piece_id=PIHZMSTO3CII7KGDLGSAILTDRSLMEBVOLBCRCZJULMCFISXIZWXA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.301902600 (81h9m22.865225606s) [noise=true , address=104.36.82.42:6811 , piece_id=A5BFEHZ3WGGK53IZ7MKI6N4CVXHOORF6L47NR3UFDCAX7LBHRHVQ, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.302234027 (81h9m22.865557029s) [noise=true , address=104.36.82.51:4084 , piece_id=D43FA7BFE54KPCWGDGDME2EGSWM7PONOVPD3DN2XY42MJFOKVDIQ, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.302496354 (81h9m22.865819363s) [noise=true , address=79.127.219.38:4023 , piece_id=BWLRCXXHBMBTNCZEVKKIIS5PB35PDQXQXCLIQB7GBLEIOWWUFBFA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.302577370 (81h9m22.865900378s) [noise=true , address=104.36.82.52:4089 , piece_id=JSOEKLNWA3W3LOVKI3EGGKJYJ6J3UZY6KGABKQPQPXTIOOPMKVHA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.302917829 (81h9m22.866240835s) [noise=true , address=104.36.82.53:4122 , piece_id=HZZQBIKHQWLUE5EHK4KSKJGIUEG7D44RQASUJHCYXX6BICCGX6EA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.303087165 (81h9m22.866410168s) [noise=true , address=77.90.31.14:4689 , piece_id=Q3HDS3BCMYDTODFRMU4XFRDCQK2QZE3ZMXVLUX4M54PDEBS7RR6A, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.303250714 (81h9m22.866573717s) [noise=true , address=79.127.205.234:5601 , piece_id=V7PMXBP2VFWBSZDG4GTLBD435T6F5DT7A43L4GQLQ5BDDTJ4WY6A, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.303579249 (81h9m22.866902255s) [noise=true , address=104.36.82.12:6158 , piece_id=LJZEOJVJCJAR2BRILIJI3ZVDGYMPBBTP2G3HCTN3MHNS4KYDPASQ, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.303599776 (81h9m22.866922784s) [noise=true , address=104.36.82.9:6023 , piece_id=SISFAUU4RPF57OLGWM2BWQWE6GRUSCMORBYHWEXETNAVM2EEMO6Q, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.304005346 (81h9m22.867328346s) [noise=true , address=45.91.116.12:5242 , piece_id=6ABT5IPYGSLFFJMBYMBYRU4EYIXABUAUHG3OVILOHBIUDS3BP27A, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.304388850 (81h9m22.867711851s) [noise=true , address=64.49.13.137:4500 , piece_id=FAE2RXD57IRP5FXA2XR6THXFW27U7IWAZ4JDV5M2WRPXKUGZ4VIA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.304881529 (81h9m22.868204536s) [noise=true , address=79.127.220.101:6048 , piece_id=AN7UQUXB42T2YXEH7ZJRZUDIW47M3RWUEOCL2VEBOGH74BZF3YKA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.305249860 (81h9m22.86857287s) [noise=true , address=104.36.82.54:4165 , piece_id=4CVOABPCWC7J7SRHZB6IN2WJJ6KGWUQ66ELUIZIFA67ETLQ7JEUQ, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.305604108 (81h9m22.868927112s) [noise=true , address=104.36.82.15:6289 , piece_id=GWL3P55BIK4IJEL5YFZKI7JCA4LADYOOHUSIQUNGJEKBJOTKMP5A, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.305930423 (81h9m22.869253425s) [noise=true , address=64.49.13.139:4476 , piece_id=P7UKBNNZGI52B3I6WKTV6MVC262FYNKYYPJUI5G47UO5S4MU7PWA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.306355311 (81h9m22.869678319s) [noise=true , address=104.36.82.55:4213 , piece_id=X4IHS2XSPH52LIMNUYT7KGWMAPUH3TEMJOEEVB3JV5N45SEOH7YA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.306788243 (81h9m22.870111252s) [noise=true , address=104.36.82.22:6539 , piece_id=GHQ2MVSTDGVYLUKLV23D6YWRR4PP7UUYHCUEQ4WWSWEMTBFO3P4A, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.392174770 (81h9m22.955497773s) [upload=64958]: Next returned nil segment Jun 22 01:32:47.392195767 (81h9m22.955518775s) [upload=64958]: Waiting for error group managing segments... Jun 22 01:32:47.480767633 (81h9m23.044090623s) [noise=true , address=104.36.82.22:6539 , piece_id=GHQ2MVSTDGVYLUKLV23D6YWRR4PP7UUYHCUEQ4WWSWEMTBFO3P4A, seg_pos={0 0}, upload=64958]: Done uploading piece. err: ecclient: upload failed (node:1QL6w8DSFR6CxDdnJmDuyVZxp6JU55VKiJofBZRfQi7q31i8Jc, address:104.36.82.22:6539): protocol: storage node overloaded, request limit: 2000 Jun 22 01:32:47.585663142 (81h9m23.148986155s) [noise=true , address=79.127.220.101:6048 , piece_id=AN7UQUXB42T2YXEH7ZJRZUDIW47M3RWUEOCL2VEBOGH74BZF3YKA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:47.640263002 (81h9m23.203586013s) [noise=true , address=45.91.116.12:5242 , piece_id=6ABT5IPYGSLFFJMBYMBYRU4EYIXABUAUHG3OVILOHBIUDS3BP27A, seg_pos={0 0}, upload=64958]: Done uploading piece. err: ecclient: upload failed (node:12JsFL62jgXmza8Md8eQ9k8cyw5V9Vz2RzN4y85ZvgHZjwrq4vQ, address:45.91.116.12:5242): protocol: storage node overloaded, request limit: 2000 Jun 22 01:32:47.984127275 (81h9m23.54745028s) [noise=true , address=64.49.13.137:4500 , piece_id=FAE2RXD57IRP5FXA2XR6THXFW27U7IWAZ4JDV5M2WRPXKUGZ4VIA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:48.100255249 (81h9m23.663578241s) [noise=true , address=77.90.31.14:4689 , piece_id=Q3HDS3BCMYDTODFRMU4XFRDCQK2QZE3ZMXVLUX4M54PDEBS7RR6A, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:48.221764035 (81h9m23.785087034s) [noise=true , address=104.36.82.54:4165 , piece_id=4CVOABPCWC7J7SRHZB6IN2WJJ6KGWUQ66ELUIZIFA67ETLQ7JEUQ, seg_pos={0 0}, upload=64958]: Done uploading piece. err: ecclient: upload failed (node:12wGUMXVEubEtki4qGnjGEsbYFB1iUauZcu3oDRKrCQsJMV3bV2, address:104.36.82.54:4165): protocol: storage node overloaded, request limit: 2000 Jun 22 01:32:48.509996527 (81h9m24.073319547s) [noise=true , address=104.36.82.9:6023 , piece_id=SISFAUU4RPF57OLGWM2BWQWE6GRUSCMORBYHWEXETNAVM2EEMO6Q, seg_pos={0 0}, upload=64958]: Done uploading piece. err: ecclient: upload failed (node:1j2S9tgWpLwmyvXZSq2qKKcbaC8r7LkzfDCaMxCejAni4CST1g, address:104.36.82.9:6023): protocol: storage node overloaded, request limit: 2000 Jun 22 01:32:48.510513169 (81h9m24.073836177s) [noise=true , address=104.36.82.42:6811 , piece_id=A5BFEHZ3WGGK53IZ7MKI6N4CVXHOORF6L47NR3UFDCAX7LBHRHVQ, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:48.565245605 (81h9m24.128568606s) [noise=true , address=79.127.205.234:5601 , piece_id=V7PMXBP2VFWBSZDG4GTLBD435T6F5DT7A43L4GQLQ5BDDTJ4WY6A, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:49.979979824 (81h9m25.543302825s) [noise=true , address=104.36.82.15:6289 , piece_id=GWL3P55BIK4IJEL5YFZKI7JCA4LADYOOHUSIQUNGJEKBJOTKMP5A, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:50.463750280 (81h9m26.02707328s) [noise=true , address=104.36.82.51:4084 , piece_id=D43FA7BFE54KPCWGDGDME2EGSWM7PONOVPD3DN2XY42MJFOKVDIQ, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:51.480599795 (81h9m27.043922798s) [noise=true , address=79.127.219.38:4023 , piece_id=BWLRCXXHBMBTNCZEVKKIIS5PB35PDQXQXCLIQB7GBLEIOWWUFBFA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:52.941200625 (81h9m28.504523627s) [noise=true , address=104.36.82.43:6841 , piece_id=PIHZMSTO3CII7KGDLGSAILTDRSLMEBVOLBCRCZJULMCFISXIZWXA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:58.281803598 (81h9m33.845126612s) [noise=true , address=64.49.13.139:4476 , piece_id=P7UKBNNZGI52B3I6WKTV6MVC262FYNKYYPJUI5G47UO5S4MU7PWA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:33:00.277271877 (81h9m35.840594867s) [noise=true , address=104.36.82.55:4213 , piece_id=X4IHS2XSPH52LIMNUYT7KGWMAPUH3TEMJOEEVB3JV5N45SEOH7YA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:33:09.435339213 (81h9m44.998662206s) [noise=true , address=104.36.82.52:4089 , piece_id=JSOEKLNWA3W3LOVKI3EGGKJYJ6J3UZY6KGABKQPQPXTIOOPMKVHA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:33:31.472868445 (81h10m7.03619143s) [noise=true , address=104.36.82.53:4122 , piece_id=HZZQBIKHQWLUE5EHK4KSKJGIUEG7D44RQASUJHCYXX6BICCGX6EA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:33:47.235404653 (81h10m22.798727642s) [noise=true , address=104.36.82.12:6158 , piece_id=LJZEOJVJCJAR2BRILIJI3ZVDGYMPBBTP2G3HCTN3MHNS4KYDPASQ, seg_pos={0 0}, upload=64958]: Done uploading piece. err: ecclient: upload failed (node:1SjM6Utm2YizeUbrBUD9Z7etE1nLadW5Fa1zYP7MioPeeqhdJ7, address:104.36.82.12:6158): protocol: expected piece hash; context deadline exceeded; EOF Jun 22 01:33:47.235448297 (81h10m22.798771306s) [seg_pos={0 0}, upload=64958]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jun 22 01:33:47.265442393 (81h10m22.828765401s) [upload=64958]: Done uploading segments... Jun 22 01:33:47.265476139 (81h10m22.82879914s) [upload=64958]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration:* 16.278067339s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [...................................X.................X......] duration: 17.514373831s FAIL (2025-06-22 02:03:17 UTC) orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded Duration: 17.514394829s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.....................X......................................] duration: 14.359187955s FAIL (2025-06-22 02:03:15 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== oad=64949]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jun 22 01:32:13.674692303 (81h8m49.238015311s) [upload=64949]: Done uploading segments... Jun 22 01:32:13.674727219 (81h8m49.238050233s) [upload=64949]: Upload finished. err: <nil> Jun 22 01:32:47.266407995 (81h9m22.829731002s) [upload=64958]: Starting upload Jun 22 01:32:47.266524767 (81h9m22.829847771s) [upload=64958]: Done starting upload Jun 22 01:32:47.266813109 (81h9m22.830136116s) [upload=64958]: Uploading segments... Jun 22 01:32:47.266969526 (81h9m22.830292526s) [upload=64958]: Got next segment. Inline: false Jun 22 01:32:47.267075475 (81h9m22.830398475s) [upload=64958]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 22 01:32:47.267081679 (81h9m22.830404667s) [upload=64958]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 22 01:32:47.301770382 (81h9m22.865093389s) [seg_pos={0 0}, upload=64958]: Begin upload segment... Jun 22 01:32:47.301823796 (81h9m22.865146799s) [seg_pos={0 0}, upload=64958]: Done begin upload segment. Jun 22 01:32:47.301855055 (81h9m22.865178087s) [noise=true , address=104.36.82.43:6841 , piece_id=PIHZMSTO3CII7KGDLGSAILTDRSLMEBVOLBCRCZJULMCFISXIZWXA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.301902600 (81h9m22.865225606s) [noise=true , address=104.36.82.42:6811 , piece_id=A5BFEHZ3WGGK53IZ7MKI6N4CVXHOORF6L47NR3UFDCAX7LBHRHVQ, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.302234027 (81h9m22.865557029s) [noise=true , address=104.36.82.51:4084 , piece_id=D43FA7BFE54KPCWGDGDME2EGSWM7PONOVPD3DN2XY42MJFOKVDIQ, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.302496354 (81h9m22.865819363s) [noise=true , address=79.127.219.38:4023 , piece_id=BWLRCXXHBMBTNCZEVKKIIS5PB35PDQXQXCLIQB7GBLEIOWWUFBFA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.302577370 (81h9m22.865900378s) [noise=true , address=104.36.82.52:4089 , piece_id=JSOEKLNWA3W3LOVKI3EGGKJYJ6J3UZY6KGABKQPQPXTIOOPMKVHA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.302917829 (81h9m22.866240835s) [noise=true , address=104.36.82.53:4122 , piece_id=HZZQBIKHQWLUE5EHK4KSKJGIUEG7D44RQASUJHCYXX6BICCGX6EA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.303087165 (81h9m22.866410168s) [noise=true , address=77.90.31.14:4689 , piece_id=Q3HDS3BCMYDTODFRMU4XFRDCQK2QZE3ZMXVLUX4M54PDEBS7RR6A, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.303250714 (81h9m22.866573717s) [noise=true , address=79.127.205.234:5601 , piece_id=V7PMXBP2VFWBSZDG4GTLBD435T6F5DT7A43L4GQLQ5BDDTJ4WY6A, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.303579249 (81h9m22.866902255s) [noise=true , address=104.36.82.12:6158 , piece_id=LJZEOJVJCJAR2BRILIJI3ZVDGYMPBBTP2G3HCTN3MHNS4KYDPASQ, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.303599776 (81h9m22.866922784s) [noise=true , address=104.36.82.9:6023 , piece_id=SISFAUU4RPF57OLGWM2BWQWE6GRUSCMORBYHWEXETNAVM2EEMO6Q, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.304005346 (81h9m22.867328346s) [noise=true , address=45.91.116.12:5242 , piece_id=6ABT5IPYGSLFFJMBYMBYRU4EYIXABUAUHG3OVILOHBIUDS3BP27A, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.304388850 (81h9m22.867711851s) [noise=true , address=64.49.13.137:4500 , piece_id=FAE2RXD57IRP5FXA2XR6THXFW27U7IWAZ4JDV5M2WRPXKUGZ4VIA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.304881529 (81h9m22.868204536s) [noise=true , address=79.127.220.101:6048 , piece_id=AN7UQUXB42T2YXEH7ZJRZUDIW47M3RWUEOCL2VEBOGH74BZF3YKA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.305249860 (81h9m22.86857287s) [noise=true , address=104.36.82.54:4165 , piece_id=4CVOABPCWC7J7SRHZB6IN2WJJ6KGWUQ66ELUIZIFA67ETLQ7JEUQ, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.305604108 (81h9m22.868927112s) [noise=true , address=104.36.82.15:6289 , piece_id=GWL3P55BIK4IJEL5YFZKI7JCA4LADYOOHUSIQUNGJEKBJOTKMP5A, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.305930423 (81h9m22.869253425s) [noise=true , address=64.49.13.139:4476 , piece_id=P7UKBNNZGI52B3I6WKTV6MVC262FYNKYYPJUI5G47UO5S4MU7PWA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.306355311 (81h9m22.869678319s) [noise=true , address=104.36.82.55:4213 , piece_id=X4IHS2XSPH52LIMNUYT7KGWMAPUH3TEMJOEEVB3JV5N45SEOH7YA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.306788243 (81h9m22.870111252s) [noise=true , address=104.36.82.22:6539 , piece_id=GHQ2MVSTDGVYLUKLV23D6YWRR4PP7UUYHCUEQ4WWSWEMTBFO3P4A, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.392174770 (81h9m22.955497773s) [upload=64958]: Next returned nil segment Jun 22 01:32:47.392195767 (81h9m22.955518775s) [upload=64958]: Waiting for error group managing segments... Jun 22 01:32:47.480767633 (81h9m23.044090623s) [noise=true , address=104.36.82.22:6539 , piece_id=GHQ2MVSTDGVYLUKLV23D6YWRR4PP7UUYHCUEQ4WWSWEMTBFO3P4A, seg_pos={0 0}, upload=64958]: Done uploading piece. err: ecclient: upload failed (node:1QL6w8DSFR6CxDdnJmDuyVZxp6JU55VKiJofBZRfQi7q31i8Jc, address:104.36.82.22:6539): protocol: storage node overloaded, request limit: 2000 Jun 22 01:32:47.585663142 (81h9m23.148986155s) [noise=true , address=79.127.220.101:6048 , piece_id=AN7UQUXB42T2YXEH7ZJRZUDIW47M3RWUEOCL2VEBOGH74BZF3YKA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:47.640263002 (81h9m23.203586013s) [noise=true , address=45.91.116.12:5242 , piece_id=6ABT5IPYGSLFFJMBYMBYRU4EYIXABUAUHG3OVILOHBIUDS3BP27A, seg_pos={0 0}, upload=64958]: Done uploading piece. err: ecclient: upload failed (node:12JsFL62jgXmza8Md8eQ9k8cyw5V9Vz2RzN4y85ZvgHZjwrq4vQ, address:45.91.116.12:5242): protocol: storage node overloaded, request limit: 2000 Jun 22 01:32:47.984127275 (81h9m23.54745028s) [noise=true , address=64.49.13.137:4500 , piece_id=FAE2RXD57IRP5FXA2XR6THXFW27U7IWAZ4JDV5M2WRPXKUGZ4VIA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:48.100255249 (81h9m23.663578241s) [noise=true , address=77.90.31.14:4689 , piece_id=Q3HDS3BCMYDTODFRMU4XFRDCQK2QZE3ZMXVLUX4M54PDEBS7RR6A, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:48.221764035 (81h9m23.785087034s) [noise=true , address=104.36.82.54:4165 , piece_id=4CVOABPCWC7J7SRHZB6IN2WJJ6KGWUQ66ELUIZIFA67ETLQ7JEUQ, seg_pos={0 0}, upload=64958]: Done uploading piece. err: ecclient: upload failed (node:12wGUMXVEubEtki4qGnjGEsbYFB1iUauZcu3oDRKrCQsJMV3bV2, address:104.36.82.54:4165): protocol: storage node overloaded, request limit: 2000 Jun 22 01:32:48.509996527 (81h9m24.073319547s) [noise=true , address=104.36.82.9:6023 , piece_id=SISFAUU4RPF57OLGWM2BWQWE6GRUSCMORBYHWEXETNAVM2EEMO6Q, seg_pos={0 0}, upload=64958]: Done uploading piece. err: ecclient: upload failed (node:1j2S9tgWpLwmyvXZSq2qKKcbaC8r7LkzfDCaMxCejAni4CST1g, address:104.36.82.9:6023): protocol: storage node overloaded, request limit: 2000 Jun 22 01:32:48.510513169 (81h9m24.073836177s) [noise=true , address=104.36.82.42:6811 , piece_id=A5BFEHZ3WGGK53IZ7MKI6N4CVXHOORF6L47NR3UFDCAX7LBHRHVQ, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:48.565245605 (81h9m24.128568606s) [noise=true , address=79.127.205.234:5601 , piece_id=V7PMXBP2VFWBSZDG4GTLBD435T6F5DT7A43L4GQLQ5BDDTJ4WY6A, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:49.979979824 (81h9m25.543302825s) [noise=true , address=104.36.82.15:6289 , piece_id=GWL3P55BIK4IJEL5YFZKI7JCA4LADYOOHUSIQUNGJEKBJOTKMP5A, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:50.463750280 (81h9m26.02707328s) [noise=true , address=104.36.82.51:4084 , piece_id=D43FA7BFE54KPCWGDGDME2EGSWM7PONOVPD3DN2XY42MJFOKVDIQ, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:51.480599795 (81h9m27.043922798s) [noise=true , address=79.127.219.38:4023 , piece_id=BWLRCXXHBMBTNCZEVKKIIS5PB35PDQXQXCLIQB7GBLEIOWWUFBFA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:52.941200625 (81h9m28.504523627s) [noise=true , address=104.36.82.43:6841 , piece_id=PIHZMSTO3CII7KGDLGSAILTDRSLMEBVOLBCRCZJULMCFISXIZWXA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:58.281803598 (81h9m33.845126612s) [noise=true , address=64.49.13.139:4476 , piece_id=P7UKBNNZGI52B3I6WKTV6MVC262FYNKYYPJUI5G47UO5S4MU7PWA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:33:00.277271877 (81h9m35.840594867s) [noise=true , address=104.36.82.55:4213 , piece_id=X4IHS2XSPH52LIMNUYT7KGWMAPUH3TEMJOEEVB3JV5N45SEOH7YA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:33:09.435339213 (81h9m44.998662206s) [noise=true , address=104.36.82.52:4089 , piece_id=JSOEKLNWA3W3LOVKI3EGGKJYJ6J3UZY6KGABKQPQPXTIOOPMKVHA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:33:31.472868445 (81h10m7.03619143s) [noise=true , address=104.36.82.53:4122 , piece_id=HZZQBIKHQWLUE5EHK4KSKJGIUEG7D44RQASUJHCYXX6BICCGX6EA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:33:47.235404653 (81h10m22.798727642s) [noise=true , address=104.36.82.12:6158 , piece_id=LJZEOJVJCJAR2BRILIJI3ZVDGYMPBBTP2G3HCTN3MHNS4KYDPASQ, seg_pos={0 0}, upload=64958]: Done uploading piece. err: ecclient: upload failed (node:1SjM6Utm2YizeUbrBUD9Z7etE1nLadW5Fa1zYP7MioPeeqhdJ7, address:104.36.82.12:6158): protocol: expected piece hash; context deadline exceeded; EOF Jun 22 01:33:47.235448297 (81h10m22.798771306s) [seg_pos={0 0}, upload=64958]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jun 22 01:33:47.265442393 (81h10m22.828765401s) [upload=64958]: Done uploading segments... Jun 22 01:33:47.265476139 (81h10m22.82879914s) [upload=64958]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration:* 14.35922073s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [....................................X.................X.....] duration: 17.664571162s FAIL (2025-06-22 02:02:58 UTC) orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded Duration: 17.664605701s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [......................X.....................................] duration: 19.91138906s FAIL (2025-06-22 02:02:31 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== oad=64949]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jun 22 01:32:13.674692303 (81h8m49.238015311s) [upload=64949]: Done uploading segments... Jun 22 01:32:13.674727219 (81h8m49.238050233s) [upload=64949]: Upload finished. err: <nil> Jun 22 01:32:47.266407995 (81h9m22.829731002s) [upload=64958]: Starting upload Jun 22 01:32:47.266524767 (81h9m22.829847771s) [upload=64958]: Done starting upload Jun 22 01:32:47.266813109 (81h9m22.830136116s) [upload=64958]: Uploading segments... Jun 22 01:32:47.266969526 (81h9m22.830292526s) [upload=64958]: Got next segment. Inline: false Jun 22 01:32:47.267075475 (81h9m22.830398475s) [upload=64958]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 22 01:32:47.267081679 (81h9m22.830404667s) [upload=64958]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 22 01:32:47.301770382 (81h9m22.865093389s) [seg_pos={0 0}, upload=64958]: Begin upload segment... Jun 22 01:32:47.301823796 (81h9m22.865146799s) [seg_pos={0 0}, upload=64958]: Done begin upload segment. Jun 22 01:32:47.301855055 (81h9m22.865178087s) [noise=true , address=104.36.82.43:6841 , piece_id=PIHZMSTO3CII7KGDLGSAILTDRSLMEBVOLBCRCZJULMCFISXIZWXA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.301902600 (81h9m22.865225606s) [noise=true , address=104.36.82.42:6811 , piece_id=A5BFEHZ3WGGK53IZ7MKI6N4CVXHOORF6L47NR3UFDCAX7LBHRHVQ, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.302234027 (81h9m22.865557029s) [noise=true , address=104.36.82.51:4084 , piece_id=D43FA7BFE54KPCWGDGDME2EGSWM7PONOVPD3DN2XY42MJFOKVDIQ, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.302496354 (81h9m22.865819363s) [noise=true , address=79.127.219.38:4023 , piece_id=BWLRCXXHBMBTNCZEVKKIIS5PB35PDQXQXCLIQB7GBLEIOWWUFBFA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.302577370 (81h9m22.865900378s) [noise=true , address=104.36.82.52:4089 , piece_id=JSOEKLNWA3W3LOVKI3EGGKJYJ6J3UZY6KGABKQPQPXTIOOPMKVHA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.302917829 (81h9m22.866240835s) [noise=true , address=104.36.82.53:4122 , piece_id=HZZQBIKHQWLUE5EHK4KSKJGIUEG7D44RQASUJHCYXX6BICCGX6EA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.303087165 (81h9m22.866410168s) [noise=true , address=77.90.31.14:4689 , piece_id=Q3HDS3BCMYDTODFRMU4XFRDCQK2QZE3ZMXVLUX4M54PDEBS7RR6A, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.303250714 (81h9m22.866573717s) [noise=true , address=79.127.205.234:5601 , piece_id=V7PMXBP2VFWBSZDG4GTLBD435T6F5DT7A43L4GQLQ5BDDTJ4WY6A, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.303579249 (81h9m22.866902255s) [noise=true , address=104.36.82.12:6158 , piece_id=LJZEOJVJCJAR2BRILIJI3ZVDGYMPBBTP2G3HCTN3MHNS4KYDPASQ, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.303599776 (81h9m22.866922784s) [noise=true , address=104.36.82.9:6023 , piece_id=SISFAUU4RPF57OLGWM2BWQWE6GRUSCMORBYHWEXETNAVM2EEMO6Q, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.304005346 (81h9m22.867328346s) [noise=true , address=45.91.116.12:5242 , piece_id=6ABT5IPYGSLFFJMBYMBYRU4EYIXABUAUHG3OVILOHBIUDS3BP27A, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.304388850 (81h9m22.867711851s) [noise=true , address=64.49.13.137:4500 , piece_id=FAE2RXD57IRP5FXA2XR6THXFW27U7IWAZ4JDV5M2WRPXKUGZ4VIA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.304881529 (81h9m22.868204536s) [noise=true , address=79.127.220.101:6048 , piece_id=AN7UQUXB42T2YXEH7ZJRZUDIW47M3RWUEOCL2VEBOGH74BZF3YKA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.305249860 (81h9m22.86857287s) [noise=true , address=104.36.82.54:4165 , piece_id=4CVOABPCWC7J7SRHZB6IN2WJJ6KGWUQ66ELUIZIFA67ETLQ7JEUQ, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.305604108 (81h9m22.868927112s) [noise=true , address=104.36.82.15:6289 , piece_id=GWL3P55BIK4IJEL5YFZKI7JCA4LADYOOHUSIQUNGJEKBJOTKMP5A, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.305930423 (81h9m22.869253425s) [noise=true , address=64.49.13.139:4476 , piece_id=P7UKBNNZGI52B3I6WKTV6MVC262FYNKYYPJUI5G47UO5S4MU7PWA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.306355311 (81h9m22.869678319s) [noise=true , address=104.36.82.55:4213 , piece_id=X4IHS2XSPH52LIMNUYT7KGWMAPUH3TEMJOEEVB3JV5N45SEOH7YA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.306788243 (81h9m22.870111252s) [noise=true , address=104.36.82.22:6539 , piece_id=GHQ2MVSTDGVYLUKLV23D6YWRR4PP7UUYHCUEQ4WWSWEMTBFO3P4A, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.392174770 (81h9m22.955497773s) [upload=64958]: Next returned nil segment Jun 22 01:32:47.392195767 (81h9m22.955518775s) [upload=64958]: Waiting for error group managing segments... Jun 22 01:32:47.480767633 (81h9m23.044090623s) [noise=true , address=104.36.82.22:6539 , piece_id=GHQ2MVSTDGVYLUKLV23D6YWRR4PP7UUYHCUEQ4WWSWEMTBFO3P4A, seg_pos={0 0}, upload=64958]: Done uploading piece. err: ecclient: upload failed (node:1QL6w8DSFR6CxDdnJmDuyVZxp6JU55VKiJofBZRfQi7q31i8Jc, address:104.36.82.22:6539): protocol: storage node overloaded, request limit: 2000 Jun 22 01:32:47.585663142 (81h9m23.148986155s) [noise=true , address=79.127.220.101:6048 , piece_id=AN7UQUXB42T2YXEH7ZJRZUDIW47M3RWUEOCL2VEBOGH74BZF3YKA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:47.640263002 (81h9m23.203586013s) [noise=true , address=45.91.116.12:5242 , piece_id=6ABT5IPYGSLFFJMBYMBYRU4EYIXABUAUHG3OVILOHBIUDS3BP27A, seg_pos={0 0}, upload=64958]: Done uploading piece. err: ecclient: upload failed (node:12JsFL62jgXmza8Md8eQ9k8cyw5V9Vz2RzN4y85ZvgHZjwrq4vQ, address:45.91.116.12:5242): protocol: storage node overloaded, request limit: 2000 Jun 22 01:32:47.984127275 (81h9m23.54745028s) [noise=true , address=64.49.13.137:4500 , piece_id=FAE2RXD57IRP5FXA2XR6THXFW27U7IWAZ4JDV5M2WRPXKUGZ4VIA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:48.100255249 (81h9m23.663578241s) [noise=true , address=77.90.31.14:4689 , piece_id=Q3HDS3BCMYDTODFRMU4XFRDCQK2QZE3ZMXVLUX4M54PDEBS7RR6A, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:48.221764035 (81h9m23.785087034s) [noise=true , address=104.36.82.54:4165 , piece_id=4CVOABPCWC7J7SRHZB6IN2WJJ6KGWUQ66ELUIZIFA67ETLQ7JEUQ, seg_pos={0 0}, upload=64958]: Done uploading piece. err: ecclient: upload failed (node:12wGUMXVEubEtki4qGnjGEsbYFB1iUauZcu3oDRKrCQsJMV3bV2, address:104.36.82.54:4165): protocol: storage node overloaded, request limit: 2000 Jun 22 01:32:48.509996527 (81h9m24.073319547s) [noise=true , address=104.36.82.9:6023 , piece_id=SISFAUU4RPF57OLGWM2BWQWE6GRUSCMORBYHWEXETNAVM2EEMO6Q, seg_pos={0 0}, upload=64958]: Done uploading piece. err: ecclient: upload failed (node:1j2S9tgWpLwmyvXZSq2qKKcbaC8r7LkzfDCaMxCejAni4CST1g, address:104.36.82.9:6023): protocol: storage node overloaded, request limit: 2000 Jun 22 01:32:48.510513169 (81h9m24.073836177s) [noise=true , address=104.36.82.42:6811 , piece_id=A5BFEHZ3WGGK53IZ7MKI6N4CVXHOORF6L47NR3UFDCAX7LBHRHVQ, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:48.565245605 (81h9m24.128568606s) [noise=true , address=79.127.205.234:5601 , piece_id=V7PMXBP2VFWBSZDG4GTLBD435T6F5DT7A43L4GQLQ5BDDTJ4WY6A, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:49.979979824 (81h9m25.543302825s) [noise=true , address=104.36.82.15:6289 , piece_id=GWL3P55BIK4IJEL5YFZKI7JCA4LADYOOHUSIQUNGJEKBJOTKMP5A, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:50.463750280 (81h9m26.02707328s) [noise=true , address=104.36.82.51:4084 , piece_id=D43FA7BFE54KPCWGDGDME2EGSWM7PONOVPD3DN2XY42MJFOKVDIQ, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:51.480599795 (81h9m27.043922798s) [noise=true , address=79.127.219.38:4023 , piece_id=BWLRCXXHBMBTNCZEVKKIIS5PB35PDQXQXCLIQB7GBLEIOWWUFBFA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:52.941200625 (81h9m28.504523627s) [noise=true , address=104.36.82.43:6841 , piece_id=PIHZMSTO3CII7KGDLGSAILTDRSLMEBVOLBCRCZJULMCFISXIZWXA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:58.281803598 (81h9m33.845126612s) [noise=true , address=64.49.13.139:4476 , piece_id=P7UKBNNZGI52B3I6WKTV6MVC262FYNKYYPJUI5G47UO5S4MU7PWA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:33:00.277271877 (81h9m35.840594867s) [noise=true , address=104.36.82.55:4213 , piece_id=X4IHS2XSPH52LIMNUYT7KGWMAPUH3TEMJOEEVB3JV5N45SEOH7YA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:33:09.435339213 (81h9m44.998662206s) [noise=true , address=104.36.82.52:4089 , piece_id=JSOEKLNWA3W3LOVKI3EGGKJYJ6J3UZY6KGABKQPQPXTIOOPMKVHA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:33:31.472868445 (81h10m7.03619143s) [noise=true , address=104.36.82.53:4122 , piece_id=HZZQBIKHQWLUE5EHK4KSKJGIUEG7D44RQASUJHCYXX6BICCGX6EA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:33:47.235404653 (81h10m22.798727642s) [noise=true , address=104.36.82.12:6158 , piece_id=LJZEOJVJCJAR2BRILIJI3ZVDGYMPBBTP2G3HCTN3MHNS4KYDPASQ, seg_pos={0 0}, upload=64958]: Done uploading piece. err: ecclient: upload failed (node:1SjM6Utm2YizeUbrBUD9Z7etE1nLadW5Fa1zYP7MioPeeqhdJ7, address:104.36.82.12:6158): protocol: expected piece hash; context deadline exceeded; EOF Jun 22 01:33:47.235448297 (81h10m22.798771306s) [seg_pos={0 0}, upload=64958]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jun 22 01:33:47.265442393 (81h10m22.828765401s) [upload=64958]: Done uploading segments... Jun 22 01:33:47.265476139 (81h10m22.82879914s) [upload=64958]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration:* 19.911432232s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [.....................................X.................X....] duration: 23.089343793s FAIL (2025-06-22 02:02:31 UTC) orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded Duration: 23.089392004s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.......................X....................................] duration: 19.851933728s FAIL (2025-06-22 02:01:45 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== oad=64949]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jun 22 01:32:13.674692303 (81h8m49.238015311s) [upload=64949]: Done uploading segments... Jun 22 01:32:13.674727219 (81h8m49.238050233s) [upload=64949]: Upload finished. err: <nil> Jun 22 01:32:47.266407995 (81h9m22.829731002s) [upload=64958]: Starting upload Jun 22 01:32:47.266524767 (81h9m22.829847771s) [upload=64958]: Done starting upload Jun 22 01:32:47.266813109 (81h9m22.830136116s) [upload=64958]: Uploading segments... Jun 22 01:32:47.266969526 (81h9m22.830292526s) [upload=64958]: Got next segment. Inline: false Jun 22 01:32:47.267075475 (81h9m22.830398475s) [upload=64958]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 22 01:32:47.267081679 (81h9m22.830404667s) [upload=64958]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 22 01:32:47.301770382 (81h9m22.865093389s) [seg_pos={0 0}, upload=64958]: Begin upload segment... Jun 22 01:32:47.301823796 (81h9m22.865146799s) [seg_pos={0 0}, upload=64958]: Done begin upload segment. Jun 22 01:32:47.301855055 (81h9m22.865178087s) [noise=true , address=104.36.82.43:6841 , piece_id=PIHZMSTO3CII7KGDLGSAILTDRSLMEBVOLBCRCZJULMCFISXIZWXA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.301902600 (81h9m22.865225606s) [noise=true , address=104.36.82.42:6811 , piece_id=A5BFEHZ3WGGK53IZ7MKI6N4CVXHOORF6L47NR3UFDCAX7LBHRHVQ, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.302234027 (81h9m22.865557029s) [noise=true , address=104.36.82.51:4084 , piece_id=D43FA7BFE54KPCWGDGDME2EGSWM7PONOVPD3DN2XY42MJFOKVDIQ, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.302496354 (81h9m22.865819363s) [noise=true , address=79.127.219.38:4023 , piece_id=BWLRCXXHBMBTNCZEVKKIIS5PB35PDQXQXCLIQB7GBLEIOWWUFBFA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.302577370 (81h9m22.865900378s) [noise=true , address=104.36.82.52:4089 , piece_id=JSOEKLNWA3W3LOVKI3EGGKJYJ6J3UZY6KGABKQPQPXTIOOPMKVHA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.302917829 (81h9m22.866240835s) [noise=true , address=104.36.82.53:4122 , piece_id=HZZQBIKHQWLUE5EHK4KSKJGIUEG7D44RQASUJHCYXX6BICCGX6EA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.303087165 (81h9m22.866410168s) [noise=true , address=77.90.31.14:4689 , piece_id=Q3HDS3BCMYDTODFRMU4XFRDCQK2QZE3ZMXVLUX4M54PDEBS7RR6A, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.303250714 (81h9m22.866573717s) [noise=true , address=79.127.205.234:5601 , piece_id=V7PMXBP2VFWBSZDG4GTLBD435T6F5DT7A43L4GQLQ5BDDTJ4WY6A, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.303579249 (81h9m22.866902255s) [noise=true , address=104.36.82.12:6158 , piece_id=LJZEOJVJCJAR2BRILIJI3ZVDGYMPBBTP2G3HCTN3MHNS4KYDPASQ, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.303599776 (81h9m22.866922784s) [noise=true , address=104.36.82.9:6023 , piece_id=SISFAUU4RPF57OLGWM2BWQWE6GRUSCMORBYHWEXETNAVM2EEMO6Q, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.304005346 (81h9m22.867328346s) [noise=true , address=45.91.116.12:5242 , piece_id=6ABT5IPYGSLFFJMBYMBYRU4EYIXABUAUHG3OVILOHBIUDS3BP27A, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.304388850 (81h9m22.867711851s) [noise=true , address=64.49.13.137:4500 , piece_id=FAE2RXD57IRP5FXA2XR6THXFW27U7IWAZ4JDV5M2WRPXKUGZ4VIA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.304881529 (81h9m22.868204536s) [noise=true , address=79.127.220.101:6048 , piece_id=AN7UQUXB42T2YXEH7ZJRZUDIW47M3RWUEOCL2VEBOGH74BZF3YKA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.305249860 (81h9m22.86857287s) [noise=true , address=104.36.82.54:4165 , piece_id=4CVOABPCWC7J7SRHZB6IN2WJJ6KGWUQ66ELUIZIFA67ETLQ7JEUQ, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.305604108 (81h9m22.868927112s) [noise=true , address=104.36.82.15:6289 , piece_id=GWL3P55BIK4IJEL5YFZKI7JCA4LADYOOHUSIQUNGJEKBJOTKMP5A, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.305930423 (81h9m22.869253425s) [noise=true , address=64.49.13.139:4476 , piece_id=P7UKBNNZGI52B3I6WKTV6MVC262FYNKYYPJUI5G47UO5S4MU7PWA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.306355311 (81h9m22.869678319s) [noise=true , address=104.36.82.55:4213 , piece_id=X4IHS2XSPH52LIMNUYT7KGWMAPUH3TEMJOEEVB3JV5N45SEOH7YA, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.306788243 (81h9m22.870111252s) [noise=true , address=104.36.82.22:6539 , piece_id=GHQ2MVSTDGVYLUKLV23D6YWRR4PP7UUYHCUEQ4WWSWEMTBFO3P4A, seg_pos={0 0}, upload=64958]: Uploading piece... Jun 22 01:32:47.392174770 (81h9m22.955497773s) [upload=64958]: Next returned nil segment Jun 22 01:32:47.392195767 (81h9m22.955518775s) [upload=64958]: Waiting for error group managing segments... Jun 22 01:32:47.480767633 (81h9m23.044090623s) [noise=true , address=104.36.82.22:6539 , piece_id=GHQ2MVSTDGVYLUKLV23D6YWRR4PP7UUYHCUEQ4WWSWEMTBFO3P4A, seg_pos={0 0}, upload=64958]: Done uploading piece. err: ecclient: upload failed (node:1QL6w8DSFR6CxDdnJmDuyVZxp6JU55VKiJofBZRfQi7q31i8Jc, address:104.36.82.22:6539): protocol: storage node overloaded, request limit: 2000 Jun 22 01:32:47.585663142 (81h9m23.148986155s) [noise=true , address=79.127.220.101:6048 , piece_id=AN7UQUXB42T2YXEH7ZJRZUDIW47M3RWUEOCL2VEBOGH74BZF3YKA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:47.640263002 (81h9m23.203586013s) [noise=true , address=45.91.116.12:5242 , piece_id=6ABT5IPYGSLFFJMBYMBYRU4EYIXABUAUHG3OVILOHBIUDS3BP27A, seg_pos={0 0}, upload=64958]: Done uploading piece. err: ecclient: upload failed (node:12JsFL62jgXmza8Md8eQ9k8cyw5V9Vz2RzN4y85ZvgHZjwrq4vQ, address:45.91.116.12:5242): protocol: storage node overloaded, request limit: 2000 Jun 22 01:32:47.984127275 (81h9m23.54745028s) [noise=true , address=64.49.13.137:4500 , piece_id=FAE2RXD57IRP5FXA2XR6THXFW27U7IWAZ4JDV5M2WRPXKUGZ4VIA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:48.100255249 (81h9m23.663578241s) [noise=true , address=77.90.31.14:4689 , piece_id=Q3HDS3BCMYDTODFRMU4XFRDCQK2QZE3ZMXVLUX4M54PDEBS7RR6A, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:48.221764035 (81h9m23.785087034s) [noise=true , address=104.36.82.54:4165 , piece_id=4CVOABPCWC7J7SRHZB6IN2WJJ6KGWUQ66ELUIZIFA67ETLQ7JEUQ, seg_pos={0 0}, upload=64958]: Done uploading piece. err: ecclient: upload failed (node:12wGUMXVEubEtki4qGnjGEsbYFB1iUauZcu3oDRKrCQsJMV3bV2, address:104.36.82.54:4165): protocol: storage node overloaded, request limit: 2000 Jun 22 01:32:48.509996527 (81h9m24.073319547s) [noise=true , address=104.36.82.9:6023 , piece_id=SISFAUU4RPF57OLGWM2BWQWE6GRUSCMORBYHWEXETNAVM2EEMO6Q, seg_pos={0 0}, upload=64958]: Done uploading piece. err: ecclient: upload failed (node:1j2S9tgWpLwmyvXZSq2qKKcbaC8r7LkzfDCaMxCejAni4CST1g, address:104.36.82.9:6023): protocol: storage node overloaded, request limit: 2000 Jun 22 01:32:48.510513169 (81h9m24.073836177s) [noise=true , address=104.36.82.42:6811 , piece_id=A5BFEHZ3WGGK53IZ7MKI6N4CVXHOORF6L47NR3UFDCAX7LBHRHVQ, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:48.565245605 (81h9m24.128568606s) [noise=true , address=79.127.205.234:5601 , piece_id=V7PMXBP2VFWBSZDG4GTLBD435T6F5DT7A43L4GQLQ5BDDTJ4WY6A, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:49.979979824 (81h9m25.543302825s) [noise=true , address=104.36.82.15:6289 , piece_id=GWL3P55BIK4IJEL5YFZKI7JCA4LADYOOHUSIQUNGJEKBJOTKMP5A, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:50.463750280 (81h9m26.02707328s) [noise=true , address=104.36.82.51:4084 , piece_id=D43FA7BFE54KPCWGDGDME2EGSWM7PONOVPD3DN2XY42MJFOKVDIQ, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:51.480599795 (81h9m27.043922798s) [noise=true , address=79.127.219.38:4023 , piece_id=BWLRCXXHBMBTNCZEVKKIIS5PB35PDQXQXCLIQB7GBLEIOWWUFBFA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:52.941200625 (81h9m28.504523627s) [noise=true , address=104.36.82.43:6841 , piece_id=PIHZMSTO3CII7KGDLGSAILTDRSLMEBVOLBCRCZJULMCFISXIZWXA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:32:58.281803598 (81h9m33.845126612s) [noise=true , address=64.49.13.139:4476 , piece_id=P7UKBNNZGI52B3I6WKTV6MVC262FYNKYYPJUI5G47UO5S4MU7PWA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:33:00.277271877 (81h9m35.840594867s) [noise=true , address=104.36.82.55:4213 , piece_id=X4IHS2XSPH52LIMNUYT7KGWMAPUH3TEMJOEEVB3JV5N45SEOH7YA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:33:09.435339213 (81h9m44.998662206s) [noise=true , address=104.36.82.52:4089 , piece_id=JSOEKLNWA3W3LOVKI3EGGKJYJ6J3UZY6KGABKQPQPXTIOOPMKVHA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:33:31.472868445 (81h10m7.03619143s) [noise=true , address=104.36.82.53:4122 , piece_id=HZZQBIKHQWLUE5EHK4KSKJGIUEG7D44RQASUJHCYXX6BICCGX6EA, seg_pos={0 0}, upload=64958]: Done uploading piece. err: <nil> Jun 22 01:33:47.235404653 (81h10m22.798727642s) [noise=true , address=104.36.82.12:6158 , piece_id=LJZEOJVJCJAR2BRILIJI3ZVDGYMPBBTP2G3HCTN3MHNS4KYDPASQ, seg_pos={0 0}, upload=64958]: Done uploading piece. err: ecclient: upload failed (node:1SjM6Utm2YizeUbrBUD9Z7etE1nLadW5Fa1zYP7MioPeeqhdJ7, address:104.36.82.12:6158): protocol: expected piece hash; context deadline exceeded; EOF Jun 22 01:33:47.235448297 (81h10m22.798771306s) [seg_pos={0 0}, upload=64958]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jun 22 01:33:47.265442393 (81h10m22.828765401s) [upload=64958]: Done uploading segments... Jun 22 01:33:47.265476139 (81h10m22.82879914s) [upload=64958]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration:* 19.851968099s core uplink us-select-1 us1-placement-20

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)