Orbiter History

checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [...........................................X................] duration: 30.890411707s FAIL (2025-06-21 21:40:55 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.890433772s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [............................................X...............] duration: 19.708629162s FAIL (2025-06-21 21:40:04 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 19.708650439s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.............................................X..............] duration: 17.88233797s FAIL (2025-06-21 21:39:18 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 17.882360521s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [..............................................X.............] duration: 20.253086236s FAIL (2025-06-21 21:38:54 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 20.253103216s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [...............................................X............] duration: 24.445808496s FAIL (2025-06-21 21:38:33 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 24.445835363s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [................................................X...........] duration: 19.102426229s FAIL (2025-06-21 21:37:46 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 19.102446993s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.................................................X..........] duration: 24.07484607s FAIL (2025-06-21 21:36:46 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 24.074866048s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [..................................................X.........] duration: 21.1767602s FAIL (2025-06-21 21:35:40 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 21.176814721s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [...................................................X........] duration: 21.438284779s FAIL (2025-06-21 21:34:29 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 21.438311126s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [....................................................X.......] duration: 17.709419871s FAIL (2025-06-21 21:33:45 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 17.709462617s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.....................................................X......] duration: 35.845665146s FAIL (2025-06-21 21:33:08 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 35.846037887s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [......................................................X.....] duration: 21.268450043s FAIL (2025-06-21 21:31:44 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 21.268529608s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.......................................................X....] duration: 16.276172785s FAIL (2025-06-21 21:30:52 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 16.27621073s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [........................................................X...] duration: 21.366172705s FAIL (2025-06-21 21:30:35 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 21.366192539s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.........................................................X..] duration: 14.442375472s FAIL (2025-06-21 21:30:10 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 14.442401506s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [..........................................................X.] duration: 37.96354314s FAIL (2025-06-21 21:29:24 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 37.963571553s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [...........................................................X] duration: 1m0.000599879s FAIL (2025-06-21 21:28:46 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1m0.000646345s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [X...........................................................] duration: 20.43711959s FAIL (2025-06-21 20:55:13 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== pload=60693]: Starting upload Jun 21 20:10:05.602866542 (75h46m41.166189557s) [upload=60693]: Done starting upload Jun 21 20:10:05.603116545 (75h46m41.166439547s) [upload=60693]: Uploading segments... Jun 21 20:10:05.603495389 (75h46m41.166818397s) [upload=60693]: Got next segment. Inline: false Jun 21 20:10:05.603566566 (75h46m41.166889558s) [upload=60693]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 21 20:10:05.603573280 (75h46m41.166896286s) [upload=60693]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 21 20:10:05.639130162 (75h46m41.202453164s) [seg_pos={0 0}, upload=60693]: Begin upload segment... Jun 21 20:10:05.639231070 (75h46m41.202554057s) [seg_pos={0 0}, upload=60693]: Done begin upload segment. Jun 21 20:10:05.639292724 (75h46m41.202615728s) [noise=true , address=104.36.82.56:4253 , piece_id=5PMVGLZETMXCQ77K3I6PXACRIMBGR2CV5JGUSU5KUMYFNXVGA2GA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.639338066 (75h46m41.202661071s) [noise=true , address=104.36.82.53:4126 , piece_id=2MKRQJEPGDJKDM7VWY5M4M6NORILAIEYYATJ7HVZEOBOMXOI2LBA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.639416264 (75h46m41.202739267s) [noise=true , address=104.36.82.5:5869 , piece_id=HAEHXXZASV3DHT3QN7NTZOS6XZKHNUWBKCHWJJWSTJKXAKJKKBAA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.639873098 (75h46m41.203196115s) [noise=true , address=104.36.82.52:4111 , piece_id=HB7QAWN6CGNIIT4HD2GZ7JDQP5NXIJ5I2USBM66YOKTYENNOWG5A, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.639887296 (75h46m41.203210304s) [noise=true , address=104.36.82.10:6087 , piece_id=5FQG7BPZC373PFB54HYXSG2SVKOBBBSGZCLC4XDJEYQFOLQY74MA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.640266623 (75h46m41.203589616s) [noise=true , address=104.36.82.13:6196 , piece_id=BCHMQO5KKT5BLNWQGOHH3V43LGSE3BUQCI2SY4TTO5M5GQNOCTMQ, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.640352274 (75h46m41.203675286s) [noise=true , address=104.36.82.37:6663 , piece_id=X7LWYIIHJFU2T2PHYR3R5TNT22UUPHEHQWZH7ZJMLS4AH4PJ6AIA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.640659767 (75h46m41.203982779s) [noise=true , address=104.36.82.42:6807 , piece_id=ZBQZPXUDJ4VBCOSVKWAJ6SK6UD7SLPON7Y4SSCK2DQ5STFG4XCNQ, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.640961701 (75h46m41.20428471s) [noise=true , address=45.91.116.6:4982 , piece_id=VP45R6KXNH5U5SGUQUMAPIBY7KFDPBIR3S3VRIX27EAT6VLFBZ7Q, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.641159578 (75h46m41.204482589s) [noise=true , address=104.36.82.9:6039 , piece_id=I4AB6N27MYYUHWAZLJJUIBFBLOQFHDIM3YLYWPVRPNUV7VMYS2SA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.641588126 (75h46m41.204911136s) [noise=true , address=77.90.31.19:4853 , piece_id=VFEON5T2KSWM2KSLTVRZTXYL42JWENZYUEUPG3LMODGZWBMVGGPQ, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.641863297 (75h46m41.205186305s) [noise=true , address=77.90.31.12:4603 , piece_id=BMUIC6KNGEDL37436GLSXW5XOO6VYOPNOSFGQCYVHSENVI2PJ7KQ, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.642124427 (75h46m41.205447437s) [noise=true , address=104.36.82.49:4982 , piece_id=UNLV5UDNWULEIPE4QSQHHZQPK3BQKFRB5T3H6GP7ZQ5QGSNK5V6A, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.642285917 (75h46m41.205608925s) [noise=true , address=104.36.82.54:4171 , piece_id=MUMABKARK4KKUKWW6XDXCQVT6O7SLFHGHE7COWC7QIYGARAA2D5A, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.642656750 (75h46m41.205979758s) [noise=true , address=104.36.82.14:6232 , piece_id=A2ZLUUZKRTBJRJ267QSK36C3LFZV7FPAYDDAXGIWDTNEGNV6MRRA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.643565583 (75h46m41.206888597s) [noise=true , address=79.127.205.245:4231 , piece_id=EJK46XUUVI5NWOAQJEW4QIJG3KRXZHB4YOQUJRCS2MIOSWKGTNLQ, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.644620291 (75h46m41.207943303s) [noise=true , address=77.90.31.14:4670 , piece_id=7FNANMUKPM3N6YWRAOAIM4WJZJUTNF7ZBI6ZUBNEPGNGMFWOE24A, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.648842639 (75h46m41.212165631s) [noise=true , address=104.36.82.55:4224 , piece_id=WRVNGSJV5BX4GJERLXNPAS7J7BZ5WFTM7R5H257YRCG5SP2HIYMA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.825828239 (75h46m41.389151241s) [noise=true , address=104.36.82.53:4126 , piece_id=2MKRQJEPGDJKDM7VWY5M4M6NORILAIEYYATJ7HVZEOBOMXOI2LBA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: ecclient: upload failed (node:12dMWFrq2NMpnMzuPAphWerntC5kkrcnzRJZBA116uzJzyvk6fH, address:104.36.82.53:4126): protocol: storage node overloaded, request limit: 2000 Jun 21 20:10:05.834666043 (75h46m41.397989033s) [upload=60693]: Next returned nil segment Jun 21 20:10:05.834686295 (75h46m41.398009302s) [upload=60693]: Waiting for error group managing segments... Jun 21 20:10:05.877010488 (75h46m41.440333486s) [noise=true , address=45.91.116.6:4982 , piece_id=VP45R6KXNH5U5SGUQUMAPIBY7KFDPBIR3S3VRIX27EAT6VLFBZ7Q, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:05.886336964 (75h46m41.449659967s) [noise=true , address=104.36.82.9:6039 , piece_id=I4AB6N27MYYUHWAZLJJUIBFBLOQFHDIM3YLYWPVRPNUV7VMYS2SA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: ecclient: upload failed (node:12P11WVW9mVXw5ra2gZ3Rezon6wZi8GrN5CchQN3MiVU4ts3L6V, address:104.36.82.9:6039): protocol: storage node overloaded, request limit: 2000 Jun 21 20:10:08.064514542 (75h46m43.627837553s) [noise=true , address=77.90.31.14:4670 , piece_id=7FNANMUKPM3N6YWRAOAIM4WJZJUTNF7ZBI6ZUBNEPGNGMFWOE24A, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:08.099098391 (75h46m43.662421396s) [noise=true , address=79.127.205.245:4231 , piece_id=EJK46XUUVI5NWOAQJEW4QIJG3KRXZHB4YOQUJRCS2MIOSWKGTNLQ, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:08.489461749 (75h46m44.052784752s) [noise=true , address=104.36.82.42:6807 , piece_id=ZBQZPXUDJ4VBCOSVKWAJ6SK6UD7SLPON7Y4SSCK2DQ5STFG4XCNQ, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:08.650922093 (75h46m44.214245093s) [noise=true , address=77.90.31.19:4853 , piece_id=VFEON5T2KSWM2KSLTVRZTXYL42JWENZYUEUPG3LMODGZWBMVGGPQ, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:09.917746439 (75h46m45.481069441s) [noise=true , address=104.36.82.13:6196 , piece_id=BCHMQO5KKT5BLNWQGOHH3V43LGSE3BUQCI2SY4TTO5M5GQNOCTMQ, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:10.830722215 (75h46m46.394045216s) [noise=true , address=104.36.82.5:5869 , piece_id=HAEHXXZASV3DHT3QN7NTZOS6XZKHNUWBKCHWJJWSTJKXAKJKKBAA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:12.738783450 (75h46m48.302106452s) [noise=true , address=77.90.31.12:4603 , piece_id=BMUIC6KNGEDL37436GLSXW5XOO6VYOPNOSFGQCYVHSENVI2PJ7KQ, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:15.460927713 (75h46m51.024250723s) [noise=true , address=104.36.82.52:4111 , piece_id=HB7QAWN6CGNIIT4HD2GZ7JDQP5NXIJ5I2USBM66YOKTYENNOWG5A, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:18.393817968 (75h46m53.95714096s) [noise=true , address=104.36.82.55:4224 , piece_id=WRVNGSJV5BX4GJERLXNPAS7J7BZ5WFTM7R5H257YRCG5SP2HIYMA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:19.549174143 (75h46m55.112497151s) [noise=true , address=104.36.82.56:4253 , piece_id=5PMVGLZETMXCQ77K3I6PXACRIMBGR2CV5JGUSU5KUMYFNXVGA2GA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:22.511944593 (75h46m58.075267584s) [noise=true , address=104.36.82.37:6663 , piece_id=X7LWYIIHJFU2T2PHYR3R5TNT22UUPHEHQWZH7ZJMLS4AH4PJ6AIA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:48.027197558 (75h47m23.590520562s) [noise=true , address=104.36.82.14:6232 , piece_id=A2ZLUUZKRTBJRJ267QSK36C3LFZV7FPAYDDAXGIWDTNEGNV6MRRA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:11:05.567692582 (75h47m41.131015574s) [noise=true , address=104.36.82.49:4982 , piece_id=UNLV5UDNWULEIPE4QSQHHZQPK3BQKFRB5T3H6GP7ZQ5QGSNK5V6A, seg_pos={0 0}, upload=60693]: Done uploading piece. err: ecclient: upload failed (node:1NxoYJXCXmTvQncbkXt6yH7mhEBBrzVdzhgL5J6W5TD4N3pkaG, address:104.36.82.49:4982): protocol: expected piece hash; context deadline exceeded; EOF Jun 21 20:11:05.567759198 (75h47m41.131082195s) [noise=true , address=104.36.82.54:4171 , piece_id=MUMABKARK4KKUKWW6XDXCQVT6O7SLFHGHE7COWC7QIYGARAA2D5A, seg_pos={0 0}, upload=60693]: Done uploading piece. err: ecclient: upload failed (node:125SMhDsEZruNu6LkmQn2M7R6WzvXTdk86DRsKf522u8BNiANq3, address:104.36.82.54:4171): protocol: expected piece hash; context deadline exceeded; EOF Jun 21 20:11:05.567869019 (75h47m41.131192033s) [noise=true , address=104.36.82.10:6087 , piece_id=5FQG7BPZC373PFB54HYXSG2SVKOBBBSGZCLC4XDJEYQFOLQY74MA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: ecclient: upload failed (node:1S9k5eejJcxkiG5pXjjBZm18uP8k5jCGfWyeW2QQxmuRU8ZzRA, address:104.36.82.10:6087): protocol: write tcp 159.223.143.39:38550->104.36.82.10:6087: use of closed network connection; write tcp 159.223.143.39:38550->104.36.82.10:6087: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:38550->104.36.82.10:6087: use of closed network connection Jun 21 20:11:05.567932150 (75h47m41.131255161s) [seg_pos={0 0}, upload=60693]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jun 21 20:11:05.593626175 (75h47m41.156949181s) [upload=60693]: Done uploading segments... Jun 21 20:11:05.593657274 (75h47m41.156980265s) [upload=60693]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 20.437153538s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.X..........................................................] duration: 13.06232196s FAIL (2025-06-21 20:54:19 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== pload=60693]: Starting upload Jun 21 20:10:05.602866542 (75h46m41.166189557s) [upload=60693]: Done starting upload Jun 21 20:10:05.603116545 (75h46m41.166439547s) [upload=60693]: Uploading segments... Jun 21 20:10:05.603495389 (75h46m41.166818397s) [upload=60693]: Got next segment. Inline: false Jun 21 20:10:05.603566566 (75h46m41.166889558s) [upload=60693]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 21 20:10:05.603573280 (75h46m41.166896286s) [upload=60693]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 21 20:10:05.639130162 (75h46m41.202453164s) [seg_pos={0 0}, upload=60693]: Begin upload segment... Jun 21 20:10:05.639231070 (75h46m41.202554057s) [seg_pos={0 0}, upload=60693]: Done begin upload segment. Jun 21 20:10:05.639292724 (75h46m41.202615728s) [noise=true , address=104.36.82.56:4253 , piece_id=5PMVGLZETMXCQ77K3I6PXACRIMBGR2CV5JGUSU5KUMYFNXVGA2GA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.639338066 (75h46m41.202661071s) [noise=true , address=104.36.82.53:4126 , piece_id=2MKRQJEPGDJKDM7VWY5M4M6NORILAIEYYATJ7HVZEOBOMXOI2LBA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.639416264 (75h46m41.202739267s) [noise=true , address=104.36.82.5:5869 , piece_id=HAEHXXZASV3DHT3QN7NTZOS6XZKHNUWBKCHWJJWSTJKXAKJKKBAA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.639873098 (75h46m41.203196115s) [noise=true , address=104.36.82.52:4111 , piece_id=HB7QAWN6CGNIIT4HD2GZ7JDQP5NXIJ5I2USBM66YOKTYENNOWG5A, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.639887296 (75h46m41.203210304s) [noise=true , address=104.36.82.10:6087 , piece_id=5FQG7BPZC373PFB54HYXSG2SVKOBBBSGZCLC4XDJEYQFOLQY74MA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.640266623 (75h46m41.203589616s) [noise=true , address=104.36.82.13:6196 , piece_id=BCHMQO5KKT5BLNWQGOHH3V43LGSE3BUQCI2SY4TTO5M5GQNOCTMQ, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.640352274 (75h46m41.203675286s) [noise=true , address=104.36.82.37:6663 , piece_id=X7LWYIIHJFU2T2PHYR3R5TNT22UUPHEHQWZH7ZJMLS4AH4PJ6AIA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.640659767 (75h46m41.203982779s) [noise=true , address=104.36.82.42:6807 , piece_id=ZBQZPXUDJ4VBCOSVKWAJ6SK6UD7SLPON7Y4SSCK2DQ5STFG4XCNQ, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.640961701 (75h46m41.20428471s) [noise=true , address=45.91.116.6:4982 , piece_id=VP45R6KXNH5U5SGUQUMAPIBY7KFDPBIR3S3VRIX27EAT6VLFBZ7Q, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.641159578 (75h46m41.204482589s) [noise=true , address=104.36.82.9:6039 , piece_id=I4AB6N27MYYUHWAZLJJUIBFBLOQFHDIM3YLYWPVRPNUV7VMYS2SA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.641588126 (75h46m41.204911136s) [noise=true , address=77.90.31.19:4853 , piece_id=VFEON5T2KSWM2KSLTVRZTXYL42JWENZYUEUPG3LMODGZWBMVGGPQ, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.641863297 (75h46m41.205186305s) [noise=true , address=77.90.31.12:4603 , piece_id=BMUIC6KNGEDL37436GLSXW5XOO6VYOPNOSFGQCYVHSENVI2PJ7KQ, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.642124427 (75h46m41.205447437s) [noise=true , address=104.36.82.49:4982 , piece_id=UNLV5UDNWULEIPE4QSQHHZQPK3BQKFRB5T3H6GP7ZQ5QGSNK5V6A, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.642285917 (75h46m41.205608925s) [noise=true , address=104.36.82.54:4171 , piece_id=MUMABKARK4KKUKWW6XDXCQVT6O7SLFHGHE7COWC7QIYGARAA2D5A, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.642656750 (75h46m41.205979758s) [noise=true , address=104.36.82.14:6232 , piece_id=A2ZLUUZKRTBJRJ267QSK36C3LFZV7FPAYDDAXGIWDTNEGNV6MRRA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.643565583 (75h46m41.206888597s) [noise=true , address=79.127.205.245:4231 , piece_id=EJK46XUUVI5NWOAQJEW4QIJG3KRXZHB4YOQUJRCS2MIOSWKGTNLQ, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.644620291 (75h46m41.207943303s) [noise=true , address=77.90.31.14:4670 , piece_id=7FNANMUKPM3N6YWRAOAIM4WJZJUTNF7ZBI6ZUBNEPGNGMFWOE24A, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.648842639 (75h46m41.212165631s) [noise=true , address=104.36.82.55:4224 , piece_id=WRVNGSJV5BX4GJERLXNPAS7J7BZ5WFTM7R5H257YRCG5SP2HIYMA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.825828239 (75h46m41.389151241s) [noise=true , address=104.36.82.53:4126 , piece_id=2MKRQJEPGDJKDM7VWY5M4M6NORILAIEYYATJ7HVZEOBOMXOI2LBA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: ecclient: upload failed (node:12dMWFrq2NMpnMzuPAphWerntC5kkrcnzRJZBA116uzJzyvk6fH, address:104.36.82.53:4126): protocol: storage node overloaded, request limit: 2000 Jun 21 20:10:05.834666043 (75h46m41.397989033s) [upload=60693]: Next returned nil segment Jun 21 20:10:05.834686295 (75h46m41.398009302s) [upload=60693]: Waiting for error group managing segments... Jun 21 20:10:05.877010488 (75h46m41.440333486s) [noise=true , address=45.91.116.6:4982 , piece_id=VP45R6KXNH5U5SGUQUMAPIBY7KFDPBIR3S3VRIX27EAT6VLFBZ7Q, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:05.886336964 (75h46m41.449659967s) [noise=true , address=104.36.82.9:6039 , piece_id=I4AB6N27MYYUHWAZLJJUIBFBLOQFHDIM3YLYWPVRPNUV7VMYS2SA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: ecclient: upload failed (node:12P11WVW9mVXw5ra2gZ3Rezon6wZi8GrN5CchQN3MiVU4ts3L6V, address:104.36.82.9:6039): protocol: storage node overloaded, request limit: 2000 Jun 21 20:10:08.064514542 (75h46m43.627837553s) [noise=true , address=77.90.31.14:4670 , piece_id=7FNANMUKPM3N6YWRAOAIM4WJZJUTNF7ZBI6ZUBNEPGNGMFWOE24A, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:08.099098391 (75h46m43.662421396s) [noise=true , address=79.127.205.245:4231 , piece_id=EJK46XUUVI5NWOAQJEW4QIJG3KRXZHB4YOQUJRCS2MIOSWKGTNLQ, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:08.489461749 (75h46m44.052784752s) [noise=true , address=104.36.82.42:6807 , piece_id=ZBQZPXUDJ4VBCOSVKWAJ6SK6UD7SLPON7Y4SSCK2DQ5STFG4XCNQ, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:08.650922093 (75h46m44.214245093s) [noise=true , address=77.90.31.19:4853 , piece_id=VFEON5T2KSWM2KSLTVRZTXYL42JWENZYUEUPG3LMODGZWBMVGGPQ, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:09.917746439 (75h46m45.481069441s) [noise=true , address=104.36.82.13:6196 , piece_id=BCHMQO5KKT5BLNWQGOHH3V43LGSE3BUQCI2SY4TTO5M5GQNOCTMQ, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:10.830722215 (75h46m46.394045216s) [noise=true , address=104.36.82.5:5869 , piece_id=HAEHXXZASV3DHT3QN7NTZOS6XZKHNUWBKCHWJJWSTJKXAKJKKBAA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:12.738783450 (75h46m48.302106452s) [noise=true , address=77.90.31.12:4603 , piece_id=BMUIC6KNGEDL37436GLSXW5XOO6VYOPNOSFGQCYVHSENVI2PJ7KQ, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:15.460927713 (75h46m51.024250723s) [noise=true , address=104.36.82.52:4111 , piece_id=HB7QAWN6CGNIIT4HD2GZ7JDQP5NXIJ5I2USBM66YOKTYENNOWG5A, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:18.393817968 (75h46m53.95714096s) [noise=true , address=104.36.82.55:4224 , piece_id=WRVNGSJV5BX4GJERLXNPAS7J7BZ5WFTM7R5H257YRCG5SP2HIYMA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:19.549174143 (75h46m55.112497151s) [noise=true , address=104.36.82.56:4253 , piece_id=5PMVGLZETMXCQ77K3I6PXACRIMBGR2CV5JGUSU5KUMYFNXVGA2GA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:22.511944593 (75h46m58.075267584s) [noise=true , address=104.36.82.37:6663 , piece_id=X7LWYIIHJFU2T2PHYR3R5TNT22UUPHEHQWZH7ZJMLS4AH4PJ6AIA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:48.027197558 (75h47m23.590520562s) [noise=true , address=104.36.82.14:6232 , piece_id=A2ZLUUZKRTBJRJ267QSK36C3LFZV7FPAYDDAXGIWDTNEGNV6MRRA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:11:05.567692582 (75h47m41.131015574s) [noise=true , address=104.36.82.49:4982 , piece_id=UNLV5UDNWULEIPE4QSQHHZQPK3BQKFRB5T3H6GP7ZQ5QGSNK5V6A, seg_pos={0 0}, upload=60693]: Done uploading piece. err: ecclient: upload failed (node:1NxoYJXCXmTvQncbkXt6yH7mhEBBrzVdzhgL5J6W5TD4N3pkaG, address:104.36.82.49:4982): protocol: expected piece hash; context deadline exceeded; EOF Jun 21 20:11:05.567759198 (75h47m41.131082195s) [noise=true , address=104.36.82.54:4171 , piece_id=MUMABKARK4KKUKWW6XDXCQVT6O7SLFHGHE7COWC7QIYGARAA2D5A, seg_pos={0 0}, upload=60693]: Done uploading piece. err: ecclient: upload failed (node:125SMhDsEZruNu6LkmQn2M7R6WzvXTdk86DRsKf522u8BNiANq3, address:104.36.82.54:4171): protocol: expected piece hash; context deadline exceeded; EOF Jun 21 20:11:05.567869019 (75h47m41.131192033s) [noise=true , address=104.36.82.10:6087 , piece_id=5FQG7BPZC373PFB54HYXSG2SVKOBBBSGZCLC4XDJEYQFOLQY74MA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: ecclient: upload failed (node:1S9k5eejJcxkiG5pXjjBZm18uP8k5jCGfWyeW2QQxmuRU8ZzRA, address:104.36.82.10:6087): protocol: write tcp 159.223.143.39:38550->104.36.82.10:6087: use of closed network connection; write tcp 159.223.143.39:38550->104.36.82.10:6087: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:38550->104.36.82.10:6087: use of closed network connection Jun 21 20:11:05.567932150 (75h47m41.131255161s) [seg_pos={0 0}, upload=60693]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jun 21 20:11:05.593626175 (75h47m41.156949181s) [upload=60693]: Done uploading segments... Jun 21 20:11:05.593657274 (75h47m41.156980265s) [upload=60693]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 13.062550142s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [..X.........................................................] duration: 18.351629936s FAIL (2025-06-21 20:53:38 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== pload=60693]: Starting upload Jun 21 20:10:05.602866542 (75h46m41.166189557s) [upload=60693]: Done starting upload Jun 21 20:10:05.603116545 (75h46m41.166439547s) [upload=60693]: Uploading segments... Jun 21 20:10:05.603495389 (75h46m41.166818397s) [upload=60693]: Got next segment. Inline: false Jun 21 20:10:05.603566566 (75h46m41.166889558s) [upload=60693]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 21 20:10:05.603573280 (75h46m41.166896286s) [upload=60693]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 21 20:10:05.639130162 (75h46m41.202453164s) [seg_pos={0 0}, upload=60693]: Begin upload segment... Jun 21 20:10:05.639231070 (75h46m41.202554057s) [seg_pos={0 0}, upload=60693]: Done begin upload segment. Jun 21 20:10:05.639292724 (75h46m41.202615728s) [noise=true , address=104.36.82.56:4253 , piece_id=5PMVGLZETMXCQ77K3I6PXACRIMBGR2CV5JGUSU5KUMYFNXVGA2GA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.639338066 (75h46m41.202661071s) [noise=true , address=104.36.82.53:4126 , piece_id=2MKRQJEPGDJKDM7VWY5M4M6NORILAIEYYATJ7HVZEOBOMXOI2LBA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.639416264 (75h46m41.202739267s) [noise=true , address=104.36.82.5:5869 , piece_id=HAEHXXZASV3DHT3QN7NTZOS6XZKHNUWBKCHWJJWSTJKXAKJKKBAA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.639873098 (75h46m41.203196115s) [noise=true , address=104.36.82.52:4111 , piece_id=HB7QAWN6CGNIIT4HD2GZ7JDQP5NXIJ5I2USBM66YOKTYENNOWG5A, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.639887296 (75h46m41.203210304s) [noise=true , address=104.36.82.10:6087 , piece_id=5FQG7BPZC373PFB54HYXSG2SVKOBBBSGZCLC4XDJEYQFOLQY74MA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.640266623 (75h46m41.203589616s) [noise=true , address=104.36.82.13:6196 , piece_id=BCHMQO5KKT5BLNWQGOHH3V43LGSE3BUQCI2SY4TTO5M5GQNOCTMQ, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.640352274 (75h46m41.203675286s) [noise=true , address=104.36.82.37:6663 , piece_id=X7LWYIIHJFU2T2PHYR3R5TNT22UUPHEHQWZH7ZJMLS4AH4PJ6AIA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.640659767 (75h46m41.203982779s) [noise=true , address=104.36.82.42:6807 , piece_id=ZBQZPXUDJ4VBCOSVKWAJ6SK6UD7SLPON7Y4SSCK2DQ5STFG4XCNQ, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.640961701 (75h46m41.20428471s) [noise=true , address=45.91.116.6:4982 , piece_id=VP45R6KXNH5U5SGUQUMAPIBY7KFDPBIR3S3VRIX27EAT6VLFBZ7Q, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.641159578 (75h46m41.204482589s) [noise=true , address=104.36.82.9:6039 , piece_id=I4AB6N27MYYUHWAZLJJUIBFBLOQFHDIM3YLYWPVRPNUV7VMYS2SA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.641588126 (75h46m41.204911136s) [noise=true , address=77.90.31.19:4853 , piece_id=VFEON5T2KSWM2KSLTVRZTXYL42JWENZYUEUPG3LMODGZWBMVGGPQ, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.641863297 (75h46m41.205186305s) [noise=true , address=77.90.31.12:4603 , piece_id=BMUIC6KNGEDL37436GLSXW5XOO6VYOPNOSFGQCYVHSENVI2PJ7KQ, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.642124427 (75h46m41.205447437s) [noise=true , address=104.36.82.49:4982 , piece_id=UNLV5UDNWULEIPE4QSQHHZQPK3BQKFRB5T3H6GP7ZQ5QGSNK5V6A, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.642285917 (75h46m41.205608925s) [noise=true , address=104.36.82.54:4171 , piece_id=MUMABKARK4KKUKWW6XDXCQVT6O7SLFHGHE7COWC7QIYGARAA2D5A, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.642656750 (75h46m41.205979758s) [noise=true , address=104.36.82.14:6232 , piece_id=A2ZLUUZKRTBJRJ267QSK36C3LFZV7FPAYDDAXGIWDTNEGNV6MRRA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.643565583 (75h46m41.206888597s) [noise=true , address=79.127.205.245:4231 , piece_id=EJK46XUUVI5NWOAQJEW4QIJG3KRXZHB4YOQUJRCS2MIOSWKGTNLQ, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.644620291 (75h46m41.207943303s) [noise=true , address=77.90.31.14:4670 , piece_id=7FNANMUKPM3N6YWRAOAIM4WJZJUTNF7ZBI6ZUBNEPGNGMFWOE24A, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.648842639 (75h46m41.212165631s) [noise=true , address=104.36.82.55:4224 , piece_id=WRVNGSJV5BX4GJERLXNPAS7J7BZ5WFTM7R5H257YRCG5SP2HIYMA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.825828239 (75h46m41.389151241s) [noise=true , address=104.36.82.53:4126 , piece_id=2MKRQJEPGDJKDM7VWY5M4M6NORILAIEYYATJ7HVZEOBOMXOI2LBA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: ecclient: upload failed (node:12dMWFrq2NMpnMzuPAphWerntC5kkrcnzRJZBA116uzJzyvk6fH, address:104.36.82.53:4126): protocol: storage node overloaded, request limit: 2000 Jun 21 20:10:05.834666043 (75h46m41.397989033s) [upload=60693]: Next returned nil segment Jun 21 20:10:05.834686295 (75h46m41.398009302s) [upload=60693]: Waiting for error group managing segments... Jun 21 20:10:05.877010488 (75h46m41.440333486s) [noise=true , address=45.91.116.6:4982 , piece_id=VP45R6KXNH5U5SGUQUMAPIBY7KFDPBIR3S3VRIX27EAT6VLFBZ7Q, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:05.886336964 (75h46m41.449659967s) [noise=true , address=104.36.82.9:6039 , piece_id=I4AB6N27MYYUHWAZLJJUIBFBLOQFHDIM3YLYWPVRPNUV7VMYS2SA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: ecclient: upload failed (node:12P11WVW9mVXw5ra2gZ3Rezon6wZi8GrN5CchQN3MiVU4ts3L6V, address:104.36.82.9:6039): protocol: storage node overloaded, request limit: 2000 Jun 21 20:10:08.064514542 (75h46m43.627837553s) [noise=true , address=77.90.31.14:4670 , piece_id=7FNANMUKPM3N6YWRAOAIM4WJZJUTNF7ZBI6ZUBNEPGNGMFWOE24A, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:08.099098391 (75h46m43.662421396s) [noise=true , address=79.127.205.245:4231 , piece_id=EJK46XUUVI5NWOAQJEW4QIJG3KRXZHB4YOQUJRCS2MIOSWKGTNLQ, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:08.489461749 (75h46m44.052784752s) [noise=true , address=104.36.82.42:6807 , piece_id=ZBQZPXUDJ4VBCOSVKWAJ6SK6UD7SLPON7Y4SSCK2DQ5STFG4XCNQ, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:08.650922093 (75h46m44.214245093s) [noise=true , address=77.90.31.19:4853 , piece_id=VFEON5T2KSWM2KSLTVRZTXYL42JWENZYUEUPG3LMODGZWBMVGGPQ, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:09.917746439 (75h46m45.481069441s) [noise=true , address=104.36.82.13:6196 , piece_id=BCHMQO5KKT5BLNWQGOHH3V43LGSE3BUQCI2SY4TTO5M5GQNOCTMQ, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:10.830722215 (75h46m46.394045216s) [noise=true , address=104.36.82.5:5869 , piece_id=HAEHXXZASV3DHT3QN7NTZOS6XZKHNUWBKCHWJJWSTJKXAKJKKBAA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:12.738783450 (75h46m48.302106452s) [noise=true , address=77.90.31.12:4603 , piece_id=BMUIC6KNGEDL37436GLSXW5XOO6VYOPNOSFGQCYVHSENVI2PJ7KQ, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:15.460927713 (75h46m51.024250723s) [noise=true , address=104.36.82.52:4111 , piece_id=HB7QAWN6CGNIIT4HD2GZ7JDQP5NXIJ5I2USBM66YOKTYENNOWG5A, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:18.393817968 (75h46m53.95714096s) [noise=true , address=104.36.82.55:4224 , piece_id=WRVNGSJV5BX4GJERLXNPAS7J7BZ5WFTM7R5H257YRCG5SP2HIYMA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:19.549174143 (75h46m55.112497151s) [noise=true , address=104.36.82.56:4253 , piece_id=5PMVGLZETMXCQ77K3I6PXACRIMBGR2CV5JGUSU5KUMYFNXVGA2GA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:22.511944593 (75h46m58.075267584s) [noise=true , address=104.36.82.37:6663 , piece_id=X7LWYIIHJFU2T2PHYR3R5TNT22UUPHEHQWZH7ZJMLS4AH4PJ6AIA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:48.027197558 (75h47m23.590520562s) [noise=true , address=104.36.82.14:6232 , piece_id=A2ZLUUZKRTBJRJ267QSK36C3LFZV7FPAYDDAXGIWDTNEGNV6MRRA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:11:05.567692582 (75h47m41.131015574s) [noise=true , address=104.36.82.49:4982 , piece_id=UNLV5UDNWULEIPE4QSQHHZQPK3BQKFRB5T3H6GP7ZQ5QGSNK5V6A, seg_pos={0 0}, upload=60693]: Done uploading piece. err: ecclient: upload failed (node:1NxoYJXCXmTvQncbkXt6yH7mhEBBrzVdzhgL5J6W5TD4N3pkaG, address:104.36.82.49:4982): protocol: expected piece hash; context deadline exceeded; EOF Jun 21 20:11:05.567759198 (75h47m41.131082195s) [noise=true , address=104.36.82.54:4171 , piece_id=MUMABKARK4KKUKWW6XDXCQVT6O7SLFHGHE7COWC7QIYGARAA2D5A, seg_pos={0 0}, upload=60693]: Done uploading piece. err: ecclient: upload failed (node:125SMhDsEZruNu6LkmQn2M7R6WzvXTdk86DRsKf522u8BNiANq3, address:104.36.82.54:4171): protocol: expected piece hash; context deadline exceeded; EOF Jun 21 20:11:05.567869019 (75h47m41.131192033s) [noise=true , address=104.36.82.10:6087 , piece_id=5FQG7BPZC373PFB54HYXSG2SVKOBBBSGZCLC4XDJEYQFOLQY74MA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: ecclient: upload failed (node:1S9k5eejJcxkiG5pXjjBZm18uP8k5jCGfWyeW2QQxmuRU8ZzRA, address:104.36.82.10:6087): protocol: write tcp 159.223.143.39:38550->104.36.82.10:6087: use of closed network connection; write tcp 159.223.143.39:38550->104.36.82.10:6087: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:38550->104.36.82.10:6087: use of closed network connection Jun 21 20:11:05.567932150 (75h47m41.131255161s) [seg_pos={0 0}, upload=60693]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jun 21 20:11:05.593626175 (75h47m41.156949181s) [upload=60693]: Done uploading segments... Jun 21 20:11:05.593657274 (75h47m41.156980265s) [upload=60693]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 18.351656557s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [...X........................................................] duration: 30.941513019s FAIL (2025-06-21 20:52:54 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== pload=60693]: Starting upload Jun 21 20:10:05.602866542 (75h46m41.166189557s) [upload=60693]: Done starting upload Jun 21 20:10:05.603116545 (75h46m41.166439547s) [upload=60693]: Uploading segments... Jun 21 20:10:05.603495389 (75h46m41.166818397s) [upload=60693]: Got next segment. Inline: false Jun 21 20:10:05.603566566 (75h46m41.166889558s) [upload=60693]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 21 20:10:05.603573280 (75h46m41.166896286s) [upload=60693]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 21 20:10:05.639130162 (75h46m41.202453164s) [seg_pos={0 0}, upload=60693]: Begin upload segment... Jun 21 20:10:05.639231070 (75h46m41.202554057s) [seg_pos={0 0}, upload=60693]: Done begin upload segment. Jun 21 20:10:05.639292724 (75h46m41.202615728s) [noise=true , address=104.36.82.56:4253 , piece_id=5PMVGLZETMXCQ77K3I6PXACRIMBGR2CV5JGUSU5KUMYFNXVGA2GA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.639338066 (75h46m41.202661071s) [noise=true , address=104.36.82.53:4126 , piece_id=2MKRQJEPGDJKDM7VWY5M4M6NORILAIEYYATJ7HVZEOBOMXOI2LBA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.639416264 (75h46m41.202739267s) [noise=true , address=104.36.82.5:5869 , piece_id=HAEHXXZASV3DHT3QN7NTZOS6XZKHNUWBKCHWJJWSTJKXAKJKKBAA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.639873098 (75h46m41.203196115s) [noise=true , address=104.36.82.52:4111 , piece_id=HB7QAWN6CGNIIT4HD2GZ7JDQP5NXIJ5I2USBM66YOKTYENNOWG5A, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.639887296 (75h46m41.203210304s) [noise=true , address=104.36.82.10:6087 , piece_id=5FQG7BPZC373PFB54HYXSG2SVKOBBBSGZCLC4XDJEYQFOLQY74MA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.640266623 (75h46m41.203589616s) [noise=true , address=104.36.82.13:6196 , piece_id=BCHMQO5KKT5BLNWQGOHH3V43LGSE3BUQCI2SY4TTO5M5GQNOCTMQ, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.640352274 (75h46m41.203675286s) [noise=true , address=104.36.82.37:6663 , piece_id=X7LWYIIHJFU2T2PHYR3R5TNT22UUPHEHQWZH7ZJMLS4AH4PJ6AIA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.640659767 (75h46m41.203982779s) [noise=true , address=104.36.82.42:6807 , piece_id=ZBQZPXUDJ4VBCOSVKWAJ6SK6UD7SLPON7Y4SSCK2DQ5STFG4XCNQ, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.640961701 (75h46m41.20428471s) [noise=true , address=45.91.116.6:4982 , piece_id=VP45R6KXNH5U5SGUQUMAPIBY7KFDPBIR3S3VRIX27EAT6VLFBZ7Q, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.641159578 (75h46m41.204482589s) [noise=true , address=104.36.82.9:6039 , piece_id=I4AB6N27MYYUHWAZLJJUIBFBLOQFHDIM3YLYWPVRPNUV7VMYS2SA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.641588126 (75h46m41.204911136s) [noise=true , address=77.90.31.19:4853 , piece_id=VFEON5T2KSWM2KSLTVRZTXYL42JWENZYUEUPG3LMODGZWBMVGGPQ, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.641863297 (75h46m41.205186305s) [noise=true , address=77.90.31.12:4603 , piece_id=BMUIC6KNGEDL37436GLSXW5XOO6VYOPNOSFGQCYVHSENVI2PJ7KQ, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.642124427 (75h46m41.205447437s) [noise=true , address=104.36.82.49:4982 , piece_id=UNLV5UDNWULEIPE4QSQHHZQPK3BQKFRB5T3H6GP7ZQ5QGSNK5V6A, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.642285917 (75h46m41.205608925s) [noise=true , address=104.36.82.54:4171 , piece_id=MUMABKARK4KKUKWW6XDXCQVT6O7SLFHGHE7COWC7QIYGARAA2D5A, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.642656750 (75h46m41.205979758s) [noise=true , address=104.36.82.14:6232 , piece_id=A2ZLUUZKRTBJRJ267QSK36C3LFZV7FPAYDDAXGIWDTNEGNV6MRRA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.643565583 (75h46m41.206888597s) [noise=true , address=79.127.205.245:4231 , piece_id=EJK46XUUVI5NWOAQJEW4QIJG3KRXZHB4YOQUJRCS2MIOSWKGTNLQ, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.644620291 (75h46m41.207943303s) [noise=true , address=77.90.31.14:4670 , piece_id=7FNANMUKPM3N6YWRAOAIM4WJZJUTNF7ZBI6ZUBNEPGNGMFWOE24A, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.648842639 (75h46m41.212165631s) [noise=true , address=104.36.82.55:4224 , piece_id=WRVNGSJV5BX4GJERLXNPAS7J7BZ5WFTM7R5H257YRCG5SP2HIYMA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.825828239 (75h46m41.389151241s) [noise=true , address=104.36.82.53:4126 , piece_id=2MKRQJEPGDJKDM7VWY5M4M6NORILAIEYYATJ7HVZEOBOMXOI2LBA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: ecclient: upload failed (node:12dMWFrq2NMpnMzuPAphWerntC5kkrcnzRJZBA116uzJzyvk6fH, address:104.36.82.53:4126): protocol: storage node overloaded, request limit: 2000 Jun 21 20:10:05.834666043 (75h46m41.397989033s) [upload=60693]: Next returned nil segment Jun 21 20:10:05.834686295 (75h46m41.398009302s) [upload=60693]: Waiting for error group managing segments... Jun 21 20:10:05.877010488 (75h46m41.440333486s) [noise=true , address=45.91.116.6:4982 , piece_id=VP45R6KXNH5U5SGUQUMAPIBY7KFDPBIR3S3VRIX27EAT6VLFBZ7Q, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:05.886336964 (75h46m41.449659967s) [noise=true , address=104.36.82.9:6039 , piece_id=I4AB6N27MYYUHWAZLJJUIBFBLOQFHDIM3YLYWPVRPNUV7VMYS2SA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: ecclient: upload failed (node:12P11WVW9mVXw5ra2gZ3Rezon6wZi8GrN5CchQN3MiVU4ts3L6V, address:104.36.82.9:6039): protocol: storage node overloaded, request limit: 2000 Jun 21 20:10:08.064514542 (75h46m43.627837553s) [noise=true , address=77.90.31.14:4670 , piece_id=7FNANMUKPM3N6YWRAOAIM4WJZJUTNF7ZBI6ZUBNEPGNGMFWOE24A, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:08.099098391 (75h46m43.662421396s) [noise=true , address=79.127.205.245:4231 , piece_id=EJK46XUUVI5NWOAQJEW4QIJG3KRXZHB4YOQUJRCS2MIOSWKGTNLQ, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:08.489461749 (75h46m44.052784752s) [noise=true , address=104.36.82.42:6807 , piece_id=ZBQZPXUDJ4VBCOSVKWAJ6SK6UD7SLPON7Y4SSCK2DQ5STFG4XCNQ, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:08.650922093 (75h46m44.214245093s) [noise=true , address=77.90.31.19:4853 , piece_id=VFEON5T2KSWM2KSLTVRZTXYL42JWENZYUEUPG3LMODGZWBMVGGPQ, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:09.917746439 (75h46m45.481069441s) [noise=true , address=104.36.82.13:6196 , piece_id=BCHMQO5KKT5BLNWQGOHH3V43LGSE3BUQCI2SY4TTO5M5GQNOCTMQ, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:10.830722215 (75h46m46.394045216s) [noise=true , address=104.36.82.5:5869 , piece_id=HAEHXXZASV3DHT3QN7NTZOS6XZKHNUWBKCHWJJWSTJKXAKJKKBAA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:12.738783450 (75h46m48.302106452s) [noise=true , address=77.90.31.12:4603 , piece_id=BMUIC6KNGEDL37436GLSXW5XOO6VYOPNOSFGQCYVHSENVI2PJ7KQ, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:15.460927713 (75h46m51.024250723s) [noise=true , address=104.36.82.52:4111 , piece_id=HB7QAWN6CGNIIT4HD2GZ7JDQP5NXIJ5I2USBM66YOKTYENNOWG5A, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:18.393817968 (75h46m53.95714096s) [noise=true , address=104.36.82.55:4224 , piece_id=WRVNGSJV5BX4GJERLXNPAS7J7BZ5WFTM7R5H257YRCG5SP2HIYMA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:19.549174143 (75h46m55.112497151s) [noise=true , address=104.36.82.56:4253 , piece_id=5PMVGLZETMXCQ77K3I6PXACRIMBGR2CV5JGUSU5KUMYFNXVGA2GA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:22.511944593 (75h46m58.075267584s) [noise=true , address=104.36.82.37:6663 , piece_id=X7LWYIIHJFU2T2PHYR3R5TNT22UUPHEHQWZH7ZJMLS4AH4PJ6AIA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:48.027197558 (75h47m23.590520562s) [noise=true , address=104.36.82.14:6232 , piece_id=A2ZLUUZKRTBJRJ267QSK36C3LFZV7FPAYDDAXGIWDTNEGNV6MRRA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:11:05.567692582 (75h47m41.131015574s) [noise=true , address=104.36.82.49:4982 , piece_id=UNLV5UDNWULEIPE4QSQHHZQPK3BQKFRB5T3H6GP7ZQ5QGSNK5V6A, seg_pos={0 0}, upload=60693]: Done uploading piece. err: ecclient: upload failed (node:1NxoYJXCXmTvQncbkXt6yH7mhEBBrzVdzhgL5J6W5TD4N3pkaG, address:104.36.82.49:4982): protocol: expected piece hash; context deadline exceeded; EOF Jun 21 20:11:05.567759198 (75h47m41.131082195s) [noise=true , address=104.36.82.54:4171 , piece_id=MUMABKARK4KKUKWW6XDXCQVT6O7SLFHGHE7COWC7QIYGARAA2D5A, seg_pos={0 0}, upload=60693]: Done uploading piece. err: ecclient: upload failed (node:125SMhDsEZruNu6LkmQn2M7R6WzvXTdk86DRsKf522u8BNiANq3, address:104.36.82.54:4171): protocol: expected piece hash; context deadline exceeded; EOF Jun 21 20:11:05.567869019 (75h47m41.131192033s) [noise=true , address=104.36.82.10:6087 , piece_id=5FQG7BPZC373PFB54HYXSG2SVKOBBBSGZCLC4XDJEYQFOLQY74MA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: ecclient: upload failed (node:1S9k5eejJcxkiG5pXjjBZm18uP8k5jCGfWyeW2QQxmuRU8ZzRA, address:104.36.82.10:6087): protocol: write tcp 159.223.143.39:38550->104.36.82.10:6087: use of closed network connection; write tcp 159.223.143.39:38550->104.36.82.10:6087: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:38550->104.36.82.10:6087: use of closed network connection Jun 21 20:11:05.567932150 (75h47m41.131255161s) [seg_pos={0 0}, upload=60693]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jun 21 20:11:05.593626175 (75h47m41.156949181s) [upload=60693]: Done uploading segments... Jun 21 20:11:05.593657274 (75h47m41.156980265s) [upload=60693]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 30.941547227s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [....X.......................................................] duration: 13.139675978s FAIL (2025-06-21 20:52:06 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== pload=60693]: Starting upload Jun 21 20:10:05.602866542 (75h46m41.166189557s) [upload=60693]: Done starting upload Jun 21 20:10:05.603116545 (75h46m41.166439547s) [upload=60693]: Uploading segments... Jun 21 20:10:05.603495389 (75h46m41.166818397s) [upload=60693]: Got next segment. Inline: false Jun 21 20:10:05.603566566 (75h46m41.166889558s) [upload=60693]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 21 20:10:05.603573280 (75h46m41.166896286s) [upload=60693]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 21 20:10:05.639130162 (75h46m41.202453164s) [seg_pos={0 0}, upload=60693]: Begin upload segment... Jun 21 20:10:05.639231070 (75h46m41.202554057s) [seg_pos={0 0}, upload=60693]: Done begin upload segment. Jun 21 20:10:05.639292724 (75h46m41.202615728s) [noise=true , address=104.36.82.56:4253 , piece_id=5PMVGLZETMXCQ77K3I6PXACRIMBGR2CV5JGUSU5KUMYFNXVGA2GA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.639338066 (75h46m41.202661071s) [noise=true , address=104.36.82.53:4126 , piece_id=2MKRQJEPGDJKDM7VWY5M4M6NORILAIEYYATJ7HVZEOBOMXOI2LBA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.639416264 (75h46m41.202739267s) [noise=true , address=104.36.82.5:5869 , piece_id=HAEHXXZASV3DHT3QN7NTZOS6XZKHNUWBKCHWJJWSTJKXAKJKKBAA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.639873098 (75h46m41.203196115s) [noise=true , address=104.36.82.52:4111 , piece_id=HB7QAWN6CGNIIT4HD2GZ7JDQP5NXIJ5I2USBM66YOKTYENNOWG5A, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.639887296 (75h46m41.203210304s) [noise=true , address=104.36.82.10:6087 , piece_id=5FQG7BPZC373PFB54HYXSG2SVKOBBBSGZCLC4XDJEYQFOLQY74MA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.640266623 (75h46m41.203589616s) [noise=true , address=104.36.82.13:6196 , piece_id=BCHMQO5KKT5BLNWQGOHH3V43LGSE3BUQCI2SY4TTO5M5GQNOCTMQ, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.640352274 (75h46m41.203675286s) [noise=true , address=104.36.82.37:6663 , piece_id=X7LWYIIHJFU2T2PHYR3R5TNT22UUPHEHQWZH7ZJMLS4AH4PJ6AIA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.640659767 (75h46m41.203982779s) [noise=true , address=104.36.82.42:6807 , piece_id=ZBQZPXUDJ4VBCOSVKWAJ6SK6UD7SLPON7Y4SSCK2DQ5STFG4XCNQ, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.640961701 (75h46m41.20428471s) [noise=true , address=45.91.116.6:4982 , piece_id=VP45R6KXNH5U5SGUQUMAPIBY7KFDPBIR3S3VRIX27EAT6VLFBZ7Q, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.641159578 (75h46m41.204482589s) [noise=true , address=104.36.82.9:6039 , piece_id=I4AB6N27MYYUHWAZLJJUIBFBLOQFHDIM3YLYWPVRPNUV7VMYS2SA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.641588126 (75h46m41.204911136s) [noise=true , address=77.90.31.19:4853 , piece_id=VFEON5T2KSWM2KSLTVRZTXYL42JWENZYUEUPG3LMODGZWBMVGGPQ, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.641863297 (75h46m41.205186305s) [noise=true , address=77.90.31.12:4603 , piece_id=BMUIC6KNGEDL37436GLSXW5XOO6VYOPNOSFGQCYVHSENVI2PJ7KQ, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.642124427 (75h46m41.205447437s) [noise=true , address=104.36.82.49:4982 , piece_id=UNLV5UDNWULEIPE4QSQHHZQPK3BQKFRB5T3H6GP7ZQ5QGSNK5V6A, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.642285917 (75h46m41.205608925s) [noise=true , address=104.36.82.54:4171 , piece_id=MUMABKARK4KKUKWW6XDXCQVT6O7SLFHGHE7COWC7QIYGARAA2D5A, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.642656750 (75h46m41.205979758s) [noise=true , address=104.36.82.14:6232 , piece_id=A2ZLUUZKRTBJRJ267QSK36C3LFZV7FPAYDDAXGIWDTNEGNV6MRRA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.643565583 (75h46m41.206888597s) [noise=true , address=79.127.205.245:4231 , piece_id=EJK46XUUVI5NWOAQJEW4QIJG3KRXZHB4YOQUJRCS2MIOSWKGTNLQ, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.644620291 (75h46m41.207943303s) [noise=true , address=77.90.31.14:4670 , piece_id=7FNANMUKPM3N6YWRAOAIM4WJZJUTNF7ZBI6ZUBNEPGNGMFWOE24A, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.648842639 (75h46m41.212165631s) [noise=true , address=104.36.82.55:4224 , piece_id=WRVNGSJV5BX4GJERLXNPAS7J7BZ5WFTM7R5H257YRCG5SP2HIYMA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.825828239 (75h46m41.389151241s) [noise=true , address=104.36.82.53:4126 , piece_id=2MKRQJEPGDJKDM7VWY5M4M6NORILAIEYYATJ7HVZEOBOMXOI2LBA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: ecclient: upload failed (node:12dMWFrq2NMpnMzuPAphWerntC5kkrcnzRJZBA116uzJzyvk6fH, address:104.36.82.53:4126): protocol: storage node overloaded, request limit: 2000 Jun 21 20:10:05.834666043 (75h46m41.397989033s) [upload=60693]: Next returned nil segment Jun 21 20:10:05.834686295 (75h46m41.398009302s) [upload=60693]: Waiting for error group managing segments... Jun 21 20:10:05.877010488 (75h46m41.440333486s) [noise=true , address=45.91.116.6:4982 , piece_id=VP45R6KXNH5U5SGUQUMAPIBY7KFDPBIR3S3VRIX27EAT6VLFBZ7Q, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:05.886336964 (75h46m41.449659967s) [noise=true , address=104.36.82.9:6039 , piece_id=I4AB6N27MYYUHWAZLJJUIBFBLOQFHDIM3YLYWPVRPNUV7VMYS2SA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: ecclient: upload failed (node:12P11WVW9mVXw5ra2gZ3Rezon6wZi8GrN5CchQN3MiVU4ts3L6V, address:104.36.82.9:6039): protocol: storage node overloaded, request limit: 2000 Jun 21 20:10:08.064514542 (75h46m43.627837553s) [noise=true , address=77.90.31.14:4670 , piece_id=7FNANMUKPM3N6YWRAOAIM4WJZJUTNF7ZBI6ZUBNEPGNGMFWOE24A, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:08.099098391 (75h46m43.662421396s) [noise=true , address=79.127.205.245:4231 , piece_id=EJK46XUUVI5NWOAQJEW4QIJG3KRXZHB4YOQUJRCS2MIOSWKGTNLQ, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:08.489461749 (75h46m44.052784752s) [noise=true , address=104.36.82.42:6807 , piece_id=ZBQZPXUDJ4VBCOSVKWAJ6SK6UD7SLPON7Y4SSCK2DQ5STFG4XCNQ, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:08.650922093 (75h46m44.214245093s) [noise=true , address=77.90.31.19:4853 , piece_id=VFEON5T2KSWM2KSLTVRZTXYL42JWENZYUEUPG3LMODGZWBMVGGPQ, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:09.917746439 (75h46m45.481069441s) [noise=true , address=104.36.82.13:6196 , piece_id=BCHMQO5KKT5BLNWQGOHH3V43LGSE3BUQCI2SY4TTO5M5GQNOCTMQ, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:10.830722215 (75h46m46.394045216s) [noise=true , address=104.36.82.5:5869 , piece_id=HAEHXXZASV3DHT3QN7NTZOS6XZKHNUWBKCHWJJWSTJKXAKJKKBAA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:12.738783450 (75h46m48.302106452s) [noise=true , address=77.90.31.12:4603 , piece_id=BMUIC6KNGEDL37436GLSXW5XOO6VYOPNOSFGQCYVHSENVI2PJ7KQ, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:15.460927713 (75h46m51.024250723s) [noise=true , address=104.36.82.52:4111 , piece_id=HB7QAWN6CGNIIT4HD2GZ7JDQP5NXIJ5I2USBM66YOKTYENNOWG5A, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:18.393817968 (75h46m53.95714096s) [noise=true , address=104.36.82.55:4224 , piece_id=WRVNGSJV5BX4GJERLXNPAS7J7BZ5WFTM7R5H257YRCG5SP2HIYMA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:19.549174143 (75h46m55.112497151s) [noise=true , address=104.36.82.56:4253 , piece_id=5PMVGLZETMXCQ77K3I6PXACRIMBGR2CV5JGUSU5KUMYFNXVGA2GA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:22.511944593 (75h46m58.075267584s) [noise=true , address=104.36.82.37:6663 , piece_id=X7LWYIIHJFU2T2PHYR3R5TNT22UUPHEHQWZH7ZJMLS4AH4PJ6AIA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:48.027197558 (75h47m23.590520562s) [noise=true , address=104.36.82.14:6232 , piece_id=A2ZLUUZKRTBJRJ267QSK36C3LFZV7FPAYDDAXGIWDTNEGNV6MRRA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:11:05.567692582 (75h47m41.131015574s) [noise=true , address=104.36.82.49:4982 , piece_id=UNLV5UDNWULEIPE4QSQHHZQPK3BQKFRB5T3H6GP7ZQ5QGSNK5V6A, seg_pos={0 0}, upload=60693]: Done uploading piece. err: ecclient: upload failed (node:1NxoYJXCXmTvQncbkXt6yH7mhEBBrzVdzhgL5J6W5TD4N3pkaG, address:104.36.82.49:4982): protocol: expected piece hash; context deadline exceeded; EOF Jun 21 20:11:05.567759198 (75h47m41.131082195s) [noise=true , address=104.36.82.54:4171 , piece_id=MUMABKARK4KKUKWW6XDXCQVT6O7SLFHGHE7COWC7QIYGARAA2D5A, seg_pos={0 0}, upload=60693]: Done uploading piece. err: ecclient: upload failed (node:125SMhDsEZruNu6LkmQn2M7R6WzvXTdk86DRsKf522u8BNiANq3, address:104.36.82.54:4171): protocol: expected piece hash; context deadline exceeded; EOF Jun 21 20:11:05.567869019 (75h47m41.131192033s) [noise=true , address=104.36.82.10:6087 , piece_id=5FQG7BPZC373PFB54HYXSG2SVKOBBBSGZCLC4XDJEYQFOLQY74MA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: ecclient: upload failed (node:1S9k5eejJcxkiG5pXjjBZm18uP8k5jCGfWyeW2QQxmuRU8ZzRA, address:104.36.82.10:6087): protocol: write tcp 159.223.143.39:38550->104.36.82.10:6087: use of closed network connection; write tcp 159.223.143.39:38550->104.36.82.10:6087: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:38550->104.36.82.10:6087: use of closed network connection Jun 21 20:11:05.567932150 (75h47m41.131255161s) [seg_pos={0 0}, upload=60693]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jun 21 20:11:05.593626175 (75h47m41.156949181s) [upload=60693]: Done uploading segments... Jun 21 20:11:05.593657274 (75h47m41.156980265s) [upload=60693]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 13.139706068s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.....X......................................................] duration: 50.241632419s FAIL (2025-06-21 20:51:39 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== pload=60693]: Starting upload Jun 21 20:10:05.602866542 (75h46m41.166189557s) [upload=60693]: Done starting upload Jun 21 20:10:05.603116545 (75h46m41.166439547s) [upload=60693]: Uploading segments... Jun 21 20:10:05.603495389 (75h46m41.166818397s) [upload=60693]: Got next segment. Inline: false Jun 21 20:10:05.603566566 (75h46m41.166889558s) [upload=60693]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 21 20:10:05.603573280 (75h46m41.166896286s) [upload=60693]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 21 20:10:05.639130162 (75h46m41.202453164s) [seg_pos={0 0}, upload=60693]: Begin upload segment... Jun 21 20:10:05.639231070 (75h46m41.202554057s) [seg_pos={0 0}, upload=60693]: Done begin upload segment. Jun 21 20:10:05.639292724 (75h46m41.202615728s) [noise=true , address=104.36.82.56:4253 , piece_id=5PMVGLZETMXCQ77K3I6PXACRIMBGR2CV5JGUSU5KUMYFNXVGA2GA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.639338066 (75h46m41.202661071s) [noise=true , address=104.36.82.53:4126 , piece_id=2MKRQJEPGDJKDM7VWY5M4M6NORILAIEYYATJ7HVZEOBOMXOI2LBA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.639416264 (75h46m41.202739267s) [noise=true , address=104.36.82.5:5869 , piece_id=HAEHXXZASV3DHT3QN7NTZOS6XZKHNUWBKCHWJJWSTJKXAKJKKBAA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.639873098 (75h46m41.203196115s) [noise=true , address=104.36.82.52:4111 , piece_id=HB7QAWN6CGNIIT4HD2GZ7JDQP5NXIJ5I2USBM66YOKTYENNOWG5A, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.639887296 (75h46m41.203210304s) [noise=true , address=104.36.82.10:6087 , piece_id=5FQG7BPZC373PFB54HYXSG2SVKOBBBSGZCLC4XDJEYQFOLQY74MA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.640266623 (75h46m41.203589616s) [noise=true , address=104.36.82.13:6196 , piece_id=BCHMQO5KKT5BLNWQGOHH3V43LGSE3BUQCI2SY4TTO5M5GQNOCTMQ, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.640352274 (75h46m41.203675286s) [noise=true , address=104.36.82.37:6663 , piece_id=X7LWYIIHJFU2T2PHYR3R5TNT22UUPHEHQWZH7ZJMLS4AH4PJ6AIA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.640659767 (75h46m41.203982779s) [noise=true , address=104.36.82.42:6807 , piece_id=ZBQZPXUDJ4VBCOSVKWAJ6SK6UD7SLPON7Y4SSCK2DQ5STFG4XCNQ, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.640961701 (75h46m41.20428471s) [noise=true , address=45.91.116.6:4982 , piece_id=VP45R6KXNH5U5SGUQUMAPIBY7KFDPBIR3S3VRIX27EAT6VLFBZ7Q, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.641159578 (75h46m41.204482589s) [noise=true , address=104.36.82.9:6039 , piece_id=I4AB6N27MYYUHWAZLJJUIBFBLOQFHDIM3YLYWPVRPNUV7VMYS2SA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.641588126 (75h46m41.204911136s) [noise=true , address=77.90.31.19:4853 , piece_id=VFEON5T2KSWM2KSLTVRZTXYL42JWENZYUEUPG3LMODGZWBMVGGPQ, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.641863297 (75h46m41.205186305s) [noise=true , address=77.90.31.12:4603 , piece_id=BMUIC6KNGEDL37436GLSXW5XOO6VYOPNOSFGQCYVHSENVI2PJ7KQ, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.642124427 (75h46m41.205447437s) [noise=true , address=104.36.82.49:4982 , piece_id=UNLV5UDNWULEIPE4QSQHHZQPK3BQKFRB5T3H6GP7ZQ5QGSNK5V6A, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.642285917 (75h46m41.205608925s) [noise=true , address=104.36.82.54:4171 , piece_id=MUMABKARK4KKUKWW6XDXCQVT6O7SLFHGHE7COWC7QIYGARAA2D5A, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.642656750 (75h46m41.205979758s) [noise=true , address=104.36.82.14:6232 , piece_id=A2ZLUUZKRTBJRJ267QSK36C3LFZV7FPAYDDAXGIWDTNEGNV6MRRA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.643565583 (75h46m41.206888597s) [noise=true , address=79.127.205.245:4231 , piece_id=EJK46XUUVI5NWOAQJEW4QIJG3KRXZHB4YOQUJRCS2MIOSWKGTNLQ, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.644620291 (75h46m41.207943303s) [noise=true , address=77.90.31.14:4670 , piece_id=7FNANMUKPM3N6YWRAOAIM4WJZJUTNF7ZBI6ZUBNEPGNGMFWOE24A, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.648842639 (75h46m41.212165631s) [noise=true , address=104.36.82.55:4224 , piece_id=WRVNGSJV5BX4GJERLXNPAS7J7BZ5WFTM7R5H257YRCG5SP2HIYMA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.825828239 (75h46m41.389151241s) [noise=true , address=104.36.82.53:4126 , piece_id=2MKRQJEPGDJKDM7VWY5M4M6NORILAIEYYATJ7HVZEOBOMXOI2LBA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: ecclient: upload failed (node:12dMWFrq2NMpnMzuPAphWerntC5kkrcnzRJZBA116uzJzyvk6fH, address:104.36.82.53:4126): protocol: storage node overloaded, request limit: 2000 Jun 21 20:10:05.834666043 (75h46m41.397989033s) [upload=60693]: Next returned nil segment Jun 21 20:10:05.834686295 (75h46m41.398009302s) [upload=60693]: Waiting for error group managing segments... Jun 21 20:10:05.877010488 (75h46m41.440333486s) [noise=true , address=45.91.116.6:4982 , piece_id=VP45R6KXNH5U5SGUQUMAPIBY7KFDPBIR3S3VRIX27EAT6VLFBZ7Q, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:05.886336964 (75h46m41.449659967s) [noise=true , address=104.36.82.9:6039 , piece_id=I4AB6N27MYYUHWAZLJJUIBFBLOQFHDIM3YLYWPVRPNUV7VMYS2SA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: ecclient: upload failed (node:12P11WVW9mVXw5ra2gZ3Rezon6wZi8GrN5CchQN3MiVU4ts3L6V, address:104.36.82.9:6039): protocol: storage node overloaded, request limit: 2000 Jun 21 20:10:08.064514542 (75h46m43.627837553s) [noise=true , address=77.90.31.14:4670 , piece_id=7FNANMUKPM3N6YWRAOAIM4WJZJUTNF7ZBI6ZUBNEPGNGMFWOE24A, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:08.099098391 (75h46m43.662421396s) [noise=true , address=79.127.205.245:4231 , piece_id=EJK46XUUVI5NWOAQJEW4QIJG3KRXZHB4YOQUJRCS2MIOSWKGTNLQ, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:08.489461749 (75h46m44.052784752s) [noise=true , address=104.36.82.42:6807 , piece_id=ZBQZPXUDJ4VBCOSVKWAJ6SK6UD7SLPON7Y4SSCK2DQ5STFG4XCNQ, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:08.650922093 (75h46m44.214245093s) [noise=true , address=77.90.31.19:4853 , piece_id=VFEON5T2KSWM2KSLTVRZTXYL42JWENZYUEUPG3LMODGZWBMVGGPQ, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:09.917746439 (75h46m45.481069441s) [noise=true , address=104.36.82.13:6196 , piece_id=BCHMQO5KKT5BLNWQGOHH3V43LGSE3BUQCI2SY4TTO5M5GQNOCTMQ, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:10.830722215 (75h46m46.394045216s) [noise=true , address=104.36.82.5:5869 , piece_id=HAEHXXZASV3DHT3QN7NTZOS6XZKHNUWBKCHWJJWSTJKXAKJKKBAA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:12.738783450 (75h46m48.302106452s) [noise=true , address=77.90.31.12:4603 , piece_id=BMUIC6KNGEDL37436GLSXW5XOO6VYOPNOSFGQCYVHSENVI2PJ7KQ, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:15.460927713 (75h46m51.024250723s) [noise=true , address=104.36.82.52:4111 , piece_id=HB7QAWN6CGNIIT4HD2GZ7JDQP5NXIJ5I2USBM66YOKTYENNOWG5A, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:18.393817968 (75h46m53.95714096s) [noise=true , address=104.36.82.55:4224 , piece_id=WRVNGSJV5BX4GJERLXNPAS7J7BZ5WFTM7R5H257YRCG5SP2HIYMA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:19.549174143 (75h46m55.112497151s) [noise=true , address=104.36.82.56:4253 , piece_id=5PMVGLZETMXCQ77K3I6PXACRIMBGR2CV5JGUSU5KUMYFNXVGA2GA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:22.511944593 (75h46m58.075267584s) [noise=true , address=104.36.82.37:6663 , piece_id=X7LWYIIHJFU2T2PHYR3R5TNT22UUPHEHQWZH7ZJMLS4AH4PJ6AIA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:48.027197558 (75h47m23.590520562s) [noise=true , address=104.36.82.14:6232 , piece_id=A2ZLUUZKRTBJRJ267QSK36C3LFZV7FPAYDDAXGIWDTNEGNV6MRRA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:11:05.567692582 (75h47m41.131015574s) [noise=true , address=104.36.82.49:4982 , piece_id=UNLV5UDNWULEIPE4QSQHHZQPK3BQKFRB5T3H6GP7ZQ5QGSNK5V6A, seg_pos={0 0}, upload=60693]: Done uploading piece. err: ecclient: upload failed (node:1NxoYJXCXmTvQncbkXt6yH7mhEBBrzVdzhgL5J6W5TD4N3pkaG, address:104.36.82.49:4982): protocol: expected piece hash; context deadline exceeded; EOF Jun 21 20:11:05.567759198 (75h47m41.131082195s) [noise=true , address=104.36.82.54:4171 , piece_id=MUMABKARK4KKUKWW6XDXCQVT6O7SLFHGHE7COWC7QIYGARAA2D5A, seg_pos={0 0}, upload=60693]: Done uploading piece. err: ecclient: upload failed (node:125SMhDsEZruNu6LkmQn2M7R6WzvXTdk86DRsKf522u8BNiANq3, address:104.36.82.54:4171): protocol: expected piece hash; context deadline exceeded; EOF Jun 21 20:11:05.567869019 (75h47m41.131192033s) [noise=true , address=104.36.82.10:6087 , piece_id=5FQG7BPZC373PFB54HYXSG2SVKOBBBSGZCLC4XDJEYQFOLQY74MA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: ecclient: upload failed (node:1S9k5eejJcxkiG5pXjjBZm18uP8k5jCGfWyeW2QQxmuRU8ZzRA, address:104.36.82.10:6087): protocol: write tcp 159.223.143.39:38550->104.36.82.10:6087: use of closed network connection; write tcp 159.223.143.39:38550->104.36.82.10:6087: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:38550->104.36.82.10:6087: use of closed network connection Jun 21 20:11:05.567932150 (75h47m41.131255161s) [seg_pos={0 0}, upload=60693]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jun 21 20:11:05.593626175 (75h47m41.156949181s) [upload=60693]: Done uploading segments... Jun 21 20:11:05.593657274 (75h47m41.156980265s) [upload=60693]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 50.241661354s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [......X.....................................................] duration: 21.522605369s FAIL (2025-06-21 20:50:48 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== pload=60693]: Starting upload Jun 21 20:10:05.602866542 (75h46m41.166189557s) [upload=60693]: Done starting upload Jun 21 20:10:05.603116545 (75h46m41.166439547s) [upload=60693]: Uploading segments... Jun 21 20:10:05.603495389 (75h46m41.166818397s) [upload=60693]: Got next segment. Inline: false Jun 21 20:10:05.603566566 (75h46m41.166889558s) [upload=60693]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 21 20:10:05.603573280 (75h46m41.166896286s) [upload=60693]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 21 20:10:05.639130162 (75h46m41.202453164s) [seg_pos={0 0}, upload=60693]: Begin upload segment... Jun 21 20:10:05.639231070 (75h46m41.202554057s) [seg_pos={0 0}, upload=60693]: Done begin upload segment. Jun 21 20:10:05.639292724 (75h46m41.202615728s) [noise=true , address=104.36.82.56:4253 , piece_id=5PMVGLZETMXCQ77K3I6PXACRIMBGR2CV5JGUSU5KUMYFNXVGA2GA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.639338066 (75h46m41.202661071s) [noise=true , address=104.36.82.53:4126 , piece_id=2MKRQJEPGDJKDM7VWY5M4M6NORILAIEYYATJ7HVZEOBOMXOI2LBA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.639416264 (75h46m41.202739267s) [noise=true , address=104.36.82.5:5869 , piece_id=HAEHXXZASV3DHT3QN7NTZOS6XZKHNUWBKCHWJJWSTJKXAKJKKBAA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.639873098 (75h46m41.203196115s) [noise=true , address=104.36.82.52:4111 , piece_id=HB7QAWN6CGNIIT4HD2GZ7JDQP5NXIJ5I2USBM66YOKTYENNOWG5A, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.639887296 (75h46m41.203210304s) [noise=true , address=104.36.82.10:6087 , piece_id=5FQG7BPZC373PFB54HYXSG2SVKOBBBSGZCLC4XDJEYQFOLQY74MA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.640266623 (75h46m41.203589616s) [noise=true , address=104.36.82.13:6196 , piece_id=BCHMQO5KKT5BLNWQGOHH3V43LGSE3BUQCI2SY4TTO5M5GQNOCTMQ, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.640352274 (75h46m41.203675286s) [noise=true , address=104.36.82.37:6663 , piece_id=X7LWYIIHJFU2T2PHYR3R5TNT22UUPHEHQWZH7ZJMLS4AH4PJ6AIA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.640659767 (75h46m41.203982779s) [noise=true , address=104.36.82.42:6807 , piece_id=ZBQZPXUDJ4VBCOSVKWAJ6SK6UD7SLPON7Y4SSCK2DQ5STFG4XCNQ, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.640961701 (75h46m41.20428471s) [noise=true , address=45.91.116.6:4982 , piece_id=VP45R6KXNH5U5SGUQUMAPIBY7KFDPBIR3S3VRIX27EAT6VLFBZ7Q, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.641159578 (75h46m41.204482589s) [noise=true , address=104.36.82.9:6039 , piece_id=I4AB6N27MYYUHWAZLJJUIBFBLOQFHDIM3YLYWPVRPNUV7VMYS2SA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.641588126 (75h46m41.204911136s) [noise=true , address=77.90.31.19:4853 , piece_id=VFEON5T2KSWM2KSLTVRZTXYL42JWENZYUEUPG3LMODGZWBMVGGPQ, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.641863297 (75h46m41.205186305s) [noise=true , address=77.90.31.12:4603 , piece_id=BMUIC6KNGEDL37436GLSXW5XOO6VYOPNOSFGQCYVHSENVI2PJ7KQ, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.642124427 (75h46m41.205447437s) [noise=true , address=104.36.82.49:4982 , piece_id=UNLV5UDNWULEIPE4QSQHHZQPK3BQKFRB5T3H6GP7ZQ5QGSNK5V6A, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.642285917 (75h46m41.205608925s) [noise=true , address=104.36.82.54:4171 , piece_id=MUMABKARK4KKUKWW6XDXCQVT6O7SLFHGHE7COWC7QIYGARAA2D5A, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.642656750 (75h46m41.205979758s) [noise=true , address=104.36.82.14:6232 , piece_id=A2ZLUUZKRTBJRJ267QSK36C3LFZV7FPAYDDAXGIWDTNEGNV6MRRA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.643565583 (75h46m41.206888597s) [noise=true , address=79.127.205.245:4231 , piece_id=EJK46XUUVI5NWOAQJEW4QIJG3KRXZHB4YOQUJRCS2MIOSWKGTNLQ, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.644620291 (75h46m41.207943303s) [noise=true , address=77.90.31.14:4670 , piece_id=7FNANMUKPM3N6YWRAOAIM4WJZJUTNF7ZBI6ZUBNEPGNGMFWOE24A, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.648842639 (75h46m41.212165631s) [noise=true , address=104.36.82.55:4224 , piece_id=WRVNGSJV5BX4GJERLXNPAS7J7BZ5WFTM7R5H257YRCG5SP2HIYMA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.825828239 (75h46m41.389151241s) [noise=true , address=104.36.82.53:4126 , piece_id=2MKRQJEPGDJKDM7VWY5M4M6NORILAIEYYATJ7HVZEOBOMXOI2LBA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: ecclient: upload failed (node:12dMWFrq2NMpnMzuPAphWerntC5kkrcnzRJZBA116uzJzyvk6fH, address:104.36.82.53:4126): protocol: storage node overloaded, request limit: 2000 Jun 21 20:10:05.834666043 (75h46m41.397989033s) [upload=60693]: Next returned nil segment Jun 21 20:10:05.834686295 (75h46m41.398009302s) [upload=60693]: Waiting for error group managing segments... Jun 21 20:10:05.877010488 (75h46m41.440333486s) [noise=true , address=45.91.116.6:4982 , piece_id=VP45R6KXNH5U5SGUQUMAPIBY7KFDPBIR3S3VRIX27EAT6VLFBZ7Q, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:05.886336964 (75h46m41.449659967s) [noise=true , address=104.36.82.9:6039 , piece_id=I4AB6N27MYYUHWAZLJJUIBFBLOQFHDIM3YLYWPVRPNUV7VMYS2SA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: ecclient: upload failed (node:12P11WVW9mVXw5ra2gZ3Rezon6wZi8GrN5CchQN3MiVU4ts3L6V, address:104.36.82.9:6039): protocol: storage node overloaded, request limit: 2000 Jun 21 20:10:08.064514542 (75h46m43.627837553s) [noise=true , address=77.90.31.14:4670 , piece_id=7FNANMUKPM3N6YWRAOAIM4WJZJUTNF7ZBI6ZUBNEPGNGMFWOE24A, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:08.099098391 (75h46m43.662421396s) [noise=true , address=79.127.205.245:4231 , piece_id=EJK46XUUVI5NWOAQJEW4QIJG3KRXZHB4YOQUJRCS2MIOSWKGTNLQ, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:08.489461749 (75h46m44.052784752s) [noise=true , address=104.36.82.42:6807 , piece_id=ZBQZPXUDJ4VBCOSVKWAJ6SK6UD7SLPON7Y4SSCK2DQ5STFG4XCNQ, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:08.650922093 (75h46m44.214245093s) [noise=true , address=77.90.31.19:4853 , piece_id=VFEON5T2KSWM2KSLTVRZTXYL42JWENZYUEUPG3LMODGZWBMVGGPQ, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:09.917746439 (75h46m45.481069441s) [noise=true , address=104.36.82.13:6196 , piece_id=BCHMQO5KKT5BLNWQGOHH3V43LGSE3BUQCI2SY4TTO5M5GQNOCTMQ, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:10.830722215 (75h46m46.394045216s) [noise=true , address=104.36.82.5:5869 , piece_id=HAEHXXZASV3DHT3QN7NTZOS6XZKHNUWBKCHWJJWSTJKXAKJKKBAA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:12.738783450 (75h46m48.302106452s) [noise=true , address=77.90.31.12:4603 , piece_id=BMUIC6KNGEDL37436GLSXW5XOO6VYOPNOSFGQCYVHSENVI2PJ7KQ, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:15.460927713 (75h46m51.024250723s) [noise=true , address=104.36.82.52:4111 , piece_id=HB7QAWN6CGNIIT4HD2GZ7JDQP5NXIJ5I2USBM66YOKTYENNOWG5A, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:18.393817968 (75h46m53.95714096s) [noise=true , address=104.36.82.55:4224 , piece_id=WRVNGSJV5BX4GJERLXNPAS7J7BZ5WFTM7R5H257YRCG5SP2HIYMA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:19.549174143 (75h46m55.112497151s) [noise=true , address=104.36.82.56:4253 , piece_id=5PMVGLZETMXCQ77K3I6PXACRIMBGR2CV5JGUSU5KUMYFNXVGA2GA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:22.511944593 (75h46m58.075267584s) [noise=true , address=104.36.82.37:6663 , piece_id=X7LWYIIHJFU2T2PHYR3R5TNT22UUPHEHQWZH7ZJMLS4AH4PJ6AIA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:48.027197558 (75h47m23.590520562s) [noise=true , address=104.36.82.14:6232 , piece_id=A2ZLUUZKRTBJRJ267QSK36C3LFZV7FPAYDDAXGIWDTNEGNV6MRRA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:11:05.567692582 (75h47m41.131015574s) [noise=true , address=104.36.82.49:4982 , piece_id=UNLV5UDNWULEIPE4QSQHHZQPK3BQKFRB5T3H6GP7ZQ5QGSNK5V6A, seg_pos={0 0}, upload=60693]: Done uploading piece. err: ecclient: upload failed (node:1NxoYJXCXmTvQncbkXt6yH7mhEBBrzVdzhgL5J6W5TD4N3pkaG, address:104.36.82.49:4982): protocol: expected piece hash; context deadline exceeded; EOF Jun 21 20:11:05.567759198 (75h47m41.131082195s) [noise=true , address=104.36.82.54:4171 , piece_id=MUMABKARK4KKUKWW6XDXCQVT6O7SLFHGHE7COWC7QIYGARAA2D5A, seg_pos={0 0}, upload=60693]: Done uploading piece. err: ecclient: upload failed (node:125SMhDsEZruNu6LkmQn2M7R6WzvXTdk86DRsKf522u8BNiANq3, address:104.36.82.54:4171): protocol: expected piece hash; context deadline exceeded; EOF Jun 21 20:11:05.567869019 (75h47m41.131192033s) [noise=true , address=104.36.82.10:6087 , piece_id=5FQG7BPZC373PFB54HYXSG2SVKOBBBSGZCLC4XDJEYQFOLQY74MA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: ecclient: upload failed (node:1S9k5eejJcxkiG5pXjjBZm18uP8k5jCGfWyeW2QQxmuRU8ZzRA, address:104.36.82.10:6087): protocol: write tcp 159.223.143.39:38550->104.36.82.10:6087: use of closed network connection; write tcp 159.223.143.39:38550->104.36.82.10:6087: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:38550->104.36.82.10:6087: use of closed network connection Jun 21 20:11:05.567932150 (75h47m41.131255161s) [seg_pos={0 0}, upload=60693]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jun 21 20:11:05.593626175 (75h47m41.156949181s) [upload=60693]: Done uploading segments... Jun 21 20:11:05.593657274 (75h47m41.156980265s) [upload=60693]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 21.522636711s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.......X....................................................] duration: 18.164311138s FAIL (2025-06-21 20:49:39 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== pload=60693]: Starting upload Jun 21 20:10:05.602866542 (75h46m41.166189557s) [upload=60693]: Done starting upload Jun 21 20:10:05.603116545 (75h46m41.166439547s) [upload=60693]: Uploading segments... Jun 21 20:10:05.603495389 (75h46m41.166818397s) [upload=60693]: Got next segment. Inline: false Jun 21 20:10:05.603566566 (75h46m41.166889558s) [upload=60693]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 21 20:10:05.603573280 (75h46m41.166896286s) [upload=60693]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 21 20:10:05.639130162 (75h46m41.202453164s) [seg_pos={0 0}, upload=60693]: Begin upload segment... Jun 21 20:10:05.639231070 (75h46m41.202554057s) [seg_pos={0 0}, upload=60693]: Done begin upload segment. Jun 21 20:10:05.639292724 (75h46m41.202615728s) [noise=true , address=104.36.82.56:4253 , piece_id=5PMVGLZETMXCQ77K3I6PXACRIMBGR2CV5JGUSU5KUMYFNXVGA2GA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.639338066 (75h46m41.202661071s) [noise=true , address=104.36.82.53:4126 , piece_id=2MKRQJEPGDJKDM7VWY5M4M6NORILAIEYYATJ7HVZEOBOMXOI2LBA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.639416264 (75h46m41.202739267s) [noise=true , address=104.36.82.5:5869 , piece_id=HAEHXXZASV3DHT3QN7NTZOS6XZKHNUWBKCHWJJWSTJKXAKJKKBAA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.639873098 (75h46m41.203196115s) [noise=true , address=104.36.82.52:4111 , piece_id=HB7QAWN6CGNIIT4HD2GZ7JDQP5NXIJ5I2USBM66YOKTYENNOWG5A, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.639887296 (75h46m41.203210304s) [noise=true , address=104.36.82.10:6087 , piece_id=5FQG7BPZC373PFB54HYXSG2SVKOBBBSGZCLC4XDJEYQFOLQY74MA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.640266623 (75h46m41.203589616s) [noise=true , address=104.36.82.13:6196 , piece_id=BCHMQO5KKT5BLNWQGOHH3V43LGSE3BUQCI2SY4TTO5M5GQNOCTMQ, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.640352274 (75h46m41.203675286s) [noise=true , address=104.36.82.37:6663 , piece_id=X7LWYIIHJFU2T2PHYR3R5TNT22UUPHEHQWZH7ZJMLS4AH4PJ6AIA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.640659767 (75h46m41.203982779s) [noise=true , address=104.36.82.42:6807 , piece_id=ZBQZPXUDJ4VBCOSVKWAJ6SK6UD7SLPON7Y4SSCK2DQ5STFG4XCNQ, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.640961701 (75h46m41.20428471s) [noise=true , address=45.91.116.6:4982 , piece_id=VP45R6KXNH5U5SGUQUMAPIBY7KFDPBIR3S3VRIX27EAT6VLFBZ7Q, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.641159578 (75h46m41.204482589s) [noise=true , address=104.36.82.9:6039 , piece_id=I4AB6N27MYYUHWAZLJJUIBFBLOQFHDIM3YLYWPVRPNUV7VMYS2SA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.641588126 (75h46m41.204911136s) [noise=true , address=77.90.31.19:4853 , piece_id=VFEON5T2KSWM2KSLTVRZTXYL42JWENZYUEUPG3LMODGZWBMVGGPQ, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.641863297 (75h46m41.205186305s) [noise=true , address=77.90.31.12:4603 , piece_id=BMUIC6KNGEDL37436GLSXW5XOO6VYOPNOSFGQCYVHSENVI2PJ7KQ, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.642124427 (75h46m41.205447437s) [noise=true , address=104.36.82.49:4982 , piece_id=UNLV5UDNWULEIPE4QSQHHZQPK3BQKFRB5T3H6GP7ZQ5QGSNK5V6A, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.642285917 (75h46m41.205608925s) [noise=true , address=104.36.82.54:4171 , piece_id=MUMABKARK4KKUKWW6XDXCQVT6O7SLFHGHE7COWC7QIYGARAA2D5A, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.642656750 (75h46m41.205979758s) [noise=true , address=104.36.82.14:6232 , piece_id=A2ZLUUZKRTBJRJ267QSK36C3LFZV7FPAYDDAXGIWDTNEGNV6MRRA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.643565583 (75h46m41.206888597s) [noise=true , address=79.127.205.245:4231 , piece_id=EJK46XUUVI5NWOAQJEW4QIJG3KRXZHB4YOQUJRCS2MIOSWKGTNLQ, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.644620291 (75h46m41.207943303s) [noise=true , address=77.90.31.14:4670 , piece_id=7FNANMUKPM3N6YWRAOAIM4WJZJUTNF7ZBI6ZUBNEPGNGMFWOE24A, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.648842639 (75h46m41.212165631s) [noise=true , address=104.36.82.55:4224 , piece_id=WRVNGSJV5BX4GJERLXNPAS7J7BZ5WFTM7R5H257YRCG5SP2HIYMA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.825828239 (75h46m41.389151241s) [noise=true , address=104.36.82.53:4126 , piece_id=2MKRQJEPGDJKDM7VWY5M4M6NORILAIEYYATJ7HVZEOBOMXOI2LBA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: ecclient: upload failed (node:12dMWFrq2NMpnMzuPAphWerntC5kkrcnzRJZBA116uzJzyvk6fH, address:104.36.82.53:4126): protocol: storage node overloaded, request limit: 2000 Jun 21 20:10:05.834666043 (75h46m41.397989033s) [upload=60693]: Next returned nil segment Jun 21 20:10:05.834686295 (75h46m41.398009302s) [upload=60693]: Waiting for error group managing segments... Jun 21 20:10:05.877010488 (75h46m41.440333486s) [noise=true , address=45.91.116.6:4982 , piece_id=VP45R6KXNH5U5SGUQUMAPIBY7KFDPBIR3S3VRIX27EAT6VLFBZ7Q, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:05.886336964 (75h46m41.449659967s) [noise=true , address=104.36.82.9:6039 , piece_id=I4AB6N27MYYUHWAZLJJUIBFBLOQFHDIM3YLYWPVRPNUV7VMYS2SA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: ecclient: upload failed (node:12P11WVW9mVXw5ra2gZ3Rezon6wZi8GrN5CchQN3MiVU4ts3L6V, address:104.36.82.9:6039): protocol: storage node overloaded, request limit: 2000 Jun 21 20:10:08.064514542 (75h46m43.627837553s) [noise=true , address=77.90.31.14:4670 , piece_id=7FNANMUKPM3N6YWRAOAIM4WJZJUTNF7ZBI6ZUBNEPGNGMFWOE24A, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:08.099098391 (75h46m43.662421396s) [noise=true , address=79.127.205.245:4231 , piece_id=EJK46XUUVI5NWOAQJEW4QIJG3KRXZHB4YOQUJRCS2MIOSWKGTNLQ, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:08.489461749 (75h46m44.052784752s) [noise=true , address=104.36.82.42:6807 , piece_id=ZBQZPXUDJ4VBCOSVKWAJ6SK6UD7SLPON7Y4SSCK2DQ5STFG4XCNQ, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:08.650922093 (75h46m44.214245093s) [noise=true , address=77.90.31.19:4853 , piece_id=VFEON5T2KSWM2KSLTVRZTXYL42JWENZYUEUPG3LMODGZWBMVGGPQ, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:09.917746439 (75h46m45.481069441s) [noise=true , address=104.36.82.13:6196 , piece_id=BCHMQO5KKT5BLNWQGOHH3V43LGSE3BUQCI2SY4TTO5M5GQNOCTMQ, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:10.830722215 (75h46m46.394045216s) [noise=true , address=104.36.82.5:5869 , piece_id=HAEHXXZASV3DHT3QN7NTZOS6XZKHNUWBKCHWJJWSTJKXAKJKKBAA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:12.738783450 (75h46m48.302106452s) [noise=true , address=77.90.31.12:4603 , piece_id=BMUIC6KNGEDL37436GLSXW5XOO6VYOPNOSFGQCYVHSENVI2PJ7KQ, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:15.460927713 (75h46m51.024250723s) [noise=true , address=104.36.82.52:4111 , piece_id=HB7QAWN6CGNIIT4HD2GZ7JDQP5NXIJ5I2USBM66YOKTYENNOWG5A, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:18.393817968 (75h46m53.95714096s) [noise=true , address=104.36.82.55:4224 , piece_id=WRVNGSJV5BX4GJERLXNPAS7J7BZ5WFTM7R5H257YRCG5SP2HIYMA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:19.549174143 (75h46m55.112497151s) [noise=true , address=104.36.82.56:4253 , piece_id=5PMVGLZETMXCQ77K3I6PXACRIMBGR2CV5JGUSU5KUMYFNXVGA2GA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:22.511944593 (75h46m58.075267584s) [noise=true , address=104.36.82.37:6663 , piece_id=X7LWYIIHJFU2T2PHYR3R5TNT22UUPHEHQWZH7ZJMLS4AH4PJ6AIA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:48.027197558 (75h47m23.590520562s) [noise=true , address=104.36.82.14:6232 , piece_id=A2ZLUUZKRTBJRJ267QSK36C3LFZV7FPAYDDAXGIWDTNEGNV6MRRA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:11:05.567692582 (75h47m41.131015574s) [noise=true , address=104.36.82.49:4982 , piece_id=UNLV5UDNWULEIPE4QSQHHZQPK3BQKFRB5T3H6GP7ZQ5QGSNK5V6A, seg_pos={0 0}, upload=60693]: Done uploading piece. err: ecclient: upload failed (node:1NxoYJXCXmTvQncbkXt6yH7mhEBBrzVdzhgL5J6W5TD4N3pkaG, address:104.36.82.49:4982): protocol: expected piece hash; context deadline exceeded; EOF Jun 21 20:11:05.567759198 (75h47m41.131082195s) [noise=true , address=104.36.82.54:4171 , piece_id=MUMABKARK4KKUKWW6XDXCQVT6O7SLFHGHE7COWC7QIYGARAA2D5A, seg_pos={0 0}, upload=60693]: Done uploading piece. err: ecclient: upload failed (node:125SMhDsEZruNu6LkmQn2M7R6WzvXTdk86DRsKf522u8BNiANq3, address:104.36.82.54:4171): protocol: expected piece hash; context deadline exceeded; EOF Jun 21 20:11:05.567869019 (75h47m41.131192033s) [noise=true , address=104.36.82.10:6087 , piece_id=5FQG7BPZC373PFB54HYXSG2SVKOBBBSGZCLC4XDJEYQFOLQY74MA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: ecclient: upload failed (node:1S9k5eejJcxkiG5pXjjBZm18uP8k5jCGfWyeW2QQxmuRU8ZzRA, address:104.36.82.10:6087): protocol: write tcp 159.223.143.39:38550->104.36.82.10:6087: use of closed network connection; write tcp 159.223.143.39:38550->104.36.82.10:6087: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:38550->104.36.82.10:6087: use of closed network connection Jun 21 20:11:05.567932150 (75h47m41.131255161s) [seg_pos={0 0}, upload=60693]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jun 21 20:11:05.593626175 (75h47m41.156949181s) [upload=60693]: Done uploading segments... Jun 21 20:11:05.593657274 (75h47m41.156980265s) [upload=60693]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 18.164339559s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [........X...................................................] duration: 22.628425475s FAIL (2025-06-21 20:49:02 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== pload=60693]: Starting upload Jun 21 20:10:05.602866542 (75h46m41.166189557s) [upload=60693]: Done starting upload Jun 21 20:10:05.603116545 (75h46m41.166439547s) [upload=60693]: Uploading segments... Jun 21 20:10:05.603495389 (75h46m41.166818397s) [upload=60693]: Got next segment. Inline: false Jun 21 20:10:05.603566566 (75h46m41.166889558s) [upload=60693]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 21 20:10:05.603573280 (75h46m41.166896286s) [upload=60693]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 21 20:10:05.639130162 (75h46m41.202453164s) [seg_pos={0 0}, upload=60693]: Begin upload segment... Jun 21 20:10:05.639231070 (75h46m41.202554057s) [seg_pos={0 0}, upload=60693]: Done begin upload segment. Jun 21 20:10:05.639292724 (75h46m41.202615728s) [noise=true , address=104.36.82.56:4253 , piece_id=5PMVGLZETMXCQ77K3I6PXACRIMBGR2CV5JGUSU5KUMYFNXVGA2GA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.639338066 (75h46m41.202661071s) [noise=true , address=104.36.82.53:4126 , piece_id=2MKRQJEPGDJKDM7VWY5M4M6NORILAIEYYATJ7HVZEOBOMXOI2LBA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.639416264 (75h46m41.202739267s) [noise=true , address=104.36.82.5:5869 , piece_id=HAEHXXZASV3DHT3QN7NTZOS6XZKHNUWBKCHWJJWSTJKXAKJKKBAA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.639873098 (75h46m41.203196115s) [noise=true , address=104.36.82.52:4111 , piece_id=HB7QAWN6CGNIIT4HD2GZ7JDQP5NXIJ5I2USBM66YOKTYENNOWG5A, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.639887296 (75h46m41.203210304s) [noise=true , address=104.36.82.10:6087 , piece_id=5FQG7BPZC373PFB54HYXSG2SVKOBBBSGZCLC4XDJEYQFOLQY74MA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.640266623 (75h46m41.203589616s) [noise=true , address=104.36.82.13:6196 , piece_id=BCHMQO5KKT5BLNWQGOHH3V43LGSE3BUQCI2SY4TTO5M5GQNOCTMQ, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.640352274 (75h46m41.203675286s) [noise=true , address=104.36.82.37:6663 , piece_id=X7LWYIIHJFU2T2PHYR3R5TNT22UUPHEHQWZH7ZJMLS4AH4PJ6AIA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.640659767 (75h46m41.203982779s) [noise=true , address=104.36.82.42:6807 , piece_id=ZBQZPXUDJ4VBCOSVKWAJ6SK6UD7SLPON7Y4SSCK2DQ5STFG4XCNQ, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.640961701 (75h46m41.20428471s) [noise=true , address=45.91.116.6:4982 , piece_id=VP45R6KXNH5U5SGUQUMAPIBY7KFDPBIR3S3VRIX27EAT6VLFBZ7Q, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.641159578 (75h46m41.204482589s) [noise=true , address=104.36.82.9:6039 , piece_id=I4AB6N27MYYUHWAZLJJUIBFBLOQFHDIM3YLYWPVRPNUV7VMYS2SA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.641588126 (75h46m41.204911136s) [noise=true , address=77.90.31.19:4853 , piece_id=VFEON5T2KSWM2KSLTVRZTXYL42JWENZYUEUPG3LMODGZWBMVGGPQ, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.641863297 (75h46m41.205186305s) [noise=true , address=77.90.31.12:4603 , piece_id=BMUIC6KNGEDL37436GLSXW5XOO6VYOPNOSFGQCYVHSENVI2PJ7KQ, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.642124427 (75h46m41.205447437s) [noise=true , address=104.36.82.49:4982 , piece_id=UNLV5UDNWULEIPE4QSQHHZQPK3BQKFRB5T3H6GP7ZQ5QGSNK5V6A, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.642285917 (75h46m41.205608925s) [noise=true , address=104.36.82.54:4171 , piece_id=MUMABKARK4KKUKWW6XDXCQVT6O7SLFHGHE7COWC7QIYGARAA2D5A, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.642656750 (75h46m41.205979758s) [noise=true , address=104.36.82.14:6232 , piece_id=A2ZLUUZKRTBJRJ267QSK36C3LFZV7FPAYDDAXGIWDTNEGNV6MRRA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.643565583 (75h46m41.206888597s) [noise=true , address=79.127.205.245:4231 , piece_id=EJK46XUUVI5NWOAQJEW4QIJG3KRXZHB4YOQUJRCS2MIOSWKGTNLQ, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.644620291 (75h46m41.207943303s) [noise=true , address=77.90.31.14:4670 , piece_id=7FNANMUKPM3N6YWRAOAIM4WJZJUTNF7ZBI6ZUBNEPGNGMFWOE24A, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.648842639 (75h46m41.212165631s) [noise=true , address=104.36.82.55:4224 , piece_id=WRVNGSJV5BX4GJERLXNPAS7J7BZ5WFTM7R5H257YRCG5SP2HIYMA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.825828239 (75h46m41.389151241s) [noise=true , address=104.36.82.53:4126 , piece_id=2MKRQJEPGDJKDM7VWY5M4M6NORILAIEYYATJ7HVZEOBOMXOI2LBA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: ecclient: upload failed (node:12dMWFrq2NMpnMzuPAphWerntC5kkrcnzRJZBA116uzJzyvk6fH, address:104.36.82.53:4126): protocol: storage node overloaded, request limit: 2000 Jun 21 20:10:05.834666043 (75h46m41.397989033s) [upload=60693]: Next returned nil segment Jun 21 20:10:05.834686295 (75h46m41.398009302s) [upload=60693]: Waiting for error group managing segments... Jun 21 20:10:05.877010488 (75h46m41.440333486s) [noise=true , address=45.91.116.6:4982 , piece_id=VP45R6KXNH5U5SGUQUMAPIBY7KFDPBIR3S3VRIX27EAT6VLFBZ7Q, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:05.886336964 (75h46m41.449659967s) [noise=true , address=104.36.82.9:6039 , piece_id=I4AB6N27MYYUHWAZLJJUIBFBLOQFHDIM3YLYWPVRPNUV7VMYS2SA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: ecclient: upload failed (node:12P11WVW9mVXw5ra2gZ3Rezon6wZi8GrN5CchQN3MiVU4ts3L6V, address:104.36.82.9:6039): protocol: storage node overloaded, request limit: 2000 Jun 21 20:10:08.064514542 (75h46m43.627837553s) [noise=true , address=77.90.31.14:4670 , piece_id=7FNANMUKPM3N6YWRAOAIM4WJZJUTNF7ZBI6ZUBNEPGNGMFWOE24A, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:08.099098391 (75h46m43.662421396s) [noise=true , address=79.127.205.245:4231 , piece_id=EJK46XUUVI5NWOAQJEW4QIJG3KRXZHB4YOQUJRCS2MIOSWKGTNLQ, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:08.489461749 (75h46m44.052784752s) [noise=true , address=104.36.82.42:6807 , piece_id=ZBQZPXUDJ4VBCOSVKWAJ6SK6UD7SLPON7Y4SSCK2DQ5STFG4XCNQ, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:08.650922093 (75h46m44.214245093s) [noise=true , address=77.90.31.19:4853 , piece_id=VFEON5T2KSWM2KSLTVRZTXYL42JWENZYUEUPG3LMODGZWBMVGGPQ, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:09.917746439 (75h46m45.481069441s) [noise=true , address=104.36.82.13:6196 , piece_id=BCHMQO5KKT5BLNWQGOHH3V43LGSE3BUQCI2SY4TTO5M5GQNOCTMQ, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:10.830722215 (75h46m46.394045216s) [noise=true , address=104.36.82.5:5869 , piece_id=HAEHXXZASV3DHT3QN7NTZOS6XZKHNUWBKCHWJJWSTJKXAKJKKBAA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:12.738783450 (75h46m48.302106452s) [noise=true , address=77.90.31.12:4603 , piece_id=BMUIC6KNGEDL37436GLSXW5XOO6VYOPNOSFGQCYVHSENVI2PJ7KQ, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:15.460927713 (75h46m51.024250723s) [noise=true , address=104.36.82.52:4111 , piece_id=HB7QAWN6CGNIIT4HD2GZ7JDQP5NXIJ5I2USBM66YOKTYENNOWG5A, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:18.393817968 (75h46m53.95714096s) [noise=true , address=104.36.82.55:4224 , piece_id=WRVNGSJV5BX4GJERLXNPAS7J7BZ5WFTM7R5H257YRCG5SP2HIYMA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:19.549174143 (75h46m55.112497151s) [noise=true , address=104.36.82.56:4253 , piece_id=5PMVGLZETMXCQ77K3I6PXACRIMBGR2CV5JGUSU5KUMYFNXVGA2GA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:22.511944593 (75h46m58.075267584s) [noise=true , address=104.36.82.37:6663 , piece_id=X7LWYIIHJFU2T2PHYR3R5TNT22UUPHEHQWZH7ZJMLS4AH4PJ6AIA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:48.027197558 (75h47m23.590520562s) [noise=true , address=104.36.82.14:6232 , piece_id=A2ZLUUZKRTBJRJ267QSK36C3LFZV7FPAYDDAXGIWDTNEGNV6MRRA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:11:05.567692582 (75h47m41.131015574s) [noise=true , address=104.36.82.49:4982 , piece_id=UNLV5UDNWULEIPE4QSQHHZQPK3BQKFRB5T3H6GP7ZQ5QGSNK5V6A, seg_pos={0 0}, upload=60693]: Done uploading piece. err: ecclient: upload failed (node:1NxoYJXCXmTvQncbkXt6yH7mhEBBrzVdzhgL5J6W5TD4N3pkaG, address:104.36.82.49:4982): protocol: expected piece hash; context deadline exceeded; EOF Jun 21 20:11:05.567759198 (75h47m41.131082195s) [noise=true , address=104.36.82.54:4171 , piece_id=MUMABKARK4KKUKWW6XDXCQVT6O7SLFHGHE7COWC7QIYGARAA2D5A, seg_pos={0 0}, upload=60693]: Done uploading piece. err: ecclient: upload failed (node:125SMhDsEZruNu6LkmQn2M7R6WzvXTdk86DRsKf522u8BNiANq3, address:104.36.82.54:4171): protocol: expected piece hash; context deadline exceeded; EOF Jun 21 20:11:05.567869019 (75h47m41.131192033s) [noise=true , address=104.36.82.10:6087 , piece_id=5FQG7BPZC373PFB54HYXSG2SVKOBBBSGZCLC4XDJEYQFOLQY74MA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: ecclient: upload failed (node:1S9k5eejJcxkiG5pXjjBZm18uP8k5jCGfWyeW2QQxmuRU8ZzRA, address:104.36.82.10:6087): protocol: write tcp 159.223.143.39:38550->104.36.82.10:6087: use of closed network connection; write tcp 159.223.143.39:38550->104.36.82.10:6087: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:38550->104.36.82.10:6087: use of closed network connection Jun 21 20:11:05.567932150 (75h47m41.131255161s) [seg_pos={0 0}, upload=60693]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jun 21 20:11:05.593626175 (75h47m41.156949181s) [upload=60693]: Done uploading segments... Jun 21 20:11:05.593657274 (75h47m41.156980265s) [upload=60693]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 22.628477319s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.........X..................................................] duration: 31.303766251s FAIL (2025-06-21 20:48:39 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== pload=60693]: Starting upload Jun 21 20:10:05.602866542 (75h46m41.166189557s) [upload=60693]: Done starting upload Jun 21 20:10:05.603116545 (75h46m41.166439547s) [upload=60693]: Uploading segments... Jun 21 20:10:05.603495389 (75h46m41.166818397s) [upload=60693]: Got next segment. Inline: false Jun 21 20:10:05.603566566 (75h46m41.166889558s) [upload=60693]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 21 20:10:05.603573280 (75h46m41.166896286s) [upload=60693]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 21 20:10:05.639130162 (75h46m41.202453164s) [seg_pos={0 0}, upload=60693]: Begin upload segment... Jun 21 20:10:05.639231070 (75h46m41.202554057s) [seg_pos={0 0}, upload=60693]: Done begin upload segment. Jun 21 20:10:05.639292724 (75h46m41.202615728s) [noise=true , address=104.36.82.56:4253 , piece_id=5PMVGLZETMXCQ77K3I6PXACRIMBGR2CV5JGUSU5KUMYFNXVGA2GA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.639338066 (75h46m41.202661071s) [noise=true , address=104.36.82.53:4126 , piece_id=2MKRQJEPGDJKDM7VWY5M4M6NORILAIEYYATJ7HVZEOBOMXOI2LBA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.639416264 (75h46m41.202739267s) [noise=true , address=104.36.82.5:5869 , piece_id=HAEHXXZASV3DHT3QN7NTZOS6XZKHNUWBKCHWJJWSTJKXAKJKKBAA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.639873098 (75h46m41.203196115s) [noise=true , address=104.36.82.52:4111 , piece_id=HB7QAWN6CGNIIT4HD2GZ7JDQP5NXIJ5I2USBM66YOKTYENNOWG5A, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.639887296 (75h46m41.203210304s) [noise=true , address=104.36.82.10:6087 , piece_id=5FQG7BPZC373PFB54HYXSG2SVKOBBBSGZCLC4XDJEYQFOLQY74MA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.640266623 (75h46m41.203589616s) [noise=true , address=104.36.82.13:6196 , piece_id=BCHMQO5KKT5BLNWQGOHH3V43LGSE3BUQCI2SY4TTO5M5GQNOCTMQ, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.640352274 (75h46m41.203675286s) [noise=true , address=104.36.82.37:6663 , piece_id=X7LWYIIHJFU2T2PHYR3R5TNT22UUPHEHQWZH7ZJMLS4AH4PJ6AIA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.640659767 (75h46m41.203982779s) [noise=true , address=104.36.82.42:6807 , piece_id=ZBQZPXUDJ4VBCOSVKWAJ6SK6UD7SLPON7Y4SSCK2DQ5STFG4XCNQ, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.640961701 (75h46m41.20428471s) [noise=true , address=45.91.116.6:4982 , piece_id=VP45R6KXNH5U5SGUQUMAPIBY7KFDPBIR3S3VRIX27EAT6VLFBZ7Q, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.641159578 (75h46m41.204482589s) [noise=true , address=104.36.82.9:6039 , piece_id=I4AB6N27MYYUHWAZLJJUIBFBLOQFHDIM3YLYWPVRPNUV7VMYS2SA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.641588126 (75h46m41.204911136s) [noise=true , address=77.90.31.19:4853 , piece_id=VFEON5T2KSWM2KSLTVRZTXYL42JWENZYUEUPG3LMODGZWBMVGGPQ, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.641863297 (75h46m41.205186305s) [noise=true , address=77.90.31.12:4603 , piece_id=BMUIC6KNGEDL37436GLSXW5XOO6VYOPNOSFGQCYVHSENVI2PJ7KQ, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.642124427 (75h46m41.205447437s) [noise=true , address=104.36.82.49:4982 , piece_id=UNLV5UDNWULEIPE4QSQHHZQPK3BQKFRB5T3H6GP7ZQ5QGSNK5V6A, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.642285917 (75h46m41.205608925s) [noise=true , address=104.36.82.54:4171 , piece_id=MUMABKARK4KKUKWW6XDXCQVT6O7SLFHGHE7COWC7QIYGARAA2D5A, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.642656750 (75h46m41.205979758s) [noise=true , address=104.36.82.14:6232 , piece_id=A2ZLUUZKRTBJRJ267QSK36C3LFZV7FPAYDDAXGIWDTNEGNV6MRRA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.643565583 (75h46m41.206888597s) [noise=true , address=79.127.205.245:4231 , piece_id=EJK46XUUVI5NWOAQJEW4QIJG3KRXZHB4YOQUJRCS2MIOSWKGTNLQ, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.644620291 (75h46m41.207943303s) [noise=true , address=77.90.31.14:4670 , piece_id=7FNANMUKPM3N6YWRAOAIM4WJZJUTNF7ZBI6ZUBNEPGNGMFWOE24A, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.648842639 (75h46m41.212165631s) [noise=true , address=104.36.82.55:4224 , piece_id=WRVNGSJV5BX4GJERLXNPAS7J7BZ5WFTM7R5H257YRCG5SP2HIYMA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.825828239 (75h46m41.389151241s) [noise=true , address=104.36.82.53:4126 , piece_id=2MKRQJEPGDJKDM7VWY5M4M6NORILAIEYYATJ7HVZEOBOMXOI2LBA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: ecclient: upload failed (node:12dMWFrq2NMpnMzuPAphWerntC5kkrcnzRJZBA116uzJzyvk6fH, address:104.36.82.53:4126): protocol: storage node overloaded, request limit: 2000 Jun 21 20:10:05.834666043 (75h46m41.397989033s) [upload=60693]: Next returned nil segment Jun 21 20:10:05.834686295 (75h46m41.398009302s) [upload=60693]: Waiting for error group managing segments... Jun 21 20:10:05.877010488 (75h46m41.440333486s) [noise=true , address=45.91.116.6:4982 , piece_id=VP45R6KXNH5U5SGUQUMAPIBY7KFDPBIR3S3VRIX27EAT6VLFBZ7Q, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:05.886336964 (75h46m41.449659967s) [noise=true , address=104.36.82.9:6039 , piece_id=I4AB6N27MYYUHWAZLJJUIBFBLOQFHDIM3YLYWPVRPNUV7VMYS2SA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: ecclient: upload failed (node:12P11WVW9mVXw5ra2gZ3Rezon6wZi8GrN5CchQN3MiVU4ts3L6V, address:104.36.82.9:6039): protocol: storage node overloaded, request limit: 2000 Jun 21 20:10:08.064514542 (75h46m43.627837553s) [noise=true , address=77.90.31.14:4670 , piece_id=7FNANMUKPM3N6YWRAOAIM4WJZJUTNF7ZBI6ZUBNEPGNGMFWOE24A, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:08.099098391 (75h46m43.662421396s) [noise=true , address=79.127.205.245:4231 , piece_id=EJK46XUUVI5NWOAQJEW4QIJG3KRXZHB4YOQUJRCS2MIOSWKGTNLQ, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:08.489461749 (75h46m44.052784752s) [noise=true , address=104.36.82.42:6807 , piece_id=ZBQZPXUDJ4VBCOSVKWAJ6SK6UD7SLPON7Y4SSCK2DQ5STFG4XCNQ, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:08.650922093 (75h46m44.214245093s) [noise=true , address=77.90.31.19:4853 , piece_id=VFEON5T2KSWM2KSLTVRZTXYL42JWENZYUEUPG3LMODGZWBMVGGPQ, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:09.917746439 (75h46m45.481069441s) [noise=true , address=104.36.82.13:6196 , piece_id=BCHMQO5KKT5BLNWQGOHH3V43LGSE3BUQCI2SY4TTO5M5GQNOCTMQ, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:10.830722215 (75h46m46.394045216s) [noise=true , address=104.36.82.5:5869 , piece_id=HAEHXXZASV3DHT3QN7NTZOS6XZKHNUWBKCHWJJWSTJKXAKJKKBAA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:12.738783450 (75h46m48.302106452s) [noise=true , address=77.90.31.12:4603 , piece_id=BMUIC6KNGEDL37436GLSXW5XOO6VYOPNOSFGQCYVHSENVI2PJ7KQ, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:15.460927713 (75h46m51.024250723s) [noise=true , address=104.36.82.52:4111 , piece_id=HB7QAWN6CGNIIT4HD2GZ7JDQP5NXIJ5I2USBM66YOKTYENNOWG5A, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:18.393817968 (75h46m53.95714096s) [noise=true , address=104.36.82.55:4224 , piece_id=WRVNGSJV5BX4GJERLXNPAS7J7BZ5WFTM7R5H257YRCG5SP2HIYMA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:19.549174143 (75h46m55.112497151s) [noise=true , address=104.36.82.56:4253 , piece_id=5PMVGLZETMXCQ77K3I6PXACRIMBGR2CV5JGUSU5KUMYFNXVGA2GA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:22.511944593 (75h46m58.075267584s) [noise=true , address=104.36.82.37:6663 , piece_id=X7LWYIIHJFU2T2PHYR3R5TNT22UUPHEHQWZH7ZJMLS4AH4PJ6AIA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:48.027197558 (75h47m23.590520562s) [noise=true , address=104.36.82.14:6232 , piece_id=A2ZLUUZKRTBJRJ267QSK36C3LFZV7FPAYDDAXGIWDTNEGNV6MRRA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:11:05.567692582 (75h47m41.131015574s) [noise=true , address=104.36.82.49:4982 , piece_id=UNLV5UDNWULEIPE4QSQHHZQPK3BQKFRB5T3H6GP7ZQ5QGSNK5V6A, seg_pos={0 0}, upload=60693]: Done uploading piece. err: ecclient: upload failed (node:1NxoYJXCXmTvQncbkXt6yH7mhEBBrzVdzhgL5J6W5TD4N3pkaG, address:104.36.82.49:4982): protocol: expected piece hash; context deadline exceeded; EOF Jun 21 20:11:05.567759198 (75h47m41.131082195s) [noise=true , address=104.36.82.54:4171 , piece_id=MUMABKARK4KKUKWW6XDXCQVT6O7SLFHGHE7COWC7QIYGARAA2D5A, seg_pos={0 0}, upload=60693]: Done uploading piece. err: ecclient: upload failed (node:125SMhDsEZruNu6LkmQn2M7R6WzvXTdk86DRsKf522u8BNiANq3, address:104.36.82.54:4171): protocol: expected piece hash; context deadline exceeded; EOF Jun 21 20:11:05.567869019 (75h47m41.131192033s) [noise=true , address=104.36.82.10:6087 , piece_id=5FQG7BPZC373PFB54HYXSG2SVKOBBBSGZCLC4XDJEYQFOLQY74MA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: ecclient: upload failed (node:1S9k5eejJcxkiG5pXjjBZm18uP8k5jCGfWyeW2QQxmuRU8ZzRA, address:104.36.82.10:6087): protocol: write tcp 159.223.143.39:38550->104.36.82.10:6087: use of closed network connection; write tcp 159.223.143.39:38550->104.36.82.10:6087: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:38550->104.36.82.10:6087: use of closed network connection Jun 21 20:11:05.567932150 (75h47m41.131255161s) [seg_pos={0 0}, upload=60693]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jun 21 20:11:05.593626175 (75h47m41.156949181s) [upload=60693]: Done uploading segments... Jun 21 20:11:05.593657274 (75h47m41.156980265s) [upload=60693]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 31.303798338s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [..........X.................................................] duration: 22.348298667s FAIL (2025-06-21 20:47:51 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== pload=60693]: Starting upload Jun 21 20:10:05.602866542 (75h46m41.166189557s) [upload=60693]: Done starting upload Jun 21 20:10:05.603116545 (75h46m41.166439547s) [upload=60693]: Uploading segments... Jun 21 20:10:05.603495389 (75h46m41.166818397s) [upload=60693]: Got next segment. Inline: false Jun 21 20:10:05.603566566 (75h46m41.166889558s) [upload=60693]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 21 20:10:05.603573280 (75h46m41.166896286s) [upload=60693]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 21 20:10:05.639130162 (75h46m41.202453164s) [seg_pos={0 0}, upload=60693]: Begin upload segment... Jun 21 20:10:05.639231070 (75h46m41.202554057s) [seg_pos={0 0}, upload=60693]: Done begin upload segment. Jun 21 20:10:05.639292724 (75h46m41.202615728s) [noise=true , address=104.36.82.56:4253 , piece_id=5PMVGLZETMXCQ77K3I6PXACRIMBGR2CV5JGUSU5KUMYFNXVGA2GA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.639338066 (75h46m41.202661071s) [noise=true , address=104.36.82.53:4126 , piece_id=2MKRQJEPGDJKDM7VWY5M4M6NORILAIEYYATJ7HVZEOBOMXOI2LBA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.639416264 (75h46m41.202739267s) [noise=true , address=104.36.82.5:5869 , piece_id=HAEHXXZASV3DHT3QN7NTZOS6XZKHNUWBKCHWJJWSTJKXAKJKKBAA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.639873098 (75h46m41.203196115s) [noise=true , address=104.36.82.52:4111 , piece_id=HB7QAWN6CGNIIT4HD2GZ7JDQP5NXIJ5I2USBM66YOKTYENNOWG5A, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.639887296 (75h46m41.203210304s) [noise=true , address=104.36.82.10:6087 , piece_id=5FQG7BPZC373PFB54HYXSG2SVKOBBBSGZCLC4XDJEYQFOLQY74MA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.640266623 (75h46m41.203589616s) [noise=true , address=104.36.82.13:6196 , piece_id=BCHMQO5KKT5BLNWQGOHH3V43LGSE3BUQCI2SY4TTO5M5GQNOCTMQ, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.640352274 (75h46m41.203675286s) [noise=true , address=104.36.82.37:6663 , piece_id=X7LWYIIHJFU2T2PHYR3R5TNT22UUPHEHQWZH7ZJMLS4AH4PJ6AIA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.640659767 (75h46m41.203982779s) [noise=true , address=104.36.82.42:6807 , piece_id=ZBQZPXUDJ4VBCOSVKWAJ6SK6UD7SLPON7Y4SSCK2DQ5STFG4XCNQ, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.640961701 (75h46m41.20428471s) [noise=true , address=45.91.116.6:4982 , piece_id=VP45R6KXNH5U5SGUQUMAPIBY7KFDPBIR3S3VRIX27EAT6VLFBZ7Q, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.641159578 (75h46m41.204482589s) [noise=true , address=104.36.82.9:6039 , piece_id=I4AB6N27MYYUHWAZLJJUIBFBLOQFHDIM3YLYWPVRPNUV7VMYS2SA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.641588126 (75h46m41.204911136s) [noise=true , address=77.90.31.19:4853 , piece_id=VFEON5T2KSWM2KSLTVRZTXYL42JWENZYUEUPG3LMODGZWBMVGGPQ, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.641863297 (75h46m41.205186305s) [noise=true , address=77.90.31.12:4603 , piece_id=BMUIC6KNGEDL37436GLSXW5XOO6VYOPNOSFGQCYVHSENVI2PJ7KQ, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.642124427 (75h46m41.205447437s) [noise=true , address=104.36.82.49:4982 , piece_id=UNLV5UDNWULEIPE4QSQHHZQPK3BQKFRB5T3H6GP7ZQ5QGSNK5V6A, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.642285917 (75h46m41.205608925s) [noise=true , address=104.36.82.54:4171 , piece_id=MUMABKARK4KKUKWW6XDXCQVT6O7SLFHGHE7COWC7QIYGARAA2D5A, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.642656750 (75h46m41.205979758s) [noise=true , address=104.36.82.14:6232 , piece_id=A2ZLUUZKRTBJRJ267QSK36C3LFZV7FPAYDDAXGIWDTNEGNV6MRRA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.643565583 (75h46m41.206888597s) [noise=true , address=79.127.205.245:4231 , piece_id=EJK46XUUVI5NWOAQJEW4QIJG3KRXZHB4YOQUJRCS2MIOSWKGTNLQ, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.644620291 (75h46m41.207943303s) [noise=true , address=77.90.31.14:4670 , piece_id=7FNANMUKPM3N6YWRAOAIM4WJZJUTNF7ZBI6ZUBNEPGNGMFWOE24A, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.648842639 (75h46m41.212165631s) [noise=true , address=104.36.82.55:4224 , piece_id=WRVNGSJV5BX4GJERLXNPAS7J7BZ5WFTM7R5H257YRCG5SP2HIYMA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.825828239 (75h46m41.389151241s) [noise=true , address=104.36.82.53:4126 , piece_id=2MKRQJEPGDJKDM7VWY5M4M6NORILAIEYYATJ7HVZEOBOMXOI2LBA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: ecclient: upload failed (node:12dMWFrq2NMpnMzuPAphWerntC5kkrcnzRJZBA116uzJzyvk6fH, address:104.36.82.53:4126): protocol: storage node overloaded, request limit: 2000 Jun 21 20:10:05.834666043 (75h46m41.397989033s) [upload=60693]: Next returned nil segment Jun 21 20:10:05.834686295 (75h46m41.398009302s) [upload=60693]: Waiting for error group managing segments... Jun 21 20:10:05.877010488 (75h46m41.440333486s) [noise=true , address=45.91.116.6:4982 , piece_id=VP45R6KXNH5U5SGUQUMAPIBY7KFDPBIR3S3VRIX27EAT6VLFBZ7Q, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:05.886336964 (75h46m41.449659967s) [noise=true , address=104.36.82.9:6039 , piece_id=I4AB6N27MYYUHWAZLJJUIBFBLOQFHDIM3YLYWPVRPNUV7VMYS2SA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: ecclient: upload failed (node:12P11WVW9mVXw5ra2gZ3Rezon6wZi8GrN5CchQN3MiVU4ts3L6V, address:104.36.82.9:6039): protocol: storage node overloaded, request limit: 2000 Jun 21 20:10:08.064514542 (75h46m43.627837553s) [noise=true , address=77.90.31.14:4670 , piece_id=7FNANMUKPM3N6YWRAOAIM4WJZJUTNF7ZBI6ZUBNEPGNGMFWOE24A, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:08.099098391 (75h46m43.662421396s) [noise=true , address=79.127.205.245:4231 , piece_id=EJK46XUUVI5NWOAQJEW4QIJG3KRXZHB4YOQUJRCS2MIOSWKGTNLQ, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:08.489461749 (75h46m44.052784752s) [noise=true , address=104.36.82.42:6807 , piece_id=ZBQZPXUDJ4VBCOSVKWAJ6SK6UD7SLPON7Y4SSCK2DQ5STFG4XCNQ, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:08.650922093 (75h46m44.214245093s) [noise=true , address=77.90.31.19:4853 , piece_id=VFEON5T2KSWM2KSLTVRZTXYL42JWENZYUEUPG3LMODGZWBMVGGPQ, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:09.917746439 (75h46m45.481069441s) [noise=true , address=104.36.82.13:6196 , piece_id=BCHMQO5KKT5BLNWQGOHH3V43LGSE3BUQCI2SY4TTO5M5GQNOCTMQ, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:10.830722215 (75h46m46.394045216s) [noise=true , address=104.36.82.5:5869 , piece_id=HAEHXXZASV3DHT3QN7NTZOS6XZKHNUWBKCHWJJWSTJKXAKJKKBAA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:12.738783450 (75h46m48.302106452s) [noise=true , address=77.90.31.12:4603 , piece_id=BMUIC6KNGEDL37436GLSXW5XOO6VYOPNOSFGQCYVHSENVI2PJ7KQ, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:15.460927713 (75h46m51.024250723s) [noise=true , address=104.36.82.52:4111 , piece_id=HB7QAWN6CGNIIT4HD2GZ7JDQP5NXIJ5I2USBM66YOKTYENNOWG5A, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:18.393817968 (75h46m53.95714096s) [noise=true , address=104.36.82.55:4224 , piece_id=WRVNGSJV5BX4GJERLXNPAS7J7BZ5WFTM7R5H257YRCG5SP2HIYMA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:19.549174143 (75h46m55.112497151s) [noise=true , address=104.36.82.56:4253 , piece_id=5PMVGLZETMXCQ77K3I6PXACRIMBGR2CV5JGUSU5KUMYFNXVGA2GA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:22.511944593 (75h46m58.075267584s) [noise=true , address=104.36.82.37:6663 , piece_id=X7LWYIIHJFU2T2PHYR3R5TNT22UUPHEHQWZH7ZJMLS4AH4PJ6AIA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:48.027197558 (75h47m23.590520562s) [noise=true , address=104.36.82.14:6232 , piece_id=A2ZLUUZKRTBJRJ267QSK36C3LFZV7FPAYDDAXGIWDTNEGNV6MRRA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:11:05.567692582 (75h47m41.131015574s) [noise=true , address=104.36.82.49:4982 , piece_id=UNLV5UDNWULEIPE4QSQHHZQPK3BQKFRB5T3H6GP7ZQ5QGSNK5V6A, seg_pos={0 0}, upload=60693]: Done uploading piece. err: ecclient: upload failed (node:1NxoYJXCXmTvQncbkXt6yH7mhEBBrzVdzhgL5J6W5TD4N3pkaG, address:104.36.82.49:4982): protocol: expected piece hash; context deadline exceeded; EOF Jun 21 20:11:05.567759198 (75h47m41.131082195s) [noise=true , address=104.36.82.54:4171 , piece_id=MUMABKARK4KKUKWW6XDXCQVT6O7SLFHGHE7COWC7QIYGARAA2D5A, seg_pos={0 0}, upload=60693]: Done uploading piece. err: ecclient: upload failed (node:125SMhDsEZruNu6LkmQn2M7R6WzvXTdk86DRsKf522u8BNiANq3, address:104.36.82.54:4171): protocol: expected piece hash; context deadline exceeded; EOF Jun 21 20:11:05.567869019 (75h47m41.131192033s) [noise=true , address=104.36.82.10:6087 , piece_id=5FQG7BPZC373PFB54HYXSG2SVKOBBBSGZCLC4XDJEYQFOLQY74MA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: ecclient: upload failed (node:1S9k5eejJcxkiG5pXjjBZm18uP8k5jCGfWyeW2QQxmuRU8ZzRA, address:104.36.82.10:6087): protocol: write tcp 159.223.143.39:38550->104.36.82.10:6087: use of closed network connection; write tcp 159.223.143.39:38550->104.36.82.10:6087: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:38550->104.36.82.10:6087: use of closed network connection Jun 21 20:11:05.567932150 (75h47m41.131255161s) [seg_pos={0 0}, upload=60693]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jun 21 20:11:05.593626175 (75h47m41.156949181s) [upload=60693]: Done uploading segments... Jun 21 20:11:05.593657274 (75h47m41.156980265s) [upload=60693]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 22.348333531s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [...........X................................................] duration: 25.495545653s FAIL (2025-06-21 20:47:12 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== pload=60693]: Starting upload Jun 21 20:10:05.602866542 (75h46m41.166189557s) [upload=60693]: Done starting upload Jun 21 20:10:05.603116545 (75h46m41.166439547s) [upload=60693]: Uploading segments... Jun 21 20:10:05.603495389 (75h46m41.166818397s) [upload=60693]: Got next segment. Inline: false Jun 21 20:10:05.603566566 (75h46m41.166889558s) [upload=60693]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 21 20:10:05.603573280 (75h46m41.166896286s) [upload=60693]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 21 20:10:05.639130162 (75h46m41.202453164s) [seg_pos={0 0}, upload=60693]: Begin upload segment... Jun 21 20:10:05.639231070 (75h46m41.202554057s) [seg_pos={0 0}, upload=60693]: Done begin upload segment. Jun 21 20:10:05.639292724 (75h46m41.202615728s) [noise=true , address=104.36.82.56:4253 , piece_id=5PMVGLZETMXCQ77K3I6PXACRIMBGR2CV5JGUSU5KUMYFNXVGA2GA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.639338066 (75h46m41.202661071s) [noise=true , address=104.36.82.53:4126 , piece_id=2MKRQJEPGDJKDM7VWY5M4M6NORILAIEYYATJ7HVZEOBOMXOI2LBA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.639416264 (75h46m41.202739267s) [noise=true , address=104.36.82.5:5869 , piece_id=HAEHXXZASV3DHT3QN7NTZOS6XZKHNUWBKCHWJJWSTJKXAKJKKBAA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.639873098 (75h46m41.203196115s) [noise=true , address=104.36.82.52:4111 , piece_id=HB7QAWN6CGNIIT4HD2GZ7JDQP5NXIJ5I2USBM66YOKTYENNOWG5A, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.639887296 (75h46m41.203210304s) [noise=true , address=104.36.82.10:6087 , piece_id=5FQG7BPZC373PFB54HYXSG2SVKOBBBSGZCLC4XDJEYQFOLQY74MA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.640266623 (75h46m41.203589616s) [noise=true , address=104.36.82.13:6196 , piece_id=BCHMQO5KKT5BLNWQGOHH3V43LGSE3BUQCI2SY4TTO5M5GQNOCTMQ, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.640352274 (75h46m41.203675286s) [noise=true , address=104.36.82.37:6663 , piece_id=X7LWYIIHJFU2T2PHYR3R5TNT22UUPHEHQWZH7ZJMLS4AH4PJ6AIA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.640659767 (75h46m41.203982779s) [noise=true , address=104.36.82.42:6807 , piece_id=ZBQZPXUDJ4VBCOSVKWAJ6SK6UD7SLPON7Y4SSCK2DQ5STFG4XCNQ, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.640961701 (75h46m41.20428471s) [noise=true , address=45.91.116.6:4982 , piece_id=VP45R6KXNH5U5SGUQUMAPIBY7KFDPBIR3S3VRIX27EAT6VLFBZ7Q, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.641159578 (75h46m41.204482589s) [noise=true , address=104.36.82.9:6039 , piece_id=I4AB6N27MYYUHWAZLJJUIBFBLOQFHDIM3YLYWPVRPNUV7VMYS2SA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.641588126 (75h46m41.204911136s) [noise=true , address=77.90.31.19:4853 , piece_id=VFEON5T2KSWM2KSLTVRZTXYL42JWENZYUEUPG3LMODGZWBMVGGPQ, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.641863297 (75h46m41.205186305s) [noise=true , address=77.90.31.12:4603 , piece_id=BMUIC6KNGEDL37436GLSXW5XOO6VYOPNOSFGQCYVHSENVI2PJ7KQ, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.642124427 (75h46m41.205447437s) [noise=true , address=104.36.82.49:4982 , piece_id=UNLV5UDNWULEIPE4QSQHHZQPK3BQKFRB5T3H6GP7ZQ5QGSNK5V6A, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.642285917 (75h46m41.205608925s) [noise=true , address=104.36.82.54:4171 , piece_id=MUMABKARK4KKUKWW6XDXCQVT6O7SLFHGHE7COWC7QIYGARAA2D5A, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.642656750 (75h46m41.205979758s) [noise=true , address=104.36.82.14:6232 , piece_id=A2ZLUUZKRTBJRJ267QSK36C3LFZV7FPAYDDAXGIWDTNEGNV6MRRA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.643565583 (75h46m41.206888597s) [noise=true , address=79.127.205.245:4231 , piece_id=EJK46XUUVI5NWOAQJEW4QIJG3KRXZHB4YOQUJRCS2MIOSWKGTNLQ, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.644620291 (75h46m41.207943303s) [noise=true , address=77.90.31.14:4670 , piece_id=7FNANMUKPM3N6YWRAOAIM4WJZJUTNF7ZBI6ZUBNEPGNGMFWOE24A, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.648842639 (75h46m41.212165631s) [noise=true , address=104.36.82.55:4224 , piece_id=WRVNGSJV5BX4GJERLXNPAS7J7BZ5WFTM7R5H257YRCG5SP2HIYMA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.825828239 (75h46m41.389151241s) [noise=true , address=104.36.82.53:4126 , piece_id=2MKRQJEPGDJKDM7VWY5M4M6NORILAIEYYATJ7HVZEOBOMXOI2LBA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: ecclient: upload failed (node:12dMWFrq2NMpnMzuPAphWerntC5kkrcnzRJZBA116uzJzyvk6fH, address:104.36.82.53:4126): protocol: storage node overloaded, request limit: 2000 Jun 21 20:10:05.834666043 (75h46m41.397989033s) [upload=60693]: Next returned nil segment Jun 21 20:10:05.834686295 (75h46m41.398009302s) [upload=60693]: Waiting for error group managing segments... Jun 21 20:10:05.877010488 (75h46m41.440333486s) [noise=true , address=45.91.116.6:4982 , piece_id=VP45R6KXNH5U5SGUQUMAPIBY7KFDPBIR3S3VRIX27EAT6VLFBZ7Q, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:05.886336964 (75h46m41.449659967s) [noise=true , address=104.36.82.9:6039 , piece_id=I4AB6N27MYYUHWAZLJJUIBFBLOQFHDIM3YLYWPVRPNUV7VMYS2SA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: ecclient: upload failed (node:12P11WVW9mVXw5ra2gZ3Rezon6wZi8GrN5CchQN3MiVU4ts3L6V, address:104.36.82.9:6039): protocol: storage node overloaded, request limit: 2000 Jun 21 20:10:08.064514542 (75h46m43.627837553s) [noise=true , address=77.90.31.14:4670 , piece_id=7FNANMUKPM3N6YWRAOAIM4WJZJUTNF7ZBI6ZUBNEPGNGMFWOE24A, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:08.099098391 (75h46m43.662421396s) [noise=true , address=79.127.205.245:4231 , piece_id=EJK46XUUVI5NWOAQJEW4QIJG3KRXZHB4YOQUJRCS2MIOSWKGTNLQ, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:08.489461749 (75h46m44.052784752s) [noise=true , address=104.36.82.42:6807 , piece_id=ZBQZPXUDJ4VBCOSVKWAJ6SK6UD7SLPON7Y4SSCK2DQ5STFG4XCNQ, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:08.650922093 (75h46m44.214245093s) [noise=true , address=77.90.31.19:4853 , piece_id=VFEON5T2KSWM2KSLTVRZTXYL42JWENZYUEUPG3LMODGZWBMVGGPQ, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:09.917746439 (75h46m45.481069441s) [noise=true , address=104.36.82.13:6196 , piece_id=BCHMQO5KKT5BLNWQGOHH3V43LGSE3BUQCI2SY4TTO5M5GQNOCTMQ, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:10.830722215 (75h46m46.394045216s) [noise=true , address=104.36.82.5:5869 , piece_id=HAEHXXZASV3DHT3QN7NTZOS6XZKHNUWBKCHWJJWSTJKXAKJKKBAA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:12.738783450 (75h46m48.302106452s) [noise=true , address=77.90.31.12:4603 , piece_id=BMUIC6KNGEDL37436GLSXW5XOO6VYOPNOSFGQCYVHSENVI2PJ7KQ, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:15.460927713 (75h46m51.024250723s) [noise=true , address=104.36.82.52:4111 , piece_id=HB7QAWN6CGNIIT4HD2GZ7JDQP5NXIJ5I2USBM66YOKTYENNOWG5A, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:18.393817968 (75h46m53.95714096s) [noise=true , address=104.36.82.55:4224 , piece_id=WRVNGSJV5BX4GJERLXNPAS7J7BZ5WFTM7R5H257YRCG5SP2HIYMA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:19.549174143 (75h46m55.112497151s) [noise=true , address=104.36.82.56:4253 , piece_id=5PMVGLZETMXCQ77K3I6PXACRIMBGR2CV5JGUSU5KUMYFNXVGA2GA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:22.511944593 (75h46m58.075267584s) [noise=true , address=104.36.82.37:6663 , piece_id=X7LWYIIHJFU2T2PHYR3R5TNT22UUPHEHQWZH7ZJMLS4AH4PJ6AIA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:48.027197558 (75h47m23.590520562s) [noise=true , address=104.36.82.14:6232 , piece_id=A2ZLUUZKRTBJRJ267QSK36C3LFZV7FPAYDDAXGIWDTNEGNV6MRRA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:11:05.567692582 (75h47m41.131015574s) [noise=true , address=104.36.82.49:4982 , piece_id=UNLV5UDNWULEIPE4QSQHHZQPK3BQKFRB5T3H6GP7ZQ5QGSNK5V6A, seg_pos={0 0}, upload=60693]: Done uploading piece. err: ecclient: upload failed (node:1NxoYJXCXmTvQncbkXt6yH7mhEBBrzVdzhgL5J6W5TD4N3pkaG, address:104.36.82.49:4982): protocol: expected piece hash; context deadline exceeded; EOF Jun 21 20:11:05.567759198 (75h47m41.131082195s) [noise=true , address=104.36.82.54:4171 , piece_id=MUMABKARK4KKUKWW6XDXCQVT6O7SLFHGHE7COWC7QIYGARAA2D5A, seg_pos={0 0}, upload=60693]: Done uploading piece. err: ecclient: upload failed (node:125SMhDsEZruNu6LkmQn2M7R6WzvXTdk86DRsKf522u8BNiANq3, address:104.36.82.54:4171): protocol: expected piece hash; context deadline exceeded; EOF Jun 21 20:11:05.567869019 (75h47m41.131192033s) [noise=true , address=104.36.82.10:6087 , piece_id=5FQG7BPZC373PFB54HYXSG2SVKOBBBSGZCLC4XDJEYQFOLQY74MA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: ecclient: upload failed (node:1S9k5eejJcxkiG5pXjjBZm18uP8k5jCGfWyeW2QQxmuRU8ZzRA, address:104.36.82.10:6087): protocol: write tcp 159.223.143.39:38550->104.36.82.10:6087: use of closed network connection; write tcp 159.223.143.39:38550->104.36.82.10:6087: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:38550->104.36.82.10:6087: use of closed network connection Jun 21 20:11:05.567932150 (75h47m41.131255161s) [seg_pos={0 0}, upload=60693]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jun 21 20:11:05.593626175 (75h47m41.156949181s) [upload=60693]: Done uploading segments... Jun 21 20:11:05.593657274 (75h47m41.156980265s) [upload=60693]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 25.495578486s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [X...........X...............................................] duration: 26.210787833s FAIL (2025-06-21 20:46:29 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== pload=60693]: Starting upload Jun 21 20:10:05.602866542 (75h46m41.166189557s) [upload=60693]: Done starting upload Jun 21 20:10:05.603116545 (75h46m41.166439547s) [upload=60693]: Uploading segments... Jun 21 20:10:05.603495389 (75h46m41.166818397s) [upload=60693]: Got next segment. Inline: false Jun 21 20:10:05.603566566 (75h46m41.166889558s) [upload=60693]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 21 20:10:05.603573280 (75h46m41.166896286s) [upload=60693]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 21 20:10:05.639130162 (75h46m41.202453164s) [seg_pos={0 0}, upload=60693]: Begin upload segment... Jun 21 20:10:05.639231070 (75h46m41.202554057s) [seg_pos={0 0}, upload=60693]: Done begin upload segment. Jun 21 20:10:05.639292724 (75h46m41.202615728s) [noise=true , address=104.36.82.56:4253 , piece_id=5PMVGLZETMXCQ77K3I6PXACRIMBGR2CV5JGUSU5KUMYFNXVGA2GA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.639338066 (75h46m41.202661071s) [noise=true , address=104.36.82.53:4126 , piece_id=2MKRQJEPGDJKDM7VWY5M4M6NORILAIEYYATJ7HVZEOBOMXOI2LBA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.639416264 (75h46m41.202739267s) [noise=true , address=104.36.82.5:5869 , piece_id=HAEHXXZASV3DHT3QN7NTZOS6XZKHNUWBKCHWJJWSTJKXAKJKKBAA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.639873098 (75h46m41.203196115s) [noise=true , address=104.36.82.52:4111 , piece_id=HB7QAWN6CGNIIT4HD2GZ7JDQP5NXIJ5I2USBM66YOKTYENNOWG5A, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.639887296 (75h46m41.203210304s) [noise=true , address=104.36.82.10:6087 , piece_id=5FQG7BPZC373PFB54HYXSG2SVKOBBBSGZCLC4XDJEYQFOLQY74MA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.640266623 (75h46m41.203589616s) [noise=true , address=104.36.82.13:6196 , piece_id=BCHMQO5KKT5BLNWQGOHH3V43LGSE3BUQCI2SY4TTO5M5GQNOCTMQ, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.640352274 (75h46m41.203675286s) [noise=true , address=104.36.82.37:6663 , piece_id=X7LWYIIHJFU2T2PHYR3R5TNT22UUPHEHQWZH7ZJMLS4AH4PJ6AIA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.640659767 (75h46m41.203982779s) [noise=true , address=104.36.82.42:6807 , piece_id=ZBQZPXUDJ4VBCOSVKWAJ6SK6UD7SLPON7Y4SSCK2DQ5STFG4XCNQ, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.640961701 (75h46m41.20428471s) [noise=true , address=45.91.116.6:4982 , piece_id=VP45R6KXNH5U5SGUQUMAPIBY7KFDPBIR3S3VRIX27EAT6VLFBZ7Q, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.641159578 (75h46m41.204482589s) [noise=true , address=104.36.82.9:6039 , piece_id=I4AB6N27MYYUHWAZLJJUIBFBLOQFHDIM3YLYWPVRPNUV7VMYS2SA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.641588126 (75h46m41.204911136s) [noise=true , address=77.90.31.19:4853 , piece_id=VFEON5T2KSWM2KSLTVRZTXYL42JWENZYUEUPG3LMODGZWBMVGGPQ, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.641863297 (75h46m41.205186305s) [noise=true , address=77.90.31.12:4603 , piece_id=BMUIC6KNGEDL37436GLSXW5XOO6VYOPNOSFGQCYVHSENVI2PJ7KQ, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.642124427 (75h46m41.205447437s) [noise=true , address=104.36.82.49:4982 , piece_id=UNLV5UDNWULEIPE4QSQHHZQPK3BQKFRB5T3H6GP7ZQ5QGSNK5V6A, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.642285917 (75h46m41.205608925s) [noise=true , address=104.36.82.54:4171 , piece_id=MUMABKARK4KKUKWW6XDXCQVT6O7SLFHGHE7COWC7QIYGARAA2D5A, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.642656750 (75h46m41.205979758s) [noise=true , address=104.36.82.14:6232 , piece_id=A2ZLUUZKRTBJRJ267QSK36C3LFZV7FPAYDDAXGIWDTNEGNV6MRRA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.643565583 (75h46m41.206888597s) [noise=true , address=79.127.205.245:4231 , piece_id=EJK46XUUVI5NWOAQJEW4QIJG3KRXZHB4YOQUJRCS2MIOSWKGTNLQ, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.644620291 (75h46m41.207943303s) [noise=true , address=77.90.31.14:4670 , piece_id=7FNANMUKPM3N6YWRAOAIM4WJZJUTNF7ZBI6ZUBNEPGNGMFWOE24A, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.648842639 (75h46m41.212165631s) [noise=true , address=104.36.82.55:4224 , piece_id=WRVNGSJV5BX4GJERLXNPAS7J7BZ5WFTM7R5H257YRCG5SP2HIYMA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.825828239 (75h46m41.389151241s) [noise=true , address=104.36.82.53:4126 , piece_id=2MKRQJEPGDJKDM7VWY5M4M6NORILAIEYYATJ7HVZEOBOMXOI2LBA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: ecclient: upload failed (node:12dMWFrq2NMpnMzuPAphWerntC5kkrcnzRJZBA116uzJzyvk6fH, address:104.36.82.53:4126): protocol: storage node overloaded, request limit: 2000 Jun 21 20:10:05.834666043 (75h46m41.397989033s) [upload=60693]: Next returned nil segment Jun 21 20:10:05.834686295 (75h46m41.398009302s) [upload=60693]: Waiting for error group managing segments... Jun 21 20:10:05.877010488 (75h46m41.440333486s) [noise=true , address=45.91.116.6:4982 , piece_id=VP45R6KXNH5U5SGUQUMAPIBY7KFDPBIR3S3VRIX27EAT6VLFBZ7Q, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:05.886336964 (75h46m41.449659967s) [noise=true , address=104.36.82.9:6039 , piece_id=I4AB6N27MYYUHWAZLJJUIBFBLOQFHDIM3YLYWPVRPNUV7VMYS2SA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: ecclient: upload failed (node:12P11WVW9mVXw5ra2gZ3Rezon6wZi8GrN5CchQN3MiVU4ts3L6V, address:104.36.82.9:6039): protocol: storage node overloaded, request limit: 2000 Jun 21 20:10:08.064514542 (75h46m43.627837553s) [noise=true , address=77.90.31.14:4670 , piece_id=7FNANMUKPM3N6YWRAOAIM4WJZJUTNF7ZBI6ZUBNEPGNGMFWOE24A, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:08.099098391 (75h46m43.662421396s) [noise=true , address=79.127.205.245:4231 , piece_id=EJK46XUUVI5NWOAQJEW4QIJG3KRXZHB4YOQUJRCS2MIOSWKGTNLQ, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:08.489461749 (75h46m44.052784752s) [noise=true , address=104.36.82.42:6807 , piece_id=ZBQZPXUDJ4VBCOSVKWAJ6SK6UD7SLPON7Y4SSCK2DQ5STFG4XCNQ, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:08.650922093 (75h46m44.214245093s) [noise=true , address=77.90.31.19:4853 , piece_id=VFEON5T2KSWM2KSLTVRZTXYL42JWENZYUEUPG3LMODGZWBMVGGPQ, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:09.917746439 (75h46m45.481069441s) [noise=true , address=104.36.82.13:6196 , piece_id=BCHMQO5KKT5BLNWQGOHH3V43LGSE3BUQCI2SY4TTO5M5GQNOCTMQ, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:10.830722215 (75h46m46.394045216s) [noise=true , address=104.36.82.5:5869 , piece_id=HAEHXXZASV3DHT3QN7NTZOS6XZKHNUWBKCHWJJWSTJKXAKJKKBAA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:12.738783450 (75h46m48.302106452s) [noise=true , address=77.90.31.12:4603 , piece_id=BMUIC6KNGEDL37436GLSXW5XOO6VYOPNOSFGQCYVHSENVI2PJ7KQ, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:15.460927713 (75h46m51.024250723s) [noise=true , address=104.36.82.52:4111 , piece_id=HB7QAWN6CGNIIT4HD2GZ7JDQP5NXIJ5I2USBM66YOKTYENNOWG5A, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:18.393817968 (75h46m53.95714096s) [noise=true , address=104.36.82.55:4224 , piece_id=WRVNGSJV5BX4GJERLXNPAS7J7BZ5WFTM7R5H257YRCG5SP2HIYMA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:19.549174143 (75h46m55.112497151s) [noise=true , address=104.36.82.56:4253 , piece_id=5PMVGLZETMXCQ77K3I6PXACRIMBGR2CV5JGUSU5KUMYFNXVGA2GA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:22.511944593 (75h46m58.075267584s) [noise=true , address=104.36.82.37:6663 , piece_id=X7LWYIIHJFU2T2PHYR3R5TNT22UUPHEHQWZH7ZJMLS4AH4PJ6AIA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:48.027197558 (75h47m23.590520562s) [noise=true , address=104.36.82.14:6232 , piece_id=A2ZLUUZKRTBJRJ267QSK36C3LFZV7FPAYDDAXGIWDTNEGNV6MRRA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:11:05.567692582 (75h47m41.131015574s) [noise=true , address=104.36.82.49:4982 , piece_id=UNLV5UDNWULEIPE4QSQHHZQPK3BQKFRB5T3H6GP7ZQ5QGSNK5V6A, seg_pos={0 0}, upload=60693]: Done uploading piece. err: ecclient: upload failed (node:1NxoYJXCXmTvQncbkXt6yH7mhEBBrzVdzhgL5J6W5TD4N3pkaG, address:104.36.82.49:4982): protocol: expected piece hash; context deadline exceeded; EOF Jun 21 20:11:05.567759198 (75h47m41.131082195s) [noise=true , address=104.36.82.54:4171 , piece_id=MUMABKARK4KKUKWW6XDXCQVT6O7SLFHGHE7COWC7QIYGARAA2D5A, seg_pos={0 0}, upload=60693]: Done uploading piece. err: ecclient: upload failed (node:125SMhDsEZruNu6LkmQn2M7R6WzvXTdk86DRsKf522u8BNiANq3, address:104.36.82.54:4171): protocol: expected piece hash; context deadline exceeded; EOF Jun 21 20:11:05.567869019 (75h47m41.131192033s) [noise=true , address=104.36.82.10:6087 , piece_id=5FQG7BPZC373PFB54HYXSG2SVKOBBBSGZCLC4XDJEYQFOLQY74MA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: ecclient: upload failed (node:1S9k5eejJcxkiG5pXjjBZm18uP8k5jCGfWyeW2QQxmuRU8ZzRA, address:104.36.82.10:6087): protocol: write tcp 159.223.143.39:38550->104.36.82.10:6087: use of closed network connection; write tcp 159.223.143.39:38550->104.36.82.10:6087: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:38550->104.36.82.10:6087: use of closed network connection Jun 21 20:11:05.567932150 (75h47m41.131255161s) [seg_pos={0 0}, upload=60693]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jun 21 20:11:05.593626175 (75h47m41.156949181s) [upload=60693]: Done uploading segments... Jun 21 20:11:05.593657274 (75h47m41.156980265s) [upload=60693]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 26.210810266s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [.X...........X..............................................] duration: 24.601036757s FAIL (2025-06-21 20:46:03 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== pload=60693]: Starting upload Jun 21 20:10:05.602866542 (75h46m41.166189557s) [upload=60693]: Done starting upload Jun 21 20:10:05.603116545 (75h46m41.166439547s) [upload=60693]: Uploading segments... Jun 21 20:10:05.603495389 (75h46m41.166818397s) [upload=60693]: Got next segment. Inline: false Jun 21 20:10:05.603566566 (75h46m41.166889558s) [upload=60693]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 21 20:10:05.603573280 (75h46m41.166896286s) [upload=60693]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 21 20:10:05.639130162 (75h46m41.202453164s) [seg_pos={0 0}, upload=60693]: Begin upload segment... Jun 21 20:10:05.639231070 (75h46m41.202554057s) [seg_pos={0 0}, upload=60693]: Done begin upload segment. Jun 21 20:10:05.639292724 (75h46m41.202615728s) [noise=true , address=104.36.82.56:4253 , piece_id=5PMVGLZETMXCQ77K3I6PXACRIMBGR2CV5JGUSU5KUMYFNXVGA2GA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.639338066 (75h46m41.202661071s) [noise=true , address=104.36.82.53:4126 , piece_id=2MKRQJEPGDJKDM7VWY5M4M6NORILAIEYYATJ7HVZEOBOMXOI2LBA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.639416264 (75h46m41.202739267s) [noise=true , address=104.36.82.5:5869 , piece_id=HAEHXXZASV3DHT3QN7NTZOS6XZKHNUWBKCHWJJWSTJKXAKJKKBAA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.639873098 (75h46m41.203196115s) [noise=true , address=104.36.82.52:4111 , piece_id=HB7QAWN6CGNIIT4HD2GZ7JDQP5NXIJ5I2USBM66YOKTYENNOWG5A, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.639887296 (75h46m41.203210304s) [noise=true , address=104.36.82.10:6087 , piece_id=5FQG7BPZC373PFB54HYXSG2SVKOBBBSGZCLC4XDJEYQFOLQY74MA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.640266623 (75h46m41.203589616s) [noise=true , address=104.36.82.13:6196 , piece_id=BCHMQO5KKT5BLNWQGOHH3V43LGSE3BUQCI2SY4TTO5M5GQNOCTMQ, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.640352274 (75h46m41.203675286s) [noise=true , address=104.36.82.37:6663 , piece_id=X7LWYIIHJFU2T2PHYR3R5TNT22UUPHEHQWZH7ZJMLS4AH4PJ6AIA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.640659767 (75h46m41.203982779s) [noise=true , address=104.36.82.42:6807 , piece_id=ZBQZPXUDJ4VBCOSVKWAJ6SK6UD7SLPON7Y4SSCK2DQ5STFG4XCNQ, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.640961701 (75h46m41.20428471s) [noise=true , address=45.91.116.6:4982 , piece_id=VP45R6KXNH5U5SGUQUMAPIBY7KFDPBIR3S3VRIX27EAT6VLFBZ7Q, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.641159578 (75h46m41.204482589s) [noise=true , address=104.36.82.9:6039 , piece_id=I4AB6N27MYYUHWAZLJJUIBFBLOQFHDIM3YLYWPVRPNUV7VMYS2SA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.641588126 (75h46m41.204911136s) [noise=true , address=77.90.31.19:4853 , piece_id=VFEON5T2KSWM2KSLTVRZTXYL42JWENZYUEUPG3LMODGZWBMVGGPQ, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.641863297 (75h46m41.205186305s) [noise=true , address=77.90.31.12:4603 , piece_id=BMUIC6KNGEDL37436GLSXW5XOO6VYOPNOSFGQCYVHSENVI2PJ7KQ, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.642124427 (75h46m41.205447437s) [noise=true , address=104.36.82.49:4982 , piece_id=UNLV5UDNWULEIPE4QSQHHZQPK3BQKFRB5T3H6GP7ZQ5QGSNK5V6A, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.642285917 (75h46m41.205608925s) [noise=true , address=104.36.82.54:4171 , piece_id=MUMABKARK4KKUKWW6XDXCQVT6O7SLFHGHE7COWC7QIYGARAA2D5A, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.642656750 (75h46m41.205979758s) [noise=true , address=104.36.82.14:6232 , piece_id=A2ZLUUZKRTBJRJ267QSK36C3LFZV7FPAYDDAXGIWDTNEGNV6MRRA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.643565583 (75h46m41.206888597s) [noise=true , address=79.127.205.245:4231 , piece_id=EJK46XUUVI5NWOAQJEW4QIJG3KRXZHB4YOQUJRCS2MIOSWKGTNLQ, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.644620291 (75h46m41.207943303s) [noise=true , address=77.90.31.14:4670 , piece_id=7FNANMUKPM3N6YWRAOAIM4WJZJUTNF7ZBI6ZUBNEPGNGMFWOE24A, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.648842639 (75h46m41.212165631s) [noise=true , address=104.36.82.55:4224 , piece_id=WRVNGSJV5BX4GJERLXNPAS7J7BZ5WFTM7R5H257YRCG5SP2HIYMA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.825828239 (75h46m41.389151241s) [noise=true , address=104.36.82.53:4126 , piece_id=2MKRQJEPGDJKDM7VWY5M4M6NORILAIEYYATJ7HVZEOBOMXOI2LBA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: ecclient: upload failed (node:12dMWFrq2NMpnMzuPAphWerntC5kkrcnzRJZBA116uzJzyvk6fH, address:104.36.82.53:4126): protocol: storage node overloaded, request limit: 2000 Jun 21 20:10:05.834666043 (75h46m41.397989033s) [upload=60693]: Next returned nil segment Jun 21 20:10:05.834686295 (75h46m41.398009302s) [upload=60693]: Waiting for error group managing segments... Jun 21 20:10:05.877010488 (75h46m41.440333486s) [noise=true , address=45.91.116.6:4982 , piece_id=VP45R6KXNH5U5SGUQUMAPIBY7KFDPBIR3S3VRIX27EAT6VLFBZ7Q, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:05.886336964 (75h46m41.449659967s) [noise=true , address=104.36.82.9:6039 , piece_id=I4AB6N27MYYUHWAZLJJUIBFBLOQFHDIM3YLYWPVRPNUV7VMYS2SA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: ecclient: upload failed (node:12P11WVW9mVXw5ra2gZ3Rezon6wZi8GrN5CchQN3MiVU4ts3L6V, address:104.36.82.9:6039): protocol: storage node overloaded, request limit: 2000 Jun 21 20:10:08.064514542 (75h46m43.627837553s) [noise=true , address=77.90.31.14:4670 , piece_id=7FNANMUKPM3N6YWRAOAIM4WJZJUTNF7ZBI6ZUBNEPGNGMFWOE24A, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:08.099098391 (75h46m43.662421396s) [noise=true , address=79.127.205.245:4231 , piece_id=EJK46XUUVI5NWOAQJEW4QIJG3KRXZHB4YOQUJRCS2MIOSWKGTNLQ, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:08.489461749 (75h46m44.052784752s) [noise=true , address=104.36.82.42:6807 , piece_id=ZBQZPXUDJ4VBCOSVKWAJ6SK6UD7SLPON7Y4SSCK2DQ5STFG4XCNQ, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:08.650922093 (75h46m44.214245093s) [noise=true , address=77.90.31.19:4853 , piece_id=VFEON5T2KSWM2KSLTVRZTXYL42JWENZYUEUPG3LMODGZWBMVGGPQ, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:09.917746439 (75h46m45.481069441s) [noise=true , address=104.36.82.13:6196 , piece_id=BCHMQO5KKT5BLNWQGOHH3V43LGSE3BUQCI2SY4TTO5M5GQNOCTMQ, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:10.830722215 (75h46m46.394045216s) [noise=true , address=104.36.82.5:5869 , piece_id=HAEHXXZASV3DHT3QN7NTZOS6XZKHNUWBKCHWJJWSTJKXAKJKKBAA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:12.738783450 (75h46m48.302106452s) [noise=true , address=77.90.31.12:4603 , piece_id=BMUIC6KNGEDL37436GLSXW5XOO6VYOPNOSFGQCYVHSENVI2PJ7KQ, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:15.460927713 (75h46m51.024250723s) [noise=true , address=104.36.82.52:4111 , piece_id=HB7QAWN6CGNIIT4HD2GZ7JDQP5NXIJ5I2USBM66YOKTYENNOWG5A, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:18.393817968 (75h46m53.95714096s) [noise=true , address=104.36.82.55:4224 , piece_id=WRVNGSJV5BX4GJERLXNPAS7J7BZ5WFTM7R5H257YRCG5SP2HIYMA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:19.549174143 (75h46m55.112497151s) [noise=true , address=104.36.82.56:4253 , piece_id=5PMVGLZETMXCQ77K3I6PXACRIMBGR2CV5JGUSU5KUMYFNXVGA2GA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:22.511944593 (75h46m58.075267584s) [noise=true , address=104.36.82.37:6663 , piece_id=X7LWYIIHJFU2T2PHYR3R5TNT22UUPHEHQWZH7ZJMLS4AH4PJ6AIA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:48.027197558 (75h47m23.590520562s) [noise=true , address=104.36.82.14:6232 , piece_id=A2ZLUUZKRTBJRJ267QSK36C3LFZV7FPAYDDAXGIWDTNEGNV6MRRA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:11:05.567692582 (75h47m41.131015574s) [noise=true , address=104.36.82.49:4982 , piece_id=UNLV5UDNWULEIPE4QSQHHZQPK3BQKFRB5T3H6GP7ZQ5QGSNK5V6A, seg_pos={0 0}, upload=60693]: Done uploading piece. err: ecclient: upload failed (node:1NxoYJXCXmTvQncbkXt6yH7mhEBBrzVdzhgL5J6W5TD4N3pkaG, address:104.36.82.49:4982): protocol: expected piece hash; context deadline exceeded; EOF Jun 21 20:11:05.567759198 (75h47m41.131082195s) [noise=true , address=104.36.82.54:4171 , piece_id=MUMABKARK4KKUKWW6XDXCQVT6O7SLFHGHE7COWC7QIYGARAA2D5A, seg_pos={0 0}, upload=60693]: Done uploading piece. err: ecclient: upload failed (node:125SMhDsEZruNu6LkmQn2M7R6WzvXTdk86DRsKf522u8BNiANq3, address:104.36.82.54:4171): protocol: expected piece hash; context deadline exceeded; EOF Jun 21 20:11:05.567869019 (75h47m41.131192033s) [noise=true , address=104.36.82.10:6087 , piece_id=5FQG7BPZC373PFB54HYXSG2SVKOBBBSGZCLC4XDJEYQFOLQY74MA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: ecclient: upload failed (node:1S9k5eejJcxkiG5pXjjBZm18uP8k5jCGfWyeW2QQxmuRU8ZzRA, address:104.36.82.10:6087): protocol: write tcp 159.223.143.39:38550->104.36.82.10:6087: use of closed network connection; write tcp 159.223.143.39:38550->104.36.82.10:6087: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:38550->104.36.82.10:6087: use of closed network connection Jun 21 20:11:05.567932150 (75h47m41.131255161s) [seg_pos={0 0}, upload=60693]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jun 21 20:11:05.593626175 (75h47m41.156949181s) [upload=60693]: Done uploading segments... Jun 21 20:11:05.593657274 (75h47m41.156980265s) [upload=60693]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 24.601067242s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [..X...........X.............................................] duration: 14.470449209s FAIL (2025-06-21 20:45:28 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== pload=60693]: Starting upload Jun 21 20:10:05.602866542 (75h46m41.166189557s) [upload=60693]: Done starting upload Jun 21 20:10:05.603116545 (75h46m41.166439547s) [upload=60693]: Uploading segments... Jun 21 20:10:05.603495389 (75h46m41.166818397s) [upload=60693]: Got next segment. Inline: false Jun 21 20:10:05.603566566 (75h46m41.166889558s) [upload=60693]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 21 20:10:05.603573280 (75h46m41.166896286s) [upload=60693]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 21 20:10:05.639130162 (75h46m41.202453164s) [seg_pos={0 0}, upload=60693]: Begin upload segment... Jun 21 20:10:05.639231070 (75h46m41.202554057s) [seg_pos={0 0}, upload=60693]: Done begin upload segment. Jun 21 20:10:05.639292724 (75h46m41.202615728s) [noise=true , address=104.36.82.56:4253 , piece_id=5PMVGLZETMXCQ77K3I6PXACRIMBGR2CV5JGUSU5KUMYFNXVGA2GA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.639338066 (75h46m41.202661071s) [noise=true , address=104.36.82.53:4126 , piece_id=2MKRQJEPGDJKDM7VWY5M4M6NORILAIEYYATJ7HVZEOBOMXOI2LBA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.639416264 (75h46m41.202739267s) [noise=true , address=104.36.82.5:5869 , piece_id=HAEHXXZASV3DHT3QN7NTZOS6XZKHNUWBKCHWJJWSTJKXAKJKKBAA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.639873098 (75h46m41.203196115s) [noise=true , address=104.36.82.52:4111 , piece_id=HB7QAWN6CGNIIT4HD2GZ7JDQP5NXIJ5I2USBM66YOKTYENNOWG5A, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.639887296 (75h46m41.203210304s) [noise=true , address=104.36.82.10:6087 , piece_id=5FQG7BPZC373PFB54HYXSG2SVKOBBBSGZCLC4XDJEYQFOLQY74MA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.640266623 (75h46m41.203589616s) [noise=true , address=104.36.82.13:6196 , piece_id=BCHMQO5KKT5BLNWQGOHH3V43LGSE3BUQCI2SY4TTO5M5GQNOCTMQ, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.640352274 (75h46m41.203675286s) [noise=true , address=104.36.82.37:6663 , piece_id=X7LWYIIHJFU2T2PHYR3R5TNT22UUPHEHQWZH7ZJMLS4AH4PJ6AIA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.640659767 (75h46m41.203982779s) [noise=true , address=104.36.82.42:6807 , piece_id=ZBQZPXUDJ4VBCOSVKWAJ6SK6UD7SLPON7Y4SSCK2DQ5STFG4XCNQ, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.640961701 (75h46m41.20428471s) [noise=true , address=45.91.116.6:4982 , piece_id=VP45R6KXNH5U5SGUQUMAPIBY7KFDPBIR3S3VRIX27EAT6VLFBZ7Q, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.641159578 (75h46m41.204482589s) [noise=true , address=104.36.82.9:6039 , piece_id=I4AB6N27MYYUHWAZLJJUIBFBLOQFHDIM3YLYWPVRPNUV7VMYS2SA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.641588126 (75h46m41.204911136s) [noise=true , address=77.90.31.19:4853 , piece_id=VFEON5T2KSWM2KSLTVRZTXYL42JWENZYUEUPG3LMODGZWBMVGGPQ, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.641863297 (75h46m41.205186305s) [noise=true , address=77.90.31.12:4603 , piece_id=BMUIC6KNGEDL37436GLSXW5XOO6VYOPNOSFGQCYVHSENVI2PJ7KQ, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.642124427 (75h46m41.205447437s) [noise=true , address=104.36.82.49:4982 , piece_id=UNLV5UDNWULEIPE4QSQHHZQPK3BQKFRB5T3H6GP7ZQ5QGSNK5V6A, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.642285917 (75h46m41.205608925s) [noise=true , address=104.36.82.54:4171 , piece_id=MUMABKARK4KKUKWW6XDXCQVT6O7SLFHGHE7COWC7QIYGARAA2D5A, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.642656750 (75h46m41.205979758s) [noise=true , address=104.36.82.14:6232 , piece_id=A2ZLUUZKRTBJRJ267QSK36C3LFZV7FPAYDDAXGIWDTNEGNV6MRRA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.643565583 (75h46m41.206888597s) [noise=true , address=79.127.205.245:4231 , piece_id=EJK46XUUVI5NWOAQJEW4QIJG3KRXZHB4YOQUJRCS2MIOSWKGTNLQ, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.644620291 (75h46m41.207943303s) [noise=true , address=77.90.31.14:4670 , piece_id=7FNANMUKPM3N6YWRAOAIM4WJZJUTNF7ZBI6ZUBNEPGNGMFWOE24A, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.648842639 (75h46m41.212165631s) [noise=true , address=104.36.82.55:4224 , piece_id=WRVNGSJV5BX4GJERLXNPAS7J7BZ5WFTM7R5H257YRCG5SP2HIYMA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.825828239 (75h46m41.389151241s) [noise=true , address=104.36.82.53:4126 , piece_id=2MKRQJEPGDJKDM7VWY5M4M6NORILAIEYYATJ7HVZEOBOMXOI2LBA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: ecclient: upload failed (node:12dMWFrq2NMpnMzuPAphWerntC5kkrcnzRJZBA116uzJzyvk6fH, address:104.36.82.53:4126): protocol: storage node overloaded, request limit: 2000 Jun 21 20:10:05.834666043 (75h46m41.397989033s) [upload=60693]: Next returned nil segment Jun 21 20:10:05.834686295 (75h46m41.398009302s) [upload=60693]: Waiting for error group managing segments... Jun 21 20:10:05.877010488 (75h46m41.440333486s) [noise=true , address=45.91.116.6:4982 , piece_id=VP45R6KXNH5U5SGUQUMAPIBY7KFDPBIR3S3VRIX27EAT6VLFBZ7Q, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:05.886336964 (75h46m41.449659967s) [noise=true , address=104.36.82.9:6039 , piece_id=I4AB6N27MYYUHWAZLJJUIBFBLOQFHDIM3YLYWPVRPNUV7VMYS2SA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: ecclient: upload failed (node:12P11WVW9mVXw5ra2gZ3Rezon6wZi8GrN5CchQN3MiVU4ts3L6V, address:104.36.82.9:6039): protocol: storage node overloaded, request limit: 2000 Jun 21 20:10:08.064514542 (75h46m43.627837553s) [noise=true , address=77.90.31.14:4670 , piece_id=7FNANMUKPM3N6YWRAOAIM4WJZJUTNF7ZBI6ZUBNEPGNGMFWOE24A, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:08.099098391 (75h46m43.662421396s) [noise=true , address=79.127.205.245:4231 , piece_id=EJK46XUUVI5NWOAQJEW4QIJG3KRXZHB4YOQUJRCS2MIOSWKGTNLQ, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:08.489461749 (75h46m44.052784752s) [noise=true , address=104.36.82.42:6807 , piece_id=ZBQZPXUDJ4VBCOSVKWAJ6SK6UD7SLPON7Y4SSCK2DQ5STFG4XCNQ, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:08.650922093 (75h46m44.214245093s) [noise=true , address=77.90.31.19:4853 , piece_id=VFEON5T2KSWM2KSLTVRZTXYL42JWENZYUEUPG3LMODGZWBMVGGPQ, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:09.917746439 (75h46m45.481069441s) [noise=true , address=104.36.82.13:6196 , piece_id=BCHMQO5KKT5BLNWQGOHH3V43LGSE3BUQCI2SY4TTO5M5GQNOCTMQ, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:10.830722215 (75h46m46.394045216s) [noise=true , address=104.36.82.5:5869 , piece_id=HAEHXXZASV3DHT3QN7NTZOS6XZKHNUWBKCHWJJWSTJKXAKJKKBAA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:12.738783450 (75h46m48.302106452s) [noise=true , address=77.90.31.12:4603 , piece_id=BMUIC6KNGEDL37436GLSXW5XOO6VYOPNOSFGQCYVHSENVI2PJ7KQ, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:15.460927713 (75h46m51.024250723s) [noise=true , address=104.36.82.52:4111 , piece_id=HB7QAWN6CGNIIT4HD2GZ7JDQP5NXIJ5I2USBM66YOKTYENNOWG5A, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:18.393817968 (75h46m53.95714096s) [noise=true , address=104.36.82.55:4224 , piece_id=WRVNGSJV5BX4GJERLXNPAS7J7BZ5WFTM7R5H257YRCG5SP2HIYMA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:19.549174143 (75h46m55.112497151s) [noise=true , address=104.36.82.56:4253 , piece_id=5PMVGLZETMXCQ77K3I6PXACRIMBGR2CV5JGUSU5KUMYFNXVGA2GA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:22.511944593 (75h46m58.075267584s) [noise=true , address=104.36.82.37:6663 , piece_id=X7LWYIIHJFU2T2PHYR3R5TNT22UUPHEHQWZH7ZJMLS4AH4PJ6AIA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:48.027197558 (75h47m23.590520562s) [noise=true , address=104.36.82.14:6232 , piece_id=A2ZLUUZKRTBJRJ267QSK36C3LFZV7FPAYDDAXGIWDTNEGNV6MRRA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:11:05.567692582 (75h47m41.131015574s) [noise=true , address=104.36.82.49:4982 , piece_id=UNLV5UDNWULEIPE4QSQHHZQPK3BQKFRB5T3H6GP7ZQ5QGSNK5V6A, seg_pos={0 0}, upload=60693]: Done uploading piece. err: ecclient: upload failed (node:1NxoYJXCXmTvQncbkXt6yH7mhEBBrzVdzhgL5J6W5TD4N3pkaG, address:104.36.82.49:4982): protocol: expected piece hash; context deadline exceeded; EOF Jun 21 20:11:05.567759198 (75h47m41.131082195s) [noise=true , address=104.36.82.54:4171 , piece_id=MUMABKARK4KKUKWW6XDXCQVT6O7SLFHGHE7COWC7QIYGARAA2D5A, seg_pos={0 0}, upload=60693]: Done uploading piece. err: ecclient: upload failed (node:125SMhDsEZruNu6LkmQn2M7R6WzvXTdk86DRsKf522u8BNiANq3, address:104.36.82.54:4171): protocol: expected piece hash; context deadline exceeded; EOF Jun 21 20:11:05.567869019 (75h47m41.131192033s) [noise=true , address=104.36.82.10:6087 , piece_id=5FQG7BPZC373PFB54HYXSG2SVKOBBBSGZCLC4XDJEYQFOLQY74MA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: ecclient: upload failed (node:1S9k5eejJcxkiG5pXjjBZm18uP8k5jCGfWyeW2QQxmuRU8ZzRA, address:104.36.82.10:6087): protocol: write tcp 159.223.143.39:38550->104.36.82.10:6087: use of closed network connection; write tcp 159.223.143.39:38550->104.36.82.10:6087: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:38550->104.36.82.10:6087: use of closed network connection Jun 21 20:11:05.567932150 (75h47m41.131255161s) [seg_pos={0 0}, upload=60693]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jun 21 20:11:05.593626175 (75h47m41.156949181s) [upload=60693]: Done uploading segments... Jun 21 20:11:05.593657274 (75h47m41.156980265s) [upload=60693]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 14.470557103s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [...X...........X............................................] duration: 19.503581206s FAIL (2025-06-21 20:44:53 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== pload=60693]: Starting upload Jun 21 20:10:05.602866542 (75h46m41.166189557s) [upload=60693]: Done starting upload Jun 21 20:10:05.603116545 (75h46m41.166439547s) [upload=60693]: Uploading segments... Jun 21 20:10:05.603495389 (75h46m41.166818397s) [upload=60693]: Got next segment. Inline: false Jun 21 20:10:05.603566566 (75h46m41.166889558s) [upload=60693]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 21 20:10:05.603573280 (75h46m41.166896286s) [upload=60693]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 21 20:10:05.639130162 (75h46m41.202453164s) [seg_pos={0 0}, upload=60693]: Begin upload segment... Jun 21 20:10:05.639231070 (75h46m41.202554057s) [seg_pos={0 0}, upload=60693]: Done begin upload segment. Jun 21 20:10:05.639292724 (75h46m41.202615728s) [noise=true , address=104.36.82.56:4253 , piece_id=5PMVGLZETMXCQ77K3I6PXACRIMBGR2CV5JGUSU5KUMYFNXVGA2GA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.639338066 (75h46m41.202661071s) [noise=true , address=104.36.82.53:4126 , piece_id=2MKRQJEPGDJKDM7VWY5M4M6NORILAIEYYATJ7HVZEOBOMXOI2LBA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.639416264 (75h46m41.202739267s) [noise=true , address=104.36.82.5:5869 , piece_id=HAEHXXZASV3DHT3QN7NTZOS6XZKHNUWBKCHWJJWSTJKXAKJKKBAA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.639873098 (75h46m41.203196115s) [noise=true , address=104.36.82.52:4111 , piece_id=HB7QAWN6CGNIIT4HD2GZ7JDQP5NXIJ5I2USBM66YOKTYENNOWG5A, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.639887296 (75h46m41.203210304s) [noise=true , address=104.36.82.10:6087 , piece_id=5FQG7BPZC373PFB54HYXSG2SVKOBBBSGZCLC4XDJEYQFOLQY74MA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.640266623 (75h46m41.203589616s) [noise=true , address=104.36.82.13:6196 , piece_id=BCHMQO5KKT5BLNWQGOHH3V43LGSE3BUQCI2SY4TTO5M5GQNOCTMQ, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.640352274 (75h46m41.203675286s) [noise=true , address=104.36.82.37:6663 , piece_id=X7LWYIIHJFU2T2PHYR3R5TNT22UUPHEHQWZH7ZJMLS4AH4PJ6AIA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.640659767 (75h46m41.203982779s) [noise=true , address=104.36.82.42:6807 , piece_id=ZBQZPXUDJ4VBCOSVKWAJ6SK6UD7SLPON7Y4SSCK2DQ5STFG4XCNQ, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.640961701 (75h46m41.20428471s) [noise=true , address=45.91.116.6:4982 , piece_id=VP45R6KXNH5U5SGUQUMAPIBY7KFDPBIR3S3VRIX27EAT6VLFBZ7Q, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.641159578 (75h46m41.204482589s) [noise=true , address=104.36.82.9:6039 , piece_id=I4AB6N27MYYUHWAZLJJUIBFBLOQFHDIM3YLYWPVRPNUV7VMYS2SA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.641588126 (75h46m41.204911136s) [noise=true , address=77.90.31.19:4853 , piece_id=VFEON5T2KSWM2KSLTVRZTXYL42JWENZYUEUPG3LMODGZWBMVGGPQ, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.641863297 (75h46m41.205186305s) [noise=true , address=77.90.31.12:4603 , piece_id=BMUIC6KNGEDL37436GLSXW5XOO6VYOPNOSFGQCYVHSENVI2PJ7KQ, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.642124427 (75h46m41.205447437s) [noise=true , address=104.36.82.49:4982 , piece_id=UNLV5UDNWULEIPE4QSQHHZQPK3BQKFRB5T3H6GP7ZQ5QGSNK5V6A, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.642285917 (75h46m41.205608925s) [noise=true , address=104.36.82.54:4171 , piece_id=MUMABKARK4KKUKWW6XDXCQVT6O7SLFHGHE7COWC7QIYGARAA2D5A, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.642656750 (75h46m41.205979758s) [noise=true , address=104.36.82.14:6232 , piece_id=A2ZLUUZKRTBJRJ267QSK36C3LFZV7FPAYDDAXGIWDTNEGNV6MRRA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.643565583 (75h46m41.206888597s) [noise=true , address=79.127.205.245:4231 , piece_id=EJK46XUUVI5NWOAQJEW4QIJG3KRXZHB4YOQUJRCS2MIOSWKGTNLQ, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.644620291 (75h46m41.207943303s) [noise=true , address=77.90.31.14:4670 , piece_id=7FNANMUKPM3N6YWRAOAIM4WJZJUTNF7ZBI6ZUBNEPGNGMFWOE24A, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.648842639 (75h46m41.212165631s) [noise=true , address=104.36.82.55:4224 , piece_id=WRVNGSJV5BX4GJERLXNPAS7J7BZ5WFTM7R5H257YRCG5SP2HIYMA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.825828239 (75h46m41.389151241s) [noise=true , address=104.36.82.53:4126 , piece_id=2MKRQJEPGDJKDM7VWY5M4M6NORILAIEYYATJ7HVZEOBOMXOI2LBA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: ecclient: upload failed (node:12dMWFrq2NMpnMzuPAphWerntC5kkrcnzRJZBA116uzJzyvk6fH, address:104.36.82.53:4126): protocol: storage node overloaded, request limit: 2000 Jun 21 20:10:05.834666043 (75h46m41.397989033s) [upload=60693]: Next returned nil segment Jun 21 20:10:05.834686295 (75h46m41.398009302s) [upload=60693]: Waiting for error group managing segments... Jun 21 20:10:05.877010488 (75h46m41.440333486s) [noise=true , address=45.91.116.6:4982 , piece_id=VP45R6KXNH5U5SGUQUMAPIBY7KFDPBIR3S3VRIX27EAT6VLFBZ7Q, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:05.886336964 (75h46m41.449659967s) [noise=true , address=104.36.82.9:6039 , piece_id=I4AB6N27MYYUHWAZLJJUIBFBLOQFHDIM3YLYWPVRPNUV7VMYS2SA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: ecclient: upload failed (node:12P11WVW9mVXw5ra2gZ3Rezon6wZi8GrN5CchQN3MiVU4ts3L6V, address:104.36.82.9:6039): protocol: storage node overloaded, request limit: 2000 Jun 21 20:10:08.064514542 (75h46m43.627837553s) [noise=true , address=77.90.31.14:4670 , piece_id=7FNANMUKPM3N6YWRAOAIM4WJZJUTNF7ZBI6ZUBNEPGNGMFWOE24A, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:08.099098391 (75h46m43.662421396s) [noise=true , address=79.127.205.245:4231 , piece_id=EJK46XUUVI5NWOAQJEW4QIJG3KRXZHB4YOQUJRCS2MIOSWKGTNLQ, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:08.489461749 (75h46m44.052784752s) [noise=true , address=104.36.82.42:6807 , piece_id=ZBQZPXUDJ4VBCOSVKWAJ6SK6UD7SLPON7Y4SSCK2DQ5STFG4XCNQ, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:08.650922093 (75h46m44.214245093s) [noise=true , address=77.90.31.19:4853 , piece_id=VFEON5T2KSWM2KSLTVRZTXYL42JWENZYUEUPG3LMODGZWBMVGGPQ, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:09.917746439 (75h46m45.481069441s) [noise=true , address=104.36.82.13:6196 , piece_id=BCHMQO5KKT5BLNWQGOHH3V43LGSE3BUQCI2SY4TTO5M5GQNOCTMQ, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:10.830722215 (75h46m46.394045216s) [noise=true , address=104.36.82.5:5869 , piece_id=HAEHXXZASV3DHT3QN7NTZOS6XZKHNUWBKCHWJJWSTJKXAKJKKBAA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:12.738783450 (75h46m48.302106452s) [noise=true , address=77.90.31.12:4603 , piece_id=BMUIC6KNGEDL37436GLSXW5XOO6VYOPNOSFGQCYVHSENVI2PJ7KQ, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:15.460927713 (75h46m51.024250723s) [noise=true , address=104.36.82.52:4111 , piece_id=HB7QAWN6CGNIIT4HD2GZ7JDQP5NXIJ5I2USBM66YOKTYENNOWG5A, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:18.393817968 (75h46m53.95714096s) [noise=true , address=104.36.82.55:4224 , piece_id=WRVNGSJV5BX4GJERLXNPAS7J7BZ5WFTM7R5H257YRCG5SP2HIYMA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:19.549174143 (75h46m55.112497151s) [noise=true , address=104.36.82.56:4253 , piece_id=5PMVGLZETMXCQ77K3I6PXACRIMBGR2CV5JGUSU5KUMYFNXVGA2GA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:22.511944593 (75h46m58.075267584s) [noise=true , address=104.36.82.37:6663 , piece_id=X7LWYIIHJFU2T2PHYR3R5TNT22UUPHEHQWZH7ZJMLS4AH4PJ6AIA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:48.027197558 (75h47m23.590520562s) [noise=true , address=104.36.82.14:6232 , piece_id=A2ZLUUZKRTBJRJ267QSK36C3LFZV7FPAYDDAXGIWDTNEGNV6MRRA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:11:05.567692582 (75h47m41.131015574s) [noise=true , address=104.36.82.49:4982 , piece_id=UNLV5UDNWULEIPE4QSQHHZQPK3BQKFRB5T3H6GP7ZQ5QGSNK5V6A, seg_pos={0 0}, upload=60693]: Done uploading piece. err: ecclient: upload failed (node:1NxoYJXCXmTvQncbkXt6yH7mhEBBrzVdzhgL5J6W5TD4N3pkaG, address:104.36.82.49:4982): protocol: expected piece hash; context deadline exceeded; EOF Jun 21 20:11:05.567759198 (75h47m41.131082195s) [noise=true , address=104.36.82.54:4171 , piece_id=MUMABKARK4KKUKWW6XDXCQVT6O7SLFHGHE7COWC7QIYGARAA2D5A, seg_pos={0 0}, upload=60693]: Done uploading piece. err: ecclient: upload failed (node:125SMhDsEZruNu6LkmQn2M7R6WzvXTdk86DRsKf522u8BNiANq3, address:104.36.82.54:4171): protocol: expected piece hash; context deadline exceeded; EOF Jun 21 20:11:05.567869019 (75h47m41.131192033s) [noise=true , address=104.36.82.10:6087 , piece_id=5FQG7BPZC373PFB54HYXSG2SVKOBBBSGZCLC4XDJEYQFOLQY74MA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: ecclient: upload failed (node:1S9k5eejJcxkiG5pXjjBZm18uP8k5jCGfWyeW2QQxmuRU8ZzRA, address:104.36.82.10:6087): protocol: write tcp 159.223.143.39:38550->104.36.82.10:6087: use of closed network connection; write tcp 159.223.143.39:38550->104.36.82.10:6087: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:38550->104.36.82.10:6087: use of closed network connection Jun 21 20:11:05.567932150 (75h47m41.131255161s) [seg_pos={0 0}, upload=60693]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jun 21 20:11:05.593626175 (75h47m41.156949181s) [upload=60693]: Done uploading segments... Jun 21 20:11:05.593657274 (75h47m41.156980265s) [upload=60693]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 19.503621654s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [....X...........X...........................................] duration: 17.666597535s FAIL (2025-06-21 20:44:01 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== pload=60693]: Starting upload Jun 21 20:10:05.602866542 (75h46m41.166189557s) [upload=60693]: Done starting upload Jun 21 20:10:05.603116545 (75h46m41.166439547s) [upload=60693]: Uploading segments... Jun 21 20:10:05.603495389 (75h46m41.166818397s) [upload=60693]: Got next segment. Inline: false Jun 21 20:10:05.603566566 (75h46m41.166889558s) [upload=60693]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 21 20:10:05.603573280 (75h46m41.166896286s) [upload=60693]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 21 20:10:05.639130162 (75h46m41.202453164s) [seg_pos={0 0}, upload=60693]: Begin upload segment... Jun 21 20:10:05.639231070 (75h46m41.202554057s) [seg_pos={0 0}, upload=60693]: Done begin upload segment. Jun 21 20:10:05.639292724 (75h46m41.202615728s) [noise=true , address=104.36.82.56:4253 , piece_id=5PMVGLZETMXCQ77K3I6PXACRIMBGR2CV5JGUSU5KUMYFNXVGA2GA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.639338066 (75h46m41.202661071s) [noise=true , address=104.36.82.53:4126 , piece_id=2MKRQJEPGDJKDM7VWY5M4M6NORILAIEYYATJ7HVZEOBOMXOI2LBA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.639416264 (75h46m41.202739267s) [noise=true , address=104.36.82.5:5869 , piece_id=HAEHXXZASV3DHT3QN7NTZOS6XZKHNUWBKCHWJJWSTJKXAKJKKBAA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.639873098 (75h46m41.203196115s) [noise=true , address=104.36.82.52:4111 , piece_id=HB7QAWN6CGNIIT4HD2GZ7JDQP5NXIJ5I2USBM66YOKTYENNOWG5A, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.639887296 (75h46m41.203210304s) [noise=true , address=104.36.82.10:6087 , piece_id=5FQG7BPZC373PFB54HYXSG2SVKOBBBSGZCLC4XDJEYQFOLQY74MA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.640266623 (75h46m41.203589616s) [noise=true , address=104.36.82.13:6196 , piece_id=BCHMQO5KKT5BLNWQGOHH3V43LGSE3BUQCI2SY4TTO5M5GQNOCTMQ, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.640352274 (75h46m41.203675286s) [noise=true , address=104.36.82.37:6663 , piece_id=X7LWYIIHJFU2T2PHYR3R5TNT22UUPHEHQWZH7ZJMLS4AH4PJ6AIA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.640659767 (75h46m41.203982779s) [noise=true , address=104.36.82.42:6807 , piece_id=ZBQZPXUDJ4VBCOSVKWAJ6SK6UD7SLPON7Y4SSCK2DQ5STFG4XCNQ, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.640961701 (75h46m41.20428471s) [noise=true , address=45.91.116.6:4982 , piece_id=VP45R6KXNH5U5SGUQUMAPIBY7KFDPBIR3S3VRIX27EAT6VLFBZ7Q, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.641159578 (75h46m41.204482589s) [noise=true , address=104.36.82.9:6039 , piece_id=I4AB6N27MYYUHWAZLJJUIBFBLOQFHDIM3YLYWPVRPNUV7VMYS2SA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.641588126 (75h46m41.204911136s) [noise=true , address=77.90.31.19:4853 , piece_id=VFEON5T2KSWM2KSLTVRZTXYL42JWENZYUEUPG3LMODGZWBMVGGPQ, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.641863297 (75h46m41.205186305s) [noise=true , address=77.90.31.12:4603 , piece_id=BMUIC6KNGEDL37436GLSXW5XOO6VYOPNOSFGQCYVHSENVI2PJ7KQ, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.642124427 (75h46m41.205447437s) [noise=true , address=104.36.82.49:4982 , piece_id=UNLV5UDNWULEIPE4QSQHHZQPK3BQKFRB5T3H6GP7ZQ5QGSNK5V6A, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.642285917 (75h46m41.205608925s) [noise=true , address=104.36.82.54:4171 , piece_id=MUMABKARK4KKUKWW6XDXCQVT6O7SLFHGHE7COWC7QIYGARAA2D5A, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.642656750 (75h46m41.205979758s) [noise=true , address=104.36.82.14:6232 , piece_id=A2ZLUUZKRTBJRJ267QSK36C3LFZV7FPAYDDAXGIWDTNEGNV6MRRA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.643565583 (75h46m41.206888597s) [noise=true , address=79.127.205.245:4231 , piece_id=EJK46XUUVI5NWOAQJEW4QIJG3KRXZHB4YOQUJRCS2MIOSWKGTNLQ, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.644620291 (75h46m41.207943303s) [noise=true , address=77.90.31.14:4670 , piece_id=7FNANMUKPM3N6YWRAOAIM4WJZJUTNF7ZBI6ZUBNEPGNGMFWOE24A, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.648842639 (75h46m41.212165631s) [noise=true , address=104.36.82.55:4224 , piece_id=WRVNGSJV5BX4GJERLXNPAS7J7BZ5WFTM7R5H257YRCG5SP2HIYMA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.825828239 (75h46m41.389151241s) [noise=true , address=104.36.82.53:4126 , piece_id=2MKRQJEPGDJKDM7VWY5M4M6NORILAIEYYATJ7HVZEOBOMXOI2LBA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: ecclient: upload failed (node:12dMWFrq2NMpnMzuPAphWerntC5kkrcnzRJZBA116uzJzyvk6fH, address:104.36.82.53:4126): protocol: storage node overloaded, request limit: 2000 Jun 21 20:10:05.834666043 (75h46m41.397989033s) [upload=60693]: Next returned nil segment Jun 21 20:10:05.834686295 (75h46m41.398009302s) [upload=60693]: Waiting for error group managing segments... Jun 21 20:10:05.877010488 (75h46m41.440333486s) [noise=true , address=45.91.116.6:4982 , piece_id=VP45R6KXNH5U5SGUQUMAPIBY7KFDPBIR3S3VRIX27EAT6VLFBZ7Q, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:05.886336964 (75h46m41.449659967s) [noise=true , address=104.36.82.9:6039 , piece_id=I4AB6N27MYYUHWAZLJJUIBFBLOQFHDIM3YLYWPVRPNUV7VMYS2SA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: ecclient: upload failed (node:12P11WVW9mVXw5ra2gZ3Rezon6wZi8GrN5CchQN3MiVU4ts3L6V, address:104.36.82.9:6039): protocol: storage node overloaded, request limit: 2000 Jun 21 20:10:08.064514542 (75h46m43.627837553s) [noise=true , address=77.90.31.14:4670 , piece_id=7FNANMUKPM3N6YWRAOAIM4WJZJUTNF7ZBI6ZUBNEPGNGMFWOE24A, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:08.099098391 (75h46m43.662421396s) [noise=true , address=79.127.205.245:4231 , piece_id=EJK46XUUVI5NWOAQJEW4QIJG3KRXZHB4YOQUJRCS2MIOSWKGTNLQ, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:08.489461749 (75h46m44.052784752s) [noise=true , address=104.36.82.42:6807 , piece_id=ZBQZPXUDJ4VBCOSVKWAJ6SK6UD7SLPON7Y4SSCK2DQ5STFG4XCNQ, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:08.650922093 (75h46m44.214245093s) [noise=true , address=77.90.31.19:4853 , piece_id=VFEON5T2KSWM2KSLTVRZTXYL42JWENZYUEUPG3LMODGZWBMVGGPQ, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:09.917746439 (75h46m45.481069441s) [noise=true , address=104.36.82.13:6196 , piece_id=BCHMQO5KKT5BLNWQGOHH3V43LGSE3BUQCI2SY4TTO5M5GQNOCTMQ, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:10.830722215 (75h46m46.394045216s) [noise=true , address=104.36.82.5:5869 , piece_id=HAEHXXZASV3DHT3QN7NTZOS6XZKHNUWBKCHWJJWSTJKXAKJKKBAA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:12.738783450 (75h46m48.302106452s) [noise=true , address=77.90.31.12:4603 , piece_id=BMUIC6KNGEDL37436GLSXW5XOO6VYOPNOSFGQCYVHSENVI2PJ7KQ, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:15.460927713 (75h46m51.024250723s) [noise=true , address=104.36.82.52:4111 , piece_id=HB7QAWN6CGNIIT4HD2GZ7JDQP5NXIJ5I2USBM66YOKTYENNOWG5A, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:18.393817968 (75h46m53.95714096s) [noise=true , address=104.36.82.55:4224 , piece_id=WRVNGSJV5BX4GJERLXNPAS7J7BZ5WFTM7R5H257YRCG5SP2HIYMA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:19.549174143 (75h46m55.112497151s) [noise=true , address=104.36.82.56:4253 , piece_id=5PMVGLZETMXCQ77K3I6PXACRIMBGR2CV5JGUSU5KUMYFNXVGA2GA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:22.511944593 (75h46m58.075267584s) [noise=true , address=104.36.82.37:6663 , piece_id=X7LWYIIHJFU2T2PHYR3R5TNT22UUPHEHQWZH7ZJMLS4AH4PJ6AIA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:48.027197558 (75h47m23.590520562s) [noise=true , address=104.36.82.14:6232 , piece_id=A2ZLUUZKRTBJRJ267QSK36C3LFZV7FPAYDDAXGIWDTNEGNV6MRRA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:11:05.567692582 (75h47m41.131015574s) [noise=true , address=104.36.82.49:4982 , piece_id=UNLV5UDNWULEIPE4QSQHHZQPK3BQKFRB5T3H6GP7ZQ5QGSNK5V6A, seg_pos={0 0}, upload=60693]: Done uploading piece. err: ecclient: upload failed (node:1NxoYJXCXmTvQncbkXt6yH7mhEBBrzVdzhgL5J6W5TD4N3pkaG, address:104.36.82.49:4982): protocol: expected piece hash; context deadline exceeded; EOF Jun 21 20:11:05.567759198 (75h47m41.131082195s) [noise=true , address=104.36.82.54:4171 , piece_id=MUMABKARK4KKUKWW6XDXCQVT6O7SLFHGHE7COWC7QIYGARAA2D5A, seg_pos={0 0}, upload=60693]: Done uploading piece. err: ecclient: upload failed (node:125SMhDsEZruNu6LkmQn2M7R6WzvXTdk86DRsKf522u8BNiANq3, address:104.36.82.54:4171): protocol: expected piece hash; context deadline exceeded; EOF Jun 21 20:11:05.567869019 (75h47m41.131192033s) [noise=true , address=104.36.82.10:6087 , piece_id=5FQG7BPZC373PFB54HYXSG2SVKOBBBSGZCLC4XDJEYQFOLQY74MA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: ecclient: upload failed (node:1S9k5eejJcxkiG5pXjjBZm18uP8k5jCGfWyeW2QQxmuRU8ZzRA, address:104.36.82.10:6087): protocol: write tcp 159.223.143.39:38550->104.36.82.10:6087: use of closed network connection; write tcp 159.223.143.39:38550->104.36.82.10:6087: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:38550->104.36.82.10:6087: use of closed network connection Jun 21 20:11:05.567932150 (75h47m41.131255161s) [seg_pos={0 0}, upload=60693]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jun 21 20:11:05.593626175 (75h47m41.156949181s) [upload=60693]: Done uploading segments... Jun 21 20:11:05.593657274 (75h47m41.156980265s) [upload=60693]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 17.666629177s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [.....X...........X..........................................] duration: 10.231440768s FAIL (2025-06-21 20:43:11 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== pload=60693]: Starting upload Jun 21 20:10:05.602866542 (75h46m41.166189557s) [upload=60693]: Done starting upload Jun 21 20:10:05.603116545 (75h46m41.166439547s) [upload=60693]: Uploading segments... Jun 21 20:10:05.603495389 (75h46m41.166818397s) [upload=60693]: Got next segment. Inline: false Jun 21 20:10:05.603566566 (75h46m41.166889558s) [upload=60693]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 21 20:10:05.603573280 (75h46m41.166896286s) [upload=60693]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 21 20:10:05.639130162 (75h46m41.202453164s) [seg_pos={0 0}, upload=60693]: Begin upload segment... Jun 21 20:10:05.639231070 (75h46m41.202554057s) [seg_pos={0 0}, upload=60693]: Done begin upload segment. Jun 21 20:10:05.639292724 (75h46m41.202615728s) [noise=true , address=104.36.82.56:4253 , piece_id=5PMVGLZETMXCQ77K3I6PXACRIMBGR2CV5JGUSU5KUMYFNXVGA2GA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.639338066 (75h46m41.202661071s) [noise=true , address=104.36.82.53:4126 , piece_id=2MKRQJEPGDJKDM7VWY5M4M6NORILAIEYYATJ7HVZEOBOMXOI2LBA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.639416264 (75h46m41.202739267s) [noise=true , address=104.36.82.5:5869 , piece_id=HAEHXXZASV3DHT3QN7NTZOS6XZKHNUWBKCHWJJWSTJKXAKJKKBAA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.639873098 (75h46m41.203196115s) [noise=true , address=104.36.82.52:4111 , piece_id=HB7QAWN6CGNIIT4HD2GZ7JDQP5NXIJ5I2USBM66YOKTYENNOWG5A, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.639887296 (75h46m41.203210304s) [noise=true , address=104.36.82.10:6087 , piece_id=5FQG7BPZC373PFB54HYXSG2SVKOBBBSGZCLC4XDJEYQFOLQY74MA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.640266623 (75h46m41.203589616s) [noise=true , address=104.36.82.13:6196 , piece_id=BCHMQO5KKT5BLNWQGOHH3V43LGSE3BUQCI2SY4TTO5M5GQNOCTMQ, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.640352274 (75h46m41.203675286s) [noise=true , address=104.36.82.37:6663 , piece_id=X7LWYIIHJFU2T2PHYR3R5TNT22UUPHEHQWZH7ZJMLS4AH4PJ6AIA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.640659767 (75h46m41.203982779s) [noise=true , address=104.36.82.42:6807 , piece_id=ZBQZPXUDJ4VBCOSVKWAJ6SK6UD7SLPON7Y4SSCK2DQ5STFG4XCNQ, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.640961701 (75h46m41.20428471s) [noise=true , address=45.91.116.6:4982 , piece_id=VP45R6KXNH5U5SGUQUMAPIBY7KFDPBIR3S3VRIX27EAT6VLFBZ7Q, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.641159578 (75h46m41.204482589s) [noise=true , address=104.36.82.9:6039 , piece_id=I4AB6N27MYYUHWAZLJJUIBFBLOQFHDIM3YLYWPVRPNUV7VMYS2SA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.641588126 (75h46m41.204911136s) [noise=true , address=77.90.31.19:4853 , piece_id=VFEON5T2KSWM2KSLTVRZTXYL42JWENZYUEUPG3LMODGZWBMVGGPQ, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.641863297 (75h46m41.205186305s) [noise=true , address=77.90.31.12:4603 , piece_id=BMUIC6KNGEDL37436GLSXW5XOO6VYOPNOSFGQCYVHSENVI2PJ7KQ, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.642124427 (75h46m41.205447437s) [noise=true , address=104.36.82.49:4982 , piece_id=UNLV5UDNWULEIPE4QSQHHZQPK3BQKFRB5T3H6GP7ZQ5QGSNK5V6A, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.642285917 (75h46m41.205608925s) [noise=true , address=104.36.82.54:4171 , piece_id=MUMABKARK4KKUKWW6XDXCQVT6O7SLFHGHE7COWC7QIYGARAA2D5A, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.642656750 (75h46m41.205979758s) [noise=true , address=104.36.82.14:6232 , piece_id=A2ZLUUZKRTBJRJ267QSK36C3LFZV7FPAYDDAXGIWDTNEGNV6MRRA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.643565583 (75h46m41.206888597s) [noise=true , address=79.127.205.245:4231 , piece_id=EJK46XUUVI5NWOAQJEW4QIJG3KRXZHB4YOQUJRCS2MIOSWKGTNLQ, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.644620291 (75h46m41.207943303s) [noise=true , address=77.90.31.14:4670 , piece_id=7FNANMUKPM3N6YWRAOAIM4WJZJUTNF7ZBI6ZUBNEPGNGMFWOE24A, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.648842639 (75h46m41.212165631s) [noise=true , address=104.36.82.55:4224 , piece_id=WRVNGSJV5BX4GJERLXNPAS7J7BZ5WFTM7R5H257YRCG5SP2HIYMA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.825828239 (75h46m41.389151241s) [noise=true , address=104.36.82.53:4126 , piece_id=2MKRQJEPGDJKDM7VWY5M4M6NORILAIEYYATJ7HVZEOBOMXOI2LBA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: ecclient: upload failed (node:12dMWFrq2NMpnMzuPAphWerntC5kkrcnzRJZBA116uzJzyvk6fH, address:104.36.82.53:4126): protocol: storage node overloaded, request limit: 2000 Jun 21 20:10:05.834666043 (75h46m41.397989033s) [upload=60693]: Next returned nil segment Jun 21 20:10:05.834686295 (75h46m41.398009302s) [upload=60693]: Waiting for error group managing segments... Jun 21 20:10:05.877010488 (75h46m41.440333486s) [noise=true , address=45.91.116.6:4982 , piece_id=VP45R6KXNH5U5SGUQUMAPIBY7KFDPBIR3S3VRIX27EAT6VLFBZ7Q, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:05.886336964 (75h46m41.449659967s) [noise=true , address=104.36.82.9:6039 , piece_id=I4AB6N27MYYUHWAZLJJUIBFBLOQFHDIM3YLYWPVRPNUV7VMYS2SA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: ecclient: upload failed (node:12P11WVW9mVXw5ra2gZ3Rezon6wZi8GrN5CchQN3MiVU4ts3L6V, address:104.36.82.9:6039): protocol: storage node overloaded, request limit: 2000 Jun 21 20:10:08.064514542 (75h46m43.627837553s) [noise=true , address=77.90.31.14:4670 , piece_id=7FNANMUKPM3N6YWRAOAIM4WJZJUTNF7ZBI6ZUBNEPGNGMFWOE24A, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:08.099098391 (75h46m43.662421396s) [noise=true , address=79.127.205.245:4231 , piece_id=EJK46XUUVI5NWOAQJEW4QIJG3KRXZHB4YOQUJRCS2MIOSWKGTNLQ, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:08.489461749 (75h46m44.052784752s) [noise=true , address=104.36.82.42:6807 , piece_id=ZBQZPXUDJ4VBCOSVKWAJ6SK6UD7SLPON7Y4SSCK2DQ5STFG4XCNQ, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:08.650922093 (75h46m44.214245093s) [noise=true , address=77.90.31.19:4853 , piece_id=VFEON5T2KSWM2KSLTVRZTXYL42JWENZYUEUPG3LMODGZWBMVGGPQ, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:09.917746439 (75h46m45.481069441s) [noise=true , address=104.36.82.13:6196 , piece_id=BCHMQO5KKT5BLNWQGOHH3V43LGSE3BUQCI2SY4TTO5M5GQNOCTMQ, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:10.830722215 (75h46m46.394045216s) [noise=true , address=104.36.82.5:5869 , piece_id=HAEHXXZASV3DHT3QN7NTZOS6XZKHNUWBKCHWJJWSTJKXAKJKKBAA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:12.738783450 (75h46m48.302106452s) [noise=true , address=77.90.31.12:4603 , piece_id=BMUIC6KNGEDL37436GLSXW5XOO6VYOPNOSFGQCYVHSENVI2PJ7KQ, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:15.460927713 (75h46m51.024250723s) [noise=true , address=104.36.82.52:4111 , piece_id=HB7QAWN6CGNIIT4HD2GZ7JDQP5NXIJ5I2USBM66YOKTYENNOWG5A, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:18.393817968 (75h46m53.95714096s) [noise=true , address=104.36.82.55:4224 , piece_id=WRVNGSJV5BX4GJERLXNPAS7J7BZ5WFTM7R5H257YRCG5SP2HIYMA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:19.549174143 (75h46m55.112497151s) [noise=true , address=104.36.82.56:4253 , piece_id=5PMVGLZETMXCQ77K3I6PXACRIMBGR2CV5JGUSU5KUMYFNXVGA2GA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:22.511944593 (75h46m58.075267584s) [noise=true , address=104.36.82.37:6663 , piece_id=X7LWYIIHJFU2T2PHYR3R5TNT22UUPHEHQWZH7ZJMLS4AH4PJ6AIA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:48.027197558 (75h47m23.590520562s) [noise=true , address=104.36.82.14:6232 , piece_id=A2ZLUUZKRTBJRJ267QSK36C3LFZV7FPAYDDAXGIWDTNEGNV6MRRA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:11:05.567692582 (75h47m41.131015574s) [noise=true , address=104.36.82.49:4982 , piece_id=UNLV5UDNWULEIPE4QSQHHZQPK3BQKFRB5T3H6GP7ZQ5QGSNK5V6A, seg_pos={0 0}, upload=60693]: Done uploading piece. err: ecclient: upload failed (node:1NxoYJXCXmTvQncbkXt6yH7mhEBBrzVdzhgL5J6W5TD4N3pkaG, address:104.36.82.49:4982): protocol: expected piece hash; context deadline exceeded; EOF Jun 21 20:11:05.567759198 (75h47m41.131082195s) [noise=true , address=104.36.82.54:4171 , piece_id=MUMABKARK4KKUKWW6XDXCQVT6O7SLFHGHE7COWC7QIYGARAA2D5A, seg_pos={0 0}, upload=60693]: Done uploading piece. err: ecclient: upload failed (node:125SMhDsEZruNu6LkmQn2M7R6WzvXTdk86DRsKf522u8BNiANq3, address:104.36.82.54:4171): protocol: expected piece hash; context deadline exceeded; EOF Jun 21 20:11:05.567869019 (75h47m41.131192033s) [noise=true , address=104.36.82.10:6087 , piece_id=5FQG7BPZC373PFB54HYXSG2SVKOBBBSGZCLC4XDJEYQFOLQY74MA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: ecclient: upload failed (node:1S9k5eejJcxkiG5pXjjBZm18uP8k5jCGfWyeW2QQxmuRU8ZzRA, address:104.36.82.10:6087): protocol: write tcp 159.223.143.39:38550->104.36.82.10:6087: use of closed network connection; write tcp 159.223.143.39:38550->104.36.82.10:6087: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:38550->104.36.82.10:6087: use of closed network connection Jun 21 20:11:05.567932150 (75h47m41.131255161s) [seg_pos={0 0}, upload=60693]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jun 21 20:11:05.593626175 (75h47m41.156949181s) [upload=60693]: Done uploading segments... Jun 21 20:11:05.593657274 (75h47m41.156980265s) [upload=60693]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 10.231504788s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [......X...........X.........................................] duration: 21.247889252s FAIL (2025-06-21 20:42:57 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== pload=60693]: Starting upload Jun 21 20:10:05.602866542 (75h46m41.166189557s) [upload=60693]: Done starting upload Jun 21 20:10:05.603116545 (75h46m41.166439547s) [upload=60693]: Uploading segments... Jun 21 20:10:05.603495389 (75h46m41.166818397s) [upload=60693]: Got next segment. Inline: false Jun 21 20:10:05.603566566 (75h46m41.166889558s) [upload=60693]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 21 20:10:05.603573280 (75h46m41.166896286s) [upload=60693]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 21 20:10:05.639130162 (75h46m41.202453164s) [seg_pos={0 0}, upload=60693]: Begin upload segment... Jun 21 20:10:05.639231070 (75h46m41.202554057s) [seg_pos={0 0}, upload=60693]: Done begin upload segment. Jun 21 20:10:05.639292724 (75h46m41.202615728s) [noise=true , address=104.36.82.56:4253 , piece_id=5PMVGLZETMXCQ77K3I6PXACRIMBGR2CV5JGUSU5KUMYFNXVGA2GA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.639338066 (75h46m41.202661071s) [noise=true , address=104.36.82.53:4126 , piece_id=2MKRQJEPGDJKDM7VWY5M4M6NORILAIEYYATJ7HVZEOBOMXOI2LBA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.639416264 (75h46m41.202739267s) [noise=true , address=104.36.82.5:5869 , piece_id=HAEHXXZASV3DHT3QN7NTZOS6XZKHNUWBKCHWJJWSTJKXAKJKKBAA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.639873098 (75h46m41.203196115s) [noise=true , address=104.36.82.52:4111 , piece_id=HB7QAWN6CGNIIT4HD2GZ7JDQP5NXIJ5I2USBM66YOKTYENNOWG5A, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.639887296 (75h46m41.203210304s) [noise=true , address=104.36.82.10:6087 , piece_id=5FQG7BPZC373PFB54HYXSG2SVKOBBBSGZCLC4XDJEYQFOLQY74MA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.640266623 (75h46m41.203589616s) [noise=true , address=104.36.82.13:6196 , piece_id=BCHMQO5KKT5BLNWQGOHH3V43LGSE3BUQCI2SY4TTO5M5GQNOCTMQ, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.640352274 (75h46m41.203675286s) [noise=true , address=104.36.82.37:6663 , piece_id=X7LWYIIHJFU2T2PHYR3R5TNT22UUPHEHQWZH7ZJMLS4AH4PJ6AIA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.640659767 (75h46m41.203982779s) [noise=true , address=104.36.82.42:6807 , piece_id=ZBQZPXUDJ4VBCOSVKWAJ6SK6UD7SLPON7Y4SSCK2DQ5STFG4XCNQ, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.640961701 (75h46m41.20428471s) [noise=true , address=45.91.116.6:4982 , piece_id=VP45R6KXNH5U5SGUQUMAPIBY7KFDPBIR3S3VRIX27EAT6VLFBZ7Q, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.641159578 (75h46m41.204482589s) [noise=true , address=104.36.82.9:6039 , piece_id=I4AB6N27MYYUHWAZLJJUIBFBLOQFHDIM3YLYWPVRPNUV7VMYS2SA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.641588126 (75h46m41.204911136s) [noise=true , address=77.90.31.19:4853 , piece_id=VFEON5T2KSWM2KSLTVRZTXYL42JWENZYUEUPG3LMODGZWBMVGGPQ, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.641863297 (75h46m41.205186305s) [noise=true , address=77.90.31.12:4603 , piece_id=BMUIC6KNGEDL37436GLSXW5XOO6VYOPNOSFGQCYVHSENVI2PJ7KQ, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.642124427 (75h46m41.205447437s) [noise=true , address=104.36.82.49:4982 , piece_id=UNLV5UDNWULEIPE4QSQHHZQPK3BQKFRB5T3H6GP7ZQ5QGSNK5V6A, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.642285917 (75h46m41.205608925s) [noise=true , address=104.36.82.54:4171 , piece_id=MUMABKARK4KKUKWW6XDXCQVT6O7SLFHGHE7COWC7QIYGARAA2D5A, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.642656750 (75h46m41.205979758s) [noise=true , address=104.36.82.14:6232 , piece_id=A2ZLUUZKRTBJRJ267QSK36C3LFZV7FPAYDDAXGIWDTNEGNV6MRRA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.643565583 (75h46m41.206888597s) [noise=true , address=79.127.205.245:4231 , piece_id=EJK46XUUVI5NWOAQJEW4QIJG3KRXZHB4YOQUJRCS2MIOSWKGTNLQ, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.644620291 (75h46m41.207943303s) [noise=true , address=77.90.31.14:4670 , piece_id=7FNANMUKPM3N6YWRAOAIM4WJZJUTNF7ZBI6ZUBNEPGNGMFWOE24A, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.648842639 (75h46m41.212165631s) [noise=true , address=104.36.82.55:4224 , piece_id=WRVNGSJV5BX4GJERLXNPAS7J7BZ5WFTM7R5H257YRCG5SP2HIYMA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.825828239 (75h46m41.389151241s) [noise=true , address=104.36.82.53:4126 , piece_id=2MKRQJEPGDJKDM7VWY5M4M6NORILAIEYYATJ7HVZEOBOMXOI2LBA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: ecclient: upload failed (node:12dMWFrq2NMpnMzuPAphWerntC5kkrcnzRJZBA116uzJzyvk6fH, address:104.36.82.53:4126): protocol: storage node overloaded, request limit: 2000 Jun 21 20:10:05.834666043 (75h46m41.397989033s) [upload=60693]: Next returned nil segment Jun 21 20:10:05.834686295 (75h46m41.398009302s) [upload=60693]: Waiting for error group managing segments... Jun 21 20:10:05.877010488 (75h46m41.440333486s) [noise=true , address=45.91.116.6:4982 , piece_id=VP45R6KXNH5U5SGUQUMAPIBY7KFDPBIR3S3VRIX27EAT6VLFBZ7Q, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:05.886336964 (75h46m41.449659967s) [noise=true , address=104.36.82.9:6039 , piece_id=I4AB6N27MYYUHWAZLJJUIBFBLOQFHDIM3YLYWPVRPNUV7VMYS2SA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: ecclient: upload failed (node:12P11WVW9mVXw5ra2gZ3Rezon6wZi8GrN5CchQN3MiVU4ts3L6V, address:104.36.82.9:6039): protocol: storage node overloaded, request limit: 2000 Jun 21 20:10:08.064514542 (75h46m43.627837553s) [noise=true , address=77.90.31.14:4670 , piece_id=7FNANMUKPM3N6YWRAOAIM4WJZJUTNF7ZBI6ZUBNEPGNGMFWOE24A, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:08.099098391 (75h46m43.662421396s) [noise=true , address=79.127.205.245:4231 , piece_id=EJK46XUUVI5NWOAQJEW4QIJG3KRXZHB4YOQUJRCS2MIOSWKGTNLQ, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:08.489461749 (75h46m44.052784752s) [noise=true , address=104.36.82.42:6807 , piece_id=ZBQZPXUDJ4VBCOSVKWAJ6SK6UD7SLPON7Y4SSCK2DQ5STFG4XCNQ, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:08.650922093 (75h46m44.214245093s) [noise=true , address=77.90.31.19:4853 , piece_id=VFEON5T2KSWM2KSLTVRZTXYL42JWENZYUEUPG3LMODGZWBMVGGPQ, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:09.917746439 (75h46m45.481069441s) [noise=true , address=104.36.82.13:6196 , piece_id=BCHMQO5KKT5BLNWQGOHH3V43LGSE3BUQCI2SY4TTO5M5GQNOCTMQ, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:10.830722215 (75h46m46.394045216s) [noise=true , address=104.36.82.5:5869 , piece_id=HAEHXXZASV3DHT3QN7NTZOS6XZKHNUWBKCHWJJWSTJKXAKJKKBAA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:12.738783450 (75h46m48.302106452s) [noise=true , address=77.90.31.12:4603 , piece_id=BMUIC6KNGEDL37436GLSXW5XOO6VYOPNOSFGQCYVHSENVI2PJ7KQ, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:15.460927713 (75h46m51.024250723s) [noise=true , address=104.36.82.52:4111 , piece_id=HB7QAWN6CGNIIT4HD2GZ7JDQP5NXIJ5I2USBM66YOKTYENNOWG5A, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:18.393817968 (75h46m53.95714096s) [noise=true , address=104.36.82.55:4224 , piece_id=WRVNGSJV5BX4GJERLXNPAS7J7BZ5WFTM7R5H257YRCG5SP2HIYMA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:19.549174143 (75h46m55.112497151s) [noise=true , address=104.36.82.56:4253 , piece_id=5PMVGLZETMXCQ77K3I6PXACRIMBGR2CV5JGUSU5KUMYFNXVGA2GA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:22.511944593 (75h46m58.075267584s) [noise=true , address=104.36.82.37:6663 , piece_id=X7LWYIIHJFU2T2PHYR3R5TNT22UUPHEHQWZH7ZJMLS4AH4PJ6AIA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:48.027197558 (75h47m23.590520562s) [noise=true , address=104.36.82.14:6232 , piece_id=A2ZLUUZKRTBJRJ267QSK36C3LFZV7FPAYDDAXGIWDTNEGNV6MRRA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:11:05.567692582 (75h47m41.131015574s) [noise=true , address=104.36.82.49:4982 , piece_id=UNLV5UDNWULEIPE4QSQHHZQPK3BQKFRB5T3H6GP7ZQ5QGSNK5V6A, seg_pos={0 0}, upload=60693]: Done uploading piece. err: ecclient: upload failed (node:1NxoYJXCXmTvQncbkXt6yH7mhEBBrzVdzhgL5J6W5TD4N3pkaG, address:104.36.82.49:4982): protocol: expected piece hash; context deadline exceeded; EOF Jun 21 20:11:05.567759198 (75h47m41.131082195s) [noise=true , address=104.36.82.54:4171 , piece_id=MUMABKARK4KKUKWW6XDXCQVT6O7SLFHGHE7COWC7QIYGARAA2D5A, seg_pos={0 0}, upload=60693]: Done uploading piece. err: ecclient: upload failed (node:125SMhDsEZruNu6LkmQn2M7R6WzvXTdk86DRsKf522u8BNiANq3, address:104.36.82.54:4171): protocol: expected piece hash; context deadline exceeded; EOF Jun 21 20:11:05.567869019 (75h47m41.131192033s) [noise=true , address=104.36.82.10:6087 , piece_id=5FQG7BPZC373PFB54HYXSG2SVKOBBBSGZCLC4XDJEYQFOLQY74MA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: ecclient: upload failed (node:1S9k5eejJcxkiG5pXjjBZm18uP8k5jCGfWyeW2QQxmuRU8ZzRA, address:104.36.82.10:6087): protocol: write tcp 159.223.143.39:38550->104.36.82.10:6087: use of closed network connection; write tcp 159.223.143.39:38550->104.36.82.10:6087: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:38550->104.36.82.10:6087: use of closed network connection Jun 21 20:11:05.567932150 (75h47m41.131255161s) [seg_pos={0 0}, upload=60693]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jun 21 20:11:05.593626175 (75h47m41.156949181s) [upload=60693]: Done uploading segments... Jun 21 20:11:05.593657274 (75h47m41.156980265s) [upload=60693]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 21.247944831s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [.......X...........X........................................] duration: 13.320181631s FAIL (2025-06-21 20:41:35 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== pload=60693]: Starting upload Jun 21 20:10:05.602866542 (75h46m41.166189557s) [upload=60693]: Done starting upload Jun 21 20:10:05.603116545 (75h46m41.166439547s) [upload=60693]: Uploading segments... Jun 21 20:10:05.603495389 (75h46m41.166818397s) [upload=60693]: Got next segment. Inline: false Jun 21 20:10:05.603566566 (75h46m41.166889558s) [upload=60693]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 21 20:10:05.603573280 (75h46m41.166896286s) [upload=60693]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 21 20:10:05.639130162 (75h46m41.202453164s) [seg_pos={0 0}, upload=60693]: Begin upload segment... Jun 21 20:10:05.639231070 (75h46m41.202554057s) [seg_pos={0 0}, upload=60693]: Done begin upload segment. Jun 21 20:10:05.639292724 (75h46m41.202615728s) [noise=true , address=104.36.82.56:4253 , piece_id=5PMVGLZETMXCQ77K3I6PXACRIMBGR2CV5JGUSU5KUMYFNXVGA2GA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.639338066 (75h46m41.202661071s) [noise=true , address=104.36.82.53:4126 , piece_id=2MKRQJEPGDJKDM7VWY5M4M6NORILAIEYYATJ7HVZEOBOMXOI2LBA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.639416264 (75h46m41.202739267s) [noise=true , address=104.36.82.5:5869 , piece_id=HAEHXXZASV3DHT3QN7NTZOS6XZKHNUWBKCHWJJWSTJKXAKJKKBAA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.639873098 (75h46m41.203196115s) [noise=true , address=104.36.82.52:4111 , piece_id=HB7QAWN6CGNIIT4HD2GZ7JDQP5NXIJ5I2USBM66YOKTYENNOWG5A, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.639887296 (75h46m41.203210304s) [noise=true , address=104.36.82.10:6087 , piece_id=5FQG7BPZC373PFB54HYXSG2SVKOBBBSGZCLC4XDJEYQFOLQY74MA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.640266623 (75h46m41.203589616s) [noise=true , address=104.36.82.13:6196 , piece_id=BCHMQO5KKT5BLNWQGOHH3V43LGSE3BUQCI2SY4TTO5M5GQNOCTMQ, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.640352274 (75h46m41.203675286s) [noise=true , address=104.36.82.37:6663 , piece_id=X7LWYIIHJFU2T2PHYR3R5TNT22UUPHEHQWZH7ZJMLS4AH4PJ6AIA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.640659767 (75h46m41.203982779s) [noise=true , address=104.36.82.42:6807 , piece_id=ZBQZPXUDJ4VBCOSVKWAJ6SK6UD7SLPON7Y4SSCK2DQ5STFG4XCNQ, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.640961701 (75h46m41.20428471s) [noise=true , address=45.91.116.6:4982 , piece_id=VP45R6KXNH5U5SGUQUMAPIBY7KFDPBIR3S3VRIX27EAT6VLFBZ7Q, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.641159578 (75h46m41.204482589s) [noise=true , address=104.36.82.9:6039 , piece_id=I4AB6N27MYYUHWAZLJJUIBFBLOQFHDIM3YLYWPVRPNUV7VMYS2SA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.641588126 (75h46m41.204911136s) [noise=true , address=77.90.31.19:4853 , piece_id=VFEON5T2KSWM2KSLTVRZTXYL42JWENZYUEUPG3LMODGZWBMVGGPQ, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.641863297 (75h46m41.205186305s) [noise=true , address=77.90.31.12:4603 , piece_id=BMUIC6KNGEDL37436GLSXW5XOO6VYOPNOSFGQCYVHSENVI2PJ7KQ, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.642124427 (75h46m41.205447437s) [noise=true , address=104.36.82.49:4982 , piece_id=UNLV5UDNWULEIPE4QSQHHZQPK3BQKFRB5T3H6GP7ZQ5QGSNK5V6A, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.642285917 (75h46m41.205608925s) [noise=true , address=104.36.82.54:4171 , piece_id=MUMABKARK4KKUKWW6XDXCQVT6O7SLFHGHE7COWC7QIYGARAA2D5A, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.642656750 (75h46m41.205979758s) [noise=true , address=104.36.82.14:6232 , piece_id=A2ZLUUZKRTBJRJ267QSK36C3LFZV7FPAYDDAXGIWDTNEGNV6MRRA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.643565583 (75h46m41.206888597s) [noise=true , address=79.127.205.245:4231 , piece_id=EJK46XUUVI5NWOAQJEW4QIJG3KRXZHB4YOQUJRCS2MIOSWKGTNLQ, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.644620291 (75h46m41.207943303s) [noise=true , address=77.90.31.14:4670 , piece_id=7FNANMUKPM3N6YWRAOAIM4WJZJUTNF7ZBI6ZUBNEPGNGMFWOE24A, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.648842639 (75h46m41.212165631s) [noise=true , address=104.36.82.55:4224 , piece_id=WRVNGSJV5BX4GJERLXNPAS7J7BZ5WFTM7R5H257YRCG5SP2HIYMA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.825828239 (75h46m41.389151241s) [noise=true , address=104.36.82.53:4126 , piece_id=2MKRQJEPGDJKDM7VWY5M4M6NORILAIEYYATJ7HVZEOBOMXOI2LBA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: ecclient: upload failed (node:12dMWFrq2NMpnMzuPAphWerntC5kkrcnzRJZBA116uzJzyvk6fH, address:104.36.82.53:4126): protocol: storage node overloaded, request limit: 2000 Jun 21 20:10:05.834666043 (75h46m41.397989033s) [upload=60693]: Next returned nil segment Jun 21 20:10:05.834686295 (75h46m41.398009302s) [upload=60693]: Waiting for error group managing segments... Jun 21 20:10:05.877010488 (75h46m41.440333486s) [noise=true , address=45.91.116.6:4982 , piece_id=VP45R6KXNH5U5SGUQUMAPIBY7KFDPBIR3S3VRIX27EAT6VLFBZ7Q, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:05.886336964 (75h46m41.449659967s) [noise=true , address=104.36.82.9:6039 , piece_id=I4AB6N27MYYUHWAZLJJUIBFBLOQFHDIM3YLYWPVRPNUV7VMYS2SA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: ecclient: upload failed (node:12P11WVW9mVXw5ra2gZ3Rezon6wZi8GrN5CchQN3MiVU4ts3L6V, address:104.36.82.9:6039): protocol: storage node overloaded, request limit: 2000 Jun 21 20:10:08.064514542 (75h46m43.627837553s) [noise=true , address=77.90.31.14:4670 , piece_id=7FNANMUKPM3N6YWRAOAIM4WJZJUTNF7ZBI6ZUBNEPGNGMFWOE24A, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:08.099098391 (75h46m43.662421396s) [noise=true , address=79.127.205.245:4231 , piece_id=EJK46XUUVI5NWOAQJEW4QIJG3KRXZHB4YOQUJRCS2MIOSWKGTNLQ, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:08.489461749 (75h46m44.052784752s) [noise=true , address=104.36.82.42:6807 , piece_id=ZBQZPXUDJ4VBCOSVKWAJ6SK6UD7SLPON7Y4SSCK2DQ5STFG4XCNQ, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:08.650922093 (75h46m44.214245093s) [noise=true , address=77.90.31.19:4853 , piece_id=VFEON5T2KSWM2KSLTVRZTXYL42JWENZYUEUPG3LMODGZWBMVGGPQ, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:09.917746439 (75h46m45.481069441s) [noise=true , address=104.36.82.13:6196 , piece_id=BCHMQO5KKT5BLNWQGOHH3V43LGSE3BUQCI2SY4TTO5M5GQNOCTMQ, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:10.830722215 (75h46m46.394045216s) [noise=true , address=104.36.82.5:5869 , piece_id=HAEHXXZASV3DHT3QN7NTZOS6XZKHNUWBKCHWJJWSTJKXAKJKKBAA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:12.738783450 (75h46m48.302106452s) [noise=true , address=77.90.31.12:4603 , piece_id=BMUIC6KNGEDL37436GLSXW5XOO6VYOPNOSFGQCYVHSENVI2PJ7KQ, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:15.460927713 (75h46m51.024250723s) [noise=true , address=104.36.82.52:4111 , piece_id=HB7QAWN6CGNIIT4HD2GZ7JDQP5NXIJ5I2USBM66YOKTYENNOWG5A, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:18.393817968 (75h46m53.95714096s) [noise=true , address=104.36.82.55:4224 , piece_id=WRVNGSJV5BX4GJERLXNPAS7J7BZ5WFTM7R5H257YRCG5SP2HIYMA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:19.549174143 (75h46m55.112497151s) [noise=true , address=104.36.82.56:4253 , piece_id=5PMVGLZETMXCQ77K3I6PXACRIMBGR2CV5JGUSU5KUMYFNXVGA2GA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:22.511944593 (75h46m58.075267584s) [noise=true , address=104.36.82.37:6663 , piece_id=X7LWYIIHJFU2T2PHYR3R5TNT22UUPHEHQWZH7ZJMLS4AH4PJ6AIA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:48.027197558 (75h47m23.590520562s) [noise=true , address=104.36.82.14:6232 , piece_id=A2ZLUUZKRTBJRJ267QSK36C3LFZV7FPAYDDAXGIWDTNEGNV6MRRA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:11:05.567692582 (75h47m41.131015574s) [noise=true , address=104.36.82.49:4982 , piece_id=UNLV5UDNWULEIPE4QSQHHZQPK3BQKFRB5T3H6GP7ZQ5QGSNK5V6A, seg_pos={0 0}, upload=60693]: Done uploading piece. err: ecclient: upload failed (node:1NxoYJXCXmTvQncbkXt6yH7mhEBBrzVdzhgL5J6W5TD4N3pkaG, address:104.36.82.49:4982): protocol: expected piece hash; context deadline exceeded; EOF Jun 21 20:11:05.567759198 (75h47m41.131082195s) [noise=true , address=104.36.82.54:4171 , piece_id=MUMABKARK4KKUKWW6XDXCQVT6O7SLFHGHE7COWC7QIYGARAA2D5A, seg_pos={0 0}, upload=60693]: Done uploading piece. err: ecclient: upload failed (node:125SMhDsEZruNu6LkmQn2M7R6WzvXTdk86DRsKf522u8BNiANq3, address:104.36.82.54:4171): protocol: expected piece hash; context deadline exceeded; EOF Jun 21 20:11:05.567869019 (75h47m41.131192033s) [noise=true , address=104.36.82.10:6087 , piece_id=5FQG7BPZC373PFB54HYXSG2SVKOBBBSGZCLC4XDJEYQFOLQY74MA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: ecclient: upload failed (node:1S9k5eejJcxkiG5pXjjBZm18uP8k5jCGfWyeW2QQxmuRU8ZzRA, address:104.36.82.10:6087): protocol: write tcp 159.223.143.39:38550->104.36.82.10:6087: use of closed network connection; write tcp 159.223.143.39:38550->104.36.82.10:6087: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:38550->104.36.82.10:6087: use of closed network connection Jun 21 20:11:05.567932150 (75h47m41.131255161s) [seg_pos={0 0}, upload=60693]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jun 21 20:11:05.593626175 (75h47m41.156949181s) [upload=60693]: Done uploading segments... Jun 21 20:11:05.593657274 (75h47m41.156980265s) [upload=60693]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 13.320204832s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [........X...........X.......................................] duration: 24.423431786s FAIL (2025-06-21 20:41:22 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== pload=60693]: Starting upload Jun 21 20:10:05.602866542 (75h46m41.166189557s) [upload=60693]: Done starting upload Jun 21 20:10:05.603116545 (75h46m41.166439547s) [upload=60693]: Uploading segments... Jun 21 20:10:05.603495389 (75h46m41.166818397s) [upload=60693]: Got next segment. Inline: false Jun 21 20:10:05.603566566 (75h46m41.166889558s) [upload=60693]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 21 20:10:05.603573280 (75h46m41.166896286s) [upload=60693]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 21 20:10:05.639130162 (75h46m41.202453164s) [seg_pos={0 0}, upload=60693]: Begin upload segment... Jun 21 20:10:05.639231070 (75h46m41.202554057s) [seg_pos={0 0}, upload=60693]: Done begin upload segment. Jun 21 20:10:05.639292724 (75h46m41.202615728s) [noise=true , address=104.36.82.56:4253 , piece_id=5PMVGLZETMXCQ77K3I6PXACRIMBGR2CV5JGUSU5KUMYFNXVGA2GA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.639338066 (75h46m41.202661071s) [noise=true , address=104.36.82.53:4126 , piece_id=2MKRQJEPGDJKDM7VWY5M4M6NORILAIEYYATJ7HVZEOBOMXOI2LBA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.639416264 (75h46m41.202739267s) [noise=true , address=104.36.82.5:5869 , piece_id=HAEHXXZASV3DHT3QN7NTZOS6XZKHNUWBKCHWJJWSTJKXAKJKKBAA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.639873098 (75h46m41.203196115s) [noise=true , address=104.36.82.52:4111 , piece_id=HB7QAWN6CGNIIT4HD2GZ7JDQP5NXIJ5I2USBM66YOKTYENNOWG5A, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.639887296 (75h46m41.203210304s) [noise=true , address=104.36.82.10:6087 , piece_id=5FQG7BPZC373PFB54HYXSG2SVKOBBBSGZCLC4XDJEYQFOLQY74MA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.640266623 (75h46m41.203589616s) [noise=true , address=104.36.82.13:6196 , piece_id=BCHMQO5KKT5BLNWQGOHH3V43LGSE3BUQCI2SY4TTO5M5GQNOCTMQ, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.640352274 (75h46m41.203675286s) [noise=true , address=104.36.82.37:6663 , piece_id=X7LWYIIHJFU2T2PHYR3R5TNT22UUPHEHQWZH7ZJMLS4AH4PJ6AIA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.640659767 (75h46m41.203982779s) [noise=true , address=104.36.82.42:6807 , piece_id=ZBQZPXUDJ4VBCOSVKWAJ6SK6UD7SLPON7Y4SSCK2DQ5STFG4XCNQ, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.640961701 (75h46m41.20428471s) [noise=true , address=45.91.116.6:4982 , piece_id=VP45R6KXNH5U5SGUQUMAPIBY7KFDPBIR3S3VRIX27EAT6VLFBZ7Q, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.641159578 (75h46m41.204482589s) [noise=true , address=104.36.82.9:6039 , piece_id=I4AB6N27MYYUHWAZLJJUIBFBLOQFHDIM3YLYWPVRPNUV7VMYS2SA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.641588126 (75h46m41.204911136s) [noise=true , address=77.90.31.19:4853 , piece_id=VFEON5T2KSWM2KSLTVRZTXYL42JWENZYUEUPG3LMODGZWBMVGGPQ, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.641863297 (75h46m41.205186305s) [noise=true , address=77.90.31.12:4603 , piece_id=BMUIC6KNGEDL37436GLSXW5XOO6VYOPNOSFGQCYVHSENVI2PJ7KQ, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.642124427 (75h46m41.205447437s) [noise=true , address=104.36.82.49:4982 , piece_id=UNLV5UDNWULEIPE4QSQHHZQPK3BQKFRB5T3H6GP7ZQ5QGSNK5V6A, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.642285917 (75h46m41.205608925s) [noise=true , address=104.36.82.54:4171 , piece_id=MUMABKARK4KKUKWW6XDXCQVT6O7SLFHGHE7COWC7QIYGARAA2D5A, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.642656750 (75h46m41.205979758s) [noise=true , address=104.36.82.14:6232 , piece_id=A2ZLUUZKRTBJRJ267QSK36C3LFZV7FPAYDDAXGIWDTNEGNV6MRRA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.643565583 (75h46m41.206888597s) [noise=true , address=79.127.205.245:4231 , piece_id=EJK46XUUVI5NWOAQJEW4QIJG3KRXZHB4YOQUJRCS2MIOSWKGTNLQ, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.644620291 (75h46m41.207943303s) [noise=true , address=77.90.31.14:4670 , piece_id=7FNANMUKPM3N6YWRAOAIM4WJZJUTNF7ZBI6ZUBNEPGNGMFWOE24A, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.648842639 (75h46m41.212165631s) [noise=true , address=104.36.82.55:4224 , piece_id=WRVNGSJV5BX4GJERLXNPAS7J7BZ5WFTM7R5H257YRCG5SP2HIYMA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.825828239 (75h46m41.389151241s) [noise=true , address=104.36.82.53:4126 , piece_id=2MKRQJEPGDJKDM7VWY5M4M6NORILAIEYYATJ7HVZEOBOMXOI2LBA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: ecclient: upload failed (node:12dMWFrq2NMpnMzuPAphWerntC5kkrcnzRJZBA116uzJzyvk6fH, address:104.36.82.53:4126): protocol: storage node overloaded, request limit: 2000 Jun 21 20:10:05.834666043 (75h46m41.397989033s) [upload=60693]: Next returned nil segment Jun 21 20:10:05.834686295 (75h46m41.398009302s) [upload=60693]: Waiting for error group managing segments... Jun 21 20:10:05.877010488 (75h46m41.440333486s) [noise=true , address=45.91.116.6:4982 , piece_id=VP45R6KXNH5U5SGUQUMAPIBY7KFDPBIR3S3VRIX27EAT6VLFBZ7Q, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:05.886336964 (75h46m41.449659967s) [noise=true , address=104.36.82.9:6039 , piece_id=I4AB6N27MYYUHWAZLJJUIBFBLOQFHDIM3YLYWPVRPNUV7VMYS2SA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: ecclient: upload failed (node:12P11WVW9mVXw5ra2gZ3Rezon6wZi8GrN5CchQN3MiVU4ts3L6V, address:104.36.82.9:6039): protocol: storage node overloaded, request limit: 2000 Jun 21 20:10:08.064514542 (75h46m43.627837553s) [noise=true , address=77.90.31.14:4670 , piece_id=7FNANMUKPM3N6YWRAOAIM4WJZJUTNF7ZBI6ZUBNEPGNGMFWOE24A, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:08.099098391 (75h46m43.662421396s) [noise=true , address=79.127.205.245:4231 , piece_id=EJK46XUUVI5NWOAQJEW4QIJG3KRXZHB4YOQUJRCS2MIOSWKGTNLQ, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:08.489461749 (75h46m44.052784752s) [noise=true , address=104.36.82.42:6807 , piece_id=ZBQZPXUDJ4VBCOSVKWAJ6SK6UD7SLPON7Y4SSCK2DQ5STFG4XCNQ, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:08.650922093 (75h46m44.214245093s) [noise=true , address=77.90.31.19:4853 , piece_id=VFEON5T2KSWM2KSLTVRZTXYL42JWENZYUEUPG3LMODGZWBMVGGPQ, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:09.917746439 (75h46m45.481069441s) [noise=true , address=104.36.82.13:6196 , piece_id=BCHMQO5KKT5BLNWQGOHH3V43LGSE3BUQCI2SY4TTO5M5GQNOCTMQ, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:10.830722215 (75h46m46.394045216s) [noise=true , address=104.36.82.5:5869 , piece_id=HAEHXXZASV3DHT3QN7NTZOS6XZKHNUWBKCHWJJWSTJKXAKJKKBAA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:12.738783450 (75h46m48.302106452s) [noise=true , address=77.90.31.12:4603 , piece_id=BMUIC6KNGEDL37436GLSXW5XOO6VYOPNOSFGQCYVHSENVI2PJ7KQ, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:15.460927713 (75h46m51.024250723s) [noise=true , address=104.36.82.52:4111 , piece_id=HB7QAWN6CGNIIT4HD2GZ7JDQP5NXIJ5I2USBM66YOKTYENNOWG5A, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:18.393817968 (75h46m53.95714096s) [noise=true , address=104.36.82.55:4224 , piece_id=WRVNGSJV5BX4GJERLXNPAS7J7BZ5WFTM7R5H257YRCG5SP2HIYMA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:19.549174143 (75h46m55.112497151s) [noise=true , address=104.36.82.56:4253 , piece_id=5PMVGLZETMXCQ77K3I6PXACRIMBGR2CV5JGUSU5KUMYFNXVGA2GA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:22.511944593 (75h46m58.075267584s) [noise=true , address=104.36.82.37:6663 , piece_id=X7LWYIIHJFU2T2PHYR3R5TNT22UUPHEHQWZH7ZJMLS4AH4PJ6AIA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:48.027197558 (75h47m23.590520562s) [noise=true , address=104.36.82.14:6232 , piece_id=A2ZLUUZKRTBJRJ267QSK36C3LFZV7FPAYDDAXGIWDTNEGNV6MRRA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:11:05.567692582 (75h47m41.131015574s) [noise=true , address=104.36.82.49:4982 , piece_id=UNLV5UDNWULEIPE4QSQHHZQPK3BQKFRB5T3H6GP7ZQ5QGSNK5V6A, seg_pos={0 0}, upload=60693]: Done uploading piece. err: ecclient: upload failed (node:1NxoYJXCXmTvQncbkXt6yH7mhEBBrzVdzhgL5J6W5TD4N3pkaG, address:104.36.82.49:4982): protocol: expected piece hash; context deadline exceeded; EOF Jun 21 20:11:05.567759198 (75h47m41.131082195s) [noise=true , address=104.36.82.54:4171 , piece_id=MUMABKARK4KKUKWW6XDXCQVT6O7SLFHGHE7COWC7QIYGARAA2D5A, seg_pos={0 0}, upload=60693]: Done uploading piece. err: ecclient: upload failed (node:125SMhDsEZruNu6LkmQn2M7R6WzvXTdk86DRsKf522u8BNiANq3, address:104.36.82.54:4171): protocol: expected piece hash; context deadline exceeded; EOF Jun 21 20:11:05.567869019 (75h47m41.131192033s) [noise=true , address=104.36.82.10:6087 , piece_id=5FQG7BPZC373PFB54HYXSG2SVKOBBBSGZCLC4XDJEYQFOLQY74MA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: ecclient: upload failed (node:1S9k5eejJcxkiG5pXjjBZm18uP8k5jCGfWyeW2QQxmuRU8ZzRA, address:104.36.82.10:6087): protocol: write tcp 159.223.143.39:38550->104.36.82.10:6087: use of closed network connection; write tcp 159.223.143.39:38550->104.36.82.10:6087: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:38550->104.36.82.10:6087: use of closed network connection Jun 21 20:11:05.567932150 (75h47m41.131255161s) [seg_pos={0 0}, upload=60693]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jun 21 20:11:05.593626175 (75h47m41.156949181s) [upload=60693]: Done uploading segments... Jun 21 20:11:05.593657274 (75h47m41.156980265s) [upload=60693]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 24.423461249s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [.........X...........X......................................] duration: 19.419107312s FAIL (2025-06-21 20:40:43 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== pload=60693]: Starting upload Jun 21 20:10:05.602866542 (75h46m41.166189557s) [upload=60693]: Done starting upload Jun 21 20:10:05.603116545 (75h46m41.166439547s) [upload=60693]: Uploading segments... Jun 21 20:10:05.603495389 (75h46m41.166818397s) [upload=60693]: Got next segment. Inline: false Jun 21 20:10:05.603566566 (75h46m41.166889558s) [upload=60693]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 21 20:10:05.603573280 (75h46m41.166896286s) [upload=60693]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 21 20:10:05.639130162 (75h46m41.202453164s) [seg_pos={0 0}, upload=60693]: Begin upload segment... Jun 21 20:10:05.639231070 (75h46m41.202554057s) [seg_pos={0 0}, upload=60693]: Done begin upload segment. Jun 21 20:10:05.639292724 (75h46m41.202615728s) [noise=true , address=104.36.82.56:4253 , piece_id=5PMVGLZETMXCQ77K3I6PXACRIMBGR2CV5JGUSU5KUMYFNXVGA2GA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.639338066 (75h46m41.202661071s) [noise=true , address=104.36.82.53:4126 , piece_id=2MKRQJEPGDJKDM7VWY5M4M6NORILAIEYYATJ7HVZEOBOMXOI2LBA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.639416264 (75h46m41.202739267s) [noise=true , address=104.36.82.5:5869 , piece_id=HAEHXXZASV3DHT3QN7NTZOS6XZKHNUWBKCHWJJWSTJKXAKJKKBAA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.639873098 (75h46m41.203196115s) [noise=true , address=104.36.82.52:4111 , piece_id=HB7QAWN6CGNIIT4HD2GZ7JDQP5NXIJ5I2USBM66YOKTYENNOWG5A, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.639887296 (75h46m41.203210304s) [noise=true , address=104.36.82.10:6087 , piece_id=5FQG7BPZC373PFB54HYXSG2SVKOBBBSGZCLC4XDJEYQFOLQY74MA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.640266623 (75h46m41.203589616s) [noise=true , address=104.36.82.13:6196 , piece_id=BCHMQO5KKT5BLNWQGOHH3V43LGSE3BUQCI2SY4TTO5M5GQNOCTMQ, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.640352274 (75h46m41.203675286s) [noise=true , address=104.36.82.37:6663 , piece_id=X7LWYIIHJFU2T2PHYR3R5TNT22UUPHEHQWZH7ZJMLS4AH4PJ6AIA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.640659767 (75h46m41.203982779s) [noise=true , address=104.36.82.42:6807 , piece_id=ZBQZPXUDJ4VBCOSVKWAJ6SK6UD7SLPON7Y4SSCK2DQ5STFG4XCNQ, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.640961701 (75h46m41.20428471s) [noise=true , address=45.91.116.6:4982 , piece_id=VP45R6KXNH5U5SGUQUMAPIBY7KFDPBIR3S3VRIX27EAT6VLFBZ7Q, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.641159578 (75h46m41.204482589s) [noise=true , address=104.36.82.9:6039 , piece_id=I4AB6N27MYYUHWAZLJJUIBFBLOQFHDIM3YLYWPVRPNUV7VMYS2SA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.641588126 (75h46m41.204911136s) [noise=true , address=77.90.31.19:4853 , piece_id=VFEON5T2KSWM2KSLTVRZTXYL42JWENZYUEUPG3LMODGZWBMVGGPQ, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.641863297 (75h46m41.205186305s) [noise=true , address=77.90.31.12:4603 , piece_id=BMUIC6KNGEDL37436GLSXW5XOO6VYOPNOSFGQCYVHSENVI2PJ7KQ, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.642124427 (75h46m41.205447437s) [noise=true , address=104.36.82.49:4982 , piece_id=UNLV5UDNWULEIPE4QSQHHZQPK3BQKFRB5T3H6GP7ZQ5QGSNK5V6A, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.642285917 (75h46m41.205608925s) [noise=true , address=104.36.82.54:4171 , piece_id=MUMABKARK4KKUKWW6XDXCQVT6O7SLFHGHE7COWC7QIYGARAA2D5A, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.642656750 (75h46m41.205979758s) [noise=true , address=104.36.82.14:6232 , piece_id=A2ZLUUZKRTBJRJ267QSK36C3LFZV7FPAYDDAXGIWDTNEGNV6MRRA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.643565583 (75h46m41.206888597s) [noise=true , address=79.127.205.245:4231 , piece_id=EJK46XUUVI5NWOAQJEW4QIJG3KRXZHB4YOQUJRCS2MIOSWKGTNLQ, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.644620291 (75h46m41.207943303s) [noise=true , address=77.90.31.14:4670 , piece_id=7FNANMUKPM3N6YWRAOAIM4WJZJUTNF7ZBI6ZUBNEPGNGMFWOE24A, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.648842639 (75h46m41.212165631s) [noise=true , address=104.36.82.55:4224 , piece_id=WRVNGSJV5BX4GJERLXNPAS7J7BZ5WFTM7R5H257YRCG5SP2HIYMA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.825828239 (75h46m41.389151241s) [noise=true , address=104.36.82.53:4126 , piece_id=2MKRQJEPGDJKDM7VWY5M4M6NORILAIEYYATJ7HVZEOBOMXOI2LBA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: ecclient: upload failed (node:12dMWFrq2NMpnMzuPAphWerntC5kkrcnzRJZBA116uzJzyvk6fH, address:104.36.82.53:4126): protocol: storage node overloaded, request limit: 2000 Jun 21 20:10:05.834666043 (75h46m41.397989033s) [upload=60693]: Next returned nil segment Jun 21 20:10:05.834686295 (75h46m41.398009302s) [upload=60693]: Waiting for error group managing segments... Jun 21 20:10:05.877010488 (75h46m41.440333486s) [noise=true , address=45.91.116.6:4982 , piece_id=VP45R6KXNH5U5SGUQUMAPIBY7KFDPBIR3S3VRIX27EAT6VLFBZ7Q, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:05.886336964 (75h46m41.449659967s) [noise=true , address=104.36.82.9:6039 , piece_id=I4AB6N27MYYUHWAZLJJUIBFBLOQFHDIM3YLYWPVRPNUV7VMYS2SA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: ecclient: upload failed (node:12P11WVW9mVXw5ra2gZ3Rezon6wZi8GrN5CchQN3MiVU4ts3L6V, address:104.36.82.9:6039): protocol: storage node overloaded, request limit: 2000 Jun 21 20:10:08.064514542 (75h46m43.627837553s) [noise=true , address=77.90.31.14:4670 , piece_id=7FNANMUKPM3N6YWRAOAIM4WJZJUTNF7ZBI6ZUBNEPGNGMFWOE24A, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:08.099098391 (75h46m43.662421396s) [noise=true , address=79.127.205.245:4231 , piece_id=EJK46XUUVI5NWOAQJEW4QIJG3KRXZHB4YOQUJRCS2MIOSWKGTNLQ, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:08.489461749 (75h46m44.052784752s) [noise=true , address=104.36.82.42:6807 , piece_id=ZBQZPXUDJ4VBCOSVKWAJ6SK6UD7SLPON7Y4SSCK2DQ5STFG4XCNQ, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:08.650922093 (75h46m44.214245093s) [noise=true , address=77.90.31.19:4853 , piece_id=VFEON5T2KSWM2KSLTVRZTXYL42JWENZYUEUPG3LMODGZWBMVGGPQ, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:09.917746439 (75h46m45.481069441s) [noise=true , address=104.36.82.13:6196 , piece_id=BCHMQO5KKT5BLNWQGOHH3V43LGSE3BUQCI2SY4TTO5M5GQNOCTMQ, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:10.830722215 (75h46m46.394045216s) [noise=true , address=104.36.82.5:5869 , piece_id=HAEHXXZASV3DHT3QN7NTZOS6XZKHNUWBKCHWJJWSTJKXAKJKKBAA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:12.738783450 (75h46m48.302106452s) [noise=true , address=77.90.31.12:4603 , piece_id=BMUIC6KNGEDL37436GLSXW5XOO6VYOPNOSFGQCYVHSENVI2PJ7KQ, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:15.460927713 (75h46m51.024250723s) [noise=true , address=104.36.82.52:4111 , piece_id=HB7QAWN6CGNIIT4HD2GZ7JDQP5NXIJ5I2USBM66YOKTYENNOWG5A, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:18.393817968 (75h46m53.95714096s) [noise=true , address=104.36.82.55:4224 , piece_id=WRVNGSJV5BX4GJERLXNPAS7J7BZ5WFTM7R5H257YRCG5SP2HIYMA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:19.549174143 (75h46m55.112497151s) [noise=true , address=104.36.82.56:4253 , piece_id=5PMVGLZETMXCQ77K3I6PXACRIMBGR2CV5JGUSU5KUMYFNXVGA2GA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:22.511944593 (75h46m58.075267584s) [noise=true , address=104.36.82.37:6663 , piece_id=X7LWYIIHJFU2T2PHYR3R5TNT22UUPHEHQWZH7ZJMLS4AH4PJ6AIA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:48.027197558 (75h47m23.590520562s) [noise=true , address=104.36.82.14:6232 , piece_id=A2ZLUUZKRTBJRJ267QSK36C3LFZV7FPAYDDAXGIWDTNEGNV6MRRA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:11:05.567692582 (75h47m41.131015574s) [noise=true , address=104.36.82.49:4982 , piece_id=UNLV5UDNWULEIPE4QSQHHZQPK3BQKFRB5T3H6GP7ZQ5QGSNK5V6A, seg_pos={0 0}, upload=60693]: Done uploading piece. err: ecclient: upload failed (node:1NxoYJXCXmTvQncbkXt6yH7mhEBBrzVdzhgL5J6W5TD4N3pkaG, address:104.36.82.49:4982): protocol: expected piece hash; context deadline exceeded; EOF Jun 21 20:11:05.567759198 (75h47m41.131082195s) [noise=true , address=104.36.82.54:4171 , piece_id=MUMABKARK4KKUKWW6XDXCQVT6O7SLFHGHE7COWC7QIYGARAA2D5A, seg_pos={0 0}, upload=60693]: Done uploading piece. err: ecclient: upload failed (node:125SMhDsEZruNu6LkmQn2M7R6WzvXTdk86DRsKf522u8BNiANq3, address:104.36.82.54:4171): protocol: expected piece hash; context deadline exceeded; EOF Jun 21 20:11:05.567869019 (75h47m41.131192033s) [noise=true , address=104.36.82.10:6087 , piece_id=5FQG7BPZC373PFB54HYXSG2SVKOBBBSGZCLC4XDJEYQFOLQY74MA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: ecclient: upload failed (node:1S9k5eejJcxkiG5pXjjBZm18uP8k5jCGfWyeW2QQxmuRU8ZzRA, address:104.36.82.10:6087): protocol: write tcp 159.223.143.39:38550->104.36.82.10:6087: use of closed network connection; write tcp 159.223.143.39:38550->104.36.82.10:6087: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:38550->104.36.82.10:6087: use of closed network connection Jun 21 20:11:05.567932150 (75h47m41.131255161s) [seg_pos={0 0}, upload=60693]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jun 21 20:11:05.593626175 (75h47m41.156949181s) [upload=60693]: Done uploading segments... Jun 21 20:11:05.593657274 (75h47m41.156980265s) [upload=60693]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 19.419138579s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [..........X...........X.....................................] duration: 24.295967657s FAIL (2025-06-21 20:39:58 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== pload=60693]: Starting upload Jun 21 20:10:05.602866542 (75h46m41.166189557s) [upload=60693]: Done starting upload Jun 21 20:10:05.603116545 (75h46m41.166439547s) [upload=60693]: Uploading segments... Jun 21 20:10:05.603495389 (75h46m41.166818397s) [upload=60693]: Got next segment. Inline: false Jun 21 20:10:05.603566566 (75h46m41.166889558s) [upload=60693]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 21 20:10:05.603573280 (75h46m41.166896286s) [upload=60693]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 21 20:10:05.639130162 (75h46m41.202453164s) [seg_pos={0 0}, upload=60693]: Begin upload segment... Jun 21 20:10:05.639231070 (75h46m41.202554057s) [seg_pos={0 0}, upload=60693]: Done begin upload segment. Jun 21 20:10:05.639292724 (75h46m41.202615728s) [noise=true , address=104.36.82.56:4253 , piece_id=5PMVGLZETMXCQ77K3I6PXACRIMBGR2CV5JGUSU5KUMYFNXVGA2GA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.639338066 (75h46m41.202661071s) [noise=true , address=104.36.82.53:4126 , piece_id=2MKRQJEPGDJKDM7VWY5M4M6NORILAIEYYATJ7HVZEOBOMXOI2LBA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.639416264 (75h46m41.202739267s) [noise=true , address=104.36.82.5:5869 , piece_id=HAEHXXZASV3DHT3QN7NTZOS6XZKHNUWBKCHWJJWSTJKXAKJKKBAA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.639873098 (75h46m41.203196115s) [noise=true , address=104.36.82.52:4111 , piece_id=HB7QAWN6CGNIIT4HD2GZ7JDQP5NXIJ5I2USBM66YOKTYENNOWG5A, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.639887296 (75h46m41.203210304s) [noise=true , address=104.36.82.10:6087 , piece_id=5FQG7BPZC373PFB54HYXSG2SVKOBBBSGZCLC4XDJEYQFOLQY74MA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.640266623 (75h46m41.203589616s) [noise=true , address=104.36.82.13:6196 , piece_id=BCHMQO5KKT5BLNWQGOHH3V43LGSE3BUQCI2SY4TTO5M5GQNOCTMQ, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.640352274 (75h46m41.203675286s) [noise=true , address=104.36.82.37:6663 , piece_id=X7LWYIIHJFU2T2PHYR3R5TNT22UUPHEHQWZH7ZJMLS4AH4PJ6AIA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.640659767 (75h46m41.203982779s) [noise=true , address=104.36.82.42:6807 , piece_id=ZBQZPXUDJ4VBCOSVKWAJ6SK6UD7SLPON7Y4SSCK2DQ5STFG4XCNQ, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.640961701 (75h46m41.20428471s) [noise=true , address=45.91.116.6:4982 , piece_id=VP45R6KXNH5U5SGUQUMAPIBY7KFDPBIR3S3VRIX27EAT6VLFBZ7Q, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.641159578 (75h46m41.204482589s) [noise=true , address=104.36.82.9:6039 , piece_id=I4AB6N27MYYUHWAZLJJUIBFBLOQFHDIM3YLYWPVRPNUV7VMYS2SA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.641588126 (75h46m41.204911136s) [noise=true , address=77.90.31.19:4853 , piece_id=VFEON5T2KSWM2KSLTVRZTXYL42JWENZYUEUPG3LMODGZWBMVGGPQ, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.641863297 (75h46m41.205186305s) [noise=true , address=77.90.31.12:4603 , piece_id=BMUIC6KNGEDL37436GLSXW5XOO6VYOPNOSFGQCYVHSENVI2PJ7KQ, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.642124427 (75h46m41.205447437s) [noise=true , address=104.36.82.49:4982 , piece_id=UNLV5UDNWULEIPE4QSQHHZQPK3BQKFRB5T3H6GP7ZQ5QGSNK5V6A, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.642285917 (75h46m41.205608925s) [noise=true , address=104.36.82.54:4171 , piece_id=MUMABKARK4KKUKWW6XDXCQVT6O7SLFHGHE7COWC7QIYGARAA2D5A, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.642656750 (75h46m41.205979758s) [noise=true , address=104.36.82.14:6232 , piece_id=A2ZLUUZKRTBJRJ267QSK36C3LFZV7FPAYDDAXGIWDTNEGNV6MRRA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.643565583 (75h46m41.206888597s) [noise=true , address=79.127.205.245:4231 , piece_id=EJK46XUUVI5NWOAQJEW4QIJG3KRXZHB4YOQUJRCS2MIOSWKGTNLQ, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.644620291 (75h46m41.207943303s) [noise=true , address=77.90.31.14:4670 , piece_id=7FNANMUKPM3N6YWRAOAIM4WJZJUTNF7ZBI6ZUBNEPGNGMFWOE24A, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.648842639 (75h46m41.212165631s) [noise=true , address=104.36.82.55:4224 , piece_id=WRVNGSJV5BX4GJERLXNPAS7J7BZ5WFTM7R5H257YRCG5SP2HIYMA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.825828239 (75h46m41.389151241s) [noise=true , address=104.36.82.53:4126 , piece_id=2MKRQJEPGDJKDM7VWY5M4M6NORILAIEYYATJ7HVZEOBOMXOI2LBA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: ecclient: upload failed (node:12dMWFrq2NMpnMzuPAphWerntC5kkrcnzRJZBA116uzJzyvk6fH, address:104.36.82.53:4126): protocol: storage node overloaded, request limit: 2000 Jun 21 20:10:05.834666043 (75h46m41.397989033s) [upload=60693]: Next returned nil segment Jun 21 20:10:05.834686295 (75h46m41.398009302s) [upload=60693]: Waiting for error group managing segments... Jun 21 20:10:05.877010488 (75h46m41.440333486s) [noise=true , address=45.91.116.6:4982 , piece_id=VP45R6KXNH5U5SGUQUMAPIBY7KFDPBIR3S3VRIX27EAT6VLFBZ7Q, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:05.886336964 (75h46m41.449659967s) [noise=true , address=104.36.82.9:6039 , piece_id=I4AB6N27MYYUHWAZLJJUIBFBLOQFHDIM3YLYWPVRPNUV7VMYS2SA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: ecclient: upload failed (node:12P11WVW9mVXw5ra2gZ3Rezon6wZi8GrN5CchQN3MiVU4ts3L6V, address:104.36.82.9:6039): protocol: storage node overloaded, request limit: 2000 Jun 21 20:10:08.064514542 (75h46m43.627837553s) [noise=true , address=77.90.31.14:4670 , piece_id=7FNANMUKPM3N6YWRAOAIM4WJZJUTNF7ZBI6ZUBNEPGNGMFWOE24A, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:08.099098391 (75h46m43.662421396s) [noise=true , address=79.127.205.245:4231 , piece_id=EJK46XUUVI5NWOAQJEW4QIJG3KRXZHB4YOQUJRCS2MIOSWKGTNLQ, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:08.489461749 (75h46m44.052784752s) [noise=true , address=104.36.82.42:6807 , piece_id=ZBQZPXUDJ4VBCOSVKWAJ6SK6UD7SLPON7Y4SSCK2DQ5STFG4XCNQ, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:08.650922093 (75h46m44.214245093s) [noise=true , address=77.90.31.19:4853 , piece_id=VFEON5T2KSWM2KSLTVRZTXYL42JWENZYUEUPG3LMODGZWBMVGGPQ, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:09.917746439 (75h46m45.481069441s) [noise=true , address=104.36.82.13:6196 , piece_id=BCHMQO5KKT5BLNWQGOHH3V43LGSE3BUQCI2SY4TTO5M5GQNOCTMQ, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:10.830722215 (75h46m46.394045216s) [noise=true , address=104.36.82.5:5869 , piece_id=HAEHXXZASV3DHT3QN7NTZOS6XZKHNUWBKCHWJJWSTJKXAKJKKBAA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:12.738783450 (75h46m48.302106452s) [noise=true , address=77.90.31.12:4603 , piece_id=BMUIC6KNGEDL37436GLSXW5XOO6VYOPNOSFGQCYVHSENVI2PJ7KQ, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:15.460927713 (75h46m51.024250723s) [noise=true , address=104.36.82.52:4111 , piece_id=HB7QAWN6CGNIIT4HD2GZ7JDQP5NXIJ5I2USBM66YOKTYENNOWG5A, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:18.393817968 (75h46m53.95714096s) [noise=true , address=104.36.82.55:4224 , piece_id=WRVNGSJV5BX4GJERLXNPAS7J7BZ5WFTM7R5H257YRCG5SP2HIYMA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:19.549174143 (75h46m55.112497151s) [noise=true , address=104.36.82.56:4253 , piece_id=5PMVGLZETMXCQ77K3I6PXACRIMBGR2CV5JGUSU5KUMYFNXVGA2GA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:22.511944593 (75h46m58.075267584s) [noise=true , address=104.36.82.37:6663 , piece_id=X7LWYIIHJFU2T2PHYR3R5TNT22UUPHEHQWZH7ZJMLS4AH4PJ6AIA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:48.027197558 (75h47m23.590520562s) [noise=true , address=104.36.82.14:6232 , piece_id=A2ZLUUZKRTBJRJ267QSK36C3LFZV7FPAYDDAXGIWDTNEGNV6MRRA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:11:05.567692582 (75h47m41.131015574s) [noise=true , address=104.36.82.49:4982 , piece_id=UNLV5UDNWULEIPE4QSQHHZQPK3BQKFRB5T3H6GP7ZQ5QGSNK5V6A, seg_pos={0 0}, upload=60693]: Done uploading piece. err: ecclient: upload failed (node:1NxoYJXCXmTvQncbkXt6yH7mhEBBrzVdzhgL5J6W5TD4N3pkaG, address:104.36.82.49:4982): protocol: expected piece hash; context deadline exceeded; EOF Jun 21 20:11:05.567759198 (75h47m41.131082195s) [noise=true , address=104.36.82.54:4171 , piece_id=MUMABKARK4KKUKWW6XDXCQVT6O7SLFHGHE7COWC7QIYGARAA2D5A, seg_pos={0 0}, upload=60693]: Done uploading piece. err: ecclient: upload failed (node:125SMhDsEZruNu6LkmQn2M7R6WzvXTdk86DRsKf522u8BNiANq3, address:104.36.82.54:4171): protocol: expected piece hash; context deadline exceeded; EOF Jun 21 20:11:05.567869019 (75h47m41.131192033s) [noise=true , address=104.36.82.10:6087 , piece_id=5FQG7BPZC373PFB54HYXSG2SVKOBBBSGZCLC4XDJEYQFOLQY74MA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: ecclient: upload failed (node:1S9k5eejJcxkiG5pXjjBZm18uP8k5jCGfWyeW2QQxmuRU8ZzRA, address:104.36.82.10:6087): protocol: write tcp 159.223.143.39:38550->104.36.82.10:6087: use of closed network connection; write tcp 159.223.143.39:38550->104.36.82.10:6087: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:38550->104.36.82.10:6087: use of closed network connection Jun 21 20:11:05.567932150 (75h47m41.131255161s) [seg_pos={0 0}, upload=60693]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jun 21 20:11:05.593626175 (75h47m41.156949181s) [upload=60693]: Done uploading segments... Jun 21 20:11:05.593657274 (75h47m41.156980265s) [upload=60693]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 24.2959875s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [...........X...........X....................................] duration: 22.772717269s FAIL (2025-06-21 20:39:33 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== pload=60693]: Starting upload Jun 21 20:10:05.602866542 (75h46m41.166189557s) [upload=60693]: Done starting upload Jun 21 20:10:05.603116545 (75h46m41.166439547s) [upload=60693]: Uploading segments... Jun 21 20:10:05.603495389 (75h46m41.166818397s) [upload=60693]: Got next segment. Inline: false Jun 21 20:10:05.603566566 (75h46m41.166889558s) [upload=60693]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 21 20:10:05.603573280 (75h46m41.166896286s) [upload=60693]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 21 20:10:05.639130162 (75h46m41.202453164s) [seg_pos={0 0}, upload=60693]: Begin upload segment... Jun 21 20:10:05.639231070 (75h46m41.202554057s) [seg_pos={0 0}, upload=60693]: Done begin upload segment. Jun 21 20:10:05.639292724 (75h46m41.202615728s) [noise=true , address=104.36.82.56:4253 , piece_id=5PMVGLZETMXCQ77K3I6PXACRIMBGR2CV5JGUSU5KUMYFNXVGA2GA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.639338066 (75h46m41.202661071s) [noise=true , address=104.36.82.53:4126 , piece_id=2MKRQJEPGDJKDM7VWY5M4M6NORILAIEYYATJ7HVZEOBOMXOI2LBA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.639416264 (75h46m41.202739267s) [noise=true , address=104.36.82.5:5869 , piece_id=HAEHXXZASV3DHT3QN7NTZOS6XZKHNUWBKCHWJJWSTJKXAKJKKBAA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.639873098 (75h46m41.203196115s) [noise=true , address=104.36.82.52:4111 , piece_id=HB7QAWN6CGNIIT4HD2GZ7JDQP5NXIJ5I2USBM66YOKTYENNOWG5A, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.639887296 (75h46m41.203210304s) [noise=true , address=104.36.82.10:6087 , piece_id=5FQG7BPZC373PFB54HYXSG2SVKOBBBSGZCLC4XDJEYQFOLQY74MA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.640266623 (75h46m41.203589616s) [noise=true , address=104.36.82.13:6196 , piece_id=BCHMQO5KKT5BLNWQGOHH3V43LGSE3BUQCI2SY4TTO5M5GQNOCTMQ, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.640352274 (75h46m41.203675286s) [noise=true , address=104.36.82.37:6663 , piece_id=X7LWYIIHJFU2T2PHYR3R5TNT22UUPHEHQWZH7ZJMLS4AH4PJ6AIA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.640659767 (75h46m41.203982779s) [noise=true , address=104.36.82.42:6807 , piece_id=ZBQZPXUDJ4VBCOSVKWAJ6SK6UD7SLPON7Y4SSCK2DQ5STFG4XCNQ, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.640961701 (75h46m41.20428471s) [noise=true , address=45.91.116.6:4982 , piece_id=VP45R6KXNH5U5SGUQUMAPIBY7KFDPBIR3S3VRIX27EAT6VLFBZ7Q, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.641159578 (75h46m41.204482589s) [noise=true , address=104.36.82.9:6039 , piece_id=I4AB6N27MYYUHWAZLJJUIBFBLOQFHDIM3YLYWPVRPNUV7VMYS2SA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.641588126 (75h46m41.204911136s) [noise=true , address=77.90.31.19:4853 , piece_id=VFEON5T2KSWM2KSLTVRZTXYL42JWENZYUEUPG3LMODGZWBMVGGPQ, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.641863297 (75h46m41.205186305s) [noise=true , address=77.90.31.12:4603 , piece_id=BMUIC6KNGEDL37436GLSXW5XOO6VYOPNOSFGQCYVHSENVI2PJ7KQ, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.642124427 (75h46m41.205447437s) [noise=true , address=104.36.82.49:4982 , piece_id=UNLV5UDNWULEIPE4QSQHHZQPK3BQKFRB5T3H6GP7ZQ5QGSNK5V6A, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.642285917 (75h46m41.205608925s) [noise=true , address=104.36.82.54:4171 , piece_id=MUMABKARK4KKUKWW6XDXCQVT6O7SLFHGHE7COWC7QIYGARAA2D5A, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.642656750 (75h46m41.205979758s) [noise=true , address=104.36.82.14:6232 , piece_id=A2ZLUUZKRTBJRJ267QSK36C3LFZV7FPAYDDAXGIWDTNEGNV6MRRA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.643565583 (75h46m41.206888597s) [noise=true , address=79.127.205.245:4231 , piece_id=EJK46XUUVI5NWOAQJEW4QIJG3KRXZHB4YOQUJRCS2MIOSWKGTNLQ, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.644620291 (75h46m41.207943303s) [noise=true , address=77.90.31.14:4670 , piece_id=7FNANMUKPM3N6YWRAOAIM4WJZJUTNF7ZBI6ZUBNEPGNGMFWOE24A, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.648842639 (75h46m41.212165631s) [noise=true , address=104.36.82.55:4224 , piece_id=WRVNGSJV5BX4GJERLXNPAS7J7BZ5WFTM7R5H257YRCG5SP2HIYMA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.825828239 (75h46m41.389151241s) [noise=true , address=104.36.82.53:4126 , piece_id=2MKRQJEPGDJKDM7VWY5M4M6NORILAIEYYATJ7HVZEOBOMXOI2LBA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: ecclient: upload failed (node:12dMWFrq2NMpnMzuPAphWerntC5kkrcnzRJZBA116uzJzyvk6fH, address:104.36.82.53:4126): protocol: storage node overloaded, request limit: 2000 Jun 21 20:10:05.834666043 (75h46m41.397989033s) [upload=60693]: Next returned nil segment Jun 21 20:10:05.834686295 (75h46m41.398009302s) [upload=60693]: Waiting for error group managing segments... Jun 21 20:10:05.877010488 (75h46m41.440333486s) [noise=true , address=45.91.116.6:4982 , piece_id=VP45R6KXNH5U5SGUQUMAPIBY7KFDPBIR3S3VRIX27EAT6VLFBZ7Q, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:05.886336964 (75h46m41.449659967s) [noise=true , address=104.36.82.9:6039 , piece_id=I4AB6N27MYYUHWAZLJJUIBFBLOQFHDIM3YLYWPVRPNUV7VMYS2SA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: ecclient: upload failed (node:12P11WVW9mVXw5ra2gZ3Rezon6wZi8GrN5CchQN3MiVU4ts3L6V, address:104.36.82.9:6039): protocol: storage node overloaded, request limit: 2000 Jun 21 20:10:08.064514542 (75h46m43.627837553s) [noise=true , address=77.90.31.14:4670 , piece_id=7FNANMUKPM3N6YWRAOAIM4WJZJUTNF7ZBI6ZUBNEPGNGMFWOE24A, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:08.099098391 (75h46m43.662421396s) [noise=true , address=79.127.205.245:4231 , piece_id=EJK46XUUVI5NWOAQJEW4QIJG3KRXZHB4YOQUJRCS2MIOSWKGTNLQ, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:08.489461749 (75h46m44.052784752s) [noise=true , address=104.36.82.42:6807 , piece_id=ZBQZPXUDJ4VBCOSVKWAJ6SK6UD7SLPON7Y4SSCK2DQ5STFG4XCNQ, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:08.650922093 (75h46m44.214245093s) [noise=true , address=77.90.31.19:4853 , piece_id=VFEON5T2KSWM2KSLTVRZTXYL42JWENZYUEUPG3LMODGZWBMVGGPQ, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:09.917746439 (75h46m45.481069441s) [noise=true , address=104.36.82.13:6196 , piece_id=BCHMQO5KKT5BLNWQGOHH3V43LGSE3BUQCI2SY4TTO5M5GQNOCTMQ, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:10.830722215 (75h46m46.394045216s) [noise=true , address=104.36.82.5:5869 , piece_id=HAEHXXZASV3DHT3QN7NTZOS6XZKHNUWBKCHWJJWSTJKXAKJKKBAA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:12.738783450 (75h46m48.302106452s) [noise=true , address=77.90.31.12:4603 , piece_id=BMUIC6KNGEDL37436GLSXW5XOO6VYOPNOSFGQCYVHSENVI2PJ7KQ, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:15.460927713 (75h46m51.024250723s) [noise=true , address=104.36.82.52:4111 , piece_id=HB7QAWN6CGNIIT4HD2GZ7JDQP5NXIJ5I2USBM66YOKTYENNOWG5A, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:18.393817968 (75h46m53.95714096s) [noise=true , address=104.36.82.55:4224 , piece_id=WRVNGSJV5BX4GJERLXNPAS7J7BZ5WFTM7R5H257YRCG5SP2HIYMA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:19.549174143 (75h46m55.112497151s) [noise=true , address=104.36.82.56:4253 , piece_id=5PMVGLZETMXCQ77K3I6PXACRIMBGR2CV5JGUSU5KUMYFNXVGA2GA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:22.511944593 (75h46m58.075267584s) [noise=true , address=104.36.82.37:6663 , piece_id=X7LWYIIHJFU2T2PHYR3R5TNT22UUPHEHQWZH7ZJMLS4AH4PJ6AIA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:48.027197558 (75h47m23.590520562s) [noise=true , address=104.36.82.14:6232 , piece_id=A2ZLUUZKRTBJRJ267QSK36C3LFZV7FPAYDDAXGIWDTNEGNV6MRRA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:11:05.567692582 (75h47m41.131015574s) [noise=true , address=104.36.82.49:4982 , piece_id=UNLV5UDNWULEIPE4QSQHHZQPK3BQKFRB5T3H6GP7ZQ5QGSNK5V6A, seg_pos={0 0}, upload=60693]: Done uploading piece. err: ecclient: upload failed (node:1NxoYJXCXmTvQncbkXt6yH7mhEBBrzVdzhgL5J6W5TD4N3pkaG, address:104.36.82.49:4982): protocol: expected piece hash; context deadline exceeded; EOF Jun 21 20:11:05.567759198 (75h47m41.131082195s) [noise=true , address=104.36.82.54:4171 , piece_id=MUMABKARK4KKUKWW6XDXCQVT6O7SLFHGHE7COWC7QIYGARAA2D5A, seg_pos={0 0}, upload=60693]: Done uploading piece. err: ecclient: upload failed (node:125SMhDsEZruNu6LkmQn2M7R6WzvXTdk86DRsKf522u8BNiANq3, address:104.36.82.54:4171): protocol: expected piece hash; context deadline exceeded; EOF Jun 21 20:11:05.567869019 (75h47m41.131192033s) [noise=true , address=104.36.82.10:6087 , piece_id=5FQG7BPZC373PFB54HYXSG2SVKOBBBSGZCLC4XDJEYQFOLQY74MA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: ecclient: upload failed (node:1S9k5eejJcxkiG5pXjjBZm18uP8k5jCGfWyeW2QQxmuRU8ZzRA, address:104.36.82.10:6087): protocol: write tcp 159.223.143.39:38550->104.36.82.10:6087: use of closed network connection; write tcp 159.223.143.39:38550->104.36.82.10:6087: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:38550->104.36.82.10:6087: use of closed network connection Jun 21 20:11:05.567932150 (75h47m41.131255161s) [seg_pos={0 0}, upload=60693]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jun 21 20:11:05.593626175 (75h47m41.156949181s) [upload=60693]: Done uploading segments... Jun 21 20:11:05.593657274 (75h47m41.156980265s) [upload=60693]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 22.772747741s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [............X...........X...................................] duration: 13.435008791s FAIL (2025-06-21 20:38:34 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== pload=60693]: Starting upload Jun 21 20:10:05.602866542 (75h46m41.166189557s) [upload=60693]: Done starting upload Jun 21 20:10:05.603116545 (75h46m41.166439547s) [upload=60693]: Uploading segments... Jun 21 20:10:05.603495389 (75h46m41.166818397s) [upload=60693]: Got next segment. Inline: false Jun 21 20:10:05.603566566 (75h46m41.166889558s) [upload=60693]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 21 20:10:05.603573280 (75h46m41.166896286s) [upload=60693]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 21 20:10:05.639130162 (75h46m41.202453164s) [seg_pos={0 0}, upload=60693]: Begin upload segment... Jun 21 20:10:05.639231070 (75h46m41.202554057s) [seg_pos={0 0}, upload=60693]: Done begin upload segment. Jun 21 20:10:05.639292724 (75h46m41.202615728s) [noise=true , address=104.36.82.56:4253 , piece_id=5PMVGLZETMXCQ77K3I6PXACRIMBGR2CV5JGUSU5KUMYFNXVGA2GA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.639338066 (75h46m41.202661071s) [noise=true , address=104.36.82.53:4126 , piece_id=2MKRQJEPGDJKDM7VWY5M4M6NORILAIEYYATJ7HVZEOBOMXOI2LBA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.639416264 (75h46m41.202739267s) [noise=true , address=104.36.82.5:5869 , piece_id=HAEHXXZASV3DHT3QN7NTZOS6XZKHNUWBKCHWJJWSTJKXAKJKKBAA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.639873098 (75h46m41.203196115s) [noise=true , address=104.36.82.52:4111 , piece_id=HB7QAWN6CGNIIT4HD2GZ7JDQP5NXIJ5I2USBM66YOKTYENNOWG5A, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.639887296 (75h46m41.203210304s) [noise=true , address=104.36.82.10:6087 , piece_id=5FQG7BPZC373PFB54HYXSG2SVKOBBBSGZCLC4XDJEYQFOLQY74MA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.640266623 (75h46m41.203589616s) [noise=true , address=104.36.82.13:6196 , piece_id=BCHMQO5KKT5BLNWQGOHH3V43LGSE3BUQCI2SY4TTO5M5GQNOCTMQ, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.640352274 (75h46m41.203675286s) [noise=true , address=104.36.82.37:6663 , piece_id=X7LWYIIHJFU2T2PHYR3R5TNT22UUPHEHQWZH7ZJMLS4AH4PJ6AIA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.640659767 (75h46m41.203982779s) [noise=true , address=104.36.82.42:6807 , piece_id=ZBQZPXUDJ4VBCOSVKWAJ6SK6UD7SLPON7Y4SSCK2DQ5STFG4XCNQ, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.640961701 (75h46m41.20428471s) [noise=true , address=45.91.116.6:4982 , piece_id=VP45R6KXNH5U5SGUQUMAPIBY7KFDPBIR3S3VRIX27EAT6VLFBZ7Q, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.641159578 (75h46m41.204482589s) [noise=true , address=104.36.82.9:6039 , piece_id=I4AB6N27MYYUHWAZLJJUIBFBLOQFHDIM3YLYWPVRPNUV7VMYS2SA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.641588126 (75h46m41.204911136s) [noise=true , address=77.90.31.19:4853 , piece_id=VFEON5T2KSWM2KSLTVRZTXYL42JWENZYUEUPG3LMODGZWBMVGGPQ, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.641863297 (75h46m41.205186305s) [noise=true , address=77.90.31.12:4603 , piece_id=BMUIC6KNGEDL37436GLSXW5XOO6VYOPNOSFGQCYVHSENVI2PJ7KQ, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.642124427 (75h46m41.205447437s) [noise=true , address=104.36.82.49:4982 , piece_id=UNLV5UDNWULEIPE4QSQHHZQPK3BQKFRB5T3H6GP7ZQ5QGSNK5V6A, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.642285917 (75h46m41.205608925s) [noise=true , address=104.36.82.54:4171 , piece_id=MUMABKARK4KKUKWW6XDXCQVT6O7SLFHGHE7COWC7QIYGARAA2D5A, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.642656750 (75h46m41.205979758s) [noise=true , address=104.36.82.14:6232 , piece_id=A2ZLUUZKRTBJRJ267QSK36C3LFZV7FPAYDDAXGIWDTNEGNV6MRRA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.643565583 (75h46m41.206888597s) [noise=true , address=79.127.205.245:4231 , piece_id=EJK46XUUVI5NWOAQJEW4QIJG3KRXZHB4YOQUJRCS2MIOSWKGTNLQ, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.644620291 (75h46m41.207943303s) [noise=true , address=77.90.31.14:4670 , piece_id=7FNANMUKPM3N6YWRAOAIM4WJZJUTNF7ZBI6ZUBNEPGNGMFWOE24A, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.648842639 (75h46m41.212165631s) [noise=true , address=104.36.82.55:4224 , piece_id=WRVNGSJV5BX4GJERLXNPAS7J7BZ5WFTM7R5H257YRCG5SP2HIYMA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.825828239 (75h46m41.389151241s) [noise=true , address=104.36.82.53:4126 , piece_id=2MKRQJEPGDJKDM7VWY5M4M6NORILAIEYYATJ7HVZEOBOMXOI2LBA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: ecclient: upload failed (node:12dMWFrq2NMpnMzuPAphWerntC5kkrcnzRJZBA116uzJzyvk6fH, address:104.36.82.53:4126): protocol: storage node overloaded, request limit: 2000 Jun 21 20:10:05.834666043 (75h46m41.397989033s) [upload=60693]: Next returned nil segment Jun 21 20:10:05.834686295 (75h46m41.398009302s) [upload=60693]: Waiting for error group managing segments... Jun 21 20:10:05.877010488 (75h46m41.440333486s) [noise=true , address=45.91.116.6:4982 , piece_id=VP45R6KXNH5U5SGUQUMAPIBY7KFDPBIR3S3VRIX27EAT6VLFBZ7Q, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:05.886336964 (75h46m41.449659967s) [noise=true , address=104.36.82.9:6039 , piece_id=I4AB6N27MYYUHWAZLJJUIBFBLOQFHDIM3YLYWPVRPNUV7VMYS2SA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: ecclient: upload failed (node:12P11WVW9mVXw5ra2gZ3Rezon6wZi8GrN5CchQN3MiVU4ts3L6V, address:104.36.82.9:6039): protocol: storage node overloaded, request limit: 2000 Jun 21 20:10:08.064514542 (75h46m43.627837553s) [noise=true , address=77.90.31.14:4670 , piece_id=7FNANMUKPM3N6YWRAOAIM4WJZJUTNF7ZBI6ZUBNEPGNGMFWOE24A, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:08.099098391 (75h46m43.662421396s) [noise=true , address=79.127.205.245:4231 , piece_id=EJK46XUUVI5NWOAQJEW4QIJG3KRXZHB4YOQUJRCS2MIOSWKGTNLQ, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:08.489461749 (75h46m44.052784752s) [noise=true , address=104.36.82.42:6807 , piece_id=ZBQZPXUDJ4VBCOSVKWAJ6SK6UD7SLPON7Y4SSCK2DQ5STFG4XCNQ, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:08.650922093 (75h46m44.214245093s) [noise=true , address=77.90.31.19:4853 , piece_id=VFEON5T2KSWM2KSLTVRZTXYL42JWENZYUEUPG3LMODGZWBMVGGPQ, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:09.917746439 (75h46m45.481069441s) [noise=true , address=104.36.82.13:6196 , piece_id=BCHMQO5KKT5BLNWQGOHH3V43LGSE3BUQCI2SY4TTO5M5GQNOCTMQ, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:10.830722215 (75h46m46.394045216s) [noise=true , address=104.36.82.5:5869 , piece_id=HAEHXXZASV3DHT3QN7NTZOS6XZKHNUWBKCHWJJWSTJKXAKJKKBAA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:12.738783450 (75h46m48.302106452s) [noise=true , address=77.90.31.12:4603 , piece_id=BMUIC6KNGEDL37436GLSXW5XOO6VYOPNOSFGQCYVHSENVI2PJ7KQ, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:15.460927713 (75h46m51.024250723s) [noise=true , address=104.36.82.52:4111 , piece_id=HB7QAWN6CGNIIT4HD2GZ7JDQP5NXIJ5I2USBM66YOKTYENNOWG5A, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:18.393817968 (75h46m53.95714096s) [noise=true , address=104.36.82.55:4224 , piece_id=WRVNGSJV5BX4GJERLXNPAS7J7BZ5WFTM7R5H257YRCG5SP2HIYMA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:19.549174143 (75h46m55.112497151s) [noise=true , address=104.36.82.56:4253 , piece_id=5PMVGLZETMXCQ77K3I6PXACRIMBGR2CV5JGUSU5KUMYFNXVGA2GA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:22.511944593 (75h46m58.075267584s) [noise=true , address=104.36.82.37:6663 , piece_id=X7LWYIIHJFU2T2PHYR3R5TNT22UUPHEHQWZH7ZJMLS4AH4PJ6AIA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:48.027197558 (75h47m23.590520562s) [noise=true , address=104.36.82.14:6232 , piece_id=A2ZLUUZKRTBJRJ267QSK36C3LFZV7FPAYDDAXGIWDTNEGNV6MRRA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:11:05.567692582 (75h47m41.131015574s) [noise=true , address=104.36.82.49:4982 , piece_id=UNLV5UDNWULEIPE4QSQHHZQPK3BQKFRB5T3H6GP7ZQ5QGSNK5V6A, seg_pos={0 0}, upload=60693]: Done uploading piece. err: ecclient: upload failed (node:1NxoYJXCXmTvQncbkXt6yH7mhEBBrzVdzhgL5J6W5TD4N3pkaG, address:104.36.82.49:4982): protocol: expected piece hash; context deadline exceeded; EOF Jun 21 20:11:05.567759198 (75h47m41.131082195s) [noise=true , address=104.36.82.54:4171 , piece_id=MUMABKARK4KKUKWW6XDXCQVT6O7SLFHGHE7COWC7QIYGARAA2D5A, seg_pos={0 0}, upload=60693]: Done uploading piece. err: ecclient: upload failed (node:125SMhDsEZruNu6LkmQn2M7R6WzvXTdk86DRsKf522u8BNiANq3, address:104.36.82.54:4171): protocol: expected piece hash; context deadline exceeded; EOF Jun 21 20:11:05.567869019 (75h47m41.131192033s) [noise=true , address=104.36.82.10:6087 , piece_id=5FQG7BPZC373PFB54HYXSG2SVKOBBBSGZCLC4XDJEYQFOLQY74MA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: ecclient: upload failed (node:1S9k5eejJcxkiG5pXjjBZm18uP8k5jCGfWyeW2QQxmuRU8ZzRA, address:104.36.82.10:6087): protocol: write tcp 159.223.143.39:38550->104.36.82.10:6087: use of closed network connection; write tcp 159.223.143.39:38550->104.36.82.10:6087: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:38550->104.36.82.10:6087: use of closed network connection Jun 21 20:11:05.567932150 (75h47m41.131255161s) [seg_pos={0 0}, upload=60693]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jun 21 20:11:05.593626175 (75h47m41.156949181s) [upload=60693]: Done uploading segments... Jun 21 20:11:05.593657274 (75h47m41.156980265s) [upload=60693]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 13.435036226s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [.............X...........X..................................] duration: 23.463841693s FAIL (2025-06-21 20:38:09 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== pload=60693]: Starting upload Jun 21 20:10:05.602866542 (75h46m41.166189557s) [upload=60693]: Done starting upload Jun 21 20:10:05.603116545 (75h46m41.166439547s) [upload=60693]: Uploading segments... Jun 21 20:10:05.603495389 (75h46m41.166818397s) [upload=60693]: Got next segment. Inline: false Jun 21 20:10:05.603566566 (75h46m41.166889558s) [upload=60693]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 21 20:10:05.603573280 (75h46m41.166896286s) [upload=60693]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 21 20:10:05.639130162 (75h46m41.202453164s) [seg_pos={0 0}, upload=60693]: Begin upload segment... Jun 21 20:10:05.639231070 (75h46m41.202554057s) [seg_pos={0 0}, upload=60693]: Done begin upload segment. Jun 21 20:10:05.639292724 (75h46m41.202615728s) [noise=true , address=104.36.82.56:4253 , piece_id=5PMVGLZETMXCQ77K3I6PXACRIMBGR2CV5JGUSU5KUMYFNXVGA2GA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.639338066 (75h46m41.202661071s) [noise=true , address=104.36.82.53:4126 , piece_id=2MKRQJEPGDJKDM7VWY5M4M6NORILAIEYYATJ7HVZEOBOMXOI2LBA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.639416264 (75h46m41.202739267s) [noise=true , address=104.36.82.5:5869 , piece_id=HAEHXXZASV3DHT3QN7NTZOS6XZKHNUWBKCHWJJWSTJKXAKJKKBAA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.639873098 (75h46m41.203196115s) [noise=true , address=104.36.82.52:4111 , piece_id=HB7QAWN6CGNIIT4HD2GZ7JDQP5NXIJ5I2USBM66YOKTYENNOWG5A, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.639887296 (75h46m41.203210304s) [noise=true , address=104.36.82.10:6087 , piece_id=5FQG7BPZC373PFB54HYXSG2SVKOBBBSGZCLC4XDJEYQFOLQY74MA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.640266623 (75h46m41.203589616s) [noise=true , address=104.36.82.13:6196 , piece_id=BCHMQO5KKT5BLNWQGOHH3V43LGSE3BUQCI2SY4TTO5M5GQNOCTMQ, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.640352274 (75h46m41.203675286s) [noise=true , address=104.36.82.37:6663 , piece_id=X7LWYIIHJFU2T2PHYR3R5TNT22UUPHEHQWZH7ZJMLS4AH4PJ6AIA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.640659767 (75h46m41.203982779s) [noise=true , address=104.36.82.42:6807 , piece_id=ZBQZPXUDJ4VBCOSVKWAJ6SK6UD7SLPON7Y4SSCK2DQ5STFG4XCNQ, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.640961701 (75h46m41.20428471s) [noise=true , address=45.91.116.6:4982 , piece_id=VP45R6KXNH5U5SGUQUMAPIBY7KFDPBIR3S3VRIX27EAT6VLFBZ7Q, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.641159578 (75h46m41.204482589s) [noise=true , address=104.36.82.9:6039 , piece_id=I4AB6N27MYYUHWAZLJJUIBFBLOQFHDIM3YLYWPVRPNUV7VMYS2SA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.641588126 (75h46m41.204911136s) [noise=true , address=77.90.31.19:4853 , piece_id=VFEON5T2KSWM2KSLTVRZTXYL42JWENZYUEUPG3LMODGZWBMVGGPQ, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.641863297 (75h46m41.205186305s) [noise=true , address=77.90.31.12:4603 , piece_id=BMUIC6KNGEDL37436GLSXW5XOO6VYOPNOSFGQCYVHSENVI2PJ7KQ, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.642124427 (75h46m41.205447437s) [noise=true , address=104.36.82.49:4982 , piece_id=UNLV5UDNWULEIPE4QSQHHZQPK3BQKFRB5T3H6GP7ZQ5QGSNK5V6A, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.642285917 (75h46m41.205608925s) [noise=true , address=104.36.82.54:4171 , piece_id=MUMABKARK4KKUKWW6XDXCQVT6O7SLFHGHE7COWC7QIYGARAA2D5A, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.642656750 (75h46m41.205979758s) [noise=true , address=104.36.82.14:6232 , piece_id=A2ZLUUZKRTBJRJ267QSK36C3LFZV7FPAYDDAXGIWDTNEGNV6MRRA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.643565583 (75h46m41.206888597s) [noise=true , address=79.127.205.245:4231 , piece_id=EJK46XUUVI5NWOAQJEW4QIJG3KRXZHB4YOQUJRCS2MIOSWKGTNLQ, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.644620291 (75h46m41.207943303s) [noise=true , address=77.90.31.14:4670 , piece_id=7FNANMUKPM3N6YWRAOAIM4WJZJUTNF7ZBI6ZUBNEPGNGMFWOE24A, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.648842639 (75h46m41.212165631s) [noise=true , address=104.36.82.55:4224 , piece_id=WRVNGSJV5BX4GJERLXNPAS7J7BZ5WFTM7R5H257YRCG5SP2HIYMA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.825828239 (75h46m41.389151241s) [noise=true , address=104.36.82.53:4126 , piece_id=2MKRQJEPGDJKDM7VWY5M4M6NORILAIEYYATJ7HVZEOBOMXOI2LBA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: ecclient: upload failed (node:12dMWFrq2NMpnMzuPAphWerntC5kkrcnzRJZBA116uzJzyvk6fH, address:104.36.82.53:4126): protocol: storage node overloaded, request limit: 2000 Jun 21 20:10:05.834666043 (75h46m41.397989033s) [upload=60693]: Next returned nil segment Jun 21 20:10:05.834686295 (75h46m41.398009302s) [upload=60693]: Waiting for error group managing segments... Jun 21 20:10:05.877010488 (75h46m41.440333486s) [noise=true , address=45.91.116.6:4982 , piece_id=VP45R6KXNH5U5SGUQUMAPIBY7KFDPBIR3S3VRIX27EAT6VLFBZ7Q, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:05.886336964 (75h46m41.449659967s) [noise=true , address=104.36.82.9:6039 , piece_id=I4AB6N27MYYUHWAZLJJUIBFBLOQFHDIM3YLYWPVRPNUV7VMYS2SA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: ecclient: upload failed (node:12P11WVW9mVXw5ra2gZ3Rezon6wZi8GrN5CchQN3MiVU4ts3L6V, address:104.36.82.9:6039): protocol: storage node overloaded, request limit: 2000 Jun 21 20:10:08.064514542 (75h46m43.627837553s) [noise=true , address=77.90.31.14:4670 , piece_id=7FNANMUKPM3N6YWRAOAIM4WJZJUTNF7ZBI6ZUBNEPGNGMFWOE24A, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:08.099098391 (75h46m43.662421396s) [noise=true , address=79.127.205.245:4231 , piece_id=EJK46XUUVI5NWOAQJEW4QIJG3KRXZHB4YOQUJRCS2MIOSWKGTNLQ, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:08.489461749 (75h46m44.052784752s) [noise=true , address=104.36.82.42:6807 , piece_id=ZBQZPXUDJ4VBCOSVKWAJ6SK6UD7SLPON7Y4SSCK2DQ5STFG4XCNQ, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:08.650922093 (75h46m44.214245093s) [noise=true , address=77.90.31.19:4853 , piece_id=VFEON5T2KSWM2KSLTVRZTXYL42JWENZYUEUPG3LMODGZWBMVGGPQ, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:09.917746439 (75h46m45.481069441s) [noise=true , address=104.36.82.13:6196 , piece_id=BCHMQO5KKT5BLNWQGOHH3V43LGSE3BUQCI2SY4TTO5M5GQNOCTMQ, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:10.830722215 (75h46m46.394045216s) [noise=true , address=104.36.82.5:5869 , piece_id=HAEHXXZASV3DHT3QN7NTZOS6XZKHNUWBKCHWJJWSTJKXAKJKKBAA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:12.738783450 (75h46m48.302106452s) [noise=true , address=77.90.31.12:4603 , piece_id=BMUIC6KNGEDL37436GLSXW5XOO6VYOPNOSFGQCYVHSENVI2PJ7KQ, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:15.460927713 (75h46m51.024250723s) [noise=true , address=104.36.82.52:4111 , piece_id=HB7QAWN6CGNIIT4HD2GZ7JDQP5NXIJ5I2USBM66YOKTYENNOWG5A, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:18.393817968 (75h46m53.95714096s) [noise=true , address=104.36.82.55:4224 , piece_id=WRVNGSJV5BX4GJERLXNPAS7J7BZ5WFTM7R5H257YRCG5SP2HIYMA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:19.549174143 (75h46m55.112497151s) [noise=true , address=104.36.82.56:4253 , piece_id=5PMVGLZETMXCQ77K3I6PXACRIMBGR2CV5JGUSU5KUMYFNXVGA2GA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:22.511944593 (75h46m58.075267584s) [noise=true , address=104.36.82.37:6663 , piece_id=X7LWYIIHJFU2T2PHYR3R5TNT22UUPHEHQWZH7ZJMLS4AH4PJ6AIA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:48.027197558 (75h47m23.590520562s) [noise=true , address=104.36.82.14:6232 , piece_id=A2ZLUUZKRTBJRJ267QSK36C3LFZV7FPAYDDAXGIWDTNEGNV6MRRA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:11:05.567692582 (75h47m41.131015574s) [noise=true , address=104.36.82.49:4982 , piece_id=UNLV5UDNWULEIPE4QSQHHZQPK3BQKFRB5T3H6GP7ZQ5QGSNK5V6A, seg_pos={0 0}, upload=60693]: Done uploading piece. err: ecclient: upload failed (node:1NxoYJXCXmTvQncbkXt6yH7mhEBBrzVdzhgL5J6W5TD4N3pkaG, address:104.36.82.49:4982): protocol: expected piece hash; context deadline exceeded; EOF Jun 21 20:11:05.567759198 (75h47m41.131082195s) [noise=true , address=104.36.82.54:4171 , piece_id=MUMABKARK4KKUKWW6XDXCQVT6O7SLFHGHE7COWC7QIYGARAA2D5A, seg_pos={0 0}, upload=60693]: Done uploading piece. err: ecclient: upload failed (node:125SMhDsEZruNu6LkmQn2M7R6WzvXTdk86DRsKf522u8BNiANq3, address:104.36.82.54:4171): protocol: expected piece hash; context deadline exceeded; EOF Jun 21 20:11:05.567869019 (75h47m41.131192033s) [noise=true , address=104.36.82.10:6087 , piece_id=5FQG7BPZC373PFB54HYXSG2SVKOBBBSGZCLC4XDJEYQFOLQY74MA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: ecclient: upload failed (node:1S9k5eejJcxkiG5pXjjBZm18uP8k5jCGfWyeW2QQxmuRU8ZzRA, address:104.36.82.10:6087): protocol: write tcp 159.223.143.39:38550->104.36.82.10:6087: use of closed network connection; write tcp 159.223.143.39:38550->104.36.82.10:6087: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:38550->104.36.82.10:6087: use of closed network connection Jun 21 20:11:05.567932150 (75h47m41.131255161s) [seg_pos={0 0}, upload=60693]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jun 21 20:11:05.593626175 (75h47m41.156949181s) [upload=60693]: Done uploading segments... Jun 21 20:11:05.593657274 (75h47m41.156980265s) [upload=60693]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 23.463887964s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [..............X...........X.................................] duration: 16.873184024s FAIL (2025-06-21 20:37:46 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== pload=60693]: Starting upload Jun 21 20:10:05.602866542 (75h46m41.166189557s) [upload=60693]: Done starting upload Jun 21 20:10:05.603116545 (75h46m41.166439547s) [upload=60693]: Uploading segments... Jun 21 20:10:05.603495389 (75h46m41.166818397s) [upload=60693]: Got next segment. Inline: false Jun 21 20:10:05.603566566 (75h46m41.166889558s) [upload=60693]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 21 20:10:05.603573280 (75h46m41.166896286s) [upload=60693]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 21 20:10:05.639130162 (75h46m41.202453164s) [seg_pos={0 0}, upload=60693]: Begin upload segment... Jun 21 20:10:05.639231070 (75h46m41.202554057s) [seg_pos={0 0}, upload=60693]: Done begin upload segment. Jun 21 20:10:05.639292724 (75h46m41.202615728s) [noise=true , address=104.36.82.56:4253 , piece_id=5PMVGLZETMXCQ77K3I6PXACRIMBGR2CV5JGUSU5KUMYFNXVGA2GA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.639338066 (75h46m41.202661071s) [noise=true , address=104.36.82.53:4126 , piece_id=2MKRQJEPGDJKDM7VWY5M4M6NORILAIEYYATJ7HVZEOBOMXOI2LBA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.639416264 (75h46m41.202739267s) [noise=true , address=104.36.82.5:5869 , piece_id=HAEHXXZASV3DHT3QN7NTZOS6XZKHNUWBKCHWJJWSTJKXAKJKKBAA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.639873098 (75h46m41.203196115s) [noise=true , address=104.36.82.52:4111 , piece_id=HB7QAWN6CGNIIT4HD2GZ7JDQP5NXIJ5I2USBM66YOKTYENNOWG5A, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.639887296 (75h46m41.203210304s) [noise=true , address=104.36.82.10:6087 , piece_id=5FQG7BPZC373PFB54HYXSG2SVKOBBBSGZCLC4XDJEYQFOLQY74MA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.640266623 (75h46m41.203589616s) [noise=true , address=104.36.82.13:6196 , piece_id=BCHMQO5KKT5BLNWQGOHH3V43LGSE3BUQCI2SY4TTO5M5GQNOCTMQ, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.640352274 (75h46m41.203675286s) [noise=true , address=104.36.82.37:6663 , piece_id=X7LWYIIHJFU2T2PHYR3R5TNT22UUPHEHQWZH7ZJMLS4AH4PJ6AIA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.640659767 (75h46m41.203982779s) [noise=true , address=104.36.82.42:6807 , piece_id=ZBQZPXUDJ4VBCOSVKWAJ6SK6UD7SLPON7Y4SSCK2DQ5STFG4XCNQ, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.640961701 (75h46m41.20428471s) [noise=true , address=45.91.116.6:4982 , piece_id=VP45R6KXNH5U5SGUQUMAPIBY7KFDPBIR3S3VRIX27EAT6VLFBZ7Q, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.641159578 (75h46m41.204482589s) [noise=true , address=104.36.82.9:6039 , piece_id=I4AB6N27MYYUHWAZLJJUIBFBLOQFHDIM3YLYWPVRPNUV7VMYS2SA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.641588126 (75h46m41.204911136s) [noise=true , address=77.90.31.19:4853 , piece_id=VFEON5T2KSWM2KSLTVRZTXYL42JWENZYUEUPG3LMODGZWBMVGGPQ, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.641863297 (75h46m41.205186305s) [noise=true , address=77.90.31.12:4603 , piece_id=BMUIC6KNGEDL37436GLSXW5XOO6VYOPNOSFGQCYVHSENVI2PJ7KQ, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.642124427 (75h46m41.205447437s) [noise=true , address=104.36.82.49:4982 , piece_id=UNLV5UDNWULEIPE4QSQHHZQPK3BQKFRB5T3H6GP7ZQ5QGSNK5V6A, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.642285917 (75h46m41.205608925s) [noise=true , address=104.36.82.54:4171 , piece_id=MUMABKARK4KKUKWW6XDXCQVT6O7SLFHGHE7COWC7QIYGARAA2D5A, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.642656750 (75h46m41.205979758s) [noise=true , address=104.36.82.14:6232 , piece_id=A2ZLUUZKRTBJRJ267QSK36C3LFZV7FPAYDDAXGIWDTNEGNV6MRRA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.643565583 (75h46m41.206888597s) [noise=true , address=79.127.205.245:4231 , piece_id=EJK46XUUVI5NWOAQJEW4QIJG3KRXZHB4YOQUJRCS2MIOSWKGTNLQ, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.644620291 (75h46m41.207943303s) [noise=true , address=77.90.31.14:4670 , piece_id=7FNANMUKPM3N6YWRAOAIM4WJZJUTNF7ZBI6ZUBNEPGNGMFWOE24A, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.648842639 (75h46m41.212165631s) [noise=true , address=104.36.82.55:4224 , piece_id=WRVNGSJV5BX4GJERLXNPAS7J7BZ5WFTM7R5H257YRCG5SP2HIYMA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.825828239 (75h46m41.389151241s) [noise=true , address=104.36.82.53:4126 , piece_id=2MKRQJEPGDJKDM7VWY5M4M6NORILAIEYYATJ7HVZEOBOMXOI2LBA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: ecclient: upload failed (node:12dMWFrq2NMpnMzuPAphWerntC5kkrcnzRJZBA116uzJzyvk6fH, address:104.36.82.53:4126): protocol: storage node overloaded, request limit: 2000 Jun 21 20:10:05.834666043 (75h46m41.397989033s) [upload=60693]: Next returned nil segment Jun 21 20:10:05.834686295 (75h46m41.398009302s) [upload=60693]: Waiting for error group managing segments... Jun 21 20:10:05.877010488 (75h46m41.440333486s) [noise=true , address=45.91.116.6:4982 , piece_id=VP45R6KXNH5U5SGUQUMAPIBY7KFDPBIR3S3VRIX27EAT6VLFBZ7Q, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:05.886336964 (75h46m41.449659967s) [noise=true , address=104.36.82.9:6039 , piece_id=I4AB6N27MYYUHWAZLJJUIBFBLOQFHDIM3YLYWPVRPNUV7VMYS2SA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: ecclient: upload failed (node:12P11WVW9mVXw5ra2gZ3Rezon6wZi8GrN5CchQN3MiVU4ts3L6V, address:104.36.82.9:6039): protocol: storage node overloaded, request limit: 2000 Jun 21 20:10:08.064514542 (75h46m43.627837553s) [noise=true , address=77.90.31.14:4670 , piece_id=7FNANMUKPM3N6YWRAOAIM4WJZJUTNF7ZBI6ZUBNEPGNGMFWOE24A, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:08.099098391 (75h46m43.662421396s) [noise=true , address=79.127.205.245:4231 , piece_id=EJK46XUUVI5NWOAQJEW4QIJG3KRXZHB4YOQUJRCS2MIOSWKGTNLQ, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:08.489461749 (75h46m44.052784752s) [noise=true , address=104.36.82.42:6807 , piece_id=ZBQZPXUDJ4VBCOSVKWAJ6SK6UD7SLPON7Y4SSCK2DQ5STFG4XCNQ, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:08.650922093 (75h46m44.214245093s) [noise=true , address=77.90.31.19:4853 , piece_id=VFEON5T2KSWM2KSLTVRZTXYL42JWENZYUEUPG3LMODGZWBMVGGPQ, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:09.917746439 (75h46m45.481069441s) [noise=true , address=104.36.82.13:6196 , piece_id=BCHMQO5KKT5BLNWQGOHH3V43LGSE3BUQCI2SY4TTO5M5GQNOCTMQ, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:10.830722215 (75h46m46.394045216s) [noise=true , address=104.36.82.5:5869 , piece_id=HAEHXXZASV3DHT3QN7NTZOS6XZKHNUWBKCHWJJWSTJKXAKJKKBAA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:12.738783450 (75h46m48.302106452s) [noise=true , address=77.90.31.12:4603 , piece_id=BMUIC6KNGEDL37436GLSXW5XOO6VYOPNOSFGQCYVHSENVI2PJ7KQ, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:15.460927713 (75h46m51.024250723s) [noise=true , address=104.36.82.52:4111 , piece_id=HB7QAWN6CGNIIT4HD2GZ7JDQP5NXIJ5I2USBM66YOKTYENNOWG5A, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:18.393817968 (75h46m53.95714096s) [noise=true , address=104.36.82.55:4224 , piece_id=WRVNGSJV5BX4GJERLXNPAS7J7BZ5WFTM7R5H257YRCG5SP2HIYMA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:19.549174143 (75h46m55.112497151s) [noise=true , address=104.36.82.56:4253 , piece_id=5PMVGLZETMXCQ77K3I6PXACRIMBGR2CV5JGUSU5KUMYFNXVGA2GA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:22.511944593 (75h46m58.075267584s) [noise=true , address=104.36.82.37:6663 , piece_id=X7LWYIIHJFU2T2PHYR3R5TNT22UUPHEHQWZH7ZJMLS4AH4PJ6AIA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:48.027197558 (75h47m23.590520562s) [noise=true , address=104.36.82.14:6232 , piece_id=A2ZLUUZKRTBJRJ267QSK36C3LFZV7FPAYDDAXGIWDTNEGNV6MRRA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:11:05.567692582 (75h47m41.131015574s) [noise=true , address=104.36.82.49:4982 , piece_id=UNLV5UDNWULEIPE4QSQHHZQPK3BQKFRB5T3H6GP7ZQ5QGSNK5V6A, seg_pos={0 0}, upload=60693]: Done uploading piece. err: ecclient: upload failed (node:1NxoYJXCXmTvQncbkXt6yH7mhEBBrzVdzhgL5J6W5TD4N3pkaG, address:104.36.82.49:4982): protocol: expected piece hash; context deadline exceeded; EOF Jun 21 20:11:05.567759198 (75h47m41.131082195s) [noise=true , address=104.36.82.54:4171 , piece_id=MUMABKARK4KKUKWW6XDXCQVT6O7SLFHGHE7COWC7QIYGARAA2D5A, seg_pos={0 0}, upload=60693]: Done uploading piece. err: ecclient: upload failed (node:125SMhDsEZruNu6LkmQn2M7R6WzvXTdk86DRsKf522u8BNiANq3, address:104.36.82.54:4171): protocol: expected piece hash; context deadline exceeded; EOF Jun 21 20:11:05.567869019 (75h47m41.131192033s) [noise=true , address=104.36.82.10:6087 , piece_id=5FQG7BPZC373PFB54HYXSG2SVKOBBBSGZCLC4XDJEYQFOLQY74MA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: ecclient: upload failed (node:1S9k5eejJcxkiG5pXjjBZm18uP8k5jCGfWyeW2QQxmuRU8ZzRA, address:104.36.82.10:6087): protocol: write tcp 159.223.143.39:38550->104.36.82.10:6087: use of closed network connection; write tcp 159.223.143.39:38550->104.36.82.10:6087: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:38550->104.36.82.10:6087: use of closed network connection Jun 21 20:11:05.567932150 (75h47m41.131255161s) [seg_pos={0 0}, upload=60693]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jun 21 20:11:05.593626175 (75h47m41.156949181s) [upload=60693]: Done uploading segments... Jun 21 20:11:05.593657274 (75h47m41.156980265s) [upload=60693]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 16.873210501s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [...............X...........X................................] duration: 15.273707081s FAIL (2025-06-21 20:37:06 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== pload=60693]: Starting upload Jun 21 20:10:05.602866542 (75h46m41.166189557s) [upload=60693]: Done starting upload Jun 21 20:10:05.603116545 (75h46m41.166439547s) [upload=60693]: Uploading segments... Jun 21 20:10:05.603495389 (75h46m41.166818397s) [upload=60693]: Got next segment. Inline: false Jun 21 20:10:05.603566566 (75h46m41.166889558s) [upload=60693]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 21 20:10:05.603573280 (75h46m41.166896286s) [upload=60693]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 21 20:10:05.639130162 (75h46m41.202453164s) [seg_pos={0 0}, upload=60693]: Begin upload segment... Jun 21 20:10:05.639231070 (75h46m41.202554057s) [seg_pos={0 0}, upload=60693]: Done begin upload segment. Jun 21 20:10:05.639292724 (75h46m41.202615728s) [noise=true , address=104.36.82.56:4253 , piece_id=5PMVGLZETMXCQ77K3I6PXACRIMBGR2CV5JGUSU5KUMYFNXVGA2GA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.639338066 (75h46m41.202661071s) [noise=true , address=104.36.82.53:4126 , piece_id=2MKRQJEPGDJKDM7VWY5M4M6NORILAIEYYATJ7HVZEOBOMXOI2LBA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.639416264 (75h46m41.202739267s) [noise=true , address=104.36.82.5:5869 , piece_id=HAEHXXZASV3DHT3QN7NTZOS6XZKHNUWBKCHWJJWSTJKXAKJKKBAA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.639873098 (75h46m41.203196115s) [noise=true , address=104.36.82.52:4111 , piece_id=HB7QAWN6CGNIIT4HD2GZ7JDQP5NXIJ5I2USBM66YOKTYENNOWG5A, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.639887296 (75h46m41.203210304s) [noise=true , address=104.36.82.10:6087 , piece_id=5FQG7BPZC373PFB54HYXSG2SVKOBBBSGZCLC4XDJEYQFOLQY74MA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.640266623 (75h46m41.203589616s) [noise=true , address=104.36.82.13:6196 , piece_id=BCHMQO5KKT5BLNWQGOHH3V43LGSE3BUQCI2SY4TTO5M5GQNOCTMQ, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.640352274 (75h46m41.203675286s) [noise=true , address=104.36.82.37:6663 , piece_id=X7LWYIIHJFU2T2PHYR3R5TNT22UUPHEHQWZH7ZJMLS4AH4PJ6AIA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.640659767 (75h46m41.203982779s) [noise=true , address=104.36.82.42:6807 , piece_id=ZBQZPXUDJ4VBCOSVKWAJ6SK6UD7SLPON7Y4SSCK2DQ5STFG4XCNQ, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.640961701 (75h46m41.20428471s) [noise=true , address=45.91.116.6:4982 , piece_id=VP45R6KXNH5U5SGUQUMAPIBY7KFDPBIR3S3VRIX27EAT6VLFBZ7Q, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.641159578 (75h46m41.204482589s) [noise=true , address=104.36.82.9:6039 , piece_id=I4AB6N27MYYUHWAZLJJUIBFBLOQFHDIM3YLYWPVRPNUV7VMYS2SA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.641588126 (75h46m41.204911136s) [noise=true , address=77.90.31.19:4853 , piece_id=VFEON5T2KSWM2KSLTVRZTXYL42JWENZYUEUPG3LMODGZWBMVGGPQ, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.641863297 (75h46m41.205186305s) [noise=true , address=77.90.31.12:4603 , piece_id=BMUIC6KNGEDL37436GLSXW5XOO6VYOPNOSFGQCYVHSENVI2PJ7KQ, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.642124427 (75h46m41.205447437s) [noise=true , address=104.36.82.49:4982 , piece_id=UNLV5UDNWULEIPE4QSQHHZQPK3BQKFRB5T3H6GP7ZQ5QGSNK5V6A, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.642285917 (75h46m41.205608925s) [noise=true , address=104.36.82.54:4171 , piece_id=MUMABKARK4KKUKWW6XDXCQVT6O7SLFHGHE7COWC7QIYGARAA2D5A, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.642656750 (75h46m41.205979758s) [noise=true , address=104.36.82.14:6232 , piece_id=A2ZLUUZKRTBJRJ267QSK36C3LFZV7FPAYDDAXGIWDTNEGNV6MRRA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.643565583 (75h46m41.206888597s) [noise=true , address=79.127.205.245:4231 , piece_id=EJK46XUUVI5NWOAQJEW4QIJG3KRXZHB4YOQUJRCS2MIOSWKGTNLQ, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.644620291 (75h46m41.207943303s) [noise=true , address=77.90.31.14:4670 , piece_id=7FNANMUKPM3N6YWRAOAIM4WJZJUTNF7ZBI6ZUBNEPGNGMFWOE24A, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.648842639 (75h46m41.212165631s) [noise=true , address=104.36.82.55:4224 , piece_id=WRVNGSJV5BX4GJERLXNPAS7J7BZ5WFTM7R5H257YRCG5SP2HIYMA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.825828239 (75h46m41.389151241s) [noise=true , address=104.36.82.53:4126 , piece_id=2MKRQJEPGDJKDM7VWY5M4M6NORILAIEYYATJ7HVZEOBOMXOI2LBA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: ecclient: upload failed (node:12dMWFrq2NMpnMzuPAphWerntC5kkrcnzRJZBA116uzJzyvk6fH, address:104.36.82.53:4126): protocol: storage node overloaded, request limit: 2000 Jun 21 20:10:05.834666043 (75h46m41.397989033s) [upload=60693]: Next returned nil segment Jun 21 20:10:05.834686295 (75h46m41.398009302s) [upload=60693]: Waiting for error group managing segments... Jun 21 20:10:05.877010488 (75h46m41.440333486s) [noise=true , address=45.91.116.6:4982 , piece_id=VP45R6KXNH5U5SGUQUMAPIBY7KFDPBIR3S3VRIX27EAT6VLFBZ7Q, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:05.886336964 (75h46m41.449659967s) [noise=true , address=104.36.82.9:6039 , piece_id=I4AB6N27MYYUHWAZLJJUIBFBLOQFHDIM3YLYWPVRPNUV7VMYS2SA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: ecclient: upload failed (node:12P11WVW9mVXw5ra2gZ3Rezon6wZi8GrN5CchQN3MiVU4ts3L6V, address:104.36.82.9:6039): protocol: storage node overloaded, request limit: 2000 Jun 21 20:10:08.064514542 (75h46m43.627837553s) [noise=true , address=77.90.31.14:4670 , piece_id=7FNANMUKPM3N6YWRAOAIM4WJZJUTNF7ZBI6ZUBNEPGNGMFWOE24A, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:08.099098391 (75h46m43.662421396s) [noise=true , address=79.127.205.245:4231 , piece_id=EJK46XUUVI5NWOAQJEW4QIJG3KRXZHB4YOQUJRCS2MIOSWKGTNLQ, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:08.489461749 (75h46m44.052784752s) [noise=true , address=104.36.82.42:6807 , piece_id=ZBQZPXUDJ4VBCOSVKWAJ6SK6UD7SLPON7Y4SSCK2DQ5STFG4XCNQ, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:08.650922093 (75h46m44.214245093s) [noise=true , address=77.90.31.19:4853 , piece_id=VFEON5T2KSWM2KSLTVRZTXYL42JWENZYUEUPG3LMODGZWBMVGGPQ, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:09.917746439 (75h46m45.481069441s) [noise=true , address=104.36.82.13:6196 , piece_id=BCHMQO5KKT5BLNWQGOHH3V43LGSE3BUQCI2SY4TTO5M5GQNOCTMQ, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:10.830722215 (75h46m46.394045216s) [noise=true , address=104.36.82.5:5869 , piece_id=HAEHXXZASV3DHT3QN7NTZOS6XZKHNUWBKCHWJJWSTJKXAKJKKBAA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:12.738783450 (75h46m48.302106452s) [noise=true , address=77.90.31.12:4603 , piece_id=BMUIC6KNGEDL37436GLSXW5XOO6VYOPNOSFGQCYVHSENVI2PJ7KQ, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:15.460927713 (75h46m51.024250723s) [noise=true , address=104.36.82.52:4111 , piece_id=HB7QAWN6CGNIIT4HD2GZ7JDQP5NXIJ5I2USBM66YOKTYENNOWG5A, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:18.393817968 (75h46m53.95714096s) [noise=true , address=104.36.82.55:4224 , piece_id=WRVNGSJV5BX4GJERLXNPAS7J7BZ5WFTM7R5H257YRCG5SP2HIYMA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:19.549174143 (75h46m55.112497151s) [noise=true , address=104.36.82.56:4253 , piece_id=5PMVGLZETMXCQ77K3I6PXACRIMBGR2CV5JGUSU5KUMYFNXVGA2GA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:22.511944593 (75h46m58.075267584s) [noise=true , address=104.36.82.37:6663 , piece_id=X7LWYIIHJFU2T2PHYR3R5TNT22UUPHEHQWZH7ZJMLS4AH4PJ6AIA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:48.027197558 (75h47m23.590520562s) [noise=true , address=104.36.82.14:6232 , piece_id=A2ZLUUZKRTBJRJ267QSK36C3LFZV7FPAYDDAXGIWDTNEGNV6MRRA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:11:05.567692582 (75h47m41.131015574s) [noise=true , address=104.36.82.49:4982 , piece_id=UNLV5UDNWULEIPE4QSQHHZQPK3BQKFRB5T3H6GP7ZQ5QGSNK5V6A, seg_pos={0 0}, upload=60693]: Done uploading piece. err: ecclient: upload failed (node:1NxoYJXCXmTvQncbkXt6yH7mhEBBrzVdzhgL5J6W5TD4N3pkaG, address:104.36.82.49:4982): protocol: expected piece hash; context deadline exceeded; EOF Jun 21 20:11:05.567759198 (75h47m41.131082195s) [noise=true , address=104.36.82.54:4171 , piece_id=MUMABKARK4KKUKWW6XDXCQVT6O7SLFHGHE7COWC7QIYGARAA2D5A, seg_pos={0 0}, upload=60693]: Done uploading piece. err: ecclient: upload failed (node:125SMhDsEZruNu6LkmQn2M7R6WzvXTdk86DRsKf522u8BNiANq3, address:104.36.82.54:4171): protocol: expected piece hash; context deadline exceeded; EOF Jun 21 20:11:05.567869019 (75h47m41.131192033s) [noise=true , address=104.36.82.10:6087 , piece_id=5FQG7BPZC373PFB54HYXSG2SVKOBBBSGZCLC4XDJEYQFOLQY74MA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: ecclient: upload failed (node:1S9k5eejJcxkiG5pXjjBZm18uP8k5jCGfWyeW2QQxmuRU8ZzRA, address:104.36.82.10:6087): protocol: write tcp 159.223.143.39:38550->104.36.82.10:6087: use of closed network connection; write tcp 159.223.143.39:38550->104.36.82.10:6087: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:38550->104.36.82.10:6087: use of closed network connection Jun 21 20:11:05.567932150 (75h47m41.131255161s) [seg_pos={0 0}, upload=60693]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jun 21 20:11:05.593626175 (75h47m41.156949181s) [upload=60693]: Done uploading segments... Jun 21 20:11:05.593657274 (75h47m41.156980265s) [upload=60693]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 15.273731147s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [................X...........X...............................] duration: 24.266877857s FAIL (2025-06-21 20:36:41 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== pload=60693]: Starting upload Jun 21 20:10:05.602866542 (75h46m41.166189557s) [upload=60693]: Done starting upload Jun 21 20:10:05.603116545 (75h46m41.166439547s) [upload=60693]: Uploading segments... Jun 21 20:10:05.603495389 (75h46m41.166818397s) [upload=60693]: Got next segment. Inline: false Jun 21 20:10:05.603566566 (75h46m41.166889558s) [upload=60693]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 21 20:10:05.603573280 (75h46m41.166896286s) [upload=60693]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 21 20:10:05.639130162 (75h46m41.202453164s) [seg_pos={0 0}, upload=60693]: Begin upload segment... Jun 21 20:10:05.639231070 (75h46m41.202554057s) [seg_pos={0 0}, upload=60693]: Done begin upload segment. Jun 21 20:10:05.639292724 (75h46m41.202615728s) [noise=true , address=104.36.82.56:4253 , piece_id=5PMVGLZETMXCQ77K3I6PXACRIMBGR2CV5JGUSU5KUMYFNXVGA2GA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.639338066 (75h46m41.202661071s) [noise=true , address=104.36.82.53:4126 , piece_id=2MKRQJEPGDJKDM7VWY5M4M6NORILAIEYYATJ7HVZEOBOMXOI2LBA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.639416264 (75h46m41.202739267s) [noise=true , address=104.36.82.5:5869 , piece_id=HAEHXXZASV3DHT3QN7NTZOS6XZKHNUWBKCHWJJWSTJKXAKJKKBAA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.639873098 (75h46m41.203196115s) [noise=true , address=104.36.82.52:4111 , piece_id=HB7QAWN6CGNIIT4HD2GZ7JDQP5NXIJ5I2USBM66YOKTYENNOWG5A, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.639887296 (75h46m41.203210304s) [noise=true , address=104.36.82.10:6087 , piece_id=5FQG7BPZC373PFB54HYXSG2SVKOBBBSGZCLC4XDJEYQFOLQY74MA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.640266623 (75h46m41.203589616s) [noise=true , address=104.36.82.13:6196 , piece_id=BCHMQO5KKT5BLNWQGOHH3V43LGSE3BUQCI2SY4TTO5M5GQNOCTMQ, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.640352274 (75h46m41.203675286s) [noise=true , address=104.36.82.37:6663 , piece_id=X7LWYIIHJFU2T2PHYR3R5TNT22UUPHEHQWZH7ZJMLS4AH4PJ6AIA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.640659767 (75h46m41.203982779s) [noise=true , address=104.36.82.42:6807 , piece_id=ZBQZPXUDJ4VBCOSVKWAJ6SK6UD7SLPON7Y4SSCK2DQ5STFG4XCNQ, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.640961701 (75h46m41.20428471s) [noise=true , address=45.91.116.6:4982 , piece_id=VP45R6KXNH5U5SGUQUMAPIBY7KFDPBIR3S3VRIX27EAT6VLFBZ7Q, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.641159578 (75h46m41.204482589s) [noise=true , address=104.36.82.9:6039 , piece_id=I4AB6N27MYYUHWAZLJJUIBFBLOQFHDIM3YLYWPVRPNUV7VMYS2SA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.641588126 (75h46m41.204911136s) [noise=true , address=77.90.31.19:4853 , piece_id=VFEON5T2KSWM2KSLTVRZTXYL42JWENZYUEUPG3LMODGZWBMVGGPQ, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.641863297 (75h46m41.205186305s) [noise=true , address=77.90.31.12:4603 , piece_id=BMUIC6KNGEDL37436GLSXW5XOO6VYOPNOSFGQCYVHSENVI2PJ7KQ, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.642124427 (75h46m41.205447437s) [noise=true , address=104.36.82.49:4982 , piece_id=UNLV5UDNWULEIPE4QSQHHZQPK3BQKFRB5T3H6GP7ZQ5QGSNK5V6A, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.642285917 (75h46m41.205608925s) [noise=true , address=104.36.82.54:4171 , piece_id=MUMABKARK4KKUKWW6XDXCQVT6O7SLFHGHE7COWC7QIYGARAA2D5A, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.642656750 (75h46m41.205979758s) [noise=true , address=104.36.82.14:6232 , piece_id=A2ZLUUZKRTBJRJ267QSK36C3LFZV7FPAYDDAXGIWDTNEGNV6MRRA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.643565583 (75h46m41.206888597s) [noise=true , address=79.127.205.245:4231 , piece_id=EJK46XUUVI5NWOAQJEW4QIJG3KRXZHB4YOQUJRCS2MIOSWKGTNLQ, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.644620291 (75h46m41.207943303s) [noise=true , address=77.90.31.14:4670 , piece_id=7FNANMUKPM3N6YWRAOAIM4WJZJUTNF7ZBI6ZUBNEPGNGMFWOE24A, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.648842639 (75h46m41.212165631s) [noise=true , address=104.36.82.55:4224 , piece_id=WRVNGSJV5BX4GJERLXNPAS7J7BZ5WFTM7R5H257YRCG5SP2HIYMA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.825828239 (75h46m41.389151241s) [noise=true , address=104.36.82.53:4126 , piece_id=2MKRQJEPGDJKDM7VWY5M4M6NORILAIEYYATJ7HVZEOBOMXOI2LBA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: ecclient: upload failed (node:12dMWFrq2NMpnMzuPAphWerntC5kkrcnzRJZBA116uzJzyvk6fH, address:104.36.82.53:4126): protocol: storage node overloaded, request limit: 2000 Jun 21 20:10:05.834666043 (75h46m41.397989033s) [upload=60693]: Next returned nil segment Jun 21 20:10:05.834686295 (75h46m41.398009302s) [upload=60693]: Waiting for error group managing segments... Jun 21 20:10:05.877010488 (75h46m41.440333486s) [noise=true , address=45.91.116.6:4982 , piece_id=VP45R6KXNH5U5SGUQUMAPIBY7KFDPBIR3S3VRIX27EAT6VLFBZ7Q, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:05.886336964 (75h46m41.449659967s) [noise=true , address=104.36.82.9:6039 , piece_id=I4AB6N27MYYUHWAZLJJUIBFBLOQFHDIM3YLYWPVRPNUV7VMYS2SA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: ecclient: upload failed (node:12P11WVW9mVXw5ra2gZ3Rezon6wZi8GrN5CchQN3MiVU4ts3L6V, address:104.36.82.9:6039): protocol: storage node overloaded, request limit: 2000 Jun 21 20:10:08.064514542 (75h46m43.627837553s) [noise=true , address=77.90.31.14:4670 , piece_id=7FNANMUKPM3N6YWRAOAIM4WJZJUTNF7ZBI6ZUBNEPGNGMFWOE24A, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:08.099098391 (75h46m43.662421396s) [noise=true , address=79.127.205.245:4231 , piece_id=EJK46XUUVI5NWOAQJEW4QIJG3KRXZHB4YOQUJRCS2MIOSWKGTNLQ, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:08.489461749 (75h46m44.052784752s) [noise=true , address=104.36.82.42:6807 , piece_id=ZBQZPXUDJ4VBCOSVKWAJ6SK6UD7SLPON7Y4SSCK2DQ5STFG4XCNQ, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:08.650922093 (75h46m44.214245093s) [noise=true , address=77.90.31.19:4853 , piece_id=VFEON5T2KSWM2KSLTVRZTXYL42JWENZYUEUPG3LMODGZWBMVGGPQ, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:09.917746439 (75h46m45.481069441s) [noise=true , address=104.36.82.13:6196 , piece_id=BCHMQO5KKT5BLNWQGOHH3V43LGSE3BUQCI2SY4TTO5M5GQNOCTMQ, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:10.830722215 (75h46m46.394045216s) [noise=true , address=104.36.82.5:5869 , piece_id=HAEHXXZASV3DHT3QN7NTZOS6XZKHNUWBKCHWJJWSTJKXAKJKKBAA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:12.738783450 (75h46m48.302106452s) [noise=true , address=77.90.31.12:4603 , piece_id=BMUIC6KNGEDL37436GLSXW5XOO6VYOPNOSFGQCYVHSENVI2PJ7KQ, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:15.460927713 (75h46m51.024250723s) [noise=true , address=104.36.82.52:4111 , piece_id=HB7QAWN6CGNIIT4HD2GZ7JDQP5NXIJ5I2USBM66YOKTYENNOWG5A, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:18.393817968 (75h46m53.95714096s) [noise=true , address=104.36.82.55:4224 , piece_id=WRVNGSJV5BX4GJERLXNPAS7J7BZ5WFTM7R5H257YRCG5SP2HIYMA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:19.549174143 (75h46m55.112497151s) [noise=true , address=104.36.82.56:4253 , piece_id=5PMVGLZETMXCQ77K3I6PXACRIMBGR2CV5JGUSU5KUMYFNXVGA2GA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:22.511944593 (75h46m58.075267584s) [noise=true , address=104.36.82.37:6663 , piece_id=X7LWYIIHJFU2T2PHYR3R5TNT22UUPHEHQWZH7ZJMLS4AH4PJ6AIA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:48.027197558 (75h47m23.590520562s) [noise=true , address=104.36.82.14:6232 , piece_id=A2ZLUUZKRTBJRJ267QSK36C3LFZV7FPAYDDAXGIWDTNEGNV6MRRA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:11:05.567692582 (75h47m41.131015574s) [noise=true , address=104.36.82.49:4982 , piece_id=UNLV5UDNWULEIPE4QSQHHZQPK3BQKFRB5T3H6GP7ZQ5QGSNK5V6A, seg_pos={0 0}, upload=60693]: Done uploading piece. err: ecclient: upload failed (node:1NxoYJXCXmTvQncbkXt6yH7mhEBBrzVdzhgL5J6W5TD4N3pkaG, address:104.36.82.49:4982): protocol: expected piece hash; context deadline exceeded; EOF Jun 21 20:11:05.567759198 (75h47m41.131082195s) [noise=true , address=104.36.82.54:4171 , piece_id=MUMABKARK4KKUKWW6XDXCQVT6O7SLFHGHE7COWC7QIYGARAA2D5A, seg_pos={0 0}, upload=60693]: Done uploading piece. err: ecclient: upload failed (node:125SMhDsEZruNu6LkmQn2M7R6WzvXTdk86DRsKf522u8BNiANq3, address:104.36.82.54:4171): protocol: expected piece hash; context deadline exceeded; EOF Jun 21 20:11:05.567869019 (75h47m41.131192033s) [noise=true , address=104.36.82.10:6087 , piece_id=5FQG7BPZC373PFB54HYXSG2SVKOBBBSGZCLC4XDJEYQFOLQY74MA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: ecclient: upload failed (node:1S9k5eejJcxkiG5pXjjBZm18uP8k5jCGfWyeW2QQxmuRU8ZzRA, address:104.36.82.10:6087): protocol: write tcp 159.223.143.39:38550->104.36.82.10:6087: use of closed network connection; write tcp 159.223.143.39:38550->104.36.82.10:6087: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:38550->104.36.82.10:6087: use of closed network connection Jun 21 20:11:05.567932150 (75h47m41.131255161s) [seg_pos={0 0}, upload=60693]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jun 21 20:11:05.593626175 (75h47m41.156949181s) [upload=60693]: Done uploading segments... Jun 21 20:11:05.593657274 (75h47m41.156980265s) [upload=60693]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 24.266913646s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [.................X...........X..............................] duration: 10.041604498s FAIL (2025-06-21 20:35:49 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== pload=60693]: Starting upload Jun 21 20:10:05.602866542 (75h46m41.166189557s) [upload=60693]: Done starting upload Jun 21 20:10:05.603116545 (75h46m41.166439547s) [upload=60693]: Uploading segments... Jun 21 20:10:05.603495389 (75h46m41.166818397s) [upload=60693]: Got next segment. Inline: false Jun 21 20:10:05.603566566 (75h46m41.166889558s) [upload=60693]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 21 20:10:05.603573280 (75h46m41.166896286s) [upload=60693]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 21 20:10:05.639130162 (75h46m41.202453164s) [seg_pos={0 0}, upload=60693]: Begin upload segment... Jun 21 20:10:05.639231070 (75h46m41.202554057s) [seg_pos={0 0}, upload=60693]: Done begin upload segment. Jun 21 20:10:05.639292724 (75h46m41.202615728s) [noise=true , address=104.36.82.56:4253 , piece_id=5PMVGLZETMXCQ77K3I6PXACRIMBGR2CV5JGUSU5KUMYFNXVGA2GA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.639338066 (75h46m41.202661071s) [noise=true , address=104.36.82.53:4126 , piece_id=2MKRQJEPGDJKDM7VWY5M4M6NORILAIEYYATJ7HVZEOBOMXOI2LBA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.639416264 (75h46m41.202739267s) [noise=true , address=104.36.82.5:5869 , piece_id=HAEHXXZASV3DHT3QN7NTZOS6XZKHNUWBKCHWJJWSTJKXAKJKKBAA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.639873098 (75h46m41.203196115s) [noise=true , address=104.36.82.52:4111 , piece_id=HB7QAWN6CGNIIT4HD2GZ7JDQP5NXIJ5I2USBM66YOKTYENNOWG5A, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.639887296 (75h46m41.203210304s) [noise=true , address=104.36.82.10:6087 , piece_id=5FQG7BPZC373PFB54HYXSG2SVKOBBBSGZCLC4XDJEYQFOLQY74MA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.640266623 (75h46m41.203589616s) [noise=true , address=104.36.82.13:6196 , piece_id=BCHMQO5KKT5BLNWQGOHH3V43LGSE3BUQCI2SY4TTO5M5GQNOCTMQ, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.640352274 (75h46m41.203675286s) [noise=true , address=104.36.82.37:6663 , piece_id=X7LWYIIHJFU2T2PHYR3R5TNT22UUPHEHQWZH7ZJMLS4AH4PJ6AIA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.640659767 (75h46m41.203982779s) [noise=true , address=104.36.82.42:6807 , piece_id=ZBQZPXUDJ4VBCOSVKWAJ6SK6UD7SLPON7Y4SSCK2DQ5STFG4XCNQ, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.640961701 (75h46m41.20428471s) [noise=true , address=45.91.116.6:4982 , piece_id=VP45R6KXNH5U5SGUQUMAPIBY7KFDPBIR3S3VRIX27EAT6VLFBZ7Q, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.641159578 (75h46m41.204482589s) [noise=true , address=104.36.82.9:6039 , piece_id=I4AB6N27MYYUHWAZLJJUIBFBLOQFHDIM3YLYWPVRPNUV7VMYS2SA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.641588126 (75h46m41.204911136s) [noise=true , address=77.90.31.19:4853 , piece_id=VFEON5T2KSWM2KSLTVRZTXYL42JWENZYUEUPG3LMODGZWBMVGGPQ, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.641863297 (75h46m41.205186305s) [noise=true , address=77.90.31.12:4603 , piece_id=BMUIC6KNGEDL37436GLSXW5XOO6VYOPNOSFGQCYVHSENVI2PJ7KQ, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.642124427 (75h46m41.205447437s) [noise=true , address=104.36.82.49:4982 , piece_id=UNLV5UDNWULEIPE4QSQHHZQPK3BQKFRB5T3H6GP7ZQ5QGSNK5V6A, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.642285917 (75h46m41.205608925s) [noise=true , address=104.36.82.54:4171 , piece_id=MUMABKARK4KKUKWW6XDXCQVT6O7SLFHGHE7COWC7QIYGARAA2D5A, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.642656750 (75h46m41.205979758s) [noise=true , address=104.36.82.14:6232 , piece_id=A2ZLUUZKRTBJRJ267QSK36C3LFZV7FPAYDDAXGIWDTNEGNV6MRRA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.643565583 (75h46m41.206888597s) [noise=true , address=79.127.205.245:4231 , piece_id=EJK46XUUVI5NWOAQJEW4QIJG3KRXZHB4YOQUJRCS2MIOSWKGTNLQ, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.644620291 (75h46m41.207943303s) [noise=true , address=77.90.31.14:4670 , piece_id=7FNANMUKPM3N6YWRAOAIM4WJZJUTNF7ZBI6ZUBNEPGNGMFWOE24A, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.648842639 (75h46m41.212165631s) [noise=true , address=104.36.82.55:4224 , piece_id=WRVNGSJV5BX4GJERLXNPAS7J7BZ5WFTM7R5H257YRCG5SP2HIYMA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.825828239 (75h46m41.389151241s) [noise=true , address=104.36.82.53:4126 , piece_id=2MKRQJEPGDJKDM7VWY5M4M6NORILAIEYYATJ7HVZEOBOMXOI2LBA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: ecclient: upload failed (node:12dMWFrq2NMpnMzuPAphWerntC5kkrcnzRJZBA116uzJzyvk6fH, address:104.36.82.53:4126): protocol: storage node overloaded, request limit: 2000 Jun 21 20:10:05.834666043 (75h46m41.397989033s) [upload=60693]: Next returned nil segment Jun 21 20:10:05.834686295 (75h46m41.398009302s) [upload=60693]: Waiting for error group managing segments... Jun 21 20:10:05.877010488 (75h46m41.440333486s) [noise=true , address=45.91.116.6:4982 , piece_id=VP45R6KXNH5U5SGUQUMAPIBY7KFDPBIR3S3VRIX27EAT6VLFBZ7Q, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:05.886336964 (75h46m41.449659967s) [noise=true , address=104.36.82.9:6039 , piece_id=I4AB6N27MYYUHWAZLJJUIBFBLOQFHDIM3YLYWPVRPNUV7VMYS2SA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: ecclient: upload failed (node:12P11WVW9mVXw5ra2gZ3Rezon6wZi8GrN5CchQN3MiVU4ts3L6V, address:104.36.82.9:6039): protocol: storage node overloaded, request limit: 2000 Jun 21 20:10:08.064514542 (75h46m43.627837553s) [noise=true , address=77.90.31.14:4670 , piece_id=7FNANMUKPM3N6YWRAOAIM4WJZJUTNF7ZBI6ZUBNEPGNGMFWOE24A, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:08.099098391 (75h46m43.662421396s) [noise=true , address=79.127.205.245:4231 , piece_id=EJK46XUUVI5NWOAQJEW4QIJG3KRXZHB4YOQUJRCS2MIOSWKGTNLQ, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:08.489461749 (75h46m44.052784752s) [noise=true , address=104.36.82.42:6807 , piece_id=ZBQZPXUDJ4VBCOSVKWAJ6SK6UD7SLPON7Y4SSCK2DQ5STFG4XCNQ, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:08.650922093 (75h46m44.214245093s) [noise=true , address=77.90.31.19:4853 , piece_id=VFEON5T2KSWM2KSLTVRZTXYL42JWENZYUEUPG3LMODGZWBMVGGPQ, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:09.917746439 (75h46m45.481069441s) [noise=true , address=104.36.82.13:6196 , piece_id=BCHMQO5KKT5BLNWQGOHH3V43LGSE3BUQCI2SY4TTO5M5GQNOCTMQ, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:10.830722215 (75h46m46.394045216s) [noise=true , address=104.36.82.5:5869 , piece_id=HAEHXXZASV3DHT3QN7NTZOS6XZKHNUWBKCHWJJWSTJKXAKJKKBAA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:12.738783450 (75h46m48.302106452s) [noise=true , address=77.90.31.12:4603 , piece_id=BMUIC6KNGEDL37436GLSXW5XOO6VYOPNOSFGQCYVHSENVI2PJ7KQ, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:15.460927713 (75h46m51.024250723s) [noise=true , address=104.36.82.52:4111 , piece_id=HB7QAWN6CGNIIT4HD2GZ7JDQP5NXIJ5I2USBM66YOKTYENNOWG5A, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:18.393817968 (75h46m53.95714096s) [noise=true , address=104.36.82.55:4224 , piece_id=WRVNGSJV5BX4GJERLXNPAS7J7BZ5WFTM7R5H257YRCG5SP2HIYMA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:19.549174143 (75h46m55.112497151s) [noise=true , address=104.36.82.56:4253 , piece_id=5PMVGLZETMXCQ77K3I6PXACRIMBGR2CV5JGUSU5KUMYFNXVGA2GA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:22.511944593 (75h46m58.075267584s) [noise=true , address=104.36.82.37:6663 , piece_id=X7LWYIIHJFU2T2PHYR3R5TNT22UUPHEHQWZH7ZJMLS4AH4PJ6AIA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:48.027197558 (75h47m23.590520562s) [noise=true , address=104.36.82.14:6232 , piece_id=A2ZLUUZKRTBJRJ267QSK36C3LFZV7FPAYDDAXGIWDTNEGNV6MRRA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:11:05.567692582 (75h47m41.131015574s) [noise=true , address=104.36.82.49:4982 , piece_id=UNLV5UDNWULEIPE4QSQHHZQPK3BQKFRB5T3H6GP7ZQ5QGSNK5V6A, seg_pos={0 0}, upload=60693]: Done uploading piece. err: ecclient: upload failed (node:1NxoYJXCXmTvQncbkXt6yH7mhEBBrzVdzhgL5J6W5TD4N3pkaG, address:104.36.82.49:4982): protocol: expected piece hash; context deadline exceeded; EOF Jun 21 20:11:05.567759198 (75h47m41.131082195s) [noise=true , address=104.36.82.54:4171 , piece_id=MUMABKARK4KKUKWW6XDXCQVT6O7SLFHGHE7COWC7QIYGARAA2D5A, seg_pos={0 0}, upload=60693]: Done uploading piece. err: ecclient: upload failed (node:125SMhDsEZruNu6LkmQn2M7R6WzvXTdk86DRsKf522u8BNiANq3, address:104.36.82.54:4171): protocol: expected piece hash; context deadline exceeded; EOF Jun 21 20:11:05.567869019 (75h47m41.131192033s) [noise=true , address=104.36.82.10:6087 , piece_id=5FQG7BPZC373PFB54HYXSG2SVKOBBBSGZCLC4XDJEYQFOLQY74MA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: ecclient: upload failed (node:1S9k5eejJcxkiG5pXjjBZm18uP8k5jCGfWyeW2QQxmuRU8ZzRA, address:104.36.82.10:6087): protocol: write tcp 159.223.143.39:38550->104.36.82.10:6087: use of closed network connection; write tcp 159.223.143.39:38550->104.36.82.10:6087: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:38550->104.36.82.10:6087: use of closed network connection Jun 21 20:11:05.567932150 (75h47m41.131255161s) [seg_pos={0 0}, upload=60693]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jun 21 20:11:05.593626175 (75h47m41.156949181s) [upload=60693]: Done uploading segments... Jun 21 20:11:05.593657274 (75h47m41.156980265s) [upload=60693]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 10.041635478s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [..................X...........X.............................] duration: 26.9442952s FAIL (2025-06-21 20:35:21 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== pload=60693]: Starting upload Jun 21 20:10:05.602866542 (75h46m41.166189557s) [upload=60693]: Done starting upload Jun 21 20:10:05.603116545 (75h46m41.166439547s) [upload=60693]: Uploading segments... Jun 21 20:10:05.603495389 (75h46m41.166818397s) [upload=60693]: Got next segment. Inline: false Jun 21 20:10:05.603566566 (75h46m41.166889558s) [upload=60693]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 21 20:10:05.603573280 (75h46m41.166896286s) [upload=60693]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 21 20:10:05.639130162 (75h46m41.202453164s) [seg_pos={0 0}, upload=60693]: Begin upload segment... Jun 21 20:10:05.639231070 (75h46m41.202554057s) [seg_pos={0 0}, upload=60693]: Done begin upload segment. Jun 21 20:10:05.639292724 (75h46m41.202615728s) [noise=true , address=104.36.82.56:4253 , piece_id=5PMVGLZETMXCQ77K3I6PXACRIMBGR2CV5JGUSU5KUMYFNXVGA2GA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.639338066 (75h46m41.202661071s) [noise=true , address=104.36.82.53:4126 , piece_id=2MKRQJEPGDJKDM7VWY5M4M6NORILAIEYYATJ7HVZEOBOMXOI2LBA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.639416264 (75h46m41.202739267s) [noise=true , address=104.36.82.5:5869 , piece_id=HAEHXXZASV3DHT3QN7NTZOS6XZKHNUWBKCHWJJWSTJKXAKJKKBAA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.639873098 (75h46m41.203196115s) [noise=true , address=104.36.82.52:4111 , piece_id=HB7QAWN6CGNIIT4HD2GZ7JDQP5NXIJ5I2USBM66YOKTYENNOWG5A, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.639887296 (75h46m41.203210304s) [noise=true , address=104.36.82.10:6087 , piece_id=5FQG7BPZC373PFB54HYXSG2SVKOBBBSGZCLC4XDJEYQFOLQY74MA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.640266623 (75h46m41.203589616s) [noise=true , address=104.36.82.13:6196 , piece_id=BCHMQO5KKT5BLNWQGOHH3V43LGSE3BUQCI2SY4TTO5M5GQNOCTMQ, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.640352274 (75h46m41.203675286s) [noise=true , address=104.36.82.37:6663 , piece_id=X7LWYIIHJFU2T2PHYR3R5TNT22UUPHEHQWZH7ZJMLS4AH4PJ6AIA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.640659767 (75h46m41.203982779s) [noise=true , address=104.36.82.42:6807 , piece_id=ZBQZPXUDJ4VBCOSVKWAJ6SK6UD7SLPON7Y4SSCK2DQ5STFG4XCNQ, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.640961701 (75h46m41.20428471s) [noise=true , address=45.91.116.6:4982 , piece_id=VP45R6KXNH5U5SGUQUMAPIBY7KFDPBIR3S3VRIX27EAT6VLFBZ7Q, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.641159578 (75h46m41.204482589s) [noise=true , address=104.36.82.9:6039 , piece_id=I4AB6N27MYYUHWAZLJJUIBFBLOQFHDIM3YLYWPVRPNUV7VMYS2SA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.641588126 (75h46m41.204911136s) [noise=true , address=77.90.31.19:4853 , piece_id=VFEON5T2KSWM2KSLTVRZTXYL42JWENZYUEUPG3LMODGZWBMVGGPQ, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.641863297 (75h46m41.205186305s) [noise=true , address=77.90.31.12:4603 , piece_id=BMUIC6KNGEDL37436GLSXW5XOO6VYOPNOSFGQCYVHSENVI2PJ7KQ, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.642124427 (75h46m41.205447437s) [noise=true , address=104.36.82.49:4982 , piece_id=UNLV5UDNWULEIPE4QSQHHZQPK3BQKFRB5T3H6GP7ZQ5QGSNK5V6A, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.642285917 (75h46m41.205608925s) [noise=true , address=104.36.82.54:4171 , piece_id=MUMABKARK4KKUKWW6XDXCQVT6O7SLFHGHE7COWC7QIYGARAA2D5A, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.642656750 (75h46m41.205979758s) [noise=true , address=104.36.82.14:6232 , piece_id=A2ZLUUZKRTBJRJ267QSK36C3LFZV7FPAYDDAXGIWDTNEGNV6MRRA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.643565583 (75h46m41.206888597s) [noise=true , address=79.127.205.245:4231 , piece_id=EJK46XUUVI5NWOAQJEW4QIJG3KRXZHB4YOQUJRCS2MIOSWKGTNLQ, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.644620291 (75h46m41.207943303s) [noise=true , address=77.90.31.14:4670 , piece_id=7FNANMUKPM3N6YWRAOAIM4WJZJUTNF7ZBI6ZUBNEPGNGMFWOE24A, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.648842639 (75h46m41.212165631s) [noise=true , address=104.36.82.55:4224 , piece_id=WRVNGSJV5BX4GJERLXNPAS7J7BZ5WFTM7R5H257YRCG5SP2HIYMA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.825828239 (75h46m41.389151241s) [noise=true , address=104.36.82.53:4126 , piece_id=2MKRQJEPGDJKDM7VWY5M4M6NORILAIEYYATJ7HVZEOBOMXOI2LBA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: ecclient: upload failed (node:12dMWFrq2NMpnMzuPAphWerntC5kkrcnzRJZBA116uzJzyvk6fH, address:104.36.82.53:4126): protocol: storage node overloaded, request limit: 2000 Jun 21 20:10:05.834666043 (75h46m41.397989033s) [upload=60693]: Next returned nil segment Jun 21 20:10:05.834686295 (75h46m41.398009302s) [upload=60693]: Waiting for error group managing segments... Jun 21 20:10:05.877010488 (75h46m41.440333486s) [noise=true , address=45.91.116.6:4982 , piece_id=VP45R6KXNH5U5SGUQUMAPIBY7KFDPBIR3S3VRIX27EAT6VLFBZ7Q, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:05.886336964 (75h46m41.449659967s) [noise=true , address=104.36.82.9:6039 , piece_id=I4AB6N27MYYUHWAZLJJUIBFBLOQFHDIM3YLYWPVRPNUV7VMYS2SA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: ecclient: upload failed (node:12P11WVW9mVXw5ra2gZ3Rezon6wZi8GrN5CchQN3MiVU4ts3L6V, address:104.36.82.9:6039): protocol: storage node overloaded, request limit: 2000 Jun 21 20:10:08.064514542 (75h46m43.627837553s) [noise=true , address=77.90.31.14:4670 , piece_id=7FNANMUKPM3N6YWRAOAIM4WJZJUTNF7ZBI6ZUBNEPGNGMFWOE24A, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:08.099098391 (75h46m43.662421396s) [noise=true , address=79.127.205.245:4231 , piece_id=EJK46XUUVI5NWOAQJEW4QIJG3KRXZHB4YOQUJRCS2MIOSWKGTNLQ, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:08.489461749 (75h46m44.052784752s) [noise=true , address=104.36.82.42:6807 , piece_id=ZBQZPXUDJ4VBCOSVKWAJ6SK6UD7SLPON7Y4SSCK2DQ5STFG4XCNQ, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:08.650922093 (75h46m44.214245093s) [noise=true , address=77.90.31.19:4853 , piece_id=VFEON5T2KSWM2KSLTVRZTXYL42JWENZYUEUPG3LMODGZWBMVGGPQ, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:09.917746439 (75h46m45.481069441s) [noise=true , address=104.36.82.13:6196 , piece_id=BCHMQO5KKT5BLNWQGOHH3V43LGSE3BUQCI2SY4TTO5M5GQNOCTMQ, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:10.830722215 (75h46m46.394045216s) [noise=true , address=104.36.82.5:5869 , piece_id=HAEHXXZASV3DHT3QN7NTZOS6XZKHNUWBKCHWJJWSTJKXAKJKKBAA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:12.738783450 (75h46m48.302106452s) [noise=true , address=77.90.31.12:4603 , piece_id=BMUIC6KNGEDL37436GLSXW5XOO6VYOPNOSFGQCYVHSENVI2PJ7KQ, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:15.460927713 (75h46m51.024250723s) [noise=true , address=104.36.82.52:4111 , piece_id=HB7QAWN6CGNIIT4HD2GZ7JDQP5NXIJ5I2USBM66YOKTYENNOWG5A, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:18.393817968 (75h46m53.95714096s) [noise=true , address=104.36.82.55:4224 , piece_id=WRVNGSJV5BX4GJERLXNPAS7J7BZ5WFTM7R5H257YRCG5SP2HIYMA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:19.549174143 (75h46m55.112497151s) [noise=true , address=104.36.82.56:4253 , piece_id=5PMVGLZETMXCQ77K3I6PXACRIMBGR2CV5JGUSU5KUMYFNXVGA2GA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:22.511944593 (75h46m58.075267584s) [noise=true , address=104.36.82.37:6663 , piece_id=X7LWYIIHJFU2T2PHYR3R5TNT22UUPHEHQWZH7ZJMLS4AH4PJ6AIA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:48.027197558 (75h47m23.590520562s) [noise=true , address=104.36.82.14:6232 , piece_id=A2ZLUUZKRTBJRJ267QSK36C3LFZV7FPAYDDAXGIWDTNEGNV6MRRA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:11:05.567692582 (75h47m41.131015574s) [noise=true , address=104.36.82.49:4982 , piece_id=UNLV5UDNWULEIPE4QSQHHZQPK3BQKFRB5T3H6GP7ZQ5QGSNK5V6A, seg_pos={0 0}, upload=60693]: Done uploading piece. err: ecclient: upload failed (node:1NxoYJXCXmTvQncbkXt6yH7mhEBBrzVdzhgL5J6W5TD4N3pkaG, address:104.36.82.49:4982): protocol: expected piece hash; context deadline exceeded; EOF Jun 21 20:11:05.567759198 (75h47m41.131082195s) [noise=true , address=104.36.82.54:4171 , piece_id=MUMABKARK4KKUKWW6XDXCQVT6O7SLFHGHE7COWC7QIYGARAA2D5A, seg_pos={0 0}, upload=60693]: Done uploading piece. err: ecclient: upload failed (node:125SMhDsEZruNu6LkmQn2M7R6WzvXTdk86DRsKf522u8BNiANq3, address:104.36.82.54:4171): protocol: expected piece hash; context deadline exceeded; EOF Jun 21 20:11:05.567869019 (75h47m41.131192033s) [noise=true , address=104.36.82.10:6087 , piece_id=5FQG7BPZC373PFB54HYXSG2SVKOBBBSGZCLC4XDJEYQFOLQY74MA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: ecclient: upload failed (node:1S9k5eejJcxkiG5pXjjBZm18uP8k5jCGfWyeW2QQxmuRU8ZzRA, address:104.36.82.10:6087): protocol: write tcp 159.223.143.39:38550->104.36.82.10:6087: use of closed network connection; write tcp 159.223.143.39:38550->104.36.82.10:6087: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:38550->104.36.82.10:6087: use of closed network connection Jun 21 20:11:05.567932150 (75h47m41.131255161s) [seg_pos={0 0}, upload=60693]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jun 21 20:11:05.593626175 (75h47m41.156949181s) [upload=60693]: Done uploading segments... Jun 21 20:11:05.593657274 (75h47m41.156980265s) [upload=60693]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 26.944340749s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [...................X...........X............................] duration: 24.944617218s FAIL (2025-06-21 20:34:25 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== pload=60693]: Starting upload Jun 21 20:10:05.602866542 (75h46m41.166189557s) [upload=60693]: Done starting upload Jun 21 20:10:05.603116545 (75h46m41.166439547s) [upload=60693]: Uploading segments... Jun 21 20:10:05.603495389 (75h46m41.166818397s) [upload=60693]: Got next segment. Inline: false Jun 21 20:10:05.603566566 (75h46m41.166889558s) [upload=60693]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 21 20:10:05.603573280 (75h46m41.166896286s) [upload=60693]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 21 20:10:05.639130162 (75h46m41.202453164s) [seg_pos={0 0}, upload=60693]: Begin upload segment... Jun 21 20:10:05.639231070 (75h46m41.202554057s) [seg_pos={0 0}, upload=60693]: Done begin upload segment. Jun 21 20:10:05.639292724 (75h46m41.202615728s) [noise=true , address=104.36.82.56:4253 , piece_id=5PMVGLZETMXCQ77K3I6PXACRIMBGR2CV5JGUSU5KUMYFNXVGA2GA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.639338066 (75h46m41.202661071s) [noise=true , address=104.36.82.53:4126 , piece_id=2MKRQJEPGDJKDM7VWY5M4M6NORILAIEYYATJ7HVZEOBOMXOI2LBA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.639416264 (75h46m41.202739267s) [noise=true , address=104.36.82.5:5869 , piece_id=HAEHXXZASV3DHT3QN7NTZOS6XZKHNUWBKCHWJJWSTJKXAKJKKBAA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.639873098 (75h46m41.203196115s) [noise=true , address=104.36.82.52:4111 , piece_id=HB7QAWN6CGNIIT4HD2GZ7JDQP5NXIJ5I2USBM66YOKTYENNOWG5A, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.639887296 (75h46m41.203210304s) [noise=true , address=104.36.82.10:6087 , piece_id=5FQG7BPZC373PFB54HYXSG2SVKOBBBSGZCLC4XDJEYQFOLQY74MA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.640266623 (75h46m41.203589616s) [noise=true , address=104.36.82.13:6196 , piece_id=BCHMQO5KKT5BLNWQGOHH3V43LGSE3BUQCI2SY4TTO5M5GQNOCTMQ, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.640352274 (75h46m41.203675286s) [noise=true , address=104.36.82.37:6663 , piece_id=X7LWYIIHJFU2T2PHYR3R5TNT22UUPHEHQWZH7ZJMLS4AH4PJ6AIA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.640659767 (75h46m41.203982779s) [noise=true , address=104.36.82.42:6807 , piece_id=ZBQZPXUDJ4VBCOSVKWAJ6SK6UD7SLPON7Y4SSCK2DQ5STFG4XCNQ, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.640961701 (75h46m41.20428471s) [noise=true , address=45.91.116.6:4982 , piece_id=VP45R6KXNH5U5SGUQUMAPIBY7KFDPBIR3S3VRIX27EAT6VLFBZ7Q, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.641159578 (75h46m41.204482589s) [noise=true , address=104.36.82.9:6039 , piece_id=I4AB6N27MYYUHWAZLJJUIBFBLOQFHDIM3YLYWPVRPNUV7VMYS2SA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.641588126 (75h46m41.204911136s) [noise=true , address=77.90.31.19:4853 , piece_id=VFEON5T2KSWM2KSLTVRZTXYL42JWENZYUEUPG3LMODGZWBMVGGPQ, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.641863297 (75h46m41.205186305s) [noise=true , address=77.90.31.12:4603 , piece_id=BMUIC6KNGEDL37436GLSXW5XOO6VYOPNOSFGQCYVHSENVI2PJ7KQ, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.642124427 (75h46m41.205447437s) [noise=true , address=104.36.82.49:4982 , piece_id=UNLV5UDNWULEIPE4QSQHHZQPK3BQKFRB5T3H6GP7ZQ5QGSNK5V6A, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.642285917 (75h46m41.205608925s) [noise=true , address=104.36.82.54:4171 , piece_id=MUMABKARK4KKUKWW6XDXCQVT6O7SLFHGHE7COWC7QIYGARAA2D5A, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.642656750 (75h46m41.205979758s) [noise=true , address=104.36.82.14:6232 , piece_id=A2ZLUUZKRTBJRJ267QSK36C3LFZV7FPAYDDAXGIWDTNEGNV6MRRA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.643565583 (75h46m41.206888597s) [noise=true , address=79.127.205.245:4231 , piece_id=EJK46XUUVI5NWOAQJEW4QIJG3KRXZHB4YOQUJRCS2MIOSWKGTNLQ, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.644620291 (75h46m41.207943303s) [noise=true , address=77.90.31.14:4670 , piece_id=7FNANMUKPM3N6YWRAOAIM4WJZJUTNF7ZBI6ZUBNEPGNGMFWOE24A, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.648842639 (75h46m41.212165631s) [noise=true , address=104.36.82.55:4224 , piece_id=WRVNGSJV5BX4GJERLXNPAS7J7BZ5WFTM7R5H257YRCG5SP2HIYMA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.825828239 (75h46m41.389151241s) [noise=true , address=104.36.82.53:4126 , piece_id=2MKRQJEPGDJKDM7VWY5M4M6NORILAIEYYATJ7HVZEOBOMXOI2LBA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: ecclient: upload failed (node:12dMWFrq2NMpnMzuPAphWerntC5kkrcnzRJZBA116uzJzyvk6fH, address:104.36.82.53:4126): protocol: storage node overloaded, request limit: 2000 Jun 21 20:10:05.834666043 (75h46m41.397989033s) [upload=60693]: Next returned nil segment Jun 21 20:10:05.834686295 (75h46m41.398009302s) [upload=60693]: Waiting for error group managing segments... Jun 21 20:10:05.877010488 (75h46m41.440333486s) [noise=true , address=45.91.116.6:4982 , piece_id=VP45R6KXNH5U5SGUQUMAPIBY7KFDPBIR3S3VRIX27EAT6VLFBZ7Q, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:05.886336964 (75h46m41.449659967s) [noise=true , address=104.36.82.9:6039 , piece_id=I4AB6N27MYYUHWAZLJJUIBFBLOQFHDIM3YLYWPVRPNUV7VMYS2SA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: ecclient: upload failed (node:12P11WVW9mVXw5ra2gZ3Rezon6wZi8GrN5CchQN3MiVU4ts3L6V, address:104.36.82.9:6039): protocol: storage node overloaded, request limit: 2000 Jun 21 20:10:08.064514542 (75h46m43.627837553s) [noise=true , address=77.90.31.14:4670 , piece_id=7FNANMUKPM3N6YWRAOAIM4WJZJUTNF7ZBI6ZUBNEPGNGMFWOE24A, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:08.099098391 (75h46m43.662421396s) [noise=true , address=79.127.205.245:4231 , piece_id=EJK46XUUVI5NWOAQJEW4QIJG3KRXZHB4YOQUJRCS2MIOSWKGTNLQ, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:08.489461749 (75h46m44.052784752s) [noise=true , address=104.36.82.42:6807 , piece_id=ZBQZPXUDJ4VBCOSVKWAJ6SK6UD7SLPON7Y4SSCK2DQ5STFG4XCNQ, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:08.650922093 (75h46m44.214245093s) [noise=true , address=77.90.31.19:4853 , piece_id=VFEON5T2KSWM2KSLTVRZTXYL42JWENZYUEUPG3LMODGZWBMVGGPQ, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:09.917746439 (75h46m45.481069441s) [noise=true , address=104.36.82.13:6196 , piece_id=BCHMQO5KKT5BLNWQGOHH3V43LGSE3BUQCI2SY4TTO5M5GQNOCTMQ, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:10.830722215 (75h46m46.394045216s) [noise=true , address=104.36.82.5:5869 , piece_id=HAEHXXZASV3DHT3QN7NTZOS6XZKHNUWBKCHWJJWSTJKXAKJKKBAA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:12.738783450 (75h46m48.302106452s) [noise=true , address=77.90.31.12:4603 , piece_id=BMUIC6KNGEDL37436GLSXW5XOO6VYOPNOSFGQCYVHSENVI2PJ7KQ, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:15.460927713 (75h46m51.024250723s) [noise=true , address=104.36.82.52:4111 , piece_id=HB7QAWN6CGNIIT4HD2GZ7JDQP5NXIJ5I2USBM66YOKTYENNOWG5A, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:18.393817968 (75h46m53.95714096s) [noise=true , address=104.36.82.55:4224 , piece_id=WRVNGSJV5BX4GJERLXNPAS7J7BZ5WFTM7R5H257YRCG5SP2HIYMA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:19.549174143 (75h46m55.112497151s) [noise=true , address=104.36.82.56:4253 , piece_id=5PMVGLZETMXCQ77K3I6PXACRIMBGR2CV5JGUSU5KUMYFNXVGA2GA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:22.511944593 (75h46m58.075267584s) [noise=true , address=104.36.82.37:6663 , piece_id=X7LWYIIHJFU2T2PHYR3R5TNT22UUPHEHQWZH7ZJMLS4AH4PJ6AIA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:48.027197558 (75h47m23.590520562s) [noise=true , address=104.36.82.14:6232 , piece_id=A2ZLUUZKRTBJRJ267QSK36C3LFZV7FPAYDDAXGIWDTNEGNV6MRRA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:11:05.567692582 (75h47m41.131015574s) [noise=true , address=104.36.82.49:4982 , piece_id=UNLV5UDNWULEIPE4QSQHHZQPK3BQKFRB5T3H6GP7ZQ5QGSNK5V6A, seg_pos={0 0}, upload=60693]: Done uploading piece. err: ecclient: upload failed (node:1NxoYJXCXmTvQncbkXt6yH7mhEBBrzVdzhgL5J6W5TD4N3pkaG, address:104.36.82.49:4982): protocol: expected piece hash; context deadline exceeded; EOF Jun 21 20:11:05.567759198 (75h47m41.131082195s) [noise=true , address=104.36.82.54:4171 , piece_id=MUMABKARK4KKUKWW6XDXCQVT6O7SLFHGHE7COWC7QIYGARAA2D5A, seg_pos={0 0}, upload=60693]: Done uploading piece. err: ecclient: upload failed (node:125SMhDsEZruNu6LkmQn2M7R6WzvXTdk86DRsKf522u8BNiANq3, address:104.36.82.54:4171): protocol: expected piece hash; context deadline exceeded; EOF Jun 21 20:11:05.567869019 (75h47m41.131192033s) [noise=true , address=104.36.82.10:6087 , piece_id=5FQG7BPZC373PFB54HYXSG2SVKOBBBSGZCLC4XDJEYQFOLQY74MA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: ecclient: upload failed (node:1S9k5eejJcxkiG5pXjjBZm18uP8k5jCGfWyeW2QQxmuRU8ZzRA, address:104.36.82.10:6087): protocol: write tcp 159.223.143.39:38550->104.36.82.10:6087: use of closed network connection; write tcp 159.223.143.39:38550->104.36.82.10:6087: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:38550->104.36.82.10:6087: use of closed network connection Jun 21 20:11:05.567932150 (75h47m41.131255161s) [seg_pos={0 0}, upload=60693]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jun 21 20:11:05.593626175 (75h47m41.156949181s) [upload=60693]: Done uploading segments... Jun 21 20:11:05.593657274 (75h47m41.156980265s) [upload=60693]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 24.944707978s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [....................X...........X...........................] duration: 15.845143985s FAIL (2025-06-21 20:33:48 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== pload=60693]: Starting upload Jun 21 20:10:05.602866542 (75h46m41.166189557s) [upload=60693]: Done starting upload Jun 21 20:10:05.603116545 (75h46m41.166439547s) [upload=60693]: Uploading segments... Jun 21 20:10:05.603495389 (75h46m41.166818397s) [upload=60693]: Got next segment. Inline: false Jun 21 20:10:05.603566566 (75h46m41.166889558s) [upload=60693]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 21 20:10:05.603573280 (75h46m41.166896286s) [upload=60693]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 21 20:10:05.639130162 (75h46m41.202453164s) [seg_pos={0 0}, upload=60693]: Begin upload segment... Jun 21 20:10:05.639231070 (75h46m41.202554057s) [seg_pos={0 0}, upload=60693]: Done begin upload segment. Jun 21 20:10:05.639292724 (75h46m41.202615728s) [noise=true , address=104.36.82.56:4253 , piece_id=5PMVGLZETMXCQ77K3I6PXACRIMBGR2CV5JGUSU5KUMYFNXVGA2GA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.639338066 (75h46m41.202661071s) [noise=true , address=104.36.82.53:4126 , piece_id=2MKRQJEPGDJKDM7VWY5M4M6NORILAIEYYATJ7HVZEOBOMXOI2LBA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.639416264 (75h46m41.202739267s) [noise=true , address=104.36.82.5:5869 , piece_id=HAEHXXZASV3DHT3QN7NTZOS6XZKHNUWBKCHWJJWSTJKXAKJKKBAA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.639873098 (75h46m41.203196115s) [noise=true , address=104.36.82.52:4111 , piece_id=HB7QAWN6CGNIIT4HD2GZ7JDQP5NXIJ5I2USBM66YOKTYENNOWG5A, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.639887296 (75h46m41.203210304s) [noise=true , address=104.36.82.10:6087 , piece_id=5FQG7BPZC373PFB54HYXSG2SVKOBBBSGZCLC4XDJEYQFOLQY74MA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.640266623 (75h46m41.203589616s) [noise=true , address=104.36.82.13:6196 , piece_id=BCHMQO5KKT5BLNWQGOHH3V43LGSE3BUQCI2SY4TTO5M5GQNOCTMQ, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.640352274 (75h46m41.203675286s) [noise=true , address=104.36.82.37:6663 , piece_id=X7LWYIIHJFU2T2PHYR3R5TNT22UUPHEHQWZH7ZJMLS4AH4PJ6AIA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.640659767 (75h46m41.203982779s) [noise=true , address=104.36.82.42:6807 , piece_id=ZBQZPXUDJ4VBCOSVKWAJ6SK6UD7SLPON7Y4SSCK2DQ5STFG4XCNQ, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.640961701 (75h46m41.20428471s) [noise=true , address=45.91.116.6:4982 , piece_id=VP45R6KXNH5U5SGUQUMAPIBY7KFDPBIR3S3VRIX27EAT6VLFBZ7Q, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.641159578 (75h46m41.204482589s) [noise=true , address=104.36.82.9:6039 , piece_id=I4AB6N27MYYUHWAZLJJUIBFBLOQFHDIM3YLYWPVRPNUV7VMYS2SA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.641588126 (75h46m41.204911136s) [noise=true , address=77.90.31.19:4853 , piece_id=VFEON5T2KSWM2KSLTVRZTXYL42JWENZYUEUPG3LMODGZWBMVGGPQ, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.641863297 (75h46m41.205186305s) [noise=true , address=77.90.31.12:4603 , piece_id=BMUIC6KNGEDL37436GLSXW5XOO6VYOPNOSFGQCYVHSENVI2PJ7KQ, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.642124427 (75h46m41.205447437s) [noise=true , address=104.36.82.49:4982 , piece_id=UNLV5UDNWULEIPE4QSQHHZQPK3BQKFRB5T3H6GP7ZQ5QGSNK5V6A, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.642285917 (75h46m41.205608925s) [noise=true , address=104.36.82.54:4171 , piece_id=MUMABKARK4KKUKWW6XDXCQVT6O7SLFHGHE7COWC7QIYGARAA2D5A, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.642656750 (75h46m41.205979758s) [noise=true , address=104.36.82.14:6232 , piece_id=A2ZLUUZKRTBJRJ267QSK36C3LFZV7FPAYDDAXGIWDTNEGNV6MRRA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.643565583 (75h46m41.206888597s) [noise=true , address=79.127.205.245:4231 , piece_id=EJK46XUUVI5NWOAQJEW4QIJG3KRXZHB4YOQUJRCS2MIOSWKGTNLQ, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.644620291 (75h46m41.207943303s) [noise=true , address=77.90.31.14:4670 , piece_id=7FNANMUKPM3N6YWRAOAIM4WJZJUTNF7ZBI6ZUBNEPGNGMFWOE24A, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.648842639 (75h46m41.212165631s) [noise=true , address=104.36.82.55:4224 , piece_id=WRVNGSJV5BX4GJERLXNPAS7J7BZ5WFTM7R5H257YRCG5SP2HIYMA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.825828239 (75h46m41.389151241s) [noise=true , address=104.36.82.53:4126 , piece_id=2MKRQJEPGDJKDM7VWY5M4M6NORILAIEYYATJ7HVZEOBOMXOI2LBA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: ecclient: upload failed (node:12dMWFrq2NMpnMzuPAphWerntC5kkrcnzRJZBA116uzJzyvk6fH, address:104.36.82.53:4126): protocol: storage node overloaded, request limit: 2000 Jun 21 20:10:05.834666043 (75h46m41.397989033s) [upload=60693]: Next returned nil segment Jun 21 20:10:05.834686295 (75h46m41.398009302s) [upload=60693]: Waiting for error group managing segments... Jun 21 20:10:05.877010488 (75h46m41.440333486s) [noise=true , address=45.91.116.6:4982 , piece_id=VP45R6KXNH5U5SGUQUMAPIBY7KFDPBIR3S3VRIX27EAT6VLFBZ7Q, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:05.886336964 (75h46m41.449659967s) [noise=true , address=104.36.82.9:6039 , piece_id=I4AB6N27MYYUHWAZLJJUIBFBLOQFHDIM3YLYWPVRPNUV7VMYS2SA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: ecclient: upload failed (node:12P11WVW9mVXw5ra2gZ3Rezon6wZi8GrN5CchQN3MiVU4ts3L6V, address:104.36.82.9:6039): protocol: storage node overloaded, request limit: 2000 Jun 21 20:10:08.064514542 (75h46m43.627837553s) [noise=true , address=77.90.31.14:4670 , piece_id=7FNANMUKPM3N6YWRAOAIM4WJZJUTNF7ZBI6ZUBNEPGNGMFWOE24A, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:08.099098391 (75h46m43.662421396s) [noise=true , address=79.127.205.245:4231 , piece_id=EJK46XUUVI5NWOAQJEW4QIJG3KRXZHB4YOQUJRCS2MIOSWKGTNLQ, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:08.489461749 (75h46m44.052784752s) [noise=true , address=104.36.82.42:6807 , piece_id=ZBQZPXUDJ4VBCOSVKWAJ6SK6UD7SLPON7Y4SSCK2DQ5STFG4XCNQ, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:08.650922093 (75h46m44.214245093s) [noise=true , address=77.90.31.19:4853 , piece_id=VFEON5T2KSWM2KSLTVRZTXYL42JWENZYUEUPG3LMODGZWBMVGGPQ, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:09.917746439 (75h46m45.481069441s) [noise=true , address=104.36.82.13:6196 , piece_id=BCHMQO5KKT5BLNWQGOHH3V43LGSE3BUQCI2SY4TTO5M5GQNOCTMQ, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:10.830722215 (75h46m46.394045216s) [noise=true , address=104.36.82.5:5869 , piece_id=HAEHXXZASV3DHT3QN7NTZOS6XZKHNUWBKCHWJJWSTJKXAKJKKBAA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:12.738783450 (75h46m48.302106452s) [noise=true , address=77.90.31.12:4603 , piece_id=BMUIC6KNGEDL37436GLSXW5XOO6VYOPNOSFGQCYVHSENVI2PJ7KQ, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:15.460927713 (75h46m51.024250723s) [noise=true , address=104.36.82.52:4111 , piece_id=HB7QAWN6CGNIIT4HD2GZ7JDQP5NXIJ5I2USBM66YOKTYENNOWG5A, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:18.393817968 (75h46m53.95714096s) [noise=true , address=104.36.82.55:4224 , piece_id=WRVNGSJV5BX4GJERLXNPAS7J7BZ5WFTM7R5H257YRCG5SP2HIYMA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:19.549174143 (75h46m55.112497151s) [noise=true , address=104.36.82.56:4253 , piece_id=5PMVGLZETMXCQ77K3I6PXACRIMBGR2CV5JGUSU5KUMYFNXVGA2GA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:22.511944593 (75h46m58.075267584s) [noise=true , address=104.36.82.37:6663 , piece_id=X7LWYIIHJFU2T2PHYR3R5TNT22UUPHEHQWZH7ZJMLS4AH4PJ6AIA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:48.027197558 (75h47m23.590520562s) [noise=true , address=104.36.82.14:6232 , piece_id=A2ZLUUZKRTBJRJ267QSK36C3LFZV7FPAYDDAXGIWDTNEGNV6MRRA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:11:05.567692582 (75h47m41.131015574s) [noise=true , address=104.36.82.49:4982 , piece_id=UNLV5UDNWULEIPE4QSQHHZQPK3BQKFRB5T3H6GP7ZQ5QGSNK5V6A, seg_pos={0 0}, upload=60693]: Done uploading piece. err: ecclient: upload failed (node:1NxoYJXCXmTvQncbkXt6yH7mhEBBrzVdzhgL5J6W5TD4N3pkaG, address:104.36.82.49:4982): protocol: expected piece hash; context deadline exceeded; EOF Jun 21 20:11:05.567759198 (75h47m41.131082195s) [noise=true , address=104.36.82.54:4171 , piece_id=MUMABKARK4KKUKWW6XDXCQVT6O7SLFHGHE7COWC7QIYGARAA2D5A, seg_pos={0 0}, upload=60693]: Done uploading piece. err: ecclient: upload failed (node:125SMhDsEZruNu6LkmQn2M7R6WzvXTdk86DRsKf522u8BNiANq3, address:104.36.82.54:4171): protocol: expected piece hash; context deadline exceeded; EOF Jun 21 20:11:05.567869019 (75h47m41.131192033s) [noise=true , address=104.36.82.10:6087 , piece_id=5FQG7BPZC373PFB54HYXSG2SVKOBBBSGZCLC4XDJEYQFOLQY74MA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: ecclient: upload failed (node:1S9k5eejJcxkiG5pXjjBZm18uP8k5jCGfWyeW2QQxmuRU8ZzRA, address:104.36.82.10:6087): protocol: write tcp 159.223.143.39:38550->104.36.82.10:6087: use of closed network connection; write tcp 159.223.143.39:38550->104.36.82.10:6087: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:38550->104.36.82.10:6087: use of closed network connection Jun 21 20:11:05.567932150 (75h47m41.131255161s) [seg_pos={0 0}, upload=60693]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jun 21 20:11:05.593626175 (75h47m41.156949181s) [upload=60693]: Done uploading segments... Jun 21 20:11:05.593657274 (75h47m41.156980265s) [upload=60693]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 15.845172011s core uplink us-select-1 us1-placement-20

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)