Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [........X...................................................] duration: 22.628425475s FAIL (2025-06-21 20:49:02 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== pload=60693]: Starting upload Jun 21 20:10:05.602866542 (75h46m41.166189557s) [upload=60693]: Done starting upload Jun 21 20:10:05.603116545 (75h46m41.166439547s) [upload=60693]: Uploading segments... Jun 21 20:10:05.603495389 (75h46m41.166818397s) [upload=60693]: Got next segment. Inline: false Jun 21 20:10:05.603566566 (75h46m41.166889558s) [upload=60693]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 21 20:10:05.603573280 (75h46m41.166896286s) [upload=60693]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 21 20:10:05.639130162 (75h46m41.202453164s) [seg_pos={0 0}, upload=60693]: Begin upload segment... Jun 21 20:10:05.639231070 (75h46m41.202554057s) [seg_pos={0 0}, upload=60693]: Done begin upload segment. Jun 21 20:10:05.639292724 (75h46m41.202615728s) [noise=true , address=104.36.82.56:4253 , piece_id=5PMVGLZETMXCQ77K3I6PXACRIMBGR2CV5JGUSU5KUMYFNXVGA2GA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.639338066 (75h46m41.202661071s) [noise=true , address=104.36.82.53:4126 , piece_id=2MKRQJEPGDJKDM7VWY5M4M6NORILAIEYYATJ7HVZEOBOMXOI2LBA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.639416264 (75h46m41.202739267s) [noise=true , address=104.36.82.5:5869 , piece_id=HAEHXXZASV3DHT3QN7NTZOS6XZKHNUWBKCHWJJWSTJKXAKJKKBAA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.639873098 (75h46m41.203196115s) [noise=true , address=104.36.82.52:4111 , piece_id=HB7QAWN6CGNIIT4HD2GZ7JDQP5NXIJ5I2USBM66YOKTYENNOWG5A, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.639887296 (75h46m41.203210304s) [noise=true , address=104.36.82.10:6087 , piece_id=5FQG7BPZC373PFB54HYXSG2SVKOBBBSGZCLC4XDJEYQFOLQY74MA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.640266623 (75h46m41.203589616s) [noise=true , address=104.36.82.13:6196 , piece_id=BCHMQO5KKT5BLNWQGOHH3V43LGSE3BUQCI2SY4TTO5M5GQNOCTMQ, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.640352274 (75h46m41.203675286s) [noise=true , address=104.36.82.37:6663 , piece_id=X7LWYIIHJFU2T2PHYR3R5TNT22UUPHEHQWZH7ZJMLS4AH4PJ6AIA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.640659767 (75h46m41.203982779s) [noise=true , address=104.36.82.42:6807 , piece_id=ZBQZPXUDJ4VBCOSVKWAJ6SK6UD7SLPON7Y4SSCK2DQ5STFG4XCNQ, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.640961701 (75h46m41.20428471s) [noise=true , address=45.91.116.6:4982 , piece_id=VP45R6KXNH5U5SGUQUMAPIBY7KFDPBIR3S3VRIX27EAT6VLFBZ7Q, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.641159578 (75h46m41.204482589s) [noise=true , address=104.36.82.9:6039 , piece_id=I4AB6N27MYYUHWAZLJJUIBFBLOQFHDIM3YLYWPVRPNUV7VMYS2SA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.641588126 (75h46m41.204911136s) [noise=true , address=77.90.31.19:4853 , piece_id=VFEON5T2KSWM2KSLTVRZTXYL42JWENZYUEUPG3LMODGZWBMVGGPQ, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.641863297 (75h46m41.205186305s) [noise=true , address=77.90.31.12:4603 , piece_id=BMUIC6KNGEDL37436GLSXW5XOO6VYOPNOSFGQCYVHSENVI2PJ7KQ, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.642124427 (75h46m41.205447437s) [noise=true , address=104.36.82.49:4982 , piece_id=UNLV5UDNWULEIPE4QSQHHZQPK3BQKFRB5T3H6GP7ZQ5QGSNK5V6A, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.642285917 (75h46m41.205608925s) [noise=true , address=104.36.82.54:4171 , piece_id=MUMABKARK4KKUKWW6XDXCQVT6O7SLFHGHE7COWC7QIYGARAA2D5A, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.642656750 (75h46m41.205979758s) [noise=true , address=104.36.82.14:6232 , piece_id=A2ZLUUZKRTBJRJ267QSK36C3LFZV7FPAYDDAXGIWDTNEGNV6MRRA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.643565583 (75h46m41.206888597s) [noise=true , address=79.127.205.245:4231 , piece_id=EJK46XUUVI5NWOAQJEW4QIJG3KRXZHB4YOQUJRCS2MIOSWKGTNLQ, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.644620291 (75h46m41.207943303s) [noise=true , address=77.90.31.14:4670 , piece_id=7FNANMUKPM3N6YWRAOAIM4WJZJUTNF7ZBI6ZUBNEPGNGMFWOE24A, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.648842639 (75h46m41.212165631s) [noise=true , address=104.36.82.55:4224 , piece_id=WRVNGSJV5BX4GJERLXNPAS7J7BZ5WFTM7R5H257YRCG5SP2HIYMA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.825828239 (75h46m41.389151241s) [noise=true , address=104.36.82.53:4126 , piece_id=2MKRQJEPGDJKDM7VWY5M4M6NORILAIEYYATJ7HVZEOBOMXOI2LBA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: ecclient: upload failed (node:12dMWFrq2NMpnMzuPAphWerntC5kkrcnzRJZBA116uzJzyvk6fH, address:104.36.82.53:4126): protocol: storage node overloaded, request limit: 2000 Jun 21 20:10:05.834666043 (75h46m41.397989033s) [upload=60693]: Next returned nil segment Jun 21 20:10:05.834686295 (75h46m41.398009302s) [upload=60693]: Waiting for error group managing segments... Jun 21 20:10:05.877010488 (75h46m41.440333486s) [noise=true , address=45.91.116.6:4982 , piece_id=VP45R6KXNH5U5SGUQUMAPIBY7KFDPBIR3S3VRIX27EAT6VLFBZ7Q, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:05.886336964 (75h46m41.449659967s) [noise=true , address=104.36.82.9:6039 , piece_id=I4AB6N27MYYUHWAZLJJUIBFBLOQFHDIM3YLYWPVRPNUV7VMYS2SA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: ecclient: upload failed (node:12P11WVW9mVXw5ra2gZ3Rezon6wZi8GrN5CchQN3MiVU4ts3L6V, address:104.36.82.9:6039): protocol: storage node overloaded, request limit: 2000 Jun 21 20:10:08.064514542 (75h46m43.627837553s) [noise=true , address=77.90.31.14:4670 , piece_id=7FNANMUKPM3N6YWRAOAIM4WJZJUTNF7ZBI6ZUBNEPGNGMFWOE24A, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:08.099098391 (75h46m43.662421396s) [noise=true , address=79.127.205.245:4231 , piece_id=EJK46XUUVI5NWOAQJEW4QIJG3KRXZHB4YOQUJRCS2MIOSWKGTNLQ, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:08.489461749 (75h46m44.052784752s) [noise=true , address=104.36.82.42:6807 , piece_id=ZBQZPXUDJ4VBCOSVKWAJ6SK6UD7SLPON7Y4SSCK2DQ5STFG4XCNQ, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:08.650922093 (75h46m44.214245093s) [noise=true , address=77.90.31.19:4853 , piece_id=VFEON5T2KSWM2KSLTVRZTXYL42JWENZYUEUPG3LMODGZWBMVGGPQ, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:09.917746439 (75h46m45.481069441s) [noise=true , address=104.36.82.13:6196 , piece_id=BCHMQO5KKT5BLNWQGOHH3V43LGSE3BUQCI2SY4TTO5M5GQNOCTMQ, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:10.830722215 (75h46m46.394045216s) [noise=true , address=104.36.82.5:5869 , piece_id=HAEHXXZASV3DHT3QN7NTZOS6XZKHNUWBKCHWJJWSTJKXAKJKKBAA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:12.738783450 (75h46m48.302106452s) [noise=true , address=77.90.31.12:4603 , piece_id=BMUIC6KNGEDL37436GLSXW5XOO6VYOPNOSFGQCYVHSENVI2PJ7KQ, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:15.460927713 (75h46m51.024250723s) [noise=true , address=104.36.82.52:4111 , piece_id=HB7QAWN6CGNIIT4HD2GZ7JDQP5NXIJ5I2USBM66YOKTYENNOWG5A, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:18.393817968 (75h46m53.95714096s) [noise=true , address=104.36.82.55:4224 , piece_id=WRVNGSJV5BX4GJERLXNPAS7J7BZ5WFTM7R5H257YRCG5SP2HIYMA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:19.549174143 (75h46m55.112497151s) [noise=true , address=104.36.82.56:4253 , piece_id=5PMVGLZETMXCQ77K3I6PXACRIMBGR2CV5JGUSU5KUMYFNXVGA2GA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:22.511944593 (75h46m58.075267584s) [noise=true , address=104.36.82.37:6663 , piece_id=X7LWYIIHJFU2T2PHYR3R5TNT22UUPHEHQWZH7ZJMLS4AH4PJ6AIA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:48.027197558 (75h47m23.590520562s) [noise=true , address=104.36.82.14:6232 , piece_id=A2ZLUUZKRTBJRJ267QSK36C3LFZV7FPAYDDAXGIWDTNEGNV6MRRA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:11:05.567692582 (75h47m41.131015574s) [noise=true , address=104.36.82.49:4982 , piece_id=UNLV5UDNWULEIPE4QSQHHZQPK3BQKFRB5T3H6GP7ZQ5QGSNK5V6A, seg_pos={0 0}, upload=60693]: Done uploading piece. err: ecclient: upload failed (node:1NxoYJXCXmTvQncbkXt6yH7mhEBBrzVdzhgL5J6W5TD4N3pkaG, address:104.36.82.49:4982): protocol: expected piece hash; context deadline exceeded; EOF Jun 21 20:11:05.567759198 (75h47m41.131082195s) [noise=true , address=104.36.82.54:4171 , piece_id=MUMABKARK4KKUKWW6XDXCQVT6O7SLFHGHE7COWC7QIYGARAA2D5A, seg_pos={0 0}, upload=60693]: Done uploading piece. err: ecclient: upload failed (node:125SMhDsEZruNu6LkmQn2M7R6WzvXTdk86DRsKf522u8BNiANq3, address:104.36.82.54:4171): protocol: expected piece hash; context deadline exceeded; EOF Jun 21 20:11:05.567869019 (75h47m41.131192033s) [noise=true , address=104.36.82.10:6087 , piece_id=5FQG7BPZC373PFB54HYXSG2SVKOBBBSGZCLC4XDJEYQFOLQY74MA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: ecclient: upload failed (node:1S9k5eejJcxkiG5pXjjBZm18uP8k5jCGfWyeW2QQxmuRU8ZzRA, address:104.36.82.10:6087): protocol: write tcp 159.223.143.39:38550->104.36.82.10:6087: use of closed network connection; write tcp 159.223.143.39:38550->104.36.82.10:6087: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:38550->104.36.82.10:6087: use of closed network connection Jun 21 20:11:05.567932150 (75h47m41.131255161s) [seg_pos={0 0}, upload=60693]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jun 21 20:11:05.593626175 (75h47m41.156949181s) [upload=60693]: Done uploading segments... Jun 21 20:11:05.593657274 (75h47m41.156980265s) [upload=60693]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 22.628477319s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.........X..................................................] duration: 31.303766251s FAIL (2025-06-21 20:48:39 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== pload=60693]: Starting upload Jun 21 20:10:05.602866542 (75h46m41.166189557s) [upload=60693]: Done starting upload Jun 21 20:10:05.603116545 (75h46m41.166439547s) [upload=60693]: Uploading segments... Jun 21 20:10:05.603495389 (75h46m41.166818397s) [upload=60693]: Got next segment. Inline: false Jun 21 20:10:05.603566566 (75h46m41.166889558s) [upload=60693]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 21 20:10:05.603573280 (75h46m41.166896286s) [upload=60693]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 21 20:10:05.639130162 (75h46m41.202453164s) [seg_pos={0 0}, upload=60693]: Begin upload segment... Jun 21 20:10:05.639231070 (75h46m41.202554057s) [seg_pos={0 0}, upload=60693]: Done begin upload segment. Jun 21 20:10:05.639292724 (75h46m41.202615728s) [noise=true , address=104.36.82.56:4253 , piece_id=5PMVGLZETMXCQ77K3I6PXACRIMBGR2CV5JGUSU5KUMYFNXVGA2GA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.639338066 (75h46m41.202661071s) [noise=true , address=104.36.82.53:4126 , piece_id=2MKRQJEPGDJKDM7VWY5M4M6NORILAIEYYATJ7HVZEOBOMXOI2LBA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.639416264 (75h46m41.202739267s) [noise=true , address=104.36.82.5:5869 , piece_id=HAEHXXZASV3DHT3QN7NTZOS6XZKHNUWBKCHWJJWSTJKXAKJKKBAA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.639873098 (75h46m41.203196115s) [noise=true , address=104.36.82.52:4111 , piece_id=HB7QAWN6CGNIIT4HD2GZ7JDQP5NXIJ5I2USBM66YOKTYENNOWG5A, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.639887296 (75h46m41.203210304s) [noise=true , address=104.36.82.10:6087 , piece_id=5FQG7BPZC373PFB54HYXSG2SVKOBBBSGZCLC4XDJEYQFOLQY74MA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.640266623 (75h46m41.203589616s) [noise=true , address=104.36.82.13:6196 , piece_id=BCHMQO5KKT5BLNWQGOHH3V43LGSE3BUQCI2SY4TTO5M5GQNOCTMQ, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.640352274 (75h46m41.203675286s) [noise=true , address=104.36.82.37:6663 , piece_id=X7LWYIIHJFU2T2PHYR3R5TNT22UUPHEHQWZH7ZJMLS4AH4PJ6AIA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.640659767 (75h46m41.203982779s) [noise=true , address=104.36.82.42:6807 , piece_id=ZBQZPXUDJ4VBCOSVKWAJ6SK6UD7SLPON7Y4SSCK2DQ5STFG4XCNQ, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.640961701 (75h46m41.20428471s) [noise=true , address=45.91.116.6:4982 , piece_id=VP45R6KXNH5U5SGUQUMAPIBY7KFDPBIR3S3VRIX27EAT6VLFBZ7Q, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.641159578 (75h46m41.204482589s) [noise=true , address=104.36.82.9:6039 , piece_id=I4AB6N27MYYUHWAZLJJUIBFBLOQFHDIM3YLYWPVRPNUV7VMYS2SA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.641588126 (75h46m41.204911136s) [noise=true , address=77.90.31.19:4853 , piece_id=VFEON5T2KSWM2KSLTVRZTXYL42JWENZYUEUPG3LMODGZWBMVGGPQ, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.641863297 (75h46m41.205186305s) [noise=true , address=77.90.31.12:4603 , piece_id=BMUIC6KNGEDL37436GLSXW5XOO6VYOPNOSFGQCYVHSENVI2PJ7KQ, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.642124427 (75h46m41.205447437s) [noise=true , address=104.36.82.49:4982 , piece_id=UNLV5UDNWULEIPE4QSQHHZQPK3BQKFRB5T3H6GP7ZQ5QGSNK5V6A, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.642285917 (75h46m41.205608925s) [noise=true , address=104.36.82.54:4171 , piece_id=MUMABKARK4KKUKWW6XDXCQVT6O7SLFHGHE7COWC7QIYGARAA2D5A, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.642656750 (75h46m41.205979758s) [noise=true , address=104.36.82.14:6232 , piece_id=A2ZLUUZKRTBJRJ267QSK36C3LFZV7FPAYDDAXGIWDTNEGNV6MRRA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.643565583 (75h46m41.206888597s) [noise=true , address=79.127.205.245:4231 , piece_id=EJK46XUUVI5NWOAQJEW4QIJG3KRXZHB4YOQUJRCS2MIOSWKGTNLQ, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.644620291 (75h46m41.207943303s) [noise=true , address=77.90.31.14:4670 , piece_id=7FNANMUKPM3N6YWRAOAIM4WJZJUTNF7ZBI6ZUBNEPGNGMFWOE24A, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.648842639 (75h46m41.212165631s) [noise=true , address=104.36.82.55:4224 , piece_id=WRVNGSJV5BX4GJERLXNPAS7J7BZ5WFTM7R5H257YRCG5SP2HIYMA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.825828239 (75h46m41.389151241s) [noise=true , address=104.36.82.53:4126 , piece_id=2MKRQJEPGDJKDM7VWY5M4M6NORILAIEYYATJ7HVZEOBOMXOI2LBA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: ecclient: upload failed (node:12dMWFrq2NMpnMzuPAphWerntC5kkrcnzRJZBA116uzJzyvk6fH, address:104.36.82.53:4126): protocol: storage node overloaded, request limit: 2000 Jun 21 20:10:05.834666043 (75h46m41.397989033s) [upload=60693]: Next returned nil segment Jun 21 20:10:05.834686295 (75h46m41.398009302s) [upload=60693]: Waiting for error group managing segments... Jun 21 20:10:05.877010488 (75h46m41.440333486s) [noise=true , address=45.91.116.6:4982 , piece_id=VP45R6KXNH5U5SGUQUMAPIBY7KFDPBIR3S3VRIX27EAT6VLFBZ7Q, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:05.886336964 (75h46m41.449659967s) [noise=true , address=104.36.82.9:6039 , piece_id=I4AB6N27MYYUHWAZLJJUIBFBLOQFHDIM3YLYWPVRPNUV7VMYS2SA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: ecclient: upload failed (node:12P11WVW9mVXw5ra2gZ3Rezon6wZi8GrN5CchQN3MiVU4ts3L6V, address:104.36.82.9:6039): protocol: storage node overloaded, request limit: 2000 Jun 21 20:10:08.064514542 (75h46m43.627837553s) [noise=true , address=77.90.31.14:4670 , piece_id=7FNANMUKPM3N6YWRAOAIM4WJZJUTNF7ZBI6ZUBNEPGNGMFWOE24A, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:08.099098391 (75h46m43.662421396s) [noise=true , address=79.127.205.245:4231 , piece_id=EJK46XUUVI5NWOAQJEW4QIJG3KRXZHB4YOQUJRCS2MIOSWKGTNLQ, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:08.489461749 (75h46m44.052784752s) [noise=true , address=104.36.82.42:6807 , piece_id=ZBQZPXUDJ4VBCOSVKWAJ6SK6UD7SLPON7Y4SSCK2DQ5STFG4XCNQ, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:08.650922093 (75h46m44.214245093s) [noise=true , address=77.90.31.19:4853 , piece_id=VFEON5T2KSWM2KSLTVRZTXYL42JWENZYUEUPG3LMODGZWBMVGGPQ, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:09.917746439 (75h46m45.481069441s) [noise=true , address=104.36.82.13:6196 , piece_id=BCHMQO5KKT5BLNWQGOHH3V43LGSE3BUQCI2SY4TTO5M5GQNOCTMQ, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:10.830722215 (75h46m46.394045216s) [noise=true , address=104.36.82.5:5869 , piece_id=HAEHXXZASV3DHT3QN7NTZOS6XZKHNUWBKCHWJJWSTJKXAKJKKBAA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:12.738783450 (75h46m48.302106452s) [noise=true , address=77.90.31.12:4603 , piece_id=BMUIC6KNGEDL37436GLSXW5XOO6VYOPNOSFGQCYVHSENVI2PJ7KQ, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:15.460927713 (75h46m51.024250723s) [noise=true , address=104.36.82.52:4111 , piece_id=HB7QAWN6CGNIIT4HD2GZ7JDQP5NXIJ5I2USBM66YOKTYENNOWG5A, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:18.393817968 (75h46m53.95714096s) [noise=true , address=104.36.82.55:4224 , piece_id=WRVNGSJV5BX4GJERLXNPAS7J7BZ5WFTM7R5H257YRCG5SP2HIYMA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:19.549174143 (75h46m55.112497151s) [noise=true , address=104.36.82.56:4253 , piece_id=5PMVGLZETMXCQ77K3I6PXACRIMBGR2CV5JGUSU5KUMYFNXVGA2GA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:22.511944593 (75h46m58.075267584s) [noise=true , address=104.36.82.37:6663 , piece_id=X7LWYIIHJFU2T2PHYR3R5TNT22UUPHEHQWZH7ZJMLS4AH4PJ6AIA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:48.027197558 (75h47m23.590520562s) [noise=true , address=104.36.82.14:6232 , piece_id=A2ZLUUZKRTBJRJ267QSK36C3LFZV7FPAYDDAXGIWDTNEGNV6MRRA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:11:05.567692582 (75h47m41.131015574s) [noise=true , address=104.36.82.49:4982 , piece_id=UNLV5UDNWULEIPE4QSQHHZQPK3BQKFRB5T3H6GP7ZQ5QGSNK5V6A, seg_pos={0 0}, upload=60693]: Done uploading piece. err: ecclient: upload failed (node:1NxoYJXCXmTvQncbkXt6yH7mhEBBrzVdzhgL5J6W5TD4N3pkaG, address:104.36.82.49:4982): protocol: expected piece hash; context deadline exceeded; EOF Jun 21 20:11:05.567759198 (75h47m41.131082195s) [noise=true , address=104.36.82.54:4171 , piece_id=MUMABKARK4KKUKWW6XDXCQVT6O7SLFHGHE7COWC7QIYGARAA2D5A, seg_pos={0 0}, upload=60693]: Done uploading piece. err: ecclient: upload failed (node:125SMhDsEZruNu6LkmQn2M7R6WzvXTdk86DRsKf522u8BNiANq3, address:104.36.82.54:4171): protocol: expected piece hash; context deadline exceeded; EOF Jun 21 20:11:05.567869019 (75h47m41.131192033s) [noise=true , address=104.36.82.10:6087 , piece_id=5FQG7BPZC373PFB54HYXSG2SVKOBBBSGZCLC4XDJEYQFOLQY74MA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: ecclient: upload failed (node:1S9k5eejJcxkiG5pXjjBZm18uP8k5jCGfWyeW2QQxmuRU8ZzRA, address:104.36.82.10:6087): protocol: write tcp 159.223.143.39:38550->104.36.82.10:6087: use of closed network connection; write tcp 159.223.143.39:38550->104.36.82.10:6087: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:38550->104.36.82.10:6087: use of closed network connection Jun 21 20:11:05.567932150 (75h47m41.131255161s) [seg_pos={0 0}, upload=60693]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jun 21 20:11:05.593626175 (75h47m41.156949181s) [upload=60693]: Done uploading segments... Jun 21 20:11:05.593657274 (75h47m41.156980265s) [upload=60693]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 31.303798338s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [..........X.................................................] duration: 22.348298667s FAIL (2025-06-21 20:47:51 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== pload=60693]: Starting upload Jun 21 20:10:05.602866542 (75h46m41.166189557s) [upload=60693]: Done starting upload Jun 21 20:10:05.603116545 (75h46m41.166439547s) [upload=60693]: Uploading segments... Jun 21 20:10:05.603495389 (75h46m41.166818397s) [upload=60693]: Got next segment. Inline: false Jun 21 20:10:05.603566566 (75h46m41.166889558s) [upload=60693]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 21 20:10:05.603573280 (75h46m41.166896286s) [upload=60693]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 21 20:10:05.639130162 (75h46m41.202453164s) [seg_pos={0 0}, upload=60693]: Begin upload segment... Jun 21 20:10:05.639231070 (75h46m41.202554057s) [seg_pos={0 0}, upload=60693]: Done begin upload segment. Jun 21 20:10:05.639292724 (75h46m41.202615728s) [noise=true , address=104.36.82.56:4253 , piece_id=5PMVGLZETMXCQ77K3I6PXACRIMBGR2CV5JGUSU5KUMYFNXVGA2GA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.639338066 (75h46m41.202661071s) [noise=true , address=104.36.82.53:4126 , piece_id=2MKRQJEPGDJKDM7VWY5M4M6NORILAIEYYATJ7HVZEOBOMXOI2LBA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.639416264 (75h46m41.202739267s) [noise=true , address=104.36.82.5:5869 , piece_id=HAEHXXZASV3DHT3QN7NTZOS6XZKHNUWBKCHWJJWSTJKXAKJKKBAA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.639873098 (75h46m41.203196115s) [noise=true , address=104.36.82.52:4111 , piece_id=HB7QAWN6CGNIIT4HD2GZ7JDQP5NXIJ5I2USBM66YOKTYENNOWG5A, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.639887296 (75h46m41.203210304s) [noise=true , address=104.36.82.10:6087 , piece_id=5FQG7BPZC373PFB54HYXSG2SVKOBBBSGZCLC4XDJEYQFOLQY74MA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.640266623 (75h46m41.203589616s) [noise=true , address=104.36.82.13:6196 , piece_id=BCHMQO5KKT5BLNWQGOHH3V43LGSE3BUQCI2SY4TTO5M5GQNOCTMQ, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.640352274 (75h46m41.203675286s) [noise=true , address=104.36.82.37:6663 , piece_id=X7LWYIIHJFU2T2PHYR3R5TNT22UUPHEHQWZH7ZJMLS4AH4PJ6AIA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.640659767 (75h46m41.203982779s) [noise=true , address=104.36.82.42:6807 , piece_id=ZBQZPXUDJ4VBCOSVKWAJ6SK6UD7SLPON7Y4SSCK2DQ5STFG4XCNQ, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.640961701 (75h46m41.20428471s) [noise=true , address=45.91.116.6:4982 , piece_id=VP45R6KXNH5U5SGUQUMAPIBY7KFDPBIR3S3VRIX27EAT6VLFBZ7Q, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.641159578 (75h46m41.204482589s) [noise=true , address=104.36.82.9:6039 , piece_id=I4AB6N27MYYUHWAZLJJUIBFBLOQFHDIM3YLYWPVRPNUV7VMYS2SA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.641588126 (75h46m41.204911136s) [noise=true , address=77.90.31.19:4853 , piece_id=VFEON5T2KSWM2KSLTVRZTXYL42JWENZYUEUPG3LMODGZWBMVGGPQ, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.641863297 (75h46m41.205186305s) [noise=true , address=77.90.31.12:4603 , piece_id=BMUIC6KNGEDL37436GLSXW5XOO6VYOPNOSFGQCYVHSENVI2PJ7KQ, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.642124427 (75h46m41.205447437s) [noise=true , address=104.36.82.49:4982 , piece_id=UNLV5UDNWULEIPE4QSQHHZQPK3BQKFRB5T3H6GP7ZQ5QGSNK5V6A, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.642285917 (75h46m41.205608925s) [noise=true , address=104.36.82.54:4171 , piece_id=MUMABKARK4KKUKWW6XDXCQVT6O7SLFHGHE7COWC7QIYGARAA2D5A, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.642656750 (75h46m41.205979758s) [noise=true , address=104.36.82.14:6232 , piece_id=A2ZLUUZKRTBJRJ267QSK36C3LFZV7FPAYDDAXGIWDTNEGNV6MRRA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.643565583 (75h46m41.206888597s) [noise=true , address=79.127.205.245:4231 , piece_id=EJK46XUUVI5NWOAQJEW4QIJG3KRXZHB4YOQUJRCS2MIOSWKGTNLQ, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.644620291 (75h46m41.207943303s) [noise=true , address=77.90.31.14:4670 , piece_id=7FNANMUKPM3N6YWRAOAIM4WJZJUTNF7ZBI6ZUBNEPGNGMFWOE24A, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.648842639 (75h46m41.212165631s) [noise=true , address=104.36.82.55:4224 , piece_id=WRVNGSJV5BX4GJERLXNPAS7J7BZ5WFTM7R5H257YRCG5SP2HIYMA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.825828239 (75h46m41.389151241s) [noise=true , address=104.36.82.53:4126 , piece_id=2MKRQJEPGDJKDM7VWY5M4M6NORILAIEYYATJ7HVZEOBOMXOI2LBA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: ecclient: upload failed (node:12dMWFrq2NMpnMzuPAphWerntC5kkrcnzRJZBA116uzJzyvk6fH, address:104.36.82.53:4126): protocol: storage node overloaded, request limit: 2000 Jun 21 20:10:05.834666043 (75h46m41.397989033s) [upload=60693]: Next returned nil segment Jun 21 20:10:05.834686295 (75h46m41.398009302s) [upload=60693]: Waiting for error group managing segments... Jun 21 20:10:05.877010488 (75h46m41.440333486s) [noise=true , address=45.91.116.6:4982 , piece_id=VP45R6KXNH5U5SGUQUMAPIBY7KFDPBIR3S3VRIX27EAT6VLFBZ7Q, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:05.886336964 (75h46m41.449659967s) [noise=true , address=104.36.82.9:6039 , piece_id=I4AB6N27MYYUHWAZLJJUIBFBLOQFHDIM3YLYWPVRPNUV7VMYS2SA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: ecclient: upload failed (node:12P11WVW9mVXw5ra2gZ3Rezon6wZi8GrN5CchQN3MiVU4ts3L6V, address:104.36.82.9:6039): protocol: storage node overloaded, request limit: 2000 Jun 21 20:10:08.064514542 (75h46m43.627837553s) [noise=true , address=77.90.31.14:4670 , piece_id=7FNANMUKPM3N6YWRAOAIM4WJZJUTNF7ZBI6ZUBNEPGNGMFWOE24A, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:08.099098391 (75h46m43.662421396s) [noise=true , address=79.127.205.245:4231 , piece_id=EJK46XUUVI5NWOAQJEW4QIJG3KRXZHB4YOQUJRCS2MIOSWKGTNLQ, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:08.489461749 (75h46m44.052784752s) [noise=true , address=104.36.82.42:6807 , piece_id=ZBQZPXUDJ4VBCOSVKWAJ6SK6UD7SLPON7Y4SSCK2DQ5STFG4XCNQ, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:08.650922093 (75h46m44.214245093s) [noise=true , address=77.90.31.19:4853 , piece_id=VFEON5T2KSWM2KSLTVRZTXYL42JWENZYUEUPG3LMODGZWBMVGGPQ, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:09.917746439 (75h46m45.481069441s) [noise=true , address=104.36.82.13:6196 , piece_id=BCHMQO5KKT5BLNWQGOHH3V43LGSE3BUQCI2SY4TTO5M5GQNOCTMQ, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:10.830722215 (75h46m46.394045216s) [noise=true , address=104.36.82.5:5869 , piece_id=HAEHXXZASV3DHT3QN7NTZOS6XZKHNUWBKCHWJJWSTJKXAKJKKBAA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:12.738783450 (75h46m48.302106452s) [noise=true , address=77.90.31.12:4603 , piece_id=BMUIC6KNGEDL37436GLSXW5XOO6VYOPNOSFGQCYVHSENVI2PJ7KQ, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:15.460927713 (75h46m51.024250723s) [noise=true , address=104.36.82.52:4111 , piece_id=HB7QAWN6CGNIIT4HD2GZ7JDQP5NXIJ5I2USBM66YOKTYENNOWG5A, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:18.393817968 (75h46m53.95714096s) [noise=true , address=104.36.82.55:4224 , piece_id=WRVNGSJV5BX4GJERLXNPAS7J7BZ5WFTM7R5H257YRCG5SP2HIYMA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:19.549174143 (75h46m55.112497151s) [noise=true , address=104.36.82.56:4253 , piece_id=5PMVGLZETMXCQ77K3I6PXACRIMBGR2CV5JGUSU5KUMYFNXVGA2GA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:22.511944593 (75h46m58.075267584s) [noise=true , address=104.36.82.37:6663 , piece_id=X7LWYIIHJFU2T2PHYR3R5TNT22UUPHEHQWZH7ZJMLS4AH4PJ6AIA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:48.027197558 (75h47m23.590520562s) [noise=true , address=104.36.82.14:6232 , piece_id=A2ZLUUZKRTBJRJ267QSK36C3LFZV7FPAYDDAXGIWDTNEGNV6MRRA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:11:05.567692582 (75h47m41.131015574s) [noise=true , address=104.36.82.49:4982 , piece_id=UNLV5UDNWULEIPE4QSQHHZQPK3BQKFRB5T3H6GP7ZQ5QGSNK5V6A, seg_pos={0 0}, upload=60693]: Done uploading piece. err: ecclient: upload failed (node:1NxoYJXCXmTvQncbkXt6yH7mhEBBrzVdzhgL5J6W5TD4N3pkaG, address:104.36.82.49:4982): protocol: expected piece hash; context deadline exceeded; EOF Jun 21 20:11:05.567759198 (75h47m41.131082195s) [noise=true , address=104.36.82.54:4171 , piece_id=MUMABKARK4KKUKWW6XDXCQVT6O7SLFHGHE7COWC7QIYGARAA2D5A, seg_pos={0 0}, upload=60693]: Done uploading piece. err: ecclient: upload failed (node:125SMhDsEZruNu6LkmQn2M7R6WzvXTdk86DRsKf522u8BNiANq3, address:104.36.82.54:4171): protocol: expected piece hash; context deadline exceeded; EOF Jun 21 20:11:05.567869019 (75h47m41.131192033s) [noise=true , address=104.36.82.10:6087 , piece_id=5FQG7BPZC373PFB54HYXSG2SVKOBBBSGZCLC4XDJEYQFOLQY74MA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: ecclient: upload failed (node:1S9k5eejJcxkiG5pXjjBZm18uP8k5jCGfWyeW2QQxmuRU8ZzRA, address:104.36.82.10:6087): protocol: write tcp 159.223.143.39:38550->104.36.82.10:6087: use of closed network connection; write tcp 159.223.143.39:38550->104.36.82.10:6087: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:38550->104.36.82.10:6087: use of closed network connection Jun 21 20:11:05.567932150 (75h47m41.131255161s) [seg_pos={0 0}, upload=60693]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jun 21 20:11:05.593626175 (75h47m41.156949181s) [upload=60693]: Done uploading segments... Jun 21 20:11:05.593657274 (75h47m41.156980265s) [upload=60693]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 22.348333531s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [...........X................................................] duration: 25.495545653s FAIL (2025-06-21 20:47:12 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== pload=60693]: Starting upload Jun 21 20:10:05.602866542 (75h46m41.166189557s) [upload=60693]: Done starting upload Jun 21 20:10:05.603116545 (75h46m41.166439547s) [upload=60693]: Uploading segments... Jun 21 20:10:05.603495389 (75h46m41.166818397s) [upload=60693]: Got next segment. Inline: false Jun 21 20:10:05.603566566 (75h46m41.166889558s) [upload=60693]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 21 20:10:05.603573280 (75h46m41.166896286s) [upload=60693]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 21 20:10:05.639130162 (75h46m41.202453164s) [seg_pos={0 0}, upload=60693]: Begin upload segment... Jun 21 20:10:05.639231070 (75h46m41.202554057s) [seg_pos={0 0}, upload=60693]: Done begin upload segment. Jun 21 20:10:05.639292724 (75h46m41.202615728s) [noise=true , address=104.36.82.56:4253 , piece_id=5PMVGLZETMXCQ77K3I6PXACRIMBGR2CV5JGUSU5KUMYFNXVGA2GA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.639338066 (75h46m41.202661071s) [noise=true , address=104.36.82.53:4126 , piece_id=2MKRQJEPGDJKDM7VWY5M4M6NORILAIEYYATJ7HVZEOBOMXOI2LBA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.639416264 (75h46m41.202739267s) [noise=true , address=104.36.82.5:5869 , piece_id=HAEHXXZASV3DHT3QN7NTZOS6XZKHNUWBKCHWJJWSTJKXAKJKKBAA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.639873098 (75h46m41.203196115s) [noise=true , address=104.36.82.52:4111 , piece_id=HB7QAWN6CGNIIT4HD2GZ7JDQP5NXIJ5I2USBM66YOKTYENNOWG5A, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.639887296 (75h46m41.203210304s) [noise=true , address=104.36.82.10:6087 , piece_id=5FQG7BPZC373PFB54HYXSG2SVKOBBBSGZCLC4XDJEYQFOLQY74MA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.640266623 (75h46m41.203589616s) [noise=true , address=104.36.82.13:6196 , piece_id=BCHMQO5KKT5BLNWQGOHH3V43LGSE3BUQCI2SY4TTO5M5GQNOCTMQ, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.640352274 (75h46m41.203675286s) [noise=true , address=104.36.82.37:6663 , piece_id=X7LWYIIHJFU2T2PHYR3R5TNT22UUPHEHQWZH7ZJMLS4AH4PJ6AIA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.640659767 (75h46m41.203982779s) [noise=true , address=104.36.82.42:6807 , piece_id=ZBQZPXUDJ4VBCOSVKWAJ6SK6UD7SLPON7Y4SSCK2DQ5STFG4XCNQ, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.640961701 (75h46m41.20428471s) [noise=true , address=45.91.116.6:4982 , piece_id=VP45R6KXNH5U5SGUQUMAPIBY7KFDPBIR3S3VRIX27EAT6VLFBZ7Q, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.641159578 (75h46m41.204482589s) [noise=true , address=104.36.82.9:6039 , piece_id=I4AB6N27MYYUHWAZLJJUIBFBLOQFHDIM3YLYWPVRPNUV7VMYS2SA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.641588126 (75h46m41.204911136s) [noise=true , address=77.90.31.19:4853 , piece_id=VFEON5T2KSWM2KSLTVRZTXYL42JWENZYUEUPG3LMODGZWBMVGGPQ, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.641863297 (75h46m41.205186305s) [noise=true , address=77.90.31.12:4603 , piece_id=BMUIC6KNGEDL37436GLSXW5XOO6VYOPNOSFGQCYVHSENVI2PJ7KQ, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.642124427 (75h46m41.205447437s) [noise=true , address=104.36.82.49:4982 , piece_id=UNLV5UDNWULEIPE4QSQHHZQPK3BQKFRB5T3H6GP7ZQ5QGSNK5V6A, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.642285917 (75h46m41.205608925s) [noise=true , address=104.36.82.54:4171 , piece_id=MUMABKARK4KKUKWW6XDXCQVT6O7SLFHGHE7COWC7QIYGARAA2D5A, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.642656750 (75h46m41.205979758s) [noise=true , address=104.36.82.14:6232 , piece_id=A2ZLUUZKRTBJRJ267QSK36C3LFZV7FPAYDDAXGIWDTNEGNV6MRRA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.643565583 (75h46m41.206888597s) [noise=true , address=79.127.205.245:4231 , piece_id=EJK46XUUVI5NWOAQJEW4QIJG3KRXZHB4YOQUJRCS2MIOSWKGTNLQ, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.644620291 (75h46m41.207943303s) [noise=true , address=77.90.31.14:4670 , piece_id=7FNANMUKPM3N6YWRAOAIM4WJZJUTNF7ZBI6ZUBNEPGNGMFWOE24A, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.648842639 (75h46m41.212165631s) [noise=true , address=104.36.82.55:4224 , piece_id=WRVNGSJV5BX4GJERLXNPAS7J7BZ5WFTM7R5H257YRCG5SP2HIYMA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.825828239 (75h46m41.389151241s) [noise=true , address=104.36.82.53:4126 , piece_id=2MKRQJEPGDJKDM7VWY5M4M6NORILAIEYYATJ7HVZEOBOMXOI2LBA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: ecclient: upload failed (node:12dMWFrq2NMpnMzuPAphWerntC5kkrcnzRJZBA116uzJzyvk6fH, address:104.36.82.53:4126): protocol: storage node overloaded, request limit: 2000 Jun 21 20:10:05.834666043 (75h46m41.397989033s) [upload=60693]: Next returned nil segment Jun 21 20:10:05.834686295 (75h46m41.398009302s) [upload=60693]: Waiting for error group managing segments... Jun 21 20:10:05.877010488 (75h46m41.440333486s) [noise=true , address=45.91.116.6:4982 , piece_id=VP45R6KXNH5U5SGUQUMAPIBY7KFDPBIR3S3VRIX27EAT6VLFBZ7Q, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:05.886336964 (75h46m41.449659967s) [noise=true , address=104.36.82.9:6039 , piece_id=I4AB6N27MYYUHWAZLJJUIBFBLOQFHDIM3YLYWPVRPNUV7VMYS2SA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: ecclient: upload failed (node:12P11WVW9mVXw5ra2gZ3Rezon6wZi8GrN5CchQN3MiVU4ts3L6V, address:104.36.82.9:6039): protocol: storage node overloaded, request limit: 2000 Jun 21 20:10:08.064514542 (75h46m43.627837553s) [noise=true , address=77.90.31.14:4670 , piece_id=7FNANMUKPM3N6YWRAOAIM4WJZJUTNF7ZBI6ZUBNEPGNGMFWOE24A, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:08.099098391 (75h46m43.662421396s) [noise=true , address=79.127.205.245:4231 , piece_id=EJK46XUUVI5NWOAQJEW4QIJG3KRXZHB4YOQUJRCS2MIOSWKGTNLQ, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:08.489461749 (75h46m44.052784752s) [noise=true , address=104.36.82.42:6807 , piece_id=ZBQZPXUDJ4VBCOSVKWAJ6SK6UD7SLPON7Y4SSCK2DQ5STFG4XCNQ, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:08.650922093 (75h46m44.214245093s) [noise=true , address=77.90.31.19:4853 , piece_id=VFEON5T2KSWM2KSLTVRZTXYL42JWENZYUEUPG3LMODGZWBMVGGPQ, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:09.917746439 (75h46m45.481069441s) [noise=true , address=104.36.82.13:6196 , piece_id=BCHMQO5KKT5BLNWQGOHH3V43LGSE3BUQCI2SY4TTO5M5GQNOCTMQ, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:10.830722215 (75h46m46.394045216s) [noise=true , address=104.36.82.5:5869 , piece_id=HAEHXXZASV3DHT3QN7NTZOS6XZKHNUWBKCHWJJWSTJKXAKJKKBAA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:12.738783450 (75h46m48.302106452s) [noise=true , address=77.90.31.12:4603 , piece_id=BMUIC6KNGEDL37436GLSXW5XOO6VYOPNOSFGQCYVHSENVI2PJ7KQ, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:15.460927713 (75h46m51.024250723s) [noise=true , address=104.36.82.52:4111 , piece_id=HB7QAWN6CGNIIT4HD2GZ7JDQP5NXIJ5I2USBM66YOKTYENNOWG5A, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:18.393817968 (75h46m53.95714096s) [noise=true , address=104.36.82.55:4224 , piece_id=WRVNGSJV5BX4GJERLXNPAS7J7BZ5WFTM7R5H257YRCG5SP2HIYMA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:19.549174143 (75h46m55.112497151s) [noise=true , address=104.36.82.56:4253 , piece_id=5PMVGLZETMXCQ77K3I6PXACRIMBGR2CV5JGUSU5KUMYFNXVGA2GA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:22.511944593 (75h46m58.075267584s) [noise=true , address=104.36.82.37:6663 , piece_id=X7LWYIIHJFU2T2PHYR3R5TNT22UUPHEHQWZH7ZJMLS4AH4PJ6AIA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:48.027197558 (75h47m23.590520562s) [noise=true , address=104.36.82.14:6232 , piece_id=A2ZLUUZKRTBJRJ267QSK36C3LFZV7FPAYDDAXGIWDTNEGNV6MRRA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:11:05.567692582 (75h47m41.131015574s) [noise=true , address=104.36.82.49:4982 , piece_id=UNLV5UDNWULEIPE4QSQHHZQPK3BQKFRB5T3H6GP7ZQ5QGSNK5V6A, seg_pos={0 0}, upload=60693]: Done uploading piece. err: ecclient: upload failed (node:1NxoYJXCXmTvQncbkXt6yH7mhEBBrzVdzhgL5J6W5TD4N3pkaG, address:104.36.82.49:4982): protocol: expected piece hash; context deadline exceeded; EOF Jun 21 20:11:05.567759198 (75h47m41.131082195s) [noise=true , address=104.36.82.54:4171 , piece_id=MUMABKARK4KKUKWW6XDXCQVT6O7SLFHGHE7COWC7QIYGARAA2D5A, seg_pos={0 0}, upload=60693]: Done uploading piece. err: ecclient: upload failed (node:125SMhDsEZruNu6LkmQn2M7R6WzvXTdk86DRsKf522u8BNiANq3, address:104.36.82.54:4171): protocol: expected piece hash; context deadline exceeded; EOF Jun 21 20:11:05.567869019 (75h47m41.131192033s) [noise=true , address=104.36.82.10:6087 , piece_id=5FQG7BPZC373PFB54HYXSG2SVKOBBBSGZCLC4XDJEYQFOLQY74MA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: ecclient: upload failed (node:1S9k5eejJcxkiG5pXjjBZm18uP8k5jCGfWyeW2QQxmuRU8ZzRA, address:104.36.82.10:6087): protocol: write tcp 159.223.143.39:38550->104.36.82.10:6087: use of closed network connection; write tcp 159.223.143.39:38550->104.36.82.10:6087: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:38550->104.36.82.10:6087: use of closed network connection Jun 21 20:11:05.567932150 (75h47m41.131255161s) [seg_pos={0 0}, upload=60693]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jun 21 20:11:05.593626175 (75h47m41.156949181s) [upload=60693]: Done uploading segments... Jun 21 20:11:05.593657274 (75h47m41.156980265s) [upload=60693]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 25.495578486s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [X...........X...............................................] duration: 26.210787833s FAIL (2025-06-21 20:46:29 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== pload=60693]: Starting upload Jun 21 20:10:05.602866542 (75h46m41.166189557s) [upload=60693]: Done starting upload Jun 21 20:10:05.603116545 (75h46m41.166439547s) [upload=60693]: Uploading segments... Jun 21 20:10:05.603495389 (75h46m41.166818397s) [upload=60693]: Got next segment. Inline: false Jun 21 20:10:05.603566566 (75h46m41.166889558s) [upload=60693]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 21 20:10:05.603573280 (75h46m41.166896286s) [upload=60693]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 21 20:10:05.639130162 (75h46m41.202453164s) [seg_pos={0 0}, upload=60693]: Begin upload segment... Jun 21 20:10:05.639231070 (75h46m41.202554057s) [seg_pos={0 0}, upload=60693]: Done begin upload segment. Jun 21 20:10:05.639292724 (75h46m41.202615728s) [noise=true , address=104.36.82.56:4253 , piece_id=5PMVGLZETMXCQ77K3I6PXACRIMBGR2CV5JGUSU5KUMYFNXVGA2GA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.639338066 (75h46m41.202661071s) [noise=true , address=104.36.82.53:4126 , piece_id=2MKRQJEPGDJKDM7VWY5M4M6NORILAIEYYATJ7HVZEOBOMXOI2LBA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.639416264 (75h46m41.202739267s) [noise=true , address=104.36.82.5:5869 , piece_id=HAEHXXZASV3DHT3QN7NTZOS6XZKHNUWBKCHWJJWSTJKXAKJKKBAA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.639873098 (75h46m41.203196115s) [noise=true , address=104.36.82.52:4111 , piece_id=HB7QAWN6CGNIIT4HD2GZ7JDQP5NXIJ5I2USBM66YOKTYENNOWG5A, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.639887296 (75h46m41.203210304s) [noise=true , address=104.36.82.10:6087 , piece_id=5FQG7BPZC373PFB54HYXSG2SVKOBBBSGZCLC4XDJEYQFOLQY74MA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.640266623 (75h46m41.203589616s) [noise=true , address=104.36.82.13:6196 , piece_id=BCHMQO5KKT5BLNWQGOHH3V43LGSE3BUQCI2SY4TTO5M5GQNOCTMQ, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.640352274 (75h46m41.203675286s) [noise=true , address=104.36.82.37:6663 , piece_id=X7LWYIIHJFU2T2PHYR3R5TNT22UUPHEHQWZH7ZJMLS4AH4PJ6AIA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.640659767 (75h46m41.203982779s) [noise=true , address=104.36.82.42:6807 , piece_id=ZBQZPXUDJ4VBCOSVKWAJ6SK6UD7SLPON7Y4SSCK2DQ5STFG4XCNQ, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.640961701 (75h46m41.20428471s) [noise=true , address=45.91.116.6:4982 , piece_id=VP45R6KXNH5U5SGUQUMAPIBY7KFDPBIR3S3VRIX27EAT6VLFBZ7Q, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.641159578 (75h46m41.204482589s) [noise=true , address=104.36.82.9:6039 , piece_id=I4AB6N27MYYUHWAZLJJUIBFBLOQFHDIM3YLYWPVRPNUV7VMYS2SA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.641588126 (75h46m41.204911136s) [noise=true , address=77.90.31.19:4853 , piece_id=VFEON5T2KSWM2KSLTVRZTXYL42JWENZYUEUPG3LMODGZWBMVGGPQ, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.641863297 (75h46m41.205186305s) [noise=true , address=77.90.31.12:4603 , piece_id=BMUIC6KNGEDL37436GLSXW5XOO6VYOPNOSFGQCYVHSENVI2PJ7KQ, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.642124427 (75h46m41.205447437s) [noise=true , address=104.36.82.49:4982 , piece_id=UNLV5UDNWULEIPE4QSQHHZQPK3BQKFRB5T3H6GP7ZQ5QGSNK5V6A, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.642285917 (75h46m41.205608925s) [noise=true , address=104.36.82.54:4171 , piece_id=MUMABKARK4KKUKWW6XDXCQVT6O7SLFHGHE7COWC7QIYGARAA2D5A, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.642656750 (75h46m41.205979758s) [noise=true , address=104.36.82.14:6232 , piece_id=A2ZLUUZKRTBJRJ267QSK36C3LFZV7FPAYDDAXGIWDTNEGNV6MRRA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.643565583 (75h46m41.206888597s) [noise=true , address=79.127.205.245:4231 , piece_id=EJK46XUUVI5NWOAQJEW4QIJG3KRXZHB4YOQUJRCS2MIOSWKGTNLQ, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.644620291 (75h46m41.207943303s) [noise=true , address=77.90.31.14:4670 , piece_id=7FNANMUKPM3N6YWRAOAIM4WJZJUTNF7ZBI6ZUBNEPGNGMFWOE24A, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.648842639 (75h46m41.212165631s) [noise=true , address=104.36.82.55:4224 , piece_id=WRVNGSJV5BX4GJERLXNPAS7J7BZ5WFTM7R5H257YRCG5SP2HIYMA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.825828239 (75h46m41.389151241s) [noise=true , address=104.36.82.53:4126 , piece_id=2MKRQJEPGDJKDM7VWY5M4M6NORILAIEYYATJ7HVZEOBOMXOI2LBA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: ecclient: upload failed (node:12dMWFrq2NMpnMzuPAphWerntC5kkrcnzRJZBA116uzJzyvk6fH, address:104.36.82.53:4126): protocol: storage node overloaded, request limit: 2000 Jun 21 20:10:05.834666043 (75h46m41.397989033s) [upload=60693]: Next returned nil segment Jun 21 20:10:05.834686295 (75h46m41.398009302s) [upload=60693]: Waiting for error group managing segments... Jun 21 20:10:05.877010488 (75h46m41.440333486s) [noise=true , address=45.91.116.6:4982 , piece_id=VP45R6KXNH5U5SGUQUMAPIBY7KFDPBIR3S3VRIX27EAT6VLFBZ7Q, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:05.886336964 (75h46m41.449659967s) [noise=true , address=104.36.82.9:6039 , piece_id=I4AB6N27MYYUHWAZLJJUIBFBLOQFHDIM3YLYWPVRPNUV7VMYS2SA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: ecclient: upload failed (node:12P11WVW9mVXw5ra2gZ3Rezon6wZi8GrN5CchQN3MiVU4ts3L6V, address:104.36.82.9:6039): protocol: storage node overloaded, request limit: 2000 Jun 21 20:10:08.064514542 (75h46m43.627837553s) [noise=true , address=77.90.31.14:4670 , piece_id=7FNANMUKPM3N6YWRAOAIM4WJZJUTNF7ZBI6ZUBNEPGNGMFWOE24A, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:08.099098391 (75h46m43.662421396s) [noise=true , address=79.127.205.245:4231 , piece_id=EJK46XUUVI5NWOAQJEW4QIJG3KRXZHB4YOQUJRCS2MIOSWKGTNLQ, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:08.489461749 (75h46m44.052784752s) [noise=true , address=104.36.82.42:6807 , piece_id=ZBQZPXUDJ4VBCOSVKWAJ6SK6UD7SLPON7Y4SSCK2DQ5STFG4XCNQ, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:08.650922093 (75h46m44.214245093s) [noise=true , address=77.90.31.19:4853 , piece_id=VFEON5T2KSWM2KSLTVRZTXYL42JWENZYUEUPG3LMODGZWBMVGGPQ, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:09.917746439 (75h46m45.481069441s) [noise=true , address=104.36.82.13:6196 , piece_id=BCHMQO5KKT5BLNWQGOHH3V43LGSE3BUQCI2SY4TTO5M5GQNOCTMQ, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:10.830722215 (75h46m46.394045216s) [noise=true , address=104.36.82.5:5869 , piece_id=HAEHXXZASV3DHT3QN7NTZOS6XZKHNUWBKCHWJJWSTJKXAKJKKBAA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:12.738783450 (75h46m48.302106452s) [noise=true , address=77.90.31.12:4603 , piece_id=BMUIC6KNGEDL37436GLSXW5XOO6VYOPNOSFGQCYVHSENVI2PJ7KQ, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:15.460927713 (75h46m51.024250723s) [noise=true , address=104.36.82.52:4111 , piece_id=HB7QAWN6CGNIIT4HD2GZ7JDQP5NXIJ5I2USBM66YOKTYENNOWG5A, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:18.393817968 (75h46m53.95714096s) [noise=true , address=104.36.82.55:4224 , piece_id=WRVNGSJV5BX4GJERLXNPAS7J7BZ5WFTM7R5H257YRCG5SP2HIYMA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:19.549174143 (75h46m55.112497151s) [noise=true , address=104.36.82.56:4253 , piece_id=5PMVGLZETMXCQ77K3I6PXACRIMBGR2CV5JGUSU5KUMYFNXVGA2GA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:22.511944593 (75h46m58.075267584s) [noise=true , address=104.36.82.37:6663 , piece_id=X7LWYIIHJFU2T2PHYR3R5TNT22UUPHEHQWZH7ZJMLS4AH4PJ6AIA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:48.027197558 (75h47m23.590520562s) [noise=true , address=104.36.82.14:6232 , piece_id=A2ZLUUZKRTBJRJ267QSK36C3LFZV7FPAYDDAXGIWDTNEGNV6MRRA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:11:05.567692582 (75h47m41.131015574s) [noise=true , address=104.36.82.49:4982 , piece_id=UNLV5UDNWULEIPE4QSQHHZQPK3BQKFRB5T3H6GP7ZQ5QGSNK5V6A, seg_pos={0 0}, upload=60693]: Done uploading piece. err: ecclient: upload failed (node:1NxoYJXCXmTvQncbkXt6yH7mhEBBrzVdzhgL5J6W5TD4N3pkaG, address:104.36.82.49:4982): protocol: expected piece hash; context deadline exceeded; EOF Jun 21 20:11:05.567759198 (75h47m41.131082195s) [noise=true , address=104.36.82.54:4171 , piece_id=MUMABKARK4KKUKWW6XDXCQVT6O7SLFHGHE7COWC7QIYGARAA2D5A, seg_pos={0 0}, upload=60693]: Done uploading piece. err: ecclient: upload failed (node:125SMhDsEZruNu6LkmQn2M7R6WzvXTdk86DRsKf522u8BNiANq3, address:104.36.82.54:4171): protocol: expected piece hash; context deadline exceeded; EOF Jun 21 20:11:05.567869019 (75h47m41.131192033s) [noise=true , address=104.36.82.10:6087 , piece_id=5FQG7BPZC373PFB54HYXSG2SVKOBBBSGZCLC4XDJEYQFOLQY74MA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: ecclient: upload failed (node:1S9k5eejJcxkiG5pXjjBZm18uP8k5jCGfWyeW2QQxmuRU8ZzRA, address:104.36.82.10:6087): protocol: write tcp 159.223.143.39:38550->104.36.82.10:6087: use of closed network connection; write tcp 159.223.143.39:38550->104.36.82.10:6087: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:38550->104.36.82.10:6087: use of closed network connection Jun 21 20:11:05.567932150 (75h47m41.131255161s) [seg_pos={0 0}, upload=60693]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jun 21 20:11:05.593626175 (75h47m41.156949181s) [upload=60693]: Done uploading segments... Jun 21 20:11:05.593657274 (75h47m41.156980265s) [upload=60693]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 26.210810266s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [.X...........X..............................................] duration: 24.601036757s FAIL (2025-06-21 20:46:03 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== pload=60693]: Starting upload Jun 21 20:10:05.602866542 (75h46m41.166189557s) [upload=60693]: Done starting upload Jun 21 20:10:05.603116545 (75h46m41.166439547s) [upload=60693]: Uploading segments... Jun 21 20:10:05.603495389 (75h46m41.166818397s) [upload=60693]: Got next segment. Inline: false Jun 21 20:10:05.603566566 (75h46m41.166889558s) [upload=60693]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 21 20:10:05.603573280 (75h46m41.166896286s) [upload=60693]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 21 20:10:05.639130162 (75h46m41.202453164s) [seg_pos={0 0}, upload=60693]: Begin upload segment... Jun 21 20:10:05.639231070 (75h46m41.202554057s) [seg_pos={0 0}, upload=60693]: Done begin upload segment. Jun 21 20:10:05.639292724 (75h46m41.202615728s) [noise=true , address=104.36.82.56:4253 , piece_id=5PMVGLZETMXCQ77K3I6PXACRIMBGR2CV5JGUSU5KUMYFNXVGA2GA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.639338066 (75h46m41.202661071s) [noise=true , address=104.36.82.53:4126 , piece_id=2MKRQJEPGDJKDM7VWY5M4M6NORILAIEYYATJ7HVZEOBOMXOI2LBA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.639416264 (75h46m41.202739267s) [noise=true , address=104.36.82.5:5869 , piece_id=HAEHXXZASV3DHT3QN7NTZOS6XZKHNUWBKCHWJJWSTJKXAKJKKBAA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.639873098 (75h46m41.203196115s) [noise=true , address=104.36.82.52:4111 , piece_id=HB7QAWN6CGNIIT4HD2GZ7JDQP5NXIJ5I2USBM66YOKTYENNOWG5A, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.639887296 (75h46m41.203210304s) [noise=true , address=104.36.82.10:6087 , piece_id=5FQG7BPZC373PFB54HYXSG2SVKOBBBSGZCLC4XDJEYQFOLQY74MA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.640266623 (75h46m41.203589616s) [noise=true , address=104.36.82.13:6196 , piece_id=BCHMQO5KKT5BLNWQGOHH3V43LGSE3BUQCI2SY4TTO5M5GQNOCTMQ, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.640352274 (75h46m41.203675286s) [noise=true , address=104.36.82.37:6663 , piece_id=X7LWYIIHJFU2T2PHYR3R5TNT22UUPHEHQWZH7ZJMLS4AH4PJ6AIA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.640659767 (75h46m41.203982779s) [noise=true , address=104.36.82.42:6807 , piece_id=ZBQZPXUDJ4VBCOSVKWAJ6SK6UD7SLPON7Y4SSCK2DQ5STFG4XCNQ, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.640961701 (75h46m41.20428471s) [noise=true , address=45.91.116.6:4982 , piece_id=VP45R6KXNH5U5SGUQUMAPIBY7KFDPBIR3S3VRIX27EAT6VLFBZ7Q, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.641159578 (75h46m41.204482589s) [noise=true , address=104.36.82.9:6039 , piece_id=I4AB6N27MYYUHWAZLJJUIBFBLOQFHDIM3YLYWPVRPNUV7VMYS2SA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.641588126 (75h46m41.204911136s) [noise=true , address=77.90.31.19:4853 , piece_id=VFEON5T2KSWM2KSLTVRZTXYL42JWENZYUEUPG3LMODGZWBMVGGPQ, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.641863297 (75h46m41.205186305s) [noise=true , address=77.90.31.12:4603 , piece_id=BMUIC6KNGEDL37436GLSXW5XOO6VYOPNOSFGQCYVHSENVI2PJ7KQ, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.642124427 (75h46m41.205447437s) [noise=true , address=104.36.82.49:4982 , piece_id=UNLV5UDNWULEIPE4QSQHHZQPK3BQKFRB5T3H6GP7ZQ5QGSNK5V6A, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.642285917 (75h46m41.205608925s) [noise=true , address=104.36.82.54:4171 , piece_id=MUMABKARK4KKUKWW6XDXCQVT6O7SLFHGHE7COWC7QIYGARAA2D5A, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.642656750 (75h46m41.205979758s) [noise=true , address=104.36.82.14:6232 , piece_id=A2ZLUUZKRTBJRJ267QSK36C3LFZV7FPAYDDAXGIWDTNEGNV6MRRA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.643565583 (75h46m41.206888597s) [noise=true , address=79.127.205.245:4231 , piece_id=EJK46XUUVI5NWOAQJEW4QIJG3KRXZHB4YOQUJRCS2MIOSWKGTNLQ, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.644620291 (75h46m41.207943303s) [noise=true , address=77.90.31.14:4670 , piece_id=7FNANMUKPM3N6YWRAOAIM4WJZJUTNF7ZBI6ZUBNEPGNGMFWOE24A, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.648842639 (75h46m41.212165631s) [noise=true , address=104.36.82.55:4224 , piece_id=WRVNGSJV5BX4GJERLXNPAS7J7BZ5WFTM7R5H257YRCG5SP2HIYMA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.825828239 (75h46m41.389151241s) [noise=true , address=104.36.82.53:4126 , piece_id=2MKRQJEPGDJKDM7VWY5M4M6NORILAIEYYATJ7HVZEOBOMXOI2LBA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: ecclient: upload failed (node:12dMWFrq2NMpnMzuPAphWerntC5kkrcnzRJZBA116uzJzyvk6fH, address:104.36.82.53:4126): protocol: storage node overloaded, request limit: 2000 Jun 21 20:10:05.834666043 (75h46m41.397989033s) [upload=60693]: Next returned nil segment Jun 21 20:10:05.834686295 (75h46m41.398009302s) [upload=60693]: Waiting for error group managing segments... Jun 21 20:10:05.877010488 (75h46m41.440333486s) [noise=true , address=45.91.116.6:4982 , piece_id=VP45R6KXNH5U5SGUQUMAPIBY7KFDPBIR3S3VRIX27EAT6VLFBZ7Q, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:05.886336964 (75h46m41.449659967s) [noise=true , address=104.36.82.9:6039 , piece_id=I4AB6N27MYYUHWAZLJJUIBFBLOQFHDIM3YLYWPVRPNUV7VMYS2SA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: ecclient: upload failed (node:12P11WVW9mVXw5ra2gZ3Rezon6wZi8GrN5CchQN3MiVU4ts3L6V, address:104.36.82.9:6039): protocol: storage node overloaded, request limit: 2000 Jun 21 20:10:08.064514542 (75h46m43.627837553s) [noise=true , address=77.90.31.14:4670 , piece_id=7FNANMUKPM3N6YWRAOAIM4WJZJUTNF7ZBI6ZUBNEPGNGMFWOE24A, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:08.099098391 (75h46m43.662421396s) [noise=true , address=79.127.205.245:4231 , piece_id=EJK46XUUVI5NWOAQJEW4QIJG3KRXZHB4YOQUJRCS2MIOSWKGTNLQ, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:08.489461749 (75h46m44.052784752s) [noise=true , address=104.36.82.42:6807 , piece_id=ZBQZPXUDJ4VBCOSVKWAJ6SK6UD7SLPON7Y4SSCK2DQ5STFG4XCNQ, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:08.650922093 (75h46m44.214245093s) [noise=true , address=77.90.31.19:4853 , piece_id=VFEON5T2KSWM2KSLTVRZTXYL42JWENZYUEUPG3LMODGZWBMVGGPQ, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:09.917746439 (75h46m45.481069441s) [noise=true , address=104.36.82.13:6196 , piece_id=BCHMQO5KKT5BLNWQGOHH3V43LGSE3BUQCI2SY4TTO5M5GQNOCTMQ, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:10.830722215 (75h46m46.394045216s) [noise=true , address=104.36.82.5:5869 , piece_id=HAEHXXZASV3DHT3QN7NTZOS6XZKHNUWBKCHWJJWSTJKXAKJKKBAA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:12.738783450 (75h46m48.302106452s) [noise=true , address=77.90.31.12:4603 , piece_id=BMUIC6KNGEDL37436GLSXW5XOO6VYOPNOSFGQCYVHSENVI2PJ7KQ, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:15.460927713 (75h46m51.024250723s) [noise=true , address=104.36.82.52:4111 , piece_id=HB7QAWN6CGNIIT4HD2GZ7JDQP5NXIJ5I2USBM66YOKTYENNOWG5A, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:18.393817968 (75h46m53.95714096s) [noise=true , address=104.36.82.55:4224 , piece_id=WRVNGSJV5BX4GJERLXNPAS7J7BZ5WFTM7R5H257YRCG5SP2HIYMA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:19.549174143 (75h46m55.112497151s) [noise=true , address=104.36.82.56:4253 , piece_id=5PMVGLZETMXCQ77K3I6PXACRIMBGR2CV5JGUSU5KUMYFNXVGA2GA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:22.511944593 (75h46m58.075267584s) [noise=true , address=104.36.82.37:6663 , piece_id=X7LWYIIHJFU2T2PHYR3R5TNT22UUPHEHQWZH7ZJMLS4AH4PJ6AIA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:48.027197558 (75h47m23.590520562s) [noise=true , address=104.36.82.14:6232 , piece_id=A2ZLUUZKRTBJRJ267QSK36C3LFZV7FPAYDDAXGIWDTNEGNV6MRRA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:11:05.567692582 (75h47m41.131015574s) [noise=true , address=104.36.82.49:4982 , piece_id=UNLV5UDNWULEIPE4QSQHHZQPK3BQKFRB5T3H6GP7ZQ5QGSNK5V6A, seg_pos={0 0}, upload=60693]: Done uploading piece. err: ecclient: upload failed (node:1NxoYJXCXmTvQncbkXt6yH7mhEBBrzVdzhgL5J6W5TD4N3pkaG, address:104.36.82.49:4982): protocol: expected piece hash; context deadline exceeded; EOF Jun 21 20:11:05.567759198 (75h47m41.131082195s) [noise=true , address=104.36.82.54:4171 , piece_id=MUMABKARK4KKUKWW6XDXCQVT6O7SLFHGHE7COWC7QIYGARAA2D5A, seg_pos={0 0}, upload=60693]: Done uploading piece. err: ecclient: upload failed (node:125SMhDsEZruNu6LkmQn2M7R6WzvXTdk86DRsKf522u8BNiANq3, address:104.36.82.54:4171): protocol: expected piece hash; context deadline exceeded; EOF Jun 21 20:11:05.567869019 (75h47m41.131192033s) [noise=true , address=104.36.82.10:6087 , piece_id=5FQG7BPZC373PFB54HYXSG2SVKOBBBSGZCLC4XDJEYQFOLQY74MA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: ecclient: upload failed (node:1S9k5eejJcxkiG5pXjjBZm18uP8k5jCGfWyeW2QQxmuRU8ZzRA, address:104.36.82.10:6087): protocol: write tcp 159.223.143.39:38550->104.36.82.10:6087: use of closed network connection; write tcp 159.223.143.39:38550->104.36.82.10:6087: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:38550->104.36.82.10:6087: use of closed network connection Jun 21 20:11:05.567932150 (75h47m41.131255161s) [seg_pos={0 0}, upload=60693]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jun 21 20:11:05.593626175 (75h47m41.156949181s) [upload=60693]: Done uploading segments... Jun 21 20:11:05.593657274 (75h47m41.156980265s) [upload=60693]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 24.601067242s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [..X...........X.............................................] duration: 14.470449209s FAIL (2025-06-21 20:45:28 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== pload=60693]: Starting upload Jun 21 20:10:05.602866542 (75h46m41.166189557s) [upload=60693]: Done starting upload Jun 21 20:10:05.603116545 (75h46m41.166439547s) [upload=60693]: Uploading segments... Jun 21 20:10:05.603495389 (75h46m41.166818397s) [upload=60693]: Got next segment. Inline: false Jun 21 20:10:05.603566566 (75h46m41.166889558s) [upload=60693]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 21 20:10:05.603573280 (75h46m41.166896286s) [upload=60693]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 21 20:10:05.639130162 (75h46m41.202453164s) [seg_pos={0 0}, upload=60693]: Begin upload segment... Jun 21 20:10:05.639231070 (75h46m41.202554057s) [seg_pos={0 0}, upload=60693]: Done begin upload segment. Jun 21 20:10:05.639292724 (75h46m41.202615728s) [noise=true , address=104.36.82.56:4253 , piece_id=5PMVGLZETMXCQ77K3I6PXACRIMBGR2CV5JGUSU5KUMYFNXVGA2GA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.639338066 (75h46m41.202661071s) [noise=true , address=104.36.82.53:4126 , piece_id=2MKRQJEPGDJKDM7VWY5M4M6NORILAIEYYATJ7HVZEOBOMXOI2LBA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.639416264 (75h46m41.202739267s) [noise=true , address=104.36.82.5:5869 , piece_id=HAEHXXZASV3DHT3QN7NTZOS6XZKHNUWBKCHWJJWSTJKXAKJKKBAA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.639873098 (75h46m41.203196115s) [noise=true , address=104.36.82.52:4111 , piece_id=HB7QAWN6CGNIIT4HD2GZ7JDQP5NXIJ5I2USBM66YOKTYENNOWG5A, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.639887296 (75h46m41.203210304s) [noise=true , address=104.36.82.10:6087 , piece_id=5FQG7BPZC373PFB54HYXSG2SVKOBBBSGZCLC4XDJEYQFOLQY74MA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.640266623 (75h46m41.203589616s) [noise=true , address=104.36.82.13:6196 , piece_id=BCHMQO5KKT5BLNWQGOHH3V43LGSE3BUQCI2SY4TTO5M5GQNOCTMQ, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.640352274 (75h46m41.203675286s) [noise=true , address=104.36.82.37:6663 , piece_id=X7LWYIIHJFU2T2PHYR3R5TNT22UUPHEHQWZH7ZJMLS4AH4PJ6AIA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.640659767 (75h46m41.203982779s) [noise=true , address=104.36.82.42:6807 , piece_id=ZBQZPXUDJ4VBCOSVKWAJ6SK6UD7SLPON7Y4SSCK2DQ5STFG4XCNQ, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.640961701 (75h46m41.20428471s) [noise=true , address=45.91.116.6:4982 , piece_id=VP45R6KXNH5U5SGUQUMAPIBY7KFDPBIR3S3VRIX27EAT6VLFBZ7Q, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.641159578 (75h46m41.204482589s) [noise=true , address=104.36.82.9:6039 , piece_id=I4AB6N27MYYUHWAZLJJUIBFBLOQFHDIM3YLYWPVRPNUV7VMYS2SA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.641588126 (75h46m41.204911136s) [noise=true , address=77.90.31.19:4853 , piece_id=VFEON5T2KSWM2KSLTVRZTXYL42JWENZYUEUPG3LMODGZWBMVGGPQ, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.641863297 (75h46m41.205186305s) [noise=true , address=77.90.31.12:4603 , piece_id=BMUIC6KNGEDL37436GLSXW5XOO6VYOPNOSFGQCYVHSENVI2PJ7KQ, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.642124427 (75h46m41.205447437s) [noise=true , address=104.36.82.49:4982 , piece_id=UNLV5UDNWULEIPE4QSQHHZQPK3BQKFRB5T3H6GP7ZQ5QGSNK5V6A, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.642285917 (75h46m41.205608925s) [noise=true , address=104.36.82.54:4171 , piece_id=MUMABKARK4KKUKWW6XDXCQVT6O7SLFHGHE7COWC7QIYGARAA2D5A, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.642656750 (75h46m41.205979758s) [noise=true , address=104.36.82.14:6232 , piece_id=A2ZLUUZKRTBJRJ267QSK36C3LFZV7FPAYDDAXGIWDTNEGNV6MRRA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.643565583 (75h46m41.206888597s) [noise=true , address=79.127.205.245:4231 , piece_id=EJK46XUUVI5NWOAQJEW4QIJG3KRXZHB4YOQUJRCS2MIOSWKGTNLQ, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.644620291 (75h46m41.207943303s) [noise=true , address=77.90.31.14:4670 , piece_id=7FNANMUKPM3N6YWRAOAIM4WJZJUTNF7ZBI6ZUBNEPGNGMFWOE24A, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.648842639 (75h46m41.212165631s) [noise=true , address=104.36.82.55:4224 , piece_id=WRVNGSJV5BX4GJERLXNPAS7J7BZ5WFTM7R5H257YRCG5SP2HIYMA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.825828239 (75h46m41.389151241s) [noise=true , address=104.36.82.53:4126 , piece_id=2MKRQJEPGDJKDM7VWY5M4M6NORILAIEYYATJ7HVZEOBOMXOI2LBA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: ecclient: upload failed (node:12dMWFrq2NMpnMzuPAphWerntC5kkrcnzRJZBA116uzJzyvk6fH, address:104.36.82.53:4126): protocol: storage node overloaded, request limit: 2000 Jun 21 20:10:05.834666043 (75h46m41.397989033s) [upload=60693]: Next returned nil segment Jun 21 20:10:05.834686295 (75h46m41.398009302s) [upload=60693]: Waiting for error group managing segments... Jun 21 20:10:05.877010488 (75h46m41.440333486s) [noise=true , address=45.91.116.6:4982 , piece_id=VP45R6KXNH5U5SGUQUMAPIBY7KFDPBIR3S3VRIX27EAT6VLFBZ7Q, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:05.886336964 (75h46m41.449659967s) [noise=true , address=104.36.82.9:6039 , piece_id=I4AB6N27MYYUHWAZLJJUIBFBLOQFHDIM3YLYWPVRPNUV7VMYS2SA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: ecclient: upload failed (node:12P11WVW9mVXw5ra2gZ3Rezon6wZi8GrN5CchQN3MiVU4ts3L6V, address:104.36.82.9:6039): protocol: storage node overloaded, request limit: 2000 Jun 21 20:10:08.064514542 (75h46m43.627837553s) [noise=true , address=77.90.31.14:4670 , piece_id=7FNANMUKPM3N6YWRAOAIM4WJZJUTNF7ZBI6ZUBNEPGNGMFWOE24A, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:08.099098391 (75h46m43.662421396s) [noise=true , address=79.127.205.245:4231 , piece_id=EJK46XUUVI5NWOAQJEW4QIJG3KRXZHB4YOQUJRCS2MIOSWKGTNLQ, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:08.489461749 (75h46m44.052784752s) [noise=true , address=104.36.82.42:6807 , piece_id=ZBQZPXUDJ4VBCOSVKWAJ6SK6UD7SLPON7Y4SSCK2DQ5STFG4XCNQ, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:08.650922093 (75h46m44.214245093s) [noise=true , address=77.90.31.19:4853 , piece_id=VFEON5T2KSWM2KSLTVRZTXYL42JWENZYUEUPG3LMODGZWBMVGGPQ, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:09.917746439 (75h46m45.481069441s) [noise=true , address=104.36.82.13:6196 , piece_id=BCHMQO5KKT5BLNWQGOHH3V43LGSE3BUQCI2SY4TTO5M5GQNOCTMQ, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:10.830722215 (75h46m46.394045216s) [noise=true , address=104.36.82.5:5869 , piece_id=HAEHXXZASV3DHT3QN7NTZOS6XZKHNUWBKCHWJJWSTJKXAKJKKBAA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:12.738783450 (75h46m48.302106452s) [noise=true , address=77.90.31.12:4603 , piece_id=BMUIC6KNGEDL37436GLSXW5XOO6VYOPNOSFGQCYVHSENVI2PJ7KQ, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:15.460927713 (75h46m51.024250723s) [noise=true , address=104.36.82.52:4111 , piece_id=HB7QAWN6CGNIIT4HD2GZ7JDQP5NXIJ5I2USBM66YOKTYENNOWG5A, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:18.393817968 (75h46m53.95714096s) [noise=true , address=104.36.82.55:4224 , piece_id=WRVNGSJV5BX4GJERLXNPAS7J7BZ5WFTM7R5H257YRCG5SP2HIYMA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:19.549174143 (75h46m55.112497151s) [noise=true , address=104.36.82.56:4253 , piece_id=5PMVGLZETMXCQ77K3I6PXACRIMBGR2CV5JGUSU5KUMYFNXVGA2GA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:22.511944593 (75h46m58.075267584s) [noise=true , address=104.36.82.37:6663 , piece_id=X7LWYIIHJFU2T2PHYR3R5TNT22UUPHEHQWZH7ZJMLS4AH4PJ6AIA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:48.027197558 (75h47m23.590520562s) [noise=true , address=104.36.82.14:6232 , piece_id=A2ZLUUZKRTBJRJ267QSK36C3LFZV7FPAYDDAXGIWDTNEGNV6MRRA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:11:05.567692582 (75h47m41.131015574s) [noise=true , address=104.36.82.49:4982 , piece_id=UNLV5UDNWULEIPE4QSQHHZQPK3BQKFRB5T3H6GP7ZQ5QGSNK5V6A, seg_pos={0 0}, upload=60693]: Done uploading piece. err: ecclient: upload failed (node:1NxoYJXCXmTvQncbkXt6yH7mhEBBrzVdzhgL5J6W5TD4N3pkaG, address:104.36.82.49:4982): protocol: expected piece hash; context deadline exceeded; EOF Jun 21 20:11:05.567759198 (75h47m41.131082195s) [noise=true , address=104.36.82.54:4171 , piece_id=MUMABKARK4KKUKWW6XDXCQVT6O7SLFHGHE7COWC7QIYGARAA2D5A, seg_pos={0 0}, upload=60693]: Done uploading piece. err: ecclient: upload failed (node:125SMhDsEZruNu6LkmQn2M7R6WzvXTdk86DRsKf522u8BNiANq3, address:104.36.82.54:4171): protocol: expected piece hash; context deadline exceeded; EOF Jun 21 20:11:05.567869019 (75h47m41.131192033s) [noise=true , address=104.36.82.10:6087 , piece_id=5FQG7BPZC373PFB54HYXSG2SVKOBBBSGZCLC4XDJEYQFOLQY74MA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: ecclient: upload failed (node:1S9k5eejJcxkiG5pXjjBZm18uP8k5jCGfWyeW2QQxmuRU8ZzRA, address:104.36.82.10:6087): protocol: write tcp 159.223.143.39:38550->104.36.82.10:6087: use of closed network connection; write tcp 159.223.143.39:38550->104.36.82.10:6087: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:38550->104.36.82.10:6087: use of closed network connection Jun 21 20:11:05.567932150 (75h47m41.131255161s) [seg_pos={0 0}, upload=60693]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jun 21 20:11:05.593626175 (75h47m41.156949181s) [upload=60693]: Done uploading segments... Jun 21 20:11:05.593657274 (75h47m41.156980265s) [upload=60693]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 14.470557103s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [...X...........X............................................] duration: 19.503581206s FAIL (2025-06-21 20:44:53 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== pload=60693]: Starting upload Jun 21 20:10:05.602866542 (75h46m41.166189557s) [upload=60693]: Done starting upload Jun 21 20:10:05.603116545 (75h46m41.166439547s) [upload=60693]: Uploading segments... Jun 21 20:10:05.603495389 (75h46m41.166818397s) [upload=60693]: Got next segment. Inline: false Jun 21 20:10:05.603566566 (75h46m41.166889558s) [upload=60693]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 21 20:10:05.603573280 (75h46m41.166896286s) [upload=60693]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 21 20:10:05.639130162 (75h46m41.202453164s) [seg_pos={0 0}, upload=60693]: Begin upload segment... Jun 21 20:10:05.639231070 (75h46m41.202554057s) [seg_pos={0 0}, upload=60693]: Done begin upload segment. Jun 21 20:10:05.639292724 (75h46m41.202615728s) [noise=true , address=104.36.82.56:4253 , piece_id=5PMVGLZETMXCQ77K3I6PXACRIMBGR2CV5JGUSU5KUMYFNXVGA2GA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.639338066 (75h46m41.202661071s) [noise=true , address=104.36.82.53:4126 , piece_id=2MKRQJEPGDJKDM7VWY5M4M6NORILAIEYYATJ7HVZEOBOMXOI2LBA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.639416264 (75h46m41.202739267s) [noise=true , address=104.36.82.5:5869 , piece_id=HAEHXXZASV3DHT3QN7NTZOS6XZKHNUWBKCHWJJWSTJKXAKJKKBAA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.639873098 (75h46m41.203196115s) [noise=true , address=104.36.82.52:4111 , piece_id=HB7QAWN6CGNIIT4HD2GZ7JDQP5NXIJ5I2USBM66YOKTYENNOWG5A, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.639887296 (75h46m41.203210304s) [noise=true , address=104.36.82.10:6087 , piece_id=5FQG7BPZC373PFB54HYXSG2SVKOBBBSGZCLC4XDJEYQFOLQY74MA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.640266623 (75h46m41.203589616s) [noise=true , address=104.36.82.13:6196 , piece_id=BCHMQO5KKT5BLNWQGOHH3V43LGSE3BUQCI2SY4TTO5M5GQNOCTMQ, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.640352274 (75h46m41.203675286s) [noise=true , address=104.36.82.37:6663 , piece_id=X7LWYIIHJFU2T2PHYR3R5TNT22UUPHEHQWZH7ZJMLS4AH4PJ6AIA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.640659767 (75h46m41.203982779s) [noise=true , address=104.36.82.42:6807 , piece_id=ZBQZPXUDJ4VBCOSVKWAJ6SK6UD7SLPON7Y4SSCK2DQ5STFG4XCNQ, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.640961701 (75h46m41.20428471s) [noise=true , address=45.91.116.6:4982 , piece_id=VP45R6KXNH5U5SGUQUMAPIBY7KFDPBIR3S3VRIX27EAT6VLFBZ7Q, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.641159578 (75h46m41.204482589s) [noise=true , address=104.36.82.9:6039 , piece_id=I4AB6N27MYYUHWAZLJJUIBFBLOQFHDIM3YLYWPVRPNUV7VMYS2SA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.641588126 (75h46m41.204911136s) [noise=true , address=77.90.31.19:4853 , piece_id=VFEON5T2KSWM2KSLTVRZTXYL42JWENZYUEUPG3LMODGZWBMVGGPQ, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.641863297 (75h46m41.205186305s) [noise=true , address=77.90.31.12:4603 , piece_id=BMUIC6KNGEDL37436GLSXW5XOO6VYOPNOSFGQCYVHSENVI2PJ7KQ, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.642124427 (75h46m41.205447437s) [noise=true , address=104.36.82.49:4982 , piece_id=UNLV5UDNWULEIPE4QSQHHZQPK3BQKFRB5T3H6GP7ZQ5QGSNK5V6A, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.642285917 (75h46m41.205608925s) [noise=true , address=104.36.82.54:4171 , piece_id=MUMABKARK4KKUKWW6XDXCQVT6O7SLFHGHE7COWC7QIYGARAA2D5A, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.642656750 (75h46m41.205979758s) [noise=true , address=104.36.82.14:6232 , piece_id=A2ZLUUZKRTBJRJ267QSK36C3LFZV7FPAYDDAXGIWDTNEGNV6MRRA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.643565583 (75h46m41.206888597s) [noise=true , address=79.127.205.245:4231 , piece_id=EJK46XUUVI5NWOAQJEW4QIJG3KRXZHB4YOQUJRCS2MIOSWKGTNLQ, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.644620291 (75h46m41.207943303s) [noise=true , address=77.90.31.14:4670 , piece_id=7FNANMUKPM3N6YWRAOAIM4WJZJUTNF7ZBI6ZUBNEPGNGMFWOE24A, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.648842639 (75h46m41.212165631s) [noise=true , address=104.36.82.55:4224 , piece_id=WRVNGSJV5BX4GJERLXNPAS7J7BZ5WFTM7R5H257YRCG5SP2HIYMA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.825828239 (75h46m41.389151241s) [noise=true , address=104.36.82.53:4126 , piece_id=2MKRQJEPGDJKDM7VWY5M4M6NORILAIEYYATJ7HVZEOBOMXOI2LBA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: ecclient: upload failed (node:12dMWFrq2NMpnMzuPAphWerntC5kkrcnzRJZBA116uzJzyvk6fH, address:104.36.82.53:4126): protocol: storage node overloaded, request limit: 2000 Jun 21 20:10:05.834666043 (75h46m41.397989033s) [upload=60693]: Next returned nil segment Jun 21 20:10:05.834686295 (75h46m41.398009302s) [upload=60693]: Waiting for error group managing segments... Jun 21 20:10:05.877010488 (75h46m41.440333486s) [noise=true , address=45.91.116.6:4982 , piece_id=VP45R6KXNH5U5SGUQUMAPIBY7KFDPBIR3S3VRIX27EAT6VLFBZ7Q, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:05.886336964 (75h46m41.449659967s) [noise=true , address=104.36.82.9:6039 , piece_id=I4AB6N27MYYUHWAZLJJUIBFBLOQFHDIM3YLYWPVRPNUV7VMYS2SA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: ecclient: upload failed (node:12P11WVW9mVXw5ra2gZ3Rezon6wZi8GrN5CchQN3MiVU4ts3L6V, address:104.36.82.9:6039): protocol: storage node overloaded, request limit: 2000 Jun 21 20:10:08.064514542 (75h46m43.627837553s) [noise=true , address=77.90.31.14:4670 , piece_id=7FNANMUKPM3N6YWRAOAIM4WJZJUTNF7ZBI6ZUBNEPGNGMFWOE24A, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:08.099098391 (75h46m43.662421396s) [noise=true , address=79.127.205.245:4231 , piece_id=EJK46XUUVI5NWOAQJEW4QIJG3KRXZHB4YOQUJRCS2MIOSWKGTNLQ, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:08.489461749 (75h46m44.052784752s) [noise=true , address=104.36.82.42:6807 , piece_id=ZBQZPXUDJ4VBCOSVKWAJ6SK6UD7SLPON7Y4SSCK2DQ5STFG4XCNQ, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:08.650922093 (75h46m44.214245093s) [noise=true , address=77.90.31.19:4853 , piece_id=VFEON5T2KSWM2KSLTVRZTXYL42JWENZYUEUPG3LMODGZWBMVGGPQ, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:09.917746439 (75h46m45.481069441s) [noise=true , address=104.36.82.13:6196 , piece_id=BCHMQO5KKT5BLNWQGOHH3V43LGSE3BUQCI2SY4TTO5M5GQNOCTMQ, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:10.830722215 (75h46m46.394045216s) [noise=true , address=104.36.82.5:5869 , piece_id=HAEHXXZASV3DHT3QN7NTZOS6XZKHNUWBKCHWJJWSTJKXAKJKKBAA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:12.738783450 (75h46m48.302106452s) [noise=true , address=77.90.31.12:4603 , piece_id=BMUIC6KNGEDL37436GLSXW5XOO6VYOPNOSFGQCYVHSENVI2PJ7KQ, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:15.460927713 (75h46m51.024250723s) [noise=true , address=104.36.82.52:4111 , piece_id=HB7QAWN6CGNIIT4HD2GZ7JDQP5NXIJ5I2USBM66YOKTYENNOWG5A, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:18.393817968 (75h46m53.95714096s) [noise=true , address=104.36.82.55:4224 , piece_id=WRVNGSJV5BX4GJERLXNPAS7J7BZ5WFTM7R5H257YRCG5SP2HIYMA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:19.549174143 (75h46m55.112497151s) [noise=true , address=104.36.82.56:4253 , piece_id=5PMVGLZETMXCQ77K3I6PXACRIMBGR2CV5JGUSU5KUMYFNXVGA2GA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:22.511944593 (75h46m58.075267584s) [noise=true , address=104.36.82.37:6663 , piece_id=X7LWYIIHJFU2T2PHYR3R5TNT22UUPHEHQWZH7ZJMLS4AH4PJ6AIA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:48.027197558 (75h47m23.590520562s) [noise=true , address=104.36.82.14:6232 , piece_id=A2ZLUUZKRTBJRJ267QSK36C3LFZV7FPAYDDAXGIWDTNEGNV6MRRA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:11:05.567692582 (75h47m41.131015574s) [noise=true , address=104.36.82.49:4982 , piece_id=UNLV5UDNWULEIPE4QSQHHZQPK3BQKFRB5T3H6GP7ZQ5QGSNK5V6A, seg_pos={0 0}, upload=60693]: Done uploading piece. err: ecclient: upload failed (node:1NxoYJXCXmTvQncbkXt6yH7mhEBBrzVdzhgL5J6W5TD4N3pkaG, address:104.36.82.49:4982): protocol: expected piece hash; context deadline exceeded; EOF Jun 21 20:11:05.567759198 (75h47m41.131082195s) [noise=true , address=104.36.82.54:4171 , piece_id=MUMABKARK4KKUKWW6XDXCQVT6O7SLFHGHE7COWC7QIYGARAA2D5A, seg_pos={0 0}, upload=60693]: Done uploading piece. err: ecclient: upload failed (node:125SMhDsEZruNu6LkmQn2M7R6WzvXTdk86DRsKf522u8BNiANq3, address:104.36.82.54:4171): protocol: expected piece hash; context deadline exceeded; EOF Jun 21 20:11:05.567869019 (75h47m41.131192033s) [noise=true , address=104.36.82.10:6087 , piece_id=5FQG7BPZC373PFB54HYXSG2SVKOBBBSGZCLC4XDJEYQFOLQY74MA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: ecclient: upload failed (node:1S9k5eejJcxkiG5pXjjBZm18uP8k5jCGfWyeW2QQxmuRU8ZzRA, address:104.36.82.10:6087): protocol: write tcp 159.223.143.39:38550->104.36.82.10:6087: use of closed network connection; write tcp 159.223.143.39:38550->104.36.82.10:6087: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:38550->104.36.82.10:6087: use of closed network connection Jun 21 20:11:05.567932150 (75h47m41.131255161s) [seg_pos={0 0}, upload=60693]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jun 21 20:11:05.593626175 (75h47m41.156949181s) [upload=60693]: Done uploading segments... Jun 21 20:11:05.593657274 (75h47m41.156980265s) [upload=60693]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 19.503621654s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [....X...........X...........................................] duration: 17.666597535s FAIL (2025-06-21 20:44:01 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== pload=60693]: Starting upload Jun 21 20:10:05.602866542 (75h46m41.166189557s) [upload=60693]: Done starting upload Jun 21 20:10:05.603116545 (75h46m41.166439547s) [upload=60693]: Uploading segments... Jun 21 20:10:05.603495389 (75h46m41.166818397s) [upload=60693]: Got next segment. Inline: false Jun 21 20:10:05.603566566 (75h46m41.166889558s) [upload=60693]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 21 20:10:05.603573280 (75h46m41.166896286s) [upload=60693]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 21 20:10:05.639130162 (75h46m41.202453164s) [seg_pos={0 0}, upload=60693]: Begin upload segment... Jun 21 20:10:05.639231070 (75h46m41.202554057s) [seg_pos={0 0}, upload=60693]: Done begin upload segment. Jun 21 20:10:05.639292724 (75h46m41.202615728s) [noise=true , address=104.36.82.56:4253 , piece_id=5PMVGLZETMXCQ77K3I6PXACRIMBGR2CV5JGUSU5KUMYFNXVGA2GA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.639338066 (75h46m41.202661071s) [noise=true , address=104.36.82.53:4126 , piece_id=2MKRQJEPGDJKDM7VWY5M4M6NORILAIEYYATJ7HVZEOBOMXOI2LBA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.639416264 (75h46m41.202739267s) [noise=true , address=104.36.82.5:5869 , piece_id=HAEHXXZASV3DHT3QN7NTZOS6XZKHNUWBKCHWJJWSTJKXAKJKKBAA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.639873098 (75h46m41.203196115s) [noise=true , address=104.36.82.52:4111 , piece_id=HB7QAWN6CGNIIT4HD2GZ7JDQP5NXIJ5I2USBM66YOKTYENNOWG5A, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.639887296 (75h46m41.203210304s) [noise=true , address=104.36.82.10:6087 , piece_id=5FQG7BPZC373PFB54HYXSG2SVKOBBBSGZCLC4XDJEYQFOLQY74MA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.640266623 (75h46m41.203589616s) [noise=true , address=104.36.82.13:6196 , piece_id=BCHMQO5KKT5BLNWQGOHH3V43LGSE3BUQCI2SY4TTO5M5GQNOCTMQ, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.640352274 (75h46m41.203675286s) [noise=true , address=104.36.82.37:6663 , piece_id=X7LWYIIHJFU2T2PHYR3R5TNT22UUPHEHQWZH7ZJMLS4AH4PJ6AIA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.640659767 (75h46m41.203982779s) [noise=true , address=104.36.82.42:6807 , piece_id=ZBQZPXUDJ4VBCOSVKWAJ6SK6UD7SLPON7Y4SSCK2DQ5STFG4XCNQ, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.640961701 (75h46m41.20428471s) [noise=true , address=45.91.116.6:4982 , piece_id=VP45R6KXNH5U5SGUQUMAPIBY7KFDPBIR3S3VRIX27EAT6VLFBZ7Q, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.641159578 (75h46m41.204482589s) [noise=true , address=104.36.82.9:6039 , piece_id=I4AB6N27MYYUHWAZLJJUIBFBLOQFHDIM3YLYWPVRPNUV7VMYS2SA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.641588126 (75h46m41.204911136s) [noise=true , address=77.90.31.19:4853 , piece_id=VFEON5T2KSWM2KSLTVRZTXYL42JWENZYUEUPG3LMODGZWBMVGGPQ, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.641863297 (75h46m41.205186305s) [noise=true , address=77.90.31.12:4603 , piece_id=BMUIC6KNGEDL37436GLSXW5XOO6VYOPNOSFGQCYVHSENVI2PJ7KQ, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.642124427 (75h46m41.205447437s) [noise=true , address=104.36.82.49:4982 , piece_id=UNLV5UDNWULEIPE4QSQHHZQPK3BQKFRB5T3H6GP7ZQ5QGSNK5V6A, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.642285917 (75h46m41.205608925s) [noise=true , address=104.36.82.54:4171 , piece_id=MUMABKARK4KKUKWW6XDXCQVT6O7SLFHGHE7COWC7QIYGARAA2D5A, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.642656750 (75h46m41.205979758s) [noise=true , address=104.36.82.14:6232 , piece_id=A2ZLUUZKRTBJRJ267QSK36C3LFZV7FPAYDDAXGIWDTNEGNV6MRRA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.643565583 (75h46m41.206888597s) [noise=true , address=79.127.205.245:4231 , piece_id=EJK46XUUVI5NWOAQJEW4QIJG3KRXZHB4YOQUJRCS2MIOSWKGTNLQ, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.644620291 (75h46m41.207943303s) [noise=true , address=77.90.31.14:4670 , piece_id=7FNANMUKPM3N6YWRAOAIM4WJZJUTNF7ZBI6ZUBNEPGNGMFWOE24A, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.648842639 (75h46m41.212165631s) [noise=true , address=104.36.82.55:4224 , piece_id=WRVNGSJV5BX4GJERLXNPAS7J7BZ5WFTM7R5H257YRCG5SP2HIYMA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.825828239 (75h46m41.389151241s) [noise=true , address=104.36.82.53:4126 , piece_id=2MKRQJEPGDJKDM7VWY5M4M6NORILAIEYYATJ7HVZEOBOMXOI2LBA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: ecclient: upload failed (node:12dMWFrq2NMpnMzuPAphWerntC5kkrcnzRJZBA116uzJzyvk6fH, address:104.36.82.53:4126): protocol: storage node overloaded, request limit: 2000 Jun 21 20:10:05.834666043 (75h46m41.397989033s) [upload=60693]: Next returned nil segment Jun 21 20:10:05.834686295 (75h46m41.398009302s) [upload=60693]: Waiting for error group managing segments... Jun 21 20:10:05.877010488 (75h46m41.440333486s) [noise=true , address=45.91.116.6:4982 , piece_id=VP45R6KXNH5U5SGUQUMAPIBY7KFDPBIR3S3VRIX27EAT6VLFBZ7Q, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:05.886336964 (75h46m41.449659967s) [noise=true , address=104.36.82.9:6039 , piece_id=I4AB6N27MYYUHWAZLJJUIBFBLOQFHDIM3YLYWPVRPNUV7VMYS2SA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: ecclient: upload failed (node:12P11WVW9mVXw5ra2gZ3Rezon6wZi8GrN5CchQN3MiVU4ts3L6V, address:104.36.82.9:6039): protocol: storage node overloaded, request limit: 2000 Jun 21 20:10:08.064514542 (75h46m43.627837553s) [noise=true , address=77.90.31.14:4670 , piece_id=7FNANMUKPM3N6YWRAOAIM4WJZJUTNF7ZBI6ZUBNEPGNGMFWOE24A, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:08.099098391 (75h46m43.662421396s) [noise=true , address=79.127.205.245:4231 , piece_id=EJK46XUUVI5NWOAQJEW4QIJG3KRXZHB4YOQUJRCS2MIOSWKGTNLQ, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:08.489461749 (75h46m44.052784752s) [noise=true , address=104.36.82.42:6807 , piece_id=ZBQZPXUDJ4VBCOSVKWAJ6SK6UD7SLPON7Y4SSCK2DQ5STFG4XCNQ, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:08.650922093 (75h46m44.214245093s) [noise=true , address=77.90.31.19:4853 , piece_id=VFEON5T2KSWM2KSLTVRZTXYL42JWENZYUEUPG3LMODGZWBMVGGPQ, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:09.917746439 (75h46m45.481069441s) [noise=true , address=104.36.82.13:6196 , piece_id=BCHMQO5KKT5BLNWQGOHH3V43LGSE3BUQCI2SY4TTO5M5GQNOCTMQ, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:10.830722215 (75h46m46.394045216s) [noise=true , address=104.36.82.5:5869 , piece_id=HAEHXXZASV3DHT3QN7NTZOS6XZKHNUWBKCHWJJWSTJKXAKJKKBAA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:12.738783450 (75h46m48.302106452s) [noise=true , address=77.90.31.12:4603 , piece_id=BMUIC6KNGEDL37436GLSXW5XOO6VYOPNOSFGQCYVHSENVI2PJ7KQ, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:15.460927713 (75h46m51.024250723s) [noise=true , address=104.36.82.52:4111 , piece_id=HB7QAWN6CGNIIT4HD2GZ7JDQP5NXIJ5I2USBM66YOKTYENNOWG5A, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:18.393817968 (75h46m53.95714096s) [noise=true , address=104.36.82.55:4224 , piece_id=WRVNGSJV5BX4GJERLXNPAS7J7BZ5WFTM7R5H257YRCG5SP2HIYMA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:19.549174143 (75h46m55.112497151s) [noise=true , address=104.36.82.56:4253 , piece_id=5PMVGLZETMXCQ77K3I6PXACRIMBGR2CV5JGUSU5KUMYFNXVGA2GA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:22.511944593 (75h46m58.075267584s) [noise=true , address=104.36.82.37:6663 , piece_id=X7LWYIIHJFU2T2PHYR3R5TNT22UUPHEHQWZH7ZJMLS4AH4PJ6AIA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:48.027197558 (75h47m23.590520562s) [noise=true , address=104.36.82.14:6232 , piece_id=A2ZLUUZKRTBJRJ267QSK36C3LFZV7FPAYDDAXGIWDTNEGNV6MRRA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:11:05.567692582 (75h47m41.131015574s) [noise=true , address=104.36.82.49:4982 , piece_id=UNLV5UDNWULEIPE4QSQHHZQPK3BQKFRB5T3H6GP7ZQ5QGSNK5V6A, seg_pos={0 0}, upload=60693]: Done uploading piece. err: ecclient: upload failed (node:1NxoYJXCXmTvQncbkXt6yH7mhEBBrzVdzhgL5J6W5TD4N3pkaG, address:104.36.82.49:4982): protocol: expected piece hash; context deadline exceeded; EOF Jun 21 20:11:05.567759198 (75h47m41.131082195s) [noise=true , address=104.36.82.54:4171 , piece_id=MUMABKARK4KKUKWW6XDXCQVT6O7SLFHGHE7COWC7QIYGARAA2D5A, seg_pos={0 0}, upload=60693]: Done uploading piece. err: ecclient: upload failed (node:125SMhDsEZruNu6LkmQn2M7R6WzvXTdk86DRsKf522u8BNiANq3, address:104.36.82.54:4171): protocol: expected piece hash; context deadline exceeded; EOF Jun 21 20:11:05.567869019 (75h47m41.131192033s) [noise=true , address=104.36.82.10:6087 , piece_id=5FQG7BPZC373PFB54HYXSG2SVKOBBBSGZCLC4XDJEYQFOLQY74MA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: ecclient: upload failed (node:1S9k5eejJcxkiG5pXjjBZm18uP8k5jCGfWyeW2QQxmuRU8ZzRA, address:104.36.82.10:6087): protocol: write tcp 159.223.143.39:38550->104.36.82.10:6087: use of closed network connection; write tcp 159.223.143.39:38550->104.36.82.10:6087: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:38550->104.36.82.10:6087: use of closed network connection Jun 21 20:11:05.567932150 (75h47m41.131255161s) [seg_pos={0 0}, upload=60693]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jun 21 20:11:05.593626175 (75h47m41.156949181s) [upload=60693]: Done uploading segments... Jun 21 20:11:05.593657274 (75h47m41.156980265s) [upload=60693]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 17.666629177s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [.....X...........X..........................................] duration: 10.231440768s FAIL (2025-06-21 20:43:11 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== pload=60693]: Starting upload Jun 21 20:10:05.602866542 (75h46m41.166189557s) [upload=60693]: Done starting upload Jun 21 20:10:05.603116545 (75h46m41.166439547s) [upload=60693]: Uploading segments... Jun 21 20:10:05.603495389 (75h46m41.166818397s) [upload=60693]: Got next segment. Inline: false Jun 21 20:10:05.603566566 (75h46m41.166889558s) [upload=60693]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 21 20:10:05.603573280 (75h46m41.166896286s) [upload=60693]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 21 20:10:05.639130162 (75h46m41.202453164s) [seg_pos={0 0}, upload=60693]: Begin upload segment... Jun 21 20:10:05.639231070 (75h46m41.202554057s) [seg_pos={0 0}, upload=60693]: Done begin upload segment. Jun 21 20:10:05.639292724 (75h46m41.202615728s) [noise=true , address=104.36.82.56:4253 , piece_id=5PMVGLZETMXCQ77K3I6PXACRIMBGR2CV5JGUSU5KUMYFNXVGA2GA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.639338066 (75h46m41.202661071s) [noise=true , address=104.36.82.53:4126 , piece_id=2MKRQJEPGDJKDM7VWY5M4M6NORILAIEYYATJ7HVZEOBOMXOI2LBA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.639416264 (75h46m41.202739267s) [noise=true , address=104.36.82.5:5869 , piece_id=HAEHXXZASV3DHT3QN7NTZOS6XZKHNUWBKCHWJJWSTJKXAKJKKBAA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.639873098 (75h46m41.203196115s) [noise=true , address=104.36.82.52:4111 , piece_id=HB7QAWN6CGNIIT4HD2GZ7JDQP5NXIJ5I2USBM66YOKTYENNOWG5A, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.639887296 (75h46m41.203210304s) [noise=true , address=104.36.82.10:6087 , piece_id=5FQG7BPZC373PFB54HYXSG2SVKOBBBSGZCLC4XDJEYQFOLQY74MA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.640266623 (75h46m41.203589616s) [noise=true , address=104.36.82.13:6196 , piece_id=BCHMQO5KKT5BLNWQGOHH3V43LGSE3BUQCI2SY4TTO5M5GQNOCTMQ, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.640352274 (75h46m41.203675286s) [noise=true , address=104.36.82.37:6663 , piece_id=X7LWYIIHJFU2T2PHYR3R5TNT22UUPHEHQWZH7ZJMLS4AH4PJ6AIA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.640659767 (75h46m41.203982779s) [noise=true , address=104.36.82.42:6807 , piece_id=ZBQZPXUDJ4VBCOSVKWAJ6SK6UD7SLPON7Y4SSCK2DQ5STFG4XCNQ, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.640961701 (75h46m41.20428471s) [noise=true , address=45.91.116.6:4982 , piece_id=VP45R6KXNH5U5SGUQUMAPIBY7KFDPBIR3S3VRIX27EAT6VLFBZ7Q, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.641159578 (75h46m41.204482589s) [noise=true , address=104.36.82.9:6039 , piece_id=I4AB6N27MYYUHWAZLJJUIBFBLOQFHDIM3YLYWPVRPNUV7VMYS2SA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.641588126 (75h46m41.204911136s) [noise=true , address=77.90.31.19:4853 , piece_id=VFEON5T2KSWM2KSLTVRZTXYL42JWENZYUEUPG3LMODGZWBMVGGPQ, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.641863297 (75h46m41.205186305s) [noise=true , address=77.90.31.12:4603 , piece_id=BMUIC6KNGEDL37436GLSXW5XOO6VYOPNOSFGQCYVHSENVI2PJ7KQ, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.642124427 (75h46m41.205447437s) [noise=true , address=104.36.82.49:4982 , piece_id=UNLV5UDNWULEIPE4QSQHHZQPK3BQKFRB5T3H6GP7ZQ5QGSNK5V6A, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.642285917 (75h46m41.205608925s) [noise=true , address=104.36.82.54:4171 , piece_id=MUMABKARK4KKUKWW6XDXCQVT6O7SLFHGHE7COWC7QIYGARAA2D5A, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.642656750 (75h46m41.205979758s) [noise=true , address=104.36.82.14:6232 , piece_id=A2ZLUUZKRTBJRJ267QSK36C3LFZV7FPAYDDAXGIWDTNEGNV6MRRA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.643565583 (75h46m41.206888597s) [noise=true , address=79.127.205.245:4231 , piece_id=EJK46XUUVI5NWOAQJEW4QIJG3KRXZHB4YOQUJRCS2MIOSWKGTNLQ, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.644620291 (75h46m41.207943303s) [noise=true , address=77.90.31.14:4670 , piece_id=7FNANMUKPM3N6YWRAOAIM4WJZJUTNF7ZBI6ZUBNEPGNGMFWOE24A, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.648842639 (75h46m41.212165631s) [noise=true , address=104.36.82.55:4224 , piece_id=WRVNGSJV5BX4GJERLXNPAS7J7BZ5WFTM7R5H257YRCG5SP2HIYMA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.825828239 (75h46m41.389151241s) [noise=true , address=104.36.82.53:4126 , piece_id=2MKRQJEPGDJKDM7VWY5M4M6NORILAIEYYATJ7HVZEOBOMXOI2LBA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: ecclient: upload failed (node:12dMWFrq2NMpnMzuPAphWerntC5kkrcnzRJZBA116uzJzyvk6fH, address:104.36.82.53:4126): protocol: storage node overloaded, request limit: 2000 Jun 21 20:10:05.834666043 (75h46m41.397989033s) [upload=60693]: Next returned nil segment Jun 21 20:10:05.834686295 (75h46m41.398009302s) [upload=60693]: Waiting for error group managing segments... Jun 21 20:10:05.877010488 (75h46m41.440333486s) [noise=true , address=45.91.116.6:4982 , piece_id=VP45R6KXNH5U5SGUQUMAPIBY7KFDPBIR3S3VRIX27EAT6VLFBZ7Q, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:05.886336964 (75h46m41.449659967s) [noise=true , address=104.36.82.9:6039 , piece_id=I4AB6N27MYYUHWAZLJJUIBFBLOQFHDIM3YLYWPVRPNUV7VMYS2SA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: ecclient: upload failed (node:12P11WVW9mVXw5ra2gZ3Rezon6wZi8GrN5CchQN3MiVU4ts3L6V, address:104.36.82.9:6039): protocol: storage node overloaded, request limit: 2000 Jun 21 20:10:08.064514542 (75h46m43.627837553s) [noise=true , address=77.90.31.14:4670 , piece_id=7FNANMUKPM3N6YWRAOAIM4WJZJUTNF7ZBI6ZUBNEPGNGMFWOE24A, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:08.099098391 (75h46m43.662421396s) [noise=true , address=79.127.205.245:4231 , piece_id=EJK46XUUVI5NWOAQJEW4QIJG3KRXZHB4YOQUJRCS2MIOSWKGTNLQ, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:08.489461749 (75h46m44.052784752s) [noise=true , address=104.36.82.42:6807 , piece_id=ZBQZPXUDJ4VBCOSVKWAJ6SK6UD7SLPON7Y4SSCK2DQ5STFG4XCNQ, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:08.650922093 (75h46m44.214245093s) [noise=true , address=77.90.31.19:4853 , piece_id=VFEON5T2KSWM2KSLTVRZTXYL42JWENZYUEUPG3LMODGZWBMVGGPQ, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:09.917746439 (75h46m45.481069441s) [noise=true , address=104.36.82.13:6196 , piece_id=BCHMQO5KKT5BLNWQGOHH3V43LGSE3BUQCI2SY4TTO5M5GQNOCTMQ, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:10.830722215 (75h46m46.394045216s) [noise=true , address=104.36.82.5:5869 , piece_id=HAEHXXZASV3DHT3QN7NTZOS6XZKHNUWBKCHWJJWSTJKXAKJKKBAA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:12.738783450 (75h46m48.302106452s) [noise=true , address=77.90.31.12:4603 , piece_id=BMUIC6KNGEDL37436GLSXW5XOO6VYOPNOSFGQCYVHSENVI2PJ7KQ, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:15.460927713 (75h46m51.024250723s) [noise=true , address=104.36.82.52:4111 , piece_id=HB7QAWN6CGNIIT4HD2GZ7JDQP5NXIJ5I2USBM66YOKTYENNOWG5A, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:18.393817968 (75h46m53.95714096s) [noise=true , address=104.36.82.55:4224 , piece_id=WRVNGSJV5BX4GJERLXNPAS7J7BZ5WFTM7R5H257YRCG5SP2HIYMA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:19.549174143 (75h46m55.112497151s) [noise=true , address=104.36.82.56:4253 , piece_id=5PMVGLZETMXCQ77K3I6PXACRIMBGR2CV5JGUSU5KUMYFNXVGA2GA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:22.511944593 (75h46m58.075267584s) [noise=true , address=104.36.82.37:6663 , piece_id=X7LWYIIHJFU2T2PHYR3R5TNT22UUPHEHQWZH7ZJMLS4AH4PJ6AIA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:48.027197558 (75h47m23.590520562s) [noise=true , address=104.36.82.14:6232 , piece_id=A2ZLUUZKRTBJRJ267QSK36C3LFZV7FPAYDDAXGIWDTNEGNV6MRRA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:11:05.567692582 (75h47m41.131015574s) [noise=true , address=104.36.82.49:4982 , piece_id=UNLV5UDNWULEIPE4QSQHHZQPK3BQKFRB5T3H6GP7ZQ5QGSNK5V6A, seg_pos={0 0}, upload=60693]: Done uploading piece. err: ecclient: upload failed (node:1NxoYJXCXmTvQncbkXt6yH7mhEBBrzVdzhgL5J6W5TD4N3pkaG, address:104.36.82.49:4982): protocol: expected piece hash; context deadline exceeded; EOF Jun 21 20:11:05.567759198 (75h47m41.131082195s) [noise=true , address=104.36.82.54:4171 , piece_id=MUMABKARK4KKUKWW6XDXCQVT6O7SLFHGHE7COWC7QIYGARAA2D5A, seg_pos={0 0}, upload=60693]: Done uploading piece. err: ecclient: upload failed (node:125SMhDsEZruNu6LkmQn2M7R6WzvXTdk86DRsKf522u8BNiANq3, address:104.36.82.54:4171): protocol: expected piece hash; context deadline exceeded; EOF Jun 21 20:11:05.567869019 (75h47m41.131192033s) [noise=true , address=104.36.82.10:6087 , piece_id=5FQG7BPZC373PFB54HYXSG2SVKOBBBSGZCLC4XDJEYQFOLQY74MA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: ecclient: upload failed (node:1S9k5eejJcxkiG5pXjjBZm18uP8k5jCGfWyeW2QQxmuRU8ZzRA, address:104.36.82.10:6087): protocol: write tcp 159.223.143.39:38550->104.36.82.10:6087: use of closed network connection; write tcp 159.223.143.39:38550->104.36.82.10:6087: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:38550->104.36.82.10:6087: use of closed network connection Jun 21 20:11:05.567932150 (75h47m41.131255161s) [seg_pos={0 0}, upload=60693]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jun 21 20:11:05.593626175 (75h47m41.156949181s) [upload=60693]: Done uploading segments... Jun 21 20:11:05.593657274 (75h47m41.156980265s) [upload=60693]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 10.231504788s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [......X...........X.........................................] duration: 21.247889252s FAIL (2025-06-21 20:42:57 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== pload=60693]: Starting upload Jun 21 20:10:05.602866542 (75h46m41.166189557s) [upload=60693]: Done starting upload Jun 21 20:10:05.603116545 (75h46m41.166439547s) [upload=60693]: Uploading segments... Jun 21 20:10:05.603495389 (75h46m41.166818397s) [upload=60693]: Got next segment. Inline: false Jun 21 20:10:05.603566566 (75h46m41.166889558s) [upload=60693]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 21 20:10:05.603573280 (75h46m41.166896286s) [upload=60693]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 21 20:10:05.639130162 (75h46m41.202453164s) [seg_pos={0 0}, upload=60693]: Begin upload segment... Jun 21 20:10:05.639231070 (75h46m41.202554057s) [seg_pos={0 0}, upload=60693]: Done begin upload segment. Jun 21 20:10:05.639292724 (75h46m41.202615728s) [noise=true , address=104.36.82.56:4253 , piece_id=5PMVGLZETMXCQ77K3I6PXACRIMBGR2CV5JGUSU5KUMYFNXVGA2GA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.639338066 (75h46m41.202661071s) [noise=true , address=104.36.82.53:4126 , piece_id=2MKRQJEPGDJKDM7VWY5M4M6NORILAIEYYATJ7HVZEOBOMXOI2LBA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.639416264 (75h46m41.202739267s) [noise=true , address=104.36.82.5:5869 , piece_id=HAEHXXZASV3DHT3QN7NTZOS6XZKHNUWBKCHWJJWSTJKXAKJKKBAA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.639873098 (75h46m41.203196115s) [noise=true , address=104.36.82.52:4111 , piece_id=HB7QAWN6CGNIIT4HD2GZ7JDQP5NXIJ5I2USBM66YOKTYENNOWG5A, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.639887296 (75h46m41.203210304s) [noise=true , address=104.36.82.10:6087 , piece_id=5FQG7BPZC373PFB54HYXSG2SVKOBBBSGZCLC4XDJEYQFOLQY74MA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.640266623 (75h46m41.203589616s) [noise=true , address=104.36.82.13:6196 , piece_id=BCHMQO5KKT5BLNWQGOHH3V43LGSE3BUQCI2SY4TTO5M5GQNOCTMQ, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.640352274 (75h46m41.203675286s) [noise=true , address=104.36.82.37:6663 , piece_id=X7LWYIIHJFU2T2PHYR3R5TNT22UUPHEHQWZH7ZJMLS4AH4PJ6AIA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.640659767 (75h46m41.203982779s) [noise=true , address=104.36.82.42:6807 , piece_id=ZBQZPXUDJ4VBCOSVKWAJ6SK6UD7SLPON7Y4SSCK2DQ5STFG4XCNQ, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.640961701 (75h46m41.20428471s) [noise=true , address=45.91.116.6:4982 , piece_id=VP45R6KXNH5U5SGUQUMAPIBY7KFDPBIR3S3VRIX27EAT6VLFBZ7Q, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.641159578 (75h46m41.204482589s) [noise=true , address=104.36.82.9:6039 , piece_id=I4AB6N27MYYUHWAZLJJUIBFBLOQFHDIM3YLYWPVRPNUV7VMYS2SA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.641588126 (75h46m41.204911136s) [noise=true , address=77.90.31.19:4853 , piece_id=VFEON5T2KSWM2KSLTVRZTXYL42JWENZYUEUPG3LMODGZWBMVGGPQ, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.641863297 (75h46m41.205186305s) [noise=true , address=77.90.31.12:4603 , piece_id=BMUIC6KNGEDL37436GLSXW5XOO6VYOPNOSFGQCYVHSENVI2PJ7KQ, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.642124427 (75h46m41.205447437s) [noise=true , address=104.36.82.49:4982 , piece_id=UNLV5UDNWULEIPE4QSQHHZQPK3BQKFRB5T3H6GP7ZQ5QGSNK5V6A, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.642285917 (75h46m41.205608925s) [noise=true , address=104.36.82.54:4171 , piece_id=MUMABKARK4KKUKWW6XDXCQVT6O7SLFHGHE7COWC7QIYGARAA2D5A, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.642656750 (75h46m41.205979758s) [noise=true , address=104.36.82.14:6232 , piece_id=A2ZLUUZKRTBJRJ267QSK36C3LFZV7FPAYDDAXGIWDTNEGNV6MRRA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.643565583 (75h46m41.206888597s) [noise=true , address=79.127.205.245:4231 , piece_id=EJK46XUUVI5NWOAQJEW4QIJG3KRXZHB4YOQUJRCS2MIOSWKGTNLQ, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.644620291 (75h46m41.207943303s) [noise=true , address=77.90.31.14:4670 , piece_id=7FNANMUKPM3N6YWRAOAIM4WJZJUTNF7ZBI6ZUBNEPGNGMFWOE24A, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.648842639 (75h46m41.212165631s) [noise=true , address=104.36.82.55:4224 , piece_id=WRVNGSJV5BX4GJERLXNPAS7J7BZ5WFTM7R5H257YRCG5SP2HIYMA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.825828239 (75h46m41.389151241s) [noise=true , address=104.36.82.53:4126 , piece_id=2MKRQJEPGDJKDM7VWY5M4M6NORILAIEYYATJ7HVZEOBOMXOI2LBA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: ecclient: upload failed (node:12dMWFrq2NMpnMzuPAphWerntC5kkrcnzRJZBA116uzJzyvk6fH, address:104.36.82.53:4126): protocol: storage node overloaded, request limit: 2000 Jun 21 20:10:05.834666043 (75h46m41.397989033s) [upload=60693]: Next returned nil segment Jun 21 20:10:05.834686295 (75h46m41.398009302s) [upload=60693]: Waiting for error group managing segments... Jun 21 20:10:05.877010488 (75h46m41.440333486s) [noise=true , address=45.91.116.6:4982 , piece_id=VP45R6KXNH5U5SGUQUMAPIBY7KFDPBIR3S3VRIX27EAT6VLFBZ7Q, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:05.886336964 (75h46m41.449659967s) [noise=true , address=104.36.82.9:6039 , piece_id=I4AB6N27MYYUHWAZLJJUIBFBLOQFHDIM3YLYWPVRPNUV7VMYS2SA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: ecclient: upload failed (node:12P11WVW9mVXw5ra2gZ3Rezon6wZi8GrN5CchQN3MiVU4ts3L6V, address:104.36.82.9:6039): protocol: storage node overloaded, request limit: 2000 Jun 21 20:10:08.064514542 (75h46m43.627837553s) [noise=true , address=77.90.31.14:4670 , piece_id=7FNANMUKPM3N6YWRAOAIM4WJZJUTNF7ZBI6ZUBNEPGNGMFWOE24A, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:08.099098391 (75h46m43.662421396s) [noise=true , address=79.127.205.245:4231 , piece_id=EJK46XUUVI5NWOAQJEW4QIJG3KRXZHB4YOQUJRCS2MIOSWKGTNLQ, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:08.489461749 (75h46m44.052784752s) [noise=true , address=104.36.82.42:6807 , piece_id=ZBQZPXUDJ4VBCOSVKWAJ6SK6UD7SLPON7Y4SSCK2DQ5STFG4XCNQ, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:08.650922093 (75h46m44.214245093s) [noise=true , address=77.90.31.19:4853 , piece_id=VFEON5T2KSWM2KSLTVRZTXYL42JWENZYUEUPG3LMODGZWBMVGGPQ, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:09.917746439 (75h46m45.481069441s) [noise=true , address=104.36.82.13:6196 , piece_id=BCHMQO5KKT5BLNWQGOHH3V43LGSE3BUQCI2SY4TTO5M5GQNOCTMQ, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:10.830722215 (75h46m46.394045216s) [noise=true , address=104.36.82.5:5869 , piece_id=HAEHXXZASV3DHT3QN7NTZOS6XZKHNUWBKCHWJJWSTJKXAKJKKBAA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:12.738783450 (75h46m48.302106452s) [noise=true , address=77.90.31.12:4603 , piece_id=BMUIC6KNGEDL37436GLSXW5XOO6VYOPNOSFGQCYVHSENVI2PJ7KQ, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:15.460927713 (75h46m51.024250723s) [noise=true , address=104.36.82.52:4111 , piece_id=HB7QAWN6CGNIIT4HD2GZ7JDQP5NXIJ5I2USBM66YOKTYENNOWG5A, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:18.393817968 (75h46m53.95714096s) [noise=true , address=104.36.82.55:4224 , piece_id=WRVNGSJV5BX4GJERLXNPAS7J7BZ5WFTM7R5H257YRCG5SP2HIYMA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:19.549174143 (75h46m55.112497151s) [noise=true , address=104.36.82.56:4253 , piece_id=5PMVGLZETMXCQ77K3I6PXACRIMBGR2CV5JGUSU5KUMYFNXVGA2GA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:22.511944593 (75h46m58.075267584s) [noise=true , address=104.36.82.37:6663 , piece_id=X7LWYIIHJFU2T2PHYR3R5TNT22UUPHEHQWZH7ZJMLS4AH4PJ6AIA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:48.027197558 (75h47m23.590520562s) [noise=true , address=104.36.82.14:6232 , piece_id=A2ZLUUZKRTBJRJ267QSK36C3LFZV7FPAYDDAXGIWDTNEGNV6MRRA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:11:05.567692582 (75h47m41.131015574s) [noise=true , address=104.36.82.49:4982 , piece_id=UNLV5UDNWULEIPE4QSQHHZQPK3BQKFRB5T3H6GP7ZQ5QGSNK5V6A, seg_pos={0 0}, upload=60693]: Done uploading piece. err: ecclient: upload failed (node:1NxoYJXCXmTvQncbkXt6yH7mhEBBrzVdzhgL5J6W5TD4N3pkaG, address:104.36.82.49:4982): protocol: expected piece hash; context deadline exceeded; EOF Jun 21 20:11:05.567759198 (75h47m41.131082195s) [noise=true , address=104.36.82.54:4171 , piece_id=MUMABKARK4KKUKWW6XDXCQVT6O7SLFHGHE7COWC7QIYGARAA2D5A, seg_pos={0 0}, upload=60693]: Done uploading piece. err: ecclient: upload failed (node:125SMhDsEZruNu6LkmQn2M7R6WzvXTdk86DRsKf522u8BNiANq3, address:104.36.82.54:4171): protocol: expected piece hash; context deadline exceeded; EOF Jun 21 20:11:05.567869019 (75h47m41.131192033s) [noise=true , address=104.36.82.10:6087 , piece_id=5FQG7BPZC373PFB54HYXSG2SVKOBBBSGZCLC4XDJEYQFOLQY74MA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: ecclient: upload failed (node:1S9k5eejJcxkiG5pXjjBZm18uP8k5jCGfWyeW2QQxmuRU8ZzRA, address:104.36.82.10:6087): protocol: write tcp 159.223.143.39:38550->104.36.82.10:6087: use of closed network connection; write tcp 159.223.143.39:38550->104.36.82.10:6087: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:38550->104.36.82.10:6087: use of closed network connection Jun 21 20:11:05.567932150 (75h47m41.131255161s) [seg_pos={0 0}, upload=60693]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jun 21 20:11:05.593626175 (75h47m41.156949181s) [upload=60693]: Done uploading segments... Jun 21 20:11:05.593657274 (75h47m41.156980265s) [upload=60693]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 21.247944831s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [.......X...........X........................................] duration: 13.320181631s FAIL (2025-06-21 20:41:35 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== pload=60693]: Starting upload Jun 21 20:10:05.602866542 (75h46m41.166189557s) [upload=60693]: Done starting upload Jun 21 20:10:05.603116545 (75h46m41.166439547s) [upload=60693]: Uploading segments... Jun 21 20:10:05.603495389 (75h46m41.166818397s) [upload=60693]: Got next segment. Inline: false Jun 21 20:10:05.603566566 (75h46m41.166889558s) [upload=60693]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 21 20:10:05.603573280 (75h46m41.166896286s) [upload=60693]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 21 20:10:05.639130162 (75h46m41.202453164s) [seg_pos={0 0}, upload=60693]: Begin upload segment... Jun 21 20:10:05.639231070 (75h46m41.202554057s) [seg_pos={0 0}, upload=60693]: Done begin upload segment. Jun 21 20:10:05.639292724 (75h46m41.202615728s) [noise=true , address=104.36.82.56:4253 , piece_id=5PMVGLZETMXCQ77K3I6PXACRIMBGR2CV5JGUSU5KUMYFNXVGA2GA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.639338066 (75h46m41.202661071s) [noise=true , address=104.36.82.53:4126 , piece_id=2MKRQJEPGDJKDM7VWY5M4M6NORILAIEYYATJ7HVZEOBOMXOI2LBA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.639416264 (75h46m41.202739267s) [noise=true , address=104.36.82.5:5869 , piece_id=HAEHXXZASV3DHT3QN7NTZOS6XZKHNUWBKCHWJJWSTJKXAKJKKBAA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.639873098 (75h46m41.203196115s) [noise=true , address=104.36.82.52:4111 , piece_id=HB7QAWN6CGNIIT4HD2GZ7JDQP5NXIJ5I2USBM66YOKTYENNOWG5A, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.639887296 (75h46m41.203210304s) [noise=true , address=104.36.82.10:6087 , piece_id=5FQG7BPZC373PFB54HYXSG2SVKOBBBSGZCLC4XDJEYQFOLQY74MA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.640266623 (75h46m41.203589616s) [noise=true , address=104.36.82.13:6196 , piece_id=BCHMQO5KKT5BLNWQGOHH3V43LGSE3BUQCI2SY4TTO5M5GQNOCTMQ, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.640352274 (75h46m41.203675286s) [noise=true , address=104.36.82.37:6663 , piece_id=X7LWYIIHJFU2T2PHYR3R5TNT22UUPHEHQWZH7ZJMLS4AH4PJ6AIA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.640659767 (75h46m41.203982779s) [noise=true , address=104.36.82.42:6807 , piece_id=ZBQZPXUDJ4VBCOSVKWAJ6SK6UD7SLPON7Y4SSCK2DQ5STFG4XCNQ, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.640961701 (75h46m41.20428471s) [noise=true , address=45.91.116.6:4982 , piece_id=VP45R6KXNH5U5SGUQUMAPIBY7KFDPBIR3S3VRIX27EAT6VLFBZ7Q, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.641159578 (75h46m41.204482589s) [noise=true , address=104.36.82.9:6039 , piece_id=I4AB6N27MYYUHWAZLJJUIBFBLOQFHDIM3YLYWPVRPNUV7VMYS2SA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.641588126 (75h46m41.204911136s) [noise=true , address=77.90.31.19:4853 , piece_id=VFEON5T2KSWM2KSLTVRZTXYL42JWENZYUEUPG3LMODGZWBMVGGPQ, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.641863297 (75h46m41.205186305s) [noise=true , address=77.90.31.12:4603 , piece_id=BMUIC6KNGEDL37436GLSXW5XOO6VYOPNOSFGQCYVHSENVI2PJ7KQ, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.642124427 (75h46m41.205447437s) [noise=true , address=104.36.82.49:4982 , piece_id=UNLV5UDNWULEIPE4QSQHHZQPK3BQKFRB5T3H6GP7ZQ5QGSNK5V6A, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.642285917 (75h46m41.205608925s) [noise=true , address=104.36.82.54:4171 , piece_id=MUMABKARK4KKUKWW6XDXCQVT6O7SLFHGHE7COWC7QIYGARAA2D5A, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.642656750 (75h46m41.205979758s) [noise=true , address=104.36.82.14:6232 , piece_id=A2ZLUUZKRTBJRJ267QSK36C3LFZV7FPAYDDAXGIWDTNEGNV6MRRA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.643565583 (75h46m41.206888597s) [noise=true , address=79.127.205.245:4231 , piece_id=EJK46XUUVI5NWOAQJEW4QIJG3KRXZHB4YOQUJRCS2MIOSWKGTNLQ, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.644620291 (75h46m41.207943303s) [noise=true , address=77.90.31.14:4670 , piece_id=7FNANMUKPM3N6YWRAOAIM4WJZJUTNF7ZBI6ZUBNEPGNGMFWOE24A, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.648842639 (75h46m41.212165631s) [noise=true , address=104.36.82.55:4224 , piece_id=WRVNGSJV5BX4GJERLXNPAS7J7BZ5WFTM7R5H257YRCG5SP2HIYMA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.825828239 (75h46m41.389151241s) [noise=true , address=104.36.82.53:4126 , piece_id=2MKRQJEPGDJKDM7VWY5M4M6NORILAIEYYATJ7HVZEOBOMXOI2LBA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: ecclient: upload failed (node:12dMWFrq2NMpnMzuPAphWerntC5kkrcnzRJZBA116uzJzyvk6fH, address:104.36.82.53:4126): protocol: storage node overloaded, request limit: 2000 Jun 21 20:10:05.834666043 (75h46m41.397989033s) [upload=60693]: Next returned nil segment Jun 21 20:10:05.834686295 (75h46m41.398009302s) [upload=60693]: Waiting for error group managing segments... Jun 21 20:10:05.877010488 (75h46m41.440333486s) [noise=true , address=45.91.116.6:4982 , piece_id=VP45R6KXNH5U5SGUQUMAPIBY7KFDPBIR3S3VRIX27EAT6VLFBZ7Q, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:05.886336964 (75h46m41.449659967s) [noise=true , address=104.36.82.9:6039 , piece_id=I4AB6N27MYYUHWAZLJJUIBFBLOQFHDIM3YLYWPVRPNUV7VMYS2SA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: ecclient: upload failed (node:12P11WVW9mVXw5ra2gZ3Rezon6wZi8GrN5CchQN3MiVU4ts3L6V, address:104.36.82.9:6039): protocol: storage node overloaded, request limit: 2000 Jun 21 20:10:08.064514542 (75h46m43.627837553s) [noise=true , address=77.90.31.14:4670 , piece_id=7FNANMUKPM3N6YWRAOAIM4WJZJUTNF7ZBI6ZUBNEPGNGMFWOE24A, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:08.099098391 (75h46m43.662421396s) [noise=true , address=79.127.205.245:4231 , piece_id=EJK46XUUVI5NWOAQJEW4QIJG3KRXZHB4YOQUJRCS2MIOSWKGTNLQ, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:08.489461749 (75h46m44.052784752s) [noise=true , address=104.36.82.42:6807 , piece_id=ZBQZPXUDJ4VBCOSVKWAJ6SK6UD7SLPON7Y4SSCK2DQ5STFG4XCNQ, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:08.650922093 (75h46m44.214245093s) [noise=true , address=77.90.31.19:4853 , piece_id=VFEON5T2KSWM2KSLTVRZTXYL42JWENZYUEUPG3LMODGZWBMVGGPQ, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:09.917746439 (75h46m45.481069441s) [noise=true , address=104.36.82.13:6196 , piece_id=BCHMQO5KKT5BLNWQGOHH3V43LGSE3BUQCI2SY4TTO5M5GQNOCTMQ, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:10.830722215 (75h46m46.394045216s) [noise=true , address=104.36.82.5:5869 , piece_id=HAEHXXZASV3DHT3QN7NTZOS6XZKHNUWBKCHWJJWSTJKXAKJKKBAA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:12.738783450 (75h46m48.302106452s) [noise=true , address=77.90.31.12:4603 , piece_id=BMUIC6KNGEDL37436GLSXW5XOO6VYOPNOSFGQCYVHSENVI2PJ7KQ, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:15.460927713 (75h46m51.024250723s) [noise=true , address=104.36.82.52:4111 , piece_id=HB7QAWN6CGNIIT4HD2GZ7JDQP5NXIJ5I2USBM66YOKTYENNOWG5A, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:18.393817968 (75h46m53.95714096s) [noise=true , address=104.36.82.55:4224 , piece_id=WRVNGSJV5BX4GJERLXNPAS7J7BZ5WFTM7R5H257YRCG5SP2HIYMA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:19.549174143 (75h46m55.112497151s) [noise=true , address=104.36.82.56:4253 , piece_id=5PMVGLZETMXCQ77K3I6PXACRIMBGR2CV5JGUSU5KUMYFNXVGA2GA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:22.511944593 (75h46m58.075267584s) [noise=true , address=104.36.82.37:6663 , piece_id=X7LWYIIHJFU2T2PHYR3R5TNT22UUPHEHQWZH7ZJMLS4AH4PJ6AIA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:48.027197558 (75h47m23.590520562s) [noise=true , address=104.36.82.14:6232 , piece_id=A2ZLUUZKRTBJRJ267QSK36C3LFZV7FPAYDDAXGIWDTNEGNV6MRRA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:11:05.567692582 (75h47m41.131015574s) [noise=true , address=104.36.82.49:4982 , piece_id=UNLV5UDNWULEIPE4QSQHHZQPK3BQKFRB5T3H6GP7ZQ5QGSNK5V6A, seg_pos={0 0}, upload=60693]: Done uploading piece. err: ecclient: upload failed (node:1NxoYJXCXmTvQncbkXt6yH7mhEBBrzVdzhgL5J6W5TD4N3pkaG, address:104.36.82.49:4982): protocol: expected piece hash; context deadline exceeded; EOF Jun 21 20:11:05.567759198 (75h47m41.131082195s) [noise=true , address=104.36.82.54:4171 , piece_id=MUMABKARK4KKUKWW6XDXCQVT6O7SLFHGHE7COWC7QIYGARAA2D5A, seg_pos={0 0}, upload=60693]: Done uploading piece. err: ecclient: upload failed (node:125SMhDsEZruNu6LkmQn2M7R6WzvXTdk86DRsKf522u8BNiANq3, address:104.36.82.54:4171): protocol: expected piece hash; context deadline exceeded; EOF Jun 21 20:11:05.567869019 (75h47m41.131192033s) [noise=true , address=104.36.82.10:6087 , piece_id=5FQG7BPZC373PFB54HYXSG2SVKOBBBSGZCLC4XDJEYQFOLQY74MA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: ecclient: upload failed (node:1S9k5eejJcxkiG5pXjjBZm18uP8k5jCGfWyeW2QQxmuRU8ZzRA, address:104.36.82.10:6087): protocol: write tcp 159.223.143.39:38550->104.36.82.10:6087: use of closed network connection; write tcp 159.223.143.39:38550->104.36.82.10:6087: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:38550->104.36.82.10:6087: use of closed network connection Jun 21 20:11:05.567932150 (75h47m41.131255161s) [seg_pos={0 0}, upload=60693]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jun 21 20:11:05.593626175 (75h47m41.156949181s) [upload=60693]: Done uploading segments... Jun 21 20:11:05.593657274 (75h47m41.156980265s) [upload=60693]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 13.320204832s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [........X...........X.......................................] duration: 24.423431786s FAIL (2025-06-21 20:41:22 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== pload=60693]: Starting upload Jun 21 20:10:05.602866542 (75h46m41.166189557s) [upload=60693]: Done starting upload Jun 21 20:10:05.603116545 (75h46m41.166439547s) [upload=60693]: Uploading segments... Jun 21 20:10:05.603495389 (75h46m41.166818397s) [upload=60693]: Got next segment. Inline: false Jun 21 20:10:05.603566566 (75h46m41.166889558s) [upload=60693]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 21 20:10:05.603573280 (75h46m41.166896286s) [upload=60693]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 21 20:10:05.639130162 (75h46m41.202453164s) [seg_pos={0 0}, upload=60693]: Begin upload segment... Jun 21 20:10:05.639231070 (75h46m41.202554057s) [seg_pos={0 0}, upload=60693]: Done begin upload segment. Jun 21 20:10:05.639292724 (75h46m41.202615728s) [noise=true , address=104.36.82.56:4253 , piece_id=5PMVGLZETMXCQ77K3I6PXACRIMBGR2CV5JGUSU5KUMYFNXVGA2GA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.639338066 (75h46m41.202661071s) [noise=true , address=104.36.82.53:4126 , piece_id=2MKRQJEPGDJKDM7VWY5M4M6NORILAIEYYATJ7HVZEOBOMXOI2LBA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.639416264 (75h46m41.202739267s) [noise=true , address=104.36.82.5:5869 , piece_id=HAEHXXZASV3DHT3QN7NTZOS6XZKHNUWBKCHWJJWSTJKXAKJKKBAA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.639873098 (75h46m41.203196115s) [noise=true , address=104.36.82.52:4111 , piece_id=HB7QAWN6CGNIIT4HD2GZ7JDQP5NXIJ5I2USBM66YOKTYENNOWG5A, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.639887296 (75h46m41.203210304s) [noise=true , address=104.36.82.10:6087 , piece_id=5FQG7BPZC373PFB54HYXSG2SVKOBBBSGZCLC4XDJEYQFOLQY74MA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.640266623 (75h46m41.203589616s) [noise=true , address=104.36.82.13:6196 , piece_id=BCHMQO5KKT5BLNWQGOHH3V43LGSE3BUQCI2SY4TTO5M5GQNOCTMQ, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.640352274 (75h46m41.203675286s) [noise=true , address=104.36.82.37:6663 , piece_id=X7LWYIIHJFU2T2PHYR3R5TNT22UUPHEHQWZH7ZJMLS4AH4PJ6AIA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.640659767 (75h46m41.203982779s) [noise=true , address=104.36.82.42:6807 , piece_id=ZBQZPXUDJ4VBCOSVKWAJ6SK6UD7SLPON7Y4SSCK2DQ5STFG4XCNQ, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.640961701 (75h46m41.20428471s) [noise=true , address=45.91.116.6:4982 , piece_id=VP45R6KXNH5U5SGUQUMAPIBY7KFDPBIR3S3VRIX27EAT6VLFBZ7Q, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.641159578 (75h46m41.204482589s) [noise=true , address=104.36.82.9:6039 , piece_id=I4AB6N27MYYUHWAZLJJUIBFBLOQFHDIM3YLYWPVRPNUV7VMYS2SA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.641588126 (75h46m41.204911136s) [noise=true , address=77.90.31.19:4853 , piece_id=VFEON5T2KSWM2KSLTVRZTXYL42JWENZYUEUPG3LMODGZWBMVGGPQ, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.641863297 (75h46m41.205186305s) [noise=true , address=77.90.31.12:4603 , piece_id=BMUIC6KNGEDL37436GLSXW5XOO6VYOPNOSFGQCYVHSENVI2PJ7KQ, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.642124427 (75h46m41.205447437s) [noise=true , address=104.36.82.49:4982 , piece_id=UNLV5UDNWULEIPE4QSQHHZQPK3BQKFRB5T3H6GP7ZQ5QGSNK5V6A, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.642285917 (75h46m41.205608925s) [noise=true , address=104.36.82.54:4171 , piece_id=MUMABKARK4KKUKWW6XDXCQVT6O7SLFHGHE7COWC7QIYGARAA2D5A, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.642656750 (75h46m41.205979758s) [noise=true , address=104.36.82.14:6232 , piece_id=A2ZLUUZKRTBJRJ267QSK36C3LFZV7FPAYDDAXGIWDTNEGNV6MRRA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.643565583 (75h46m41.206888597s) [noise=true , address=79.127.205.245:4231 , piece_id=EJK46XUUVI5NWOAQJEW4QIJG3KRXZHB4YOQUJRCS2MIOSWKGTNLQ, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.644620291 (75h46m41.207943303s) [noise=true , address=77.90.31.14:4670 , piece_id=7FNANMUKPM3N6YWRAOAIM4WJZJUTNF7ZBI6ZUBNEPGNGMFWOE24A, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.648842639 (75h46m41.212165631s) [noise=true , address=104.36.82.55:4224 , piece_id=WRVNGSJV5BX4GJERLXNPAS7J7BZ5WFTM7R5H257YRCG5SP2HIYMA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.825828239 (75h46m41.389151241s) [noise=true , address=104.36.82.53:4126 , piece_id=2MKRQJEPGDJKDM7VWY5M4M6NORILAIEYYATJ7HVZEOBOMXOI2LBA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: ecclient: upload failed (node:12dMWFrq2NMpnMzuPAphWerntC5kkrcnzRJZBA116uzJzyvk6fH, address:104.36.82.53:4126): protocol: storage node overloaded, request limit: 2000 Jun 21 20:10:05.834666043 (75h46m41.397989033s) [upload=60693]: Next returned nil segment Jun 21 20:10:05.834686295 (75h46m41.398009302s) [upload=60693]: Waiting for error group managing segments... Jun 21 20:10:05.877010488 (75h46m41.440333486s) [noise=true , address=45.91.116.6:4982 , piece_id=VP45R6KXNH5U5SGUQUMAPIBY7KFDPBIR3S3VRIX27EAT6VLFBZ7Q, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:05.886336964 (75h46m41.449659967s) [noise=true , address=104.36.82.9:6039 , piece_id=I4AB6N27MYYUHWAZLJJUIBFBLOQFHDIM3YLYWPVRPNUV7VMYS2SA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: ecclient: upload failed (node:12P11WVW9mVXw5ra2gZ3Rezon6wZi8GrN5CchQN3MiVU4ts3L6V, address:104.36.82.9:6039): protocol: storage node overloaded, request limit: 2000 Jun 21 20:10:08.064514542 (75h46m43.627837553s) [noise=true , address=77.90.31.14:4670 , piece_id=7FNANMUKPM3N6YWRAOAIM4WJZJUTNF7ZBI6ZUBNEPGNGMFWOE24A, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:08.099098391 (75h46m43.662421396s) [noise=true , address=79.127.205.245:4231 , piece_id=EJK46XUUVI5NWOAQJEW4QIJG3KRXZHB4YOQUJRCS2MIOSWKGTNLQ, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:08.489461749 (75h46m44.052784752s) [noise=true , address=104.36.82.42:6807 , piece_id=ZBQZPXUDJ4VBCOSVKWAJ6SK6UD7SLPON7Y4SSCK2DQ5STFG4XCNQ, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:08.650922093 (75h46m44.214245093s) [noise=true , address=77.90.31.19:4853 , piece_id=VFEON5T2KSWM2KSLTVRZTXYL42JWENZYUEUPG3LMODGZWBMVGGPQ, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:09.917746439 (75h46m45.481069441s) [noise=true , address=104.36.82.13:6196 , piece_id=BCHMQO5KKT5BLNWQGOHH3V43LGSE3BUQCI2SY4TTO5M5GQNOCTMQ, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:10.830722215 (75h46m46.394045216s) [noise=true , address=104.36.82.5:5869 , piece_id=HAEHXXZASV3DHT3QN7NTZOS6XZKHNUWBKCHWJJWSTJKXAKJKKBAA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:12.738783450 (75h46m48.302106452s) [noise=true , address=77.90.31.12:4603 , piece_id=BMUIC6KNGEDL37436GLSXW5XOO6VYOPNOSFGQCYVHSENVI2PJ7KQ, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:15.460927713 (75h46m51.024250723s) [noise=true , address=104.36.82.52:4111 , piece_id=HB7QAWN6CGNIIT4HD2GZ7JDQP5NXIJ5I2USBM66YOKTYENNOWG5A, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:18.393817968 (75h46m53.95714096s) [noise=true , address=104.36.82.55:4224 , piece_id=WRVNGSJV5BX4GJERLXNPAS7J7BZ5WFTM7R5H257YRCG5SP2HIYMA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:19.549174143 (75h46m55.112497151s) [noise=true , address=104.36.82.56:4253 , piece_id=5PMVGLZETMXCQ77K3I6PXACRIMBGR2CV5JGUSU5KUMYFNXVGA2GA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:22.511944593 (75h46m58.075267584s) [noise=true , address=104.36.82.37:6663 , piece_id=X7LWYIIHJFU2T2PHYR3R5TNT22UUPHEHQWZH7ZJMLS4AH4PJ6AIA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:48.027197558 (75h47m23.590520562s) [noise=true , address=104.36.82.14:6232 , piece_id=A2ZLUUZKRTBJRJ267QSK36C3LFZV7FPAYDDAXGIWDTNEGNV6MRRA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:11:05.567692582 (75h47m41.131015574s) [noise=true , address=104.36.82.49:4982 , piece_id=UNLV5UDNWULEIPE4QSQHHZQPK3BQKFRB5T3H6GP7ZQ5QGSNK5V6A, seg_pos={0 0}, upload=60693]: Done uploading piece. err: ecclient: upload failed (node:1NxoYJXCXmTvQncbkXt6yH7mhEBBrzVdzhgL5J6W5TD4N3pkaG, address:104.36.82.49:4982): protocol: expected piece hash; context deadline exceeded; EOF Jun 21 20:11:05.567759198 (75h47m41.131082195s) [noise=true , address=104.36.82.54:4171 , piece_id=MUMABKARK4KKUKWW6XDXCQVT6O7SLFHGHE7COWC7QIYGARAA2D5A, seg_pos={0 0}, upload=60693]: Done uploading piece. err: ecclient: upload failed (node:125SMhDsEZruNu6LkmQn2M7R6WzvXTdk86DRsKf522u8BNiANq3, address:104.36.82.54:4171): protocol: expected piece hash; context deadline exceeded; EOF Jun 21 20:11:05.567869019 (75h47m41.131192033s) [noise=true , address=104.36.82.10:6087 , piece_id=5FQG7BPZC373PFB54HYXSG2SVKOBBBSGZCLC4XDJEYQFOLQY74MA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: ecclient: upload failed (node:1S9k5eejJcxkiG5pXjjBZm18uP8k5jCGfWyeW2QQxmuRU8ZzRA, address:104.36.82.10:6087): protocol: write tcp 159.223.143.39:38550->104.36.82.10:6087: use of closed network connection; write tcp 159.223.143.39:38550->104.36.82.10:6087: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:38550->104.36.82.10:6087: use of closed network connection Jun 21 20:11:05.567932150 (75h47m41.131255161s) [seg_pos={0 0}, upload=60693]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jun 21 20:11:05.593626175 (75h47m41.156949181s) [upload=60693]: Done uploading segments... Jun 21 20:11:05.593657274 (75h47m41.156980265s) [upload=60693]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 24.423461249s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [.........X...........X......................................] duration: 19.419107312s FAIL (2025-06-21 20:40:43 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== pload=60693]: Starting upload Jun 21 20:10:05.602866542 (75h46m41.166189557s) [upload=60693]: Done starting upload Jun 21 20:10:05.603116545 (75h46m41.166439547s) [upload=60693]: Uploading segments... Jun 21 20:10:05.603495389 (75h46m41.166818397s) [upload=60693]: Got next segment. Inline: false Jun 21 20:10:05.603566566 (75h46m41.166889558s) [upload=60693]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 21 20:10:05.603573280 (75h46m41.166896286s) [upload=60693]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 21 20:10:05.639130162 (75h46m41.202453164s) [seg_pos={0 0}, upload=60693]: Begin upload segment... Jun 21 20:10:05.639231070 (75h46m41.202554057s) [seg_pos={0 0}, upload=60693]: Done begin upload segment. Jun 21 20:10:05.639292724 (75h46m41.202615728s) [noise=true , address=104.36.82.56:4253 , piece_id=5PMVGLZETMXCQ77K3I6PXACRIMBGR2CV5JGUSU5KUMYFNXVGA2GA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.639338066 (75h46m41.202661071s) [noise=true , address=104.36.82.53:4126 , piece_id=2MKRQJEPGDJKDM7VWY5M4M6NORILAIEYYATJ7HVZEOBOMXOI2LBA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.639416264 (75h46m41.202739267s) [noise=true , address=104.36.82.5:5869 , piece_id=HAEHXXZASV3DHT3QN7NTZOS6XZKHNUWBKCHWJJWSTJKXAKJKKBAA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.639873098 (75h46m41.203196115s) [noise=true , address=104.36.82.52:4111 , piece_id=HB7QAWN6CGNIIT4HD2GZ7JDQP5NXIJ5I2USBM66YOKTYENNOWG5A, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.639887296 (75h46m41.203210304s) [noise=true , address=104.36.82.10:6087 , piece_id=5FQG7BPZC373PFB54HYXSG2SVKOBBBSGZCLC4XDJEYQFOLQY74MA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.640266623 (75h46m41.203589616s) [noise=true , address=104.36.82.13:6196 , piece_id=BCHMQO5KKT5BLNWQGOHH3V43LGSE3BUQCI2SY4TTO5M5GQNOCTMQ, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.640352274 (75h46m41.203675286s) [noise=true , address=104.36.82.37:6663 , piece_id=X7LWYIIHJFU2T2PHYR3R5TNT22UUPHEHQWZH7ZJMLS4AH4PJ6AIA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.640659767 (75h46m41.203982779s) [noise=true , address=104.36.82.42:6807 , piece_id=ZBQZPXUDJ4VBCOSVKWAJ6SK6UD7SLPON7Y4SSCK2DQ5STFG4XCNQ, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.640961701 (75h46m41.20428471s) [noise=true , address=45.91.116.6:4982 , piece_id=VP45R6KXNH5U5SGUQUMAPIBY7KFDPBIR3S3VRIX27EAT6VLFBZ7Q, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.641159578 (75h46m41.204482589s) [noise=true , address=104.36.82.9:6039 , piece_id=I4AB6N27MYYUHWAZLJJUIBFBLOQFHDIM3YLYWPVRPNUV7VMYS2SA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.641588126 (75h46m41.204911136s) [noise=true , address=77.90.31.19:4853 , piece_id=VFEON5T2KSWM2KSLTVRZTXYL42JWENZYUEUPG3LMODGZWBMVGGPQ, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.641863297 (75h46m41.205186305s) [noise=true , address=77.90.31.12:4603 , piece_id=BMUIC6KNGEDL37436GLSXW5XOO6VYOPNOSFGQCYVHSENVI2PJ7KQ, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.642124427 (75h46m41.205447437s) [noise=true , address=104.36.82.49:4982 , piece_id=UNLV5UDNWULEIPE4QSQHHZQPK3BQKFRB5T3H6GP7ZQ5QGSNK5V6A, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.642285917 (75h46m41.205608925s) [noise=true , address=104.36.82.54:4171 , piece_id=MUMABKARK4KKUKWW6XDXCQVT6O7SLFHGHE7COWC7QIYGARAA2D5A, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.642656750 (75h46m41.205979758s) [noise=true , address=104.36.82.14:6232 , piece_id=A2ZLUUZKRTBJRJ267QSK36C3LFZV7FPAYDDAXGIWDTNEGNV6MRRA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.643565583 (75h46m41.206888597s) [noise=true , address=79.127.205.245:4231 , piece_id=EJK46XUUVI5NWOAQJEW4QIJG3KRXZHB4YOQUJRCS2MIOSWKGTNLQ, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.644620291 (75h46m41.207943303s) [noise=true , address=77.90.31.14:4670 , piece_id=7FNANMUKPM3N6YWRAOAIM4WJZJUTNF7ZBI6ZUBNEPGNGMFWOE24A, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.648842639 (75h46m41.212165631s) [noise=true , address=104.36.82.55:4224 , piece_id=WRVNGSJV5BX4GJERLXNPAS7J7BZ5WFTM7R5H257YRCG5SP2HIYMA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.825828239 (75h46m41.389151241s) [noise=true , address=104.36.82.53:4126 , piece_id=2MKRQJEPGDJKDM7VWY5M4M6NORILAIEYYATJ7HVZEOBOMXOI2LBA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: ecclient: upload failed (node:12dMWFrq2NMpnMzuPAphWerntC5kkrcnzRJZBA116uzJzyvk6fH, address:104.36.82.53:4126): protocol: storage node overloaded, request limit: 2000 Jun 21 20:10:05.834666043 (75h46m41.397989033s) [upload=60693]: Next returned nil segment Jun 21 20:10:05.834686295 (75h46m41.398009302s) [upload=60693]: Waiting for error group managing segments... Jun 21 20:10:05.877010488 (75h46m41.440333486s) [noise=true , address=45.91.116.6:4982 , piece_id=VP45R6KXNH5U5SGUQUMAPIBY7KFDPBIR3S3VRIX27EAT6VLFBZ7Q, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:05.886336964 (75h46m41.449659967s) [noise=true , address=104.36.82.9:6039 , piece_id=I4AB6N27MYYUHWAZLJJUIBFBLOQFHDIM3YLYWPVRPNUV7VMYS2SA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: ecclient: upload failed (node:12P11WVW9mVXw5ra2gZ3Rezon6wZi8GrN5CchQN3MiVU4ts3L6V, address:104.36.82.9:6039): protocol: storage node overloaded, request limit: 2000 Jun 21 20:10:08.064514542 (75h46m43.627837553s) [noise=true , address=77.90.31.14:4670 , piece_id=7FNANMUKPM3N6YWRAOAIM4WJZJUTNF7ZBI6ZUBNEPGNGMFWOE24A, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:08.099098391 (75h46m43.662421396s) [noise=true , address=79.127.205.245:4231 , piece_id=EJK46XUUVI5NWOAQJEW4QIJG3KRXZHB4YOQUJRCS2MIOSWKGTNLQ, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:08.489461749 (75h46m44.052784752s) [noise=true , address=104.36.82.42:6807 , piece_id=ZBQZPXUDJ4VBCOSVKWAJ6SK6UD7SLPON7Y4SSCK2DQ5STFG4XCNQ, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:08.650922093 (75h46m44.214245093s) [noise=true , address=77.90.31.19:4853 , piece_id=VFEON5T2KSWM2KSLTVRZTXYL42JWENZYUEUPG3LMODGZWBMVGGPQ, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:09.917746439 (75h46m45.481069441s) [noise=true , address=104.36.82.13:6196 , piece_id=BCHMQO5KKT5BLNWQGOHH3V43LGSE3BUQCI2SY4TTO5M5GQNOCTMQ, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:10.830722215 (75h46m46.394045216s) [noise=true , address=104.36.82.5:5869 , piece_id=HAEHXXZASV3DHT3QN7NTZOS6XZKHNUWBKCHWJJWSTJKXAKJKKBAA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:12.738783450 (75h46m48.302106452s) [noise=true , address=77.90.31.12:4603 , piece_id=BMUIC6KNGEDL37436GLSXW5XOO6VYOPNOSFGQCYVHSENVI2PJ7KQ, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:15.460927713 (75h46m51.024250723s) [noise=true , address=104.36.82.52:4111 , piece_id=HB7QAWN6CGNIIT4HD2GZ7JDQP5NXIJ5I2USBM66YOKTYENNOWG5A, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:18.393817968 (75h46m53.95714096s) [noise=true , address=104.36.82.55:4224 , piece_id=WRVNGSJV5BX4GJERLXNPAS7J7BZ5WFTM7R5H257YRCG5SP2HIYMA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:19.549174143 (75h46m55.112497151s) [noise=true , address=104.36.82.56:4253 , piece_id=5PMVGLZETMXCQ77K3I6PXACRIMBGR2CV5JGUSU5KUMYFNXVGA2GA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:22.511944593 (75h46m58.075267584s) [noise=true , address=104.36.82.37:6663 , piece_id=X7LWYIIHJFU2T2PHYR3R5TNT22UUPHEHQWZH7ZJMLS4AH4PJ6AIA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:48.027197558 (75h47m23.590520562s) [noise=true , address=104.36.82.14:6232 , piece_id=A2ZLUUZKRTBJRJ267QSK36C3LFZV7FPAYDDAXGIWDTNEGNV6MRRA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:11:05.567692582 (75h47m41.131015574s) [noise=true , address=104.36.82.49:4982 , piece_id=UNLV5UDNWULEIPE4QSQHHZQPK3BQKFRB5T3H6GP7ZQ5QGSNK5V6A, seg_pos={0 0}, upload=60693]: Done uploading piece. err: ecclient: upload failed (node:1NxoYJXCXmTvQncbkXt6yH7mhEBBrzVdzhgL5J6W5TD4N3pkaG, address:104.36.82.49:4982): protocol: expected piece hash; context deadline exceeded; EOF Jun 21 20:11:05.567759198 (75h47m41.131082195s) [noise=true , address=104.36.82.54:4171 , piece_id=MUMABKARK4KKUKWW6XDXCQVT6O7SLFHGHE7COWC7QIYGARAA2D5A, seg_pos={0 0}, upload=60693]: Done uploading piece. err: ecclient: upload failed (node:125SMhDsEZruNu6LkmQn2M7R6WzvXTdk86DRsKf522u8BNiANq3, address:104.36.82.54:4171): protocol: expected piece hash; context deadline exceeded; EOF Jun 21 20:11:05.567869019 (75h47m41.131192033s) [noise=true , address=104.36.82.10:6087 , piece_id=5FQG7BPZC373PFB54HYXSG2SVKOBBBSGZCLC4XDJEYQFOLQY74MA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: ecclient: upload failed (node:1S9k5eejJcxkiG5pXjjBZm18uP8k5jCGfWyeW2QQxmuRU8ZzRA, address:104.36.82.10:6087): protocol: write tcp 159.223.143.39:38550->104.36.82.10:6087: use of closed network connection; write tcp 159.223.143.39:38550->104.36.82.10:6087: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:38550->104.36.82.10:6087: use of closed network connection Jun 21 20:11:05.567932150 (75h47m41.131255161s) [seg_pos={0 0}, upload=60693]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jun 21 20:11:05.593626175 (75h47m41.156949181s) [upload=60693]: Done uploading segments... Jun 21 20:11:05.593657274 (75h47m41.156980265s) [upload=60693]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 19.419138579s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [..........X...........X.....................................] duration: 24.295967657s FAIL (2025-06-21 20:39:58 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== pload=60693]: Starting upload Jun 21 20:10:05.602866542 (75h46m41.166189557s) [upload=60693]: Done starting upload Jun 21 20:10:05.603116545 (75h46m41.166439547s) [upload=60693]: Uploading segments... Jun 21 20:10:05.603495389 (75h46m41.166818397s) [upload=60693]: Got next segment. Inline: false Jun 21 20:10:05.603566566 (75h46m41.166889558s) [upload=60693]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 21 20:10:05.603573280 (75h46m41.166896286s) [upload=60693]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 21 20:10:05.639130162 (75h46m41.202453164s) [seg_pos={0 0}, upload=60693]: Begin upload segment... Jun 21 20:10:05.639231070 (75h46m41.202554057s) [seg_pos={0 0}, upload=60693]: Done begin upload segment. Jun 21 20:10:05.639292724 (75h46m41.202615728s) [noise=true , address=104.36.82.56:4253 , piece_id=5PMVGLZETMXCQ77K3I6PXACRIMBGR2CV5JGUSU5KUMYFNXVGA2GA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.639338066 (75h46m41.202661071s) [noise=true , address=104.36.82.53:4126 , piece_id=2MKRQJEPGDJKDM7VWY5M4M6NORILAIEYYATJ7HVZEOBOMXOI2LBA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.639416264 (75h46m41.202739267s) [noise=true , address=104.36.82.5:5869 , piece_id=HAEHXXZASV3DHT3QN7NTZOS6XZKHNUWBKCHWJJWSTJKXAKJKKBAA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.639873098 (75h46m41.203196115s) [noise=true , address=104.36.82.52:4111 , piece_id=HB7QAWN6CGNIIT4HD2GZ7JDQP5NXIJ5I2USBM66YOKTYENNOWG5A, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.639887296 (75h46m41.203210304s) [noise=true , address=104.36.82.10:6087 , piece_id=5FQG7BPZC373PFB54HYXSG2SVKOBBBSGZCLC4XDJEYQFOLQY74MA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.640266623 (75h46m41.203589616s) [noise=true , address=104.36.82.13:6196 , piece_id=BCHMQO5KKT5BLNWQGOHH3V43LGSE3BUQCI2SY4TTO5M5GQNOCTMQ, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.640352274 (75h46m41.203675286s) [noise=true , address=104.36.82.37:6663 , piece_id=X7LWYIIHJFU2T2PHYR3R5TNT22UUPHEHQWZH7ZJMLS4AH4PJ6AIA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.640659767 (75h46m41.203982779s) [noise=true , address=104.36.82.42:6807 , piece_id=ZBQZPXUDJ4VBCOSVKWAJ6SK6UD7SLPON7Y4SSCK2DQ5STFG4XCNQ, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.640961701 (75h46m41.20428471s) [noise=true , address=45.91.116.6:4982 , piece_id=VP45R6KXNH5U5SGUQUMAPIBY7KFDPBIR3S3VRIX27EAT6VLFBZ7Q, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.641159578 (75h46m41.204482589s) [noise=true , address=104.36.82.9:6039 , piece_id=I4AB6N27MYYUHWAZLJJUIBFBLOQFHDIM3YLYWPVRPNUV7VMYS2SA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.641588126 (75h46m41.204911136s) [noise=true , address=77.90.31.19:4853 , piece_id=VFEON5T2KSWM2KSLTVRZTXYL42JWENZYUEUPG3LMODGZWBMVGGPQ, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.641863297 (75h46m41.205186305s) [noise=true , address=77.90.31.12:4603 , piece_id=BMUIC6KNGEDL37436GLSXW5XOO6VYOPNOSFGQCYVHSENVI2PJ7KQ, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.642124427 (75h46m41.205447437s) [noise=true , address=104.36.82.49:4982 , piece_id=UNLV5UDNWULEIPE4QSQHHZQPK3BQKFRB5T3H6GP7ZQ5QGSNK5V6A, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.642285917 (75h46m41.205608925s) [noise=true , address=104.36.82.54:4171 , piece_id=MUMABKARK4KKUKWW6XDXCQVT6O7SLFHGHE7COWC7QIYGARAA2D5A, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.642656750 (75h46m41.205979758s) [noise=true , address=104.36.82.14:6232 , piece_id=A2ZLUUZKRTBJRJ267QSK36C3LFZV7FPAYDDAXGIWDTNEGNV6MRRA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.643565583 (75h46m41.206888597s) [noise=true , address=79.127.205.245:4231 , piece_id=EJK46XUUVI5NWOAQJEW4QIJG3KRXZHB4YOQUJRCS2MIOSWKGTNLQ, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.644620291 (75h46m41.207943303s) [noise=true , address=77.90.31.14:4670 , piece_id=7FNANMUKPM3N6YWRAOAIM4WJZJUTNF7ZBI6ZUBNEPGNGMFWOE24A, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.648842639 (75h46m41.212165631s) [noise=true , address=104.36.82.55:4224 , piece_id=WRVNGSJV5BX4GJERLXNPAS7J7BZ5WFTM7R5H257YRCG5SP2HIYMA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.825828239 (75h46m41.389151241s) [noise=true , address=104.36.82.53:4126 , piece_id=2MKRQJEPGDJKDM7VWY5M4M6NORILAIEYYATJ7HVZEOBOMXOI2LBA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: ecclient: upload failed (node:12dMWFrq2NMpnMzuPAphWerntC5kkrcnzRJZBA116uzJzyvk6fH, address:104.36.82.53:4126): protocol: storage node overloaded, request limit: 2000 Jun 21 20:10:05.834666043 (75h46m41.397989033s) [upload=60693]: Next returned nil segment Jun 21 20:10:05.834686295 (75h46m41.398009302s) [upload=60693]: Waiting for error group managing segments... Jun 21 20:10:05.877010488 (75h46m41.440333486s) [noise=true , address=45.91.116.6:4982 , piece_id=VP45R6KXNH5U5SGUQUMAPIBY7KFDPBIR3S3VRIX27EAT6VLFBZ7Q, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:05.886336964 (75h46m41.449659967s) [noise=true , address=104.36.82.9:6039 , piece_id=I4AB6N27MYYUHWAZLJJUIBFBLOQFHDIM3YLYWPVRPNUV7VMYS2SA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: ecclient: upload failed (node:12P11WVW9mVXw5ra2gZ3Rezon6wZi8GrN5CchQN3MiVU4ts3L6V, address:104.36.82.9:6039): protocol: storage node overloaded, request limit: 2000 Jun 21 20:10:08.064514542 (75h46m43.627837553s) [noise=true , address=77.90.31.14:4670 , piece_id=7FNANMUKPM3N6YWRAOAIM4WJZJUTNF7ZBI6ZUBNEPGNGMFWOE24A, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:08.099098391 (75h46m43.662421396s) [noise=true , address=79.127.205.245:4231 , piece_id=EJK46XUUVI5NWOAQJEW4QIJG3KRXZHB4YOQUJRCS2MIOSWKGTNLQ, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:08.489461749 (75h46m44.052784752s) [noise=true , address=104.36.82.42:6807 , piece_id=ZBQZPXUDJ4VBCOSVKWAJ6SK6UD7SLPON7Y4SSCK2DQ5STFG4XCNQ, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:08.650922093 (75h46m44.214245093s) [noise=true , address=77.90.31.19:4853 , piece_id=VFEON5T2KSWM2KSLTVRZTXYL42JWENZYUEUPG3LMODGZWBMVGGPQ, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:09.917746439 (75h46m45.481069441s) [noise=true , address=104.36.82.13:6196 , piece_id=BCHMQO5KKT5BLNWQGOHH3V43LGSE3BUQCI2SY4TTO5M5GQNOCTMQ, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:10.830722215 (75h46m46.394045216s) [noise=true , address=104.36.82.5:5869 , piece_id=HAEHXXZASV3DHT3QN7NTZOS6XZKHNUWBKCHWJJWSTJKXAKJKKBAA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:12.738783450 (75h46m48.302106452s) [noise=true , address=77.90.31.12:4603 , piece_id=BMUIC6KNGEDL37436GLSXW5XOO6VYOPNOSFGQCYVHSENVI2PJ7KQ, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:15.460927713 (75h46m51.024250723s) [noise=true , address=104.36.82.52:4111 , piece_id=HB7QAWN6CGNIIT4HD2GZ7JDQP5NXIJ5I2USBM66YOKTYENNOWG5A, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:18.393817968 (75h46m53.95714096s) [noise=true , address=104.36.82.55:4224 , piece_id=WRVNGSJV5BX4GJERLXNPAS7J7BZ5WFTM7R5H257YRCG5SP2HIYMA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:19.549174143 (75h46m55.112497151s) [noise=true , address=104.36.82.56:4253 , piece_id=5PMVGLZETMXCQ77K3I6PXACRIMBGR2CV5JGUSU5KUMYFNXVGA2GA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:22.511944593 (75h46m58.075267584s) [noise=true , address=104.36.82.37:6663 , piece_id=X7LWYIIHJFU2T2PHYR3R5TNT22UUPHEHQWZH7ZJMLS4AH4PJ6AIA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:48.027197558 (75h47m23.590520562s) [noise=true , address=104.36.82.14:6232 , piece_id=A2ZLUUZKRTBJRJ267QSK36C3LFZV7FPAYDDAXGIWDTNEGNV6MRRA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:11:05.567692582 (75h47m41.131015574s) [noise=true , address=104.36.82.49:4982 , piece_id=UNLV5UDNWULEIPE4QSQHHZQPK3BQKFRB5T3H6GP7ZQ5QGSNK5V6A, seg_pos={0 0}, upload=60693]: Done uploading piece. err: ecclient: upload failed (node:1NxoYJXCXmTvQncbkXt6yH7mhEBBrzVdzhgL5J6W5TD4N3pkaG, address:104.36.82.49:4982): protocol: expected piece hash; context deadline exceeded; EOF Jun 21 20:11:05.567759198 (75h47m41.131082195s) [noise=true , address=104.36.82.54:4171 , piece_id=MUMABKARK4KKUKWW6XDXCQVT6O7SLFHGHE7COWC7QIYGARAA2D5A, seg_pos={0 0}, upload=60693]: Done uploading piece. err: ecclient: upload failed (node:125SMhDsEZruNu6LkmQn2M7R6WzvXTdk86DRsKf522u8BNiANq3, address:104.36.82.54:4171): protocol: expected piece hash; context deadline exceeded; EOF Jun 21 20:11:05.567869019 (75h47m41.131192033s) [noise=true , address=104.36.82.10:6087 , piece_id=5FQG7BPZC373PFB54HYXSG2SVKOBBBSGZCLC4XDJEYQFOLQY74MA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: ecclient: upload failed (node:1S9k5eejJcxkiG5pXjjBZm18uP8k5jCGfWyeW2QQxmuRU8ZzRA, address:104.36.82.10:6087): protocol: write tcp 159.223.143.39:38550->104.36.82.10:6087: use of closed network connection; write tcp 159.223.143.39:38550->104.36.82.10:6087: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:38550->104.36.82.10:6087: use of closed network connection Jun 21 20:11:05.567932150 (75h47m41.131255161s) [seg_pos={0 0}, upload=60693]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jun 21 20:11:05.593626175 (75h47m41.156949181s) [upload=60693]: Done uploading segments... Jun 21 20:11:05.593657274 (75h47m41.156980265s) [upload=60693]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 24.2959875s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [...........X...........X....................................] duration: 22.772717269s FAIL (2025-06-21 20:39:33 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== pload=60693]: Starting upload Jun 21 20:10:05.602866542 (75h46m41.166189557s) [upload=60693]: Done starting upload Jun 21 20:10:05.603116545 (75h46m41.166439547s) [upload=60693]: Uploading segments... Jun 21 20:10:05.603495389 (75h46m41.166818397s) [upload=60693]: Got next segment. Inline: false Jun 21 20:10:05.603566566 (75h46m41.166889558s) [upload=60693]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 21 20:10:05.603573280 (75h46m41.166896286s) [upload=60693]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 21 20:10:05.639130162 (75h46m41.202453164s) [seg_pos={0 0}, upload=60693]: Begin upload segment... Jun 21 20:10:05.639231070 (75h46m41.202554057s) [seg_pos={0 0}, upload=60693]: Done begin upload segment. Jun 21 20:10:05.639292724 (75h46m41.202615728s) [noise=true , address=104.36.82.56:4253 , piece_id=5PMVGLZETMXCQ77K3I6PXACRIMBGR2CV5JGUSU5KUMYFNXVGA2GA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.639338066 (75h46m41.202661071s) [noise=true , address=104.36.82.53:4126 , piece_id=2MKRQJEPGDJKDM7VWY5M4M6NORILAIEYYATJ7HVZEOBOMXOI2LBA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.639416264 (75h46m41.202739267s) [noise=true , address=104.36.82.5:5869 , piece_id=HAEHXXZASV3DHT3QN7NTZOS6XZKHNUWBKCHWJJWSTJKXAKJKKBAA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.639873098 (75h46m41.203196115s) [noise=true , address=104.36.82.52:4111 , piece_id=HB7QAWN6CGNIIT4HD2GZ7JDQP5NXIJ5I2USBM66YOKTYENNOWG5A, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.639887296 (75h46m41.203210304s) [noise=true , address=104.36.82.10:6087 , piece_id=5FQG7BPZC373PFB54HYXSG2SVKOBBBSGZCLC4XDJEYQFOLQY74MA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.640266623 (75h46m41.203589616s) [noise=true , address=104.36.82.13:6196 , piece_id=BCHMQO5KKT5BLNWQGOHH3V43LGSE3BUQCI2SY4TTO5M5GQNOCTMQ, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.640352274 (75h46m41.203675286s) [noise=true , address=104.36.82.37:6663 , piece_id=X7LWYIIHJFU2T2PHYR3R5TNT22UUPHEHQWZH7ZJMLS4AH4PJ6AIA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.640659767 (75h46m41.203982779s) [noise=true , address=104.36.82.42:6807 , piece_id=ZBQZPXUDJ4VBCOSVKWAJ6SK6UD7SLPON7Y4SSCK2DQ5STFG4XCNQ, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.640961701 (75h46m41.20428471s) [noise=true , address=45.91.116.6:4982 , piece_id=VP45R6KXNH5U5SGUQUMAPIBY7KFDPBIR3S3VRIX27EAT6VLFBZ7Q, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.641159578 (75h46m41.204482589s) [noise=true , address=104.36.82.9:6039 , piece_id=I4AB6N27MYYUHWAZLJJUIBFBLOQFHDIM3YLYWPVRPNUV7VMYS2SA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.641588126 (75h46m41.204911136s) [noise=true , address=77.90.31.19:4853 , piece_id=VFEON5T2KSWM2KSLTVRZTXYL42JWENZYUEUPG3LMODGZWBMVGGPQ, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.641863297 (75h46m41.205186305s) [noise=true , address=77.90.31.12:4603 , piece_id=BMUIC6KNGEDL37436GLSXW5XOO6VYOPNOSFGQCYVHSENVI2PJ7KQ, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.642124427 (75h46m41.205447437s) [noise=true , address=104.36.82.49:4982 , piece_id=UNLV5UDNWULEIPE4QSQHHZQPK3BQKFRB5T3H6GP7ZQ5QGSNK5V6A, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.642285917 (75h46m41.205608925s) [noise=true , address=104.36.82.54:4171 , piece_id=MUMABKARK4KKUKWW6XDXCQVT6O7SLFHGHE7COWC7QIYGARAA2D5A, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.642656750 (75h46m41.205979758s) [noise=true , address=104.36.82.14:6232 , piece_id=A2ZLUUZKRTBJRJ267QSK36C3LFZV7FPAYDDAXGIWDTNEGNV6MRRA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.643565583 (75h46m41.206888597s) [noise=true , address=79.127.205.245:4231 , piece_id=EJK46XUUVI5NWOAQJEW4QIJG3KRXZHB4YOQUJRCS2MIOSWKGTNLQ, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.644620291 (75h46m41.207943303s) [noise=true , address=77.90.31.14:4670 , piece_id=7FNANMUKPM3N6YWRAOAIM4WJZJUTNF7ZBI6ZUBNEPGNGMFWOE24A, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.648842639 (75h46m41.212165631s) [noise=true , address=104.36.82.55:4224 , piece_id=WRVNGSJV5BX4GJERLXNPAS7J7BZ5WFTM7R5H257YRCG5SP2HIYMA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.825828239 (75h46m41.389151241s) [noise=true , address=104.36.82.53:4126 , piece_id=2MKRQJEPGDJKDM7VWY5M4M6NORILAIEYYATJ7HVZEOBOMXOI2LBA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: ecclient: upload failed (node:12dMWFrq2NMpnMzuPAphWerntC5kkrcnzRJZBA116uzJzyvk6fH, address:104.36.82.53:4126): protocol: storage node overloaded, request limit: 2000 Jun 21 20:10:05.834666043 (75h46m41.397989033s) [upload=60693]: Next returned nil segment Jun 21 20:10:05.834686295 (75h46m41.398009302s) [upload=60693]: Waiting for error group managing segments... Jun 21 20:10:05.877010488 (75h46m41.440333486s) [noise=true , address=45.91.116.6:4982 , piece_id=VP45R6KXNH5U5SGUQUMAPIBY7KFDPBIR3S3VRIX27EAT6VLFBZ7Q, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:05.886336964 (75h46m41.449659967s) [noise=true , address=104.36.82.9:6039 , piece_id=I4AB6N27MYYUHWAZLJJUIBFBLOQFHDIM3YLYWPVRPNUV7VMYS2SA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: ecclient: upload failed (node:12P11WVW9mVXw5ra2gZ3Rezon6wZi8GrN5CchQN3MiVU4ts3L6V, address:104.36.82.9:6039): protocol: storage node overloaded, request limit: 2000 Jun 21 20:10:08.064514542 (75h46m43.627837553s) [noise=true , address=77.90.31.14:4670 , piece_id=7FNANMUKPM3N6YWRAOAIM4WJZJUTNF7ZBI6ZUBNEPGNGMFWOE24A, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:08.099098391 (75h46m43.662421396s) [noise=true , address=79.127.205.245:4231 , piece_id=EJK46XUUVI5NWOAQJEW4QIJG3KRXZHB4YOQUJRCS2MIOSWKGTNLQ, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:08.489461749 (75h46m44.052784752s) [noise=true , address=104.36.82.42:6807 , piece_id=ZBQZPXUDJ4VBCOSVKWAJ6SK6UD7SLPON7Y4SSCK2DQ5STFG4XCNQ, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:08.650922093 (75h46m44.214245093s) [noise=true , address=77.90.31.19:4853 , piece_id=VFEON5T2KSWM2KSLTVRZTXYL42JWENZYUEUPG3LMODGZWBMVGGPQ, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:09.917746439 (75h46m45.481069441s) [noise=true , address=104.36.82.13:6196 , piece_id=BCHMQO5KKT5BLNWQGOHH3V43LGSE3BUQCI2SY4TTO5M5GQNOCTMQ, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:10.830722215 (75h46m46.394045216s) [noise=true , address=104.36.82.5:5869 , piece_id=HAEHXXZASV3DHT3QN7NTZOS6XZKHNUWBKCHWJJWSTJKXAKJKKBAA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:12.738783450 (75h46m48.302106452s) [noise=true , address=77.90.31.12:4603 , piece_id=BMUIC6KNGEDL37436GLSXW5XOO6VYOPNOSFGQCYVHSENVI2PJ7KQ, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:15.460927713 (75h46m51.024250723s) [noise=true , address=104.36.82.52:4111 , piece_id=HB7QAWN6CGNIIT4HD2GZ7JDQP5NXIJ5I2USBM66YOKTYENNOWG5A, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:18.393817968 (75h46m53.95714096s) [noise=true , address=104.36.82.55:4224 , piece_id=WRVNGSJV5BX4GJERLXNPAS7J7BZ5WFTM7R5H257YRCG5SP2HIYMA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:19.549174143 (75h46m55.112497151s) [noise=true , address=104.36.82.56:4253 , piece_id=5PMVGLZETMXCQ77K3I6PXACRIMBGR2CV5JGUSU5KUMYFNXVGA2GA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:22.511944593 (75h46m58.075267584s) [noise=true , address=104.36.82.37:6663 , piece_id=X7LWYIIHJFU2T2PHYR3R5TNT22UUPHEHQWZH7ZJMLS4AH4PJ6AIA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:48.027197558 (75h47m23.590520562s) [noise=true , address=104.36.82.14:6232 , piece_id=A2ZLUUZKRTBJRJ267QSK36C3LFZV7FPAYDDAXGIWDTNEGNV6MRRA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:11:05.567692582 (75h47m41.131015574s) [noise=true , address=104.36.82.49:4982 , piece_id=UNLV5UDNWULEIPE4QSQHHZQPK3BQKFRB5T3H6GP7ZQ5QGSNK5V6A, seg_pos={0 0}, upload=60693]: Done uploading piece. err: ecclient: upload failed (node:1NxoYJXCXmTvQncbkXt6yH7mhEBBrzVdzhgL5J6W5TD4N3pkaG, address:104.36.82.49:4982): protocol: expected piece hash; context deadline exceeded; EOF Jun 21 20:11:05.567759198 (75h47m41.131082195s) [noise=true , address=104.36.82.54:4171 , piece_id=MUMABKARK4KKUKWW6XDXCQVT6O7SLFHGHE7COWC7QIYGARAA2D5A, seg_pos={0 0}, upload=60693]: Done uploading piece. err: ecclient: upload failed (node:125SMhDsEZruNu6LkmQn2M7R6WzvXTdk86DRsKf522u8BNiANq3, address:104.36.82.54:4171): protocol: expected piece hash; context deadline exceeded; EOF Jun 21 20:11:05.567869019 (75h47m41.131192033s) [noise=true , address=104.36.82.10:6087 , piece_id=5FQG7BPZC373PFB54HYXSG2SVKOBBBSGZCLC4XDJEYQFOLQY74MA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: ecclient: upload failed (node:1S9k5eejJcxkiG5pXjjBZm18uP8k5jCGfWyeW2QQxmuRU8ZzRA, address:104.36.82.10:6087): protocol: write tcp 159.223.143.39:38550->104.36.82.10:6087: use of closed network connection; write tcp 159.223.143.39:38550->104.36.82.10:6087: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:38550->104.36.82.10:6087: use of closed network connection Jun 21 20:11:05.567932150 (75h47m41.131255161s) [seg_pos={0 0}, upload=60693]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jun 21 20:11:05.593626175 (75h47m41.156949181s) [upload=60693]: Done uploading segments... Jun 21 20:11:05.593657274 (75h47m41.156980265s) [upload=60693]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 22.772747741s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [............X...........X...................................] duration: 13.435008791s FAIL (2025-06-21 20:38:34 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== pload=60693]: Starting upload Jun 21 20:10:05.602866542 (75h46m41.166189557s) [upload=60693]: Done starting upload Jun 21 20:10:05.603116545 (75h46m41.166439547s) [upload=60693]: Uploading segments... Jun 21 20:10:05.603495389 (75h46m41.166818397s) [upload=60693]: Got next segment. Inline: false Jun 21 20:10:05.603566566 (75h46m41.166889558s) [upload=60693]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 21 20:10:05.603573280 (75h46m41.166896286s) [upload=60693]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 21 20:10:05.639130162 (75h46m41.202453164s) [seg_pos={0 0}, upload=60693]: Begin upload segment... Jun 21 20:10:05.639231070 (75h46m41.202554057s) [seg_pos={0 0}, upload=60693]: Done begin upload segment. Jun 21 20:10:05.639292724 (75h46m41.202615728s) [noise=true , address=104.36.82.56:4253 , piece_id=5PMVGLZETMXCQ77K3I6PXACRIMBGR2CV5JGUSU5KUMYFNXVGA2GA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.639338066 (75h46m41.202661071s) [noise=true , address=104.36.82.53:4126 , piece_id=2MKRQJEPGDJKDM7VWY5M4M6NORILAIEYYATJ7HVZEOBOMXOI2LBA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.639416264 (75h46m41.202739267s) [noise=true , address=104.36.82.5:5869 , piece_id=HAEHXXZASV3DHT3QN7NTZOS6XZKHNUWBKCHWJJWSTJKXAKJKKBAA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.639873098 (75h46m41.203196115s) [noise=true , address=104.36.82.52:4111 , piece_id=HB7QAWN6CGNIIT4HD2GZ7JDQP5NXIJ5I2USBM66YOKTYENNOWG5A, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.639887296 (75h46m41.203210304s) [noise=true , address=104.36.82.10:6087 , piece_id=5FQG7BPZC373PFB54HYXSG2SVKOBBBSGZCLC4XDJEYQFOLQY74MA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.640266623 (75h46m41.203589616s) [noise=true , address=104.36.82.13:6196 , piece_id=BCHMQO5KKT5BLNWQGOHH3V43LGSE3BUQCI2SY4TTO5M5GQNOCTMQ, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.640352274 (75h46m41.203675286s) [noise=true , address=104.36.82.37:6663 , piece_id=X7LWYIIHJFU2T2PHYR3R5TNT22UUPHEHQWZH7ZJMLS4AH4PJ6AIA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.640659767 (75h46m41.203982779s) [noise=true , address=104.36.82.42:6807 , piece_id=ZBQZPXUDJ4VBCOSVKWAJ6SK6UD7SLPON7Y4SSCK2DQ5STFG4XCNQ, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.640961701 (75h46m41.20428471s) [noise=true , address=45.91.116.6:4982 , piece_id=VP45R6KXNH5U5SGUQUMAPIBY7KFDPBIR3S3VRIX27EAT6VLFBZ7Q, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.641159578 (75h46m41.204482589s) [noise=true , address=104.36.82.9:6039 , piece_id=I4AB6N27MYYUHWAZLJJUIBFBLOQFHDIM3YLYWPVRPNUV7VMYS2SA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.641588126 (75h46m41.204911136s) [noise=true , address=77.90.31.19:4853 , piece_id=VFEON5T2KSWM2KSLTVRZTXYL42JWENZYUEUPG3LMODGZWBMVGGPQ, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.641863297 (75h46m41.205186305s) [noise=true , address=77.90.31.12:4603 , piece_id=BMUIC6KNGEDL37436GLSXW5XOO6VYOPNOSFGQCYVHSENVI2PJ7KQ, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.642124427 (75h46m41.205447437s) [noise=true , address=104.36.82.49:4982 , piece_id=UNLV5UDNWULEIPE4QSQHHZQPK3BQKFRB5T3H6GP7ZQ5QGSNK5V6A, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.642285917 (75h46m41.205608925s) [noise=true , address=104.36.82.54:4171 , piece_id=MUMABKARK4KKUKWW6XDXCQVT6O7SLFHGHE7COWC7QIYGARAA2D5A, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.642656750 (75h46m41.205979758s) [noise=true , address=104.36.82.14:6232 , piece_id=A2ZLUUZKRTBJRJ267QSK36C3LFZV7FPAYDDAXGIWDTNEGNV6MRRA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.643565583 (75h46m41.206888597s) [noise=true , address=79.127.205.245:4231 , piece_id=EJK46XUUVI5NWOAQJEW4QIJG3KRXZHB4YOQUJRCS2MIOSWKGTNLQ, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.644620291 (75h46m41.207943303s) [noise=true , address=77.90.31.14:4670 , piece_id=7FNANMUKPM3N6YWRAOAIM4WJZJUTNF7ZBI6ZUBNEPGNGMFWOE24A, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.648842639 (75h46m41.212165631s) [noise=true , address=104.36.82.55:4224 , piece_id=WRVNGSJV5BX4GJERLXNPAS7J7BZ5WFTM7R5H257YRCG5SP2HIYMA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.825828239 (75h46m41.389151241s) [noise=true , address=104.36.82.53:4126 , piece_id=2MKRQJEPGDJKDM7VWY5M4M6NORILAIEYYATJ7HVZEOBOMXOI2LBA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: ecclient: upload failed (node:12dMWFrq2NMpnMzuPAphWerntC5kkrcnzRJZBA116uzJzyvk6fH, address:104.36.82.53:4126): protocol: storage node overloaded, request limit: 2000 Jun 21 20:10:05.834666043 (75h46m41.397989033s) [upload=60693]: Next returned nil segment Jun 21 20:10:05.834686295 (75h46m41.398009302s) [upload=60693]: Waiting for error group managing segments... Jun 21 20:10:05.877010488 (75h46m41.440333486s) [noise=true , address=45.91.116.6:4982 , piece_id=VP45R6KXNH5U5SGUQUMAPIBY7KFDPBIR3S3VRIX27EAT6VLFBZ7Q, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:05.886336964 (75h46m41.449659967s) [noise=true , address=104.36.82.9:6039 , piece_id=I4AB6N27MYYUHWAZLJJUIBFBLOQFHDIM3YLYWPVRPNUV7VMYS2SA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: ecclient: upload failed (node:12P11WVW9mVXw5ra2gZ3Rezon6wZi8GrN5CchQN3MiVU4ts3L6V, address:104.36.82.9:6039): protocol: storage node overloaded, request limit: 2000 Jun 21 20:10:08.064514542 (75h46m43.627837553s) [noise=true , address=77.90.31.14:4670 , piece_id=7FNANMUKPM3N6YWRAOAIM4WJZJUTNF7ZBI6ZUBNEPGNGMFWOE24A, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:08.099098391 (75h46m43.662421396s) [noise=true , address=79.127.205.245:4231 , piece_id=EJK46XUUVI5NWOAQJEW4QIJG3KRXZHB4YOQUJRCS2MIOSWKGTNLQ, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:08.489461749 (75h46m44.052784752s) [noise=true , address=104.36.82.42:6807 , piece_id=ZBQZPXUDJ4VBCOSVKWAJ6SK6UD7SLPON7Y4SSCK2DQ5STFG4XCNQ, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:08.650922093 (75h46m44.214245093s) [noise=true , address=77.90.31.19:4853 , piece_id=VFEON5T2KSWM2KSLTVRZTXYL42JWENZYUEUPG3LMODGZWBMVGGPQ, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:09.917746439 (75h46m45.481069441s) [noise=true , address=104.36.82.13:6196 , piece_id=BCHMQO5KKT5BLNWQGOHH3V43LGSE3BUQCI2SY4TTO5M5GQNOCTMQ, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:10.830722215 (75h46m46.394045216s) [noise=true , address=104.36.82.5:5869 , piece_id=HAEHXXZASV3DHT3QN7NTZOS6XZKHNUWBKCHWJJWSTJKXAKJKKBAA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:12.738783450 (75h46m48.302106452s) [noise=true , address=77.90.31.12:4603 , piece_id=BMUIC6KNGEDL37436GLSXW5XOO6VYOPNOSFGQCYVHSENVI2PJ7KQ, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:15.460927713 (75h46m51.024250723s) [noise=true , address=104.36.82.52:4111 , piece_id=HB7QAWN6CGNIIT4HD2GZ7JDQP5NXIJ5I2USBM66YOKTYENNOWG5A, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:18.393817968 (75h46m53.95714096s) [noise=true , address=104.36.82.55:4224 , piece_id=WRVNGSJV5BX4GJERLXNPAS7J7BZ5WFTM7R5H257YRCG5SP2HIYMA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:19.549174143 (75h46m55.112497151s) [noise=true , address=104.36.82.56:4253 , piece_id=5PMVGLZETMXCQ77K3I6PXACRIMBGR2CV5JGUSU5KUMYFNXVGA2GA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:22.511944593 (75h46m58.075267584s) [noise=true , address=104.36.82.37:6663 , piece_id=X7LWYIIHJFU2T2PHYR3R5TNT22UUPHEHQWZH7ZJMLS4AH4PJ6AIA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:48.027197558 (75h47m23.590520562s) [noise=true , address=104.36.82.14:6232 , piece_id=A2ZLUUZKRTBJRJ267QSK36C3LFZV7FPAYDDAXGIWDTNEGNV6MRRA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:11:05.567692582 (75h47m41.131015574s) [noise=true , address=104.36.82.49:4982 , piece_id=UNLV5UDNWULEIPE4QSQHHZQPK3BQKFRB5T3H6GP7ZQ5QGSNK5V6A, seg_pos={0 0}, upload=60693]: Done uploading piece. err: ecclient: upload failed (node:1NxoYJXCXmTvQncbkXt6yH7mhEBBrzVdzhgL5J6W5TD4N3pkaG, address:104.36.82.49:4982): protocol: expected piece hash; context deadline exceeded; EOF Jun 21 20:11:05.567759198 (75h47m41.131082195s) [noise=true , address=104.36.82.54:4171 , piece_id=MUMABKARK4KKUKWW6XDXCQVT6O7SLFHGHE7COWC7QIYGARAA2D5A, seg_pos={0 0}, upload=60693]: Done uploading piece. err: ecclient: upload failed (node:125SMhDsEZruNu6LkmQn2M7R6WzvXTdk86DRsKf522u8BNiANq3, address:104.36.82.54:4171): protocol: expected piece hash; context deadline exceeded; EOF Jun 21 20:11:05.567869019 (75h47m41.131192033s) [noise=true , address=104.36.82.10:6087 , piece_id=5FQG7BPZC373PFB54HYXSG2SVKOBBBSGZCLC4XDJEYQFOLQY74MA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: ecclient: upload failed (node:1S9k5eejJcxkiG5pXjjBZm18uP8k5jCGfWyeW2QQxmuRU8ZzRA, address:104.36.82.10:6087): protocol: write tcp 159.223.143.39:38550->104.36.82.10:6087: use of closed network connection; write tcp 159.223.143.39:38550->104.36.82.10:6087: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:38550->104.36.82.10:6087: use of closed network connection Jun 21 20:11:05.567932150 (75h47m41.131255161s) [seg_pos={0 0}, upload=60693]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jun 21 20:11:05.593626175 (75h47m41.156949181s) [upload=60693]: Done uploading segments... Jun 21 20:11:05.593657274 (75h47m41.156980265s) [upload=60693]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 13.435036226s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [.............X...........X..................................] duration: 23.463841693s FAIL (2025-06-21 20:38:09 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== pload=60693]: Starting upload Jun 21 20:10:05.602866542 (75h46m41.166189557s) [upload=60693]: Done starting upload Jun 21 20:10:05.603116545 (75h46m41.166439547s) [upload=60693]: Uploading segments... Jun 21 20:10:05.603495389 (75h46m41.166818397s) [upload=60693]: Got next segment. Inline: false Jun 21 20:10:05.603566566 (75h46m41.166889558s) [upload=60693]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 21 20:10:05.603573280 (75h46m41.166896286s) [upload=60693]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 21 20:10:05.639130162 (75h46m41.202453164s) [seg_pos={0 0}, upload=60693]: Begin upload segment... Jun 21 20:10:05.639231070 (75h46m41.202554057s) [seg_pos={0 0}, upload=60693]: Done begin upload segment. Jun 21 20:10:05.639292724 (75h46m41.202615728s) [noise=true , address=104.36.82.56:4253 , piece_id=5PMVGLZETMXCQ77K3I6PXACRIMBGR2CV5JGUSU5KUMYFNXVGA2GA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.639338066 (75h46m41.202661071s) [noise=true , address=104.36.82.53:4126 , piece_id=2MKRQJEPGDJKDM7VWY5M4M6NORILAIEYYATJ7HVZEOBOMXOI2LBA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.639416264 (75h46m41.202739267s) [noise=true , address=104.36.82.5:5869 , piece_id=HAEHXXZASV3DHT3QN7NTZOS6XZKHNUWBKCHWJJWSTJKXAKJKKBAA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.639873098 (75h46m41.203196115s) [noise=true , address=104.36.82.52:4111 , piece_id=HB7QAWN6CGNIIT4HD2GZ7JDQP5NXIJ5I2USBM66YOKTYENNOWG5A, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.639887296 (75h46m41.203210304s) [noise=true , address=104.36.82.10:6087 , piece_id=5FQG7BPZC373PFB54HYXSG2SVKOBBBSGZCLC4XDJEYQFOLQY74MA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.640266623 (75h46m41.203589616s) [noise=true , address=104.36.82.13:6196 , piece_id=BCHMQO5KKT5BLNWQGOHH3V43LGSE3BUQCI2SY4TTO5M5GQNOCTMQ, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.640352274 (75h46m41.203675286s) [noise=true , address=104.36.82.37:6663 , piece_id=X7LWYIIHJFU2T2PHYR3R5TNT22UUPHEHQWZH7ZJMLS4AH4PJ6AIA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.640659767 (75h46m41.203982779s) [noise=true , address=104.36.82.42:6807 , piece_id=ZBQZPXUDJ4VBCOSVKWAJ6SK6UD7SLPON7Y4SSCK2DQ5STFG4XCNQ, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.640961701 (75h46m41.20428471s) [noise=true , address=45.91.116.6:4982 , piece_id=VP45R6KXNH5U5SGUQUMAPIBY7KFDPBIR3S3VRIX27EAT6VLFBZ7Q, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.641159578 (75h46m41.204482589s) [noise=true , address=104.36.82.9:6039 , piece_id=I4AB6N27MYYUHWAZLJJUIBFBLOQFHDIM3YLYWPVRPNUV7VMYS2SA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.641588126 (75h46m41.204911136s) [noise=true , address=77.90.31.19:4853 , piece_id=VFEON5T2KSWM2KSLTVRZTXYL42JWENZYUEUPG3LMODGZWBMVGGPQ, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.641863297 (75h46m41.205186305s) [noise=true , address=77.90.31.12:4603 , piece_id=BMUIC6KNGEDL37436GLSXW5XOO6VYOPNOSFGQCYVHSENVI2PJ7KQ, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.642124427 (75h46m41.205447437s) [noise=true , address=104.36.82.49:4982 , piece_id=UNLV5UDNWULEIPE4QSQHHZQPK3BQKFRB5T3H6GP7ZQ5QGSNK5V6A, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.642285917 (75h46m41.205608925s) [noise=true , address=104.36.82.54:4171 , piece_id=MUMABKARK4KKUKWW6XDXCQVT6O7SLFHGHE7COWC7QIYGARAA2D5A, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.642656750 (75h46m41.205979758s) [noise=true , address=104.36.82.14:6232 , piece_id=A2ZLUUZKRTBJRJ267QSK36C3LFZV7FPAYDDAXGIWDTNEGNV6MRRA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.643565583 (75h46m41.206888597s) [noise=true , address=79.127.205.245:4231 , piece_id=EJK46XUUVI5NWOAQJEW4QIJG3KRXZHB4YOQUJRCS2MIOSWKGTNLQ, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.644620291 (75h46m41.207943303s) [noise=true , address=77.90.31.14:4670 , piece_id=7FNANMUKPM3N6YWRAOAIM4WJZJUTNF7ZBI6ZUBNEPGNGMFWOE24A, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.648842639 (75h46m41.212165631s) [noise=true , address=104.36.82.55:4224 , piece_id=WRVNGSJV5BX4GJERLXNPAS7J7BZ5WFTM7R5H257YRCG5SP2HIYMA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.825828239 (75h46m41.389151241s) [noise=true , address=104.36.82.53:4126 , piece_id=2MKRQJEPGDJKDM7VWY5M4M6NORILAIEYYATJ7HVZEOBOMXOI2LBA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: ecclient: upload failed (node:12dMWFrq2NMpnMzuPAphWerntC5kkrcnzRJZBA116uzJzyvk6fH, address:104.36.82.53:4126): protocol: storage node overloaded, request limit: 2000 Jun 21 20:10:05.834666043 (75h46m41.397989033s) [upload=60693]: Next returned nil segment Jun 21 20:10:05.834686295 (75h46m41.398009302s) [upload=60693]: Waiting for error group managing segments... Jun 21 20:10:05.877010488 (75h46m41.440333486s) [noise=true , address=45.91.116.6:4982 , piece_id=VP45R6KXNH5U5SGUQUMAPIBY7KFDPBIR3S3VRIX27EAT6VLFBZ7Q, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:05.886336964 (75h46m41.449659967s) [noise=true , address=104.36.82.9:6039 , piece_id=I4AB6N27MYYUHWAZLJJUIBFBLOQFHDIM3YLYWPVRPNUV7VMYS2SA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: ecclient: upload failed (node:12P11WVW9mVXw5ra2gZ3Rezon6wZi8GrN5CchQN3MiVU4ts3L6V, address:104.36.82.9:6039): protocol: storage node overloaded, request limit: 2000 Jun 21 20:10:08.064514542 (75h46m43.627837553s) [noise=true , address=77.90.31.14:4670 , piece_id=7FNANMUKPM3N6YWRAOAIM4WJZJUTNF7ZBI6ZUBNEPGNGMFWOE24A, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:08.099098391 (75h46m43.662421396s) [noise=true , address=79.127.205.245:4231 , piece_id=EJK46XUUVI5NWOAQJEW4QIJG3KRXZHB4YOQUJRCS2MIOSWKGTNLQ, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:08.489461749 (75h46m44.052784752s) [noise=true , address=104.36.82.42:6807 , piece_id=ZBQZPXUDJ4VBCOSVKWAJ6SK6UD7SLPON7Y4SSCK2DQ5STFG4XCNQ, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:08.650922093 (75h46m44.214245093s) [noise=true , address=77.90.31.19:4853 , piece_id=VFEON5T2KSWM2KSLTVRZTXYL42JWENZYUEUPG3LMODGZWBMVGGPQ, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:09.917746439 (75h46m45.481069441s) [noise=true , address=104.36.82.13:6196 , piece_id=BCHMQO5KKT5BLNWQGOHH3V43LGSE3BUQCI2SY4TTO5M5GQNOCTMQ, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:10.830722215 (75h46m46.394045216s) [noise=true , address=104.36.82.5:5869 , piece_id=HAEHXXZASV3DHT3QN7NTZOS6XZKHNUWBKCHWJJWSTJKXAKJKKBAA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:12.738783450 (75h46m48.302106452s) [noise=true , address=77.90.31.12:4603 , piece_id=BMUIC6KNGEDL37436GLSXW5XOO6VYOPNOSFGQCYVHSENVI2PJ7KQ, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:15.460927713 (75h46m51.024250723s) [noise=true , address=104.36.82.52:4111 , piece_id=HB7QAWN6CGNIIT4HD2GZ7JDQP5NXIJ5I2USBM66YOKTYENNOWG5A, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:18.393817968 (75h46m53.95714096s) [noise=true , address=104.36.82.55:4224 , piece_id=WRVNGSJV5BX4GJERLXNPAS7J7BZ5WFTM7R5H257YRCG5SP2HIYMA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:19.549174143 (75h46m55.112497151s) [noise=true , address=104.36.82.56:4253 , piece_id=5PMVGLZETMXCQ77K3I6PXACRIMBGR2CV5JGUSU5KUMYFNXVGA2GA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:22.511944593 (75h46m58.075267584s) [noise=true , address=104.36.82.37:6663 , piece_id=X7LWYIIHJFU2T2PHYR3R5TNT22UUPHEHQWZH7ZJMLS4AH4PJ6AIA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:48.027197558 (75h47m23.590520562s) [noise=true , address=104.36.82.14:6232 , piece_id=A2ZLUUZKRTBJRJ267QSK36C3LFZV7FPAYDDAXGIWDTNEGNV6MRRA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:11:05.567692582 (75h47m41.131015574s) [noise=true , address=104.36.82.49:4982 , piece_id=UNLV5UDNWULEIPE4QSQHHZQPK3BQKFRB5T3H6GP7ZQ5QGSNK5V6A, seg_pos={0 0}, upload=60693]: Done uploading piece. err: ecclient: upload failed (node:1NxoYJXCXmTvQncbkXt6yH7mhEBBrzVdzhgL5J6W5TD4N3pkaG, address:104.36.82.49:4982): protocol: expected piece hash; context deadline exceeded; EOF Jun 21 20:11:05.567759198 (75h47m41.131082195s) [noise=true , address=104.36.82.54:4171 , piece_id=MUMABKARK4KKUKWW6XDXCQVT6O7SLFHGHE7COWC7QIYGARAA2D5A, seg_pos={0 0}, upload=60693]: Done uploading piece. err: ecclient: upload failed (node:125SMhDsEZruNu6LkmQn2M7R6WzvXTdk86DRsKf522u8BNiANq3, address:104.36.82.54:4171): protocol: expected piece hash; context deadline exceeded; EOF Jun 21 20:11:05.567869019 (75h47m41.131192033s) [noise=true , address=104.36.82.10:6087 , piece_id=5FQG7BPZC373PFB54HYXSG2SVKOBBBSGZCLC4XDJEYQFOLQY74MA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: ecclient: upload failed (node:1S9k5eejJcxkiG5pXjjBZm18uP8k5jCGfWyeW2QQxmuRU8ZzRA, address:104.36.82.10:6087): protocol: write tcp 159.223.143.39:38550->104.36.82.10:6087: use of closed network connection; write tcp 159.223.143.39:38550->104.36.82.10:6087: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:38550->104.36.82.10:6087: use of closed network connection Jun 21 20:11:05.567932150 (75h47m41.131255161s) [seg_pos={0 0}, upload=60693]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jun 21 20:11:05.593626175 (75h47m41.156949181s) [upload=60693]: Done uploading segments... Jun 21 20:11:05.593657274 (75h47m41.156980265s) [upload=60693]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 23.463887964s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [..............X...........X.................................] duration: 16.873184024s FAIL (2025-06-21 20:37:46 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== pload=60693]: Starting upload Jun 21 20:10:05.602866542 (75h46m41.166189557s) [upload=60693]: Done starting upload Jun 21 20:10:05.603116545 (75h46m41.166439547s) [upload=60693]: Uploading segments... Jun 21 20:10:05.603495389 (75h46m41.166818397s) [upload=60693]: Got next segment. Inline: false Jun 21 20:10:05.603566566 (75h46m41.166889558s) [upload=60693]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 21 20:10:05.603573280 (75h46m41.166896286s) [upload=60693]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 21 20:10:05.639130162 (75h46m41.202453164s) [seg_pos={0 0}, upload=60693]: Begin upload segment... Jun 21 20:10:05.639231070 (75h46m41.202554057s) [seg_pos={0 0}, upload=60693]: Done begin upload segment. Jun 21 20:10:05.639292724 (75h46m41.202615728s) [noise=true , address=104.36.82.56:4253 , piece_id=5PMVGLZETMXCQ77K3I6PXACRIMBGR2CV5JGUSU5KUMYFNXVGA2GA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.639338066 (75h46m41.202661071s) [noise=true , address=104.36.82.53:4126 , piece_id=2MKRQJEPGDJKDM7VWY5M4M6NORILAIEYYATJ7HVZEOBOMXOI2LBA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.639416264 (75h46m41.202739267s) [noise=true , address=104.36.82.5:5869 , piece_id=HAEHXXZASV3DHT3QN7NTZOS6XZKHNUWBKCHWJJWSTJKXAKJKKBAA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.639873098 (75h46m41.203196115s) [noise=true , address=104.36.82.52:4111 , piece_id=HB7QAWN6CGNIIT4HD2GZ7JDQP5NXIJ5I2USBM66YOKTYENNOWG5A, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.639887296 (75h46m41.203210304s) [noise=true , address=104.36.82.10:6087 , piece_id=5FQG7BPZC373PFB54HYXSG2SVKOBBBSGZCLC4XDJEYQFOLQY74MA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.640266623 (75h46m41.203589616s) [noise=true , address=104.36.82.13:6196 , piece_id=BCHMQO5KKT5BLNWQGOHH3V43LGSE3BUQCI2SY4TTO5M5GQNOCTMQ, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.640352274 (75h46m41.203675286s) [noise=true , address=104.36.82.37:6663 , piece_id=X7LWYIIHJFU2T2PHYR3R5TNT22UUPHEHQWZH7ZJMLS4AH4PJ6AIA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.640659767 (75h46m41.203982779s) [noise=true , address=104.36.82.42:6807 , piece_id=ZBQZPXUDJ4VBCOSVKWAJ6SK6UD7SLPON7Y4SSCK2DQ5STFG4XCNQ, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.640961701 (75h46m41.20428471s) [noise=true , address=45.91.116.6:4982 , piece_id=VP45R6KXNH5U5SGUQUMAPIBY7KFDPBIR3S3VRIX27EAT6VLFBZ7Q, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.641159578 (75h46m41.204482589s) [noise=true , address=104.36.82.9:6039 , piece_id=I4AB6N27MYYUHWAZLJJUIBFBLOQFHDIM3YLYWPVRPNUV7VMYS2SA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.641588126 (75h46m41.204911136s) [noise=true , address=77.90.31.19:4853 , piece_id=VFEON5T2KSWM2KSLTVRZTXYL42JWENZYUEUPG3LMODGZWBMVGGPQ, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.641863297 (75h46m41.205186305s) [noise=true , address=77.90.31.12:4603 , piece_id=BMUIC6KNGEDL37436GLSXW5XOO6VYOPNOSFGQCYVHSENVI2PJ7KQ, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.642124427 (75h46m41.205447437s) [noise=true , address=104.36.82.49:4982 , piece_id=UNLV5UDNWULEIPE4QSQHHZQPK3BQKFRB5T3H6GP7ZQ5QGSNK5V6A, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.642285917 (75h46m41.205608925s) [noise=true , address=104.36.82.54:4171 , piece_id=MUMABKARK4KKUKWW6XDXCQVT6O7SLFHGHE7COWC7QIYGARAA2D5A, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.642656750 (75h46m41.205979758s) [noise=true , address=104.36.82.14:6232 , piece_id=A2ZLUUZKRTBJRJ267QSK36C3LFZV7FPAYDDAXGIWDTNEGNV6MRRA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.643565583 (75h46m41.206888597s) [noise=true , address=79.127.205.245:4231 , piece_id=EJK46XUUVI5NWOAQJEW4QIJG3KRXZHB4YOQUJRCS2MIOSWKGTNLQ, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.644620291 (75h46m41.207943303s) [noise=true , address=77.90.31.14:4670 , piece_id=7FNANMUKPM3N6YWRAOAIM4WJZJUTNF7ZBI6ZUBNEPGNGMFWOE24A, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.648842639 (75h46m41.212165631s) [noise=true , address=104.36.82.55:4224 , piece_id=WRVNGSJV5BX4GJERLXNPAS7J7BZ5WFTM7R5H257YRCG5SP2HIYMA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.825828239 (75h46m41.389151241s) [noise=true , address=104.36.82.53:4126 , piece_id=2MKRQJEPGDJKDM7VWY5M4M6NORILAIEYYATJ7HVZEOBOMXOI2LBA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: ecclient: upload failed (node:12dMWFrq2NMpnMzuPAphWerntC5kkrcnzRJZBA116uzJzyvk6fH, address:104.36.82.53:4126): protocol: storage node overloaded, request limit: 2000 Jun 21 20:10:05.834666043 (75h46m41.397989033s) [upload=60693]: Next returned nil segment Jun 21 20:10:05.834686295 (75h46m41.398009302s) [upload=60693]: Waiting for error group managing segments... Jun 21 20:10:05.877010488 (75h46m41.440333486s) [noise=true , address=45.91.116.6:4982 , piece_id=VP45R6KXNH5U5SGUQUMAPIBY7KFDPBIR3S3VRIX27EAT6VLFBZ7Q, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:05.886336964 (75h46m41.449659967s) [noise=true , address=104.36.82.9:6039 , piece_id=I4AB6N27MYYUHWAZLJJUIBFBLOQFHDIM3YLYWPVRPNUV7VMYS2SA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: ecclient: upload failed (node:12P11WVW9mVXw5ra2gZ3Rezon6wZi8GrN5CchQN3MiVU4ts3L6V, address:104.36.82.9:6039): protocol: storage node overloaded, request limit: 2000 Jun 21 20:10:08.064514542 (75h46m43.627837553s) [noise=true , address=77.90.31.14:4670 , piece_id=7FNANMUKPM3N6YWRAOAIM4WJZJUTNF7ZBI6ZUBNEPGNGMFWOE24A, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:08.099098391 (75h46m43.662421396s) [noise=true , address=79.127.205.245:4231 , piece_id=EJK46XUUVI5NWOAQJEW4QIJG3KRXZHB4YOQUJRCS2MIOSWKGTNLQ, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:08.489461749 (75h46m44.052784752s) [noise=true , address=104.36.82.42:6807 , piece_id=ZBQZPXUDJ4VBCOSVKWAJ6SK6UD7SLPON7Y4SSCK2DQ5STFG4XCNQ, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:08.650922093 (75h46m44.214245093s) [noise=true , address=77.90.31.19:4853 , piece_id=VFEON5T2KSWM2KSLTVRZTXYL42JWENZYUEUPG3LMODGZWBMVGGPQ, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:09.917746439 (75h46m45.481069441s) [noise=true , address=104.36.82.13:6196 , piece_id=BCHMQO5KKT5BLNWQGOHH3V43LGSE3BUQCI2SY4TTO5M5GQNOCTMQ, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:10.830722215 (75h46m46.394045216s) [noise=true , address=104.36.82.5:5869 , piece_id=HAEHXXZASV3DHT3QN7NTZOS6XZKHNUWBKCHWJJWSTJKXAKJKKBAA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:12.738783450 (75h46m48.302106452s) [noise=true , address=77.90.31.12:4603 , piece_id=BMUIC6KNGEDL37436GLSXW5XOO6VYOPNOSFGQCYVHSENVI2PJ7KQ, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:15.460927713 (75h46m51.024250723s) [noise=true , address=104.36.82.52:4111 , piece_id=HB7QAWN6CGNIIT4HD2GZ7JDQP5NXIJ5I2USBM66YOKTYENNOWG5A, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:18.393817968 (75h46m53.95714096s) [noise=true , address=104.36.82.55:4224 , piece_id=WRVNGSJV5BX4GJERLXNPAS7J7BZ5WFTM7R5H257YRCG5SP2HIYMA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:19.549174143 (75h46m55.112497151s) [noise=true , address=104.36.82.56:4253 , piece_id=5PMVGLZETMXCQ77K3I6PXACRIMBGR2CV5JGUSU5KUMYFNXVGA2GA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:22.511944593 (75h46m58.075267584s) [noise=true , address=104.36.82.37:6663 , piece_id=X7LWYIIHJFU2T2PHYR3R5TNT22UUPHEHQWZH7ZJMLS4AH4PJ6AIA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:48.027197558 (75h47m23.590520562s) [noise=true , address=104.36.82.14:6232 , piece_id=A2ZLUUZKRTBJRJ267QSK36C3LFZV7FPAYDDAXGIWDTNEGNV6MRRA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:11:05.567692582 (75h47m41.131015574s) [noise=true , address=104.36.82.49:4982 , piece_id=UNLV5UDNWULEIPE4QSQHHZQPK3BQKFRB5T3H6GP7ZQ5QGSNK5V6A, seg_pos={0 0}, upload=60693]: Done uploading piece. err: ecclient: upload failed (node:1NxoYJXCXmTvQncbkXt6yH7mhEBBrzVdzhgL5J6W5TD4N3pkaG, address:104.36.82.49:4982): protocol: expected piece hash; context deadline exceeded; EOF Jun 21 20:11:05.567759198 (75h47m41.131082195s) [noise=true , address=104.36.82.54:4171 , piece_id=MUMABKARK4KKUKWW6XDXCQVT6O7SLFHGHE7COWC7QIYGARAA2D5A, seg_pos={0 0}, upload=60693]: Done uploading piece. err: ecclient: upload failed (node:125SMhDsEZruNu6LkmQn2M7R6WzvXTdk86DRsKf522u8BNiANq3, address:104.36.82.54:4171): protocol: expected piece hash; context deadline exceeded; EOF Jun 21 20:11:05.567869019 (75h47m41.131192033s) [noise=true , address=104.36.82.10:6087 , piece_id=5FQG7BPZC373PFB54HYXSG2SVKOBBBSGZCLC4XDJEYQFOLQY74MA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: ecclient: upload failed (node:1S9k5eejJcxkiG5pXjjBZm18uP8k5jCGfWyeW2QQxmuRU8ZzRA, address:104.36.82.10:6087): protocol: write tcp 159.223.143.39:38550->104.36.82.10:6087: use of closed network connection; write tcp 159.223.143.39:38550->104.36.82.10:6087: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:38550->104.36.82.10:6087: use of closed network connection Jun 21 20:11:05.567932150 (75h47m41.131255161s) [seg_pos={0 0}, upload=60693]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jun 21 20:11:05.593626175 (75h47m41.156949181s) [upload=60693]: Done uploading segments... Jun 21 20:11:05.593657274 (75h47m41.156980265s) [upload=60693]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 16.873210501s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [...............X...........X................................] duration: 15.273707081s FAIL (2025-06-21 20:37:06 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== pload=60693]: Starting upload Jun 21 20:10:05.602866542 (75h46m41.166189557s) [upload=60693]: Done starting upload Jun 21 20:10:05.603116545 (75h46m41.166439547s) [upload=60693]: Uploading segments... Jun 21 20:10:05.603495389 (75h46m41.166818397s) [upload=60693]: Got next segment. Inline: false Jun 21 20:10:05.603566566 (75h46m41.166889558s) [upload=60693]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 21 20:10:05.603573280 (75h46m41.166896286s) [upload=60693]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 21 20:10:05.639130162 (75h46m41.202453164s) [seg_pos={0 0}, upload=60693]: Begin upload segment... Jun 21 20:10:05.639231070 (75h46m41.202554057s) [seg_pos={0 0}, upload=60693]: Done begin upload segment. Jun 21 20:10:05.639292724 (75h46m41.202615728s) [noise=true , address=104.36.82.56:4253 , piece_id=5PMVGLZETMXCQ77K3I6PXACRIMBGR2CV5JGUSU5KUMYFNXVGA2GA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.639338066 (75h46m41.202661071s) [noise=true , address=104.36.82.53:4126 , piece_id=2MKRQJEPGDJKDM7VWY5M4M6NORILAIEYYATJ7HVZEOBOMXOI2LBA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.639416264 (75h46m41.202739267s) [noise=true , address=104.36.82.5:5869 , piece_id=HAEHXXZASV3DHT3QN7NTZOS6XZKHNUWBKCHWJJWSTJKXAKJKKBAA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.639873098 (75h46m41.203196115s) [noise=true , address=104.36.82.52:4111 , piece_id=HB7QAWN6CGNIIT4HD2GZ7JDQP5NXIJ5I2USBM66YOKTYENNOWG5A, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.639887296 (75h46m41.203210304s) [noise=true , address=104.36.82.10:6087 , piece_id=5FQG7BPZC373PFB54HYXSG2SVKOBBBSGZCLC4XDJEYQFOLQY74MA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.640266623 (75h46m41.203589616s) [noise=true , address=104.36.82.13:6196 , piece_id=BCHMQO5KKT5BLNWQGOHH3V43LGSE3BUQCI2SY4TTO5M5GQNOCTMQ, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.640352274 (75h46m41.203675286s) [noise=true , address=104.36.82.37:6663 , piece_id=X7LWYIIHJFU2T2PHYR3R5TNT22UUPHEHQWZH7ZJMLS4AH4PJ6AIA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.640659767 (75h46m41.203982779s) [noise=true , address=104.36.82.42:6807 , piece_id=ZBQZPXUDJ4VBCOSVKWAJ6SK6UD7SLPON7Y4SSCK2DQ5STFG4XCNQ, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.640961701 (75h46m41.20428471s) [noise=true , address=45.91.116.6:4982 , piece_id=VP45R6KXNH5U5SGUQUMAPIBY7KFDPBIR3S3VRIX27EAT6VLFBZ7Q, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.641159578 (75h46m41.204482589s) [noise=true , address=104.36.82.9:6039 , piece_id=I4AB6N27MYYUHWAZLJJUIBFBLOQFHDIM3YLYWPVRPNUV7VMYS2SA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.641588126 (75h46m41.204911136s) [noise=true , address=77.90.31.19:4853 , piece_id=VFEON5T2KSWM2KSLTVRZTXYL42JWENZYUEUPG3LMODGZWBMVGGPQ, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.641863297 (75h46m41.205186305s) [noise=true , address=77.90.31.12:4603 , piece_id=BMUIC6KNGEDL37436GLSXW5XOO6VYOPNOSFGQCYVHSENVI2PJ7KQ, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.642124427 (75h46m41.205447437s) [noise=true , address=104.36.82.49:4982 , piece_id=UNLV5UDNWULEIPE4QSQHHZQPK3BQKFRB5T3H6GP7ZQ5QGSNK5V6A, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.642285917 (75h46m41.205608925s) [noise=true , address=104.36.82.54:4171 , piece_id=MUMABKARK4KKUKWW6XDXCQVT6O7SLFHGHE7COWC7QIYGARAA2D5A, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.642656750 (75h46m41.205979758s) [noise=true , address=104.36.82.14:6232 , piece_id=A2ZLUUZKRTBJRJ267QSK36C3LFZV7FPAYDDAXGIWDTNEGNV6MRRA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.643565583 (75h46m41.206888597s) [noise=true , address=79.127.205.245:4231 , piece_id=EJK46XUUVI5NWOAQJEW4QIJG3KRXZHB4YOQUJRCS2MIOSWKGTNLQ, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.644620291 (75h46m41.207943303s) [noise=true , address=77.90.31.14:4670 , piece_id=7FNANMUKPM3N6YWRAOAIM4WJZJUTNF7ZBI6ZUBNEPGNGMFWOE24A, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.648842639 (75h46m41.212165631s) [noise=true , address=104.36.82.55:4224 , piece_id=WRVNGSJV5BX4GJERLXNPAS7J7BZ5WFTM7R5H257YRCG5SP2HIYMA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.825828239 (75h46m41.389151241s) [noise=true , address=104.36.82.53:4126 , piece_id=2MKRQJEPGDJKDM7VWY5M4M6NORILAIEYYATJ7HVZEOBOMXOI2LBA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: ecclient: upload failed (node:12dMWFrq2NMpnMzuPAphWerntC5kkrcnzRJZBA116uzJzyvk6fH, address:104.36.82.53:4126): protocol: storage node overloaded, request limit: 2000 Jun 21 20:10:05.834666043 (75h46m41.397989033s) [upload=60693]: Next returned nil segment Jun 21 20:10:05.834686295 (75h46m41.398009302s) [upload=60693]: Waiting for error group managing segments... Jun 21 20:10:05.877010488 (75h46m41.440333486s) [noise=true , address=45.91.116.6:4982 , piece_id=VP45R6KXNH5U5SGUQUMAPIBY7KFDPBIR3S3VRIX27EAT6VLFBZ7Q, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:05.886336964 (75h46m41.449659967s) [noise=true , address=104.36.82.9:6039 , piece_id=I4AB6N27MYYUHWAZLJJUIBFBLOQFHDIM3YLYWPVRPNUV7VMYS2SA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: ecclient: upload failed (node:12P11WVW9mVXw5ra2gZ3Rezon6wZi8GrN5CchQN3MiVU4ts3L6V, address:104.36.82.9:6039): protocol: storage node overloaded, request limit: 2000 Jun 21 20:10:08.064514542 (75h46m43.627837553s) [noise=true , address=77.90.31.14:4670 , piece_id=7FNANMUKPM3N6YWRAOAIM4WJZJUTNF7ZBI6ZUBNEPGNGMFWOE24A, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:08.099098391 (75h46m43.662421396s) [noise=true , address=79.127.205.245:4231 , piece_id=EJK46XUUVI5NWOAQJEW4QIJG3KRXZHB4YOQUJRCS2MIOSWKGTNLQ, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:08.489461749 (75h46m44.052784752s) [noise=true , address=104.36.82.42:6807 , piece_id=ZBQZPXUDJ4VBCOSVKWAJ6SK6UD7SLPON7Y4SSCK2DQ5STFG4XCNQ, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:08.650922093 (75h46m44.214245093s) [noise=true , address=77.90.31.19:4853 , piece_id=VFEON5T2KSWM2KSLTVRZTXYL42JWENZYUEUPG3LMODGZWBMVGGPQ, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:09.917746439 (75h46m45.481069441s) [noise=true , address=104.36.82.13:6196 , piece_id=BCHMQO5KKT5BLNWQGOHH3V43LGSE3BUQCI2SY4TTO5M5GQNOCTMQ, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:10.830722215 (75h46m46.394045216s) [noise=true , address=104.36.82.5:5869 , piece_id=HAEHXXZASV3DHT3QN7NTZOS6XZKHNUWBKCHWJJWSTJKXAKJKKBAA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:12.738783450 (75h46m48.302106452s) [noise=true , address=77.90.31.12:4603 , piece_id=BMUIC6KNGEDL37436GLSXW5XOO6VYOPNOSFGQCYVHSENVI2PJ7KQ, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:15.460927713 (75h46m51.024250723s) [noise=true , address=104.36.82.52:4111 , piece_id=HB7QAWN6CGNIIT4HD2GZ7JDQP5NXIJ5I2USBM66YOKTYENNOWG5A, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:18.393817968 (75h46m53.95714096s) [noise=true , address=104.36.82.55:4224 , piece_id=WRVNGSJV5BX4GJERLXNPAS7J7BZ5WFTM7R5H257YRCG5SP2HIYMA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:19.549174143 (75h46m55.112497151s) [noise=true , address=104.36.82.56:4253 , piece_id=5PMVGLZETMXCQ77K3I6PXACRIMBGR2CV5JGUSU5KUMYFNXVGA2GA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:22.511944593 (75h46m58.075267584s) [noise=true , address=104.36.82.37:6663 , piece_id=X7LWYIIHJFU2T2PHYR3R5TNT22UUPHEHQWZH7ZJMLS4AH4PJ6AIA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:48.027197558 (75h47m23.590520562s) [noise=true , address=104.36.82.14:6232 , piece_id=A2ZLUUZKRTBJRJ267QSK36C3LFZV7FPAYDDAXGIWDTNEGNV6MRRA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:11:05.567692582 (75h47m41.131015574s) [noise=true , address=104.36.82.49:4982 , piece_id=UNLV5UDNWULEIPE4QSQHHZQPK3BQKFRB5T3H6GP7ZQ5QGSNK5V6A, seg_pos={0 0}, upload=60693]: Done uploading piece. err: ecclient: upload failed (node:1NxoYJXCXmTvQncbkXt6yH7mhEBBrzVdzhgL5J6W5TD4N3pkaG, address:104.36.82.49:4982): protocol: expected piece hash; context deadline exceeded; EOF Jun 21 20:11:05.567759198 (75h47m41.131082195s) [noise=true , address=104.36.82.54:4171 , piece_id=MUMABKARK4KKUKWW6XDXCQVT6O7SLFHGHE7COWC7QIYGARAA2D5A, seg_pos={0 0}, upload=60693]: Done uploading piece. err: ecclient: upload failed (node:125SMhDsEZruNu6LkmQn2M7R6WzvXTdk86DRsKf522u8BNiANq3, address:104.36.82.54:4171): protocol: expected piece hash; context deadline exceeded; EOF Jun 21 20:11:05.567869019 (75h47m41.131192033s) [noise=true , address=104.36.82.10:6087 , piece_id=5FQG7BPZC373PFB54HYXSG2SVKOBBBSGZCLC4XDJEYQFOLQY74MA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: ecclient: upload failed (node:1S9k5eejJcxkiG5pXjjBZm18uP8k5jCGfWyeW2QQxmuRU8ZzRA, address:104.36.82.10:6087): protocol: write tcp 159.223.143.39:38550->104.36.82.10:6087: use of closed network connection; write tcp 159.223.143.39:38550->104.36.82.10:6087: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:38550->104.36.82.10:6087: use of closed network connection Jun 21 20:11:05.567932150 (75h47m41.131255161s) [seg_pos={0 0}, upload=60693]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jun 21 20:11:05.593626175 (75h47m41.156949181s) [upload=60693]: Done uploading segments... Jun 21 20:11:05.593657274 (75h47m41.156980265s) [upload=60693]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 15.273731147s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [................X...........X...............................] duration: 24.266877857s FAIL (2025-06-21 20:36:41 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== pload=60693]: Starting upload Jun 21 20:10:05.602866542 (75h46m41.166189557s) [upload=60693]: Done starting upload Jun 21 20:10:05.603116545 (75h46m41.166439547s) [upload=60693]: Uploading segments... Jun 21 20:10:05.603495389 (75h46m41.166818397s) [upload=60693]: Got next segment. Inline: false Jun 21 20:10:05.603566566 (75h46m41.166889558s) [upload=60693]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 21 20:10:05.603573280 (75h46m41.166896286s) [upload=60693]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 21 20:10:05.639130162 (75h46m41.202453164s) [seg_pos={0 0}, upload=60693]: Begin upload segment... Jun 21 20:10:05.639231070 (75h46m41.202554057s) [seg_pos={0 0}, upload=60693]: Done begin upload segment. Jun 21 20:10:05.639292724 (75h46m41.202615728s) [noise=true , address=104.36.82.56:4253 , piece_id=5PMVGLZETMXCQ77K3I6PXACRIMBGR2CV5JGUSU5KUMYFNXVGA2GA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.639338066 (75h46m41.202661071s) [noise=true , address=104.36.82.53:4126 , piece_id=2MKRQJEPGDJKDM7VWY5M4M6NORILAIEYYATJ7HVZEOBOMXOI2LBA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.639416264 (75h46m41.202739267s) [noise=true , address=104.36.82.5:5869 , piece_id=HAEHXXZASV3DHT3QN7NTZOS6XZKHNUWBKCHWJJWSTJKXAKJKKBAA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.639873098 (75h46m41.203196115s) [noise=true , address=104.36.82.52:4111 , piece_id=HB7QAWN6CGNIIT4HD2GZ7JDQP5NXIJ5I2USBM66YOKTYENNOWG5A, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.639887296 (75h46m41.203210304s) [noise=true , address=104.36.82.10:6087 , piece_id=5FQG7BPZC373PFB54HYXSG2SVKOBBBSGZCLC4XDJEYQFOLQY74MA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.640266623 (75h46m41.203589616s) [noise=true , address=104.36.82.13:6196 , piece_id=BCHMQO5KKT5BLNWQGOHH3V43LGSE3BUQCI2SY4TTO5M5GQNOCTMQ, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.640352274 (75h46m41.203675286s) [noise=true , address=104.36.82.37:6663 , piece_id=X7LWYIIHJFU2T2PHYR3R5TNT22UUPHEHQWZH7ZJMLS4AH4PJ6AIA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.640659767 (75h46m41.203982779s) [noise=true , address=104.36.82.42:6807 , piece_id=ZBQZPXUDJ4VBCOSVKWAJ6SK6UD7SLPON7Y4SSCK2DQ5STFG4XCNQ, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.640961701 (75h46m41.20428471s) [noise=true , address=45.91.116.6:4982 , piece_id=VP45R6KXNH5U5SGUQUMAPIBY7KFDPBIR3S3VRIX27EAT6VLFBZ7Q, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.641159578 (75h46m41.204482589s) [noise=true , address=104.36.82.9:6039 , piece_id=I4AB6N27MYYUHWAZLJJUIBFBLOQFHDIM3YLYWPVRPNUV7VMYS2SA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.641588126 (75h46m41.204911136s) [noise=true , address=77.90.31.19:4853 , piece_id=VFEON5T2KSWM2KSLTVRZTXYL42JWENZYUEUPG3LMODGZWBMVGGPQ, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.641863297 (75h46m41.205186305s) [noise=true , address=77.90.31.12:4603 , piece_id=BMUIC6KNGEDL37436GLSXW5XOO6VYOPNOSFGQCYVHSENVI2PJ7KQ, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.642124427 (75h46m41.205447437s) [noise=true , address=104.36.82.49:4982 , piece_id=UNLV5UDNWULEIPE4QSQHHZQPK3BQKFRB5T3H6GP7ZQ5QGSNK5V6A, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.642285917 (75h46m41.205608925s) [noise=true , address=104.36.82.54:4171 , piece_id=MUMABKARK4KKUKWW6XDXCQVT6O7SLFHGHE7COWC7QIYGARAA2D5A, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.642656750 (75h46m41.205979758s) [noise=true , address=104.36.82.14:6232 , piece_id=A2ZLUUZKRTBJRJ267QSK36C3LFZV7FPAYDDAXGIWDTNEGNV6MRRA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.643565583 (75h46m41.206888597s) [noise=true , address=79.127.205.245:4231 , piece_id=EJK46XUUVI5NWOAQJEW4QIJG3KRXZHB4YOQUJRCS2MIOSWKGTNLQ, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.644620291 (75h46m41.207943303s) [noise=true , address=77.90.31.14:4670 , piece_id=7FNANMUKPM3N6YWRAOAIM4WJZJUTNF7ZBI6ZUBNEPGNGMFWOE24A, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.648842639 (75h46m41.212165631s) [noise=true , address=104.36.82.55:4224 , piece_id=WRVNGSJV5BX4GJERLXNPAS7J7BZ5WFTM7R5H257YRCG5SP2HIYMA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.825828239 (75h46m41.389151241s) [noise=true , address=104.36.82.53:4126 , piece_id=2MKRQJEPGDJKDM7VWY5M4M6NORILAIEYYATJ7HVZEOBOMXOI2LBA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: ecclient: upload failed (node:12dMWFrq2NMpnMzuPAphWerntC5kkrcnzRJZBA116uzJzyvk6fH, address:104.36.82.53:4126): protocol: storage node overloaded, request limit: 2000 Jun 21 20:10:05.834666043 (75h46m41.397989033s) [upload=60693]: Next returned nil segment Jun 21 20:10:05.834686295 (75h46m41.398009302s) [upload=60693]: Waiting for error group managing segments... Jun 21 20:10:05.877010488 (75h46m41.440333486s) [noise=true , address=45.91.116.6:4982 , piece_id=VP45R6KXNH5U5SGUQUMAPIBY7KFDPBIR3S3VRIX27EAT6VLFBZ7Q, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:05.886336964 (75h46m41.449659967s) [noise=true , address=104.36.82.9:6039 , piece_id=I4AB6N27MYYUHWAZLJJUIBFBLOQFHDIM3YLYWPVRPNUV7VMYS2SA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: ecclient: upload failed (node:12P11WVW9mVXw5ra2gZ3Rezon6wZi8GrN5CchQN3MiVU4ts3L6V, address:104.36.82.9:6039): protocol: storage node overloaded, request limit: 2000 Jun 21 20:10:08.064514542 (75h46m43.627837553s) [noise=true , address=77.90.31.14:4670 , piece_id=7FNANMUKPM3N6YWRAOAIM4WJZJUTNF7ZBI6ZUBNEPGNGMFWOE24A, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:08.099098391 (75h46m43.662421396s) [noise=true , address=79.127.205.245:4231 , piece_id=EJK46XUUVI5NWOAQJEW4QIJG3KRXZHB4YOQUJRCS2MIOSWKGTNLQ, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:08.489461749 (75h46m44.052784752s) [noise=true , address=104.36.82.42:6807 , piece_id=ZBQZPXUDJ4VBCOSVKWAJ6SK6UD7SLPON7Y4SSCK2DQ5STFG4XCNQ, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:08.650922093 (75h46m44.214245093s) [noise=true , address=77.90.31.19:4853 , piece_id=VFEON5T2KSWM2KSLTVRZTXYL42JWENZYUEUPG3LMODGZWBMVGGPQ, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:09.917746439 (75h46m45.481069441s) [noise=true , address=104.36.82.13:6196 , piece_id=BCHMQO5KKT5BLNWQGOHH3V43LGSE3BUQCI2SY4TTO5M5GQNOCTMQ, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:10.830722215 (75h46m46.394045216s) [noise=true , address=104.36.82.5:5869 , piece_id=HAEHXXZASV3DHT3QN7NTZOS6XZKHNUWBKCHWJJWSTJKXAKJKKBAA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:12.738783450 (75h46m48.302106452s) [noise=true , address=77.90.31.12:4603 , piece_id=BMUIC6KNGEDL37436GLSXW5XOO6VYOPNOSFGQCYVHSENVI2PJ7KQ, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:15.460927713 (75h46m51.024250723s) [noise=true , address=104.36.82.52:4111 , piece_id=HB7QAWN6CGNIIT4HD2GZ7JDQP5NXIJ5I2USBM66YOKTYENNOWG5A, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:18.393817968 (75h46m53.95714096s) [noise=true , address=104.36.82.55:4224 , piece_id=WRVNGSJV5BX4GJERLXNPAS7J7BZ5WFTM7R5H257YRCG5SP2HIYMA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:19.549174143 (75h46m55.112497151s) [noise=true , address=104.36.82.56:4253 , piece_id=5PMVGLZETMXCQ77K3I6PXACRIMBGR2CV5JGUSU5KUMYFNXVGA2GA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:22.511944593 (75h46m58.075267584s) [noise=true , address=104.36.82.37:6663 , piece_id=X7LWYIIHJFU2T2PHYR3R5TNT22UUPHEHQWZH7ZJMLS4AH4PJ6AIA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:48.027197558 (75h47m23.590520562s) [noise=true , address=104.36.82.14:6232 , piece_id=A2ZLUUZKRTBJRJ267QSK36C3LFZV7FPAYDDAXGIWDTNEGNV6MRRA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:11:05.567692582 (75h47m41.131015574s) [noise=true , address=104.36.82.49:4982 , piece_id=UNLV5UDNWULEIPE4QSQHHZQPK3BQKFRB5T3H6GP7ZQ5QGSNK5V6A, seg_pos={0 0}, upload=60693]: Done uploading piece. err: ecclient: upload failed (node:1NxoYJXCXmTvQncbkXt6yH7mhEBBrzVdzhgL5J6W5TD4N3pkaG, address:104.36.82.49:4982): protocol: expected piece hash; context deadline exceeded; EOF Jun 21 20:11:05.567759198 (75h47m41.131082195s) [noise=true , address=104.36.82.54:4171 , piece_id=MUMABKARK4KKUKWW6XDXCQVT6O7SLFHGHE7COWC7QIYGARAA2D5A, seg_pos={0 0}, upload=60693]: Done uploading piece. err: ecclient: upload failed (node:125SMhDsEZruNu6LkmQn2M7R6WzvXTdk86DRsKf522u8BNiANq3, address:104.36.82.54:4171): protocol: expected piece hash; context deadline exceeded; EOF Jun 21 20:11:05.567869019 (75h47m41.131192033s) [noise=true , address=104.36.82.10:6087 , piece_id=5FQG7BPZC373PFB54HYXSG2SVKOBBBSGZCLC4XDJEYQFOLQY74MA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: ecclient: upload failed (node:1S9k5eejJcxkiG5pXjjBZm18uP8k5jCGfWyeW2QQxmuRU8ZzRA, address:104.36.82.10:6087): protocol: write tcp 159.223.143.39:38550->104.36.82.10:6087: use of closed network connection; write tcp 159.223.143.39:38550->104.36.82.10:6087: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:38550->104.36.82.10:6087: use of closed network connection Jun 21 20:11:05.567932150 (75h47m41.131255161s) [seg_pos={0 0}, upload=60693]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jun 21 20:11:05.593626175 (75h47m41.156949181s) [upload=60693]: Done uploading segments... Jun 21 20:11:05.593657274 (75h47m41.156980265s) [upload=60693]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 24.266913646s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [.................X...........X..............................] duration: 10.041604498s FAIL (2025-06-21 20:35:49 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== pload=60693]: Starting upload Jun 21 20:10:05.602866542 (75h46m41.166189557s) [upload=60693]: Done starting upload Jun 21 20:10:05.603116545 (75h46m41.166439547s) [upload=60693]: Uploading segments... Jun 21 20:10:05.603495389 (75h46m41.166818397s) [upload=60693]: Got next segment. Inline: false Jun 21 20:10:05.603566566 (75h46m41.166889558s) [upload=60693]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 21 20:10:05.603573280 (75h46m41.166896286s) [upload=60693]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 21 20:10:05.639130162 (75h46m41.202453164s) [seg_pos={0 0}, upload=60693]: Begin upload segment... Jun 21 20:10:05.639231070 (75h46m41.202554057s) [seg_pos={0 0}, upload=60693]: Done begin upload segment. Jun 21 20:10:05.639292724 (75h46m41.202615728s) [noise=true , address=104.36.82.56:4253 , piece_id=5PMVGLZETMXCQ77K3I6PXACRIMBGR2CV5JGUSU5KUMYFNXVGA2GA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.639338066 (75h46m41.202661071s) [noise=true , address=104.36.82.53:4126 , piece_id=2MKRQJEPGDJKDM7VWY5M4M6NORILAIEYYATJ7HVZEOBOMXOI2LBA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.639416264 (75h46m41.202739267s) [noise=true , address=104.36.82.5:5869 , piece_id=HAEHXXZASV3DHT3QN7NTZOS6XZKHNUWBKCHWJJWSTJKXAKJKKBAA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.639873098 (75h46m41.203196115s) [noise=true , address=104.36.82.52:4111 , piece_id=HB7QAWN6CGNIIT4HD2GZ7JDQP5NXIJ5I2USBM66YOKTYENNOWG5A, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.639887296 (75h46m41.203210304s) [noise=true , address=104.36.82.10:6087 , piece_id=5FQG7BPZC373PFB54HYXSG2SVKOBBBSGZCLC4XDJEYQFOLQY74MA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.640266623 (75h46m41.203589616s) [noise=true , address=104.36.82.13:6196 , piece_id=BCHMQO5KKT5BLNWQGOHH3V43LGSE3BUQCI2SY4TTO5M5GQNOCTMQ, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.640352274 (75h46m41.203675286s) [noise=true , address=104.36.82.37:6663 , piece_id=X7LWYIIHJFU2T2PHYR3R5TNT22UUPHEHQWZH7ZJMLS4AH4PJ6AIA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.640659767 (75h46m41.203982779s) [noise=true , address=104.36.82.42:6807 , piece_id=ZBQZPXUDJ4VBCOSVKWAJ6SK6UD7SLPON7Y4SSCK2DQ5STFG4XCNQ, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.640961701 (75h46m41.20428471s) [noise=true , address=45.91.116.6:4982 , piece_id=VP45R6KXNH5U5SGUQUMAPIBY7KFDPBIR3S3VRIX27EAT6VLFBZ7Q, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.641159578 (75h46m41.204482589s) [noise=true , address=104.36.82.9:6039 , piece_id=I4AB6N27MYYUHWAZLJJUIBFBLOQFHDIM3YLYWPVRPNUV7VMYS2SA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.641588126 (75h46m41.204911136s) [noise=true , address=77.90.31.19:4853 , piece_id=VFEON5T2KSWM2KSLTVRZTXYL42JWENZYUEUPG3LMODGZWBMVGGPQ, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.641863297 (75h46m41.205186305s) [noise=true , address=77.90.31.12:4603 , piece_id=BMUIC6KNGEDL37436GLSXW5XOO6VYOPNOSFGQCYVHSENVI2PJ7KQ, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.642124427 (75h46m41.205447437s) [noise=true , address=104.36.82.49:4982 , piece_id=UNLV5UDNWULEIPE4QSQHHZQPK3BQKFRB5T3H6GP7ZQ5QGSNK5V6A, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.642285917 (75h46m41.205608925s) [noise=true , address=104.36.82.54:4171 , piece_id=MUMABKARK4KKUKWW6XDXCQVT6O7SLFHGHE7COWC7QIYGARAA2D5A, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.642656750 (75h46m41.205979758s) [noise=true , address=104.36.82.14:6232 , piece_id=A2ZLUUZKRTBJRJ267QSK36C3LFZV7FPAYDDAXGIWDTNEGNV6MRRA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.643565583 (75h46m41.206888597s) [noise=true , address=79.127.205.245:4231 , piece_id=EJK46XUUVI5NWOAQJEW4QIJG3KRXZHB4YOQUJRCS2MIOSWKGTNLQ, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.644620291 (75h46m41.207943303s) [noise=true , address=77.90.31.14:4670 , piece_id=7FNANMUKPM3N6YWRAOAIM4WJZJUTNF7ZBI6ZUBNEPGNGMFWOE24A, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.648842639 (75h46m41.212165631s) [noise=true , address=104.36.82.55:4224 , piece_id=WRVNGSJV5BX4GJERLXNPAS7J7BZ5WFTM7R5H257YRCG5SP2HIYMA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.825828239 (75h46m41.389151241s) [noise=true , address=104.36.82.53:4126 , piece_id=2MKRQJEPGDJKDM7VWY5M4M6NORILAIEYYATJ7HVZEOBOMXOI2LBA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: ecclient: upload failed (node:12dMWFrq2NMpnMzuPAphWerntC5kkrcnzRJZBA116uzJzyvk6fH, address:104.36.82.53:4126): protocol: storage node overloaded, request limit: 2000 Jun 21 20:10:05.834666043 (75h46m41.397989033s) [upload=60693]: Next returned nil segment Jun 21 20:10:05.834686295 (75h46m41.398009302s) [upload=60693]: Waiting for error group managing segments... Jun 21 20:10:05.877010488 (75h46m41.440333486s) [noise=true , address=45.91.116.6:4982 , piece_id=VP45R6KXNH5U5SGUQUMAPIBY7KFDPBIR3S3VRIX27EAT6VLFBZ7Q, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:05.886336964 (75h46m41.449659967s) [noise=true , address=104.36.82.9:6039 , piece_id=I4AB6N27MYYUHWAZLJJUIBFBLOQFHDIM3YLYWPVRPNUV7VMYS2SA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: ecclient: upload failed (node:12P11WVW9mVXw5ra2gZ3Rezon6wZi8GrN5CchQN3MiVU4ts3L6V, address:104.36.82.9:6039): protocol: storage node overloaded, request limit: 2000 Jun 21 20:10:08.064514542 (75h46m43.627837553s) [noise=true , address=77.90.31.14:4670 , piece_id=7FNANMUKPM3N6YWRAOAIM4WJZJUTNF7ZBI6ZUBNEPGNGMFWOE24A, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:08.099098391 (75h46m43.662421396s) [noise=true , address=79.127.205.245:4231 , piece_id=EJK46XUUVI5NWOAQJEW4QIJG3KRXZHB4YOQUJRCS2MIOSWKGTNLQ, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:08.489461749 (75h46m44.052784752s) [noise=true , address=104.36.82.42:6807 , piece_id=ZBQZPXUDJ4VBCOSVKWAJ6SK6UD7SLPON7Y4SSCK2DQ5STFG4XCNQ, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:08.650922093 (75h46m44.214245093s) [noise=true , address=77.90.31.19:4853 , piece_id=VFEON5T2KSWM2KSLTVRZTXYL42JWENZYUEUPG3LMODGZWBMVGGPQ, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:09.917746439 (75h46m45.481069441s) [noise=true , address=104.36.82.13:6196 , piece_id=BCHMQO5KKT5BLNWQGOHH3V43LGSE3BUQCI2SY4TTO5M5GQNOCTMQ, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:10.830722215 (75h46m46.394045216s) [noise=true , address=104.36.82.5:5869 , piece_id=HAEHXXZASV3DHT3QN7NTZOS6XZKHNUWBKCHWJJWSTJKXAKJKKBAA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:12.738783450 (75h46m48.302106452s) [noise=true , address=77.90.31.12:4603 , piece_id=BMUIC6KNGEDL37436GLSXW5XOO6VYOPNOSFGQCYVHSENVI2PJ7KQ, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:15.460927713 (75h46m51.024250723s) [noise=true , address=104.36.82.52:4111 , piece_id=HB7QAWN6CGNIIT4HD2GZ7JDQP5NXIJ5I2USBM66YOKTYENNOWG5A, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:18.393817968 (75h46m53.95714096s) [noise=true , address=104.36.82.55:4224 , piece_id=WRVNGSJV5BX4GJERLXNPAS7J7BZ5WFTM7R5H257YRCG5SP2HIYMA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:19.549174143 (75h46m55.112497151s) [noise=true , address=104.36.82.56:4253 , piece_id=5PMVGLZETMXCQ77K3I6PXACRIMBGR2CV5JGUSU5KUMYFNXVGA2GA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:22.511944593 (75h46m58.075267584s) [noise=true , address=104.36.82.37:6663 , piece_id=X7LWYIIHJFU2T2PHYR3R5TNT22UUPHEHQWZH7ZJMLS4AH4PJ6AIA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:48.027197558 (75h47m23.590520562s) [noise=true , address=104.36.82.14:6232 , piece_id=A2ZLUUZKRTBJRJ267QSK36C3LFZV7FPAYDDAXGIWDTNEGNV6MRRA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:11:05.567692582 (75h47m41.131015574s) [noise=true , address=104.36.82.49:4982 , piece_id=UNLV5UDNWULEIPE4QSQHHZQPK3BQKFRB5T3H6GP7ZQ5QGSNK5V6A, seg_pos={0 0}, upload=60693]: Done uploading piece. err: ecclient: upload failed (node:1NxoYJXCXmTvQncbkXt6yH7mhEBBrzVdzhgL5J6W5TD4N3pkaG, address:104.36.82.49:4982): protocol: expected piece hash; context deadline exceeded; EOF Jun 21 20:11:05.567759198 (75h47m41.131082195s) [noise=true , address=104.36.82.54:4171 , piece_id=MUMABKARK4KKUKWW6XDXCQVT6O7SLFHGHE7COWC7QIYGARAA2D5A, seg_pos={0 0}, upload=60693]: Done uploading piece. err: ecclient: upload failed (node:125SMhDsEZruNu6LkmQn2M7R6WzvXTdk86DRsKf522u8BNiANq3, address:104.36.82.54:4171): protocol: expected piece hash; context deadline exceeded; EOF Jun 21 20:11:05.567869019 (75h47m41.131192033s) [noise=true , address=104.36.82.10:6087 , piece_id=5FQG7BPZC373PFB54HYXSG2SVKOBBBSGZCLC4XDJEYQFOLQY74MA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: ecclient: upload failed (node:1S9k5eejJcxkiG5pXjjBZm18uP8k5jCGfWyeW2QQxmuRU8ZzRA, address:104.36.82.10:6087): protocol: write tcp 159.223.143.39:38550->104.36.82.10:6087: use of closed network connection; write tcp 159.223.143.39:38550->104.36.82.10:6087: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:38550->104.36.82.10:6087: use of closed network connection Jun 21 20:11:05.567932150 (75h47m41.131255161s) [seg_pos={0 0}, upload=60693]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jun 21 20:11:05.593626175 (75h47m41.156949181s) [upload=60693]: Done uploading segments... Jun 21 20:11:05.593657274 (75h47m41.156980265s) [upload=60693]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 10.041635478s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [..................X...........X.............................] duration: 26.9442952s FAIL (2025-06-21 20:35:21 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== pload=60693]: Starting upload Jun 21 20:10:05.602866542 (75h46m41.166189557s) [upload=60693]: Done starting upload Jun 21 20:10:05.603116545 (75h46m41.166439547s) [upload=60693]: Uploading segments... Jun 21 20:10:05.603495389 (75h46m41.166818397s) [upload=60693]: Got next segment. Inline: false Jun 21 20:10:05.603566566 (75h46m41.166889558s) [upload=60693]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 21 20:10:05.603573280 (75h46m41.166896286s) [upload=60693]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 21 20:10:05.639130162 (75h46m41.202453164s) [seg_pos={0 0}, upload=60693]: Begin upload segment... Jun 21 20:10:05.639231070 (75h46m41.202554057s) [seg_pos={0 0}, upload=60693]: Done begin upload segment. Jun 21 20:10:05.639292724 (75h46m41.202615728s) [noise=true , address=104.36.82.56:4253 , piece_id=5PMVGLZETMXCQ77K3I6PXACRIMBGR2CV5JGUSU5KUMYFNXVGA2GA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.639338066 (75h46m41.202661071s) [noise=true , address=104.36.82.53:4126 , piece_id=2MKRQJEPGDJKDM7VWY5M4M6NORILAIEYYATJ7HVZEOBOMXOI2LBA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.639416264 (75h46m41.202739267s) [noise=true , address=104.36.82.5:5869 , piece_id=HAEHXXZASV3DHT3QN7NTZOS6XZKHNUWBKCHWJJWSTJKXAKJKKBAA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.639873098 (75h46m41.203196115s) [noise=true , address=104.36.82.52:4111 , piece_id=HB7QAWN6CGNIIT4HD2GZ7JDQP5NXIJ5I2USBM66YOKTYENNOWG5A, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.639887296 (75h46m41.203210304s) [noise=true , address=104.36.82.10:6087 , piece_id=5FQG7BPZC373PFB54HYXSG2SVKOBBBSGZCLC4XDJEYQFOLQY74MA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.640266623 (75h46m41.203589616s) [noise=true , address=104.36.82.13:6196 , piece_id=BCHMQO5KKT5BLNWQGOHH3V43LGSE3BUQCI2SY4TTO5M5GQNOCTMQ, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.640352274 (75h46m41.203675286s) [noise=true , address=104.36.82.37:6663 , piece_id=X7LWYIIHJFU2T2PHYR3R5TNT22UUPHEHQWZH7ZJMLS4AH4PJ6AIA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.640659767 (75h46m41.203982779s) [noise=true , address=104.36.82.42:6807 , piece_id=ZBQZPXUDJ4VBCOSVKWAJ6SK6UD7SLPON7Y4SSCK2DQ5STFG4XCNQ, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.640961701 (75h46m41.20428471s) [noise=true , address=45.91.116.6:4982 , piece_id=VP45R6KXNH5U5SGUQUMAPIBY7KFDPBIR3S3VRIX27EAT6VLFBZ7Q, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.641159578 (75h46m41.204482589s) [noise=true , address=104.36.82.9:6039 , piece_id=I4AB6N27MYYUHWAZLJJUIBFBLOQFHDIM3YLYWPVRPNUV7VMYS2SA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.641588126 (75h46m41.204911136s) [noise=true , address=77.90.31.19:4853 , piece_id=VFEON5T2KSWM2KSLTVRZTXYL42JWENZYUEUPG3LMODGZWBMVGGPQ, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.641863297 (75h46m41.205186305s) [noise=true , address=77.90.31.12:4603 , piece_id=BMUIC6KNGEDL37436GLSXW5XOO6VYOPNOSFGQCYVHSENVI2PJ7KQ, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.642124427 (75h46m41.205447437s) [noise=true , address=104.36.82.49:4982 , piece_id=UNLV5UDNWULEIPE4QSQHHZQPK3BQKFRB5T3H6GP7ZQ5QGSNK5V6A, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.642285917 (75h46m41.205608925s) [noise=true , address=104.36.82.54:4171 , piece_id=MUMABKARK4KKUKWW6XDXCQVT6O7SLFHGHE7COWC7QIYGARAA2D5A, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.642656750 (75h46m41.205979758s) [noise=true , address=104.36.82.14:6232 , piece_id=A2ZLUUZKRTBJRJ267QSK36C3LFZV7FPAYDDAXGIWDTNEGNV6MRRA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.643565583 (75h46m41.206888597s) [noise=true , address=79.127.205.245:4231 , piece_id=EJK46XUUVI5NWOAQJEW4QIJG3KRXZHB4YOQUJRCS2MIOSWKGTNLQ, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.644620291 (75h46m41.207943303s) [noise=true , address=77.90.31.14:4670 , piece_id=7FNANMUKPM3N6YWRAOAIM4WJZJUTNF7ZBI6ZUBNEPGNGMFWOE24A, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.648842639 (75h46m41.212165631s) [noise=true , address=104.36.82.55:4224 , piece_id=WRVNGSJV5BX4GJERLXNPAS7J7BZ5WFTM7R5H257YRCG5SP2HIYMA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.825828239 (75h46m41.389151241s) [noise=true , address=104.36.82.53:4126 , piece_id=2MKRQJEPGDJKDM7VWY5M4M6NORILAIEYYATJ7HVZEOBOMXOI2LBA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: ecclient: upload failed (node:12dMWFrq2NMpnMzuPAphWerntC5kkrcnzRJZBA116uzJzyvk6fH, address:104.36.82.53:4126): protocol: storage node overloaded, request limit: 2000 Jun 21 20:10:05.834666043 (75h46m41.397989033s) [upload=60693]: Next returned nil segment Jun 21 20:10:05.834686295 (75h46m41.398009302s) [upload=60693]: Waiting for error group managing segments... Jun 21 20:10:05.877010488 (75h46m41.440333486s) [noise=true , address=45.91.116.6:4982 , piece_id=VP45R6KXNH5U5SGUQUMAPIBY7KFDPBIR3S3VRIX27EAT6VLFBZ7Q, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:05.886336964 (75h46m41.449659967s) [noise=true , address=104.36.82.9:6039 , piece_id=I4AB6N27MYYUHWAZLJJUIBFBLOQFHDIM3YLYWPVRPNUV7VMYS2SA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: ecclient: upload failed (node:12P11WVW9mVXw5ra2gZ3Rezon6wZi8GrN5CchQN3MiVU4ts3L6V, address:104.36.82.9:6039): protocol: storage node overloaded, request limit: 2000 Jun 21 20:10:08.064514542 (75h46m43.627837553s) [noise=true , address=77.90.31.14:4670 , piece_id=7FNANMUKPM3N6YWRAOAIM4WJZJUTNF7ZBI6ZUBNEPGNGMFWOE24A, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:08.099098391 (75h46m43.662421396s) [noise=true , address=79.127.205.245:4231 , piece_id=EJK46XUUVI5NWOAQJEW4QIJG3KRXZHB4YOQUJRCS2MIOSWKGTNLQ, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:08.489461749 (75h46m44.052784752s) [noise=true , address=104.36.82.42:6807 , piece_id=ZBQZPXUDJ4VBCOSVKWAJ6SK6UD7SLPON7Y4SSCK2DQ5STFG4XCNQ, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:08.650922093 (75h46m44.214245093s) [noise=true , address=77.90.31.19:4853 , piece_id=VFEON5T2KSWM2KSLTVRZTXYL42JWENZYUEUPG3LMODGZWBMVGGPQ, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:09.917746439 (75h46m45.481069441s) [noise=true , address=104.36.82.13:6196 , piece_id=BCHMQO5KKT5BLNWQGOHH3V43LGSE3BUQCI2SY4TTO5M5GQNOCTMQ, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:10.830722215 (75h46m46.394045216s) [noise=true , address=104.36.82.5:5869 , piece_id=HAEHXXZASV3DHT3QN7NTZOS6XZKHNUWBKCHWJJWSTJKXAKJKKBAA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:12.738783450 (75h46m48.302106452s) [noise=true , address=77.90.31.12:4603 , piece_id=BMUIC6KNGEDL37436GLSXW5XOO6VYOPNOSFGQCYVHSENVI2PJ7KQ, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:15.460927713 (75h46m51.024250723s) [noise=true , address=104.36.82.52:4111 , piece_id=HB7QAWN6CGNIIT4HD2GZ7JDQP5NXIJ5I2USBM66YOKTYENNOWG5A, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:18.393817968 (75h46m53.95714096s) [noise=true , address=104.36.82.55:4224 , piece_id=WRVNGSJV5BX4GJERLXNPAS7J7BZ5WFTM7R5H257YRCG5SP2HIYMA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:19.549174143 (75h46m55.112497151s) [noise=true , address=104.36.82.56:4253 , piece_id=5PMVGLZETMXCQ77K3I6PXACRIMBGR2CV5JGUSU5KUMYFNXVGA2GA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:22.511944593 (75h46m58.075267584s) [noise=true , address=104.36.82.37:6663 , piece_id=X7LWYIIHJFU2T2PHYR3R5TNT22UUPHEHQWZH7ZJMLS4AH4PJ6AIA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:48.027197558 (75h47m23.590520562s) [noise=true , address=104.36.82.14:6232 , piece_id=A2ZLUUZKRTBJRJ267QSK36C3LFZV7FPAYDDAXGIWDTNEGNV6MRRA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:11:05.567692582 (75h47m41.131015574s) [noise=true , address=104.36.82.49:4982 , piece_id=UNLV5UDNWULEIPE4QSQHHZQPK3BQKFRB5T3H6GP7ZQ5QGSNK5V6A, seg_pos={0 0}, upload=60693]: Done uploading piece. err: ecclient: upload failed (node:1NxoYJXCXmTvQncbkXt6yH7mhEBBrzVdzhgL5J6W5TD4N3pkaG, address:104.36.82.49:4982): protocol: expected piece hash; context deadline exceeded; EOF Jun 21 20:11:05.567759198 (75h47m41.131082195s) [noise=true , address=104.36.82.54:4171 , piece_id=MUMABKARK4KKUKWW6XDXCQVT6O7SLFHGHE7COWC7QIYGARAA2D5A, seg_pos={0 0}, upload=60693]: Done uploading piece. err: ecclient: upload failed (node:125SMhDsEZruNu6LkmQn2M7R6WzvXTdk86DRsKf522u8BNiANq3, address:104.36.82.54:4171): protocol: expected piece hash; context deadline exceeded; EOF Jun 21 20:11:05.567869019 (75h47m41.131192033s) [noise=true , address=104.36.82.10:6087 , piece_id=5FQG7BPZC373PFB54HYXSG2SVKOBBBSGZCLC4XDJEYQFOLQY74MA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: ecclient: upload failed (node:1S9k5eejJcxkiG5pXjjBZm18uP8k5jCGfWyeW2QQxmuRU8ZzRA, address:104.36.82.10:6087): protocol: write tcp 159.223.143.39:38550->104.36.82.10:6087: use of closed network connection; write tcp 159.223.143.39:38550->104.36.82.10:6087: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:38550->104.36.82.10:6087: use of closed network connection Jun 21 20:11:05.567932150 (75h47m41.131255161s) [seg_pos={0 0}, upload=60693]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jun 21 20:11:05.593626175 (75h47m41.156949181s) [upload=60693]: Done uploading segments... Jun 21 20:11:05.593657274 (75h47m41.156980265s) [upload=60693]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 26.944340749s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [...................X...........X............................] duration: 24.944617218s FAIL (2025-06-21 20:34:25 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== pload=60693]: Starting upload Jun 21 20:10:05.602866542 (75h46m41.166189557s) [upload=60693]: Done starting upload Jun 21 20:10:05.603116545 (75h46m41.166439547s) [upload=60693]: Uploading segments... Jun 21 20:10:05.603495389 (75h46m41.166818397s) [upload=60693]: Got next segment. Inline: false Jun 21 20:10:05.603566566 (75h46m41.166889558s) [upload=60693]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 21 20:10:05.603573280 (75h46m41.166896286s) [upload=60693]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 21 20:10:05.639130162 (75h46m41.202453164s) [seg_pos={0 0}, upload=60693]: Begin upload segment... Jun 21 20:10:05.639231070 (75h46m41.202554057s) [seg_pos={0 0}, upload=60693]: Done begin upload segment. Jun 21 20:10:05.639292724 (75h46m41.202615728s) [noise=true , address=104.36.82.56:4253 , piece_id=5PMVGLZETMXCQ77K3I6PXACRIMBGR2CV5JGUSU5KUMYFNXVGA2GA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.639338066 (75h46m41.202661071s) [noise=true , address=104.36.82.53:4126 , piece_id=2MKRQJEPGDJKDM7VWY5M4M6NORILAIEYYATJ7HVZEOBOMXOI2LBA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.639416264 (75h46m41.202739267s) [noise=true , address=104.36.82.5:5869 , piece_id=HAEHXXZASV3DHT3QN7NTZOS6XZKHNUWBKCHWJJWSTJKXAKJKKBAA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.639873098 (75h46m41.203196115s) [noise=true , address=104.36.82.52:4111 , piece_id=HB7QAWN6CGNIIT4HD2GZ7JDQP5NXIJ5I2USBM66YOKTYENNOWG5A, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.639887296 (75h46m41.203210304s) [noise=true , address=104.36.82.10:6087 , piece_id=5FQG7BPZC373PFB54HYXSG2SVKOBBBSGZCLC4XDJEYQFOLQY74MA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.640266623 (75h46m41.203589616s) [noise=true , address=104.36.82.13:6196 , piece_id=BCHMQO5KKT5BLNWQGOHH3V43LGSE3BUQCI2SY4TTO5M5GQNOCTMQ, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.640352274 (75h46m41.203675286s) [noise=true , address=104.36.82.37:6663 , piece_id=X7LWYIIHJFU2T2PHYR3R5TNT22UUPHEHQWZH7ZJMLS4AH4PJ6AIA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.640659767 (75h46m41.203982779s) [noise=true , address=104.36.82.42:6807 , piece_id=ZBQZPXUDJ4VBCOSVKWAJ6SK6UD7SLPON7Y4SSCK2DQ5STFG4XCNQ, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.640961701 (75h46m41.20428471s) [noise=true , address=45.91.116.6:4982 , piece_id=VP45R6KXNH5U5SGUQUMAPIBY7KFDPBIR3S3VRIX27EAT6VLFBZ7Q, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.641159578 (75h46m41.204482589s) [noise=true , address=104.36.82.9:6039 , piece_id=I4AB6N27MYYUHWAZLJJUIBFBLOQFHDIM3YLYWPVRPNUV7VMYS2SA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.641588126 (75h46m41.204911136s) [noise=true , address=77.90.31.19:4853 , piece_id=VFEON5T2KSWM2KSLTVRZTXYL42JWENZYUEUPG3LMODGZWBMVGGPQ, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.641863297 (75h46m41.205186305s) [noise=true , address=77.90.31.12:4603 , piece_id=BMUIC6KNGEDL37436GLSXW5XOO6VYOPNOSFGQCYVHSENVI2PJ7KQ, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.642124427 (75h46m41.205447437s) [noise=true , address=104.36.82.49:4982 , piece_id=UNLV5UDNWULEIPE4QSQHHZQPK3BQKFRB5T3H6GP7ZQ5QGSNK5V6A, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.642285917 (75h46m41.205608925s) [noise=true , address=104.36.82.54:4171 , piece_id=MUMABKARK4KKUKWW6XDXCQVT6O7SLFHGHE7COWC7QIYGARAA2D5A, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.642656750 (75h46m41.205979758s) [noise=true , address=104.36.82.14:6232 , piece_id=A2ZLUUZKRTBJRJ267QSK36C3LFZV7FPAYDDAXGIWDTNEGNV6MRRA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.643565583 (75h46m41.206888597s) [noise=true , address=79.127.205.245:4231 , piece_id=EJK46XUUVI5NWOAQJEW4QIJG3KRXZHB4YOQUJRCS2MIOSWKGTNLQ, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.644620291 (75h46m41.207943303s) [noise=true , address=77.90.31.14:4670 , piece_id=7FNANMUKPM3N6YWRAOAIM4WJZJUTNF7ZBI6ZUBNEPGNGMFWOE24A, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.648842639 (75h46m41.212165631s) [noise=true , address=104.36.82.55:4224 , piece_id=WRVNGSJV5BX4GJERLXNPAS7J7BZ5WFTM7R5H257YRCG5SP2HIYMA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.825828239 (75h46m41.389151241s) [noise=true , address=104.36.82.53:4126 , piece_id=2MKRQJEPGDJKDM7VWY5M4M6NORILAIEYYATJ7HVZEOBOMXOI2LBA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: ecclient: upload failed (node:12dMWFrq2NMpnMzuPAphWerntC5kkrcnzRJZBA116uzJzyvk6fH, address:104.36.82.53:4126): protocol: storage node overloaded, request limit: 2000 Jun 21 20:10:05.834666043 (75h46m41.397989033s) [upload=60693]: Next returned nil segment Jun 21 20:10:05.834686295 (75h46m41.398009302s) [upload=60693]: Waiting for error group managing segments... Jun 21 20:10:05.877010488 (75h46m41.440333486s) [noise=true , address=45.91.116.6:4982 , piece_id=VP45R6KXNH5U5SGUQUMAPIBY7KFDPBIR3S3VRIX27EAT6VLFBZ7Q, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:05.886336964 (75h46m41.449659967s) [noise=true , address=104.36.82.9:6039 , piece_id=I4AB6N27MYYUHWAZLJJUIBFBLOQFHDIM3YLYWPVRPNUV7VMYS2SA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: ecclient: upload failed (node:12P11WVW9mVXw5ra2gZ3Rezon6wZi8GrN5CchQN3MiVU4ts3L6V, address:104.36.82.9:6039): protocol: storage node overloaded, request limit: 2000 Jun 21 20:10:08.064514542 (75h46m43.627837553s) [noise=true , address=77.90.31.14:4670 , piece_id=7FNANMUKPM3N6YWRAOAIM4WJZJUTNF7ZBI6ZUBNEPGNGMFWOE24A, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:08.099098391 (75h46m43.662421396s) [noise=true , address=79.127.205.245:4231 , piece_id=EJK46XUUVI5NWOAQJEW4QIJG3KRXZHB4YOQUJRCS2MIOSWKGTNLQ, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:08.489461749 (75h46m44.052784752s) [noise=true , address=104.36.82.42:6807 , piece_id=ZBQZPXUDJ4VBCOSVKWAJ6SK6UD7SLPON7Y4SSCK2DQ5STFG4XCNQ, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:08.650922093 (75h46m44.214245093s) [noise=true , address=77.90.31.19:4853 , piece_id=VFEON5T2KSWM2KSLTVRZTXYL42JWENZYUEUPG3LMODGZWBMVGGPQ, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:09.917746439 (75h46m45.481069441s) [noise=true , address=104.36.82.13:6196 , piece_id=BCHMQO5KKT5BLNWQGOHH3V43LGSE3BUQCI2SY4TTO5M5GQNOCTMQ, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:10.830722215 (75h46m46.394045216s) [noise=true , address=104.36.82.5:5869 , piece_id=HAEHXXZASV3DHT3QN7NTZOS6XZKHNUWBKCHWJJWSTJKXAKJKKBAA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:12.738783450 (75h46m48.302106452s) [noise=true , address=77.90.31.12:4603 , piece_id=BMUIC6KNGEDL37436GLSXW5XOO6VYOPNOSFGQCYVHSENVI2PJ7KQ, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:15.460927713 (75h46m51.024250723s) [noise=true , address=104.36.82.52:4111 , piece_id=HB7QAWN6CGNIIT4HD2GZ7JDQP5NXIJ5I2USBM66YOKTYENNOWG5A, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:18.393817968 (75h46m53.95714096s) [noise=true , address=104.36.82.55:4224 , piece_id=WRVNGSJV5BX4GJERLXNPAS7J7BZ5WFTM7R5H257YRCG5SP2HIYMA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:19.549174143 (75h46m55.112497151s) [noise=true , address=104.36.82.56:4253 , piece_id=5PMVGLZETMXCQ77K3I6PXACRIMBGR2CV5JGUSU5KUMYFNXVGA2GA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:22.511944593 (75h46m58.075267584s) [noise=true , address=104.36.82.37:6663 , piece_id=X7LWYIIHJFU2T2PHYR3R5TNT22UUPHEHQWZH7ZJMLS4AH4PJ6AIA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:48.027197558 (75h47m23.590520562s) [noise=true , address=104.36.82.14:6232 , piece_id=A2ZLUUZKRTBJRJ267QSK36C3LFZV7FPAYDDAXGIWDTNEGNV6MRRA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:11:05.567692582 (75h47m41.131015574s) [noise=true , address=104.36.82.49:4982 , piece_id=UNLV5UDNWULEIPE4QSQHHZQPK3BQKFRB5T3H6GP7ZQ5QGSNK5V6A, seg_pos={0 0}, upload=60693]: Done uploading piece. err: ecclient: upload failed (node:1NxoYJXCXmTvQncbkXt6yH7mhEBBrzVdzhgL5J6W5TD4N3pkaG, address:104.36.82.49:4982): protocol: expected piece hash; context deadline exceeded; EOF Jun 21 20:11:05.567759198 (75h47m41.131082195s) [noise=true , address=104.36.82.54:4171 , piece_id=MUMABKARK4KKUKWW6XDXCQVT6O7SLFHGHE7COWC7QIYGARAA2D5A, seg_pos={0 0}, upload=60693]: Done uploading piece. err: ecclient: upload failed (node:125SMhDsEZruNu6LkmQn2M7R6WzvXTdk86DRsKf522u8BNiANq3, address:104.36.82.54:4171): protocol: expected piece hash; context deadline exceeded; EOF Jun 21 20:11:05.567869019 (75h47m41.131192033s) [noise=true , address=104.36.82.10:6087 , piece_id=5FQG7BPZC373PFB54HYXSG2SVKOBBBSGZCLC4XDJEYQFOLQY74MA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: ecclient: upload failed (node:1S9k5eejJcxkiG5pXjjBZm18uP8k5jCGfWyeW2QQxmuRU8ZzRA, address:104.36.82.10:6087): protocol: write tcp 159.223.143.39:38550->104.36.82.10:6087: use of closed network connection; write tcp 159.223.143.39:38550->104.36.82.10:6087: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:38550->104.36.82.10:6087: use of closed network connection Jun 21 20:11:05.567932150 (75h47m41.131255161s) [seg_pos={0 0}, upload=60693]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jun 21 20:11:05.593626175 (75h47m41.156949181s) [upload=60693]: Done uploading segments... Jun 21 20:11:05.593657274 (75h47m41.156980265s) [upload=60693]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 24.944707978s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [....................X...........X...........................] duration: 15.845143985s FAIL (2025-06-21 20:33:48 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== pload=60693]: Starting upload Jun 21 20:10:05.602866542 (75h46m41.166189557s) [upload=60693]: Done starting upload Jun 21 20:10:05.603116545 (75h46m41.166439547s) [upload=60693]: Uploading segments... Jun 21 20:10:05.603495389 (75h46m41.166818397s) [upload=60693]: Got next segment. Inline: false Jun 21 20:10:05.603566566 (75h46m41.166889558s) [upload=60693]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 21 20:10:05.603573280 (75h46m41.166896286s) [upload=60693]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 21 20:10:05.639130162 (75h46m41.202453164s) [seg_pos={0 0}, upload=60693]: Begin upload segment... Jun 21 20:10:05.639231070 (75h46m41.202554057s) [seg_pos={0 0}, upload=60693]: Done begin upload segment. Jun 21 20:10:05.639292724 (75h46m41.202615728s) [noise=true , address=104.36.82.56:4253 , piece_id=5PMVGLZETMXCQ77K3I6PXACRIMBGR2CV5JGUSU5KUMYFNXVGA2GA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.639338066 (75h46m41.202661071s) [noise=true , address=104.36.82.53:4126 , piece_id=2MKRQJEPGDJKDM7VWY5M4M6NORILAIEYYATJ7HVZEOBOMXOI2LBA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.639416264 (75h46m41.202739267s) [noise=true , address=104.36.82.5:5869 , piece_id=HAEHXXZASV3DHT3QN7NTZOS6XZKHNUWBKCHWJJWSTJKXAKJKKBAA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.639873098 (75h46m41.203196115s) [noise=true , address=104.36.82.52:4111 , piece_id=HB7QAWN6CGNIIT4HD2GZ7JDQP5NXIJ5I2USBM66YOKTYENNOWG5A, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.639887296 (75h46m41.203210304s) [noise=true , address=104.36.82.10:6087 , piece_id=5FQG7BPZC373PFB54HYXSG2SVKOBBBSGZCLC4XDJEYQFOLQY74MA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.640266623 (75h46m41.203589616s) [noise=true , address=104.36.82.13:6196 , piece_id=BCHMQO5KKT5BLNWQGOHH3V43LGSE3BUQCI2SY4TTO5M5GQNOCTMQ, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.640352274 (75h46m41.203675286s) [noise=true , address=104.36.82.37:6663 , piece_id=X7LWYIIHJFU2T2PHYR3R5TNT22UUPHEHQWZH7ZJMLS4AH4PJ6AIA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.640659767 (75h46m41.203982779s) [noise=true , address=104.36.82.42:6807 , piece_id=ZBQZPXUDJ4VBCOSVKWAJ6SK6UD7SLPON7Y4SSCK2DQ5STFG4XCNQ, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.640961701 (75h46m41.20428471s) [noise=true , address=45.91.116.6:4982 , piece_id=VP45R6KXNH5U5SGUQUMAPIBY7KFDPBIR3S3VRIX27EAT6VLFBZ7Q, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.641159578 (75h46m41.204482589s) [noise=true , address=104.36.82.9:6039 , piece_id=I4AB6N27MYYUHWAZLJJUIBFBLOQFHDIM3YLYWPVRPNUV7VMYS2SA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.641588126 (75h46m41.204911136s) [noise=true , address=77.90.31.19:4853 , piece_id=VFEON5T2KSWM2KSLTVRZTXYL42JWENZYUEUPG3LMODGZWBMVGGPQ, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.641863297 (75h46m41.205186305s) [noise=true , address=77.90.31.12:4603 , piece_id=BMUIC6KNGEDL37436GLSXW5XOO6VYOPNOSFGQCYVHSENVI2PJ7KQ, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.642124427 (75h46m41.205447437s) [noise=true , address=104.36.82.49:4982 , piece_id=UNLV5UDNWULEIPE4QSQHHZQPK3BQKFRB5T3H6GP7ZQ5QGSNK5V6A, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.642285917 (75h46m41.205608925s) [noise=true , address=104.36.82.54:4171 , piece_id=MUMABKARK4KKUKWW6XDXCQVT6O7SLFHGHE7COWC7QIYGARAA2D5A, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.642656750 (75h46m41.205979758s) [noise=true , address=104.36.82.14:6232 , piece_id=A2ZLUUZKRTBJRJ267QSK36C3LFZV7FPAYDDAXGIWDTNEGNV6MRRA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.643565583 (75h46m41.206888597s) [noise=true , address=79.127.205.245:4231 , piece_id=EJK46XUUVI5NWOAQJEW4QIJG3KRXZHB4YOQUJRCS2MIOSWKGTNLQ, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.644620291 (75h46m41.207943303s) [noise=true , address=77.90.31.14:4670 , piece_id=7FNANMUKPM3N6YWRAOAIM4WJZJUTNF7ZBI6ZUBNEPGNGMFWOE24A, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.648842639 (75h46m41.212165631s) [noise=true , address=104.36.82.55:4224 , piece_id=WRVNGSJV5BX4GJERLXNPAS7J7BZ5WFTM7R5H257YRCG5SP2HIYMA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.825828239 (75h46m41.389151241s) [noise=true , address=104.36.82.53:4126 , piece_id=2MKRQJEPGDJKDM7VWY5M4M6NORILAIEYYATJ7HVZEOBOMXOI2LBA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: ecclient: upload failed (node:12dMWFrq2NMpnMzuPAphWerntC5kkrcnzRJZBA116uzJzyvk6fH, address:104.36.82.53:4126): protocol: storage node overloaded, request limit: 2000 Jun 21 20:10:05.834666043 (75h46m41.397989033s) [upload=60693]: Next returned nil segment Jun 21 20:10:05.834686295 (75h46m41.398009302s) [upload=60693]: Waiting for error group managing segments... Jun 21 20:10:05.877010488 (75h46m41.440333486s) [noise=true , address=45.91.116.6:4982 , piece_id=VP45R6KXNH5U5SGUQUMAPIBY7KFDPBIR3S3VRIX27EAT6VLFBZ7Q, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:05.886336964 (75h46m41.449659967s) [noise=true , address=104.36.82.9:6039 , piece_id=I4AB6N27MYYUHWAZLJJUIBFBLOQFHDIM3YLYWPVRPNUV7VMYS2SA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: ecclient: upload failed (node:12P11WVW9mVXw5ra2gZ3Rezon6wZi8GrN5CchQN3MiVU4ts3L6V, address:104.36.82.9:6039): protocol: storage node overloaded, request limit: 2000 Jun 21 20:10:08.064514542 (75h46m43.627837553s) [noise=true , address=77.90.31.14:4670 , piece_id=7FNANMUKPM3N6YWRAOAIM4WJZJUTNF7ZBI6ZUBNEPGNGMFWOE24A, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:08.099098391 (75h46m43.662421396s) [noise=true , address=79.127.205.245:4231 , piece_id=EJK46XUUVI5NWOAQJEW4QIJG3KRXZHB4YOQUJRCS2MIOSWKGTNLQ, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:08.489461749 (75h46m44.052784752s) [noise=true , address=104.36.82.42:6807 , piece_id=ZBQZPXUDJ4VBCOSVKWAJ6SK6UD7SLPON7Y4SSCK2DQ5STFG4XCNQ, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:08.650922093 (75h46m44.214245093s) [noise=true , address=77.90.31.19:4853 , piece_id=VFEON5T2KSWM2KSLTVRZTXYL42JWENZYUEUPG3LMODGZWBMVGGPQ, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:09.917746439 (75h46m45.481069441s) [noise=true , address=104.36.82.13:6196 , piece_id=BCHMQO5KKT5BLNWQGOHH3V43LGSE3BUQCI2SY4TTO5M5GQNOCTMQ, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:10.830722215 (75h46m46.394045216s) [noise=true , address=104.36.82.5:5869 , piece_id=HAEHXXZASV3DHT3QN7NTZOS6XZKHNUWBKCHWJJWSTJKXAKJKKBAA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:12.738783450 (75h46m48.302106452s) [noise=true , address=77.90.31.12:4603 , piece_id=BMUIC6KNGEDL37436GLSXW5XOO6VYOPNOSFGQCYVHSENVI2PJ7KQ, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:15.460927713 (75h46m51.024250723s) [noise=true , address=104.36.82.52:4111 , piece_id=HB7QAWN6CGNIIT4HD2GZ7JDQP5NXIJ5I2USBM66YOKTYENNOWG5A, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:18.393817968 (75h46m53.95714096s) [noise=true , address=104.36.82.55:4224 , piece_id=WRVNGSJV5BX4GJERLXNPAS7J7BZ5WFTM7R5H257YRCG5SP2HIYMA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:19.549174143 (75h46m55.112497151s) [noise=true , address=104.36.82.56:4253 , piece_id=5PMVGLZETMXCQ77K3I6PXACRIMBGR2CV5JGUSU5KUMYFNXVGA2GA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:22.511944593 (75h46m58.075267584s) [noise=true , address=104.36.82.37:6663 , piece_id=X7LWYIIHJFU2T2PHYR3R5TNT22UUPHEHQWZH7ZJMLS4AH4PJ6AIA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:48.027197558 (75h47m23.590520562s) [noise=true , address=104.36.82.14:6232 , piece_id=A2ZLUUZKRTBJRJ267QSK36C3LFZV7FPAYDDAXGIWDTNEGNV6MRRA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:11:05.567692582 (75h47m41.131015574s) [noise=true , address=104.36.82.49:4982 , piece_id=UNLV5UDNWULEIPE4QSQHHZQPK3BQKFRB5T3H6GP7ZQ5QGSNK5V6A, seg_pos={0 0}, upload=60693]: Done uploading piece. err: ecclient: upload failed (node:1NxoYJXCXmTvQncbkXt6yH7mhEBBrzVdzhgL5J6W5TD4N3pkaG, address:104.36.82.49:4982): protocol: expected piece hash; context deadline exceeded; EOF Jun 21 20:11:05.567759198 (75h47m41.131082195s) [noise=true , address=104.36.82.54:4171 , piece_id=MUMABKARK4KKUKWW6XDXCQVT6O7SLFHGHE7COWC7QIYGARAA2D5A, seg_pos={0 0}, upload=60693]: Done uploading piece. err: ecclient: upload failed (node:125SMhDsEZruNu6LkmQn2M7R6WzvXTdk86DRsKf522u8BNiANq3, address:104.36.82.54:4171): protocol: expected piece hash; context deadline exceeded; EOF Jun 21 20:11:05.567869019 (75h47m41.131192033s) [noise=true , address=104.36.82.10:6087 , piece_id=5FQG7BPZC373PFB54HYXSG2SVKOBBBSGZCLC4XDJEYQFOLQY74MA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: ecclient: upload failed (node:1S9k5eejJcxkiG5pXjjBZm18uP8k5jCGfWyeW2QQxmuRU8ZzRA, address:104.36.82.10:6087): protocol: write tcp 159.223.143.39:38550->104.36.82.10:6087: use of closed network connection; write tcp 159.223.143.39:38550->104.36.82.10:6087: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:38550->104.36.82.10:6087: use of closed network connection Jun 21 20:11:05.567932150 (75h47m41.131255161s) [seg_pos={0 0}, upload=60693]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jun 21 20:11:05.593626175 (75h47m41.156949181s) [upload=60693]: Done uploading segments... Jun 21 20:11:05.593657274 (75h47m41.156980265s) [upload=60693]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 15.845172011s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [.....................X...........X..........................] duration: 24.291828123s FAIL (2025-06-21 20:33:14 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== pload=60693]: Starting upload Jun 21 20:10:05.602866542 (75h46m41.166189557s) [upload=60693]: Done starting upload Jun 21 20:10:05.603116545 (75h46m41.166439547s) [upload=60693]: Uploading segments... Jun 21 20:10:05.603495389 (75h46m41.166818397s) [upload=60693]: Got next segment. Inline: false Jun 21 20:10:05.603566566 (75h46m41.166889558s) [upload=60693]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 21 20:10:05.603573280 (75h46m41.166896286s) [upload=60693]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 21 20:10:05.639130162 (75h46m41.202453164s) [seg_pos={0 0}, upload=60693]: Begin upload segment... Jun 21 20:10:05.639231070 (75h46m41.202554057s) [seg_pos={0 0}, upload=60693]: Done begin upload segment. Jun 21 20:10:05.639292724 (75h46m41.202615728s) [noise=true , address=104.36.82.56:4253 , piece_id=5PMVGLZETMXCQ77K3I6PXACRIMBGR2CV5JGUSU5KUMYFNXVGA2GA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.639338066 (75h46m41.202661071s) [noise=true , address=104.36.82.53:4126 , piece_id=2MKRQJEPGDJKDM7VWY5M4M6NORILAIEYYATJ7HVZEOBOMXOI2LBA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.639416264 (75h46m41.202739267s) [noise=true , address=104.36.82.5:5869 , piece_id=HAEHXXZASV3DHT3QN7NTZOS6XZKHNUWBKCHWJJWSTJKXAKJKKBAA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.639873098 (75h46m41.203196115s) [noise=true , address=104.36.82.52:4111 , piece_id=HB7QAWN6CGNIIT4HD2GZ7JDQP5NXIJ5I2USBM66YOKTYENNOWG5A, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.639887296 (75h46m41.203210304s) [noise=true , address=104.36.82.10:6087 , piece_id=5FQG7BPZC373PFB54HYXSG2SVKOBBBSGZCLC4XDJEYQFOLQY74MA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.640266623 (75h46m41.203589616s) [noise=true , address=104.36.82.13:6196 , piece_id=BCHMQO5KKT5BLNWQGOHH3V43LGSE3BUQCI2SY4TTO5M5GQNOCTMQ, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.640352274 (75h46m41.203675286s) [noise=true , address=104.36.82.37:6663 , piece_id=X7LWYIIHJFU2T2PHYR3R5TNT22UUPHEHQWZH7ZJMLS4AH4PJ6AIA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.640659767 (75h46m41.203982779s) [noise=true , address=104.36.82.42:6807 , piece_id=ZBQZPXUDJ4VBCOSVKWAJ6SK6UD7SLPON7Y4SSCK2DQ5STFG4XCNQ, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.640961701 (75h46m41.20428471s) [noise=true , address=45.91.116.6:4982 , piece_id=VP45R6KXNH5U5SGUQUMAPIBY7KFDPBIR3S3VRIX27EAT6VLFBZ7Q, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.641159578 (75h46m41.204482589s) [noise=true , address=104.36.82.9:6039 , piece_id=I4AB6N27MYYUHWAZLJJUIBFBLOQFHDIM3YLYWPVRPNUV7VMYS2SA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.641588126 (75h46m41.204911136s) [noise=true , address=77.90.31.19:4853 , piece_id=VFEON5T2KSWM2KSLTVRZTXYL42JWENZYUEUPG3LMODGZWBMVGGPQ, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.641863297 (75h46m41.205186305s) [noise=true , address=77.90.31.12:4603 , piece_id=BMUIC6KNGEDL37436GLSXW5XOO6VYOPNOSFGQCYVHSENVI2PJ7KQ, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.642124427 (75h46m41.205447437s) [noise=true , address=104.36.82.49:4982 , piece_id=UNLV5UDNWULEIPE4QSQHHZQPK3BQKFRB5T3H6GP7ZQ5QGSNK5V6A, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.642285917 (75h46m41.205608925s) [noise=true , address=104.36.82.54:4171 , piece_id=MUMABKARK4KKUKWW6XDXCQVT6O7SLFHGHE7COWC7QIYGARAA2D5A, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.642656750 (75h46m41.205979758s) [noise=true , address=104.36.82.14:6232 , piece_id=A2ZLUUZKRTBJRJ267QSK36C3LFZV7FPAYDDAXGIWDTNEGNV6MRRA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.643565583 (75h46m41.206888597s) [noise=true , address=79.127.205.245:4231 , piece_id=EJK46XUUVI5NWOAQJEW4QIJG3KRXZHB4YOQUJRCS2MIOSWKGTNLQ, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.644620291 (75h46m41.207943303s) [noise=true , address=77.90.31.14:4670 , piece_id=7FNANMUKPM3N6YWRAOAIM4WJZJUTNF7ZBI6ZUBNEPGNGMFWOE24A, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.648842639 (75h46m41.212165631s) [noise=true , address=104.36.82.55:4224 , piece_id=WRVNGSJV5BX4GJERLXNPAS7J7BZ5WFTM7R5H257YRCG5SP2HIYMA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.825828239 (75h46m41.389151241s) [noise=true , address=104.36.82.53:4126 , piece_id=2MKRQJEPGDJKDM7VWY5M4M6NORILAIEYYATJ7HVZEOBOMXOI2LBA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: ecclient: upload failed (node:12dMWFrq2NMpnMzuPAphWerntC5kkrcnzRJZBA116uzJzyvk6fH, address:104.36.82.53:4126): protocol: storage node overloaded, request limit: 2000 Jun 21 20:10:05.834666043 (75h46m41.397989033s) [upload=60693]: Next returned nil segment Jun 21 20:10:05.834686295 (75h46m41.398009302s) [upload=60693]: Waiting for error group managing segments... Jun 21 20:10:05.877010488 (75h46m41.440333486s) [noise=true , address=45.91.116.6:4982 , piece_id=VP45R6KXNH5U5SGUQUMAPIBY7KFDPBIR3S3VRIX27EAT6VLFBZ7Q, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:05.886336964 (75h46m41.449659967s) [noise=true , address=104.36.82.9:6039 , piece_id=I4AB6N27MYYUHWAZLJJUIBFBLOQFHDIM3YLYWPVRPNUV7VMYS2SA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: ecclient: upload failed (node:12P11WVW9mVXw5ra2gZ3Rezon6wZi8GrN5CchQN3MiVU4ts3L6V, address:104.36.82.9:6039): protocol: storage node overloaded, request limit: 2000 Jun 21 20:10:08.064514542 (75h46m43.627837553s) [noise=true , address=77.90.31.14:4670 , piece_id=7FNANMUKPM3N6YWRAOAIM4WJZJUTNF7ZBI6ZUBNEPGNGMFWOE24A, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:08.099098391 (75h46m43.662421396s) [noise=true , address=79.127.205.245:4231 , piece_id=EJK46XUUVI5NWOAQJEW4QIJG3KRXZHB4YOQUJRCS2MIOSWKGTNLQ, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:08.489461749 (75h46m44.052784752s) [noise=true , address=104.36.82.42:6807 , piece_id=ZBQZPXUDJ4VBCOSVKWAJ6SK6UD7SLPON7Y4SSCK2DQ5STFG4XCNQ, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:08.650922093 (75h46m44.214245093s) [noise=true , address=77.90.31.19:4853 , piece_id=VFEON5T2KSWM2KSLTVRZTXYL42JWENZYUEUPG3LMODGZWBMVGGPQ, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:09.917746439 (75h46m45.481069441s) [noise=true , address=104.36.82.13:6196 , piece_id=BCHMQO5KKT5BLNWQGOHH3V43LGSE3BUQCI2SY4TTO5M5GQNOCTMQ, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:10.830722215 (75h46m46.394045216s) [noise=true , address=104.36.82.5:5869 , piece_id=HAEHXXZASV3DHT3QN7NTZOS6XZKHNUWBKCHWJJWSTJKXAKJKKBAA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:12.738783450 (75h46m48.302106452s) [noise=true , address=77.90.31.12:4603 , piece_id=BMUIC6KNGEDL37436GLSXW5XOO6VYOPNOSFGQCYVHSENVI2PJ7KQ, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:15.460927713 (75h46m51.024250723s) [noise=true , address=104.36.82.52:4111 , piece_id=HB7QAWN6CGNIIT4HD2GZ7JDQP5NXIJ5I2USBM66YOKTYENNOWG5A, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:18.393817968 (75h46m53.95714096s) [noise=true , address=104.36.82.55:4224 , piece_id=WRVNGSJV5BX4GJERLXNPAS7J7BZ5WFTM7R5H257YRCG5SP2HIYMA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:19.549174143 (75h46m55.112497151s) [noise=true , address=104.36.82.56:4253 , piece_id=5PMVGLZETMXCQ77K3I6PXACRIMBGR2CV5JGUSU5KUMYFNXVGA2GA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:22.511944593 (75h46m58.075267584s) [noise=true , address=104.36.82.37:6663 , piece_id=X7LWYIIHJFU2T2PHYR3R5TNT22UUPHEHQWZH7ZJMLS4AH4PJ6AIA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:48.027197558 (75h47m23.590520562s) [noise=true , address=104.36.82.14:6232 , piece_id=A2ZLUUZKRTBJRJ267QSK36C3LFZV7FPAYDDAXGIWDTNEGNV6MRRA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:11:05.567692582 (75h47m41.131015574s) [noise=true , address=104.36.82.49:4982 , piece_id=UNLV5UDNWULEIPE4QSQHHZQPK3BQKFRB5T3H6GP7ZQ5QGSNK5V6A, seg_pos={0 0}, upload=60693]: Done uploading piece. err: ecclient: upload failed (node:1NxoYJXCXmTvQncbkXt6yH7mhEBBrzVdzhgL5J6W5TD4N3pkaG, address:104.36.82.49:4982): protocol: expected piece hash; context deadline exceeded; EOF Jun 21 20:11:05.567759198 (75h47m41.131082195s) [noise=true , address=104.36.82.54:4171 , piece_id=MUMABKARK4KKUKWW6XDXCQVT6O7SLFHGHE7COWC7QIYGARAA2D5A, seg_pos={0 0}, upload=60693]: Done uploading piece. err: ecclient: upload failed (node:125SMhDsEZruNu6LkmQn2M7R6WzvXTdk86DRsKf522u8BNiANq3, address:104.36.82.54:4171): protocol: expected piece hash; context deadline exceeded; EOF Jun 21 20:11:05.567869019 (75h47m41.131192033s) [noise=true , address=104.36.82.10:6087 , piece_id=5FQG7BPZC373PFB54HYXSG2SVKOBBBSGZCLC4XDJEYQFOLQY74MA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: ecclient: upload failed (node:1S9k5eejJcxkiG5pXjjBZm18uP8k5jCGfWyeW2QQxmuRU8ZzRA, address:104.36.82.10:6087): protocol: write tcp 159.223.143.39:38550->104.36.82.10:6087: use of closed network connection; write tcp 159.223.143.39:38550->104.36.82.10:6087: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:38550->104.36.82.10:6087: use of closed network connection Jun 21 20:11:05.567932150 (75h47m41.131255161s) [seg_pos={0 0}, upload=60693]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jun 21 20:11:05.593626175 (75h47m41.156949181s) [upload=60693]: Done uploading segments... Jun 21 20:11:05.593657274 (75h47m41.156980265s) [upload=60693]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 24.291963321s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [......................X...........X.........................] duration: 32.785095055s FAIL (2025-06-21 20:32:27 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== pload=60693]: Starting upload Jun 21 20:10:05.602866542 (75h46m41.166189557s) [upload=60693]: Done starting upload Jun 21 20:10:05.603116545 (75h46m41.166439547s) [upload=60693]: Uploading segments... Jun 21 20:10:05.603495389 (75h46m41.166818397s) [upload=60693]: Got next segment. Inline: false Jun 21 20:10:05.603566566 (75h46m41.166889558s) [upload=60693]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 21 20:10:05.603573280 (75h46m41.166896286s) [upload=60693]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 21 20:10:05.639130162 (75h46m41.202453164s) [seg_pos={0 0}, upload=60693]: Begin upload segment... Jun 21 20:10:05.639231070 (75h46m41.202554057s) [seg_pos={0 0}, upload=60693]: Done begin upload segment. Jun 21 20:10:05.639292724 (75h46m41.202615728s) [noise=true , address=104.36.82.56:4253 , piece_id=5PMVGLZETMXCQ77K3I6PXACRIMBGR2CV5JGUSU5KUMYFNXVGA2GA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.639338066 (75h46m41.202661071s) [noise=true , address=104.36.82.53:4126 , piece_id=2MKRQJEPGDJKDM7VWY5M4M6NORILAIEYYATJ7HVZEOBOMXOI2LBA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.639416264 (75h46m41.202739267s) [noise=true , address=104.36.82.5:5869 , piece_id=HAEHXXZASV3DHT3QN7NTZOS6XZKHNUWBKCHWJJWSTJKXAKJKKBAA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.639873098 (75h46m41.203196115s) [noise=true , address=104.36.82.52:4111 , piece_id=HB7QAWN6CGNIIT4HD2GZ7JDQP5NXIJ5I2USBM66YOKTYENNOWG5A, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.639887296 (75h46m41.203210304s) [noise=true , address=104.36.82.10:6087 , piece_id=5FQG7BPZC373PFB54HYXSG2SVKOBBBSGZCLC4XDJEYQFOLQY74MA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.640266623 (75h46m41.203589616s) [noise=true , address=104.36.82.13:6196 , piece_id=BCHMQO5KKT5BLNWQGOHH3V43LGSE3BUQCI2SY4TTO5M5GQNOCTMQ, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.640352274 (75h46m41.203675286s) [noise=true , address=104.36.82.37:6663 , piece_id=X7LWYIIHJFU2T2PHYR3R5TNT22UUPHEHQWZH7ZJMLS4AH4PJ6AIA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.640659767 (75h46m41.203982779s) [noise=true , address=104.36.82.42:6807 , piece_id=ZBQZPXUDJ4VBCOSVKWAJ6SK6UD7SLPON7Y4SSCK2DQ5STFG4XCNQ, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.640961701 (75h46m41.20428471s) [noise=true , address=45.91.116.6:4982 , piece_id=VP45R6KXNH5U5SGUQUMAPIBY7KFDPBIR3S3VRIX27EAT6VLFBZ7Q, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.641159578 (75h46m41.204482589s) [noise=true , address=104.36.82.9:6039 , piece_id=I4AB6N27MYYUHWAZLJJUIBFBLOQFHDIM3YLYWPVRPNUV7VMYS2SA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.641588126 (75h46m41.204911136s) [noise=true , address=77.90.31.19:4853 , piece_id=VFEON5T2KSWM2KSLTVRZTXYL42JWENZYUEUPG3LMODGZWBMVGGPQ, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.641863297 (75h46m41.205186305s) [noise=true , address=77.90.31.12:4603 , piece_id=BMUIC6KNGEDL37436GLSXW5XOO6VYOPNOSFGQCYVHSENVI2PJ7KQ, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.642124427 (75h46m41.205447437s) [noise=true , address=104.36.82.49:4982 , piece_id=UNLV5UDNWULEIPE4QSQHHZQPK3BQKFRB5T3H6GP7ZQ5QGSNK5V6A, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.642285917 (75h46m41.205608925s) [noise=true , address=104.36.82.54:4171 , piece_id=MUMABKARK4KKUKWW6XDXCQVT6O7SLFHGHE7COWC7QIYGARAA2D5A, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.642656750 (75h46m41.205979758s) [noise=true , address=104.36.82.14:6232 , piece_id=A2ZLUUZKRTBJRJ267QSK36C3LFZV7FPAYDDAXGIWDTNEGNV6MRRA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.643565583 (75h46m41.206888597s) [noise=true , address=79.127.205.245:4231 , piece_id=EJK46XUUVI5NWOAQJEW4QIJG3KRXZHB4YOQUJRCS2MIOSWKGTNLQ, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.644620291 (75h46m41.207943303s) [noise=true , address=77.90.31.14:4670 , piece_id=7FNANMUKPM3N6YWRAOAIM4WJZJUTNF7ZBI6ZUBNEPGNGMFWOE24A, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.648842639 (75h46m41.212165631s) [noise=true , address=104.36.82.55:4224 , piece_id=WRVNGSJV5BX4GJERLXNPAS7J7BZ5WFTM7R5H257YRCG5SP2HIYMA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.825828239 (75h46m41.389151241s) [noise=true , address=104.36.82.53:4126 , piece_id=2MKRQJEPGDJKDM7VWY5M4M6NORILAIEYYATJ7HVZEOBOMXOI2LBA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: ecclient: upload failed (node:12dMWFrq2NMpnMzuPAphWerntC5kkrcnzRJZBA116uzJzyvk6fH, address:104.36.82.53:4126): protocol: storage node overloaded, request limit: 2000 Jun 21 20:10:05.834666043 (75h46m41.397989033s) [upload=60693]: Next returned nil segment Jun 21 20:10:05.834686295 (75h46m41.398009302s) [upload=60693]: Waiting for error group managing segments... Jun 21 20:10:05.877010488 (75h46m41.440333486s) [noise=true , address=45.91.116.6:4982 , piece_id=VP45R6KXNH5U5SGUQUMAPIBY7KFDPBIR3S3VRIX27EAT6VLFBZ7Q, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:05.886336964 (75h46m41.449659967s) [noise=true , address=104.36.82.9:6039 , piece_id=I4AB6N27MYYUHWAZLJJUIBFBLOQFHDIM3YLYWPVRPNUV7VMYS2SA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: ecclient: upload failed (node:12P11WVW9mVXw5ra2gZ3Rezon6wZi8GrN5CchQN3MiVU4ts3L6V, address:104.36.82.9:6039): protocol: storage node overloaded, request limit: 2000 Jun 21 20:10:08.064514542 (75h46m43.627837553s) [noise=true , address=77.90.31.14:4670 , piece_id=7FNANMUKPM3N6YWRAOAIM4WJZJUTNF7ZBI6ZUBNEPGNGMFWOE24A, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:08.099098391 (75h46m43.662421396s) [noise=true , address=79.127.205.245:4231 , piece_id=EJK46XUUVI5NWOAQJEW4QIJG3KRXZHB4YOQUJRCS2MIOSWKGTNLQ, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:08.489461749 (75h46m44.052784752s) [noise=true , address=104.36.82.42:6807 , piece_id=ZBQZPXUDJ4VBCOSVKWAJ6SK6UD7SLPON7Y4SSCK2DQ5STFG4XCNQ, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:08.650922093 (75h46m44.214245093s) [noise=true , address=77.90.31.19:4853 , piece_id=VFEON5T2KSWM2KSLTVRZTXYL42JWENZYUEUPG3LMODGZWBMVGGPQ, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:09.917746439 (75h46m45.481069441s) [noise=true , address=104.36.82.13:6196 , piece_id=BCHMQO5KKT5BLNWQGOHH3V43LGSE3BUQCI2SY4TTO5M5GQNOCTMQ, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:10.830722215 (75h46m46.394045216s) [noise=true , address=104.36.82.5:5869 , piece_id=HAEHXXZASV3DHT3QN7NTZOS6XZKHNUWBKCHWJJWSTJKXAKJKKBAA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:12.738783450 (75h46m48.302106452s) [noise=true , address=77.90.31.12:4603 , piece_id=BMUIC6KNGEDL37436GLSXW5XOO6VYOPNOSFGQCYVHSENVI2PJ7KQ, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:15.460927713 (75h46m51.024250723s) [noise=true , address=104.36.82.52:4111 , piece_id=HB7QAWN6CGNIIT4HD2GZ7JDQP5NXIJ5I2USBM66YOKTYENNOWG5A, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:18.393817968 (75h46m53.95714096s) [noise=true , address=104.36.82.55:4224 , piece_id=WRVNGSJV5BX4GJERLXNPAS7J7BZ5WFTM7R5H257YRCG5SP2HIYMA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:19.549174143 (75h46m55.112497151s) [noise=true , address=104.36.82.56:4253 , piece_id=5PMVGLZETMXCQ77K3I6PXACRIMBGR2CV5JGUSU5KUMYFNXVGA2GA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:22.511944593 (75h46m58.075267584s) [noise=true , address=104.36.82.37:6663 , piece_id=X7LWYIIHJFU2T2PHYR3R5TNT22UUPHEHQWZH7ZJMLS4AH4PJ6AIA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:48.027197558 (75h47m23.590520562s) [noise=true , address=104.36.82.14:6232 , piece_id=A2ZLUUZKRTBJRJ267QSK36C3LFZV7FPAYDDAXGIWDTNEGNV6MRRA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:11:05.567692582 (75h47m41.131015574s) [noise=true , address=104.36.82.49:4982 , piece_id=UNLV5UDNWULEIPE4QSQHHZQPK3BQKFRB5T3H6GP7ZQ5QGSNK5V6A, seg_pos={0 0}, upload=60693]: Done uploading piece. err: ecclient: upload failed (node:1NxoYJXCXmTvQncbkXt6yH7mhEBBrzVdzhgL5J6W5TD4N3pkaG, address:104.36.82.49:4982): protocol: expected piece hash; context deadline exceeded; EOF Jun 21 20:11:05.567759198 (75h47m41.131082195s) [noise=true , address=104.36.82.54:4171 , piece_id=MUMABKARK4KKUKWW6XDXCQVT6O7SLFHGHE7COWC7QIYGARAA2D5A, seg_pos={0 0}, upload=60693]: Done uploading piece. err: ecclient: upload failed (node:125SMhDsEZruNu6LkmQn2M7R6WzvXTdk86DRsKf522u8BNiANq3, address:104.36.82.54:4171): protocol: expected piece hash; context deadline exceeded; EOF Jun 21 20:11:05.567869019 (75h47m41.131192033s) [noise=true , address=104.36.82.10:6087 , piece_id=5FQG7BPZC373PFB54HYXSG2SVKOBBBSGZCLC4XDJEYQFOLQY74MA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: ecclient: upload failed (node:1S9k5eejJcxkiG5pXjjBZm18uP8k5jCGfWyeW2QQxmuRU8ZzRA, address:104.36.82.10:6087): protocol: write tcp 159.223.143.39:38550->104.36.82.10:6087: use of closed network connection; write tcp 159.223.143.39:38550->104.36.82.10:6087: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:38550->104.36.82.10:6087: use of closed network connection Jun 21 20:11:05.567932150 (75h47m41.131255161s) [seg_pos={0 0}, upload=60693]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jun 21 20:11:05.593626175 (75h47m41.156949181s) [upload=60693]: Done uploading segments... Jun 21 20:11:05.593657274 (75h47m41.156980265s) [upload=60693]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 32.785126815s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [.......................X...........X........................] duration: 28.600434712s FAIL (2025-06-21 20:31:34 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== pload=60693]: Starting upload Jun 21 20:10:05.602866542 (75h46m41.166189557s) [upload=60693]: Done starting upload Jun 21 20:10:05.603116545 (75h46m41.166439547s) [upload=60693]: Uploading segments... Jun 21 20:10:05.603495389 (75h46m41.166818397s) [upload=60693]: Got next segment. Inline: false Jun 21 20:10:05.603566566 (75h46m41.166889558s) [upload=60693]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 21 20:10:05.603573280 (75h46m41.166896286s) [upload=60693]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 21 20:10:05.639130162 (75h46m41.202453164s) [seg_pos={0 0}, upload=60693]: Begin upload segment... Jun 21 20:10:05.639231070 (75h46m41.202554057s) [seg_pos={0 0}, upload=60693]: Done begin upload segment. Jun 21 20:10:05.639292724 (75h46m41.202615728s) [noise=true , address=104.36.82.56:4253 , piece_id=5PMVGLZETMXCQ77K3I6PXACRIMBGR2CV5JGUSU5KUMYFNXVGA2GA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.639338066 (75h46m41.202661071s) [noise=true , address=104.36.82.53:4126 , piece_id=2MKRQJEPGDJKDM7VWY5M4M6NORILAIEYYATJ7HVZEOBOMXOI2LBA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.639416264 (75h46m41.202739267s) [noise=true , address=104.36.82.5:5869 , piece_id=HAEHXXZASV3DHT3QN7NTZOS6XZKHNUWBKCHWJJWSTJKXAKJKKBAA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.639873098 (75h46m41.203196115s) [noise=true , address=104.36.82.52:4111 , piece_id=HB7QAWN6CGNIIT4HD2GZ7JDQP5NXIJ5I2USBM66YOKTYENNOWG5A, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.639887296 (75h46m41.203210304s) [noise=true , address=104.36.82.10:6087 , piece_id=5FQG7BPZC373PFB54HYXSG2SVKOBBBSGZCLC4XDJEYQFOLQY74MA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.640266623 (75h46m41.203589616s) [noise=true , address=104.36.82.13:6196 , piece_id=BCHMQO5KKT5BLNWQGOHH3V43LGSE3BUQCI2SY4TTO5M5GQNOCTMQ, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.640352274 (75h46m41.203675286s) [noise=true , address=104.36.82.37:6663 , piece_id=X7LWYIIHJFU2T2PHYR3R5TNT22UUPHEHQWZH7ZJMLS4AH4PJ6AIA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.640659767 (75h46m41.203982779s) [noise=true , address=104.36.82.42:6807 , piece_id=ZBQZPXUDJ4VBCOSVKWAJ6SK6UD7SLPON7Y4SSCK2DQ5STFG4XCNQ, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.640961701 (75h46m41.20428471s) [noise=true , address=45.91.116.6:4982 , piece_id=VP45R6KXNH5U5SGUQUMAPIBY7KFDPBIR3S3VRIX27EAT6VLFBZ7Q, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.641159578 (75h46m41.204482589s) [noise=true , address=104.36.82.9:6039 , piece_id=I4AB6N27MYYUHWAZLJJUIBFBLOQFHDIM3YLYWPVRPNUV7VMYS2SA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.641588126 (75h46m41.204911136s) [noise=true , address=77.90.31.19:4853 , piece_id=VFEON5T2KSWM2KSLTVRZTXYL42JWENZYUEUPG3LMODGZWBMVGGPQ, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.641863297 (75h46m41.205186305s) [noise=true , address=77.90.31.12:4603 , piece_id=BMUIC6KNGEDL37436GLSXW5XOO6VYOPNOSFGQCYVHSENVI2PJ7KQ, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.642124427 (75h46m41.205447437s) [noise=true , address=104.36.82.49:4982 , piece_id=UNLV5UDNWULEIPE4QSQHHZQPK3BQKFRB5T3H6GP7ZQ5QGSNK5V6A, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.642285917 (75h46m41.205608925s) [noise=true , address=104.36.82.54:4171 , piece_id=MUMABKARK4KKUKWW6XDXCQVT6O7SLFHGHE7COWC7QIYGARAA2D5A, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.642656750 (75h46m41.205979758s) [noise=true , address=104.36.82.14:6232 , piece_id=A2ZLUUZKRTBJRJ267QSK36C3LFZV7FPAYDDAXGIWDTNEGNV6MRRA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.643565583 (75h46m41.206888597s) [noise=true , address=79.127.205.245:4231 , piece_id=EJK46XUUVI5NWOAQJEW4QIJG3KRXZHB4YOQUJRCS2MIOSWKGTNLQ, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.644620291 (75h46m41.207943303s) [noise=true , address=77.90.31.14:4670 , piece_id=7FNANMUKPM3N6YWRAOAIM4WJZJUTNF7ZBI6ZUBNEPGNGMFWOE24A, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.648842639 (75h46m41.212165631s) [noise=true , address=104.36.82.55:4224 , piece_id=WRVNGSJV5BX4GJERLXNPAS7J7BZ5WFTM7R5H257YRCG5SP2HIYMA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.825828239 (75h46m41.389151241s) [noise=true , address=104.36.82.53:4126 , piece_id=2MKRQJEPGDJKDM7VWY5M4M6NORILAIEYYATJ7HVZEOBOMXOI2LBA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: ecclient: upload failed (node:12dMWFrq2NMpnMzuPAphWerntC5kkrcnzRJZBA116uzJzyvk6fH, address:104.36.82.53:4126): protocol: storage node overloaded, request limit: 2000 Jun 21 20:10:05.834666043 (75h46m41.397989033s) [upload=60693]: Next returned nil segment Jun 21 20:10:05.834686295 (75h46m41.398009302s) [upload=60693]: Waiting for error group managing segments... Jun 21 20:10:05.877010488 (75h46m41.440333486s) [noise=true , address=45.91.116.6:4982 , piece_id=VP45R6KXNH5U5SGUQUMAPIBY7KFDPBIR3S3VRIX27EAT6VLFBZ7Q, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:05.886336964 (75h46m41.449659967s) [noise=true , address=104.36.82.9:6039 , piece_id=I4AB6N27MYYUHWAZLJJUIBFBLOQFHDIM3YLYWPVRPNUV7VMYS2SA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: ecclient: upload failed (node:12P11WVW9mVXw5ra2gZ3Rezon6wZi8GrN5CchQN3MiVU4ts3L6V, address:104.36.82.9:6039): protocol: storage node overloaded, request limit: 2000 Jun 21 20:10:08.064514542 (75h46m43.627837553s) [noise=true , address=77.90.31.14:4670 , piece_id=7FNANMUKPM3N6YWRAOAIM4WJZJUTNF7ZBI6ZUBNEPGNGMFWOE24A, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:08.099098391 (75h46m43.662421396s) [noise=true , address=79.127.205.245:4231 , piece_id=EJK46XUUVI5NWOAQJEW4QIJG3KRXZHB4YOQUJRCS2MIOSWKGTNLQ, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:08.489461749 (75h46m44.052784752s) [noise=true , address=104.36.82.42:6807 , piece_id=ZBQZPXUDJ4VBCOSVKWAJ6SK6UD7SLPON7Y4SSCK2DQ5STFG4XCNQ, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:08.650922093 (75h46m44.214245093s) [noise=true , address=77.90.31.19:4853 , piece_id=VFEON5T2KSWM2KSLTVRZTXYL42JWENZYUEUPG3LMODGZWBMVGGPQ, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:09.917746439 (75h46m45.481069441s) [noise=true , address=104.36.82.13:6196 , piece_id=BCHMQO5KKT5BLNWQGOHH3V43LGSE3BUQCI2SY4TTO5M5GQNOCTMQ, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:10.830722215 (75h46m46.394045216s) [noise=true , address=104.36.82.5:5869 , piece_id=HAEHXXZASV3DHT3QN7NTZOS6XZKHNUWBKCHWJJWSTJKXAKJKKBAA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:12.738783450 (75h46m48.302106452s) [noise=true , address=77.90.31.12:4603 , piece_id=BMUIC6KNGEDL37436GLSXW5XOO6VYOPNOSFGQCYVHSENVI2PJ7KQ, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:15.460927713 (75h46m51.024250723s) [noise=true , address=104.36.82.52:4111 , piece_id=HB7QAWN6CGNIIT4HD2GZ7JDQP5NXIJ5I2USBM66YOKTYENNOWG5A, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:18.393817968 (75h46m53.95714096s) [noise=true , address=104.36.82.55:4224 , piece_id=WRVNGSJV5BX4GJERLXNPAS7J7BZ5WFTM7R5H257YRCG5SP2HIYMA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:19.549174143 (75h46m55.112497151s) [noise=true , address=104.36.82.56:4253 , piece_id=5PMVGLZETMXCQ77K3I6PXACRIMBGR2CV5JGUSU5KUMYFNXVGA2GA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:22.511944593 (75h46m58.075267584s) [noise=true , address=104.36.82.37:6663 , piece_id=X7LWYIIHJFU2T2PHYR3R5TNT22UUPHEHQWZH7ZJMLS4AH4PJ6AIA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:48.027197558 (75h47m23.590520562s) [noise=true , address=104.36.82.14:6232 , piece_id=A2ZLUUZKRTBJRJ267QSK36C3LFZV7FPAYDDAXGIWDTNEGNV6MRRA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:11:05.567692582 (75h47m41.131015574s) [noise=true , address=104.36.82.49:4982 , piece_id=UNLV5UDNWULEIPE4QSQHHZQPK3BQKFRB5T3H6GP7ZQ5QGSNK5V6A, seg_pos={0 0}, upload=60693]: Done uploading piece. err: ecclient: upload failed (node:1NxoYJXCXmTvQncbkXt6yH7mhEBBrzVdzhgL5J6W5TD4N3pkaG, address:104.36.82.49:4982): protocol: expected piece hash; context deadline exceeded; EOF Jun 21 20:11:05.567759198 (75h47m41.131082195s) [noise=true , address=104.36.82.54:4171 , piece_id=MUMABKARK4KKUKWW6XDXCQVT6O7SLFHGHE7COWC7QIYGARAA2D5A, seg_pos={0 0}, upload=60693]: Done uploading piece. err: ecclient: upload failed (node:125SMhDsEZruNu6LkmQn2M7R6WzvXTdk86DRsKf522u8BNiANq3, address:104.36.82.54:4171): protocol: expected piece hash; context deadline exceeded; EOF Jun 21 20:11:05.567869019 (75h47m41.131192033s) [noise=true , address=104.36.82.10:6087 , piece_id=5FQG7BPZC373PFB54HYXSG2SVKOBBBSGZCLC4XDJEYQFOLQY74MA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: ecclient: upload failed (node:1S9k5eejJcxkiG5pXjjBZm18uP8k5jCGfWyeW2QQxmuRU8ZzRA, address:104.36.82.10:6087): protocol: write tcp 159.223.143.39:38550->104.36.82.10:6087: use of closed network connection; write tcp 159.223.143.39:38550->104.36.82.10:6087: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:38550->104.36.82.10:6087: use of closed network connection Jun 21 20:11:05.567932150 (75h47m41.131255161s) [seg_pos={0 0}, upload=60693]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jun 21 20:11:05.593626175 (75h47m41.156949181s) [upload=60693]: Done uploading segments... Jun 21 20:11:05.593657274 (75h47m41.156980265s) [upload=60693]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 28.600465624s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [........................X...........X.......................] duration: 9.835063044s FAIL (2025-06-21 20:30:11 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== pload=60693]: Starting upload Jun 21 20:10:05.602866542 (75h46m41.166189557s) [upload=60693]: Done starting upload Jun 21 20:10:05.603116545 (75h46m41.166439547s) [upload=60693]: Uploading segments... Jun 21 20:10:05.603495389 (75h46m41.166818397s) [upload=60693]: Got next segment. Inline: false Jun 21 20:10:05.603566566 (75h46m41.166889558s) [upload=60693]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 21 20:10:05.603573280 (75h46m41.166896286s) [upload=60693]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 21 20:10:05.639130162 (75h46m41.202453164s) [seg_pos={0 0}, upload=60693]: Begin upload segment... Jun 21 20:10:05.639231070 (75h46m41.202554057s) [seg_pos={0 0}, upload=60693]: Done begin upload segment. Jun 21 20:10:05.639292724 (75h46m41.202615728s) [noise=true , address=104.36.82.56:4253 , piece_id=5PMVGLZETMXCQ77K3I6PXACRIMBGR2CV5JGUSU5KUMYFNXVGA2GA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.639338066 (75h46m41.202661071s) [noise=true , address=104.36.82.53:4126 , piece_id=2MKRQJEPGDJKDM7VWY5M4M6NORILAIEYYATJ7HVZEOBOMXOI2LBA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.639416264 (75h46m41.202739267s) [noise=true , address=104.36.82.5:5869 , piece_id=HAEHXXZASV3DHT3QN7NTZOS6XZKHNUWBKCHWJJWSTJKXAKJKKBAA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.639873098 (75h46m41.203196115s) [noise=true , address=104.36.82.52:4111 , piece_id=HB7QAWN6CGNIIT4HD2GZ7JDQP5NXIJ5I2USBM66YOKTYENNOWG5A, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.639887296 (75h46m41.203210304s) [noise=true , address=104.36.82.10:6087 , piece_id=5FQG7BPZC373PFB54HYXSG2SVKOBBBSGZCLC4XDJEYQFOLQY74MA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.640266623 (75h46m41.203589616s) [noise=true , address=104.36.82.13:6196 , piece_id=BCHMQO5KKT5BLNWQGOHH3V43LGSE3BUQCI2SY4TTO5M5GQNOCTMQ, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.640352274 (75h46m41.203675286s) [noise=true , address=104.36.82.37:6663 , piece_id=X7LWYIIHJFU2T2PHYR3R5TNT22UUPHEHQWZH7ZJMLS4AH4PJ6AIA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.640659767 (75h46m41.203982779s) [noise=true , address=104.36.82.42:6807 , piece_id=ZBQZPXUDJ4VBCOSVKWAJ6SK6UD7SLPON7Y4SSCK2DQ5STFG4XCNQ, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.640961701 (75h46m41.20428471s) [noise=true , address=45.91.116.6:4982 , piece_id=VP45R6KXNH5U5SGUQUMAPIBY7KFDPBIR3S3VRIX27EAT6VLFBZ7Q, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.641159578 (75h46m41.204482589s) [noise=true , address=104.36.82.9:6039 , piece_id=I4AB6N27MYYUHWAZLJJUIBFBLOQFHDIM3YLYWPVRPNUV7VMYS2SA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.641588126 (75h46m41.204911136s) [noise=true , address=77.90.31.19:4853 , piece_id=VFEON5T2KSWM2KSLTVRZTXYL42JWENZYUEUPG3LMODGZWBMVGGPQ, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.641863297 (75h46m41.205186305s) [noise=true , address=77.90.31.12:4603 , piece_id=BMUIC6KNGEDL37436GLSXW5XOO6VYOPNOSFGQCYVHSENVI2PJ7KQ, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.642124427 (75h46m41.205447437s) [noise=true , address=104.36.82.49:4982 , piece_id=UNLV5UDNWULEIPE4QSQHHZQPK3BQKFRB5T3H6GP7ZQ5QGSNK5V6A, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.642285917 (75h46m41.205608925s) [noise=true , address=104.36.82.54:4171 , piece_id=MUMABKARK4KKUKWW6XDXCQVT6O7SLFHGHE7COWC7QIYGARAA2D5A, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.642656750 (75h46m41.205979758s) [noise=true , address=104.36.82.14:6232 , piece_id=A2ZLUUZKRTBJRJ267QSK36C3LFZV7FPAYDDAXGIWDTNEGNV6MRRA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.643565583 (75h46m41.206888597s) [noise=true , address=79.127.205.245:4231 , piece_id=EJK46XUUVI5NWOAQJEW4QIJG3KRXZHB4YOQUJRCS2MIOSWKGTNLQ, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.644620291 (75h46m41.207943303s) [noise=true , address=77.90.31.14:4670 , piece_id=7FNANMUKPM3N6YWRAOAIM4WJZJUTNF7ZBI6ZUBNEPGNGMFWOE24A, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.648842639 (75h46m41.212165631s) [noise=true , address=104.36.82.55:4224 , piece_id=WRVNGSJV5BX4GJERLXNPAS7J7BZ5WFTM7R5H257YRCG5SP2HIYMA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.825828239 (75h46m41.389151241s) [noise=true , address=104.36.82.53:4126 , piece_id=2MKRQJEPGDJKDM7VWY5M4M6NORILAIEYYATJ7HVZEOBOMXOI2LBA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: ecclient: upload failed (node:12dMWFrq2NMpnMzuPAphWerntC5kkrcnzRJZBA116uzJzyvk6fH, address:104.36.82.53:4126): protocol: storage node overloaded, request limit: 2000 Jun 21 20:10:05.834666043 (75h46m41.397989033s) [upload=60693]: Next returned nil segment Jun 21 20:10:05.834686295 (75h46m41.398009302s) [upload=60693]: Waiting for error group managing segments... Jun 21 20:10:05.877010488 (75h46m41.440333486s) [noise=true , address=45.91.116.6:4982 , piece_id=VP45R6KXNH5U5SGUQUMAPIBY7KFDPBIR3S3VRIX27EAT6VLFBZ7Q, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:05.886336964 (75h46m41.449659967s) [noise=true , address=104.36.82.9:6039 , piece_id=I4AB6N27MYYUHWAZLJJUIBFBLOQFHDIM3YLYWPVRPNUV7VMYS2SA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: ecclient: upload failed (node:12P11WVW9mVXw5ra2gZ3Rezon6wZi8GrN5CchQN3MiVU4ts3L6V, address:104.36.82.9:6039): protocol: storage node overloaded, request limit: 2000 Jun 21 20:10:08.064514542 (75h46m43.627837553s) [noise=true , address=77.90.31.14:4670 , piece_id=7FNANMUKPM3N6YWRAOAIM4WJZJUTNF7ZBI6ZUBNEPGNGMFWOE24A, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:08.099098391 (75h46m43.662421396s) [noise=true , address=79.127.205.245:4231 , piece_id=EJK46XUUVI5NWOAQJEW4QIJG3KRXZHB4YOQUJRCS2MIOSWKGTNLQ, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:08.489461749 (75h46m44.052784752s) [noise=true , address=104.36.82.42:6807 , piece_id=ZBQZPXUDJ4VBCOSVKWAJ6SK6UD7SLPON7Y4SSCK2DQ5STFG4XCNQ, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:08.650922093 (75h46m44.214245093s) [noise=true , address=77.90.31.19:4853 , piece_id=VFEON5T2KSWM2KSLTVRZTXYL42JWENZYUEUPG3LMODGZWBMVGGPQ, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:09.917746439 (75h46m45.481069441s) [noise=true , address=104.36.82.13:6196 , piece_id=BCHMQO5KKT5BLNWQGOHH3V43LGSE3BUQCI2SY4TTO5M5GQNOCTMQ, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:10.830722215 (75h46m46.394045216s) [noise=true , address=104.36.82.5:5869 , piece_id=HAEHXXZASV3DHT3QN7NTZOS6XZKHNUWBKCHWJJWSTJKXAKJKKBAA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:12.738783450 (75h46m48.302106452s) [noise=true , address=77.90.31.12:4603 , piece_id=BMUIC6KNGEDL37436GLSXW5XOO6VYOPNOSFGQCYVHSENVI2PJ7KQ, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:15.460927713 (75h46m51.024250723s) [noise=true , address=104.36.82.52:4111 , piece_id=HB7QAWN6CGNIIT4HD2GZ7JDQP5NXIJ5I2USBM66YOKTYENNOWG5A, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:18.393817968 (75h46m53.95714096s) [noise=true , address=104.36.82.55:4224 , piece_id=WRVNGSJV5BX4GJERLXNPAS7J7BZ5WFTM7R5H257YRCG5SP2HIYMA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:19.549174143 (75h46m55.112497151s) [noise=true , address=104.36.82.56:4253 , piece_id=5PMVGLZETMXCQ77K3I6PXACRIMBGR2CV5JGUSU5KUMYFNXVGA2GA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:22.511944593 (75h46m58.075267584s) [noise=true , address=104.36.82.37:6663 , piece_id=X7LWYIIHJFU2T2PHYR3R5TNT22UUPHEHQWZH7ZJMLS4AH4PJ6AIA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:48.027197558 (75h47m23.590520562s) [noise=true , address=104.36.82.14:6232 , piece_id=A2ZLUUZKRTBJRJ267QSK36C3LFZV7FPAYDDAXGIWDTNEGNV6MRRA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:11:05.567692582 (75h47m41.131015574s) [noise=true , address=104.36.82.49:4982 , piece_id=UNLV5UDNWULEIPE4QSQHHZQPK3BQKFRB5T3H6GP7ZQ5QGSNK5V6A, seg_pos={0 0}, upload=60693]: Done uploading piece. err: ecclient: upload failed (node:1NxoYJXCXmTvQncbkXt6yH7mhEBBrzVdzhgL5J6W5TD4N3pkaG, address:104.36.82.49:4982): protocol: expected piece hash; context deadline exceeded; EOF Jun 21 20:11:05.567759198 (75h47m41.131082195s) [noise=true , address=104.36.82.54:4171 , piece_id=MUMABKARK4KKUKWW6XDXCQVT6O7SLFHGHE7COWC7QIYGARAA2D5A, seg_pos={0 0}, upload=60693]: Done uploading piece. err: ecclient: upload failed (node:125SMhDsEZruNu6LkmQn2M7R6WzvXTdk86DRsKf522u8BNiANq3, address:104.36.82.54:4171): protocol: expected piece hash; context deadline exceeded; EOF Jun 21 20:11:05.567869019 (75h47m41.131192033s) [noise=true , address=104.36.82.10:6087 , piece_id=5FQG7BPZC373PFB54HYXSG2SVKOBBBSGZCLC4XDJEYQFOLQY74MA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: ecclient: upload failed (node:1S9k5eejJcxkiG5pXjjBZm18uP8k5jCGfWyeW2QQxmuRU8ZzRA, address:104.36.82.10:6087): protocol: write tcp 159.223.143.39:38550->104.36.82.10:6087: use of closed network connection; write tcp 159.223.143.39:38550->104.36.82.10:6087: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:38550->104.36.82.10:6087: use of closed network connection Jun 21 20:11:05.567932150 (75h47m41.131255161s) [seg_pos={0 0}, upload=60693]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jun 21 20:11:05.593626175 (75h47m41.156949181s) [upload=60693]: Done uploading segments... Jun 21 20:11:05.593657274 (75h47m41.156980265s) [upload=60693]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 9.835094645s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [.........................X...........X......................] duration: 18.154400925s FAIL (2025-06-21 20:29:30 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== pload=60693]: Starting upload Jun 21 20:10:05.602866542 (75h46m41.166189557s) [upload=60693]: Done starting upload Jun 21 20:10:05.603116545 (75h46m41.166439547s) [upload=60693]: Uploading segments... Jun 21 20:10:05.603495389 (75h46m41.166818397s) [upload=60693]: Got next segment. Inline: false Jun 21 20:10:05.603566566 (75h46m41.166889558s) [upload=60693]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 21 20:10:05.603573280 (75h46m41.166896286s) [upload=60693]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 21 20:10:05.639130162 (75h46m41.202453164s) [seg_pos={0 0}, upload=60693]: Begin upload segment... Jun 21 20:10:05.639231070 (75h46m41.202554057s) [seg_pos={0 0}, upload=60693]: Done begin upload segment. Jun 21 20:10:05.639292724 (75h46m41.202615728s) [noise=true , address=104.36.82.56:4253 , piece_id=5PMVGLZETMXCQ77K3I6PXACRIMBGR2CV5JGUSU5KUMYFNXVGA2GA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.639338066 (75h46m41.202661071s) [noise=true , address=104.36.82.53:4126 , piece_id=2MKRQJEPGDJKDM7VWY5M4M6NORILAIEYYATJ7HVZEOBOMXOI2LBA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.639416264 (75h46m41.202739267s) [noise=true , address=104.36.82.5:5869 , piece_id=HAEHXXZASV3DHT3QN7NTZOS6XZKHNUWBKCHWJJWSTJKXAKJKKBAA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.639873098 (75h46m41.203196115s) [noise=true , address=104.36.82.52:4111 , piece_id=HB7QAWN6CGNIIT4HD2GZ7JDQP5NXIJ5I2USBM66YOKTYENNOWG5A, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.639887296 (75h46m41.203210304s) [noise=true , address=104.36.82.10:6087 , piece_id=5FQG7BPZC373PFB54HYXSG2SVKOBBBSGZCLC4XDJEYQFOLQY74MA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.640266623 (75h46m41.203589616s) [noise=true , address=104.36.82.13:6196 , piece_id=BCHMQO5KKT5BLNWQGOHH3V43LGSE3BUQCI2SY4TTO5M5GQNOCTMQ, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.640352274 (75h46m41.203675286s) [noise=true , address=104.36.82.37:6663 , piece_id=X7LWYIIHJFU2T2PHYR3R5TNT22UUPHEHQWZH7ZJMLS4AH4PJ6AIA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.640659767 (75h46m41.203982779s) [noise=true , address=104.36.82.42:6807 , piece_id=ZBQZPXUDJ4VBCOSVKWAJ6SK6UD7SLPON7Y4SSCK2DQ5STFG4XCNQ, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.640961701 (75h46m41.20428471s) [noise=true , address=45.91.116.6:4982 , piece_id=VP45R6KXNH5U5SGUQUMAPIBY7KFDPBIR3S3VRIX27EAT6VLFBZ7Q, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.641159578 (75h46m41.204482589s) [noise=true , address=104.36.82.9:6039 , piece_id=I4AB6N27MYYUHWAZLJJUIBFBLOQFHDIM3YLYWPVRPNUV7VMYS2SA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.641588126 (75h46m41.204911136s) [noise=true , address=77.90.31.19:4853 , piece_id=VFEON5T2KSWM2KSLTVRZTXYL42JWENZYUEUPG3LMODGZWBMVGGPQ, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.641863297 (75h46m41.205186305s) [noise=true , address=77.90.31.12:4603 , piece_id=BMUIC6KNGEDL37436GLSXW5XOO6VYOPNOSFGQCYVHSENVI2PJ7KQ, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.642124427 (75h46m41.205447437s) [noise=true , address=104.36.82.49:4982 , piece_id=UNLV5UDNWULEIPE4QSQHHZQPK3BQKFRB5T3H6GP7ZQ5QGSNK5V6A, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.642285917 (75h46m41.205608925s) [noise=true , address=104.36.82.54:4171 , piece_id=MUMABKARK4KKUKWW6XDXCQVT6O7SLFHGHE7COWC7QIYGARAA2D5A, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.642656750 (75h46m41.205979758s) [noise=true , address=104.36.82.14:6232 , piece_id=A2ZLUUZKRTBJRJ267QSK36C3LFZV7FPAYDDAXGIWDTNEGNV6MRRA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.643565583 (75h46m41.206888597s) [noise=true , address=79.127.205.245:4231 , piece_id=EJK46XUUVI5NWOAQJEW4QIJG3KRXZHB4YOQUJRCS2MIOSWKGTNLQ, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.644620291 (75h46m41.207943303s) [noise=true , address=77.90.31.14:4670 , piece_id=7FNANMUKPM3N6YWRAOAIM4WJZJUTNF7ZBI6ZUBNEPGNGMFWOE24A, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.648842639 (75h46m41.212165631s) [noise=true , address=104.36.82.55:4224 , piece_id=WRVNGSJV5BX4GJERLXNPAS7J7BZ5WFTM7R5H257YRCG5SP2HIYMA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.825828239 (75h46m41.389151241s) [noise=true , address=104.36.82.53:4126 , piece_id=2MKRQJEPGDJKDM7VWY5M4M6NORILAIEYYATJ7HVZEOBOMXOI2LBA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: ecclient: upload failed (node:12dMWFrq2NMpnMzuPAphWerntC5kkrcnzRJZBA116uzJzyvk6fH, address:104.36.82.53:4126): protocol: storage node overloaded, request limit: 2000 Jun 21 20:10:05.834666043 (75h46m41.397989033s) [upload=60693]: Next returned nil segment Jun 21 20:10:05.834686295 (75h46m41.398009302s) [upload=60693]: Waiting for error group managing segments... Jun 21 20:10:05.877010488 (75h46m41.440333486s) [noise=true , address=45.91.116.6:4982 , piece_id=VP45R6KXNH5U5SGUQUMAPIBY7KFDPBIR3S3VRIX27EAT6VLFBZ7Q, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:05.886336964 (75h46m41.449659967s) [noise=true , address=104.36.82.9:6039 , piece_id=I4AB6N27MYYUHWAZLJJUIBFBLOQFHDIM3YLYWPVRPNUV7VMYS2SA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: ecclient: upload failed (node:12P11WVW9mVXw5ra2gZ3Rezon6wZi8GrN5CchQN3MiVU4ts3L6V, address:104.36.82.9:6039): protocol: storage node overloaded, request limit: 2000 Jun 21 20:10:08.064514542 (75h46m43.627837553s) [noise=true , address=77.90.31.14:4670 , piece_id=7FNANMUKPM3N6YWRAOAIM4WJZJUTNF7ZBI6ZUBNEPGNGMFWOE24A, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:08.099098391 (75h46m43.662421396s) [noise=true , address=79.127.205.245:4231 , piece_id=EJK46XUUVI5NWOAQJEW4QIJG3KRXZHB4YOQUJRCS2MIOSWKGTNLQ, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:08.489461749 (75h46m44.052784752s) [noise=true , address=104.36.82.42:6807 , piece_id=ZBQZPXUDJ4VBCOSVKWAJ6SK6UD7SLPON7Y4SSCK2DQ5STFG4XCNQ, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:08.650922093 (75h46m44.214245093s) [noise=true , address=77.90.31.19:4853 , piece_id=VFEON5T2KSWM2KSLTVRZTXYL42JWENZYUEUPG3LMODGZWBMVGGPQ, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:09.917746439 (75h46m45.481069441s) [noise=true , address=104.36.82.13:6196 , piece_id=BCHMQO5KKT5BLNWQGOHH3V43LGSE3BUQCI2SY4TTO5M5GQNOCTMQ, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:10.830722215 (75h46m46.394045216s) [noise=true , address=104.36.82.5:5869 , piece_id=HAEHXXZASV3DHT3QN7NTZOS6XZKHNUWBKCHWJJWSTJKXAKJKKBAA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:12.738783450 (75h46m48.302106452s) [noise=true , address=77.90.31.12:4603 , piece_id=BMUIC6KNGEDL37436GLSXW5XOO6VYOPNOSFGQCYVHSENVI2PJ7KQ, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:15.460927713 (75h46m51.024250723s) [noise=true , address=104.36.82.52:4111 , piece_id=HB7QAWN6CGNIIT4HD2GZ7JDQP5NXIJ5I2USBM66YOKTYENNOWG5A, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:18.393817968 (75h46m53.95714096s) [noise=true , address=104.36.82.55:4224 , piece_id=WRVNGSJV5BX4GJERLXNPAS7J7BZ5WFTM7R5H257YRCG5SP2HIYMA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:19.549174143 (75h46m55.112497151s) [noise=true , address=104.36.82.56:4253 , piece_id=5PMVGLZETMXCQ77K3I6PXACRIMBGR2CV5JGUSU5KUMYFNXVGA2GA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:22.511944593 (75h46m58.075267584s) [noise=true , address=104.36.82.37:6663 , piece_id=X7LWYIIHJFU2T2PHYR3R5TNT22UUPHEHQWZH7ZJMLS4AH4PJ6AIA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:48.027197558 (75h47m23.590520562s) [noise=true , address=104.36.82.14:6232 , piece_id=A2ZLUUZKRTBJRJ267QSK36C3LFZV7FPAYDDAXGIWDTNEGNV6MRRA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:11:05.567692582 (75h47m41.131015574s) [noise=true , address=104.36.82.49:4982 , piece_id=UNLV5UDNWULEIPE4QSQHHZQPK3BQKFRB5T3H6GP7ZQ5QGSNK5V6A, seg_pos={0 0}, upload=60693]: Done uploading piece. err: ecclient: upload failed (node:1NxoYJXCXmTvQncbkXt6yH7mhEBBrzVdzhgL5J6W5TD4N3pkaG, address:104.36.82.49:4982): protocol: expected piece hash; context deadline exceeded; EOF Jun 21 20:11:05.567759198 (75h47m41.131082195s) [noise=true , address=104.36.82.54:4171 , piece_id=MUMABKARK4KKUKWW6XDXCQVT6O7SLFHGHE7COWC7QIYGARAA2D5A, seg_pos={0 0}, upload=60693]: Done uploading piece. err: ecclient: upload failed (node:125SMhDsEZruNu6LkmQn2M7R6WzvXTdk86DRsKf522u8BNiANq3, address:104.36.82.54:4171): protocol: expected piece hash; context deadline exceeded; EOF Jun 21 20:11:05.567869019 (75h47m41.131192033s) [noise=true , address=104.36.82.10:6087 , piece_id=5FQG7BPZC373PFB54HYXSG2SVKOBBBSGZCLC4XDJEYQFOLQY74MA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: ecclient: upload failed (node:1S9k5eejJcxkiG5pXjjBZm18uP8k5jCGfWyeW2QQxmuRU8ZzRA, address:104.36.82.10:6087): protocol: write tcp 159.223.143.39:38550->104.36.82.10:6087: use of closed network connection; write tcp 159.223.143.39:38550->104.36.82.10:6087: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:38550->104.36.82.10:6087: use of closed network connection Jun 21 20:11:05.567932150 (75h47m41.131255161s) [seg_pos={0 0}, upload=60693]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jun 21 20:11:05.593626175 (75h47m41.156949181s) [upload=60693]: Done uploading segments... Jun 21 20:11:05.593657274 (75h47m41.156980265s) [upload=60693]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 18.154433043s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [..........................X...........X.....................] duration: 23.102523773s FAIL (2025-06-21 20:28:30 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== pload=60693]: Starting upload Jun 21 20:10:05.602866542 (75h46m41.166189557s) [upload=60693]: Done starting upload Jun 21 20:10:05.603116545 (75h46m41.166439547s) [upload=60693]: Uploading segments... Jun 21 20:10:05.603495389 (75h46m41.166818397s) [upload=60693]: Got next segment. Inline: false Jun 21 20:10:05.603566566 (75h46m41.166889558s) [upload=60693]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 21 20:10:05.603573280 (75h46m41.166896286s) [upload=60693]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 21 20:10:05.639130162 (75h46m41.202453164s) [seg_pos={0 0}, upload=60693]: Begin upload segment... Jun 21 20:10:05.639231070 (75h46m41.202554057s) [seg_pos={0 0}, upload=60693]: Done begin upload segment. Jun 21 20:10:05.639292724 (75h46m41.202615728s) [noise=true , address=104.36.82.56:4253 , piece_id=5PMVGLZETMXCQ77K3I6PXACRIMBGR2CV5JGUSU5KUMYFNXVGA2GA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.639338066 (75h46m41.202661071s) [noise=true , address=104.36.82.53:4126 , piece_id=2MKRQJEPGDJKDM7VWY5M4M6NORILAIEYYATJ7HVZEOBOMXOI2LBA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.639416264 (75h46m41.202739267s) [noise=true , address=104.36.82.5:5869 , piece_id=HAEHXXZASV3DHT3QN7NTZOS6XZKHNUWBKCHWJJWSTJKXAKJKKBAA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.639873098 (75h46m41.203196115s) [noise=true , address=104.36.82.52:4111 , piece_id=HB7QAWN6CGNIIT4HD2GZ7JDQP5NXIJ5I2USBM66YOKTYENNOWG5A, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.639887296 (75h46m41.203210304s) [noise=true , address=104.36.82.10:6087 , piece_id=5FQG7BPZC373PFB54HYXSG2SVKOBBBSGZCLC4XDJEYQFOLQY74MA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.640266623 (75h46m41.203589616s) [noise=true , address=104.36.82.13:6196 , piece_id=BCHMQO5KKT5BLNWQGOHH3V43LGSE3BUQCI2SY4TTO5M5GQNOCTMQ, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.640352274 (75h46m41.203675286s) [noise=true , address=104.36.82.37:6663 , piece_id=X7LWYIIHJFU2T2PHYR3R5TNT22UUPHEHQWZH7ZJMLS4AH4PJ6AIA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.640659767 (75h46m41.203982779s) [noise=true , address=104.36.82.42:6807 , piece_id=ZBQZPXUDJ4VBCOSVKWAJ6SK6UD7SLPON7Y4SSCK2DQ5STFG4XCNQ, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.640961701 (75h46m41.20428471s) [noise=true , address=45.91.116.6:4982 , piece_id=VP45R6KXNH5U5SGUQUMAPIBY7KFDPBIR3S3VRIX27EAT6VLFBZ7Q, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.641159578 (75h46m41.204482589s) [noise=true , address=104.36.82.9:6039 , piece_id=I4AB6N27MYYUHWAZLJJUIBFBLOQFHDIM3YLYWPVRPNUV7VMYS2SA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.641588126 (75h46m41.204911136s) [noise=true , address=77.90.31.19:4853 , piece_id=VFEON5T2KSWM2KSLTVRZTXYL42JWENZYUEUPG3LMODGZWBMVGGPQ, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.641863297 (75h46m41.205186305s) [noise=true , address=77.90.31.12:4603 , piece_id=BMUIC6KNGEDL37436GLSXW5XOO6VYOPNOSFGQCYVHSENVI2PJ7KQ, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.642124427 (75h46m41.205447437s) [noise=true , address=104.36.82.49:4982 , piece_id=UNLV5UDNWULEIPE4QSQHHZQPK3BQKFRB5T3H6GP7ZQ5QGSNK5V6A, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.642285917 (75h46m41.205608925s) [noise=true , address=104.36.82.54:4171 , piece_id=MUMABKARK4KKUKWW6XDXCQVT6O7SLFHGHE7COWC7QIYGARAA2D5A, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.642656750 (75h46m41.205979758s) [noise=true , address=104.36.82.14:6232 , piece_id=A2ZLUUZKRTBJRJ267QSK36C3LFZV7FPAYDDAXGIWDTNEGNV6MRRA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.643565583 (75h46m41.206888597s) [noise=true , address=79.127.205.245:4231 , piece_id=EJK46XUUVI5NWOAQJEW4QIJG3KRXZHB4YOQUJRCS2MIOSWKGTNLQ, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.644620291 (75h46m41.207943303s) [noise=true , address=77.90.31.14:4670 , piece_id=7FNANMUKPM3N6YWRAOAIM4WJZJUTNF7ZBI6ZUBNEPGNGMFWOE24A, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.648842639 (75h46m41.212165631s) [noise=true , address=104.36.82.55:4224 , piece_id=WRVNGSJV5BX4GJERLXNPAS7J7BZ5WFTM7R5H257YRCG5SP2HIYMA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.825828239 (75h46m41.389151241s) [noise=true , address=104.36.82.53:4126 , piece_id=2MKRQJEPGDJKDM7VWY5M4M6NORILAIEYYATJ7HVZEOBOMXOI2LBA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: ecclient: upload failed (node:12dMWFrq2NMpnMzuPAphWerntC5kkrcnzRJZBA116uzJzyvk6fH, address:104.36.82.53:4126): protocol: storage node overloaded, request limit: 2000 Jun 21 20:10:05.834666043 (75h46m41.397989033s) [upload=60693]: Next returned nil segment Jun 21 20:10:05.834686295 (75h46m41.398009302s) [upload=60693]: Waiting for error group managing segments... Jun 21 20:10:05.877010488 (75h46m41.440333486s) [noise=true , address=45.91.116.6:4982 , piece_id=VP45R6KXNH5U5SGUQUMAPIBY7KFDPBIR3S3VRIX27EAT6VLFBZ7Q, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:05.886336964 (75h46m41.449659967s) [noise=true , address=104.36.82.9:6039 , piece_id=I4AB6N27MYYUHWAZLJJUIBFBLOQFHDIM3YLYWPVRPNUV7VMYS2SA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: ecclient: upload failed (node:12P11WVW9mVXw5ra2gZ3Rezon6wZi8GrN5CchQN3MiVU4ts3L6V, address:104.36.82.9:6039): protocol: storage node overloaded, request limit: 2000 Jun 21 20:10:08.064514542 (75h46m43.627837553s) [noise=true , address=77.90.31.14:4670 , piece_id=7FNANMUKPM3N6YWRAOAIM4WJZJUTNF7ZBI6ZUBNEPGNGMFWOE24A, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:08.099098391 (75h46m43.662421396s) [noise=true , address=79.127.205.245:4231 , piece_id=EJK46XUUVI5NWOAQJEW4QIJG3KRXZHB4YOQUJRCS2MIOSWKGTNLQ, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:08.489461749 (75h46m44.052784752s) [noise=true , address=104.36.82.42:6807 , piece_id=ZBQZPXUDJ4VBCOSVKWAJ6SK6UD7SLPON7Y4SSCK2DQ5STFG4XCNQ, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:08.650922093 (75h46m44.214245093s) [noise=true , address=77.90.31.19:4853 , piece_id=VFEON5T2KSWM2KSLTVRZTXYL42JWENZYUEUPG3LMODGZWBMVGGPQ, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:09.917746439 (75h46m45.481069441s) [noise=true , address=104.36.82.13:6196 , piece_id=BCHMQO5KKT5BLNWQGOHH3V43LGSE3BUQCI2SY4TTO5M5GQNOCTMQ, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:10.830722215 (75h46m46.394045216s) [noise=true , address=104.36.82.5:5869 , piece_id=HAEHXXZASV3DHT3QN7NTZOS6XZKHNUWBKCHWJJWSTJKXAKJKKBAA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:12.738783450 (75h46m48.302106452s) [noise=true , address=77.90.31.12:4603 , piece_id=BMUIC6KNGEDL37436GLSXW5XOO6VYOPNOSFGQCYVHSENVI2PJ7KQ, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:15.460927713 (75h46m51.024250723s) [noise=true , address=104.36.82.52:4111 , piece_id=HB7QAWN6CGNIIT4HD2GZ7JDQP5NXIJ5I2USBM66YOKTYENNOWG5A, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:18.393817968 (75h46m53.95714096s) [noise=true , address=104.36.82.55:4224 , piece_id=WRVNGSJV5BX4GJERLXNPAS7J7BZ5WFTM7R5H257YRCG5SP2HIYMA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:19.549174143 (75h46m55.112497151s) [noise=true , address=104.36.82.56:4253 , piece_id=5PMVGLZETMXCQ77K3I6PXACRIMBGR2CV5JGUSU5KUMYFNXVGA2GA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:22.511944593 (75h46m58.075267584s) [noise=true , address=104.36.82.37:6663 , piece_id=X7LWYIIHJFU2T2PHYR3R5TNT22UUPHEHQWZH7ZJMLS4AH4PJ6AIA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:48.027197558 (75h47m23.590520562s) [noise=true , address=104.36.82.14:6232 , piece_id=A2ZLUUZKRTBJRJ267QSK36C3LFZV7FPAYDDAXGIWDTNEGNV6MRRA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:11:05.567692582 (75h47m41.131015574s) [noise=true , address=104.36.82.49:4982 , piece_id=UNLV5UDNWULEIPE4QSQHHZQPK3BQKFRB5T3H6GP7ZQ5QGSNK5V6A, seg_pos={0 0}, upload=60693]: Done uploading piece. err: ecclient: upload failed (node:1NxoYJXCXmTvQncbkXt6yH7mhEBBrzVdzhgL5J6W5TD4N3pkaG, address:104.36.82.49:4982): protocol: expected piece hash; context deadline exceeded; EOF Jun 21 20:11:05.567759198 (75h47m41.131082195s) [noise=true , address=104.36.82.54:4171 , piece_id=MUMABKARK4KKUKWW6XDXCQVT6O7SLFHGHE7COWC7QIYGARAA2D5A, seg_pos={0 0}, upload=60693]: Done uploading piece. err: ecclient: upload failed (node:125SMhDsEZruNu6LkmQn2M7R6WzvXTdk86DRsKf522u8BNiANq3, address:104.36.82.54:4171): protocol: expected piece hash; context deadline exceeded; EOF Jun 21 20:11:05.567869019 (75h47m41.131192033s) [noise=true , address=104.36.82.10:6087 , piece_id=5FQG7BPZC373PFB54HYXSG2SVKOBBBSGZCLC4XDJEYQFOLQY74MA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: ecclient: upload failed (node:1S9k5eejJcxkiG5pXjjBZm18uP8k5jCGfWyeW2QQxmuRU8ZzRA, address:104.36.82.10:6087): protocol: write tcp 159.223.143.39:38550->104.36.82.10:6087: use of closed network connection; write tcp 159.223.143.39:38550->104.36.82.10:6087: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:38550->104.36.82.10:6087: use of closed network connection Jun 21 20:11:05.567932150 (75h47m41.131255161s) [seg_pos={0 0}, upload=60693]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jun 21 20:11:05.593626175 (75h47m41.156949181s) [upload=60693]: Done uploading segments... Jun 21 20:11:05.593657274 (75h47m41.156980265s) [upload=60693]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 23.10255471s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [...........................X...........X....................] duration: 27.092209567s FAIL (2025-06-21 20:27:36 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== pload=60693]: Starting upload Jun 21 20:10:05.602866542 (75h46m41.166189557s) [upload=60693]: Done starting upload Jun 21 20:10:05.603116545 (75h46m41.166439547s) [upload=60693]: Uploading segments... Jun 21 20:10:05.603495389 (75h46m41.166818397s) [upload=60693]: Got next segment. Inline: false Jun 21 20:10:05.603566566 (75h46m41.166889558s) [upload=60693]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 21 20:10:05.603573280 (75h46m41.166896286s) [upload=60693]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 21 20:10:05.639130162 (75h46m41.202453164s) [seg_pos={0 0}, upload=60693]: Begin upload segment... Jun 21 20:10:05.639231070 (75h46m41.202554057s) [seg_pos={0 0}, upload=60693]: Done begin upload segment. Jun 21 20:10:05.639292724 (75h46m41.202615728s) [noise=true , address=104.36.82.56:4253 , piece_id=5PMVGLZETMXCQ77K3I6PXACRIMBGR2CV5JGUSU5KUMYFNXVGA2GA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.639338066 (75h46m41.202661071s) [noise=true , address=104.36.82.53:4126 , piece_id=2MKRQJEPGDJKDM7VWY5M4M6NORILAIEYYATJ7HVZEOBOMXOI2LBA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.639416264 (75h46m41.202739267s) [noise=true , address=104.36.82.5:5869 , piece_id=HAEHXXZASV3DHT3QN7NTZOS6XZKHNUWBKCHWJJWSTJKXAKJKKBAA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.639873098 (75h46m41.203196115s) [noise=true , address=104.36.82.52:4111 , piece_id=HB7QAWN6CGNIIT4HD2GZ7JDQP5NXIJ5I2USBM66YOKTYENNOWG5A, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.639887296 (75h46m41.203210304s) [noise=true , address=104.36.82.10:6087 , piece_id=5FQG7BPZC373PFB54HYXSG2SVKOBBBSGZCLC4XDJEYQFOLQY74MA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.640266623 (75h46m41.203589616s) [noise=true , address=104.36.82.13:6196 , piece_id=BCHMQO5KKT5BLNWQGOHH3V43LGSE3BUQCI2SY4TTO5M5GQNOCTMQ, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.640352274 (75h46m41.203675286s) [noise=true , address=104.36.82.37:6663 , piece_id=X7LWYIIHJFU2T2PHYR3R5TNT22UUPHEHQWZH7ZJMLS4AH4PJ6AIA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.640659767 (75h46m41.203982779s) [noise=true , address=104.36.82.42:6807 , piece_id=ZBQZPXUDJ4VBCOSVKWAJ6SK6UD7SLPON7Y4SSCK2DQ5STFG4XCNQ, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.640961701 (75h46m41.20428471s) [noise=true , address=45.91.116.6:4982 , piece_id=VP45R6KXNH5U5SGUQUMAPIBY7KFDPBIR3S3VRIX27EAT6VLFBZ7Q, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.641159578 (75h46m41.204482589s) [noise=true , address=104.36.82.9:6039 , piece_id=I4AB6N27MYYUHWAZLJJUIBFBLOQFHDIM3YLYWPVRPNUV7VMYS2SA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.641588126 (75h46m41.204911136s) [noise=true , address=77.90.31.19:4853 , piece_id=VFEON5T2KSWM2KSLTVRZTXYL42JWENZYUEUPG3LMODGZWBMVGGPQ, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.641863297 (75h46m41.205186305s) [noise=true , address=77.90.31.12:4603 , piece_id=BMUIC6KNGEDL37436GLSXW5XOO6VYOPNOSFGQCYVHSENVI2PJ7KQ, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.642124427 (75h46m41.205447437s) [noise=true , address=104.36.82.49:4982 , piece_id=UNLV5UDNWULEIPE4QSQHHZQPK3BQKFRB5T3H6GP7ZQ5QGSNK5V6A, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.642285917 (75h46m41.205608925s) [noise=true , address=104.36.82.54:4171 , piece_id=MUMABKARK4KKUKWW6XDXCQVT6O7SLFHGHE7COWC7QIYGARAA2D5A, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.642656750 (75h46m41.205979758s) [noise=true , address=104.36.82.14:6232 , piece_id=A2ZLUUZKRTBJRJ267QSK36C3LFZV7FPAYDDAXGIWDTNEGNV6MRRA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.643565583 (75h46m41.206888597s) [noise=true , address=79.127.205.245:4231 , piece_id=EJK46XUUVI5NWOAQJEW4QIJG3KRXZHB4YOQUJRCS2MIOSWKGTNLQ, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.644620291 (75h46m41.207943303s) [noise=true , address=77.90.31.14:4670 , piece_id=7FNANMUKPM3N6YWRAOAIM4WJZJUTNF7ZBI6ZUBNEPGNGMFWOE24A, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.648842639 (75h46m41.212165631s) [noise=true , address=104.36.82.55:4224 , piece_id=WRVNGSJV5BX4GJERLXNPAS7J7BZ5WFTM7R5H257YRCG5SP2HIYMA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.825828239 (75h46m41.389151241s) [noise=true , address=104.36.82.53:4126 , piece_id=2MKRQJEPGDJKDM7VWY5M4M6NORILAIEYYATJ7HVZEOBOMXOI2LBA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: ecclient: upload failed (node:12dMWFrq2NMpnMzuPAphWerntC5kkrcnzRJZBA116uzJzyvk6fH, address:104.36.82.53:4126): protocol: storage node overloaded, request limit: 2000 Jun 21 20:10:05.834666043 (75h46m41.397989033s) [upload=60693]: Next returned nil segment Jun 21 20:10:05.834686295 (75h46m41.398009302s) [upload=60693]: Waiting for error group managing segments... Jun 21 20:10:05.877010488 (75h46m41.440333486s) [noise=true , address=45.91.116.6:4982 , piece_id=VP45R6KXNH5U5SGUQUMAPIBY7KFDPBIR3S3VRIX27EAT6VLFBZ7Q, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:05.886336964 (75h46m41.449659967s) [noise=true , address=104.36.82.9:6039 , piece_id=I4AB6N27MYYUHWAZLJJUIBFBLOQFHDIM3YLYWPVRPNUV7VMYS2SA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: ecclient: upload failed (node:12P11WVW9mVXw5ra2gZ3Rezon6wZi8GrN5CchQN3MiVU4ts3L6V, address:104.36.82.9:6039): protocol: storage node overloaded, request limit: 2000 Jun 21 20:10:08.064514542 (75h46m43.627837553s) [noise=true , address=77.90.31.14:4670 , piece_id=7FNANMUKPM3N6YWRAOAIM4WJZJUTNF7ZBI6ZUBNEPGNGMFWOE24A, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:08.099098391 (75h46m43.662421396s) [noise=true , address=79.127.205.245:4231 , piece_id=EJK46XUUVI5NWOAQJEW4QIJG3KRXZHB4YOQUJRCS2MIOSWKGTNLQ, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:08.489461749 (75h46m44.052784752s) [noise=true , address=104.36.82.42:6807 , piece_id=ZBQZPXUDJ4VBCOSVKWAJ6SK6UD7SLPON7Y4SSCK2DQ5STFG4XCNQ, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:08.650922093 (75h46m44.214245093s) [noise=true , address=77.90.31.19:4853 , piece_id=VFEON5T2KSWM2KSLTVRZTXYL42JWENZYUEUPG3LMODGZWBMVGGPQ, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:09.917746439 (75h46m45.481069441s) [noise=true , address=104.36.82.13:6196 , piece_id=BCHMQO5KKT5BLNWQGOHH3V43LGSE3BUQCI2SY4TTO5M5GQNOCTMQ, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:10.830722215 (75h46m46.394045216s) [noise=true , address=104.36.82.5:5869 , piece_id=HAEHXXZASV3DHT3QN7NTZOS6XZKHNUWBKCHWJJWSTJKXAKJKKBAA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:12.738783450 (75h46m48.302106452s) [noise=true , address=77.90.31.12:4603 , piece_id=BMUIC6KNGEDL37436GLSXW5XOO6VYOPNOSFGQCYVHSENVI2PJ7KQ, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:15.460927713 (75h46m51.024250723s) [noise=true , address=104.36.82.52:4111 , piece_id=HB7QAWN6CGNIIT4HD2GZ7JDQP5NXIJ5I2USBM66YOKTYENNOWG5A, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:18.393817968 (75h46m53.95714096s) [noise=true , address=104.36.82.55:4224 , piece_id=WRVNGSJV5BX4GJERLXNPAS7J7BZ5WFTM7R5H257YRCG5SP2HIYMA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:19.549174143 (75h46m55.112497151s) [noise=true , address=104.36.82.56:4253 , piece_id=5PMVGLZETMXCQ77K3I6PXACRIMBGR2CV5JGUSU5KUMYFNXVGA2GA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:22.511944593 (75h46m58.075267584s) [noise=true , address=104.36.82.37:6663 , piece_id=X7LWYIIHJFU2T2PHYR3R5TNT22UUPHEHQWZH7ZJMLS4AH4PJ6AIA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:48.027197558 (75h47m23.590520562s) [noise=true , address=104.36.82.14:6232 , piece_id=A2ZLUUZKRTBJRJ267QSK36C3LFZV7FPAYDDAXGIWDTNEGNV6MRRA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:11:05.567692582 (75h47m41.131015574s) [noise=true , address=104.36.82.49:4982 , piece_id=UNLV5UDNWULEIPE4QSQHHZQPK3BQKFRB5T3H6GP7ZQ5QGSNK5V6A, seg_pos={0 0}, upload=60693]: Done uploading piece. err: ecclient: upload failed (node:1NxoYJXCXmTvQncbkXt6yH7mhEBBrzVdzhgL5J6W5TD4N3pkaG, address:104.36.82.49:4982): protocol: expected piece hash; context deadline exceeded; EOF Jun 21 20:11:05.567759198 (75h47m41.131082195s) [noise=true , address=104.36.82.54:4171 , piece_id=MUMABKARK4KKUKWW6XDXCQVT6O7SLFHGHE7COWC7QIYGARAA2D5A, seg_pos={0 0}, upload=60693]: Done uploading piece. err: ecclient: upload failed (node:125SMhDsEZruNu6LkmQn2M7R6WzvXTdk86DRsKf522u8BNiANq3, address:104.36.82.54:4171): protocol: expected piece hash; context deadline exceeded; EOF Jun 21 20:11:05.567869019 (75h47m41.131192033s) [noise=true , address=104.36.82.10:6087 , piece_id=5FQG7BPZC373PFB54HYXSG2SVKOBBBSGZCLC4XDJEYQFOLQY74MA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: ecclient: upload failed (node:1S9k5eejJcxkiG5pXjjBZm18uP8k5jCGfWyeW2QQxmuRU8ZzRA, address:104.36.82.10:6087): protocol: write tcp 159.223.143.39:38550->104.36.82.10:6087: use of closed network connection; write tcp 159.223.143.39:38550->104.36.82.10:6087: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:38550->104.36.82.10:6087: use of closed network connection Jun 21 20:11:05.567932150 (75h47m41.131255161s) [seg_pos={0 0}, upload=60693]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jun 21 20:11:05.593626175 (75h47m41.156949181s) [upload=60693]: Done uploading segments... Jun 21 20:11:05.593657274 (75h47m41.156980265s) [upload=60693]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 27.092240928s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [............................X...........X...................] duration: 24.287283946s FAIL (2025-06-21 20:26:43 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== pload=60693]: Starting upload Jun 21 20:10:05.602866542 (75h46m41.166189557s) [upload=60693]: Done starting upload Jun 21 20:10:05.603116545 (75h46m41.166439547s) [upload=60693]: Uploading segments... Jun 21 20:10:05.603495389 (75h46m41.166818397s) [upload=60693]: Got next segment. Inline: false Jun 21 20:10:05.603566566 (75h46m41.166889558s) [upload=60693]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 21 20:10:05.603573280 (75h46m41.166896286s) [upload=60693]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 21 20:10:05.639130162 (75h46m41.202453164s) [seg_pos={0 0}, upload=60693]: Begin upload segment... Jun 21 20:10:05.639231070 (75h46m41.202554057s) [seg_pos={0 0}, upload=60693]: Done begin upload segment. Jun 21 20:10:05.639292724 (75h46m41.202615728s) [noise=true , address=104.36.82.56:4253 , piece_id=5PMVGLZETMXCQ77K3I6PXACRIMBGR2CV5JGUSU5KUMYFNXVGA2GA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.639338066 (75h46m41.202661071s) [noise=true , address=104.36.82.53:4126 , piece_id=2MKRQJEPGDJKDM7VWY5M4M6NORILAIEYYATJ7HVZEOBOMXOI2LBA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.639416264 (75h46m41.202739267s) [noise=true , address=104.36.82.5:5869 , piece_id=HAEHXXZASV3DHT3QN7NTZOS6XZKHNUWBKCHWJJWSTJKXAKJKKBAA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.639873098 (75h46m41.203196115s) [noise=true , address=104.36.82.52:4111 , piece_id=HB7QAWN6CGNIIT4HD2GZ7JDQP5NXIJ5I2USBM66YOKTYENNOWG5A, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.639887296 (75h46m41.203210304s) [noise=true , address=104.36.82.10:6087 , piece_id=5FQG7BPZC373PFB54HYXSG2SVKOBBBSGZCLC4XDJEYQFOLQY74MA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.640266623 (75h46m41.203589616s) [noise=true , address=104.36.82.13:6196 , piece_id=BCHMQO5KKT5BLNWQGOHH3V43LGSE3BUQCI2SY4TTO5M5GQNOCTMQ, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.640352274 (75h46m41.203675286s) [noise=true , address=104.36.82.37:6663 , piece_id=X7LWYIIHJFU2T2PHYR3R5TNT22UUPHEHQWZH7ZJMLS4AH4PJ6AIA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.640659767 (75h46m41.203982779s) [noise=true , address=104.36.82.42:6807 , piece_id=ZBQZPXUDJ4VBCOSVKWAJ6SK6UD7SLPON7Y4SSCK2DQ5STFG4XCNQ, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.640961701 (75h46m41.20428471s) [noise=true , address=45.91.116.6:4982 , piece_id=VP45R6KXNH5U5SGUQUMAPIBY7KFDPBIR3S3VRIX27EAT6VLFBZ7Q, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.641159578 (75h46m41.204482589s) [noise=true , address=104.36.82.9:6039 , piece_id=I4AB6N27MYYUHWAZLJJUIBFBLOQFHDIM3YLYWPVRPNUV7VMYS2SA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.641588126 (75h46m41.204911136s) [noise=true , address=77.90.31.19:4853 , piece_id=VFEON5T2KSWM2KSLTVRZTXYL42JWENZYUEUPG3LMODGZWBMVGGPQ, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.641863297 (75h46m41.205186305s) [noise=true , address=77.90.31.12:4603 , piece_id=BMUIC6KNGEDL37436GLSXW5XOO6VYOPNOSFGQCYVHSENVI2PJ7KQ, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.642124427 (75h46m41.205447437s) [noise=true , address=104.36.82.49:4982 , piece_id=UNLV5UDNWULEIPE4QSQHHZQPK3BQKFRB5T3H6GP7ZQ5QGSNK5V6A, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.642285917 (75h46m41.205608925s) [noise=true , address=104.36.82.54:4171 , piece_id=MUMABKARK4KKUKWW6XDXCQVT6O7SLFHGHE7COWC7QIYGARAA2D5A, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.642656750 (75h46m41.205979758s) [noise=true , address=104.36.82.14:6232 , piece_id=A2ZLUUZKRTBJRJ267QSK36C3LFZV7FPAYDDAXGIWDTNEGNV6MRRA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.643565583 (75h46m41.206888597s) [noise=true , address=79.127.205.245:4231 , piece_id=EJK46XUUVI5NWOAQJEW4QIJG3KRXZHB4YOQUJRCS2MIOSWKGTNLQ, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.644620291 (75h46m41.207943303s) [noise=true , address=77.90.31.14:4670 , piece_id=7FNANMUKPM3N6YWRAOAIM4WJZJUTNF7ZBI6ZUBNEPGNGMFWOE24A, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.648842639 (75h46m41.212165631s) [noise=true , address=104.36.82.55:4224 , piece_id=WRVNGSJV5BX4GJERLXNPAS7J7BZ5WFTM7R5H257YRCG5SP2HIYMA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.825828239 (75h46m41.389151241s) [noise=true , address=104.36.82.53:4126 , piece_id=2MKRQJEPGDJKDM7VWY5M4M6NORILAIEYYATJ7HVZEOBOMXOI2LBA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: ecclient: upload failed (node:12dMWFrq2NMpnMzuPAphWerntC5kkrcnzRJZBA116uzJzyvk6fH, address:104.36.82.53:4126): protocol: storage node overloaded, request limit: 2000 Jun 21 20:10:05.834666043 (75h46m41.397989033s) [upload=60693]: Next returned nil segment Jun 21 20:10:05.834686295 (75h46m41.398009302s) [upload=60693]: Waiting for error group managing segments... Jun 21 20:10:05.877010488 (75h46m41.440333486s) [noise=true , address=45.91.116.6:4982 , piece_id=VP45R6KXNH5U5SGUQUMAPIBY7KFDPBIR3S3VRIX27EAT6VLFBZ7Q, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:05.886336964 (75h46m41.449659967s) [noise=true , address=104.36.82.9:6039 , piece_id=I4AB6N27MYYUHWAZLJJUIBFBLOQFHDIM3YLYWPVRPNUV7VMYS2SA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: ecclient: upload failed (node:12P11WVW9mVXw5ra2gZ3Rezon6wZi8GrN5CchQN3MiVU4ts3L6V, address:104.36.82.9:6039): protocol: storage node overloaded, request limit: 2000 Jun 21 20:10:08.064514542 (75h46m43.627837553s) [noise=true , address=77.90.31.14:4670 , piece_id=7FNANMUKPM3N6YWRAOAIM4WJZJUTNF7ZBI6ZUBNEPGNGMFWOE24A, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:08.099098391 (75h46m43.662421396s) [noise=true , address=79.127.205.245:4231 , piece_id=EJK46XUUVI5NWOAQJEW4QIJG3KRXZHB4YOQUJRCS2MIOSWKGTNLQ, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:08.489461749 (75h46m44.052784752s) [noise=true , address=104.36.82.42:6807 , piece_id=ZBQZPXUDJ4VBCOSVKWAJ6SK6UD7SLPON7Y4SSCK2DQ5STFG4XCNQ, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:08.650922093 (75h46m44.214245093s) [noise=true , address=77.90.31.19:4853 , piece_id=VFEON5T2KSWM2KSLTVRZTXYL42JWENZYUEUPG3LMODGZWBMVGGPQ, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:09.917746439 (75h46m45.481069441s) [noise=true , address=104.36.82.13:6196 , piece_id=BCHMQO5KKT5BLNWQGOHH3V43LGSE3BUQCI2SY4TTO5M5GQNOCTMQ, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:10.830722215 (75h46m46.394045216s) [noise=true , address=104.36.82.5:5869 , piece_id=HAEHXXZASV3DHT3QN7NTZOS6XZKHNUWBKCHWJJWSTJKXAKJKKBAA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:12.738783450 (75h46m48.302106452s) [noise=true , address=77.90.31.12:4603 , piece_id=BMUIC6KNGEDL37436GLSXW5XOO6VYOPNOSFGQCYVHSENVI2PJ7KQ, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:15.460927713 (75h46m51.024250723s) [noise=true , address=104.36.82.52:4111 , piece_id=HB7QAWN6CGNIIT4HD2GZ7JDQP5NXIJ5I2USBM66YOKTYENNOWG5A, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:18.393817968 (75h46m53.95714096s) [noise=true , address=104.36.82.55:4224 , piece_id=WRVNGSJV5BX4GJERLXNPAS7J7BZ5WFTM7R5H257YRCG5SP2HIYMA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:19.549174143 (75h46m55.112497151s) [noise=true , address=104.36.82.56:4253 , piece_id=5PMVGLZETMXCQ77K3I6PXACRIMBGR2CV5JGUSU5KUMYFNXVGA2GA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:22.511944593 (75h46m58.075267584s) [noise=true , address=104.36.82.37:6663 , piece_id=X7LWYIIHJFU2T2PHYR3R5TNT22UUPHEHQWZH7ZJMLS4AH4PJ6AIA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:48.027197558 (75h47m23.590520562s) [noise=true , address=104.36.82.14:6232 , piece_id=A2ZLUUZKRTBJRJ267QSK36C3LFZV7FPAYDDAXGIWDTNEGNV6MRRA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:11:05.567692582 (75h47m41.131015574s) [noise=true , address=104.36.82.49:4982 , piece_id=UNLV5UDNWULEIPE4QSQHHZQPK3BQKFRB5T3H6GP7ZQ5QGSNK5V6A, seg_pos={0 0}, upload=60693]: Done uploading piece. err: ecclient: upload failed (node:1NxoYJXCXmTvQncbkXt6yH7mhEBBrzVdzhgL5J6W5TD4N3pkaG, address:104.36.82.49:4982): protocol: expected piece hash; context deadline exceeded; EOF Jun 21 20:11:05.567759198 (75h47m41.131082195s) [noise=true , address=104.36.82.54:4171 , piece_id=MUMABKARK4KKUKWW6XDXCQVT6O7SLFHGHE7COWC7QIYGARAA2D5A, seg_pos={0 0}, upload=60693]: Done uploading piece. err: ecclient: upload failed (node:125SMhDsEZruNu6LkmQn2M7R6WzvXTdk86DRsKf522u8BNiANq3, address:104.36.82.54:4171): protocol: expected piece hash; context deadline exceeded; EOF Jun 21 20:11:05.567869019 (75h47m41.131192033s) [noise=true , address=104.36.82.10:6087 , piece_id=5FQG7BPZC373PFB54HYXSG2SVKOBBBSGZCLC4XDJEYQFOLQY74MA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: ecclient: upload failed (node:1S9k5eejJcxkiG5pXjjBZm18uP8k5jCGfWyeW2QQxmuRU8ZzRA, address:104.36.82.10:6087): protocol: write tcp 159.223.143.39:38550->104.36.82.10:6087: use of closed network connection; write tcp 159.223.143.39:38550->104.36.82.10:6087: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:38550->104.36.82.10:6087: use of closed network connection Jun 21 20:11:05.567932150 (75h47m41.131255161s) [seg_pos={0 0}, upload=60693]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jun 21 20:11:05.593626175 (75h47m41.156949181s) [upload=60693]: Done uploading segments... Jun 21 20:11:05.593657274 (75h47m41.156980265s) [upload=60693]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 24.287313277s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [.............................X...........X..................] duration: 19.412019074s FAIL (2025-06-21 20:25:39 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== pload=60693]: Starting upload Jun 21 20:10:05.602866542 (75h46m41.166189557s) [upload=60693]: Done starting upload Jun 21 20:10:05.603116545 (75h46m41.166439547s) [upload=60693]: Uploading segments... Jun 21 20:10:05.603495389 (75h46m41.166818397s) [upload=60693]: Got next segment. Inline: false Jun 21 20:10:05.603566566 (75h46m41.166889558s) [upload=60693]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 21 20:10:05.603573280 (75h46m41.166896286s) [upload=60693]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 21 20:10:05.639130162 (75h46m41.202453164s) [seg_pos={0 0}, upload=60693]: Begin upload segment... Jun 21 20:10:05.639231070 (75h46m41.202554057s) [seg_pos={0 0}, upload=60693]: Done begin upload segment. Jun 21 20:10:05.639292724 (75h46m41.202615728s) [noise=true , address=104.36.82.56:4253 , piece_id=5PMVGLZETMXCQ77K3I6PXACRIMBGR2CV5JGUSU5KUMYFNXVGA2GA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.639338066 (75h46m41.202661071s) [noise=true , address=104.36.82.53:4126 , piece_id=2MKRQJEPGDJKDM7VWY5M4M6NORILAIEYYATJ7HVZEOBOMXOI2LBA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.639416264 (75h46m41.202739267s) [noise=true , address=104.36.82.5:5869 , piece_id=HAEHXXZASV3DHT3QN7NTZOS6XZKHNUWBKCHWJJWSTJKXAKJKKBAA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.639873098 (75h46m41.203196115s) [noise=true , address=104.36.82.52:4111 , piece_id=HB7QAWN6CGNIIT4HD2GZ7JDQP5NXIJ5I2USBM66YOKTYENNOWG5A, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.639887296 (75h46m41.203210304s) [noise=true , address=104.36.82.10:6087 , piece_id=5FQG7BPZC373PFB54HYXSG2SVKOBBBSGZCLC4XDJEYQFOLQY74MA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.640266623 (75h46m41.203589616s) [noise=true , address=104.36.82.13:6196 , piece_id=BCHMQO5KKT5BLNWQGOHH3V43LGSE3BUQCI2SY4TTO5M5GQNOCTMQ, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.640352274 (75h46m41.203675286s) [noise=true , address=104.36.82.37:6663 , piece_id=X7LWYIIHJFU2T2PHYR3R5TNT22UUPHEHQWZH7ZJMLS4AH4PJ6AIA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.640659767 (75h46m41.203982779s) [noise=true , address=104.36.82.42:6807 , piece_id=ZBQZPXUDJ4VBCOSVKWAJ6SK6UD7SLPON7Y4SSCK2DQ5STFG4XCNQ, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.640961701 (75h46m41.20428471s) [noise=true , address=45.91.116.6:4982 , piece_id=VP45R6KXNH5U5SGUQUMAPIBY7KFDPBIR3S3VRIX27EAT6VLFBZ7Q, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.641159578 (75h46m41.204482589s) [noise=true , address=104.36.82.9:6039 , piece_id=I4AB6N27MYYUHWAZLJJUIBFBLOQFHDIM3YLYWPVRPNUV7VMYS2SA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.641588126 (75h46m41.204911136s) [noise=true , address=77.90.31.19:4853 , piece_id=VFEON5T2KSWM2KSLTVRZTXYL42JWENZYUEUPG3LMODGZWBMVGGPQ, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.641863297 (75h46m41.205186305s) [noise=true , address=77.90.31.12:4603 , piece_id=BMUIC6KNGEDL37436GLSXW5XOO6VYOPNOSFGQCYVHSENVI2PJ7KQ, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.642124427 (75h46m41.205447437s) [noise=true , address=104.36.82.49:4982 , piece_id=UNLV5UDNWULEIPE4QSQHHZQPK3BQKFRB5T3H6GP7ZQ5QGSNK5V6A, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.642285917 (75h46m41.205608925s) [noise=true , address=104.36.82.54:4171 , piece_id=MUMABKARK4KKUKWW6XDXCQVT6O7SLFHGHE7COWC7QIYGARAA2D5A, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.642656750 (75h46m41.205979758s) [noise=true , address=104.36.82.14:6232 , piece_id=A2ZLUUZKRTBJRJ267QSK36C3LFZV7FPAYDDAXGIWDTNEGNV6MRRA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.643565583 (75h46m41.206888597s) [noise=true , address=79.127.205.245:4231 , piece_id=EJK46XUUVI5NWOAQJEW4QIJG3KRXZHB4YOQUJRCS2MIOSWKGTNLQ, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.644620291 (75h46m41.207943303s) [noise=true , address=77.90.31.14:4670 , piece_id=7FNANMUKPM3N6YWRAOAIM4WJZJUTNF7ZBI6ZUBNEPGNGMFWOE24A, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.648842639 (75h46m41.212165631s) [noise=true , address=104.36.82.55:4224 , piece_id=WRVNGSJV5BX4GJERLXNPAS7J7BZ5WFTM7R5H257YRCG5SP2HIYMA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.825828239 (75h46m41.389151241s) [noise=true , address=104.36.82.53:4126 , piece_id=2MKRQJEPGDJKDM7VWY5M4M6NORILAIEYYATJ7HVZEOBOMXOI2LBA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: ecclient: upload failed (node:12dMWFrq2NMpnMzuPAphWerntC5kkrcnzRJZBA116uzJzyvk6fH, address:104.36.82.53:4126): protocol: storage node overloaded, request limit: 2000 Jun 21 20:10:05.834666043 (75h46m41.397989033s) [upload=60693]: Next returned nil segment Jun 21 20:10:05.834686295 (75h46m41.398009302s) [upload=60693]: Waiting for error group managing segments... Jun 21 20:10:05.877010488 (75h46m41.440333486s) [noise=true , address=45.91.116.6:4982 , piece_id=VP45R6KXNH5U5SGUQUMAPIBY7KFDPBIR3S3VRIX27EAT6VLFBZ7Q, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:05.886336964 (75h46m41.449659967s) [noise=true , address=104.36.82.9:6039 , piece_id=I4AB6N27MYYUHWAZLJJUIBFBLOQFHDIM3YLYWPVRPNUV7VMYS2SA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: ecclient: upload failed (node:12P11WVW9mVXw5ra2gZ3Rezon6wZi8GrN5CchQN3MiVU4ts3L6V, address:104.36.82.9:6039): protocol: storage node overloaded, request limit: 2000 Jun 21 20:10:08.064514542 (75h46m43.627837553s) [noise=true , address=77.90.31.14:4670 , piece_id=7FNANMUKPM3N6YWRAOAIM4WJZJUTNF7ZBI6ZUBNEPGNGMFWOE24A, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:08.099098391 (75h46m43.662421396s) [noise=true , address=79.127.205.245:4231 , piece_id=EJK46XUUVI5NWOAQJEW4QIJG3KRXZHB4YOQUJRCS2MIOSWKGTNLQ, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:08.489461749 (75h46m44.052784752s) [noise=true , address=104.36.82.42:6807 , piece_id=ZBQZPXUDJ4VBCOSVKWAJ6SK6UD7SLPON7Y4SSCK2DQ5STFG4XCNQ, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:08.650922093 (75h46m44.214245093s) [noise=true , address=77.90.31.19:4853 , piece_id=VFEON5T2KSWM2KSLTVRZTXYL42JWENZYUEUPG3LMODGZWBMVGGPQ, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:09.917746439 (75h46m45.481069441s) [noise=true , address=104.36.82.13:6196 , piece_id=BCHMQO5KKT5BLNWQGOHH3V43LGSE3BUQCI2SY4TTO5M5GQNOCTMQ, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:10.830722215 (75h46m46.394045216s) [noise=true , address=104.36.82.5:5869 , piece_id=HAEHXXZASV3DHT3QN7NTZOS6XZKHNUWBKCHWJJWSTJKXAKJKKBAA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:12.738783450 (75h46m48.302106452s) [noise=true , address=77.90.31.12:4603 , piece_id=BMUIC6KNGEDL37436GLSXW5XOO6VYOPNOSFGQCYVHSENVI2PJ7KQ, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:15.460927713 (75h46m51.024250723s) [noise=true , address=104.36.82.52:4111 , piece_id=HB7QAWN6CGNIIT4HD2GZ7JDQP5NXIJ5I2USBM66YOKTYENNOWG5A, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:18.393817968 (75h46m53.95714096s) [noise=true , address=104.36.82.55:4224 , piece_id=WRVNGSJV5BX4GJERLXNPAS7J7BZ5WFTM7R5H257YRCG5SP2HIYMA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:19.549174143 (75h46m55.112497151s) [noise=true , address=104.36.82.56:4253 , piece_id=5PMVGLZETMXCQ77K3I6PXACRIMBGR2CV5JGUSU5KUMYFNXVGA2GA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:22.511944593 (75h46m58.075267584s) [noise=true , address=104.36.82.37:6663 , piece_id=X7LWYIIHJFU2T2PHYR3R5TNT22UUPHEHQWZH7ZJMLS4AH4PJ6AIA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:48.027197558 (75h47m23.590520562s) [noise=true , address=104.36.82.14:6232 , piece_id=A2ZLUUZKRTBJRJ267QSK36C3LFZV7FPAYDDAXGIWDTNEGNV6MRRA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:11:05.567692582 (75h47m41.131015574s) [noise=true , address=104.36.82.49:4982 , piece_id=UNLV5UDNWULEIPE4QSQHHZQPK3BQKFRB5T3H6GP7ZQ5QGSNK5V6A, seg_pos={0 0}, upload=60693]: Done uploading piece. err: ecclient: upload failed (node:1NxoYJXCXmTvQncbkXt6yH7mhEBBrzVdzhgL5J6W5TD4N3pkaG, address:104.36.82.49:4982): protocol: expected piece hash; context deadline exceeded; EOF Jun 21 20:11:05.567759198 (75h47m41.131082195s) [noise=true , address=104.36.82.54:4171 , piece_id=MUMABKARK4KKUKWW6XDXCQVT6O7SLFHGHE7COWC7QIYGARAA2D5A, seg_pos={0 0}, upload=60693]: Done uploading piece. err: ecclient: upload failed (node:125SMhDsEZruNu6LkmQn2M7R6WzvXTdk86DRsKf522u8BNiANq3, address:104.36.82.54:4171): protocol: expected piece hash; context deadline exceeded; EOF Jun 21 20:11:05.567869019 (75h47m41.131192033s) [noise=true , address=104.36.82.10:6087 , piece_id=5FQG7BPZC373PFB54HYXSG2SVKOBBBSGZCLC4XDJEYQFOLQY74MA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: ecclient: upload failed (node:1S9k5eejJcxkiG5pXjjBZm18uP8k5jCGfWyeW2QQxmuRU8ZzRA, address:104.36.82.10:6087): protocol: write tcp 159.223.143.39:38550->104.36.82.10:6087: use of closed network connection; write tcp 159.223.143.39:38550->104.36.82.10:6087: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:38550->104.36.82.10:6087: use of closed network connection Jun 21 20:11:05.567932150 (75h47m41.131255161s) [seg_pos={0 0}, upload=60693]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jun 21 20:11:05.593626175 (75h47m41.156949181s) [upload=60693]: Done uploading segments... Jun 21 20:11:05.593657274 (75h47m41.156980265s) [upload=60693]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 19.412066965s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [..............................X...........X.................] duration: 15.377422554s FAIL (2025-06-21 20:24:39 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== pload=60693]: Starting upload Jun 21 20:10:05.602866542 (75h46m41.166189557s) [upload=60693]: Done starting upload Jun 21 20:10:05.603116545 (75h46m41.166439547s) [upload=60693]: Uploading segments... Jun 21 20:10:05.603495389 (75h46m41.166818397s) [upload=60693]: Got next segment. Inline: false Jun 21 20:10:05.603566566 (75h46m41.166889558s) [upload=60693]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 21 20:10:05.603573280 (75h46m41.166896286s) [upload=60693]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 21 20:10:05.639130162 (75h46m41.202453164s) [seg_pos={0 0}, upload=60693]: Begin upload segment... Jun 21 20:10:05.639231070 (75h46m41.202554057s) [seg_pos={0 0}, upload=60693]: Done begin upload segment. Jun 21 20:10:05.639292724 (75h46m41.202615728s) [noise=true , address=104.36.82.56:4253 , piece_id=5PMVGLZETMXCQ77K3I6PXACRIMBGR2CV5JGUSU5KUMYFNXVGA2GA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.639338066 (75h46m41.202661071s) [noise=true , address=104.36.82.53:4126 , piece_id=2MKRQJEPGDJKDM7VWY5M4M6NORILAIEYYATJ7HVZEOBOMXOI2LBA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.639416264 (75h46m41.202739267s) [noise=true , address=104.36.82.5:5869 , piece_id=HAEHXXZASV3DHT3QN7NTZOS6XZKHNUWBKCHWJJWSTJKXAKJKKBAA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.639873098 (75h46m41.203196115s) [noise=true , address=104.36.82.52:4111 , piece_id=HB7QAWN6CGNIIT4HD2GZ7JDQP5NXIJ5I2USBM66YOKTYENNOWG5A, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.639887296 (75h46m41.203210304s) [noise=true , address=104.36.82.10:6087 , piece_id=5FQG7BPZC373PFB54HYXSG2SVKOBBBSGZCLC4XDJEYQFOLQY74MA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.640266623 (75h46m41.203589616s) [noise=true , address=104.36.82.13:6196 , piece_id=BCHMQO5KKT5BLNWQGOHH3V43LGSE3BUQCI2SY4TTO5M5GQNOCTMQ, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.640352274 (75h46m41.203675286s) [noise=true , address=104.36.82.37:6663 , piece_id=X7LWYIIHJFU2T2PHYR3R5TNT22UUPHEHQWZH7ZJMLS4AH4PJ6AIA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.640659767 (75h46m41.203982779s) [noise=true , address=104.36.82.42:6807 , piece_id=ZBQZPXUDJ4VBCOSVKWAJ6SK6UD7SLPON7Y4SSCK2DQ5STFG4XCNQ, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.640961701 (75h46m41.20428471s) [noise=true , address=45.91.116.6:4982 , piece_id=VP45R6KXNH5U5SGUQUMAPIBY7KFDPBIR3S3VRIX27EAT6VLFBZ7Q, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.641159578 (75h46m41.204482589s) [noise=true , address=104.36.82.9:6039 , piece_id=I4AB6N27MYYUHWAZLJJUIBFBLOQFHDIM3YLYWPVRPNUV7VMYS2SA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.641588126 (75h46m41.204911136s) [noise=true , address=77.90.31.19:4853 , piece_id=VFEON5T2KSWM2KSLTVRZTXYL42JWENZYUEUPG3LMODGZWBMVGGPQ, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.641863297 (75h46m41.205186305s) [noise=true , address=77.90.31.12:4603 , piece_id=BMUIC6KNGEDL37436GLSXW5XOO6VYOPNOSFGQCYVHSENVI2PJ7KQ, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.642124427 (75h46m41.205447437s) [noise=true , address=104.36.82.49:4982 , piece_id=UNLV5UDNWULEIPE4QSQHHZQPK3BQKFRB5T3H6GP7ZQ5QGSNK5V6A, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.642285917 (75h46m41.205608925s) [noise=true , address=104.36.82.54:4171 , piece_id=MUMABKARK4KKUKWW6XDXCQVT6O7SLFHGHE7COWC7QIYGARAA2D5A, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.642656750 (75h46m41.205979758s) [noise=true , address=104.36.82.14:6232 , piece_id=A2ZLUUZKRTBJRJ267QSK36C3LFZV7FPAYDDAXGIWDTNEGNV6MRRA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.643565583 (75h46m41.206888597s) [noise=true , address=79.127.205.245:4231 , piece_id=EJK46XUUVI5NWOAQJEW4QIJG3KRXZHB4YOQUJRCS2MIOSWKGTNLQ, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.644620291 (75h46m41.207943303s) [noise=true , address=77.90.31.14:4670 , piece_id=7FNANMUKPM3N6YWRAOAIM4WJZJUTNF7ZBI6ZUBNEPGNGMFWOE24A, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.648842639 (75h46m41.212165631s) [noise=true , address=104.36.82.55:4224 , piece_id=WRVNGSJV5BX4GJERLXNPAS7J7BZ5WFTM7R5H257YRCG5SP2HIYMA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.825828239 (75h46m41.389151241s) [noise=true , address=104.36.82.53:4126 , piece_id=2MKRQJEPGDJKDM7VWY5M4M6NORILAIEYYATJ7HVZEOBOMXOI2LBA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: ecclient: upload failed (node:12dMWFrq2NMpnMzuPAphWerntC5kkrcnzRJZBA116uzJzyvk6fH, address:104.36.82.53:4126): protocol: storage node overloaded, request limit: 2000 Jun 21 20:10:05.834666043 (75h46m41.397989033s) [upload=60693]: Next returned nil segment Jun 21 20:10:05.834686295 (75h46m41.398009302s) [upload=60693]: Waiting for error group managing segments... Jun 21 20:10:05.877010488 (75h46m41.440333486s) [noise=true , address=45.91.116.6:4982 , piece_id=VP45R6KXNH5U5SGUQUMAPIBY7KFDPBIR3S3VRIX27EAT6VLFBZ7Q, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:05.886336964 (75h46m41.449659967s) [noise=true , address=104.36.82.9:6039 , piece_id=I4AB6N27MYYUHWAZLJJUIBFBLOQFHDIM3YLYWPVRPNUV7VMYS2SA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: ecclient: upload failed (node:12P11WVW9mVXw5ra2gZ3Rezon6wZi8GrN5CchQN3MiVU4ts3L6V, address:104.36.82.9:6039): protocol: storage node overloaded, request limit: 2000 Jun 21 20:10:08.064514542 (75h46m43.627837553s) [noise=true , address=77.90.31.14:4670 , piece_id=7FNANMUKPM3N6YWRAOAIM4WJZJUTNF7ZBI6ZUBNEPGNGMFWOE24A, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:08.099098391 (75h46m43.662421396s) [noise=true , address=79.127.205.245:4231 , piece_id=EJK46XUUVI5NWOAQJEW4QIJG3KRXZHB4YOQUJRCS2MIOSWKGTNLQ, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:08.489461749 (75h46m44.052784752s) [noise=true , address=104.36.82.42:6807 , piece_id=ZBQZPXUDJ4VBCOSVKWAJ6SK6UD7SLPON7Y4SSCK2DQ5STFG4XCNQ, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:08.650922093 (75h46m44.214245093s) [noise=true , address=77.90.31.19:4853 , piece_id=VFEON5T2KSWM2KSLTVRZTXYL42JWENZYUEUPG3LMODGZWBMVGGPQ, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:09.917746439 (75h46m45.481069441s) [noise=true , address=104.36.82.13:6196 , piece_id=BCHMQO5KKT5BLNWQGOHH3V43LGSE3BUQCI2SY4TTO5M5GQNOCTMQ, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:10.830722215 (75h46m46.394045216s) [noise=true , address=104.36.82.5:5869 , piece_id=HAEHXXZASV3DHT3QN7NTZOS6XZKHNUWBKCHWJJWSTJKXAKJKKBAA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:12.738783450 (75h46m48.302106452s) [noise=true , address=77.90.31.12:4603 , piece_id=BMUIC6KNGEDL37436GLSXW5XOO6VYOPNOSFGQCYVHSENVI2PJ7KQ, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:15.460927713 (75h46m51.024250723s) [noise=true , address=104.36.82.52:4111 , piece_id=HB7QAWN6CGNIIT4HD2GZ7JDQP5NXIJ5I2USBM66YOKTYENNOWG5A, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:18.393817968 (75h46m53.95714096s) [noise=true , address=104.36.82.55:4224 , piece_id=WRVNGSJV5BX4GJERLXNPAS7J7BZ5WFTM7R5H257YRCG5SP2HIYMA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:19.549174143 (75h46m55.112497151s) [noise=true , address=104.36.82.56:4253 , piece_id=5PMVGLZETMXCQ77K3I6PXACRIMBGR2CV5JGUSU5KUMYFNXVGA2GA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:22.511944593 (75h46m58.075267584s) [noise=true , address=104.36.82.37:6663 , piece_id=X7LWYIIHJFU2T2PHYR3R5TNT22UUPHEHQWZH7ZJMLS4AH4PJ6AIA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:48.027197558 (75h47m23.590520562s) [noise=true , address=104.36.82.14:6232 , piece_id=A2ZLUUZKRTBJRJ267QSK36C3LFZV7FPAYDDAXGIWDTNEGNV6MRRA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:11:05.567692582 (75h47m41.131015574s) [noise=true , address=104.36.82.49:4982 , piece_id=UNLV5UDNWULEIPE4QSQHHZQPK3BQKFRB5T3H6GP7ZQ5QGSNK5V6A, seg_pos={0 0}, upload=60693]: Done uploading piece. err: ecclient: upload failed (node:1NxoYJXCXmTvQncbkXt6yH7mhEBBrzVdzhgL5J6W5TD4N3pkaG, address:104.36.82.49:4982): protocol: expected piece hash; context deadline exceeded; EOF Jun 21 20:11:05.567759198 (75h47m41.131082195s) [noise=true , address=104.36.82.54:4171 , piece_id=MUMABKARK4KKUKWW6XDXCQVT6O7SLFHGHE7COWC7QIYGARAA2D5A, seg_pos={0 0}, upload=60693]: Done uploading piece. err: ecclient: upload failed (node:125SMhDsEZruNu6LkmQn2M7R6WzvXTdk86DRsKf522u8BNiANq3, address:104.36.82.54:4171): protocol: expected piece hash; context deadline exceeded; EOF Jun 21 20:11:05.567869019 (75h47m41.131192033s) [noise=true , address=104.36.82.10:6087 , piece_id=5FQG7BPZC373PFB54HYXSG2SVKOBBBSGZCLC4XDJEYQFOLQY74MA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: ecclient: upload failed (node:1S9k5eejJcxkiG5pXjjBZm18uP8k5jCGfWyeW2QQxmuRU8ZzRA, address:104.36.82.10:6087): protocol: write tcp 159.223.143.39:38550->104.36.82.10:6087: use of closed network connection; write tcp 159.223.143.39:38550->104.36.82.10:6087: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:38550->104.36.82.10:6087: use of closed network connection Jun 21 20:11:05.567932150 (75h47m41.131255161s) [seg_pos={0 0}, upload=60693]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jun 21 20:11:05.593626175 (75h47m41.156949181s) [upload=60693]: Done uploading segments... Jun 21 20:11:05.593657274 (75h47m41.156980265s) [upload=60693]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 15.377446313s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [...............................X...........X................] duration: 18.882619453s FAIL (2025-06-21 20:23:28 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== pload=60693]: Starting upload Jun 21 20:10:05.602866542 (75h46m41.166189557s) [upload=60693]: Done starting upload Jun 21 20:10:05.603116545 (75h46m41.166439547s) [upload=60693]: Uploading segments... Jun 21 20:10:05.603495389 (75h46m41.166818397s) [upload=60693]: Got next segment. Inline: false Jun 21 20:10:05.603566566 (75h46m41.166889558s) [upload=60693]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 21 20:10:05.603573280 (75h46m41.166896286s) [upload=60693]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 21 20:10:05.639130162 (75h46m41.202453164s) [seg_pos={0 0}, upload=60693]: Begin upload segment... Jun 21 20:10:05.639231070 (75h46m41.202554057s) [seg_pos={0 0}, upload=60693]: Done begin upload segment. Jun 21 20:10:05.639292724 (75h46m41.202615728s) [noise=true , address=104.36.82.56:4253 , piece_id=5PMVGLZETMXCQ77K3I6PXACRIMBGR2CV5JGUSU5KUMYFNXVGA2GA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.639338066 (75h46m41.202661071s) [noise=true , address=104.36.82.53:4126 , piece_id=2MKRQJEPGDJKDM7VWY5M4M6NORILAIEYYATJ7HVZEOBOMXOI2LBA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.639416264 (75h46m41.202739267s) [noise=true , address=104.36.82.5:5869 , piece_id=HAEHXXZASV3DHT3QN7NTZOS6XZKHNUWBKCHWJJWSTJKXAKJKKBAA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.639873098 (75h46m41.203196115s) [noise=true , address=104.36.82.52:4111 , piece_id=HB7QAWN6CGNIIT4HD2GZ7JDQP5NXIJ5I2USBM66YOKTYENNOWG5A, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.639887296 (75h46m41.203210304s) [noise=true , address=104.36.82.10:6087 , piece_id=5FQG7BPZC373PFB54HYXSG2SVKOBBBSGZCLC4XDJEYQFOLQY74MA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.640266623 (75h46m41.203589616s) [noise=true , address=104.36.82.13:6196 , piece_id=BCHMQO5KKT5BLNWQGOHH3V43LGSE3BUQCI2SY4TTO5M5GQNOCTMQ, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.640352274 (75h46m41.203675286s) [noise=true , address=104.36.82.37:6663 , piece_id=X7LWYIIHJFU2T2PHYR3R5TNT22UUPHEHQWZH7ZJMLS4AH4PJ6AIA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.640659767 (75h46m41.203982779s) [noise=true , address=104.36.82.42:6807 , piece_id=ZBQZPXUDJ4VBCOSVKWAJ6SK6UD7SLPON7Y4SSCK2DQ5STFG4XCNQ, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.640961701 (75h46m41.20428471s) [noise=true , address=45.91.116.6:4982 , piece_id=VP45R6KXNH5U5SGUQUMAPIBY7KFDPBIR3S3VRIX27EAT6VLFBZ7Q, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.641159578 (75h46m41.204482589s) [noise=true , address=104.36.82.9:6039 , piece_id=I4AB6N27MYYUHWAZLJJUIBFBLOQFHDIM3YLYWPVRPNUV7VMYS2SA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.641588126 (75h46m41.204911136s) [noise=true , address=77.90.31.19:4853 , piece_id=VFEON5T2KSWM2KSLTVRZTXYL42JWENZYUEUPG3LMODGZWBMVGGPQ, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.641863297 (75h46m41.205186305s) [noise=true , address=77.90.31.12:4603 , piece_id=BMUIC6KNGEDL37436GLSXW5XOO6VYOPNOSFGQCYVHSENVI2PJ7KQ, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.642124427 (75h46m41.205447437s) [noise=true , address=104.36.82.49:4982 , piece_id=UNLV5UDNWULEIPE4QSQHHZQPK3BQKFRB5T3H6GP7ZQ5QGSNK5V6A, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.642285917 (75h46m41.205608925s) [noise=true , address=104.36.82.54:4171 , piece_id=MUMABKARK4KKUKWW6XDXCQVT6O7SLFHGHE7COWC7QIYGARAA2D5A, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.642656750 (75h46m41.205979758s) [noise=true , address=104.36.82.14:6232 , piece_id=A2ZLUUZKRTBJRJ267QSK36C3LFZV7FPAYDDAXGIWDTNEGNV6MRRA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.643565583 (75h46m41.206888597s) [noise=true , address=79.127.205.245:4231 , piece_id=EJK46XUUVI5NWOAQJEW4QIJG3KRXZHB4YOQUJRCS2MIOSWKGTNLQ, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.644620291 (75h46m41.207943303s) [noise=true , address=77.90.31.14:4670 , piece_id=7FNANMUKPM3N6YWRAOAIM4WJZJUTNF7ZBI6ZUBNEPGNGMFWOE24A, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.648842639 (75h46m41.212165631s) [noise=true , address=104.36.82.55:4224 , piece_id=WRVNGSJV5BX4GJERLXNPAS7J7BZ5WFTM7R5H257YRCG5SP2HIYMA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.825828239 (75h46m41.389151241s) [noise=true , address=104.36.82.53:4126 , piece_id=2MKRQJEPGDJKDM7VWY5M4M6NORILAIEYYATJ7HVZEOBOMXOI2LBA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: ecclient: upload failed (node:12dMWFrq2NMpnMzuPAphWerntC5kkrcnzRJZBA116uzJzyvk6fH, address:104.36.82.53:4126): protocol: storage node overloaded, request limit: 2000 Jun 21 20:10:05.834666043 (75h46m41.397989033s) [upload=60693]: Next returned nil segment Jun 21 20:10:05.834686295 (75h46m41.398009302s) [upload=60693]: Waiting for error group managing segments... Jun 21 20:10:05.877010488 (75h46m41.440333486s) [noise=true , address=45.91.116.6:4982 , piece_id=VP45R6KXNH5U5SGUQUMAPIBY7KFDPBIR3S3VRIX27EAT6VLFBZ7Q, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:05.886336964 (75h46m41.449659967s) [noise=true , address=104.36.82.9:6039 , piece_id=I4AB6N27MYYUHWAZLJJUIBFBLOQFHDIM3YLYWPVRPNUV7VMYS2SA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: ecclient: upload failed (node:12P11WVW9mVXw5ra2gZ3Rezon6wZi8GrN5CchQN3MiVU4ts3L6V, address:104.36.82.9:6039): protocol: storage node overloaded, request limit: 2000 Jun 21 20:10:08.064514542 (75h46m43.627837553s) [noise=true , address=77.90.31.14:4670 , piece_id=7FNANMUKPM3N6YWRAOAIM4WJZJUTNF7ZBI6ZUBNEPGNGMFWOE24A, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:08.099098391 (75h46m43.662421396s) [noise=true , address=79.127.205.245:4231 , piece_id=EJK46XUUVI5NWOAQJEW4QIJG3KRXZHB4YOQUJRCS2MIOSWKGTNLQ, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:08.489461749 (75h46m44.052784752s) [noise=true , address=104.36.82.42:6807 , piece_id=ZBQZPXUDJ4VBCOSVKWAJ6SK6UD7SLPON7Y4SSCK2DQ5STFG4XCNQ, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:08.650922093 (75h46m44.214245093s) [noise=true , address=77.90.31.19:4853 , piece_id=VFEON5T2KSWM2KSLTVRZTXYL42JWENZYUEUPG3LMODGZWBMVGGPQ, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:09.917746439 (75h46m45.481069441s) [noise=true , address=104.36.82.13:6196 , piece_id=BCHMQO5KKT5BLNWQGOHH3V43LGSE3BUQCI2SY4TTO5M5GQNOCTMQ, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:10.830722215 (75h46m46.394045216s) [noise=true , address=104.36.82.5:5869 , piece_id=HAEHXXZASV3DHT3QN7NTZOS6XZKHNUWBKCHWJJWSTJKXAKJKKBAA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:12.738783450 (75h46m48.302106452s) [noise=true , address=77.90.31.12:4603 , piece_id=BMUIC6KNGEDL37436GLSXW5XOO6VYOPNOSFGQCYVHSENVI2PJ7KQ, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:15.460927713 (75h46m51.024250723s) [noise=true , address=104.36.82.52:4111 , piece_id=HB7QAWN6CGNIIT4HD2GZ7JDQP5NXIJ5I2USBM66YOKTYENNOWG5A, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:18.393817968 (75h46m53.95714096s) [noise=true , address=104.36.82.55:4224 , piece_id=WRVNGSJV5BX4GJERLXNPAS7J7BZ5WFTM7R5H257YRCG5SP2HIYMA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:19.549174143 (75h46m55.112497151s) [noise=true , address=104.36.82.56:4253 , piece_id=5PMVGLZETMXCQ77K3I6PXACRIMBGR2CV5JGUSU5KUMYFNXVGA2GA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:22.511944593 (75h46m58.075267584s) [noise=true , address=104.36.82.37:6663 , piece_id=X7LWYIIHJFU2T2PHYR3R5TNT22UUPHEHQWZH7ZJMLS4AH4PJ6AIA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:48.027197558 (75h47m23.590520562s) [noise=true , address=104.36.82.14:6232 , piece_id=A2ZLUUZKRTBJRJ267QSK36C3LFZV7FPAYDDAXGIWDTNEGNV6MRRA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:11:05.567692582 (75h47m41.131015574s) [noise=true , address=104.36.82.49:4982 , piece_id=UNLV5UDNWULEIPE4QSQHHZQPK3BQKFRB5T3H6GP7ZQ5QGSNK5V6A, seg_pos={0 0}, upload=60693]: Done uploading piece. err: ecclient: upload failed (node:1NxoYJXCXmTvQncbkXt6yH7mhEBBrzVdzhgL5J6W5TD4N3pkaG, address:104.36.82.49:4982): protocol: expected piece hash; context deadline exceeded; EOF Jun 21 20:11:05.567759198 (75h47m41.131082195s) [noise=true , address=104.36.82.54:4171 , piece_id=MUMABKARK4KKUKWW6XDXCQVT6O7SLFHGHE7COWC7QIYGARAA2D5A, seg_pos={0 0}, upload=60693]: Done uploading piece. err: ecclient: upload failed (node:125SMhDsEZruNu6LkmQn2M7R6WzvXTdk86DRsKf522u8BNiANq3, address:104.36.82.54:4171): protocol: expected piece hash; context deadline exceeded; EOF Jun 21 20:11:05.567869019 (75h47m41.131192033s) [noise=true , address=104.36.82.10:6087 , piece_id=5FQG7BPZC373PFB54HYXSG2SVKOBBBSGZCLC4XDJEYQFOLQY74MA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: ecclient: upload failed (node:1S9k5eejJcxkiG5pXjjBZm18uP8k5jCGfWyeW2QQxmuRU8ZzRA, address:104.36.82.10:6087): protocol: write tcp 159.223.143.39:38550->104.36.82.10:6087: use of closed network connection; write tcp 159.223.143.39:38550->104.36.82.10:6087: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:38550->104.36.82.10:6087: use of closed network connection Jun 21 20:11:05.567932150 (75h47m41.131255161s) [seg_pos={0 0}, upload=60693]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jun 21 20:11:05.593626175 (75h47m41.156949181s) [upload=60693]: Done uploading segments... Jun 21 20:11:05.593657274 (75h47m41.156980265s) [upload=60693]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 18.882692674s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [................................X...........X...............] duration: 22.256548969s FAIL (2025-06-21 20:22:44 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== pload=60693]: Starting upload Jun 21 20:10:05.602866542 (75h46m41.166189557s) [upload=60693]: Done starting upload Jun 21 20:10:05.603116545 (75h46m41.166439547s) [upload=60693]: Uploading segments... Jun 21 20:10:05.603495389 (75h46m41.166818397s) [upload=60693]: Got next segment. Inline: false Jun 21 20:10:05.603566566 (75h46m41.166889558s) [upload=60693]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 21 20:10:05.603573280 (75h46m41.166896286s) [upload=60693]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 21 20:10:05.639130162 (75h46m41.202453164s) [seg_pos={0 0}, upload=60693]: Begin upload segment... Jun 21 20:10:05.639231070 (75h46m41.202554057s) [seg_pos={0 0}, upload=60693]: Done begin upload segment. Jun 21 20:10:05.639292724 (75h46m41.202615728s) [noise=true , address=104.36.82.56:4253 , piece_id=5PMVGLZETMXCQ77K3I6PXACRIMBGR2CV5JGUSU5KUMYFNXVGA2GA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.639338066 (75h46m41.202661071s) [noise=true , address=104.36.82.53:4126 , piece_id=2MKRQJEPGDJKDM7VWY5M4M6NORILAIEYYATJ7HVZEOBOMXOI2LBA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.639416264 (75h46m41.202739267s) [noise=true , address=104.36.82.5:5869 , piece_id=HAEHXXZASV3DHT3QN7NTZOS6XZKHNUWBKCHWJJWSTJKXAKJKKBAA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.639873098 (75h46m41.203196115s) [noise=true , address=104.36.82.52:4111 , piece_id=HB7QAWN6CGNIIT4HD2GZ7JDQP5NXIJ5I2USBM66YOKTYENNOWG5A, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.639887296 (75h46m41.203210304s) [noise=true , address=104.36.82.10:6087 , piece_id=5FQG7BPZC373PFB54HYXSG2SVKOBBBSGZCLC4XDJEYQFOLQY74MA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.640266623 (75h46m41.203589616s) [noise=true , address=104.36.82.13:6196 , piece_id=BCHMQO5KKT5BLNWQGOHH3V43LGSE3BUQCI2SY4TTO5M5GQNOCTMQ, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.640352274 (75h46m41.203675286s) [noise=true , address=104.36.82.37:6663 , piece_id=X7LWYIIHJFU2T2PHYR3R5TNT22UUPHEHQWZH7ZJMLS4AH4PJ6AIA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.640659767 (75h46m41.203982779s) [noise=true , address=104.36.82.42:6807 , piece_id=ZBQZPXUDJ4VBCOSVKWAJ6SK6UD7SLPON7Y4SSCK2DQ5STFG4XCNQ, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.640961701 (75h46m41.20428471s) [noise=true , address=45.91.116.6:4982 , piece_id=VP45R6KXNH5U5SGUQUMAPIBY7KFDPBIR3S3VRIX27EAT6VLFBZ7Q, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.641159578 (75h46m41.204482589s) [noise=true , address=104.36.82.9:6039 , piece_id=I4AB6N27MYYUHWAZLJJUIBFBLOQFHDIM3YLYWPVRPNUV7VMYS2SA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.641588126 (75h46m41.204911136s) [noise=true , address=77.90.31.19:4853 , piece_id=VFEON5T2KSWM2KSLTVRZTXYL42JWENZYUEUPG3LMODGZWBMVGGPQ, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.641863297 (75h46m41.205186305s) [noise=true , address=77.90.31.12:4603 , piece_id=BMUIC6KNGEDL37436GLSXW5XOO6VYOPNOSFGQCYVHSENVI2PJ7KQ, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.642124427 (75h46m41.205447437s) [noise=true , address=104.36.82.49:4982 , piece_id=UNLV5UDNWULEIPE4QSQHHZQPK3BQKFRB5T3H6GP7ZQ5QGSNK5V6A, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.642285917 (75h46m41.205608925s) [noise=true , address=104.36.82.54:4171 , piece_id=MUMABKARK4KKUKWW6XDXCQVT6O7SLFHGHE7COWC7QIYGARAA2D5A, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.642656750 (75h46m41.205979758s) [noise=true , address=104.36.82.14:6232 , piece_id=A2ZLUUZKRTBJRJ267QSK36C3LFZV7FPAYDDAXGIWDTNEGNV6MRRA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.643565583 (75h46m41.206888597s) [noise=true , address=79.127.205.245:4231 , piece_id=EJK46XUUVI5NWOAQJEW4QIJG3KRXZHB4YOQUJRCS2MIOSWKGTNLQ, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.644620291 (75h46m41.207943303s) [noise=true , address=77.90.31.14:4670 , piece_id=7FNANMUKPM3N6YWRAOAIM4WJZJUTNF7ZBI6ZUBNEPGNGMFWOE24A, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.648842639 (75h46m41.212165631s) [noise=true , address=104.36.82.55:4224 , piece_id=WRVNGSJV5BX4GJERLXNPAS7J7BZ5WFTM7R5H257YRCG5SP2HIYMA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.825828239 (75h46m41.389151241s) [noise=true , address=104.36.82.53:4126 , piece_id=2MKRQJEPGDJKDM7VWY5M4M6NORILAIEYYATJ7HVZEOBOMXOI2LBA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: ecclient: upload failed (node:12dMWFrq2NMpnMzuPAphWerntC5kkrcnzRJZBA116uzJzyvk6fH, address:104.36.82.53:4126): protocol: storage node overloaded, request limit: 2000 Jun 21 20:10:05.834666043 (75h46m41.397989033s) [upload=60693]: Next returned nil segment Jun 21 20:10:05.834686295 (75h46m41.398009302s) [upload=60693]: Waiting for error group managing segments... Jun 21 20:10:05.877010488 (75h46m41.440333486s) [noise=true , address=45.91.116.6:4982 , piece_id=VP45R6KXNH5U5SGUQUMAPIBY7KFDPBIR3S3VRIX27EAT6VLFBZ7Q, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:05.886336964 (75h46m41.449659967s) [noise=true , address=104.36.82.9:6039 , piece_id=I4AB6N27MYYUHWAZLJJUIBFBLOQFHDIM3YLYWPVRPNUV7VMYS2SA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: ecclient: upload failed (node:12P11WVW9mVXw5ra2gZ3Rezon6wZi8GrN5CchQN3MiVU4ts3L6V, address:104.36.82.9:6039): protocol: storage node overloaded, request limit: 2000 Jun 21 20:10:08.064514542 (75h46m43.627837553s) [noise=true , address=77.90.31.14:4670 , piece_id=7FNANMUKPM3N6YWRAOAIM4WJZJUTNF7ZBI6ZUBNEPGNGMFWOE24A, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:08.099098391 (75h46m43.662421396s) [noise=true , address=79.127.205.245:4231 , piece_id=EJK46XUUVI5NWOAQJEW4QIJG3KRXZHB4YOQUJRCS2MIOSWKGTNLQ, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:08.489461749 (75h46m44.052784752s) [noise=true , address=104.36.82.42:6807 , piece_id=ZBQZPXUDJ4VBCOSVKWAJ6SK6UD7SLPON7Y4SSCK2DQ5STFG4XCNQ, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:08.650922093 (75h46m44.214245093s) [noise=true , address=77.90.31.19:4853 , piece_id=VFEON5T2KSWM2KSLTVRZTXYL42JWENZYUEUPG3LMODGZWBMVGGPQ, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:09.917746439 (75h46m45.481069441s) [noise=true , address=104.36.82.13:6196 , piece_id=BCHMQO5KKT5BLNWQGOHH3V43LGSE3BUQCI2SY4TTO5M5GQNOCTMQ, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:10.830722215 (75h46m46.394045216s) [noise=true , address=104.36.82.5:5869 , piece_id=HAEHXXZASV3DHT3QN7NTZOS6XZKHNUWBKCHWJJWSTJKXAKJKKBAA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:12.738783450 (75h46m48.302106452s) [noise=true , address=77.90.31.12:4603 , piece_id=BMUIC6KNGEDL37436GLSXW5XOO6VYOPNOSFGQCYVHSENVI2PJ7KQ, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:15.460927713 (75h46m51.024250723s) [noise=true , address=104.36.82.52:4111 , piece_id=HB7QAWN6CGNIIT4HD2GZ7JDQP5NXIJ5I2USBM66YOKTYENNOWG5A, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:18.393817968 (75h46m53.95714096s) [noise=true , address=104.36.82.55:4224 , piece_id=WRVNGSJV5BX4GJERLXNPAS7J7BZ5WFTM7R5H257YRCG5SP2HIYMA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:19.549174143 (75h46m55.112497151s) [noise=true , address=104.36.82.56:4253 , piece_id=5PMVGLZETMXCQ77K3I6PXACRIMBGR2CV5JGUSU5KUMYFNXVGA2GA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:22.511944593 (75h46m58.075267584s) [noise=true , address=104.36.82.37:6663 , piece_id=X7LWYIIHJFU2T2PHYR3R5TNT22UUPHEHQWZH7ZJMLS4AH4PJ6AIA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:48.027197558 (75h47m23.590520562s) [noise=true , address=104.36.82.14:6232 , piece_id=A2ZLUUZKRTBJRJ267QSK36C3LFZV7FPAYDDAXGIWDTNEGNV6MRRA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:11:05.567692582 (75h47m41.131015574s) [noise=true , address=104.36.82.49:4982 , piece_id=UNLV5UDNWULEIPE4QSQHHZQPK3BQKFRB5T3H6GP7ZQ5QGSNK5V6A, seg_pos={0 0}, upload=60693]: Done uploading piece. err: ecclient: upload failed (node:1NxoYJXCXmTvQncbkXt6yH7mhEBBrzVdzhgL5J6W5TD4N3pkaG, address:104.36.82.49:4982): protocol: expected piece hash; context deadline exceeded; EOF Jun 21 20:11:05.567759198 (75h47m41.131082195s) [noise=true , address=104.36.82.54:4171 , piece_id=MUMABKARK4KKUKWW6XDXCQVT6O7SLFHGHE7COWC7QIYGARAA2D5A, seg_pos={0 0}, upload=60693]: Done uploading piece. err: ecclient: upload failed (node:125SMhDsEZruNu6LkmQn2M7R6WzvXTdk86DRsKf522u8BNiANq3, address:104.36.82.54:4171): protocol: expected piece hash; context deadline exceeded; EOF Jun 21 20:11:05.567869019 (75h47m41.131192033s) [noise=true , address=104.36.82.10:6087 , piece_id=5FQG7BPZC373PFB54HYXSG2SVKOBBBSGZCLC4XDJEYQFOLQY74MA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: ecclient: upload failed (node:1S9k5eejJcxkiG5pXjjBZm18uP8k5jCGfWyeW2QQxmuRU8ZzRA, address:104.36.82.10:6087): protocol: write tcp 159.223.143.39:38550->104.36.82.10:6087: use of closed network connection; write tcp 159.223.143.39:38550->104.36.82.10:6087: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:38550->104.36.82.10:6087: use of closed network connection Jun 21 20:11:05.567932150 (75h47m41.131255161s) [seg_pos={0 0}, upload=60693]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jun 21 20:11:05.593626175 (75h47m41.156949181s) [upload=60693]: Done uploading segments... Jun 21 20:11:05.593657274 (75h47m41.156980265s) [upload=60693]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 22.256578286s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [.................................X...........X..............] duration: 19.789731478s FAIL (2025-06-21 20:21:29 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== pload=60693]: Starting upload Jun 21 20:10:05.602866542 (75h46m41.166189557s) [upload=60693]: Done starting upload Jun 21 20:10:05.603116545 (75h46m41.166439547s) [upload=60693]: Uploading segments... Jun 21 20:10:05.603495389 (75h46m41.166818397s) [upload=60693]: Got next segment. Inline: false Jun 21 20:10:05.603566566 (75h46m41.166889558s) [upload=60693]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 21 20:10:05.603573280 (75h46m41.166896286s) [upload=60693]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 21 20:10:05.639130162 (75h46m41.202453164s) [seg_pos={0 0}, upload=60693]: Begin upload segment... Jun 21 20:10:05.639231070 (75h46m41.202554057s) [seg_pos={0 0}, upload=60693]: Done begin upload segment. Jun 21 20:10:05.639292724 (75h46m41.202615728s) [noise=true , address=104.36.82.56:4253 , piece_id=5PMVGLZETMXCQ77K3I6PXACRIMBGR2CV5JGUSU5KUMYFNXVGA2GA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.639338066 (75h46m41.202661071s) [noise=true , address=104.36.82.53:4126 , piece_id=2MKRQJEPGDJKDM7VWY5M4M6NORILAIEYYATJ7HVZEOBOMXOI2LBA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.639416264 (75h46m41.202739267s) [noise=true , address=104.36.82.5:5869 , piece_id=HAEHXXZASV3DHT3QN7NTZOS6XZKHNUWBKCHWJJWSTJKXAKJKKBAA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.639873098 (75h46m41.203196115s) [noise=true , address=104.36.82.52:4111 , piece_id=HB7QAWN6CGNIIT4HD2GZ7JDQP5NXIJ5I2USBM66YOKTYENNOWG5A, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.639887296 (75h46m41.203210304s) [noise=true , address=104.36.82.10:6087 , piece_id=5FQG7BPZC373PFB54HYXSG2SVKOBBBSGZCLC4XDJEYQFOLQY74MA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.640266623 (75h46m41.203589616s) [noise=true , address=104.36.82.13:6196 , piece_id=BCHMQO5KKT5BLNWQGOHH3V43LGSE3BUQCI2SY4TTO5M5GQNOCTMQ, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.640352274 (75h46m41.203675286s) [noise=true , address=104.36.82.37:6663 , piece_id=X7LWYIIHJFU2T2PHYR3R5TNT22UUPHEHQWZH7ZJMLS4AH4PJ6AIA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.640659767 (75h46m41.203982779s) [noise=true , address=104.36.82.42:6807 , piece_id=ZBQZPXUDJ4VBCOSVKWAJ6SK6UD7SLPON7Y4SSCK2DQ5STFG4XCNQ, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.640961701 (75h46m41.20428471s) [noise=true , address=45.91.116.6:4982 , piece_id=VP45R6KXNH5U5SGUQUMAPIBY7KFDPBIR3S3VRIX27EAT6VLFBZ7Q, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.641159578 (75h46m41.204482589s) [noise=true , address=104.36.82.9:6039 , piece_id=I4AB6N27MYYUHWAZLJJUIBFBLOQFHDIM3YLYWPVRPNUV7VMYS2SA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.641588126 (75h46m41.204911136s) [noise=true , address=77.90.31.19:4853 , piece_id=VFEON5T2KSWM2KSLTVRZTXYL42JWENZYUEUPG3LMODGZWBMVGGPQ, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.641863297 (75h46m41.205186305s) [noise=true , address=77.90.31.12:4603 , piece_id=BMUIC6KNGEDL37436GLSXW5XOO6VYOPNOSFGQCYVHSENVI2PJ7KQ, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.642124427 (75h46m41.205447437s) [noise=true , address=104.36.82.49:4982 , piece_id=UNLV5UDNWULEIPE4QSQHHZQPK3BQKFRB5T3H6GP7ZQ5QGSNK5V6A, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.642285917 (75h46m41.205608925s) [noise=true , address=104.36.82.54:4171 , piece_id=MUMABKARK4KKUKWW6XDXCQVT6O7SLFHGHE7COWC7QIYGARAA2D5A, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.642656750 (75h46m41.205979758s) [noise=true , address=104.36.82.14:6232 , piece_id=A2ZLUUZKRTBJRJ267QSK36C3LFZV7FPAYDDAXGIWDTNEGNV6MRRA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.643565583 (75h46m41.206888597s) [noise=true , address=79.127.205.245:4231 , piece_id=EJK46XUUVI5NWOAQJEW4QIJG3KRXZHB4YOQUJRCS2MIOSWKGTNLQ, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.644620291 (75h46m41.207943303s) [noise=true , address=77.90.31.14:4670 , piece_id=7FNANMUKPM3N6YWRAOAIM4WJZJUTNF7ZBI6ZUBNEPGNGMFWOE24A, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.648842639 (75h46m41.212165631s) [noise=true , address=104.36.82.55:4224 , piece_id=WRVNGSJV5BX4GJERLXNPAS7J7BZ5WFTM7R5H257YRCG5SP2HIYMA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.825828239 (75h46m41.389151241s) [noise=true , address=104.36.82.53:4126 , piece_id=2MKRQJEPGDJKDM7VWY5M4M6NORILAIEYYATJ7HVZEOBOMXOI2LBA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: ecclient: upload failed (node:12dMWFrq2NMpnMzuPAphWerntC5kkrcnzRJZBA116uzJzyvk6fH, address:104.36.82.53:4126): protocol: storage node overloaded, request limit: 2000 Jun 21 20:10:05.834666043 (75h46m41.397989033s) [upload=60693]: Next returned nil segment Jun 21 20:10:05.834686295 (75h46m41.398009302s) [upload=60693]: Waiting for error group managing segments... Jun 21 20:10:05.877010488 (75h46m41.440333486s) [noise=true , address=45.91.116.6:4982 , piece_id=VP45R6KXNH5U5SGUQUMAPIBY7KFDPBIR3S3VRIX27EAT6VLFBZ7Q, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:05.886336964 (75h46m41.449659967s) [noise=true , address=104.36.82.9:6039 , piece_id=I4AB6N27MYYUHWAZLJJUIBFBLOQFHDIM3YLYWPVRPNUV7VMYS2SA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: ecclient: upload failed (node:12P11WVW9mVXw5ra2gZ3Rezon6wZi8GrN5CchQN3MiVU4ts3L6V, address:104.36.82.9:6039): protocol: storage node overloaded, request limit: 2000 Jun 21 20:10:08.064514542 (75h46m43.627837553s) [noise=true , address=77.90.31.14:4670 , piece_id=7FNANMUKPM3N6YWRAOAIM4WJZJUTNF7ZBI6ZUBNEPGNGMFWOE24A, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:08.099098391 (75h46m43.662421396s) [noise=true , address=79.127.205.245:4231 , piece_id=EJK46XUUVI5NWOAQJEW4QIJG3KRXZHB4YOQUJRCS2MIOSWKGTNLQ, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:08.489461749 (75h46m44.052784752s) [noise=true , address=104.36.82.42:6807 , piece_id=ZBQZPXUDJ4VBCOSVKWAJ6SK6UD7SLPON7Y4SSCK2DQ5STFG4XCNQ, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:08.650922093 (75h46m44.214245093s) [noise=true , address=77.90.31.19:4853 , piece_id=VFEON5T2KSWM2KSLTVRZTXYL42JWENZYUEUPG3LMODGZWBMVGGPQ, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:09.917746439 (75h46m45.481069441s) [noise=true , address=104.36.82.13:6196 , piece_id=BCHMQO5KKT5BLNWQGOHH3V43LGSE3BUQCI2SY4TTO5M5GQNOCTMQ, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:10.830722215 (75h46m46.394045216s) [noise=true , address=104.36.82.5:5869 , piece_id=HAEHXXZASV3DHT3QN7NTZOS6XZKHNUWBKCHWJJWSTJKXAKJKKBAA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:12.738783450 (75h46m48.302106452s) [noise=true , address=77.90.31.12:4603 , piece_id=BMUIC6KNGEDL37436GLSXW5XOO6VYOPNOSFGQCYVHSENVI2PJ7KQ, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:15.460927713 (75h46m51.024250723s) [noise=true , address=104.36.82.52:4111 , piece_id=HB7QAWN6CGNIIT4HD2GZ7JDQP5NXIJ5I2USBM66YOKTYENNOWG5A, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:18.393817968 (75h46m53.95714096s) [noise=true , address=104.36.82.55:4224 , piece_id=WRVNGSJV5BX4GJERLXNPAS7J7BZ5WFTM7R5H257YRCG5SP2HIYMA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:19.549174143 (75h46m55.112497151s) [noise=true , address=104.36.82.56:4253 , piece_id=5PMVGLZETMXCQ77K3I6PXACRIMBGR2CV5JGUSU5KUMYFNXVGA2GA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:22.511944593 (75h46m58.075267584s) [noise=true , address=104.36.82.37:6663 , piece_id=X7LWYIIHJFU2T2PHYR3R5TNT22UUPHEHQWZH7ZJMLS4AH4PJ6AIA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:48.027197558 (75h47m23.590520562s) [noise=true , address=104.36.82.14:6232 , piece_id=A2ZLUUZKRTBJRJ267QSK36C3LFZV7FPAYDDAXGIWDTNEGNV6MRRA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:11:05.567692582 (75h47m41.131015574s) [noise=true , address=104.36.82.49:4982 , piece_id=UNLV5UDNWULEIPE4QSQHHZQPK3BQKFRB5T3H6GP7ZQ5QGSNK5V6A, seg_pos={0 0}, upload=60693]: Done uploading piece. err: ecclient: upload failed (node:1NxoYJXCXmTvQncbkXt6yH7mhEBBrzVdzhgL5J6W5TD4N3pkaG, address:104.36.82.49:4982): protocol: expected piece hash; context deadline exceeded; EOF Jun 21 20:11:05.567759198 (75h47m41.131082195s) [noise=true , address=104.36.82.54:4171 , piece_id=MUMABKARK4KKUKWW6XDXCQVT6O7SLFHGHE7COWC7QIYGARAA2D5A, seg_pos={0 0}, upload=60693]: Done uploading piece. err: ecclient: upload failed (node:125SMhDsEZruNu6LkmQn2M7R6WzvXTdk86DRsKf522u8BNiANq3, address:104.36.82.54:4171): protocol: expected piece hash; context deadline exceeded; EOF Jun 21 20:11:05.567869019 (75h47m41.131192033s) [noise=true , address=104.36.82.10:6087 , piece_id=5FQG7BPZC373PFB54HYXSG2SVKOBBBSGZCLC4XDJEYQFOLQY74MA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: ecclient: upload failed (node:1S9k5eejJcxkiG5pXjjBZm18uP8k5jCGfWyeW2QQxmuRU8ZzRA, address:104.36.82.10:6087): protocol: write tcp 159.223.143.39:38550->104.36.82.10:6087: use of closed network connection; write tcp 159.223.143.39:38550->104.36.82.10:6087: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:38550->104.36.82.10:6087: use of closed network connection Jun 21 20:11:05.567932150 (75h47m41.131255161s) [seg_pos={0 0}, upload=60693]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jun 21 20:11:05.593626175 (75h47m41.156949181s) [upload=60693]: Done uploading segments... Jun 21 20:11:05.593657274 (75h47m41.156980265s) [upload=60693]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 19.789757582s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [..................................X...........X.............] duration: 25.879294238s FAIL (2025-06-21 20:20:38 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== pload=60693]: Starting upload Jun 21 20:10:05.602866542 (75h46m41.166189557s) [upload=60693]: Done starting upload Jun 21 20:10:05.603116545 (75h46m41.166439547s) [upload=60693]: Uploading segments... Jun 21 20:10:05.603495389 (75h46m41.166818397s) [upload=60693]: Got next segment. Inline: false Jun 21 20:10:05.603566566 (75h46m41.166889558s) [upload=60693]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 21 20:10:05.603573280 (75h46m41.166896286s) [upload=60693]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 21 20:10:05.639130162 (75h46m41.202453164s) [seg_pos={0 0}, upload=60693]: Begin upload segment... Jun 21 20:10:05.639231070 (75h46m41.202554057s) [seg_pos={0 0}, upload=60693]: Done begin upload segment. Jun 21 20:10:05.639292724 (75h46m41.202615728s) [noise=true , address=104.36.82.56:4253 , piece_id=5PMVGLZETMXCQ77K3I6PXACRIMBGR2CV5JGUSU5KUMYFNXVGA2GA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.639338066 (75h46m41.202661071s) [noise=true , address=104.36.82.53:4126 , piece_id=2MKRQJEPGDJKDM7VWY5M4M6NORILAIEYYATJ7HVZEOBOMXOI2LBA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.639416264 (75h46m41.202739267s) [noise=true , address=104.36.82.5:5869 , piece_id=HAEHXXZASV3DHT3QN7NTZOS6XZKHNUWBKCHWJJWSTJKXAKJKKBAA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.639873098 (75h46m41.203196115s) [noise=true , address=104.36.82.52:4111 , piece_id=HB7QAWN6CGNIIT4HD2GZ7JDQP5NXIJ5I2USBM66YOKTYENNOWG5A, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.639887296 (75h46m41.203210304s) [noise=true , address=104.36.82.10:6087 , piece_id=5FQG7BPZC373PFB54HYXSG2SVKOBBBSGZCLC4XDJEYQFOLQY74MA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.640266623 (75h46m41.203589616s) [noise=true , address=104.36.82.13:6196 , piece_id=BCHMQO5KKT5BLNWQGOHH3V43LGSE3BUQCI2SY4TTO5M5GQNOCTMQ, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.640352274 (75h46m41.203675286s) [noise=true , address=104.36.82.37:6663 , piece_id=X7LWYIIHJFU2T2PHYR3R5TNT22UUPHEHQWZH7ZJMLS4AH4PJ6AIA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.640659767 (75h46m41.203982779s) [noise=true , address=104.36.82.42:6807 , piece_id=ZBQZPXUDJ4VBCOSVKWAJ6SK6UD7SLPON7Y4SSCK2DQ5STFG4XCNQ, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.640961701 (75h46m41.20428471s) [noise=true , address=45.91.116.6:4982 , piece_id=VP45R6KXNH5U5SGUQUMAPIBY7KFDPBIR3S3VRIX27EAT6VLFBZ7Q, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.641159578 (75h46m41.204482589s) [noise=true , address=104.36.82.9:6039 , piece_id=I4AB6N27MYYUHWAZLJJUIBFBLOQFHDIM3YLYWPVRPNUV7VMYS2SA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.641588126 (75h46m41.204911136s) [noise=true , address=77.90.31.19:4853 , piece_id=VFEON5T2KSWM2KSLTVRZTXYL42JWENZYUEUPG3LMODGZWBMVGGPQ, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.641863297 (75h46m41.205186305s) [noise=true , address=77.90.31.12:4603 , piece_id=BMUIC6KNGEDL37436GLSXW5XOO6VYOPNOSFGQCYVHSENVI2PJ7KQ, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.642124427 (75h46m41.205447437s) [noise=true , address=104.36.82.49:4982 , piece_id=UNLV5UDNWULEIPE4QSQHHZQPK3BQKFRB5T3H6GP7ZQ5QGSNK5V6A, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.642285917 (75h46m41.205608925s) [noise=true , address=104.36.82.54:4171 , piece_id=MUMABKARK4KKUKWW6XDXCQVT6O7SLFHGHE7COWC7QIYGARAA2D5A, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.642656750 (75h46m41.205979758s) [noise=true , address=104.36.82.14:6232 , piece_id=A2ZLUUZKRTBJRJ267QSK36C3LFZV7FPAYDDAXGIWDTNEGNV6MRRA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.643565583 (75h46m41.206888597s) [noise=true , address=79.127.205.245:4231 , piece_id=EJK46XUUVI5NWOAQJEW4QIJG3KRXZHB4YOQUJRCS2MIOSWKGTNLQ, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.644620291 (75h46m41.207943303s) [noise=true , address=77.90.31.14:4670 , piece_id=7FNANMUKPM3N6YWRAOAIM4WJZJUTNF7ZBI6ZUBNEPGNGMFWOE24A, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.648842639 (75h46m41.212165631s) [noise=true , address=104.36.82.55:4224 , piece_id=WRVNGSJV5BX4GJERLXNPAS7J7BZ5WFTM7R5H257YRCG5SP2HIYMA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.825828239 (75h46m41.389151241s) [noise=true , address=104.36.82.53:4126 , piece_id=2MKRQJEPGDJKDM7VWY5M4M6NORILAIEYYATJ7HVZEOBOMXOI2LBA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: ecclient: upload failed (node:12dMWFrq2NMpnMzuPAphWerntC5kkrcnzRJZBA116uzJzyvk6fH, address:104.36.82.53:4126): protocol: storage node overloaded, request limit: 2000 Jun 21 20:10:05.834666043 (75h46m41.397989033s) [upload=60693]: Next returned nil segment Jun 21 20:10:05.834686295 (75h46m41.398009302s) [upload=60693]: Waiting for error group managing segments... Jun 21 20:10:05.877010488 (75h46m41.440333486s) [noise=true , address=45.91.116.6:4982 , piece_id=VP45R6KXNH5U5SGUQUMAPIBY7KFDPBIR3S3VRIX27EAT6VLFBZ7Q, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:05.886336964 (75h46m41.449659967s) [noise=true , address=104.36.82.9:6039 , piece_id=I4AB6N27MYYUHWAZLJJUIBFBLOQFHDIM3YLYWPVRPNUV7VMYS2SA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: ecclient: upload failed (node:12P11WVW9mVXw5ra2gZ3Rezon6wZi8GrN5CchQN3MiVU4ts3L6V, address:104.36.82.9:6039): protocol: storage node overloaded, request limit: 2000 Jun 21 20:10:08.064514542 (75h46m43.627837553s) [noise=true , address=77.90.31.14:4670 , piece_id=7FNANMUKPM3N6YWRAOAIM4WJZJUTNF7ZBI6ZUBNEPGNGMFWOE24A, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:08.099098391 (75h46m43.662421396s) [noise=true , address=79.127.205.245:4231 , piece_id=EJK46XUUVI5NWOAQJEW4QIJG3KRXZHB4YOQUJRCS2MIOSWKGTNLQ, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:08.489461749 (75h46m44.052784752s) [noise=true , address=104.36.82.42:6807 , piece_id=ZBQZPXUDJ4VBCOSVKWAJ6SK6UD7SLPON7Y4SSCK2DQ5STFG4XCNQ, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:08.650922093 (75h46m44.214245093s) [noise=true , address=77.90.31.19:4853 , piece_id=VFEON5T2KSWM2KSLTVRZTXYL42JWENZYUEUPG3LMODGZWBMVGGPQ, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:09.917746439 (75h46m45.481069441s) [noise=true , address=104.36.82.13:6196 , piece_id=BCHMQO5KKT5BLNWQGOHH3V43LGSE3BUQCI2SY4TTO5M5GQNOCTMQ, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:10.830722215 (75h46m46.394045216s) [noise=true , address=104.36.82.5:5869 , piece_id=HAEHXXZASV3DHT3QN7NTZOS6XZKHNUWBKCHWJJWSTJKXAKJKKBAA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:12.738783450 (75h46m48.302106452s) [noise=true , address=77.90.31.12:4603 , piece_id=BMUIC6KNGEDL37436GLSXW5XOO6VYOPNOSFGQCYVHSENVI2PJ7KQ, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:15.460927713 (75h46m51.024250723s) [noise=true , address=104.36.82.52:4111 , piece_id=HB7QAWN6CGNIIT4HD2GZ7JDQP5NXIJ5I2USBM66YOKTYENNOWG5A, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:18.393817968 (75h46m53.95714096s) [noise=true , address=104.36.82.55:4224 , piece_id=WRVNGSJV5BX4GJERLXNPAS7J7BZ5WFTM7R5H257YRCG5SP2HIYMA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:19.549174143 (75h46m55.112497151s) [noise=true , address=104.36.82.56:4253 , piece_id=5PMVGLZETMXCQ77K3I6PXACRIMBGR2CV5JGUSU5KUMYFNXVGA2GA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:22.511944593 (75h46m58.075267584s) [noise=true , address=104.36.82.37:6663 , piece_id=X7LWYIIHJFU2T2PHYR3R5TNT22UUPHEHQWZH7ZJMLS4AH4PJ6AIA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:48.027197558 (75h47m23.590520562s) [noise=true , address=104.36.82.14:6232 , piece_id=A2ZLUUZKRTBJRJ267QSK36C3LFZV7FPAYDDAXGIWDTNEGNV6MRRA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:11:05.567692582 (75h47m41.131015574s) [noise=true , address=104.36.82.49:4982 , piece_id=UNLV5UDNWULEIPE4QSQHHZQPK3BQKFRB5T3H6GP7ZQ5QGSNK5V6A, seg_pos={0 0}, upload=60693]: Done uploading piece. err: ecclient: upload failed (node:1NxoYJXCXmTvQncbkXt6yH7mhEBBrzVdzhgL5J6W5TD4N3pkaG, address:104.36.82.49:4982): protocol: expected piece hash; context deadline exceeded; EOF Jun 21 20:11:05.567759198 (75h47m41.131082195s) [noise=true , address=104.36.82.54:4171 , piece_id=MUMABKARK4KKUKWW6XDXCQVT6O7SLFHGHE7COWC7QIYGARAA2D5A, seg_pos={0 0}, upload=60693]: Done uploading piece. err: ecclient: upload failed (node:125SMhDsEZruNu6LkmQn2M7R6WzvXTdk86DRsKf522u8BNiANq3, address:104.36.82.54:4171): protocol: expected piece hash; context deadline exceeded; EOF Jun 21 20:11:05.567869019 (75h47m41.131192033s) [noise=true , address=104.36.82.10:6087 , piece_id=5FQG7BPZC373PFB54HYXSG2SVKOBBBSGZCLC4XDJEYQFOLQY74MA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: ecclient: upload failed (node:1S9k5eejJcxkiG5pXjjBZm18uP8k5jCGfWyeW2QQxmuRU8ZzRA, address:104.36.82.10:6087): protocol: write tcp 159.223.143.39:38550->104.36.82.10:6087: use of closed network connection; write tcp 159.223.143.39:38550->104.36.82.10:6087: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:38550->104.36.82.10:6087: use of closed network connection Jun 21 20:11:05.567932150 (75h47m41.131255161s) [seg_pos={0 0}, upload=60693]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jun 21 20:11:05.593626175 (75h47m41.156949181s) [upload=60693]: Done uploading segments... Jun 21 20:11:05.593657274 (75h47m41.156980265s) [upload=60693]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 25.879321862s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [...................................X...........X............] duration: 9.85608251s FAIL (2025-06-21 20:19:34 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== pload=60693]: Starting upload Jun 21 20:10:05.602866542 (75h46m41.166189557s) [upload=60693]: Done starting upload Jun 21 20:10:05.603116545 (75h46m41.166439547s) [upload=60693]: Uploading segments... Jun 21 20:10:05.603495389 (75h46m41.166818397s) [upload=60693]: Got next segment. Inline: false Jun 21 20:10:05.603566566 (75h46m41.166889558s) [upload=60693]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 21 20:10:05.603573280 (75h46m41.166896286s) [upload=60693]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 21 20:10:05.639130162 (75h46m41.202453164s) [seg_pos={0 0}, upload=60693]: Begin upload segment... Jun 21 20:10:05.639231070 (75h46m41.202554057s) [seg_pos={0 0}, upload=60693]: Done begin upload segment. Jun 21 20:10:05.639292724 (75h46m41.202615728s) [noise=true , address=104.36.82.56:4253 , piece_id=5PMVGLZETMXCQ77K3I6PXACRIMBGR2CV5JGUSU5KUMYFNXVGA2GA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.639338066 (75h46m41.202661071s) [noise=true , address=104.36.82.53:4126 , piece_id=2MKRQJEPGDJKDM7VWY5M4M6NORILAIEYYATJ7HVZEOBOMXOI2LBA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.639416264 (75h46m41.202739267s) [noise=true , address=104.36.82.5:5869 , piece_id=HAEHXXZASV3DHT3QN7NTZOS6XZKHNUWBKCHWJJWSTJKXAKJKKBAA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.639873098 (75h46m41.203196115s) [noise=true , address=104.36.82.52:4111 , piece_id=HB7QAWN6CGNIIT4HD2GZ7JDQP5NXIJ5I2USBM66YOKTYENNOWG5A, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.639887296 (75h46m41.203210304s) [noise=true , address=104.36.82.10:6087 , piece_id=5FQG7BPZC373PFB54HYXSG2SVKOBBBSGZCLC4XDJEYQFOLQY74MA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.640266623 (75h46m41.203589616s) [noise=true , address=104.36.82.13:6196 , piece_id=BCHMQO5KKT5BLNWQGOHH3V43LGSE3BUQCI2SY4TTO5M5GQNOCTMQ, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.640352274 (75h46m41.203675286s) [noise=true , address=104.36.82.37:6663 , piece_id=X7LWYIIHJFU2T2PHYR3R5TNT22UUPHEHQWZH7ZJMLS4AH4PJ6AIA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.640659767 (75h46m41.203982779s) [noise=true , address=104.36.82.42:6807 , piece_id=ZBQZPXUDJ4VBCOSVKWAJ6SK6UD7SLPON7Y4SSCK2DQ5STFG4XCNQ, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.640961701 (75h46m41.20428471s) [noise=true , address=45.91.116.6:4982 , piece_id=VP45R6KXNH5U5SGUQUMAPIBY7KFDPBIR3S3VRIX27EAT6VLFBZ7Q, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.641159578 (75h46m41.204482589s) [noise=true , address=104.36.82.9:6039 , piece_id=I4AB6N27MYYUHWAZLJJUIBFBLOQFHDIM3YLYWPVRPNUV7VMYS2SA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.641588126 (75h46m41.204911136s) [noise=true , address=77.90.31.19:4853 , piece_id=VFEON5T2KSWM2KSLTVRZTXYL42JWENZYUEUPG3LMODGZWBMVGGPQ, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.641863297 (75h46m41.205186305s) [noise=true , address=77.90.31.12:4603 , piece_id=BMUIC6KNGEDL37436GLSXW5XOO6VYOPNOSFGQCYVHSENVI2PJ7KQ, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.642124427 (75h46m41.205447437s) [noise=true , address=104.36.82.49:4982 , piece_id=UNLV5UDNWULEIPE4QSQHHZQPK3BQKFRB5T3H6GP7ZQ5QGSNK5V6A, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.642285917 (75h46m41.205608925s) [noise=true , address=104.36.82.54:4171 , piece_id=MUMABKARK4KKUKWW6XDXCQVT6O7SLFHGHE7COWC7QIYGARAA2D5A, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.642656750 (75h46m41.205979758s) [noise=true , address=104.36.82.14:6232 , piece_id=A2ZLUUZKRTBJRJ267QSK36C3LFZV7FPAYDDAXGIWDTNEGNV6MRRA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.643565583 (75h46m41.206888597s) [noise=true , address=79.127.205.245:4231 , piece_id=EJK46XUUVI5NWOAQJEW4QIJG3KRXZHB4YOQUJRCS2MIOSWKGTNLQ, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.644620291 (75h46m41.207943303s) [noise=true , address=77.90.31.14:4670 , piece_id=7FNANMUKPM3N6YWRAOAIM4WJZJUTNF7ZBI6ZUBNEPGNGMFWOE24A, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.648842639 (75h46m41.212165631s) [noise=true , address=104.36.82.55:4224 , piece_id=WRVNGSJV5BX4GJERLXNPAS7J7BZ5WFTM7R5H257YRCG5SP2HIYMA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.825828239 (75h46m41.389151241s) [noise=true , address=104.36.82.53:4126 , piece_id=2MKRQJEPGDJKDM7VWY5M4M6NORILAIEYYATJ7HVZEOBOMXOI2LBA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: ecclient: upload failed (node:12dMWFrq2NMpnMzuPAphWerntC5kkrcnzRJZBA116uzJzyvk6fH, address:104.36.82.53:4126): protocol: storage node overloaded, request limit: 2000 Jun 21 20:10:05.834666043 (75h46m41.397989033s) [upload=60693]: Next returned nil segment Jun 21 20:10:05.834686295 (75h46m41.398009302s) [upload=60693]: Waiting for error group managing segments... Jun 21 20:10:05.877010488 (75h46m41.440333486s) [noise=true , address=45.91.116.6:4982 , piece_id=VP45R6KXNH5U5SGUQUMAPIBY7KFDPBIR3S3VRIX27EAT6VLFBZ7Q, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:05.886336964 (75h46m41.449659967s) [noise=true , address=104.36.82.9:6039 , piece_id=I4AB6N27MYYUHWAZLJJUIBFBLOQFHDIM3YLYWPVRPNUV7VMYS2SA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: ecclient: upload failed (node:12P11WVW9mVXw5ra2gZ3Rezon6wZi8GrN5CchQN3MiVU4ts3L6V, address:104.36.82.9:6039): protocol: storage node overloaded, request limit: 2000 Jun 21 20:10:08.064514542 (75h46m43.627837553s) [noise=true , address=77.90.31.14:4670 , piece_id=7FNANMUKPM3N6YWRAOAIM4WJZJUTNF7ZBI6ZUBNEPGNGMFWOE24A, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:08.099098391 (75h46m43.662421396s) [noise=true , address=79.127.205.245:4231 , piece_id=EJK46XUUVI5NWOAQJEW4QIJG3KRXZHB4YOQUJRCS2MIOSWKGTNLQ, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:08.489461749 (75h46m44.052784752s) [noise=true , address=104.36.82.42:6807 , piece_id=ZBQZPXUDJ4VBCOSVKWAJ6SK6UD7SLPON7Y4SSCK2DQ5STFG4XCNQ, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:08.650922093 (75h46m44.214245093s) [noise=true , address=77.90.31.19:4853 , piece_id=VFEON5T2KSWM2KSLTVRZTXYL42JWENZYUEUPG3LMODGZWBMVGGPQ, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:09.917746439 (75h46m45.481069441s) [noise=true , address=104.36.82.13:6196 , piece_id=BCHMQO5KKT5BLNWQGOHH3V43LGSE3BUQCI2SY4TTO5M5GQNOCTMQ, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:10.830722215 (75h46m46.394045216s) [noise=true , address=104.36.82.5:5869 , piece_id=HAEHXXZASV3DHT3QN7NTZOS6XZKHNUWBKCHWJJWSTJKXAKJKKBAA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:12.738783450 (75h46m48.302106452s) [noise=true , address=77.90.31.12:4603 , piece_id=BMUIC6KNGEDL37436GLSXW5XOO6VYOPNOSFGQCYVHSENVI2PJ7KQ, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:15.460927713 (75h46m51.024250723s) [noise=true , address=104.36.82.52:4111 , piece_id=HB7QAWN6CGNIIT4HD2GZ7JDQP5NXIJ5I2USBM66YOKTYENNOWG5A, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:18.393817968 (75h46m53.95714096s) [noise=true , address=104.36.82.55:4224 , piece_id=WRVNGSJV5BX4GJERLXNPAS7J7BZ5WFTM7R5H257YRCG5SP2HIYMA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:19.549174143 (75h46m55.112497151s) [noise=true , address=104.36.82.56:4253 , piece_id=5PMVGLZETMXCQ77K3I6PXACRIMBGR2CV5JGUSU5KUMYFNXVGA2GA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:22.511944593 (75h46m58.075267584s) [noise=true , address=104.36.82.37:6663 , piece_id=X7LWYIIHJFU2T2PHYR3R5TNT22UUPHEHQWZH7ZJMLS4AH4PJ6AIA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:48.027197558 (75h47m23.590520562s) [noise=true , address=104.36.82.14:6232 , piece_id=A2ZLUUZKRTBJRJ267QSK36C3LFZV7FPAYDDAXGIWDTNEGNV6MRRA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:11:05.567692582 (75h47m41.131015574s) [noise=true , address=104.36.82.49:4982 , piece_id=UNLV5UDNWULEIPE4QSQHHZQPK3BQKFRB5T3H6GP7ZQ5QGSNK5V6A, seg_pos={0 0}, upload=60693]: Done uploading piece. err: ecclient: upload failed (node:1NxoYJXCXmTvQncbkXt6yH7mhEBBrzVdzhgL5J6W5TD4N3pkaG, address:104.36.82.49:4982): protocol: expected piece hash; context deadline exceeded; EOF Jun 21 20:11:05.567759198 (75h47m41.131082195s) [noise=true , address=104.36.82.54:4171 , piece_id=MUMABKARK4KKUKWW6XDXCQVT6O7SLFHGHE7COWC7QIYGARAA2D5A, seg_pos={0 0}, upload=60693]: Done uploading piece. err: ecclient: upload failed (node:125SMhDsEZruNu6LkmQn2M7R6WzvXTdk86DRsKf522u8BNiANq3, address:104.36.82.54:4171): protocol: expected piece hash; context deadline exceeded; EOF Jun 21 20:11:05.567869019 (75h47m41.131192033s) [noise=true , address=104.36.82.10:6087 , piece_id=5FQG7BPZC373PFB54HYXSG2SVKOBBBSGZCLC4XDJEYQFOLQY74MA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: ecclient: upload failed (node:1S9k5eejJcxkiG5pXjjBZm18uP8k5jCGfWyeW2QQxmuRU8ZzRA, address:104.36.82.10:6087): protocol: write tcp 159.223.143.39:38550->104.36.82.10:6087: use of closed network connection; write tcp 159.223.143.39:38550->104.36.82.10:6087: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:38550->104.36.82.10:6087: use of closed network connection Jun 21 20:11:05.567932150 (75h47m41.131255161s) [seg_pos={0 0}, upload=60693]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jun 21 20:11:05.593626175 (75h47m41.156949181s) [upload=60693]: Done uploading segments... Jun 21 20:11:05.593657274 (75h47m41.156980265s) [upload=60693]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 9.856105724s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [....................................X...........X...........] duration: 28.050748236s FAIL (2025-06-21 20:19:24 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== pload=60693]: Starting upload Jun 21 20:10:05.602866542 (75h46m41.166189557s) [upload=60693]: Done starting upload Jun 21 20:10:05.603116545 (75h46m41.166439547s) [upload=60693]: Uploading segments... Jun 21 20:10:05.603495389 (75h46m41.166818397s) [upload=60693]: Got next segment. Inline: false Jun 21 20:10:05.603566566 (75h46m41.166889558s) [upload=60693]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 21 20:10:05.603573280 (75h46m41.166896286s) [upload=60693]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 21 20:10:05.639130162 (75h46m41.202453164s) [seg_pos={0 0}, upload=60693]: Begin upload segment... Jun 21 20:10:05.639231070 (75h46m41.202554057s) [seg_pos={0 0}, upload=60693]: Done begin upload segment. Jun 21 20:10:05.639292724 (75h46m41.202615728s) [noise=true , address=104.36.82.56:4253 , piece_id=5PMVGLZETMXCQ77K3I6PXACRIMBGR2CV5JGUSU5KUMYFNXVGA2GA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.639338066 (75h46m41.202661071s) [noise=true , address=104.36.82.53:4126 , piece_id=2MKRQJEPGDJKDM7VWY5M4M6NORILAIEYYATJ7HVZEOBOMXOI2LBA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.639416264 (75h46m41.202739267s) [noise=true , address=104.36.82.5:5869 , piece_id=HAEHXXZASV3DHT3QN7NTZOS6XZKHNUWBKCHWJJWSTJKXAKJKKBAA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.639873098 (75h46m41.203196115s) [noise=true , address=104.36.82.52:4111 , piece_id=HB7QAWN6CGNIIT4HD2GZ7JDQP5NXIJ5I2USBM66YOKTYENNOWG5A, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.639887296 (75h46m41.203210304s) [noise=true , address=104.36.82.10:6087 , piece_id=5FQG7BPZC373PFB54HYXSG2SVKOBBBSGZCLC4XDJEYQFOLQY74MA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.640266623 (75h46m41.203589616s) [noise=true , address=104.36.82.13:6196 , piece_id=BCHMQO5KKT5BLNWQGOHH3V43LGSE3BUQCI2SY4TTO5M5GQNOCTMQ, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.640352274 (75h46m41.203675286s) [noise=true , address=104.36.82.37:6663 , piece_id=X7LWYIIHJFU2T2PHYR3R5TNT22UUPHEHQWZH7ZJMLS4AH4PJ6AIA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.640659767 (75h46m41.203982779s) [noise=true , address=104.36.82.42:6807 , piece_id=ZBQZPXUDJ4VBCOSVKWAJ6SK6UD7SLPON7Y4SSCK2DQ5STFG4XCNQ, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.640961701 (75h46m41.20428471s) [noise=true , address=45.91.116.6:4982 , piece_id=VP45R6KXNH5U5SGUQUMAPIBY7KFDPBIR3S3VRIX27EAT6VLFBZ7Q, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.641159578 (75h46m41.204482589s) [noise=true , address=104.36.82.9:6039 , piece_id=I4AB6N27MYYUHWAZLJJUIBFBLOQFHDIM3YLYWPVRPNUV7VMYS2SA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.641588126 (75h46m41.204911136s) [noise=true , address=77.90.31.19:4853 , piece_id=VFEON5T2KSWM2KSLTVRZTXYL42JWENZYUEUPG3LMODGZWBMVGGPQ, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.641863297 (75h46m41.205186305s) [noise=true , address=77.90.31.12:4603 , piece_id=BMUIC6KNGEDL37436GLSXW5XOO6VYOPNOSFGQCYVHSENVI2PJ7KQ, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.642124427 (75h46m41.205447437s) [noise=true , address=104.36.82.49:4982 , piece_id=UNLV5UDNWULEIPE4QSQHHZQPK3BQKFRB5T3H6GP7ZQ5QGSNK5V6A, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.642285917 (75h46m41.205608925s) [noise=true , address=104.36.82.54:4171 , piece_id=MUMABKARK4KKUKWW6XDXCQVT6O7SLFHGHE7COWC7QIYGARAA2D5A, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.642656750 (75h46m41.205979758s) [noise=true , address=104.36.82.14:6232 , piece_id=A2ZLUUZKRTBJRJ267QSK36C3LFZV7FPAYDDAXGIWDTNEGNV6MRRA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.643565583 (75h46m41.206888597s) [noise=true , address=79.127.205.245:4231 , piece_id=EJK46XUUVI5NWOAQJEW4QIJG3KRXZHB4YOQUJRCS2MIOSWKGTNLQ, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.644620291 (75h46m41.207943303s) [noise=true , address=77.90.31.14:4670 , piece_id=7FNANMUKPM3N6YWRAOAIM4WJZJUTNF7ZBI6ZUBNEPGNGMFWOE24A, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.648842639 (75h46m41.212165631s) [noise=true , address=104.36.82.55:4224 , piece_id=WRVNGSJV5BX4GJERLXNPAS7J7BZ5WFTM7R5H257YRCG5SP2HIYMA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.825828239 (75h46m41.389151241s) [noise=true , address=104.36.82.53:4126 , piece_id=2MKRQJEPGDJKDM7VWY5M4M6NORILAIEYYATJ7HVZEOBOMXOI2LBA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: ecclient: upload failed (node:12dMWFrq2NMpnMzuPAphWerntC5kkrcnzRJZBA116uzJzyvk6fH, address:104.36.82.53:4126): protocol: storage node overloaded, request limit: 2000 Jun 21 20:10:05.834666043 (75h46m41.397989033s) [upload=60693]: Next returned nil segment Jun 21 20:10:05.834686295 (75h46m41.398009302s) [upload=60693]: Waiting for error group managing segments... Jun 21 20:10:05.877010488 (75h46m41.440333486s) [noise=true , address=45.91.116.6:4982 , piece_id=VP45R6KXNH5U5SGUQUMAPIBY7KFDPBIR3S3VRIX27EAT6VLFBZ7Q, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:05.886336964 (75h46m41.449659967s) [noise=true , address=104.36.82.9:6039 , piece_id=I4AB6N27MYYUHWAZLJJUIBFBLOQFHDIM3YLYWPVRPNUV7VMYS2SA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: ecclient: upload failed (node:12P11WVW9mVXw5ra2gZ3Rezon6wZi8GrN5CchQN3MiVU4ts3L6V, address:104.36.82.9:6039): protocol: storage node overloaded, request limit: 2000 Jun 21 20:10:08.064514542 (75h46m43.627837553s) [noise=true , address=77.90.31.14:4670 , piece_id=7FNANMUKPM3N6YWRAOAIM4WJZJUTNF7ZBI6ZUBNEPGNGMFWOE24A, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:08.099098391 (75h46m43.662421396s) [noise=true , address=79.127.205.245:4231 , piece_id=EJK46XUUVI5NWOAQJEW4QIJG3KRXZHB4YOQUJRCS2MIOSWKGTNLQ, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:08.489461749 (75h46m44.052784752s) [noise=true , address=104.36.82.42:6807 , piece_id=ZBQZPXUDJ4VBCOSVKWAJ6SK6UD7SLPON7Y4SSCK2DQ5STFG4XCNQ, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:08.650922093 (75h46m44.214245093s) [noise=true , address=77.90.31.19:4853 , piece_id=VFEON5T2KSWM2KSLTVRZTXYL42JWENZYUEUPG3LMODGZWBMVGGPQ, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:09.917746439 (75h46m45.481069441s) [noise=true , address=104.36.82.13:6196 , piece_id=BCHMQO5KKT5BLNWQGOHH3V43LGSE3BUQCI2SY4TTO5M5GQNOCTMQ, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:10.830722215 (75h46m46.394045216s) [noise=true , address=104.36.82.5:5869 , piece_id=HAEHXXZASV3DHT3QN7NTZOS6XZKHNUWBKCHWJJWSTJKXAKJKKBAA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:12.738783450 (75h46m48.302106452s) [noise=true , address=77.90.31.12:4603 , piece_id=BMUIC6KNGEDL37436GLSXW5XOO6VYOPNOSFGQCYVHSENVI2PJ7KQ, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:15.460927713 (75h46m51.024250723s) [noise=true , address=104.36.82.52:4111 , piece_id=HB7QAWN6CGNIIT4HD2GZ7JDQP5NXIJ5I2USBM66YOKTYENNOWG5A, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:18.393817968 (75h46m53.95714096s) [noise=true , address=104.36.82.55:4224 , piece_id=WRVNGSJV5BX4GJERLXNPAS7J7BZ5WFTM7R5H257YRCG5SP2HIYMA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:19.549174143 (75h46m55.112497151s) [noise=true , address=104.36.82.56:4253 , piece_id=5PMVGLZETMXCQ77K3I6PXACRIMBGR2CV5JGUSU5KUMYFNXVGA2GA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:22.511944593 (75h46m58.075267584s) [noise=true , address=104.36.82.37:6663 , piece_id=X7LWYIIHJFU2T2PHYR3R5TNT22UUPHEHQWZH7ZJMLS4AH4PJ6AIA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:48.027197558 (75h47m23.590520562s) [noise=true , address=104.36.82.14:6232 , piece_id=A2ZLUUZKRTBJRJ267QSK36C3LFZV7FPAYDDAXGIWDTNEGNV6MRRA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:11:05.567692582 (75h47m41.131015574s) [noise=true , address=104.36.82.49:4982 , piece_id=UNLV5UDNWULEIPE4QSQHHZQPK3BQKFRB5T3H6GP7ZQ5QGSNK5V6A, seg_pos={0 0}, upload=60693]: Done uploading piece. err: ecclient: upload failed (node:1NxoYJXCXmTvQncbkXt6yH7mhEBBrzVdzhgL5J6W5TD4N3pkaG, address:104.36.82.49:4982): protocol: expected piece hash; context deadline exceeded; EOF Jun 21 20:11:05.567759198 (75h47m41.131082195s) [noise=true , address=104.36.82.54:4171 , piece_id=MUMABKARK4KKUKWW6XDXCQVT6O7SLFHGHE7COWC7QIYGARAA2D5A, seg_pos={0 0}, upload=60693]: Done uploading piece. err: ecclient: upload failed (node:125SMhDsEZruNu6LkmQn2M7R6WzvXTdk86DRsKf522u8BNiANq3, address:104.36.82.54:4171): protocol: expected piece hash; context deadline exceeded; EOF Jun 21 20:11:05.567869019 (75h47m41.131192033s) [noise=true , address=104.36.82.10:6087 , piece_id=5FQG7BPZC373PFB54HYXSG2SVKOBBBSGZCLC4XDJEYQFOLQY74MA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: ecclient: upload failed (node:1S9k5eejJcxkiG5pXjjBZm18uP8k5jCGfWyeW2QQxmuRU8ZzRA, address:104.36.82.10:6087): protocol: write tcp 159.223.143.39:38550->104.36.82.10:6087: use of closed network connection; write tcp 159.223.143.39:38550->104.36.82.10:6087: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:38550->104.36.82.10:6087: use of closed network connection Jun 21 20:11:05.567932150 (75h47m41.131255161s) [seg_pos={0 0}, upload=60693]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jun 21 20:11:05.593626175 (75h47m41.156949181s) [upload=60693]: Done uploading segments... Jun 21 20:11:05.593657274 (75h47m41.156980265s) [upload=60693]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 28.050802946s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [.....................................X...........X..........] duration: 15.58526699s FAIL (2025-06-21 20:18:15 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== pload=60693]: Starting upload Jun 21 20:10:05.602866542 (75h46m41.166189557s) [upload=60693]: Done starting upload Jun 21 20:10:05.603116545 (75h46m41.166439547s) [upload=60693]: Uploading segments... Jun 21 20:10:05.603495389 (75h46m41.166818397s) [upload=60693]: Got next segment. Inline: false Jun 21 20:10:05.603566566 (75h46m41.166889558s) [upload=60693]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 21 20:10:05.603573280 (75h46m41.166896286s) [upload=60693]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 21 20:10:05.639130162 (75h46m41.202453164s) [seg_pos={0 0}, upload=60693]: Begin upload segment... Jun 21 20:10:05.639231070 (75h46m41.202554057s) [seg_pos={0 0}, upload=60693]: Done begin upload segment. Jun 21 20:10:05.639292724 (75h46m41.202615728s) [noise=true , address=104.36.82.56:4253 , piece_id=5PMVGLZETMXCQ77K3I6PXACRIMBGR2CV5JGUSU5KUMYFNXVGA2GA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.639338066 (75h46m41.202661071s) [noise=true , address=104.36.82.53:4126 , piece_id=2MKRQJEPGDJKDM7VWY5M4M6NORILAIEYYATJ7HVZEOBOMXOI2LBA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.639416264 (75h46m41.202739267s) [noise=true , address=104.36.82.5:5869 , piece_id=HAEHXXZASV3DHT3QN7NTZOS6XZKHNUWBKCHWJJWSTJKXAKJKKBAA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.639873098 (75h46m41.203196115s) [noise=true , address=104.36.82.52:4111 , piece_id=HB7QAWN6CGNIIT4HD2GZ7JDQP5NXIJ5I2USBM66YOKTYENNOWG5A, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.639887296 (75h46m41.203210304s) [noise=true , address=104.36.82.10:6087 , piece_id=5FQG7BPZC373PFB54HYXSG2SVKOBBBSGZCLC4XDJEYQFOLQY74MA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.640266623 (75h46m41.203589616s) [noise=true , address=104.36.82.13:6196 , piece_id=BCHMQO5KKT5BLNWQGOHH3V43LGSE3BUQCI2SY4TTO5M5GQNOCTMQ, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.640352274 (75h46m41.203675286s) [noise=true , address=104.36.82.37:6663 , piece_id=X7LWYIIHJFU2T2PHYR3R5TNT22UUPHEHQWZH7ZJMLS4AH4PJ6AIA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.640659767 (75h46m41.203982779s) [noise=true , address=104.36.82.42:6807 , piece_id=ZBQZPXUDJ4VBCOSVKWAJ6SK6UD7SLPON7Y4SSCK2DQ5STFG4XCNQ, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.640961701 (75h46m41.20428471s) [noise=true , address=45.91.116.6:4982 , piece_id=VP45R6KXNH5U5SGUQUMAPIBY7KFDPBIR3S3VRIX27EAT6VLFBZ7Q, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.641159578 (75h46m41.204482589s) [noise=true , address=104.36.82.9:6039 , piece_id=I4AB6N27MYYUHWAZLJJUIBFBLOQFHDIM3YLYWPVRPNUV7VMYS2SA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.641588126 (75h46m41.204911136s) [noise=true , address=77.90.31.19:4853 , piece_id=VFEON5T2KSWM2KSLTVRZTXYL42JWENZYUEUPG3LMODGZWBMVGGPQ, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.641863297 (75h46m41.205186305s) [noise=true , address=77.90.31.12:4603 , piece_id=BMUIC6KNGEDL37436GLSXW5XOO6VYOPNOSFGQCYVHSENVI2PJ7KQ, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.642124427 (75h46m41.205447437s) [noise=true , address=104.36.82.49:4982 , piece_id=UNLV5UDNWULEIPE4QSQHHZQPK3BQKFRB5T3H6GP7ZQ5QGSNK5V6A, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.642285917 (75h46m41.205608925s) [noise=true , address=104.36.82.54:4171 , piece_id=MUMABKARK4KKUKWW6XDXCQVT6O7SLFHGHE7COWC7QIYGARAA2D5A, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.642656750 (75h46m41.205979758s) [noise=true , address=104.36.82.14:6232 , piece_id=A2ZLUUZKRTBJRJ267QSK36C3LFZV7FPAYDDAXGIWDTNEGNV6MRRA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.643565583 (75h46m41.206888597s) [noise=true , address=79.127.205.245:4231 , piece_id=EJK46XUUVI5NWOAQJEW4QIJG3KRXZHB4YOQUJRCS2MIOSWKGTNLQ, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.644620291 (75h46m41.207943303s) [noise=true , address=77.90.31.14:4670 , piece_id=7FNANMUKPM3N6YWRAOAIM4WJZJUTNF7ZBI6ZUBNEPGNGMFWOE24A, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.648842639 (75h46m41.212165631s) [noise=true , address=104.36.82.55:4224 , piece_id=WRVNGSJV5BX4GJERLXNPAS7J7BZ5WFTM7R5H257YRCG5SP2HIYMA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.825828239 (75h46m41.389151241s) [noise=true , address=104.36.82.53:4126 , piece_id=2MKRQJEPGDJKDM7VWY5M4M6NORILAIEYYATJ7HVZEOBOMXOI2LBA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: ecclient: upload failed (node:12dMWFrq2NMpnMzuPAphWerntC5kkrcnzRJZBA116uzJzyvk6fH, address:104.36.82.53:4126): protocol: storage node overloaded, request limit: 2000 Jun 21 20:10:05.834666043 (75h46m41.397989033s) [upload=60693]: Next returned nil segment Jun 21 20:10:05.834686295 (75h46m41.398009302s) [upload=60693]: Waiting for error group managing segments... Jun 21 20:10:05.877010488 (75h46m41.440333486s) [noise=true , address=45.91.116.6:4982 , piece_id=VP45R6KXNH5U5SGUQUMAPIBY7KFDPBIR3S3VRIX27EAT6VLFBZ7Q, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:05.886336964 (75h46m41.449659967s) [noise=true , address=104.36.82.9:6039 , piece_id=I4AB6N27MYYUHWAZLJJUIBFBLOQFHDIM3YLYWPVRPNUV7VMYS2SA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: ecclient: upload failed (node:12P11WVW9mVXw5ra2gZ3Rezon6wZi8GrN5CchQN3MiVU4ts3L6V, address:104.36.82.9:6039): protocol: storage node overloaded, request limit: 2000 Jun 21 20:10:08.064514542 (75h46m43.627837553s) [noise=true , address=77.90.31.14:4670 , piece_id=7FNANMUKPM3N6YWRAOAIM4WJZJUTNF7ZBI6ZUBNEPGNGMFWOE24A, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:08.099098391 (75h46m43.662421396s) [noise=true , address=79.127.205.245:4231 , piece_id=EJK46XUUVI5NWOAQJEW4QIJG3KRXZHB4YOQUJRCS2MIOSWKGTNLQ, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:08.489461749 (75h46m44.052784752s) [noise=true , address=104.36.82.42:6807 , piece_id=ZBQZPXUDJ4VBCOSVKWAJ6SK6UD7SLPON7Y4SSCK2DQ5STFG4XCNQ, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:08.650922093 (75h46m44.214245093s) [noise=true , address=77.90.31.19:4853 , piece_id=VFEON5T2KSWM2KSLTVRZTXYL42JWENZYUEUPG3LMODGZWBMVGGPQ, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:09.917746439 (75h46m45.481069441s) [noise=true , address=104.36.82.13:6196 , piece_id=BCHMQO5KKT5BLNWQGOHH3V43LGSE3BUQCI2SY4TTO5M5GQNOCTMQ, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:10.830722215 (75h46m46.394045216s) [noise=true , address=104.36.82.5:5869 , piece_id=HAEHXXZASV3DHT3QN7NTZOS6XZKHNUWBKCHWJJWSTJKXAKJKKBAA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:12.738783450 (75h46m48.302106452s) [noise=true , address=77.90.31.12:4603 , piece_id=BMUIC6KNGEDL37436GLSXW5XOO6VYOPNOSFGQCYVHSENVI2PJ7KQ, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:15.460927713 (75h46m51.024250723s) [noise=true , address=104.36.82.52:4111 , piece_id=HB7QAWN6CGNIIT4HD2GZ7JDQP5NXIJ5I2USBM66YOKTYENNOWG5A, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:18.393817968 (75h46m53.95714096s) [noise=true , address=104.36.82.55:4224 , piece_id=WRVNGSJV5BX4GJERLXNPAS7J7BZ5WFTM7R5H257YRCG5SP2HIYMA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:19.549174143 (75h46m55.112497151s) [noise=true , address=104.36.82.56:4253 , piece_id=5PMVGLZETMXCQ77K3I6PXACRIMBGR2CV5JGUSU5KUMYFNXVGA2GA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:22.511944593 (75h46m58.075267584s) [noise=true , address=104.36.82.37:6663 , piece_id=X7LWYIIHJFU2T2PHYR3R5TNT22UUPHEHQWZH7ZJMLS4AH4PJ6AIA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:48.027197558 (75h47m23.590520562s) [noise=true , address=104.36.82.14:6232 , piece_id=A2ZLUUZKRTBJRJ267QSK36C3LFZV7FPAYDDAXGIWDTNEGNV6MRRA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:11:05.567692582 (75h47m41.131015574s) [noise=true , address=104.36.82.49:4982 , piece_id=UNLV5UDNWULEIPE4QSQHHZQPK3BQKFRB5T3H6GP7ZQ5QGSNK5V6A, seg_pos={0 0}, upload=60693]: Done uploading piece. err: ecclient: upload failed (node:1NxoYJXCXmTvQncbkXt6yH7mhEBBrzVdzhgL5J6W5TD4N3pkaG, address:104.36.82.49:4982): protocol: expected piece hash; context deadline exceeded; EOF Jun 21 20:11:05.567759198 (75h47m41.131082195s) [noise=true , address=104.36.82.54:4171 , piece_id=MUMABKARK4KKUKWW6XDXCQVT6O7SLFHGHE7COWC7QIYGARAA2D5A, seg_pos={0 0}, upload=60693]: Done uploading piece. err: ecclient: upload failed (node:125SMhDsEZruNu6LkmQn2M7R6WzvXTdk86DRsKf522u8BNiANq3, address:104.36.82.54:4171): protocol: expected piece hash; context deadline exceeded; EOF Jun 21 20:11:05.567869019 (75h47m41.131192033s) [noise=true , address=104.36.82.10:6087 , piece_id=5FQG7BPZC373PFB54HYXSG2SVKOBBBSGZCLC4XDJEYQFOLQY74MA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: ecclient: upload failed (node:1S9k5eejJcxkiG5pXjjBZm18uP8k5jCGfWyeW2QQxmuRU8ZzRA, address:104.36.82.10:6087): protocol: write tcp 159.223.143.39:38550->104.36.82.10:6087: use of closed network connection; write tcp 159.223.143.39:38550->104.36.82.10:6087: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:38550->104.36.82.10:6087: use of closed network connection Jun 21 20:11:05.567932150 (75h47m41.131255161s) [seg_pos={0 0}, upload=60693]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jun 21 20:11:05.593626175 (75h47m41.156949181s) [upload=60693]: Done uploading segments... Jun 21 20:11:05.593657274 (75h47m41.156980265s) [upload=60693]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 15.585297471s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [......................................X...........X.........] duration: 19.678500308s FAIL (2025-06-21 20:17:32 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== pload=60693]: Starting upload Jun 21 20:10:05.602866542 (75h46m41.166189557s) [upload=60693]: Done starting upload Jun 21 20:10:05.603116545 (75h46m41.166439547s) [upload=60693]: Uploading segments... Jun 21 20:10:05.603495389 (75h46m41.166818397s) [upload=60693]: Got next segment. Inline: false Jun 21 20:10:05.603566566 (75h46m41.166889558s) [upload=60693]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 21 20:10:05.603573280 (75h46m41.166896286s) [upload=60693]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 21 20:10:05.639130162 (75h46m41.202453164s) [seg_pos={0 0}, upload=60693]: Begin upload segment... Jun 21 20:10:05.639231070 (75h46m41.202554057s) [seg_pos={0 0}, upload=60693]: Done begin upload segment. Jun 21 20:10:05.639292724 (75h46m41.202615728s) [noise=true , address=104.36.82.56:4253 , piece_id=5PMVGLZETMXCQ77K3I6PXACRIMBGR2CV5JGUSU5KUMYFNXVGA2GA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.639338066 (75h46m41.202661071s) [noise=true , address=104.36.82.53:4126 , piece_id=2MKRQJEPGDJKDM7VWY5M4M6NORILAIEYYATJ7HVZEOBOMXOI2LBA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.639416264 (75h46m41.202739267s) [noise=true , address=104.36.82.5:5869 , piece_id=HAEHXXZASV3DHT3QN7NTZOS6XZKHNUWBKCHWJJWSTJKXAKJKKBAA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.639873098 (75h46m41.203196115s) [noise=true , address=104.36.82.52:4111 , piece_id=HB7QAWN6CGNIIT4HD2GZ7JDQP5NXIJ5I2USBM66YOKTYENNOWG5A, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.639887296 (75h46m41.203210304s) [noise=true , address=104.36.82.10:6087 , piece_id=5FQG7BPZC373PFB54HYXSG2SVKOBBBSGZCLC4XDJEYQFOLQY74MA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.640266623 (75h46m41.203589616s) [noise=true , address=104.36.82.13:6196 , piece_id=BCHMQO5KKT5BLNWQGOHH3V43LGSE3BUQCI2SY4TTO5M5GQNOCTMQ, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.640352274 (75h46m41.203675286s) [noise=true , address=104.36.82.37:6663 , piece_id=X7LWYIIHJFU2T2PHYR3R5TNT22UUPHEHQWZH7ZJMLS4AH4PJ6AIA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.640659767 (75h46m41.203982779s) [noise=true , address=104.36.82.42:6807 , piece_id=ZBQZPXUDJ4VBCOSVKWAJ6SK6UD7SLPON7Y4SSCK2DQ5STFG4XCNQ, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.640961701 (75h46m41.20428471s) [noise=true , address=45.91.116.6:4982 , piece_id=VP45R6KXNH5U5SGUQUMAPIBY7KFDPBIR3S3VRIX27EAT6VLFBZ7Q, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.641159578 (75h46m41.204482589s) [noise=true , address=104.36.82.9:6039 , piece_id=I4AB6N27MYYUHWAZLJJUIBFBLOQFHDIM3YLYWPVRPNUV7VMYS2SA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.641588126 (75h46m41.204911136s) [noise=true , address=77.90.31.19:4853 , piece_id=VFEON5T2KSWM2KSLTVRZTXYL42JWENZYUEUPG3LMODGZWBMVGGPQ, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.641863297 (75h46m41.205186305s) [noise=true , address=77.90.31.12:4603 , piece_id=BMUIC6KNGEDL37436GLSXW5XOO6VYOPNOSFGQCYVHSENVI2PJ7KQ, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.642124427 (75h46m41.205447437s) [noise=true , address=104.36.82.49:4982 , piece_id=UNLV5UDNWULEIPE4QSQHHZQPK3BQKFRB5T3H6GP7ZQ5QGSNK5V6A, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.642285917 (75h46m41.205608925s) [noise=true , address=104.36.82.54:4171 , piece_id=MUMABKARK4KKUKWW6XDXCQVT6O7SLFHGHE7COWC7QIYGARAA2D5A, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.642656750 (75h46m41.205979758s) [noise=true , address=104.36.82.14:6232 , piece_id=A2ZLUUZKRTBJRJ267QSK36C3LFZV7FPAYDDAXGIWDTNEGNV6MRRA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.643565583 (75h46m41.206888597s) [noise=true , address=79.127.205.245:4231 , piece_id=EJK46XUUVI5NWOAQJEW4QIJG3KRXZHB4YOQUJRCS2MIOSWKGTNLQ, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.644620291 (75h46m41.207943303s) [noise=true , address=77.90.31.14:4670 , piece_id=7FNANMUKPM3N6YWRAOAIM4WJZJUTNF7ZBI6ZUBNEPGNGMFWOE24A, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.648842639 (75h46m41.212165631s) [noise=true , address=104.36.82.55:4224 , piece_id=WRVNGSJV5BX4GJERLXNPAS7J7BZ5WFTM7R5H257YRCG5SP2HIYMA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.825828239 (75h46m41.389151241s) [noise=true , address=104.36.82.53:4126 , piece_id=2MKRQJEPGDJKDM7VWY5M4M6NORILAIEYYATJ7HVZEOBOMXOI2LBA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: ecclient: upload failed (node:12dMWFrq2NMpnMzuPAphWerntC5kkrcnzRJZBA116uzJzyvk6fH, address:104.36.82.53:4126): protocol: storage node overloaded, request limit: 2000 Jun 21 20:10:05.834666043 (75h46m41.397989033s) [upload=60693]: Next returned nil segment Jun 21 20:10:05.834686295 (75h46m41.398009302s) [upload=60693]: Waiting for error group managing segments... Jun 21 20:10:05.877010488 (75h46m41.440333486s) [noise=true , address=45.91.116.6:4982 , piece_id=VP45R6KXNH5U5SGUQUMAPIBY7KFDPBIR3S3VRIX27EAT6VLFBZ7Q, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:05.886336964 (75h46m41.449659967s) [noise=true , address=104.36.82.9:6039 , piece_id=I4AB6N27MYYUHWAZLJJUIBFBLOQFHDIM3YLYWPVRPNUV7VMYS2SA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: ecclient: upload failed (node:12P11WVW9mVXw5ra2gZ3Rezon6wZi8GrN5CchQN3MiVU4ts3L6V, address:104.36.82.9:6039): protocol: storage node overloaded, request limit: 2000 Jun 21 20:10:08.064514542 (75h46m43.627837553s) [noise=true , address=77.90.31.14:4670 , piece_id=7FNANMUKPM3N6YWRAOAIM4WJZJUTNF7ZBI6ZUBNEPGNGMFWOE24A, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:08.099098391 (75h46m43.662421396s) [noise=true , address=79.127.205.245:4231 , piece_id=EJK46XUUVI5NWOAQJEW4QIJG3KRXZHB4YOQUJRCS2MIOSWKGTNLQ, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:08.489461749 (75h46m44.052784752s) [noise=true , address=104.36.82.42:6807 , piece_id=ZBQZPXUDJ4VBCOSVKWAJ6SK6UD7SLPON7Y4SSCK2DQ5STFG4XCNQ, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:08.650922093 (75h46m44.214245093s) [noise=true , address=77.90.31.19:4853 , piece_id=VFEON5T2KSWM2KSLTVRZTXYL42JWENZYUEUPG3LMODGZWBMVGGPQ, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:09.917746439 (75h46m45.481069441s) [noise=true , address=104.36.82.13:6196 , piece_id=BCHMQO5KKT5BLNWQGOHH3V43LGSE3BUQCI2SY4TTO5M5GQNOCTMQ, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:10.830722215 (75h46m46.394045216s) [noise=true , address=104.36.82.5:5869 , piece_id=HAEHXXZASV3DHT3QN7NTZOS6XZKHNUWBKCHWJJWSTJKXAKJKKBAA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:12.738783450 (75h46m48.302106452s) [noise=true , address=77.90.31.12:4603 , piece_id=BMUIC6KNGEDL37436GLSXW5XOO6VYOPNOSFGQCYVHSENVI2PJ7KQ, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:15.460927713 (75h46m51.024250723s) [noise=true , address=104.36.82.52:4111 , piece_id=HB7QAWN6CGNIIT4HD2GZ7JDQP5NXIJ5I2USBM66YOKTYENNOWG5A, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:18.393817968 (75h46m53.95714096s) [noise=true , address=104.36.82.55:4224 , piece_id=WRVNGSJV5BX4GJERLXNPAS7J7BZ5WFTM7R5H257YRCG5SP2HIYMA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:19.549174143 (75h46m55.112497151s) [noise=true , address=104.36.82.56:4253 , piece_id=5PMVGLZETMXCQ77K3I6PXACRIMBGR2CV5JGUSU5KUMYFNXVGA2GA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:22.511944593 (75h46m58.075267584s) [noise=true , address=104.36.82.37:6663 , piece_id=X7LWYIIHJFU2T2PHYR3R5TNT22UUPHEHQWZH7ZJMLS4AH4PJ6AIA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:48.027197558 (75h47m23.590520562s) [noise=true , address=104.36.82.14:6232 , piece_id=A2ZLUUZKRTBJRJ267QSK36C3LFZV7FPAYDDAXGIWDTNEGNV6MRRA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:11:05.567692582 (75h47m41.131015574s) [noise=true , address=104.36.82.49:4982 , piece_id=UNLV5UDNWULEIPE4QSQHHZQPK3BQKFRB5T3H6GP7ZQ5QGSNK5V6A, seg_pos={0 0}, upload=60693]: Done uploading piece. err: ecclient: upload failed (node:1NxoYJXCXmTvQncbkXt6yH7mhEBBrzVdzhgL5J6W5TD4N3pkaG, address:104.36.82.49:4982): protocol: expected piece hash; context deadline exceeded; EOF Jun 21 20:11:05.567759198 (75h47m41.131082195s) [noise=true , address=104.36.82.54:4171 , piece_id=MUMABKARK4KKUKWW6XDXCQVT6O7SLFHGHE7COWC7QIYGARAA2D5A, seg_pos={0 0}, upload=60693]: Done uploading piece. err: ecclient: upload failed (node:125SMhDsEZruNu6LkmQn2M7R6WzvXTdk86DRsKf522u8BNiANq3, address:104.36.82.54:4171): protocol: expected piece hash; context deadline exceeded; EOF Jun 21 20:11:05.567869019 (75h47m41.131192033s) [noise=true , address=104.36.82.10:6087 , piece_id=5FQG7BPZC373PFB54HYXSG2SVKOBBBSGZCLC4XDJEYQFOLQY74MA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: ecclient: upload failed (node:1S9k5eejJcxkiG5pXjjBZm18uP8k5jCGfWyeW2QQxmuRU8ZzRA, address:104.36.82.10:6087): protocol: write tcp 159.223.143.39:38550->104.36.82.10:6087: use of closed network connection; write tcp 159.223.143.39:38550->104.36.82.10:6087: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:38550->104.36.82.10:6087: use of closed network connection Jun 21 20:11:05.567932150 (75h47m41.131255161s) [seg_pos={0 0}, upload=60693]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jun 21 20:11:05.593626175 (75h47m41.156949181s) [upload=60693]: Done uploading segments... Jun 21 20:11:05.593657274 (75h47m41.156980265s) [upload=60693]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 19.678529138s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [.......................................X...........X........] duration: 15.830333344s FAIL (2025-06-21 20:16:18 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== pload=60693]: Starting upload Jun 21 20:10:05.602866542 (75h46m41.166189557s) [upload=60693]: Done starting upload Jun 21 20:10:05.603116545 (75h46m41.166439547s) [upload=60693]: Uploading segments... Jun 21 20:10:05.603495389 (75h46m41.166818397s) [upload=60693]: Got next segment. Inline: false Jun 21 20:10:05.603566566 (75h46m41.166889558s) [upload=60693]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 21 20:10:05.603573280 (75h46m41.166896286s) [upload=60693]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 21 20:10:05.639130162 (75h46m41.202453164s) [seg_pos={0 0}, upload=60693]: Begin upload segment... Jun 21 20:10:05.639231070 (75h46m41.202554057s) [seg_pos={0 0}, upload=60693]: Done begin upload segment. Jun 21 20:10:05.639292724 (75h46m41.202615728s) [noise=true , address=104.36.82.56:4253 , piece_id=5PMVGLZETMXCQ77K3I6PXACRIMBGR2CV5JGUSU5KUMYFNXVGA2GA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.639338066 (75h46m41.202661071s) [noise=true , address=104.36.82.53:4126 , piece_id=2MKRQJEPGDJKDM7VWY5M4M6NORILAIEYYATJ7HVZEOBOMXOI2LBA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.639416264 (75h46m41.202739267s) [noise=true , address=104.36.82.5:5869 , piece_id=HAEHXXZASV3DHT3QN7NTZOS6XZKHNUWBKCHWJJWSTJKXAKJKKBAA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.639873098 (75h46m41.203196115s) [noise=true , address=104.36.82.52:4111 , piece_id=HB7QAWN6CGNIIT4HD2GZ7JDQP5NXIJ5I2USBM66YOKTYENNOWG5A, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.639887296 (75h46m41.203210304s) [noise=true , address=104.36.82.10:6087 , piece_id=5FQG7BPZC373PFB54HYXSG2SVKOBBBSGZCLC4XDJEYQFOLQY74MA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.640266623 (75h46m41.203589616s) [noise=true , address=104.36.82.13:6196 , piece_id=BCHMQO5KKT5BLNWQGOHH3V43LGSE3BUQCI2SY4TTO5M5GQNOCTMQ, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.640352274 (75h46m41.203675286s) [noise=true , address=104.36.82.37:6663 , piece_id=X7LWYIIHJFU2T2PHYR3R5TNT22UUPHEHQWZH7ZJMLS4AH4PJ6AIA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.640659767 (75h46m41.203982779s) [noise=true , address=104.36.82.42:6807 , piece_id=ZBQZPXUDJ4VBCOSVKWAJ6SK6UD7SLPON7Y4SSCK2DQ5STFG4XCNQ, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.640961701 (75h46m41.20428471s) [noise=true , address=45.91.116.6:4982 , piece_id=VP45R6KXNH5U5SGUQUMAPIBY7KFDPBIR3S3VRIX27EAT6VLFBZ7Q, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.641159578 (75h46m41.204482589s) [noise=true , address=104.36.82.9:6039 , piece_id=I4AB6N27MYYUHWAZLJJUIBFBLOQFHDIM3YLYWPVRPNUV7VMYS2SA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.641588126 (75h46m41.204911136s) [noise=true , address=77.90.31.19:4853 , piece_id=VFEON5T2KSWM2KSLTVRZTXYL42JWENZYUEUPG3LMODGZWBMVGGPQ, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.641863297 (75h46m41.205186305s) [noise=true , address=77.90.31.12:4603 , piece_id=BMUIC6KNGEDL37436GLSXW5XOO6VYOPNOSFGQCYVHSENVI2PJ7KQ, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.642124427 (75h46m41.205447437s) [noise=true , address=104.36.82.49:4982 , piece_id=UNLV5UDNWULEIPE4QSQHHZQPK3BQKFRB5T3H6GP7ZQ5QGSNK5V6A, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.642285917 (75h46m41.205608925s) [noise=true , address=104.36.82.54:4171 , piece_id=MUMABKARK4KKUKWW6XDXCQVT6O7SLFHGHE7COWC7QIYGARAA2D5A, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.642656750 (75h46m41.205979758s) [noise=true , address=104.36.82.14:6232 , piece_id=A2ZLUUZKRTBJRJ267QSK36C3LFZV7FPAYDDAXGIWDTNEGNV6MRRA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.643565583 (75h46m41.206888597s) [noise=true , address=79.127.205.245:4231 , piece_id=EJK46XUUVI5NWOAQJEW4QIJG3KRXZHB4YOQUJRCS2MIOSWKGTNLQ, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.644620291 (75h46m41.207943303s) [noise=true , address=77.90.31.14:4670 , piece_id=7FNANMUKPM3N6YWRAOAIM4WJZJUTNF7ZBI6ZUBNEPGNGMFWOE24A, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.648842639 (75h46m41.212165631s) [noise=true , address=104.36.82.55:4224 , piece_id=WRVNGSJV5BX4GJERLXNPAS7J7BZ5WFTM7R5H257YRCG5SP2HIYMA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.825828239 (75h46m41.389151241s) [noise=true , address=104.36.82.53:4126 , piece_id=2MKRQJEPGDJKDM7VWY5M4M6NORILAIEYYATJ7HVZEOBOMXOI2LBA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: ecclient: upload failed (node:12dMWFrq2NMpnMzuPAphWerntC5kkrcnzRJZBA116uzJzyvk6fH, address:104.36.82.53:4126): protocol: storage node overloaded, request limit: 2000 Jun 21 20:10:05.834666043 (75h46m41.397989033s) [upload=60693]: Next returned nil segment Jun 21 20:10:05.834686295 (75h46m41.398009302s) [upload=60693]: Waiting for error group managing segments... Jun 21 20:10:05.877010488 (75h46m41.440333486s) [noise=true , address=45.91.116.6:4982 , piece_id=VP45R6KXNH5U5SGUQUMAPIBY7KFDPBIR3S3VRIX27EAT6VLFBZ7Q, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:05.886336964 (75h46m41.449659967s) [noise=true , address=104.36.82.9:6039 , piece_id=I4AB6N27MYYUHWAZLJJUIBFBLOQFHDIM3YLYWPVRPNUV7VMYS2SA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: ecclient: upload failed (node:12P11WVW9mVXw5ra2gZ3Rezon6wZi8GrN5CchQN3MiVU4ts3L6V, address:104.36.82.9:6039): protocol: storage node overloaded, request limit: 2000 Jun 21 20:10:08.064514542 (75h46m43.627837553s) [noise=true , address=77.90.31.14:4670 , piece_id=7FNANMUKPM3N6YWRAOAIM4WJZJUTNF7ZBI6ZUBNEPGNGMFWOE24A, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:08.099098391 (75h46m43.662421396s) [noise=true , address=79.127.205.245:4231 , piece_id=EJK46XUUVI5NWOAQJEW4QIJG3KRXZHB4YOQUJRCS2MIOSWKGTNLQ, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:08.489461749 (75h46m44.052784752s) [noise=true , address=104.36.82.42:6807 , piece_id=ZBQZPXUDJ4VBCOSVKWAJ6SK6UD7SLPON7Y4SSCK2DQ5STFG4XCNQ, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:08.650922093 (75h46m44.214245093s) [noise=true , address=77.90.31.19:4853 , piece_id=VFEON5T2KSWM2KSLTVRZTXYL42JWENZYUEUPG3LMODGZWBMVGGPQ, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:09.917746439 (75h46m45.481069441s) [noise=true , address=104.36.82.13:6196 , piece_id=BCHMQO5KKT5BLNWQGOHH3V43LGSE3BUQCI2SY4TTO5M5GQNOCTMQ, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:10.830722215 (75h46m46.394045216s) [noise=true , address=104.36.82.5:5869 , piece_id=HAEHXXZASV3DHT3QN7NTZOS6XZKHNUWBKCHWJJWSTJKXAKJKKBAA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:12.738783450 (75h46m48.302106452s) [noise=true , address=77.90.31.12:4603 , piece_id=BMUIC6KNGEDL37436GLSXW5XOO6VYOPNOSFGQCYVHSENVI2PJ7KQ, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:15.460927713 (75h46m51.024250723s) [noise=true , address=104.36.82.52:4111 , piece_id=HB7QAWN6CGNIIT4HD2GZ7JDQP5NXIJ5I2USBM66YOKTYENNOWG5A, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:18.393817968 (75h46m53.95714096s) [noise=true , address=104.36.82.55:4224 , piece_id=WRVNGSJV5BX4GJERLXNPAS7J7BZ5WFTM7R5H257YRCG5SP2HIYMA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:19.549174143 (75h46m55.112497151s) [noise=true , address=104.36.82.56:4253 , piece_id=5PMVGLZETMXCQ77K3I6PXACRIMBGR2CV5JGUSU5KUMYFNXVGA2GA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:22.511944593 (75h46m58.075267584s) [noise=true , address=104.36.82.37:6663 , piece_id=X7LWYIIHJFU2T2PHYR3R5TNT22UUPHEHQWZH7ZJMLS4AH4PJ6AIA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:48.027197558 (75h47m23.590520562s) [noise=true , address=104.36.82.14:6232 , piece_id=A2ZLUUZKRTBJRJ267QSK36C3LFZV7FPAYDDAXGIWDTNEGNV6MRRA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:11:05.567692582 (75h47m41.131015574s) [noise=true , address=104.36.82.49:4982 , piece_id=UNLV5UDNWULEIPE4QSQHHZQPK3BQKFRB5T3H6GP7ZQ5QGSNK5V6A, seg_pos={0 0}, upload=60693]: Done uploading piece. err: ecclient: upload failed (node:1NxoYJXCXmTvQncbkXt6yH7mhEBBrzVdzhgL5J6W5TD4N3pkaG, address:104.36.82.49:4982): protocol: expected piece hash; context deadline exceeded; EOF Jun 21 20:11:05.567759198 (75h47m41.131082195s) [noise=true , address=104.36.82.54:4171 , piece_id=MUMABKARK4KKUKWW6XDXCQVT6O7SLFHGHE7COWC7QIYGARAA2D5A, seg_pos={0 0}, upload=60693]: Done uploading piece. err: ecclient: upload failed (node:125SMhDsEZruNu6LkmQn2M7R6WzvXTdk86DRsKf522u8BNiANq3, address:104.36.82.54:4171): protocol: expected piece hash; context deadline exceeded; EOF Jun 21 20:11:05.567869019 (75h47m41.131192033s) [noise=true , address=104.36.82.10:6087 , piece_id=5FQG7BPZC373PFB54HYXSG2SVKOBBBSGZCLC4XDJEYQFOLQY74MA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: ecclient: upload failed (node:1S9k5eejJcxkiG5pXjjBZm18uP8k5jCGfWyeW2QQxmuRU8ZzRA, address:104.36.82.10:6087): protocol: write tcp 159.223.143.39:38550->104.36.82.10:6087: use of closed network connection; write tcp 159.223.143.39:38550->104.36.82.10:6087: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:38550->104.36.82.10:6087: use of closed network connection Jun 21 20:11:05.567932150 (75h47m41.131255161s) [seg_pos={0 0}, upload=60693]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jun 21 20:11:05.593626175 (75h47m41.156949181s) [upload=60693]: Done uploading segments... Jun 21 20:11:05.593657274 (75h47m41.156980265s) [upload=60693]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 15.83036763s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [........................................X...........X.......] duration: 18.899167806s FAIL (2025-06-21 20:15:57 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== pload=60693]: Starting upload Jun 21 20:10:05.602866542 (75h46m41.166189557s) [upload=60693]: Done starting upload Jun 21 20:10:05.603116545 (75h46m41.166439547s) [upload=60693]: Uploading segments... Jun 21 20:10:05.603495389 (75h46m41.166818397s) [upload=60693]: Got next segment. Inline: false Jun 21 20:10:05.603566566 (75h46m41.166889558s) [upload=60693]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 21 20:10:05.603573280 (75h46m41.166896286s) [upload=60693]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 21 20:10:05.639130162 (75h46m41.202453164s) [seg_pos={0 0}, upload=60693]: Begin upload segment... Jun 21 20:10:05.639231070 (75h46m41.202554057s) [seg_pos={0 0}, upload=60693]: Done begin upload segment. Jun 21 20:10:05.639292724 (75h46m41.202615728s) [noise=true , address=104.36.82.56:4253 , piece_id=5PMVGLZETMXCQ77K3I6PXACRIMBGR2CV5JGUSU5KUMYFNXVGA2GA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.639338066 (75h46m41.202661071s) [noise=true , address=104.36.82.53:4126 , piece_id=2MKRQJEPGDJKDM7VWY5M4M6NORILAIEYYATJ7HVZEOBOMXOI2LBA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.639416264 (75h46m41.202739267s) [noise=true , address=104.36.82.5:5869 , piece_id=HAEHXXZASV3DHT3QN7NTZOS6XZKHNUWBKCHWJJWSTJKXAKJKKBAA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.639873098 (75h46m41.203196115s) [noise=true , address=104.36.82.52:4111 , piece_id=HB7QAWN6CGNIIT4HD2GZ7JDQP5NXIJ5I2USBM66YOKTYENNOWG5A, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.639887296 (75h46m41.203210304s) [noise=true , address=104.36.82.10:6087 , piece_id=5FQG7BPZC373PFB54HYXSG2SVKOBBBSGZCLC4XDJEYQFOLQY74MA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.640266623 (75h46m41.203589616s) [noise=true , address=104.36.82.13:6196 , piece_id=BCHMQO5KKT5BLNWQGOHH3V43LGSE3BUQCI2SY4TTO5M5GQNOCTMQ, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.640352274 (75h46m41.203675286s) [noise=true , address=104.36.82.37:6663 , piece_id=X7LWYIIHJFU2T2PHYR3R5TNT22UUPHEHQWZH7ZJMLS4AH4PJ6AIA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.640659767 (75h46m41.203982779s) [noise=true , address=104.36.82.42:6807 , piece_id=ZBQZPXUDJ4VBCOSVKWAJ6SK6UD7SLPON7Y4SSCK2DQ5STFG4XCNQ, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.640961701 (75h46m41.20428471s) [noise=true , address=45.91.116.6:4982 , piece_id=VP45R6KXNH5U5SGUQUMAPIBY7KFDPBIR3S3VRIX27EAT6VLFBZ7Q, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.641159578 (75h46m41.204482589s) [noise=true , address=104.36.82.9:6039 , piece_id=I4AB6N27MYYUHWAZLJJUIBFBLOQFHDIM3YLYWPVRPNUV7VMYS2SA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.641588126 (75h46m41.204911136s) [noise=true , address=77.90.31.19:4853 , piece_id=VFEON5T2KSWM2KSLTVRZTXYL42JWENZYUEUPG3LMODGZWBMVGGPQ, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.641863297 (75h46m41.205186305s) [noise=true , address=77.90.31.12:4603 , piece_id=BMUIC6KNGEDL37436GLSXW5XOO6VYOPNOSFGQCYVHSENVI2PJ7KQ, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.642124427 (75h46m41.205447437s) [noise=true , address=104.36.82.49:4982 , piece_id=UNLV5UDNWULEIPE4QSQHHZQPK3BQKFRB5T3H6GP7ZQ5QGSNK5V6A, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.642285917 (75h46m41.205608925s) [noise=true , address=104.36.82.54:4171 , piece_id=MUMABKARK4KKUKWW6XDXCQVT6O7SLFHGHE7COWC7QIYGARAA2D5A, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.642656750 (75h46m41.205979758s) [noise=true , address=104.36.82.14:6232 , piece_id=A2ZLUUZKRTBJRJ267QSK36C3LFZV7FPAYDDAXGIWDTNEGNV6MRRA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.643565583 (75h46m41.206888597s) [noise=true , address=79.127.205.245:4231 , piece_id=EJK46XUUVI5NWOAQJEW4QIJG3KRXZHB4YOQUJRCS2MIOSWKGTNLQ, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.644620291 (75h46m41.207943303s) [noise=true , address=77.90.31.14:4670 , piece_id=7FNANMUKPM3N6YWRAOAIM4WJZJUTNF7ZBI6ZUBNEPGNGMFWOE24A, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.648842639 (75h46m41.212165631s) [noise=true , address=104.36.82.55:4224 , piece_id=WRVNGSJV5BX4GJERLXNPAS7J7BZ5WFTM7R5H257YRCG5SP2HIYMA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.825828239 (75h46m41.389151241s) [noise=true , address=104.36.82.53:4126 , piece_id=2MKRQJEPGDJKDM7VWY5M4M6NORILAIEYYATJ7HVZEOBOMXOI2LBA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: ecclient: upload failed (node:12dMWFrq2NMpnMzuPAphWerntC5kkrcnzRJZBA116uzJzyvk6fH, address:104.36.82.53:4126): protocol: storage node overloaded, request limit: 2000 Jun 21 20:10:05.834666043 (75h46m41.397989033s) [upload=60693]: Next returned nil segment Jun 21 20:10:05.834686295 (75h46m41.398009302s) [upload=60693]: Waiting for error group managing segments... Jun 21 20:10:05.877010488 (75h46m41.440333486s) [noise=true , address=45.91.116.6:4982 , piece_id=VP45R6KXNH5U5SGUQUMAPIBY7KFDPBIR3S3VRIX27EAT6VLFBZ7Q, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:05.886336964 (75h46m41.449659967s) [noise=true , address=104.36.82.9:6039 , piece_id=I4AB6N27MYYUHWAZLJJUIBFBLOQFHDIM3YLYWPVRPNUV7VMYS2SA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: ecclient: upload failed (node:12P11WVW9mVXw5ra2gZ3Rezon6wZi8GrN5CchQN3MiVU4ts3L6V, address:104.36.82.9:6039): protocol: storage node overloaded, request limit: 2000 Jun 21 20:10:08.064514542 (75h46m43.627837553s) [noise=true , address=77.90.31.14:4670 , piece_id=7FNANMUKPM3N6YWRAOAIM4WJZJUTNF7ZBI6ZUBNEPGNGMFWOE24A, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:08.099098391 (75h46m43.662421396s) [noise=true , address=79.127.205.245:4231 , piece_id=EJK46XUUVI5NWOAQJEW4QIJG3KRXZHB4YOQUJRCS2MIOSWKGTNLQ, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:08.489461749 (75h46m44.052784752s) [noise=true , address=104.36.82.42:6807 , piece_id=ZBQZPXUDJ4VBCOSVKWAJ6SK6UD7SLPON7Y4SSCK2DQ5STFG4XCNQ, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:08.650922093 (75h46m44.214245093s) [noise=true , address=77.90.31.19:4853 , piece_id=VFEON5T2KSWM2KSLTVRZTXYL42JWENZYUEUPG3LMODGZWBMVGGPQ, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:09.917746439 (75h46m45.481069441s) [noise=true , address=104.36.82.13:6196 , piece_id=BCHMQO5KKT5BLNWQGOHH3V43LGSE3BUQCI2SY4TTO5M5GQNOCTMQ, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:10.830722215 (75h46m46.394045216s) [noise=true , address=104.36.82.5:5869 , piece_id=HAEHXXZASV3DHT3QN7NTZOS6XZKHNUWBKCHWJJWSTJKXAKJKKBAA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:12.738783450 (75h46m48.302106452s) [noise=true , address=77.90.31.12:4603 , piece_id=BMUIC6KNGEDL37436GLSXW5XOO6VYOPNOSFGQCYVHSENVI2PJ7KQ, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:15.460927713 (75h46m51.024250723s) [noise=true , address=104.36.82.52:4111 , piece_id=HB7QAWN6CGNIIT4HD2GZ7JDQP5NXIJ5I2USBM66YOKTYENNOWG5A, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:18.393817968 (75h46m53.95714096s) [noise=true , address=104.36.82.55:4224 , piece_id=WRVNGSJV5BX4GJERLXNPAS7J7BZ5WFTM7R5H257YRCG5SP2HIYMA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:19.549174143 (75h46m55.112497151s) [noise=true , address=104.36.82.56:4253 , piece_id=5PMVGLZETMXCQ77K3I6PXACRIMBGR2CV5JGUSU5KUMYFNXVGA2GA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:22.511944593 (75h46m58.075267584s) [noise=true , address=104.36.82.37:6663 , piece_id=X7LWYIIHJFU2T2PHYR3R5TNT22UUPHEHQWZH7ZJMLS4AH4PJ6AIA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:48.027197558 (75h47m23.590520562s) [noise=true , address=104.36.82.14:6232 , piece_id=A2ZLUUZKRTBJRJ267QSK36C3LFZV7FPAYDDAXGIWDTNEGNV6MRRA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:11:05.567692582 (75h47m41.131015574s) [noise=true , address=104.36.82.49:4982 , piece_id=UNLV5UDNWULEIPE4QSQHHZQPK3BQKFRB5T3H6GP7ZQ5QGSNK5V6A, seg_pos={0 0}, upload=60693]: Done uploading piece. err: ecclient: upload failed (node:1NxoYJXCXmTvQncbkXt6yH7mhEBBrzVdzhgL5J6W5TD4N3pkaG, address:104.36.82.49:4982): protocol: expected piece hash; context deadline exceeded; EOF Jun 21 20:11:05.567759198 (75h47m41.131082195s) [noise=true , address=104.36.82.54:4171 , piece_id=MUMABKARK4KKUKWW6XDXCQVT6O7SLFHGHE7COWC7QIYGARAA2D5A, seg_pos={0 0}, upload=60693]: Done uploading piece. err: ecclient: upload failed (node:125SMhDsEZruNu6LkmQn2M7R6WzvXTdk86DRsKf522u8BNiANq3, address:104.36.82.54:4171): protocol: expected piece hash; context deadline exceeded; EOF Jun 21 20:11:05.567869019 (75h47m41.131192033s) [noise=true , address=104.36.82.10:6087 , piece_id=5FQG7BPZC373PFB54HYXSG2SVKOBBBSGZCLC4XDJEYQFOLQY74MA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: ecclient: upload failed (node:1S9k5eejJcxkiG5pXjjBZm18uP8k5jCGfWyeW2QQxmuRU8ZzRA, address:104.36.82.10:6087): protocol: write tcp 159.223.143.39:38550->104.36.82.10:6087: use of closed network connection; write tcp 159.223.143.39:38550->104.36.82.10:6087: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:38550->104.36.82.10:6087: use of closed network connection Jun 21 20:11:05.567932150 (75h47m41.131255161s) [seg_pos={0 0}, upload=60693]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jun 21 20:11:05.593626175 (75h47m41.156949181s) [upload=60693]: Done uploading segments... Jun 21 20:11:05.593657274 (75h47m41.156980265s) [upload=60693]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 18.899192573s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [.........................................X...........X......] duration: 29.201249838s FAIL (2025-06-21 20:14:54 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== pload=60693]: Starting upload Jun 21 20:10:05.602866542 (75h46m41.166189557s) [upload=60693]: Done starting upload Jun 21 20:10:05.603116545 (75h46m41.166439547s) [upload=60693]: Uploading segments... Jun 21 20:10:05.603495389 (75h46m41.166818397s) [upload=60693]: Got next segment. Inline: false Jun 21 20:10:05.603566566 (75h46m41.166889558s) [upload=60693]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 21 20:10:05.603573280 (75h46m41.166896286s) [upload=60693]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 21 20:10:05.639130162 (75h46m41.202453164s) [seg_pos={0 0}, upload=60693]: Begin upload segment... Jun 21 20:10:05.639231070 (75h46m41.202554057s) [seg_pos={0 0}, upload=60693]: Done begin upload segment. Jun 21 20:10:05.639292724 (75h46m41.202615728s) [noise=true , address=104.36.82.56:4253 , piece_id=5PMVGLZETMXCQ77K3I6PXACRIMBGR2CV5JGUSU5KUMYFNXVGA2GA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.639338066 (75h46m41.202661071s) [noise=true , address=104.36.82.53:4126 , piece_id=2MKRQJEPGDJKDM7VWY5M4M6NORILAIEYYATJ7HVZEOBOMXOI2LBA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.639416264 (75h46m41.202739267s) [noise=true , address=104.36.82.5:5869 , piece_id=HAEHXXZASV3DHT3QN7NTZOS6XZKHNUWBKCHWJJWSTJKXAKJKKBAA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.639873098 (75h46m41.203196115s) [noise=true , address=104.36.82.52:4111 , piece_id=HB7QAWN6CGNIIT4HD2GZ7JDQP5NXIJ5I2USBM66YOKTYENNOWG5A, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.639887296 (75h46m41.203210304s) [noise=true , address=104.36.82.10:6087 , piece_id=5FQG7BPZC373PFB54HYXSG2SVKOBBBSGZCLC4XDJEYQFOLQY74MA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.640266623 (75h46m41.203589616s) [noise=true , address=104.36.82.13:6196 , piece_id=BCHMQO5KKT5BLNWQGOHH3V43LGSE3BUQCI2SY4TTO5M5GQNOCTMQ, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.640352274 (75h46m41.203675286s) [noise=true , address=104.36.82.37:6663 , piece_id=X7LWYIIHJFU2T2PHYR3R5TNT22UUPHEHQWZH7ZJMLS4AH4PJ6AIA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.640659767 (75h46m41.203982779s) [noise=true , address=104.36.82.42:6807 , piece_id=ZBQZPXUDJ4VBCOSVKWAJ6SK6UD7SLPON7Y4SSCK2DQ5STFG4XCNQ, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.640961701 (75h46m41.20428471s) [noise=true , address=45.91.116.6:4982 , piece_id=VP45R6KXNH5U5SGUQUMAPIBY7KFDPBIR3S3VRIX27EAT6VLFBZ7Q, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.641159578 (75h46m41.204482589s) [noise=true , address=104.36.82.9:6039 , piece_id=I4AB6N27MYYUHWAZLJJUIBFBLOQFHDIM3YLYWPVRPNUV7VMYS2SA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.641588126 (75h46m41.204911136s) [noise=true , address=77.90.31.19:4853 , piece_id=VFEON5T2KSWM2KSLTVRZTXYL42JWENZYUEUPG3LMODGZWBMVGGPQ, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.641863297 (75h46m41.205186305s) [noise=true , address=77.90.31.12:4603 , piece_id=BMUIC6KNGEDL37436GLSXW5XOO6VYOPNOSFGQCYVHSENVI2PJ7KQ, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.642124427 (75h46m41.205447437s) [noise=true , address=104.36.82.49:4982 , piece_id=UNLV5UDNWULEIPE4QSQHHZQPK3BQKFRB5T3H6GP7ZQ5QGSNK5V6A, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.642285917 (75h46m41.205608925s) [noise=true , address=104.36.82.54:4171 , piece_id=MUMABKARK4KKUKWW6XDXCQVT6O7SLFHGHE7COWC7QIYGARAA2D5A, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.642656750 (75h46m41.205979758s) [noise=true , address=104.36.82.14:6232 , piece_id=A2ZLUUZKRTBJRJ267QSK36C3LFZV7FPAYDDAXGIWDTNEGNV6MRRA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.643565583 (75h46m41.206888597s) [noise=true , address=79.127.205.245:4231 , piece_id=EJK46XUUVI5NWOAQJEW4QIJG3KRXZHB4YOQUJRCS2MIOSWKGTNLQ, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.644620291 (75h46m41.207943303s) [noise=true , address=77.90.31.14:4670 , piece_id=7FNANMUKPM3N6YWRAOAIM4WJZJUTNF7ZBI6ZUBNEPGNGMFWOE24A, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.648842639 (75h46m41.212165631s) [noise=true , address=104.36.82.55:4224 , piece_id=WRVNGSJV5BX4GJERLXNPAS7J7BZ5WFTM7R5H257YRCG5SP2HIYMA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.825828239 (75h46m41.389151241s) [noise=true , address=104.36.82.53:4126 , piece_id=2MKRQJEPGDJKDM7VWY5M4M6NORILAIEYYATJ7HVZEOBOMXOI2LBA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: ecclient: upload failed (node:12dMWFrq2NMpnMzuPAphWerntC5kkrcnzRJZBA116uzJzyvk6fH, address:104.36.82.53:4126): protocol: storage node overloaded, request limit: 2000 Jun 21 20:10:05.834666043 (75h46m41.397989033s) [upload=60693]: Next returned nil segment Jun 21 20:10:05.834686295 (75h46m41.398009302s) [upload=60693]: Waiting for error group managing segments... Jun 21 20:10:05.877010488 (75h46m41.440333486s) [noise=true , address=45.91.116.6:4982 , piece_id=VP45R6KXNH5U5SGUQUMAPIBY7KFDPBIR3S3VRIX27EAT6VLFBZ7Q, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:05.886336964 (75h46m41.449659967s) [noise=true , address=104.36.82.9:6039 , piece_id=I4AB6N27MYYUHWAZLJJUIBFBLOQFHDIM3YLYWPVRPNUV7VMYS2SA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: ecclient: upload failed (node:12P11WVW9mVXw5ra2gZ3Rezon6wZi8GrN5CchQN3MiVU4ts3L6V, address:104.36.82.9:6039): protocol: storage node overloaded, request limit: 2000 Jun 21 20:10:08.064514542 (75h46m43.627837553s) [noise=true , address=77.90.31.14:4670 , piece_id=7FNANMUKPM3N6YWRAOAIM4WJZJUTNF7ZBI6ZUBNEPGNGMFWOE24A, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:08.099098391 (75h46m43.662421396s) [noise=true , address=79.127.205.245:4231 , piece_id=EJK46XUUVI5NWOAQJEW4QIJG3KRXZHB4YOQUJRCS2MIOSWKGTNLQ, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:08.489461749 (75h46m44.052784752s) [noise=true , address=104.36.82.42:6807 , piece_id=ZBQZPXUDJ4VBCOSVKWAJ6SK6UD7SLPON7Y4SSCK2DQ5STFG4XCNQ, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:08.650922093 (75h46m44.214245093s) [noise=true , address=77.90.31.19:4853 , piece_id=VFEON5T2KSWM2KSLTVRZTXYL42JWENZYUEUPG3LMODGZWBMVGGPQ, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:09.917746439 (75h46m45.481069441s) [noise=true , address=104.36.82.13:6196 , piece_id=BCHMQO5KKT5BLNWQGOHH3V43LGSE3BUQCI2SY4TTO5M5GQNOCTMQ, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:10.830722215 (75h46m46.394045216s) [noise=true , address=104.36.82.5:5869 , piece_id=HAEHXXZASV3DHT3QN7NTZOS6XZKHNUWBKCHWJJWSTJKXAKJKKBAA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:12.738783450 (75h46m48.302106452s) [noise=true , address=77.90.31.12:4603 , piece_id=BMUIC6KNGEDL37436GLSXW5XOO6VYOPNOSFGQCYVHSENVI2PJ7KQ, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:15.460927713 (75h46m51.024250723s) [noise=true , address=104.36.82.52:4111 , piece_id=HB7QAWN6CGNIIT4HD2GZ7JDQP5NXIJ5I2USBM66YOKTYENNOWG5A, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:18.393817968 (75h46m53.95714096s) [noise=true , address=104.36.82.55:4224 , piece_id=WRVNGSJV5BX4GJERLXNPAS7J7BZ5WFTM7R5H257YRCG5SP2HIYMA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:19.549174143 (75h46m55.112497151s) [noise=true , address=104.36.82.56:4253 , piece_id=5PMVGLZETMXCQ77K3I6PXACRIMBGR2CV5JGUSU5KUMYFNXVGA2GA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:22.511944593 (75h46m58.075267584s) [noise=true , address=104.36.82.37:6663 , piece_id=X7LWYIIHJFU2T2PHYR3R5TNT22UUPHEHQWZH7ZJMLS4AH4PJ6AIA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:48.027197558 (75h47m23.590520562s) [noise=true , address=104.36.82.14:6232 , piece_id=A2ZLUUZKRTBJRJ267QSK36C3LFZV7FPAYDDAXGIWDTNEGNV6MRRA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:11:05.567692582 (75h47m41.131015574s) [noise=true , address=104.36.82.49:4982 , piece_id=UNLV5UDNWULEIPE4QSQHHZQPK3BQKFRB5T3H6GP7ZQ5QGSNK5V6A, seg_pos={0 0}, upload=60693]: Done uploading piece. err: ecclient: upload failed (node:1NxoYJXCXmTvQncbkXt6yH7mhEBBrzVdzhgL5J6W5TD4N3pkaG, address:104.36.82.49:4982): protocol: expected piece hash; context deadline exceeded; EOF Jun 21 20:11:05.567759198 (75h47m41.131082195s) [noise=true , address=104.36.82.54:4171 , piece_id=MUMABKARK4KKUKWW6XDXCQVT6O7SLFHGHE7COWC7QIYGARAA2D5A, seg_pos={0 0}, upload=60693]: Done uploading piece. err: ecclient: upload failed (node:125SMhDsEZruNu6LkmQn2M7R6WzvXTdk86DRsKf522u8BNiANq3, address:104.36.82.54:4171): protocol: expected piece hash; context deadline exceeded; EOF Jun 21 20:11:05.567869019 (75h47m41.131192033s) [noise=true , address=104.36.82.10:6087 , piece_id=5FQG7BPZC373PFB54HYXSG2SVKOBBBSGZCLC4XDJEYQFOLQY74MA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: ecclient: upload failed (node:1S9k5eejJcxkiG5pXjjBZm18uP8k5jCGfWyeW2QQxmuRU8ZzRA, address:104.36.82.10:6087): protocol: write tcp 159.223.143.39:38550->104.36.82.10:6087: use of closed network connection; write tcp 159.223.143.39:38550->104.36.82.10:6087: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:38550->104.36.82.10:6087: use of closed network connection Jun 21 20:11:05.567932150 (75h47m41.131255161s) [seg_pos={0 0}, upload=60693]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jun 21 20:11:05.593626175 (75h47m41.156949181s) [upload=60693]: Done uploading segments... Jun 21 20:11:05.593657274 (75h47m41.156980265s) [upload=60693]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 29.201276461s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [..........................................X...........X.....] duration: 9.443636861s FAIL (2025-06-21 20:14:12 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== pload=60693]: Starting upload Jun 21 20:10:05.602866542 (75h46m41.166189557s) [upload=60693]: Done starting upload Jun 21 20:10:05.603116545 (75h46m41.166439547s) [upload=60693]: Uploading segments... Jun 21 20:10:05.603495389 (75h46m41.166818397s) [upload=60693]: Got next segment. Inline: false Jun 21 20:10:05.603566566 (75h46m41.166889558s) [upload=60693]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 21 20:10:05.603573280 (75h46m41.166896286s) [upload=60693]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 21 20:10:05.639130162 (75h46m41.202453164s) [seg_pos={0 0}, upload=60693]: Begin upload segment... Jun 21 20:10:05.639231070 (75h46m41.202554057s) [seg_pos={0 0}, upload=60693]: Done begin upload segment. Jun 21 20:10:05.639292724 (75h46m41.202615728s) [noise=true , address=104.36.82.56:4253 , piece_id=5PMVGLZETMXCQ77K3I6PXACRIMBGR2CV5JGUSU5KUMYFNXVGA2GA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.639338066 (75h46m41.202661071s) [noise=true , address=104.36.82.53:4126 , piece_id=2MKRQJEPGDJKDM7VWY5M4M6NORILAIEYYATJ7HVZEOBOMXOI2LBA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.639416264 (75h46m41.202739267s) [noise=true , address=104.36.82.5:5869 , piece_id=HAEHXXZASV3DHT3QN7NTZOS6XZKHNUWBKCHWJJWSTJKXAKJKKBAA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.639873098 (75h46m41.203196115s) [noise=true , address=104.36.82.52:4111 , piece_id=HB7QAWN6CGNIIT4HD2GZ7JDQP5NXIJ5I2USBM66YOKTYENNOWG5A, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.639887296 (75h46m41.203210304s) [noise=true , address=104.36.82.10:6087 , piece_id=5FQG7BPZC373PFB54HYXSG2SVKOBBBSGZCLC4XDJEYQFOLQY74MA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.640266623 (75h46m41.203589616s) [noise=true , address=104.36.82.13:6196 , piece_id=BCHMQO5KKT5BLNWQGOHH3V43LGSE3BUQCI2SY4TTO5M5GQNOCTMQ, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.640352274 (75h46m41.203675286s) [noise=true , address=104.36.82.37:6663 , piece_id=X7LWYIIHJFU2T2PHYR3R5TNT22UUPHEHQWZH7ZJMLS4AH4PJ6AIA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.640659767 (75h46m41.203982779s) [noise=true , address=104.36.82.42:6807 , piece_id=ZBQZPXUDJ4VBCOSVKWAJ6SK6UD7SLPON7Y4SSCK2DQ5STFG4XCNQ, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.640961701 (75h46m41.20428471s) [noise=true , address=45.91.116.6:4982 , piece_id=VP45R6KXNH5U5SGUQUMAPIBY7KFDPBIR3S3VRIX27EAT6VLFBZ7Q, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.641159578 (75h46m41.204482589s) [noise=true , address=104.36.82.9:6039 , piece_id=I4AB6N27MYYUHWAZLJJUIBFBLOQFHDIM3YLYWPVRPNUV7VMYS2SA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.641588126 (75h46m41.204911136s) [noise=true , address=77.90.31.19:4853 , piece_id=VFEON5T2KSWM2KSLTVRZTXYL42JWENZYUEUPG3LMODGZWBMVGGPQ, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.641863297 (75h46m41.205186305s) [noise=true , address=77.90.31.12:4603 , piece_id=BMUIC6KNGEDL37436GLSXW5XOO6VYOPNOSFGQCYVHSENVI2PJ7KQ, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.642124427 (75h46m41.205447437s) [noise=true , address=104.36.82.49:4982 , piece_id=UNLV5UDNWULEIPE4QSQHHZQPK3BQKFRB5T3H6GP7ZQ5QGSNK5V6A, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.642285917 (75h46m41.205608925s) [noise=true , address=104.36.82.54:4171 , piece_id=MUMABKARK4KKUKWW6XDXCQVT6O7SLFHGHE7COWC7QIYGARAA2D5A, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.642656750 (75h46m41.205979758s) [noise=true , address=104.36.82.14:6232 , piece_id=A2ZLUUZKRTBJRJ267QSK36C3LFZV7FPAYDDAXGIWDTNEGNV6MRRA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.643565583 (75h46m41.206888597s) [noise=true , address=79.127.205.245:4231 , piece_id=EJK46XUUVI5NWOAQJEW4QIJG3KRXZHB4YOQUJRCS2MIOSWKGTNLQ, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.644620291 (75h46m41.207943303s) [noise=true , address=77.90.31.14:4670 , piece_id=7FNANMUKPM3N6YWRAOAIM4WJZJUTNF7ZBI6ZUBNEPGNGMFWOE24A, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.648842639 (75h46m41.212165631s) [noise=true , address=104.36.82.55:4224 , piece_id=WRVNGSJV5BX4GJERLXNPAS7J7BZ5WFTM7R5H257YRCG5SP2HIYMA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.825828239 (75h46m41.389151241s) [noise=true , address=104.36.82.53:4126 , piece_id=2MKRQJEPGDJKDM7VWY5M4M6NORILAIEYYATJ7HVZEOBOMXOI2LBA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: ecclient: upload failed (node:12dMWFrq2NMpnMzuPAphWerntC5kkrcnzRJZBA116uzJzyvk6fH, address:104.36.82.53:4126): protocol: storage node overloaded, request limit: 2000 Jun 21 20:10:05.834666043 (75h46m41.397989033s) [upload=60693]: Next returned nil segment Jun 21 20:10:05.834686295 (75h46m41.398009302s) [upload=60693]: Waiting for error group managing segments... Jun 21 20:10:05.877010488 (75h46m41.440333486s) [noise=true , address=45.91.116.6:4982 , piece_id=VP45R6KXNH5U5SGUQUMAPIBY7KFDPBIR3S3VRIX27EAT6VLFBZ7Q, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:05.886336964 (75h46m41.449659967s) [noise=true , address=104.36.82.9:6039 , piece_id=I4AB6N27MYYUHWAZLJJUIBFBLOQFHDIM3YLYWPVRPNUV7VMYS2SA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: ecclient: upload failed (node:12P11WVW9mVXw5ra2gZ3Rezon6wZi8GrN5CchQN3MiVU4ts3L6V, address:104.36.82.9:6039): protocol: storage node overloaded, request limit: 2000 Jun 21 20:10:08.064514542 (75h46m43.627837553s) [noise=true , address=77.90.31.14:4670 , piece_id=7FNANMUKPM3N6YWRAOAIM4WJZJUTNF7ZBI6ZUBNEPGNGMFWOE24A, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:08.099098391 (75h46m43.662421396s) [noise=true , address=79.127.205.245:4231 , piece_id=EJK46XUUVI5NWOAQJEW4QIJG3KRXZHB4YOQUJRCS2MIOSWKGTNLQ, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:08.489461749 (75h46m44.052784752s) [noise=true , address=104.36.82.42:6807 , piece_id=ZBQZPXUDJ4VBCOSVKWAJ6SK6UD7SLPON7Y4SSCK2DQ5STFG4XCNQ, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:08.650922093 (75h46m44.214245093s) [noise=true , address=77.90.31.19:4853 , piece_id=VFEON5T2KSWM2KSLTVRZTXYL42JWENZYUEUPG3LMODGZWBMVGGPQ, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:09.917746439 (75h46m45.481069441s) [noise=true , address=104.36.82.13:6196 , piece_id=BCHMQO5KKT5BLNWQGOHH3V43LGSE3BUQCI2SY4TTO5M5GQNOCTMQ, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:10.830722215 (75h46m46.394045216s) [noise=true , address=104.36.82.5:5869 , piece_id=HAEHXXZASV3DHT3QN7NTZOS6XZKHNUWBKCHWJJWSTJKXAKJKKBAA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:12.738783450 (75h46m48.302106452s) [noise=true , address=77.90.31.12:4603 , piece_id=BMUIC6KNGEDL37436GLSXW5XOO6VYOPNOSFGQCYVHSENVI2PJ7KQ, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:15.460927713 (75h46m51.024250723s) [noise=true , address=104.36.82.52:4111 , piece_id=HB7QAWN6CGNIIT4HD2GZ7JDQP5NXIJ5I2USBM66YOKTYENNOWG5A, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:18.393817968 (75h46m53.95714096s) [noise=true , address=104.36.82.55:4224 , piece_id=WRVNGSJV5BX4GJERLXNPAS7J7BZ5WFTM7R5H257YRCG5SP2HIYMA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:19.549174143 (75h46m55.112497151s) [noise=true , address=104.36.82.56:4253 , piece_id=5PMVGLZETMXCQ77K3I6PXACRIMBGR2CV5JGUSU5KUMYFNXVGA2GA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:22.511944593 (75h46m58.075267584s) [noise=true , address=104.36.82.37:6663 , piece_id=X7LWYIIHJFU2T2PHYR3R5TNT22UUPHEHQWZH7ZJMLS4AH4PJ6AIA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:48.027197558 (75h47m23.590520562s) [noise=true , address=104.36.82.14:6232 , piece_id=A2ZLUUZKRTBJRJ267QSK36C3LFZV7FPAYDDAXGIWDTNEGNV6MRRA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:11:05.567692582 (75h47m41.131015574s) [noise=true , address=104.36.82.49:4982 , piece_id=UNLV5UDNWULEIPE4QSQHHZQPK3BQKFRB5T3H6GP7ZQ5QGSNK5V6A, seg_pos={0 0}, upload=60693]: Done uploading piece. err: ecclient: upload failed (node:1NxoYJXCXmTvQncbkXt6yH7mhEBBrzVdzhgL5J6W5TD4N3pkaG, address:104.36.82.49:4982): protocol: expected piece hash; context deadline exceeded; EOF Jun 21 20:11:05.567759198 (75h47m41.131082195s) [noise=true , address=104.36.82.54:4171 , piece_id=MUMABKARK4KKUKWW6XDXCQVT6O7SLFHGHE7COWC7QIYGARAA2D5A, seg_pos={0 0}, upload=60693]: Done uploading piece. err: ecclient: upload failed (node:125SMhDsEZruNu6LkmQn2M7R6WzvXTdk86DRsKf522u8BNiANq3, address:104.36.82.54:4171): protocol: expected piece hash; context deadline exceeded; EOF Jun 21 20:11:05.567869019 (75h47m41.131192033s) [noise=true , address=104.36.82.10:6087 , piece_id=5FQG7BPZC373PFB54HYXSG2SVKOBBBSGZCLC4XDJEYQFOLQY74MA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: ecclient: upload failed (node:1S9k5eejJcxkiG5pXjjBZm18uP8k5jCGfWyeW2QQxmuRU8ZzRA, address:104.36.82.10:6087): protocol: write tcp 159.223.143.39:38550->104.36.82.10:6087: use of closed network connection; write tcp 159.223.143.39:38550->104.36.82.10:6087: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:38550->104.36.82.10:6087: use of closed network connection Jun 21 20:11:05.567932150 (75h47m41.131255161s) [seg_pos={0 0}, upload=60693]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jun 21 20:11:05.593626175 (75h47m41.156949181s) [upload=60693]: Done uploading segments... Jun 21 20:11:05.593657274 (75h47m41.156980265s) [upload=60693]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 9.443668744s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [...........................................X...........X....] duration: 14.064363297s FAIL (2025-06-21 20:13:37 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== pload=60693]: Starting upload Jun 21 20:10:05.602866542 (75h46m41.166189557s) [upload=60693]: Done starting upload Jun 21 20:10:05.603116545 (75h46m41.166439547s) [upload=60693]: Uploading segments... Jun 21 20:10:05.603495389 (75h46m41.166818397s) [upload=60693]: Got next segment. Inline: false Jun 21 20:10:05.603566566 (75h46m41.166889558s) [upload=60693]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 21 20:10:05.603573280 (75h46m41.166896286s) [upload=60693]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 21 20:10:05.639130162 (75h46m41.202453164s) [seg_pos={0 0}, upload=60693]: Begin upload segment... Jun 21 20:10:05.639231070 (75h46m41.202554057s) [seg_pos={0 0}, upload=60693]: Done begin upload segment. Jun 21 20:10:05.639292724 (75h46m41.202615728s) [noise=true , address=104.36.82.56:4253 , piece_id=5PMVGLZETMXCQ77K3I6PXACRIMBGR2CV5JGUSU5KUMYFNXVGA2GA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.639338066 (75h46m41.202661071s) [noise=true , address=104.36.82.53:4126 , piece_id=2MKRQJEPGDJKDM7VWY5M4M6NORILAIEYYATJ7HVZEOBOMXOI2LBA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.639416264 (75h46m41.202739267s) [noise=true , address=104.36.82.5:5869 , piece_id=HAEHXXZASV3DHT3QN7NTZOS6XZKHNUWBKCHWJJWSTJKXAKJKKBAA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.639873098 (75h46m41.203196115s) [noise=true , address=104.36.82.52:4111 , piece_id=HB7QAWN6CGNIIT4HD2GZ7JDQP5NXIJ5I2USBM66YOKTYENNOWG5A, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.639887296 (75h46m41.203210304s) [noise=true , address=104.36.82.10:6087 , piece_id=5FQG7BPZC373PFB54HYXSG2SVKOBBBSGZCLC4XDJEYQFOLQY74MA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.640266623 (75h46m41.203589616s) [noise=true , address=104.36.82.13:6196 , piece_id=BCHMQO5KKT5BLNWQGOHH3V43LGSE3BUQCI2SY4TTO5M5GQNOCTMQ, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.640352274 (75h46m41.203675286s) [noise=true , address=104.36.82.37:6663 , piece_id=X7LWYIIHJFU2T2PHYR3R5TNT22UUPHEHQWZH7ZJMLS4AH4PJ6AIA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.640659767 (75h46m41.203982779s) [noise=true , address=104.36.82.42:6807 , piece_id=ZBQZPXUDJ4VBCOSVKWAJ6SK6UD7SLPON7Y4SSCK2DQ5STFG4XCNQ, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.640961701 (75h46m41.20428471s) [noise=true , address=45.91.116.6:4982 , piece_id=VP45R6KXNH5U5SGUQUMAPIBY7KFDPBIR3S3VRIX27EAT6VLFBZ7Q, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.641159578 (75h46m41.204482589s) [noise=true , address=104.36.82.9:6039 , piece_id=I4AB6N27MYYUHWAZLJJUIBFBLOQFHDIM3YLYWPVRPNUV7VMYS2SA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.641588126 (75h46m41.204911136s) [noise=true , address=77.90.31.19:4853 , piece_id=VFEON5T2KSWM2KSLTVRZTXYL42JWENZYUEUPG3LMODGZWBMVGGPQ, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.641863297 (75h46m41.205186305s) [noise=true , address=77.90.31.12:4603 , piece_id=BMUIC6KNGEDL37436GLSXW5XOO6VYOPNOSFGQCYVHSENVI2PJ7KQ, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.642124427 (75h46m41.205447437s) [noise=true , address=104.36.82.49:4982 , piece_id=UNLV5UDNWULEIPE4QSQHHZQPK3BQKFRB5T3H6GP7ZQ5QGSNK5V6A, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.642285917 (75h46m41.205608925s) [noise=true , address=104.36.82.54:4171 , piece_id=MUMABKARK4KKUKWW6XDXCQVT6O7SLFHGHE7COWC7QIYGARAA2D5A, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.642656750 (75h46m41.205979758s) [noise=true , address=104.36.82.14:6232 , piece_id=A2ZLUUZKRTBJRJ267QSK36C3LFZV7FPAYDDAXGIWDTNEGNV6MRRA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.643565583 (75h46m41.206888597s) [noise=true , address=79.127.205.245:4231 , piece_id=EJK46XUUVI5NWOAQJEW4QIJG3KRXZHB4YOQUJRCS2MIOSWKGTNLQ, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.644620291 (75h46m41.207943303s) [noise=true , address=77.90.31.14:4670 , piece_id=7FNANMUKPM3N6YWRAOAIM4WJZJUTNF7ZBI6ZUBNEPGNGMFWOE24A, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.648842639 (75h46m41.212165631s) [noise=true , address=104.36.82.55:4224 , piece_id=WRVNGSJV5BX4GJERLXNPAS7J7BZ5WFTM7R5H257YRCG5SP2HIYMA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.825828239 (75h46m41.389151241s) [noise=true , address=104.36.82.53:4126 , piece_id=2MKRQJEPGDJKDM7VWY5M4M6NORILAIEYYATJ7HVZEOBOMXOI2LBA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: ecclient: upload failed (node:12dMWFrq2NMpnMzuPAphWerntC5kkrcnzRJZBA116uzJzyvk6fH, address:104.36.82.53:4126): protocol: storage node overloaded, request limit: 2000 Jun 21 20:10:05.834666043 (75h46m41.397989033s) [upload=60693]: Next returned nil segment Jun 21 20:10:05.834686295 (75h46m41.398009302s) [upload=60693]: Waiting for error group managing segments... Jun 21 20:10:05.877010488 (75h46m41.440333486s) [noise=true , address=45.91.116.6:4982 , piece_id=VP45R6KXNH5U5SGUQUMAPIBY7KFDPBIR3S3VRIX27EAT6VLFBZ7Q, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:05.886336964 (75h46m41.449659967s) [noise=true , address=104.36.82.9:6039 , piece_id=I4AB6N27MYYUHWAZLJJUIBFBLOQFHDIM3YLYWPVRPNUV7VMYS2SA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: ecclient: upload failed (node:12P11WVW9mVXw5ra2gZ3Rezon6wZi8GrN5CchQN3MiVU4ts3L6V, address:104.36.82.9:6039): protocol: storage node overloaded, request limit: 2000 Jun 21 20:10:08.064514542 (75h46m43.627837553s) [noise=true , address=77.90.31.14:4670 , piece_id=7FNANMUKPM3N6YWRAOAIM4WJZJUTNF7ZBI6ZUBNEPGNGMFWOE24A, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:08.099098391 (75h46m43.662421396s) [noise=true , address=79.127.205.245:4231 , piece_id=EJK46XUUVI5NWOAQJEW4QIJG3KRXZHB4YOQUJRCS2MIOSWKGTNLQ, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:08.489461749 (75h46m44.052784752s) [noise=true , address=104.36.82.42:6807 , piece_id=ZBQZPXUDJ4VBCOSVKWAJ6SK6UD7SLPON7Y4SSCK2DQ5STFG4XCNQ, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:08.650922093 (75h46m44.214245093s) [noise=true , address=77.90.31.19:4853 , piece_id=VFEON5T2KSWM2KSLTVRZTXYL42JWENZYUEUPG3LMODGZWBMVGGPQ, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:09.917746439 (75h46m45.481069441s) [noise=true , address=104.36.82.13:6196 , piece_id=BCHMQO5KKT5BLNWQGOHH3V43LGSE3BUQCI2SY4TTO5M5GQNOCTMQ, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:10.830722215 (75h46m46.394045216s) [noise=true , address=104.36.82.5:5869 , piece_id=HAEHXXZASV3DHT3QN7NTZOS6XZKHNUWBKCHWJJWSTJKXAKJKKBAA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:12.738783450 (75h46m48.302106452s) [noise=true , address=77.90.31.12:4603 , piece_id=BMUIC6KNGEDL37436GLSXW5XOO6VYOPNOSFGQCYVHSENVI2PJ7KQ, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:15.460927713 (75h46m51.024250723s) [noise=true , address=104.36.82.52:4111 , piece_id=HB7QAWN6CGNIIT4HD2GZ7JDQP5NXIJ5I2USBM66YOKTYENNOWG5A, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:18.393817968 (75h46m53.95714096s) [noise=true , address=104.36.82.55:4224 , piece_id=WRVNGSJV5BX4GJERLXNPAS7J7BZ5WFTM7R5H257YRCG5SP2HIYMA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:19.549174143 (75h46m55.112497151s) [noise=true , address=104.36.82.56:4253 , piece_id=5PMVGLZETMXCQ77K3I6PXACRIMBGR2CV5JGUSU5KUMYFNXVGA2GA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:22.511944593 (75h46m58.075267584s) [noise=true , address=104.36.82.37:6663 , piece_id=X7LWYIIHJFU2T2PHYR3R5TNT22UUPHEHQWZH7ZJMLS4AH4PJ6AIA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:48.027197558 (75h47m23.590520562s) [noise=true , address=104.36.82.14:6232 , piece_id=A2ZLUUZKRTBJRJ267QSK36C3LFZV7FPAYDDAXGIWDTNEGNV6MRRA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:11:05.567692582 (75h47m41.131015574s) [noise=true , address=104.36.82.49:4982 , piece_id=UNLV5UDNWULEIPE4QSQHHZQPK3BQKFRB5T3H6GP7ZQ5QGSNK5V6A, seg_pos={0 0}, upload=60693]: Done uploading piece. err: ecclient: upload failed (node:1NxoYJXCXmTvQncbkXt6yH7mhEBBrzVdzhgL5J6W5TD4N3pkaG, address:104.36.82.49:4982): protocol: expected piece hash; context deadline exceeded; EOF Jun 21 20:11:05.567759198 (75h47m41.131082195s) [noise=true , address=104.36.82.54:4171 , piece_id=MUMABKARK4KKUKWW6XDXCQVT6O7SLFHGHE7COWC7QIYGARAA2D5A, seg_pos={0 0}, upload=60693]: Done uploading piece. err: ecclient: upload failed (node:125SMhDsEZruNu6LkmQn2M7R6WzvXTdk86DRsKf522u8BNiANq3, address:104.36.82.54:4171): protocol: expected piece hash; context deadline exceeded; EOF Jun 21 20:11:05.567869019 (75h47m41.131192033s) [noise=true , address=104.36.82.10:6087 , piece_id=5FQG7BPZC373PFB54HYXSG2SVKOBBBSGZCLC4XDJEYQFOLQY74MA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: ecclient: upload failed (node:1S9k5eejJcxkiG5pXjjBZm18uP8k5jCGfWyeW2QQxmuRU8ZzRA, address:104.36.82.10:6087): protocol: write tcp 159.223.143.39:38550->104.36.82.10:6087: use of closed network connection; write tcp 159.223.143.39:38550->104.36.82.10:6087: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:38550->104.36.82.10:6087: use of closed network connection Jun 21 20:11:05.567932150 (75h47m41.131255161s) [seg_pos={0 0}, upload=60693]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jun 21 20:11:05.593626175 (75h47m41.156949181s) [upload=60693]: Done uploading segments... Jun 21 20:11:05.593657274 (75h47m41.156980265s) [upload=60693]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 14.064396921s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [............................................X...........X...] duration: 17.726500126s FAIL (2025-06-21 20:12:55 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== pload=60693]: Starting upload Jun 21 20:10:05.602866542 (75h46m41.166189557s) [upload=60693]: Done starting upload Jun 21 20:10:05.603116545 (75h46m41.166439547s) [upload=60693]: Uploading segments... Jun 21 20:10:05.603495389 (75h46m41.166818397s) [upload=60693]: Got next segment. Inline: false Jun 21 20:10:05.603566566 (75h46m41.166889558s) [upload=60693]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 21 20:10:05.603573280 (75h46m41.166896286s) [upload=60693]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 21 20:10:05.639130162 (75h46m41.202453164s) [seg_pos={0 0}, upload=60693]: Begin upload segment... Jun 21 20:10:05.639231070 (75h46m41.202554057s) [seg_pos={0 0}, upload=60693]: Done begin upload segment. Jun 21 20:10:05.639292724 (75h46m41.202615728s) [noise=true , address=104.36.82.56:4253 , piece_id=5PMVGLZETMXCQ77K3I6PXACRIMBGR2CV5JGUSU5KUMYFNXVGA2GA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.639338066 (75h46m41.202661071s) [noise=true , address=104.36.82.53:4126 , piece_id=2MKRQJEPGDJKDM7VWY5M4M6NORILAIEYYATJ7HVZEOBOMXOI2LBA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.639416264 (75h46m41.202739267s) [noise=true , address=104.36.82.5:5869 , piece_id=HAEHXXZASV3DHT3QN7NTZOS6XZKHNUWBKCHWJJWSTJKXAKJKKBAA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.639873098 (75h46m41.203196115s) [noise=true , address=104.36.82.52:4111 , piece_id=HB7QAWN6CGNIIT4HD2GZ7JDQP5NXIJ5I2USBM66YOKTYENNOWG5A, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.639887296 (75h46m41.203210304s) [noise=true , address=104.36.82.10:6087 , piece_id=5FQG7BPZC373PFB54HYXSG2SVKOBBBSGZCLC4XDJEYQFOLQY74MA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.640266623 (75h46m41.203589616s) [noise=true , address=104.36.82.13:6196 , piece_id=BCHMQO5KKT5BLNWQGOHH3V43LGSE3BUQCI2SY4TTO5M5GQNOCTMQ, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.640352274 (75h46m41.203675286s) [noise=true , address=104.36.82.37:6663 , piece_id=X7LWYIIHJFU2T2PHYR3R5TNT22UUPHEHQWZH7ZJMLS4AH4PJ6AIA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.640659767 (75h46m41.203982779s) [noise=true , address=104.36.82.42:6807 , piece_id=ZBQZPXUDJ4VBCOSVKWAJ6SK6UD7SLPON7Y4SSCK2DQ5STFG4XCNQ, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.640961701 (75h46m41.20428471s) [noise=true , address=45.91.116.6:4982 , piece_id=VP45R6KXNH5U5SGUQUMAPIBY7KFDPBIR3S3VRIX27EAT6VLFBZ7Q, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.641159578 (75h46m41.204482589s) [noise=true , address=104.36.82.9:6039 , piece_id=I4AB6N27MYYUHWAZLJJUIBFBLOQFHDIM3YLYWPVRPNUV7VMYS2SA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.641588126 (75h46m41.204911136s) [noise=true , address=77.90.31.19:4853 , piece_id=VFEON5T2KSWM2KSLTVRZTXYL42JWENZYUEUPG3LMODGZWBMVGGPQ, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.641863297 (75h46m41.205186305s) [noise=true , address=77.90.31.12:4603 , piece_id=BMUIC6KNGEDL37436GLSXW5XOO6VYOPNOSFGQCYVHSENVI2PJ7KQ, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.642124427 (75h46m41.205447437s) [noise=true , address=104.36.82.49:4982 , piece_id=UNLV5UDNWULEIPE4QSQHHZQPK3BQKFRB5T3H6GP7ZQ5QGSNK5V6A, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.642285917 (75h46m41.205608925s) [noise=true , address=104.36.82.54:4171 , piece_id=MUMABKARK4KKUKWW6XDXCQVT6O7SLFHGHE7COWC7QIYGARAA2D5A, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.642656750 (75h46m41.205979758s) [noise=true , address=104.36.82.14:6232 , piece_id=A2ZLUUZKRTBJRJ267QSK36C3LFZV7FPAYDDAXGIWDTNEGNV6MRRA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.643565583 (75h46m41.206888597s) [noise=true , address=79.127.205.245:4231 , piece_id=EJK46XUUVI5NWOAQJEW4QIJG3KRXZHB4YOQUJRCS2MIOSWKGTNLQ, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.644620291 (75h46m41.207943303s) [noise=true , address=77.90.31.14:4670 , piece_id=7FNANMUKPM3N6YWRAOAIM4WJZJUTNF7ZBI6ZUBNEPGNGMFWOE24A, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.648842639 (75h46m41.212165631s) [noise=true , address=104.36.82.55:4224 , piece_id=WRVNGSJV5BX4GJERLXNPAS7J7BZ5WFTM7R5H257YRCG5SP2HIYMA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.825828239 (75h46m41.389151241s) [noise=true , address=104.36.82.53:4126 , piece_id=2MKRQJEPGDJKDM7VWY5M4M6NORILAIEYYATJ7HVZEOBOMXOI2LBA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: ecclient: upload failed (node:12dMWFrq2NMpnMzuPAphWerntC5kkrcnzRJZBA116uzJzyvk6fH, address:104.36.82.53:4126): protocol: storage node overloaded, request limit: 2000 Jun 21 20:10:05.834666043 (75h46m41.397989033s) [upload=60693]: Next returned nil segment Jun 21 20:10:05.834686295 (75h46m41.398009302s) [upload=60693]: Waiting for error group managing segments... Jun 21 20:10:05.877010488 (75h46m41.440333486s) [noise=true , address=45.91.116.6:4982 , piece_id=VP45R6KXNH5U5SGUQUMAPIBY7KFDPBIR3S3VRIX27EAT6VLFBZ7Q, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:05.886336964 (75h46m41.449659967s) [noise=true , address=104.36.82.9:6039 , piece_id=I4AB6N27MYYUHWAZLJJUIBFBLOQFHDIM3YLYWPVRPNUV7VMYS2SA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: ecclient: upload failed (node:12P11WVW9mVXw5ra2gZ3Rezon6wZi8GrN5CchQN3MiVU4ts3L6V, address:104.36.82.9:6039): protocol: storage node overloaded, request limit: 2000 Jun 21 20:10:08.064514542 (75h46m43.627837553s) [noise=true , address=77.90.31.14:4670 , piece_id=7FNANMUKPM3N6YWRAOAIM4WJZJUTNF7ZBI6ZUBNEPGNGMFWOE24A, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:08.099098391 (75h46m43.662421396s) [noise=true , address=79.127.205.245:4231 , piece_id=EJK46XUUVI5NWOAQJEW4QIJG3KRXZHB4YOQUJRCS2MIOSWKGTNLQ, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:08.489461749 (75h46m44.052784752s) [noise=true , address=104.36.82.42:6807 , piece_id=ZBQZPXUDJ4VBCOSVKWAJ6SK6UD7SLPON7Y4SSCK2DQ5STFG4XCNQ, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:08.650922093 (75h46m44.214245093s) [noise=true , address=77.90.31.19:4853 , piece_id=VFEON5T2KSWM2KSLTVRZTXYL42JWENZYUEUPG3LMODGZWBMVGGPQ, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:09.917746439 (75h46m45.481069441s) [noise=true , address=104.36.82.13:6196 , piece_id=BCHMQO5KKT5BLNWQGOHH3V43LGSE3BUQCI2SY4TTO5M5GQNOCTMQ, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:10.830722215 (75h46m46.394045216s) [noise=true , address=104.36.82.5:5869 , piece_id=HAEHXXZASV3DHT3QN7NTZOS6XZKHNUWBKCHWJJWSTJKXAKJKKBAA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:12.738783450 (75h46m48.302106452s) [noise=true , address=77.90.31.12:4603 , piece_id=BMUIC6KNGEDL37436GLSXW5XOO6VYOPNOSFGQCYVHSENVI2PJ7KQ, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:15.460927713 (75h46m51.024250723s) [noise=true , address=104.36.82.52:4111 , piece_id=HB7QAWN6CGNIIT4HD2GZ7JDQP5NXIJ5I2USBM66YOKTYENNOWG5A, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:18.393817968 (75h46m53.95714096s) [noise=true , address=104.36.82.55:4224 , piece_id=WRVNGSJV5BX4GJERLXNPAS7J7BZ5WFTM7R5H257YRCG5SP2HIYMA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:19.549174143 (75h46m55.112497151s) [noise=true , address=104.36.82.56:4253 , piece_id=5PMVGLZETMXCQ77K3I6PXACRIMBGR2CV5JGUSU5KUMYFNXVGA2GA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:22.511944593 (75h46m58.075267584s) [noise=true , address=104.36.82.37:6663 , piece_id=X7LWYIIHJFU2T2PHYR3R5TNT22UUPHEHQWZH7ZJMLS4AH4PJ6AIA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:48.027197558 (75h47m23.590520562s) [noise=true , address=104.36.82.14:6232 , piece_id=A2ZLUUZKRTBJRJ267QSK36C3LFZV7FPAYDDAXGIWDTNEGNV6MRRA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:11:05.567692582 (75h47m41.131015574s) [noise=true , address=104.36.82.49:4982 , piece_id=UNLV5UDNWULEIPE4QSQHHZQPK3BQKFRB5T3H6GP7ZQ5QGSNK5V6A, seg_pos={0 0}, upload=60693]: Done uploading piece. err: ecclient: upload failed (node:1NxoYJXCXmTvQncbkXt6yH7mhEBBrzVdzhgL5J6W5TD4N3pkaG, address:104.36.82.49:4982): protocol: expected piece hash; context deadline exceeded; EOF Jun 21 20:11:05.567759198 (75h47m41.131082195s) [noise=true , address=104.36.82.54:4171 , piece_id=MUMABKARK4KKUKWW6XDXCQVT6O7SLFHGHE7COWC7QIYGARAA2D5A, seg_pos={0 0}, upload=60693]: Done uploading piece. err: ecclient: upload failed (node:125SMhDsEZruNu6LkmQn2M7R6WzvXTdk86DRsKf522u8BNiANq3, address:104.36.82.54:4171): protocol: expected piece hash; context deadline exceeded; EOF Jun 21 20:11:05.567869019 (75h47m41.131192033s) [noise=true , address=104.36.82.10:6087 , piece_id=5FQG7BPZC373PFB54HYXSG2SVKOBBBSGZCLC4XDJEYQFOLQY74MA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: ecclient: upload failed (node:1S9k5eejJcxkiG5pXjjBZm18uP8k5jCGfWyeW2QQxmuRU8ZzRA, address:104.36.82.10:6087): protocol: write tcp 159.223.143.39:38550->104.36.82.10:6087: use of closed network connection; write tcp 159.223.143.39:38550->104.36.82.10:6087: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:38550->104.36.82.10:6087: use of closed network connection Jun 21 20:11:05.567932150 (75h47m41.131255161s) [seg_pos={0 0}, upload=60693]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jun 21 20:11:05.593626175 (75h47m41.156949181s) [upload=60693]: Done uploading segments... Jun 21 20:11:05.593657274 (75h47m41.156980265s) [upload=60693]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 17.72661827s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [.............................................X...........X..] duration: 14.719725463s FAIL (2025-06-21 20:11:53 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== pload=60693]: Starting upload Jun 21 20:10:05.602866542 (75h46m41.166189557s) [upload=60693]: Done starting upload Jun 21 20:10:05.603116545 (75h46m41.166439547s) [upload=60693]: Uploading segments... Jun 21 20:10:05.603495389 (75h46m41.166818397s) [upload=60693]: Got next segment. Inline: false Jun 21 20:10:05.603566566 (75h46m41.166889558s) [upload=60693]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 21 20:10:05.603573280 (75h46m41.166896286s) [upload=60693]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 21 20:10:05.639130162 (75h46m41.202453164s) [seg_pos={0 0}, upload=60693]: Begin upload segment... Jun 21 20:10:05.639231070 (75h46m41.202554057s) [seg_pos={0 0}, upload=60693]: Done begin upload segment. Jun 21 20:10:05.639292724 (75h46m41.202615728s) [noise=true , address=104.36.82.56:4253 , piece_id=5PMVGLZETMXCQ77K3I6PXACRIMBGR2CV5JGUSU5KUMYFNXVGA2GA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.639338066 (75h46m41.202661071s) [noise=true , address=104.36.82.53:4126 , piece_id=2MKRQJEPGDJKDM7VWY5M4M6NORILAIEYYATJ7HVZEOBOMXOI2LBA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.639416264 (75h46m41.202739267s) [noise=true , address=104.36.82.5:5869 , piece_id=HAEHXXZASV3DHT3QN7NTZOS6XZKHNUWBKCHWJJWSTJKXAKJKKBAA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.639873098 (75h46m41.203196115s) [noise=true , address=104.36.82.52:4111 , piece_id=HB7QAWN6CGNIIT4HD2GZ7JDQP5NXIJ5I2USBM66YOKTYENNOWG5A, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.639887296 (75h46m41.203210304s) [noise=true , address=104.36.82.10:6087 , piece_id=5FQG7BPZC373PFB54HYXSG2SVKOBBBSGZCLC4XDJEYQFOLQY74MA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.640266623 (75h46m41.203589616s) [noise=true , address=104.36.82.13:6196 , piece_id=BCHMQO5KKT5BLNWQGOHH3V43LGSE3BUQCI2SY4TTO5M5GQNOCTMQ, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.640352274 (75h46m41.203675286s) [noise=true , address=104.36.82.37:6663 , piece_id=X7LWYIIHJFU2T2PHYR3R5TNT22UUPHEHQWZH7ZJMLS4AH4PJ6AIA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.640659767 (75h46m41.203982779s) [noise=true , address=104.36.82.42:6807 , piece_id=ZBQZPXUDJ4VBCOSVKWAJ6SK6UD7SLPON7Y4SSCK2DQ5STFG4XCNQ, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.640961701 (75h46m41.20428471s) [noise=true , address=45.91.116.6:4982 , piece_id=VP45R6KXNH5U5SGUQUMAPIBY7KFDPBIR3S3VRIX27EAT6VLFBZ7Q, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.641159578 (75h46m41.204482589s) [noise=true , address=104.36.82.9:6039 , piece_id=I4AB6N27MYYUHWAZLJJUIBFBLOQFHDIM3YLYWPVRPNUV7VMYS2SA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.641588126 (75h46m41.204911136s) [noise=true , address=77.90.31.19:4853 , piece_id=VFEON5T2KSWM2KSLTVRZTXYL42JWENZYUEUPG3LMODGZWBMVGGPQ, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.641863297 (75h46m41.205186305s) [noise=true , address=77.90.31.12:4603 , piece_id=BMUIC6KNGEDL37436GLSXW5XOO6VYOPNOSFGQCYVHSENVI2PJ7KQ, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.642124427 (75h46m41.205447437s) [noise=true , address=104.36.82.49:4982 , piece_id=UNLV5UDNWULEIPE4QSQHHZQPK3BQKFRB5T3H6GP7ZQ5QGSNK5V6A, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.642285917 (75h46m41.205608925s) [noise=true , address=104.36.82.54:4171 , piece_id=MUMABKARK4KKUKWW6XDXCQVT6O7SLFHGHE7COWC7QIYGARAA2D5A, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.642656750 (75h46m41.205979758s) [noise=true , address=104.36.82.14:6232 , piece_id=A2ZLUUZKRTBJRJ267QSK36C3LFZV7FPAYDDAXGIWDTNEGNV6MRRA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.643565583 (75h46m41.206888597s) [noise=true , address=79.127.205.245:4231 , piece_id=EJK46XUUVI5NWOAQJEW4QIJG3KRXZHB4YOQUJRCS2MIOSWKGTNLQ, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.644620291 (75h46m41.207943303s) [noise=true , address=77.90.31.14:4670 , piece_id=7FNANMUKPM3N6YWRAOAIM4WJZJUTNF7ZBI6ZUBNEPGNGMFWOE24A, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.648842639 (75h46m41.212165631s) [noise=true , address=104.36.82.55:4224 , piece_id=WRVNGSJV5BX4GJERLXNPAS7J7BZ5WFTM7R5H257YRCG5SP2HIYMA, seg_pos={0 0}, upload=60693]: Uploading piece... Jun 21 20:10:05.825828239 (75h46m41.389151241s) [noise=true , address=104.36.82.53:4126 , piece_id=2MKRQJEPGDJKDM7VWY5M4M6NORILAIEYYATJ7HVZEOBOMXOI2LBA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: ecclient: upload failed (node:12dMWFrq2NMpnMzuPAphWerntC5kkrcnzRJZBA116uzJzyvk6fH, address:104.36.82.53:4126): protocol: storage node overloaded, request limit: 2000 Jun 21 20:10:05.834666043 (75h46m41.397989033s) [upload=60693]: Next returned nil segment Jun 21 20:10:05.834686295 (75h46m41.398009302s) [upload=60693]: Waiting for error group managing segments... Jun 21 20:10:05.877010488 (75h46m41.440333486s) [noise=true , address=45.91.116.6:4982 , piece_id=VP45R6KXNH5U5SGUQUMAPIBY7KFDPBIR3S3VRIX27EAT6VLFBZ7Q, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:05.886336964 (75h46m41.449659967s) [noise=true , address=104.36.82.9:6039 , piece_id=I4AB6N27MYYUHWAZLJJUIBFBLOQFHDIM3YLYWPVRPNUV7VMYS2SA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: ecclient: upload failed (node:12P11WVW9mVXw5ra2gZ3Rezon6wZi8GrN5CchQN3MiVU4ts3L6V, address:104.36.82.9:6039): protocol: storage node overloaded, request limit: 2000 Jun 21 20:10:08.064514542 (75h46m43.627837553s) [noise=true , address=77.90.31.14:4670 , piece_id=7FNANMUKPM3N6YWRAOAIM4WJZJUTNF7ZBI6ZUBNEPGNGMFWOE24A, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:08.099098391 (75h46m43.662421396s) [noise=true , address=79.127.205.245:4231 , piece_id=EJK46XUUVI5NWOAQJEW4QIJG3KRXZHB4YOQUJRCS2MIOSWKGTNLQ, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:08.489461749 (75h46m44.052784752s) [noise=true , address=104.36.82.42:6807 , piece_id=ZBQZPXUDJ4VBCOSVKWAJ6SK6UD7SLPON7Y4SSCK2DQ5STFG4XCNQ, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:08.650922093 (75h46m44.214245093s) [noise=true , address=77.90.31.19:4853 , piece_id=VFEON5T2KSWM2KSLTVRZTXYL42JWENZYUEUPG3LMODGZWBMVGGPQ, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:09.917746439 (75h46m45.481069441s) [noise=true , address=104.36.82.13:6196 , piece_id=BCHMQO5KKT5BLNWQGOHH3V43LGSE3BUQCI2SY4TTO5M5GQNOCTMQ, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:10.830722215 (75h46m46.394045216s) [noise=true , address=104.36.82.5:5869 , piece_id=HAEHXXZASV3DHT3QN7NTZOS6XZKHNUWBKCHWJJWSTJKXAKJKKBAA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:12.738783450 (75h46m48.302106452s) [noise=true , address=77.90.31.12:4603 , piece_id=BMUIC6KNGEDL37436GLSXW5XOO6VYOPNOSFGQCYVHSENVI2PJ7KQ, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:15.460927713 (75h46m51.024250723s) [noise=true , address=104.36.82.52:4111 , piece_id=HB7QAWN6CGNIIT4HD2GZ7JDQP5NXIJ5I2USBM66YOKTYENNOWG5A, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:18.393817968 (75h46m53.95714096s) [noise=true , address=104.36.82.55:4224 , piece_id=WRVNGSJV5BX4GJERLXNPAS7J7BZ5WFTM7R5H257YRCG5SP2HIYMA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:19.549174143 (75h46m55.112497151s) [noise=true , address=104.36.82.56:4253 , piece_id=5PMVGLZETMXCQ77K3I6PXACRIMBGR2CV5JGUSU5KUMYFNXVGA2GA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:22.511944593 (75h46m58.075267584s) [noise=true , address=104.36.82.37:6663 , piece_id=X7LWYIIHJFU2T2PHYR3R5TNT22UUPHEHQWZH7ZJMLS4AH4PJ6AIA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:10:48.027197558 (75h47m23.590520562s) [noise=true , address=104.36.82.14:6232 , piece_id=A2ZLUUZKRTBJRJ267QSK36C3LFZV7FPAYDDAXGIWDTNEGNV6MRRA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: <nil> Jun 21 20:11:05.567692582 (75h47m41.131015574s) [noise=true , address=104.36.82.49:4982 , piece_id=UNLV5UDNWULEIPE4QSQHHZQPK3BQKFRB5T3H6GP7ZQ5QGSNK5V6A, seg_pos={0 0}, upload=60693]: Done uploading piece. err: ecclient: upload failed (node:1NxoYJXCXmTvQncbkXt6yH7mhEBBrzVdzhgL5J6W5TD4N3pkaG, address:104.36.82.49:4982): protocol: expected piece hash; context deadline exceeded; EOF Jun 21 20:11:05.567759198 (75h47m41.131082195s) [noise=true , address=104.36.82.54:4171 , piece_id=MUMABKARK4KKUKWW6XDXCQVT6O7SLFHGHE7COWC7QIYGARAA2D5A, seg_pos={0 0}, upload=60693]: Done uploading piece. err: ecclient: upload failed (node:125SMhDsEZruNu6LkmQn2M7R6WzvXTdk86DRsKf522u8BNiANq3, address:104.36.82.54:4171): protocol: expected piece hash; context deadline exceeded; EOF Jun 21 20:11:05.567869019 (75h47m41.131192033s) [noise=true , address=104.36.82.10:6087 , piece_id=5FQG7BPZC373PFB54HYXSG2SVKOBBBSGZCLC4XDJEYQFOLQY74MA, seg_pos={0 0}, upload=60693]: Done uploading piece. err: ecclient: upload failed (node:1S9k5eejJcxkiG5pXjjBZm18uP8k5jCGfWyeW2QQxmuRU8ZzRA, address:104.36.82.10:6087): protocol: write tcp 159.223.143.39:38550->104.36.82.10:6087: use of closed network connection; write tcp 159.223.143.39:38550->104.36.82.10:6087: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:38550->104.36.82.10:6087: use of closed network connection Jun 21 20:11:05.567932150 (75h47m41.131255161s) [seg_pos={0 0}, upload=60693]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jun 21 20:11:05.593626175 (75h47m41.156949181s) [upload=60693]: Done uploading segments... Jun 21 20:11:05.593657274 (75h47m41.156980265s) [upload=60693]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 14.719753612s core uplink us-select-1 us1-placement-20

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)