Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [....................X.......................................] duration: 24.902522025s FAIL (2025-06-22 01:02:35 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== ing upload Jun 22 00:30:55.303646184 (80h7m30.866969171s) [upload=64142]: Uploading segments... Jun 22 00:30:55.303706406 (80h7m30.867029416s) [upload=64142]: Got next segment. Inline: false Jun 22 00:30:55.303762479 (80h7m30.867085468s) [upload=64142]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 22 00:30:55.303768627 (80h7m30.867091636s) [upload=64142]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 22 00:30:55.349058140 (80h7m30.912381168s) [seg_pos={0 0}, upload=64142]: Begin upload segment... Jun 22 00:30:55.349113768 (80h7m30.912436772s) [seg_pos={0 0}, upload=64142]: Done begin upload segment. Jun 22 00:30:55.349143121 (80h7m30.912466131s) [noise=true , address=45.91.116.5:5310 , piece_id=AXJOKDZRPVNL7HFT4QKJWRAHFM2UCM3GEVQB2KNYD6CXB6TH4D4Q, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.349660917 (80h7m30.912983928s) [noise=true , address=77.90.31.10:4533 , piece_id=SE6YAKKFDL3BV677CDDUC32UNMPXJHTG23GWWYK5B4YIPKER6GAQ, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.350093077 (80h7m30.913416089s) [noise=true , address=79.127.201.215:4326 , piece_id=USEWZDX6P52U6MCALTBNQG3SV7N2JRKNU6O23TLS6JEWAS4D7ARA, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.350480894 (80h7m30.913803916s) [noise=true , address=104.36.82.13:6199 , piece_id=YP62N2TO446YR3QRG5YLXECD7TN6SJ4UNNLZSK5H4CMV6JIF4AKA, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.350669998 (80h7m30.913993011s) [noise=true , address=104.36.82.54:4184 , piece_id=IZICVGZFMAYLIJTNUJ2OFLZVNZ6IPPYDZO4BSTJJPLHVCVOE5EAA, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.351201352 (80h7m30.914524366s) [noise=true , address=104.36.82.55:4224 , piece_id=XF7H2BYSNNNZOS2T77VIN75N76AYUXAQEI6D4LRXACU2KU62V2TA, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.351668071 (80h7m30.914991083s) [noise=true , address=104.36.82.56:4261 , piece_id=EYOYOYLDACGMPGJYX6ILQT3XHMUTQNXZEDC7ANHLZXIKYVDQFFVQ, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.352193126 (80h7m30.915516138s) [noise=true , address=104.36.82.52:4089 , piece_id=X2VG4S3UHVNWA4AQJS7FLKWNVLTOZ3AKWI5ZDH24QCA3ENQW7BXQ, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.352587096 (80h7m30.915910103s) [noise=true , address=79.127.220.98:4402 , piece_id=TE2WQO5LK45AVMPIAIGVSIBOVZLC4FG4ESQ72X6KFYC7IAEN2DRA, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.353420253 (80h7m30.916743265s) [noise=true , address=104.36.82.11:6125 , piece_id=2CX7RJOKNQA6ODZZZXYPSCYAQY5P35TWX7I56CMYAPFLJE3JEO5Q, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.353791565 (80h7m30.917114578s) [noise=true , address=104.36.82.42:6803 , piece_id=R626UFD326S2UBJEZ2WMHVR7EN4SK427VXPDJZOOZ32MX25JWZWQ, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.354171226 (80h7m30.917494233s) [noise=true , address=104.36.82.14:6245 , piece_id=II36G46OA6TINLR244U6D7Q6LGV754UA5M62K3EX7ZOBESMJUEHA, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.354542282 (80h7m30.917865288s) [noise=true , address=79.127.205.235:5660 , piece_id=UGFPVS7VNB7U2YSFPSGYDIRZV5R2DHCN57PJP6VOXO52WIAYHLXA, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.354891655 (80h7m30.918214678s) [noise=true , address=104.36.82.15:6283 , piece_id=N7TH2EMIOPATVHCSNH2WXUGU7KMMEJ5LCHR22SWEHSA5OV4CUCXQ, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.355499503 (80h7m30.918822519s) [noise=true , address=77.90.31.14:4666 , piece_id=H4DTFDAVCN266PB235BJIQTQ32FBXTIB5AZSX3B4R2VIK554AFSQ, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.356575547 (80h7m30.919898539s) [noise=true , address=104.36.82.39:6756 , piece_id=OKKWMDJPZO6D5VRMGOE3GOCYGHM725JZTNOVYAWKJW3WYZPI4V5Q, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.357039590 (80h7m30.920362596s) [noise=true , address=104.36.82.53:4146 , piece_id=3V3DVGUOPJKDWIGQ2TU7M6VFYFVAS6ODY62ERGSU6LI3LSTWQ4YQ, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.360343566 (80h7m30.923666582s) [noise=true , address=104.36.82.9:6033 , piece_id=T5XA23LZELMSTWOLBSHOX4AHKDACF65OYCKTMZYTXDIBSXNA2T4A, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.452444064 (80h7m31.015767055s) [upload=64142]: Next returned nil segment Jun 22 00:30:55.452495148 (80h7m31.015818149s) [upload=64142]: Waiting for error group managing segments... Jun 22 00:30:55.532958808 (80h7m31.096281883s) [noise=true , address=104.36.82.9:6033 , piece_id=T5XA23LZELMSTWOLBSHOX4AHKDACF65OYCKTMZYTXDIBSXNA2T4A, seg_pos={0 0}, upload=64142]: Done uploading piece. err: ecclient: upload failed (node:1SV3LrcQW16wLewDzfyUWFB6sUgnERNJVdhz72CJmyde95M79C, address:104.36.82.9:6033): protocol: storage node overloaded, request limit: 2000 Jun 22 00:30:55.547802184 (80h7m31.111125188s) [noise=true , address=79.127.220.98:4402 , piece_id=TE2WQO5LK45AVMPIAIGVSIBOVZLC4FG4ESQ72X6KFYC7IAEN2DRA, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:30:55.554223621 (80h7m31.117546616s) [noise=true , address=77.90.31.14:4666 , piece_id=H4DTFDAVCN266PB235BJIQTQ32FBXTIB5AZSX3B4R2VIK554AFSQ, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:30:55.762514255 (80h7m31.325837245s) [noise=true , address=104.36.82.14:6245 , piece_id=II36G46OA6TINLR244U6D7Q6LGV754UA5M62K3EX7ZOBESMJUEHA, seg_pos={0 0}, upload=64142]: Done uploading piece. err: ecclient: upload failed (node:1Zu8teNPTXn638grHKwtJqz4UEY7oFYLx1DHvuVP6RZXcatVqX, address:104.36.82.14:6245): protocol: storage node overloaded, request limit: 2000 Jun 22 00:30:55.773669583 (80h7m31.3369926s) [noise=true , address=79.127.205.235:5660 , piece_id=UGFPVS7VNB7U2YSFPSGYDIRZV5R2DHCN57PJP6VOXO52WIAYHLXA, seg_pos={0 0}, upload=64142]: Done uploading piece. err: ecclient: upload failed (node:12E6uCX9KL4Ud7YpxMXepvMXuqTFJCiSXByDjW94Am3U9H2Nf7f, address:79.127.205.235:5660): protocol: storage node overloaded, request limit: 2000 Jun 22 00:30:56.968366655 (80h7m32.53168964s) [noise=true , address=77.90.31.10:4533 , piece_id=SE6YAKKFDL3BV677CDDUC32UNMPXJHTG23GWWYK5B4YIPKER6GAQ, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:30:57.403572874 (80h7m32.966895883s) [noise=true , address=45.91.116.5:5310 , piece_id=AXJOKDZRPVNL7HFT4QKJWRAHFM2UCM3GEVQB2KNYD6CXB6TH4D4Q, seg_pos={0 0}, upload=64142]: Done uploading piece. err: ecclient: upload failed (node:127RMUc62xRq9RU5z3XfkZsq5sehmv3kP8o53tr9BxVcPE3GQCY, address:45.91.116.5:5310): protocol: storage node overloaded, request limit: 2000 Jun 22 00:30:58.210389994 (80h7m33.773712995s) [noise=true , address=104.36.82.13:6199 , piece_id=YP62N2TO446YR3QRG5YLXECD7TN6SJ4UNNLZSK5H4CMV6JIF4AKA, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:30:58.693037961 (80h7m34.256360966s) [noise=true , address=104.36.82.39:6756 , piece_id=OKKWMDJPZO6D5VRMGOE3GOCYGHM725JZTNOVYAWKJW3WYZPI4V5Q, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:30:59.124834814 (80h7m34.688157819s) [noise=true , address=104.36.82.15:6283 , piece_id=N7TH2EMIOPATVHCSNH2WXUGU7KMMEJ5LCHR22SWEHSA5OV4CUCXQ, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:31:00.629657982 (80h7m36.192980977s) [noise=true , address=104.36.82.42:6803 , piece_id=R626UFD326S2UBJEZ2WMHVR7EN4SK427VXPDJZOOZ32MX25JWZWQ, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:31:00.933110546 (80h7m36.496433536s) [noise=true , address=79.127.201.215:4326 , piece_id=USEWZDX6P52U6MCALTBNQG3SV7N2JRKNU6O23TLS6JEWAS4D7ARA, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:31:05.669167243 (80h7m41.232490237s) [noise=true , address=104.36.82.56:4261 , piece_id=EYOYOYLDACGMPGJYX6ILQT3XHMUTQNXZEDC7ANHLZXIKYVDQFFVQ, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:31:08.443582763 (80h7m44.006905763s) [noise=true , address=104.36.82.54:4184 , piece_id=IZICVGZFMAYLIJTNUJ2OFLZVNZ6IPPYDZO4BSTJJPLHVCVOE5EAA, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:31:26.459138014 (80h8m2.022461013s) [noise=true , address=104.36.82.53:4146 , piece_id=3V3DVGUOPJKDWIGQ2TU7M6VFYFVAS6ODY62ERGSU6LI3LSTWQ4YQ, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:31:27.456050336 (80h8m3.019373336s) [noise=true , address=104.36.82.52:4089 , piece_id=X2VG4S3UHVNWA4AQJS7FLKWNVLTOZ3AKWI5ZDH24QCA3ENQW7BXQ, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:31:55.270234251 (80h8m30.833557256s) [noise=true , address=104.36.82.11:6125 , piece_id=2CX7RJOKNQA6ODZZZXYPSCYAQY5P35TWX7I56CMYAPFLJE3JEO5Q, seg_pos={0 0}, upload=64142]: Done uploading piece. err: ecclient: upload failed (node:12vricofwx67FRaxAGLbmgT1TYyKkqfJbxdsyeEPL4R7kg8pKzT, address:104.36.82.11:6125): protocol: expected piece hash; context deadline exceeded; EOF Jun 22 00:31:55.270319418 (80h8m30.833642407s) [noise=true , address=104.36.82.55:4224 , piece_id=XF7H2BYSNNNZOS2T77VIN75N76AYUXAQEI6D4LRXACU2KU62V2TA, seg_pos={0 0}, upload=64142]: Done uploading piece. err: ecclient: upload failed (node:12EjKmUiv4pxkVCB4WoiXim4uFpxrBPyfiSS4efTY3HKSE3jb5y, address:104.36.82.55:4224): protocol: write tcp 159.223.143.39:39152->104.36.82.55:4224: use of closed network connection; write tcp 159.223.143.39:39152->104.36.82.55:4224: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:39152->104.36.82.55:4224: use of closed network connection Jun 22 00:31:55.270353811 (80h8m30.833676799s) [seg_pos={0 0}, upload=64142]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Jun 22 00:31:55.301099877 (80h8m30.864422884s) [upload=64142]: Done uploading segments... Jun 22 00:31:55.301121567 (80h8m30.864444566s) [upload=64142]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 24.902549167s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [..X...................X.....................................] duration: 9.237321957s FAIL (2025-06-22 01:02:18 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 9.237352997s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [...X...................X....................................] duration: 25.95253686s FAIL (2025-06-22 01:02:06 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 25.952577396s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.....................X......................................] duration: 19.775796143s FAIL (2025-06-22 01:01:42 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== ing upload Jun 22 00:30:55.303646184 (80h7m30.866969171s) [upload=64142]: Uploading segments... Jun 22 00:30:55.303706406 (80h7m30.867029416s) [upload=64142]: Got next segment. Inline: false Jun 22 00:30:55.303762479 (80h7m30.867085468s) [upload=64142]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 22 00:30:55.303768627 (80h7m30.867091636s) [upload=64142]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 22 00:30:55.349058140 (80h7m30.912381168s) [seg_pos={0 0}, upload=64142]: Begin upload segment... Jun 22 00:30:55.349113768 (80h7m30.912436772s) [seg_pos={0 0}, upload=64142]: Done begin upload segment. Jun 22 00:30:55.349143121 (80h7m30.912466131s) [noise=true , address=45.91.116.5:5310 , piece_id=AXJOKDZRPVNL7HFT4QKJWRAHFM2UCM3GEVQB2KNYD6CXB6TH4D4Q, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.349660917 (80h7m30.912983928s) [noise=true , address=77.90.31.10:4533 , piece_id=SE6YAKKFDL3BV677CDDUC32UNMPXJHTG23GWWYK5B4YIPKER6GAQ, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.350093077 (80h7m30.913416089s) [noise=true , address=79.127.201.215:4326 , piece_id=USEWZDX6P52U6MCALTBNQG3SV7N2JRKNU6O23TLS6JEWAS4D7ARA, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.350480894 (80h7m30.913803916s) [noise=true , address=104.36.82.13:6199 , piece_id=YP62N2TO446YR3QRG5YLXECD7TN6SJ4UNNLZSK5H4CMV6JIF4AKA, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.350669998 (80h7m30.913993011s) [noise=true , address=104.36.82.54:4184 , piece_id=IZICVGZFMAYLIJTNUJ2OFLZVNZ6IPPYDZO4BSTJJPLHVCVOE5EAA, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.351201352 (80h7m30.914524366s) [noise=true , address=104.36.82.55:4224 , piece_id=XF7H2BYSNNNZOS2T77VIN75N76AYUXAQEI6D4LRXACU2KU62V2TA, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.351668071 (80h7m30.914991083s) [noise=true , address=104.36.82.56:4261 , piece_id=EYOYOYLDACGMPGJYX6ILQT3XHMUTQNXZEDC7ANHLZXIKYVDQFFVQ, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.352193126 (80h7m30.915516138s) [noise=true , address=104.36.82.52:4089 , piece_id=X2VG4S3UHVNWA4AQJS7FLKWNVLTOZ3AKWI5ZDH24QCA3ENQW7BXQ, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.352587096 (80h7m30.915910103s) [noise=true , address=79.127.220.98:4402 , piece_id=TE2WQO5LK45AVMPIAIGVSIBOVZLC4FG4ESQ72X6KFYC7IAEN2DRA, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.353420253 (80h7m30.916743265s) [noise=true , address=104.36.82.11:6125 , piece_id=2CX7RJOKNQA6ODZZZXYPSCYAQY5P35TWX7I56CMYAPFLJE3JEO5Q, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.353791565 (80h7m30.917114578s) [noise=true , address=104.36.82.42:6803 , piece_id=R626UFD326S2UBJEZ2WMHVR7EN4SK427VXPDJZOOZ32MX25JWZWQ, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.354171226 (80h7m30.917494233s) [noise=true , address=104.36.82.14:6245 , piece_id=II36G46OA6TINLR244U6D7Q6LGV754UA5M62K3EX7ZOBESMJUEHA, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.354542282 (80h7m30.917865288s) [noise=true , address=79.127.205.235:5660 , piece_id=UGFPVS7VNB7U2YSFPSGYDIRZV5R2DHCN57PJP6VOXO52WIAYHLXA, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.354891655 (80h7m30.918214678s) [noise=true , address=104.36.82.15:6283 , piece_id=N7TH2EMIOPATVHCSNH2WXUGU7KMMEJ5LCHR22SWEHSA5OV4CUCXQ, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.355499503 (80h7m30.918822519s) [noise=true , address=77.90.31.14:4666 , piece_id=H4DTFDAVCN266PB235BJIQTQ32FBXTIB5AZSX3B4R2VIK554AFSQ, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.356575547 (80h7m30.919898539s) [noise=true , address=104.36.82.39:6756 , piece_id=OKKWMDJPZO6D5VRMGOE3GOCYGHM725JZTNOVYAWKJW3WYZPI4V5Q, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.357039590 (80h7m30.920362596s) [noise=true , address=104.36.82.53:4146 , piece_id=3V3DVGUOPJKDWIGQ2TU7M6VFYFVAS6ODY62ERGSU6LI3LSTWQ4YQ, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.360343566 (80h7m30.923666582s) [noise=true , address=104.36.82.9:6033 , piece_id=T5XA23LZELMSTWOLBSHOX4AHKDACF65OYCKTMZYTXDIBSXNA2T4A, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.452444064 (80h7m31.015767055s) [upload=64142]: Next returned nil segment Jun 22 00:30:55.452495148 (80h7m31.015818149s) [upload=64142]: Waiting for error group managing segments... Jun 22 00:30:55.532958808 (80h7m31.096281883s) [noise=true , address=104.36.82.9:6033 , piece_id=T5XA23LZELMSTWOLBSHOX4AHKDACF65OYCKTMZYTXDIBSXNA2T4A, seg_pos={0 0}, upload=64142]: Done uploading piece. err: ecclient: upload failed (node:1SV3LrcQW16wLewDzfyUWFB6sUgnERNJVdhz72CJmyde95M79C, address:104.36.82.9:6033): protocol: storage node overloaded, request limit: 2000 Jun 22 00:30:55.547802184 (80h7m31.111125188s) [noise=true , address=79.127.220.98:4402 , piece_id=TE2WQO5LK45AVMPIAIGVSIBOVZLC4FG4ESQ72X6KFYC7IAEN2DRA, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:30:55.554223621 (80h7m31.117546616s) [noise=true , address=77.90.31.14:4666 , piece_id=H4DTFDAVCN266PB235BJIQTQ32FBXTIB5AZSX3B4R2VIK554AFSQ, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:30:55.762514255 (80h7m31.325837245s) [noise=true , address=104.36.82.14:6245 , piece_id=II36G46OA6TINLR244U6D7Q6LGV754UA5M62K3EX7ZOBESMJUEHA, seg_pos={0 0}, upload=64142]: Done uploading piece. err: ecclient: upload failed (node:1Zu8teNPTXn638grHKwtJqz4UEY7oFYLx1DHvuVP6RZXcatVqX, address:104.36.82.14:6245): protocol: storage node overloaded, request limit: 2000 Jun 22 00:30:55.773669583 (80h7m31.3369926s) [noise=true , address=79.127.205.235:5660 , piece_id=UGFPVS7VNB7U2YSFPSGYDIRZV5R2DHCN57PJP6VOXO52WIAYHLXA, seg_pos={0 0}, upload=64142]: Done uploading piece. err: ecclient: upload failed (node:12E6uCX9KL4Ud7YpxMXepvMXuqTFJCiSXByDjW94Am3U9H2Nf7f, address:79.127.205.235:5660): protocol: storage node overloaded, request limit: 2000 Jun 22 00:30:56.968366655 (80h7m32.53168964s) [noise=true , address=77.90.31.10:4533 , piece_id=SE6YAKKFDL3BV677CDDUC32UNMPXJHTG23GWWYK5B4YIPKER6GAQ, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:30:57.403572874 (80h7m32.966895883s) [noise=true , address=45.91.116.5:5310 , piece_id=AXJOKDZRPVNL7HFT4QKJWRAHFM2UCM3GEVQB2KNYD6CXB6TH4D4Q, seg_pos={0 0}, upload=64142]: Done uploading piece. err: ecclient: upload failed (node:127RMUc62xRq9RU5z3XfkZsq5sehmv3kP8o53tr9BxVcPE3GQCY, address:45.91.116.5:5310): protocol: storage node overloaded, request limit: 2000 Jun 22 00:30:58.210389994 (80h7m33.773712995s) [noise=true , address=104.36.82.13:6199 , piece_id=YP62N2TO446YR3QRG5YLXECD7TN6SJ4UNNLZSK5H4CMV6JIF4AKA, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:30:58.693037961 (80h7m34.256360966s) [noise=true , address=104.36.82.39:6756 , piece_id=OKKWMDJPZO6D5VRMGOE3GOCYGHM725JZTNOVYAWKJW3WYZPI4V5Q, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:30:59.124834814 (80h7m34.688157819s) [noise=true , address=104.36.82.15:6283 , piece_id=N7TH2EMIOPATVHCSNH2WXUGU7KMMEJ5LCHR22SWEHSA5OV4CUCXQ, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:31:00.629657982 (80h7m36.192980977s) [noise=true , address=104.36.82.42:6803 , piece_id=R626UFD326S2UBJEZ2WMHVR7EN4SK427VXPDJZOOZ32MX25JWZWQ, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:31:00.933110546 (80h7m36.496433536s) [noise=true , address=79.127.201.215:4326 , piece_id=USEWZDX6P52U6MCALTBNQG3SV7N2JRKNU6O23TLS6JEWAS4D7ARA, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:31:05.669167243 (80h7m41.232490237s) [noise=true , address=104.36.82.56:4261 , piece_id=EYOYOYLDACGMPGJYX6ILQT3XHMUTQNXZEDC7ANHLZXIKYVDQFFVQ, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:31:08.443582763 (80h7m44.006905763s) [noise=true , address=104.36.82.54:4184 , piece_id=IZICVGZFMAYLIJTNUJ2OFLZVNZ6IPPYDZO4BSTJJPLHVCVOE5EAA, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:31:26.459138014 (80h8m2.022461013s) [noise=true , address=104.36.82.53:4146 , piece_id=3V3DVGUOPJKDWIGQ2TU7M6VFYFVAS6ODY62ERGSU6LI3LSTWQ4YQ, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:31:27.456050336 (80h8m3.019373336s) [noise=true , address=104.36.82.52:4089 , piece_id=X2VG4S3UHVNWA4AQJS7FLKWNVLTOZ3AKWI5ZDH24QCA3ENQW7BXQ, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:31:55.270234251 (80h8m30.833557256s) [noise=true , address=104.36.82.11:6125 , piece_id=2CX7RJOKNQA6ODZZZXYPSCYAQY5P35TWX7I56CMYAPFLJE3JEO5Q, seg_pos={0 0}, upload=64142]: Done uploading piece. err: ecclient: upload failed (node:12vricofwx67FRaxAGLbmgT1TYyKkqfJbxdsyeEPL4R7kg8pKzT, address:104.36.82.11:6125): protocol: expected piece hash; context deadline exceeded; EOF Jun 22 00:31:55.270319418 (80h8m30.833642407s) [noise=true , address=104.36.82.55:4224 , piece_id=XF7H2BYSNNNZOS2T77VIN75N76AYUXAQEI6D4LRXACU2KU62V2TA, seg_pos={0 0}, upload=64142]: Done uploading piece. err: ecclient: upload failed (node:12EjKmUiv4pxkVCB4WoiXim4uFpxrBPyfiSS4efTY3HKSE3jb5y, address:104.36.82.55:4224): protocol: write tcp 159.223.143.39:39152->104.36.82.55:4224: use of closed network connection; write tcp 159.223.143.39:39152->104.36.82.55:4224: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:39152->104.36.82.55:4224: use of closed network connection Jun 22 00:31:55.270353811 (80h8m30.833676799s) [seg_pos={0 0}, upload=64142]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Jun 22 00:31:55.301099877 (80h8m30.864422884s) [upload=64142]: Done uploading segments... Jun 22 00:31:55.301121567 (80h8m30.864444566s) [upload=64142]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 19.775828086s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [....X...................X...................................] duration: 13.575515478s FAIL (2025-06-22 01:00:52 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 13.57553998s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [......................X.....................................] duration: 14.648701621s FAIL (2025-06-22 01:00:48 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== ing upload Jun 22 00:30:55.303646184 (80h7m30.866969171s) [upload=64142]: Uploading segments... Jun 22 00:30:55.303706406 (80h7m30.867029416s) [upload=64142]: Got next segment. Inline: false Jun 22 00:30:55.303762479 (80h7m30.867085468s) [upload=64142]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 22 00:30:55.303768627 (80h7m30.867091636s) [upload=64142]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 22 00:30:55.349058140 (80h7m30.912381168s) [seg_pos={0 0}, upload=64142]: Begin upload segment... Jun 22 00:30:55.349113768 (80h7m30.912436772s) [seg_pos={0 0}, upload=64142]: Done begin upload segment. Jun 22 00:30:55.349143121 (80h7m30.912466131s) [noise=true , address=45.91.116.5:5310 , piece_id=AXJOKDZRPVNL7HFT4QKJWRAHFM2UCM3GEVQB2KNYD6CXB6TH4D4Q, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.349660917 (80h7m30.912983928s) [noise=true , address=77.90.31.10:4533 , piece_id=SE6YAKKFDL3BV677CDDUC32UNMPXJHTG23GWWYK5B4YIPKER6GAQ, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.350093077 (80h7m30.913416089s) [noise=true , address=79.127.201.215:4326 , piece_id=USEWZDX6P52U6MCALTBNQG3SV7N2JRKNU6O23TLS6JEWAS4D7ARA, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.350480894 (80h7m30.913803916s) [noise=true , address=104.36.82.13:6199 , piece_id=YP62N2TO446YR3QRG5YLXECD7TN6SJ4UNNLZSK5H4CMV6JIF4AKA, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.350669998 (80h7m30.913993011s) [noise=true , address=104.36.82.54:4184 , piece_id=IZICVGZFMAYLIJTNUJ2OFLZVNZ6IPPYDZO4BSTJJPLHVCVOE5EAA, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.351201352 (80h7m30.914524366s) [noise=true , address=104.36.82.55:4224 , piece_id=XF7H2BYSNNNZOS2T77VIN75N76AYUXAQEI6D4LRXACU2KU62V2TA, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.351668071 (80h7m30.914991083s) [noise=true , address=104.36.82.56:4261 , piece_id=EYOYOYLDACGMPGJYX6ILQT3XHMUTQNXZEDC7ANHLZXIKYVDQFFVQ, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.352193126 (80h7m30.915516138s) [noise=true , address=104.36.82.52:4089 , piece_id=X2VG4S3UHVNWA4AQJS7FLKWNVLTOZ3AKWI5ZDH24QCA3ENQW7BXQ, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.352587096 (80h7m30.915910103s) [noise=true , address=79.127.220.98:4402 , piece_id=TE2WQO5LK45AVMPIAIGVSIBOVZLC4FG4ESQ72X6KFYC7IAEN2DRA, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.353420253 (80h7m30.916743265s) [noise=true , address=104.36.82.11:6125 , piece_id=2CX7RJOKNQA6ODZZZXYPSCYAQY5P35TWX7I56CMYAPFLJE3JEO5Q, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.353791565 (80h7m30.917114578s) [noise=true , address=104.36.82.42:6803 , piece_id=R626UFD326S2UBJEZ2WMHVR7EN4SK427VXPDJZOOZ32MX25JWZWQ, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.354171226 (80h7m30.917494233s) [noise=true , address=104.36.82.14:6245 , piece_id=II36G46OA6TINLR244U6D7Q6LGV754UA5M62K3EX7ZOBESMJUEHA, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.354542282 (80h7m30.917865288s) [noise=true , address=79.127.205.235:5660 , piece_id=UGFPVS7VNB7U2YSFPSGYDIRZV5R2DHCN57PJP6VOXO52WIAYHLXA, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.354891655 (80h7m30.918214678s) [noise=true , address=104.36.82.15:6283 , piece_id=N7TH2EMIOPATVHCSNH2WXUGU7KMMEJ5LCHR22SWEHSA5OV4CUCXQ, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.355499503 (80h7m30.918822519s) [noise=true , address=77.90.31.14:4666 , piece_id=H4DTFDAVCN266PB235BJIQTQ32FBXTIB5AZSX3B4R2VIK554AFSQ, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.356575547 (80h7m30.919898539s) [noise=true , address=104.36.82.39:6756 , piece_id=OKKWMDJPZO6D5VRMGOE3GOCYGHM725JZTNOVYAWKJW3WYZPI4V5Q, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.357039590 (80h7m30.920362596s) [noise=true , address=104.36.82.53:4146 , piece_id=3V3DVGUOPJKDWIGQ2TU7M6VFYFVAS6ODY62ERGSU6LI3LSTWQ4YQ, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.360343566 (80h7m30.923666582s) [noise=true , address=104.36.82.9:6033 , piece_id=T5XA23LZELMSTWOLBSHOX4AHKDACF65OYCKTMZYTXDIBSXNA2T4A, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.452444064 (80h7m31.015767055s) [upload=64142]: Next returned nil segment Jun 22 00:30:55.452495148 (80h7m31.015818149s) [upload=64142]: Waiting for error group managing segments... Jun 22 00:30:55.532958808 (80h7m31.096281883s) [noise=true , address=104.36.82.9:6033 , piece_id=T5XA23LZELMSTWOLBSHOX4AHKDACF65OYCKTMZYTXDIBSXNA2T4A, seg_pos={0 0}, upload=64142]: Done uploading piece. err: ecclient: upload failed (node:1SV3LrcQW16wLewDzfyUWFB6sUgnERNJVdhz72CJmyde95M79C, address:104.36.82.9:6033): protocol: storage node overloaded, request limit: 2000 Jun 22 00:30:55.547802184 (80h7m31.111125188s) [noise=true , address=79.127.220.98:4402 , piece_id=TE2WQO5LK45AVMPIAIGVSIBOVZLC4FG4ESQ72X6KFYC7IAEN2DRA, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:30:55.554223621 (80h7m31.117546616s) [noise=true , address=77.90.31.14:4666 , piece_id=H4DTFDAVCN266PB235BJIQTQ32FBXTIB5AZSX3B4R2VIK554AFSQ, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:30:55.762514255 (80h7m31.325837245s) [noise=true , address=104.36.82.14:6245 , piece_id=II36G46OA6TINLR244U6D7Q6LGV754UA5M62K3EX7ZOBESMJUEHA, seg_pos={0 0}, upload=64142]: Done uploading piece. err: ecclient: upload failed (node:1Zu8teNPTXn638grHKwtJqz4UEY7oFYLx1DHvuVP6RZXcatVqX, address:104.36.82.14:6245): protocol: storage node overloaded, request limit: 2000 Jun 22 00:30:55.773669583 (80h7m31.3369926s) [noise=true , address=79.127.205.235:5660 , piece_id=UGFPVS7VNB7U2YSFPSGYDIRZV5R2DHCN57PJP6VOXO52WIAYHLXA, seg_pos={0 0}, upload=64142]: Done uploading piece. err: ecclient: upload failed (node:12E6uCX9KL4Ud7YpxMXepvMXuqTFJCiSXByDjW94Am3U9H2Nf7f, address:79.127.205.235:5660): protocol: storage node overloaded, request limit: 2000 Jun 22 00:30:56.968366655 (80h7m32.53168964s) [noise=true , address=77.90.31.10:4533 , piece_id=SE6YAKKFDL3BV677CDDUC32UNMPXJHTG23GWWYK5B4YIPKER6GAQ, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:30:57.403572874 (80h7m32.966895883s) [noise=true , address=45.91.116.5:5310 , piece_id=AXJOKDZRPVNL7HFT4QKJWRAHFM2UCM3GEVQB2KNYD6CXB6TH4D4Q, seg_pos={0 0}, upload=64142]: Done uploading piece. err: ecclient: upload failed (node:127RMUc62xRq9RU5z3XfkZsq5sehmv3kP8o53tr9BxVcPE3GQCY, address:45.91.116.5:5310): protocol: storage node overloaded, request limit: 2000 Jun 22 00:30:58.210389994 (80h7m33.773712995s) [noise=true , address=104.36.82.13:6199 , piece_id=YP62N2TO446YR3QRG5YLXECD7TN6SJ4UNNLZSK5H4CMV6JIF4AKA, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:30:58.693037961 (80h7m34.256360966s) [noise=true , address=104.36.82.39:6756 , piece_id=OKKWMDJPZO6D5VRMGOE3GOCYGHM725JZTNOVYAWKJW3WYZPI4V5Q, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:30:59.124834814 (80h7m34.688157819s) [noise=true , address=104.36.82.15:6283 , piece_id=N7TH2EMIOPATVHCSNH2WXUGU7KMMEJ5LCHR22SWEHSA5OV4CUCXQ, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:31:00.629657982 (80h7m36.192980977s) [noise=true , address=104.36.82.42:6803 , piece_id=R626UFD326S2UBJEZ2WMHVR7EN4SK427VXPDJZOOZ32MX25JWZWQ, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:31:00.933110546 (80h7m36.496433536s) [noise=true , address=79.127.201.215:4326 , piece_id=USEWZDX6P52U6MCALTBNQG3SV7N2JRKNU6O23TLS6JEWAS4D7ARA, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:31:05.669167243 (80h7m41.232490237s) [noise=true , address=104.36.82.56:4261 , piece_id=EYOYOYLDACGMPGJYX6ILQT3XHMUTQNXZEDC7ANHLZXIKYVDQFFVQ, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:31:08.443582763 (80h7m44.006905763s) [noise=true , address=104.36.82.54:4184 , piece_id=IZICVGZFMAYLIJTNUJ2OFLZVNZ6IPPYDZO4BSTJJPLHVCVOE5EAA, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:31:26.459138014 (80h8m2.022461013s) [noise=true , address=104.36.82.53:4146 , piece_id=3V3DVGUOPJKDWIGQ2TU7M6VFYFVAS6ODY62ERGSU6LI3LSTWQ4YQ, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:31:27.456050336 (80h8m3.019373336s) [noise=true , address=104.36.82.52:4089 , piece_id=X2VG4S3UHVNWA4AQJS7FLKWNVLTOZ3AKWI5ZDH24QCA3ENQW7BXQ, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:31:55.270234251 (80h8m30.833557256s) [noise=true , address=104.36.82.11:6125 , piece_id=2CX7RJOKNQA6ODZZZXYPSCYAQY5P35TWX7I56CMYAPFLJE3JEO5Q, seg_pos={0 0}, upload=64142]: Done uploading piece. err: ecclient: upload failed (node:12vricofwx67FRaxAGLbmgT1TYyKkqfJbxdsyeEPL4R7kg8pKzT, address:104.36.82.11:6125): protocol: expected piece hash; context deadline exceeded; EOF Jun 22 00:31:55.270319418 (80h8m30.833642407s) [noise=true , address=104.36.82.55:4224 , piece_id=XF7H2BYSNNNZOS2T77VIN75N76AYUXAQEI6D4LRXACU2KU62V2TA, seg_pos={0 0}, upload=64142]: Done uploading piece. err: ecclient: upload failed (node:12EjKmUiv4pxkVCB4WoiXim4uFpxrBPyfiSS4efTY3HKSE3jb5y, address:104.36.82.55:4224): protocol: write tcp 159.223.143.39:39152->104.36.82.55:4224: use of closed network connection; write tcp 159.223.143.39:39152->104.36.82.55:4224: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:39152->104.36.82.55:4224: use of closed network connection Jun 22 00:31:55.270353811 (80h8m30.833676799s) [seg_pos={0 0}, upload=64142]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Jun 22 00:31:55.301099877 (80h8m30.864422884s) [upload=64142]: Done uploading segments... Jun 22 00:31:55.301121567 (80h8m30.864444566s) [upload=64142]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 14.648739164s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [.....X...................X..................................] duration: 15.727294744s FAIL (2025-06-22 01:00:10 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 15.727329692s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.......................X....................................] duration: 24.200004931s FAIL (2025-06-22 01:00:06 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== ing upload Jun 22 00:30:55.303646184 (80h7m30.866969171s) [upload=64142]: Uploading segments... Jun 22 00:30:55.303706406 (80h7m30.867029416s) [upload=64142]: Got next segment. Inline: false Jun 22 00:30:55.303762479 (80h7m30.867085468s) [upload=64142]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 22 00:30:55.303768627 (80h7m30.867091636s) [upload=64142]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 22 00:30:55.349058140 (80h7m30.912381168s) [seg_pos={0 0}, upload=64142]: Begin upload segment... Jun 22 00:30:55.349113768 (80h7m30.912436772s) [seg_pos={0 0}, upload=64142]: Done begin upload segment. Jun 22 00:30:55.349143121 (80h7m30.912466131s) [noise=true , address=45.91.116.5:5310 , piece_id=AXJOKDZRPVNL7HFT4QKJWRAHFM2UCM3GEVQB2KNYD6CXB6TH4D4Q, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.349660917 (80h7m30.912983928s) [noise=true , address=77.90.31.10:4533 , piece_id=SE6YAKKFDL3BV677CDDUC32UNMPXJHTG23GWWYK5B4YIPKER6GAQ, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.350093077 (80h7m30.913416089s) [noise=true , address=79.127.201.215:4326 , piece_id=USEWZDX6P52U6MCALTBNQG3SV7N2JRKNU6O23TLS6JEWAS4D7ARA, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.350480894 (80h7m30.913803916s) [noise=true , address=104.36.82.13:6199 , piece_id=YP62N2TO446YR3QRG5YLXECD7TN6SJ4UNNLZSK5H4CMV6JIF4AKA, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.350669998 (80h7m30.913993011s) [noise=true , address=104.36.82.54:4184 , piece_id=IZICVGZFMAYLIJTNUJ2OFLZVNZ6IPPYDZO4BSTJJPLHVCVOE5EAA, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.351201352 (80h7m30.914524366s) [noise=true , address=104.36.82.55:4224 , piece_id=XF7H2BYSNNNZOS2T77VIN75N76AYUXAQEI6D4LRXACU2KU62V2TA, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.351668071 (80h7m30.914991083s) [noise=true , address=104.36.82.56:4261 , piece_id=EYOYOYLDACGMPGJYX6ILQT3XHMUTQNXZEDC7ANHLZXIKYVDQFFVQ, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.352193126 (80h7m30.915516138s) [noise=true , address=104.36.82.52:4089 , piece_id=X2VG4S3UHVNWA4AQJS7FLKWNVLTOZ3AKWI5ZDH24QCA3ENQW7BXQ, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.352587096 (80h7m30.915910103s) [noise=true , address=79.127.220.98:4402 , piece_id=TE2WQO5LK45AVMPIAIGVSIBOVZLC4FG4ESQ72X6KFYC7IAEN2DRA, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.353420253 (80h7m30.916743265s) [noise=true , address=104.36.82.11:6125 , piece_id=2CX7RJOKNQA6ODZZZXYPSCYAQY5P35TWX7I56CMYAPFLJE3JEO5Q, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.353791565 (80h7m30.917114578s) [noise=true , address=104.36.82.42:6803 , piece_id=R626UFD326S2UBJEZ2WMHVR7EN4SK427VXPDJZOOZ32MX25JWZWQ, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.354171226 (80h7m30.917494233s) [noise=true , address=104.36.82.14:6245 , piece_id=II36G46OA6TINLR244U6D7Q6LGV754UA5M62K3EX7ZOBESMJUEHA, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.354542282 (80h7m30.917865288s) [noise=true , address=79.127.205.235:5660 , piece_id=UGFPVS7VNB7U2YSFPSGYDIRZV5R2DHCN57PJP6VOXO52WIAYHLXA, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.354891655 (80h7m30.918214678s) [noise=true , address=104.36.82.15:6283 , piece_id=N7TH2EMIOPATVHCSNH2WXUGU7KMMEJ5LCHR22SWEHSA5OV4CUCXQ, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.355499503 (80h7m30.918822519s) [noise=true , address=77.90.31.14:4666 , piece_id=H4DTFDAVCN266PB235BJIQTQ32FBXTIB5AZSX3B4R2VIK554AFSQ, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.356575547 (80h7m30.919898539s) [noise=true , address=104.36.82.39:6756 , piece_id=OKKWMDJPZO6D5VRMGOE3GOCYGHM725JZTNOVYAWKJW3WYZPI4V5Q, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.357039590 (80h7m30.920362596s) [noise=true , address=104.36.82.53:4146 , piece_id=3V3DVGUOPJKDWIGQ2TU7M6VFYFVAS6ODY62ERGSU6LI3LSTWQ4YQ, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.360343566 (80h7m30.923666582s) [noise=true , address=104.36.82.9:6033 , piece_id=T5XA23LZELMSTWOLBSHOX4AHKDACF65OYCKTMZYTXDIBSXNA2T4A, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.452444064 (80h7m31.015767055s) [upload=64142]: Next returned nil segment Jun 22 00:30:55.452495148 (80h7m31.015818149s) [upload=64142]: Waiting for error group managing segments... Jun 22 00:30:55.532958808 (80h7m31.096281883s) [noise=true , address=104.36.82.9:6033 , piece_id=T5XA23LZELMSTWOLBSHOX4AHKDACF65OYCKTMZYTXDIBSXNA2T4A, seg_pos={0 0}, upload=64142]: Done uploading piece. err: ecclient: upload failed (node:1SV3LrcQW16wLewDzfyUWFB6sUgnERNJVdhz72CJmyde95M79C, address:104.36.82.9:6033): protocol: storage node overloaded, request limit: 2000 Jun 22 00:30:55.547802184 (80h7m31.111125188s) [noise=true , address=79.127.220.98:4402 , piece_id=TE2WQO5LK45AVMPIAIGVSIBOVZLC4FG4ESQ72X6KFYC7IAEN2DRA, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:30:55.554223621 (80h7m31.117546616s) [noise=true , address=77.90.31.14:4666 , piece_id=H4DTFDAVCN266PB235BJIQTQ32FBXTIB5AZSX3B4R2VIK554AFSQ, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:30:55.762514255 (80h7m31.325837245s) [noise=true , address=104.36.82.14:6245 , piece_id=II36G46OA6TINLR244U6D7Q6LGV754UA5M62K3EX7ZOBESMJUEHA, seg_pos={0 0}, upload=64142]: Done uploading piece. err: ecclient: upload failed (node:1Zu8teNPTXn638grHKwtJqz4UEY7oFYLx1DHvuVP6RZXcatVqX, address:104.36.82.14:6245): protocol: storage node overloaded, request limit: 2000 Jun 22 00:30:55.773669583 (80h7m31.3369926s) [noise=true , address=79.127.205.235:5660 , piece_id=UGFPVS7VNB7U2YSFPSGYDIRZV5R2DHCN57PJP6VOXO52WIAYHLXA, seg_pos={0 0}, upload=64142]: Done uploading piece. err: ecclient: upload failed (node:12E6uCX9KL4Ud7YpxMXepvMXuqTFJCiSXByDjW94Am3U9H2Nf7f, address:79.127.205.235:5660): protocol: storage node overloaded, request limit: 2000 Jun 22 00:30:56.968366655 (80h7m32.53168964s) [noise=true , address=77.90.31.10:4533 , piece_id=SE6YAKKFDL3BV677CDDUC32UNMPXJHTG23GWWYK5B4YIPKER6GAQ, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:30:57.403572874 (80h7m32.966895883s) [noise=true , address=45.91.116.5:5310 , piece_id=AXJOKDZRPVNL7HFT4QKJWRAHFM2UCM3GEVQB2KNYD6CXB6TH4D4Q, seg_pos={0 0}, upload=64142]: Done uploading piece. err: ecclient: upload failed (node:127RMUc62xRq9RU5z3XfkZsq5sehmv3kP8o53tr9BxVcPE3GQCY, address:45.91.116.5:5310): protocol: storage node overloaded, request limit: 2000 Jun 22 00:30:58.210389994 (80h7m33.773712995s) [noise=true , address=104.36.82.13:6199 , piece_id=YP62N2TO446YR3QRG5YLXECD7TN6SJ4UNNLZSK5H4CMV6JIF4AKA, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:30:58.693037961 (80h7m34.256360966s) [noise=true , address=104.36.82.39:6756 , piece_id=OKKWMDJPZO6D5VRMGOE3GOCYGHM725JZTNOVYAWKJW3WYZPI4V5Q, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:30:59.124834814 (80h7m34.688157819s) [noise=true , address=104.36.82.15:6283 , piece_id=N7TH2EMIOPATVHCSNH2WXUGU7KMMEJ5LCHR22SWEHSA5OV4CUCXQ, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:31:00.629657982 (80h7m36.192980977s) [noise=true , address=104.36.82.42:6803 , piece_id=R626UFD326S2UBJEZ2WMHVR7EN4SK427VXPDJZOOZ32MX25JWZWQ, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:31:00.933110546 (80h7m36.496433536s) [noise=true , address=79.127.201.215:4326 , piece_id=USEWZDX6P52U6MCALTBNQG3SV7N2JRKNU6O23TLS6JEWAS4D7ARA, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:31:05.669167243 (80h7m41.232490237s) [noise=true , address=104.36.82.56:4261 , piece_id=EYOYOYLDACGMPGJYX6ILQT3XHMUTQNXZEDC7ANHLZXIKYVDQFFVQ, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:31:08.443582763 (80h7m44.006905763s) [noise=true , address=104.36.82.54:4184 , piece_id=IZICVGZFMAYLIJTNUJ2OFLZVNZ6IPPYDZO4BSTJJPLHVCVOE5EAA, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:31:26.459138014 (80h8m2.022461013s) [noise=true , address=104.36.82.53:4146 , piece_id=3V3DVGUOPJKDWIGQ2TU7M6VFYFVAS6ODY62ERGSU6LI3LSTWQ4YQ, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:31:27.456050336 (80h8m3.019373336s) [noise=true , address=104.36.82.52:4089 , piece_id=X2VG4S3UHVNWA4AQJS7FLKWNVLTOZ3AKWI5ZDH24QCA3ENQW7BXQ, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:31:55.270234251 (80h8m30.833557256s) [noise=true , address=104.36.82.11:6125 , piece_id=2CX7RJOKNQA6ODZZZXYPSCYAQY5P35TWX7I56CMYAPFLJE3JEO5Q, seg_pos={0 0}, upload=64142]: Done uploading piece. err: ecclient: upload failed (node:12vricofwx67FRaxAGLbmgT1TYyKkqfJbxdsyeEPL4R7kg8pKzT, address:104.36.82.11:6125): protocol: expected piece hash; context deadline exceeded; EOF Jun 22 00:31:55.270319418 (80h8m30.833642407s) [noise=true , address=104.36.82.55:4224 , piece_id=XF7H2BYSNNNZOS2T77VIN75N76AYUXAQEI6D4LRXACU2KU62V2TA, seg_pos={0 0}, upload=64142]: Done uploading piece. err: ecclient: upload failed (node:12EjKmUiv4pxkVCB4WoiXim4uFpxrBPyfiSS4efTY3HKSE3jb5y, address:104.36.82.55:4224): protocol: write tcp 159.223.143.39:39152->104.36.82.55:4224: use of closed network connection; write tcp 159.223.143.39:39152->104.36.82.55:4224: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:39152->104.36.82.55:4224: use of closed network connection Jun 22 00:31:55.270353811 (80h8m30.833676799s) [seg_pos={0 0}, upload=64142]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Jun 22 00:31:55.301099877 (80h8m30.864422884s) [upload=64142]: Done uploading segments... Jun 22 00:31:55.301121567 (80h8m30.864444566s) [upload=64142]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 24.200036395s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [......X...................X.................................] duration: 21.767553741s FAIL (2025-06-22 00:59:41 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 21.767578932s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [........................X...................................] duration: 22.026591465s FAIL (2025-06-22 00:59:36 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== ing upload Jun 22 00:30:55.303646184 (80h7m30.866969171s) [upload=64142]: Uploading segments... Jun 22 00:30:55.303706406 (80h7m30.867029416s) [upload=64142]: Got next segment. Inline: false Jun 22 00:30:55.303762479 (80h7m30.867085468s) [upload=64142]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 22 00:30:55.303768627 (80h7m30.867091636s) [upload=64142]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 22 00:30:55.349058140 (80h7m30.912381168s) [seg_pos={0 0}, upload=64142]: Begin upload segment... Jun 22 00:30:55.349113768 (80h7m30.912436772s) [seg_pos={0 0}, upload=64142]: Done begin upload segment. Jun 22 00:30:55.349143121 (80h7m30.912466131s) [noise=true , address=45.91.116.5:5310 , piece_id=AXJOKDZRPVNL7HFT4QKJWRAHFM2UCM3GEVQB2KNYD6CXB6TH4D4Q, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.349660917 (80h7m30.912983928s) [noise=true , address=77.90.31.10:4533 , piece_id=SE6YAKKFDL3BV677CDDUC32UNMPXJHTG23GWWYK5B4YIPKER6GAQ, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.350093077 (80h7m30.913416089s) [noise=true , address=79.127.201.215:4326 , piece_id=USEWZDX6P52U6MCALTBNQG3SV7N2JRKNU6O23TLS6JEWAS4D7ARA, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.350480894 (80h7m30.913803916s) [noise=true , address=104.36.82.13:6199 , piece_id=YP62N2TO446YR3QRG5YLXECD7TN6SJ4UNNLZSK5H4CMV6JIF4AKA, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.350669998 (80h7m30.913993011s) [noise=true , address=104.36.82.54:4184 , piece_id=IZICVGZFMAYLIJTNUJ2OFLZVNZ6IPPYDZO4BSTJJPLHVCVOE5EAA, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.351201352 (80h7m30.914524366s) [noise=true , address=104.36.82.55:4224 , piece_id=XF7H2BYSNNNZOS2T77VIN75N76AYUXAQEI6D4LRXACU2KU62V2TA, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.351668071 (80h7m30.914991083s) [noise=true , address=104.36.82.56:4261 , piece_id=EYOYOYLDACGMPGJYX6ILQT3XHMUTQNXZEDC7ANHLZXIKYVDQFFVQ, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.352193126 (80h7m30.915516138s) [noise=true , address=104.36.82.52:4089 , piece_id=X2VG4S3UHVNWA4AQJS7FLKWNVLTOZ3AKWI5ZDH24QCA3ENQW7BXQ, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.352587096 (80h7m30.915910103s) [noise=true , address=79.127.220.98:4402 , piece_id=TE2WQO5LK45AVMPIAIGVSIBOVZLC4FG4ESQ72X6KFYC7IAEN2DRA, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.353420253 (80h7m30.916743265s) [noise=true , address=104.36.82.11:6125 , piece_id=2CX7RJOKNQA6ODZZZXYPSCYAQY5P35TWX7I56CMYAPFLJE3JEO5Q, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.353791565 (80h7m30.917114578s) [noise=true , address=104.36.82.42:6803 , piece_id=R626UFD326S2UBJEZ2WMHVR7EN4SK427VXPDJZOOZ32MX25JWZWQ, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.354171226 (80h7m30.917494233s) [noise=true , address=104.36.82.14:6245 , piece_id=II36G46OA6TINLR244U6D7Q6LGV754UA5M62K3EX7ZOBESMJUEHA, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.354542282 (80h7m30.917865288s) [noise=true , address=79.127.205.235:5660 , piece_id=UGFPVS7VNB7U2YSFPSGYDIRZV5R2DHCN57PJP6VOXO52WIAYHLXA, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.354891655 (80h7m30.918214678s) [noise=true , address=104.36.82.15:6283 , piece_id=N7TH2EMIOPATVHCSNH2WXUGU7KMMEJ5LCHR22SWEHSA5OV4CUCXQ, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.355499503 (80h7m30.918822519s) [noise=true , address=77.90.31.14:4666 , piece_id=H4DTFDAVCN266PB235BJIQTQ32FBXTIB5AZSX3B4R2VIK554AFSQ, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.356575547 (80h7m30.919898539s) [noise=true , address=104.36.82.39:6756 , piece_id=OKKWMDJPZO6D5VRMGOE3GOCYGHM725JZTNOVYAWKJW3WYZPI4V5Q, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.357039590 (80h7m30.920362596s) [noise=true , address=104.36.82.53:4146 , piece_id=3V3DVGUOPJKDWIGQ2TU7M6VFYFVAS6ODY62ERGSU6LI3LSTWQ4YQ, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.360343566 (80h7m30.923666582s) [noise=true , address=104.36.82.9:6033 , piece_id=T5XA23LZELMSTWOLBSHOX4AHKDACF65OYCKTMZYTXDIBSXNA2T4A, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.452444064 (80h7m31.015767055s) [upload=64142]: Next returned nil segment Jun 22 00:30:55.452495148 (80h7m31.015818149s) [upload=64142]: Waiting for error group managing segments... Jun 22 00:30:55.532958808 (80h7m31.096281883s) [noise=true , address=104.36.82.9:6033 , piece_id=T5XA23LZELMSTWOLBSHOX4AHKDACF65OYCKTMZYTXDIBSXNA2T4A, seg_pos={0 0}, upload=64142]: Done uploading piece. err: ecclient: upload failed (node:1SV3LrcQW16wLewDzfyUWFB6sUgnERNJVdhz72CJmyde95M79C, address:104.36.82.9:6033): protocol: storage node overloaded, request limit: 2000 Jun 22 00:30:55.547802184 (80h7m31.111125188s) [noise=true , address=79.127.220.98:4402 , piece_id=TE2WQO5LK45AVMPIAIGVSIBOVZLC4FG4ESQ72X6KFYC7IAEN2DRA, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:30:55.554223621 (80h7m31.117546616s) [noise=true , address=77.90.31.14:4666 , piece_id=H4DTFDAVCN266PB235BJIQTQ32FBXTIB5AZSX3B4R2VIK554AFSQ, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:30:55.762514255 (80h7m31.325837245s) [noise=true , address=104.36.82.14:6245 , piece_id=II36G46OA6TINLR244U6D7Q6LGV754UA5M62K3EX7ZOBESMJUEHA, seg_pos={0 0}, upload=64142]: Done uploading piece. err: ecclient: upload failed (node:1Zu8teNPTXn638grHKwtJqz4UEY7oFYLx1DHvuVP6RZXcatVqX, address:104.36.82.14:6245): protocol: storage node overloaded, request limit: 2000 Jun 22 00:30:55.773669583 (80h7m31.3369926s) [noise=true , address=79.127.205.235:5660 , piece_id=UGFPVS7VNB7U2YSFPSGYDIRZV5R2DHCN57PJP6VOXO52WIAYHLXA, seg_pos={0 0}, upload=64142]: Done uploading piece. err: ecclient: upload failed (node:12E6uCX9KL4Ud7YpxMXepvMXuqTFJCiSXByDjW94Am3U9H2Nf7f, address:79.127.205.235:5660): protocol: storage node overloaded, request limit: 2000 Jun 22 00:30:56.968366655 (80h7m32.53168964s) [noise=true , address=77.90.31.10:4533 , piece_id=SE6YAKKFDL3BV677CDDUC32UNMPXJHTG23GWWYK5B4YIPKER6GAQ, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:30:57.403572874 (80h7m32.966895883s) [noise=true , address=45.91.116.5:5310 , piece_id=AXJOKDZRPVNL7HFT4QKJWRAHFM2UCM3GEVQB2KNYD6CXB6TH4D4Q, seg_pos={0 0}, upload=64142]: Done uploading piece. err: ecclient: upload failed (node:127RMUc62xRq9RU5z3XfkZsq5sehmv3kP8o53tr9BxVcPE3GQCY, address:45.91.116.5:5310): protocol: storage node overloaded, request limit: 2000 Jun 22 00:30:58.210389994 (80h7m33.773712995s) [noise=true , address=104.36.82.13:6199 , piece_id=YP62N2TO446YR3QRG5YLXECD7TN6SJ4UNNLZSK5H4CMV6JIF4AKA, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:30:58.693037961 (80h7m34.256360966s) [noise=true , address=104.36.82.39:6756 , piece_id=OKKWMDJPZO6D5VRMGOE3GOCYGHM725JZTNOVYAWKJW3WYZPI4V5Q, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:30:59.124834814 (80h7m34.688157819s) [noise=true , address=104.36.82.15:6283 , piece_id=N7TH2EMIOPATVHCSNH2WXUGU7KMMEJ5LCHR22SWEHSA5OV4CUCXQ, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:31:00.629657982 (80h7m36.192980977s) [noise=true , address=104.36.82.42:6803 , piece_id=R626UFD326S2UBJEZ2WMHVR7EN4SK427VXPDJZOOZ32MX25JWZWQ, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:31:00.933110546 (80h7m36.496433536s) [noise=true , address=79.127.201.215:4326 , piece_id=USEWZDX6P52U6MCALTBNQG3SV7N2JRKNU6O23TLS6JEWAS4D7ARA, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:31:05.669167243 (80h7m41.232490237s) [noise=true , address=104.36.82.56:4261 , piece_id=EYOYOYLDACGMPGJYX6ILQT3XHMUTQNXZEDC7ANHLZXIKYVDQFFVQ, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:31:08.443582763 (80h7m44.006905763s) [noise=true , address=104.36.82.54:4184 , piece_id=IZICVGZFMAYLIJTNUJ2OFLZVNZ6IPPYDZO4BSTJJPLHVCVOE5EAA, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:31:26.459138014 (80h8m2.022461013s) [noise=true , address=104.36.82.53:4146 , piece_id=3V3DVGUOPJKDWIGQ2TU7M6VFYFVAS6ODY62ERGSU6LI3LSTWQ4YQ, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:31:27.456050336 (80h8m3.019373336s) [noise=true , address=104.36.82.52:4089 , piece_id=X2VG4S3UHVNWA4AQJS7FLKWNVLTOZ3AKWI5ZDH24QCA3ENQW7BXQ, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:31:55.270234251 (80h8m30.833557256s) [noise=true , address=104.36.82.11:6125 , piece_id=2CX7RJOKNQA6ODZZZXYPSCYAQY5P35TWX7I56CMYAPFLJE3JEO5Q, seg_pos={0 0}, upload=64142]: Done uploading piece. err: ecclient: upload failed (node:12vricofwx67FRaxAGLbmgT1TYyKkqfJbxdsyeEPL4R7kg8pKzT, address:104.36.82.11:6125): protocol: expected piece hash; context deadline exceeded; EOF Jun 22 00:31:55.270319418 (80h8m30.833642407s) [noise=true , address=104.36.82.55:4224 , piece_id=XF7H2BYSNNNZOS2T77VIN75N76AYUXAQEI6D4LRXACU2KU62V2TA, seg_pos={0 0}, upload=64142]: Done uploading piece. err: ecclient: upload failed (node:12EjKmUiv4pxkVCB4WoiXim4uFpxrBPyfiSS4efTY3HKSE3jb5y, address:104.36.82.55:4224): protocol: write tcp 159.223.143.39:39152->104.36.82.55:4224: use of closed network connection; write tcp 159.223.143.39:39152->104.36.82.55:4224: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:39152->104.36.82.55:4224: use of closed network connection Jun 22 00:31:55.270353811 (80h8m30.833676799s) [seg_pos={0 0}, upload=64142]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Jun 22 00:31:55.301099877 (80h8m30.864422884s) [upload=64142]: Done uploading segments... Jun 22 00:31:55.301121567 (80h8m30.864444566s) [upload=64142]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 22.026661142s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [.......X...................X................................] duration: 14.681752741s FAIL (2025-06-22 00:58:41 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 14.6819845s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [........X...................X...............................] duration: 31.501366625s FAIL (2025-06-22 00:58:27 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 31.501385259s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.........................X..................................] duration: 19.496961873s FAIL (2025-06-22 00:58:21 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== ing upload Jun 22 00:30:55.303646184 (80h7m30.866969171s) [upload=64142]: Uploading segments... Jun 22 00:30:55.303706406 (80h7m30.867029416s) [upload=64142]: Got next segment. Inline: false Jun 22 00:30:55.303762479 (80h7m30.867085468s) [upload=64142]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 22 00:30:55.303768627 (80h7m30.867091636s) [upload=64142]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 22 00:30:55.349058140 (80h7m30.912381168s) [seg_pos={0 0}, upload=64142]: Begin upload segment... Jun 22 00:30:55.349113768 (80h7m30.912436772s) [seg_pos={0 0}, upload=64142]: Done begin upload segment. Jun 22 00:30:55.349143121 (80h7m30.912466131s) [noise=true , address=45.91.116.5:5310 , piece_id=AXJOKDZRPVNL7HFT4QKJWRAHFM2UCM3GEVQB2KNYD6CXB6TH4D4Q, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.349660917 (80h7m30.912983928s) [noise=true , address=77.90.31.10:4533 , piece_id=SE6YAKKFDL3BV677CDDUC32UNMPXJHTG23GWWYK5B4YIPKER6GAQ, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.350093077 (80h7m30.913416089s) [noise=true , address=79.127.201.215:4326 , piece_id=USEWZDX6P52U6MCALTBNQG3SV7N2JRKNU6O23TLS6JEWAS4D7ARA, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.350480894 (80h7m30.913803916s) [noise=true , address=104.36.82.13:6199 , piece_id=YP62N2TO446YR3QRG5YLXECD7TN6SJ4UNNLZSK5H4CMV6JIF4AKA, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.350669998 (80h7m30.913993011s) [noise=true , address=104.36.82.54:4184 , piece_id=IZICVGZFMAYLIJTNUJ2OFLZVNZ6IPPYDZO4BSTJJPLHVCVOE5EAA, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.351201352 (80h7m30.914524366s) [noise=true , address=104.36.82.55:4224 , piece_id=XF7H2BYSNNNZOS2T77VIN75N76AYUXAQEI6D4LRXACU2KU62V2TA, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.351668071 (80h7m30.914991083s) [noise=true , address=104.36.82.56:4261 , piece_id=EYOYOYLDACGMPGJYX6ILQT3XHMUTQNXZEDC7ANHLZXIKYVDQFFVQ, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.352193126 (80h7m30.915516138s) [noise=true , address=104.36.82.52:4089 , piece_id=X2VG4S3UHVNWA4AQJS7FLKWNVLTOZ3AKWI5ZDH24QCA3ENQW7BXQ, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.352587096 (80h7m30.915910103s) [noise=true , address=79.127.220.98:4402 , piece_id=TE2WQO5LK45AVMPIAIGVSIBOVZLC4FG4ESQ72X6KFYC7IAEN2DRA, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.353420253 (80h7m30.916743265s) [noise=true , address=104.36.82.11:6125 , piece_id=2CX7RJOKNQA6ODZZZXYPSCYAQY5P35TWX7I56CMYAPFLJE3JEO5Q, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.353791565 (80h7m30.917114578s) [noise=true , address=104.36.82.42:6803 , piece_id=R626UFD326S2UBJEZ2WMHVR7EN4SK427VXPDJZOOZ32MX25JWZWQ, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.354171226 (80h7m30.917494233s) [noise=true , address=104.36.82.14:6245 , piece_id=II36G46OA6TINLR244U6D7Q6LGV754UA5M62K3EX7ZOBESMJUEHA, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.354542282 (80h7m30.917865288s) [noise=true , address=79.127.205.235:5660 , piece_id=UGFPVS7VNB7U2YSFPSGYDIRZV5R2DHCN57PJP6VOXO52WIAYHLXA, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.354891655 (80h7m30.918214678s) [noise=true , address=104.36.82.15:6283 , piece_id=N7TH2EMIOPATVHCSNH2WXUGU7KMMEJ5LCHR22SWEHSA5OV4CUCXQ, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.355499503 (80h7m30.918822519s) [noise=true , address=77.90.31.14:4666 , piece_id=H4DTFDAVCN266PB235BJIQTQ32FBXTIB5AZSX3B4R2VIK554AFSQ, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.356575547 (80h7m30.919898539s) [noise=true , address=104.36.82.39:6756 , piece_id=OKKWMDJPZO6D5VRMGOE3GOCYGHM725JZTNOVYAWKJW3WYZPI4V5Q, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.357039590 (80h7m30.920362596s) [noise=true , address=104.36.82.53:4146 , piece_id=3V3DVGUOPJKDWIGQ2TU7M6VFYFVAS6ODY62ERGSU6LI3LSTWQ4YQ, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.360343566 (80h7m30.923666582s) [noise=true , address=104.36.82.9:6033 , piece_id=T5XA23LZELMSTWOLBSHOX4AHKDACF65OYCKTMZYTXDIBSXNA2T4A, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.452444064 (80h7m31.015767055s) [upload=64142]: Next returned nil segment Jun 22 00:30:55.452495148 (80h7m31.015818149s) [upload=64142]: Waiting for error group managing segments... Jun 22 00:30:55.532958808 (80h7m31.096281883s) [noise=true , address=104.36.82.9:6033 , piece_id=T5XA23LZELMSTWOLBSHOX4AHKDACF65OYCKTMZYTXDIBSXNA2T4A, seg_pos={0 0}, upload=64142]: Done uploading piece. err: ecclient: upload failed (node:1SV3LrcQW16wLewDzfyUWFB6sUgnERNJVdhz72CJmyde95M79C, address:104.36.82.9:6033): protocol: storage node overloaded, request limit: 2000 Jun 22 00:30:55.547802184 (80h7m31.111125188s) [noise=true , address=79.127.220.98:4402 , piece_id=TE2WQO5LK45AVMPIAIGVSIBOVZLC4FG4ESQ72X6KFYC7IAEN2DRA, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:30:55.554223621 (80h7m31.117546616s) [noise=true , address=77.90.31.14:4666 , piece_id=H4DTFDAVCN266PB235BJIQTQ32FBXTIB5AZSX3B4R2VIK554AFSQ, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:30:55.762514255 (80h7m31.325837245s) [noise=true , address=104.36.82.14:6245 , piece_id=II36G46OA6TINLR244U6D7Q6LGV754UA5M62K3EX7ZOBESMJUEHA, seg_pos={0 0}, upload=64142]: Done uploading piece. err: ecclient: upload failed (node:1Zu8teNPTXn638grHKwtJqz4UEY7oFYLx1DHvuVP6RZXcatVqX, address:104.36.82.14:6245): protocol: storage node overloaded, request limit: 2000 Jun 22 00:30:55.773669583 (80h7m31.3369926s) [noise=true , address=79.127.205.235:5660 , piece_id=UGFPVS7VNB7U2YSFPSGYDIRZV5R2DHCN57PJP6VOXO52WIAYHLXA, seg_pos={0 0}, upload=64142]: Done uploading piece. err: ecclient: upload failed (node:12E6uCX9KL4Ud7YpxMXepvMXuqTFJCiSXByDjW94Am3U9H2Nf7f, address:79.127.205.235:5660): protocol: storage node overloaded, request limit: 2000 Jun 22 00:30:56.968366655 (80h7m32.53168964s) [noise=true , address=77.90.31.10:4533 , piece_id=SE6YAKKFDL3BV677CDDUC32UNMPXJHTG23GWWYK5B4YIPKER6GAQ, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:30:57.403572874 (80h7m32.966895883s) [noise=true , address=45.91.116.5:5310 , piece_id=AXJOKDZRPVNL7HFT4QKJWRAHFM2UCM3GEVQB2KNYD6CXB6TH4D4Q, seg_pos={0 0}, upload=64142]: Done uploading piece. err: ecclient: upload failed (node:127RMUc62xRq9RU5z3XfkZsq5sehmv3kP8o53tr9BxVcPE3GQCY, address:45.91.116.5:5310): protocol: storage node overloaded, request limit: 2000 Jun 22 00:30:58.210389994 (80h7m33.773712995s) [noise=true , address=104.36.82.13:6199 , piece_id=YP62N2TO446YR3QRG5YLXECD7TN6SJ4UNNLZSK5H4CMV6JIF4AKA, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:30:58.693037961 (80h7m34.256360966s) [noise=true , address=104.36.82.39:6756 , piece_id=OKKWMDJPZO6D5VRMGOE3GOCYGHM725JZTNOVYAWKJW3WYZPI4V5Q, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:30:59.124834814 (80h7m34.688157819s) [noise=true , address=104.36.82.15:6283 , piece_id=N7TH2EMIOPATVHCSNH2WXUGU7KMMEJ5LCHR22SWEHSA5OV4CUCXQ, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:31:00.629657982 (80h7m36.192980977s) [noise=true , address=104.36.82.42:6803 , piece_id=R626UFD326S2UBJEZ2WMHVR7EN4SK427VXPDJZOOZ32MX25JWZWQ, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:31:00.933110546 (80h7m36.496433536s) [noise=true , address=79.127.201.215:4326 , piece_id=USEWZDX6P52U6MCALTBNQG3SV7N2JRKNU6O23TLS6JEWAS4D7ARA, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:31:05.669167243 (80h7m41.232490237s) [noise=true , address=104.36.82.56:4261 , piece_id=EYOYOYLDACGMPGJYX6ILQT3XHMUTQNXZEDC7ANHLZXIKYVDQFFVQ, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:31:08.443582763 (80h7m44.006905763s) [noise=true , address=104.36.82.54:4184 , piece_id=IZICVGZFMAYLIJTNUJ2OFLZVNZ6IPPYDZO4BSTJJPLHVCVOE5EAA, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:31:26.459138014 (80h8m2.022461013s) [noise=true , address=104.36.82.53:4146 , piece_id=3V3DVGUOPJKDWIGQ2TU7M6VFYFVAS6ODY62ERGSU6LI3LSTWQ4YQ, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:31:27.456050336 (80h8m3.019373336s) [noise=true , address=104.36.82.52:4089 , piece_id=X2VG4S3UHVNWA4AQJS7FLKWNVLTOZ3AKWI5ZDH24QCA3ENQW7BXQ, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:31:55.270234251 (80h8m30.833557256s) [noise=true , address=104.36.82.11:6125 , piece_id=2CX7RJOKNQA6ODZZZXYPSCYAQY5P35TWX7I56CMYAPFLJE3JEO5Q, seg_pos={0 0}, upload=64142]: Done uploading piece. err: ecclient: upload failed (node:12vricofwx67FRaxAGLbmgT1TYyKkqfJbxdsyeEPL4R7kg8pKzT, address:104.36.82.11:6125): protocol: expected piece hash; context deadline exceeded; EOF Jun 22 00:31:55.270319418 (80h8m30.833642407s) [noise=true , address=104.36.82.55:4224 , piece_id=XF7H2BYSNNNZOS2T77VIN75N76AYUXAQEI6D4LRXACU2KU62V2TA, seg_pos={0 0}, upload=64142]: Done uploading piece. err: ecclient: upload failed (node:12EjKmUiv4pxkVCB4WoiXim4uFpxrBPyfiSS4efTY3HKSE3jb5y, address:104.36.82.55:4224): protocol: write tcp 159.223.143.39:39152->104.36.82.55:4224: use of closed network connection; write tcp 159.223.143.39:39152->104.36.82.55:4224: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:39152->104.36.82.55:4224: use of closed network connection Jun 22 00:31:55.270353811 (80h8m30.833676799s) [seg_pos={0 0}, upload=64142]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Jun 22 00:31:55.301099877 (80h8m30.864422884s) [upload=64142]: Done uploading segments... Jun 22 00:31:55.301121567 (80h8m30.864444566s) [upload=64142]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 19.496983119s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [..........................X.................................] duration: 55.514129309s FAIL (2025-06-22 00:58:01 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== ing upload Jun 22 00:30:55.303646184 (80h7m30.866969171s) [upload=64142]: Uploading segments... Jun 22 00:30:55.303706406 (80h7m30.867029416s) [upload=64142]: Got next segment. Inline: false Jun 22 00:30:55.303762479 (80h7m30.867085468s) [upload=64142]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 22 00:30:55.303768627 (80h7m30.867091636s) [upload=64142]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 22 00:30:55.349058140 (80h7m30.912381168s) [seg_pos={0 0}, upload=64142]: Begin upload segment... Jun 22 00:30:55.349113768 (80h7m30.912436772s) [seg_pos={0 0}, upload=64142]: Done begin upload segment. Jun 22 00:30:55.349143121 (80h7m30.912466131s) [noise=true , address=45.91.116.5:5310 , piece_id=AXJOKDZRPVNL7HFT4QKJWRAHFM2UCM3GEVQB2KNYD6CXB6TH4D4Q, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.349660917 (80h7m30.912983928s) [noise=true , address=77.90.31.10:4533 , piece_id=SE6YAKKFDL3BV677CDDUC32UNMPXJHTG23GWWYK5B4YIPKER6GAQ, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.350093077 (80h7m30.913416089s) [noise=true , address=79.127.201.215:4326 , piece_id=USEWZDX6P52U6MCALTBNQG3SV7N2JRKNU6O23TLS6JEWAS4D7ARA, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.350480894 (80h7m30.913803916s) [noise=true , address=104.36.82.13:6199 , piece_id=YP62N2TO446YR3QRG5YLXECD7TN6SJ4UNNLZSK5H4CMV6JIF4AKA, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.350669998 (80h7m30.913993011s) [noise=true , address=104.36.82.54:4184 , piece_id=IZICVGZFMAYLIJTNUJ2OFLZVNZ6IPPYDZO4BSTJJPLHVCVOE5EAA, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.351201352 (80h7m30.914524366s) [noise=true , address=104.36.82.55:4224 , piece_id=XF7H2BYSNNNZOS2T77VIN75N76AYUXAQEI6D4LRXACU2KU62V2TA, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.351668071 (80h7m30.914991083s) [noise=true , address=104.36.82.56:4261 , piece_id=EYOYOYLDACGMPGJYX6ILQT3XHMUTQNXZEDC7ANHLZXIKYVDQFFVQ, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.352193126 (80h7m30.915516138s) [noise=true , address=104.36.82.52:4089 , piece_id=X2VG4S3UHVNWA4AQJS7FLKWNVLTOZ3AKWI5ZDH24QCA3ENQW7BXQ, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.352587096 (80h7m30.915910103s) [noise=true , address=79.127.220.98:4402 , piece_id=TE2WQO5LK45AVMPIAIGVSIBOVZLC4FG4ESQ72X6KFYC7IAEN2DRA, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.353420253 (80h7m30.916743265s) [noise=true , address=104.36.82.11:6125 , piece_id=2CX7RJOKNQA6ODZZZXYPSCYAQY5P35TWX7I56CMYAPFLJE3JEO5Q, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.353791565 (80h7m30.917114578s) [noise=true , address=104.36.82.42:6803 , piece_id=R626UFD326S2UBJEZ2WMHVR7EN4SK427VXPDJZOOZ32MX25JWZWQ, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.354171226 (80h7m30.917494233s) [noise=true , address=104.36.82.14:6245 , piece_id=II36G46OA6TINLR244U6D7Q6LGV754UA5M62K3EX7ZOBESMJUEHA, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.354542282 (80h7m30.917865288s) [noise=true , address=79.127.205.235:5660 , piece_id=UGFPVS7VNB7U2YSFPSGYDIRZV5R2DHCN57PJP6VOXO52WIAYHLXA, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.354891655 (80h7m30.918214678s) [noise=true , address=104.36.82.15:6283 , piece_id=N7TH2EMIOPATVHCSNH2WXUGU7KMMEJ5LCHR22SWEHSA5OV4CUCXQ, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.355499503 (80h7m30.918822519s) [noise=true , address=77.90.31.14:4666 , piece_id=H4DTFDAVCN266PB235BJIQTQ32FBXTIB5AZSX3B4R2VIK554AFSQ, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.356575547 (80h7m30.919898539s) [noise=true , address=104.36.82.39:6756 , piece_id=OKKWMDJPZO6D5VRMGOE3GOCYGHM725JZTNOVYAWKJW3WYZPI4V5Q, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.357039590 (80h7m30.920362596s) [noise=true , address=104.36.82.53:4146 , piece_id=3V3DVGUOPJKDWIGQ2TU7M6VFYFVAS6ODY62ERGSU6LI3LSTWQ4YQ, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.360343566 (80h7m30.923666582s) [noise=true , address=104.36.82.9:6033 , piece_id=T5XA23LZELMSTWOLBSHOX4AHKDACF65OYCKTMZYTXDIBSXNA2T4A, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.452444064 (80h7m31.015767055s) [upload=64142]: Next returned nil segment Jun 22 00:30:55.452495148 (80h7m31.015818149s) [upload=64142]: Waiting for error group managing segments... Jun 22 00:30:55.532958808 (80h7m31.096281883s) [noise=true , address=104.36.82.9:6033 , piece_id=T5XA23LZELMSTWOLBSHOX4AHKDACF65OYCKTMZYTXDIBSXNA2T4A, seg_pos={0 0}, upload=64142]: Done uploading piece. err: ecclient: upload failed (node:1SV3LrcQW16wLewDzfyUWFB6sUgnERNJVdhz72CJmyde95M79C, address:104.36.82.9:6033): protocol: storage node overloaded, request limit: 2000 Jun 22 00:30:55.547802184 (80h7m31.111125188s) [noise=true , address=79.127.220.98:4402 , piece_id=TE2WQO5LK45AVMPIAIGVSIBOVZLC4FG4ESQ72X6KFYC7IAEN2DRA, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:30:55.554223621 (80h7m31.117546616s) [noise=true , address=77.90.31.14:4666 , piece_id=H4DTFDAVCN266PB235BJIQTQ32FBXTIB5AZSX3B4R2VIK554AFSQ, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:30:55.762514255 (80h7m31.325837245s) [noise=true , address=104.36.82.14:6245 , piece_id=II36G46OA6TINLR244U6D7Q6LGV754UA5M62K3EX7ZOBESMJUEHA, seg_pos={0 0}, upload=64142]: Done uploading piece. err: ecclient: upload failed (node:1Zu8teNPTXn638grHKwtJqz4UEY7oFYLx1DHvuVP6RZXcatVqX, address:104.36.82.14:6245): protocol: storage node overloaded, request limit: 2000 Jun 22 00:30:55.773669583 (80h7m31.3369926s) [noise=true , address=79.127.205.235:5660 , piece_id=UGFPVS7VNB7U2YSFPSGYDIRZV5R2DHCN57PJP6VOXO52WIAYHLXA, seg_pos={0 0}, upload=64142]: Done uploading piece. err: ecclient: upload failed (node:12E6uCX9KL4Ud7YpxMXepvMXuqTFJCiSXByDjW94Am3U9H2Nf7f, address:79.127.205.235:5660): protocol: storage node overloaded, request limit: 2000 Jun 22 00:30:56.968366655 (80h7m32.53168964s) [noise=true , address=77.90.31.10:4533 , piece_id=SE6YAKKFDL3BV677CDDUC32UNMPXJHTG23GWWYK5B4YIPKER6GAQ, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:30:57.403572874 (80h7m32.966895883s) [noise=true , address=45.91.116.5:5310 , piece_id=AXJOKDZRPVNL7HFT4QKJWRAHFM2UCM3GEVQB2KNYD6CXB6TH4D4Q, seg_pos={0 0}, upload=64142]: Done uploading piece. err: ecclient: upload failed (node:127RMUc62xRq9RU5z3XfkZsq5sehmv3kP8o53tr9BxVcPE3GQCY, address:45.91.116.5:5310): protocol: storage node overloaded, request limit: 2000 Jun 22 00:30:58.210389994 (80h7m33.773712995s) [noise=true , address=104.36.82.13:6199 , piece_id=YP62N2TO446YR3QRG5YLXECD7TN6SJ4UNNLZSK5H4CMV6JIF4AKA, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:30:58.693037961 (80h7m34.256360966s) [noise=true , address=104.36.82.39:6756 , piece_id=OKKWMDJPZO6D5VRMGOE3GOCYGHM725JZTNOVYAWKJW3WYZPI4V5Q, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:30:59.124834814 (80h7m34.688157819s) [noise=true , address=104.36.82.15:6283 , piece_id=N7TH2EMIOPATVHCSNH2WXUGU7KMMEJ5LCHR22SWEHSA5OV4CUCXQ, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:31:00.629657982 (80h7m36.192980977s) [noise=true , address=104.36.82.42:6803 , piece_id=R626UFD326S2UBJEZ2WMHVR7EN4SK427VXPDJZOOZ32MX25JWZWQ, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:31:00.933110546 (80h7m36.496433536s) [noise=true , address=79.127.201.215:4326 , piece_id=USEWZDX6P52U6MCALTBNQG3SV7N2JRKNU6O23TLS6JEWAS4D7ARA, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:31:05.669167243 (80h7m41.232490237s) [noise=true , address=104.36.82.56:4261 , piece_id=EYOYOYLDACGMPGJYX6ILQT3XHMUTQNXZEDC7ANHLZXIKYVDQFFVQ, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:31:08.443582763 (80h7m44.006905763s) [noise=true , address=104.36.82.54:4184 , piece_id=IZICVGZFMAYLIJTNUJ2OFLZVNZ6IPPYDZO4BSTJJPLHVCVOE5EAA, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:31:26.459138014 (80h8m2.022461013s) [noise=true , address=104.36.82.53:4146 , piece_id=3V3DVGUOPJKDWIGQ2TU7M6VFYFVAS6ODY62ERGSU6LI3LSTWQ4YQ, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:31:27.456050336 (80h8m3.019373336s) [noise=true , address=104.36.82.52:4089 , piece_id=X2VG4S3UHVNWA4AQJS7FLKWNVLTOZ3AKWI5ZDH24QCA3ENQW7BXQ, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:31:55.270234251 (80h8m30.833557256s) [noise=true , address=104.36.82.11:6125 , piece_id=2CX7RJOKNQA6ODZZZXYPSCYAQY5P35TWX7I56CMYAPFLJE3JEO5Q, seg_pos={0 0}, upload=64142]: Done uploading piece. err: ecclient: upload failed (node:12vricofwx67FRaxAGLbmgT1TYyKkqfJbxdsyeEPL4R7kg8pKzT, address:104.36.82.11:6125): protocol: expected piece hash; context deadline exceeded; EOF Jun 22 00:31:55.270319418 (80h8m30.833642407s) [noise=true , address=104.36.82.55:4224 , piece_id=XF7H2BYSNNNZOS2T77VIN75N76AYUXAQEI6D4LRXACU2KU62V2TA, seg_pos={0 0}, upload=64142]: Done uploading piece. err: ecclient: upload failed (node:12EjKmUiv4pxkVCB4WoiXim4uFpxrBPyfiSS4efTY3HKSE3jb5y, address:104.36.82.55:4224): protocol: write tcp 159.223.143.39:39152->104.36.82.55:4224: use of closed network connection; write tcp 159.223.143.39:39152->104.36.82.55:4224: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:39152->104.36.82.55:4224: use of closed network connection Jun 22 00:31:55.270353811 (80h8m30.833676799s) [seg_pos={0 0}, upload=64142]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Jun 22 00:31:55.301099877 (80h8m30.864422884s) [upload=64142]: Done uploading segments... Jun 22 00:31:55.301121567 (80h8m30.864444566s) [upload=64142]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 55.514152688s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [.........X...................X..............................] duration: 14.501403367s FAIL (2025-06-22 00:57:11 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 14.501449099s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [...........................X................................] duration: 19.939418112s FAIL (2025-06-22 00:57:06 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== ing upload Jun 22 00:30:55.303646184 (80h7m30.866969171s) [upload=64142]: Uploading segments... Jun 22 00:30:55.303706406 (80h7m30.867029416s) [upload=64142]: Got next segment. Inline: false Jun 22 00:30:55.303762479 (80h7m30.867085468s) [upload=64142]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 22 00:30:55.303768627 (80h7m30.867091636s) [upload=64142]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 22 00:30:55.349058140 (80h7m30.912381168s) [seg_pos={0 0}, upload=64142]: Begin upload segment... Jun 22 00:30:55.349113768 (80h7m30.912436772s) [seg_pos={0 0}, upload=64142]: Done begin upload segment. Jun 22 00:30:55.349143121 (80h7m30.912466131s) [noise=true , address=45.91.116.5:5310 , piece_id=AXJOKDZRPVNL7HFT4QKJWRAHFM2UCM3GEVQB2KNYD6CXB6TH4D4Q, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.349660917 (80h7m30.912983928s) [noise=true , address=77.90.31.10:4533 , piece_id=SE6YAKKFDL3BV677CDDUC32UNMPXJHTG23GWWYK5B4YIPKER6GAQ, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.350093077 (80h7m30.913416089s) [noise=true , address=79.127.201.215:4326 , piece_id=USEWZDX6P52U6MCALTBNQG3SV7N2JRKNU6O23TLS6JEWAS4D7ARA, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.350480894 (80h7m30.913803916s) [noise=true , address=104.36.82.13:6199 , piece_id=YP62N2TO446YR3QRG5YLXECD7TN6SJ4UNNLZSK5H4CMV6JIF4AKA, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.350669998 (80h7m30.913993011s) [noise=true , address=104.36.82.54:4184 , piece_id=IZICVGZFMAYLIJTNUJ2OFLZVNZ6IPPYDZO4BSTJJPLHVCVOE5EAA, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.351201352 (80h7m30.914524366s) [noise=true , address=104.36.82.55:4224 , piece_id=XF7H2BYSNNNZOS2T77VIN75N76AYUXAQEI6D4LRXACU2KU62V2TA, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.351668071 (80h7m30.914991083s) [noise=true , address=104.36.82.56:4261 , piece_id=EYOYOYLDACGMPGJYX6ILQT3XHMUTQNXZEDC7ANHLZXIKYVDQFFVQ, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.352193126 (80h7m30.915516138s) [noise=true , address=104.36.82.52:4089 , piece_id=X2VG4S3UHVNWA4AQJS7FLKWNVLTOZ3AKWI5ZDH24QCA3ENQW7BXQ, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.352587096 (80h7m30.915910103s) [noise=true , address=79.127.220.98:4402 , piece_id=TE2WQO5LK45AVMPIAIGVSIBOVZLC4FG4ESQ72X6KFYC7IAEN2DRA, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.353420253 (80h7m30.916743265s) [noise=true , address=104.36.82.11:6125 , piece_id=2CX7RJOKNQA6ODZZZXYPSCYAQY5P35TWX7I56CMYAPFLJE3JEO5Q, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.353791565 (80h7m30.917114578s) [noise=true , address=104.36.82.42:6803 , piece_id=R626UFD326S2UBJEZ2WMHVR7EN4SK427VXPDJZOOZ32MX25JWZWQ, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.354171226 (80h7m30.917494233s) [noise=true , address=104.36.82.14:6245 , piece_id=II36G46OA6TINLR244U6D7Q6LGV754UA5M62K3EX7ZOBESMJUEHA, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.354542282 (80h7m30.917865288s) [noise=true , address=79.127.205.235:5660 , piece_id=UGFPVS7VNB7U2YSFPSGYDIRZV5R2DHCN57PJP6VOXO52WIAYHLXA, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.354891655 (80h7m30.918214678s) [noise=true , address=104.36.82.15:6283 , piece_id=N7TH2EMIOPATVHCSNH2WXUGU7KMMEJ5LCHR22SWEHSA5OV4CUCXQ, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.355499503 (80h7m30.918822519s) [noise=true , address=77.90.31.14:4666 , piece_id=H4DTFDAVCN266PB235BJIQTQ32FBXTIB5AZSX3B4R2VIK554AFSQ, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.356575547 (80h7m30.919898539s) [noise=true , address=104.36.82.39:6756 , piece_id=OKKWMDJPZO6D5VRMGOE3GOCYGHM725JZTNOVYAWKJW3WYZPI4V5Q, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.357039590 (80h7m30.920362596s) [noise=true , address=104.36.82.53:4146 , piece_id=3V3DVGUOPJKDWIGQ2TU7M6VFYFVAS6ODY62ERGSU6LI3LSTWQ4YQ, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.360343566 (80h7m30.923666582s) [noise=true , address=104.36.82.9:6033 , piece_id=T5XA23LZELMSTWOLBSHOX4AHKDACF65OYCKTMZYTXDIBSXNA2T4A, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.452444064 (80h7m31.015767055s) [upload=64142]: Next returned nil segment Jun 22 00:30:55.452495148 (80h7m31.015818149s) [upload=64142]: Waiting for error group managing segments... Jun 22 00:30:55.532958808 (80h7m31.096281883s) [noise=true , address=104.36.82.9:6033 , piece_id=T5XA23LZELMSTWOLBSHOX4AHKDACF65OYCKTMZYTXDIBSXNA2T4A, seg_pos={0 0}, upload=64142]: Done uploading piece. err: ecclient: upload failed (node:1SV3LrcQW16wLewDzfyUWFB6sUgnERNJVdhz72CJmyde95M79C, address:104.36.82.9:6033): protocol: storage node overloaded, request limit: 2000 Jun 22 00:30:55.547802184 (80h7m31.111125188s) [noise=true , address=79.127.220.98:4402 , piece_id=TE2WQO5LK45AVMPIAIGVSIBOVZLC4FG4ESQ72X6KFYC7IAEN2DRA, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:30:55.554223621 (80h7m31.117546616s) [noise=true , address=77.90.31.14:4666 , piece_id=H4DTFDAVCN266PB235BJIQTQ32FBXTIB5AZSX3B4R2VIK554AFSQ, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:30:55.762514255 (80h7m31.325837245s) [noise=true , address=104.36.82.14:6245 , piece_id=II36G46OA6TINLR244U6D7Q6LGV754UA5M62K3EX7ZOBESMJUEHA, seg_pos={0 0}, upload=64142]: Done uploading piece. err: ecclient: upload failed (node:1Zu8teNPTXn638grHKwtJqz4UEY7oFYLx1DHvuVP6RZXcatVqX, address:104.36.82.14:6245): protocol: storage node overloaded, request limit: 2000 Jun 22 00:30:55.773669583 (80h7m31.3369926s) [noise=true , address=79.127.205.235:5660 , piece_id=UGFPVS7VNB7U2YSFPSGYDIRZV5R2DHCN57PJP6VOXO52WIAYHLXA, seg_pos={0 0}, upload=64142]: Done uploading piece. err: ecclient: upload failed (node:12E6uCX9KL4Ud7YpxMXepvMXuqTFJCiSXByDjW94Am3U9H2Nf7f, address:79.127.205.235:5660): protocol: storage node overloaded, request limit: 2000 Jun 22 00:30:56.968366655 (80h7m32.53168964s) [noise=true , address=77.90.31.10:4533 , piece_id=SE6YAKKFDL3BV677CDDUC32UNMPXJHTG23GWWYK5B4YIPKER6GAQ, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:30:57.403572874 (80h7m32.966895883s) [noise=true , address=45.91.116.5:5310 , piece_id=AXJOKDZRPVNL7HFT4QKJWRAHFM2UCM3GEVQB2KNYD6CXB6TH4D4Q, seg_pos={0 0}, upload=64142]: Done uploading piece. err: ecclient: upload failed (node:127RMUc62xRq9RU5z3XfkZsq5sehmv3kP8o53tr9BxVcPE3GQCY, address:45.91.116.5:5310): protocol: storage node overloaded, request limit: 2000 Jun 22 00:30:58.210389994 (80h7m33.773712995s) [noise=true , address=104.36.82.13:6199 , piece_id=YP62N2TO446YR3QRG5YLXECD7TN6SJ4UNNLZSK5H4CMV6JIF4AKA, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:30:58.693037961 (80h7m34.256360966s) [noise=true , address=104.36.82.39:6756 , piece_id=OKKWMDJPZO6D5VRMGOE3GOCYGHM725JZTNOVYAWKJW3WYZPI4V5Q, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:30:59.124834814 (80h7m34.688157819s) [noise=true , address=104.36.82.15:6283 , piece_id=N7TH2EMIOPATVHCSNH2WXUGU7KMMEJ5LCHR22SWEHSA5OV4CUCXQ, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:31:00.629657982 (80h7m36.192980977s) [noise=true , address=104.36.82.42:6803 , piece_id=R626UFD326S2UBJEZ2WMHVR7EN4SK427VXPDJZOOZ32MX25JWZWQ, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:31:00.933110546 (80h7m36.496433536s) [noise=true , address=79.127.201.215:4326 , piece_id=USEWZDX6P52U6MCALTBNQG3SV7N2JRKNU6O23TLS6JEWAS4D7ARA, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:31:05.669167243 (80h7m41.232490237s) [noise=true , address=104.36.82.56:4261 , piece_id=EYOYOYLDACGMPGJYX6ILQT3XHMUTQNXZEDC7ANHLZXIKYVDQFFVQ, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:31:08.443582763 (80h7m44.006905763s) [noise=true , address=104.36.82.54:4184 , piece_id=IZICVGZFMAYLIJTNUJ2OFLZVNZ6IPPYDZO4BSTJJPLHVCVOE5EAA, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:31:26.459138014 (80h8m2.022461013s) [noise=true , address=104.36.82.53:4146 , piece_id=3V3DVGUOPJKDWIGQ2TU7M6VFYFVAS6ODY62ERGSU6LI3LSTWQ4YQ, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:31:27.456050336 (80h8m3.019373336s) [noise=true , address=104.36.82.52:4089 , piece_id=X2VG4S3UHVNWA4AQJS7FLKWNVLTOZ3AKWI5ZDH24QCA3ENQW7BXQ, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:31:55.270234251 (80h8m30.833557256s) [noise=true , address=104.36.82.11:6125 , piece_id=2CX7RJOKNQA6ODZZZXYPSCYAQY5P35TWX7I56CMYAPFLJE3JEO5Q, seg_pos={0 0}, upload=64142]: Done uploading piece. err: ecclient: upload failed (node:12vricofwx67FRaxAGLbmgT1TYyKkqfJbxdsyeEPL4R7kg8pKzT, address:104.36.82.11:6125): protocol: expected piece hash; context deadline exceeded; EOF Jun 22 00:31:55.270319418 (80h8m30.833642407s) [noise=true , address=104.36.82.55:4224 , piece_id=XF7H2BYSNNNZOS2T77VIN75N76AYUXAQEI6D4LRXACU2KU62V2TA, seg_pos={0 0}, upload=64142]: Done uploading piece. err: ecclient: upload failed (node:12EjKmUiv4pxkVCB4WoiXim4uFpxrBPyfiSS4efTY3HKSE3jb5y, address:104.36.82.55:4224): protocol: write tcp 159.223.143.39:39152->104.36.82.55:4224: use of closed network connection; write tcp 159.223.143.39:39152->104.36.82.55:4224: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:39152->104.36.82.55:4224: use of closed network connection Jun 22 00:31:55.270353811 (80h8m30.833676799s) [seg_pos={0 0}, upload=64142]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Jun 22 00:31:55.301099877 (80h8m30.864422884s) [upload=64142]: Done uploading segments... Jun 22 00:31:55.301121567 (80h8m30.864444566s) [upload=64142]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 19.939469187s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [X.........X...................X.............................] duration: 44.208652841s FAIL (2025-06-22 00:56:29 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 44.208742003s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [............................X...............................] duration: 29.168386893s FAIL (2025-06-22 00:56:24 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== ing upload Jun 22 00:30:55.303646184 (80h7m30.866969171s) [upload=64142]: Uploading segments... Jun 22 00:30:55.303706406 (80h7m30.867029416s) [upload=64142]: Got next segment. Inline: false Jun 22 00:30:55.303762479 (80h7m30.867085468s) [upload=64142]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 22 00:30:55.303768627 (80h7m30.867091636s) [upload=64142]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 22 00:30:55.349058140 (80h7m30.912381168s) [seg_pos={0 0}, upload=64142]: Begin upload segment... Jun 22 00:30:55.349113768 (80h7m30.912436772s) [seg_pos={0 0}, upload=64142]: Done begin upload segment. Jun 22 00:30:55.349143121 (80h7m30.912466131s) [noise=true , address=45.91.116.5:5310 , piece_id=AXJOKDZRPVNL7HFT4QKJWRAHFM2UCM3GEVQB2KNYD6CXB6TH4D4Q, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.349660917 (80h7m30.912983928s) [noise=true , address=77.90.31.10:4533 , piece_id=SE6YAKKFDL3BV677CDDUC32UNMPXJHTG23GWWYK5B4YIPKER6GAQ, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.350093077 (80h7m30.913416089s) [noise=true , address=79.127.201.215:4326 , piece_id=USEWZDX6P52U6MCALTBNQG3SV7N2JRKNU6O23TLS6JEWAS4D7ARA, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.350480894 (80h7m30.913803916s) [noise=true , address=104.36.82.13:6199 , piece_id=YP62N2TO446YR3QRG5YLXECD7TN6SJ4UNNLZSK5H4CMV6JIF4AKA, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.350669998 (80h7m30.913993011s) [noise=true , address=104.36.82.54:4184 , piece_id=IZICVGZFMAYLIJTNUJ2OFLZVNZ6IPPYDZO4BSTJJPLHVCVOE5EAA, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.351201352 (80h7m30.914524366s) [noise=true , address=104.36.82.55:4224 , piece_id=XF7H2BYSNNNZOS2T77VIN75N76AYUXAQEI6D4LRXACU2KU62V2TA, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.351668071 (80h7m30.914991083s) [noise=true , address=104.36.82.56:4261 , piece_id=EYOYOYLDACGMPGJYX6ILQT3XHMUTQNXZEDC7ANHLZXIKYVDQFFVQ, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.352193126 (80h7m30.915516138s) [noise=true , address=104.36.82.52:4089 , piece_id=X2VG4S3UHVNWA4AQJS7FLKWNVLTOZ3AKWI5ZDH24QCA3ENQW7BXQ, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.352587096 (80h7m30.915910103s) [noise=true , address=79.127.220.98:4402 , piece_id=TE2WQO5LK45AVMPIAIGVSIBOVZLC4FG4ESQ72X6KFYC7IAEN2DRA, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.353420253 (80h7m30.916743265s) [noise=true , address=104.36.82.11:6125 , piece_id=2CX7RJOKNQA6ODZZZXYPSCYAQY5P35TWX7I56CMYAPFLJE3JEO5Q, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.353791565 (80h7m30.917114578s) [noise=true , address=104.36.82.42:6803 , piece_id=R626UFD326S2UBJEZ2WMHVR7EN4SK427VXPDJZOOZ32MX25JWZWQ, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.354171226 (80h7m30.917494233s) [noise=true , address=104.36.82.14:6245 , piece_id=II36G46OA6TINLR244U6D7Q6LGV754UA5M62K3EX7ZOBESMJUEHA, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.354542282 (80h7m30.917865288s) [noise=true , address=79.127.205.235:5660 , piece_id=UGFPVS7VNB7U2YSFPSGYDIRZV5R2DHCN57PJP6VOXO52WIAYHLXA, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.354891655 (80h7m30.918214678s) [noise=true , address=104.36.82.15:6283 , piece_id=N7TH2EMIOPATVHCSNH2WXUGU7KMMEJ5LCHR22SWEHSA5OV4CUCXQ, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.355499503 (80h7m30.918822519s) [noise=true , address=77.90.31.14:4666 , piece_id=H4DTFDAVCN266PB235BJIQTQ32FBXTIB5AZSX3B4R2VIK554AFSQ, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.356575547 (80h7m30.919898539s) [noise=true , address=104.36.82.39:6756 , piece_id=OKKWMDJPZO6D5VRMGOE3GOCYGHM725JZTNOVYAWKJW3WYZPI4V5Q, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.357039590 (80h7m30.920362596s) [noise=true , address=104.36.82.53:4146 , piece_id=3V3DVGUOPJKDWIGQ2TU7M6VFYFVAS6ODY62ERGSU6LI3LSTWQ4YQ, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.360343566 (80h7m30.923666582s) [noise=true , address=104.36.82.9:6033 , piece_id=T5XA23LZELMSTWOLBSHOX4AHKDACF65OYCKTMZYTXDIBSXNA2T4A, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.452444064 (80h7m31.015767055s) [upload=64142]: Next returned nil segment Jun 22 00:30:55.452495148 (80h7m31.015818149s) [upload=64142]: Waiting for error group managing segments... Jun 22 00:30:55.532958808 (80h7m31.096281883s) [noise=true , address=104.36.82.9:6033 , piece_id=T5XA23LZELMSTWOLBSHOX4AHKDACF65OYCKTMZYTXDIBSXNA2T4A, seg_pos={0 0}, upload=64142]: Done uploading piece. err: ecclient: upload failed (node:1SV3LrcQW16wLewDzfyUWFB6sUgnERNJVdhz72CJmyde95M79C, address:104.36.82.9:6033): protocol: storage node overloaded, request limit: 2000 Jun 22 00:30:55.547802184 (80h7m31.111125188s) [noise=true , address=79.127.220.98:4402 , piece_id=TE2WQO5LK45AVMPIAIGVSIBOVZLC4FG4ESQ72X6KFYC7IAEN2DRA, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:30:55.554223621 (80h7m31.117546616s) [noise=true , address=77.90.31.14:4666 , piece_id=H4DTFDAVCN266PB235BJIQTQ32FBXTIB5AZSX3B4R2VIK554AFSQ, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:30:55.762514255 (80h7m31.325837245s) [noise=true , address=104.36.82.14:6245 , piece_id=II36G46OA6TINLR244U6D7Q6LGV754UA5M62K3EX7ZOBESMJUEHA, seg_pos={0 0}, upload=64142]: Done uploading piece. err: ecclient: upload failed (node:1Zu8teNPTXn638grHKwtJqz4UEY7oFYLx1DHvuVP6RZXcatVqX, address:104.36.82.14:6245): protocol: storage node overloaded, request limit: 2000 Jun 22 00:30:55.773669583 (80h7m31.3369926s) [noise=true , address=79.127.205.235:5660 , piece_id=UGFPVS7VNB7U2YSFPSGYDIRZV5R2DHCN57PJP6VOXO52WIAYHLXA, seg_pos={0 0}, upload=64142]: Done uploading piece. err: ecclient: upload failed (node:12E6uCX9KL4Ud7YpxMXepvMXuqTFJCiSXByDjW94Am3U9H2Nf7f, address:79.127.205.235:5660): protocol: storage node overloaded, request limit: 2000 Jun 22 00:30:56.968366655 (80h7m32.53168964s) [noise=true , address=77.90.31.10:4533 , piece_id=SE6YAKKFDL3BV677CDDUC32UNMPXJHTG23GWWYK5B4YIPKER6GAQ, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:30:57.403572874 (80h7m32.966895883s) [noise=true , address=45.91.116.5:5310 , piece_id=AXJOKDZRPVNL7HFT4QKJWRAHFM2UCM3GEVQB2KNYD6CXB6TH4D4Q, seg_pos={0 0}, upload=64142]: Done uploading piece. err: ecclient: upload failed (node:127RMUc62xRq9RU5z3XfkZsq5sehmv3kP8o53tr9BxVcPE3GQCY, address:45.91.116.5:5310): protocol: storage node overloaded, request limit: 2000 Jun 22 00:30:58.210389994 (80h7m33.773712995s) [noise=true , address=104.36.82.13:6199 , piece_id=YP62N2TO446YR3QRG5YLXECD7TN6SJ4UNNLZSK5H4CMV6JIF4AKA, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:30:58.693037961 (80h7m34.256360966s) [noise=true , address=104.36.82.39:6756 , piece_id=OKKWMDJPZO6D5VRMGOE3GOCYGHM725JZTNOVYAWKJW3WYZPI4V5Q, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:30:59.124834814 (80h7m34.688157819s) [noise=true , address=104.36.82.15:6283 , piece_id=N7TH2EMIOPATVHCSNH2WXUGU7KMMEJ5LCHR22SWEHSA5OV4CUCXQ, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:31:00.629657982 (80h7m36.192980977s) [noise=true , address=104.36.82.42:6803 , piece_id=R626UFD326S2UBJEZ2WMHVR7EN4SK427VXPDJZOOZ32MX25JWZWQ, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:31:00.933110546 (80h7m36.496433536s) [noise=true , address=79.127.201.215:4326 , piece_id=USEWZDX6P52U6MCALTBNQG3SV7N2JRKNU6O23TLS6JEWAS4D7ARA, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:31:05.669167243 (80h7m41.232490237s) [noise=true , address=104.36.82.56:4261 , piece_id=EYOYOYLDACGMPGJYX6ILQT3XHMUTQNXZEDC7ANHLZXIKYVDQFFVQ, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:31:08.443582763 (80h7m44.006905763s) [noise=true , address=104.36.82.54:4184 , piece_id=IZICVGZFMAYLIJTNUJ2OFLZVNZ6IPPYDZO4BSTJJPLHVCVOE5EAA, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:31:26.459138014 (80h8m2.022461013s) [noise=true , address=104.36.82.53:4146 , piece_id=3V3DVGUOPJKDWIGQ2TU7M6VFYFVAS6ODY62ERGSU6LI3LSTWQ4YQ, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:31:27.456050336 (80h8m3.019373336s) [noise=true , address=104.36.82.52:4089 , piece_id=X2VG4S3UHVNWA4AQJS7FLKWNVLTOZ3AKWI5ZDH24QCA3ENQW7BXQ, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:31:55.270234251 (80h8m30.833557256s) [noise=true , address=104.36.82.11:6125 , piece_id=2CX7RJOKNQA6ODZZZXYPSCYAQY5P35TWX7I56CMYAPFLJE3JEO5Q, seg_pos={0 0}, upload=64142]: Done uploading piece. err: ecclient: upload failed (node:12vricofwx67FRaxAGLbmgT1TYyKkqfJbxdsyeEPL4R7kg8pKzT, address:104.36.82.11:6125): protocol: expected piece hash; context deadline exceeded; EOF Jun 22 00:31:55.270319418 (80h8m30.833642407s) [noise=true , address=104.36.82.55:4224 , piece_id=XF7H2BYSNNNZOS2T77VIN75N76AYUXAQEI6D4LRXACU2KU62V2TA, seg_pos={0 0}, upload=64142]: Done uploading piece. err: ecclient: upload failed (node:12EjKmUiv4pxkVCB4WoiXim4uFpxrBPyfiSS4efTY3HKSE3jb5y, address:104.36.82.55:4224): protocol: write tcp 159.223.143.39:39152->104.36.82.55:4224: use of closed network connection; write tcp 159.223.143.39:39152->104.36.82.55:4224: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:39152->104.36.82.55:4224: use of closed network connection Jun 22 00:31:55.270353811 (80h8m30.833676799s) [seg_pos={0 0}, upload=64142]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Jun 22 00:31:55.301099877 (80h8m30.864422884s) [upload=64142]: Done uploading segments... Jun 22 00:31:55.301121567 (80h8m30.864444566s) [upload=64142]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 29.168540913s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.............................X..............................] duration: 18.603542637s FAIL (2025-06-22 00:55:55 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== ing upload Jun 22 00:30:55.303646184 (80h7m30.866969171s) [upload=64142]: Uploading segments... Jun 22 00:30:55.303706406 (80h7m30.867029416s) [upload=64142]: Got next segment. Inline: false Jun 22 00:30:55.303762479 (80h7m30.867085468s) [upload=64142]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 22 00:30:55.303768627 (80h7m30.867091636s) [upload=64142]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 22 00:30:55.349058140 (80h7m30.912381168s) [seg_pos={0 0}, upload=64142]: Begin upload segment... Jun 22 00:30:55.349113768 (80h7m30.912436772s) [seg_pos={0 0}, upload=64142]: Done begin upload segment. Jun 22 00:30:55.349143121 (80h7m30.912466131s) [noise=true , address=45.91.116.5:5310 , piece_id=AXJOKDZRPVNL7HFT4QKJWRAHFM2UCM3GEVQB2KNYD6CXB6TH4D4Q, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.349660917 (80h7m30.912983928s) [noise=true , address=77.90.31.10:4533 , piece_id=SE6YAKKFDL3BV677CDDUC32UNMPXJHTG23GWWYK5B4YIPKER6GAQ, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.350093077 (80h7m30.913416089s) [noise=true , address=79.127.201.215:4326 , piece_id=USEWZDX6P52U6MCALTBNQG3SV7N2JRKNU6O23TLS6JEWAS4D7ARA, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.350480894 (80h7m30.913803916s) [noise=true , address=104.36.82.13:6199 , piece_id=YP62N2TO446YR3QRG5YLXECD7TN6SJ4UNNLZSK5H4CMV6JIF4AKA, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.350669998 (80h7m30.913993011s) [noise=true , address=104.36.82.54:4184 , piece_id=IZICVGZFMAYLIJTNUJ2OFLZVNZ6IPPYDZO4BSTJJPLHVCVOE5EAA, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.351201352 (80h7m30.914524366s) [noise=true , address=104.36.82.55:4224 , piece_id=XF7H2BYSNNNZOS2T77VIN75N76AYUXAQEI6D4LRXACU2KU62V2TA, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.351668071 (80h7m30.914991083s) [noise=true , address=104.36.82.56:4261 , piece_id=EYOYOYLDACGMPGJYX6ILQT3XHMUTQNXZEDC7ANHLZXIKYVDQFFVQ, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.352193126 (80h7m30.915516138s) [noise=true , address=104.36.82.52:4089 , piece_id=X2VG4S3UHVNWA4AQJS7FLKWNVLTOZ3AKWI5ZDH24QCA3ENQW7BXQ, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.352587096 (80h7m30.915910103s) [noise=true , address=79.127.220.98:4402 , piece_id=TE2WQO5LK45AVMPIAIGVSIBOVZLC4FG4ESQ72X6KFYC7IAEN2DRA, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.353420253 (80h7m30.916743265s) [noise=true , address=104.36.82.11:6125 , piece_id=2CX7RJOKNQA6ODZZZXYPSCYAQY5P35TWX7I56CMYAPFLJE3JEO5Q, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.353791565 (80h7m30.917114578s) [noise=true , address=104.36.82.42:6803 , piece_id=R626UFD326S2UBJEZ2WMHVR7EN4SK427VXPDJZOOZ32MX25JWZWQ, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.354171226 (80h7m30.917494233s) [noise=true , address=104.36.82.14:6245 , piece_id=II36G46OA6TINLR244U6D7Q6LGV754UA5M62K3EX7ZOBESMJUEHA, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.354542282 (80h7m30.917865288s) [noise=true , address=79.127.205.235:5660 , piece_id=UGFPVS7VNB7U2YSFPSGYDIRZV5R2DHCN57PJP6VOXO52WIAYHLXA, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.354891655 (80h7m30.918214678s) [noise=true , address=104.36.82.15:6283 , piece_id=N7TH2EMIOPATVHCSNH2WXUGU7KMMEJ5LCHR22SWEHSA5OV4CUCXQ, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.355499503 (80h7m30.918822519s) [noise=true , address=77.90.31.14:4666 , piece_id=H4DTFDAVCN266PB235BJIQTQ32FBXTIB5AZSX3B4R2VIK554AFSQ, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.356575547 (80h7m30.919898539s) [noise=true , address=104.36.82.39:6756 , piece_id=OKKWMDJPZO6D5VRMGOE3GOCYGHM725JZTNOVYAWKJW3WYZPI4V5Q, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.357039590 (80h7m30.920362596s) [noise=true , address=104.36.82.53:4146 , piece_id=3V3DVGUOPJKDWIGQ2TU7M6VFYFVAS6ODY62ERGSU6LI3LSTWQ4YQ, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.360343566 (80h7m30.923666582s) [noise=true , address=104.36.82.9:6033 , piece_id=T5XA23LZELMSTWOLBSHOX4AHKDACF65OYCKTMZYTXDIBSXNA2T4A, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.452444064 (80h7m31.015767055s) [upload=64142]: Next returned nil segment Jun 22 00:30:55.452495148 (80h7m31.015818149s) [upload=64142]: Waiting for error group managing segments... Jun 22 00:30:55.532958808 (80h7m31.096281883s) [noise=true , address=104.36.82.9:6033 , piece_id=T5XA23LZELMSTWOLBSHOX4AHKDACF65OYCKTMZYTXDIBSXNA2T4A, seg_pos={0 0}, upload=64142]: Done uploading piece. err: ecclient: upload failed (node:1SV3LrcQW16wLewDzfyUWFB6sUgnERNJVdhz72CJmyde95M79C, address:104.36.82.9:6033): protocol: storage node overloaded, request limit: 2000 Jun 22 00:30:55.547802184 (80h7m31.111125188s) [noise=true , address=79.127.220.98:4402 , piece_id=TE2WQO5LK45AVMPIAIGVSIBOVZLC4FG4ESQ72X6KFYC7IAEN2DRA, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:30:55.554223621 (80h7m31.117546616s) [noise=true , address=77.90.31.14:4666 , piece_id=H4DTFDAVCN266PB235BJIQTQ32FBXTIB5AZSX3B4R2VIK554AFSQ, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:30:55.762514255 (80h7m31.325837245s) [noise=true , address=104.36.82.14:6245 , piece_id=II36G46OA6TINLR244U6D7Q6LGV754UA5M62K3EX7ZOBESMJUEHA, seg_pos={0 0}, upload=64142]: Done uploading piece. err: ecclient: upload failed (node:1Zu8teNPTXn638grHKwtJqz4UEY7oFYLx1DHvuVP6RZXcatVqX, address:104.36.82.14:6245): protocol: storage node overloaded, request limit: 2000 Jun 22 00:30:55.773669583 (80h7m31.3369926s) [noise=true , address=79.127.205.235:5660 , piece_id=UGFPVS7VNB7U2YSFPSGYDIRZV5R2DHCN57PJP6VOXO52WIAYHLXA, seg_pos={0 0}, upload=64142]: Done uploading piece. err: ecclient: upload failed (node:12E6uCX9KL4Ud7YpxMXepvMXuqTFJCiSXByDjW94Am3U9H2Nf7f, address:79.127.205.235:5660): protocol: storage node overloaded, request limit: 2000 Jun 22 00:30:56.968366655 (80h7m32.53168964s) [noise=true , address=77.90.31.10:4533 , piece_id=SE6YAKKFDL3BV677CDDUC32UNMPXJHTG23GWWYK5B4YIPKER6GAQ, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:30:57.403572874 (80h7m32.966895883s) [noise=true , address=45.91.116.5:5310 , piece_id=AXJOKDZRPVNL7HFT4QKJWRAHFM2UCM3GEVQB2KNYD6CXB6TH4D4Q, seg_pos={0 0}, upload=64142]: Done uploading piece. err: ecclient: upload failed (node:127RMUc62xRq9RU5z3XfkZsq5sehmv3kP8o53tr9BxVcPE3GQCY, address:45.91.116.5:5310): protocol: storage node overloaded, request limit: 2000 Jun 22 00:30:58.210389994 (80h7m33.773712995s) [noise=true , address=104.36.82.13:6199 , piece_id=YP62N2TO446YR3QRG5YLXECD7TN6SJ4UNNLZSK5H4CMV6JIF4AKA, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:30:58.693037961 (80h7m34.256360966s) [noise=true , address=104.36.82.39:6756 , piece_id=OKKWMDJPZO6D5VRMGOE3GOCYGHM725JZTNOVYAWKJW3WYZPI4V5Q, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:30:59.124834814 (80h7m34.688157819s) [noise=true , address=104.36.82.15:6283 , piece_id=N7TH2EMIOPATVHCSNH2WXUGU7KMMEJ5LCHR22SWEHSA5OV4CUCXQ, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:31:00.629657982 (80h7m36.192980977s) [noise=true , address=104.36.82.42:6803 , piece_id=R626UFD326S2UBJEZ2WMHVR7EN4SK427VXPDJZOOZ32MX25JWZWQ, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:31:00.933110546 (80h7m36.496433536s) [noise=true , address=79.127.201.215:4326 , piece_id=USEWZDX6P52U6MCALTBNQG3SV7N2JRKNU6O23TLS6JEWAS4D7ARA, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:31:05.669167243 (80h7m41.232490237s) [noise=true , address=104.36.82.56:4261 , piece_id=EYOYOYLDACGMPGJYX6ILQT3XHMUTQNXZEDC7ANHLZXIKYVDQFFVQ, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:31:08.443582763 (80h7m44.006905763s) [noise=true , address=104.36.82.54:4184 , piece_id=IZICVGZFMAYLIJTNUJ2OFLZVNZ6IPPYDZO4BSTJJPLHVCVOE5EAA, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:31:26.459138014 (80h8m2.022461013s) [noise=true , address=104.36.82.53:4146 , piece_id=3V3DVGUOPJKDWIGQ2TU7M6VFYFVAS6ODY62ERGSU6LI3LSTWQ4YQ, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:31:27.456050336 (80h8m3.019373336s) [noise=true , address=104.36.82.52:4089 , piece_id=X2VG4S3UHVNWA4AQJS7FLKWNVLTOZ3AKWI5ZDH24QCA3ENQW7BXQ, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:31:55.270234251 (80h8m30.833557256s) [noise=true , address=104.36.82.11:6125 , piece_id=2CX7RJOKNQA6ODZZZXYPSCYAQY5P35TWX7I56CMYAPFLJE3JEO5Q, seg_pos={0 0}, upload=64142]: Done uploading piece. err: ecclient: upload failed (node:12vricofwx67FRaxAGLbmgT1TYyKkqfJbxdsyeEPL4R7kg8pKzT, address:104.36.82.11:6125): protocol: expected piece hash; context deadline exceeded; EOF Jun 22 00:31:55.270319418 (80h8m30.833642407s) [noise=true , address=104.36.82.55:4224 , piece_id=XF7H2BYSNNNZOS2T77VIN75N76AYUXAQEI6D4LRXACU2KU62V2TA, seg_pos={0 0}, upload=64142]: Done uploading piece. err: ecclient: upload failed (node:12EjKmUiv4pxkVCB4WoiXim4uFpxrBPyfiSS4efTY3HKSE3jb5y, address:104.36.82.55:4224): protocol: write tcp 159.223.143.39:39152->104.36.82.55:4224: use of closed network connection; write tcp 159.223.143.39:39152->104.36.82.55:4224: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:39152->104.36.82.55:4224: use of closed network connection Jun 22 00:31:55.270353811 (80h8m30.833676799s) [seg_pos={0 0}, upload=64142]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Jun 22 00:31:55.301099877 (80h8m30.864422884s) [upload=64142]: Done uploading segments... Jun 22 00:31:55.301121567 (80h8m30.864444566s) [upload=64142]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 18.603584695s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [..............................X.............................] duration: 23.710883324s FAIL (2025-06-22 00:55:36 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== ing upload Jun 22 00:30:55.303646184 (80h7m30.866969171s) [upload=64142]: Uploading segments... Jun 22 00:30:55.303706406 (80h7m30.867029416s) [upload=64142]: Got next segment. Inline: false Jun 22 00:30:55.303762479 (80h7m30.867085468s) [upload=64142]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 22 00:30:55.303768627 (80h7m30.867091636s) [upload=64142]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 22 00:30:55.349058140 (80h7m30.912381168s) [seg_pos={0 0}, upload=64142]: Begin upload segment... Jun 22 00:30:55.349113768 (80h7m30.912436772s) [seg_pos={0 0}, upload=64142]: Done begin upload segment. Jun 22 00:30:55.349143121 (80h7m30.912466131s) [noise=true , address=45.91.116.5:5310 , piece_id=AXJOKDZRPVNL7HFT4QKJWRAHFM2UCM3GEVQB2KNYD6CXB6TH4D4Q, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.349660917 (80h7m30.912983928s) [noise=true , address=77.90.31.10:4533 , piece_id=SE6YAKKFDL3BV677CDDUC32UNMPXJHTG23GWWYK5B4YIPKER6GAQ, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.350093077 (80h7m30.913416089s) [noise=true , address=79.127.201.215:4326 , piece_id=USEWZDX6P52U6MCALTBNQG3SV7N2JRKNU6O23TLS6JEWAS4D7ARA, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.350480894 (80h7m30.913803916s) [noise=true , address=104.36.82.13:6199 , piece_id=YP62N2TO446YR3QRG5YLXECD7TN6SJ4UNNLZSK5H4CMV6JIF4AKA, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.350669998 (80h7m30.913993011s) [noise=true , address=104.36.82.54:4184 , piece_id=IZICVGZFMAYLIJTNUJ2OFLZVNZ6IPPYDZO4BSTJJPLHVCVOE5EAA, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.351201352 (80h7m30.914524366s) [noise=true , address=104.36.82.55:4224 , piece_id=XF7H2BYSNNNZOS2T77VIN75N76AYUXAQEI6D4LRXACU2KU62V2TA, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.351668071 (80h7m30.914991083s) [noise=true , address=104.36.82.56:4261 , piece_id=EYOYOYLDACGMPGJYX6ILQT3XHMUTQNXZEDC7ANHLZXIKYVDQFFVQ, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.352193126 (80h7m30.915516138s) [noise=true , address=104.36.82.52:4089 , piece_id=X2VG4S3UHVNWA4AQJS7FLKWNVLTOZ3AKWI5ZDH24QCA3ENQW7BXQ, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.352587096 (80h7m30.915910103s) [noise=true , address=79.127.220.98:4402 , piece_id=TE2WQO5LK45AVMPIAIGVSIBOVZLC4FG4ESQ72X6KFYC7IAEN2DRA, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.353420253 (80h7m30.916743265s) [noise=true , address=104.36.82.11:6125 , piece_id=2CX7RJOKNQA6ODZZZXYPSCYAQY5P35TWX7I56CMYAPFLJE3JEO5Q, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.353791565 (80h7m30.917114578s) [noise=true , address=104.36.82.42:6803 , piece_id=R626UFD326S2UBJEZ2WMHVR7EN4SK427VXPDJZOOZ32MX25JWZWQ, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.354171226 (80h7m30.917494233s) [noise=true , address=104.36.82.14:6245 , piece_id=II36G46OA6TINLR244U6D7Q6LGV754UA5M62K3EX7ZOBESMJUEHA, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.354542282 (80h7m30.917865288s) [noise=true , address=79.127.205.235:5660 , piece_id=UGFPVS7VNB7U2YSFPSGYDIRZV5R2DHCN57PJP6VOXO52WIAYHLXA, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.354891655 (80h7m30.918214678s) [noise=true , address=104.36.82.15:6283 , piece_id=N7TH2EMIOPATVHCSNH2WXUGU7KMMEJ5LCHR22SWEHSA5OV4CUCXQ, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.355499503 (80h7m30.918822519s) [noise=true , address=77.90.31.14:4666 , piece_id=H4DTFDAVCN266PB235BJIQTQ32FBXTIB5AZSX3B4R2VIK554AFSQ, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.356575547 (80h7m30.919898539s) [noise=true , address=104.36.82.39:6756 , piece_id=OKKWMDJPZO6D5VRMGOE3GOCYGHM725JZTNOVYAWKJW3WYZPI4V5Q, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.357039590 (80h7m30.920362596s) [noise=true , address=104.36.82.53:4146 , piece_id=3V3DVGUOPJKDWIGQ2TU7M6VFYFVAS6ODY62ERGSU6LI3LSTWQ4YQ, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.360343566 (80h7m30.923666582s) [noise=true , address=104.36.82.9:6033 , piece_id=T5XA23LZELMSTWOLBSHOX4AHKDACF65OYCKTMZYTXDIBSXNA2T4A, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.452444064 (80h7m31.015767055s) [upload=64142]: Next returned nil segment Jun 22 00:30:55.452495148 (80h7m31.015818149s) [upload=64142]: Waiting for error group managing segments... Jun 22 00:30:55.532958808 (80h7m31.096281883s) [noise=true , address=104.36.82.9:6033 , piece_id=T5XA23LZELMSTWOLBSHOX4AHKDACF65OYCKTMZYTXDIBSXNA2T4A, seg_pos={0 0}, upload=64142]: Done uploading piece. err: ecclient: upload failed (node:1SV3LrcQW16wLewDzfyUWFB6sUgnERNJVdhz72CJmyde95M79C, address:104.36.82.9:6033): protocol: storage node overloaded, request limit: 2000 Jun 22 00:30:55.547802184 (80h7m31.111125188s) [noise=true , address=79.127.220.98:4402 , piece_id=TE2WQO5LK45AVMPIAIGVSIBOVZLC4FG4ESQ72X6KFYC7IAEN2DRA, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:30:55.554223621 (80h7m31.117546616s) [noise=true , address=77.90.31.14:4666 , piece_id=H4DTFDAVCN266PB235BJIQTQ32FBXTIB5AZSX3B4R2VIK554AFSQ, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:30:55.762514255 (80h7m31.325837245s) [noise=true , address=104.36.82.14:6245 , piece_id=II36G46OA6TINLR244U6D7Q6LGV754UA5M62K3EX7ZOBESMJUEHA, seg_pos={0 0}, upload=64142]: Done uploading piece. err: ecclient: upload failed (node:1Zu8teNPTXn638grHKwtJqz4UEY7oFYLx1DHvuVP6RZXcatVqX, address:104.36.82.14:6245): protocol: storage node overloaded, request limit: 2000 Jun 22 00:30:55.773669583 (80h7m31.3369926s) [noise=true , address=79.127.205.235:5660 , piece_id=UGFPVS7VNB7U2YSFPSGYDIRZV5R2DHCN57PJP6VOXO52WIAYHLXA, seg_pos={0 0}, upload=64142]: Done uploading piece. err: ecclient: upload failed (node:12E6uCX9KL4Ud7YpxMXepvMXuqTFJCiSXByDjW94Am3U9H2Nf7f, address:79.127.205.235:5660): protocol: storage node overloaded, request limit: 2000 Jun 22 00:30:56.968366655 (80h7m32.53168964s) [noise=true , address=77.90.31.10:4533 , piece_id=SE6YAKKFDL3BV677CDDUC32UNMPXJHTG23GWWYK5B4YIPKER6GAQ, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:30:57.403572874 (80h7m32.966895883s) [noise=true , address=45.91.116.5:5310 , piece_id=AXJOKDZRPVNL7HFT4QKJWRAHFM2UCM3GEVQB2KNYD6CXB6TH4D4Q, seg_pos={0 0}, upload=64142]: Done uploading piece. err: ecclient: upload failed (node:127RMUc62xRq9RU5z3XfkZsq5sehmv3kP8o53tr9BxVcPE3GQCY, address:45.91.116.5:5310): protocol: storage node overloaded, request limit: 2000 Jun 22 00:30:58.210389994 (80h7m33.773712995s) [noise=true , address=104.36.82.13:6199 , piece_id=YP62N2TO446YR3QRG5YLXECD7TN6SJ4UNNLZSK5H4CMV6JIF4AKA, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:30:58.693037961 (80h7m34.256360966s) [noise=true , address=104.36.82.39:6756 , piece_id=OKKWMDJPZO6D5VRMGOE3GOCYGHM725JZTNOVYAWKJW3WYZPI4V5Q, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:30:59.124834814 (80h7m34.688157819s) [noise=true , address=104.36.82.15:6283 , piece_id=N7TH2EMIOPATVHCSNH2WXUGU7KMMEJ5LCHR22SWEHSA5OV4CUCXQ, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:31:00.629657982 (80h7m36.192980977s) [noise=true , address=104.36.82.42:6803 , piece_id=R626UFD326S2UBJEZ2WMHVR7EN4SK427VXPDJZOOZ32MX25JWZWQ, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:31:00.933110546 (80h7m36.496433536s) [noise=true , address=79.127.201.215:4326 , piece_id=USEWZDX6P52U6MCALTBNQG3SV7N2JRKNU6O23TLS6JEWAS4D7ARA, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:31:05.669167243 (80h7m41.232490237s) [noise=true , address=104.36.82.56:4261 , piece_id=EYOYOYLDACGMPGJYX6ILQT3XHMUTQNXZEDC7ANHLZXIKYVDQFFVQ, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:31:08.443582763 (80h7m44.006905763s) [noise=true , address=104.36.82.54:4184 , piece_id=IZICVGZFMAYLIJTNUJ2OFLZVNZ6IPPYDZO4BSTJJPLHVCVOE5EAA, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:31:26.459138014 (80h8m2.022461013s) [noise=true , address=104.36.82.53:4146 , piece_id=3V3DVGUOPJKDWIGQ2TU7M6VFYFVAS6ODY62ERGSU6LI3LSTWQ4YQ, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:31:27.456050336 (80h8m3.019373336s) [noise=true , address=104.36.82.52:4089 , piece_id=X2VG4S3UHVNWA4AQJS7FLKWNVLTOZ3AKWI5ZDH24QCA3ENQW7BXQ, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:31:55.270234251 (80h8m30.833557256s) [noise=true , address=104.36.82.11:6125 , piece_id=2CX7RJOKNQA6ODZZZXYPSCYAQY5P35TWX7I56CMYAPFLJE3JEO5Q, seg_pos={0 0}, upload=64142]: Done uploading piece. err: ecclient: upload failed (node:12vricofwx67FRaxAGLbmgT1TYyKkqfJbxdsyeEPL4R7kg8pKzT, address:104.36.82.11:6125): protocol: expected piece hash; context deadline exceeded; EOF Jun 22 00:31:55.270319418 (80h8m30.833642407s) [noise=true , address=104.36.82.55:4224 , piece_id=XF7H2BYSNNNZOS2T77VIN75N76AYUXAQEI6D4LRXACU2KU62V2TA, seg_pos={0 0}, upload=64142]: Done uploading piece. err: ecclient: upload failed (node:12EjKmUiv4pxkVCB4WoiXim4uFpxrBPyfiSS4efTY3HKSE3jb5y, address:104.36.82.55:4224): protocol: write tcp 159.223.143.39:39152->104.36.82.55:4224: use of closed network connection; write tcp 159.223.143.39:39152->104.36.82.55:4224: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:39152->104.36.82.55:4224: use of closed network connection Jun 22 00:31:55.270353811 (80h8m30.833676799s) [seg_pos={0 0}, upload=64142]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Jun 22 00:31:55.301099877 (80h8m30.864422884s) [upload=64142]: Done uploading segments... Jun 22 00:31:55.301121567 (80h8m30.864444566s) [upload=64142]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 23.71090663s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [.X.........X...................X............................] duration: 16.369442592s FAIL (2025-06-22 00:55:19 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 16.369484577s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [X..............................X............................] duration: 35.217875761s FAIL (2025-06-22 00:55:12 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== ing upload Jun 22 00:30:55.303646184 (80h7m30.866969171s) [upload=64142]: Uploading segments... Jun 22 00:30:55.303706406 (80h7m30.867029416s) [upload=64142]: Got next segment. Inline: false Jun 22 00:30:55.303762479 (80h7m30.867085468s) [upload=64142]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 22 00:30:55.303768627 (80h7m30.867091636s) [upload=64142]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 22 00:30:55.349058140 (80h7m30.912381168s) [seg_pos={0 0}, upload=64142]: Begin upload segment... Jun 22 00:30:55.349113768 (80h7m30.912436772s) [seg_pos={0 0}, upload=64142]: Done begin upload segment. Jun 22 00:30:55.349143121 (80h7m30.912466131s) [noise=true , address=45.91.116.5:5310 , piece_id=AXJOKDZRPVNL7HFT4QKJWRAHFM2UCM3GEVQB2KNYD6CXB6TH4D4Q, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.349660917 (80h7m30.912983928s) [noise=true , address=77.90.31.10:4533 , piece_id=SE6YAKKFDL3BV677CDDUC32UNMPXJHTG23GWWYK5B4YIPKER6GAQ, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.350093077 (80h7m30.913416089s) [noise=true , address=79.127.201.215:4326 , piece_id=USEWZDX6P52U6MCALTBNQG3SV7N2JRKNU6O23TLS6JEWAS4D7ARA, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.350480894 (80h7m30.913803916s) [noise=true , address=104.36.82.13:6199 , piece_id=YP62N2TO446YR3QRG5YLXECD7TN6SJ4UNNLZSK5H4CMV6JIF4AKA, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.350669998 (80h7m30.913993011s) [noise=true , address=104.36.82.54:4184 , piece_id=IZICVGZFMAYLIJTNUJ2OFLZVNZ6IPPYDZO4BSTJJPLHVCVOE5EAA, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.351201352 (80h7m30.914524366s) [noise=true , address=104.36.82.55:4224 , piece_id=XF7H2BYSNNNZOS2T77VIN75N76AYUXAQEI6D4LRXACU2KU62V2TA, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.351668071 (80h7m30.914991083s) [noise=true , address=104.36.82.56:4261 , piece_id=EYOYOYLDACGMPGJYX6ILQT3XHMUTQNXZEDC7ANHLZXIKYVDQFFVQ, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.352193126 (80h7m30.915516138s) [noise=true , address=104.36.82.52:4089 , piece_id=X2VG4S3UHVNWA4AQJS7FLKWNVLTOZ3AKWI5ZDH24QCA3ENQW7BXQ, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.352587096 (80h7m30.915910103s) [noise=true , address=79.127.220.98:4402 , piece_id=TE2WQO5LK45AVMPIAIGVSIBOVZLC4FG4ESQ72X6KFYC7IAEN2DRA, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.353420253 (80h7m30.916743265s) [noise=true , address=104.36.82.11:6125 , piece_id=2CX7RJOKNQA6ODZZZXYPSCYAQY5P35TWX7I56CMYAPFLJE3JEO5Q, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.353791565 (80h7m30.917114578s) [noise=true , address=104.36.82.42:6803 , piece_id=R626UFD326S2UBJEZ2WMHVR7EN4SK427VXPDJZOOZ32MX25JWZWQ, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.354171226 (80h7m30.917494233s) [noise=true , address=104.36.82.14:6245 , piece_id=II36G46OA6TINLR244U6D7Q6LGV754UA5M62K3EX7ZOBESMJUEHA, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.354542282 (80h7m30.917865288s) [noise=true , address=79.127.205.235:5660 , piece_id=UGFPVS7VNB7U2YSFPSGYDIRZV5R2DHCN57PJP6VOXO52WIAYHLXA, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.354891655 (80h7m30.918214678s) [noise=true , address=104.36.82.15:6283 , piece_id=N7TH2EMIOPATVHCSNH2WXUGU7KMMEJ5LCHR22SWEHSA5OV4CUCXQ, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.355499503 (80h7m30.918822519s) [noise=true , address=77.90.31.14:4666 , piece_id=H4DTFDAVCN266PB235BJIQTQ32FBXTIB5AZSX3B4R2VIK554AFSQ, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.356575547 (80h7m30.919898539s) [noise=true , address=104.36.82.39:6756 , piece_id=OKKWMDJPZO6D5VRMGOE3GOCYGHM725JZTNOVYAWKJW3WYZPI4V5Q, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.357039590 (80h7m30.920362596s) [noise=true , address=104.36.82.53:4146 , piece_id=3V3DVGUOPJKDWIGQ2TU7M6VFYFVAS6ODY62ERGSU6LI3LSTWQ4YQ, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.360343566 (80h7m30.923666582s) [noise=true , address=104.36.82.9:6033 , piece_id=T5XA23LZELMSTWOLBSHOX4AHKDACF65OYCKTMZYTXDIBSXNA2T4A, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.452444064 (80h7m31.015767055s) [upload=64142]: Next returned nil segment Jun 22 00:30:55.452495148 (80h7m31.015818149s) [upload=64142]: Waiting for error group managing segments... Jun 22 00:30:55.532958808 (80h7m31.096281883s) [noise=true , address=104.36.82.9:6033 , piece_id=T5XA23LZELMSTWOLBSHOX4AHKDACF65OYCKTMZYTXDIBSXNA2T4A, seg_pos={0 0}, upload=64142]: Done uploading piece. err: ecclient: upload failed (node:1SV3LrcQW16wLewDzfyUWFB6sUgnERNJVdhz72CJmyde95M79C, address:104.36.82.9:6033): protocol: storage node overloaded, request limit: 2000 Jun 22 00:30:55.547802184 (80h7m31.111125188s) [noise=true , address=79.127.220.98:4402 , piece_id=TE2WQO5LK45AVMPIAIGVSIBOVZLC4FG4ESQ72X6KFYC7IAEN2DRA, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:30:55.554223621 (80h7m31.117546616s) [noise=true , address=77.90.31.14:4666 , piece_id=H4DTFDAVCN266PB235BJIQTQ32FBXTIB5AZSX3B4R2VIK554AFSQ, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:30:55.762514255 (80h7m31.325837245s) [noise=true , address=104.36.82.14:6245 , piece_id=II36G46OA6TINLR244U6D7Q6LGV754UA5M62K3EX7ZOBESMJUEHA, seg_pos={0 0}, upload=64142]: Done uploading piece. err: ecclient: upload failed (node:1Zu8teNPTXn638grHKwtJqz4UEY7oFYLx1DHvuVP6RZXcatVqX, address:104.36.82.14:6245): protocol: storage node overloaded, request limit: 2000 Jun 22 00:30:55.773669583 (80h7m31.3369926s) [noise=true , address=79.127.205.235:5660 , piece_id=UGFPVS7VNB7U2YSFPSGYDIRZV5R2DHCN57PJP6VOXO52WIAYHLXA, seg_pos={0 0}, upload=64142]: Done uploading piece. err: ecclient: upload failed (node:12E6uCX9KL4Ud7YpxMXepvMXuqTFJCiSXByDjW94Am3U9H2Nf7f, address:79.127.205.235:5660): protocol: storage node overloaded, request limit: 2000 Jun 22 00:30:56.968366655 (80h7m32.53168964s) [noise=true , address=77.90.31.10:4533 , piece_id=SE6YAKKFDL3BV677CDDUC32UNMPXJHTG23GWWYK5B4YIPKER6GAQ, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:30:57.403572874 (80h7m32.966895883s) [noise=true , address=45.91.116.5:5310 , piece_id=AXJOKDZRPVNL7HFT4QKJWRAHFM2UCM3GEVQB2KNYD6CXB6TH4D4Q, seg_pos={0 0}, upload=64142]: Done uploading piece. err: ecclient: upload failed (node:127RMUc62xRq9RU5z3XfkZsq5sehmv3kP8o53tr9BxVcPE3GQCY, address:45.91.116.5:5310): protocol: storage node overloaded, request limit: 2000 Jun 22 00:30:58.210389994 (80h7m33.773712995s) [noise=true , address=104.36.82.13:6199 , piece_id=YP62N2TO446YR3QRG5YLXECD7TN6SJ4UNNLZSK5H4CMV6JIF4AKA, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:30:58.693037961 (80h7m34.256360966s) [noise=true , address=104.36.82.39:6756 , piece_id=OKKWMDJPZO6D5VRMGOE3GOCYGHM725JZTNOVYAWKJW3WYZPI4V5Q, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:30:59.124834814 (80h7m34.688157819s) [noise=true , address=104.36.82.15:6283 , piece_id=N7TH2EMIOPATVHCSNH2WXUGU7KMMEJ5LCHR22SWEHSA5OV4CUCXQ, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:31:00.629657982 (80h7m36.192980977s) [noise=true , address=104.36.82.42:6803 , piece_id=R626UFD326S2UBJEZ2WMHVR7EN4SK427VXPDJZOOZ32MX25JWZWQ, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:31:00.933110546 (80h7m36.496433536s) [noise=true , address=79.127.201.215:4326 , piece_id=USEWZDX6P52U6MCALTBNQG3SV7N2JRKNU6O23TLS6JEWAS4D7ARA, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:31:05.669167243 (80h7m41.232490237s) [noise=true , address=104.36.82.56:4261 , piece_id=EYOYOYLDACGMPGJYX6ILQT3XHMUTQNXZEDC7ANHLZXIKYVDQFFVQ, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:31:08.443582763 (80h7m44.006905763s) [noise=true , address=104.36.82.54:4184 , piece_id=IZICVGZFMAYLIJTNUJ2OFLZVNZ6IPPYDZO4BSTJJPLHVCVOE5EAA, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:31:26.459138014 (80h8m2.022461013s) [noise=true , address=104.36.82.53:4146 , piece_id=3V3DVGUOPJKDWIGQ2TU7M6VFYFVAS6ODY62ERGSU6LI3LSTWQ4YQ, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:31:27.456050336 (80h8m3.019373336s) [noise=true , address=104.36.82.52:4089 , piece_id=X2VG4S3UHVNWA4AQJS7FLKWNVLTOZ3AKWI5ZDH24QCA3ENQW7BXQ, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:31:55.270234251 (80h8m30.833557256s) [noise=true , address=104.36.82.11:6125 , piece_id=2CX7RJOKNQA6ODZZZXYPSCYAQY5P35TWX7I56CMYAPFLJE3JEO5Q, seg_pos={0 0}, upload=64142]: Done uploading piece. err: ecclient: upload failed (node:12vricofwx67FRaxAGLbmgT1TYyKkqfJbxdsyeEPL4R7kg8pKzT, address:104.36.82.11:6125): protocol: expected piece hash; context deadline exceeded; EOF Jun 22 00:31:55.270319418 (80h8m30.833642407s) [noise=true , address=104.36.82.55:4224 , piece_id=XF7H2BYSNNNZOS2T77VIN75N76AYUXAQEI6D4LRXACU2KU62V2TA, seg_pos={0 0}, upload=64142]: Done uploading piece. err: ecclient: upload failed (node:12EjKmUiv4pxkVCB4WoiXim4uFpxrBPyfiSS4efTY3HKSE3jb5y, address:104.36.82.55:4224): protocol: write tcp 159.223.143.39:39152->104.36.82.55:4224: use of closed network connection; write tcp 159.223.143.39:39152->104.36.82.55:4224: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:39152->104.36.82.55:4224: use of closed network connection Jun 22 00:31:55.270353811 (80h8m30.833676799s) [seg_pos={0 0}, upload=64142]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Jun 22 00:31:55.301099877 (80h8m30.864422884s) [upload=64142]: Done uploading segments... Jun 22 00:31:55.301121567 (80h8m30.864444566s) [upload=64142]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 35.217908084s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [..X.........X...................X...........................] duration: 12.496495413s FAIL (2025-06-22 00:54:13 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 12.496528968s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [.X..............................X...........................] duration: 28.616751004s FAIL (2025-06-22 00:53:51 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== ing upload Jun 22 00:30:55.303646184 (80h7m30.866969171s) [upload=64142]: Uploading segments... Jun 22 00:30:55.303706406 (80h7m30.867029416s) [upload=64142]: Got next segment. Inline: false Jun 22 00:30:55.303762479 (80h7m30.867085468s) [upload=64142]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 22 00:30:55.303768627 (80h7m30.867091636s) [upload=64142]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 22 00:30:55.349058140 (80h7m30.912381168s) [seg_pos={0 0}, upload=64142]: Begin upload segment... Jun 22 00:30:55.349113768 (80h7m30.912436772s) [seg_pos={0 0}, upload=64142]: Done begin upload segment. Jun 22 00:30:55.349143121 (80h7m30.912466131s) [noise=true , address=45.91.116.5:5310 , piece_id=AXJOKDZRPVNL7HFT4QKJWRAHFM2UCM3GEVQB2KNYD6CXB6TH4D4Q, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.349660917 (80h7m30.912983928s) [noise=true , address=77.90.31.10:4533 , piece_id=SE6YAKKFDL3BV677CDDUC32UNMPXJHTG23GWWYK5B4YIPKER6GAQ, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.350093077 (80h7m30.913416089s) [noise=true , address=79.127.201.215:4326 , piece_id=USEWZDX6P52U6MCALTBNQG3SV7N2JRKNU6O23TLS6JEWAS4D7ARA, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.350480894 (80h7m30.913803916s) [noise=true , address=104.36.82.13:6199 , piece_id=YP62N2TO446YR3QRG5YLXECD7TN6SJ4UNNLZSK5H4CMV6JIF4AKA, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.350669998 (80h7m30.913993011s) [noise=true , address=104.36.82.54:4184 , piece_id=IZICVGZFMAYLIJTNUJ2OFLZVNZ6IPPYDZO4BSTJJPLHVCVOE5EAA, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.351201352 (80h7m30.914524366s) [noise=true , address=104.36.82.55:4224 , piece_id=XF7H2BYSNNNZOS2T77VIN75N76AYUXAQEI6D4LRXACU2KU62V2TA, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.351668071 (80h7m30.914991083s) [noise=true , address=104.36.82.56:4261 , piece_id=EYOYOYLDACGMPGJYX6ILQT3XHMUTQNXZEDC7ANHLZXIKYVDQFFVQ, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.352193126 (80h7m30.915516138s) [noise=true , address=104.36.82.52:4089 , piece_id=X2VG4S3UHVNWA4AQJS7FLKWNVLTOZ3AKWI5ZDH24QCA3ENQW7BXQ, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.352587096 (80h7m30.915910103s) [noise=true , address=79.127.220.98:4402 , piece_id=TE2WQO5LK45AVMPIAIGVSIBOVZLC4FG4ESQ72X6KFYC7IAEN2DRA, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.353420253 (80h7m30.916743265s) [noise=true , address=104.36.82.11:6125 , piece_id=2CX7RJOKNQA6ODZZZXYPSCYAQY5P35TWX7I56CMYAPFLJE3JEO5Q, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.353791565 (80h7m30.917114578s) [noise=true , address=104.36.82.42:6803 , piece_id=R626UFD326S2UBJEZ2WMHVR7EN4SK427VXPDJZOOZ32MX25JWZWQ, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.354171226 (80h7m30.917494233s) [noise=true , address=104.36.82.14:6245 , piece_id=II36G46OA6TINLR244U6D7Q6LGV754UA5M62K3EX7ZOBESMJUEHA, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.354542282 (80h7m30.917865288s) [noise=true , address=79.127.205.235:5660 , piece_id=UGFPVS7VNB7U2YSFPSGYDIRZV5R2DHCN57PJP6VOXO52WIAYHLXA, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.354891655 (80h7m30.918214678s) [noise=true , address=104.36.82.15:6283 , piece_id=N7TH2EMIOPATVHCSNH2WXUGU7KMMEJ5LCHR22SWEHSA5OV4CUCXQ, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.355499503 (80h7m30.918822519s) [noise=true , address=77.90.31.14:4666 , piece_id=H4DTFDAVCN266PB235BJIQTQ32FBXTIB5AZSX3B4R2VIK554AFSQ, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.356575547 (80h7m30.919898539s) [noise=true , address=104.36.82.39:6756 , piece_id=OKKWMDJPZO6D5VRMGOE3GOCYGHM725JZTNOVYAWKJW3WYZPI4V5Q, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.357039590 (80h7m30.920362596s) [noise=true , address=104.36.82.53:4146 , piece_id=3V3DVGUOPJKDWIGQ2TU7M6VFYFVAS6ODY62ERGSU6LI3LSTWQ4YQ, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.360343566 (80h7m30.923666582s) [noise=true , address=104.36.82.9:6033 , piece_id=T5XA23LZELMSTWOLBSHOX4AHKDACF65OYCKTMZYTXDIBSXNA2T4A, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.452444064 (80h7m31.015767055s) [upload=64142]: Next returned nil segment Jun 22 00:30:55.452495148 (80h7m31.015818149s) [upload=64142]: Waiting for error group managing segments... Jun 22 00:30:55.532958808 (80h7m31.096281883s) [noise=true , address=104.36.82.9:6033 , piece_id=T5XA23LZELMSTWOLBSHOX4AHKDACF65OYCKTMZYTXDIBSXNA2T4A, seg_pos={0 0}, upload=64142]: Done uploading piece. err: ecclient: upload failed (node:1SV3LrcQW16wLewDzfyUWFB6sUgnERNJVdhz72CJmyde95M79C, address:104.36.82.9:6033): protocol: storage node overloaded, request limit: 2000 Jun 22 00:30:55.547802184 (80h7m31.111125188s) [noise=true , address=79.127.220.98:4402 , piece_id=TE2WQO5LK45AVMPIAIGVSIBOVZLC4FG4ESQ72X6KFYC7IAEN2DRA, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:30:55.554223621 (80h7m31.117546616s) [noise=true , address=77.90.31.14:4666 , piece_id=H4DTFDAVCN266PB235BJIQTQ32FBXTIB5AZSX3B4R2VIK554AFSQ, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:30:55.762514255 (80h7m31.325837245s) [noise=true , address=104.36.82.14:6245 , piece_id=II36G46OA6TINLR244U6D7Q6LGV754UA5M62K3EX7ZOBESMJUEHA, seg_pos={0 0}, upload=64142]: Done uploading piece. err: ecclient: upload failed (node:1Zu8teNPTXn638grHKwtJqz4UEY7oFYLx1DHvuVP6RZXcatVqX, address:104.36.82.14:6245): protocol: storage node overloaded, request limit: 2000 Jun 22 00:30:55.773669583 (80h7m31.3369926s) [noise=true , address=79.127.205.235:5660 , piece_id=UGFPVS7VNB7U2YSFPSGYDIRZV5R2DHCN57PJP6VOXO52WIAYHLXA, seg_pos={0 0}, upload=64142]: Done uploading piece. err: ecclient: upload failed (node:12E6uCX9KL4Ud7YpxMXepvMXuqTFJCiSXByDjW94Am3U9H2Nf7f, address:79.127.205.235:5660): protocol: storage node overloaded, request limit: 2000 Jun 22 00:30:56.968366655 (80h7m32.53168964s) [noise=true , address=77.90.31.10:4533 , piece_id=SE6YAKKFDL3BV677CDDUC32UNMPXJHTG23GWWYK5B4YIPKER6GAQ, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:30:57.403572874 (80h7m32.966895883s) [noise=true , address=45.91.116.5:5310 , piece_id=AXJOKDZRPVNL7HFT4QKJWRAHFM2UCM3GEVQB2KNYD6CXB6TH4D4Q, seg_pos={0 0}, upload=64142]: Done uploading piece. err: ecclient: upload failed (node:127RMUc62xRq9RU5z3XfkZsq5sehmv3kP8o53tr9BxVcPE3GQCY, address:45.91.116.5:5310): protocol: storage node overloaded, request limit: 2000 Jun 22 00:30:58.210389994 (80h7m33.773712995s) [noise=true , address=104.36.82.13:6199 , piece_id=YP62N2TO446YR3QRG5YLXECD7TN6SJ4UNNLZSK5H4CMV6JIF4AKA, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:30:58.693037961 (80h7m34.256360966s) [noise=true , address=104.36.82.39:6756 , piece_id=OKKWMDJPZO6D5VRMGOE3GOCYGHM725JZTNOVYAWKJW3WYZPI4V5Q, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:30:59.124834814 (80h7m34.688157819s) [noise=true , address=104.36.82.15:6283 , piece_id=N7TH2EMIOPATVHCSNH2WXUGU7KMMEJ5LCHR22SWEHSA5OV4CUCXQ, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:31:00.629657982 (80h7m36.192980977s) [noise=true , address=104.36.82.42:6803 , piece_id=R626UFD326S2UBJEZ2WMHVR7EN4SK427VXPDJZOOZ32MX25JWZWQ, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:31:00.933110546 (80h7m36.496433536s) [noise=true , address=79.127.201.215:4326 , piece_id=USEWZDX6P52U6MCALTBNQG3SV7N2JRKNU6O23TLS6JEWAS4D7ARA, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:31:05.669167243 (80h7m41.232490237s) [noise=true , address=104.36.82.56:4261 , piece_id=EYOYOYLDACGMPGJYX6ILQT3XHMUTQNXZEDC7ANHLZXIKYVDQFFVQ, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:31:08.443582763 (80h7m44.006905763s) [noise=true , address=104.36.82.54:4184 , piece_id=IZICVGZFMAYLIJTNUJ2OFLZVNZ6IPPYDZO4BSTJJPLHVCVOE5EAA, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:31:26.459138014 (80h8m2.022461013s) [noise=true , address=104.36.82.53:4146 , piece_id=3V3DVGUOPJKDWIGQ2TU7M6VFYFVAS6ODY62ERGSU6LI3LSTWQ4YQ, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:31:27.456050336 (80h8m3.019373336s) [noise=true , address=104.36.82.52:4089 , piece_id=X2VG4S3UHVNWA4AQJS7FLKWNVLTOZ3AKWI5ZDH24QCA3ENQW7BXQ, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:31:55.270234251 (80h8m30.833557256s) [noise=true , address=104.36.82.11:6125 , piece_id=2CX7RJOKNQA6ODZZZXYPSCYAQY5P35TWX7I56CMYAPFLJE3JEO5Q, seg_pos={0 0}, upload=64142]: Done uploading piece. err: ecclient: upload failed (node:12vricofwx67FRaxAGLbmgT1TYyKkqfJbxdsyeEPL4R7kg8pKzT, address:104.36.82.11:6125): protocol: expected piece hash; context deadline exceeded; EOF Jun 22 00:31:55.270319418 (80h8m30.833642407s) [noise=true , address=104.36.82.55:4224 , piece_id=XF7H2BYSNNNZOS2T77VIN75N76AYUXAQEI6D4LRXACU2KU62V2TA, seg_pos={0 0}, upload=64142]: Done uploading piece. err: ecclient: upload failed (node:12EjKmUiv4pxkVCB4WoiXim4uFpxrBPyfiSS4efTY3HKSE3jb5y, address:104.36.82.55:4224): protocol: write tcp 159.223.143.39:39152->104.36.82.55:4224: use of closed network connection; write tcp 159.223.143.39:39152->104.36.82.55:4224: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:39152->104.36.82.55:4224: use of closed network connection Jun 22 00:31:55.270353811 (80h8m30.833676799s) [seg_pos={0 0}, upload=64142]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Jun 22 00:31:55.301099877 (80h8m30.864422884s) [upload=64142]: Done uploading segments... Jun 22 00:31:55.301121567 (80h8m30.864444566s) [upload=64142]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 28.616786041s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [...X.........X...................X..........................] duration: 16.762915244s FAIL (2025-06-22 00:53:34 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 16.762933793s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [X.X..............................X..........................] duration: 10.192584334s FAIL (2025-06-22 00:52:25 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== ing upload Jun 22 00:30:55.303646184 (80h7m30.866969171s) [upload=64142]: Uploading segments... Jun 22 00:30:55.303706406 (80h7m30.867029416s) [upload=64142]: Got next segment. Inline: false Jun 22 00:30:55.303762479 (80h7m30.867085468s) [upload=64142]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 22 00:30:55.303768627 (80h7m30.867091636s) [upload=64142]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 22 00:30:55.349058140 (80h7m30.912381168s) [seg_pos={0 0}, upload=64142]: Begin upload segment... Jun 22 00:30:55.349113768 (80h7m30.912436772s) [seg_pos={0 0}, upload=64142]: Done begin upload segment. Jun 22 00:30:55.349143121 (80h7m30.912466131s) [noise=true , address=45.91.116.5:5310 , piece_id=AXJOKDZRPVNL7HFT4QKJWRAHFM2UCM3GEVQB2KNYD6CXB6TH4D4Q, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.349660917 (80h7m30.912983928s) [noise=true , address=77.90.31.10:4533 , piece_id=SE6YAKKFDL3BV677CDDUC32UNMPXJHTG23GWWYK5B4YIPKER6GAQ, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.350093077 (80h7m30.913416089s) [noise=true , address=79.127.201.215:4326 , piece_id=USEWZDX6P52U6MCALTBNQG3SV7N2JRKNU6O23TLS6JEWAS4D7ARA, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.350480894 (80h7m30.913803916s) [noise=true , address=104.36.82.13:6199 , piece_id=YP62N2TO446YR3QRG5YLXECD7TN6SJ4UNNLZSK5H4CMV6JIF4AKA, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.350669998 (80h7m30.913993011s) [noise=true , address=104.36.82.54:4184 , piece_id=IZICVGZFMAYLIJTNUJ2OFLZVNZ6IPPYDZO4BSTJJPLHVCVOE5EAA, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.351201352 (80h7m30.914524366s) [noise=true , address=104.36.82.55:4224 , piece_id=XF7H2BYSNNNZOS2T77VIN75N76AYUXAQEI6D4LRXACU2KU62V2TA, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.351668071 (80h7m30.914991083s) [noise=true , address=104.36.82.56:4261 , piece_id=EYOYOYLDACGMPGJYX6ILQT3XHMUTQNXZEDC7ANHLZXIKYVDQFFVQ, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.352193126 (80h7m30.915516138s) [noise=true , address=104.36.82.52:4089 , piece_id=X2VG4S3UHVNWA4AQJS7FLKWNVLTOZ3AKWI5ZDH24QCA3ENQW7BXQ, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.352587096 (80h7m30.915910103s) [noise=true , address=79.127.220.98:4402 , piece_id=TE2WQO5LK45AVMPIAIGVSIBOVZLC4FG4ESQ72X6KFYC7IAEN2DRA, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.353420253 (80h7m30.916743265s) [noise=true , address=104.36.82.11:6125 , piece_id=2CX7RJOKNQA6ODZZZXYPSCYAQY5P35TWX7I56CMYAPFLJE3JEO5Q, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.353791565 (80h7m30.917114578s) [noise=true , address=104.36.82.42:6803 , piece_id=R626UFD326S2UBJEZ2WMHVR7EN4SK427VXPDJZOOZ32MX25JWZWQ, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.354171226 (80h7m30.917494233s) [noise=true , address=104.36.82.14:6245 , piece_id=II36G46OA6TINLR244U6D7Q6LGV754UA5M62K3EX7ZOBESMJUEHA, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.354542282 (80h7m30.917865288s) [noise=true , address=79.127.205.235:5660 , piece_id=UGFPVS7VNB7U2YSFPSGYDIRZV5R2DHCN57PJP6VOXO52WIAYHLXA, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.354891655 (80h7m30.918214678s) [noise=true , address=104.36.82.15:6283 , piece_id=N7TH2EMIOPATVHCSNH2WXUGU7KMMEJ5LCHR22SWEHSA5OV4CUCXQ, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.355499503 (80h7m30.918822519s) [noise=true , address=77.90.31.14:4666 , piece_id=H4DTFDAVCN266PB235BJIQTQ32FBXTIB5AZSX3B4R2VIK554AFSQ, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.356575547 (80h7m30.919898539s) [noise=true , address=104.36.82.39:6756 , piece_id=OKKWMDJPZO6D5VRMGOE3GOCYGHM725JZTNOVYAWKJW3WYZPI4V5Q, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.357039590 (80h7m30.920362596s) [noise=true , address=104.36.82.53:4146 , piece_id=3V3DVGUOPJKDWIGQ2TU7M6VFYFVAS6ODY62ERGSU6LI3LSTWQ4YQ, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.360343566 (80h7m30.923666582s) [noise=true , address=104.36.82.9:6033 , piece_id=T5XA23LZELMSTWOLBSHOX4AHKDACF65OYCKTMZYTXDIBSXNA2T4A, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.452444064 (80h7m31.015767055s) [upload=64142]: Next returned nil segment Jun 22 00:30:55.452495148 (80h7m31.015818149s) [upload=64142]: Waiting for error group managing segments... Jun 22 00:30:55.532958808 (80h7m31.096281883s) [noise=true , address=104.36.82.9:6033 , piece_id=T5XA23LZELMSTWOLBSHOX4AHKDACF65OYCKTMZYTXDIBSXNA2T4A, seg_pos={0 0}, upload=64142]: Done uploading piece. err: ecclient: upload failed (node:1SV3LrcQW16wLewDzfyUWFB6sUgnERNJVdhz72CJmyde95M79C, address:104.36.82.9:6033): protocol: storage node overloaded, request limit: 2000 Jun 22 00:30:55.547802184 (80h7m31.111125188s) [noise=true , address=79.127.220.98:4402 , piece_id=TE2WQO5LK45AVMPIAIGVSIBOVZLC4FG4ESQ72X6KFYC7IAEN2DRA, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:30:55.554223621 (80h7m31.117546616s) [noise=true , address=77.90.31.14:4666 , piece_id=H4DTFDAVCN266PB235BJIQTQ32FBXTIB5AZSX3B4R2VIK554AFSQ, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:30:55.762514255 (80h7m31.325837245s) [noise=true , address=104.36.82.14:6245 , piece_id=II36G46OA6TINLR244U6D7Q6LGV754UA5M62K3EX7ZOBESMJUEHA, seg_pos={0 0}, upload=64142]: Done uploading piece. err: ecclient: upload failed (node:1Zu8teNPTXn638grHKwtJqz4UEY7oFYLx1DHvuVP6RZXcatVqX, address:104.36.82.14:6245): protocol: storage node overloaded, request limit: 2000 Jun 22 00:30:55.773669583 (80h7m31.3369926s) [noise=true , address=79.127.205.235:5660 , piece_id=UGFPVS7VNB7U2YSFPSGYDIRZV5R2DHCN57PJP6VOXO52WIAYHLXA, seg_pos={0 0}, upload=64142]: Done uploading piece. err: ecclient: upload failed (node:12E6uCX9KL4Ud7YpxMXepvMXuqTFJCiSXByDjW94Am3U9H2Nf7f, address:79.127.205.235:5660): protocol: storage node overloaded, request limit: 2000 Jun 22 00:30:56.968366655 (80h7m32.53168964s) [noise=true , address=77.90.31.10:4533 , piece_id=SE6YAKKFDL3BV677CDDUC32UNMPXJHTG23GWWYK5B4YIPKER6GAQ, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:30:57.403572874 (80h7m32.966895883s) [noise=true , address=45.91.116.5:5310 , piece_id=AXJOKDZRPVNL7HFT4QKJWRAHFM2UCM3GEVQB2KNYD6CXB6TH4D4Q, seg_pos={0 0}, upload=64142]: Done uploading piece. err: ecclient: upload failed (node:127RMUc62xRq9RU5z3XfkZsq5sehmv3kP8o53tr9BxVcPE3GQCY, address:45.91.116.5:5310): protocol: storage node overloaded, request limit: 2000 Jun 22 00:30:58.210389994 (80h7m33.773712995s) [noise=true , address=104.36.82.13:6199 , piece_id=YP62N2TO446YR3QRG5YLXECD7TN6SJ4UNNLZSK5H4CMV6JIF4AKA, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:30:58.693037961 (80h7m34.256360966s) [noise=true , address=104.36.82.39:6756 , piece_id=OKKWMDJPZO6D5VRMGOE3GOCYGHM725JZTNOVYAWKJW3WYZPI4V5Q, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:30:59.124834814 (80h7m34.688157819s) [noise=true , address=104.36.82.15:6283 , piece_id=N7TH2EMIOPATVHCSNH2WXUGU7KMMEJ5LCHR22SWEHSA5OV4CUCXQ, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:31:00.629657982 (80h7m36.192980977s) [noise=true , address=104.36.82.42:6803 , piece_id=R626UFD326S2UBJEZ2WMHVR7EN4SK427VXPDJZOOZ32MX25JWZWQ, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:31:00.933110546 (80h7m36.496433536s) [noise=true , address=79.127.201.215:4326 , piece_id=USEWZDX6P52U6MCALTBNQG3SV7N2JRKNU6O23TLS6JEWAS4D7ARA, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:31:05.669167243 (80h7m41.232490237s) [noise=true , address=104.36.82.56:4261 , piece_id=EYOYOYLDACGMPGJYX6ILQT3XHMUTQNXZEDC7ANHLZXIKYVDQFFVQ, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:31:08.443582763 (80h7m44.006905763s) [noise=true , address=104.36.82.54:4184 , piece_id=IZICVGZFMAYLIJTNUJ2OFLZVNZ6IPPYDZO4BSTJJPLHVCVOE5EAA, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:31:26.459138014 (80h8m2.022461013s) [noise=true , address=104.36.82.53:4146 , piece_id=3V3DVGUOPJKDWIGQ2TU7M6VFYFVAS6ODY62ERGSU6LI3LSTWQ4YQ, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:31:27.456050336 (80h8m3.019373336s) [noise=true , address=104.36.82.52:4089 , piece_id=X2VG4S3UHVNWA4AQJS7FLKWNVLTOZ3AKWI5ZDH24QCA3ENQW7BXQ, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:31:55.270234251 (80h8m30.833557256s) [noise=true , address=104.36.82.11:6125 , piece_id=2CX7RJOKNQA6ODZZZXYPSCYAQY5P35TWX7I56CMYAPFLJE3JEO5Q, seg_pos={0 0}, upload=64142]: Done uploading piece. err: ecclient: upload failed (node:12vricofwx67FRaxAGLbmgT1TYyKkqfJbxdsyeEPL4R7kg8pKzT, address:104.36.82.11:6125): protocol: expected piece hash; context deadline exceeded; EOF Jun 22 00:31:55.270319418 (80h8m30.833642407s) [noise=true , address=104.36.82.55:4224 , piece_id=XF7H2BYSNNNZOS2T77VIN75N76AYUXAQEI6D4LRXACU2KU62V2TA, seg_pos={0 0}, upload=64142]: Done uploading piece. err: ecclient: upload failed (node:12EjKmUiv4pxkVCB4WoiXim4uFpxrBPyfiSS4efTY3HKSE3jb5y, address:104.36.82.55:4224): protocol: write tcp 159.223.143.39:39152->104.36.82.55:4224: use of closed network connection; write tcp 159.223.143.39:39152->104.36.82.55:4224: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:39152->104.36.82.55:4224: use of closed network connection Jun 22 00:31:55.270353811 (80h8m30.833676799s) [seg_pos={0 0}, upload=64142]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Jun 22 00:31:55.301099877 (80h8m30.864422884s) [upload=64142]: Done uploading segments... Jun 22 00:31:55.301121567 (80h8m30.864444566s) [upload=64142]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 10.192613385s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [....X.........X...................X.........................] duration: 14.654475377s FAIL (2025-06-22 00:52:19 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 14.654505953s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [.....X.........X...................X........................] duration: 56.059876711s FAIL (2025-06-22 00:51:55 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 56.05991261s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [.X.X..............................X.........................] duration: 18.872217666s FAIL (2025-06-22 00:51:20 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== ing upload Jun 22 00:30:55.303646184 (80h7m30.866969171s) [upload=64142]: Uploading segments... Jun 22 00:30:55.303706406 (80h7m30.867029416s) [upload=64142]: Got next segment. Inline: false Jun 22 00:30:55.303762479 (80h7m30.867085468s) [upload=64142]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 22 00:30:55.303768627 (80h7m30.867091636s) [upload=64142]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 22 00:30:55.349058140 (80h7m30.912381168s) [seg_pos={0 0}, upload=64142]: Begin upload segment... Jun 22 00:30:55.349113768 (80h7m30.912436772s) [seg_pos={0 0}, upload=64142]: Done begin upload segment. Jun 22 00:30:55.349143121 (80h7m30.912466131s) [noise=true , address=45.91.116.5:5310 , piece_id=AXJOKDZRPVNL7HFT4QKJWRAHFM2UCM3GEVQB2KNYD6CXB6TH4D4Q, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.349660917 (80h7m30.912983928s) [noise=true , address=77.90.31.10:4533 , piece_id=SE6YAKKFDL3BV677CDDUC32UNMPXJHTG23GWWYK5B4YIPKER6GAQ, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.350093077 (80h7m30.913416089s) [noise=true , address=79.127.201.215:4326 , piece_id=USEWZDX6P52U6MCALTBNQG3SV7N2JRKNU6O23TLS6JEWAS4D7ARA, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.350480894 (80h7m30.913803916s) [noise=true , address=104.36.82.13:6199 , piece_id=YP62N2TO446YR3QRG5YLXECD7TN6SJ4UNNLZSK5H4CMV6JIF4AKA, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.350669998 (80h7m30.913993011s) [noise=true , address=104.36.82.54:4184 , piece_id=IZICVGZFMAYLIJTNUJ2OFLZVNZ6IPPYDZO4BSTJJPLHVCVOE5EAA, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.351201352 (80h7m30.914524366s) [noise=true , address=104.36.82.55:4224 , piece_id=XF7H2BYSNNNZOS2T77VIN75N76AYUXAQEI6D4LRXACU2KU62V2TA, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.351668071 (80h7m30.914991083s) [noise=true , address=104.36.82.56:4261 , piece_id=EYOYOYLDACGMPGJYX6ILQT3XHMUTQNXZEDC7ANHLZXIKYVDQFFVQ, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.352193126 (80h7m30.915516138s) [noise=true , address=104.36.82.52:4089 , piece_id=X2VG4S3UHVNWA4AQJS7FLKWNVLTOZ3AKWI5ZDH24QCA3ENQW7BXQ, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.352587096 (80h7m30.915910103s) [noise=true , address=79.127.220.98:4402 , piece_id=TE2WQO5LK45AVMPIAIGVSIBOVZLC4FG4ESQ72X6KFYC7IAEN2DRA, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.353420253 (80h7m30.916743265s) [noise=true , address=104.36.82.11:6125 , piece_id=2CX7RJOKNQA6ODZZZXYPSCYAQY5P35TWX7I56CMYAPFLJE3JEO5Q, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.353791565 (80h7m30.917114578s) [noise=true , address=104.36.82.42:6803 , piece_id=R626UFD326S2UBJEZ2WMHVR7EN4SK427VXPDJZOOZ32MX25JWZWQ, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.354171226 (80h7m30.917494233s) [noise=true , address=104.36.82.14:6245 , piece_id=II36G46OA6TINLR244U6D7Q6LGV754UA5M62K3EX7ZOBESMJUEHA, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.354542282 (80h7m30.917865288s) [noise=true , address=79.127.205.235:5660 , piece_id=UGFPVS7VNB7U2YSFPSGYDIRZV5R2DHCN57PJP6VOXO52WIAYHLXA, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.354891655 (80h7m30.918214678s) [noise=true , address=104.36.82.15:6283 , piece_id=N7TH2EMIOPATVHCSNH2WXUGU7KMMEJ5LCHR22SWEHSA5OV4CUCXQ, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.355499503 (80h7m30.918822519s) [noise=true , address=77.90.31.14:4666 , piece_id=H4DTFDAVCN266PB235BJIQTQ32FBXTIB5AZSX3B4R2VIK554AFSQ, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.356575547 (80h7m30.919898539s) [noise=true , address=104.36.82.39:6756 , piece_id=OKKWMDJPZO6D5VRMGOE3GOCYGHM725JZTNOVYAWKJW3WYZPI4V5Q, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.357039590 (80h7m30.920362596s) [noise=true , address=104.36.82.53:4146 , piece_id=3V3DVGUOPJKDWIGQ2TU7M6VFYFVAS6ODY62ERGSU6LI3LSTWQ4YQ, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.360343566 (80h7m30.923666582s) [noise=true , address=104.36.82.9:6033 , piece_id=T5XA23LZELMSTWOLBSHOX4AHKDACF65OYCKTMZYTXDIBSXNA2T4A, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.452444064 (80h7m31.015767055s) [upload=64142]: Next returned nil segment Jun 22 00:30:55.452495148 (80h7m31.015818149s) [upload=64142]: Waiting for error group managing segments... Jun 22 00:30:55.532958808 (80h7m31.096281883s) [noise=true , address=104.36.82.9:6033 , piece_id=T5XA23LZELMSTWOLBSHOX4AHKDACF65OYCKTMZYTXDIBSXNA2T4A, seg_pos={0 0}, upload=64142]: Done uploading piece. err: ecclient: upload failed (node:1SV3LrcQW16wLewDzfyUWFB6sUgnERNJVdhz72CJmyde95M79C, address:104.36.82.9:6033): protocol: storage node overloaded, request limit: 2000 Jun 22 00:30:55.547802184 (80h7m31.111125188s) [noise=true , address=79.127.220.98:4402 , piece_id=TE2WQO5LK45AVMPIAIGVSIBOVZLC4FG4ESQ72X6KFYC7IAEN2DRA, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:30:55.554223621 (80h7m31.117546616s) [noise=true , address=77.90.31.14:4666 , piece_id=H4DTFDAVCN266PB235BJIQTQ32FBXTIB5AZSX3B4R2VIK554AFSQ, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:30:55.762514255 (80h7m31.325837245s) [noise=true , address=104.36.82.14:6245 , piece_id=II36G46OA6TINLR244U6D7Q6LGV754UA5M62K3EX7ZOBESMJUEHA, seg_pos={0 0}, upload=64142]: Done uploading piece. err: ecclient: upload failed (node:1Zu8teNPTXn638grHKwtJqz4UEY7oFYLx1DHvuVP6RZXcatVqX, address:104.36.82.14:6245): protocol: storage node overloaded, request limit: 2000 Jun 22 00:30:55.773669583 (80h7m31.3369926s) [noise=true , address=79.127.205.235:5660 , piece_id=UGFPVS7VNB7U2YSFPSGYDIRZV5R2DHCN57PJP6VOXO52WIAYHLXA, seg_pos={0 0}, upload=64142]: Done uploading piece. err: ecclient: upload failed (node:12E6uCX9KL4Ud7YpxMXepvMXuqTFJCiSXByDjW94Am3U9H2Nf7f, address:79.127.205.235:5660): protocol: storage node overloaded, request limit: 2000 Jun 22 00:30:56.968366655 (80h7m32.53168964s) [noise=true , address=77.90.31.10:4533 , piece_id=SE6YAKKFDL3BV677CDDUC32UNMPXJHTG23GWWYK5B4YIPKER6GAQ, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:30:57.403572874 (80h7m32.966895883s) [noise=true , address=45.91.116.5:5310 , piece_id=AXJOKDZRPVNL7HFT4QKJWRAHFM2UCM3GEVQB2KNYD6CXB6TH4D4Q, seg_pos={0 0}, upload=64142]: Done uploading piece. err: ecclient: upload failed (node:127RMUc62xRq9RU5z3XfkZsq5sehmv3kP8o53tr9BxVcPE3GQCY, address:45.91.116.5:5310): protocol: storage node overloaded, request limit: 2000 Jun 22 00:30:58.210389994 (80h7m33.773712995s) [noise=true , address=104.36.82.13:6199 , piece_id=YP62N2TO446YR3QRG5YLXECD7TN6SJ4UNNLZSK5H4CMV6JIF4AKA, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:30:58.693037961 (80h7m34.256360966s) [noise=true , address=104.36.82.39:6756 , piece_id=OKKWMDJPZO6D5VRMGOE3GOCYGHM725JZTNOVYAWKJW3WYZPI4V5Q, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:30:59.124834814 (80h7m34.688157819s) [noise=true , address=104.36.82.15:6283 , piece_id=N7TH2EMIOPATVHCSNH2WXUGU7KMMEJ5LCHR22SWEHSA5OV4CUCXQ, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:31:00.629657982 (80h7m36.192980977s) [noise=true , address=104.36.82.42:6803 , piece_id=R626UFD326S2UBJEZ2WMHVR7EN4SK427VXPDJZOOZ32MX25JWZWQ, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:31:00.933110546 (80h7m36.496433536s) [noise=true , address=79.127.201.215:4326 , piece_id=USEWZDX6P52U6MCALTBNQG3SV7N2JRKNU6O23TLS6JEWAS4D7ARA, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:31:05.669167243 (80h7m41.232490237s) [noise=true , address=104.36.82.56:4261 , piece_id=EYOYOYLDACGMPGJYX6ILQT3XHMUTQNXZEDC7ANHLZXIKYVDQFFVQ, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:31:08.443582763 (80h7m44.006905763s) [noise=true , address=104.36.82.54:4184 , piece_id=IZICVGZFMAYLIJTNUJ2OFLZVNZ6IPPYDZO4BSTJJPLHVCVOE5EAA, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:31:26.459138014 (80h8m2.022461013s) [noise=true , address=104.36.82.53:4146 , piece_id=3V3DVGUOPJKDWIGQ2TU7M6VFYFVAS6ODY62ERGSU6LI3LSTWQ4YQ, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:31:27.456050336 (80h8m3.019373336s) [noise=true , address=104.36.82.52:4089 , piece_id=X2VG4S3UHVNWA4AQJS7FLKWNVLTOZ3AKWI5ZDH24QCA3ENQW7BXQ, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:31:55.270234251 (80h8m30.833557256s) [noise=true , address=104.36.82.11:6125 , piece_id=2CX7RJOKNQA6ODZZZXYPSCYAQY5P35TWX7I56CMYAPFLJE3JEO5Q, seg_pos={0 0}, upload=64142]: Done uploading piece. err: ecclient: upload failed (node:12vricofwx67FRaxAGLbmgT1TYyKkqfJbxdsyeEPL4R7kg8pKzT, address:104.36.82.11:6125): protocol: expected piece hash; context deadline exceeded; EOF Jun 22 00:31:55.270319418 (80h8m30.833642407s) [noise=true , address=104.36.82.55:4224 , piece_id=XF7H2BYSNNNZOS2T77VIN75N76AYUXAQEI6D4LRXACU2KU62V2TA, seg_pos={0 0}, upload=64142]: Done uploading piece. err: ecclient: upload failed (node:12EjKmUiv4pxkVCB4WoiXim4uFpxrBPyfiSS4efTY3HKSE3jb5y, address:104.36.82.55:4224): protocol: write tcp 159.223.143.39:39152->104.36.82.55:4224: use of closed network connection; write tcp 159.223.143.39:39152->104.36.82.55:4224: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:39152->104.36.82.55:4224: use of closed network connection Jun 22 00:31:55.270353811 (80h8m30.833676799s) [seg_pos={0 0}, upload=64142]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Jun 22 00:31:55.301099877 (80h8m30.864422884s) [upload=64142]: Done uploading segments... Jun 22 00:31:55.301121567 (80h8m30.864444566s) [upload=64142]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 18.872246395s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [..X.X..............................X........................] duration: 28.376883527s FAIL (2025-06-22 00:50:47 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== ing upload Jun 22 00:30:55.303646184 (80h7m30.866969171s) [upload=64142]: Uploading segments... Jun 22 00:30:55.303706406 (80h7m30.867029416s) [upload=64142]: Got next segment. Inline: false Jun 22 00:30:55.303762479 (80h7m30.867085468s) [upload=64142]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 22 00:30:55.303768627 (80h7m30.867091636s) [upload=64142]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 22 00:30:55.349058140 (80h7m30.912381168s) [seg_pos={0 0}, upload=64142]: Begin upload segment... Jun 22 00:30:55.349113768 (80h7m30.912436772s) [seg_pos={0 0}, upload=64142]: Done begin upload segment. Jun 22 00:30:55.349143121 (80h7m30.912466131s) [noise=true , address=45.91.116.5:5310 , piece_id=AXJOKDZRPVNL7HFT4QKJWRAHFM2UCM3GEVQB2KNYD6CXB6TH4D4Q, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.349660917 (80h7m30.912983928s) [noise=true , address=77.90.31.10:4533 , piece_id=SE6YAKKFDL3BV677CDDUC32UNMPXJHTG23GWWYK5B4YIPKER6GAQ, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.350093077 (80h7m30.913416089s) [noise=true , address=79.127.201.215:4326 , piece_id=USEWZDX6P52U6MCALTBNQG3SV7N2JRKNU6O23TLS6JEWAS4D7ARA, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.350480894 (80h7m30.913803916s) [noise=true , address=104.36.82.13:6199 , piece_id=YP62N2TO446YR3QRG5YLXECD7TN6SJ4UNNLZSK5H4CMV6JIF4AKA, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.350669998 (80h7m30.913993011s) [noise=true , address=104.36.82.54:4184 , piece_id=IZICVGZFMAYLIJTNUJ2OFLZVNZ6IPPYDZO4BSTJJPLHVCVOE5EAA, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.351201352 (80h7m30.914524366s) [noise=true , address=104.36.82.55:4224 , piece_id=XF7H2BYSNNNZOS2T77VIN75N76AYUXAQEI6D4LRXACU2KU62V2TA, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.351668071 (80h7m30.914991083s) [noise=true , address=104.36.82.56:4261 , piece_id=EYOYOYLDACGMPGJYX6ILQT3XHMUTQNXZEDC7ANHLZXIKYVDQFFVQ, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.352193126 (80h7m30.915516138s) [noise=true , address=104.36.82.52:4089 , piece_id=X2VG4S3UHVNWA4AQJS7FLKWNVLTOZ3AKWI5ZDH24QCA3ENQW7BXQ, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.352587096 (80h7m30.915910103s) [noise=true , address=79.127.220.98:4402 , piece_id=TE2WQO5LK45AVMPIAIGVSIBOVZLC4FG4ESQ72X6KFYC7IAEN2DRA, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.353420253 (80h7m30.916743265s) [noise=true , address=104.36.82.11:6125 , piece_id=2CX7RJOKNQA6ODZZZXYPSCYAQY5P35TWX7I56CMYAPFLJE3JEO5Q, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.353791565 (80h7m30.917114578s) [noise=true , address=104.36.82.42:6803 , piece_id=R626UFD326S2UBJEZ2WMHVR7EN4SK427VXPDJZOOZ32MX25JWZWQ, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.354171226 (80h7m30.917494233s) [noise=true , address=104.36.82.14:6245 , piece_id=II36G46OA6TINLR244U6D7Q6LGV754UA5M62K3EX7ZOBESMJUEHA, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.354542282 (80h7m30.917865288s) [noise=true , address=79.127.205.235:5660 , piece_id=UGFPVS7VNB7U2YSFPSGYDIRZV5R2DHCN57PJP6VOXO52WIAYHLXA, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.354891655 (80h7m30.918214678s) [noise=true , address=104.36.82.15:6283 , piece_id=N7TH2EMIOPATVHCSNH2WXUGU7KMMEJ5LCHR22SWEHSA5OV4CUCXQ, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.355499503 (80h7m30.918822519s) [noise=true , address=77.90.31.14:4666 , piece_id=H4DTFDAVCN266PB235BJIQTQ32FBXTIB5AZSX3B4R2VIK554AFSQ, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.356575547 (80h7m30.919898539s) [noise=true , address=104.36.82.39:6756 , piece_id=OKKWMDJPZO6D5VRMGOE3GOCYGHM725JZTNOVYAWKJW3WYZPI4V5Q, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.357039590 (80h7m30.920362596s) [noise=true , address=104.36.82.53:4146 , piece_id=3V3DVGUOPJKDWIGQ2TU7M6VFYFVAS6ODY62ERGSU6LI3LSTWQ4YQ, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.360343566 (80h7m30.923666582s) [noise=true , address=104.36.82.9:6033 , piece_id=T5XA23LZELMSTWOLBSHOX4AHKDACF65OYCKTMZYTXDIBSXNA2T4A, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.452444064 (80h7m31.015767055s) [upload=64142]: Next returned nil segment Jun 22 00:30:55.452495148 (80h7m31.015818149s) [upload=64142]: Waiting for error group managing segments... Jun 22 00:30:55.532958808 (80h7m31.096281883s) [noise=true , address=104.36.82.9:6033 , piece_id=T5XA23LZELMSTWOLBSHOX4AHKDACF65OYCKTMZYTXDIBSXNA2T4A, seg_pos={0 0}, upload=64142]: Done uploading piece. err: ecclient: upload failed (node:1SV3LrcQW16wLewDzfyUWFB6sUgnERNJVdhz72CJmyde95M79C, address:104.36.82.9:6033): protocol: storage node overloaded, request limit: 2000 Jun 22 00:30:55.547802184 (80h7m31.111125188s) [noise=true , address=79.127.220.98:4402 , piece_id=TE2WQO5LK45AVMPIAIGVSIBOVZLC4FG4ESQ72X6KFYC7IAEN2DRA, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:30:55.554223621 (80h7m31.117546616s) [noise=true , address=77.90.31.14:4666 , piece_id=H4DTFDAVCN266PB235BJIQTQ32FBXTIB5AZSX3B4R2VIK554AFSQ, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:30:55.762514255 (80h7m31.325837245s) [noise=true , address=104.36.82.14:6245 , piece_id=II36G46OA6TINLR244U6D7Q6LGV754UA5M62K3EX7ZOBESMJUEHA, seg_pos={0 0}, upload=64142]: Done uploading piece. err: ecclient: upload failed (node:1Zu8teNPTXn638grHKwtJqz4UEY7oFYLx1DHvuVP6RZXcatVqX, address:104.36.82.14:6245): protocol: storage node overloaded, request limit: 2000 Jun 22 00:30:55.773669583 (80h7m31.3369926s) [noise=true , address=79.127.205.235:5660 , piece_id=UGFPVS7VNB7U2YSFPSGYDIRZV5R2DHCN57PJP6VOXO52WIAYHLXA, seg_pos={0 0}, upload=64142]: Done uploading piece. err: ecclient: upload failed (node:12E6uCX9KL4Ud7YpxMXepvMXuqTFJCiSXByDjW94Am3U9H2Nf7f, address:79.127.205.235:5660): protocol: storage node overloaded, request limit: 2000 Jun 22 00:30:56.968366655 (80h7m32.53168964s) [noise=true , address=77.90.31.10:4533 , piece_id=SE6YAKKFDL3BV677CDDUC32UNMPXJHTG23GWWYK5B4YIPKER6GAQ, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:30:57.403572874 (80h7m32.966895883s) [noise=true , address=45.91.116.5:5310 , piece_id=AXJOKDZRPVNL7HFT4QKJWRAHFM2UCM3GEVQB2KNYD6CXB6TH4D4Q, seg_pos={0 0}, upload=64142]: Done uploading piece. err: ecclient: upload failed (node:127RMUc62xRq9RU5z3XfkZsq5sehmv3kP8o53tr9BxVcPE3GQCY, address:45.91.116.5:5310): protocol: storage node overloaded, request limit: 2000 Jun 22 00:30:58.210389994 (80h7m33.773712995s) [noise=true , address=104.36.82.13:6199 , piece_id=YP62N2TO446YR3QRG5YLXECD7TN6SJ4UNNLZSK5H4CMV6JIF4AKA, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:30:58.693037961 (80h7m34.256360966s) [noise=true , address=104.36.82.39:6756 , piece_id=OKKWMDJPZO6D5VRMGOE3GOCYGHM725JZTNOVYAWKJW3WYZPI4V5Q, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:30:59.124834814 (80h7m34.688157819s) [noise=true , address=104.36.82.15:6283 , piece_id=N7TH2EMIOPATVHCSNH2WXUGU7KMMEJ5LCHR22SWEHSA5OV4CUCXQ, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:31:00.629657982 (80h7m36.192980977s) [noise=true , address=104.36.82.42:6803 , piece_id=R626UFD326S2UBJEZ2WMHVR7EN4SK427VXPDJZOOZ32MX25JWZWQ, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:31:00.933110546 (80h7m36.496433536s) [noise=true , address=79.127.201.215:4326 , piece_id=USEWZDX6P52U6MCALTBNQG3SV7N2JRKNU6O23TLS6JEWAS4D7ARA, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:31:05.669167243 (80h7m41.232490237s) [noise=true , address=104.36.82.56:4261 , piece_id=EYOYOYLDACGMPGJYX6ILQT3XHMUTQNXZEDC7ANHLZXIKYVDQFFVQ, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:31:08.443582763 (80h7m44.006905763s) [noise=true , address=104.36.82.54:4184 , piece_id=IZICVGZFMAYLIJTNUJ2OFLZVNZ6IPPYDZO4BSTJJPLHVCVOE5EAA, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:31:26.459138014 (80h8m2.022461013s) [noise=true , address=104.36.82.53:4146 , piece_id=3V3DVGUOPJKDWIGQ2TU7M6VFYFVAS6ODY62ERGSU6LI3LSTWQ4YQ, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:31:27.456050336 (80h8m3.019373336s) [noise=true , address=104.36.82.52:4089 , piece_id=X2VG4S3UHVNWA4AQJS7FLKWNVLTOZ3AKWI5ZDH24QCA3ENQW7BXQ, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:31:55.270234251 (80h8m30.833557256s) [noise=true , address=104.36.82.11:6125 , piece_id=2CX7RJOKNQA6ODZZZXYPSCYAQY5P35TWX7I56CMYAPFLJE3JEO5Q, seg_pos={0 0}, upload=64142]: Done uploading piece. err: ecclient: upload failed (node:12vricofwx67FRaxAGLbmgT1TYyKkqfJbxdsyeEPL4R7kg8pKzT, address:104.36.82.11:6125): protocol: expected piece hash; context deadline exceeded; EOF Jun 22 00:31:55.270319418 (80h8m30.833642407s) [noise=true , address=104.36.82.55:4224 , piece_id=XF7H2BYSNNNZOS2T77VIN75N76AYUXAQEI6D4LRXACU2KU62V2TA, seg_pos={0 0}, upload=64142]: Done uploading piece. err: ecclient: upload failed (node:12EjKmUiv4pxkVCB4WoiXim4uFpxrBPyfiSS4efTY3HKSE3jb5y, address:104.36.82.55:4224): protocol: write tcp 159.223.143.39:39152->104.36.82.55:4224: use of closed network connection; write tcp 159.223.143.39:39152->104.36.82.55:4224: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:39152->104.36.82.55:4224: use of closed network connection Jun 22 00:31:55.270353811 (80h8m30.833676799s) [seg_pos={0 0}, upload=64142]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Jun 22 00:31:55.301099877 (80h8m30.864422884s) [upload=64142]: Done uploading segments... Jun 22 00:31:55.301121567 (80h8m30.864444566s) [upload=64142]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 28.376913473s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [......X.........X...................X.......................] duration: 14.991395087s FAIL (2025-06-22 00:50:01 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 14.991417919s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [...X.X..............................X.......................] duration: 20.367589506s FAIL (2025-06-22 00:49:35 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== ing upload Jun 22 00:30:55.303646184 (80h7m30.866969171s) [upload=64142]: Uploading segments... Jun 22 00:30:55.303706406 (80h7m30.867029416s) [upload=64142]: Got next segment. Inline: false Jun 22 00:30:55.303762479 (80h7m30.867085468s) [upload=64142]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 22 00:30:55.303768627 (80h7m30.867091636s) [upload=64142]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 22 00:30:55.349058140 (80h7m30.912381168s) [seg_pos={0 0}, upload=64142]: Begin upload segment... Jun 22 00:30:55.349113768 (80h7m30.912436772s) [seg_pos={0 0}, upload=64142]: Done begin upload segment. Jun 22 00:30:55.349143121 (80h7m30.912466131s) [noise=true , address=45.91.116.5:5310 , piece_id=AXJOKDZRPVNL7HFT4QKJWRAHFM2UCM3GEVQB2KNYD6CXB6TH4D4Q, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.349660917 (80h7m30.912983928s) [noise=true , address=77.90.31.10:4533 , piece_id=SE6YAKKFDL3BV677CDDUC32UNMPXJHTG23GWWYK5B4YIPKER6GAQ, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.350093077 (80h7m30.913416089s) [noise=true , address=79.127.201.215:4326 , piece_id=USEWZDX6P52U6MCALTBNQG3SV7N2JRKNU6O23TLS6JEWAS4D7ARA, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.350480894 (80h7m30.913803916s) [noise=true , address=104.36.82.13:6199 , piece_id=YP62N2TO446YR3QRG5YLXECD7TN6SJ4UNNLZSK5H4CMV6JIF4AKA, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.350669998 (80h7m30.913993011s) [noise=true , address=104.36.82.54:4184 , piece_id=IZICVGZFMAYLIJTNUJ2OFLZVNZ6IPPYDZO4BSTJJPLHVCVOE5EAA, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.351201352 (80h7m30.914524366s) [noise=true , address=104.36.82.55:4224 , piece_id=XF7H2BYSNNNZOS2T77VIN75N76AYUXAQEI6D4LRXACU2KU62V2TA, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.351668071 (80h7m30.914991083s) [noise=true , address=104.36.82.56:4261 , piece_id=EYOYOYLDACGMPGJYX6ILQT3XHMUTQNXZEDC7ANHLZXIKYVDQFFVQ, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.352193126 (80h7m30.915516138s) [noise=true , address=104.36.82.52:4089 , piece_id=X2VG4S3UHVNWA4AQJS7FLKWNVLTOZ3AKWI5ZDH24QCA3ENQW7BXQ, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.352587096 (80h7m30.915910103s) [noise=true , address=79.127.220.98:4402 , piece_id=TE2WQO5LK45AVMPIAIGVSIBOVZLC4FG4ESQ72X6KFYC7IAEN2DRA, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.353420253 (80h7m30.916743265s) [noise=true , address=104.36.82.11:6125 , piece_id=2CX7RJOKNQA6ODZZZXYPSCYAQY5P35TWX7I56CMYAPFLJE3JEO5Q, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.353791565 (80h7m30.917114578s) [noise=true , address=104.36.82.42:6803 , piece_id=R626UFD326S2UBJEZ2WMHVR7EN4SK427VXPDJZOOZ32MX25JWZWQ, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.354171226 (80h7m30.917494233s) [noise=true , address=104.36.82.14:6245 , piece_id=II36G46OA6TINLR244U6D7Q6LGV754UA5M62K3EX7ZOBESMJUEHA, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.354542282 (80h7m30.917865288s) [noise=true , address=79.127.205.235:5660 , piece_id=UGFPVS7VNB7U2YSFPSGYDIRZV5R2DHCN57PJP6VOXO52WIAYHLXA, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.354891655 (80h7m30.918214678s) [noise=true , address=104.36.82.15:6283 , piece_id=N7TH2EMIOPATVHCSNH2WXUGU7KMMEJ5LCHR22SWEHSA5OV4CUCXQ, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.355499503 (80h7m30.918822519s) [noise=true , address=77.90.31.14:4666 , piece_id=H4DTFDAVCN266PB235BJIQTQ32FBXTIB5AZSX3B4R2VIK554AFSQ, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.356575547 (80h7m30.919898539s) [noise=true , address=104.36.82.39:6756 , piece_id=OKKWMDJPZO6D5VRMGOE3GOCYGHM725JZTNOVYAWKJW3WYZPI4V5Q, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.357039590 (80h7m30.920362596s) [noise=true , address=104.36.82.53:4146 , piece_id=3V3DVGUOPJKDWIGQ2TU7M6VFYFVAS6ODY62ERGSU6LI3LSTWQ4YQ, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.360343566 (80h7m30.923666582s) [noise=true , address=104.36.82.9:6033 , piece_id=T5XA23LZELMSTWOLBSHOX4AHKDACF65OYCKTMZYTXDIBSXNA2T4A, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.452444064 (80h7m31.015767055s) [upload=64142]: Next returned nil segment Jun 22 00:30:55.452495148 (80h7m31.015818149s) [upload=64142]: Waiting for error group managing segments... Jun 22 00:30:55.532958808 (80h7m31.096281883s) [noise=true , address=104.36.82.9:6033 , piece_id=T5XA23LZELMSTWOLBSHOX4AHKDACF65OYCKTMZYTXDIBSXNA2T4A, seg_pos={0 0}, upload=64142]: Done uploading piece. err: ecclient: upload failed (node:1SV3LrcQW16wLewDzfyUWFB6sUgnERNJVdhz72CJmyde95M79C, address:104.36.82.9:6033): protocol: storage node overloaded, request limit: 2000 Jun 22 00:30:55.547802184 (80h7m31.111125188s) [noise=true , address=79.127.220.98:4402 , piece_id=TE2WQO5LK45AVMPIAIGVSIBOVZLC4FG4ESQ72X6KFYC7IAEN2DRA, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:30:55.554223621 (80h7m31.117546616s) [noise=true , address=77.90.31.14:4666 , piece_id=H4DTFDAVCN266PB235BJIQTQ32FBXTIB5AZSX3B4R2VIK554AFSQ, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:30:55.762514255 (80h7m31.325837245s) [noise=true , address=104.36.82.14:6245 , piece_id=II36G46OA6TINLR244U6D7Q6LGV754UA5M62K3EX7ZOBESMJUEHA, seg_pos={0 0}, upload=64142]: Done uploading piece. err: ecclient: upload failed (node:1Zu8teNPTXn638grHKwtJqz4UEY7oFYLx1DHvuVP6RZXcatVqX, address:104.36.82.14:6245): protocol: storage node overloaded, request limit: 2000 Jun 22 00:30:55.773669583 (80h7m31.3369926s) [noise=true , address=79.127.205.235:5660 , piece_id=UGFPVS7VNB7U2YSFPSGYDIRZV5R2DHCN57PJP6VOXO52WIAYHLXA, seg_pos={0 0}, upload=64142]: Done uploading piece. err: ecclient: upload failed (node:12E6uCX9KL4Ud7YpxMXepvMXuqTFJCiSXByDjW94Am3U9H2Nf7f, address:79.127.205.235:5660): protocol: storage node overloaded, request limit: 2000 Jun 22 00:30:56.968366655 (80h7m32.53168964s) [noise=true , address=77.90.31.10:4533 , piece_id=SE6YAKKFDL3BV677CDDUC32UNMPXJHTG23GWWYK5B4YIPKER6GAQ, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:30:57.403572874 (80h7m32.966895883s) [noise=true , address=45.91.116.5:5310 , piece_id=AXJOKDZRPVNL7HFT4QKJWRAHFM2UCM3GEVQB2KNYD6CXB6TH4D4Q, seg_pos={0 0}, upload=64142]: Done uploading piece. err: ecclient: upload failed (node:127RMUc62xRq9RU5z3XfkZsq5sehmv3kP8o53tr9BxVcPE3GQCY, address:45.91.116.5:5310): protocol: storage node overloaded, request limit: 2000 Jun 22 00:30:58.210389994 (80h7m33.773712995s) [noise=true , address=104.36.82.13:6199 , piece_id=YP62N2TO446YR3QRG5YLXECD7TN6SJ4UNNLZSK5H4CMV6JIF4AKA, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:30:58.693037961 (80h7m34.256360966s) [noise=true , address=104.36.82.39:6756 , piece_id=OKKWMDJPZO6D5VRMGOE3GOCYGHM725JZTNOVYAWKJW3WYZPI4V5Q, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:30:59.124834814 (80h7m34.688157819s) [noise=true , address=104.36.82.15:6283 , piece_id=N7TH2EMIOPATVHCSNH2WXUGU7KMMEJ5LCHR22SWEHSA5OV4CUCXQ, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:31:00.629657982 (80h7m36.192980977s) [noise=true , address=104.36.82.42:6803 , piece_id=R626UFD326S2UBJEZ2WMHVR7EN4SK427VXPDJZOOZ32MX25JWZWQ, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:31:00.933110546 (80h7m36.496433536s) [noise=true , address=79.127.201.215:4326 , piece_id=USEWZDX6P52U6MCALTBNQG3SV7N2JRKNU6O23TLS6JEWAS4D7ARA, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:31:05.669167243 (80h7m41.232490237s) [noise=true , address=104.36.82.56:4261 , piece_id=EYOYOYLDACGMPGJYX6ILQT3XHMUTQNXZEDC7ANHLZXIKYVDQFFVQ, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:31:08.443582763 (80h7m44.006905763s) [noise=true , address=104.36.82.54:4184 , piece_id=IZICVGZFMAYLIJTNUJ2OFLZVNZ6IPPYDZO4BSTJJPLHVCVOE5EAA, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:31:26.459138014 (80h8m2.022461013s) [noise=true , address=104.36.82.53:4146 , piece_id=3V3DVGUOPJKDWIGQ2TU7M6VFYFVAS6ODY62ERGSU6LI3LSTWQ4YQ, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:31:27.456050336 (80h8m3.019373336s) [noise=true , address=104.36.82.52:4089 , piece_id=X2VG4S3UHVNWA4AQJS7FLKWNVLTOZ3AKWI5ZDH24QCA3ENQW7BXQ, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:31:55.270234251 (80h8m30.833557256s) [noise=true , address=104.36.82.11:6125 , piece_id=2CX7RJOKNQA6ODZZZXYPSCYAQY5P35TWX7I56CMYAPFLJE3JEO5Q, seg_pos={0 0}, upload=64142]: Done uploading piece. err: ecclient: upload failed (node:12vricofwx67FRaxAGLbmgT1TYyKkqfJbxdsyeEPL4R7kg8pKzT, address:104.36.82.11:6125): protocol: expected piece hash; context deadline exceeded; EOF Jun 22 00:31:55.270319418 (80h8m30.833642407s) [noise=true , address=104.36.82.55:4224 , piece_id=XF7H2BYSNNNZOS2T77VIN75N76AYUXAQEI6D4LRXACU2KU62V2TA, seg_pos={0 0}, upload=64142]: Done uploading piece. err: ecclient: upload failed (node:12EjKmUiv4pxkVCB4WoiXim4uFpxrBPyfiSS4efTY3HKSE3jb5y, address:104.36.82.55:4224): protocol: write tcp 159.223.143.39:39152->104.36.82.55:4224: use of closed network connection; write tcp 159.223.143.39:39152->104.36.82.55:4224: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:39152->104.36.82.55:4224: use of closed network connection Jun 22 00:31:55.270353811 (80h8m30.833676799s) [seg_pos={0 0}, upload=64142]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Jun 22 00:31:55.301099877 (80h8m30.864422884s) [upload=64142]: Done uploading segments... Jun 22 00:31:55.301121567 (80h8m30.864444566s) [upload=64142]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 20.367618367s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [.......X.........X...................X......................] duration: 18.811395684s FAIL (2025-06-22 00:49:17 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 18.811413319s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [....X.X..............................X......................] duration: 23.147145032s FAIL (2025-06-22 00:49:12 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== ing upload Jun 22 00:30:55.303646184 (80h7m30.866969171s) [upload=64142]: Uploading segments... Jun 22 00:30:55.303706406 (80h7m30.867029416s) [upload=64142]: Got next segment. Inline: false Jun 22 00:30:55.303762479 (80h7m30.867085468s) [upload=64142]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 22 00:30:55.303768627 (80h7m30.867091636s) [upload=64142]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 22 00:30:55.349058140 (80h7m30.912381168s) [seg_pos={0 0}, upload=64142]: Begin upload segment... Jun 22 00:30:55.349113768 (80h7m30.912436772s) [seg_pos={0 0}, upload=64142]: Done begin upload segment. Jun 22 00:30:55.349143121 (80h7m30.912466131s) [noise=true , address=45.91.116.5:5310 , piece_id=AXJOKDZRPVNL7HFT4QKJWRAHFM2UCM3GEVQB2KNYD6CXB6TH4D4Q, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.349660917 (80h7m30.912983928s) [noise=true , address=77.90.31.10:4533 , piece_id=SE6YAKKFDL3BV677CDDUC32UNMPXJHTG23GWWYK5B4YIPKER6GAQ, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.350093077 (80h7m30.913416089s) [noise=true , address=79.127.201.215:4326 , piece_id=USEWZDX6P52U6MCALTBNQG3SV7N2JRKNU6O23TLS6JEWAS4D7ARA, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.350480894 (80h7m30.913803916s) [noise=true , address=104.36.82.13:6199 , piece_id=YP62N2TO446YR3QRG5YLXECD7TN6SJ4UNNLZSK5H4CMV6JIF4AKA, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.350669998 (80h7m30.913993011s) [noise=true , address=104.36.82.54:4184 , piece_id=IZICVGZFMAYLIJTNUJ2OFLZVNZ6IPPYDZO4BSTJJPLHVCVOE5EAA, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.351201352 (80h7m30.914524366s) [noise=true , address=104.36.82.55:4224 , piece_id=XF7H2BYSNNNZOS2T77VIN75N76AYUXAQEI6D4LRXACU2KU62V2TA, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.351668071 (80h7m30.914991083s) [noise=true , address=104.36.82.56:4261 , piece_id=EYOYOYLDACGMPGJYX6ILQT3XHMUTQNXZEDC7ANHLZXIKYVDQFFVQ, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.352193126 (80h7m30.915516138s) [noise=true , address=104.36.82.52:4089 , piece_id=X2VG4S3UHVNWA4AQJS7FLKWNVLTOZ3AKWI5ZDH24QCA3ENQW7BXQ, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.352587096 (80h7m30.915910103s) [noise=true , address=79.127.220.98:4402 , piece_id=TE2WQO5LK45AVMPIAIGVSIBOVZLC4FG4ESQ72X6KFYC7IAEN2DRA, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.353420253 (80h7m30.916743265s) [noise=true , address=104.36.82.11:6125 , piece_id=2CX7RJOKNQA6ODZZZXYPSCYAQY5P35TWX7I56CMYAPFLJE3JEO5Q, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.353791565 (80h7m30.917114578s) [noise=true , address=104.36.82.42:6803 , piece_id=R626UFD326S2UBJEZ2WMHVR7EN4SK427VXPDJZOOZ32MX25JWZWQ, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.354171226 (80h7m30.917494233s) [noise=true , address=104.36.82.14:6245 , piece_id=II36G46OA6TINLR244U6D7Q6LGV754UA5M62K3EX7ZOBESMJUEHA, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.354542282 (80h7m30.917865288s) [noise=true , address=79.127.205.235:5660 , piece_id=UGFPVS7VNB7U2YSFPSGYDIRZV5R2DHCN57PJP6VOXO52WIAYHLXA, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.354891655 (80h7m30.918214678s) [noise=true , address=104.36.82.15:6283 , piece_id=N7TH2EMIOPATVHCSNH2WXUGU7KMMEJ5LCHR22SWEHSA5OV4CUCXQ, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.355499503 (80h7m30.918822519s) [noise=true , address=77.90.31.14:4666 , piece_id=H4DTFDAVCN266PB235BJIQTQ32FBXTIB5AZSX3B4R2VIK554AFSQ, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.356575547 (80h7m30.919898539s) [noise=true , address=104.36.82.39:6756 , piece_id=OKKWMDJPZO6D5VRMGOE3GOCYGHM725JZTNOVYAWKJW3WYZPI4V5Q, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.357039590 (80h7m30.920362596s) [noise=true , address=104.36.82.53:4146 , piece_id=3V3DVGUOPJKDWIGQ2TU7M6VFYFVAS6ODY62ERGSU6LI3LSTWQ4YQ, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.360343566 (80h7m30.923666582s) [noise=true , address=104.36.82.9:6033 , piece_id=T5XA23LZELMSTWOLBSHOX4AHKDACF65OYCKTMZYTXDIBSXNA2T4A, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.452444064 (80h7m31.015767055s) [upload=64142]: Next returned nil segment Jun 22 00:30:55.452495148 (80h7m31.015818149s) [upload=64142]: Waiting for error group managing segments... Jun 22 00:30:55.532958808 (80h7m31.096281883s) [noise=true , address=104.36.82.9:6033 , piece_id=T5XA23LZELMSTWOLBSHOX4AHKDACF65OYCKTMZYTXDIBSXNA2T4A, seg_pos={0 0}, upload=64142]: Done uploading piece. err: ecclient: upload failed (node:1SV3LrcQW16wLewDzfyUWFB6sUgnERNJVdhz72CJmyde95M79C, address:104.36.82.9:6033): protocol: storage node overloaded, request limit: 2000 Jun 22 00:30:55.547802184 (80h7m31.111125188s) [noise=true , address=79.127.220.98:4402 , piece_id=TE2WQO5LK45AVMPIAIGVSIBOVZLC4FG4ESQ72X6KFYC7IAEN2DRA, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:30:55.554223621 (80h7m31.117546616s) [noise=true , address=77.90.31.14:4666 , piece_id=H4DTFDAVCN266PB235BJIQTQ32FBXTIB5AZSX3B4R2VIK554AFSQ, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:30:55.762514255 (80h7m31.325837245s) [noise=true , address=104.36.82.14:6245 , piece_id=II36G46OA6TINLR244U6D7Q6LGV754UA5M62K3EX7ZOBESMJUEHA, seg_pos={0 0}, upload=64142]: Done uploading piece. err: ecclient: upload failed (node:1Zu8teNPTXn638grHKwtJqz4UEY7oFYLx1DHvuVP6RZXcatVqX, address:104.36.82.14:6245): protocol: storage node overloaded, request limit: 2000 Jun 22 00:30:55.773669583 (80h7m31.3369926s) [noise=true , address=79.127.205.235:5660 , piece_id=UGFPVS7VNB7U2YSFPSGYDIRZV5R2DHCN57PJP6VOXO52WIAYHLXA, seg_pos={0 0}, upload=64142]: Done uploading piece. err: ecclient: upload failed (node:12E6uCX9KL4Ud7YpxMXepvMXuqTFJCiSXByDjW94Am3U9H2Nf7f, address:79.127.205.235:5660): protocol: storage node overloaded, request limit: 2000 Jun 22 00:30:56.968366655 (80h7m32.53168964s) [noise=true , address=77.90.31.10:4533 , piece_id=SE6YAKKFDL3BV677CDDUC32UNMPXJHTG23GWWYK5B4YIPKER6GAQ, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:30:57.403572874 (80h7m32.966895883s) [noise=true , address=45.91.116.5:5310 , piece_id=AXJOKDZRPVNL7HFT4QKJWRAHFM2UCM3GEVQB2KNYD6CXB6TH4D4Q, seg_pos={0 0}, upload=64142]: Done uploading piece. err: ecclient: upload failed (node:127RMUc62xRq9RU5z3XfkZsq5sehmv3kP8o53tr9BxVcPE3GQCY, address:45.91.116.5:5310): protocol: storage node overloaded, request limit: 2000 Jun 22 00:30:58.210389994 (80h7m33.773712995s) [noise=true , address=104.36.82.13:6199 , piece_id=YP62N2TO446YR3QRG5YLXECD7TN6SJ4UNNLZSK5H4CMV6JIF4AKA, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:30:58.693037961 (80h7m34.256360966s) [noise=true , address=104.36.82.39:6756 , piece_id=OKKWMDJPZO6D5VRMGOE3GOCYGHM725JZTNOVYAWKJW3WYZPI4V5Q, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:30:59.124834814 (80h7m34.688157819s) [noise=true , address=104.36.82.15:6283 , piece_id=N7TH2EMIOPATVHCSNH2WXUGU7KMMEJ5LCHR22SWEHSA5OV4CUCXQ, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:31:00.629657982 (80h7m36.192980977s) [noise=true , address=104.36.82.42:6803 , piece_id=R626UFD326S2UBJEZ2WMHVR7EN4SK427VXPDJZOOZ32MX25JWZWQ, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:31:00.933110546 (80h7m36.496433536s) [noise=true , address=79.127.201.215:4326 , piece_id=USEWZDX6P52U6MCALTBNQG3SV7N2JRKNU6O23TLS6JEWAS4D7ARA, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:31:05.669167243 (80h7m41.232490237s) [noise=true , address=104.36.82.56:4261 , piece_id=EYOYOYLDACGMPGJYX6ILQT3XHMUTQNXZEDC7ANHLZXIKYVDQFFVQ, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:31:08.443582763 (80h7m44.006905763s) [noise=true , address=104.36.82.54:4184 , piece_id=IZICVGZFMAYLIJTNUJ2OFLZVNZ6IPPYDZO4BSTJJPLHVCVOE5EAA, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:31:26.459138014 (80h8m2.022461013s) [noise=true , address=104.36.82.53:4146 , piece_id=3V3DVGUOPJKDWIGQ2TU7M6VFYFVAS6ODY62ERGSU6LI3LSTWQ4YQ, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:31:27.456050336 (80h8m3.019373336s) [noise=true , address=104.36.82.52:4089 , piece_id=X2VG4S3UHVNWA4AQJS7FLKWNVLTOZ3AKWI5ZDH24QCA3ENQW7BXQ, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:31:55.270234251 (80h8m30.833557256s) [noise=true , address=104.36.82.11:6125 , piece_id=2CX7RJOKNQA6ODZZZXYPSCYAQY5P35TWX7I56CMYAPFLJE3JEO5Q, seg_pos={0 0}, upload=64142]: Done uploading piece. err: ecclient: upload failed (node:12vricofwx67FRaxAGLbmgT1TYyKkqfJbxdsyeEPL4R7kg8pKzT, address:104.36.82.11:6125): protocol: expected piece hash; context deadline exceeded; EOF Jun 22 00:31:55.270319418 (80h8m30.833642407s) [noise=true , address=104.36.82.55:4224 , piece_id=XF7H2BYSNNNZOS2T77VIN75N76AYUXAQEI6D4LRXACU2KU62V2TA, seg_pos={0 0}, upload=64142]: Done uploading piece. err: ecclient: upload failed (node:12EjKmUiv4pxkVCB4WoiXim4uFpxrBPyfiSS4efTY3HKSE3jb5y, address:104.36.82.55:4224): protocol: write tcp 159.223.143.39:39152->104.36.82.55:4224: use of closed network connection; write tcp 159.223.143.39:39152->104.36.82.55:4224: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:39152->104.36.82.55:4224: use of closed network connection Jun 22 00:31:55.270353811 (80h8m30.833676799s) [seg_pos={0 0}, upload=64142]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Jun 22 00:31:55.301099877 (80h8m30.864422884s) [upload=64142]: Done uploading segments... Jun 22 00:31:55.301121567 (80h8m30.864444566s) [upload=64142]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 23.147175644s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [........X.........X...................X.....................] duration: 27.060433261s FAIL (2025-06-22 00:48:55 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 27.060458989s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [.....X.X..............................X.....................] duration: 30.988804091s FAIL (2025-06-22 00:48:07 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== ing upload Jun 22 00:30:55.303646184 (80h7m30.866969171s) [upload=64142]: Uploading segments... Jun 22 00:30:55.303706406 (80h7m30.867029416s) [upload=64142]: Got next segment. Inline: false Jun 22 00:30:55.303762479 (80h7m30.867085468s) [upload=64142]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 22 00:30:55.303768627 (80h7m30.867091636s) [upload=64142]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 22 00:30:55.349058140 (80h7m30.912381168s) [seg_pos={0 0}, upload=64142]: Begin upload segment... Jun 22 00:30:55.349113768 (80h7m30.912436772s) [seg_pos={0 0}, upload=64142]: Done begin upload segment. Jun 22 00:30:55.349143121 (80h7m30.912466131s) [noise=true , address=45.91.116.5:5310 , piece_id=AXJOKDZRPVNL7HFT4QKJWRAHFM2UCM3GEVQB2KNYD6CXB6TH4D4Q, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.349660917 (80h7m30.912983928s) [noise=true , address=77.90.31.10:4533 , piece_id=SE6YAKKFDL3BV677CDDUC32UNMPXJHTG23GWWYK5B4YIPKER6GAQ, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.350093077 (80h7m30.913416089s) [noise=true , address=79.127.201.215:4326 , piece_id=USEWZDX6P52U6MCALTBNQG3SV7N2JRKNU6O23TLS6JEWAS4D7ARA, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.350480894 (80h7m30.913803916s) [noise=true , address=104.36.82.13:6199 , piece_id=YP62N2TO446YR3QRG5YLXECD7TN6SJ4UNNLZSK5H4CMV6JIF4AKA, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.350669998 (80h7m30.913993011s) [noise=true , address=104.36.82.54:4184 , piece_id=IZICVGZFMAYLIJTNUJ2OFLZVNZ6IPPYDZO4BSTJJPLHVCVOE5EAA, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.351201352 (80h7m30.914524366s) [noise=true , address=104.36.82.55:4224 , piece_id=XF7H2BYSNNNZOS2T77VIN75N76AYUXAQEI6D4LRXACU2KU62V2TA, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.351668071 (80h7m30.914991083s) [noise=true , address=104.36.82.56:4261 , piece_id=EYOYOYLDACGMPGJYX6ILQT3XHMUTQNXZEDC7ANHLZXIKYVDQFFVQ, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.352193126 (80h7m30.915516138s) [noise=true , address=104.36.82.52:4089 , piece_id=X2VG4S3UHVNWA4AQJS7FLKWNVLTOZ3AKWI5ZDH24QCA3ENQW7BXQ, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.352587096 (80h7m30.915910103s) [noise=true , address=79.127.220.98:4402 , piece_id=TE2WQO5LK45AVMPIAIGVSIBOVZLC4FG4ESQ72X6KFYC7IAEN2DRA, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.353420253 (80h7m30.916743265s) [noise=true , address=104.36.82.11:6125 , piece_id=2CX7RJOKNQA6ODZZZXYPSCYAQY5P35TWX7I56CMYAPFLJE3JEO5Q, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.353791565 (80h7m30.917114578s) [noise=true , address=104.36.82.42:6803 , piece_id=R626UFD326S2UBJEZ2WMHVR7EN4SK427VXPDJZOOZ32MX25JWZWQ, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.354171226 (80h7m30.917494233s) [noise=true , address=104.36.82.14:6245 , piece_id=II36G46OA6TINLR244U6D7Q6LGV754UA5M62K3EX7ZOBESMJUEHA, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.354542282 (80h7m30.917865288s) [noise=true , address=79.127.205.235:5660 , piece_id=UGFPVS7VNB7U2YSFPSGYDIRZV5R2DHCN57PJP6VOXO52WIAYHLXA, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.354891655 (80h7m30.918214678s) [noise=true , address=104.36.82.15:6283 , piece_id=N7TH2EMIOPATVHCSNH2WXUGU7KMMEJ5LCHR22SWEHSA5OV4CUCXQ, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.355499503 (80h7m30.918822519s) [noise=true , address=77.90.31.14:4666 , piece_id=H4DTFDAVCN266PB235BJIQTQ32FBXTIB5AZSX3B4R2VIK554AFSQ, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.356575547 (80h7m30.919898539s) [noise=true , address=104.36.82.39:6756 , piece_id=OKKWMDJPZO6D5VRMGOE3GOCYGHM725JZTNOVYAWKJW3WYZPI4V5Q, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.357039590 (80h7m30.920362596s) [noise=true , address=104.36.82.53:4146 , piece_id=3V3DVGUOPJKDWIGQ2TU7M6VFYFVAS6ODY62ERGSU6LI3LSTWQ4YQ, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.360343566 (80h7m30.923666582s) [noise=true , address=104.36.82.9:6033 , piece_id=T5XA23LZELMSTWOLBSHOX4AHKDACF65OYCKTMZYTXDIBSXNA2T4A, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.452444064 (80h7m31.015767055s) [upload=64142]: Next returned nil segment Jun 22 00:30:55.452495148 (80h7m31.015818149s) [upload=64142]: Waiting for error group managing segments... Jun 22 00:30:55.532958808 (80h7m31.096281883s) [noise=true , address=104.36.82.9:6033 , piece_id=T5XA23LZELMSTWOLBSHOX4AHKDACF65OYCKTMZYTXDIBSXNA2T4A, seg_pos={0 0}, upload=64142]: Done uploading piece. err: ecclient: upload failed (node:1SV3LrcQW16wLewDzfyUWFB6sUgnERNJVdhz72CJmyde95M79C, address:104.36.82.9:6033): protocol: storage node overloaded, request limit: 2000 Jun 22 00:30:55.547802184 (80h7m31.111125188s) [noise=true , address=79.127.220.98:4402 , piece_id=TE2WQO5LK45AVMPIAIGVSIBOVZLC4FG4ESQ72X6KFYC7IAEN2DRA, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:30:55.554223621 (80h7m31.117546616s) [noise=true , address=77.90.31.14:4666 , piece_id=H4DTFDAVCN266PB235BJIQTQ32FBXTIB5AZSX3B4R2VIK554AFSQ, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:30:55.762514255 (80h7m31.325837245s) [noise=true , address=104.36.82.14:6245 , piece_id=II36G46OA6TINLR244U6D7Q6LGV754UA5M62K3EX7ZOBESMJUEHA, seg_pos={0 0}, upload=64142]: Done uploading piece. err: ecclient: upload failed (node:1Zu8teNPTXn638grHKwtJqz4UEY7oFYLx1DHvuVP6RZXcatVqX, address:104.36.82.14:6245): protocol: storage node overloaded, request limit: 2000 Jun 22 00:30:55.773669583 (80h7m31.3369926s) [noise=true , address=79.127.205.235:5660 , piece_id=UGFPVS7VNB7U2YSFPSGYDIRZV5R2DHCN57PJP6VOXO52WIAYHLXA, seg_pos={0 0}, upload=64142]: Done uploading piece. err: ecclient: upload failed (node:12E6uCX9KL4Ud7YpxMXepvMXuqTFJCiSXByDjW94Am3U9H2Nf7f, address:79.127.205.235:5660): protocol: storage node overloaded, request limit: 2000 Jun 22 00:30:56.968366655 (80h7m32.53168964s) [noise=true , address=77.90.31.10:4533 , piece_id=SE6YAKKFDL3BV677CDDUC32UNMPXJHTG23GWWYK5B4YIPKER6GAQ, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:30:57.403572874 (80h7m32.966895883s) [noise=true , address=45.91.116.5:5310 , piece_id=AXJOKDZRPVNL7HFT4QKJWRAHFM2UCM3GEVQB2KNYD6CXB6TH4D4Q, seg_pos={0 0}, upload=64142]: Done uploading piece. err: ecclient: upload failed (node:127RMUc62xRq9RU5z3XfkZsq5sehmv3kP8o53tr9BxVcPE3GQCY, address:45.91.116.5:5310): protocol: storage node overloaded, request limit: 2000 Jun 22 00:30:58.210389994 (80h7m33.773712995s) [noise=true , address=104.36.82.13:6199 , piece_id=YP62N2TO446YR3QRG5YLXECD7TN6SJ4UNNLZSK5H4CMV6JIF4AKA, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:30:58.693037961 (80h7m34.256360966s) [noise=true , address=104.36.82.39:6756 , piece_id=OKKWMDJPZO6D5VRMGOE3GOCYGHM725JZTNOVYAWKJW3WYZPI4V5Q, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:30:59.124834814 (80h7m34.688157819s) [noise=true , address=104.36.82.15:6283 , piece_id=N7TH2EMIOPATVHCSNH2WXUGU7KMMEJ5LCHR22SWEHSA5OV4CUCXQ, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:31:00.629657982 (80h7m36.192980977s) [noise=true , address=104.36.82.42:6803 , piece_id=R626UFD326S2UBJEZ2WMHVR7EN4SK427VXPDJZOOZ32MX25JWZWQ, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:31:00.933110546 (80h7m36.496433536s) [noise=true , address=79.127.201.215:4326 , piece_id=USEWZDX6P52U6MCALTBNQG3SV7N2JRKNU6O23TLS6JEWAS4D7ARA, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:31:05.669167243 (80h7m41.232490237s) [noise=true , address=104.36.82.56:4261 , piece_id=EYOYOYLDACGMPGJYX6ILQT3XHMUTQNXZEDC7ANHLZXIKYVDQFFVQ, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:31:08.443582763 (80h7m44.006905763s) [noise=true , address=104.36.82.54:4184 , piece_id=IZICVGZFMAYLIJTNUJ2OFLZVNZ6IPPYDZO4BSTJJPLHVCVOE5EAA, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:31:26.459138014 (80h8m2.022461013s) [noise=true , address=104.36.82.53:4146 , piece_id=3V3DVGUOPJKDWIGQ2TU7M6VFYFVAS6ODY62ERGSU6LI3LSTWQ4YQ, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:31:27.456050336 (80h8m3.019373336s) [noise=true , address=104.36.82.52:4089 , piece_id=X2VG4S3UHVNWA4AQJS7FLKWNVLTOZ3AKWI5ZDH24QCA3ENQW7BXQ, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:31:55.270234251 (80h8m30.833557256s) [noise=true , address=104.36.82.11:6125 , piece_id=2CX7RJOKNQA6ODZZZXYPSCYAQY5P35TWX7I56CMYAPFLJE3JEO5Q, seg_pos={0 0}, upload=64142]: Done uploading piece. err: ecclient: upload failed (node:12vricofwx67FRaxAGLbmgT1TYyKkqfJbxdsyeEPL4R7kg8pKzT, address:104.36.82.11:6125): protocol: expected piece hash; context deadline exceeded; EOF Jun 22 00:31:55.270319418 (80h8m30.833642407s) [noise=true , address=104.36.82.55:4224 , piece_id=XF7H2BYSNNNZOS2T77VIN75N76AYUXAQEI6D4LRXACU2KU62V2TA, seg_pos={0 0}, upload=64142]: Done uploading piece. err: ecclient: upload failed (node:12EjKmUiv4pxkVCB4WoiXim4uFpxrBPyfiSS4efTY3HKSE3jb5y, address:104.36.82.55:4224): protocol: write tcp 159.223.143.39:39152->104.36.82.55:4224: use of closed network connection; write tcp 159.223.143.39:39152->104.36.82.55:4224: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:39152->104.36.82.55:4224: use of closed network connection Jun 22 00:31:55.270353811 (80h8m30.833676799s) [seg_pos={0 0}, upload=64142]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Jun 22 00:31:55.301099877 (80h8m30.864422884s) [upload=64142]: Done uploading segments... Jun 22 00:31:55.301121567 (80h8m30.864444566s) [upload=64142]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 30.988832931s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [.........X.........X...................X....................] duration: 17.877330509s FAIL (2025-06-22 00:47:41 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 17.877382324s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [..........X.........X...................X...................] duration: 33.030832776s FAIL (2025-06-22 00:46:58 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 33.030902019s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [......X.X..............................X....................] duration: 7.951971649s FAIL (2025-06-22 00:46:37 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== ing upload Jun 22 00:30:55.303646184 (80h7m30.866969171s) [upload=64142]: Uploading segments... Jun 22 00:30:55.303706406 (80h7m30.867029416s) [upload=64142]: Got next segment. Inline: false Jun 22 00:30:55.303762479 (80h7m30.867085468s) [upload=64142]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 22 00:30:55.303768627 (80h7m30.867091636s) [upload=64142]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 22 00:30:55.349058140 (80h7m30.912381168s) [seg_pos={0 0}, upload=64142]: Begin upload segment... Jun 22 00:30:55.349113768 (80h7m30.912436772s) [seg_pos={0 0}, upload=64142]: Done begin upload segment. Jun 22 00:30:55.349143121 (80h7m30.912466131s) [noise=true , address=45.91.116.5:5310 , piece_id=AXJOKDZRPVNL7HFT4QKJWRAHFM2UCM3GEVQB2KNYD6CXB6TH4D4Q, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.349660917 (80h7m30.912983928s) [noise=true , address=77.90.31.10:4533 , piece_id=SE6YAKKFDL3BV677CDDUC32UNMPXJHTG23GWWYK5B4YIPKER6GAQ, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.350093077 (80h7m30.913416089s) [noise=true , address=79.127.201.215:4326 , piece_id=USEWZDX6P52U6MCALTBNQG3SV7N2JRKNU6O23TLS6JEWAS4D7ARA, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.350480894 (80h7m30.913803916s) [noise=true , address=104.36.82.13:6199 , piece_id=YP62N2TO446YR3QRG5YLXECD7TN6SJ4UNNLZSK5H4CMV6JIF4AKA, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.350669998 (80h7m30.913993011s) [noise=true , address=104.36.82.54:4184 , piece_id=IZICVGZFMAYLIJTNUJ2OFLZVNZ6IPPYDZO4BSTJJPLHVCVOE5EAA, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.351201352 (80h7m30.914524366s) [noise=true , address=104.36.82.55:4224 , piece_id=XF7H2BYSNNNZOS2T77VIN75N76AYUXAQEI6D4LRXACU2KU62V2TA, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.351668071 (80h7m30.914991083s) [noise=true , address=104.36.82.56:4261 , piece_id=EYOYOYLDACGMPGJYX6ILQT3XHMUTQNXZEDC7ANHLZXIKYVDQFFVQ, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.352193126 (80h7m30.915516138s) [noise=true , address=104.36.82.52:4089 , piece_id=X2VG4S3UHVNWA4AQJS7FLKWNVLTOZ3AKWI5ZDH24QCA3ENQW7BXQ, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.352587096 (80h7m30.915910103s) [noise=true , address=79.127.220.98:4402 , piece_id=TE2WQO5LK45AVMPIAIGVSIBOVZLC4FG4ESQ72X6KFYC7IAEN2DRA, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.353420253 (80h7m30.916743265s) [noise=true , address=104.36.82.11:6125 , piece_id=2CX7RJOKNQA6ODZZZXYPSCYAQY5P35TWX7I56CMYAPFLJE3JEO5Q, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.353791565 (80h7m30.917114578s) [noise=true , address=104.36.82.42:6803 , piece_id=R626UFD326S2UBJEZ2WMHVR7EN4SK427VXPDJZOOZ32MX25JWZWQ, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.354171226 (80h7m30.917494233s) [noise=true , address=104.36.82.14:6245 , piece_id=II36G46OA6TINLR244U6D7Q6LGV754UA5M62K3EX7ZOBESMJUEHA, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.354542282 (80h7m30.917865288s) [noise=true , address=79.127.205.235:5660 , piece_id=UGFPVS7VNB7U2YSFPSGYDIRZV5R2DHCN57PJP6VOXO52WIAYHLXA, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.354891655 (80h7m30.918214678s) [noise=true , address=104.36.82.15:6283 , piece_id=N7TH2EMIOPATVHCSNH2WXUGU7KMMEJ5LCHR22SWEHSA5OV4CUCXQ, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.355499503 (80h7m30.918822519s) [noise=true , address=77.90.31.14:4666 , piece_id=H4DTFDAVCN266PB235BJIQTQ32FBXTIB5AZSX3B4R2VIK554AFSQ, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.356575547 (80h7m30.919898539s) [noise=true , address=104.36.82.39:6756 , piece_id=OKKWMDJPZO6D5VRMGOE3GOCYGHM725JZTNOVYAWKJW3WYZPI4V5Q, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.357039590 (80h7m30.920362596s) [noise=true , address=104.36.82.53:4146 , piece_id=3V3DVGUOPJKDWIGQ2TU7M6VFYFVAS6ODY62ERGSU6LI3LSTWQ4YQ, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.360343566 (80h7m30.923666582s) [noise=true , address=104.36.82.9:6033 , piece_id=T5XA23LZELMSTWOLBSHOX4AHKDACF65OYCKTMZYTXDIBSXNA2T4A, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.452444064 (80h7m31.015767055s) [upload=64142]: Next returned nil segment Jun 22 00:30:55.452495148 (80h7m31.015818149s) [upload=64142]: Waiting for error group managing segments... Jun 22 00:30:55.532958808 (80h7m31.096281883s) [noise=true , address=104.36.82.9:6033 , piece_id=T5XA23LZELMSTWOLBSHOX4AHKDACF65OYCKTMZYTXDIBSXNA2T4A, seg_pos={0 0}, upload=64142]: Done uploading piece. err: ecclient: upload failed (node:1SV3LrcQW16wLewDzfyUWFB6sUgnERNJVdhz72CJmyde95M79C, address:104.36.82.9:6033): protocol: storage node overloaded, request limit: 2000 Jun 22 00:30:55.547802184 (80h7m31.111125188s) [noise=true , address=79.127.220.98:4402 , piece_id=TE2WQO5LK45AVMPIAIGVSIBOVZLC4FG4ESQ72X6KFYC7IAEN2DRA, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:30:55.554223621 (80h7m31.117546616s) [noise=true , address=77.90.31.14:4666 , piece_id=H4DTFDAVCN266PB235BJIQTQ32FBXTIB5AZSX3B4R2VIK554AFSQ, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:30:55.762514255 (80h7m31.325837245s) [noise=true , address=104.36.82.14:6245 , piece_id=II36G46OA6TINLR244U6D7Q6LGV754UA5M62K3EX7ZOBESMJUEHA, seg_pos={0 0}, upload=64142]: Done uploading piece. err: ecclient: upload failed (node:1Zu8teNPTXn638grHKwtJqz4UEY7oFYLx1DHvuVP6RZXcatVqX, address:104.36.82.14:6245): protocol: storage node overloaded, request limit: 2000 Jun 22 00:30:55.773669583 (80h7m31.3369926s) [noise=true , address=79.127.205.235:5660 , piece_id=UGFPVS7VNB7U2YSFPSGYDIRZV5R2DHCN57PJP6VOXO52WIAYHLXA, seg_pos={0 0}, upload=64142]: Done uploading piece. err: ecclient: upload failed (node:12E6uCX9KL4Ud7YpxMXepvMXuqTFJCiSXByDjW94Am3U9H2Nf7f, address:79.127.205.235:5660): protocol: storage node overloaded, request limit: 2000 Jun 22 00:30:56.968366655 (80h7m32.53168964s) [noise=true , address=77.90.31.10:4533 , piece_id=SE6YAKKFDL3BV677CDDUC32UNMPXJHTG23GWWYK5B4YIPKER6GAQ, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:30:57.403572874 (80h7m32.966895883s) [noise=true , address=45.91.116.5:5310 , piece_id=AXJOKDZRPVNL7HFT4QKJWRAHFM2UCM3GEVQB2KNYD6CXB6TH4D4Q, seg_pos={0 0}, upload=64142]: Done uploading piece. err: ecclient: upload failed (node:127RMUc62xRq9RU5z3XfkZsq5sehmv3kP8o53tr9BxVcPE3GQCY, address:45.91.116.5:5310): protocol: storage node overloaded, request limit: 2000 Jun 22 00:30:58.210389994 (80h7m33.773712995s) [noise=true , address=104.36.82.13:6199 , piece_id=YP62N2TO446YR3QRG5YLXECD7TN6SJ4UNNLZSK5H4CMV6JIF4AKA, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:30:58.693037961 (80h7m34.256360966s) [noise=true , address=104.36.82.39:6756 , piece_id=OKKWMDJPZO6D5VRMGOE3GOCYGHM725JZTNOVYAWKJW3WYZPI4V5Q, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:30:59.124834814 (80h7m34.688157819s) [noise=true , address=104.36.82.15:6283 , piece_id=N7TH2EMIOPATVHCSNH2WXUGU7KMMEJ5LCHR22SWEHSA5OV4CUCXQ, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:31:00.629657982 (80h7m36.192980977s) [noise=true , address=104.36.82.42:6803 , piece_id=R626UFD326S2UBJEZ2WMHVR7EN4SK427VXPDJZOOZ32MX25JWZWQ, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:31:00.933110546 (80h7m36.496433536s) [noise=true , address=79.127.201.215:4326 , piece_id=USEWZDX6P52U6MCALTBNQG3SV7N2JRKNU6O23TLS6JEWAS4D7ARA, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:31:05.669167243 (80h7m41.232490237s) [noise=true , address=104.36.82.56:4261 , piece_id=EYOYOYLDACGMPGJYX6ILQT3XHMUTQNXZEDC7ANHLZXIKYVDQFFVQ, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:31:08.443582763 (80h7m44.006905763s) [noise=true , address=104.36.82.54:4184 , piece_id=IZICVGZFMAYLIJTNUJ2OFLZVNZ6IPPYDZO4BSTJJPLHVCVOE5EAA, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:31:26.459138014 (80h8m2.022461013s) [noise=true , address=104.36.82.53:4146 , piece_id=3V3DVGUOPJKDWIGQ2TU7M6VFYFVAS6ODY62ERGSU6LI3LSTWQ4YQ, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:31:27.456050336 (80h8m3.019373336s) [noise=true , address=104.36.82.52:4089 , piece_id=X2VG4S3UHVNWA4AQJS7FLKWNVLTOZ3AKWI5ZDH24QCA3ENQW7BXQ, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:31:55.270234251 (80h8m30.833557256s) [noise=true , address=104.36.82.11:6125 , piece_id=2CX7RJOKNQA6ODZZZXYPSCYAQY5P35TWX7I56CMYAPFLJE3JEO5Q, seg_pos={0 0}, upload=64142]: Done uploading piece. err: ecclient: upload failed (node:12vricofwx67FRaxAGLbmgT1TYyKkqfJbxdsyeEPL4R7kg8pKzT, address:104.36.82.11:6125): protocol: expected piece hash; context deadline exceeded; EOF Jun 22 00:31:55.270319418 (80h8m30.833642407s) [noise=true , address=104.36.82.55:4224 , piece_id=XF7H2BYSNNNZOS2T77VIN75N76AYUXAQEI6D4LRXACU2KU62V2TA, seg_pos={0 0}, upload=64142]: Done uploading piece. err: ecclient: upload failed (node:12EjKmUiv4pxkVCB4WoiXim4uFpxrBPyfiSS4efTY3HKSE3jb5y, address:104.36.82.55:4224): protocol: write tcp 159.223.143.39:39152->104.36.82.55:4224: use of closed network connection; write tcp 159.223.143.39:39152->104.36.82.55:4224: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:39152->104.36.82.55:4224: use of closed network connection Jun 22 00:31:55.270353811 (80h8m30.833676799s) [seg_pos={0 0}, upload=64142]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Jun 22 00:31:55.301099877 (80h8m30.864422884s) [upload=64142]: Done uploading segments... Jun 22 00:31:55.301121567 (80h8m30.864444566s) [upload=64142]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 7.951999549s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [...........X.........X...................X..................] duration: 16.777241362s FAIL (2025-06-22 00:46:14 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 16.777260822s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [.......X.X..............................X...................] duration: 35.961084498s FAIL (2025-06-22 00:45:57 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== ing upload Jun 22 00:30:55.303646184 (80h7m30.866969171s) [upload=64142]: Uploading segments... Jun 22 00:30:55.303706406 (80h7m30.867029416s) [upload=64142]: Got next segment. Inline: false Jun 22 00:30:55.303762479 (80h7m30.867085468s) [upload=64142]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 22 00:30:55.303768627 (80h7m30.867091636s) [upload=64142]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 22 00:30:55.349058140 (80h7m30.912381168s) [seg_pos={0 0}, upload=64142]: Begin upload segment... Jun 22 00:30:55.349113768 (80h7m30.912436772s) [seg_pos={0 0}, upload=64142]: Done begin upload segment. Jun 22 00:30:55.349143121 (80h7m30.912466131s) [noise=true , address=45.91.116.5:5310 , piece_id=AXJOKDZRPVNL7HFT4QKJWRAHFM2UCM3GEVQB2KNYD6CXB6TH4D4Q, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.349660917 (80h7m30.912983928s) [noise=true , address=77.90.31.10:4533 , piece_id=SE6YAKKFDL3BV677CDDUC32UNMPXJHTG23GWWYK5B4YIPKER6GAQ, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.350093077 (80h7m30.913416089s) [noise=true , address=79.127.201.215:4326 , piece_id=USEWZDX6P52U6MCALTBNQG3SV7N2JRKNU6O23TLS6JEWAS4D7ARA, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.350480894 (80h7m30.913803916s) [noise=true , address=104.36.82.13:6199 , piece_id=YP62N2TO446YR3QRG5YLXECD7TN6SJ4UNNLZSK5H4CMV6JIF4AKA, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.350669998 (80h7m30.913993011s) [noise=true , address=104.36.82.54:4184 , piece_id=IZICVGZFMAYLIJTNUJ2OFLZVNZ6IPPYDZO4BSTJJPLHVCVOE5EAA, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.351201352 (80h7m30.914524366s) [noise=true , address=104.36.82.55:4224 , piece_id=XF7H2BYSNNNZOS2T77VIN75N76AYUXAQEI6D4LRXACU2KU62V2TA, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.351668071 (80h7m30.914991083s) [noise=true , address=104.36.82.56:4261 , piece_id=EYOYOYLDACGMPGJYX6ILQT3XHMUTQNXZEDC7ANHLZXIKYVDQFFVQ, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.352193126 (80h7m30.915516138s) [noise=true , address=104.36.82.52:4089 , piece_id=X2VG4S3UHVNWA4AQJS7FLKWNVLTOZ3AKWI5ZDH24QCA3ENQW7BXQ, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.352587096 (80h7m30.915910103s) [noise=true , address=79.127.220.98:4402 , piece_id=TE2WQO5LK45AVMPIAIGVSIBOVZLC4FG4ESQ72X6KFYC7IAEN2DRA, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.353420253 (80h7m30.916743265s) [noise=true , address=104.36.82.11:6125 , piece_id=2CX7RJOKNQA6ODZZZXYPSCYAQY5P35TWX7I56CMYAPFLJE3JEO5Q, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.353791565 (80h7m30.917114578s) [noise=true , address=104.36.82.42:6803 , piece_id=R626UFD326S2UBJEZ2WMHVR7EN4SK427VXPDJZOOZ32MX25JWZWQ, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.354171226 (80h7m30.917494233s) [noise=true , address=104.36.82.14:6245 , piece_id=II36G46OA6TINLR244U6D7Q6LGV754UA5M62K3EX7ZOBESMJUEHA, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.354542282 (80h7m30.917865288s) [noise=true , address=79.127.205.235:5660 , piece_id=UGFPVS7VNB7U2YSFPSGYDIRZV5R2DHCN57PJP6VOXO52WIAYHLXA, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.354891655 (80h7m30.918214678s) [noise=true , address=104.36.82.15:6283 , piece_id=N7TH2EMIOPATVHCSNH2WXUGU7KMMEJ5LCHR22SWEHSA5OV4CUCXQ, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.355499503 (80h7m30.918822519s) [noise=true , address=77.90.31.14:4666 , piece_id=H4DTFDAVCN266PB235BJIQTQ32FBXTIB5AZSX3B4R2VIK554AFSQ, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.356575547 (80h7m30.919898539s) [noise=true , address=104.36.82.39:6756 , piece_id=OKKWMDJPZO6D5VRMGOE3GOCYGHM725JZTNOVYAWKJW3WYZPI4V5Q, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.357039590 (80h7m30.920362596s) [noise=true , address=104.36.82.53:4146 , piece_id=3V3DVGUOPJKDWIGQ2TU7M6VFYFVAS6ODY62ERGSU6LI3LSTWQ4YQ, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.360343566 (80h7m30.923666582s) [noise=true , address=104.36.82.9:6033 , piece_id=T5XA23LZELMSTWOLBSHOX4AHKDACF65OYCKTMZYTXDIBSXNA2T4A, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.452444064 (80h7m31.015767055s) [upload=64142]: Next returned nil segment Jun 22 00:30:55.452495148 (80h7m31.015818149s) [upload=64142]: Waiting for error group managing segments... Jun 22 00:30:55.532958808 (80h7m31.096281883s) [noise=true , address=104.36.82.9:6033 , piece_id=T5XA23LZELMSTWOLBSHOX4AHKDACF65OYCKTMZYTXDIBSXNA2T4A, seg_pos={0 0}, upload=64142]: Done uploading piece. err: ecclient: upload failed (node:1SV3LrcQW16wLewDzfyUWFB6sUgnERNJVdhz72CJmyde95M79C, address:104.36.82.9:6033): protocol: storage node overloaded, request limit: 2000 Jun 22 00:30:55.547802184 (80h7m31.111125188s) [noise=true , address=79.127.220.98:4402 , piece_id=TE2WQO5LK45AVMPIAIGVSIBOVZLC4FG4ESQ72X6KFYC7IAEN2DRA, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:30:55.554223621 (80h7m31.117546616s) [noise=true , address=77.90.31.14:4666 , piece_id=H4DTFDAVCN266PB235BJIQTQ32FBXTIB5AZSX3B4R2VIK554AFSQ, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:30:55.762514255 (80h7m31.325837245s) [noise=true , address=104.36.82.14:6245 , piece_id=II36G46OA6TINLR244U6D7Q6LGV754UA5M62K3EX7ZOBESMJUEHA, seg_pos={0 0}, upload=64142]: Done uploading piece. err: ecclient: upload failed (node:1Zu8teNPTXn638grHKwtJqz4UEY7oFYLx1DHvuVP6RZXcatVqX, address:104.36.82.14:6245): protocol: storage node overloaded, request limit: 2000 Jun 22 00:30:55.773669583 (80h7m31.3369926s) [noise=true , address=79.127.205.235:5660 , piece_id=UGFPVS7VNB7U2YSFPSGYDIRZV5R2DHCN57PJP6VOXO52WIAYHLXA, seg_pos={0 0}, upload=64142]: Done uploading piece. err: ecclient: upload failed (node:12E6uCX9KL4Ud7YpxMXepvMXuqTFJCiSXByDjW94Am3U9H2Nf7f, address:79.127.205.235:5660): protocol: storage node overloaded, request limit: 2000 Jun 22 00:30:56.968366655 (80h7m32.53168964s) [noise=true , address=77.90.31.10:4533 , piece_id=SE6YAKKFDL3BV677CDDUC32UNMPXJHTG23GWWYK5B4YIPKER6GAQ, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:30:57.403572874 (80h7m32.966895883s) [noise=true , address=45.91.116.5:5310 , piece_id=AXJOKDZRPVNL7HFT4QKJWRAHFM2UCM3GEVQB2KNYD6CXB6TH4D4Q, seg_pos={0 0}, upload=64142]: Done uploading piece. err: ecclient: upload failed (node:127RMUc62xRq9RU5z3XfkZsq5sehmv3kP8o53tr9BxVcPE3GQCY, address:45.91.116.5:5310): protocol: storage node overloaded, request limit: 2000 Jun 22 00:30:58.210389994 (80h7m33.773712995s) [noise=true , address=104.36.82.13:6199 , piece_id=YP62N2TO446YR3QRG5YLXECD7TN6SJ4UNNLZSK5H4CMV6JIF4AKA, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:30:58.693037961 (80h7m34.256360966s) [noise=true , address=104.36.82.39:6756 , piece_id=OKKWMDJPZO6D5VRMGOE3GOCYGHM725JZTNOVYAWKJW3WYZPI4V5Q, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:30:59.124834814 (80h7m34.688157819s) [noise=true , address=104.36.82.15:6283 , piece_id=N7TH2EMIOPATVHCSNH2WXUGU7KMMEJ5LCHR22SWEHSA5OV4CUCXQ, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:31:00.629657982 (80h7m36.192980977s) [noise=true , address=104.36.82.42:6803 , piece_id=R626UFD326S2UBJEZ2WMHVR7EN4SK427VXPDJZOOZ32MX25JWZWQ, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:31:00.933110546 (80h7m36.496433536s) [noise=true , address=79.127.201.215:4326 , piece_id=USEWZDX6P52U6MCALTBNQG3SV7N2JRKNU6O23TLS6JEWAS4D7ARA, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:31:05.669167243 (80h7m41.232490237s) [noise=true , address=104.36.82.56:4261 , piece_id=EYOYOYLDACGMPGJYX6ILQT3XHMUTQNXZEDC7ANHLZXIKYVDQFFVQ, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:31:08.443582763 (80h7m44.006905763s) [noise=true , address=104.36.82.54:4184 , piece_id=IZICVGZFMAYLIJTNUJ2OFLZVNZ6IPPYDZO4BSTJJPLHVCVOE5EAA, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:31:26.459138014 (80h8m2.022461013s) [noise=true , address=104.36.82.53:4146 , piece_id=3V3DVGUOPJKDWIGQ2TU7M6VFYFVAS6ODY62ERGSU6LI3LSTWQ4YQ, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:31:27.456050336 (80h8m3.019373336s) [noise=true , address=104.36.82.52:4089 , piece_id=X2VG4S3UHVNWA4AQJS7FLKWNVLTOZ3AKWI5ZDH24QCA3ENQW7BXQ, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:31:55.270234251 (80h8m30.833557256s) [noise=true , address=104.36.82.11:6125 , piece_id=2CX7RJOKNQA6ODZZZXYPSCYAQY5P35TWX7I56CMYAPFLJE3JEO5Q, seg_pos={0 0}, upload=64142]: Done uploading piece. err: ecclient: upload failed (node:12vricofwx67FRaxAGLbmgT1TYyKkqfJbxdsyeEPL4R7kg8pKzT, address:104.36.82.11:6125): protocol: expected piece hash; context deadline exceeded; EOF Jun 22 00:31:55.270319418 (80h8m30.833642407s) [noise=true , address=104.36.82.55:4224 , piece_id=XF7H2BYSNNNZOS2T77VIN75N76AYUXAQEI6D4LRXACU2KU62V2TA, seg_pos={0 0}, upload=64142]: Done uploading piece. err: ecclient: upload failed (node:12EjKmUiv4pxkVCB4WoiXim4uFpxrBPyfiSS4efTY3HKSE3jb5y, address:104.36.82.55:4224): protocol: write tcp 159.223.143.39:39152->104.36.82.55:4224: use of closed network connection; write tcp 159.223.143.39:39152->104.36.82.55:4224: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:39152->104.36.82.55:4224: use of closed network connection Jun 22 00:31:55.270353811 (80h8m30.833676799s) [seg_pos={0 0}, upload=64142]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Jun 22 00:31:55.301099877 (80h8m30.864422884s) [upload=64142]: Done uploading segments... Jun 22 00:31:55.301121567 (80h8m30.864444566s) [upload=64142]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 35.961161936s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [............X.........X...................X.................] duration: 37.288639641s FAIL (2025-06-22 00:45:37 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 37.288663274s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [........X.X..............................X..................] duration: 44.348547745s FAIL (2025-06-22 00:45:00 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== ing upload Jun 22 00:30:55.303646184 (80h7m30.866969171s) [upload=64142]: Uploading segments... Jun 22 00:30:55.303706406 (80h7m30.867029416s) [upload=64142]: Got next segment. Inline: false Jun 22 00:30:55.303762479 (80h7m30.867085468s) [upload=64142]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 22 00:30:55.303768627 (80h7m30.867091636s) [upload=64142]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 22 00:30:55.349058140 (80h7m30.912381168s) [seg_pos={0 0}, upload=64142]: Begin upload segment... Jun 22 00:30:55.349113768 (80h7m30.912436772s) [seg_pos={0 0}, upload=64142]: Done begin upload segment. Jun 22 00:30:55.349143121 (80h7m30.912466131s) [noise=true , address=45.91.116.5:5310 , piece_id=AXJOKDZRPVNL7HFT4QKJWRAHFM2UCM3GEVQB2KNYD6CXB6TH4D4Q, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.349660917 (80h7m30.912983928s) [noise=true , address=77.90.31.10:4533 , piece_id=SE6YAKKFDL3BV677CDDUC32UNMPXJHTG23GWWYK5B4YIPKER6GAQ, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.350093077 (80h7m30.913416089s) [noise=true , address=79.127.201.215:4326 , piece_id=USEWZDX6P52U6MCALTBNQG3SV7N2JRKNU6O23TLS6JEWAS4D7ARA, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.350480894 (80h7m30.913803916s) [noise=true , address=104.36.82.13:6199 , piece_id=YP62N2TO446YR3QRG5YLXECD7TN6SJ4UNNLZSK5H4CMV6JIF4AKA, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.350669998 (80h7m30.913993011s) [noise=true , address=104.36.82.54:4184 , piece_id=IZICVGZFMAYLIJTNUJ2OFLZVNZ6IPPYDZO4BSTJJPLHVCVOE5EAA, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.351201352 (80h7m30.914524366s) [noise=true , address=104.36.82.55:4224 , piece_id=XF7H2BYSNNNZOS2T77VIN75N76AYUXAQEI6D4LRXACU2KU62V2TA, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.351668071 (80h7m30.914991083s) [noise=true , address=104.36.82.56:4261 , piece_id=EYOYOYLDACGMPGJYX6ILQT3XHMUTQNXZEDC7ANHLZXIKYVDQFFVQ, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.352193126 (80h7m30.915516138s) [noise=true , address=104.36.82.52:4089 , piece_id=X2VG4S3UHVNWA4AQJS7FLKWNVLTOZ3AKWI5ZDH24QCA3ENQW7BXQ, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.352587096 (80h7m30.915910103s) [noise=true , address=79.127.220.98:4402 , piece_id=TE2WQO5LK45AVMPIAIGVSIBOVZLC4FG4ESQ72X6KFYC7IAEN2DRA, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.353420253 (80h7m30.916743265s) [noise=true , address=104.36.82.11:6125 , piece_id=2CX7RJOKNQA6ODZZZXYPSCYAQY5P35TWX7I56CMYAPFLJE3JEO5Q, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.353791565 (80h7m30.917114578s) [noise=true , address=104.36.82.42:6803 , piece_id=R626UFD326S2UBJEZ2WMHVR7EN4SK427VXPDJZOOZ32MX25JWZWQ, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.354171226 (80h7m30.917494233s) [noise=true , address=104.36.82.14:6245 , piece_id=II36G46OA6TINLR244U6D7Q6LGV754UA5M62K3EX7ZOBESMJUEHA, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.354542282 (80h7m30.917865288s) [noise=true , address=79.127.205.235:5660 , piece_id=UGFPVS7VNB7U2YSFPSGYDIRZV5R2DHCN57PJP6VOXO52WIAYHLXA, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.354891655 (80h7m30.918214678s) [noise=true , address=104.36.82.15:6283 , piece_id=N7TH2EMIOPATVHCSNH2WXUGU7KMMEJ5LCHR22SWEHSA5OV4CUCXQ, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.355499503 (80h7m30.918822519s) [noise=true , address=77.90.31.14:4666 , piece_id=H4DTFDAVCN266PB235BJIQTQ32FBXTIB5AZSX3B4R2VIK554AFSQ, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.356575547 (80h7m30.919898539s) [noise=true , address=104.36.82.39:6756 , piece_id=OKKWMDJPZO6D5VRMGOE3GOCYGHM725JZTNOVYAWKJW3WYZPI4V5Q, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.357039590 (80h7m30.920362596s) [noise=true , address=104.36.82.53:4146 , piece_id=3V3DVGUOPJKDWIGQ2TU7M6VFYFVAS6ODY62ERGSU6LI3LSTWQ4YQ, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.360343566 (80h7m30.923666582s) [noise=true , address=104.36.82.9:6033 , piece_id=T5XA23LZELMSTWOLBSHOX4AHKDACF65OYCKTMZYTXDIBSXNA2T4A, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.452444064 (80h7m31.015767055s) [upload=64142]: Next returned nil segment Jun 22 00:30:55.452495148 (80h7m31.015818149s) [upload=64142]: Waiting for error group managing segments... Jun 22 00:30:55.532958808 (80h7m31.096281883s) [noise=true , address=104.36.82.9:6033 , piece_id=T5XA23LZELMSTWOLBSHOX4AHKDACF65OYCKTMZYTXDIBSXNA2T4A, seg_pos={0 0}, upload=64142]: Done uploading piece. err: ecclient: upload failed (node:1SV3LrcQW16wLewDzfyUWFB6sUgnERNJVdhz72CJmyde95M79C, address:104.36.82.9:6033): protocol: storage node overloaded, request limit: 2000 Jun 22 00:30:55.547802184 (80h7m31.111125188s) [noise=true , address=79.127.220.98:4402 , piece_id=TE2WQO5LK45AVMPIAIGVSIBOVZLC4FG4ESQ72X6KFYC7IAEN2DRA, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:30:55.554223621 (80h7m31.117546616s) [noise=true , address=77.90.31.14:4666 , piece_id=H4DTFDAVCN266PB235BJIQTQ32FBXTIB5AZSX3B4R2VIK554AFSQ, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:30:55.762514255 (80h7m31.325837245s) [noise=true , address=104.36.82.14:6245 , piece_id=II36G46OA6TINLR244U6D7Q6LGV754UA5M62K3EX7ZOBESMJUEHA, seg_pos={0 0}, upload=64142]: Done uploading piece. err: ecclient: upload failed (node:1Zu8teNPTXn638grHKwtJqz4UEY7oFYLx1DHvuVP6RZXcatVqX, address:104.36.82.14:6245): protocol: storage node overloaded, request limit: 2000 Jun 22 00:30:55.773669583 (80h7m31.3369926s) [noise=true , address=79.127.205.235:5660 , piece_id=UGFPVS7VNB7U2YSFPSGYDIRZV5R2DHCN57PJP6VOXO52WIAYHLXA, seg_pos={0 0}, upload=64142]: Done uploading piece. err: ecclient: upload failed (node:12E6uCX9KL4Ud7YpxMXepvMXuqTFJCiSXByDjW94Am3U9H2Nf7f, address:79.127.205.235:5660): protocol: storage node overloaded, request limit: 2000 Jun 22 00:30:56.968366655 (80h7m32.53168964s) [noise=true , address=77.90.31.10:4533 , piece_id=SE6YAKKFDL3BV677CDDUC32UNMPXJHTG23GWWYK5B4YIPKER6GAQ, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:30:57.403572874 (80h7m32.966895883s) [noise=true , address=45.91.116.5:5310 , piece_id=AXJOKDZRPVNL7HFT4QKJWRAHFM2UCM3GEVQB2KNYD6CXB6TH4D4Q, seg_pos={0 0}, upload=64142]: Done uploading piece. err: ecclient: upload failed (node:127RMUc62xRq9RU5z3XfkZsq5sehmv3kP8o53tr9BxVcPE3GQCY, address:45.91.116.5:5310): protocol: storage node overloaded, request limit: 2000 Jun 22 00:30:58.210389994 (80h7m33.773712995s) [noise=true , address=104.36.82.13:6199 , piece_id=YP62N2TO446YR3QRG5YLXECD7TN6SJ4UNNLZSK5H4CMV6JIF4AKA, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:30:58.693037961 (80h7m34.256360966s) [noise=true , address=104.36.82.39:6756 , piece_id=OKKWMDJPZO6D5VRMGOE3GOCYGHM725JZTNOVYAWKJW3WYZPI4V5Q, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:30:59.124834814 (80h7m34.688157819s) [noise=true , address=104.36.82.15:6283 , piece_id=N7TH2EMIOPATVHCSNH2WXUGU7KMMEJ5LCHR22SWEHSA5OV4CUCXQ, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:31:00.629657982 (80h7m36.192980977s) [noise=true , address=104.36.82.42:6803 , piece_id=R626UFD326S2UBJEZ2WMHVR7EN4SK427VXPDJZOOZ32MX25JWZWQ, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:31:00.933110546 (80h7m36.496433536s) [noise=true , address=79.127.201.215:4326 , piece_id=USEWZDX6P52U6MCALTBNQG3SV7N2JRKNU6O23TLS6JEWAS4D7ARA, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:31:05.669167243 (80h7m41.232490237s) [noise=true , address=104.36.82.56:4261 , piece_id=EYOYOYLDACGMPGJYX6ILQT3XHMUTQNXZEDC7ANHLZXIKYVDQFFVQ, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:31:08.443582763 (80h7m44.006905763s) [noise=true , address=104.36.82.54:4184 , piece_id=IZICVGZFMAYLIJTNUJ2OFLZVNZ6IPPYDZO4BSTJJPLHVCVOE5EAA, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:31:26.459138014 (80h8m2.022461013s) [noise=true , address=104.36.82.53:4146 , piece_id=3V3DVGUOPJKDWIGQ2TU7M6VFYFVAS6ODY62ERGSU6LI3LSTWQ4YQ, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:31:27.456050336 (80h8m3.019373336s) [noise=true , address=104.36.82.52:4089 , piece_id=X2VG4S3UHVNWA4AQJS7FLKWNVLTOZ3AKWI5ZDH24QCA3ENQW7BXQ, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:31:55.270234251 (80h8m30.833557256s) [noise=true , address=104.36.82.11:6125 , piece_id=2CX7RJOKNQA6ODZZZXYPSCYAQY5P35TWX7I56CMYAPFLJE3JEO5Q, seg_pos={0 0}, upload=64142]: Done uploading piece. err: ecclient: upload failed (node:12vricofwx67FRaxAGLbmgT1TYyKkqfJbxdsyeEPL4R7kg8pKzT, address:104.36.82.11:6125): protocol: expected piece hash; context deadline exceeded; EOF Jun 22 00:31:55.270319418 (80h8m30.833642407s) [noise=true , address=104.36.82.55:4224 , piece_id=XF7H2BYSNNNZOS2T77VIN75N76AYUXAQEI6D4LRXACU2KU62V2TA, seg_pos={0 0}, upload=64142]: Done uploading piece. err: ecclient: upload failed (node:12EjKmUiv4pxkVCB4WoiXim4uFpxrBPyfiSS4efTY3HKSE3jb5y, address:104.36.82.55:4224): protocol: write tcp 159.223.143.39:39152->104.36.82.55:4224: use of closed network connection; write tcp 159.223.143.39:39152->104.36.82.55:4224: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:39152->104.36.82.55:4224: use of closed network connection Jun 22 00:31:55.270353811 (80h8m30.833676799s) [seg_pos={0 0}, upload=64142]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Jun 22 00:31:55.301099877 (80h8m30.864422884s) [upload=64142]: Done uploading segments... Jun 22 00:31:55.301121567 (80h8m30.864444566s) [upload=64142]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 44.348572591s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [.........X.X..............................X.................] duration: 46.208624836s FAIL (2025-06-22 00:44:16 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== ing upload Jun 22 00:30:55.303646184 (80h7m30.866969171s) [upload=64142]: Uploading segments... Jun 22 00:30:55.303706406 (80h7m30.867029416s) [upload=64142]: Got next segment. Inline: false Jun 22 00:30:55.303762479 (80h7m30.867085468s) [upload=64142]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 22 00:30:55.303768627 (80h7m30.867091636s) [upload=64142]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 22 00:30:55.349058140 (80h7m30.912381168s) [seg_pos={0 0}, upload=64142]: Begin upload segment... Jun 22 00:30:55.349113768 (80h7m30.912436772s) [seg_pos={0 0}, upload=64142]: Done begin upload segment. Jun 22 00:30:55.349143121 (80h7m30.912466131s) [noise=true , address=45.91.116.5:5310 , piece_id=AXJOKDZRPVNL7HFT4QKJWRAHFM2UCM3GEVQB2KNYD6CXB6TH4D4Q, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.349660917 (80h7m30.912983928s) [noise=true , address=77.90.31.10:4533 , piece_id=SE6YAKKFDL3BV677CDDUC32UNMPXJHTG23GWWYK5B4YIPKER6GAQ, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.350093077 (80h7m30.913416089s) [noise=true , address=79.127.201.215:4326 , piece_id=USEWZDX6P52U6MCALTBNQG3SV7N2JRKNU6O23TLS6JEWAS4D7ARA, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.350480894 (80h7m30.913803916s) [noise=true , address=104.36.82.13:6199 , piece_id=YP62N2TO446YR3QRG5YLXECD7TN6SJ4UNNLZSK5H4CMV6JIF4AKA, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.350669998 (80h7m30.913993011s) [noise=true , address=104.36.82.54:4184 , piece_id=IZICVGZFMAYLIJTNUJ2OFLZVNZ6IPPYDZO4BSTJJPLHVCVOE5EAA, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.351201352 (80h7m30.914524366s) [noise=true , address=104.36.82.55:4224 , piece_id=XF7H2BYSNNNZOS2T77VIN75N76AYUXAQEI6D4LRXACU2KU62V2TA, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.351668071 (80h7m30.914991083s) [noise=true , address=104.36.82.56:4261 , piece_id=EYOYOYLDACGMPGJYX6ILQT3XHMUTQNXZEDC7ANHLZXIKYVDQFFVQ, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.352193126 (80h7m30.915516138s) [noise=true , address=104.36.82.52:4089 , piece_id=X2VG4S3UHVNWA4AQJS7FLKWNVLTOZ3AKWI5ZDH24QCA3ENQW7BXQ, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.352587096 (80h7m30.915910103s) [noise=true , address=79.127.220.98:4402 , piece_id=TE2WQO5LK45AVMPIAIGVSIBOVZLC4FG4ESQ72X6KFYC7IAEN2DRA, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.353420253 (80h7m30.916743265s) [noise=true , address=104.36.82.11:6125 , piece_id=2CX7RJOKNQA6ODZZZXYPSCYAQY5P35TWX7I56CMYAPFLJE3JEO5Q, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.353791565 (80h7m30.917114578s) [noise=true , address=104.36.82.42:6803 , piece_id=R626UFD326S2UBJEZ2WMHVR7EN4SK427VXPDJZOOZ32MX25JWZWQ, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.354171226 (80h7m30.917494233s) [noise=true , address=104.36.82.14:6245 , piece_id=II36G46OA6TINLR244U6D7Q6LGV754UA5M62K3EX7ZOBESMJUEHA, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.354542282 (80h7m30.917865288s) [noise=true , address=79.127.205.235:5660 , piece_id=UGFPVS7VNB7U2YSFPSGYDIRZV5R2DHCN57PJP6VOXO52WIAYHLXA, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.354891655 (80h7m30.918214678s) [noise=true , address=104.36.82.15:6283 , piece_id=N7TH2EMIOPATVHCSNH2WXUGU7KMMEJ5LCHR22SWEHSA5OV4CUCXQ, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.355499503 (80h7m30.918822519s) [noise=true , address=77.90.31.14:4666 , piece_id=H4DTFDAVCN266PB235BJIQTQ32FBXTIB5AZSX3B4R2VIK554AFSQ, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.356575547 (80h7m30.919898539s) [noise=true , address=104.36.82.39:6756 , piece_id=OKKWMDJPZO6D5VRMGOE3GOCYGHM725JZTNOVYAWKJW3WYZPI4V5Q, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.357039590 (80h7m30.920362596s) [noise=true , address=104.36.82.53:4146 , piece_id=3V3DVGUOPJKDWIGQ2TU7M6VFYFVAS6ODY62ERGSU6LI3LSTWQ4YQ, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.360343566 (80h7m30.923666582s) [noise=true , address=104.36.82.9:6033 , piece_id=T5XA23LZELMSTWOLBSHOX4AHKDACF65OYCKTMZYTXDIBSXNA2T4A, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.452444064 (80h7m31.015767055s) [upload=64142]: Next returned nil segment Jun 22 00:30:55.452495148 (80h7m31.015818149s) [upload=64142]: Waiting for error group managing segments... Jun 22 00:30:55.532958808 (80h7m31.096281883s) [noise=true , address=104.36.82.9:6033 , piece_id=T5XA23LZELMSTWOLBSHOX4AHKDACF65OYCKTMZYTXDIBSXNA2T4A, seg_pos={0 0}, upload=64142]: Done uploading piece. err: ecclient: upload failed (node:1SV3LrcQW16wLewDzfyUWFB6sUgnERNJVdhz72CJmyde95M79C, address:104.36.82.9:6033): protocol: storage node overloaded, request limit: 2000 Jun 22 00:30:55.547802184 (80h7m31.111125188s) [noise=true , address=79.127.220.98:4402 , piece_id=TE2WQO5LK45AVMPIAIGVSIBOVZLC4FG4ESQ72X6KFYC7IAEN2DRA, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:30:55.554223621 (80h7m31.117546616s) [noise=true , address=77.90.31.14:4666 , piece_id=H4DTFDAVCN266PB235BJIQTQ32FBXTIB5AZSX3B4R2VIK554AFSQ, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:30:55.762514255 (80h7m31.325837245s) [noise=true , address=104.36.82.14:6245 , piece_id=II36G46OA6TINLR244U6D7Q6LGV754UA5M62K3EX7ZOBESMJUEHA, seg_pos={0 0}, upload=64142]: Done uploading piece. err: ecclient: upload failed (node:1Zu8teNPTXn638grHKwtJqz4UEY7oFYLx1DHvuVP6RZXcatVqX, address:104.36.82.14:6245): protocol: storage node overloaded, request limit: 2000 Jun 22 00:30:55.773669583 (80h7m31.3369926s) [noise=true , address=79.127.205.235:5660 , piece_id=UGFPVS7VNB7U2YSFPSGYDIRZV5R2DHCN57PJP6VOXO52WIAYHLXA, seg_pos={0 0}, upload=64142]: Done uploading piece. err: ecclient: upload failed (node:12E6uCX9KL4Ud7YpxMXepvMXuqTFJCiSXByDjW94Am3U9H2Nf7f, address:79.127.205.235:5660): protocol: storage node overloaded, request limit: 2000 Jun 22 00:30:56.968366655 (80h7m32.53168964s) [noise=true , address=77.90.31.10:4533 , piece_id=SE6YAKKFDL3BV677CDDUC32UNMPXJHTG23GWWYK5B4YIPKER6GAQ, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:30:57.403572874 (80h7m32.966895883s) [noise=true , address=45.91.116.5:5310 , piece_id=AXJOKDZRPVNL7HFT4QKJWRAHFM2UCM3GEVQB2KNYD6CXB6TH4D4Q, seg_pos={0 0}, upload=64142]: Done uploading piece. err: ecclient: upload failed (node:127RMUc62xRq9RU5z3XfkZsq5sehmv3kP8o53tr9BxVcPE3GQCY, address:45.91.116.5:5310): protocol: storage node overloaded, request limit: 2000 Jun 22 00:30:58.210389994 (80h7m33.773712995s) [noise=true , address=104.36.82.13:6199 , piece_id=YP62N2TO446YR3QRG5YLXECD7TN6SJ4UNNLZSK5H4CMV6JIF4AKA, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:30:58.693037961 (80h7m34.256360966s) [noise=true , address=104.36.82.39:6756 , piece_id=OKKWMDJPZO6D5VRMGOE3GOCYGHM725JZTNOVYAWKJW3WYZPI4V5Q, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:30:59.124834814 (80h7m34.688157819s) [noise=true , address=104.36.82.15:6283 , piece_id=N7TH2EMIOPATVHCSNH2WXUGU7KMMEJ5LCHR22SWEHSA5OV4CUCXQ, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:31:00.629657982 (80h7m36.192980977s) [noise=true , address=104.36.82.42:6803 , piece_id=R626UFD326S2UBJEZ2WMHVR7EN4SK427VXPDJZOOZ32MX25JWZWQ, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:31:00.933110546 (80h7m36.496433536s) [noise=true , address=79.127.201.215:4326 , piece_id=USEWZDX6P52U6MCALTBNQG3SV7N2JRKNU6O23TLS6JEWAS4D7ARA, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:31:05.669167243 (80h7m41.232490237s) [noise=true , address=104.36.82.56:4261 , piece_id=EYOYOYLDACGMPGJYX6ILQT3XHMUTQNXZEDC7ANHLZXIKYVDQFFVQ, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:31:08.443582763 (80h7m44.006905763s) [noise=true , address=104.36.82.54:4184 , piece_id=IZICVGZFMAYLIJTNUJ2OFLZVNZ6IPPYDZO4BSTJJPLHVCVOE5EAA, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:31:26.459138014 (80h8m2.022461013s) [noise=true , address=104.36.82.53:4146 , piece_id=3V3DVGUOPJKDWIGQ2TU7M6VFYFVAS6ODY62ERGSU6LI3LSTWQ4YQ, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:31:27.456050336 (80h8m3.019373336s) [noise=true , address=104.36.82.52:4089 , piece_id=X2VG4S3UHVNWA4AQJS7FLKWNVLTOZ3AKWI5ZDH24QCA3ENQW7BXQ, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:31:55.270234251 (80h8m30.833557256s) [noise=true , address=104.36.82.11:6125 , piece_id=2CX7RJOKNQA6ODZZZXYPSCYAQY5P35TWX7I56CMYAPFLJE3JEO5Q, seg_pos={0 0}, upload=64142]: Done uploading piece. err: ecclient: upload failed (node:12vricofwx67FRaxAGLbmgT1TYyKkqfJbxdsyeEPL4R7kg8pKzT, address:104.36.82.11:6125): protocol: expected piece hash; context deadline exceeded; EOF Jun 22 00:31:55.270319418 (80h8m30.833642407s) [noise=true , address=104.36.82.55:4224 , piece_id=XF7H2BYSNNNZOS2T77VIN75N76AYUXAQEI6D4LRXACU2KU62V2TA, seg_pos={0 0}, upload=64142]: Done uploading piece. err: ecclient: upload failed (node:12EjKmUiv4pxkVCB4WoiXim4uFpxrBPyfiSS4efTY3HKSE3jb5y, address:104.36.82.55:4224): protocol: write tcp 159.223.143.39:39152->104.36.82.55:4224: use of closed network connection; write tcp 159.223.143.39:39152->104.36.82.55:4224: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:39152->104.36.82.55:4224: use of closed network connection Jun 22 00:31:55.270353811 (80h8m30.833676799s) [seg_pos={0 0}, upload=64142]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Jun 22 00:31:55.301099877 (80h8m30.864422884s) [upload=64142]: Done uploading segments... Jun 22 00:31:55.301121567 (80h8m30.864444566s) [upload=64142]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 46.20865663s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [.............X.........X...................X................] duration: 7.990247875s FAIL (2025-06-22 00:44:03 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 7.99026716s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [..............X.........X...................X...............] duration: 30.942158075s FAIL (2025-06-22 00:43:35 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.942193886s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [..........X.X..............................X................] duration: 15.743945482s FAIL (2025-06-22 00:43:29 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== ing upload Jun 22 00:30:55.303646184 (80h7m30.866969171s) [upload=64142]: Uploading segments... Jun 22 00:30:55.303706406 (80h7m30.867029416s) [upload=64142]: Got next segment. Inline: false Jun 22 00:30:55.303762479 (80h7m30.867085468s) [upload=64142]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 22 00:30:55.303768627 (80h7m30.867091636s) [upload=64142]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 22 00:30:55.349058140 (80h7m30.912381168s) [seg_pos={0 0}, upload=64142]: Begin upload segment... Jun 22 00:30:55.349113768 (80h7m30.912436772s) [seg_pos={0 0}, upload=64142]: Done begin upload segment. Jun 22 00:30:55.349143121 (80h7m30.912466131s) [noise=true , address=45.91.116.5:5310 , piece_id=AXJOKDZRPVNL7HFT4QKJWRAHFM2UCM3GEVQB2KNYD6CXB6TH4D4Q, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.349660917 (80h7m30.912983928s) [noise=true , address=77.90.31.10:4533 , piece_id=SE6YAKKFDL3BV677CDDUC32UNMPXJHTG23GWWYK5B4YIPKER6GAQ, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.350093077 (80h7m30.913416089s) [noise=true , address=79.127.201.215:4326 , piece_id=USEWZDX6P52U6MCALTBNQG3SV7N2JRKNU6O23TLS6JEWAS4D7ARA, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.350480894 (80h7m30.913803916s) [noise=true , address=104.36.82.13:6199 , piece_id=YP62N2TO446YR3QRG5YLXECD7TN6SJ4UNNLZSK5H4CMV6JIF4AKA, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.350669998 (80h7m30.913993011s) [noise=true , address=104.36.82.54:4184 , piece_id=IZICVGZFMAYLIJTNUJ2OFLZVNZ6IPPYDZO4BSTJJPLHVCVOE5EAA, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.351201352 (80h7m30.914524366s) [noise=true , address=104.36.82.55:4224 , piece_id=XF7H2BYSNNNZOS2T77VIN75N76AYUXAQEI6D4LRXACU2KU62V2TA, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.351668071 (80h7m30.914991083s) [noise=true , address=104.36.82.56:4261 , piece_id=EYOYOYLDACGMPGJYX6ILQT3XHMUTQNXZEDC7ANHLZXIKYVDQFFVQ, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.352193126 (80h7m30.915516138s) [noise=true , address=104.36.82.52:4089 , piece_id=X2VG4S3UHVNWA4AQJS7FLKWNVLTOZ3AKWI5ZDH24QCA3ENQW7BXQ, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.352587096 (80h7m30.915910103s) [noise=true , address=79.127.220.98:4402 , piece_id=TE2WQO5LK45AVMPIAIGVSIBOVZLC4FG4ESQ72X6KFYC7IAEN2DRA, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.353420253 (80h7m30.916743265s) [noise=true , address=104.36.82.11:6125 , piece_id=2CX7RJOKNQA6ODZZZXYPSCYAQY5P35TWX7I56CMYAPFLJE3JEO5Q, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.353791565 (80h7m30.917114578s) [noise=true , address=104.36.82.42:6803 , piece_id=R626UFD326S2UBJEZ2WMHVR7EN4SK427VXPDJZOOZ32MX25JWZWQ, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.354171226 (80h7m30.917494233s) [noise=true , address=104.36.82.14:6245 , piece_id=II36G46OA6TINLR244U6D7Q6LGV754UA5M62K3EX7ZOBESMJUEHA, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.354542282 (80h7m30.917865288s) [noise=true , address=79.127.205.235:5660 , piece_id=UGFPVS7VNB7U2YSFPSGYDIRZV5R2DHCN57PJP6VOXO52WIAYHLXA, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.354891655 (80h7m30.918214678s) [noise=true , address=104.36.82.15:6283 , piece_id=N7TH2EMIOPATVHCSNH2WXUGU7KMMEJ5LCHR22SWEHSA5OV4CUCXQ, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.355499503 (80h7m30.918822519s) [noise=true , address=77.90.31.14:4666 , piece_id=H4DTFDAVCN266PB235BJIQTQ32FBXTIB5AZSX3B4R2VIK554AFSQ, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.356575547 (80h7m30.919898539s) [noise=true , address=104.36.82.39:6756 , piece_id=OKKWMDJPZO6D5VRMGOE3GOCYGHM725JZTNOVYAWKJW3WYZPI4V5Q, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.357039590 (80h7m30.920362596s) [noise=true , address=104.36.82.53:4146 , piece_id=3V3DVGUOPJKDWIGQ2TU7M6VFYFVAS6ODY62ERGSU6LI3LSTWQ4YQ, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.360343566 (80h7m30.923666582s) [noise=true , address=104.36.82.9:6033 , piece_id=T5XA23LZELMSTWOLBSHOX4AHKDACF65OYCKTMZYTXDIBSXNA2T4A, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.452444064 (80h7m31.015767055s) [upload=64142]: Next returned nil segment Jun 22 00:30:55.452495148 (80h7m31.015818149s) [upload=64142]: Waiting for error group managing segments... Jun 22 00:30:55.532958808 (80h7m31.096281883s) [noise=true , address=104.36.82.9:6033 , piece_id=T5XA23LZELMSTWOLBSHOX4AHKDACF65OYCKTMZYTXDIBSXNA2T4A, seg_pos={0 0}, upload=64142]: Done uploading piece. err: ecclient: upload failed (node:1SV3LrcQW16wLewDzfyUWFB6sUgnERNJVdhz72CJmyde95M79C, address:104.36.82.9:6033): protocol: storage node overloaded, request limit: 2000 Jun 22 00:30:55.547802184 (80h7m31.111125188s) [noise=true , address=79.127.220.98:4402 , piece_id=TE2WQO5LK45AVMPIAIGVSIBOVZLC4FG4ESQ72X6KFYC7IAEN2DRA, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:30:55.554223621 (80h7m31.117546616s) [noise=true , address=77.90.31.14:4666 , piece_id=H4DTFDAVCN266PB235BJIQTQ32FBXTIB5AZSX3B4R2VIK554AFSQ, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:30:55.762514255 (80h7m31.325837245s) [noise=true , address=104.36.82.14:6245 , piece_id=II36G46OA6TINLR244U6D7Q6LGV754UA5M62K3EX7ZOBESMJUEHA, seg_pos={0 0}, upload=64142]: Done uploading piece. err: ecclient: upload failed (node:1Zu8teNPTXn638grHKwtJqz4UEY7oFYLx1DHvuVP6RZXcatVqX, address:104.36.82.14:6245): protocol: storage node overloaded, request limit: 2000 Jun 22 00:30:55.773669583 (80h7m31.3369926s) [noise=true , address=79.127.205.235:5660 , piece_id=UGFPVS7VNB7U2YSFPSGYDIRZV5R2DHCN57PJP6VOXO52WIAYHLXA, seg_pos={0 0}, upload=64142]: Done uploading piece. err: ecclient: upload failed (node:12E6uCX9KL4Ud7YpxMXepvMXuqTFJCiSXByDjW94Am3U9H2Nf7f, address:79.127.205.235:5660): protocol: storage node overloaded, request limit: 2000 Jun 22 00:30:56.968366655 (80h7m32.53168964s) [noise=true , address=77.90.31.10:4533 , piece_id=SE6YAKKFDL3BV677CDDUC32UNMPXJHTG23GWWYK5B4YIPKER6GAQ, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:30:57.403572874 (80h7m32.966895883s) [noise=true , address=45.91.116.5:5310 , piece_id=AXJOKDZRPVNL7HFT4QKJWRAHFM2UCM3GEVQB2KNYD6CXB6TH4D4Q, seg_pos={0 0}, upload=64142]: Done uploading piece. err: ecclient: upload failed (node:127RMUc62xRq9RU5z3XfkZsq5sehmv3kP8o53tr9BxVcPE3GQCY, address:45.91.116.5:5310): protocol: storage node overloaded, request limit: 2000 Jun 22 00:30:58.210389994 (80h7m33.773712995s) [noise=true , address=104.36.82.13:6199 , piece_id=YP62N2TO446YR3QRG5YLXECD7TN6SJ4UNNLZSK5H4CMV6JIF4AKA, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:30:58.693037961 (80h7m34.256360966s) [noise=true , address=104.36.82.39:6756 , piece_id=OKKWMDJPZO6D5VRMGOE3GOCYGHM725JZTNOVYAWKJW3WYZPI4V5Q, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:30:59.124834814 (80h7m34.688157819s) [noise=true , address=104.36.82.15:6283 , piece_id=N7TH2EMIOPATVHCSNH2WXUGU7KMMEJ5LCHR22SWEHSA5OV4CUCXQ, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:31:00.629657982 (80h7m36.192980977s) [noise=true , address=104.36.82.42:6803 , piece_id=R626UFD326S2UBJEZ2WMHVR7EN4SK427VXPDJZOOZ32MX25JWZWQ, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:31:00.933110546 (80h7m36.496433536s) [noise=true , address=79.127.201.215:4326 , piece_id=USEWZDX6P52U6MCALTBNQG3SV7N2JRKNU6O23TLS6JEWAS4D7ARA, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:31:05.669167243 (80h7m41.232490237s) [noise=true , address=104.36.82.56:4261 , piece_id=EYOYOYLDACGMPGJYX6ILQT3XHMUTQNXZEDC7ANHLZXIKYVDQFFVQ, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:31:08.443582763 (80h7m44.006905763s) [noise=true , address=104.36.82.54:4184 , piece_id=IZICVGZFMAYLIJTNUJ2OFLZVNZ6IPPYDZO4BSTJJPLHVCVOE5EAA, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:31:26.459138014 (80h8m2.022461013s) [noise=true , address=104.36.82.53:4146 , piece_id=3V3DVGUOPJKDWIGQ2TU7M6VFYFVAS6ODY62ERGSU6LI3LSTWQ4YQ, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:31:27.456050336 (80h8m3.019373336s) [noise=true , address=104.36.82.52:4089 , piece_id=X2VG4S3UHVNWA4AQJS7FLKWNVLTOZ3AKWI5ZDH24QCA3ENQW7BXQ, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:31:55.270234251 (80h8m30.833557256s) [noise=true , address=104.36.82.11:6125 , piece_id=2CX7RJOKNQA6ODZZZXYPSCYAQY5P35TWX7I56CMYAPFLJE3JEO5Q, seg_pos={0 0}, upload=64142]: Done uploading piece. err: ecclient: upload failed (node:12vricofwx67FRaxAGLbmgT1TYyKkqfJbxdsyeEPL4R7kg8pKzT, address:104.36.82.11:6125): protocol: expected piece hash; context deadline exceeded; EOF Jun 22 00:31:55.270319418 (80h8m30.833642407s) [noise=true , address=104.36.82.55:4224 , piece_id=XF7H2BYSNNNZOS2T77VIN75N76AYUXAQEI6D4LRXACU2KU62V2TA, seg_pos={0 0}, upload=64142]: Done uploading piece. err: ecclient: upload failed (node:12EjKmUiv4pxkVCB4WoiXim4uFpxrBPyfiSS4efTY3HKSE3jb5y, address:104.36.82.55:4224): protocol: write tcp 159.223.143.39:39152->104.36.82.55:4224: use of closed network connection; write tcp 159.223.143.39:39152->104.36.82.55:4224: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:39152->104.36.82.55:4224: use of closed network connection Jun 22 00:31:55.270353811 (80h8m30.833676799s) [seg_pos={0 0}, upload=64142]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Jun 22 00:31:55.301099877 (80h8m30.864422884s) [upload=64142]: Done uploading segments... Jun 22 00:31:55.301121567 (80h8m30.864444566s) [upload=64142]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 15.743974652s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [...............X.........X...................X..............] duration: 21.90592926s FAIL (2025-06-22 00:42:28 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 21.905997231s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [...........X.X..............................X...............] duration: 19.651563164s FAIL (2025-06-22 00:42:27 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== ing upload Jun 22 00:30:55.303646184 (80h7m30.866969171s) [upload=64142]: Uploading segments... Jun 22 00:30:55.303706406 (80h7m30.867029416s) [upload=64142]: Got next segment. Inline: false Jun 22 00:30:55.303762479 (80h7m30.867085468s) [upload=64142]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 22 00:30:55.303768627 (80h7m30.867091636s) [upload=64142]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 22 00:30:55.349058140 (80h7m30.912381168s) [seg_pos={0 0}, upload=64142]: Begin upload segment... Jun 22 00:30:55.349113768 (80h7m30.912436772s) [seg_pos={0 0}, upload=64142]: Done begin upload segment. Jun 22 00:30:55.349143121 (80h7m30.912466131s) [noise=true , address=45.91.116.5:5310 , piece_id=AXJOKDZRPVNL7HFT4QKJWRAHFM2UCM3GEVQB2KNYD6CXB6TH4D4Q, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.349660917 (80h7m30.912983928s) [noise=true , address=77.90.31.10:4533 , piece_id=SE6YAKKFDL3BV677CDDUC32UNMPXJHTG23GWWYK5B4YIPKER6GAQ, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.350093077 (80h7m30.913416089s) [noise=true , address=79.127.201.215:4326 , piece_id=USEWZDX6P52U6MCALTBNQG3SV7N2JRKNU6O23TLS6JEWAS4D7ARA, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.350480894 (80h7m30.913803916s) [noise=true , address=104.36.82.13:6199 , piece_id=YP62N2TO446YR3QRG5YLXECD7TN6SJ4UNNLZSK5H4CMV6JIF4AKA, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.350669998 (80h7m30.913993011s) [noise=true , address=104.36.82.54:4184 , piece_id=IZICVGZFMAYLIJTNUJ2OFLZVNZ6IPPYDZO4BSTJJPLHVCVOE5EAA, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.351201352 (80h7m30.914524366s) [noise=true , address=104.36.82.55:4224 , piece_id=XF7H2BYSNNNZOS2T77VIN75N76AYUXAQEI6D4LRXACU2KU62V2TA, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.351668071 (80h7m30.914991083s) [noise=true , address=104.36.82.56:4261 , piece_id=EYOYOYLDACGMPGJYX6ILQT3XHMUTQNXZEDC7ANHLZXIKYVDQFFVQ, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.352193126 (80h7m30.915516138s) [noise=true , address=104.36.82.52:4089 , piece_id=X2VG4S3UHVNWA4AQJS7FLKWNVLTOZ3AKWI5ZDH24QCA3ENQW7BXQ, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.352587096 (80h7m30.915910103s) [noise=true , address=79.127.220.98:4402 , piece_id=TE2WQO5LK45AVMPIAIGVSIBOVZLC4FG4ESQ72X6KFYC7IAEN2DRA, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.353420253 (80h7m30.916743265s) [noise=true , address=104.36.82.11:6125 , piece_id=2CX7RJOKNQA6ODZZZXYPSCYAQY5P35TWX7I56CMYAPFLJE3JEO5Q, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.353791565 (80h7m30.917114578s) [noise=true , address=104.36.82.42:6803 , piece_id=R626UFD326S2UBJEZ2WMHVR7EN4SK427VXPDJZOOZ32MX25JWZWQ, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.354171226 (80h7m30.917494233s) [noise=true , address=104.36.82.14:6245 , piece_id=II36G46OA6TINLR244U6D7Q6LGV754UA5M62K3EX7ZOBESMJUEHA, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.354542282 (80h7m30.917865288s) [noise=true , address=79.127.205.235:5660 , piece_id=UGFPVS7VNB7U2YSFPSGYDIRZV5R2DHCN57PJP6VOXO52WIAYHLXA, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.354891655 (80h7m30.918214678s) [noise=true , address=104.36.82.15:6283 , piece_id=N7TH2EMIOPATVHCSNH2WXUGU7KMMEJ5LCHR22SWEHSA5OV4CUCXQ, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.355499503 (80h7m30.918822519s) [noise=true , address=77.90.31.14:4666 , piece_id=H4DTFDAVCN266PB235BJIQTQ32FBXTIB5AZSX3B4R2VIK554AFSQ, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.356575547 (80h7m30.919898539s) [noise=true , address=104.36.82.39:6756 , piece_id=OKKWMDJPZO6D5VRMGOE3GOCYGHM725JZTNOVYAWKJW3WYZPI4V5Q, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.357039590 (80h7m30.920362596s) [noise=true , address=104.36.82.53:4146 , piece_id=3V3DVGUOPJKDWIGQ2TU7M6VFYFVAS6ODY62ERGSU6LI3LSTWQ4YQ, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.360343566 (80h7m30.923666582s) [noise=true , address=104.36.82.9:6033 , piece_id=T5XA23LZELMSTWOLBSHOX4AHKDACF65OYCKTMZYTXDIBSXNA2T4A, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.452444064 (80h7m31.015767055s) [upload=64142]: Next returned nil segment Jun 22 00:30:55.452495148 (80h7m31.015818149s) [upload=64142]: Waiting for error group managing segments... Jun 22 00:30:55.532958808 (80h7m31.096281883s) [noise=true , address=104.36.82.9:6033 , piece_id=T5XA23LZELMSTWOLBSHOX4AHKDACF65OYCKTMZYTXDIBSXNA2T4A, seg_pos={0 0}, upload=64142]: Done uploading piece. err: ecclient: upload failed (node:1SV3LrcQW16wLewDzfyUWFB6sUgnERNJVdhz72CJmyde95M79C, address:104.36.82.9:6033): protocol: storage node overloaded, request limit: 2000 Jun 22 00:30:55.547802184 (80h7m31.111125188s) [noise=true , address=79.127.220.98:4402 , piece_id=TE2WQO5LK45AVMPIAIGVSIBOVZLC4FG4ESQ72X6KFYC7IAEN2DRA, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:30:55.554223621 (80h7m31.117546616s) [noise=true , address=77.90.31.14:4666 , piece_id=H4DTFDAVCN266PB235BJIQTQ32FBXTIB5AZSX3B4R2VIK554AFSQ, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:30:55.762514255 (80h7m31.325837245s) [noise=true , address=104.36.82.14:6245 , piece_id=II36G46OA6TINLR244U6D7Q6LGV754UA5M62K3EX7ZOBESMJUEHA, seg_pos={0 0}, upload=64142]: Done uploading piece. err: ecclient: upload failed (node:1Zu8teNPTXn638grHKwtJqz4UEY7oFYLx1DHvuVP6RZXcatVqX, address:104.36.82.14:6245): protocol: storage node overloaded, request limit: 2000 Jun 22 00:30:55.773669583 (80h7m31.3369926s) [noise=true , address=79.127.205.235:5660 , piece_id=UGFPVS7VNB7U2YSFPSGYDIRZV5R2DHCN57PJP6VOXO52WIAYHLXA, seg_pos={0 0}, upload=64142]: Done uploading piece. err: ecclient: upload failed (node:12E6uCX9KL4Ud7YpxMXepvMXuqTFJCiSXByDjW94Am3U9H2Nf7f, address:79.127.205.235:5660): protocol: storage node overloaded, request limit: 2000 Jun 22 00:30:56.968366655 (80h7m32.53168964s) [noise=true , address=77.90.31.10:4533 , piece_id=SE6YAKKFDL3BV677CDDUC32UNMPXJHTG23GWWYK5B4YIPKER6GAQ, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:30:57.403572874 (80h7m32.966895883s) [noise=true , address=45.91.116.5:5310 , piece_id=AXJOKDZRPVNL7HFT4QKJWRAHFM2UCM3GEVQB2KNYD6CXB6TH4D4Q, seg_pos={0 0}, upload=64142]: Done uploading piece. err: ecclient: upload failed (node:127RMUc62xRq9RU5z3XfkZsq5sehmv3kP8o53tr9BxVcPE3GQCY, address:45.91.116.5:5310): protocol: storage node overloaded, request limit: 2000 Jun 22 00:30:58.210389994 (80h7m33.773712995s) [noise=true , address=104.36.82.13:6199 , piece_id=YP62N2TO446YR3QRG5YLXECD7TN6SJ4UNNLZSK5H4CMV6JIF4AKA, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:30:58.693037961 (80h7m34.256360966s) [noise=true , address=104.36.82.39:6756 , piece_id=OKKWMDJPZO6D5VRMGOE3GOCYGHM725JZTNOVYAWKJW3WYZPI4V5Q, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:30:59.124834814 (80h7m34.688157819s) [noise=true , address=104.36.82.15:6283 , piece_id=N7TH2EMIOPATVHCSNH2WXUGU7KMMEJ5LCHR22SWEHSA5OV4CUCXQ, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:31:00.629657982 (80h7m36.192980977s) [noise=true , address=104.36.82.42:6803 , piece_id=R626UFD326S2UBJEZ2WMHVR7EN4SK427VXPDJZOOZ32MX25JWZWQ, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:31:00.933110546 (80h7m36.496433536s) [noise=true , address=79.127.201.215:4326 , piece_id=USEWZDX6P52U6MCALTBNQG3SV7N2JRKNU6O23TLS6JEWAS4D7ARA, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:31:05.669167243 (80h7m41.232490237s) [noise=true , address=104.36.82.56:4261 , piece_id=EYOYOYLDACGMPGJYX6ILQT3XHMUTQNXZEDC7ANHLZXIKYVDQFFVQ, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:31:08.443582763 (80h7m44.006905763s) [noise=true , address=104.36.82.54:4184 , piece_id=IZICVGZFMAYLIJTNUJ2OFLZVNZ6IPPYDZO4BSTJJPLHVCVOE5EAA, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:31:26.459138014 (80h8m2.022461013s) [noise=true , address=104.36.82.53:4146 , piece_id=3V3DVGUOPJKDWIGQ2TU7M6VFYFVAS6ODY62ERGSU6LI3LSTWQ4YQ, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:31:27.456050336 (80h8m3.019373336s) [noise=true , address=104.36.82.52:4089 , piece_id=X2VG4S3UHVNWA4AQJS7FLKWNVLTOZ3AKWI5ZDH24QCA3ENQW7BXQ, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:31:55.270234251 (80h8m30.833557256s) [noise=true , address=104.36.82.11:6125 , piece_id=2CX7RJOKNQA6ODZZZXYPSCYAQY5P35TWX7I56CMYAPFLJE3JEO5Q, seg_pos={0 0}, upload=64142]: Done uploading piece. err: ecclient: upload failed (node:12vricofwx67FRaxAGLbmgT1TYyKkqfJbxdsyeEPL4R7kg8pKzT, address:104.36.82.11:6125): protocol: expected piece hash; context deadline exceeded; EOF Jun 22 00:31:55.270319418 (80h8m30.833642407s) [noise=true , address=104.36.82.55:4224 , piece_id=XF7H2BYSNNNZOS2T77VIN75N76AYUXAQEI6D4LRXACU2KU62V2TA, seg_pos={0 0}, upload=64142]: Done uploading piece. err: ecclient: upload failed (node:12EjKmUiv4pxkVCB4WoiXim4uFpxrBPyfiSS4efTY3HKSE3jb5y, address:104.36.82.55:4224): protocol: write tcp 159.223.143.39:39152->104.36.82.55:4224: use of closed network connection; write tcp 159.223.143.39:39152->104.36.82.55:4224: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:39152->104.36.82.55:4224: use of closed network connection Jun 22 00:31:55.270353811 (80h8m30.833676799s) [seg_pos={0 0}, upload=64142]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Jun 22 00:31:55.301099877 (80h8m30.864422884s) [upload=64142]: Done uploading segments... Jun 22 00:31:55.301121567 (80h8m30.864444566s) [upload=64142]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 19.651590199s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [............X.X..............................X..............] duration: 18.635829736s FAIL (2025-06-22 00:41:43 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== ing upload Jun 22 00:30:55.303646184 (80h7m30.866969171s) [upload=64142]: Uploading segments... Jun 22 00:30:55.303706406 (80h7m30.867029416s) [upload=64142]: Got next segment. Inline: false Jun 22 00:30:55.303762479 (80h7m30.867085468s) [upload=64142]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 22 00:30:55.303768627 (80h7m30.867091636s) [upload=64142]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 22 00:30:55.349058140 (80h7m30.912381168s) [seg_pos={0 0}, upload=64142]: Begin upload segment... Jun 22 00:30:55.349113768 (80h7m30.912436772s) [seg_pos={0 0}, upload=64142]: Done begin upload segment. Jun 22 00:30:55.349143121 (80h7m30.912466131s) [noise=true , address=45.91.116.5:5310 , piece_id=AXJOKDZRPVNL7HFT4QKJWRAHFM2UCM3GEVQB2KNYD6CXB6TH4D4Q, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.349660917 (80h7m30.912983928s) [noise=true , address=77.90.31.10:4533 , piece_id=SE6YAKKFDL3BV677CDDUC32UNMPXJHTG23GWWYK5B4YIPKER6GAQ, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.350093077 (80h7m30.913416089s) [noise=true , address=79.127.201.215:4326 , piece_id=USEWZDX6P52U6MCALTBNQG3SV7N2JRKNU6O23TLS6JEWAS4D7ARA, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.350480894 (80h7m30.913803916s) [noise=true , address=104.36.82.13:6199 , piece_id=YP62N2TO446YR3QRG5YLXECD7TN6SJ4UNNLZSK5H4CMV6JIF4AKA, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.350669998 (80h7m30.913993011s) [noise=true , address=104.36.82.54:4184 , piece_id=IZICVGZFMAYLIJTNUJ2OFLZVNZ6IPPYDZO4BSTJJPLHVCVOE5EAA, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.351201352 (80h7m30.914524366s) [noise=true , address=104.36.82.55:4224 , piece_id=XF7H2BYSNNNZOS2T77VIN75N76AYUXAQEI6D4LRXACU2KU62V2TA, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.351668071 (80h7m30.914991083s) [noise=true , address=104.36.82.56:4261 , piece_id=EYOYOYLDACGMPGJYX6ILQT3XHMUTQNXZEDC7ANHLZXIKYVDQFFVQ, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.352193126 (80h7m30.915516138s) [noise=true , address=104.36.82.52:4089 , piece_id=X2VG4S3UHVNWA4AQJS7FLKWNVLTOZ3AKWI5ZDH24QCA3ENQW7BXQ, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.352587096 (80h7m30.915910103s) [noise=true , address=79.127.220.98:4402 , piece_id=TE2WQO5LK45AVMPIAIGVSIBOVZLC4FG4ESQ72X6KFYC7IAEN2DRA, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.353420253 (80h7m30.916743265s) [noise=true , address=104.36.82.11:6125 , piece_id=2CX7RJOKNQA6ODZZZXYPSCYAQY5P35TWX7I56CMYAPFLJE3JEO5Q, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.353791565 (80h7m30.917114578s) [noise=true , address=104.36.82.42:6803 , piece_id=R626UFD326S2UBJEZ2WMHVR7EN4SK427VXPDJZOOZ32MX25JWZWQ, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.354171226 (80h7m30.917494233s) [noise=true , address=104.36.82.14:6245 , piece_id=II36G46OA6TINLR244U6D7Q6LGV754UA5M62K3EX7ZOBESMJUEHA, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.354542282 (80h7m30.917865288s) [noise=true , address=79.127.205.235:5660 , piece_id=UGFPVS7VNB7U2YSFPSGYDIRZV5R2DHCN57PJP6VOXO52WIAYHLXA, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.354891655 (80h7m30.918214678s) [noise=true , address=104.36.82.15:6283 , piece_id=N7TH2EMIOPATVHCSNH2WXUGU7KMMEJ5LCHR22SWEHSA5OV4CUCXQ, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.355499503 (80h7m30.918822519s) [noise=true , address=77.90.31.14:4666 , piece_id=H4DTFDAVCN266PB235BJIQTQ32FBXTIB5AZSX3B4R2VIK554AFSQ, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.356575547 (80h7m30.919898539s) [noise=true , address=104.36.82.39:6756 , piece_id=OKKWMDJPZO6D5VRMGOE3GOCYGHM725JZTNOVYAWKJW3WYZPI4V5Q, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.357039590 (80h7m30.920362596s) [noise=true , address=104.36.82.53:4146 , piece_id=3V3DVGUOPJKDWIGQ2TU7M6VFYFVAS6ODY62ERGSU6LI3LSTWQ4YQ, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.360343566 (80h7m30.923666582s) [noise=true , address=104.36.82.9:6033 , piece_id=T5XA23LZELMSTWOLBSHOX4AHKDACF65OYCKTMZYTXDIBSXNA2T4A, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.452444064 (80h7m31.015767055s) [upload=64142]: Next returned nil segment Jun 22 00:30:55.452495148 (80h7m31.015818149s) [upload=64142]: Waiting for error group managing segments... Jun 22 00:30:55.532958808 (80h7m31.096281883s) [noise=true , address=104.36.82.9:6033 , piece_id=T5XA23LZELMSTWOLBSHOX4AHKDACF65OYCKTMZYTXDIBSXNA2T4A, seg_pos={0 0}, upload=64142]: Done uploading piece. err: ecclient: upload failed (node:1SV3LrcQW16wLewDzfyUWFB6sUgnERNJVdhz72CJmyde95M79C, address:104.36.82.9:6033): protocol: storage node overloaded, request limit: 2000 Jun 22 00:30:55.547802184 (80h7m31.111125188s) [noise=true , address=79.127.220.98:4402 , piece_id=TE2WQO5LK45AVMPIAIGVSIBOVZLC4FG4ESQ72X6KFYC7IAEN2DRA, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:30:55.554223621 (80h7m31.117546616s) [noise=true , address=77.90.31.14:4666 , piece_id=H4DTFDAVCN266PB235BJIQTQ32FBXTIB5AZSX3B4R2VIK554AFSQ, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:30:55.762514255 (80h7m31.325837245s) [noise=true , address=104.36.82.14:6245 , piece_id=II36G46OA6TINLR244U6D7Q6LGV754UA5M62K3EX7ZOBESMJUEHA, seg_pos={0 0}, upload=64142]: Done uploading piece. err: ecclient: upload failed (node:1Zu8teNPTXn638grHKwtJqz4UEY7oFYLx1DHvuVP6RZXcatVqX, address:104.36.82.14:6245): protocol: storage node overloaded, request limit: 2000 Jun 22 00:30:55.773669583 (80h7m31.3369926s) [noise=true , address=79.127.205.235:5660 , piece_id=UGFPVS7VNB7U2YSFPSGYDIRZV5R2DHCN57PJP6VOXO52WIAYHLXA, seg_pos={0 0}, upload=64142]: Done uploading piece. err: ecclient: upload failed (node:12E6uCX9KL4Ud7YpxMXepvMXuqTFJCiSXByDjW94Am3U9H2Nf7f, address:79.127.205.235:5660): protocol: storage node overloaded, request limit: 2000 Jun 22 00:30:56.968366655 (80h7m32.53168964s) [noise=true , address=77.90.31.10:4533 , piece_id=SE6YAKKFDL3BV677CDDUC32UNMPXJHTG23GWWYK5B4YIPKER6GAQ, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:30:57.403572874 (80h7m32.966895883s) [noise=true , address=45.91.116.5:5310 , piece_id=AXJOKDZRPVNL7HFT4QKJWRAHFM2UCM3GEVQB2KNYD6CXB6TH4D4Q, seg_pos={0 0}, upload=64142]: Done uploading piece. err: ecclient: upload failed (node:127RMUc62xRq9RU5z3XfkZsq5sehmv3kP8o53tr9BxVcPE3GQCY, address:45.91.116.5:5310): protocol: storage node overloaded, request limit: 2000 Jun 22 00:30:58.210389994 (80h7m33.773712995s) [noise=true , address=104.36.82.13:6199 , piece_id=YP62N2TO446YR3QRG5YLXECD7TN6SJ4UNNLZSK5H4CMV6JIF4AKA, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:30:58.693037961 (80h7m34.256360966s) [noise=true , address=104.36.82.39:6756 , piece_id=OKKWMDJPZO6D5VRMGOE3GOCYGHM725JZTNOVYAWKJW3WYZPI4V5Q, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:30:59.124834814 (80h7m34.688157819s) [noise=true , address=104.36.82.15:6283 , piece_id=N7TH2EMIOPATVHCSNH2WXUGU7KMMEJ5LCHR22SWEHSA5OV4CUCXQ, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:31:00.629657982 (80h7m36.192980977s) [noise=true , address=104.36.82.42:6803 , piece_id=R626UFD326S2UBJEZ2WMHVR7EN4SK427VXPDJZOOZ32MX25JWZWQ, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:31:00.933110546 (80h7m36.496433536s) [noise=true , address=79.127.201.215:4326 , piece_id=USEWZDX6P52U6MCALTBNQG3SV7N2JRKNU6O23TLS6JEWAS4D7ARA, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:31:05.669167243 (80h7m41.232490237s) [noise=true , address=104.36.82.56:4261 , piece_id=EYOYOYLDACGMPGJYX6ILQT3XHMUTQNXZEDC7ANHLZXIKYVDQFFVQ, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:31:08.443582763 (80h7m44.006905763s) [noise=true , address=104.36.82.54:4184 , piece_id=IZICVGZFMAYLIJTNUJ2OFLZVNZ6IPPYDZO4BSTJJPLHVCVOE5EAA, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:31:26.459138014 (80h8m2.022461013s) [noise=true , address=104.36.82.53:4146 , piece_id=3V3DVGUOPJKDWIGQ2TU7M6VFYFVAS6ODY62ERGSU6LI3LSTWQ4YQ, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:31:27.456050336 (80h8m3.019373336s) [noise=true , address=104.36.82.52:4089 , piece_id=X2VG4S3UHVNWA4AQJS7FLKWNVLTOZ3AKWI5ZDH24QCA3ENQW7BXQ, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:31:55.270234251 (80h8m30.833557256s) [noise=true , address=104.36.82.11:6125 , piece_id=2CX7RJOKNQA6ODZZZXYPSCYAQY5P35TWX7I56CMYAPFLJE3JEO5Q, seg_pos={0 0}, upload=64142]: Done uploading piece. err: ecclient: upload failed (node:12vricofwx67FRaxAGLbmgT1TYyKkqfJbxdsyeEPL4R7kg8pKzT, address:104.36.82.11:6125): protocol: expected piece hash; context deadline exceeded; EOF Jun 22 00:31:55.270319418 (80h8m30.833642407s) [noise=true , address=104.36.82.55:4224 , piece_id=XF7H2BYSNNNZOS2T77VIN75N76AYUXAQEI6D4LRXACU2KU62V2TA, seg_pos={0 0}, upload=64142]: Done uploading piece. err: ecclient: upload failed (node:12EjKmUiv4pxkVCB4WoiXim4uFpxrBPyfiSS4efTY3HKSE3jb5y, address:104.36.82.55:4224): protocol: write tcp 159.223.143.39:39152->104.36.82.55:4224: use of closed network connection; write tcp 159.223.143.39:39152->104.36.82.55:4224: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:39152->104.36.82.55:4224: use of closed network connection Jun 22 00:31:55.270353811 (80h8m30.833676799s) [seg_pos={0 0}, upload=64142]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Jun 22 00:31:55.301099877 (80h8m30.864422884s) [upload=64142]: Done uploading segments... Jun 22 00:31:55.301121567 (80h8m30.864444566s) [upload=64142]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 18.635855491s core uplink us-select-1 us1-placement-20

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)