Orbiter History

checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [................X.........X...................X.............] duration: 34.774792313s FAIL (2025-06-22 00:41:38 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 34.774820573s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [.............X.X..............................X.............] duration: 26.173937439s FAIL (2025-06-22 00:41:06 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== ing upload Jun 22 00:30:55.303646184 (80h7m30.866969171s) [upload=64142]: Uploading segments... Jun 22 00:30:55.303706406 (80h7m30.867029416s) [upload=64142]: Got next segment. Inline: false Jun 22 00:30:55.303762479 (80h7m30.867085468s) [upload=64142]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 22 00:30:55.303768627 (80h7m30.867091636s) [upload=64142]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 22 00:30:55.349058140 (80h7m30.912381168s) [seg_pos={0 0}, upload=64142]: Begin upload segment... Jun 22 00:30:55.349113768 (80h7m30.912436772s) [seg_pos={0 0}, upload=64142]: Done begin upload segment. Jun 22 00:30:55.349143121 (80h7m30.912466131s) [noise=true , address=45.91.116.5:5310 , piece_id=AXJOKDZRPVNL7HFT4QKJWRAHFM2UCM3GEVQB2KNYD6CXB6TH4D4Q, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.349660917 (80h7m30.912983928s) [noise=true , address=77.90.31.10:4533 , piece_id=SE6YAKKFDL3BV677CDDUC32UNMPXJHTG23GWWYK5B4YIPKER6GAQ, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.350093077 (80h7m30.913416089s) [noise=true , address=79.127.201.215:4326 , piece_id=USEWZDX6P52U6MCALTBNQG3SV7N2JRKNU6O23TLS6JEWAS4D7ARA, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.350480894 (80h7m30.913803916s) [noise=true , address=104.36.82.13:6199 , piece_id=YP62N2TO446YR3QRG5YLXECD7TN6SJ4UNNLZSK5H4CMV6JIF4AKA, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.350669998 (80h7m30.913993011s) [noise=true , address=104.36.82.54:4184 , piece_id=IZICVGZFMAYLIJTNUJ2OFLZVNZ6IPPYDZO4BSTJJPLHVCVOE5EAA, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.351201352 (80h7m30.914524366s) [noise=true , address=104.36.82.55:4224 , piece_id=XF7H2BYSNNNZOS2T77VIN75N76AYUXAQEI6D4LRXACU2KU62V2TA, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.351668071 (80h7m30.914991083s) [noise=true , address=104.36.82.56:4261 , piece_id=EYOYOYLDACGMPGJYX6ILQT3XHMUTQNXZEDC7ANHLZXIKYVDQFFVQ, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.352193126 (80h7m30.915516138s) [noise=true , address=104.36.82.52:4089 , piece_id=X2VG4S3UHVNWA4AQJS7FLKWNVLTOZ3AKWI5ZDH24QCA3ENQW7BXQ, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.352587096 (80h7m30.915910103s) [noise=true , address=79.127.220.98:4402 , piece_id=TE2WQO5LK45AVMPIAIGVSIBOVZLC4FG4ESQ72X6KFYC7IAEN2DRA, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.353420253 (80h7m30.916743265s) [noise=true , address=104.36.82.11:6125 , piece_id=2CX7RJOKNQA6ODZZZXYPSCYAQY5P35TWX7I56CMYAPFLJE3JEO5Q, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.353791565 (80h7m30.917114578s) [noise=true , address=104.36.82.42:6803 , piece_id=R626UFD326S2UBJEZ2WMHVR7EN4SK427VXPDJZOOZ32MX25JWZWQ, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.354171226 (80h7m30.917494233s) [noise=true , address=104.36.82.14:6245 , piece_id=II36G46OA6TINLR244U6D7Q6LGV754UA5M62K3EX7ZOBESMJUEHA, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.354542282 (80h7m30.917865288s) [noise=true , address=79.127.205.235:5660 , piece_id=UGFPVS7VNB7U2YSFPSGYDIRZV5R2DHCN57PJP6VOXO52WIAYHLXA, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.354891655 (80h7m30.918214678s) [noise=true , address=104.36.82.15:6283 , piece_id=N7TH2EMIOPATVHCSNH2WXUGU7KMMEJ5LCHR22SWEHSA5OV4CUCXQ, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.355499503 (80h7m30.918822519s) [noise=true , address=77.90.31.14:4666 , piece_id=H4DTFDAVCN266PB235BJIQTQ32FBXTIB5AZSX3B4R2VIK554AFSQ, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.356575547 (80h7m30.919898539s) [noise=true , address=104.36.82.39:6756 , piece_id=OKKWMDJPZO6D5VRMGOE3GOCYGHM725JZTNOVYAWKJW3WYZPI4V5Q, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.357039590 (80h7m30.920362596s) [noise=true , address=104.36.82.53:4146 , piece_id=3V3DVGUOPJKDWIGQ2TU7M6VFYFVAS6ODY62ERGSU6LI3LSTWQ4YQ, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.360343566 (80h7m30.923666582s) [noise=true , address=104.36.82.9:6033 , piece_id=T5XA23LZELMSTWOLBSHOX4AHKDACF65OYCKTMZYTXDIBSXNA2T4A, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.452444064 (80h7m31.015767055s) [upload=64142]: Next returned nil segment Jun 22 00:30:55.452495148 (80h7m31.015818149s) [upload=64142]: Waiting for error group managing segments... Jun 22 00:30:55.532958808 (80h7m31.096281883s) [noise=true , address=104.36.82.9:6033 , piece_id=T5XA23LZELMSTWOLBSHOX4AHKDACF65OYCKTMZYTXDIBSXNA2T4A, seg_pos={0 0}, upload=64142]: Done uploading piece. err: ecclient: upload failed (node:1SV3LrcQW16wLewDzfyUWFB6sUgnERNJVdhz72CJmyde95M79C, address:104.36.82.9:6033): protocol: storage node overloaded, request limit: 2000 Jun 22 00:30:55.547802184 (80h7m31.111125188s) [noise=true , address=79.127.220.98:4402 , piece_id=TE2WQO5LK45AVMPIAIGVSIBOVZLC4FG4ESQ72X6KFYC7IAEN2DRA, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:30:55.554223621 (80h7m31.117546616s) [noise=true , address=77.90.31.14:4666 , piece_id=H4DTFDAVCN266PB235BJIQTQ32FBXTIB5AZSX3B4R2VIK554AFSQ, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:30:55.762514255 (80h7m31.325837245s) [noise=true , address=104.36.82.14:6245 , piece_id=II36G46OA6TINLR244U6D7Q6LGV754UA5M62K3EX7ZOBESMJUEHA, seg_pos={0 0}, upload=64142]: Done uploading piece. err: ecclient: upload failed (node:1Zu8teNPTXn638grHKwtJqz4UEY7oFYLx1DHvuVP6RZXcatVqX, address:104.36.82.14:6245): protocol: storage node overloaded, request limit: 2000 Jun 22 00:30:55.773669583 (80h7m31.3369926s) [noise=true , address=79.127.205.235:5660 , piece_id=UGFPVS7VNB7U2YSFPSGYDIRZV5R2DHCN57PJP6VOXO52WIAYHLXA, seg_pos={0 0}, upload=64142]: Done uploading piece. err: ecclient: upload failed (node:12E6uCX9KL4Ud7YpxMXepvMXuqTFJCiSXByDjW94Am3U9H2Nf7f, address:79.127.205.235:5660): protocol: storage node overloaded, request limit: 2000 Jun 22 00:30:56.968366655 (80h7m32.53168964s) [noise=true , address=77.90.31.10:4533 , piece_id=SE6YAKKFDL3BV677CDDUC32UNMPXJHTG23GWWYK5B4YIPKER6GAQ, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:30:57.403572874 (80h7m32.966895883s) [noise=true , address=45.91.116.5:5310 , piece_id=AXJOKDZRPVNL7HFT4QKJWRAHFM2UCM3GEVQB2KNYD6CXB6TH4D4Q, seg_pos={0 0}, upload=64142]: Done uploading piece. err: ecclient: upload failed (node:127RMUc62xRq9RU5z3XfkZsq5sehmv3kP8o53tr9BxVcPE3GQCY, address:45.91.116.5:5310): protocol: storage node overloaded, request limit: 2000 Jun 22 00:30:58.210389994 (80h7m33.773712995s) [noise=true , address=104.36.82.13:6199 , piece_id=YP62N2TO446YR3QRG5YLXECD7TN6SJ4UNNLZSK5H4CMV6JIF4AKA, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:30:58.693037961 (80h7m34.256360966s) [noise=true , address=104.36.82.39:6756 , piece_id=OKKWMDJPZO6D5VRMGOE3GOCYGHM725JZTNOVYAWKJW3WYZPI4V5Q, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:30:59.124834814 (80h7m34.688157819s) [noise=true , address=104.36.82.15:6283 , piece_id=N7TH2EMIOPATVHCSNH2WXUGU7KMMEJ5LCHR22SWEHSA5OV4CUCXQ, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:31:00.629657982 (80h7m36.192980977s) [noise=true , address=104.36.82.42:6803 , piece_id=R626UFD326S2UBJEZ2WMHVR7EN4SK427VXPDJZOOZ32MX25JWZWQ, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:31:00.933110546 (80h7m36.496433536s) [noise=true , address=79.127.201.215:4326 , piece_id=USEWZDX6P52U6MCALTBNQG3SV7N2JRKNU6O23TLS6JEWAS4D7ARA, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:31:05.669167243 (80h7m41.232490237s) [noise=true , address=104.36.82.56:4261 , piece_id=EYOYOYLDACGMPGJYX6ILQT3XHMUTQNXZEDC7ANHLZXIKYVDQFFVQ, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:31:08.443582763 (80h7m44.006905763s) [noise=true , address=104.36.82.54:4184 , piece_id=IZICVGZFMAYLIJTNUJ2OFLZVNZ6IPPYDZO4BSTJJPLHVCVOE5EAA, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:31:26.459138014 (80h8m2.022461013s) [noise=true , address=104.36.82.53:4146 , piece_id=3V3DVGUOPJKDWIGQ2TU7M6VFYFVAS6ODY62ERGSU6LI3LSTWQ4YQ, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:31:27.456050336 (80h8m3.019373336s) [noise=true , address=104.36.82.52:4089 , piece_id=X2VG4S3UHVNWA4AQJS7FLKWNVLTOZ3AKWI5ZDH24QCA3ENQW7BXQ, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:31:55.270234251 (80h8m30.833557256s) [noise=true , address=104.36.82.11:6125 , piece_id=2CX7RJOKNQA6ODZZZXYPSCYAQY5P35TWX7I56CMYAPFLJE3JEO5Q, seg_pos={0 0}, upload=64142]: Done uploading piece. err: ecclient: upload failed (node:12vricofwx67FRaxAGLbmgT1TYyKkqfJbxdsyeEPL4R7kg8pKzT, address:104.36.82.11:6125): protocol: expected piece hash; context deadline exceeded; EOF Jun 22 00:31:55.270319418 (80h8m30.833642407s) [noise=true , address=104.36.82.55:4224 , piece_id=XF7H2BYSNNNZOS2T77VIN75N76AYUXAQEI6D4LRXACU2KU62V2TA, seg_pos={0 0}, upload=64142]: Done uploading piece. err: ecclient: upload failed (node:12EjKmUiv4pxkVCB4WoiXim4uFpxrBPyfiSS4efTY3HKSE3jb5y, address:104.36.82.55:4224): protocol: write tcp 159.223.143.39:39152->104.36.82.55:4224: use of closed network connection; write tcp 159.223.143.39:39152->104.36.82.55:4224: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:39152->104.36.82.55:4224: use of closed network connection Jun 22 00:31:55.270353811 (80h8m30.833676799s) [seg_pos={0 0}, upload=64142]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Jun 22 00:31:55.301099877 (80h8m30.864422884s) [upload=64142]: Done uploading segments... Jun 22 00:31:55.301121567 (80h8m30.864444566s) [upload=64142]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 26.173969484s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [..............X.X..............................X............] duration: 46.994169541s FAIL (2025-06-22 00:40:22 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== ing upload Jun 22 00:30:55.303646184 (80h7m30.866969171s) [upload=64142]: Uploading segments... Jun 22 00:30:55.303706406 (80h7m30.867029416s) [upload=64142]: Got next segment. Inline: false Jun 22 00:30:55.303762479 (80h7m30.867085468s) [upload=64142]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 22 00:30:55.303768627 (80h7m30.867091636s) [upload=64142]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 22 00:30:55.349058140 (80h7m30.912381168s) [seg_pos={0 0}, upload=64142]: Begin upload segment... Jun 22 00:30:55.349113768 (80h7m30.912436772s) [seg_pos={0 0}, upload=64142]: Done begin upload segment. Jun 22 00:30:55.349143121 (80h7m30.912466131s) [noise=true , address=45.91.116.5:5310 , piece_id=AXJOKDZRPVNL7HFT4QKJWRAHFM2UCM3GEVQB2KNYD6CXB6TH4D4Q, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.349660917 (80h7m30.912983928s) [noise=true , address=77.90.31.10:4533 , piece_id=SE6YAKKFDL3BV677CDDUC32UNMPXJHTG23GWWYK5B4YIPKER6GAQ, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.350093077 (80h7m30.913416089s) [noise=true , address=79.127.201.215:4326 , piece_id=USEWZDX6P52U6MCALTBNQG3SV7N2JRKNU6O23TLS6JEWAS4D7ARA, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.350480894 (80h7m30.913803916s) [noise=true , address=104.36.82.13:6199 , piece_id=YP62N2TO446YR3QRG5YLXECD7TN6SJ4UNNLZSK5H4CMV6JIF4AKA, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.350669998 (80h7m30.913993011s) [noise=true , address=104.36.82.54:4184 , piece_id=IZICVGZFMAYLIJTNUJ2OFLZVNZ6IPPYDZO4BSTJJPLHVCVOE5EAA, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.351201352 (80h7m30.914524366s) [noise=true , address=104.36.82.55:4224 , piece_id=XF7H2BYSNNNZOS2T77VIN75N76AYUXAQEI6D4LRXACU2KU62V2TA, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.351668071 (80h7m30.914991083s) [noise=true , address=104.36.82.56:4261 , piece_id=EYOYOYLDACGMPGJYX6ILQT3XHMUTQNXZEDC7ANHLZXIKYVDQFFVQ, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.352193126 (80h7m30.915516138s) [noise=true , address=104.36.82.52:4089 , piece_id=X2VG4S3UHVNWA4AQJS7FLKWNVLTOZ3AKWI5ZDH24QCA3ENQW7BXQ, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.352587096 (80h7m30.915910103s) [noise=true , address=79.127.220.98:4402 , piece_id=TE2WQO5LK45AVMPIAIGVSIBOVZLC4FG4ESQ72X6KFYC7IAEN2DRA, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.353420253 (80h7m30.916743265s) [noise=true , address=104.36.82.11:6125 , piece_id=2CX7RJOKNQA6ODZZZXYPSCYAQY5P35TWX7I56CMYAPFLJE3JEO5Q, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.353791565 (80h7m30.917114578s) [noise=true , address=104.36.82.42:6803 , piece_id=R626UFD326S2UBJEZ2WMHVR7EN4SK427VXPDJZOOZ32MX25JWZWQ, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.354171226 (80h7m30.917494233s) [noise=true , address=104.36.82.14:6245 , piece_id=II36G46OA6TINLR244U6D7Q6LGV754UA5M62K3EX7ZOBESMJUEHA, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.354542282 (80h7m30.917865288s) [noise=true , address=79.127.205.235:5660 , piece_id=UGFPVS7VNB7U2YSFPSGYDIRZV5R2DHCN57PJP6VOXO52WIAYHLXA, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.354891655 (80h7m30.918214678s) [noise=true , address=104.36.82.15:6283 , piece_id=N7TH2EMIOPATVHCSNH2WXUGU7KMMEJ5LCHR22SWEHSA5OV4CUCXQ, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.355499503 (80h7m30.918822519s) [noise=true , address=77.90.31.14:4666 , piece_id=H4DTFDAVCN266PB235BJIQTQ32FBXTIB5AZSX3B4R2VIK554AFSQ, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.356575547 (80h7m30.919898539s) [noise=true , address=104.36.82.39:6756 , piece_id=OKKWMDJPZO6D5VRMGOE3GOCYGHM725JZTNOVYAWKJW3WYZPI4V5Q, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.357039590 (80h7m30.920362596s) [noise=true , address=104.36.82.53:4146 , piece_id=3V3DVGUOPJKDWIGQ2TU7M6VFYFVAS6ODY62ERGSU6LI3LSTWQ4YQ, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.360343566 (80h7m30.923666582s) [noise=true , address=104.36.82.9:6033 , piece_id=T5XA23LZELMSTWOLBSHOX4AHKDACF65OYCKTMZYTXDIBSXNA2T4A, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.452444064 (80h7m31.015767055s) [upload=64142]: Next returned nil segment Jun 22 00:30:55.452495148 (80h7m31.015818149s) [upload=64142]: Waiting for error group managing segments... Jun 22 00:30:55.532958808 (80h7m31.096281883s) [noise=true , address=104.36.82.9:6033 , piece_id=T5XA23LZELMSTWOLBSHOX4AHKDACF65OYCKTMZYTXDIBSXNA2T4A, seg_pos={0 0}, upload=64142]: Done uploading piece. err: ecclient: upload failed (node:1SV3LrcQW16wLewDzfyUWFB6sUgnERNJVdhz72CJmyde95M79C, address:104.36.82.9:6033): protocol: storage node overloaded, request limit: 2000 Jun 22 00:30:55.547802184 (80h7m31.111125188s) [noise=true , address=79.127.220.98:4402 , piece_id=TE2WQO5LK45AVMPIAIGVSIBOVZLC4FG4ESQ72X6KFYC7IAEN2DRA, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:30:55.554223621 (80h7m31.117546616s) [noise=true , address=77.90.31.14:4666 , piece_id=H4DTFDAVCN266PB235BJIQTQ32FBXTIB5AZSX3B4R2VIK554AFSQ, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:30:55.762514255 (80h7m31.325837245s) [noise=true , address=104.36.82.14:6245 , piece_id=II36G46OA6TINLR244U6D7Q6LGV754UA5M62K3EX7ZOBESMJUEHA, seg_pos={0 0}, upload=64142]: Done uploading piece. err: ecclient: upload failed (node:1Zu8teNPTXn638grHKwtJqz4UEY7oFYLx1DHvuVP6RZXcatVqX, address:104.36.82.14:6245): protocol: storage node overloaded, request limit: 2000 Jun 22 00:30:55.773669583 (80h7m31.3369926s) [noise=true , address=79.127.205.235:5660 , piece_id=UGFPVS7VNB7U2YSFPSGYDIRZV5R2DHCN57PJP6VOXO52WIAYHLXA, seg_pos={0 0}, upload=64142]: Done uploading piece. err: ecclient: upload failed (node:12E6uCX9KL4Ud7YpxMXepvMXuqTFJCiSXByDjW94Am3U9H2Nf7f, address:79.127.205.235:5660): protocol: storage node overloaded, request limit: 2000 Jun 22 00:30:56.968366655 (80h7m32.53168964s) [noise=true , address=77.90.31.10:4533 , piece_id=SE6YAKKFDL3BV677CDDUC32UNMPXJHTG23GWWYK5B4YIPKER6GAQ, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:30:57.403572874 (80h7m32.966895883s) [noise=true , address=45.91.116.5:5310 , piece_id=AXJOKDZRPVNL7HFT4QKJWRAHFM2UCM3GEVQB2KNYD6CXB6TH4D4Q, seg_pos={0 0}, upload=64142]: Done uploading piece. err: ecclient: upload failed (node:127RMUc62xRq9RU5z3XfkZsq5sehmv3kP8o53tr9BxVcPE3GQCY, address:45.91.116.5:5310): protocol: storage node overloaded, request limit: 2000 Jun 22 00:30:58.210389994 (80h7m33.773712995s) [noise=true , address=104.36.82.13:6199 , piece_id=YP62N2TO446YR3QRG5YLXECD7TN6SJ4UNNLZSK5H4CMV6JIF4AKA, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:30:58.693037961 (80h7m34.256360966s) [noise=true , address=104.36.82.39:6756 , piece_id=OKKWMDJPZO6D5VRMGOE3GOCYGHM725JZTNOVYAWKJW3WYZPI4V5Q, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:30:59.124834814 (80h7m34.688157819s) [noise=true , address=104.36.82.15:6283 , piece_id=N7TH2EMIOPATVHCSNH2WXUGU7KMMEJ5LCHR22SWEHSA5OV4CUCXQ, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:31:00.629657982 (80h7m36.192980977s) [noise=true , address=104.36.82.42:6803 , piece_id=R626UFD326S2UBJEZ2WMHVR7EN4SK427VXPDJZOOZ32MX25JWZWQ, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:31:00.933110546 (80h7m36.496433536s) [noise=true , address=79.127.201.215:4326 , piece_id=USEWZDX6P52U6MCALTBNQG3SV7N2JRKNU6O23TLS6JEWAS4D7ARA, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:31:05.669167243 (80h7m41.232490237s) [noise=true , address=104.36.82.56:4261 , piece_id=EYOYOYLDACGMPGJYX6ILQT3XHMUTQNXZEDC7ANHLZXIKYVDQFFVQ, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:31:08.443582763 (80h7m44.006905763s) [noise=true , address=104.36.82.54:4184 , piece_id=IZICVGZFMAYLIJTNUJ2OFLZVNZ6IPPYDZO4BSTJJPLHVCVOE5EAA, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:31:26.459138014 (80h8m2.022461013s) [noise=true , address=104.36.82.53:4146 , piece_id=3V3DVGUOPJKDWIGQ2TU7M6VFYFVAS6ODY62ERGSU6LI3LSTWQ4YQ, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:31:27.456050336 (80h8m3.019373336s) [noise=true , address=104.36.82.52:4089 , piece_id=X2VG4S3UHVNWA4AQJS7FLKWNVLTOZ3AKWI5ZDH24QCA3ENQW7BXQ, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:31:55.270234251 (80h8m30.833557256s) [noise=true , address=104.36.82.11:6125 , piece_id=2CX7RJOKNQA6ODZZZXYPSCYAQY5P35TWX7I56CMYAPFLJE3JEO5Q, seg_pos={0 0}, upload=64142]: Done uploading piece. err: ecclient: upload failed (node:12vricofwx67FRaxAGLbmgT1TYyKkqfJbxdsyeEPL4R7kg8pKzT, address:104.36.82.11:6125): protocol: expected piece hash; context deadline exceeded; EOF Jun 22 00:31:55.270319418 (80h8m30.833642407s) [noise=true , address=104.36.82.55:4224 , piece_id=XF7H2BYSNNNZOS2T77VIN75N76AYUXAQEI6D4LRXACU2KU62V2TA, seg_pos={0 0}, upload=64142]: Done uploading piece. err: ecclient: upload failed (node:12EjKmUiv4pxkVCB4WoiXim4uFpxrBPyfiSS4efTY3HKSE3jb5y, address:104.36.82.55:4224): protocol: write tcp 159.223.143.39:39152->104.36.82.55:4224: use of closed network connection; write tcp 159.223.143.39:39152->104.36.82.55:4224: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:39152->104.36.82.55:4224: use of closed network connection Jun 22 00:31:55.270353811 (80h8m30.833676799s) [seg_pos={0 0}, upload=64142]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Jun 22 00:31:55.301099877 (80h8m30.864422884s) [upload=64142]: Done uploading segments... Jun 22 00:31:55.301121567 (80h8m30.864444566s) [upload=64142]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 46.994284255s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [.................X.........X...................X............] duration: 13.703978467s FAIL (2025-06-22 00:40:12 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 13.704020156s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [...............X.X..............................X...........] duration: 47.377492555s FAIL (2025-06-22 00:39:27 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== ing upload Jun 22 00:30:55.303646184 (80h7m30.866969171s) [upload=64142]: Uploading segments... Jun 22 00:30:55.303706406 (80h7m30.867029416s) [upload=64142]: Got next segment. Inline: false Jun 22 00:30:55.303762479 (80h7m30.867085468s) [upload=64142]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 22 00:30:55.303768627 (80h7m30.867091636s) [upload=64142]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 22 00:30:55.349058140 (80h7m30.912381168s) [seg_pos={0 0}, upload=64142]: Begin upload segment... Jun 22 00:30:55.349113768 (80h7m30.912436772s) [seg_pos={0 0}, upload=64142]: Done begin upload segment. Jun 22 00:30:55.349143121 (80h7m30.912466131s) [noise=true , address=45.91.116.5:5310 , piece_id=AXJOKDZRPVNL7HFT4QKJWRAHFM2UCM3GEVQB2KNYD6CXB6TH4D4Q, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.349660917 (80h7m30.912983928s) [noise=true , address=77.90.31.10:4533 , piece_id=SE6YAKKFDL3BV677CDDUC32UNMPXJHTG23GWWYK5B4YIPKER6GAQ, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.350093077 (80h7m30.913416089s) [noise=true , address=79.127.201.215:4326 , piece_id=USEWZDX6P52U6MCALTBNQG3SV7N2JRKNU6O23TLS6JEWAS4D7ARA, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.350480894 (80h7m30.913803916s) [noise=true , address=104.36.82.13:6199 , piece_id=YP62N2TO446YR3QRG5YLXECD7TN6SJ4UNNLZSK5H4CMV6JIF4AKA, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.350669998 (80h7m30.913993011s) [noise=true , address=104.36.82.54:4184 , piece_id=IZICVGZFMAYLIJTNUJ2OFLZVNZ6IPPYDZO4BSTJJPLHVCVOE5EAA, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.351201352 (80h7m30.914524366s) [noise=true , address=104.36.82.55:4224 , piece_id=XF7H2BYSNNNZOS2T77VIN75N76AYUXAQEI6D4LRXACU2KU62V2TA, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.351668071 (80h7m30.914991083s) [noise=true , address=104.36.82.56:4261 , piece_id=EYOYOYLDACGMPGJYX6ILQT3XHMUTQNXZEDC7ANHLZXIKYVDQFFVQ, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.352193126 (80h7m30.915516138s) [noise=true , address=104.36.82.52:4089 , piece_id=X2VG4S3UHVNWA4AQJS7FLKWNVLTOZ3AKWI5ZDH24QCA3ENQW7BXQ, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.352587096 (80h7m30.915910103s) [noise=true , address=79.127.220.98:4402 , piece_id=TE2WQO5LK45AVMPIAIGVSIBOVZLC4FG4ESQ72X6KFYC7IAEN2DRA, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.353420253 (80h7m30.916743265s) [noise=true , address=104.36.82.11:6125 , piece_id=2CX7RJOKNQA6ODZZZXYPSCYAQY5P35TWX7I56CMYAPFLJE3JEO5Q, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.353791565 (80h7m30.917114578s) [noise=true , address=104.36.82.42:6803 , piece_id=R626UFD326S2UBJEZ2WMHVR7EN4SK427VXPDJZOOZ32MX25JWZWQ, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.354171226 (80h7m30.917494233s) [noise=true , address=104.36.82.14:6245 , piece_id=II36G46OA6TINLR244U6D7Q6LGV754UA5M62K3EX7ZOBESMJUEHA, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.354542282 (80h7m30.917865288s) [noise=true , address=79.127.205.235:5660 , piece_id=UGFPVS7VNB7U2YSFPSGYDIRZV5R2DHCN57PJP6VOXO52WIAYHLXA, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.354891655 (80h7m30.918214678s) [noise=true , address=104.36.82.15:6283 , piece_id=N7TH2EMIOPATVHCSNH2WXUGU7KMMEJ5LCHR22SWEHSA5OV4CUCXQ, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.355499503 (80h7m30.918822519s) [noise=true , address=77.90.31.14:4666 , piece_id=H4DTFDAVCN266PB235BJIQTQ32FBXTIB5AZSX3B4R2VIK554AFSQ, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.356575547 (80h7m30.919898539s) [noise=true , address=104.36.82.39:6756 , piece_id=OKKWMDJPZO6D5VRMGOE3GOCYGHM725JZTNOVYAWKJW3WYZPI4V5Q, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.357039590 (80h7m30.920362596s) [noise=true , address=104.36.82.53:4146 , piece_id=3V3DVGUOPJKDWIGQ2TU7M6VFYFVAS6ODY62ERGSU6LI3LSTWQ4YQ, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.360343566 (80h7m30.923666582s) [noise=true , address=104.36.82.9:6033 , piece_id=T5XA23LZELMSTWOLBSHOX4AHKDACF65OYCKTMZYTXDIBSXNA2T4A, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.452444064 (80h7m31.015767055s) [upload=64142]: Next returned nil segment Jun 22 00:30:55.452495148 (80h7m31.015818149s) [upload=64142]: Waiting for error group managing segments... Jun 22 00:30:55.532958808 (80h7m31.096281883s) [noise=true , address=104.36.82.9:6033 , piece_id=T5XA23LZELMSTWOLBSHOX4AHKDACF65OYCKTMZYTXDIBSXNA2T4A, seg_pos={0 0}, upload=64142]: Done uploading piece. err: ecclient: upload failed (node:1SV3LrcQW16wLewDzfyUWFB6sUgnERNJVdhz72CJmyde95M79C, address:104.36.82.9:6033): protocol: storage node overloaded, request limit: 2000 Jun 22 00:30:55.547802184 (80h7m31.111125188s) [noise=true , address=79.127.220.98:4402 , piece_id=TE2WQO5LK45AVMPIAIGVSIBOVZLC4FG4ESQ72X6KFYC7IAEN2DRA, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:30:55.554223621 (80h7m31.117546616s) [noise=true , address=77.90.31.14:4666 , piece_id=H4DTFDAVCN266PB235BJIQTQ32FBXTIB5AZSX3B4R2VIK554AFSQ, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:30:55.762514255 (80h7m31.325837245s) [noise=true , address=104.36.82.14:6245 , piece_id=II36G46OA6TINLR244U6D7Q6LGV754UA5M62K3EX7ZOBESMJUEHA, seg_pos={0 0}, upload=64142]: Done uploading piece. err: ecclient: upload failed (node:1Zu8teNPTXn638grHKwtJqz4UEY7oFYLx1DHvuVP6RZXcatVqX, address:104.36.82.14:6245): protocol: storage node overloaded, request limit: 2000 Jun 22 00:30:55.773669583 (80h7m31.3369926s) [noise=true , address=79.127.205.235:5660 , piece_id=UGFPVS7VNB7U2YSFPSGYDIRZV5R2DHCN57PJP6VOXO52WIAYHLXA, seg_pos={0 0}, upload=64142]: Done uploading piece. err: ecclient: upload failed (node:12E6uCX9KL4Ud7YpxMXepvMXuqTFJCiSXByDjW94Am3U9H2Nf7f, address:79.127.205.235:5660): protocol: storage node overloaded, request limit: 2000 Jun 22 00:30:56.968366655 (80h7m32.53168964s) [noise=true , address=77.90.31.10:4533 , piece_id=SE6YAKKFDL3BV677CDDUC32UNMPXJHTG23GWWYK5B4YIPKER6GAQ, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:30:57.403572874 (80h7m32.966895883s) [noise=true , address=45.91.116.5:5310 , piece_id=AXJOKDZRPVNL7HFT4QKJWRAHFM2UCM3GEVQB2KNYD6CXB6TH4D4Q, seg_pos={0 0}, upload=64142]: Done uploading piece. err: ecclient: upload failed (node:127RMUc62xRq9RU5z3XfkZsq5sehmv3kP8o53tr9BxVcPE3GQCY, address:45.91.116.5:5310): protocol: storage node overloaded, request limit: 2000 Jun 22 00:30:58.210389994 (80h7m33.773712995s) [noise=true , address=104.36.82.13:6199 , piece_id=YP62N2TO446YR3QRG5YLXECD7TN6SJ4UNNLZSK5H4CMV6JIF4AKA, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:30:58.693037961 (80h7m34.256360966s) [noise=true , address=104.36.82.39:6756 , piece_id=OKKWMDJPZO6D5VRMGOE3GOCYGHM725JZTNOVYAWKJW3WYZPI4V5Q, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:30:59.124834814 (80h7m34.688157819s) [noise=true , address=104.36.82.15:6283 , piece_id=N7TH2EMIOPATVHCSNH2WXUGU7KMMEJ5LCHR22SWEHSA5OV4CUCXQ, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:31:00.629657982 (80h7m36.192980977s) [noise=true , address=104.36.82.42:6803 , piece_id=R626UFD326S2UBJEZ2WMHVR7EN4SK427VXPDJZOOZ32MX25JWZWQ, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:31:00.933110546 (80h7m36.496433536s) [noise=true , address=79.127.201.215:4326 , piece_id=USEWZDX6P52U6MCALTBNQG3SV7N2JRKNU6O23TLS6JEWAS4D7ARA, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:31:05.669167243 (80h7m41.232490237s) [noise=true , address=104.36.82.56:4261 , piece_id=EYOYOYLDACGMPGJYX6ILQT3XHMUTQNXZEDC7ANHLZXIKYVDQFFVQ, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:31:08.443582763 (80h7m44.006905763s) [noise=true , address=104.36.82.54:4184 , piece_id=IZICVGZFMAYLIJTNUJ2OFLZVNZ6IPPYDZO4BSTJJPLHVCVOE5EAA, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:31:26.459138014 (80h8m2.022461013s) [noise=true , address=104.36.82.53:4146 , piece_id=3V3DVGUOPJKDWIGQ2TU7M6VFYFVAS6ODY62ERGSU6LI3LSTWQ4YQ, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:31:27.456050336 (80h8m3.019373336s) [noise=true , address=104.36.82.52:4089 , piece_id=X2VG4S3UHVNWA4AQJS7FLKWNVLTOZ3AKWI5ZDH24QCA3ENQW7BXQ, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:31:55.270234251 (80h8m30.833557256s) [noise=true , address=104.36.82.11:6125 , piece_id=2CX7RJOKNQA6ODZZZXYPSCYAQY5P35TWX7I56CMYAPFLJE3JEO5Q, seg_pos={0 0}, upload=64142]: Done uploading piece. err: ecclient: upload failed (node:12vricofwx67FRaxAGLbmgT1TYyKkqfJbxdsyeEPL4R7kg8pKzT, address:104.36.82.11:6125): protocol: expected piece hash; context deadline exceeded; EOF Jun 22 00:31:55.270319418 (80h8m30.833642407s) [noise=true , address=104.36.82.55:4224 , piece_id=XF7H2BYSNNNZOS2T77VIN75N76AYUXAQEI6D4LRXACU2KU62V2TA, seg_pos={0 0}, upload=64142]: Done uploading piece. err: ecclient: upload failed (node:12EjKmUiv4pxkVCB4WoiXim4uFpxrBPyfiSS4efTY3HKSE3jb5y, address:104.36.82.55:4224): protocol: write tcp 159.223.143.39:39152->104.36.82.55:4224: use of closed network connection; write tcp 159.223.143.39:39152->104.36.82.55:4224: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:39152->104.36.82.55:4224: use of closed network connection Jun 22 00:31:55.270353811 (80h8m30.833676799s) [seg_pos={0 0}, upload=64142]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Jun 22 00:31:55.301099877 (80h8m30.864422884s) [upload=64142]: Done uploading segments... Jun 22 00:31:55.301121567 (80h8m30.864444566s) [upload=64142]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 47.377520579s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [..................X.........X...................X...........] duration: 14.665340889s FAIL (2025-06-22 00:39:14 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 14.665372254s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [................X.X..............................X..........] duration: 14.36907497s FAIL (2025-06-22 00:38:30 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== ing upload Jun 22 00:30:55.303646184 (80h7m30.866969171s) [upload=64142]: Uploading segments... Jun 22 00:30:55.303706406 (80h7m30.867029416s) [upload=64142]: Got next segment. Inline: false Jun 22 00:30:55.303762479 (80h7m30.867085468s) [upload=64142]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 22 00:30:55.303768627 (80h7m30.867091636s) [upload=64142]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 22 00:30:55.349058140 (80h7m30.912381168s) [seg_pos={0 0}, upload=64142]: Begin upload segment... Jun 22 00:30:55.349113768 (80h7m30.912436772s) [seg_pos={0 0}, upload=64142]: Done begin upload segment. Jun 22 00:30:55.349143121 (80h7m30.912466131s) [noise=true , address=45.91.116.5:5310 , piece_id=AXJOKDZRPVNL7HFT4QKJWRAHFM2UCM3GEVQB2KNYD6CXB6TH4D4Q, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.349660917 (80h7m30.912983928s) [noise=true , address=77.90.31.10:4533 , piece_id=SE6YAKKFDL3BV677CDDUC32UNMPXJHTG23GWWYK5B4YIPKER6GAQ, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.350093077 (80h7m30.913416089s) [noise=true , address=79.127.201.215:4326 , piece_id=USEWZDX6P52U6MCALTBNQG3SV7N2JRKNU6O23TLS6JEWAS4D7ARA, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.350480894 (80h7m30.913803916s) [noise=true , address=104.36.82.13:6199 , piece_id=YP62N2TO446YR3QRG5YLXECD7TN6SJ4UNNLZSK5H4CMV6JIF4AKA, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.350669998 (80h7m30.913993011s) [noise=true , address=104.36.82.54:4184 , piece_id=IZICVGZFMAYLIJTNUJ2OFLZVNZ6IPPYDZO4BSTJJPLHVCVOE5EAA, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.351201352 (80h7m30.914524366s) [noise=true , address=104.36.82.55:4224 , piece_id=XF7H2BYSNNNZOS2T77VIN75N76AYUXAQEI6D4LRXACU2KU62V2TA, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.351668071 (80h7m30.914991083s) [noise=true , address=104.36.82.56:4261 , piece_id=EYOYOYLDACGMPGJYX6ILQT3XHMUTQNXZEDC7ANHLZXIKYVDQFFVQ, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.352193126 (80h7m30.915516138s) [noise=true , address=104.36.82.52:4089 , piece_id=X2VG4S3UHVNWA4AQJS7FLKWNVLTOZ3AKWI5ZDH24QCA3ENQW7BXQ, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.352587096 (80h7m30.915910103s) [noise=true , address=79.127.220.98:4402 , piece_id=TE2WQO5LK45AVMPIAIGVSIBOVZLC4FG4ESQ72X6KFYC7IAEN2DRA, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.353420253 (80h7m30.916743265s) [noise=true , address=104.36.82.11:6125 , piece_id=2CX7RJOKNQA6ODZZZXYPSCYAQY5P35TWX7I56CMYAPFLJE3JEO5Q, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.353791565 (80h7m30.917114578s) [noise=true , address=104.36.82.42:6803 , piece_id=R626UFD326S2UBJEZ2WMHVR7EN4SK427VXPDJZOOZ32MX25JWZWQ, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.354171226 (80h7m30.917494233s) [noise=true , address=104.36.82.14:6245 , piece_id=II36G46OA6TINLR244U6D7Q6LGV754UA5M62K3EX7ZOBESMJUEHA, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.354542282 (80h7m30.917865288s) [noise=true , address=79.127.205.235:5660 , piece_id=UGFPVS7VNB7U2YSFPSGYDIRZV5R2DHCN57PJP6VOXO52WIAYHLXA, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.354891655 (80h7m30.918214678s) [noise=true , address=104.36.82.15:6283 , piece_id=N7TH2EMIOPATVHCSNH2WXUGU7KMMEJ5LCHR22SWEHSA5OV4CUCXQ, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.355499503 (80h7m30.918822519s) [noise=true , address=77.90.31.14:4666 , piece_id=H4DTFDAVCN266PB235BJIQTQ32FBXTIB5AZSX3B4R2VIK554AFSQ, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.356575547 (80h7m30.919898539s) [noise=true , address=104.36.82.39:6756 , piece_id=OKKWMDJPZO6D5VRMGOE3GOCYGHM725JZTNOVYAWKJW3WYZPI4V5Q, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.357039590 (80h7m30.920362596s) [noise=true , address=104.36.82.53:4146 , piece_id=3V3DVGUOPJKDWIGQ2TU7M6VFYFVAS6ODY62ERGSU6LI3LSTWQ4YQ, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.360343566 (80h7m30.923666582s) [noise=true , address=104.36.82.9:6033 , piece_id=T5XA23LZELMSTWOLBSHOX4AHKDACF65OYCKTMZYTXDIBSXNA2T4A, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.452444064 (80h7m31.015767055s) [upload=64142]: Next returned nil segment Jun 22 00:30:55.452495148 (80h7m31.015818149s) [upload=64142]: Waiting for error group managing segments... Jun 22 00:30:55.532958808 (80h7m31.096281883s) [noise=true , address=104.36.82.9:6033 , piece_id=T5XA23LZELMSTWOLBSHOX4AHKDACF65OYCKTMZYTXDIBSXNA2T4A, seg_pos={0 0}, upload=64142]: Done uploading piece. err: ecclient: upload failed (node:1SV3LrcQW16wLewDzfyUWFB6sUgnERNJVdhz72CJmyde95M79C, address:104.36.82.9:6033): protocol: storage node overloaded, request limit: 2000 Jun 22 00:30:55.547802184 (80h7m31.111125188s) [noise=true , address=79.127.220.98:4402 , piece_id=TE2WQO5LK45AVMPIAIGVSIBOVZLC4FG4ESQ72X6KFYC7IAEN2DRA, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:30:55.554223621 (80h7m31.117546616s) [noise=true , address=77.90.31.14:4666 , piece_id=H4DTFDAVCN266PB235BJIQTQ32FBXTIB5AZSX3B4R2VIK554AFSQ, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:30:55.762514255 (80h7m31.325837245s) [noise=true , address=104.36.82.14:6245 , piece_id=II36G46OA6TINLR244U6D7Q6LGV754UA5M62K3EX7ZOBESMJUEHA, seg_pos={0 0}, upload=64142]: Done uploading piece. err: ecclient: upload failed (node:1Zu8teNPTXn638grHKwtJqz4UEY7oFYLx1DHvuVP6RZXcatVqX, address:104.36.82.14:6245): protocol: storage node overloaded, request limit: 2000 Jun 22 00:30:55.773669583 (80h7m31.3369926s) [noise=true , address=79.127.205.235:5660 , piece_id=UGFPVS7VNB7U2YSFPSGYDIRZV5R2DHCN57PJP6VOXO52WIAYHLXA, seg_pos={0 0}, upload=64142]: Done uploading piece. err: ecclient: upload failed (node:12E6uCX9KL4Ud7YpxMXepvMXuqTFJCiSXByDjW94Am3U9H2Nf7f, address:79.127.205.235:5660): protocol: storage node overloaded, request limit: 2000 Jun 22 00:30:56.968366655 (80h7m32.53168964s) [noise=true , address=77.90.31.10:4533 , piece_id=SE6YAKKFDL3BV677CDDUC32UNMPXJHTG23GWWYK5B4YIPKER6GAQ, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:30:57.403572874 (80h7m32.966895883s) [noise=true , address=45.91.116.5:5310 , piece_id=AXJOKDZRPVNL7HFT4QKJWRAHFM2UCM3GEVQB2KNYD6CXB6TH4D4Q, seg_pos={0 0}, upload=64142]: Done uploading piece. err: ecclient: upload failed (node:127RMUc62xRq9RU5z3XfkZsq5sehmv3kP8o53tr9BxVcPE3GQCY, address:45.91.116.5:5310): protocol: storage node overloaded, request limit: 2000 Jun 22 00:30:58.210389994 (80h7m33.773712995s) [noise=true , address=104.36.82.13:6199 , piece_id=YP62N2TO446YR3QRG5YLXECD7TN6SJ4UNNLZSK5H4CMV6JIF4AKA, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:30:58.693037961 (80h7m34.256360966s) [noise=true , address=104.36.82.39:6756 , piece_id=OKKWMDJPZO6D5VRMGOE3GOCYGHM725JZTNOVYAWKJW3WYZPI4V5Q, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:30:59.124834814 (80h7m34.688157819s) [noise=true , address=104.36.82.15:6283 , piece_id=N7TH2EMIOPATVHCSNH2WXUGU7KMMEJ5LCHR22SWEHSA5OV4CUCXQ, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:31:00.629657982 (80h7m36.192980977s) [noise=true , address=104.36.82.42:6803 , piece_id=R626UFD326S2UBJEZ2WMHVR7EN4SK427VXPDJZOOZ32MX25JWZWQ, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:31:00.933110546 (80h7m36.496433536s) [noise=true , address=79.127.201.215:4326 , piece_id=USEWZDX6P52U6MCALTBNQG3SV7N2JRKNU6O23TLS6JEWAS4D7ARA, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:31:05.669167243 (80h7m41.232490237s) [noise=true , address=104.36.82.56:4261 , piece_id=EYOYOYLDACGMPGJYX6ILQT3XHMUTQNXZEDC7ANHLZXIKYVDQFFVQ, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:31:08.443582763 (80h7m44.006905763s) [noise=true , address=104.36.82.54:4184 , piece_id=IZICVGZFMAYLIJTNUJ2OFLZVNZ6IPPYDZO4BSTJJPLHVCVOE5EAA, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:31:26.459138014 (80h8m2.022461013s) [noise=true , address=104.36.82.53:4146 , piece_id=3V3DVGUOPJKDWIGQ2TU7M6VFYFVAS6ODY62ERGSU6LI3LSTWQ4YQ, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:31:27.456050336 (80h8m3.019373336s) [noise=true , address=104.36.82.52:4089 , piece_id=X2VG4S3UHVNWA4AQJS7FLKWNVLTOZ3AKWI5ZDH24QCA3ENQW7BXQ, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:31:55.270234251 (80h8m30.833557256s) [noise=true , address=104.36.82.11:6125 , piece_id=2CX7RJOKNQA6ODZZZXYPSCYAQY5P35TWX7I56CMYAPFLJE3JEO5Q, seg_pos={0 0}, upload=64142]: Done uploading piece. err: ecclient: upload failed (node:12vricofwx67FRaxAGLbmgT1TYyKkqfJbxdsyeEPL4R7kg8pKzT, address:104.36.82.11:6125): protocol: expected piece hash; context deadline exceeded; EOF Jun 22 00:31:55.270319418 (80h8m30.833642407s) [noise=true , address=104.36.82.55:4224 , piece_id=XF7H2BYSNNNZOS2T77VIN75N76AYUXAQEI6D4LRXACU2KU62V2TA, seg_pos={0 0}, upload=64142]: Done uploading piece. err: ecclient: upload failed (node:12EjKmUiv4pxkVCB4WoiXim4uFpxrBPyfiSS4efTY3HKSE3jb5y, address:104.36.82.55:4224): protocol: write tcp 159.223.143.39:39152->104.36.82.55:4224: use of closed network connection; write tcp 159.223.143.39:39152->104.36.82.55:4224: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:39152->104.36.82.55:4224: use of closed network connection Jun 22 00:31:55.270353811 (80h8m30.833676799s) [seg_pos={0 0}, upload=64142]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Jun 22 00:31:55.301099877 (80h8m30.864422884s) [upload=64142]: Done uploading segments... Jun 22 00:31:55.301121567 (80h8m30.864444566s) [upload=64142]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 14.369101955s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [...................X.........X...................X..........] duration: 16.859170272s FAIL (2025-06-22 00:38:22 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 16.859191473s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [.................X.X..............................X.........] duration: 15.41775267s FAIL (2025-06-22 00:37:56 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== ing upload Jun 22 00:30:55.303646184 (80h7m30.866969171s) [upload=64142]: Uploading segments... Jun 22 00:30:55.303706406 (80h7m30.867029416s) [upload=64142]: Got next segment. Inline: false Jun 22 00:30:55.303762479 (80h7m30.867085468s) [upload=64142]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 22 00:30:55.303768627 (80h7m30.867091636s) [upload=64142]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 22 00:30:55.349058140 (80h7m30.912381168s) [seg_pos={0 0}, upload=64142]: Begin upload segment... Jun 22 00:30:55.349113768 (80h7m30.912436772s) [seg_pos={0 0}, upload=64142]: Done begin upload segment. Jun 22 00:30:55.349143121 (80h7m30.912466131s) [noise=true , address=45.91.116.5:5310 , piece_id=AXJOKDZRPVNL7HFT4QKJWRAHFM2UCM3GEVQB2KNYD6CXB6TH4D4Q, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.349660917 (80h7m30.912983928s) [noise=true , address=77.90.31.10:4533 , piece_id=SE6YAKKFDL3BV677CDDUC32UNMPXJHTG23GWWYK5B4YIPKER6GAQ, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.350093077 (80h7m30.913416089s) [noise=true , address=79.127.201.215:4326 , piece_id=USEWZDX6P52U6MCALTBNQG3SV7N2JRKNU6O23TLS6JEWAS4D7ARA, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.350480894 (80h7m30.913803916s) [noise=true , address=104.36.82.13:6199 , piece_id=YP62N2TO446YR3QRG5YLXECD7TN6SJ4UNNLZSK5H4CMV6JIF4AKA, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.350669998 (80h7m30.913993011s) [noise=true , address=104.36.82.54:4184 , piece_id=IZICVGZFMAYLIJTNUJ2OFLZVNZ6IPPYDZO4BSTJJPLHVCVOE5EAA, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.351201352 (80h7m30.914524366s) [noise=true , address=104.36.82.55:4224 , piece_id=XF7H2BYSNNNZOS2T77VIN75N76AYUXAQEI6D4LRXACU2KU62V2TA, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.351668071 (80h7m30.914991083s) [noise=true , address=104.36.82.56:4261 , piece_id=EYOYOYLDACGMPGJYX6ILQT3XHMUTQNXZEDC7ANHLZXIKYVDQFFVQ, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.352193126 (80h7m30.915516138s) [noise=true , address=104.36.82.52:4089 , piece_id=X2VG4S3UHVNWA4AQJS7FLKWNVLTOZ3AKWI5ZDH24QCA3ENQW7BXQ, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.352587096 (80h7m30.915910103s) [noise=true , address=79.127.220.98:4402 , piece_id=TE2WQO5LK45AVMPIAIGVSIBOVZLC4FG4ESQ72X6KFYC7IAEN2DRA, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.353420253 (80h7m30.916743265s) [noise=true , address=104.36.82.11:6125 , piece_id=2CX7RJOKNQA6ODZZZXYPSCYAQY5P35TWX7I56CMYAPFLJE3JEO5Q, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.353791565 (80h7m30.917114578s) [noise=true , address=104.36.82.42:6803 , piece_id=R626UFD326S2UBJEZ2WMHVR7EN4SK427VXPDJZOOZ32MX25JWZWQ, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.354171226 (80h7m30.917494233s) [noise=true , address=104.36.82.14:6245 , piece_id=II36G46OA6TINLR244U6D7Q6LGV754UA5M62K3EX7ZOBESMJUEHA, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.354542282 (80h7m30.917865288s) [noise=true , address=79.127.205.235:5660 , piece_id=UGFPVS7VNB7U2YSFPSGYDIRZV5R2DHCN57PJP6VOXO52WIAYHLXA, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.354891655 (80h7m30.918214678s) [noise=true , address=104.36.82.15:6283 , piece_id=N7TH2EMIOPATVHCSNH2WXUGU7KMMEJ5LCHR22SWEHSA5OV4CUCXQ, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.355499503 (80h7m30.918822519s) [noise=true , address=77.90.31.14:4666 , piece_id=H4DTFDAVCN266PB235BJIQTQ32FBXTIB5AZSX3B4R2VIK554AFSQ, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.356575547 (80h7m30.919898539s) [noise=true , address=104.36.82.39:6756 , piece_id=OKKWMDJPZO6D5VRMGOE3GOCYGHM725JZTNOVYAWKJW3WYZPI4V5Q, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.357039590 (80h7m30.920362596s) [noise=true , address=104.36.82.53:4146 , piece_id=3V3DVGUOPJKDWIGQ2TU7M6VFYFVAS6ODY62ERGSU6LI3LSTWQ4YQ, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.360343566 (80h7m30.923666582s) [noise=true , address=104.36.82.9:6033 , piece_id=T5XA23LZELMSTWOLBSHOX4AHKDACF65OYCKTMZYTXDIBSXNA2T4A, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.452444064 (80h7m31.015767055s) [upload=64142]: Next returned nil segment Jun 22 00:30:55.452495148 (80h7m31.015818149s) [upload=64142]: Waiting for error group managing segments... Jun 22 00:30:55.532958808 (80h7m31.096281883s) [noise=true , address=104.36.82.9:6033 , piece_id=T5XA23LZELMSTWOLBSHOX4AHKDACF65OYCKTMZYTXDIBSXNA2T4A, seg_pos={0 0}, upload=64142]: Done uploading piece. err: ecclient: upload failed (node:1SV3LrcQW16wLewDzfyUWFB6sUgnERNJVdhz72CJmyde95M79C, address:104.36.82.9:6033): protocol: storage node overloaded, request limit: 2000 Jun 22 00:30:55.547802184 (80h7m31.111125188s) [noise=true , address=79.127.220.98:4402 , piece_id=TE2WQO5LK45AVMPIAIGVSIBOVZLC4FG4ESQ72X6KFYC7IAEN2DRA, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:30:55.554223621 (80h7m31.117546616s) [noise=true , address=77.90.31.14:4666 , piece_id=H4DTFDAVCN266PB235BJIQTQ32FBXTIB5AZSX3B4R2VIK554AFSQ, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:30:55.762514255 (80h7m31.325837245s) [noise=true , address=104.36.82.14:6245 , piece_id=II36G46OA6TINLR244U6D7Q6LGV754UA5M62K3EX7ZOBESMJUEHA, seg_pos={0 0}, upload=64142]: Done uploading piece. err: ecclient: upload failed (node:1Zu8teNPTXn638grHKwtJqz4UEY7oFYLx1DHvuVP6RZXcatVqX, address:104.36.82.14:6245): protocol: storage node overloaded, request limit: 2000 Jun 22 00:30:55.773669583 (80h7m31.3369926s) [noise=true , address=79.127.205.235:5660 , piece_id=UGFPVS7VNB7U2YSFPSGYDIRZV5R2DHCN57PJP6VOXO52WIAYHLXA, seg_pos={0 0}, upload=64142]: Done uploading piece. err: ecclient: upload failed (node:12E6uCX9KL4Ud7YpxMXepvMXuqTFJCiSXByDjW94Am3U9H2Nf7f, address:79.127.205.235:5660): protocol: storage node overloaded, request limit: 2000 Jun 22 00:30:56.968366655 (80h7m32.53168964s) [noise=true , address=77.90.31.10:4533 , piece_id=SE6YAKKFDL3BV677CDDUC32UNMPXJHTG23GWWYK5B4YIPKER6GAQ, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:30:57.403572874 (80h7m32.966895883s) [noise=true , address=45.91.116.5:5310 , piece_id=AXJOKDZRPVNL7HFT4QKJWRAHFM2UCM3GEVQB2KNYD6CXB6TH4D4Q, seg_pos={0 0}, upload=64142]: Done uploading piece. err: ecclient: upload failed (node:127RMUc62xRq9RU5z3XfkZsq5sehmv3kP8o53tr9BxVcPE3GQCY, address:45.91.116.5:5310): protocol: storage node overloaded, request limit: 2000 Jun 22 00:30:58.210389994 (80h7m33.773712995s) [noise=true , address=104.36.82.13:6199 , piece_id=YP62N2TO446YR3QRG5YLXECD7TN6SJ4UNNLZSK5H4CMV6JIF4AKA, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:30:58.693037961 (80h7m34.256360966s) [noise=true , address=104.36.82.39:6756 , piece_id=OKKWMDJPZO6D5VRMGOE3GOCYGHM725JZTNOVYAWKJW3WYZPI4V5Q, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:30:59.124834814 (80h7m34.688157819s) [noise=true , address=104.36.82.15:6283 , piece_id=N7TH2EMIOPATVHCSNH2WXUGU7KMMEJ5LCHR22SWEHSA5OV4CUCXQ, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:31:00.629657982 (80h7m36.192980977s) [noise=true , address=104.36.82.42:6803 , piece_id=R626UFD326S2UBJEZ2WMHVR7EN4SK427VXPDJZOOZ32MX25JWZWQ, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:31:00.933110546 (80h7m36.496433536s) [noise=true , address=79.127.201.215:4326 , piece_id=USEWZDX6P52U6MCALTBNQG3SV7N2JRKNU6O23TLS6JEWAS4D7ARA, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:31:05.669167243 (80h7m41.232490237s) [noise=true , address=104.36.82.56:4261 , piece_id=EYOYOYLDACGMPGJYX6ILQT3XHMUTQNXZEDC7ANHLZXIKYVDQFFVQ, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:31:08.443582763 (80h7m44.006905763s) [noise=true , address=104.36.82.54:4184 , piece_id=IZICVGZFMAYLIJTNUJ2OFLZVNZ6IPPYDZO4BSTJJPLHVCVOE5EAA, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:31:26.459138014 (80h8m2.022461013s) [noise=true , address=104.36.82.53:4146 , piece_id=3V3DVGUOPJKDWIGQ2TU7M6VFYFVAS6ODY62ERGSU6LI3LSTWQ4YQ, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:31:27.456050336 (80h8m3.019373336s) [noise=true , address=104.36.82.52:4089 , piece_id=X2VG4S3UHVNWA4AQJS7FLKWNVLTOZ3AKWI5ZDH24QCA3ENQW7BXQ, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:31:55.270234251 (80h8m30.833557256s) [noise=true , address=104.36.82.11:6125 , piece_id=2CX7RJOKNQA6ODZZZXYPSCYAQY5P35TWX7I56CMYAPFLJE3JEO5Q, seg_pos={0 0}, upload=64142]: Done uploading piece. err: ecclient: upload failed (node:12vricofwx67FRaxAGLbmgT1TYyKkqfJbxdsyeEPL4R7kg8pKzT, address:104.36.82.11:6125): protocol: expected piece hash; context deadline exceeded; EOF Jun 22 00:31:55.270319418 (80h8m30.833642407s) [noise=true , address=104.36.82.55:4224 , piece_id=XF7H2BYSNNNZOS2T77VIN75N76AYUXAQEI6D4LRXACU2KU62V2TA, seg_pos={0 0}, upload=64142]: Done uploading piece. err: ecclient: upload failed (node:12EjKmUiv4pxkVCB4WoiXim4uFpxrBPyfiSS4efTY3HKSE3jb5y, address:104.36.82.55:4224): protocol: write tcp 159.223.143.39:39152->104.36.82.55:4224: use of closed network connection; write tcp 159.223.143.39:39152->104.36.82.55:4224: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:39152->104.36.82.55:4224: use of closed network connection Jun 22 00:31:55.270353811 (80h8m30.833676799s) [seg_pos={0 0}, upload=64142]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Jun 22 00:31:55.301099877 (80h8m30.864422884s) [upload=64142]: Done uploading segments... Jun 22 00:31:55.301121567 (80h8m30.864444566s) [upload=64142]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 15.417783339s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [....................X.........X...................X.........] duration: 42.573822708s FAIL (2025-06-22 00:37:41 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 42.573848063s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [..................X.X..............................X........] duration: 36.460524825s FAIL (2025-06-22 00:37:20 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== ing upload Jun 22 00:30:55.303646184 (80h7m30.866969171s) [upload=64142]: Uploading segments... Jun 22 00:30:55.303706406 (80h7m30.867029416s) [upload=64142]: Got next segment. Inline: false Jun 22 00:30:55.303762479 (80h7m30.867085468s) [upload=64142]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 22 00:30:55.303768627 (80h7m30.867091636s) [upload=64142]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 22 00:30:55.349058140 (80h7m30.912381168s) [seg_pos={0 0}, upload=64142]: Begin upload segment... Jun 22 00:30:55.349113768 (80h7m30.912436772s) [seg_pos={0 0}, upload=64142]: Done begin upload segment. Jun 22 00:30:55.349143121 (80h7m30.912466131s) [noise=true , address=45.91.116.5:5310 , piece_id=AXJOKDZRPVNL7HFT4QKJWRAHFM2UCM3GEVQB2KNYD6CXB6TH4D4Q, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.349660917 (80h7m30.912983928s) [noise=true , address=77.90.31.10:4533 , piece_id=SE6YAKKFDL3BV677CDDUC32UNMPXJHTG23GWWYK5B4YIPKER6GAQ, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.350093077 (80h7m30.913416089s) [noise=true , address=79.127.201.215:4326 , piece_id=USEWZDX6P52U6MCALTBNQG3SV7N2JRKNU6O23TLS6JEWAS4D7ARA, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.350480894 (80h7m30.913803916s) [noise=true , address=104.36.82.13:6199 , piece_id=YP62N2TO446YR3QRG5YLXECD7TN6SJ4UNNLZSK5H4CMV6JIF4AKA, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.350669998 (80h7m30.913993011s) [noise=true , address=104.36.82.54:4184 , piece_id=IZICVGZFMAYLIJTNUJ2OFLZVNZ6IPPYDZO4BSTJJPLHVCVOE5EAA, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.351201352 (80h7m30.914524366s) [noise=true , address=104.36.82.55:4224 , piece_id=XF7H2BYSNNNZOS2T77VIN75N76AYUXAQEI6D4LRXACU2KU62V2TA, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.351668071 (80h7m30.914991083s) [noise=true , address=104.36.82.56:4261 , piece_id=EYOYOYLDACGMPGJYX6ILQT3XHMUTQNXZEDC7ANHLZXIKYVDQFFVQ, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.352193126 (80h7m30.915516138s) [noise=true , address=104.36.82.52:4089 , piece_id=X2VG4S3UHVNWA4AQJS7FLKWNVLTOZ3AKWI5ZDH24QCA3ENQW7BXQ, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.352587096 (80h7m30.915910103s) [noise=true , address=79.127.220.98:4402 , piece_id=TE2WQO5LK45AVMPIAIGVSIBOVZLC4FG4ESQ72X6KFYC7IAEN2DRA, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.353420253 (80h7m30.916743265s) [noise=true , address=104.36.82.11:6125 , piece_id=2CX7RJOKNQA6ODZZZXYPSCYAQY5P35TWX7I56CMYAPFLJE3JEO5Q, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.353791565 (80h7m30.917114578s) [noise=true , address=104.36.82.42:6803 , piece_id=R626UFD326S2UBJEZ2WMHVR7EN4SK427VXPDJZOOZ32MX25JWZWQ, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.354171226 (80h7m30.917494233s) [noise=true , address=104.36.82.14:6245 , piece_id=II36G46OA6TINLR244U6D7Q6LGV754UA5M62K3EX7ZOBESMJUEHA, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.354542282 (80h7m30.917865288s) [noise=true , address=79.127.205.235:5660 , piece_id=UGFPVS7VNB7U2YSFPSGYDIRZV5R2DHCN57PJP6VOXO52WIAYHLXA, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.354891655 (80h7m30.918214678s) [noise=true , address=104.36.82.15:6283 , piece_id=N7TH2EMIOPATVHCSNH2WXUGU7KMMEJ5LCHR22SWEHSA5OV4CUCXQ, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.355499503 (80h7m30.918822519s) [noise=true , address=77.90.31.14:4666 , piece_id=H4DTFDAVCN266PB235BJIQTQ32FBXTIB5AZSX3B4R2VIK554AFSQ, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.356575547 (80h7m30.919898539s) [noise=true , address=104.36.82.39:6756 , piece_id=OKKWMDJPZO6D5VRMGOE3GOCYGHM725JZTNOVYAWKJW3WYZPI4V5Q, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.357039590 (80h7m30.920362596s) [noise=true , address=104.36.82.53:4146 , piece_id=3V3DVGUOPJKDWIGQ2TU7M6VFYFVAS6ODY62ERGSU6LI3LSTWQ4YQ, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.360343566 (80h7m30.923666582s) [noise=true , address=104.36.82.9:6033 , piece_id=T5XA23LZELMSTWOLBSHOX4AHKDACF65OYCKTMZYTXDIBSXNA2T4A, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.452444064 (80h7m31.015767055s) [upload=64142]: Next returned nil segment Jun 22 00:30:55.452495148 (80h7m31.015818149s) [upload=64142]: Waiting for error group managing segments... Jun 22 00:30:55.532958808 (80h7m31.096281883s) [noise=true , address=104.36.82.9:6033 , piece_id=T5XA23LZELMSTWOLBSHOX4AHKDACF65OYCKTMZYTXDIBSXNA2T4A, seg_pos={0 0}, upload=64142]: Done uploading piece. err: ecclient: upload failed (node:1SV3LrcQW16wLewDzfyUWFB6sUgnERNJVdhz72CJmyde95M79C, address:104.36.82.9:6033): protocol: storage node overloaded, request limit: 2000 Jun 22 00:30:55.547802184 (80h7m31.111125188s) [noise=true , address=79.127.220.98:4402 , piece_id=TE2WQO5LK45AVMPIAIGVSIBOVZLC4FG4ESQ72X6KFYC7IAEN2DRA, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:30:55.554223621 (80h7m31.117546616s) [noise=true , address=77.90.31.14:4666 , piece_id=H4DTFDAVCN266PB235BJIQTQ32FBXTIB5AZSX3B4R2VIK554AFSQ, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:30:55.762514255 (80h7m31.325837245s) [noise=true , address=104.36.82.14:6245 , piece_id=II36G46OA6TINLR244U6D7Q6LGV754UA5M62K3EX7ZOBESMJUEHA, seg_pos={0 0}, upload=64142]: Done uploading piece. err: ecclient: upload failed (node:1Zu8teNPTXn638grHKwtJqz4UEY7oFYLx1DHvuVP6RZXcatVqX, address:104.36.82.14:6245): protocol: storage node overloaded, request limit: 2000 Jun 22 00:30:55.773669583 (80h7m31.3369926s) [noise=true , address=79.127.205.235:5660 , piece_id=UGFPVS7VNB7U2YSFPSGYDIRZV5R2DHCN57PJP6VOXO52WIAYHLXA, seg_pos={0 0}, upload=64142]: Done uploading piece. err: ecclient: upload failed (node:12E6uCX9KL4Ud7YpxMXepvMXuqTFJCiSXByDjW94Am3U9H2Nf7f, address:79.127.205.235:5660): protocol: storage node overloaded, request limit: 2000 Jun 22 00:30:56.968366655 (80h7m32.53168964s) [noise=true , address=77.90.31.10:4533 , piece_id=SE6YAKKFDL3BV677CDDUC32UNMPXJHTG23GWWYK5B4YIPKER6GAQ, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:30:57.403572874 (80h7m32.966895883s) [noise=true , address=45.91.116.5:5310 , piece_id=AXJOKDZRPVNL7HFT4QKJWRAHFM2UCM3GEVQB2KNYD6CXB6TH4D4Q, seg_pos={0 0}, upload=64142]: Done uploading piece. err: ecclient: upload failed (node:127RMUc62xRq9RU5z3XfkZsq5sehmv3kP8o53tr9BxVcPE3GQCY, address:45.91.116.5:5310): protocol: storage node overloaded, request limit: 2000 Jun 22 00:30:58.210389994 (80h7m33.773712995s) [noise=true , address=104.36.82.13:6199 , piece_id=YP62N2TO446YR3QRG5YLXECD7TN6SJ4UNNLZSK5H4CMV6JIF4AKA, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:30:58.693037961 (80h7m34.256360966s) [noise=true , address=104.36.82.39:6756 , piece_id=OKKWMDJPZO6D5VRMGOE3GOCYGHM725JZTNOVYAWKJW3WYZPI4V5Q, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:30:59.124834814 (80h7m34.688157819s) [noise=true , address=104.36.82.15:6283 , piece_id=N7TH2EMIOPATVHCSNH2WXUGU7KMMEJ5LCHR22SWEHSA5OV4CUCXQ, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:31:00.629657982 (80h7m36.192980977s) [noise=true , address=104.36.82.42:6803 , piece_id=R626UFD326S2UBJEZ2WMHVR7EN4SK427VXPDJZOOZ32MX25JWZWQ, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:31:00.933110546 (80h7m36.496433536s) [noise=true , address=79.127.201.215:4326 , piece_id=USEWZDX6P52U6MCALTBNQG3SV7N2JRKNU6O23TLS6JEWAS4D7ARA, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:31:05.669167243 (80h7m41.232490237s) [noise=true , address=104.36.82.56:4261 , piece_id=EYOYOYLDACGMPGJYX6ILQT3XHMUTQNXZEDC7ANHLZXIKYVDQFFVQ, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:31:08.443582763 (80h7m44.006905763s) [noise=true , address=104.36.82.54:4184 , piece_id=IZICVGZFMAYLIJTNUJ2OFLZVNZ6IPPYDZO4BSTJJPLHVCVOE5EAA, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:31:26.459138014 (80h8m2.022461013s) [noise=true , address=104.36.82.53:4146 , piece_id=3V3DVGUOPJKDWIGQ2TU7M6VFYFVAS6ODY62ERGSU6LI3LSTWQ4YQ, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:31:27.456050336 (80h8m3.019373336s) [noise=true , address=104.36.82.52:4089 , piece_id=X2VG4S3UHVNWA4AQJS7FLKWNVLTOZ3AKWI5ZDH24QCA3ENQW7BXQ, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:31:55.270234251 (80h8m30.833557256s) [noise=true , address=104.36.82.11:6125 , piece_id=2CX7RJOKNQA6ODZZZXYPSCYAQY5P35TWX7I56CMYAPFLJE3JEO5Q, seg_pos={0 0}, upload=64142]: Done uploading piece. err: ecclient: upload failed (node:12vricofwx67FRaxAGLbmgT1TYyKkqfJbxdsyeEPL4R7kg8pKzT, address:104.36.82.11:6125): protocol: expected piece hash; context deadline exceeded; EOF Jun 22 00:31:55.270319418 (80h8m30.833642407s) [noise=true , address=104.36.82.55:4224 , piece_id=XF7H2BYSNNNZOS2T77VIN75N76AYUXAQEI6D4LRXACU2KU62V2TA, seg_pos={0 0}, upload=64142]: Done uploading piece. err: ecclient: upload failed (node:12EjKmUiv4pxkVCB4WoiXim4uFpxrBPyfiSS4efTY3HKSE3jb5y, address:104.36.82.55:4224): protocol: write tcp 159.223.143.39:39152->104.36.82.55:4224: use of closed network connection; write tcp 159.223.143.39:39152->104.36.82.55:4224: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:39152->104.36.82.55:4224: use of closed network connection Jun 22 00:31:55.270353811 (80h8m30.833676799s) [seg_pos={0 0}, upload=64142]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Jun 22 00:31:55.301099877 (80h8m30.864422884s) [upload=64142]: Done uploading segments... Jun 22 00:31:55.301121567 (80h8m30.864444566s) [upload=64142]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 36.460549033s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [...................X.X..............................X.......] duration: 27.620381361s FAIL (2025-06-22 00:36:44 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== ing upload Jun 22 00:30:55.303646184 (80h7m30.866969171s) [upload=64142]: Uploading segments... Jun 22 00:30:55.303706406 (80h7m30.867029416s) [upload=64142]: Got next segment. Inline: false Jun 22 00:30:55.303762479 (80h7m30.867085468s) [upload=64142]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 22 00:30:55.303768627 (80h7m30.867091636s) [upload=64142]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 22 00:30:55.349058140 (80h7m30.912381168s) [seg_pos={0 0}, upload=64142]: Begin upload segment... Jun 22 00:30:55.349113768 (80h7m30.912436772s) [seg_pos={0 0}, upload=64142]: Done begin upload segment. Jun 22 00:30:55.349143121 (80h7m30.912466131s) [noise=true , address=45.91.116.5:5310 , piece_id=AXJOKDZRPVNL7HFT4QKJWRAHFM2UCM3GEVQB2KNYD6CXB6TH4D4Q, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.349660917 (80h7m30.912983928s) [noise=true , address=77.90.31.10:4533 , piece_id=SE6YAKKFDL3BV677CDDUC32UNMPXJHTG23GWWYK5B4YIPKER6GAQ, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.350093077 (80h7m30.913416089s) [noise=true , address=79.127.201.215:4326 , piece_id=USEWZDX6P52U6MCALTBNQG3SV7N2JRKNU6O23TLS6JEWAS4D7ARA, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.350480894 (80h7m30.913803916s) [noise=true , address=104.36.82.13:6199 , piece_id=YP62N2TO446YR3QRG5YLXECD7TN6SJ4UNNLZSK5H4CMV6JIF4AKA, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.350669998 (80h7m30.913993011s) [noise=true , address=104.36.82.54:4184 , piece_id=IZICVGZFMAYLIJTNUJ2OFLZVNZ6IPPYDZO4BSTJJPLHVCVOE5EAA, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.351201352 (80h7m30.914524366s) [noise=true , address=104.36.82.55:4224 , piece_id=XF7H2BYSNNNZOS2T77VIN75N76AYUXAQEI6D4LRXACU2KU62V2TA, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.351668071 (80h7m30.914991083s) [noise=true , address=104.36.82.56:4261 , piece_id=EYOYOYLDACGMPGJYX6ILQT3XHMUTQNXZEDC7ANHLZXIKYVDQFFVQ, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.352193126 (80h7m30.915516138s) [noise=true , address=104.36.82.52:4089 , piece_id=X2VG4S3UHVNWA4AQJS7FLKWNVLTOZ3AKWI5ZDH24QCA3ENQW7BXQ, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.352587096 (80h7m30.915910103s) [noise=true , address=79.127.220.98:4402 , piece_id=TE2WQO5LK45AVMPIAIGVSIBOVZLC4FG4ESQ72X6KFYC7IAEN2DRA, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.353420253 (80h7m30.916743265s) [noise=true , address=104.36.82.11:6125 , piece_id=2CX7RJOKNQA6ODZZZXYPSCYAQY5P35TWX7I56CMYAPFLJE3JEO5Q, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.353791565 (80h7m30.917114578s) [noise=true , address=104.36.82.42:6803 , piece_id=R626UFD326S2UBJEZ2WMHVR7EN4SK427VXPDJZOOZ32MX25JWZWQ, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.354171226 (80h7m30.917494233s) [noise=true , address=104.36.82.14:6245 , piece_id=II36G46OA6TINLR244U6D7Q6LGV754UA5M62K3EX7ZOBESMJUEHA, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.354542282 (80h7m30.917865288s) [noise=true , address=79.127.205.235:5660 , piece_id=UGFPVS7VNB7U2YSFPSGYDIRZV5R2DHCN57PJP6VOXO52WIAYHLXA, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.354891655 (80h7m30.918214678s) [noise=true , address=104.36.82.15:6283 , piece_id=N7TH2EMIOPATVHCSNH2WXUGU7KMMEJ5LCHR22SWEHSA5OV4CUCXQ, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.355499503 (80h7m30.918822519s) [noise=true , address=77.90.31.14:4666 , piece_id=H4DTFDAVCN266PB235BJIQTQ32FBXTIB5AZSX3B4R2VIK554AFSQ, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.356575547 (80h7m30.919898539s) [noise=true , address=104.36.82.39:6756 , piece_id=OKKWMDJPZO6D5VRMGOE3GOCYGHM725JZTNOVYAWKJW3WYZPI4V5Q, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.357039590 (80h7m30.920362596s) [noise=true , address=104.36.82.53:4146 , piece_id=3V3DVGUOPJKDWIGQ2TU7M6VFYFVAS6ODY62ERGSU6LI3LSTWQ4YQ, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.360343566 (80h7m30.923666582s) [noise=true , address=104.36.82.9:6033 , piece_id=T5XA23LZELMSTWOLBSHOX4AHKDACF65OYCKTMZYTXDIBSXNA2T4A, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.452444064 (80h7m31.015767055s) [upload=64142]: Next returned nil segment Jun 22 00:30:55.452495148 (80h7m31.015818149s) [upload=64142]: Waiting for error group managing segments... Jun 22 00:30:55.532958808 (80h7m31.096281883s) [noise=true , address=104.36.82.9:6033 , piece_id=T5XA23LZELMSTWOLBSHOX4AHKDACF65OYCKTMZYTXDIBSXNA2T4A, seg_pos={0 0}, upload=64142]: Done uploading piece. err: ecclient: upload failed (node:1SV3LrcQW16wLewDzfyUWFB6sUgnERNJVdhz72CJmyde95M79C, address:104.36.82.9:6033): protocol: storage node overloaded, request limit: 2000 Jun 22 00:30:55.547802184 (80h7m31.111125188s) [noise=true , address=79.127.220.98:4402 , piece_id=TE2WQO5LK45AVMPIAIGVSIBOVZLC4FG4ESQ72X6KFYC7IAEN2DRA, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:30:55.554223621 (80h7m31.117546616s) [noise=true , address=77.90.31.14:4666 , piece_id=H4DTFDAVCN266PB235BJIQTQ32FBXTIB5AZSX3B4R2VIK554AFSQ, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:30:55.762514255 (80h7m31.325837245s) [noise=true , address=104.36.82.14:6245 , piece_id=II36G46OA6TINLR244U6D7Q6LGV754UA5M62K3EX7ZOBESMJUEHA, seg_pos={0 0}, upload=64142]: Done uploading piece. err: ecclient: upload failed (node:1Zu8teNPTXn638grHKwtJqz4UEY7oFYLx1DHvuVP6RZXcatVqX, address:104.36.82.14:6245): protocol: storage node overloaded, request limit: 2000 Jun 22 00:30:55.773669583 (80h7m31.3369926s) [noise=true , address=79.127.205.235:5660 , piece_id=UGFPVS7VNB7U2YSFPSGYDIRZV5R2DHCN57PJP6VOXO52WIAYHLXA, seg_pos={0 0}, upload=64142]: Done uploading piece. err: ecclient: upload failed (node:12E6uCX9KL4Ud7YpxMXepvMXuqTFJCiSXByDjW94Am3U9H2Nf7f, address:79.127.205.235:5660): protocol: storage node overloaded, request limit: 2000 Jun 22 00:30:56.968366655 (80h7m32.53168964s) [noise=true , address=77.90.31.10:4533 , piece_id=SE6YAKKFDL3BV677CDDUC32UNMPXJHTG23GWWYK5B4YIPKER6GAQ, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:30:57.403572874 (80h7m32.966895883s) [noise=true , address=45.91.116.5:5310 , piece_id=AXJOKDZRPVNL7HFT4QKJWRAHFM2UCM3GEVQB2KNYD6CXB6TH4D4Q, seg_pos={0 0}, upload=64142]: Done uploading piece. err: ecclient: upload failed (node:127RMUc62xRq9RU5z3XfkZsq5sehmv3kP8o53tr9BxVcPE3GQCY, address:45.91.116.5:5310): protocol: storage node overloaded, request limit: 2000 Jun 22 00:30:58.210389994 (80h7m33.773712995s) [noise=true , address=104.36.82.13:6199 , piece_id=YP62N2TO446YR3QRG5YLXECD7TN6SJ4UNNLZSK5H4CMV6JIF4AKA, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:30:58.693037961 (80h7m34.256360966s) [noise=true , address=104.36.82.39:6756 , piece_id=OKKWMDJPZO6D5VRMGOE3GOCYGHM725JZTNOVYAWKJW3WYZPI4V5Q, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:30:59.124834814 (80h7m34.688157819s) [noise=true , address=104.36.82.15:6283 , piece_id=N7TH2EMIOPATVHCSNH2WXUGU7KMMEJ5LCHR22SWEHSA5OV4CUCXQ, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:31:00.629657982 (80h7m36.192980977s) [noise=true , address=104.36.82.42:6803 , piece_id=R626UFD326S2UBJEZ2WMHVR7EN4SK427VXPDJZOOZ32MX25JWZWQ, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:31:00.933110546 (80h7m36.496433536s) [noise=true , address=79.127.201.215:4326 , piece_id=USEWZDX6P52U6MCALTBNQG3SV7N2JRKNU6O23TLS6JEWAS4D7ARA, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:31:05.669167243 (80h7m41.232490237s) [noise=true , address=104.36.82.56:4261 , piece_id=EYOYOYLDACGMPGJYX6ILQT3XHMUTQNXZEDC7ANHLZXIKYVDQFFVQ, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:31:08.443582763 (80h7m44.006905763s) [noise=true , address=104.36.82.54:4184 , piece_id=IZICVGZFMAYLIJTNUJ2OFLZVNZ6IPPYDZO4BSTJJPLHVCVOE5EAA, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:31:26.459138014 (80h8m2.022461013s) [noise=true , address=104.36.82.53:4146 , piece_id=3V3DVGUOPJKDWIGQ2TU7M6VFYFVAS6ODY62ERGSU6LI3LSTWQ4YQ, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:31:27.456050336 (80h8m3.019373336s) [noise=true , address=104.36.82.52:4089 , piece_id=X2VG4S3UHVNWA4AQJS7FLKWNVLTOZ3AKWI5ZDH24QCA3ENQW7BXQ, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:31:55.270234251 (80h8m30.833557256s) [noise=true , address=104.36.82.11:6125 , piece_id=2CX7RJOKNQA6ODZZZXYPSCYAQY5P35TWX7I56CMYAPFLJE3JEO5Q, seg_pos={0 0}, upload=64142]: Done uploading piece. err: ecclient: upload failed (node:12vricofwx67FRaxAGLbmgT1TYyKkqfJbxdsyeEPL4R7kg8pKzT, address:104.36.82.11:6125): protocol: expected piece hash; context deadline exceeded; EOF Jun 22 00:31:55.270319418 (80h8m30.833642407s) [noise=true , address=104.36.82.55:4224 , piece_id=XF7H2BYSNNNZOS2T77VIN75N76AYUXAQEI6D4LRXACU2KU62V2TA, seg_pos={0 0}, upload=64142]: Done uploading piece. err: ecclient: upload failed (node:12EjKmUiv4pxkVCB4WoiXim4uFpxrBPyfiSS4efTY3HKSE3jb5y, address:104.36.82.55:4224): protocol: write tcp 159.223.143.39:39152->104.36.82.55:4224: use of closed network connection; write tcp 159.223.143.39:39152->104.36.82.55:4224: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:39152->104.36.82.55:4224: use of closed network connection Jun 22 00:31:55.270353811 (80h8m30.833676799s) [seg_pos={0 0}, upload=64142]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Jun 22 00:31:55.301099877 (80h8m30.864422884s) [upload=64142]: Done uploading segments... Jun 22 00:31:55.301121567 (80h8m30.864444566s) [upload=64142]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 27.620409302s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [.....................X.........X...................X........] duration: 15.145485614s FAIL (2025-06-22 00:36:26 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 15.1455095s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [....................X.X..............................X......] duration: 22.654524233s FAIL (2025-06-22 00:36:16 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== ing upload Jun 22 00:30:55.303646184 (80h7m30.866969171s) [upload=64142]: Uploading segments... Jun 22 00:30:55.303706406 (80h7m30.867029416s) [upload=64142]: Got next segment. Inline: false Jun 22 00:30:55.303762479 (80h7m30.867085468s) [upload=64142]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 22 00:30:55.303768627 (80h7m30.867091636s) [upload=64142]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 22 00:30:55.349058140 (80h7m30.912381168s) [seg_pos={0 0}, upload=64142]: Begin upload segment... Jun 22 00:30:55.349113768 (80h7m30.912436772s) [seg_pos={0 0}, upload=64142]: Done begin upload segment. Jun 22 00:30:55.349143121 (80h7m30.912466131s) [noise=true , address=45.91.116.5:5310 , piece_id=AXJOKDZRPVNL7HFT4QKJWRAHFM2UCM3GEVQB2KNYD6CXB6TH4D4Q, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.349660917 (80h7m30.912983928s) [noise=true , address=77.90.31.10:4533 , piece_id=SE6YAKKFDL3BV677CDDUC32UNMPXJHTG23GWWYK5B4YIPKER6GAQ, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.350093077 (80h7m30.913416089s) [noise=true , address=79.127.201.215:4326 , piece_id=USEWZDX6P52U6MCALTBNQG3SV7N2JRKNU6O23TLS6JEWAS4D7ARA, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.350480894 (80h7m30.913803916s) [noise=true , address=104.36.82.13:6199 , piece_id=YP62N2TO446YR3QRG5YLXECD7TN6SJ4UNNLZSK5H4CMV6JIF4AKA, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.350669998 (80h7m30.913993011s) [noise=true , address=104.36.82.54:4184 , piece_id=IZICVGZFMAYLIJTNUJ2OFLZVNZ6IPPYDZO4BSTJJPLHVCVOE5EAA, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.351201352 (80h7m30.914524366s) [noise=true , address=104.36.82.55:4224 , piece_id=XF7H2BYSNNNZOS2T77VIN75N76AYUXAQEI6D4LRXACU2KU62V2TA, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.351668071 (80h7m30.914991083s) [noise=true , address=104.36.82.56:4261 , piece_id=EYOYOYLDACGMPGJYX6ILQT3XHMUTQNXZEDC7ANHLZXIKYVDQFFVQ, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.352193126 (80h7m30.915516138s) [noise=true , address=104.36.82.52:4089 , piece_id=X2VG4S3UHVNWA4AQJS7FLKWNVLTOZ3AKWI5ZDH24QCA3ENQW7BXQ, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.352587096 (80h7m30.915910103s) [noise=true , address=79.127.220.98:4402 , piece_id=TE2WQO5LK45AVMPIAIGVSIBOVZLC4FG4ESQ72X6KFYC7IAEN2DRA, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.353420253 (80h7m30.916743265s) [noise=true , address=104.36.82.11:6125 , piece_id=2CX7RJOKNQA6ODZZZXYPSCYAQY5P35TWX7I56CMYAPFLJE3JEO5Q, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.353791565 (80h7m30.917114578s) [noise=true , address=104.36.82.42:6803 , piece_id=R626UFD326S2UBJEZ2WMHVR7EN4SK427VXPDJZOOZ32MX25JWZWQ, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.354171226 (80h7m30.917494233s) [noise=true , address=104.36.82.14:6245 , piece_id=II36G46OA6TINLR244U6D7Q6LGV754UA5M62K3EX7ZOBESMJUEHA, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.354542282 (80h7m30.917865288s) [noise=true , address=79.127.205.235:5660 , piece_id=UGFPVS7VNB7U2YSFPSGYDIRZV5R2DHCN57PJP6VOXO52WIAYHLXA, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.354891655 (80h7m30.918214678s) [noise=true , address=104.36.82.15:6283 , piece_id=N7TH2EMIOPATVHCSNH2WXUGU7KMMEJ5LCHR22SWEHSA5OV4CUCXQ, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.355499503 (80h7m30.918822519s) [noise=true , address=77.90.31.14:4666 , piece_id=H4DTFDAVCN266PB235BJIQTQ32FBXTIB5AZSX3B4R2VIK554AFSQ, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.356575547 (80h7m30.919898539s) [noise=true , address=104.36.82.39:6756 , piece_id=OKKWMDJPZO6D5VRMGOE3GOCYGHM725JZTNOVYAWKJW3WYZPI4V5Q, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.357039590 (80h7m30.920362596s) [noise=true , address=104.36.82.53:4146 , piece_id=3V3DVGUOPJKDWIGQ2TU7M6VFYFVAS6ODY62ERGSU6LI3LSTWQ4YQ, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.360343566 (80h7m30.923666582s) [noise=true , address=104.36.82.9:6033 , piece_id=T5XA23LZELMSTWOLBSHOX4AHKDACF65OYCKTMZYTXDIBSXNA2T4A, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.452444064 (80h7m31.015767055s) [upload=64142]: Next returned nil segment Jun 22 00:30:55.452495148 (80h7m31.015818149s) [upload=64142]: Waiting for error group managing segments... Jun 22 00:30:55.532958808 (80h7m31.096281883s) [noise=true , address=104.36.82.9:6033 , piece_id=T5XA23LZELMSTWOLBSHOX4AHKDACF65OYCKTMZYTXDIBSXNA2T4A, seg_pos={0 0}, upload=64142]: Done uploading piece. err: ecclient: upload failed (node:1SV3LrcQW16wLewDzfyUWFB6sUgnERNJVdhz72CJmyde95M79C, address:104.36.82.9:6033): protocol: storage node overloaded, request limit: 2000 Jun 22 00:30:55.547802184 (80h7m31.111125188s) [noise=true , address=79.127.220.98:4402 , piece_id=TE2WQO5LK45AVMPIAIGVSIBOVZLC4FG4ESQ72X6KFYC7IAEN2DRA, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:30:55.554223621 (80h7m31.117546616s) [noise=true , address=77.90.31.14:4666 , piece_id=H4DTFDAVCN266PB235BJIQTQ32FBXTIB5AZSX3B4R2VIK554AFSQ, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:30:55.762514255 (80h7m31.325837245s) [noise=true , address=104.36.82.14:6245 , piece_id=II36G46OA6TINLR244U6D7Q6LGV754UA5M62K3EX7ZOBESMJUEHA, seg_pos={0 0}, upload=64142]: Done uploading piece. err: ecclient: upload failed (node:1Zu8teNPTXn638grHKwtJqz4UEY7oFYLx1DHvuVP6RZXcatVqX, address:104.36.82.14:6245): protocol: storage node overloaded, request limit: 2000 Jun 22 00:30:55.773669583 (80h7m31.3369926s) [noise=true , address=79.127.205.235:5660 , piece_id=UGFPVS7VNB7U2YSFPSGYDIRZV5R2DHCN57PJP6VOXO52WIAYHLXA, seg_pos={0 0}, upload=64142]: Done uploading piece. err: ecclient: upload failed (node:12E6uCX9KL4Ud7YpxMXepvMXuqTFJCiSXByDjW94Am3U9H2Nf7f, address:79.127.205.235:5660): protocol: storage node overloaded, request limit: 2000 Jun 22 00:30:56.968366655 (80h7m32.53168964s) [noise=true , address=77.90.31.10:4533 , piece_id=SE6YAKKFDL3BV677CDDUC32UNMPXJHTG23GWWYK5B4YIPKER6GAQ, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:30:57.403572874 (80h7m32.966895883s) [noise=true , address=45.91.116.5:5310 , piece_id=AXJOKDZRPVNL7HFT4QKJWRAHFM2UCM3GEVQB2KNYD6CXB6TH4D4Q, seg_pos={0 0}, upload=64142]: Done uploading piece. err: ecclient: upload failed (node:127RMUc62xRq9RU5z3XfkZsq5sehmv3kP8o53tr9BxVcPE3GQCY, address:45.91.116.5:5310): protocol: storage node overloaded, request limit: 2000 Jun 22 00:30:58.210389994 (80h7m33.773712995s) [noise=true , address=104.36.82.13:6199 , piece_id=YP62N2TO446YR3QRG5YLXECD7TN6SJ4UNNLZSK5H4CMV6JIF4AKA, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:30:58.693037961 (80h7m34.256360966s) [noise=true , address=104.36.82.39:6756 , piece_id=OKKWMDJPZO6D5VRMGOE3GOCYGHM725JZTNOVYAWKJW3WYZPI4V5Q, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:30:59.124834814 (80h7m34.688157819s) [noise=true , address=104.36.82.15:6283 , piece_id=N7TH2EMIOPATVHCSNH2WXUGU7KMMEJ5LCHR22SWEHSA5OV4CUCXQ, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:31:00.629657982 (80h7m36.192980977s) [noise=true , address=104.36.82.42:6803 , piece_id=R626UFD326S2UBJEZ2WMHVR7EN4SK427VXPDJZOOZ32MX25JWZWQ, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:31:00.933110546 (80h7m36.496433536s) [noise=true , address=79.127.201.215:4326 , piece_id=USEWZDX6P52U6MCALTBNQG3SV7N2JRKNU6O23TLS6JEWAS4D7ARA, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:31:05.669167243 (80h7m41.232490237s) [noise=true , address=104.36.82.56:4261 , piece_id=EYOYOYLDACGMPGJYX6ILQT3XHMUTQNXZEDC7ANHLZXIKYVDQFFVQ, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:31:08.443582763 (80h7m44.006905763s) [noise=true , address=104.36.82.54:4184 , piece_id=IZICVGZFMAYLIJTNUJ2OFLZVNZ6IPPYDZO4BSTJJPLHVCVOE5EAA, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:31:26.459138014 (80h8m2.022461013s) [noise=true , address=104.36.82.53:4146 , piece_id=3V3DVGUOPJKDWIGQ2TU7M6VFYFVAS6ODY62ERGSU6LI3LSTWQ4YQ, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:31:27.456050336 (80h8m3.019373336s) [noise=true , address=104.36.82.52:4089 , piece_id=X2VG4S3UHVNWA4AQJS7FLKWNVLTOZ3AKWI5ZDH24QCA3ENQW7BXQ, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:31:55.270234251 (80h8m30.833557256s) [noise=true , address=104.36.82.11:6125 , piece_id=2CX7RJOKNQA6ODZZZXYPSCYAQY5P35TWX7I56CMYAPFLJE3JEO5Q, seg_pos={0 0}, upload=64142]: Done uploading piece. err: ecclient: upload failed (node:12vricofwx67FRaxAGLbmgT1TYyKkqfJbxdsyeEPL4R7kg8pKzT, address:104.36.82.11:6125): protocol: expected piece hash; context deadline exceeded; EOF Jun 22 00:31:55.270319418 (80h8m30.833642407s) [noise=true , address=104.36.82.55:4224 , piece_id=XF7H2BYSNNNZOS2T77VIN75N76AYUXAQEI6D4LRXACU2KU62V2TA, seg_pos={0 0}, upload=64142]: Done uploading piece. err: ecclient: upload failed (node:12EjKmUiv4pxkVCB4WoiXim4uFpxrBPyfiSS4efTY3HKSE3jb5y, address:104.36.82.55:4224): protocol: write tcp 159.223.143.39:39152->104.36.82.55:4224: use of closed network connection; write tcp 159.223.143.39:39152->104.36.82.55:4224: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:39152->104.36.82.55:4224: use of closed network connection Jun 22 00:31:55.270353811 (80h8m30.833676799s) [seg_pos={0 0}, upload=64142]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Jun 22 00:31:55.301099877 (80h8m30.864422884s) [upload=64142]: Done uploading segments... Jun 22 00:31:55.301121567 (80h8m30.864444566s) [upload=64142]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 22.654576361s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [......................X.........X...................X.......] duration: 12.22285312s FAIL (2025-06-22 00:35:58 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 12.222903884s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [.....................X.X..............................X.....] duration: 18.128970027s FAIL (2025-06-22 00:35:26 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== ing upload Jun 22 00:30:55.303646184 (80h7m30.866969171s) [upload=64142]: Uploading segments... Jun 22 00:30:55.303706406 (80h7m30.867029416s) [upload=64142]: Got next segment. Inline: false Jun 22 00:30:55.303762479 (80h7m30.867085468s) [upload=64142]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 22 00:30:55.303768627 (80h7m30.867091636s) [upload=64142]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 22 00:30:55.349058140 (80h7m30.912381168s) [seg_pos={0 0}, upload=64142]: Begin upload segment... Jun 22 00:30:55.349113768 (80h7m30.912436772s) [seg_pos={0 0}, upload=64142]: Done begin upload segment. Jun 22 00:30:55.349143121 (80h7m30.912466131s) [noise=true , address=45.91.116.5:5310 , piece_id=AXJOKDZRPVNL7HFT4QKJWRAHFM2UCM3GEVQB2KNYD6CXB6TH4D4Q, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.349660917 (80h7m30.912983928s) [noise=true , address=77.90.31.10:4533 , piece_id=SE6YAKKFDL3BV677CDDUC32UNMPXJHTG23GWWYK5B4YIPKER6GAQ, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.350093077 (80h7m30.913416089s) [noise=true , address=79.127.201.215:4326 , piece_id=USEWZDX6P52U6MCALTBNQG3SV7N2JRKNU6O23TLS6JEWAS4D7ARA, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.350480894 (80h7m30.913803916s) [noise=true , address=104.36.82.13:6199 , piece_id=YP62N2TO446YR3QRG5YLXECD7TN6SJ4UNNLZSK5H4CMV6JIF4AKA, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.350669998 (80h7m30.913993011s) [noise=true , address=104.36.82.54:4184 , piece_id=IZICVGZFMAYLIJTNUJ2OFLZVNZ6IPPYDZO4BSTJJPLHVCVOE5EAA, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.351201352 (80h7m30.914524366s) [noise=true , address=104.36.82.55:4224 , piece_id=XF7H2BYSNNNZOS2T77VIN75N76AYUXAQEI6D4LRXACU2KU62V2TA, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.351668071 (80h7m30.914991083s) [noise=true , address=104.36.82.56:4261 , piece_id=EYOYOYLDACGMPGJYX6ILQT3XHMUTQNXZEDC7ANHLZXIKYVDQFFVQ, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.352193126 (80h7m30.915516138s) [noise=true , address=104.36.82.52:4089 , piece_id=X2VG4S3UHVNWA4AQJS7FLKWNVLTOZ3AKWI5ZDH24QCA3ENQW7BXQ, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.352587096 (80h7m30.915910103s) [noise=true , address=79.127.220.98:4402 , piece_id=TE2WQO5LK45AVMPIAIGVSIBOVZLC4FG4ESQ72X6KFYC7IAEN2DRA, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.353420253 (80h7m30.916743265s) [noise=true , address=104.36.82.11:6125 , piece_id=2CX7RJOKNQA6ODZZZXYPSCYAQY5P35TWX7I56CMYAPFLJE3JEO5Q, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.353791565 (80h7m30.917114578s) [noise=true , address=104.36.82.42:6803 , piece_id=R626UFD326S2UBJEZ2WMHVR7EN4SK427VXPDJZOOZ32MX25JWZWQ, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.354171226 (80h7m30.917494233s) [noise=true , address=104.36.82.14:6245 , piece_id=II36G46OA6TINLR244U6D7Q6LGV754UA5M62K3EX7ZOBESMJUEHA, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.354542282 (80h7m30.917865288s) [noise=true , address=79.127.205.235:5660 , piece_id=UGFPVS7VNB7U2YSFPSGYDIRZV5R2DHCN57PJP6VOXO52WIAYHLXA, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.354891655 (80h7m30.918214678s) [noise=true , address=104.36.82.15:6283 , piece_id=N7TH2EMIOPATVHCSNH2WXUGU7KMMEJ5LCHR22SWEHSA5OV4CUCXQ, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.355499503 (80h7m30.918822519s) [noise=true , address=77.90.31.14:4666 , piece_id=H4DTFDAVCN266PB235BJIQTQ32FBXTIB5AZSX3B4R2VIK554AFSQ, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.356575547 (80h7m30.919898539s) [noise=true , address=104.36.82.39:6756 , piece_id=OKKWMDJPZO6D5VRMGOE3GOCYGHM725JZTNOVYAWKJW3WYZPI4V5Q, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.357039590 (80h7m30.920362596s) [noise=true , address=104.36.82.53:4146 , piece_id=3V3DVGUOPJKDWIGQ2TU7M6VFYFVAS6ODY62ERGSU6LI3LSTWQ4YQ, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.360343566 (80h7m30.923666582s) [noise=true , address=104.36.82.9:6033 , piece_id=T5XA23LZELMSTWOLBSHOX4AHKDACF65OYCKTMZYTXDIBSXNA2T4A, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.452444064 (80h7m31.015767055s) [upload=64142]: Next returned nil segment Jun 22 00:30:55.452495148 (80h7m31.015818149s) [upload=64142]: Waiting for error group managing segments... Jun 22 00:30:55.532958808 (80h7m31.096281883s) [noise=true , address=104.36.82.9:6033 , piece_id=T5XA23LZELMSTWOLBSHOX4AHKDACF65OYCKTMZYTXDIBSXNA2T4A, seg_pos={0 0}, upload=64142]: Done uploading piece. err: ecclient: upload failed (node:1SV3LrcQW16wLewDzfyUWFB6sUgnERNJVdhz72CJmyde95M79C, address:104.36.82.9:6033): protocol: storage node overloaded, request limit: 2000 Jun 22 00:30:55.547802184 (80h7m31.111125188s) [noise=true , address=79.127.220.98:4402 , piece_id=TE2WQO5LK45AVMPIAIGVSIBOVZLC4FG4ESQ72X6KFYC7IAEN2DRA, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:30:55.554223621 (80h7m31.117546616s) [noise=true , address=77.90.31.14:4666 , piece_id=H4DTFDAVCN266PB235BJIQTQ32FBXTIB5AZSX3B4R2VIK554AFSQ, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:30:55.762514255 (80h7m31.325837245s) [noise=true , address=104.36.82.14:6245 , piece_id=II36G46OA6TINLR244U6D7Q6LGV754UA5M62K3EX7ZOBESMJUEHA, seg_pos={0 0}, upload=64142]: Done uploading piece. err: ecclient: upload failed (node:1Zu8teNPTXn638grHKwtJqz4UEY7oFYLx1DHvuVP6RZXcatVqX, address:104.36.82.14:6245): protocol: storage node overloaded, request limit: 2000 Jun 22 00:30:55.773669583 (80h7m31.3369926s) [noise=true , address=79.127.205.235:5660 , piece_id=UGFPVS7VNB7U2YSFPSGYDIRZV5R2DHCN57PJP6VOXO52WIAYHLXA, seg_pos={0 0}, upload=64142]: Done uploading piece. err: ecclient: upload failed (node:12E6uCX9KL4Ud7YpxMXepvMXuqTFJCiSXByDjW94Am3U9H2Nf7f, address:79.127.205.235:5660): protocol: storage node overloaded, request limit: 2000 Jun 22 00:30:56.968366655 (80h7m32.53168964s) [noise=true , address=77.90.31.10:4533 , piece_id=SE6YAKKFDL3BV677CDDUC32UNMPXJHTG23GWWYK5B4YIPKER6GAQ, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:30:57.403572874 (80h7m32.966895883s) [noise=true , address=45.91.116.5:5310 , piece_id=AXJOKDZRPVNL7HFT4QKJWRAHFM2UCM3GEVQB2KNYD6CXB6TH4D4Q, seg_pos={0 0}, upload=64142]: Done uploading piece. err: ecclient: upload failed (node:127RMUc62xRq9RU5z3XfkZsq5sehmv3kP8o53tr9BxVcPE3GQCY, address:45.91.116.5:5310): protocol: storage node overloaded, request limit: 2000 Jun 22 00:30:58.210389994 (80h7m33.773712995s) [noise=true , address=104.36.82.13:6199 , piece_id=YP62N2TO446YR3QRG5YLXECD7TN6SJ4UNNLZSK5H4CMV6JIF4AKA, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:30:58.693037961 (80h7m34.256360966s) [noise=true , address=104.36.82.39:6756 , piece_id=OKKWMDJPZO6D5VRMGOE3GOCYGHM725JZTNOVYAWKJW3WYZPI4V5Q, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:30:59.124834814 (80h7m34.688157819s) [noise=true , address=104.36.82.15:6283 , piece_id=N7TH2EMIOPATVHCSNH2WXUGU7KMMEJ5LCHR22SWEHSA5OV4CUCXQ, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:31:00.629657982 (80h7m36.192980977s) [noise=true , address=104.36.82.42:6803 , piece_id=R626UFD326S2UBJEZ2WMHVR7EN4SK427VXPDJZOOZ32MX25JWZWQ, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:31:00.933110546 (80h7m36.496433536s) [noise=true , address=79.127.201.215:4326 , piece_id=USEWZDX6P52U6MCALTBNQG3SV7N2JRKNU6O23TLS6JEWAS4D7ARA, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:31:05.669167243 (80h7m41.232490237s) [noise=true , address=104.36.82.56:4261 , piece_id=EYOYOYLDACGMPGJYX6ILQT3XHMUTQNXZEDC7ANHLZXIKYVDQFFVQ, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:31:08.443582763 (80h7m44.006905763s) [noise=true , address=104.36.82.54:4184 , piece_id=IZICVGZFMAYLIJTNUJ2OFLZVNZ6IPPYDZO4BSTJJPLHVCVOE5EAA, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:31:26.459138014 (80h8m2.022461013s) [noise=true , address=104.36.82.53:4146 , piece_id=3V3DVGUOPJKDWIGQ2TU7M6VFYFVAS6ODY62ERGSU6LI3LSTWQ4YQ, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:31:27.456050336 (80h8m3.019373336s) [noise=true , address=104.36.82.52:4089 , piece_id=X2VG4S3UHVNWA4AQJS7FLKWNVLTOZ3AKWI5ZDH24QCA3ENQW7BXQ, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:31:55.270234251 (80h8m30.833557256s) [noise=true , address=104.36.82.11:6125 , piece_id=2CX7RJOKNQA6ODZZZXYPSCYAQY5P35TWX7I56CMYAPFLJE3JEO5Q, seg_pos={0 0}, upload=64142]: Done uploading piece. err: ecclient: upload failed (node:12vricofwx67FRaxAGLbmgT1TYyKkqfJbxdsyeEPL4R7kg8pKzT, address:104.36.82.11:6125): protocol: expected piece hash; context deadline exceeded; EOF Jun 22 00:31:55.270319418 (80h8m30.833642407s) [noise=true , address=104.36.82.55:4224 , piece_id=XF7H2BYSNNNZOS2T77VIN75N76AYUXAQEI6D4LRXACU2KU62V2TA, seg_pos={0 0}, upload=64142]: Done uploading piece. err: ecclient: upload failed (node:12EjKmUiv4pxkVCB4WoiXim4uFpxrBPyfiSS4efTY3HKSE3jb5y, address:104.36.82.55:4224): protocol: write tcp 159.223.143.39:39152->104.36.82.55:4224: use of closed network connection; write tcp 159.223.143.39:39152->104.36.82.55:4224: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:39152->104.36.82.55:4224: use of closed network connection Jun 22 00:31:55.270353811 (80h8m30.833676799s) [seg_pos={0 0}, upload=64142]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Jun 22 00:31:55.301099877 (80h8m30.864422884s) [upload=64142]: Done uploading segments... Jun 22 00:31:55.301121567 (80h8m30.864444566s) [upload=64142]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 18.129019909s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [.......................X.........X...................X......] duration: 21.493039022s FAIL (2025-06-22 00:35:19 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 21.493058768s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [........................X.........X...................X.....] duration: 9.457573138s FAIL (2025-06-22 00:34:34 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 9.457632163s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [......................X.X..............................X....] duration: 26.53570595s FAIL (2025-06-22 00:34:22 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== ing upload Jun 22 00:30:55.303646184 (80h7m30.866969171s) [upload=64142]: Uploading segments... Jun 22 00:30:55.303706406 (80h7m30.867029416s) [upload=64142]: Got next segment. Inline: false Jun 22 00:30:55.303762479 (80h7m30.867085468s) [upload=64142]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 22 00:30:55.303768627 (80h7m30.867091636s) [upload=64142]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 22 00:30:55.349058140 (80h7m30.912381168s) [seg_pos={0 0}, upload=64142]: Begin upload segment... Jun 22 00:30:55.349113768 (80h7m30.912436772s) [seg_pos={0 0}, upload=64142]: Done begin upload segment. Jun 22 00:30:55.349143121 (80h7m30.912466131s) [noise=true , address=45.91.116.5:5310 , piece_id=AXJOKDZRPVNL7HFT4QKJWRAHFM2UCM3GEVQB2KNYD6CXB6TH4D4Q, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.349660917 (80h7m30.912983928s) [noise=true , address=77.90.31.10:4533 , piece_id=SE6YAKKFDL3BV677CDDUC32UNMPXJHTG23GWWYK5B4YIPKER6GAQ, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.350093077 (80h7m30.913416089s) [noise=true , address=79.127.201.215:4326 , piece_id=USEWZDX6P52U6MCALTBNQG3SV7N2JRKNU6O23TLS6JEWAS4D7ARA, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.350480894 (80h7m30.913803916s) [noise=true , address=104.36.82.13:6199 , piece_id=YP62N2TO446YR3QRG5YLXECD7TN6SJ4UNNLZSK5H4CMV6JIF4AKA, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.350669998 (80h7m30.913993011s) [noise=true , address=104.36.82.54:4184 , piece_id=IZICVGZFMAYLIJTNUJ2OFLZVNZ6IPPYDZO4BSTJJPLHVCVOE5EAA, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.351201352 (80h7m30.914524366s) [noise=true , address=104.36.82.55:4224 , piece_id=XF7H2BYSNNNZOS2T77VIN75N76AYUXAQEI6D4LRXACU2KU62V2TA, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.351668071 (80h7m30.914991083s) [noise=true , address=104.36.82.56:4261 , piece_id=EYOYOYLDACGMPGJYX6ILQT3XHMUTQNXZEDC7ANHLZXIKYVDQFFVQ, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.352193126 (80h7m30.915516138s) [noise=true , address=104.36.82.52:4089 , piece_id=X2VG4S3UHVNWA4AQJS7FLKWNVLTOZ3AKWI5ZDH24QCA3ENQW7BXQ, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.352587096 (80h7m30.915910103s) [noise=true , address=79.127.220.98:4402 , piece_id=TE2WQO5LK45AVMPIAIGVSIBOVZLC4FG4ESQ72X6KFYC7IAEN2DRA, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.353420253 (80h7m30.916743265s) [noise=true , address=104.36.82.11:6125 , piece_id=2CX7RJOKNQA6ODZZZXYPSCYAQY5P35TWX7I56CMYAPFLJE3JEO5Q, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.353791565 (80h7m30.917114578s) [noise=true , address=104.36.82.42:6803 , piece_id=R626UFD326S2UBJEZ2WMHVR7EN4SK427VXPDJZOOZ32MX25JWZWQ, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.354171226 (80h7m30.917494233s) [noise=true , address=104.36.82.14:6245 , piece_id=II36G46OA6TINLR244U6D7Q6LGV754UA5M62K3EX7ZOBESMJUEHA, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.354542282 (80h7m30.917865288s) [noise=true , address=79.127.205.235:5660 , piece_id=UGFPVS7VNB7U2YSFPSGYDIRZV5R2DHCN57PJP6VOXO52WIAYHLXA, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.354891655 (80h7m30.918214678s) [noise=true , address=104.36.82.15:6283 , piece_id=N7TH2EMIOPATVHCSNH2WXUGU7KMMEJ5LCHR22SWEHSA5OV4CUCXQ, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.355499503 (80h7m30.918822519s) [noise=true , address=77.90.31.14:4666 , piece_id=H4DTFDAVCN266PB235BJIQTQ32FBXTIB5AZSX3B4R2VIK554AFSQ, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.356575547 (80h7m30.919898539s) [noise=true , address=104.36.82.39:6756 , piece_id=OKKWMDJPZO6D5VRMGOE3GOCYGHM725JZTNOVYAWKJW3WYZPI4V5Q, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.357039590 (80h7m30.920362596s) [noise=true , address=104.36.82.53:4146 , piece_id=3V3DVGUOPJKDWIGQ2TU7M6VFYFVAS6ODY62ERGSU6LI3LSTWQ4YQ, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.360343566 (80h7m30.923666582s) [noise=true , address=104.36.82.9:6033 , piece_id=T5XA23LZELMSTWOLBSHOX4AHKDACF65OYCKTMZYTXDIBSXNA2T4A, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.452444064 (80h7m31.015767055s) [upload=64142]: Next returned nil segment Jun 22 00:30:55.452495148 (80h7m31.015818149s) [upload=64142]: Waiting for error group managing segments... Jun 22 00:30:55.532958808 (80h7m31.096281883s) [noise=true , address=104.36.82.9:6033 , piece_id=T5XA23LZELMSTWOLBSHOX4AHKDACF65OYCKTMZYTXDIBSXNA2T4A, seg_pos={0 0}, upload=64142]: Done uploading piece. err: ecclient: upload failed (node:1SV3LrcQW16wLewDzfyUWFB6sUgnERNJVdhz72CJmyde95M79C, address:104.36.82.9:6033): protocol: storage node overloaded, request limit: 2000 Jun 22 00:30:55.547802184 (80h7m31.111125188s) [noise=true , address=79.127.220.98:4402 , piece_id=TE2WQO5LK45AVMPIAIGVSIBOVZLC4FG4ESQ72X6KFYC7IAEN2DRA, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:30:55.554223621 (80h7m31.117546616s) [noise=true , address=77.90.31.14:4666 , piece_id=H4DTFDAVCN266PB235BJIQTQ32FBXTIB5AZSX3B4R2VIK554AFSQ, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:30:55.762514255 (80h7m31.325837245s) [noise=true , address=104.36.82.14:6245 , piece_id=II36G46OA6TINLR244U6D7Q6LGV754UA5M62K3EX7ZOBESMJUEHA, seg_pos={0 0}, upload=64142]: Done uploading piece. err: ecclient: upload failed (node:1Zu8teNPTXn638grHKwtJqz4UEY7oFYLx1DHvuVP6RZXcatVqX, address:104.36.82.14:6245): protocol: storage node overloaded, request limit: 2000 Jun 22 00:30:55.773669583 (80h7m31.3369926s) [noise=true , address=79.127.205.235:5660 , piece_id=UGFPVS7VNB7U2YSFPSGYDIRZV5R2DHCN57PJP6VOXO52WIAYHLXA, seg_pos={0 0}, upload=64142]: Done uploading piece. err: ecclient: upload failed (node:12E6uCX9KL4Ud7YpxMXepvMXuqTFJCiSXByDjW94Am3U9H2Nf7f, address:79.127.205.235:5660): protocol: storage node overloaded, request limit: 2000 Jun 22 00:30:56.968366655 (80h7m32.53168964s) [noise=true , address=77.90.31.10:4533 , piece_id=SE6YAKKFDL3BV677CDDUC32UNMPXJHTG23GWWYK5B4YIPKER6GAQ, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:30:57.403572874 (80h7m32.966895883s) [noise=true , address=45.91.116.5:5310 , piece_id=AXJOKDZRPVNL7HFT4QKJWRAHFM2UCM3GEVQB2KNYD6CXB6TH4D4Q, seg_pos={0 0}, upload=64142]: Done uploading piece. err: ecclient: upload failed (node:127RMUc62xRq9RU5z3XfkZsq5sehmv3kP8o53tr9BxVcPE3GQCY, address:45.91.116.5:5310): protocol: storage node overloaded, request limit: 2000 Jun 22 00:30:58.210389994 (80h7m33.773712995s) [noise=true , address=104.36.82.13:6199 , piece_id=YP62N2TO446YR3QRG5YLXECD7TN6SJ4UNNLZSK5H4CMV6JIF4AKA, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:30:58.693037961 (80h7m34.256360966s) [noise=true , address=104.36.82.39:6756 , piece_id=OKKWMDJPZO6D5VRMGOE3GOCYGHM725JZTNOVYAWKJW3WYZPI4V5Q, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:30:59.124834814 (80h7m34.688157819s) [noise=true , address=104.36.82.15:6283 , piece_id=N7TH2EMIOPATVHCSNH2WXUGU7KMMEJ5LCHR22SWEHSA5OV4CUCXQ, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:31:00.629657982 (80h7m36.192980977s) [noise=true , address=104.36.82.42:6803 , piece_id=R626UFD326S2UBJEZ2WMHVR7EN4SK427VXPDJZOOZ32MX25JWZWQ, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:31:00.933110546 (80h7m36.496433536s) [noise=true , address=79.127.201.215:4326 , piece_id=USEWZDX6P52U6MCALTBNQG3SV7N2JRKNU6O23TLS6JEWAS4D7ARA, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:31:05.669167243 (80h7m41.232490237s) [noise=true , address=104.36.82.56:4261 , piece_id=EYOYOYLDACGMPGJYX6ILQT3XHMUTQNXZEDC7ANHLZXIKYVDQFFVQ, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:31:08.443582763 (80h7m44.006905763s) [noise=true , address=104.36.82.54:4184 , piece_id=IZICVGZFMAYLIJTNUJ2OFLZVNZ6IPPYDZO4BSTJJPLHVCVOE5EAA, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:31:26.459138014 (80h8m2.022461013s) [noise=true , address=104.36.82.53:4146 , piece_id=3V3DVGUOPJKDWIGQ2TU7M6VFYFVAS6ODY62ERGSU6LI3LSTWQ4YQ, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:31:27.456050336 (80h8m3.019373336s) [noise=true , address=104.36.82.52:4089 , piece_id=X2VG4S3UHVNWA4AQJS7FLKWNVLTOZ3AKWI5ZDH24QCA3ENQW7BXQ, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:31:55.270234251 (80h8m30.833557256s) [noise=true , address=104.36.82.11:6125 , piece_id=2CX7RJOKNQA6ODZZZXYPSCYAQY5P35TWX7I56CMYAPFLJE3JEO5Q, seg_pos={0 0}, upload=64142]: Done uploading piece. err: ecclient: upload failed (node:12vricofwx67FRaxAGLbmgT1TYyKkqfJbxdsyeEPL4R7kg8pKzT, address:104.36.82.11:6125): protocol: expected piece hash; context deadline exceeded; EOF Jun 22 00:31:55.270319418 (80h8m30.833642407s) [noise=true , address=104.36.82.55:4224 , piece_id=XF7H2BYSNNNZOS2T77VIN75N76AYUXAQEI6D4LRXACU2KU62V2TA, seg_pos={0 0}, upload=64142]: Done uploading piece. err: ecclient: upload failed (node:12EjKmUiv4pxkVCB4WoiXim4uFpxrBPyfiSS4efTY3HKSE3jb5y, address:104.36.82.55:4224): protocol: write tcp 159.223.143.39:39152->104.36.82.55:4224: use of closed network connection; write tcp 159.223.143.39:39152->104.36.82.55:4224: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:39152->104.36.82.55:4224: use of closed network connection Jun 22 00:31:55.270353811 (80h8m30.833676799s) [seg_pos={0 0}, upload=64142]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Jun 22 00:31:55.301099877 (80h8m30.864422884s) [upload=64142]: Done uploading segments... Jun 22 00:31:55.301121567 (80h8m30.864444566s) [upload=64142]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 26.535731489s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [.........................X.........X...................X....] duration: 14.492606706s FAIL (2025-06-22 00:34:12 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 14.492625988s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [.......................X.X..............................X...] duration: 19.580223567s FAIL (2025-06-22 00:33:46 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== ing upload Jun 22 00:30:55.303646184 (80h7m30.866969171s) [upload=64142]: Uploading segments... Jun 22 00:30:55.303706406 (80h7m30.867029416s) [upload=64142]: Got next segment. Inline: false Jun 22 00:30:55.303762479 (80h7m30.867085468s) [upload=64142]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 22 00:30:55.303768627 (80h7m30.867091636s) [upload=64142]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 22 00:30:55.349058140 (80h7m30.912381168s) [seg_pos={0 0}, upload=64142]: Begin upload segment... Jun 22 00:30:55.349113768 (80h7m30.912436772s) [seg_pos={0 0}, upload=64142]: Done begin upload segment. Jun 22 00:30:55.349143121 (80h7m30.912466131s) [noise=true , address=45.91.116.5:5310 , piece_id=AXJOKDZRPVNL7HFT4QKJWRAHFM2UCM3GEVQB2KNYD6CXB6TH4D4Q, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.349660917 (80h7m30.912983928s) [noise=true , address=77.90.31.10:4533 , piece_id=SE6YAKKFDL3BV677CDDUC32UNMPXJHTG23GWWYK5B4YIPKER6GAQ, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.350093077 (80h7m30.913416089s) [noise=true , address=79.127.201.215:4326 , piece_id=USEWZDX6P52U6MCALTBNQG3SV7N2JRKNU6O23TLS6JEWAS4D7ARA, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.350480894 (80h7m30.913803916s) [noise=true , address=104.36.82.13:6199 , piece_id=YP62N2TO446YR3QRG5YLXECD7TN6SJ4UNNLZSK5H4CMV6JIF4AKA, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.350669998 (80h7m30.913993011s) [noise=true , address=104.36.82.54:4184 , piece_id=IZICVGZFMAYLIJTNUJ2OFLZVNZ6IPPYDZO4BSTJJPLHVCVOE5EAA, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.351201352 (80h7m30.914524366s) [noise=true , address=104.36.82.55:4224 , piece_id=XF7H2BYSNNNZOS2T77VIN75N76AYUXAQEI6D4LRXACU2KU62V2TA, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.351668071 (80h7m30.914991083s) [noise=true , address=104.36.82.56:4261 , piece_id=EYOYOYLDACGMPGJYX6ILQT3XHMUTQNXZEDC7ANHLZXIKYVDQFFVQ, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.352193126 (80h7m30.915516138s) [noise=true , address=104.36.82.52:4089 , piece_id=X2VG4S3UHVNWA4AQJS7FLKWNVLTOZ3AKWI5ZDH24QCA3ENQW7BXQ, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.352587096 (80h7m30.915910103s) [noise=true , address=79.127.220.98:4402 , piece_id=TE2WQO5LK45AVMPIAIGVSIBOVZLC4FG4ESQ72X6KFYC7IAEN2DRA, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.353420253 (80h7m30.916743265s) [noise=true , address=104.36.82.11:6125 , piece_id=2CX7RJOKNQA6ODZZZXYPSCYAQY5P35TWX7I56CMYAPFLJE3JEO5Q, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.353791565 (80h7m30.917114578s) [noise=true , address=104.36.82.42:6803 , piece_id=R626UFD326S2UBJEZ2WMHVR7EN4SK427VXPDJZOOZ32MX25JWZWQ, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.354171226 (80h7m30.917494233s) [noise=true , address=104.36.82.14:6245 , piece_id=II36G46OA6TINLR244U6D7Q6LGV754UA5M62K3EX7ZOBESMJUEHA, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.354542282 (80h7m30.917865288s) [noise=true , address=79.127.205.235:5660 , piece_id=UGFPVS7VNB7U2YSFPSGYDIRZV5R2DHCN57PJP6VOXO52WIAYHLXA, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.354891655 (80h7m30.918214678s) [noise=true , address=104.36.82.15:6283 , piece_id=N7TH2EMIOPATVHCSNH2WXUGU7KMMEJ5LCHR22SWEHSA5OV4CUCXQ, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.355499503 (80h7m30.918822519s) [noise=true , address=77.90.31.14:4666 , piece_id=H4DTFDAVCN266PB235BJIQTQ32FBXTIB5AZSX3B4R2VIK554AFSQ, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.356575547 (80h7m30.919898539s) [noise=true , address=104.36.82.39:6756 , piece_id=OKKWMDJPZO6D5VRMGOE3GOCYGHM725JZTNOVYAWKJW3WYZPI4V5Q, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.357039590 (80h7m30.920362596s) [noise=true , address=104.36.82.53:4146 , piece_id=3V3DVGUOPJKDWIGQ2TU7M6VFYFVAS6ODY62ERGSU6LI3LSTWQ4YQ, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.360343566 (80h7m30.923666582s) [noise=true , address=104.36.82.9:6033 , piece_id=T5XA23LZELMSTWOLBSHOX4AHKDACF65OYCKTMZYTXDIBSXNA2T4A, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.452444064 (80h7m31.015767055s) [upload=64142]: Next returned nil segment Jun 22 00:30:55.452495148 (80h7m31.015818149s) [upload=64142]: Waiting for error group managing segments... Jun 22 00:30:55.532958808 (80h7m31.096281883s) [noise=true , address=104.36.82.9:6033 , piece_id=T5XA23LZELMSTWOLBSHOX4AHKDACF65OYCKTMZYTXDIBSXNA2T4A, seg_pos={0 0}, upload=64142]: Done uploading piece. err: ecclient: upload failed (node:1SV3LrcQW16wLewDzfyUWFB6sUgnERNJVdhz72CJmyde95M79C, address:104.36.82.9:6033): protocol: storage node overloaded, request limit: 2000 Jun 22 00:30:55.547802184 (80h7m31.111125188s) [noise=true , address=79.127.220.98:4402 , piece_id=TE2WQO5LK45AVMPIAIGVSIBOVZLC4FG4ESQ72X6KFYC7IAEN2DRA, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:30:55.554223621 (80h7m31.117546616s) [noise=true , address=77.90.31.14:4666 , piece_id=H4DTFDAVCN266PB235BJIQTQ32FBXTIB5AZSX3B4R2VIK554AFSQ, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:30:55.762514255 (80h7m31.325837245s) [noise=true , address=104.36.82.14:6245 , piece_id=II36G46OA6TINLR244U6D7Q6LGV754UA5M62K3EX7ZOBESMJUEHA, seg_pos={0 0}, upload=64142]: Done uploading piece. err: ecclient: upload failed (node:1Zu8teNPTXn638grHKwtJqz4UEY7oFYLx1DHvuVP6RZXcatVqX, address:104.36.82.14:6245): protocol: storage node overloaded, request limit: 2000 Jun 22 00:30:55.773669583 (80h7m31.3369926s) [noise=true , address=79.127.205.235:5660 , piece_id=UGFPVS7VNB7U2YSFPSGYDIRZV5R2DHCN57PJP6VOXO52WIAYHLXA, seg_pos={0 0}, upload=64142]: Done uploading piece. err: ecclient: upload failed (node:12E6uCX9KL4Ud7YpxMXepvMXuqTFJCiSXByDjW94Am3U9H2Nf7f, address:79.127.205.235:5660): protocol: storage node overloaded, request limit: 2000 Jun 22 00:30:56.968366655 (80h7m32.53168964s) [noise=true , address=77.90.31.10:4533 , piece_id=SE6YAKKFDL3BV677CDDUC32UNMPXJHTG23GWWYK5B4YIPKER6GAQ, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:30:57.403572874 (80h7m32.966895883s) [noise=true , address=45.91.116.5:5310 , piece_id=AXJOKDZRPVNL7HFT4QKJWRAHFM2UCM3GEVQB2KNYD6CXB6TH4D4Q, seg_pos={0 0}, upload=64142]: Done uploading piece. err: ecclient: upload failed (node:127RMUc62xRq9RU5z3XfkZsq5sehmv3kP8o53tr9BxVcPE3GQCY, address:45.91.116.5:5310): protocol: storage node overloaded, request limit: 2000 Jun 22 00:30:58.210389994 (80h7m33.773712995s) [noise=true , address=104.36.82.13:6199 , piece_id=YP62N2TO446YR3QRG5YLXECD7TN6SJ4UNNLZSK5H4CMV6JIF4AKA, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:30:58.693037961 (80h7m34.256360966s) [noise=true , address=104.36.82.39:6756 , piece_id=OKKWMDJPZO6D5VRMGOE3GOCYGHM725JZTNOVYAWKJW3WYZPI4V5Q, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:30:59.124834814 (80h7m34.688157819s) [noise=true , address=104.36.82.15:6283 , piece_id=N7TH2EMIOPATVHCSNH2WXUGU7KMMEJ5LCHR22SWEHSA5OV4CUCXQ, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:31:00.629657982 (80h7m36.192980977s) [noise=true , address=104.36.82.42:6803 , piece_id=R626UFD326S2UBJEZ2WMHVR7EN4SK427VXPDJZOOZ32MX25JWZWQ, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:31:00.933110546 (80h7m36.496433536s) [noise=true , address=79.127.201.215:4326 , piece_id=USEWZDX6P52U6MCALTBNQG3SV7N2JRKNU6O23TLS6JEWAS4D7ARA, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:31:05.669167243 (80h7m41.232490237s) [noise=true , address=104.36.82.56:4261 , piece_id=EYOYOYLDACGMPGJYX6ILQT3XHMUTQNXZEDC7ANHLZXIKYVDQFFVQ, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:31:08.443582763 (80h7m44.006905763s) [noise=true , address=104.36.82.54:4184 , piece_id=IZICVGZFMAYLIJTNUJ2OFLZVNZ6IPPYDZO4BSTJJPLHVCVOE5EAA, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:31:26.459138014 (80h8m2.022461013s) [noise=true , address=104.36.82.53:4146 , piece_id=3V3DVGUOPJKDWIGQ2TU7M6VFYFVAS6ODY62ERGSU6LI3LSTWQ4YQ, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:31:27.456050336 (80h8m3.019373336s) [noise=true , address=104.36.82.52:4089 , piece_id=X2VG4S3UHVNWA4AQJS7FLKWNVLTOZ3AKWI5ZDH24QCA3ENQW7BXQ, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:31:55.270234251 (80h8m30.833557256s) [noise=true , address=104.36.82.11:6125 , piece_id=2CX7RJOKNQA6ODZZZXYPSCYAQY5P35TWX7I56CMYAPFLJE3JEO5Q, seg_pos={0 0}, upload=64142]: Done uploading piece. err: ecclient: upload failed (node:12vricofwx67FRaxAGLbmgT1TYyKkqfJbxdsyeEPL4R7kg8pKzT, address:104.36.82.11:6125): protocol: expected piece hash; context deadline exceeded; EOF Jun 22 00:31:55.270319418 (80h8m30.833642407s) [noise=true , address=104.36.82.55:4224 , piece_id=XF7H2BYSNNNZOS2T77VIN75N76AYUXAQEI6D4LRXACU2KU62V2TA, seg_pos={0 0}, upload=64142]: Done uploading piece. err: ecclient: upload failed (node:12EjKmUiv4pxkVCB4WoiXim4uFpxrBPyfiSS4efTY3HKSE3jb5y, address:104.36.82.55:4224): protocol: write tcp 159.223.143.39:39152->104.36.82.55:4224: use of closed network connection; write tcp 159.223.143.39:39152->104.36.82.55:4224: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:39152->104.36.82.55:4224: use of closed network connection Jun 22 00:31:55.270353811 (80h8m30.833676799s) [seg_pos={0 0}, upload=64142]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Jun 22 00:31:55.301099877 (80h8m30.864422884s) [upload=64142]: Done uploading segments... Jun 22 00:31:55.301121567 (80h8m30.864444566s) [upload=64142]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 19.580249664s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 4 out of 60 is failed (threshold=3) [X.........................X.........X...................X...] duration: 16.167150792s FAIL (2025-06-22 00:33:00 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 16.167178177s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [........................X.X..............................X..] duration: 19.481023091s FAIL (2025-06-22 00:32:48 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== ing upload Jun 22 00:30:55.303646184 (80h7m30.866969171s) [upload=64142]: Uploading segments... Jun 22 00:30:55.303706406 (80h7m30.867029416s) [upload=64142]: Got next segment. Inline: false Jun 22 00:30:55.303762479 (80h7m30.867085468s) [upload=64142]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 22 00:30:55.303768627 (80h7m30.867091636s) [upload=64142]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 22 00:30:55.349058140 (80h7m30.912381168s) [seg_pos={0 0}, upload=64142]: Begin upload segment... Jun 22 00:30:55.349113768 (80h7m30.912436772s) [seg_pos={0 0}, upload=64142]: Done begin upload segment. Jun 22 00:30:55.349143121 (80h7m30.912466131s) [noise=true , address=45.91.116.5:5310 , piece_id=AXJOKDZRPVNL7HFT4QKJWRAHFM2UCM3GEVQB2KNYD6CXB6TH4D4Q, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.349660917 (80h7m30.912983928s) [noise=true , address=77.90.31.10:4533 , piece_id=SE6YAKKFDL3BV677CDDUC32UNMPXJHTG23GWWYK5B4YIPKER6GAQ, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.350093077 (80h7m30.913416089s) [noise=true , address=79.127.201.215:4326 , piece_id=USEWZDX6P52U6MCALTBNQG3SV7N2JRKNU6O23TLS6JEWAS4D7ARA, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.350480894 (80h7m30.913803916s) [noise=true , address=104.36.82.13:6199 , piece_id=YP62N2TO446YR3QRG5YLXECD7TN6SJ4UNNLZSK5H4CMV6JIF4AKA, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.350669998 (80h7m30.913993011s) [noise=true , address=104.36.82.54:4184 , piece_id=IZICVGZFMAYLIJTNUJ2OFLZVNZ6IPPYDZO4BSTJJPLHVCVOE5EAA, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.351201352 (80h7m30.914524366s) [noise=true , address=104.36.82.55:4224 , piece_id=XF7H2BYSNNNZOS2T77VIN75N76AYUXAQEI6D4LRXACU2KU62V2TA, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.351668071 (80h7m30.914991083s) [noise=true , address=104.36.82.56:4261 , piece_id=EYOYOYLDACGMPGJYX6ILQT3XHMUTQNXZEDC7ANHLZXIKYVDQFFVQ, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.352193126 (80h7m30.915516138s) [noise=true , address=104.36.82.52:4089 , piece_id=X2VG4S3UHVNWA4AQJS7FLKWNVLTOZ3AKWI5ZDH24QCA3ENQW7BXQ, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.352587096 (80h7m30.915910103s) [noise=true , address=79.127.220.98:4402 , piece_id=TE2WQO5LK45AVMPIAIGVSIBOVZLC4FG4ESQ72X6KFYC7IAEN2DRA, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.353420253 (80h7m30.916743265s) [noise=true , address=104.36.82.11:6125 , piece_id=2CX7RJOKNQA6ODZZZXYPSCYAQY5P35TWX7I56CMYAPFLJE3JEO5Q, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.353791565 (80h7m30.917114578s) [noise=true , address=104.36.82.42:6803 , piece_id=R626UFD326S2UBJEZ2WMHVR7EN4SK427VXPDJZOOZ32MX25JWZWQ, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.354171226 (80h7m30.917494233s) [noise=true , address=104.36.82.14:6245 , piece_id=II36G46OA6TINLR244U6D7Q6LGV754UA5M62K3EX7ZOBESMJUEHA, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.354542282 (80h7m30.917865288s) [noise=true , address=79.127.205.235:5660 , piece_id=UGFPVS7VNB7U2YSFPSGYDIRZV5R2DHCN57PJP6VOXO52WIAYHLXA, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.354891655 (80h7m30.918214678s) [noise=true , address=104.36.82.15:6283 , piece_id=N7TH2EMIOPATVHCSNH2WXUGU7KMMEJ5LCHR22SWEHSA5OV4CUCXQ, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.355499503 (80h7m30.918822519s) [noise=true , address=77.90.31.14:4666 , piece_id=H4DTFDAVCN266PB235BJIQTQ32FBXTIB5AZSX3B4R2VIK554AFSQ, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.356575547 (80h7m30.919898539s) [noise=true , address=104.36.82.39:6756 , piece_id=OKKWMDJPZO6D5VRMGOE3GOCYGHM725JZTNOVYAWKJW3WYZPI4V5Q, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.357039590 (80h7m30.920362596s) [noise=true , address=104.36.82.53:4146 , piece_id=3V3DVGUOPJKDWIGQ2TU7M6VFYFVAS6ODY62ERGSU6LI3LSTWQ4YQ, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.360343566 (80h7m30.923666582s) [noise=true , address=104.36.82.9:6033 , piece_id=T5XA23LZELMSTWOLBSHOX4AHKDACF65OYCKTMZYTXDIBSXNA2T4A, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.452444064 (80h7m31.015767055s) [upload=64142]: Next returned nil segment Jun 22 00:30:55.452495148 (80h7m31.015818149s) [upload=64142]: Waiting for error group managing segments... Jun 22 00:30:55.532958808 (80h7m31.096281883s) [noise=true , address=104.36.82.9:6033 , piece_id=T5XA23LZELMSTWOLBSHOX4AHKDACF65OYCKTMZYTXDIBSXNA2T4A, seg_pos={0 0}, upload=64142]: Done uploading piece. err: ecclient: upload failed (node:1SV3LrcQW16wLewDzfyUWFB6sUgnERNJVdhz72CJmyde95M79C, address:104.36.82.9:6033): protocol: storage node overloaded, request limit: 2000 Jun 22 00:30:55.547802184 (80h7m31.111125188s) [noise=true , address=79.127.220.98:4402 , piece_id=TE2WQO5LK45AVMPIAIGVSIBOVZLC4FG4ESQ72X6KFYC7IAEN2DRA, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:30:55.554223621 (80h7m31.117546616s) [noise=true , address=77.90.31.14:4666 , piece_id=H4DTFDAVCN266PB235BJIQTQ32FBXTIB5AZSX3B4R2VIK554AFSQ, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:30:55.762514255 (80h7m31.325837245s) [noise=true , address=104.36.82.14:6245 , piece_id=II36G46OA6TINLR244U6D7Q6LGV754UA5M62K3EX7ZOBESMJUEHA, seg_pos={0 0}, upload=64142]: Done uploading piece. err: ecclient: upload failed (node:1Zu8teNPTXn638grHKwtJqz4UEY7oFYLx1DHvuVP6RZXcatVqX, address:104.36.82.14:6245): protocol: storage node overloaded, request limit: 2000 Jun 22 00:30:55.773669583 (80h7m31.3369926s) [noise=true , address=79.127.205.235:5660 , piece_id=UGFPVS7VNB7U2YSFPSGYDIRZV5R2DHCN57PJP6VOXO52WIAYHLXA, seg_pos={0 0}, upload=64142]: Done uploading piece. err: ecclient: upload failed (node:12E6uCX9KL4Ud7YpxMXepvMXuqTFJCiSXByDjW94Am3U9H2Nf7f, address:79.127.205.235:5660): protocol: storage node overloaded, request limit: 2000 Jun 22 00:30:56.968366655 (80h7m32.53168964s) [noise=true , address=77.90.31.10:4533 , piece_id=SE6YAKKFDL3BV677CDDUC32UNMPXJHTG23GWWYK5B4YIPKER6GAQ, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:30:57.403572874 (80h7m32.966895883s) [noise=true , address=45.91.116.5:5310 , piece_id=AXJOKDZRPVNL7HFT4QKJWRAHFM2UCM3GEVQB2KNYD6CXB6TH4D4Q, seg_pos={0 0}, upload=64142]: Done uploading piece. err: ecclient: upload failed (node:127RMUc62xRq9RU5z3XfkZsq5sehmv3kP8o53tr9BxVcPE3GQCY, address:45.91.116.5:5310): protocol: storage node overloaded, request limit: 2000 Jun 22 00:30:58.210389994 (80h7m33.773712995s) [noise=true , address=104.36.82.13:6199 , piece_id=YP62N2TO446YR3QRG5YLXECD7TN6SJ4UNNLZSK5H4CMV6JIF4AKA, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:30:58.693037961 (80h7m34.256360966s) [noise=true , address=104.36.82.39:6756 , piece_id=OKKWMDJPZO6D5VRMGOE3GOCYGHM725JZTNOVYAWKJW3WYZPI4V5Q, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:30:59.124834814 (80h7m34.688157819s) [noise=true , address=104.36.82.15:6283 , piece_id=N7TH2EMIOPATVHCSNH2WXUGU7KMMEJ5LCHR22SWEHSA5OV4CUCXQ, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:31:00.629657982 (80h7m36.192980977s) [noise=true , address=104.36.82.42:6803 , piece_id=R626UFD326S2UBJEZ2WMHVR7EN4SK427VXPDJZOOZ32MX25JWZWQ, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:31:00.933110546 (80h7m36.496433536s) [noise=true , address=79.127.201.215:4326 , piece_id=USEWZDX6P52U6MCALTBNQG3SV7N2JRKNU6O23TLS6JEWAS4D7ARA, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:31:05.669167243 (80h7m41.232490237s) [noise=true , address=104.36.82.56:4261 , piece_id=EYOYOYLDACGMPGJYX6ILQT3XHMUTQNXZEDC7ANHLZXIKYVDQFFVQ, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:31:08.443582763 (80h7m44.006905763s) [noise=true , address=104.36.82.54:4184 , piece_id=IZICVGZFMAYLIJTNUJ2OFLZVNZ6IPPYDZO4BSTJJPLHVCVOE5EAA, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:31:26.459138014 (80h8m2.022461013s) [noise=true , address=104.36.82.53:4146 , piece_id=3V3DVGUOPJKDWIGQ2TU7M6VFYFVAS6ODY62ERGSU6LI3LSTWQ4YQ, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:31:27.456050336 (80h8m3.019373336s) [noise=true , address=104.36.82.52:4089 , piece_id=X2VG4S3UHVNWA4AQJS7FLKWNVLTOZ3AKWI5ZDH24QCA3ENQW7BXQ, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:31:55.270234251 (80h8m30.833557256s) [noise=true , address=104.36.82.11:6125 , piece_id=2CX7RJOKNQA6ODZZZXYPSCYAQY5P35TWX7I56CMYAPFLJE3JEO5Q, seg_pos={0 0}, upload=64142]: Done uploading piece. err: ecclient: upload failed (node:12vricofwx67FRaxAGLbmgT1TYyKkqfJbxdsyeEPL4R7kg8pKzT, address:104.36.82.11:6125): protocol: expected piece hash; context deadline exceeded; EOF Jun 22 00:31:55.270319418 (80h8m30.833642407s) [noise=true , address=104.36.82.55:4224 , piece_id=XF7H2BYSNNNZOS2T77VIN75N76AYUXAQEI6D4LRXACU2KU62V2TA, seg_pos={0 0}, upload=64142]: Done uploading piece. err: ecclient: upload failed (node:12EjKmUiv4pxkVCB4WoiXim4uFpxrBPyfiSS4efTY3HKSE3jb5y, address:104.36.82.55:4224): protocol: write tcp 159.223.143.39:39152->104.36.82.55:4224: use of closed network connection; write tcp 159.223.143.39:39152->104.36.82.55:4224: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:39152->104.36.82.55:4224: use of closed network connection Jun 22 00:31:55.270353811 (80h8m30.833676799s) [seg_pos={0 0}, upload=64142]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Jun 22 00:31:55.301099877 (80h8m30.864422884s) [upload=64142]: Done uploading segments... Jun 22 00:31:55.301121567 (80h8m30.864444566s) [upload=64142]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 19.481052233s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [.........................X.X..............................X.] duration: 19.931372892s FAIL (2025-06-22 00:32:15 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== ing upload Jun 22 00:30:55.303646184 (80h7m30.866969171s) [upload=64142]: Uploading segments... Jun 22 00:30:55.303706406 (80h7m30.867029416s) [upload=64142]: Got next segment. Inline: false Jun 22 00:30:55.303762479 (80h7m30.867085468s) [upload=64142]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 22 00:30:55.303768627 (80h7m30.867091636s) [upload=64142]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 22 00:30:55.349058140 (80h7m30.912381168s) [seg_pos={0 0}, upload=64142]: Begin upload segment... Jun 22 00:30:55.349113768 (80h7m30.912436772s) [seg_pos={0 0}, upload=64142]: Done begin upload segment. Jun 22 00:30:55.349143121 (80h7m30.912466131s) [noise=true , address=45.91.116.5:5310 , piece_id=AXJOKDZRPVNL7HFT4QKJWRAHFM2UCM3GEVQB2KNYD6CXB6TH4D4Q, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.349660917 (80h7m30.912983928s) [noise=true , address=77.90.31.10:4533 , piece_id=SE6YAKKFDL3BV677CDDUC32UNMPXJHTG23GWWYK5B4YIPKER6GAQ, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.350093077 (80h7m30.913416089s) [noise=true , address=79.127.201.215:4326 , piece_id=USEWZDX6P52U6MCALTBNQG3SV7N2JRKNU6O23TLS6JEWAS4D7ARA, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.350480894 (80h7m30.913803916s) [noise=true , address=104.36.82.13:6199 , piece_id=YP62N2TO446YR3QRG5YLXECD7TN6SJ4UNNLZSK5H4CMV6JIF4AKA, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.350669998 (80h7m30.913993011s) [noise=true , address=104.36.82.54:4184 , piece_id=IZICVGZFMAYLIJTNUJ2OFLZVNZ6IPPYDZO4BSTJJPLHVCVOE5EAA, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.351201352 (80h7m30.914524366s) [noise=true , address=104.36.82.55:4224 , piece_id=XF7H2BYSNNNZOS2T77VIN75N76AYUXAQEI6D4LRXACU2KU62V2TA, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.351668071 (80h7m30.914991083s) [noise=true , address=104.36.82.56:4261 , piece_id=EYOYOYLDACGMPGJYX6ILQT3XHMUTQNXZEDC7ANHLZXIKYVDQFFVQ, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.352193126 (80h7m30.915516138s) [noise=true , address=104.36.82.52:4089 , piece_id=X2VG4S3UHVNWA4AQJS7FLKWNVLTOZ3AKWI5ZDH24QCA3ENQW7BXQ, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.352587096 (80h7m30.915910103s) [noise=true , address=79.127.220.98:4402 , piece_id=TE2WQO5LK45AVMPIAIGVSIBOVZLC4FG4ESQ72X6KFYC7IAEN2DRA, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.353420253 (80h7m30.916743265s) [noise=true , address=104.36.82.11:6125 , piece_id=2CX7RJOKNQA6ODZZZXYPSCYAQY5P35TWX7I56CMYAPFLJE3JEO5Q, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.353791565 (80h7m30.917114578s) [noise=true , address=104.36.82.42:6803 , piece_id=R626UFD326S2UBJEZ2WMHVR7EN4SK427VXPDJZOOZ32MX25JWZWQ, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.354171226 (80h7m30.917494233s) [noise=true , address=104.36.82.14:6245 , piece_id=II36G46OA6TINLR244U6D7Q6LGV754UA5M62K3EX7ZOBESMJUEHA, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.354542282 (80h7m30.917865288s) [noise=true , address=79.127.205.235:5660 , piece_id=UGFPVS7VNB7U2YSFPSGYDIRZV5R2DHCN57PJP6VOXO52WIAYHLXA, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.354891655 (80h7m30.918214678s) [noise=true , address=104.36.82.15:6283 , piece_id=N7TH2EMIOPATVHCSNH2WXUGU7KMMEJ5LCHR22SWEHSA5OV4CUCXQ, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.355499503 (80h7m30.918822519s) [noise=true , address=77.90.31.14:4666 , piece_id=H4DTFDAVCN266PB235BJIQTQ32FBXTIB5AZSX3B4R2VIK554AFSQ, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.356575547 (80h7m30.919898539s) [noise=true , address=104.36.82.39:6756 , piece_id=OKKWMDJPZO6D5VRMGOE3GOCYGHM725JZTNOVYAWKJW3WYZPI4V5Q, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.357039590 (80h7m30.920362596s) [noise=true , address=104.36.82.53:4146 , piece_id=3V3DVGUOPJKDWIGQ2TU7M6VFYFVAS6ODY62ERGSU6LI3LSTWQ4YQ, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.360343566 (80h7m30.923666582s) [noise=true , address=104.36.82.9:6033 , piece_id=T5XA23LZELMSTWOLBSHOX4AHKDACF65OYCKTMZYTXDIBSXNA2T4A, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.452444064 (80h7m31.015767055s) [upload=64142]: Next returned nil segment Jun 22 00:30:55.452495148 (80h7m31.015818149s) [upload=64142]: Waiting for error group managing segments... Jun 22 00:30:55.532958808 (80h7m31.096281883s) [noise=true , address=104.36.82.9:6033 , piece_id=T5XA23LZELMSTWOLBSHOX4AHKDACF65OYCKTMZYTXDIBSXNA2T4A, seg_pos={0 0}, upload=64142]: Done uploading piece. err: ecclient: upload failed (node:1SV3LrcQW16wLewDzfyUWFB6sUgnERNJVdhz72CJmyde95M79C, address:104.36.82.9:6033): protocol: storage node overloaded, request limit: 2000 Jun 22 00:30:55.547802184 (80h7m31.111125188s) [noise=true , address=79.127.220.98:4402 , piece_id=TE2WQO5LK45AVMPIAIGVSIBOVZLC4FG4ESQ72X6KFYC7IAEN2DRA, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:30:55.554223621 (80h7m31.117546616s) [noise=true , address=77.90.31.14:4666 , piece_id=H4DTFDAVCN266PB235BJIQTQ32FBXTIB5AZSX3B4R2VIK554AFSQ, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:30:55.762514255 (80h7m31.325837245s) [noise=true , address=104.36.82.14:6245 , piece_id=II36G46OA6TINLR244U6D7Q6LGV754UA5M62K3EX7ZOBESMJUEHA, seg_pos={0 0}, upload=64142]: Done uploading piece. err: ecclient: upload failed (node:1Zu8teNPTXn638grHKwtJqz4UEY7oFYLx1DHvuVP6RZXcatVqX, address:104.36.82.14:6245): protocol: storage node overloaded, request limit: 2000 Jun 22 00:30:55.773669583 (80h7m31.3369926s) [noise=true , address=79.127.205.235:5660 , piece_id=UGFPVS7VNB7U2YSFPSGYDIRZV5R2DHCN57PJP6VOXO52WIAYHLXA, seg_pos={0 0}, upload=64142]: Done uploading piece. err: ecclient: upload failed (node:12E6uCX9KL4Ud7YpxMXepvMXuqTFJCiSXByDjW94Am3U9H2Nf7f, address:79.127.205.235:5660): protocol: storage node overloaded, request limit: 2000 Jun 22 00:30:56.968366655 (80h7m32.53168964s) [noise=true , address=77.90.31.10:4533 , piece_id=SE6YAKKFDL3BV677CDDUC32UNMPXJHTG23GWWYK5B4YIPKER6GAQ, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:30:57.403572874 (80h7m32.966895883s) [noise=true , address=45.91.116.5:5310 , piece_id=AXJOKDZRPVNL7HFT4QKJWRAHFM2UCM3GEVQB2KNYD6CXB6TH4D4Q, seg_pos={0 0}, upload=64142]: Done uploading piece. err: ecclient: upload failed (node:127RMUc62xRq9RU5z3XfkZsq5sehmv3kP8o53tr9BxVcPE3GQCY, address:45.91.116.5:5310): protocol: storage node overloaded, request limit: 2000 Jun 22 00:30:58.210389994 (80h7m33.773712995s) [noise=true , address=104.36.82.13:6199 , piece_id=YP62N2TO446YR3QRG5YLXECD7TN6SJ4UNNLZSK5H4CMV6JIF4AKA, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:30:58.693037961 (80h7m34.256360966s) [noise=true , address=104.36.82.39:6756 , piece_id=OKKWMDJPZO6D5VRMGOE3GOCYGHM725JZTNOVYAWKJW3WYZPI4V5Q, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:30:59.124834814 (80h7m34.688157819s) [noise=true , address=104.36.82.15:6283 , piece_id=N7TH2EMIOPATVHCSNH2WXUGU7KMMEJ5LCHR22SWEHSA5OV4CUCXQ, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:31:00.629657982 (80h7m36.192980977s) [noise=true , address=104.36.82.42:6803 , piece_id=R626UFD326S2UBJEZ2WMHVR7EN4SK427VXPDJZOOZ32MX25JWZWQ, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:31:00.933110546 (80h7m36.496433536s) [noise=true , address=79.127.201.215:4326 , piece_id=USEWZDX6P52U6MCALTBNQG3SV7N2JRKNU6O23TLS6JEWAS4D7ARA, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:31:05.669167243 (80h7m41.232490237s) [noise=true , address=104.36.82.56:4261 , piece_id=EYOYOYLDACGMPGJYX6ILQT3XHMUTQNXZEDC7ANHLZXIKYVDQFFVQ, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:31:08.443582763 (80h7m44.006905763s) [noise=true , address=104.36.82.54:4184 , piece_id=IZICVGZFMAYLIJTNUJ2OFLZVNZ6IPPYDZO4BSTJJPLHVCVOE5EAA, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:31:26.459138014 (80h8m2.022461013s) [noise=true , address=104.36.82.53:4146 , piece_id=3V3DVGUOPJKDWIGQ2TU7M6VFYFVAS6ODY62ERGSU6LI3LSTWQ4YQ, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:31:27.456050336 (80h8m3.019373336s) [noise=true , address=104.36.82.52:4089 , piece_id=X2VG4S3UHVNWA4AQJS7FLKWNVLTOZ3AKWI5ZDH24QCA3ENQW7BXQ, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:31:55.270234251 (80h8m30.833557256s) [noise=true , address=104.36.82.11:6125 , piece_id=2CX7RJOKNQA6ODZZZXYPSCYAQY5P35TWX7I56CMYAPFLJE3JEO5Q, seg_pos={0 0}, upload=64142]: Done uploading piece. err: ecclient: upload failed (node:12vricofwx67FRaxAGLbmgT1TYyKkqfJbxdsyeEPL4R7kg8pKzT, address:104.36.82.11:6125): protocol: expected piece hash; context deadline exceeded; EOF Jun 22 00:31:55.270319418 (80h8m30.833642407s) [noise=true , address=104.36.82.55:4224 , piece_id=XF7H2BYSNNNZOS2T77VIN75N76AYUXAQEI6D4LRXACU2KU62V2TA, seg_pos={0 0}, upload=64142]: Done uploading piece. err: ecclient: upload failed (node:12EjKmUiv4pxkVCB4WoiXim4uFpxrBPyfiSS4efTY3HKSE3jb5y, address:104.36.82.55:4224): protocol: write tcp 159.223.143.39:39152->104.36.82.55:4224: use of closed network connection; write tcp 159.223.143.39:39152->104.36.82.55:4224: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:39152->104.36.82.55:4224: use of closed network connection Jun 22 00:31:55.270353811 (80h8m30.833676799s) [seg_pos={0 0}, upload=64142]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Jun 22 00:31:55.301099877 (80h8m30.864422884s) [upload=64142]: Done uploading segments... Jun 22 00:31:55.301121567 (80h8m30.864444566s) [upload=64142]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 19.931411027s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 4 out of 60 is failed (threshold=3) [.X.........................X.........X...................X..] duration: 23.952534939s FAIL (2025-06-22 00:31:58 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 23.952576013s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [..........................X.X..............................X] duration: 1m0.033069762s FAIL (2025-06-22 00:31:55 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== ing upload Jun 22 00:30:55.303646184 (80h7m30.866969171s) [upload=64142]: Uploading segments... Jun 22 00:30:55.303706406 (80h7m30.867029416s) [upload=64142]: Got next segment. Inline: false Jun 22 00:30:55.303762479 (80h7m30.867085468s) [upload=64142]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 22 00:30:55.303768627 (80h7m30.867091636s) [upload=64142]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 22 00:30:55.349058140 (80h7m30.912381168s) [seg_pos={0 0}, upload=64142]: Begin upload segment... Jun 22 00:30:55.349113768 (80h7m30.912436772s) [seg_pos={0 0}, upload=64142]: Done begin upload segment. Jun 22 00:30:55.349143121 (80h7m30.912466131s) [noise=true , address=45.91.116.5:5310 , piece_id=AXJOKDZRPVNL7HFT4QKJWRAHFM2UCM3GEVQB2KNYD6CXB6TH4D4Q, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.349660917 (80h7m30.912983928s) [noise=true , address=77.90.31.10:4533 , piece_id=SE6YAKKFDL3BV677CDDUC32UNMPXJHTG23GWWYK5B4YIPKER6GAQ, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.350093077 (80h7m30.913416089s) [noise=true , address=79.127.201.215:4326 , piece_id=USEWZDX6P52U6MCALTBNQG3SV7N2JRKNU6O23TLS6JEWAS4D7ARA, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.350480894 (80h7m30.913803916s) [noise=true , address=104.36.82.13:6199 , piece_id=YP62N2TO446YR3QRG5YLXECD7TN6SJ4UNNLZSK5H4CMV6JIF4AKA, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.350669998 (80h7m30.913993011s) [noise=true , address=104.36.82.54:4184 , piece_id=IZICVGZFMAYLIJTNUJ2OFLZVNZ6IPPYDZO4BSTJJPLHVCVOE5EAA, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.351201352 (80h7m30.914524366s) [noise=true , address=104.36.82.55:4224 , piece_id=XF7H2BYSNNNZOS2T77VIN75N76AYUXAQEI6D4LRXACU2KU62V2TA, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.351668071 (80h7m30.914991083s) [noise=true , address=104.36.82.56:4261 , piece_id=EYOYOYLDACGMPGJYX6ILQT3XHMUTQNXZEDC7ANHLZXIKYVDQFFVQ, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.352193126 (80h7m30.915516138s) [noise=true , address=104.36.82.52:4089 , piece_id=X2VG4S3UHVNWA4AQJS7FLKWNVLTOZ3AKWI5ZDH24QCA3ENQW7BXQ, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.352587096 (80h7m30.915910103s) [noise=true , address=79.127.220.98:4402 , piece_id=TE2WQO5LK45AVMPIAIGVSIBOVZLC4FG4ESQ72X6KFYC7IAEN2DRA, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.353420253 (80h7m30.916743265s) [noise=true , address=104.36.82.11:6125 , piece_id=2CX7RJOKNQA6ODZZZXYPSCYAQY5P35TWX7I56CMYAPFLJE3JEO5Q, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.353791565 (80h7m30.917114578s) [noise=true , address=104.36.82.42:6803 , piece_id=R626UFD326S2UBJEZ2WMHVR7EN4SK427VXPDJZOOZ32MX25JWZWQ, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.354171226 (80h7m30.917494233s) [noise=true , address=104.36.82.14:6245 , piece_id=II36G46OA6TINLR244U6D7Q6LGV754UA5M62K3EX7ZOBESMJUEHA, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.354542282 (80h7m30.917865288s) [noise=true , address=79.127.205.235:5660 , piece_id=UGFPVS7VNB7U2YSFPSGYDIRZV5R2DHCN57PJP6VOXO52WIAYHLXA, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.354891655 (80h7m30.918214678s) [noise=true , address=104.36.82.15:6283 , piece_id=N7TH2EMIOPATVHCSNH2WXUGU7KMMEJ5LCHR22SWEHSA5OV4CUCXQ, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.355499503 (80h7m30.918822519s) [noise=true , address=77.90.31.14:4666 , piece_id=H4DTFDAVCN266PB235BJIQTQ32FBXTIB5AZSX3B4R2VIK554AFSQ, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.356575547 (80h7m30.919898539s) [noise=true , address=104.36.82.39:6756 , piece_id=OKKWMDJPZO6D5VRMGOE3GOCYGHM725JZTNOVYAWKJW3WYZPI4V5Q, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.357039590 (80h7m30.920362596s) [noise=true , address=104.36.82.53:4146 , piece_id=3V3DVGUOPJKDWIGQ2TU7M6VFYFVAS6ODY62ERGSU6LI3LSTWQ4YQ, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.360343566 (80h7m30.923666582s) [noise=true , address=104.36.82.9:6033 , piece_id=T5XA23LZELMSTWOLBSHOX4AHKDACF65OYCKTMZYTXDIBSXNA2T4A, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.452444064 (80h7m31.015767055s) [upload=64142]: Next returned nil segment Jun 22 00:30:55.452495148 (80h7m31.015818149s) [upload=64142]: Waiting for error group managing segments... Jun 22 00:30:55.532958808 (80h7m31.096281883s) [noise=true , address=104.36.82.9:6033 , piece_id=T5XA23LZELMSTWOLBSHOX4AHKDACF65OYCKTMZYTXDIBSXNA2T4A, seg_pos={0 0}, upload=64142]: Done uploading piece. err: ecclient: upload failed (node:1SV3LrcQW16wLewDzfyUWFB6sUgnERNJVdhz72CJmyde95M79C, address:104.36.82.9:6033): protocol: storage node overloaded, request limit: 2000 Jun 22 00:30:55.547802184 (80h7m31.111125188s) [noise=true , address=79.127.220.98:4402 , piece_id=TE2WQO5LK45AVMPIAIGVSIBOVZLC4FG4ESQ72X6KFYC7IAEN2DRA, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:30:55.554223621 (80h7m31.117546616s) [noise=true , address=77.90.31.14:4666 , piece_id=H4DTFDAVCN266PB235BJIQTQ32FBXTIB5AZSX3B4R2VIK554AFSQ, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:30:55.762514255 (80h7m31.325837245s) [noise=true , address=104.36.82.14:6245 , piece_id=II36G46OA6TINLR244U6D7Q6LGV754UA5M62K3EX7ZOBESMJUEHA, seg_pos={0 0}, upload=64142]: Done uploading piece. err: ecclient: upload failed (node:1Zu8teNPTXn638grHKwtJqz4UEY7oFYLx1DHvuVP6RZXcatVqX, address:104.36.82.14:6245): protocol: storage node overloaded, request limit: 2000 Jun 22 00:30:55.773669583 (80h7m31.3369926s) [noise=true , address=79.127.205.235:5660 , piece_id=UGFPVS7VNB7U2YSFPSGYDIRZV5R2DHCN57PJP6VOXO52WIAYHLXA, seg_pos={0 0}, upload=64142]: Done uploading piece. err: ecclient: upload failed (node:12E6uCX9KL4Ud7YpxMXepvMXuqTFJCiSXByDjW94Am3U9H2Nf7f, address:79.127.205.235:5660): protocol: storage node overloaded, request limit: 2000 Jun 22 00:30:56.968366655 (80h7m32.53168964s) [noise=true , address=77.90.31.10:4533 , piece_id=SE6YAKKFDL3BV677CDDUC32UNMPXJHTG23GWWYK5B4YIPKER6GAQ, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:30:57.403572874 (80h7m32.966895883s) [noise=true , address=45.91.116.5:5310 , piece_id=AXJOKDZRPVNL7HFT4QKJWRAHFM2UCM3GEVQB2KNYD6CXB6TH4D4Q, seg_pos={0 0}, upload=64142]: Done uploading piece. err: ecclient: upload failed (node:127RMUc62xRq9RU5z3XfkZsq5sehmv3kP8o53tr9BxVcPE3GQCY, address:45.91.116.5:5310): protocol: storage node overloaded, request limit: 2000 Jun 22 00:30:58.210389994 (80h7m33.773712995s) [noise=true , address=104.36.82.13:6199 , piece_id=YP62N2TO446YR3QRG5YLXECD7TN6SJ4UNNLZSK5H4CMV6JIF4AKA, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:30:58.693037961 (80h7m34.256360966s) [noise=true , address=104.36.82.39:6756 , piece_id=OKKWMDJPZO6D5VRMGOE3GOCYGHM725JZTNOVYAWKJW3WYZPI4V5Q, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:30:59.124834814 (80h7m34.688157819s) [noise=true , address=104.36.82.15:6283 , piece_id=N7TH2EMIOPATVHCSNH2WXUGU7KMMEJ5LCHR22SWEHSA5OV4CUCXQ, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:31:00.629657982 (80h7m36.192980977s) [noise=true , address=104.36.82.42:6803 , piece_id=R626UFD326S2UBJEZ2WMHVR7EN4SK427VXPDJZOOZ32MX25JWZWQ, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:31:00.933110546 (80h7m36.496433536s) [noise=true , address=79.127.201.215:4326 , piece_id=USEWZDX6P52U6MCALTBNQG3SV7N2JRKNU6O23TLS6JEWAS4D7ARA, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:31:05.669167243 (80h7m41.232490237s) [noise=true , address=104.36.82.56:4261 , piece_id=EYOYOYLDACGMPGJYX6ILQT3XHMUTQNXZEDC7ANHLZXIKYVDQFFVQ, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:31:08.443582763 (80h7m44.006905763s) [noise=true , address=104.36.82.54:4184 , piece_id=IZICVGZFMAYLIJTNUJ2OFLZVNZ6IPPYDZO4BSTJJPLHVCVOE5EAA, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:31:26.459138014 (80h8m2.022461013s) [noise=true , address=104.36.82.53:4146 , piece_id=3V3DVGUOPJKDWIGQ2TU7M6VFYFVAS6ODY62ERGSU6LI3LSTWQ4YQ, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:31:27.456050336 (80h8m3.019373336s) [noise=true , address=104.36.82.52:4089 , piece_id=X2VG4S3UHVNWA4AQJS7FLKWNVLTOZ3AKWI5ZDH24QCA3ENQW7BXQ, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:31:55.270234251 (80h8m30.833557256s) [noise=true , address=104.36.82.11:6125 , piece_id=2CX7RJOKNQA6ODZZZXYPSCYAQY5P35TWX7I56CMYAPFLJE3JEO5Q, seg_pos={0 0}, upload=64142]: Done uploading piece. err: ecclient: upload failed (node:12vricofwx67FRaxAGLbmgT1TYyKkqfJbxdsyeEPL4R7kg8pKzT, address:104.36.82.11:6125): protocol: expected piece hash; context deadline exceeded; EOF Jun 22 00:31:55.270319418 (80h8m30.833642407s) [noise=true , address=104.36.82.55:4224 , piece_id=XF7H2BYSNNNZOS2T77VIN75N76AYUXAQEI6D4LRXACU2KU62V2TA, seg_pos={0 0}, upload=64142]: Done uploading piece. err: ecclient: upload failed (node:12EjKmUiv4pxkVCB4WoiXim4uFpxrBPyfiSS4efTY3HKSE3jb5y, address:104.36.82.55:4224): protocol: write tcp 159.223.143.39:39152->104.36.82.55:4224: use of closed network connection; write tcp 159.223.143.39:39152->104.36.82.55:4224: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:39152->104.36.82.55:4224: use of closed network connection Jun 22 00:31:55.270353811 (80h8m30.833676799s) [seg_pos={0 0}, upload=64142]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Jun 22 00:31:55.301099877 (80h8m30.864422884s) [upload=64142]: Done uploading segments... Jun 22 00:31:55.301121567 (80h8m30.864444566s) [upload=64142]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 1m0.033098853s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 4 out of 60 is failed (threshold=3) [..X.........................X.........X...................X.] duration: 50.268276425s FAIL (2025-06-22 00:31:10 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 50.26829286s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [...........................X.X..............................] duration: 17.222335142s FAIL (2025-06-22 00:30:42 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== g_pos={0 0}, upload=63843]: Uploading piece... Jun 22 00:08:16.324082782 (79h44m51.88740579s) [noise=true , address=104.36.82.52:4103 , piece_id=MWVPOZMAAFPZD6HDK4SGJOKW5IMNYG75R3YA7LOLMNJVQOLP3SBQ, seg_pos={0 0}, upload=63843]: Uploading piece... Jun 22 00:08:16.324387615 (79h44m51.887710618s) [noise=true , address=45.91.116.5:5308 , piece_id=QZA34FLZEJNVE2TUHVF5ZMTG2WP7RTNNOJF7AZ4YLYCA22XJPCZA, seg_pos={0 0}, upload=63843]: Uploading piece... Jun 22 00:08:16.324693763 (79h44m51.888016771s) [noise=true , address=77.90.31.14:4693 , piece_id=QAVXJYMYM3XOYRGUPHMK257ZNL7CUMR6UBGSJM647C73ECJDCVIA, seg_pos={0 0}, upload=63843]: Uploading piece... Jun 22 00:08:16.324987498 (79h44m51.888310498s) [noise=true , address=104.36.82.7:5938 , piece_id=YRRM6JLCSPEV2UZI5CICTMVPW2MTIK7P5K5F5S53AEV2DEMJFA3Q, seg_pos={0 0}, upload=63843]: Uploading piece... Jun 22 00:08:16.325315636 (79h44m51.888638643s) [noise=true , address=104.36.82.21:6473 , piece_id=3M34BSPUAFRF5NXPWIXKH25KX67RCHVXQCS5MFS5S7X7RYF676RQ, seg_pos={0 0}, upload=63843]: Uploading piece... Jun 22 00:08:16.394058454 (79h44m51.957381445s) [upload=63843]: Next returned nil segment Jun 22 00:08:16.394072985 (79h44m51.957395986s) [upload=63843]: Waiting for error group managing segments... Jun 22 00:08:16.397574195 (79h44m51.960897187s) [noise=true , address=45.91.116.5:5308 , piece_id=QZA34FLZEJNVE2TUHVF5ZMTG2WP7RTNNOJF7AZ4YLYCA22XJPCZA, seg_pos={0 0}, upload=63843]: Done uploading piece. err: <nil> Jun 22 00:08:16.459349430 (79h44m52.022672432s) [noise=true , address=104.36.82.54:4173 , piece_id=O2FBSZLX7FIDIDJWGWV2RMIZ7DZ5WYKS7JLZIOK64D5TRKSP3U7Q, seg_pos={0 0}, upload=63843]: Done uploading piece. err: ecclient: upload failed (node:1w2v4iyYtJtcdz7shYhG8CjCDQ1dWjeJVwY5Ubi7Ex5Y38e2i1, address:104.36.82.54:4173): protocol: storage node overloaded, request limit: 2000 Jun 22 00:08:16.518220521 (79h44m52.081543528s) [noise=true , address=77.90.31.12:4605 , piece_id=AITH3LEQCYAWVX4JXU7LDBV4LRIA3Y2LI3FNTOISOMRQ4LWNZ2VQ, seg_pos={0 0}, upload=63843]: Done uploading piece. err: <nil> Jun 22 00:08:17.014258958 (79h44m52.577581952s) [noise=true , address=64.49.13.137:4500 , piece_id=FMD3ECZINI2YWY32VYSH7GKCITBALAEDUMYCNTWTAJQVY55Y2MLQ, seg_pos={0 0}, upload=63843]: Done uploading piece. err: ecclient: upload failed (node:15TwVQ6CHDbtqS4XW86n9BuyjdqpaJo4ZKPgPnmhSm6EPDrnd9, address:64.49.13.137:4500): protocol: storage node overloaded, request limit: 2000 Jun 22 00:08:17.153129516 (79h44m52.716452522s) [noise=true , address=104.36.82.16:6328 , piece_id=4TEPB6TBOWSY5KR52EVGRYVGHAKBEDTUHMFF3KJPQKZXYRGDRBJQ, seg_pos={0 0}, upload=63843]: Done uploading piece. err: ecclient: upload failed (node:1S8PSdkLyaiURWkPGhcABSh3481hoswzukT1NKvpTkSp75vKDJ, address:104.36.82.16:6328): protocol: storage node overloaded, request limit: 2000 Jun 22 00:08:17.275208950 (79h44m52.83853195s) [noise=true , address=79.127.205.235:5664 , piece_id=XB53MDJKRRNXK6SAMHX2RHAEO6SAHTKI7732ERADDN7JCWDZKNVA, seg_pos={0 0}, upload=63843]: Done uploading piece. err: ecclient: upload failed (node:12p8Gg6yYNUbFiEkHLHDFdnPC2w4AHgguBYv2ce34AYWXQzhHER, address:79.127.205.235:5664): protocol: storage node overloaded, request limit: 2000 Jun 22 00:08:17.605904252 (79h44m53.169227253s) [noise=true , address=77.90.31.21:4887 , piece_id=3JB4YN5JQ2KHD3LXGYAH2VQDBYAM63SYBNIVNUK2BAVW6TG6L6EA, seg_pos={0 0}, upload=63843]: Done uploading piece. err: ecclient: upload failed (node:12hE2JH2wY7TepnbCVpSxkTcVifqui3QK42tLh1neU2BxZ2BRFJ, address:77.90.31.21:4887): protocol: storage node overloaded, request limit: 2000 Jun 22 00:08:17.865919302 (79h44m53.429242294s) [noise=true , address=77.90.31.14:4693 , piece_id=QAVXJYMYM3XOYRGUPHMK257ZNL7CUMR6UBGSJM647C73ECJDCVIA, seg_pos={0 0}, upload=63843]: Done uploading piece. err: <nil> Jun 22 00:08:19.517357026 (79h44m55.080680046s) [noise=true , address=104.36.82.10:6108 , piece_id=QTWYQJQ22NWG7I6WRKMC6AL7YYPHVEBN2LIUVIFZHC5WMDPKPANQ, seg_pos={0 0}, upload=63843]: Done uploading piece. err: <nil> Jun 22 00:08:20.580254458 (79h44m56.143577448s) [noise=true , address=104.36.82.7:5938 , piece_id=YRRM6JLCSPEV2UZI5CICTMVPW2MTIK7P5K5F5S53AEV2DEMJFA3Q, seg_pos={0 0}, upload=63843]: Done uploading piece. err: <nil> Jun 22 00:08:20.752577874 (79h44m56.315900867s) [noise=true , address=104.36.82.20:6466 , piece_id=P7UZBSRQEY3ORJAXMYSNYBPTYNGDBSSKZL74Z2GSJNMON3VZBOZQ, seg_pos={0 0}, upload=63843]: Done uploading piece. err: <nil> Jun 22 00:08:24.515800663 (79h45m0.079123665s) [noise=true , address=79.127.219.38:4028 , piece_id=QZDOXSRJI6MXMIUNJLVRKZBSCRMDI54KPFDSR4D4JF6S2RZ4SV7A, seg_pos={0 0}, upload=63843]: Done uploading piece. err: <nil> Jun 22 00:08:25.948556776 (79h45m1.511879777s) [noise=true , address=104.36.82.21:6473 , piece_id=3M34BSPUAFRF5NXPWIXKH25KX67RCHVXQCS5MFS5S7X7RYF676RQ, seg_pos={0 0}, upload=63843]: Done uploading piece. err: <nil> Jun 22 00:08:32.636880825 (79h45m8.200203818s) [noise=true , address=104.36.82.56:4261 , piece_id=2RGJ5T5RFUR67IIUA5CUVXFL4RVBO7GVLDLAVUZCRHWAITSBOEWA, seg_pos={0 0}, upload=63843]: Done uploading piece. err: <nil> Jun 22 00:08:35.525047894 (79h45m11.088370884s) [noise=true , address=104.36.82.19:6406 , piece_id=ERRJIBJ6LCOK3QRXCJSASUGTFUL76SS3HZJI2VQ64HWTSUMGVHZA, seg_pos={0 0}, upload=63843]: Done uploading piece. err: <nil> Jun 22 00:08:40.407828721 (79h45m15.971151709s) [noise=true , address=104.36.82.14:6234 , piece_id=WUFOSCTTAKQZLII6GTESOCKX4MJZM6CA45XBQUUKGS6S7JQASLUQ, seg_pos={0 0}, upload=63843]: Done uploading piece. err: <nil> Jun 22 00:08:52.755279221 (79h45m28.318602228s) [noise=true , address=104.36.82.52:4103 , piece_id=MWVPOZMAAFPZD6HDK4SGJOKW5IMNYG75R3YA7LOLMNJVQOLP3SBQ, seg_pos={0 0}, upload=63843]: Done uploading piece. err: <nil> Jun 22 00:09:16.049083116 (79h45m51.612406117s) [noise=true , address=104.36.82.55:4219 , piece_id=BMAN2MOGHY3DLY5O3GYQUSAUQ7A77YSTD7GEHUAUSSPE5EE7LJQQ, seg_pos={0 0}, upload=63843]: Done uploading piece. err: <nil> Jun 22 00:09:16.067949159 (79h45m51.631272164s) [noise=true , address=45.91.116.7:5012 , piece_id=7TSTZEQJHXFCT4UIUEOF6Q3JML3YJRJRJWHO3ZYPGVY66BCSDZ7Q, seg_pos={0 0}, upload=63843]: Uploading piece... Jun 22 00:09:16.067962327 (79h45m51.63128533s) [noise=true , address=77.90.31.17:4803 , piece_id=DXJWERKQZP445OUZQF6W33T5JXREOIMFYAMWDMDDMJS736PXWAXQ, seg_pos={0 0}, upload=63843]: Uploading piece... Jun 22 00:09:16.068067279 (79h45m51.631390289s) [noise=true , address=104.36.82.20:6468 , piece_id=SQUT67FBQGV423UW4LOAECRWUJWCJ7AQ4ED3ACD3MZUTBRXUSYMA, seg_pos={0 0}, upload=63843]: Uploading piece... Jun 22 00:09:16.068445650 (79h45m51.631768659s) [noise=true , address=104.36.82.11:6139 , piece_id=QQUWKI2WTCYU44QDNA4OOBWSPVTCMMMLEXEPOASV35A4UZVPW3TQ, seg_pos={0 0}, upload=63843]: Uploading piece... Jun 22 00:09:16.068751557 (79h45m51.632074575s) [noise=true , address=79.127.205.235:5664 , piece_id=NXG2ACQ4GJGVX7PQYTR5MCRWRPK6Y5FXNUN7YVETYC3U4ROXMOFA, seg_pos={0 0}, upload=63843]: Uploading piece... Jun 22 00:09:16.254420059 (79h45m51.817743058s) [noise=true , address=45.91.116.7:5012 , piece_id=7TSTZEQJHXFCT4UIUEOF6Q3JML3YJRJRJWHO3ZYPGVY66BCSDZ7Q, seg_pos={0 0}, upload=63843]: Done uploading piece. err: ecclient: upload failed (node:1smg5DiC8TRGnnmoQrRpvScvehv8aqtA2DaKtD1JtAhrhnm96t, address:45.91.116.7:5012): protocol: expected piece hash; context deadline exceeded; EOF Jun 22 00:09:16.254721861 (79h45m51.818044861s) [noise=true , address=104.36.82.11:6139 , piece_id=QQUWKI2WTCYU44QDNA4OOBWSPVTCMMMLEXEPOASV35A4UZVPW3TQ, seg_pos={0 0}, upload=63843]: Done uploading piece. err: ecclient: upload failed (node:1EiZjV2QMYue3KQDsNRhhdGpCQjybDv54wTuXsJTNtyeV8hHeh, address:104.36.82.11:6139): protocol: write tcp 159.223.143.39:45694->104.36.82.11:6139: use of closed network connection; write tcp 159.223.143.39:45694->104.36.82.11:6139: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:45694->104.36.82.11:6139: use of closed network connection Jun 22 00:09:16.254797295 (79h45m51.818120295s) [noise=true , address=104.36.82.20:6468 , piece_id=SQUT67FBQGV423UW4LOAECRWUJWCJ7AQ4ED3ACD3MZUTBRXUSYMA, seg_pos={0 0}, upload=63843]: Done uploading piece. err: ecclient: upload failed (node:123Cc2RmkYzkkbder6auanNvu1sTSEM6QD5aagnNC28DviG41Gi, address:104.36.82.20:6468): protocol: read tcp 159.223.143.39:35576->104.36.82.20:6468: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 22 00:09:16.254808564 (79h45m51.818131557s) [noise=true , address=77.90.31.17:4803 , piece_id=DXJWERKQZP445OUZQF6W33T5JXREOIMFYAMWDMDDMJS736PXWAXQ, seg_pos={0 0}, upload=63843]: Done uploading piece. err: ecclient: upload failed (node:12onacKUY32faZTFuyEjvMCZA4nVKiRKmTKh1WRFMfif2Kw1Q7Y, address:77.90.31.17:4803): protocol: read tcp 159.223.143.39:42574->77.90.31.17:4803: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 22 00:09:16.254859521 (79h45m51.81818252s) [noise=true , address=79.127.205.235:5664 , piece_id=NXG2ACQ4GJGVX7PQYTR5MCRWRPK6Y5FXNUN7YVETYC3U4ROXMOFA, seg_pos={0 0}, upload=63843]: Done uploading piece. err: ecclient: upload failed (node:12p8Gg6yYNUbFiEkHLHDFdnPC2w4AHgguBYv2ce34AYWXQzhHER, address:79.127.205.235:5664): protocol: write tcp 159.223.143.39:54186->79.127.205.235:5664: use of closed network connection; write tcp 159.223.143.39:54186->79.127.205.235:5664: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:54186->79.127.205.235:5664: use of closed network connection Jun 22 00:09:16.254895863 (79h45m51.818218866s) [seg_pos={0 0}, upload=63843]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jun 22 00:09:16.276852311 (79h45m51.840175305s) [upload=63843]: Done uploading segments... Jun 22 00:09:16.276903113 (79h45m51.840226102s) [upload=63843]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 17.2223653s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 4 out of 60 is failed (threshold=3) [...X.........................X.........X...................X] duration: 1m0.001007942s FAIL (2025-06-22 00:30:20 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1m0.001032884s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [............................X.X.............................] duration: 21.851481196s FAIL (2025-06-22 00:29:36 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== g_pos={0 0}, upload=63843]: Uploading piece... Jun 22 00:08:16.324082782 (79h44m51.88740579s) [noise=true , address=104.36.82.52:4103 , piece_id=MWVPOZMAAFPZD6HDK4SGJOKW5IMNYG75R3YA7LOLMNJVQOLP3SBQ, seg_pos={0 0}, upload=63843]: Uploading piece... Jun 22 00:08:16.324387615 (79h44m51.887710618s) [noise=true , address=45.91.116.5:5308 , piece_id=QZA34FLZEJNVE2TUHVF5ZMTG2WP7RTNNOJF7AZ4YLYCA22XJPCZA, seg_pos={0 0}, upload=63843]: Uploading piece... Jun 22 00:08:16.324693763 (79h44m51.888016771s) [noise=true , address=77.90.31.14:4693 , piece_id=QAVXJYMYM3XOYRGUPHMK257ZNL7CUMR6UBGSJM647C73ECJDCVIA, seg_pos={0 0}, upload=63843]: Uploading piece... Jun 22 00:08:16.324987498 (79h44m51.888310498s) [noise=true , address=104.36.82.7:5938 , piece_id=YRRM6JLCSPEV2UZI5CICTMVPW2MTIK7P5K5F5S53AEV2DEMJFA3Q, seg_pos={0 0}, upload=63843]: Uploading piece... Jun 22 00:08:16.325315636 (79h44m51.888638643s) [noise=true , address=104.36.82.21:6473 , piece_id=3M34BSPUAFRF5NXPWIXKH25KX67RCHVXQCS5MFS5S7X7RYF676RQ, seg_pos={0 0}, upload=63843]: Uploading piece... Jun 22 00:08:16.394058454 (79h44m51.957381445s) [upload=63843]: Next returned nil segment Jun 22 00:08:16.394072985 (79h44m51.957395986s) [upload=63843]: Waiting for error group managing segments... Jun 22 00:08:16.397574195 (79h44m51.960897187s) [noise=true , address=45.91.116.5:5308 , piece_id=QZA34FLZEJNVE2TUHVF5ZMTG2WP7RTNNOJF7AZ4YLYCA22XJPCZA, seg_pos={0 0}, upload=63843]: Done uploading piece. err: <nil> Jun 22 00:08:16.459349430 (79h44m52.022672432s) [noise=true , address=104.36.82.54:4173 , piece_id=O2FBSZLX7FIDIDJWGWV2RMIZ7DZ5WYKS7JLZIOK64D5TRKSP3U7Q, seg_pos={0 0}, upload=63843]: Done uploading piece. err: ecclient: upload failed (node:1w2v4iyYtJtcdz7shYhG8CjCDQ1dWjeJVwY5Ubi7Ex5Y38e2i1, address:104.36.82.54:4173): protocol: storage node overloaded, request limit: 2000 Jun 22 00:08:16.518220521 (79h44m52.081543528s) [noise=true , address=77.90.31.12:4605 , piece_id=AITH3LEQCYAWVX4JXU7LDBV4LRIA3Y2LI3FNTOISOMRQ4LWNZ2VQ, seg_pos={0 0}, upload=63843]: Done uploading piece. err: <nil> Jun 22 00:08:17.014258958 (79h44m52.577581952s) [noise=true , address=64.49.13.137:4500 , piece_id=FMD3ECZINI2YWY32VYSH7GKCITBALAEDUMYCNTWTAJQVY55Y2MLQ, seg_pos={0 0}, upload=63843]: Done uploading piece. err: ecclient: upload failed (node:15TwVQ6CHDbtqS4XW86n9BuyjdqpaJo4ZKPgPnmhSm6EPDrnd9, address:64.49.13.137:4500): protocol: storage node overloaded, request limit: 2000 Jun 22 00:08:17.153129516 (79h44m52.716452522s) [noise=true , address=104.36.82.16:6328 , piece_id=4TEPB6TBOWSY5KR52EVGRYVGHAKBEDTUHMFF3KJPQKZXYRGDRBJQ, seg_pos={0 0}, upload=63843]: Done uploading piece. err: ecclient: upload failed (node:1S8PSdkLyaiURWkPGhcABSh3481hoswzukT1NKvpTkSp75vKDJ, address:104.36.82.16:6328): protocol: storage node overloaded, request limit: 2000 Jun 22 00:08:17.275208950 (79h44m52.83853195s) [noise=true , address=79.127.205.235:5664 , piece_id=XB53MDJKRRNXK6SAMHX2RHAEO6SAHTKI7732ERADDN7JCWDZKNVA, seg_pos={0 0}, upload=63843]: Done uploading piece. err: ecclient: upload failed (node:12p8Gg6yYNUbFiEkHLHDFdnPC2w4AHgguBYv2ce34AYWXQzhHER, address:79.127.205.235:5664): protocol: storage node overloaded, request limit: 2000 Jun 22 00:08:17.605904252 (79h44m53.169227253s) [noise=true , address=77.90.31.21:4887 , piece_id=3JB4YN5JQ2KHD3LXGYAH2VQDBYAM63SYBNIVNUK2BAVW6TG6L6EA, seg_pos={0 0}, upload=63843]: Done uploading piece. err: ecclient: upload failed (node:12hE2JH2wY7TepnbCVpSxkTcVifqui3QK42tLh1neU2BxZ2BRFJ, address:77.90.31.21:4887): protocol: storage node overloaded, request limit: 2000 Jun 22 00:08:17.865919302 (79h44m53.429242294s) [noise=true , address=77.90.31.14:4693 , piece_id=QAVXJYMYM3XOYRGUPHMK257ZNL7CUMR6UBGSJM647C73ECJDCVIA, seg_pos={0 0}, upload=63843]: Done uploading piece. err: <nil> Jun 22 00:08:19.517357026 (79h44m55.080680046s) [noise=true , address=104.36.82.10:6108 , piece_id=QTWYQJQ22NWG7I6WRKMC6AL7YYPHVEBN2LIUVIFZHC5WMDPKPANQ, seg_pos={0 0}, upload=63843]: Done uploading piece. err: <nil> Jun 22 00:08:20.580254458 (79h44m56.143577448s) [noise=true , address=104.36.82.7:5938 , piece_id=YRRM6JLCSPEV2UZI5CICTMVPW2MTIK7P5K5F5S53AEV2DEMJFA3Q, seg_pos={0 0}, upload=63843]: Done uploading piece. err: <nil> Jun 22 00:08:20.752577874 (79h44m56.315900867s) [noise=true , address=104.36.82.20:6466 , piece_id=P7UZBSRQEY3ORJAXMYSNYBPTYNGDBSSKZL74Z2GSJNMON3VZBOZQ, seg_pos={0 0}, upload=63843]: Done uploading piece. err: <nil> Jun 22 00:08:24.515800663 (79h45m0.079123665s) [noise=true , address=79.127.219.38:4028 , piece_id=QZDOXSRJI6MXMIUNJLVRKZBSCRMDI54KPFDSR4D4JF6S2RZ4SV7A, seg_pos={0 0}, upload=63843]: Done uploading piece. err: <nil> Jun 22 00:08:25.948556776 (79h45m1.511879777s) [noise=true , address=104.36.82.21:6473 , piece_id=3M34BSPUAFRF5NXPWIXKH25KX67RCHVXQCS5MFS5S7X7RYF676RQ, seg_pos={0 0}, upload=63843]: Done uploading piece. err: <nil> Jun 22 00:08:32.636880825 (79h45m8.200203818s) [noise=true , address=104.36.82.56:4261 , piece_id=2RGJ5T5RFUR67IIUA5CUVXFL4RVBO7GVLDLAVUZCRHWAITSBOEWA, seg_pos={0 0}, upload=63843]: Done uploading piece. err: <nil> Jun 22 00:08:35.525047894 (79h45m11.088370884s) [noise=true , address=104.36.82.19:6406 , piece_id=ERRJIBJ6LCOK3QRXCJSASUGTFUL76SS3HZJI2VQ64HWTSUMGVHZA, seg_pos={0 0}, upload=63843]: Done uploading piece. err: <nil> Jun 22 00:08:40.407828721 (79h45m15.971151709s) [noise=true , address=104.36.82.14:6234 , piece_id=WUFOSCTTAKQZLII6GTESOCKX4MJZM6CA45XBQUUKGS6S7JQASLUQ, seg_pos={0 0}, upload=63843]: Done uploading piece. err: <nil> Jun 22 00:08:52.755279221 (79h45m28.318602228s) [noise=true , address=104.36.82.52:4103 , piece_id=MWVPOZMAAFPZD6HDK4SGJOKW5IMNYG75R3YA7LOLMNJVQOLP3SBQ, seg_pos={0 0}, upload=63843]: Done uploading piece. err: <nil> Jun 22 00:09:16.049083116 (79h45m51.612406117s) [noise=true , address=104.36.82.55:4219 , piece_id=BMAN2MOGHY3DLY5O3GYQUSAUQ7A77YSTD7GEHUAUSSPE5EE7LJQQ, seg_pos={0 0}, upload=63843]: Done uploading piece. err: <nil> Jun 22 00:09:16.067949159 (79h45m51.631272164s) [noise=true , address=45.91.116.7:5012 , piece_id=7TSTZEQJHXFCT4UIUEOF6Q3JML3YJRJRJWHO3ZYPGVY66BCSDZ7Q, seg_pos={0 0}, upload=63843]: Uploading piece... Jun 22 00:09:16.067962327 (79h45m51.63128533s) [noise=true , address=77.90.31.17:4803 , piece_id=DXJWERKQZP445OUZQF6W33T5JXREOIMFYAMWDMDDMJS736PXWAXQ, seg_pos={0 0}, upload=63843]: Uploading piece... Jun 22 00:09:16.068067279 (79h45m51.631390289s) [noise=true , address=104.36.82.20:6468 , piece_id=SQUT67FBQGV423UW4LOAECRWUJWCJ7AQ4ED3ACD3MZUTBRXUSYMA, seg_pos={0 0}, upload=63843]: Uploading piece... Jun 22 00:09:16.068445650 (79h45m51.631768659s) [noise=true , address=104.36.82.11:6139 , piece_id=QQUWKI2WTCYU44QDNA4OOBWSPVTCMMMLEXEPOASV35A4UZVPW3TQ, seg_pos={0 0}, upload=63843]: Uploading piece... Jun 22 00:09:16.068751557 (79h45m51.632074575s) [noise=true , address=79.127.205.235:5664 , piece_id=NXG2ACQ4GJGVX7PQYTR5MCRWRPK6Y5FXNUN7YVETYC3U4ROXMOFA, seg_pos={0 0}, upload=63843]: Uploading piece... Jun 22 00:09:16.254420059 (79h45m51.817743058s) [noise=true , address=45.91.116.7:5012 , piece_id=7TSTZEQJHXFCT4UIUEOF6Q3JML3YJRJRJWHO3ZYPGVY66BCSDZ7Q, seg_pos={0 0}, upload=63843]: Done uploading piece. err: ecclient: upload failed (node:1smg5DiC8TRGnnmoQrRpvScvehv8aqtA2DaKtD1JtAhrhnm96t, address:45.91.116.7:5012): protocol: expected piece hash; context deadline exceeded; EOF Jun 22 00:09:16.254721861 (79h45m51.818044861s) [noise=true , address=104.36.82.11:6139 , piece_id=QQUWKI2WTCYU44QDNA4OOBWSPVTCMMMLEXEPOASV35A4UZVPW3TQ, seg_pos={0 0}, upload=63843]: Done uploading piece. err: ecclient: upload failed (node:1EiZjV2QMYue3KQDsNRhhdGpCQjybDv54wTuXsJTNtyeV8hHeh, address:104.36.82.11:6139): protocol: write tcp 159.223.143.39:45694->104.36.82.11:6139: use of closed network connection; write tcp 159.223.143.39:45694->104.36.82.11:6139: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:45694->104.36.82.11:6139: use of closed network connection Jun 22 00:09:16.254797295 (79h45m51.818120295s) [noise=true , address=104.36.82.20:6468 , piece_id=SQUT67FBQGV423UW4LOAECRWUJWCJ7AQ4ED3ACD3MZUTBRXUSYMA, seg_pos={0 0}, upload=63843]: Done uploading piece. err: ecclient: upload failed (node:123Cc2RmkYzkkbder6auanNvu1sTSEM6QD5aagnNC28DviG41Gi, address:104.36.82.20:6468): protocol: read tcp 159.223.143.39:35576->104.36.82.20:6468: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 22 00:09:16.254808564 (79h45m51.818131557s) [noise=true , address=77.90.31.17:4803 , piece_id=DXJWERKQZP445OUZQF6W33T5JXREOIMFYAMWDMDDMJS736PXWAXQ, seg_pos={0 0}, upload=63843]: Done uploading piece. err: ecclient: upload failed (node:12onacKUY32faZTFuyEjvMCZA4nVKiRKmTKh1WRFMfif2Kw1Q7Y, address:77.90.31.17:4803): protocol: read tcp 159.223.143.39:42574->77.90.31.17:4803: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 22 00:09:16.254859521 (79h45m51.81818252s) [noise=true , address=79.127.205.235:5664 , piece_id=NXG2ACQ4GJGVX7PQYTR5MCRWRPK6Y5FXNUN7YVETYC3U4ROXMOFA, seg_pos={0 0}, upload=63843]: Done uploading piece. err: ecclient: upload failed (node:12p8Gg6yYNUbFiEkHLHDFdnPC2w4AHgguBYv2ce34AYWXQzhHER, address:79.127.205.235:5664): protocol: write tcp 159.223.143.39:54186->79.127.205.235:5664: use of closed network connection; write tcp 159.223.143.39:54186->79.127.205.235:5664: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:54186->79.127.205.235:5664: use of closed network connection Jun 22 00:09:16.254895863 (79h45m51.818218866s) [seg_pos={0 0}, upload=63843]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jun 22 00:09:16.276852311 (79h45m51.840175305s) [upload=63843]: Done uploading segments... Jun 22 00:09:16.276903113 (79h45m51.840226102s) [upload=63843]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 21.851523489s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [.............................X.X............................] duration: 28.220679504s FAIL (2025-06-22 00:29:14 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== g_pos={0 0}, upload=63843]: Uploading piece... Jun 22 00:08:16.324082782 (79h44m51.88740579s) [noise=true , address=104.36.82.52:4103 , piece_id=MWVPOZMAAFPZD6HDK4SGJOKW5IMNYG75R3YA7LOLMNJVQOLP3SBQ, seg_pos={0 0}, upload=63843]: Uploading piece... Jun 22 00:08:16.324387615 (79h44m51.887710618s) [noise=true , address=45.91.116.5:5308 , piece_id=QZA34FLZEJNVE2TUHVF5ZMTG2WP7RTNNOJF7AZ4YLYCA22XJPCZA, seg_pos={0 0}, upload=63843]: Uploading piece... Jun 22 00:08:16.324693763 (79h44m51.888016771s) [noise=true , address=77.90.31.14:4693 , piece_id=QAVXJYMYM3XOYRGUPHMK257ZNL7CUMR6UBGSJM647C73ECJDCVIA, seg_pos={0 0}, upload=63843]: Uploading piece... Jun 22 00:08:16.324987498 (79h44m51.888310498s) [noise=true , address=104.36.82.7:5938 , piece_id=YRRM6JLCSPEV2UZI5CICTMVPW2MTIK7P5K5F5S53AEV2DEMJFA3Q, seg_pos={0 0}, upload=63843]: Uploading piece... Jun 22 00:08:16.325315636 (79h44m51.888638643s) [noise=true , address=104.36.82.21:6473 , piece_id=3M34BSPUAFRF5NXPWIXKH25KX67RCHVXQCS5MFS5S7X7RYF676RQ, seg_pos={0 0}, upload=63843]: Uploading piece... Jun 22 00:08:16.394058454 (79h44m51.957381445s) [upload=63843]: Next returned nil segment Jun 22 00:08:16.394072985 (79h44m51.957395986s) [upload=63843]: Waiting for error group managing segments... Jun 22 00:08:16.397574195 (79h44m51.960897187s) [noise=true , address=45.91.116.5:5308 , piece_id=QZA34FLZEJNVE2TUHVF5ZMTG2WP7RTNNOJF7AZ4YLYCA22XJPCZA, seg_pos={0 0}, upload=63843]: Done uploading piece. err: <nil> Jun 22 00:08:16.459349430 (79h44m52.022672432s) [noise=true , address=104.36.82.54:4173 , piece_id=O2FBSZLX7FIDIDJWGWV2RMIZ7DZ5WYKS7JLZIOK64D5TRKSP3U7Q, seg_pos={0 0}, upload=63843]: Done uploading piece. err: ecclient: upload failed (node:1w2v4iyYtJtcdz7shYhG8CjCDQ1dWjeJVwY5Ubi7Ex5Y38e2i1, address:104.36.82.54:4173): protocol: storage node overloaded, request limit: 2000 Jun 22 00:08:16.518220521 (79h44m52.081543528s) [noise=true , address=77.90.31.12:4605 , piece_id=AITH3LEQCYAWVX4JXU7LDBV4LRIA3Y2LI3FNTOISOMRQ4LWNZ2VQ, seg_pos={0 0}, upload=63843]: Done uploading piece. err: <nil> Jun 22 00:08:17.014258958 (79h44m52.577581952s) [noise=true , address=64.49.13.137:4500 , piece_id=FMD3ECZINI2YWY32VYSH7GKCITBALAEDUMYCNTWTAJQVY55Y2MLQ, seg_pos={0 0}, upload=63843]: Done uploading piece. err: ecclient: upload failed (node:15TwVQ6CHDbtqS4XW86n9BuyjdqpaJo4ZKPgPnmhSm6EPDrnd9, address:64.49.13.137:4500): protocol: storage node overloaded, request limit: 2000 Jun 22 00:08:17.153129516 (79h44m52.716452522s) [noise=true , address=104.36.82.16:6328 , piece_id=4TEPB6TBOWSY5KR52EVGRYVGHAKBEDTUHMFF3KJPQKZXYRGDRBJQ, seg_pos={0 0}, upload=63843]: Done uploading piece. err: ecclient: upload failed (node:1S8PSdkLyaiURWkPGhcABSh3481hoswzukT1NKvpTkSp75vKDJ, address:104.36.82.16:6328): protocol: storage node overloaded, request limit: 2000 Jun 22 00:08:17.275208950 (79h44m52.83853195s) [noise=true , address=79.127.205.235:5664 , piece_id=XB53MDJKRRNXK6SAMHX2RHAEO6SAHTKI7732ERADDN7JCWDZKNVA, seg_pos={0 0}, upload=63843]: Done uploading piece. err: ecclient: upload failed (node:12p8Gg6yYNUbFiEkHLHDFdnPC2w4AHgguBYv2ce34AYWXQzhHER, address:79.127.205.235:5664): protocol: storage node overloaded, request limit: 2000 Jun 22 00:08:17.605904252 (79h44m53.169227253s) [noise=true , address=77.90.31.21:4887 , piece_id=3JB4YN5JQ2KHD3LXGYAH2VQDBYAM63SYBNIVNUK2BAVW6TG6L6EA, seg_pos={0 0}, upload=63843]: Done uploading piece. err: ecclient: upload failed (node:12hE2JH2wY7TepnbCVpSxkTcVifqui3QK42tLh1neU2BxZ2BRFJ, address:77.90.31.21:4887): protocol: storage node overloaded, request limit: 2000 Jun 22 00:08:17.865919302 (79h44m53.429242294s) [noise=true , address=77.90.31.14:4693 , piece_id=QAVXJYMYM3XOYRGUPHMK257ZNL7CUMR6UBGSJM647C73ECJDCVIA, seg_pos={0 0}, upload=63843]: Done uploading piece. err: <nil> Jun 22 00:08:19.517357026 (79h44m55.080680046s) [noise=true , address=104.36.82.10:6108 , piece_id=QTWYQJQ22NWG7I6WRKMC6AL7YYPHVEBN2LIUVIFZHC5WMDPKPANQ, seg_pos={0 0}, upload=63843]: Done uploading piece. err: <nil> Jun 22 00:08:20.580254458 (79h44m56.143577448s) [noise=true , address=104.36.82.7:5938 , piece_id=YRRM6JLCSPEV2UZI5CICTMVPW2MTIK7P5K5F5S53AEV2DEMJFA3Q, seg_pos={0 0}, upload=63843]: Done uploading piece. err: <nil> Jun 22 00:08:20.752577874 (79h44m56.315900867s) [noise=true , address=104.36.82.20:6466 , piece_id=P7UZBSRQEY3ORJAXMYSNYBPTYNGDBSSKZL74Z2GSJNMON3VZBOZQ, seg_pos={0 0}, upload=63843]: Done uploading piece. err: <nil> Jun 22 00:08:24.515800663 (79h45m0.079123665s) [noise=true , address=79.127.219.38:4028 , piece_id=QZDOXSRJI6MXMIUNJLVRKZBSCRMDI54KPFDSR4D4JF6S2RZ4SV7A, seg_pos={0 0}, upload=63843]: Done uploading piece. err: <nil> Jun 22 00:08:25.948556776 (79h45m1.511879777s) [noise=true , address=104.36.82.21:6473 , piece_id=3M34BSPUAFRF5NXPWIXKH25KX67RCHVXQCS5MFS5S7X7RYF676RQ, seg_pos={0 0}, upload=63843]: Done uploading piece. err: <nil> Jun 22 00:08:32.636880825 (79h45m8.200203818s) [noise=true , address=104.36.82.56:4261 , piece_id=2RGJ5T5RFUR67IIUA5CUVXFL4RVBO7GVLDLAVUZCRHWAITSBOEWA, seg_pos={0 0}, upload=63843]: Done uploading piece. err: <nil> Jun 22 00:08:35.525047894 (79h45m11.088370884s) [noise=true , address=104.36.82.19:6406 , piece_id=ERRJIBJ6LCOK3QRXCJSASUGTFUL76SS3HZJI2VQ64HWTSUMGVHZA, seg_pos={0 0}, upload=63843]: Done uploading piece. err: <nil> Jun 22 00:08:40.407828721 (79h45m15.971151709s) [noise=true , address=104.36.82.14:6234 , piece_id=WUFOSCTTAKQZLII6GTESOCKX4MJZM6CA45XBQUUKGS6S7JQASLUQ, seg_pos={0 0}, upload=63843]: Done uploading piece. err: <nil> Jun 22 00:08:52.755279221 (79h45m28.318602228s) [noise=true , address=104.36.82.52:4103 , piece_id=MWVPOZMAAFPZD6HDK4SGJOKW5IMNYG75R3YA7LOLMNJVQOLP3SBQ, seg_pos={0 0}, upload=63843]: Done uploading piece. err: <nil> Jun 22 00:09:16.049083116 (79h45m51.612406117s) [noise=true , address=104.36.82.55:4219 , piece_id=BMAN2MOGHY3DLY5O3GYQUSAUQ7A77YSTD7GEHUAUSSPE5EE7LJQQ, seg_pos={0 0}, upload=63843]: Done uploading piece. err: <nil> Jun 22 00:09:16.067949159 (79h45m51.631272164s) [noise=true , address=45.91.116.7:5012 , piece_id=7TSTZEQJHXFCT4UIUEOF6Q3JML3YJRJRJWHO3ZYPGVY66BCSDZ7Q, seg_pos={0 0}, upload=63843]: Uploading piece... Jun 22 00:09:16.067962327 (79h45m51.63128533s) [noise=true , address=77.90.31.17:4803 , piece_id=DXJWERKQZP445OUZQF6W33T5JXREOIMFYAMWDMDDMJS736PXWAXQ, seg_pos={0 0}, upload=63843]: Uploading piece... Jun 22 00:09:16.068067279 (79h45m51.631390289s) [noise=true , address=104.36.82.20:6468 , piece_id=SQUT67FBQGV423UW4LOAECRWUJWCJ7AQ4ED3ACD3MZUTBRXUSYMA, seg_pos={0 0}, upload=63843]: Uploading piece... Jun 22 00:09:16.068445650 (79h45m51.631768659s) [noise=true , address=104.36.82.11:6139 , piece_id=QQUWKI2WTCYU44QDNA4OOBWSPVTCMMMLEXEPOASV35A4UZVPW3TQ, seg_pos={0 0}, upload=63843]: Uploading piece... Jun 22 00:09:16.068751557 (79h45m51.632074575s) [noise=true , address=79.127.205.235:5664 , piece_id=NXG2ACQ4GJGVX7PQYTR5MCRWRPK6Y5FXNUN7YVETYC3U4ROXMOFA, seg_pos={0 0}, upload=63843]: Uploading piece... Jun 22 00:09:16.254420059 (79h45m51.817743058s) [noise=true , address=45.91.116.7:5012 , piece_id=7TSTZEQJHXFCT4UIUEOF6Q3JML3YJRJRJWHO3ZYPGVY66BCSDZ7Q, seg_pos={0 0}, upload=63843]: Done uploading piece. err: ecclient: upload failed (node:1smg5DiC8TRGnnmoQrRpvScvehv8aqtA2DaKtD1JtAhrhnm96t, address:45.91.116.7:5012): protocol: expected piece hash; context deadline exceeded; EOF Jun 22 00:09:16.254721861 (79h45m51.818044861s) [noise=true , address=104.36.82.11:6139 , piece_id=QQUWKI2WTCYU44QDNA4OOBWSPVTCMMMLEXEPOASV35A4UZVPW3TQ, seg_pos={0 0}, upload=63843]: Done uploading piece. err: ecclient: upload failed (node:1EiZjV2QMYue3KQDsNRhhdGpCQjybDv54wTuXsJTNtyeV8hHeh, address:104.36.82.11:6139): protocol: write tcp 159.223.143.39:45694->104.36.82.11:6139: use of closed network connection; write tcp 159.223.143.39:45694->104.36.82.11:6139: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:45694->104.36.82.11:6139: use of closed network connection Jun 22 00:09:16.254797295 (79h45m51.818120295s) [noise=true , address=104.36.82.20:6468 , piece_id=SQUT67FBQGV423UW4LOAECRWUJWCJ7AQ4ED3ACD3MZUTBRXUSYMA, seg_pos={0 0}, upload=63843]: Done uploading piece. err: ecclient: upload failed (node:123Cc2RmkYzkkbder6auanNvu1sTSEM6QD5aagnNC28DviG41Gi, address:104.36.82.20:6468): protocol: read tcp 159.223.143.39:35576->104.36.82.20:6468: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 22 00:09:16.254808564 (79h45m51.818131557s) [noise=true , address=77.90.31.17:4803 , piece_id=DXJWERKQZP445OUZQF6W33T5JXREOIMFYAMWDMDDMJS736PXWAXQ, seg_pos={0 0}, upload=63843]: Done uploading piece. err: ecclient: upload failed (node:12onacKUY32faZTFuyEjvMCZA4nVKiRKmTKh1WRFMfif2Kw1Q7Y, address:77.90.31.17:4803): protocol: read tcp 159.223.143.39:42574->77.90.31.17:4803: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 22 00:09:16.254859521 (79h45m51.81818252s) [noise=true , address=79.127.205.235:5664 , piece_id=NXG2ACQ4GJGVX7PQYTR5MCRWRPK6Y5FXNUN7YVETYC3U4ROXMOFA, seg_pos={0 0}, upload=63843]: Done uploading piece. err: ecclient: upload failed (node:12p8Gg6yYNUbFiEkHLHDFdnPC2w4AHgguBYv2ce34AYWXQzhHER, address:79.127.205.235:5664): protocol: write tcp 159.223.143.39:54186->79.127.205.235:5664: use of closed network connection; write tcp 159.223.143.39:54186->79.127.205.235:5664: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:54186->79.127.205.235:5664: use of closed network connection Jun 22 00:09:16.254895863 (79h45m51.818218866s) [seg_pos={0 0}, upload=63843]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jun 22 00:09:16.276852311 (79h45m51.840175305s) [upload=63843]: Done uploading segments... Jun 22 00:09:16.276903113 (79h45m51.840226102s) [upload=63843]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 28.220704385s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [..............................X.X...........................] duration: 53.439563093s FAIL (2025-06-22 00:28:46 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== g_pos={0 0}, upload=63843]: Uploading piece... Jun 22 00:08:16.324082782 (79h44m51.88740579s) [noise=true , address=104.36.82.52:4103 , piece_id=MWVPOZMAAFPZD6HDK4SGJOKW5IMNYG75R3YA7LOLMNJVQOLP3SBQ, seg_pos={0 0}, upload=63843]: Uploading piece... Jun 22 00:08:16.324387615 (79h44m51.887710618s) [noise=true , address=45.91.116.5:5308 , piece_id=QZA34FLZEJNVE2TUHVF5ZMTG2WP7RTNNOJF7AZ4YLYCA22XJPCZA, seg_pos={0 0}, upload=63843]: Uploading piece... Jun 22 00:08:16.324693763 (79h44m51.888016771s) [noise=true , address=77.90.31.14:4693 , piece_id=QAVXJYMYM3XOYRGUPHMK257ZNL7CUMR6UBGSJM647C73ECJDCVIA, seg_pos={0 0}, upload=63843]: Uploading piece... Jun 22 00:08:16.324987498 (79h44m51.888310498s) [noise=true , address=104.36.82.7:5938 , piece_id=YRRM6JLCSPEV2UZI5CICTMVPW2MTIK7P5K5F5S53AEV2DEMJFA3Q, seg_pos={0 0}, upload=63843]: Uploading piece... Jun 22 00:08:16.325315636 (79h44m51.888638643s) [noise=true , address=104.36.82.21:6473 , piece_id=3M34BSPUAFRF5NXPWIXKH25KX67RCHVXQCS5MFS5S7X7RYF676RQ, seg_pos={0 0}, upload=63843]: Uploading piece... Jun 22 00:08:16.394058454 (79h44m51.957381445s) [upload=63843]: Next returned nil segment Jun 22 00:08:16.394072985 (79h44m51.957395986s) [upload=63843]: Waiting for error group managing segments... Jun 22 00:08:16.397574195 (79h44m51.960897187s) [noise=true , address=45.91.116.5:5308 , piece_id=QZA34FLZEJNVE2TUHVF5ZMTG2WP7RTNNOJF7AZ4YLYCA22XJPCZA, seg_pos={0 0}, upload=63843]: Done uploading piece. err: <nil> Jun 22 00:08:16.459349430 (79h44m52.022672432s) [noise=true , address=104.36.82.54:4173 , piece_id=O2FBSZLX7FIDIDJWGWV2RMIZ7DZ5WYKS7JLZIOK64D5TRKSP3U7Q, seg_pos={0 0}, upload=63843]: Done uploading piece. err: ecclient: upload failed (node:1w2v4iyYtJtcdz7shYhG8CjCDQ1dWjeJVwY5Ubi7Ex5Y38e2i1, address:104.36.82.54:4173): protocol: storage node overloaded, request limit: 2000 Jun 22 00:08:16.518220521 (79h44m52.081543528s) [noise=true , address=77.90.31.12:4605 , piece_id=AITH3LEQCYAWVX4JXU7LDBV4LRIA3Y2LI3FNTOISOMRQ4LWNZ2VQ, seg_pos={0 0}, upload=63843]: Done uploading piece. err: <nil> Jun 22 00:08:17.014258958 (79h44m52.577581952s) [noise=true , address=64.49.13.137:4500 , piece_id=FMD3ECZINI2YWY32VYSH7GKCITBALAEDUMYCNTWTAJQVY55Y2MLQ, seg_pos={0 0}, upload=63843]: Done uploading piece. err: ecclient: upload failed (node:15TwVQ6CHDbtqS4XW86n9BuyjdqpaJo4ZKPgPnmhSm6EPDrnd9, address:64.49.13.137:4500): protocol: storage node overloaded, request limit: 2000 Jun 22 00:08:17.153129516 (79h44m52.716452522s) [noise=true , address=104.36.82.16:6328 , piece_id=4TEPB6TBOWSY5KR52EVGRYVGHAKBEDTUHMFF3KJPQKZXYRGDRBJQ, seg_pos={0 0}, upload=63843]: Done uploading piece. err: ecclient: upload failed (node:1S8PSdkLyaiURWkPGhcABSh3481hoswzukT1NKvpTkSp75vKDJ, address:104.36.82.16:6328): protocol: storage node overloaded, request limit: 2000 Jun 22 00:08:17.275208950 (79h44m52.83853195s) [noise=true , address=79.127.205.235:5664 , piece_id=XB53MDJKRRNXK6SAMHX2RHAEO6SAHTKI7732ERADDN7JCWDZKNVA, seg_pos={0 0}, upload=63843]: Done uploading piece. err: ecclient: upload failed (node:12p8Gg6yYNUbFiEkHLHDFdnPC2w4AHgguBYv2ce34AYWXQzhHER, address:79.127.205.235:5664): protocol: storage node overloaded, request limit: 2000 Jun 22 00:08:17.605904252 (79h44m53.169227253s) [noise=true , address=77.90.31.21:4887 , piece_id=3JB4YN5JQ2KHD3LXGYAH2VQDBYAM63SYBNIVNUK2BAVW6TG6L6EA, seg_pos={0 0}, upload=63843]: Done uploading piece. err: ecclient: upload failed (node:12hE2JH2wY7TepnbCVpSxkTcVifqui3QK42tLh1neU2BxZ2BRFJ, address:77.90.31.21:4887): protocol: storage node overloaded, request limit: 2000 Jun 22 00:08:17.865919302 (79h44m53.429242294s) [noise=true , address=77.90.31.14:4693 , piece_id=QAVXJYMYM3XOYRGUPHMK257ZNL7CUMR6UBGSJM647C73ECJDCVIA, seg_pos={0 0}, upload=63843]: Done uploading piece. err: <nil> Jun 22 00:08:19.517357026 (79h44m55.080680046s) [noise=true , address=104.36.82.10:6108 , piece_id=QTWYQJQ22NWG7I6WRKMC6AL7YYPHVEBN2LIUVIFZHC5WMDPKPANQ, seg_pos={0 0}, upload=63843]: Done uploading piece. err: <nil> Jun 22 00:08:20.580254458 (79h44m56.143577448s) [noise=true , address=104.36.82.7:5938 , piece_id=YRRM6JLCSPEV2UZI5CICTMVPW2MTIK7P5K5F5S53AEV2DEMJFA3Q, seg_pos={0 0}, upload=63843]: Done uploading piece. err: <nil> Jun 22 00:08:20.752577874 (79h44m56.315900867s) [noise=true , address=104.36.82.20:6466 , piece_id=P7UZBSRQEY3ORJAXMYSNYBPTYNGDBSSKZL74Z2GSJNMON3VZBOZQ, seg_pos={0 0}, upload=63843]: Done uploading piece. err: <nil> Jun 22 00:08:24.515800663 (79h45m0.079123665s) [noise=true , address=79.127.219.38:4028 , piece_id=QZDOXSRJI6MXMIUNJLVRKZBSCRMDI54KPFDSR4D4JF6S2RZ4SV7A, seg_pos={0 0}, upload=63843]: Done uploading piece. err: <nil> Jun 22 00:08:25.948556776 (79h45m1.511879777s) [noise=true , address=104.36.82.21:6473 , piece_id=3M34BSPUAFRF5NXPWIXKH25KX67RCHVXQCS5MFS5S7X7RYF676RQ, seg_pos={0 0}, upload=63843]: Done uploading piece. err: <nil> Jun 22 00:08:32.636880825 (79h45m8.200203818s) [noise=true , address=104.36.82.56:4261 , piece_id=2RGJ5T5RFUR67IIUA5CUVXFL4RVBO7GVLDLAVUZCRHWAITSBOEWA, seg_pos={0 0}, upload=63843]: Done uploading piece. err: <nil> Jun 22 00:08:35.525047894 (79h45m11.088370884s) [noise=true , address=104.36.82.19:6406 , piece_id=ERRJIBJ6LCOK3QRXCJSASUGTFUL76SS3HZJI2VQ64HWTSUMGVHZA, seg_pos={0 0}, upload=63843]: Done uploading piece. err: <nil> Jun 22 00:08:40.407828721 (79h45m15.971151709s) [noise=true , address=104.36.82.14:6234 , piece_id=WUFOSCTTAKQZLII6GTESOCKX4MJZM6CA45XBQUUKGS6S7JQASLUQ, seg_pos={0 0}, upload=63843]: Done uploading piece. err: <nil> Jun 22 00:08:52.755279221 (79h45m28.318602228s) [noise=true , address=104.36.82.52:4103 , piece_id=MWVPOZMAAFPZD6HDK4SGJOKW5IMNYG75R3YA7LOLMNJVQOLP3SBQ, seg_pos={0 0}, upload=63843]: Done uploading piece. err: <nil> Jun 22 00:09:16.049083116 (79h45m51.612406117s) [noise=true , address=104.36.82.55:4219 , piece_id=BMAN2MOGHY3DLY5O3GYQUSAUQ7A77YSTD7GEHUAUSSPE5EE7LJQQ, seg_pos={0 0}, upload=63843]: Done uploading piece. err: <nil> Jun 22 00:09:16.067949159 (79h45m51.631272164s) [noise=true , address=45.91.116.7:5012 , piece_id=7TSTZEQJHXFCT4UIUEOF6Q3JML3YJRJRJWHO3ZYPGVY66BCSDZ7Q, seg_pos={0 0}, upload=63843]: Uploading piece... Jun 22 00:09:16.067962327 (79h45m51.63128533s) [noise=true , address=77.90.31.17:4803 , piece_id=DXJWERKQZP445OUZQF6W33T5JXREOIMFYAMWDMDDMJS736PXWAXQ, seg_pos={0 0}, upload=63843]: Uploading piece... Jun 22 00:09:16.068067279 (79h45m51.631390289s) [noise=true , address=104.36.82.20:6468 , piece_id=SQUT67FBQGV423UW4LOAECRWUJWCJ7AQ4ED3ACD3MZUTBRXUSYMA, seg_pos={0 0}, upload=63843]: Uploading piece... Jun 22 00:09:16.068445650 (79h45m51.631768659s) [noise=true , address=104.36.82.11:6139 , piece_id=QQUWKI2WTCYU44QDNA4OOBWSPVTCMMMLEXEPOASV35A4UZVPW3TQ, seg_pos={0 0}, upload=63843]: Uploading piece... Jun 22 00:09:16.068751557 (79h45m51.632074575s) [noise=true , address=79.127.205.235:5664 , piece_id=NXG2ACQ4GJGVX7PQYTR5MCRWRPK6Y5FXNUN7YVETYC3U4ROXMOFA, seg_pos={0 0}, upload=63843]: Uploading piece... Jun 22 00:09:16.254420059 (79h45m51.817743058s) [noise=true , address=45.91.116.7:5012 , piece_id=7TSTZEQJHXFCT4UIUEOF6Q3JML3YJRJRJWHO3ZYPGVY66BCSDZ7Q, seg_pos={0 0}, upload=63843]: Done uploading piece. err: ecclient: upload failed (node:1smg5DiC8TRGnnmoQrRpvScvehv8aqtA2DaKtD1JtAhrhnm96t, address:45.91.116.7:5012): protocol: expected piece hash; context deadline exceeded; EOF Jun 22 00:09:16.254721861 (79h45m51.818044861s) [noise=true , address=104.36.82.11:6139 , piece_id=QQUWKI2WTCYU44QDNA4OOBWSPVTCMMMLEXEPOASV35A4UZVPW3TQ, seg_pos={0 0}, upload=63843]: Done uploading piece. err: ecclient: upload failed (node:1EiZjV2QMYue3KQDsNRhhdGpCQjybDv54wTuXsJTNtyeV8hHeh, address:104.36.82.11:6139): protocol: write tcp 159.223.143.39:45694->104.36.82.11:6139: use of closed network connection; write tcp 159.223.143.39:45694->104.36.82.11:6139: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:45694->104.36.82.11:6139: use of closed network connection Jun 22 00:09:16.254797295 (79h45m51.818120295s) [noise=true , address=104.36.82.20:6468 , piece_id=SQUT67FBQGV423UW4LOAECRWUJWCJ7AQ4ED3ACD3MZUTBRXUSYMA, seg_pos={0 0}, upload=63843]: Done uploading piece. err: ecclient: upload failed (node:123Cc2RmkYzkkbder6auanNvu1sTSEM6QD5aagnNC28DviG41Gi, address:104.36.82.20:6468): protocol: read tcp 159.223.143.39:35576->104.36.82.20:6468: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 22 00:09:16.254808564 (79h45m51.818131557s) [noise=true , address=77.90.31.17:4803 , piece_id=DXJWERKQZP445OUZQF6W33T5JXREOIMFYAMWDMDDMJS736PXWAXQ, seg_pos={0 0}, upload=63843]: Done uploading piece. err: ecclient: upload failed (node:12onacKUY32faZTFuyEjvMCZA4nVKiRKmTKh1WRFMfif2Kw1Q7Y, address:77.90.31.17:4803): protocol: read tcp 159.223.143.39:42574->77.90.31.17:4803: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 22 00:09:16.254859521 (79h45m51.81818252s) [noise=true , address=79.127.205.235:5664 , piece_id=NXG2ACQ4GJGVX7PQYTR5MCRWRPK6Y5FXNUN7YVETYC3U4ROXMOFA, seg_pos={0 0}, upload=63843]: Done uploading piece. err: ecclient: upload failed (node:12p8Gg6yYNUbFiEkHLHDFdnPC2w4AHgguBYv2ce34AYWXQzhHER, address:79.127.205.235:5664): protocol: write tcp 159.223.143.39:54186->79.127.205.235:5664: use of closed network connection; write tcp 159.223.143.39:54186->79.127.205.235:5664: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:54186->79.127.205.235:5664: use of closed network connection Jun 22 00:09:16.254895863 (79h45m51.818218866s) [seg_pos={0 0}, upload=63843]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jun 22 00:09:16.276852311 (79h45m51.840175305s) [upload=63843]: Done uploading segments... Jun 22 00:09:16.276903113 (79h45m51.840226102s) [upload=63843]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 53.439591126s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [....X.........................X.........X...................] duration: 18.558121645s FAIL (2025-06-22 00:28:42 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 18.5581467s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [.....X.........................X.........X..................] duration: 32.210429977s FAIL (2025-06-22 00:28:17 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 32.210454444s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [...............................X.X..........................] duration: 18.703455464s FAIL (2025-06-22 00:27:45 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== g_pos={0 0}, upload=63843]: Uploading piece... Jun 22 00:08:16.324082782 (79h44m51.88740579s) [noise=true , address=104.36.82.52:4103 , piece_id=MWVPOZMAAFPZD6HDK4SGJOKW5IMNYG75R3YA7LOLMNJVQOLP3SBQ, seg_pos={0 0}, upload=63843]: Uploading piece... Jun 22 00:08:16.324387615 (79h44m51.887710618s) [noise=true , address=45.91.116.5:5308 , piece_id=QZA34FLZEJNVE2TUHVF5ZMTG2WP7RTNNOJF7AZ4YLYCA22XJPCZA, seg_pos={0 0}, upload=63843]: Uploading piece... Jun 22 00:08:16.324693763 (79h44m51.888016771s) [noise=true , address=77.90.31.14:4693 , piece_id=QAVXJYMYM3XOYRGUPHMK257ZNL7CUMR6UBGSJM647C73ECJDCVIA, seg_pos={0 0}, upload=63843]: Uploading piece... Jun 22 00:08:16.324987498 (79h44m51.888310498s) [noise=true , address=104.36.82.7:5938 , piece_id=YRRM6JLCSPEV2UZI5CICTMVPW2MTIK7P5K5F5S53AEV2DEMJFA3Q, seg_pos={0 0}, upload=63843]: Uploading piece... Jun 22 00:08:16.325315636 (79h44m51.888638643s) [noise=true , address=104.36.82.21:6473 , piece_id=3M34BSPUAFRF5NXPWIXKH25KX67RCHVXQCS5MFS5S7X7RYF676RQ, seg_pos={0 0}, upload=63843]: Uploading piece... Jun 22 00:08:16.394058454 (79h44m51.957381445s) [upload=63843]: Next returned nil segment Jun 22 00:08:16.394072985 (79h44m51.957395986s) [upload=63843]: Waiting for error group managing segments... Jun 22 00:08:16.397574195 (79h44m51.960897187s) [noise=true , address=45.91.116.5:5308 , piece_id=QZA34FLZEJNVE2TUHVF5ZMTG2WP7RTNNOJF7AZ4YLYCA22XJPCZA, seg_pos={0 0}, upload=63843]: Done uploading piece. err: <nil> Jun 22 00:08:16.459349430 (79h44m52.022672432s) [noise=true , address=104.36.82.54:4173 , piece_id=O2FBSZLX7FIDIDJWGWV2RMIZ7DZ5WYKS7JLZIOK64D5TRKSP3U7Q, seg_pos={0 0}, upload=63843]: Done uploading piece. err: ecclient: upload failed (node:1w2v4iyYtJtcdz7shYhG8CjCDQ1dWjeJVwY5Ubi7Ex5Y38e2i1, address:104.36.82.54:4173): protocol: storage node overloaded, request limit: 2000 Jun 22 00:08:16.518220521 (79h44m52.081543528s) [noise=true , address=77.90.31.12:4605 , piece_id=AITH3LEQCYAWVX4JXU7LDBV4LRIA3Y2LI3FNTOISOMRQ4LWNZ2VQ, seg_pos={0 0}, upload=63843]: Done uploading piece. err: <nil> Jun 22 00:08:17.014258958 (79h44m52.577581952s) [noise=true , address=64.49.13.137:4500 , piece_id=FMD3ECZINI2YWY32VYSH7GKCITBALAEDUMYCNTWTAJQVY55Y2MLQ, seg_pos={0 0}, upload=63843]: Done uploading piece. err: ecclient: upload failed (node:15TwVQ6CHDbtqS4XW86n9BuyjdqpaJo4ZKPgPnmhSm6EPDrnd9, address:64.49.13.137:4500): protocol: storage node overloaded, request limit: 2000 Jun 22 00:08:17.153129516 (79h44m52.716452522s) [noise=true , address=104.36.82.16:6328 , piece_id=4TEPB6TBOWSY5KR52EVGRYVGHAKBEDTUHMFF3KJPQKZXYRGDRBJQ, seg_pos={0 0}, upload=63843]: Done uploading piece. err: ecclient: upload failed (node:1S8PSdkLyaiURWkPGhcABSh3481hoswzukT1NKvpTkSp75vKDJ, address:104.36.82.16:6328): protocol: storage node overloaded, request limit: 2000 Jun 22 00:08:17.275208950 (79h44m52.83853195s) [noise=true , address=79.127.205.235:5664 , piece_id=XB53MDJKRRNXK6SAMHX2RHAEO6SAHTKI7732ERADDN7JCWDZKNVA, seg_pos={0 0}, upload=63843]: Done uploading piece. err: ecclient: upload failed (node:12p8Gg6yYNUbFiEkHLHDFdnPC2w4AHgguBYv2ce34AYWXQzhHER, address:79.127.205.235:5664): protocol: storage node overloaded, request limit: 2000 Jun 22 00:08:17.605904252 (79h44m53.169227253s) [noise=true , address=77.90.31.21:4887 , piece_id=3JB4YN5JQ2KHD3LXGYAH2VQDBYAM63SYBNIVNUK2BAVW6TG6L6EA, seg_pos={0 0}, upload=63843]: Done uploading piece. err: ecclient: upload failed (node:12hE2JH2wY7TepnbCVpSxkTcVifqui3QK42tLh1neU2BxZ2BRFJ, address:77.90.31.21:4887): protocol: storage node overloaded, request limit: 2000 Jun 22 00:08:17.865919302 (79h44m53.429242294s) [noise=true , address=77.90.31.14:4693 , piece_id=QAVXJYMYM3XOYRGUPHMK257ZNL7CUMR6UBGSJM647C73ECJDCVIA, seg_pos={0 0}, upload=63843]: Done uploading piece. err: <nil> Jun 22 00:08:19.517357026 (79h44m55.080680046s) [noise=true , address=104.36.82.10:6108 , piece_id=QTWYQJQ22NWG7I6WRKMC6AL7YYPHVEBN2LIUVIFZHC5WMDPKPANQ, seg_pos={0 0}, upload=63843]: Done uploading piece. err: <nil> Jun 22 00:08:20.580254458 (79h44m56.143577448s) [noise=true , address=104.36.82.7:5938 , piece_id=YRRM6JLCSPEV2UZI5CICTMVPW2MTIK7P5K5F5S53AEV2DEMJFA3Q, seg_pos={0 0}, upload=63843]: Done uploading piece. err: <nil> Jun 22 00:08:20.752577874 (79h44m56.315900867s) [noise=true , address=104.36.82.20:6466 , piece_id=P7UZBSRQEY3ORJAXMYSNYBPTYNGDBSSKZL74Z2GSJNMON3VZBOZQ, seg_pos={0 0}, upload=63843]: Done uploading piece. err: <nil> Jun 22 00:08:24.515800663 (79h45m0.079123665s) [noise=true , address=79.127.219.38:4028 , piece_id=QZDOXSRJI6MXMIUNJLVRKZBSCRMDI54KPFDSR4D4JF6S2RZ4SV7A, seg_pos={0 0}, upload=63843]: Done uploading piece. err: <nil> Jun 22 00:08:25.948556776 (79h45m1.511879777s) [noise=true , address=104.36.82.21:6473 , piece_id=3M34BSPUAFRF5NXPWIXKH25KX67RCHVXQCS5MFS5S7X7RYF676RQ, seg_pos={0 0}, upload=63843]: Done uploading piece. err: <nil> Jun 22 00:08:32.636880825 (79h45m8.200203818s) [noise=true , address=104.36.82.56:4261 , piece_id=2RGJ5T5RFUR67IIUA5CUVXFL4RVBO7GVLDLAVUZCRHWAITSBOEWA, seg_pos={0 0}, upload=63843]: Done uploading piece. err: <nil> Jun 22 00:08:35.525047894 (79h45m11.088370884s) [noise=true , address=104.36.82.19:6406 , piece_id=ERRJIBJ6LCOK3QRXCJSASUGTFUL76SS3HZJI2VQ64HWTSUMGVHZA, seg_pos={0 0}, upload=63843]: Done uploading piece. err: <nil> Jun 22 00:08:40.407828721 (79h45m15.971151709s) [noise=true , address=104.36.82.14:6234 , piece_id=WUFOSCTTAKQZLII6GTESOCKX4MJZM6CA45XBQUUKGS6S7JQASLUQ, seg_pos={0 0}, upload=63843]: Done uploading piece. err: <nil> Jun 22 00:08:52.755279221 (79h45m28.318602228s) [noise=true , address=104.36.82.52:4103 , piece_id=MWVPOZMAAFPZD6HDK4SGJOKW5IMNYG75R3YA7LOLMNJVQOLP3SBQ, seg_pos={0 0}, upload=63843]: Done uploading piece. err: <nil> Jun 22 00:09:16.049083116 (79h45m51.612406117s) [noise=true , address=104.36.82.55:4219 , piece_id=BMAN2MOGHY3DLY5O3GYQUSAUQ7A77YSTD7GEHUAUSSPE5EE7LJQQ, seg_pos={0 0}, upload=63843]: Done uploading piece. err: <nil> Jun 22 00:09:16.067949159 (79h45m51.631272164s) [noise=true , address=45.91.116.7:5012 , piece_id=7TSTZEQJHXFCT4UIUEOF6Q3JML3YJRJRJWHO3ZYPGVY66BCSDZ7Q, seg_pos={0 0}, upload=63843]: Uploading piece... Jun 22 00:09:16.067962327 (79h45m51.63128533s) [noise=true , address=77.90.31.17:4803 , piece_id=DXJWERKQZP445OUZQF6W33T5JXREOIMFYAMWDMDDMJS736PXWAXQ, seg_pos={0 0}, upload=63843]: Uploading piece... Jun 22 00:09:16.068067279 (79h45m51.631390289s) [noise=true , address=104.36.82.20:6468 , piece_id=SQUT67FBQGV423UW4LOAECRWUJWCJ7AQ4ED3ACD3MZUTBRXUSYMA, seg_pos={0 0}, upload=63843]: Uploading piece... Jun 22 00:09:16.068445650 (79h45m51.631768659s) [noise=true , address=104.36.82.11:6139 , piece_id=QQUWKI2WTCYU44QDNA4OOBWSPVTCMMMLEXEPOASV35A4UZVPW3TQ, seg_pos={0 0}, upload=63843]: Uploading piece... Jun 22 00:09:16.068751557 (79h45m51.632074575s) [noise=true , address=79.127.205.235:5664 , piece_id=NXG2ACQ4GJGVX7PQYTR5MCRWRPK6Y5FXNUN7YVETYC3U4ROXMOFA, seg_pos={0 0}, upload=63843]: Uploading piece... Jun 22 00:09:16.254420059 (79h45m51.817743058s) [noise=true , address=45.91.116.7:5012 , piece_id=7TSTZEQJHXFCT4UIUEOF6Q3JML3YJRJRJWHO3ZYPGVY66BCSDZ7Q, seg_pos={0 0}, upload=63843]: Done uploading piece. err: ecclient: upload failed (node:1smg5DiC8TRGnnmoQrRpvScvehv8aqtA2DaKtD1JtAhrhnm96t, address:45.91.116.7:5012): protocol: expected piece hash; context deadline exceeded; EOF Jun 22 00:09:16.254721861 (79h45m51.818044861s) [noise=true , address=104.36.82.11:6139 , piece_id=QQUWKI2WTCYU44QDNA4OOBWSPVTCMMMLEXEPOASV35A4UZVPW3TQ, seg_pos={0 0}, upload=63843]: Done uploading piece. err: ecclient: upload failed (node:1EiZjV2QMYue3KQDsNRhhdGpCQjybDv54wTuXsJTNtyeV8hHeh, address:104.36.82.11:6139): protocol: write tcp 159.223.143.39:45694->104.36.82.11:6139: use of closed network connection; write tcp 159.223.143.39:45694->104.36.82.11:6139: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:45694->104.36.82.11:6139: use of closed network connection Jun 22 00:09:16.254797295 (79h45m51.818120295s) [noise=true , address=104.36.82.20:6468 , piece_id=SQUT67FBQGV423UW4LOAECRWUJWCJ7AQ4ED3ACD3MZUTBRXUSYMA, seg_pos={0 0}, upload=63843]: Done uploading piece. err: ecclient: upload failed (node:123Cc2RmkYzkkbder6auanNvu1sTSEM6QD5aagnNC28DviG41Gi, address:104.36.82.20:6468): protocol: read tcp 159.223.143.39:35576->104.36.82.20:6468: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 22 00:09:16.254808564 (79h45m51.818131557s) [noise=true , address=77.90.31.17:4803 , piece_id=DXJWERKQZP445OUZQF6W33T5JXREOIMFYAMWDMDDMJS736PXWAXQ, seg_pos={0 0}, upload=63843]: Done uploading piece. err: ecclient: upload failed (node:12onacKUY32faZTFuyEjvMCZA4nVKiRKmTKh1WRFMfif2Kw1Q7Y, address:77.90.31.17:4803): protocol: read tcp 159.223.143.39:42574->77.90.31.17:4803: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 22 00:09:16.254859521 (79h45m51.81818252s) [noise=true , address=79.127.205.235:5664 , piece_id=NXG2ACQ4GJGVX7PQYTR5MCRWRPK6Y5FXNUN7YVETYC3U4ROXMOFA, seg_pos={0 0}, upload=63843]: Done uploading piece. err: ecclient: upload failed (node:12p8Gg6yYNUbFiEkHLHDFdnPC2w4AHgguBYv2ce34AYWXQzhHER, address:79.127.205.235:5664): protocol: write tcp 159.223.143.39:54186->79.127.205.235:5664: use of closed network connection; write tcp 159.223.143.39:54186->79.127.205.235:5664: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:54186->79.127.205.235:5664: use of closed network connection Jun 22 00:09:16.254895863 (79h45m51.818218866s) [seg_pos={0 0}, upload=63843]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jun 22 00:09:16.276852311 (79h45m51.840175305s) [upload=63843]: Done uploading segments... Jun 22 00:09:16.276903113 (79h45m51.840226102s) [upload=63843]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 18.703495921s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [................................X.X.........................] duration: 36.047086142s FAIL (2025-06-22 00:27:26 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== g_pos={0 0}, upload=63843]: Uploading piece... Jun 22 00:08:16.324082782 (79h44m51.88740579s) [noise=true , address=104.36.82.52:4103 , piece_id=MWVPOZMAAFPZD6HDK4SGJOKW5IMNYG75R3YA7LOLMNJVQOLP3SBQ, seg_pos={0 0}, upload=63843]: Uploading piece... Jun 22 00:08:16.324387615 (79h44m51.887710618s) [noise=true , address=45.91.116.5:5308 , piece_id=QZA34FLZEJNVE2TUHVF5ZMTG2WP7RTNNOJF7AZ4YLYCA22XJPCZA, seg_pos={0 0}, upload=63843]: Uploading piece... Jun 22 00:08:16.324693763 (79h44m51.888016771s) [noise=true , address=77.90.31.14:4693 , piece_id=QAVXJYMYM3XOYRGUPHMK257ZNL7CUMR6UBGSJM647C73ECJDCVIA, seg_pos={0 0}, upload=63843]: Uploading piece... Jun 22 00:08:16.324987498 (79h44m51.888310498s) [noise=true , address=104.36.82.7:5938 , piece_id=YRRM6JLCSPEV2UZI5CICTMVPW2MTIK7P5K5F5S53AEV2DEMJFA3Q, seg_pos={0 0}, upload=63843]: Uploading piece... Jun 22 00:08:16.325315636 (79h44m51.888638643s) [noise=true , address=104.36.82.21:6473 , piece_id=3M34BSPUAFRF5NXPWIXKH25KX67RCHVXQCS5MFS5S7X7RYF676RQ, seg_pos={0 0}, upload=63843]: Uploading piece... Jun 22 00:08:16.394058454 (79h44m51.957381445s) [upload=63843]: Next returned nil segment Jun 22 00:08:16.394072985 (79h44m51.957395986s) [upload=63843]: Waiting for error group managing segments... Jun 22 00:08:16.397574195 (79h44m51.960897187s) [noise=true , address=45.91.116.5:5308 , piece_id=QZA34FLZEJNVE2TUHVF5ZMTG2WP7RTNNOJF7AZ4YLYCA22XJPCZA, seg_pos={0 0}, upload=63843]: Done uploading piece. err: <nil> Jun 22 00:08:16.459349430 (79h44m52.022672432s) [noise=true , address=104.36.82.54:4173 , piece_id=O2FBSZLX7FIDIDJWGWV2RMIZ7DZ5WYKS7JLZIOK64D5TRKSP3U7Q, seg_pos={0 0}, upload=63843]: Done uploading piece. err: ecclient: upload failed (node:1w2v4iyYtJtcdz7shYhG8CjCDQ1dWjeJVwY5Ubi7Ex5Y38e2i1, address:104.36.82.54:4173): protocol: storage node overloaded, request limit: 2000 Jun 22 00:08:16.518220521 (79h44m52.081543528s) [noise=true , address=77.90.31.12:4605 , piece_id=AITH3LEQCYAWVX4JXU7LDBV4LRIA3Y2LI3FNTOISOMRQ4LWNZ2VQ, seg_pos={0 0}, upload=63843]: Done uploading piece. err: <nil> Jun 22 00:08:17.014258958 (79h44m52.577581952s) [noise=true , address=64.49.13.137:4500 , piece_id=FMD3ECZINI2YWY32VYSH7GKCITBALAEDUMYCNTWTAJQVY55Y2MLQ, seg_pos={0 0}, upload=63843]: Done uploading piece. err: ecclient: upload failed (node:15TwVQ6CHDbtqS4XW86n9BuyjdqpaJo4ZKPgPnmhSm6EPDrnd9, address:64.49.13.137:4500): protocol: storage node overloaded, request limit: 2000 Jun 22 00:08:17.153129516 (79h44m52.716452522s) [noise=true , address=104.36.82.16:6328 , piece_id=4TEPB6TBOWSY5KR52EVGRYVGHAKBEDTUHMFF3KJPQKZXYRGDRBJQ, seg_pos={0 0}, upload=63843]: Done uploading piece. err: ecclient: upload failed (node:1S8PSdkLyaiURWkPGhcABSh3481hoswzukT1NKvpTkSp75vKDJ, address:104.36.82.16:6328): protocol: storage node overloaded, request limit: 2000 Jun 22 00:08:17.275208950 (79h44m52.83853195s) [noise=true , address=79.127.205.235:5664 , piece_id=XB53MDJKRRNXK6SAMHX2RHAEO6SAHTKI7732ERADDN7JCWDZKNVA, seg_pos={0 0}, upload=63843]: Done uploading piece. err: ecclient: upload failed (node:12p8Gg6yYNUbFiEkHLHDFdnPC2w4AHgguBYv2ce34AYWXQzhHER, address:79.127.205.235:5664): protocol: storage node overloaded, request limit: 2000 Jun 22 00:08:17.605904252 (79h44m53.169227253s) [noise=true , address=77.90.31.21:4887 , piece_id=3JB4YN5JQ2KHD3LXGYAH2VQDBYAM63SYBNIVNUK2BAVW6TG6L6EA, seg_pos={0 0}, upload=63843]: Done uploading piece. err: ecclient: upload failed (node:12hE2JH2wY7TepnbCVpSxkTcVifqui3QK42tLh1neU2BxZ2BRFJ, address:77.90.31.21:4887): protocol: storage node overloaded, request limit: 2000 Jun 22 00:08:17.865919302 (79h44m53.429242294s) [noise=true , address=77.90.31.14:4693 , piece_id=QAVXJYMYM3XOYRGUPHMK257ZNL7CUMR6UBGSJM647C73ECJDCVIA, seg_pos={0 0}, upload=63843]: Done uploading piece. err: <nil> Jun 22 00:08:19.517357026 (79h44m55.080680046s) [noise=true , address=104.36.82.10:6108 , piece_id=QTWYQJQ22NWG7I6WRKMC6AL7YYPHVEBN2LIUVIFZHC5WMDPKPANQ, seg_pos={0 0}, upload=63843]: Done uploading piece. err: <nil> Jun 22 00:08:20.580254458 (79h44m56.143577448s) [noise=true , address=104.36.82.7:5938 , piece_id=YRRM6JLCSPEV2UZI5CICTMVPW2MTIK7P5K5F5S53AEV2DEMJFA3Q, seg_pos={0 0}, upload=63843]: Done uploading piece. err: <nil> Jun 22 00:08:20.752577874 (79h44m56.315900867s) [noise=true , address=104.36.82.20:6466 , piece_id=P7UZBSRQEY3ORJAXMYSNYBPTYNGDBSSKZL74Z2GSJNMON3VZBOZQ, seg_pos={0 0}, upload=63843]: Done uploading piece. err: <nil> Jun 22 00:08:24.515800663 (79h45m0.079123665s) [noise=true , address=79.127.219.38:4028 , piece_id=QZDOXSRJI6MXMIUNJLVRKZBSCRMDI54KPFDSR4D4JF6S2RZ4SV7A, seg_pos={0 0}, upload=63843]: Done uploading piece. err: <nil> Jun 22 00:08:25.948556776 (79h45m1.511879777s) [noise=true , address=104.36.82.21:6473 , piece_id=3M34BSPUAFRF5NXPWIXKH25KX67RCHVXQCS5MFS5S7X7RYF676RQ, seg_pos={0 0}, upload=63843]: Done uploading piece. err: <nil> Jun 22 00:08:32.636880825 (79h45m8.200203818s) [noise=true , address=104.36.82.56:4261 , piece_id=2RGJ5T5RFUR67IIUA5CUVXFL4RVBO7GVLDLAVUZCRHWAITSBOEWA, seg_pos={0 0}, upload=63843]: Done uploading piece. err: <nil> Jun 22 00:08:35.525047894 (79h45m11.088370884s) [noise=true , address=104.36.82.19:6406 , piece_id=ERRJIBJ6LCOK3QRXCJSASUGTFUL76SS3HZJI2VQ64HWTSUMGVHZA, seg_pos={0 0}, upload=63843]: Done uploading piece. err: <nil> Jun 22 00:08:40.407828721 (79h45m15.971151709s) [noise=true , address=104.36.82.14:6234 , piece_id=WUFOSCTTAKQZLII6GTESOCKX4MJZM6CA45XBQUUKGS6S7JQASLUQ, seg_pos={0 0}, upload=63843]: Done uploading piece. err: <nil> Jun 22 00:08:52.755279221 (79h45m28.318602228s) [noise=true , address=104.36.82.52:4103 , piece_id=MWVPOZMAAFPZD6HDK4SGJOKW5IMNYG75R3YA7LOLMNJVQOLP3SBQ, seg_pos={0 0}, upload=63843]: Done uploading piece. err: <nil> Jun 22 00:09:16.049083116 (79h45m51.612406117s) [noise=true , address=104.36.82.55:4219 , piece_id=BMAN2MOGHY3DLY5O3GYQUSAUQ7A77YSTD7GEHUAUSSPE5EE7LJQQ, seg_pos={0 0}, upload=63843]: Done uploading piece. err: <nil> Jun 22 00:09:16.067949159 (79h45m51.631272164s) [noise=true , address=45.91.116.7:5012 , piece_id=7TSTZEQJHXFCT4UIUEOF6Q3JML3YJRJRJWHO3ZYPGVY66BCSDZ7Q, seg_pos={0 0}, upload=63843]: Uploading piece... Jun 22 00:09:16.067962327 (79h45m51.63128533s) [noise=true , address=77.90.31.17:4803 , piece_id=DXJWERKQZP445OUZQF6W33T5JXREOIMFYAMWDMDDMJS736PXWAXQ, seg_pos={0 0}, upload=63843]: Uploading piece... Jun 22 00:09:16.068067279 (79h45m51.631390289s) [noise=true , address=104.36.82.20:6468 , piece_id=SQUT67FBQGV423UW4LOAECRWUJWCJ7AQ4ED3ACD3MZUTBRXUSYMA, seg_pos={0 0}, upload=63843]: Uploading piece... Jun 22 00:09:16.068445650 (79h45m51.631768659s) [noise=true , address=104.36.82.11:6139 , piece_id=QQUWKI2WTCYU44QDNA4OOBWSPVTCMMMLEXEPOASV35A4UZVPW3TQ, seg_pos={0 0}, upload=63843]: Uploading piece... Jun 22 00:09:16.068751557 (79h45m51.632074575s) [noise=true , address=79.127.205.235:5664 , piece_id=NXG2ACQ4GJGVX7PQYTR5MCRWRPK6Y5FXNUN7YVETYC3U4ROXMOFA, seg_pos={0 0}, upload=63843]: Uploading piece... Jun 22 00:09:16.254420059 (79h45m51.817743058s) [noise=true , address=45.91.116.7:5012 , piece_id=7TSTZEQJHXFCT4UIUEOF6Q3JML3YJRJRJWHO3ZYPGVY66BCSDZ7Q, seg_pos={0 0}, upload=63843]: Done uploading piece. err: ecclient: upload failed (node:1smg5DiC8TRGnnmoQrRpvScvehv8aqtA2DaKtD1JtAhrhnm96t, address:45.91.116.7:5012): protocol: expected piece hash; context deadline exceeded; EOF Jun 22 00:09:16.254721861 (79h45m51.818044861s) [noise=true , address=104.36.82.11:6139 , piece_id=QQUWKI2WTCYU44QDNA4OOBWSPVTCMMMLEXEPOASV35A4UZVPW3TQ, seg_pos={0 0}, upload=63843]: Done uploading piece. err: ecclient: upload failed (node:1EiZjV2QMYue3KQDsNRhhdGpCQjybDv54wTuXsJTNtyeV8hHeh, address:104.36.82.11:6139): protocol: write tcp 159.223.143.39:45694->104.36.82.11:6139: use of closed network connection; write tcp 159.223.143.39:45694->104.36.82.11:6139: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:45694->104.36.82.11:6139: use of closed network connection Jun 22 00:09:16.254797295 (79h45m51.818120295s) [noise=true , address=104.36.82.20:6468 , piece_id=SQUT67FBQGV423UW4LOAECRWUJWCJ7AQ4ED3ACD3MZUTBRXUSYMA, seg_pos={0 0}, upload=63843]: Done uploading piece. err: ecclient: upload failed (node:123Cc2RmkYzkkbder6auanNvu1sTSEM6QD5aagnNC28DviG41Gi, address:104.36.82.20:6468): protocol: read tcp 159.223.143.39:35576->104.36.82.20:6468: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 22 00:09:16.254808564 (79h45m51.818131557s) [noise=true , address=77.90.31.17:4803 , piece_id=DXJWERKQZP445OUZQF6W33T5JXREOIMFYAMWDMDDMJS736PXWAXQ, seg_pos={0 0}, upload=63843]: Done uploading piece. err: ecclient: upload failed (node:12onacKUY32faZTFuyEjvMCZA4nVKiRKmTKh1WRFMfif2Kw1Q7Y, address:77.90.31.17:4803): protocol: read tcp 159.223.143.39:42574->77.90.31.17:4803: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 22 00:09:16.254859521 (79h45m51.81818252s) [noise=true , address=79.127.205.235:5664 , piece_id=NXG2ACQ4GJGVX7PQYTR5MCRWRPK6Y5FXNUN7YVETYC3U4ROXMOFA, seg_pos={0 0}, upload=63843]: Done uploading piece. err: ecclient: upload failed (node:12p8Gg6yYNUbFiEkHLHDFdnPC2w4AHgguBYv2ce34AYWXQzhHER, address:79.127.205.235:5664): protocol: write tcp 159.223.143.39:54186->79.127.205.235:5664: use of closed network connection; write tcp 159.223.143.39:54186->79.127.205.235:5664: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:54186->79.127.205.235:5664: use of closed network connection Jun 22 00:09:16.254895863 (79h45m51.818218866s) [seg_pos={0 0}, upload=63843]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jun 22 00:09:16.276852311 (79h45m51.840175305s) [upload=63843]: Done uploading segments... Jun 22 00:09:16.276903113 (79h45m51.840226102s) [upload=63843]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 36.047116953s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [......X.........................X.........X.................] duration: 23.195997013s FAIL (2025-06-22 00:26:55 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 23.196019712s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [.................................X.X........................] duration: 25.266316112s FAIL (2025-06-22 00:26:37 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== g_pos={0 0}, upload=63843]: Uploading piece... Jun 22 00:08:16.324082782 (79h44m51.88740579s) [noise=true , address=104.36.82.52:4103 , piece_id=MWVPOZMAAFPZD6HDK4SGJOKW5IMNYG75R3YA7LOLMNJVQOLP3SBQ, seg_pos={0 0}, upload=63843]: Uploading piece... Jun 22 00:08:16.324387615 (79h44m51.887710618s) [noise=true , address=45.91.116.5:5308 , piece_id=QZA34FLZEJNVE2TUHVF5ZMTG2WP7RTNNOJF7AZ4YLYCA22XJPCZA, seg_pos={0 0}, upload=63843]: Uploading piece... Jun 22 00:08:16.324693763 (79h44m51.888016771s) [noise=true , address=77.90.31.14:4693 , piece_id=QAVXJYMYM3XOYRGUPHMK257ZNL7CUMR6UBGSJM647C73ECJDCVIA, seg_pos={0 0}, upload=63843]: Uploading piece... Jun 22 00:08:16.324987498 (79h44m51.888310498s) [noise=true , address=104.36.82.7:5938 , piece_id=YRRM6JLCSPEV2UZI5CICTMVPW2MTIK7P5K5F5S53AEV2DEMJFA3Q, seg_pos={0 0}, upload=63843]: Uploading piece... Jun 22 00:08:16.325315636 (79h44m51.888638643s) [noise=true , address=104.36.82.21:6473 , piece_id=3M34BSPUAFRF5NXPWIXKH25KX67RCHVXQCS5MFS5S7X7RYF676RQ, seg_pos={0 0}, upload=63843]: Uploading piece... Jun 22 00:08:16.394058454 (79h44m51.957381445s) [upload=63843]: Next returned nil segment Jun 22 00:08:16.394072985 (79h44m51.957395986s) [upload=63843]: Waiting for error group managing segments... Jun 22 00:08:16.397574195 (79h44m51.960897187s) [noise=true , address=45.91.116.5:5308 , piece_id=QZA34FLZEJNVE2TUHVF5ZMTG2WP7RTNNOJF7AZ4YLYCA22XJPCZA, seg_pos={0 0}, upload=63843]: Done uploading piece. err: <nil> Jun 22 00:08:16.459349430 (79h44m52.022672432s) [noise=true , address=104.36.82.54:4173 , piece_id=O2FBSZLX7FIDIDJWGWV2RMIZ7DZ5WYKS7JLZIOK64D5TRKSP3U7Q, seg_pos={0 0}, upload=63843]: Done uploading piece. err: ecclient: upload failed (node:1w2v4iyYtJtcdz7shYhG8CjCDQ1dWjeJVwY5Ubi7Ex5Y38e2i1, address:104.36.82.54:4173): protocol: storage node overloaded, request limit: 2000 Jun 22 00:08:16.518220521 (79h44m52.081543528s) [noise=true , address=77.90.31.12:4605 , piece_id=AITH3LEQCYAWVX4JXU7LDBV4LRIA3Y2LI3FNTOISOMRQ4LWNZ2VQ, seg_pos={0 0}, upload=63843]: Done uploading piece. err: <nil> Jun 22 00:08:17.014258958 (79h44m52.577581952s) [noise=true , address=64.49.13.137:4500 , piece_id=FMD3ECZINI2YWY32VYSH7GKCITBALAEDUMYCNTWTAJQVY55Y2MLQ, seg_pos={0 0}, upload=63843]: Done uploading piece. err: ecclient: upload failed (node:15TwVQ6CHDbtqS4XW86n9BuyjdqpaJo4ZKPgPnmhSm6EPDrnd9, address:64.49.13.137:4500): protocol: storage node overloaded, request limit: 2000 Jun 22 00:08:17.153129516 (79h44m52.716452522s) [noise=true , address=104.36.82.16:6328 , piece_id=4TEPB6TBOWSY5KR52EVGRYVGHAKBEDTUHMFF3KJPQKZXYRGDRBJQ, seg_pos={0 0}, upload=63843]: Done uploading piece. err: ecclient: upload failed (node:1S8PSdkLyaiURWkPGhcABSh3481hoswzukT1NKvpTkSp75vKDJ, address:104.36.82.16:6328): protocol: storage node overloaded, request limit: 2000 Jun 22 00:08:17.275208950 (79h44m52.83853195s) [noise=true , address=79.127.205.235:5664 , piece_id=XB53MDJKRRNXK6SAMHX2RHAEO6SAHTKI7732ERADDN7JCWDZKNVA, seg_pos={0 0}, upload=63843]: Done uploading piece. err: ecclient: upload failed (node:12p8Gg6yYNUbFiEkHLHDFdnPC2w4AHgguBYv2ce34AYWXQzhHER, address:79.127.205.235:5664): protocol: storage node overloaded, request limit: 2000 Jun 22 00:08:17.605904252 (79h44m53.169227253s) [noise=true , address=77.90.31.21:4887 , piece_id=3JB4YN5JQ2KHD3LXGYAH2VQDBYAM63SYBNIVNUK2BAVW6TG6L6EA, seg_pos={0 0}, upload=63843]: Done uploading piece. err: ecclient: upload failed (node:12hE2JH2wY7TepnbCVpSxkTcVifqui3QK42tLh1neU2BxZ2BRFJ, address:77.90.31.21:4887): protocol: storage node overloaded, request limit: 2000 Jun 22 00:08:17.865919302 (79h44m53.429242294s) [noise=true , address=77.90.31.14:4693 , piece_id=QAVXJYMYM3XOYRGUPHMK257ZNL7CUMR6UBGSJM647C73ECJDCVIA, seg_pos={0 0}, upload=63843]: Done uploading piece. err: <nil> Jun 22 00:08:19.517357026 (79h44m55.080680046s) [noise=true , address=104.36.82.10:6108 , piece_id=QTWYQJQ22NWG7I6WRKMC6AL7YYPHVEBN2LIUVIFZHC5WMDPKPANQ, seg_pos={0 0}, upload=63843]: Done uploading piece. err: <nil> Jun 22 00:08:20.580254458 (79h44m56.143577448s) [noise=true , address=104.36.82.7:5938 , piece_id=YRRM6JLCSPEV2UZI5CICTMVPW2MTIK7P5K5F5S53AEV2DEMJFA3Q, seg_pos={0 0}, upload=63843]: Done uploading piece. err: <nil> Jun 22 00:08:20.752577874 (79h44m56.315900867s) [noise=true , address=104.36.82.20:6466 , piece_id=P7UZBSRQEY3ORJAXMYSNYBPTYNGDBSSKZL74Z2GSJNMON3VZBOZQ, seg_pos={0 0}, upload=63843]: Done uploading piece. err: <nil> Jun 22 00:08:24.515800663 (79h45m0.079123665s) [noise=true , address=79.127.219.38:4028 , piece_id=QZDOXSRJI6MXMIUNJLVRKZBSCRMDI54KPFDSR4D4JF6S2RZ4SV7A, seg_pos={0 0}, upload=63843]: Done uploading piece. err: <nil> Jun 22 00:08:25.948556776 (79h45m1.511879777s) [noise=true , address=104.36.82.21:6473 , piece_id=3M34BSPUAFRF5NXPWIXKH25KX67RCHVXQCS5MFS5S7X7RYF676RQ, seg_pos={0 0}, upload=63843]: Done uploading piece. err: <nil> Jun 22 00:08:32.636880825 (79h45m8.200203818s) [noise=true , address=104.36.82.56:4261 , piece_id=2RGJ5T5RFUR67IIUA5CUVXFL4RVBO7GVLDLAVUZCRHWAITSBOEWA, seg_pos={0 0}, upload=63843]: Done uploading piece. err: <nil> Jun 22 00:08:35.525047894 (79h45m11.088370884s) [noise=true , address=104.36.82.19:6406 , piece_id=ERRJIBJ6LCOK3QRXCJSASUGTFUL76SS3HZJI2VQ64HWTSUMGVHZA, seg_pos={0 0}, upload=63843]: Done uploading piece. err: <nil> Jun 22 00:08:40.407828721 (79h45m15.971151709s) [noise=true , address=104.36.82.14:6234 , piece_id=WUFOSCTTAKQZLII6GTESOCKX4MJZM6CA45XBQUUKGS6S7JQASLUQ, seg_pos={0 0}, upload=63843]: Done uploading piece. err: <nil> Jun 22 00:08:52.755279221 (79h45m28.318602228s) [noise=true , address=104.36.82.52:4103 , piece_id=MWVPOZMAAFPZD6HDK4SGJOKW5IMNYG75R3YA7LOLMNJVQOLP3SBQ, seg_pos={0 0}, upload=63843]: Done uploading piece. err: <nil> Jun 22 00:09:16.049083116 (79h45m51.612406117s) [noise=true , address=104.36.82.55:4219 , piece_id=BMAN2MOGHY3DLY5O3GYQUSAUQ7A77YSTD7GEHUAUSSPE5EE7LJQQ, seg_pos={0 0}, upload=63843]: Done uploading piece. err: <nil> Jun 22 00:09:16.067949159 (79h45m51.631272164s) [noise=true , address=45.91.116.7:5012 , piece_id=7TSTZEQJHXFCT4UIUEOF6Q3JML3YJRJRJWHO3ZYPGVY66BCSDZ7Q, seg_pos={0 0}, upload=63843]: Uploading piece... Jun 22 00:09:16.067962327 (79h45m51.63128533s) [noise=true , address=77.90.31.17:4803 , piece_id=DXJWERKQZP445OUZQF6W33T5JXREOIMFYAMWDMDDMJS736PXWAXQ, seg_pos={0 0}, upload=63843]: Uploading piece... Jun 22 00:09:16.068067279 (79h45m51.631390289s) [noise=true , address=104.36.82.20:6468 , piece_id=SQUT67FBQGV423UW4LOAECRWUJWCJ7AQ4ED3ACD3MZUTBRXUSYMA, seg_pos={0 0}, upload=63843]: Uploading piece... Jun 22 00:09:16.068445650 (79h45m51.631768659s) [noise=true , address=104.36.82.11:6139 , piece_id=QQUWKI2WTCYU44QDNA4OOBWSPVTCMMMLEXEPOASV35A4UZVPW3TQ, seg_pos={0 0}, upload=63843]: Uploading piece... Jun 22 00:09:16.068751557 (79h45m51.632074575s) [noise=true , address=79.127.205.235:5664 , piece_id=NXG2ACQ4GJGVX7PQYTR5MCRWRPK6Y5FXNUN7YVETYC3U4ROXMOFA, seg_pos={0 0}, upload=63843]: Uploading piece... Jun 22 00:09:16.254420059 (79h45m51.817743058s) [noise=true , address=45.91.116.7:5012 , piece_id=7TSTZEQJHXFCT4UIUEOF6Q3JML3YJRJRJWHO3ZYPGVY66BCSDZ7Q, seg_pos={0 0}, upload=63843]: Done uploading piece. err: ecclient: upload failed (node:1smg5DiC8TRGnnmoQrRpvScvehv8aqtA2DaKtD1JtAhrhnm96t, address:45.91.116.7:5012): protocol: expected piece hash; context deadline exceeded; EOF Jun 22 00:09:16.254721861 (79h45m51.818044861s) [noise=true , address=104.36.82.11:6139 , piece_id=QQUWKI2WTCYU44QDNA4OOBWSPVTCMMMLEXEPOASV35A4UZVPW3TQ, seg_pos={0 0}, upload=63843]: Done uploading piece. err: ecclient: upload failed (node:1EiZjV2QMYue3KQDsNRhhdGpCQjybDv54wTuXsJTNtyeV8hHeh, address:104.36.82.11:6139): protocol: write tcp 159.223.143.39:45694->104.36.82.11:6139: use of closed network connection; write tcp 159.223.143.39:45694->104.36.82.11:6139: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:45694->104.36.82.11:6139: use of closed network connection Jun 22 00:09:16.254797295 (79h45m51.818120295s) [noise=true , address=104.36.82.20:6468 , piece_id=SQUT67FBQGV423UW4LOAECRWUJWCJ7AQ4ED3ACD3MZUTBRXUSYMA, seg_pos={0 0}, upload=63843]: Done uploading piece. err: ecclient: upload failed (node:123Cc2RmkYzkkbder6auanNvu1sTSEM6QD5aagnNC28DviG41Gi, address:104.36.82.20:6468): protocol: read tcp 159.223.143.39:35576->104.36.82.20:6468: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 22 00:09:16.254808564 (79h45m51.818131557s) [noise=true , address=77.90.31.17:4803 , piece_id=DXJWERKQZP445OUZQF6W33T5JXREOIMFYAMWDMDDMJS736PXWAXQ, seg_pos={0 0}, upload=63843]: Done uploading piece. err: ecclient: upload failed (node:12onacKUY32faZTFuyEjvMCZA4nVKiRKmTKh1WRFMfif2Kw1Q7Y, address:77.90.31.17:4803): protocol: read tcp 159.223.143.39:42574->77.90.31.17:4803: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 22 00:09:16.254859521 (79h45m51.81818252s) [noise=true , address=79.127.205.235:5664 , piece_id=NXG2ACQ4GJGVX7PQYTR5MCRWRPK6Y5FXNUN7YVETYC3U4ROXMOFA, seg_pos={0 0}, upload=63843]: Done uploading piece. err: ecclient: upload failed (node:12p8Gg6yYNUbFiEkHLHDFdnPC2w4AHgguBYv2ce34AYWXQzhHER, address:79.127.205.235:5664): protocol: write tcp 159.223.143.39:54186->79.127.205.235:5664: use of closed network connection; write tcp 159.223.143.39:54186->79.127.205.235:5664: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:54186->79.127.205.235:5664: use of closed network connection Jun 22 00:09:16.254895863 (79h45m51.818218866s) [seg_pos={0 0}, upload=63843]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jun 22 00:09:16.276852311 (79h45m51.840175305s) [upload=63843]: Done uploading segments... Jun 22 00:09:16.276903113 (79h45m51.840226102s) [upload=63843]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 25.266337345s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [..................................X.X.......................] duration: 51.03626133s FAIL (2025-06-22 00:26:11 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== g_pos={0 0}, upload=63843]: Uploading piece... Jun 22 00:08:16.324082782 (79h44m51.88740579s) [noise=true , address=104.36.82.52:4103 , piece_id=MWVPOZMAAFPZD6HDK4SGJOKW5IMNYG75R3YA7LOLMNJVQOLP3SBQ, seg_pos={0 0}, upload=63843]: Uploading piece... Jun 22 00:08:16.324387615 (79h44m51.887710618s) [noise=true , address=45.91.116.5:5308 , piece_id=QZA34FLZEJNVE2TUHVF5ZMTG2WP7RTNNOJF7AZ4YLYCA22XJPCZA, seg_pos={0 0}, upload=63843]: Uploading piece... Jun 22 00:08:16.324693763 (79h44m51.888016771s) [noise=true , address=77.90.31.14:4693 , piece_id=QAVXJYMYM3XOYRGUPHMK257ZNL7CUMR6UBGSJM647C73ECJDCVIA, seg_pos={0 0}, upload=63843]: Uploading piece... Jun 22 00:08:16.324987498 (79h44m51.888310498s) [noise=true , address=104.36.82.7:5938 , piece_id=YRRM6JLCSPEV2UZI5CICTMVPW2MTIK7P5K5F5S53AEV2DEMJFA3Q, seg_pos={0 0}, upload=63843]: Uploading piece... Jun 22 00:08:16.325315636 (79h44m51.888638643s) [noise=true , address=104.36.82.21:6473 , piece_id=3M34BSPUAFRF5NXPWIXKH25KX67RCHVXQCS5MFS5S7X7RYF676RQ, seg_pos={0 0}, upload=63843]: Uploading piece... Jun 22 00:08:16.394058454 (79h44m51.957381445s) [upload=63843]: Next returned nil segment Jun 22 00:08:16.394072985 (79h44m51.957395986s) [upload=63843]: Waiting for error group managing segments... Jun 22 00:08:16.397574195 (79h44m51.960897187s) [noise=true , address=45.91.116.5:5308 , piece_id=QZA34FLZEJNVE2TUHVF5ZMTG2WP7RTNNOJF7AZ4YLYCA22XJPCZA, seg_pos={0 0}, upload=63843]: Done uploading piece. err: <nil> Jun 22 00:08:16.459349430 (79h44m52.022672432s) [noise=true , address=104.36.82.54:4173 , piece_id=O2FBSZLX7FIDIDJWGWV2RMIZ7DZ5WYKS7JLZIOK64D5TRKSP3U7Q, seg_pos={0 0}, upload=63843]: Done uploading piece. err: ecclient: upload failed (node:1w2v4iyYtJtcdz7shYhG8CjCDQ1dWjeJVwY5Ubi7Ex5Y38e2i1, address:104.36.82.54:4173): protocol: storage node overloaded, request limit: 2000 Jun 22 00:08:16.518220521 (79h44m52.081543528s) [noise=true , address=77.90.31.12:4605 , piece_id=AITH3LEQCYAWVX4JXU7LDBV4LRIA3Y2LI3FNTOISOMRQ4LWNZ2VQ, seg_pos={0 0}, upload=63843]: Done uploading piece. err: <nil> Jun 22 00:08:17.014258958 (79h44m52.577581952s) [noise=true , address=64.49.13.137:4500 , piece_id=FMD3ECZINI2YWY32VYSH7GKCITBALAEDUMYCNTWTAJQVY55Y2MLQ, seg_pos={0 0}, upload=63843]: Done uploading piece. err: ecclient: upload failed (node:15TwVQ6CHDbtqS4XW86n9BuyjdqpaJo4ZKPgPnmhSm6EPDrnd9, address:64.49.13.137:4500): protocol: storage node overloaded, request limit: 2000 Jun 22 00:08:17.153129516 (79h44m52.716452522s) [noise=true , address=104.36.82.16:6328 , piece_id=4TEPB6TBOWSY5KR52EVGRYVGHAKBEDTUHMFF3KJPQKZXYRGDRBJQ, seg_pos={0 0}, upload=63843]: Done uploading piece. err: ecclient: upload failed (node:1S8PSdkLyaiURWkPGhcABSh3481hoswzukT1NKvpTkSp75vKDJ, address:104.36.82.16:6328): protocol: storage node overloaded, request limit: 2000 Jun 22 00:08:17.275208950 (79h44m52.83853195s) [noise=true , address=79.127.205.235:5664 , piece_id=XB53MDJKRRNXK6SAMHX2RHAEO6SAHTKI7732ERADDN7JCWDZKNVA, seg_pos={0 0}, upload=63843]: Done uploading piece. err: ecclient: upload failed (node:12p8Gg6yYNUbFiEkHLHDFdnPC2w4AHgguBYv2ce34AYWXQzhHER, address:79.127.205.235:5664): protocol: storage node overloaded, request limit: 2000 Jun 22 00:08:17.605904252 (79h44m53.169227253s) [noise=true , address=77.90.31.21:4887 , piece_id=3JB4YN5JQ2KHD3LXGYAH2VQDBYAM63SYBNIVNUK2BAVW6TG6L6EA, seg_pos={0 0}, upload=63843]: Done uploading piece. err: ecclient: upload failed (node:12hE2JH2wY7TepnbCVpSxkTcVifqui3QK42tLh1neU2BxZ2BRFJ, address:77.90.31.21:4887): protocol: storage node overloaded, request limit: 2000 Jun 22 00:08:17.865919302 (79h44m53.429242294s) [noise=true , address=77.90.31.14:4693 , piece_id=QAVXJYMYM3XOYRGUPHMK257ZNL7CUMR6UBGSJM647C73ECJDCVIA, seg_pos={0 0}, upload=63843]: Done uploading piece. err: <nil> Jun 22 00:08:19.517357026 (79h44m55.080680046s) [noise=true , address=104.36.82.10:6108 , piece_id=QTWYQJQ22NWG7I6WRKMC6AL7YYPHVEBN2LIUVIFZHC5WMDPKPANQ, seg_pos={0 0}, upload=63843]: Done uploading piece. err: <nil> Jun 22 00:08:20.580254458 (79h44m56.143577448s) [noise=true , address=104.36.82.7:5938 , piece_id=YRRM6JLCSPEV2UZI5CICTMVPW2MTIK7P5K5F5S53AEV2DEMJFA3Q, seg_pos={0 0}, upload=63843]: Done uploading piece. err: <nil> Jun 22 00:08:20.752577874 (79h44m56.315900867s) [noise=true , address=104.36.82.20:6466 , piece_id=P7UZBSRQEY3ORJAXMYSNYBPTYNGDBSSKZL74Z2GSJNMON3VZBOZQ, seg_pos={0 0}, upload=63843]: Done uploading piece. err: <nil> Jun 22 00:08:24.515800663 (79h45m0.079123665s) [noise=true , address=79.127.219.38:4028 , piece_id=QZDOXSRJI6MXMIUNJLVRKZBSCRMDI54KPFDSR4D4JF6S2RZ4SV7A, seg_pos={0 0}, upload=63843]: Done uploading piece. err: <nil> Jun 22 00:08:25.948556776 (79h45m1.511879777s) [noise=true , address=104.36.82.21:6473 , piece_id=3M34BSPUAFRF5NXPWIXKH25KX67RCHVXQCS5MFS5S7X7RYF676RQ, seg_pos={0 0}, upload=63843]: Done uploading piece. err: <nil> Jun 22 00:08:32.636880825 (79h45m8.200203818s) [noise=true , address=104.36.82.56:4261 , piece_id=2RGJ5T5RFUR67IIUA5CUVXFL4RVBO7GVLDLAVUZCRHWAITSBOEWA, seg_pos={0 0}, upload=63843]: Done uploading piece. err: <nil> Jun 22 00:08:35.525047894 (79h45m11.088370884s) [noise=true , address=104.36.82.19:6406 , piece_id=ERRJIBJ6LCOK3QRXCJSASUGTFUL76SS3HZJI2VQ64HWTSUMGVHZA, seg_pos={0 0}, upload=63843]: Done uploading piece. err: <nil> Jun 22 00:08:40.407828721 (79h45m15.971151709s) [noise=true , address=104.36.82.14:6234 , piece_id=WUFOSCTTAKQZLII6GTESOCKX4MJZM6CA45XBQUUKGS6S7JQASLUQ, seg_pos={0 0}, upload=63843]: Done uploading piece. err: <nil> Jun 22 00:08:52.755279221 (79h45m28.318602228s) [noise=true , address=104.36.82.52:4103 , piece_id=MWVPOZMAAFPZD6HDK4SGJOKW5IMNYG75R3YA7LOLMNJVQOLP3SBQ, seg_pos={0 0}, upload=63843]: Done uploading piece. err: <nil> Jun 22 00:09:16.049083116 (79h45m51.612406117s) [noise=true , address=104.36.82.55:4219 , piece_id=BMAN2MOGHY3DLY5O3GYQUSAUQ7A77YSTD7GEHUAUSSPE5EE7LJQQ, seg_pos={0 0}, upload=63843]: Done uploading piece. err: <nil> Jun 22 00:09:16.067949159 (79h45m51.631272164s) [noise=true , address=45.91.116.7:5012 , piece_id=7TSTZEQJHXFCT4UIUEOF6Q3JML3YJRJRJWHO3ZYPGVY66BCSDZ7Q, seg_pos={0 0}, upload=63843]: Uploading piece... Jun 22 00:09:16.067962327 (79h45m51.63128533s) [noise=true , address=77.90.31.17:4803 , piece_id=DXJWERKQZP445OUZQF6W33T5JXREOIMFYAMWDMDDMJS736PXWAXQ, seg_pos={0 0}, upload=63843]: Uploading piece... Jun 22 00:09:16.068067279 (79h45m51.631390289s) [noise=true , address=104.36.82.20:6468 , piece_id=SQUT67FBQGV423UW4LOAECRWUJWCJ7AQ4ED3ACD3MZUTBRXUSYMA, seg_pos={0 0}, upload=63843]: Uploading piece... Jun 22 00:09:16.068445650 (79h45m51.631768659s) [noise=true , address=104.36.82.11:6139 , piece_id=QQUWKI2WTCYU44QDNA4OOBWSPVTCMMMLEXEPOASV35A4UZVPW3TQ, seg_pos={0 0}, upload=63843]: Uploading piece... Jun 22 00:09:16.068751557 (79h45m51.632074575s) [noise=true , address=79.127.205.235:5664 , piece_id=NXG2ACQ4GJGVX7PQYTR5MCRWRPK6Y5FXNUN7YVETYC3U4ROXMOFA, seg_pos={0 0}, upload=63843]: Uploading piece... Jun 22 00:09:16.254420059 (79h45m51.817743058s) [noise=true , address=45.91.116.7:5012 , piece_id=7TSTZEQJHXFCT4UIUEOF6Q3JML3YJRJRJWHO3ZYPGVY66BCSDZ7Q, seg_pos={0 0}, upload=63843]: Done uploading piece. err: ecclient: upload failed (node:1smg5DiC8TRGnnmoQrRpvScvehv8aqtA2DaKtD1JtAhrhnm96t, address:45.91.116.7:5012): protocol: expected piece hash; context deadline exceeded; EOF Jun 22 00:09:16.254721861 (79h45m51.818044861s) [noise=true , address=104.36.82.11:6139 , piece_id=QQUWKI2WTCYU44QDNA4OOBWSPVTCMMMLEXEPOASV35A4UZVPW3TQ, seg_pos={0 0}, upload=63843]: Done uploading piece. err: ecclient: upload failed (node:1EiZjV2QMYue3KQDsNRhhdGpCQjybDv54wTuXsJTNtyeV8hHeh, address:104.36.82.11:6139): protocol: write tcp 159.223.143.39:45694->104.36.82.11:6139: use of closed network connection; write tcp 159.223.143.39:45694->104.36.82.11:6139: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:45694->104.36.82.11:6139: use of closed network connection Jun 22 00:09:16.254797295 (79h45m51.818120295s) [noise=true , address=104.36.82.20:6468 , piece_id=SQUT67FBQGV423UW4LOAECRWUJWCJ7AQ4ED3ACD3MZUTBRXUSYMA, seg_pos={0 0}, upload=63843]: Done uploading piece. err: ecclient: upload failed (node:123Cc2RmkYzkkbder6auanNvu1sTSEM6QD5aagnNC28DviG41Gi, address:104.36.82.20:6468): protocol: read tcp 159.223.143.39:35576->104.36.82.20:6468: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 22 00:09:16.254808564 (79h45m51.818131557s) [noise=true , address=77.90.31.17:4803 , piece_id=DXJWERKQZP445OUZQF6W33T5JXREOIMFYAMWDMDDMJS736PXWAXQ, seg_pos={0 0}, upload=63843]: Done uploading piece. err: ecclient: upload failed (node:12onacKUY32faZTFuyEjvMCZA4nVKiRKmTKh1WRFMfif2Kw1Q7Y, address:77.90.31.17:4803): protocol: read tcp 159.223.143.39:42574->77.90.31.17:4803: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 22 00:09:16.254859521 (79h45m51.81818252s) [noise=true , address=79.127.205.235:5664 , piece_id=NXG2ACQ4GJGVX7PQYTR5MCRWRPK6Y5FXNUN7YVETYC3U4ROXMOFA, seg_pos={0 0}, upload=63843]: Done uploading piece. err: ecclient: upload failed (node:12p8Gg6yYNUbFiEkHLHDFdnPC2w4AHgguBYv2ce34AYWXQzhHER, address:79.127.205.235:5664): protocol: write tcp 159.223.143.39:54186->79.127.205.235:5664: use of closed network connection; write tcp 159.223.143.39:54186->79.127.205.235:5664: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:54186->79.127.205.235:5664: use of closed network connection Jun 22 00:09:16.254895863 (79h45m51.818218866s) [seg_pos={0 0}, upload=63843]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jun 22 00:09:16.276852311 (79h45m51.840175305s) [upload=63843]: Done uploading segments... Jun 22 00:09:16.276903113 (79h45m51.840226102s) [upload=63843]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 51.036291659s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [.......X.........................X.........X................] duration: 16.640108931s FAIL (2025-06-22 00:26:05 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 16.640135112s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [........X.........................X.........X...............] duration: 23.867406051s FAIL (2025-06-22 00:25:29 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 23.867900196s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [.........X.........................X.........X..............] duration: 42.367400952s FAIL (2025-06-22 00:25:05 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 42.367418904s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [...................................X.X......................] duration: 15.695022985s FAIL (2025-06-22 00:24:52 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== g_pos={0 0}, upload=63843]: Uploading piece... Jun 22 00:08:16.324082782 (79h44m51.88740579s) [noise=true , address=104.36.82.52:4103 , piece_id=MWVPOZMAAFPZD6HDK4SGJOKW5IMNYG75R3YA7LOLMNJVQOLP3SBQ, seg_pos={0 0}, upload=63843]: Uploading piece... Jun 22 00:08:16.324387615 (79h44m51.887710618s) [noise=true , address=45.91.116.5:5308 , piece_id=QZA34FLZEJNVE2TUHVF5ZMTG2WP7RTNNOJF7AZ4YLYCA22XJPCZA, seg_pos={0 0}, upload=63843]: Uploading piece... Jun 22 00:08:16.324693763 (79h44m51.888016771s) [noise=true , address=77.90.31.14:4693 , piece_id=QAVXJYMYM3XOYRGUPHMK257ZNL7CUMR6UBGSJM647C73ECJDCVIA, seg_pos={0 0}, upload=63843]: Uploading piece... Jun 22 00:08:16.324987498 (79h44m51.888310498s) [noise=true , address=104.36.82.7:5938 , piece_id=YRRM6JLCSPEV2UZI5CICTMVPW2MTIK7P5K5F5S53AEV2DEMJFA3Q, seg_pos={0 0}, upload=63843]: Uploading piece... Jun 22 00:08:16.325315636 (79h44m51.888638643s) [noise=true , address=104.36.82.21:6473 , piece_id=3M34BSPUAFRF5NXPWIXKH25KX67RCHVXQCS5MFS5S7X7RYF676RQ, seg_pos={0 0}, upload=63843]: Uploading piece... Jun 22 00:08:16.394058454 (79h44m51.957381445s) [upload=63843]: Next returned nil segment Jun 22 00:08:16.394072985 (79h44m51.957395986s) [upload=63843]: Waiting for error group managing segments... Jun 22 00:08:16.397574195 (79h44m51.960897187s) [noise=true , address=45.91.116.5:5308 , piece_id=QZA34FLZEJNVE2TUHVF5ZMTG2WP7RTNNOJF7AZ4YLYCA22XJPCZA, seg_pos={0 0}, upload=63843]: Done uploading piece. err: <nil> Jun 22 00:08:16.459349430 (79h44m52.022672432s) [noise=true , address=104.36.82.54:4173 , piece_id=O2FBSZLX7FIDIDJWGWV2RMIZ7DZ5WYKS7JLZIOK64D5TRKSP3U7Q, seg_pos={0 0}, upload=63843]: Done uploading piece. err: ecclient: upload failed (node:1w2v4iyYtJtcdz7shYhG8CjCDQ1dWjeJVwY5Ubi7Ex5Y38e2i1, address:104.36.82.54:4173): protocol: storage node overloaded, request limit: 2000 Jun 22 00:08:16.518220521 (79h44m52.081543528s) [noise=true , address=77.90.31.12:4605 , piece_id=AITH3LEQCYAWVX4JXU7LDBV4LRIA3Y2LI3FNTOISOMRQ4LWNZ2VQ, seg_pos={0 0}, upload=63843]: Done uploading piece. err: <nil> Jun 22 00:08:17.014258958 (79h44m52.577581952s) [noise=true , address=64.49.13.137:4500 , piece_id=FMD3ECZINI2YWY32VYSH7GKCITBALAEDUMYCNTWTAJQVY55Y2MLQ, seg_pos={0 0}, upload=63843]: Done uploading piece. err: ecclient: upload failed (node:15TwVQ6CHDbtqS4XW86n9BuyjdqpaJo4ZKPgPnmhSm6EPDrnd9, address:64.49.13.137:4500): protocol: storage node overloaded, request limit: 2000 Jun 22 00:08:17.153129516 (79h44m52.716452522s) [noise=true , address=104.36.82.16:6328 , piece_id=4TEPB6TBOWSY5KR52EVGRYVGHAKBEDTUHMFF3KJPQKZXYRGDRBJQ, seg_pos={0 0}, upload=63843]: Done uploading piece. err: ecclient: upload failed (node:1S8PSdkLyaiURWkPGhcABSh3481hoswzukT1NKvpTkSp75vKDJ, address:104.36.82.16:6328): protocol: storage node overloaded, request limit: 2000 Jun 22 00:08:17.275208950 (79h44m52.83853195s) [noise=true , address=79.127.205.235:5664 , piece_id=XB53MDJKRRNXK6SAMHX2RHAEO6SAHTKI7732ERADDN7JCWDZKNVA, seg_pos={0 0}, upload=63843]: Done uploading piece. err: ecclient: upload failed (node:12p8Gg6yYNUbFiEkHLHDFdnPC2w4AHgguBYv2ce34AYWXQzhHER, address:79.127.205.235:5664): protocol: storage node overloaded, request limit: 2000 Jun 22 00:08:17.605904252 (79h44m53.169227253s) [noise=true , address=77.90.31.21:4887 , piece_id=3JB4YN5JQ2KHD3LXGYAH2VQDBYAM63SYBNIVNUK2BAVW6TG6L6EA, seg_pos={0 0}, upload=63843]: Done uploading piece. err: ecclient: upload failed (node:12hE2JH2wY7TepnbCVpSxkTcVifqui3QK42tLh1neU2BxZ2BRFJ, address:77.90.31.21:4887): protocol: storage node overloaded, request limit: 2000 Jun 22 00:08:17.865919302 (79h44m53.429242294s) [noise=true , address=77.90.31.14:4693 , piece_id=QAVXJYMYM3XOYRGUPHMK257ZNL7CUMR6UBGSJM647C73ECJDCVIA, seg_pos={0 0}, upload=63843]: Done uploading piece. err: <nil> Jun 22 00:08:19.517357026 (79h44m55.080680046s) [noise=true , address=104.36.82.10:6108 , piece_id=QTWYQJQ22NWG7I6WRKMC6AL7YYPHVEBN2LIUVIFZHC5WMDPKPANQ, seg_pos={0 0}, upload=63843]: Done uploading piece. err: <nil> Jun 22 00:08:20.580254458 (79h44m56.143577448s) [noise=true , address=104.36.82.7:5938 , piece_id=YRRM6JLCSPEV2UZI5CICTMVPW2MTIK7P5K5F5S53AEV2DEMJFA3Q, seg_pos={0 0}, upload=63843]: Done uploading piece. err: <nil> Jun 22 00:08:20.752577874 (79h44m56.315900867s) [noise=true , address=104.36.82.20:6466 , piece_id=P7UZBSRQEY3ORJAXMYSNYBPTYNGDBSSKZL74Z2GSJNMON3VZBOZQ, seg_pos={0 0}, upload=63843]: Done uploading piece. err: <nil> Jun 22 00:08:24.515800663 (79h45m0.079123665s) [noise=true , address=79.127.219.38:4028 , piece_id=QZDOXSRJI6MXMIUNJLVRKZBSCRMDI54KPFDSR4D4JF6S2RZ4SV7A, seg_pos={0 0}, upload=63843]: Done uploading piece. err: <nil> Jun 22 00:08:25.948556776 (79h45m1.511879777s) [noise=true , address=104.36.82.21:6473 , piece_id=3M34BSPUAFRF5NXPWIXKH25KX67RCHVXQCS5MFS5S7X7RYF676RQ, seg_pos={0 0}, upload=63843]: Done uploading piece. err: <nil> Jun 22 00:08:32.636880825 (79h45m8.200203818s) [noise=true , address=104.36.82.56:4261 , piece_id=2RGJ5T5RFUR67IIUA5CUVXFL4RVBO7GVLDLAVUZCRHWAITSBOEWA, seg_pos={0 0}, upload=63843]: Done uploading piece. err: <nil> Jun 22 00:08:35.525047894 (79h45m11.088370884s) [noise=true , address=104.36.82.19:6406 , piece_id=ERRJIBJ6LCOK3QRXCJSASUGTFUL76SS3HZJI2VQ64HWTSUMGVHZA, seg_pos={0 0}, upload=63843]: Done uploading piece. err: <nil> Jun 22 00:08:40.407828721 (79h45m15.971151709s) [noise=true , address=104.36.82.14:6234 , piece_id=WUFOSCTTAKQZLII6GTESOCKX4MJZM6CA45XBQUUKGS6S7JQASLUQ, seg_pos={0 0}, upload=63843]: Done uploading piece. err: <nil> Jun 22 00:08:52.755279221 (79h45m28.318602228s) [noise=true , address=104.36.82.52:4103 , piece_id=MWVPOZMAAFPZD6HDK4SGJOKW5IMNYG75R3YA7LOLMNJVQOLP3SBQ, seg_pos={0 0}, upload=63843]: Done uploading piece. err: <nil> Jun 22 00:09:16.049083116 (79h45m51.612406117s) [noise=true , address=104.36.82.55:4219 , piece_id=BMAN2MOGHY3DLY5O3GYQUSAUQ7A77YSTD7GEHUAUSSPE5EE7LJQQ, seg_pos={0 0}, upload=63843]: Done uploading piece. err: <nil> Jun 22 00:09:16.067949159 (79h45m51.631272164s) [noise=true , address=45.91.116.7:5012 , piece_id=7TSTZEQJHXFCT4UIUEOF6Q3JML3YJRJRJWHO3ZYPGVY66BCSDZ7Q, seg_pos={0 0}, upload=63843]: Uploading piece... Jun 22 00:09:16.067962327 (79h45m51.63128533s) [noise=true , address=77.90.31.17:4803 , piece_id=DXJWERKQZP445OUZQF6W33T5JXREOIMFYAMWDMDDMJS736PXWAXQ, seg_pos={0 0}, upload=63843]: Uploading piece... Jun 22 00:09:16.068067279 (79h45m51.631390289s) [noise=true , address=104.36.82.20:6468 , piece_id=SQUT67FBQGV423UW4LOAECRWUJWCJ7AQ4ED3ACD3MZUTBRXUSYMA, seg_pos={0 0}, upload=63843]: Uploading piece... Jun 22 00:09:16.068445650 (79h45m51.631768659s) [noise=true , address=104.36.82.11:6139 , piece_id=QQUWKI2WTCYU44QDNA4OOBWSPVTCMMMLEXEPOASV35A4UZVPW3TQ, seg_pos={0 0}, upload=63843]: Uploading piece... Jun 22 00:09:16.068751557 (79h45m51.632074575s) [noise=true , address=79.127.205.235:5664 , piece_id=NXG2ACQ4GJGVX7PQYTR5MCRWRPK6Y5FXNUN7YVETYC3U4ROXMOFA, seg_pos={0 0}, upload=63843]: Uploading piece... Jun 22 00:09:16.254420059 (79h45m51.817743058s) [noise=true , address=45.91.116.7:5012 , piece_id=7TSTZEQJHXFCT4UIUEOF6Q3JML3YJRJRJWHO3ZYPGVY66BCSDZ7Q, seg_pos={0 0}, upload=63843]: Done uploading piece. err: ecclient: upload failed (node:1smg5DiC8TRGnnmoQrRpvScvehv8aqtA2DaKtD1JtAhrhnm96t, address:45.91.116.7:5012): protocol: expected piece hash; context deadline exceeded; EOF Jun 22 00:09:16.254721861 (79h45m51.818044861s) [noise=true , address=104.36.82.11:6139 , piece_id=QQUWKI2WTCYU44QDNA4OOBWSPVTCMMMLEXEPOASV35A4UZVPW3TQ, seg_pos={0 0}, upload=63843]: Done uploading piece. err: ecclient: upload failed (node:1EiZjV2QMYue3KQDsNRhhdGpCQjybDv54wTuXsJTNtyeV8hHeh, address:104.36.82.11:6139): protocol: write tcp 159.223.143.39:45694->104.36.82.11:6139: use of closed network connection; write tcp 159.223.143.39:45694->104.36.82.11:6139: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:45694->104.36.82.11:6139: use of closed network connection Jun 22 00:09:16.254797295 (79h45m51.818120295s) [noise=true , address=104.36.82.20:6468 , piece_id=SQUT67FBQGV423UW4LOAECRWUJWCJ7AQ4ED3ACD3MZUTBRXUSYMA, seg_pos={0 0}, upload=63843]: Done uploading piece. err: ecclient: upload failed (node:123Cc2RmkYzkkbder6auanNvu1sTSEM6QD5aagnNC28DviG41Gi, address:104.36.82.20:6468): protocol: read tcp 159.223.143.39:35576->104.36.82.20:6468: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 22 00:09:16.254808564 (79h45m51.818131557s) [noise=true , address=77.90.31.17:4803 , piece_id=DXJWERKQZP445OUZQF6W33T5JXREOIMFYAMWDMDDMJS736PXWAXQ, seg_pos={0 0}, upload=63843]: Done uploading piece. err: ecclient: upload failed (node:12onacKUY32faZTFuyEjvMCZA4nVKiRKmTKh1WRFMfif2Kw1Q7Y, address:77.90.31.17:4803): protocol: read tcp 159.223.143.39:42574->77.90.31.17:4803: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 22 00:09:16.254859521 (79h45m51.81818252s) [noise=true , address=79.127.205.235:5664 , piece_id=NXG2ACQ4GJGVX7PQYTR5MCRWRPK6Y5FXNUN7YVETYC3U4ROXMOFA, seg_pos={0 0}, upload=63843]: Done uploading piece. err: ecclient: upload failed (node:12p8Gg6yYNUbFiEkHLHDFdnPC2w4AHgguBYv2ce34AYWXQzhHER, address:79.127.205.235:5664): protocol: write tcp 159.223.143.39:54186->79.127.205.235:5664: use of closed network connection; write tcp 159.223.143.39:54186->79.127.205.235:5664: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:54186->79.127.205.235:5664: use of closed network connection Jun 22 00:09:16.254895863 (79h45m51.818218866s) [seg_pos={0 0}, upload=63843]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jun 22 00:09:16.276852311 (79h45m51.840175305s) [upload=63843]: Done uploading segments... Jun 22 00:09:16.276903113 (79h45m51.840226102s) [upload=63843]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 15.6950527s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [....................................X.X.....................] duration: 16.605138815s FAIL (2025-06-22 00:24:16 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== g_pos={0 0}, upload=63843]: Uploading piece... Jun 22 00:08:16.324082782 (79h44m51.88740579s) [noise=true , address=104.36.82.52:4103 , piece_id=MWVPOZMAAFPZD6HDK4SGJOKW5IMNYG75R3YA7LOLMNJVQOLP3SBQ, seg_pos={0 0}, upload=63843]: Uploading piece... Jun 22 00:08:16.324387615 (79h44m51.887710618s) [noise=true , address=45.91.116.5:5308 , piece_id=QZA34FLZEJNVE2TUHVF5ZMTG2WP7RTNNOJF7AZ4YLYCA22XJPCZA, seg_pos={0 0}, upload=63843]: Uploading piece... Jun 22 00:08:16.324693763 (79h44m51.888016771s) [noise=true , address=77.90.31.14:4693 , piece_id=QAVXJYMYM3XOYRGUPHMK257ZNL7CUMR6UBGSJM647C73ECJDCVIA, seg_pos={0 0}, upload=63843]: Uploading piece... Jun 22 00:08:16.324987498 (79h44m51.888310498s) [noise=true , address=104.36.82.7:5938 , piece_id=YRRM6JLCSPEV2UZI5CICTMVPW2MTIK7P5K5F5S53AEV2DEMJFA3Q, seg_pos={0 0}, upload=63843]: Uploading piece... Jun 22 00:08:16.325315636 (79h44m51.888638643s) [noise=true , address=104.36.82.21:6473 , piece_id=3M34BSPUAFRF5NXPWIXKH25KX67RCHVXQCS5MFS5S7X7RYF676RQ, seg_pos={0 0}, upload=63843]: Uploading piece... Jun 22 00:08:16.394058454 (79h44m51.957381445s) [upload=63843]: Next returned nil segment Jun 22 00:08:16.394072985 (79h44m51.957395986s) [upload=63843]: Waiting for error group managing segments... Jun 22 00:08:16.397574195 (79h44m51.960897187s) [noise=true , address=45.91.116.5:5308 , piece_id=QZA34FLZEJNVE2TUHVF5ZMTG2WP7RTNNOJF7AZ4YLYCA22XJPCZA, seg_pos={0 0}, upload=63843]: Done uploading piece. err: <nil> Jun 22 00:08:16.459349430 (79h44m52.022672432s) [noise=true , address=104.36.82.54:4173 , piece_id=O2FBSZLX7FIDIDJWGWV2RMIZ7DZ5WYKS7JLZIOK64D5TRKSP3U7Q, seg_pos={0 0}, upload=63843]: Done uploading piece. err: ecclient: upload failed (node:1w2v4iyYtJtcdz7shYhG8CjCDQ1dWjeJVwY5Ubi7Ex5Y38e2i1, address:104.36.82.54:4173): protocol: storage node overloaded, request limit: 2000 Jun 22 00:08:16.518220521 (79h44m52.081543528s) [noise=true , address=77.90.31.12:4605 , piece_id=AITH3LEQCYAWVX4JXU7LDBV4LRIA3Y2LI3FNTOISOMRQ4LWNZ2VQ, seg_pos={0 0}, upload=63843]: Done uploading piece. err: <nil> Jun 22 00:08:17.014258958 (79h44m52.577581952s) [noise=true , address=64.49.13.137:4500 , piece_id=FMD3ECZINI2YWY32VYSH7GKCITBALAEDUMYCNTWTAJQVY55Y2MLQ, seg_pos={0 0}, upload=63843]: Done uploading piece. err: ecclient: upload failed (node:15TwVQ6CHDbtqS4XW86n9BuyjdqpaJo4ZKPgPnmhSm6EPDrnd9, address:64.49.13.137:4500): protocol: storage node overloaded, request limit: 2000 Jun 22 00:08:17.153129516 (79h44m52.716452522s) [noise=true , address=104.36.82.16:6328 , piece_id=4TEPB6TBOWSY5KR52EVGRYVGHAKBEDTUHMFF3KJPQKZXYRGDRBJQ, seg_pos={0 0}, upload=63843]: Done uploading piece. err: ecclient: upload failed (node:1S8PSdkLyaiURWkPGhcABSh3481hoswzukT1NKvpTkSp75vKDJ, address:104.36.82.16:6328): protocol: storage node overloaded, request limit: 2000 Jun 22 00:08:17.275208950 (79h44m52.83853195s) [noise=true , address=79.127.205.235:5664 , piece_id=XB53MDJKRRNXK6SAMHX2RHAEO6SAHTKI7732ERADDN7JCWDZKNVA, seg_pos={0 0}, upload=63843]: Done uploading piece. err: ecclient: upload failed (node:12p8Gg6yYNUbFiEkHLHDFdnPC2w4AHgguBYv2ce34AYWXQzhHER, address:79.127.205.235:5664): protocol: storage node overloaded, request limit: 2000 Jun 22 00:08:17.605904252 (79h44m53.169227253s) [noise=true , address=77.90.31.21:4887 , piece_id=3JB4YN5JQ2KHD3LXGYAH2VQDBYAM63SYBNIVNUK2BAVW6TG6L6EA, seg_pos={0 0}, upload=63843]: Done uploading piece. err: ecclient: upload failed (node:12hE2JH2wY7TepnbCVpSxkTcVifqui3QK42tLh1neU2BxZ2BRFJ, address:77.90.31.21:4887): protocol: storage node overloaded, request limit: 2000 Jun 22 00:08:17.865919302 (79h44m53.429242294s) [noise=true , address=77.90.31.14:4693 , piece_id=QAVXJYMYM3XOYRGUPHMK257ZNL7CUMR6UBGSJM647C73ECJDCVIA, seg_pos={0 0}, upload=63843]: Done uploading piece. err: <nil> Jun 22 00:08:19.517357026 (79h44m55.080680046s) [noise=true , address=104.36.82.10:6108 , piece_id=QTWYQJQ22NWG7I6WRKMC6AL7YYPHVEBN2LIUVIFZHC5WMDPKPANQ, seg_pos={0 0}, upload=63843]: Done uploading piece. err: <nil> Jun 22 00:08:20.580254458 (79h44m56.143577448s) [noise=true , address=104.36.82.7:5938 , piece_id=YRRM6JLCSPEV2UZI5CICTMVPW2MTIK7P5K5F5S53AEV2DEMJFA3Q, seg_pos={0 0}, upload=63843]: Done uploading piece. err: <nil> Jun 22 00:08:20.752577874 (79h44m56.315900867s) [noise=true , address=104.36.82.20:6466 , piece_id=P7UZBSRQEY3ORJAXMYSNYBPTYNGDBSSKZL74Z2GSJNMON3VZBOZQ, seg_pos={0 0}, upload=63843]: Done uploading piece. err: <nil> Jun 22 00:08:24.515800663 (79h45m0.079123665s) [noise=true , address=79.127.219.38:4028 , piece_id=QZDOXSRJI6MXMIUNJLVRKZBSCRMDI54KPFDSR4D4JF6S2RZ4SV7A, seg_pos={0 0}, upload=63843]: Done uploading piece. err: <nil> Jun 22 00:08:25.948556776 (79h45m1.511879777s) [noise=true , address=104.36.82.21:6473 , piece_id=3M34BSPUAFRF5NXPWIXKH25KX67RCHVXQCS5MFS5S7X7RYF676RQ, seg_pos={0 0}, upload=63843]: Done uploading piece. err: <nil> Jun 22 00:08:32.636880825 (79h45m8.200203818s) [noise=true , address=104.36.82.56:4261 , piece_id=2RGJ5T5RFUR67IIUA5CUVXFL4RVBO7GVLDLAVUZCRHWAITSBOEWA, seg_pos={0 0}, upload=63843]: Done uploading piece. err: <nil> Jun 22 00:08:35.525047894 (79h45m11.088370884s) [noise=true , address=104.36.82.19:6406 , piece_id=ERRJIBJ6LCOK3QRXCJSASUGTFUL76SS3HZJI2VQ64HWTSUMGVHZA, seg_pos={0 0}, upload=63843]: Done uploading piece. err: <nil> Jun 22 00:08:40.407828721 (79h45m15.971151709s) [noise=true , address=104.36.82.14:6234 , piece_id=WUFOSCTTAKQZLII6GTESOCKX4MJZM6CA45XBQUUKGS6S7JQASLUQ, seg_pos={0 0}, upload=63843]: Done uploading piece. err: <nil> Jun 22 00:08:52.755279221 (79h45m28.318602228s) [noise=true , address=104.36.82.52:4103 , piece_id=MWVPOZMAAFPZD6HDK4SGJOKW5IMNYG75R3YA7LOLMNJVQOLP3SBQ, seg_pos={0 0}, upload=63843]: Done uploading piece. err: <nil> Jun 22 00:09:16.049083116 (79h45m51.612406117s) [noise=true , address=104.36.82.55:4219 , piece_id=BMAN2MOGHY3DLY5O3GYQUSAUQ7A77YSTD7GEHUAUSSPE5EE7LJQQ, seg_pos={0 0}, upload=63843]: Done uploading piece. err: <nil> Jun 22 00:09:16.067949159 (79h45m51.631272164s) [noise=true , address=45.91.116.7:5012 , piece_id=7TSTZEQJHXFCT4UIUEOF6Q3JML3YJRJRJWHO3ZYPGVY66BCSDZ7Q, seg_pos={0 0}, upload=63843]: Uploading piece... Jun 22 00:09:16.067962327 (79h45m51.63128533s) [noise=true , address=77.90.31.17:4803 , piece_id=DXJWERKQZP445OUZQF6W33T5JXREOIMFYAMWDMDDMJS736PXWAXQ, seg_pos={0 0}, upload=63843]: Uploading piece... Jun 22 00:09:16.068067279 (79h45m51.631390289s) [noise=true , address=104.36.82.20:6468 , piece_id=SQUT67FBQGV423UW4LOAECRWUJWCJ7AQ4ED3ACD3MZUTBRXUSYMA, seg_pos={0 0}, upload=63843]: Uploading piece... Jun 22 00:09:16.068445650 (79h45m51.631768659s) [noise=true , address=104.36.82.11:6139 , piece_id=QQUWKI2WTCYU44QDNA4OOBWSPVTCMMMLEXEPOASV35A4UZVPW3TQ, seg_pos={0 0}, upload=63843]: Uploading piece... Jun 22 00:09:16.068751557 (79h45m51.632074575s) [noise=true , address=79.127.205.235:5664 , piece_id=NXG2ACQ4GJGVX7PQYTR5MCRWRPK6Y5FXNUN7YVETYC3U4ROXMOFA, seg_pos={0 0}, upload=63843]: Uploading piece... Jun 22 00:09:16.254420059 (79h45m51.817743058s) [noise=true , address=45.91.116.7:5012 , piece_id=7TSTZEQJHXFCT4UIUEOF6Q3JML3YJRJRJWHO3ZYPGVY66BCSDZ7Q, seg_pos={0 0}, upload=63843]: Done uploading piece. err: ecclient: upload failed (node:1smg5DiC8TRGnnmoQrRpvScvehv8aqtA2DaKtD1JtAhrhnm96t, address:45.91.116.7:5012): protocol: expected piece hash; context deadline exceeded; EOF Jun 22 00:09:16.254721861 (79h45m51.818044861s) [noise=true , address=104.36.82.11:6139 , piece_id=QQUWKI2WTCYU44QDNA4OOBWSPVTCMMMLEXEPOASV35A4UZVPW3TQ, seg_pos={0 0}, upload=63843]: Done uploading piece. err: ecclient: upload failed (node:1EiZjV2QMYue3KQDsNRhhdGpCQjybDv54wTuXsJTNtyeV8hHeh, address:104.36.82.11:6139): protocol: write tcp 159.223.143.39:45694->104.36.82.11:6139: use of closed network connection; write tcp 159.223.143.39:45694->104.36.82.11:6139: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:45694->104.36.82.11:6139: use of closed network connection Jun 22 00:09:16.254797295 (79h45m51.818120295s) [noise=true , address=104.36.82.20:6468 , piece_id=SQUT67FBQGV423UW4LOAECRWUJWCJ7AQ4ED3ACD3MZUTBRXUSYMA, seg_pos={0 0}, upload=63843]: Done uploading piece. err: ecclient: upload failed (node:123Cc2RmkYzkkbder6auanNvu1sTSEM6QD5aagnNC28DviG41Gi, address:104.36.82.20:6468): protocol: read tcp 159.223.143.39:35576->104.36.82.20:6468: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 22 00:09:16.254808564 (79h45m51.818131557s) [noise=true , address=77.90.31.17:4803 , piece_id=DXJWERKQZP445OUZQF6W33T5JXREOIMFYAMWDMDDMJS736PXWAXQ, seg_pos={0 0}, upload=63843]: Done uploading piece. err: ecclient: upload failed (node:12onacKUY32faZTFuyEjvMCZA4nVKiRKmTKh1WRFMfif2Kw1Q7Y, address:77.90.31.17:4803): protocol: read tcp 159.223.143.39:42574->77.90.31.17:4803: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 22 00:09:16.254859521 (79h45m51.81818252s) [noise=true , address=79.127.205.235:5664 , piece_id=NXG2ACQ4GJGVX7PQYTR5MCRWRPK6Y5FXNUN7YVETYC3U4ROXMOFA, seg_pos={0 0}, upload=63843]: Done uploading piece. err: ecclient: upload failed (node:12p8Gg6yYNUbFiEkHLHDFdnPC2w4AHgguBYv2ce34AYWXQzhHER, address:79.127.205.235:5664): protocol: write tcp 159.223.143.39:54186->79.127.205.235:5664: use of closed network connection; write tcp 159.223.143.39:54186->79.127.205.235:5664: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:54186->79.127.205.235:5664: use of closed network connection Jun 22 00:09:16.254895863 (79h45m51.818218866s) [seg_pos={0 0}, upload=63843]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jun 22 00:09:16.276852311 (79h45m51.840175305s) [upload=63843]: Done uploading segments... Jun 22 00:09:16.276903113 (79h45m51.840226102s) [upload=63843]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 16.605160362s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [.....................................X.X....................] duration: 28.852358695s FAIL (2025-06-22 00:24:00 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== g_pos={0 0}, upload=63843]: Uploading piece... Jun 22 00:08:16.324082782 (79h44m51.88740579s) [noise=true , address=104.36.82.52:4103 , piece_id=MWVPOZMAAFPZD6HDK4SGJOKW5IMNYG75R3YA7LOLMNJVQOLP3SBQ, seg_pos={0 0}, upload=63843]: Uploading piece... Jun 22 00:08:16.324387615 (79h44m51.887710618s) [noise=true , address=45.91.116.5:5308 , piece_id=QZA34FLZEJNVE2TUHVF5ZMTG2WP7RTNNOJF7AZ4YLYCA22XJPCZA, seg_pos={0 0}, upload=63843]: Uploading piece... Jun 22 00:08:16.324693763 (79h44m51.888016771s) [noise=true , address=77.90.31.14:4693 , piece_id=QAVXJYMYM3XOYRGUPHMK257ZNL7CUMR6UBGSJM647C73ECJDCVIA, seg_pos={0 0}, upload=63843]: Uploading piece... Jun 22 00:08:16.324987498 (79h44m51.888310498s) [noise=true , address=104.36.82.7:5938 , piece_id=YRRM6JLCSPEV2UZI5CICTMVPW2MTIK7P5K5F5S53AEV2DEMJFA3Q, seg_pos={0 0}, upload=63843]: Uploading piece... Jun 22 00:08:16.325315636 (79h44m51.888638643s) [noise=true , address=104.36.82.21:6473 , piece_id=3M34BSPUAFRF5NXPWIXKH25KX67RCHVXQCS5MFS5S7X7RYF676RQ, seg_pos={0 0}, upload=63843]: Uploading piece... Jun 22 00:08:16.394058454 (79h44m51.957381445s) [upload=63843]: Next returned nil segment Jun 22 00:08:16.394072985 (79h44m51.957395986s) [upload=63843]: Waiting for error group managing segments... Jun 22 00:08:16.397574195 (79h44m51.960897187s) [noise=true , address=45.91.116.5:5308 , piece_id=QZA34FLZEJNVE2TUHVF5ZMTG2WP7RTNNOJF7AZ4YLYCA22XJPCZA, seg_pos={0 0}, upload=63843]: Done uploading piece. err: <nil> Jun 22 00:08:16.459349430 (79h44m52.022672432s) [noise=true , address=104.36.82.54:4173 , piece_id=O2FBSZLX7FIDIDJWGWV2RMIZ7DZ5WYKS7JLZIOK64D5TRKSP3U7Q, seg_pos={0 0}, upload=63843]: Done uploading piece. err: ecclient: upload failed (node:1w2v4iyYtJtcdz7shYhG8CjCDQ1dWjeJVwY5Ubi7Ex5Y38e2i1, address:104.36.82.54:4173): protocol: storage node overloaded, request limit: 2000 Jun 22 00:08:16.518220521 (79h44m52.081543528s) [noise=true , address=77.90.31.12:4605 , piece_id=AITH3LEQCYAWVX4JXU7LDBV4LRIA3Y2LI3FNTOISOMRQ4LWNZ2VQ, seg_pos={0 0}, upload=63843]: Done uploading piece. err: <nil> Jun 22 00:08:17.014258958 (79h44m52.577581952s) [noise=true , address=64.49.13.137:4500 , piece_id=FMD3ECZINI2YWY32VYSH7GKCITBALAEDUMYCNTWTAJQVY55Y2MLQ, seg_pos={0 0}, upload=63843]: Done uploading piece. err: ecclient: upload failed (node:15TwVQ6CHDbtqS4XW86n9BuyjdqpaJo4ZKPgPnmhSm6EPDrnd9, address:64.49.13.137:4500): protocol: storage node overloaded, request limit: 2000 Jun 22 00:08:17.153129516 (79h44m52.716452522s) [noise=true , address=104.36.82.16:6328 , piece_id=4TEPB6TBOWSY5KR52EVGRYVGHAKBEDTUHMFF3KJPQKZXYRGDRBJQ, seg_pos={0 0}, upload=63843]: Done uploading piece. err: ecclient: upload failed (node:1S8PSdkLyaiURWkPGhcABSh3481hoswzukT1NKvpTkSp75vKDJ, address:104.36.82.16:6328): protocol: storage node overloaded, request limit: 2000 Jun 22 00:08:17.275208950 (79h44m52.83853195s) [noise=true , address=79.127.205.235:5664 , piece_id=XB53MDJKRRNXK6SAMHX2RHAEO6SAHTKI7732ERADDN7JCWDZKNVA, seg_pos={0 0}, upload=63843]: Done uploading piece. err: ecclient: upload failed (node:12p8Gg6yYNUbFiEkHLHDFdnPC2w4AHgguBYv2ce34AYWXQzhHER, address:79.127.205.235:5664): protocol: storage node overloaded, request limit: 2000 Jun 22 00:08:17.605904252 (79h44m53.169227253s) [noise=true , address=77.90.31.21:4887 , piece_id=3JB4YN5JQ2KHD3LXGYAH2VQDBYAM63SYBNIVNUK2BAVW6TG6L6EA, seg_pos={0 0}, upload=63843]: Done uploading piece. err: ecclient: upload failed (node:12hE2JH2wY7TepnbCVpSxkTcVifqui3QK42tLh1neU2BxZ2BRFJ, address:77.90.31.21:4887): protocol: storage node overloaded, request limit: 2000 Jun 22 00:08:17.865919302 (79h44m53.429242294s) [noise=true , address=77.90.31.14:4693 , piece_id=QAVXJYMYM3XOYRGUPHMK257ZNL7CUMR6UBGSJM647C73ECJDCVIA, seg_pos={0 0}, upload=63843]: Done uploading piece. err: <nil> Jun 22 00:08:19.517357026 (79h44m55.080680046s) [noise=true , address=104.36.82.10:6108 , piece_id=QTWYQJQ22NWG7I6WRKMC6AL7YYPHVEBN2LIUVIFZHC5WMDPKPANQ, seg_pos={0 0}, upload=63843]: Done uploading piece. err: <nil> Jun 22 00:08:20.580254458 (79h44m56.143577448s) [noise=true , address=104.36.82.7:5938 , piece_id=YRRM6JLCSPEV2UZI5CICTMVPW2MTIK7P5K5F5S53AEV2DEMJFA3Q, seg_pos={0 0}, upload=63843]: Done uploading piece. err: <nil> Jun 22 00:08:20.752577874 (79h44m56.315900867s) [noise=true , address=104.36.82.20:6466 , piece_id=P7UZBSRQEY3ORJAXMYSNYBPTYNGDBSSKZL74Z2GSJNMON3VZBOZQ, seg_pos={0 0}, upload=63843]: Done uploading piece. err: <nil> Jun 22 00:08:24.515800663 (79h45m0.079123665s) [noise=true , address=79.127.219.38:4028 , piece_id=QZDOXSRJI6MXMIUNJLVRKZBSCRMDI54KPFDSR4D4JF6S2RZ4SV7A, seg_pos={0 0}, upload=63843]: Done uploading piece. err: <nil> Jun 22 00:08:25.948556776 (79h45m1.511879777s) [noise=true , address=104.36.82.21:6473 , piece_id=3M34BSPUAFRF5NXPWIXKH25KX67RCHVXQCS5MFS5S7X7RYF676RQ, seg_pos={0 0}, upload=63843]: Done uploading piece. err: <nil> Jun 22 00:08:32.636880825 (79h45m8.200203818s) [noise=true , address=104.36.82.56:4261 , piece_id=2RGJ5T5RFUR67IIUA5CUVXFL4RVBO7GVLDLAVUZCRHWAITSBOEWA, seg_pos={0 0}, upload=63843]: Done uploading piece. err: <nil> Jun 22 00:08:35.525047894 (79h45m11.088370884s) [noise=true , address=104.36.82.19:6406 , piece_id=ERRJIBJ6LCOK3QRXCJSASUGTFUL76SS3HZJI2VQ64HWTSUMGVHZA, seg_pos={0 0}, upload=63843]: Done uploading piece. err: <nil> Jun 22 00:08:40.407828721 (79h45m15.971151709s) [noise=true , address=104.36.82.14:6234 , piece_id=WUFOSCTTAKQZLII6GTESOCKX4MJZM6CA45XBQUUKGS6S7JQASLUQ, seg_pos={0 0}, upload=63843]: Done uploading piece. err: <nil> Jun 22 00:08:52.755279221 (79h45m28.318602228s) [noise=true , address=104.36.82.52:4103 , piece_id=MWVPOZMAAFPZD6HDK4SGJOKW5IMNYG75R3YA7LOLMNJVQOLP3SBQ, seg_pos={0 0}, upload=63843]: Done uploading piece. err: <nil> Jun 22 00:09:16.049083116 (79h45m51.612406117s) [noise=true , address=104.36.82.55:4219 , piece_id=BMAN2MOGHY3DLY5O3GYQUSAUQ7A77YSTD7GEHUAUSSPE5EE7LJQQ, seg_pos={0 0}, upload=63843]: Done uploading piece. err: <nil> Jun 22 00:09:16.067949159 (79h45m51.631272164s) [noise=true , address=45.91.116.7:5012 , piece_id=7TSTZEQJHXFCT4UIUEOF6Q3JML3YJRJRJWHO3ZYPGVY66BCSDZ7Q, seg_pos={0 0}, upload=63843]: Uploading piece... Jun 22 00:09:16.067962327 (79h45m51.63128533s) [noise=true , address=77.90.31.17:4803 , piece_id=DXJWERKQZP445OUZQF6W33T5JXREOIMFYAMWDMDDMJS736PXWAXQ, seg_pos={0 0}, upload=63843]: Uploading piece... Jun 22 00:09:16.068067279 (79h45m51.631390289s) [noise=true , address=104.36.82.20:6468 , piece_id=SQUT67FBQGV423UW4LOAECRWUJWCJ7AQ4ED3ACD3MZUTBRXUSYMA, seg_pos={0 0}, upload=63843]: Uploading piece... Jun 22 00:09:16.068445650 (79h45m51.631768659s) [noise=true , address=104.36.82.11:6139 , piece_id=QQUWKI2WTCYU44QDNA4OOBWSPVTCMMMLEXEPOASV35A4UZVPW3TQ, seg_pos={0 0}, upload=63843]: Uploading piece... Jun 22 00:09:16.068751557 (79h45m51.632074575s) [noise=true , address=79.127.205.235:5664 , piece_id=NXG2ACQ4GJGVX7PQYTR5MCRWRPK6Y5FXNUN7YVETYC3U4ROXMOFA, seg_pos={0 0}, upload=63843]: Uploading piece... Jun 22 00:09:16.254420059 (79h45m51.817743058s) [noise=true , address=45.91.116.7:5012 , piece_id=7TSTZEQJHXFCT4UIUEOF6Q3JML3YJRJRJWHO3ZYPGVY66BCSDZ7Q, seg_pos={0 0}, upload=63843]: Done uploading piece. err: ecclient: upload failed (node:1smg5DiC8TRGnnmoQrRpvScvehv8aqtA2DaKtD1JtAhrhnm96t, address:45.91.116.7:5012): protocol: expected piece hash; context deadline exceeded; EOF Jun 22 00:09:16.254721861 (79h45m51.818044861s) [noise=true , address=104.36.82.11:6139 , piece_id=QQUWKI2WTCYU44QDNA4OOBWSPVTCMMMLEXEPOASV35A4UZVPW3TQ, seg_pos={0 0}, upload=63843]: Done uploading piece. err: ecclient: upload failed (node:1EiZjV2QMYue3KQDsNRhhdGpCQjybDv54wTuXsJTNtyeV8hHeh, address:104.36.82.11:6139): protocol: write tcp 159.223.143.39:45694->104.36.82.11:6139: use of closed network connection; write tcp 159.223.143.39:45694->104.36.82.11:6139: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:45694->104.36.82.11:6139: use of closed network connection Jun 22 00:09:16.254797295 (79h45m51.818120295s) [noise=true , address=104.36.82.20:6468 , piece_id=SQUT67FBQGV423UW4LOAECRWUJWCJ7AQ4ED3ACD3MZUTBRXUSYMA, seg_pos={0 0}, upload=63843]: Done uploading piece. err: ecclient: upload failed (node:123Cc2RmkYzkkbder6auanNvu1sTSEM6QD5aagnNC28DviG41Gi, address:104.36.82.20:6468): protocol: read tcp 159.223.143.39:35576->104.36.82.20:6468: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 22 00:09:16.254808564 (79h45m51.818131557s) [noise=true , address=77.90.31.17:4803 , piece_id=DXJWERKQZP445OUZQF6W33T5JXREOIMFYAMWDMDDMJS736PXWAXQ, seg_pos={0 0}, upload=63843]: Done uploading piece. err: ecclient: upload failed (node:12onacKUY32faZTFuyEjvMCZA4nVKiRKmTKh1WRFMfif2Kw1Q7Y, address:77.90.31.17:4803): protocol: read tcp 159.223.143.39:42574->77.90.31.17:4803: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 22 00:09:16.254859521 (79h45m51.81818252s) [noise=true , address=79.127.205.235:5664 , piece_id=NXG2ACQ4GJGVX7PQYTR5MCRWRPK6Y5FXNUN7YVETYC3U4ROXMOFA, seg_pos={0 0}, upload=63843]: Done uploading piece. err: ecclient: upload failed (node:12p8Gg6yYNUbFiEkHLHDFdnPC2w4AHgguBYv2ce34AYWXQzhHER, address:79.127.205.235:5664): protocol: write tcp 159.223.143.39:54186->79.127.205.235:5664: use of closed network connection; write tcp 159.223.143.39:54186->79.127.205.235:5664: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:54186->79.127.205.235:5664: use of closed network connection Jun 22 00:09:16.254895863 (79h45m51.818218866s) [seg_pos={0 0}, upload=63843]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jun 22 00:09:16.276852311 (79h45m51.840175305s) [upload=63843]: Done uploading segments... Jun 22 00:09:16.276903113 (79h45m51.840226102s) [upload=63843]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 28.852438134s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [..........X.........................X.........X.............] duration: 10.488787772s FAIL (2025-06-22 00:23:52 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 10.488816236s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [......................................X.X...................] duration: 11.569443448s FAIL (2025-06-22 00:23:05 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== g_pos={0 0}, upload=63843]: Uploading piece... Jun 22 00:08:16.324082782 (79h44m51.88740579s) [noise=true , address=104.36.82.52:4103 , piece_id=MWVPOZMAAFPZD6HDK4SGJOKW5IMNYG75R3YA7LOLMNJVQOLP3SBQ, seg_pos={0 0}, upload=63843]: Uploading piece... Jun 22 00:08:16.324387615 (79h44m51.887710618s) [noise=true , address=45.91.116.5:5308 , piece_id=QZA34FLZEJNVE2TUHVF5ZMTG2WP7RTNNOJF7AZ4YLYCA22XJPCZA, seg_pos={0 0}, upload=63843]: Uploading piece... Jun 22 00:08:16.324693763 (79h44m51.888016771s) [noise=true , address=77.90.31.14:4693 , piece_id=QAVXJYMYM3XOYRGUPHMK257ZNL7CUMR6UBGSJM647C73ECJDCVIA, seg_pos={0 0}, upload=63843]: Uploading piece... Jun 22 00:08:16.324987498 (79h44m51.888310498s) [noise=true , address=104.36.82.7:5938 , piece_id=YRRM6JLCSPEV2UZI5CICTMVPW2MTIK7P5K5F5S53AEV2DEMJFA3Q, seg_pos={0 0}, upload=63843]: Uploading piece... Jun 22 00:08:16.325315636 (79h44m51.888638643s) [noise=true , address=104.36.82.21:6473 , piece_id=3M34BSPUAFRF5NXPWIXKH25KX67RCHVXQCS5MFS5S7X7RYF676RQ, seg_pos={0 0}, upload=63843]: Uploading piece... Jun 22 00:08:16.394058454 (79h44m51.957381445s) [upload=63843]: Next returned nil segment Jun 22 00:08:16.394072985 (79h44m51.957395986s) [upload=63843]: Waiting for error group managing segments... Jun 22 00:08:16.397574195 (79h44m51.960897187s) [noise=true , address=45.91.116.5:5308 , piece_id=QZA34FLZEJNVE2TUHVF5ZMTG2WP7RTNNOJF7AZ4YLYCA22XJPCZA, seg_pos={0 0}, upload=63843]: Done uploading piece. err: <nil> Jun 22 00:08:16.459349430 (79h44m52.022672432s) [noise=true , address=104.36.82.54:4173 , piece_id=O2FBSZLX7FIDIDJWGWV2RMIZ7DZ5WYKS7JLZIOK64D5TRKSP3U7Q, seg_pos={0 0}, upload=63843]: Done uploading piece. err: ecclient: upload failed (node:1w2v4iyYtJtcdz7shYhG8CjCDQ1dWjeJVwY5Ubi7Ex5Y38e2i1, address:104.36.82.54:4173): protocol: storage node overloaded, request limit: 2000 Jun 22 00:08:16.518220521 (79h44m52.081543528s) [noise=true , address=77.90.31.12:4605 , piece_id=AITH3LEQCYAWVX4JXU7LDBV4LRIA3Y2LI3FNTOISOMRQ4LWNZ2VQ, seg_pos={0 0}, upload=63843]: Done uploading piece. err: <nil> Jun 22 00:08:17.014258958 (79h44m52.577581952s) [noise=true , address=64.49.13.137:4500 , piece_id=FMD3ECZINI2YWY32VYSH7GKCITBALAEDUMYCNTWTAJQVY55Y2MLQ, seg_pos={0 0}, upload=63843]: Done uploading piece. err: ecclient: upload failed (node:15TwVQ6CHDbtqS4XW86n9BuyjdqpaJo4ZKPgPnmhSm6EPDrnd9, address:64.49.13.137:4500): protocol: storage node overloaded, request limit: 2000 Jun 22 00:08:17.153129516 (79h44m52.716452522s) [noise=true , address=104.36.82.16:6328 , piece_id=4TEPB6TBOWSY5KR52EVGRYVGHAKBEDTUHMFF3KJPQKZXYRGDRBJQ, seg_pos={0 0}, upload=63843]: Done uploading piece. err: ecclient: upload failed (node:1S8PSdkLyaiURWkPGhcABSh3481hoswzukT1NKvpTkSp75vKDJ, address:104.36.82.16:6328): protocol: storage node overloaded, request limit: 2000 Jun 22 00:08:17.275208950 (79h44m52.83853195s) [noise=true , address=79.127.205.235:5664 , piece_id=XB53MDJKRRNXK6SAMHX2RHAEO6SAHTKI7732ERADDN7JCWDZKNVA, seg_pos={0 0}, upload=63843]: Done uploading piece. err: ecclient: upload failed (node:12p8Gg6yYNUbFiEkHLHDFdnPC2w4AHgguBYv2ce34AYWXQzhHER, address:79.127.205.235:5664): protocol: storage node overloaded, request limit: 2000 Jun 22 00:08:17.605904252 (79h44m53.169227253s) [noise=true , address=77.90.31.21:4887 , piece_id=3JB4YN5JQ2KHD3LXGYAH2VQDBYAM63SYBNIVNUK2BAVW6TG6L6EA, seg_pos={0 0}, upload=63843]: Done uploading piece. err: ecclient: upload failed (node:12hE2JH2wY7TepnbCVpSxkTcVifqui3QK42tLh1neU2BxZ2BRFJ, address:77.90.31.21:4887): protocol: storage node overloaded, request limit: 2000 Jun 22 00:08:17.865919302 (79h44m53.429242294s) [noise=true , address=77.90.31.14:4693 , piece_id=QAVXJYMYM3XOYRGUPHMK257ZNL7CUMR6UBGSJM647C73ECJDCVIA, seg_pos={0 0}, upload=63843]: Done uploading piece. err: <nil> Jun 22 00:08:19.517357026 (79h44m55.080680046s) [noise=true , address=104.36.82.10:6108 , piece_id=QTWYQJQ22NWG7I6WRKMC6AL7YYPHVEBN2LIUVIFZHC5WMDPKPANQ, seg_pos={0 0}, upload=63843]: Done uploading piece. err: <nil> Jun 22 00:08:20.580254458 (79h44m56.143577448s) [noise=true , address=104.36.82.7:5938 , piece_id=YRRM6JLCSPEV2UZI5CICTMVPW2MTIK7P5K5F5S53AEV2DEMJFA3Q, seg_pos={0 0}, upload=63843]: Done uploading piece. err: <nil> Jun 22 00:08:20.752577874 (79h44m56.315900867s) [noise=true , address=104.36.82.20:6466 , piece_id=P7UZBSRQEY3ORJAXMYSNYBPTYNGDBSSKZL74Z2GSJNMON3VZBOZQ, seg_pos={0 0}, upload=63843]: Done uploading piece. err: <nil> Jun 22 00:08:24.515800663 (79h45m0.079123665s) [noise=true , address=79.127.219.38:4028 , piece_id=QZDOXSRJI6MXMIUNJLVRKZBSCRMDI54KPFDSR4D4JF6S2RZ4SV7A, seg_pos={0 0}, upload=63843]: Done uploading piece. err: <nil> Jun 22 00:08:25.948556776 (79h45m1.511879777s) [noise=true , address=104.36.82.21:6473 , piece_id=3M34BSPUAFRF5NXPWIXKH25KX67RCHVXQCS5MFS5S7X7RYF676RQ, seg_pos={0 0}, upload=63843]: Done uploading piece. err: <nil> Jun 22 00:08:32.636880825 (79h45m8.200203818s) [noise=true , address=104.36.82.56:4261 , piece_id=2RGJ5T5RFUR67IIUA5CUVXFL4RVBO7GVLDLAVUZCRHWAITSBOEWA, seg_pos={0 0}, upload=63843]: Done uploading piece. err: <nil> Jun 22 00:08:35.525047894 (79h45m11.088370884s) [noise=true , address=104.36.82.19:6406 , piece_id=ERRJIBJ6LCOK3QRXCJSASUGTFUL76SS3HZJI2VQ64HWTSUMGVHZA, seg_pos={0 0}, upload=63843]: Done uploading piece. err: <nil> Jun 22 00:08:40.407828721 (79h45m15.971151709s) [noise=true , address=104.36.82.14:6234 , piece_id=WUFOSCTTAKQZLII6GTESOCKX4MJZM6CA45XBQUUKGS6S7JQASLUQ, seg_pos={0 0}, upload=63843]: Done uploading piece. err: <nil> Jun 22 00:08:52.755279221 (79h45m28.318602228s) [noise=true , address=104.36.82.52:4103 , piece_id=MWVPOZMAAFPZD6HDK4SGJOKW5IMNYG75R3YA7LOLMNJVQOLP3SBQ, seg_pos={0 0}, upload=63843]: Done uploading piece. err: <nil> Jun 22 00:09:16.049083116 (79h45m51.612406117s) [noise=true , address=104.36.82.55:4219 , piece_id=BMAN2MOGHY3DLY5O3GYQUSAUQ7A77YSTD7GEHUAUSSPE5EE7LJQQ, seg_pos={0 0}, upload=63843]: Done uploading piece. err: <nil> Jun 22 00:09:16.067949159 (79h45m51.631272164s) [noise=true , address=45.91.116.7:5012 , piece_id=7TSTZEQJHXFCT4UIUEOF6Q3JML3YJRJRJWHO3ZYPGVY66BCSDZ7Q, seg_pos={0 0}, upload=63843]: Uploading piece... Jun 22 00:09:16.067962327 (79h45m51.63128533s) [noise=true , address=77.90.31.17:4803 , piece_id=DXJWERKQZP445OUZQF6W33T5JXREOIMFYAMWDMDDMJS736PXWAXQ, seg_pos={0 0}, upload=63843]: Uploading piece... Jun 22 00:09:16.068067279 (79h45m51.631390289s) [noise=true , address=104.36.82.20:6468 , piece_id=SQUT67FBQGV423UW4LOAECRWUJWCJ7AQ4ED3ACD3MZUTBRXUSYMA, seg_pos={0 0}, upload=63843]: Uploading piece... Jun 22 00:09:16.068445650 (79h45m51.631768659s) [noise=true , address=104.36.82.11:6139 , piece_id=QQUWKI2WTCYU44QDNA4OOBWSPVTCMMMLEXEPOASV35A4UZVPW3TQ, seg_pos={0 0}, upload=63843]: Uploading piece... Jun 22 00:09:16.068751557 (79h45m51.632074575s) [noise=true , address=79.127.205.235:5664 , piece_id=NXG2ACQ4GJGVX7PQYTR5MCRWRPK6Y5FXNUN7YVETYC3U4ROXMOFA, seg_pos={0 0}, upload=63843]: Uploading piece... Jun 22 00:09:16.254420059 (79h45m51.817743058s) [noise=true , address=45.91.116.7:5012 , piece_id=7TSTZEQJHXFCT4UIUEOF6Q3JML3YJRJRJWHO3ZYPGVY66BCSDZ7Q, seg_pos={0 0}, upload=63843]: Done uploading piece. err: ecclient: upload failed (node:1smg5DiC8TRGnnmoQrRpvScvehv8aqtA2DaKtD1JtAhrhnm96t, address:45.91.116.7:5012): protocol: expected piece hash; context deadline exceeded; EOF Jun 22 00:09:16.254721861 (79h45m51.818044861s) [noise=true , address=104.36.82.11:6139 , piece_id=QQUWKI2WTCYU44QDNA4OOBWSPVTCMMMLEXEPOASV35A4UZVPW3TQ, seg_pos={0 0}, upload=63843]: Done uploading piece. err: ecclient: upload failed (node:1EiZjV2QMYue3KQDsNRhhdGpCQjybDv54wTuXsJTNtyeV8hHeh, address:104.36.82.11:6139): protocol: write tcp 159.223.143.39:45694->104.36.82.11:6139: use of closed network connection; write tcp 159.223.143.39:45694->104.36.82.11:6139: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:45694->104.36.82.11:6139: use of closed network connection Jun 22 00:09:16.254797295 (79h45m51.818120295s) [noise=true , address=104.36.82.20:6468 , piece_id=SQUT67FBQGV423UW4LOAECRWUJWCJ7AQ4ED3ACD3MZUTBRXUSYMA, seg_pos={0 0}, upload=63843]: Done uploading piece. err: ecclient: upload failed (node:123Cc2RmkYzkkbder6auanNvu1sTSEM6QD5aagnNC28DviG41Gi, address:104.36.82.20:6468): protocol: read tcp 159.223.143.39:35576->104.36.82.20:6468: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 22 00:09:16.254808564 (79h45m51.818131557s) [noise=true , address=77.90.31.17:4803 , piece_id=DXJWERKQZP445OUZQF6W33T5JXREOIMFYAMWDMDDMJS736PXWAXQ, seg_pos={0 0}, upload=63843]: Done uploading piece. err: ecclient: upload failed (node:12onacKUY32faZTFuyEjvMCZA4nVKiRKmTKh1WRFMfif2Kw1Q7Y, address:77.90.31.17:4803): protocol: read tcp 159.223.143.39:42574->77.90.31.17:4803: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 22 00:09:16.254859521 (79h45m51.81818252s) [noise=true , address=79.127.205.235:5664 , piece_id=NXG2ACQ4GJGVX7PQYTR5MCRWRPK6Y5FXNUN7YVETYC3U4ROXMOFA, seg_pos={0 0}, upload=63843]: Done uploading piece. err: ecclient: upload failed (node:12p8Gg6yYNUbFiEkHLHDFdnPC2w4AHgguBYv2ce34AYWXQzhHER, address:79.127.205.235:5664): protocol: write tcp 159.223.143.39:54186->79.127.205.235:5664: use of closed network connection; write tcp 159.223.143.39:54186->79.127.205.235:5664: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:54186->79.127.205.235:5664: use of closed network connection Jun 22 00:09:16.254895863 (79h45m51.818218866s) [seg_pos={0 0}, upload=63843]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jun 22 00:09:16.276852311 (79h45m51.840175305s) [upload=63843]: Done uploading segments... Jun 22 00:09:16.276903113 (79h45m51.840226102s) [upload=63843]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 11.569463926s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [.......................................X.X..................] duration: 20.673039015s FAIL (2025-06-22 00:22:53 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== g_pos={0 0}, upload=63843]: Uploading piece... Jun 22 00:08:16.324082782 (79h44m51.88740579s) [noise=true , address=104.36.82.52:4103 , piece_id=MWVPOZMAAFPZD6HDK4SGJOKW5IMNYG75R3YA7LOLMNJVQOLP3SBQ, seg_pos={0 0}, upload=63843]: Uploading piece... Jun 22 00:08:16.324387615 (79h44m51.887710618s) [noise=true , address=45.91.116.5:5308 , piece_id=QZA34FLZEJNVE2TUHVF5ZMTG2WP7RTNNOJF7AZ4YLYCA22XJPCZA, seg_pos={0 0}, upload=63843]: Uploading piece... Jun 22 00:08:16.324693763 (79h44m51.888016771s) [noise=true , address=77.90.31.14:4693 , piece_id=QAVXJYMYM3XOYRGUPHMK257ZNL7CUMR6UBGSJM647C73ECJDCVIA, seg_pos={0 0}, upload=63843]: Uploading piece... Jun 22 00:08:16.324987498 (79h44m51.888310498s) [noise=true , address=104.36.82.7:5938 , piece_id=YRRM6JLCSPEV2UZI5CICTMVPW2MTIK7P5K5F5S53AEV2DEMJFA3Q, seg_pos={0 0}, upload=63843]: Uploading piece... Jun 22 00:08:16.325315636 (79h44m51.888638643s) [noise=true , address=104.36.82.21:6473 , piece_id=3M34BSPUAFRF5NXPWIXKH25KX67RCHVXQCS5MFS5S7X7RYF676RQ, seg_pos={0 0}, upload=63843]: Uploading piece... Jun 22 00:08:16.394058454 (79h44m51.957381445s) [upload=63843]: Next returned nil segment Jun 22 00:08:16.394072985 (79h44m51.957395986s) [upload=63843]: Waiting for error group managing segments... Jun 22 00:08:16.397574195 (79h44m51.960897187s) [noise=true , address=45.91.116.5:5308 , piece_id=QZA34FLZEJNVE2TUHVF5ZMTG2WP7RTNNOJF7AZ4YLYCA22XJPCZA, seg_pos={0 0}, upload=63843]: Done uploading piece. err: <nil> Jun 22 00:08:16.459349430 (79h44m52.022672432s) [noise=true , address=104.36.82.54:4173 , piece_id=O2FBSZLX7FIDIDJWGWV2RMIZ7DZ5WYKS7JLZIOK64D5TRKSP3U7Q, seg_pos={0 0}, upload=63843]: Done uploading piece. err: ecclient: upload failed (node:1w2v4iyYtJtcdz7shYhG8CjCDQ1dWjeJVwY5Ubi7Ex5Y38e2i1, address:104.36.82.54:4173): protocol: storage node overloaded, request limit: 2000 Jun 22 00:08:16.518220521 (79h44m52.081543528s) [noise=true , address=77.90.31.12:4605 , piece_id=AITH3LEQCYAWVX4JXU7LDBV4LRIA3Y2LI3FNTOISOMRQ4LWNZ2VQ, seg_pos={0 0}, upload=63843]: Done uploading piece. err: <nil> Jun 22 00:08:17.014258958 (79h44m52.577581952s) [noise=true , address=64.49.13.137:4500 , piece_id=FMD3ECZINI2YWY32VYSH7GKCITBALAEDUMYCNTWTAJQVY55Y2MLQ, seg_pos={0 0}, upload=63843]: Done uploading piece. err: ecclient: upload failed (node:15TwVQ6CHDbtqS4XW86n9BuyjdqpaJo4ZKPgPnmhSm6EPDrnd9, address:64.49.13.137:4500): protocol: storage node overloaded, request limit: 2000 Jun 22 00:08:17.153129516 (79h44m52.716452522s) [noise=true , address=104.36.82.16:6328 , piece_id=4TEPB6TBOWSY5KR52EVGRYVGHAKBEDTUHMFF3KJPQKZXYRGDRBJQ, seg_pos={0 0}, upload=63843]: Done uploading piece. err: ecclient: upload failed (node:1S8PSdkLyaiURWkPGhcABSh3481hoswzukT1NKvpTkSp75vKDJ, address:104.36.82.16:6328): protocol: storage node overloaded, request limit: 2000 Jun 22 00:08:17.275208950 (79h44m52.83853195s) [noise=true , address=79.127.205.235:5664 , piece_id=XB53MDJKRRNXK6SAMHX2RHAEO6SAHTKI7732ERADDN7JCWDZKNVA, seg_pos={0 0}, upload=63843]: Done uploading piece. err: ecclient: upload failed (node:12p8Gg6yYNUbFiEkHLHDFdnPC2w4AHgguBYv2ce34AYWXQzhHER, address:79.127.205.235:5664): protocol: storage node overloaded, request limit: 2000 Jun 22 00:08:17.605904252 (79h44m53.169227253s) [noise=true , address=77.90.31.21:4887 , piece_id=3JB4YN5JQ2KHD3LXGYAH2VQDBYAM63SYBNIVNUK2BAVW6TG6L6EA, seg_pos={0 0}, upload=63843]: Done uploading piece. err: ecclient: upload failed (node:12hE2JH2wY7TepnbCVpSxkTcVifqui3QK42tLh1neU2BxZ2BRFJ, address:77.90.31.21:4887): protocol: storage node overloaded, request limit: 2000 Jun 22 00:08:17.865919302 (79h44m53.429242294s) [noise=true , address=77.90.31.14:4693 , piece_id=QAVXJYMYM3XOYRGUPHMK257ZNL7CUMR6UBGSJM647C73ECJDCVIA, seg_pos={0 0}, upload=63843]: Done uploading piece. err: <nil> Jun 22 00:08:19.517357026 (79h44m55.080680046s) [noise=true , address=104.36.82.10:6108 , piece_id=QTWYQJQ22NWG7I6WRKMC6AL7YYPHVEBN2LIUVIFZHC5WMDPKPANQ, seg_pos={0 0}, upload=63843]: Done uploading piece. err: <nil> Jun 22 00:08:20.580254458 (79h44m56.143577448s) [noise=true , address=104.36.82.7:5938 , piece_id=YRRM6JLCSPEV2UZI5CICTMVPW2MTIK7P5K5F5S53AEV2DEMJFA3Q, seg_pos={0 0}, upload=63843]: Done uploading piece. err: <nil> Jun 22 00:08:20.752577874 (79h44m56.315900867s) [noise=true , address=104.36.82.20:6466 , piece_id=P7UZBSRQEY3ORJAXMYSNYBPTYNGDBSSKZL74Z2GSJNMON3VZBOZQ, seg_pos={0 0}, upload=63843]: Done uploading piece. err: <nil> Jun 22 00:08:24.515800663 (79h45m0.079123665s) [noise=true , address=79.127.219.38:4028 , piece_id=QZDOXSRJI6MXMIUNJLVRKZBSCRMDI54KPFDSR4D4JF6S2RZ4SV7A, seg_pos={0 0}, upload=63843]: Done uploading piece. err: <nil> Jun 22 00:08:25.948556776 (79h45m1.511879777s) [noise=true , address=104.36.82.21:6473 , piece_id=3M34BSPUAFRF5NXPWIXKH25KX67RCHVXQCS5MFS5S7X7RYF676RQ, seg_pos={0 0}, upload=63843]: Done uploading piece. err: <nil> Jun 22 00:08:32.636880825 (79h45m8.200203818s) [noise=true , address=104.36.82.56:4261 , piece_id=2RGJ5T5RFUR67IIUA5CUVXFL4RVBO7GVLDLAVUZCRHWAITSBOEWA, seg_pos={0 0}, upload=63843]: Done uploading piece. err: <nil> Jun 22 00:08:35.525047894 (79h45m11.088370884s) [noise=true , address=104.36.82.19:6406 , piece_id=ERRJIBJ6LCOK3QRXCJSASUGTFUL76SS3HZJI2VQ64HWTSUMGVHZA, seg_pos={0 0}, upload=63843]: Done uploading piece. err: <nil> Jun 22 00:08:40.407828721 (79h45m15.971151709s) [noise=true , address=104.36.82.14:6234 , piece_id=WUFOSCTTAKQZLII6GTESOCKX4MJZM6CA45XBQUUKGS6S7JQASLUQ, seg_pos={0 0}, upload=63843]: Done uploading piece. err: <nil> Jun 22 00:08:52.755279221 (79h45m28.318602228s) [noise=true , address=104.36.82.52:4103 , piece_id=MWVPOZMAAFPZD6HDK4SGJOKW5IMNYG75R3YA7LOLMNJVQOLP3SBQ, seg_pos={0 0}, upload=63843]: Done uploading piece. err: <nil> Jun 22 00:09:16.049083116 (79h45m51.612406117s) [noise=true , address=104.36.82.55:4219 , piece_id=BMAN2MOGHY3DLY5O3GYQUSAUQ7A77YSTD7GEHUAUSSPE5EE7LJQQ, seg_pos={0 0}, upload=63843]: Done uploading piece. err: <nil> Jun 22 00:09:16.067949159 (79h45m51.631272164s) [noise=true , address=45.91.116.7:5012 , piece_id=7TSTZEQJHXFCT4UIUEOF6Q3JML3YJRJRJWHO3ZYPGVY66BCSDZ7Q, seg_pos={0 0}, upload=63843]: Uploading piece... Jun 22 00:09:16.067962327 (79h45m51.63128533s) [noise=true , address=77.90.31.17:4803 , piece_id=DXJWERKQZP445OUZQF6W33T5JXREOIMFYAMWDMDDMJS736PXWAXQ, seg_pos={0 0}, upload=63843]: Uploading piece... Jun 22 00:09:16.068067279 (79h45m51.631390289s) [noise=true , address=104.36.82.20:6468 , piece_id=SQUT67FBQGV423UW4LOAECRWUJWCJ7AQ4ED3ACD3MZUTBRXUSYMA, seg_pos={0 0}, upload=63843]: Uploading piece... Jun 22 00:09:16.068445650 (79h45m51.631768659s) [noise=true , address=104.36.82.11:6139 , piece_id=QQUWKI2WTCYU44QDNA4OOBWSPVTCMMMLEXEPOASV35A4UZVPW3TQ, seg_pos={0 0}, upload=63843]: Uploading piece... Jun 22 00:09:16.068751557 (79h45m51.632074575s) [noise=true , address=79.127.205.235:5664 , piece_id=NXG2ACQ4GJGVX7PQYTR5MCRWRPK6Y5FXNUN7YVETYC3U4ROXMOFA, seg_pos={0 0}, upload=63843]: Uploading piece... Jun 22 00:09:16.254420059 (79h45m51.817743058s) [noise=true , address=45.91.116.7:5012 , piece_id=7TSTZEQJHXFCT4UIUEOF6Q3JML3YJRJRJWHO3ZYPGVY66BCSDZ7Q, seg_pos={0 0}, upload=63843]: Done uploading piece. err: ecclient: upload failed (node:1smg5DiC8TRGnnmoQrRpvScvehv8aqtA2DaKtD1JtAhrhnm96t, address:45.91.116.7:5012): protocol: expected piece hash; context deadline exceeded; EOF Jun 22 00:09:16.254721861 (79h45m51.818044861s) [noise=true , address=104.36.82.11:6139 , piece_id=QQUWKI2WTCYU44QDNA4OOBWSPVTCMMMLEXEPOASV35A4UZVPW3TQ, seg_pos={0 0}, upload=63843]: Done uploading piece. err: ecclient: upload failed (node:1EiZjV2QMYue3KQDsNRhhdGpCQjybDv54wTuXsJTNtyeV8hHeh, address:104.36.82.11:6139): protocol: write tcp 159.223.143.39:45694->104.36.82.11:6139: use of closed network connection; write tcp 159.223.143.39:45694->104.36.82.11:6139: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:45694->104.36.82.11:6139: use of closed network connection Jun 22 00:09:16.254797295 (79h45m51.818120295s) [noise=true , address=104.36.82.20:6468 , piece_id=SQUT67FBQGV423UW4LOAECRWUJWCJ7AQ4ED3ACD3MZUTBRXUSYMA, seg_pos={0 0}, upload=63843]: Done uploading piece. err: ecclient: upload failed (node:123Cc2RmkYzkkbder6auanNvu1sTSEM6QD5aagnNC28DviG41Gi, address:104.36.82.20:6468): protocol: read tcp 159.223.143.39:35576->104.36.82.20:6468: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 22 00:09:16.254808564 (79h45m51.818131557s) [noise=true , address=77.90.31.17:4803 , piece_id=DXJWERKQZP445OUZQF6W33T5JXREOIMFYAMWDMDDMJS736PXWAXQ, seg_pos={0 0}, upload=63843]: Done uploading piece. err: ecclient: upload failed (node:12onacKUY32faZTFuyEjvMCZA4nVKiRKmTKh1WRFMfif2Kw1Q7Y, address:77.90.31.17:4803): protocol: read tcp 159.223.143.39:42574->77.90.31.17:4803: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 22 00:09:16.254859521 (79h45m51.81818252s) [noise=true , address=79.127.205.235:5664 , piece_id=NXG2ACQ4GJGVX7PQYTR5MCRWRPK6Y5FXNUN7YVETYC3U4ROXMOFA, seg_pos={0 0}, upload=63843]: Done uploading piece. err: ecclient: upload failed (node:12p8Gg6yYNUbFiEkHLHDFdnPC2w4AHgguBYv2ce34AYWXQzhHER, address:79.127.205.235:5664): protocol: write tcp 159.223.143.39:54186->79.127.205.235:5664: use of closed network connection; write tcp 159.223.143.39:54186->79.127.205.235:5664: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:54186->79.127.205.235:5664: use of closed network connection Jun 22 00:09:16.254895863 (79h45m51.818218866s) [seg_pos={0 0}, upload=63843]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jun 22 00:09:16.276852311 (79h45m51.840175305s) [upload=63843]: Done uploading segments... Jun 22 00:09:16.276903113 (79h45m51.840226102s) [upload=63843]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 20.673068993s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [...........X.........................X.........X............] duration: 10.886140624s FAIL (2025-06-22 00:22:48 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 10.886165608s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [........................................X.X.................] duration: 16.164957794s FAIL (2025-06-22 00:22:27 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== g_pos={0 0}, upload=63843]: Uploading piece... Jun 22 00:08:16.324082782 (79h44m51.88740579s) [noise=true , address=104.36.82.52:4103 , piece_id=MWVPOZMAAFPZD6HDK4SGJOKW5IMNYG75R3YA7LOLMNJVQOLP3SBQ, seg_pos={0 0}, upload=63843]: Uploading piece... Jun 22 00:08:16.324387615 (79h44m51.887710618s) [noise=true , address=45.91.116.5:5308 , piece_id=QZA34FLZEJNVE2TUHVF5ZMTG2WP7RTNNOJF7AZ4YLYCA22XJPCZA, seg_pos={0 0}, upload=63843]: Uploading piece... Jun 22 00:08:16.324693763 (79h44m51.888016771s) [noise=true , address=77.90.31.14:4693 , piece_id=QAVXJYMYM3XOYRGUPHMK257ZNL7CUMR6UBGSJM647C73ECJDCVIA, seg_pos={0 0}, upload=63843]: Uploading piece... Jun 22 00:08:16.324987498 (79h44m51.888310498s) [noise=true , address=104.36.82.7:5938 , piece_id=YRRM6JLCSPEV2UZI5CICTMVPW2MTIK7P5K5F5S53AEV2DEMJFA3Q, seg_pos={0 0}, upload=63843]: Uploading piece... Jun 22 00:08:16.325315636 (79h44m51.888638643s) [noise=true , address=104.36.82.21:6473 , piece_id=3M34BSPUAFRF5NXPWIXKH25KX67RCHVXQCS5MFS5S7X7RYF676RQ, seg_pos={0 0}, upload=63843]: Uploading piece... Jun 22 00:08:16.394058454 (79h44m51.957381445s) [upload=63843]: Next returned nil segment Jun 22 00:08:16.394072985 (79h44m51.957395986s) [upload=63843]: Waiting for error group managing segments... Jun 22 00:08:16.397574195 (79h44m51.960897187s) [noise=true , address=45.91.116.5:5308 , piece_id=QZA34FLZEJNVE2TUHVF5ZMTG2WP7RTNNOJF7AZ4YLYCA22XJPCZA, seg_pos={0 0}, upload=63843]: Done uploading piece. err: <nil> Jun 22 00:08:16.459349430 (79h44m52.022672432s) [noise=true , address=104.36.82.54:4173 , piece_id=O2FBSZLX7FIDIDJWGWV2RMIZ7DZ5WYKS7JLZIOK64D5TRKSP3U7Q, seg_pos={0 0}, upload=63843]: Done uploading piece. err: ecclient: upload failed (node:1w2v4iyYtJtcdz7shYhG8CjCDQ1dWjeJVwY5Ubi7Ex5Y38e2i1, address:104.36.82.54:4173): protocol: storage node overloaded, request limit: 2000 Jun 22 00:08:16.518220521 (79h44m52.081543528s) [noise=true , address=77.90.31.12:4605 , piece_id=AITH3LEQCYAWVX4JXU7LDBV4LRIA3Y2LI3FNTOISOMRQ4LWNZ2VQ, seg_pos={0 0}, upload=63843]: Done uploading piece. err: <nil> Jun 22 00:08:17.014258958 (79h44m52.577581952s) [noise=true , address=64.49.13.137:4500 , piece_id=FMD3ECZINI2YWY32VYSH7GKCITBALAEDUMYCNTWTAJQVY55Y2MLQ, seg_pos={0 0}, upload=63843]: Done uploading piece. err: ecclient: upload failed (node:15TwVQ6CHDbtqS4XW86n9BuyjdqpaJo4ZKPgPnmhSm6EPDrnd9, address:64.49.13.137:4500): protocol: storage node overloaded, request limit: 2000 Jun 22 00:08:17.153129516 (79h44m52.716452522s) [noise=true , address=104.36.82.16:6328 , piece_id=4TEPB6TBOWSY5KR52EVGRYVGHAKBEDTUHMFF3KJPQKZXYRGDRBJQ, seg_pos={0 0}, upload=63843]: Done uploading piece. err: ecclient: upload failed (node:1S8PSdkLyaiURWkPGhcABSh3481hoswzukT1NKvpTkSp75vKDJ, address:104.36.82.16:6328): protocol: storage node overloaded, request limit: 2000 Jun 22 00:08:17.275208950 (79h44m52.83853195s) [noise=true , address=79.127.205.235:5664 , piece_id=XB53MDJKRRNXK6SAMHX2RHAEO6SAHTKI7732ERADDN7JCWDZKNVA, seg_pos={0 0}, upload=63843]: Done uploading piece. err: ecclient: upload failed (node:12p8Gg6yYNUbFiEkHLHDFdnPC2w4AHgguBYv2ce34AYWXQzhHER, address:79.127.205.235:5664): protocol: storage node overloaded, request limit: 2000 Jun 22 00:08:17.605904252 (79h44m53.169227253s) [noise=true , address=77.90.31.21:4887 , piece_id=3JB4YN5JQ2KHD3LXGYAH2VQDBYAM63SYBNIVNUK2BAVW6TG6L6EA, seg_pos={0 0}, upload=63843]: Done uploading piece. err: ecclient: upload failed (node:12hE2JH2wY7TepnbCVpSxkTcVifqui3QK42tLh1neU2BxZ2BRFJ, address:77.90.31.21:4887): protocol: storage node overloaded, request limit: 2000 Jun 22 00:08:17.865919302 (79h44m53.429242294s) [noise=true , address=77.90.31.14:4693 , piece_id=QAVXJYMYM3XOYRGUPHMK257ZNL7CUMR6UBGSJM647C73ECJDCVIA, seg_pos={0 0}, upload=63843]: Done uploading piece. err: <nil> Jun 22 00:08:19.517357026 (79h44m55.080680046s) [noise=true , address=104.36.82.10:6108 , piece_id=QTWYQJQ22NWG7I6WRKMC6AL7YYPHVEBN2LIUVIFZHC5WMDPKPANQ, seg_pos={0 0}, upload=63843]: Done uploading piece. err: <nil> Jun 22 00:08:20.580254458 (79h44m56.143577448s) [noise=true , address=104.36.82.7:5938 , piece_id=YRRM6JLCSPEV2UZI5CICTMVPW2MTIK7P5K5F5S53AEV2DEMJFA3Q, seg_pos={0 0}, upload=63843]: Done uploading piece. err: <nil> Jun 22 00:08:20.752577874 (79h44m56.315900867s) [noise=true , address=104.36.82.20:6466 , piece_id=P7UZBSRQEY3ORJAXMYSNYBPTYNGDBSSKZL74Z2GSJNMON3VZBOZQ, seg_pos={0 0}, upload=63843]: Done uploading piece. err: <nil> Jun 22 00:08:24.515800663 (79h45m0.079123665s) [noise=true , address=79.127.219.38:4028 , piece_id=QZDOXSRJI6MXMIUNJLVRKZBSCRMDI54KPFDSR4D4JF6S2RZ4SV7A, seg_pos={0 0}, upload=63843]: Done uploading piece. err: <nil> Jun 22 00:08:25.948556776 (79h45m1.511879777s) [noise=true , address=104.36.82.21:6473 , piece_id=3M34BSPUAFRF5NXPWIXKH25KX67RCHVXQCS5MFS5S7X7RYF676RQ, seg_pos={0 0}, upload=63843]: Done uploading piece. err: <nil> Jun 22 00:08:32.636880825 (79h45m8.200203818s) [noise=true , address=104.36.82.56:4261 , piece_id=2RGJ5T5RFUR67IIUA5CUVXFL4RVBO7GVLDLAVUZCRHWAITSBOEWA, seg_pos={0 0}, upload=63843]: Done uploading piece. err: <nil> Jun 22 00:08:35.525047894 (79h45m11.088370884s) [noise=true , address=104.36.82.19:6406 , piece_id=ERRJIBJ6LCOK3QRXCJSASUGTFUL76SS3HZJI2VQ64HWTSUMGVHZA, seg_pos={0 0}, upload=63843]: Done uploading piece. err: <nil> Jun 22 00:08:40.407828721 (79h45m15.971151709s) [noise=true , address=104.36.82.14:6234 , piece_id=WUFOSCTTAKQZLII6GTESOCKX4MJZM6CA45XBQUUKGS6S7JQASLUQ, seg_pos={0 0}, upload=63843]: Done uploading piece. err: <nil> Jun 22 00:08:52.755279221 (79h45m28.318602228s) [noise=true , address=104.36.82.52:4103 , piece_id=MWVPOZMAAFPZD6HDK4SGJOKW5IMNYG75R3YA7LOLMNJVQOLP3SBQ, seg_pos={0 0}, upload=63843]: Done uploading piece. err: <nil> Jun 22 00:09:16.049083116 (79h45m51.612406117s) [noise=true , address=104.36.82.55:4219 , piece_id=BMAN2MOGHY3DLY5O3GYQUSAUQ7A77YSTD7GEHUAUSSPE5EE7LJQQ, seg_pos={0 0}, upload=63843]: Done uploading piece. err: <nil> Jun 22 00:09:16.067949159 (79h45m51.631272164s) [noise=true , address=45.91.116.7:5012 , piece_id=7TSTZEQJHXFCT4UIUEOF6Q3JML3YJRJRJWHO3ZYPGVY66BCSDZ7Q, seg_pos={0 0}, upload=63843]: Uploading piece... Jun 22 00:09:16.067962327 (79h45m51.63128533s) [noise=true , address=77.90.31.17:4803 , piece_id=DXJWERKQZP445OUZQF6W33T5JXREOIMFYAMWDMDDMJS736PXWAXQ, seg_pos={0 0}, upload=63843]: Uploading piece... Jun 22 00:09:16.068067279 (79h45m51.631390289s) [noise=true , address=104.36.82.20:6468 , piece_id=SQUT67FBQGV423UW4LOAECRWUJWCJ7AQ4ED3ACD3MZUTBRXUSYMA, seg_pos={0 0}, upload=63843]: Uploading piece... Jun 22 00:09:16.068445650 (79h45m51.631768659s) [noise=true , address=104.36.82.11:6139 , piece_id=QQUWKI2WTCYU44QDNA4OOBWSPVTCMMMLEXEPOASV35A4UZVPW3TQ, seg_pos={0 0}, upload=63843]: Uploading piece... Jun 22 00:09:16.068751557 (79h45m51.632074575s) [noise=true , address=79.127.205.235:5664 , piece_id=NXG2ACQ4GJGVX7PQYTR5MCRWRPK6Y5FXNUN7YVETYC3U4ROXMOFA, seg_pos={0 0}, upload=63843]: Uploading piece... Jun 22 00:09:16.254420059 (79h45m51.817743058s) [noise=true , address=45.91.116.7:5012 , piece_id=7TSTZEQJHXFCT4UIUEOF6Q3JML3YJRJRJWHO3ZYPGVY66BCSDZ7Q, seg_pos={0 0}, upload=63843]: Done uploading piece. err: ecclient: upload failed (node:1smg5DiC8TRGnnmoQrRpvScvehv8aqtA2DaKtD1JtAhrhnm96t, address:45.91.116.7:5012): protocol: expected piece hash; context deadline exceeded; EOF Jun 22 00:09:16.254721861 (79h45m51.818044861s) [noise=true , address=104.36.82.11:6139 , piece_id=QQUWKI2WTCYU44QDNA4OOBWSPVTCMMMLEXEPOASV35A4UZVPW3TQ, seg_pos={0 0}, upload=63843]: Done uploading piece. err: ecclient: upload failed (node:1EiZjV2QMYue3KQDsNRhhdGpCQjybDv54wTuXsJTNtyeV8hHeh, address:104.36.82.11:6139): protocol: write tcp 159.223.143.39:45694->104.36.82.11:6139: use of closed network connection; write tcp 159.223.143.39:45694->104.36.82.11:6139: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:45694->104.36.82.11:6139: use of closed network connection Jun 22 00:09:16.254797295 (79h45m51.818120295s) [noise=true , address=104.36.82.20:6468 , piece_id=SQUT67FBQGV423UW4LOAECRWUJWCJ7AQ4ED3ACD3MZUTBRXUSYMA, seg_pos={0 0}, upload=63843]: Done uploading piece. err: ecclient: upload failed (node:123Cc2RmkYzkkbder6auanNvu1sTSEM6QD5aagnNC28DviG41Gi, address:104.36.82.20:6468): protocol: read tcp 159.223.143.39:35576->104.36.82.20:6468: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 22 00:09:16.254808564 (79h45m51.818131557s) [noise=true , address=77.90.31.17:4803 , piece_id=DXJWERKQZP445OUZQF6W33T5JXREOIMFYAMWDMDDMJS736PXWAXQ, seg_pos={0 0}, upload=63843]: Done uploading piece. err: ecclient: upload failed (node:12onacKUY32faZTFuyEjvMCZA4nVKiRKmTKh1WRFMfif2Kw1Q7Y, address:77.90.31.17:4803): protocol: read tcp 159.223.143.39:42574->77.90.31.17:4803: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 22 00:09:16.254859521 (79h45m51.81818252s) [noise=true , address=79.127.205.235:5664 , piece_id=NXG2ACQ4GJGVX7PQYTR5MCRWRPK6Y5FXNUN7YVETYC3U4ROXMOFA, seg_pos={0 0}, upload=63843]: Done uploading piece. err: ecclient: upload failed (node:12p8Gg6yYNUbFiEkHLHDFdnPC2w4AHgguBYv2ce34AYWXQzhHER, address:79.127.205.235:5664): protocol: write tcp 159.223.143.39:54186->79.127.205.235:5664: use of closed network connection; write tcp 159.223.143.39:54186->79.127.205.235:5664: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:54186->79.127.205.235:5664: use of closed network connection Jun 22 00:09:16.254895863 (79h45m51.818218866s) [seg_pos={0 0}, upload=63843]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jun 22 00:09:16.276852311 (79h45m51.840175305s) [upload=63843]: Done uploading segments... Jun 22 00:09:16.276903113 (79h45m51.840226102s) [upload=63843]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 16.164982811s core uplink us-select-1 us1-placement-20

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)