Orbiter History

checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [........X...................................................] duration: 12.247205055s FAIL (2025-06-22 01:12:32 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 12.247227606s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.........X..................................................] duration: 12.753490981s FAIL (2025-06-22 01:12:20 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== ing upload Jun 22 00:30:55.303646184 (80h7m30.866969171s) [upload=64142]: Uploading segments... Jun 22 00:30:55.303706406 (80h7m30.867029416s) [upload=64142]: Got next segment. Inline: false Jun 22 00:30:55.303762479 (80h7m30.867085468s) [upload=64142]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 22 00:30:55.303768627 (80h7m30.867091636s) [upload=64142]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 22 00:30:55.349058140 (80h7m30.912381168s) [seg_pos={0 0}, upload=64142]: Begin upload segment... Jun 22 00:30:55.349113768 (80h7m30.912436772s) [seg_pos={0 0}, upload=64142]: Done begin upload segment. Jun 22 00:30:55.349143121 (80h7m30.912466131s) [noise=true , address=45.91.116.5:5310 , piece_id=AXJOKDZRPVNL7HFT4QKJWRAHFM2UCM3GEVQB2KNYD6CXB6TH4D4Q, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.349660917 (80h7m30.912983928s) [noise=true , address=77.90.31.10:4533 , piece_id=SE6YAKKFDL3BV677CDDUC32UNMPXJHTG23GWWYK5B4YIPKER6GAQ, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.350093077 (80h7m30.913416089s) [noise=true , address=79.127.201.215:4326 , piece_id=USEWZDX6P52U6MCALTBNQG3SV7N2JRKNU6O23TLS6JEWAS4D7ARA, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.350480894 (80h7m30.913803916s) [noise=true , address=104.36.82.13:6199 , piece_id=YP62N2TO446YR3QRG5YLXECD7TN6SJ4UNNLZSK5H4CMV6JIF4AKA, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.350669998 (80h7m30.913993011s) [noise=true , address=104.36.82.54:4184 , piece_id=IZICVGZFMAYLIJTNUJ2OFLZVNZ6IPPYDZO4BSTJJPLHVCVOE5EAA, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.351201352 (80h7m30.914524366s) [noise=true , address=104.36.82.55:4224 , piece_id=XF7H2BYSNNNZOS2T77VIN75N76AYUXAQEI6D4LRXACU2KU62V2TA, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.351668071 (80h7m30.914991083s) [noise=true , address=104.36.82.56:4261 , piece_id=EYOYOYLDACGMPGJYX6ILQT3XHMUTQNXZEDC7ANHLZXIKYVDQFFVQ, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.352193126 (80h7m30.915516138s) [noise=true , address=104.36.82.52:4089 , piece_id=X2VG4S3UHVNWA4AQJS7FLKWNVLTOZ3AKWI5ZDH24QCA3ENQW7BXQ, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.352587096 (80h7m30.915910103s) [noise=true , address=79.127.220.98:4402 , piece_id=TE2WQO5LK45AVMPIAIGVSIBOVZLC4FG4ESQ72X6KFYC7IAEN2DRA, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.353420253 (80h7m30.916743265s) [noise=true , address=104.36.82.11:6125 , piece_id=2CX7RJOKNQA6ODZZZXYPSCYAQY5P35TWX7I56CMYAPFLJE3JEO5Q, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.353791565 (80h7m30.917114578s) [noise=true , address=104.36.82.42:6803 , piece_id=R626UFD326S2UBJEZ2WMHVR7EN4SK427VXPDJZOOZ32MX25JWZWQ, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.354171226 (80h7m30.917494233s) [noise=true , address=104.36.82.14:6245 , piece_id=II36G46OA6TINLR244U6D7Q6LGV754UA5M62K3EX7ZOBESMJUEHA, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.354542282 (80h7m30.917865288s) [noise=true , address=79.127.205.235:5660 , piece_id=UGFPVS7VNB7U2YSFPSGYDIRZV5R2DHCN57PJP6VOXO52WIAYHLXA, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.354891655 (80h7m30.918214678s) [noise=true , address=104.36.82.15:6283 , piece_id=N7TH2EMIOPATVHCSNH2WXUGU7KMMEJ5LCHR22SWEHSA5OV4CUCXQ, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.355499503 (80h7m30.918822519s) [noise=true , address=77.90.31.14:4666 , piece_id=H4DTFDAVCN266PB235BJIQTQ32FBXTIB5AZSX3B4R2VIK554AFSQ, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.356575547 (80h7m30.919898539s) [noise=true , address=104.36.82.39:6756 , piece_id=OKKWMDJPZO6D5VRMGOE3GOCYGHM725JZTNOVYAWKJW3WYZPI4V5Q, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.357039590 (80h7m30.920362596s) [noise=true , address=104.36.82.53:4146 , piece_id=3V3DVGUOPJKDWIGQ2TU7M6VFYFVAS6ODY62ERGSU6LI3LSTWQ4YQ, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.360343566 (80h7m30.923666582s) [noise=true , address=104.36.82.9:6033 , piece_id=T5XA23LZELMSTWOLBSHOX4AHKDACF65OYCKTMZYTXDIBSXNA2T4A, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.452444064 (80h7m31.015767055s) [upload=64142]: Next returned nil segment Jun 22 00:30:55.452495148 (80h7m31.015818149s) [upload=64142]: Waiting for error group managing segments... Jun 22 00:30:55.532958808 (80h7m31.096281883s) [noise=true , address=104.36.82.9:6033 , piece_id=T5XA23LZELMSTWOLBSHOX4AHKDACF65OYCKTMZYTXDIBSXNA2T4A, seg_pos={0 0}, upload=64142]: Done uploading piece. err: ecclient: upload failed (node:1SV3LrcQW16wLewDzfyUWFB6sUgnERNJVdhz72CJmyde95M79C, address:104.36.82.9:6033): protocol: storage node overloaded, request limit: 2000 Jun 22 00:30:55.547802184 (80h7m31.111125188s) [noise=true , address=79.127.220.98:4402 , piece_id=TE2WQO5LK45AVMPIAIGVSIBOVZLC4FG4ESQ72X6KFYC7IAEN2DRA, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:30:55.554223621 (80h7m31.117546616s) [noise=true , address=77.90.31.14:4666 , piece_id=H4DTFDAVCN266PB235BJIQTQ32FBXTIB5AZSX3B4R2VIK554AFSQ, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:30:55.762514255 (80h7m31.325837245s) [noise=true , address=104.36.82.14:6245 , piece_id=II36G46OA6TINLR244U6D7Q6LGV754UA5M62K3EX7ZOBESMJUEHA, seg_pos={0 0}, upload=64142]: Done uploading piece. err: ecclient: upload failed (node:1Zu8teNPTXn638grHKwtJqz4UEY7oFYLx1DHvuVP6RZXcatVqX, address:104.36.82.14:6245): protocol: storage node overloaded, request limit: 2000 Jun 22 00:30:55.773669583 (80h7m31.3369926s) [noise=true , address=79.127.205.235:5660 , piece_id=UGFPVS7VNB7U2YSFPSGYDIRZV5R2DHCN57PJP6VOXO52WIAYHLXA, seg_pos={0 0}, upload=64142]: Done uploading piece. err: ecclient: upload failed (node:12E6uCX9KL4Ud7YpxMXepvMXuqTFJCiSXByDjW94Am3U9H2Nf7f, address:79.127.205.235:5660): protocol: storage node overloaded, request limit: 2000 Jun 22 00:30:56.968366655 (80h7m32.53168964s) [noise=true , address=77.90.31.10:4533 , piece_id=SE6YAKKFDL3BV677CDDUC32UNMPXJHTG23GWWYK5B4YIPKER6GAQ, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:30:57.403572874 (80h7m32.966895883s) [noise=true , address=45.91.116.5:5310 , piece_id=AXJOKDZRPVNL7HFT4QKJWRAHFM2UCM3GEVQB2KNYD6CXB6TH4D4Q, seg_pos={0 0}, upload=64142]: Done uploading piece. err: ecclient: upload failed (node:127RMUc62xRq9RU5z3XfkZsq5sehmv3kP8o53tr9BxVcPE3GQCY, address:45.91.116.5:5310): protocol: storage node overloaded, request limit: 2000 Jun 22 00:30:58.210389994 (80h7m33.773712995s) [noise=true , address=104.36.82.13:6199 , piece_id=YP62N2TO446YR3QRG5YLXECD7TN6SJ4UNNLZSK5H4CMV6JIF4AKA, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:30:58.693037961 (80h7m34.256360966s) [noise=true , address=104.36.82.39:6756 , piece_id=OKKWMDJPZO6D5VRMGOE3GOCYGHM725JZTNOVYAWKJW3WYZPI4V5Q, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:30:59.124834814 (80h7m34.688157819s) [noise=true , address=104.36.82.15:6283 , piece_id=N7TH2EMIOPATVHCSNH2WXUGU7KMMEJ5LCHR22SWEHSA5OV4CUCXQ, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:31:00.629657982 (80h7m36.192980977s) [noise=true , address=104.36.82.42:6803 , piece_id=R626UFD326S2UBJEZ2WMHVR7EN4SK427VXPDJZOOZ32MX25JWZWQ, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:31:00.933110546 (80h7m36.496433536s) [noise=true , address=79.127.201.215:4326 , piece_id=USEWZDX6P52U6MCALTBNQG3SV7N2JRKNU6O23TLS6JEWAS4D7ARA, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:31:05.669167243 (80h7m41.232490237s) [noise=true , address=104.36.82.56:4261 , piece_id=EYOYOYLDACGMPGJYX6ILQT3XHMUTQNXZEDC7ANHLZXIKYVDQFFVQ, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:31:08.443582763 (80h7m44.006905763s) [noise=true , address=104.36.82.54:4184 , piece_id=IZICVGZFMAYLIJTNUJ2OFLZVNZ6IPPYDZO4BSTJJPLHVCVOE5EAA, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:31:26.459138014 (80h8m2.022461013s) [noise=true , address=104.36.82.53:4146 , piece_id=3V3DVGUOPJKDWIGQ2TU7M6VFYFVAS6ODY62ERGSU6LI3LSTWQ4YQ, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:31:27.456050336 (80h8m3.019373336s) [noise=true , address=104.36.82.52:4089 , piece_id=X2VG4S3UHVNWA4AQJS7FLKWNVLTOZ3AKWI5ZDH24QCA3ENQW7BXQ, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:31:55.270234251 (80h8m30.833557256s) [noise=true , address=104.36.82.11:6125 , piece_id=2CX7RJOKNQA6ODZZZXYPSCYAQY5P35TWX7I56CMYAPFLJE3JEO5Q, seg_pos={0 0}, upload=64142]: Done uploading piece. err: ecclient: upload failed (node:12vricofwx67FRaxAGLbmgT1TYyKkqfJbxdsyeEPL4R7kg8pKzT, address:104.36.82.11:6125): protocol: expected piece hash; context deadline exceeded; EOF Jun 22 00:31:55.270319418 (80h8m30.833642407s) [noise=true , address=104.36.82.55:4224 , piece_id=XF7H2BYSNNNZOS2T77VIN75N76AYUXAQEI6D4LRXACU2KU62V2TA, seg_pos={0 0}, upload=64142]: Done uploading piece. err: ecclient: upload failed (node:12EjKmUiv4pxkVCB4WoiXim4uFpxrBPyfiSS4efTY3HKSE3jb5y, address:104.36.82.55:4224): protocol: write tcp 159.223.143.39:39152->104.36.82.55:4224: use of closed network connection; write tcp 159.223.143.39:39152->104.36.82.55:4224: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:39152->104.36.82.55:4224: use of closed network connection Jun 22 00:31:55.270353811 (80h8m30.833676799s) [seg_pos={0 0}, upload=64142]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Jun 22 00:31:55.301099877 (80h8m30.864422884s) [upload=64142]: Done uploading segments... Jun 22 00:31:55.301121567 (80h8m30.864444566s) [upload=64142]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 12.753518204s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [..........X.................................................] duration: 35.865374409s FAIL (2025-06-22 01:12:08 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== ing upload Jun 22 00:30:55.303646184 (80h7m30.866969171s) [upload=64142]: Uploading segments... Jun 22 00:30:55.303706406 (80h7m30.867029416s) [upload=64142]: Got next segment. Inline: false Jun 22 00:30:55.303762479 (80h7m30.867085468s) [upload=64142]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 22 00:30:55.303768627 (80h7m30.867091636s) [upload=64142]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 22 00:30:55.349058140 (80h7m30.912381168s) [seg_pos={0 0}, upload=64142]: Begin upload segment... Jun 22 00:30:55.349113768 (80h7m30.912436772s) [seg_pos={0 0}, upload=64142]: Done begin upload segment. Jun 22 00:30:55.349143121 (80h7m30.912466131s) [noise=true , address=45.91.116.5:5310 , piece_id=AXJOKDZRPVNL7HFT4QKJWRAHFM2UCM3GEVQB2KNYD6CXB6TH4D4Q, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.349660917 (80h7m30.912983928s) [noise=true , address=77.90.31.10:4533 , piece_id=SE6YAKKFDL3BV677CDDUC32UNMPXJHTG23GWWYK5B4YIPKER6GAQ, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.350093077 (80h7m30.913416089s) [noise=true , address=79.127.201.215:4326 , piece_id=USEWZDX6P52U6MCALTBNQG3SV7N2JRKNU6O23TLS6JEWAS4D7ARA, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.350480894 (80h7m30.913803916s) [noise=true , address=104.36.82.13:6199 , piece_id=YP62N2TO446YR3QRG5YLXECD7TN6SJ4UNNLZSK5H4CMV6JIF4AKA, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.350669998 (80h7m30.913993011s) [noise=true , address=104.36.82.54:4184 , piece_id=IZICVGZFMAYLIJTNUJ2OFLZVNZ6IPPYDZO4BSTJJPLHVCVOE5EAA, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.351201352 (80h7m30.914524366s) [noise=true , address=104.36.82.55:4224 , piece_id=XF7H2BYSNNNZOS2T77VIN75N76AYUXAQEI6D4LRXACU2KU62V2TA, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.351668071 (80h7m30.914991083s) [noise=true , address=104.36.82.56:4261 , piece_id=EYOYOYLDACGMPGJYX6ILQT3XHMUTQNXZEDC7ANHLZXIKYVDQFFVQ, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.352193126 (80h7m30.915516138s) [noise=true , address=104.36.82.52:4089 , piece_id=X2VG4S3UHVNWA4AQJS7FLKWNVLTOZ3AKWI5ZDH24QCA3ENQW7BXQ, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.352587096 (80h7m30.915910103s) [noise=true , address=79.127.220.98:4402 , piece_id=TE2WQO5LK45AVMPIAIGVSIBOVZLC4FG4ESQ72X6KFYC7IAEN2DRA, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.353420253 (80h7m30.916743265s) [noise=true , address=104.36.82.11:6125 , piece_id=2CX7RJOKNQA6ODZZZXYPSCYAQY5P35TWX7I56CMYAPFLJE3JEO5Q, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.353791565 (80h7m30.917114578s) [noise=true , address=104.36.82.42:6803 , piece_id=R626UFD326S2UBJEZ2WMHVR7EN4SK427VXPDJZOOZ32MX25JWZWQ, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.354171226 (80h7m30.917494233s) [noise=true , address=104.36.82.14:6245 , piece_id=II36G46OA6TINLR244U6D7Q6LGV754UA5M62K3EX7ZOBESMJUEHA, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.354542282 (80h7m30.917865288s) [noise=true , address=79.127.205.235:5660 , piece_id=UGFPVS7VNB7U2YSFPSGYDIRZV5R2DHCN57PJP6VOXO52WIAYHLXA, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.354891655 (80h7m30.918214678s) [noise=true , address=104.36.82.15:6283 , piece_id=N7TH2EMIOPATVHCSNH2WXUGU7KMMEJ5LCHR22SWEHSA5OV4CUCXQ, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.355499503 (80h7m30.918822519s) [noise=true , address=77.90.31.14:4666 , piece_id=H4DTFDAVCN266PB235BJIQTQ32FBXTIB5AZSX3B4R2VIK554AFSQ, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.356575547 (80h7m30.919898539s) [noise=true , address=104.36.82.39:6756 , piece_id=OKKWMDJPZO6D5VRMGOE3GOCYGHM725JZTNOVYAWKJW3WYZPI4V5Q, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.357039590 (80h7m30.920362596s) [noise=true , address=104.36.82.53:4146 , piece_id=3V3DVGUOPJKDWIGQ2TU7M6VFYFVAS6ODY62ERGSU6LI3LSTWQ4YQ, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.360343566 (80h7m30.923666582s) [noise=true , address=104.36.82.9:6033 , piece_id=T5XA23LZELMSTWOLBSHOX4AHKDACF65OYCKTMZYTXDIBSXNA2T4A, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.452444064 (80h7m31.015767055s) [upload=64142]: Next returned nil segment Jun 22 00:30:55.452495148 (80h7m31.015818149s) [upload=64142]: Waiting for error group managing segments... Jun 22 00:30:55.532958808 (80h7m31.096281883s) [noise=true , address=104.36.82.9:6033 , piece_id=T5XA23LZELMSTWOLBSHOX4AHKDACF65OYCKTMZYTXDIBSXNA2T4A, seg_pos={0 0}, upload=64142]: Done uploading piece. err: ecclient: upload failed (node:1SV3LrcQW16wLewDzfyUWFB6sUgnERNJVdhz72CJmyde95M79C, address:104.36.82.9:6033): protocol: storage node overloaded, request limit: 2000 Jun 22 00:30:55.547802184 (80h7m31.111125188s) [noise=true , address=79.127.220.98:4402 , piece_id=TE2WQO5LK45AVMPIAIGVSIBOVZLC4FG4ESQ72X6KFYC7IAEN2DRA, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:30:55.554223621 (80h7m31.117546616s) [noise=true , address=77.90.31.14:4666 , piece_id=H4DTFDAVCN266PB235BJIQTQ32FBXTIB5AZSX3B4R2VIK554AFSQ, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:30:55.762514255 (80h7m31.325837245s) [noise=true , address=104.36.82.14:6245 , piece_id=II36G46OA6TINLR244U6D7Q6LGV754UA5M62K3EX7ZOBESMJUEHA, seg_pos={0 0}, upload=64142]: Done uploading piece. err: ecclient: upload failed (node:1Zu8teNPTXn638grHKwtJqz4UEY7oFYLx1DHvuVP6RZXcatVqX, address:104.36.82.14:6245): protocol: storage node overloaded, request limit: 2000 Jun 22 00:30:55.773669583 (80h7m31.3369926s) [noise=true , address=79.127.205.235:5660 , piece_id=UGFPVS7VNB7U2YSFPSGYDIRZV5R2DHCN57PJP6VOXO52WIAYHLXA, seg_pos={0 0}, upload=64142]: Done uploading piece. err: ecclient: upload failed (node:12E6uCX9KL4Ud7YpxMXepvMXuqTFJCiSXByDjW94Am3U9H2Nf7f, address:79.127.205.235:5660): protocol: storage node overloaded, request limit: 2000 Jun 22 00:30:56.968366655 (80h7m32.53168964s) [noise=true , address=77.90.31.10:4533 , piece_id=SE6YAKKFDL3BV677CDDUC32UNMPXJHTG23GWWYK5B4YIPKER6GAQ, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:30:57.403572874 (80h7m32.966895883s) [noise=true , address=45.91.116.5:5310 , piece_id=AXJOKDZRPVNL7HFT4QKJWRAHFM2UCM3GEVQB2KNYD6CXB6TH4D4Q, seg_pos={0 0}, upload=64142]: Done uploading piece. err: ecclient: upload failed (node:127RMUc62xRq9RU5z3XfkZsq5sehmv3kP8o53tr9BxVcPE3GQCY, address:45.91.116.5:5310): protocol: storage node overloaded, request limit: 2000 Jun 22 00:30:58.210389994 (80h7m33.773712995s) [noise=true , address=104.36.82.13:6199 , piece_id=YP62N2TO446YR3QRG5YLXECD7TN6SJ4UNNLZSK5H4CMV6JIF4AKA, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:30:58.693037961 (80h7m34.256360966s) [noise=true , address=104.36.82.39:6756 , piece_id=OKKWMDJPZO6D5VRMGOE3GOCYGHM725JZTNOVYAWKJW3WYZPI4V5Q, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:30:59.124834814 (80h7m34.688157819s) [noise=true , address=104.36.82.15:6283 , piece_id=N7TH2EMIOPATVHCSNH2WXUGU7KMMEJ5LCHR22SWEHSA5OV4CUCXQ, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:31:00.629657982 (80h7m36.192980977s) [noise=true , address=104.36.82.42:6803 , piece_id=R626UFD326S2UBJEZ2WMHVR7EN4SK427VXPDJZOOZ32MX25JWZWQ, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:31:00.933110546 (80h7m36.496433536s) [noise=true , address=79.127.201.215:4326 , piece_id=USEWZDX6P52U6MCALTBNQG3SV7N2JRKNU6O23TLS6JEWAS4D7ARA, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:31:05.669167243 (80h7m41.232490237s) [noise=true , address=104.36.82.56:4261 , piece_id=EYOYOYLDACGMPGJYX6ILQT3XHMUTQNXZEDC7ANHLZXIKYVDQFFVQ, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:31:08.443582763 (80h7m44.006905763s) [noise=true , address=104.36.82.54:4184 , piece_id=IZICVGZFMAYLIJTNUJ2OFLZVNZ6IPPYDZO4BSTJJPLHVCVOE5EAA, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:31:26.459138014 (80h8m2.022461013s) [noise=true , address=104.36.82.53:4146 , piece_id=3V3DVGUOPJKDWIGQ2TU7M6VFYFVAS6ODY62ERGSU6LI3LSTWQ4YQ, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:31:27.456050336 (80h8m3.019373336s) [noise=true , address=104.36.82.52:4089 , piece_id=X2VG4S3UHVNWA4AQJS7FLKWNVLTOZ3AKWI5ZDH24QCA3ENQW7BXQ, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:31:55.270234251 (80h8m30.833557256s) [noise=true , address=104.36.82.11:6125 , piece_id=2CX7RJOKNQA6ODZZZXYPSCYAQY5P35TWX7I56CMYAPFLJE3JEO5Q, seg_pos={0 0}, upload=64142]: Done uploading piece. err: ecclient: upload failed (node:12vricofwx67FRaxAGLbmgT1TYyKkqfJbxdsyeEPL4R7kg8pKzT, address:104.36.82.11:6125): protocol: expected piece hash; context deadline exceeded; EOF Jun 22 00:31:55.270319418 (80h8m30.833642407s) [noise=true , address=104.36.82.55:4224 , piece_id=XF7H2BYSNNNZOS2T77VIN75N76AYUXAQEI6D4LRXACU2KU62V2TA, seg_pos={0 0}, upload=64142]: Done uploading piece. err: ecclient: upload failed (node:12EjKmUiv4pxkVCB4WoiXim4uFpxrBPyfiSS4efTY3HKSE3jb5y, address:104.36.82.55:4224): protocol: write tcp 159.223.143.39:39152->104.36.82.55:4224: use of closed network connection; write tcp 159.223.143.39:39152->104.36.82.55:4224: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:39152->104.36.82.55:4224: use of closed network connection Jun 22 00:31:55.270353811 (80h8m30.833676799s) [seg_pos={0 0}, upload=64142]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Jun 22 00:31:55.301099877 (80h8m30.864422884s) [upload=64142]: Done uploading segments... Jun 22 00:31:55.301121567 (80h8m30.864444566s) [upload=64142]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 35.865428762s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.........X..................................................] duration: 10.49735483s FAIL (2025-06-22 01:11:50 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 10.497382858s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [..........X.................................................] duration: 13.45774447s FAIL (2025-06-22 01:11:23 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 13.457771099s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [...........X................................................] duration: 16.346396607s FAIL (2025-06-22 01:10:59 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 16.346426888s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [...........X................................................] duration: 19.890426522s FAIL (2025-06-22 01:10:43 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== ing upload Jun 22 00:30:55.303646184 (80h7m30.866969171s) [upload=64142]: Uploading segments... Jun 22 00:30:55.303706406 (80h7m30.867029416s) [upload=64142]: Got next segment. Inline: false Jun 22 00:30:55.303762479 (80h7m30.867085468s) [upload=64142]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 22 00:30:55.303768627 (80h7m30.867091636s) [upload=64142]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 22 00:30:55.349058140 (80h7m30.912381168s) [seg_pos={0 0}, upload=64142]: Begin upload segment... Jun 22 00:30:55.349113768 (80h7m30.912436772s) [seg_pos={0 0}, upload=64142]: Done begin upload segment. Jun 22 00:30:55.349143121 (80h7m30.912466131s) [noise=true , address=45.91.116.5:5310 , piece_id=AXJOKDZRPVNL7HFT4QKJWRAHFM2UCM3GEVQB2KNYD6CXB6TH4D4Q, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.349660917 (80h7m30.912983928s) [noise=true , address=77.90.31.10:4533 , piece_id=SE6YAKKFDL3BV677CDDUC32UNMPXJHTG23GWWYK5B4YIPKER6GAQ, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.350093077 (80h7m30.913416089s) [noise=true , address=79.127.201.215:4326 , piece_id=USEWZDX6P52U6MCALTBNQG3SV7N2JRKNU6O23TLS6JEWAS4D7ARA, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.350480894 (80h7m30.913803916s) [noise=true , address=104.36.82.13:6199 , piece_id=YP62N2TO446YR3QRG5YLXECD7TN6SJ4UNNLZSK5H4CMV6JIF4AKA, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.350669998 (80h7m30.913993011s) [noise=true , address=104.36.82.54:4184 , piece_id=IZICVGZFMAYLIJTNUJ2OFLZVNZ6IPPYDZO4BSTJJPLHVCVOE5EAA, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.351201352 (80h7m30.914524366s) [noise=true , address=104.36.82.55:4224 , piece_id=XF7H2BYSNNNZOS2T77VIN75N76AYUXAQEI6D4LRXACU2KU62V2TA, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.351668071 (80h7m30.914991083s) [noise=true , address=104.36.82.56:4261 , piece_id=EYOYOYLDACGMPGJYX6ILQT3XHMUTQNXZEDC7ANHLZXIKYVDQFFVQ, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.352193126 (80h7m30.915516138s) [noise=true , address=104.36.82.52:4089 , piece_id=X2VG4S3UHVNWA4AQJS7FLKWNVLTOZ3AKWI5ZDH24QCA3ENQW7BXQ, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.352587096 (80h7m30.915910103s) [noise=true , address=79.127.220.98:4402 , piece_id=TE2WQO5LK45AVMPIAIGVSIBOVZLC4FG4ESQ72X6KFYC7IAEN2DRA, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.353420253 (80h7m30.916743265s) [noise=true , address=104.36.82.11:6125 , piece_id=2CX7RJOKNQA6ODZZZXYPSCYAQY5P35TWX7I56CMYAPFLJE3JEO5Q, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.353791565 (80h7m30.917114578s) [noise=true , address=104.36.82.42:6803 , piece_id=R626UFD326S2UBJEZ2WMHVR7EN4SK427VXPDJZOOZ32MX25JWZWQ, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.354171226 (80h7m30.917494233s) [noise=true , address=104.36.82.14:6245 , piece_id=II36G46OA6TINLR244U6D7Q6LGV754UA5M62K3EX7ZOBESMJUEHA, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.354542282 (80h7m30.917865288s) [noise=true , address=79.127.205.235:5660 , piece_id=UGFPVS7VNB7U2YSFPSGYDIRZV5R2DHCN57PJP6VOXO52WIAYHLXA, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.354891655 (80h7m30.918214678s) [noise=true , address=104.36.82.15:6283 , piece_id=N7TH2EMIOPATVHCSNH2WXUGU7KMMEJ5LCHR22SWEHSA5OV4CUCXQ, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.355499503 (80h7m30.918822519s) [noise=true , address=77.90.31.14:4666 , piece_id=H4DTFDAVCN266PB235BJIQTQ32FBXTIB5AZSX3B4R2VIK554AFSQ, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.356575547 (80h7m30.919898539s) [noise=true , address=104.36.82.39:6756 , piece_id=OKKWMDJPZO6D5VRMGOE3GOCYGHM725JZTNOVYAWKJW3WYZPI4V5Q, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.357039590 (80h7m30.920362596s) [noise=true , address=104.36.82.53:4146 , piece_id=3V3DVGUOPJKDWIGQ2TU7M6VFYFVAS6ODY62ERGSU6LI3LSTWQ4YQ, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.360343566 (80h7m30.923666582s) [noise=true , address=104.36.82.9:6033 , piece_id=T5XA23LZELMSTWOLBSHOX4AHKDACF65OYCKTMZYTXDIBSXNA2T4A, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.452444064 (80h7m31.015767055s) [upload=64142]: Next returned nil segment Jun 22 00:30:55.452495148 (80h7m31.015818149s) [upload=64142]: Waiting for error group managing segments... Jun 22 00:30:55.532958808 (80h7m31.096281883s) [noise=true , address=104.36.82.9:6033 , piece_id=T5XA23LZELMSTWOLBSHOX4AHKDACF65OYCKTMZYTXDIBSXNA2T4A, seg_pos={0 0}, upload=64142]: Done uploading piece. err: ecclient: upload failed (node:1SV3LrcQW16wLewDzfyUWFB6sUgnERNJVdhz72CJmyde95M79C, address:104.36.82.9:6033): protocol: storage node overloaded, request limit: 2000 Jun 22 00:30:55.547802184 (80h7m31.111125188s) [noise=true , address=79.127.220.98:4402 , piece_id=TE2WQO5LK45AVMPIAIGVSIBOVZLC4FG4ESQ72X6KFYC7IAEN2DRA, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:30:55.554223621 (80h7m31.117546616s) [noise=true , address=77.90.31.14:4666 , piece_id=H4DTFDAVCN266PB235BJIQTQ32FBXTIB5AZSX3B4R2VIK554AFSQ, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:30:55.762514255 (80h7m31.325837245s) [noise=true , address=104.36.82.14:6245 , piece_id=II36G46OA6TINLR244U6D7Q6LGV754UA5M62K3EX7ZOBESMJUEHA, seg_pos={0 0}, upload=64142]: Done uploading piece. err: ecclient: upload failed (node:1Zu8teNPTXn638grHKwtJqz4UEY7oFYLx1DHvuVP6RZXcatVqX, address:104.36.82.14:6245): protocol: storage node overloaded, request limit: 2000 Jun 22 00:30:55.773669583 (80h7m31.3369926s) [noise=true , address=79.127.205.235:5660 , piece_id=UGFPVS7VNB7U2YSFPSGYDIRZV5R2DHCN57PJP6VOXO52WIAYHLXA, seg_pos={0 0}, upload=64142]: Done uploading piece. err: ecclient: upload failed (node:12E6uCX9KL4Ud7YpxMXepvMXuqTFJCiSXByDjW94Am3U9H2Nf7f, address:79.127.205.235:5660): protocol: storage node overloaded, request limit: 2000 Jun 22 00:30:56.968366655 (80h7m32.53168964s) [noise=true , address=77.90.31.10:4533 , piece_id=SE6YAKKFDL3BV677CDDUC32UNMPXJHTG23GWWYK5B4YIPKER6GAQ, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:30:57.403572874 (80h7m32.966895883s) [noise=true , address=45.91.116.5:5310 , piece_id=AXJOKDZRPVNL7HFT4QKJWRAHFM2UCM3GEVQB2KNYD6CXB6TH4D4Q, seg_pos={0 0}, upload=64142]: Done uploading piece. err: ecclient: upload failed (node:127RMUc62xRq9RU5z3XfkZsq5sehmv3kP8o53tr9BxVcPE3GQCY, address:45.91.116.5:5310): protocol: storage node overloaded, request limit: 2000 Jun 22 00:30:58.210389994 (80h7m33.773712995s) [noise=true , address=104.36.82.13:6199 , piece_id=YP62N2TO446YR3QRG5YLXECD7TN6SJ4UNNLZSK5H4CMV6JIF4AKA, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:30:58.693037961 (80h7m34.256360966s) [noise=true , address=104.36.82.39:6756 , piece_id=OKKWMDJPZO6D5VRMGOE3GOCYGHM725JZTNOVYAWKJW3WYZPI4V5Q, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:30:59.124834814 (80h7m34.688157819s) [noise=true , address=104.36.82.15:6283 , piece_id=N7TH2EMIOPATVHCSNH2WXUGU7KMMEJ5LCHR22SWEHSA5OV4CUCXQ, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:31:00.629657982 (80h7m36.192980977s) [noise=true , address=104.36.82.42:6803 , piece_id=R626UFD326S2UBJEZ2WMHVR7EN4SK427VXPDJZOOZ32MX25JWZWQ, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:31:00.933110546 (80h7m36.496433536s) [noise=true , address=79.127.201.215:4326 , piece_id=USEWZDX6P52U6MCALTBNQG3SV7N2JRKNU6O23TLS6JEWAS4D7ARA, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:31:05.669167243 (80h7m41.232490237s) [noise=true , address=104.36.82.56:4261 , piece_id=EYOYOYLDACGMPGJYX6ILQT3XHMUTQNXZEDC7ANHLZXIKYVDQFFVQ, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:31:08.443582763 (80h7m44.006905763s) [noise=true , address=104.36.82.54:4184 , piece_id=IZICVGZFMAYLIJTNUJ2OFLZVNZ6IPPYDZO4BSTJJPLHVCVOE5EAA, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:31:26.459138014 (80h8m2.022461013s) [noise=true , address=104.36.82.53:4146 , piece_id=3V3DVGUOPJKDWIGQ2TU7M6VFYFVAS6ODY62ERGSU6LI3LSTWQ4YQ, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:31:27.456050336 (80h8m3.019373336s) [noise=true , address=104.36.82.52:4089 , piece_id=X2VG4S3UHVNWA4AQJS7FLKWNVLTOZ3AKWI5ZDH24QCA3ENQW7BXQ, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:31:55.270234251 (80h8m30.833557256s) [noise=true , address=104.36.82.11:6125 , piece_id=2CX7RJOKNQA6ODZZZXYPSCYAQY5P35TWX7I56CMYAPFLJE3JEO5Q, seg_pos={0 0}, upload=64142]: Done uploading piece. err: ecclient: upload failed (node:12vricofwx67FRaxAGLbmgT1TYyKkqfJbxdsyeEPL4R7kg8pKzT, address:104.36.82.11:6125): protocol: expected piece hash; context deadline exceeded; EOF Jun 22 00:31:55.270319418 (80h8m30.833642407s) [noise=true , address=104.36.82.55:4224 , piece_id=XF7H2BYSNNNZOS2T77VIN75N76AYUXAQEI6D4LRXACU2KU62V2TA, seg_pos={0 0}, upload=64142]: Done uploading piece. err: ecclient: upload failed (node:12EjKmUiv4pxkVCB4WoiXim4uFpxrBPyfiSS4efTY3HKSE3jb5y, address:104.36.82.55:4224): protocol: write tcp 159.223.143.39:39152->104.36.82.55:4224: use of closed network connection; write tcp 159.223.143.39:39152->104.36.82.55:4224: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:39152->104.36.82.55:4224: use of closed network connection Jun 22 00:31:55.270353811 (80h8m30.833676799s) [seg_pos={0 0}, upload=64142]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Jun 22 00:31:55.301099877 (80h8m30.864422884s) [upload=64142]: Done uploading segments... Jun 22 00:31:55.301121567 (80h8m30.864444566s) [upload=64142]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 19.890458703s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [............X...............................................] duration: 21.223906079s FAIL (2025-06-22 01:10:08 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 21.223936997s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [............X...............................................] duration: 37.73994125s FAIL (2025-06-22 01:09:53 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== ing upload Jun 22 00:30:55.303646184 (80h7m30.866969171s) [upload=64142]: Uploading segments... Jun 22 00:30:55.303706406 (80h7m30.867029416s) [upload=64142]: Got next segment. Inline: false Jun 22 00:30:55.303762479 (80h7m30.867085468s) [upload=64142]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 22 00:30:55.303768627 (80h7m30.867091636s) [upload=64142]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 22 00:30:55.349058140 (80h7m30.912381168s) [seg_pos={0 0}, upload=64142]: Begin upload segment... Jun 22 00:30:55.349113768 (80h7m30.912436772s) [seg_pos={0 0}, upload=64142]: Done begin upload segment. Jun 22 00:30:55.349143121 (80h7m30.912466131s) [noise=true , address=45.91.116.5:5310 , piece_id=AXJOKDZRPVNL7HFT4QKJWRAHFM2UCM3GEVQB2KNYD6CXB6TH4D4Q, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.349660917 (80h7m30.912983928s) [noise=true , address=77.90.31.10:4533 , piece_id=SE6YAKKFDL3BV677CDDUC32UNMPXJHTG23GWWYK5B4YIPKER6GAQ, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.350093077 (80h7m30.913416089s) [noise=true , address=79.127.201.215:4326 , piece_id=USEWZDX6P52U6MCALTBNQG3SV7N2JRKNU6O23TLS6JEWAS4D7ARA, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.350480894 (80h7m30.913803916s) [noise=true , address=104.36.82.13:6199 , piece_id=YP62N2TO446YR3QRG5YLXECD7TN6SJ4UNNLZSK5H4CMV6JIF4AKA, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.350669998 (80h7m30.913993011s) [noise=true , address=104.36.82.54:4184 , piece_id=IZICVGZFMAYLIJTNUJ2OFLZVNZ6IPPYDZO4BSTJJPLHVCVOE5EAA, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.351201352 (80h7m30.914524366s) [noise=true , address=104.36.82.55:4224 , piece_id=XF7H2BYSNNNZOS2T77VIN75N76AYUXAQEI6D4LRXACU2KU62V2TA, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.351668071 (80h7m30.914991083s) [noise=true , address=104.36.82.56:4261 , piece_id=EYOYOYLDACGMPGJYX6ILQT3XHMUTQNXZEDC7ANHLZXIKYVDQFFVQ, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.352193126 (80h7m30.915516138s) [noise=true , address=104.36.82.52:4089 , piece_id=X2VG4S3UHVNWA4AQJS7FLKWNVLTOZ3AKWI5ZDH24QCA3ENQW7BXQ, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.352587096 (80h7m30.915910103s) [noise=true , address=79.127.220.98:4402 , piece_id=TE2WQO5LK45AVMPIAIGVSIBOVZLC4FG4ESQ72X6KFYC7IAEN2DRA, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.353420253 (80h7m30.916743265s) [noise=true , address=104.36.82.11:6125 , piece_id=2CX7RJOKNQA6ODZZZXYPSCYAQY5P35TWX7I56CMYAPFLJE3JEO5Q, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.353791565 (80h7m30.917114578s) [noise=true , address=104.36.82.42:6803 , piece_id=R626UFD326S2UBJEZ2WMHVR7EN4SK427VXPDJZOOZ32MX25JWZWQ, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.354171226 (80h7m30.917494233s) [noise=true , address=104.36.82.14:6245 , piece_id=II36G46OA6TINLR244U6D7Q6LGV754UA5M62K3EX7ZOBESMJUEHA, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.354542282 (80h7m30.917865288s) [noise=true , address=79.127.205.235:5660 , piece_id=UGFPVS7VNB7U2YSFPSGYDIRZV5R2DHCN57PJP6VOXO52WIAYHLXA, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.354891655 (80h7m30.918214678s) [noise=true , address=104.36.82.15:6283 , piece_id=N7TH2EMIOPATVHCSNH2WXUGU7KMMEJ5LCHR22SWEHSA5OV4CUCXQ, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.355499503 (80h7m30.918822519s) [noise=true , address=77.90.31.14:4666 , piece_id=H4DTFDAVCN266PB235BJIQTQ32FBXTIB5AZSX3B4R2VIK554AFSQ, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.356575547 (80h7m30.919898539s) [noise=true , address=104.36.82.39:6756 , piece_id=OKKWMDJPZO6D5VRMGOE3GOCYGHM725JZTNOVYAWKJW3WYZPI4V5Q, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.357039590 (80h7m30.920362596s) [noise=true , address=104.36.82.53:4146 , piece_id=3V3DVGUOPJKDWIGQ2TU7M6VFYFVAS6ODY62ERGSU6LI3LSTWQ4YQ, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.360343566 (80h7m30.923666582s) [noise=true , address=104.36.82.9:6033 , piece_id=T5XA23LZELMSTWOLBSHOX4AHKDACF65OYCKTMZYTXDIBSXNA2T4A, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.452444064 (80h7m31.015767055s) [upload=64142]: Next returned nil segment Jun 22 00:30:55.452495148 (80h7m31.015818149s) [upload=64142]: Waiting for error group managing segments... Jun 22 00:30:55.532958808 (80h7m31.096281883s) [noise=true , address=104.36.82.9:6033 , piece_id=T5XA23LZELMSTWOLBSHOX4AHKDACF65OYCKTMZYTXDIBSXNA2T4A, seg_pos={0 0}, upload=64142]: Done uploading piece. err: ecclient: upload failed (node:1SV3LrcQW16wLewDzfyUWFB6sUgnERNJVdhz72CJmyde95M79C, address:104.36.82.9:6033): protocol: storage node overloaded, request limit: 2000 Jun 22 00:30:55.547802184 (80h7m31.111125188s) [noise=true , address=79.127.220.98:4402 , piece_id=TE2WQO5LK45AVMPIAIGVSIBOVZLC4FG4ESQ72X6KFYC7IAEN2DRA, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:30:55.554223621 (80h7m31.117546616s) [noise=true , address=77.90.31.14:4666 , piece_id=H4DTFDAVCN266PB235BJIQTQ32FBXTIB5AZSX3B4R2VIK554AFSQ, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:30:55.762514255 (80h7m31.325837245s) [noise=true , address=104.36.82.14:6245 , piece_id=II36G46OA6TINLR244U6D7Q6LGV754UA5M62K3EX7ZOBESMJUEHA, seg_pos={0 0}, upload=64142]: Done uploading piece. err: ecclient: upload failed (node:1Zu8teNPTXn638grHKwtJqz4UEY7oFYLx1DHvuVP6RZXcatVqX, address:104.36.82.14:6245): protocol: storage node overloaded, request limit: 2000 Jun 22 00:30:55.773669583 (80h7m31.3369926s) [noise=true , address=79.127.205.235:5660 , piece_id=UGFPVS7VNB7U2YSFPSGYDIRZV5R2DHCN57PJP6VOXO52WIAYHLXA, seg_pos={0 0}, upload=64142]: Done uploading piece. err: ecclient: upload failed (node:12E6uCX9KL4Ud7YpxMXepvMXuqTFJCiSXByDjW94Am3U9H2Nf7f, address:79.127.205.235:5660): protocol: storage node overloaded, request limit: 2000 Jun 22 00:30:56.968366655 (80h7m32.53168964s) [noise=true , address=77.90.31.10:4533 , piece_id=SE6YAKKFDL3BV677CDDUC32UNMPXJHTG23GWWYK5B4YIPKER6GAQ, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:30:57.403572874 (80h7m32.966895883s) [noise=true , address=45.91.116.5:5310 , piece_id=AXJOKDZRPVNL7HFT4QKJWRAHFM2UCM3GEVQB2KNYD6CXB6TH4D4Q, seg_pos={0 0}, upload=64142]: Done uploading piece. err: ecclient: upload failed (node:127RMUc62xRq9RU5z3XfkZsq5sehmv3kP8o53tr9BxVcPE3GQCY, address:45.91.116.5:5310): protocol: storage node overloaded, request limit: 2000 Jun 22 00:30:58.210389994 (80h7m33.773712995s) [noise=true , address=104.36.82.13:6199 , piece_id=YP62N2TO446YR3QRG5YLXECD7TN6SJ4UNNLZSK5H4CMV6JIF4AKA, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:30:58.693037961 (80h7m34.256360966s) [noise=true , address=104.36.82.39:6756 , piece_id=OKKWMDJPZO6D5VRMGOE3GOCYGHM725JZTNOVYAWKJW3WYZPI4V5Q, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:30:59.124834814 (80h7m34.688157819s) [noise=true , address=104.36.82.15:6283 , piece_id=N7TH2EMIOPATVHCSNH2WXUGU7KMMEJ5LCHR22SWEHSA5OV4CUCXQ, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:31:00.629657982 (80h7m36.192980977s) [noise=true , address=104.36.82.42:6803 , piece_id=R626UFD326S2UBJEZ2WMHVR7EN4SK427VXPDJZOOZ32MX25JWZWQ, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:31:00.933110546 (80h7m36.496433536s) [noise=true , address=79.127.201.215:4326 , piece_id=USEWZDX6P52U6MCALTBNQG3SV7N2JRKNU6O23TLS6JEWAS4D7ARA, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:31:05.669167243 (80h7m41.232490237s) [noise=true , address=104.36.82.56:4261 , piece_id=EYOYOYLDACGMPGJYX6ILQT3XHMUTQNXZEDC7ANHLZXIKYVDQFFVQ, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:31:08.443582763 (80h7m44.006905763s) [noise=true , address=104.36.82.54:4184 , piece_id=IZICVGZFMAYLIJTNUJ2OFLZVNZ6IPPYDZO4BSTJJPLHVCVOE5EAA, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:31:26.459138014 (80h8m2.022461013s) [noise=true , address=104.36.82.53:4146 , piece_id=3V3DVGUOPJKDWIGQ2TU7M6VFYFVAS6ODY62ERGSU6LI3LSTWQ4YQ, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:31:27.456050336 (80h8m3.019373336s) [noise=true , address=104.36.82.52:4089 , piece_id=X2VG4S3UHVNWA4AQJS7FLKWNVLTOZ3AKWI5ZDH24QCA3ENQW7BXQ, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:31:55.270234251 (80h8m30.833557256s) [noise=true , address=104.36.82.11:6125 , piece_id=2CX7RJOKNQA6ODZZZXYPSCYAQY5P35TWX7I56CMYAPFLJE3JEO5Q, seg_pos={0 0}, upload=64142]: Done uploading piece. err: ecclient: upload failed (node:12vricofwx67FRaxAGLbmgT1TYyKkqfJbxdsyeEPL4R7kg8pKzT, address:104.36.82.11:6125): protocol: expected piece hash; context deadline exceeded; EOF Jun 22 00:31:55.270319418 (80h8m30.833642407s) [noise=true , address=104.36.82.55:4224 , piece_id=XF7H2BYSNNNZOS2T77VIN75N76AYUXAQEI6D4LRXACU2KU62V2TA, seg_pos={0 0}, upload=64142]: Done uploading piece. err: ecclient: upload failed (node:12EjKmUiv4pxkVCB4WoiXim4uFpxrBPyfiSS4efTY3HKSE3jb5y, address:104.36.82.55:4224): protocol: write tcp 159.223.143.39:39152->104.36.82.55:4224: use of closed network connection; write tcp 159.223.143.39:39152->104.36.82.55:4224: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:39152->104.36.82.55:4224: use of closed network connection Jun 22 00:31:55.270353811 (80h8m30.833676799s) [seg_pos={0 0}, upload=64142]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Jun 22 00:31:55.301099877 (80h8m30.864422884s) [upload=64142]: Done uploading segments... Jun 22 00:31:55.301121567 (80h8m30.864444566s) [upload=64142]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 37.739960768s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.............X..............................................] duration: 30.305072265s FAIL (2025-06-22 01:09:15 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== ing upload Jun 22 00:30:55.303646184 (80h7m30.866969171s) [upload=64142]: Uploading segments... Jun 22 00:30:55.303706406 (80h7m30.867029416s) [upload=64142]: Got next segment. Inline: false Jun 22 00:30:55.303762479 (80h7m30.867085468s) [upload=64142]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 22 00:30:55.303768627 (80h7m30.867091636s) [upload=64142]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 22 00:30:55.349058140 (80h7m30.912381168s) [seg_pos={0 0}, upload=64142]: Begin upload segment... Jun 22 00:30:55.349113768 (80h7m30.912436772s) [seg_pos={0 0}, upload=64142]: Done begin upload segment. Jun 22 00:30:55.349143121 (80h7m30.912466131s) [noise=true , address=45.91.116.5:5310 , piece_id=AXJOKDZRPVNL7HFT4QKJWRAHFM2UCM3GEVQB2KNYD6CXB6TH4D4Q, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.349660917 (80h7m30.912983928s) [noise=true , address=77.90.31.10:4533 , piece_id=SE6YAKKFDL3BV677CDDUC32UNMPXJHTG23GWWYK5B4YIPKER6GAQ, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.350093077 (80h7m30.913416089s) [noise=true , address=79.127.201.215:4326 , piece_id=USEWZDX6P52U6MCALTBNQG3SV7N2JRKNU6O23TLS6JEWAS4D7ARA, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.350480894 (80h7m30.913803916s) [noise=true , address=104.36.82.13:6199 , piece_id=YP62N2TO446YR3QRG5YLXECD7TN6SJ4UNNLZSK5H4CMV6JIF4AKA, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.350669998 (80h7m30.913993011s) [noise=true , address=104.36.82.54:4184 , piece_id=IZICVGZFMAYLIJTNUJ2OFLZVNZ6IPPYDZO4BSTJJPLHVCVOE5EAA, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.351201352 (80h7m30.914524366s) [noise=true , address=104.36.82.55:4224 , piece_id=XF7H2BYSNNNZOS2T77VIN75N76AYUXAQEI6D4LRXACU2KU62V2TA, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.351668071 (80h7m30.914991083s) [noise=true , address=104.36.82.56:4261 , piece_id=EYOYOYLDACGMPGJYX6ILQT3XHMUTQNXZEDC7ANHLZXIKYVDQFFVQ, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.352193126 (80h7m30.915516138s) [noise=true , address=104.36.82.52:4089 , piece_id=X2VG4S3UHVNWA4AQJS7FLKWNVLTOZ3AKWI5ZDH24QCA3ENQW7BXQ, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.352587096 (80h7m30.915910103s) [noise=true , address=79.127.220.98:4402 , piece_id=TE2WQO5LK45AVMPIAIGVSIBOVZLC4FG4ESQ72X6KFYC7IAEN2DRA, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.353420253 (80h7m30.916743265s) [noise=true , address=104.36.82.11:6125 , piece_id=2CX7RJOKNQA6ODZZZXYPSCYAQY5P35TWX7I56CMYAPFLJE3JEO5Q, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.353791565 (80h7m30.917114578s) [noise=true , address=104.36.82.42:6803 , piece_id=R626UFD326S2UBJEZ2WMHVR7EN4SK427VXPDJZOOZ32MX25JWZWQ, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.354171226 (80h7m30.917494233s) [noise=true , address=104.36.82.14:6245 , piece_id=II36G46OA6TINLR244U6D7Q6LGV754UA5M62K3EX7ZOBESMJUEHA, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.354542282 (80h7m30.917865288s) [noise=true , address=79.127.205.235:5660 , piece_id=UGFPVS7VNB7U2YSFPSGYDIRZV5R2DHCN57PJP6VOXO52WIAYHLXA, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.354891655 (80h7m30.918214678s) [noise=true , address=104.36.82.15:6283 , piece_id=N7TH2EMIOPATVHCSNH2WXUGU7KMMEJ5LCHR22SWEHSA5OV4CUCXQ, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.355499503 (80h7m30.918822519s) [noise=true , address=77.90.31.14:4666 , piece_id=H4DTFDAVCN266PB235BJIQTQ32FBXTIB5AZSX3B4R2VIK554AFSQ, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.356575547 (80h7m30.919898539s) [noise=true , address=104.36.82.39:6756 , piece_id=OKKWMDJPZO6D5VRMGOE3GOCYGHM725JZTNOVYAWKJW3WYZPI4V5Q, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.357039590 (80h7m30.920362596s) [noise=true , address=104.36.82.53:4146 , piece_id=3V3DVGUOPJKDWIGQ2TU7M6VFYFVAS6ODY62ERGSU6LI3LSTWQ4YQ, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.360343566 (80h7m30.923666582s) [noise=true , address=104.36.82.9:6033 , piece_id=T5XA23LZELMSTWOLBSHOX4AHKDACF65OYCKTMZYTXDIBSXNA2T4A, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.452444064 (80h7m31.015767055s) [upload=64142]: Next returned nil segment Jun 22 00:30:55.452495148 (80h7m31.015818149s) [upload=64142]: Waiting for error group managing segments... Jun 22 00:30:55.532958808 (80h7m31.096281883s) [noise=true , address=104.36.82.9:6033 , piece_id=T5XA23LZELMSTWOLBSHOX4AHKDACF65OYCKTMZYTXDIBSXNA2T4A, seg_pos={0 0}, upload=64142]: Done uploading piece. err: ecclient: upload failed (node:1SV3LrcQW16wLewDzfyUWFB6sUgnERNJVdhz72CJmyde95M79C, address:104.36.82.9:6033): protocol: storage node overloaded, request limit: 2000 Jun 22 00:30:55.547802184 (80h7m31.111125188s) [noise=true , address=79.127.220.98:4402 , piece_id=TE2WQO5LK45AVMPIAIGVSIBOVZLC4FG4ESQ72X6KFYC7IAEN2DRA, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:30:55.554223621 (80h7m31.117546616s) [noise=true , address=77.90.31.14:4666 , piece_id=H4DTFDAVCN266PB235BJIQTQ32FBXTIB5AZSX3B4R2VIK554AFSQ, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:30:55.762514255 (80h7m31.325837245s) [noise=true , address=104.36.82.14:6245 , piece_id=II36G46OA6TINLR244U6D7Q6LGV754UA5M62K3EX7ZOBESMJUEHA, seg_pos={0 0}, upload=64142]: Done uploading piece. err: ecclient: upload failed (node:1Zu8teNPTXn638grHKwtJqz4UEY7oFYLx1DHvuVP6RZXcatVqX, address:104.36.82.14:6245): protocol: storage node overloaded, request limit: 2000 Jun 22 00:30:55.773669583 (80h7m31.3369926s) [noise=true , address=79.127.205.235:5660 , piece_id=UGFPVS7VNB7U2YSFPSGYDIRZV5R2DHCN57PJP6VOXO52WIAYHLXA, seg_pos={0 0}, upload=64142]: Done uploading piece. err: ecclient: upload failed (node:12E6uCX9KL4Ud7YpxMXepvMXuqTFJCiSXByDjW94Am3U9H2Nf7f, address:79.127.205.235:5660): protocol: storage node overloaded, request limit: 2000 Jun 22 00:30:56.968366655 (80h7m32.53168964s) [noise=true , address=77.90.31.10:4533 , piece_id=SE6YAKKFDL3BV677CDDUC32UNMPXJHTG23GWWYK5B4YIPKER6GAQ, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:30:57.403572874 (80h7m32.966895883s) [noise=true , address=45.91.116.5:5310 , piece_id=AXJOKDZRPVNL7HFT4QKJWRAHFM2UCM3GEVQB2KNYD6CXB6TH4D4Q, seg_pos={0 0}, upload=64142]: Done uploading piece. err: ecclient: upload failed (node:127RMUc62xRq9RU5z3XfkZsq5sehmv3kP8o53tr9BxVcPE3GQCY, address:45.91.116.5:5310): protocol: storage node overloaded, request limit: 2000 Jun 22 00:30:58.210389994 (80h7m33.773712995s) [noise=true , address=104.36.82.13:6199 , piece_id=YP62N2TO446YR3QRG5YLXECD7TN6SJ4UNNLZSK5H4CMV6JIF4AKA, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:30:58.693037961 (80h7m34.256360966s) [noise=true , address=104.36.82.39:6756 , piece_id=OKKWMDJPZO6D5VRMGOE3GOCYGHM725JZTNOVYAWKJW3WYZPI4V5Q, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:30:59.124834814 (80h7m34.688157819s) [noise=true , address=104.36.82.15:6283 , piece_id=N7TH2EMIOPATVHCSNH2WXUGU7KMMEJ5LCHR22SWEHSA5OV4CUCXQ, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:31:00.629657982 (80h7m36.192980977s) [noise=true , address=104.36.82.42:6803 , piece_id=R626UFD326S2UBJEZ2WMHVR7EN4SK427VXPDJZOOZ32MX25JWZWQ, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:31:00.933110546 (80h7m36.496433536s) [noise=true , address=79.127.201.215:4326 , piece_id=USEWZDX6P52U6MCALTBNQG3SV7N2JRKNU6O23TLS6JEWAS4D7ARA, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:31:05.669167243 (80h7m41.232490237s) [noise=true , address=104.36.82.56:4261 , piece_id=EYOYOYLDACGMPGJYX6ILQT3XHMUTQNXZEDC7ANHLZXIKYVDQFFVQ, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:31:08.443582763 (80h7m44.006905763s) [noise=true , address=104.36.82.54:4184 , piece_id=IZICVGZFMAYLIJTNUJ2OFLZVNZ6IPPYDZO4BSTJJPLHVCVOE5EAA, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:31:26.459138014 (80h8m2.022461013s) [noise=true , address=104.36.82.53:4146 , piece_id=3V3DVGUOPJKDWIGQ2TU7M6VFYFVAS6ODY62ERGSU6LI3LSTWQ4YQ, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:31:27.456050336 (80h8m3.019373336s) [noise=true , address=104.36.82.52:4089 , piece_id=X2VG4S3UHVNWA4AQJS7FLKWNVLTOZ3AKWI5ZDH24QCA3ENQW7BXQ, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:31:55.270234251 (80h8m30.833557256s) [noise=true , address=104.36.82.11:6125 , piece_id=2CX7RJOKNQA6ODZZZXYPSCYAQY5P35TWX7I56CMYAPFLJE3JEO5Q, seg_pos={0 0}, upload=64142]: Done uploading piece. err: ecclient: upload failed (node:12vricofwx67FRaxAGLbmgT1TYyKkqfJbxdsyeEPL4R7kg8pKzT, address:104.36.82.11:6125): protocol: expected piece hash; context deadline exceeded; EOF Jun 22 00:31:55.270319418 (80h8m30.833642407s) [noise=true , address=104.36.82.55:4224 , piece_id=XF7H2BYSNNNZOS2T77VIN75N76AYUXAQEI6D4LRXACU2KU62V2TA, seg_pos={0 0}, upload=64142]: Done uploading piece. err: ecclient: upload failed (node:12EjKmUiv4pxkVCB4WoiXim4uFpxrBPyfiSS4efTY3HKSE3jb5y, address:104.36.82.55:4224): protocol: write tcp 159.223.143.39:39152->104.36.82.55:4224: use of closed network connection; write tcp 159.223.143.39:39152->104.36.82.55:4224: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:39152->104.36.82.55:4224: use of closed network connection Jun 22 00:31:55.270353811 (80h8m30.833676799s) [seg_pos={0 0}, upload=64142]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Jun 22 00:31:55.301099877 (80h8m30.864422884s) [upload=64142]: Done uploading segments... Jun 22 00:31:55.301121567 (80h8m30.864444566s) [upload=64142]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 30.305135739s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.............X..............................................] duration: 13.385556374s FAIL (2025-06-22 01:08:49 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 13.3858354s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [..............X.............................................] duration: 20.278147333s FAIL (2025-06-22 01:08:11 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 20.278197311s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [..............X.............................................] duration: 25.769160114s FAIL (2025-06-22 01:07:56 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== ing upload Jun 22 00:30:55.303646184 (80h7m30.866969171s) [upload=64142]: Uploading segments... Jun 22 00:30:55.303706406 (80h7m30.867029416s) [upload=64142]: Got next segment. Inline: false Jun 22 00:30:55.303762479 (80h7m30.867085468s) [upload=64142]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 22 00:30:55.303768627 (80h7m30.867091636s) [upload=64142]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 22 00:30:55.349058140 (80h7m30.912381168s) [seg_pos={0 0}, upload=64142]: Begin upload segment... Jun 22 00:30:55.349113768 (80h7m30.912436772s) [seg_pos={0 0}, upload=64142]: Done begin upload segment. Jun 22 00:30:55.349143121 (80h7m30.912466131s) [noise=true , address=45.91.116.5:5310 , piece_id=AXJOKDZRPVNL7HFT4QKJWRAHFM2UCM3GEVQB2KNYD6CXB6TH4D4Q, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.349660917 (80h7m30.912983928s) [noise=true , address=77.90.31.10:4533 , piece_id=SE6YAKKFDL3BV677CDDUC32UNMPXJHTG23GWWYK5B4YIPKER6GAQ, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.350093077 (80h7m30.913416089s) [noise=true , address=79.127.201.215:4326 , piece_id=USEWZDX6P52U6MCALTBNQG3SV7N2JRKNU6O23TLS6JEWAS4D7ARA, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.350480894 (80h7m30.913803916s) [noise=true , address=104.36.82.13:6199 , piece_id=YP62N2TO446YR3QRG5YLXECD7TN6SJ4UNNLZSK5H4CMV6JIF4AKA, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.350669998 (80h7m30.913993011s) [noise=true , address=104.36.82.54:4184 , piece_id=IZICVGZFMAYLIJTNUJ2OFLZVNZ6IPPYDZO4BSTJJPLHVCVOE5EAA, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.351201352 (80h7m30.914524366s) [noise=true , address=104.36.82.55:4224 , piece_id=XF7H2BYSNNNZOS2T77VIN75N76AYUXAQEI6D4LRXACU2KU62V2TA, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.351668071 (80h7m30.914991083s) [noise=true , address=104.36.82.56:4261 , piece_id=EYOYOYLDACGMPGJYX6ILQT3XHMUTQNXZEDC7ANHLZXIKYVDQFFVQ, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.352193126 (80h7m30.915516138s) [noise=true , address=104.36.82.52:4089 , piece_id=X2VG4S3UHVNWA4AQJS7FLKWNVLTOZ3AKWI5ZDH24QCA3ENQW7BXQ, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.352587096 (80h7m30.915910103s) [noise=true , address=79.127.220.98:4402 , piece_id=TE2WQO5LK45AVMPIAIGVSIBOVZLC4FG4ESQ72X6KFYC7IAEN2DRA, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.353420253 (80h7m30.916743265s) [noise=true , address=104.36.82.11:6125 , piece_id=2CX7RJOKNQA6ODZZZXYPSCYAQY5P35TWX7I56CMYAPFLJE3JEO5Q, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.353791565 (80h7m30.917114578s) [noise=true , address=104.36.82.42:6803 , piece_id=R626UFD326S2UBJEZ2WMHVR7EN4SK427VXPDJZOOZ32MX25JWZWQ, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.354171226 (80h7m30.917494233s) [noise=true , address=104.36.82.14:6245 , piece_id=II36G46OA6TINLR244U6D7Q6LGV754UA5M62K3EX7ZOBESMJUEHA, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.354542282 (80h7m30.917865288s) [noise=true , address=79.127.205.235:5660 , piece_id=UGFPVS7VNB7U2YSFPSGYDIRZV5R2DHCN57PJP6VOXO52WIAYHLXA, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.354891655 (80h7m30.918214678s) [noise=true , address=104.36.82.15:6283 , piece_id=N7TH2EMIOPATVHCSNH2WXUGU7KMMEJ5LCHR22SWEHSA5OV4CUCXQ, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.355499503 (80h7m30.918822519s) [noise=true , address=77.90.31.14:4666 , piece_id=H4DTFDAVCN266PB235BJIQTQ32FBXTIB5AZSX3B4R2VIK554AFSQ, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.356575547 (80h7m30.919898539s) [noise=true , address=104.36.82.39:6756 , piece_id=OKKWMDJPZO6D5VRMGOE3GOCYGHM725JZTNOVYAWKJW3WYZPI4V5Q, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.357039590 (80h7m30.920362596s) [noise=true , address=104.36.82.53:4146 , piece_id=3V3DVGUOPJKDWIGQ2TU7M6VFYFVAS6ODY62ERGSU6LI3LSTWQ4YQ, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.360343566 (80h7m30.923666582s) [noise=true , address=104.36.82.9:6033 , piece_id=T5XA23LZELMSTWOLBSHOX4AHKDACF65OYCKTMZYTXDIBSXNA2T4A, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.452444064 (80h7m31.015767055s) [upload=64142]: Next returned nil segment Jun 22 00:30:55.452495148 (80h7m31.015818149s) [upload=64142]: Waiting for error group managing segments... Jun 22 00:30:55.532958808 (80h7m31.096281883s) [noise=true , address=104.36.82.9:6033 , piece_id=T5XA23LZELMSTWOLBSHOX4AHKDACF65OYCKTMZYTXDIBSXNA2T4A, seg_pos={0 0}, upload=64142]: Done uploading piece. err: ecclient: upload failed (node:1SV3LrcQW16wLewDzfyUWFB6sUgnERNJVdhz72CJmyde95M79C, address:104.36.82.9:6033): protocol: storage node overloaded, request limit: 2000 Jun 22 00:30:55.547802184 (80h7m31.111125188s) [noise=true , address=79.127.220.98:4402 , piece_id=TE2WQO5LK45AVMPIAIGVSIBOVZLC4FG4ESQ72X6KFYC7IAEN2DRA, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:30:55.554223621 (80h7m31.117546616s) [noise=true , address=77.90.31.14:4666 , piece_id=H4DTFDAVCN266PB235BJIQTQ32FBXTIB5AZSX3B4R2VIK554AFSQ, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:30:55.762514255 (80h7m31.325837245s) [noise=true , address=104.36.82.14:6245 , piece_id=II36G46OA6TINLR244U6D7Q6LGV754UA5M62K3EX7ZOBESMJUEHA, seg_pos={0 0}, upload=64142]: Done uploading piece. err: ecclient: upload failed (node:1Zu8teNPTXn638grHKwtJqz4UEY7oFYLx1DHvuVP6RZXcatVqX, address:104.36.82.14:6245): protocol: storage node overloaded, request limit: 2000 Jun 22 00:30:55.773669583 (80h7m31.3369926s) [noise=true , address=79.127.205.235:5660 , piece_id=UGFPVS7VNB7U2YSFPSGYDIRZV5R2DHCN57PJP6VOXO52WIAYHLXA, seg_pos={0 0}, upload=64142]: Done uploading piece. err: ecclient: upload failed (node:12E6uCX9KL4Ud7YpxMXepvMXuqTFJCiSXByDjW94Am3U9H2Nf7f, address:79.127.205.235:5660): protocol: storage node overloaded, request limit: 2000 Jun 22 00:30:56.968366655 (80h7m32.53168964s) [noise=true , address=77.90.31.10:4533 , piece_id=SE6YAKKFDL3BV677CDDUC32UNMPXJHTG23GWWYK5B4YIPKER6GAQ, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:30:57.403572874 (80h7m32.966895883s) [noise=true , address=45.91.116.5:5310 , piece_id=AXJOKDZRPVNL7HFT4QKJWRAHFM2UCM3GEVQB2KNYD6CXB6TH4D4Q, seg_pos={0 0}, upload=64142]: Done uploading piece. err: ecclient: upload failed (node:127RMUc62xRq9RU5z3XfkZsq5sehmv3kP8o53tr9BxVcPE3GQCY, address:45.91.116.5:5310): protocol: storage node overloaded, request limit: 2000 Jun 22 00:30:58.210389994 (80h7m33.773712995s) [noise=true , address=104.36.82.13:6199 , piece_id=YP62N2TO446YR3QRG5YLXECD7TN6SJ4UNNLZSK5H4CMV6JIF4AKA, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:30:58.693037961 (80h7m34.256360966s) [noise=true , address=104.36.82.39:6756 , piece_id=OKKWMDJPZO6D5VRMGOE3GOCYGHM725JZTNOVYAWKJW3WYZPI4V5Q, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:30:59.124834814 (80h7m34.688157819s) [noise=true , address=104.36.82.15:6283 , piece_id=N7TH2EMIOPATVHCSNH2WXUGU7KMMEJ5LCHR22SWEHSA5OV4CUCXQ, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:31:00.629657982 (80h7m36.192980977s) [noise=true , address=104.36.82.42:6803 , piece_id=R626UFD326S2UBJEZ2WMHVR7EN4SK427VXPDJZOOZ32MX25JWZWQ, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:31:00.933110546 (80h7m36.496433536s) [noise=true , address=79.127.201.215:4326 , piece_id=USEWZDX6P52U6MCALTBNQG3SV7N2JRKNU6O23TLS6JEWAS4D7ARA, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:31:05.669167243 (80h7m41.232490237s) [noise=true , address=104.36.82.56:4261 , piece_id=EYOYOYLDACGMPGJYX6ILQT3XHMUTQNXZEDC7ANHLZXIKYVDQFFVQ, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:31:08.443582763 (80h7m44.006905763s) [noise=true , address=104.36.82.54:4184 , piece_id=IZICVGZFMAYLIJTNUJ2OFLZVNZ6IPPYDZO4BSTJJPLHVCVOE5EAA, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:31:26.459138014 (80h8m2.022461013s) [noise=true , address=104.36.82.53:4146 , piece_id=3V3DVGUOPJKDWIGQ2TU7M6VFYFVAS6ODY62ERGSU6LI3LSTWQ4YQ, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:31:27.456050336 (80h8m3.019373336s) [noise=true , address=104.36.82.52:4089 , piece_id=X2VG4S3UHVNWA4AQJS7FLKWNVLTOZ3AKWI5ZDH24QCA3ENQW7BXQ, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:31:55.270234251 (80h8m30.833557256s) [noise=true , address=104.36.82.11:6125 , piece_id=2CX7RJOKNQA6ODZZZXYPSCYAQY5P35TWX7I56CMYAPFLJE3JEO5Q, seg_pos={0 0}, upload=64142]: Done uploading piece. err: ecclient: upload failed (node:12vricofwx67FRaxAGLbmgT1TYyKkqfJbxdsyeEPL4R7kg8pKzT, address:104.36.82.11:6125): protocol: expected piece hash; context deadline exceeded; EOF Jun 22 00:31:55.270319418 (80h8m30.833642407s) [noise=true , address=104.36.82.55:4224 , piece_id=XF7H2BYSNNNZOS2T77VIN75N76AYUXAQEI6D4LRXACU2KU62V2TA, seg_pos={0 0}, upload=64142]: Done uploading piece. err: ecclient: upload failed (node:12EjKmUiv4pxkVCB4WoiXim4uFpxrBPyfiSS4efTY3HKSE3jb5y, address:104.36.82.55:4224): protocol: write tcp 159.223.143.39:39152->104.36.82.55:4224: use of closed network connection; write tcp 159.223.143.39:39152->104.36.82.55:4224: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:39152->104.36.82.55:4224: use of closed network connection Jun 22 00:31:55.270353811 (80h8m30.833676799s) [seg_pos={0 0}, upload=64142]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Jun 22 00:31:55.301099877 (80h8m30.864422884s) [upload=64142]: Done uploading segments... Jun 22 00:31:55.301121567 (80h8m30.864444566s) [upload=64142]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 25.769188932s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [...............X............................................] duration: 14.018306581s FAIL (2025-06-22 01:07:11 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 14.018337687s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [...............X............................................] duration: 18.597753182s FAIL (2025-06-22 01:06:39 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== ing upload Jun 22 00:30:55.303646184 (80h7m30.866969171s) [upload=64142]: Uploading segments... Jun 22 00:30:55.303706406 (80h7m30.867029416s) [upload=64142]: Got next segment. Inline: false Jun 22 00:30:55.303762479 (80h7m30.867085468s) [upload=64142]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 22 00:30:55.303768627 (80h7m30.867091636s) [upload=64142]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 22 00:30:55.349058140 (80h7m30.912381168s) [seg_pos={0 0}, upload=64142]: Begin upload segment... Jun 22 00:30:55.349113768 (80h7m30.912436772s) [seg_pos={0 0}, upload=64142]: Done begin upload segment. Jun 22 00:30:55.349143121 (80h7m30.912466131s) [noise=true , address=45.91.116.5:5310 , piece_id=AXJOKDZRPVNL7HFT4QKJWRAHFM2UCM3GEVQB2KNYD6CXB6TH4D4Q, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.349660917 (80h7m30.912983928s) [noise=true , address=77.90.31.10:4533 , piece_id=SE6YAKKFDL3BV677CDDUC32UNMPXJHTG23GWWYK5B4YIPKER6GAQ, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.350093077 (80h7m30.913416089s) [noise=true , address=79.127.201.215:4326 , piece_id=USEWZDX6P52U6MCALTBNQG3SV7N2JRKNU6O23TLS6JEWAS4D7ARA, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.350480894 (80h7m30.913803916s) [noise=true , address=104.36.82.13:6199 , piece_id=YP62N2TO446YR3QRG5YLXECD7TN6SJ4UNNLZSK5H4CMV6JIF4AKA, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.350669998 (80h7m30.913993011s) [noise=true , address=104.36.82.54:4184 , piece_id=IZICVGZFMAYLIJTNUJ2OFLZVNZ6IPPYDZO4BSTJJPLHVCVOE5EAA, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.351201352 (80h7m30.914524366s) [noise=true , address=104.36.82.55:4224 , piece_id=XF7H2BYSNNNZOS2T77VIN75N76AYUXAQEI6D4LRXACU2KU62V2TA, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.351668071 (80h7m30.914991083s) [noise=true , address=104.36.82.56:4261 , piece_id=EYOYOYLDACGMPGJYX6ILQT3XHMUTQNXZEDC7ANHLZXIKYVDQFFVQ, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.352193126 (80h7m30.915516138s) [noise=true , address=104.36.82.52:4089 , piece_id=X2VG4S3UHVNWA4AQJS7FLKWNVLTOZ3AKWI5ZDH24QCA3ENQW7BXQ, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.352587096 (80h7m30.915910103s) [noise=true , address=79.127.220.98:4402 , piece_id=TE2WQO5LK45AVMPIAIGVSIBOVZLC4FG4ESQ72X6KFYC7IAEN2DRA, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.353420253 (80h7m30.916743265s) [noise=true , address=104.36.82.11:6125 , piece_id=2CX7RJOKNQA6ODZZZXYPSCYAQY5P35TWX7I56CMYAPFLJE3JEO5Q, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.353791565 (80h7m30.917114578s) [noise=true , address=104.36.82.42:6803 , piece_id=R626UFD326S2UBJEZ2WMHVR7EN4SK427VXPDJZOOZ32MX25JWZWQ, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.354171226 (80h7m30.917494233s) [noise=true , address=104.36.82.14:6245 , piece_id=II36G46OA6TINLR244U6D7Q6LGV754UA5M62K3EX7ZOBESMJUEHA, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.354542282 (80h7m30.917865288s) [noise=true , address=79.127.205.235:5660 , piece_id=UGFPVS7VNB7U2YSFPSGYDIRZV5R2DHCN57PJP6VOXO52WIAYHLXA, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.354891655 (80h7m30.918214678s) [noise=true , address=104.36.82.15:6283 , piece_id=N7TH2EMIOPATVHCSNH2WXUGU7KMMEJ5LCHR22SWEHSA5OV4CUCXQ, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.355499503 (80h7m30.918822519s) [noise=true , address=77.90.31.14:4666 , piece_id=H4DTFDAVCN266PB235BJIQTQ32FBXTIB5AZSX3B4R2VIK554AFSQ, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.356575547 (80h7m30.919898539s) [noise=true , address=104.36.82.39:6756 , piece_id=OKKWMDJPZO6D5VRMGOE3GOCYGHM725JZTNOVYAWKJW3WYZPI4V5Q, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.357039590 (80h7m30.920362596s) [noise=true , address=104.36.82.53:4146 , piece_id=3V3DVGUOPJKDWIGQ2TU7M6VFYFVAS6ODY62ERGSU6LI3LSTWQ4YQ, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.360343566 (80h7m30.923666582s) [noise=true , address=104.36.82.9:6033 , piece_id=T5XA23LZELMSTWOLBSHOX4AHKDACF65OYCKTMZYTXDIBSXNA2T4A, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.452444064 (80h7m31.015767055s) [upload=64142]: Next returned nil segment Jun 22 00:30:55.452495148 (80h7m31.015818149s) [upload=64142]: Waiting for error group managing segments... Jun 22 00:30:55.532958808 (80h7m31.096281883s) [noise=true , address=104.36.82.9:6033 , piece_id=T5XA23LZELMSTWOLBSHOX4AHKDACF65OYCKTMZYTXDIBSXNA2T4A, seg_pos={0 0}, upload=64142]: Done uploading piece. err: ecclient: upload failed (node:1SV3LrcQW16wLewDzfyUWFB6sUgnERNJVdhz72CJmyde95M79C, address:104.36.82.9:6033): protocol: storage node overloaded, request limit: 2000 Jun 22 00:30:55.547802184 (80h7m31.111125188s) [noise=true , address=79.127.220.98:4402 , piece_id=TE2WQO5LK45AVMPIAIGVSIBOVZLC4FG4ESQ72X6KFYC7IAEN2DRA, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:30:55.554223621 (80h7m31.117546616s) [noise=true , address=77.90.31.14:4666 , piece_id=H4DTFDAVCN266PB235BJIQTQ32FBXTIB5AZSX3B4R2VIK554AFSQ, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:30:55.762514255 (80h7m31.325837245s) [noise=true , address=104.36.82.14:6245 , piece_id=II36G46OA6TINLR244U6D7Q6LGV754UA5M62K3EX7ZOBESMJUEHA, seg_pos={0 0}, upload=64142]: Done uploading piece. err: ecclient: upload failed (node:1Zu8teNPTXn638grHKwtJqz4UEY7oFYLx1DHvuVP6RZXcatVqX, address:104.36.82.14:6245): protocol: storage node overloaded, request limit: 2000 Jun 22 00:30:55.773669583 (80h7m31.3369926s) [noise=true , address=79.127.205.235:5660 , piece_id=UGFPVS7VNB7U2YSFPSGYDIRZV5R2DHCN57PJP6VOXO52WIAYHLXA, seg_pos={0 0}, upload=64142]: Done uploading piece. err: ecclient: upload failed (node:12E6uCX9KL4Ud7YpxMXepvMXuqTFJCiSXByDjW94Am3U9H2Nf7f, address:79.127.205.235:5660): protocol: storage node overloaded, request limit: 2000 Jun 22 00:30:56.968366655 (80h7m32.53168964s) [noise=true , address=77.90.31.10:4533 , piece_id=SE6YAKKFDL3BV677CDDUC32UNMPXJHTG23GWWYK5B4YIPKER6GAQ, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:30:57.403572874 (80h7m32.966895883s) [noise=true , address=45.91.116.5:5310 , piece_id=AXJOKDZRPVNL7HFT4QKJWRAHFM2UCM3GEVQB2KNYD6CXB6TH4D4Q, seg_pos={0 0}, upload=64142]: Done uploading piece. err: ecclient: upload failed (node:127RMUc62xRq9RU5z3XfkZsq5sehmv3kP8o53tr9BxVcPE3GQCY, address:45.91.116.5:5310): protocol: storage node overloaded, request limit: 2000 Jun 22 00:30:58.210389994 (80h7m33.773712995s) [noise=true , address=104.36.82.13:6199 , piece_id=YP62N2TO446YR3QRG5YLXECD7TN6SJ4UNNLZSK5H4CMV6JIF4AKA, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:30:58.693037961 (80h7m34.256360966s) [noise=true , address=104.36.82.39:6756 , piece_id=OKKWMDJPZO6D5VRMGOE3GOCYGHM725JZTNOVYAWKJW3WYZPI4V5Q, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:30:59.124834814 (80h7m34.688157819s) [noise=true , address=104.36.82.15:6283 , piece_id=N7TH2EMIOPATVHCSNH2WXUGU7KMMEJ5LCHR22SWEHSA5OV4CUCXQ, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:31:00.629657982 (80h7m36.192980977s) [noise=true , address=104.36.82.42:6803 , piece_id=R626UFD326S2UBJEZ2WMHVR7EN4SK427VXPDJZOOZ32MX25JWZWQ, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:31:00.933110546 (80h7m36.496433536s) [noise=true , address=79.127.201.215:4326 , piece_id=USEWZDX6P52U6MCALTBNQG3SV7N2JRKNU6O23TLS6JEWAS4D7ARA, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:31:05.669167243 (80h7m41.232490237s) [noise=true , address=104.36.82.56:4261 , piece_id=EYOYOYLDACGMPGJYX6ILQT3XHMUTQNXZEDC7ANHLZXIKYVDQFFVQ, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:31:08.443582763 (80h7m44.006905763s) [noise=true , address=104.36.82.54:4184 , piece_id=IZICVGZFMAYLIJTNUJ2OFLZVNZ6IPPYDZO4BSTJJPLHVCVOE5EAA, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:31:26.459138014 (80h8m2.022461013s) [noise=true , address=104.36.82.53:4146 , piece_id=3V3DVGUOPJKDWIGQ2TU7M6VFYFVAS6ODY62ERGSU6LI3LSTWQ4YQ, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:31:27.456050336 (80h8m3.019373336s) [noise=true , address=104.36.82.52:4089 , piece_id=X2VG4S3UHVNWA4AQJS7FLKWNVLTOZ3AKWI5ZDH24QCA3ENQW7BXQ, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:31:55.270234251 (80h8m30.833557256s) [noise=true , address=104.36.82.11:6125 , piece_id=2CX7RJOKNQA6ODZZZXYPSCYAQY5P35TWX7I56CMYAPFLJE3JEO5Q, seg_pos={0 0}, upload=64142]: Done uploading piece. err: ecclient: upload failed (node:12vricofwx67FRaxAGLbmgT1TYyKkqfJbxdsyeEPL4R7kg8pKzT, address:104.36.82.11:6125): protocol: expected piece hash; context deadline exceeded; EOF Jun 22 00:31:55.270319418 (80h8m30.833642407s) [noise=true , address=104.36.82.55:4224 , piece_id=XF7H2BYSNNNZOS2T77VIN75N76AYUXAQEI6D4LRXACU2KU62V2TA, seg_pos={0 0}, upload=64142]: Done uploading piece. err: ecclient: upload failed (node:12EjKmUiv4pxkVCB4WoiXim4uFpxrBPyfiSS4efTY3HKSE3jb5y, address:104.36.82.55:4224): protocol: write tcp 159.223.143.39:39152->104.36.82.55:4224: use of closed network connection; write tcp 159.223.143.39:39152->104.36.82.55:4224: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:39152->104.36.82.55:4224: use of closed network connection Jun 22 00:31:55.270353811 (80h8m30.833676799s) [seg_pos={0 0}, upload=64142]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Jun 22 00:31:55.301099877 (80h8m30.864422884s) [upload=64142]: Done uploading segments... Jun 22 00:31:55.301121567 (80h8m30.864444566s) [upload=64142]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 18.597853708s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [................X...........................................] duration: 15.857920927s FAIL (2025-06-22 01:06:26 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 15.857948439s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.................X..........................................] duration: 14.565556947s FAIL (2025-06-22 01:05:33 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 14.565583805s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [................X...........................................] duration: 15.670718998s FAIL (2025-06-22 01:05:31 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== ing upload Jun 22 00:30:55.303646184 (80h7m30.866969171s) [upload=64142]: Uploading segments... Jun 22 00:30:55.303706406 (80h7m30.867029416s) [upload=64142]: Got next segment. Inline: false Jun 22 00:30:55.303762479 (80h7m30.867085468s) [upload=64142]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 22 00:30:55.303768627 (80h7m30.867091636s) [upload=64142]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 22 00:30:55.349058140 (80h7m30.912381168s) [seg_pos={0 0}, upload=64142]: Begin upload segment... Jun 22 00:30:55.349113768 (80h7m30.912436772s) [seg_pos={0 0}, upload=64142]: Done begin upload segment. Jun 22 00:30:55.349143121 (80h7m30.912466131s) [noise=true , address=45.91.116.5:5310 , piece_id=AXJOKDZRPVNL7HFT4QKJWRAHFM2UCM3GEVQB2KNYD6CXB6TH4D4Q, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.349660917 (80h7m30.912983928s) [noise=true , address=77.90.31.10:4533 , piece_id=SE6YAKKFDL3BV677CDDUC32UNMPXJHTG23GWWYK5B4YIPKER6GAQ, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.350093077 (80h7m30.913416089s) [noise=true , address=79.127.201.215:4326 , piece_id=USEWZDX6P52U6MCALTBNQG3SV7N2JRKNU6O23TLS6JEWAS4D7ARA, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.350480894 (80h7m30.913803916s) [noise=true , address=104.36.82.13:6199 , piece_id=YP62N2TO446YR3QRG5YLXECD7TN6SJ4UNNLZSK5H4CMV6JIF4AKA, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.350669998 (80h7m30.913993011s) [noise=true , address=104.36.82.54:4184 , piece_id=IZICVGZFMAYLIJTNUJ2OFLZVNZ6IPPYDZO4BSTJJPLHVCVOE5EAA, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.351201352 (80h7m30.914524366s) [noise=true , address=104.36.82.55:4224 , piece_id=XF7H2BYSNNNZOS2T77VIN75N76AYUXAQEI6D4LRXACU2KU62V2TA, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.351668071 (80h7m30.914991083s) [noise=true , address=104.36.82.56:4261 , piece_id=EYOYOYLDACGMPGJYX6ILQT3XHMUTQNXZEDC7ANHLZXIKYVDQFFVQ, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.352193126 (80h7m30.915516138s) [noise=true , address=104.36.82.52:4089 , piece_id=X2VG4S3UHVNWA4AQJS7FLKWNVLTOZ3AKWI5ZDH24QCA3ENQW7BXQ, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.352587096 (80h7m30.915910103s) [noise=true , address=79.127.220.98:4402 , piece_id=TE2WQO5LK45AVMPIAIGVSIBOVZLC4FG4ESQ72X6KFYC7IAEN2DRA, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.353420253 (80h7m30.916743265s) [noise=true , address=104.36.82.11:6125 , piece_id=2CX7RJOKNQA6ODZZZXYPSCYAQY5P35TWX7I56CMYAPFLJE3JEO5Q, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.353791565 (80h7m30.917114578s) [noise=true , address=104.36.82.42:6803 , piece_id=R626UFD326S2UBJEZ2WMHVR7EN4SK427VXPDJZOOZ32MX25JWZWQ, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.354171226 (80h7m30.917494233s) [noise=true , address=104.36.82.14:6245 , piece_id=II36G46OA6TINLR244U6D7Q6LGV754UA5M62K3EX7ZOBESMJUEHA, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.354542282 (80h7m30.917865288s) [noise=true , address=79.127.205.235:5660 , piece_id=UGFPVS7VNB7U2YSFPSGYDIRZV5R2DHCN57PJP6VOXO52WIAYHLXA, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.354891655 (80h7m30.918214678s) [noise=true , address=104.36.82.15:6283 , piece_id=N7TH2EMIOPATVHCSNH2WXUGU7KMMEJ5LCHR22SWEHSA5OV4CUCXQ, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.355499503 (80h7m30.918822519s) [noise=true , address=77.90.31.14:4666 , piece_id=H4DTFDAVCN266PB235BJIQTQ32FBXTIB5AZSX3B4R2VIK554AFSQ, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.356575547 (80h7m30.919898539s) [noise=true , address=104.36.82.39:6756 , piece_id=OKKWMDJPZO6D5VRMGOE3GOCYGHM725JZTNOVYAWKJW3WYZPI4V5Q, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.357039590 (80h7m30.920362596s) [noise=true , address=104.36.82.53:4146 , piece_id=3V3DVGUOPJKDWIGQ2TU7M6VFYFVAS6ODY62ERGSU6LI3LSTWQ4YQ, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.360343566 (80h7m30.923666582s) [noise=true , address=104.36.82.9:6033 , piece_id=T5XA23LZELMSTWOLBSHOX4AHKDACF65OYCKTMZYTXDIBSXNA2T4A, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.452444064 (80h7m31.015767055s) [upload=64142]: Next returned nil segment Jun 22 00:30:55.452495148 (80h7m31.015818149s) [upload=64142]: Waiting for error group managing segments... Jun 22 00:30:55.532958808 (80h7m31.096281883s) [noise=true , address=104.36.82.9:6033 , piece_id=T5XA23LZELMSTWOLBSHOX4AHKDACF65OYCKTMZYTXDIBSXNA2T4A, seg_pos={0 0}, upload=64142]: Done uploading piece. err: ecclient: upload failed (node:1SV3LrcQW16wLewDzfyUWFB6sUgnERNJVdhz72CJmyde95M79C, address:104.36.82.9:6033): protocol: storage node overloaded, request limit: 2000 Jun 22 00:30:55.547802184 (80h7m31.111125188s) [noise=true , address=79.127.220.98:4402 , piece_id=TE2WQO5LK45AVMPIAIGVSIBOVZLC4FG4ESQ72X6KFYC7IAEN2DRA, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:30:55.554223621 (80h7m31.117546616s) [noise=true , address=77.90.31.14:4666 , piece_id=H4DTFDAVCN266PB235BJIQTQ32FBXTIB5AZSX3B4R2VIK554AFSQ, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:30:55.762514255 (80h7m31.325837245s) [noise=true , address=104.36.82.14:6245 , piece_id=II36G46OA6TINLR244U6D7Q6LGV754UA5M62K3EX7ZOBESMJUEHA, seg_pos={0 0}, upload=64142]: Done uploading piece. err: ecclient: upload failed (node:1Zu8teNPTXn638grHKwtJqz4UEY7oFYLx1DHvuVP6RZXcatVqX, address:104.36.82.14:6245): protocol: storage node overloaded, request limit: 2000 Jun 22 00:30:55.773669583 (80h7m31.3369926s) [noise=true , address=79.127.205.235:5660 , piece_id=UGFPVS7VNB7U2YSFPSGYDIRZV5R2DHCN57PJP6VOXO52WIAYHLXA, seg_pos={0 0}, upload=64142]: Done uploading piece. err: ecclient: upload failed (node:12E6uCX9KL4Ud7YpxMXepvMXuqTFJCiSXByDjW94Am3U9H2Nf7f, address:79.127.205.235:5660): protocol: storage node overloaded, request limit: 2000 Jun 22 00:30:56.968366655 (80h7m32.53168964s) [noise=true , address=77.90.31.10:4533 , piece_id=SE6YAKKFDL3BV677CDDUC32UNMPXJHTG23GWWYK5B4YIPKER6GAQ, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:30:57.403572874 (80h7m32.966895883s) [noise=true , address=45.91.116.5:5310 , piece_id=AXJOKDZRPVNL7HFT4QKJWRAHFM2UCM3GEVQB2KNYD6CXB6TH4D4Q, seg_pos={0 0}, upload=64142]: Done uploading piece. err: ecclient: upload failed (node:127RMUc62xRq9RU5z3XfkZsq5sehmv3kP8o53tr9BxVcPE3GQCY, address:45.91.116.5:5310): protocol: storage node overloaded, request limit: 2000 Jun 22 00:30:58.210389994 (80h7m33.773712995s) [noise=true , address=104.36.82.13:6199 , piece_id=YP62N2TO446YR3QRG5YLXECD7TN6SJ4UNNLZSK5H4CMV6JIF4AKA, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:30:58.693037961 (80h7m34.256360966s) [noise=true , address=104.36.82.39:6756 , piece_id=OKKWMDJPZO6D5VRMGOE3GOCYGHM725JZTNOVYAWKJW3WYZPI4V5Q, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:30:59.124834814 (80h7m34.688157819s) [noise=true , address=104.36.82.15:6283 , piece_id=N7TH2EMIOPATVHCSNH2WXUGU7KMMEJ5LCHR22SWEHSA5OV4CUCXQ, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:31:00.629657982 (80h7m36.192980977s) [noise=true , address=104.36.82.42:6803 , piece_id=R626UFD326S2UBJEZ2WMHVR7EN4SK427VXPDJZOOZ32MX25JWZWQ, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:31:00.933110546 (80h7m36.496433536s) [noise=true , address=79.127.201.215:4326 , piece_id=USEWZDX6P52U6MCALTBNQG3SV7N2JRKNU6O23TLS6JEWAS4D7ARA, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:31:05.669167243 (80h7m41.232490237s) [noise=true , address=104.36.82.56:4261 , piece_id=EYOYOYLDACGMPGJYX6ILQT3XHMUTQNXZEDC7ANHLZXIKYVDQFFVQ, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:31:08.443582763 (80h7m44.006905763s) [noise=true , address=104.36.82.54:4184 , piece_id=IZICVGZFMAYLIJTNUJ2OFLZVNZ6IPPYDZO4BSTJJPLHVCVOE5EAA, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:31:26.459138014 (80h8m2.022461013s) [noise=true , address=104.36.82.53:4146 , piece_id=3V3DVGUOPJKDWIGQ2TU7M6VFYFVAS6ODY62ERGSU6LI3LSTWQ4YQ, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:31:27.456050336 (80h8m3.019373336s) [noise=true , address=104.36.82.52:4089 , piece_id=X2VG4S3UHVNWA4AQJS7FLKWNVLTOZ3AKWI5ZDH24QCA3ENQW7BXQ, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:31:55.270234251 (80h8m30.833557256s) [noise=true , address=104.36.82.11:6125 , piece_id=2CX7RJOKNQA6ODZZZXYPSCYAQY5P35TWX7I56CMYAPFLJE3JEO5Q, seg_pos={0 0}, upload=64142]: Done uploading piece. err: ecclient: upload failed (node:12vricofwx67FRaxAGLbmgT1TYyKkqfJbxdsyeEPL4R7kg8pKzT, address:104.36.82.11:6125): protocol: expected piece hash; context deadline exceeded; EOF Jun 22 00:31:55.270319418 (80h8m30.833642407s) [noise=true , address=104.36.82.55:4224 , piece_id=XF7H2BYSNNNZOS2T77VIN75N76AYUXAQEI6D4LRXACU2KU62V2TA, seg_pos={0 0}, upload=64142]: Done uploading piece. err: ecclient: upload failed (node:12EjKmUiv4pxkVCB4WoiXim4uFpxrBPyfiSS4efTY3HKSE3jb5y, address:104.36.82.55:4224): protocol: write tcp 159.223.143.39:39152->104.36.82.55:4224: use of closed network connection; write tcp 159.223.143.39:39152->104.36.82.55:4224: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:39152->104.36.82.55:4224: use of closed network connection Jun 22 00:31:55.270353811 (80h8m30.833676799s) [seg_pos={0 0}, upload=64142]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Jun 22 00:31:55.301099877 (80h8m30.864422884s) [upload=64142]: Done uploading segments... Jun 22 00:31:55.301121567 (80h8m30.864444566s) [upload=64142]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 15.670744778s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.................X..........................................] duration: 21.53017151s FAIL (2025-06-22 01:05:11 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== ing upload Jun 22 00:30:55.303646184 (80h7m30.866969171s) [upload=64142]: Uploading segments... Jun 22 00:30:55.303706406 (80h7m30.867029416s) [upload=64142]: Got next segment. Inline: false Jun 22 00:30:55.303762479 (80h7m30.867085468s) [upload=64142]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 22 00:30:55.303768627 (80h7m30.867091636s) [upload=64142]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 22 00:30:55.349058140 (80h7m30.912381168s) [seg_pos={0 0}, upload=64142]: Begin upload segment... Jun 22 00:30:55.349113768 (80h7m30.912436772s) [seg_pos={0 0}, upload=64142]: Done begin upload segment. Jun 22 00:30:55.349143121 (80h7m30.912466131s) [noise=true , address=45.91.116.5:5310 , piece_id=AXJOKDZRPVNL7HFT4QKJWRAHFM2UCM3GEVQB2KNYD6CXB6TH4D4Q, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.349660917 (80h7m30.912983928s) [noise=true , address=77.90.31.10:4533 , piece_id=SE6YAKKFDL3BV677CDDUC32UNMPXJHTG23GWWYK5B4YIPKER6GAQ, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.350093077 (80h7m30.913416089s) [noise=true , address=79.127.201.215:4326 , piece_id=USEWZDX6P52U6MCALTBNQG3SV7N2JRKNU6O23TLS6JEWAS4D7ARA, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.350480894 (80h7m30.913803916s) [noise=true , address=104.36.82.13:6199 , piece_id=YP62N2TO446YR3QRG5YLXECD7TN6SJ4UNNLZSK5H4CMV6JIF4AKA, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.350669998 (80h7m30.913993011s) [noise=true , address=104.36.82.54:4184 , piece_id=IZICVGZFMAYLIJTNUJ2OFLZVNZ6IPPYDZO4BSTJJPLHVCVOE5EAA, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.351201352 (80h7m30.914524366s) [noise=true , address=104.36.82.55:4224 , piece_id=XF7H2BYSNNNZOS2T77VIN75N76AYUXAQEI6D4LRXACU2KU62V2TA, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.351668071 (80h7m30.914991083s) [noise=true , address=104.36.82.56:4261 , piece_id=EYOYOYLDACGMPGJYX6ILQT3XHMUTQNXZEDC7ANHLZXIKYVDQFFVQ, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.352193126 (80h7m30.915516138s) [noise=true , address=104.36.82.52:4089 , piece_id=X2VG4S3UHVNWA4AQJS7FLKWNVLTOZ3AKWI5ZDH24QCA3ENQW7BXQ, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.352587096 (80h7m30.915910103s) [noise=true , address=79.127.220.98:4402 , piece_id=TE2WQO5LK45AVMPIAIGVSIBOVZLC4FG4ESQ72X6KFYC7IAEN2DRA, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.353420253 (80h7m30.916743265s) [noise=true , address=104.36.82.11:6125 , piece_id=2CX7RJOKNQA6ODZZZXYPSCYAQY5P35TWX7I56CMYAPFLJE3JEO5Q, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.353791565 (80h7m30.917114578s) [noise=true , address=104.36.82.42:6803 , piece_id=R626UFD326S2UBJEZ2WMHVR7EN4SK427VXPDJZOOZ32MX25JWZWQ, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.354171226 (80h7m30.917494233s) [noise=true , address=104.36.82.14:6245 , piece_id=II36G46OA6TINLR244U6D7Q6LGV754UA5M62K3EX7ZOBESMJUEHA, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.354542282 (80h7m30.917865288s) [noise=true , address=79.127.205.235:5660 , piece_id=UGFPVS7VNB7U2YSFPSGYDIRZV5R2DHCN57PJP6VOXO52WIAYHLXA, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.354891655 (80h7m30.918214678s) [noise=true , address=104.36.82.15:6283 , piece_id=N7TH2EMIOPATVHCSNH2WXUGU7KMMEJ5LCHR22SWEHSA5OV4CUCXQ, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.355499503 (80h7m30.918822519s) [noise=true , address=77.90.31.14:4666 , piece_id=H4DTFDAVCN266PB235BJIQTQ32FBXTIB5AZSX3B4R2VIK554AFSQ, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.356575547 (80h7m30.919898539s) [noise=true , address=104.36.82.39:6756 , piece_id=OKKWMDJPZO6D5VRMGOE3GOCYGHM725JZTNOVYAWKJW3WYZPI4V5Q, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.357039590 (80h7m30.920362596s) [noise=true , address=104.36.82.53:4146 , piece_id=3V3DVGUOPJKDWIGQ2TU7M6VFYFVAS6ODY62ERGSU6LI3LSTWQ4YQ, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.360343566 (80h7m30.923666582s) [noise=true , address=104.36.82.9:6033 , piece_id=T5XA23LZELMSTWOLBSHOX4AHKDACF65OYCKTMZYTXDIBSXNA2T4A, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.452444064 (80h7m31.015767055s) [upload=64142]: Next returned nil segment Jun 22 00:30:55.452495148 (80h7m31.015818149s) [upload=64142]: Waiting for error group managing segments... Jun 22 00:30:55.532958808 (80h7m31.096281883s) [noise=true , address=104.36.82.9:6033 , piece_id=T5XA23LZELMSTWOLBSHOX4AHKDACF65OYCKTMZYTXDIBSXNA2T4A, seg_pos={0 0}, upload=64142]: Done uploading piece. err: ecclient: upload failed (node:1SV3LrcQW16wLewDzfyUWFB6sUgnERNJVdhz72CJmyde95M79C, address:104.36.82.9:6033): protocol: storage node overloaded, request limit: 2000 Jun 22 00:30:55.547802184 (80h7m31.111125188s) [noise=true , address=79.127.220.98:4402 , piece_id=TE2WQO5LK45AVMPIAIGVSIBOVZLC4FG4ESQ72X6KFYC7IAEN2DRA, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:30:55.554223621 (80h7m31.117546616s) [noise=true , address=77.90.31.14:4666 , piece_id=H4DTFDAVCN266PB235BJIQTQ32FBXTIB5AZSX3B4R2VIK554AFSQ, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:30:55.762514255 (80h7m31.325837245s) [noise=true , address=104.36.82.14:6245 , piece_id=II36G46OA6TINLR244U6D7Q6LGV754UA5M62K3EX7ZOBESMJUEHA, seg_pos={0 0}, upload=64142]: Done uploading piece. err: ecclient: upload failed (node:1Zu8teNPTXn638grHKwtJqz4UEY7oFYLx1DHvuVP6RZXcatVqX, address:104.36.82.14:6245): protocol: storage node overloaded, request limit: 2000 Jun 22 00:30:55.773669583 (80h7m31.3369926s) [noise=true , address=79.127.205.235:5660 , piece_id=UGFPVS7VNB7U2YSFPSGYDIRZV5R2DHCN57PJP6VOXO52WIAYHLXA, seg_pos={0 0}, upload=64142]: Done uploading piece. err: ecclient: upload failed (node:12E6uCX9KL4Ud7YpxMXepvMXuqTFJCiSXByDjW94Am3U9H2Nf7f, address:79.127.205.235:5660): protocol: storage node overloaded, request limit: 2000 Jun 22 00:30:56.968366655 (80h7m32.53168964s) [noise=true , address=77.90.31.10:4533 , piece_id=SE6YAKKFDL3BV677CDDUC32UNMPXJHTG23GWWYK5B4YIPKER6GAQ, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:30:57.403572874 (80h7m32.966895883s) [noise=true , address=45.91.116.5:5310 , piece_id=AXJOKDZRPVNL7HFT4QKJWRAHFM2UCM3GEVQB2KNYD6CXB6TH4D4Q, seg_pos={0 0}, upload=64142]: Done uploading piece. err: ecclient: upload failed (node:127RMUc62xRq9RU5z3XfkZsq5sehmv3kP8o53tr9BxVcPE3GQCY, address:45.91.116.5:5310): protocol: storage node overloaded, request limit: 2000 Jun 22 00:30:58.210389994 (80h7m33.773712995s) [noise=true , address=104.36.82.13:6199 , piece_id=YP62N2TO446YR3QRG5YLXECD7TN6SJ4UNNLZSK5H4CMV6JIF4AKA, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:30:58.693037961 (80h7m34.256360966s) [noise=true , address=104.36.82.39:6756 , piece_id=OKKWMDJPZO6D5VRMGOE3GOCYGHM725JZTNOVYAWKJW3WYZPI4V5Q, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:30:59.124834814 (80h7m34.688157819s) [noise=true , address=104.36.82.15:6283 , piece_id=N7TH2EMIOPATVHCSNH2WXUGU7KMMEJ5LCHR22SWEHSA5OV4CUCXQ, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:31:00.629657982 (80h7m36.192980977s) [noise=true , address=104.36.82.42:6803 , piece_id=R626UFD326S2UBJEZ2WMHVR7EN4SK427VXPDJZOOZ32MX25JWZWQ, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:31:00.933110546 (80h7m36.496433536s) [noise=true , address=79.127.201.215:4326 , piece_id=USEWZDX6P52U6MCALTBNQG3SV7N2JRKNU6O23TLS6JEWAS4D7ARA, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:31:05.669167243 (80h7m41.232490237s) [noise=true , address=104.36.82.56:4261 , piece_id=EYOYOYLDACGMPGJYX6ILQT3XHMUTQNXZEDC7ANHLZXIKYVDQFFVQ, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:31:08.443582763 (80h7m44.006905763s) [noise=true , address=104.36.82.54:4184 , piece_id=IZICVGZFMAYLIJTNUJ2OFLZVNZ6IPPYDZO4BSTJJPLHVCVOE5EAA, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:31:26.459138014 (80h8m2.022461013s) [noise=true , address=104.36.82.53:4146 , piece_id=3V3DVGUOPJKDWIGQ2TU7M6VFYFVAS6ODY62ERGSU6LI3LSTWQ4YQ, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:31:27.456050336 (80h8m3.019373336s) [noise=true , address=104.36.82.52:4089 , piece_id=X2VG4S3UHVNWA4AQJS7FLKWNVLTOZ3AKWI5ZDH24QCA3ENQW7BXQ, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:31:55.270234251 (80h8m30.833557256s) [noise=true , address=104.36.82.11:6125 , piece_id=2CX7RJOKNQA6ODZZZXYPSCYAQY5P35TWX7I56CMYAPFLJE3JEO5Q, seg_pos={0 0}, upload=64142]: Done uploading piece. err: ecclient: upload failed (node:12vricofwx67FRaxAGLbmgT1TYyKkqfJbxdsyeEPL4R7kg8pKzT, address:104.36.82.11:6125): protocol: expected piece hash; context deadline exceeded; EOF Jun 22 00:31:55.270319418 (80h8m30.833642407s) [noise=true , address=104.36.82.55:4224 , piece_id=XF7H2BYSNNNZOS2T77VIN75N76AYUXAQEI6D4LRXACU2KU62V2TA, seg_pos={0 0}, upload=64142]: Done uploading piece. err: ecclient: upload failed (node:12EjKmUiv4pxkVCB4WoiXim4uFpxrBPyfiSS4efTY3HKSE3jb5y, address:104.36.82.55:4224): protocol: write tcp 159.223.143.39:39152->104.36.82.55:4224: use of closed network connection; write tcp 159.223.143.39:39152->104.36.82.55:4224: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:39152->104.36.82.55:4224: use of closed network connection Jun 22 00:31:55.270353811 (80h8m30.833676799s) [seg_pos={0 0}, upload=64142]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Jun 22 00:31:55.301099877 (80h8m30.864422884s) [upload=64142]: Done uploading segments... Jun 22 00:31:55.301121567 (80h8m30.864444566s) [upload=64142]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 21.530197756s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [..................X.........................................] duration: 28.427178296s FAIL (2025-06-22 01:04:49 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== ing upload Jun 22 00:30:55.303646184 (80h7m30.866969171s) [upload=64142]: Uploading segments... Jun 22 00:30:55.303706406 (80h7m30.867029416s) [upload=64142]: Got next segment. Inline: false Jun 22 00:30:55.303762479 (80h7m30.867085468s) [upload=64142]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 22 00:30:55.303768627 (80h7m30.867091636s) [upload=64142]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 22 00:30:55.349058140 (80h7m30.912381168s) [seg_pos={0 0}, upload=64142]: Begin upload segment... Jun 22 00:30:55.349113768 (80h7m30.912436772s) [seg_pos={0 0}, upload=64142]: Done begin upload segment. Jun 22 00:30:55.349143121 (80h7m30.912466131s) [noise=true , address=45.91.116.5:5310 , piece_id=AXJOKDZRPVNL7HFT4QKJWRAHFM2UCM3GEVQB2KNYD6CXB6TH4D4Q, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.349660917 (80h7m30.912983928s) [noise=true , address=77.90.31.10:4533 , piece_id=SE6YAKKFDL3BV677CDDUC32UNMPXJHTG23GWWYK5B4YIPKER6GAQ, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.350093077 (80h7m30.913416089s) [noise=true , address=79.127.201.215:4326 , piece_id=USEWZDX6P52U6MCALTBNQG3SV7N2JRKNU6O23TLS6JEWAS4D7ARA, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.350480894 (80h7m30.913803916s) [noise=true , address=104.36.82.13:6199 , piece_id=YP62N2TO446YR3QRG5YLXECD7TN6SJ4UNNLZSK5H4CMV6JIF4AKA, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.350669998 (80h7m30.913993011s) [noise=true , address=104.36.82.54:4184 , piece_id=IZICVGZFMAYLIJTNUJ2OFLZVNZ6IPPYDZO4BSTJJPLHVCVOE5EAA, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.351201352 (80h7m30.914524366s) [noise=true , address=104.36.82.55:4224 , piece_id=XF7H2BYSNNNZOS2T77VIN75N76AYUXAQEI6D4LRXACU2KU62V2TA, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.351668071 (80h7m30.914991083s) [noise=true , address=104.36.82.56:4261 , piece_id=EYOYOYLDACGMPGJYX6ILQT3XHMUTQNXZEDC7ANHLZXIKYVDQFFVQ, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.352193126 (80h7m30.915516138s) [noise=true , address=104.36.82.52:4089 , piece_id=X2VG4S3UHVNWA4AQJS7FLKWNVLTOZ3AKWI5ZDH24QCA3ENQW7BXQ, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.352587096 (80h7m30.915910103s) [noise=true , address=79.127.220.98:4402 , piece_id=TE2WQO5LK45AVMPIAIGVSIBOVZLC4FG4ESQ72X6KFYC7IAEN2DRA, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.353420253 (80h7m30.916743265s) [noise=true , address=104.36.82.11:6125 , piece_id=2CX7RJOKNQA6ODZZZXYPSCYAQY5P35TWX7I56CMYAPFLJE3JEO5Q, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.353791565 (80h7m30.917114578s) [noise=true , address=104.36.82.42:6803 , piece_id=R626UFD326S2UBJEZ2WMHVR7EN4SK427VXPDJZOOZ32MX25JWZWQ, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.354171226 (80h7m30.917494233s) [noise=true , address=104.36.82.14:6245 , piece_id=II36G46OA6TINLR244U6D7Q6LGV754UA5M62K3EX7ZOBESMJUEHA, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.354542282 (80h7m30.917865288s) [noise=true , address=79.127.205.235:5660 , piece_id=UGFPVS7VNB7U2YSFPSGYDIRZV5R2DHCN57PJP6VOXO52WIAYHLXA, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.354891655 (80h7m30.918214678s) [noise=true , address=104.36.82.15:6283 , piece_id=N7TH2EMIOPATVHCSNH2WXUGU7KMMEJ5LCHR22SWEHSA5OV4CUCXQ, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.355499503 (80h7m30.918822519s) [noise=true , address=77.90.31.14:4666 , piece_id=H4DTFDAVCN266PB235BJIQTQ32FBXTIB5AZSX3B4R2VIK554AFSQ, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.356575547 (80h7m30.919898539s) [noise=true , address=104.36.82.39:6756 , piece_id=OKKWMDJPZO6D5VRMGOE3GOCYGHM725JZTNOVYAWKJW3WYZPI4V5Q, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.357039590 (80h7m30.920362596s) [noise=true , address=104.36.82.53:4146 , piece_id=3V3DVGUOPJKDWIGQ2TU7M6VFYFVAS6ODY62ERGSU6LI3LSTWQ4YQ, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.360343566 (80h7m30.923666582s) [noise=true , address=104.36.82.9:6033 , piece_id=T5XA23LZELMSTWOLBSHOX4AHKDACF65OYCKTMZYTXDIBSXNA2T4A, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.452444064 (80h7m31.015767055s) [upload=64142]: Next returned nil segment Jun 22 00:30:55.452495148 (80h7m31.015818149s) [upload=64142]: Waiting for error group managing segments... Jun 22 00:30:55.532958808 (80h7m31.096281883s) [noise=true , address=104.36.82.9:6033 , piece_id=T5XA23LZELMSTWOLBSHOX4AHKDACF65OYCKTMZYTXDIBSXNA2T4A, seg_pos={0 0}, upload=64142]: Done uploading piece. err: ecclient: upload failed (node:1SV3LrcQW16wLewDzfyUWFB6sUgnERNJVdhz72CJmyde95M79C, address:104.36.82.9:6033): protocol: storage node overloaded, request limit: 2000 Jun 22 00:30:55.547802184 (80h7m31.111125188s) [noise=true , address=79.127.220.98:4402 , piece_id=TE2WQO5LK45AVMPIAIGVSIBOVZLC4FG4ESQ72X6KFYC7IAEN2DRA, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:30:55.554223621 (80h7m31.117546616s) [noise=true , address=77.90.31.14:4666 , piece_id=H4DTFDAVCN266PB235BJIQTQ32FBXTIB5AZSX3B4R2VIK554AFSQ, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:30:55.762514255 (80h7m31.325837245s) [noise=true , address=104.36.82.14:6245 , piece_id=II36G46OA6TINLR244U6D7Q6LGV754UA5M62K3EX7ZOBESMJUEHA, seg_pos={0 0}, upload=64142]: Done uploading piece. err: ecclient: upload failed (node:1Zu8teNPTXn638grHKwtJqz4UEY7oFYLx1DHvuVP6RZXcatVqX, address:104.36.82.14:6245): protocol: storage node overloaded, request limit: 2000 Jun 22 00:30:55.773669583 (80h7m31.3369926s) [noise=true , address=79.127.205.235:5660 , piece_id=UGFPVS7VNB7U2YSFPSGYDIRZV5R2DHCN57PJP6VOXO52WIAYHLXA, seg_pos={0 0}, upload=64142]: Done uploading piece. err: ecclient: upload failed (node:12E6uCX9KL4Ud7YpxMXepvMXuqTFJCiSXByDjW94Am3U9H2Nf7f, address:79.127.205.235:5660): protocol: storage node overloaded, request limit: 2000 Jun 22 00:30:56.968366655 (80h7m32.53168964s) [noise=true , address=77.90.31.10:4533 , piece_id=SE6YAKKFDL3BV677CDDUC32UNMPXJHTG23GWWYK5B4YIPKER6GAQ, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:30:57.403572874 (80h7m32.966895883s) [noise=true , address=45.91.116.5:5310 , piece_id=AXJOKDZRPVNL7HFT4QKJWRAHFM2UCM3GEVQB2KNYD6CXB6TH4D4Q, seg_pos={0 0}, upload=64142]: Done uploading piece. err: ecclient: upload failed (node:127RMUc62xRq9RU5z3XfkZsq5sehmv3kP8o53tr9BxVcPE3GQCY, address:45.91.116.5:5310): protocol: storage node overloaded, request limit: 2000 Jun 22 00:30:58.210389994 (80h7m33.773712995s) [noise=true , address=104.36.82.13:6199 , piece_id=YP62N2TO446YR3QRG5YLXECD7TN6SJ4UNNLZSK5H4CMV6JIF4AKA, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:30:58.693037961 (80h7m34.256360966s) [noise=true , address=104.36.82.39:6756 , piece_id=OKKWMDJPZO6D5VRMGOE3GOCYGHM725JZTNOVYAWKJW3WYZPI4V5Q, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:30:59.124834814 (80h7m34.688157819s) [noise=true , address=104.36.82.15:6283 , piece_id=N7TH2EMIOPATVHCSNH2WXUGU7KMMEJ5LCHR22SWEHSA5OV4CUCXQ, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:31:00.629657982 (80h7m36.192980977s) [noise=true , address=104.36.82.42:6803 , piece_id=R626UFD326S2UBJEZ2WMHVR7EN4SK427VXPDJZOOZ32MX25JWZWQ, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:31:00.933110546 (80h7m36.496433536s) [noise=true , address=79.127.201.215:4326 , piece_id=USEWZDX6P52U6MCALTBNQG3SV7N2JRKNU6O23TLS6JEWAS4D7ARA, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:31:05.669167243 (80h7m41.232490237s) [noise=true , address=104.36.82.56:4261 , piece_id=EYOYOYLDACGMPGJYX6ILQT3XHMUTQNXZEDC7ANHLZXIKYVDQFFVQ, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:31:08.443582763 (80h7m44.006905763s) [noise=true , address=104.36.82.54:4184 , piece_id=IZICVGZFMAYLIJTNUJ2OFLZVNZ6IPPYDZO4BSTJJPLHVCVOE5EAA, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:31:26.459138014 (80h8m2.022461013s) [noise=true , address=104.36.82.53:4146 , piece_id=3V3DVGUOPJKDWIGQ2TU7M6VFYFVAS6ODY62ERGSU6LI3LSTWQ4YQ, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:31:27.456050336 (80h8m3.019373336s) [noise=true , address=104.36.82.52:4089 , piece_id=X2VG4S3UHVNWA4AQJS7FLKWNVLTOZ3AKWI5ZDH24QCA3ENQW7BXQ, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:31:55.270234251 (80h8m30.833557256s) [noise=true , address=104.36.82.11:6125 , piece_id=2CX7RJOKNQA6ODZZZXYPSCYAQY5P35TWX7I56CMYAPFLJE3JEO5Q, seg_pos={0 0}, upload=64142]: Done uploading piece. err: ecclient: upload failed (node:12vricofwx67FRaxAGLbmgT1TYyKkqfJbxdsyeEPL4R7kg8pKzT, address:104.36.82.11:6125): protocol: expected piece hash; context deadline exceeded; EOF Jun 22 00:31:55.270319418 (80h8m30.833642407s) [noise=true , address=104.36.82.55:4224 , piece_id=XF7H2BYSNNNZOS2T77VIN75N76AYUXAQEI6D4LRXACU2KU62V2TA, seg_pos={0 0}, upload=64142]: Done uploading piece. err: ecclient: upload failed (node:12EjKmUiv4pxkVCB4WoiXim4uFpxrBPyfiSS4efTY3HKSE3jb5y, address:104.36.82.55:4224): protocol: write tcp 159.223.143.39:39152->104.36.82.55:4224: use of closed network connection; write tcp 159.223.143.39:39152->104.36.82.55:4224: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:39152->104.36.82.55:4224: use of closed network connection Jun 22 00:31:55.270353811 (80h8m30.833676799s) [seg_pos={0 0}, upload=64142]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Jun 22 00:31:55.301099877 (80h8m30.864422884s) [upload=64142]: Done uploading segments... Jun 22 00:31:55.301121567 (80h8m30.864444566s) [upload=64142]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 28.427205843s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [..................X.........................................] duration: 15.99986869s FAIL (2025-06-22 01:04:39 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 15.99989024s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [...................X........................................] duration: 14.447622338s FAIL (2025-06-22 01:04:20 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 14.447641045s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [...................X........................................] duration: 42.006901021s FAIL (2025-06-22 01:03:57 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== ing upload Jun 22 00:30:55.303646184 (80h7m30.866969171s) [upload=64142]: Uploading segments... Jun 22 00:30:55.303706406 (80h7m30.867029416s) [upload=64142]: Got next segment. Inline: false Jun 22 00:30:55.303762479 (80h7m30.867085468s) [upload=64142]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 22 00:30:55.303768627 (80h7m30.867091636s) [upload=64142]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 22 00:30:55.349058140 (80h7m30.912381168s) [seg_pos={0 0}, upload=64142]: Begin upload segment... Jun 22 00:30:55.349113768 (80h7m30.912436772s) [seg_pos={0 0}, upload=64142]: Done begin upload segment. Jun 22 00:30:55.349143121 (80h7m30.912466131s) [noise=true , address=45.91.116.5:5310 , piece_id=AXJOKDZRPVNL7HFT4QKJWRAHFM2UCM3GEVQB2KNYD6CXB6TH4D4Q, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.349660917 (80h7m30.912983928s) [noise=true , address=77.90.31.10:4533 , piece_id=SE6YAKKFDL3BV677CDDUC32UNMPXJHTG23GWWYK5B4YIPKER6GAQ, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.350093077 (80h7m30.913416089s) [noise=true , address=79.127.201.215:4326 , piece_id=USEWZDX6P52U6MCALTBNQG3SV7N2JRKNU6O23TLS6JEWAS4D7ARA, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.350480894 (80h7m30.913803916s) [noise=true , address=104.36.82.13:6199 , piece_id=YP62N2TO446YR3QRG5YLXECD7TN6SJ4UNNLZSK5H4CMV6JIF4AKA, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.350669998 (80h7m30.913993011s) [noise=true , address=104.36.82.54:4184 , piece_id=IZICVGZFMAYLIJTNUJ2OFLZVNZ6IPPYDZO4BSTJJPLHVCVOE5EAA, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.351201352 (80h7m30.914524366s) [noise=true , address=104.36.82.55:4224 , piece_id=XF7H2BYSNNNZOS2T77VIN75N76AYUXAQEI6D4LRXACU2KU62V2TA, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.351668071 (80h7m30.914991083s) [noise=true , address=104.36.82.56:4261 , piece_id=EYOYOYLDACGMPGJYX6ILQT3XHMUTQNXZEDC7ANHLZXIKYVDQFFVQ, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.352193126 (80h7m30.915516138s) [noise=true , address=104.36.82.52:4089 , piece_id=X2VG4S3UHVNWA4AQJS7FLKWNVLTOZ3AKWI5ZDH24QCA3ENQW7BXQ, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.352587096 (80h7m30.915910103s) [noise=true , address=79.127.220.98:4402 , piece_id=TE2WQO5LK45AVMPIAIGVSIBOVZLC4FG4ESQ72X6KFYC7IAEN2DRA, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.353420253 (80h7m30.916743265s) [noise=true , address=104.36.82.11:6125 , piece_id=2CX7RJOKNQA6ODZZZXYPSCYAQY5P35TWX7I56CMYAPFLJE3JEO5Q, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.353791565 (80h7m30.917114578s) [noise=true , address=104.36.82.42:6803 , piece_id=R626UFD326S2UBJEZ2WMHVR7EN4SK427VXPDJZOOZ32MX25JWZWQ, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.354171226 (80h7m30.917494233s) [noise=true , address=104.36.82.14:6245 , piece_id=II36G46OA6TINLR244U6D7Q6LGV754UA5M62K3EX7ZOBESMJUEHA, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.354542282 (80h7m30.917865288s) [noise=true , address=79.127.205.235:5660 , piece_id=UGFPVS7VNB7U2YSFPSGYDIRZV5R2DHCN57PJP6VOXO52WIAYHLXA, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.354891655 (80h7m30.918214678s) [noise=true , address=104.36.82.15:6283 , piece_id=N7TH2EMIOPATVHCSNH2WXUGU7KMMEJ5LCHR22SWEHSA5OV4CUCXQ, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.355499503 (80h7m30.918822519s) [noise=true , address=77.90.31.14:4666 , piece_id=H4DTFDAVCN266PB235BJIQTQ32FBXTIB5AZSX3B4R2VIK554AFSQ, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.356575547 (80h7m30.919898539s) [noise=true , address=104.36.82.39:6756 , piece_id=OKKWMDJPZO6D5VRMGOE3GOCYGHM725JZTNOVYAWKJW3WYZPI4V5Q, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.357039590 (80h7m30.920362596s) [noise=true , address=104.36.82.53:4146 , piece_id=3V3DVGUOPJKDWIGQ2TU7M6VFYFVAS6ODY62ERGSU6LI3LSTWQ4YQ, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.360343566 (80h7m30.923666582s) [noise=true , address=104.36.82.9:6033 , piece_id=T5XA23LZELMSTWOLBSHOX4AHKDACF65OYCKTMZYTXDIBSXNA2T4A, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.452444064 (80h7m31.015767055s) [upload=64142]: Next returned nil segment Jun 22 00:30:55.452495148 (80h7m31.015818149s) [upload=64142]: Waiting for error group managing segments... Jun 22 00:30:55.532958808 (80h7m31.096281883s) [noise=true , address=104.36.82.9:6033 , piece_id=T5XA23LZELMSTWOLBSHOX4AHKDACF65OYCKTMZYTXDIBSXNA2T4A, seg_pos={0 0}, upload=64142]: Done uploading piece. err: ecclient: upload failed (node:1SV3LrcQW16wLewDzfyUWFB6sUgnERNJVdhz72CJmyde95M79C, address:104.36.82.9:6033): protocol: storage node overloaded, request limit: 2000 Jun 22 00:30:55.547802184 (80h7m31.111125188s) [noise=true , address=79.127.220.98:4402 , piece_id=TE2WQO5LK45AVMPIAIGVSIBOVZLC4FG4ESQ72X6KFYC7IAEN2DRA, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:30:55.554223621 (80h7m31.117546616s) [noise=true , address=77.90.31.14:4666 , piece_id=H4DTFDAVCN266PB235BJIQTQ32FBXTIB5AZSX3B4R2VIK554AFSQ, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:30:55.762514255 (80h7m31.325837245s) [noise=true , address=104.36.82.14:6245 , piece_id=II36G46OA6TINLR244U6D7Q6LGV754UA5M62K3EX7ZOBESMJUEHA, seg_pos={0 0}, upload=64142]: Done uploading piece. err: ecclient: upload failed (node:1Zu8teNPTXn638grHKwtJqz4UEY7oFYLx1DHvuVP6RZXcatVqX, address:104.36.82.14:6245): protocol: storage node overloaded, request limit: 2000 Jun 22 00:30:55.773669583 (80h7m31.3369926s) [noise=true , address=79.127.205.235:5660 , piece_id=UGFPVS7VNB7U2YSFPSGYDIRZV5R2DHCN57PJP6VOXO52WIAYHLXA, seg_pos={0 0}, upload=64142]: Done uploading piece. err: ecclient: upload failed (node:12E6uCX9KL4Ud7YpxMXepvMXuqTFJCiSXByDjW94Am3U9H2Nf7f, address:79.127.205.235:5660): protocol: storage node overloaded, request limit: 2000 Jun 22 00:30:56.968366655 (80h7m32.53168964s) [noise=true , address=77.90.31.10:4533 , piece_id=SE6YAKKFDL3BV677CDDUC32UNMPXJHTG23GWWYK5B4YIPKER6GAQ, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:30:57.403572874 (80h7m32.966895883s) [noise=true , address=45.91.116.5:5310 , piece_id=AXJOKDZRPVNL7HFT4QKJWRAHFM2UCM3GEVQB2KNYD6CXB6TH4D4Q, seg_pos={0 0}, upload=64142]: Done uploading piece. err: ecclient: upload failed (node:127RMUc62xRq9RU5z3XfkZsq5sehmv3kP8o53tr9BxVcPE3GQCY, address:45.91.116.5:5310): protocol: storage node overloaded, request limit: 2000 Jun 22 00:30:58.210389994 (80h7m33.773712995s) [noise=true , address=104.36.82.13:6199 , piece_id=YP62N2TO446YR3QRG5YLXECD7TN6SJ4UNNLZSK5H4CMV6JIF4AKA, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:30:58.693037961 (80h7m34.256360966s) [noise=true , address=104.36.82.39:6756 , piece_id=OKKWMDJPZO6D5VRMGOE3GOCYGHM725JZTNOVYAWKJW3WYZPI4V5Q, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:30:59.124834814 (80h7m34.688157819s) [noise=true , address=104.36.82.15:6283 , piece_id=N7TH2EMIOPATVHCSNH2WXUGU7KMMEJ5LCHR22SWEHSA5OV4CUCXQ, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:31:00.629657982 (80h7m36.192980977s) [noise=true , address=104.36.82.42:6803 , piece_id=R626UFD326S2UBJEZ2WMHVR7EN4SK427VXPDJZOOZ32MX25JWZWQ, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:31:00.933110546 (80h7m36.496433536s) [noise=true , address=79.127.201.215:4326 , piece_id=USEWZDX6P52U6MCALTBNQG3SV7N2JRKNU6O23TLS6JEWAS4D7ARA, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:31:05.669167243 (80h7m41.232490237s) [noise=true , address=104.36.82.56:4261 , piece_id=EYOYOYLDACGMPGJYX6ILQT3XHMUTQNXZEDC7ANHLZXIKYVDQFFVQ, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:31:08.443582763 (80h7m44.006905763s) [noise=true , address=104.36.82.54:4184 , piece_id=IZICVGZFMAYLIJTNUJ2OFLZVNZ6IPPYDZO4BSTJJPLHVCVOE5EAA, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:31:26.459138014 (80h8m2.022461013s) [noise=true , address=104.36.82.53:4146 , piece_id=3V3DVGUOPJKDWIGQ2TU7M6VFYFVAS6ODY62ERGSU6LI3LSTWQ4YQ, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:31:27.456050336 (80h8m3.019373336s) [noise=true , address=104.36.82.52:4089 , piece_id=X2VG4S3UHVNWA4AQJS7FLKWNVLTOZ3AKWI5ZDH24QCA3ENQW7BXQ, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:31:55.270234251 (80h8m30.833557256s) [noise=true , address=104.36.82.11:6125 , piece_id=2CX7RJOKNQA6ODZZZXYPSCYAQY5P35TWX7I56CMYAPFLJE3JEO5Q, seg_pos={0 0}, upload=64142]: Done uploading piece. err: ecclient: upload failed (node:12vricofwx67FRaxAGLbmgT1TYyKkqfJbxdsyeEPL4R7kg8pKzT, address:104.36.82.11:6125): protocol: expected piece hash; context deadline exceeded; EOF Jun 22 00:31:55.270319418 (80h8m30.833642407s) [noise=true , address=104.36.82.55:4224 , piece_id=XF7H2BYSNNNZOS2T77VIN75N76AYUXAQEI6D4LRXACU2KU62V2TA, seg_pos={0 0}, upload=64142]: Done uploading piece. err: ecclient: upload failed (node:12EjKmUiv4pxkVCB4WoiXim4uFpxrBPyfiSS4efTY3HKSE3jb5y, address:104.36.82.55:4224): protocol: write tcp 159.223.143.39:39152->104.36.82.55:4224: use of closed network connection; write tcp 159.223.143.39:39152->104.36.82.55:4224: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:39152->104.36.82.55:4224: use of closed network connection Jun 22 00:31:55.270353811 (80h8m30.833676799s) [seg_pos={0 0}, upload=64142]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Jun 22 00:31:55.301099877 (80h8m30.864422884s) [upload=64142]: Done uploading segments... Jun 22 00:31:55.301121567 (80h8m30.864444566s) [upload=64142]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 42.006941682s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [X...................X.......................................] duration: 31.032676087s FAIL (2025-06-22 01:03:57 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 31.032722766s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [.X...................X......................................] duration: 15.324687476s FAIL (2025-06-22 01:03:03 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 15.324710636s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [....................X.......................................] duration: 24.902522025s FAIL (2025-06-22 01:02:35 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== ing upload Jun 22 00:30:55.303646184 (80h7m30.866969171s) [upload=64142]: Uploading segments... Jun 22 00:30:55.303706406 (80h7m30.867029416s) [upload=64142]: Got next segment. Inline: false Jun 22 00:30:55.303762479 (80h7m30.867085468s) [upload=64142]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 22 00:30:55.303768627 (80h7m30.867091636s) [upload=64142]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 22 00:30:55.349058140 (80h7m30.912381168s) [seg_pos={0 0}, upload=64142]: Begin upload segment... Jun 22 00:30:55.349113768 (80h7m30.912436772s) [seg_pos={0 0}, upload=64142]: Done begin upload segment. Jun 22 00:30:55.349143121 (80h7m30.912466131s) [noise=true , address=45.91.116.5:5310 , piece_id=AXJOKDZRPVNL7HFT4QKJWRAHFM2UCM3GEVQB2KNYD6CXB6TH4D4Q, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.349660917 (80h7m30.912983928s) [noise=true , address=77.90.31.10:4533 , piece_id=SE6YAKKFDL3BV677CDDUC32UNMPXJHTG23GWWYK5B4YIPKER6GAQ, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.350093077 (80h7m30.913416089s) [noise=true , address=79.127.201.215:4326 , piece_id=USEWZDX6P52U6MCALTBNQG3SV7N2JRKNU6O23TLS6JEWAS4D7ARA, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.350480894 (80h7m30.913803916s) [noise=true , address=104.36.82.13:6199 , piece_id=YP62N2TO446YR3QRG5YLXECD7TN6SJ4UNNLZSK5H4CMV6JIF4AKA, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.350669998 (80h7m30.913993011s) [noise=true , address=104.36.82.54:4184 , piece_id=IZICVGZFMAYLIJTNUJ2OFLZVNZ6IPPYDZO4BSTJJPLHVCVOE5EAA, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.351201352 (80h7m30.914524366s) [noise=true , address=104.36.82.55:4224 , piece_id=XF7H2BYSNNNZOS2T77VIN75N76AYUXAQEI6D4LRXACU2KU62V2TA, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.351668071 (80h7m30.914991083s) [noise=true , address=104.36.82.56:4261 , piece_id=EYOYOYLDACGMPGJYX6ILQT3XHMUTQNXZEDC7ANHLZXIKYVDQFFVQ, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.352193126 (80h7m30.915516138s) [noise=true , address=104.36.82.52:4089 , piece_id=X2VG4S3UHVNWA4AQJS7FLKWNVLTOZ3AKWI5ZDH24QCA3ENQW7BXQ, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.352587096 (80h7m30.915910103s) [noise=true , address=79.127.220.98:4402 , piece_id=TE2WQO5LK45AVMPIAIGVSIBOVZLC4FG4ESQ72X6KFYC7IAEN2DRA, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.353420253 (80h7m30.916743265s) [noise=true , address=104.36.82.11:6125 , piece_id=2CX7RJOKNQA6ODZZZXYPSCYAQY5P35TWX7I56CMYAPFLJE3JEO5Q, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.353791565 (80h7m30.917114578s) [noise=true , address=104.36.82.42:6803 , piece_id=R626UFD326S2UBJEZ2WMHVR7EN4SK427VXPDJZOOZ32MX25JWZWQ, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.354171226 (80h7m30.917494233s) [noise=true , address=104.36.82.14:6245 , piece_id=II36G46OA6TINLR244U6D7Q6LGV754UA5M62K3EX7ZOBESMJUEHA, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.354542282 (80h7m30.917865288s) [noise=true , address=79.127.205.235:5660 , piece_id=UGFPVS7VNB7U2YSFPSGYDIRZV5R2DHCN57PJP6VOXO52WIAYHLXA, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.354891655 (80h7m30.918214678s) [noise=true , address=104.36.82.15:6283 , piece_id=N7TH2EMIOPATVHCSNH2WXUGU7KMMEJ5LCHR22SWEHSA5OV4CUCXQ, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.355499503 (80h7m30.918822519s) [noise=true , address=77.90.31.14:4666 , piece_id=H4DTFDAVCN266PB235BJIQTQ32FBXTIB5AZSX3B4R2VIK554AFSQ, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.356575547 (80h7m30.919898539s) [noise=true , address=104.36.82.39:6756 , piece_id=OKKWMDJPZO6D5VRMGOE3GOCYGHM725JZTNOVYAWKJW3WYZPI4V5Q, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.357039590 (80h7m30.920362596s) [noise=true , address=104.36.82.53:4146 , piece_id=3V3DVGUOPJKDWIGQ2TU7M6VFYFVAS6ODY62ERGSU6LI3LSTWQ4YQ, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.360343566 (80h7m30.923666582s) [noise=true , address=104.36.82.9:6033 , piece_id=T5XA23LZELMSTWOLBSHOX4AHKDACF65OYCKTMZYTXDIBSXNA2T4A, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.452444064 (80h7m31.015767055s) [upload=64142]: Next returned nil segment Jun 22 00:30:55.452495148 (80h7m31.015818149s) [upload=64142]: Waiting for error group managing segments... Jun 22 00:30:55.532958808 (80h7m31.096281883s) [noise=true , address=104.36.82.9:6033 , piece_id=T5XA23LZELMSTWOLBSHOX4AHKDACF65OYCKTMZYTXDIBSXNA2T4A, seg_pos={0 0}, upload=64142]: Done uploading piece. err: ecclient: upload failed (node:1SV3LrcQW16wLewDzfyUWFB6sUgnERNJVdhz72CJmyde95M79C, address:104.36.82.9:6033): protocol: storage node overloaded, request limit: 2000 Jun 22 00:30:55.547802184 (80h7m31.111125188s) [noise=true , address=79.127.220.98:4402 , piece_id=TE2WQO5LK45AVMPIAIGVSIBOVZLC4FG4ESQ72X6KFYC7IAEN2DRA, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:30:55.554223621 (80h7m31.117546616s) [noise=true , address=77.90.31.14:4666 , piece_id=H4DTFDAVCN266PB235BJIQTQ32FBXTIB5AZSX3B4R2VIK554AFSQ, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:30:55.762514255 (80h7m31.325837245s) [noise=true , address=104.36.82.14:6245 , piece_id=II36G46OA6TINLR244U6D7Q6LGV754UA5M62K3EX7ZOBESMJUEHA, seg_pos={0 0}, upload=64142]: Done uploading piece. err: ecclient: upload failed (node:1Zu8teNPTXn638grHKwtJqz4UEY7oFYLx1DHvuVP6RZXcatVqX, address:104.36.82.14:6245): protocol: storage node overloaded, request limit: 2000 Jun 22 00:30:55.773669583 (80h7m31.3369926s) [noise=true , address=79.127.205.235:5660 , piece_id=UGFPVS7VNB7U2YSFPSGYDIRZV5R2DHCN57PJP6VOXO52WIAYHLXA, seg_pos={0 0}, upload=64142]: Done uploading piece. err: ecclient: upload failed (node:12E6uCX9KL4Ud7YpxMXepvMXuqTFJCiSXByDjW94Am3U9H2Nf7f, address:79.127.205.235:5660): protocol: storage node overloaded, request limit: 2000 Jun 22 00:30:56.968366655 (80h7m32.53168964s) [noise=true , address=77.90.31.10:4533 , piece_id=SE6YAKKFDL3BV677CDDUC32UNMPXJHTG23GWWYK5B4YIPKER6GAQ, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:30:57.403572874 (80h7m32.966895883s) [noise=true , address=45.91.116.5:5310 , piece_id=AXJOKDZRPVNL7HFT4QKJWRAHFM2UCM3GEVQB2KNYD6CXB6TH4D4Q, seg_pos={0 0}, upload=64142]: Done uploading piece. err: ecclient: upload failed (node:127RMUc62xRq9RU5z3XfkZsq5sehmv3kP8o53tr9BxVcPE3GQCY, address:45.91.116.5:5310): protocol: storage node overloaded, request limit: 2000 Jun 22 00:30:58.210389994 (80h7m33.773712995s) [noise=true , address=104.36.82.13:6199 , piece_id=YP62N2TO446YR3QRG5YLXECD7TN6SJ4UNNLZSK5H4CMV6JIF4AKA, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:30:58.693037961 (80h7m34.256360966s) [noise=true , address=104.36.82.39:6756 , piece_id=OKKWMDJPZO6D5VRMGOE3GOCYGHM725JZTNOVYAWKJW3WYZPI4V5Q, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:30:59.124834814 (80h7m34.688157819s) [noise=true , address=104.36.82.15:6283 , piece_id=N7TH2EMIOPATVHCSNH2WXUGU7KMMEJ5LCHR22SWEHSA5OV4CUCXQ, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:31:00.629657982 (80h7m36.192980977s) [noise=true , address=104.36.82.42:6803 , piece_id=R626UFD326S2UBJEZ2WMHVR7EN4SK427VXPDJZOOZ32MX25JWZWQ, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:31:00.933110546 (80h7m36.496433536s) [noise=true , address=79.127.201.215:4326 , piece_id=USEWZDX6P52U6MCALTBNQG3SV7N2JRKNU6O23TLS6JEWAS4D7ARA, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:31:05.669167243 (80h7m41.232490237s) [noise=true , address=104.36.82.56:4261 , piece_id=EYOYOYLDACGMPGJYX6ILQT3XHMUTQNXZEDC7ANHLZXIKYVDQFFVQ, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:31:08.443582763 (80h7m44.006905763s) [noise=true , address=104.36.82.54:4184 , piece_id=IZICVGZFMAYLIJTNUJ2OFLZVNZ6IPPYDZO4BSTJJPLHVCVOE5EAA, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:31:26.459138014 (80h8m2.022461013s) [noise=true , address=104.36.82.53:4146 , piece_id=3V3DVGUOPJKDWIGQ2TU7M6VFYFVAS6ODY62ERGSU6LI3LSTWQ4YQ, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:31:27.456050336 (80h8m3.019373336s) [noise=true , address=104.36.82.52:4089 , piece_id=X2VG4S3UHVNWA4AQJS7FLKWNVLTOZ3AKWI5ZDH24QCA3ENQW7BXQ, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:31:55.270234251 (80h8m30.833557256s) [noise=true , address=104.36.82.11:6125 , piece_id=2CX7RJOKNQA6ODZZZXYPSCYAQY5P35TWX7I56CMYAPFLJE3JEO5Q, seg_pos={0 0}, upload=64142]: Done uploading piece. err: ecclient: upload failed (node:12vricofwx67FRaxAGLbmgT1TYyKkqfJbxdsyeEPL4R7kg8pKzT, address:104.36.82.11:6125): protocol: expected piece hash; context deadline exceeded; EOF Jun 22 00:31:55.270319418 (80h8m30.833642407s) [noise=true , address=104.36.82.55:4224 , piece_id=XF7H2BYSNNNZOS2T77VIN75N76AYUXAQEI6D4LRXACU2KU62V2TA, seg_pos={0 0}, upload=64142]: Done uploading piece. err: ecclient: upload failed (node:12EjKmUiv4pxkVCB4WoiXim4uFpxrBPyfiSS4efTY3HKSE3jb5y, address:104.36.82.55:4224): protocol: write tcp 159.223.143.39:39152->104.36.82.55:4224: use of closed network connection; write tcp 159.223.143.39:39152->104.36.82.55:4224: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:39152->104.36.82.55:4224: use of closed network connection Jun 22 00:31:55.270353811 (80h8m30.833676799s) [seg_pos={0 0}, upload=64142]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Jun 22 00:31:55.301099877 (80h8m30.864422884s) [upload=64142]: Done uploading segments... Jun 22 00:31:55.301121567 (80h8m30.864444566s) [upload=64142]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 24.902549167s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [..X...................X.....................................] duration: 9.237321957s FAIL (2025-06-22 01:02:18 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 9.237352997s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [...X...................X....................................] duration: 25.95253686s FAIL (2025-06-22 01:02:06 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 25.952577396s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.....................X......................................] duration: 19.775796143s FAIL (2025-06-22 01:01:42 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== ing upload Jun 22 00:30:55.303646184 (80h7m30.866969171s) [upload=64142]: Uploading segments... Jun 22 00:30:55.303706406 (80h7m30.867029416s) [upload=64142]: Got next segment. Inline: false Jun 22 00:30:55.303762479 (80h7m30.867085468s) [upload=64142]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 22 00:30:55.303768627 (80h7m30.867091636s) [upload=64142]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 22 00:30:55.349058140 (80h7m30.912381168s) [seg_pos={0 0}, upload=64142]: Begin upload segment... Jun 22 00:30:55.349113768 (80h7m30.912436772s) [seg_pos={0 0}, upload=64142]: Done begin upload segment. Jun 22 00:30:55.349143121 (80h7m30.912466131s) [noise=true , address=45.91.116.5:5310 , piece_id=AXJOKDZRPVNL7HFT4QKJWRAHFM2UCM3GEVQB2KNYD6CXB6TH4D4Q, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.349660917 (80h7m30.912983928s) [noise=true , address=77.90.31.10:4533 , piece_id=SE6YAKKFDL3BV677CDDUC32UNMPXJHTG23GWWYK5B4YIPKER6GAQ, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.350093077 (80h7m30.913416089s) [noise=true , address=79.127.201.215:4326 , piece_id=USEWZDX6P52U6MCALTBNQG3SV7N2JRKNU6O23TLS6JEWAS4D7ARA, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.350480894 (80h7m30.913803916s) [noise=true , address=104.36.82.13:6199 , piece_id=YP62N2TO446YR3QRG5YLXECD7TN6SJ4UNNLZSK5H4CMV6JIF4AKA, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.350669998 (80h7m30.913993011s) [noise=true , address=104.36.82.54:4184 , piece_id=IZICVGZFMAYLIJTNUJ2OFLZVNZ6IPPYDZO4BSTJJPLHVCVOE5EAA, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.351201352 (80h7m30.914524366s) [noise=true , address=104.36.82.55:4224 , piece_id=XF7H2BYSNNNZOS2T77VIN75N76AYUXAQEI6D4LRXACU2KU62V2TA, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.351668071 (80h7m30.914991083s) [noise=true , address=104.36.82.56:4261 , piece_id=EYOYOYLDACGMPGJYX6ILQT3XHMUTQNXZEDC7ANHLZXIKYVDQFFVQ, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.352193126 (80h7m30.915516138s) [noise=true , address=104.36.82.52:4089 , piece_id=X2VG4S3UHVNWA4AQJS7FLKWNVLTOZ3AKWI5ZDH24QCA3ENQW7BXQ, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.352587096 (80h7m30.915910103s) [noise=true , address=79.127.220.98:4402 , piece_id=TE2WQO5LK45AVMPIAIGVSIBOVZLC4FG4ESQ72X6KFYC7IAEN2DRA, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.353420253 (80h7m30.916743265s) [noise=true , address=104.36.82.11:6125 , piece_id=2CX7RJOKNQA6ODZZZXYPSCYAQY5P35TWX7I56CMYAPFLJE3JEO5Q, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.353791565 (80h7m30.917114578s) [noise=true , address=104.36.82.42:6803 , piece_id=R626UFD326S2UBJEZ2WMHVR7EN4SK427VXPDJZOOZ32MX25JWZWQ, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.354171226 (80h7m30.917494233s) [noise=true , address=104.36.82.14:6245 , piece_id=II36G46OA6TINLR244U6D7Q6LGV754UA5M62K3EX7ZOBESMJUEHA, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.354542282 (80h7m30.917865288s) [noise=true , address=79.127.205.235:5660 , piece_id=UGFPVS7VNB7U2YSFPSGYDIRZV5R2DHCN57PJP6VOXO52WIAYHLXA, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.354891655 (80h7m30.918214678s) [noise=true , address=104.36.82.15:6283 , piece_id=N7TH2EMIOPATVHCSNH2WXUGU7KMMEJ5LCHR22SWEHSA5OV4CUCXQ, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.355499503 (80h7m30.918822519s) [noise=true , address=77.90.31.14:4666 , piece_id=H4DTFDAVCN266PB235BJIQTQ32FBXTIB5AZSX3B4R2VIK554AFSQ, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.356575547 (80h7m30.919898539s) [noise=true , address=104.36.82.39:6756 , piece_id=OKKWMDJPZO6D5VRMGOE3GOCYGHM725JZTNOVYAWKJW3WYZPI4V5Q, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.357039590 (80h7m30.920362596s) [noise=true , address=104.36.82.53:4146 , piece_id=3V3DVGUOPJKDWIGQ2TU7M6VFYFVAS6ODY62ERGSU6LI3LSTWQ4YQ, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.360343566 (80h7m30.923666582s) [noise=true , address=104.36.82.9:6033 , piece_id=T5XA23LZELMSTWOLBSHOX4AHKDACF65OYCKTMZYTXDIBSXNA2T4A, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.452444064 (80h7m31.015767055s) [upload=64142]: Next returned nil segment Jun 22 00:30:55.452495148 (80h7m31.015818149s) [upload=64142]: Waiting for error group managing segments... Jun 22 00:30:55.532958808 (80h7m31.096281883s) [noise=true , address=104.36.82.9:6033 , piece_id=T5XA23LZELMSTWOLBSHOX4AHKDACF65OYCKTMZYTXDIBSXNA2T4A, seg_pos={0 0}, upload=64142]: Done uploading piece. err: ecclient: upload failed (node:1SV3LrcQW16wLewDzfyUWFB6sUgnERNJVdhz72CJmyde95M79C, address:104.36.82.9:6033): protocol: storage node overloaded, request limit: 2000 Jun 22 00:30:55.547802184 (80h7m31.111125188s) [noise=true , address=79.127.220.98:4402 , piece_id=TE2WQO5LK45AVMPIAIGVSIBOVZLC4FG4ESQ72X6KFYC7IAEN2DRA, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:30:55.554223621 (80h7m31.117546616s) [noise=true , address=77.90.31.14:4666 , piece_id=H4DTFDAVCN266PB235BJIQTQ32FBXTIB5AZSX3B4R2VIK554AFSQ, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:30:55.762514255 (80h7m31.325837245s) [noise=true , address=104.36.82.14:6245 , piece_id=II36G46OA6TINLR244U6D7Q6LGV754UA5M62K3EX7ZOBESMJUEHA, seg_pos={0 0}, upload=64142]: Done uploading piece. err: ecclient: upload failed (node:1Zu8teNPTXn638grHKwtJqz4UEY7oFYLx1DHvuVP6RZXcatVqX, address:104.36.82.14:6245): protocol: storage node overloaded, request limit: 2000 Jun 22 00:30:55.773669583 (80h7m31.3369926s) [noise=true , address=79.127.205.235:5660 , piece_id=UGFPVS7VNB7U2YSFPSGYDIRZV5R2DHCN57PJP6VOXO52WIAYHLXA, seg_pos={0 0}, upload=64142]: Done uploading piece. err: ecclient: upload failed (node:12E6uCX9KL4Ud7YpxMXepvMXuqTFJCiSXByDjW94Am3U9H2Nf7f, address:79.127.205.235:5660): protocol: storage node overloaded, request limit: 2000 Jun 22 00:30:56.968366655 (80h7m32.53168964s) [noise=true , address=77.90.31.10:4533 , piece_id=SE6YAKKFDL3BV677CDDUC32UNMPXJHTG23GWWYK5B4YIPKER6GAQ, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:30:57.403572874 (80h7m32.966895883s) [noise=true , address=45.91.116.5:5310 , piece_id=AXJOKDZRPVNL7HFT4QKJWRAHFM2UCM3GEVQB2KNYD6CXB6TH4D4Q, seg_pos={0 0}, upload=64142]: Done uploading piece. err: ecclient: upload failed (node:127RMUc62xRq9RU5z3XfkZsq5sehmv3kP8o53tr9BxVcPE3GQCY, address:45.91.116.5:5310): protocol: storage node overloaded, request limit: 2000 Jun 22 00:30:58.210389994 (80h7m33.773712995s) [noise=true , address=104.36.82.13:6199 , piece_id=YP62N2TO446YR3QRG5YLXECD7TN6SJ4UNNLZSK5H4CMV6JIF4AKA, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:30:58.693037961 (80h7m34.256360966s) [noise=true , address=104.36.82.39:6756 , piece_id=OKKWMDJPZO6D5VRMGOE3GOCYGHM725JZTNOVYAWKJW3WYZPI4V5Q, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:30:59.124834814 (80h7m34.688157819s) [noise=true , address=104.36.82.15:6283 , piece_id=N7TH2EMIOPATVHCSNH2WXUGU7KMMEJ5LCHR22SWEHSA5OV4CUCXQ, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:31:00.629657982 (80h7m36.192980977s) [noise=true , address=104.36.82.42:6803 , piece_id=R626UFD326S2UBJEZ2WMHVR7EN4SK427VXPDJZOOZ32MX25JWZWQ, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:31:00.933110546 (80h7m36.496433536s) [noise=true , address=79.127.201.215:4326 , piece_id=USEWZDX6P52U6MCALTBNQG3SV7N2JRKNU6O23TLS6JEWAS4D7ARA, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:31:05.669167243 (80h7m41.232490237s) [noise=true , address=104.36.82.56:4261 , piece_id=EYOYOYLDACGMPGJYX6ILQT3XHMUTQNXZEDC7ANHLZXIKYVDQFFVQ, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:31:08.443582763 (80h7m44.006905763s) [noise=true , address=104.36.82.54:4184 , piece_id=IZICVGZFMAYLIJTNUJ2OFLZVNZ6IPPYDZO4BSTJJPLHVCVOE5EAA, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:31:26.459138014 (80h8m2.022461013s) [noise=true , address=104.36.82.53:4146 , piece_id=3V3DVGUOPJKDWIGQ2TU7M6VFYFVAS6ODY62ERGSU6LI3LSTWQ4YQ, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:31:27.456050336 (80h8m3.019373336s) [noise=true , address=104.36.82.52:4089 , piece_id=X2VG4S3UHVNWA4AQJS7FLKWNVLTOZ3AKWI5ZDH24QCA3ENQW7BXQ, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:31:55.270234251 (80h8m30.833557256s) [noise=true , address=104.36.82.11:6125 , piece_id=2CX7RJOKNQA6ODZZZXYPSCYAQY5P35TWX7I56CMYAPFLJE3JEO5Q, seg_pos={0 0}, upload=64142]: Done uploading piece. err: ecclient: upload failed (node:12vricofwx67FRaxAGLbmgT1TYyKkqfJbxdsyeEPL4R7kg8pKzT, address:104.36.82.11:6125): protocol: expected piece hash; context deadline exceeded; EOF Jun 22 00:31:55.270319418 (80h8m30.833642407s) [noise=true , address=104.36.82.55:4224 , piece_id=XF7H2BYSNNNZOS2T77VIN75N76AYUXAQEI6D4LRXACU2KU62V2TA, seg_pos={0 0}, upload=64142]: Done uploading piece. err: ecclient: upload failed (node:12EjKmUiv4pxkVCB4WoiXim4uFpxrBPyfiSS4efTY3HKSE3jb5y, address:104.36.82.55:4224): protocol: write tcp 159.223.143.39:39152->104.36.82.55:4224: use of closed network connection; write tcp 159.223.143.39:39152->104.36.82.55:4224: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:39152->104.36.82.55:4224: use of closed network connection Jun 22 00:31:55.270353811 (80h8m30.833676799s) [seg_pos={0 0}, upload=64142]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Jun 22 00:31:55.301099877 (80h8m30.864422884s) [upload=64142]: Done uploading segments... Jun 22 00:31:55.301121567 (80h8m30.864444566s) [upload=64142]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 19.775828086s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [....X...................X...................................] duration: 13.575515478s FAIL (2025-06-22 01:00:52 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 13.57553998s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [......................X.....................................] duration: 14.648701621s FAIL (2025-06-22 01:00:48 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== ing upload Jun 22 00:30:55.303646184 (80h7m30.866969171s) [upload=64142]: Uploading segments... Jun 22 00:30:55.303706406 (80h7m30.867029416s) [upload=64142]: Got next segment. Inline: false Jun 22 00:30:55.303762479 (80h7m30.867085468s) [upload=64142]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 22 00:30:55.303768627 (80h7m30.867091636s) [upload=64142]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 22 00:30:55.349058140 (80h7m30.912381168s) [seg_pos={0 0}, upload=64142]: Begin upload segment... Jun 22 00:30:55.349113768 (80h7m30.912436772s) [seg_pos={0 0}, upload=64142]: Done begin upload segment. Jun 22 00:30:55.349143121 (80h7m30.912466131s) [noise=true , address=45.91.116.5:5310 , piece_id=AXJOKDZRPVNL7HFT4QKJWRAHFM2UCM3GEVQB2KNYD6CXB6TH4D4Q, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.349660917 (80h7m30.912983928s) [noise=true , address=77.90.31.10:4533 , piece_id=SE6YAKKFDL3BV677CDDUC32UNMPXJHTG23GWWYK5B4YIPKER6GAQ, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.350093077 (80h7m30.913416089s) [noise=true , address=79.127.201.215:4326 , piece_id=USEWZDX6P52U6MCALTBNQG3SV7N2JRKNU6O23TLS6JEWAS4D7ARA, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.350480894 (80h7m30.913803916s) [noise=true , address=104.36.82.13:6199 , piece_id=YP62N2TO446YR3QRG5YLXECD7TN6SJ4UNNLZSK5H4CMV6JIF4AKA, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.350669998 (80h7m30.913993011s) [noise=true , address=104.36.82.54:4184 , piece_id=IZICVGZFMAYLIJTNUJ2OFLZVNZ6IPPYDZO4BSTJJPLHVCVOE5EAA, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.351201352 (80h7m30.914524366s) [noise=true , address=104.36.82.55:4224 , piece_id=XF7H2BYSNNNZOS2T77VIN75N76AYUXAQEI6D4LRXACU2KU62V2TA, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.351668071 (80h7m30.914991083s) [noise=true , address=104.36.82.56:4261 , piece_id=EYOYOYLDACGMPGJYX6ILQT3XHMUTQNXZEDC7ANHLZXIKYVDQFFVQ, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.352193126 (80h7m30.915516138s) [noise=true , address=104.36.82.52:4089 , piece_id=X2VG4S3UHVNWA4AQJS7FLKWNVLTOZ3AKWI5ZDH24QCA3ENQW7BXQ, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.352587096 (80h7m30.915910103s) [noise=true , address=79.127.220.98:4402 , piece_id=TE2WQO5LK45AVMPIAIGVSIBOVZLC4FG4ESQ72X6KFYC7IAEN2DRA, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.353420253 (80h7m30.916743265s) [noise=true , address=104.36.82.11:6125 , piece_id=2CX7RJOKNQA6ODZZZXYPSCYAQY5P35TWX7I56CMYAPFLJE3JEO5Q, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.353791565 (80h7m30.917114578s) [noise=true , address=104.36.82.42:6803 , piece_id=R626UFD326S2UBJEZ2WMHVR7EN4SK427VXPDJZOOZ32MX25JWZWQ, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.354171226 (80h7m30.917494233s) [noise=true , address=104.36.82.14:6245 , piece_id=II36G46OA6TINLR244U6D7Q6LGV754UA5M62K3EX7ZOBESMJUEHA, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.354542282 (80h7m30.917865288s) [noise=true , address=79.127.205.235:5660 , piece_id=UGFPVS7VNB7U2YSFPSGYDIRZV5R2DHCN57PJP6VOXO52WIAYHLXA, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.354891655 (80h7m30.918214678s) [noise=true , address=104.36.82.15:6283 , piece_id=N7TH2EMIOPATVHCSNH2WXUGU7KMMEJ5LCHR22SWEHSA5OV4CUCXQ, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.355499503 (80h7m30.918822519s) [noise=true , address=77.90.31.14:4666 , piece_id=H4DTFDAVCN266PB235BJIQTQ32FBXTIB5AZSX3B4R2VIK554AFSQ, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.356575547 (80h7m30.919898539s) [noise=true , address=104.36.82.39:6756 , piece_id=OKKWMDJPZO6D5VRMGOE3GOCYGHM725JZTNOVYAWKJW3WYZPI4V5Q, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.357039590 (80h7m30.920362596s) [noise=true , address=104.36.82.53:4146 , piece_id=3V3DVGUOPJKDWIGQ2TU7M6VFYFVAS6ODY62ERGSU6LI3LSTWQ4YQ, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.360343566 (80h7m30.923666582s) [noise=true , address=104.36.82.9:6033 , piece_id=T5XA23LZELMSTWOLBSHOX4AHKDACF65OYCKTMZYTXDIBSXNA2T4A, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.452444064 (80h7m31.015767055s) [upload=64142]: Next returned nil segment Jun 22 00:30:55.452495148 (80h7m31.015818149s) [upload=64142]: Waiting for error group managing segments... Jun 22 00:30:55.532958808 (80h7m31.096281883s) [noise=true , address=104.36.82.9:6033 , piece_id=T5XA23LZELMSTWOLBSHOX4AHKDACF65OYCKTMZYTXDIBSXNA2T4A, seg_pos={0 0}, upload=64142]: Done uploading piece. err: ecclient: upload failed (node:1SV3LrcQW16wLewDzfyUWFB6sUgnERNJVdhz72CJmyde95M79C, address:104.36.82.9:6033): protocol: storage node overloaded, request limit: 2000 Jun 22 00:30:55.547802184 (80h7m31.111125188s) [noise=true , address=79.127.220.98:4402 , piece_id=TE2WQO5LK45AVMPIAIGVSIBOVZLC4FG4ESQ72X6KFYC7IAEN2DRA, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:30:55.554223621 (80h7m31.117546616s) [noise=true , address=77.90.31.14:4666 , piece_id=H4DTFDAVCN266PB235BJIQTQ32FBXTIB5AZSX3B4R2VIK554AFSQ, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:30:55.762514255 (80h7m31.325837245s) [noise=true , address=104.36.82.14:6245 , piece_id=II36G46OA6TINLR244U6D7Q6LGV754UA5M62K3EX7ZOBESMJUEHA, seg_pos={0 0}, upload=64142]: Done uploading piece. err: ecclient: upload failed (node:1Zu8teNPTXn638grHKwtJqz4UEY7oFYLx1DHvuVP6RZXcatVqX, address:104.36.82.14:6245): protocol: storage node overloaded, request limit: 2000 Jun 22 00:30:55.773669583 (80h7m31.3369926s) [noise=true , address=79.127.205.235:5660 , piece_id=UGFPVS7VNB7U2YSFPSGYDIRZV5R2DHCN57PJP6VOXO52WIAYHLXA, seg_pos={0 0}, upload=64142]: Done uploading piece. err: ecclient: upload failed (node:12E6uCX9KL4Ud7YpxMXepvMXuqTFJCiSXByDjW94Am3U9H2Nf7f, address:79.127.205.235:5660): protocol: storage node overloaded, request limit: 2000 Jun 22 00:30:56.968366655 (80h7m32.53168964s) [noise=true , address=77.90.31.10:4533 , piece_id=SE6YAKKFDL3BV677CDDUC32UNMPXJHTG23GWWYK5B4YIPKER6GAQ, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:30:57.403572874 (80h7m32.966895883s) [noise=true , address=45.91.116.5:5310 , piece_id=AXJOKDZRPVNL7HFT4QKJWRAHFM2UCM3GEVQB2KNYD6CXB6TH4D4Q, seg_pos={0 0}, upload=64142]: Done uploading piece. err: ecclient: upload failed (node:127RMUc62xRq9RU5z3XfkZsq5sehmv3kP8o53tr9BxVcPE3GQCY, address:45.91.116.5:5310): protocol: storage node overloaded, request limit: 2000 Jun 22 00:30:58.210389994 (80h7m33.773712995s) [noise=true , address=104.36.82.13:6199 , piece_id=YP62N2TO446YR3QRG5YLXECD7TN6SJ4UNNLZSK5H4CMV6JIF4AKA, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:30:58.693037961 (80h7m34.256360966s) [noise=true , address=104.36.82.39:6756 , piece_id=OKKWMDJPZO6D5VRMGOE3GOCYGHM725JZTNOVYAWKJW3WYZPI4V5Q, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:30:59.124834814 (80h7m34.688157819s) [noise=true , address=104.36.82.15:6283 , piece_id=N7TH2EMIOPATVHCSNH2WXUGU7KMMEJ5LCHR22SWEHSA5OV4CUCXQ, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:31:00.629657982 (80h7m36.192980977s) [noise=true , address=104.36.82.42:6803 , piece_id=R626UFD326S2UBJEZ2WMHVR7EN4SK427VXPDJZOOZ32MX25JWZWQ, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:31:00.933110546 (80h7m36.496433536s) [noise=true , address=79.127.201.215:4326 , piece_id=USEWZDX6P52U6MCALTBNQG3SV7N2JRKNU6O23TLS6JEWAS4D7ARA, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:31:05.669167243 (80h7m41.232490237s) [noise=true , address=104.36.82.56:4261 , piece_id=EYOYOYLDACGMPGJYX6ILQT3XHMUTQNXZEDC7ANHLZXIKYVDQFFVQ, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:31:08.443582763 (80h7m44.006905763s) [noise=true , address=104.36.82.54:4184 , piece_id=IZICVGZFMAYLIJTNUJ2OFLZVNZ6IPPYDZO4BSTJJPLHVCVOE5EAA, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:31:26.459138014 (80h8m2.022461013s) [noise=true , address=104.36.82.53:4146 , piece_id=3V3DVGUOPJKDWIGQ2TU7M6VFYFVAS6ODY62ERGSU6LI3LSTWQ4YQ, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:31:27.456050336 (80h8m3.019373336s) [noise=true , address=104.36.82.52:4089 , piece_id=X2VG4S3UHVNWA4AQJS7FLKWNVLTOZ3AKWI5ZDH24QCA3ENQW7BXQ, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:31:55.270234251 (80h8m30.833557256s) [noise=true , address=104.36.82.11:6125 , piece_id=2CX7RJOKNQA6ODZZZXYPSCYAQY5P35TWX7I56CMYAPFLJE3JEO5Q, seg_pos={0 0}, upload=64142]: Done uploading piece. err: ecclient: upload failed (node:12vricofwx67FRaxAGLbmgT1TYyKkqfJbxdsyeEPL4R7kg8pKzT, address:104.36.82.11:6125): protocol: expected piece hash; context deadline exceeded; EOF Jun 22 00:31:55.270319418 (80h8m30.833642407s) [noise=true , address=104.36.82.55:4224 , piece_id=XF7H2BYSNNNZOS2T77VIN75N76AYUXAQEI6D4LRXACU2KU62V2TA, seg_pos={0 0}, upload=64142]: Done uploading piece. err: ecclient: upload failed (node:12EjKmUiv4pxkVCB4WoiXim4uFpxrBPyfiSS4efTY3HKSE3jb5y, address:104.36.82.55:4224): protocol: write tcp 159.223.143.39:39152->104.36.82.55:4224: use of closed network connection; write tcp 159.223.143.39:39152->104.36.82.55:4224: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:39152->104.36.82.55:4224: use of closed network connection Jun 22 00:31:55.270353811 (80h8m30.833676799s) [seg_pos={0 0}, upload=64142]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Jun 22 00:31:55.301099877 (80h8m30.864422884s) [upload=64142]: Done uploading segments... Jun 22 00:31:55.301121567 (80h8m30.864444566s) [upload=64142]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 14.648739164s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [.....X...................X..................................] duration: 15.727294744s FAIL (2025-06-22 01:00:10 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 15.727329692s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.......................X....................................] duration: 24.200004931s FAIL (2025-06-22 01:00:06 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== ing upload Jun 22 00:30:55.303646184 (80h7m30.866969171s) [upload=64142]: Uploading segments... Jun 22 00:30:55.303706406 (80h7m30.867029416s) [upload=64142]: Got next segment. Inline: false Jun 22 00:30:55.303762479 (80h7m30.867085468s) [upload=64142]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 22 00:30:55.303768627 (80h7m30.867091636s) [upload=64142]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 22 00:30:55.349058140 (80h7m30.912381168s) [seg_pos={0 0}, upload=64142]: Begin upload segment... Jun 22 00:30:55.349113768 (80h7m30.912436772s) [seg_pos={0 0}, upload=64142]: Done begin upload segment. Jun 22 00:30:55.349143121 (80h7m30.912466131s) [noise=true , address=45.91.116.5:5310 , piece_id=AXJOKDZRPVNL7HFT4QKJWRAHFM2UCM3GEVQB2KNYD6CXB6TH4D4Q, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.349660917 (80h7m30.912983928s) [noise=true , address=77.90.31.10:4533 , piece_id=SE6YAKKFDL3BV677CDDUC32UNMPXJHTG23GWWYK5B4YIPKER6GAQ, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.350093077 (80h7m30.913416089s) [noise=true , address=79.127.201.215:4326 , piece_id=USEWZDX6P52U6MCALTBNQG3SV7N2JRKNU6O23TLS6JEWAS4D7ARA, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.350480894 (80h7m30.913803916s) [noise=true , address=104.36.82.13:6199 , piece_id=YP62N2TO446YR3QRG5YLXECD7TN6SJ4UNNLZSK5H4CMV6JIF4AKA, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.350669998 (80h7m30.913993011s) [noise=true , address=104.36.82.54:4184 , piece_id=IZICVGZFMAYLIJTNUJ2OFLZVNZ6IPPYDZO4BSTJJPLHVCVOE5EAA, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.351201352 (80h7m30.914524366s) [noise=true , address=104.36.82.55:4224 , piece_id=XF7H2BYSNNNZOS2T77VIN75N76AYUXAQEI6D4LRXACU2KU62V2TA, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.351668071 (80h7m30.914991083s) [noise=true , address=104.36.82.56:4261 , piece_id=EYOYOYLDACGMPGJYX6ILQT3XHMUTQNXZEDC7ANHLZXIKYVDQFFVQ, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.352193126 (80h7m30.915516138s) [noise=true , address=104.36.82.52:4089 , piece_id=X2VG4S3UHVNWA4AQJS7FLKWNVLTOZ3AKWI5ZDH24QCA3ENQW7BXQ, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.352587096 (80h7m30.915910103s) [noise=true , address=79.127.220.98:4402 , piece_id=TE2WQO5LK45AVMPIAIGVSIBOVZLC4FG4ESQ72X6KFYC7IAEN2DRA, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.353420253 (80h7m30.916743265s) [noise=true , address=104.36.82.11:6125 , piece_id=2CX7RJOKNQA6ODZZZXYPSCYAQY5P35TWX7I56CMYAPFLJE3JEO5Q, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.353791565 (80h7m30.917114578s) [noise=true , address=104.36.82.42:6803 , piece_id=R626UFD326S2UBJEZ2WMHVR7EN4SK427VXPDJZOOZ32MX25JWZWQ, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.354171226 (80h7m30.917494233s) [noise=true , address=104.36.82.14:6245 , piece_id=II36G46OA6TINLR244U6D7Q6LGV754UA5M62K3EX7ZOBESMJUEHA, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.354542282 (80h7m30.917865288s) [noise=true , address=79.127.205.235:5660 , piece_id=UGFPVS7VNB7U2YSFPSGYDIRZV5R2DHCN57PJP6VOXO52WIAYHLXA, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.354891655 (80h7m30.918214678s) [noise=true , address=104.36.82.15:6283 , piece_id=N7TH2EMIOPATVHCSNH2WXUGU7KMMEJ5LCHR22SWEHSA5OV4CUCXQ, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.355499503 (80h7m30.918822519s) [noise=true , address=77.90.31.14:4666 , piece_id=H4DTFDAVCN266PB235BJIQTQ32FBXTIB5AZSX3B4R2VIK554AFSQ, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.356575547 (80h7m30.919898539s) [noise=true , address=104.36.82.39:6756 , piece_id=OKKWMDJPZO6D5VRMGOE3GOCYGHM725JZTNOVYAWKJW3WYZPI4V5Q, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.357039590 (80h7m30.920362596s) [noise=true , address=104.36.82.53:4146 , piece_id=3V3DVGUOPJKDWIGQ2TU7M6VFYFVAS6ODY62ERGSU6LI3LSTWQ4YQ, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.360343566 (80h7m30.923666582s) [noise=true , address=104.36.82.9:6033 , piece_id=T5XA23LZELMSTWOLBSHOX4AHKDACF65OYCKTMZYTXDIBSXNA2T4A, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.452444064 (80h7m31.015767055s) [upload=64142]: Next returned nil segment Jun 22 00:30:55.452495148 (80h7m31.015818149s) [upload=64142]: Waiting for error group managing segments... Jun 22 00:30:55.532958808 (80h7m31.096281883s) [noise=true , address=104.36.82.9:6033 , piece_id=T5XA23LZELMSTWOLBSHOX4AHKDACF65OYCKTMZYTXDIBSXNA2T4A, seg_pos={0 0}, upload=64142]: Done uploading piece. err: ecclient: upload failed (node:1SV3LrcQW16wLewDzfyUWFB6sUgnERNJVdhz72CJmyde95M79C, address:104.36.82.9:6033): protocol: storage node overloaded, request limit: 2000 Jun 22 00:30:55.547802184 (80h7m31.111125188s) [noise=true , address=79.127.220.98:4402 , piece_id=TE2WQO5LK45AVMPIAIGVSIBOVZLC4FG4ESQ72X6KFYC7IAEN2DRA, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:30:55.554223621 (80h7m31.117546616s) [noise=true , address=77.90.31.14:4666 , piece_id=H4DTFDAVCN266PB235BJIQTQ32FBXTIB5AZSX3B4R2VIK554AFSQ, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:30:55.762514255 (80h7m31.325837245s) [noise=true , address=104.36.82.14:6245 , piece_id=II36G46OA6TINLR244U6D7Q6LGV754UA5M62K3EX7ZOBESMJUEHA, seg_pos={0 0}, upload=64142]: Done uploading piece. err: ecclient: upload failed (node:1Zu8teNPTXn638grHKwtJqz4UEY7oFYLx1DHvuVP6RZXcatVqX, address:104.36.82.14:6245): protocol: storage node overloaded, request limit: 2000 Jun 22 00:30:55.773669583 (80h7m31.3369926s) [noise=true , address=79.127.205.235:5660 , piece_id=UGFPVS7VNB7U2YSFPSGYDIRZV5R2DHCN57PJP6VOXO52WIAYHLXA, seg_pos={0 0}, upload=64142]: Done uploading piece. err: ecclient: upload failed (node:12E6uCX9KL4Ud7YpxMXepvMXuqTFJCiSXByDjW94Am3U9H2Nf7f, address:79.127.205.235:5660): protocol: storage node overloaded, request limit: 2000 Jun 22 00:30:56.968366655 (80h7m32.53168964s) [noise=true , address=77.90.31.10:4533 , piece_id=SE6YAKKFDL3BV677CDDUC32UNMPXJHTG23GWWYK5B4YIPKER6GAQ, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:30:57.403572874 (80h7m32.966895883s) [noise=true , address=45.91.116.5:5310 , piece_id=AXJOKDZRPVNL7HFT4QKJWRAHFM2UCM3GEVQB2KNYD6CXB6TH4D4Q, seg_pos={0 0}, upload=64142]: Done uploading piece. err: ecclient: upload failed (node:127RMUc62xRq9RU5z3XfkZsq5sehmv3kP8o53tr9BxVcPE3GQCY, address:45.91.116.5:5310): protocol: storage node overloaded, request limit: 2000 Jun 22 00:30:58.210389994 (80h7m33.773712995s) [noise=true , address=104.36.82.13:6199 , piece_id=YP62N2TO446YR3QRG5YLXECD7TN6SJ4UNNLZSK5H4CMV6JIF4AKA, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:30:58.693037961 (80h7m34.256360966s) [noise=true , address=104.36.82.39:6756 , piece_id=OKKWMDJPZO6D5VRMGOE3GOCYGHM725JZTNOVYAWKJW3WYZPI4V5Q, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:30:59.124834814 (80h7m34.688157819s) [noise=true , address=104.36.82.15:6283 , piece_id=N7TH2EMIOPATVHCSNH2WXUGU7KMMEJ5LCHR22SWEHSA5OV4CUCXQ, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:31:00.629657982 (80h7m36.192980977s) [noise=true , address=104.36.82.42:6803 , piece_id=R626UFD326S2UBJEZ2WMHVR7EN4SK427VXPDJZOOZ32MX25JWZWQ, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:31:00.933110546 (80h7m36.496433536s) [noise=true , address=79.127.201.215:4326 , piece_id=USEWZDX6P52U6MCALTBNQG3SV7N2JRKNU6O23TLS6JEWAS4D7ARA, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:31:05.669167243 (80h7m41.232490237s) [noise=true , address=104.36.82.56:4261 , piece_id=EYOYOYLDACGMPGJYX6ILQT3XHMUTQNXZEDC7ANHLZXIKYVDQFFVQ, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:31:08.443582763 (80h7m44.006905763s) [noise=true , address=104.36.82.54:4184 , piece_id=IZICVGZFMAYLIJTNUJ2OFLZVNZ6IPPYDZO4BSTJJPLHVCVOE5EAA, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:31:26.459138014 (80h8m2.022461013s) [noise=true , address=104.36.82.53:4146 , piece_id=3V3DVGUOPJKDWIGQ2TU7M6VFYFVAS6ODY62ERGSU6LI3LSTWQ4YQ, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:31:27.456050336 (80h8m3.019373336s) [noise=true , address=104.36.82.52:4089 , piece_id=X2VG4S3UHVNWA4AQJS7FLKWNVLTOZ3AKWI5ZDH24QCA3ENQW7BXQ, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:31:55.270234251 (80h8m30.833557256s) [noise=true , address=104.36.82.11:6125 , piece_id=2CX7RJOKNQA6ODZZZXYPSCYAQY5P35TWX7I56CMYAPFLJE3JEO5Q, seg_pos={0 0}, upload=64142]: Done uploading piece. err: ecclient: upload failed (node:12vricofwx67FRaxAGLbmgT1TYyKkqfJbxdsyeEPL4R7kg8pKzT, address:104.36.82.11:6125): protocol: expected piece hash; context deadline exceeded; EOF Jun 22 00:31:55.270319418 (80h8m30.833642407s) [noise=true , address=104.36.82.55:4224 , piece_id=XF7H2BYSNNNZOS2T77VIN75N76AYUXAQEI6D4LRXACU2KU62V2TA, seg_pos={0 0}, upload=64142]: Done uploading piece. err: ecclient: upload failed (node:12EjKmUiv4pxkVCB4WoiXim4uFpxrBPyfiSS4efTY3HKSE3jb5y, address:104.36.82.55:4224): protocol: write tcp 159.223.143.39:39152->104.36.82.55:4224: use of closed network connection; write tcp 159.223.143.39:39152->104.36.82.55:4224: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:39152->104.36.82.55:4224: use of closed network connection Jun 22 00:31:55.270353811 (80h8m30.833676799s) [seg_pos={0 0}, upload=64142]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Jun 22 00:31:55.301099877 (80h8m30.864422884s) [upload=64142]: Done uploading segments... Jun 22 00:31:55.301121567 (80h8m30.864444566s) [upload=64142]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 24.200036395s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [......X...................X.................................] duration: 21.767553741s FAIL (2025-06-22 00:59:41 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 21.767578932s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [........................X...................................] duration: 22.026591465s FAIL (2025-06-22 00:59:36 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== ing upload Jun 22 00:30:55.303646184 (80h7m30.866969171s) [upload=64142]: Uploading segments... Jun 22 00:30:55.303706406 (80h7m30.867029416s) [upload=64142]: Got next segment. Inline: false Jun 22 00:30:55.303762479 (80h7m30.867085468s) [upload=64142]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 22 00:30:55.303768627 (80h7m30.867091636s) [upload=64142]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 22 00:30:55.349058140 (80h7m30.912381168s) [seg_pos={0 0}, upload=64142]: Begin upload segment... Jun 22 00:30:55.349113768 (80h7m30.912436772s) [seg_pos={0 0}, upload=64142]: Done begin upload segment. Jun 22 00:30:55.349143121 (80h7m30.912466131s) [noise=true , address=45.91.116.5:5310 , piece_id=AXJOKDZRPVNL7HFT4QKJWRAHFM2UCM3GEVQB2KNYD6CXB6TH4D4Q, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.349660917 (80h7m30.912983928s) [noise=true , address=77.90.31.10:4533 , piece_id=SE6YAKKFDL3BV677CDDUC32UNMPXJHTG23GWWYK5B4YIPKER6GAQ, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.350093077 (80h7m30.913416089s) [noise=true , address=79.127.201.215:4326 , piece_id=USEWZDX6P52U6MCALTBNQG3SV7N2JRKNU6O23TLS6JEWAS4D7ARA, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.350480894 (80h7m30.913803916s) [noise=true , address=104.36.82.13:6199 , piece_id=YP62N2TO446YR3QRG5YLXECD7TN6SJ4UNNLZSK5H4CMV6JIF4AKA, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.350669998 (80h7m30.913993011s) [noise=true , address=104.36.82.54:4184 , piece_id=IZICVGZFMAYLIJTNUJ2OFLZVNZ6IPPYDZO4BSTJJPLHVCVOE5EAA, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.351201352 (80h7m30.914524366s) [noise=true , address=104.36.82.55:4224 , piece_id=XF7H2BYSNNNZOS2T77VIN75N76AYUXAQEI6D4LRXACU2KU62V2TA, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.351668071 (80h7m30.914991083s) [noise=true , address=104.36.82.56:4261 , piece_id=EYOYOYLDACGMPGJYX6ILQT3XHMUTQNXZEDC7ANHLZXIKYVDQFFVQ, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.352193126 (80h7m30.915516138s) [noise=true , address=104.36.82.52:4089 , piece_id=X2VG4S3UHVNWA4AQJS7FLKWNVLTOZ3AKWI5ZDH24QCA3ENQW7BXQ, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.352587096 (80h7m30.915910103s) [noise=true , address=79.127.220.98:4402 , piece_id=TE2WQO5LK45AVMPIAIGVSIBOVZLC4FG4ESQ72X6KFYC7IAEN2DRA, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.353420253 (80h7m30.916743265s) [noise=true , address=104.36.82.11:6125 , piece_id=2CX7RJOKNQA6ODZZZXYPSCYAQY5P35TWX7I56CMYAPFLJE3JEO5Q, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.353791565 (80h7m30.917114578s) [noise=true , address=104.36.82.42:6803 , piece_id=R626UFD326S2UBJEZ2WMHVR7EN4SK427VXPDJZOOZ32MX25JWZWQ, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.354171226 (80h7m30.917494233s) [noise=true , address=104.36.82.14:6245 , piece_id=II36G46OA6TINLR244U6D7Q6LGV754UA5M62K3EX7ZOBESMJUEHA, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.354542282 (80h7m30.917865288s) [noise=true , address=79.127.205.235:5660 , piece_id=UGFPVS7VNB7U2YSFPSGYDIRZV5R2DHCN57PJP6VOXO52WIAYHLXA, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.354891655 (80h7m30.918214678s) [noise=true , address=104.36.82.15:6283 , piece_id=N7TH2EMIOPATVHCSNH2WXUGU7KMMEJ5LCHR22SWEHSA5OV4CUCXQ, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.355499503 (80h7m30.918822519s) [noise=true , address=77.90.31.14:4666 , piece_id=H4DTFDAVCN266PB235BJIQTQ32FBXTIB5AZSX3B4R2VIK554AFSQ, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.356575547 (80h7m30.919898539s) [noise=true , address=104.36.82.39:6756 , piece_id=OKKWMDJPZO6D5VRMGOE3GOCYGHM725JZTNOVYAWKJW3WYZPI4V5Q, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.357039590 (80h7m30.920362596s) [noise=true , address=104.36.82.53:4146 , piece_id=3V3DVGUOPJKDWIGQ2TU7M6VFYFVAS6ODY62ERGSU6LI3LSTWQ4YQ, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.360343566 (80h7m30.923666582s) [noise=true , address=104.36.82.9:6033 , piece_id=T5XA23LZELMSTWOLBSHOX4AHKDACF65OYCKTMZYTXDIBSXNA2T4A, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.452444064 (80h7m31.015767055s) [upload=64142]: Next returned nil segment Jun 22 00:30:55.452495148 (80h7m31.015818149s) [upload=64142]: Waiting for error group managing segments... Jun 22 00:30:55.532958808 (80h7m31.096281883s) [noise=true , address=104.36.82.9:6033 , piece_id=T5XA23LZELMSTWOLBSHOX4AHKDACF65OYCKTMZYTXDIBSXNA2T4A, seg_pos={0 0}, upload=64142]: Done uploading piece. err: ecclient: upload failed (node:1SV3LrcQW16wLewDzfyUWFB6sUgnERNJVdhz72CJmyde95M79C, address:104.36.82.9:6033): protocol: storage node overloaded, request limit: 2000 Jun 22 00:30:55.547802184 (80h7m31.111125188s) [noise=true , address=79.127.220.98:4402 , piece_id=TE2WQO5LK45AVMPIAIGVSIBOVZLC4FG4ESQ72X6KFYC7IAEN2DRA, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:30:55.554223621 (80h7m31.117546616s) [noise=true , address=77.90.31.14:4666 , piece_id=H4DTFDAVCN266PB235BJIQTQ32FBXTIB5AZSX3B4R2VIK554AFSQ, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:30:55.762514255 (80h7m31.325837245s) [noise=true , address=104.36.82.14:6245 , piece_id=II36G46OA6TINLR244U6D7Q6LGV754UA5M62K3EX7ZOBESMJUEHA, seg_pos={0 0}, upload=64142]: Done uploading piece. err: ecclient: upload failed (node:1Zu8teNPTXn638grHKwtJqz4UEY7oFYLx1DHvuVP6RZXcatVqX, address:104.36.82.14:6245): protocol: storage node overloaded, request limit: 2000 Jun 22 00:30:55.773669583 (80h7m31.3369926s) [noise=true , address=79.127.205.235:5660 , piece_id=UGFPVS7VNB7U2YSFPSGYDIRZV5R2DHCN57PJP6VOXO52WIAYHLXA, seg_pos={0 0}, upload=64142]: Done uploading piece. err: ecclient: upload failed (node:12E6uCX9KL4Ud7YpxMXepvMXuqTFJCiSXByDjW94Am3U9H2Nf7f, address:79.127.205.235:5660): protocol: storage node overloaded, request limit: 2000 Jun 22 00:30:56.968366655 (80h7m32.53168964s) [noise=true , address=77.90.31.10:4533 , piece_id=SE6YAKKFDL3BV677CDDUC32UNMPXJHTG23GWWYK5B4YIPKER6GAQ, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:30:57.403572874 (80h7m32.966895883s) [noise=true , address=45.91.116.5:5310 , piece_id=AXJOKDZRPVNL7HFT4QKJWRAHFM2UCM3GEVQB2KNYD6CXB6TH4D4Q, seg_pos={0 0}, upload=64142]: Done uploading piece. err: ecclient: upload failed (node:127RMUc62xRq9RU5z3XfkZsq5sehmv3kP8o53tr9BxVcPE3GQCY, address:45.91.116.5:5310): protocol: storage node overloaded, request limit: 2000 Jun 22 00:30:58.210389994 (80h7m33.773712995s) [noise=true , address=104.36.82.13:6199 , piece_id=YP62N2TO446YR3QRG5YLXECD7TN6SJ4UNNLZSK5H4CMV6JIF4AKA, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:30:58.693037961 (80h7m34.256360966s) [noise=true , address=104.36.82.39:6756 , piece_id=OKKWMDJPZO6D5VRMGOE3GOCYGHM725JZTNOVYAWKJW3WYZPI4V5Q, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:30:59.124834814 (80h7m34.688157819s) [noise=true , address=104.36.82.15:6283 , piece_id=N7TH2EMIOPATVHCSNH2WXUGU7KMMEJ5LCHR22SWEHSA5OV4CUCXQ, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:31:00.629657982 (80h7m36.192980977s) [noise=true , address=104.36.82.42:6803 , piece_id=R626UFD326S2UBJEZ2WMHVR7EN4SK427VXPDJZOOZ32MX25JWZWQ, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:31:00.933110546 (80h7m36.496433536s) [noise=true , address=79.127.201.215:4326 , piece_id=USEWZDX6P52U6MCALTBNQG3SV7N2JRKNU6O23TLS6JEWAS4D7ARA, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:31:05.669167243 (80h7m41.232490237s) [noise=true , address=104.36.82.56:4261 , piece_id=EYOYOYLDACGMPGJYX6ILQT3XHMUTQNXZEDC7ANHLZXIKYVDQFFVQ, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:31:08.443582763 (80h7m44.006905763s) [noise=true , address=104.36.82.54:4184 , piece_id=IZICVGZFMAYLIJTNUJ2OFLZVNZ6IPPYDZO4BSTJJPLHVCVOE5EAA, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:31:26.459138014 (80h8m2.022461013s) [noise=true , address=104.36.82.53:4146 , piece_id=3V3DVGUOPJKDWIGQ2TU7M6VFYFVAS6ODY62ERGSU6LI3LSTWQ4YQ, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:31:27.456050336 (80h8m3.019373336s) [noise=true , address=104.36.82.52:4089 , piece_id=X2VG4S3UHVNWA4AQJS7FLKWNVLTOZ3AKWI5ZDH24QCA3ENQW7BXQ, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:31:55.270234251 (80h8m30.833557256s) [noise=true , address=104.36.82.11:6125 , piece_id=2CX7RJOKNQA6ODZZZXYPSCYAQY5P35TWX7I56CMYAPFLJE3JEO5Q, seg_pos={0 0}, upload=64142]: Done uploading piece. err: ecclient: upload failed (node:12vricofwx67FRaxAGLbmgT1TYyKkqfJbxdsyeEPL4R7kg8pKzT, address:104.36.82.11:6125): protocol: expected piece hash; context deadline exceeded; EOF Jun 22 00:31:55.270319418 (80h8m30.833642407s) [noise=true , address=104.36.82.55:4224 , piece_id=XF7H2BYSNNNZOS2T77VIN75N76AYUXAQEI6D4LRXACU2KU62V2TA, seg_pos={0 0}, upload=64142]: Done uploading piece. err: ecclient: upload failed (node:12EjKmUiv4pxkVCB4WoiXim4uFpxrBPyfiSS4efTY3HKSE3jb5y, address:104.36.82.55:4224): protocol: write tcp 159.223.143.39:39152->104.36.82.55:4224: use of closed network connection; write tcp 159.223.143.39:39152->104.36.82.55:4224: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:39152->104.36.82.55:4224: use of closed network connection Jun 22 00:31:55.270353811 (80h8m30.833676799s) [seg_pos={0 0}, upload=64142]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Jun 22 00:31:55.301099877 (80h8m30.864422884s) [upload=64142]: Done uploading segments... Jun 22 00:31:55.301121567 (80h8m30.864444566s) [upload=64142]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 22.026661142s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [.......X...................X................................] duration: 14.681752741s FAIL (2025-06-22 00:58:41 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 14.6819845s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [........X...................X...............................] duration: 31.501366625s FAIL (2025-06-22 00:58:27 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 31.501385259s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.........................X..................................] duration: 19.496961873s FAIL (2025-06-22 00:58:21 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== ing upload Jun 22 00:30:55.303646184 (80h7m30.866969171s) [upload=64142]: Uploading segments... Jun 22 00:30:55.303706406 (80h7m30.867029416s) [upload=64142]: Got next segment. Inline: false Jun 22 00:30:55.303762479 (80h7m30.867085468s) [upload=64142]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 22 00:30:55.303768627 (80h7m30.867091636s) [upload=64142]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 22 00:30:55.349058140 (80h7m30.912381168s) [seg_pos={0 0}, upload=64142]: Begin upload segment... Jun 22 00:30:55.349113768 (80h7m30.912436772s) [seg_pos={0 0}, upload=64142]: Done begin upload segment. Jun 22 00:30:55.349143121 (80h7m30.912466131s) [noise=true , address=45.91.116.5:5310 , piece_id=AXJOKDZRPVNL7HFT4QKJWRAHFM2UCM3GEVQB2KNYD6CXB6TH4D4Q, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.349660917 (80h7m30.912983928s) [noise=true , address=77.90.31.10:4533 , piece_id=SE6YAKKFDL3BV677CDDUC32UNMPXJHTG23GWWYK5B4YIPKER6GAQ, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.350093077 (80h7m30.913416089s) [noise=true , address=79.127.201.215:4326 , piece_id=USEWZDX6P52U6MCALTBNQG3SV7N2JRKNU6O23TLS6JEWAS4D7ARA, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.350480894 (80h7m30.913803916s) [noise=true , address=104.36.82.13:6199 , piece_id=YP62N2TO446YR3QRG5YLXECD7TN6SJ4UNNLZSK5H4CMV6JIF4AKA, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.350669998 (80h7m30.913993011s) [noise=true , address=104.36.82.54:4184 , piece_id=IZICVGZFMAYLIJTNUJ2OFLZVNZ6IPPYDZO4BSTJJPLHVCVOE5EAA, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.351201352 (80h7m30.914524366s) [noise=true , address=104.36.82.55:4224 , piece_id=XF7H2BYSNNNZOS2T77VIN75N76AYUXAQEI6D4LRXACU2KU62V2TA, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.351668071 (80h7m30.914991083s) [noise=true , address=104.36.82.56:4261 , piece_id=EYOYOYLDACGMPGJYX6ILQT3XHMUTQNXZEDC7ANHLZXIKYVDQFFVQ, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.352193126 (80h7m30.915516138s) [noise=true , address=104.36.82.52:4089 , piece_id=X2VG4S3UHVNWA4AQJS7FLKWNVLTOZ3AKWI5ZDH24QCA3ENQW7BXQ, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.352587096 (80h7m30.915910103s) [noise=true , address=79.127.220.98:4402 , piece_id=TE2WQO5LK45AVMPIAIGVSIBOVZLC4FG4ESQ72X6KFYC7IAEN2DRA, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.353420253 (80h7m30.916743265s) [noise=true , address=104.36.82.11:6125 , piece_id=2CX7RJOKNQA6ODZZZXYPSCYAQY5P35TWX7I56CMYAPFLJE3JEO5Q, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.353791565 (80h7m30.917114578s) [noise=true , address=104.36.82.42:6803 , piece_id=R626UFD326S2UBJEZ2WMHVR7EN4SK427VXPDJZOOZ32MX25JWZWQ, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.354171226 (80h7m30.917494233s) [noise=true , address=104.36.82.14:6245 , piece_id=II36G46OA6TINLR244U6D7Q6LGV754UA5M62K3EX7ZOBESMJUEHA, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.354542282 (80h7m30.917865288s) [noise=true , address=79.127.205.235:5660 , piece_id=UGFPVS7VNB7U2YSFPSGYDIRZV5R2DHCN57PJP6VOXO52WIAYHLXA, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.354891655 (80h7m30.918214678s) [noise=true , address=104.36.82.15:6283 , piece_id=N7TH2EMIOPATVHCSNH2WXUGU7KMMEJ5LCHR22SWEHSA5OV4CUCXQ, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.355499503 (80h7m30.918822519s) [noise=true , address=77.90.31.14:4666 , piece_id=H4DTFDAVCN266PB235BJIQTQ32FBXTIB5AZSX3B4R2VIK554AFSQ, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.356575547 (80h7m30.919898539s) [noise=true , address=104.36.82.39:6756 , piece_id=OKKWMDJPZO6D5VRMGOE3GOCYGHM725JZTNOVYAWKJW3WYZPI4V5Q, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.357039590 (80h7m30.920362596s) [noise=true , address=104.36.82.53:4146 , piece_id=3V3DVGUOPJKDWIGQ2TU7M6VFYFVAS6ODY62ERGSU6LI3LSTWQ4YQ, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.360343566 (80h7m30.923666582s) [noise=true , address=104.36.82.9:6033 , piece_id=T5XA23LZELMSTWOLBSHOX4AHKDACF65OYCKTMZYTXDIBSXNA2T4A, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.452444064 (80h7m31.015767055s) [upload=64142]: Next returned nil segment Jun 22 00:30:55.452495148 (80h7m31.015818149s) [upload=64142]: Waiting for error group managing segments... Jun 22 00:30:55.532958808 (80h7m31.096281883s) [noise=true , address=104.36.82.9:6033 , piece_id=T5XA23LZELMSTWOLBSHOX4AHKDACF65OYCKTMZYTXDIBSXNA2T4A, seg_pos={0 0}, upload=64142]: Done uploading piece. err: ecclient: upload failed (node:1SV3LrcQW16wLewDzfyUWFB6sUgnERNJVdhz72CJmyde95M79C, address:104.36.82.9:6033): protocol: storage node overloaded, request limit: 2000 Jun 22 00:30:55.547802184 (80h7m31.111125188s) [noise=true , address=79.127.220.98:4402 , piece_id=TE2WQO5LK45AVMPIAIGVSIBOVZLC4FG4ESQ72X6KFYC7IAEN2DRA, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:30:55.554223621 (80h7m31.117546616s) [noise=true , address=77.90.31.14:4666 , piece_id=H4DTFDAVCN266PB235BJIQTQ32FBXTIB5AZSX3B4R2VIK554AFSQ, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:30:55.762514255 (80h7m31.325837245s) [noise=true , address=104.36.82.14:6245 , piece_id=II36G46OA6TINLR244U6D7Q6LGV754UA5M62K3EX7ZOBESMJUEHA, seg_pos={0 0}, upload=64142]: Done uploading piece. err: ecclient: upload failed (node:1Zu8teNPTXn638grHKwtJqz4UEY7oFYLx1DHvuVP6RZXcatVqX, address:104.36.82.14:6245): protocol: storage node overloaded, request limit: 2000 Jun 22 00:30:55.773669583 (80h7m31.3369926s) [noise=true , address=79.127.205.235:5660 , piece_id=UGFPVS7VNB7U2YSFPSGYDIRZV5R2DHCN57PJP6VOXO52WIAYHLXA, seg_pos={0 0}, upload=64142]: Done uploading piece. err: ecclient: upload failed (node:12E6uCX9KL4Ud7YpxMXepvMXuqTFJCiSXByDjW94Am3U9H2Nf7f, address:79.127.205.235:5660): protocol: storage node overloaded, request limit: 2000 Jun 22 00:30:56.968366655 (80h7m32.53168964s) [noise=true , address=77.90.31.10:4533 , piece_id=SE6YAKKFDL3BV677CDDUC32UNMPXJHTG23GWWYK5B4YIPKER6GAQ, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:30:57.403572874 (80h7m32.966895883s) [noise=true , address=45.91.116.5:5310 , piece_id=AXJOKDZRPVNL7HFT4QKJWRAHFM2UCM3GEVQB2KNYD6CXB6TH4D4Q, seg_pos={0 0}, upload=64142]: Done uploading piece. err: ecclient: upload failed (node:127RMUc62xRq9RU5z3XfkZsq5sehmv3kP8o53tr9BxVcPE3GQCY, address:45.91.116.5:5310): protocol: storage node overloaded, request limit: 2000 Jun 22 00:30:58.210389994 (80h7m33.773712995s) [noise=true , address=104.36.82.13:6199 , piece_id=YP62N2TO446YR3QRG5YLXECD7TN6SJ4UNNLZSK5H4CMV6JIF4AKA, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:30:58.693037961 (80h7m34.256360966s) [noise=true , address=104.36.82.39:6756 , piece_id=OKKWMDJPZO6D5VRMGOE3GOCYGHM725JZTNOVYAWKJW3WYZPI4V5Q, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:30:59.124834814 (80h7m34.688157819s) [noise=true , address=104.36.82.15:6283 , piece_id=N7TH2EMIOPATVHCSNH2WXUGU7KMMEJ5LCHR22SWEHSA5OV4CUCXQ, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:31:00.629657982 (80h7m36.192980977s) [noise=true , address=104.36.82.42:6803 , piece_id=R626UFD326S2UBJEZ2WMHVR7EN4SK427VXPDJZOOZ32MX25JWZWQ, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:31:00.933110546 (80h7m36.496433536s) [noise=true , address=79.127.201.215:4326 , piece_id=USEWZDX6P52U6MCALTBNQG3SV7N2JRKNU6O23TLS6JEWAS4D7ARA, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:31:05.669167243 (80h7m41.232490237s) [noise=true , address=104.36.82.56:4261 , piece_id=EYOYOYLDACGMPGJYX6ILQT3XHMUTQNXZEDC7ANHLZXIKYVDQFFVQ, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:31:08.443582763 (80h7m44.006905763s) [noise=true , address=104.36.82.54:4184 , piece_id=IZICVGZFMAYLIJTNUJ2OFLZVNZ6IPPYDZO4BSTJJPLHVCVOE5EAA, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:31:26.459138014 (80h8m2.022461013s) [noise=true , address=104.36.82.53:4146 , piece_id=3V3DVGUOPJKDWIGQ2TU7M6VFYFVAS6ODY62ERGSU6LI3LSTWQ4YQ, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:31:27.456050336 (80h8m3.019373336s) [noise=true , address=104.36.82.52:4089 , piece_id=X2VG4S3UHVNWA4AQJS7FLKWNVLTOZ3AKWI5ZDH24QCA3ENQW7BXQ, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:31:55.270234251 (80h8m30.833557256s) [noise=true , address=104.36.82.11:6125 , piece_id=2CX7RJOKNQA6ODZZZXYPSCYAQY5P35TWX7I56CMYAPFLJE3JEO5Q, seg_pos={0 0}, upload=64142]: Done uploading piece. err: ecclient: upload failed (node:12vricofwx67FRaxAGLbmgT1TYyKkqfJbxdsyeEPL4R7kg8pKzT, address:104.36.82.11:6125): protocol: expected piece hash; context deadline exceeded; EOF Jun 22 00:31:55.270319418 (80h8m30.833642407s) [noise=true , address=104.36.82.55:4224 , piece_id=XF7H2BYSNNNZOS2T77VIN75N76AYUXAQEI6D4LRXACU2KU62V2TA, seg_pos={0 0}, upload=64142]: Done uploading piece. err: ecclient: upload failed (node:12EjKmUiv4pxkVCB4WoiXim4uFpxrBPyfiSS4efTY3HKSE3jb5y, address:104.36.82.55:4224): protocol: write tcp 159.223.143.39:39152->104.36.82.55:4224: use of closed network connection; write tcp 159.223.143.39:39152->104.36.82.55:4224: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:39152->104.36.82.55:4224: use of closed network connection Jun 22 00:31:55.270353811 (80h8m30.833676799s) [seg_pos={0 0}, upload=64142]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Jun 22 00:31:55.301099877 (80h8m30.864422884s) [upload=64142]: Done uploading segments... Jun 22 00:31:55.301121567 (80h8m30.864444566s) [upload=64142]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 19.496983119s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [..........................X.................................] duration: 55.514129309s FAIL (2025-06-22 00:58:01 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== ing upload Jun 22 00:30:55.303646184 (80h7m30.866969171s) [upload=64142]: Uploading segments... Jun 22 00:30:55.303706406 (80h7m30.867029416s) [upload=64142]: Got next segment. Inline: false Jun 22 00:30:55.303762479 (80h7m30.867085468s) [upload=64142]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 22 00:30:55.303768627 (80h7m30.867091636s) [upload=64142]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 22 00:30:55.349058140 (80h7m30.912381168s) [seg_pos={0 0}, upload=64142]: Begin upload segment... Jun 22 00:30:55.349113768 (80h7m30.912436772s) [seg_pos={0 0}, upload=64142]: Done begin upload segment. Jun 22 00:30:55.349143121 (80h7m30.912466131s) [noise=true , address=45.91.116.5:5310 , piece_id=AXJOKDZRPVNL7HFT4QKJWRAHFM2UCM3GEVQB2KNYD6CXB6TH4D4Q, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.349660917 (80h7m30.912983928s) [noise=true , address=77.90.31.10:4533 , piece_id=SE6YAKKFDL3BV677CDDUC32UNMPXJHTG23GWWYK5B4YIPKER6GAQ, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.350093077 (80h7m30.913416089s) [noise=true , address=79.127.201.215:4326 , piece_id=USEWZDX6P52U6MCALTBNQG3SV7N2JRKNU6O23TLS6JEWAS4D7ARA, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.350480894 (80h7m30.913803916s) [noise=true , address=104.36.82.13:6199 , piece_id=YP62N2TO446YR3QRG5YLXECD7TN6SJ4UNNLZSK5H4CMV6JIF4AKA, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.350669998 (80h7m30.913993011s) [noise=true , address=104.36.82.54:4184 , piece_id=IZICVGZFMAYLIJTNUJ2OFLZVNZ6IPPYDZO4BSTJJPLHVCVOE5EAA, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.351201352 (80h7m30.914524366s) [noise=true , address=104.36.82.55:4224 , piece_id=XF7H2BYSNNNZOS2T77VIN75N76AYUXAQEI6D4LRXACU2KU62V2TA, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.351668071 (80h7m30.914991083s) [noise=true , address=104.36.82.56:4261 , piece_id=EYOYOYLDACGMPGJYX6ILQT3XHMUTQNXZEDC7ANHLZXIKYVDQFFVQ, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.352193126 (80h7m30.915516138s) [noise=true , address=104.36.82.52:4089 , piece_id=X2VG4S3UHVNWA4AQJS7FLKWNVLTOZ3AKWI5ZDH24QCA3ENQW7BXQ, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.352587096 (80h7m30.915910103s) [noise=true , address=79.127.220.98:4402 , piece_id=TE2WQO5LK45AVMPIAIGVSIBOVZLC4FG4ESQ72X6KFYC7IAEN2DRA, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.353420253 (80h7m30.916743265s) [noise=true , address=104.36.82.11:6125 , piece_id=2CX7RJOKNQA6ODZZZXYPSCYAQY5P35TWX7I56CMYAPFLJE3JEO5Q, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.353791565 (80h7m30.917114578s) [noise=true , address=104.36.82.42:6803 , piece_id=R626UFD326S2UBJEZ2WMHVR7EN4SK427VXPDJZOOZ32MX25JWZWQ, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.354171226 (80h7m30.917494233s) [noise=true , address=104.36.82.14:6245 , piece_id=II36G46OA6TINLR244U6D7Q6LGV754UA5M62K3EX7ZOBESMJUEHA, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.354542282 (80h7m30.917865288s) [noise=true , address=79.127.205.235:5660 , piece_id=UGFPVS7VNB7U2YSFPSGYDIRZV5R2DHCN57PJP6VOXO52WIAYHLXA, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.354891655 (80h7m30.918214678s) [noise=true , address=104.36.82.15:6283 , piece_id=N7TH2EMIOPATVHCSNH2WXUGU7KMMEJ5LCHR22SWEHSA5OV4CUCXQ, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.355499503 (80h7m30.918822519s) [noise=true , address=77.90.31.14:4666 , piece_id=H4DTFDAVCN266PB235BJIQTQ32FBXTIB5AZSX3B4R2VIK554AFSQ, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.356575547 (80h7m30.919898539s) [noise=true , address=104.36.82.39:6756 , piece_id=OKKWMDJPZO6D5VRMGOE3GOCYGHM725JZTNOVYAWKJW3WYZPI4V5Q, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.357039590 (80h7m30.920362596s) [noise=true , address=104.36.82.53:4146 , piece_id=3V3DVGUOPJKDWIGQ2TU7M6VFYFVAS6ODY62ERGSU6LI3LSTWQ4YQ, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.360343566 (80h7m30.923666582s) [noise=true , address=104.36.82.9:6033 , piece_id=T5XA23LZELMSTWOLBSHOX4AHKDACF65OYCKTMZYTXDIBSXNA2T4A, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.452444064 (80h7m31.015767055s) [upload=64142]: Next returned nil segment Jun 22 00:30:55.452495148 (80h7m31.015818149s) [upload=64142]: Waiting for error group managing segments... Jun 22 00:30:55.532958808 (80h7m31.096281883s) [noise=true , address=104.36.82.9:6033 , piece_id=T5XA23LZELMSTWOLBSHOX4AHKDACF65OYCKTMZYTXDIBSXNA2T4A, seg_pos={0 0}, upload=64142]: Done uploading piece. err: ecclient: upload failed (node:1SV3LrcQW16wLewDzfyUWFB6sUgnERNJVdhz72CJmyde95M79C, address:104.36.82.9:6033): protocol: storage node overloaded, request limit: 2000 Jun 22 00:30:55.547802184 (80h7m31.111125188s) [noise=true , address=79.127.220.98:4402 , piece_id=TE2WQO5LK45AVMPIAIGVSIBOVZLC4FG4ESQ72X6KFYC7IAEN2DRA, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:30:55.554223621 (80h7m31.117546616s) [noise=true , address=77.90.31.14:4666 , piece_id=H4DTFDAVCN266PB235BJIQTQ32FBXTIB5AZSX3B4R2VIK554AFSQ, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:30:55.762514255 (80h7m31.325837245s) [noise=true , address=104.36.82.14:6245 , piece_id=II36G46OA6TINLR244U6D7Q6LGV754UA5M62K3EX7ZOBESMJUEHA, seg_pos={0 0}, upload=64142]: Done uploading piece. err: ecclient: upload failed (node:1Zu8teNPTXn638grHKwtJqz4UEY7oFYLx1DHvuVP6RZXcatVqX, address:104.36.82.14:6245): protocol: storage node overloaded, request limit: 2000 Jun 22 00:30:55.773669583 (80h7m31.3369926s) [noise=true , address=79.127.205.235:5660 , piece_id=UGFPVS7VNB7U2YSFPSGYDIRZV5R2DHCN57PJP6VOXO52WIAYHLXA, seg_pos={0 0}, upload=64142]: Done uploading piece. err: ecclient: upload failed (node:12E6uCX9KL4Ud7YpxMXepvMXuqTFJCiSXByDjW94Am3U9H2Nf7f, address:79.127.205.235:5660): protocol: storage node overloaded, request limit: 2000 Jun 22 00:30:56.968366655 (80h7m32.53168964s) [noise=true , address=77.90.31.10:4533 , piece_id=SE6YAKKFDL3BV677CDDUC32UNMPXJHTG23GWWYK5B4YIPKER6GAQ, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:30:57.403572874 (80h7m32.966895883s) [noise=true , address=45.91.116.5:5310 , piece_id=AXJOKDZRPVNL7HFT4QKJWRAHFM2UCM3GEVQB2KNYD6CXB6TH4D4Q, seg_pos={0 0}, upload=64142]: Done uploading piece. err: ecclient: upload failed (node:127RMUc62xRq9RU5z3XfkZsq5sehmv3kP8o53tr9BxVcPE3GQCY, address:45.91.116.5:5310): protocol: storage node overloaded, request limit: 2000 Jun 22 00:30:58.210389994 (80h7m33.773712995s) [noise=true , address=104.36.82.13:6199 , piece_id=YP62N2TO446YR3QRG5YLXECD7TN6SJ4UNNLZSK5H4CMV6JIF4AKA, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:30:58.693037961 (80h7m34.256360966s) [noise=true , address=104.36.82.39:6756 , piece_id=OKKWMDJPZO6D5VRMGOE3GOCYGHM725JZTNOVYAWKJW3WYZPI4V5Q, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:30:59.124834814 (80h7m34.688157819s) [noise=true , address=104.36.82.15:6283 , piece_id=N7TH2EMIOPATVHCSNH2WXUGU7KMMEJ5LCHR22SWEHSA5OV4CUCXQ, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:31:00.629657982 (80h7m36.192980977s) [noise=true , address=104.36.82.42:6803 , piece_id=R626UFD326S2UBJEZ2WMHVR7EN4SK427VXPDJZOOZ32MX25JWZWQ, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:31:00.933110546 (80h7m36.496433536s) [noise=true , address=79.127.201.215:4326 , piece_id=USEWZDX6P52U6MCALTBNQG3SV7N2JRKNU6O23TLS6JEWAS4D7ARA, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:31:05.669167243 (80h7m41.232490237s) [noise=true , address=104.36.82.56:4261 , piece_id=EYOYOYLDACGMPGJYX6ILQT3XHMUTQNXZEDC7ANHLZXIKYVDQFFVQ, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:31:08.443582763 (80h7m44.006905763s) [noise=true , address=104.36.82.54:4184 , piece_id=IZICVGZFMAYLIJTNUJ2OFLZVNZ6IPPYDZO4BSTJJPLHVCVOE5EAA, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:31:26.459138014 (80h8m2.022461013s) [noise=true , address=104.36.82.53:4146 , piece_id=3V3DVGUOPJKDWIGQ2TU7M6VFYFVAS6ODY62ERGSU6LI3LSTWQ4YQ, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:31:27.456050336 (80h8m3.019373336s) [noise=true , address=104.36.82.52:4089 , piece_id=X2VG4S3UHVNWA4AQJS7FLKWNVLTOZ3AKWI5ZDH24QCA3ENQW7BXQ, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:31:55.270234251 (80h8m30.833557256s) [noise=true , address=104.36.82.11:6125 , piece_id=2CX7RJOKNQA6ODZZZXYPSCYAQY5P35TWX7I56CMYAPFLJE3JEO5Q, seg_pos={0 0}, upload=64142]: Done uploading piece. err: ecclient: upload failed (node:12vricofwx67FRaxAGLbmgT1TYyKkqfJbxdsyeEPL4R7kg8pKzT, address:104.36.82.11:6125): protocol: expected piece hash; context deadline exceeded; EOF Jun 22 00:31:55.270319418 (80h8m30.833642407s) [noise=true , address=104.36.82.55:4224 , piece_id=XF7H2BYSNNNZOS2T77VIN75N76AYUXAQEI6D4LRXACU2KU62V2TA, seg_pos={0 0}, upload=64142]: Done uploading piece. err: ecclient: upload failed (node:12EjKmUiv4pxkVCB4WoiXim4uFpxrBPyfiSS4efTY3HKSE3jb5y, address:104.36.82.55:4224): protocol: write tcp 159.223.143.39:39152->104.36.82.55:4224: use of closed network connection; write tcp 159.223.143.39:39152->104.36.82.55:4224: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:39152->104.36.82.55:4224: use of closed network connection Jun 22 00:31:55.270353811 (80h8m30.833676799s) [seg_pos={0 0}, upload=64142]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Jun 22 00:31:55.301099877 (80h8m30.864422884s) [upload=64142]: Done uploading segments... Jun 22 00:31:55.301121567 (80h8m30.864444566s) [upload=64142]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 55.514152688s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [.........X...................X..............................] duration: 14.501403367s FAIL (2025-06-22 00:57:11 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 14.501449099s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [...........................X................................] duration: 19.939418112s FAIL (2025-06-22 00:57:06 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== ing upload Jun 22 00:30:55.303646184 (80h7m30.866969171s) [upload=64142]: Uploading segments... Jun 22 00:30:55.303706406 (80h7m30.867029416s) [upload=64142]: Got next segment. Inline: false Jun 22 00:30:55.303762479 (80h7m30.867085468s) [upload=64142]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 22 00:30:55.303768627 (80h7m30.867091636s) [upload=64142]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 22 00:30:55.349058140 (80h7m30.912381168s) [seg_pos={0 0}, upload=64142]: Begin upload segment... Jun 22 00:30:55.349113768 (80h7m30.912436772s) [seg_pos={0 0}, upload=64142]: Done begin upload segment. Jun 22 00:30:55.349143121 (80h7m30.912466131s) [noise=true , address=45.91.116.5:5310 , piece_id=AXJOKDZRPVNL7HFT4QKJWRAHFM2UCM3GEVQB2KNYD6CXB6TH4D4Q, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.349660917 (80h7m30.912983928s) [noise=true , address=77.90.31.10:4533 , piece_id=SE6YAKKFDL3BV677CDDUC32UNMPXJHTG23GWWYK5B4YIPKER6GAQ, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.350093077 (80h7m30.913416089s) [noise=true , address=79.127.201.215:4326 , piece_id=USEWZDX6P52U6MCALTBNQG3SV7N2JRKNU6O23TLS6JEWAS4D7ARA, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.350480894 (80h7m30.913803916s) [noise=true , address=104.36.82.13:6199 , piece_id=YP62N2TO446YR3QRG5YLXECD7TN6SJ4UNNLZSK5H4CMV6JIF4AKA, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.350669998 (80h7m30.913993011s) [noise=true , address=104.36.82.54:4184 , piece_id=IZICVGZFMAYLIJTNUJ2OFLZVNZ6IPPYDZO4BSTJJPLHVCVOE5EAA, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.351201352 (80h7m30.914524366s) [noise=true , address=104.36.82.55:4224 , piece_id=XF7H2BYSNNNZOS2T77VIN75N76AYUXAQEI6D4LRXACU2KU62V2TA, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.351668071 (80h7m30.914991083s) [noise=true , address=104.36.82.56:4261 , piece_id=EYOYOYLDACGMPGJYX6ILQT3XHMUTQNXZEDC7ANHLZXIKYVDQFFVQ, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.352193126 (80h7m30.915516138s) [noise=true , address=104.36.82.52:4089 , piece_id=X2VG4S3UHVNWA4AQJS7FLKWNVLTOZ3AKWI5ZDH24QCA3ENQW7BXQ, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.352587096 (80h7m30.915910103s) [noise=true , address=79.127.220.98:4402 , piece_id=TE2WQO5LK45AVMPIAIGVSIBOVZLC4FG4ESQ72X6KFYC7IAEN2DRA, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.353420253 (80h7m30.916743265s) [noise=true , address=104.36.82.11:6125 , piece_id=2CX7RJOKNQA6ODZZZXYPSCYAQY5P35TWX7I56CMYAPFLJE3JEO5Q, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.353791565 (80h7m30.917114578s) [noise=true , address=104.36.82.42:6803 , piece_id=R626UFD326S2UBJEZ2WMHVR7EN4SK427VXPDJZOOZ32MX25JWZWQ, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.354171226 (80h7m30.917494233s) [noise=true , address=104.36.82.14:6245 , piece_id=II36G46OA6TINLR244U6D7Q6LGV754UA5M62K3EX7ZOBESMJUEHA, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.354542282 (80h7m30.917865288s) [noise=true , address=79.127.205.235:5660 , piece_id=UGFPVS7VNB7U2YSFPSGYDIRZV5R2DHCN57PJP6VOXO52WIAYHLXA, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.354891655 (80h7m30.918214678s) [noise=true , address=104.36.82.15:6283 , piece_id=N7TH2EMIOPATVHCSNH2WXUGU7KMMEJ5LCHR22SWEHSA5OV4CUCXQ, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.355499503 (80h7m30.918822519s) [noise=true , address=77.90.31.14:4666 , piece_id=H4DTFDAVCN266PB235BJIQTQ32FBXTIB5AZSX3B4R2VIK554AFSQ, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.356575547 (80h7m30.919898539s) [noise=true , address=104.36.82.39:6756 , piece_id=OKKWMDJPZO6D5VRMGOE3GOCYGHM725JZTNOVYAWKJW3WYZPI4V5Q, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.357039590 (80h7m30.920362596s) [noise=true , address=104.36.82.53:4146 , piece_id=3V3DVGUOPJKDWIGQ2TU7M6VFYFVAS6ODY62ERGSU6LI3LSTWQ4YQ, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.360343566 (80h7m30.923666582s) [noise=true , address=104.36.82.9:6033 , piece_id=T5XA23LZELMSTWOLBSHOX4AHKDACF65OYCKTMZYTXDIBSXNA2T4A, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.452444064 (80h7m31.015767055s) [upload=64142]: Next returned nil segment Jun 22 00:30:55.452495148 (80h7m31.015818149s) [upload=64142]: Waiting for error group managing segments... Jun 22 00:30:55.532958808 (80h7m31.096281883s) [noise=true , address=104.36.82.9:6033 , piece_id=T5XA23LZELMSTWOLBSHOX4AHKDACF65OYCKTMZYTXDIBSXNA2T4A, seg_pos={0 0}, upload=64142]: Done uploading piece. err: ecclient: upload failed (node:1SV3LrcQW16wLewDzfyUWFB6sUgnERNJVdhz72CJmyde95M79C, address:104.36.82.9:6033): protocol: storage node overloaded, request limit: 2000 Jun 22 00:30:55.547802184 (80h7m31.111125188s) [noise=true , address=79.127.220.98:4402 , piece_id=TE2WQO5LK45AVMPIAIGVSIBOVZLC4FG4ESQ72X6KFYC7IAEN2DRA, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:30:55.554223621 (80h7m31.117546616s) [noise=true , address=77.90.31.14:4666 , piece_id=H4DTFDAVCN266PB235BJIQTQ32FBXTIB5AZSX3B4R2VIK554AFSQ, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:30:55.762514255 (80h7m31.325837245s) [noise=true , address=104.36.82.14:6245 , piece_id=II36G46OA6TINLR244U6D7Q6LGV754UA5M62K3EX7ZOBESMJUEHA, seg_pos={0 0}, upload=64142]: Done uploading piece. err: ecclient: upload failed (node:1Zu8teNPTXn638grHKwtJqz4UEY7oFYLx1DHvuVP6RZXcatVqX, address:104.36.82.14:6245): protocol: storage node overloaded, request limit: 2000 Jun 22 00:30:55.773669583 (80h7m31.3369926s) [noise=true , address=79.127.205.235:5660 , piece_id=UGFPVS7VNB7U2YSFPSGYDIRZV5R2DHCN57PJP6VOXO52WIAYHLXA, seg_pos={0 0}, upload=64142]: Done uploading piece. err: ecclient: upload failed (node:12E6uCX9KL4Ud7YpxMXepvMXuqTFJCiSXByDjW94Am3U9H2Nf7f, address:79.127.205.235:5660): protocol: storage node overloaded, request limit: 2000 Jun 22 00:30:56.968366655 (80h7m32.53168964s) [noise=true , address=77.90.31.10:4533 , piece_id=SE6YAKKFDL3BV677CDDUC32UNMPXJHTG23GWWYK5B4YIPKER6GAQ, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:30:57.403572874 (80h7m32.966895883s) [noise=true , address=45.91.116.5:5310 , piece_id=AXJOKDZRPVNL7HFT4QKJWRAHFM2UCM3GEVQB2KNYD6CXB6TH4D4Q, seg_pos={0 0}, upload=64142]: Done uploading piece. err: ecclient: upload failed (node:127RMUc62xRq9RU5z3XfkZsq5sehmv3kP8o53tr9BxVcPE3GQCY, address:45.91.116.5:5310): protocol: storage node overloaded, request limit: 2000 Jun 22 00:30:58.210389994 (80h7m33.773712995s) [noise=true , address=104.36.82.13:6199 , piece_id=YP62N2TO446YR3QRG5YLXECD7TN6SJ4UNNLZSK5H4CMV6JIF4AKA, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:30:58.693037961 (80h7m34.256360966s) [noise=true , address=104.36.82.39:6756 , piece_id=OKKWMDJPZO6D5VRMGOE3GOCYGHM725JZTNOVYAWKJW3WYZPI4V5Q, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:30:59.124834814 (80h7m34.688157819s) [noise=true , address=104.36.82.15:6283 , piece_id=N7TH2EMIOPATVHCSNH2WXUGU7KMMEJ5LCHR22SWEHSA5OV4CUCXQ, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:31:00.629657982 (80h7m36.192980977s) [noise=true , address=104.36.82.42:6803 , piece_id=R626UFD326S2UBJEZ2WMHVR7EN4SK427VXPDJZOOZ32MX25JWZWQ, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:31:00.933110546 (80h7m36.496433536s) [noise=true , address=79.127.201.215:4326 , piece_id=USEWZDX6P52U6MCALTBNQG3SV7N2JRKNU6O23TLS6JEWAS4D7ARA, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:31:05.669167243 (80h7m41.232490237s) [noise=true , address=104.36.82.56:4261 , piece_id=EYOYOYLDACGMPGJYX6ILQT3XHMUTQNXZEDC7ANHLZXIKYVDQFFVQ, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:31:08.443582763 (80h7m44.006905763s) [noise=true , address=104.36.82.54:4184 , piece_id=IZICVGZFMAYLIJTNUJ2OFLZVNZ6IPPYDZO4BSTJJPLHVCVOE5EAA, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:31:26.459138014 (80h8m2.022461013s) [noise=true , address=104.36.82.53:4146 , piece_id=3V3DVGUOPJKDWIGQ2TU7M6VFYFVAS6ODY62ERGSU6LI3LSTWQ4YQ, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:31:27.456050336 (80h8m3.019373336s) [noise=true , address=104.36.82.52:4089 , piece_id=X2VG4S3UHVNWA4AQJS7FLKWNVLTOZ3AKWI5ZDH24QCA3ENQW7BXQ, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:31:55.270234251 (80h8m30.833557256s) [noise=true , address=104.36.82.11:6125 , piece_id=2CX7RJOKNQA6ODZZZXYPSCYAQY5P35TWX7I56CMYAPFLJE3JEO5Q, seg_pos={0 0}, upload=64142]: Done uploading piece. err: ecclient: upload failed (node:12vricofwx67FRaxAGLbmgT1TYyKkqfJbxdsyeEPL4R7kg8pKzT, address:104.36.82.11:6125): protocol: expected piece hash; context deadline exceeded; EOF Jun 22 00:31:55.270319418 (80h8m30.833642407s) [noise=true , address=104.36.82.55:4224 , piece_id=XF7H2BYSNNNZOS2T77VIN75N76AYUXAQEI6D4LRXACU2KU62V2TA, seg_pos={0 0}, upload=64142]: Done uploading piece. err: ecclient: upload failed (node:12EjKmUiv4pxkVCB4WoiXim4uFpxrBPyfiSS4efTY3HKSE3jb5y, address:104.36.82.55:4224): protocol: write tcp 159.223.143.39:39152->104.36.82.55:4224: use of closed network connection; write tcp 159.223.143.39:39152->104.36.82.55:4224: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:39152->104.36.82.55:4224: use of closed network connection Jun 22 00:31:55.270353811 (80h8m30.833676799s) [seg_pos={0 0}, upload=64142]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Jun 22 00:31:55.301099877 (80h8m30.864422884s) [upload=64142]: Done uploading segments... Jun 22 00:31:55.301121567 (80h8m30.864444566s) [upload=64142]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 19.939469187s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [X.........X...................X.............................] duration: 44.208652841s FAIL (2025-06-22 00:56:29 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 44.208742003s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [............................X...............................] duration: 29.168386893s FAIL (2025-06-22 00:56:24 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== ing upload Jun 22 00:30:55.303646184 (80h7m30.866969171s) [upload=64142]: Uploading segments... Jun 22 00:30:55.303706406 (80h7m30.867029416s) [upload=64142]: Got next segment. Inline: false Jun 22 00:30:55.303762479 (80h7m30.867085468s) [upload=64142]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 22 00:30:55.303768627 (80h7m30.867091636s) [upload=64142]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 22 00:30:55.349058140 (80h7m30.912381168s) [seg_pos={0 0}, upload=64142]: Begin upload segment... Jun 22 00:30:55.349113768 (80h7m30.912436772s) [seg_pos={0 0}, upload=64142]: Done begin upload segment. Jun 22 00:30:55.349143121 (80h7m30.912466131s) [noise=true , address=45.91.116.5:5310 , piece_id=AXJOKDZRPVNL7HFT4QKJWRAHFM2UCM3GEVQB2KNYD6CXB6TH4D4Q, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.349660917 (80h7m30.912983928s) [noise=true , address=77.90.31.10:4533 , piece_id=SE6YAKKFDL3BV677CDDUC32UNMPXJHTG23GWWYK5B4YIPKER6GAQ, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.350093077 (80h7m30.913416089s) [noise=true , address=79.127.201.215:4326 , piece_id=USEWZDX6P52U6MCALTBNQG3SV7N2JRKNU6O23TLS6JEWAS4D7ARA, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.350480894 (80h7m30.913803916s) [noise=true , address=104.36.82.13:6199 , piece_id=YP62N2TO446YR3QRG5YLXECD7TN6SJ4UNNLZSK5H4CMV6JIF4AKA, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.350669998 (80h7m30.913993011s) [noise=true , address=104.36.82.54:4184 , piece_id=IZICVGZFMAYLIJTNUJ2OFLZVNZ6IPPYDZO4BSTJJPLHVCVOE5EAA, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.351201352 (80h7m30.914524366s) [noise=true , address=104.36.82.55:4224 , piece_id=XF7H2BYSNNNZOS2T77VIN75N76AYUXAQEI6D4LRXACU2KU62V2TA, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.351668071 (80h7m30.914991083s) [noise=true , address=104.36.82.56:4261 , piece_id=EYOYOYLDACGMPGJYX6ILQT3XHMUTQNXZEDC7ANHLZXIKYVDQFFVQ, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.352193126 (80h7m30.915516138s) [noise=true , address=104.36.82.52:4089 , piece_id=X2VG4S3UHVNWA4AQJS7FLKWNVLTOZ3AKWI5ZDH24QCA3ENQW7BXQ, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.352587096 (80h7m30.915910103s) [noise=true , address=79.127.220.98:4402 , piece_id=TE2WQO5LK45AVMPIAIGVSIBOVZLC4FG4ESQ72X6KFYC7IAEN2DRA, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.353420253 (80h7m30.916743265s) [noise=true , address=104.36.82.11:6125 , piece_id=2CX7RJOKNQA6ODZZZXYPSCYAQY5P35TWX7I56CMYAPFLJE3JEO5Q, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.353791565 (80h7m30.917114578s) [noise=true , address=104.36.82.42:6803 , piece_id=R626UFD326S2UBJEZ2WMHVR7EN4SK427VXPDJZOOZ32MX25JWZWQ, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.354171226 (80h7m30.917494233s) [noise=true , address=104.36.82.14:6245 , piece_id=II36G46OA6TINLR244U6D7Q6LGV754UA5M62K3EX7ZOBESMJUEHA, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.354542282 (80h7m30.917865288s) [noise=true , address=79.127.205.235:5660 , piece_id=UGFPVS7VNB7U2YSFPSGYDIRZV5R2DHCN57PJP6VOXO52WIAYHLXA, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.354891655 (80h7m30.918214678s) [noise=true , address=104.36.82.15:6283 , piece_id=N7TH2EMIOPATVHCSNH2WXUGU7KMMEJ5LCHR22SWEHSA5OV4CUCXQ, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.355499503 (80h7m30.918822519s) [noise=true , address=77.90.31.14:4666 , piece_id=H4DTFDAVCN266PB235BJIQTQ32FBXTIB5AZSX3B4R2VIK554AFSQ, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.356575547 (80h7m30.919898539s) [noise=true , address=104.36.82.39:6756 , piece_id=OKKWMDJPZO6D5VRMGOE3GOCYGHM725JZTNOVYAWKJW3WYZPI4V5Q, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.357039590 (80h7m30.920362596s) [noise=true , address=104.36.82.53:4146 , piece_id=3V3DVGUOPJKDWIGQ2TU7M6VFYFVAS6ODY62ERGSU6LI3LSTWQ4YQ, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.360343566 (80h7m30.923666582s) [noise=true , address=104.36.82.9:6033 , piece_id=T5XA23LZELMSTWOLBSHOX4AHKDACF65OYCKTMZYTXDIBSXNA2T4A, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.452444064 (80h7m31.015767055s) [upload=64142]: Next returned nil segment Jun 22 00:30:55.452495148 (80h7m31.015818149s) [upload=64142]: Waiting for error group managing segments... Jun 22 00:30:55.532958808 (80h7m31.096281883s) [noise=true , address=104.36.82.9:6033 , piece_id=T5XA23LZELMSTWOLBSHOX4AHKDACF65OYCKTMZYTXDIBSXNA2T4A, seg_pos={0 0}, upload=64142]: Done uploading piece. err: ecclient: upload failed (node:1SV3LrcQW16wLewDzfyUWFB6sUgnERNJVdhz72CJmyde95M79C, address:104.36.82.9:6033): protocol: storage node overloaded, request limit: 2000 Jun 22 00:30:55.547802184 (80h7m31.111125188s) [noise=true , address=79.127.220.98:4402 , piece_id=TE2WQO5LK45AVMPIAIGVSIBOVZLC4FG4ESQ72X6KFYC7IAEN2DRA, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:30:55.554223621 (80h7m31.117546616s) [noise=true , address=77.90.31.14:4666 , piece_id=H4DTFDAVCN266PB235BJIQTQ32FBXTIB5AZSX3B4R2VIK554AFSQ, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:30:55.762514255 (80h7m31.325837245s) [noise=true , address=104.36.82.14:6245 , piece_id=II36G46OA6TINLR244U6D7Q6LGV754UA5M62K3EX7ZOBESMJUEHA, seg_pos={0 0}, upload=64142]: Done uploading piece. err: ecclient: upload failed (node:1Zu8teNPTXn638grHKwtJqz4UEY7oFYLx1DHvuVP6RZXcatVqX, address:104.36.82.14:6245): protocol: storage node overloaded, request limit: 2000 Jun 22 00:30:55.773669583 (80h7m31.3369926s) [noise=true , address=79.127.205.235:5660 , piece_id=UGFPVS7VNB7U2YSFPSGYDIRZV5R2DHCN57PJP6VOXO52WIAYHLXA, seg_pos={0 0}, upload=64142]: Done uploading piece. err: ecclient: upload failed (node:12E6uCX9KL4Ud7YpxMXepvMXuqTFJCiSXByDjW94Am3U9H2Nf7f, address:79.127.205.235:5660): protocol: storage node overloaded, request limit: 2000 Jun 22 00:30:56.968366655 (80h7m32.53168964s) [noise=true , address=77.90.31.10:4533 , piece_id=SE6YAKKFDL3BV677CDDUC32UNMPXJHTG23GWWYK5B4YIPKER6GAQ, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:30:57.403572874 (80h7m32.966895883s) [noise=true , address=45.91.116.5:5310 , piece_id=AXJOKDZRPVNL7HFT4QKJWRAHFM2UCM3GEVQB2KNYD6CXB6TH4D4Q, seg_pos={0 0}, upload=64142]: Done uploading piece. err: ecclient: upload failed (node:127RMUc62xRq9RU5z3XfkZsq5sehmv3kP8o53tr9BxVcPE3GQCY, address:45.91.116.5:5310): protocol: storage node overloaded, request limit: 2000 Jun 22 00:30:58.210389994 (80h7m33.773712995s) [noise=true , address=104.36.82.13:6199 , piece_id=YP62N2TO446YR3QRG5YLXECD7TN6SJ4UNNLZSK5H4CMV6JIF4AKA, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:30:58.693037961 (80h7m34.256360966s) [noise=true , address=104.36.82.39:6756 , piece_id=OKKWMDJPZO6D5VRMGOE3GOCYGHM725JZTNOVYAWKJW3WYZPI4V5Q, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:30:59.124834814 (80h7m34.688157819s) [noise=true , address=104.36.82.15:6283 , piece_id=N7TH2EMIOPATVHCSNH2WXUGU7KMMEJ5LCHR22SWEHSA5OV4CUCXQ, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:31:00.629657982 (80h7m36.192980977s) [noise=true , address=104.36.82.42:6803 , piece_id=R626UFD326S2UBJEZ2WMHVR7EN4SK427VXPDJZOOZ32MX25JWZWQ, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:31:00.933110546 (80h7m36.496433536s) [noise=true , address=79.127.201.215:4326 , piece_id=USEWZDX6P52U6MCALTBNQG3SV7N2JRKNU6O23TLS6JEWAS4D7ARA, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:31:05.669167243 (80h7m41.232490237s) [noise=true , address=104.36.82.56:4261 , piece_id=EYOYOYLDACGMPGJYX6ILQT3XHMUTQNXZEDC7ANHLZXIKYVDQFFVQ, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:31:08.443582763 (80h7m44.006905763s) [noise=true , address=104.36.82.54:4184 , piece_id=IZICVGZFMAYLIJTNUJ2OFLZVNZ6IPPYDZO4BSTJJPLHVCVOE5EAA, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:31:26.459138014 (80h8m2.022461013s) [noise=true , address=104.36.82.53:4146 , piece_id=3V3DVGUOPJKDWIGQ2TU7M6VFYFVAS6ODY62ERGSU6LI3LSTWQ4YQ, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:31:27.456050336 (80h8m3.019373336s) [noise=true , address=104.36.82.52:4089 , piece_id=X2VG4S3UHVNWA4AQJS7FLKWNVLTOZ3AKWI5ZDH24QCA3ENQW7BXQ, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:31:55.270234251 (80h8m30.833557256s) [noise=true , address=104.36.82.11:6125 , piece_id=2CX7RJOKNQA6ODZZZXYPSCYAQY5P35TWX7I56CMYAPFLJE3JEO5Q, seg_pos={0 0}, upload=64142]: Done uploading piece. err: ecclient: upload failed (node:12vricofwx67FRaxAGLbmgT1TYyKkqfJbxdsyeEPL4R7kg8pKzT, address:104.36.82.11:6125): protocol: expected piece hash; context deadline exceeded; EOF Jun 22 00:31:55.270319418 (80h8m30.833642407s) [noise=true , address=104.36.82.55:4224 , piece_id=XF7H2BYSNNNZOS2T77VIN75N76AYUXAQEI6D4LRXACU2KU62V2TA, seg_pos={0 0}, upload=64142]: Done uploading piece. err: ecclient: upload failed (node:12EjKmUiv4pxkVCB4WoiXim4uFpxrBPyfiSS4efTY3HKSE3jb5y, address:104.36.82.55:4224): protocol: write tcp 159.223.143.39:39152->104.36.82.55:4224: use of closed network connection; write tcp 159.223.143.39:39152->104.36.82.55:4224: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:39152->104.36.82.55:4224: use of closed network connection Jun 22 00:31:55.270353811 (80h8m30.833676799s) [seg_pos={0 0}, upload=64142]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Jun 22 00:31:55.301099877 (80h8m30.864422884s) [upload=64142]: Done uploading segments... Jun 22 00:31:55.301121567 (80h8m30.864444566s) [upload=64142]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 29.168540913s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.............................X..............................] duration: 18.603542637s FAIL (2025-06-22 00:55:55 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== ing upload Jun 22 00:30:55.303646184 (80h7m30.866969171s) [upload=64142]: Uploading segments... Jun 22 00:30:55.303706406 (80h7m30.867029416s) [upload=64142]: Got next segment. Inline: false Jun 22 00:30:55.303762479 (80h7m30.867085468s) [upload=64142]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 22 00:30:55.303768627 (80h7m30.867091636s) [upload=64142]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 22 00:30:55.349058140 (80h7m30.912381168s) [seg_pos={0 0}, upload=64142]: Begin upload segment... Jun 22 00:30:55.349113768 (80h7m30.912436772s) [seg_pos={0 0}, upload=64142]: Done begin upload segment. Jun 22 00:30:55.349143121 (80h7m30.912466131s) [noise=true , address=45.91.116.5:5310 , piece_id=AXJOKDZRPVNL7HFT4QKJWRAHFM2UCM3GEVQB2KNYD6CXB6TH4D4Q, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.349660917 (80h7m30.912983928s) [noise=true , address=77.90.31.10:4533 , piece_id=SE6YAKKFDL3BV677CDDUC32UNMPXJHTG23GWWYK5B4YIPKER6GAQ, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.350093077 (80h7m30.913416089s) [noise=true , address=79.127.201.215:4326 , piece_id=USEWZDX6P52U6MCALTBNQG3SV7N2JRKNU6O23TLS6JEWAS4D7ARA, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.350480894 (80h7m30.913803916s) [noise=true , address=104.36.82.13:6199 , piece_id=YP62N2TO446YR3QRG5YLXECD7TN6SJ4UNNLZSK5H4CMV6JIF4AKA, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.350669998 (80h7m30.913993011s) [noise=true , address=104.36.82.54:4184 , piece_id=IZICVGZFMAYLIJTNUJ2OFLZVNZ6IPPYDZO4BSTJJPLHVCVOE5EAA, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.351201352 (80h7m30.914524366s) [noise=true , address=104.36.82.55:4224 , piece_id=XF7H2BYSNNNZOS2T77VIN75N76AYUXAQEI6D4LRXACU2KU62V2TA, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.351668071 (80h7m30.914991083s) [noise=true , address=104.36.82.56:4261 , piece_id=EYOYOYLDACGMPGJYX6ILQT3XHMUTQNXZEDC7ANHLZXIKYVDQFFVQ, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.352193126 (80h7m30.915516138s) [noise=true , address=104.36.82.52:4089 , piece_id=X2VG4S3UHVNWA4AQJS7FLKWNVLTOZ3AKWI5ZDH24QCA3ENQW7BXQ, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.352587096 (80h7m30.915910103s) [noise=true , address=79.127.220.98:4402 , piece_id=TE2WQO5LK45AVMPIAIGVSIBOVZLC4FG4ESQ72X6KFYC7IAEN2DRA, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.353420253 (80h7m30.916743265s) [noise=true , address=104.36.82.11:6125 , piece_id=2CX7RJOKNQA6ODZZZXYPSCYAQY5P35TWX7I56CMYAPFLJE3JEO5Q, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.353791565 (80h7m30.917114578s) [noise=true , address=104.36.82.42:6803 , piece_id=R626UFD326S2UBJEZ2WMHVR7EN4SK427VXPDJZOOZ32MX25JWZWQ, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.354171226 (80h7m30.917494233s) [noise=true , address=104.36.82.14:6245 , piece_id=II36G46OA6TINLR244U6D7Q6LGV754UA5M62K3EX7ZOBESMJUEHA, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.354542282 (80h7m30.917865288s) [noise=true , address=79.127.205.235:5660 , piece_id=UGFPVS7VNB7U2YSFPSGYDIRZV5R2DHCN57PJP6VOXO52WIAYHLXA, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.354891655 (80h7m30.918214678s) [noise=true , address=104.36.82.15:6283 , piece_id=N7TH2EMIOPATVHCSNH2WXUGU7KMMEJ5LCHR22SWEHSA5OV4CUCXQ, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.355499503 (80h7m30.918822519s) [noise=true , address=77.90.31.14:4666 , piece_id=H4DTFDAVCN266PB235BJIQTQ32FBXTIB5AZSX3B4R2VIK554AFSQ, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.356575547 (80h7m30.919898539s) [noise=true , address=104.36.82.39:6756 , piece_id=OKKWMDJPZO6D5VRMGOE3GOCYGHM725JZTNOVYAWKJW3WYZPI4V5Q, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.357039590 (80h7m30.920362596s) [noise=true , address=104.36.82.53:4146 , piece_id=3V3DVGUOPJKDWIGQ2TU7M6VFYFVAS6ODY62ERGSU6LI3LSTWQ4YQ, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.360343566 (80h7m30.923666582s) [noise=true , address=104.36.82.9:6033 , piece_id=T5XA23LZELMSTWOLBSHOX4AHKDACF65OYCKTMZYTXDIBSXNA2T4A, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.452444064 (80h7m31.015767055s) [upload=64142]: Next returned nil segment Jun 22 00:30:55.452495148 (80h7m31.015818149s) [upload=64142]: Waiting for error group managing segments... Jun 22 00:30:55.532958808 (80h7m31.096281883s) [noise=true , address=104.36.82.9:6033 , piece_id=T5XA23LZELMSTWOLBSHOX4AHKDACF65OYCKTMZYTXDIBSXNA2T4A, seg_pos={0 0}, upload=64142]: Done uploading piece. err: ecclient: upload failed (node:1SV3LrcQW16wLewDzfyUWFB6sUgnERNJVdhz72CJmyde95M79C, address:104.36.82.9:6033): protocol: storage node overloaded, request limit: 2000 Jun 22 00:30:55.547802184 (80h7m31.111125188s) [noise=true , address=79.127.220.98:4402 , piece_id=TE2WQO5LK45AVMPIAIGVSIBOVZLC4FG4ESQ72X6KFYC7IAEN2DRA, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:30:55.554223621 (80h7m31.117546616s) [noise=true , address=77.90.31.14:4666 , piece_id=H4DTFDAVCN266PB235BJIQTQ32FBXTIB5AZSX3B4R2VIK554AFSQ, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:30:55.762514255 (80h7m31.325837245s) [noise=true , address=104.36.82.14:6245 , piece_id=II36G46OA6TINLR244U6D7Q6LGV754UA5M62K3EX7ZOBESMJUEHA, seg_pos={0 0}, upload=64142]: Done uploading piece. err: ecclient: upload failed (node:1Zu8teNPTXn638grHKwtJqz4UEY7oFYLx1DHvuVP6RZXcatVqX, address:104.36.82.14:6245): protocol: storage node overloaded, request limit: 2000 Jun 22 00:30:55.773669583 (80h7m31.3369926s) [noise=true , address=79.127.205.235:5660 , piece_id=UGFPVS7VNB7U2YSFPSGYDIRZV5R2DHCN57PJP6VOXO52WIAYHLXA, seg_pos={0 0}, upload=64142]: Done uploading piece. err: ecclient: upload failed (node:12E6uCX9KL4Ud7YpxMXepvMXuqTFJCiSXByDjW94Am3U9H2Nf7f, address:79.127.205.235:5660): protocol: storage node overloaded, request limit: 2000 Jun 22 00:30:56.968366655 (80h7m32.53168964s) [noise=true , address=77.90.31.10:4533 , piece_id=SE6YAKKFDL3BV677CDDUC32UNMPXJHTG23GWWYK5B4YIPKER6GAQ, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:30:57.403572874 (80h7m32.966895883s) [noise=true , address=45.91.116.5:5310 , piece_id=AXJOKDZRPVNL7HFT4QKJWRAHFM2UCM3GEVQB2KNYD6CXB6TH4D4Q, seg_pos={0 0}, upload=64142]: Done uploading piece. err: ecclient: upload failed (node:127RMUc62xRq9RU5z3XfkZsq5sehmv3kP8o53tr9BxVcPE3GQCY, address:45.91.116.5:5310): protocol: storage node overloaded, request limit: 2000 Jun 22 00:30:58.210389994 (80h7m33.773712995s) [noise=true , address=104.36.82.13:6199 , piece_id=YP62N2TO446YR3QRG5YLXECD7TN6SJ4UNNLZSK5H4CMV6JIF4AKA, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:30:58.693037961 (80h7m34.256360966s) [noise=true , address=104.36.82.39:6756 , piece_id=OKKWMDJPZO6D5VRMGOE3GOCYGHM725JZTNOVYAWKJW3WYZPI4V5Q, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:30:59.124834814 (80h7m34.688157819s) [noise=true , address=104.36.82.15:6283 , piece_id=N7TH2EMIOPATVHCSNH2WXUGU7KMMEJ5LCHR22SWEHSA5OV4CUCXQ, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:31:00.629657982 (80h7m36.192980977s) [noise=true , address=104.36.82.42:6803 , piece_id=R626UFD326S2UBJEZ2WMHVR7EN4SK427VXPDJZOOZ32MX25JWZWQ, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:31:00.933110546 (80h7m36.496433536s) [noise=true , address=79.127.201.215:4326 , piece_id=USEWZDX6P52U6MCALTBNQG3SV7N2JRKNU6O23TLS6JEWAS4D7ARA, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:31:05.669167243 (80h7m41.232490237s) [noise=true , address=104.36.82.56:4261 , piece_id=EYOYOYLDACGMPGJYX6ILQT3XHMUTQNXZEDC7ANHLZXIKYVDQFFVQ, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:31:08.443582763 (80h7m44.006905763s) [noise=true , address=104.36.82.54:4184 , piece_id=IZICVGZFMAYLIJTNUJ2OFLZVNZ6IPPYDZO4BSTJJPLHVCVOE5EAA, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:31:26.459138014 (80h8m2.022461013s) [noise=true , address=104.36.82.53:4146 , piece_id=3V3DVGUOPJKDWIGQ2TU7M6VFYFVAS6ODY62ERGSU6LI3LSTWQ4YQ, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:31:27.456050336 (80h8m3.019373336s) [noise=true , address=104.36.82.52:4089 , piece_id=X2VG4S3UHVNWA4AQJS7FLKWNVLTOZ3AKWI5ZDH24QCA3ENQW7BXQ, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:31:55.270234251 (80h8m30.833557256s) [noise=true , address=104.36.82.11:6125 , piece_id=2CX7RJOKNQA6ODZZZXYPSCYAQY5P35TWX7I56CMYAPFLJE3JEO5Q, seg_pos={0 0}, upload=64142]: Done uploading piece. err: ecclient: upload failed (node:12vricofwx67FRaxAGLbmgT1TYyKkqfJbxdsyeEPL4R7kg8pKzT, address:104.36.82.11:6125): protocol: expected piece hash; context deadline exceeded; EOF Jun 22 00:31:55.270319418 (80h8m30.833642407s) [noise=true , address=104.36.82.55:4224 , piece_id=XF7H2BYSNNNZOS2T77VIN75N76AYUXAQEI6D4LRXACU2KU62V2TA, seg_pos={0 0}, upload=64142]: Done uploading piece. err: ecclient: upload failed (node:12EjKmUiv4pxkVCB4WoiXim4uFpxrBPyfiSS4efTY3HKSE3jb5y, address:104.36.82.55:4224): protocol: write tcp 159.223.143.39:39152->104.36.82.55:4224: use of closed network connection; write tcp 159.223.143.39:39152->104.36.82.55:4224: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:39152->104.36.82.55:4224: use of closed network connection Jun 22 00:31:55.270353811 (80h8m30.833676799s) [seg_pos={0 0}, upload=64142]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Jun 22 00:31:55.301099877 (80h8m30.864422884s) [upload=64142]: Done uploading segments... Jun 22 00:31:55.301121567 (80h8m30.864444566s) [upload=64142]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 18.603584695s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [..............................X.............................] duration: 23.710883324s FAIL (2025-06-22 00:55:36 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== ing upload Jun 22 00:30:55.303646184 (80h7m30.866969171s) [upload=64142]: Uploading segments... Jun 22 00:30:55.303706406 (80h7m30.867029416s) [upload=64142]: Got next segment. Inline: false Jun 22 00:30:55.303762479 (80h7m30.867085468s) [upload=64142]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 22 00:30:55.303768627 (80h7m30.867091636s) [upload=64142]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 22 00:30:55.349058140 (80h7m30.912381168s) [seg_pos={0 0}, upload=64142]: Begin upload segment... Jun 22 00:30:55.349113768 (80h7m30.912436772s) [seg_pos={0 0}, upload=64142]: Done begin upload segment. Jun 22 00:30:55.349143121 (80h7m30.912466131s) [noise=true , address=45.91.116.5:5310 , piece_id=AXJOKDZRPVNL7HFT4QKJWRAHFM2UCM3GEVQB2KNYD6CXB6TH4D4Q, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.349660917 (80h7m30.912983928s) [noise=true , address=77.90.31.10:4533 , piece_id=SE6YAKKFDL3BV677CDDUC32UNMPXJHTG23GWWYK5B4YIPKER6GAQ, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.350093077 (80h7m30.913416089s) [noise=true , address=79.127.201.215:4326 , piece_id=USEWZDX6P52U6MCALTBNQG3SV7N2JRKNU6O23TLS6JEWAS4D7ARA, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.350480894 (80h7m30.913803916s) [noise=true , address=104.36.82.13:6199 , piece_id=YP62N2TO446YR3QRG5YLXECD7TN6SJ4UNNLZSK5H4CMV6JIF4AKA, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.350669998 (80h7m30.913993011s) [noise=true , address=104.36.82.54:4184 , piece_id=IZICVGZFMAYLIJTNUJ2OFLZVNZ6IPPYDZO4BSTJJPLHVCVOE5EAA, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.351201352 (80h7m30.914524366s) [noise=true , address=104.36.82.55:4224 , piece_id=XF7H2BYSNNNZOS2T77VIN75N76AYUXAQEI6D4LRXACU2KU62V2TA, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.351668071 (80h7m30.914991083s) [noise=true , address=104.36.82.56:4261 , piece_id=EYOYOYLDACGMPGJYX6ILQT3XHMUTQNXZEDC7ANHLZXIKYVDQFFVQ, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.352193126 (80h7m30.915516138s) [noise=true , address=104.36.82.52:4089 , piece_id=X2VG4S3UHVNWA4AQJS7FLKWNVLTOZ3AKWI5ZDH24QCA3ENQW7BXQ, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.352587096 (80h7m30.915910103s) [noise=true , address=79.127.220.98:4402 , piece_id=TE2WQO5LK45AVMPIAIGVSIBOVZLC4FG4ESQ72X6KFYC7IAEN2DRA, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.353420253 (80h7m30.916743265s) [noise=true , address=104.36.82.11:6125 , piece_id=2CX7RJOKNQA6ODZZZXYPSCYAQY5P35TWX7I56CMYAPFLJE3JEO5Q, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.353791565 (80h7m30.917114578s) [noise=true , address=104.36.82.42:6803 , piece_id=R626UFD326S2UBJEZ2WMHVR7EN4SK427VXPDJZOOZ32MX25JWZWQ, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.354171226 (80h7m30.917494233s) [noise=true , address=104.36.82.14:6245 , piece_id=II36G46OA6TINLR244U6D7Q6LGV754UA5M62K3EX7ZOBESMJUEHA, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.354542282 (80h7m30.917865288s) [noise=true , address=79.127.205.235:5660 , piece_id=UGFPVS7VNB7U2YSFPSGYDIRZV5R2DHCN57PJP6VOXO52WIAYHLXA, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.354891655 (80h7m30.918214678s) [noise=true , address=104.36.82.15:6283 , piece_id=N7TH2EMIOPATVHCSNH2WXUGU7KMMEJ5LCHR22SWEHSA5OV4CUCXQ, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.355499503 (80h7m30.918822519s) [noise=true , address=77.90.31.14:4666 , piece_id=H4DTFDAVCN266PB235BJIQTQ32FBXTIB5AZSX3B4R2VIK554AFSQ, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.356575547 (80h7m30.919898539s) [noise=true , address=104.36.82.39:6756 , piece_id=OKKWMDJPZO6D5VRMGOE3GOCYGHM725JZTNOVYAWKJW3WYZPI4V5Q, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.357039590 (80h7m30.920362596s) [noise=true , address=104.36.82.53:4146 , piece_id=3V3DVGUOPJKDWIGQ2TU7M6VFYFVAS6ODY62ERGSU6LI3LSTWQ4YQ, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.360343566 (80h7m30.923666582s) [noise=true , address=104.36.82.9:6033 , piece_id=T5XA23LZELMSTWOLBSHOX4AHKDACF65OYCKTMZYTXDIBSXNA2T4A, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.452444064 (80h7m31.015767055s) [upload=64142]: Next returned nil segment Jun 22 00:30:55.452495148 (80h7m31.015818149s) [upload=64142]: Waiting for error group managing segments... Jun 22 00:30:55.532958808 (80h7m31.096281883s) [noise=true , address=104.36.82.9:6033 , piece_id=T5XA23LZELMSTWOLBSHOX4AHKDACF65OYCKTMZYTXDIBSXNA2T4A, seg_pos={0 0}, upload=64142]: Done uploading piece. err: ecclient: upload failed (node:1SV3LrcQW16wLewDzfyUWFB6sUgnERNJVdhz72CJmyde95M79C, address:104.36.82.9:6033): protocol: storage node overloaded, request limit: 2000 Jun 22 00:30:55.547802184 (80h7m31.111125188s) [noise=true , address=79.127.220.98:4402 , piece_id=TE2WQO5LK45AVMPIAIGVSIBOVZLC4FG4ESQ72X6KFYC7IAEN2DRA, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:30:55.554223621 (80h7m31.117546616s) [noise=true , address=77.90.31.14:4666 , piece_id=H4DTFDAVCN266PB235BJIQTQ32FBXTIB5AZSX3B4R2VIK554AFSQ, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:30:55.762514255 (80h7m31.325837245s) [noise=true , address=104.36.82.14:6245 , piece_id=II36G46OA6TINLR244U6D7Q6LGV754UA5M62K3EX7ZOBESMJUEHA, seg_pos={0 0}, upload=64142]: Done uploading piece. err: ecclient: upload failed (node:1Zu8teNPTXn638grHKwtJqz4UEY7oFYLx1DHvuVP6RZXcatVqX, address:104.36.82.14:6245): protocol: storage node overloaded, request limit: 2000 Jun 22 00:30:55.773669583 (80h7m31.3369926s) [noise=true , address=79.127.205.235:5660 , piece_id=UGFPVS7VNB7U2YSFPSGYDIRZV5R2DHCN57PJP6VOXO52WIAYHLXA, seg_pos={0 0}, upload=64142]: Done uploading piece. err: ecclient: upload failed (node:12E6uCX9KL4Ud7YpxMXepvMXuqTFJCiSXByDjW94Am3U9H2Nf7f, address:79.127.205.235:5660): protocol: storage node overloaded, request limit: 2000 Jun 22 00:30:56.968366655 (80h7m32.53168964s) [noise=true , address=77.90.31.10:4533 , piece_id=SE6YAKKFDL3BV677CDDUC32UNMPXJHTG23GWWYK5B4YIPKER6GAQ, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:30:57.403572874 (80h7m32.966895883s) [noise=true , address=45.91.116.5:5310 , piece_id=AXJOKDZRPVNL7HFT4QKJWRAHFM2UCM3GEVQB2KNYD6CXB6TH4D4Q, seg_pos={0 0}, upload=64142]: Done uploading piece. err: ecclient: upload failed (node:127RMUc62xRq9RU5z3XfkZsq5sehmv3kP8o53tr9BxVcPE3GQCY, address:45.91.116.5:5310): protocol: storage node overloaded, request limit: 2000 Jun 22 00:30:58.210389994 (80h7m33.773712995s) [noise=true , address=104.36.82.13:6199 , piece_id=YP62N2TO446YR3QRG5YLXECD7TN6SJ4UNNLZSK5H4CMV6JIF4AKA, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:30:58.693037961 (80h7m34.256360966s) [noise=true , address=104.36.82.39:6756 , piece_id=OKKWMDJPZO6D5VRMGOE3GOCYGHM725JZTNOVYAWKJW3WYZPI4V5Q, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:30:59.124834814 (80h7m34.688157819s) [noise=true , address=104.36.82.15:6283 , piece_id=N7TH2EMIOPATVHCSNH2WXUGU7KMMEJ5LCHR22SWEHSA5OV4CUCXQ, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:31:00.629657982 (80h7m36.192980977s) [noise=true , address=104.36.82.42:6803 , piece_id=R626UFD326S2UBJEZ2WMHVR7EN4SK427VXPDJZOOZ32MX25JWZWQ, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:31:00.933110546 (80h7m36.496433536s) [noise=true , address=79.127.201.215:4326 , piece_id=USEWZDX6P52U6MCALTBNQG3SV7N2JRKNU6O23TLS6JEWAS4D7ARA, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:31:05.669167243 (80h7m41.232490237s) [noise=true , address=104.36.82.56:4261 , piece_id=EYOYOYLDACGMPGJYX6ILQT3XHMUTQNXZEDC7ANHLZXIKYVDQFFVQ, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:31:08.443582763 (80h7m44.006905763s) [noise=true , address=104.36.82.54:4184 , piece_id=IZICVGZFMAYLIJTNUJ2OFLZVNZ6IPPYDZO4BSTJJPLHVCVOE5EAA, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:31:26.459138014 (80h8m2.022461013s) [noise=true , address=104.36.82.53:4146 , piece_id=3V3DVGUOPJKDWIGQ2TU7M6VFYFVAS6ODY62ERGSU6LI3LSTWQ4YQ, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:31:27.456050336 (80h8m3.019373336s) [noise=true , address=104.36.82.52:4089 , piece_id=X2VG4S3UHVNWA4AQJS7FLKWNVLTOZ3AKWI5ZDH24QCA3ENQW7BXQ, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:31:55.270234251 (80h8m30.833557256s) [noise=true , address=104.36.82.11:6125 , piece_id=2CX7RJOKNQA6ODZZZXYPSCYAQY5P35TWX7I56CMYAPFLJE3JEO5Q, seg_pos={0 0}, upload=64142]: Done uploading piece. err: ecclient: upload failed (node:12vricofwx67FRaxAGLbmgT1TYyKkqfJbxdsyeEPL4R7kg8pKzT, address:104.36.82.11:6125): protocol: expected piece hash; context deadline exceeded; EOF Jun 22 00:31:55.270319418 (80h8m30.833642407s) [noise=true , address=104.36.82.55:4224 , piece_id=XF7H2BYSNNNZOS2T77VIN75N76AYUXAQEI6D4LRXACU2KU62V2TA, seg_pos={0 0}, upload=64142]: Done uploading piece. err: ecclient: upload failed (node:12EjKmUiv4pxkVCB4WoiXim4uFpxrBPyfiSS4efTY3HKSE3jb5y, address:104.36.82.55:4224): protocol: write tcp 159.223.143.39:39152->104.36.82.55:4224: use of closed network connection; write tcp 159.223.143.39:39152->104.36.82.55:4224: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:39152->104.36.82.55:4224: use of closed network connection Jun 22 00:31:55.270353811 (80h8m30.833676799s) [seg_pos={0 0}, upload=64142]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Jun 22 00:31:55.301099877 (80h8m30.864422884s) [upload=64142]: Done uploading segments... Jun 22 00:31:55.301121567 (80h8m30.864444566s) [upload=64142]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 23.71090663s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [.X.........X...................X............................] duration: 16.369442592s FAIL (2025-06-22 00:55:19 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 16.369484577s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [X..............................X............................] duration: 35.217875761s FAIL (2025-06-22 00:55:12 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== ing upload Jun 22 00:30:55.303646184 (80h7m30.866969171s) [upload=64142]: Uploading segments... Jun 22 00:30:55.303706406 (80h7m30.867029416s) [upload=64142]: Got next segment. Inline: false Jun 22 00:30:55.303762479 (80h7m30.867085468s) [upload=64142]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 22 00:30:55.303768627 (80h7m30.867091636s) [upload=64142]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 22 00:30:55.349058140 (80h7m30.912381168s) [seg_pos={0 0}, upload=64142]: Begin upload segment... Jun 22 00:30:55.349113768 (80h7m30.912436772s) [seg_pos={0 0}, upload=64142]: Done begin upload segment. Jun 22 00:30:55.349143121 (80h7m30.912466131s) [noise=true , address=45.91.116.5:5310 , piece_id=AXJOKDZRPVNL7HFT4QKJWRAHFM2UCM3GEVQB2KNYD6CXB6TH4D4Q, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.349660917 (80h7m30.912983928s) [noise=true , address=77.90.31.10:4533 , piece_id=SE6YAKKFDL3BV677CDDUC32UNMPXJHTG23GWWYK5B4YIPKER6GAQ, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.350093077 (80h7m30.913416089s) [noise=true , address=79.127.201.215:4326 , piece_id=USEWZDX6P52U6MCALTBNQG3SV7N2JRKNU6O23TLS6JEWAS4D7ARA, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.350480894 (80h7m30.913803916s) [noise=true , address=104.36.82.13:6199 , piece_id=YP62N2TO446YR3QRG5YLXECD7TN6SJ4UNNLZSK5H4CMV6JIF4AKA, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.350669998 (80h7m30.913993011s) [noise=true , address=104.36.82.54:4184 , piece_id=IZICVGZFMAYLIJTNUJ2OFLZVNZ6IPPYDZO4BSTJJPLHVCVOE5EAA, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.351201352 (80h7m30.914524366s) [noise=true , address=104.36.82.55:4224 , piece_id=XF7H2BYSNNNZOS2T77VIN75N76AYUXAQEI6D4LRXACU2KU62V2TA, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.351668071 (80h7m30.914991083s) [noise=true , address=104.36.82.56:4261 , piece_id=EYOYOYLDACGMPGJYX6ILQT3XHMUTQNXZEDC7ANHLZXIKYVDQFFVQ, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.352193126 (80h7m30.915516138s) [noise=true , address=104.36.82.52:4089 , piece_id=X2VG4S3UHVNWA4AQJS7FLKWNVLTOZ3AKWI5ZDH24QCA3ENQW7BXQ, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.352587096 (80h7m30.915910103s) [noise=true , address=79.127.220.98:4402 , piece_id=TE2WQO5LK45AVMPIAIGVSIBOVZLC4FG4ESQ72X6KFYC7IAEN2DRA, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.353420253 (80h7m30.916743265s) [noise=true , address=104.36.82.11:6125 , piece_id=2CX7RJOKNQA6ODZZZXYPSCYAQY5P35TWX7I56CMYAPFLJE3JEO5Q, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.353791565 (80h7m30.917114578s) [noise=true , address=104.36.82.42:6803 , piece_id=R626UFD326S2UBJEZ2WMHVR7EN4SK427VXPDJZOOZ32MX25JWZWQ, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.354171226 (80h7m30.917494233s) [noise=true , address=104.36.82.14:6245 , piece_id=II36G46OA6TINLR244U6D7Q6LGV754UA5M62K3EX7ZOBESMJUEHA, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.354542282 (80h7m30.917865288s) [noise=true , address=79.127.205.235:5660 , piece_id=UGFPVS7VNB7U2YSFPSGYDIRZV5R2DHCN57PJP6VOXO52WIAYHLXA, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.354891655 (80h7m30.918214678s) [noise=true , address=104.36.82.15:6283 , piece_id=N7TH2EMIOPATVHCSNH2WXUGU7KMMEJ5LCHR22SWEHSA5OV4CUCXQ, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.355499503 (80h7m30.918822519s) [noise=true , address=77.90.31.14:4666 , piece_id=H4DTFDAVCN266PB235BJIQTQ32FBXTIB5AZSX3B4R2VIK554AFSQ, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.356575547 (80h7m30.919898539s) [noise=true , address=104.36.82.39:6756 , piece_id=OKKWMDJPZO6D5VRMGOE3GOCYGHM725JZTNOVYAWKJW3WYZPI4V5Q, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.357039590 (80h7m30.920362596s) [noise=true , address=104.36.82.53:4146 , piece_id=3V3DVGUOPJKDWIGQ2TU7M6VFYFVAS6ODY62ERGSU6LI3LSTWQ4YQ, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.360343566 (80h7m30.923666582s) [noise=true , address=104.36.82.9:6033 , piece_id=T5XA23LZELMSTWOLBSHOX4AHKDACF65OYCKTMZYTXDIBSXNA2T4A, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.452444064 (80h7m31.015767055s) [upload=64142]: Next returned nil segment Jun 22 00:30:55.452495148 (80h7m31.015818149s) [upload=64142]: Waiting for error group managing segments... Jun 22 00:30:55.532958808 (80h7m31.096281883s) [noise=true , address=104.36.82.9:6033 , piece_id=T5XA23LZELMSTWOLBSHOX4AHKDACF65OYCKTMZYTXDIBSXNA2T4A, seg_pos={0 0}, upload=64142]: Done uploading piece. err: ecclient: upload failed (node:1SV3LrcQW16wLewDzfyUWFB6sUgnERNJVdhz72CJmyde95M79C, address:104.36.82.9:6033): protocol: storage node overloaded, request limit: 2000 Jun 22 00:30:55.547802184 (80h7m31.111125188s) [noise=true , address=79.127.220.98:4402 , piece_id=TE2WQO5LK45AVMPIAIGVSIBOVZLC4FG4ESQ72X6KFYC7IAEN2DRA, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:30:55.554223621 (80h7m31.117546616s) [noise=true , address=77.90.31.14:4666 , piece_id=H4DTFDAVCN266PB235BJIQTQ32FBXTIB5AZSX3B4R2VIK554AFSQ, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:30:55.762514255 (80h7m31.325837245s) [noise=true , address=104.36.82.14:6245 , piece_id=II36G46OA6TINLR244U6D7Q6LGV754UA5M62K3EX7ZOBESMJUEHA, seg_pos={0 0}, upload=64142]: Done uploading piece. err: ecclient: upload failed (node:1Zu8teNPTXn638grHKwtJqz4UEY7oFYLx1DHvuVP6RZXcatVqX, address:104.36.82.14:6245): protocol: storage node overloaded, request limit: 2000 Jun 22 00:30:55.773669583 (80h7m31.3369926s) [noise=true , address=79.127.205.235:5660 , piece_id=UGFPVS7VNB7U2YSFPSGYDIRZV5R2DHCN57PJP6VOXO52WIAYHLXA, seg_pos={0 0}, upload=64142]: Done uploading piece. err: ecclient: upload failed (node:12E6uCX9KL4Ud7YpxMXepvMXuqTFJCiSXByDjW94Am3U9H2Nf7f, address:79.127.205.235:5660): protocol: storage node overloaded, request limit: 2000 Jun 22 00:30:56.968366655 (80h7m32.53168964s) [noise=true , address=77.90.31.10:4533 , piece_id=SE6YAKKFDL3BV677CDDUC32UNMPXJHTG23GWWYK5B4YIPKER6GAQ, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:30:57.403572874 (80h7m32.966895883s) [noise=true , address=45.91.116.5:5310 , piece_id=AXJOKDZRPVNL7HFT4QKJWRAHFM2UCM3GEVQB2KNYD6CXB6TH4D4Q, seg_pos={0 0}, upload=64142]: Done uploading piece. err: ecclient: upload failed (node:127RMUc62xRq9RU5z3XfkZsq5sehmv3kP8o53tr9BxVcPE3GQCY, address:45.91.116.5:5310): protocol: storage node overloaded, request limit: 2000 Jun 22 00:30:58.210389994 (80h7m33.773712995s) [noise=true , address=104.36.82.13:6199 , piece_id=YP62N2TO446YR3QRG5YLXECD7TN6SJ4UNNLZSK5H4CMV6JIF4AKA, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:30:58.693037961 (80h7m34.256360966s) [noise=true , address=104.36.82.39:6756 , piece_id=OKKWMDJPZO6D5VRMGOE3GOCYGHM725JZTNOVYAWKJW3WYZPI4V5Q, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:30:59.124834814 (80h7m34.688157819s) [noise=true , address=104.36.82.15:6283 , piece_id=N7TH2EMIOPATVHCSNH2WXUGU7KMMEJ5LCHR22SWEHSA5OV4CUCXQ, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:31:00.629657982 (80h7m36.192980977s) [noise=true , address=104.36.82.42:6803 , piece_id=R626UFD326S2UBJEZ2WMHVR7EN4SK427VXPDJZOOZ32MX25JWZWQ, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:31:00.933110546 (80h7m36.496433536s) [noise=true , address=79.127.201.215:4326 , piece_id=USEWZDX6P52U6MCALTBNQG3SV7N2JRKNU6O23TLS6JEWAS4D7ARA, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:31:05.669167243 (80h7m41.232490237s) [noise=true , address=104.36.82.56:4261 , piece_id=EYOYOYLDACGMPGJYX6ILQT3XHMUTQNXZEDC7ANHLZXIKYVDQFFVQ, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:31:08.443582763 (80h7m44.006905763s) [noise=true , address=104.36.82.54:4184 , piece_id=IZICVGZFMAYLIJTNUJ2OFLZVNZ6IPPYDZO4BSTJJPLHVCVOE5EAA, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:31:26.459138014 (80h8m2.022461013s) [noise=true , address=104.36.82.53:4146 , piece_id=3V3DVGUOPJKDWIGQ2TU7M6VFYFVAS6ODY62ERGSU6LI3LSTWQ4YQ, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:31:27.456050336 (80h8m3.019373336s) [noise=true , address=104.36.82.52:4089 , piece_id=X2VG4S3UHVNWA4AQJS7FLKWNVLTOZ3AKWI5ZDH24QCA3ENQW7BXQ, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:31:55.270234251 (80h8m30.833557256s) [noise=true , address=104.36.82.11:6125 , piece_id=2CX7RJOKNQA6ODZZZXYPSCYAQY5P35TWX7I56CMYAPFLJE3JEO5Q, seg_pos={0 0}, upload=64142]: Done uploading piece. err: ecclient: upload failed (node:12vricofwx67FRaxAGLbmgT1TYyKkqfJbxdsyeEPL4R7kg8pKzT, address:104.36.82.11:6125): protocol: expected piece hash; context deadline exceeded; EOF Jun 22 00:31:55.270319418 (80h8m30.833642407s) [noise=true , address=104.36.82.55:4224 , piece_id=XF7H2BYSNNNZOS2T77VIN75N76AYUXAQEI6D4LRXACU2KU62V2TA, seg_pos={0 0}, upload=64142]: Done uploading piece. err: ecclient: upload failed (node:12EjKmUiv4pxkVCB4WoiXim4uFpxrBPyfiSS4efTY3HKSE3jb5y, address:104.36.82.55:4224): protocol: write tcp 159.223.143.39:39152->104.36.82.55:4224: use of closed network connection; write tcp 159.223.143.39:39152->104.36.82.55:4224: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:39152->104.36.82.55:4224: use of closed network connection Jun 22 00:31:55.270353811 (80h8m30.833676799s) [seg_pos={0 0}, upload=64142]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Jun 22 00:31:55.301099877 (80h8m30.864422884s) [upload=64142]: Done uploading segments... Jun 22 00:31:55.301121567 (80h8m30.864444566s) [upload=64142]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 35.217908084s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [..X.........X...................X...........................] duration: 12.496495413s FAIL (2025-06-22 00:54:13 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 12.496528968s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [.X..............................X...........................] duration: 28.616751004s FAIL (2025-06-22 00:53:51 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== ing upload Jun 22 00:30:55.303646184 (80h7m30.866969171s) [upload=64142]: Uploading segments... Jun 22 00:30:55.303706406 (80h7m30.867029416s) [upload=64142]: Got next segment. Inline: false Jun 22 00:30:55.303762479 (80h7m30.867085468s) [upload=64142]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 22 00:30:55.303768627 (80h7m30.867091636s) [upload=64142]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 22 00:30:55.349058140 (80h7m30.912381168s) [seg_pos={0 0}, upload=64142]: Begin upload segment... Jun 22 00:30:55.349113768 (80h7m30.912436772s) [seg_pos={0 0}, upload=64142]: Done begin upload segment. Jun 22 00:30:55.349143121 (80h7m30.912466131s) [noise=true , address=45.91.116.5:5310 , piece_id=AXJOKDZRPVNL7HFT4QKJWRAHFM2UCM3GEVQB2KNYD6CXB6TH4D4Q, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.349660917 (80h7m30.912983928s) [noise=true , address=77.90.31.10:4533 , piece_id=SE6YAKKFDL3BV677CDDUC32UNMPXJHTG23GWWYK5B4YIPKER6GAQ, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.350093077 (80h7m30.913416089s) [noise=true , address=79.127.201.215:4326 , piece_id=USEWZDX6P52U6MCALTBNQG3SV7N2JRKNU6O23TLS6JEWAS4D7ARA, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.350480894 (80h7m30.913803916s) [noise=true , address=104.36.82.13:6199 , piece_id=YP62N2TO446YR3QRG5YLXECD7TN6SJ4UNNLZSK5H4CMV6JIF4AKA, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.350669998 (80h7m30.913993011s) [noise=true , address=104.36.82.54:4184 , piece_id=IZICVGZFMAYLIJTNUJ2OFLZVNZ6IPPYDZO4BSTJJPLHVCVOE5EAA, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.351201352 (80h7m30.914524366s) [noise=true , address=104.36.82.55:4224 , piece_id=XF7H2BYSNNNZOS2T77VIN75N76AYUXAQEI6D4LRXACU2KU62V2TA, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.351668071 (80h7m30.914991083s) [noise=true , address=104.36.82.56:4261 , piece_id=EYOYOYLDACGMPGJYX6ILQT3XHMUTQNXZEDC7ANHLZXIKYVDQFFVQ, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.352193126 (80h7m30.915516138s) [noise=true , address=104.36.82.52:4089 , piece_id=X2VG4S3UHVNWA4AQJS7FLKWNVLTOZ3AKWI5ZDH24QCA3ENQW7BXQ, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.352587096 (80h7m30.915910103s) [noise=true , address=79.127.220.98:4402 , piece_id=TE2WQO5LK45AVMPIAIGVSIBOVZLC4FG4ESQ72X6KFYC7IAEN2DRA, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.353420253 (80h7m30.916743265s) [noise=true , address=104.36.82.11:6125 , piece_id=2CX7RJOKNQA6ODZZZXYPSCYAQY5P35TWX7I56CMYAPFLJE3JEO5Q, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.353791565 (80h7m30.917114578s) [noise=true , address=104.36.82.42:6803 , piece_id=R626UFD326S2UBJEZ2WMHVR7EN4SK427VXPDJZOOZ32MX25JWZWQ, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.354171226 (80h7m30.917494233s) [noise=true , address=104.36.82.14:6245 , piece_id=II36G46OA6TINLR244U6D7Q6LGV754UA5M62K3EX7ZOBESMJUEHA, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.354542282 (80h7m30.917865288s) [noise=true , address=79.127.205.235:5660 , piece_id=UGFPVS7VNB7U2YSFPSGYDIRZV5R2DHCN57PJP6VOXO52WIAYHLXA, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.354891655 (80h7m30.918214678s) [noise=true , address=104.36.82.15:6283 , piece_id=N7TH2EMIOPATVHCSNH2WXUGU7KMMEJ5LCHR22SWEHSA5OV4CUCXQ, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.355499503 (80h7m30.918822519s) [noise=true , address=77.90.31.14:4666 , piece_id=H4DTFDAVCN266PB235BJIQTQ32FBXTIB5AZSX3B4R2VIK554AFSQ, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.356575547 (80h7m30.919898539s) [noise=true , address=104.36.82.39:6756 , piece_id=OKKWMDJPZO6D5VRMGOE3GOCYGHM725JZTNOVYAWKJW3WYZPI4V5Q, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.357039590 (80h7m30.920362596s) [noise=true , address=104.36.82.53:4146 , piece_id=3V3DVGUOPJKDWIGQ2TU7M6VFYFVAS6ODY62ERGSU6LI3LSTWQ4YQ, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.360343566 (80h7m30.923666582s) [noise=true , address=104.36.82.9:6033 , piece_id=T5XA23LZELMSTWOLBSHOX4AHKDACF65OYCKTMZYTXDIBSXNA2T4A, seg_pos={0 0}, upload=64142]: Uploading piece... Jun 22 00:30:55.452444064 (80h7m31.015767055s) [upload=64142]: Next returned nil segment Jun 22 00:30:55.452495148 (80h7m31.015818149s) [upload=64142]: Waiting for error group managing segments... Jun 22 00:30:55.532958808 (80h7m31.096281883s) [noise=true , address=104.36.82.9:6033 , piece_id=T5XA23LZELMSTWOLBSHOX4AHKDACF65OYCKTMZYTXDIBSXNA2T4A, seg_pos={0 0}, upload=64142]: Done uploading piece. err: ecclient: upload failed (node:1SV3LrcQW16wLewDzfyUWFB6sUgnERNJVdhz72CJmyde95M79C, address:104.36.82.9:6033): protocol: storage node overloaded, request limit: 2000 Jun 22 00:30:55.547802184 (80h7m31.111125188s) [noise=true , address=79.127.220.98:4402 , piece_id=TE2WQO5LK45AVMPIAIGVSIBOVZLC4FG4ESQ72X6KFYC7IAEN2DRA, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:30:55.554223621 (80h7m31.117546616s) [noise=true , address=77.90.31.14:4666 , piece_id=H4DTFDAVCN266PB235BJIQTQ32FBXTIB5AZSX3B4R2VIK554AFSQ, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:30:55.762514255 (80h7m31.325837245s) [noise=true , address=104.36.82.14:6245 , piece_id=II36G46OA6TINLR244U6D7Q6LGV754UA5M62K3EX7ZOBESMJUEHA, seg_pos={0 0}, upload=64142]: Done uploading piece. err: ecclient: upload failed (node:1Zu8teNPTXn638grHKwtJqz4UEY7oFYLx1DHvuVP6RZXcatVqX, address:104.36.82.14:6245): protocol: storage node overloaded, request limit: 2000 Jun 22 00:30:55.773669583 (80h7m31.3369926s) [noise=true , address=79.127.205.235:5660 , piece_id=UGFPVS7VNB7U2YSFPSGYDIRZV5R2DHCN57PJP6VOXO52WIAYHLXA, seg_pos={0 0}, upload=64142]: Done uploading piece. err: ecclient: upload failed (node:12E6uCX9KL4Ud7YpxMXepvMXuqTFJCiSXByDjW94Am3U9H2Nf7f, address:79.127.205.235:5660): protocol: storage node overloaded, request limit: 2000 Jun 22 00:30:56.968366655 (80h7m32.53168964s) [noise=true , address=77.90.31.10:4533 , piece_id=SE6YAKKFDL3BV677CDDUC32UNMPXJHTG23GWWYK5B4YIPKER6GAQ, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:30:57.403572874 (80h7m32.966895883s) [noise=true , address=45.91.116.5:5310 , piece_id=AXJOKDZRPVNL7HFT4QKJWRAHFM2UCM3GEVQB2KNYD6CXB6TH4D4Q, seg_pos={0 0}, upload=64142]: Done uploading piece. err: ecclient: upload failed (node:127RMUc62xRq9RU5z3XfkZsq5sehmv3kP8o53tr9BxVcPE3GQCY, address:45.91.116.5:5310): protocol: storage node overloaded, request limit: 2000 Jun 22 00:30:58.210389994 (80h7m33.773712995s) [noise=true , address=104.36.82.13:6199 , piece_id=YP62N2TO446YR3QRG5YLXECD7TN6SJ4UNNLZSK5H4CMV6JIF4AKA, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:30:58.693037961 (80h7m34.256360966s) [noise=true , address=104.36.82.39:6756 , piece_id=OKKWMDJPZO6D5VRMGOE3GOCYGHM725JZTNOVYAWKJW3WYZPI4V5Q, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:30:59.124834814 (80h7m34.688157819s) [noise=true , address=104.36.82.15:6283 , piece_id=N7TH2EMIOPATVHCSNH2WXUGU7KMMEJ5LCHR22SWEHSA5OV4CUCXQ, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:31:00.629657982 (80h7m36.192980977s) [noise=true , address=104.36.82.42:6803 , piece_id=R626UFD326S2UBJEZ2WMHVR7EN4SK427VXPDJZOOZ32MX25JWZWQ, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:31:00.933110546 (80h7m36.496433536s) [noise=true , address=79.127.201.215:4326 , piece_id=USEWZDX6P52U6MCALTBNQG3SV7N2JRKNU6O23TLS6JEWAS4D7ARA, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:31:05.669167243 (80h7m41.232490237s) [noise=true , address=104.36.82.56:4261 , piece_id=EYOYOYLDACGMPGJYX6ILQT3XHMUTQNXZEDC7ANHLZXIKYVDQFFVQ, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:31:08.443582763 (80h7m44.006905763s) [noise=true , address=104.36.82.54:4184 , piece_id=IZICVGZFMAYLIJTNUJ2OFLZVNZ6IPPYDZO4BSTJJPLHVCVOE5EAA, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:31:26.459138014 (80h8m2.022461013s) [noise=true , address=104.36.82.53:4146 , piece_id=3V3DVGUOPJKDWIGQ2TU7M6VFYFVAS6ODY62ERGSU6LI3LSTWQ4YQ, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:31:27.456050336 (80h8m3.019373336s) [noise=true , address=104.36.82.52:4089 , piece_id=X2VG4S3UHVNWA4AQJS7FLKWNVLTOZ3AKWI5ZDH24QCA3ENQW7BXQ, seg_pos={0 0}, upload=64142]: Done uploading piece. err: <nil> Jun 22 00:31:55.270234251 (80h8m30.833557256s) [noise=true , address=104.36.82.11:6125 , piece_id=2CX7RJOKNQA6ODZZZXYPSCYAQY5P35TWX7I56CMYAPFLJE3JEO5Q, seg_pos={0 0}, upload=64142]: Done uploading piece. err: ecclient: upload failed (node:12vricofwx67FRaxAGLbmgT1TYyKkqfJbxdsyeEPL4R7kg8pKzT, address:104.36.82.11:6125): protocol: expected piece hash; context deadline exceeded; EOF Jun 22 00:31:55.270319418 (80h8m30.833642407s) [noise=true , address=104.36.82.55:4224 , piece_id=XF7H2BYSNNNZOS2T77VIN75N76AYUXAQEI6D4LRXACU2KU62V2TA, seg_pos={0 0}, upload=64142]: Done uploading piece. err: ecclient: upload failed (node:12EjKmUiv4pxkVCB4WoiXim4uFpxrBPyfiSS4efTY3HKSE3jb5y, address:104.36.82.55:4224): protocol: write tcp 159.223.143.39:39152->104.36.82.55:4224: use of closed network connection; write tcp 159.223.143.39:39152->104.36.82.55:4224: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:39152->104.36.82.55:4224: use of closed network connection Jun 22 00:31:55.270353811 (80h8m30.833676799s) [seg_pos={0 0}, upload=64142]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Jun 22 00:31:55.301099877 (80h8m30.864422884s) [upload=64142]: Done uploading segments... Jun 22 00:31:55.301121567 (80h8m30.864444566s) [upload=64142]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 28.616786041s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [...X.........X...................X..........................] duration: 16.762915244s FAIL (2025-06-22 00:53:34 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 16.762933793s core uplink us-select-1 us1-placement-20

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)