Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [X.............................X...........X.................] duration: 11.616804285s FAIL (2025-06-23 01:41:43 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== oad=84082]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jun 23 01:27:43.602110205 (105h4m19.165433214s) [upload=84082]: Done uploading segments... Jun 23 01:27:43.602149595 (105h4m19.165472598s) [upload=84082]: Upload finished. err: <nil> Jun 23 01:28:26.568078065 (105h5m2.131401068s) [upload=84097]: Starting upload Jun 23 01:28:26.568176623 (105h5m2.131499629s) [upload=84097]: Done starting upload Jun 23 01:28:26.568333698 (105h5m2.131656699s) [upload=84097]: Uploading segments... Jun 23 01:28:26.569025334 (105h5m2.132348341s) [upload=84097]: Got next segment. Inline: false Jun 23 01:28:26.569144232 (105h5m2.132467237s) [upload=84097]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 23 01:28:26.569226194 (105h5m2.132549206s) [upload=84097]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 23 01:28:26.603382637 (105h5m2.166705641s) [seg_pos={0 0}, upload=84097]: Begin upload segment... Jun 23 01:28:26.603439808 (105h5m2.16676282s) [seg_pos={0 0}, upload=84097]: Done begin upload segment. Jun 23 01:28:26.603481538 (105h5m2.166804559s) [noise=true , address=79.127.219.38:4031 , piece_id=TBIY2ZMYVVKMYV4LVGJEKCJXMXYVJKSZKKBXASVNEUU2P7DKZKIQ, seg_pos={0 0}, upload=84097]: Uploading piece... Jun 23 01:28:26.603498030 (105h5m2.166821033s) [noise=true , address=77.90.31.15:4729 , piece_id=AAVBJ7VEQX67HQBXQ5DO3O5XJ37TRKU3PS2XXEH3WARMSEAQGQ2A, seg_pos={0 0}, upload=84097]: Uploading piece... Jun 23 01:28:26.603958736 (105h5m2.16728174s) [noise=true , address=79.127.219.51:4209 , piece_id=ZHJC5QUBGOHT3HRY4XPPADV7CYINQRXT4ID53KVC6LYXM7IV533A, seg_pos={0 0}, upload=84097]: Uploading piece... Jun 23 01:28:26.603997417 (105h5m2.167320407s) [noise=true , address=79.127.220.98:4410 , piece_id=DLW3G47XXWAJ4FZAZDEL6TCIN5FKG26LUNVELDWSUE2GDH6IPBLQ, seg_pos={0 0}, upload=84097]: Uploading piece... Jun 23 01:28:26.604297025 (105h5m2.167620026s) [noise=true , address=45.91.116.10:5084 , piece_id=WCIMJN6XDXUO2VCRKLAVOPVVQUEQDGHR4NP4KQGY27NEUPCD5OFA, seg_pos={0 0}, upload=84097]: Uploading piece... Jun 23 01:28:26.604692251 (105h5m2.16801525s) [noise=true , address=104.36.82.55:4217 , piece_id=WQIZ3WHXBLCKNJ4ZVQRC2PQIRH6IVVO7J3UMM5V5GXEMODV36KLQ, seg_pos={0 0}, upload=84097]: Uploading piece... Jun 23 01:28:26.605044347 (105h5m2.168367349s) [noise=true , address=104.36.82.53:4142 , piece_id=CL2WQNZHZNZM5BFJQIBUBQ7FQCIJSK4AR7PR2OLBIXFULPMBTGIQ, seg_pos={0 0}, upload=84097]: Uploading piece... Jun 23 01:28:26.605369657 (105h5m2.168692667s) [noise=true , address=45.91.116.8:5050 , piece_id=M4P3RKV2BUWYZOP2TOLNSMYOYDTOEG5NXXM6XZUYPLFQMLUGUIOA, seg_pos={0 0}, upload=84097]: Uploading piece... Jun 23 01:28:26.605753081 (105h5m2.169076095s) [noise=true , address=77.90.31.14:4693 , piece_id=YGA5IDHRAXIFCELCNCQF23PEFGJMFQOQHFWY3D7UODW5HYEIEWXA, seg_pos={0 0}, upload=84097]: Uploading piece... Jun 23 01:28:26.606212150 (105h5m2.169535158s) [noise=true , address=104.36.82.9:6021 , piece_id=4JQGGBGDZMQ6FDS35A7BZLB5OLZL2JHMD5Q3TXAETVX7TYA74EHQ, seg_pos={0 0}, upload=84097]: Uploading piece... Jun 23 01:28:26.606654203 (105h5m2.169977203s) [noise=true , address=104.36.82.16:6310 , piece_id=CRHVWU4XYDGUG7XVGC3TI6DC2CLBTSTHND4YB2YFEFPBD6TOAXKQ, seg_pos={0 0}, upload=84097]: Uploading piece... Jun 23 01:28:26.606961844 (105h5m2.170284849s) [noise=true , address=156.146.43.225:4448 , piece_id=5SH6UKJRS56UOVGLLW7JKTSAQANRVHYVOMEXZ56UWYM3AEB7QGVA, seg_pos={0 0}, upload=84097]: Uploading piece... Jun 23 01:28:26.607374002 (105h5m2.170697008s) [noise=true , address=104.36.82.12:6174 , piece_id=VQG2SHTAARK4FOSQER2YRKHZV7ODG7ZDPHNF6AESZGE3GMITKZZA, seg_pos={0 0}, upload=84097]: Uploading piece... Jun 23 01:28:26.607734796 (105h5m2.171057797s) [noise=true , address=45.91.116.9:5288 , piece_id=EQWQVX4JDLVTNRTKYNUIFGJYQUXPVUCUCYJPNVW2IYM3CGQY5D6A, seg_pos={0 0}, upload=84097]: Uploading piece... Jun 23 01:28:26.608082032 (105h5m2.171405034s) [noise=true , address=104.36.82.11:6131 , piece_id=BI4HK2NGEXXNEDYGRXRL77E5CONZG6JH4IWMONSB64DHOJLQQZPA, seg_pos={0 0}, upload=84097]: Uploading piece... Jun 23 01:28:26.608428729 (105h5m2.171751737s) [noise=true , address=104.36.82.42:6830 , piece_id=KW3RNOZYENVRUR663PVSC4WFCLI3RG5YBITFXOYC6W2GTGEFXTVA, seg_pos={0 0}, upload=84097]: Uploading piece... Jun 23 01:28:26.610585268 (105h5m2.173908273s) [noise=true , address=104.36.82.21:6483 , piece_id=WTUXQB3XLSHBMZZQNK745AK5U3ZLLNKC3TLW4CEFA7OX4DJCOTVQ, seg_pos={0 0}, upload=84097]: Uploading piece... Jun 23 01:28:26.612001063 (105h5m2.175324064s) [noise=true , address=104.36.82.3:5797 , piece_id=CJD4BHNHEVAGUJ5K55QM7OLAQQLEYRD4C6KJQBZ4FKJLDREGLNRQ, seg_pos={0 0}, upload=84097]: Uploading piece... Jun 23 01:28:26.687672089 (105h5m2.250995077s) [upload=84097]: Next returned nil segment Jun 23 01:28:26.687687969 (105h5m2.25101097s) [upload=84097]: Waiting for error group managing segments... Jun 23 01:28:26.707815366 (105h5m2.271138367s) [noise=true , address=79.127.220.98:4410 , piece_id=DLW3G47XXWAJ4FZAZDEL6TCIN5FKG26LUNVELDWSUE2GDH6IPBLQ, seg_pos={0 0}, upload=84097]: Done uploading piece. err: ecclient: upload failed (node:1Lay6oZEhDEeDmmkisrLND1j82PurLHV95nGJoKkk5wKUaK5S7, address:79.127.220.98:4410): protocol: storage node overloaded, request limit: 500 Jun 23 01:28:26.735633260 (105h5m2.298956259s) [noise=true , address=104.36.82.21:6483 , piece_id=WTUXQB3XLSHBMZZQNK745AK5U3ZLLNKC3TLW4CEFA7OX4DJCOTVQ, seg_pos={0 0}, upload=84097]: Done uploading piece. err: ecclient: upload failed (node:12gxKuBALDLDVDfRGBvMZymwGyPfPtV1BsHdBPqZs34ugvvYacG, address:104.36.82.21:6483): protocol: storage node overloaded, request limit: 2000 Jun 23 01:28:26.793765419 (105h5m2.357088412s) [noise=true , address=156.146.43.225:4448 , piece_id=5SH6UKJRS56UOVGLLW7JKTSAQANRVHYVOMEXZ56UWYM3AEB7QGVA, seg_pos={0 0}, upload=84097]: Done uploading piece. err: ecclient: upload failed (node:1irai8UTYknGTUkKHtKbBNTJTYMQsNTFnzrTEaWCMRDq1NNpGh, address:156.146.43.225:4448): protocol: storage node overloaded, request limit: 2000 Jun 23 01:28:26.868408971 (105h5m2.43173196s) [noise=true , address=77.90.31.15:4729 , piece_id=AAVBJ7VEQX67HQBXQ5DO3O5XJ37TRKU3PS2XXEH3WARMSEAQGQ2A, seg_pos={0 0}, upload=84097]: Done uploading piece. err: <nil> Jun 23 01:28:26.989727871 (105h5m2.553050876s) [noise=true , address=45.91.116.8:5050 , piece_id=M4P3RKV2BUWYZOP2TOLNSMYOYDTOEG5NXXM6XZUYPLFQMLUGUIOA, seg_pos={0 0}, upload=84097]: Done uploading piece. err: <nil> Jun 23 01:28:27.669519830 (105h5m3.23284283s) [noise=true , address=77.90.31.14:4693 , piece_id=YGA5IDHRAXIFCELCNCQF23PEFGJMFQOQHFWY3D7UODW5HYEIEWXA, seg_pos={0 0}, upload=84097]: Done uploading piece. err: <nil> Jun 23 01:28:27.697322244 (105h5m3.260645244s) [noise=true , address=104.36.82.42:6830 , piece_id=KW3RNOZYENVRUR663PVSC4WFCLI3RG5YBITFXOYC6W2GTGEFXTVA, seg_pos={0 0}, upload=84097]: Done uploading piece. err: <nil> Jun 23 01:28:28.071845632 (105h5m3.635168633s) [noise=true , address=104.36.82.55:4217 , piece_id=WQIZ3WHXBLCKNJ4ZVQRC2PQIRH6IVVO7J3UMM5V5GXEMODV36KLQ, seg_pos={0 0}, upload=84097]: Done uploading piece. err: <nil> Jun 23 01:28:28.589172169 (105h5m4.152495155s) [noise=true , address=45.91.116.9:5288 , piece_id=EQWQVX4JDLVTNRTKYNUIFGJYQUXPVUCUCYJPNVW2IYM3CGQY5D6A, seg_pos={0 0}, upload=84097]: Done uploading piece. err: <nil> Jun 23 01:28:29.203146980 (105h5m4.76646998s) [noise=true , address=104.36.82.16:6310 , piece_id=CRHVWU4XYDGUG7XVGC3TI6DC2CLBTSTHND4YB2YFEFPBD6TOAXKQ, seg_pos={0 0}, upload=84097]: Done uploading piece. err: ecclient: upload failed (node:1g14evoaeUnhJThf5QKbQT1Gd8WW9MHvaYQhQ1GRgwPfDLoC5R, address:104.36.82.16:6310): protocol: storage node overloaded, request limit: 2000 Jun 23 01:28:29.434402213 (105h5m4.997725203s) [noise=true , address=45.91.116.10:5084 , piece_id=WCIMJN6XDXUO2VCRKLAVOPVVQUEQDGHR4NP4KQGY27NEUPCD5OFA, seg_pos={0 0}, upload=84097]: Done uploading piece. err: <nil> Jun 23 01:28:30.293775103 (105h5m5.857098094s) [noise=true , address=104.36.82.9:6021 , piece_id=4JQGGBGDZMQ6FDS35A7BZLB5OLZL2JHMD5Q3TXAETVX7TYA74EHQ, seg_pos={0 0}, upload=84097]: Done uploading piece. err: <nil> Jun 23 01:28:30.347258498 (105h5m5.910581502s) [noise=true , address=104.36.82.12:6174 , piece_id=VQG2SHTAARK4FOSQER2YRKHZV7ODG7ZDPHNF6AESZGE3GMITKZZA, seg_pos={0 0}, upload=84097]: Done uploading piece. err: <nil> Jun 23 01:28:30.547306548 (105h5m6.110629586s) [noise=true , address=79.127.219.51:4209 , piece_id=ZHJC5QUBGOHT3HRY4XPPADV7CYINQRXT4ID53KVC6LYXM7IV533A, seg_pos={0 0}, upload=84097]: Done uploading piece. err: <nil> Jun 23 01:28:34.333414200 (105h5m9.896737197s) [noise=true , address=79.127.219.38:4031 , piece_id=TBIY2ZMYVVKMYV4LVGJEKCJXMXYVJKSZKKBXASVNEUU2P7DKZKIQ, seg_pos={0 0}, upload=84097]: Done uploading piece. err: <nil> Jun 23 01:28:42.464111369 (105h5m18.02743436s) [noise=true , address=104.36.82.53:4142 , piece_id=CL2WQNZHZNZM5BFJQIBUBQ7FQCIJSK4AR7PR2OLBIXFULPMBTGIQ, seg_pos={0 0}, upload=84097]: Done uploading piece. err: <nil> Jun 23 01:28:49.581287518 (105h5m25.144610509s) [noise=true , address=104.36.82.3:5797 , piece_id=CJD4BHNHEVAGUJ5K55QM7OLAQQLEYRD4C6KJQBZ4FKJLDREGLNRQ, seg_pos={0 0}, upload=84097]: Done uploading piece. err: <nil> Jun 23 01:29:26.535692548 (105h6m2.099015547s) [noise=true , address=104.36.82.11:6131 , piece_id=BI4HK2NGEXXNEDYGRXRL77E5CONZG6JH4IWMONSB64DHOJLQQZPA, seg_pos={0 0}, upload=84097]: Done uploading piece. err: ecclient: upload failed (node:129jahwAmyPHqyYaXz7XTCK8FakBrYZw94pBKGhPzzX23yjM3RQ, address:104.36.82.11:6131): protocol: expected piece hash; context deadline exceeded; EOF Jun 23 01:29:26.535746358 (105h6m2.099069362s) [seg_pos={0 0}, upload=84097]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jun 23 01:29:26.562546209 (105h6m2.125869225s) [upload=84097]: Done uploading segments... Jun 23 01:29:26.562593265 (105h6m2.125916275s) [upload=84097]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration:* 11.616881955s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [.....X.....................X...................X............] duration: 7.784124902s FAIL (2025-06-23 01:41:31 UTC) operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 7.784147305s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [.X.............................X...........X................] duration: 12.645275794s FAIL (2025-06-23 01:41:13 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== oad=84082]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jun 23 01:27:43.602110205 (105h4m19.165433214s) [upload=84082]: Done uploading segments... Jun 23 01:27:43.602149595 (105h4m19.165472598s) [upload=84082]: Upload finished. err: <nil> Jun 23 01:28:26.568078065 (105h5m2.131401068s) [upload=84097]: Starting upload Jun 23 01:28:26.568176623 (105h5m2.131499629s) [upload=84097]: Done starting upload Jun 23 01:28:26.568333698 (105h5m2.131656699s) [upload=84097]: Uploading segments... Jun 23 01:28:26.569025334 (105h5m2.132348341s) [upload=84097]: Got next segment. Inline: false Jun 23 01:28:26.569144232 (105h5m2.132467237s) [upload=84097]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 23 01:28:26.569226194 (105h5m2.132549206s) [upload=84097]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 23 01:28:26.603382637 (105h5m2.166705641s) [seg_pos={0 0}, upload=84097]: Begin upload segment... Jun 23 01:28:26.603439808 (105h5m2.16676282s) [seg_pos={0 0}, upload=84097]: Done begin upload segment. Jun 23 01:28:26.603481538 (105h5m2.166804559s) [noise=true , address=79.127.219.38:4031 , piece_id=TBIY2ZMYVVKMYV4LVGJEKCJXMXYVJKSZKKBXASVNEUU2P7DKZKIQ, seg_pos={0 0}, upload=84097]: Uploading piece... Jun 23 01:28:26.603498030 (105h5m2.166821033s) [noise=true , address=77.90.31.15:4729 , piece_id=AAVBJ7VEQX67HQBXQ5DO3O5XJ37TRKU3PS2XXEH3WARMSEAQGQ2A, seg_pos={0 0}, upload=84097]: Uploading piece... Jun 23 01:28:26.603958736 (105h5m2.16728174s) [noise=true , address=79.127.219.51:4209 , piece_id=ZHJC5QUBGOHT3HRY4XPPADV7CYINQRXT4ID53KVC6LYXM7IV533A, seg_pos={0 0}, upload=84097]: Uploading piece... Jun 23 01:28:26.603997417 (105h5m2.167320407s) [noise=true , address=79.127.220.98:4410 , piece_id=DLW3G47XXWAJ4FZAZDEL6TCIN5FKG26LUNVELDWSUE2GDH6IPBLQ, seg_pos={0 0}, upload=84097]: Uploading piece... Jun 23 01:28:26.604297025 (105h5m2.167620026s) [noise=true , address=45.91.116.10:5084 , piece_id=WCIMJN6XDXUO2VCRKLAVOPVVQUEQDGHR4NP4KQGY27NEUPCD5OFA, seg_pos={0 0}, upload=84097]: Uploading piece... Jun 23 01:28:26.604692251 (105h5m2.16801525s) [noise=true , address=104.36.82.55:4217 , piece_id=WQIZ3WHXBLCKNJ4ZVQRC2PQIRH6IVVO7J3UMM5V5GXEMODV36KLQ, seg_pos={0 0}, upload=84097]: Uploading piece... Jun 23 01:28:26.605044347 (105h5m2.168367349s) [noise=true , address=104.36.82.53:4142 , piece_id=CL2WQNZHZNZM5BFJQIBUBQ7FQCIJSK4AR7PR2OLBIXFULPMBTGIQ, seg_pos={0 0}, upload=84097]: Uploading piece... Jun 23 01:28:26.605369657 (105h5m2.168692667s) [noise=true , address=45.91.116.8:5050 , piece_id=M4P3RKV2BUWYZOP2TOLNSMYOYDTOEG5NXXM6XZUYPLFQMLUGUIOA, seg_pos={0 0}, upload=84097]: Uploading piece... Jun 23 01:28:26.605753081 (105h5m2.169076095s) [noise=true , address=77.90.31.14:4693 , piece_id=YGA5IDHRAXIFCELCNCQF23PEFGJMFQOQHFWY3D7UODW5HYEIEWXA, seg_pos={0 0}, upload=84097]: Uploading piece... Jun 23 01:28:26.606212150 (105h5m2.169535158s) [noise=true , address=104.36.82.9:6021 , piece_id=4JQGGBGDZMQ6FDS35A7BZLB5OLZL2JHMD5Q3TXAETVX7TYA74EHQ, seg_pos={0 0}, upload=84097]: Uploading piece... Jun 23 01:28:26.606654203 (105h5m2.169977203s) [noise=true , address=104.36.82.16:6310 , piece_id=CRHVWU4XYDGUG7XVGC3TI6DC2CLBTSTHND4YB2YFEFPBD6TOAXKQ, seg_pos={0 0}, upload=84097]: Uploading piece... Jun 23 01:28:26.606961844 (105h5m2.170284849s) [noise=true , address=156.146.43.225:4448 , piece_id=5SH6UKJRS56UOVGLLW7JKTSAQANRVHYVOMEXZ56UWYM3AEB7QGVA, seg_pos={0 0}, upload=84097]: Uploading piece... Jun 23 01:28:26.607374002 (105h5m2.170697008s) [noise=true , address=104.36.82.12:6174 , piece_id=VQG2SHTAARK4FOSQER2YRKHZV7ODG7ZDPHNF6AESZGE3GMITKZZA, seg_pos={0 0}, upload=84097]: Uploading piece... Jun 23 01:28:26.607734796 (105h5m2.171057797s) [noise=true , address=45.91.116.9:5288 , piece_id=EQWQVX4JDLVTNRTKYNUIFGJYQUXPVUCUCYJPNVW2IYM3CGQY5D6A, seg_pos={0 0}, upload=84097]: Uploading piece... Jun 23 01:28:26.608082032 (105h5m2.171405034s) [noise=true , address=104.36.82.11:6131 , piece_id=BI4HK2NGEXXNEDYGRXRL77E5CONZG6JH4IWMONSB64DHOJLQQZPA, seg_pos={0 0}, upload=84097]: Uploading piece... Jun 23 01:28:26.608428729 (105h5m2.171751737s) [noise=true , address=104.36.82.42:6830 , piece_id=KW3RNOZYENVRUR663PVSC4WFCLI3RG5YBITFXOYC6W2GTGEFXTVA, seg_pos={0 0}, upload=84097]: Uploading piece... Jun 23 01:28:26.610585268 (105h5m2.173908273s) [noise=true , address=104.36.82.21:6483 , piece_id=WTUXQB3XLSHBMZZQNK745AK5U3ZLLNKC3TLW4CEFA7OX4DJCOTVQ, seg_pos={0 0}, upload=84097]: Uploading piece... Jun 23 01:28:26.612001063 (105h5m2.175324064s) [noise=true , address=104.36.82.3:5797 , piece_id=CJD4BHNHEVAGUJ5K55QM7OLAQQLEYRD4C6KJQBZ4FKJLDREGLNRQ, seg_pos={0 0}, upload=84097]: Uploading piece... Jun 23 01:28:26.687672089 (105h5m2.250995077s) [upload=84097]: Next returned nil segment Jun 23 01:28:26.687687969 (105h5m2.25101097s) [upload=84097]: Waiting for error group managing segments... Jun 23 01:28:26.707815366 (105h5m2.271138367s) [noise=true , address=79.127.220.98:4410 , piece_id=DLW3G47XXWAJ4FZAZDEL6TCIN5FKG26LUNVELDWSUE2GDH6IPBLQ, seg_pos={0 0}, upload=84097]: Done uploading piece. err: ecclient: upload failed (node:1Lay6oZEhDEeDmmkisrLND1j82PurLHV95nGJoKkk5wKUaK5S7, address:79.127.220.98:4410): protocol: storage node overloaded, request limit: 500 Jun 23 01:28:26.735633260 (105h5m2.298956259s) [noise=true , address=104.36.82.21:6483 , piece_id=WTUXQB3XLSHBMZZQNK745AK5U3ZLLNKC3TLW4CEFA7OX4DJCOTVQ, seg_pos={0 0}, upload=84097]: Done uploading piece. err: ecclient: upload failed (node:12gxKuBALDLDVDfRGBvMZymwGyPfPtV1BsHdBPqZs34ugvvYacG, address:104.36.82.21:6483): protocol: storage node overloaded, request limit: 2000 Jun 23 01:28:26.793765419 (105h5m2.357088412s) [noise=true , address=156.146.43.225:4448 , piece_id=5SH6UKJRS56UOVGLLW7JKTSAQANRVHYVOMEXZ56UWYM3AEB7QGVA, seg_pos={0 0}, upload=84097]: Done uploading piece. err: ecclient: upload failed (node:1irai8UTYknGTUkKHtKbBNTJTYMQsNTFnzrTEaWCMRDq1NNpGh, address:156.146.43.225:4448): protocol: storage node overloaded, request limit: 2000 Jun 23 01:28:26.868408971 (105h5m2.43173196s) [noise=true , address=77.90.31.15:4729 , piece_id=AAVBJ7VEQX67HQBXQ5DO3O5XJ37TRKU3PS2XXEH3WARMSEAQGQ2A, seg_pos={0 0}, upload=84097]: Done uploading piece. err: <nil> Jun 23 01:28:26.989727871 (105h5m2.553050876s) [noise=true , address=45.91.116.8:5050 , piece_id=M4P3RKV2BUWYZOP2TOLNSMYOYDTOEG5NXXM6XZUYPLFQMLUGUIOA, seg_pos={0 0}, upload=84097]: Done uploading piece. err: <nil> Jun 23 01:28:27.669519830 (105h5m3.23284283s) [noise=true , address=77.90.31.14:4693 , piece_id=YGA5IDHRAXIFCELCNCQF23PEFGJMFQOQHFWY3D7UODW5HYEIEWXA, seg_pos={0 0}, upload=84097]: Done uploading piece. err: <nil> Jun 23 01:28:27.697322244 (105h5m3.260645244s) [noise=true , address=104.36.82.42:6830 , piece_id=KW3RNOZYENVRUR663PVSC4WFCLI3RG5YBITFXOYC6W2GTGEFXTVA, seg_pos={0 0}, upload=84097]: Done uploading piece. err: <nil> Jun 23 01:28:28.071845632 (105h5m3.635168633s) [noise=true , address=104.36.82.55:4217 , piece_id=WQIZ3WHXBLCKNJ4ZVQRC2PQIRH6IVVO7J3UMM5V5GXEMODV36KLQ, seg_pos={0 0}, upload=84097]: Done uploading piece. err: <nil> Jun 23 01:28:28.589172169 (105h5m4.152495155s) [noise=true , address=45.91.116.9:5288 , piece_id=EQWQVX4JDLVTNRTKYNUIFGJYQUXPVUCUCYJPNVW2IYM3CGQY5D6A, seg_pos={0 0}, upload=84097]: Done uploading piece. err: <nil> Jun 23 01:28:29.203146980 (105h5m4.76646998s) [noise=true , address=104.36.82.16:6310 , piece_id=CRHVWU4XYDGUG7XVGC3TI6DC2CLBTSTHND4YB2YFEFPBD6TOAXKQ, seg_pos={0 0}, upload=84097]: Done uploading piece. err: ecclient: upload failed (node:1g14evoaeUnhJThf5QKbQT1Gd8WW9MHvaYQhQ1GRgwPfDLoC5R, address:104.36.82.16:6310): protocol: storage node overloaded, request limit: 2000 Jun 23 01:28:29.434402213 (105h5m4.997725203s) [noise=true , address=45.91.116.10:5084 , piece_id=WCIMJN6XDXUO2VCRKLAVOPVVQUEQDGHR4NP4KQGY27NEUPCD5OFA, seg_pos={0 0}, upload=84097]: Done uploading piece. err: <nil> Jun 23 01:28:30.293775103 (105h5m5.857098094s) [noise=true , address=104.36.82.9:6021 , piece_id=4JQGGBGDZMQ6FDS35A7BZLB5OLZL2JHMD5Q3TXAETVX7TYA74EHQ, seg_pos={0 0}, upload=84097]: Done uploading piece. err: <nil> Jun 23 01:28:30.347258498 (105h5m5.910581502s) [noise=true , address=104.36.82.12:6174 , piece_id=VQG2SHTAARK4FOSQER2YRKHZV7ODG7ZDPHNF6AESZGE3GMITKZZA, seg_pos={0 0}, upload=84097]: Done uploading piece. err: <nil> Jun 23 01:28:30.547306548 (105h5m6.110629586s) [noise=true , address=79.127.219.51:4209 , piece_id=ZHJC5QUBGOHT3HRY4XPPADV7CYINQRXT4ID53KVC6LYXM7IV533A, seg_pos={0 0}, upload=84097]: Done uploading piece. err: <nil> Jun 23 01:28:34.333414200 (105h5m9.896737197s) [noise=true , address=79.127.219.38:4031 , piece_id=TBIY2ZMYVVKMYV4LVGJEKCJXMXYVJKSZKKBXASVNEUU2P7DKZKIQ, seg_pos={0 0}, upload=84097]: Done uploading piece. err: <nil> Jun 23 01:28:42.464111369 (105h5m18.02743436s) [noise=true , address=104.36.82.53:4142 , piece_id=CL2WQNZHZNZM5BFJQIBUBQ7FQCIJSK4AR7PR2OLBIXFULPMBTGIQ, seg_pos={0 0}, upload=84097]: Done uploading piece. err: <nil> Jun 23 01:28:49.581287518 (105h5m25.144610509s) [noise=true , address=104.36.82.3:5797 , piece_id=CJD4BHNHEVAGUJ5K55QM7OLAQQLEYRD4C6KJQBZ4FKJLDREGLNRQ, seg_pos={0 0}, upload=84097]: Done uploading piece. err: <nil> Jun 23 01:29:26.535692548 (105h6m2.099015547s) [noise=true , address=104.36.82.11:6131 , piece_id=BI4HK2NGEXXNEDYGRXRL77E5CONZG6JH4IWMONSB64DHOJLQQZPA, seg_pos={0 0}, upload=84097]: Done uploading piece. err: ecclient: upload failed (node:129jahwAmyPHqyYaXz7XTCK8FakBrYZw94pBKGhPzzX23yjM3RQ, address:104.36.82.11:6131): protocol: expected piece hash; context deadline exceeded; EOF Jun 23 01:29:26.535746358 (105h6m2.099069362s) [seg_pos={0 0}, upload=84097]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jun 23 01:29:26.562546209 (105h6m2.125869225s) [upload=84097]: Done uploading segments... Jun 23 01:29:26.562593265 (105h6m2.125916275s) [upload=84097]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration:* 12.645306194s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [......X.....................X...................X...........] duration: 23.70815764s FAIL (2025-06-23 01:40:54 UTC) operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 23.708184652s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [..X.............................X...........X...............] duration: 27.896461115s FAIL (2025-06-23 01:40:23 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== oad=84082]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jun 23 01:27:43.602110205 (105h4m19.165433214s) [upload=84082]: Done uploading segments... Jun 23 01:27:43.602149595 (105h4m19.165472598s) [upload=84082]: Upload finished. err: <nil> Jun 23 01:28:26.568078065 (105h5m2.131401068s) [upload=84097]: Starting upload Jun 23 01:28:26.568176623 (105h5m2.131499629s) [upload=84097]: Done starting upload Jun 23 01:28:26.568333698 (105h5m2.131656699s) [upload=84097]: Uploading segments... Jun 23 01:28:26.569025334 (105h5m2.132348341s) [upload=84097]: Got next segment. Inline: false Jun 23 01:28:26.569144232 (105h5m2.132467237s) [upload=84097]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 23 01:28:26.569226194 (105h5m2.132549206s) [upload=84097]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 23 01:28:26.603382637 (105h5m2.166705641s) [seg_pos={0 0}, upload=84097]: Begin upload segment... Jun 23 01:28:26.603439808 (105h5m2.16676282s) [seg_pos={0 0}, upload=84097]: Done begin upload segment. Jun 23 01:28:26.603481538 (105h5m2.166804559s) [noise=true , address=79.127.219.38:4031 , piece_id=TBIY2ZMYVVKMYV4LVGJEKCJXMXYVJKSZKKBXASVNEUU2P7DKZKIQ, seg_pos={0 0}, upload=84097]: Uploading piece... Jun 23 01:28:26.603498030 (105h5m2.166821033s) [noise=true , address=77.90.31.15:4729 , piece_id=AAVBJ7VEQX67HQBXQ5DO3O5XJ37TRKU3PS2XXEH3WARMSEAQGQ2A, seg_pos={0 0}, upload=84097]: Uploading piece... Jun 23 01:28:26.603958736 (105h5m2.16728174s) [noise=true , address=79.127.219.51:4209 , piece_id=ZHJC5QUBGOHT3HRY4XPPADV7CYINQRXT4ID53KVC6LYXM7IV533A, seg_pos={0 0}, upload=84097]: Uploading piece... Jun 23 01:28:26.603997417 (105h5m2.167320407s) [noise=true , address=79.127.220.98:4410 , piece_id=DLW3G47XXWAJ4FZAZDEL6TCIN5FKG26LUNVELDWSUE2GDH6IPBLQ, seg_pos={0 0}, upload=84097]: Uploading piece... Jun 23 01:28:26.604297025 (105h5m2.167620026s) [noise=true , address=45.91.116.10:5084 , piece_id=WCIMJN6XDXUO2VCRKLAVOPVVQUEQDGHR4NP4KQGY27NEUPCD5OFA, seg_pos={0 0}, upload=84097]: Uploading piece... Jun 23 01:28:26.604692251 (105h5m2.16801525s) [noise=true , address=104.36.82.55:4217 , piece_id=WQIZ3WHXBLCKNJ4ZVQRC2PQIRH6IVVO7J3UMM5V5GXEMODV36KLQ, seg_pos={0 0}, upload=84097]: Uploading piece... Jun 23 01:28:26.605044347 (105h5m2.168367349s) [noise=true , address=104.36.82.53:4142 , piece_id=CL2WQNZHZNZM5BFJQIBUBQ7FQCIJSK4AR7PR2OLBIXFULPMBTGIQ, seg_pos={0 0}, upload=84097]: Uploading piece... Jun 23 01:28:26.605369657 (105h5m2.168692667s) [noise=true , address=45.91.116.8:5050 , piece_id=M4P3RKV2BUWYZOP2TOLNSMYOYDTOEG5NXXM6XZUYPLFQMLUGUIOA, seg_pos={0 0}, upload=84097]: Uploading piece... Jun 23 01:28:26.605753081 (105h5m2.169076095s) [noise=true , address=77.90.31.14:4693 , piece_id=YGA5IDHRAXIFCELCNCQF23PEFGJMFQOQHFWY3D7UODW5HYEIEWXA, seg_pos={0 0}, upload=84097]: Uploading piece... Jun 23 01:28:26.606212150 (105h5m2.169535158s) [noise=true , address=104.36.82.9:6021 , piece_id=4JQGGBGDZMQ6FDS35A7BZLB5OLZL2JHMD5Q3TXAETVX7TYA74EHQ, seg_pos={0 0}, upload=84097]: Uploading piece... Jun 23 01:28:26.606654203 (105h5m2.169977203s) [noise=true , address=104.36.82.16:6310 , piece_id=CRHVWU4XYDGUG7XVGC3TI6DC2CLBTSTHND4YB2YFEFPBD6TOAXKQ, seg_pos={0 0}, upload=84097]: Uploading piece... Jun 23 01:28:26.606961844 (105h5m2.170284849s) [noise=true , address=156.146.43.225:4448 , piece_id=5SH6UKJRS56UOVGLLW7JKTSAQANRVHYVOMEXZ56UWYM3AEB7QGVA, seg_pos={0 0}, upload=84097]: Uploading piece... Jun 23 01:28:26.607374002 (105h5m2.170697008s) [noise=true , address=104.36.82.12:6174 , piece_id=VQG2SHTAARK4FOSQER2YRKHZV7ODG7ZDPHNF6AESZGE3GMITKZZA, seg_pos={0 0}, upload=84097]: Uploading piece... Jun 23 01:28:26.607734796 (105h5m2.171057797s) [noise=true , address=45.91.116.9:5288 , piece_id=EQWQVX4JDLVTNRTKYNUIFGJYQUXPVUCUCYJPNVW2IYM3CGQY5D6A, seg_pos={0 0}, upload=84097]: Uploading piece... Jun 23 01:28:26.608082032 (105h5m2.171405034s) [noise=true , address=104.36.82.11:6131 , piece_id=BI4HK2NGEXXNEDYGRXRL77E5CONZG6JH4IWMONSB64DHOJLQQZPA, seg_pos={0 0}, upload=84097]: Uploading piece... Jun 23 01:28:26.608428729 (105h5m2.171751737s) [noise=true , address=104.36.82.42:6830 , piece_id=KW3RNOZYENVRUR663PVSC4WFCLI3RG5YBITFXOYC6W2GTGEFXTVA, seg_pos={0 0}, upload=84097]: Uploading piece... Jun 23 01:28:26.610585268 (105h5m2.173908273s) [noise=true , address=104.36.82.21:6483 , piece_id=WTUXQB3XLSHBMZZQNK745AK5U3ZLLNKC3TLW4CEFA7OX4DJCOTVQ, seg_pos={0 0}, upload=84097]: Uploading piece... Jun 23 01:28:26.612001063 (105h5m2.175324064s) [noise=true , address=104.36.82.3:5797 , piece_id=CJD4BHNHEVAGUJ5K55QM7OLAQQLEYRD4C6KJQBZ4FKJLDREGLNRQ, seg_pos={0 0}, upload=84097]: Uploading piece... Jun 23 01:28:26.687672089 (105h5m2.250995077s) [upload=84097]: Next returned nil segment Jun 23 01:28:26.687687969 (105h5m2.25101097s) [upload=84097]: Waiting for error group managing segments... Jun 23 01:28:26.707815366 (105h5m2.271138367s) [noise=true , address=79.127.220.98:4410 , piece_id=DLW3G47XXWAJ4FZAZDEL6TCIN5FKG26LUNVELDWSUE2GDH6IPBLQ, seg_pos={0 0}, upload=84097]: Done uploading piece. err: ecclient: upload failed (node:1Lay6oZEhDEeDmmkisrLND1j82PurLHV95nGJoKkk5wKUaK5S7, address:79.127.220.98:4410): protocol: storage node overloaded, request limit: 500 Jun 23 01:28:26.735633260 (105h5m2.298956259s) [noise=true , address=104.36.82.21:6483 , piece_id=WTUXQB3XLSHBMZZQNK745AK5U3ZLLNKC3TLW4CEFA7OX4DJCOTVQ, seg_pos={0 0}, upload=84097]: Done uploading piece. err: ecclient: upload failed (node:12gxKuBALDLDVDfRGBvMZymwGyPfPtV1BsHdBPqZs34ugvvYacG, address:104.36.82.21:6483): protocol: storage node overloaded, request limit: 2000 Jun 23 01:28:26.793765419 (105h5m2.357088412s) [noise=true , address=156.146.43.225:4448 , piece_id=5SH6UKJRS56UOVGLLW7JKTSAQANRVHYVOMEXZ56UWYM3AEB7QGVA, seg_pos={0 0}, upload=84097]: Done uploading piece. err: ecclient: upload failed (node:1irai8UTYknGTUkKHtKbBNTJTYMQsNTFnzrTEaWCMRDq1NNpGh, address:156.146.43.225:4448): protocol: storage node overloaded, request limit: 2000 Jun 23 01:28:26.868408971 (105h5m2.43173196s) [noise=true , address=77.90.31.15:4729 , piece_id=AAVBJ7VEQX67HQBXQ5DO3O5XJ37TRKU3PS2XXEH3WARMSEAQGQ2A, seg_pos={0 0}, upload=84097]: Done uploading piece. err: <nil> Jun 23 01:28:26.989727871 (105h5m2.553050876s) [noise=true , address=45.91.116.8:5050 , piece_id=M4P3RKV2BUWYZOP2TOLNSMYOYDTOEG5NXXM6XZUYPLFQMLUGUIOA, seg_pos={0 0}, upload=84097]: Done uploading piece. err: <nil> Jun 23 01:28:27.669519830 (105h5m3.23284283s) [noise=true , address=77.90.31.14:4693 , piece_id=YGA5IDHRAXIFCELCNCQF23PEFGJMFQOQHFWY3D7UODW5HYEIEWXA, seg_pos={0 0}, upload=84097]: Done uploading piece. err: <nil> Jun 23 01:28:27.697322244 (105h5m3.260645244s) [noise=true , address=104.36.82.42:6830 , piece_id=KW3RNOZYENVRUR663PVSC4WFCLI3RG5YBITFXOYC6W2GTGEFXTVA, seg_pos={0 0}, upload=84097]: Done uploading piece. err: <nil> Jun 23 01:28:28.071845632 (105h5m3.635168633s) [noise=true , address=104.36.82.55:4217 , piece_id=WQIZ3WHXBLCKNJ4ZVQRC2PQIRH6IVVO7J3UMM5V5GXEMODV36KLQ, seg_pos={0 0}, upload=84097]: Done uploading piece. err: <nil> Jun 23 01:28:28.589172169 (105h5m4.152495155s) [noise=true , address=45.91.116.9:5288 , piece_id=EQWQVX4JDLVTNRTKYNUIFGJYQUXPVUCUCYJPNVW2IYM3CGQY5D6A, seg_pos={0 0}, upload=84097]: Done uploading piece. err: <nil> Jun 23 01:28:29.203146980 (105h5m4.76646998s) [noise=true , address=104.36.82.16:6310 , piece_id=CRHVWU4XYDGUG7XVGC3TI6DC2CLBTSTHND4YB2YFEFPBD6TOAXKQ, seg_pos={0 0}, upload=84097]: Done uploading piece. err: ecclient: upload failed (node:1g14evoaeUnhJThf5QKbQT1Gd8WW9MHvaYQhQ1GRgwPfDLoC5R, address:104.36.82.16:6310): protocol: storage node overloaded, request limit: 2000 Jun 23 01:28:29.434402213 (105h5m4.997725203s) [noise=true , address=45.91.116.10:5084 , piece_id=WCIMJN6XDXUO2VCRKLAVOPVVQUEQDGHR4NP4KQGY27NEUPCD5OFA, seg_pos={0 0}, upload=84097]: Done uploading piece. err: <nil> Jun 23 01:28:30.293775103 (105h5m5.857098094s) [noise=true , address=104.36.82.9:6021 , piece_id=4JQGGBGDZMQ6FDS35A7BZLB5OLZL2JHMD5Q3TXAETVX7TYA74EHQ, seg_pos={0 0}, upload=84097]: Done uploading piece. err: <nil> Jun 23 01:28:30.347258498 (105h5m5.910581502s) [noise=true , address=104.36.82.12:6174 , piece_id=VQG2SHTAARK4FOSQER2YRKHZV7ODG7ZDPHNF6AESZGE3GMITKZZA, seg_pos={0 0}, upload=84097]: Done uploading piece. err: <nil> Jun 23 01:28:30.547306548 (105h5m6.110629586s) [noise=true , address=79.127.219.51:4209 , piece_id=ZHJC5QUBGOHT3HRY4XPPADV7CYINQRXT4ID53KVC6LYXM7IV533A, seg_pos={0 0}, upload=84097]: Done uploading piece. err: <nil> Jun 23 01:28:34.333414200 (105h5m9.896737197s) [noise=true , address=79.127.219.38:4031 , piece_id=TBIY2ZMYVVKMYV4LVGJEKCJXMXYVJKSZKKBXASVNEUU2P7DKZKIQ, seg_pos={0 0}, upload=84097]: Done uploading piece. err: <nil> Jun 23 01:28:42.464111369 (105h5m18.02743436s) [noise=true , address=104.36.82.53:4142 , piece_id=CL2WQNZHZNZM5BFJQIBUBQ7FQCIJSK4AR7PR2OLBIXFULPMBTGIQ, seg_pos={0 0}, upload=84097]: Done uploading piece. err: <nil> Jun 23 01:28:49.581287518 (105h5m25.144610509s) [noise=true , address=104.36.82.3:5797 , piece_id=CJD4BHNHEVAGUJ5K55QM7OLAQQLEYRD4C6KJQBZ4FKJLDREGLNRQ, seg_pos={0 0}, upload=84097]: Done uploading piece. err: <nil> Jun 23 01:29:26.535692548 (105h6m2.099015547s) [noise=true , address=104.36.82.11:6131 , piece_id=BI4HK2NGEXXNEDYGRXRL77E5CONZG6JH4IWMONSB64DHOJLQQZPA, seg_pos={0 0}, upload=84097]: Done uploading piece. err: ecclient: upload failed (node:129jahwAmyPHqyYaXz7XTCK8FakBrYZw94pBKGhPzzX23yjM3RQ, address:104.36.82.11:6131): protocol: expected piece hash; context deadline exceeded; EOF Jun 23 01:29:26.535746358 (105h6m2.099069362s) [seg_pos={0 0}, upload=84097]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jun 23 01:29:26.562546209 (105h6m2.125869225s) [upload=84097]: Done uploading segments... Jun 23 01:29:26.562593265 (105h6m2.125916275s) [upload=84097]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration:* 27.896491458s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [.......X.....................X...................X..........] duration: 18.807458729s FAIL (2025-06-23 01:39:34 UTC) operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 18.807504145s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [...X.............................X...........X..............] duration: 8.267518535s FAIL (2025-06-23 01:39:11 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== oad=84082]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jun 23 01:27:43.602110205 (105h4m19.165433214s) [upload=84082]: Done uploading segments... Jun 23 01:27:43.602149595 (105h4m19.165472598s) [upload=84082]: Upload finished. err: <nil> Jun 23 01:28:26.568078065 (105h5m2.131401068s) [upload=84097]: Starting upload Jun 23 01:28:26.568176623 (105h5m2.131499629s) [upload=84097]: Done starting upload Jun 23 01:28:26.568333698 (105h5m2.131656699s) [upload=84097]: Uploading segments... Jun 23 01:28:26.569025334 (105h5m2.132348341s) [upload=84097]: Got next segment. Inline: false Jun 23 01:28:26.569144232 (105h5m2.132467237s) [upload=84097]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 23 01:28:26.569226194 (105h5m2.132549206s) [upload=84097]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 23 01:28:26.603382637 (105h5m2.166705641s) [seg_pos={0 0}, upload=84097]: Begin upload segment... Jun 23 01:28:26.603439808 (105h5m2.16676282s) [seg_pos={0 0}, upload=84097]: Done begin upload segment. Jun 23 01:28:26.603481538 (105h5m2.166804559s) [noise=true , address=79.127.219.38:4031 , piece_id=TBIY2ZMYVVKMYV4LVGJEKCJXMXYVJKSZKKBXASVNEUU2P7DKZKIQ, seg_pos={0 0}, upload=84097]: Uploading piece... Jun 23 01:28:26.603498030 (105h5m2.166821033s) [noise=true , address=77.90.31.15:4729 , piece_id=AAVBJ7VEQX67HQBXQ5DO3O5XJ37TRKU3PS2XXEH3WARMSEAQGQ2A, seg_pos={0 0}, upload=84097]: Uploading piece... Jun 23 01:28:26.603958736 (105h5m2.16728174s) [noise=true , address=79.127.219.51:4209 , piece_id=ZHJC5QUBGOHT3HRY4XPPADV7CYINQRXT4ID53KVC6LYXM7IV533A, seg_pos={0 0}, upload=84097]: Uploading piece... Jun 23 01:28:26.603997417 (105h5m2.167320407s) [noise=true , address=79.127.220.98:4410 , piece_id=DLW3G47XXWAJ4FZAZDEL6TCIN5FKG26LUNVELDWSUE2GDH6IPBLQ, seg_pos={0 0}, upload=84097]: Uploading piece... Jun 23 01:28:26.604297025 (105h5m2.167620026s) [noise=true , address=45.91.116.10:5084 , piece_id=WCIMJN6XDXUO2VCRKLAVOPVVQUEQDGHR4NP4KQGY27NEUPCD5OFA, seg_pos={0 0}, upload=84097]: Uploading piece... Jun 23 01:28:26.604692251 (105h5m2.16801525s) [noise=true , address=104.36.82.55:4217 , piece_id=WQIZ3WHXBLCKNJ4ZVQRC2PQIRH6IVVO7J3UMM5V5GXEMODV36KLQ, seg_pos={0 0}, upload=84097]: Uploading piece... Jun 23 01:28:26.605044347 (105h5m2.168367349s) [noise=true , address=104.36.82.53:4142 , piece_id=CL2WQNZHZNZM5BFJQIBUBQ7FQCIJSK4AR7PR2OLBIXFULPMBTGIQ, seg_pos={0 0}, upload=84097]: Uploading piece... Jun 23 01:28:26.605369657 (105h5m2.168692667s) [noise=true , address=45.91.116.8:5050 , piece_id=M4P3RKV2BUWYZOP2TOLNSMYOYDTOEG5NXXM6XZUYPLFQMLUGUIOA, seg_pos={0 0}, upload=84097]: Uploading piece... Jun 23 01:28:26.605753081 (105h5m2.169076095s) [noise=true , address=77.90.31.14:4693 , piece_id=YGA5IDHRAXIFCELCNCQF23PEFGJMFQOQHFWY3D7UODW5HYEIEWXA, seg_pos={0 0}, upload=84097]: Uploading piece... Jun 23 01:28:26.606212150 (105h5m2.169535158s) [noise=true , address=104.36.82.9:6021 , piece_id=4JQGGBGDZMQ6FDS35A7BZLB5OLZL2JHMD5Q3TXAETVX7TYA74EHQ, seg_pos={0 0}, upload=84097]: Uploading piece... Jun 23 01:28:26.606654203 (105h5m2.169977203s) [noise=true , address=104.36.82.16:6310 , piece_id=CRHVWU4XYDGUG7XVGC3TI6DC2CLBTSTHND4YB2YFEFPBD6TOAXKQ, seg_pos={0 0}, upload=84097]: Uploading piece... Jun 23 01:28:26.606961844 (105h5m2.170284849s) [noise=true , address=156.146.43.225:4448 , piece_id=5SH6UKJRS56UOVGLLW7JKTSAQANRVHYVOMEXZ56UWYM3AEB7QGVA, seg_pos={0 0}, upload=84097]: Uploading piece... Jun 23 01:28:26.607374002 (105h5m2.170697008s) [noise=true , address=104.36.82.12:6174 , piece_id=VQG2SHTAARK4FOSQER2YRKHZV7ODG7ZDPHNF6AESZGE3GMITKZZA, seg_pos={0 0}, upload=84097]: Uploading piece... Jun 23 01:28:26.607734796 (105h5m2.171057797s) [noise=true , address=45.91.116.9:5288 , piece_id=EQWQVX4JDLVTNRTKYNUIFGJYQUXPVUCUCYJPNVW2IYM3CGQY5D6A, seg_pos={0 0}, upload=84097]: Uploading piece... Jun 23 01:28:26.608082032 (105h5m2.171405034s) [noise=true , address=104.36.82.11:6131 , piece_id=BI4HK2NGEXXNEDYGRXRL77E5CONZG6JH4IWMONSB64DHOJLQQZPA, seg_pos={0 0}, upload=84097]: Uploading piece... Jun 23 01:28:26.608428729 (105h5m2.171751737s) [noise=true , address=104.36.82.42:6830 , piece_id=KW3RNOZYENVRUR663PVSC4WFCLI3RG5YBITFXOYC6W2GTGEFXTVA, seg_pos={0 0}, upload=84097]: Uploading piece... Jun 23 01:28:26.610585268 (105h5m2.173908273s) [noise=true , address=104.36.82.21:6483 , piece_id=WTUXQB3XLSHBMZZQNK745AK5U3ZLLNKC3TLW4CEFA7OX4DJCOTVQ, seg_pos={0 0}, upload=84097]: Uploading piece... Jun 23 01:28:26.612001063 (105h5m2.175324064s) [noise=true , address=104.36.82.3:5797 , piece_id=CJD4BHNHEVAGUJ5K55QM7OLAQQLEYRD4C6KJQBZ4FKJLDREGLNRQ, seg_pos={0 0}, upload=84097]: Uploading piece... Jun 23 01:28:26.687672089 (105h5m2.250995077s) [upload=84097]: Next returned nil segment Jun 23 01:28:26.687687969 (105h5m2.25101097s) [upload=84097]: Waiting for error group managing segments... Jun 23 01:28:26.707815366 (105h5m2.271138367s) [noise=true , address=79.127.220.98:4410 , piece_id=DLW3G47XXWAJ4FZAZDEL6TCIN5FKG26LUNVELDWSUE2GDH6IPBLQ, seg_pos={0 0}, upload=84097]: Done uploading piece. err: ecclient: upload failed (node:1Lay6oZEhDEeDmmkisrLND1j82PurLHV95nGJoKkk5wKUaK5S7, address:79.127.220.98:4410): protocol: storage node overloaded, request limit: 500 Jun 23 01:28:26.735633260 (105h5m2.298956259s) [noise=true , address=104.36.82.21:6483 , piece_id=WTUXQB3XLSHBMZZQNK745AK5U3ZLLNKC3TLW4CEFA7OX4DJCOTVQ, seg_pos={0 0}, upload=84097]: Done uploading piece. err: ecclient: upload failed (node:12gxKuBALDLDVDfRGBvMZymwGyPfPtV1BsHdBPqZs34ugvvYacG, address:104.36.82.21:6483): protocol: storage node overloaded, request limit: 2000 Jun 23 01:28:26.793765419 (105h5m2.357088412s) [noise=true , address=156.146.43.225:4448 , piece_id=5SH6UKJRS56UOVGLLW7JKTSAQANRVHYVOMEXZ56UWYM3AEB7QGVA, seg_pos={0 0}, upload=84097]: Done uploading piece. err: ecclient: upload failed (node:1irai8UTYknGTUkKHtKbBNTJTYMQsNTFnzrTEaWCMRDq1NNpGh, address:156.146.43.225:4448): protocol: storage node overloaded, request limit: 2000 Jun 23 01:28:26.868408971 (105h5m2.43173196s) [noise=true , address=77.90.31.15:4729 , piece_id=AAVBJ7VEQX67HQBXQ5DO3O5XJ37TRKU3PS2XXEH3WARMSEAQGQ2A, seg_pos={0 0}, upload=84097]: Done uploading piece. err: <nil> Jun 23 01:28:26.989727871 (105h5m2.553050876s) [noise=true , address=45.91.116.8:5050 , piece_id=M4P3RKV2BUWYZOP2TOLNSMYOYDTOEG5NXXM6XZUYPLFQMLUGUIOA, seg_pos={0 0}, upload=84097]: Done uploading piece. err: <nil> Jun 23 01:28:27.669519830 (105h5m3.23284283s) [noise=true , address=77.90.31.14:4693 , piece_id=YGA5IDHRAXIFCELCNCQF23PEFGJMFQOQHFWY3D7UODW5HYEIEWXA, seg_pos={0 0}, upload=84097]: Done uploading piece. err: <nil> Jun 23 01:28:27.697322244 (105h5m3.260645244s) [noise=true , address=104.36.82.42:6830 , piece_id=KW3RNOZYENVRUR663PVSC4WFCLI3RG5YBITFXOYC6W2GTGEFXTVA, seg_pos={0 0}, upload=84097]: Done uploading piece. err: <nil> Jun 23 01:28:28.071845632 (105h5m3.635168633s) [noise=true , address=104.36.82.55:4217 , piece_id=WQIZ3WHXBLCKNJ4ZVQRC2PQIRH6IVVO7J3UMM5V5GXEMODV36KLQ, seg_pos={0 0}, upload=84097]: Done uploading piece. err: <nil> Jun 23 01:28:28.589172169 (105h5m4.152495155s) [noise=true , address=45.91.116.9:5288 , piece_id=EQWQVX4JDLVTNRTKYNUIFGJYQUXPVUCUCYJPNVW2IYM3CGQY5D6A, seg_pos={0 0}, upload=84097]: Done uploading piece. err: <nil> Jun 23 01:28:29.203146980 (105h5m4.76646998s) [noise=true , address=104.36.82.16:6310 , piece_id=CRHVWU4XYDGUG7XVGC3TI6DC2CLBTSTHND4YB2YFEFPBD6TOAXKQ, seg_pos={0 0}, upload=84097]: Done uploading piece. err: ecclient: upload failed (node:1g14evoaeUnhJThf5QKbQT1Gd8WW9MHvaYQhQ1GRgwPfDLoC5R, address:104.36.82.16:6310): protocol: storage node overloaded, request limit: 2000 Jun 23 01:28:29.434402213 (105h5m4.997725203s) [noise=true , address=45.91.116.10:5084 , piece_id=WCIMJN6XDXUO2VCRKLAVOPVVQUEQDGHR4NP4KQGY27NEUPCD5OFA, seg_pos={0 0}, upload=84097]: Done uploading piece. err: <nil> Jun 23 01:28:30.293775103 (105h5m5.857098094s) [noise=true , address=104.36.82.9:6021 , piece_id=4JQGGBGDZMQ6FDS35A7BZLB5OLZL2JHMD5Q3TXAETVX7TYA74EHQ, seg_pos={0 0}, upload=84097]: Done uploading piece. err: <nil> Jun 23 01:28:30.347258498 (105h5m5.910581502s) [noise=true , address=104.36.82.12:6174 , piece_id=VQG2SHTAARK4FOSQER2YRKHZV7ODG7ZDPHNF6AESZGE3GMITKZZA, seg_pos={0 0}, upload=84097]: Done uploading piece. err: <nil> Jun 23 01:28:30.547306548 (105h5m6.110629586s) [noise=true , address=79.127.219.51:4209 , piece_id=ZHJC5QUBGOHT3HRY4XPPADV7CYINQRXT4ID53KVC6LYXM7IV533A, seg_pos={0 0}, upload=84097]: Done uploading piece. err: <nil> Jun 23 01:28:34.333414200 (105h5m9.896737197s) [noise=true , address=79.127.219.38:4031 , piece_id=TBIY2ZMYVVKMYV4LVGJEKCJXMXYVJKSZKKBXASVNEUU2P7DKZKIQ, seg_pos={0 0}, upload=84097]: Done uploading piece. err: <nil> Jun 23 01:28:42.464111369 (105h5m18.02743436s) [noise=true , address=104.36.82.53:4142 , piece_id=CL2WQNZHZNZM5BFJQIBUBQ7FQCIJSK4AR7PR2OLBIXFULPMBTGIQ, seg_pos={0 0}, upload=84097]: Done uploading piece. err: <nil> Jun 23 01:28:49.581287518 (105h5m25.144610509s) [noise=true , address=104.36.82.3:5797 , piece_id=CJD4BHNHEVAGUJ5K55QM7OLAQQLEYRD4C6KJQBZ4FKJLDREGLNRQ, seg_pos={0 0}, upload=84097]: Done uploading piece. err: <nil> Jun 23 01:29:26.535692548 (105h6m2.099015547s) [noise=true , address=104.36.82.11:6131 , piece_id=BI4HK2NGEXXNEDYGRXRL77E5CONZG6JH4IWMONSB64DHOJLQQZPA, seg_pos={0 0}, upload=84097]: Done uploading piece. err: ecclient: upload failed (node:129jahwAmyPHqyYaXz7XTCK8FakBrYZw94pBKGhPzzX23yjM3RQ, address:104.36.82.11:6131): protocol: expected piece hash; context deadline exceeded; EOF Jun 23 01:29:26.535746358 (105h6m2.099069362s) [seg_pos={0 0}, upload=84097]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jun 23 01:29:26.562546209 (105h6m2.125869225s) [upload=84097]: Done uploading segments... Jun 23 01:29:26.562593265 (105h6m2.125916275s) [upload=84097]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration:* 8.267545434s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [....X.............................X...........X.............] duration: 5.514235147s FAIL (2025-06-23 01:38:45 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== oad=84082]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jun 23 01:27:43.602110205 (105h4m19.165433214s) [upload=84082]: Done uploading segments... Jun 23 01:27:43.602149595 (105h4m19.165472598s) [upload=84082]: Upload finished. err: <nil> Jun 23 01:28:26.568078065 (105h5m2.131401068s) [upload=84097]: Starting upload Jun 23 01:28:26.568176623 (105h5m2.131499629s) [upload=84097]: Done starting upload Jun 23 01:28:26.568333698 (105h5m2.131656699s) [upload=84097]: Uploading segments... Jun 23 01:28:26.569025334 (105h5m2.132348341s) [upload=84097]: Got next segment. Inline: false Jun 23 01:28:26.569144232 (105h5m2.132467237s) [upload=84097]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 23 01:28:26.569226194 (105h5m2.132549206s) [upload=84097]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 23 01:28:26.603382637 (105h5m2.166705641s) [seg_pos={0 0}, upload=84097]: Begin upload segment... Jun 23 01:28:26.603439808 (105h5m2.16676282s) [seg_pos={0 0}, upload=84097]: Done begin upload segment. Jun 23 01:28:26.603481538 (105h5m2.166804559s) [noise=true , address=79.127.219.38:4031 , piece_id=TBIY2ZMYVVKMYV4LVGJEKCJXMXYVJKSZKKBXASVNEUU2P7DKZKIQ, seg_pos={0 0}, upload=84097]: Uploading piece... Jun 23 01:28:26.603498030 (105h5m2.166821033s) [noise=true , address=77.90.31.15:4729 , piece_id=AAVBJ7VEQX67HQBXQ5DO3O5XJ37TRKU3PS2XXEH3WARMSEAQGQ2A, seg_pos={0 0}, upload=84097]: Uploading piece... Jun 23 01:28:26.603958736 (105h5m2.16728174s) [noise=true , address=79.127.219.51:4209 , piece_id=ZHJC5QUBGOHT3HRY4XPPADV7CYINQRXT4ID53KVC6LYXM7IV533A, seg_pos={0 0}, upload=84097]: Uploading piece... Jun 23 01:28:26.603997417 (105h5m2.167320407s) [noise=true , address=79.127.220.98:4410 , piece_id=DLW3G47XXWAJ4FZAZDEL6TCIN5FKG26LUNVELDWSUE2GDH6IPBLQ, seg_pos={0 0}, upload=84097]: Uploading piece... Jun 23 01:28:26.604297025 (105h5m2.167620026s) [noise=true , address=45.91.116.10:5084 , piece_id=WCIMJN6XDXUO2VCRKLAVOPVVQUEQDGHR4NP4KQGY27NEUPCD5OFA, seg_pos={0 0}, upload=84097]: Uploading piece... Jun 23 01:28:26.604692251 (105h5m2.16801525s) [noise=true , address=104.36.82.55:4217 , piece_id=WQIZ3WHXBLCKNJ4ZVQRC2PQIRH6IVVO7J3UMM5V5GXEMODV36KLQ, seg_pos={0 0}, upload=84097]: Uploading piece... Jun 23 01:28:26.605044347 (105h5m2.168367349s) [noise=true , address=104.36.82.53:4142 , piece_id=CL2WQNZHZNZM5BFJQIBUBQ7FQCIJSK4AR7PR2OLBIXFULPMBTGIQ, seg_pos={0 0}, upload=84097]: Uploading piece... Jun 23 01:28:26.605369657 (105h5m2.168692667s) [noise=true , address=45.91.116.8:5050 , piece_id=M4P3RKV2BUWYZOP2TOLNSMYOYDTOEG5NXXM6XZUYPLFQMLUGUIOA, seg_pos={0 0}, upload=84097]: Uploading piece... Jun 23 01:28:26.605753081 (105h5m2.169076095s) [noise=true , address=77.90.31.14:4693 , piece_id=YGA5IDHRAXIFCELCNCQF23PEFGJMFQOQHFWY3D7UODW5HYEIEWXA, seg_pos={0 0}, upload=84097]: Uploading piece... Jun 23 01:28:26.606212150 (105h5m2.169535158s) [noise=true , address=104.36.82.9:6021 , piece_id=4JQGGBGDZMQ6FDS35A7BZLB5OLZL2JHMD5Q3TXAETVX7TYA74EHQ, seg_pos={0 0}, upload=84097]: Uploading piece... Jun 23 01:28:26.606654203 (105h5m2.169977203s) [noise=true , address=104.36.82.16:6310 , piece_id=CRHVWU4XYDGUG7XVGC3TI6DC2CLBTSTHND4YB2YFEFPBD6TOAXKQ, seg_pos={0 0}, upload=84097]: Uploading piece... Jun 23 01:28:26.606961844 (105h5m2.170284849s) [noise=true , address=156.146.43.225:4448 , piece_id=5SH6UKJRS56UOVGLLW7JKTSAQANRVHYVOMEXZ56UWYM3AEB7QGVA, seg_pos={0 0}, upload=84097]: Uploading piece... Jun 23 01:28:26.607374002 (105h5m2.170697008s) [noise=true , address=104.36.82.12:6174 , piece_id=VQG2SHTAARK4FOSQER2YRKHZV7ODG7ZDPHNF6AESZGE3GMITKZZA, seg_pos={0 0}, upload=84097]: Uploading piece... Jun 23 01:28:26.607734796 (105h5m2.171057797s) [noise=true , address=45.91.116.9:5288 , piece_id=EQWQVX4JDLVTNRTKYNUIFGJYQUXPVUCUCYJPNVW2IYM3CGQY5D6A, seg_pos={0 0}, upload=84097]: Uploading piece... Jun 23 01:28:26.608082032 (105h5m2.171405034s) [noise=true , address=104.36.82.11:6131 , piece_id=BI4HK2NGEXXNEDYGRXRL77E5CONZG6JH4IWMONSB64DHOJLQQZPA, seg_pos={0 0}, upload=84097]: Uploading piece... Jun 23 01:28:26.608428729 (105h5m2.171751737s) [noise=true , address=104.36.82.42:6830 , piece_id=KW3RNOZYENVRUR663PVSC4WFCLI3RG5YBITFXOYC6W2GTGEFXTVA, seg_pos={0 0}, upload=84097]: Uploading piece... Jun 23 01:28:26.610585268 (105h5m2.173908273s) [noise=true , address=104.36.82.21:6483 , piece_id=WTUXQB3XLSHBMZZQNK745AK5U3ZLLNKC3TLW4CEFA7OX4DJCOTVQ, seg_pos={0 0}, upload=84097]: Uploading piece... Jun 23 01:28:26.612001063 (105h5m2.175324064s) [noise=true , address=104.36.82.3:5797 , piece_id=CJD4BHNHEVAGUJ5K55QM7OLAQQLEYRD4C6KJQBZ4FKJLDREGLNRQ, seg_pos={0 0}, upload=84097]: Uploading piece... Jun 23 01:28:26.687672089 (105h5m2.250995077s) [upload=84097]: Next returned nil segment Jun 23 01:28:26.687687969 (105h5m2.25101097s) [upload=84097]: Waiting for error group managing segments... Jun 23 01:28:26.707815366 (105h5m2.271138367s) [noise=true , address=79.127.220.98:4410 , piece_id=DLW3G47XXWAJ4FZAZDEL6TCIN5FKG26LUNVELDWSUE2GDH6IPBLQ, seg_pos={0 0}, upload=84097]: Done uploading piece. err: ecclient: upload failed (node:1Lay6oZEhDEeDmmkisrLND1j82PurLHV95nGJoKkk5wKUaK5S7, address:79.127.220.98:4410): protocol: storage node overloaded, request limit: 500 Jun 23 01:28:26.735633260 (105h5m2.298956259s) [noise=true , address=104.36.82.21:6483 , piece_id=WTUXQB3XLSHBMZZQNK745AK5U3ZLLNKC3TLW4CEFA7OX4DJCOTVQ, seg_pos={0 0}, upload=84097]: Done uploading piece. err: ecclient: upload failed (node:12gxKuBALDLDVDfRGBvMZymwGyPfPtV1BsHdBPqZs34ugvvYacG, address:104.36.82.21:6483): protocol: storage node overloaded, request limit: 2000 Jun 23 01:28:26.793765419 (105h5m2.357088412s) [noise=true , address=156.146.43.225:4448 , piece_id=5SH6UKJRS56UOVGLLW7JKTSAQANRVHYVOMEXZ56UWYM3AEB7QGVA, seg_pos={0 0}, upload=84097]: Done uploading piece. err: ecclient: upload failed (node:1irai8UTYknGTUkKHtKbBNTJTYMQsNTFnzrTEaWCMRDq1NNpGh, address:156.146.43.225:4448): protocol: storage node overloaded, request limit: 2000 Jun 23 01:28:26.868408971 (105h5m2.43173196s) [noise=true , address=77.90.31.15:4729 , piece_id=AAVBJ7VEQX67HQBXQ5DO3O5XJ37TRKU3PS2XXEH3WARMSEAQGQ2A, seg_pos={0 0}, upload=84097]: Done uploading piece. err: <nil> Jun 23 01:28:26.989727871 (105h5m2.553050876s) [noise=true , address=45.91.116.8:5050 , piece_id=M4P3RKV2BUWYZOP2TOLNSMYOYDTOEG5NXXM6XZUYPLFQMLUGUIOA, seg_pos={0 0}, upload=84097]: Done uploading piece. err: <nil> Jun 23 01:28:27.669519830 (105h5m3.23284283s) [noise=true , address=77.90.31.14:4693 , piece_id=YGA5IDHRAXIFCELCNCQF23PEFGJMFQOQHFWY3D7UODW5HYEIEWXA, seg_pos={0 0}, upload=84097]: Done uploading piece. err: <nil> Jun 23 01:28:27.697322244 (105h5m3.260645244s) [noise=true , address=104.36.82.42:6830 , piece_id=KW3RNOZYENVRUR663PVSC4WFCLI3RG5YBITFXOYC6W2GTGEFXTVA, seg_pos={0 0}, upload=84097]: Done uploading piece. err: <nil> Jun 23 01:28:28.071845632 (105h5m3.635168633s) [noise=true , address=104.36.82.55:4217 , piece_id=WQIZ3WHXBLCKNJ4ZVQRC2PQIRH6IVVO7J3UMM5V5GXEMODV36KLQ, seg_pos={0 0}, upload=84097]: Done uploading piece. err: <nil> Jun 23 01:28:28.589172169 (105h5m4.152495155s) [noise=true , address=45.91.116.9:5288 , piece_id=EQWQVX4JDLVTNRTKYNUIFGJYQUXPVUCUCYJPNVW2IYM3CGQY5D6A, seg_pos={0 0}, upload=84097]: Done uploading piece. err: <nil> Jun 23 01:28:29.203146980 (105h5m4.76646998s) [noise=true , address=104.36.82.16:6310 , piece_id=CRHVWU4XYDGUG7XVGC3TI6DC2CLBTSTHND4YB2YFEFPBD6TOAXKQ, seg_pos={0 0}, upload=84097]: Done uploading piece. err: ecclient: upload failed (node:1g14evoaeUnhJThf5QKbQT1Gd8WW9MHvaYQhQ1GRgwPfDLoC5R, address:104.36.82.16:6310): protocol: storage node overloaded, request limit: 2000 Jun 23 01:28:29.434402213 (105h5m4.997725203s) [noise=true , address=45.91.116.10:5084 , piece_id=WCIMJN6XDXUO2VCRKLAVOPVVQUEQDGHR4NP4KQGY27NEUPCD5OFA, seg_pos={0 0}, upload=84097]: Done uploading piece. err: <nil> Jun 23 01:28:30.293775103 (105h5m5.857098094s) [noise=true , address=104.36.82.9:6021 , piece_id=4JQGGBGDZMQ6FDS35A7BZLB5OLZL2JHMD5Q3TXAETVX7TYA74EHQ, seg_pos={0 0}, upload=84097]: Done uploading piece. err: <nil> Jun 23 01:28:30.347258498 (105h5m5.910581502s) [noise=true , address=104.36.82.12:6174 , piece_id=VQG2SHTAARK4FOSQER2YRKHZV7ODG7ZDPHNF6AESZGE3GMITKZZA, seg_pos={0 0}, upload=84097]: Done uploading piece. err: <nil> Jun 23 01:28:30.547306548 (105h5m6.110629586s) [noise=true , address=79.127.219.51:4209 , piece_id=ZHJC5QUBGOHT3HRY4XPPADV7CYINQRXT4ID53KVC6LYXM7IV533A, seg_pos={0 0}, upload=84097]: Done uploading piece. err: <nil> Jun 23 01:28:34.333414200 (105h5m9.896737197s) [noise=true , address=79.127.219.38:4031 , piece_id=TBIY2ZMYVVKMYV4LVGJEKCJXMXYVJKSZKKBXASVNEUU2P7DKZKIQ, seg_pos={0 0}, upload=84097]: Done uploading piece. err: <nil> Jun 23 01:28:42.464111369 (105h5m18.02743436s) [noise=true , address=104.36.82.53:4142 , piece_id=CL2WQNZHZNZM5BFJQIBUBQ7FQCIJSK4AR7PR2OLBIXFULPMBTGIQ, seg_pos={0 0}, upload=84097]: Done uploading piece. err: <nil> Jun 23 01:28:49.581287518 (105h5m25.144610509s) [noise=true , address=104.36.82.3:5797 , piece_id=CJD4BHNHEVAGUJ5K55QM7OLAQQLEYRD4C6KJQBZ4FKJLDREGLNRQ, seg_pos={0 0}, upload=84097]: Done uploading piece. err: <nil> Jun 23 01:29:26.535692548 (105h6m2.099015547s) [noise=true , address=104.36.82.11:6131 , piece_id=BI4HK2NGEXXNEDYGRXRL77E5CONZG6JH4IWMONSB64DHOJLQQZPA, seg_pos={0 0}, upload=84097]: Done uploading piece. err: ecclient: upload failed (node:129jahwAmyPHqyYaXz7XTCK8FakBrYZw94pBKGhPzzX23yjM3RQ, address:104.36.82.11:6131): protocol: expected piece hash; context deadline exceeded; EOF Jun 23 01:29:26.535746358 (105h6m2.099069362s) [seg_pos={0 0}, upload=84097]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jun 23 01:29:26.562546209 (105h6m2.125869225s) [upload=84097]: Done uploading segments... Jun 23 01:29:26.562593265 (105h6m2.125916275s) [upload=84097]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration:* 5.514268907s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [........X.....................X...................X.........] duration: 21.226612572s FAIL (2025-06-23 01:38:35 UTC) operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 21.226635286s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [.....X.............................X...........X............] duration: 24.216732979s FAIL (2025-06-23 01:38:25 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== oad=84082]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jun 23 01:27:43.602110205 (105h4m19.165433214s) [upload=84082]: Done uploading segments... Jun 23 01:27:43.602149595 (105h4m19.165472598s) [upload=84082]: Upload finished. err: <nil> Jun 23 01:28:26.568078065 (105h5m2.131401068s) [upload=84097]: Starting upload Jun 23 01:28:26.568176623 (105h5m2.131499629s) [upload=84097]: Done starting upload Jun 23 01:28:26.568333698 (105h5m2.131656699s) [upload=84097]: Uploading segments... Jun 23 01:28:26.569025334 (105h5m2.132348341s) [upload=84097]: Got next segment. Inline: false Jun 23 01:28:26.569144232 (105h5m2.132467237s) [upload=84097]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 23 01:28:26.569226194 (105h5m2.132549206s) [upload=84097]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 23 01:28:26.603382637 (105h5m2.166705641s) [seg_pos={0 0}, upload=84097]: Begin upload segment... Jun 23 01:28:26.603439808 (105h5m2.16676282s) [seg_pos={0 0}, upload=84097]: Done begin upload segment. Jun 23 01:28:26.603481538 (105h5m2.166804559s) [noise=true , address=79.127.219.38:4031 , piece_id=TBIY2ZMYVVKMYV4LVGJEKCJXMXYVJKSZKKBXASVNEUU2P7DKZKIQ, seg_pos={0 0}, upload=84097]: Uploading piece... Jun 23 01:28:26.603498030 (105h5m2.166821033s) [noise=true , address=77.90.31.15:4729 , piece_id=AAVBJ7VEQX67HQBXQ5DO3O5XJ37TRKU3PS2XXEH3WARMSEAQGQ2A, seg_pos={0 0}, upload=84097]: Uploading piece... Jun 23 01:28:26.603958736 (105h5m2.16728174s) [noise=true , address=79.127.219.51:4209 , piece_id=ZHJC5QUBGOHT3HRY4XPPADV7CYINQRXT4ID53KVC6LYXM7IV533A, seg_pos={0 0}, upload=84097]: Uploading piece... Jun 23 01:28:26.603997417 (105h5m2.167320407s) [noise=true , address=79.127.220.98:4410 , piece_id=DLW3G47XXWAJ4FZAZDEL6TCIN5FKG26LUNVELDWSUE2GDH6IPBLQ, seg_pos={0 0}, upload=84097]: Uploading piece... Jun 23 01:28:26.604297025 (105h5m2.167620026s) [noise=true , address=45.91.116.10:5084 , piece_id=WCIMJN6XDXUO2VCRKLAVOPVVQUEQDGHR4NP4KQGY27NEUPCD5OFA, seg_pos={0 0}, upload=84097]: Uploading piece... Jun 23 01:28:26.604692251 (105h5m2.16801525s) [noise=true , address=104.36.82.55:4217 , piece_id=WQIZ3WHXBLCKNJ4ZVQRC2PQIRH6IVVO7J3UMM5V5GXEMODV36KLQ, seg_pos={0 0}, upload=84097]: Uploading piece... Jun 23 01:28:26.605044347 (105h5m2.168367349s) [noise=true , address=104.36.82.53:4142 , piece_id=CL2WQNZHZNZM5BFJQIBUBQ7FQCIJSK4AR7PR2OLBIXFULPMBTGIQ, seg_pos={0 0}, upload=84097]: Uploading piece... Jun 23 01:28:26.605369657 (105h5m2.168692667s) [noise=true , address=45.91.116.8:5050 , piece_id=M4P3RKV2BUWYZOP2TOLNSMYOYDTOEG5NXXM6XZUYPLFQMLUGUIOA, seg_pos={0 0}, upload=84097]: Uploading piece... Jun 23 01:28:26.605753081 (105h5m2.169076095s) [noise=true , address=77.90.31.14:4693 , piece_id=YGA5IDHRAXIFCELCNCQF23PEFGJMFQOQHFWY3D7UODW5HYEIEWXA, seg_pos={0 0}, upload=84097]: Uploading piece... Jun 23 01:28:26.606212150 (105h5m2.169535158s) [noise=true , address=104.36.82.9:6021 , piece_id=4JQGGBGDZMQ6FDS35A7BZLB5OLZL2JHMD5Q3TXAETVX7TYA74EHQ, seg_pos={0 0}, upload=84097]: Uploading piece... Jun 23 01:28:26.606654203 (105h5m2.169977203s) [noise=true , address=104.36.82.16:6310 , piece_id=CRHVWU4XYDGUG7XVGC3TI6DC2CLBTSTHND4YB2YFEFPBD6TOAXKQ, seg_pos={0 0}, upload=84097]: Uploading piece... Jun 23 01:28:26.606961844 (105h5m2.170284849s) [noise=true , address=156.146.43.225:4448 , piece_id=5SH6UKJRS56UOVGLLW7JKTSAQANRVHYVOMEXZ56UWYM3AEB7QGVA, seg_pos={0 0}, upload=84097]: Uploading piece... Jun 23 01:28:26.607374002 (105h5m2.170697008s) [noise=true , address=104.36.82.12:6174 , piece_id=VQG2SHTAARK4FOSQER2YRKHZV7ODG7ZDPHNF6AESZGE3GMITKZZA, seg_pos={0 0}, upload=84097]: Uploading piece... Jun 23 01:28:26.607734796 (105h5m2.171057797s) [noise=true , address=45.91.116.9:5288 , piece_id=EQWQVX4JDLVTNRTKYNUIFGJYQUXPVUCUCYJPNVW2IYM3CGQY5D6A, seg_pos={0 0}, upload=84097]: Uploading piece... Jun 23 01:28:26.608082032 (105h5m2.171405034s) [noise=true , address=104.36.82.11:6131 , piece_id=BI4HK2NGEXXNEDYGRXRL77E5CONZG6JH4IWMONSB64DHOJLQQZPA, seg_pos={0 0}, upload=84097]: Uploading piece... Jun 23 01:28:26.608428729 (105h5m2.171751737s) [noise=true , address=104.36.82.42:6830 , piece_id=KW3RNOZYENVRUR663PVSC4WFCLI3RG5YBITFXOYC6W2GTGEFXTVA, seg_pos={0 0}, upload=84097]: Uploading piece... Jun 23 01:28:26.610585268 (105h5m2.173908273s) [noise=true , address=104.36.82.21:6483 , piece_id=WTUXQB3XLSHBMZZQNK745AK5U3ZLLNKC3TLW4CEFA7OX4DJCOTVQ, seg_pos={0 0}, upload=84097]: Uploading piece... Jun 23 01:28:26.612001063 (105h5m2.175324064s) [noise=true , address=104.36.82.3:5797 , piece_id=CJD4BHNHEVAGUJ5K55QM7OLAQQLEYRD4C6KJQBZ4FKJLDREGLNRQ, seg_pos={0 0}, upload=84097]: Uploading piece... Jun 23 01:28:26.687672089 (105h5m2.250995077s) [upload=84097]: Next returned nil segment Jun 23 01:28:26.687687969 (105h5m2.25101097s) [upload=84097]: Waiting for error group managing segments... Jun 23 01:28:26.707815366 (105h5m2.271138367s) [noise=true , address=79.127.220.98:4410 , piece_id=DLW3G47XXWAJ4FZAZDEL6TCIN5FKG26LUNVELDWSUE2GDH6IPBLQ, seg_pos={0 0}, upload=84097]: Done uploading piece. err: ecclient: upload failed (node:1Lay6oZEhDEeDmmkisrLND1j82PurLHV95nGJoKkk5wKUaK5S7, address:79.127.220.98:4410): protocol: storage node overloaded, request limit: 500 Jun 23 01:28:26.735633260 (105h5m2.298956259s) [noise=true , address=104.36.82.21:6483 , piece_id=WTUXQB3XLSHBMZZQNK745AK5U3ZLLNKC3TLW4CEFA7OX4DJCOTVQ, seg_pos={0 0}, upload=84097]: Done uploading piece. err: ecclient: upload failed (node:12gxKuBALDLDVDfRGBvMZymwGyPfPtV1BsHdBPqZs34ugvvYacG, address:104.36.82.21:6483): protocol: storage node overloaded, request limit: 2000 Jun 23 01:28:26.793765419 (105h5m2.357088412s) [noise=true , address=156.146.43.225:4448 , piece_id=5SH6UKJRS56UOVGLLW7JKTSAQANRVHYVOMEXZ56UWYM3AEB7QGVA, seg_pos={0 0}, upload=84097]: Done uploading piece. err: ecclient: upload failed (node:1irai8UTYknGTUkKHtKbBNTJTYMQsNTFnzrTEaWCMRDq1NNpGh, address:156.146.43.225:4448): protocol: storage node overloaded, request limit: 2000 Jun 23 01:28:26.868408971 (105h5m2.43173196s) [noise=true , address=77.90.31.15:4729 , piece_id=AAVBJ7VEQX67HQBXQ5DO3O5XJ37TRKU3PS2XXEH3WARMSEAQGQ2A, seg_pos={0 0}, upload=84097]: Done uploading piece. err: <nil> Jun 23 01:28:26.989727871 (105h5m2.553050876s) [noise=true , address=45.91.116.8:5050 , piece_id=M4P3RKV2BUWYZOP2TOLNSMYOYDTOEG5NXXM6XZUYPLFQMLUGUIOA, seg_pos={0 0}, upload=84097]: Done uploading piece. err: <nil> Jun 23 01:28:27.669519830 (105h5m3.23284283s) [noise=true , address=77.90.31.14:4693 , piece_id=YGA5IDHRAXIFCELCNCQF23PEFGJMFQOQHFWY3D7UODW5HYEIEWXA, seg_pos={0 0}, upload=84097]: Done uploading piece. err: <nil> Jun 23 01:28:27.697322244 (105h5m3.260645244s) [noise=true , address=104.36.82.42:6830 , piece_id=KW3RNOZYENVRUR663PVSC4WFCLI3RG5YBITFXOYC6W2GTGEFXTVA, seg_pos={0 0}, upload=84097]: Done uploading piece. err: <nil> Jun 23 01:28:28.071845632 (105h5m3.635168633s) [noise=true , address=104.36.82.55:4217 , piece_id=WQIZ3WHXBLCKNJ4ZVQRC2PQIRH6IVVO7J3UMM5V5GXEMODV36KLQ, seg_pos={0 0}, upload=84097]: Done uploading piece. err: <nil> Jun 23 01:28:28.589172169 (105h5m4.152495155s) [noise=true , address=45.91.116.9:5288 , piece_id=EQWQVX4JDLVTNRTKYNUIFGJYQUXPVUCUCYJPNVW2IYM3CGQY5D6A, seg_pos={0 0}, upload=84097]: Done uploading piece. err: <nil> Jun 23 01:28:29.203146980 (105h5m4.76646998s) [noise=true , address=104.36.82.16:6310 , piece_id=CRHVWU4XYDGUG7XVGC3TI6DC2CLBTSTHND4YB2YFEFPBD6TOAXKQ, seg_pos={0 0}, upload=84097]: Done uploading piece. err: ecclient: upload failed (node:1g14evoaeUnhJThf5QKbQT1Gd8WW9MHvaYQhQ1GRgwPfDLoC5R, address:104.36.82.16:6310): protocol: storage node overloaded, request limit: 2000 Jun 23 01:28:29.434402213 (105h5m4.997725203s) [noise=true , address=45.91.116.10:5084 , piece_id=WCIMJN6XDXUO2VCRKLAVOPVVQUEQDGHR4NP4KQGY27NEUPCD5OFA, seg_pos={0 0}, upload=84097]: Done uploading piece. err: <nil> Jun 23 01:28:30.293775103 (105h5m5.857098094s) [noise=true , address=104.36.82.9:6021 , piece_id=4JQGGBGDZMQ6FDS35A7BZLB5OLZL2JHMD5Q3TXAETVX7TYA74EHQ, seg_pos={0 0}, upload=84097]: Done uploading piece. err: <nil> Jun 23 01:28:30.347258498 (105h5m5.910581502s) [noise=true , address=104.36.82.12:6174 , piece_id=VQG2SHTAARK4FOSQER2YRKHZV7ODG7ZDPHNF6AESZGE3GMITKZZA, seg_pos={0 0}, upload=84097]: Done uploading piece. err: <nil> Jun 23 01:28:30.547306548 (105h5m6.110629586s) [noise=true , address=79.127.219.51:4209 , piece_id=ZHJC5QUBGOHT3HRY4XPPADV7CYINQRXT4ID53KVC6LYXM7IV533A, seg_pos={0 0}, upload=84097]: Done uploading piece. err: <nil> Jun 23 01:28:34.333414200 (105h5m9.896737197s) [noise=true , address=79.127.219.38:4031 , piece_id=TBIY2ZMYVVKMYV4LVGJEKCJXMXYVJKSZKKBXASVNEUU2P7DKZKIQ, seg_pos={0 0}, upload=84097]: Done uploading piece. err: <nil> Jun 23 01:28:42.464111369 (105h5m18.02743436s) [noise=true , address=104.36.82.53:4142 , piece_id=CL2WQNZHZNZM5BFJQIBUBQ7FQCIJSK4AR7PR2OLBIXFULPMBTGIQ, seg_pos={0 0}, upload=84097]: Done uploading piece. err: <nil> Jun 23 01:28:49.581287518 (105h5m25.144610509s) [noise=true , address=104.36.82.3:5797 , piece_id=CJD4BHNHEVAGUJ5K55QM7OLAQQLEYRD4C6KJQBZ4FKJLDREGLNRQ, seg_pos={0 0}, upload=84097]: Done uploading piece. err: <nil> Jun 23 01:29:26.535692548 (105h6m2.099015547s) [noise=true , address=104.36.82.11:6131 , piece_id=BI4HK2NGEXXNEDYGRXRL77E5CONZG6JH4IWMONSB64DHOJLQQZPA, seg_pos={0 0}, upload=84097]: Done uploading piece. err: ecclient: upload failed (node:129jahwAmyPHqyYaXz7XTCK8FakBrYZw94pBKGhPzzX23yjM3RQ, address:104.36.82.11:6131): protocol: expected piece hash; context deadline exceeded; EOF Jun 23 01:29:26.535746358 (105h6m2.099069362s) [seg_pos={0 0}, upload=84097]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jun 23 01:29:26.562546209 (105h6m2.125869225s) [upload=84097]: Done uploading segments... Jun 23 01:29:26.562593265 (105h6m2.125916275s) [upload=84097]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration:* 24.216770268s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [......X.............................X...........X...........] duration: 19.876475267s FAIL (2025-06-23 01:37:56 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== oad=84082]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jun 23 01:27:43.602110205 (105h4m19.165433214s) [upload=84082]: Done uploading segments... Jun 23 01:27:43.602149595 (105h4m19.165472598s) [upload=84082]: Upload finished. err: <nil> Jun 23 01:28:26.568078065 (105h5m2.131401068s) [upload=84097]: Starting upload Jun 23 01:28:26.568176623 (105h5m2.131499629s) [upload=84097]: Done starting upload Jun 23 01:28:26.568333698 (105h5m2.131656699s) [upload=84097]: Uploading segments... Jun 23 01:28:26.569025334 (105h5m2.132348341s) [upload=84097]: Got next segment. Inline: false Jun 23 01:28:26.569144232 (105h5m2.132467237s) [upload=84097]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 23 01:28:26.569226194 (105h5m2.132549206s) [upload=84097]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 23 01:28:26.603382637 (105h5m2.166705641s) [seg_pos={0 0}, upload=84097]: Begin upload segment... Jun 23 01:28:26.603439808 (105h5m2.16676282s) [seg_pos={0 0}, upload=84097]: Done begin upload segment. Jun 23 01:28:26.603481538 (105h5m2.166804559s) [noise=true , address=79.127.219.38:4031 , piece_id=TBIY2ZMYVVKMYV4LVGJEKCJXMXYVJKSZKKBXASVNEUU2P7DKZKIQ, seg_pos={0 0}, upload=84097]: Uploading piece... Jun 23 01:28:26.603498030 (105h5m2.166821033s) [noise=true , address=77.90.31.15:4729 , piece_id=AAVBJ7VEQX67HQBXQ5DO3O5XJ37TRKU3PS2XXEH3WARMSEAQGQ2A, seg_pos={0 0}, upload=84097]: Uploading piece... Jun 23 01:28:26.603958736 (105h5m2.16728174s) [noise=true , address=79.127.219.51:4209 , piece_id=ZHJC5QUBGOHT3HRY4XPPADV7CYINQRXT4ID53KVC6LYXM7IV533A, seg_pos={0 0}, upload=84097]: Uploading piece... Jun 23 01:28:26.603997417 (105h5m2.167320407s) [noise=true , address=79.127.220.98:4410 , piece_id=DLW3G47XXWAJ4FZAZDEL6TCIN5FKG26LUNVELDWSUE2GDH6IPBLQ, seg_pos={0 0}, upload=84097]: Uploading piece... Jun 23 01:28:26.604297025 (105h5m2.167620026s) [noise=true , address=45.91.116.10:5084 , piece_id=WCIMJN6XDXUO2VCRKLAVOPVVQUEQDGHR4NP4KQGY27NEUPCD5OFA, seg_pos={0 0}, upload=84097]: Uploading piece... Jun 23 01:28:26.604692251 (105h5m2.16801525s) [noise=true , address=104.36.82.55:4217 , piece_id=WQIZ3WHXBLCKNJ4ZVQRC2PQIRH6IVVO7J3UMM5V5GXEMODV36KLQ, seg_pos={0 0}, upload=84097]: Uploading piece... Jun 23 01:28:26.605044347 (105h5m2.168367349s) [noise=true , address=104.36.82.53:4142 , piece_id=CL2WQNZHZNZM5BFJQIBUBQ7FQCIJSK4AR7PR2OLBIXFULPMBTGIQ, seg_pos={0 0}, upload=84097]: Uploading piece... Jun 23 01:28:26.605369657 (105h5m2.168692667s) [noise=true , address=45.91.116.8:5050 , piece_id=M4P3RKV2BUWYZOP2TOLNSMYOYDTOEG5NXXM6XZUYPLFQMLUGUIOA, seg_pos={0 0}, upload=84097]: Uploading piece... Jun 23 01:28:26.605753081 (105h5m2.169076095s) [noise=true , address=77.90.31.14:4693 , piece_id=YGA5IDHRAXIFCELCNCQF23PEFGJMFQOQHFWY3D7UODW5HYEIEWXA, seg_pos={0 0}, upload=84097]: Uploading piece... Jun 23 01:28:26.606212150 (105h5m2.169535158s) [noise=true , address=104.36.82.9:6021 , piece_id=4JQGGBGDZMQ6FDS35A7BZLB5OLZL2JHMD5Q3TXAETVX7TYA74EHQ, seg_pos={0 0}, upload=84097]: Uploading piece... Jun 23 01:28:26.606654203 (105h5m2.169977203s) [noise=true , address=104.36.82.16:6310 , piece_id=CRHVWU4XYDGUG7XVGC3TI6DC2CLBTSTHND4YB2YFEFPBD6TOAXKQ, seg_pos={0 0}, upload=84097]: Uploading piece... Jun 23 01:28:26.606961844 (105h5m2.170284849s) [noise=true , address=156.146.43.225:4448 , piece_id=5SH6UKJRS56UOVGLLW7JKTSAQANRVHYVOMEXZ56UWYM3AEB7QGVA, seg_pos={0 0}, upload=84097]: Uploading piece... Jun 23 01:28:26.607374002 (105h5m2.170697008s) [noise=true , address=104.36.82.12:6174 , piece_id=VQG2SHTAARK4FOSQER2YRKHZV7ODG7ZDPHNF6AESZGE3GMITKZZA, seg_pos={0 0}, upload=84097]: Uploading piece... Jun 23 01:28:26.607734796 (105h5m2.171057797s) [noise=true , address=45.91.116.9:5288 , piece_id=EQWQVX4JDLVTNRTKYNUIFGJYQUXPVUCUCYJPNVW2IYM3CGQY5D6A, seg_pos={0 0}, upload=84097]: Uploading piece... Jun 23 01:28:26.608082032 (105h5m2.171405034s) [noise=true , address=104.36.82.11:6131 , piece_id=BI4HK2NGEXXNEDYGRXRL77E5CONZG6JH4IWMONSB64DHOJLQQZPA, seg_pos={0 0}, upload=84097]: Uploading piece... Jun 23 01:28:26.608428729 (105h5m2.171751737s) [noise=true , address=104.36.82.42:6830 , piece_id=KW3RNOZYENVRUR663PVSC4WFCLI3RG5YBITFXOYC6W2GTGEFXTVA, seg_pos={0 0}, upload=84097]: Uploading piece... Jun 23 01:28:26.610585268 (105h5m2.173908273s) [noise=true , address=104.36.82.21:6483 , piece_id=WTUXQB3XLSHBMZZQNK745AK5U3ZLLNKC3TLW4CEFA7OX4DJCOTVQ, seg_pos={0 0}, upload=84097]: Uploading piece... Jun 23 01:28:26.612001063 (105h5m2.175324064s) [noise=true , address=104.36.82.3:5797 , piece_id=CJD4BHNHEVAGUJ5K55QM7OLAQQLEYRD4C6KJQBZ4FKJLDREGLNRQ, seg_pos={0 0}, upload=84097]: Uploading piece... Jun 23 01:28:26.687672089 (105h5m2.250995077s) [upload=84097]: Next returned nil segment Jun 23 01:28:26.687687969 (105h5m2.25101097s) [upload=84097]: Waiting for error group managing segments... Jun 23 01:28:26.707815366 (105h5m2.271138367s) [noise=true , address=79.127.220.98:4410 , piece_id=DLW3G47XXWAJ4FZAZDEL6TCIN5FKG26LUNVELDWSUE2GDH6IPBLQ, seg_pos={0 0}, upload=84097]: Done uploading piece. err: ecclient: upload failed (node:1Lay6oZEhDEeDmmkisrLND1j82PurLHV95nGJoKkk5wKUaK5S7, address:79.127.220.98:4410): protocol: storage node overloaded, request limit: 500 Jun 23 01:28:26.735633260 (105h5m2.298956259s) [noise=true , address=104.36.82.21:6483 , piece_id=WTUXQB3XLSHBMZZQNK745AK5U3ZLLNKC3TLW4CEFA7OX4DJCOTVQ, seg_pos={0 0}, upload=84097]: Done uploading piece. err: ecclient: upload failed (node:12gxKuBALDLDVDfRGBvMZymwGyPfPtV1BsHdBPqZs34ugvvYacG, address:104.36.82.21:6483): protocol: storage node overloaded, request limit: 2000 Jun 23 01:28:26.793765419 (105h5m2.357088412s) [noise=true , address=156.146.43.225:4448 , piece_id=5SH6UKJRS56UOVGLLW7JKTSAQANRVHYVOMEXZ56UWYM3AEB7QGVA, seg_pos={0 0}, upload=84097]: Done uploading piece. err: ecclient: upload failed (node:1irai8UTYknGTUkKHtKbBNTJTYMQsNTFnzrTEaWCMRDq1NNpGh, address:156.146.43.225:4448): protocol: storage node overloaded, request limit: 2000 Jun 23 01:28:26.868408971 (105h5m2.43173196s) [noise=true , address=77.90.31.15:4729 , piece_id=AAVBJ7VEQX67HQBXQ5DO3O5XJ37TRKU3PS2XXEH3WARMSEAQGQ2A, seg_pos={0 0}, upload=84097]: Done uploading piece. err: <nil> Jun 23 01:28:26.989727871 (105h5m2.553050876s) [noise=true , address=45.91.116.8:5050 , piece_id=M4P3RKV2BUWYZOP2TOLNSMYOYDTOEG5NXXM6XZUYPLFQMLUGUIOA, seg_pos={0 0}, upload=84097]: Done uploading piece. err: <nil> Jun 23 01:28:27.669519830 (105h5m3.23284283s) [noise=true , address=77.90.31.14:4693 , piece_id=YGA5IDHRAXIFCELCNCQF23PEFGJMFQOQHFWY3D7UODW5HYEIEWXA, seg_pos={0 0}, upload=84097]: Done uploading piece. err: <nil> Jun 23 01:28:27.697322244 (105h5m3.260645244s) [noise=true , address=104.36.82.42:6830 , piece_id=KW3RNOZYENVRUR663PVSC4WFCLI3RG5YBITFXOYC6W2GTGEFXTVA, seg_pos={0 0}, upload=84097]: Done uploading piece. err: <nil> Jun 23 01:28:28.071845632 (105h5m3.635168633s) [noise=true , address=104.36.82.55:4217 , piece_id=WQIZ3WHXBLCKNJ4ZVQRC2PQIRH6IVVO7J3UMM5V5GXEMODV36KLQ, seg_pos={0 0}, upload=84097]: Done uploading piece. err: <nil> Jun 23 01:28:28.589172169 (105h5m4.152495155s) [noise=true , address=45.91.116.9:5288 , piece_id=EQWQVX4JDLVTNRTKYNUIFGJYQUXPVUCUCYJPNVW2IYM3CGQY5D6A, seg_pos={0 0}, upload=84097]: Done uploading piece. err: <nil> Jun 23 01:28:29.203146980 (105h5m4.76646998s) [noise=true , address=104.36.82.16:6310 , piece_id=CRHVWU4XYDGUG7XVGC3TI6DC2CLBTSTHND4YB2YFEFPBD6TOAXKQ, seg_pos={0 0}, upload=84097]: Done uploading piece. err: ecclient: upload failed (node:1g14evoaeUnhJThf5QKbQT1Gd8WW9MHvaYQhQ1GRgwPfDLoC5R, address:104.36.82.16:6310): protocol: storage node overloaded, request limit: 2000 Jun 23 01:28:29.434402213 (105h5m4.997725203s) [noise=true , address=45.91.116.10:5084 , piece_id=WCIMJN6XDXUO2VCRKLAVOPVVQUEQDGHR4NP4KQGY27NEUPCD5OFA, seg_pos={0 0}, upload=84097]: Done uploading piece. err: <nil> Jun 23 01:28:30.293775103 (105h5m5.857098094s) [noise=true , address=104.36.82.9:6021 , piece_id=4JQGGBGDZMQ6FDS35A7BZLB5OLZL2JHMD5Q3TXAETVX7TYA74EHQ, seg_pos={0 0}, upload=84097]: Done uploading piece. err: <nil> Jun 23 01:28:30.347258498 (105h5m5.910581502s) [noise=true , address=104.36.82.12:6174 , piece_id=VQG2SHTAARK4FOSQER2YRKHZV7ODG7ZDPHNF6AESZGE3GMITKZZA, seg_pos={0 0}, upload=84097]: Done uploading piece. err: <nil> Jun 23 01:28:30.547306548 (105h5m6.110629586s) [noise=true , address=79.127.219.51:4209 , piece_id=ZHJC5QUBGOHT3HRY4XPPADV7CYINQRXT4ID53KVC6LYXM7IV533A, seg_pos={0 0}, upload=84097]: Done uploading piece. err: <nil> Jun 23 01:28:34.333414200 (105h5m9.896737197s) [noise=true , address=79.127.219.38:4031 , piece_id=TBIY2ZMYVVKMYV4LVGJEKCJXMXYVJKSZKKBXASVNEUU2P7DKZKIQ, seg_pos={0 0}, upload=84097]: Done uploading piece. err: <nil> Jun 23 01:28:42.464111369 (105h5m18.02743436s) [noise=true , address=104.36.82.53:4142 , piece_id=CL2WQNZHZNZM5BFJQIBUBQ7FQCIJSK4AR7PR2OLBIXFULPMBTGIQ, seg_pos={0 0}, upload=84097]: Done uploading piece. err: <nil> Jun 23 01:28:49.581287518 (105h5m25.144610509s) [noise=true , address=104.36.82.3:5797 , piece_id=CJD4BHNHEVAGUJ5K55QM7OLAQQLEYRD4C6KJQBZ4FKJLDREGLNRQ, seg_pos={0 0}, upload=84097]: Done uploading piece. err: <nil> Jun 23 01:29:26.535692548 (105h6m2.099015547s) [noise=true , address=104.36.82.11:6131 , piece_id=BI4HK2NGEXXNEDYGRXRL77E5CONZG6JH4IWMONSB64DHOJLQQZPA, seg_pos={0 0}, upload=84097]: Done uploading piece. err: ecclient: upload failed (node:129jahwAmyPHqyYaXz7XTCK8FakBrYZw94pBKGhPzzX23yjM3RQ, address:104.36.82.11:6131): protocol: expected piece hash; context deadline exceeded; EOF Jun 23 01:29:26.535746358 (105h6m2.099069362s) [seg_pos={0 0}, upload=84097]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jun 23 01:29:26.562546209 (105h6m2.125869225s) [upload=84097]: Done uploading segments... Jun 23 01:29:26.562593265 (105h6m2.125916275s) [upload=84097]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration:* 19.876504323s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [.........X.....................X...................X........] duration: 47.057898706s FAIL (2025-06-23 01:37:54 UTC) operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 47.057973318s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [.......X.............................X...........X..........] duration: 8.337105811s FAIL (2025-06-23 01:37:07 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== oad=84082]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jun 23 01:27:43.602110205 (105h4m19.165433214s) [upload=84082]: Done uploading segments... Jun 23 01:27:43.602149595 (105h4m19.165472598s) [upload=84082]: Upload finished. err: <nil> Jun 23 01:28:26.568078065 (105h5m2.131401068s) [upload=84097]: Starting upload Jun 23 01:28:26.568176623 (105h5m2.131499629s) [upload=84097]: Done starting upload Jun 23 01:28:26.568333698 (105h5m2.131656699s) [upload=84097]: Uploading segments... Jun 23 01:28:26.569025334 (105h5m2.132348341s) [upload=84097]: Got next segment. Inline: false Jun 23 01:28:26.569144232 (105h5m2.132467237s) [upload=84097]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 23 01:28:26.569226194 (105h5m2.132549206s) [upload=84097]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 23 01:28:26.603382637 (105h5m2.166705641s) [seg_pos={0 0}, upload=84097]: Begin upload segment... Jun 23 01:28:26.603439808 (105h5m2.16676282s) [seg_pos={0 0}, upload=84097]: Done begin upload segment. Jun 23 01:28:26.603481538 (105h5m2.166804559s) [noise=true , address=79.127.219.38:4031 , piece_id=TBIY2ZMYVVKMYV4LVGJEKCJXMXYVJKSZKKBXASVNEUU2P7DKZKIQ, seg_pos={0 0}, upload=84097]: Uploading piece... Jun 23 01:28:26.603498030 (105h5m2.166821033s) [noise=true , address=77.90.31.15:4729 , piece_id=AAVBJ7VEQX67HQBXQ5DO3O5XJ37TRKU3PS2XXEH3WARMSEAQGQ2A, seg_pos={0 0}, upload=84097]: Uploading piece... Jun 23 01:28:26.603958736 (105h5m2.16728174s) [noise=true , address=79.127.219.51:4209 , piece_id=ZHJC5QUBGOHT3HRY4XPPADV7CYINQRXT4ID53KVC6LYXM7IV533A, seg_pos={0 0}, upload=84097]: Uploading piece... Jun 23 01:28:26.603997417 (105h5m2.167320407s) [noise=true , address=79.127.220.98:4410 , piece_id=DLW3G47XXWAJ4FZAZDEL6TCIN5FKG26LUNVELDWSUE2GDH6IPBLQ, seg_pos={0 0}, upload=84097]: Uploading piece... Jun 23 01:28:26.604297025 (105h5m2.167620026s) [noise=true , address=45.91.116.10:5084 , piece_id=WCIMJN6XDXUO2VCRKLAVOPVVQUEQDGHR4NP4KQGY27NEUPCD5OFA, seg_pos={0 0}, upload=84097]: Uploading piece... Jun 23 01:28:26.604692251 (105h5m2.16801525s) [noise=true , address=104.36.82.55:4217 , piece_id=WQIZ3WHXBLCKNJ4ZVQRC2PQIRH6IVVO7J3UMM5V5GXEMODV36KLQ, seg_pos={0 0}, upload=84097]: Uploading piece... Jun 23 01:28:26.605044347 (105h5m2.168367349s) [noise=true , address=104.36.82.53:4142 , piece_id=CL2WQNZHZNZM5BFJQIBUBQ7FQCIJSK4AR7PR2OLBIXFULPMBTGIQ, seg_pos={0 0}, upload=84097]: Uploading piece... Jun 23 01:28:26.605369657 (105h5m2.168692667s) [noise=true , address=45.91.116.8:5050 , piece_id=M4P3RKV2BUWYZOP2TOLNSMYOYDTOEG5NXXM6XZUYPLFQMLUGUIOA, seg_pos={0 0}, upload=84097]: Uploading piece... Jun 23 01:28:26.605753081 (105h5m2.169076095s) [noise=true , address=77.90.31.14:4693 , piece_id=YGA5IDHRAXIFCELCNCQF23PEFGJMFQOQHFWY3D7UODW5HYEIEWXA, seg_pos={0 0}, upload=84097]: Uploading piece... Jun 23 01:28:26.606212150 (105h5m2.169535158s) [noise=true , address=104.36.82.9:6021 , piece_id=4JQGGBGDZMQ6FDS35A7BZLB5OLZL2JHMD5Q3TXAETVX7TYA74EHQ, seg_pos={0 0}, upload=84097]: Uploading piece... Jun 23 01:28:26.606654203 (105h5m2.169977203s) [noise=true , address=104.36.82.16:6310 , piece_id=CRHVWU4XYDGUG7XVGC3TI6DC2CLBTSTHND4YB2YFEFPBD6TOAXKQ, seg_pos={0 0}, upload=84097]: Uploading piece... Jun 23 01:28:26.606961844 (105h5m2.170284849s) [noise=true , address=156.146.43.225:4448 , piece_id=5SH6UKJRS56UOVGLLW7JKTSAQANRVHYVOMEXZ56UWYM3AEB7QGVA, seg_pos={0 0}, upload=84097]: Uploading piece... Jun 23 01:28:26.607374002 (105h5m2.170697008s) [noise=true , address=104.36.82.12:6174 , piece_id=VQG2SHTAARK4FOSQER2YRKHZV7ODG7ZDPHNF6AESZGE3GMITKZZA, seg_pos={0 0}, upload=84097]: Uploading piece... Jun 23 01:28:26.607734796 (105h5m2.171057797s) [noise=true , address=45.91.116.9:5288 , piece_id=EQWQVX4JDLVTNRTKYNUIFGJYQUXPVUCUCYJPNVW2IYM3CGQY5D6A, seg_pos={0 0}, upload=84097]: Uploading piece... Jun 23 01:28:26.608082032 (105h5m2.171405034s) [noise=true , address=104.36.82.11:6131 , piece_id=BI4HK2NGEXXNEDYGRXRL77E5CONZG6JH4IWMONSB64DHOJLQQZPA, seg_pos={0 0}, upload=84097]: Uploading piece... Jun 23 01:28:26.608428729 (105h5m2.171751737s) [noise=true , address=104.36.82.42:6830 , piece_id=KW3RNOZYENVRUR663PVSC4WFCLI3RG5YBITFXOYC6W2GTGEFXTVA, seg_pos={0 0}, upload=84097]: Uploading piece... Jun 23 01:28:26.610585268 (105h5m2.173908273s) [noise=true , address=104.36.82.21:6483 , piece_id=WTUXQB3XLSHBMZZQNK745AK5U3ZLLNKC3TLW4CEFA7OX4DJCOTVQ, seg_pos={0 0}, upload=84097]: Uploading piece... Jun 23 01:28:26.612001063 (105h5m2.175324064s) [noise=true , address=104.36.82.3:5797 , piece_id=CJD4BHNHEVAGUJ5K55QM7OLAQQLEYRD4C6KJQBZ4FKJLDREGLNRQ, seg_pos={0 0}, upload=84097]: Uploading piece... Jun 23 01:28:26.687672089 (105h5m2.250995077s) [upload=84097]: Next returned nil segment Jun 23 01:28:26.687687969 (105h5m2.25101097s) [upload=84097]: Waiting for error group managing segments... Jun 23 01:28:26.707815366 (105h5m2.271138367s) [noise=true , address=79.127.220.98:4410 , piece_id=DLW3G47XXWAJ4FZAZDEL6TCIN5FKG26LUNVELDWSUE2GDH6IPBLQ, seg_pos={0 0}, upload=84097]: Done uploading piece. err: ecclient: upload failed (node:1Lay6oZEhDEeDmmkisrLND1j82PurLHV95nGJoKkk5wKUaK5S7, address:79.127.220.98:4410): protocol: storage node overloaded, request limit: 500 Jun 23 01:28:26.735633260 (105h5m2.298956259s) [noise=true , address=104.36.82.21:6483 , piece_id=WTUXQB3XLSHBMZZQNK745AK5U3ZLLNKC3TLW4CEFA7OX4DJCOTVQ, seg_pos={0 0}, upload=84097]: Done uploading piece. err: ecclient: upload failed (node:12gxKuBALDLDVDfRGBvMZymwGyPfPtV1BsHdBPqZs34ugvvYacG, address:104.36.82.21:6483): protocol: storage node overloaded, request limit: 2000 Jun 23 01:28:26.793765419 (105h5m2.357088412s) [noise=true , address=156.146.43.225:4448 , piece_id=5SH6UKJRS56UOVGLLW7JKTSAQANRVHYVOMEXZ56UWYM3AEB7QGVA, seg_pos={0 0}, upload=84097]: Done uploading piece. err: ecclient: upload failed (node:1irai8UTYknGTUkKHtKbBNTJTYMQsNTFnzrTEaWCMRDq1NNpGh, address:156.146.43.225:4448): protocol: storage node overloaded, request limit: 2000 Jun 23 01:28:26.868408971 (105h5m2.43173196s) [noise=true , address=77.90.31.15:4729 , piece_id=AAVBJ7VEQX67HQBXQ5DO3O5XJ37TRKU3PS2XXEH3WARMSEAQGQ2A, seg_pos={0 0}, upload=84097]: Done uploading piece. err: <nil> Jun 23 01:28:26.989727871 (105h5m2.553050876s) [noise=true , address=45.91.116.8:5050 , piece_id=M4P3RKV2BUWYZOP2TOLNSMYOYDTOEG5NXXM6XZUYPLFQMLUGUIOA, seg_pos={0 0}, upload=84097]: Done uploading piece. err: <nil> Jun 23 01:28:27.669519830 (105h5m3.23284283s) [noise=true , address=77.90.31.14:4693 , piece_id=YGA5IDHRAXIFCELCNCQF23PEFGJMFQOQHFWY3D7UODW5HYEIEWXA, seg_pos={0 0}, upload=84097]: Done uploading piece. err: <nil> Jun 23 01:28:27.697322244 (105h5m3.260645244s) [noise=true , address=104.36.82.42:6830 , piece_id=KW3RNOZYENVRUR663PVSC4WFCLI3RG5YBITFXOYC6W2GTGEFXTVA, seg_pos={0 0}, upload=84097]: Done uploading piece. err: <nil> Jun 23 01:28:28.071845632 (105h5m3.635168633s) [noise=true , address=104.36.82.55:4217 , piece_id=WQIZ3WHXBLCKNJ4ZVQRC2PQIRH6IVVO7J3UMM5V5GXEMODV36KLQ, seg_pos={0 0}, upload=84097]: Done uploading piece. err: <nil> Jun 23 01:28:28.589172169 (105h5m4.152495155s) [noise=true , address=45.91.116.9:5288 , piece_id=EQWQVX4JDLVTNRTKYNUIFGJYQUXPVUCUCYJPNVW2IYM3CGQY5D6A, seg_pos={0 0}, upload=84097]: Done uploading piece. err: <nil> Jun 23 01:28:29.203146980 (105h5m4.76646998s) [noise=true , address=104.36.82.16:6310 , piece_id=CRHVWU4XYDGUG7XVGC3TI6DC2CLBTSTHND4YB2YFEFPBD6TOAXKQ, seg_pos={0 0}, upload=84097]: Done uploading piece. err: ecclient: upload failed (node:1g14evoaeUnhJThf5QKbQT1Gd8WW9MHvaYQhQ1GRgwPfDLoC5R, address:104.36.82.16:6310): protocol: storage node overloaded, request limit: 2000 Jun 23 01:28:29.434402213 (105h5m4.997725203s) [noise=true , address=45.91.116.10:5084 , piece_id=WCIMJN6XDXUO2VCRKLAVOPVVQUEQDGHR4NP4KQGY27NEUPCD5OFA, seg_pos={0 0}, upload=84097]: Done uploading piece. err: <nil> Jun 23 01:28:30.293775103 (105h5m5.857098094s) [noise=true , address=104.36.82.9:6021 , piece_id=4JQGGBGDZMQ6FDS35A7BZLB5OLZL2JHMD5Q3TXAETVX7TYA74EHQ, seg_pos={0 0}, upload=84097]: Done uploading piece. err: <nil> Jun 23 01:28:30.347258498 (105h5m5.910581502s) [noise=true , address=104.36.82.12:6174 , piece_id=VQG2SHTAARK4FOSQER2YRKHZV7ODG7ZDPHNF6AESZGE3GMITKZZA, seg_pos={0 0}, upload=84097]: Done uploading piece. err: <nil> Jun 23 01:28:30.547306548 (105h5m6.110629586s) [noise=true , address=79.127.219.51:4209 , piece_id=ZHJC5QUBGOHT3HRY4XPPADV7CYINQRXT4ID53KVC6LYXM7IV533A, seg_pos={0 0}, upload=84097]: Done uploading piece. err: <nil> Jun 23 01:28:34.333414200 (105h5m9.896737197s) [noise=true , address=79.127.219.38:4031 , piece_id=TBIY2ZMYVVKMYV4LVGJEKCJXMXYVJKSZKKBXASVNEUU2P7DKZKIQ, seg_pos={0 0}, upload=84097]: Done uploading piece. err: <nil> Jun 23 01:28:42.464111369 (105h5m18.02743436s) [noise=true , address=104.36.82.53:4142 , piece_id=CL2WQNZHZNZM5BFJQIBUBQ7FQCIJSK4AR7PR2OLBIXFULPMBTGIQ, seg_pos={0 0}, upload=84097]: Done uploading piece. err: <nil> Jun 23 01:28:49.581287518 (105h5m25.144610509s) [noise=true , address=104.36.82.3:5797 , piece_id=CJD4BHNHEVAGUJ5K55QM7OLAQQLEYRD4C6KJQBZ4FKJLDREGLNRQ, seg_pos={0 0}, upload=84097]: Done uploading piece. err: <nil> Jun 23 01:29:26.535692548 (105h6m2.099015547s) [noise=true , address=104.36.82.11:6131 , piece_id=BI4HK2NGEXXNEDYGRXRL77E5CONZG6JH4IWMONSB64DHOJLQQZPA, seg_pos={0 0}, upload=84097]: Done uploading piece. err: ecclient: upload failed (node:129jahwAmyPHqyYaXz7XTCK8FakBrYZw94pBKGhPzzX23yjM3RQ, address:104.36.82.11:6131): protocol: expected piece hash; context deadline exceeded; EOF Jun 23 01:29:26.535746358 (105h6m2.099069362s) [seg_pos={0 0}, upload=84097]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jun 23 01:29:26.562546209 (105h6m2.125869225s) [upload=84097]: Done uploading segments... Jun 23 01:29:26.562593265 (105h6m2.125916275s) [upload=84097]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration:* 8.337137678s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [........X.............................X...........X.........] duration: 21.648757231s FAIL (2025-06-23 01:36:16 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== oad=84082]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jun 23 01:27:43.602110205 (105h4m19.165433214s) [upload=84082]: Done uploading segments... Jun 23 01:27:43.602149595 (105h4m19.165472598s) [upload=84082]: Upload finished. err: <nil> Jun 23 01:28:26.568078065 (105h5m2.131401068s) [upload=84097]: Starting upload Jun 23 01:28:26.568176623 (105h5m2.131499629s) [upload=84097]: Done starting upload Jun 23 01:28:26.568333698 (105h5m2.131656699s) [upload=84097]: Uploading segments... Jun 23 01:28:26.569025334 (105h5m2.132348341s) [upload=84097]: Got next segment. Inline: false Jun 23 01:28:26.569144232 (105h5m2.132467237s) [upload=84097]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 23 01:28:26.569226194 (105h5m2.132549206s) [upload=84097]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 23 01:28:26.603382637 (105h5m2.166705641s) [seg_pos={0 0}, upload=84097]: Begin upload segment... Jun 23 01:28:26.603439808 (105h5m2.16676282s) [seg_pos={0 0}, upload=84097]: Done begin upload segment. Jun 23 01:28:26.603481538 (105h5m2.166804559s) [noise=true , address=79.127.219.38:4031 , piece_id=TBIY2ZMYVVKMYV4LVGJEKCJXMXYVJKSZKKBXASVNEUU2P7DKZKIQ, seg_pos={0 0}, upload=84097]: Uploading piece... Jun 23 01:28:26.603498030 (105h5m2.166821033s) [noise=true , address=77.90.31.15:4729 , piece_id=AAVBJ7VEQX67HQBXQ5DO3O5XJ37TRKU3PS2XXEH3WARMSEAQGQ2A, seg_pos={0 0}, upload=84097]: Uploading piece... Jun 23 01:28:26.603958736 (105h5m2.16728174s) [noise=true , address=79.127.219.51:4209 , piece_id=ZHJC5QUBGOHT3HRY4XPPADV7CYINQRXT4ID53KVC6LYXM7IV533A, seg_pos={0 0}, upload=84097]: Uploading piece... Jun 23 01:28:26.603997417 (105h5m2.167320407s) [noise=true , address=79.127.220.98:4410 , piece_id=DLW3G47XXWAJ4FZAZDEL6TCIN5FKG26LUNVELDWSUE2GDH6IPBLQ, seg_pos={0 0}, upload=84097]: Uploading piece... Jun 23 01:28:26.604297025 (105h5m2.167620026s) [noise=true , address=45.91.116.10:5084 , piece_id=WCIMJN6XDXUO2VCRKLAVOPVVQUEQDGHR4NP4KQGY27NEUPCD5OFA, seg_pos={0 0}, upload=84097]: Uploading piece... Jun 23 01:28:26.604692251 (105h5m2.16801525s) [noise=true , address=104.36.82.55:4217 , piece_id=WQIZ3WHXBLCKNJ4ZVQRC2PQIRH6IVVO7J3UMM5V5GXEMODV36KLQ, seg_pos={0 0}, upload=84097]: Uploading piece... Jun 23 01:28:26.605044347 (105h5m2.168367349s) [noise=true , address=104.36.82.53:4142 , piece_id=CL2WQNZHZNZM5BFJQIBUBQ7FQCIJSK4AR7PR2OLBIXFULPMBTGIQ, seg_pos={0 0}, upload=84097]: Uploading piece... Jun 23 01:28:26.605369657 (105h5m2.168692667s) [noise=true , address=45.91.116.8:5050 , piece_id=M4P3RKV2BUWYZOP2TOLNSMYOYDTOEG5NXXM6XZUYPLFQMLUGUIOA, seg_pos={0 0}, upload=84097]: Uploading piece... Jun 23 01:28:26.605753081 (105h5m2.169076095s) [noise=true , address=77.90.31.14:4693 , piece_id=YGA5IDHRAXIFCELCNCQF23PEFGJMFQOQHFWY3D7UODW5HYEIEWXA, seg_pos={0 0}, upload=84097]: Uploading piece... Jun 23 01:28:26.606212150 (105h5m2.169535158s) [noise=true , address=104.36.82.9:6021 , piece_id=4JQGGBGDZMQ6FDS35A7BZLB5OLZL2JHMD5Q3TXAETVX7TYA74EHQ, seg_pos={0 0}, upload=84097]: Uploading piece... Jun 23 01:28:26.606654203 (105h5m2.169977203s) [noise=true , address=104.36.82.16:6310 , piece_id=CRHVWU4XYDGUG7XVGC3TI6DC2CLBTSTHND4YB2YFEFPBD6TOAXKQ, seg_pos={0 0}, upload=84097]: Uploading piece... Jun 23 01:28:26.606961844 (105h5m2.170284849s) [noise=true , address=156.146.43.225:4448 , piece_id=5SH6UKJRS56UOVGLLW7JKTSAQANRVHYVOMEXZ56UWYM3AEB7QGVA, seg_pos={0 0}, upload=84097]: Uploading piece... Jun 23 01:28:26.607374002 (105h5m2.170697008s) [noise=true , address=104.36.82.12:6174 , piece_id=VQG2SHTAARK4FOSQER2YRKHZV7ODG7ZDPHNF6AESZGE3GMITKZZA, seg_pos={0 0}, upload=84097]: Uploading piece... Jun 23 01:28:26.607734796 (105h5m2.171057797s) [noise=true , address=45.91.116.9:5288 , piece_id=EQWQVX4JDLVTNRTKYNUIFGJYQUXPVUCUCYJPNVW2IYM3CGQY5D6A, seg_pos={0 0}, upload=84097]: Uploading piece... Jun 23 01:28:26.608082032 (105h5m2.171405034s) [noise=true , address=104.36.82.11:6131 , piece_id=BI4HK2NGEXXNEDYGRXRL77E5CONZG6JH4IWMONSB64DHOJLQQZPA, seg_pos={0 0}, upload=84097]: Uploading piece... Jun 23 01:28:26.608428729 (105h5m2.171751737s) [noise=true , address=104.36.82.42:6830 , piece_id=KW3RNOZYENVRUR663PVSC4WFCLI3RG5YBITFXOYC6W2GTGEFXTVA, seg_pos={0 0}, upload=84097]: Uploading piece... Jun 23 01:28:26.610585268 (105h5m2.173908273s) [noise=true , address=104.36.82.21:6483 , piece_id=WTUXQB3XLSHBMZZQNK745AK5U3ZLLNKC3TLW4CEFA7OX4DJCOTVQ, seg_pos={0 0}, upload=84097]: Uploading piece... Jun 23 01:28:26.612001063 (105h5m2.175324064s) [noise=true , address=104.36.82.3:5797 , piece_id=CJD4BHNHEVAGUJ5K55QM7OLAQQLEYRD4C6KJQBZ4FKJLDREGLNRQ, seg_pos={0 0}, upload=84097]: Uploading piece... Jun 23 01:28:26.687672089 (105h5m2.250995077s) [upload=84097]: Next returned nil segment Jun 23 01:28:26.687687969 (105h5m2.25101097s) [upload=84097]: Waiting for error group managing segments... Jun 23 01:28:26.707815366 (105h5m2.271138367s) [noise=true , address=79.127.220.98:4410 , piece_id=DLW3G47XXWAJ4FZAZDEL6TCIN5FKG26LUNVELDWSUE2GDH6IPBLQ, seg_pos={0 0}, upload=84097]: Done uploading piece. err: ecclient: upload failed (node:1Lay6oZEhDEeDmmkisrLND1j82PurLHV95nGJoKkk5wKUaK5S7, address:79.127.220.98:4410): protocol: storage node overloaded, request limit: 500 Jun 23 01:28:26.735633260 (105h5m2.298956259s) [noise=true , address=104.36.82.21:6483 , piece_id=WTUXQB3XLSHBMZZQNK745AK5U3ZLLNKC3TLW4CEFA7OX4DJCOTVQ, seg_pos={0 0}, upload=84097]: Done uploading piece. err: ecclient: upload failed (node:12gxKuBALDLDVDfRGBvMZymwGyPfPtV1BsHdBPqZs34ugvvYacG, address:104.36.82.21:6483): protocol: storage node overloaded, request limit: 2000 Jun 23 01:28:26.793765419 (105h5m2.357088412s) [noise=true , address=156.146.43.225:4448 , piece_id=5SH6UKJRS56UOVGLLW7JKTSAQANRVHYVOMEXZ56UWYM3AEB7QGVA, seg_pos={0 0}, upload=84097]: Done uploading piece. err: ecclient: upload failed (node:1irai8UTYknGTUkKHtKbBNTJTYMQsNTFnzrTEaWCMRDq1NNpGh, address:156.146.43.225:4448): protocol: storage node overloaded, request limit: 2000 Jun 23 01:28:26.868408971 (105h5m2.43173196s) [noise=true , address=77.90.31.15:4729 , piece_id=AAVBJ7VEQX67HQBXQ5DO3O5XJ37TRKU3PS2XXEH3WARMSEAQGQ2A, seg_pos={0 0}, upload=84097]: Done uploading piece. err: <nil> Jun 23 01:28:26.989727871 (105h5m2.553050876s) [noise=true , address=45.91.116.8:5050 , piece_id=M4P3RKV2BUWYZOP2TOLNSMYOYDTOEG5NXXM6XZUYPLFQMLUGUIOA, seg_pos={0 0}, upload=84097]: Done uploading piece. err: <nil> Jun 23 01:28:27.669519830 (105h5m3.23284283s) [noise=true , address=77.90.31.14:4693 , piece_id=YGA5IDHRAXIFCELCNCQF23PEFGJMFQOQHFWY3D7UODW5HYEIEWXA, seg_pos={0 0}, upload=84097]: Done uploading piece. err: <nil> Jun 23 01:28:27.697322244 (105h5m3.260645244s) [noise=true , address=104.36.82.42:6830 , piece_id=KW3RNOZYENVRUR663PVSC4WFCLI3RG5YBITFXOYC6W2GTGEFXTVA, seg_pos={0 0}, upload=84097]: Done uploading piece. err: <nil> Jun 23 01:28:28.071845632 (105h5m3.635168633s) [noise=true , address=104.36.82.55:4217 , piece_id=WQIZ3WHXBLCKNJ4ZVQRC2PQIRH6IVVO7J3UMM5V5GXEMODV36KLQ, seg_pos={0 0}, upload=84097]: Done uploading piece. err: <nil> Jun 23 01:28:28.589172169 (105h5m4.152495155s) [noise=true , address=45.91.116.9:5288 , piece_id=EQWQVX4JDLVTNRTKYNUIFGJYQUXPVUCUCYJPNVW2IYM3CGQY5D6A, seg_pos={0 0}, upload=84097]: Done uploading piece. err: <nil> Jun 23 01:28:29.203146980 (105h5m4.76646998s) [noise=true , address=104.36.82.16:6310 , piece_id=CRHVWU4XYDGUG7XVGC3TI6DC2CLBTSTHND4YB2YFEFPBD6TOAXKQ, seg_pos={0 0}, upload=84097]: Done uploading piece. err: ecclient: upload failed (node:1g14evoaeUnhJThf5QKbQT1Gd8WW9MHvaYQhQ1GRgwPfDLoC5R, address:104.36.82.16:6310): protocol: storage node overloaded, request limit: 2000 Jun 23 01:28:29.434402213 (105h5m4.997725203s) [noise=true , address=45.91.116.10:5084 , piece_id=WCIMJN6XDXUO2VCRKLAVOPVVQUEQDGHR4NP4KQGY27NEUPCD5OFA, seg_pos={0 0}, upload=84097]: Done uploading piece. err: <nil> Jun 23 01:28:30.293775103 (105h5m5.857098094s) [noise=true , address=104.36.82.9:6021 , piece_id=4JQGGBGDZMQ6FDS35A7BZLB5OLZL2JHMD5Q3TXAETVX7TYA74EHQ, seg_pos={0 0}, upload=84097]: Done uploading piece. err: <nil> Jun 23 01:28:30.347258498 (105h5m5.910581502s) [noise=true , address=104.36.82.12:6174 , piece_id=VQG2SHTAARK4FOSQER2YRKHZV7ODG7ZDPHNF6AESZGE3GMITKZZA, seg_pos={0 0}, upload=84097]: Done uploading piece. err: <nil> Jun 23 01:28:30.547306548 (105h5m6.110629586s) [noise=true , address=79.127.219.51:4209 , piece_id=ZHJC5QUBGOHT3HRY4XPPADV7CYINQRXT4ID53KVC6LYXM7IV533A, seg_pos={0 0}, upload=84097]: Done uploading piece. err: <nil> Jun 23 01:28:34.333414200 (105h5m9.896737197s) [noise=true , address=79.127.219.38:4031 , piece_id=TBIY2ZMYVVKMYV4LVGJEKCJXMXYVJKSZKKBXASVNEUU2P7DKZKIQ, seg_pos={0 0}, upload=84097]: Done uploading piece. err: <nil> Jun 23 01:28:42.464111369 (105h5m18.02743436s) [noise=true , address=104.36.82.53:4142 , piece_id=CL2WQNZHZNZM5BFJQIBUBQ7FQCIJSK4AR7PR2OLBIXFULPMBTGIQ, seg_pos={0 0}, upload=84097]: Done uploading piece. err: <nil> Jun 23 01:28:49.581287518 (105h5m25.144610509s) [noise=true , address=104.36.82.3:5797 , piece_id=CJD4BHNHEVAGUJ5K55QM7OLAQQLEYRD4C6KJQBZ4FKJLDREGLNRQ, seg_pos={0 0}, upload=84097]: Done uploading piece. err: <nil> Jun 23 01:29:26.535692548 (105h6m2.099015547s) [noise=true , address=104.36.82.11:6131 , piece_id=BI4HK2NGEXXNEDYGRXRL77E5CONZG6JH4IWMONSB64DHOJLQQZPA, seg_pos={0 0}, upload=84097]: Done uploading piece. err: ecclient: upload failed (node:129jahwAmyPHqyYaXz7XTCK8FakBrYZw94pBKGhPzzX23yjM3RQ, address:104.36.82.11:6131): protocol: expected piece hash; context deadline exceeded; EOF Jun 23 01:29:26.535746358 (105h6m2.099069362s) [seg_pos={0 0}, upload=84097]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jun 23 01:29:26.562546209 (105h6m2.125869225s) [upload=84097]: Done uploading segments... Jun 23 01:29:26.562593265 (105h6m2.125916275s) [upload=84097]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration:* 21.648785251s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [..........X.....................X...................X.......] duration: 16.024807081s FAIL (2025-06-23 01:36:14 UTC) operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 16.024827928s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [...........X.....................X...................X......] duration: 38.313331276s FAIL (2025-06-23 01:35:58 UTC) operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 38.313444371s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [.........X.............................X...........X........] duration: 18.916766936s FAIL (2025-06-23 01:35:40 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== oad=84082]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jun 23 01:27:43.602110205 (105h4m19.165433214s) [upload=84082]: Done uploading segments... Jun 23 01:27:43.602149595 (105h4m19.165472598s) [upload=84082]: Upload finished. err: <nil> Jun 23 01:28:26.568078065 (105h5m2.131401068s) [upload=84097]: Starting upload Jun 23 01:28:26.568176623 (105h5m2.131499629s) [upload=84097]: Done starting upload Jun 23 01:28:26.568333698 (105h5m2.131656699s) [upload=84097]: Uploading segments... Jun 23 01:28:26.569025334 (105h5m2.132348341s) [upload=84097]: Got next segment. Inline: false Jun 23 01:28:26.569144232 (105h5m2.132467237s) [upload=84097]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 23 01:28:26.569226194 (105h5m2.132549206s) [upload=84097]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 23 01:28:26.603382637 (105h5m2.166705641s) [seg_pos={0 0}, upload=84097]: Begin upload segment... Jun 23 01:28:26.603439808 (105h5m2.16676282s) [seg_pos={0 0}, upload=84097]: Done begin upload segment. Jun 23 01:28:26.603481538 (105h5m2.166804559s) [noise=true , address=79.127.219.38:4031 , piece_id=TBIY2ZMYVVKMYV4LVGJEKCJXMXYVJKSZKKBXASVNEUU2P7DKZKIQ, seg_pos={0 0}, upload=84097]: Uploading piece... Jun 23 01:28:26.603498030 (105h5m2.166821033s) [noise=true , address=77.90.31.15:4729 , piece_id=AAVBJ7VEQX67HQBXQ5DO3O5XJ37TRKU3PS2XXEH3WARMSEAQGQ2A, seg_pos={0 0}, upload=84097]: Uploading piece... Jun 23 01:28:26.603958736 (105h5m2.16728174s) [noise=true , address=79.127.219.51:4209 , piece_id=ZHJC5QUBGOHT3HRY4XPPADV7CYINQRXT4ID53KVC6LYXM7IV533A, seg_pos={0 0}, upload=84097]: Uploading piece... Jun 23 01:28:26.603997417 (105h5m2.167320407s) [noise=true , address=79.127.220.98:4410 , piece_id=DLW3G47XXWAJ4FZAZDEL6TCIN5FKG26LUNVELDWSUE2GDH6IPBLQ, seg_pos={0 0}, upload=84097]: Uploading piece... Jun 23 01:28:26.604297025 (105h5m2.167620026s) [noise=true , address=45.91.116.10:5084 , piece_id=WCIMJN6XDXUO2VCRKLAVOPVVQUEQDGHR4NP4KQGY27NEUPCD5OFA, seg_pos={0 0}, upload=84097]: Uploading piece... Jun 23 01:28:26.604692251 (105h5m2.16801525s) [noise=true , address=104.36.82.55:4217 , piece_id=WQIZ3WHXBLCKNJ4ZVQRC2PQIRH6IVVO7J3UMM5V5GXEMODV36KLQ, seg_pos={0 0}, upload=84097]: Uploading piece... Jun 23 01:28:26.605044347 (105h5m2.168367349s) [noise=true , address=104.36.82.53:4142 , piece_id=CL2WQNZHZNZM5BFJQIBUBQ7FQCIJSK4AR7PR2OLBIXFULPMBTGIQ, seg_pos={0 0}, upload=84097]: Uploading piece... Jun 23 01:28:26.605369657 (105h5m2.168692667s) [noise=true , address=45.91.116.8:5050 , piece_id=M4P3RKV2BUWYZOP2TOLNSMYOYDTOEG5NXXM6XZUYPLFQMLUGUIOA, seg_pos={0 0}, upload=84097]: Uploading piece... Jun 23 01:28:26.605753081 (105h5m2.169076095s) [noise=true , address=77.90.31.14:4693 , piece_id=YGA5IDHRAXIFCELCNCQF23PEFGJMFQOQHFWY3D7UODW5HYEIEWXA, seg_pos={0 0}, upload=84097]: Uploading piece... Jun 23 01:28:26.606212150 (105h5m2.169535158s) [noise=true , address=104.36.82.9:6021 , piece_id=4JQGGBGDZMQ6FDS35A7BZLB5OLZL2JHMD5Q3TXAETVX7TYA74EHQ, seg_pos={0 0}, upload=84097]: Uploading piece... Jun 23 01:28:26.606654203 (105h5m2.169977203s) [noise=true , address=104.36.82.16:6310 , piece_id=CRHVWU4XYDGUG7XVGC3TI6DC2CLBTSTHND4YB2YFEFPBD6TOAXKQ, seg_pos={0 0}, upload=84097]: Uploading piece... Jun 23 01:28:26.606961844 (105h5m2.170284849s) [noise=true , address=156.146.43.225:4448 , piece_id=5SH6UKJRS56UOVGLLW7JKTSAQANRVHYVOMEXZ56UWYM3AEB7QGVA, seg_pos={0 0}, upload=84097]: Uploading piece... Jun 23 01:28:26.607374002 (105h5m2.170697008s) [noise=true , address=104.36.82.12:6174 , piece_id=VQG2SHTAARK4FOSQER2YRKHZV7ODG7ZDPHNF6AESZGE3GMITKZZA, seg_pos={0 0}, upload=84097]: Uploading piece... Jun 23 01:28:26.607734796 (105h5m2.171057797s) [noise=true , address=45.91.116.9:5288 , piece_id=EQWQVX4JDLVTNRTKYNUIFGJYQUXPVUCUCYJPNVW2IYM3CGQY5D6A, seg_pos={0 0}, upload=84097]: Uploading piece... Jun 23 01:28:26.608082032 (105h5m2.171405034s) [noise=true , address=104.36.82.11:6131 , piece_id=BI4HK2NGEXXNEDYGRXRL77E5CONZG6JH4IWMONSB64DHOJLQQZPA, seg_pos={0 0}, upload=84097]: Uploading piece... Jun 23 01:28:26.608428729 (105h5m2.171751737s) [noise=true , address=104.36.82.42:6830 , piece_id=KW3RNOZYENVRUR663PVSC4WFCLI3RG5YBITFXOYC6W2GTGEFXTVA, seg_pos={0 0}, upload=84097]: Uploading piece... Jun 23 01:28:26.610585268 (105h5m2.173908273s) [noise=true , address=104.36.82.21:6483 , piece_id=WTUXQB3XLSHBMZZQNK745AK5U3ZLLNKC3TLW4CEFA7OX4DJCOTVQ, seg_pos={0 0}, upload=84097]: Uploading piece... Jun 23 01:28:26.612001063 (105h5m2.175324064s) [noise=true , address=104.36.82.3:5797 , piece_id=CJD4BHNHEVAGUJ5K55QM7OLAQQLEYRD4C6KJQBZ4FKJLDREGLNRQ, seg_pos={0 0}, upload=84097]: Uploading piece... Jun 23 01:28:26.687672089 (105h5m2.250995077s) [upload=84097]: Next returned nil segment Jun 23 01:28:26.687687969 (105h5m2.25101097s) [upload=84097]: Waiting for error group managing segments... Jun 23 01:28:26.707815366 (105h5m2.271138367s) [noise=true , address=79.127.220.98:4410 , piece_id=DLW3G47XXWAJ4FZAZDEL6TCIN5FKG26LUNVELDWSUE2GDH6IPBLQ, seg_pos={0 0}, upload=84097]: Done uploading piece. err: ecclient: upload failed (node:1Lay6oZEhDEeDmmkisrLND1j82PurLHV95nGJoKkk5wKUaK5S7, address:79.127.220.98:4410): protocol: storage node overloaded, request limit: 500 Jun 23 01:28:26.735633260 (105h5m2.298956259s) [noise=true , address=104.36.82.21:6483 , piece_id=WTUXQB3XLSHBMZZQNK745AK5U3ZLLNKC3TLW4CEFA7OX4DJCOTVQ, seg_pos={0 0}, upload=84097]: Done uploading piece. err: ecclient: upload failed (node:12gxKuBALDLDVDfRGBvMZymwGyPfPtV1BsHdBPqZs34ugvvYacG, address:104.36.82.21:6483): protocol: storage node overloaded, request limit: 2000 Jun 23 01:28:26.793765419 (105h5m2.357088412s) [noise=true , address=156.146.43.225:4448 , piece_id=5SH6UKJRS56UOVGLLW7JKTSAQANRVHYVOMEXZ56UWYM3AEB7QGVA, seg_pos={0 0}, upload=84097]: Done uploading piece. err: ecclient: upload failed (node:1irai8UTYknGTUkKHtKbBNTJTYMQsNTFnzrTEaWCMRDq1NNpGh, address:156.146.43.225:4448): protocol: storage node overloaded, request limit: 2000 Jun 23 01:28:26.868408971 (105h5m2.43173196s) [noise=true , address=77.90.31.15:4729 , piece_id=AAVBJ7VEQX67HQBXQ5DO3O5XJ37TRKU3PS2XXEH3WARMSEAQGQ2A, seg_pos={0 0}, upload=84097]: Done uploading piece. err: <nil> Jun 23 01:28:26.989727871 (105h5m2.553050876s) [noise=true , address=45.91.116.8:5050 , piece_id=M4P3RKV2BUWYZOP2TOLNSMYOYDTOEG5NXXM6XZUYPLFQMLUGUIOA, seg_pos={0 0}, upload=84097]: Done uploading piece. err: <nil> Jun 23 01:28:27.669519830 (105h5m3.23284283s) [noise=true , address=77.90.31.14:4693 , piece_id=YGA5IDHRAXIFCELCNCQF23PEFGJMFQOQHFWY3D7UODW5HYEIEWXA, seg_pos={0 0}, upload=84097]: Done uploading piece. err: <nil> Jun 23 01:28:27.697322244 (105h5m3.260645244s) [noise=true , address=104.36.82.42:6830 , piece_id=KW3RNOZYENVRUR663PVSC4WFCLI3RG5YBITFXOYC6W2GTGEFXTVA, seg_pos={0 0}, upload=84097]: Done uploading piece. err: <nil> Jun 23 01:28:28.071845632 (105h5m3.635168633s) [noise=true , address=104.36.82.55:4217 , piece_id=WQIZ3WHXBLCKNJ4ZVQRC2PQIRH6IVVO7J3UMM5V5GXEMODV36KLQ, seg_pos={0 0}, upload=84097]: Done uploading piece. err: <nil> Jun 23 01:28:28.589172169 (105h5m4.152495155s) [noise=true , address=45.91.116.9:5288 , piece_id=EQWQVX4JDLVTNRTKYNUIFGJYQUXPVUCUCYJPNVW2IYM3CGQY5D6A, seg_pos={0 0}, upload=84097]: Done uploading piece. err: <nil> Jun 23 01:28:29.203146980 (105h5m4.76646998s) [noise=true , address=104.36.82.16:6310 , piece_id=CRHVWU4XYDGUG7XVGC3TI6DC2CLBTSTHND4YB2YFEFPBD6TOAXKQ, seg_pos={0 0}, upload=84097]: Done uploading piece. err: ecclient: upload failed (node:1g14evoaeUnhJThf5QKbQT1Gd8WW9MHvaYQhQ1GRgwPfDLoC5R, address:104.36.82.16:6310): protocol: storage node overloaded, request limit: 2000 Jun 23 01:28:29.434402213 (105h5m4.997725203s) [noise=true , address=45.91.116.10:5084 , piece_id=WCIMJN6XDXUO2VCRKLAVOPVVQUEQDGHR4NP4KQGY27NEUPCD5OFA, seg_pos={0 0}, upload=84097]: Done uploading piece. err: <nil> Jun 23 01:28:30.293775103 (105h5m5.857098094s) [noise=true , address=104.36.82.9:6021 , piece_id=4JQGGBGDZMQ6FDS35A7BZLB5OLZL2JHMD5Q3TXAETVX7TYA74EHQ, seg_pos={0 0}, upload=84097]: Done uploading piece. err: <nil> Jun 23 01:28:30.347258498 (105h5m5.910581502s) [noise=true , address=104.36.82.12:6174 , piece_id=VQG2SHTAARK4FOSQER2YRKHZV7ODG7ZDPHNF6AESZGE3GMITKZZA, seg_pos={0 0}, upload=84097]: Done uploading piece. err: <nil> Jun 23 01:28:30.547306548 (105h5m6.110629586s) [noise=true , address=79.127.219.51:4209 , piece_id=ZHJC5QUBGOHT3HRY4XPPADV7CYINQRXT4ID53KVC6LYXM7IV533A, seg_pos={0 0}, upload=84097]: Done uploading piece. err: <nil> Jun 23 01:28:34.333414200 (105h5m9.896737197s) [noise=true , address=79.127.219.38:4031 , piece_id=TBIY2ZMYVVKMYV4LVGJEKCJXMXYVJKSZKKBXASVNEUU2P7DKZKIQ, seg_pos={0 0}, upload=84097]: Done uploading piece. err: <nil> Jun 23 01:28:42.464111369 (105h5m18.02743436s) [noise=true , address=104.36.82.53:4142 , piece_id=CL2WQNZHZNZM5BFJQIBUBQ7FQCIJSK4AR7PR2OLBIXFULPMBTGIQ, seg_pos={0 0}, upload=84097]: Done uploading piece. err: <nil> Jun 23 01:28:49.581287518 (105h5m25.144610509s) [noise=true , address=104.36.82.3:5797 , piece_id=CJD4BHNHEVAGUJ5K55QM7OLAQQLEYRD4C6KJQBZ4FKJLDREGLNRQ, seg_pos={0 0}, upload=84097]: Done uploading piece. err: <nil> Jun 23 01:29:26.535692548 (105h6m2.099015547s) [noise=true , address=104.36.82.11:6131 , piece_id=BI4HK2NGEXXNEDYGRXRL77E5CONZG6JH4IWMONSB64DHOJLQQZPA, seg_pos={0 0}, upload=84097]: Done uploading piece. err: ecclient: upload failed (node:129jahwAmyPHqyYaXz7XTCK8FakBrYZw94pBKGhPzzX23yjM3RQ, address:104.36.82.11:6131): protocol: expected piece hash; context deadline exceeded; EOF Jun 23 01:29:26.535746358 (105h6m2.099069362s) [seg_pos={0 0}, upload=84097]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jun 23 01:29:26.562546209 (105h6m2.125869225s) [upload=84097]: Done uploading segments... Jun 23 01:29:26.562593265 (105h6m2.125916275s) [upload=84097]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration:* 18.916796414s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [............X.....................X...................X.....] duration: 12.321546177s FAIL (2025-06-23 01:35:01 UTC) operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 12.321572744s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [..........X.............................X...........X.......] duration: 19.211464764s FAIL (2025-06-23 01:34:47 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== oad=84082]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jun 23 01:27:43.602110205 (105h4m19.165433214s) [upload=84082]: Done uploading segments... Jun 23 01:27:43.602149595 (105h4m19.165472598s) [upload=84082]: Upload finished. err: <nil> Jun 23 01:28:26.568078065 (105h5m2.131401068s) [upload=84097]: Starting upload Jun 23 01:28:26.568176623 (105h5m2.131499629s) [upload=84097]: Done starting upload Jun 23 01:28:26.568333698 (105h5m2.131656699s) [upload=84097]: Uploading segments... Jun 23 01:28:26.569025334 (105h5m2.132348341s) [upload=84097]: Got next segment. Inline: false Jun 23 01:28:26.569144232 (105h5m2.132467237s) [upload=84097]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 23 01:28:26.569226194 (105h5m2.132549206s) [upload=84097]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 23 01:28:26.603382637 (105h5m2.166705641s) [seg_pos={0 0}, upload=84097]: Begin upload segment... Jun 23 01:28:26.603439808 (105h5m2.16676282s) [seg_pos={0 0}, upload=84097]: Done begin upload segment. Jun 23 01:28:26.603481538 (105h5m2.166804559s) [noise=true , address=79.127.219.38:4031 , piece_id=TBIY2ZMYVVKMYV4LVGJEKCJXMXYVJKSZKKBXASVNEUU2P7DKZKIQ, seg_pos={0 0}, upload=84097]: Uploading piece... Jun 23 01:28:26.603498030 (105h5m2.166821033s) [noise=true , address=77.90.31.15:4729 , piece_id=AAVBJ7VEQX67HQBXQ5DO3O5XJ37TRKU3PS2XXEH3WARMSEAQGQ2A, seg_pos={0 0}, upload=84097]: Uploading piece... Jun 23 01:28:26.603958736 (105h5m2.16728174s) [noise=true , address=79.127.219.51:4209 , piece_id=ZHJC5QUBGOHT3HRY4XPPADV7CYINQRXT4ID53KVC6LYXM7IV533A, seg_pos={0 0}, upload=84097]: Uploading piece... Jun 23 01:28:26.603997417 (105h5m2.167320407s) [noise=true , address=79.127.220.98:4410 , piece_id=DLW3G47XXWAJ4FZAZDEL6TCIN5FKG26LUNVELDWSUE2GDH6IPBLQ, seg_pos={0 0}, upload=84097]: Uploading piece... Jun 23 01:28:26.604297025 (105h5m2.167620026s) [noise=true , address=45.91.116.10:5084 , piece_id=WCIMJN6XDXUO2VCRKLAVOPVVQUEQDGHR4NP4KQGY27NEUPCD5OFA, seg_pos={0 0}, upload=84097]: Uploading piece... Jun 23 01:28:26.604692251 (105h5m2.16801525s) [noise=true , address=104.36.82.55:4217 , piece_id=WQIZ3WHXBLCKNJ4ZVQRC2PQIRH6IVVO7J3UMM5V5GXEMODV36KLQ, seg_pos={0 0}, upload=84097]: Uploading piece... Jun 23 01:28:26.605044347 (105h5m2.168367349s) [noise=true , address=104.36.82.53:4142 , piece_id=CL2WQNZHZNZM5BFJQIBUBQ7FQCIJSK4AR7PR2OLBIXFULPMBTGIQ, seg_pos={0 0}, upload=84097]: Uploading piece... Jun 23 01:28:26.605369657 (105h5m2.168692667s) [noise=true , address=45.91.116.8:5050 , piece_id=M4P3RKV2BUWYZOP2TOLNSMYOYDTOEG5NXXM6XZUYPLFQMLUGUIOA, seg_pos={0 0}, upload=84097]: Uploading piece... Jun 23 01:28:26.605753081 (105h5m2.169076095s) [noise=true , address=77.90.31.14:4693 , piece_id=YGA5IDHRAXIFCELCNCQF23PEFGJMFQOQHFWY3D7UODW5HYEIEWXA, seg_pos={0 0}, upload=84097]: Uploading piece... Jun 23 01:28:26.606212150 (105h5m2.169535158s) [noise=true , address=104.36.82.9:6021 , piece_id=4JQGGBGDZMQ6FDS35A7BZLB5OLZL2JHMD5Q3TXAETVX7TYA74EHQ, seg_pos={0 0}, upload=84097]: Uploading piece... Jun 23 01:28:26.606654203 (105h5m2.169977203s) [noise=true , address=104.36.82.16:6310 , piece_id=CRHVWU4XYDGUG7XVGC3TI6DC2CLBTSTHND4YB2YFEFPBD6TOAXKQ, seg_pos={0 0}, upload=84097]: Uploading piece... Jun 23 01:28:26.606961844 (105h5m2.170284849s) [noise=true , address=156.146.43.225:4448 , piece_id=5SH6UKJRS56UOVGLLW7JKTSAQANRVHYVOMEXZ56UWYM3AEB7QGVA, seg_pos={0 0}, upload=84097]: Uploading piece... Jun 23 01:28:26.607374002 (105h5m2.170697008s) [noise=true , address=104.36.82.12:6174 , piece_id=VQG2SHTAARK4FOSQER2YRKHZV7ODG7ZDPHNF6AESZGE3GMITKZZA, seg_pos={0 0}, upload=84097]: Uploading piece... Jun 23 01:28:26.607734796 (105h5m2.171057797s) [noise=true , address=45.91.116.9:5288 , piece_id=EQWQVX4JDLVTNRTKYNUIFGJYQUXPVUCUCYJPNVW2IYM3CGQY5D6A, seg_pos={0 0}, upload=84097]: Uploading piece... Jun 23 01:28:26.608082032 (105h5m2.171405034s) [noise=true , address=104.36.82.11:6131 , piece_id=BI4HK2NGEXXNEDYGRXRL77E5CONZG6JH4IWMONSB64DHOJLQQZPA, seg_pos={0 0}, upload=84097]: Uploading piece... Jun 23 01:28:26.608428729 (105h5m2.171751737s) [noise=true , address=104.36.82.42:6830 , piece_id=KW3RNOZYENVRUR663PVSC4WFCLI3RG5YBITFXOYC6W2GTGEFXTVA, seg_pos={0 0}, upload=84097]: Uploading piece... Jun 23 01:28:26.610585268 (105h5m2.173908273s) [noise=true , address=104.36.82.21:6483 , piece_id=WTUXQB3XLSHBMZZQNK745AK5U3ZLLNKC3TLW4CEFA7OX4DJCOTVQ, seg_pos={0 0}, upload=84097]: Uploading piece... Jun 23 01:28:26.612001063 (105h5m2.175324064s) [noise=true , address=104.36.82.3:5797 , piece_id=CJD4BHNHEVAGUJ5K55QM7OLAQQLEYRD4C6KJQBZ4FKJLDREGLNRQ, seg_pos={0 0}, upload=84097]: Uploading piece... Jun 23 01:28:26.687672089 (105h5m2.250995077s) [upload=84097]: Next returned nil segment Jun 23 01:28:26.687687969 (105h5m2.25101097s) [upload=84097]: Waiting for error group managing segments... Jun 23 01:28:26.707815366 (105h5m2.271138367s) [noise=true , address=79.127.220.98:4410 , piece_id=DLW3G47XXWAJ4FZAZDEL6TCIN5FKG26LUNVELDWSUE2GDH6IPBLQ, seg_pos={0 0}, upload=84097]: Done uploading piece. err: ecclient: upload failed (node:1Lay6oZEhDEeDmmkisrLND1j82PurLHV95nGJoKkk5wKUaK5S7, address:79.127.220.98:4410): protocol: storage node overloaded, request limit: 500 Jun 23 01:28:26.735633260 (105h5m2.298956259s) [noise=true , address=104.36.82.21:6483 , piece_id=WTUXQB3XLSHBMZZQNK745AK5U3ZLLNKC3TLW4CEFA7OX4DJCOTVQ, seg_pos={0 0}, upload=84097]: Done uploading piece. err: ecclient: upload failed (node:12gxKuBALDLDVDfRGBvMZymwGyPfPtV1BsHdBPqZs34ugvvYacG, address:104.36.82.21:6483): protocol: storage node overloaded, request limit: 2000 Jun 23 01:28:26.793765419 (105h5m2.357088412s) [noise=true , address=156.146.43.225:4448 , piece_id=5SH6UKJRS56UOVGLLW7JKTSAQANRVHYVOMEXZ56UWYM3AEB7QGVA, seg_pos={0 0}, upload=84097]: Done uploading piece. err: ecclient: upload failed (node:1irai8UTYknGTUkKHtKbBNTJTYMQsNTFnzrTEaWCMRDq1NNpGh, address:156.146.43.225:4448): protocol: storage node overloaded, request limit: 2000 Jun 23 01:28:26.868408971 (105h5m2.43173196s) [noise=true , address=77.90.31.15:4729 , piece_id=AAVBJ7VEQX67HQBXQ5DO3O5XJ37TRKU3PS2XXEH3WARMSEAQGQ2A, seg_pos={0 0}, upload=84097]: Done uploading piece. err: <nil> Jun 23 01:28:26.989727871 (105h5m2.553050876s) [noise=true , address=45.91.116.8:5050 , piece_id=M4P3RKV2BUWYZOP2TOLNSMYOYDTOEG5NXXM6XZUYPLFQMLUGUIOA, seg_pos={0 0}, upload=84097]: Done uploading piece. err: <nil> Jun 23 01:28:27.669519830 (105h5m3.23284283s) [noise=true , address=77.90.31.14:4693 , piece_id=YGA5IDHRAXIFCELCNCQF23PEFGJMFQOQHFWY3D7UODW5HYEIEWXA, seg_pos={0 0}, upload=84097]: Done uploading piece. err: <nil> Jun 23 01:28:27.697322244 (105h5m3.260645244s) [noise=true , address=104.36.82.42:6830 , piece_id=KW3RNOZYENVRUR663PVSC4WFCLI3RG5YBITFXOYC6W2GTGEFXTVA, seg_pos={0 0}, upload=84097]: Done uploading piece. err: <nil> Jun 23 01:28:28.071845632 (105h5m3.635168633s) [noise=true , address=104.36.82.55:4217 , piece_id=WQIZ3WHXBLCKNJ4ZVQRC2PQIRH6IVVO7J3UMM5V5GXEMODV36KLQ, seg_pos={0 0}, upload=84097]: Done uploading piece. err: <nil> Jun 23 01:28:28.589172169 (105h5m4.152495155s) [noise=true , address=45.91.116.9:5288 , piece_id=EQWQVX4JDLVTNRTKYNUIFGJYQUXPVUCUCYJPNVW2IYM3CGQY5D6A, seg_pos={0 0}, upload=84097]: Done uploading piece. err: <nil> Jun 23 01:28:29.203146980 (105h5m4.76646998s) [noise=true , address=104.36.82.16:6310 , piece_id=CRHVWU4XYDGUG7XVGC3TI6DC2CLBTSTHND4YB2YFEFPBD6TOAXKQ, seg_pos={0 0}, upload=84097]: Done uploading piece. err: ecclient: upload failed (node:1g14evoaeUnhJThf5QKbQT1Gd8WW9MHvaYQhQ1GRgwPfDLoC5R, address:104.36.82.16:6310): protocol: storage node overloaded, request limit: 2000 Jun 23 01:28:29.434402213 (105h5m4.997725203s) [noise=true , address=45.91.116.10:5084 , piece_id=WCIMJN6XDXUO2VCRKLAVOPVVQUEQDGHR4NP4KQGY27NEUPCD5OFA, seg_pos={0 0}, upload=84097]: Done uploading piece. err: <nil> Jun 23 01:28:30.293775103 (105h5m5.857098094s) [noise=true , address=104.36.82.9:6021 , piece_id=4JQGGBGDZMQ6FDS35A7BZLB5OLZL2JHMD5Q3TXAETVX7TYA74EHQ, seg_pos={0 0}, upload=84097]: Done uploading piece. err: <nil> Jun 23 01:28:30.347258498 (105h5m5.910581502s) [noise=true , address=104.36.82.12:6174 , piece_id=VQG2SHTAARK4FOSQER2YRKHZV7ODG7ZDPHNF6AESZGE3GMITKZZA, seg_pos={0 0}, upload=84097]: Done uploading piece. err: <nil> Jun 23 01:28:30.547306548 (105h5m6.110629586s) [noise=true , address=79.127.219.51:4209 , piece_id=ZHJC5QUBGOHT3HRY4XPPADV7CYINQRXT4ID53KVC6LYXM7IV533A, seg_pos={0 0}, upload=84097]: Done uploading piece. err: <nil> Jun 23 01:28:34.333414200 (105h5m9.896737197s) [noise=true , address=79.127.219.38:4031 , piece_id=TBIY2ZMYVVKMYV4LVGJEKCJXMXYVJKSZKKBXASVNEUU2P7DKZKIQ, seg_pos={0 0}, upload=84097]: Done uploading piece. err: <nil> Jun 23 01:28:42.464111369 (105h5m18.02743436s) [noise=true , address=104.36.82.53:4142 , piece_id=CL2WQNZHZNZM5BFJQIBUBQ7FQCIJSK4AR7PR2OLBIXFULPMBTGIQ, seg_pos={0 0}, upload=84097]: Done uploading piece. err: <nil> Jun 23 01:28:49.581287518 (105h5m25.144610509s) [noise=true , address=104.36.82.3:5797 , piece_id=CJD4BHNHEVAGUJ5K55QM7OLAQQLEYRD4C6KJQBZ4FKJLDREGLNRQ, seg_pos={0 0}, upload=84097]: Done uploading piece. err: <nil> Jun 23 01:29:26.535692548 (105h6m2.099015547s) [noise=true , address=104.36.82.11:6131 , piece_id=BI4HK2NGEXXNEDYGRXRL77E5CONZG6JH4IWMONSB64DHOJLQQZPA, seg_pos={0 0}, upload=84097]: Done uploading piece. err: ecclient: upload failed (node:129jahwAmyPHqyYaXz7XTCK8FakBrYZw94pBKGhPzzX23yjM3RQ, address:104.36.82.11:6131): protocol: expected piece hash; context deadline exceeded; EOF Jun 23 01:29:26.535746358 (105h6m2.099069362s) [seg_pos={0 0}, upload=84097]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jun 23 01:29:26.562546209 (105h6m2.125869225s) [upload=84097]: Done uploading segments... Jun 23 01:29:26.562593265 (105h6m2.125916275s) [upload=84097]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration:* 19.211494327s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [.............X.....................X...................X....] duration: 30.018537453s FAIL (2025-06-23 01:34:35 UTC) operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.018563554s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [...........X.............................X...........X......] duration: 13.671516936s FAIL (2025-06-23 01:34:23 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== oad=84082]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jun 23 01:27:43.602110205 (105h4m19.165433214s) [upload=84082]: Done uploading segments... Jun 23 01:27:43.602149595 (105h4m19.165472598s) [upload=84082]: Upload finished. err: <nil> Jun 23 01:28:26.568078065 (105h5m2.131401068s) [upload=84097]: Starting upload Jun 23 01:28:26.568176623 (105h5m2.131499629s) [upload=84097]: Done starting upload Jun 23 01:28:26.568333698 (105h5m2.131656699s) [upload=84097]: Uploading segments... Jun 23 01:28:26.569025334 (105h5m2.132348341s) [upload=84097]: Got next segment. Inline: false Jun 23 01:28:26.569144232 (105h5m2.132467237s) [upload=84097]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 23 01:28:26.569226194 (105h5m2.132549206s) [upload=84097]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 23 01:28:26.603382637 (105h5m2.166705641s) [seg_pos={0 0}, upload=84097]: Begin upload segment... Jun 23 01:28:26.603439808 (105h5m2.16676282s) [seg_pos={0 0}, upload=84097]: Done begin upload segment. Jun 23 01:28:26.603481538 (105h5m2.166804559s) [noise=true , address=79.127.219.38:4031 , piece_id=TBIY2ZMYVVKMYV4LVGJEKCJXMXYVJKSZKKBXASVNEUU2P7DKZKIQ, seg_pos={0 0}, upload=84097]: Uploading piece... Jun 23 01:28:26.603498030 (105h5m2.166821033s) [noise=true , address=77.90.31.15:4729 , piece_id=AAVBJ7VEQX67HQBXQ5DO3O5XJ37TRKU3PS2XXEH3WARMSEAQGQ2A, seg_pos={0 0}, upload=84097]: Uploading piece... Jun 23 01:28:26.603958736 (105h5m2.16728174s) [noise=true , address=79.127.219.51:4209 , piece_id=ZHJC5QUBGOHT3HRY4XPPADV7CYINQRXT4ID53KVC6LYXM7IV533A, seg_pos={0 0}, upload=84097]: Uploading piece... Jun 23 01:28:26.603997417 (105h5m2.167320407s) [noise=true , address=79.127.220.98:4410 , piece_id=DLW3G47XXWAJ4FZAZDEL6TCIN5FKG26LUNVELDWSUE2GDH6IPBLQ, seg_pos={0 0}, upload=84097]: Uploading piece... Jun 23 01:28:26.604297025 (105h5m2.167620026s) [noise=true , address=45.91.116.10:5084 , piece_id=WCIMJN6XDXUO2VCRKLAVOPVVQUEQDGHR4NP4KQGY27NEUPCD5OFA, seg_pos={0 0}, upload=84097]: Uploading piece... Jun 23 01:28:26.604692251 (105h5m2.16801525s) [noise=true , address=104.36.82.55:4217 , piece_id=WQIZ3WHXBLCKNJ4ZVQRC2PQIRH6IVVO7J3UMM5V5GXEMODV36KLQ, seg_pos={0 0}, upload=84097]: Uploading piece... Jun 23 01:28:26.605044347 (105h5m2.168367349s) [noise=true , address=104.36.82.53:4142 , piece_id=CL2WQNZHZNZM5BFJQIBUBQ7FQCIJSK4AR7PR2OLBIXFULPMBTGIQ, seg_pos={0 0}, upload=84097]: Uploading piece... Jun 23 01:28:26.605369657 (105h5m2.168692667s) [noise=true , address=45.91.116.8:5050 , piece_id=M4P3RKV2BUWYZOP2TOLNSMYOYDTOEG5NXXM6XZUYPLFQMLUGUIOA, seg_pos={0 0}, upload=84097]: Uploading piece... Jun 23 01:28:26.605753081 (105h5m2.169076095s) [noise=true , address=77.90.31.14:4693 , piece_id=YGA5IDHRAXIFCELCNCQF23PEFGJMFQOQHFWY3D7UODW5HYEIEWXA, seg_pos={0 0}, upload=84097]: Uploading piece... Jun 23 01:28:26.606212150 (105h5m2.169535158s) [noise=true , address=104.36.82.9:6021 , piece_id=4JQGGBGDZMQ6FDS35A7BZLB5OLZL2JHMD5Q3TXAETVX7TYA74EHQ, seg_pos={0 0}, upload=84097]: Uploading piece... Jun 23 01:28:26.606654203 (105h5m2.169977203s) [noise=true , address=104.36.82.16:6310 , piece_id=CRHVWU4XYDGUG7XVGC3TI6DC2CLBTSTHND4YB2YFEFPBD6TOAXKQ, seg_pos={0 0}, upload=84097]: Uploading piece... Jun 23 01:28:26.606961844 (105h5m2.170284849s) [noise=true , address=156.146.43.225:4448 , piece_id=5SH6UKJRS56UOVGLLW7JKTSAQANRVHYVOMEXZ56UWYM3AEB7QGVA, seg_pos={0 0}, upload=84097]: Uploading piece... Jun 23 01:28:26.607374002 (105h5m2.170697008s) [noise=true , address=104.36.82.12:6174 , piece_id=VQG2SHTAARK4FOSQER2YRKHZV7ODG7ZDPHNF6AESZGE3GMITKZZA, seg_pos={0 0}, upload=84097]: Uploading piece... Jun 23 01:28:26.607734796 (105h5m2.171057797s) [noise=true , address=45.91.116.9:5288 , piece_id=EQWQVX4JDLVTNRTKYNUIFGJYQUXPVUCUCYJPNVW2IYM3CGQY5D6A, seg_pos={0 0}, upload=84097]: Uploading piece... Jun 23 01:28:26.608082032 (105h5m2.171405034s) [noise=true , address=104.36.82.11:6131 , piece_id=BI4HK2NGEXXNEDYGRXRL77E5CONZG6JH4IWMONSB64DHOJLQQZPA, seg_pos={0 0}, upload=84097]: Uploading piece... Jun 23 01:28:26.608428729 (105h5m2.171751737s) [noise=true , address=104.36.82.42:6830 , piece_id=KW3RNOZYENVRUR663PVSC4WFCLI3RG5YBITFXOYC6W2GTGEFXTVA, seg_pos={0 0}, upload=84097]: Uploading piece... Jun 23 01:28:26.610585268 (105h5m2.173908273s) [noise=true , address=104.36.82.21:6483 , piece_id=WTUXQB3XLSHBMZZQNK745AK5U3ZLLNKC3TLW4CEFA7OX4DJCOTVQ, seg_pos={0 0}, upload=84097]: Uploading piece... Jun 23 01:28:26.612001063 (105h5m2.175324064s) [noise=true , address=104.36.82.3:5797 , piece_id=CJD4BHNHEVAGUJ5K55QM7OLAQQLEYRD4C6KJQBZ4FKJLDREGLNRQ, seg_pos={0 0}, upload=84097]: Uploading piece... Jun 23 01:28:26.687672089 (105h5m2.250995077s) [upload=84097]: Next returned nil segment Jun 23 01:28:26.687687969 (105h5m2.25101097s) [upload=84097]: Waiting for error group managing segments... Jun 23 01:28:26.707815366 (105h5m2.271138367s) [noise=true , address=79.127.220.98:4410 , piece_id=DLW3G47XXWAJ4FZAZDEL6TCIN5FKG26LUNVELDWSUE2GDH6IPBLQ, seg_pos={0 0}, upload=84097]: Done uploading piece. err: ecclient: upload failed (node:1Lay6oZEhDEeDmmkisrLND1j82PurLHV95nGJoKkk5wKUaK5S7, address:79.127.220.98:4410): protocol: storage node overloaded, request limit: 500 Jun 23 01:28:26.735633260 (105h5m2.298956259s) [noise=true , address=104.36.82.21:6483 , piece_id=WTUXQB3XLSHBMZZQNK745AK5U3ZLLNKC3TLW4CEFA7OX4DJCOTVQ, seg_pos={0 0}, upload=84097]: Done uploading piece. err: ecclient: upload failed (node:12gxKuBALDLDVDfRGBvMZymwGyPfPtV1BsHdBPqZs34ugvvYacG, address:104.36.82.21:6483): protocol: storage node overloaded, request limit: 2000 Jun 23 01:28:26.793765419 (105h5m2.357088412s) [noise=true , address=156.146.43.225:4448 , piece_id=5SH6UKJRS56UOVGLLW7JKTSAQANRVHYVOMEXZ56UWYM3AEB7QGVA, seg_pos={0 0}, upload=84097]: Done uploading piece. err: ecclient: upload failed (node:1irai8UTYknGTUkKHtKbBNTJTYMQsNTFnzrTEaWCMRDq1NNpGh, address:156.146.43.225:4448): protocol: storage node overloaded, request limit: 2000 Jun 23 01:28:26.868408971 (105h5m2.43173196s) [noise=true , address=77.90.31.15:4729 , piece_id=AAVBJ7VEQX67HQBXQ5DO3O5XJ37TRKU3PS2XXEH3WARMSEAQGQ2A, seg_pos={0 0}, upload=84097]: Done uploading piece. err: <nil> Jun 23 01:28:26.989727871 (105h5m2.553050876s) [noise=true , address=45.91.116.8:5050 , piece_id=M4P3RKV2BUWYZOP2TOLNSMYOYDTOEG5NXXM6XZUYPLFQMLUGUIOA, seg_pos={0 0}, upload=84097]: Done uploading piece. err: <nil> Jun 23 01:28:27.669519830 (105h5m3.23284283s) [noise=true , address=77.90.31.14:4693 , piece_id=YGA5IDHRAXIFCELCNCQF23PEFGJMFQOQHFWY3D7UODW5HYEIEWXA, seg_pos={0 0}, upload=84097]: Done uploading piece. err: <nil> Jun 23 01:28:27.697322244 (105h5m3.260645244s) [noise=true , address=104.36.82.42:6830 , piece_id=KW3RNOZYENVRUR663PVSC4WFCLI3RG5YBITFXOYC6W2GTGEFXTVA, seg_pos={0 0}, upload=84097]: Done uploading piece. err: <nil> Jun 23 01:28:28.071845632 (105h5m3.635168633s) [noise=true , address=104.36.82.55:4217 , piece_id=WQIZ3WHXBLCKNJ4ZVQRC2PQIRH6IVVO7J3UMM5V5GXEMODV36KLQ, seg_pos={0 0}, upload=84097]: Done uploading piece. err: <nil> Jun 23 01:28:28.589172169 (105h5m4.152495155s) [noise=true , address=45.91.116.9:5288 , piece_id=EQWQVX4JDLVTNRTKYNUIFGJYQUXPVUCUCYJPNVW2IYM3CGQY5D6A, seg_pos={0 0}, upload=84097]: Done uploading piece. err: <nil> Jun 23 01:28:29.203146980 (105h5m4.76646998s) [noise=true , address=104.36.82.16:6310 , piece_id=CRHVWU4XYDGUG7XVGC3TI6DC2CLBTSTHND4YB2YFEFPBD6TOAXKQ, seg_pos={0 0}, upload=84097]: Done uploading piece. err: ecclient: upload failed (node:1g14evoaeUnhJThf5QKbQT1Gd8WW9MHvaYQhQ1GRgwPfDLoC5R, address:104.36.82.16:6310): protocol: storage node overloaded, request limit: 2000 Jun 23 01:28:29.434402213 (105h5m4.997725203s) [noise=true , address=45.91.116.10:5084 , piece_id=WCIMJN6XDXUO2VCRKLAVOPVVQUEQDGHR4NP4KQGY27NEUPCD5OFA, seg_pos={0 0}, upload=84097]: Done uploading piece. err: <nil> Jun 23 01:28:30.293775103 (105h5m5.857098094s) [noise=true , address=104.36.82.9:6021 , piece_id=4JQGGBGDZMQ6FDS35A7BZLB5OLZL2JHMD5Q3TXAETVX7TYA74EHQ, seg_pos={0 0}, upload=84097]: Done uploading piece. err: <nil> Jun 23 01:28:30.347258498 (105h5m5.910581502s) [noise=true , address=104.36.82.12:6174 , piece_id=VQG2SHTAARK4FOSQER2YRKHZV7ODG7ZDPHNF6AESZGE3GMITKZZA, seg_pos={0 0}, upload=84097]: Done uploading piece. err: <nil> Jun 23 01:28:30.547306548 (105h5m6.110629586s) [noise=true , address=79.127.219.51:4209 , piece_id=ZHJC5QUBGOHT3HRY4XPPADV7CYINQRXT4ID53KVC6LYXM7IV533A, seg_pos={0 0}, upload=84097]: Done uploading piece. err: <nil> Jun 23 01:28:34.333414200 (105h5m9.896737197s) [noise=true , address=79.127.219.38:4031 , piece_id=TBIY2ZMYVVKMYV4LVGJEKCJXMXYVJKSZKKBXASVNEUU2P7DKZKIQ, seg_pos={0 0}, upload=84097]: Done uploading piece. err: <nil> Jun 23 01:28:42.464111369 (105h5m18.02743436s) [noise=true , address=104.36.82.53:4142 , piece_id=CL2WQNZHZNZM5BFJQIBUBQ7FQCIJSK4AR7PR2OLBIXFULPMBTGIQ, seg_pos={0 0}, upload=84097]: Done uploading piece. err: <nil> Jun 23 01:28:49.581287518 (105h5m25.144610509s) [noise=true , address=104.36.82.3:5797 , piece_id=CJD4BHNHEVAGUJ5K55QM7OLAQQLEYRD4C6KJQBZ4FKJLDREGLNRQ, seg_pos={0 0}, upload=84097]: Done uploading piece. err: <nil> Jun 23 01:29:26.535692548 (105h6m2.099015547s) [noise=true , address=104.36.82.11:6131 , piece_id=BI4HK2NGEXXNEDYGRXRL77E5CONZG6JH4IWMONSB64DHOJLQQZPA, seg_pos={0 0}, upload=84097]: Done uploading piece. err: ecclient: upload failed (node:129jahwAmyPHqyYaXz7XTCK8FakBrYZw94pBKGhPzzX23yjM3RQ, address:104.36.82.11:6131): protocol: expected piece hash; context deadline exceeded; EOF Jun 23 01:29:26.535746358 (105h6m2.099069362s) [seg_pos={0 0}, upload=84097]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jun 23 01:29:26.562546209 (105h6m2.125869225s) [upload=84097]: Done uploading segments... Jun 23 01:29:26.562593265 (105h6m2.125916275s) [upload=84097]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration:* 13.671548225s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [..............X.....................X...................X...] duration: 17.269750058s FAIL (2025-06-23 01:33:49 UTC) operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 17.269773533s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [............X.............................X...........X.....] duration: 12.756946067s FAIL (2025-06-23 01:33:37 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== oad=84082]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jun 23 01:27:43.602110205 (105h4m19.165433214s) [upload=84082]: Done uploading segments... Jun 23 01:27:43.602149595 (105h4m19.165472598s) [upload=84082]: Upload finished. err: <nil> Jun 23 01:28:26.568078065 (105h5m2.131401068s) [upload=84097]: Starting upload Jun 23 01:28:26.568176623 (105h5m2.131499629s) [upload=84097]: Done starting upload Jun 23 01:28:26.568333698 (105h5m2.131656699s) [upload=84097]: Uploading segments... Jun 23 01:28:26.569025334 (105h5m2.132348341s) [upload=84097]: Got next segment. Inline: false Jun 23 01:28:26.569144232 (105h5m2.132467237s) [upload=84097]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 23 01:28:26.569226194 (105h5m2.132549206s) [upload=84097]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 23 01:28:26.603382637 (105h5m2.166705641s) [seg_pos={0 0}, upload=84097]: Begin upload segment... Jun 23 01:28:26.603439808 (105h5m2.16676282s) [seg_pos={0 0}, upload=84097]: Done begin upload segment. Jun 23 01:28:26.603481538 (105h5m2.166804559s) [noise=true , address=79.127.219.38:4031 , piece_id=TBIY2ZMYVVKMYV4LVGJEKCJXMXYVJKSZKKBXASVNEUU2P7DKZKIQ, seg_pos={0 0}, upload=84097]: Uploading piece... Jun 23 01:28:26.603498030 (105h5m2.166821033s) [noise=true , address=77.90.31.15:4729 , piece_id=AAVBJ7VEQX67HQBXQ5DO3O5XJ37TRKU3PS2XXEH3WARMSEAQGQ2A, seg_pos={0 0}, upload=84097]: Uploading piece... Jun 23 01:28:26.603958736 (105h5m2.16728174s) [noise=true , address=79.127.219.51:4209 , piece_id=ZHJC5QUBGOHT3HRY4XPPADV7CYINQRXT4ID53KVC6LYXM7IV533A, seg_pos={0 0}, upload=84097]: Uploading piece... Jun 23 01:28:26.603997417 (105h5m2.167320407s) [noise=true , address=79.127.220.98:4410 , piece_id=DLW3G47XXWAJ4FZAZDEL6TCIN5FKG26LUNVELDWSUE2GDH6IPBLQ, seg_pos={0 0}, upload=84097]: Uploading piece... Jun 23 01:28:26.604297025 (105h5m2.167620026s) [noise=true , address=45.91.116.10:5084 , piece_id=WCIMJN6XDXUO2VCRKLAVOPVVQUEQDGHR4NP4KQGY27NEUPCD5OFA, seg_pos={0 0}, upload=84097]: Uploading piece... Jun 23 01:28:26.604692251 (105h5m2.16801525s) [noise=true , address=104.36.82.55:4217 , piece_id=WQIZ3WHXBLCKNJ4ZVQRC2PQIRH6IVVO7J3UMM5V5GXEMODV36KLQ, seg_pos={0 0}, upload=84097]: Uploading piece... Jun 23 01:28:26.605044347 (105h5m2.168367349s) [noise=true , address=104.36.82.53:4142 , piece_id=CL2WQNZHZNZM5BFJQIBUBQ7FQCIJSK4AR7PR2OLBIXFULPMBTGIQ, seg_pos={0 0}, upload=84097]: Uploading piece... Jun 23 01:28:26.605369657 (105h5m2.168692667s) [noise=true , address=45.91.116.8:5050 , piece_id=M4P3RKV2BUWYZOP2TOLNSMYOYDTOEG5NXXM6XZUYPLFQMLUGUIOA, seg_pos={0 0}, upload=84097]: Uploading piece... Jun 23 01:28:26.605753081 (105h5m2.169076095s) [noise=true , address=77.90.31.14:4693 , piece_id=YGA5IDHRAXIFCELCNCQF23PEFGJMFQOQHFWY3D7UODW5HYEIEWXA, seg_pos={0 0}, upload=84097]: Uploading piece... Jun 23 01:28:26.606212150 (105h5m2.169535158s) [noise=true , address=104.36.82.9:6021 , piece_id=4JQGGBGDZMQ6FDS35A7BZLB5OLZL2JHMD5Q3TXAETVX7TYA74EHQ, seg_pos={0 0}, upload=84097]: Uploading piece... Jun 23 01:28:26.606654203 (105h5m2.169977203s) [noise=true , address=104.36.82.16:6310 , piece_id=CRHVWU4XYDGUG7XVGC3TI6DC2CLBTSTHND4YB2YFEFPBD6TOAXKQ, seg_pos={0 0}, upload=84097]: Uploading piece... Jun 23 01:28:26.606961844 (105h5m2.170284849s) [noise=true , address=156.146.43.225:4448 , piece_id=5SH6UKJRS56UOVGLLW7JKTSAQANRVHYVOMEXZ56UWYM3AEB7QGVA, seg_pos={0 0}, upload=84097]: Uploading piece... Jun 23 01:28:26.607374002 (105h5m2.170697008s) [noise=true , address=104.36.82.12:6174 , piece_id=VQG2SHTAARK4FOSQER2YRKHZV7ODG7ZDPHNF6AESZGE3GMITKZZA, seg_pos={0 0}, upload=84097]: Uploading piece... Jun 23 01:28:26.607734796 (105h5m2.171057797s) [noise=true , address=45.91.116.9:5288 , piece_id=EQWQVX4JDLVTNRTKYNUIFGJYQUXPVUCUCYJPNVW2IYM3CGQY5D6A, seg_pos={0 0}, upload=84097]: Uploading piece... Jun 23 01:28:26.608082032 (105h5m2.171405034s) [noise=true , address=104.36.82.11:6131 , piece_id=BI4HK2NGEXXNEDYGRXRL77E5CONZG6JH4IWMONSB64DHOJLQQZPA, seg_pos={0 0}, upload=84097]: Uploading piece... Jun 23 01:28:26.608428729 (105h5m2.171751737s) [noise=true , address=104.36.82.42:6830 , piece_id=KW3RNOZYENVRUR663PVSC4WFCLI3RG5YBITFXOYC6W2GTGEFXTVA, seg_pos={0 0}, upload=84097]: Uploading piece... Jun 23 01:28:26.610585268 (105h5m2.173908273s) [noise=true , address=104.36.82.21:6483 , piece_id=WTUXQB3XLSHBMZZQNK745AK5U3ZLLNKC3TLW4CEFA7OX4DJCOTVQ, seg_pos={0 0}, upload=84097]: Uploading piece... Jun 23 01:28:26.612001063 (105h5m2.175324064s) [noise=true , address=104.36.82.3:5797 , piece_id=CJD4BHNHEVAGUJ5K55QM7OLAQQLEYRD4C6KJQBZ4FKJLDREGLNRQ, seg_pos={0 0}, upload=84097]: Uploading piece... Jun 23 01:28:26.687672089 (105h5m2.250995077s) [upload=84097]: Next returned nil segment Jun 23 01:28:26.687687969 (105h5m2.25101097s) [upload=84097]: Waiting for error group managing segments... Jun 23 01:28:26.707815366 (105h5m2.271138367s) [noise=true , address=79.127.220.98:4410 , piece_id=DLW3G47XXWAJ4FZAZDEL6TCIN5FKG26LUNVELDWSUE2GDH6IPBLQ, seg_pos={0 0}, upload=84097]: Done uploading piece. err: ecclient: upload failed (node:1Lay6oZEhDEeDmmkisrLND1j82PurLHV95nGJoKkk5wKUaK5S7, address:79.127.220.98:4410): protocol: storage node overloaded, request limit: 500 Jun 23 01:28:26.735633260 (105h5m2.298956259s) [noise=true , address=104.36.82.21:6483 , piece_id=WTUXQB3XLSHBMZZQNK745AK5U3ZLLNKC3TLW4CEFA7OX4DJCOTVQ, seg_pos={0 0}, upload=84097]: Done uploading piece. err: ecclient: upload failed (node:12gxKuBALDLDVDfRGBvMZymwGyPfPtV1BsHdBPqZs34ugvvYacG, address:104.36.82.21:6483): protocol: storage node overloaded, request limit: 2000 Jun 23 01:28:26.793765419 (105h5m2.357088412s) [noise=true , address=156.146.43.225:4448 , piece_id=5SH6UKJRS56UOVGLLW7JKTSAQANRVHYVOMEXZ56UWYM3AEB7QGVA, seg_pos={0 0}, upload=84097]: Done uploading piece. err: ecclient: upload failed (node:1irai8UTYknGTUkKHtKbBNTJTYMQsNTFnzrTEaWCMRDq1NNpGh, address:156.146.43.225:4448): protocol: storage node overloaded, request limit: 2000 Jun 23 01:28:26.868408971 (105h5m2.43173196s) [noise=true , address=77.90.31.15:4729 , piece_id=AAVBJ7VEQX67HQBXQ5DO3O5XJ37TRKU3PS2XXEH3WARMSEAQGQ2A, seg_pos={0 0}, upload=84097]: Done uploading piece. err: <nil> Jun 23 01:28:26.989727871 (105h5m2.553050876s) [noise=true , address=45.91.116.8:5050 , piece_id=M4P3RKV2BUWYZOP2TOLNSMYOYDTOEG5NXXM6XZUYPLFQMLUGUIOA, seg_pos={0 0}, upload=84097]: Done uploading piece. err: <nil> Jun 23 01:28:27.669519830 (105h5m3.23284283s) [noise=true , address=77.90.31.14:4693 , piece_id=YGA5IDHRAXIFCELCNCQF23PEFGJMFQOQHFWY3D7UODW5HYEIEWXA, seg_pos={0 0}, upload=84097]: Done uploading piece. err: <nil> Jun 23 01:28:27.697322244 (105h5m3.260645244s) [noise=true , address=104.36.82.42:6830 , piece_id=KW3RNOZYENVRUR663PVSC4WFCLI3RG5YBITFXOYC6W2GTGEFXTVA, seg_pos={0 0}, upload=84097]: Done uploading piece. err: <nil> Jun 23 01:28:28.071845632 (105h5m3.635168633s) [noise=true , address=104.36.82.55:4217 , piece_id=WQIZ3WHXBLCKNJ4ZVQRC2PQIRH6IVVO7J3UMM5V5GXEMODV36KLQ, seg_pos={0 0}, upload=84097]: Done uploading piece. err: <nil> Jun 23 01:28:28.589172169 (105h5m4.152495155s) [noise=true , address=45.91.116.9:5288 , piece_id=EQWQVX4JDLVTNRTKYNUIFGJYQUXPVUCUCYJPNVW2IYM3CGQY5D6A, seg_pos={0 0}, upload=84097]: Done uploading piece. err: <nil> Jun 23 01:28:29.203146980 (105h5m4.76646998s) [noise=true , address=104.36.82.16:6310 , piece_id=CRHVWU4XYDGUG7XVGC3TI6DC2CLBTSTHND4YB2YFEFPBD6TOAXKQ, seg_pos={0 0}, upload=84097]: Done uploading piece. err: ecclient: upload failed (node:1g14evoaeUnhJThf5QKbQT1Gd8WW9MHvaYQhQ1GRgwPfDLoC5R, address:104.36.82.16:6310): protocol: storage node overloaded, request limit: 2000 Jun 23 01:28:29.434402213 (105h5m4.997725203s) [noise=true , address=45.91.116.10:5084 , piece_id=WCIMJN6XDXUO2VCRKLAVOPVVQUEQDGHR4NP4KQGY27NEUPCD5OFA, seg_pos={0 0}, upload=84097]: Done uploading piece. err: <nil> Jun 23 01:28:30.293775103 (105h5m5.857098094s) [noise=true , address=104.36.82.9:6021 , piece_id=4JQGGBGDZMQ6FDS35A7BZLB5OLZL2JHMD5Q3TXAETVX7TYA74EHQ, seg_pos={0 0}, upload=84097]: Done uploading piece. err: <nil> Jun 23 01:28:30.347258498 (105h5m5.910581502s) [noise=true , address=104.36.82.12:6174 , piece_id=VQG2SHTAARK4FOSQER2YRKHZV7ODG7ZDPHNF6AESZGE3GMITKZZA, seg_pos={0 0}, upload=84097]: Done uploading piece. err: <nil> Jun 23 01:28:30.547306548 (105h5m6.110629586s) [noise=true , address=79.127.219.51:4209 , piece_id=ZHJC5QUBGOHT3HRY4XPPADV7CYINQRXT4ID53KVC6LYXM7IV533A, seg_pos={0 0}, upload=84097]: Done uploading piece. err: <nil> Jun 23 01:28:34.333414200 (105h5m9.896737197s) [noise=true , address=79.127.219.38:4031 , piece_id=TBIY2ZMYVVKMYV4LVGJEKCJXMXYVJKSZKKBXASVNEUU2P7DKZKIQ, seg_pos={0 0}, upload=84097]: Done uploading piece. err: <nil> Jun 23 01:28:42.464111369 (105h5m18.02743436s) [noise=true , address=104.36.82.53:4142 , piece_id=CL2WQNZHZNZM5BFJQIBUBQ7FQCIJSK4AR7PR2OLBIXFULPMBTGIQ, seg_pos={0 0}, upload=84097]: Done uploading piece. err: <nil> Jun 23 01:28:49.581287518 (105h5m25.144610509s) [noise=true , address=104.36.82.3:5797 , piece_id=CJD4BHNHEVAGUJ5K55QM7OLAQQLEYRD4C6KJQBZ4FKJLDREGLNRQ, seg_pos={0 0}, upload=84097]: Done uploading piece. err: <nil> Jun 23 01:29:26.535692548 (105h6m2.099015547s) [noise=true , address=104.36.82.11:6131 , piece_id=BI4HK2NGEXXNEDYGRXRL77E5CONZG6JH4IWMONSB64DHOJLQQZPA, seg_pos={0 0}, upload=84097]: Done uploading piece. err: ecclient: upload failed (node:129jahwAmyPHqyYaXz7XTCK8FakBrYZw94pBKGhPzzX23yjM3RQ, address:104.36.82.11:6131): protocol: expected piece hash; context deadline exceeded; EOF Jun 23 01:29:26.535746358 (105h6m2.099069362s) [seg_pos={0 0}, upload=84097]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jun 23 01:29:26.562546209 (105h6m2.125869225s) [upload=84097]: Done uploading segments... Jun 23 01:29:26.562593265 (105h6m2.125916275s) [upload=84097]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration:* 12.756979465s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [...............X.....................X...................X..] duration: 8.749351797s FAIL (2025-06-23 01:32:53 UTC) operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 8.749390009s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [.............X.............................X...........X....] duration: 28.378888347s FAIL (2025-06-23 01:32:44 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== oad=84082]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jun 23 01:27:43.602110205 (105h4m19.165433214s) [upload=84082]: Done uploading segments... Jun 23 01:27:43.602149595 (105h4m19.165472598s) [upload=84082]: Upload finished. err: <nil> Jun 23 01:28:26.568078065 (105h5m2.131401068s) [upload=84097]: Starting upload Jun 23 01:28:26.568176623 (105h5m2.131499629s) [upload=84097]: Done starting upload Jun 23 01:28:26.568333698 (105h5m2.131656699s) [upload=84097]: Uploading segments... Jun 23 01:28:26.569025334 (105h5m2.132348341s) [upload=84097]: Got next segment. Inline: false Jun 23 01:28:26.569144232 (105h5m2.132467237s) [upload=84097]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 23 01:28:26.569226194 (105h5m2.132549206s) [upload=84097]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 23 01:28:26.603382637 (105h5m2.166705641s) [seg_pos={0 0}, upload=84097]: Begin upload segment... Jun 23 01:28:26.603439808 (105h5m2.16676282s) [seg_pos={0 0}, upload=84097]: Done begin upload segment. Jun 23 01:28:26.603481538 (105h5m2.166804559s) [noise=true , address=79.127.219.38:4031 , piece_id=TBIY2ZMYVVKMYV4LVGJEKCJXMXYVJKSZKKBXASVNEUU2P7DKZKIQ, seg_pos={0 0}, upload=84097]: Uploading piece... Jun 23 01:28:26.603498030 (105h5m2.166821033s) [noise=true , address=77.90.31.15:4729 , piece_id=AAVBJ7VEQX67HQBXQ5DO3O5XJ37TRKU3PS2XXEH3WARMSEAQGQ2A, seg_pos={0 0}, upload=84097]: Uploading piece... Jun 23 01:28:26.603958736 (105h5m2.16728174s) [noise=true , address=79.127.219.51:4209 , piece_id=ZHJC5QUBGOHT3HRY4XPPADV7CYINQRXT4ID53KVC6LYXM7IV533A, seg_pos={0 0}, upload=84097]: Uploading piece... Jun 23 01:28:26.603997417 (105h5m2.167320407s) [noise=true , address=79.127.220.98:4410 , piece_id=DLW3G47XXWAJ4FZAZDEL6TCIN5FKG26LUNVELDWSUE2GDH6IPBLQ, seg_pos={0 0}, upload=84097]: Uploading piece... Jun 23 01:28:26.604297025 (105h5m2.167620026s) [noise=true , address=45.91.116.10:5084 , piece_id=WCIMJN6XDXUO2VCRKLAVOPVVQUEQDGHR4NP4KQGY27NEUPCD5OFA, seg_pos={0 0}, upload=84097]: Uploading piece... Jun 23 01:28:26.604692251 (105h5m2.16801525s) [noise=true , address=104.36.82.55:4217 , piece_id=WQIZ3WHXBLCKNJ4ZVQRC2PQIRH6IVVO7J3UMM5V5GXEMODV36KLQ, seg_pos={0 0}, upload=84097]: Uploading piece... Jun 23 01:28:26.605044347 (105h5m2.168367349s) [noise=true , address=104.36.82.53:4142 , piece_id=CL2WQNZHZNZM5BFJQIBUBQ7FQCIJSK4AR7PR2OLBIXFULPMBTGIQ, seg_pos={0 0}, upload=84097]: Uploading piece... Jun 23 01:28:26.605369657 (105h5m2.168692667s) [noise=true , address=45.91.116.8:5050 , piece_id=M4P3RKV2BUWYZOP2TOLNSMYOYDTOEG5NXXM6XZUYPLFQMLUGUIOA, seg_pos={0 0}, upload=84097]: Uploading piece... Jun 23 01:28:26.605753081 (105h5m2.169076095s) [noise=true , address=77.90.31.14:4693 , piece_id=YGA5IDHRAXIFCELCNCQF23PEFGJMFQOQHFWY3D7UODW5HYEIEWXA, seg_pos={0 0}, upload=84097]: Uploading piece... Jun 23 01:28:26.606212150 (105h5m2.169535158s) [noise=true , address=104.36.82.9:6021 , piece_id=4JQGGBGDZMQ6FDS35A7BZLB5OLZL2JHMD5Q3TXAETVX7TYA74EHQ, seg_pos={0 0}, upload=84097]: Uploading piece... Jun 23 01:28:26.606654203 (105h5m2.169977203s) [noise=true , address=104.36.82.16:6310 , piece_id=CRHVWU4XYDGUG7XVGC3TI6DC2CLBTSTHND4YB2YFEFPBD6TOAXKQ, seg_pos={0 0}, upload=84097]: Uploading piece... Jun 23 01:28:26.606961844 (105h5m2.170284849s) [noise=true , address=156.146.43.225:4448 , piece_id=5SH6UKJRS56UOVGLLW7JKTSAQANRVHYVOMEXZ56UWYM3AEB7QGVA, seg_pos={0 0}, upload=84097]: Uploading piece... Jun 23 01:28:26.607374002 (105h5m2.170697008s) [noise=true , address=104.36.82.12:6174 , piece_id=VQG2SHTAARK4FOSQER2YRKHZV7ODG7ZDPHNF6AESZGE3GMITKZZA, seg_pos={0 0}, upload=84097]: Uploading piece... Jun 23 01:28:26.607734796 (105h5m2.171057797s) [noise=true , address=45.91.116.9:5288 , piece_id=EQWQVX4JDLVTNRTKYNUIFGJYQUXPVUCUCYJPNVW2IYM3CGQY5D6A, seg_pos={0 0}, upload=84097]: Uploading piece... Jun 23 01:28:26.608082032 (105h5m2.171405034s) [noise=true , address=104.36.82.11:6131 , piece_id=BI4HK2NGEXXNEDYGRXRL77E5CONZG6JH4IWMONSB64DHOJLQQZPA, seg_pos={0 0}, upload=84097]: Uploading piece... Jun 23 01:28:26.608428729 (105h5m2.171751737s) [noise=true , address=104.36.82.42:6830 , piece_id=KW3RNOZYENVRUR663PVSC4WFCLI3RG5YBITFXOYC6W2GTGEFXTVA, seg_pos={0 0}, upload=84097]: Uploading piece... Jun 23 01:28:26.610585268 (105h5m2.173908273s) [noise=true , address=104.36.82.21:6483 , piece_id=WTUXQB3XLSHBMZZQNK745AK5U3ZLLNKC3TLW4CEFA7OX4DJCOTVQ, seg_pos={0 0}, upload=84097]: Uploading piece... Jun 23 01:28:26.612001063 (105h5m2.175324064s) [noise=true , address=104.36.82.3:5797 , piece_id=CJD4BHNHEVAGUJ5K55QM7OLAQQLEYRD4C6KJQBZ4FKJLDREGLNRQ, seg_pos={0 0}, upload=84097]: Uploading piece... Jun 23 01:28:26.687672089 (105h5m2.250995077s) [upload=84097]: Next returned nil segment Jun 23 01:28:26.687687969 (105h5m2.25101097s) [upload=84097]: Waiting for error group managing segments... Jun 23 01:28:26.707815366 (105h5m2.271138367s) [noise=true , address=79.127.220.98:4410 , piece_id=DLW3G47XXWAJ4FZAZDEL6TCIN5FKG26LUNVELDWSUE2GDH6IPBLQ, seg_pos={0 0}, upload=84097]: Done uploading piece. err: ecclient: upload failed (node:1Lay6oZEhDEeDmmkisrLND1j82PurLHV95nGJoKkk5wKUaK5S7, address:79.127.220.98:4410): protocol: storage node overloaded, request limit: 500 Jun 23 01:28:26.735633260 (105h5m2.298956259s) [noise=true , address=104.36.82.21:6483 , piece_id=WTUXQB3XLSHBMZZQNK745AK5U3ZLLNKC3TLW4CEFA7OX4DJCOTVQ, seg_pos={0 0}, upload=84097]: Done uploading piece. err: ecclient: upload failed (node:12gxKuBALDLDVDfRGBvMZymwGyPfPtV1BsHdBPqZs34ugvvYacG, address:104.36.82.21:6483): protocol: storage node overloaded, request limit: 2000 Jun 23 01:28:26.793765419 (105h5m2.357088412s) [noise=true , address=156.146.43.225:4448 , piece_id=5SH6UKJRS56UOVGLLW7JKTSAQANRVHYVOMEXZ56UWYM3AEB7QGVA, seg_pos={0 0}, upload=84097]: Done uploading piece. err: ecclient: upload failed (node:1irai8UTYknGTUkKHtKbBNTJTYMQsNTFnzrTEaWCMRDq1NNpGh, address:156.146.43.225:4448): protocol: storage node overloaded, request limit: 2000 Jun 23 01:28:26.868408971 (105h5m2.43173196s) [noise=true , address=77.90.31.15:4729 , piece_id=AAVBJ7VEQX67HQBXQ5DO3O5XJ37TRKU3PS2XXEH3WARMSEAQGQ2A, seg_pos={0 0}, upload=84097]: Done uploading piece. err: <nil> Jun 23 01:28:26.989727871 (105h5m2.553050876s) [noise=true , address=45.91.116.8:5050 , piece_id=M4P3RKV2BUWYZOP2TOLNSMYOYDTOEG5NXXM6XZUYPLFQMLUGUIOA, seg_pos={0 0}, upload=84097]: Done uploading piece. err: <nil> Jun 23 01:28:27.669519830 (105h5m3.23284283s) [noise=true , address=77.90.31.14:4693 , piece_id=YGA5IDHRAXIFCELCNCQF23PEFGJMFQOQHFWY3D7UODW5HYEIEWXA, seg_pos={0 0}, upload=84097]: Done uploading piece. err: <nil> Jun 23 01:28:27.697322244 (105h5m3.260645244s) [noise=true , address=104.36.82.42:6830 , piece_id=KW3RNOZYENVRUR663PVSC4WFCLI3RG5YBITFXOYC6W2GTGEFXTVA, seg_pos={0 0}, upload=84097]: Done uploading piece. err: <nil> Jun 23 01:28:28.071845632 (105h5m3.635168633s) [noise=true , address=104.36.82.55:4217 , piece_id=WQIZ3WHXBLCKNJ4ZVQRC2PQIRH6IVVO7J3UMM5V5GXEMODV36KLQ, seg_pos={0 0}, upload=84097]: Done uploading piece. err: <nil> Jun 23 01:28:28.589172169 (105h5m4.152495155s) [noise=true , address=45.91.116.9:5288 , piece_id=EQWQVX4JDLVTNRTKYNUIFGJYQUXPVUCUCYJPNVW2IYM3CGQY5D6A, seg_pos={0 0}, upload=84097]: Done uploading piece. err: <nil> Jun 23 01:28:29.203146980 (105h5m4.76646998s) [noise=true , address=104.36.82.16:6310 , piece_id=CRHVWU4XYDGUG7XVGC3TI6DC2CLBTSTHND4YB2YFEFPBD6TOAXKQ, seg_pos={0 0}, upload=84097]: Done uploading piece. err: ecclient: upload failed (node:1g14evoaeUnhJThf5QKbQT1Gd8WW9MHvaYQhQ1GRgwPfDLoC5R, address:104.36.82.16:6310): protocol: storage node overloaded, request limit: 2000 Jun 23 01:28:29.434402213 (105h5m4.997725203s) [noise=true , address=45.91.116.10:5084 , piece_id=WCIMJN6XDXUO2VCRKLAVOPVVQUEQDGHR4NP4KQGY27NEUPCD5OFA, seg_pos={0 0}, upload=84097]: Done uploading piece. err: <nil> Jun 23 01:28:30.293775103 (105h5m5.857098094s) [noise=true , address=104.36.82.9:6021 , piece_id=4JQGGBGDZMQ6FDS35A7BZLB5OLZL2JHMD5Q3TXAETVX7TYA74EHQ, seg_pos={0 0}, upload=84097]: Done uploading piece. err: <nil> Jun 23 01:28:30.347258498 (105h5m5.910581502s) [noise=true , address=104.36.82.12:6174 , piece_id=VQG2SHTAARK4FOSQER2YRKHZV7ODG7ZDPHNF6AESZGE3GMITKZZA, seg_pos={0 0}, upload=84097]: Done uploading piece. err: <nil> Jun 23 01:28:30.547306548 (105h5m6.110629586s) [noise=true , address=79.127.219.51:4209 , piece_id=ZHJC5QUBGOHT3HRY4XPPADV7CYINQRXT4ID53KVC6LYXM7IV533A, seg_pos={0 0}, upload=84097]: Done uploading piece. err: <nil> Jun 23 01:28:34.333414200 (105h5m9.896737197s) [noise=true , address=79.127.219.38:4031 , piece_id=TBIY2ZMYVVKMYV4LVGJEKCJXMXYVJKSZKKBXASVNEUU2P7DKZKIQ, seg_pos={0 0}, upload=84097]: Done uploading piece. err: <nil> Jun 23 01:28:42.464111369 (105h5m18.02743436s) [noise=true , address=104.36.82.53:4142 , piece_id=CL2WQNZHZNZM5BFJQIBUBQ7FQCIJSK4AR7PR2OLBIXFULPMBTGIQ, seg_pos={0 0}, upload=84097]: Done uploading piece. err: <nil> Jun 23 01:28:49.581287518 (105h5m25.144610509s) [noise=true , address=104.36.82.3:5797 , piece_id=CJD4BHNHEVAGUJ5K55QM7OLAQQLEYRD4C6KJQBZ4FKJLDREGLNRQ, seg_pos={0 0}, upload=84097]: Done uploading piece. err: <nil> Jun 23 01:29:26.535692548 (105h6m2.099015547s) [noise=true , address=104.36.82.11:6131 , piece_id=BI4HK2NGEXXNEDYGRXRL77E5CONZG6JH4IWMONSB64DHOJLQQZPA, seg_pos={0 0}, upload=84097]: Done uploading piece. err: ecclient: upload failed (node:129jahwAmyPHqyYaXz7XTCK8FakBrYZw94pBKGhPzzX23yjM3RQ, address:104.36.82.11:6131): protocol: expected piece hash; context deadline exceeded; EOF Jun 23 01:29:26.535746358 (105h6m2.099069362s) [seg_pos={0 0}, upload=84097]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jun 23 01:29:26.562546209 (105h6m2.125869225s) [upload=84097]: Done uploading segments... Jun 23 01:29:26.562593265 (105h6m2.125916275s) [upload=84097]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration:* 28.37900043s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [................X.....................X...................X.] duration: 33.599857424s FAIL (2025-06-23 01:32:13 UTC) operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 33.599875748s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [..............X.............................X...........X...] duration: 35.233463837s FAIL (2025-06-23 01:31:50 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== oad=84082]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jun 23 01:27:43.602110205 (105h4m19.165433214s) [upload=84082]: Done uploading segments... Jun 23 01:27:43.602149595 (105h4m19.165472598s) [upload=84082]: Upload finished. err: <nil> Jun 23 01:28:26.568078065 (105h5m2.131401068s) [upload=84097]: Starting upload Jun 23 01:28:26.568176623 (105h5m2.131499629s) [upload=84097]: Done starting upload Jun 23 01:28:26.568333698 (105h5m2.131656699s) [upload=84097]: Uploading segments... Jun 23 01:28:26.569025334 (105h5m2.132348341s) [upload=84097]: Got next segment. Inline: false Jun 23 01:28:26.569144232 (105h5m2.132467237s) [upload=84097]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 23 01:28:26.569226194 (105h5m2.132549206s) [upload=84097]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 23 01:28:26.603382637 (105h5m2.166705641s) [seg_pos={0 0}, upload=84097]: Begin upload segment... Jun 23 01:28:26.603439808 (105h5m2.16676282s) [seg_pos={0 0}, upload=84097]: Done begin upload segment. Jun 23 01:28:26.603481538 (105h5m2.166804559s) [noise=true , address=79.127.219.38:4031 , piece_id=TBIY2ZMYVVKMYV4LVGJEKCJXMXYVJKSZKKBXASVNEUU2P7DKZKIQ, seg_pos={0 0}, upload=84097]: Uploading piece... Jun 23 01:28:26.603498030 (105h5m2.166821033s) [noise=true , address=77.90.31.15:4729 , piece_id=AAVBJ7VEQX67HQBXQ5DO3O5XJ37TRKU3PS2XXEH3WARMSEAQGQ2A, seg_pos={0 0}, upload=84097]: Uploading piece... Jun 23 01:28:26.603958736 (105h5m2.16728174s) [noise=true , address=79.127.219.51:4209 , piece_id=ZHJC5QUBGOHT3HRY4XPPADV7CYINQRXT4ID53KVC6LYXM7IV533A, seg_pos={0 0}, upload=84097]: Uploading piece... Jun 23 01:28:26.603997417 (105h5m2.167320407s) [noise=true , address=79.127.220.98:4410 , piece_id=DLW3G47XXWAJ4FZAZDEL6TCIN5FKG26LUNVELDWSUE2GDH6IPBLQ, seg_pos={0 0}, upload=84097]: Uploading piece... Jun 23 01:28:26.604297025 (105h5m2.167620026s) [noise=true , address=45.91.116.10:5084 , piece_id=WCIMJN6XDXUO2VCRKLAVOPVVQUEQDGHR4NP4KQGY27NEUPCD5OFA, seg_pos={0 0}, upload=84097]: Uploading piece... Jun 23 01:28:26.604692251 (105h5m2.16801525s) [noise=true , address=104.36.82.55:4217 , piece_id=WQIZ3WHXBLCKNJ4ZVQRC2PQIRH6IVVO7J3UMM5V5GXEMODV36KLQ, seg_pos={0 0}, upload=84097]: Uploading piece... Jun 23 01:28:26.605044347 (105h5m2.168367349s) [noise=true , address=104.36.82.53:4142 , piece_id=CL2WQNZHZNZM5BFJQIBUBQ7FQCIJSK4AR7PR2OLBIXFULPMBTGIQ, seg_pos={0 0}, upload=84097]: Uploading piece... Jun 23 01:28:26.605369657 (105h5m2.168692667s) [noise=true , address=45.91.116.8:5050 , piece_id=M4P3RKV2BUWYZOP2TOLNSMYOYDTOEG5NXXM6XZUYPLFQMLUGUIOA, seg_pos={0 0}, upload=84097]: Uploading piece... Jun 23 01:28:26.605753081 (105h5m2.169076095s) [noise=true , address=77.90.31.14:4693 , piece_id=YGA5IDHRAXIFCELCNCQF23PEFGJMFQOQHFWY3D7UODW5HYEIEWXA, seg_pos={0 0}, upload=84097]: Uploading piece... Jun 23 01:28:26.606212150 (105h5m2.169535158s) [noise=true , address=104.36.82.9:6021 , piece_id=4JQGGBGDZMQ6FDS35A7BZLB5OLZL2JHMD5Q3TXAETVX7TYA74EHQ, seg_pos={0 0}, upload=84097]: Uploading piece... Jun 23 01:28:26.606654203 (105h5m2.169977203s) [noise=true , address=104.36.82.16:6310 , piece_id=CRHVWU4XYDGUG7XVGC3TI6DC2CLBTSTHND4YB2YFEFPBD6TOAXKQ, seg_pos={0 0}, upload=84097]: Uploading piece... Jun 23 01:28:26.606961844 (105h5m2.170284849s) [noise=true , address=156.146.43.225:4448 , piece_id=5SH6UKJRS56UOVGLLW7JKTSAQANRVHYVOMEXZ56UWYM3AEB7QGVA, seg_pos={0 0}, upload=84097]: Uploading piece... Jun 23 01:28:26.607374002 (105h5m2.170697008s) [noise=true , address=104.36.82.12:6174 , piece_id=VQG2SHTAARK4FOSQER2YRKHZV7ODG7ZDPHNF6AESZGE3GMITKZZA, seg_pos={0 0}, upload=84097]: Uploading piece... Jun 23 01:28:26.607734796 (105h5m2.171057797s) [noise=true , address=45.91.116.9:5288 , piece_id=EQWQVX4JDLVTNRTKYNUIFGJYQUXPVUCUCYJPNVW2IYM3CGQY5D6A, seg_pos={0 0}, upload=84097]: Uploading piece... Jun 23 01:28:26.608082032 (105h5m2.171405034s) [noise=true , address=104.36.82.11:6131 , piece_id=BI4HK2NGEXXNEDYGRXRL77E5CONZG6JH4IWMONSB64DHOJLQQZPA, seg_pos={0 0}, upload=84097]: Uploading piece... Jun 23 01:28:26.608428729 (105h5m2.171751737s) [noise=true , address=104.36.82.42:6830 , piece_id=KW3RNOZYENVRUR663PVSC4WFCLI3RG5YBITFXOYC6W2GTGEFXTVA, seg_pos={0 0}, upload=84097]: Uploading piece... Jun 23 01:28:26.610585268 (105h5m2.173908273s) [noise=true , address=104.36.82.21:6483 , piece_id=WTUXQB3XLSHBMZZQNK745AK5U3ZLLNKC3TLW4CEFA7OX4DJCOTVQ, seg_pos={0 0}, upload=84097]: Uploading piece... Jun 23 01:28:26.612001063 (105h5m2.175324064s) [noise=true , address=104.36.82.3:5797 , piece_id=CJD4BHNHEVAGUJ5K55QM7OLAQQLEYRD4C6KJQBZ4FKJLDREGLNRQ, seg_pos={0 0}, upload=84097]: Uploading piece... Jun 23 01:28:26.687672089 (105h5m2.250995077s) [upload=84097]: Next returned nil segment Jun 23 01:28:26.687687969 (105h5m2.25101097s) [upload=84097]: Waiting for error group managing segments... Jun 23 01:28:26.707815366 (105h5m2.271138367s) [noise=true , address=79.127.220.98:4410 , piece_id=DLW3G47XXWAJ4FZAZDEL6TCIN5FKG26LUNVELDWSUE2GDH6IPBLQ, seg_pos={0 0}, upload=84097]: Done uploading piece. err: ecclient: upload failed (node:1Lay6oZEhDEeDmmkisrLND1j82PurLHV95nGJoKkk5wKUaK5S7, address:79.127.220.98:4410): protocol: storage node overloaded, request limit: 500 Jun 23 01:28:26.735633260 (105h5m2.298956259s) [noise=true , address=104.36.82.21:6483 , piece_id=WTUXQB3XLSHBMZZQNK745AK5U3ZLLNKC3TLW4CEFA7OX4DJCOTVQ, seg_pos={0 0}, upload=84097]: Done uploading piece. err: ecclient: upload failed (node:12gxKuBALDLDVDfRGBvMZymwGyPfPtV1BsHdBPqZs34ugvvYacG, address:104.36.82.21:6483): protocol: storage node overloaded, request limit: 2000 Jun 23 01:28:26.793765419 (105h5m2.357088412s) [noise=true , address=156.146.43.225:4448 , piece_id=5SH6UKJRS56UOVGLLW7JKTSAQANRVHYVOMEXZ56UWYM3AEB7QGVA, seg_pos={0 0}, upload=84097]: Done uploading piece. err: ecclient: upload failed (node:1irai8UTYknGTUkKHtKbBNTJTYMQsNTFnzrTEaWCMRDq1NNpGh, address:156.146.43.225:4448): protocol: storage node overloaded, request limit: 2000 Jun 23 01:28:26.868408971 (105h5m2.43173196s) [noise=true , address=77.90.31.15:4729 , piece_id=AAVBJ7VEQX67HQBXQ5DO3O5XJ37TRKU3PS2XXEH3WARMSEAQGQ2A, seg_pos={0 0}, upload=84097]: Done uploading piece. err: <nil> Jun 23 01:28:26.989727871 (105h5m2.553050876s) [noise=true , address=45.91.116.8:5050 , piece_id=M4P3RKV2BUWYZOP2TOLNSMYOYDTOEG5NXXM6XZUYPLFQMLUGUIOA, seg_pos={0 0}, upload=84097]: Done uploading piece. err: <nil> Jun 23 01:28:27.669519830 (105h5m3.23284283s) [noise=true , address=77.90.31.14:4693 , piece_id=YGA5IDHRAXIFCELCNCQF23PEFGJMFQOQHFWY3D7UODW5HYEIEWXA, seg_pos={0 0}, upload=84097]: Done uploading piece. err: <nil> Jun 23 01:28:27.697322244 (105h5m3.260645244s) [noise=true , address=104.36.82.42:6830 , piece_id=KW3RNOZYENVRUR663PVSC4WFCLI3RG5YBITFXOYC6W2GTGEFXTVA, seg_pos={0 0}, upload=84097]: Done uploading piece. err: <nil> Jun 23 01:28:28.071845632 (105h5m3.635168633s) [noise=true , address=104.36.82.55:4217 , piece_id=WQIZ3WHXBLCKNJ4ZVQRC2PQIRH6IVVO7J3UMM5V5GXEMODV36KLQ, seg_pos={0 0}, upload=84097]: Done uploading piece. err: <nil> Jun 23 01:28:28.589172169 (105h5m4.152495155s) [noise=true , address=45.91.116.9:5288 , piece_id=EQWQVX4JDLVTNRTKYNUIFGJYQUXPVUCUCYJPNVW2IYM3CGQY5D6A, seg_pos={0 0}, upload=84097]: Done uploading piece. err: <nil> Jun 23 01:28:29.203146980 (105h5m4.76646998s) [noise=true , address=104.36.82.16:6310 , piece_id=CRHVWU4XYDGUG7XVGC3TI6DC2CLBTSTHND4YB2YFEFPBD6TOAXKQ, seg_pos={0 0}, upload=84097]: Done uploading piece. err: ecclient: upload failed (node:1g14evoaeUnhJThf5QKbQT1Gd8WW9MHvaYQhQ1GRgwPfDLoC5R, address:104.36.82.16:6310): protocol: storage node overloaded, request limit: 2000 Jun 23 01:28:29.434402213 (105h5m4.997725203s) [noise=true , address=45.91.116.10:5084 , piece_id=WCIMJN6XDXUO2VCRKLAVOPVVQUEQDGHR4NP4KQGY27NEUPCD5OFA, seg_pos={0 0}, upload=84097]: Done uploading piece. err: <nil> Jun 23 01:28:30.293775103 (105h5m5.857098094s) [noise=true , address=104.36.82.9:6021 , piece_id=4JQGGBGDZMQ6FDS35A7BZLB5OLZL2JHMD5Q3TXAETVX7TYA74EHQ, seg_pos={0 0}, upload=84097]: Done uploading piece. err: <nil> Jun 23 01:28:30.347258498 (105h5m5.910581502s) [noise=true , address=104.36.82.12:6174 , piece_id=VQG2SHTAARK4FOSQER2YRKHZV7ODG7ZDPHNF6AESZGE3GMITKZZA, seg_pos={0 0}, upload=84097]: Done uploading piece. err: <nil> Jun 23 01:28:30.547306548 (105h5m6.110629586s) [noise=true , address=79.127.219.51:4209 , piece_id=ZHJC5QUBGOHT3HRY4XPPADV7CYINQRXT4ID53KVC6LYXM7IV533A, seg_pos={0 0}, upload=84097]: Done uploading piece. err: <nil> Jun 23 01:28:34.333414200 (105h5m9.896737197s) [noise=true , address=79.127.219.38:4031 , piece_id=TBIY2ZMYVVKMYV4LVGJEKCJXMXYVJKSZKKBXASVNEUU2P7DKZKIQ, seg_pos={0 0}, upload=84097]: Done uploading piece. err: <nil> Jun 23 01:28:42.464111369 (105h5m18.02743436s) [noise=true , address=104.36.82.53:4142 , piece_id=CL2WQNZHZNZM5BFJQIBUBQ7FQCIJSK4AR7PR2OLBIXFULPMBTGIQ, seg_pos={0 0}, upload=84097]: Done uploading piece. err: <nil> Jun 23 01:28:49.581287518 (105h5m25.144610509s) [noise=true , address=104.36.82.3:5797 , piece_id=CJD4BHNHEVAGUJ5K55QM7OLAQQLEYRD4C6KJQBZ4FKJLDREGLNRQ, seg_pos={0 0}, upload=84097]: Done uploading piece. err: <nil> Jun 23 01:29:26.535692548 (105h6m2.099015547s) [noise=true , address=104.36.82.11:6131 , piece_id=BI4HK2NGEXXNEDYGRXRL77E5CONZG6JH4IWMONSB64DHOJLQQZPA, seg_pos={0 0}, upload=84097]: Done uploading piece. err: ecclient: upload failed (node:129jahwAmyPHqyYaXz7XTCK8FakBrYZw94pBKGhPzzX23yjM3RQ, address:104.36.82.11:6131): protocol: expected piece hash; context deadline exceeded; EOF Jun 23 01:29:26.535746358 (105h6m2.099069362s) [seg_pos={0 0}, upload=84097]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jun 23 01:29:26.562546209 (105h6m2.125869225s) [upload=84097]: Done uploading segments... Jun 23 01:29:26.562593265 (105h6m2.125916275s) [upload=84097]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration:* 35.233489701s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [.................X.....................X...................X] duration: 1m0.001941109s FAIL (2025-06-23 01:31:39 UTC) operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1m0.001965509s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [...............X.............................X...........X..] duration: 16.528253114s FAIL (2025-06-23 01:30:50 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== oad=84082]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jun 23 01:27:43.602110205 (105h4m19.165433214s) [upload=84082]: Done uploading segments... Jun 23 01:27:43.602149595 (105h4m19.165472598s) [upload=84082]: Upload finished. err: <nil> Jun 23 01:28:26.568078065 (105h5m2.131401068s) [upload=84097]: Starting upload Jun 23 01:28:26.568176623 (105h5m2.131499629s) [upload=84097]: Done starting upload Jun 23 01:28:26.568333698 (105h5m2.131656699s) [upload=84097]: Uploading segments... Jun 23 01:28:26.569025334 (105h5m2.132348341s) [upload=84097]: Got next segment. Inline: false Jun 23 01:28:26.569144232 (105h5m2.132467237s) [upload=84097]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 23 01:28:26.569226194 (105h5m2.132549206s) [upload=84097]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 23 01:28:26.603382637 (105h5m2.166705641s) [seg_pos={0 0}, upload=84097]: Begin upload segment... Jun 23 01:28:26.603439808 (105h5m2.16676282s) [seg_pos={0 0}, upload=84097]: Done begin upload segment. Jun 23 01:28:26.603481538 (105h5m2.166804559s) [noise=true , address=79.127.219.38:4031 , piece_id=TBIY2ZMYVVKMYV4LVGJEKCJXMXYVJKSZKKBXASVNEUU2P7DKZKIQ, seg_pos={0 0}, upload=84097]: Uploading piece... Jun 23 01:28:26.603498030 (105h5m2.166821033s) [noise=true , address=77.90.31.15:4729 , piece_id=AAVBJ7VEQX67HQBXQ5DO3O5XJ37TRKU3PS2XXEH3WARMSEAQGQ2A, seg_pos={0 0}, upload=84097]: Uploading piece... Jun 23 01:28:26.603958736 (105h5m2.16728174s) [noise=true , address=79.127.219.51:4209 , piece_id=ZHJC5QUBGOHT3HRY4XPPADV7CYINQRXT4ID53KVC6LYXM7IV533A, seg_pos={0 0}, upload=84097]: Uploading piece... Jun 23 01:28:26.603997417 (105h5m2.167320407s) [noise=true , address=79.127.220.98:4410 , piece_id=DLW3G47XXWAJ4FZAZDEL6TCIN5FKG26LUNVELDWSUE2GDH6IPBLQ, seg_pos={0 0}, upload=84097]: Uploading piece... Jun 23 01:28:26.604297025 (105h5m2.167620026s) [noise=true , address=45.91.116.10:5084 , piece_id=WCIMJN6XDXUO2VCRKLAVOPVVQUEQDGHR4NP4KQGY27NEUPCD5OFA, seg_pos={0 0}, upload=84097]: Uploading piece... Jun 23 01:28:26.604692251 (105h5m2.16801525s) [noise=true , address=104.36.82.55:4217 , piece_id=WQIZ3WHXBLCKNJ4ZVQRC2PQIRH6IVVO7J3UMM5V5GXEMODV36KLQ, seg_pos={0 0}, upload=84097]: Uploading piece... Jun 23 01:28:26.605044347 (105h5m2.168367349s) [noise=true , address=104.36.82.53:4142 , piece_id=CL2WQNZHZNZM5BFJQIBUBQ7FQCIJSK4AR7PR2OLBIXFULPMBTGIQ, seg_pos={0 0}, upload=84097]: Uploading piece... Jun 23 01:28:26.605369657 (105h5m2.168692667s) [noise=true , address=45.91.116.8:5050 , piece_id=M4P3RKV2BUWYZOP2TOLNSMYOYDTOEG5NXXM6XZUYPLFQMLUGUIOA, seg_pos={0 0}, upload=84097]: Uploading piece... Jun 23 01:28:26.605753081 (105h5m2.169076095s) [noise=true , address=77.90.31.14:4693 , piece_id=YGA5IDHRAXIFCELCNCQF23PEFGJMFQOQHFWY3D7UODW5HYEIEWXA, seg_pos={0 0}, upload=84097]: Uploading piece... Jun 23 01:28:26.606212150 (105h5m2.169535158s) [noise=true , address=104.36.82.9:6021 , piece_id=4JQGGBGDZMQ6FDS35A7BZLB5OLZL2JHMD5Q3TXAETVX7TYA74EHQ, seg_pos={0 0}, upload=84097]: Uploading piece... Jun 23 01:28:26.606654203 (105h5m2.169977203s) [noise=true , address=104.36.82.16:6310 , piece_id=CRHVWU4XYDGUG7XVGC3TI6DC2CLBTSTHND4YB2YFEFPBD6TOAXKQ, seg_pos={0 0}, upload=84097]: Uploading piece... Jun 23 01:28:26.606961844 (105h5m2.170284849s) [noise=true , address=156.146.43.225:4448 , piece_id=5SH6UKJRS56UOVGLLW7JKTSAQANRVHYVOMEXZ56UWYM3AEB7QGVA, seg_pos={0 0}, upload=84097]: Uploading piece... Jun 23 01:28:26.607374002 (105h5m2.170697008s) [noise=true , address=104.36.82.12:6174 , piece_id=VQG2SHTAARK4FOSQER2YRKHZV7ODG7ZDPHNF6AESZGE3GMITKZZA, seg_pos={0 0}, upload=84097]: Uploading piece... Jun 23 01:28:26.607734796 (105h5m2.171057797s) [noise=true , address=45.91.116.9:5288 , piece_id=EQWQVX4JDLVTNRTKYNUIFGJYQUXPVUCUCYJPNVW2IYM3CGQY5D6A, seg_pos={0 0}, upload=84097]: Uploading piece... Jun 23 01:28:26.608082032 (105h5m2.171405034s) [noise=true , address=104.36.82.11:6131 , piece_id=BI4HK2NGEXXNEDYGRXRL77E5CONZG6JH4IWMONSB64DHOJLQQZPA, seg_pos={0 0}, upload=84097]: Uploading piece... Jun 23 01:28:26.608428729 (105h5m2.171751737s) [noise=true , address=104.36.82.42:6830 , piece_id=KW3RNOZYENVRUR663PVSC4WFCLI3RG5YBITFXOYC6W2GTGEFXTVA, seg_pos={0 0}, upload=84097]: Uploading piece... Jun 23 01:28:26.610585268 (105h5m2.173908273s) [noise=true , address=104.36.82.21:6483 , piece_id=WTUXQB3XLSHBMZZQNK745AK5U3ZLLNKC3TLW4CEFA7OX4DJCOTVQ, seg_pos={0 0}, upload=84097]: Uploading piece... Jun 23 01:28:26.612001063 (105h5m2.175324064s) [noise=true , address=104.36.82.3:5797 , piece_id=CJD4BHNHEVAGUJ5K55QM7OLAQQLEYRD4C6KJQBZ4FKJLDREGLNRQ, seg_pos={0 0}, upload=84097]: Uploading piece... Jun 23 01:28:26.687672089 (105h5m2.250995077s) [upload=84097]: Next returned nil segment Jun 23 01:28:26.687687969 (105h5m2.25101097s) [upload=84097]: Waiting for error group managing segments... Jun 23 01:28:26.707815366 (105h5m2.271138367s) [noise=true , address=79.127.220.98:4410 , piece_id=DLW3G47XXWAJ4FZAZDEL6TCIN5FKG26LUNVELDWSUE2GDH6IPBLQ, seg_pos={0 0}, upload=84097]: Done uploading piece. err: ecclient: upload failed (node:1Lay6oZEhDEeDmmkisrLND1j82PurLHV95nGJoKkk5wKUaK5S7, address:79.127.220.98:4410): protocol: storage node overloaded, request limit: 500 Jun 23 01:28:26.735633260 (105h5m2.298956259s) [noise=true , address=104.36.82.21:6483 , piece_id=WTUXQB3XLSHBMZZQNK745AK5U3ZLLNKC3TLW4CEFA7OX4DJCOTVQ, seg_pos={0 0}, upload=84097]: Done uploading piece. err: ecclient: upload failed (node:12gxKuBALDLDVDfRGBvMZymwGyPfPtV1BsHdBPqZs34ugvvYacG, address:104.36.82.21:6483): protocol: storage node overloaded, request limit: 2000 Jun 23 01:28:26.793765419 (105h5m2.357088412s) [noise=true , address=156.146.43.225:4448 , piece_id=5SH6UKJRS56UOVGLLW7JKTSAQANRVHYVOMEXZ56UWYM3AEB7QGVA, seg_pos={0 0}, upload=84097]: Done uploading piece. err: ecclient: upload failed (node:1irai8UTYknGTUkKHtKbBNTJTYMQsNTFnzrTEaWCMRDq1NNpGh, address:156.146.43.225:4448): protocol: storage node overloaded, request limit: 2000 Jun 23 01:28:26.868408971 (105h5m2.43173196s) [noise=true , address=77.90.31.15:4729 , piece_id=AAVBJ7VEQX67HQBXQ5DO3O5XJ37TRKU3PS2XXEH3WARMSEAQGQ2A, seg_pos={0 0}, upload=84097]: Done uploading piece. err: <nil> Jun 23 01:28:26.989727871 (105h5m2.553050876s) [noise=true , address=45.91.116.8:5050 , piece_id=M4P3RKV2BUWYZOP2TOLNSMYOYDTOEG5NXXM6XZUYPLFQMLUGUIOA, seg_pos={0 0}, upload=84097]: Done uploading piece. err: <nil> Jun 23 01:28:27.669519830 (105h5m3.23284283s) [noise=true , address=77.90.31.14:4693 , piece_id=YGA5IDHRAXIFCELCNCQF23PEFGJMFQOQHFWY3D7UODW5HYEIEWXA, seg_pos={0 0}, upload=84097]: Done uploading piece. err: <nil> Jun 23 01:28:27.697322244 (105h5m3.260645244s) [noise=true , address=104.36.82.42:6830 , piece_id=KW3RNOZYENVRUR663PVSC4WFCLI3RG5YBITFXOYC6W2GTGEFXTVA, seg_pos={0 0}, upload=84097]: Done uploading piece. err: <nil> Jun 23 01:28:28.071845632 (105h5m3.635168633s) [noise=true , address=104.36.82.55:4217 , piece_id=WQIZ3WHXBLCKNJ4ZVQRC2PQIRH6IVVO7J3UMM5V5GXEMODV36KLQ, seg_pos={0 0}, upload=84097]: Done uploading piece. err: <nil> Jun 23 01:28:28.589172169 (105h5m4.152495155s) [noise=true , address=45.91.116.9:5288 , piece_id=EQWQVX4JDLVTNRTKYNUIFGJYQUXPVUCUCYJPNVW2IYM3CGQY5D6A, seg_pos={0 0}, upload=84097]: Done uploading piece. err: <nil> Jun 23 01:28:29.203146980 (105h5m4.76646998s) [noise=true , address=104.36.82.16:6310 , piece_id=CRHVWU4XYDGUG7XVGC3TI6DC2CLBTSTHND4YB2YFEFPBD6TOAXKQ, seg_pos={0 0}, upload=84097]: Done uploading piece. err: ecclient: upload failed (node:1g14evoaeUnhJThf5QKbQT1Gd8WW9MHvaYQhQ1GRgwPfDLoC5R, address:104.36.82.16:6310): protocol: storage node overloaded, request limit: 2000 Jun 23 01:28:29.434402213 (105h5m4.997725203s) [noise=true , address=45.91.116.10:5084 , piece_id=WCIMJN6XDXUO2VCRKLAVOPVVQUEQDGHR4NP4KQGY27NEUPCD5OFA, seg_pos={0 0}, upload=84097]: Done uploading piece. err: <nil> Jun 23 01:28:30.293775103 (105h5m5.857098094s) [noise=true , address=104.36.82.9:6021 , piece_id=4JQGGBGDZMQ6FDS35A7BZLB5OLZL2JHMD5Q3TXAETVX7TYA74EHQ, seg_pos={0 0}, upload=84097]: Done uploading piece. err: <nil> Jun 23 01:28:30.347258498 (105h5m5.910581502s) [noise=true , address=104.36.82.12:6174 , piece_id=VQG2SHTAARK4FOSQER2YRKHZV7ODG7ZDPHNF6AESZGE3GMITKZZA, seg_pos={0 0}, upload=84097]: Done uploading piece. err: <nil> Jun 23 01:28:30.547306548 (105h5m6.110629586s) [noise=true , address=79.127.219.51:4209 , piece_id=ZHJC5QUBGOHT3HRY4XPPADV7CYINQRXT4ID53KVC6LYXM7IV533A, seg_pos={0 0}, upload=84097]: Done uploading piece. err: <nil> Jun 23 01:28:34.333414200 (105h5m9.896737197s) [noise=true , address=79.127.219.38:4031 , piece_id=TBIY2ZMYVVKMYV4LVGJEKCJXMXYVJKSZKKBXASVNEUU2P7DKZKIQ, seg_pos={0 0}, upload=84097]: Done uploading piece. err: <nil> Jun 23 01:28:42.464111369 (105h5m18.02743436s) [noise=true , address=104.36.82.53:4142 , piece_id=CL2WQNZHZNZM5BFJQIBUBQ7FQCIJSK4AR7PR2OLBIXFULPMBTGIQ, seg_pos={0 0}, upload=84097]: Done uploading piece. err: <nil> Jun 23 01:28:49.581287518 (105h5m25.144610509s) [noise=true , address=104.36.82.3:5797 , piece_id=CJD4BHNHEVAGUJ5K55QM7OLAQQLEYRD4C6KJQBZ4FKJLDREGLNRQ, seg_pos={0 0}, upload=84097]: Done uploading piece. err: <nil> Jun 23 01:29:26.535692548 (105h6m2.099015547s) [noise=true , address=104.36.82.11:6131 , piece_id=BI4HK2NGEXXNEDYGRXRL77E5CONZG6JH4IWMONSB64DHOJLQQZPA, seg_pos={0 0}, upload=84097]: Done uploading piece. err: ecclient: upload failed (node:129jahwAmyPHqyYaXz7XTCK8FakBrYZw94pBKGhPzzX23yjM3RQ, address:104.36.82.11:6131): protocol: expected piece hash; context deadline exceeded; EOF Jun 23 01:29:26.535746358 (105h6m2.099069362s) [seg_pos={0 0}, upload=84097]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jun 23 01:29:26.562546209 (105h6m2.125869225s) [upload=84097]: Done uploading segments... Jun 23 01:29:26.562593265 (105h6m2.125916275s) [upload=84097]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration:* 16.528330326s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [..................X.....................X...................] duration: 17.836334167s FAIL (2025-06-23 01:30:13 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 17.836361664s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [................X.............................X...........X.] duration: 7.461345433s FAIL (2025-06-23 01:29:44 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== oad=84082]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jun 23 01:27:43.602110205 (105h4m19.165433214s) [upload=84082]: Done uploading segments... Jun 23 01:27:43.602149595 (105h4m19.165472598s) [upload=84082]: Upload finished. err: <nil> Jun 23 01:28:26.568078065 (105h5m2.131401068s) [upload=84097]: Starting upload Jun 23 01:28:26.568176623 (105h5m2.131499629s) [upload=84097]: Done starting upload Jun 23 01:28:26.568333698 (105h5m2.131656699s) [upload=84097]: Uploading segments... Jun 23 01:28:26.569025334 (105h5m2.132348341s) [upload=84097]: Got next segment. Inline: false Jun 23 01:28:26.569144232 (105h5m2.132467237s) [upload=84097]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 23 01:28:26.569226194 (105h5m2.132549206s) [upload=84097]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 23 01:28:26.603382637 (105h5m2.166705641s) [seg_pos={0 0}, upload=84097]: Begin upload segment... Jun 23 01:28:26.603439808 (105h5m2.16676282s) [seg_pos={0 0}, upload=84097]: Done begin upload segment. Jun 23 01:28:26.603481538 (105h5m2.166804559s) [noise=true , address=79.127.219.38:4031 , piece_id=TBIY2ZMYVVKMYV4LVGJEKCJXMXYVJKSZKKBXASVNEUU2P7DKZKIQ, seg_pos={0 0}, upload=84097]: Uploading piece... Jun 23 01:28:26.603498030 (105h5m2.166821033s) [noise=true , address=77.90.31.15:4729 , piece_id=AAVBJ7VEQX67HQBXQ5DO3O5XJ37TRKU3PS2XXEH3WARMSEAQGQ2A, seg_pos={0 0}, upload=84097]: Uploading piece... Jun 23 01:28:26.603958736 (105h5m2.16728174s) [noise=true , address=79.127.219.51:4209 , piece_id=ZHJC5QUBGOHT3HRY4XPPADV7CYINQRXT4ID53KVC6LYXM7IV533A, seg_pos={0 0}, upload=84097]: Uploading piece... Jun 23 01:28:26.603997417 (105h5m2.167320407s) [noise=true , address=79.127.220.98:4410 , piece_id=DLW3G47XXWAJ4FZAZDEL6TCIN5FKG26LUNVELDWSUE2GDH6IPBLQ, seg_pos={0 0}, upload=84097]: Uploading piece... Jun 23 01:28:26.604297025 (105h5m2.167620026s) [noise=true , address=45.91.116.10:5084 , piece_id=WCIMJN6XDXUO2VCRKLAVOPVVQUEQDGHR4NP4KQGY27NEUPCD5OFA, seg_pos={0 0}, upload=84097]: Uploading piece... Jun 23 01:28:26.604692251 (105h5m2.16801525s) [noise=true , address=104.36.82.55:4217 , piece_id=WQIZ3WHXBLCKNJ4ZVQRC2PQIRH6IVVO7J3UMM5V5GXEMODV36KLQ, seg_pos={0 0}, upload=84097]: Uploading piece... Jun 23 01:28:26.605044347 (105h5m2.168367349s) [noise=true , address=104.36.82.53:4142 , piece_id=CL2WQNZHZNZM5BFJQIBUBQ7FQCIJSK4AR7PR2OLBIXFULPMBTGIQ, seg_pos={0 0}, upload=84097]: Uploading piece... Jun 23 01:28:26.605369657 (105h5m2.168692667s) [noise=true , address=45.91.116.8:5050 , piece_id=M4P3RKV2BUWYZOP2TOLNSMYOYDTOEG5NXXM6XZUYPLFQMLUGUIOA, seg_pos={0 0}, upload=84097]: Uploading piece... Jun 23 01:28:26.605753081 (105h5m2.169076095s) [noise=true , address=77.90.31.14:4693 , piece_id=YGA5IDHRAXIFCELCNCQF23PEFGJMFQOQHFWY3D7UODW5HYEIEWXA, seg_pos={0 0}, upload=84097]: Uploading piece... Jun 23 01:28:26.606212150 (105h5m2.169535158s) [noise=true , address=104.36.82.9:6021 , piece_id=4JQGGBGDZMQ6FDS35A7BZLB5OLZL2JHMD5Q3TXAETVX7TYA74EHQ, seg_pos={0 0}, upload=84097]: Uploading piece... Jun 23 01:28:26.606654203 (105h5m2.169977203s) [noise=true , address=104.36.82.16:6310 , piece_id=CRHVWU4XYDGUG7XVGC3TI6DC2CLBTSTHND4YB2YFEFPBD6TOAXKQ, seg_pos={0 0}, upload=84097]: Uploading piece... Jun 23 01:28:26.606961844 (105h5m2.170284849s) [noise=true , address=156.146.43.225:4448 , piece_id=5SH6UKJRS56UOVGLLW7JKTSAQANRVHYVOMEXZ56UWYM3AEB7QGVA, seg_pos={0 0}, upload=84097]: Uploading piece... Jun 23 01:28:26.607374002 (105h5m2.170697008s) [noise=true , address=104.36.82.12:6174 , piece_id=VQG2SHTAARK4FOSQER2YRKHZV7ODG7ZDPHNF6AESZGE3GMITKZZA, seg_pos={0 0}, upload=84097]: Uploading piece... Jun 23 01:28:26.607734796 (105h5m2.171057797s) [noise=true , address=45.91.116.9:5288 , piece_id=EQWQVX4JDLVTNRTKYNUIFGJYQUXPVUCUCYJPNVW2IYM3CGQY5D6A, seg_pos={0 0}, upload=84097]: Uploading piece... Jun 23 01:28:26.608082032 (105h5m2.171405034s) [noise=true , address=104.36.82.11:6131 , piece_id=BI4HK2NGEXXNEDYGRXRL77E5CONZG6JH4IWMONSB64DHOJLQQZPA, seg_pos={0 0}, upload=84097]: Uploading piece... Jun 23 01:28:26.608428729 (105h5m2.171751737s) [noise=true , address=104.36.82.42:6830 , piece_id=KW3RNOZYENVRUR663PVSC4WFCLI3RG5YBITFXOYC6W2GTGEFXTVA, seg_pos={0 0}, upload=84097]: Uploading piece... Jun 23 01:28:26.610585268 (105h5m2.173908273s) [noise=true , address=104.36.82.21:6483 , piece_id=WTUXQB3XLSHBMZZQNK745AK5U3ZLLNKC3TLW4CEFA7OX4DJCOTVQ, seg_pos={0 0}, upload=84097]: Uploading piece... Jun 23 01:28:26.612001063 (105h5m2.175324064s) [noise=true , address=104.36.82.3:5797 , piece_id=CJD4BHNHEVAGUJ5K55QM7OLAQQLEYRD4C6KJQBZ4FKJLDREGLNRQ, seg_pos={0 0}, upload=84097]: Uploading piece... Jun 23 01:28:26.687672089 (105h5m2.250995077s) [upload=84097]: Next returned nil segment Jun 23 01:28:26.687687969 (105h5m2.25101097s) [upload=84097]: Waiting for error group managing segments... Jun 23 01:28:26.707815366 (105h5m2.271138367s) [noise=true , address=79.127.220.98:4410 , piece_id=DLW3G47XXWAJ4FZAZDEL6TCIN5FKG26LUNVELDWSUE2GDH6IPBLQ, seg_pos={0 0}, upload=84097]: Done uploading piece. err: ecclient: upload failed (node:1Lay6oZEhDEeDmmkisrLND1j82PurLHV95nGJoKkk5wKUaK5S7, address:79.127.220.98:4410): protocol: storage node overloaded, request limit: 500 Jun 23 01:28:26.735633260 (105h5m2.298956259s) [noise=true , address=104.36.82.21:6483 , piece_id=WTUXQB3XLSHBMZZQNK745AK5U3ZLLNKC3TLW4CEFA7OX4DJCOTVQ, seg_pos={0 0}, upload=84097]: Done uploading piece. err: ecclient: upload failed (node:12gxKuBALDLDVDfRGBvMZymwGyPfPtV1BsHdBPqZs34ugvvYacG, address:104.36.82.21:6483): protocol: storage node overloaded, request limit: 2000 Jun 23 01:28:26.793765419 (105h5m2.357088412s) [noise=true , address=156.146.43.225:4448 , piece_id=5SH6UKJRS56UOVGLLW7JKTSAQANRVHYVOMEXZ56UWYM3AEB7QGVA, seg_pos={0 0}, upload=84097]: Done uploading piece. err: ecclient: upload failed (node:1irai8UTYknGTUkKHtKbBNTJTYMQsNTFnzrTEaWCMRDq1NNpGh, address:156.146.43.225:4448): protocol: storage node overloaded, request limit: 2000 Jun 23 01:28:26.868408971 (105h5m2.43173196s) [noise=true , address=77.90.31.15:4729 , piece_id=AAVBJ7VEQX67HQBXQ5DO3O5XJ37TRKU3PS2XXEH3WARMSEAQGQ2A, seg_pos={0 0}, upload=84097]: Done uploading piece. err: <nil> Jun 23 01:28:26.989727871 (105h5m2.553050876s) [noise=true , address=45.91.116.8:5050 , piece_id=M4P3RKV2BUWYZOP2TOLNSMYOYDTOEG5NXXM6XZUYPLFQMLUGUIOA, seg_pos={0 0}, upload=84097]: Done uploading piece. err: <nil> Jun 23 01:28:27.669519830 (105h5m3.23284283s) [noise=true , address=77.90.31.14:4693 , piece_id=YGA5IDHRAXIFCELCNCQF23PEFGJMFQOQHFWY3D7UODW5HYEIEWXA, seg_pos={0 0}, upload=84097]: Done uploading piece. err: <nil> Jun 23 01:28:27.697322244 (105h5m3.260645244s) [noise=true , address=104.36.82.42:6830 , piece_id=KW3RNOZYENVRUR663PVSC4WFCLI3RG5YBITFXOYC6W2GTGEFXTVA, seg_pos={0 0}, upload=84097]: Done uploading piece. err: <nil> Jun 23 01:28:28.071845632 (105h5m3.635168633s) [noise=true , address=104.36.82.55:4217 , piece_id=WQIZ3WHXBLCKNJ4ZVQRC2PQIRH6IVVO7J3UMM5V5GXEMODV36KLQ, seg_pos={0 0}, upload=84097]: Done uploading piece. err: <nil> Jun 23 01:28:28.589172169 (105h5m4.152495155s) [noise=true , address=45.91.116.9:5288 , piece_id=EQWQVX4JDLVTNRTKYNUIFGJYQUXPVUCUCYJPNVW2IYM3CGQY5D6A, seg_pos={0 0}, upload=84097]: Done uploading piece. err: <nil> Jun 23 01:28:29.203146980 (105h5m4.76646998s) [noise=true , address=104.36.82.16:6310 , piece_id=CRHVWU4XYDGUG7XVGC3TI6DC2CLBTSTHND4YB2YFEFPBD6TOAXKQ, seg_pos={0 0}, upload=84097]: Done uploading piece. err: ecclient: upload failed (node:1g14evoaeUnhJThf5QKbQT1Gd8WW9MHvaYQhQ1GRgwPfDLoC5R, address:104.36.82.16:6310): protocol: storage node overloaded, request limit: 2000 Jun 23 01:28:29.434402213 (105h5m4.997725203s) [noise=true , address=45.91.116.10:5084 , piece_id=WCIMJN6XDXUO2VCRKLAVOPVVQUEQDGHR4NP4KQGY27NEUPCD5OFA, seg_pos={0 0}, upload=84097]: Done uploading piece. err: <nil> Jun 23 01:28:30.293775103 (105h5m5.857098094s) [noise=true , address=104.36.82.9:6021 , piece_id=4JQGGBGDZMQ6FDS35A7BZLB5OLZL2JHMD5Q3TXAETVX7TYA74EHQ, seg_pos={0 0}, upload=84097]: Done uploading piece. err: <nil> Jun 23 01:28:30.347258498 (105h5m5.910581502s) [noise=true , address=104.36.82.12:6174 , piece_id=VQG2SHTAARK4FOSQER2YRKHZV7ODG7ZDPHNF6AESZGE3GMITKZZA, seg_pos={0 0}, upload=84097]: Done uploading piece. err: <nil> Jun 23 01:28:30.547306548 (105h5m6.110629586s) [noise=true , address=79.127.219.51:4209 , piece_id=ZHJC5QUBGOHT3HRY4XPPADV7CYINQRXT4ID53KVC6LYXM7IV533A, seg_pos={0 0}, upload=84097]: Done uploading piece. err: <nil> Jun 23 01:28:34.333414200 (105h5m9.896737197s) [noise=true , address=79.127.219.38:4031 , piece_id=TBIY2ZMYVVKMYV4LVGJEKCJXMXYVJKSZKKBXASVNEUU2P7DKZKIQ, seg_pos={0 0}, upload=84097]: Done uploading piece. err: <nil> Jun 23 01:28:42.464111369 (105h5m18.02743436s) [noise=true , address=104.36.82.53:4142 , piece_id=CL2WQNZHZNZM5BFJQIBUBQ7FQCIJSK4AR7PR2OLBIXFULPMBTGIQ, seg_pos={0 0}, upload=84097]: Done uploading piece. err: <nil> Jun 23 01:28:49.581287518 (105h5m25.144610509s) [noise=true , address=104.36.82.3:5797 , piece_id=CJD4BHNHEVAGUJ5K55QM7OLAQQLEYRD4C6KJQBZ4FKJLDREGLNRQ, seg_pos={0 0}, upload=84097]: Done uploading piece. err: <nil> Jun 23 01:29:26.535692548 (105h6m2.099015547s) [noise=true , address=104.36.82.11:6131 , piece_id=BI4HK2NGEXXNEDYGRXRL77E5CONZG6JH4IWMONSB64DHOJLQQZPA, seg_pos={0 0}, upload=84097]: Done uploading piece. err: ecclient: upload failed (node:129jahwAmyPHqyYaXz7XTCK8FakBrYZw94pBKGhPzzX23yjM3RQ, address:104.36.82.11:6131): protocol: expected piece hash; context deadline exceeded; EOF Jun 23 01:29:26.535746358 (105h6m2.099069362s) [seg_pos={0 0}, upload=84097]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jun 23 01:29:26.562546209 (105h6m2.125869225s) [upload=84097]: Done uploading segments... Jun 23 01:29:26.562593265 (105h6m2.125916275s) [upload=84097]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration:* 7.461374654s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [...................X.....................X..................] duration: 9.298177956s FAIL (2025-06-23 01:29:28 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 9.298219502s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [.................X.............................X...........X] duration: 1m0.028569905s FAIL (2025-06-23 01:29:26 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== oad=84082]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jun 23 01:27:43.602110205 (105h4m19.165433214s) [upload=84082]: Done uploading segments... Jun 23 01:27:43.602149595 (105h4m19.165472598s) [upload=84082]: Upload finished. err: <nil> Jun 23 01:28:26.568078065 (105h5m2.131401068s) [upload=84097]: Starting upload Jun 23 01:28:26.568176623 (105h5m2.131499629s) [upload=84097]: Done starting upload Jun 23 01:28:26.568333698 (105h5m2.131656699s) [upload=84097]: Uploading segments... Jun 23 01:28:26.569025334 (105h5m2.132348341s) [upload=84097]: Got next segment. Inline: false Jun 23 01:28:26.569144232 (105h5m2.132467237s) [upload=84097]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 23 01:28:26.569226194 (105h5m2.132549206s) [upload=84097]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 23 01:28:26.603382637 (105h5m2.166705641s) [seg_pos={0 0}, upload=84097]: Begin upload segment... Jun 23 01:28:26.603439808 (105h5m2.16676282s) [seg_pos={0 0}, upload=84097]: Done begin upload segment. Jun 23 01:28:26.603481538 (105h5m2.166804559s) [noise=true , address=79.127.219.38:4031 , piece_id=TBIY2ZMYVVKMYV4LVGJEKCJXMXYVJKSZKKBXASVNEUU2P7DKZKIQ, seg_pos={0 0}, upload=84097]: Uploading piece... Jun 23 01:28:26.603498030 (105h5m2.166821033s) [noise=true , address=77.90.31.15:4729 , piece_id=AAVBJ7VEQX67HQBXQ5DO3O5XJ37TRKU3PS2XXEH3WARMSEAQGQ2A, seg_pos={0 0}, upload=84097]: Uploading piece... Jun 23 01:28:26.603958736 (105h5m2.16728174s) [noise=true , address=79.127.219.51:4209 , piece_id=ZHJC5QUBGOHT3HRY4XPPADV7CYINQRXT4ID53KVC6LYXM7IV533A, seg_pos={0 0}, upload=84097]: Uploading piece... Jun 23 01:28:26.603997417 (105h5m2.167320407s) [noise=true , address=79.127.220.98:4410 , piece_id=DLW3G47XXWAJ4FZAZDEL6TCIN5FKG26LUNVELDWSUE2GDH6IPBLQ, seg_pos={0 0}, upload=84097]: Uploading piece... Jun 23 01:28:26.604297025 (105h5m2.167620026s) [noise=true , address=45.91.116.10:5084 , piece_id=WCIMJN6XDXUO2VCRKLAVOPVVQUEQDGHR4NP4KQGY27NEUPCD5OFA, seg_pos={0 0}, upload=84097]: Uploading piece... Jun 23 01:28:26.604692251 (105h5m2.16801525s) [noise=true , address=104.36.82.55:4217 , piece_id=WQIZ3WHXBLCKNJ4ZVQRC2PQIRH6IVVO7J3UMM5V5GXEMODV36KLQ, seg_pos={0 0}, upload=84097]: Uploading piece... Jun 23 01:28:26.605044347 (105h5m2.168367349s) [noise=true , address=104.36.82.53:4142 , piece_id=CL2WQNZHZNZM5BFJQIBUBQ7FQCIJSK4AR7PR2OLBIXFULPMBTGIQ, seg_pos={0 0}, upload=84097]: Uploading piece... Jun 23 01:28:26.605369657 (105h5m2.168692667s) [noise=true , address=45.91.116.8:5050 , piece_id=M4P3RKV2BUWYZOP2TOLNSMYOYDTOEG5NXXM6XZUYPLFQMLUGUIOA, seg_pos={0 0}, upload=84097]: Uploading piece... Jun 23 01:28:26.605753081 (105h5m2.169076095s) [noise=true , address=77.90.31.14:4693 , piece_id=YGA5IDHRAXIFCELCNCQF23PEFGJMFQOQHFWY3D7UODW5HYEIEWXA, seg_pos={0 0}, upload=84097]: Uploading piece... Jun 23 01:28:26.606212150 (105h5m2.169535158s) [noise=true , address=104.36.82.9:6021 , piece_id=4JQGGBGDZMQ6FDS35A7BZLB5OLZL2JHMD5Q3TXAETVX7TYA74EHQ, seg_pos={0 0}, upload=84097]: Uploading piece... Jun 23 01:28:26.606654203 (105h5m2.169977203s) [noise=true , address=104.36.82.16:6310 , piece_id=CRHVWU4XYDGUG7XVGC3TI6DC2CLBTSTHND4YB2YFEFPBD6TOAXKQ, seg_pos={0 0}, upload=84097]: Uploading piece... Jun 23 01:28:26.606961844 (105h5m2.170284849s) [noise=true , address=156.146.43.225:4448 , piece_id=5SH6UKJRS56UOVGLLW7JKTSAQANRVHYVOMEXZ56UWYM3AEB7QGVA, seg_pos={0 0}, upload=84097]: Uploading piece... Jun 23 01:28:26.607374002 (105h5m2.170697008s) [noise=true , address=104.36.82.12:6174 , piece_id=VQG2SHTAARK4FOSQER2YRKHZV7ODG7ZDPHNF6AESZGE3GMITKZZA, seg_pos={0 0}, upload=84097]: Uploading piece... Jun 23 01:28:26.607734796 (105h5m2.171057797s) [noise=true , address=45.91.116.9:5288 , piece_id=EQWQVX4JDLVTNRTKYNUIFGJYQUXPVUCUCYJPNVW2IYM3CGQY5D6A, seg_pos={0 0}, upload=84097]: Uploading piece... Jun 23 01:28:26.608082032 (105h5m2.171405034s) [noise=true , address=104.36.82.11:6131 , piece_id=BI4HK2NGEXXNEDYGRXRL77E5CONZG6JH4IWMONSB64DHOJLQQZPA, seg_pos={0 0}, upload=84097]: Uploading piece... Jun 23 01:28:26.608428729 (105h5m2.171751737s) [noise=true , address=104.36.82.42:6830 , piece_id=KW3RNOZYENVRUR663PVSC4WFCLI3RG5YBITFXOYC6W2GTGEFXTVA, seg_pos={0 0}, upload=84097]: Uploading piece... Jun 23 01:28:26.610585268 (105h5m2.173908273s) [noise=true , address=104.36.82.21:6483 , piece_id=WTUXQB3XLSHBMZZQNK745AK5U3ZLLNKC3TLW4CEFA7OX4DJCOTVQ, seg_pos={0 0}, upload=84097]: Uploading piece... Jun 23 01:28:26.612001063 (105h5m2.175324064s) [noise=true , address=104.36.82.3:5797 , piece_id=CJD4BHNHEVAGUJ5K55QM7OLAQQLEYRD4C6KJQBZ4FKJLDREGLNRQ, seg_pos={0 0}, upload=84097]: Uploading piece... Jun 23 01:28:26.687672089 (105h5m2.250995077s) [upload=84097]: Next returned nil segment Jun 23 01:28:26.687687969 (105h5m2.25101097s) [upload=84097]: Waiting for error group managing segments... Jun 23 01:28:26.707815366 (105h5m2.271138367s) [noise=true , address=79.127.220.98:4410 , piece_id=DLW3G47XXWAJ4FZAZDEL6TCIN5FKG26LUNVELDWSUE2GDH6IPBLQ, seg_pos={0 0}, upload=84097]: Done uploading piece. err: ecclient: upload failed (node:1Lay6oZEhDEeDmmkisrLND1j82PurLHV95nGJoKkk5wKUaK5S7, address:79.127.220.98:4410): protocol: storage node overloaded, request limit: 500 Jun 23 01:28:26.735633260 (105h5m2.298956259s) [noise=true , address=104.36.82.21:6483 , piece_id=WTUXQB3XLSHBMZZQNK745AK5U3ZLLNKC3TLW4CEFA7OX4DJCOTVQ, seg_pos={0 0}, upload=84097]: Done uploading piece. err: ecclient: upload failed (node:12gxKuBALDLDVDfRGBvMZymwGyPfPtV1BsHdBPqZs34ugvvYacG, address:104.36.82.21:6483): protocol: storage node overloaded, request limit: 2000 Jun 23 01:28:26.793765419 (105h5m2.357088412s) [noise=true , address=156.146.43.225:4448 , piece_id=5SH6UKJRS56UOVGLLW7JKTSAQANRVHYVOMEXZ56UWYM3AEB7QGVA, seg_pos={0 0}, upload=84097]: Done uploading piece. err: ecclient: upload failed (node:1irai8UTYknGTUkKHtKbBNTJTYMQsNTFnzrTEaWCMRDq1NNpGh, address:156.146.43.225:4448): protocol: storage node overloaded, request limit: 2000 Jun 23 01:28:26.868408971 (105h5m2.43173196s) [noise=true , address=77.90.31.15:4729 , piece_id=AAVBJ7VEQX67HQBXQ5DO3O5XJ37TRKU3PS2XXEH3WARMSEAQGQ2A, seg_pos={0 0}, upload=84097]: Done uploading piece. err: <nil> Jun 23 01:28:26.989727871 (105h5m2.553050876s) [noise=true , address=45.91.116.8:5050 , piece_id=M4P3RKV2BUWYZOP2TOLNSMYOYDTOEG5NXXM6XZUYPLFQMLUGUIOA, seg_pos={0 0}, upload=84097]: Done uploading piece. err: <nil> Jun 23 01:28:27.669519830 (105h5m3.23284283s) [noise=true , address=77.90.31.14:4693 , piece_id=YGA5IDHRAXIFCELCNCQF23PEFGJMFQOQHFWY3D7UODW5HYEIEWXA, seg_pos={0 0}, upload=84097]: Done uploading piece. err: <nil> Jun 23 01:28:27.697322244 (105h5m3.260645244s) [noise=true , address=104.36.82.42:6830 , piece_id=KW3RNOZYENVRUR663PVSC4WFCLI3RG5YBITFXOYC6W2GTGEFXTVA, seg_pos={0 0}, upload=84097]: Done uploading piece. err: <nil> Jun 23 01:28:28.071845632 (105h5m3.635168633s) [noise=true , address=104.36.82.55:4217 , piece_id=WQIZ3WHXBLCKNJ4ZVQRC2PQIRH6IVVO7J3UMM5V5GXEMODV36KLQ, seg_pos={0 0}, upload=84097]: Done uploading piece. err: <nil> Jun 23 01:28:28.589172169 (105h5m4.152495155s) [noise=true , address=45.91.116.9:5288 , piece_id=EQWQVX4JDLVTNRTKYNUIFGJYQUXPVUCUCYJPNVW2IYM3CGQY5D6A, seg_pos={0 0}, upload=84097]: Done uploading piece. err: <nil> Jun 23 01:28:29.203146980 (105h5m4.76646998s) [noise=true , address=104.36.82.16:6310 , piece_id=CRHVWU4XYDGUG7XVGC3TI6DC2CLBTSTHND4YB2YFEFPBD6TOAXKQ, seg_pos={0 0}, upload=84097]: Done uploading piece. err: ecclient: upload failed (node:1g14evoaeUnhJThf5QKbQT1Gd8WW9MHvaYQhQ1GRgwPfDLoC5R, address:104.36.82.16:6310): protocol: storage node overloaded, request limit: 2000 Jun 23 01:28:29.434402213 (105h5m4.997725203s) [noise=true , address=45.91.116.10:5084 , piece_id=WCIMJN6XDXUO2VCRKLAVOPVVQUEQDGHR4NP4KQGY27NEUPCD5OFA, seg_pos={0 0}, upload=84097]: Done uploading piece. err: <nil> Jun 23 01:28:30.293775103 (105h5m5.857098094s) [noise=true , address=104.36.82.9:6021 , piece_id=4JQGGBGDZMQ6FDS35A7BZLB5OLZL2JHMD5Q3TXAETVX7TYA74EHQ, seg_pos={0 0}, upload=84097]: Done uploading piece. err: <nil> Jun 23 01:28:30.347258498 (105h5m5.910581502s) [noise=true , address=104.36.82.12:6174 , piece_id=VQG2SHTAARK4FOSQER2YRKHZV7ODG7ZDPHNF6AESZGE3GMITKZZA, seg_pos={0 0}, upload=84097]: Done uploading piece. err: <nil> Jun 23 01:28:30.547306548 (105h5m6.110629586s) [noise=true , address=79.127.219.51:4209 , piece_id=ZHJC5QUBGOHT3HRY4XPPADV7CYINQRXT4ID53KVC6LYXM7IV533A, seg_pos={0 0}, upload=84097]: Done uploading piece. err: <nil> Jun 23 01:28:34.333414200 (105h5m9.896737197s) [noise=true , address=79.127.219.38:4031 , piece_id=TBIY2ZMYVVKMYV4LVGJEKCJXMXYVJKSZKKBXASVNEUU2P7DKZKIQ, seg_pos={0 0}, upload=84097]: Done uploading piece. err: <nil> Jun 23 01:28:42.464111369 (105h5m18.02743436s) [noise=true , address=104.36.82.53:4142 , piece_id=CL2WQNZHZNZM5BFJQIBUBQ7FQCIJSK4AR7PR2OLBIXFULPMBTGIQ, seg_pos={0 0}, upload=84097]: Done uploading piece. err: <nil> Jun 23 01:28:49.581287518 (105h5m25.144610509s) [noise=true , address=104.36.82.3:5797 , piece_id=CJD4BHNHEVAGUJ5K55QM7OLAQQLEYRD4C6KJQBZ4FKJLDREGLNRQ, seg_pos={0 0}, upload=84097]: Done uploading piece. err: <nil> Jun 23 01:29:26.535692548 (105h6m2.099015547s) [noise=true , address=104.36.82.11:6131 , piece_id=BI4HK2NGEXXNEDYGRXRL77E5CONZG6JH4IWMONSB64DHOJLQQZPA, seg_pos={0 0}, upload=84097]: Done uploading piece. err: ecclient: upload failed (node:129jahwAmyPHqyYaXz7XTCK8FakBrYZw94pBKGhPzzX23yjM3RQ, address:104.36.82.11:6131): protocol: expected piece hash; context deadline exceeded; EOF Jun 23 01:29:26.535746358 (105h6m2.099069362s) [seg_pos={0 0}, upload=84097]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jun 23 01:29:26.562546209 (105h6m2.125869225s) [upload=84097]: Done uploading segments... Jun 23 01:29:26.562593265 (105h6m2.125916275s) [upload=84097]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration:* 1m0.028586995s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [....................X.....................X.................] duration: 18.562819465s FAIL (2025-06-23 01:28:43 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 18.562844072s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [.....................X.....................X................] duration: 22.61366977s FAIL (2025-06-23 01:27:49 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 22.613695494s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [..................X.............................X...........] duration: 7.017098653s FAIL (2025-06-23 01:27:44 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== nil> Jun 23 01:18:36.741346057 (104h55m12.304669068s) [upload=83962]: Starting upload Jun 23 01:18:36.741457026 (104h55m12.304780037s) [upload=83962]: Done starting upload Jun 23 01:18:36.741574186 (104h55m12.304897176s) [upload=83962]: Uploading segments... Jun 23 01:18:36.741945356 (104h55m12.305268357s) [upload=83962]: Got next segment. Inline: false Jun 23 01:18:36.742017220 (104h55m12.305340209s) [upload=83962]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 23 01:18:36.742022591 (104h55m12.305345591s) [upload=83962]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 23 01:18:36.777635436 (104h55m12.34095844s) [seg_pos={0 0}, upload=83962]: Begin upload segment... Jun 23 01:18:36.777700720 (104h55m12.341023723s) [seg_pos={0 0}, upload=83962]: Done begin upload segment. Jun 23 01:18:36.777707373 (104h55m12.341030382s) [noise=true , address=104.36.82.34:6569 , piece_id=EGMJEZMPGKMR5JHGP3TH3MJ7SJ5XWVMTVXCAFDX4RKPIVVMP4BAA, seg_pos={0 0}, upload=83962]: Uploading piece... Jun 23 01:18:36.777724726 (104h55m12.341047737s) [noise=true , address=77.90.31.10:4535 , piece_id=BORXDT7QJFNH6WJR6GEMRXATEOJFLESKT5XMWY6UOER6R2RW6KBQ, seg_pos={0 0}, upload=83962]: Uploading piece... Jun 23 01:18:36.777804915 (104h55m12.341127924s) [noise=true , address=45.91.116.9:5277 , piece_id=U7K7GEX3GPXNULZP35E65E52YL2BUNEXEKTRRZUP6GAALWWPS6KA, seg_pos={0 0}, upload=83962]: Uploading piece... Jun 23 01:18:36.778210171 (104h55m12.341533182s) [noise=true , address=104.36.82.53:4128 , piece_id=YZHGLRKEISP3W4KUK3LL3EQEJUMX2LFPIDX2O676LS3GY3PNZ4TA, seg_pos={0 0}, upload=83962]: Uploading piece... Jun 23 01:18:36.778614738 (104h55m12.341937746s) [noise=true , address=104.36.82.56:4245 , piece_id=CGUQ357JLAQJCIDNQ23BNV4Q3R4RGVJI2Q5Z7B56LJKZIFENJM3Q, seg_pos={0 0}, upload=83962]: Uploading piece... Jun 23 01:18:36.778987954 (104h55m12.342310963s) [noise=true , address=104.36.82.7:5933 , piece_id=Q27EIDAGLD5IZGDJ6QBB3ZA5ULFSIPXY4JWFTNBMLX346CK2ADBQ, seg_pos={0 0}, upload=83962]: Uploading piece... Jun 23 01:18:36.779391219 (104h55m12.342714228s) [noise=true , address=104.36.82.54:4157 , piece_id=ESGS5AM5NBIEFUH7GGK7QV3PIQDMBLIKLIIGEBO7TJML72TF4EAQ, seg_pos={0 0}, upload=83962]: Uploading piece... Jun 23 01:18:36.779521731 (104h55m12.342844734s) [noise=true , address=104.36.82.20:6444 , piece_id=CWBFFDJCLPZFFSP7NVOKELPXCR3T3DV43FGSSWNGJO6W2IHXEJ3A, seg_pos={0 0}, upload=83962]: Uploading piece... Jun 23 01:18:36.779962650 (104h55m12.34328566s) [noise=true , address=104.36.82.14:6228 , piece_id=QMSNWZWKIZ4DQPXOZJC67HOTDUBWJNKY45M5QZF7I7ENC4TZE2VA, seg_pos={0 0}, upload=83962]: Uploading piece... Jun 23 01:18:36.780465195 (104h55m12.343788212s) [noise=true , address=77.90.31.18:4825 , piece_id=DIQKUPGRMOVTGWHU5TVS4O2DXIBO74SKVFU3WHJN5PDU4R4XZI7Q, seg_pos={0 0}, upload=83962]: Uploading piece... Jun 23 01:18:36.780615746 (104h55m12.343938757s) [noise=true , address=104.36.82.22:6530 , piece_id=ZHRUIZ5BZCM67DGFQZS6NI2DV732UCVDYV3GM55EISACHUWN5ZLQ, seg_pos={0 0}, upload=83962]: Uploading piece... Jun 23 01:18:36.781165217 (104h55m12.344488225s) [noise=true , address=104.36.82.51:4084 , piece_id=DODGPPX2WWL6OWRP56V6QX2JZ2PMGHXG3AEQEBRACLAHLWGJTJWA, seg_pos={0 0}, upload=83962]: Uploading piece... Jun 23 01:18:36.781550748 (104h55m12.344873758s) [noise=true , address=79.127.205.234:5591 , piece_id=U2VCI7JO4IAU4B3LXOGMLQT452CIP2LLT2T53HBQFCR6U6UDQ2RA, seg_pos={0 0}, upload=83962]: Uploading piece... Jun 23 01:18:36.782014436 (104h55m12.345337441s) [noise=true , address=45.91.116.8:5050 , piece_id=6HAF64M3236TINYRYVIYSG27BF2QW6XGCIKE6SBVVZ527LFOP2ZQ, seg_pos={0 0}, upload=83962]: Uploading piece... Jun 23 01:18:36.782496184 (104h55m12.345819196s) [noise=true , address=104.36.82.15:6286 , piece_id=ETSDIEVDHOYGJC62M3M6JUQQLZL5PWPYRLUAVWHU7HE7FGX4YE6Q, seg_pos={0 0}, upload=83962]: Uploading piece... Jun 23 01:18:36.782881896 (104h55m12.346204905s) [noise=true , address=79.127.219.38:4026 , piece_id=EQUPNI4CUJP6MTH6MODLH6M5YJ4ZNO5ALN6GPSX22RNBCRRIPUOQ, seg_pos={0 0}, upload=83962]: Uploading piece... Jun 23 01:18:36.783409077 (104h55m12.346732086s) [noise=true , address=104.36.82.55:4225 , piece_id=M77UT6JBGUAVSXFMDUMYPRQWN4X7TOSX35LIWATYKMDXE6CSYPDA, seg_pos={0 0}, upload=83962]: Uploading piece... Jun 23 01:18:36.783797923 (104h55m12.34712093s) [noise=true , address=77.90.31.14:4695 , piece_id=RLOHBVSNWCGZU4YE2HNCBKV6V7BY7JDCDKJFWCM4YYFH5EZXWCFQ, seg_pos={0 0}, upload=83962]: Uploading piece... Jun 23 01:18:36.863659010 (104h55m12.426982006s) [upload=83962]: Next returned nil segment Jun 23 01:18:36.863678971 (104h55m12.427001984s) [upload=83962]: Waiting for error group managing segments... Jun 23 01:18:36.873250869 (104h55m12.436573877s) [noise=true , address=77.90.31.18:4825 , piece_id=DIQKUPGRMOVTGWHU5TVS4O2DXIBO74SKVFU3WHJN5PDU4R4XZI7Q, seg_pos={0 0}, upload=83962]: Done uploading piece. err: <nil> Jun 23 01:18:36.896075025 (104h55m12.459398017s) [noise=true , address=77.90.31.14:4695 , piece_id=RLOHBVSNWCGZU4YE2HNCBKV6V7BY7JDCDKJFWCM4YYFH5EZXWCFQ, seg_pos={0 0}, upload=83962]: Done uploading piece. err: <nil> Jun 23 01:18:36.930177382 (104h55m12.493500388s) [noise=true , address=104.36.82.14:6228 , piece_id=QMSNWZWKIZ4DQPXOZJC67HOTDUBWJNKY45M5QZF7I7ENC4TZE2VA, seg_pos={0 0}, upload=83962]: Done uploading piece. err: ecclient: upload failed (node:12pkADNFaYa2SqFALQdxwf1CfR1ktUwshQkw7iBJzhdhaRYaszF, address:104.36.82.14:6228): protocol: storage node overloaded, request limit: 2000 Jun 23 01:18:36.953640851 (104h55m12.51696386s) [noise=true , address=104.36.82.20:6444 , piece_id=CWBFFDJCLPZFFSP7NVOKELPXCR3T3DV43FGSSWNGJO6W2IHXEJ3A, seg_pos={0 0}, upload=83962]: Done uploading piece. err: ecclient: upload failed (node:12m5PditUsvMEejqnuEBZ6jLunbkTw9dEkmGzLcBg4y2XAwhBTD, address:104.36.82.20:6444): protocol: storage node overloaded, request limit: 2000 Jun 23 01:18:37.109817807 (104h55m12.673140818s) [noise=true , address=104.36.82.22:6530 , piece_id=ZHRUIZ5BZCM67DGFQZS6NI2DV732UCVDYV3GM55EISACHUWN5ZLQ, seg_pos={0 0}, upload=83962]: Done uploading piece. err: ecclient: upload failed (node:1akHUSoQFafzkGTJ385iYaowc87pJBDNfU5hodSCUDFa5ojN3v, address:104.36.82.22:6530): protocol: storage node overloaded, request limit: 2000 Jun 23 01:18:37.110741515 (104h55m12.674064529s) [noise=true , address=79.127.205.234:5591 , piece_id=U2VCI7JO4IAU4B3LXOGMLQT452CIP2LLT2T53HBQFCR6U6UDQ2RA, seg_pos={0 0}, upload=83962]: Done uploading piece. err: ecclient: upload failed (node:1ZK9ZUTAh12ywDRBtr54pdiWZi5WwkHXTmZkgoh6SpwvLAJTRi, address:79.127.205.234:5591): protocol: storage node overloaded, request limit: 2000 Jun 23 01:18:37.860754351 (104h55m13.424077339s) [noise=true , address=104.36.82.7:5933 , piece_id=Q27EIDAGLD5IZGDJ6QBB3ZA5ULFSIPXY4JWFTNBMLX346CK2ADBQ, seg_pos={0 0}, upload=83962]: Done uploading piece. err: <nil> Jun 23 01:18:38.383822537 (104h55m13.947145526s) [noise=true , address=45.91.116.8:5050 , piece_id=6HAF64M3236TINYRYVIYSG27BF2QW6XGCIKE6SBVVZ527LFOP2ZQ, seg_pos={0 0}, upload=83962]: Done uploading piece. err: <nil> Jun 23 01:18:38.848556195 (104h55m14.411879183s) [noise=true , address=104.36.82.34:6569 , piece_id=EGMJEZMPGKMR5JHGP3TH3MJ7SJ5XWVMTVXCAFDX4RKPIVVMP4BAA, seg_pos={0 0}, upload=83962]: Done uploading piece. err: <nil> Jun 23 01:18:38.877262383 (104h55m14.440585376s) [noise=true , address=104.36.82.51:4084 , piece_id=DODGPPX2WWL6OWRP56V6QX2JZ2PMGHXG3AEQEBRACLAHLWGJTJWA, seg_pos={0 0}, upload=83962]: Done uploading piece. err: <nil> Jun 23 01:18:39.590142305 (104h55m15.15346531s) [noise=true , address=45.91.116.9:5277 , piece_id=U7K7GEX3GPXNULZP35E65E52YL2BUNEXEKTRRZUP6GAALWWPS6KA, seg_pos={0 0}, upload=83962]: Done uploading piece. err: <nil> Jun 23 01:18:41.397250467 (104h55m16.960573472s) [noise=true , address=104.36.82.55:4225 , piece_id=M77UT6JBGUAVSXFMDUMYPRQWN4X7TOSX35LIWATYKMDXE6CSYPDA, seg_pos={0 0}, upload=83962]: Done uploading piece. err: <nil> Jun 23 01:18:41.578659382 (104h55m17.141982372s) [noise=true , address=104.36.82.15:6286 , piece_id=ETSDIEVDHOYGJC62M3M6JUQQLZL5PWPYRLUAVWHU7HE7FGX4YE6Q, seg_pos={0 0}, upload=83962]: Done uploading piece. err: <nil> Jun 23 01:18:44.086228467 (104h55m19.649551478s) [noise=true , address=79.127.219.38:4026 , piece_id=EQUPNI4CUJP6MTH6MODLH6M5YJ4ZNO5ALN6GPSX22RNBCRRIPUOQ, seg_pos={0 0}, upload=83962]: Done uploading piece. err: <nil> Jun 23 01:18:44.523443200 (104h55m20.086766195s) [noise=true , address=77.90.31.10:4535 , piece_id=BORXDT7QJFNH6WJR6GEMRXATEOJFLESKT5XMWY6UOER6R2RW6KBQ, seg_pos={0 0}, upload=83962]: Done uploading piece. err: <nil> Jun 23 01:18:46.233887646 (104h55m21.797210646s) [noise=true , address=104.36.82.56:4245 , piece_id=CGUQ357JLAQJCIDNQ23BNV4Q3R4RGVJI2Q5Z7B56LJKZIFENJM3Q, seg_pos={0 0}, upload=83962]: Done uploading piece. err: <nil> Jun 23 01:19:21.357923809 (104h55m56.921246809s) [noise=true , address=104.36.82.54:4157 , piece_id=ESGS5AM5NBIEFUH7GGK7QV3PIQDMBLIKLIIGEBO7TJML72TF4EAQ, seg_pos={0 0}, upload=83962]: Done uploading piece. err: <nil> Jun 23 01:19:36.705243994 (104h56m12.268566993s) [noise=true , address=104.36.82.53:4128 , piece_id=YZHGLRKEISP3W4KUK3LL3EQEJUMX2LFPIDX2O676LS3GY3PNZ4TA, seg_pos={0 0}, upload=83962]: Done uploading piece. err: ecclient: upload failed (node:125ZMc6ed9qgBRtBC5N4zNvGonip3GNAEBPt3auF6dBGteX46Pf, address:104.36.82.53:4128): protocol: expected piece hash; write tcp 159.223.143.39:39324->104.36.82.53:4128: use of closed network connection; write tcp 159.223.143.39:39324->104.36.82.53:4128: use of closed network connection; EOF Jun 23 01:19:36.705278262 (104h56m12.26860127s) [seg_pos={0 0}, upload=83962]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jun 23 01:19:36.728720635 (104h56m12.292043639s) [upload=83962]: Done uploading segments... Jun 23 01:19:36.728814444 (104h56m12.292137449s) [upload=83962]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 7.017119148s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [......................X.....................X...............] duration: 13.300493072s FAIL (2025-06-23 01:27:24 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 13.300511878s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [...................X.............................X..........] duration: 15.87952571s FAIL (2025-06-23 01:27:18 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== nil> Jun 23 01:18:36.741346057 (104h55m12.304669068s) [upload=83962]: Starting upload Jun 23 01:18:36.741457026 (104h55m12.304780037s) [upload=83962]: Done starting upload Jun 23 01:18:36.741574186 (104h55m12.304897176s) [upload=83962]: Uploading segments... Jun 23 01:18:36.741945356 (104h55m12.305268357s) [upload=83962]: Got next segment. Inline: false Jun 23 01:18:36.742017220 (104h55m12.305340209s) [upload=83962]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 23 01:18:36.742022591 (104h55m12.305345591s) [upload=83962]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 23 01:18:36.777635436 (104h55m12.34095844s) [seg_pos={0 0}, upload=83962]: Begin upload segment... Jun 23 01:18:36.777700720 (104h55m12.341023723s) [seg_pos={0 0}, upload=83962]: Done begin upload segment. Jun 23 01:18:36.777707373 (104h55m12.341030382s) [noise=true , address=104.36.82.34:6569 , piece_id=EGMJEZMPGKMR5JHGP3TH3MJ7SJ5XWVMTVXCAFDX4RKPIVVMP4BAA, seg_pos={0 0}, upload=83962]: Uploading piece... Jun 23 01:18:36.777724726 (104h55m12.341047737s) [noise=true , address=77.90.31.10:4535 , piece_id=BORXDT7QJFNH6WJR6GEMRXATEOJFLESKT5XMWY6UOER6R2RW6KBQ, seg_pos={0 0}, upload=83962]: Uploading piece... Jun 23 01:18:36.777804915 (104h55m12.341127924s) [noise=true , address=45.91.116.9:5277 , piece_id=U7K7GEX3GPXNULZP35E65E52YL2BUNEXEKTRRZUP6GAALWWPS6KA, seg_pos={0 0}, upload=83962]: Uploading piece... Jun 23 01:18:36.778210171 (104h55m12.341533182s) [noise=true , address=104.36.82.53:4128 , piece_id=YZHGLRKEISP3W4KUK3LL3EQEJUMX2LFPIDX2O676LS3GY3PNZ4TA, seg_pos={0 0}, upload=83962]: Uploading piece... Jun 23 01:18:36.778614738 (104h55m12.341937746s) [noise=true , address=104.36.82.56:4245 , piece_id=CGUQ357JLAQJCIDNQ23BNV4Q3R4RGVJI2Q5Z7B56LJKZIFENJM3Q, seg_pos={0 0}, upload=83962]: Uploading piece... Jun 23 01:18:36.778987954 (104h55m12.342310963s) [noise=true , address=104.36.82.7:5933 , piece_id=Q27EIDAGLD5IZGDJ6QBB3ZA5ULFSIPXY4JWFTNBMLX346CK2ADBQ, seg_pos={0 0}, upload=83962]: Uploading piece... Jun 23 01:18:36.779391219 (104h55m12.342714228s) [noise=true , address=104.36.82.54:4157 , piece_id=ESGS5AM5NBIEFUH7GGK7QV3PIQDMBLIKLIIGEBO7TJML72TF4EAQ, seg_pos={0 0}, upload=83962]: Uploading piece... Jun 23 01:18:36.779521731 (104h55m12.342844734s) [noise=true , address=104.36.82.20:6444 , piece_id=CWBFFDJCLPZFFSP7NVOKELPXCR3T3DV43FGSSWNGJO6W2IHXEJ3A, seg_pos={0 0}, upload=83962]: Uploading piece... Jun 23 01:18:36.779962650 (104h55m12.34328566s) [noise=true , address=104.36.82.14:6228 , piece_id=QMSNWZWKIZ4DQPXOZJC67HOTDUBWJNKY45M5QZF7I7ENC4TZE2VA, seg_pos={0 0}, upload=83962]: Uploading piece... Jun 23 01:18:36.780465195 (104h55m12.343788212s) [noise=true , address=77.90.31.18:4825 , piece_id=DIQKUPGRMOVTGWHU5TVS4O2DXIBO74SKVFU3WHJN5PDU4R4XZI7Q, seg_pos={0 0}, upload=83962]: Uploading piece... Jun 23 01:18:36.780615746 (104h55m12.343938757s) [noise=true , address=104.36.82.22:6530 , piece_id=ZHRUIZ5BZCM67DGFQZS6NI2DV732UCVDYV3GM55EISACHUWN5ZLQ, seg_pos={0 0}, upload=83962]: Uploading piece... Jun 23 01:18:36.781165217 (104h55m12.344488225s) [noise=true , address=104.36.82.51:4084 , piece_id=DODGPPX2WWL6OWRP56V6QX2JZ2PMGHXG3AEQEBRACLAHLWGJTJWA, seg_pos={0 0}, upload=83962]: Uploading piece... Jun 23 01:18:36.781550748 (104h55m12.344873758s) [noise=true , address=79.127.205.234:5591 , piece_id=U2VCI7JO4IAU4B3LXOGMLQT452CIP2LLT2T53HBQFCR6U6UDQ2RA, seg_pos={0 0}, upload=83962]: Uploading piece... Jun 23 01:18:36.782014436 (104h55m12.345337441s) [noise=true , address=45.91.116.8:5050 , piece_id=6HAF64M3236TINYRYVIYSG27BF2QW6XGCIKE6SBVVZ527LFOP2ZQ, seg_pos={0 0}, upload=83962]: Uploading piece... Jun 23 01:18:36.782496184 (104h55m12.345819196s) [noise=true , address=104.36.82.15:6286 , piece_id=ETSDIEVDHOYGJC62M3M6JUQQLZL5PWPYRLUAVWHU7HE7FGX4YE6Q, seg_pos={0 0}, upload=83962]: Uploading piece... Jun 23 01:18:36.782881896 (104h55m12.346204905s) [noise=true , address=79.127.219.38:4026 , piece_id=EQUPNI4CUJP6MTH6MODLH6M5YJ4ZNO5ALN6GPSX22RNBCRRIPUOQ, seg_pos={0 0}, upload=83962]: Uploading piece... Jun 23 01:18:36.783409077 (104h55m12.346732086s) [noise=true , address=104.36.82.55:4225 , piece_id=M77UT6JBGUAVSXFMDUMYPRQWN4X7TOSX35LIWATYKMDXE6CSYPDA, seg_pos={0 0}, upload=83962]: Uploading piece... Jun 23 01:18:36.783797923 (104h55m12.34712093s) [noise=true , address=77.90.31.14:4695 , piece_id=RLOHBVSNWCGZU4YE2HNCBKV6V7BY7JDCDKJFWCM4YYFH5EZXWCFQ, seg_pos={0 0}, upload=83962]: Uploading piece... Jun 23 01:18:36.863659010 (104h55m12.426982006s) [upload=83962]: Next returned nil segment Jun 23 01:18:36.863678971 (104h55m12.427001984s) [upload=83962]: Waiting for error group managing segments... Jun 23 01:18:36.873250869 (104h55m12.436573877s) [noise=true , address=77.90.31.18:4825 , piece_id=DIQKUPGRMOVTGWHU5TVS4O2DXIBO74SKVFU3WHJN5PDU4R4XZI7Q, seg_pos={0 0}, upload=83962]: Done uploading piece. err: <nil> Jun 23 01:18:36.896075025 (104h55m12.459398017s) [noise=true , address=77.90.31.14:4695 , piece_id=RLOHBVSNWCGZU4YE2HNCBKV6V7BY7JDCDKJFWCM4YYFH5EZXWCFQ, seg_pos={0 0}, upload=83962]: Done uploading piece. err: <nil> Jun 23 01:18:36.930177382 (104h55m12.493500388s) [noise=true , address=104.36.82.14:6228 , piece_id=QMSNWZWKIZ4DQPXOZJC67HOTDUBWJNKY45M5QZF7I7ENC4TZE2VA, seg_pos={0 0}, upload=83962]: Done uploading piece. err: ecclient: upload failed (node:12pkADNFaYa2SqFALQdxwf1CfR1ktUwshQkw7iBJzhdhaRYaszF, address:104.36.82.14:6228): protocol: storage node overloaded, request limit: 2000 Jun 23 01:18:36.953640851 (104h55m12.51696386s) [noise=true , address=104.36.82.20:6444 , piece_id=CWBFFDJCLPZFFSP7NVOKELPXCR3T3DV43FGSSWNGJO6W2IHXEJ3A, seg_pos={0 0}, upload=83962]: Done uploading piece. err: ecclient: upload failed (node:12m5PditUsvMEejqnuEBZ6jLunbkTw9dEkmGzLcBg4y2XAwhBTD, address:104.36.82.20:6444): protocol: storage node overloaded, request limit: 2000 Jun 23 01:18:37.109817807 (104h55m12.673140818s) [noise=true , address=104.36.82.22:6530 , piece_id=ZHRUIZ5BZCM67DGFQZS6NI2DV732UCVDYV3GM55EISACHUWN5ZLQ, seg_pos={0 0}, upload=83962]: Done uploading piece. err: ecclient: upload failed (node:1akHUSoQFafzkGTJ385iYaowc87pJBDNfU5hodSCUDFa5ojN3v, address:104.36.82.22:6530): protocol: storage node overloaded, request limit: 2000 Jun 23 01:18:37.110741515 (104h55m12.674064529s) [noise=true , address=79.127.205.234:5591 , piece_id=U2VCI7JO4IAU4B3LXOGMLQT452CIP2LLT2T53HBQFCR6U6UDQ2RA, seg_pos={0 0}, upload=83962]: Done uploading piece. err: ecclient: upload failed (node:1ZK9ZUTAh12ywDRBtr54pdiWZi5WwkHXTmZkgoh6SpwvLAJTRi, address:79.127.205.234:5591): protocol: storage node overloaded, request limit: 2000 Jun 23 01:18:37.860754351 (104h55m13.424077339s) [noise=true , address=104.36.82.7:5933 , piece_id=Q27EIDAGLD5IZGDJ6QBB3ZA5ULFSIPXY4JWFTNBMLX346CK2ADBQ, seg_pos={0 0}, upload=83962]: Done uploading piece. err: <nil> Jun 23 01:18:38.383822537 (104h55m13.947145526s) [noise=true , address=45.91.116.8:5050 , piece_id=6HAF64M3236TINYRYVIYSG27BF2QW6XGCIKE6SBVVZ527LFOP2ZQ, seg_pos={0 0}, upload=83962]: Done uploading piece. err: <nil> Jun 23 01:18:38.848556195 (104h55m14.411879183s) [noise=true , address=104.36.82.34:6569 , piece_id=EGMJEZMPGKMR5JHGP3TH3MJ7SJ5XWVMTVXCAFDX4RKPIVVMP4BAA, seg_pos={0 0}, upload=83962]: Done uploading piece. err: <nil> Jun 23 01:18:38.877262383 (104h55m14.440585376s) [noise=true , address=104.36.82.51:4084 , piece_id=DODGPPX2WWL6OWRP56V6QX2JZ2PMGHXG3AEQEBRACLAHLWGJTJWA, seg_pos={0 0}, upload=83962]: Done uploading piece. err: <nil> Jun 23 01:18:39.590142305 (104h55m15.15346531s) [noise=true , address=45.91.116.9:5277 , piece_id=U7K7GEX3GPXNULZP35E65E52YL2BUNEXEKTRRZUP6GAALWWPS6KA, seg_pos={0 0}, upload=83962]: Done uploading piece. err: <nil> Jun 23 01:18:41.397250467 (104h55m16.960573472s) [noise=true , address=104.36.82.55:4225 , piece_id=M77UT6JBGUAVSXFMDUMYPRQWN4X7TOSX35LIWATYKMDXE6CSYPDA, seg_pos={0 0}, upload=83962]: Done uploading piece. err: <nil> Jun 23 01:18:41.578659382 (104h55m17.141982372s) [noise=true , address=104.36.82.15:6286 , piece_id=ETSDIEVDHOYGJC62M3M6JUQQLZL5PWPYRLUAVWHU7HE7FGX4YE6Q, seg_pos={0 0}, upload=83962]: Done uploading piece. err: <nil> Jun 23 01:18:44.086228467 (104h55m19.649551478s) [noise=true , address=79.127.219.38:4026 , piece_id=EQUPNI4CUJP6MTH6MODLH6M5YJ4ZNO5ALN6GPSX22RNBCRRIPUOQ, seg_pos={0 0}, upload=83962]: Done uploading piece. err: <nil> Jun 23 01:18:44.523443200 (104h55m20.086766195s) [noise=true , address=77.90.31.10:4535 , piece_id=BORXDT7QJFNH6WJR6GEMRXATEOJFLESKT5XMWY6UOER6R2RW6KBQ, seg_pos={0 0}, upload=83962]: Done uploading piece. err: <nil> Jun 23 01:18:46.233887646 (104h55m21.797210646s) [noise=true , address=104.36.82.56:4245 , piece_id=CGUQ357JLAQJCIDNQ23BNV4Q3R4RGVJI2Q5Z7B56LJKZIFENJM3Q, seg_pos={0 0}, upload=83962]: Done uploading piece. err: <nil> Jun 23 01:19:21.357923809 (104h55m56.921246809s) [noise=true , address=104.36.82.54:4157 , piece_id=ESGS5AM5NBIEFUH7GGK7QV3PIQDMBLIKLIIGEBO7TJML72TF4EAQ, seg_pos={0 0}, upload=83962]: Done uploading piece. err: <nil> Jun 23 01:19:36.705243994 (104h56m12.268566993s) [noise=true , address=104.36.82.53:4128 , piece_id=YZHGLRKEISP3W4KUK3LL3EQEJUMX2LFPIDX2O676LS3GY3PNZ4TA, seg_pos={0 0}, upload=83962]: Done uploading piece. err: ecclient: upload failed (node:125ZMc6ed9qgBRtBC5N4zNvGonip3GNAEBPt3auF6dBGteX46Pf, address:104.36.82.53:4128): protocol: expected piece hash; write tcp 159.223.143.39:39324->104.36.82.53:4128: use of closed network connection; write tcp 159.223.143.39:39324->104.36.82.53:4128: use of closed network connection; EOF Jun 23 01:19:36.705278262 (104h56m12.26860127s) [seg_pos={0 0}, upload=83962]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jun 23 01:19:36.728720635 (104h56m12.292043639s) [upload=83962]: Done uploading segments... Jun 23 01:19:36.728814444 (104h56m12.292137449s) [upload=83962]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 15.879662155s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [.......................X.....................X..............] duration: 53.136216159s FAIL (2025-06-23 01:27:11 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 53.136246483s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [....................X.............................X.........] duration: 25.989511292s FAIL (2025-06-23 01:26:41 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== nil> Jun 23 01:18:36.741346057 (104h55m12.304669068s) [upload=83962]: Starting upload Jun 23 01:18:36.741457026 (104h55m12.304780037s) [upload=83962]: Done starting upload Jun 23 01:18:36.741574186 (104h55m12.304897176s) [upload=83962]: Uploading segments... Jun 23 01:18:36.741945356 (104h55m12.305268357s) [upload=83962]: Got next segment. Inline: false Jun 23 01:18:36.742017220 (104h55m12.305340209s) [upload=83962]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 23 01:18:36.742022591 (104h55m12.305345591s) [upload=83962]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 23 01:18:36.777635436 (104h55m12.34095844s) [seg_pos={0 0}, upload=83962]: Begin upload segment... Jun 23 01:18:36.777700720 (104h55m12.341023723s) [seg_pos={0 0}, upload=83962]: Done begin upload segment. Jun 23 01:18:36.777707373 (104h55m12.341030382s) [noise=true , address=104.36.82.34:6569 , piece_id=EGMJEZMPGKMR5JHGP3TH3MJ7SJ5XWVMTVXCAFDX4RKPIVVMP4BAA, seg_pos={0 0}, upload=83962]: Uploading piece... Jun 23 01:18:36.777724726 (104h55m12.341047737s) [noise=true , address=77.90.31.10:4535 , piece_id=BORXDT7QJFNH6WJR6GEMRXATEOJFLESKT5XMWY6UOER6R2RW6KBQ, seg_pos={0 0}, upload=83962]: Uploading piece... Jun 23 01:18:36.777804915 (104h55m12.341127924s) [noise=true , address=45.91.116.9:5277 , piece_id=U7K7GEX3GPXNULZP35E65E52YL2BUNEXEKTRRZUP6GAALWWPS6KA, seg_pos={0 0}, upload=83962]: Uploading piece... Jun 23 01:18:36.778210171 (104h55m12.341533182s) [noise=true , address=104.36.82.53:4128 , piece_id=YZHGLRKEISP3W4KUK3LL3EQEJUMX2LFPIDX2O676LS3GY3PNZ4TA, seg_pos={0 0}, upload=83962]: Uploading piece... Jun 23 01:18:36.778614738 (104h55m12.341937746s) [noise=true , address=104.36.82.56:4245 , piece_id=CGUQ357JLAQJCIDNQ23BNV4Q3R4RGVJI2Q5Z7B56LJKZIFENJM3Q, seg_pos={0 0}, upload=83962]: Uploading piece... Jun 23 01:18:36.778987954 (104h55m12.342310963s) [noise=true , address=104.36.82.7:5933 , piece_id=Q27EIDAGLD5IZGDJ6QBB3ZA5ULFSIPXY4JWFTNBMLX346CK2ADBQ, seg_pos={0 0}, upload=83962]: Uploading piece... Jun 23 01:18:36.779391219 (104h55m12.342714228s) [noise=true , address=104.36.82.54:4157 , piece_id=ESGS5AM5NBIEFUH7GGK7QV3PIQDMBLIKLIIGEBO7TJML72TF4EAQ, seg_pos={0 0}, upload=83962]: Uploading piece... Jun 23 01:18:36.779521731 (104h55m12.342844734s) [noise=true , address=104.36.82.20:6444 , piece_id=CWBFFDJCLPZFFSP7NVOKELPXCR3T3DV43FGSSWNGJO6W2IHXEJ3A, seg_pos={0 0}, upload=83962]: Uploading piece... Jun 23 01:18:36.779962650 (104h55m12.34328566s) [noise=true , address=104.36.82.14:6228 , piece_id=QMSNWZWKIZ4DQPXOZJC67HOTDUBWJNKY45M5QZF7I7ENC4TZE2VA, seg_pos={0 0}, upload=83962]: Uploading piece... Jun 23 01:18:36.780465195 (104h55m12.343788212s) [noise=true , address=77.90.31.18:4825 , piece_id=DIQKUPGRMOVTGWHU5TVS4O2DXIBO74SKVFU3WHJN5PDU4R4XZI7Q, seg_pos={0 0}, upload=83962]: Uploading piece... Jun 23 01:18:36.780615746 (104h55m12.343938757s) [noise=true , address=104.36.82.22:6530 , piece_id=ZHRUIZ5BZCM67DGFQZS6NI2DV732UCVDYV3GM55EISACHUWN5ZLQ, seg_pos={0 0}, upload=83962]: Uploading piece... Jun 23 01:18:36.781165217 (104h55m12.344488225s) [noise=true , address=104.36.82.51:4084 , piece_id=DODGPPX2WWL6OWRP56V6QX2JZ2PMGHXG3AEQEBRACLAHLWGJTJWA, seg_pos={0 0}, upload=83962]: Uploading piece... Jun 23 01:18:36.781550748 (104h55m12.344873758s) [noise=true , address=79.127.205.234:5591 , piece_id=U2VCI7JO4IAU4B3LXOGMLQT452CIP2LLT2T53HBQFCR6U6UDQ2RA, seg_pos={0 0}, upload=83962]: Uploading piece... Jun 23 01:18:36.782014436 (104h55m12.345337441s) [noise=true , address=45.91.116.8:5050 , piece_id=6HAF64M3236TINYRYVIYSG27BF2QW6XGCIKE6SBVVZ527LFOP2ZQ, seg_pos={0 0}, upload=83962]: Uploading piece... Jun 23 01:18:36.782496184 (104h55m12.345819196s) [noise=true , address=104.36.82.15:6286 , piece_id=ETSDIEVDHOYGJC62M3M6JUQQLZL5PWPYRLUAVWHU7HE7FGX4YE6Q, seg_pos={0 0}, upload=83962]: Uploading piece... Jun 23 01:18:36.782881896 (104h55m12.346204905s) [noise=true , address=79.127.219.38:4026 , piece_id=EQUPNI4CUJP6MTH6MODLH6M5YJ4ZNO5ALN6GPSX22RNBCRRIPUOQ, seg_pos={0 0}, upload=83962]: Uploading piece... Jun 23 01:18:36.783409077 (104h55m12.346732086s) [noise=true , address=104.36.82.55:4225 , piece_id=M77UT6JBGUAVSXFMDUMYPRQWN4X7TOSX35LIWATYKMDXE6CSYPDA, seg_pos={0 0}, upload=83962]: Uploading piece... Jun 23 01:18:36.783797923 (104h55m12.34712093s) [noise=true , address=77.90.31.14:4695 , piece_id=RLOHBVSNWCGZU4YE2HNCBKV6V7BY7JDCDKJFWCM4YYFH5EZXWCFQ, seg_pos={0 0}, upload=83962]: Uploading piece... Jun 23 01:18:36.863659010 (104h55m12.426982006s) [upload=83962]: Next returned nil segment Jun 23 01:18:36.863678971 (104h55m12.427001984s) [upload=83962]: Waiting for error group managing segments... Jun 23 01:18:36.873250869 (104h55m12.436573877s) [noise=true , address=77.90.31.18:4825 , piece_id=DIQKUPGRMOVTGWHU5TVS4O2DXIBO74SKVFU3WHJN5PDU4R4XZI7Q, seg_pos={0 0}, upload=83962]: Done uploading piece. err: <nil> Jun 23 01:18:36.896075025 (104h55m12.459398017s) [noise=true , address=77.90.31.14:4695 , piece_id=RLOHBVSNWCGZU4YE2HNCBKV6V7BY7JDCDKJFWCM4YYFH5EZXWCFQ, seg_pos={0 0}, upload=83962]: Done uploading piece. err: <nil> Jun 23 01:18:36.930177382 (104h55m12.493500388s) [noise=true , address=104.36.82.14:6228 , piece_id=QMSNWZWKIZ4DQPXOZJC67HOTDUBWJNKY45M5QZF7I7ENC4TZE2VA, seg_pos={0 0}, upload=83962]: Done uploading piece. err: ecclient: upload failed (node:12pkADNFaYa2SqFALQdxwf1CfR1ktUwshQkw7iBJzhdhaRYaszF, address:104.36.82.14:6228): protocol: storage node overloaded, request limit: 2000 Jun 23 01:18:36.953640851 (104h55m12.51696386s) [noise=true , address=104.36.82.20:6444 , piece_id=CWBFFDJCLPZFFSP7NVOKELPXCR3T3DV43FGSSWNGJO6W2IHXEJ3A, seg_pos={0 0}, upload=83962]: Done uploading piece. err: ecclient: upload failed (node:12m5PditUsvMEejqnuEBZ6jLunbkTw9dEkmGzLcBg4y2XAwhBTD, address:104.36.82.20:6444): protocol: storage node overloaded, request limit: 2000 Jun 23 01:18:37.109817807 (104h55m12.673140818s) [noise=true , address=104.36.82.22:6530 , piece_id=ZHRUIZ5BZCM67DGFQZS6NI2DV732UCVDYV3GM55EISACHUWN5ZLQ, seg_pos={0 0}, upload=83962]: Done uploading piece. err: ecclient: upload failed (node:1akHUSoQFafzkGTJ385iYaowc87pJBDNfU5hodSCUDFa5ojN3v, address:104.36.82.22:6530): protocol: storage node overloaded, request limit: 2000 Jun 23 01:18:37.110741515 (104h55m12.674064529s) [noise=true , address=79.127.205.234:5591 , piece_id=U2VCI7JO4IAU4B3LXOGMLQT452CIP2LLT2T53HBQFCR6U6UDQ2RA, seg_pos={0 0}, upload=83962]: Done uploading piece. err: ecclient: upload failed (node:1ZK9ZUTAh12ywDRBtr54pdiWZi5WwkHXTmZkgoh6SpwvLAJTRi, address:79.127.205.234:5591): protocol: storage node overloaded, request limit: 2000 Jun 23 01:18:37.860754351 (104h55m13.424077339s) [noise=true , address=104.36.82.7:5933 , piece_id=Q27EIDAGLD5IZGDJ6QBB3ZA5ULFSIPXY4JWFTNBMLX346CK2ADBQ, seg_pos={0 0}, upload=83962]: Done uploading piece. err: <nil> Jun 23 01:18:38.383822537 (104h55m13.947145526s) [noise=true , address=45.91.116.8:5050 , piece_id=6HAF64M3236TINYRYVIYSG27BF2QW6XGCIKE6SBVVZ527LFOP2ZQ, seg_pos={0 0}, upload=83962]: Done uploading piece. err: <nil> Jun 23 01:18:38.848556195 (104h55m14.411879183s) [noise=true , address=104.36.82.34:6569 , piece_id=EGMJEZMPGKMR5JHGP3TH3MJ7SJ5XWVMTVXCAFDX4RKPIVVMP4BAA, seg_pos={0 0}, upload=83962]: Done uploading piece. err: <nil> Jun 23 01:18:38.877262383 (104h55m14.440585376s) [noise=true , address=104.36.82.51:4084 , piece_id=DODGPPX2WWL6OWRP56V6QX2JZ2PMGHXG3AEQEBRACLAHLWGJTJWA, seg_pos={0 0}, upload=83962]: Done uploading piece. err: <nil> Jun 23 01:18:39.590142305 (104h55m15.15346531s) [noise=true , address=45.91.116.9:5277 , piece_id=U7K7GEX3GPXNULZP35E65E52YL2BUNEXEKTRRZUP6GAALWWPS6KA, seg_pos={0 0}, upload=83962]: Done uploading piece. err: <nil> Jun 23 01:18:41.397250467 (104h55m16.960573472s) [noise=true , address=104.36.82.55:4225 , piece_id=M77UT6JBGUAVSXFMDUMYPRQWN4X7TOSX35LIWATYKMDXE6CSYPDA, seg_pos={0 0}, upload=83962]: Done uploading piece. err: <nil> Jun 23 01:18:41.578659382 (104h55m17.141982372s) [noise=true , address=104.36.82.15:6286 , piece_id=ETSDIEVDHOYGJC62M3M6JUQQLZL5PWPYRLUAVWHU7HE7FGX4YE6Q, seg_pos={0 0}, upload=83962]: Done uploading piece. err: <nil> Jun 23 01:18:44.086228467 (104h55m19.649551478s) [noise=true , address=79.127.219.38:4026 , piece_id=EQUPNI4CUJP6MTH6MODLH6M5YJ4ZNO5ALN6GPSX22RNBCRRIPUOQ, seg_pos={0 0}, upload=83962]: Done uploading piece. err: <nil> Jun 23 01:18:44.523443200 (104h55m20.086766195s) [noise=true , address=77.90.31.10:4535 , piece_id=BORXDT7QJFNH6WJR6GEMRXATEOJFLESKT5XMWY6UOER6R2RW6KBQ, seg_pos={0 0}, upload=83962]: Done uploading piece. err: <nil> Jun 23 01:18:46.233887646 (104h55m21.797210646s) [noise=true , address=104.36.82.56:4245 , piece_id=CGUQ357JLAQJCIDNQ23BNV4Q3R4RGVJI2Q5Z7B56LJKZIFENJM3Q, seg_pos={0 0}, upload=83962]: Done uploading piece. err: <nil> Jun 23 01:19:21.357923809 (104h55m56.921246809s) [noise=true , address=104.36.82.54:4157 , piece_id=ESGS5AM5NBIEFUH7GGK7QV3PIQDMBLIKLIIGEBO7TJML72TF4EAQ, seg_pos={0 0}, upload=83962]: Done uploading piece. err: <nil> Jun 23 01:19:36.705243994 (104h56m12.268566993s) [noise=true , address=104.36.82.53:4128 , piece_id=YZHGLRKEISP3W4KUK3LL3EQEJUMX2LFPIDX2O676LS3GY3PNZ4TA, seg_pos={0 0}, upload=83962]: Done uploading piece. err: ecclient: upload failed (node:125ZMc6ed9qgBRtBC5N4zNvGonip3GNAEBPt3auF6dBGteX46Pf, address:104.36.82.53:4128): protocol: expected piece hash; write tcp 159.223.143.39:39324->104.36.82.53:4128: use of closed network connection; write tcp 159.223.143.39:39324->104.36.82.53:4128: use of closed network connection; EOF Jun 23 01:19:36.705278262 (104h56m12.26860127s) [seg_pos={0 0}, upload=83962]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jun 23 01:19:36.728720635 (104h56m12.292043639s) [upload=83962]: Done uploading segments... Jun 23 01:19:36.728814444 (104h56m12.292137449s) [upload=83962]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 25.989544036s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [........................X.....................X.............] duration: 21.378961427s FAIL (2025-06-23 01:25:57 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 21.378975969s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [.........................X.....................X............] duration: 24.088960219s FAIL (2025-06-23 01:25:36 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 24.088989067s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [.....................X.............................X........] duration: 26.620760794s FAIL (2025-06-23 01:25:33 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== nil> Jun 23 01:18:36.741346057 (104h55m12.304669068s) [upload=83962]: Starting upload Jun 23 01:18:36.741457026 (104h55m12.304780037s) [upload=83962]: Done starting upload Jun 23 01:18:36.741574186 (104h55m12.304897176s) [upload=83962]: Uploading segments... Jun 23 01:18:36.741945356 (104h55m12.305268357s) [upload=83962]: Got next segment. Inline: false Jun 23 01:18:36.742017220 (104h55m12.305340209s) [upload=83962]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 23 01:18:36.742022591 (104h55m12.305345591s) [upload=83962]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 23 01:18:36.777635436 (104h55m12.34095844s) [seg_pos={0 0}, upload=83962]: Begin upload segment... Jun 23 01:18:36.777700720 (104h55m12.341023723s) [seg_pos={0 0}, upload=83962]: Done begin upload segment. Jun 23 01:18:36.777707373 (104h55m12.341030382s) [noise=true , address=104.36.82.34:6569 , piece_id=EGMJEZMPGKMR5JHGP3TH3MJ7SJ5XWVMTVXCAFDX4RKPIVVMP4BAA, seg_pos={0 0}, upload=83962]: Uploading piece... Jun 23 01:18:36.777724726 (104h55m12.341047737s) [noise=true , address=77.90.31.10:4535 , piece_id=BORXDT7QJFNH6WJR6GEMRXATEOJFLESKT5XMWY6UOER6R2RW6KBQ, seg_pos={0 0}, upload=83962]: Uploading piece... Jun 23 01:18:36.777804915 (104h55m12.341127924s) [noise=true , address=45.91.116.9:5277 , piece_id=U7K7GEX3GPXNULZP35E65E52YL2BUNEXEKTRRZUP6GAALWWPS6KA, seg_pos={0 0}, upload=83962]: Uploading piece... Jun 23 01:18:36.778210171 (104h55m12.341533182s) [noise=true , address=104.36.82.53:4128 , piece_id=YZHGLRKEISP3W4KUK3LL3EQEJUMX2LFPIDX2O676LS3GY3PNZ4TA, seg_pos={0 0}, upload=83962]: Uploading piece... Jun 23 01:18:36.778614738 (104h55m12.341937746s) [noise=true , address=104.36.82.56:4245 , piece_id=CGUQ357JLAQJCIDNQ23BNV4Q3R4RGVJI2Q5Z7B56LJKZIFENJM3Q, seg_pos={0 0}, upload=83962]: Uploading piece... Jun 23 01:18:36.778987954 (104h55m12.342310963s) [noise=true , address=104.36.82.7:5933 , piece_id=Q27EIDAGLD5IZGDJ6QBB3ZA5ULFSIPXY4JWFTNBMLX346CK2ADBQ, seg_pos={0 0}, upload=83962]: Uploading piece... Jun 23 01:18:36.779391219 (104h55m12.342714228s) [noise=true , address=104.36.82.54:4157 , piece_id=ESGS5AM5NBIEFUH7GGK7QV3PIQDMBLIKLIIGEBO7TJML72TF4EAQ, seg_pos={0 0}, upload=83962]: Uploading piece... Jun 23 01:18:36.779521731 (104h55m12.342844734s) [noise=true , address=104.36.82.20:6444 , piece_id=CWBFFDJCLPZFFSP7NVOKELPXCR3T3DV43FGSSWNGJO6W2IHXEJ3A, seg_pos={0 0}, upload=83962]: Uploading piece... Jun 23 01:18:36.779962650 (104h55m12.34328566s) [noise=true , address=104.36.82.14:6228 , piece_id=QMSNWZWKIZ4DQPXOZJC67HOTDUBWJNKY45M5QZF7I7ENC4TZE2VA, seg_pos={0 0}, upload=83962]: Uploading piece... Jun 23 01:18:36.780465195 (104h55m12.343788212s) [noise=true , address=77.90.31.18:4825 , piece_id=DIQKUPGRMOVTGWHU5TVS4O2DXIBO74SKVFU3WHJN5PDU4R4XZI7Q, seg_pos={0 0}, upload=83962]: Uploading piece... Jun 23 01:18:36.780615746 (104h55m12.343938757s) [noise=true , address=104.36.82.22:6530 , piece_id=ZHRUIZ5BZCM67DGFQZS6NI2DV732UCVDYV3GM55EISACHUWN5ZLQ, seg_pos={0 0}, upload=83962]: Uploading piece... Jun 23 01:18:36.781165217 (104h55m12.344488225s) [noise=true , address=104.36.82.51:4084 , piece_id=DODGPPX2WWL6OWRP56V6QX2JZ2PMGHXG3AEQEBRACLAHLWGJTJWA, seg_pos={0 0}, upload=83962]: Uploading piece... Jun 23 01:18:36.781550748 (104h55m12.344873758s) [noise=true , address=79.127.205.234:5591 , piece_id=U2VCI7JO4IAU4B3LXOGMLQT452CIP2LLT2T53HBQFCR6U6UDQ2RA, seg_pos={0 0}, upload=83962]: Uploading piece... Jun 23 01:18:36.782014436 (104h55m12.345337441s) [noise=true , address=45.91.116.8:5050 , piece_id=6HAF64M3236TINYRYVIYSG27BF2QW6XGCIKE6SBVVZ527LFOP2ZQ, seg_pos={0 0}, upload=83962]: Uploading piece... Jun 23 01:18:36.782496184 (104h55m12.345819196s) [noise=true , address=104.36.82.15:6286 , piece_id=ETSDIEVDHOYGJC62M3M6JUQQLZL5PWPYRLUAVWHU7HE7FGX4YE6Q, seg_pos={0 0}, upload=83962]: Uploading piece... Jun 23 01:18:36.782881896 (104h55m12.346204905s) [noise=true , address=79.127.219.38:4026 , piece_id=EQUPNI4CUJP6MTH6MODLH6M5YJ4ZNO5ALN6GPSX22RNBCRRIPUOQ, seg_pos={0 0}, upload=83962]: Uploading piece... Jun 23 01:18:36.783409077 (104h55m12.346732086s) [noise=true , address=104.36.82.55:4225 , piece_id=M77UT6JBGUAVSXFMDUMYPRQWN4X7TOSX35LIWATYKMDXE6CSYPDA, seg_pos={0 0}, upload=83962]: Uploading piece... Jun 23 01:18:36.783797923 (104h55m12.34712093s) [noise=true , address=77.90.31.14:4695 , piece_id=RLOHBVSNWCGZU4YE2HNCBKV6V7BY7JDCDKJFWCM4YYFH5EZXWCFQ, seg_pos={0 0}, upload=83962]: Uploading piece... Jun 23 01:18:36.863659010 (104h55m12.426982006s) [upload=83962]: Next returned nil segment Jun 23 01:18:36.863678971 (104h55m12.427001984s) [upload=83962]: Waiting for error group managing segments... Jun 23 01:18:36.873250869 (104h55m12.436573877s) [noise=true , address=77.90.31.18:4825 , piece_id=DIQKUPGRMOVTGWHU5TVS4O2DXIBO74SKVFU3WHJN5PDU4R4XZI7Q, seg_pos={0 0}, upload=83962]: Done uploading piece. err: <nil> Jun 23 01:18:36.896075025 (104h55m12.459398017s) [noise=true , address=77.90.31.14:4695 , piece_id=RLOHBVSNWCGZU4YE2HNCBKV6V7BY7JDCDKJFWCM4YYFH5EZXWCFQ, seg_pos={0 0}, upload=83962]: Done uploading piece. err: <nil> Jun 23 01:18:36.930177382 (104h55m12.493500388s) [noise=true , address=104.36.82.14:6228 , piece_id=QMSNWZWKIZ4DQPXOZJC67HOTDUBWJNKY45M5QZF7I7ENC4TZE2VA, seg_pos={0 0}, upload=83962]: Done uploading piece. err: ecclient: upload failed (node:12pkADNFaYa2SqFALQdxwf1CfR1ktUwshQkw7iBJzhdhaRYaszF, address:104.36.82.14:6228): protocol: storage node overloaded, request limit: 2000 Jun 23 01:18:36.953640851 (104h55m12.51696386s) [noise=true , address=104.36.82.20:6444 , piece_id=CWBFFDJCLPZFFSP7NVOKELPXCR3T3DV43FGSSWNGJO6W2IHXEJ3A, seg_pos={0 0}, upload=83962]: Done uploading piece. err: ecclient: upload failed (node:12m5PditUsvMEejqnuEBZ6jLunbkTw9dEkmGzLcBg4y2XAwhBTD, address:104.36.82.20:6444): protocol: storage node overloaded, request limit: 2000 Jun 23 01:18:37.109817807 (104h55m12.673140818s) [noise=true , address=104.36.82.22:6530 , piece_id=ZHRUIZ5BZCM67DGFQZS6NI2DV732UCVDYV3GM55EISACHUWN5ZLQ, seg_pos={0 0}, upload=83962]: Done uploading piece. err: ecclient: upload failed (node:1akHUSoQFafzkGTJ385iYaowc87pJBDNfU5hodSCUDFa5ojN3v, address:104.36.82.22:6530): protocol: storage node overloaded, request limit: 2000 Jun 23 01:18:37.110741515 (104h55m12.674064529s) [noise=true , address=79.127.205.234:5591 , piece_id=U2VCI7JO4IAU4B3LXOGMLQT452CIP2LLT2T53HBQFCR6U6UDQ2RA, seg_pos={0 0}, upload=83962]: Done uploading piece. err: ecclient: upload failed (node:1ZK9ZUTAh12ywDRBtr54pdiWZi5WwkHXTmZkgoh6SpwvLAJTRi, address:79.127.205.234:5591): protocol: storage node overloaded, request limit: 2000 Jun 23 01:18:37.860754351 (104h55m13.424077339s) [noise=true , address=104.36.82.7:5933 , piece_id=Q27EIDAGLD5IZGDJ6QBB3ZA5ULFSIPXY4JWFTNBMLX346CK2ADBQ, seg_pos={0 0}, upload=83962]: Done uploading piece. err: <nil> Jun 23 01:18:38.383822537 (104h55m13.947145526s) [noise=true , address=45.91.116.8:5050 , piece_id=6HAF64M3236TINYRYVIYSG27BF2QW6XGCIKE6SBVVZ527LFOP2ZQ, seg_pos={0 0}, upload=83962]: Done uploading piece. err: <nil> Jun 23 01:18:38.848556195 (104h55m14.411879183s) [noise=true , address=104.36.82.34:6569 , piece_id=EGMJEZMPGKMR5JHGP3TH3MJ7SJ5XWVMTVXCAFDX4RKPIVVMP4BAA, seg_pos={0 0}, upload=83962]: Done uploading piece. err: <nil> Jun 23 01:18:38.877262383 (104h55m14.440585376s) [noise=true , address=104.36.82.51:4084 , piece_id=DODGPPX2WWL6OWRP56V6QX2JZ2PMGHXG3AEQEBRACLAHLWGJTJWA, seg_pos={0 0}, upload=83962]: Done uploading piece. err: <nil> Jun 23 01:18:39.590142305 (104h55m15.15346531s) [noise=true , address=45.91.116.9:5277 , piece_id=U7K7GEX3GPXNULZP35E65E52YL2BUNEXEKTRRZUP6GAALWWPS6KA, seg_pos={0 0}, upload=83962]: Done uploading piece. err: <nil> Jun 23 01:18:41.397250467 (104h55m16.960573472s) [noise=true , address=104.36.82.55:4225 , piece_id=M77UT6JBGUAVSXFMDUMYPRQWN4X7TOSX35LIWATYKMDXE6CSYPDA, seg_pos={0 0}, upload=83962]: Done uploading piece. err: <nil> Jun 23 01:18:41.578659382 (104h55m17.141982372s) [noise=true , address=104.36.82.15:6286 , piece_id=ETSDIEVDHOYGJC62M3M6JUQQLZL5PWPYRLUAVWHU7HE7FGX4YE6Q, seg_pos={0 0}, upload=83962]: Done uploading piece. err: <nil> Jun 23 01:18:44.086228467 (104h55m19.649551478s) [noise=true , address=79.127.219.38:4026 , piece_id=EQUPNI4CUJP6MTH6MODLH6M5YJ4ZNO5ALN6GPSX22RNBCRRIPUOQ, seg_pos={0 0}, upload=83962]: Done uploading piece. err: <nil> Jun 23 01:18:44.523443200 (104h55m20.086766195s) [noise=true , address=77.90.31.10:4535 , piece_id=BORXDT7QJFNH6WJR6GEMRXATEOJFLESKT5XMWY6UOER6R2RW6KBQ, seg_pos={0 0}, upload=83962]: Done uploading piece. err: <nil> Jun 23 01:18:46.233887646 (104h55m21.797210646s) [noise=true , address=104.36.82.56:4245 , piece_id=CGUQ357JLAQJCIDNQ23BNV4Q3R4RGVJI2Q5Z7B56LJKZIFENJM3Q, seg_pos={0 0}, upload=83962]: Done uploading piece. err: <nil> Jun 23 01:19:21.357923809 (104h55m56.921246809s) [noise=true , address=104.36.82.54:4157 , piece_id=ESGS5AM5NBIEFUH7GGK7QV3PIQDMBLIKLIIGEBO7TJML72TF4EAQ, seg_pos={0 0}, upload=83962]: Done uploading piece. err: <nil> Jun 23 01:19:36.705243994 (104h56m12.268566993s) [noise=true , address=104.36.82.53:4128 , piece_id=YZHGLRKEISP3W4KUK3LL3EQEJUMX2LFPIDX2O676LS3GY3PNZ4TA, seg_pos={0 0}, upload=83962]: Done uploading piece. err: ecclient: upload failed (node:125ZMc6ed9qgBRtBC5N4zNvGonip3GNAEBPt3auF6dBGteX46Pf, address:104.36.82.53:4128): protocol: expected piece hash; write tcp 159.223.143.39:39324->104.36.82.53:4128: use of closed network connection; write tcp 159.223.143.39:39324->104.36.82.53:4128: use of closed network connection; EOF Jun 23 01:19:36.705278262 (104h56m12.26860127s) [seg_pos={0 0}, upload=83962]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jun 23 01:19:36.728720635 (104h56m12.292043639s) [upload=83962]: Done uploading segments... Jun 23 01:19:36.728814444 (104h56m12.292137449s) [upload=83962]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 26.620790947s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [......................X.............................X.......] duration: 14.976098289s FAIL (2025-06-23 01:24:46 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== nil> Jun 23 01:18:36.741346057 (104h55m12.304669068s) [upload=83962]: Starting upload Jun 23 01:18:36.741457026 (104h55m12.304780037s) [upload=83962]: Done starting upload Jun 23 01:18:36.741574186 (104h55m12.304897176s) [upload=83962]: Uploading segments... Jun 23 01:18:36.741945356 (104h55m12.305268357s) [upload=83962]: Got next segment. Inline: false Jun 23 01:18:36.742017220 (104h55m12.305340209s) [upload=83962]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 23 01:18:36.742022591 (104h55m12.305345591s) [upload=83962]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 23 01:18:36.777635436 (104h55m12.34095844s) [seg_pos={0 0}, upload=83962]: Begin upload segment... Jun 23 01:18:36.777700720 (104h55m12.341023723s) [seg_pos={0 0}, upload=83962]: Done begin upload segment. Jun 23 01:18:36.777707373 (104h55m12.341030382s) [noise=true , address=104.36.82.34:6569 , piece_id=EGMJEZMPGKMR5JHGP3TH3MJ7SJ5XWVMTVXCAFDX4RKPIVVMP4BAA, seg_pos={0 0}, upload=83962]: Uploading piece... Jun 23 01:18:36.777724726 (104h55m12.341047737s) [noise=true , address=77.90.31.10:4535 , piece_id=BORXDT7QJFNH6WJR6GEMRXATEOJFLESKT5XMWY6UOER6R2RW6KBQ, seg_pos={0 0}, upload=83962]: Uploading piece... Jun 23 01:18:36.777804915 (104h55m12.341127924s) [noise=true , address=45.91.116.9:5277 , piece_id=U7K7GEX3GPXNULZP35E65E52YL2BUNEXEKTRRZUP6GAALWWPS6KA, seg_pos={0 0}, upload=83962]: Uploading piece... Jun 23 01:18:36.778210171 (104h55m12.341533182s) [noise=true , address=104.36.82.53:4128 , piece_id=YZHGLRKEISP3W4KUK3LL3EQEJUMX2LFPIDX2O676LS3GY3PNZ4TA, seg_pos={0 0}, upload=83962]: Uploading piece... Jun 23 01:18:36.778614738 (104h55m12.341937746s) [noise=true , address=104.36.82.56:4245 , piece_id=CGUQ357JLAQJCIDNQ23BNV4Q3R4RGVJI2Q5Z7B56LJKZIFENJM3Q, seg_pos={0 0}, upload=83962]: Uploading piece... Jun 23 01:18:36.778987954 (104h55m12.342310963s) [noise=true , address=104.36.82.7:5933 , piece_id=Q27EIDAGLD5IZGDJ6QBB3ZA5ULFSIPXY4JWFTNBMLX346CK2ADBQ, seg_pos={0 0}, upload=83962]: Uploading piece... Jun 23 01:18:36.779391219 (104h55m12.342714228s) [noise=true , address=104.36.82.54:4157 , piece_id=ESGS5AM5NBIEFUH7GGK7QV3PIQDMBLIKLIIGEBO7TJML72TF4EAQ, seg_pos={0 0}, upload=83962]: Uploading piece... Jun 23 01:18:36.779521731 (104h55m12.342844734s) [noise=true , address=104.36.82.20:6444 , piece_id=CWBFFDJCLPZFFSP7NVOKELPXCR3T3DV43FGSSWNGJO6W2IHXEJ3A, seg_pos={0 0}, upload=83962]: Uploading piece... Jun 23 01:18:36.779962650 (104h55m12.34328566s) [noise=true , address=104.36.82.14:6228 , piece_id=QMSNWZWKIZ4DQPXOZJC67HOTDUBWJNKY45M5QZF7I7ENC4TZE2VA, seg_pos={0 0}, upload=83962]: Uploading piece... Jun 23 01:18:36.780465195 (104h55m12.343788212s) [noise=true , address=77.90.31.18:4825 , piece_id=DIQKUPGRMOVTGWHU5TVS4O2DXIBO74SKVFU3WHJN5PDU4R4XZI7Q, seg_pos={0 0}, upload=83962]: Uploading piece... Jun 23 01:18:36.780615746 (104h55m12.343938757s) [noise=true , address=104.36.82.22:6530 , piece_id=ZHRUIZ5BZCM67DGFQZS6NI2DV732UCVDYV3GM55EISACHUWN5ZLQ, seg_pos={0 0}, upload=83962]: Uploading piece... Jun 23 01:18:36.781165217 (104h55m12.344488225s) [noise=true , address=104.36.82.51:4084 , piece_id=DODGPPX2WWL6OWRP56V6QX2JZ2PMGHXG3AEQEBRACLAHLWGJTJWA, seg_pos={0 0}, upload=83962]: Uploading piece... Jun 23 01:18:36.781550748 (104h55m12.344873758s) [noise=true , address=79.127.205.234:5591 , piece_id=U2VCI7JO4IAU4B3LXOGMLQT452CIP2LLT2T53HBQFCR6U6UDQ2RA, seg_pos={0 0}, upload=83962]: Uploading piece... Jun 23 01:18:36.782014436 (104h55m12.345337441s) [noise=true , address=45.91.116.8:5050 , piece_id=6HAF64M3236TINYRYVIYSG27BF2QW6XGCIKE6SBVVZ527LFOP2ZQ, seg_pos={0 0}, upload=83962]: Uploading piece... Jun 23 01:18:36.782496184 (104h55m12.345819196s) [noise=true , address=104.36.82.15:6286 , piece_id=ETSDIEVDHOYGJC62M3M6JUQQLZL5PWPYRLUAVWHU7HE7FGX4YE6Q, seg_pos={0 0}, upload=83962]: Uploading piece... Jun 23 01:18:36.782881896 (104h55m12.346204905s) [noise=true , address=79.127.219.38:4026 , piece_id=EQUPNI4CUJP6MTH6MODLH6M5YJ4ZNO5ALN6GPSX22RNBCRRIPUOQ, seg_pos={0 0}, upload=83962]: Uploading piece... Jun 23 01:18:36.783409077 (104h55m12.346732086s) [noise=true , address=104.36.82.55:4225 , piece_id=M77UT6JBGUAVSXFMDUMYPRQWN4X7TOSX35LIWATYKMDXE6CSYPDA, seg_pos={0 0}, upload=83962]: Uploading piece... Jun 23 01:18:36.783797923 (104h55m12.34712093s) [noise=true , address=77.90.31.14:4695 , piece_id=RLOHBVSNWCGZU4YE2HNCBKV6V7BY7JDCDKJFWCM4YYFH5EZXWCFQ, seg_pos={0 0}, upload=83962]: Uploading piece... Jun 23 01:18:36.863659010 (104h55m12.426982006s) [upload=83962]: Next returned nil segment Jun 23 01:18:36.863678971 (104h55m12.427001984s) [upload=83962]: Waiting for error group managing segments... Jun 23 01:18:36.873250869 (104h55m12.436573877s) [noise=true , address=77.90.31.18:4825 , piece_id=DIQKUPGRMOVTGWHU5TVS4O2DXIBO74SKVFU3WHJN5PDU4R4XZI7Q, seg_pos={0 0}, upload=83962]: Done uploading piece. err: <nil> Jun 23 01:18:36.896075025 (104h55m12.459398017s) [noise=true , address=77.90.31.14:4695 , piece_id=RLOHBVSNWCGZU4YE2HNCBKV6V7BY7JDCDKJFWCM4YYFH5EZXWCFQ, seg_pos={0 0}, upload=83962]: Done uploading piece. err: <nil> Jun 23 01:18:36.930177382 (104h55m12.493500388s) [noise=true , address=104.36.82.14:6228 , piece_id=QMSNWZWKIZ4DQPXOZJC67HOTDUBWJNKY45M5QZF7I7ENC4TZE2VA, seg_pos={0 0}, upload=83962]: Done uploading piece. err: ecclient: upload failed (node:12pkADNFaYa2SqFALQdxwf1CfR1ktUwshQkw7iBJzhdhaRYaszF, address:104.36.82.14:6228): protocol: storage node overloaded, request limit: 2000 Jun 23 01:18:36.953640851 (104h55m12.51696386s) [noise=true , address=104.36.82.20:6444 , piece_id=CWBFFDJCLPZFFSP7NVOKELPXCR3T3DV43FGSSWNGJO6W2IHXEJ3A, seg_pos={0 0}, upload=83962]: Done uploading piece. err: ecclient: upload failed (node:12m5PditUsvMEejqnuEBZ6jLunbkTw9dEkmGzLcBg4y2XAwhBTD, address:104.36.82.20:6444): protocol: storage node overloaded, request limit: 2000 Jun 23 01:18:37.109817807 (104h55m12.673140818s) [noise=true , address=104.36.82.22:6530 , piece_id=ZHRUIZ5BZCM67DGFQZS6NI2DV732UCVDYV3GM55EISACHUWN5ZLQ, seg_pos={0 0}, upload=83962]: Done uploading piece. err: ecclient: upload failed (node:1akHUSoQFafzkGTJ385iYaowc87pJBDNfU5hodSCUDFa5ojN3v, address:104.36.82.22:6530): protocol: storage node overloaded, request limit: 2000 Jun 23 01:18:37.110741515 (104h55m12.674064529s) [noise=true , address=79.127.205.234:5591 , piece_id=U2VCI7JO4IAU4B3LXOGMLQT452CIP2LLT2T53HBQFCR6U6UDQ2RA, seg_pos={0 0}, upload=83962]: Done uploading piece. err: ecclient: upload failed (node:1ZK9ZUTAh12ywDRBtr54pdiWZi5WwkHXTmZkgoh6SpwvLAJTRi, address:79.127.205.234:5591): protocol: storage node overloaded, request limit: 2000 Jun 23 01:18:37.860754351 (104h55m13.424077339s) [noise=true , address=104.36.82.7:5933 , piece_id=Q27EIDAGLD5IZGDJ6QBB3ZA5ULFSIPXY4JWFTNBMLX346CK2ADBQ, seg_pos={0 0}, upload=83962]: Done uploading piece. err: <nil> Jun 23 01:18:38.383822537 (104h55m13.947145526s) [noise=true , address=45.91.116.8:5050 , piece_id=6HAF64M3236TINYRYVIYSG27BF2QW6XGCIKE6SBVVZ527LFOP2ZQ, seg_pos={0 0}, upload=83962]: Done uploading piece. err: <nil> Jun 23 01:18:38.848556195 (104h55m14.411879183s) [noise=true , address=104.36.82.34:6569 , piece_id=EGMJEZMPGKMR5JHGP3TH3MJ7SJ5XWVMTVXCAFDX4RKPIVVMP4BAA, seg_pos={0 0}, upload=83962]: Done uploading piece. err: <nil> Jun 23 01:18:38.877262383 (104h55m14.440585376s) [noise=true , address=104.36.82.51:4084 , piece_id=DODGPPX2WWL6OWRP56V6QX2JZ2PMGHXG3AEQEBRACLAHLWGJTJWA, seg_pos={0 0}, upload=83962]: Done uploading piece. err: <nil> Jun 23 01:18:39.590142305 (104h55m15.15346531s) [noise=true , address=45.91.116.9:5277 , piece_id=U7K7GEX3GPXNULZP35E65E52YL2BUNEXEKTRRZUP6GAALWWPS6KA, seg_pos={0 0}, upload=83962]: Done uploading piece. err: <nil> Jun 23 01:18:41.397250467 (104h55m16.960573472s) [noise=true , address=104.36.82.55:4225 , piece_id=M77UT6JBGUAVSXFMDUMYPRQWN4X7TOSX35LIWATYKMDXE6CSYPDA, seg_pos={0 0}, upload=83962]: Done uploading piece. err: <nil> Jun 23 01:18:41.578659382 (104h55m17.141982372s) [noise=true , address=104.36.82.15:6286 , piece_id=ETSDIEVDHOYGJC62M3M6JUQQLZL5PWPYRLUAVWHU7HE7FGX4YE6Q, seg_pos={0 0}, upload=83962]: Done uploading piece. err: <nil> Jun 23 01:18:44.086228467 (104h55m19.649551478s) [noise=true , address=79.127.219.38:4026 , piece_id=EQUPNI4CUJP6MTH6MODLH6M5YJ4ZNO5ALN6GPSX22RNBCRRIPUOQ, seg_pos={0 0}, upload=83962]: Done uploading piece. err: <nil> Jun 23 01:18:44.523443200 (104h55m20.086766195s) [noise=true , address=77.90.31.10:4535 , piece_id=BORXDT7QJFNH6WJR6GEMRXATEOJFLESKT5XMWY6UOER6R2RW6KBQ, seg_pos={0 0}, upload=83962]: Done uploading piece. err: <nil> Jun 23 01:18:46.233887646 (104h55m21.797210646s) [noise=true , address=104.36.82.56:4245 , piece_id=CGUQ357JLAQJCIDNQ23BNV4Q3R4RGVJI2Q5Z7B56LJKZIFENJM3Q, seg_pos={0 0}, upload=83962]: Done uploading piece. err: <nil> Jun 23 01:19:21.357923809 (104h55m56.921246809s) [noise=true , address=104.36.82.54:4157 , piece_id=ESGS5AM5NBIEFUH7GGK7QV3PIQDMBLIKLIIGEBO7TJML72TF4EAQ, seg_pos={0 0}, upload=83962]: Done uploading piece. err: <nil> Jun 23 01:19:36.705243994 (104h56m12.268566993s) [noise=true , address=104.36.82.53:4128 , piece_id=YZHGLRKEISP3W4KUK3LL3EQEJUMX2LFPIDX2O676LS3GY3PNZ4TA, seg_pos={0 0}, upload=83962]: Done uploading piece. err: ecclient: upload failed (node:125ZMc6ed9qgBRtBC5N4zNvGonip3GNAEBPt3auF6dBGteX46Pf, address:104.36.82.53:4128): protocol: expected piece hash; write tcp 159.223.143.39:39324->104.36.82.53:4128: use of closed network connection; write tcp 159.223.143.39:39324->104.36.82.53:4128: use of closed network connection; EOF Jun 23 01:19:36.705278262 (104h56m12.26860127s) [seg_pos={0 0}, upload=83962]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jun 23 01:19:36.728720635 (104h56m12.292043639s) [upload=83962]: Done uploading segments... Jun 23 01:19:36.728814444 (104h56m12.292137449s) [upload=83962]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 14.976128591s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [..........................X.....................X...........] duration: 8.912591364s FAIL (2025-06-23 01:24:44 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 8.91260788s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [...........................X.....................X..........] duration: 44.365983002s FAIL (2025-06-23 01:24:36 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 44.366006966s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [.......................X.............................X......] duration: 20.245309906s FAIL (2025-06-23 01:23:42 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== nil> Jun 23 01:18:36.741346057 (104h55m12.304669068s) [upload=83962]: Starting upload Jun 23 01:18:36.741457026 (104h55m12.304780037s) [upload=83962]: Done starting upload Jun 23 01:18:36.741574186 (104h55m12.304897176s) [upload=83962]: Uploading segments... Jun 23 01:18:36.741945356 (104h55m12.305268357s) [upload=83962]: Got next segment. Inline: false Jun 23 01:18:36.742017220 (104h55m12.305340209s) [upload=83962]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 23 01:18:36.742022591 (104h55m12.305345591s) [upload=83962]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 23 01:18:36.777635436 (104h55m12.34095844s) [seg_pos={0 0}, upload=83962]: Begin upload segment... Jun 23 01:18:36.777700720 (104h55m12.341023723s) [seg_pos={0 0}, upload=83962]: Done begin upload segment. Jun 23 01:18:36.777707373 (104h55m12.341030382s) [noise=true , address=104.36.82.34:6569 , piece_id=EGMJEZMPGKMR5JHGP3TH3MJ7SJ5XWVMTVXCAFDX4RKPIVVMP4BAA, seg_pos={0 0}, upload=83962]: Uploading piece... Jun 23 01:18:36.777724726 (104h55m12.341047737s) [noise=true , address=77.90.31.10:4535 , piece_id=BORXDT7QJFNH6WJR6GEMRXATEOJFLESKT5XMWY6UOER6R2RW6KBQ, seg_pos={0 0}, upload=83962]: Uploading piece... Jun 23 01:18:36.777804915 (104h55m12.341127924s) [noise=true , address=45.91.116.9:5277 , piece_id=U7K7GEX3GPXNULZP35E65E52YL2BUNEXEKTRRZUP6GAALWWPS6KA, seg_pos={0 0}, upload=83962]: Uploading piece... Jun 23 01:18:36.778210171 (104h55m12.341533182s) [noise=true , address=104.36.82.53:4128 , piece_id=YZHGLRKEISP3W4KUK3LL3EQEJUMX2LFPIDX2O676LS3GY3PNZ4TA, seg_pos={0 0}, upload=83962]: Uploading piece... Jun 23 01:18:36.778614738 (104h55m12.341937746s) [noise=true , address=104.36.82.56:4245 , piece_id=CGUQ357JLAQJCIDNQ23BNV4Q3R4RGVJI2Q5Z7B56LJKZIFENJM3Q, seg_pos={0 0}, upload=83962]: Uploading piece... Jun 23 01:18:36.778987954 (104h55m12.342310963s) [noise=true , address=104.36.82.7:5933 , piece_id=Q27EIDAGLD5IZGDJ6QBB3ZA5ULFSIPXY4JWFTNBMLX346CK2ADBQ, seg_pos={0 0}, upload=83962]: Uploading piece... Jun 23 01:18:36.779391219 (104h55m12.342714228s) [noise=true , address=104.36.82.54:4157 , piece_id=ESGS5AM5NBIEFUH7GGK7QV3PIQDMBLIKLIIGEBO7TJML72TF4EAQ, seg_pos={0 0}, upload=83962]: Uploading piece... Jun 23 01:18:36.779521731 (104h55m12.342844734s) [noise=true , address=104.36.82.20:6444 , piece_id=CWBFFDJCLPZFFSP7NVOKELPXCR3T3DV43FGSSWNGJO6W2IHXEJ3A, seg_pos={0 0}, upload=83962]: Uploading piece... Jun 23 01:18:36.779962650 (104h55m12.34328566s) [noise=true , address=104.36.82.14:6228 , piece_id=QMSNWZWKIZ4DQPXOZJC67HOTDUBWJNKY45M5QZF7I7ENC4TZE2VA, seg_pos={0 0}, upload=83962]: Uploading piece... Jun 23 01:18:36.780465195 (104h55m12.343788212s) [noise=true , address=77.90.31.18:4825 , piece_id=DIQKUPGRMOVTGWHU5TVS4O2DXIBO74SKVFU3WHJN5PDU4R4XZI7Q, seg_pos={0 0}, upload=83962]: Uploading piece... Jun 23 01:18:36.780615746 (104h55m12.343938757s) [noise=true , address=104.36.82.22:6530 , piece_id=ZHRUIZ5BZCM67DGFQZS6NI2DV732UCVDYV3GM55EISACHUWN5ZLQ, seg_pos={0 0}, upload=83962]: Uploading piece... Jun 23 01:18:36.781165217 (104h55m12.344488225s) [noise=true , address=104.36.82.51:4084 , piece_id=DODGPPX2WWL6OWRP56V6QX2JZ2PMGHXG3AEQEBRACLAHLWGJTJWA, seg_pos={0 0}, upload=83962]: Uploading piece... Jun 23 01:18:36.781550748 (104h55m12.344873758s) [noise=true , address=79.127.205.234:5591 , piece_id=U2VCI7JO4IAU4B3LXOGMLQT452CIP2LLT2T53HBQFCR6U6UDQ2RA, seg_pos={0 0}, upload=83962]: Uploading piece... Jun 23 01:18:36.782014436 (104h55m12.345337441s) [noise=true , address=45.91.116.8:5050 , piece_id=6HAF64M3236TINYRYVIYSG27BF2QW6XGCIKE6SBVVZ527LFOP2ZQ, seg_pos={0 0}, upload=83962]: Uploading piece... Jun 23 01:18:36.782496184 (104h55m12.345819196s) [noise=true , address=104.36.82.15:6286 , piece_id=ETSDIEVDHOYGJC62M3M6JUQQLZL5PWPYRLUAVWHU7HE7FGX4YE6Q, seg_pos={0 0}, upload=83962]: Uploading piece... Jun 23 01:18:36.782881896 (104h55m12.346204905s) [noise=true , address=79.127.219.38:4026 , piece_id=EQUPNI4CUJP6MTH6MODLH6M5YJ4ZNO5ALN6GPSX22RNBCRRIPUOQ, seg_pos={0 0}, upload=83962]: Uploading piece... Jun 23 01:18:36.783409077 (104h55m12.346732086s) [noise=true , address=104.36.82.55:4225 , piece_id=M77UT6JBGUAVSXFMDUMYPRQWN4X7TOSX35LIWATYKMDXE6CSYPDA, seg_pos={0 0}, upload=83962]: Uploading piece... Jun 23 01:18:36.783797923 (104h55m12.34712093s) [noise=true , address=77.90.31.14:4695 , piece_id=RLOHBVSNWCGZU4YE2HNCBKV6V7BY7JDCDKJFWCM4YYFH5EZXWCFQ, seg_pos={0 0}, upload=83962]: Uploading piece... Jun 23 01:18:36.863659010 (104h55m12.426982006s) [upload=83962]: Next returned nil segment Jun 23 01:18:36.863678971 (104h55m12.427001984s) [upload=83962]: Waiting for error group managing segments... Jun 23 01:18:36.873250869 (104h55m12.436573877s) [noise=true , address=77.90.31.18:4825 , piece_id=DIQKUPGRMOVTGWHU5TVS4O2DXIBO74SKVFU3WHJN5PDU4R4XZI7Q, seg_pos={0 0}, upload=83962]: Done uploading piece. err: <nil> Jun 23 01:18:36.896075025 (104h55m12.459398017s) [noise=true , address=77.90.31.14:4695 , piece_id=RLOHBVSNWCGZU4YE2HNCBKV6V7BY7JDCDKJFWCM4YYFH5EZXWCFQ, seg_pos={0 0}, upload=83962]: Done uploading piece. err: <nil> Jun 23 01:18:36.930177382 (104h55m12.493500388s) [noise=true , address=104.36.82.14:6228 , piece_id=QMSNWZWKIZ4DQPXOZJC67HOTDUBWJNKY45M5QZF7I7ENC4TZE2VA, seg_pos={0 0}, upload=83962]: Done uploading piece. err: ecclient: upload failed (node:12pkADNFaYa2SqFALQdxwf1CfR1ktUwshQkw7iBJzhdhaRYaszF, address:104.36.82.14:6228): protocol: storage node overloaded, request limit: 2000 Jun 23 01:18:36.953640851 (104h55m12.51696386s) [noise=true , address=104.36.82.20:6444 , piece_id=CWBFFDJCLPZFFSP7NVOKELPXCR3T3DV43FGSSWNGJO6W2IHXEJ3A, seg_pos={0 0}, upload=83962]: Done uploading piece. err: ecclient: upload failed (node:12m5PditUsvMEejqnuEBZ6jLunbkTw9dEkmGzLcBg4y2XAwhBTD, address:104.36.82.20:6444): protocol: storage node overloaded, request limit: 2000 Jun 23 01:18:37.109817807 (104h55m12.673140818s) [noise=true , address=104.36.82.22:6530 , piece_id=ZHRUIZ5BZCM67DGFQZS6NI2DV732UCVDYV3GM55EISACHUWN5ZLQ, seg_pos={0 0}, upload=83962]: Done uploading piece. err: ecclient: upload failed (node:1akHUSoQFafzkGTJ385iYaowc87pJBDNfU5hodSCUDFa5ojN3v, address:104.36.82.22:6530): protocol: storage node overloaded, request limit: 2000 Jun 23 01:18:37.110741515 (104h55m12.674064529s) [noise=true , address=79.127.205.234:5591 , piece_id=U2VCI7JO4IAU4B3LXOGMLQT452CIP2LLT2T53HBQFCR6U6UDQ2RA, seg_pos={0 0}, upload=83962]: Done uploading piece. err: ecclient: upload failed (node:1ZK9ZUTAh12ywDRBtr54pdiWZi5WwkHXTmZkgoh6SpwvLAJTRi, address:79.127.205.234:5591): protocol: storage node overloaded, request limit: 2000 Jun 23 01:18:37.860754351 (104h55m13.424077339s) [noise=true , address=104.36.82.7:5933 , piece_id=Q27EIDAGLD5IZGDJ6QBB3ZA5ULFSIPXY4JWFTNBMLX346CK2ADBQ, seg_pos={0 0}, upload=83962]: Done uploading piece. err: <nil> Jun 23 01:18:38.383822537 (104h55m13.947145526s) [noise=true , address=45.91.116.8:5050 , piece_id=6HAF64M3236TINYRYVIYSG27BF2QW6XGCIKE6SBVVZ527LFOP2ZQ, seg_pos={0 0}, upload=83962]: Done uploading piece. err: <nil> Jun 23 01:18:38.848556195 (104h55m14.411879183s) [noise=true , address=104.36.82.34:6569 , piece_id=EGMJEZMPGKMR5JHGP3TH3MJ7SJ5XWVMTVXCAFDX4RKPIVVMP4BAA, seg_pos={0 0}, upload=83962]: Done uploading piece. err: <nil> Jun 23 01:18:38.877262383 (104h55m14.440585376s) [noise=true , address=104.36.82.51:4084 , piece_id=DODGPPX2WWL6OWRP56V6QX2JZ2PMGHXG3AEQEBRACLAHLWGJTJWA, seg_pos={0 0}, upload=83962]: Done uploading piece. err: <nil> Jun 23 01:18:39.590142305 (104h55m15.15346531s) [noise=true , address=45.91.116.9:5277 , piece_id=U7K7GEX3GPXNULZP35E65E52YL2BUNEXEKTRRZUP6GAALWWPS6KA, seg_pos={0 0}, upload=83962]: Done uploading piece. err: <nil> Jun 23 01:18:41.397250467 (104h55m16.960573472s) [noise=true , address=104.36.82.55:4225 , piece_id=M77UT6JBGUAVSXFMDUMYPRQWN4X7TOSX35LIWATYKMDXE6CSYPDA, seg_pos={0 0}, upload=83962]: Done uploading piece. err: <nil> Jun 23 01:18:41.578659382 (104h55m17.141982372s) [noise=true , address=104.36.82.15:6286 , piece_id=ETSDIEVDHOYGJC62M3M6JUQQLZL5PWPYRLUAVWHU7HE7FGX4YE6Q, seg_pos={0 0}, upload=83962]: Done uploading piece. err: <nil> Jun 23 01:18:44.086228467 (104h55m19.649551478s) [noise=true , address=79.127.219.38:4026 , piece_id=EQUPNI4CUJP6MTH6MODLH6M5YJ4ZNO5ALN6GPSX22RNBCRRIPUOQ, seg_pos={0 0}, upload=83962]: Done uploading piece. err: <nil> Jun 23 01:18:44.523443200 (104h55m20.086766195s) [noise=true , address=77.90.31.10:4535 , piece_id=BORXDT7QJFNH6WJR6GEMRXATEOJFLESKT5XMWY6UOER6R2RW6KBQ, seg_pos={0 0}, upload=83962]: Done uploading piece. err: <nil> Jun 23 01:18:46.233887646 (104h55m21.797210646s) [noise=true , address=104.36.82.56:4245 , piece_id=CGUQ357JLAQJCIDNQ23BNV4Q3R4RGVJI2Q5Z7B56LJKZIFENJM3Q, seg_pos={0 0}, upload=83962]: Done uploading piece. err: <nil> Jun 23 01:19:21.357923809 (104h55m56.921246809s) [noise=true , address=104.36.82.54:4157 , piece_id=ESGS5AM5NBIEFUH7GGK7QV3PIQDMBLIKLIIGEBO7TJML72TF4EAQ, seg_pos={0 0}, upload=83962]: Done uploading piece. err: <nil> Jun 23 01:19:36.705243994 (104h56m12.268566993s) [noise=true , address=104.36.82.53:4128 , piece_id=YZHGLRKEISP3W4KUK3LL3EQEJUMX2LFPIDX2O676LS3GY3PNZ4TA, seg_pos={0 0}, upload=83962]: Done uploading piece. err: ecclient: upload failed (node:125ZMc6ed9qgBRtBC5N4zNvGonip3GNAEBPt3auF6dBGteX46Pf, address:104.36.82.53:4128): protocol: expected piece hash; write tcp 159.223.143.39:39324->104.36.82.53:4128: use of closed network connection; write tcp 159.223.143.39:39324->104.36.82.53:4128: use of closed network connection; EOF Jun 23 01:19:36.705278262 (104h56m12.26860127s) [seg_pos={0 0}, upload=83962]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jun 23 01:19:36.728720635 (104h56m12.292043639s) [upload=83962]: Done uploading segments... Jun 23 01:19:36.728814444 (104h56m12.292137449s) [upload=83962]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 20.245340943s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [............................X.....................X.........] duration: 12.504209135s FAIL (2025-06-23 01:23:19 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 12.505041633s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [........................X.............................X.....] duration: 13.876876127s FAIL (2025-06-23 01:23:16 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== nil> Jun 23 01:18:36.741346057 (104h55m12.304669068s) [upload=83962]: Starting upload Jun 23 01:18:36.741457026 (104h55m12.304780037s) [upload=83962]: Done starting upload Jun 23 01:18:36.741574186 (104h55m12.304897176s) [upload=83962]: Uploading segments... Jun 23 01:18:36.741945356 (104h55m12.305268357s) [upload=83962]: Got next segment. Inline: false Jun 23 01:18:36.742017220 (104h55m12.305340209s) [upload=83962]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 23 01:18:36.742022591 (104h55m12.305345591s) [upload=83962]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 23 01:18:36.777635436 (104h55m12.34095844s) [seg_pos={0 0}, upload=83962]: Begin upload segment... Jun 23 01:18:36.777700720 (104h55m12.341023723s) [seg_pos={0 0}, upload=83962]: Done begin upload segment. Jun 23 01:18:36.777707373 (104h55m12.341030382s) [noise=true , address=104.36.82.34:6569 , piece_id=EGMJEZMPGKMR5JHGP3TH3MJ7SJ5XWVMTVXCAFDX4RKPIVVMP4BAA, seg_pos={0 0}, upload=83962]: Uploading piece... Jun 23 01:18:36.777724726 (104h55m12.341047737s) [noise=true , address=77.90.31.10:4535 , piece_id=BORXDT7QJFNH6WJR6GEMRXATEOJFLESKT5XMWY6UOER6R2RW6KBQ, seg_pos={0 0}, upload=83962]: Uploading piece... Jun 23 01:18:36.777804915 (104h55m12.341127924s) [noise=true , address=45.91.116.9:5277 , piece_id=U7K7GEX3GPXNULZP35E65E52YL2BUNEXEKTRRZUP6GAALWWPS6KA, seg_pos={0 0}, upload=83962]: Uploading piece... Jun 23 01:18:36.778210171 (104h55m12.341533182s) [noise=true , address=104.36.82.53:4128 , piece_id=YZHGLRKEISP3W4KUK3LL3EQEJUMX2LFPIDX2O676LS3GY3PNZ4TA, seg_pos={0 0}, upload=83962]: Uploading piece... Jun 23 01:18:36.778614738 (104h55m12.341937746s) [noise=true , address=104.36.82.56:4245 , piece_id=CGUQ357JLAQJCIDNQ23BNV4Q3R4RGVJI2Q5Z7B56LJKZIFENJM3Q, seg_pos={0 0}, upload=83962]: Uploading piece... Jun 23 01:18:36.778987954 (104h55m12.342310963s) [noise=true , address=104.36.82.7:5933 , piece_id=Q27EIDAGLD5IZGDJ6QBB3ZA5ULFSIPXY4JWFTNBMLX346CK2ADBQ, seg_pos={0 0}, upload=83962]: Uploading piece... Jun 23 01:18:36.779391219 (104h55m12.342714228s) [noise=true , address=104.36.82.54:4157 , piece_id=ESGS5AM5NBIEFUH7GGK7QV3PIQDMBLIKLIIGEBO7TJML72TF4EAQ, seg_pos={0 0}, upload=83962]: Uploading piece... Jun 23 01:18:36.779521731 (104h55m12.342844734s) [noise=true , address=104.36.82.20:6444 , piece_id=CWBFFDJCLPZFFSP7NVOKELPXCR3T3DV43FGSSWNGJO6W2IHXEJ3A, seg_pos={0 0}, upload=83962]: Uploading piece... Jun 23 01:18:36.779962650 (104h55m12.34328566s) [noise=true , address=104.36.82.14:6228 , piece_id=QMSNWZWKIZ4DQPXOZJC67HOTDUBWJNKY45M5QZF7I7ENC4TZE2VA, seg_pos={0 0}, upload=83962]: Uploading piece... Jun 23 01:18:36.780465195 (104h55m12.343788212s) [noise=true , address=77.90.31.18:4825 , piece_id=DIQKUPGRMOVTGWHU5TVS4O2DXIBO74SKVFU3WHJN5PDU4R4XZI7Q, seg_pos={0 0}, upload=83962]: Uploading piece... Jun 23 01:18:36.780615746 (104h55m12.343938757s) [noise=true , address=104.36.82.22:6530 , piece_id=ZHRUIZ5BZCM67DGFQZS6NI2DV732UCVDYV3GM55EISACHUWN5ZLQ, seg_pos={0 0}, upload=83962]: Uploading piece... Jun 23 01:18:36.781165217 (104h55m12.344488225s) [noise=true , address=104.36.82.51:4084 , piece_id=DODGPPX2WWL6OWRP56V6QX2JZ2PMGHXG3AEQEBRACLAHLWGJTJWA, seg_pos={0 0}, upload=83962]: Uploading piece... Jun 23 01:18:36.781550748 (104h55m12.344873758s) [noise=true , address=79.127.205.234:5591 , piece_id=U2VCI7JO4IAU4B3LXOGMLQT452CIP2LLT2T53HBQFCR6U6UDQ2RA, seg_pos={0 0}, upload=83962]: Uploading piece... Jun 23 01:18:36.782014436 (104h55m12.345337441s) [noise=true , address=45.91.116.8:5050 , piece_id=6HAF64M3236TINYRYVIYSG27BF2QW6XGCIKE6SBVVZ527LFOP2ZQ, seg_pos={0 0}, upload=83962]: Uploading piece... Jun 23 01:18:36.782496184 (104h55m12.345819196s) [noise=true , address=104.36.82.15:6286 , piece_id=ETSDIEVDHOYGJC62M3M6JUQQLZL5PWPYRLUAVWHU7HE7FGX4YE6Q, seg_pos={0 0}, upload=83962]: Uploading piece... Jun 23 01:18:36.782881896 (104h55m12.346204905s) [noise=true , address=79.127.219.38:4026 , piece_id=EQUPNI4CUJP6MTH6MODLH6M5YJ4ZNO5ALN6GPSX22RNBCRRIPUOQ, seg_pos={0 0}, upload=83962]: Uploading piece... Jun 23 01:18:36.783409077 (104h55m12.346732086s) [noise=true , address=104.36.82.55:4225 , piece_id=M77UT6JBGUAVSXFMDUMYPRQWN4X7TOSX35LIWATYKMDXE6CSYPDA, seg_pos={0 0}, upload=83962]: Uploading piece... Jun 23 01:18:36.783797923 (104h55m12.34712093s) [noise=true , address=77.90.31.14:4695 , piece_id=RLOHBVSNWCGZU4YE2HNCBKV6V7BY7JDCDKJFWCM4YYFH5EZXWCFQ, seg_pos={0 0}, upload=83962]: Uploading piece... Jun 23 01:18:36.863659010 (104h55m12.426982006s) [upload=83962]: Next returned nil segment Jun 23 01:18:36.863678971 (104h55m12.427001984s) [upload=83962]: Waiting for error group managing segments... Jun 23 01:18:36.873250869 (104h55m12.436573877s) [noise=true , address=77.90.31.18:4825 , piece_id=DIQKUPGRMOVTGWHU5TVS4O2DXIBO74SKVFU3WHJN5PDU4R4XZI7Q, seg_pos={0 0}, upload=83962]: Done uploading piece. err: <nil> Jun 23 01:18:36.896075025 (104h55m12.459398017s) [noise=true , address=77.90.31.14:4695 , piece_id=RLOHBVSNWCGZU4YE2HNCBKV6V7BY7JDCDKJFWCM4YYFH5EZXWCFQ, seg_pos={0 0}, upload=83962]: Done uploading piece. err: <nil> Jun 23 01:18:36.930177382 (104h55m12.493500388s) [noise=true , address=104.36.82.14:6228 , piece_id=QMSNWZWKIZ4DQPXOZJC67HOTDUBWJNKY45M5QZF7I7ENC4TZE2VA, seg_pos={0 0}, upload=83962]: Done uploading piece. err: ecclient: upload failed (node:12pkADNFaYa2SqFALQdxwf1CfR1ktUwshQkw7iBJzhdhaRYaszF, address:104.36.82.14:6228): protocol: storage node overloaded, request limit: 2000 Jun 23 01:18:36.953640851 (104h55m12.51696386s) [noise=true , address=104.36.82.20:6444 , piece_id=CWBFFDJCLPZFFSP7NVOKELPXCR3T3DV43FGSSWNGJO6W2IHXEJ3A, seg_pos={0 0}, upload=83962]: Done uploading piece. err: ecclient: upload failed (node:12m5PditUsvMEejqnuEBZ6jLunbkTw9dEkmGzLcBg4y2XAwhBTD, address:104.36.82.20:6444): protocol: storage node overloaded, request limit: 2000 Jun 23 01:18:37.109817807 (104h55m12.673140818s) [noise=true , address=104.36.82.22:6530 , piece_id=ZHRUIZ5BZCM67DGFQZS6NI2DV732UCVDYV3GM55EISACHUWN5ZLQ, seg_pos={0 0}, upload=83962]: Done uploading piece. err: ecclient: upload failed (node:1akHUSoQFafzkGTJ385iYaowc87pJBDNfU5hodSCUDFa5ojN3v, address:104.36.82.22:6530): protocol: storage node overloaded, request limit: 2000 Jun 23 01:18:37.110741515 (104h55m12.674064529s) [noise=true , address=79.127.205.234:5591 , piece_id=U2VCI7JO4IAU4B3LXOGMLQT452CIP2LLT2T53HBQFCR6U6UDQ2RA, seg_pos={0 0}, upload=83962]: Done uploading piece. err: ecclient: upload failed (node:1ZK9ZUTAh12ywDRBtr54pdiWZi5WwkHXTmZkgoh6SpwvLAJTRi, address:79.127.205.234:5591): protocol: storage node overloaded, request limit: 2000 Jun 23 01:18:37.860754351 (104h55m13.424077339s) [noise=true , address=104.36.82.7:5933 , piece_id=Q27EIDAGLD5IZGDJ6QBB3ZA5ULFSIPXY4JWFTNBMLX346CK2ADBQ, seg_pos={0 0}, upload=83962]: Done uploading piece. err: <nil> Jun 23 01:18:38.383822537 (104h55m13.947145526s) [noise=true , address=45.91.116.8:5050 , piece_id=6HAF64M3236TINYRYVIYSG27BF2QW6XGCIKE6SBVVZ527LFOP2ZQ, seg_pos={0 0}, upload=83962]: Done uploading piece. err: <nil> Jun 23 01:18:38.848556195 (104h55m14.411879183s) [noise=true , address=104.36.82.34:6569 , piece_id=EGMJEZMPGKMR5JHGP3TH3MJ7SJ5XWVMTVXCAFDX4RKPIVVMP4BAA, seg_pos={0 0}, upload=83962]: Done uploading piece. err: <nil> Jun 23 01:18:38.877262383 (104h55m14.440585376s) [noise=true , address=104.36.82.51:4084 , piece_id=DODGPPX2WWL6OWRP56V6QX2JZ2PMGHXG3AEQEBRACLAHLWGJTJWA, seg_pos={0 0}, upload=83962]: Done uploading piece. err: <nil> Jun 23 01:18:39.590142305 (104h55m15.15346531s) [noise=true , address=45.91.116.9:5277 , piece_id=U7K7GEX3GPXNULZP35E65E52YL2BUNEXEKTRRZUP6GAALWWPS6KA, seg_pos={0 0}, upload=83962]: Done uploading piece. err: <nil> Jun 23 01:18:41.397250467 (104h55m16.960573472s) [noise=true , address=104.36.82.55:4225 , piece_id=M77UT6JBGUAVSXFMDUMYPRQWN4X7TOSX35LIWATYKMDXE6CSYPDA, seg_pos={0 0}, upload=83962]: Done uploading piece. err: <nil> Jun 23 01:18:41.578659382 (104h55m17.141982372s) [noise=true , address=104.36.82.15:6286 , piece_id=ETSDIEVDHOYGJC62M3M6JUQQLZL5PWPYRLUAVWHU7HE7FGX4YE6Q, seg_pos={0 0}, upload=83962]: Done uploading piece. err: <nil> Jun 23 01:18:44.086228467 (104h55m19.649551478s) [noise=true , address=79.127.219.38:4026 , piece_id=EQUPNI4CUJP6MTH6MODLH6M5YJ4ZNO5ALN6GPSX22RNBCRRIPUOQ, seg_pos={0 0}, upload=83962]: Done uploading piece. err: <nil> Jun 23 01:18:44.523443200 (104h55m20.086766195s) [noise=true , address=77.90.31.10:4535 , piece_id=BORXDT7QJFNH6WJR6GEMRXATEOJFLESKT5XMWY6UOER6R2RW6KBQ, seg_pos={0 0}, upload=83962]: Done uploading piece. err: <nil> Jun 23 01:18:46.233887646 (104h55m21.797210646s) [noise=true , address=104.36.82.56:4245 , piece_id=CGUQ357JLAQJCIDNQ23BNV4Q3R4RGVJI2Q5Z7B56LJKZIFENJM3Q, seg_pos={0 0}, upload=83962]: Done uploading piece. err: <nil> Jun 23 01:19:21.357923809 (104h55m56.921246809s) [noise=true , address=104.36.82.54:4157 , piece_id=ESGS5AM5NBIEFUH7GGK7QV3PIQDMBLIKLIIGEBO7TJML72TF4EAQ, seg_pos={0 0}, upload=83962]: Done uploading piece. err: <nil> Jun 23 01:19:36.705243994 (104h56m12.268566993s) [noise=true , address=104.36.82.53:4128 , piece_id=YZHGLRKEISP3W4KUK3LL3EQEJUMX2LFPIDX2O676LS3GY3PNZ4TA, seg_pos={0 0}, upload=83962]: Done uploading piece. err: ecclient: upload failed (node:125ZMc6ed9qgBRtBC5N4zNvGonip3GNAEBPt3auF6dBGteX46Pf, address:104.36.82.53:4128): protocol: expected piece hash; write tcp 159.223.143.39:39324->104.36.82.53:4128: use of closed network connection; write tcp 159.223.143.39:39324->104.36.82.53:4128: use of closed network connection; EOF Jun 23 01:19:36.705278262 (104h56m12.26860127s) [seg_pos={0 0}, upload=83962]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jun 23 01:19:36.728720635 (104h56m12.292043639s) [upload=83962]: Done uploading segments... Jun 23 01:19:36.728814444 (104h56m12.292137449s) [upload=83962]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 13.876906087s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [.........................X.............................X....] duration: 23.177329485s FAIL (2025-06-23 01:22:37 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== nil> Jun 23 01:18:36.741346057 (104h55m12.304669068s) [upload=83962]: Starting upload Jun 23 01:18:36.741457026 (104h55m12.304780037s) [upload=83962]: Done starting upload Jun 23 01:18:36.741574186 (104h55m12.304897176s) [upload=83962]: Uploading segments... Jun 23 01:18:36.741945356 (104h55m12.305268357s) [upload=83962]: Got next segment. Inline: false Jun 23 01:18:36.742017220 (104h55m12.305340209s) [upload=83962]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 23 01:18:36.742022591 (104h55m12.305345591s) [upload=83962]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 23 01:18:36.777635436 (104h55m12.34095844s) [seg_pos={0 0}, upload=83962]: Begin upload segment... Jun 23 01:18:36.777700720 (104h55m12.341023723s) [seg_pos={0 0}, upload=83962]: Done begin upload segment. Jun 23 01:18:36.777707373 (104h55m12.341030382s) [noise=true , address=104.36.82.34:6569 , piece_id=EGMJEZMPGKMR5JHGP3TH3MJ7SJ5XWVMTVXCAFDX4RKPIVVMP4BAA, seg_pos={0 0}, upload=83962]: Uploading piece... Jun 23 01:18:36.777724726 (104h55m12.341047737s) [noise=true , address=77.90.31.10:4535 , piece_id=BORXDT7QJFNH6WJR6GEMRXATEOJFLESKT5XMWY6UOER6R2RW6KBQ, seg_pos={0 0}, upload=83962]: Uploading piece... Jun 23 01:18:36.777804915 (104h55m12.341127924s) [noise=true , address=45.91.116.9:5277 , piece_id=U7K7GEX3GPXNULZP35E65E52YL2BUNEXEKTRRZUP6GAALWWPS6KA, seg_pos={0 0}, upload=83962]: Uploading piece... Jun 23 01:18:36.778210171 (104h55m12.341533182s) [noise=true , address=104.36.82.53:4128 , piece_id=YZHGLRKEISP3W4KUK3LL3EQEJUMX2LFPIDX2O676LS3GY3PNZ4TA, seg_pos={0 0}, upload=83962]: Uploading piece... Jun 23 01:18:36.778614738 (104h55m12.341937746s) [noise=true , address=104.36.82.56:4245 , piece_id=CGUQ357JLAQJCIDNQ23BNV4Q3R4RGVJI2Q5Z7B56LJKZIFENJM3Q, seg_pos={0 0}, upload=83962]: Uploading piece... Jun 23 01:18:36.778987954 (104h55m12.342310963s) [noise=true , address=104.36.82.7:5933 , piece_id=Q27EIDAGLD5IZGDJ6QBB3ZA5ULFSIPXY4JWFTNBMLX346CK2ADBQ, seg_pos={0 0}, upload=83962]: Uploading piece... Jun 23 01:18:36.779391219 (104h55m12.342714228s) [noise=true , address=104.36.82.54:4157 , piece_id=ESGS5AM5NBIEFUH7GGK7QV3PIQDMBLIKLIIGEBO7TJML72TF4EAQ, seg_pos={0 0}, upload=83962]: Uploading piece... Jun 23 01:18:36.779521731 (104h55m12.342844734s) [noise=true , address=104.36.82.20:6444 , piece_id=CWBFFDJCLPZFFSP7NVOKELPXCR3T3DV43FGSSWNGJO6W2IHXEJ3A, seg_pos={0 0}, upload=83962]: Uploading piece... Jun 23 01:18:36.779962650 (104h55m12.34328566s) [noise=true , address=104.36.82.14:6228 , piece_id=QMSNWZWKIZ4DQPXOZJC67HOTDUBWJNKY45M5QZF7I7ENC4TZE2VA, seg_pos={0 0}, upload=83962]: Uploading piece... Jun 23 01:18:36.780465195 (104h55m12.343788212s) [noise=true , address=77.90.31.18:4825 , piece_id=DIQKUPGRMOVTGWHU5TVS4O2DXIBO74SKVFU3WHJN5PDU4R4XZI7Q, seg_pos={0 0}, upload=83962]: Uploading piece... Jun 23 01:18:36.780615746 (104h55m12.343938757s) [noise=true , address=104.36.82.22:6530 , piece_id=ZHRUIZ5BZCM67DGFQZS6NI2DV732UCVDYV3GM55EISACHUWN5ZLQ, seg_pos={0 0}, upload=83962]: Uploading piece... Jun 23 01:18:36.781165217 (104h55m12.344488225s) [noise=true , address=104.36.82.51:4084 , piece_id=DODGPPX2WWL6OWRP56V6QX2JZ2PMGHXG3AEQEBRACLAHLWGJTJWA, seg_pos={0 0}, upload=83962]: Uploading piece... Jun 23 01:18:36.781550748 (104h55m12.344873758s) [noise=true , address=79.127.205.234:5591 , piece_id=U2VCI7JO4IAU4B3LXOGMLQT452CIP2LLT2T53HBQFCR6U6UDQ2RA, seg_pos={0 0}, upload=83962]: Uploading piece... Jun 23 01:18:36.782014436 (104h55m12.345337441s) [noise=true , address=45.91.116.8:5050 , piece_id=6HAF64M3236TINYRYVIYSG27BF2QW6XGCIKE6SBVVZ527LFOP2ZQ, seg_pos={0 0}, upload=83962]: Uploading piece... Jun 23 01:18:36.782496184 (104h55m12.345819196s) [noise=true , address=104.36.82.15:6286 , piece_id=ETSDIEVDHOYGJC62M3M6JUQQLZL5PWPYRLUAVWHU7HE7FGX4YE6Q, seg_pos={0 0}, upload=83962]: Uploading piece... Jun 23 01:18:36.782881896 (104h55m12.346204905s) [noise=true , address=79.127.219.38:4026 , piece_id=EQUPNI4CUJP6MTH6MODLH6M5YJ4ZNO5ALN6GPSX22RNBCRRIPUOQ, seg_pos={0 0}, upload=83962]: Uploading piece... Jun 23 01:18:36.783409077 (104h55m12.346732086s) [noise=true , address=104.36.82.55:4225 , piece_id=M77UT6JBGUAVSXFMDUMYPRQWN4X7TOSX35LIWATYKMDXE6CSYPDA, seg_pos={0 0}, upload=83962]: Uploading piece... Jun 23 01:18:36.783797923 (104h55m12.34712093s) [noise=true , address=77.90.31.14:4695 , piece_id=RLOHBVSNWCGZU4YE2HNCBKV6V7BY7JDCDKJFWCM4YYFH5EZXWCFQ, seg_pos={0 0}, upload=83962]: Uploading piece... Jun 23 01:18:36.863659010 (104h55m12.426982006s) [upload=83962]: Next returned nil segment Jun 23 01:18:36.863678971 (104h55m12.427001984s) [upload=83962]: Waiting for error group managing segments... Jun 23 01:18:36.873250869 (104h55m12.436573877s) [noise=true , address=77.90.31.18:4825 , piece_id=DIQKUPGRMOVTGWHU5TVS4O2DXIBO74SKVFU3WHJN5PDU4R4XZI7Q, seg_pos={0 0}, upload=83962]: Done uploading piece. err: <nil> Jun 23 01:18:36.896075025 (104h55m12.459398017s) [noise=true , address=77.90.31.14:4695 , piece_id=RLOHBVSNWCGZU4YE2HNCBKV6V7BY7JDCDKJFWCM4YYFH5EZXWCFQ, seg_pos={0 0}, upload=83962]: Done uploading piece. err: <nil> Jun 23 01:18:36.930177382 (104h55m12.493500388s) [noise=true , address=104.36.82.14:6228 , piece_id=QMSNWZWKIZ4DQPXOZJC67HOTDUBWJNKY45M5QZF7I7ENC4TZE2VA, seg_pos={0 0}, upload=83962]: Done uploading piece. err: ecclient: upload failed (node:12pkADNFaYa2SqFALQdxwf1CfR1ktUwshQkw7iBJzhdhaRYaszF, address:104.36.82.14:6228): protocol: storage node overloaded, request limit: 2000 Jun 23 01:18:36.953640851 (104h55m12.51696386s) [noise=true , address=104.36.82.20:6444 , piece_id=CWBFFDJCLPZFFSP7NVOKELPXCR3T3DV43FGSSWNGJO6W2IHXEJ3A, seg_pos={0 0}, upload=83962]: Done uploading piece. err: ecclient: upload failed (node:12m5PditUsvMEejqnuEBZ6jLunbkTw9dEkmGzLcBg4y2XAwhBTD, address:104.36.82.20:6444): protocol: storage node overloaded, request limit: 2000 Jun 23 01:18:37.109817807 (104h55m12.673140818s) [noise=true , address=104.36.82.22:6530 , piece_id=ZHRUIZ5BZCM67DGFQZS6NI2DV732UCVDYV3GM55EISACHUWN5ZLQ, seg_pos={0 0}, upload=83962]: Done uploading piece. err: ecclient: upload failed (node:1akHUSoQFafzkGTJ385iYaowc87pJBDNfU5hodSCUDFa5ojN3v, address:104.36.82.22:6530): protocol: storage node overloaded, request limit: 2000 Jun 23 01:18:37.110741515 (104h55m12.674064529s) [noise=true , address=79.127.205.234:5591 , piece_id=U2VCI7JO4IAU4B3LXOGMLQT452CIP2LLT2T53HBQFCR6U6UDQ2RA, seg_pos={0 0}, upload=83962]: Done uploading piece. err: ecclient: upload failed (node:1ZK9ZUTAh12ywDRBtr54pdiWZi5WwkHXTmZkgoh6SpwvLAJTRi, address:79.127.205.234:5591): protocol: storage node overloaded, request limit: 2000 Jun 23 01:18:37.860754351 (104h55m13.424077339s) [noise=true , address=104.36.82.7:5933 , piece_id=Q27EIDAGLD5IZGDJ6QBB3ZA5ULFSIPXY4JWFTNBMLX346CK2ADBQ, seg_pos={0 0}, upload=83962]: Done uploading piece. err: <nil> Jun 23 01:18:38.383822537 (104h55m13.947145526s) [noise=true , address=45.91.116.8:5050 , piece_id=6HAF64M3236TINYRYVIYSG27BF2QW6XGCIKE6SBVVZ527LFOP2ZQ, seg_pos={0 0}, upload=83962]: Done uploading piece. err: <nil> Jun 23 01:18:38.848556195 (104h55m14.411879183s) [noise=true , address=104.36.82.34:6569 , piece_id=EGMJEZMPGKMR5JHGP3TH3MJ7SJ5XWVMTVXCAFDX4RKPIVVMP4BAA, seg_pos={0 0}, upload=83962]: Done uploading piece. err: <nil> Jun 23 01:18:38.877262383 (104h55m14.440585376s) [noise=true , address=104.36.82.51:4084 , piece_id=DODGPPX2WWL6OWRP56V6QX2JZ2PMGHXG3AEQEBRACLAHLWGJTJWA, seg_pos={0 0}, upload=83962]: Done uploading piece. err: <nil> Jun 23 01:18:39.590142305 (104h55m15.15346531s) [noise=true , address=45.91.116.9:5277 , piece_id=U7K7GEX3GPXNULZP35E65E52YL2BUNEXEKTRRZUP6GAALWWPS6KA, seg_pos={0 0}, upload=83962]: Done uploading piece. err: <nil> Jun 23 01:18:41.397250467 (104h55m16.960573472s) [noise=true , address=104.36.82.55:4225 , piece_id=M77UT6JBGUAVSXFMDUMYPRQWN4X7TOSX35LIWATYKMDXE6CSYPDA, seg_pos={0 0}, upload=83962]: Done uploading piece. err: <nil> Jun 23 01:18:41.578659382 (104h55m17.141982372s) [noise=true , address=104.36.82.15:6286 , piece_id=ETSDIEVDHOYGJC62M3M6JUQQLZL5PWPYRLUAVWHU7HE7FGX4YE6Q, seg_pos={0 0}, upload=83962]: Done uploading piece. err: <nil> Jun 23 01:18:44.086228467 (104h55m19.649551478s) [noise=true , address=79.127.219.38:4026 , piece_id=EQUPNI4CUJP6MTH6MODLH6M5YJ4ZNO5ALN6GPSX22RNBCRRIPUOQ, seg_pos={0 0}, upload=83962]: Done uploading piece. err: <nil> Jun 23 01:18:44.523443200 (104h55m20.086766195s) [noise=true , address=77.90.31.10:4535 , piece_id=BORXDT7QJFNH6WJR6GEMRXATEOJFLESKT5XMWY6UOER6R2RW6KBQ, seg_pos={0 0}, upload=83962]: Done uploading piece. err: <nil> Jun 23 01:18:46.233887646 (104h55m21.797210646s) [noise=true , address=104.36.82.56:4245 , piece_id=CGUQ357JLAQJCIDNQ23BNV4Q3R4RGVJI2Q5Z7B56LJKZIFENJM3Q, seg_pos={0 0}, upload=83962]: Done uploading piece. err: <nil> Jun 23 01:19:21.357923809 (104h55m56.921246809s) [noise=true , address=104.36.82.54:4157 , piece_id=ESGS5AM5NBIEFUH7GGK7QV3PIQDMBLIKLIIGEBO7TJML72TF4EAQ, seg_pos={0 0}, upload=83962]: Done uploading piece. err: <nil> Jun 23 01:19:36.705243994 (104h56m12.268566993s) [noise=true , address=104.36.82.53:4128 , piece_id=YZHGLRKEISP3W4KUK3LL3EQEJUMX2LFPIDX2O676LS3GY3PNZ4TA, seg_pos={0 0}, upload=83962]: Done uploading piece. err: ecclient: upload failed (node:125ZMc6ed9qgBRtBC5N4zNvGonip3GNAEBPt3auF6dBGteX46Pf, address:104.36.82.53:4128): protocol: expected piece hash; write tcp 159.223.143.39:39324->104.36.82.53:4128: use of closed network connection; write tcp 159.223.143.39:39324->104.36.82.53:4128: use of closed network connection; EOF Jun 23 01:19:36.705278262 (104h56m12.26860127s) [seg_pos={0 0}, upload=83962]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jun 23 01:19:36.728720635 (104h56m12.292043639s) [upload=83962]: Done uploading segments... Jun 23 01:19:36.728814444 (104h56m12.292137449s) [upload=83962]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 23.177380246s core uplink us-select-1 us1-placement-20

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)