Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [XX..........................................................] duration: 396.176759ms FAIL (2025-06-26 06:52:51 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 7) ======LOGS====== pload=55354]: Got next segment. Inline: false Jun 26 06:08:21.105827735 (68h56m48.032122411s) [upload=55354]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 26 06:08:21.105833588 (68h56m48.032128274s) [upload=55354]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 26 06:08:21.135318313 (68h56m48.061613009s) [seg_pos={0 0}, upload=55354]: Begin upload segment... Jun 26 06:08:21.135366534 (68h56m48.061661214s) [seg_pos={0 0}, upload=55354]: Done begin upload segment. Jun 26 06:08:21.135390342 (68h56m48.061685028s) [noise=true , address=45.91.116.9:5261 , piece_id=CZWB3XJVA7B7R7GWFYNW3RCGNVRXQ3E37XDXD4SUXTEMQFZ73HHA, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.135405801 (68h56m48.061700497s) [noise=true , address=104.36.82.6:4873 , piece_id=TIIMEQUSDLRFF3OK3WBB74F7SWJ5MEC4Z6IQNELDMRZLNTMQT7LA, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.135774729 (68h56m48.062069418s) [noise=true , address=77.90.31.13:4626 , piece_id=MXYBIEDFAFK53CVTE2C6CKEZWOL3MLW4S7LOHK4IHW6QEIZJL76Q, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.135944436 (68h56m48.06223913s) [noise=true , address=77.90.31.10:4517 , piece_id=VJZ5T4QNVAPNUHNFNK24S6TAS4V4NKNFMQE2FZMCWWLCCB2ELXXQ, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.136080402 (68h56m48.062375094s) [noise=true , address=77.90.31.15:4697 , piece_id=CUXGVL7PS3H7ZQKUCSYXJCBR3ETYFDPZFYIEAWEYL7C2S3RJY3AA, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.136382603 (68h56m48.062677289s) [noise=true , address=104.36.82.13:4186 , piece_id=5SVABN3KS6AUL6HILHNQCVF74BA75VPKTTAK7IQAPKRRME3MOZEA, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.136399304 (68h56m48.062693995s) [noise=true , address=104.36.82.14:4223 , piece_id=LP74YFI2SRSYPYEIKREVINCOFMVCXLNDS2555YKUVFN63N6TBJDA, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.136669576 (68h56m48.062964262s) [noise=true , address=104.36.82.6:4876 , piece_id=SSMGONONY34EHF5W6QMKO7YXB7RLO5LZVTJMMFKTP6GQNGYFYQ4A, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.136809784 (68h56m48.063104477s) [noise=true , address=104.36.82.15:4258 , piece_id=YXS4J7YSG6JV33JG4BZLK5AXKE3JXLBTGZJXPXZQ4LBYBT3MU2AA, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.136950556 (68h56m48.063245242s) [noise=true , address=77.90.31.11:4554 , piece_id=R2CIFLWS4LXTGSDSIMIRC4UYVJTXC7T35HPWZ2S7VYUTJ2AMY3BQ, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.137226497 (68h56m48.06352119s) [noise=true , address=77.90.31.14:4661 , piece_id=VXI2ZUSGR4N3H7TPIK75RIDFP3SBAOWVRGHNXTZOHFT5FNG4JWJA, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.138099107 (68h56m48.064393792s) [noise=true , address=45.91.116.10:4082 , piece_id=7NVEGA4QQOE7KYPHXUUX42SJPQL6LP46RGM37VD6A2AWN7PAOM3Q, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.139330448 (68h56m48.065625141s) [noise=true , address=104.36.82.14:4222 , piece_id=U3MBMAKFBFTOPJZ3HS3FXLPDBLQTN577XNCZLUBHRO6IU4BE5VNA, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.140580734 (68h56m48.06687543s) [noise=true , address=77.90.31.19:4842 , piece_id=G3AG375DATHB7MGEEWONE5GMMIC7QHG2VVDKF5AEVOSXV5N4S6AA, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.141452300 (68h56m48.067747027s) [noise=true , address=45.91.116.13:4154 , piece_id=STFPXN2LUJYUIAYROWZIRTCMT2Y3XET6KWA4GZWUHENXRJURP62Q, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.142917062 (68h56m48.069211739s) [noise=true , address=104.36.82.7:4912 , piece_id=A5HPC7CEU66IFNS47HO3AQX22LUZ6AZ4L4XAMZ6T5NCVE4D3TAZA, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.221824189 (68h56m48.148118865s) [upload=55354]: Next returned nil segment Jun 26 06:08:21.221837958 (68h56m48.148132644s) [upload=55354]: Waiting for error group managing segments... Jun 26 06:08:21.229550628 (68h56m48.155845315s) [noise=true , address=104.36.82.14:4223 , piece_id=LP74YFI2SRSYPYEIKREVINCOFMVCXLNDS2555YKUVFN63N6TBJDA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: <nil> Jun 26 06:08:21.230245145 (68h56m48.156539832s) [noise=true , address=104.36.82.14:4222 , piece_id=U3MBMAKFBFTOPJZ3HS3FXLPDBLQTN577XNCZLUBHRO6IU4BE5VNA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: <nil> Jun 26 06:08:21.230801067 (68h56m48.15709574s) [noise=true , address=104.36.82.6:4873 , piece_id=TIIMEQUSDLRFF3OK3WBB74F7SWJ5MEC4Z6IQNELDMRZLNTMQT7LA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: <nil> Jun 26 06:08:21.231457382 (68h56m48.157752071s) [noise=true , address=104.36.82.6:4876 , piece_id=SSMGONONY34EHF5W6QMKO7YXB7RLO5LZVTJMMFKTP6GQNGYFYQ4A, seg_pos={0 0}, upload=55354]: Done uploading piece. err: <nil> Jun 26 06:08:21.231563205 (68h56m48.157857893s) [noise=true , address=104.36.82.13:4186 , piece_id=5SVABN3KS6AUL6HILHNQCVF74BA75VPKTTAK7IQAPKRRME3MOZEA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: <nil> Jun 26 06:08:21.239131119 (68h56m48.165425799s) [noise=true , address=104.36.82.7:4912 , piece_id=A5HPC7CEU66IFNS47HO3AQX22LUZ6AZ4L4XAMZ6T5NCVE4D3TAZA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: <nil> Jun 26 06:08:21.534686665 (68h56m48.460981337s) [noise=true , address=104.36.82.15:4258 , piece_id=YXS4J7YSG6JV33JG4BZLK5AXKE3JXLBTGZJXPXZQ4LBYBT3MU2AA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: <nil> Jun 26 06:08:51.073673533 (68h57m17.999968219s) [noise=true , address=45.91.116.9:5261 , piece_id=CZWB3XJVA7B7R7GWFYNW3RCGNVRXQ3E37XDXD4SUXTEMQFZ73HHA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi, address:45.91.116.9:5261): protocol: rpc: dial tcp 45.91.116.9:5261: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:51.073739594 (68h57m18.000034272s) [noise=true , address=45.91.116.10:4082 , piece_id=7NVEGA4QQOE7KYPHXUUX42SJPQL6LP46RGM37VD6A2AWN7PAOM3Q, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj, address:45.91.116.10:4082): protocol: rpc: dial tcp 45.91.116.10:4082: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:51.073789897 (68h57m18.000084584s) [noise=true , address=77.90.31.19:4842 , piece_id=G3AG375DATHB7MGEEWONE5GMMIC7QHG2VVDKF5AEVOSXV5N4S6AA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: rpc: dial tcp 77.90.31.19:4842: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:51.073849570 (68h57m18.000144246s) [noise=true , address=45.91.116.13:4154 , piece_id=STFPXN2LUJYUIAYROWZIRTCMT2Y3XET6KWA4GZWUHENXRJURP62Q, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN, address:45.91.116.13:4154): protocol: write tcp 159.223.143.39:35392->45.91.116.13:4154: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:51.073904716 (68h57m18.000199391s) [noise=true , address=77.90.31.13:4626 , piece_id=MXYBIEDFAFK53CVTE2C6CKEZWOL3MLW4S7LOHK4IHW6QEIZJL76Q, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: write tcp 159.223.143.39:58864->77.90.31.13:4626: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:51.073952402 (68h57m18.000247082s) [noise=true , address=77.90.31.10:4517 , piece_id=VJZ5T4QNVAPNUHNFNK24S6TAS4V4NKNFMQE2FZMCWWLCCB2ELXXQ, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, address:77.90.31.10:4517): protocol: write tcp 159.223.143.39:36382->77.90.31.10:4517: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:51.074022481 (68h57m18.000317156s) [noise=true , address=77.90.31.15:4697 , piece_id=CUXGVL7PS3H7ZQKUCSYXJCBR3ETYFDPZFYIEAWEYL7C2S3RJY3AA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5, address:77.90.31.15:4697): protocol: write tcp 159.223.143.39:52730->77.90.31.15:4697: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:51.074071824 (68h57m18.000366503s) [noise=true , address=77.90.31.11:4554 , piece_id=R2CIFLWS4LXTGSDSIMIRC4UYVJTXC7T35HPWZ2S7VYUTJ2AMY3BQ, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, address:77.90.31.11:4554): protocol: rpc: dial tcp 77.90.31.11:4554: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:51.074102177 (68h57m18.000396855s) [noise=true , address=77.90.31.14:4661 , piece_id=VXI2ZUSGR4N3H7TPIK75RIDFP3SBAOWVRGHNXTZOHFT5FNG4JWJA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue, address:77.90.31.14:4661): protocol: rpc: dial tcp 77.90.31.14:4661: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:51.074119276 (68h57m18.000413966s) [seg_pos={0 0}, upload=55354]: Done waiting for segment. successful: 7 optimal: 13 errs: <nil> Jun 26 06:08:51.100079867 (68h57m18.026374541s) [upload=55354]: Done uploading segments... Jun 26 06:08:51.100114505 (68h57m18.026409191s) [upload=55354]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 7) Duration: 396.201455ms core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [X...........................................................] duration: 562.933467ms FAIL (2025-06-26 06:52:27 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 562.961025ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [XXX.........................................................] duration: 465.406849ms FAIL (2025-06-26 06:52:14 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 7) ======LOGS====== pload=55354]: Got next segment. Inline: false Jun 26 06:08:21.105827735 (68h56m48.032122411s) [upload=55354]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 26 06:08:21.105833588 (68h56m48.032128274s) [upload=55354]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 26 06:08:21.135318313 (68h56m48.061613009s) [seg_pos={0 0}, upload=55354]: Begin upload segment... Jun 26 06:08:21.135366534 (68h56m48.061661214s) [seg_pos={0 0}, upload=55354]: Done begin upload segment. Jun 26 06:08:21.135390342 (68h56m48.061685028s) [noise=true , address=45.91.116.9:5261 , piece_id=CZWB3XJVA7B7R7GWFYNW3RCGNVRXQ3E37XDXD4SUXTEMQFZ73HHA, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.135405801 (68h56m48.061700497s) [noise=true , address=104.36.82.6:4873 , piece_id=TIIMEQUSDLRFF3OK3WBB74F7SWJ5MEC4Z6IQNELDMRZLNTMQT7LA, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.135774729 (68h56m48.062069418s) [noise=true , address=77.90.31.13:4626 , piece_id=MXYBIEDFAFK53CVTE2C6CKEZWOL3MLW4S7LOHK4IHW6QEIZJL76Q, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.135944436 (68h56m48.06223913s) [noise=true , address=77.90.31.10:4517 , piece_id=VJZ5T4QNVAPNUHNFNK24S6TAS4V4NKNFMQE2FZMCWWLCCB2ELXXQ, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.136080402 (68h56m48.062375094s) [noise=true , address=77.90.31.15:4697 , piece_id=CUXGVL7PS3H7ZQKUCSYXJCBR3ETYFDPZFYIEAWEYL7C2S3RJY3AA, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.136382603 (68h56m48.062677289s) [noise=true , address=104.36.82.13:4186 , piece_id=5SVABN3KS6AUL6HILHNQCVF74BA75VPKTTAK7IQAPKRRME3MOZEA, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.136399304 (68h56m48.062693995s) [noise=true , address=104.36.82.14:4223 , piece_id=LP74YFI2SRSYPYEIKREVINCOFMVCXLNDS2555YKUVFN63N6TBJDA, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.136669576 (68h56m48.062964262s) [noise=true , address=104.36.82.6:4876 , piece_id=SSMGONONY34EHF5W6QMKO7YXB7RLO5LZVTJMMFKTP6GQNGYFYQ4A, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.136809784 (68h56m48.063104477s) [noise=true , address=104.36.82.15:4258 , piece_id=YXS4J7YSG6JV33JG4BZLK5AXKE3JXLBTGZJXPXZQ4LBYBT3MU2AA, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.136950556 (68h56m48.063245242s) [noise=true , address=77.90.31.11:4554 , piece_id=R2CIFLWS4LXTGSDSIMIRC4UYVJTXC7T35HPWZ2S7VYUTJ2AMY3BQ, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.137226497 (68h56m48.06352119s) [noise=true , address=77.90.31.14:4661 , piece_id=VXI2ZUSGR4N3H7TPIK75RIDFP3SBAOWVRGHNXTZOHFT5FNG4JWJA, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.138099107 (68h56m48.064393792s) [noise=true , address=45.91.116.10:4082 , piece_id=7NVEGA4QQOE7KYPHXUUX42SJPQL6LP46RGM37VD6A2AWN7PAOM3Q, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.139330448 (68h56m48.065625141s) [noise=true , address=104.36.82.14:4222 , piece_id=U3MBMAKFBFTOPJZ3HS3FXLPDBLQTN577XNCZLUBHRO6IU4BE5VNA, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.140580734 (68h56m48.06687543s) [noise=true , address=77.90.31.19:4842 , piece_id=G3AG375DATHB7MGEEWONE5GMMIC7QHG2VVDKF5AEVOSXV5N4S6AA, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.141452300 (68h56m48.067747027s) [noise=true , address=45.91.116.13:4154 , piece_id=STFPXN2LUJYUIAYROWZIRTCMT2Y3XET6KWA4GZWUHENXRJURP62Q, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.142917062 (68h56m48.069211739s) [noise=true , address=104.36.82.7:4912 , piece_id=A5HPC7CEU66IFNS47HO3AQX22LUZ6AZ4L4XAMZ6T5NCVE4D3TAZA, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.221824189 (68h56m48.148118865s) [upload=55354]: Next returned nil segment Jun 26 06:08:21.221837958 (68h56m48.148132644s) [upload=55354]: Waiting for error group managing segments... Jun 26 06:08:21.229550628 (68h56m48.155845315s) [noise=true , address=104.36.82.14:4223 , piece_id=LP74YFI2SRSYPYEIKREVINCOFMVCXLNDS2555YKUVFN63N6TBJDA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: <nil> Jun 26 06:08:21.230245145 (68h56m48.156539832s) [noise=true , address=104.36.82.14:4222 , piece_id=U3MBMAKFBFTOPJZ3HS3FXLPDBLQTN577XNCZLUBHRO6IU4BE5VNA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: <nil> Jun 26 06:08:21.230801067 (68h56m48.15709574s) [noise=true , address=104.36.82.6:4873 , piece_id=TIIMEQUSDLRFF3OK3WBB74F7SWJ5MEC4Z6IQNELDMRZLNTMQT7LA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: <nil> Jun 26 06:08:21.231457382 (68h56m48.157752071s) [noise=true , address=104.36.82.6:4876 , piece_id=SSMGONONY34EHF5W6QMKO7YXB7RLO5LZVTJMMFKTP6GQNGYFYQ4A, seg_pos={0 0}, upload=55354]: Done uploading piece. err: <nil> Jun 26 06:08:21.231563205 (68h56m48.157857893s) [noise=true , address=104.36.82.13:4186 , piece_id=5SVABN3KS6AUL6HILHNQCVF74BA75VPKTTAK7IQAPKRRME3MOZEA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: <nil> Jun 26 06:08:21.239131119 (68h56m48.165425799s) [noise=true , address=104.36.82.7:4912 , piece_id=A5HPC7CEU66IFNS47HO3AQX22LUZ6AZ4L4XAMZ6T5NCVE4D3TAZA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: <nil> Jun 26 06:08:21.534686665 (68h56m48.460981337s) [noise=true , address=104.36.82.15:4258 , piece_id=YXS4J7YSG6JV33JG4BZLK5AXKE3JXLBTGZJXPXZQ4LBYBT3MU2AA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: <nil> Jun 26 06:08:51.073673533 (68h57m17.999968219s) [noise=true , address=45.91.116.9:5261 , piece_id=CZWB3XJVA7B7R7GWFYNW3RCGNVRXQ3E37XDXD4SUXTEMQFZ73HHA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi, address:45.91.116.9:5261): protocol: rpc: dial tcp 45.91.116.9:5261: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:51.073739594 (68h57m18.000034272s) [noise=true , address=45.91.116.10:4082 , piece_id=7NVEGA4QQOE7KYPHXUUX42SJPQL6LP46RGM37VD6A2AWN7PAOM3Q, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj, address:45.91.116.10:4082): protocol: rpc: dial tcp 45.91.116.10:4082: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:51.073789897 (68h57m18.000084584s) [noise=true , address=77.90.31.19:4842 , piece_id=G3AG375DATHB7MGEEWONE5GMMIC7QHG2VVDKF5AEVOSXV5N4S6AA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: rpc: dial tcp 77.90.31.19:4842: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:51.073849570 (68h57m18.000144246s) [noise=true , address=45.91.116.13:4154 , piece_id=STFPXN2LUJYUIAYROWZIRTCMT2Y3XET6KWA4GZWUHENXRJURP62Q, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN, address:45.91.116.13:4154): protocol: write tcp 159.223.143.39:35392->45.91.116.13:4154: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:51.073904716 (68h57m18.000199391s) [noise=true , address=77.90.31.13:4626 , piece_id=MXYBIEDFAFK53CVTE2C6CKEZWOL3MLW4S7LOHK4IHW6QEIZJL76Q, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: write tcp 159.223.143.39:58864->77.90.31.13:4626: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:51.073952402 (68h57m18.000247082s) [noise=true , address=77.90.31.10:4517 , piece_id=VJZ5T4QNVAPNUHNFNK24S6TAS4V4NKNFMQE2FZMCWWLCCB2ELXXQ, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, address:77.90.31.10:4517): protocol: write tcp 159.223.143.39:36382->77.90.31.10:4517: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:51.074022481 (68h57m18.000317156s) [noise=true , address=77.90.31.15:4697 , piece_id=CUXGVL7PS3H7ZQKUCSYXJCBR3ETYFDPZFYIEAWEYL7C2S3RJY3AA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5, address:77.90.31.15:4697): protocol: write tcp 159.223.143.39:52730->77.90.31.15:4697: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:51.074071824 (68h57m18.000366503s) [noise=true , address=77.90.31.11:4554 , piece_id=R2CIFLWS4LXTGSDSIMIRC4UYVJTXC7T35HPWZ2S7VYUTJ2AMY3BQ, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, address:77.90.31.11:4554): protocol: rpc: dial tcp 77.90.31.11:4554: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:51.074102177 (68h57m18.000396855s) [noise=true , address=77.90.31.14:4661 , piece_id=VXI2ZUSGR4N3H7TPIK75RIDFP3SBAOWVRGHNXTZOHFT5FNG4JWJA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue, address:77.90.31.14:4661): protocol: rpc: dial tcp 77.90.31.14:4661: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:51.074119276 (68h57m18.000413966s) [seg_pos={0 0}, upload=55354]: Done waiting for segment. successful: 7 optimal: 13 errs: <nil> Jun 26 06:08:51.100079867 (68h57m18.026374541s) [upload=55354]: Done uploading segments... Jun 26 06:08:51.100114505 (68h57m18.026409191s) [upload=55354]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 7) Duration: 465.459683ms core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [XX..........................................................] duration: 494.034035ms FAIL (2025-06-26 06:51:51 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 494.058778ms edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.XX.........................................................] duration: 525.915498ms FAIL (2025-06-26 06:51:35 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 525.98924ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.XXX........................................................] duration: 472.448763ms FAIL (2025-06-26 06:51:07 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 7) ======LOGS====== pload=55354]: Got next segment. Inline: false Jun 26 06:08:21.105827735 (68h56m48.032122411s) [upload=55354]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 26 06:08:21.105833588 (68h56m48.032128274s) [upload=55354]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 26 06:08:21.135318313 (68h56m48.061613009s) [seg_pos={0 0}, upload=55354]: Begin upload segment... Jun 26 06:08:21.135366534 (68h56m48.061661214s) [seg_pos={0 0}, upload=55354]: Done begin upload segment. Jun 26 06:08:21.135390342 (68h56m48.061685028s) [noise=true , address=45.91.116.9:5261 , piece_id=CZWB3XJVA7B7R7GWFYNW3RCGNVRXQ3E37XDXD4SUXTEMQFZ73HHA, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.135405801 (68h56m48.061700497s) [noise=true , address=104.36.82.6:4873 , piece_id=TIIMEQUSDLRFF3OK3WBB74F7SWJ5MEC4Z6IQNELDMRZLNTMQT7LA, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.135774729 (68h56m48.062069418s) [noise=true , address=77.90.31.13:4626 , piece_id=MXYBIEDFAFK53CVTE2C6CKEZWOL3MLW4S7LOHK4IHW6QEIZJL76Q, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.135944436 (68h56m48.06223913s) [noise=true , address=77.90.31.10:4517 , piece_id=VJZ5T4QNVAPNUHNFNK24S6TAS4V4NKNFMQE2FZMCWWLCCB2ELXXQ, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.136080402 (68h56m48.062375094s) [noise=true , address=77.90.31.15:4697 , piece_id=CUXGVL7PS3H7ZQKUCSYXJCBR3ETYFDPZFYIEAWEYL7C2S3RJY3AA, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.136382603 (68h56m48.062677289s) [noise=true , address=104.36.82.13:4186 , piece_id=5SVABN3KS6AUL6HILHNQCVF74BA75VPKTTAK7IQAPKRRME3MOZEA, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.136399304 (68h56m48.062693995s) [noise=true , address=104.36.82.14:4223 , piece_id=LP74YFI2SRSYPYEIKREVINCOFMVCXLNDS2555YKUVFN63N6TBJDA, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.136669576 (68h56m48.062964262s) [noise=true , address=104.36.82.6:4876 , piece_id=SSMGONONY34EHF5W6QMKO7YXB7RLO5LZVTJMMFKTP6GQNGYFYQ4A, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.136809784 (68h56m48.063104477s) [noise=true , address=104.36.82.15:4258 , piece_id=YXS4J7YSG6JV33JG4BZLK5AXKE3JXLBTGZJXPXZQ4LBYBT3MU2AA, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.136950556 (68h56m48.063245242s) [noise=true , address=77.90.31.11:4554 , piece_id=R2CIFLWS4LXTGSDSIMIRC4UYVJTXC7T35HPWZ2S7VYUTJ2AMY3BQ, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.137226497 (68h56m48.06352119s) [noise=true , address=77.90.31.14:4661 , piece_id=VXI2ZUSGR4N3H7TPIK75RIDFP3SBAOWVRGHNXTZOHFT5FNG4JWJA, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.138099107 (68h56m48.064393792s) [noise=true , address=45.91.116.10:4082 , piece_id=7NVEGA4QQOE7KYPHXUUX42SJPQL6LP46RGM37VD6A2AWN7PAOM3Q, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.139330448 (68h56m48.065625141s) [noise=true , address=104.36.82.14:4222 , piece_id=U3MBMAKFBFTOPJZ3HS3FXLPDBLQTN577XNCZLUBHRO6IU4BE5VNA, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.140580734 (68h56m48.06687543s) [noise=true , address=77.90.31.19:4842 , piece_id=G3AG375DATHB7MGEEWONE5GMMIC7QHG2VVDKF5AEVOSXV5N4S6AA, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.141452300 (68h56m48.067747027s) [noise=true , address=45.91.116.13:4154 , piece_id=STFPXN2LUJYUIAYROWZIRTCMT2Y3XET6KWA4GZWUHENXRJURP62Q, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.142917062 (68h56m48.069211739s) [noise=true , address=104.36.82.7:4912 , piece_id=A5HPC7CEU66IFNS47HO3AQX22LUZ6AZ4L4XAMZ6T5NCVE4D3TAZA, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.221824189 (68h56m48.148118865s) [upload=55354]: Next returned nil segment Jun 26 06:08:21.221837958 (68h56m48.148132644s) [upload=55354]: Waiting for error group managing segments... Jun 26 06:08:21.229550628 (68h56m48.155845315s) [noise=true , address=104.36.82.14:4223 , piece_id=LP74YFI2SRSYPYEIKREVINCOFMVCXLNDS2555YKUVFN63N6TBJDA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: <nil> Jun 26 06:08:21.230245145 (68h56m48.156539832s) [noise=true , address=104.36.82.14:4222 , piece_id=U3MBMAKFBFTOPJZ3HS3FXLPDBLQTN577XNCZLUBHRO6IU4BE5VNA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: <nil> Jun 26 06:08:21.230801067 (68h56m48.15709574s) [noise=true , address=104.36.82.6:4873 , piece_id=TIIMEQUSDLRFF3OK3WBB74F7SWJ5MEC4Z6IQNELDMRZLNTMQT7LA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: <nil> Jun 26 06:08:21.231457382 (68h56m48.157752071s) [noise=true , address=104.36.82.6:4876 , piece_id=SSMGONONY34EHF5W6QMKO7YXB7RLO5LZVTJMMFKTP6GQNGYFYQ4A, seg_pos={0 0}, upload=55354]: Done uploading piece. err: <nil> Jun 26 06:08:21.231563205 (68h56m48.157857893s) [noise=true , address=104.36.82.13:4186 , piece_id=5SVABN3KS6AUL6HILHNQCVF74BA75VPKTTAK7IQAPKRRME3MOZEA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: <nil> Jun 26 06:08:21.239131119 (68h56m48.165425799s) [noise=true , address=104.36.82.7:4912 , piece_id=A5HPC7CEU66IFNS47HO3AQX22LUZ6AZ4L4XAMZ6T5NCVE4D3TAZA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: <nil> Jun 26 06:08:21.534686665 (68h56m48.460981337s) [noise=true , address=104.36.82.15:4258 , piece_id=YXS4J7YSG6JV33JG4BZLK5AXKE3JXLBTGZJXPXZQ4LBYBT3MU2AA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: <nil> Jun 26 06:08:51.073673533 (68h57m17.999968219s) [noise=true , address=45.91.116.9:5261 , piece_id=CZWB3XJVA7B7R7GWFYNW3RCGNVRXQ3E37XDXD4SUXTEMQFZ73HHA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi, address:45.91.116.9:5261): protocol: rpc: dial tcp 45.91.116.9:5261: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:51.073739594 (68h57m18.000034272s) [noise=true , address=45.91.116.10:4082 , piece_id=7NVEGA4QQOE7KYPHXUUX42SJPQL6LP46RGM37VD6A2AWN7PAOM3Q, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj, address:45.91.116.10:4082): protocol: rpc: dial tcp 45.91.116.10:4082: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:51.073789897 (68h57m18.000084584s) [noise=true , address=77.90.31.19:4842 , piece_id=G3AG375DATHB7MGEEWONE5GMMIC7QHG2VVDKF5AEVOSXV5N4S6AA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: rpc: dial tcp 77.90.31.19:4842: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:51.073849570 (68h57m18.000144246s) [noise=true , address=45.91.116.13:4154 , piece_id=STFPXN2LUJYUIAYROWZIRTCMT2Y3XET6KWA4GZWUHENXRJURP62Q, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN, address:45.91.116.13:4154): protocol: write tcp 159.223.143.39:35392->45.91.116.13:4154: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:51.073904716 (68h57m18.000199391s) [noise=true , address=77.90.31.13:4626 , piece_id=MXYBIEDFAFK53CVTE2C6CKEZWOL3MLW4S7LOHK4IHW6QEIZJL76Q, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: write tcp 159.223.143.39:58864->77.90.31.13:4626: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:51.073952402 (68h57m18.000247082s) [noise=true , address=77.90.31.10:4517 , piece_id=VJZ5T4QNVAPNUHNFNK24S6TAS4V4NKNFMQE2FZMCWWLCCB2ELXXQ, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, address:77.90.31.10:4517): protocol: write tcp 159.223.143.39:36382->77.90.31.10:4517: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:51.074022481 (68h57m18.000317156s) [noise=true , address=77.90.31.15:4697 , piece_id=CUXGVL7PS3H7ZQKUCSYXJCBR3ETYFDPZFYIEAWEYL7C2S3RJY3AA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5, address:77.90.31.15:4697): protocol: write tcp 159.223.143.39:52730->77.90.31.15:4697: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:51.074071824 (68h57m18.000366503s) [noise=true , address=77.90.31.11:4554 , piece_id=R2CIFLWS4LXTGSDSIMIRC4UYVJTXC7T35HPWZ2S7VYUTJ2AMY3BQ, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, address:77.90.31.11:4554): protocol: rpc: dial tcp 77.90.31.11:4554: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:51.074102177 (68h57m18.000396855s) [noise=true , address=77.90.31.14:4661 , piece_id=VXI2ZUSGR4N3H7TPIK75RIDFP3SBAOWVRGHNXTZOHFT5FNG4JWJA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue, address:77.90.31.14:4661): protocol: rpc: dial tcp 77.90.31.14:4661: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:51.074119276 (68h57m18.000413966s) [seg_pos={0 0}, upload=55354]: Done waiting for segment. successful: 7 optimal: 13 errs: <nil> Jun 26 06:08:51.100079867 (68h57m18.026374541s) [upload=55354]: Done uploading segments... Jun 26 06:08:51.100114505 (68h57m18.026409191s) [upload=55354]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 7) Duration: 472.477271ms core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..XX........................................................] duration: 549.021842ms FAIL (2025-06-26 06:50:40 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 549.072058ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [..XXX.......................................................] duration: 447.657341ms FAIL (2025-06-26 06:50:33 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 7) ======LOGS====== pload=55354]: Got next segment. Inline: false Jun 26 06:08:21.105827735 (68h56m48.032122411s) [upload=55354]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 26 06:08:21.105833588 (68h56m48.032128274s) [upload=55354]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 26 06:08:21.135318313 (68h56m48.061613009s) [seg_pos={0 0}, upload=55354]: Begin upload segment... Jun 26 06:08:21.135366534 (68h56m48.061661214s) [seg_pos={0 0}, upload=55354]: Done begin upload segment. Jun 26 06:08:21.135390342 (68h56m48.061685028s) [noise=true , address=45.91.116.9:5261 , piece_id=CZWB3XJVA7B7R7GWFYNW3RCGNVRXQ3E37XDXD4SUXTEMQFZ73HHA, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.135405801 (68h56m48.061700497s) [noise=true , address=104.36.82.6:4873 , piece_id=TIIMEQUSDLRFF3OK3WBB74F7SWJ5MEC4Z6IQNELDMRZLNTMQT7LA, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.135774729 (68h56m48.062069418s) [noise=true , address=77.90.31.13:4626 , piece_id=MXYBIEDFAFK53CVTE2C6CKEZWOL3MLW4S7LOHK4IHW6QEIZJL76Q, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.135944436 (68h56m48.06223913s) [noise=true , address=77.90.31.10:4517 , piece_id=VJZ5T4QNVAPNUHNFNK24S6TAS4V4NKNFMQE2FZMCWWLCCB2ELXXQ, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.136080402 (68h56m48.062375094s) [noise=true , address=77.90.31.15:4697 , piece_id=CUXGVL7PS3H7ZQKUCSYXJCBR3ETYFDPZFYIEAWEYL7C2S3RJY3AA, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.136382603 (68h56m48.062677289s) [noise=true , address=104.36.82.13:4186 , piece_id=5SVABN3KS6AUL6HILHNQCVF74BA75VPKTTAK7IQAPKRRME3MOZEA, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.136399304 (68h56m48.062693995s) [noise=true , address=104.36.82.14:4223 , piece_id=LP74YFI2SRSYPYEIKREVINCOFMVCXLNDS2555YKUVFN63N6TBJDA, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.136669576 (68h56m48.062964262s) [noise=true , address=104.36.82.6:4876 , piece_id=SSMGONONY34EHF5W6QMKO7YXB7RLO5LZVTJMMFKTP6GQNGYFYQ4A, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.136809784 (68h56m48.063104477s) [noise=true , address=104.36.82.15:4258 , piece_id=YXS4J7YSG6JV33JG4BZLK5AXKE3JXLBTGZJXPXZQ4LBYBT3MU2AA, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.136950556 (68h56m48.063245242s) [noise=true , address=77.90.31.11:4554 , piece_id=R2CIFLWS4LXTGSDSIMIRC4UYVJTXC7T35HPWZ2S7VYUTJ2AMY3BQ, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.137226497 (68h56m48.06352119s) [noise=true , address=77.90.31.14:4661 , piece_id=VXI2ZUSGR4N3H7TPIK75RIDFP3SBAOWVRGHNXTZOHFT5FNG4JWJA, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.138099107 (68h56m48.064393792s) [noise=true , address=45.91.116.10:4082 , piece_id=7NVEGA4QQOE7KYPHXUUX42SJPQL6LP46RGM37VD6A2AWN7PAOM3Q, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.139330448 (68h56m48.065625141s) [noise=true , address=104.36.82.14:4222 , piece_id=U3MBMAKFBFTOPJZ3HS3FXLPDBLQTN577XNCZLUBHRO6IU4BE5VNA, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.140580734 (68h56m48.06687543s) [noise=true , address=77.90.31.19:4842 , piece_id=G3AG375DATHB7MGEEWONE5GMMIC7QHG2VVDKF5AEVOSXV5N4S6AA, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.141452300 (68h56m48.067747027s) [noise=true , address=45.91.116.13:4154 , piece_id=STFPXN2LUJYUIAYROWZIRTCMT2Y3XET6KWA4GZWUHENXRJURP62Q, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.142917062 (68h56m48.069211739s) [noise=true , address=104.36.82.7:4912 , piece_id=A5HPC7CEU66IFNS47HO3AQX22LUZ6AZ4L4XAMZ6T5NCVE4D3TAZA, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.221824189 (68h56m48.148118865s) [upload=55354]: Next returned nil segment Jun 26 06:08:21.221837958 (68h56m48.148132644s) [upload=55354]: Waiting for error group managing segments... Jun 26 06:08:21.229550628 (68h56m48.155845315s) [noise=true , address=104.36.82.14:4223 , piece_id=LP74YFI2SRSYPYEIKREVINCOFMVCXLNDS2555YKUVFN63N6TBJDA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: <nil> Jun 26 06:08:21.230245145 (68h56m48.156539832s) [noise=true , address=104.36.82.14:4222 , piece_id=U3MBMAKFBFTOPJZ3HS3FXLPDBLQTN577XNCZLUBHRO6IU4BE5VNA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: <nil> Jun 26 06:08:21.230801067 (68h56m48.15709574s) [noise=true , address=104.36.82.6:4873 , piece_id=TIIMEQUSDLRFF3OK3WBB74F7SWJ5MEC4Z6IQNELDMRZLNTMQT7LA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: <nil> Jun 26 06:08:21.231457382 (68h56m48.157752071s) [noise=true , address=104.36.82.6:4876 , piece_id=SSMGONONY34EHF5W6QMKO7YXB7RLO5LZVTJMMFKTP6GQNGYFYQ4A, seg_pos={0 0}, upload=55354]: Done uploading piece. err: <nil> Jun 26 06:08:21.231563205 (68h56m48.157857893s) [noise=true , address=104.36.82.13:4186 , piece_id=5SVABN3KS6AUL6HILHNQCVF74BA75VPKTTAK7IQAPKRRME3MOZEA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: <nil> Jun 26 06:08:21.239131119 (68h56m48.165425799s) [noise=true , address=104.36.82.7:4912 , piece_id=A5HPC7CEU66IFNS47HO3AQX22LUZ6AZ4L4XAMZ6T5NCVE4D3TAZA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: <nil> Jun 26 06:08:21.534686665 (68h56m48.460981337s) [noise=true , address=104.36.82.15:4258 , piece_id=YXS4J7YSG6JV33JG4BZLK5AXKE3JXLBTGZJXPXZQ4LBYBT3MU2AA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: <nil> Jun 26 06:08:51.073673533 (68h57m17.999968219s) [noise=true , address=45.91.116.9:5261 , piece_id=CZWB3XJVA7B7R7GWFYNW3RCGNVRXQ3E37XDXD4SUXTEMQFZ73HHA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi, address:45.91.116.9:5261): protocol: rpc: dial tcp 45.91.116.9:5261: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:51.073739594 (68h57m18.000034272s) [noise=true , address=45.91.116.10:4082 , piece_id=7NVEGA4QQOE7KYPHXUUX42SJPQL6LP46RGM37VD6A2AWN7PAOM3Q, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj, address:45.91.116.10:4082): protocol: rpc: dial tcp 45.91.116.10:4082: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:51.073789897 (68h57m18.000084584s) [noise=true , address=77.90.31.19:4842 , piece_id=G3AG375DATHB7MGEEWONE5GMMIC7QHG2VVDKF5AEVOSXV5N4S6AA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: rpc: dial tcp 77.90.31.19:4842: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:51.073849570 (68h57m18.000144246s) [noise=true , address=45.91.116.13:4154 , piece_id=STFPXN2LUJYUIAYROWZIRTCMT2Y3XET6KWA4GZWUHENXRJURP62Q, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN, address:45.91.116.13:4154): protocol: write tcp 159.223.143.39:35392->45.91.116.13:4154: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:51.073904716 (68h57m18.000199391s) [noise=true , address=77.90.31.13:4626 , piece_id=MXYBIEDFAFK53CVTE2C6CKEZWOL3MLW4S7LOHK4IHW6QEIZJL76Q, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: write tcp 159.223.143.39:58864->77.90.31.13:4626: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:51.073952402 (68h57m18.000247082s) [noise=true , address=77.90.31.10:4517 , piece_id=VJZ5T4QNVAPNUHNFNK24S6TAS4V4NKNFMQE2FZMCWWLCCB2ELXXQ, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, address:77.90.31.10:4517): protocol: write tcp 159.223.143.39:36382->77.90.31.10:4517: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:51.074022481 (68h57m18.000317156s) [noise=true , address=77.90.31.15:4697 , piece_id=CUXGVL7PS3H7ZQKUCSYXJCBR3ETYFDPZFYIEAWEYL7C2S3RJY3AA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5, address:77.90.31.15:4697): protocol: write tcp 159.223.143.39:52730->77.90.31.15:4697: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:51.074071824 (68h57m18.000366503s) [noise=true , address=77.90.31.11:4554 , piece_id=R2CIFLWS4LXTGSDSIMIRC4UYVJTXC7T35HPWZ2S7VYUTJ2AMY3BQ, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, address:77.90.31.11:4554): protocol: rpc: dial tcp 77.90.31.11:4554: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:51.074102177 (68h57m18.000396855s) [noise=true , address=77.90.31.14:4661 , piece_id=VXI2ZUSGR4N3H7TPIK75RIDFP3SBAOWVRGHNXTZOHFT5FNG4JWJA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue, address:77.90.31.14:4661): protocol: rpc: dial tcp 77.90.31.14:4661: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:51.074119276 (68h57m18.000413966s) [seg_pos={0 0}, upload=55354]: Done waiting for segment. successful: 7 optimal: 13 errs: <nil> Jun 26 06:08:51.100079867 (68h57m18.026374541s) [upload=55354]: Done uploading segments... Jun 26 06:08:51.100114505 (68h57m18.026409191s) [upload=55354]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 7) Duration: 447.69027ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [...XXX......................................................] duration: 408.395974ms FAIL (2025-06-26 06:49:47 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 7) ======LOGS====== pload=55354]: Got next segment. Inline: false Jun 26 06:08:21.105827735 (68h56m48.032122411s) [upload=55354]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 26 06:08:21.105833588 (68h56m48.032128274s) [upload=55354]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 26 06:08:21.135318313 (68h56m48.061613009s) [seg_pos={0 0}, upload=55354]: Begin upload segment... Jun 26 06:08:21.135366534 (68h56m48.061661214s) [seg_pos={0 0}, upload=55354]: Done begin upload segment. Jun 26 06:08:21.135390342 (68h56m48.061685028s) [noise=true , address=45.91.116.9:5261 , piece_id=CZWB3XJVA7B7R7GWFYNW3RCGNVRXQ3E37XDXD4SUXTEMQFZ73HHA, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.135405801 (68h56m48.061700497s) [noise=true , address=104.36.82.6:4873 , piece_id=TIIMEQUSDLRFF3OK3WBB74F7SWJ5MEC4Z6IQNELDMRZLNTMQT7LA, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.135774729 (68h56m48.062069418s) [noise=true , address=77.90.31.13:4626 , piece_id=MXYBIEDFAFK53CVTE2C6CKEZWOL3MLW4S7LOHK4IHW6QEIZJL76Q, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.135944436 (68h56m48.06223913s) [noise=true , address=77.90.31.10:4517 , piece_id=VJZ5T4QNVAPNUHNFNK24S6TAS4V4NKNFMQE2FZMCWWLCCB2ELXXQ, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.136080402 (68h56m48.062375094s) [noise=true , address=77.90.31.15:4697 , piece_id=CUXGVL7PS3H7ZQKUCSYXJCBR3ETYFDPZFYIEAWEYL7C2S3RJY3AA, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.136382603 (68h56m48.062677289s) [noise=true , address=104.36.82.13:4186 , piece_id=5SVABN3KS6AUL6HILHNQCVF74BA75VPKTTAK7IQAPKRRME3MOZEA, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.136399304 (68h56m48.062693995s) [noise=true , address=104.36.82.14:4223 , piece_id=LP74YFI2SRSYPYEIKREVINCOFMVCXLNDS2555YKUVFN63N6TBJDA, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.136669576 (68h56m48.062964262s) [noise=true , address=104.36.82.6:4876 , piece_id=SSMGONONY34EHF5W6QMKO7YXB7RLO5LZVTJMMFKTP6GQNGYFYQ4A, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.136809784 (68h56m48.063104477s) [noise=true , address=104.36.82.15:4258 , piece_id=YXS4J7YSG6JV33JG4BZLK5AXKE3JXLBTGZJXPXZQ4LBYBT3MU2AA, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.136950556 (68h56m48.063245242s) [noise=true , address=77.90.31.11:4554 , piece_id=R2CIFLWS4LXTGSDSIMIRC4UYVJTXC7T35HPWZ2S7VYUTJ2AMY3BQ, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.137226497 (68h56m48.06352119s) [noise=true , address=77.90.31.14:4661 , piece_id=VXI2ZUSGR4N3H7TPIK75RIDFP3SBAOWVRGHNXTZOHFT5FNG4JWJA, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.138099107 (68h56m48.064393792s) [noise=true , address=45.91.116.10:4082 , piece_id=7NVEGA4QQOE7KYPHXUUX42SJPQL6LP46RGM37VD6A2AWN7PAOM3Q, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.139330448 (68h56m48.065625141s) [noise=true , address=104.36.82.14:4222 , piece_id=U3MBMAKFBFTOPJZ3HS3FXLPDBLQTN577XNCZLUBHRO6IU4BE5VNA, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.140580734 (68h56m48.06687543s) [noise=true , address=77.90.31.19:4842 , piece_id=G3AG375DATHB7MGEEWONE5GMMIC7QHG2VVDKF5AEVOSXV5N4S6AA, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.141452300 (68h56m48.067747027s) [noise=true , address=45.91.116.13:4154 , piece_id=STFPXN2LUJYUIAYROWZIRTCMT2Y3XET6KWA4GZWUHENXRJURP62Q, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.142917062 (68h56m48.069211739s) [noise=true , address=104.36.82.7:4912 , piece_id=A5HPC7CEU66IFNS47HO3AQX22LUZ6AZ4L4XAMZ6T5NCVE4D3TAZA, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.221824189 (68h56m48.148118865s) [upload=55354]: Next returned nil segment Jun 26 06:08:21.221837958 (68h56m48.148132644s) [upload=55354]: Waiting for error group managing segments... Jun 26 06:08:21.229550628 (68h56m48.155845315s) [noise=true , address=104.36.82.14:4223 , piece_id=LP74YFI2SRSYPYEIKREVINCOFMVCXLNDS2555YKUVFN63N6TBJDA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: <nil> Jun 26 06:08:21.230245145 (68h56m48.156539832s) [noise=true , address=104.36.82.14:4222 , piece_id=U3MBMAKFBFTOPJZ3HS3FXLPDBLQTN577XNCZLUBHRO6IU4BE5VNA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: <nil> Jun 26 06:08:21.230801067 (68h56m48.15709574s) [noise=true , address=104.36.82.6:4873 , piece_id=TIIMEQUSDLRFF3OK3WBB74F7SWJ5MEC4Z6IQNELDMRZLNTMQT7LA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: <nil> Jun 26 06:08:21.231457382 (68h56m48.157752071s) [noise=true , address=104.36.82.6:4876 , piece_id=SSMGONONY34EHF5W6QMKO7YXB7RLO5LZVTJMMFKTP6GQNGYFYQ4A, seg_pos={0 0}, upload=55354]: Done uploading piece. err: <nil> Jun 26 06:08:21.231563205 (68h56m48.157857893s) [noise=true , address=104.36.82.13:4186 , piece_id=5SVABN3KS6AUL6HILHNQCVF74BA75VPKTTAK7IQAPKRRME3MOZEA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: <nil> Jun 26 06:08:21.239131119 (68h56m48.165425799s) [noise=true , address=104.36.82.7:4912 , piece_id=A5HPC7CEU66IFNS47HO3AQX22LUZ6AZ4L4XAMZ6T5NCVE4D3TAZA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: <nil> Jun 26 06:08:21.534686665 (68h56m48.460981337s) [noise=true , address=104.36.82.15:4258 , piece_id=YXS4J7YSG6JV33JG4BZLK5AXKE3JXLBTGZJXPXZQ4LBYBT3MU2AA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: <nil> Jun 26 06:08:51.073673533 (68h57m17.999968219s) [noise=true , address=45.91.116.9:5261 , piece_id=CZWB3XJVA7B7R7GWFYNW3RCGNVRXQ3E37XDXD4SUXTEMQFZ73HHA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi, address:45.91.116.9:5261): protocol: rpc: dial tcp 45.91.116.9:5261: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:51.073739594 (68h57m18.000034272s) [noise=true , address=45.91.116.10:4082 , piece_id=7NVEGA4QQOE7KYPHXUUX42SJPQL6LP46RGM37VD6A2AWN7PAOM3Q, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj, address:45.91.116.10:4082): protocol: rpc: dial tcp 45.91.116.10:4082: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:51.073789897 (68h57m18.000084584s) [noise=true , address=77.90.31.19:4842 , piece_id=G3AG375DATHB7MGEEWONE5GMMIC7QHG2VVDKF5AEVOSXV5N4S6AA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: rpc: dial tcp 77.90.31.19:4842: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:51.073849570 (68h57m18.000144246s) [noise=true , address=45.91.116.13:4154 , piece_id=STFPXN2LUJYUIAYROWZIRTCMT2Y3XET6KWA4GZWUHENXRJURP62Q, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN, address:45.91.116.13:4154): protocol: write tcp 159.223.143.39:35392->45.91.116.13:4154: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:51.073904716 (68h57m18.000199391s) [noise=true , address=77.90.31.13:4626 , piece_id=MXYBIEDFAFK53CVTE2C6CKEZWOL3MLW4S7LOHK4IHW6QEIZJL76Q, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: write tcp 159.223.143.39:58864->77.90.31.13:4626: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:51.073952402 (68h57m18.000247082s) [noise=true , address=77.90.31.10:4517 , piece_id=VJZ5T4QNVAPNUHNFNK24S6TAS4V4NKNFMQE2FZMCWWLCCB2ELXXQ, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, address:77.90.31.10:4517): protocol: write tcp 159.223.143.39:36382->77.90.31.10:4517: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:51.074022481 (68h57m18.000317156s) [noise=true , address=77.90.31.15:4697 , piece_id=CUXGVL7PS3H7ZQKUCSYXJCBR3ETYFDPZFYIEAWEYL7C2S3RJY3AA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5, address:77.90.31.15:4697): protocol: write tcp 159.223.143.39:52730->77.90.31.15:4697: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:51.074071824 (68h57m18.000366503s) [noise=true , address=77.90.31.11:4554 , piece_id=R2CIFLWS4LXTGSDSIMIRC4UYVJTXC7T35HPWZ2S7VYUTJ2AMY3BQ, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, address:77.90.31.11:4554): protocol: rpc: dial tcp 77.90.31.11:4554: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:51.074102177 (68h57m18.000396855s) [noise=true , address=77.90.31.14:4661 , piece_id=VXI2ZUSGR4N3H7TPIK75RIDFP3SBAOWVRGHNXTZOHFT5FNG4JWJA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue, address:77.90.31.14:4661): protocol: rpc: dial tcp 77.90.31.14:4661: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:51.074119276 (68h57m18.000413966s) [seg_pos={0 0}, upload=55354]: Done waiting for segment. successful: 7 optimal: 13 errs: <nil> Jun 26 06:08:51.100079867 (68h57m18.026374541s) [upload=55354]: Done uploading segments... Jun 26 06:08:51.100114505 (68h57m18.026409191s) [upload=55354]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 7) Duration: 408.489039ms core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...XX.......................................................] duration: 651.850819ms FAIL (2025-06-26 06:49:34 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 651.923228ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [....XXX.....................................................] duration: 455.054884ms FAIL (2025-06-26 06:49:21 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 7) ======LOGS====== pload=55354]: Got next segment. Inline: false Jun 26 06:08:21.105827735 (68h56m48.032122411s) [upload=55354]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 26 06:08:21.105833588 (68h56m48.032128274s) [upload=55354]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 26 06:08:21.135318313 (68h56m48.061613009s) [seg_pos={0 0}, upload=55354]: Begin upload segment... Jun 26 06:08:21.135366534 (68h56m48.061661214s) [seg_pos={0 0}, upload=55354]: Done begin upload segment. Jun 26 06:08:21.135390342 (68h56m48.061685028s) [noise=true , address=45.91.116.9:5261 , piece_id=CZWB3XJVA7B7R7GWFYNW3RCGNVRXQ3E37XDXD4SUXTEMQFZ73HHA, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.135405801 (68h56m48.061700497s) [noise=true , address=104.36.82.6:4873 , piece_id=TIIMEQUSDLRFF3OK3WBB74F7SWJ5MEC4Z6IQNELDMRZLNTMQT7LA, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.135774729 (68h56m48.062069418s) [noise=true , address=77.90.31.13:4626 , piece_id=MXYBIEDFAFK53CVTE2C6CKEZWOL3MLW4S7LOHK4IHW6QEIZJL76Q, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.135944436 (68h56m48.06223913s) [noise=true , address=77.90.31.10:4517 , piece_id=VJZ5T4QNVAPNUHNFNK24S6TAS4V4NKNFMQE2FZMCWWLCCB2ELXXQ, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.136080402 (68h56m48.062375094s) [noise=true , address=77.90.31.15:4697 , piece_id=CUXGVL7PS3H7ZQKUCSYXJCBR3ETYFDPZFYIEAWEYL7C2S3RJY3AA, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.136382603 (68h56m48.062677289s) [noise=true , address=104.36.82.13:4186 , piece_id=5SVABN3KS6AUL6HILHNQCVF74BA75VPKTTAK7IQAPKRRME3MOZEA, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.136399304 (68h56m48.062693995s) [noise=true , address=104.36.82.14:4223 , piece_id=LP74YFI2SRSYPYEIKREVINCOFMVCXLNDS2555YKUVFN63N6TBJDA, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.136669576 (68h56m48.062964262s) [noise=true , address=104.36.82.6:4876 , piece_id=SSMGONONY34EHF5W6QMKO7YXB7RLO5LZVTJMMFKTP6GQNGYFYQ4A, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.136809784 (68h56m48.063104477s) [noise=true , address=104.36.82.15:4258 , piece_id=YXS4J7YSG6JV33JG4BZLK5AXKE3JXLBTGZJXPXZQ4LBYBT3MU2AA, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.136950556 (68h56m48.063245242s) [noise=true , address=77.90.31.11:4554 , piece_id=R2CIFLWS4LXTGSDSIMIRC4UYVJTXC7T35HPWZ2S7VYUTJ2AMY3BQ, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.137226497 (68h56m48.06352119s) [noise=true , address=77.90.31.14:4661 , piece_id=VXI2ZUSGR4N3H7TPIK75RIDFP3SBAOWVRGHNXTZOHFT5FNG4JWJA, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.138099107 (68h56m48.064393792s) [noise=true , address=45.91.116.10:4082 , piece_id=7NVEGA4QQOE7KYPHXUUX42SJPQL6LP46RGM37VD6A2AWN7PAOM3Q, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.139330448 (68h56m48.065625141s) [noise=true , address=104.36.82.14:4222 , piece_id=U3MBMAKFBFTOPJZ3HS3FXLPDBLQTN577XNCZLUBHRO6IU4BE5VNA, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.140580734 (68h56m48.06687543s) [noise=true , address=77.90.31.19:4842 , piece_id=G3AG375DATHB7MGEEWONE5GMMIC7QHG2VVDKF5AEVOSXV5N4S6AA, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.141452300 (68h56m48.067747027s) [noise=true , address=45.91.116.13:4154 , piece_id=STFPXN2LUJYUIAYROWZIRTCMT2Y3XET6KWA4GZWUHENXRJURP62Q, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.142917062 (68h56m48.069211739s) [noise=true , address=104.36.82.7:4912 , piece_id=A5HPC7CEU66IFNS47HO3AQX22LUZ6AZ4L4XAMZ6T5NCVE4D3TAZA, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.221824189 (68h56m48.148118865s) [upload=55354]: Next returned nil segment Jun 26 06:08:21.221837958 (68h56m48.148132644s) [upload=55354]: Waiting for error group managing segments... Jun 26 06:08:21.229550628 (68h56m48.155845315s) [noise=true , address=104.36.82.14:4223 , piece_id=LP74YFI2SRSYPYEIKREVINCOFMVCXLNDS2555YKUVFN63N6TBJDA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: <nil> Jun 26 06:08:21.230245145 (68h56m48.156539832s) [noise=true , address=104.36.82.14:4222 , piece_id=U3MBMAKFBFTOPJZ3HS3FXLPDBLQTN577XNCZLUBHRO6IU4BE5VNA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: <nil> Jun 26 06:08:21.230801067 (68h56m48.15709574s) [noise=true , address=104.36.82.6:4873 , piece_id=TIIMEQUSDLRFF3OK3WBB74F7SWJ5MEC4Z6IQNELDMRZLNTMQT7LA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: <nil> Jun 26 06:08:21.231457382 (68h56m48.157752071s) [noise=true , address=104.36.82.6:4876 , piece_id=SSMGONONY34EHF5W6QMKO7YXB7RLO5LZVTJMMFKTP6GQNGYFYQ4A, seg_pos={0 0}, upload=55354]: Done uploading piece. err: <nil> Jun 26 06:08:21.231563205 (68h56m48.157857893s) [noise=true , address=104.36.82.13:4186 , piece_id=5SVABN3KS6AUL6HILHNQCVF74BA75VPKTTAK7IQAPKRRME3MOZEA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: <nil> Jun 26 06:08:21.239131119 (68h56m48.165425799s) [noise=true , address=104.36.82.7:4912 , piece_id=A5HPC7CEU66IFNS47HO3AQX22LUZ6AZ4L4XAMZ6T5NCVE4D3TAZA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: <nil> Jun 26 06:08:21.534686665 (68h56m48.460981337s) [noise=true , address=104.36.82.15:4258 , piece_id=YXS4J7YSG6JV33JG4BZLK5AXKE3JXLBTGZJXPXZQ4LBYBT3MU2AA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: <nil> Jun 26 06:08:51.073673533 (68h57m17.999968219s) [noise=true , address=45.91.116.9:5261 , piece_id=CZWB3XJVA7B7R7GWFYNW3RCGNVRXQ3E37XDXD4SUXTEMQFZ73HHA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi, address:45.91.116.9:5261): protocol: rpc: dial tcp 45.91.116.9:5261: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:51.073739594 (68h57m18.000034272s) [noise=true , address=45.91.116.10:4082 , piece_id=7NVEGA4QQOE7KYPHXUUX42SJPQL6LP46RGM37VD6A2AWN7PAOM3Q, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj, address:45.91.116.10:4082): protocol: rpc: dial tcp 45.91.116.10:4082: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:51.073789897 (68h57m18.000084584s) [noise=true , address=77.90.31.19:4842 , piece_id=G3AG375DATHB7MGEEWONE5GMMIC7QHG2VVDKF5AEVOSXV5N4S6AA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: rpc: dial tcp 77.90.31.19:4842: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:51.073849570 (68h57m18.000144246s) [noise=true , address=45.91.116.13:4154 , piece_id=STFPXN2LUJYUIAYROWZIRTCMT2Y3XET6KWA4GZWUHENXRJURP62Q, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN, address:45.91.116.13:4154): protocol: write tcp 159.223.143.39:35392->45.91.116.13:4154: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:51.073904716 (68h57m18.000199391s) [noise=true , address=77.90.31.13:4626 , piece_id=MXYBIEDFAFK53CVTE2C6CKEZWOL3MLW4S7LOHK4IHW6QEIZJL76Q, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: write tcp 159.223.143.39:58864->77.90.31.13:4626: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:51.073952402 (68h57m18.000247082s) [noise=true , address=77.90.31.10:4517 , piece_id=VJZ5T4QNVAPNUHNFNK24S6TAS4V4NKNFMQE2FZMCWWLCCB2ELXXQ, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, address:77.90.31.10:4517): protocol: write tcp 159.223.143.39:36382->77.90.31.10:4517: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:51.074022481 (68h57m18.000317156s) [noise=true , address=77.90.31.15:4697 , piece_id=CUXGVL7PS3H7ZQKUCSYXJCBR3ETYFDPZFYIEAWEYL7C2S3RJY3AA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5, address:77.90.31.15:4697): protocol: write tcp 159.223.143.39:52730->77.90.31.15:4697: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:51.074071824 (68h57m18.000366503s) [noise=true , address=77.90.31.11:4554 , piece_id=R2CIFLWS4LXTGSDSIMIRC4UYVJTXC7T35HPWZ2S7VYUTJ2AMY3BQ, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, address:77.90.31.11:4554): protocol: rpc: dial tcp 77.90.31.11:4554: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:51.074102177 (68h57m18.000396855s) [noise=true , address=77.90.31.14:4661 , piece_id=VXI2ZUSGR4N3H7TPIK75RIDFP3SBAOWVRGHNXTZOHFT5FNG4JWJA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue, address:77.90.31.14:4661): protocol: rpc: dial tcp 77.90.31.14:4661: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:51.074119276 (68h57m18.000413966s) [seg_pos={0 0}, upload=55354]: Done waiting for segment. successful: 7 optimal: 13 errs: <nil> Jun 26 06:08:51.100079867 (68h57m18.026374541s) [upload=55354]: Done uploading segments... Jun 26 06:08:51.100114505 (68h57m18.026409191s) [upload=55354]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 7) Duration: 455.083853ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.....XXX....................................................] duration: 413.629032ms FAIL (2025-06-26 06:49:01 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 7) ======LOGS====== pload=55354]: Got next segment. Inline: false Jun 26 06:08:21.105827735 (68h56m48.032122411s) [upload=55354]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 26 06:08:21.105833588 (68h56m48.032128274s) [upload=55354]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 26 06:08:21.135318313 (68h56m48.061613009s) [seg_pos={0 0}, upload=55354]: Begin upload segment... Jun 26 06:08:21.135366534 (68h56m48.061661214s) [seg_pos={0 0}, upload=55354]: Done begin upload segment. Jun 26 06:08:21.135390342 (68h56m48.061685028s) [noise=true , address=45.91.116.9:5261 , piece_id=CZWB3XJVA7B7R7GWFYNW3RCGNVRXQ3E37XDXD4SUXTEMQFZ73HHA, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.135405801 (68h56m48.061700497s) [noise=true , address=104.36.82.6:4873 , piece_id=TIIMEQUSDLRFF3OK3WBB74F7SWJ5MEC4Z6IQNELDMRZLNTMQT7LA, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.135774729 (68h56m48.062069418s) [noise=true , address=77.90.31.13:4626 , piece_id=MXYBIEDFAFK53CVTE2C6CKEZWOL3MLW4S7LOHK4IHW6QEIZJL76Q, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.135944436 (68h56m48.06223913s) [noise=true , address=77.90.31.10:4517 , piece_id=VJZ5T4QNVAPNUHNFNK24S6TAS4V4NKNFMQE2FZMCWWLCCB2ELXXQ, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.136080402 (68h56m48.062375094s) [noise=true , address=77.90.31.15:4697 , piece_id=CUXGVL7PS3H7ZQKUCSYXJCBR3ETYFDPZFYIEAWEYL7C2S3RJY3AA, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.136382603 (68h56m48.062677289s) [noise=true , address=104.36.82.13:4186 , piece_id=5SVABN3KS6AUL6HILHNQCVF74BA75VPKTTAK7IQAPKRRME3MOZEA, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.136399304 (68h56m48.062693995s) [noise=true , address=104.36.82.14:4223 , piece_id=LP74YFI2SRSYPYEIKREVINCOFMVCXLNDS2555YKUVFN63N6TBJDA, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.136669576 (68h56m48.062964262s) [noise=true , address=104.36.82.6:4876 , piece_id=SSMGONONY34EHF5W6QMKO7YXB7RLO5LZVTJMMFKTP6GQNGYFYQ4A, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.136809784 (68h56m48.063104477s) [noise=true , address=104.36.82.15:4258 , piece_id=YXS4J7YSG6JV33JG4BZLK5AXKE3JXLBTGZJXPXZQ4LBYBT3MU2AA, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.136950556 (68h56m48.063245242s) [noise=true , address=77.90.31.11:4554 , piece_id=R2CIFLWS4LXTGSDSIMIRC4UYVJTXC7T35HPWZ2S7VYUTJ2AMY3BQ, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.137226497 (68h56m48.06352119s) [noise=true , address=77.90.31.14:4661 , piece_id=VXI2ZUSGR4N3H7TPIK75RIDFP3SBAOWVRGHNXTZOHFT5FNG4JWJA, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.138099107 (68h56m48.064393792s) [noise=true , address=45.91.116.10:4082 , piece_id=7NVEGA4QQOE7KYPHXUUX42SJPQL6LP46RGM37VD6A2AWN7PAOM3Q, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.139330448 (68h56m48.065625141s) [noise=true , address=104.36.82.14:4222 , piece_id=U3MBMAKFBFTOPJZ3HS3FXLPDBLQTN577XNCZLUBHRO6IU4BE5VNA, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.140580734 (68h56m48.06687543s) [noise=true , address=77.90.31.19:4842 , piece_id=G3AG375DATHB7MGEEWONE5GMMIC7QHG2VVDKF5AEVOSXV5N4S6AA, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.141452300 (68h56m48.067747027s) [noise=true , address=45.91.116.13:4154 , piece_id=STFPXN2LUJYUIAYROWZIRTCMT2Y3XET6KWA4GZWUHENXRJURP62Q, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.142917062 (68h56m48.069211739s) [noise=true , address=104.36.82.7:4912 , piece_id=A5HPC7CEU66IFNS47HO3AQX22LUZ6AZ4L4XAMZ6T5NCVE4D3TAZA, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.221824189 (68h56m48.148118865s) [upload=55354]: Next returned nil segment Jun 26 06:08:21.221837958 (68h56m48.148132644s) [upload=55354]: Waiting for error group managing segments... Jun 26 06:08:21.229550628 (68h56m48.155845315s) [noise=true , address=104.36.82.14:4223 , piece_id=LP74YFI2SRSYPYEIKREVINCOFMVCXLNDS2555YKUVFN63N6TBJDA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: <nil> Jun 26 06:08:21.230245145 (68h56m48.156539832s) [noise=true , address=104.36.82.14:4222 , piece_id=U3MBMAKFBFTOPJZ3HS3FXLPDBLQTN577XNCZLUBHRO6IU4BE5VNA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: <nil> Jun 26 06:08:21.230801067 (68h56m48.15709574s) [noise=true , address=104.36.82.6:4873 , piece_id=TIIMEQUSDLRFF3OK3WBB74F7SWJ5MEC4Z6IQNELDMRZLNTMQT7LA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: <nil> Jun 26 06:08:21.231457382 (68h56m48.157752071s) [noise=true , address=104.36.82.6:4876 , piece_id=SSMGONONY34EHF5W6QMKO7YXB7RLO5LZVTJMMFKTP6GQNGYFYQ4A, seg_pos={0 0}, upload=55354]: Done uploading piece. err: <nil> Jun 26 06:08:21.231563205 (68h56m48.157857893s) [noise=true , address=104.36.82.13:4186 , piece_id=5SVABN3KS6AUL6HILHNQCVF74BA75VPKTTAK7IQAPKRRME3MOZEA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: <nil> Jun 26 06:08:21.239131119 (68h56m48.165425799s) [noise=true , address=104.36.82.7:4912 , piece_id=A5HPC7CEU66IFNS47HO3AQX22LUZ6AZ4L4XAMZ6T5NCVE4D3TAZA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: <nil> Jun 26 06:08:21.534686665 (68h56m48.460981337s) [noise=true , address=104.36.82.15:4258 , piece_id=YXS4J7YSG6JV33JG4BZLK5AXKE3JXLBTGZJXPXZQ4LBYBT3MU2AA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: <nil> Jun 26 06:08:51.073673533 (68h57m17.999968219s) [noise=true , address=45.91.116.9:5261 , piece_id=CZWB3XJVA7B7R7GWFYNW3RCGNVRXQ3E37XDXD4SUXTEMQFZ73HHA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi, address:45.91.116.9:5261): protocol: rpc: dial tcp 45.91.116.9:5261: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:51.073739594 (68h57m18.000034272s) [noise=true , address=45.91.116.10:4082 , piece_id=7NVEGA4QQOE7KYPHXUUX42SJPQL6LP46RGM37VD6A2AWN7PAOM3Q, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj, address:45.91.116.10:4082): protocol: rpc: dial tcp 45.91.116.10:4082: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:51.073789897 (68h57m18.000084584s) [noise=true , address=77.90.31.19:4842 , piece_id=G3AG375DATHB7MGEEWONE5GMMIC7QHG2VVDKF5AEVOSXV5N4S6AA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: rpc: dial tcp 77.90.31.19:4842: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:51.073849570 (68h57m18.000144246s) [noise=true , address=45.91.116.13:4154 , piece_id=STFPXN2LUJYUIAYROWZIRTCMT2Y3XET6KWA4GZWUHENXRJURP62Q, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN, address:45.91.116.13:4154): protocol: write tcp 159.223.143.39:35392->45.91.116.13:4154: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:51.073904716 (68h57m18.000199391s) [noise=true , address=77.90.31.13:4626 , piece_id=MXYBIEDFAFK53CVTE2C6CKEZWOL3MLW4S7LOHK4IHW6QEIZJL76Q, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: write tcp 159.223.143.39:58864->77.90.31.13:4626: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:51.073952402 (68h57m18.000247082s) [noise=true , address=77.90.31.10:4517 , piece_id=VJZ5T4QNVAPNUHNFNK24S6TAS4V4NKNFMQE2FZMCWWLCCB2ELXXQ, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, address:77.90.31.10:4517): protocol: write tcp 159.223.143.39:36382->77.90.31.10:4517: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:51.074022481 (68h57m18.000317156s) [noise=true , address=77.90.31.15:4697 , piece_id=CUXGVL7PS3H7ZQKUCSYXJCBR3ETYFDPZFYIEAWEYL7C2S3RJY3AA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5, address:77.90.31.15:4697): protocol: write tcp 159.223.143.39:52730->77.90.31.15:4697: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:51.074071824 (68h57m18.000366503s) [noise=true , address=77.90.31.11:4554 , piece_id=R2CIFLWS4LXTGSDSIMIRC4UYVJTXC7T35HPWZ2S7VYUTJ2AMY3BQ, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, address:77.90.31.11:4554): protocol: rpc: dial tcp 77.90.31.11:4554: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:51.074102177 (68h57m18.000396855s) [noise=true , address=77.90.31.14:4661 , piece_id=VXI2ZUSGR4N3H7TPIK75RIDFP3SBAOWVRGHNXTZOHFT5FNG4JWJA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue, address:77.90.31.14:4661): protocol: rpc: dial tcp 77.90.31.14:4661: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:51.074119276 (68h57m18.000413966s) [seg_pos={0 0}, upload=55354]: Done waiting for segment. successful: 7 optimal: 13 errs: <nil> Jun 26 06:08:51.100079867 (68h57m18.026374541s) [upload=55354]: Done uploading segments... Jun 26 06:08:51.100114505 (68h57m18.026409191s) [upload=55354]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 7) Duration: 413.674916ms core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [....XX......................................................] duration: 510.420211ms FAIL (2025-06-26 06:48:45 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 510.474542ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [......XXX...................................................] duration: 503.883226ms FAIL (2025-06-26 06:48:26 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 7) ======LOGS====== pload=55354]: Got next segment. Inline: false Jun 26 06:08:21.105827735 (68h56m48.032122411s) [upload=55354]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 26 06:08:21.105833588 (68h56m48.032128274s) [upload=55354]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 26 06:08:21.135318313 (68h56m48.061613009s) [seg_pos={0 0}, upload=55354]: Begin upload segment... Jun 26 06:08:21.135366534 (68h56m48.061661214s) [seg_pos={0 0}, upload=55354]: Done begin upload segment. Jun 26 06:08:21.135390342 (68h56m48.061685028s) [noise=true , address=45.91.116.9:5261 , piece_id=CZWB3XJVA7B7R7GWFYNW3RCGNVRXQ3E37XDXD4SUXTEMQFZ73HHA, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.135405801 (68h56m48.061700497s) [noise=true , address=104.36.82.6:4873 , piece_id=TIIMEQUSDLRFF3OK3WBB74F7SWJ5MEC4Z6IQNELDMRZLNTMQT7LA, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.135774729 (68h56m48.062069418s) [noise=true , address=77.90.31.13:4626 , piece_id=MXYBIEDFAFK53CVTE2C6CKEZWOL3MLW4S7LOHK4IHW6QEIZJL76Q, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.135944436 (68h56m48.06223913s) [noise=true , address=77.90.31.10:4517 , piece_id=VJZ5T4QNVAPNUHNFNK24S6TAS4V4NKNFMQE2FZMCWWLCCB2ELXXQ, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.136080402 (68h56m48.062375094s) [noise=true , address=77.90.31.15:4697 , piece_id=CUXGVL7PS3H7ZQKUCSYXJCBR3ETYFDPZFYIEAWEYL7C2S3RJY3AA, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.136382603 (68h56m48.062677289s) [noise=true , address=104.36.82.13:4186 , piece_id=5SVABN3KS6AUL6HILHNQCVF74BA75VPKTTAK7IQAPKRRME3MOZEA, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.136399304 (68h56m48.062693995s) [noise=true , address=104.36.82.14:4223 , piece_id=LP74YFI2SRSYPYEIKREVINCOFMVCXLNDS2555YKUVFN63N6TBJDA, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.136669576 (68h56m48.062964262s) [noise=true , address=104.36.82.6:4876 , piece_id=SSMGONONY34EHF5W6QMKO7YXB7RLO5LZVTJMMFKTP6GQNGYFYQ4A, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.136809784 (68h56m48.063104477s) [noise=true , address=104.36.82.15:4258 , piece_id=YXS4J7YSG6JV33JG4BZLK5AXKE3JXLBTGZJXPXZQ4LBYBT3MU2AA, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.136950556 (68h56m48.063245242s) [noise=true , address=77.90.31.11:4554 , piece_id=R2CIFLWS4LXTGSDSIMIRC4UYVJTXC7T35HPWZ2S7VYUTJ2AMY3BQ, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.137226497 (68h56m48.06352119s) [noise=true , address=77.90.31.14:4661 , piece_id=VXI2ZUSGR4N3H7TPIK75RIDFP3SBAOWVRGHNXTZOHFT5FNG4JWJA, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.138099107 (68h56m48.064393792s) [noise=true , address=45.91.116.10:4082 , piece_id=7NVEGA4QQOE7KYPHXUUX42SJPQL6LP46RGM37VD6A2AWN7PAOM3Q, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.139330448 (68h56m48.065625141s) [noise=true , address=104.36.82.14:4222 , piece_id=U3MBMAKFBFTOPJZ3HS3FXLPDBLQTN577XNCZLUBHRO6IU4BE5VNA, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.140580734 (68h56m48.06687543s) [noise=true , address=77.90.31.19:4842 , piece_id=G3AG375DATHB7MGEEWONE5GMMIC7QHG2VVDKF5AEVOSXV5N4S6AA, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.141452300 (68h56m48.067747027s) [noise=true , address=45.91.116.13:4154 , piece_id=STFPXN2LUJYUIAYROWZIRTCMT2Y3XET6KWA4GZWUHENXRJURP62Q, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.142917062 (68h56m48.069211739s) [noise=true , address=104.36.82.7:4912 , piece_id=A5HPC7CEU66IFNS47HO3AQX22LUZ6AZ4L4XAMZ6T5NCVE4D3TAZA, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.221824189 (68h56m48.148118865s) [upload=55354]: Next returned nil segment Jun 26 06:08:21.221837958 (68h56m48.148132644s) [upload=55354]: Waiting for error group managing segments... Jun 26 06:08:21.229550628 (68h56m48.155845315s) [noise=true , address=104.36.82.14:4223 , piece_id=LP74YFI2SRSYPYEIKREVINCOFMVCXLNDS2555YKUVFN63N6TBJDA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: <nil> Jun 26 06:08:21.230245145 (68h56m48.156539832s) [noise=true , address=104.36.82.14:4222 , piece_id=U3MBMAKFBFTOPJZ3HS3FXLPDBLQTN577XNCZLUBHRO6IU4BE5VNA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: <nil> Jun 26 06:08:21.230801067 (68h56m48.15709574s) [noise=true , address=104.36.82.6:4873 , piece_id=TIIMEQUSDLRFF3OK3WBB74F7SWJ5MEC4Z6IQNELDMRZLNTMQT7LA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: <nil> Jun 26 06:08:21.231457382 (68h56m48.157752071s) [noise=true , address=104.36.82.6:4876 , piece_id=SSMGONONY34EHF5W6QMKO7YXB7RLO5LZVTJMMFKTP6GQNGYFYQ4A, seg_pos={0 0}, upload=55354]: Done uploading piece. err: <nil> Jun 26 06:08:21.231563205 (68h56m48.157857893s) [noise=true , address=104.36.82.13:4186 , piece_id=5SVABN3KS6AUL6HILHNQCVF74BA75VPKTTAK7IQAPKRRME3MOZEA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: <nil> Jun 26 06:08:21.239131119 (68h56m48.165425799s) [noise=true , address=104.36.82.7:4912 , piece_id=A5HPC7CEU66IFNS47HO3AQX22LUZ6AZ4L4XAMZ6T5NCVE4D3TAZA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: <nil> Jun 26 06:08:21.534686665 (68h56m48.460981337s) [noise=true , address=104.36.82.15:4258 , piece_id=YXS4J7YSG6JV33JG4BZLK5AXKE3JXLBTGZJXPXZQ4LBYBT3MU2AA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: <nil> Jun 26 06:08:51.073673533 (68h57m17.999968219s) [noise=true , address=45.91.116.9:5261 , piece_id=CZWB3XJVA7B7R7GWFYNW3RCGNVRXQ3E37XDXD4SUXTEMQFZ73HHA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi, address:45.91.116.9:5261): protocol: rpc: dial tcp 45.91.116.9:5261: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:51.073739594 (68h57m18.000034272s) [noise=true , address=45.91.116.10:4082 , piece_id=7NVEGA4QQOE7KYPHXUUX42SJPQL6LP46RGM37VD6A2AWN7PAOM3Q, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj, address:45.91.116.10:4082): protocol: rpc: dial tcp 45.91.116.10:4082: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:51.073789897 (68h57m18.000084584s) [noise=true , address=77.90.31.19:4842 , piece_id=G3AG375DATHB7MGEEWONE5GMMIC7QHG2VVDKF5AEVOSXV5N4S6AA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: rpc: dial tcp 77.90.31.19:4842: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:51.073849570 (68h57m18.000144246s) [noise=true , address=45.91.116.13:4154 , piece_id=STFPXN2LUJYUIAYROWZIRTCMT2Y3XET6KWA4GZWUHENXRJURP62Q, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN, address:45.91.116.13:4154): protocol: write tcp 159.223.143.39:35392->45.91.116.13:4154: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:51.073904716 (68h57m18.000199391s) [noise=true , address=77.90.31.13:4626 , piece_id=MXYBIEDFAFK53CVTE2C6CKEZWOL3MLW4S7LOHK4IHW6QEIZJL76Q, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: write tcp 159.223.143.39:58864->77.90.31.13:4626: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:51.073952402 (68h57m18.000247082s) [noise=true , address=77.90.31.10:4517 , piece_id=VJZ5T4QNVAPNUHNFNK24S6TAS4V4NKNFMQE2FZMCWWLCCB2ELXXQ, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, address:77.90.31.10:4517): protocol: write tcp 159.223.143.39:36382->77.90.31.10:4517: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:51.074022481 (68h57m18.000317156s) [noise=true , address=77.90.31.15:4697 , piece_id=CUXGVL7PS3H7ZQKUCSYXJCBR3ETYFDPZFYIEAWEYL7C2S3RJY3AA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5, address:77.90.31.15:4697): protocol: write tcp 159.223.143.39:52730->77.90.31.15:4697: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:51.074071824 (68h57m18.000366503s) [noise=true , address=77.90.31.11:4554 , piece_id=R2CIFLWS4LXTGSDSIMIRC4UYVJTXC7T35HPWZ2S7VYUTJ2AMY3BQ, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, address:77.90.31.11:4554): protocol: rpc: dial tcp 77.90.31.11:4554: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:51.074102177 (68h57m18.000396855s) [noise=true , address=77.90.31.14:4661 , piece_id=VXI2ZUSGR4N3H7TPIK75RIDFP3SBAOWVRGHNXTZOHFT5FNG4JWJA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue, address:77.90.31.14:4661): protocol: rpc: dial tcp 77.90.31.14:4661: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:51.074119276 (68h57m18.000413966s) [seg_pos={0 0}, upload=55354]: Done waiting for segment. successful: 7 optimal: 13 errs: <nil> Jun 26 06:08:51.100079867 (68h57m18.026374541s) [upload=55354]: Done uploading segments... Jun 26 06:08:51.100114505 (68h57m18.026409191s) [upload=55354]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 7) Duration: 503.948355ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.......XXX..................................................] duration: 453.38883ms FAIL (2025-06-26 06:48:04 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 7) ======LOGS====== pload=55354]: Got next segment. Inline: false Jun 26 06:08:21.105827735 (68h56m48.032122411s) [upload=55354]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 26 06:08:21.105833588 (68h56m48.032128274s) [upload=55354]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 26 06:08:21.135318313 (68h56m48.061613009s) [seg_pos={0 0}, upload=55354]: Begin upload segment... Jun 26 06:08:21.135366534 (68h56m48.061661214s) [seg_pos={0 0}, upload=55354]: Done begin upload segment. Jun 26 06:08:21.135390342 (68h56m48.061685028s) [noise=true , address=45.91.116.9:5261 , piece_id=CZWB3XJVA7B7R7GWFYNW3RCGNVRXQ3E37XDXD4SUXTEMQFZ73HHA, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.135405801 (68h56m48.061700497s) [noise=true , address=104.36.82.6:4873 , piece_id=TIIMEQUSDLRFF3OK3WBB74F7SWJ5MEC4Z6IQNELDMRZLNTMQT7LA, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.135774729 (68h56m48.062069418s) [noise=true , address=77.90.31.13:4626 , piece_id=MXYBIEDFAFK53CVTE2C6CKEZWOL3MLW4S7LOHK4IHW6QEIZJL76Q, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.135944436 (68h56m48.06223913s) [noise=true , address=77.90.31.10:4517 , piece_id=VJZ5T4QNVAPNUHNFNK24S6TAS4V4NKNFMQE2FZMCWWLCCB2ELXXQ, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.136080402 (68h56m48.062375094s) [noise=true , address=77.90.31.15:4697 , piece_id=CUXGVL7PS3H7ZQKUCSYXJCBR3ETYFDPZFYIEAWEYL7C2S3RJY3AA, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.136382603 (68h56m48.062677289s) [noise=true , address=104.36.82.13:4186 , piece_id=5SVABN3KS6AUL6HILHNQCVF74BA75VPKTTAK7IQAPKRRME3MOZEA, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.136399304 (68h56m48.062693995s) [noise=true , address=104.36.82.14:4223 , piece_id=LP74YFI2SRSYPYEIKREVINCOFMVCXLNDS2555YKUVFN63N6TBJDA, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.136669576 (68h56m48.062964262s) [noise=true , address=104.36.82.6:4876 , piece_id=SSMGONONY34EHF5W6QMKO7YXB7RLO5LZVTJMMFKTP6GQNGYFYQ4A, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.136809784 (68h56m48.063104477s) [noise=true , address=104.36.82.15:4258 , piece_id=YXS4J7YSG6JV33JG4BZLK5AXKE3JXLBTGZJXPXZQ4LBYBT3MU2AA, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.136950556 (68h56m48.063245242s) [noise=true , address=77.90.31.11:4554 , piece_id=R2CIFLWS4LXTGSDSIMIRC4UYVJTXC7T35HPWZ2S7VYUTJ2AMY3BQ, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.137226497 (68h56m48.06352119s) [noise=true , address=77.90.31.14:4661 , piece_id=VXI2ZUSGR4N3H7TPIK75RIDFP3SBAOWVRGHNXTZOHFT5FNG4JWJA, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.138099107 (68h56m48.064393792s) [noise=true , address=45.91.116.10:4082 , piece_id=7NVEGA4QQOE7KYPHXUUX42SJPQL6LP46RGM37VD6A2AWN7PAOM3Q, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.139330448 (68h56m48.065625141s) [noise=true , address=104.36.82.14:4222 , piece_id=U3MBMAKFBFTOPJZ3HS3FXLPDBLQTN577XNCZLUBHRO6IU4BE5VNA, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.140580734 (68h56m48.06687543s) [noise=true , address=77.90.31.19:4842 , piece_id=G3AG375DATHB7MGEEWONE5GMMIC7QHG2VVDKF5AEVOSXV5N4S6AA, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.141452300 (68h56m48.067747027s) [noise=true , address=45.91.116.13:4154 , piece_id=STFPXN2LUJYUIAYROWZIRTCMT2Y3XET6KWA4GZWUHENXRJURP62Q, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.142917062 (68h56m48.069211739s) [noise=true , address=104.36.82.7:4912 , piece_id=A5HPC7CEU66IFNS47HO3AQX22LUZ6AZ4L4XAMZ6T5NCVE4D3TAZA, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.221824189 (68h56m48.148118865s) [upload=55354]: Next returned nil segment Jun 26 06:08:21.221837958 (68h56m48.148132644s) [upload=55354]: Waiting for error group managing segments... Jun 26 06:08:21.229550628 (68h56m48.155845315s) [noise=true , address=104.36.82.14:4223 , piece_id=LP74YFI2SRSYPYEIKREVINCOFMVCXLNDS2555YKUVFN63N6TBJDA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: <nil> Jun 26 06:08:21.230245145 (68h56m48.156539832s) [noise=true , address=104.36.82.14:4222 , piece_id=U3MBMAKFBFTOPJZ3HS3FXLPDBLQTN577XNCZLUBHRO6IU4BE5VNA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: <nil> Jun 26 06:08:21.230801067 (68h56m48.15709574s) [noise=true , address=104.36.82.6:4873 , piece_id=TIIMEQUSDLRFF3OK3WBB74F7SWJ5MEC4Z6IQNELDMRZLNTMQT7LA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: <nil> Jun 26 06:08:21.231457382 (68h56m48.157752071s) [noise=true , address=104.36.82.6:4876 , piece_id=SSMGONONY34EHF5W6QMKO7YXB7RLO5LZVTJMMFKTP6GQNGYFYQ4A, seg_pos={0 0}, upload=55354]: Done uploading piece. err: <nil> Jun 26 06:08:21.231563205 (68h56m48.157857893s) [noise=true , address=104.36.82.13:4186 , piece_id=5SVABN3KS6AUL6HILHNQCVF74BA75VPKTTAK7IQAPKRRME3MOZEA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: <nil> Jun 26 06:08:21.239131119 (68h56m48.165425799s) [noise=true , address=104.36.82.7:4912 , piece_id=A5HPC7CEU66IFNS47HO3AQX22LUZ6AZ4L4XAMZ6T5NCVE4D3TAZA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: <nil> Jun 26 06:08:21.534686665 (68h56m48.460981337s) [noise=true , address=104.36.82.15:4258 , piece_id=YXS4J7YSG6JV33JG4BZLK5AXKE3JXLBTGZJXPXZQ4LBYBT3MU2AA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: <nil> Jun 26 06:08:51.073673533 (68h57m17.999968219s) [noise=true , address=45.91.116.9:5261 , piece_id=CZWB3XJVA7B7R7GWFYNW3RCGNVRXQ3E37XDXD4SUXTEMQFZ73HHA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi, address:45.91.116.9:5261): protocol: rpc: dial tcp 45.91.116.9:5261: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:51.073739594 (68h57m18.000034272s) [noise=true , address=45.91.116.10:4082 , piece_id=7NVEGA4QQOE7KYPHXUUX42SJPQL6LP46RGM37VD6A2AWN7PAOM3Q, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj, address:45.91.116.10:4082): protocol: rpc: dial tcp 45.91.116.10:4082: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:51.073789897 (68h57m18.000084584s) [noise=true , address=77.90.31.19:4842 , piece_id=G3AG375DATHB7MGEEWONE5GMMIC7QHG2VVDKF5AEVOSXV5N4S6AA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: rpc: dial tcp 77.90.31.19:4842: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:51.073849570 (68h57m18.000144246s) [noise=true , address=45.91.116.13:4154 , piece_id=STFPXN2LUJYUIAYROWZIRTCMT2Y3XET6KWA4GZWUHENXRJURP62Q, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN, address:45.91.116.13:4154): protocol: write tcp 159.223.143.39:35392->45.91.116.13:4154: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:51.073904716 (68h57m18.000199391s) [noise=true , address=77.90.31.13:4626 , piece_id=MXYBIEDFAFK53CVTE2C6CKEZWOL3MLW4S7LOHK4IHW6QEIZJL76Q, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: write tcp 159.223.143.39:58864->77.90.31.13:4626: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:51.073952402 (68h57m18.000247082s) [noise=true , address=77.90.31.10:4517 , piece_id=VJZ5T4QNVAPNUHNFNK24S6TAS4V4NKNFMQE2FZMCWWLCCB2ELXXQ, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, address:77.90.31.10:4517): protocol: write tcp 159.223.143.39:36382->77.90.31.10:4517: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:51.074022481 (68h57m18.000317156s) [noise=true , address=77.90.31.15:4697 , piece_id=CUXGVL7PS3H7ZQKUCSYXJCBR3ETYFDPZFYIEAWEYL7C2S3RJY3AA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5, address:77.90.31.15:4697): protocol: write tcp 159.223.143.39:52730->77.90.31.15:4697: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:51.074071824 (68h57m18.000366503s) [noise=true , address=77.90.31.11:4554 , piece_id=R2CIFLWS4LXTGSDSIMIRC4UYVJTXC7T35HPWZ2S7VYUTJ2AMY3BQ, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, address:77.90.31.11:4554): protocol: rpc: dial tcp 77.90.31.11:4554: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:51.074102177 (68h57m18.000396855s) [noise=true , address=77.90.31.14:4661 , piece_id=VXI2ZUSGR4N3H7TPIK75RIDFP3SBAOWVRGHNXTZOHFT5FNG4JWJA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue, address:77.90.31.14:4661): protocol: rpc: dial tcp 77.90.31.14:4661: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:51.074119276 (68h57m18.000413966s) [seg_pos={0 0}, upload=55354]: Done waiting for segment. successful: 7 optimal: 13 errs: <nil> Jun 26 06:08:51.100079867 (68h57m18.026374541s) [upload=55354]: Done uploading segments... Jun 26 06:08:51.100114505 (68h57m18.026409191s) [upload=55354]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 7) Duration: 453.44586ms core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.....XX.....................................................] duration: 512.681996ms FAIL (2025-06-26 06:48:03 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 512.711067ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [........XXX.................................................] duration: 430.698605ms FAIL (2025-06-26 06:47:46 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 7) ======LOGS====== pload=55354]: Got next segment. Inline: false Jun 26 06:08:21.105827735 (68h56m48.032122411s) [upload=55354]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 26 06:08:21.105833588 (68h56m48.032128274s) [upload=55354]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 26 06:08:21.135318313 (68h56m48.061613009s) [seg_pos={0 0}, upload=55354]: Begin upload segment... Jun 26 06:08:21.135366534 (68h56m48.061661214s) [seg_pos={0 0}, upload=55354]: Done begin upload segment. Jun 26 06:08:21.135390342 (68h56m48.061685028s) [noise=true , address=45.91.116.9:5261 , piece_id=CZWB3XJVA7B7R7GWFYNW3RCGNVRXQ3E37XDXD4SUXTEMQFZ73HHA, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.135405801 (68h56m48.061700497s) [noise=true , address=104.36.82.6:4873 , piece_id=TIIMEQUSDLRFF3OK3WBB74F7SWJ5MEC4Z6IQNELDMRZLNTMQT7LA, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.135774729 (68h56m48.062069418s) [noise=true , address=77.90.31.13:4626 , piece_id=MXYBIEDFAFK53CVTE2C6CKEZWOL3MLW4S7LOHK4IHW6QEIZJL76Q, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.135944436 (68h56m48.06223913s) [noise=true , address=77.90.31.10:4517 , piece_id=VJZ5T4QNVAPNUHNFNK24S6TAS4V4NKNFMQE2FZMCWWLCCB2ELXXQ, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.136080402 (68h56m48.062375094s) [noise=true , address=77.90.31.15:4697 , piece_id=CUXGVL7PS3H7ZQKUCSYXJCBR3ETYFDPZFYIEAWEYL7C2S3RJY3AA, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.136382603 (68h56m48.062677289s) [noise=true , address=104.36.82.13:4186 , piece_id=5SVABN3KS6AUL6HILHNQCVF74BA75VPKTTAK7IQAPKRRME3MOZEA, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.136399304 (68h56m48.062693995s) [noise=true , address=104.36.82.14:4223 , piece_id=LP74YFI2SRSYPYEIKREVINCOFMVCXLNDS2555YKUVFN63N6TBJDA, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.136669576 (68h56m48.062964262s) [noise=true , address=104.36.82.6:4876 , piece_id=SSMGONONY34EHF5W6QMKO7YXB7RLO5LZVTJMMFKTP6GQNGYFYQ4A, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.136809784 (68h56m48.063104477s) [noise=true , address=104.36.82.15:4258 , piece_id=YXS4J7YSG6JV33JG4BZLK5AXKE3JXLBTGZJXPXZQ4LBYBT3MU2AA, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.136950556 (68h56m48.063245242s) [noise=true , address=77.90.31.11:4554 , piece_id=R2CIFLWS4LXTGSDSIMIRC4UYVJTXC7T35HPWZ2S7VYUTJ2AMY3BQ, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.137226497 (68h56m48.06352119s) [noise=true , address=77.90.31.14:4661 , piece_id=VXI2ZUSGR4N3H7TPIK75RIDFP3SBAOWVRGHNXTZOHFT5FNG4JWJA, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.138099107 (68h56m48.064393792s) [noise=true , address=45.91.116.10:4082 , piece_id=7NVEGA4QQOE7KYPHXUUX42SJPQL6LP46RGM37VD6A2AWN7PAOM3Q, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.139330448 (68h56m48.065625141s) [noise=true , address=104.36.82.14:4222 , piece_id=U3MBMAKFBFTOPJZ3HS3FXLPDBLQTN577XNCZLUBHRO6IU4BE5VNA, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.140580734 (68h56m48.06687543s) [noise=true , address=77.90.31.19:4842 , piece_id=G3AG375DATHB7MGEEWONE5GMMIC7QHG2VVDKF5AEVOSXV5N4S6AA, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.141452300 (68h56m48.067747027s) [noise=true , address=45.91.116.13:4154 , piece_id=STFPXN2LUJYUIAYROWZIRTCMT2Y3XET6KWA4GZWUHENXRJURP62Q, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.142917062 (68h56m48.069211739s) [noise=true , address=104.36.82.7:4912 , piece_id=A5HPC7CEU66IFNS47HO3AQX22LUZ6AZ4L4XAMZ6T5NCVE4D3TAZA, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.221824189 (68h56m48.148118865s) [upload=55354]: Next returned nil segment Jun 26 06:08:21.221837958 (68h56m48.148132644s) [upload=55354]: Waiting for error group managing segments... Jun 26 06:08:21.229550628 (68h56m48.155845315s) [noise=true , address=104.36.82.14:4223 , piece_id=LP74YFI2SRSYPYEIKREVINCOFMVCXLNDS2555YKUVFN63N6TBJDA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: <nil> Jun 26 06:08:21.230245145 (68h56m48.156539832s) [noise=true , address=104.36.82.14:4222 , piece_id=U3MBMAKFBFTOPJZ3HS3FXLPDBLQTN577XNCZLUBHRO6IU4BE5VNA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: <nil> Jun 26 06:08:21.230801067 (68h56m48.15709574s) [noise=true , address=104.36.82.6:4873 , piece_id=TIIMEQUSDLRFF3OK3WBB74F7SWJ5MEC4Z6IQNELDMRZLNTMQT7LA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: <nil> Jun 26 06:08:21.231457382 (68h56m48.157752071s) [noise=true , address=104.36.82.6:4876 , piece_id=SSMGONONY34EHF5W6QMKO7YXB7RLO5LZVTJMMFKTP6GQNGYFYQ4A, seg_pos={0 0}, upload=55354]: Done uploading piece. err: <nil> Jun 26 06:08:21.231563205 (68h56m48.157857893s) [noise=true , address=104.36.82.13:4186 , piece_id=5SVABN3KS6AUL6HILHNQCVF74BA75VPKTTAK7IQAPKRRME3MOZEA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: <nil> Jun 26 06:08:21.239131119 (68h56m48.165425799s) [noise=true , address=104.36.82.7:4912 , piece_id=A5HPC7CEU66IFNS47HO3AQX22LUZ6AZ4L4XAMZ6T5NCVE4D3TAZA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: <nil> Jun 26 06:08:21.534686665 (68h56m48.460981337s) [noise=true , address=104.36.82.15:4258 , piece_id=YXS4J7YSG6JV33JG4BZLK5AXKE3JXLBTGZJXPXZQ4LBYBT3MU2AA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: <nil> Jun 26 06:08:51.073673533 (68h57m17.999968219s) [noise=true , address=45.91.116.9:5261 , piece_id=CZWB3XJVA7B7R7GWFYNW3RCGNVRXQ3E37XDXD4SUXTEMQFZ73HHA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi, address:45.91.116.9:5261): protocol: rpc: dial tcp 45.91.116.9:5261: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:51.073739594 (68h57m18.000034272s) [noise=true , address=45.91.116.10:4082 , piece_id=7NVEGA4QQOE7KYPHXUUX42SJPQL6LP46RGM37VD6A2AWN7PAOM3Q, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj, address:45.91.116.10:4082): protocol: rpc: dial tcp 45.91.116.10:4082: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:51.073789897 (68h57m18.000084584s) [noise=true , address=77.90.31.19:4842 , piece_id=G3AG375DATHB7MGEEWONE5GMMIC7QHG2VVDKF5AEVOSXV5N4S6AA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: rpc: dial tcp 77.90.31.19:4842: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:51.073849570 (68h57m18.000144246s) [noise=true , address=45.91.116.13:4154 , piece_id=STFPXN2LUJYUIAYROWZIRTCMT2Y3XET6KWA4GZWUHENXRJURP62Q, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN, address:45.91.116.13:4154): protocol: write tcp 159.223.143.39:35392->45.91.116.13:4154: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:51.073904716 (68h57m18.000199391s) [noise=true , address=77.90.31.13:4626 , piece_id=MXYBIEDFAFK53CVTE2C6CKEZWOL3MLW4S7LOHK4IHW6QEIZJL76Q, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: write tcp 159.223.143.39:58864->77.90.31.13:4626: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:51.073952402 (68h57m18.000247082s) [noise=true , address=77.90.31.10:4517 , piece_id=VJZ5T4QNVAPNUHNFNK24S6TAS4V4NKNFMQE2FZMCWWLCCB2ELXXQ, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, address:77.90.31.10:4517): protocol: write tcp 159.223.143.39:36382->77.90.31.10:4517: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:51.074022481 (68h57m18.000317156s) [noise=true , address=77.90.31.15:4697 , piece_id=CUXGVL7PS3H7ZQKUCSYXJCBR3ETYFDPZFYIEAWEYL7C2S3RJY3AA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5, address:77.90.31.15:4697): protocol: write tcp 159.223.143.39:52730->77.90.31.15:4697: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:51.074071824 (68h57m18.000366503s) [noise=true , address=77.90.31.11:4554 , piece_id=R2CIFLWS4LXTGSDSIMIRC4UYVJTXC7T35HPWZ2S7VYUTJ2AMY3BQ, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, address:77.90.31.11:4554): protocol: rpc: dial tcp 77.90.31.11:4554: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:51.074102177 (68h57m18.000396855s) [noise=true , address=77.90.31.14:4661 , piece_id=VXI2ZUSGR4N3H7TPIK75RIDFP3SBAOWVRGHNXTZOHFT5FNG4JWJA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue, address:77.90.31.14:4661): protocol: rpc: dial tcp 77.90.31.14:4661: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:51.074119276 (68h57m18.000413966s) [seg_pos={0 0}, upload=55354]: Done waiting for segment. successful: 7 optimal: 13 errs: <nil> Jun 26 06:08:51.100079867 (68h57m18.026374541s) [upload=55354]: Done uploading segments... Jun 26 06:08:51.100114505 (68h57m18.026409191s) [upload=55354]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 7) Duration: 430.73049ms core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [......XX....................................................] duration: 479.343454ms FAIL (2025-06-26 06:46:58 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 479.385579ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.........XXX................................................] duration: 566.105663ms FAIL (2025-06-26 06:46:32 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 7) ======LOGS====== pload=55354]: Got next segment. Inline: false Jun 26 06:08:21.105827735 (68h56m48.032122411s) [upload=55354]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 26 06:08:21.105833588 (68h56m48.032128274s) [upload=55354]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 26 06:08:21.135318313 (68h56m48.061613009s) [seg_pos={0 0}, upload=55354]: Begin upload segment... Jun 26 06:08:21.135366534 (68h56m48.061661214s) [seg_pos={0 0}, upload=55354]: Done begin upload segment. Jun 26 06:08:21.135390342 (68h56m48.061685028s) [noise=true , address=45.91.116.9:5261 , piece_id=CZWB3XJVA7B7R7GWFYNW3RCGNVRXQ3E37XDXD4SUXTEMQFZ73HHA, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.135405801 (68h56m48.061700497s) [noise=true , address=104.36.82.6:4873 , piece_id=TIIMEQUSDLRFF3OK3WBB74F7SWJ5MEC4Z6IQNELDMRZLNTMQT7LA, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.135774729 (68h56m48.062069418s) [noise=true , address=77.90.31.13:4626 , piece_id=MXYBIEDFAFK53CVTE2C6CKEZWOL3MLW4S7LOHK4IHW6QEIZJL76Q, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.135944436 (68h56m48.06223913s) [noise=true , address=77.90.31.10:4517 , piece_id=VJZ5T4QNVAPNUHNFNK24S6TAS4V4NKNFMQE2FZMCWWLCCB2ELXXQ, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.136080402 (68h56m48.062375094s) [noise=true , address=77.90.31.15:4697 , piece_id=CUXGVL7PS3H7ZQKUCSYXJCBR3ETYFDPZFYIEAWEYL7C2S3RJY3AA, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.136382603 (68h56m48.062677289s) [noise=true , address=104.36.82.13:4186 , piece_id=5SVABN3KS6AUL6HILHNQCVF74BA75VPKTTAK7IQAPKRRME3MOZEA, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.136399304 (68h56m48.062693995s) [noise=true , address=104.36.82.14:4223 , piece_id=LP74YFI2SRSYPYEIKREVINCOFMVCXLNDS2555YKUVFN63N6TBJDA, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.136669576 (68h56m48.062964262s) [noise=true , address=104.36.82.6:4876 , piece_id=SSMGONONY34EHF5W6QMKO7YXB7RLO5LZVTJMMFKTP6GQNGYFYQ4A, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.136809784 (68h56m48.063104477s) [noise=true , address=104.36.82.15:4258 , piece_id=YXS4J7YSG6JV33JG4BZLK5AXKE3JXLBTGZJXPXZQ4LBYBT3MU2AA, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.136950556 (68h56m48.063245242s) [noise=true , address=77.90.31.11:4554 , piece_id=R2CIFLWS4LXTGSDSIMIRC4UYVJTXC7T35HPWZ2S7VYUTJ2AMY3BQ, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.137226497 (68h56m48.06352119s) [noise=true , address=77.90.31.14:4661 , piece_id=VXI2ZUSGR4N3H7TPIK75RIDFP3SBAOWVRGHNXTZOHFT5FNG4JWJA, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.138099107 (68h56m48.064393792s) [noise=true , address=45.91.116.10:4082 , piece_id=7NVEGA4QQOE7KYPHXUUX42SJPQL6LP46RGM37VD6A2AWN7PAOM3Q, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.139330448 (68h56m48.065625141s) [noise=true , address=104.36.82.14:4222 , piece_id=U3MBMAKFBFTOPJZ3HS3FXLPDBLQTN577XNCZLUBHRO6IU4BE5VNA, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.140580734 (68h56m48.06687543s) [noise=true , address=77.90.31.19:4842 , piece_id=G3AG375DATHB7MGEEWONE5GMMIC7QHG2VVDKF5AEVOSXV5N4S6AA, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.141452300 (68h56m48.067747027s) [noise=true , address=45.91.116.13:4154 , piece_id=STFPXN2LUJYUIAYROWZIRTCMT2Y3XET6KWA4GZWUHENXRJURP62Q, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.142917062 (68h56m48.069211739s) [noise=true , address=104.36.82.7:4912 , piece_id=A5HPC7CEU66IFNS47HO3AQX22LUZ6AZ4L4XAMZ6T5NCVE4D3TAZA, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.221824189 (68h56m48.148118865s) [upload=55354]: Next returned nil segment Jun 26 06:08:21.221837958 (68h56m48.148132644s) [upload=55354]: Waiting for error group managing segments... Jun 26 06:08:21.229550628 (68h56m48.155845315s) [noise=true , address=104.36.82.14:4223 , piece_id=LP74YFI2SRSYPYEIKREVINCOFMVCXLNDS2555YKUVFN63N6TBJDA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: <nil> Jun 26 06:08:21.230245145 (68h56m48.156539832s) [noise=true , address=104.36.82.14:4222 , piece_id=U3MBMAKFBFTOPJZ3HS3FXLPDBLQTN577XNCZLUBHRO6IU4BE5VNA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: <nil> Jun 26 06:08:21.230801067 (68h56m48.15709574s) [noise=true , address=104.36.82.6:4873 , piece_id=TIIMEQUSDLRFF3OK3WBB74F7SWJ5MEC4Z6IQNELDMRZLNTMQT7LA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: <nil> Jun 26 06:08:21.231457382 (68h56m48.157752071s) [noise=true , address=104.36.82.6:4876 , piece_id=SSMGONONY34EHF5W6QMKO7YXB7RLO5LZVTJMMFKTP6GQNGYFYQ4A, seg_pos={0 0}, upload=55354]: Done uploading piece. err: <nil> Jun 26 06:08:21.231563205 (68h56m48.157857893s) [noise=true , address=104.36.82.13:4186 , piece_id=5SVABN3KS6AUL6HILHNQCVF74BA75VPKTTAK7IQAPKRRME3MOZEA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: <nil> Jun 26 06:08:21.239131119 (68h56m48.165425799s) [noise=true , address=104.36.82.7:4912 , piece_id=A5HPC7CEU66IFNS47HO3AQX22LUZ6AZ4L4XAMZ6T5NCVE4D3TAZA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: <nil> Jun 26 06:08:21.534686665 (68h56m48.460981337s) [noise=true , address=104.36.82.15:4258 , piece_id=YXS4J7YSG6JV33JG4BZLK5AXKE3JXLBTGZJXPXZQ4LBYBT3MU2AA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: <nil> Jun 26 06:08:51.073673533 (68h57m17.999968219s) [noise=true , address=45.91.116.9:5261 , piece_id=CZWB3XJVA7B7R7GWFYNW3RCGNVRXQ3E37XDXD4SUXTEMQFZ73HHA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi, address:45.91.116.9:5261): protocol: rpc: dial tcp 45.91.116.9:5261: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:51.073739594 (68h57m18.000034272s) [noise=true , address=45.91.116.10:4082 , piece_id=7NVEGA4QQOE7KYPHXUUX42SJPQL6LP46RGM37VD6A2AWN7PAOM3Q, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj, address:45.91.116.10:4082): protocol: rpc: dial tcp 45.91.116.10:4082: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:51.073789897 (68h57m18.000084584s) [noise=true , address=77.90.31.19:4842 , piece_id=G3AG375DATHB7MGEEWONE5GMMIC7QHG2VVDKF5AEVOSXV5N4S6AA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: rpc: dial tcp 77.90.31.19:4842: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:51.073849570 (68h57m18.000144246s) [noise=true , address=45.91.116.13:4154 , piece_id=STFPXN2LUJYUIAYROWZIRTCMT2Y3XET6KWA4GZWUHENXRJURP62Q, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN, address:45.91.116.13:4154): protocol: write tcp 159.223.143.39:35392->45.91.116.13:4154: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:51.073904716 (68h57m18.000199391s) [noise=true , address=77.90.31.13:4626 , piece_id=MXYBIEDFAFK53CVTE2C6CKEZWOL3MLW4S7LOHK4IHW6QEIZJL76Q, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: write tcp 159.223.143.39:58864->77.90.31.13:4626: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:51.073952402 (68h57m18.000247082s) [noise=true , address=77.90.31.10:4517 , piece_id=VJZ5T4QNVAPNUHNFNK24S6TAS4V4NKNFMQE2FZMCWWLCCB2ELXXQ, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, address:77.90.31.10:4517): protocol: write tcp 159.223.143.39:36382->77.90.31.10:4517: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:51.074022481 (68h57m18.000317156s) [noise=true , address=77.90.31.15:4697 , piece_id=CUXGVL7PS3H7ZQKUCSYXJCBR3ETYFDPZFYIEAWEYL7C2S3RJY3AA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5, address:77.90.31.15:4697): protocol: write tcp 159.223.143.39:52730->77.90.31.15:4697: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:51.074071824 (68h57m18.000366503s) [noise=true , address=77.90.31.11:4554 , piece_id=R2CIFLWS4LXTGSDSIMIRC4UYVJTXC7T35HPWZ2S7VYUTJ2AMY3BQ, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, address:77.90.31.11:4554): protocol: rpc: dial tcp 77.90.31.11:4554: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:51.074102177 (68h57m18.000396855s) [noise=true , address=77.90.31.14:4661 , piece_id=VXI2ZUSGR4N3H7TPIK75RIDFP3SBAOWVRGHNXTZOHFT5FNG4JWJA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue, address:77.90.31.14:4661): protocol: rpc: dial tcp 77.90.31.14:4661: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:51.074119276 (68h57m18.000413966s) [seg_pos={0 0}, upload=55354]: Done waiting for segment. successful: 7 optimal: 13 errs: <nil> Jun 26 06:08:51.100079867 (68h57m18.026374541s) [upload=55354]: Done uploading segments... Jun 26 06:08:51.100114505 (68h57m18.026409191s) [upload=55354]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 7) Duration: 566.15861ms core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.......XX...................................................] duration: 521.709239ms FAIL (2025-06-26 06:46:21 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 521.743041ms edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [........XX..................................................] duration: 658.915449ms FAIL (2025-06-26 06:45:49 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 659.562877ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [..........XXX...............................................] duration: 477.772796ms FAIL (2025-06-26 06:45:35 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 7) ======LOGS====== pload=55354]: Got next segment. Inline: false Jun 26 06:08:21.105827735 (68h56m48.032122411s) [upload=55354]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 26 06:08:21.105833588 (68h56m48.032128274s) [upload=55354]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 26 06:08:21.135318313 (68h56m48.061613009s) [seg_pos={0 0}, upload=55354]: Begin upload segment... Jun 26 06:08:21.135366534 (68h56m48.061661214s) [seg_pos={0 0}, upload=55354]: Done begin upload segment. Jun 26 06:08:21.135390342 (68h56m48.061685028s) [noise=true , address=45.91.116.9:5261 , piece_id=CZWB3XJVA7B7R7GWFYNW3RCGNVRXQ3E37XDXD4SUXTEMQFZ73HHA, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.135405801 (68h56m48.061700497s) [noise=true , address=104.36.82.6:4873 , piece_id=TIIMEQUSDLRFF3OK3WBB74F7SWJ5MEC4Z6IQNELDMRZLNTMQT7LA, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.135774729 (68h56m48.062069418s) [noise=true , address=77.90.31.13:4626 , piece_id=MXYBIEDFAFK53CVTE2C6CKEZWOL3MLW4S7LOHK4IHW6QEIZJL76Q, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.135944436 (68h56m48.06223913s) [noise=true , address=77.90.31.10:4517 , piece_id=VJZ5T4QNVAPNUHNFNK24S6TAS4V4NKNFMQE2FZMCWWLCCB2ELXXQ, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.136080402 (68h56m48.062375094s) [noise=true , address=77.90.31.15:4697 , piece_id=CUXGVL7PS3H7ZQKUCSYXJCBR3ETYFDPZFYIEAWEYL7C2S3RJY3AA, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.136382603 (68h56m48.062677289s) [noise=true , address=104.36.82.13:4186 , piece_id=5SVABN3KS6AUL6HILHNQCVF74BA75VPKTTAK7IQAPKRRME3MOZEA, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.136399304 (68h56m48.062693995s) [noise=true , address=104.36.82.14:4223 , piece_id=LP74YFI2SRSYPYEIKREVINCOFMVCXLNDS2555YKUVFN63N6TBJDA, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.136669576 (68h56m48.062964262s) [noise=true , address=104.36.82.6:4876 , piece_id=SSMGONONY34EHF5W6QMKO7YXB7RLO5LZVTJMMFKTP6GQNGYFYQ4A, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.136809784 (68h56m48.063104477s) [noise=true , address=104.36.82.15:4258 , piece_id=YXS4J7YSG6JV33JG4BZLK5AXKE3JXLBTGZJXPXZQ4LBYBT3MU2AA, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.136950556 (68h56m48.063245242s) [noise=true , address=77.90.31.11:4554 , piece_id=R2CIFLWS4LXTGSDSIMIRC4UYVJTXC7T35HPWZ2S7VYUTJ2AMY3BQ, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.137226497 (68h56m48.06352119s) [noise=true , address=77.90.31.14:4661 , piece_id=VXI2ZUSGR4N3H7TPIK75RIDFP3SBAOWVRGHNXTZOHFT5FNG4JWJA, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.138099107 (68h56m48.064393792s) [noise=true , address=45.91.116.10:4082 , piece_id=7NVEGA4QQOE7KYPHXUUX42SJPQL6LP46RGM37VD6A2AWN7PAOM3Q, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.139330448 (68h56m48.065625141s) [noise=true , address=104.36.82.14:4222 , piece_id=U3MBMAKFBFTOPJZ3HS3FXLPDBLQTN577XNCZLUBHRO6IU4BE5VNA, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.140580734 (68h56m48.06687543s) [noise=true , address=77.90.31.19:4842 , piece_id=G3AG375DATHB7MGEEWONE5GMMIC7QHG2VVDKF5AEVOSXV5N4S6AA, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.141452300 (68h56m48.067747027s) [noise=true , address=45.91.116.13:4154 , piece_id=STFPXN2LUJYUIAYROWZIRTCMT2Y3XET6KWA4GZWUHENXRJURP62Q, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.142917062 (68h56m48.069211739s) [noise=true , address=104.36.82.7:4912 , piece_id=A5HPC7CEU66IFNS47HO3AQX22LUZ6AZ4L4XAMZ6T5NCVE4D3TAZA, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.221824189 (68h56m48.148118865s) [upload=55354]: Next returned nil segment Jun 26 06:08:21.221837958 (68h56m48.148132644s) [upload=55354]: Waiting for error group managing segments... Jun 26 06:08:21.229550628 (68h56m48.155845315s) [noise=true , address=104.36.82.14:4223 , piece_id=LP74YFI2SRSYPYEIKREVINCOFMVCXLNDS2555YKUVFN63N6TBJDA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: <nil> Jun 26 06:08:21.230245145 (68h56m48.156539832s) [noise=true , address=104.36.82.14:4222 , piece_id=U3MBMAKFBFTOPJZ3HS3FXLPDBLQTN577XNCZLUBHRO6IU4BE5VNA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: <nil> Jun 26 06:08:21.230801067 (68h56m48.15709574s) [noise=true , address=104.36.82.6:4873 , piece_id=TIIMEQUSDLRFF3OK3WBB74F7SWJ5MEC4Z6IQNELDMRZLNTMQT7LA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: <nil> Jun 26 06:08:21.231457382 (68h56m48.157752071s) [noise=true , address=104.36.82.6:4876 , piece_id=SSMGONONY34EHF5W6QMKO7YXB7RLO5LZVTJMMFKTP6GQNGYFYQ4A, seg_pos={0 0}, upload=55354]: Done uploading piece. err: <nil> Jun 26 06:08:21.231563205 (68h56m48.157857893s) [noise=true , address=104.36.82.13:4186 , piece_id=5SVABN3KS6AUL6HILHNQCVF74BA75VPKTTAK7IQAPKRRME3MOZEA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: <nil> Jun 26 06:08:21.239131119 (68h56m48.165425799s) [noise=true , address=104.36.82.7:4912 , piece_id=A5HPC7CEU66IFNS47HO3AQX22LUZ6AZ4L4XAMZ6T5NCVE4D3TAZA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: <nil> Jun 26 06:08:21.534686665 (68h56m48.460981337s) [noise=true , address=104.36.82.15:4258 , piece_id=YXS4J7YSG6JV33JG4BZLK5AXKE3JXLBTGZJXPXZQ4LBYBT3MU2AA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: <nil> Jun 26 06:08:51.073673533 (68h57m17.999968219s) [noise=true , address=45.91.116.9:5261 , piece_id=CZWB3XJVA7B7R7GWFYNW3RCGNVRXQ3E37XDXD4SUXTEMQFZ73HHA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi, address:45.91.116.9:5261): protocol: rpc: dial tcp 45.91.116.9:5261: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:51.073739594 (68h57m18.000034272s) [noise=true , address=45.91.116.10:4082 , piece_id=7NVEGA4QQOE7KYPHXUUX42SJPQL6LP46RGM37VD6A2AWN7PAOM3Q, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj, address:45.91.116.10:4082): protocol: rpc: dial tcp 45.91.116.10:4082: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:51.073789897 (68h57m18.000084584s) [noise=true , address=77.90.31.19:4842 , piece_id=G3AG375DATHB7MGEEWONE5GMMIC7QHG2VVDKF5AEVOSXV5N4S6AA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: rpc: dial tcp 77.90.31.19:4842: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:51.073849570 (68h57m18.000144246s) [noise=true , address=45.91.116.13:4154 , piece_id=STFPXN2LUJYUIAYROWZIRTCMT2Y3XET6KWA4GZWUHENXRJURP62Q, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN, address:45.91.116.13:4154): protocol: write tcp 159.223.143.39:35392->45.91.116.13:4154: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:51.073904716 (68h57m18.000199391s) [noise=true , address=77.90.31.13:4626 , piece_id=MXYBIEDFAFK53CVTE2C6CKEZWOL3MLW4S7LOHK4IHW6QEIZJL76Q, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: write tcp 159.223.143.39:58864->77.90.31.13:4626: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:51.073952402 (68h57m18.000247082s) [noise=true , address=77.90.31.10:4517 , piece_id=VJZ5T4QNVAPNUHNFNK24S6TAS4V4NKNFMQE2FZMCWWLCCB2ELXXQ, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, address:77.90.31.10:4517): protocol: write tcp 159.223.143.39:36382->77.90.31.10:4517: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:51.074022481 (68h57m18.000317156s) [noise=true , address=77.90.31.15:4697 , piece_id=CUXGVL7PS3H7ZQKUCSYXJCBR3ETYFDPZFYIEAWEYL7C2S3RJY3AA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5, address:77.90.31.15:4697): protocol: write tcp 159.223.143.39:52730->77.90.31.15:4697: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:51.074071824 (68h57m18.000366503s) [noise=true , address=77.90.31.11:4554 , piece_id=R2CIFLWS4LXTGSDSIMIRC4UYVJTXC7T35HPWZ2S7VYUTJ2AMY3BQ, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, address:77.90.31.11:4554): protocol: rpc: dial tcp 77.90.31.11:4554: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:51.074102177 (68h57m18.000396855s) [noise=true , address=77.90.31.14:4661 , piece_id=VXI2ZUSGR4N3H7TPIK75RIDFP3SBAOWVRGHNXTZOHFT5FNG4JWJA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue, address:77.90.31.14:4661): protocol: rpc: dial tcp 77.90.31.14:4661: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:51.074119276 (68h57m18.000413966s) [seg_pos={0 0}, upload=55354]: Done waiting for segment. successful: 7 optimal: 13 errs: <nil> Jun 26 06:08:51.100079867 (68h57m18.026374541s) [upload=55354]: Done uploading segments... Jun 26 06:08:51.100114505 (68h57m18.026409191s) [upload=55354]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 7) Duration: 477.803712ms core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.........XX.................................................] duration: 561.024801ms FAIL (2025-06-26 06:45:33 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 561.055059ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [...........XXX..............................................] duration: 405.426803ms FAIL (2025-06-26 06:44:25 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 7) ======LOGS====== pload=55354]: Got next segment. Inline: false Jun 26 06:08:21.105827735 (68h56m48.032122411s) [upload=55354]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 26 06:08:21.105833588 (68h56m48.032128274s) [upload=55354]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 26 06:08:21.135318313 (68h56m48.061613009s) [seg_pos={0 0}, upload=55354]: Begin upload segment... Jun 26 06:08:21.135366534 (68h56m48.061661214s) [seg_pos={0 0}, upload=55354]: Done begin upload segment. Jun 26 06:08:21.135390342 (68h56m48.061685028s) [noise=true , address=45.91.116.9:5261 , piece_id=CZWB3XJVA7B7R7GWFYNW3RCGNVRXQ3E37XDXD4SUXTEMQFZ73HHA, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.135405801 (68h56m48.061700497s) [noise=true , address=104.36.82.6:4873 , piece_id=TIIMEQUSDLRFF3OK3WBB74F7SWJ5MEC4Z6IQNELDMRZLNTMQT7LA, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.135774729 (68h56m48.062069418s) [noise=true , address=77.90.31.13:4626 , piece_id=MXYBIEDFAFK53CVTE2C6CKEZWOL3MLW4S7LOHK4IHW6QEIZJL76Q, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.135944436 (68h56m48.06223913s) [noise=true , address=77.90.31.10:4517 , piece_id=VJZ5T4QNVAPNUHNFNK24S6TAS4V4NKNFMQE2FZMCWWLCCB2ELXXQ, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.136080402 (68h56m48.062375094s) [noise=true , address=77.90.31.15:4697 , piece_id=CUXGVL7PS3H7ZQKUCSYXJCBR3ETYFDPZFYIEAWEYL7C2S3RJY3AA, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.136382603 (68h56m48.062677289s) [noise=true , address=104.36.82.13:4186 , piece_id=5SVABN3KS6AUL6HILHNQCVF74BA75VPKTTAK7IQAPKRRME3MOZEA, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.136399304 (68h56m48.062693995s) [noise=true , address=104.36.82.14:4223 , piece_id=LP74YFI2SRSYPYEIKREVINCOFMVCXLNDS2555YKUVFN63N6TBJDA, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.136669576 (68h56m48.062964262s) [noise=true , address=104.36.82.6:4876 , piece_id=SSMGONONY34EHF5W6QMKO7YXB7RLO5LZVTJMMFKTP6GQNGYFYQ4A, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.136809784 (68h56m48.063104477s) [noise=true , address=104.36.82.15:4258 , piece_id=YXS4J7YSG6JV33JG4BZLK5AXKE3JXLBTGZJXPXZQ4LBYBT3MU2AA, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.136950556 (68h56m48.063245242s) [noise=true , address=77.90.31.11:4554 , piece_id=R2CIFLWS4LXTGSDSIMIRC4UYVJTXC7T35HPWZ2S7VYUTJ2AMY3BQ, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.137226497 (68h56m48.06352119s) [noise=true , address=77.90.31.14:4661 , piece_id=VXI2ZUSGR4N3H7TPIK75RIDFP3SBAOWVRGHNXTZOHFT5FNG4JWJA, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.138099107 (68h56m48.064393792s) [noise=true , address=45.91.116.10:4082 , piece_id=7NVEGA4QQOE7KYPHXUUX42SJPQL6LP46RGM37VD6A2AWN7PAOM3Q, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.139330448 (68h56m48.065625141s) [noise=true , address=104.36.82.14:4222 , piece_id=U3MBMAKFBFTOPJZ3HS3FXLPDBLQTN577XNCZLUBHRO6IU4BE5VNA, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.140580734 (68h56m48.06687543s) [noise=true , address=77.90.31.19:4842 , piece_id=G3AG375DATHB7MGEEWONE5GMMIC7QHG2VVDKF5AEVOSXV5N4S6AA, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.141452300 (68h56m48.067747027s) [noise=true , address=45.91.116.13:4154 , piece_id=STFPXN2LUJYUIAYROWZIRTCMT2Y3XET6KWA4GZWUHENXRJURP62Q, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.142917062 (68h56m48.069211739s) [noise=true , address=104.36.82.7:4912 , piece_id=A5HPC7CEU66IFNS47HO3AQX22LUZ6AZ4L4XAMZ6T5NCVE4D3TAZA, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.221824189 (68h56m48.148118865s) [upload=55354]: Next returned nil segment Jun 26 06:08:21.221837958 (68h56m48.148132644s) [upload=55354]: Waiting for error group managing segments... Jun 26 06:08:21.229550628 (68h56m48.155845315s) [noise=true , address=104.36.82.14:4223 , piece_id=LP74YFI2SRSYPYEIKREVINCOFMVCXLNDS2555YKUVFN63N6TBJDA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: <nil> Jun 26 06:08:21.230245145 (68h56m48.156539832s) [noise=true , address=104.36.82.14:4222 , piece_id=U3MBMAKFBFTOPJZ3HS3FXLPDBLQTN577XNCZLUBHRO6IU4BE5VNA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: <nil> Jun 26 06:08:21.230801067 (68h56m48.15709574s) [noise=true , address=104.36.82.6:4873 , piece_id=TIIMEQUSDLRFF3OK3WBB74F7SWJ5MEC4Z6IQNELDMRZLNTMQT7LA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: <nil> Jun 26 06:08:21.231457382 (68h56m48.157752071s) [noise=true , address=104.36.82.6:4876 , piece_id=SSMGONONY34EHF5W6QMKO7YXB7RLO5LZVTJMMFKTP6GQNGYFYQ4A, seg_pos={0 0}, upload=55354]: Done uploading piece. err: <nil> Jun 26 06:08:21.231563205 (68h56m48.157857893s) [noise=true , address=104.36.82.13:4186 , piece_id=5SVABN3KS6AUL6HILHNQCVF74BA75VPKTTAK7IQAPKRRME3MOZEA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: <nil> Jun 26 06:08:21.239131119 (68h56m48.165425799s) [noise=true , address=104.36.82.7:4912 , piece_id=A5HPC7CEU66IFNS47HO3AQX22LUZ6AZ4L4XAMZ6T5NCVE4D3TAZA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: <nil> Jun 26 06:08:21.534686665 (68h56m48.460981337s) [noise=true , address=104.36.82.15:4258 , piece_id=YXS4J7YSG6JV33JG4BZLK5AXKE3JXLBTGZJXPXZQ4LBYBT3MU2AA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: <nil> Jun 26 06:08:51.073673533 (68h57m17.999968219s) [noise=true , address=45.91.116.9:5261 , piece_id=CZWB3XJVA7B7R7GWFYNW3RCGNVRXQ3E37XDXD4SUXTEMQFZ73HHA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi, address:45.91.116.9:5261): protocol: rpc: dial tcp 45.91.116.9:5261: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:51.073739594 (68h57m18.000034272s) [noise=true , address=45.91.116.10:4082 , piece_id=7NVEGA4QQOE7KYPHXUUX42SJPQL6LP46RGM37VD6A2AWN7PAOM3Q, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj, address:45.91.116.10:4082): protocol: rpc: dial tcp 45.91.116.10:4082: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:51.073789897 (68h57m18.000084584s) [noise=true , address=77.90.31.19:4842 , piece_id=G3AG375DATHB7MGEEWONE5GMMIC7QHG2VVDKF5AEVOSXV5N4S6AA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: rpc: dial tcp 77.90.31.19:4842: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:51.073849570 (68h57m18.000144246s) [noise=true , address=45.91.116.13:4154 , piece_id=STFPXN2LUJYUIAYROWZIRTCMT2Y3XET6KWA4GZWUHENXRJURP62Q, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN, address:45.91.116.13:4154): protocol: write tcp 159.223.143.39:35392->45.91.116.13:4154: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:51.073904716 (68h57m18.000199391s) [noise=true , address=77.90.31.13:4626 , piece_id=MXYBIEDFAFK53CVTE2C6CKEZWOL3MLW4S7LOHK4IHW6QEIZJL76Q, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: write tcp 159.223.143.39:58864->77.90.31.13:4626: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:51.073952402 (68h57m18.000247082s) [noise=true , address=77.90.31.10:4517 , piece_id=VJZ5T4QNVAPNUHNFNK24S6TAS4V4NKNFMQE2FZMCWWLCCB2ELXXQ, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, address:77.90.31.10:4517): protocol: write tcp 159.223.143.39:36382->77.90.31.10:4517: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:51.074022481 (68h57m18.000317156s) [noise=true , address=77.90.31.15:4697 , piece_id=CUXGVL7PS3H7ZQKUCSYXJCBR3ETYFDPZFYIEAWEYL7C2S3RJY3AA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5, address:77.90.31.15:4697): protocol: write tcp 159.223.143.39:52730->77.90.31.15:4697: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:51.074071824 (68h57m18.000366503s) [noise=true , address=77.90.31.11:4554 , piece_id=R2CIFLWS4LXTGSDSIMIRC4UYVJTXC7T35HPWZ2S7VYUTJ2AMY3BQ, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, address:77.90.31.11:4554): protocol: rpc: dial tcp 77.90.31.11:4554: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:51.074102177 (68h57m18.000396855s) [noise=true , address=77.90.31.14:4661 , piece_id=VXI2ZUSGR4N3H7TPIK75RIDFP3SBAOWVRGHNXTZOHFT5FNG4JWJA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue, address:77.90.31.14:4661): protocol: rpc: dial tcp 77.90.31.14:4661: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:51.074119276 (68h57m18.000413966s) [seg_pos={0 0}, upload=55354]: Done waiting for segment. successful: 7 optimal: 13 errs: <nil> Jun 26 06:08:51.100079867 (68h57m18.026374541s) [upload=55354]: Done uploading segments... Jun 26 06:08:51.100114505 (68h57m18.026409191s) [upload=55354]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 7) Duration: 405.456708ms core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..........XX................................................] duration: 582.507409ms FAIL (2025-06-26 06:44:21 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 582.553878ms edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...........XX...............................................] duration: 506.135157ms FAIL (2025-06-26 06:43:33 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 506.368884ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [............XXX.............................................] duration: 446.903165ms FAIL (2025-06-26 06:43:19 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 7) ======LOGS====== pload=55354]: Got next segment. Inline: false Jun 26 06:08:21.105827735 (68h56m48.032122411s) [upload=55354]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 26 06:08:21.105833588 (68h56m48.032128274s) [upload=55354]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 26 06:08:21.135318313 (68h56m48.061613009s) [seg_pos={0 0}, upload=55354]: Begin upload segment... Jun 26 06:08:21.135366534 (68h56m48.061661214s) [seg_pos={0 0}, upload=55354]: Done begin upload segment. Jun 26 06:08:21.135390342 (68h56m48.061685028s) [noise=true , address=45.91.116.9:5261 , piece_id=CZWB3XJVA7B7R7GWFYNW3RCGNVRXQ3E37XDXD4SUXTEMQFZ73HHA, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.135405801 (68h56m48.061700497s) [noise=true , address=104.36.82.6:4873 , piece_id=TIIMEQUSDLRFF3OK3WBB74F7SWJ5MEC4Z6IQNELDMRZLNTMQT7LA, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.135774729 (68h56m48.062069418s) [noise=true , address=77.90.31.13:4626 , piece_id=MXYBIEDFAFK53CVTE2C6CKEZWOL3MLW4S7LOHK4IHW6QEIZJL76Q, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.135944436 (68h56m48.06223913s) [noise=true , address=77.90.31.10:4517 , piece_id=VJZ5T4QNVAPNUHNFNK24S6TAS4V4NKNFMQE2FZMCWWLCCB2ELXXQ, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.136080402 (68h56m48.062375094s) [noise=true , address=77.90.31.15:4697 , piece_id=CUXGVL7PS3H7ZQKUCSYXJCBR3ETYFDPZFYIEAWEYL7C2S3RJY3AA, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.136382603 (68h56m48.062677289s) [noise=true , address=104.36.82.13:4186 , piece_id=5SVABN3KS6AUL6HILHNQCVF74BA75VPKTTAK7IQAPKRRME3MOZEA, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.136399304 (68h56m48.062693995s) [noise=true , address=104.36.82.14:4223 , piece_id=LP74YFI2SRSYPYEIKREVINCOFMVCXLNDS2555YKUVFN63N6TBJDA, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.136669576 (68h56m48.062964262s) [noise=true , address=104.36.82.6:4876 , piece_id=SSMGONONY34EHF5W6QMKO7YXB7RLO5LZVTJMMFKTP6GQNGYFYQ4A, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.136809784 (68h56m48.063104477s) [noise=true , address=104.36.82.15:4258 , piece_id=YXS4J7YSG6JV33JG4BZLK5AXKE3JXLBTGZJXPXZQ4LBYBT3MU2AA, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.136950556 (68h56m48.063245242s) [noise=true , address=77.90.31.11:4554 , piece_id=R2CIFLWS4LXTGSDSIMIRC4UYVJTXC7T35HPWZ2S7VYUTJ2AMY3BQ, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.137226497 (68h56m48.06352119s) [noise=true , address=77.90.31.14:4661 , piece_id=VXI2ZUSGR4N3H7TPIK75RIDFP3SBAOWVRGHNXTZOHFT5FNG4JWJA, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.138099107 (68h56m48.064393792s) [noise=true , address=45.91.116.10:4082 , piece_id=7NVEGA4QQOE7KYPHXUUX42SJPQL6LP46RGM37VD6A2AWN7PAOM3Q, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.139330448 (68h56m48.065625141s) [noise=true , address=104.36.82.14:4222 , piece_id=U3MBMAKFBFTOPJZ3HS3FXLPDBLQTN577XNCZLUBHRO6IU4BE5VNA, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.140580734 (68h56m48.06687543s) [noise=true , address=77.90.31.19:4842 , piece_id=G3AG375DATHB7MGEEWONE5GMMIC7QHG2VVDKF5AEVOSXV5N4S6AA, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.141452300 (68h56m48.067747027s) [noise=true , address=45.91.116.13:4154 , piece_id=STFPXN2LUJYUIAYROWZIRTCMT2Y3XET6KWA4GZWUHENXRJURP62Q, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.142917062 (68h56m48.069211739s) [noise=true , address=104.36.82.7:4912 , piece_id=A5HPC7CEU66IFNS47HO3AQX22LUZ6AZ4L4XAMZ6T5NCVE4D3TAZA, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.221824189 (68h56m48.148118865s) [upload=55354]: Next returned nil segment Jun 26 06:08:21.221837958 (68h56m48.148132644s) [upload=55354]: Waiting for error group managing segments... Jun 26 06:08:21.229550628 (68h56m48.155845315s) [noise=true , address=104.36.82.14:4223 , piece_id=LP74YFI2SRSYPYEIKREVINCOFMVCXLNDS2555YKUVFN63N6TBJDA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: <nil> Jun 26 06:08:21.230245145 (68h56m48.156539832s) [noise=true , address=104.36.82.14:4222 , piece_id=U3MBMAKFBFTOPJZ3HS3FXLPDBLQTN577XNCZLUBHRO6IU4BE5VNA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: <nil> Jun 26 06:08:21.230801067 (68h56m48.15709574s) [noise=true , address=104.36.82.6:4873 , piece_id=TIIMEQUSDLRFF3OK3WBB74F7SWJ5MEC4Z6IQNELDMRZLNTMQT7LA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: <nil> Jun 26 06:08:21.231457382 (68h56m48.157752071s) [noise=true , address=104.36.82.6:4876 , piece_id=SSMGONONY34EHF5W6QMKO7YXB7RLO5LZVTJMMFKTP6GQNGYFYQ4A, seg_pos={0 0}, upload=55354]: Done uploading piece. err: <nil> Jun 26 06:08:21.231563205 (68h56m48.157857893s) [noise=true , address=104.36.82.13:4186 , piece_id=5SVABN3KS6AUL6HILHNQCVF74BA75VPKTTAK7IQAPKRRME3MOZEA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: <nil> Jun 26 06:08:21.239131119 (68h56m48.165425799s) [noise=true , address=104.36.82.7:4912 , piece_id=A5HPC7CEU66IFNS47HO3AQX22LUZ6AZ4L4XAMZ6T5NCVE4D3TAZA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: <nil> Jun 26 06:08:21.534686665 (68h56m48.460981337s) [noise=true , address=104.36.82.15:4258 , piece_id=YXS4J7YSG6JV33JG4BZLK5AXKE3JXLBTGZJXPXZQ4LBYBT3MU2AA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: <nil> Jun 26 06:08:51.073673533 (68h57m17.999968219s) [noise=true , address=45.91.116.9:5261 , piece_id=CZWB3XJVA7B7R7GWFYNW3RCGNVRXQ3E37XDXD4SUXTEMQFZ73HHA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi, address:45.91.116.9:5261): protocol: rpc: dial tcp 45.91.116.9:5261: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:51.073739594 (68h57m18.000034272s) [noise=true , address=45.91.116.10:4082 , piece_id=7NVEGA4QQOE7KYPHXUUX42SJPQL6LP46RGM37VD6A2AWN7PAOM3Q, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj, address:45.91.116.10:4082): protocol: rpc: dial tcp 45.91.116.10:4082: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:51.073789897 (68h57m18.000084584s) [noise=true , address=77.90.31.19:4842 , piece_id=G3AG375DATHB7MGEEWONE5GMMIC7QHG2VVDKF5AEVOSXV5N4S6AA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: rpc: dial tcp 77.90.31.19:4842: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:51.073849570 (68h57m18.000144246s) [noise=true , address=45.91.116.13:4154 , piece_id=STFPXN2LUJYUIAYROWZIRTCMT2Y3XET6KWA4GZWUHENXRJURP62Q, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN, address:45.91.116.13:4154): protocol: write tcp 159.223.143.39:35392->45.91.116.13:4154: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:51.073904716 (68h57m18.000199391s) [noise=true , address=77.90.31.13:4626 , piece_id=MXYBIEDFAFK53CVTE2C6CKEZWOL3MLW4S7LOHK4IHW6QEIZJL76Q, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: write tcp 159.223.143.39:58864->77.90.31.13:4626: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:51.073952402 (68h57m18.000247082s) [noise=true , address=77.90.31.10:4517 , piece_id=VJZ5T4QNVAPNUHNFNK24S6TAS4V4NKNFMQE2FZMCWWLCCB2ELXXQ, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, address:77.90.31.10:4517): protocol: write tcp 159.223.143.39:36382->77.90.31.10:4517: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:51.074022481 (68h57m18.000317156s) [noise=true , address=77.90.31.15:4697 , piece_id=CUXGVL7PS3H7ZQKUCSYXJCBR3ETYFDPZFYIEAWEYL7C2S3RJY3AA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5, address:77.90.31.15:4697): protocol: write tcp 159.223.143.39:52730->77.90.31.15:4697: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:51.074071824 (68h57m18.000366503s) [noise=true , address=77.90.31.11:4554 , piece_id=R2CIFLWS4LXTGSDSIMIRC4UYVJTXC7T35HPWZ2S7VYUTJ2AMY3BQ, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, address:77.90.31.11:4554): protocol: rpc: dial tcp 77.90.31.11:4554: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:51.074102177 (68h57m18.000396855s) [noise=true , address=77.90.31.14:4661 , piece_id=VXI2ZUSGR4N3H7TPIK75RIDFP3SBAOWVRGHNXTZOHFT5FNG4JWJA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue, address:77.90.31.14:4661): protocol: rpc: dial tcp 77.90.31.14:4661: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:51.074119276 (68h57m18.000413966s) [seg_pos={0 0}, upload=55354]: Done waiting for segment. successful: 7 optimal: 13 errs: <nil> Jun 26 06:08:51.100079867 (68h57m18.026374541s) [upload=55354]: Done uploading segments... Jun 26 06:08:51.100114505 (68h57m18.026409191s) [upload=55354]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 7) Duration: 446.928901ms core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [............XX..............................................] duration: 485.697298ms FAIL (2025-06-26 06:43:13 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 485.721529ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.............XXX............................................] duration: 418.684488ms FAIL (2025-06-26 06:42:49 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 7) ======LOGS====== pload=55354]: Got next segment. Inline: false Jun 26 06:08:21.105827735 (68h56m48.032122411s) [upload=55354]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 26 06:08:21.105833588 (68h56m48.032128274s) [upload=55354]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 26 06:08:21.135318313 (68h56m48.061613009s) [seg_pos={0 0}, upload=55354]: Begin upload segment... Jun 26 06:08:21.135366534 (68h56m48.061661214s) [seg_pos={0 0}, upload=55354]: Done begin upload segment. Jun 26 06:08:21.135390342 (68h56m48.061685028s) [noise=true , address=45.91.116.9:5261 , piece_id=CZWB3XJVA7B7R7GWFYNW3RCGNVRXQ3E37XDXD4SUXTEMQFZ73HHA, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.135405801 (68h56m48.061700497s) [noise=true , address=104.36.82.6:4873 , piece_id=TIIMEQUSDLRFF3OK3WBB74F7SWJ5MEC4Z6IQNELDMRZLNTMQT7LA, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.135774729 (68h56m48.062069418s) [noise=true , address=77.90.31.13:4626 , piece_id=MXYBIEDFAFK53CVTE2C6CKEZWOL3MLW4S7LOHK4IHW6QEIZJL76Q, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.135944436 (68h56m48.06223913s) [noise=true , address=77.90.31.10:4517 , piece_id=VJZ5T4QNVAPNUHNFNK24S6TAS4V4NKNFMQE2FZMCWWLCCB2ELXXQ, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.136080402 (68h56m48.062375094s) [noise=true , address=77.90.31.15:4697 , piece_id=CUXGVL7PS3H7ZQKUCSYXJCBR3ETYFDPZFYIEAWEYL7C2S3RJY3AA, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.136382603 (68h56m48.062677289s) [noise=true , address=104.36.82.13:4186 , piece_id=5SVABN3KS6AUL6HILHNQCVF74BA75VPKTTAK7IQAPKRRME3MOZEA, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.136399304 (68h56m48.062693995s) [noise=true , address=104.36.82.14:4223 , piece_id=LP74YFI2SRSYPYEIKREVINCOFMVCXLNDS2555YKUVFN63N6TBJDA, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.136669576 (68h56m48.062964262s) [noise=true , address=104.36.82.6:4876 , piece_id=SSMGONONY34EHF5W6QMKO7YXB7RLO5LZVTJMMFKTP6GQNGYFYQ4A, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.136809784 (68h56m48.063104477s) [noise=true , address=104.36.82.15:4258 , piece_id=YXS4J7YSG6JV33JG4BZLK5AXKE3JXLBTGZJXPXZQ4LBYBT3MU2AA, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.136950556 (68h56m48.063245242s) [noise=true , address=77.90.31.11:4554 , piece_id=R2CIFLWS4LXTGSDSIMIRC4UYVJTXC7T35HPWZ2S7VYUTJ2AMY3BQ, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.137226497 (68h56m48.06352119s) [noise=true , address=77.90.31.14:4661 , piece_id=VXI2ZUSGR4N3H7TPIK75RIDFP3SBAOWVRGHNXTZOHFT5FNG4JWJA, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.138099107 (68h56m48.064393792s) [noise=true , address=45.91.116.10:4082 , piece_id=7NVEGA4QQOE7KYPHXUUX42SJPQL6LP46RGM37VD6A2AWN7PAOM3Q, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.139330448 (68h56m48.065625141s) [noise=true , address=104.36.82.14:4222 , piece_id=U3MBMAKFBFTOPJZ3HS3FXLPDBLQTN577XNCZLUBHRO6IU4BE5VNA, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.140580734 (68h56m48.06687543s) [noise=true , address=77.90.31.19:4842 , piece_id=G3AG375DATHB7MGEEWONE5GMMIC7QHG2VVDKF5AEVOSXV5N4S6AA, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.141452300 (68h56m48.067747027s) [noise=true , address=45.91.116.13:4154 , piece_id=STFPXN2LUJYUIAYROWZIRTCMT2Y3XET6KWA4GZWUHENXRJURP62Q, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.142917062 (68h56m48.069211739s) [noise=true , address=104.36.82.7:4912 , piece_id=A5HPC7CEU66IFNS47HO3AQX22LUZ6AZ4L4XAMZ6T5NCVE4D3TAZA, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.221824189 (68h56m48.148118865s) [upload=55354]: Next returned nil segment Jun 26 06:08:21.221837958 (68h56m48.148132644s) [upload=55354]: Waiting for error group managing segments... Jun 26 06:08:21.229550628 (68h56m48.155845315s) [noise=true , address=104.36.82.14:4223 , piece_id=LP74YFI2SRSYPYEIKREVINCOFMVCXLNDS2555YKUVFN63N6TBJDA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: <nil> Jun 26 06:08:21.230245145 (68h56m48.156539832s) [noise=true , address=104.36.82.14:4222 , piece_id=U3MBMAKFBFTOPJZ3HS3FXLPDBLQTN577XNCZLUBHRO6IU4BE5VNA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: <nil> Jun 26 06:08:21.230801067 (68h56m48.15709574s) [noise=true , address=104.36.82.6:4873 , piece_id=TIIMEQUSDLRFF3OK3WBB74F7SWJ5MEC4Z6IQNELDMRZLNTMQT7LA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: <nil> Jun 26 06:08:21.231457382 (68h56m48.157752071s) [noise=true , address=104.36.82.6:4876 , piece_id=SSMGONONY34EHF5W6QMKO7YXB7RLO5LZVTJMMFKTP6GQNGYFYQ4A, seg_pos={0 0}, upload=55354]: Done uploading piece. err: <nil> Jun 26 06:08:21.231563205 (68h56m48.157857893s) [noise=true , address=104.36.82.13:4186 , piece_id=5SVABN3KS6AUL6HILHNQCVF74BA75VPKTTAK7IQAPKRRME3MOZEA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: <nil> Jun 26 06:08:21.239131119 (68h56m48.165425799s) [noise=true , address=104.36.82.7:4912 , piece_id=A5HPC7CEU66IFNS47HO3AQX22LUZ6AZ4L4XAMZ6T5NCVE4D3TAZA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: <nil> Jun 26 06:08:21.534686665 (68h56m48.460981337s) [noise=true , address=104.36.82.15:4258 , piece_id=YXS4J7YSG6JV33JG4BZLK5AXKE3JXLBTGZJXPXZQ4LBYBT3MU2AA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: <nil> Jun 26 06:08:51.073673533 (68h57m17.999968219s) [noise=true , address=45.91.116.9:5261 , piece_id=CZWB3XJVA7B7R7GWFYNW3RCGNVRXQ3E37XDXD4SUXTEMQFZ73HHA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi, address:45.91.116.9:5261): protocol: rpc: dial tcp 45.91.116.9:5261: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:51.073739594 (68h57m18.000034272s) [noise=true , address=45.91.116.10:4082 , piece_id=7NVEGA4QQOE7KYPHXUUX42SJPQL6LP46RGM37VD6A2AWN7PAOM3Q, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj, address:45.91.116.10:4082): protocol: rpc: dial tcp 45.91.116.10:4082: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:51.073789897 (68h57m18.000084584s) [noise=true , address=77.90.31.19:4842 , piece_id=G3AG375DATHB7MGEEWONE5GMMIC7QHG2VVDKF5AEVOSXV5N4S6AA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: rpc: dial tcp 77.90.31.19:4842: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:51.073849570 (68h57m18.000144246s) [noise=true , address=45.91.116.13:4154 , piece_id=STFPXN2LUJYUIAYROWZIRTCMT2Y3XET6KWA4GZWUHENXRJURP62Q, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN, address:45.91.116.13:4154): protocol: write tcp 159.223.143.39:35392->45.91.116.13:4154: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:51.073904716 (68h57m18.000199391s) [noise=true , address=77.90.31.13:4626 , piece_id=MXYBIEDFAFK53CVTE2C6CKEZWOL3MLW4S7LOHK4IHW6QEIZJL76Q, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: write tcp 159.223.143.39:58864->77.90.31.13:4626: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:51.073952402 (68h57m18.000247082s) [noise=true , address=77.90.31.10:4517 , piece_id=VJZ5T4QNVAPNUHNFNK24S6TAS4V4NKNFMQE2FZMCWWLCCB2ELXXQ, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, address:77.90.31.10:4517): protocol: write tcp 159.223.143.39:36382->77.90.31.10:4517: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:51.074022481 (68h57m18.000317156s) [noise=true , address=77.90.31.15:4697 , piece_id=CUXGVL7PS3H7ZQKUCSYXJCBR3ETYFDPZFYIEAWEYL7C2S3RJY3AA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5, address:77.90.31.15:4697): protocol: write tcp 159.223.143.39:52730->77.90.31.15:4697: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:51.074071824 (68h57m18.000366503s) [noise=true , address=77.90.31.11:4554 , piece_id=R2CIFLWS4LXTGSDSIMIRC4UYVJTXC7T35HPWZ2S7VYUTJ2AMY3BQ, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, address:77.90.31.11:4554): protocol: rpc: dial tcp 77.90.31.11:4554: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:51.074102177 (68h57m18.000396855s) [noise=true , address=77.90.31.14:4661 , piece_id=VXI2ZUSGR4N3H7TPIK75RIDFP3SBAOWVRGHNXTZOHFT5FNG4JWJA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue, address:77.90.31.14:4661): protocol: rpc: dial tcp 77.90.31.14:4661: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:51.074119276 (68h57m18.000413966s) [seg_pos={0 0}, upload=55354]: Done waiting for segment. successful: 7 optimal: 13 errs: <nil> Jun 26 06:08:51.100079867 (68h57m18.026374541s) [upload=55354]: Done uploading segments... Jun 26 06:08:51.100114505 (68h57m18.026409191s) [upload=55354]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 7) Duration: 418.708989ms core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.............XX.............................................] duration: 459.607381ms FAIL (2025-06-26 06:42:00 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 459.632506ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [..............XXX...........................................] duration: 421.118155ms FAIL (2025-06-26 06:41:55 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 7) ======LOGS====== pload=55354]: Got next segment. Inline: false Jun 26 06:08:21.105827735 (68h56m48.032122411s) [upload=55354]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 26 06:08:21.105833588 (68h56m48.032128274s) [upload=55354]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 26 06:08:21.135318313 (68h56m48.061613009s) [seg_pos={0 0}, upload=55354]: Begin upload segment... Jun 26 06:08:21.135366534 (68h56m48.061661214s) [seg_pos={0 0}, upload=55354]: Done begin upload segment. Jun 26 06:08:21.135390342 (68h56m48.061685028s) [noise=true , address=45.91.116.9:5261 , piece_id=CZWB3XJVA7B7R7GWFYNW3RCGNVRXQ3E37XDXD4SUXTEMQFZ73HHA, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.135405801 (68h56m48.061700497s) [noise=true , address=104.36.82.6:4873 , piece_id=TIIMEQUSDLRFF3OK3WBB74F7SWJ5MEC4Z6IQNELDMRZLNTMQT7LA, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.135774729 (68h56m48.062069418s) [noise=true , address=77.90.31.13:4626 , piece_id=MXYBIEDFAFK53CVTE2C6CKEZWOL3MLW4S7LOHK4IHW6QEIZJL76Q, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.135944436 (68h56m48.06223913s) [noise=true , address=77.90.31.10:4517 , piece_id=VJZ5T4QNVAPNUHNFNK24S6TAS4V4NKNFMQE2FZMCWWLCCB2ELXXQ, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.136080402 (68h56m48.062375094s) [noise=true , address=77.90.31.15:4697 , piece_id=CUXGVL7PS3H7ZQKUCSYXJCBR3ETYFDPZFYIEAWEYL7C2S3RJY3AA, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.136382603 (68h56m48.062677289s) [noise=true , address=104.36.82.13:4186 , piece_id=5SVABN3KS6AUL6HILHNQCVF74BA75VPKTTAK7IQAPKRRME3MOZEA, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.136399304 (68h56m48.062693995s) [noise=true , address=104.36.82.14:4223 , piece_id=LP74YFI2SRSYPYEIKREVINCOFMVCXLNDS2555YKUVFN63N6TBJDA, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.136669576 (68h56m48.062964262s) [noise=true , address=104.36.82.6:4876 , piece_id=SSMGONONY34EHF5W6QMKO7YXB7RLO5LZVTJMMFKTP6GQNGYFYQ4A, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.136809784 (68h56m48.063104477s) [noise=true , address=104.36.82.15:4258 , piece_id=YXS4J7YSG6JV33JG4BZLK5AXKE3JXLBTGZJXPXZQ4LBYBT3MU2AA, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.136950556 (68h56m48.063245242s) [noise=true , address=77.90.31.11:4554 , piece_id=R2CIFLWS4LXTGSDSIMIRC4UYVJTXC7T35HPWZ2S7VYUTJ2AMY3BQ, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.137226497 (68h56m48.06352119s) [noise=true , address=77.90.31.14:4661 , piece_id=VXI2ZUSGR4N3H7TPIK75RIDFP3SBAOWVRGHNXTZOHFT5FNG4JWJA, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.138099107 (68h56m48.064393792s) [noise=true , address=45.91.116.10:4082 , piece_id=7NVEGA4QQOE7KYPHXUUX42SJPQL6LP46RGM37VD6A2AWN7PAOM3Q, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.139330448 (68h56m48.065625141s) [noise=true , address=104.36.82.14:4222 , piece_id=U3MBMAKFBFTOPJZ3HS3FXLPDBLQTN577XNCZLUBHRO6IU4BE5VNA, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.140580734 (68h56m48.06687543s) [noise=true , address=77.90.31.19:4842 , piece_id=G3AG375DATHB7MGEEWONE5GMMIC7QHG2VVDKF5AEVOSXV5N4S6AA, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.141452300 (68h56m48.067747027s) [noise=true , address=45.91.116.13:4154 , piece_id=STFPXN2LUJYUIAYROWZIRTCMT2Y3XET6KWA4GZWUHENXRJURP62Q, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.142917062 (68h56m48.069211739s) [noise=true , address=104.36.82.7:4912 , piece_id=A5HPC7CEU66IFNS47HO3AQX22LUZ6AZ4L4XAMZ6T5NCVE4D3TAZA, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.221824189 (68h56m48.148118865s) [upload=55354]: Next returned nil segment Jun 26 06:08:21.221837958 (68h56m48.148132644s) [upload=55354]: Waiting for error group managing segments... Jun 26 06:08:21.229550628 (68h56m48.155845315s) [noise=true , address=104.36.82.14:4223 , piece_id=LP74YFI2SRSYPYEIKREVINCOFMVCXLNDS2555YKUVFN63N6TBJDA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: <nil> Jun 26 06:08:21.230245145 (68h56m48.156539832s) [noise=true , address=104.36.82.14:4222 , piece_id=U3MBMAKFBFTOPJZ3HS3FXLPDBLQTN577XNCZLUBHRO6IU4BE5VNA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: <nil> Jun 26 06:08:21.230801067 (68h56m48.15709574s) [noise=true , address=104.36.82.6:4873 , piece_id=TIIMEQUSDLRFF3OK3WBB74F7SWJ5MEC4Z6IQNELDMRZLNTMQT7LA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: <nil> Jun 26 06:08:21.231457382 (68h56m48.157752071s) [noise=true , address=104.36.82.6:4876 , piece_id=SSMGONONY34EHF5W6QMKO7YXB7RLO5LZVTJMMFKTP6GQNGYFYQ4A, seg_pos={0 0}, upload=55354]: Done uploading piece. err: <nil> Jun 26 06:08:21.231563205 (68h56m48.157857893s) [noise=true , address=104.36.82.13:4186 , piece_id=5SVABN3KS6AUL6HILHNQCVF74BA75VPKTTAK7IQAPKRRME3MOZEA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: <nil> Jun 26 06:08:21.239131119 (68h56m48.165425799s) [noise=true , address=104.36.82.7:4912 , piece_id=A5HPC7CEU66IFNS47HO3AQX22LUZ6AZ4L4XAMZ6T5NCVE4D3TAZA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: <nil> Jun 26 06:08:21.534686665 (68h56m48.460981337s) [noise=true , address=104.36.82.15:4258 , piece_id=YXS4J7YSG6JV33JG4BZLK5AXKE3JXLBTGZJXPXZQ4LBYBT3MU2AA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: <nil> Jun 26 06:08:51.073673533 (68h57m17.999968219s) [noise=true , address=45.91.116.9:5261 , piece_id=CZWB3XJVA7B7R7GWFYNW3RCGNVRXQ3E37XDXD4SUXTEMQFZ73HHA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi, address:45.91.116.9:5261): protocol: rpc: dial tcp 45.91.116.9:5261: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:51.073739594 (68h57m18.000034272s) [noise=true , address=45.91.116.10:4082 , piece_id=7NVEGA4QQOE7KYPHXUUX42SJPQL6LP46RGM37VD6A2AWN7PAOM3Q, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj, address:45.91.116.10:4082): protocol: rpc: dial tcp 45.91.116.10:4082: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:51.073789897 (68h57m18.000084584s) [noise=true , address=77.90.31.19:4842 , piece_id=G3AG375DATHB7MGEEWONE5GMMIC7QHG2VVDKF5AEVOSXV5N4S6AA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: rpc: dial tcp 77.90.31.19:4842: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:51.073849570 (68h57m18.000144246s) [noise=true , address=45.91.116.13:4154 , piece_id=STFPXN2LUJYUIAYROWZIRTCMT2Y3XET6KWA4GZWUHENXRJURP62Q, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN, address:45.91.116.13:4154): protocol: write tcp 159.223.143.39:35392->45.91.116.13:4154: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:51.073904716 (68h57m18.000199391s) [noise=true , address=77.90.31.13:4626 , piece_id=MXYBIEDFAFK53CVTE2C6CKEZWOL3MLW4S7LOHK4IHW6QEIZJL76Q, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: write tcp 159.223.143.39:58864->77.90.31.13:4626: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:51.073952402 (68h57m18.000247082s) [noise=true , address=77.90.31.10:4517 , piece_id=VJZ5T4QNVAPNUHNFNK24S6TAS4V4NKNFMQE2FZMCWWLCCB2ELXXQ, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, address:77.90.31.10:4517): protocol: write tcp 159.223.143.39:36382->77.90.31.10:4517: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:51.074022481 (68h57m18.000317156s) [noise=true , address=77.90.31.15:4697 , piece_id=CUXGVL7PS3H7ZQKUCSYXJCBR3ETYFDPZFYIEAWEYL7C2S3RJY3AA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5, address:77.90.31.15:4697): protocol: write tcp 159.223.143.39:52730->77.90.31.15:4697: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:51.074071824 (68h57m18.000366503s) [noise=true , address=77.90.31.11:4554 , piece_id=R2CIFLWS4LXTGSDSIMIRC4UYVJTXC7T35HPWZ2S7VYUTJ2AMY3BQ, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, address:77.90.31.11:4554): protocol: rpc: dial tcp 77.90.31.11:4554: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:51.074102177 (68h57m18.000396855s) [noise=true , address=77.90.31.14:4661 , piece_id=VXI2ZUSGR4N3H7TPIK75RIDFP3SBAOWVRGHNXTZOHFT5FNG4JWJA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue, address:77.90.31.14:4661): protocol: rpc: dial tcp 77.90.31.14:4661: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:51.074119276 (68h57m18.000413966s) [seg_pos={0 0}, upload=55354]: Done waiting for segment. successful: 7 optimal: 13 errs: <nil> Jun 26 06:08:51.100079867 (68h57m18.026374541s) [upload=55354]: Done uploading segments... Jun 26 06:08:51.100114505 (68h57m18.026409191s) [upload=55354]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 7) Duration: 421.147264ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [...............XXX..........................................] duration: 450.470196ms FAIL (2025-06-26 06:41:30 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 7) ======LOGS====== pload=55354]: Got next segment. Inline: false Jun 26 06:08:21.105827735 (68h56m48.032122411s) [upload=55354]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 26 06:08:21.105833588 (68h56m48.032128274s) [upload=55354]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 26 06:08:21.135318313 (68h56m48.061613009s) [seg_pos={0 0}, upload=55354]: Begin upload segment... Jun 26 06:08:21.135366534 (68h56m48.061661214s) [seg_pos={0 0}, upload=55354]: Done begin upload segment. Jun 26 06:08:21.135390342 (68h56m48.061685028s) [noise=true , address=45.91.116.9:5261 , piece_id=CZWB3XJVA7B7R7GWFYNW3RCGNVRXQ3E37XDXD4SUXTEMQFZ73HHA, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.135405801 (68h56m48.061700497s) [noise=true , address=104.36.82.6:4873 , piece_id=TIIMEQUSDLRFF3OK3WBB74F7SWJ5MEC4Z6IQNELDMRZLNTMQT7LA, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.135774729 (68h56m48.062069418s) [noise=true , address=77.90.31.13:4626 , piece_id=MXYBIEDFAFK53CVTE2C6CKEZWOL3MLW4S7LOHK4IHW6QEIZJL76Q, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.135944436 (68h56m48.06223913s) [noise=true , address=77.90.31.10:4517 , piece_id=VJZ5T4QNVAPNUHNFNK24S6TAS4V4NKNFMQE2FZMCWWLCCB2ELXXQ, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.136080402 (68h56m48.062375094s) [noise=true , address=77.90.31.15:4697 , piece_id=CUXGVL7PS3H7ZQKUCSYXJCBR3ETYFDPZFYIEAWEYL7C2S3RJY3AA, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.136382603 (68h56m48.062677289s) [noise=true , address=104.36.82.13:4186 , piece_id=5SVABN3KS6AUL6HILHNQCVF74BA75VPKTTAK7IQAPKRRME3MOZEA, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.136399304 (68h56m48.062693995s) [noise=true , address=104.36.82.14:4223 , piece_id=LP74YFI2SRSYPYEIKREVINCOFMVCXLNDS2555YKUVFN63N6TBJDA, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.136669576 (68h56m48.062964262s) [noise=true , address=104.36.82.6:4876 , piece_id=SSMGONONY34EHF5W6QMKO7YXB7RLO5LZVTJMMFKTP6GQNGYFYQ4A, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.136809784 (68h56m48.063104477s) [noise=true , address=104.36.82.15:4258 , piece_id=YXS4J7YSG6JV33JG4BZLK5AXKE3JXLBTGZJXPXZQ4LBYBT3MU2AA, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.136950556 (68h56m48.063245242s) [noise=true , address=77.90.31.11:4554 , piece_id=R2CIFLWS4LXTGSDSIMIRC4UYVJTXC7T35HPWZ2S7VYUTJ2AMY3BQ, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.137226497 (68h56m48.06352119s) [noise=true , address=77.90.31.14:4661 , piece_id=VXI2ZUSGR4N3H7TPIK75RIDFP3SBAOWVRGHNXTZOHFT5FNG4JWJA, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.138099107 (68h56m48.064393792s) [noise=true , address=45.91.116.10:4082 , piece_id=7NVEGA4QQOE7KYPHXUUX42SJPQL6LP46RGM37VD6A2AWN7PAOM3Q, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.139330448 (68h56m48.065625141s) [noise=true , address=104.36.82.14:4222 , piece_id=U3MBMAKFBFTOPJZ3HS3FXLPDBLQTN577XNCZLUBHRO6IU4BE5VNA, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.140580734 (68h56m48.06687543s) [noise=true , address=77.90.31.19:4842 , piece_id=G3AG375DATHB7MGEEWONE5GMMIC7QHG2VVDKF5AEVOSXV5N4S6AA, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.141452300 (68h56m48.067747027s) [noise=true , address=45.91.116.13:4154 , piece_id=STFPXN2LUJYUIAYROWZIRTCMT2Y3XET6KWA4GZWUHENXRJURP62Q, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.142917062 (68h56m48.069211739s) [noise=true , address=104.36.82.7:4912 , piece_id=A5HPC7CEU66IFNS47HO3AQX22LUZ6AZ4L4XAMZ6T5NCVE4D3TAZA, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.221824189 (68h56m48.148118865s) [upload=55354]: Next returned nil segment Jun 26 06:08:21.221837958 (68h56m48.148132644s) [upload=55354]: Waiting for error group managing segments... Jun 26 06:08:21.229550628 (68h56m48.155845315s) [noise=true , address=104.36.82.14:4223 , piece_id=LP74YFI2SRSYPYEIKREVINCOFMVCXLNDS2555YKUVFN63N6TBJDA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: <nil> Jun 26 06:08:21.230245145 (68h56m48.156539832s) [noise=true , address=104.36.82.14:4222 , piece_id=U3MBMAKFBFTOPJZ3HS3FXLPDBLQTN577XNCZLUBHRO6IU4BE5VNA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: <nil> Jun 26 06:08:21.230801067 (68h56m48.15709574s) [noise=true , address=104.36.82.6:4873 , piece_id=TIIMEQUSDLRFF3OK3WBB74F7SWJ5MEC4Z6IQNELDMRZLNTMQT7LA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: <nil> Jun 26 06:08:21.231457382 (68h56m48.157752071s) [noise=true , address=104.36.82.6:4876 , piece_id=SSMGONONY34EHF5W6QMKO7YXB7RLO5LZVTJMMFKTP6GQNGYFYQ4A, seg_pos={0 0}, upload=55354]: Done uploading piece. err: <nil> Jun 26 06:08:21.231563205 (68h56m48.157857893s) [noise=true , address=104.36.82.13:4186 , piece_id=5SVABN3KS6AUL6HILHNQCVF74BA75VPKTTAK7IQAPKRRME3MOZEA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: <nil> Jun 26 06:08:21.239131119 (68h56m48.165425799s) [noise=true , address=104.36.82.7:4912 , piece_id=A5HPC7CEU66IFNS47HO3AQX22LUZ6AZ4L4XAMZ6T5NCVE4D3TAZA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: <nil> Jun 26 06:08:21.534686665 (68h56m48.460981337s) [noise=true , address=104.36.82.15:4258 , piece_id=YXS4J7YSG6JV33JG4BZLK5AXKE3JXLBTGZJXPXZQ4LBYBT3MU2AA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: <nil> Jun 26 06:08:51.073673533 (68h57m17.999968219s) [noise=true , address=45.91.116.9:5261 , piece_id=CZWB3XJVA7B7R7GWFYNW3RCGNVRXQ3E37XDXD4SUXTEMQFZ73HHA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi, address:45.91.116.9:5261): protocol: rpc: dial tcp 45.91.116.9:5261: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:51.073739594 (68h57m18.000034272s) [noise=true , address=45.91.116.10:4082 , piece_id=7NVEGA4QQOE7KYPHXUUX42SJPQL6LP46RGM37VD6A2AWN7PAOM3Q, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj, address:45.91.116.10:4082): protocol: rpc: dial tcp 45.91.116.10:4082: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:51.073789897 (68h57m18.000084584s) [noise=true , address=77.90.31.19:4842 , piece_id=G3AG375DATHB7MGEEWONE5GMMIC7QHG2VVDKF5AEVOSXV5N4S6AA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: rpc: dial tcp 77.90.31.19:4842: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:51.073849570 (68h57m18.000144246s) [noise=true , address=45.91.116.13:4154 , piece_id=STFPXN2LUJYUIAYROWZIRTCMT2Y3XET6KWA4GZWUHENXRJURP62Q, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN, address:45.91.116.13:4154): protocol: write tcp 159.223.143.39:35392->45.91.116.13:4154: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:51.073904716 (68h57m18.000199391s) [noise=true , address=77.90.31.13:4626 , piece_id=MXYBIEDFAFK53CVTE2C6CKEZWOL3MLW4S7LOHK4IHW6QEIZJL76Q, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: write tcp 159.223.143.39:58864->77.90.31.13:4626: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:51.073952402 (68h57m18.000247082s) [noise=true , address=77.90.31.10:4517 , piece_id=VJZ5T4QNVAPNUHNFNK24S6TAS4V4NKNFMQE2FZMCWWLCCB2ELXXQ, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, address:77.90.31.10:4517): protocol: write tcp 159.223.143.39:36382->77.90.31.10:4517: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:51.074022481 (68h57m18.000317156s) [noise=true , address=77.90.31.15:4697 , piece_id=CUXGVL7PS3H7ZQKUCSYXJCBR3ETYFDPZFYIEAWEYL7C2S3RJY3AA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5, address:77.90.31.15:4697): protocol: write tcp 159.223.143.39:52730->77.90.31.15:4697: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:51.074071824 (68h57m18.000366503s) [noise=true , address=77.90.31.11:4554 , piece_id=R2CIFLWS4LXTGSDSIMIRC4UYVJTXC7T35HPWZ2S7VYUTJ2AMY3BQ, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, address:77.90.31.11:4554): protocol: rpc: dial tcp 77.90.31.11:4554: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:51.074102177 (68h57m18.000396855s) [noise=true , address=77.90.31.14:4661 , piece_id=VXI2ZUSGR4N3H7TPIK75RIDFP3SBAOWVRGHNXTZOHFT5FNG4JWJA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue, address:77.90.31.14:4661): protocol: rpc: dial tcp 77.90.31.14:4661: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:51.074119276 (68h57m18.000413966s) [seg_pos={0 0}, upload=55354]: Done waiting for segment. successful: 7 optimal: 13 errs: <nil> Jun 26 06:08:51.100079867 (68h57m18.026374541s) [upload=55354]: Done uploading segments... Jun 26 06:08:51.100114505 (68h57m18.026409191s) [upload=55354]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 7) Duration: 450.500752ms core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..............XX............................................] duration: 496.190701ms FAIL (2025-06-26 06:41:27 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 496.305232ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [................XXX.........................................] duration: 541.437037ms FAIL (2025-06-26 06:40:54 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 7) ======LOGS====== pload=55354]: Got next segment. Inline: false Jun 26 06:08:21.105827735 (68h56m48.032122411s) [upload=55354]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 26 06:08:21.105833588 (68h56m48.032128274s) [upload=55354]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 26 06:08:21.135318313 (68h56m48.061613009s) [seg_pos={0 0}, upload=55354]: Begin upload segment... Jun 26 06:08:21.135366534 (68h56m48.061661214s) [seg_pos={0 0}, upload=55354]: Done begin upload segment. Jun 26 06:08:21.135390342 (68h56m48.061685028s) [noise=true , address=45.91.116.9:5261 , piece_id=CZWB3XJVA7B7R7GWFYNW3RCGNVRXQ3E37XDXD4SUXTEMQFZ73HHA, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.135405801 (68h56m48.061700497s) [noise=true , address=104.36.82.6:4873 , piece_id=TIIMEQUSDLRFF3OK3WBB74F7SWJ5MEC4Z6IQNELDMRZLNTMQT7LA, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.135774729 (68h56m48.062069418s) [noise=true , address=77.90.31.13:4626 , piece_id=MXYBIEDFAFK53CVTE2C6CKEZWOL3MLW4S7LOHK4IHW6QEIZJL76Q, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.135944436 (68h56m48.06223913s) [noise=true , address=77.90.31.10:4517 , piece_id=VJZ5T4QNVAPNUHNFNK24S6TAS4V4NKNFMQE2FZMCWWLCCB2ELXXQ, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.136080402 (68h56m48.062375094s) [noise=true , address=77.90.31.15:4697 , piece_id=CUXGVL7PS3H7ZQKUCSYXJCBR3ETYFDPZFYIEAWEYL7C2S3RJY3AA, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.136382603 (68h56m48.062677289s) [noise=true , address=104.36.82.13:4186 , piece_id=5SVABN3KS6AUL6HILHNQCVF74BA75VPKTTAK7IQAPKRRME3MOZEA, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.136399304 (68h56m48.062693995s) [noise=true , address=104.36.82.14:4223 , piece_id=LP74YFI2SRSYPYEIKREVINCOFMVCXLNDS2555YKUVFN63N6TBJDA, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.136669576 (68h56m48.062964262s) [noise=true , address=104.36.82.6:4876 , piece_id=SSMGONONY34EHF5W6QMKO7YXB7RLO5LZVTJMMFKTP6GQNGYFYQ4A, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.136809784 (68h56m48.063104477s) [noise=true , address=104.36.82.15:4258 , piece_id=YXS4J7YSG6JV33JG4BZLK5AXKE3JXLBTGZJXPXZQ4LBYBT3MU2AA, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.136950556 (68h56m48.063245242s) [noise=true , address=77.90.31.11:4554 , piece_id=R2CIFLWS4LXTGSDSIMIRC4UYVJTXC7T35HPWZ2S7VYUTJ2AMY3BQ, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.137226497 (68h56m48.06352119s) [noise=true , address=77.90.31.14:4661 , piece_id=VXI2ZUSGR4N3H7TPIK75RIDFP3SBAOWVRGHNXTZOHFT5FNG4JWJA, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.138099107 (68h56m48.064393792s) [noise=true , address=45.91.116.10:4082 , piece_id=7NVEGA4QQOE7KYPHXUUX42SJPQL6LP46RGM37VD6A2AWN7PAOM3Q, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.139330448 (68h56m48.065625141s) [noise=true , address=104.36.82.14:4222 , piece_id=U3MBMAKFBFTOPJZ3HS3FXLPDBLQTN577XNCZLUBHRO6IU4BE5VNA, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.140580734 (68h56m48.06687543s) [noise=true , address=77.90.31.19:4842 , piece_id=G3AG375DATHB7MGEEWONE5GMMIC7QHG2VVDKF5AEVOSXV5N4S6AA, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.141452300 (68h56m48.067747027s) [noise=true , address=45.91.116.13:4154 , piece_id=STFPXN2LUJYUIAYROWZIRTCMT2Y3XET6KWA4GZWUHENXRJURP62Q, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.142917062 (68h56m48.069211739s) [noise=true , address=104.36.82.7:4912 , piece_id=A5HPC7CEU66IFNS47HO3AQX22LUZ6AZ4L4XAMZ6T5NCVE4D3TAZA, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.221824189 (68h56m48.148118865s) [upload=55354]: Next returned nil segment Jun 26 06:08:21.221837958 (68h56m48.148132644s) [upload=55354]: Waiting for error group managing segments... Jun 26 06:08:21.229550628 (68h56m48.155845315s) [noise=true , address=104.36.82.14:4223 , piece_id=LP74YFI2SRSYPYEIKREVINCOFMVCXLNDS2555YKUVFN63N6TBJDA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: <nil> Jun 26 06:08:21.230245145 (68h56m48.156539832s) [noise=true , address=104.36.82.14:4222 , piece_id=U3MBMAKFBFTOPJZ3HS3FXLPDBLQTN577XNCZLUBHRO6IU4BE5VNA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: <nil> Jun 26 06:08:21.230801067 (68h56m48.15709574s) [noise=true , address=104.36.82.6:4873 , piece_id=TIIMEQUSDLRFF3OK3WBB74F7SWJ5MEC4Z6IQNELDMRZLNTMQT7LA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: <nil> Jun 26 06:08:21.231457382 (68h56m48.157752071s) [noise=true , address=104.36.82.6:4876 , piece_id=SSMGONONY34EHF5W6QMKO7YXB7RLO5LZVTJMMFKTP6GQNGYFYQ4A, seg_pos={0 0}, upload=55354]: Done uploading piece. err: <nil> Jun 26 06:08:21.231563205 (68h56m48.157857893s) [noise=true , address=104.36.82.13:4186 , piece_id=5SVABN3KS6AUL6HILHNQCVF74BA75VPKTTAK7IQAPKRRME3MOZEA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: <nil> Jun 26 06:08:21.239131119 (68h56m48.165425799s) [noise=true , address=104.36.82.7:4912 , piece_id=A5HPC7CEU66IFNS47HO3AQX22LUZ6AZ4L4XAMZ6T5NCVE4D3TAZA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: <nil> Jun 26 06:08:21.534686665 (68h56m48.460981337s) [noise=true , address=104.36.82.15:4258 , piece_id=YXS4J7YSG6JV33JG4BZLK5AXKE3JXLBTGZJXPXZQ4LBYBT3MU2AA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: <nil> Jun 26 06:08:51.073673533 (68h57m17.999968219s) [noise=true , address=45.91.116.9:5261 , piece_id=CZWB3XJVA7B7R7GWFYNW3RCGNVRXQ3E37XDXD4SUXTEMQFZ73HHA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi, address:45.91.116.9:5261): protocol: rpc: dial tcp 45.91.116.9:5261: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:51.073739594 (68h57m18.000034272s) [noise=true , address=45.91.116.10:4082 , piece_id=7NVEGA4QQOE7KYPHXUUX42SJPQL6LP46RGM37VD6A2AWN7PAOM3Q, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj, address:45.91.116.10:4082): protocol: rpc: dial tcp 45.91.116.10:4082: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:51.073789897 (68h57m18.000084584s) [noise=true , address=77.90.31.19:4842 , piece_id=G3AG375DATHB7MGEEWONE5GMMIC7QHG2VVDKF5AEVOSXV5N4S6AA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: rpc: dial tcp 77.90.31.19:4842: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:51.073849570 (68h57m18.000144246s) [noise=true , address=45.91.116.13:4154 , piece_id=STFPXN2LUJYUIAYROWZIRTCMT2Y3XET6KWA4GZWUHENXRJURP62Q, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN, address:45.91.116.13:4154): protocol: write tcp 159.223.143.39:35392->45.91.116.13:4154: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:51.073904716 (68h57m18.000199391s) [noise=true , address=77.90.31.13:4626 , piece_id=MXYBIEDFAFK53CVTE2C6CKEZWOL3MLW4S7LOHK4IHW6QEIZJL76Q, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: write tcp 159.223.143.39:58864->77.90.31.13:4626: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:51.073952402 (68h57m18.000247082s) [noise=true , address=77.90.31.10:4517 , piece_id=VJZ5T4QNVAPNUHNFNK24S6TAS4V4NKNFMQE2FZMCWWLCCB2ELXXQ, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, address:77.90.31.10:4517): protocol: write tcp 159.223.143.39:36382->77.90.31.10:4517: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:51.074022481 (68h57m18.000317156s) [noise=true , address=77.90.31.15:4697 , piece_id=CUXGVL7PS3H7ZQKUCSYXJCBR3ETYFDPZFYIEAWEYL7C2S3RJY3AA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5, address:77.90.31.15:4697): protocol: write tcp 159.223.143.39:52730->77.90.31.15:4697: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:51.074071824 (68h57m18.000366503s) [noise=true , address=77.90.31.11:4554 , piece_id=R2CIFLWS4LXTGSDSIMIRC4UYVJTXC7T35HPWZ2S7VYUTJ2AMY3BQ, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, address:77.90.31.11:4554): protocol: rpc: dial tcp 77.90.31.11:4554: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:51.074102177 (68h57m18.000396855s) [noise=true , address=77.90.31.14:4661 , piece_id=VXI2ZUSGR4N3H7TPIK75RIDFP3SBAOWVRGHNXTZOHFT5FNG4JWJA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue, address:77.90.31.14:4661): protocol: rpc: dial tcp 77.90.31.14:4661: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:51.074119276 (68h57m18.000413966s) [seg_pos={0 0}, upload=55354]: Done waiting for segment. successful: 7 optimal: 13 errs: <nil> Jun 26 06:08:51.100079867 (68h57m18.026374541s) [upload=55354]: Done uploading segments... Jun 26 06:08:51.100114505 (68h57m18.026409191s) [upload=55354]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 7) Duration: 541.46628ms core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...............XX...........................................] duration: 523.6065ms FAIL (2025-06-26 06:40:40 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 523.643946ms edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [................XX..........................................] duration: 705.174687ms FAIL (2025-06-26 06:40:04 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 705.414691ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.................XXX........................................] duration: 438.544068ms FAIL (2025-06-26 06:39:46 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 7) ======LOGS====== pload=55354]: Got next segment. Inline: false Jun 26 06:08:21.105827735 (68h56m48.032122411s) [upload=55354]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 26 06:08:21.105833588 (68h56m48.032128274s) [upload=55354]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 26 06:08:21.135318313 (68h56m48.061613009s) [seg_pos={0 0}, upload=55354]: Begin upload segment... Jun 26 06:08:21.135366534 (68h56m48.061661214s) [seg_pos={0 0}, upload=55354]: Done begin upload segment. Jun 26 06:08:21.135390342 (68h56m48.061685028s) [noise=true , address=45.91.116.9:5261 , piece_id=CZWB3XJVA7B7R7GWFYNW3RCGNVRXQ3E37XDXD4SUXTEMQFZ73HHA, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.135405801 (68h56m48.061700497s) [noise=true , address=104.36.82.6:4873 , piece_id=TIIMEQUSDLRFF3OK3WBB74F7SWJ5MEC4Z6IQNELDMRZLNTMQT7LA, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.135774729 (68h56m48.062069418s) [noise=true , address=77.90.31.13:4626 , piece_id=MXYBIEDFAFK53CVTE2C6CKEZWOL3MLW4S7LOHK4IHW6QEIZJL76Q, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.135944436 (68h56m48.06223913s) [noise=true , address=77.90.31.10:4517 , piece_id=VJZ5T4QNVAPNUHNFNK24S6TAS4V4NKNFMQE2FZMCWWLCCB2ELXXQ, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.136080402 (68h56m48.062375094s) [noise=true , address=77.90.31.15:4697 , piece_id=CUXGVL7PS3H7ZQKUCSYXJCBR3ETYFDPZFYIEAWEYL7C2S3RJY3AA, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.136382603 (68h56m48.062677289s) [noise=true , address=104.36.82.13:4186 , piece_id=5SVABN3KS6AUL6HILHNQCVF74BA75VPKTTAK7IQAPKRRME3MOZEA, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.136399304 (68h56m48.062693995s) [noise=true , address=104.36.82.14:4223 , piece_id=LP74YFI2SRSYPYEIKREVINCOFMVCXLNDS2555YKUVFN63N6TBJDA, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.136669576 (68h56m48.062964262s) [noise=true , address=104.36.82.6:4876 , piece_id=SSMGONONY34EHF5W6QMKO7YXB7RLO5LZVTJMMFKTP6GQNGYFYQ4A, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.136809784 (68h56m48.063104477s) [noise=true , address=104.36.82.15:4258 , piece_id=YXS4J7YSG6JV33JG4BZLK5AXKE3JXLBTGZJXPXZQ4LBYBT3MU2AA, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.136950556 (68h56m48.063245242s) [noise=true , address=77.90.31.11:4554 , piece_id=R2CIFLWS4LXTGSDSIMIRC4UYVJTXC7T35HPWZ2S7VYUTJ2AMY3BQ, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.137226497 (68h56m48.06352119s) [noise=true , address=77.90.31.14:4661 , piece_id=VXI2ZUSGR4N3H7TPIK75RIDFP3SBAOWVRGHNXTZOHFT5FNG4JWJA, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.138099107 (68h56m48.064393792s) [noise=true , address=45.91.116.10:4082 , piece_id=7NVEGA4QQOE7KYPHXUUX42SJPQL6LP46RGM37VD6A2AWN7PAOM3Q, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.139330448 (68h56m48.065625141s) [noise=true , address=104.36.82.14:4222 , piece_id=U3MBMAKFBFTOPJZ3HS3FXLPDBLQTN577XNCZLUBHRO6IU4BE5VNA, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.140580734 (68h56m48.06687543s) [noise=true , address=77.90.31.19:4842 , piece_id=G3AG375DATHB7MGEEWONE5GMMIC7QHG2VVDKF5AEVOSXV5N4S6AA, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.141452300 (68h56m48.067747027s) [noise=true , address=45.91.116.13:4154 , piece_id=STFPXN2LUJYUIAYROWZIRTCMT2Y3XET6KWA4GZWUHENXRJURP62Q, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.142917062 (68h56m48.069211739s) [noise=true , address=104.36.82.7:4912 , piece_id=A5HPC7CEU66IFNS47HO3AQX22LUZ6AZ4L4XAMZ6T5NCVE4D3TAZA, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.221824189 (68h56m48.148118865s) [upload=55354]: Next returned nil segment Jun 26 06:08:21.221837958 (68h56m48.148132644s) [upload=55354]: Waiting for error group managing segments... Jun 26 06:08:21.229550628 (68h56m48.155845315s) [noise=true , address=104.36.82.14:4223 , piece_id=LP74YFI2SRSYPYEIKREVINCOFMVCXLNDS2555YKUVFN63N6TBJDA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: <nil> Jun 26 06:08:21.230245145 (68h56m48.156539832s) [noise=true , address=104.36.82.14:4222 , piece_id=U3MBMAKFBFTOPJZ3HS3FXLPDBLQTN577XNCZLUBHRO6IU4BE5VNA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: <nil> Jun 26 06:08:21.230801067 (68h56m48.15709574s) [noise=true , address=104.36.82.6:4873 , piece_id=TIIMEQUSDLRFF3OK3WBB74F7SWJ5MEC4Z6IQNELDMRZLNTMQT7LA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: <nil> Jun 26 06:08:21.231457382 (68h56m48.157752071s) [noise=true , address=104.36.82.6:4876 , piece_id=SSMGONONY34EHF5W6QMKO7YXB7RLO5LZVTJMMFKTP6GQNGYFYQ4A, seg_pos={0 0}, upload=55354]: Done uploading piece. err: <nil> Jun 26 06:08:21.231563205 (68h56m48.157857893s) [noise=true , address=104.36.82.13:4186 , piece_id=5SVABN3KS6AUL6HILHNQCVF74BA75VPKTTAK7IQAPKRRME3MOZEA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: <nil> Jun 26 06:08:21.239131119 (68h56m48.165425799s) [noise=true , address=104.36.82.7:4912 , piece_id=A5HPC7CEU66IFNS47HO3AQX22LUZ6AZ4L4XAMZ6T5NCVE4D3TAZA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: <nil> Jun 26 06:08:21.534686665 (68h56m48.460981337s) [noise=true , address=104.36.82.15:4258 , piece_id=YXS4J7YSG6JV33JG4BZLK5AXKE3JXLBTGZJXPXZQ4LBYBT3MU2AA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: <nil> Jun 26 06:08:51.073673533 (68h57m17.999968219s) [noise=true , address=45.91.116.9:5261 , piece_id=CZWB3XJVA7B7R7GWFYNW3RCGNVRXQ3E37XDXD4SUXTEMQFZ73HHA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi, address:45.91.116.9:5261): protocol: rpc: dial tcp 45.91.116.9:5261: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:51.073739594 (68h57m18.000034272s) [noise=true , address=45.91.116.10:4082 , piece_id=7NVEGA4QQOE7KYPHXUUX42SJPQL6LP46RGM37VD6A2AWN7PAOM3Q, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj, address:45.91.116.10:4082): protocol: rpc: dial tcp 45.91.116.10:4082: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:51.073789897 (68h57m18.000084584s) [noise=true , address=77.90.31.19:4842 , piece_id=G3AG375DATHB7MGEEWONE5GMMIC7QHG2VVDKF5AEVOSXV5N4S6AA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: rpc: dial tcp 77.90.31.19:4842: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:51.073849570 (68h57m18.000144246s) [noise=true , address=45.91.116.13:4154 , piece_id=STFPXN2LUJYUIAYROWZIRTCMT2Y3XET6KWA4GZWUHENXRJURP62Q, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN, address:45.91.116.13:4154): protocol: write tcp 159.223.143.39:35392->45.91.116.13:4154: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:51.073904716 (68h57m18.000199391s) [noise=true , address=77.90.31.13:4626 , piece_id=MXYBIEDFAFK53CVTE2C6CKEZWOL3MLW4S7LOHK4IHW6QEIZJL76Q, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: write tcp 159.223.143.39:58864->77.90.31.13:4626: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:51.073952402 (68h57m18.000247082s) [noise=true , address=77.90.31.10:4517 , piece_id=VJZ5T4QNVAPNUHNFNK24S6TAS4V4NKNFMQE2FZMCWWLCCB2ELXXQ, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, address:77.90.31.10:4517): protocol: write tcp 159.223.143.39:36382->77.90.31.10:4517: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:51.074022481 (68h57m18.000317156s) [noise=true , address=77.90.31.15:4697 , piece_id=CUXGVL7PS3H7ZQKUCSYXJCBR3ETYFDPZFYIEAWEYL7C2S3RJY3AA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5, address:77.90.31.15:4697): protocol: write tcp 159.223.143.39:52730->77.90.31.15:4697: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:51.074071824 (68h57m18.000366503s) [noise=true , address=77.90.31.11:4554 , piece_id=R2CIFLWS4LXTGSDSIMIRC4UYVJTXC7T35HPWZ2S7VYUTJ2AMY3BQ, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, address:77.90.31.11:4554): protocol: rpc: dial tcp 77.90.31.11:4554: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:51.074102177 (68h57m18.000396855s) [noise=true , address=77.90.31.14:4661 , piece_id=VXI2ZUSGR4N3H7TPIK75RIDFP3SBAOWVRGHNXTZOHFT5FNG4JWJA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue, address:77.90.31.14:4661): protocol: rpc: dial tcp 77.90.31.14:4661: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:51.074119276 (68h57m18.000413966s) [seg_pos={0 0}, upload=55354]: Done waiting for segment. successful: 7 optimal: 13 errs: <nil> Jun 26 06:08:51.100079867 (68h57m18.026374541s) [upload=55354]: Done uploading segments... Jun 26 06:08:51.100114505 (68h57m18.026409191s) [upload=55354]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 7) Duration: 438.577183ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [..................XXX.......................................] duration: 440.253709ms FAIL (2025-06-26 06:39:31 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 7) ======LOGS====== pload=55354]: Got next segment. Inline: false Jun 26 06:08:21.105827735 (68h56m48.032122411s) [upload=55354]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 26 06:08:21.105833588 (68h56m48.032128274s) [upload=55354]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 26 06:08:21.135318313 (68h56m48.061613009s) [seg_pos={0 0}, upload=55354]: Begin upload segment... Jun 26 06:08:21.135366534 (68h56m48.061661214s) [seg_pos={0 0}, upload=55354]: Done begin upload segment. Jun 26 06:08:21.135390342 (68h56m48.061685028s) [noise=true , address=45.91.116.9:5261 , piece_id=CZWB3XJVA7B7R7GWFYNW3RCGNVRXQ3E37XDXD4SUXTEMQFZ73HHA, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.135405801 (68h56m48.061700497s) [noise=true , address=104.36.82.6:4873 , piece_id=TIIMEQUSDLRFF3OK3WBB74F7SWJ5MEC4Z6IQNELDMRZLNTMQT7LA, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.135774729 (68h56m48.062069418s) [noise=true , address=77.90.31.13:4626 , piece_id=MXYBIEDFAFK53CVTE2C6CKEZWOL3MLW4S7LOHK4IHW6QEIZJL76Q, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.135944436 (68h56m48.06223913s) [noise=true , address=77.90.31.10:4517 , piece_id=VJZ5T4QNVAPNUHNFNK24S6TAS4V4NKNFMQE2FZMCWWLCCB2ELXXQ, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.136080402 (68h56m48.062375094s) [noise=true , address=77.90.31.15:4697 , piece_id=CUXGVL7PS3H7ZQKUCSYXJCBR3ETYFDPZFYIEAWEYL7C2S3RJY3AA, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.136382603 (68h56m48.062677289s) [noise=true , address=104.36.82.13:4186 , piece_id=5SVABN3KS6AUL6HILHNQCVF74BA75VPKTTAK7IQAPKRRME3MOZEA, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.136399304 (68h56m48.062693995s) [noise=true , address=104.36.82.14:4223 , piece_id=LP74YFI2SRSYPYEIKREVINCOFMVCXLNDS2555YKUVFN63N6TBJDA, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.136669576 (68h56m48.062964262s) [noise=true , address=104.36.82.6:4876 , piece_id=SSMGONONY34EHF5W6QMKO7YXB7RLO5LZVTJMMFKTP6GQNGYFYQ4A, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.136809784 (68h56m48.063104477s) [noise=true , address=104.36.82.15:4258 , piece_id=YXS4J7YSG6JV33JG4BZLK5AXKE3JXLBTGZJXPXZQ4LBYBT3MU2AA, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.136950556 (68h56m48.063245242s) [noise=true , address=77.90.31.11:4554 , piece_id=R2CIFLWS4LXTGSDSIMIRC4UYVJTXC7T35HPWZ2S7VYUTJ2AMY3BQ, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.137226497 (68h56m48.06352119s) [noise=true , address=77.90.31.14:4661 , piece_id=VXI2ZUSGR4N3H7TPIK75RIDFP3SBAOWVRGHNXTZOHFT5FNG4JWJA, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.138099107 (68h56m48.064393792s) [noise=true , address=45.91.116.10:4082 , piece_id=7NVEGA4QQOE7KYPHXUUX42SJPQL6LP46RGM37VD6A2AWN7PAOM3Q, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.139330448 (68h56m48.065625141s) [noise=true , address=104.36.82.14:4222 , piece_id=U3MBMAKFBFTOPJZ3HS3FXLPDBLQTN577XNCZLUBHRO6IU4BE5VNA, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.140580734 (68h56m48.06687543s) [noise=true , address=77.90.31.19:4842 , piece_id=G3AG375DATHB7MGEEWONE5GMMIC7QHG2VVDKF5AEVOSXV5N4S6AA, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.141452300 (68h56m48.067747027s) [noise=true , address=45.91.116.13:4154 , piece_id=STFPXN2LUJYUIAYROWZIRTCMT2Y3XET6KWA4GZWUHENXRJURP62Q, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.142917062 (68h56m48.069211739s) [noise=true , address=104.36.82.7:4912 , piece_id=A5HPC7CEU66IFNS47HO3AQX22LUZ6AZ4L4XAMZ6T5NCVE4D3TAZA, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.221824189 (68h56m48.148118865s) [upload=55354]: Next returned nil segment Jun 26 06:08:21.221837958 (68h56m48.148132644s) [upload=55354]: Waiting for error group managing segments... Jun 26 06:08:21.229550628 (68h56m48.155845315s) [noise=true , address=104.36.82.14:4223 , piece_id=LP74YFI2SRSYPYEIKREVINCOFMVCXLNDS2555YKUVFN63N6TBJDA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: <nil> Jun 26 06:08:21.230245145 (68h56m48.156539832s) [noise=true , address=104.36.82.14:4222 , piece_id=U3MBMAKFBFTOPJZ3HS3FXLPDBLQTN577XNCZLUBHRO6IU4BE5VNA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: <nil> Jun 26 06:08:21.230801067 (68h56m48.15709574s) [noise=true , address=104.36.82.6:4873 , piece_id=TIIMEQUSDLRFF3OK3WBB74F7SWJ5MEC4Z6IQNELDMRZLNTMQT7LA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: <nil> Jun 26 06:08:21.231457382 (68h56m48.157752071s) [noise=true , address=104.36.82.6:4876 , piece_id=SSMGONONY34EHF5W6QMKO7YXB7RLO5LZVTJMMFKTP6GQNGYFYQ4A, seg_pos={0 0}, upload=55354]: Done uploading piece. err: <nil> Jun 26 06:08:21.231563205 (68h56m48.157857893s) [noise=true , address=104.36.82.13:4186 , piece_id=5SVABN3KS6AUL6HILHNQCVF74BA75VPKTTAK7IQAPKRRME3MOZEA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: <nil> Jun 26 06:08:21.239131119 (68h56m48.165425799s) [noise=true , address=104.36.82.7:4912 , piece_id=A5HPC7CEU66IFNS47HO3AQX22LUZ6AZ4L4XAMZ6T5NCVE4D3TAZA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: <nil> Jun 26 06:08:21.534686665 (68h56m48.460981337s) [noise=true , address=104.36.82.15:4258 , piece_id=YXS4J7YSG6JV33JG4BZLK5AXKE3JXLBTGZJXPXZQ4LBYBT3MU2AA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: <nil> Jun 26 06:08:51.073673533 (68h57m17.999968219s) [noise=true , address=45.91.116.9:5261 , piece_id=CZWB3XJVA7B7R7GWFYNW3RCGNVRXQ3E37XDXD4SUXTEMQFZ73HHA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi, address:45.91.116.9:5261): protocol: rpc: dial tcp 45.91.116.9:5261: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:51.073739594 (68h57m18.000034272s) [noise=true , address=45.91.116.10:4082 , piece_id=7NVEGA4QQOE7KYPHXUUX42SJPQL6LP46RGM37VD6A2AWN7PAOM3Q, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj, address:45.91.116.10:4082): protocol: rpc: dial tcp 45.91.116.10:4082: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:51.073789897 (68h57m18.000084584s) [noise=true , address=77.90.31.19:4842 , piece_id=G3AG375DATHB7MGEEWONE5GMMIC7QHG2VVDKF5AEVOSXV5N4S6AA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: rpc: dial tcp 77.90.31.19:4842: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:51.073849570 (68h57m18.000144246s) [noise=true , address=45.91.116.13:4154 , piece_id=STFPXN2LUJYUIAYROWZIRTCMT2Y3XET6KWA4GZWUHENXRJURP62Q, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN, address:45.91.116.13:4154): protocol: write tcp 159.223.143.39:35392->45.91.116.13:4154: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:51.073904716 (68h57m18.000199391s) [noise=true , address=77.90.31.13:4626 , piece_id=MXYBIEDFAFK53CVTE2C6CKEZWOL3MLW4S7LOHK4IHW6QEIZJL76Q, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: write tcp 159.223.143.39:58864->77.90.31.13:4626: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:51.073952402 (68h57m18.000247082s) [noise=true , address=77.90.31.10:4517 , piece_id=VJZ5T4QNVAPNUHNFNK24S6TAS4V4NKNFMQE2FZMCWWLCCB2ELXXQ, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, address:77.90.31.10:4517): protocol: write tcp 159.223.143.39:36382->77.90.31.10:4517: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:51.074022481 (68h57m18.000317156s) [noise=true , address=77.90.31.15:4697 , piece_id=CUXGVL7PS3H7ZQKUCSYXJCBR3ETYFDPZFYIEAWEYL7C2S3RJY3AA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5, address:77.90.31.15:4697): protocol: write tcp 159.223.143.39:52730->77.90.31.15:4697: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:51.074071824 (68h57m18.000366503s) [noise=true , address=77.90.31.11:4554 , piece_id=R2CIFLWS4LXTGSDSIMIRC4UYVJTXC7T35HPWZ2S7VYUTJ2AMY3BQ, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, address:77.90.31.11:4554): protocol: rpc: dial tcp 77.90.31.11:4554: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:51.074102177 (68h57m18.000396855s) [noise=true , address=77.90.31.14:4661 , piece_id=VXI2ZUSGR4N3H7TPIK75RIDFP3SBAOWVRGHNXTZOHFT5FNG4JWJA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue, address:77.90.31.14:4661): protocol: rpc: dial tcp 77.90.31.14:4661: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:51.074119276 (68h57m18.000413966s) [seg_pos={0 0}, upload=55354]: Done waiting for segment. successful: 7 optimal: 13 errs: <nil> Jun 26 06:08:51.100079867 (68h57m18.026374541s) [upload=55354]: Done uploading segments... Jun 26 06:08:51.100114505 (68h57m18.026409191s) [upload=55354]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 7) Duration: 440.27681ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [...................XXX......................................] duration: 591.912319ms FAIL (2025-06-26 06:38:54 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 7) ======LOGS====== pload=55354]: Got next segment. Inline: false Jun 26 06:08:21.105827735 (68h56m48.032122411s) [upload=55354]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 26 06:08:21.105833588 (68h56m48.032128274s) [upload=55354]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 26 06:08:21.135318313 (68h56m48.061613009s) [seg_pos={0 0}, upload=55354]: Begin upload segment... Jun 26 06:08:21.135366534 (68h56m48.061661214s) [seg_pos={0 0}, upload=55354]: Done begin upload segment. Jun 26 06:08:21.135390342 (68h56m48.061685028s) [noise=true , address=45.91.116.9:5261 , piece_id=CZWB3XJVA7B7R7GWFYNW3RCGNVRXQ3E37XDXD4SUXTEMQFZ73HHA, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.135405801 (68h56m48.061700497s) [noise=true , address=104.36.82.6:4873 , piece_id=TIIMEQUSDLRFF3OK3WBB74F7SWJ5MEC4Z6IQNELDMRZLNTMQT7LA, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.135774729 (68h56m48.062069418s) [noise=true , address=77.90.31.13:4626 , piece_id=MXYBIEDFAFK53CVTE2C6CKEZWOL3MLW4S7LOHK4IHW6QEIZJL76Q, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.135944436 (68h56m48.06223913s) [noise=true , address=77.90.31.10:4517 , piece_id=VJZ5T4QNVAPNUHNFNK24S6TAS4V4NKNFMQE2FZMCWWLCCB2ELXXQ, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.136080402 (68h56m48.062375094s) [noise=true , address=77.90.31.15:4697 , piece_id=CUXGVL7PS3H7ZQKUCSYXJCBR3ETYFDPZFYIEAWEYL7C2S3RJY3AA, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.136382603 (68h56m48.062677289s) [noise=true , address=104.36.82.13:4186 , piece_id=5SVABN3KS6AUL6HILHNQCVF74BA75VPKTTAK7IQAPKRRME3MOZEA, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.136399304 (68h56m48.062693995s) [noise=true , address=104.36.82.14:4223 , piece_id=LP74YFI2SRSYPYEIKREVINCOFMVCXLNDS2555YKUVFN63N6TBJDA, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.136669576 (68h56m48.062964262s) [noise=true , address=104.36.82.6:4876 , piece_id=SSMGONONY34EHF5W6QMKO7YXB7RLO5LZVTJMMFKTP6GQNGYFYQ4A, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.136809784 (68h56m48.063104477s) [noise=true , address=104.36.82.15:4258 , piece_id=YXS4J7YSG6JV33JG4BZLK5AXKE3JXLBTGZJXPXZQ4LBYBT3MU2AA, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.136950556 (68h56m48.063245242s) [noise=true , address=77.90.31.11:4554 , piece_id=R2CIFLWS4LXTGSDSIMIRC4UYVJTXC7T35HPWZ2S7VYUTJ2AMY3BQ, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.137226497 (68h56m48.06352119s) [noise=true , address=77.90.31.14:4661 , piece_id=VXI2ZUSGR4N3H7TPIK75RIDFP3SBAOWVRGHNXTZOHFT5FNG4JWJA, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.138099107 (68h56m48.064393792s) [noise=true , address=45.91.116.10:4082 , piece_id=7NVEGA4QQOE7KYPHXUUX42SJPQL6LP46RGM37VD6A2AWN7PAOM3Q, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.139330448 (68h56m48.065625141s) [noise=true , address=104.36.82.14:4222 , piece_id=U3MBMAKFBFTOPJZ3HS3FXLPDBLQTN577XNCZLUBHRO6IU4BE5VNA, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.140580734 (68h56m48.06687543s) [noise=true , address=77.90.31.19:4842 , piece_id=G3AG375DATHB7MGEEWONE5GMMIC7QHG2VVDKF5AEVOSXV5N4S6AA, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.141452300 (68h56m48.067747027s) [noise=true , address=45.91.116.13:4154 , piece_id=STFPXN2LUJYUIAYROWZIRTCMT2Y3XET6KWA4GZWUHENXRJURP62Q, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.142917062 (68h56m48.069211739s) [noise=true , address=104.36.82.7:4912 , piece_id=A5HPC7CEU66IFNS47HO3AQX22LUZ6AZ4L4XAMZ6T5NCVE4D3TAZA, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.221824189 (68h56m48.148118865s) [upload=55354]: Next returned nil segment Jun 26 06:08:21.221837958 (68h56m48.148132644s) [upload=55354]: Waiting for error group managing segments... Jun 26 06:08:21.229550628 (68h56m48.155845315s) [noise=true , address=104.36.82.14:4223 , piece_id=LP74YFI2SRSYPYEIKREVINCOFMVCXLNDS2555YKUVFN63N6TBJDA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: <nil> Jun 26 06:08:21.230245145 (68h56m48.156539832s) [noise=true , address=104.36.82.14:4222 , piece_id=U3MBMAKFBFTOPJZ3HS3FXLPDBLQTN577XNCZLUBHRO6IU4BE5VNA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: <nil> Jun 26 06:08:21.230801067 (68h56m48.15709574s) [noise=true , address=104.36.82.6:4873 , piece_id=TIIMEQUSDLRFF3OK3WBB74F7SWJ5MEC4Z6IQNELDMRZLNTMQT7LA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: <nil> Jun 26 06:08:21.231457382 (68h56m48.157752071s) [noise=true , address=104.36.82.6:4876 , piece_id=SSMGONONY34EHF5W6QMKO7YXB7RLO5LZVTJMMFKTP6GQNGYFYQ4A, seg_pos={0 0}, upload=55354]: Done uploading piece. err: <nil> Jun 26 06:08:21.231563205 (68h56m48.157857893s) [noise=true , address=104.36.82.13:4186 , piece_id=5SVABN3KS6AUL6HILHNQCVF74BA75VPKTTAK7IQAPKRRME3MOZEA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: <nil> Jun 26 06:08:21.239131119 (68h56m48.165425799s) [noise=true , address=104.36.82.7:4912 , piece_id=A5HPC7CEU66IFNS47HO3AQX22LUZ6AZ4L4XAMZ6T5NCVE4D3TAZA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: <nil> Jun 26 06:08:21.534686665 (68h56m48.460981337s) [noise=true , address=104.36.82.15:4258 , piece_id=YXS4J7YSG6JV33JG4BZLK5AXKE3JXLBTGZJXPXZQ4LBYBT3MU2AA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: <nil> Jun 26 06:08:51.073673533 (68h57m17.999968219s) [noise=true , address=45.91.116.9:5261 , piece_id=CZWB3XJVA7B7R7GWFYNW3RCGNVRXQ3E37XDXD4SUXTEMQFZ73HHA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi, address:45.91.116.9:5261): protocol: rpc: dial tcp 45.91.116.9:5261: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:51.073739594 (68h57m18.000034272s) [noise=true , address=45.91.116.10:4082 , piece_id=7NVEGA4QQOE7KYPHXUUX42SJPQL6LP46RGM37VD6A2AWN7PAOM3Q, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj, address:45.91.116.10:4082): protocol: rpc: dial tcp 45.91.116.10:4082: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:51.073789897 (68h57m18.000084584s) [noise=true , address=77.90.31.19:4842 , piece_id=G3AG375DATHB7MGEEWONE5GMMIC7QHG2VVDKF5AEVOSXV5N4S6AA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: rpc: dial tcp 77.90.31.19:4842: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:51.073849570 (68h57m18.000144246s) [noise=true , address=45.91.116.13:4154 , piece_id=STFPXN2LUJYUIAYROWZIRTCMT2Y3XET6KWA4GZWUHENXRJURP62Q, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN, address:45.91.116.13:4154): protocol: write tcp 159.223.143.39:35392->45.91.116.13:4154: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:51.073904716 (68h57m18.000199391s) [noise=true , address=77.90.31.13:4626 , piece_id=MXYBIEDFAFK53CVTE2C6CKEZWOL3MLW4S7LOHK4IHW6QEIZJL76Q, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: write tcp 159.223.143.39:58864->77.90.31.13:4626: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:51.073952402 (68h57m18.000247082s) [noise=true , address=77.90.31.10:4517 , piece_id=VJZ5T4QNVAPNUHNFNK24S6TAS4V4NKNFMQE2FZMCWWLCCB2ELXXQ, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, address:77.90.31.10:4517): protocol: write tcp 159.223.143.39:36382->77.90.31.10:4517: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:51.074022481 (68h57m18.000317156s) [noise=true , address=77.90.31.15:4697 , piece_id=CUXGVL7PS3H7ZQKUCSYXJCBR3ETYFDPZFYIEAWEYL7C2S3RJY3AA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5, address:77.90.31.15:4697): protocol: write tcp 159.223.143.39:52730->77.90.31.15:4697: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:51.074071824 (68h57m18.000366503s) [noise=true , address=77.90.31.11:4554 , piece_id=R2CIFLWS4LXTGSDSIMIRC4UYVJTXC7T35HPWZ2S7VYUTJ2AMY3BQ, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, address:77.90.31.11:4554): protocol: rpc: dial tcp 77.90.31.11:4554: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:51.074102177 (68h57m18.000396855s) [noise=true , address=77.90.31.14:4661 , piece_id=VXI2ZUSGR4N3H7TPIK75RIDFP3SBAOWVRGHNXTZOHFT5FNG4JWJA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue, address:77.90.31.14:4661): protocol: rpc: dial tcp 77.90.31.14:4661: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:51.074119276 (68h57m18.000413966s) [seg_pos={0 0}, upload=55354]: Done waiting for segment. successful: 7 optimal: 13 errs: <nil> Jun 26 06:08:51.100079867 (68h57m18.026374541s) [upload=55354]: Done uploading segments... Jun 26 06:08:51.100114505 (68h57m18.026409191s) [upload=55354]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 7) Duration: 591.939476ms core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.................XX.........................................] duration: 460.490923ms FAIL (2025-06-26 06:38:53 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 460.516102ms edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..................XX........................................] duration: 539.062682ms FAIL (2025-06-26 06:38:25 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 540.564016ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [....................XXX.....................................] duration: 460.889738ms FAIL (2025-06-26 06:37:43 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 7) ======LOGS====== pload=55354]: Got next segment. Inline: false Jun 26 06:08:21.105827735 (68h56m48.032122411s) [upload=55354]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 26 06:08:21.105833588 (68h56m48.032128274s) [upload=55354]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 26 06:08:21.135318313 (68h56m48.061613009s) [seg_pos={0 0}, upload=55354]: Begin upload segment... Jun 26 06:08:21.135366534 (68h56m48.061661214s) [seg_pos={0 0}, upload=55354]: Done begin upload segment. Jun 26 06:08:21.135390342 (68h56m48.061685028s) [noise=true , address=45.91.116.9:5261 , piece_id=CZWB3XJVA7B7R7GWFYNW3RCGNVRXQ3E37XDXD4SUXTEMQFZ73HHA, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.135405801 (68h56m48.061700497s) [noise=true , address=104.36.82.6:4873 , piece_id=TIIMEQUSDLRFF3OK3WBB74F7SWJ5MEC4Z6IQNELDMRZLNTMQT7LA, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.135774729 (68h56m48.062069418s) [noise=true , address=77.90.31.13:4626 , piece_id=MXYBIEDFAFK53CVTE2C6CKEZWOL3MLW4S7LOHK4IHW6QEIZJL76Q, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.135944436 (68h56m48.06223913s) [noise=true , address=77.90.31.10:4517 , piece_id=VJZ5T4QNVAPNUHNFNK24S6TAS4V4NKNFMQE2FZMCWWLCCB2ELXXQ, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.136080402 (68h56m48.062375094s) [noise=true , address=77.90.31.15:4697 , piece_id=CUXGVL7PS3H7ZQKUCSYXJCBR3ETYFDPZFYIEAWEYL7C2S3RJY3AA, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.136382603 (68h56m48.062677289s) [noise=true , address=104.36.82.13:4186 , piece_id=5SVABN3KS6AUL6HILHNQCVF74BA75VPKTTAK7IQAPKRRME3MOZEA, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.136399304 (68h56m48.062693995s) [noise=true , address=104.36.82.14:4223 , piece_id=LP74YFI2SRSYPYEIKREVINCOFMVCXLNDS2555YKUVFN63N6TBJDA, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.136669576 (68h56m48.062964262s) [noise=true , address=104.36.82.6:4876 , piece_id=SSMGONONY34EHF5W6QMKO7YXB7RLO5LZVTJMMFKTP6GQNGYFYQ4A, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.136809784 (68h56m48.063104477s) [noise=true , address=104.36.82.15:4258 , piece_id=YXS4J7YSG6JV33JG4BZLK5AXKE3JXLBTGZJXPXZQ4LBYBT3MU2AA, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.136950556 (68h56m48.063245242s) [noise=true , address=77.90.31.11:4554 , piece_id=R2CIFLWS4LXTGSDSIMIRC4UYVJTXC7T35HPWZ2S7VYUTJ2AMY3BQ, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.137226497 (68h56m48.06352119s) [noise=true , address=77.90.31.14:4661 , piece_id=VXI2ZUSGR4N3H7TPIK75RIDFP3SBAOWVRGHNXTZOHFT5FNG4JWJA, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.138099107 (68h56m48.064393792s) [noise=true , address=45.91.116.10:4082 , piece_id=7NVEGA4QQOE7KYPHXUUX42SJPQL6LP46RGM37VD6A2AWN7PAOM3Q, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.139330448 (68h56m48.065625141s) [noise=true , address=104.36.82.14:4222 , piece_id=U3MBMAKFBFTOPJZ3HS3FXLPDBLQTN577XNCZLUBHRO6IU4BE5VNA, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.140580734 (68h56m48.06687543s) [noise=true , address=77.90.31.19:4842 , piece_id=G3AG375DATHB7MGEEWONE5GMMIC7QHG2VVDKF5AEVOSXV5N4S6AA, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.141452300 (68h56m48.067747027s) [noise=true , address=45.91.116.13:4154 , piece_id=STFPXN2LUJYUIAYROWZIRTCMT2Y3XET6KWA4GZWUHENXRJURP62Q, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.142917062 (68h56m48.069211739s) [noise=true , address=104.36.82.7:4912 , piece_id=A5HPC7CEU66IFNS47HO3AQX22LUZ6AZ4L4XAMZ6T5NCVE4D3TAZA, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.221824189 (68h56m48.148118865s) [upload=55354]: Next returned nil segment Jun 26 06:08:21.221837958 (68h56m48.148132644s) [upload=55354]: Waiting for error group managing segments... Jun 26 06:08:21.229550628 (68h56m48.155845315s) [noise=true , address=104.36.82.14:4223 , piece_id=LP74YFI2SRSYPYEIKREVINCOFMVCXLNDS2555YKUVFN63N6TBJDA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: <nil> Jun 26 06:08:21.230245145 (68h56m48.156539832s) [noise=true , address=104.36.82.14:4222 , piece_id=U3MBMAKFBFTOPJZ3HS3FXLPDBLQTN577XNCZLUBHRO6IU4BE5VNA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: <nil> Jun 26 06:08:21.230801067 (68h56m48.15709574s) [noise=true , address=104.36.82.6:4873 , piece_id=TIIMEQUSDLRFF3OK3WBB74F7SWJ5MEC4Z6IQNELDMRZLNTMQT7LA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: <nil> Jun 26 06:08:21.231457382 (68h56m48.157752071s) [noise=true , address=104.36.82.6:4876 , piece_id=SSMGONONY34EHF5W6QMKO7YXB7RLO5LZVTJMMFKTP6GQNGYFYQ4A, seg_pos={0 0}, upload=55354]: Done uploading piece. err: <nil> Jun 26 06:08:21.231563205 (68h56m48.157857893s) [noise=true , address=104.36.82.13:4186 , piece_id=5SVABN3KS6AUL6HILHNQCVF74BA75VPKTTAK7IQAPKRRME3MOZEA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: <nil> Jun 26 06:08:21.239131119 (68h56m48.165425799s) [noise=true , address=104.36.82.7:4912 , piece_id=A5HPC7CEU66IFNS47HO3AQX22LUZ6AZ4L4XAMZ6T5NCVE4D3TAZA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: <nil> Jun 26 06:08:21.534686665 (68h56m48.460981337s) [noise=true , address=104.36.82.15:4258 , piece_id=YXS4J7YSG6JV33JG4BZLK5AXKE3JXLBTGZJXPXZQ4LBYBT3MU2AA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: <nil> Jun 26 06:08:51.073673533 (68h57m17.999968219s) [noise=true , address=45.91.116.9:5261 , piece_id=CZWB3XJVA7B7R7GWFYNW3RCGNVRXQ3E37XDXD4SUXTEMQFZ73HHA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi, address:45.91.116.9:5261): protocol: rpc: dial tcp 45.91.116.9:5261: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:51.073739594 (68h57m18.000034272s) [noise=true , address=45.91.116.10:4082 , piece_id=7NVEGA4QQOE7KYPHXUUX42SJPQL6LP46RGM37VD6A2AWN7PAOM3Q, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj, address:45.91.116.10:4082): protocol: rpc: dial tcp 45.91.116.10:4082: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:51.073789897 (68h57m18.000084584s) [noise=true , address=77.90.31.19:4842 , piece_id=G3AG375DATHB7MGEEWONE5GMMIC7QHG2VVDKF5AEVOSXV5N4S6AA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: rpc: dial tcp 77.90.31.19:4842: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:51.073849570 (68h57m18.000144246s) [noise=true , address=45.91.116.13:4154 , piece_id=STFPXN2LUJYUIAYROWZIRTCMT2Y3XET6KWA4GZWUHENXRJURP62Q, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN, address:45.91.116.13:4154): protocol: write tcp 159.223.143.39:35392->45.91.116.13:4154: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:51.073904716 (68h57m18.000199391s) [noise=true , address=77.90.31.13:4626 , piece_id=MXYBIEDFAFK53CVTE2C6CKEZWOL3MLW4S7LOHK4IHW6QEIZJL76Q, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: write tcp 159.223.143.39:58864->77.90.31.13:4626: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:51.073952402 (68h57m18.000247082s) [noise=true , address=77.90.31.10:4517 , piece_id=VJZ5T4QNVAPNUHNFNK24S6TAS4V4NKNFMQE2FZMCWWLCCB2ELXXQ, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, address:77.90.31.10:4517): protocol: write tcp 159.223.143.39:36382->77.90.31.10:4517: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:51.074022481 (68h57m18.000317156s) [noise=true , address=77.90.31.15:4697 , piece_id=CUXGVL7PS3H7ZQKUCSYXJCBR3ETYFDPZFYIEAWEYL7C2S3RJY3AA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5, address:77.90.31.15:4697): protocol: write tcp 159.223.143.39:52730->77.90.31.15:4697: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:51.074071824 (68h57m18.000366503s) [noise=true , address=77.90.31.11:4554 , piece_id=R2CIFLWS4LXTGSDSIMIRC4UYVJTXC7T35HPWZ2S7VYUTJ2AMY3BQ, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, address:77.90.31.11:4554): protocol: rpc: dial tcp 77.90.31.11:4554: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:51.074102177 (68h57m18.000396855s) [noise=true , address=77.90.31.14:4661 , piece_id=VXI2ZUSGR4N3H7TPIK75RIDFP3SBAOWVRGHNXTZOHFT5FNG4JWJA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue, address:77.90.31.14:4661): protocol: rpc: dial tcp 77.90.31.14:4661: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:51.074119276 (68h57m18.000413966s) [seg_pos={0 0}, upload=55354]: Done waiting for segment. successful: 7 optimal: 13 errs: <nil> Jun 26 06:08:51.100079867 (68h57m18.026374541s) [upload=55354]: Done uploading segments... Jun 26 06:08:51.100114505 (68h57m18.026409191s) [upload=55354]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 7) Duration: 460.921557ms core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...................XX.......................................] duration: 499.047596ms FAIL (2025-06-26 06:37:36 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 499.069913ms edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [....................XX......................................] duration: 546.727301ms FAIL (2025-06-26 06:37:13 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 546.750878ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.....................XXX....................................] duration: 430.185744ms FAIL (2025-06-26 06:36:56 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 7) ======LOGS====== pload=55354]: Got next segment. Inline: false Jun 26 06:08:21.105827735 (68h56m48.032122411s) [upload=55354]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 26 06:08:21.105833588 (68h56m48.032128274s) [upload=55354]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 26 06:08:21.135318313 (68h56m48.061613009s) [seg_pos={0 0}, upload=55354]: Begin upload segment... Jun 26 06:08:21.135366534 (68h56m48.061661214s) [seg_pos={0 0}, upload=55354]: Done begin upload segment. Jun 26 06:08:21.135390342 (68h56m48.061685028s) [noise=true , address=45.91.116.9:5261 , piece_id=CZWB3XJVA7B7R7GWFYNW3RCGNVRXQ3E37XDXD4SUXTEMQFZ73HHA, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.135405801 (68h56m48.061700497s) [noise=true , address=104.36.82.6:4873 , piece_id=TIIMEQUSDLRFF3OK3WBB74F7SWJ5MEC4Z6IQNELDMRZLNTMQT7LA, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.135774729 (68h56m48.062069418s) [noise=true , address=77.90.31.13:4626 , piece_id=MXYBIEDFAFK53CVTE2C6CKEZWOL3MLW4S7LOHK4IHW6QEIZJL76Q, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.135944436 (68h56m48.06223913s) [noise=true , address=77.90.31.10:4517 , piece_id=VJZ5T4QNVAPNUHNFNK24S6TAS4V4NKNFMQE2FZMCWWLCCB2ELXXQ, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.136080402 (68h56m48.062375094s) [noise=true , address=77.90.31.15:4697 , piece_id=CUXGVL7PS3H7ZQKUCSYXJCBR3ETYFDPZFYIEAWEYL7C2S3RJY3AA, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.136382603 (68h56m48.062677289s) [noise=true , address=104.36.82.13:4186 , piece_id=5SVABN3KS6AUL6HILHNQCVF74BA75VPKTTAK7IQAPKRRME3MOZEA, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.136399304 (68h56m48.062693995s) [noise=true , address=104.36.82.14:4223 , piece_id=LP74YFI2SRSYPYEIKREVINCOFMVCXLNDS2555YKUVFN63N6TBJDA, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.136669576 (68h56m48.062964262s) [noise=true , address=104.36.82.6:4876 , piece_id=SSMGONONY34EHF5W6QMKO7YXB7RLO5LZVTJMMFKTP6GQNGYFYQ4A, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.136809784 (68h56m48.063104477s) [noise=true , address=104.36.82.15:4258 , piece_id=YXS4J7YSG6JV33JG4BZLK5AXKE3JXLBTGZJXPXZQ4LBYBT3MU2AA, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.136950556 (68h56m48.063245242s) [noise=true , address=77.90.31.11:4554 , piece_id=R2CIFLWS4LXTGSDSIMIRC4UYVJTXC7T35HPWZ2S7VYUTJ2AMY3BQ, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.137226497 (68h56m48.06352119s) [noise=true , address=77.90.31.14:4661 , piece_id=VXI2ZUSGR4N3H7TPIK75RIDFP3SBAOWVRGHNXTZOHFT5FNG4JWJA, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.138099107 (68h56m48.064393792s) [noise=true , address=45.91.116.10:4082 , piece_id=7NVEGA4QQOE7KYPHXUUX42SJPQL6LP46RGM37VD6A2AWN7PAOM3Q, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.139330448 (68h56m48.065625141s) [noise=true , address=104.36.82.14:4222 , piece_id=U3MBMAKFBFTOPJZ3HS3FXLPDBLQTN577XNCZLUBHRO6IU4BE5VNA, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.140580734 (68h56m48.06687543s) [noise=true , address=77.90.31.19:4842 , piece_id=G3AG375DATHB7MGEEWONE5GMMIC7QHG2VVDKF5AEVOSXV5N4S6AA, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.141452300 (68h56m48.067747027s) [noise=true , address=45.91.116.13:4154 , piece_id=STFPXN2LUJYUIAYROWZIRTCMT2Y3XET6KWA4GZWUHENXRJURP62Q, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.142917062 (68h56m48.069211739s) [noise=true , address=104.36.82.7:4912 , piece_id=A5HPC7CEU66IFNS47HO3AQX22LUZ6AZ4L4XAMZ6T5NCVE4D3TAZA, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.221824189 (68h56m48.148118865s) [upload=55354]: Next returned nil segment Jun 26 06:08:21.221837958 (68h56m48.148132644s) [upload=55354]: Waiting for error group managing segments... Jun 26 06:08:21.229550628 (68h56m48.155845315s) [noise=true , address=104.36.82.14:4223 , piece_id=LP74YFI2SRSYPYEIKREVINCOFMVCXLNDS2555YKUVFN63N6TBJDA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: <nil> Jun 26 06:08:21.230245145 (68h56m48.156539832s) [noise=true , address=104.36.82.14:4222 , piece_id=U3MBMAKFBFTOPJZ3HS3FXLPDBLQTN577XNCZLUBHRO6IU4BE5VNA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: <nil> Jun 26 06:08:21.230801067 (68h56m48.15709574s) [noise=true , address=104.36.82.6:4873 , piece_id=TIIMEQUSDLRFF3OK3WBB74F7SWJ5MEC4Z6IQNELDMRZLNTMQT7LA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: <nil> Jun 26 06:08:21.231457382 (68h56m48.157752071s) [noise=true , address=104.36.82.6:4876 , piece_id=SSMGONONY34EHF5W6QMKO7YXB7RLO5LZVTJMMFKTP6GQNGYFYQ4A, seg_pos={0 0}, upload=55354]: Done uploading piece. err: <nil> Jun 26 06:08:21.231563205 (68h56m48.157857893s) [noise=true , address=104.36.82.13:4186 , piece_id=5SVABN3KS6AUL6HILHNQCVF74BA75VPKTTAK7IQAPKRRME3MOZEA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: <nil> Jun 26 06:08:21.239131119 (68h56m48.165425799s) [noise=true , address=104.36.82.7:4912 , piece_id=A5HPC7CEU66IFNS47HO3AQX22LUZ6AZ4L4XAMZ6T5NCVE4D3TAZA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: <nil> Jun 26 06:08:21.534686665 (68h56m48.460981337s) [noise=true , address=104.36.82.15:4258 , piece_id=YXS4J7YSG6JV33JG4BZLK5AXKE3JXLBTGZJXPXZQ4LBYBT3MU2AA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: <nil> Jun 26 06:08:51.073673533 (68h57m17.999968219s) [noise=true , address=45.91.116.9:5261 , piece_id=CZWB3XJVA7B7R7GWFYNW3RCGNVRXQ3E37XDXD4SUXTEMQFZ73HHA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi, address:45.91.116.9:5261): protocol: rpc: dial tcp 45.91.116.9:5261: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:51.073739594 (68h57m18.000034272s) [noise=true , address=45.91.116.10:4082 , piece_id=7NVEGA4QQOE7KYPHXUUX42SJPQL6LP46RGM37VD6A2AWN7PAOM3Q, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj, address:45.91.116.10:4082): protocol: rpc: dial tcp 45.91.116.10:4082: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:51.073789897 (68h57m18.000084584s) [noise=true , address=77.90.31.19:4842 , piece_id=G3AG375DATHB7MGEEWONE5GMMIC7QHG2VVDKF5AEVOSXV5N4S6AA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: rpc: dial tcp 77.90.31.19:4842: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:51.073849570 (68h57m18.000144246s) [noise=true , address=45.91.116.13:4154 , piece_id=STFPXN2LUJYUIAYROWZIRTCMT2Y3XET6KWA4GZWUHENXRJURP62Q, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN, address:45.91.116.13:4154): protocol: write tcp 159.223.143.39:35392->45.91.116.13:4154: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:51.073904716 (68h57m18.000199391s) [noise=true , address=77.90.31.13:4626 , piece_id=MXYBIEDFAFK53CVTE2C6CKEZWOL3MLW4S7LOHK4IHW6QEIZJL76Q, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: write tcp 159.223.143.39:58864->77.90.31.13:4626: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:51.073952402 (68h57m18.000247082s) [noise=true , address=77.90.31.10:4517 , piece_id=VJZ5T4QNVAPNUHNFNK24S6TAS4V4NKNFMQE2FZMCWWLCCB2ELXXQ, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, address:77.90.31.10:4517): protocol: write tcp 159.223.143.39:36382->77.90.31.10:4517: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:51.074022481 (68h57m18.000317156s) [noise=true , address=77.90.31.15:4697 , piece_id=CUXGVL7PS3H7ZQKUCSYXJCBR3ETYFDPZFYIEAWEYL7C2S3RJY3AA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5, address:77.90.31.15:4697): protocol: write tcp 159.223.143.39:52730->77.90.31.15:4697: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:51.074071824 (68h57m18.000366503s) [noise=true , address=77.90.31.11:4554 , piece_id=R2CIFLWS4LXTGSDSIMIRC4UYVJTXC7T35HPWZ2S7VYUTJ2AMY3BQ, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, address:77.90.31.11:4554): protocol: rpc: dial tcp 77.90.31.11:4554: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:51.074102177 (68h57m18.000396855s) [noise=true , address=77.90.31.14:4661 , piece_id=VXI2ZUSGR4N3H7TPIK75RIDFP3SBAOWVRGHNXTZOHFT5FNG4JWJA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue, address:77.90.31.14:4661): protocol: rpc: dial tcp 77.90.31.14:4661: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:51.074119276 (68h57m18.000413966s) [seg_pos={0 0}, upload=55354]: Done waiting for segment. successful: 7 optimal: 13 errs: <nil> Jun 26 06:08:51.100079867 (68h57m18.026374541s) [upload=55354]: Done uploading segments... Jun 26 06:08:51.100114505 (68h57m18.026409191s) [upload=55354]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 7) Duration: 430.383945ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [......................XXX...................................] duration: 446.04059ms FAIL (2025-06-26 06:36:00 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 7) ======LOGS====== pload=55354]: Got next segment. Inline: false Jun 26 06:08:21.105827735 (68h56m48.032122411s) [upload=55354]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 26 06:08:21.105833588 (68h56m48.032128274s) [upload=55354]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 26 06:08:21.135318313 (68h56m48.061613009s) [seg_pos={0 0}, upload=55354]: Begin upload segment... Jun 26 06:08:21.135366534 (68h56m48.061661214s) [seg_pos={0 0}, upload=55354]: Done begin upload segment. Jun 26 06:08:21.135390342 (68h56m48.061685028s) [noise=true , address=45.91.116.9:5261 , piece_id=CZWB3XJVA7B7R7GWFYNW3RCGNVRXQ3E37XDXD4SUXTEMQFZ73HHA, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.135405801 (68h56m48.061700497s) [noise=true , address=104.36.82.6:4873 , piece_id=TIIMEQUSDLRFF3OK3WBB74F7SWJ5MEC4Z6IQNELDMRZLNTMQT7LA, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.135774729 (68h56m48.062069418s) [noise=true , address=77.90.31.13:4626 , piece_id=MXYBIEDFAFK53CVTE2C6CKEZWOL3MLW4S7LOHK4IHW6QEIZJL76Q, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.135944436 (68h56m48.06223913s) [noise=true , address=77.90.31.10:4517 , piece_id=VJZ5T4QNVAPNUHNFNK24S6TAS4V4NKNFMQE2FZMCWWLCCB2ELXXQ, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.136080402 (68h56m48.062375094s) [noise=true , address=77.90.31.15:4697 , piece_id=CUXGVL7PS3H7ZQKUCSYXJCBR3ETYFDPZFYIEAWEYL7C2S3RJY3AA, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.136382603 (68h56m48.062677289s) [noise=true , address=104.36.82.13:4186 , piece_id=5SVABN3KS6AUL6HILHNQCVF74BA75VPKTTAK7IQAPKRRME3MOZEA, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.136399304 (68h56m48.062693995s) [noise=true , address=104.36.82.14:4223 , piece_id=LP74YFI2SRSYPYEIKREVINCOFMVCXLNDS2555YKUVFN63N6TBJDA, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.136669576 (68h56m48.062964262s) [noise=true , address=104.36.82.6:4876 , piece_id=SSMGONONY34EHF5W6QMKO7YXB7RLO5LZVTJMMFKTP6GQNGYFYQ4A, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.136809784 (68h56m48.063104477s) [noise=true , address=104.36.82.15:4258 , piece_id=YXS4J7YSG6JV33JG4BZLK5AXKE3JXLBTGZJXPXZQ4LBYBT3MU2AA, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.136950556 (68h56m48.063245242s) [noise=true , address=77.90.31.11:4554 , piece_id=R2CIFLWS4LXTGSDSIMIRC4UYVJTXC7T35HPWZ2S7VYUTJ2AMY3BQ, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.137226497 (68h56m48.06352119s) [noise=true , address=77.90.31.14:4661 , piece_id=VXI2ZUSGR4N3H7TPIK75RIDFP3SBAOWVRGHNXTZOHFT5FNG4JWJA, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.138099107 (68h56m48.064393792s) [noise=true , address=45.91.116.10:4082 , piece_id=7NVEGA4QQOE7KYPHXUUX42SJPQL6LP46RGM37VD6A2AWN7PAOM3Q, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.139330448 (68h56m48.065625141s) [noise=true , address=104.36.82.14:4222 , piece_id=U3MBMAKFBFTOPJZ3HS3FXLPDBLQTN577XNCZLUBHRO6IU4BE5VNA, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.140580734 (68h56m48.06687543s) [noise=true , address=77.90.31.19:4842 , piece_id=G3AG375DATHB7MGEEWONE5GMMIC7QHG2VVDKF5AEVOSXV5N4S6AA, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.141452300 (68h56m48.067747027s) [noise=true , address=45.91.116.13:4154 , piece_id=STFPXN2LUJYUIAYROWZIRTCMT2Y3XET6KWA4GZWUHENXRJURP62Q, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.142917062 (68h56m48.069211739s) [noise=true , address=104.36.82.7:4912 , piece_id=A5HPC7CEU66IFNS47HO3AQX22LUZ6AZ4L4XAMZ6T5NCVE4D3TAZA, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.221824189 (68h56m48.148118865s) [upload=55354]: Next returned nil segment Jun 26 06:08:21.221837958 (68h56m48.148132644s) [upload=55354]: Waiting for error group managing segments... Jun 26 06:08:21.229550628 (68h56m48.155845315s) [noise=true , address=104.36.82.14:4223 , piece_id=LP74YFI2SRSYPYEIKREVINCOFMVCXLNDS2555YKUVFN63N6TBJDA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: <nil> Jun 26 06:08:21.230245145 (68h56m48.156539832s) [noise=true , address=104.36.82.14:4222 , piece_id=U3MBMAKFBFTOPJZ3HS3FXLPDBLQTN577XNCZLUBHRO6IU4BE5VNA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: <nil> Jun 26 06:08:21.230801067 (68h56m48.15709574s) [noise=true , address=104.36.82.6:4873 , piece_id=TIIMEQUSDLRFF3OK3WBB74F7SWJ5MEC4Z6IQNELDMRZLNTMQT7LA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: <nil> Jun 26 06:08:21.231457382 (68h56m48.157752071s) [noise=true , address=104.36.82.6:4876 , piece_id=SSMGONONY34EHF5W6QMKO7YXB7RLO5LZVTJMMFKTP6GQNGYFYQ4A, seg_pos={0 0}, upload=55354]: Done uploading piece. err: <nil> Jun 26 06:08:21.231563205 (68h56m48.157857893s) [noise=true , address=104.36.82.13:4186 , piece_id=5SVABN3KS6AUL6HILHNQCVF74BA75VPKTTAK7IQAPKRRME3MOZEA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: <nil> Jun 26 06:08:21.239131119 (68h56m48.165425799s) [noise=true , address=104.36.82.7:4912 , piece_id=A5HPC7CEU66IFNS47HO3AQX22LUZ6AZ4L4XAMZ6T5NCVE4D3TAZA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: <nil> Jun 26 06:08:21.534686665 (68h56m48.460981337s) [noise=true , address=104.36.82.15:4258 , piece_id=YXS4J7YSG6JV33JG4BZLK5AXKE3JXLBTGZJXPXZQ4LBYBT3MU2AA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: <nil> Jun 26 06:08:51.073673533 (68h57m17.999968219s) [noise=true , address=45.91.116.9:5261 , piece_id=CZWB3XJVA7B7R7GWFYNW3RCGNVRXQ3E37XDXD4SUXTEMQFZ73HHA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi, address:45.91.116.9:5261): protocol: rpc: dial tcp 45.91.116.9:5261: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:51.073739594 (68h57m18.000034272s) [noise=true , address=45.91.116.10:4082 , piece_id=7NVEGA4QQOE7KYPHXUUX42SJPQL6LP46RGM37VD6A2AWN7PAOM3Q, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj, address:45.91.116.10:4082): protocol: rpc: dial tcp 45.91.116.10:4082: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:51.073789897 (68h57m18.000084584s) [noise=true , address=77.90.31.19:4842 , piece_id=G3AG375DATHB7MGEEWONE5GMMIC7QHG2VVDKF5AEVOSXV5N4S6AA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: rpc: dial tcp 77.90.31.19:4842: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:51.073849570 (68h57m18.000144246s) [noise=true , address=45.91.116.13:4154 , piece_id=STFPXN2LUJYUIAYROWZIRTCMT2Y3XET6KWA4GZWUHENXRJURP62Q, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN, address:45.91.116.13:4154): protocol: write tcp 159.223.143.39:35392->45.91.116.13:4154: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:51.073904716 (68h57m18.000199391s) [noise=true , address=77.90.31.13:4626 , piece_id=MXYBIEDFAFK53CVTE2C6CKEZWOL3MLW4S7LOHK4IHW6QEIZJL76Q, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: write tcp 159.223.143.39:58864->77.90.31.13:4626: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:51.073952402 (68h57m18.000247082s) [noise=true , address=77.90.31.10:4517 , piece_id=VJZ5T4QNVAPNUHNFNK24S6TAS4V4NKNFMQE2FZMCWWLCCB2ELXXQ, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, address:77.90.31.10:4517): protocol: write tcp 159.223.143.39:36382->77.90.31.10:4517: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:51.074022481 (68h57m18.000317156s) [noise=true , address=77.90.31.15:4697 , piece_id=CUXGVL7PS3H7ZQKUCSYXJCBR3ETYFDPZFYIEAWEYL7C2S3RJY3AA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5, address:77.90.31.15:4697): protocol: write tcp 159.223.143.39:52730->77.90.31.15:4697: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:51.074071824 (68h57m18.000366503s) [noise=true , address=77.90.31.11:4554 , piece_id=R2CIFLWS4LXTGSDSIMIRC4UYVJTXC7T35HPWZ2S7VYUTJ2AMY3BQ, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, address:77.90.31.11:4554): protocol: rpc: dial tcp 77.90.31.11:4554: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:51.074102177 (68h57m18.000396855s) [noise=true , address=77.90.31.14:4661 , piece_id=VXI2ZUSGR4N3H7TPIK75RIDFP3SBAOWVRGHNXTZOHFT5FNG4JWJA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue, address:77.90.31.14:4661): protocol: rpc: dial tcp 77.90.31.14:4661: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:51.074119276 (68h57m18.000413966s) [seg_pos={0 0}, upload=55354]: Done waiting for segment. successful: 7 optimal: 13 errs: <nil> Jun 26 06:08:51.100079867 (68h57m18.026374541s) [upload=55354]: Done uploading segments... Jun 26 06:08:51.100114505 (68h57m18.026409191s) [upload=55354]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 7) Duration: 446.083104ms core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.....................XX.....................................] duration: 530.793885ms FAIL (2025-06-26 06:35:59 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 530.815285ms edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [X...........................................................] duration: 560.085938ms FAIL (2025-06-26 06:35:22 UTC) orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded Duration: 560.149666ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.......................XXX..................................] duration: 494.572676ms FAIL (2025-06-26 06:35:16 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 7) ======LOGS====== pload=55354]: Got next segment. Inline: false Jun 26 06:08:21.105827735 (68h56m48.032122411s) [upload=55354]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 26 06:08:21.105833588 (68h56m48.032128274s) [upload=55354]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 26 06:08:21.135318313 (68h56m48.061613009s) [seg_pos={0 0}, upload=55354]: Begin upload segment... Jun 26 06:08:21.135366534 (68h56m48.061661214s) [seg_pos={0 0}, upload=55354]: Done begin upload segment. Jun 26 06:08:21.135390342 (68h56m48.061685028s) [noise=true , address=45.91.116.9:5261 , piece_id=CZWB3XJVA7B7R7GWFYNW3RCGNVRXQ3E37XDXD4SUXTEMQFZ73HHA, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.135405801 (68h56m48.061700497s) [noise=true , address=104.36.82.6:4873 , piece_id=TIIMEQUSDLRFF3OK3WBB74F7SWJ5MEC4Z6IQNELDMRZLNTMQT7LA, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.135774729 (68h56m48.062069418s) [noise=true , address=77.90.31.13:4626 , piece_id=MXYBIEDFAFK53CVTE2C6CKEZWOL3MLW4S7LOHK4IHW6QEIZJL76Q, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.135944436 (68h56m48.06223913s) [noise=true , address=77.90.31.10:4517 , piece_id=VJZ5T4QNVAPNUHNFNK24S6TAS4V4NKNFMQE2FZMCWWLCCB2ELXXQ, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.136080402 (68h56m48.062375094s) [noise=true , address=77.90.31.15:4697 , piece_id=CUXGVL7PS3H7ZQKUCSYXJCBR3ETYFDPZFYIEAWEYL7C2S3RJY3AA, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.136382603 (68h56m48.062677289s) [noise=true , address=104.36.82.13:4186 , piece_id=5SVABN3KS6AUL6HILHNQCVF74BA75VPKTTAK7IQAPKRRME3MOZEA, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.136399304 (68h56m48.062693995s) [noise=true , address=104.36.82.14:4223 , piece_id=LP74YFI2SRSYPYEIKREVINCOFMVCXLNDS2555YKUVFN63N6TBJDA, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.136669576 (68h56m48.062964262s) [noise=true , address=104.36.82.6:4876 , piece_id=SSMGONONY34EHF5W6QMKO7YXB7RLO5LZVTJMMFKTP6GQNGYFYQ4A, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.136809784 (68h56m48.063104477s) [noise=true , address=104.36.82.15:4258 , piece_id=YXS4J7YSG6JV33JG4BZLK5AXKE3JXLBTGZJXPXZQ4LBYBT3MU2AA, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.136950556 (68h56m48.063245242s) [noise=true , address=77.90.31.11:4554 , piece_id=R2CIFLWS4LXTGSDSIMIRC4UYVJTXC7T35HPWZ2S7VYUTJ2AMY3BQ, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.137226497 (68h56m48.06352119s) [noise=true , address=77.90.31.14:4661 , piece_id=VXI2ZUSGR4N3H7TPIK75RIDFP3SBAOWVRGHNXTZOHFT5FNG4JWJA, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.138099107 (68h56m48.064393792s) [noise=true , address=45.91.116.10:4082 , piece_id=7NVEGA4QQOE7KYPHXUUX42SJPQL6LP46RGM37VD6A2AWN7PAOM3Q, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.139330448 (68h56m48.065625141s) [noise=true , address=104.36.82.14:4222 , piece_id=U3MBMAKFBFTOPJZ3HS3FXLPDBLQTN577XNCZLUBHRO6IU4BE5VNA, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.140580734 (68h56m48.06687543s) [noise=true , address=77.90.31.19:4842 , piece_id=G3AG375DATHB7MGEEWONE5GMMIC7QHG2VVDKF5AEVOSXV5N4S6AA, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.141452300 (68h56m48.067747027s) [noise=true , address=45.91.116.13:4154 , piece_id=STFPXN2LUJYUIAYROWZIRTCMT2Y3XET6KWA4GZWUHENXRJURP62Q, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.142917062 (68h56m48.069211739s) [noise=true , address=104.36.82.7:4912 , piece_id=A5HPC7CEU66IFNS47HO3AQX22LUZ6AZ4L4XAMZ6T5NCVE4D3TAZA, seg_pos={0 0}, upload=55354]: Uploading piece... Jun 26 06:08:21.221824189 (68h56m48.148118865s) [upload=55354]: Next returned nil segment Jun 26 06:08:21.221837958 (68h56m48.148132644s) [upload=55354]: Waiting for error group managing segments... Jun 26 06:08:21.229550628 (68h56m48.155845315s) [noise=true , address=104.36.82.14:4223 , piece_id=LP74YFI2SRSYPYEIKREVINCOFMVCXLNDS2555YKUVFN63N6TBJDA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: <nil> Jun 26 06:08:21.230245145 (68h56m48.156539832s) [noise=true , address=104.36.82.14:4222 , piece_id=U3MBMAKFBFTOPJZ3HS3FXLPDBLQTN577XNCZLUBHRO6IU4BE5VNA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: <nil> Jun 26 06:08:21.230801067 (68h56m48.15709574s) [noise=true , address=104.36.82.6:4873 , piece_id=TIIMEQUSDLRFF3OK3WBB74F7SWJ5MEC4Z6IQNELDMRZLNTMQT7LA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: <nil> Jun 26 06:08:21.231457382 (68h56m48.157752071s) [noise=true , address=104.36.82.6:4876 , piece_id=SSMGONONY34EHF5W6QMKO7YXB7RLO5LZVTJMMFKTP6GQNGYFYQ4A, seg_pos={0 0}, upload=55354]: Done uploading piece. err: <nil> Jun 26 06:08:21.231563205 (68h56m48.157857893s) [noise=true , address=104.36.82.13:4186 , piece_id=5SVABN3KS6AUL6HILHNQCVF74BA75VPKTTAK7IQAPKRRME3MOZEA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: <nil> Jun 26 06:08:21.239131119 (68h56m48.165425799s) [noise=true , address=104.36.82.7:4912 , piece_id=A5HPC7CEU66IFNS47HO3AQX22LUZ6AZ4L4XAMZ6T5NCVE4D3TAZA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: <nil> Jun 26 06:08:21.534686665 (68h56m48.460981337s) [noise=true , address=104.36.82.15:4258 , piece_id=YXS4J7YSG6JV33JG4BZLK5AXKE3JXLBTGZJXPXZQ4LBYBT3MU2AA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: <nil> Jun 26 06:08:51.073673533 (68h57m17.999968219s) [noise=true , address=45.91.116.9:5261 , piece_id=CZWB3XJVA7B7R7GWFYNW3RCGNVRXQ3E37XDXD4SUXTEMQFZ73HHA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi, address:45.91.116.9:5261): protocol: rpc: dial tcp 45.91.116.9:5261: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:51.073739594 (68h57m18.000034272s) [noise=true , address=45.91.116.10:4082 , piece_id=7NVEGA4QQOE7KYPHXUUX42SJPQL6LP46RGM37VD6A2AWN7PAOM3Q, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj, address:45.91.116.10:4082): protocol: rpc: dial tcp 45.91.116.10:4082: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:51.073789897 (68h57m18.000084584s) [noise=true , address=77.90.31.19:4842 , piece_id=G3AG375DATHB7MGEEWONE5GMMIC7QHG2VVDKF5AEVOSXV5N4S6AA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: rpc: dial tcp 77.90.31.19:4842: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:51.073849570 (68h57m18.000144246s) [noise=true , address=45.91.116.13:4154 , piece_id=STFPXN2LUJYUIAYROWZIRTCMT2Y3XET6KWA4GZWUHENXRJURP62Q, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN, address:45.91.116.13:4154): protocol: write tcp 159.223.143.39:35392->45.91.116.13:4154: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:51.073904716 (68h57m18.000199391s) [noise=true , address=77.90.31.13:4626 , piece_id=MXYBIEDFAFK53CVTE2C6CKEZWOL3MLW4S7LOHK4IHW6QEIZJL76Q, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: write tcp 159.223.143.39:58864->77.90.31.13:4626: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:51.073952402 (68h57m18.000247082s) [noise=true , address=77.90.31.10:4517 , piece_id=VJZ5T4QNVAPNUHNFNK24S6TAS4V4NKNFMQE2FZMCWWLCCB2ELXXQ, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, address:77.90.31.10:4517): protocol: write tcp 159.223.143.39:36382->77.90.31.10:4517: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:51.074022481 (68h57m18.000317156s) [noise=true , address=77.90.31.15:4697 , piece_id=CUXGVL7PS3H7ZQKUCSYXJCBR3ETYFDPZFYIEAWEYL7C2S3RJY3AA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5, address:77.90.31.15:4697): protocol: write tcp 159.223.143.39:52730->77.90.31.15:4697: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:51.074071824 (68h57m18.000366503s) [noise=true , address=77.90.31.11:4554 , piece_id=R2CIFLWS4LXTGSDSIMIRC4UYVJTXC7T35HPWZ2S7VYUTJ2AMY3BQ, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, address:77.90.31.11:4554): protocol: rpc: dial tcp 77.90.31.11:4554: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:51.074102177 (68h57m18.000396855s) [noise=true , address=77.90.31.14:4661 , piece_id=VXI2ZUSGR4N3H7TPIK75RIDFP3SBAOWVRGHNXTZOHFT5FNG4JWJA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue, address:77.90.31.14:4661): protocol: rpc: dial tcp 77.90.31.14:4661: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jun 26 06:08:51.074119276 (68h57m18.000413966s) [seg_pos={0 0}, upload=55354]: Done waiting for segment. successful: 7 optimal: 13 errs: <nil> Jun 26 06:08:51.100079867 (68h57m18.026374541s) [upload=55354]: Done uploading segments... Jun 26 06:08:51.100114505 (68h57m18.026409191s) [upload=55354]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 7) Duration: 494.604481ms core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [......................XX....................................] duration: 509.186173ms FAIL (2025-06-26 06:35:04 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 509.217302ms edge gateway us-select-1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)