Orbiter: EU/US/AP | Code | Config | Playbook | Edge Deployment | Escalation Procedure
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
3 out of 60 is failed (threshold=3) [........................XXX.................................] duration:
456.838513ms
FAIL (2025-06-26 06:35:00 UTC)
orbiter warning:
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 7)
======LOGS======
pload=55354]: Got next segment. Inline: false
Jun 26 06:08:21.105827735 (68h56m48.032122411s) [upload=55354]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jun 26 06:08:21.105833588 (68h56m48.032128274s) [upload=55354]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jun 26 06:08:21.135318313 (68h56m48.061613009s) [seg_pos={0 0}, upload=55354]: Begin upload segment...
Jun 26 06:08:21.135366534 (68h56m48.061661214s) [seg_pos={0 0}, upload=55354]: Done begin upload segment.
Jun 26 06:08:21.135390342 (68h56m48.061685028s) [noise=true , address=45.91.116.9:5261 , piece_id=CZWB3XJVA7B7R7GWFYNW3RCGNVRXQ3E37XDXD4SUXTEMQFZ73HHA, seg_pos={0 0}, upload=55354]: Uploading piece...
Jun 26 06:08:21.135405801 (68h56m48.061700497s) [noise=true , address=104.36.82.6:4873 , piece_id=TIIMEQUSDLRFF3OK3WBB74F7SWJ5MEC4Z6IQNELDMRZLNTMQT7LA, seg_pos={0 0}, upload=55354]: Uploading piece...
Jun 26 06:08:21.135774729 (68h56m48.062069418s) [noise=true , address=77.90.31.13:4626 , piece_id=MXYBIEDFAFK53CVTE2C6CKEZWOL3MLW4S7LOHK4IHW6QEIZJL76Q, seg_pos={0 0}, upload=55354]: Uploading piece...
Jun 26 06:08:21.135944436 (68h56m48.06223913s) [noise=true , address=77.90.31.10:4517 , piece_id=VJZ5T4QNVAPNUHNFNK24S6TAS4V4NKNFMQE2FZMCWWLCCB2ELXXQ, seg_pos={0 0}, upload=55354]: Uploading piece...
Jun 26 06:08:21.136080402 (68h56m48.062375094s) [noise=true , address=77.90.31.15:4697 , piece_id=CUXGVL7PS3H7ZQKUCSYXJCBR3ETYFDPZFYIEAWEYL7C2S3RJY3AA, seg_pos={0 0}, upload=55354]: Uploading piece...
Jun 26 06:08:21.136382603 (68h56m48.062677289s) [noise=true , address=104.36.82.13:4186 , piece_id=5SVABN3KS6AUL6HILHNQCVF74BA75VPKTTAK7IQAPKRRME3MOZEA, seg_pos={0 0}, upload=55354]: Uploading piece...
Jun 26 06:08:21.136399304 (68h56m48.062693995s) [noise=true , address=104.36.82.14:4223 , piece_id=LP74YFI2SRSYPYEIKREVINCOFMVCXLNDS2555YKUVFN63N6TBJDA, seg_pos={0 0}, upload=55354]: Uploading piece...
Jun 26 06:08:21.136669576 (68h56m48.062964262s) [noise=true , address=104.36.82.6:4876 , piece_id=SSMGONONY34EHF5W6QMKO7YXB7RLO5LZVTJMMFKTP6GQNGYFYQ4A, seg_pos={0 0}, upload=55354]: Uploading piece...
Jun 26 06:08:21.136809784 (68h56m48.063104477s) [noise=true , address=104.36.82.15:4258 , piece_id=YXS4J7YSG6JV33JG4BZLK5AXKE3JXLBTGZJXPXZQ4LBYBT3MU2AA, seg_pos={0 0}, upload=55354]: Uploading piece...
Jun 26 06:08:21.136950556 (68h56m48.063245242s) [noise=true , address=77.90.31.11:4554 , piece_id=R2CIFLWS4LXTGSDSIMIRC4UYVJTXC7T35HPWZ2S7VYUTJ2AMY3BQ, seg_pos={0 0}, upload=55354]: Uploading piece...
Jun 26 06:08:21.137226497 (68h56m48.06352119s) [noise=true , address=77.90.31.14:4661 , piece_id=VXI2ZUSGR4N3H7TPIK75RIDFP3SBAOWVRGHNXTZOHFT5FNG4JWJA, seg_pos={0 0}, upload=55354]: Uploading piece...
Jun 26 06:08:21.138099107 (68h56m48.064393792s) [noise=true , address=45.91.116.10:4082 , piece_id=7NVEGA4QQOE7KYPHXUUX42SJPQL6LP46RGM37VD6A2AWN7PAOM3Q, seg_pos={0 0}, upload=55354]: Uploading piece...
Jun 26 06:08:21.139330448 (68h56m48.065625141s) [noise=true , address=104.36.82.14:4222 , piece_id=U3MBMAKFBFTOPJZ3HS3FXLPDBLQTN577XNCZLUBHRO6IU4BE5VNA, seg_pos={0 0}, upload=55354]: Uploading piece...
Jun 26 06:08:21.140580734 (68h56m48.06687543s) [noise=true , address=77.90.31.19:4842 , piece_id=G3AG375DATHB7MGEEWONE5GMMIC7QHG2VVDKF5AEVOSXV5N4S6AA, seg_pos={0 0}, upload=55354]: Uploading piece...
Jun 26 06:08:21.141452300 (68h56m48.067747027s) [noise=true , address=45.91.116.13:4154 , piece_id=STFPXN2LUJYUIAYROWZIRTCMT2Y3XET6KWA4GZWUHENXRJURP62Q, seg_pos={0 0}, upload=55354]: Uploading piece...
Jun 26 06:08:21.142917062 (68h56m48.069211739s) [noise=true , address=104.36.82.7:4912 , piece_id=A5HPC7CEU66IFNS47HO3AQX22LUZ6AZ4L4XAMZ6T5NCVE4D3TAZA, seg_pos={0 0}, upload=55354]: Uploading piece...
Jun 26 06:08:21.221824189 (68h56m48.148118865s) [upload=55354]: Next returned nil segment
Jun 26 06:08:21.221837958 (68h56m48.148132644s) [upload=55354]: Waiting for error group managing segments...
Jun 26 06:08:21.229550628 (68h56m48.155845315s) [noise=true , address=104.36.82.14:4223 , piece_id=LP74YFI2SRSYPYEIKREVINCOFMVCXLNDS2555YKUVFN63N6TBJDA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: <nil>
Jun 26 06:08:21.230245145 (68h56m48.156539832s) [noise=true , address=104.36.82.14:4222 , piece_id=U3MBMAKFBFTOPJZ3HS3FXLPDBLQTN577XNCZLUBHRO6IU4BE5VNA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: <nil>
Jun 26 06:08:21.230801067 (68h56m48.15709574s) [noise=true , address=104.36.82.6:4873 , piece_id=TIIMEQUSDLRFF3OK3WBB74F7SWJ5MEC4Z6IQNELDMRZLNTMQT7LA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: <nil>
Jun 26 06:08:21.231457382 (68h56m48.157752071s) [noise=true , address=104.36.82.6:4876 , piece_id=SSMGONONY34EHF5W6QMKO7YXB7RLO5LZVTJMMFKTP6GQNGYFYQ4A, seg_pos={0 0}, upload=55354]: Done uploading piece. err: <nil>
Jun 26 06:08:21.231563205 (68h56m48.157857893s) [noise=true , address=104.36.82.13:4186 , piece_id=5SVABN3KS6AUL6HILHNQCVF74BA75VPKTTAK7IQAPKRRME3MOZEA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: <nil>
Jun 26 06:08:21.239131119 (68h56m48.165425799s) [noise=true , address=104.36.82.7:4912 , piece_id=A5HPC7CEU66IFNS47HO3AQX22LUZ6AZ4L4XAMZ6T5NCVE4D3TAZA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: <nil>
Jun 26 06:08:21.534686665 (68h56m48.460981337s) [noise=true , address=104.36.82.15:4258 , piece_id=YXS4J7YSG6JV33JG4BZLK5AXKE3JXLBTGZJXPXZQ4LBYBT3MU2AA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: <nil>
Jun 26 06:08:51.073673533 (68h57m17.999968219s) [noise=true , address=45.91.116.9:5261 , piece_id=CZWB3XJVA7B7R7GWFYNW3RCGNVRXQ3E37XDXD4SUXTEMQFZ73HHA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi, address:45.91.116.9:5261): protocol: rpc: dial tcp 45.91.116.9:5261: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jun 26 06:08:51.073739594 (68h57m18.000034272s) [noise=true , address=45.91.116.10:4082 , piece_id=7NVEGA4QQOE7KYPHXUUX42SJPQL6LP46RGM37VD6A2AWN7PAOM3Q, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj, address:45.91.116.10:4082): protocol: rpc: dial tcp 45.91.116.10:4082: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jun 26 06:08:51.073789897 (68h57m18.000084584s) [noise=true , address=77.90.31.19:4842 , piece_id=G3AG375DATHB7MGEEWONE5GMMIC7QHG2VVDKF5AEVOSXV5N4S6AA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: rpc: dial tcp 77.90.31.19:4842: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jun 26 06:08:51.073849570 (68h57m18.000144246s) [noise=true , address=45.91.116.13:4154 , piece_id=STFPXN2LUJYUIAYROWZIRTCMT2Y3XET6KWA4GZWUHENXRJURP62Q, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN, address:45.91.116.13:4154): protocol: write tcp 159.223.143.39:35392->45.91.116.13:4154: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jun 26 06:08:51.073904716 (68h57m18.000199391s) [noise=true , address=77.90.31.13:4626 , piece_id=MXYBIEDFAFK53CVTE2C6CKEZWOL3MLW4S7LOHK4IHW6QEIZJL76Q, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: write tcp 159.223.143.39:58864->77.90.31.13:4626: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jun 26 06:08:51.073952402 (68h57m18.000247082s) [noise=true , address=77.90.31.10:4517 , piece_id=VJZ5T4QNVAPNUHNFNK24S6TAS4V4NKNFMQE2FZMCWWLCCB2ELXXQ, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, address:77.90.31.10:4517): protocol: write tcp 159.223.143.39:36382->77.90.31.10:4517: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jun 26 06:08:51.074022481 (68h57m18.000317156s) [noise=true , address=77.90.31.15:4697 , piece_id=CUXGVL7PS3H7ZQKUCSYXJCBR3ETYFDPZFYIEAWEYL7C2S3RJY3AA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5, address:77.90.31.15:4697): protocol: write tcp 159.223.143.39:52730->77.90.31.15:4697: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jun 26 06:08:51.074071824 (68h57m18.000366503s) [noise=true , address=77.90.31.11:4554 , piece_id=R2CIFLWS4LXTGSDSIMIRC4UYVJTXC7T35HPWZ2S7VYUTJ2AMY3BQ, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, address:77.90.31.11:4554): protocol: rpc: dial tcp 77.90.31.11:4554: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jun 26 06:08:51.074102177 (68h57m18.000396855s) [noise=true , address=77.90.31.14:4661 , piece_id=VXI2ZUSGR4N3H7TPIK75RIDFP3SBAOWVRGHNXTZOHFT5FNG4JWJA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue, address:77.90.31.14:4661): protocol: rpc: dial tcp 77.90.31.14:4661: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jun 26 06:08:51.074119276 (68h57m18.000413966s) [seg_pos={0 0}, upload=55354]: Done waiting for segment. successful: 7 optimal: 13 errs: <nil>
Jun 26 06:08:51.100079867 (68h57m18.026374541s) [upload=55354]: Done uploading segments...
Jun 26 06:08:51.100114505 (68h57m18.026409191s) [upload=55354]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 7)
Duration: 456.87047ms
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [.......................XX...................................] duration:
474.700516ms
FAIL (2025-06-26 06:34:38 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 474.724088ms
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [.X..........................................................] duration:
563.239798ms
FAIL (2025-06-26 06:34:20 UTC)
orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded
Duration: 563.285088ms
core
uplink
us-select-1
us1-placement-20
|
checkers.AuthServiceHTTP{"https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us1)antiblip:
1 out of 60 is failed (threshold=3) [X...........................................................] duration:
148.647611ms
FAIL (2025-06-26 06:34:07 UTC)
orbiter warning: unexpected status code "500 Internal Server Error" "\n"
Duration: 148.677682ms
edge
authservice
http
bgp
us1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
3 out of 60 is failed (threshold=3) [.........................XXX................................] duration:
436.55885ms
FAIL (2025-06-26 06:33:57 UTC)
orbiter warning:
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 7)
======LOGS======
pload=55354]: Got next segment. Inline: false
Jun 26 06:08:21.105827735 (68h56m48.032122411s) [upload=55354]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jun 26 06:08:21.105833588 (68h56m48.032128274s) [upload=55354]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jun 26 06:08:21.135318313 (68h56m48.061613009s) [seg_pos={0 0}, upload=55354]: Begin upload segment...
Jun 26 06:08:21.135366534 (68h56m48.061661214s) [seg_pos={0 0}, upload=55354]: Done begin upload segment.
Jun 26 06:08:21.135390342 (68h56m48.061685028s) [noise=true , address=45.91.116.9:5261 , piece_id=CZWB3XJVA7B7R7GWFYNW3RCGNVRXQ3E37XDXD4SUXTEMQFZ73HHA, seg_pos={0 0}, upload=55354]: Uploading piece...
Jun 26 06:08:21.135405801 (68h56m48.061700497s) [noise=true , address=104.36.82.6:4873 , piece_id=TIIMEQUSDLRFF3OK3WBB74F7SWJ5MEC4Z6IQNELDMRZLNTMQT7LA, seg_pos={0 0}, upload=55354]: Uploading piece...
Jun 26 06:08:21.135774729 (68h56m48.062069418s) [noise=true , address=77.90.31.13:4626 , piece_id=MXYBIEDFAFK53CVTE2C6CKEZWOL3MLW4S7LOHK4IHW6QEIZJL76Q, seg_pos={0 0}, upload=55354]: Uploading piece...
Jun 26 06:08:21.135944436 (68h56m48.06223913s) [noise=true , address=77.90.31.10:4517 , piece_id=VJZ5T4QNVAPNUHNFNK24S6TAS4V4NKNFMQE2FZMCWWLCCB2ELXXQ, seg_pos={0 0}, upload=55354]: Uploading piece...
Jun 26 06:08:21.136080402 (68h56m48.062375094s) [noise=true , address=77.90.31.15:4697 , piece_id=CUXGVL7PS3H7ZQKUCSYXJCBR3ETYFDPZFYIEAWEYL7C2S3RJY3AA, seg_pos={0 0}, upload=55354]: Uploading piece...
Jun 26 06:08:21.136382603 (68h56m48.062677289s) [noise=true , address=104.36.82.13:4186 , piece_id=5SVABN3KS6AUL6HILHNQCVF74BA75VPKTTAK7IQAPKRRME3MOZEA, seg_pos={0 0}, upload=55354]: Uploading piece...
Jun 26 06:08:21.136399304 (68h56m48.062693995s) [noise=true , address=104.36.82.14:4223 , piece_id=LP74YFI2SRSYPYEIKREVINCOFMVCXLNDS2555YKUVFN63N6TBJDA, seg_pos={0 0}, upload=55354]: Uploading piece...
Jun 26 06:08:21.136669576 (68h56m48.062964262s) [noise=true , address=104.36.82.6:4876 , piece_id=SSMGONONY34EHF5W6QMKO7YXB7RLO5LZVTJMMFKTP6GQNGYFYQ4A, seg_pos={0 0}, upload=55354]: Uploading piece...
Jun 26 06:08:21.136809784 (68h56m48.063104477s) [noise=true , address=104.36.82.15:4258 , piece_id=YXS4J7YSG6JV33JG4BZLK5AXKE3JXLBTGZJXPXZQ4LBYBT3MU2AA, seg_pos={0 0}, upload=55354]: Uploading piece...
Jun 26 06:08:21.136950556 (68h56m48.063245242s) [noise=true , address=77.90.31.11:4554 , piece_id=R2CIFLWS4LXTGSDSIMIRC4UYVJTXC7T35HPWZ2S7VYUTJ2AMY3BQ, seg_pos={0 0}, upload=55354]: Uploading piece...
Jun 26 06:08:21.137226497 (68h56m48.06352119s) [noise=true , address=77.90.31.14:4661 , piece_id=VXI2ZUSGR4N3H7TPIK75RIDFP3SBAOWVRGHNXTZOHFT5FNG4JWJA, seg_pos={0 0}, upload=55354]: Uploading piece...
Jun 26 06:08:21.138099107 (68h56m48.064393792s) [noise=true , address=45.91.116.10:4082 , piece_id=7NVEGA4QQOE7KYPHXUUX42SJPQL6LP46RGM37VD6A2AWN7PAOM3Q, seg_pos={0 0}, upload=55354]: Uploading piece...
Jun 26 06:08:21.139330448 (68h56m48.065625141s) [noise=true , address=104.36.82.14:4222 , piece_id=U3MBMAKFBFTOPJZ3HS3FXLPDBLQTN577XNCZLUBHRO6IU4BE5VNA, seg_pos={0 0}, upload=55354]: Uploading piece...
Jun 26 06:08:21.140580734 (68h56m48.06687543s) [noise=true , address=77.90.31.19:4842 , piece_id=G3AG375DATHB7MGEEWONE5GMMIC7QHG2VVDKF5AEVOSXV5N4S6AA, seg_pos={0 0}, upload=55354]: Uploading piece...
Jun 26 06:08:21.141452300 (68h56m48.067747027s) [noise=true , address=45.91.116.13:4154 , piece_id=STFPXN2LUJYUIAYROWZIRTCMT2Y3XET6KWA4GZWUHENXRJURP62Q, seg_pos={0 0}, upload=55354]: Uploading piece...
Jun 26 06:08:21.142917062 (68h56m48.069211739s) [noise=true , address=104.36.82.7:4912 , piece_id=A5HPC7CEU66IFNS47HO3AQX22LUZ6AZ4L4XAMZ6T5NCVE4D3TAZA, seg_pos={0 0}, upload=55354]: Uploading piece...
Jun 26 06:08:21.221824189 (68h56m48.148118865s) [upload=55354]: Next returned nil segment
Jun 26 06:08:21.221837958 (68h56m48.148132644s) [upload=55354]: Waiting for error group managing segments...
Jun 26 06:08:21.229550628 (68h56m48.155845315s) [noise=true , address=104.36.82.14:4223 , piece_id=LP74YFI2SRSYPYEIKREVINCOFMVCXLNDS2555YKUVFN63N6TBJDA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: <nil>
Jun 26 06:08:21.230245145 (68h56m48.156539832s) [noise=true , address=104.36.82.14:4222 , piece_id=U3MBMAKFBFTOPJZ3HS3FXLPDBLQTN577XNCZLUBHRO6IU4BE5VNA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: <nil>
Jun 26 06:08:21.230801067 (68h56m48.15709574s) [noise=true , address=104.36.82.6:4873 , piece_id=TIIMEQUSDLRFF3OK3WBB74F7SWJ5MEC4Z6IQNELDMRZLNTMQT7LA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: <nil>
Jun 26 06:08:21.231457382 (68h56m48.157752071s) [noise=true , address=104.36.82.6:4876 , piece_id=SSMGONONY34EHF5W6QMKO7YXB7RLO5LZVTJMMFKTP6GQNGYFYQ4A, seg_pos={0 0}, upload=55354]: Done uploading piece. err: <nil>
Jun 26 06:08:21.231563205 (68h56m48.157857893s) [noise=true , address=104.36.82.13:4186 , piece_id=5SVABN3KS6AUL6HILHNQCVF74BA75VPKTTAK7IQAPKRRME3MOZEA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: <nil>
Jun 26 06:08:21.239131119 (68h56m48.165425799s) [noise=true , address=104.36.82.7:4912 , piece_id=A5HPC7CEU66IFNS47HO3AQX22LUZ6AZ4L4XAMZ6T5NCVE4D3TAZA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: <nil>
Jun 26 06:08:21.534686665 (68h56m48.460981337s) [noise=true , address=104.36.82.15:4258 , piece_id=YXS4J7YSG6JV33JG4BZLK5AXKE3JXLBTGZJXPXZQ4LBYBT3MU2AA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: <nil>
Jun 26 06:08:51.073673533 (68h57m17.999968219s) [noise=true , address=45.91.116.9:5261 , piece_id=CZWB3XJVA7B7R7GWFYNW3RCGNVRXQ3E37XDXD4SUXTEMQFZ73HHA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi, address:45.91.116.9:5261): protocol: rpc: dial tcp 45.91.116.9:5261: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jun 26 06:08:51.073739594 (68h57m18.000034272s) [noise=true , address=45.91.116.10:4082 , piece_id=7NVEGA4QQOE7KYPHXUUX42SJPQL6LP46RGM37VD6A2AWN7PAOM3Q, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj, address:45.91.116.10:4082): protocol: rpc: dial tcp 45.91.116.10:4082: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jun 26 06:08:51.073789897 (68h57m18.000084584s) [noise=true , address=77.90.31.19:4842 , piece_id=G3AG375DATHB7MGEEWONE5GMMIC7QHG2VVDKF5AEVOSXV5N4S6AA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: rpc: dial tcp 77.90.31.19:4842: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jun 26 06:08:51.073849570 (68h57m18.000144246s) [noise=true , address=45.91.116.13:4154 , piece_id=STFPXN2LUJYUIAYROWZIRTCMT2Y3XET6KWA4GZWUHENXRJURP62Q, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN, address:45.91.116.13:4154): protocol: write tcp 159.223.143.39:35392->45.91.116.13:4154: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jun 26 06:08:51.073904716 (68h57m18.000199391s) [noise=true , address=77.90.31.13:4626 , piece_id=MXYBIEDFAFK53CVTE2C6CKEZWOL3MLW4S7LOHK4IHW6QEIZJL76Q, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: write tcp 159.223.143.39:58864->77.90.31.13:4626: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jun 26 06:08:51.073952402 (68h57m18.000247082s) [noise=true , address=77.90.31.10:4517 , piece_id=VJZ5T4QNVAPNUHNFNK24S6TAS4V4NKNFMQE2FZMCWWLCCB2ELXXQ, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, address:77.90.31.10:4517): protocol: write tcp 159.223.143.39:36382->77.90.31.10:4517: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jun 26 06:08:51.074022481 (68h57m18.000317156s) [noise=true , address=77.90.31.15:4697 , piece_id=CUXGVL7PS3H7ZQKUCSYXJCBR3ETYFDPZFYIEAWEYL7C2S3RJY3AA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5, address:77.90.31.15:4697): protocol: write tcp 159.223.143.39:52730->77.90.31.15:4697: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jun 26 06:08:51.074071824 (68h57m18.000366503s) [noise=true , address=77.90.31.11:4554 , piece_id=R2CIFLWS4LXTGSDSIMIRC4UYVJTXC7T35HPWZ2S7VYUTJ2AMY3BQ, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, address:77.90.31.11:4554): protocol: rpc: dial tcp 77.90.31.11:4554: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jun 26 06:08:51.074102177 (68h57m18.000396855s) [noise=true , address=77.90.31.14:4661 , piece_id=VXI2ZUSGR4N3H7TPIK75RIDFP3SBAOWVRGHNXTZOHFT5FNG4JWJA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue, address:77.90.31.14:4661): protocol: rpc: dial tcp 77.90.31.14:4661: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jun 26 06:08:51.074119276 (68h57m18.000413966s) [seg_pos={0 0}, upload=55354]: Done waiting for segment. successful: 7 optimal: 13 errs: <nil>
Jun 26 06:08:51.100079867 (68h57m18.026374541s) [upload=55354]: Done uploading segments...
Jun 26 06:08:51.100114505 (68h57m18.026409191s) [upload=55354]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 7)
Duration: 436.581573ms
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [..X.........................................................] duration:
642.812567ms
FAIL (2025-06-26 06:33:48 UTC)
orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded
Duration: 642.835829ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [........................XX..................................] duration:
477.61469ms
FAIL (2025-06-26 06:33:41 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 477.64038ms
edge
gateway
us-select-1
|
checkers.AuthServiceHTTP{"https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us1)antiblip:
1 out of 60 is failed (threshold=3) [.X..........................................................] duration:
193.616283ms
FAIL (2025-06-26 06:33:25 UTC)
orbiter warning: unexpected status code "500 Internal Server Error" "\n"
Duration: 193.675958ms
edge
authservice
http
bgp
us1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [.........................XX.................................] duration:
580.539933ms
FAIL (2025-06-26 06:33:09 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 580.569239ms
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
3 out of 60 is failed (threshold=3) [..........................XXX...............................] duration:
529.331828ms
FAIL (2025-06-26 06:33:07 UTC)
orbiter warning:
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 7)
======LOGS======
pload=55354]: Got next segment. Inline: false
Jun 26 06:08:21.105827735 (68h56m48.032122411s) [upload=55354]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jun 26 06:08:21.105833588 (68h56m48.032128274s) [upload=55354]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jun 26 06:08:21.135318313 (68h56m48.061613009s) [seg_pos={0 0}, upload=55354]: Begin upload segment...
Jun 26 06:08:21.135366534 (68h56m48.061661214s) [seg_pos={0 0}, upload=55354]: Done begin upload segment.
Jun 26 06:08:21.135390342 (68h56m48.061685028s) [noise=true , address=45.91.116.9:5261 , piece_id=CZWB3XJVA7B7R7GWFYNW3RCGNVRXQ3E37XDXD4SUXTEMQFZ73HHA, seg_pos={0 0}, upload=55354]: Uploading piece...
Jun 26 06:08:21.135405801 (68h56m48.061700497s) [noise=true , address=104.36.82.6:4873 , piece_id=TIIMEQUSDLRFF3OK3WBB74F7SWJ5MEC4Z6IQNELDMRZLNTMQT7LA, seg_pos={0 0}, upload=55354]: Uploading piece...
Jun 26 06:08:21.135774729 (68h56m48.062069418s) [noise=true , address=77.90.31.13:4626 , piece_id=MXYBIEDFAFK53CVTE2C6CKEZWOL3MLW4S7LOHK4IHW6QEIZJL76Q, seg_pos={0 0}, upload=55354]: Uploading piece...
Jun 26 06:08:21.135944436 (68h56m48.06223913s) [noise=true , address=77.90.31.10:4517 , piece_id=VJZ5T4QNVAPNUHNFNK24S6TAS4V4NKNFMQE2FZMCWWLCCB2ELXXQ, seg_pos={0 0}, upload=55354]: Uploading piece...
Jun 26 06:08:21.136080402 (68h56m48.062375094s) [noise=true , address=77.90.31.15:4697 , piece_id=CUXGVL7PS3H7ZQKUCSYXJCBR3ETYFDPZFYIEAWEYL7C2S3RJY3AA, seg_pos={0 0}, upload=55354]: Uploading piece...
Jun 26 06:08:21.136382603 (68h56m48.062677289s) [noise=true , address=104.36.82.13:4186 , piece_id=5SVABN3KS6AUL6HILHNQCVF74BA75VPKTTAK7IQAPKRRME3MOZEA, seg_pos={0 0}, upload=55354]: Uploading piece...
Jun 26 06:08:21.136399304 (68h56m48.062693995s) [noise=true , address=104.36.82.14:4223 , piece_id=LP74YFI2SRSYPYEIKREVINCOFMVCXLNDS2555YKUVFN63N6TBJDA, seg_pos={0 0}, upload=55354]: Uploading piece...
Jun 26 06:08:21.136669576 (68h56m48.062964262s) [noise=true , address=104.36.82.6:4876 , piece_id=SSMGONONY34EHF5W6QMKO7YXB7RLO5LZVTJMMFKTP6GQNGYFYQ4A, seg_pos={0 0}, upload=55354]: Uploading piece...
Jun 26 06:08:21.136809784 (68h56m48.063104477s) [noise=true , address=104.36.82.15:4258 , piece_id=YXS4J7YSG6JV33JG4BZLK5AXKE3JXLBTGZJXPXZQ4LBYBT3MU2AA, seg_pos={0 0}, upload=55354]: Uploading piece...
Jun 26 06:08:21.136950556 (68h56m48.063245242s) [noise=true , address=77.90.31.11:4554 , piece_id=R2CIFLWS4LXTGSDSIMIRC4UYVJTXC7T35HPWZ2S7VYUTJ2AMY3BQ, seg_pos={0 0}, upload=55354]: Uploading piece...
Jun 26 06:08:21.137226497 (68h56m48.06352119s) [noise=true , address=77.90.31.14:4661 , piece_id=VXI2ZUSGR4N3H7TPIK75RIDFP3SBAOWVRGHNXTZOHFT5FNG4JWJA, seg_pos={0 0}, upload=55354]: Uploading piece...
Jun 26 06:08:21.138099107 (68h56m48.064393792s) [noise=true , address=45.91.116.10:4082 , piece_id=7NVEGA4QQOE7KYPHXUUX42SJPQL6LP46RGM37VD6A2AWN7PAOM3Q, seg_pos={0 0}, upload=55354]: Uploading piece...
Jun 26 06:08:21.139330448 (68h56m48.065625141s) [noise=true , address=104.36.82.14:4222 , piece_id=U3MBMAKFBFTOPJZ3HS3FXLPDBLQTN577XNCZLUBHRO6IU4BE5VNA, seg_pos={0 0}, upload=55354]: Uploading piece...
Jun 26 06:08:21.140580734 (68h56m48.06687543s) [noise=true , address=77.90.31.19:4842 , piece_id=G3AG375DATHB7MGEEWONE5GMMIC7QHG2VVDKF5AEVOSXV5N4S6AA, seg_pos={0 0}, upload=55354]: Uploading piece...
Jun 26 06:08:21.141452300 (68h56m48.067747027s) [noise=true , address=45.91.116.13:4154 , piece_id=STFPXN2LUJYUIAYROWZIRTCMT2Y3XET6KWA4GZWUHENXRJURP62Q, seg_pos={0 0}, upload=55354]: Uploading piece...
Jun 26 06:08:21.142917062 (68h56m48.069211739s) [noise=true , address=104.36.82.7:4912 , piece_id=A5HPC7CEU66IFNS47HO3AQX22LUZ6AZ4L4XAMZ6T5NCVE4D3TAZA, seg_pos={0 0}, upload=55354]: Uploading piece...
Jun 26 06:08:21.221824189 (68h56m48.148118865s) [upload=55354]: Next returned nil segment
Jun 26 06:08:21.221837958 (68h56m48.148132644s) [upload=55354]: Waiting for error group managing segments...
Jun 26 06:08:21.229550628 (68h56m48.155845315s) [noise=true , address=104.36.82.14:4223 , piece_id=LP74YFI2SRSYPYEIKREVINCOFMVCXLNDS2555YKUVFN63N6TBJDA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: <nil>
Jun 26 06:08:21.230245145 (68h56m48.156539832s) [noise=true , address=104.36.82.14:4222 , piece_id=U3MBMAKFBFTOPJZ3HS3FXLPDBLQTN577XNCZLUBHRO6IU4BE5VNA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: <nil>
Jun 26 06:08:21.230801067 (68h56m48.15709574s) [noise=true , address=104.36.82.6:4873 , piece_id=TIIMEQUSDLRFF3OK3WBB74F7SWJ5MEC4Z6IQNELDMRZLNTMQT7LA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: <nil>
Jun 26 06:08:21.231457382 (68h56m48.157752071s) [noise=true , address=104.36.82.6:4876 , piece_id=SSMGONONY34EHF5W6QMKO7YXB7RLO5LZVTJMMFKTP6GQNGYFYQ4A, seg_pos={0 0}, upload=55354]: Done uploading piece. err: <nil>
Jun 26 06:08:21.231563205 (68h56m48.157857893s) [noise=true , address=104.36.82.13:4186 , piece_id=5SVABN3KS6AUL6HILHNQCVF74BA75VPKTTAK7IQAPKRRME3MOZEA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: <nil>
Jun 26 06:08:21.239131119 (68h56m48.165425799s) [noise=true , address=104.36.82.7:4912 , piece_id=A5HPC7CEU66IFNS47HO3AQX22LUZ6AZ4L4XAMZ6T5NCVE4D3TAZA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: <nil>
Jun 26 06:08:21.534686665 (68h56m48.460981337s) [noise=true , address=104.36.82.15:4258 , piece_id=YXS4J7YSG6JV33JG4BZLK5AXKE3JXLBTGZJXPXZQ4LBYBT3MU2AA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: <nil>
Jun 26 06:08:51.073673533 (68h57m17.999968219s) [noise=true , address=45.91.116.9:5261 , piece_id=CZWB3XJVA7B7R7GWFYNW3RCGNVRXQ3E37XDXD4SUXTEMQFZ73HHA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi, address:45.91.116.9:5261): protocol: rpc: dial tcp 45.91.116.9:5261: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jun 26 06:08:51.073739594 (68h57m18.000034272s) [noise=true , address=45.91.116.10:4082 , piece_id=7NVEGA4QQOE7KYPHXUUX42SJPQL6LP46RGM37VD6A2AWN7PAOM3Q, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj, address:45.91.116.10:4082): protocol: rpc: dial tcp 45.91.116.10:4082: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jun 26 06:08:51.073789897 (68h57m18.000084584s) [noise=true , address=77.90.31.19:4842 , piece_id=G3AG375DATHB7MGEEWONE5GMMIC7QHG2VVDKF5AEVOSXV5N4S6AA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: rpc: dial tcp 77.90.31.19:4842: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jun 26 06:08:51.073849570 (68h57m18.000144246s) [noise=true , address=45.91.116.13:4154 , piece_id=STFPXN2LUJYUIAYROWZIRTCMT2Y3XET6KWA4GZWUHENXRJURP62Q, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN, address:45.91.116.13:4154): protocol: write tcp 159.223.143.39:35392->45.91.116.13:4154: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jun 26 06:08:51.073904716 (68h57m18.000199391s) [noise=true , address=77.90.31.13:4626 , piece_id=MXYBIEDFAFK53CVTE2C6CKEZWOL3MLW4S7LOHK4IHW6QEIZJL76Q, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: write tcp 159.223.143.39:58864->77.90.31.13:4626: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jun 26 06:08:51.073952402 (68h57m18.000247082s) [noise=true , address=77.90.31.10:4517 , piece_id=VJZ5T4QNVAPNUHNFNK24S6TAS4V4NKNFMQE2FZMCWWLCCB2ELXXQ, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, address:77.90.31.10:4517): protocol: write tcp 159.223.143.39:36382->77.90.31.10:4517: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jun 26 06:08:51.074022481 (68h57m18.000317156s) [noise=true , address=77.90.31.15:4697 , piece_id=CUXGVL7PS3H7ZQKUCSYXJCBR3ETYFDPZFYIEAWEYL7C2S3RJY3AA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5, address:77.90.31.15:4697): protocol: write tcp 159.223.143.39:52730->77.90.31.15:4697: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jun 26 06:08:51.074071824 (68h57m18.000366503s) [noise=true , address=77.90.31.11:4554 , piece_id=R2CIFLWS4LXTGSDSIMIRC4UYVJTXC7T35HPWZ2S7VYUTJ2AMY3BQ, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, address:77.90.31.11:4554): protocol: rpc: dial tcp 77.90.31.11:4554: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jun 26 06:08:51.074102177 (68h57m18.000396855s) [noise=true , address=77.90.31.14:4661 , piece_id=VXI2ZUSGR4N3H7TPIK75RIDFP3SBAOWVRGHNXTZOHFT5FNG4JWJA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue, address:77.90.31.14:4661): protocol: rpc: dial tcp 77.90.31.14:4661: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jun 26 06:08:51.074119276 (68h57m18.000413966s) [seg_pos={0 0}, upload=55354]: Done waiting for segment. successful: 7 optimal: 13 errs: <nil>
Jun 26 06:08:51.100079867 (68h57m18.026374541s) [upload=55354]: Done uploading segments...
Jun 26 06:08:51.100114505 (68h57m18.026409191s) [upload=55354]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 7)
Duration: 529.363595ms
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [...X........................................................] duration:
565.972148ms
FAIL (2025-06-26 06:33:06 UTC)
orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded
Duration: 565.991845ms
core
uplink
us-select-1
us1-placement-20
|
checkers.AuthServiceHTTP{"https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us1)antiblip:
1 out of 60 is failed (threshold=3) [..X.........................................................] duration:
413.66797ms
FAIL (2025-06-26 06:32:35 UTC)
orbiter warning: unexpected status code "500 Internal Server Error" "\n"
Duration: 413.691881ms
edge
authservice
http
bgp
us1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [....X.......................................................] duration:
544.287477ms
FAIL (2025-06-26 06:32:25 UTC)
orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded
Duration: 544.314306ms
core
uplink
us-select-1
us1-placement-20
|
checkers.AuthServiceHTTP{"https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us1)antiblip:
1 out of 60 is failed (threshold=3) [...X........................................................] duration:
186.819394ms
FAIL (2025-06-26 06:32:15 UTC)
orbiter warning: unexpected status code "500 Internal Server Error" "\n"
Duration: 186.842669ms
edge
authservice
http
bgp
us1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
3 out of 60 is failed (threshold=3) [...........................XXX..............................] duration:
581.600376ms
FAIL (2025-06-26 06:32:14 UTC)
orbiter warning:
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 7)
======LOGS======
pload=55354]: Got next segment. Inline: false
Jun 26 06:08:21.105827735 (68h56m48.032122411s) [upload=55354]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jun 26 06:08:21.105833588 (68h56m48.032128274s) [upload=55354]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jun 26 06:08:21.135318313 (68h56m48.061613009s) [seg_pos={0 0}, upload=55354]: Begin upload segment...
Jun 26 06:08:21.135366534 (68h56m48.061661214s) [seg_pos={0 0}, upload=55354]: Done begin upload segment.
Jun 26 06:08:21.135390342 (68h56m48.061685028s) [noise=true , address=45.91.116.9:5261 , piece_id=CZWB3XJVA7B7R7GWFYNW3RCGNVRXQ3E37XDXD4SUXTEMQFZ73HHA, seg_pos={0 0}, upload=55354]: Uploading piece...
Jun 26 06:08:21.135405801 (68h56m48.061700497s) [noise=true , address=104.36.82.6:4873 , piece_id=TIIMEQUSDLRFF3OK3WBB74F7SWJ5MEC4Z6IQNELDMRZLNTMQT7LA, seg_pos={0 0}, upload=55354]: Uploading piece...
Jun 26 06:08:21.135774729 (68h56m48.062069418s) [noise=true , address=77.90.31.13:4626 , piece_id=MXYBIEDFAFK53CVTE2C6CKEZWOL3MLW4S7LOHK4IHW6QEIZJL76Q, seg_pos={0 0}, upload=55354]: Uploading piece...
Jun 26 06:08:21.135944436 (68h56m48.06223913s) [noise=true , address=77.90.31.10:4517 , piece_id=VJZ5T4QNVAPNUHNFNK24S6TAS4V4NKNFMQE2FZMCWWLCCB2ELXXQ, seg_pos={0 0}, upload=55354]: Uploading piece...
Jun 26 06:08:21.136080402 (68h56m48.062375094s) [noise=true , address=77.90.31.15:4697 , piece_id=CUXGVL7PS3H7ZQKUCSYXJCBR3ETYFDPZFYIEAWEYL7C2S3RJY3AA, seg_pos={0 0}, upload=55354]: Uploading piece...
Jun 26 06:08:21.136382603 (68h56m48.062677289s) [noise=true , address=104.36.82.13:4186 , piece_id=5SVABN3KS6AUL6HILHNQCVF74BA75VPKTTAK7IQAPKRRME3MOZEA, seg_pos={0 0}, upload=55354]: Uploading piece...
Jun 26 06:08:21.136399304 (68h56m48.062693995s) [noise=true , address=104.36.82.14:4223 , piece_id=LP74YFI2SRSYPYEIKREVINCOFMVCXLNDS2555YKUVFN63N6TBJDA, seg_pos={0 0}, upload=55354]: Uploading piece...
Jun 26 06:08:21.136669576 (68h56m48.062964262s) [noise=true , address=104.36.82.6:4876 , piece_id=SSMGONONY34EHF5W6QMKO7YXB7RLO5LZVTJMMFKTP6GQNGYFYQ4A, seg_pos={0 0}, upload=55354]: Uploading piece...
Jun 26 06:08:21.136809784 (68h56m48.063104477s) [noise=true , address=104.36.82.15:4258 , piece_id=YXS4J7YSG6JV33JG4BZLK5AXKE3JXLBTGZJXPXZQ4LBYBT3MU2AA, seg_pos={0 0}, upload=55354]: Uploading piece...
Jun 26 06:08:21.136950556 (68h56m48.063245242s) [noise=true , address=77.90.31.11:4554 , piece_id=R2CIFLWS4LXTGSDSIMIRC4UYVJTXC7T35HPWZ2S7VYUTJ2AMY3BQ, seg_pos={0 0}, upload=55354]: Uploading piece...
Jun 26 06:08:21.137226497 (68h56m48.06352119s) [noise=true , address=77.90.31.14:4661 , piece_id=VXI2ZUSGR4N3H7TPIK75RIDFP3SBAOWVRGHNXTZOHFT5FNG4JWJA, seg_pos={0 0}, upload=55354]: Uploading piece...
Jun 26 06:08:21.138099107 (68h56m48.064393792s) [noise=true , address=45.91.116.10:4082 , piece_id=7NVEGA4QQOE7KYPHXUUX42SJPQL6LP46RGM37VD6A2AWN7PAOM3Q, seg_pos={0 0}, upload=55354]: Uploading piece...
Jun 26 06:08:21.139330448 (68h56m48.065625141s) [noise=true , address=104.36.82.14:4222 , piece_id=U3MBMAKFBFTOPJZ3HS3FXLPDBLQTN577XNCZLUBHRO6IU4BE5VNA, seg_pos={0 0}, upload=55354]: Uploading piece...
Jun 26 06:08:21.140580734 (68h56m48.06687543s) [noise=true , address=77.90.31.19:4842 , piece_id=G3AG375DATHB7MGEEWONE5GMMIC7QHG2VVDKF5AEVOSXV5N4S6AA, seg_pos={0 0}, upload=55354]: Uploading piece...
Jun 26 06:08:21.141452300 (68h56m48.067747027s) [noise=true , address=45.91.116.13:4154 , piece_id=STFPXN2LUJYUIAYROWZIRTCMT2Y3XET6KWA4GZWUHENXRJURP62Q, seg_pos={0 0}, upload=55354]: Uploading piece...
Jun 26 06:08:21.142917062 (68h56m48.069211739s) [noise=true , address=104.36.82.7:4912 , piece_id=A5HPC7CEU66IFNS47HO3AQX22LUZ6AZ4L4XAMZ6T5NCVE4D3TAZA, seg_pos={0 0}, upload=55354]: Uploading piece...
Jun 26 06:08:21.221824189 (68h56m48.148118865s) [upload=55354]: Next returned nil segment
Jun 26 06:08:21.221837958 (68h56m48.148132644s) [upload=55354]: Waiting for error group managing segments...
Jun 26 06:08:21.229550628 (68h56m48.155845315s) [noise=true , address=104.36.82.14:4223 , piece_id=LP74YFI2SRSYPYEIKREVINCOFMVCXLNDS2555YKUVFN63N6TBJDA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: <nil>
Jun 26 06:08:21.230245145 (68h56m48.156539832s) [noise=true , address=104.36.82.14:4222 , piece_id=U3MBMAKFBFTOPJZ3HS3FXLPDBLQTN577XNCZLUBHRO6IU4BE5VNA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: <nil>
Jun 26 06:08:21.230801067 (68h56m48.15709574s) [noise=true , address=104.36.82.6:4873 , piece_id=TIIMEQUSDLRFF3OK3WBB74F7SWJ5MEC4Z6IQNELDMRZLNTMQT7LA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: <nil>
Jun 26 06:08:21.231457382 (68h56m48.157752071s) [noise=true , address=104.36.82.6:4876 , piece_id=SSMGONONY34EHF5W6QMKO7YXB7RLO5LZVTJMMFKTP6GQNGYFYQ4A, seg_pos={0 0}, upload=55354]: Done uploading piece. err: <nil>
Jun 26 06:08:21.231563205 (68h56m48.157857893s) [noise=true , address=104.36.82.13:4186 , piece_id=5SVABN3KS6AUL6HILHNQCVF74BA75VPKTTAK7IQAPKRRME3MOZEA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: <nil>
Jun 26 06:08:21.239131119 (68h56m48.165425799s) [noise=true , address=104.36.82.7:4912 , piece_id=A5HPC7CEU66IFNS47HO3AQX22LUZ6AZ4L4XAMZ6T5NCVE4D3TAZA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: <nil>
Jun 26 06:08:21.534686665 (68h56m48.460981337s) [noise=true , address=104.36.82.15:4258 , piece_id=YXS4J7YSG6JV33JG4BZLK5AXKE3JXLBTGZJXPXZQ4LBYBT3MU2AA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: <nil>
Jun 26 06:08:51.073673533 (68h57m17.999968219s) [noise=true , address=45.91.116.9:5261 , piece_id=CZWB3XJVA7B7R7GWFYNW3RCGNVRXQ3E37XDXD4SUXTEMQFZ73HHA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi, address:45.91.116.9:5261): protocol: rpc: dial tcp 45.91.116.9:5261: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jun 26 06:08:51.073739594 (68h57m18.000034272s) [noise=true , address=45.91.116.10:4082 , piece_id=7NVEGA4QQOE7KYPHXUUX42SJPQL6LP46RGM37VD6A2AWN7PAOM3Q, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj, address:45.91.116.10:4082): protocol: rpc: dial tcp 45.91.116.10:4082: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jun 26 06:08:51.073789897 (68h57m18.000084584s) [noise=true , address=77.90.31.19:4842 , piece_id=G3AG375DATHB7MGEEWONE5GMMIC7QHG2VVDKF5AEVOSXV5N4S6AA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: rpc: dial tcp 77.90.31.19:4842: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jun 26 06:08:51.073849570 (68h57m18.000144246s) [noise=true , address=45.91.116.13:4154 , piece_id=STFPXN2LUJYUIAYROWZIRTCMT2Y3XET6KWA4GZWUHENXRJURP62Q, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN, address:45.91.116.13:4154): protocol: write tcp 159.223.143.39:35392->45.91.116.13:4154: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jun 26 06:08:51.073904716 (68h57m18.000199391s) [noise=true , address=77.90.31.13:4626 , piece_id=MXYBIEDFAFK53CVTE2C6CKEZWOL3MLW4S7LOHK4IHW6QEIZJL76Q, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: write tcp 159.223.143.39:58864->77.90.31.13:4626: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jun 26 06:08:51.073952402 (68h57m18.000247082s) [noise=true , address=77.90.31.10:4517 , piece_id=VJZ5T4QNVAPNUHNFNK24S6TAS4V4NKNFMQE2FZMCWWLCCB2ELXXQ, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, address:77.90.31.10:4517): protocol: write tcp 159.223.143.39:36382->77.90.31.10:4517: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jun 26 06:08:51.074022481 (68h57m18.000317156s) [noise=true , address=77.90.31.15:4697 , piece_id=CUXGVL7PS3H7ZQKUCSYXJCBR3ETYFDPZFYIEAWEYL7C2S3RJY3AA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5, address:77.90.31.15:4697): protocol: write tcp 159.223.143.39:52730->77.90.31.15:4697: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jun 26 06:08:51.074071824 (68h57m18.000366503s) [noise=true , address=77.90.31.11:4554 , piece_id=R2CIFLWS4LXTGSDSIMIRC4UYVJTXC7T35HPWZ2S7VYUTJ2AMY3BQ, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, address:77.90.31.11:4554): protocol: rpc: dial tcp 77.90.31.11:4554: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jun 26 06:08:51.074102177 (68h57m18.000396855s) [noise=true , address=77.90.31.14:4661 , piece_id=VXI2ZUSGR4N3H7TPIK75RIDFP3SBAOWVRGHNXTZOHFT5FNG4JWJA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue, address:77.90.31.14:4661): protocol: rpc: dial tcp 77.90.31.14:4661: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jun 26 06:08:51.074119276 (68h57m18.000413966s) [seg_pos={0 0}, upload=55354]: Done waiting for segment. successful: 7 optimal: 13 errs: <nil>
Jun 26 06:08:51.100079867 (68h57m18.026374541s) [upload=55354]: Done uploading segments...
Jun 26 06:08:51.100114505 (68h57m18.026409191s) [upload=55354]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 7)
Duration: 581.632424ms
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [..........................XX................................] duration:
660.851138ms
FAIL (2025-06-26 06:32:14 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 660.940497ms
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [...........................XX...............................] duration:
501.448794ms
FAIL (2025-06-26 06:31:59 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 501.477222ms
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [.....X......................................................] duration:
548.740291ms
FAIL (2025-06-26 06:31:48 UTC)
orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded
Duration: 548.766045ms
core
uplink
us-select-1
us1-placement-20
|
checkers.AuthServiceHTTP{"https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us1)antiblip:
1 out of 60 is failed (threshold=3) [....X.......................................................] duration:
174.721761ms
FAIL (2025-06-26 06:31:46 UTC)
orbiter warning: unexpected status code "500 Internal Server Error" "\n"
Duration: 174.753329ms
edge
authservice
http
bgp
us1
|
checkers.AuthServiceHTTP{"https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us1)antiblip:
1 out of 60 is failed (threshold=3) [.....X......................................................] duration:
209.969273ms
FAIL (2025-06-26 06:31:25 UTC)
orbiter warning: unexpected status code "500 Internal Server Error" "\n"
Duration: 210.001565ms
edge
authservice
http
bgp
us1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
3 out of 60 is failed (threshold=3) [............................XXX.............................] duration:
483.255859ms
FAIL (2025-06-26 06:31:23 UTC)
orbiter warning:
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 7)
======LOGS======
pload=55354]: Got next segment. Inline: false
Jun 26 06:08:21.105827735 (68h56m48.032122411s) [upload=55354]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jun 26 06:08:21.105833588 (68h56m48.032128274s) [upload=55354]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jun 26 06:08:21.135318313 (68h56m48.061613009s) [seg_pos={0 0}, upload=55354]: Begin upload segment...
Jun 26 06:08:21.135366534 (68h56m48.061661214s) [seg_pos={0 0}, upload=55354]: Done begin upload segment.
Jun 26 06:08:21.135390342 (68h56m48.061685028s) [noise=true , address=45.91.116.9:5261 , piece_id=CZWB3XJVA7B7R7GWFYNW3RCGNVRXQ3E37XDXD4SUXTEMQFZ73HHA, seg_pos={0 0}, upload=55354]: Uploading piece...
Jun 26 06:08:21.135405801 (68h56m48.061700497s) [noise=true , address=104.36.82.6:4873 , piece_id=TIIMEQUSDLRFF3OK3WBB74F7SWJ5MEC4Z6IQNELDMRZLNTMQT7LA, seg_pos={0 0}, upload=55354]: Uploading piece...
Jun 26 06:08:21.135774729 (68h56m48.062069418s) [noise=true , address=77.90.31.13:4626 , piece_id=MXYBIEDFAFK53CVTE2C6CKEZWOL3MLW4S7LOHK4IHW6QEIZJL76Q, seg_pos={0 0}, upload=55354]: Uploading piece...
Jun 26 06:08:21.135944436 (68h56m48.06223913s) [noise=true , address=77.90.31.10:4517 , piece_id=VJZ5T4QNVAPNUHNFNK24S6TAS4V4NKNFMQE2FZMCWWLCCB2ELXXQ, seg_pos={0 0}, upload=55354]: Uploading piece...
Jun 26 06:08:21.136080402 (68h56m48.062375094s) [noise=true , address=77.90.31.15:4697 , piece_id=CUXGVL7PS3H7ZQKUCSYXJCBR3ETYFDPZFYIEAWEYL7C2S3RJY3AA, seg_pos={0 0}, upload=55354]: Uploading piece...
Jun 26 06:08:21.136382603 (68h56m48.062677289s) [noise=true , address=104.36.82.13:4186 , piece_id=5SVABN3KS6AUL6HILHNQCVF74BA75VPKTTAK7IQAPKRRME3MOZEA, seg_pos={0 0}, upload=55354]: Uploading piece...
Jun 26 06:08:21.136399304 (68h56m48.062693995s) [noise=true , address=104.36.82.14:4223 , piece_id=LP74YFI2SRSYPYEIKREVINCOFMVCXLNDS2555YKUVFN63N6TBJDA, seg_pos={0 0}, upload=55354]: Uploading piece...
Jun 26 06:08:21.136669576 (68h56m48.062964262s) [noise=true , address=104.36.82.6:4876 , piece_id=SSMGONONY34EHF5W6QMKO7YXB7RLO5LZVTJMMFKTP6GQNGYFYQ4A, seg_pos={0 0}, upload=55354]: Uploading piece...
Jun 26 06:08:21.136809784 (68h56m48.063104477s) [noise=true , address=104.36.82.15:4258 , piece_id=YXS4J7YSG6JV33JG4BZLK5AXKE3JXLBTGZJXPXZQ4LBYBT3MU2AA, seg_pos={0 0}, upload=55354]: Uploading piece...
Jun 26 06:08:21.136950556 (68h56m48.063245242s) [noise=true , address=77.90.31.11:4554 , piece_id=R2CIFLWS4LXTGSDSIMIRC4UYVJTXC7T35HPWZ2S7VYUTJ2AMY3BQ, seg_pos={0 0}, upload=55354]: Uploading piece...
Jun 26 06:08:21.137226497 (68h56m48.06352119s) [noise=true , address=77.90.31.14:4661 , piece_id=VXI2ZUSGR4N3H7TPIK75RIDFP3SBAOWVRGHNXTZOHFT5FNG4JWJA, seg_pos={0 0}, upload=55354]: Uploading piece...
Jun 26 06:08:21.138099107 (68h56m48.064393792s) [noise=true , address=45.91.116.10:4082 , piece_id=7NVEGA4QQOE7KYPHXUUX42SJPQL6LP46RGM37VD6A2AWN7PAOM3Q, seg_pos={0 0}, upload=55354]: Uploading piece...
Jun 26 06:08:21.139330448 (68h56m48.065625141s) [noise=true , address=104.36.82.14:4222 , piece_id=U3MBMAKFBFTOPJZ3HS3FXLPDBLQTN577XNCZLUBHRO6IU4BE5VNA, seg_pos={0 0}, upload=55354]: Uploading piece...
Jun 26 06:08:21.140580734 (68h56m48.06687543s) [noise=true , address=77.90.31.19:4842 , piece_id=G3AG375DATHB7MGEEWONE5GMMIC7QHG2VVDKF5AEVOSXV5N4S6AA, seg_pos={0 0}, upload=55354]: Uploading piece...
Jun 26 06:08:21.141452300 (68h56m48.067747027s) [noise=true , address=45.91.116.13:4154 , piece_id=STFPXN2LUJYUIAYROWZIRTCMT2Y3XET6KWA4GZWUHENXRJURP62Q, seg_pos={0 0}, upload=55354]: Uploading piece...
Jun 26 06:08:21.142917062 (68h56m48.069211739s) [noise=true , address=104.36.82.7:4912 , piece_id=A5HPC7CEU66IFNS47HO3AQX22LUZ6AZ4L4XAMZ6T5NCVE4D3TAZA, seg_pos={0 0}, upload=55354]: Uploading piece...
Jun 26 06:08:21.221824189 (68h56m48.148118865s) [upload=55354]: Next returned nil segment
Jun 26 06:08:21.221837958 (68h56m48.148132644s) [upload=55354]: Waiting for error group managing segments...
Jun 26 06:08:21.229550628 (68h56m48.155845315s) [noise=true , address=104.36.82.14:4223 , piece_id=LP74YFI2SRSYPYEIKREVINCOFMVCXLNDS2555YKUVFN63N6TBJDA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: <nil>
Jun 26 06:08:21.230245145 (68h56m48.156539832s) [noise=true , address=104.36.82.14:4222 , piece_id=U3MBMAKFBFTOPJZ3HS3FXLPDBLQTN577XNCZLUBHRO6IU4BE5VNA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: <nil>
Jun 26 06:08:21.230801067 (68h56m48.15709574s) [noise=true , address=104.36.82.6:4873 , piece_id=TIIMEQUSDLRFF3OK3WBB74F7SWJ5MEC4Z6IQNELDMRZLNTMQT7LA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: <nil>
Jun 26 06:08:21.231457382 (68h56m48.157752071s) [noise=true , address=104.36.82.6:4876 , piece_id=SSMGONONY34EHF5W6QMKO7YXB7RLO5LZVTJMMFKTP6GQNGYFYQ4A, seg_pos={0 0}, upload=55354]: Done uploading piece. err: <nil>
Jun 26 06:08:21.231563205 (68h56m48.157857893s) [noise=true , address=104.36.82.13:4186 , piece_id=5SVABN3KS6AUL6HILHNQCVF74BA75VPKTTAK7IQAPKRRME3MOZEA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: <nil>
Jun 26 06:08:21.239131119 (68h56m48.165425799s) [noise=true , address=104.36.82.7:4912 , piece_id=A5HPC7CEU66IFNS47HO3AQX22LUZ6AZ4L4XAMZ6T5NCVE4D3TAZA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: <nil>
Jun 26 06:08:21.534686665 (68h56m48.460981337s) [noise=true , address=104.36.82.15:4258 , piece_id=YXS4J7YSG6JV33JG4BZLK5AXKE3JXLBTGZJXPXZQ4LBYBT3MU2AA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: <nil>
Jun 26 06:08:51.073673533 (68h57m17.999968219s) [noise=true , address=45.91.116.9:5261 , piece_id=CZWB3XJVA7B7R7GWFYNW3RCGNVRXQ3E37XDXD4SUXTEMQFZ73HHA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi, address:45.91.116.9:5261): protocol: rpc: dial tcp 45.91.116.9:5261: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jun 26 06:08:51.073739594 (68h57m18.000034272s) [noise=true , address=45.91.116.10:4082 , piece_id=7NVEGA4QQOE7KYPHXUUX42SJPQL6LP46RGM37VD6A2AWN7PAOM3Q, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj, address:45.91.116.10:4082): protocol: rpc: dial tcp 45.91.116.10:4082: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jun 26 06:08:51.073789897 (68h57m18.000084584s) [noise=true , address=77.90.31.19:4842 , piece_id=G3AG375DATHB7MGEEWONE5GMMIC7QHG2VVDKF5AEVOSXV5N4S6AA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: rpc: dial tcp 77.90.31.19:4842: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jun 26 06:08:51.073849570 (68h57m18.000144246s) [noise=true , address=45.91.116.13:4154 , piece_id=STFPXN2LUJYUIAYROWZIRTCMT2Y3XET6KWA4GZWUHENXRJURP62Q, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN, address:45.91.116.13:4154): protocol: write tcp 159.223.143.39:35392->45.91.116.13:4154: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jun 26 06:08:51.073904716 (68h57m18.000199391s) [noise=true , address=77.90.31.13:4626 , piece_id=MXYBIEDFAFK53CVTE2C6CKEZWOL3MLW4S7LOHK4IHW6QEIZJL76Q, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: write tcp 159.223.143.39:58864->77.90.31.13:4626: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jun 26 06:08:51.073952402 (68h57m18.000247082s) [noise=true , address=77.90.31.10:4517 , piece_id=VJZ5T4QNVAPNUHNFNK24S6TAS4V4NKNFMQE2FZMCWWLCCB2ELXXQ, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, address:77.90.31.10:4517): protocol: write tcp 159.223.143.39:36382->77.90.31.10:4517: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jun 26 06:08:51.074022481 (68h57m18.000317156s) [noise=true , address=77.90.31.15:4697 , piece_id=CUXGVL7PS3H7ZQKUCSYXJCBR3ETYFDPZFYIEAWEYL7C2S3RJY3AA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5, address:77.90.31.15:4697): protocol: write tcp 159.223.143.39:52730->77.90.31.15:4697: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jun 26 06:08:51.074071824 (68h57m18.000366503s) [noise=true , address=77.90.31.11:4554 , piece_id=R2CIFLWS4LXTGSDSIMIRC4UYVJTXC7T35HPWZ2S7VYUTJ2AMY3BQ, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, address:77.90.31.11:4554): protocol: rpc: dial tcp 77.90.31.11:4554: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jun 26 06:08:51.074102177 (68h57m18.000396855s) [noise=true , address=77.90.31.14:4661 , piece_id=VXI2ZUSGR4N3H7TPIK75RIDFP3SBAOWVRGHNXTZOHFT5FNG4JWJA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue, address:77.90.31.14:4661): protocol: rpc: dial tcp 77.90.31.14:4661: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jun 26 06:08:51.074119276 (68h57m18.000413966s) [seg_pos={0 0}, upload=55354]: Done waiting for segment. successful: 7 optimal: 13 errs: <nil>
Jun 26 06:08:51.100079867 (68h57m18.026374541s) [upload=55354]: Done uploading segments...
Jun 26 06:08:51.100114505 (68h57m18.026409191s) [upload=55354]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 7)
Duration: 483.287382ms
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [............................XX..............................] duration:
506.797923ms
FAIL (2025-06-26 06:31:08 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 506.822259ms
edge
gateway
us-select-1
|
checkers.AuthServiceHTTP{"https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us1)antiblip:
1 out of 60 is failed (threshold=3) [......X.....................................................] duration:
175.542841ms
FAIL (2025-06-26 06:30:58 UTC)
orbiter warning: unexpected status code "500 Internal Server Error" "\n"
Duration: 175.566926ms
edge
authservice
http
bgp
us1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
3 out of 60 is failed (threshold=3) [.............................XXX............................] duration:
426.73208ms
FAIL (2025-06-26 06:30:38 UTC)
orbiter warning:
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 7)
======LOGS======
pload=55354]: Got next segment. Inline: false
Jun 26 06:08:21.105827735 (68h56m48.032122411s) [upload=55354]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jun 26 06:08:21.105833588 (68h56m48.032128274s) [upload=55354]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jun 26 06:08:21.135318313 (68h56m48.061613009s) [seg_pos={0 0}, upload=55354]: Begin upload segment...
Jun 26 06:08:21.135366534 (68h56m48.061661214s) [seg_pos={0 0}, upload=55354]: Done begin upload segment.
Jun 26 06:08:21.135390342 (68h56m48.061685028s) [noise=true , address=45.91.116.9:5261 , piece_id=CZWB3XJVA7B7R7GWFYNW3RCGNVRXQ3E37XDXD4SUXTEMQFZ73HHA, seg_pos={0 0}, upload=55354]: Uploading piece...
Jun 26 06:08:21.135405801 (68h56m48.061700497s) [noise=true , address=104.36.82.6:4873 , piece_id=TIIMEQUSDLRFF3OK3WBB74F7SWJ5MEC4Z6IQNELDMRZLNTMQT7LA, seg_pos={0 0}, upload=55354]: Uploading piece...
Jun 26 06:08:21.135774729 (68h56m48.062069418s) [noise=true , address=77.90.31.13:4626 , piece_id=MXYBIEDFAFK53CVTE2C6CKEZWOL3MLW4S7LOHK4IHW6QEIZJL76Q, seg_pos={0 0}, upload=55354]: Uploading piece...
Jun 26 06:08:21.135944436 (68h56m48.06223913s) [noise=true , address=77.90.31.10:4517 , piece_id=VJZ5T4QNVAPNUHNFNK24S6TAS4V4NKNFMQE2FZMCWWLCCB2ELXXQ, seg_pos={0 0}, upload=55354]: Uploading piece...
Jun 26 06:08:21.136080402 (68h56m48.062375094s) [noise=true , address=77.90.31.15:4697 , piece_id=CUXGVL7PS3H7ZQKUCSYXJCBR3ETYFDPZFYIEAWEYL7C2S3RJY3AA, seg_pos={0 0}, upload=55354]: Uploading piece...
Jun 26 06:08:21.136382603 (68h56m48.062677289s) [noise=true , address=104.36.82.13:4186 , piece_id=5SVABN3KS6AUL6HILHNQCVF74BA75VPKTTAK7IQAPKRRME3MOZEA, seg_pos={0 0}, upload=55354]: Uploading piece...
Jun 26 06:08:21.136399304 (68h56m48.062693995s) [noise=true , address=104.36.82.14:4223 , piece_id=LP74YFI2SRSYPYEIKREVINCOFMVCXLNDS2555YKUVFN63N6TBJDA, seg_pos={0 0}, upload=55354]: Uploading piece...
Jun 26 06:08:21.136669576 (68h56m48.062964262s) [noise=true , address=104.36.82.6:4876 , piece_id=SSMGONONY34EHF5W6QMKO7YXB7RLO5LZVTJMMFKTP6GQNGYFYQ4A, seg_pos={0 0}, upload=55354]: Uploading piece...
Jun 26 06:08:21.136809784 (68h56m48.063104477s) [noise=true , address=104.36.82.15:4258 , piece_id=YXS4J7YSG6JV33JG4BZLK5AXKE3JXLBTGZJXPXZQ4LBYBT3MU2AA, seg_pos={0 0}, upload=55354]: Uploading piece...
Jun 26 06:08:21.136950556 (68h56m48.063245242s) [noise=true , address=77.90.31.11:4554 , piece_id=R2CIFLWS4LXTGSDSIMIRC4UYVJTXC7T35HPWZ2S7VYUTJ2AMY3BQ, seg_pos={0 0}, upload=55354]: Uploading piece...
Jun 26 06:08:21.137226497 (68h56m48.06352119s) [noise=true , address=77.90.31.14:4661 , piece_id=VXI2ZUSGR4N3H7TPIK75RIDFP3SBAOWVRGHNXTZOHFT5FNG4JWJA, seg_pos={0 0}, upload=55354]: Uploading piece...
Jun 26 06:08:21.138099107 (68h56m48.064393792s) [noise=true , address=45.91.116.10:4082 , piece_id=7NVEGA4QQOE7KYPHXUUX42SJPQL6LP46RGM37VD6A2AWN7PAOM3Q, seg_pos={0 0}, upload=55354]: Uploading piece...
Jun 26 06:08:21.139330448 (68h56m48.065625141s) [noise=true , address=104.36.82.14:4222 , piece_id=U3MBMAKFBFTOPJZ3HS3FXLPDBLQTN577XNCZLUBHRO6IU4BE5VNA, seg_pos={0 0}, upload=55354]: Uploading piece...
Jun 26 06:08:21.140580734 (68h56m48.06687543s) [noise=true , address=77.90.31.19:4842 , piece_id=G3AG375DATHB7MGEEWONE5GMMIC7QHG2VVDKF5AEVOSXV5N4S6AA, seg_pos={0 0}, upload=55354]: Uploading piece...
Jun 26 06:08:21.141452300 (68h56m48.067747027s) [noise=true , address=45.91.116.13:4154 , piece_id=STFPXN2LUJYUIAYROWZIRTCMT2Y3XET6KWA4GZWUHENXRJURP62Q, seg_pos={0 0}, upload=55354]: Uploading piece...
Jun 26 06:08:21.142917062 (68h56m48.069211739s) [noise=true , address=104.36.82.7:4912 , piece_id=A5HPC7CEU66IFNS47HO3AQX22LUZ6AZ4L4XAMZ6T5NCVE4D3TAZA, seg_pos={0 0}, upload=55354]: Uploading piece...
Jun 26 06:08:21.221824189 (68h56m48.148118865s) [upload=55354]: Next returned nil segment
Jun 26 06:08:21.221837958 (68h56m48.148132644s) [upload=55354]: Waiting for error group managing segments...
Jun 26 06:08:21.229550628 (68h56m48.155845315s) [noise=true , address=104.36.82.14:4223 , piece_id=LP74YFI2SRSYPYEIKREVINCOFMVCXLNDS2555YKUVFN63N6TBJDA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: <nil>
Jun 26 06:08:21.230245145 (68h56m48.156539832s) [noise=true , address=104.36.82.14:4222 , piece_id=U3MBMAKFBFTOPJZ3HS3FXLPDBLQTN577XNCZLUBHRO6IU4BE5VNA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: <nil>
Jun 26 06:08:21.230801067 (68h56m48.15709574s) [noise=true , address=104.36.82.6:4873 , piece_id=TIIMEQUSDLRFF3OK3WBB74F7SWJ5MEC4Z6IQNELDMRZLNTMQT7LA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: <nil>
Jun 26 06:08:21.231457382 (68h56m48.157752071s) [noise=true , address=104.36.82.6:4876 , piece_id=SSMGONONY34EHF5W6QMKO7YXB7RLO5LZVTJMMFKTP6GQNGYFYQ4A, seg_pos={0 0}, upload=55354]: Done uploading piece. err: <nil>
Jun 26 06:08:21.231563205 (68h56m48.157857893s) [noise=true , address=104.36.82.13:4186 , piece_id=5SVABN3KS6AUL6HILHNQCVF74BA75VPKTTAK7IQAPKRRME3MOZEA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: <nil>
Jun 26 06:08:21.239131119 (68h56m48.165425799s) [noise=true , address=104.36.82.7:4912 , piece_id=A5HPC7CEU66IFNS47HO3AQX22LUZ6AZ4L4XAMZ6T5NCVE4D3TAZA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: <nil>
Jun 26 06:08:21.534686665 (68h56m48.460981337s) [noise=true , address=104.36.82.15:4258 , piece_id=YXS4J7YSG6JV33JG4BZLK5AXKE3JXLBTGZJXPXZQ4LBYBT3MU2AA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: <nil>
Jun 26 06:08:51.073673533 (68h57m17.999968219s) [noise=true , address=45.91.116.9:5261 , piece_id=CZWB3XJVA7B7R7GWFYNW3RCGNVRXQ3E37XDXD4SUXTEMQFZ73HHA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi, address:45.91.116.9:5261): protocol: rpc: dial tcp 45.91.116.9:5261: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jun 26 06:08:51.073739594 (68h57m18.000034272s) [noise=true , address=45.91.116.10:4082 , piece_id=7NVEGA4QQOE7KYPHXUUX42SJPQL6LP46RGM37VD6A2AWN7PAOM3Q, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj, address:45.91.116.10:4082): protocol: rpc: dial tcp 45.91.116.10:4082: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jun 26 06:08:51.073789897 (68h57m18.000084584s) [noise=true , address=77.90.31.19:4842 , piece_id=G3AG375DATHB7MGEEWONE5GMMIC7QHG2VVDKF5AEVOSXV5N4S6AA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: rpc: dial tcp 77.90.31.19:4842: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jun 26 06:08:51.073849570 (68h57m18.000144246s) [noise=true , address=45.91.116.13:4154 , piece_id=STFPXN2LUJYUIAYROWZIRTCMT2Y3XET6KWA4GZWUHENXRJURP62Q, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN, address:45.91.116.13:4154): protocol: write tcp 159.223.143.39:35392->45.91.116.13:4154: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jun 26 06:08:51.073904716 (68h57m18.000199391s) [noise=true , address=77.90.31.13:4626 , piece_id=MXYBIEDFAFK53CVTE2C6CKEZWOL3MLW4S7LOHK4IHW6QEIZJL76Q, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: write tcp 159.223.143.39:58864->77.90.31.13:4626: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jun 26 06:08:51.073952402 (68h57m18.000247082s) [noise=true , address=77.90.31.10:4517 , piece_id=VJZ5T4QNVAPNUHNFNK24S6TAS4V4NKNFMQE2FZMCWWLCCB2ELXXQ, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, address:77.90.31.10:4517): protocol: write tcp 159.223.143.39:36382->77.90.31.10:4517: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jun 26 06:08:51.074022481 (68h57m18.000317156s) [noise=true , address=77.90.31.15:4697 , piece_id=CUXGVL7PS3H7ZQKUCSYXJCBR3ETYFDPZFYIEAWEYL7C2S3RJY3AA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5, address:77.90.31.15:4697): protocol: write tcp 159.223.143.39:52730->77.90.31.15:4697: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jun 26 06:08:51.074071824 (68h57m18.000366503s) [noise=true , address=77.90.31.11:4554 , piece_id=R2CIFLWS4LXTGSDSIMIRC4UYVJTXC7T35HPWZ2S7VYUTJ2AMY3BQ, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, address:77.90.31.11:4554): protocol: rpc: dial tcp 77.90.31.11:4554: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jun 26 06:08:51.074102177 (68h57m18.000396855s) [noise=true , address=77.90.31.14:4661 , piece_id=VXI2ZUSGR4N3H7TPIK75RIDFP3SBAOWVRGHNXTZOHFT5FNG4JWJA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue, address:77.90.31.14:4661): protocol: rpc: dial tcp 77.90.31.14:4661: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jun 26 06:08:51.074119276 (68h57m18.000413966s) [seg_pos={0 0}, upload=55354]: Done waiting for segment. successful: 7 optimal: 13 errs: <nil>
Jun 26 06:08:51.100079867 (68h57m18.026374541s) [upload=55354]: Done uploading segments...
Jun 26 06:08:51.100114505 (68h57m18.026409191s) [upload=55354]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 7)
Duration: 426.759857ms
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [......X.....................................................] duration:
713.647742ms
FAIL (2025-06-26 06:30:38 UTC)
orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded
Duration: 713.66681ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
3 out of 60 is failed (threshold=3) [..............................XXX...........................] duration:
588.73312ms
FAIL (2025-06-26 06:30:15 UTC)
orbiter warning:
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 7)
======LOGS======
pload=55354]: Got next segment. Inline: false
Jun 26 06:08:21.105827735 (68h56m48.032122411s) [upload=55354]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jun 26 06:08:21.105833588 (68h56m48.032128274s) [upload=55354]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jun 26 06:08:21.135318313 (68h56m48.061613009s) [seg_pos={0 0}, upload=55354]: Begin upload segment...
Jun 26 06:08:21.135366534 (68h56m48.061661214s) [seg_pos={0 0}, upload=55354]: Done begin upload segment.
Jun 26 06:08:21.135390342 (68h56m48.061685028s) [noise=true , address=45.91.116.9:5261 , piece_id=CZWB3XJVA7B7R7GWFYNW3RCGNVRXQ3E37XDXD4SUXTEMQFZ73HHA, seg_pos={0 0}, upload=55354]: Uploading piece...
Jun 26 06:08:21.135405801 (68h56m48.061700497s) [noise=true , address=104.36.82.6:4873 , piece_id=TIIMEQUSDLRFF3OK3WBB74F7SWJ5MEC4Z6IQNELDMRZLNTMQT7LA, seg_pos={0 0}, upload=55354]: Uploading piece...
Jun 26 06:08:21.135774729 (68h56m48.062069418s) [noise=true , address=77.90.31.13:4626 , piece_id=MXYBIEDFAFK53CVTE2C6CKEZWOL3MLW4S7LOHK4IHW6QEIZJL76Q, seg_pos={0 0}, upload=55354]: Uploading piece...
Jun 26 06:08:21.135944436 (68h56m48.06223913s) [noise=true , address=77.90.31.10:4517 , piece_id=VJZ5T4QNVAPNUHNFNK24S6TAS4V4NKNFMQE2FZMCWWLCCB2ELXXQ, seg_pos={0 0}, upload=55354]: Uploading piece...
Jun 26 06:08:21.136080402 (68h56m48.062375094s) [noise=true , address=77.90.31.15:4697 , piece_id=CUXGVL7PS3H7ZQKUCSYXJCBR3ETYFDPZFYIEAWEYL7C2S3RJY3AA, seg_pos={0 0}, upload=55354]: Uploading piece...
Jun 26 06:08:21.136382603 (68h56m48.062677289s) [noise=true , address=104.36.82.13:4186 , piece_id=5SVABN3KS6AUL6HILHNQCVF74BA75VPKTTAK7IQAPKRRME3MOZEA, seg_pos={0 0}, upload=55354]: Uploading piece...
Jun 26 06:08:21.136399304 (68h56m48.062693995s) [noise=true , address=104.36.82.14:4223 , piece_id=LP74YFI2SRSYPYEIKREVINCOFMVCXLNDS2555YKUVFN63N6TBJDA, seg_pos={0 0}, upload=55354]: Uploading piece...
Jun 26 06:08:21.136669576 (68h56m48.062964262s) [noise=true , address=104.36.82.6:4876 , piece_id=SSMGONONY34EHF5W6QMKO7YXB7RLO5LZVTJMMFKTP6GQNGYFYQ4A, seg_pos={0 0}, upload=55354]: Uploading piece...
Jun 26 06:08:21.136809784 (68h56m48.063104477s) [noise=true , address=104.36.82.15:4258 , piece_id=YXS4J7YSG6JV33JG4BZLK5AXKE3JXLBTGZJXPXZQ4LBYBT3MU2AA, seg_pos={0 0}, upload=55354]: Uploading piece...
Jun 26 06:08:21.136950556 (68h56m48.063245242s) [noise=true , address=77.90.31.11:4554 , piece_id=R2CIFLWS4LXTGSDSIMIRC4UYVJTXC7T35HPWZ2S7VYUTJ2AMY3BQ, seg_pos={0 0}, upload=55354]: Uploading piece...
Jun 26 06:08:21.137226497 (68h56m48.06352119s) [noise=true , address=77.90.31.14:4661 , piece_id=VXI2ZUSGR4N3H7TPIK75RIDFP3SBAOWVRGHNXTZOHFT5FNG4JWJA, seg_pos={0 0}, upload=55354]: Uploading piece...
Jun 26 06:08:21.138099107 (68h56m48.064393792s) [noise=true , address=45.91.116.10:4082 , piece_id=7NVEGA4QQOE7KYPHXUUX42SJPQL6LP46RGM37VD6A2AWN7PAOM3Q, seg_pos={0 0}, upload=55354]: Uploading piece...
Jun 26 06:08:21.139330448 (68h56m48.065625141s) [noise=true , address=104.36.82.14:4222 , piece_id=U3MBMAKFBFTOPJZ3HS3FXLPDBLQTN577XNCZLUBHRO6IU4BE5VNA, seg_pos={0 0}, upload=55354]: Uploading piece...
Jun 26 06:08:21.140580734 (68h56m48.06687543s) [noise=true , address=77.90.31.19:4842 , piece_id=G3AG375DATHB7MGEEWONE5GMMIC7QHG2VVDKF5AEVOSXV5N4S6AA, seg_pos={0 0}, upload=55354]: Uploading piece...
Jun 26 06:08:21.141452300 (68h56m48.067747027s) [noise=true , address=45.91.116.13:4154 , piece_id=STFPXN2LUJYUIAYROWZIRTCMT2Y3XET6KWA4GZWUHENXRJURP62Q, seg_pos={0 0}, upload=55354]: Uploading piece...
Jun 26 06:08:21.142917062 (68h56m48.069211739s) [noise=true , address=104.36.82.7:4912 , piece_id=A5HPC7CEU66IFNS47HO3AQX22LUZ6AZ4L4XAMZ6T5NCVE4D3TAZA, seg_pos={0 0}, upload=55354]: Uploading piece...
Jun 26 06:08:21.221824189 (68h56m48.148118865s) [upload=55354]: Next returned nil segment
Jun 26 06:08:21.221837958 (68h56m48.148132644s) [upload=55354]: Waiting for error group managing segments...
Jun 26 06:08:21.229550628 (68h56m48.155845315s) [noise=true , address=104.36.82.14:4223 , piece_id=LP74YFI2SRSYPYEIKREVINCOFMVCXLNDS2555YKUVFN63N6TBJDA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: <nil>
Jun 26 06:08:21.230245145 (68h56m48.156539832s) [noise=true , address=104.36.82.14:4222 , piece_id=U3MBMAKFBFTOPJZ3HS3FXLPDBLQTN577XNCZLUBHRO6IU4BE5VNA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: <nil>
Jun 26 06:08:21.230801067 (68h56m48.15709574s) [noise=true , address=104.36.82.6:4873 , piece_id=TIIMEQUSDLRFF3OK3WBB74F7SWJ5MEC4Z6IQNELDMRZLNTMQT7LA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: <nil>
Jun 26 06:08:21.231457382 (68h56m48.157752071s) [noise=true , address=104.36.82.6:4876 , piece_id=SSMGONONY34EHF5W6QMKO7YXB7RLO5LZVTJMMFKTP6GQNGYFYQ4A, seg_pos={0 0}, upload=55354]: Done uploading piece. err: <nil>
Jun 26 06:08:21.231563205 (68h56m48.157857893s) [noise=true , address=104.36.82.13:4186 , piece_id=5SVABN3KS6AUL6HILHNQCVF74BA75VPKTTAK7IQAPKRRME3MOZEA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: <nil>
Jun 26 06:08:21.239131119 (68h56m48.165425799s) [noise=true , address=104.36.82.7:4912 , piece_id=A5HPC7CEU66IFNS47HO3AQX22LUZ6AZ4L4XAMZ6T5NCVE4D3TAZA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: <nil>
Jun 26 06:08:21.534686665 (68h56m48.460981337s) [noise=true , address=104.36.82.15:4258 , piece_id=YXS4J7YSG6JV33JG4BZLK5AXKE3JXLBTGZJXPXZQ4LBYBT3MU2AA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: <nil>
Jun 26 06:08:51.073673533 (68h57m17.999968219s) [noise=true , address=45.91.116.9:5261 , piece_id=CZWB3XJVA7B7R7GWFYNW3RCGNVRXQ3E37XDXD4SUXTEMQFZ73HHA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi, address:45.91.116.9:5261): protocol: rpc: dial tcp 45.91.116.9:5261: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jun 26 06:08:51.073739594 (68h57m18.000034272s) [noise=true , address=45.91.116.10:4082 , piece_id=7NVEGA4QQOE7KYPHXUUX42SJPQL6LP46RGM37VD6A2AWN7PAOM3Q, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj, address:45.91.116.10:4082): protocol: rpc: dial tcp 45.91.116.10:4082: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jun 26 06:08:51.073789897 (68h57m18.000084584s) [noise=true , address=77.90.31.19:4842 , piece_id=G3AG375DATHB7MGEEWONE5GMMIC7QHG2VVDKF5AEVOSXV5N4S6AA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: rpc: dial tcp 77.90.31.19:4842: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jun 26 06:08:51.073849570 (68h57m18.000144246s) [noise=true , address=45.91.116.13:4154 , piece_id=STFPXN2LUJYUIAYROWZIRTCMT2Y3XET6KWA4GZWUHENXRJURP62Q, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN, address:45.91.116.13:4154): protocol: write tcp 159.223.143.39:35392->45.91.116.13:4154: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jun 26 06:08:51.073904716 (68h57m18.000199391s) [noise=true , address=77.90.31.13:4626 , piece_id=MXYBIEDFAFK53CVTE2C6CKEZWOL3MLW4S7LOHK4IHW6QEIZJL76Q, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: write tcp 159.223.143.39:58864->77.90.31.13:4626: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jun 26 06:08:51.073952402 (68h57m18.000247082s) [noise=true , address=77.90.31.10:4517 , piece_id=VJZ5T4QNVAPNUHNFNK24S6TAS4V4NKNFMQE2FZMCWWLCCB2ELXXQ, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, address:77.90.31.10:4517): protocol: write tcp 159.223.143.39:36382->77.90.31.10:4517: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jun 26 06:08:51.074022481 (68h57m18.000317156s) [noise=true , address=77.90.31.15:4697 , piece_id=CUXGVL7PS3H7ZQKUCSYXJCBR3ETYFDPZFYIEAWEYL7C2S3RJY3AA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5, address:77.90.31.15:4697): protocol: write tcp 159.223.143.39:52730->77.90.31.15:4697: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jun 26 06:08:51.074071824 (68h57m18.000366503s) [noise=true , address=77.90.31.11:4554 , piece_id=R2CIFLWS4LXTGSDSIMIRC4UYVJTXC7T35HPWZ2S7VYUTJ2AMY3BQ, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, address:77.90.31.11:4554): protocol: rpc: dial tcp 77.90.31.11:4554: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jun 26 06:08:51.074102177 (68h57m18.000396855s) [noise=true , address=77.90.31.14:4661 , piece_id=VXI2ZUSGR4N3H7TPIK75RIDFP3SBAOWVRGHNXTZOHFT5FNG4JWJA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue, address:77.90.31.14:4661): protocol: rpc: dial tcp 77.90.31.14:4661: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jun 26 06:08:51.074119276 (68h57m18.000413966s) [seg_pos={0 0}, upload=55354]: Done waiting for segment. successful: 7 optimal: 13 errs: <nil>
Jun 26 06:08:51.100079867 (68h57m18.026374541s) [upload=55354]: Done uploading segments...
Jun 26 06:08:51.100114505 (68h57m18.026409191s) [upload=55354]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 7)
Duration: 588.76917ms
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [.............................XX.............................] duration:
642.994363ms
FAIL (2025-06-26 06:30:03 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 643.021087ms
edge
gateway
us-select-1
|
checkers.AuthServiceHTTP{"https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us1)antiblip:
1 out of 60 is failed (threshold=3) [.......X....................................................] duration:
206.599568ms
FAIL (2025-06-26 06:29:49 UTC)
orbiter warning: unexpected status code "500 Internal Server Error" "\n"
Duration: 206.630758ms
edge
authservice
http
bgp
us1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [.......X....................................................] duration:
867.508083ms
FAIL (2025-06-26 06:29:32 UTC)
orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded
Duration: 867.530954ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
3 out of 60 is failed (threshold=3) [...............................XXX..........................] duration:
488.829092ms
FAIL (2025-06-26 06:29:16 UTC)
orbiter warning:
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 7)
======LOGS======
pload=55354]: Got next segment. Inline: false
Jun 26 06:08:21.105827735 (68h56m48.032122411s) [upload=55354]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jun 26 06:08:21.105833588 (68h56m48.032128274s) [upload=55354]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jun 26 06:08:21.135318313 (68h56m48.061613009s) [seg_pos={0 0}, upload=55354]: Begin upload segment...
Jun 26 06:08:21.135366534 (68h56m48.061661214s) [seg_pos={0 0}, upload=55354]: Done begin upload segment.
Jun 26 06:08:21.135390342 (68h56m48.061685028s) [noise=true , address=45.91.116.9:5261 , piece_id=CZWB3XJVA7B7R7GWFYNW3RCGNVRXQ3E37XDXD4SUXTEMQFZ73HHA, seg_pos={0 0}, upload=55354]: Uploading piece...
Jun 26 06:08:21.135405801 (68h56m48.061700497s) [noise=true , address=104.36.82.6:4873 , piece_id=TIIMEQUSDLRFF3OK3WBB74F7SWJ5MEC4Z6IQNELDMRZLNTMQT7LA, seg_pos={0 0}, upload=55354]: Uploading piece...
Jun 26 06:08:21.135774729 (68h56m48.062069418s) [noise=true , address=77.90.31.13:4626 , piece_id=MXYBIEDFAFK53CVTE2C6CKEZWOL3MLW4S7LOHK4IHW6QEIZJL76Q, seg_pos={0 0}, upload=55354]: Uploading piece...
Jun 26 06:08:21.135944436 (68h56m48.06223913s) [noise=true , address=77.90.31.10:4517 , piece_id=VJZ5T4QNVAPNUHNFNK24S6TAS4V4NKNFMQE2FZMCWWLCCB2ELXXQ, seg_pos={0 0}, upload=55354]: Uploading piece...
Jun 26 06:08:21.136080402 (68h56m48.062375094s) [noise=true , address=77.90.31.15:4697 , piece_id=CUXGVL7PS3H7ZQKUCSYXJCBR3ETYFDPZFYIEAWEYL7C2S3RJY3AA, seg_pos={0 0}, upload=55354]: Uploading piece...
Jun 26 06:08:21.136382603 (68h56m48.062677289s) [noise=true , address=104.36.82.13:4186 , piece_id=5SVABN3KS6AUL6HILHNQCVF74BA75VPKTTAK7IQAPKRRME3MOZEA, seg_pos={0 0}, upload=55354]: Uploading piece...
Jun 26 06:08:21.136399304 (68h56m48.062693995s) [noise=true , address=104.36.82.14:4223 , piece_id=LP74YFI2SRSYPYEIKREVINCOFMVCXLNDS2555YKUVFN63N6TBJDA, seg_pos={0 0}, upload=55354]: Uploading piece...
Jun 26 06:08:21.136669576 (68h56m48.062964262s) [noise=true , address=104.36.82.6:4876 , piece_id=SSMGONONY34EHF5W6QMKO7YXB7RLO5LZVTJMMFKTP6GQNGYFYQ4A, seg_pos={0 0}, upload=55354]: Uploading piece...
Jun 26 06:08:21.136809784 (68h56m48.063104477s) [noise=true , address=104.36.82.15:4258 , piece_id=YXS4J7YSG6JV33JG4BZLK5AXKE3JXLBTGZJXPXZQ4LBYBT3MU2AA, seg_pos={0 0}, upload=55354]: Uploading piece...
Jun 26 06:08:21.136950556 (68h56m48.063245242s) [noise=true , address=77.90.31.11:4554 , piece_id=R2CIFLWS4LXTGSDSIMIRC4UYVJTXC7T35HPWZ2S7VYUTJ2AMY3BQ, seg_pos={0 0}, upload=55354]: Uploading piece...
Jun 26 06:08:21.137226497 (68h56m48.06352119s) [noise=true , address=77.90.31.14:4661 , piece_id=VXI2ZUSGR4N3H7TPIK75RIDFP3SBAOWVRGHNXTZOHFT5FNG4JWJA, seg_pos={0 0}, upload=55354]: Uploading piece...
Jun 26 06:08:21.138099107 (68h56m48.064393792s) [noise=true , address=45.91.116.10:4082 , piece_id=7NVEGA4QQOE7KYPHXUUX42SJPQL6LP46RGM37VD6A2AWN7PAOM3Q, seg_pos={0 0}, upload=55354]: Uploading piece...
Jun 26 06:08:21.139330448 (68h56m48.065625141s) [noise=true , address=104.36.82.14:4222 , piece_id=U3MBMAKFBFTOPJZ3HS3FXLPDBLQTN577XNCZLUBHRO6IU4BE5VNA, seg_pos={0 0}, upload=55354]: Uploading piece...
Jun 26 06:08:21.140580734 (68h56m48.06687543s) [noise=true , address=77.90.31.19:4842 , piece_id=G3AG375DATHB7MGEEWONE5GMMIC7QHG2VVDKF5AEVOSXV5N4S6AA, seg_pos={0 0}, upload=55354]: Uploading piece...
Jun 26 06:08:21.141452300 (68h56m48.067747027s) [noise=true , address=45.91.116.13:4154 , piece_id=STFPXN2LUJYUIAYROWZIRTCMT2Y3XET6KWA4GZWUHENXRJURP62Q, seg_pos={0 0}, upload=55354]: Uploading piece...
Jun 26 06:08:21.142917062 (68h56m48.069211739s) [noise=true , address=104.36.82.7:4912 , piece_id=A5HPC7CEU66IFNS47HO3AQX22LUZ6AZ4L4XAMZ6T5NCVE4D3TAZA, seg_pos={0 0}, upload=55354]: Uploading piece...
Jun 26 06:08:21.221824189 (68h56m48.148118865s) [upload=55354]: Next returned nil segment
Jun 26 06:08:21.221837958 (68h56m48.148132644s) [upload=55354]: Waiting for error group managing segments...
Jun 26 06:08:21.229550628 (68h56m48.155845315s) [noise=true , address=104.36.82.14:4223 , piece_id=LP74YFI2SRSYPYEIKREVINCOFMVCXLNDS2555YKUVFN63N6TBJDA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: <nil>
Jun 26 06:08:21.230245145 (68h56m48.156539832s) [noise=true , address=104.36.82.14:4222 , piece_id=U3MBMAKFBFTOPJZ3HS3FXLPDBLQTN577XNCZLUBHRO6IU4BE5VNA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: <nil>
Jun 26 06:08:21.230801067 (68h56m48.15709574s) [noise=true , address=104.36.82.6:4873 , piece_id=TIIMEQUSDLRFF3OK3WBB74F7SWJ5MEC4Z6IQNELDMRZLNTMQT7LA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: <nil>
Jun 26 06:08:21.231457382 (68h56m48.157752071s) [noise=true , address=104.36.82.6:4876 , piece_id=SSMGONONY34EHF5W6QMKO7YXB7RLO5LZVTJMMFKTP6GQNGYFYQ4A, seg_pos={0 0}, upload=55354]: Done uploading piece. err: <nil>
Jun 26 06:08:21.231563205 (68h56m48.157857893s) [noise=true , address=104.36.82.13:4186 , piece_id=5SVABN3KS6AUL6HILHNQCVF74BA75VPKTTAK7IQAPKRRME3MOZEA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: <nil>
Jun 26 06:08:21.239131119 (68h56m48.165425799s) [noise=true , address=104.36.82.7:4912 , piece_id=A5HPC7CEU66IFNS47HO3AQX22LUZ6AZ4L4XAMZ6T5NCVE4D3TAZA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: <nil>
Jun 26 06:08:21.534686665 (68h56m48.460981337s) [noise=true , address=104.36.82.15:4258 , piece_id=YXS4J7YSG6JV33JG4BZLK5AXKE3JXLBTGZJXPXZQ4LBYBT3MU2AA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: <nil>
Jun 26 06:08:51.073673533 (68h57m17.999968219s) [noise=true , address=45.91.116.9:5261 , piece_id=CZWB3XJVA7B7R7GWFYNW3RCGNVRXQ3E37XDXD4SUXTEMQFZ73HHA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi, address:45.91.116.9:5261): protocol: rpc: dial tcp 45.91.116.9:5261: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jun 26 06:08:51.073739594 (68h57m18.000034272s) [noise=true , address=45.91.116.10:4082 , piece_id=7NVEGA4QQOE7KYPHXUUX42SJPQL6LP46RGM37VD6A2AWN7PAOM3Q, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj, address:45.91.116.10:4082): protocol: rpc: dial tcp 45.91.116.10:4082: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jun 26 06:08:51.073789897 (68h57m18.000084584s) [noise=true , address=77.90.31.19:4842 , piece_id=G3AG375DATHB7MGEEWONE5GMMIC7QHG2VVDKF5AEVOSXV5N4S6AA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: rpc: dial tcp 77.90.31.19:4842: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jun 26 06:08:51.073849570 (68h57m18.000144246s) [noise=true , address=45.91.116.13:4154 , piece_id=STFPXN2LUJYUIAYROWZIRTCMT2Y3XET6KWA4GZWUHENXRJURP62Q, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN, address:45.91.116.13:4154): protocol: write tcp 159.223.143.39:35392->45.91.116.13:4154: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jun 26 06:08:51.073904716 (68h57m18.000199391s) [noise=true , address=77.90.31.13:4626 , piece_id=MXYBIEDFAFK53CVTE2C6CKEZWOL3MLW4S7LOHK4IHW6QEIZJL76Q, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: write tcp 159.223.143.39:58864->77.90.31.13:4626: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jun 26 06:08:51.073952402 (68h57m18.000247082s) [noise=true , address=77.90.31.10:4517 , piece_id=VJZ5T4QNVAPNUHNFNK24S6TAS4V4NKNFMQE2FZMCWWLCCB2ELXXQ, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, address:77.90.31.10:4517): protocol: write tcp 159.223.143.39:36382->77.90.31.10:4517: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jun 26 06:08:51.074022481 (68h57m18.000317156s) [noise=true , address=77.90.31.15:4697 , piece_id=CUXGVL7PS3H7ZQKUCSYXJCBR3ETYFDPZFYIEAWEYL7C2S3RJY3AA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5, address:77.90.31.15:4697): protocol: write tcp 159.223.143.39:52730->77.90.31.15:4697: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jun 26 06:08:51.074071824 (68h57m18.000366503s) [noise=true , address=77.90.31.11:4554 , piece_id=R2CIFLWS4LXTGSDSIMIRC4UYVJTXC7T35HPWZ2S7VYUTJ2AMY3BQ, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, address:77.90.31.11:4554): protocol: rpc: dial tcp 77.90.31.11:4554: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jun 26 06:08:51.074102177 (68h57m18.000396855s) [noise=true , address=77.90.31.14:4661 , piece_id=VXI2ZUSGR4N3H7TPIK75RIDFP3SBAOWVRGHNXTZOHFT5FNG4JWJA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue, address:77.90.31.14:4661): protocol: rpc: dial tcp 77.90.31.14:4661: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jun 26 06:08:51.074119276 (68h57m18.000413966s) [seg_pos={0 0}, upload=55354]: Done waiting for segment. successful: 7 optimal: 13 errs: <nil>
Jun 26 06:08:51.100079867 (68h57m18.026374541s) [upload=55354]: Done uploading segments...
Jun 26 06:08:51.100114505 (68h57m18.026409191s) [upload=55354]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 7)
Duration: 488.860685ms
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [..............................XX............................] duration:
559.588373ms
FAIL (2025-06-26 06:28:59 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 559.612602ms
edge
gateway
us-select-1
|
checkers.AuthServiceHTTP{"https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us1)antiblip:
1 out of 60 is failed (threshold=3) [........X...................................................] duration:
182.761856ms
FAIL (2025-06-26 06:28:45 UTC)
orbiter warning: unexpected status code "500 Internal Server Error" "\n"
Duration: 182.792988ms
edge
authservice
http
bgp
us1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [........X...................................................] duration:
876.34063ms
FAIL (2025-06-26 06:28:18 UTC)
orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded
Duration: 876.389744ms
core
uplink
us-select-1
us1-placement-20
|
checkers.AuthServiceHTTP{"https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us1)antiblip:
1 out of 60 is failed (threshold=3) [.........X..................................................] duration:
545.415237ms
FAIL (2025-06-26 06:28:11 UTC)
orbiter warning: unexpected status code "500 Internal Server Error" "\n"
Duration: 545.443718ms
edge
authservice
http
bgp
us1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [...............................XX...........................] duration:
689.566148ms
FAIL (2025-06-26 06:28:07 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 689.706543ms
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
3 out of 60 is failed (threshold=3) [................................XXX.........................] duration:
604.806095ms
FAIL (2025-06-26 06:28:06 UTC)
orbiter warning:
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 7)
======LOGS======
pload=55354]: Got next segment. Inline: false
Jun 26 06:08:21.105827735 (68h56m48.032122411s) [upload=55354]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jun 26 06:08:21.105833588 (68h56m48.032128274s) [upload=55354]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jun 26 06:08:21.135318313 (68h56m48.061613009s) [seg_pos={0 0}, upload=55354]: Begin upload segment...
Jun 26 06:08:21.135366534 (68h56m48.061661214s) [seg_pos={0 0}, upload=55354]: Done begin upload segment.
Jun 26 06:08:21.135390342 (68h56m48.061685028s) [noise=true , address=45.91.116.9:5261 , piece_id=CZWB3XJVA7B7R7GWFYNW3RCGNVRXQ3E37XDXD4SUXTEMQFZ73HHA, seg_pos={0 0}, upload=55354]: Uploading piece...
Jun 26 06:08:21.135405801 (68h56m48.061700497s) [noise=true , address=104.36.82.6:4873 , piece_id=TIIMEQUSDLRFF3OK3WBB74F7SWJ5MEC4Z6IQNELDMRZLNTMQT7LA, seg_pos={0 0}, upload=55354]: Uploading piece...
Jun 26 06:08:21.135774729 (68h56m48.062069418s) [noise=true , address=77.90.31.13:4626 , piece_id=MXYBIEDFAFK53CVTE2C6CKEZWOL3MLW4S7LOHK4IHW6QEIZJL76Q, seg_pos={0 0}, upload=55354]: Uploading piece...
Jun 26 06:08:21.135944436 (68h56m48.06223913s) [noise=true , address=77.90.31.10:4517 , piece_id=VJZ5T4QNVAPNUHNFNK24S6TAS4V4NKNFMQE2FZMCWWLCCB2ELXXQ, seg_pos={0 0}, upload=55354]: Uploading piece...
Jun 26 06:08:21.136080402 (68h56m48.062375094s) [noise=true , address=77.90.31.15:4697 , piece_id=CUXGVL7PS3H7ZQKUCSYXJCBR3ETYFDPZFYIEAWEYL7C2S3RJY3AA, seg_pos={0 0}, upload=55354]: Uploading piece...
Jun 26 06:08:21.136382603 (68h56m48.062677289s) [noise=true , address=104.36.82.13:4186 , piece_id=5SVABN3KS6AUL6HILHNQCVF74BA75VPKTTAK7IQAPKRRME3MOZEA, seg_pos={0 0}, upload=55354]: Uploading piece...
Jun 26 06:08:21.136399304 (68h56m48.062693995s) [noise=true , address=104.36.82.14:4223 , piece_id=LP74YFI2SRSYPYEIKREVINCOFMVCXLNDS2555YKUVFN63N6TBJDA, seg_pos={0 0}, upload=55354]: Uploading piece...
Jun 26 06:08:21.136669576 (68h56m48.062964262s) [noise=true , address=104.36.82.6:4876 , piece_id=SSMGONONY34EHF5W6QMKO7YXB7RLO5LZVTJMMFKTP6GQNGYFYQ4A, seg_pos={0 0}, upload=55354]: Uploading piece...
Jun 26 06:08:21.136809784 (68h56m48.063104477s) [noise=true , address=104.36.82.15:4258 , piece_id=YXS4J7YSG6JV33JG4BZLK5AXKE3JXLBTGZJXPXZQ4LBYBT3MU2AA, seg_pos={0 0}, upload=55354]: Uploading piece...
Jun 26 06:08:21.136950556 (68h56m48.063245242s) [noise=true , address=77.90.31.11:4554 , piece_id=R2CIFLWS4LXTGSDSIMIRC4UYVJTXC7T35HPWZ2S7VYUTJ2AMY3BQ, seg_pos={0 0}, upload=55354]: Uploading piece...
Jun 26 06:08:21.137226497 (68h56m48.06352119s) [noise=true , address=77.90.31.14:4661 , piece_id=VXI2ZUSGR4N3H7TPIK75RIDFP3SBAOWVRGHNXTZOHFT5FNG4JWJA, seg_pos={0 0}, upload=55354]: Uploading piece...
Jun 26 06:08:21.138099107 (68h56m48.064393792s) [noise=true , address=45.91.116.10:4082 , piece_id=7NVEGA4QQOE7KYPHXUUX42SJPQL6LP46RGM37VD6A2AWN7PAOM3Q, seg_pos={0 0}, upload=55354]: Uploading piece...
Jun 26 06:08:21.139330448 (68h56m48.065625141s) [noise=true , address=104.36.82.14:4222 , piece_id=U3MBMAKFBFTOPJZ3HS3FXLPDBLQTN577XNCZLUBHRO6IU4BE5VNA, seg_pos={0 0}, upload=55354]: Uploading piece...
Jun 26 06:08:21.140580734 (68h56m48.06687543s) [noise=true , address=77.90.31.19:4842 , piece_id=G3AG375DATHB7MGEEWONE5GMMIC7QHG2VVDKF5AEVOSXV5N4S6AA, seg_pos={0 0}, upload=55354]: Uploading piece...
Jun 26 06:08:21.141452300 (68h56m48.067747027s) [noise=true , address=45.91.116.13:4154 , piece_id=STFPXN2LUJYUIAYROWZIRTCMT2Y3XET6KWA4GZWUHENXRJURP62Q, seg_pos={0 0}, upload=55354]: Uploading piece...
Jun 26 06:08:21.142917062 (68h56m48.069211739s) [noise=true , address=104.36.82.7:4912 , piece_id=A5HPC7CEU66IFNS47HO3AQX22LUZ6AZ4L4XAMZ6T5NCVE4D3TAZA, seg_pos={0 0}, upload=55354]: Uploading piece...
Jun 26 06:08:21.221824189 (68h56m48.148118865s) [upload=55354]: Next returned nil segment
Jun 26 06:08:21.221837958 (68h56m48.148132644s) [upload=55354]: Waiting for error group managing segments...
Jun 26 06:08:21.229550628 (68h56m48.155845315s) [noise=true , address=104.36.82.14:4223 , piece_id=LP74YFI2SRSYPYEIKREVINCOFMVCXLNDS2555YKUVFN63N6TBJDA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: <nil>
Jun 26 06:08:21.230245145 (68h56m48.156539832s) [noise=true , address=104.36.82.14:4222 , piece_id=U3MBMAKFBFTOPJZ3HS3FXLPDBLQTN577XNCZLUBHRO6IU4BE5VNA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: <nil>
Jun 26 06:08:21.230801067 (68h56m48.15709574s) [noise=true , address=104.36.82.6:4873 , piece_id=TIIMEQUSDLRFF3OK3WBB74F7SWJ5MEC4Z6IQNELDMRZLNTMQT7LA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: <nil>
Jun 26 06:08:21.231457382 (68h56m48.157752071s) [noise=true , address=104.36.82.6:4876 , piece_id=SSMGONONY34EHF5W6QMKO7YXB7RLO5LZVTJMMFKTP6GQNGYFYQ4A, seg_pos={0 0}, upload=55354]: Done uploading piece. err: <nil>
Jun 26 06:08:21.231563205 (68h56m48.157857893s) [noise=true , address=104.36.82.13:4186 , piece_id=5SVABN3KS6AUL6HILHNQCVF74BA75VPKTTAK7IQAPKRRME3MOZEA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: <nil>
Jun 26 06:08:21.239131119 (68h56m48.165425799s) [noise=true , address=104.36.82.7:4912 , piece_id=A5HPC7CEU66IFNS47HO3AQX22LUZ6AZ4L4XAMZ6T5NCVE4D3TAZA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: <nil>
Jun 26 06:08:21.534686665 (68h56m48.460981337s) [noise=true , address=104.36.82.15:4258 , piece_id=YXS4J7YSG6JV33JG4BZLK5AXKE3JXLBTGZJXPXZQ4LBYBT3MU2AA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: <nil>
Jun 26 06:08:51.073673533 (68h57m17.999968219s) [noise=true , address=45.91.116.9:5261 , piece_id=CZWB3XJVA7B7R7GWFYNW3RCGNVRXQ3E37XDXD4SUXTEMQFZ73HHA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi, address:45.91.116.9:5261): protocol: rpc: dial tcp 45.91.116.9:5261: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jun 26 06:08:51.073739594 (68h57m18.000034272s) [noise=true , address=45.91.116.10:4082 , piece_id=7NVEGA4QQOE7KYPHXUUX42SJPQL6LP46RGM37VD6A2AWN7PAOM3Q, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj, address:45.91.116.10:4082): protocol: rpc: dial tcp 45.91.116.10:4082: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jun 26 06:08:51.073789897 (68h57m18.000084584s) [noise=true , address=77.90.31.19:4842 , piece_id=G3AG375DATHB7MGEEWONE5GMMIC7QHG2VVDKF5AEVOSXV5N4S6AA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: rpc: dial tcp 77.90.31.19:4842: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jun 26 06:08:51.073849570 (68h57m18.000144246s) [noise=true , address=45.91.116.13:4154 , piece_id=STFPXN2LUJYUIAYROWZIRTCMT2Y3XET6KWA4GZWUHENXRJURP62Q, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN, address:45.91.116.13:4154): protocol: write tcp 159.223.143.39:35392->45.91.116.13:4154: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jun 26 06:08:51.073904716 (68h57m18.000199391s) [noise=true , address=77.90.31.13:4626 , piece_id=MXYBIEDFAFK53CVTE2C6CKEZWOL3MLW4S7LOHK4IHW6QEIZJL76Q, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: write tcp 159.223.143.39:58864->77.90.31.13:4626: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jun 26 06:08:51.073952402 (68h57m18.000247082s) [noise=true , address=77.90.31.10:4517 , piece_id=VJZ5T4QNVAPNUHNFNK24S6TAS4V4NKNFMQE2FZMCWWLCCB2ELXXQ, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, address:77.90.31.10:4517): protocol: write tcp 159.223.143.39:36382->77.90.31.10:4517: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jun 26 06:08:51.074022481 (68h57m18.000317156s) [noise=true , address=77.90.31.15:4697 , piece_id=CUXGVL7PS3H7ZQKUCSYXJCBR3ETYFDPZFYIEAWEYL7C2S3RJY3AA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5, address:77.90.31.15:4697): protocol: write tcp 159.223.143.39:52730->77.90.31.15:4697: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jun 26 06:08:51.074071824 (68h57m18.000366503s) [noise=true , address=77.90.31.11:4554 , piece_id=R2CIFLWS4LXTGSDSIMIRC4UYVJTXC7T35HPWZ2S7VYUTJ2AMY3BQ, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, address:77.90.31.11:4554): protocol: rpc: dial tcp 77.90.31.11:4554: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jun 26 06:08:51.074102177 (68h57m18.000396855s) [noise=true , address=77.90.31.14:4661 , piece_id=VXI2ZUSGR4N3H7TPIK75RIDFP3SBAOWVRGHNXTZOHFT5FNG4JWJA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue, address:77.90.31.14:4661): protocol: rpc: dial tcp 77.90.31.14:4661: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jun 26 06:08:51.074119276 (68h57m18.000413966s) [seg_pos={0 0}, upload=55354]: Done waiting for segment. successful: 7 optimal: 13 errs: <nil>
Jun 26 06:08:51.100079867 (68h57m18.026374541s) [upload=55354]: Done uploading segments...
Jun 26 06:08:51.100114505 (68h57m18.026409191s) [upload=55354]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 7)
Duration: 604.830459ms
core
uplink
us-select-1
|
checkers.AuthServiceHTTP{"https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us1)antiblip:
1 out of 60 is failed (threshold=3) [..........X.................................................] duration:
181.914186ms
FAIL (2025-06-26 06:27:54 UTC)
orbiter warning: unexpected status code "500 Internal Server Error" "\n"
Duration: 181.941983ms
edge
authservice
http
bgp
us1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [................................XX..........................] duration:
533.560182ms
FAIL (2025-06-26 06:27:50 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 533.582885ms
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [.........X..................................................] duration:
656.109041ms
FAIL (2025-06-26 06:27:50 UTC)
orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded
Duration: 656.127418ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
3 out of 60 is failed (threshold=3) [.................................XXX........................] duration:
520.087677ms
FAIL (2025-06-26 06:27:36 UTC)
orbiter warning:
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 7)
======LOGS======
pload=55354]: Got next segment. Inline: false
Jun 26 06:08:21.105827735 (68h56m48.032122411s) [upload=55354]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jun 26 06:08:21.105833588 (68h56m48.032128274s) [upload=55354]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jun 26 06:08:21.135318313 (68h56m48.061613009s) [seg_pos={0 0}, upload=55354]: Begin upload segment...
Jun 26 06:08:21.135366534 (68h56m48.061661214s) [seg_pos={0 0}, upload=55354]: Done begin upload segment.
Jun 26 06:08:21.135390342 (68h56m48.061685028s) [noise=true , address=45.91.116.9:5261 , piece_id=CZWB3XJVA7B7R7GWFYNW3RCGNVRXQ3E37XDXD4SUXTEMQFZ73HHA, seg_pos={0 0}, upload=55354]: Uploading piece...
Jun 26 06:08:21.135405801 (68h56m48.061700497s) [noise=true , address=104.36.82.6:4873 , piece_id=TIIMEQUSDLRFF3OK3WBB74F7SWJ5MEC4Z6IQNELDMRZLNTMQT7LA, seg_pos={0 0}, upload=55354]: Uploading piece...
Jun 26 06:08:21.135774729 (68h56m48.062069418s) [noise=true , address=77.90.31.13:4626 , piece_id=MXYBIEDFAFK53CVTE2C6CKEZWOL3MLW4S7LOHK4IHW6QEIZJL76Q, seg_pos={0 0}, upload=55354]: Uploading piece...
Jun 26 06:08:21.135944436 (68h56m48.06223913s) [noise=true , address=77.90.31.10:4517 , piece_id=VJZ5T4QNVAPNUHNFNK24S6TAS4V4NKNFMQE2FZMCWWLCCB2ELXXQ, seg_pos={0 0}, upload=55354]: Uploading piece...
Jun 26 06:08:21.136080402 (68h56m48.062375094s) [noise=true , address=77.90.31.15:4697 , piece_id=CUXGVL7PS3H7ZQKUCSYXJCBR3ETYFDPZFYIEAWEYL7C2S3RJY3AA, seg_pos={0 0}, upload=55354]: Uploading piece...
Jun 26 06:08:21.136382603 (68h56m48.062677289s) [noise=true , address=104.36.82.13:4186 , piece_id=5SVABN3KS6AUL6HILHNQCVF74BA75VPKTTAK7IQAPKRRME3MOZEA, seg_pos={0 0}, upload=55354]: Uploading piece...
Jun 26 06:08:21.136399304 (68h56m48.062693995s) [noise=true , address=104.36.82.14:4223 , piece_id=LP74YFI2SRSYPYEIKREVINCOFMVCXLNDS2555YKUVFN63N6TBJDA, seg_pos={0 0}, upload=55354]: Uploading piece...
Jun 26 06:08:21.136669576 (68h56m48.062964262s) [noise=true , address=104.36.82.6:4876 , piece_id=SSMGONONY34EHF5W6QMKO7YXB7RLO5LZVTJMMFKTP6GQNGYFYQ4A, seg_pos={0 0}, upload=55354]: Uploading piece...
Jun 26 06:08:21.136809784 (68h56m48.063104477s) [noise=true , address=104.36.82.15:4258 , piece_id=YXS4J7YSG6JV33JG4BZLK5AXKE3JXLBTGZJXPXZQ4LBYBT3MU2AA, seg_pos={0 0}, upload=55354]: Uploading piece...
Jun 26 06:08:21.136950556 (68h56m48.063245242s) [noise=true , address=77.90.31.11:4554 , piece_id=R2CIFLWS4LXTGSDSIMIRC4UYVJTXC7T35HPWZ2S7VYUTJ2AMY3BQ, seg_pos={0 0}, upload=55354]: Uploading piece...
Jun 26 06:08:21.137226497 (68h56m48.06352119s) [noise=true , address=77.90.31.14:4661 , piece_id=VXI2ZUSGR4N3H7TPIK75RIDFP3SBAOWVRGHNXTZOHFT5FNG4JWJA, seg_pos={0 0}, upload=55354]: Uploading piece...
Jun 26 06:08:21.138099107 (68h56m48.064393792s) [noise=true , address=45.91.116.10:4082 , piece_id=7NVEGA4QQOE7KYPHXUUX42SJPQL6LP46RGM37VD6A2AWN7PAOM3Q, seg_pos={0 0}, upload=55354]: Uploading piece...
Jun 26 06:08:21.139330448 (68h56m48.065625141s) [noise=true , address=104.36.82.14:4222 , piece_id=U3MBMAKFBFTOPJZ3HS3FXLPDBLQTN577XNCZLUBHRO6IU4BE5VNA, seg_pos={0 0}, upload=55354]: Uploading piece...
Jun 26 06:08:21.140580734 (68h56m48.06687543s) [noise=true , address=77.90.31.19:4842 , piece_id=G3AG375DATHB7MGEEWONE5GMMIC7QHG2VVDKF5AEVOSXV5N4S6AA, seg_pos={0 0}, upload=55354]: Uploading piece...
Jun 26 06:08:21.141452300 (68h56m48.067747027s) [noise=true , address=45.91.116.13:4154 , piece_id=STFPXN2LUJYUIAYROWZIRTCMT2Y3XET6KWA4GZWUHENXRJURP62Q, seg_pos={0 0}, upload=55354]: Uploading piece...
Jun 26 06:08:21.142917062 (68h56m48.069211739s) [noise=true , address=104.36.82.7:4912 , piece_id=A5HPC7CEU66IFNS47HO3AQX22LUZ6AZ4L4XAMZ6T5NCVE4D3TAZA, seg_pos={0 0}, upload=55354]: Uploading piece...
Jun 26 06:08:21.221824189 (68h56m48.148118865s) [upload=55354]: Next returned nil segment
Jun 26 06:08:21.221837958 (68h56m48.148132644s) [upload=55354]: Waiting for error group managing segments...
Jun 26 06:08:21.229550628 (68h56m48.155845315s) [noise=true , address=104.36.82.14:4223 , piece_id=LP74YFI2SRSYPYEIKREVINCOFMVCXLNDS2555YKUVFN63N6TBJDA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: <nil>
Jun 26 06:08:21.230245145 (68h56m48.156539832s) [noise=true , address=104.36.82.14:4222 , piece_id=U3MBMAKFBFTOPJZ3HS3FXLPDBLQTN577XNCZLUBHRO6IU4BE5VNA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: <nil>
Jun 26 06:08:21.230801067 (68h56m48.15709574s) [noise=true , address=104.36.82.6:4873 , piece_id=TIIMEQUSDLRFF3OK3WBB74F7SWJ5MEC4Z6IQNELDMRZLNTMQT7LA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: <nil>
Jun 26 06:08:21.231457382 (68h56m48.157752071s) [noise=true , address=104.36.82.6:4876 , piece_id=SSMGONONY34EHF5W6QMKO7YXB7RLO5LZVTJMMFKTP6GQNGYFYQ4A, seg_pos={0 0}, upload=55354]: Done uploading piece. err: <nil>
Jun 26 06:08:21.231563205 (68h56m48.157857893s) [noise=true , address=104.36.82.13:4186 , piece_id=5SVABN3KS6AUL6HILHNQCVF74BA75VPKTTAK7IQAPKRRME3MOZEA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: <nil>
Jun 26 06:08:21.239131119 (68h56m48.165425799s) [noise=true , address=104.36.82.7:4912 , piece_id=A5HPC7CEU66IFNS47HO3AQX22LUZ6AZ4L4XAMZ6T5NCVE4D3TAZA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: <nil>
Jun 26 06:08:21.534686665 (68h56m48.460981337s) [noise=true , address=104.36.82.15:4258 , piece_id=YXS4J7YSG6JV33JG4BZLK5AXKE3JXLBTGZJXPXZQ4LBYBT3MU2AA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: <nil>
Jun 26 06:08:51.073673533 (68h57m17.999968219s) [noise=true , address=45.91.116.9:5261 , piece_id=CZWB3XJVA7B7R7GWFYNW3RCGNVRXQ3E37XDXD4SUXTEMQFZ73HHA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi, address:45.91.116.9:5261): protocol: rpc: dial tcp 45.91.116.9:5261: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jun 26 06:08:51.073739594 (68h57m18.000034272s) [noise=true , address=45.91.116.10:4082 , piece_id=7NVEGA4QQOE7KYPHXUUX42SJPQL6LP46RGM37VD6A2AWN7PAOM3Q, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj, address:45.91.116.10:4082): protocol: rpc: dial tcp 45.91.116.10:4082: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jun 26 06:08:51.073789897 (68h57m18.000084584s) [noise=true , address=77.90.31.19:4842 , piece_id=G3AG375DATHB7MGEEWONE5GMMIC7QHG2VVDKF5AEVOSXV5N4S6AA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: rpc: dial tcp 77.90.31.19:4842: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jun 26 06:08:51.073849570 (68h57m18.000144246s) [noise=true , address=45.91.116.13:4154 , piece_id=STFPXN2LUJYUIAYROWZIRTCMT2Y3XET6KWA4GZWUHENXRJURP62Q, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN, address:45.91.116.13:4154): protocol: write tcp 159.223.143.39:35392->45.91.116.13:4154: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jun 26 06:08:51.073904716 (68h57m18.000199391s) [noise=true , address=77.90.31.13:4626 , piece_id=MXYBIEDFAFK53CVTE2C6CKEZWOL3MLW4S7LOHK4IHW6QEIZJL76Q, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: write tcp 159.223.143.39:58864->77.90.31.13:4626: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jun 26 06:08:51.073952402 (68h57m18.000247082s) [noise=true , address=77.90.31.10:4517 , piece_id=VJZ5T4QNVAPNUHNFNK24S6TAS4V4NKNFMQE2FZMCWWLCCB2ELXXQ, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, address:77.90.31.10:4517): protocol: write tcp 159.223.143.39:36382->77.90.31.10:4517: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jun 26 06:08:51.074022481 (68h57m18.000317156s) [noise=true , address=77.90.31.15:4697 , piece_id=CUXGVL7PS3H7ZQKUCSYXJCBR3ETYFDPZFYIEAWEYL7C2S3RJY3AA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5, address:77.90.31.15:4697): protocol: write tcp 159.223.143.39:52730->77.90.31.15:4697: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jun 26 06:08:51.074071824 (68h57m18.000366503s) [noise=true , address=77.90.31.11:4554 , piece_id=R2CIFLWS4LXTGSDSIMIRC4UYVJTXC7T35HPWZ2S7VYUTJ2AMY3BQ, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, address:77.90.31.11:4554): protocol: rpc: dial tcp 77.90.31.11:4554: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jun 26 06:08:51.074102177 (68h57m18.000396855s) [noise=true , address=77.90.31.14:4661 , piece_id=VXI2ZUSGR4N3H7TPIK75RIDFP3SBAOWVRGHNXTZOHFT5FNG4JWJA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue, address:77.90.31.14:4661): protocol: rpc: dial tcp 77.90.31.14:4661: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jun 26 06:08:51.074119276 (68h57m18.000413966s) [seg_pos={0 0}, upload=55354]: Done waiting for segment. successful: 7 optimal: 13 errs: <nil>
Jun 26 06:08:51.100079867 (68h57m18.026374541s) [upload=55354]: Done uploading segments...
Jun 26 06:08:51.100114505 (68h57m18.026409191s) [upload=55354]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 7)
Duration: 520.134955ms
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [.................................XX.........................] duration:
505.830286ms
FAIL (2025-06-26 06:27:25 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 505.850618ms
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [..........X.................................................] duration:
620.731449ms
FAIL (2025-06-26 06:27:02 UTC)
orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded
Duration: 620.760218ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
3 out of 60 is failed (threshold=3) [..................................XXX.......................] duration:
3.526802061s
FAIL (2025-06-26 06:27:01 UTC)
orbiter warning:
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 7)
======LOGS======
pload=55354]: Got next segment. Inline: false
Jun 26 06:08:21.105827735 (68h56m48.032122411s) [upload=55354]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jun 26 06:08:21.105833588 (68h56m48.032128274s) [upload=55354]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jun 26 06:08:21.135318313 (68h56m48.061613009s) [seg_pos={0 0}, upload=55354]: Begin upload segment...
Jun 26 06:08:21.135366534 (68h56m48.061661214s) [seg_pos={0 0}, upload=55354]: Done begin upload segment.
Jun 26 06:08:21.135390342 (68h56m48.061685028s) [noise=true , address=45.91.116.9:5261 , piece_id=CZWB3XJVA7B7R7GWFYNW3RCGNVRXQ3E37XDXD4SUXTEMQFZ73HHA, seg_pos={0 0}, upload=55354]: Uploading piece...
Jun 26 06:08:21.135405801 (68h56m48.061700497s) [noise=true , address=104.36.82.6:4873 , piece_id=TIIMEQUSDLRFF3OK3WBB74F7SWJ5MEC4Z6IQNELDMRZLNTMQT7LA, seg_pos={0 0}, upload=55354]: Uploading piece...
Jun 26 06:08:21.135774729 (68h56m48.062069418s) [noise=true , address=77.90.31.13:4626 , piece_id=MXYBIEDFAFK53CVTE2C6CKEZWOL3MLW4S7LOHK4IHW6QEIZJL76Q, seg_pos={0 0}, upload=55354]: Uploading piece...
Jun 26 06:08:21.135944436 (68h56m48.06223913s) [noise=true , address=77.90.31.10:4517 , piece_id=VJZ5T4QNVAPNUHNFNK24S6TAS4V4NKNFMQE2FZMCWWLCCB2ELXXQ, seg_pos={0 0}, upload=55354]: Uploading piece...
Jun 26 06:08:21.136080402 (68h56m48.062375094s) [noise=true , address=77.90.31.15:4697 , piece_id=CUXGVL7PS3H7ZQKUCSYXJCBR3ETYFDPZFYIEAWEYL7C2S3RJY3AA, seg_pos={0 0}, upload=55354]: Uploading piece...
Jun 26 06:08:21.136382603 (68h56m48.062677289s) [noise=true , address=104.36.82.13:4186 , piece_id=5SVABN3KS6AUL6HILHNQCVF74BA75VPKTTAK7IQAPKRRME3MOZEA, seg_pos={0 0}, upload=55354]: Uploading piece...
Jun 26 06:08:21.136399304 (68h56m48.062693995s) [noise=true , address=104.36.82.14:4223 , piece_id=LP74YFI2SRSYPYEIKREVINCOFMVCXLNDS2555YKUVFN63N6TBJDA, seg_pos={0 0}, upload=55354]: Uploading piece...
Jun 26 06:08:21.136669576 (68h56m48.062964262s) [noise=true , address=104.36.82.6:4876 , piece_id=SSMGONONY34EHF5W6QMKO7YXB7RLO5LZVTJMMFKTP6GQNGYFYQ4A, seg_pos={0 0}, upload=55354]: Uploading piece...
Jun 26 06:08:21.136809784 (68h56m48.063104477s) [noise=true , address=104.36.82.15:4258 , piece_id=YXS4J7YSG6JV33JG4BZLK5AXKE3JXLBTGZJXPXZQ4LBYBT3MU2AA, seg_pos={0 0}, upload=55354]: Uploading piece...
Jun 26 06:08:21.136950556 (68h56m48.063245242s) [noise=true , address=77.90.31.11:4554 , piece_id=R2CIFLWS4LXTGSDSIMIRC4UYVJTXC7T35HPWZ2S7VYUTJ2AMY3BQ, seg_pos={0 0}, upload=55354]: Uploading piece...
Jun 26 06:08:21.137226497 (68h56m48.06352119s) [noise=true , address=77.90.31.14:4661 , piece_id=VXI2ZUSGR4N3H7TPIK75RIDFP3SBAOWVRGHNXTZOHFT5FNG4JWJA, seg_pos={0 0}, upload=55354]: Uploading piece...
Jun 26 06:08:21.138099107 (68h56m48.064393792s) [noise=true , address=45.91.116.10:4082 , piece_id=7NVEGA4QQOE7KYPHXUUX42SJPQL6LP46RGM37VD6A2AWN7PAOM3Q, seg_pos={0 0}, upload=55354]: Uploading piece...
Jun 26 06:08:21.139330448 (68h56m48.065625141s) [noise=true , address=104.36.82.14:4222 , piece_id=U3MBMAKFBFTOPJZ3HS3FXLPDBLQTN577XNCZLUBHRO6IU4BE5VNA, seg_pos={0 0}, upload=55354]: Uploading piece...
Jun 26 06:08:21.140580734 (68h56m48.06687543s) [noise=true , address=77.90.31.19:4842 , piece_id=G3AG375DATHB7MGEEWONE5GMMIC7QHG2VVDKF5AEVOSXV5N4S6AA, seg_pos={0 0}, upload=55354]: Uploading piece...
Jun 26 06:08:21.141452300 (68h56m48.067747027s) [noise=true , address=45.91.116.13:4154 , piece_id=STFPXN2LUJYUIAYROWZIRTCMT2Y3XET6KWA4GZWUHENXRJURP62Q, seg_pos={0 0}, upload=55354]: Uploading piece...
Jun 26 06:08:21.142917062 (68h56m48.069211739s) [noise=true , address=104.36.82.7:4912 , piece_id=A5HPC7CEU66IFNS47HO3AQX22LUZ6AZ4L4XAMZ6T5NCVE4D3TAZA, seg_pos={0 0}, upload=55354]: Uploading piece...
Jun 26 06:08:21.221824189 (68h56m48.148118865s) [upload=55354]: Next returned nil segment
Jun 26 06:08:21.221837958 (68h56m48.148132644s) [upload=55354]: Waiting for error group managing segments...
Jun 26 06:08:21.229550628 (68h56m48.155845315s) [noise=true , address=104.36.82.14:4223 , piece_id=LP74YFI2SRSYPYEIKREVINCOFMVCXLNDS2555YKUVFN63N6TBJDA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: <nil>
Jun 26 06:08:21.230245145 (68h56m48.156539832s) [noise=true , address=104.36.82.14:4222 , piece_id=U3MBMAKFBFTOPJZ3HS3FXLPDBLQTN577XNCZLUBHRO6IU4BE5VNA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: <nil>
Jun 26 06:08:21.230801067 (68h56m48.15709574s) [noise=true , address=104.36.82.6:4873 , piece_id=TIIMEQUSDLRFF3OK3WBB74F7SWJ5MEC4Z6IQNELDMRZLNTMQT7LA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: <nil>
Jun 26 06:08:21.231457382 (68h56m48.157752071s) [noise=true , address=104.36.82.6:4876 , piece_id=SSMGONONY34EHF5W6QMKO7YXB7RLO5LZVTJMMFKTP6GQNGYFYQ4A, seg_pos={0 0}, upload=55354]: Done uploading piece. err: <nil>
Jun 26 06:08:21.231563205 (68h56m48.157857893s) [noise=true , address=104.36.82.13:4186 , piece_id=5SVABN3KS6AUL6HILHNQCVF74BA75VPKTTAK7IQAPKRRME3MOZEA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: <nil>
Jun 26 06:08:21.239131119 (68h56m48.165425799s) [noise=true , address=104.36.82.7:4912 , piece_id=A5HPC7CEU66IFNS47HO3AQX22LUZ6AZ4L4XAMZ6T5NCVE4D3TAZA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: <nil>
Jun 26 06:08:21.534686665 (68h56m48.460981337s) [noise=true , address=104.36.82.15:4258 , piece_id=YXS4J7YSG6JV33JG4BZLK5AXKE3JXLBTGZJXPXZQ4LBYBT3MU2AA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: <nil>
Jun 26 06:08:51.073673533 (68h57m17.999968219s) [noise=true , address=45.91.116.9:5261 , piece_id=CZWB3XJVA7B7R7GWFYNW3RCGNVRXQ3E37XDXD4SUXTEMQFZ73HHA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi, address:45.91.116.9:5261): protocol: rpc: dial tcp 45.91.116.9:5261: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jun 26 06:08:51.073739594 (68h57m18.000034272s) [noise=true , address=45.91.116.10:4082 , piece_id=7NVEGA4QQOE7KYPHXUUX42SJPQL6LP46RGM37VD6A2AWN7PAOM3Q, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj, address:45.91.116.10:4082): protocol: rpc: dial tcp 45.91.116.10:4082: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jun 26 06:08:51.073789897 (68h57m18.000084584s) [noise=true , address=77.90.31.19:4842 , piece_id=G3AG375DATHB7MGEEWONE5GMMIC7QHG2VVDKF5AEVOSXV5N4S6AA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: rpc: dial tcp 77.90.31.19:4842: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jun 26 06:08:51.073849570 (68h57m18.000144246s) [noise=true , address=45.91.116.13:4154 , piece_id=STFPXN2LUJYUIAYROWZIRTCMT2Y3XET6KWA4GZWUHENXRJURP62Q, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN, address:45.91.116.13:4154): protocol: write tcp 159.223.143.39:35392->45.91.116.13:4154: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jun 26 06:08:51.073904716 (68h57m18.000199391s) [noise=true , address=77.90.31.13:4626 , piece_id=MXYBIEDFAFK53CVTE2C6CKEZWOL3MLW4S7LOHK4IHW6QEIZJL76Q, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: write tcp 159.223.143.39:58864->77.90.31.13:4626: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jun 26 06:08:51.073952402 (68h57m18.000247082s) [noise=true , address=77.90.31.10:4517 , piece_id=VJZ5T4QNVAPNUHNFNK24S6TAS4V4NKNFMQE2FZMCWWLCCB2ELXXQ, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, address:77.90.31.10:4517): protocol: write tcp 159.223.143.39:36382->77.90.31.10:4517: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jun 26 06:08:51.074022481 (68h57m18.000317156s) [noise=true , address=77.90.31.15:4697 , piece_id=CUXGVL7PS3H7ZQKUCSYXJCBR3ETYFDPZFYIEAWEYL7C2S3RJY3AA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5, address:77.90.31.15:4697): protocol: write tcp 159.223.143.39:52730->77.90.31.15:4697: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jun 26 06:08:51.074071824 (68h57m18.000366503s) [noise=true , address=77.90.31.11:4554 , piece_id=R2CIFLWS4LXTGSDSIMIRC4UYVJTXC7T35HPWZ2S7VYUTJ2AMY3BQ, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, address:77.90.31.11:4554): protocol: rpc: dial tcp 77.90.31.11:4554: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jun 26 06:08:51.074102177 (68h57m18.000396855s) [noise=true , address=77.90.31.14:4661 , piece_id=VXI2ZUSGR4N3H7TPIK75RIDFP3SBAOWVRGHNXTZOHFT5FNG4JWJA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue, address:77.90.31.14:4661): protocol: rpc: dial tcp 77.90.31.14:4661: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jun 26 06:08:51.074119276 (68h57m18.000413966s) [seg_pos={0 0}, upload=55354]: Done waiting for segment. successful: 7 optimal: 13 errs: <nil>
Jun 26 06:08:51.100079867 (68h57m18.026374541s) [upload=55354]: Done uploading segments...
Jun 26 06:08:51.100114505 (68h57m18.026409191s) [upload=55354]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 7)
Duration: 3.526826696s
core
uplink
us-select-1
|
checkers.AuthServiceHTTP{"https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us1)antiblip:
1 out of 60 is failed (threshold=3) [...........X................................................] duration:
172.374769ms
FAIL (2025-06-26 06:26:55 UTC)
orbiter warning: unexpected status code "500 Internal Server Error" "\n"
Duration: 172.410802ms
edge
authservice
http
bgp
us1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [..................................XX........................] duration:
664.132628ms
FAIL (2025-06-26 06:26:54 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 664.189309ms
edge
gateway
us-select-1
|
checkers.AuthServiceHTTP{"https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us1)antiblip:
1 out of 60 is failed (threshold=3) [............X...............................................] duration:
196.860956ms
FAIL (2025-06-26 06:26:23 UTC)
orbiter warning: unexpected status code "500 Internal Server Error" "\n"
Duration: 196.887287ms
edge
authservice
http
bgp
us1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [...........X................................................] duration:
691.535289ms
FAIL (2025-06-26 06:26:01 UTC)
orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded
Duration: 691.55895ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
3 out of 60 is failed (threshold=3) [...................................XXX......................] duration:
453.354171ms
FAIL (2025-06-26 06:25:53 UTC)
orbiter warning:
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 7)
======LOGS======
pload=55354]: Got next segment. Inline: false
Jun 26 06:08:21.105827735 (68h56m48.032122411s) [upload=55354]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jun 26 06:08:21.105833588 (68h56m48.032128274s) [upload=55354]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jun 26 06:08:21.135318313 (68h56m48.061613009s) [seg_pos={0 0}, upload=55354]: Begin upload segment...
Jun 26 06:08:21.135366534 (68h56m48.061661214s) [seg_pos={0 0}, upload=55354]: Done begin upload segment.
Jun 26 06:08:21.135390342 (68h56m48.061685028s) [noise=true , address=45.91.116.9:5261 , piece_id=CZWB3XJVA7B7R7GWFYNW3RCGNVRXQ3E37XDXD4SUXTEMQFZ73HHA, seg_pos={0 0}, upload=55354]: Uploading piece...
Jun 26 06:08:21.135405801 (68h56m48.061700497s) [noise=true , address=104.36.82.6:4873 , piece_id=TIIMEQUSDLRFF3OK3WBB74F7SWJ5MEC4Z6IQNELDMRZLNTMQT7LA, seg_pos={0 0}, upload=55354]: Uploading piece...
Jun 26 06:08:21.135774729 (68h56m48.062069418s) [noise=true , address=77.90.31.13:4626 , piece_id=MXYBIEDFAFK53CVTE2C6CKEZWOL3MLW4S7LOHK4IHW6QEIZJL76Q, seg_pos={0 0}, upload=55354]: Uploading piece...
Jun 26 06:08:21.135944436 (68h56m48.06223913s) [noise=true , address=77.90.31.10:4517 , piece_id=VJZ5T4QNVAPNUHNFNK24S6TAS4V4NKNFMQE2FZMCWWLCCB2ELXXQ, seg_pos={0 0}, upload=55354]: Uploading piece...
Jun 26 06:08:21.136080402 (68h56m48.062375094s) [noise=true , address=77.90.31.15:4697 , piece_id=CUXGVL7PS3H7ZQKUCSYXJCBR3ETYFDPZFYIEAWEYL7C2S3RJY3AA, seg_pos={0 0}, upload=55354]: Uploading piece...
Jun 26 06:08:21.136382603 (68h56m48.062677289s) [noise=true , address=104.36.82.13:4186 , piece_id=5SVABN3KS6AUL6HILHNQCVF74BA75VPKTTAK7IQAPKRRME3MOZEA, seg_pos={0 0}, upload=55354]: Uploading piece...
Jun 26 06:08:21.136399304 (68h56m48.062693995s) [noise=true , address=104.36.82.14:4223 , piece_id=LP74YFI2SRSYPYEIKREVINCOFMVCXLNDS2555YKUVFN63N6TBJDA, seg_pos={0 0}, upload=55354]: Uploading piece...
Jun 26 06:08:21.136669576 (68h56m48.062964262s) [noise=true , address=104.36.82.6:4876 , piece_id=SSMGONONY34EHF5W6QMKO7YXB7RLO5LZVTJMMFKTP6GQNGYFYQ4A, seg_pos={0 0}, upload=55354]: Uploading piece...
Jun 26 06:08:21.136809784 (68h56m48.063104477s) [noise=true , address=104.36.82.15:4258 , piece_id=YXS4J7YSG6JV33JG4BZLK5AXKE3JXLBTGZJXPXZQ4LBYBT3MU2AA, seg_pos={0 0}, upload=55354]: Uploading piece...
Jun 26 06:08:21.136950556 (68h56m48.063245242s) [noise=true , address=77.90.31.11:4554 , piece_id=R2CIFLWS4LXTGSDSIMIRC4UYVJTXC7T35HPWZ2S7VYUTJ2AMY3BQ, seg_pos={0 0}, upload=55354]: Uploading piece...
Jun 26 06:08:21.137226497 (68h56m48.06352119s) [noise=true , address=77.90.31.14:4661 , piece_id=VXI2ZUSGR4N3H7TPIK75RIDFP3SBAOWVRGHNXTZOHFT5FNG4JWJA, seg_pos={0 0}, upload=55354]: Uploading piece...
Jun 26 06:08:21.138099107 (68h56m48.064393792s) [noise=true , address=45.91.116.10:4082 , piece_id=7NVEGA4QQOE7KYPHXUUX42SJPQL6LP46RGM37VD6A2AWN7PAOM3Q, seg_pos={0 0}, upload=55354]: Uploading piece...
Jun 26 06:08:21.139330448 (68h56m48.065625141s) [noise=true , address=104.36.82.14:4222 , piece_id=U3MBMAKFBFTOPJZ3HS3FXLPDBLQTN577XNCZLUBHRO6IU4BE5VNA, seg_pos={0 0}, upload=55354]: Uploading piece...
Jun 26 06:08:21.140580734 (68h56m48.06687543s) [noise=true , address=77.90.31.19:4842 , piece_id=G3AG375DATHB7MGEEWONE5GMMIC7QHG2VVDKF5AEVOSXV5N4S6AA, seg_pos={0 0}, upload=55354]: Uploading piece...
Jun 26 06:08:21.141452300 (68h56m48.067747027s) [noise=true , address=45.91.116.13:4154 , piece_id=STFPXN2LUJYUIAYROWZIRTCMT2Y3XET6KWA4GZWUHENXRJURP62Q, seg_pos={0 0}, upload=55354]: Uploading piece...
Jun 26 06:08:21.142917062 (68h56m48.069211739s) [noise=true , address=104.36.82.7:4912 , piece_id=A5HPC7CEU66IFNS47HO3AQX22LUZ6AZ4L4XAMZ6T5NCVE4D3TAZA, seg_pos={0 0}, upload=55354]: Uploading piece...
Jun 26 06:08:21.221824189 (68h56m48.148118865s) [upload=55354]: Next returned nil segment
Jun 26 06:08:21.221837958 (68h56m48.148132644s) [upload=55354]: Waiting for error group managing segments...
Jun 26 06:08:21.229550628 (68h56m48.155845315s) [noise=true , address=104.36.82.14:4223 , piece_id=LP74YFI2SRSYPYEIKREVINCOFMVCXLNDS2555YKUVFN63N6TBJDA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: <nil>
Jun 26 06:08:21.230245145 (68h56m48.156539832s) [noise=true , address=104.36.82.14:4222 , piece_id=U3MBMAKFBFTOPJZ3HS3FXLPDBLQTN577XNCZLUBHRO6IU4BE5VNA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: <nil>
Jun 26 06:08:21.230801067 (68h56m48.15709574s) [noise=true , address=104.36.82.6:4873 , piece_id=TIIMEQUSDLRFF3OK3WBB74F7SWJ5MEC4Z6IQNELDMRZLNTMQT7LA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: <nil>
Jun 26 06:08:21.231457382 (68h56m48.157752071s) [noise=true , address=104.36.82.6:4876 , piece_id=SSMGONONY34EHF5W6QMKO7YXB7RLO5LZVTJMMFKTP6GQNGYFYQ4A, seg_pos={0 0}, upload=55354]: Done uploading piece. err: <nil>
Jun 26 06:08:21.231563205 (68h56m48.157857893s) [noise=true , address=104.36.82.13:4186 , piece_id=5SVABN3KS6AUL6HILHNQCVF74BA75VPKTTAK7IQAPKRRME3MOZEA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: <nil>
Jun 26 06:08:21.239131119 (68h56m48.165425799s) [noise=true , address=104.36.82.7:4912 , piece_id=A5HPC7CEU66IFNS47HO3AQX22LUZ6AZ4L4XAMZ6T5NCVE4D3TAZA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: <nil>
Jun 26 06:08:21.534686665 (68h56m48.460981337s) [noise=true , address=104.36.82.15:4258 , piece_id=YXS4J7YSG6JV33JG4BZLK5AXKE3JXLBTGZJXPXZQ4LBYBT3MU2AA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: <nil>
Jun 26 06:08:51.073673533 (68h57m17.999968219s) [noise=true , address=45.91.116.9:5261 , piece_id=CZWB3XJVA7B7R7GWFYNW3RCGNVRXQ3E37XDXD4SUXTEMQFZ73HHA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi, address:45.91.116.9:5261): protocol: rpc: dial tcp 45.91.116.9:5261: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jun 26 06:08:51.073739594 (68h57m18.000034272s) [noise=true , address=45.91.116.10:4082 , piece_id=7NVEGA4QQOE7KYPHXUUX42SJPQL6LP46RGM37VD6A2AWN7PAOM3Q, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj, address:45.91.116.10:4082): protocol: rpc: dial tcp 45.91.116.10:4082: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jun 26 06:08:51.073789897 (68h57m18.000084584s) [noise=true , address=77.90.31.19:4842 , piece_id=G3AG375DATHB7MGEEWONE5GMMIC7QHG2VVDKF5AEVOSXV5N4S6AA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: rpc: dial tcp 77.90.31.19:4842: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jun 26 06:08:51.073849570 (68h57m18.000144246s) [noise=true , address=45.91.116.13:4154 , piece_id=STFPXN2LUJYUIAYROWZIRTCMT2Y3XET6KWA4GZWUHENXRJURP62Q, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN, address:45.91.116.13:4154): protocol: write tcp 159.223.143.39:35392->45.91.116.13:4154: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jun 26 06:08:51.073904716 (68h57m18.000199391s) [noise=true , address=77.90.31.13:4626 , piece_id=MXYBIEDFAFK53CVTE2C6CKEZWOL3MLW4S7LOHK4IHW6QEIZJL76Q, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: write tcp 159.223.143.39:58864->77.90.31.13:4626: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jun 26 06:08:51.073952402 (68h57m18.000247082s) [noise=true , address=77.90.31.10:4517 , piece_id=VJZ5T4QNVAPNUHNFNK24S6TAS4V4NKNFMQE2FZMCWWLCCB2ELXXQ, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, address:77.90.31.10:4517): protocol: write tcp 159.223.143.39:36382->77.90.31.10:4517: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jun 26 06:08:51.074022481 (68h57m18.000317156s) [noise=true , address=77.90.31.15:4697 , piece_id=CUXGVL7PS3H7ZQKUCSYXJCBR3ETYFDPZFYIEAWEYL7C2S3RJY3AA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5, address:77.90.31.15:4697): protocol: write tcp 159.223.143.39:52730->77.90.31.15:4697: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jun 26 06:08:51.074071824 (68h57m18.000366503s) [noise=true , address=77.90.31.11:4554 , piece_id=R2CIFLWS4LXTGSDSIMIRC4UYVJTXC7T35HPWZ2S7VYUTJ2AMY3BQ, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, address:77.90.31.11:4554): protocol: rpc: dial tcp 77.90.31.11:4554: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jun 26 06:08:51.074102177 (68h57m18.000396855s) [noise=true , address=77.90.31.14:4661 , piece_id=VXI2ZUSGR4N3H7TPIK75RIDFP3SBAOWVRGHNXTZOHFT5FNG4JWJA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue, address:77.90.31.14:4661): protocol: rpc: dial tcp 77.90.31.14:4661: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jun 26 06:08:51.074119276 (68h57m18.000413966s) [seg_pos={0 0}, upload=55354]: Done waiting for segment. successful: 7 optimal: 13 errs: <nil>
Jun 26 06:08:51.100079867 (68h57m18.026374541s) [upload=55354]: Done uploading segments...
Jun 26 06:08:51.100114505 (68h57m18.026409191s) [upload=55354]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 7)
Duration: 453.383304ms
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [...................................XX.......................] duration:
461.667476ms
FAIL (2025-06-26 06:25:48 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 461.736262ms
edge
gateway
us-select-1
|
checkers.AuthServiceHTTP{"https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us1)antiblip:
1 out of 60 is failed (threshold=3) [.............X..............................................] duration:
192.613177ms
FAIL (2025-06-26 06:25:37 UTC)
orbiter warning: unexpected status code "500 Internal Server Error" "\n"
Duration: 192.638395ms
edge
authservice
http
bgp
us1
|