Orbiter: EU/US/AP | Code | Config | Playbook | Edge Deployment | Escalation Procedure
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
3 out of 60 is failed (threshold=3) [....................................XXX.....................] duration:
442.027698ms
FAIL (2025-06-26 06:25:20 UTC)
orbiter warning:
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 7)
======LOGS======
pload=55354]: Got next segment. Inline: false
Jun 26 06:08:21.105827735 (68h56m48.032122411s) [upload=55354]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jun 26 06:08:21.105833588 (68h56m48.032128274s) [upload=55354]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jun 26 06:08:21.135318313 (68h56m48.061613009s) [seg_pos={0 0}, upload=55354]: Begin upload segment...
Jun 26 06:08:21.135366534 (68h56m48.061661214s) [seg_pos={0 0}, upload=55354]: Done begin upload segment.
Jun 26 06:08:21.135390342 (68h56m48.061685028s) [noise=true , address=45.91.116.9:5261 , piece_id=CZWB3XJVA7B7R7GWFYNW3RCGNVRXQ3E37XDXD4SUXTEMQFZ73HHA, seg_pos={0 0}, upload=55354]: Uploading piece...
Jun 26 06:08:21.135405801 (68h56m48.061700497s) [noise=true , address=104.36.82.6:4873 , piece_id=TIIMEQUSDLRFF3OK3WBB74F7SWJ5MEC4Z6IQNELDMRZLNTMQT7LA, seg_pos={0 0}, upload=55354]: Uploading piece...
Jun 26 06:08:21.135774729 (68h56m48.062069418s) [noise=true , address=77.90.31.13:4626 , piece_id=MXYBIEDFAFK53CVTE2C6CKEZWOL3MLW4S7LOHK4IHW6QEIZJL76Q, seg_pos={0 0}, upload=55354]: Uploading piece...
Jun 26 06:08:21.135944436 (68h56m48.06223913s) [noise=true , address=77.90.31.10:4517 , piece_id=VJZ5T4QNVAPNUHNFNK24S6TAS4V4NKNFMQE2FZMCWWLCCB2ELXXQ, seg_pos={0 0}, upload=55354]: Uploading piece...
Jun 26 06:08:21.136080402 (68h56m48.062375094s) [noise=true , address=77.90.31.15:4697 , piece_id=CUXGVL7PS3H7ZQKUCSYXJCBR3ETYFDPZFYIEAWEYL7C2S3RJY3AA, seg_pos={0 0}, upload=55354]: Uploading piece...
Jun 26 06:08:21.136382603 (68h56m48.062677289s) [noise=true , address=104.36.82.13:4186 , piece_id=5SVABN3KS6AUL6HILHNQCVF74BA75VPKTTAK7IQAPKRRME3MOZEA, seg_pos={0 0}, upload=55354]: Uploading piece...
Jun 26 06:08:21.136399304 (68h56m48.062693995s) [noise=true , address=104.36.82.14:4223 , piece_id=LP74YFI2SRSYPYEIKREVINCOFMVCXLNDS2555YKUVFN63N6TBJDA, seg_pos={0 0}, upload=55354]: Uploading piece...
Jun 26 06:08:21.136669576 (68h56m48.062964262s) [noise=true , address=104.36.82.6:4876 , piece_id=SSMGONONY34EHF5W6QMKO7YXB7RLO5LZVTJMMFKTP6GQNGYFYQ4A, seg_pos={0 0}, upload=55354]: Uploading piece...
Jun 26 06:08:21.136809784 (68h56m48.063104477s) [noise=true , address=104.36.82.15:4258 , piece_id=YXS4J7YSG6JV33JG4BZLK5AXKE3JXLBTGZJXPXZQ4LBYBT3MU2AA, seg_pos={0 0}, upload=55354]: Uploading piece...
Jun 26 06:08:21.136950556 (68h56m48.063245242s) [noise=true , address=77.90.31.11:4554 , piece_id=R2CIFLWS4LXTGSDSIMIRC4UYVJTXC7T35HPWZ2S7VYUTJ2AMY3BQ, seg_pos={0 0}, upload=55354]: Uploading piece...
Jun 26 06:08:21.137226497 (68h56m48.06352119s) [noise=true , address=77.90.31.14:4661 , piece_id=VXI2ZUSGR4N3H7TPIK75RIDFP3SBAOWVRGHNXTZOHFT5FNG4JWJA, seg_pos={0 0}, upload=55354]: Uploading piece...
Jun 26 06:08:21.138099107 (68h56m48.064393792s) [noise=true , address=45.91.116.10:4082 , piece_id=7NVEGA4QQOE7KYPHXUUX42SJPQL6LP46RGM37VD6A2AWN7PAOM3Q, seg_pos={0 0}, upload=55354]: Uploading piece...
Jun 26 06:08:21.139330448 (68h56m48.065625141s) [noise=true , address=104.36.82.14:4222 , piece_id=U3MBMAKFBFTOPJZ3HS3FXLPDBLQTN577XNCZLUBHRO6IU4BE5VNA, seg_pos={0 0}, upload=55354]: Uploading piece...
Jun 26 06:08:21.140580734 (68h56m48.06687543s) [noise=true , address=77.90.31.19:4842 , piece_id=G3AG375DATHB7MGEEWONE5GMMIC7QHG2VVDKF5AEVOSXV5N4S6AA, seg_pos={0 0}, upload=55354]: Uploading piece...
Jun 26 06:08:21.141452300 (68h56m48.067747027s) [noise=true , address=45.91.116.13:4154 , piece_id=STFPXN2LUJYUIAYROWZIRTCMT2Y3XET6KWA4GZWUHENXRJURP62Q, seg_pos={0 0}, upload=55354]: Uploading piece...
Jun 26 06:08:21.142917062 (68h56m48.069211739s) [noise=true , address=104.36.82.7:4912 , piece_id=A5HPC7CEU66IFNS47HO3AQX22LUZ6AZ4L4XAMZ6T5NCVE4D3TAZA, seg_pos={0 0}, upload=55354]: Uploading piece...
Jun 26 06:08:21.221824189 (68h56m48.148118865s) [upload=55354]: Next returned nil segment
Jun 26 06:08:21.221837958 (68h56m48.148132644s) [upload=55354]: Waiting for error group managing segments...
Jun 26 06:08:21.229550628 (68h56m48.155845315s) [noise=true , address=104.36.82.14:4223 , piece_id=LP74YFI2SRSYPYEIKREVINCOFMVCXLNDS2555YKUVFN63N6TBJDA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: <nil>
Jun 26 06:08:21.230245145 (68h56m48.156539832s) [noise=true , address=104.36.82.14:4222 , piece_id=U3MBMAKFBFTOPJZ3HS3FXLPDBLQTN577XNCZLUBHRO6IU4BE5VNA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: <nil>
Jun 26 06:08:21.230801067 (68h56m48.15709574s) [noise=true , address=104.36.82.6:4873 , piece_id=TIIMEQUSDLRFF3OK3WBB74F7SWJ5MEC4Z6IQNELDMRZLNTMQT7LA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: <nil>
Jun 26 06:08:21.231457382 (68h56m48.157752071s) [noise=true , address=104.36.82.6:4876 , piece_id=SSMGONONY34EHF5W6QMKO7YXB7RLO5LZVTJMMFKTP6GQNGYFYQ4A, seg_pos={0 0}, upload=55354]: Done uploading piece. err: <nil>
Jun 26 06:08:21.231563205 (68h56m48.157857893s) [noise=true , address=104.36.82.13:4186 , piece_id=5SVABN3KS6AUL6HILHNQCVF74BA75VPKTTAK7IQAPKRRME3MOZEA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: <nil>
Jun 26 06:08:21.239131119 (68h56m48.165425799s) [noise=true , address=104.36.82.7:4912 , piece_id=A5HPC7CEU66IFNS47HO3AQX22LUZ6AZ4L4XAMZ6T5NCVE4D3TAZA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: <nil>
Jun 26 06:08:21.534686665 (68h56m48.460981337s) [noise=true , address=104.36.82.15:4258 , piece_id=YXS4J7YSG6JV33JG4BZLK5AXKE3JXLBTGZJXPXZQ4LBYBT3MU2AA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: <nil>
Jun 26 06:08:51.073673533 (68h57m17.999968219s) [noise=true , address=45.91.116.9:5261 , piece_id=CZWB3XJVA7B7R7GWFYNW3RCGNVRXQ3E37XDXD4SUXTEMQFZ73HHA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi, address:45.91.116.9:5261): protocol: rpc: dial tcp 45.91.116.9:5261: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jun 26 06:08:51.073739594 (68h57m18.000034272s) [noise=true , address=45.91.116.10:4082 , piece_id=7NVEGA4QQOE7KYPHXUUX42SJPQL6LP46RGM37VD6A2AWN7PAOM3Q, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj, address:45.91.116.10:4082): protocol: rpc: dial tcp 45.91.116.10:4082: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jun 26 06:08:51.073789897 (68h57m18.000084584s) [noise=true , address=77.90.31.19:4842 , piece_id=G3AG375DATHB7MGEEWONE5GMMIC7QHG2VVDKF5AEVOSXV5N4S6AA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: rpc: dial tcp 77.90.31.19:4842: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jun 26 06:08:51.073849570 (68h57m18.000144246s) [noise=true , address=45.91.116.13:4154 , piece_id=STFPXN2LUJYUIAYROWZIRTCMT2Y3XET6KWA4GZWUHENXRJURP62Q, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN, address:45.91.116.13:4154): protocol: write tcp 159.223.143.39:35392->45.91.116.13:4154: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jun 26 06:08:51.073904716 (68h57m18.000199391s) [noise=true , address=77.90.31.13:4626 , piece_id=MXYBIEDFAFK53CVTE2C6CKEZWOL3MLW4S7LOHK4IHW6QEIZJL76Q, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: write tcp 159.223.143.39:58864->77.90.31.13:4626: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jun 26 06:08:51.073952402 (68h57m18.000247082s) [noise=true , address=77.90.31.10:4517 , piece_id=VJZ5T4QNVAPNUHNFNK24S6TAS4V4NKNFMQE2FZMCWWLCCB2ELXXQ, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, address:77.90.31.10:4517): protocol: write tcp 159.223.143.39:36382->77.90.31.10:4517: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jun 26 06:08:51.074022481 (68h57m18.000317156s) [noise=true , address=77.90.31.15:4697 , piece_id=CUXGVL7PS3H7ZQKUCSYXJCBR3ETYFDPZFYIEAWEYL7C2S3RJY3AA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5, address:77.90.31.15:4697): protocol: write tcp 159.223.143.39:52730->77.90.31.15:4697: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jun 26 06:08:51.074071824 (68h57m18.000366503s) [noise=true , address=77.90.31.11:4554 , piece_id=R2CIFLWS4LXTGSDSIMIRC4UYVJTXC7T35HPWZ2S7VYUTJ2AMY3BQ, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, address:77.90.31.11:4554): protocol: rpc: dial tcp 77.90.31.11:4554: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jun 26 06:08:51.074102177 (68h57m18.000396855s) [noise=true , address=77.90.31.14:4661 , piece_id=VXI2ZUSGR4N3H7TPIK75RIDFP3SBAOWVRGHNXTZOHFT5FNG4JWJA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue, address:77.90.31.14:4661): protocol: rpc: dial tcp 77.90.31.14:4661: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jun 26 06:08:51.074119276 (68h57m18.000413966s) [seg_pos={0 0}, upload=55354]: Done waiting for segment. successful: 7 optimal: 13 errs: <nil>
Jun 26 06:08:51.100079867 (68h57m18.026374541s) [upload=55354]: Done uploading segments...
Jun 26 06:08:51.100114505 (68h57m18.026409191s) [upload=55354]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 7)
Duration: 442.063857ms
core
uplink
us-select-1
|
checkers.AuthServiceHTTP{"https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us1)antiblip:
1 out of 60 is failed (threshold=3) [..............X.............................................] duration:
170.325279ms
FAIL (2025-06-26 06:25:18 UTC)
orbiter warning: unexpected status code "500 Internal Server Error" "\n"
Duration: 170.352974ms
edge
authservice
http
bgp
us1
|
checkers.AuthServiceHTTP{"https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us1)antiblip:
1 out of 60 is failed (threshold=3) [...............X............................................] duration:
178.137521ms
FAIL (2025-06-26 06:24:59 UTC)
orbiter warning: unexpected status code "500 Internal Server Error" "\n"
Duration: 178.166335ms
edge
authservice
http
bgp
us1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [............X...............................................] duration:
701.7438ms
FAIL (2025-06-26 06:24:53 UTC)
orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded
Duration: 701.764982ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [....................................XX......................] duration:
513.202816ms
FAIL (2025-06-26 06:24:38 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 513.227323ms
edge
gateway
us-select-1
|
checkers.AuthServiceHTTP{"https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us1)antiblip:
1 out of 60 is failed (threshold=3) [................X...........................................] duration:
222.291633ms
FAIL (2025-06-26 06:24:19 UTC)
orbiter warning: unexpected status code "500 Internal Server Error" "\n"
Duration: 222.318135ms
edge
authservice
http
bgp
us1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
3 out of 60 is failed (threshold=3) [.....................................XXX....................] duration:
434.644655ms
FAIL (2025-06-26 06:24:15 UTC)
orbiter warning:
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 7)
======LOGS======
pload=55354]: Got next segment. Inline: false
Jun 26 06:08:21.105827735 (68h56m48.032122411s) [upload=55354]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jun 26 06:08:21.105833588 (68h56m48.032128274s) [upload=55354]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jun 26 06:08:21.135318313 (68h56m48.061613009s) [seg_pos={0 0}, upload=55354]: Begin upload segment...
Jun 26 06:08:21.135366534 (68h56m48.061661214s) [seg_pos={0 0}, upload=55354]: Done begin upload segment.
Jun 26 06:08:21.135390342 (68h56m48.061685028s) [noise=true , address=45.91.116.9:5261 , piece_id=CZWB3XJVA7B7R7GWFYNW3RCGNVRXQ3E37XDXD4SUXTEMQFZ73HHA, seg_pos={0 0}, upload=55354]: Uploading piece...
Jun 26 06:08:21.135405801 (68h56m48.061700497s) [noise=true , address=104.36.82.6:4873 , piece_id=TIIMEQUSDLRFF3OK3WBB74F7SWJ5MEC4Z6IQNELDMRZLNTMQT7LA, seg_pos={0 0}, upload=55354]: Uploading piece...
Jun 26 06:08:21.135774729 (68h56m48.062069418s) [noise=true , address=77.90.31.13:4626 , piece_id=MXYBIEDFAFK53CVTE2C6CKEZWOL3MLW4S7LOHK4IHW6QEIZJL76Q, seg_pos={0 0}, upload=55354]: Uploading piece...
Jun 26 06:08:21.135944436 (68h56m48.06223913s) [noise=true , address=77.90.31.10:4517 , piece_id=VJZ5T4QNVAPNUHNFNK24S6TAS4V4NKNFMQE2FZMCWWLCCB2ELXXQ, seg_pos={0 0}, upload=55354]: Uploading piece...
Jun 26 06:08:21.136080402 (68h56m48.062375094s) [noise=true , address=77.90.31.15:4697 , piece_id=CUXGVL7PS3H7ZQKUCSYXJCBR3ETYFDPZFYIEAWEYL7C2S3RJY3AA, seg_pos={0 0}, upload=55354]: Uploading piece...
Jun 26 06:08:21.136382603 (68h56m48.062677289s) [noise=true , address=104.36.82.13:4186 , piece_id=5SVABN3KS6AUL6HILHNQCVF74BA75VPKTTAK7IQAPKRRME3MOZEA, seg_pos={0 0}, upload=55354]: Uploading piece...
Jun 26 06:08:21.136399304 (68h56m48.062693995s) [noise=true , address=104.36.82.14:4223 , piece_id=LP74YFI2SRSYPYEIKREVINCOFMVCXLNDS2555YKUVFN63N6TBJDA, seg_pos={0 0}, upload=55354]: Uploading piece...
Jun 26 06:08:21.136669576 (68h56m48.062964262s) [noise=true , address=104.36.82.6:4876 , piece_id=SSMGONONY34EHF5W6QMKO7YXB7RLO5LZVTJMMFKTP6GQNGYFYQ4A, seg_pos={0 0}, upload=55354]: Uploading piece...
Jun 26 06:08:21.136809784 (68h56m48.063104477s) [noise=true , address=104.36.82.15:4258 , piece_id=YXS4J7YSG6JV33JG4BZLK5AXKE3JXLBTGZJXPXZQ4LBYBT3MU2AA, seg_pos={0 0}, upload=55354]: Uploading piece...
Jun 26 06:08:21.136950556 (68h56m48.063245242s) [noise=true , address=77.90.31.11:4554 , piece_id=R2CIFLWS4LXTGSDSIMIRC4UYVJTXC7T35HPWZ2S7VYUTJ2AMY3BQ, seg_pos={0 0}, upload=55354]: Uploading piece...
Jun 26 06:08:21.137226497 (68h56m48.06352119s) [noise=true , address=77.90.31.14:4661 , piece_id=VXI2ZUSGR4N3H7TPIK75RIDFP3SBAOWVRGHNXTZOHFT5FNG4JWJA, seg_pos={0 0}, upload=55354]: Uploading piece...
Jun 26 06:08:21.138099107 (68h56m48.064393792s) [noise=true , address=45.91.116.10:4082 , piece_id=7NVEGA4QQOE7KYPHXUUX42SJPQL6LP46RGM37VD6A2AWN7PAOM3Q, seg_pos={0 0}, upload=55354]: Uploading piece...
Jun 26 06:08:21.139330448 (68h56m48.065625141s) [noise=true , address=104.36.82.14:4222 , piece_id=U3MBMAKFBFTOPJZ3HS3FXLPDBLQTN577XNCZLUBHRO6IU4BE5VNA, seg_pos={0 0}, upload=55354]: Uploading piece...
Jun 26 06:08:21.140580734 (68h56m48.06687543s) [noise=true , address=77.90.31.19:4842 , piece_id=G3AG375DATHB7MGEEWONE5GMMIC7QHG2VVDKF5AEVOSXV5N4S6AA, seg_pos={0 0}, upload=55354]: Uploading piece...
Jun 26 06:08:21.141452300 (68h56m48.067747027s) [noise=true , address=45.91.116.13:4154 , piece_id=STFPXN2LUJYUIAYROWZIRTCMT2Y3XET6KWA4GZWUHENXRJURP62Q, seg_pos={0 0}, upload=55354]: Uploading piece...
Jun 26 06:08:21.142917062 (68h56m48.069211739s) [noise=true , address=104.36.82.7:4912 , piece_id=A5HPC7CEU66IFNS47HO3AQX22LUZ6AZ4L4XAMZ6T5NCVE4D3TAZA, seg_pos={0 0}, upload=55354]: Uploading piece...
Jun 26 06:08:21.221824189 (68h56m48.148118865s) [upload=55354]: Next returned nil segment
Jun 26 06:08:21.221837958 (68h56m48.148132644s) [upload=55354]: Waiting for error group managing segments...
Jun 26 06:08:21.229550628 (68h56m48.155845315s) [noise=true , address=104.36.82.14:4223 , piece_id=LP74YFI2SRSYPYEIKREVINCOFMVCXLNDS2555YKUVFN63N6TBJDA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: <nil>
Jun 26 06:08:21.230245145 (68h56m48.156539832s) [noise=true , address=104.36.82.14:4222 , piece_id=U3MBMAKFBFTOPJZ3HS3FXLPDBLQTN577XNCZLUBHRO6IU4BE5VNA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: <nil>
Jun 26 06:08:21.230801067 (68h56m48.15709574s) [noise=true , address=104.36.82.6:4873 , piece_id=TIIMEQUSDLRFF3OK3WBB74F7SWJ5MEC4Z6IQNELDMRZLNTMQT7LA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: <nil>
Jun 26 06:08:21.231457382 (68h56m48.157752071s) [noise=true , address=104.36.82.6:4876 , piece_id=SSMGONONY34EHF5W6QMKO7YXB7RLO5LZVTJMMFKTP6GQNGYFYQ4A, seg_pos={0 0}, upload=55354]: Done uploading piece. err: <nil>
Jun 26 06:08:21.231563205 (68h56m48.157857893s) [noise=true , address=104.36.82.13:4186 , piece_id=5SVABN3KS6AUL6HILHNQCVF74BA75VPKTTAK7IQAPKRRME3MOZEA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: <nil>
Jun 26 06:08:21.239131119 (68h56m48.165425799s) [noise=true , address=104.36.82.7:4912 , piece_id=A5HPC7CEU66IFNS47HO3AQX22LUZ6AZ4L4XAMZ6T5NCVE4D3TAZA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: <nil>
Jun 26 06:08:21.534686665 (68h56m48.460981337s) [noise=true , address=104.36.82.15:4258 , piece_id=YXS4J7YSG6JV33JG4BZLK5AXKE3JXLBTGZJXPXZQ4LBYBT3MU2AA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: <nil>
Jun 26 06:08:51.073673533 (68h57m17.999968219s) [noise=true , address=45.91.116.9:5261 , piece_id=CZWB3XJVA7B7R7GWFYNW3RCGNVRXQ3E37XDXD4SUXTEMQFZ73HHA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi, address:45.91.116.9:5261): protocol: rpc: dial tcp 45.91.116.9:5261: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jun 26 06:08:51.073739594 (68h57m18.000034272s) [noise=true , address=45.91.116.10:4082 , piece_id=7NVEGA4QQOE7KYPHXUUX42SJPQL6LP46RGM37VD6A2AWN7PAOM3Q, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj, address:45.91.116.10:4082): protocol: rpc: dial tcp 45.91.116.10:4082: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jun 26 06:08:51.073789897 (68h57m18.000084584s) [noise=true , address=77.90.31.19:4842 , piece_id=G3AG375DATHB7MGEEWONE5GMMIC7QHG2VVDKF5AEVOSXV5N4S6AA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: rpc: dial tcp 77.90.31.19:4842: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jun 26 06:08:51.073849570 (68h57m18.000144246s) [noise=true , address=45.91.116.13:4154 , piece_id=STFPXN2LUJYUIAYROWZIRTCMT2Y3XET6KWA4GZWUHENXRJURP62Q, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN, address:45.91.116.13:4154): protocol: write tcp 159.223.143.39:35392->45.91.116.13:4154: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jun 26 06:08:51.073904716 (68h57m18.000199391s) [noise=true , address=77.90.31.13:4626 , piece_id=MXYBIEDFAFK53CVTE2C6CKEZWOL3MLW4S7LOHK4IHW6QEIZJL76Q, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: write tcp 159.223.143.39:58864->77.90.31.13:4626: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jun 26 06:08:51.073952402 (68h57m18.000247082s) [noise=true , address=77.90.31.10:4517 , piece_id=VJZ5T4QNVAPNUHNFNK24S6TAS4V4NKNFMQE2FZMCWWLCCB2ELXXQ, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, address:77.90.31.10:4517): protocol: write tcp 159.223.143.39:36382->77.90.31.10:4517: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jun 26 06:08:51.074022481 (68h57m18.000317156s) [noise=true , address=77.90.31.15:4697 , piece_id=CUXGVL7PS3H7ZQKUCSYXJCBR3ETYFDPZFYIEAWEYL7C2S3RJY3AA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5, address:77.90.31.15:4697): protocol: write tcp 159.223.143.39:52730->77.90.31.15:4697: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jun 26 06:08:51.074071824 (68h57m18.000366503s) [noise=true , address=77.90.31.11:4554 , piece_id=R2CIFLWS4LXTGSDSIMIRC4UYVJTXC7T35HPWZ2S7VYUTJ2AMY3BQ, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, address:77.90.31.11:4554): protocol: rpc: dial tcp 77.90.31.11:4554: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jun 26 06:08:51.074102177 (68h57m18.000396855s) [noise=true , address=77.90.31.14:4661 , piece_id=VXI2ZUSGR4N3H7TPIK75RIDFP3SBAOWVRGHNXTZOHFT5FNG4JWJA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue, address:77.90.31.14:4661): protocol: rpc: dial tcp 77.90.31.14:4661: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jun 26 06:08:51.074119276 (68h57m18.000413966s) [seg_pos={0 0}, upload=55354]: Done waiting for segment. successful: 7 optimal: 13 errs: <nil>
Jun 26 06:08:51.100079867 (68h57m18.026374541s) [upload=55354]: Done uploading segments...
Jun 26 06:08:51.100114505 (68h57m18.026409191s) [upload=55354]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 7)
Duration: 434.679904ms
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [.....................................XX.....................] duration:
470.473083ms
FAIL (2025-06-26 06:23:59 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 470.501668ms
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
3 out of 60 is failed (threshold=3) [......................................XXX...................] duration:
584.266768ms
FAIL (2025-06-26 06:23:47 UTC)
orbiter warning:
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 7)
======LOGS======
pload=55354]: Got next segment. Inline: false
Jun 26 06:08:21.105827735 (68h56m48.032122411s) [upload=55354]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jun 26 06:08:21.105833588 (68h56m48.032128274s) [upload=55354]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jun 26 06:08:21.135318313 (68h56m48.061613009s) [seg_pos={0 0}, upload=55354]: Begin upload segment...
Jun 26 06:08:21.135366534 (68h56m48.061661214s) [seg_pos={0 0}, upload=55354]: Done begin upload segment.
Jun 26 06:08:21.135390342 (68h56m48.061685028s) [noise=true , address=45.91.116.9:5261 , piece_id=CZWB3XJVA7B7R7GWFYNW3RCGNVRXQ3E37XDXD4SUXTEMQFZ73HHA, seg_pos={0 0}, upload=55354]: Uploading piece...
Jun 26 06:08:21.135405801 (68h56m48.061700497s) [noise=true , address=104.36.82.6:4873 , piece_id=TIIMEQUSDLRFF3OK3WBB74F7SWJ5MEC4Z6IQNELDMRZLNTMQT7LA, seg_pos={0 0}, upload=55354]: Uploading piece...
Jun 26 06:08:21.135774729 (68h56m48.062069418s) [noise=true , address=77.90.31.13:4626 , piece_id=MXYBIEDFAFK53CVTE2C6CKEZWOL3MLW4S7LOHK4IHW6QEIZJL76Q, seg_pos={0 0}, upload=55354]: Uploading piece...
Jun 26 06:08:21.135944436 (68h56m48.06223913s) [noise=true , address=77.90.31.10:4517 , piece_id=VJZ5T4QNVAPNUHNFNK24S6TAS4V4NKNFMQE2FZMCWWLCCB2ELXXQ, seg_pos={0 0}, upload=55354]: Uploading piece...
Jun 26 06:08:21.136080402 (68h56m48.062375094s) [noise=true , address=77.90.31.15:4697 , piece_id=CUXGVL7PS3H7ZQKUCSYXJCBR3ETYFDPZFYIEAWEYL7C2S3RJY3AA, seg_pos={0 0}, upload=55354]: Uploading piece...
Jun 26 06:08:21.136382603 (68h56m48.062677289s) [noise=true , address=104.36.82.13:4186 , piece_id=5SVABN3KS6AUL6HILHNQCVF74BA75VPKTTAK7IQAPKRRME3MOZEA, seg_pos={0 0}, upload=55354]: Uploading piece...
Jun 26 06:08:21.136399304 (68h56m48.062693995s) [noise=true , address=104.36.82.14:4223 , piece_id=LP74YFI2SRSYPYEIKREVINCOFMVCXLNDS2555YKUVFN63N6TBJDA, seg_pos={0 0}, upload=55354]: Uploading piece...
Jun 26 06:08:21.136669576 (68h56m48.062964262s) [noise=true , address=104.36.82.6:4876 , piece_id=SSMGONONY34EHF5W6QMKO7YXB7RLO5LZVTJMMFKTP6GQNGYFYQ4A, seg_pos={0 0}, upload=55354]: Uploading piece...
Jun 26 06:08:21.136809784 (68h56m48.063104477s) [noise=true , address=104.36.82.15:4258 , piece_id=YXS4J7YSG6JV33JG4BZLK5AXKE3JXLBTGZJXPXZQ4LBYBT3MU2AA, seg_pos={0 0}, upload=55354]: Uploading piece...
Jun 26 06:08:21.136950556 (68h56m48.063245242s) [noise=true , address=77.90.31.11:4554 , piece_id=R2CIFLWS4LXTGSDSIMIRC4UYVJTXC7T35HPWZ2S7VYUTJ2AMY3BQ, seg_pos={0 0}, upload=55354]: Uploading piece...
Jun 26 06:08:21.137226497 (68h56m48.06352119s) [noise=true , address=77.90.31.14:4661 , piece_id=VXI2ZUSGR4N3H7TPIK75RIDFP3SBAOWVRGHNXTZOHFT5FNG4JWJA, seg_pos={0 0}, upload=55354]: Uploading piece...
Jun 26 06:08:21.138099107 (68h56m48.064393792s) [noise=true , address=45.91.116.10:4082 , piece_id=7NVEGA4QQOE7KYPHXUUX42SJPQL6LP46RGM37VD6A2AWN7PAOM3Q, seg_pos={0 0}, upload=55354]: Uploading piece...
Jun 26 06:08:21.139330448 (68h56m48.065625141s) [noise=true , address=104.36.82.14:4222 , piece_id=U3MBMAKFBFTOPJZ3HS3FXLPDBLQTN577XNCZLUBHRO6IU4BE5VNA, seg_pos={0 0}, upload=55354]: Uploading piece...
Jun 26 06:08:21.140580734 (68h56m48.06687543s) [noise=true , address=77.90.31.19:4842 , piece_id=G3AG375DATHB7MGEEWONE5GMMIC7QHG2VVDKF5AEVOSXV5N4S6AA, seg_pos={0 0}, upload=55354]: Uploading piece...
Jun 26 06:08:21.141452300 (68h56m48.067747027s) [noise=true , address=45.91.116.13:4154 , piece_id=STFPXN2LUJYUIAYROWZIRTCMT2Y3XET6KWA4GZWUHENXRJURP62Q, seg_pos={0 0}, upload=55354]: Uploading piece...
Jun 26 06:08:21.142917062 (68h56m48.069211739s) [noise=true , address=104.36.82.7:4912 , piece_id=A5HPC7CEU66IFNS47HO3AQX22LUZ6AZ4L4XAMZ6T5NCVE4D3TAZA, seg_pos={0 0}, upload=55354]: Uploading piece...
Jun 26 06:08:21.221824189 (68h56m48.148118865s) [upload=55354]: Next returned nil segment
Jun 26 06:08:21.221837958 (68h56m48.148132644s) [upload=55354]: Waiting for error group managing segments...
Jun 26 06:08:21.229550628 (68h56m48.155845315s) [noise=true , address=104.36.82.14:4223 , piece_id=LP74YFI2SRSYPYEIKREVINCOFMVCXLNDS2555YKUVFN63N6TBJDA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: <nil>
Jun 26 06:08:21.230245145 (68h56m48.156539832s) [noise=true , address=104.36.82.14:4222 , piece_id=U3MBMAKFBFTOPJZ3HS3FXLPDBLQTN577XNCZLUBHRO6IU4BE5VNA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: <nil>
Jun 26 06:08:21.230801067 (68h56m48.15709574s) [noise=true , address=104.36.82.6:4873 , piece_id=TIIMEQUSDLRFF3OK3WBB74F7SWJ5MEC4Z6IQNELDMRZLNTMQT7LA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: <nil>
Jun 26 06:08:21.231457382 (68h56m48.157752071s) [noise=true , address=104.36.82.6:4876 , piece_id=SSMGONONY34EHF5W6QMKO7YXB7RLO5LZVTJMMFKTP6GQNGYFYQ4A, seg_pos={0 0}, upload=55354]: Done uploading piece. err: <nil>
Jun 26 06:08:21.231563205 (68h56m48.157857893s) [noise=true , address=104.36.82.13:4186 , piece_id=5SVABN3KS6AUL6HILHNQCVF74BA75VPKTTAK7IQAPKRRME3MOZEA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: <nil>
Jun 26 06:08:21.239131119 (68h56m48.165425799s) [noise=true , address=104.36.82.7:4912 , piece_id=A5HPC7CEU66IFNS47HO3AQX22LUZ6AZ4L4XAMZ6T5NCVE4D3TAZA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: <nil>
Jun 26 06:08:21.534686665 (68h56m48.460981337s) [noise=true , address=104.36.82.15:4258 , piece_id=YXS4J7YSG6JV33JG4BZLK5AXKE3JXLBTGZJXPXZQ4LBYBT3MU2AA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: <nil>
Jun 26 06:08:51.073673533 (68h57m17.999968219s) [noise=true , address=45.91.116.9:5261 , piece_id=CZWB3XJVA7B7R7GWFYNW3RCGNVRXQ3E37XDXD4SUXTEMQFZ73HHA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi, address:45.91.116.9:5261): protocol: rpc: dial tcp 45.91.116.9:5261: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jun 26 06:08:51.073739594 (68h57m18.000034272s) [noise=true , address=45.91.116.10:4082 , piece_id=7NVEGA4QQOE7KYPHXUUX42SJPQL6LP46RGM37VD6A2AWN7PAOM3Q, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj, address:45.91.116.10:4082): protocol: rpc: dial tcp 45.91.116.10:4082: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jun 26 06:08:51.073789897 (68h57m18.000084584s) [noise=true , address=77.90.31.19:4842 , piece_id=G3AG375DATHB7MGEEWONE5GMMIC7QHG2VVDKF5AEVOSXV5N4S6AA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: rpc: dial tcp 77.90.31.19:4842: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jun 26 06:08:51.073849570 (68h57m18.000144246s) [noise=true , address=45.91.116.13:4154 , piece_id=STFPXN2LUJYUIAYROWZIRTCMT2Y3XET6KWA4GZWUHENXRJURP62Q, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN, address:45.91.116.13:4154): protocol: write tcp 159.223.143.39:35392->45.91.116.13:4154: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jun 26 06:08:51.073904716 (68h57m18.000199391s) [noise=true , address=77.90.31.13:4626 , piece_id=MXYBIEDFAFK53CVTE2C6CKEZWOL3MLW4S7LOHK4IHW6QEIZJL76Q, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: write tcp 159.223.143.39:58864->77.90.31.13:4626: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jun 26 06:08:51.073952402 (68h57m18.000247082s) [noise=true , address=77.90.31.10:4517 , piece_id=VJZ5T4QNVAPNUHNFNK24S6TAS4V4NKNFMQE2FZMCWWLCCB2ELXXQ, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, address:77.90.31.10:4517): protocol: write tcp 159.223.143.39:36382->77.90.31.10:4517: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jun 26 06:08:51.074022481 (68h57m18.000317156s) [noise=true , address=77.90.31.15:4697 , piece_id=CUXGVL7PS3H7ZQKUCSYXJCBR3ETYFDPZFYIEAWEYL7C2S3RJY3AA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5, address:77.90.31.15:4697): protocol: write tcp 159.223.143.39:52730->77.90.31.15:4697: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jun 26 06:08:51.074071824 (68h57m18.000366503s) [noise=true , address=77.90.31.11:4554 , piece_id=R2CIFLWS4LXTGSDSIMIRC4UYVJTXC7T35HPWZ2S7VYUTJ2AMY3BQ, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, address:77.90.31.11:4554): protocol: rpc: dial tcp 77.90.31.11:4554: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jun 26 06:08:51.074102177 (68h57m18.000396855s) [noise=true , address=77.90.31.14:4661 , piece_id=VXI2ZUSGR4N3H7TPIK75RIDFP3SBAOWVRGHNXTZOHFT5FNG4JWJA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue, address:77.90.31.14:4661): protocol: rpc: dial tcp 77.90.31.14:4661: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jun 26 06:08:51.074119276 (68h57m18.000413966s) [seg_pos={0 0}, upload=55354]: Done waiting for segment. successful: 7 optimal: 13 errs: <nil>
Jun 26 06:08:51.100079867 (68h57m18.026374541s) [upload=55354]: Done uploading segments...
Jun 26 06:08:51.100114505 (68h57m18.026409191s) [upload=55354]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 7)
Duration: 584.298843ms
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [.............X..............................................] duration:
672.880204ms
FAIL (2025-06-26 06:23:39 UTC)
orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded
Duration: 672.903111ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
3 out of 60 is failed (threshold=3) [.......................................XXX..................] duration:
446.54512ms
FAIL (2025-06-26 06:23:30 UTC)
orbiter warning:
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 7)
======LOGS======
pload=55354]: Got next segment. Inline: false
Jun 26 06:08:21.105827735 (68h56m48.032122411s) [upload=55354]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jun 26 06:08:21.105833588 (68h56m48.032128274s) [upload=55354]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jun 26 06:08:21.135318313 (68h56m48.061613009s) [seg_pos={0 0}, upload=55354]: Begin upload segment...
Jun 26 06:08:21.135366534 (68h56m48.061661214s) [seg_pos={0 0}, upload=55354]: Done begin upload segment.
Jun 26 06:08:21.135390342 (68h56m48.061685028s) [noise=true , address=45.91.116.9:5261 , piece_id=CZWB3XJVA7B7R7GWFYNW3RCGNVRXQ3E37XDXD4SUXTEMQFZ73HHA, seg_pos={0 0}, upload=55354]: Uploading piece...
Jun 26 06:08:21.135405801 (68h56m48.061700497s) [noise=true , address=104.36.82.6:4873 , piece_id=TIIMEQUSDLRFF3OK3WBB74F7SWJ5MEC4Z6IQNELDMRZLNTMQT7LA, seg_pos={0 0}, upload=55354]: Uploading piece...
Jun 26 06:08:21.135774729 (68h56m48.062069418s) [noise=true , address=77.90.31.13:4626 , piece_id=MXYBIEDFAFK53CVTE2C6CKEZWOL3MLW4S7LOHK4IHW6QEIZJL76Q, seg_pos={0 0}, upload=55354]: Uploading piece...
Jun 26 06:08:21.135944436 (68h56m48.06223913s) [noise=true , address=77.90.31.10:4517 , piece_id=VJZ5T4QNVAPNUHNFNK24S6TAS4V4NKNFMQE2FZMCWWLCCB2ELXXQ, seg_pos={0 0}, upload=55354]: Uploading piece...
Jun 26 06:08:21.136080402 (68h56m48.062375094s) [noise=true , address=77.90.31.15:4697 , piece_id=CUXGVL7PS3H7ZQKUCSYXJCBR3ETYFDPZFYIEAWEYL7C2S3RJY3AA, seg_pos={0 0}, upload=55354]: Uploading piece...
Jun 26 06:08:21.136382603 (68h56m48.062677289s) [noise=true , address=104.36.82.13:4186 , piece_id=5SVABN3KS6AUL6HILHNQCVF74BA75VPKTTAK7IQAPKRRME3MOZEA, seg_pos={0 0}, upload=55354]: Uploading piece...
Jun 26 06:08:21.136399304 (68h56m48.062693995s) [noise=true , address=104.36.82.14:4223 , piece_id=LP74YFI2SRSYPYEIKREVINCOFMVCXLNDS2555YKUVFN63N6TBJDA, seg_pos={0 0}, upload=55354]: Uploading piece...
Jun 26 06:08:21.136669576 (68h56m48.062964262s) [noise=true , address=104.36.82.6:4876 , piece_id=SSMGONONY34EHF5W6QMKO7YXB7RLO5LZVTJMMFKTP6GQNGYFYQ4A, seg_pos={0 0}, upload=55354]: Uploading piece...
Jun 26 06:08:21.136809784 (68h56m48.063104477s) [noise=true , address=104.36.82.15:4258 , piece_id=YXS4J7YSG6JV33JG4BZLK5AXKE3JXLBTGZJXPXZQ4LBYBT3MU2AA, seg_pos={0 0}, upload=55354]: Uploading piece...
Jun 26 06:08:21.136950556 (68h56m48.063245242s) [noise=true , address=77.90.31.11:4554 , piece_id=R2CIFLWS4LXTGSDSIMIRC4UYVJTXC7T35HPWZ2S7VYUTJ2AMY3BQ, seg_pos={0 0}, upload=55354]: Uploading piece...
Jun 26 06:08:21.137226497 (68h56m48.06352119s) [noise=true , address=77.90.31.14:4661 , piece_id=VXI2ZUSGR4N3H7TPIK75RIDFP3SBAOWVRGHNXTZOHFT5FNG4JWJA, seg_pos={0 0}, upload=55354]: Uploading piece...
Jun 26 06:08:21.138099107 (68h56m48.064393792s) [noise=true , address=45.91.116.10:4082 , piece_id=7NVEGA4QQOE7KYPHXUUX42SJPQL6LP46RGM37VD6A2AWN7PAOM3Q, seg_pos={0 0}, upload=55354]: Uploading piece...
Jun 26 06:08:21.139330448 (68h56m48.065625141s) [noise=true , address=104.36.82.14:4222 , piece_id=U3MBMAKFBFTOPJZ3HS3FXLPDBLQTN577XNCZLUBHRO6IU4BE5VNA, seg_pos={0 0}, upload=55354]: Uploading piece...
Jun 26 06:08:21.140580734 (68h56m48.06687543s) [noise=true , address=77.90.31.19:4842 , piece_id=G3AG375DATHB7MGEEWONE5GMMIC7QHG2VVDKF5AEVOSXV5N4S6AA, seg_pos={0 0}, upload=55354]: Uploading piece...
Jun 26 06:08:21.141452300 (68h56m48.067747027s) [noise=true , address=45.91.116.13:4154 , piece_id=STFPXN2LUJYUIAYROWZIRTCMT2Y3XET6KWA4GZWUHENXRJURP62Q, seg_pos={0 0}, upload=55354]: Uploading piece...
Jun 26 06:08:21.142917062 (68h56m48.069211739s) [noise=true , address=104.36.82.7:4912 , piece_id=A5HPC7CEU66IFNS47HO3AQX22LUZ6AZ4L4XAMZ6T5NCVE4D3TAZA, seg_pos={0 0}, upload=55354]: Uploading piece...
Jun 26 06:08:21.221824189 (68h56m48.148118865s) [upload=55354]: Next returned nil segment
Jun 26 06:08:21.221837958 (68h56m48.148132644s) [upload=55354]: Waiting for error group managing segments...
Jun 26 06:08:21.229550628 (68h56m48.155845315s) [noise=true , address=104.36.82.14:4223 , piece_id=LP74YFI2SRSYPYEIKREVINCOFMVCXLNDS2555YKUVFN63N6TBJDA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: <nil>
Jun 26 06:08:21.230245145 (68h56m48.156539832s) [noise=true , address=104.36.82.14:4222 , piece_id=U3MBMAKFBFTOPJZ3HS3FXLPDBLQTN577XNCZLUBHRO6IU4BE5VNA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: <nil>
Jun 26 06:08:21.230801067 (68h56m48.15709574s) [noise=true , address=104.36.82.6:4873 , piece_id=TIIMEQUSDLRFF3OK3WBB74F7SWJ5MEC4Z6IQNELDMRZLNTMQT7LA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: <nil>
Jun 26 06:08:21.231457382 (68h56m48.157752071s) [noise=true , address=104.36.82.6:4876 , piece_id=SSMGONONY34EHF5W6QMKO7YXB7RLO5LZVTJMMFKTP6GQNGYFYQ4A, seg_pos={0 0}, upload=55354]: Done uploading piece. err: <nil>
Jun 26 06:08:21.231563205 (68h56m48.157857893s) [noise=true , address=104.36.82.13:4186 , piece_id=5SVABN3KS6AUL6HILHNQCVF74BA75VPKTTAK7IQAPKRRME3MOZEA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: <nil>
Jun 26 06:08:21.239131119 (68h56m48.165425799s) [noise=true , address=104.36.82.7:4912 , piece_id=A5HPC7CEU66IFNS47HO3AQX22LUZ6AZ4L4XAMZ6T5NCVE4D3TAZA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: <nil>
Jun 26 06:08:21.534686665 (68h56m48.460981337s) [noise=true , address=104.36.82.15:4258 , piece_id=YXS4J7YSG6JV33JG4BZLK5AXKE3JXLBTGZJXPXZQ4LBYBT3MU2AA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: <nil>
Jun 26 06:08:51.073673533 (68h57m17.999968219s) [noise=true , address=45.91.116.9:5261 , piece_id=CZWB3XJVA7B7R7GWFYNW3RCGNVRXQ3E37XDXD4SUXTEMQFZ73HHA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi, address:45.91.116.9:5261): protocol: rpc: dial tcp 45.91.116.9:5261: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jun 26 06:08:51.073739594 (68h57m18.000034272s) [noise=true , address=45.91.116.10:4082 , piece_id=7NVEGA4QQOE7KYPHXUUX42SJPQL6LP46RGM37VD6A2AWN7PAOM3Q, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj, address:45.91.116.10:4082): protocol: rpc: dial tcp 45.91.116.10:4082: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jun 26 06:08:51.073789897 (68h57m18.000084584s) [noise=true , address=77.90.31.19:4842 , piece_id=G3AG375DATHB7MGEEWONE5GMMIC7QHG2VVDKF5AEVOSXV5N4S6AA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: rpc: dial tcp 77.90.31.19:4842: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jun 26 06:08:51.073849570 (68h57m18.000144246s) [noise=true , address=45.91.116.13:4154 , piece_id=STFPXN2LUJYUIAYROWZIRTCMT2Y3XET6KWA4GZWUHENXRJURP62Q, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN, address:45.91.116.13:4154): protocol: write tcp 159.223.143.39:35392->45.91.116.13:4154: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jun 26 06:08:51.073904716 (68h57m18.000199391s) [noise=true , address=77.90.31.13:4626 , piece_id=MXYBIEDFAFK53CVTE2C6CKEZWOL3MLW4S7LOHK4IHW6QEIZJL76Q, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: write tcp 159.223.143.39:58864->77.90.31.13:4626: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jun 26 06:08:51.073952402 (68h57m18.000247082s) [noise=true , address=77.90.31.10:4517 , piece_id=VJZ5T4QNVAPNUHNFNK24S6TAS4V4NKNFMQE2FZMCWWLCCB2ELXXQ, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, address:77.90.31.10:4517): protocol: write tcp 159.223.143.39:36382->77.90.31.10:4517: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jun 26 06:08:51.074022481 (68h57m18.000317156s) [noise=true , address=77.90.31.15:4697 , piece_id=CUXGVL7PS3H7ZQKUCSYXJCBR3ETYFDPZFYIEAWEYL7C2S3RJY3AA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5, address:77.90.31.15:4697): protocol: write tcp 159.223.143.39:52730->77.90.31.15:4697: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jun 26 06:08:51.074071824 (68h57m18.000366503s) [noise=true , address=77.90.31.11:4554 , piece_id=R2CIFLWS4LXTGSDSIMIRC4UYVJTXC7T35HPWZ2S7VYUTJ2AMY3BQ, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, address:77.90.31.11:4554): protocol: rpc: dial tcp 77.90.31.11:4554: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jun 26 06:08:51.074102177 (68h57m18.000396855s) [noise=true , address=77.90.31.14:4661 , piece_id=VXI2ZUSGR4N3H7TPIK75RIDFP3SBAOWVRGHNXTZOHFT5FNG4JWJA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue, address:77.90.31.14:4661): protocol: rpc: dial tcp 77.90.31.14:4661: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jun 26 06:08:51.074119276 (68h57m18.000413966s) [seg_pos={0 0}, upload=55354]: Done waiting for segment. successful: 7 optimal: 13 errs: <nil>
Jun 26 06:08:51.100079867 (68h57m18.026374541s) [upload=55354]: Done uploading segments...
Jun 26 06:08:51.100114505 (68h57m18.026409191s) [upload=55354]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 7)
Duration: 446.57281ms
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [......................................XX....................] duration:
475.528438ms
FAIL (2025-06-26 06:23:26 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 475.550721ms
edge
gateway
us-select-1
|
checkers.AuthServiceHTTP{"https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us1)antiblip:
1 out of 60 is failed (threshold=3) [.................X..........................................] duration:
159.723504ms
FAIL (2025-06-26 06:23:10 UTC)
orbiter warning: unexpected status code "500 Internal Server Error" "\n"
Duration: 159.748007ms
edge
authservice
http
bgp
us1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [.......................................XX...................] duration:
459.748524ms
FAIL (2025-06-26 06:22:41 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 459.773018ms
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [..............X.............................................] duration:
666.557171ms
FAIL (2025-06-26 06:22:35 UTC)
orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded
Duration: 666.578646ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
3 out of 60 is failed (threshold=3) [........................................XXX.................] duration:
602.176722ms
FAIL (2025-06-26 06:22:29 UTC)
orbiter warning:
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 7)
======LOGS======
pload=55354]: Got next segment. Inline: false
Jun 26 06:08:21.105827735 (68h56m48.032122411s) [upload=55354]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jun 26 06:08:21.105833588 (68h56m48.032128274s) [upload=55354]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jun 26 06:08:21.135318313 (68h56m48.061613009s) [seg_pos={0 0}, upload=55354]: Begin upload segment...
Jun 26 06:08:21.135366534 (68h56m48.061661214s) [seg_pos={0 0}, upload=55354]: Done begin upload segment.
Jun 26 06:08:21.135390342 (68h56m48.061685028s) [noise=true , address=45.91.116.9:5261 , piece_id=CZWB3XJVA7B7R7GWFYNW3RCGNVRXQ3E37XDXD4SUXTEMQFZ73HHA, seg_pos={0 0}, upload=55354]: Uploading piece...
Jun 26 06:08:21.135405801 (68h56m48.061700497s) [noise=true , address=104.36.82.6:4873 , piece_id=TIIMEQUSDLRFF3OK3WBB74F7SWJ5MEC4Z6IQNELDMRZLNTMQT7LA, seg_pos={0 0}, upload=55354]: Uploading piece...
Jun 26 06:08:21.135774729 (68h56m48.062069418s) [noise=true , address=77.90.31.13:4626 , piece_id=MXYBIEDFAFK53CVTE2C6CKEZWOL3MLW4S7LOHK4IHW6QEIZJL76Q, seg_pos={0 0}, upload=55354]: Uploading piece...
Jun 26 06:08:21.135944436 (68h56m48.06223913s) [noise=true , address=77.90.31.10:4517 , piece_id=VJZ5T4QNVAPNUHNFNK24S6TAS4V4NKNFMQE2FZMCWWLCCB2ELXXQ, seg_pos={0 0}, upload=55354]: Uploading piece...
Jun 26 06:08:21.136080402 (68h56m48.062375094s) [noise=true , address=77.90.31.15:4697 , piece_id=CUXGVL7PS3H7ZQKUCSYXJCBR3ETYFDPZFYIEAWEYL7C2S3RJY3AA, seg_pos={0 0}, upload=55354]: Uploading piece...
Jun 26 06:08:21.136382603 (68h56m48.062677289s) [noise=true , address=104.36.82.13:4186 , piece_id=5SVABN3KS6AUL6HILHNQCVF74BA75VPKTTAK7IQAPKRRME3MOZEA, seg_pos={0 0}, upload=55354]: Uploading piece...
Jun 26 06:08:21.136399304 (68h56m48.062693995s) [noise=true , address=104.36.82.14:4223 , piece_id=LP74YFI2SRSYPYEIKREVINCOFMVCXLNDS2555YKUVFN63N6TBJDA, seg_pos={0 0}, upload=55354]: Uploading piece...
Jun 26 06:08:21.136669576 (68h56m48.062964262s) [noise=true , address=104.36.82.6:4876 , piece_id=SSMGONONY34EHF5W6QMKO7YXB7RLO5LZVTJMMFKTP6GQNGYFYQ4A, seg_pos={0 0}, upload=55354]: Uploading piece...
Jun 26 06:08:21.136809784 (68h56m48.063104477s) [noise=true , address=104.36.82.15:4258 , piece_id=YXS4J7YSG6JV33JG4BZLK5AXKE3JXLBTGZJXPXZQ4LBYBT3MU2AA, seg_pos={0 0}, upload=55354]: Uploading piece...
Jun 26 06:08:21.136950556 (68h56m48.063245242s) [noise=true , address=77.90.31.11:4554 , piece_id=R2CIFLWS4LXTGSDSIMIRC4UYVJTXC7T35HPWZ2S7VYUTJ2AMY3BQ, seg_pos={0 0}, upload=55354]: Uploading piece...
Jun 26 06:08:21.137226497 (68h56m48.06352119s) [noise=true , address=77.90.31.14:4661 , piece_id=VXI2ZUSGR4N3H7TPIK75RIDFP3SBAOWVRGHNXTZOHFT5FNG4JWJA, seg_pos={0 0}, upload=55354]: Uploading piece...
Jun 26 06:08:21.138099107 (68h56m48.064393792s) [noise=true , address=45.91.116.10:4082 , piece_id=7NVEGA4QQOE7KYPHXUUX42SJPQL6LP46RGM37VD6A2AWN7PAOM3Q, seg_pos={0 0}, upload=55354]: Uploading piece...
Jun 26 06:08:21.139330448 (68h56m48.065625141s) [noise=true , address=104.36.82.14:4222 , piece_id=U3MBMAKFBFTOPJZ3HS3FXLPDBLQTN577XNCZLUBHRO6IU4BE5VNA, seg_pos={0 0}, upload=55354]: Uploading piece...
Jun 26 06:08:21.140580734 (68h56m48.06687543s) [noise=true , address=77.90.31.19:4842 , piece_id=G3AG375DATHB7MGEEWONE5GMMIC7QHG2VVDKF5AEVOSXV5N4S6AA, seg_pos={0 0}, upload=55354]: Uploading piece...
Jun 26 06:08:21.141452300 (68h56m48.067747027s) [noise=true , address=45.91.116.13:4154 , piece_id=STFPXN2LUJYUIAYROWZIRTCMT2Y3XET6KWA4GZWUHENXRJURP62Q, seg_pos={0 0}, upload=55354]: Uploading piece...
Jun 26 06:08:21.142917062 (68h56m48.069211739s) [noise=true , address=104.36.82.7:4912 , piece_id=A5HPC7CEU66IFNS47HO3AQX22LUZ6AZ4L4XAMZ6T5NCVE4D3TAZA, seg_pos={0 0}, upload=55354]: Uploading piece...
Jun 26 06:08:21.221824189 (68h56m48.148118865s) [upload=55354]: Next returned nil segment
Jun 26 06:08:21.221837958 (68h56m48.148132644s) [upload=55354]: Waiting for error group managing segments...
Jun 26 06:08:21.229550628 (68h56m48.155845315s) [noise=true , address=104.36.82.14:4223 , piece_id=LP74YFI2SRSYPYEIKREVINCOFMVCXLNDS2555YKUVFN63N6TBJDA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: <nil>
Jun 26 06:08:21.230245145 (68h56m48.156539832s) [noise=true , address=104.36.82.14:4222 , piece_id=U3MBMAKFBFTOPJZ3HS3FXLPDBLQTN577XNCZLUBHRO6IU4BE5VNA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: <nil>
Jun 26 06:08:21.230801067 (68h56m48.15709574s) [noise=true , address=104.36.82.6:4873 , piece_id=TIIMEQUSDLRFF3OK3WBB74F7SWJ5MEC4Z6IQNELDMRZLNTMQT7LA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: <nil>
Jun 26 06:08:21.231457382 (68h56m48.157752071s) [noise=true , address=104.36.82.6:4876 , piece_id=SSMGONONY34EHF5W6QMKO7YXB7RLO5LZVTJMMFKTP6GQNGYFYQ4A, seg_pos={0 0}, upload=55354]: Done uploading piece. err: <nil>
Jun 26 06:08:21.231563205 (68h56m48.157857893s) [noise=true , address=104.36.82.13:4186 , piece_id=5SVABN3KS6AUL6HILHNQCVF74BA75VPKTTAK7IQAPKRRME3MOZEA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: <nil>
Jun 26 06:08:21.239131119 (68h56m48.165425799s) [noise=true , address=104.36.82.7:4912 , piece_id=A5HPC7CEU66IFNS47HO3AQX22LUZ6AZ4L4XAMZ6T5NCVE4D3TAZA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: <nil>
Jun 26 06:08:21.534686665 (68h56m48.460981337s) [noise=true , address=104.36.82.15:4258 , piece_id=YXS4J7YSG6JV33JG4BZLK5AXKE3JXLBTGZJXPXZQ4LBYBT3MU2AA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: <nil>
Jun 26 06:08:51.073673533 (68h57m17.999968219s) [noise=true , address=45.91.116.9:5261 , piece_id=CZWB3XJVA7B7R7GWFYNW3RCGNVRXQ3E37XDXD4SUXTEMQFZ73HHA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi, address:45.91.116.9:5261): protocol: rpc: dial tcp 45.91.116.9:5261: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jun 26 06:08:51.073739594 (68h57m18.000034272s) [noise=true , address=45.91.116.10:4082 , piece_id=7NVEGA4QQOE7KYPHXUUX42SJPQL6LP46RGM37VD6A2AWN7PAOM3Q, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj, address:45.91.116.10:4082): protocol: rpc: dial tcp 45.91.116.10:4082: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jun 26 06:08:51.073789897 (68h57m18.000084584s) [noise=true , address=77.90.31.19:4842 , piece_id=G3AG375DATHB7MGEEWONE5GMMIC7QHG2VVDKF5AEVOSXV5N4S6AA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: rpc: dial tcp 77.90.31.19:4842: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jun 26 06:08:51.073849570 (68h57m18.000144246s) [noise=true , address=45.91.116.13:4154 , piece_id=STFPXN2LUJYUIAYROWZIRTCMT2Y3XET6KWA4GZWUHENXRJURP62Q, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN, address:45.91.116.13:4154): protocol: write tcp 159.223.143.39:35392->45.91.116.13:4154: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jun 26 06:08:51.073904716 (68h57m18.000199391s) [noise=true , address=77.90.31.13:4626 , piece_id=MXYBIEDFAFK53CVTE2C6CKEZWOL3MLW4S7LOHK4IHW6QEIZJL76Q, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: write tcp 159.223.143.39:58864->77.90.31.13:4626: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jun 26 06:08:51.073952402 (68h57m18.000247082s) [noise=true , address=77.90.31.10:4517 , piece_id=VJZ5T4QNVAPNUHNFNK24S6TAS4V4NKNFMQE2FZMCWWLCCB2ELXXQ, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, address:77.90.31.10:4517): protocol: write tcp 159.223.143.39:36382->77.90.31.10:4517: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jun 26 06:08:51.074022481 (68h57m18.000317156s) [noise=true , address=77.90.31.15:4697 , piece_id=CUXGVL7PS3H7ZQKUCSYXJCBR3ETYFDPZFYIEAWEYL7C2S3RJY3AA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5, address:77.90.31.15:4697): protocol: write tcp 159.223.143.39:52730->77.90.31.15:4697: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jun 26 06:08:51.074071824 (68h57m18.000366503s) [noise=true , address=77.90.31.11:4554 , piece_id=R2CIFLWS4LXTGSDSIMIRC4UYVJTXC7T35HPWZ2S7VYUTJ2AMY3BQ, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, address:77.90.31.11:4554): protocol: rpc: dial tcp 77.90.31.11:4554: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jun 26 06:08:51.074102177 (68h57m18.000396855s) [noise=true , address=77.90.31.14:4661 , piece_id=VXI2ZUSGR4N3H7TPIK75RIDFP3SBAOWVRGHNXTZOHFT5FNG4JWJA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue, address:77.90.31.14:4661): protocol: rpc: dial tcp 77.90.31.14:4661: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jun 26 06:08:51.074119276 (68h57m18.000413966s) [seg_pos={0 0}, upload=55354]: Done waiting for segment. successful: 7 optimal: 13 errs: <nil>
Jun 26 06:08:51.100079867 (68h57m18.026374541s) [upload=55354]: Done uploading segments...
Jun 26 06:08:51.100114505 (68h57m18.026409191s) [upload=55354]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 7)
Duration: 602.203573ms
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
3 out of 60 is failed (threshold=3) [.........................................XXX................] duration:
480.190685ms
FAIL (2025-06-26 06:22:11 UTC)
orbiter warning:
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 7)
======LOGS======
pload=55354]: Got next segment. Inline: false
Jun 26 06:08:21.105827735 (68h56m48.032122411s) [upload=55354]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jun 26 06:08:21.105833588 (68h56m48.032128274s) [upload=55354]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jun 26 06:08:21.135318313 (68h56m48.061613009s) [seg_pos={0 0}, upload=55354]: Begin upload segment...
Jun 26 06:08:21.135366534 (68h56m48.061661214s) [seg_pos={0 0}, upload=55354]: Done begin upload segment.
Jun 26 06:08:21.135390342 (68h56m48.061685028s) [noise=true , address=45.91.116.9:5261 , piece_id=CZWB3XJVA7B7R7GWFYNW3RCGNVRXQ3E37XDXD4SUXTEMQFZ73HHA, seg_pos={0 0}, upload=55354]: Uploading piece...
Jun 26 06:08:21.135405801 (68h56m48.061700497s) [noise=true , address=104.36.82.6:4873 , piece_id=TIIMEQUSDLRFF3OK3WBB74F7SWJ5MEC4Z6IQNELDMRZLNTMQT7LA, seg_pos={0 0}, upload=55354]: Uploading piece...
Jun 26 06:08:21.135774729 (68h56m48.062069418s) [noise=true , address=77.90.31.13:4626 , piece_id=MXYBIEDFAFK53CVTE2C6CKEZWOL3MLW4S7LOHK4IHW6QEIZJL76Q, seg_pos={0 0}, upload=55354]: Uploading piece...
Jun 26 06:08:21.135944436 (68h56m48.06223913s) [noise=true , address=77.90.31.10:4517 , piece_id=VJZ5T4QNVAPNUHNFNK24S6TAS4V4NKNFMQE2FZMCWWLCCB2ELXXQ, seg_pos={0 0}, upload=55354]: Uploading piece...
Jun 26 06:08:21.136080402 (68h56m48.062375094s) [noise=true , address=77.90.31.15:4697 , piece_id=CUXGVL7PS3H7ZQKUCSYXJCBR3ETYFDPZFYIEAWEYL7C2S3RJY3AA, seg_pos={0 0}, upload=55354]: Uploading piece...
Jun 26 06:08:21.136382603 (68h56m48.062677289s) [noise=true , address=104.36.82.13:4186 , piece_id=5SVABN3KS6AUL6HILHNQCVF74BA75VPKTTAK7IQAPKRRME3MOZEA, seg_pos={0 0}, upload=55354]: Uploading piece...
Jun 26 06:08:21.136399304 (68h56m48.062693995s) [noise=true , address=104.36.82.14:4223 , piece_id=LP74YFI2SRSYPYEIKREVINCOFMVCXLNDS2555YKUVFN63N6TBJDA, seg_pos={0 0}, upload=55354]: Uploading piece...
Jun 26 06:08:21.136669576 (68h56m48.062964262s) [noise=true , address=104.36.82.6:4876 , piece_id=SSMGONONY34EHF5W6QMKO7YXB7RLO5LZVTJMMFKTP6GQNGYFYQ4A, seg_pos={0 0}, upload=55354]: Uploading piece...
Jun 26 06:08:21.136809784 (68h56m48.063104477s) [noise=true , address=104.36.82.15:4258 , piece_id=YXS4J7YSG6JV33JG4BZLK5AXKE3JXLBTGZJXPXZQ4LBYBT3MU2AA, seg_pos={0 0}, upload=55354]: Uploading piece...
Jun 26 06:08:21.136950556 (68h56m48.063245242s) [noise=true , address=77.90.31.11:4554 , piece_id=R2CIFLWS4LXTGSDSIMIRC4UYVJTXC7T35HPWZ2S7VYUTJ2AMY3BQ, seg_pos={0 0}, upload=55354]: Uploading piece...
Jun 26 06:08:21.137226497 (68h56m48.06352119s) [noise=true , address=77.90.31.14:4661 , piece_id=VXI2ZUSGR4N3H7TPIK75RIDFP3SBAOWVRGHNXTZOHFT5FNG4JWJA, seg_pos={0 0}, upload=55354]: Uploading piece...
Jun 26 06:08:21.138099107 (68h56m48.064393792s) [noise=true , address=45.91.116.10:4082 , piece_id=7NVEGA4QQOE7KYPHXUUX42SJPQL6LP46RGM37VD6A2AWN7PAOM3Q, seg_pos={0 0}, upload=55354]: Uploading piece...
Jun 26 06:08:21.139330448 (68h56m48.065625141s) [noise=true , address=104.36.82.14:4222 , piece_id=U3MBMAKFBFTOPJZ3HS3FXLPDBLQTN577XNCZLUBHRO6IU4BE5VNA, seg_pos={0 0}, upload=55354]: Uploading piece...
Jun 26 06:08:21.140580734 (68h56m48.06687543s) [noise=true , address=77.90.31.19:4842 , piece_id=G3AG375DATHB7MGEEWONE5GMMIC7QHG2VVDKF5AEVOSXV5N4S6AA, seg_pos={0 0}, upload=55354]: Uploading piece...
Jun 26 06:08:21.141452300 (68h56m48.067747027s) [noise=true , address=45.91.116.13:4154 , piece_id=STFPXN2LUJYUIAYROWZIRTCMT2Y3XET6KWA4GZWUHENXRJURP62Q, seg_pos={0 0}, upload=55354]: Uploading piece...
Jun 26 06:08:21.142917062 (68h56m48.069211739s) [noise=true , address=104.36.82.7:4912 , piece_id=A5HPC7CEU66IFNS47HO3AQX22LUZ6AZ4L4XAMZ6T5NCVE4D3TAZA, seg_pos={0 0}, upload=55354]: Uploading piece...
Jun 26 06:08:21.221824189 (68h56m48.148118865s) [upload=55354]: Next returned nil segment
Jun 26 06:08:21.221837958 (68h56m48.148132644s) [upload=55354]: Waiting for error group managing segments...
Jun 26 06:08:21.229550628 (68h56m48.155845315s) [noise=true , address=104.36.82.14:4223 , piece_id=LP74YFI2SRSYPYEIKREVINCOFMVCXLNDS2555YKUVFN63N6TBJDA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: <nil>
Jun 26 06:08:21.230245145 (68h56m48.156539832s) [noise=true , address=104.36.82.14:4222 , piece_id=U3MBMAKFBFTOPJZ3HS3FXLPDBLQTN577XNCZLUBHRO6IU4BE5VNA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: <nil>
Jun 26 06:08:21.230801067 (68h56m48.15709574s) [noise=true , address=104.36.82.6:4873 , piece_id=TIIMEQUSDLRFF3OK3WBB74F7SWJ5MEC4Z6IQNELDMRZLNTMQT7LA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: <nil>
Jun 26 06:08:21.231457382 (68h56m48.157752071s) [noise=true , address=104.36.82.6:4876 , piece_id=SSMGONONY34EHF5W6QMKO7YXB7RLO5LZVTJMMFKTP6GQNGYFYQ4A, seg_pos={0 0}, upload=55354]: Done uploading piece. err: <nil>
Jun 26 06:08:21.231563205 (68h56m48.157857893s) [noise=true , address=104.36.82.13:4186 , piece_id=5SVABN3KS6AUL6HILHNQCVF74BA75VPKTTAK7IQAPKRRME3MOZEA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: <nil>
Jun 26 06:08:21.239131119 (68h56m48.165425799s) [noise=true , address=104.36.82.7:4912 , piece_id=A5HPC7CEU66IFNS47HO3AQX22LUZ6AZ4L4XAMZ6T5NCVE4D3TAZA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: <nil>
Jun 26 06:08:21.534686665 (68h56m48.460981337s) [noise=true , address=104.36.82.15:4258 , piece_id=YXS4J7YSG6JV33JG4BZLK5AXKE3JXLBTGZJXPXZQ4LBYBT3MU2AA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: <nil>
Jun 26 06:08:51.073673533 (68h57m17.999968219s) [noise=true , address=45.91.116.9:5261 , piece_id=CZWB3XJVA7B7R7GWFYNW3RCGNVRXQ3E37XDXD4SUXTEMQFZ73HHA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi, address:45.91.116.9:5261): protocol: rpc: dial tcp 45.91.116.9:5261: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jun 26 06:08:51.073739594 (68h57m18.000034272s) [noise=true , address=45.91.116.10:4082 , piece_id=7NVEGA4QQOE7KYPHXUUX42SJPQL6LP46RGM37VD6A2AWN7PAOM3Q, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj, address:45.91.116.10:4082): protocol: rpc: dial tcp 45.91.116.10:4082: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jun 26 06:08:51.073789897 (68h57m18.000084584s) [noise=true , address=77.90.31.19:4842 , piece_id=G3AG375DATHB7MGEEWONE5GMMIC7QHG2VVDKF5AEVOSXV5N4S6AA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: rpc: dial tcp 77.90.31.19:4842: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jun 26 06:08:51.073849570 (68h57m18.000144246s) [noise=true , address=45.91.116.13:4154 , piece_id=STFPXN2LUJYUIAYROWZIRTCMT2Y3XET6KWA4GZWUHENXRJURP62Q, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN, address:45.91.116.13:4154): protocol: write tcp 159.223.143.39:35392->45.91.116.13:4154: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jun 26 06:08:51.073904716 (68h57m18.000199391s) [noise=true , address=77.90.31.13:4626 , piece_id=MXYBIEDFAFK53CVTE2C6CKEZWOL3MLW4S7LOHK4IHW6QEIZJL76Q, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: write tcp 159.223.143.39:58864->77.90.31.13:4626: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jun 26 06:08:51.073952402 (68h57m18.000247082s) [noise=true , address=77.90.31.10:4517 , piece_id=VJZ5T4QNVAPNUHNFNK24S6TAS4V4NKNFMQE2FZMCWWLCCB2ELXXQ, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, address:77.90.31.10:4517): protocol: write tcp 159.223.143.39:36382->77.90.31.10:4517: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jun 26 06:08:51.074022481 (68h57m18.000317156s) [noise=true , address=77.90.31.15:4697 , piece_id=CUXGVL7PS3H7ZQKUCSYXJCBR3ETYFDPZFYIEAWEYL7C2S3RJY3AA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5, address:77.90.31.15:4697): protocol: write tcp 159.223.143.39:52730->77.90.31.15:4697: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jun 26 06:08:51.074071824 (68h57m18.000366503s) [noise=true , address=77.90.31.11:4554 , piece_id=R2CIFLWS4LXTGSDSIMIRC4UYVJTXC7T35HPWZ2S7VYUTJ2AMY3BQ, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, address:77.90.31.11:4554): protocol: rpc: dial tcp 77.90.31.11:4554: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jun 26 06:08:51.074102177 (68h57m18.000396855s) [noise=true , address=77.90.31.14:4661 , piece_id=VXI2ZUSGR4N3H7TPIK75RIDFP3SBAOWVRGHNXTZOHFT5FNG4JWJA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue, address:77.90.31.14:4661): protocol: rpc: dial tcp 77.90.31.14:4661: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jun 26 06:08:51.074119276 (68h57m18.000413966s) [seg_pos={0 0}, upload=55354]: Done waiting for segment. successful: 7 optimal: 13 errs: <nil>
Jun 26 06:08:51.100079867 (68h57m18.026374541s) [upload=55354]: Done uploading segments...
Jun 26 06:08:51.100114505 (68h57m18.026409191s) [upload=55354]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 7)
Duration: 480.247757ms
core
uplink
us-select-1
|
checkers.AuthServiceHTTP{"https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us1)antiblip:
1 out of 60 is failed (threshold=3) [..................X.........................................] duration:
170.843442ms
FAIL (2025-06-26 06:22:09 UTC)
orbiter warning: unexpected status code "500 Internal Server Error" "\n"
Duration: 170.870462ms
edge
authservice
http
bgp
us1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [........................................XX..................] duration:
508.73608ms
FAIL (2025-06-26 06:22:05 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 508.758318ms
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [.........................................XX.................] duration:
483.473976ms
FAIL (2025-06-26 06:21:46 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 483.51361ms
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [...............X............................................] duration:
632.159503ms
FAIL (2025-06-26 06:21:44 UTC)
orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded
Duration: 632.189574ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [..........................................XX................] duration:
517.037746ms
FAIL (2025-06-26 06:21:25 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 517.058203ms
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
3 out of 60 is failed (threshold=3) [..........................................XXX...............] duration:
486.594442ms
FAIL (2025-06-26 06:21:03 UTC)
orbiter warning:
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 7)
======LOGS======
pload=55354]: Got next segment. Inline: false
Jun 26 06:08:21.105827735 (68h56m48.032122411s) [upload=55354]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jun 26 06:08:21.105833588 (68h56m48.032128274s) [upload=55354]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jun 26 06:08:21.135318313 (68h56m48.061613009s) [seg_pos={0 0}, upload=55354]: Begin upload segment...
Jun 26 06:08:21.135366534 (68h56m48.061661214s) [seg_pos={0 0}, upload=55354]: Done begin upload segment.
Jun 26 06:08:21.135390342 (68h56m48.061685028s) [noise=true , address=45.91.116.9:5261 , piece_id=CZWB3XJVA7B7R7GWFYNW3RCGNVRXQ3E37XDXD4SUXTEMQFZ73HHA, seg_pos={0 0}, upload=55354]: Uploading piece...
Jun 26 06:08:21.135405801 (68h56m48.061700497s) [noise=true , address=104.36.82.6:4873 , piece_id=TIIMEQUSDLRFF3OK3WBB74F7SWJ5MEC4Z6IQNELDMRZLNTMQT7LA, seg_pos={0 0}, upload=55354]: Uploading piece...
Jun 26 06:08:21.135774729 (68h56m48.062069418s) [noise=true , address=77.90.31.13:4626 , piece_id=MXYBIEDFAFK53CVTE2C6CKEZWOL3MLW4S7LOHK4IHW6QEIZJL76Q, seg_pos={0 0}, upload=55354]: Uploading piece...
Jun 26 06:08:21.135944436 (68h56m48.06223913s) [noise=true , address=77.90.31.10:4517 , piece_id=VJZ5T4QNVAPNUHNFNK24S6TAS4V4NKNFMQE2FZMCWWLCCB2ELXXQ, seg_pos={0 0}, upload=55354]: Uploading piece...
Jun 26 06:08:21.136080402 (68h56m48.062375094s) [noise=true , address=77.90.31.15:4697 , piece_id=CUXGVL7PS3H7ZQKUCSYXJCBR3ETYFDPZFYIEAWEYL7C2S3RJY3AA, seg_pos={0 0}, upload=55354]: Uploading piece...
Jun 26 06:08:21.136382603 (68h56m48.062677289s) [noise=true , address=104.36.82.13:4186 , piece_id=5SVABN3KS6AUL6HILHNQCVF74BA75VPKTTAK7IQAPKRRME3MOZEA, seg_pos={0 0}, upload=55354]: Uploading piece...
Jun 26 06:08:21.136399304 (68h56m48.062693995s) [noise=true , address=104.36.82.14:4223 , piece_id=LP74YFI2SRSYPYEIKREVINCOFMVCXLNDS2555YKUVFN63N6TBJDA, seg_pos={0 0}, upload=55354]: Uploading piece...
Jun 26 06:08:21.136669576 (68h56m48.062964262s) [noise=true , address=104.36.82.6:4876 , piece_id=SSMGONONY34EHF5W6QMKO7YXB7RLO5LZVTJMMFKTP6GQNGYFYQ4A, seg_pos={0 0}, upload=55354]: Uploading piece...
Jun 26 06:08:21.136809784 (68h56m48.063104477s) [noise=true , address=104.36.82.15:4258 , piece_id=YXS4J7YSG6JV33JG4BZLK5AXKE3JXLBTGZJXPXZQ4LBYBT3MU2AA, seg_pos={0 0}, upload=55354]: Uploading piece...
Jun 26 06:08:21.136950556 (68h56m48.063245242s) [noise=true , address=77.90.31.11:4554 , piece_id=R2CIFLWS4LXTGSDSIMIRC4UYVJTXC7T35HPWZ2S7VYUTJ2AMY3BQ, seg_pos={0 0}, upload=55354]: Uploading piece...
Jun 26 06:08:21.137226497 (68h56m48.06352119s) [noise=true , address=77.90.31.14:4661 , piece_id=VXI2ZUSGR4N3H7TPIK75RIDFP3SBAOWVRGHNXTZOHFT5FNG4JWJA, seg_pos={0 0}, upload=55354]: Uploading piece...
Jun 26 06:08:21.138099107 (68h56m48.064393792s) [noise=true , address=45.91.116.10:4082 , piece_id=7NVEGA4QQOE7KYPHXUUX42SJPQL6LP46RGM37VD6A2AWN7PAOM3Q, seg_pos={0 0}, upload=55354]: Uploading piece...
Jun 26 06:08:21.139330448 (68h56m48.065625141s) [noise=true , address=104.36.82.14:4222 , piece_id=U3MBMAKFBFTOPJZ3HS3FXLPDBLQTN577XNCZLUBHRO6IU4BE5VNA, seg_pos={0 0}, upload=55354]: Uploading piece...
Jun 26 06:08:21.140580734 (68h56m48.06687543s) [noise=true , address=77.90.31.19:4842 , piece_id=G3AG375DATHB7MGEEWONE5GMMIC7QHG2VVDKF5AEVOSXV5N4S6AA, seg_pos={0 0}, upload=55354]: Uploading piece...
Jun 26 06:08:21.141452300 (68h56m48.067747027s) [noise=true , address=45.91.116.13:4154 , piece_id=STFPXN2LUJYUIAYROWZIRTCMT2Y3XET6KWA4GZWUHENXRJURP62Q, seg_pos={0 0}, upload=55354]: Uploading piece...
Jun 26 06:08:21.142917062 (68h56m48.069211739s) [noise=true , address=104.36.82.7:4912 , piece_id=A5HPC7CEU66IFNS47HO3AQX22LUZ6AZ4L4XAMZ6T5NCVE4D3TAZA, seg_pos={0 0}, upload=55354]: Uploading piece...
Jun 26 06:08:21.221824189 (68h56m48.148118865s) [upload=55354]: Next returned nil segment
Jun 26 06:08:21.221837958 (68h56m48.148132644s) [upload=55354]: Waiting for error group managing segments...
Jun 26 06:08:21.229550628 (68h56m48.155845315s) [noise=true , address=104.36.82.14:4223 , piece_id=LP74YFI2SRSYPYEIKREVINCOFMVCXLNDS2555YKUVFN63N6TBJDA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: <nil>
Jun 26 06:08:21.230245145 (68h56m48.156539832s) [noise=true , address=104.36.82.14:4222 , piece_id=U3MBMAKFBFTOPJZ3HS3FXLPDBLQTN577XNCZLUBHRO6IU4BE5VNA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: <nil>
Jun 26 06:08:21.230801067 (68h56m48.15709574s) [noise=true , address=104.36.82.6:4873 , piece_id=TIIMEQUSDLRFF3OK3WBB74F7SWJ5MEC4Z6IQNELDMRZLNTMQT7LA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: <nil>
Jun 26 06:08:21.231457382 (68h56m48.157752071s) [noise=true , address=104.36.82.6:4876 , piece_id=SSMGONONY34EHF5W6QMKO7YXB7RLO5LZVTJMMFKTP6GQNGYFYQ4A, seg_pos={0 0}, upload=55354]: Done uploading piece. err: <nil>
Jun 26 06:08:21.231563205 (68h56m48.157857893s) [noise=true , address=104.36.82.13:4186 , piece_id=5SVABN3KS6AUL6HILHNQCVF74BA75VPKTTAK7IQAPKRRME3MOZEA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: <nil>
Jun 26 06:08:21.239131119 (68h56m48.165425799s) [noise=true , address=104.36.82.7:4912 , piece_id=A5HPC7CEU66IFNS47HO3AQX22LUZ6AZ4L4XAMZ6T5NCVE4D3TAZA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: <nil>
Jun 26 06:08:21.534686665 (68h56m48.460981337s) [noise=true , address=104.36.82.15:4258 , piece_id=YXS4J7YSG6JV33JG4BZLK5AXKE3JXLBTGZJXPXZQ4LBYBT3MU2AA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: <nil>
Jun 26 06:08:51.073673533 (68h57m17.999968219s) [noise=true , address=45.91.116.9:5261 , piece_id=CZWB3XJVA7B7R7GWFYNW3RCGNVRXQ3E37XDXD4SUXTEMQFZ73HHA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi, address:45.91.116.9:5261): protocol: rpc: dial tcp 45.91.116.9:5261: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jun 26 06:08:51.073739594 (68h57m18.000034272s) [noise=true , address=45.91.116.10:4082 , piece_id=7NVEGA4QQOE7KYPHXUUX42SJPQL6LP46RGM37VD6A2AWN7PAOM3Q, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj, address:45.91.116.10:4082): protocol: rpc: dial tcp 45.91.116.10:4082: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jun 26 06:08:51.073789897 (68h57m18.000084584s) [noise=true , address=77.90.31.19:4842 , piece_id=G3AG375DATHB7MGEEWONE5GMMIC7QHG2VVDKF5AEVOSXV5N4S6AA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: rpc: dial tcp 77.90.31.19:4842: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jun 26 06:08:51.073849570 (68h57m18.000144246s) [noise=true , address=45.91.116.13:4154 , piece_id=STFPXN2LUJYUIAYROWZIRTCMT2Y3XET6KWA4GZWUHENXRJURP62Q, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN, address:45.91.116.13:4154): protocol: write tcp 159.223.143.39:35392->45.91.116.13:4154: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jun 26 06:08:51.073904716 (68h57m18.000199391s) [noise=true , address=77.90.31.13:4626 , piece_id=MXYBIEDFAFK53CVTE2C6CKEZWOL3MLW4S7LOHK4IHW6QEIZJL76Q, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: write tcp 159.223.143.39:58864->77.90.31.13:4626: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jun 26 06:08:51.073952402 (68h57m18.000247082s) [noise=true , address=77.90.31.10:4517 , piece_id=VJZ5T4QNVAPNUHNFNK24S6TAS4V4NKNFMQE2FZMCWWLCCB2ELXXQ, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, address:77.90.31.10:4517): protocol: write tcp 159.223.143.39:36382->77.90.31.10:4517: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jun 26 06:08:51.074022481 (68h57m18.000317156s) [noise=true , address=77.90.31.15:4697 , piece_id=CUXGVL7PS3H7ZQKUCSYXJCBR3ETYFDPZFYIEAWEYL7C2S3RJY3AA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5, address:77.90.31.15:4697): protocol: write tcp 159.223.143.39:52730->77.90.31.15:4697: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jun 26 06:08:51.074071824 (68h57m18.000366503s) [noise=true , address=77.90.31.11:4554 , piece_id=R2CIFLWS4LXTGSDSIMIRC4UYVJTXC7T35HPWZ2S7VYUTJ2AMY3BQ, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, address:77.90.31.11:4554): protocol: rpc: dial tcp 77.90.31.11:4554: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jun 26 06:08:51.074102177 (68h57m18.000396855s) [noise=true , address=77.90.31.14:4661 , piece_id=VXI2ZUSGR4N3H7TPIK75RIDFP3SBAOWVRGHNXTZOHFT5FNG4JWJA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue, address:77.90.31.14:4661): protocol: rpc: dial tcp 77.90.31.14:4661: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jun 26 06:08:51.074119276 (68h57m18.000413966s) [seg_pos={0 0}, upload=55354]: Done waiting for segment. successful: 7 optimal: 13 errs: <nil>
Jun 26 06:08:51.100079867 (68h57m18.026374541s) [upload=55354]: Done uploading segments...
Jun 26 06:08:51.100114505 (68h57m18.026409191s) [upload=55354]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 7)
Duration: 486.707546ms
core
uplink
us-select-1
|
checkers.AuthServiceHTTP{"https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us1)antiblip:
1 out of 60 is failed (threshold=3) [...................X........................................] duration:
276.484365ms
FAIL (2025-06-26 06:21:02 UTC)
orbiter warning: unexpected status code "500 Internal Server Error" "\n"
Duration: 276.532874ms
edge
authservice
http
bgp
us1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [................X...........................................] duration:
869.640548ms
FAIL (2025-06-26 06:20:37 UTC)
orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded
Duration: 869.66237ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
3 out of 60 is failed (threshold=3) [...........................................XXX..............] duration:
452.553353ms
FAIL (2025-06-26 06:20:34 UTC)
orbiter warning:
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 7)
======LOGS======
pload=55354]: Got next segment. Inline: false
Jun 26 06:08:21.105827735 (68h56m48.032122411s) [upload=55354]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jun 26 06:08:21.105833588 (68h56m48.032128274s) [upload=55354]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jun 26 06:08:21.135318313 (68h56m48.061613009s) [seg_pos={0 0}, upload=55354]: Begin upload segment...
Jun 26 06:08:21.135366534 (68h56m48.061661214s) [seg_pos={0 0}, upload=55354]: Done begin upload segment.
Jun 26 06:08:21.135390342 (68h56m48.061685028s) [noise=true , address=45.91.116.9:5261 , piece_id=CZWB3XJVA7B7R7GWFYNW3RCGNVRXQ3E37XDXD4SUXTEMQFZ73HHA, seg_pos={0 0}, upload=55354]: Uploading piece...
Jun 26 06:08:21.135405801 (68h56m48.061700497s) [noise=true , address=104.36.82.6:4873 , piece_id=TIIMEQUSDLRFF3OK3WBB74F7SWJ5MEC4Z6IQNELDMRZLNTMQT7LA, seg_pos={0 0}, upload=55354]: Uploading piece...
Jun 26 06:08:21.135774729 (68h56m48.062069418s) [noise=true , address=77.90.31.13:4626 , piece_id=MXYBIEDFAFK53CVTE2C6CKEZWOL3MLW4S7LOHK4IHW6QEIZJL76Q, seg_pos={0 0}, upload=55354]: Uploading piece...
Jun 26 06:08:21.135944436 (68h56m48.06223913s) [noise=true , address=77.90.31.10:4517 , piece_id=VJZ5T4QNVAPNUHNFNK24S6TAS4V4NKNFMQE2FZMCWWLCCB2ELXXQ, seg_pos={0 0}, upload=55354]: Uploading piece...
Jun 26 06:08:21.136080402 (68h56m48.062375094s) [noise=true , address=77.90.31.15:4697 , piece_id=CUXGVL7PS3H7ZQKUCSYXJCBR3ETYFDPZFYIEAWEYL7C2S3RJY3AA, seg_pos={0 0}, upload=55354]: Uploading piece...
Jun 26 06:08:21.136382603 (68h56m48.062677289s) [noise=true , address=104.36.82.13:4186 , piece_id=5SVABN3KS6AUL6HILHNQCVF74BA75VPKTTAK7IQAPKRRME3MOZEA, seg_pos={0 0}, upload=55354]: Uploading piece...
Jun 26 06:08:21.136399304 (68h56m48.062693995s) [noise=true , address=104.36.82.14:4223 , piece_id=LP74YFI2SRSYPYEIKREVINCOFMVCXLNDS2555YKUVFN63N6TBJDA, seg_pos={0 0}, upload=55354]: Uploading piece...
Jun 26 06:08:21.136669576 (68h56m48.062964262s) [noise=true , address=104.36.82.6:4876 , piece_id=SSMGONONY34EHF5W6QMKO7YXB7RLO5LZVTJMMFKTP6GQNGYFYQ4A, seg_pos={0 0}, upload=55354]: Uploading piece...
Jun 26 06:08:21.136809784 (68h56m48.063104477s) [noise=true , address=104.36.82.15:4258 , piece_id=YXS4J7YSG6JV33JG4BZLK5AXKE3JXLBTGZJXPXZQ4LBYBT3MU2AA, seg_pos={0 0}, upload=55354]: Uploading piece...
Jun 26 06:08:21.136950556 (68h56m48.063245242s) [noise=true , address=77.90.31.11:4554 , piece_id=R2CIFLWS4LXTGSDSIMIRC4UYVJTXC7T35HPWZ2S7VYUTJ2AMY3BQ, seg_pos={0 0}, upload=55354]: Uploading piece...
Jun 26 06:08:21.137226497 (68h56m48.06352119s) [noise=true , address=77.90.31.14:4661 , piece_id=VXI2ZUSGR4N3H7TPIK75RIDFP3SBAOWVRGHNXTZOHFT5FNG4JWJA, seg_pos={0 0}, upload=55354]: Uploading piece...
Jun 26 06:08:21.138099107 (68h56m48.064393792s) [noise=true , address=45.91.116.10:4082 , piece_id=7NVEGA4QQOE7KYPHXUUX42SJPQL6LP46RGM37VD6A2AWN7PAOM3Q, seg_pos={0 0}, upload=55354]: Uploading piece...
Jun 26 06:08:21.139330448 (68h56m48.065625141s) [noise=true , address=104.36.82.14:4222 , piece_id=U3MBMAKFBFTOPJZ3HS3FXLPDBLQTN577XNCZLUBHRO6IU4BE5VNA, seg_pos={0 0}, upload=55354]: Uploading piece...
Jun 26 06:08:21.140580734 (68h56m48.06687543s) [noise=true , address=77.90.31.19:4842 , piece_id=G3AG375DATHB7MGEEWONE5GMMIC7QHG2VVDKF5AEVOSXV5N4S6AA, seg_pos={0 0}, upload=55354]: Uploading piece...
Jun 26 06:08:21.141452300 (68h56m48.067747027s) [noise=true , address=45.91.116.13:4154 , piece_id=STFPXN2LUJYUIAYROWZIRTCMT2Y3XET6KWA4GZWUHENXRJURP62Q, seg_pos={0 0}, upload=55354]: Uploading piece...
Jun 26 06:08:21.142917062 (68h56m48.069211739s) [noise=true , address=104.36.82.7:4912 , piece_id=A5HPC7CEU66IFNS47HO3AQX22LUZ6AZ4L4XAMZ6T5NCVE4D3TAZA, seg_pos={0 0}, upload=55354]: Uploading piece...
Jun 26 06:08:21.221824189 (68h56m48.148118865s) [upload=55354]: Next returned nil segment
Jun 26 06:08:21.221837958 (68h56m48.148132644s) [upload=55354]: Waiting for error group managing segments...
Jun 26 06:08:21.229550628 (68h56m48.155845315s) [noise=true , address=104.36.82.14:4223 , piece_id=LP74YFI2SRSYPYEIKREVINCOFMVCXLNDS2555YKUVFN63N6TBJDA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: <nil>
Jun 26 06:08:21.230245145 (68h56m48.156539832s) [noise=true , address=104.36.82.14:4222 , piece_id=U3MBMAKFBFTOPJZ3HS3FXLPDBLQTN577XNCZLUBHRO6IU4BE5VNA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: <nil>
Jun 26 06:08:21.230801067 (68h56m48.15709574s) [noise=true , address=104.36.82.6:4873 , piece_id=TIIMEQUSDLRFF3OK3WBB74F7SWJ5MEC4Z6IQNELDMRZLNTMQT7LA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: <nil>
Jun 26 06:08:21.231457382 (68h56m48.157752071s) [noise=true , address=104.36.82.6:4876 , piece_id=SSMGONONY34EHF5W6QMKO7YXB7RLO5LZVTJMMFKTP6GQNGYFYQ4A, seg_pos={0 0}, upload=55354]: Done uploading piece. err: <nil>
Jun 26 06:08:21.231563205 (68h56m48.157857893s) [noise=true , address=104.36.82.13:4186 , piece_id=5SVABN3KS6AUL6HILHNQCVF74BA75VPKTTAK7IQAPKRRME3MOZEA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: <nil>
Jun 26 06:08:21.239131119 (68h56m48.165425799s) [noise=true , address=104.36.82.7:4912 , piece_id=A5HPC7CEU66IFNS47HO3AQX22LUZ6AZ4L4XAMZ6T5NCVE4D3TAZA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: <nil>
Jun 26 06:08:21.534686665 (68h56m48.460981337s) [noise=true , address=104.36.82.15:4258 , piece_id=YXS4J7YSG6JV33JG4BZLK5AXKE3JXLBTGZJXPXZQ4LBYBT3MU2AA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: <nil>
Jun 26 06:08:51.073673533 (68h57m17.999968219s) [noise=true , address=45.91.116.9:5261 , piece_id=CZWB3XJVA7B7R7GWFYNW3RCGNVRXQ3E37XDXD4SUXTEMQFZ73HHA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi, address:45.91.116.9:5261): protocol: rpc: dial tcp 45.91.116.9:5261: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jun 26 06:08:51.073739594 (68h57m18.000034272s) [noise=true , address=45.91.116.10:4082 , piece_id=7NVEGA4QQOE7KYPHXUUX42SJPQL6LP46RGM37VD6A2AWN7PAOM3Q, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj, address:45.91.116.10:4082): protocol: rpc: dial tcp 45.91.116.10:4082: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jun 26 06:08:51.073789897 (68h57m18.000084584s) [noise=true , address=77.90.31.19:4842 , piece_id=G3AG375DATHB7MGEEWONE5GMMIC7QHG2VVDKF5AEVOSXV5N4S6AA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: rpc: dial tcp 77.90.31.19:4842: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jun 26 06:08:51.073849570 (68h57m18.000144246s) [noise=true , address=45.91.116.13:4154 , piece_id=STFPXN2LUJYUIAYROWZIRTCMT2Y3XET6KWA4GZWUHENXRJURP62Q, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN, address:45.91.116.13:4154): protocol: write tcp 159.223.143.39:35392->45.91.116.13:4154: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jun 26 06:08:51.073904716 (68h57m18.000199391s) [noise=true , address=77.90.31.13:4626 , piece_id=MXYBIEDFAFK53CVTE2C6CKEZWOL3MLW4S7LOHK4IHW6QEIZJL76Q, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: write tcp 159.223.143.39:58864->77.90.31.13:4626: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jun 26 06:08:51.073952402 (68h57m18.000247082s) [noise=true , address=77.90.31.10:4517 , piece_id=VJZ5T4QNVAPNUHNFNK24S6TAS4V4NKNFMQE2FZMCWWLCCB2ELXXQ, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, address:77.90.31.10:4517): protocol: write tcp 159.223.143.39:36382->77.90.31.10:4517: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jun 26 06:08:51.074022481 (68h57m18.000317156s) [noise=true , address=77.90.31.15:4697 , piece_id=CUXGVL7PS3H7ZQKUCSYXJCBR3ETYFDPZFYIEAWEYL7C2S3RJY3AA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5, address:77.90.31.15:4697): protocol: write tcp 159.223.143.39:52730->77.90.31.15:4697: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jun 26 06:08:51.074071824 (68h57m18.000366503s) [noise=true , address=77.90.31.11:4554 , piece_id=R2CIFLWS4LXTGSDSIMIRC4UYVJTXC7T35HPWZ2S7VYUTJ2AMY3BQ, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, address:77.90.31.11:4554): protocol: rpc: dial tcp 77.90.31.11:4554: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jun 26 06:08:51.074102177 (68h57m18.000396855s) [noise=true , address=77.90.31.14:4661 , piece_id=VXI2ZUSGR4N3H7TPIK75RIDFP3SBAOWVRGHNXTZOHFT5FNG4JWJA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue, address:77.90.31.14:4661): protocol: rpc: dial tcp 77.90.31.14:4661: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jun 26 06:08:51.074119276 (68h57m18.000413966s) [seg_pos={0 0}, upload=55354]: Done waiting for segment. successful: 7 optimal: 13 errs: <nil>
Jun 26 06:08:51.100079867 (68h57m18.026374541s) [upload=55354]: Done uploading segments...
Jun 26 06:08:51.100114505 (68h57m18.026409191s) [upload=55354]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 7)
Duration: 452.607788ms
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [...........................................XX...............] duration:
602.719791ms
FAIL (2025-06-26 06:20:33 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 602.742295ms
edge
gateway
us-select-1
|
checkers.AuthServiceHTTP{"https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us1)antiblip:
1 out of 60 is failed (threshold=3) [....................X.......................................] duration:
225.103577ms
FAIL (2025-06-26 06:20:11 UTC)
orbiter warning: unexpected status code "500 Internal Server Error" "\n"
Duration: 225.132153ms
edge
authservice
http
bgp
us1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
3 out of 60 is failed (threshold=3) [............................................XXX.............] duration:
557.870751ms
FAIL (2025-06-26 06:19:48 UTC)
orbiter warning:
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 7)
======LOGS======
pload=55354]: Got next segment. Inline: false
Jun 26 06:08:21.105827735 (68h56m48.032122411s) [upload=55354]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jun 26 06:08:21.105833588 (68h56m48.032128274s) [upload=55354]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jun 26 06:08:21.135318313 (68h56m48.061613009s) [seg_pos={0 0}, upload=55354]: Begin upload segment...
Jun 26 06:08:21.135366534 (68h56m48.061661214s) [seg_pos={0 0}, upload=55354]: Done begin upload segment.
Jun 26 06:08:21.135390342 (68h56m48.061685028s) [noise=true , address=45.91.116.9:5261 , piece_id=CZWB3XJVA7B7R7GWFYNW3RCGNVRXQ3E37XDXD4SUXTEMQFZ73HHA, seg_pos={0 0}, upload=55354]: Uploading piece...
Jun 26 06:08:21.135405801 (68h56m48.061700497s) [noise=true , address=104.36.82.6:4873 , piece_id=TIIMEQUSDLRFF3OK3WBB74F7SWJ5MEC4Z6IQNELDMRZLNTMQT7LA, seg_pos={0 0}, upload=55354]: Uploading piece...
Jun 26 06:08:21.135774729 (68h56m48.062069418s) [noise=true , address=77.90.31.13:4626 , piece_id=MXYBIEDFAFK53CVTE2C6CKEZWOL3MLW4S7LOHK4IHW6QEIZJL76Q, seg_pos={0 0}, upload=55354]: Uploading piece...
Jun 26 06:08:21.135944436 (68h56m48.06223913s) [noise=true , address=77.90.31.10:4517 , piece_id=VJZ5T4QNVAPNUHNFNK24S6TAS4V4NKNFMQE2FZMCWWLCCB2ELXXQ, seg_pos={0 0}, upload=55354]: Uploading piece...
Jun 26 06:08:21.136080402 (68h56m48.062375094s) [noise=true , address=77.90.31.15:4697 , piece_id=CUXGVL7PS3H7ZQKUCSYXJCBR3ETYFDPZFYIEAWEYL7C2S3RJY3AA, seg_pos={0 0}, upload=55354]: Uploading piece...
Jun 26 06:08:21.136382603 (68h56m48.062677289s) [noise=true , address=104.36.82.13:4186 , piece_id=5SVABN3KS6AUL6HILHNQCVF74BA75VPKTTAK7IQAPKRRME3MOZEA, seg_pos={0 0}, upload=55354]: Uploading piece...
Jun 26 06:08:21.136399304 (68h56m48.062693995s) [noise=true , address=104.36.82.14:4223 , piece_id=LP74YFI2SRSYPYEIKREVINCOFMVCXLNDS2555YKUVFN63N6TBJDA, seg_pos={0 0}, upload=55354]: Uploading piece...
Jun 26 06:08:21.136669576 (68h56m48.062964262s) [noise=true , address=104.36.82.6:4876 , piece_id=SSMGONONY34EHF5W6QMKO7YXB7RLO5LZVTJMMFKTP6GQNGYFYQ4A, seg_pos={0 0}, upload=55354]: Uploading piece...
Jun 26 06:08:21.136809784 (68h56m48.063104477s) [noise=true , address=104.36.82.15:4258 , piece_id=YXS4J7YSG6JV33JG4BZLK5AXKE3JXLBTGZJXPXZQ4LBYBT3MU2AA, seg_pos={0 0}, upload=55354]: Uploading piece...
Jun 26 06:08:21.136950556 (68h56m48.063245242s) [noise=true , address=77.90.31.11:4554 , piece_id=R2CIFLWS4LXTGSDSIMIRC4UYVJTXC7T35HPWZ2S7VYUTJ2AMY3BQ, seg_pos={0 0}, upload=55354]: Uploading piece...
Jun 26 06:08:21.137226497 (68h56m48.06352119s) [noise=true , address=77.90.31.14:4661 , piece_id=VXI2ZUSGR4N3H7TPIK75RIDFP3SBAOWVRGHNXTZOHFT5FNG4JWJA, seg_pos={0 0}, upload=55354]: Uploading piece...
Jun 26 06:08:21.138099107 (68h56m48.064393792s) [noise=true , address=45.91.116.10:4082 , piece_id=7NVEGA4QQOE7KYPHXUUX42SJPQL6LP46RGM37VD6A2AWN7PAOM3Q, seg_pos={0 0}, upload=55354]: Uploading piece...
Jun 26 06:08:21.139330448 (68h56m48.065625141s) [noise=true , address=104.36.82.14:4222 , piece_id=U3MBMAKFBFTOPJZ3HS3FXLPDBLQTN577XNCZLUBHRO6IU4BE5VNA, seg_pos={0 0}, upload=55354]: Uploading piece...
Jun 26 06:08:21.140580734 (68h56m48.06687543s) [noise=true , address=77.90.31.19:4842 , piece_id=G3AG375DATHB7MGEEWONE5GMMIC7QHG2VVDKF5AEVOSXV5N4S6AA, seg_pos={0 0}, upload=55354]: Uploading piece...
Jun 26 06:08:21.141452300 (68h56m48.067747027s) [noise=true , address=45.91.116.13:4154 , piece_id=STFPXN2LUJYUIAYROWZIRTCMT2Y3XET6KWA4GZWUHENXRJURP62Q, seg_pos={0 0}, upload=55354]: Uploading piece...
Jun 26 06:08:21.142917062 (68h56m48.069211739s) [noise=true , address=104.36.82.7:4912 , piece_id=A5HPC7CEU66IFNS47HO3AQX22LUZ6AZ4L4XAMZ6T5NCVE4D3TAZA, seg_pos={0 0}, upload=55354]: Uploading piece...
Jun 26 06:08:21.221824189 (68h56m48.148118865s) [upload=55354]: Next returned nil segment
Jun 26 06:08:21.221837958 (68h56m48.148132644s) [upload=55354]: Waiting for error group managing segments...
Jun 26 06:08:21.229550628 (68h56m48.155845315s) [noise=true , address=104.36.82.14:4223 , piece_id=LP74YFI2SRSYPYEIKREVINCOFMVCXLNDS2555YKUVFN63N6TBJDA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: <nil>
Jun 26 06:08:21.230245145 (68h56m48.156539832s) [noise=true , address=104.36.82.14:4222 , piece_id=U3MBMAKFBFTOPJZ3HS3FXLPDBLQTN577XNCZLUBHRO6IU4BE5VNA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: <nil>
Jun 26 06:08:21.230801067 (68h56m48.15709574s) [noise=true , address=104.36.82.6:4873 , piece_id=TIIMEQUSDLRFF3OK3WBB74F7SWJ5MEC4Z6IQNELDMRZLNTMQT7LA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: <nil>
Jun 26 06:08:21.231457382 (68h56m48.157752071s) [noise=true , address=104.36.82.6:4876 , piece_id=SSMGONONY34EHF5W6QMKO7YXB7RLO5LZVTJMMFKTP6GQNGYFYQ4A, seg_pos={0 0}, upload=55354]: Done uploading piece. err: <nil>
Jun 26 06:08:21.231563205 (68h56m48.157857893s) [noise=true , address=104.36.82.13:4186 , piece_id=5SVABN3KS6AUL6HILHNQCVF74BA75VPKTTAK7IQAPKRRME3MOZEA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: <nil>
Jun 26 06:08:21.239131119 (68h56m48.165425799s) [noise=true , address=104.36.82.7:4912 , piece_id=A5HPC7CEU66IFNS47HO3AQX22LUZ6AZ4L4XAMZ6T5NCVE4D3TAZA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: <nil>
Jun 26 06:08:21.534686665 (68h56m48.460981337s) [noise=true , address=104.36.82.15:4258 , piece_id=YXS4J7YSG6JV33JG4BZLK5AXKE3JXLBTGZJXPXZQ4LBYBT3MU2AA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: <nil>
Jun 26 06:08:51.073673533 (68h57m17.999968219s) [noise=true , address=45.91.116.9:5261 , piece_id=CZWB3XJVA7B7R7GWFYNW3RCGNVRXQ3E37XDXD4SUXTEMQFZ73HHA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi, address:45.91.116.9:5261): protocol: rpc: dial tcp 45.91.116.9:5261: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jun 26 06:08:51.073739594 (68h57m18.000034272s) [noise=true , address=45.91.116.10:4082 , piece_id=7NVEGA4QQOE7KYPHXUUX42SJPQL6LP46RGM37VD6A2AWN7PAOM3Q, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj, address:45.91.116.10:4082): protocol: rpc: dial tcp 45.91.116.10:4082: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jun 26 06:08:51.073789897 (68h57m18.000084584s) [noise=true , address=77.90.31.19:4842 , piece_id=G3AG375DATHB7MGEEWONE5GMMIC7QHG2VVDKF5AEVOSXV5N4S6AA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: rpc: dial tcp 77.90.31.19:4842: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jun 26 06:08:51.073849570 (68h57m18.000144246s) [noise=true , address=45.91.116.13:4154 , piece_id=STFPXN2LUJYUIAYROWZIRTCMT2Y3XET6KWA4GZWUHENXRJURP62Q, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN, address:45.91.116.13:4154): protocol: write tcp 159.223.143.39:35392->45.91.116.13:4154: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jun 26 06:08:51.073904716 (68h57m18.000199391s) [noise=true , address=77.90.31.13:4626 , piece_id=MXYBIEDFAFK53CVTE2C6CKEZWOL3MLW4S7LOHK4IHW6QEIZJL76Q, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: write tcp 159.223.143.39:58864->77.90.31.13:4626: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jun 26 06:08:51.073952402 (68h57m18.000247082s) [noise=true , address=77.90.31.10:4517 , piece_id=VJZ5T4QNVAPNUHNFNK24S6TAS4V4NKNFMQE2FZMCWWLCCB2ELXXQ, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, address:77.90.31.10:4517): protocol: write tcp 159.223.143.39:36382->77.90.31.10:4517: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jun 26 06:08:51.074022481 (68h57m18.000317156s) [noise=true , address=77.90.31.15:4697 , piece_id=CUXGVL7PS3H7ZQKUCSYXJCBR3ETYFDPZFYIEAWEYL7C2S3RJY3AA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5, address:77.90.31.15:4697): protocol: write tcp 159.223.143.39:52730->77.90.31.15:4697: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jun 26 06:08:51.074071824 (68h57m18.000366503s) [noise=true , address=77.90.31.11:4554 , piece_id=R2CIFLWS4LXTGSDSIMIRC4UYVJTXC7T35HPWZ2S7VYUTJ2AMY3BQ, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, address:77.90.31.11:4554): protocol: rpc: dial tcp 77.90.31.11:4554: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jun 26 06:08:51.074102177 (68h57m18.000396855s) [noise=true , address=77.90.31.14:4661 , piece_id=VXI2ZUSGR4N3H7TPIK75RIDFP3SBAOWVRGHNXTZOHFT5FNG4JWJA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue, address:77.90.31.14:4661): protocol: rpc: dial tcp 77.90.31.14:4661: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jun 26 06:08:51.074119276 (68h57m18.000413966s) [seg_pos={0 0}, upload=55354]: Done waiting for segment. successful: 7 optimal: 13 errs: <nil>
Jun 26 06:08:51.100079867 (68h57m18.026374541s) [upload=55354]: Done uploading segments...
Jun 26 06:08:51.100114505 (68h57m18.026409191s) [upload=55354]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 7)
Duration: 557.900279ms
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [............................................XX..............] duration:
493.574356ms
FAIL (2025-06-26 06:19:44 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 493.614864ms
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [.................X..........................................] duration:
650.447836ms
FAIL (2025-06-26 06:19:38 UTC)
orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded
Duration: 650.477817ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [.............................................XX.............] duration:
560.971246ms
FAIL (2025-06-26 06:19:27 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 561.093282ms
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
3 out of 60 is failed (threshold=3) [.............................................XXX............] duration:
496.348393ms
FAIL (2025-06-26 06:19:12 UTC)
orbiter warning:
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 7)
======LOGS======
pload=55354]: Got next segment. Inline: false
Jun 26 06:08:21.105827735 (68h56m48.032122411s) [upload=55354]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jun 26 06:08:21.105833588 (68h56m48.032128274s) [upload=55354]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jun 26 06:08:21.135318313 (68h56m48.061613009s) [seg_pos={0 0}, upload=55354]: Begin upload segment...
Jun 26 06:08:21.135366534 (68h56m48.061661214s) [seg_pos={0 0}, upload=55354]: Done begin upload segment.
Jun 26 06:08:21.135390342 (68h56m48.061685028s) [noise=true , address=45.91.116.9:5261 , piece_id=CZWB3XJVA7B7R7GWFYNW3RCGNVRXQ3E37XDXD4SUXTEMQFZ73HHA, seg_pos={0 0}, upload=55354]: Uploading piece...
Jun 26 06:08:21.135405801 (68h56m48.061700497s) [noise=true , address=104.36.82.6:4873 , piece_id=TIIMEQUSDLRFF3OK3WBB74F7SWJ5MEC4Z6IQNELDMRZLNTMQT7LA, seg_pos={0 0}, upload=55354]: Uploading piece...
Jun 26 06:08:21.135774729 (68h56m48.062069418s) [noise=true , address=77.90.31.13:4626 , piece_id=MXYBIEDFAFK53CVTE2C6CKEZWOL3MLW4S7LOHK4IHW6QEIZJL76Q, seg_pos={0 0}, upload=55354]: Uploading piece...
Jun 26 06:08:21.135944436 (68h56m48.06223913s) [noise=true , address=77.90.31.10:4517 , piece_id=VJZ5T4QNVAPNUHNFNK24S6TAS4V4NKNFMQE2FZMCWWLCCB2ELXXQ, seg_pos={0 0}, upload=55354]: Uploading piece...
Jun 26 06:08:21.136080402 (68h56m48.062375094s) [noise=true , address=77.90.31.15:4697 , piece_id=CUXGVL7PS3H7ZQKUCSYXJCBR3ETYFDPZFYIEAWEYL7C2S3RJY3AA, seg_pos={0 0}, upload=55354]: Uploading piece...
Jun 26 06:08:21.136382603 (68h56m48.062677289s) [noise=true , address=104.36.82.13:4186 , piece_id=5SVABN3KS6AUL6HILHNQCVF74BA75VPKTTAK7IQAPKRRME3MOZEA, seg_pos={0 0}, upload=55354]: Uploading piece...
Jun 26 06:08:21.136399304 (68h56m48.062693995s) [noise=true , address=104.36.82.14:4223 , piece_id=LP74YFI2SRSYPYEIKREVINCOFMVCXLNDS2555YKUVFN63N6TBJDA, seg_pos={0 0}, upload=55354]: Uploading piece...
Jun 26 06:08:21.136669576 (68h56m48.062964262s) [noise=true , address=104.36.82.6:4876 , piece_id=SSMGONONY34EHF5W6QMKO7YXB7RLO5LZVTJMMFKTP6GQNGYFYQ4A, seg_pos={0 0}, upload=55354]: Uploading piece...
Jun 26 06:08:21.136809784 (68h56m48.063104477s) [noise=true , address=104.36.82.15:4258 , piece_id=YXS4J7YSG6JV33JG4BZLK5AXKE3JXLBTGZJXPXZQ4LBYBT3MU2AA, seg_pos={0 0}, upload=55354]: Uploading piece...
Jun 26 06:08:21.136950556 (68h56m48.063245242s) [noise=true , address=77.90.31.11:4554 , piece_id=R2CIFLWS4LXTGSDSIMIRC4UYVJTXC7T35HPWZ2S7VYUTJ2AMY3BQ, seg_pos={0 0}, upload=55354]: Uploading piece...
Jun 26 06:08:21.137226497 (68h56m48.06352119s) [noise=true , address=77.90.31.14:4661 , piece_id=VXI2ZUSGR4N3H7TPIK75RIDFP3SBAOWVRGHNXTZOHFT5FNG4JWJA, seg_pos={0 0}, upload=55354]: Uploading piece...
Jun 26 06:08:21.138099107 (68h56m48.064393792s) [noise=true , address=45.91.116.10:4082 , piece_id=7NVEGA4QQOE7KYPHXUUX42SJPQL6LP46RGM37VD6A2AWN7PAOM3Q, seg_pos={0 0}, upload=55354]: Uploading piece...
Jun 26 06:08:21.139330448 (68h56m48.065625141s) [noise=true , address=104.36.82.14:4222 , piece_id=U3MBMAKFBFTOPJZ3HS3FXLPDBLQTN577XNCZLUBHRO6IU4BE5VNA, seg_pos={0 0}, upload=55354]: Uploading piece...
Jun 26 06:08:21.140580734 (68h56m48.06687543s) [noise=true , address=77.90.31.19:4842 , piece_id=G3AG375DATHB7MGEEWONE5GMMIC7QHG2VVDKF5AEVOSXV5N4S6AA, seg_pos={0 0}, upload=55354]: Uploading piece...
Jun 26 06:08:21.141452300 (68h56m48.067747027s) [noise=true , address=45.91.116.13:4154 , piece_id=STFPXN2LUJYUIAYROWZIRTCMT2Y3XET6KWA4GZWUHENXRJURP62Q, seg_pos={0 0}, upload=55354]: Uploading piece...
Jun 26 06:08:21.142917062 (68h56m48.069211739s) [noise=true , address=104.36.82.7:4912 , piece_id=A5HPC7CEU66IFNS47HO3AQX22LUZ6AZ4L4XAMZ6T5NCVE4D3TAZA, seg_pos={0 0}, upload=55354]: Uploading piece...
Jun 26 06:08:21.221824189 (68h56m48.148118865s) [upload=55354]: Next returned nil segment
Jun 26 06:08:21.221837958 (68h56m48.148132644s) [upload=55354]: Waiting for error group managing segments...
Jun 26 06:08:21.229550628 (68h56m48.155845315s) [noise=true , address=104.36.82.14:4223 , piece_id=LP74YFI2SRSYPYEIKREVINCOFMVCXLNDS2555YKUVFN63N6TBJDA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: <nil>
Jun 26 06:08:21.230245145 (68h56m48.156539832s) [noise=true , address=104.36.82.14:4222 , piece_id=U3MBMAKFBFTOPJZ3HS3FXLPDBLQTN577XNCZLUBHRO6IU4BE5VNA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: <nil>
Jun 26 06:08:21.230801067 (68h56m48.15709574s) [noise=true , address=104.36.82.6:4873 , piece_id=TIIMEQUSDLRFF3OK3WBB74F7SWJ5MEC4Z6IQNELDMRZLNTMQT7LA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: <nil>
Jun 26 06:08:21.231457382 (68h56m48.157752071s) [noise=true , address=104.36.82.6:4876 , piece_id=SSMGONONY34EHF5W6QMKO7YXB7RLO5LZVTJMMFKTP6GQNGYFYQ4A, seg_pos={0 0}, upload=55354]: Done uploading piece. err: <nil>
Jun 26 06:08:21.231563205 (68h56m48.157857893s) [noise=true , address=104.36.82.13:4186 , piece_id=5SVABN3KS6AUL6HILHNQCVF74BA75VPKTTAK7IQAPKRRME3MOZEA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: <nil>
Jun 26 06:08:21.239131119 (68h56m48.165425799s) [noise=true , address=104.36.82.7:4912 , piece_id=A5HPC7CEU66IFNS47HO3AQX22LUZ6AZ4L4XAMZ6T5NCVE4D3TAZA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: <nil>
Jun 26 06:08:21.534686665 (68h56m48.460981337s) [noise=true , address=104.36.82.15:4258 , piece_id=YXS4J7YSG6JV33JG4BZLK5AXKE3JXLBTGZJXPXZQ4LBYBT3MU2AA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: <nil>
Jun 26 06:08:51.073673533 (68h57m17.999968219s) [noise=true , address=45.91.116.9:5261 , piece_id=CZWB3XJVA7B7R7GWFYNW3RCGNVRXQ3E37XDXD4SUXTEMQFZ73HHA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi, address:45.91.116.9:5261): protocol: rpc: dial tcp 45.91.116.9:5261: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jun 26 06:08:51.073739594 (68h57m18.000034272s) [noise=true , address=45.91.116.10:4082 , piece_id=7NVEGA4QQOE7KYPHXUUX42SJPQL6LP46RGM37VD6A2AWN7PAOM3Q, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj, address:45.91.116.10:4082): protocol: rpc: dial tcp 45.91.116.10:4082: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jun 26 06:08:51.073789897 (68h57m18.000084584s) [noise=true , address=77.90.31.19:4842 , piece_id=G3AG375DATHB7MGEEWONE5GMMIC7QHG2VVDKF5AEVOSXV5N4S6AA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: rpc: dial tcp 77.90.31.19:4842: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jun 26 06:08:51.073849570 (68h57m18.000144246s) [noise=true , address=45.91.116.13:4154 , piece_id=STFPXN2LUJYUIAYROWZIRTCMT2Y3XET6KWA4GZWUHENXRJURP62Q, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN, address:45.91.116.13:4154): protocol: write tcp 159.223.143.39:35392->45.91.116.13:4154: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jun 26 06:08:51.073904716 (68h57m18.000199391s) [noise=true , address=77.90.31.13:4626 , piece_id=MXYBIEDFAFK53CVTE2C6CKEZWOL3MLW4S7LOHK4IHW6QEIZJL76Q, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: write tcp 159.223.143.39:58864->77.90.31.13:4626: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jun 26 06:08:51.073952402 (68h57m18.000247082s) [noise=true , address=77.90.31.10:4517 , piece_id=VJZ5T4QNVAPNUHNFNK24S6TAS4V4NKNFMQE2FZMCWWLCCB2ELXXQ, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, address:77.90.31.10:4517): protocol: write tcp 159.223.143.39:36382->77.90.31.10:4517: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jun 26 06:08:51.074022481 (68h57m18.000317156s) [noise=true , address=77.90.31.15:4697 , piece_id=CUXGVL7PS3H7ZQKUCSYXJCBR3ETYFDPZFYIEAWEYL7C2S3RJY3AA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5, address:77.90.31.15:4697): protocol: write tcp 159.223.143.39:52730->77.90.31.15:4697: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jun 26 06:08:51.074071824 (68h57m18.000366503s) [noise=true , address=77.90.31.11:4554 , piece_id=R2CIFLWS4LXTGSDSIMIRC4UYVJTXC7T35HPWZ2S7VYUTJ2AMY3BQ, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, address:77.90.31.11:4554): protocol: rpc: dial tcp 77.90.31.11:4554: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jun 26 06:08:51.074102177 (68h57m18.000396855s) [noise=true , address=77.90.31.14:4661 , piece_id=VXI2ZUSGR4N3H7TPIK75RIDFP3SBAOWVRGHNXTZOHFT5FNG4JWJA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue, address:77.90.31.14:4661): protocol: rpc: dial tcp 77.90.31.14:4661: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jun 26 06:08:51.074119276 (68h57m18.000413966s) [seg_pos={0 0}, upload=55354]: Done waiting for segment. successful: 7 optimal: 13 errs: <nil>
Jun 26 06:08:51.100079867 (68h57m18.026374541s) [upload=55354]: Done uploading segments...
Jun 26 06:08:51.100114505 (68h57m18.026409191s) [upload=55354]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 7)
Duration: 496.37591ms
core
uplink
us-select-1
|
checkers.AuthServiceHTTP{"https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us1)antiblip:
1 out of 60 is failed (threshold=3) [.....................X......................................] duration:
156.728493ms
FAIL (2025-06-26 06:19:00 UTC)
orbiter warning: unexpected status code "500 Internal Server Error" "\n"
Duration: 156.752834ms
edge
authservice
http
bgp
us1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [..................X.........................................] duration:
700.337655ms
FAIL (2025-06-26 06:18:51 UTC)
orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded
Duration: 700.353984ms
core
uplink
us-select-1
us1-placement-20
|
checkers.AuthServiceHTTP{"https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us1)antiblip:
1 out of 60 is failed (threshold=3) [......................X.....................................] duration:
166.007728ms
FAIL (2025-06-26 06:18:44 UTC)
orbiter warning: unexpected status code "500 Internal Server Error" "\n"
Duration: 166.032983ms
edge
authservice
http
bgp
us1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [...................X........................................] duration:
767.102698ms
FAIL (2025-06-26 06:18:35 UTC)
orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded
Duration: 767.13701ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [..............................................XX............] duration:
472.713754ms
FAIL (2025-06-26 06:18:19 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 472.748901ms
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
3 out of 60 is failed (threshold=3) [..............................................XXX...........] duration:
619.690044ms
FAIL (2025-06-26 06:18:00 UTC)
orbiter warning:
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 7)
======LOGS======
pload=55354]: Got next segment. Inline: false
Jun 26 06:08:21.105827735 (68h56m48.032122411s) [upload=55354]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jun 26 06:08:21.105833588 (68h56m48.032128274s) [upload=55354]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jun 26 06:08:21.135318313 (68h56m48.061613009s) [seg_pos={0 0}, upload=55354]: Begin upload segment...
Jun 26 06:08:21.135366534 (68h56m48.061661214s) [seg_pos={0 0}, upload=55354]: Done begin upload segment.
Jun 26 06:08:21.135390342 (68h56m48.061685028s) [noise=true , address=45.91.116.9:5261 , piece_id=CZWB3XJVA7B7R7GWFYNW3RCGNVRXQ3E37XDXD4SUXTEMQFZ73HHA, seg_pos={0 0}, upload=55354]: Uploading piece...
Jun 26 06:08:21.135405801 (68h56m48.061700497s) [noise=true , address=104.36.82.6:4873 , piece_id=TIIMEQUSDLRFF3OK3WBB74F7SWJ5MEC4Z6IQNELDMRZLNTMQT7LA, seg_pos={0 0}, upload=55354]: Uploading piece...
Jun 26 06:08:21.135774729 (68h56m48.062069418s) [noise=true , address=77.90.31.13:4626 , piece_id=MXYBIEDFAFK53CVTE2C6CKEZWOL3MLW4S7LOHK4IHW6QEIZJL76Q, seg_pos={0 0}, upload=55354]: Uploading piece...
Jun 26 06:08:21.135944436 (68h56m48.06223913s) [noise=true , address=77.90.31.10:4517 , piece_id=VJZ5T4QNVAPNUHNFNK24S6TAS4V4NKNFMQE2FZMCWWLCCB2ELXXQ, seg_pos={0 0}, upload=55354]: Uploading piece...
Jun 26 06:08:21.136080402 (68h56m48.062375094s) [noise=true , address=77.90.31.15:4697 , piece_id=CUXGVL7PS3H7ZQKUCSYXJCBR3ETYFDPZFYIEAWEYL7C2S3RJY3AA, seg_pos={0 0}, upload=55354]: Uploading piece...
Jun 26 06:08:21.136382603 (68h56m48.062677289s) [noise=true , address=104.36.82.13:4186 , piece_id=5SVABN3KS6AUL6HILHNQCVF74BA75VPKTTAK7IQAPKRRME3MOZEA, seg_pos={0 0}, upload=55354]: Uploading piece...
Jun 26 06:08:21.136399304 (68h56m48.062693995s) [noise=true , address=104.36.82.14:4223 , piece_id=LP74YFI2SRSYPYEIKREVINCOFMVCXLNDS2555YKUVFN63N6TBJDA, seg_pos={0 0}, upload=55354]: Uploading piece...
Jun 26 06:08:21.136669576 (68h56m48.062964262s) [noise=true , address=104.36.82.6:4876 , piece_id=SSMGONONY34EHF5W6QMKO7YXB7RLO5LZVTJMMFKTP6GQNGYFYQ4A, seg_pos={0 0}, upload=55354]: Uploading piece...
Jun 26 06:08:21.136809784 (68h56m48.063104477s) [noise=true , address=104.36.82.15:4258 , piece_id=YXS4J7YSG6JV33JG4BZLK5AXKE3JXLBTGZJXPXZQ4LBYBT3MU2AA, seg_pos={0 0}, upload=55354]: Uploading piece...
Jun 26 06:08:21.136950556 (68h56m48.063245242s) [noise=true , address=77.90.31.11:4554 , piece_id=R2CIFLWS4LXTGSDSIMIRC4UYVJTXC7T35HPWZ2S7VYUTJ2AMY3BQ, seg_pos={0 0}, upload=55354]: Uploading piece...
Jun 26 06:08:21.137226497 (68h56m48.06352119s) [noise=true , address=77.90.31.14:4661 , piece_id=VXI2ZUSGR4N3H7TPIK75RIDFP3SBAOWVRGHNXTZOHFT5FNG4JWJA, seg_pos={0 0}, upload=55354]: Uploading piece...
Jun 26 06:08:21.138099107 (68h56m48.064393792s) [noise=true , address=45.91.116.10:4082 , piece_id=7NVEGA4QQOE7KYPHXUUX42SJPQL6LP46RGM37VD6A2AWN7PAOM3Q, seg_pos={0 0}, upload=55354]: Uploading piece...
Jun 26 06:08:21.139330448 (68h56m48.065625141s) [noise=true , address=104.36.82.14:4222 , piece_id=U3MBMAKFBFTOPJZ3HS3FXLPDBLQTN577XNCZLUBHRO6IU4BE5VNA, seg_pos={0 0}, upload=55354]: Uploading piece...
Jun 26 06:08:21.140580734 (68h56m48.06687543s) [noise=true , address=77.90.31.19:4842 , piece_id=G3AG375DATHB7MGEEWONE5GMMIC7QHG2VVDKF5AEVOSXV5N4S6AA, seg_pos={0 0}, upload=55354]: Uploading piece...
Jun 26 06:08:21.141452300 (68h56m48.067747027s) [noise=true , address=45.91.116.13:4154 , piece_id=STFPXN2LUJYUIAYROWZIRTCMT2Y3XET6KWA4GZWUHENXRJURP62Q, seg_pos={0 0}, upload=55354]: Uploading piece...
Jun 26 06:08:21.142917062 (68h56m48.069211739s) [noise=true , address=104.36.82.7:4912 , piece_id=A5HPC7CEU66IFNS47HO3AQX22LUZ6AZ4L4XAMZ6T5NCVE4D3TAZA, seg_pos={0 0}, upload=55354]: Uploading piece...
Jun 26 06:08:21.221824189 (68h56m48.148118865s) [upload=55354]: Next returned nil segment
Jun 26 06:08:21.221837958 (68h56m48.148132644s) [upload=55354]: Waiting for error group managing segments...
Jun 26 06:08:21.229550628 (68h56m48.155845315s) [noise=true , address=104.36.82.14:4223 , piece_id=LP74YFI2SRSYPYEIKREVINCOFMVCXLNDS2555YKUVFN63N6TBJDA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: <nil>
Jun 26 06:08:21.230245145 (68h56m48.156539832s) [noise=true , address=104.36.82.14:4222 , piece_id=U3MBMAKFBFTOPJZ3HS3FXLPDBLQTN577XNCZLUBHRO6IU4BE5VNA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: <nil>
Jun 26 06:08:21.230801067 (68h56m48.15709574s) [noise=true , address=104.36.82.6:4873 , piece_id=TIIMEQUSDLRFF3OK3WBB74F7SWJ5MEC4Z6IQNELDMRZLNTMQT7LA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: <nil>
Jun 26 06:08:21.231457382 (68h56m48.157752071s) [noise=true , address=104.36.82.6:4876 , piece_id=SSMGONONY34EHF5W6QMKO7YXB7RLO5LZVTJMMFKTP6GQNGYFYQ4A, seg_pos={0 0}, upload=55354]: Done uploading piece. err: <nil>
Jun 26 06:08:21.231563205 (68h56m48.157857893s) [noise=true , address=104.36.82.13:4186 , piece_id=5SVABN3KS6AUL6HILHNQCVF74BA75VPKTTAK7IQAPKRRME3MOZEA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: <nil>
Jun 26 06:08:21.239131119 (68h56m48.165425799s) [noise=true , address=104.36.82.7:4912 , piece_id=A5HPC7CEU66IFNS47HO3AQX22LUZ6AZ4L4XAMZ6T5NCVE4D3TAZA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: <nil>
Jun 26 06:08:21.534686665 (68h56m48.460981337s) [noise=true , address=104.36.82.15:4258 , piece_id=YXS4J7YSG6JV33JG4BZLK5AXKE3JXLBTGZJXPXZQ4LBYBT3MU2AA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: <nil>
Jun 26 06:08:51.073673533 (68h57m17.999968219s) [noise=true , address=45.91.116.9:5261 , piece_id=CZWB3XJVA7B7R7GWFYNW3RCGNVRXQ3E37XDXD4SUXTEMQFZ73HHA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi, address:45.91.116.9:5261): protocol: rpc: dial tcp 45.91.116.9:5261: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jun 26 06:08:51.073739594 (68h57m18.000034272s) [noise=true , address=45.91.116.10:4082 , piece_id=7NVEGA4QQOE7KYPHXUUX42SJPQL6LP46RGM37VD6A2AWN7PAOM3Q, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj, address:45.91.116.10:4082): protocol: rpc: dial tcp 45.91.116.10:4082: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jun 26 06:08:51.073789897 (68h57m18.000084584s) [noise=true , address=77.90.31.19:4842 , piece_id=G3AG375DATHB7MGEEWONE5GMMIC7QHG2VVDKF5AEVOSXV5N4S6AA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: rpc: dial tcp 77.90.31.19:4842: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jun 26 06:08:51.073849570 (68h57m18.000144246s) [noise=true , address=45.91.116.13:4154 , piece_id=STFPXN2LUJYUIAYROWZIRTCMT2Y3XET6KWA4GZWUHENXRJURP62Q, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN, address:45.91.116.13:4154): protocol: write tcp 159.223.143.39:35392->45.91.116.13:4154: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jun 26 06:08:51.073904716 (68h57m18.000199391s) [noise=true , address=77.90.31.13:4626 , piece_id=MXYBIEDFAFK53CVTE2C6CKEZWOL3MLW4S7LOHK4IHW6QEIZJL76Q, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: write tcp 159.223.143.39:58864->77.90.31.13:4626: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jun 26 06:08:51.073952402 (68h57m18.000247082s) [noise=true , address=77.90.31.10:4517 , piece_id=VJZ5T4QNVAPNUHNFNK24S6TAS4V4NKNFMQE2FZMCWWLCCB2ELXXQ, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, address:77.90.31.10:4517): protocol: write tcp 159.223.143.39:36382->77.90.31.10:4517: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jun 26 06:08:51.074022481 (68h57m18.000317156s) [noise=true , address=77.90.31.15:4697 , piece_id=CUXGVL7PS3H7ZQKUCSYXJCBR3ETYFDPZFYIEAWEYL7C2S3RJY3AA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5, address:77.90.31.15:4697): protocol: write tcp 159.223.143.39:52730->77.90.31.15:4697: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jun 26 06:08:51.074071824 (68h57m18.000366503s) [noise=true , address=77.90.31.11:4554 , piece_id=R2CIFLWS4LXTGSDSIMIRC4UYVJTXC7T35HPWZ2S7VYUTJ2AMY3BQ, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, address:77.90.31.11:4554): protocol: rpc: dial tcp 77.90.31.11:4554: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jun 26 06:08:51.074102177 (68h57m18.000396855s) [noise=true , address=77.90.31.14:4661 , piece_id=VXI2ZUSGR4N3H7TPIK75RIDFP3SBAOWVRGHNXTZOHFT5FNG4JWJA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue, address:77.90.31.14:4661): protocol: rpc: dial tcp 77.90.31.14:4661: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jun 26 06:08:51.074119276 (68h57m18.000413966s) [seg_pos={0 0}, upload=55354]: Done waiting for segment. successful: 7 optimal: 13 errs: <nil>
Jun 26 06:08:51.100079867 (68h57m18.026374541s) [upload=55354]: Done uploading segments...
Jun 26 06:08:51.100114505 (68h57m18.026409191s) [upload=55354]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 7)
Duration: 619.720733ms
core
uplink
us-select-1
|
checkers.AuthServiceHTTP{"https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us1)antiblip:
1 out of 60 is failed (threshold=3) [.......................X....................................] duration:
175.280261ms
FAIL (2025-06-26 06:17:45 UTC)
orbiter warning: unexpected status code "500 Internal Server Error" "\n"
Duration: 175.314905ms
edge
authservice
http
bgp
us1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [....................X.......................................] duration:
776.722692ms
FAIL (2025-06-26 06:17:31 UTC)
orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded
Duration: 776.742138ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [...............................................XX...........] duration:
501.793588ms
FAIL (2025-06-26 06:17:08 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 501.858391ms
edge
gateway
us-select-1
|
checkers.AuthServiceHTTP{"https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us1)antiblip:
1 out of 60 is failed (threshold=3) [........................X...................................] duration:
226.374257ms
FAIL (2025-06-26 06:17:04 UTC)
orbiter warning: unexpected status code "500 Internal Server Error" "\n"
Duration: 226.407246ms
edge
authservice
http
bgp
us1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
3 out of 60 is failed (threshold=3) [...............................................XXX..........] duration:
501.814262ms
FAIL (2025-06-26 06:16:47 UTC)
orbiter warning:
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 7)
======LOGS======
pload=55354]: Got next segment. Inline: false
Jun 26 06:08:21.105827735 (68h56m48.032122411s) [upload=55354]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jun 26 06:08:21.105833588 (68h56m48.032128274s) [upload=55354]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jun 26 06:08:21.135318313 (68h56m48.061613009s) [seg_pos={0 0}, upload=55354]: Begin upload segment...
Jun 26 06:08:21.135366534 (68h56m48.061661214s) [seg_pos={0 0}, upload=55354]: Done begin upload segment.
Jun 26 06:08:21.135390342 (68h56m48.061685028s) [noise=true , address=45.91.116.9:5261 , piece_id=CZWB3XJVA7B7R7GWFYNW3RCGNVRXQ3E37XDXD4SUXTEMQFZ73HHA, seg_pos={0 0}, upload=55354]: Uploading piece...
Jun 26 06:08:21.135405801 (68h56m48.061700497s) [noise=true , address=104.36.82.6:4873 , piece_id=TIIMEQUSDLRFF3OK3WBB74F7SWJ5MEC4Z6IQNELDMRZLNTMQT7LA, seg_pos={0 0}, upload=55354]: Uploading piece...
Jun 26 06:08:21.135774729 (68h56m48.062069418s) [noise=true , address=77.90.31.13:4626 , piece_id=MXYBIEDFAFK53CVTE2C6CKEZWOL3MLW4S7LOHK4IHW6QEIZJL76Q, seg_pos={0 0}, upload=55354]: Uploading piece...
Jun 26 06:08:21.135944436 (68h56m48.06223913s) [noise=true , address=77.90.31.10:4517 , piece_id=VJZ5T4QNVAPNUHNFNK24S6TAS4V4NKNFMQE2FZMCWWLCCB2ELXXQ, seg_pos={0 0}, upload=55354]: Uploading piece...
Jun 26 06:08:21.136080402 (68h56m48.062375094s) [noise=true , address=77.90.31.15:4697 , piece_id=CUXGVL7PS3H7ZQKUCSYXJCBR3ETYFDPZFYIEAWEYL7C2S3RJY3AA, seg_pos={0 0}, upload=55354]: Uploading piece...
Jun 26 06:08:21.136382603 (68h56m48.062677289s) [noise=true , address=104.36.82.13:4186 , piece_id=5SVABN3KS6AUL6HILHNQCVF74BA75VPKTTAK7IQAPKRRME3MOZEA, seg_pos={0 0}, upload=55354]: Uploading piece...
Jun 26 06:08:21.136399304 (68h56m48.062693995s) [noise=true , address=104.36.82.14:4223 , piece_id=LP74YFI2SRSYPYEIKREVINCOFMVCXLNDS2555YKUVFN63N6TBJDA, seg_pos={0 0}, upload=55354]: Uploading piece...
Jun 26 06:08:21.136669576 (68h56m48.062964262s) [noise=true , address=104.36.82.6:4876 , piece_id=SSMGONONY34EHF5W6QMKO7YXB7RLO5LZVTJMMFKTP6GQNGYFYQ4A, seg_pos={0 0}, upload=55354]: Uploading piece...
Jun 26 06:08:21.136809784 (68h56m48.063104477s) [noise=true , address=104.36.82.15:4258 , piece_id=YXS4J7YSG6JV33JG4BZLK5AXKE3JXLBTGZJXPXZQ4LBYBT3MU2AA, seg_pos={0 0}, upload=55354]: Uploading piece...
Jun 26 06:08:21.136950556 (68h56m48.063245242s) [noise=true , address=77.90.31.11:4554 , piece_id=R2CIFLWS4LXTGSDSIMIRC4UYVJTXC7T35HPWZ2S7VYUTJ2AMY3BQ, seg_pos={0 0}, upload=55354]: Uploading piece...
Jun 26 06:08:21.137226497 (68h56m48.06352119s) [noise=true , address=77.90.31.14:4661 , piece_id=VXI2ZUSGR4N3H7TPIK75RIDFP3SBAOWVRGHNXTZOHFT5FNG4JWJA, seg_pos={0 0}, upload=55354]: Uploading piece...
Jun 26 06:08:21.138099107 (68h56m48.064393792s) [noise=true , address=45.91.116.10:4082 , piece_id=7NVEGA4QQOE7KYPHXUUX42SJPQL6LP46RGM37VD6A2AWN7PAOM3Q, seg_pos={0 0}, upload=55354]: Uploading piece...
Jun 26 06:08:21.139330448 (68h56m48.065625141s) [noise=true , address=104.36.82.14:4222 , piece_id=U3MBMAKFBFTOPJZ3HS3FXLPDBLQTN577XNCZLUBHRO6IU4BE5VNA, seg_pos={0 0}, upload=55354]: Uploading piece...
Jun 26 06:08:21.140580734 (68h56m48.06687543s) [noise=true , address=77.90.31.19:4842 , piece_id=G3AG375DATHB7MGEEWONE5GMMIC7QHG2VVDKF5AEVOSXV5N4S6AA, seg_pos={0 0}, upload=55354]: Uploading piece...
Jun 26 06:08:21.141452300 (68h56m48.067747027s) [noise=true , address=45.91.116.13:4154 , piece_id=STFPXN2LUJYUIAYROWZIRTCMT2Y3XET6KWA4GZWUHENXRJURP62Q, seg_pos={0 0}, upload=55354]: Uploading piece...
Jun 26 06:08:21.142917062 (68h56m48.069211739s) [noise=true , address=104.36.82.7:4912 , piece_id=A5HPC7CEU66IFNS47HO3AQX22LUZ6AZ4L4XAMZ6T5NCVE4D3TAZA, seg_pos={0 0}, upload=55354]: Uploading piece...
Jun 26 06:08:21.221824189 (68h56m48.148118865s) [upload=55354]: Next returned nil segment
Jun 26 06:08:21.221837958 (68h56m48.148132644s) [upload=55354]: Waiting for error group managing segments...
Jun 26 06:08:21.229550628 (68h56m48.155845315s) [noise=true , address=104.36.82.14:4223 , piece_id=LP74YFI2SRSYPYEIKREVINCOFMVCXLNDS2555YKUVFN63N6TBJDA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: <nil>
Jun 26 06:08:21.230245145 (68h56m48.156539832s) [noise=true , address=104.36.82.14:4222 , piece_id=U3MBMAKFBFTOPJZ3HS3FXLPDBLQTN577XNCZLUBHRO6IU4BE5VNA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: <nil>
Jun 26 06:08:21.230801067 (68h56m48.15709574s) [noise=true , address=104.36.82.6:4873 , piece_id=TIIMEQUSDLRFF3OK3WBB74F7SWJ5MEC4Z6IQNELDMRZLNTMQT7LA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: <nil>
Jun 26 06:08:21.231457382 (68h56m48.157752071s) [noise=true , address=104.36.82.6:4876 , piece_id=SSMGONONY34EHF5W6QMKO7YXB7RLO5LZVTJMMFKTP6GQNGYFYQ4A, seg_pos={0 0}, upload=55354]: Done uploading piece. err: <nil>
Jun 26 06:08:21.231563205 (68h56m48.157857893s) [noise=true , address=104.36.82.13:4186 , piece_id=5SVABN3KS6AUL6HILHNQCVF74BA75VPKTTAK7IQAPKRRME3MOZEA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: <nil>
Jun 26 06:08:21.239131119 (68h56m48.165425799s) [noise=true , address=104.36.82.7:4912 , piece_id=A5HPC7CEU66IFNS47HO3AQX22LUZ6AZ4L4XAMZ6T5NCVE4D3TAZA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: <nil>
Jun 26 06:08:21.534686665 (68h56m48.460981337s) [noise=true , address=104.36.82.15:4258 , piece_id=YXS4J7YSG6JV33JG4BZLK5AXKE3JXLBTGZJXPXZQ4LBYBT3MU2AA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: <nil>
Jun 26 06:08:51.073673533 (68h57m17.999968219s) [noise=true , address=45.91.116.9:5261 , piece_id=CZWB3XJVA7B7R7GWFYNW3RCGNVRXQ3E37XDXD4SUXTEMQFZ73HHA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi, address:45.91.116.9:5261): protocol: rpc: dial tcp 45.91.116.9:5261: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jun 26 06:08:51.073739594 (68h57m18.000034272s) [noise=true , address=45.91.116.10:4082 , piece_id=7NVEGA4QQOE7KYPHXUUX42SJPQL6LP46RGM37VD6A2AWN7PAOM3Q, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj, address:45.91.116.10:4082): protocol: rpc: dial tcp 45.91.116.10:4082: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jun 26 06:08:51.073789897 (68h57m18.000084584s) [noise=true , address=77.90.31.19:4842 , piece_id=G3AG375DATHB7MGEEWONE5GMMIC7QHG2VVDKF5AEVOSXV5N4S6AA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: rpc: dial tcp 77.90.31.19:4842: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jun 26 06:08:51.073849570 (68h57m18.000144246s) [noise=true , address=45.91.116.13:4154 , piece_id=STFPXN2LUJYUIAYROWZIRTCMT2Y3XET6KWA4GZWUHENXRJURP62Q, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN, address:45.91.116.13:4154): protocol: write tcp 159.223.143.39:35392->45.91.116.13:4154: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jun 26 06:08:51.073904716 (68h57m18.000199391s) [noise=true , address=77.90.31.13:4626 , piece_id=MXYBIEDFAFK53CVTE2C6CKEZWOL3MLW4S7LOHK4IHW6QEIZJL76Q, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: write tcp 159.223.143.39:58864->77.90.31.13:4626: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jun 26 06:08:51.073952402 (68h57m18.000247082s) [noise=true , address=77.90.31.10:4517 , piece_id=VJZ5T4QNVAPNUHNFNK24S6TAS4V4NKNFMQE2FZMCWWLCCB2ELXXQ, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, address:77.90.31.10:4517): protocol: write tcp 159.223.143.39:36382->77.90.31.10:4517: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jun 26 06:08:51.074022481 (68h57m18.000317156s) [noise=true , address=77.90.31.15:4697 , piece_id=CUXGVL7PS3H7ZQKUCSYXJCBR3ETYFDPZFYIEAWEYL7C2S3RJY3AA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5, address:77.90.31.15:4697): protocol: write tcp 159.223.143.39:52730->77.90.31.15:4697: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jun 26 06:08:51.074071824 (68h57m18.000366503s) [noise=true , address=77.90.31.11:4554 , piece_id=R2CIFLWS4LXTGSDSIMIRC4UYVJTXC7T35HPWZ2S7VYUTJ2AMY3BQ, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, address:77.90.31.11:4554): protocol: rpc: dial tcp 77.90.31.11:4554: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jun 26 06:08:51.074102177 (68h57m18.000396855s) [noise=true , address=77.90.31.14:4661 , piece_id=VXI2ZUSGR4N3H7TPIK75RIDFP3SBAOWVRGHNXTZOHFT5FNG4JWJA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue, address:77.90.31.14:4661): protocol: rpc: dial tcp 77.90.31.14:4661: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jun 26 06:08:51.074119276 (68h57m18.000413966s) [seg_pos={0 0}, upload=55354]: Done waiting for segment. successful: 7 optimal: 13 errs: <nil>
Jun 26 06:08:51.100079867 (68h57m18.026374541s) [upload=55354]: Done uploading segments...
Jun 26 06:08:51.100114505 (68h57m18.026409191s) [upload=55354]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 7)
Duration: 501.839212ms
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [.....................X......................................] duration:
640.091197ms
FAIL (2025-06-26 06:16:41 UTC)
orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded
Duration: 640.114386ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
3 out of 60 is failed (threshold=3) [................................................XXX.........] duration:
504.828494ms
FAIL (2025-06-26 06:16:29 UTC)
orbiter warning:
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 7)
======LOGS======
pload=55354]: Got next segment. Inline: false
Jun 26 06:08:21.105827735 (68h56m48.032122411s) [upload=55354]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jun 26 06:08:21.105833588 (68h56m48.032128274s) [upload=55354]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jun 26 06:08:21.135318313 (68h56m48.061613009s) [seg_pos={0 0}, upload=55354]: Begin upload segment...
Jun 26 06:08:21.135366534 (68h56m48.061661214s) [seg_pos={0 0}, upload=55354]: Done begin upload segment.
Jun 26 06:08:21.135390342 (68h56m48.061685028s) [noise=true , address=45.91.116.9:5261 , piece_id=CZWB3XJVA7B7R7GWFYNW3RCGNVRXQ3E37XDXD4SUXTEMQFZ73HHA, seg_pos={0 0}, upload=55354]: Uploading piece...
Jun 26 06:08:21.135405801 (68h56m48.061700497s) [noise=true , address=104.36.82.6:4873 , piece_id=TIIMEQUSDLRFF3OK3WBB74F7SWJ5MEC4Z6IQNELDMRZLNTMQT7LA, seg_pos={0 0}, upload=55354]: Uploading piece...
Jun 26 06:08:21.135774729 (68h56m48.062069418s) [noise=true , address=77.90.31.13:4626 , piece_id=MXYBIEDFAFK53CVTE2C6CKEZWOL3MLW4S7LOHK4IHW6QEIZJL76Q, seg_pos={0 0}, upload=55354]: Uploading piece...
Jun 26 06:08:21.135944436 (68h56m48.06223913s) [noise=true , address=77.90.31.10:4517 , piece_id=VJZ5T4QNVAPNUHNFNK24S6TAS4V4NKNFMQE2FZMCWWLCCB2ELXXQ, seg_pos={0 0}, upload=55354]: Uploading piece...
Jun 26 06:08:21.136080402 (68h56m48.062375094s) [noise=true , address=77.90.31.15:4697 , piece_id=CUXGVL7PS3H7ZQKUCSYXJCBR3ETYFDPZFYIEAWEYL7C2S3RJY3AA, seg_pos={0 0}, upload=55354]: Uploading piece...
Jun 26 06:08:21.136382603 (68h56m48.062677289s) [noise=true , address=104.36.82.13:4186 , piece_id=5SVABN3KS6AUL6HILHNQCVF74BA75VPKTTAK7IQAPKRRME3MOZEA, seg_pos={0 0}, upload=55354]: Uploading piece...
Jun 26 06:08:21.136399304 (68h56m48.062693995s) [noise=true , address=104.36.82.14:4223 , piece_id=LP74YFI2SRSYPYEIKREVINCOFMVCXLNDS2555YKUVFN63N6TBJDA, seg_pos={0 0}, upload=55354]: Uploading piece...
Jun 26 06:08:21.136669576 (68h56m48.062964262s) [noise=true , address=104.36.82.6:4876 , piece_id=SSMGONONY34EHF5W6QMKO7YXB7RLO5LZVTJMMFKTP6GQNGYFYQ4A, seg_pos={0 0}, upload=55354]: Uploading piece...
Jun 26 06:08:21.136809784 (68h56m48.063104477s) [noise=true , address=104.36.82.15:4258 , piece_id=YXS4J7YSG6JV33JG4BZLK5AXKE3JXLBTGZJXPXZQ4LBYBT3MU2AA, seg_pos={0 0}, upload=55354]: Uploading piece...
Jun 26 06:08:21.136950556 (68h56m48.063245242s) [noise=true , address=77.90.31.11:4554 , piece_id=R2CIFLWS4LXTGSDSIMIRC4UYVJTXC7T35HPWZ2S7VYUTJ2AMY3BQ, seg_pos={0 0}, upload=55354]: Uploading piece...
Jun 26 06:08:21.137226497 (68h56m48.06352119s) [noise=true , address=77.90.31.14:4661 , piece_id=VXI2ZUSGR4N3H7TPIK75RIDFP3SBAOWVRGHNXTZOHFT5FNG4JWJA, seg_pos={0 0}, upload=55354]: Uploading piece...
Jun 26 06:08:21.138099107 (68h56m48.064393792s) [noise=true , address=45.91.116.10:4082 , piece_id=7NVEGA4QQOE7KYPHXUUX42SJPQL6LP46RGM37VD6A2AWN7PAOM3Q, seg_pos={0 0}, upload=55354]: Uploading piece...
Jun 26 06:08:21.139330448 (68h56m48.065625141s) [noise=true , address=104.36.82.14:4222 , piece_id=U3MBMAKFBFTOPJZ3HS3FXLPDBLQTN577XNCZLUBHRO6IU4BE5VNA, seg_pos={0 0}, upload=55354]: Uploading piece...
Jun 26 06:08:21.140580734 (68h56m48.06687543s) [noise=true , address=77.90.31.19:4842 , piece_id=G3AG375DATHB7MGEEWONE5GMMIC7QHG2VVDKF5AEVOSXV5N4S6AA, seg_pos={0 0}, upload=55354]: Uploading piece...
Jun 26 06:08:21.141452300 (68h56m48.067747027s) [noise=true , address=45.91.116.13:4154 , piece_id=STFPXN2LUJYUIAYROWZIRTCMT2Y3XET6KWA4GZWUHENXRJURP62Q, seg_pos={0 0}, upload=55354]: Uploading piece...
Jun 26 06:08:21.142917062 (68h56m48.069211739s) [noise=true , address=104.36.82.7:4912 , piece_id=A5HPC7CEU66IFNS47HO3AQX22LUZ6AZ4L4XAMZ6T5NCVE4D3TAZA, seg_pos={0 0}, upload=55354]: Uploading piece...
Jun 26 06:08:21.221824189 (68h56m48.148118865s) [upload=55354]: Next returned nil segment
Jun 26 06:08:21.221837958 (68h56m48.148132644s) [upload=55354]: Waiting for error group managing segments...
Jun 26 06:08:21.229550628 (68h56m48.155845315s) [noise=true , address=104.36.82.14:4223 , piece_id=LP74YFI2SRSYPYEIKREVINCOFMVCXLNDS2555YKUVFN63N6TBJDA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: <nil>
Jun 26 06:08:21.230245145 (68h56m48.156539832s) [noise=true , address=104.36.82.14:4222 , piece_id=U3MBMAKFBFTOPJZ3HS3FXLPDBLQTN577XNCZLUBHRO6IU4BE5VNA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: <nil>
Jun 26 06:08:21.230801067 (68h56m48.15709574s) [noise=true , address=104.36.82.6:4873 , piece_id=TIIMEQUSDLRFF3OK3WBB74F7SWJ5MEC4Z6IQNELDMRZLNTMQT7LA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: <nil>
Jun 26 06:08:21.231457382 (68h56m48.157752071s) [noise=true , address=104.36.82.6:4876 , piece_id=SSMGONONY34EHF5W6QMKO7YXB7RLO5LZVTJMMFKTP6GQNGYFYQ4A, seg_pos={0 0}, upload=55354]: Done uploading piece. err: <nil>
Jun 26 06:08:21.231563205 (68h56m48.157857893s) [noise=true , address=104.36.82.13:4186 , piece_id=5SVABN3KS6AUL6HILHNQCVF74BA75VPKTTAK7IQAPKRRME3MOZEA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: <nil>
Jun 26 06:08:21.239131119 (68h56m48.165425799s) [noise=true , address=104.36.82.7:4912 , piece_id=A5HPC7CEU66IFNS47HO3AQX22LUZ6AZ4L4XAMZ6T5NCVE4D3TAZA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: <nil>
Jun 26 06:08:21.534686665 (68h56m48.460981337s) [noise=true , address=104.36.82.15:4258 , piece_id=YXS4J7YSG6JV33JG4BZLK5AXKE3JXLBTGZJXPXZQ4LBYBT3MU2AA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: <nil>
Jun 26 06:08:51.073673533 (68h57m17.999968219s) [noise=true , address=45.91.116.9:5261 , piece_id=CZWB3XJVA7B7R7GWFYNW3RCGNVRXQ3E37XDXD4SUXTEMQFZ73HHA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi, address:45.91.116.9:5261): protocol: rpc: dial tcp 45.91.116.9:5261: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jun 26 06:08:51.073739594 (68h57m18.000034272s) [noise=true , address=45.91.116.10:4082 , piece_id=7NVEGA4QQOE7KYPHXUUX42SJPQL6LP46RGM37VD6A2AWN7PAOM3Q, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj, address:45.91.116.10:4082): protocol: rpc: dial tcp 45.91.116.10:4082: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jun 26 06:08:51.073789897 (68h57m18.000084584s) [noise=true , address=77.90.31.19:4842 , piece_id=G3AG375DATHB7MGEEWONE5GMMIC7QHG2VVDKF5AEVOSXV5N4S6AA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: rpc: dial tcp 77.90.31.19:4842: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jun 26 06:08:51.073849570 (68h57m18.000144246s) [noise=true , address=45.91.116.13:4154 , piece_id=STFPXN2LUJYUIAYROWZIRTCMT2Y3XET6KWA4GZWUHENXRJURP62Q, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN, address:45.91.116.13:4154): protocol: write tcp 159.223.143.39:35392->45.91.116.13:4154: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jun 26 06:08:51.073904716 (68h57m18.000199391s) [noise=true , address=77.90.31.13:4626 , piece_id=MXYBIEDFAFK53CVTE2C6CKEZWOL3MLW4S7LOHK4IHW6QEIZJL76Q, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: write tcp 159.223.143.39:58864->77.90.31.13:4626: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jun 26 06:08:51.073952402 (68h57m18.000247082s) [noise=true , address=77.90.31.10:4517 , piece_id=VJZ5T4QNVAPNUHNFNK24S6TAS4V4NKNFMQE2FZMCWWLCCB2ELXXQ, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, address:77.90.31.10:4517): protocol: write tcp 159.223.143.39:36382->77.90.31.10:4517: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jun 26 06:08:51.074022481 (68h57m18.000317156s) [noise=true , address=77.90.31.15:4697 , piece_id=CUXGVL7PS3H7ZQKUCSYXJCBR3ETYFDPZFYIEAWEYL7C2S3RJY3AA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5, address:77.90.31.15:4697): protocol: write tcp 159.223.143.39:52730->77.90.31.15:4697: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jun 26 06:08:51.074071824 (68h57m18.000366503s) [noise=true , address=77.90.31.11:4554 , piece_id=R2CIFLWS4LXTGSDSIMIRC4UYVJTXC7T35HPWZ2S7VYUTJ2AMY3BQ, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, address:77.90.31.11:4554): protocol: rpc: dial tcp 77.90.31.11:4554: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jun 26 06:08:51.074102177 (68h57m18.000396855s) [noise=true , address=77.90.31.14:4661 , piece_id=VXI2ZUSGR4N3H7TPIK75RIDFP3SBAOWVRGHNXTZOHFT5FNG4JWJA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue, address:77.90.31.14:4661): protocol: rpc: dial tcp 77.90.31.14:4661: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jun 26 06:08:51.074119276 (68h57m18.000413966s) [seg_pos={0 0}, upload=55354]: Done waiting for segment. successful: 7 optimal: 13 errs: <nil>
Jun 26 06:08:51.100079867 (68h57m18.026374541s) [upload=55354]: Done uploading segments...
Jun 26 06:08:51.100114505 (68h57m18.026409191s) [upload=55354]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 7)
Duration: 504.855348ms
core
uplink
us-select-1
|
checkers.AuthServiceHTTP{"https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us1)antiblip:
1 out of 60 is failed (threshold=3) [.........................X..................................] duration:
132.119721ms
FAIL (2025-06-26 06:16:05 UTC)
orbiter warning: unexpected status code "500 Internal Server Error" "\n"
Duration: 132.146192ms
edge
authservice
http
bgp
us1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [................................................XX..........] duration:
662.131786ms
FAIL (2025-06-26 06:16:04 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 662.297727ms
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [......................X.....................................] duration:
627.5732ms
FAIL (2025-06-26 06:15:34 UTC)
orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded
Duration: 627.619443ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
3 out of 60 is failed (threshold=3) [.................................................XXX........] duration:
479.123595ms
FAIL (2025-06-26 06:15:19 UTC)
orbiter warning:
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 7)
======LOGS======
pload=55354]: Got next segment. Inline: false
Jun 26 06:08:21.105827735 (68h56m48.032122411s) [upload=55354]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jun 26 06:08:21.105833588 (68h56m48.032128274s) [upload=55354]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jun 26 06:08:21.135318313 (68h56m48.061613009s) [seg_pos={0 0}, upload=55354]: Begin upload segment...
Jun 26 06:08:21.135366534 (68h56m48.061661214s) [seg_pos={0 0}, upload=55354]: Done begin upload segment.
Jun 26 06:08:21.135390342 (68h56m48.061685028s) [noise=true , address=45.91.116.9:5261 , piece_id=CZWB3XJVA7B7R7GWFYNW3RCGNVRXQ3E37XDXD4SUXTEMQFZ73HHA, seg_pos={0 0}, upload=55354]: Uploading piece...
Jun 26 06:08:21.135405801 (68h56m48.061700497s) [noise=true , address=104.36.82.6:4873 , piece_id=TIIMEQUSDLRFF3OK3WBB74F7SWJ5MEC4Z6IQNELDMRZLNTMQT7LA, seg_pos={0 0}, upload=55354]: Uploading piece...
Jun 26 06:08:21.135774729 (68h56m48.062069418s) [noise=true , address=77.90.31.13:4626 , piece_id=MXYBIEDFAFK53CVTE2C6CKEZWOL3MLW4S7LOHK4IHW6QEIZJL76Q, seg_pos={0 0}, upload=55354]: Uploading piece...
Jun 26 06:08:21.135944436 (68h56m48.06223913s) [noise=true , address=77.90.31.10:4517 , piece_id=VJZ5T4QNVAPNUHNFNK24S6TAS4V4NKNFMQE2FZMCWWLCCB2ELXXQ, seg_pos={0 0}, upload=55354]: Uploading piece...
Jun 26 06:08:21.136080402 (68h56m48.062375094s) [noise=true , address=77.90.31.15:4697 , piece_id=CUXGVL7PS3H7ZQKUCSYXJCBR3ETYFDPZFYIEAWEYL7C2S3RJY3AA, seg_pos={0 0}, upload=55354]: Uploading piece...
Jun 26 06:08:21.136382603 (68h56m48.062677289s) [noise=true , address=104.36.82.13:4186 , piece_id=5SVABN3KS6AUL6HILHNQCVF74BA75VPKTTAK7IQAPKRRME3MOZEA, seg_pos={0 0}, upload=55354]: Uploading piece...
Jun 26 06:08:21.136399304 (68h56m48.062693995s) [noise=true , address=104.36.82.14:4223 , piece_id=LP74YFI2SRSYPYEIKREVINCOFMVCXLNDS2555YKUVFN63N6TBJDA, seg_pos={0 0}, upload=55354]: Uploading piece...
Jun 26 06:08:21.136669576 (68h56m48.062964262s) [noise=true , address=104.36.82.6:4876 , piece_id=SSMGONONY34EHF5W6QMKO7YXB7RLO5LZVTJMMFKTP6GQNGYFYQ4A, seg_pos={0 0}, upload=55354]: Uploading piece...
Jun 26 06:08:21.136809784 (68h56m48.063104477s) [noise=true , address=104.36.82.15:4258 , piece_id=YXS4J7YSG6JV33JG4BZLK5AXKE3JXLBTGZJXPXZQ4LBYBT3MU2AA, seg_pos={0 0}, upload=55354]: Uploading piece...
Jun 26 06:08:21.136950556 (68h56m48.063245242s) [noise=true , address=77.90.31.11:4554 , piece_id=R2CIFLWS4LXTGSDSIMIRC4UYVJTXC7T35HPWZ2S7VYUTJ2AMY3BQ, seg_pos={0 0}, upload=55354]: Uploading piece...
Jun 26 06:08:21.137226497 (68h56m48.06352119s) [noise=true , address=77.90.31.14:4661 , piece_id=VXI2ZUSGR4N3H7TPIK75RIDFP3SBAOWVRGHNXTZOHFT5FNG4JWJA, seg_pos={0 0}, upload=55354]: Uploading piece...
Jun 26 06:08:21.138099107 (68h56m48.064393792s) [noise=true , address=45.91.116.10:4082 , piece_id=7NVEGA4QQOE7KYPHXUUX42SJPQL6LP46RGM37VD6A2AWN7PAOM3Q, seg_pos={0 0}, upload=55354]: Uploading piece...
Jun 26 06:08:21.139330448 (68h56m48.065625141s) [noise=true , address=104.36.82.14:4222 , piece_id=U3MBMAKFBFTOPJZ3HS3FXLPDBLQTN577XNCZLUBHRO6IU4BE5VNA, seg_pos={0 0}, upload=55354]: Uploading piece...
Jun 26 06:08:21.140580734 (68h56m48.06687543s) [noise=true , address=77.90.31.19:4842 , piece_id=G3AG375DATHB7MGEEWONE5GMMIC7QHG2VVDKF5AEVOSXV5N4S6AA, seg_pos={0 0}, upload=55354]: Uploading piece...
Jun 26 06:08:21.141452300 (68h56m48.067747027s) [noise=true , address=45.91.116.13:4154 , piece_id=STFPXN2LUJYUIAYROWZIRTCMT2Y3XET6KWA4GZWUHENXRJURP62Q, seg_pos={0 0}, upload=55354]: Uploading piece...
Jun 26 06:08:21.142917062 (68h56m48.069211739s) [noise=true , address=104.36.82.7:4912 , piece_id=A5HPC7CEU66IFNS47HO3AQX22LUZ6AZ4L4XAMZ6T5NCVE4D3TAZA, seg_pos={0 0}, upload=55354]: Uploading piece...
Jun 26 06:08:21.221824189 (68h56m48.148118865s) [upload=55354]: Next returned nil segment
Jun 26 06:08:21.221837958 (68h56m48.148132644s) [upload=55354]: Waiting for error group managing segments...
Jun 26 06:08:21.229550628 (68h56m48.155845315s) [noise=true , address=104.36.82.14:4223 , piece_id=LP74YFI2SRSYPYEIKREVINCOFMVCXLNDS2555YKUVFN63N6TBJDA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: <nil>
Jun 26 06:08:21.230245145 (68h56m48.156539832s) [noise=true , address=104.36.82.14:4222 , piece_id=U3MBMAKFBFTOPJZ3HS3FXLPDBLQTN577XNCZLUBHRO6IU4BE5VNA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: <nil>
Jun 26 06:08:21.230801067 (68h56m48.15709574s) [noise=true , address=104.36.82.6:4873 , piece_id=TIIMEQUSDLRFF3OK3WBB74F7SWJ5MEC4Z6IQNELDMRZLNTMQT7LA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: <nil>
Jun 26 06:08:21.231457382 (68h56m48.157752071s) [noise=true , address=104.36.82.6:4876 , piece_id=SSMGONONY34EHF5W6QMKO7YXB7RLO5LZVTJMMFKTP6GQNGYFYQ4A, seg_pos={0 0}, upload=55354]: Done uploading piece. err: <nil>
Jun 26 06:08:21.231563205 (68h56m48.157857893s) [noise=true , address=104.36.82.13:4186 , piece_id=5SVABN3KS6AUL6HILHNQCVF74BA75VPKTTAK7IQAPKRRME3MOZEA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: <nil>
Jun 26 06:08:21.239131119 (68h56m48.165425799s) [noise=true , address=104.36.82.7:4912 , piece_id=A5HPC7CEU66IFNS47HO3AQX22LUZ6AZ4L4XAMZ6T5NCVE4D3TAZA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: <nil>
Jun 26 06:08:21.534686665 (68h56m48.460981337s) [noise=true , address=104.36.82.15:4258 , piece_id=YXS4J7YSG6JV33JG4BZLK5AXKE3JXLBTGZJXPXZQ4LBYBT3MU2AA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: <nil>
Jun 26 06:08:51.073673533 (68h57m17.999968219s) [noise=true , address=45.91.116.9:5261 , piece_id=CZWB3XJVA7B7R7GWFYNW3RCGNVRXQ3E37XDXD4SUXTEMQFZ73HHA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi, address:45.91.116.9:5261): protocol: rpc: dial tcp 45.91.116.9:5261: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jun 26 06:08:51.073739594 (68h57m18.000034272s) [noise=true , address=45.91.116.10:4082 , piece_id=7NVEGA4QQOE7KYPHXUUX42SJPQL6LP46RGM37VD6A2AWN7PAOM3Q, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj, address:45.91.116.10:4082): protocol: rpc: dial tcp 45.91.116.10:4082: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jun 26 06:08:51.073789897 (68h57m18.000084584s) [noise=true , address=77.90.31.19:4842 , piece_id=G3AG375DATHB7MGEEWONE5GMMIC7QHG2VVDKF5AEVOSXV5N4S6AA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: rpc: dial tcp 77.90.31.19:4842: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jun 26 06:08:51.073849570 (68h57m18.000144246s) [noise=true , address=45.91.116.13:4154 , piece_id=STFPXN2LUJYUIAYROWZIRTCMT2Y3XET6KWA4GZWUHENXRJURP62Q, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN, address:45.91.116.13:4154): protocol: write tcp 159.223.143.39:35392->45.91.116.13:4154: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jun 26 06:08:51.073904716 (68h57m18.000199391s) [noise=true , address=77.90.31.13:4626 , piece_id=MXYBIEDFAFK53CVTE2C6CKEZWOL3MLW4S7LOHK4IHW6QEIZJL76Q, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: write tcp 159.223.143.39:58864->77.90.31.13:4626: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jun 26 06:08:51.073952402 (68h57m18.000247082s) [noise=true , address=77.90.31.10:4517 , piece_id=VJZ5T4QNVAPNUHNFNK24S6TAS4V4NKNFMQE2FZMCWWLCCB2ELXXQ, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, address:77.90.31.10:4517): protocol: write tcp 159.223.143.39:36382->77.90.31.10:4517: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jun 26 06:08:51.074022481 (68h57m18.000317156s) [noise=true , address=77.90.31.15:4697 , piece_id=CUXGVL7PS3H7ZQKUCSYXJCBR3ETYFDPZFYIEAWEYL7C2S3RJY3AA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5, address:77.90.31.15:4697): protocol: write tcp 159.223.143.39:52730->77.90.31.15:4697: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jun 26 06:08:51.074071824 (68h57m18.000366503s) [noise=true , address=77.90.31.11:4554 , piece_id=R2CIFLWS4LXTGSDSIMIRC4UYVJTXC7T35HPWZ2S7VYUTJ2AMY3BQ, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, address:77.90.31.11:4554): protocol: rpc: dial tcp 77.90.31.11:4554: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jun 26 06:08:51.074102177 (68h57m18.000396855s) [noise=true , address=77.90.31.14:4661 , piece_id=VXI2ZUSGR4N3H7TPIK75RIDFP3SBAOWVRGHNXTZOHFT5FNG4JWJA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue, address:77.90.31.14:4661): protocol: rpc: dial tcp 77.90.31.14:4661: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jun 26 06:08:51.074119276 (68h57m18.000413966s) [seg_pos={0 0}, upload=55354]: Done waiting for segment. successful: 7 optimal: 13 errs: <nil>
Jun 26 06:08:51.100079867 (68h57m18.026374541s) [upload=55354]: Done uploading segments...
Jun 26 06:08:51.100114505 (68h57m18.026409191s) [upload=55354]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 7)
Duration: 479.163406ms
core
uplink
us-select-1
|