Orbiter: EU/US/AP | Code | Config | Playbook | Edge Deployment | Escalation Procedure
checkers.AuthServiceHTTP{"https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us1)antiblip:
1 out of 60 is failed (threshold=3) [..........................X.................................] duration:
209.060177ms
FAIL (2025-06-26 06:15:11 UTC)
orbiter warning: unexpected status code "500 Internal Server Error" "\n"
Duration: 209.105894ms
edge
authservice
http
bgp
us1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [.......................X....................................] duration:
734.020842ms
FAIL (2025-06-26 06:15:03 UTC)
orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded
Duration: 734.082445ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [.................................................XX.........] duration:
490.866826ms
FAIL (2025-06-26 06:14:52 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 490.890781ms
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [..................................................XX........] duration:
498.603684ms
FAIL (2025-06-26 06:14:34 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 498.741085ms
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
3 out of 60 is failed (threshold=3) [..................................................XXX.......] duration:
461.576447ms
FAIL (2025-06-26 06:14:23 UTC)
orbiter warning:
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 7)
======LOGS======
pload=55354]: Got next segment. Inline: false
Jun 26 06:08:21.105827735 (68h56m48.032122411s) [upload=55354]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jun 26 06:08:21.105833588 (68h56m48.032128274s) [upload=55354]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jun 26 06:08:21.135318313 (68h56m48.061613009s) [seg_pos={0 0}, upload=55354]: Begin upload segment...
Jun 26 06:08:21.135366534 (68h56m48.061661214s) [seg_pos={0 0}, upload=55354]: Done begin upload segment.
Jun 26 06:08:21.135390342 (68h56m48.061685028s) [noise=true , address=45.91.116.9:5261 , piece_id=CZWB3XJVA7B7R7GWFYNW3RCGNVRXQ3E37XDXD4SUXTEMQFZ73HHA, seg_pos={0 0}, upload=55354]: Uploading piece...
Jun 26 06:08:21.135405801 (68h56m48.061700497s) [noise=true , address=104.36.82.6:4873 , piece_id=TIIMEQUSDLRFF3OK3WBB74F7SWJ5MEC4Z6IQNELDMRZLNTMQT7LA, seg_pos={0 0}, upload=55354]: Uploading piece...
Jun 26 06:08:21.135774729 (68h56m48.062069418s) [noise=true , address=77.90.31.13:4626 , piece_id=MXYBIEDFAFK53CVTE2C6CKEZWOL3MLW4S7LOHK4IHW6QEIZJL76Q, seg_pos={0 0}, upload=55354]: Uploading piece...
Jun 26 06:08:21.135944436 (68h56m48.06223913s) [noise=true , address=77.90.31.10:4517 , piece_id=VJZ5T4QNVAPNUHNFNK24S6TAS4V4NKNFMQE2FZMCWWLCCB2ELXXQ, seg_pos={0 0}, upload=55354]: Uploading piece...
Jun 26 06:08:21.136080402 (68h56m48.062375094s) [noise=true , address=77.90.31.15:4697 , piece_id=CUXGVL7PS3H7ZQKUCSYXJCBR3ETYFDPZFYIEAWEYL7C2S3RJY3AA, seg_pos={0 0}, upload=55354]: Uploading piece...
Jun 26 06:08:21.136382603 (68h56m48.062677289s) [noise=true , address=104.36.82.13:4186 , piece_id=5SVABN3KS6AUL6HILHNQCVF74BA75VPKTTAK7IQAPKRRME3MOZEA, seg_pos={0 0}, upload=55354]: Uploading piece...
Jun 26 06:08:21.136399304 (68h56m48.062693995s) [noise=true , address=104.36.82.14:4223 , piece_id=LP74YFI2SRSYPYEIKREVINCOFMVCXLNDS2555YKUVFN63N6TBJDA, seg_pos={0 0}, upload=55354]: Uploading piece...
Jun 26 06:08:21.136669576 (68h56m48.062964262s) [noise=true , address=104.36.82.6:4876 , piece_id=SSMGONONY34EHF5W6QMKO7YXB7RLO5LZVTJMMFKTP6GQNGYFYQ4A, seg_pos={0 0}, upload=55354]: Uploading piece...
Jun 26 06:08:21.136809784 (68h56m48.063104477s) [noise=true , address=104.36.82.15:4258 , piece_id=YXS4J7YSG6JV33JG4BZLK5AXKE3JXLBTGZJXPXZQ4LBYBT3MU2AA, seg_pos={0 0}, upload=55354]: Uploading piece...
Jun 26 06:08:21.136950556 (68h56m48.063245242s) [noise=true , address=77.90.31.11:4554 , piece_id=R2CIFLWS4LXTGSDSIMIRC4UYVJTXC7T35HPWZ2S7VYUTJ2AMY3BQ, seg_pos={0 0}, upload=55354]: Uploading piece...
Jun 26 06:08:21.137226497 (68h56m48.06352119s) [noise=true , address=77.90.31.14:4661 , piece_id=VXI2ZUSGR4N3H7TPIK75RIDFP3SBAOWVRGHNXTZOHFT5FNG4JWJA, seg_pos={0 0}, upload=55354]: Uploading piece...
Jun 26 06:08:21.138099107 (68h56m48.064393792s) [noise=true , address=45.91.116.10:4082 , piece_id=7NVEGA4QQOE7KYPHXUUX42SJPQL6LP46RGM37VD6A2AWN7PAOM3Q, seg_pos={0 0}, upload=55354]: Uploading piece...
Jun 26 06:08:21.139330448 (68h56m48.065625141s) [noise=true , address=104.36.82.14:4222 , piece_id=U3MBMAKFBFTOPJZ3HS3FXLPDBLQTN577XNCZLUBHRO6IU4BE5VNA, seg_pos={0 0}, upload=55354]: Uploading piece...
Jun 26 06:08:21.140580734 (68h56m48.06687543s) [noise=true , address=77.90.31.19:4842 , piece_id=G3AG375DATHB7MGEEWONE5GMMIC7QHG2VVDKF5AEVOSXV5N4S6AA, seg_pos={0 0}, upload=55354]: Uploading piece...
Jun 26 06:08:21.141452300 (68h56m48.067747027s) [noise=true , address=45.91.116.13:4154 , piece_id=STFPXN2LUJYUIAYROWZIRTCMT2Y3XET6KWA4GZWUHENXRJURP62Q, seg_pos={0 0}, upload=55354]: Uploading piece...
Jun 26 06:08:21.142917062 (68h56m48.069211739s) [noise=true , address=104.36.82.7:4912 , piece_id=A5HPC7CEU66IFNS47HO3AQX22LUZ6AZ4L4XAMZ6T5NCVE4D3TAZA, seg_pos={0 0}, upload=55354]: Uploading piece...
Jun 26 06:08:21.221824189 (68h56m48.148118865s) [upload=55354]: Next returned nil segment
Jun 26 06:08:21.221837958 (68h56m48.148132644s) [upload=55354]: Waiting for error group managing segments...
Jun 26 06:08:21.229550628 (68h56m48.155845315s) [noise=true , address=104.36.82.14:4223 , piece_id=LP74YFI2SRSYPYEIKREVINCOFMVCXLNDS2555YKUVFN63N6TBJDA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: <nil>
Jun 26 06:08:21.230245145 (68h56m48.156539832s) [noise=true , address=104.36.82.14:4222 , piece_id=U3MBMAKFBFTOPJZ3HS3FXLPDBLQTN577XNCZLUBHRO6IU4BE5VNA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: <nil>
Jun 26 06:08:21.230801067 (68h56m48.15709574s) [noise=true , address=104.36.82.6:4873 , piece_id=TIIMEQUSDLRFF3OK3WBB74F7SWJ5MEC4Z6IQNELDMRZLNTMQT7LA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: <nil>
Jun 26 06:08:21.231457382 (68h56m48.157752071s) [noise=true , address=104.36.82.6:4876 , piece_id=SSMGONONY34EHF5W6QMKO7YXB7RLO5LZVTJMMFKTP6GQNGYFYQ4A, seg_pos={0 0}, upload=55354]: Done uploading piece. err: <nil>
Jun 26 06:08:21.231563205 (68h56m48.157857893s) [noise=true , address=104.36.82.13:4186 , piece_id=5SVABN3KS6AUL6HILHNQCVF74BA75VPKTTAK7IQAPKRRME3MOZEA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: <nil>
Jun 26 06:08:21.239131119 (68h56m48.165425799s) [noise=true , address=104.36.82.7:4912 , piece_id=A5HPC7CEU66IFNS47HO3AQX22LUZ6AZ4L4XAMZ6T5NCVE4D3TAZA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: <nil>
Jun 26 06:08:21.534686665 (68h56m48.460981337s) [noise=true , address=104.36.82.15:4258 , piece_id=YXS4J7YSG6JV33JG4BZLK5AXKE3JXLBTGZJXPXZQ4LBYBT3MU2AA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: <nil>
Jun 26 06:08:51.073673533 (68h57m17.999968219s) [noise=true , address=45.91.116.9:5261 , piece_id=CZWB3XJVA7B7R7GWFYNW3RCGNVRXQ3E37XDXD4SUXTEMQFZ73HHA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi, address:45.91.116.9:5261): protocol: rpc: dial tcp 45.91.116.9:5261: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jun 26 06:08:51.073739594 (68h57m18.000034272s) [noise=true , address=45.91.116.10:4082 , piece_id=7NVEGA4QQOE7KYPHXUUX42SJPQL6LP46RGM37VD6A2AWN7PAOM3Q, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj, address:45.91.116.10:4082): protocol: rpc: dial tcp 45.91.116.10:4082: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jun 26 06:08:51.073789897 (68h57m18.000084584s) [noise=true , address=77.90.31.19:4842 , piece_id=G3AG375DATHB7MGEEWONE5GMMIC7QHG2VVDKF5AEVOSXV5N4S6AA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: rpc: dial tcp 77.90.31.19:4842: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jun 26 06:08:51.073849570 (68h57m18.000144246s) [noise=true , address=45.91.116.13:4154 , piece_id=STFPXN2LUJYUIAYROWZIRTCMT2Y3XET6KWA4GZWUHENXRJURP62Q, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN, address:45.91.116.13:4154): protocol: write tcp 159.223.143.39:35392->45.91.116.13:4154: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jun 26 06:08:51.073904716 (68h57m18.000199391s) [noise=true , address=77.90.31.13:4626 , piece_id=MXYBIEDFAFK53CVTE2C6CKEZWOL3MLW4S7LOHK4IHW6QEIZJL76Q, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: write tcp 159.223.143.39:58864->77.90.31.13:4626: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jun 26 06:08:51.073952402 (68h57m18.000247082s) [noise=true , address=77.90.31.10:4517 , piece_id=VJZ5T4QNVAPNUHNFNK24S6TAS4V4NKNFMQE2FZMCWWLCCB2ELXXQ, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, address:77.90.31.10:4517): protocol: write tcp 159.223.143.39:36382->77.90.31.10:4517: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jun 26 06:08:51.074022481 (68h57m18.000317156s) [noise=true , address=77.90.31.15:4697 , piece_id=CUXGVL7PS3H7ZQKUCSYXJCBR3ETYFDPZFYIEAWEYL7C2S3RJY3AA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5, address:77.90.31.15:4697): protocol: write tcp 159.223.143.39:52730->77.90.31.15:4697: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jun 26 06:08:51.074071824 (68h57m18.000366503s) [noise=true , address=77.90.31.11:4554 , piece_id=R2CIFLWS4LXTGSDSIMIRC4UYVJTXC7T35HPWZ2S7VYUTJ2AMY3BQ, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, address:77.90.31.11:4554): protocol: rpc: dial tcp 77.90.31.11:4554: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jun 26 06:08:51.074102177 (68h57m18.000396855s) [noise=true , address=77.90.31.14:4661 , piece_id=VXI2ZUSGR4N3H7TPIK75RIDFP3SBAOWVRGHNXTZOHFT5FNG4JWJA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue, address:77.90.31.14:4661): protocol: rpc: dial tcp 77.90.31.14:4661: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jun 26 06:08:51.074119276 (68h57m18.000413966s) [seg_pos={0 0}, upload=55354]: Done waiting for segment. successful: 7 optimal: 13 errs: <nil>
Jun 26 06:08:51.100079867 (68h57m18.026374541s) [upload=55354]: Done uploading segments...
Jun 26 06:08:51.100114505 (68h57m18.026409191s) [upload=55354]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 7)
Duration: 461.602496ms
core
uplink
us-select-1
|
checkers.AuthServiceHTTP{"https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us1)antiblip:
1 out of 60 is failed (threshold=3) [...........................X................................] duration:
191.436071ms
FAIL (2025-06-26 06:14:15 UTC)
orbiter warning: unexpected status code "500 Internal Server Error" "\n"
Duration: 191.461672ms
edge
authservice
http
bgp
us1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [........................X...................................] duration:
959.19218ms
FAIL (2025-06-26 06:14:14 UTC)
orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded
Duration: 959.219021ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [...................................................XX.......] duration:
529.952998ms
FAIL (2025-06-26 06:13:32 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 530.015862ms
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
3 out of 60 is failed (threshold=3) [...................................................XXX......] duration:
456.636484ms
FAIL (2025-06-26 06:13:26 UTC)
orbiter warning:
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 7)
======LOGS======
pload=55354]: Got next segment. Inline: false
Jun 26 06:08:21.105827735 (68h56m48.032122411s) [upload=55354]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jun 26 06:08:21.105833588 (68h56m48.032128274s) [upload=55354]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jun 26 06:08:21.135318313 (68h56m48.061613009s) [seg_pos={0 0}, upload=55354]: Begin upload segment...
Jun 26 06:08:21.135366534 (68h56m48.061661214s) [seg_pos={0 0}, upload=55354]: Done begin upload segment.
Jun 26 06:08:21.135390342 (68h56m48.061685028s) [noise=true , address=45.91.116.9:5261 , piece_id=CZWB3XJVA7B7R7GWFYNW3RCGNVRXQ3E37XDXD4SUXTEMQFZ73HHA, seg_pos={0 0}, upload=55354]: Uploading piece...
Jun 26 06:08:21.135405801 (68h56m48.061700497s) [noise=true , address=104.36.82.6:4873 , piece_id=TIIMEQUSDLRFF3OK3WBB74F7SWJ5MEC4Z6IQNELDMRZLNTMQT7LA, seg_pos={0 0}, upload=55354]: Uploading piece...
Jun 26 06:08:21.135774729 (68h56m48.062069418s) [noise=true , address=77.90.31.13:4626 , piece_id=MXYBIEDFAFK53CVTE2C6CKEZWOL3MLW4S7LOHK4IHW6QEIZJL76Q, seg_pos={0 0}, upload=55354]: Uploading piece...
Jun 26 06:08:21.135944436 (68h56m48.06223913s) [noise=true , address=77.90.31.10:4517 , piece_id=VJZ5T4QNVAPNUHNFNK24S6TAS4V4NKNFMQE2FZMCWWLCCB2ELXXQ, seg_pos={0 0}, upload=55354]: Uploading piece...
Jun 26 06:08:21.136080402 (68h56m48.062375094s) [noise=true , address=77.90.31.15:4697 , piece_id=CUXGVL7PS3H7ZQKUCSYXJCBR3ETYFDPZFYIEAWEYL7C2S3RJY3AA, seg_pos={0 0}, upload=55354]: Uploading piece...
Jun 26 06:08:21.136382603 (68h56m48.062677289s) [noise=true , address=104.36.82.13:4186 , piece_id=5SVABN3KS6AUL6HILHNQCVF74BA75VPKTTAK7IQAPKRRME3MOZEA, seg_pos={0 0}, upload=55354]: Uploading piece...
Jun 26 06:08:21.136399304 (68h56m48.062693995s) [noise=true , address=104.36.82.14:4223 , piece_id=LP74YFI2SRSYPYEIKREVINCOFMVCXLNDS2555YKUVFN63N6TBJDA, seg_pos={0 0}, upload=55354]: Uploading piece...
Jun 26 06:08:21.136669576 (68h56m48.062964262s) [noise=true , address=104.36.82.6:4876 , piece_id=SSMGONONY34EHF5W6QMKO7YXB7RLO5LZVTJMMFKTP6GQNGYFYQ4A, seg_pos={0 0}, upload=55354]: Uploading piece...
Jun 26 06:08:21.136809784 (68h56m48.063104477s) [noise=true , address=104.36.82.15:4258 , piece_id=YXS4J7YSG6JV33JG4BZLK5AXKE3JXLBTGZJXPXZQ4LBYBT3MU2AA, seg_pos={0 0}, upload=55354]: Uploading piece...
Jun 26 06:08:21.136950556 (68h56m48.063245242s) [noise=true , address=77.90.31.11:4554 , piece_id=R2CIFLWS4LXTGSDSIMIRC4UYVJTXC7T35HPWZ2S7VYUTJ2AMY3BQ, seg_pos={0 0}, upload=55354]: Uploading piece...
Jun 26 06:08:21.137226497 (68h56m48.06352119s) [noise=true , address=77.90.31.14:4661 , piece_id=VXI2ZUSGR4N3H7TPIK75RIDFP3SBAOWVRGHNXTZOHFT5FNG4JWJA, seg_pos={0 0}, upload=55354]: Uploading piece...
Jun 26 06:08:21.138099107 (68h56m48.064393792s) [noise=true , address=45.91.116.10:4082 , piece_id=7NVEGA4QQOE7KYPHXUUX42SJPQL6LP46RGM37VD6A2AWN7PAOM3Q, seg_pos={0 0}, upload=55354]: Uploading piece...
Jun 26 06:08:21.139330448 (68h56m48.065625141s) [noise=true , address=104.36.82.14:4222 , piece_id=U3MBMAKFBFTOPJZ3HS3FXLPDBLQTN577XNCZLUBHRO6IU4BE5VNA, seg_pos={0 0}, upload=55354]: Uploading piece...
Jun 26 06:08:21.140580734 (68h56m48.06687543s) [noise=true , address=77.90.31.19:4842 , piece_id=G3AG375DATHB7MGEEWONE5GMMIC7QHG2VVDKF5AEVOSXV5N4S6AA, seg_pos={0 0}, upload=55354]: Uploading piece...
Jun 26 06:08:21.141452300 (68h56m48.067747027s) [noise=true , address=45.91.116.13:4154 , piece_id=STFPXN2LUJYUIAYROWZIRTCMT2Y3XET6KWA4GZWUHENXRJURP62Q, seg_pos={0 0}, upload=55354]: Uploading piece...
Jun 26 06:08:21.142917062 (68h56m48.069211739s) [noise=true , address=104.36.82.7:4912 , piece_id=A5HPC7CEU66IFNS47HO3AQX22LUZ6AZ4L4XAMZ6T5NCVE4D3TAZA, seg_pos={0 0}, upload=55354]: Uploading piece...
Jun 26 06:08:21.221824189 (68h56m48.148118865s) [upload=55354]: Next returned nil segment
Jun 26 06:08:21.221837958 (68h56m48.148132644s) [upload=55354]: Waiting for error group managing segments...
Jun 26 06:08:21.229550628 (68h56m48.155845315s) [noise=true , address=104.36.82.14:4223 , piece_id=LP74YFI2SRSYPYEIKREVINCOFMVCXLNDS2555YKUVFN63N6TBJDA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: <nil>
Jun 26 06:08:21.230245145 (68h56m48.156539832s) [noise=true , address=104.36.82.14:4222 , piece_id=U3MBMAKFBFTOPJZ3HS3FXLPDBLQTN577XNCZLUBHRO6IU4BE5VNA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: <nil>
Jun 26 06:08:21.230801067 (68h56m48.15709574s) [noise=true , address=104.36.82.6:4873 , piece_id=TIIMEQUSDLRFF3OK3WBB74F7SWJ5MEC4Z6IQNELDMRZLNTMQT7LA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: <nil>
Jun 26 06:08:21.231457382 (68h56m48.157752071s) [noise=true , address=104.36.82.6:4876 , piece_id=SSMGONONY34EHF5W6QMKO7YXB7RLO5LZVTJMMFKTP6GQNGYFYQ4A, seg_pos={0 0}, upload=55354]: Done uploading piece. err: <nil>
Jun 26 06:08:21.231563205 (68h56m48.157857893s) [noise=true , address=104.36.82.13:4186 , piece_id=5SVABN3KS6AUL6HILHNQCVF74BA75VPKTTAK7IQAPKRRME3MOZEA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: <nil>
Jun 26 06:08:21.239131119 (68h56m48.165425799s) [noise=true , address=104.36.82.7:4912 , piece_id=A5HPC7CEU66IFNS47HO3AQX22LUZ6AZ4L4XAMZ6T5NCVE4D3TAZA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: <nil>
Jun 26 06:08:21.534686665 (68h56m48.460981337s) [noise=true , address=104.36.82.15:4258 , piece_id=YXS4J7YSG6JV33JG4BZLK5AXKE3JXLBTGZJXPXZQ4LBYBT3MU2AA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: <nil>
Jun 26 06:08:51.073673533 (68h57m17.999968219s) [noise=true , address=45.91.116.9:5261 , piece_id=CZWB3XJVA7B7R7GWFYNW3RCGNVRXQ3E37XDXD4SUXTEMQFZ73HHA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi, address:45.91.116.9:5261): protocol: rpc: dial tcp 45.91.116.9:5261: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jun 26 06:08:51.073739594 (68h57m18.000034272s) [noise=true , address=45.91.116.10:4082 , piece_id=7NVEGA4QQOE7KYPHXUUX42SJPQL6LP46RGM37VD6A2AWN7PAOM3Q, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj, address:45.91.116.10:4082): protocol: rpc: dial tcp 45.91.116.10:4082: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jun 26 06:08:51.073789897 (68h57m18.000084584s) [noise=true , address=77.90.31.19:4842 , piece_id=G3AG375DATHB7MGEEWONE5GMMIC7QHG2VVDKF5AEVOSXV5N4S6AA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: rpc: dial tcp 77.90.31.19:4842: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jun 26 06:08:51.073849570 (68h57m18.000144246s) [noise=true , address=45.91.116.13:4154 , piece_id=STFPXN2LUJYUIAYROWZIRTCMT2Y3XET6KWA4GZWUHENXRJURP62Q, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN, address:45.91.116.13:4154): protocol: write tcp 159.223.143.39:35392->45.91.116.13:4154: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jun 26 06:08:51.073904716 (68h57m18.000199391s) [noise=true , address=77.90.31.13:4626 , piece_id=MXYBIEDFAFK53CVTE2C6CKEZWOL3MLW4S7LOHK4IHW6QEIZJL76Q, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: write tcp 159.223.143.39:58864->77.90.31.13:4626: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jun 26 06:08:51.073952402 (68h57m18.000247082s) [noise=true , address=77.90.31.10:4517 , piece_id=VJZ5T4QNVAPNUHNFNK24S6TAS4V4NKNFMQE2FZMCWWLCCB2ELXXQ, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, address:77.90.31.10:4517): protocol: write tcp 159.223.143.39:36382->77.90.31.10:4517: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jun 26 06:08:51.074022481 (68h57m18.000317156s) [noise=true , address=77.90.31.15:4697 , piece_id=CUXGVL7PS3H7ZQKUCSYXJCBR3ETYFDPZFYIEAWEYL7C2S3RJY3AA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5, address:77.90.31.15:4697): protocol: write tcp 159.223.143.39:52730->77.90.31.15:4697: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jun 26 06:08:51.074071824 (68h57m18.000366503s) [noise=true , address=77.90.31.11:4554 , piece_id=R2CIFLWS4LXTGSDSIMIRC4UYVJTXC7T35HPWZ2S7VYUTJ2AMY3BQ, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, address:77.90.31.11:4554): protocol: rpc: dial tcp 77.90.31.11:4554: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jun 26 06:08:51.074102177 (68h57m18.000396855s) [noise=true , address=77.90.31.14:4661 , piece_id=VXI2ZUSGR4N3H7TPIK75RIDFP3SBAOWVRGHNXTZOHFT5FNG4JWJA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue, address:77.90.31.14:4661): protocol: rpc: dial tcp 77.90.31.14:4661: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jun 26 06:08:51.074119276 (68h57m18.000413966s) [seg_pos={0 0}, upload=55354]: Done waiting for segment. successful: 7 optimal: 13 errs: <nil>
Jun 26 06:08:51.100079867 (68h57m18.026374541s) [upload=55354]: Done uploading segments...
Jun 26 06:08:51.100114505 (68h57m18.026409191s) [upload=55354]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 7)
Duration: 456.668788ms
core
uplink
us-select-1
|
checkers.AuthServiceHTTP{"https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us1)antiblip:
1 out of 60 is failed (threshold=3) [............................X...............................] duration:
133.42542ms
FAIL (2025-06-26 06:13:15 UTC)
orbiter warning: unexpected status code "500 Internal Server Error" "\n"
Duration: 133.451024ms
edge
authservice
http
bgp
us1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [.........................X..................................] duration:
643.842469ms
FAIL (2025-06-26 06:13:07 UTC)
orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded
Duration: 643.864612ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [....................................................XX......] duration:
568.438533ms
FAIL (2025-06-26 06:13:06 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 568.463243ms
edge
gateway
us-select-1
|
checkers.AuthServiceHTTP{"https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us1)antiblip:
1 out of 60 is failed (threshold=3) [.............................X..............................] duration:
222.63119ms
FAIL (2025-06-26 06:12:55 UTC)
orbiter warning: unexpected status code "500 Internal Server Error" "\n"
Duration: 222.653175ms
edge
authservice
http
bgp
us1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
3 out of 60 is failed (threshold=3) [....................................................XXX.....] duration:
473.775638ms
FAIL (2025-06-26 06:12:33 UTC)
orbiter warning:
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 7)
======LOGS======
pload=55354]: Got next segment. Inline: false
Jun 26 06:08:21.105827735 (68h56m48.032122411s) [upload=55354]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jun 26 06:08:21.105833588 (68h56m48.032128274s) [upload=55354]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jun 26 06:08:21.135318313 (68h56m48.061613009s) [seg_pos={0 0}, upload=55354]: Begin upload segment...
Jun 26 06:08:21.135366534 (68h56m48.061661214s) [seg_pos={0 0}, upload=55354]: Done begin upload segment.
Jun 26 06:08:21.135390342 (68h56m48.061685028s) [noise=true , address=45.91.116.9:5261 , piece_id=CZWB3XJVA7B7R7GWFYNW3RCGNVRXQ3E37XDXD4SUXTEMQFZ73HHA, seg_pos={0 0}, upload=55354]: Uploading piece...
Jun 26 06:08:21.135405801 (68h56m48.061700497s) [noise=true , address=104.36.82.6:4873 , piece_id=TIIMEQUSDLRFF3OK3WBB74F7SWJ5MEC4Z6IQNELDMRZLNTMQT7LA, seg_pos={0 0}, upload=55354]: Uploading piece...
Jun 26 06:08:21.135774729 (68h56m48.062069418s) [noise=true , address=77.90.31.13:4626 , piece_id=MXYBIEDFAFK53CVTE2C6CKEZWOL3MLW4S7LOHK4IHW6QEIZJL76Q, seg_pos={0 0}, upload=55354]: Uploading piece...
Jun 26 06:08:21.135944436 (68h56m48.06223913s) [noise=true , address=77.90.31.10:4517 , piece_id=VJZ5T4QNVAPNUHNFNK24S6TAS4V4NKNFMQE2FZMCWWLCCB2ELXXQ, seg_pos={0 0}, upload=55354]: Uploading piece...
Jun 26 06:08:21.136080402 (68h56m48.062375094s) [noise=true , address=77.90.31.15:4697 , piece_id=CUXGVL7PS3H7ZQKUCSYXJCBR3ETYFDPZFYIEAWEYL7C2S3RJY3AA, seg_pos={0 0}, upload=55354]: Uploading piece...
Jun 26 06:08:21.136382603 (68h56m48.062677289s) [noise=true , address=104.36.82.13:4186 , piece_id=5SVABN3KS6AUL6HILHNQCVF74BA75VPKTTAK7IQAPKRRME3MOZEA, seg_pos={0 0}, upload=55354]: Uploading piece...
Jun 26 06:08:21.136399304 (68h56m48.062693995s) [noise=true , address=104.36.82.14:4223 , piece_id=LP74YFI2SRSYPYEIKREVINCOFMVCXLNDS2555YKUVFN63N6TBJDA, seg_pos={0 0}, upload=55354]: Uploading piece...
Jun 26 06:08:21.136669576 (68h56m48.062964262s) [noise=true , address=104.36.82.6:4876 , piece_id=SSMGONONY34EHF5W6QMKO7YXB7RLO5LZVTJMMFKTP6GQNGYFYQ4A, seg_pos={0 0}, upload=55354]: Uploading piece...
Jun 26 06:08:21.136809784 (68h56m48.063104477s) [noise=true , address=104.36.82.15:4258 , piece_id=YXS4J7YSG6JV33JG4BZLK5AXKE3JXLBTGZJXPXZQ4LBYBT3MU2AA, seg_pos={0 0}, upload=55354]: Uploading piece...
Jun 26 06:08:21.136950556 (68h56m48.063245242s) [noise=true , address=77.90.31.11:4554 , piece_id=R2CIFLWS4LXTGSDSIMIRC4UYVJTXC7T35HPWZ2S7VYUTJ2AMY3BQ, seg_pos={0 0}, upload=55354]: Uploading piece...
Jun 26 06:08:21.137226497 (68h56m48.06352119s) [noise=true , address=77.90.31.14:4661 , piece_id=VXI2ZUSGR4N3H7TPIK75RIDFP3SBAOWVRGHNXTZOHFT5FNG4JWJA, seg_pos={0 0}, upload=55354]: Uploading piece...
Jun 26 06:08:21.138099107 (68h56m48.064393792s) [noise=true , address=45.91.116.10:4082 , piece_id=7NVEGA4QQOE7KYPHXUUX42SJPQL6LP46RGM37VD6A2AWN7PAOM3Q, seg_pos={0 0}, upload=55354]: Uploading piece...
Jun 26 06:08:21.139330448 (68h56m48.065625141s) [noise=true , address=104.36.82.14:4222 , piece_id=U3MBMAKFBFTOPJZ3HS3FXLPDBLQTN577XNCZLUBHRO6IU4BE5VNA, seg_pos={0 0}, upload=55354]: Uploading piece...
Jun 26 06:08:21.140580734 (68h56m48.06687543s) [noise=true , address=77.90.31.19:4842 , piece_id=G3AG375DATHB7MGEEWONE5GMMIC7QHG2VVDKF5AEVOSXV5N4S6AA, seg_pos={0 0}, upload=55354]: Uploading piece...
Jun 26 06:08:21.141452300 (68h56m48.067747027s) [noise=true , address=45.91.116.13:4154 , piece_id=STFPXN2LUJYUIAYROWZIRTCMT2Y3XET6KWA4GZWUHENXRJURP62Q, seg_pos={0 0}, upload=55354]: Uploading piece...
Jun 26 06:08:21.142917062 (68h56m48.069211739s) [noise=true , address=104.36.82.7:4912 , piece_id=A5HPC7CEU66IFNS47HO3AQX22LUZ6AZ4L4XAMZ6T5NCVE4D3TAZA, seg_pos={0 0}, upload=55354]: Uploading piece...
Jun 26 06:08:21.221824189 (68h56m48.148118865s) [upload=55354]: Next returned nil segment
Jun 26 06:08:21.221837958 (68h56m48.148132644s) [upload=55354]: Waiting for error group managing segments...
Jun 26 06:08:21.229550628 (68h56m48.155845315s) [noise=true , address=104.36.82.14:4223 , piece_id=LP74YFI2SRSYPYEIKREVINCOFMVCXLNDS2555YKUVFN63N6TBJDA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: <nil>
Jun 26 06:08:21.230245145 (68h56m48.156539832s) [noise=true , address=104.36.82.14:4222 , piece_id=U3MBMAKFBFTOPJZ3HS3FXLPDBLQTN577XNCZLUBHRO6IU4BE5VNA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: <nil>
Jun 26 06:08:21.230801067 (68h56m48.15709574s) [noise=true , address=104.36.82.6:4873 , piece_id=TIIMEQUSDLRFF3OK3WBB74F7SWJ5MEC4Z6IQNELDMRZLNTMQT7LA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: <nil>
Jun 26 06:08:21.231457382 (68h56m48.157752071s) [noise=true , address=104.36.82.6:4876 , piece_id=SSMGONONY34EHF5W6QMKO7YXB7RLO5LZVTJMMFKTP6GQNGYFYQ4A, seg_pos={0 0}, upload=55354]: Done uploading piece. err: <nil>
Jun 26 06:08:21.231563205 (68h56m48.157857893s) [noise=true , address=104.36.82.13:4186 , piece_id=5SVABN3KS6AUL6HILHNQCVF74BA75VPKTTAK7IQAPKRRME3MOZEA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: <nil>
Jun 26 06:08:21.239131119 (68h56m48.165425799s) [noise=true , address=104.36.82.7:4912 , piece_id=A5HPC7CEU66IFNS47HO3AQX22LUZ6AZ4L4XAMZ6T5NCVE4D3TAZA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: <nil>
Jun 26 06:08:21.534686665 (68h56m48.460981337s) [noise=true , address=104.36.82.15:4258 , piece_id=YXS4J7YSG6JV33JG4BZLK5AXKE3JXLBTGZJXPXZQ4LBYBT3MU2AA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: <nil>
Jun 26 06:08:51.073673533 (68h57m17.999968219s) [noise=true , address=45.91.116.9:5261 , piece_id=CZWB3XJVA7B7R7GWFYNW3RCGNVRXQ3E37XDXD4SUXTEMQFZ73HHA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi, address:45.91.116.9:5261): protocol: rpc: dial tcp 45.91.116.9:5261: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jun 26 06:08:51.073739594 (68h57m18.000034272s) [noise=true , address=45.91.116.10:4082 , piece_id=7NVEGA4QQOE7KYPHXUUX42SJPQL6LP46RGM37VD6A2AWN7PAOM3Q, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj, address:45.91.116.10:4082): protocol: rpc: dial tcp 45.91.116.10:4082: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jun 26 06:08:51.073789897 (68h57m18.000084584s) [noise=true , address=77.90.31.19:4842 , piece_id=G3AG375DATHB7MGEEWONE5GMMIC7QHG2VVDKF5AEVOSXV5N4S6AA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: rpc: dial tcp 77.90.31.19:4842: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jun 26 06:08:51.073849570 (68h57m18.000144246s) [noise=true , address=45.91.116.13:4154 , piece_id=STFPXN2LUJYUIAYROWZIRTCMT2Y3XET6KWA4GZWUHENXRJURP62Q, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN, address:45.91.116.13:4154): protocol: write tcp 159.223.143.39:35392->45.91.116.13:4154: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jun 26 06:08:51.073904716 (68h57m18.000199391s) [noise=true , address=77.90.31.13:4626 , piece_id=MXYBIEDFAFK53CVTE2C6CKEZWOL3MLW4S7LOHK4IHW6QEIZJL76Q, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: write tcp 159.223.143.39:58864->77.90.31.13:4626: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jun 26 06:08:51.073952402 (68h57m18.000247082s) [noise=true , address=77.90.31.10:4517 , piece_id=VJZ5T4QNVAPNUHNFNK24S6TAS4V4NKNFMQE2FZMCWWLCCB2ELXXQ, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, address:77.90.31.10:4517): protocol: write tcp 159.223.143.39:36382->77.90.31.10:4517: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jun 26 06:08:51.074022481 (68h57m18.000317156s) [noise=true , address=77.90.31.15:4697 , piece_id=CUXGVL7PS3H7ZQKUCSYXJCBR3ETYFDPZFYIEAWEYL7C2S3RJY3AA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5, address:77.90.31.15:4697): protocol: write tcp 159.223.143.39:52730->77.90.31.15:4697: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jun 26 06:08:51.074071824 (68h57m18.000366503s) [noise=true , address=77.90.31.11:4554 , piece_id=R2CIFLWS4LXTGSDSIMIRC4UYVJTXC7T35HPWZ2S7VYUTJ2AMY3BQ, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, address:77.90.31.11:4554): protocol: rpc: dial tcp 77.90.31.11:4554: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jun 26 06:08:51.074102177 (68h57m18.000396855s) [noise=true , address=77.90.31.14:4661 , piece_id=VXI2ZUSGR4N3H7TPIK75RIDFP3SBAOWVRGHNXTZOHFT5FNG4JWJA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue, address:77.90.31.14:4661): protocol: rpc: dial tcp 77.90.31.14:4661: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jun 26 06:08:51.074119276 (68h57m18.000413966s) [seg_pos={0 0}, upload=55354]: Done waiting for segment. successful: 7 optimal: 13 errs: <nil>
Jun 26 06:08:51.100079867 (68h57m18.026374541s) [upload=55354]: Done uploading segments...
Jun 26 06:08:51.100114505 (68h57m18.026409191s) [upload=55354]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 7)
Duration: 473.807955ms
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [.....................................................XX.....] duration:
708.442548ms
FAIL (2025-06-26 06:12:16 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 708.469463ms
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [..........................X.................................] duration:
647.040165ms
FAIL (2025-06-26 06:12:07 UTC)
orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded
Duration: 647.058678ms
core
uplink
us-select-1
us1-placement-20
|
checkers.AuthServiceHTTP{"https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us1)antiblip:
1 out of 60 is failed (threshold=3) [..............................X.............................] duration:
182.520178ms
FAIL (2025-06-26 06:11:55 UTC)
orbiter warning: unexpected status code "500 Internal Server Error" "\n"
Duration: 182.544361ms
edge
authservice
http
bgp
us1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
3 out of 60 is failed (threshold=3) [.....................................................XXX....] duration:
570.384914ms
FAIL (2025-06-26 06:11:54 UTC)
orbiter warning:
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 7)
======LOGS======
pload=55354]: Got next segment. Inline: false
Jun 26 06:08:21.105827735 (68h56m48.032122411s) [upload=55354]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jun 26 06:08:21.105833588 (68h56m48.032128274s) [upload=55354]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jun 26 06:08:21.135318313 (68h56m48.061613009s) [seg_pos={0 0}, upload=55354]: Begin upload segment...
Jun 26 06:08:21.135366534 (68h56m48.061661214s) [seg_pos={0 0}, upload=55354]: Done begin upload segment.
Jun 26 06:08:21.135390342 (68h56m48.061685028s) [noise=true , address=45.91.116.9:5261 , piece_id=CZWB3XJVA7B7R7GWFYNW3RCGNVRXQ3E37XDXD4SUXTEMQFZ73HHA, seg_pos={0 0}, upload=55354]: Uploading piece...
Jun 26 06:08:21.135405801 (68h56m48.061700497s) [noise=true , address=104.36.82.6:4873 , piece_id=TIIMEQUSDLRFF3OK3WBB74F7SWJ5MEC4Z6IQNELDMRZLNTMQT7LA, seg_pos={0 0}, upload=55354]: Uploading piece...
Jun 26 06:08:21.135774729 (68h56m48.062069418s) [noise=true , address=77.90.31.13:4626 , piece_id=MXYBIEDFAFK53CVTE2C6CKEZWOL3MLW4S7LOHK4IHW6QEIZJL76Q, seg_pos={0 0}, upload=55354]: Uploading piece...
Jun 26 06:08:21.135944436 (68h56m48.06223913s) [noise=true , address=77.90.31.10:4517 , piece_id=VJZ5T4QNVAPNUHNFNK24S6TAS4V4NKNFMQE2FZMCWWLCCB2ELXXQ, seg_pos={0 0}, upload=55354]: Uploading piece...
Jun 26 06:08:21.136080402 (68h56m48.062375094s) [noise=true , address=77.90.31.15:4697 , piece_id=CUXGVL7PS3H7ZQKUCSYXJCBR3ETYFDPZFYIEAWEYL7C2S3RJY3AA, seg_pos={0 0}, upload=55354]: Uploading piece...
Jun 26 06:08:21.136382603 (68h56m48.062677289s) [noise=true , address=104.36.82.13:4186 , piece_id=5SVABN3KS6AUL6HILHNQCVF74BA75VPKTTAK7IQAPKRRME3MOZEA, seg_pos={0 0}, upload=55354]: Uploading piece...
Jun 26 06:08:21.136399304 (68h56m48.062693995s) [noise=true , address=104.36.82.14:4223 , piece_id=LP74YFI2SRSYPYEIKREVINCOFMVCXLNDS2555YKUVFN63N6TBJDA, seg_pos={0 0}, upload=55354]: Uploading piece...
Jun 26 06:08:21.136669576 (68h56m48.062964262s) [noise=true , address=104.36.82.6:4876 , piece_id=SSMGONONY34EHF5W6QMKO7YXB7RLO5LZVTJMMFKTP6GQNGYFYQ4A, seg_pos={0 0}, upload=55354]: Uploading piece...
Jun 26 06:08:21.136809784 (68h56m48.063104477s) [noise=true , address=104.36.82.15:4258 , piece_id=YXS4J7YSG6JV33JG4BZLK5AXKE3JXLBTGZJXPXZQ4LBYBT3MU2AA, seg_pos={0 0}, upload=55354]: Uploading piece...
Jun 26 06:08:21.136950556 (68h56m48.063245242s) [noise=true , address=77.90.31.11:4554 , piece_id=R2CIFLWS4LXTGSDSIMIRC4UYVJTXC7T35HPWZ2S7VYUTJ2AMY3BQ, seg_pos={0 0}, upload=55354]: Uploading piece...
Jun 26 06:08:21.137226497 (68h56m48.06352119s) [noise=true , address=77.90.31.14:4661 , piece_id=VXI2ZUSGR4N3H7TPIK75RIDFP3SBAOWVRGHNXTZOHFT5FNG4JWJA, seg_pos={0 0}, upload=55354]: Uploading piece...
Jun 26 06:08:21.138099107 (68h56m48.064393792s) [noise=true , address=45.91.116.10:4082 , piece_id=7NVEGA4QQOE7KYPHXUUX42SJPQL6LP46RGM37VD6A2AWN7PAOM3Q, seg_pos={0 0}, upload=55354]: Uploading piece...
Jun 26 06:08:21.139330448 (68h56m48.065625141s) [noise=true , address=104.36.82.14:4222 , piece_id=U3MBMAKFBFTOPJZ3HS3FXLPDBLQTN577XNCZLUBHRO6IU4BE5VNA, seg_pos={0 0}, upload=55354]: Uploading piece...
Jun 26 06:08:21.140580734 (68h56m48.06687543s) [noise=true , address=77.90.31.19:4842 , piece_id=G3AG375DATHB7MGEEWONE5GMMIC7QHG2VVDKF5AEVOSXV5N4S6AA, seg_pos={0 0}, upload=55354]: Uploading piece...
Jun 26 06:08:21.141452300 (68h56m48.067747027s) [noise=true , address=45.91.116.13:4154 , piece_id=STFPXN2LUJYUIAYROWZIRTCMT2Y3XET6KWA4GZWUHENXRJURP62Q, seg_pos={0 0}, upload=55354]: Uploading piece...
Jun 26 06:08:21.142917062 (68h56m48.069211739s) [noise=true , address=104.36.82.7:4912 , piece_id=A5HPC7CEU66IFNS47HO3AQX22LUZ6AZ4L4XAMZ6T5NCVE4D3TAZA, seg_pos={0 0}, upload=55354]: Uploading piece...
Jun 26 06:08:21.221824189 (68h56m48.148118865s) [upload=55354]: Next returned nil segment
Jun 26 06:08:21.221837958 (68h56m48.148132644s) [upload=55354]: Waiting for error group managing segments...
Jun 26 06:08:21.229550628 (68h56m48.155845315s) [noise=true , address=104.36.82.14:4223 , piece_id=LP74YFI2SRSYPYEIKREVINCOFMVCXLNDS2555YKUVFN63N6TBJDA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: <nil>
Jun 26 06:08:21.230245145 (68h56m48.156539832s) [noise=true , address=104.36.82.14:4222 , piece_id=U3MBMAKFBFTOPJZ3HS3FXLPDBLQTN577XNCZLUBHRO6IU4BE5VNA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: <nil>
Jun 26 06:08:21.230801067 (68h56m48.15709574s) [noise=true , address=104.36.82.6:4873 , piece_id=TIIMEQUSDLRFF3OK3WBB74F7SWJ5MEC4Z6IQNELDMRZLNTMQT7LA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: <nil>
Jun 26 06:08:21.231457382 (68h56m48.157752071s) [noise=true , address=104.36.82.6:4876 , piece_id=SSMGONONY34EHF5W6QMKO7YXB7RLO5LZVTJMMFKTP6GQNGYFYQ4A, seg_pos={0 0}, upload=55354]: Done uploading piece. err: <nil>
Jun 26 06:08:21.231563205 (68h56m48.157857893s) [noise=true , address=104.36.82.13:4186 , piece_id=5SVABN3KS6AUL6HILHNQCVF74BA75VPKTTAK7IQAPKRRME3MOZEA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: <nil>
Jun 26 06:08:21.239131119 (68h56m48.165425799s) [noise=true , address=104.36.82.7:4912 , piece_id=A5HPC7CEU66IFNS47HO3AQX22LUZ6AZ4L4XAMZ6T5NCVE4D3TAZA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: <nil>
Jun 26 06:08:21.534686665 (68h56m48.460981337s) [noise=true , address=104.36.82.15:4258 , piece_id=YXS4J7YSG6JV33JG4BZLK5AXKE3JXLBTGZJXPXZQ4LBYBT3MU2AA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: <nil>
Jun 26 06:08:51.073673533 (68h57m17.999968219s) [noise=true , address=45.91.116.9:5261 , piece_id=CZWB3XJVA7B7R7GWFYNW3RCGNVRXQ3E37XDXD4SUXTEMQFZ73HHA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi, address:45.91.116.9:5261): protocol: rpc: dial tcp 45.91.116.9:5261: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jun 26 06:08:51.073739594 (68h57m18.000034272s) [noise=true , address=45.91.116.10:4082 , piece_id=7NVEGA4QQOE7KYPHXUUX42SJPQL6LP46RGM37VD6A2AWN7PAOM3Q, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj, address:45.91.116.10:4082): protocol: rpc: dial tcp 45.91.116.10:4082: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jun 26 06:08:51.073789897 (68h57m18.000084584s) [noise=true , address=77.90.31.19:4842 , piece_id=G3AG375DATHB7MGEEWONE5GMMIC7QHG2VVDKF5AEVOSXV5N4S6AA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: rpc: dial tcp 77.90.31.19:4842: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jun 26 06:08:51.073849570 (68h57m18.000144246s) [noise=true , address=45.91.116.13:4154 , piece_id=STFPXN2LUJYUIAYROWZIRTCMT2Y3XET6KWA4GZWUHENXRJURP62Q, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN, address:45.91.116.13:4154): protocol: write tcp 159.223.143.39:35392->45.91.116.13:4154: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jun 26 06:08:51.073904716 (68h57m18.000199391s) [noise=true , address=77.90.31.13:4626 , piece_id=MXYBIEDFAFK53CVTE2C6CKEZWOL3MLW4S7LOHK4IHW6QEIZJL76Q, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: write tcp 159.223.143.39:58864->77.90.31.13:4626: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jun 26 06:08:51.073952402 (68h57m18.000247082s) [noise=true , address=77.90.31.10:4517 , piece_id=VJZ5T4QNVAPNUHNFNK24S6TAS4V4NKNFMQE2FZMCWWLCCB2ELXXQ, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, address:77.90.31.10:4517): protocol: write tcp 159.223.143.39:36382->77.90.31.10:4517: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jun 26 06:08:51.074022481 (68h57m18.000317156s) [noise=true , address=77.90.31.15:4697 , piece_id=CUXGVL7PS3H7ZQKUCSYXJCBR3ETYFDPZFYIEAWEYL7C2S3RJY3AA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5, address:77.90.31.15:4697): protocol: write tcp 159.223.143.39:52730->77.90.31.15:4697: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jun 26 06:08:51.074071824 (68h57m18.000366503s) [noise=true , address=77.90.31.11:4554 , piece_id=R2CIFLWS4LXTGSDSIMIRC4UYVJTXC7T35HPWZ2S7VYUTJ2AMY3BQ, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, address:77.90.31.11:4554): protocol: rpc: dial tcp 77.90.31.11:4554: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jun 26 06:08:51.074102177 (68h57m18.000396855s) [noise=true , address=77.90.31.14:4661 , piece_id=VXI2ZUSGR4N3H7TPIK75RIDFP3SBAOWVRGHNXTZOHFT5FNG4JWJA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue, address:77.90.31.14:4661): protocol: rpc: dial tcp 77.90.31.14:4661: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jun 26 06:08:51.074119276 (68h57m18.000413966s) [seg_pos={0 0}, upload=55354]: Done waiting for segment. successful: 7 optimal: 13 errs: <nil>
Jun 26 06:08:51.100079867 (68h57m18.026374541s) [upload=55354]: Done uploading segments...
Jun 26 06:08:51.100114505 (68h57m18.026409191s) [upload=55354]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 7)
Duration: 570.414948ms
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [...........................X................................] duration:
695.710072ms
FAIL (2025-06-26 06:11:13 UTC)
orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded
Duration: 695.739829ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [......................................................XX....] duration:
791.494065ms
FAIL (2025-06-26 06:11:08 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 791.512731ms
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
3 out of 60 is failed (threshold=3) [......................................................XXX...] duration:
604.543051ms
FAIL (2025-06-26 06:11:02 UTC)
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 7)
======LOGS======
pload=55354]: Got next segment. Inline: false
Jun 26 06:08:21.105827735 (68h56m48.032122411s) [upload=55354]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jun 26 06:08:21.105833588 (68h56m48.032128274s) [upload=55354]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jun 26 06:08:21.135318313 (68h56m48.061613009s) [seg_pos={0 0}, upload=55354]: Begin upload segment...
Jun 26 06:08:21.135366534 (68h56m48.061661214s) [seg_pos={0 0}, upload=55354]: Done begin upload segment.
Jun 26 06:08:21.135390342 (68h56m48.061685028s) [noise=true , address=45.91.116.9:5261 , piece_id=CZWB3XJVA7B7R7GWFYNW3RCGNVRXQ3E37XDXD4SUXTEMQFZ73HHA, seg_pos={0 0}, upload=55354]: Uploading piece...
Jun 26 06:08:21.135405801 (68h56m48.061700497s) [noise=true , address=104.36.82.6:4873 , piece_id=TIIMEQUSDLRFF3OK3WBB74F7SWJ5MEC4Z6IQNELDMRZLNTMQT7LA, seg_pos={0 0}, upload=55354]: Uploading piece...
Jun 26 06:08:21.135774729 (68h56m48.062069418s) [noise=true , address=77.90.31.13:4626 , piece_id=MXYBIEDFAFK53CVTE2C6CKEZWOL3MLW4S7LOHK4IHW6QEIZJL76Q, seg_pos={0 0}, upload=55354]: Uploading piece...
Jun 26 06:08:21.135944436 (68h56m48.06223913s) [noise=true , address=77.90.31.10:4517 , piece_id=VJZ5T4QNVAPNUHNFNK24S6TAS4V4NKNFMQE2FZMCWWLCCB2ELXXQ, seg_pos={0 0}, upload=55354]: Uploading piece...
Jun 26 06:08:21.136080402 (68h56m48.062375094s) [noise=true , address=77.90.31.15:4697 , piece_id=CUXGVL7PS3H7ZQKUCSYXJCBR3ETYFDPZFYIEAWEYL7C2S3RJY3AA, seg_pos={0 0}, upload=55354]: Uploading piece...
Jun 26 06:08:21.136382603 (68h56m48.062677289s) [noise=true , address=104.36.82.13:4186 , piece_id=5SVABN3KS6AUL6HILHNQCVF74BA75VPKTTAK7IQAPKRRME3MOZEA, seg_pos={0 0}, upload=55354]: Uploading piece...
Jun 26 06:08:21.136399304 (68h56m48.062693995s) [noise=true , address=104.36.82.14:4223 , piece_id=LP74YFI2SRSYPYEIKREVINCOFMVCXLNDS2555YKUVFN63N6TBJDA, seg_pos={0 0}, upload=55354]: Uploading piece...
Jun 26 06:08:21.136669576 (68h56m48.062964262s) [noise=true , address=104.36.82.6:4876 , piece_id=SSMGONONY34EHF5W6QMKO7YXB7RLO5LZVTJMMFKTP6GQNGYFYQ4A, seg_pos={0 0}, upload=55354]: Uploading piece...
Jun 26 06:08:21.136809784 (68h56m48.063104477s) [noise=true , address=104.36.82.15:4258 , piece_id=YXS4J7YSG6JV33JG4BZLK5AXKE3JXLBTGZJXPXZQ4LBYBT3MU2AA, seg_pos={0 0}, upload=55354]: Uploading piece...
Jun 26 06:08:21.136950556 (68h56m48.063245242s) [noise=true , address=77.90.31.11:4554 , piece_id=R2CIFLWS4LXTGSDSIMIRC4UYVJTXC7T35HPWZ2S7VYUTJ2AMY3BQ, seg_pos={0 0}, upload=55354]: Uploading piece...
Jun 26 06:08:21.137226497 (68h56m48.06352119s) [noise=true , address=77.90.31.14:4661 , piece_id=VXI2ZUSGR4N3H7TPIK75RIDFP3SBAOWVRGHNXTZOHFT5FNG4JWJA, seg_pos={0 0}, upload=55354]: Uploading piece...
Jun 26 06:08:21.138099107 (68h56m48.064393792s) [noise=true , address=45.91.116.10:4082 , piece_id=7NVEGA4QQOE7KYPHXUUX42SJPQL6LP46RGM37VD6A2AWN7PAOM3Q, seg_pos={0 0}, upload=55354]: Uploading piece...
Jun 26 06:08:21.139330448 (68h56m48.065625141s) [noise=true , address=104.36.82.14:4222 , piece_id=U3MBMAKFBFTOPJZ3HS3FXLPDBLQTN577XNCZLUBHRO6IU4BE5VNA, seg_pos={0 0}, upload=55354]: Uploading piece...
Jun 26 06:08:21.140580734 (68h56m48.06687543s) [noise=true , address=77.90.31.19:4842 , piece_id=G3AG375DATHB7MGEEWONE5GMMIC7QHG2VVDKF5AEVOSXV5N4S6AA, seg_pos={0 0}, upload=55354]: Uploading piece...
Jun 26 06:08:21.141452300 (68h56m48.067747027s) [noise=true , address=45.91.116.13:4154 , piece_id=STFPXN2LUJYUIAYROWZIRTCMT2Y3XET6KWA4GZWUHENXRJURP62Q, seg_pos={0 0}, upload=55354]: Uploading piece...
Jun 26 06:08:21.142917062 (68h56m48.069211739s) [noise=true , address=104.36.82.7:4912 , piece_id=A5HPC7CEU66IFNS47HO3AQX22LUZ6AZ4L4XAMZ6T5NCVE4D3TAZA, seg_pos={0 0}, upload=55354]: Uploading piece...
Jun 26 06:08:21.221824189 (68h56m48.148118865s) [upload=55354]: Next returned nil segment
Jun 26 06:08:21.221837958 (68h56m48.148132644s) [upload=55354]: Waiting for error group managing segments...
Jun 26 06:08:21.229550628 (68h56m48.155845315s) [noise=true , address=104.36.82.14:4223 , piece_id=LP74YFI2SRSYPYEIKREVINCOFMVCXLNDS2555YKUVFN63N6TBJDA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: <nil>
Jun 26 06:08:21.230245145 (68h56m48.156539832s) [noise=true , address=104.36.82.14:4222 , piece_id=U3MBMAKFBFTOPJZ3HS3FXLPDBLQTN577XNCZLUBHRO6IU4BE5VNA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: <nil>
Jun 26 06:08:21.230801067 (68h56m48.15709574s) [noise=true , address=104.36.82.6:4873 , piece_id=TIIMEQUSDLRFF3OK3WBB74F7SWJ5MEC4Z6IQNELDMRZLNTMQT7LA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: <nil>
Jun 26 06:08:21.231457382 (68h56m48.157752071s) [noise=true , address=104.36.82.6:4876 , piece_id=SSMGONONY34EHF5W6QMKO7YXB7RLO5LZVTJMMFKTP6GQNGYFYQ4A, seg_pos={0 0}, upload=55354]: Done uploading piece. err: <nil>
Jun 26 06:08:21.231563205 (68h56m48.157857893s) [noise=true , address=104.36.82.13:4186 , piece_id=5SVABN3KS6AUL6HILHNQCVF74BA75VPKTTAK7IQAPKRRME3MOZEA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: <nil>
Jun 26 06:08:21.239131119 (68h56m48.165425799s) [noise=true , address=104.36.82.7:4912 , piece_id=A5HPC7CEU66IFNS47HO3AQX22LUZ6AZ4L4XAMZ6T5NCVE4D3TAZA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: <nil>
Jun 26 06:08:21.534686665 (68h56m48.460981337s) [noise=true , address=104.36.82.15:4258 , piece_id=YXS4J7YSG6JV33JG4BZLK5AXKE3JXLBTGZJXPXZQ4LBYBT3MU2AA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: <nil>
Jun 26 06:08:51.073673533 (68h57m17.999968219s) [noise=true , address=45.91.116.9:5261 , piece_id=CZWB3XJVA7B7R7GWFYNW3RCGNVRXQ3E37XDXD4SUXTEMQFZ73HHA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi, address:45.91.116.9:5261): protocol: rpc: dial tcp 45.91.116.9:5261: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jun 26 06:08:51.073739594 (68h57m18.000034272s) [noise=true , address=45.91.116.10:4082 , piece_id=7NVEGA4QQOE7KYPHXUUX42SJPQL6LP46RGM37VD6A2AWN7PAOM3Q, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj, address:45.91.116.10:4082): protocol: rpc: dial tcp 45.91.116.10:4082: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jun 26 06:08:51.073789897 (68h57m18.000084584s) [noise=true , address=77.90.31.19:4842 , piece_id=G3AG375DATHB7MGEEWONE5GMMIC7QHG2VVDKF5AEVOSXV5N4S6AA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: rpc: dial tcp 77.90.31.19:4842: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jun 26 06:08:51.073849570 (68h57m18.000144246s) [noise=true , address=45.91.116.13:4154 , piece_id=STFPXN2LUJYUIAYROWZIRTCMT2Y3XET6KWA4GZWUHENXRJURP62Q, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN, address:45.91.116.13:4154): protocol: write tcp 159.223.143.39:35392->45.91.116.13:4154: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jun 26 06:08:51.073904716 (68h57m18.000199391s) [noise=true , address=77.90.31.13:4626 , piece_id=MXYBIEDFAFK53CVTE2C6CKEZWOL3MLW4S7LOHK4IHW6QEIZJL76Q, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: write tcp 159.223.143.39:58864->77.90.31.13:4626: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jun 26 06:08:51.073952402 (68h57m18.000247082s) [noise=true , address=77.90.31.10:4517 , piece_id=VJZ5T4QNVAPNUHNFNK24S6TAS4V4NKNFMQE2FZMCWWLCCB2ELXXQ, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, address:77.90.31.10:4517): protocol: write tcp 159.223.143.39:36382->77.90.31.10:4517: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jun 26 06:08:51.074022481 (68h57m18.000317156s) [noise=true , address=77.90.31.15:4697 , piece_id=CUXGVL7PS3H7ZQKUCSYXJCBR3ETYFDPZFYIEAWEYL7C2S3RJY3AA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5, address:77.90.31.15:4697): protocol: write tcp 159.223.143.39:52730->77.90.31.15:4697: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jun 26 06:08:51.074071824 (68h57m18.000366503s) [noise=true , address=77.90.31.11:4554 , piece_id=R2CIFLWS4LXTGSDSIMIRC4UYVJTXC7T35HPWZ2S7VYUTJ2AMY3BQ, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, address:77.90.31.11:4554): protocol: rpc: dial tcp 77.90.31.11:4554: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jun 26 06:08:51.074102177 (68h57m18.000396855s) [noise=true , address=77.90.31.14:4661 , piece_id=VXI2ZUSGR4N3H7TPIK75RIDFP3SBAOWVRGHNXTZOHFT5FNG4JWJA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue, address:77.90.31.14:4661): protocol: rpc: dial tcp 77.90.31.14:4661: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jun 26 06:08:51.074119276 (68h57m18.000413966s) [seg_pos={0 0}, upload=55354]: Done waiting for segment. successful: 7 optimal: 13 errs: <nil>
Jun 26 06:08:51.100079867 (68h57m18.026374541s) [upload=55354]: Done uploading segments...
Jun 26 06:08:51.100114505 (68h57m18.026409191s) [upload=55354]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 7)
Duration: 604.642074ms
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [............................X...............................] duration:
706.615882ms
FAIL (2025-06-26 06:10:46 UTC)
orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded
Duration: 706.644825ms
core
uplink
us-select-1
us1-placement-20
|
checkers.AuthServiceHTTP{"https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us1)antiblip:
1 out of 60 is failed (threshold=3) [...............................X............................] duration:
214.827842ms
FAIL (2025-06-26 06:10:42 UTC)
orbiter warning: unexpected status code "500 Internal Server Error" "\n"
Duration: 214.864833ms
edge
authservice
http
bgp
us1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [.............................X..............................] duration:
731.591564ms
FAIL (2025-06-26 06:10:30 UTC)
orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded
Duration: 731.617741ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
3 out of 60 is failed (threshold=3) [.......................................................XXX..] duration:
701.107774ms
FAIL (2025-06-26 06:10:14 UTC)
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 7)
======LOGS======
pload=55354]: Got next segment. Inline: false
Jun 26 06:08:21.105827735 (68h56m48.032122411s) [upload=55354]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jun 26 06:08:21.105833588 (68h56m48.032128274s) [upload=55354]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jun 26 06:08:21.135318313 (68h56m48.061613009s) [seg_pos={0 0}, upload=55354]: Begin upload segment...
Jun 26 06:08:21.135366534 (68h56m48.061661214s) [seg_pos={0 0}, upload=55354]: Done begin upload segment.
Jun 26 06:08:21.135390342 (68h56m48.061685028s) [noise=true , address=45.91.116.9:5261 , piece_id=CZWB3XJVA7B7R7GWFYNW3RCGNVRXQ3E37XDXD4SUXTEMQFZ73HHA, seg_pos={0 0}, upload=55354]: Uploading piece...
Jun 26 06:08:21.135405801 (68h56m48.061700497s) [noise=true , address=104.36.82.6:4873 , piece_id=TIIMEQUSDLRFF3OK3WBB74F7SWJ5MEC4Z6IQNELDMRZLNTMQT7LA, seg_pos={0 0}, upload=55354]: Uploading piece...
Jun 26 06:08:21.135774729 (68h56m48.062069418s) [noise=true , address=77.90.31.13:4626 , piece_id=MXYBIEDFAFK53CVTE2C6CKEZWOL3MLW4S7LOHK4IHW6QEIZJL76Q, seg_pos={0 0}, upload=55354]: Uploading piece...
Jun 26 06:08:21.135944436 (68h56m48.06223913s) [noise=true , address=77.90.31.10:4517 , piece_id=VJZ5T4QNVAPNUHNFNK24S6TAS4V4NKNFMQE2FZMCWWLCCB2ELXXQ, seg_pos={0 0}, upload=55354]: Uploading piece...
Jun 26 06:08:21.136080402 (68h56m48.062375094s) [noise=true , address=77.90.31.15:4697 , piece_id=CUXGVL7PS3H7ZQKUCSYXJCBR3ETYFDPZFYIEAWEYL7C2S3RJY3AA, seg_pos={0 0}, upload=55354]: Uploading piece...
Jun 26 06:08:21.136382603 (68h56m48.062677289s) [noise=true , address=104.36.82.13:4186 , piece_id=5SVABN3KS6AUL6HILHNQCVF74BA75VPKTTAK7IQAPKRRME3MOZEA, seg_pos={0 0}, upload=55354]: Uploading piece...
Jun 26 06:08:21.136399304 (68h56m48.062693995s) [noise=true , address=104.36.82.14:4223 , piece_id=LP74YFI2SRSYPYEIKREVINCOFMVCXLNDS2555YKUVFN63N6TBJDA, seg_pos={0 0}, upload=55354]: Uploading piece...
Jun 26 06:08:21.136669576 (68h56m48.062964262s) [noise=true , address=104.36.82.6:4876 , piece_id=SSMGONONY34EHF5W6QMKO7YXB7RLO5LZVTJMMFKTP6GQNGYFYQ4A, seg_pos={0 0}, upload=55354]: Uploading piece...
Jun 26 06:08:21.136809784 (68h56m48.063104477s) [noise=true , address=104.36.82.15:4258 , piece_id=YXS4J7YSG6JV33JG4BZLK5AXKE3JXLBTGZJXPXZQ4LBYBT3MU2AA, seg_pos={0 0}, upload=55354]: Uploading piece...
Jun 26 06:08:21.136950556 (68h56m48.063245242s) [noise=true , address=77.90.31.11:4554 , piece_id=R2CIFLWS4LXTGSDSIMIRC4UYVJTXC7T35HPWZ2S7VYUTJ2AMY3BQ, seg_pos={0 0}, upload=55354]: Uploading piece...
Jun 26 06:08:21.137226497 (68h56m48.06352119s) [noise=true , address=77.90.31.14:4661 , piece_id=VXI2ZUSGR4N3H7TPIK75RIDFP3SBAOWVRGHNXTZOHFT5FNG4JWJA, seg_pos={0 0}, upload=55354]: Uploading piece...
Jun 26 06:08:21.138099107 (68h56m48.064393792s) [noise=true , address=45.91.116.10:4082 , piece_id=7NVEGA4QQOE7KYPHXUUX42SJPQL6LP46RGM37VD6A2AWN7PAOM3Q, seg_pos={0 0}, upload=55354]: Uploading piece...
Jun 26 06:08:21.139330448 (68h56m48.065625141s) [noise=true , address=104.36.82.14:4222 , piece_id=U3MBMAKFBFTOPJZ3HS3FXLPDBLQTN577XNCZLUBHRO6IU4BE5VNA, seg_pos={0 0}, upload=55354]: Uploading piece...
Jun 26 06:08:21.140580734 (68h56m48.06687543s) [noise=true , address=77.90.31.19:4842 , piece_id=G3AG375DATHB7MGEEWONE5GMMIC7QHG2VVDKF5AEVOSXV5N4S6AA, seg_pos={0 0}, upload=55354]: Uploading piece...
Jun 26 06:08:21.141452300 (68h56m48.067747027s) [noise=true , address=45.91.116.13:4154 , piece_id=STFPXN2LUJYUIAYROWZIRTCMT2Y3XET6KWA4GZWUHENXRJURP62Q, seg_pos={0 0}, upload=55354]: Uploading piece...
Jun 26 06:08:21.142917062 (68h56m48.069211739s) [noise=true , address=104.36.82.7:4912 , piece_id=A5HPC7CEU66IFNS47HO3AQX22LUZ6AZ4L4XAMZ6T5NCVE4D3TAZA, seg_pos={0 0}, upload=55354]: Uploading piece...
Jun 26 06:08:21.221824189 (68h56m48.148118865s) [upload=55354]: Next returned nil segment
Jun 26 06:08:21.221837958 (68h56m48.148132644s) [upload=55354]: Waiting for error group managing segments...
Jun 26 06:08:21.229550628 (68h56m48.155845315s) [noise=true , address=104.36.82.14:4223 , piece_id=LP74YFI2SRSYPYEIKREVINCOFMVCXLNDS2555YKUVFN63N6TBJDA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: <nil>
Jun 26 06:08:21.230245145 (68h56m48.156539832s) [noise=true , address=104.36.82.14:4222 , piece_id=U3MBMAKFBFTOPJZ3HS3FXLPDBLQTN577XNCZLUBHRO6IU4BE5VNA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: <nil>
Jun 26 06:08:21.230801067 (68h56m48.15709574s) [noise=true , address=104.36.82.6:4873 , piece_id=TIIMEQUSDLRFF3OK3WBB74F7SWJ5MEC4Z6IQNELDMRZLNTMQT7LA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: <nil>
Jun 26 06:08:21.231457382 (68h56m48.157752071s) [noise=true , address=104.36.82.6:4876 , piece_id=SSMGONONY34EHF5W6QMKO7YXB7RLO5LZVTJMMFKTP6GQNGYFYQ4A, seg_pos={0 0}, upload=55354]: Done uploading piece. err: <nil>
Jun 26 06:08:21.231563205 (68h56m48.157857893s) [noise=true , address=104.36.82.13:4186 , piece_id=5SVABN3KS6AUL6HILHNQCVF74BA75VPKTTAK7IQAPKRRME3MOZEA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: <nil>
Jun 26 06:08:21.239131119 (68h56m48.165425799s) [noise=true , address=104.36.82.7:4912 , piece_id=A5HPC7CEU66IFNS47HO3AQX22LUZ6AZ4L4XAMZ6T5NCVE4D3TAZA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: <nil>
Jun 26 06:08:21.534686665 (68h56m48.460981337s) [noise=true , address=104.36.82.15:4258 , piece_id=YXS4J7YSG6JV33JG4BZLK5AXKE3JXLBTGZJXPXZQ4LBYBT3MU2AA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: <nil>
Jun 26 06:08:51.073673533 (68h57m17.999968219s) [noise=true , address=45.91.116.9:5261 , piece_id=CZWB3XJVA7B7R7GWFYNW3RCGNVRXQ3E37XDXD4SUXTEMQFZ73HHA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi, address:45.91.116.9:5261): protocol: rpc: dial tcp 45.91.116.9:5261: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jun 26 06:08:51.073739594 (68h57m18.000034272s) [noise=true , address=45.91.116.10:4082 , piece_id=7NVEGA4QQOE7KYPHXUUX42SJPQL6LP46RGM37VD6A2AWN7PAOM3Q, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj, address:45.91.116.10:4082): protocol: rpc: dial tcp 45.91.116.10:4082: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jun 26 06:08:51.073789897 (68h57m18.000084584s) [noise=true , address=77.90.31.19:4842 , piece_id=G3AG375DATHB7MGEEWONE5GMMIC7QHG2VVDKF5AEVOSXV5N4S6AA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: rpc: dial tcp 77.90.31.19:4842: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jun 26 06:08:51.073849570 (68h57m18.000144246s) [noise=true , address=45.91.116.13:4154 , piece_id=STFPXN2LUJYUIAYROWZIRTCMT2Y3XET6KWA4GZWUHENXRJURP62Q, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN, address:45.91.116.13:4154): protocol: write tcp 159.223.143.39:35392->45.91.116.13:4154: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jun 26 06:08:51.073904716 (68h57m18.000199391s) [noise=true , address=77.90.31.13:4626 , piece_id=MXYBIEDFAFK53CVTE2C6CKEZWOL3MLW4S7LOHK4IHW6QEIZJL76Q, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: write tcp 159.223.143.39:58864->77.90.31.13:4626: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jun 26 06:08:51.073952402 (68h57m18.000247082s) [noise=true , address=77.90.31.10:4517 , piece_id=VJZ5T4QNVAPNUHNFNK24S6TAS4V4NKNFMQE2FZMCWWLCCB2ELXXQ, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, address:77.90.31.10:4517): protocol: write tcp 159.223.143.39:36382->77.90.31.10:4517: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jun 26 06:08:51.074022481 (68h57m18.000317156s) [noise=true , address=77.90.31.15:4697 , piece_id=CUXGVL7PS3H7ZQKUCSYXJCBR3ETYFDPZFYIEAWEYL7C2S3RJY3AA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5, address:77.90.31.15:4697): protocol: write tcp 159.223.143.39:52730->77.90.31.15:4697: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jun 26 06:08:51.074071824 (68h57m18.000366503s) [noise=true , address=77.90.31.11:4554 , piece_id=R2CIFLWS4LXTGSDSIMIRC4UYVJTXC7T35HPWZ2S7VYUTJ2AMY3BQ, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, address:77.90.31.11:4554): protocol: rpc: dial tcp 77.90.31.11:4554: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jun 26 06:08:51.074102177 (68h57m18.000396855s) [noise=true , address=77.90.31.14:4661 , piece_id=VXI2ZUSGR4N3H7TPIK75RIDFP3SBAOWVRGHNXTZOHFT5FNG4JWJA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue, address:77.90.31.14:4661): protocol: rpc: dial tcp 77.90.31.14:4661: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jun 26 06:08:51.074119276 (68h57m18.000413966s) [seg_pos={0 0}, upload=55354]: Done waiting for segment. successful: 7 optimal: 13 errs: <nil>
Jun 26 06:08:51.100079867 (68h57m18.026374541s) [upload=55354]: Done uploading segments...
Jun 26 06:08:51.100114505 (68h57m18.026409191s) [upload=55354]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 7)
Duration: 701.140498ms
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [.......................................................XX...] duration:
1.181988907s
FAIL (2025-06-26 06:10:01 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 1.182010588s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [........................................................XX..] duration:
1.489905792s
FAIL (2025-06-26 06:09:35 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 1.489994649s
edge
gateway
us-select-1
|
checkers.AuthServiceHTTP{"https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us1)antiblip:
1 out of 60 is failed (threshold=3) [................................X...........................] duration:
253.67907ms
FAIL (2025-06-26 06:09:29 UTC)
orbiter warning: unexpected status code "500 Internal Server Error" "\n"
Duration: 253.709565ms
edge
authservice
http
bgp
us1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [..............................X.............................] duration:
636.986869ms
FAIL (2025-06-26 06:09:26 UTC)
orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded
Duration: 637.037378ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
3 out of 60 is failed (threshold=3) [........................................................XXX.] duration:
10.784777647s
FAIL (2025-06-26 06:09:24 UTC)
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 7)
======LOGS======
pload=55354]: Got next segment. Inline: false
Jun 26 06:08:21.105827735 (68h56m48.032122411s) [upload=55354]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jun 26 06:08:21.105833588 (68h56m48.032128274s) [upload=55354]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jun 26 06:08:21.135318313 (68h56m48.061613009s) [seg_pos={0 0}, upload=55354]: Begin upload segment...
Jun 26 06:08:21.135366534 (68h56m48.061661214s) [seg_pos={0 0}, upload=55354]: Done begin upload segment.
Jun 26 06:08:21.135390342 (68h56m48.061685028s) [noise=true , address=45.91.116.9:5261 , piece_id=CZWB3XJVA7B7R7GWFYNW3RCGNVRXQ3E37XDXD4SUXTEMQFZ73HHA, seg_pos={0 0}, upload=55354]: Uploading piece...
Jun 26 06:08:21.135405801 (68h56m48.061700497s) [noise=true , address=104.36.82.6:4873 , piece_id=TIIMEQUSDLRFF3OK3WBB74F7SWJ5MEC4Z6IQNELDMRZLNTMQT7LA, seg_pos={0 0}, upload=55354]: Uploading piece...
Jun 26 06:08:21.135774729 (68h56m48.062069418s) [noise=true , address=77.90.31.13:4626 , piece_id=MXYBIEDFAFK53CVTE2C6CKEZWOL3MLW4S7LOHK4IHW6QEIZJL76Q, seg_pos={0 0}, upload=55354]: Uploading piece...
Jun 26 06:08:21.135944436 (68h56m48.06223913s) [noise=true , address=77.90.31.10:4517 , piece_id=VJZ5T4QNVAPNUHNFNK24S6TAS4V4NKNFMQE2FZMCWWLCCB2ELXXQ, seg_pos={0 0}, upload=55354]: Uploading piece...
Jun 26 06:08:21.136080402 (68h56m48.062375094s) [noise=true , address=77.90.31.15:4697 , piece_id=CUXGVL7PS3H7ZQKUCSYXJCBR3ETYFDPZFYIEAWEYL7C2S3RJY3AA, seg_pos={0 0}, upload=55354]: Uploading piece...
Jun 26 06:08:21.136382603 (68h56m48.062677289s) [noise=true , address=104.36.82.13:4186 , piece_id=5SVABN3KS6AUL6HILHNQCVF74BA75VPKTTAK7IQAPKRRME3MOZEA, seg_pos={0 0}, upload=55354]: Uploading piece...
Jun 26 06:08:21.136399304 (68h56m48.062693995s) [noise=true , address=104.36.82.14:4223 , piece_id=LP74YFI2SRSYPYEIKREVINCOFMVCXLNDS2555YKUVFN63N6TBJDA, seg_pos={0 0}, upload=55354]: Uploading piece...
Jun 26 06:08:21.136669576 (68h56m48.062964262s) [noise=true , address=104.36.82.6:4876 , piece_id=SSMGONONY34EHF5W6QMKO7YXB7RLO5LZVTJMMFKTP6GQNGYFYQ4A, seg_pos={0 0}, upload=55354]: Uploading piece...
Jun 26 06:08:21.136809784 (68h56m48.063104477s) [noise=true , address=104.36.82.15:4258 , piece_id=YXS4J7YSG6JV33JG4BZLK5AXKE3JXLBTGZJXPXZQ4LBYBT3MU2AA, seg_pos={0 0}, upload=55354]: Uploading piece...
Jun 26 06:08:21.136950556 (68h56m48.063245242s) [noise=true , address=77.90.31.11:4554 , piece_id=R2CIFLWS4LXTGSDSIMIRC4UYVJTXC7T35HPWZ2S7VYUTJ2AMY3BQ, seg_pos={0 0}, upload=55354]: Uploading piece...
Jun 26 06:08:21.137226497 (68h56m48.06352119s) [noise=true , address=77.90.31.14:4661 , piece_id=VXI2ZUSGR4N3H7TPIK75RIDFP3SBAOWVRGHNXTZOHFT5FNG4JWJA, seg_pos={0 0}, upload=55354]: Uploading piece...
Jun 26 06:08:21.138099107 (68h56m48.064393792s) [noise=true , address=45.91.116.10:4082 , piece_id=7NVEGA4QQOE7KYPHXUUX42SJPQL6LP46RGM37VD6A2AWN7PAOM3Q, seg_pos={0 0}, upload=55354]: Uploading piece...
Jun 26 06:08:21.139330448 (68h56m48.065625141s) [noise=true , address=104.36.82.14:4222 , piece_id=U3MBMAKFBFTOPJZ3HS3FXLPDBLQTN577XNCZLUBHRO6IU4BE5VNA, seg_pos={0 0}, upload=55354]: Uploading piece...
Jun 26 06:08:21.140580734 (68h56m48.06687543s) [noise=true , address=77.90.31.19:4842 , piece_id=G3AG375DATHB7MGEEWONE5GMMIC7QHG2VVDKF5AEVOSXV5N4S6AA, seg_pos={0 0}, upload=55354]: Uploading piece...
Jun 26 06:08:21.141452300 (68h56m48.067747027s) [noise=true , address=45.91.116.13:4154 , piece_id=STFPXN2LUJYUIAYROWZIRTCMT2Y3XET6KWA4GZWUHENXRJURP62Q, seg_pos={0 0}, upload=55354]: Uploading piece...
Jun 26 06:08:21.142917062 (68h56m48.069211739s) [noise=true , address=104.36.82.7:4912 , piece_id=A5HPC7CEU66IFNS47HO3AQX22LUZ6AZ4L4XAMZ6T5NCVE4D3TAZA, seg_pos={0 0}, upload=55354]: Uploading piece...
Jun 26 06:08:21.221824189 (68h56m48.148118865s) [upload=55354]: Next returned nil segment
Jun 26 06:08:21.221837958 (68h56m48.148132644s) [upload=55354]: Waiting for error group managing segments...
Jun 26 06:08:21.229550628 (68h56m48.155845315s) [noise=true , address=104.36.82.14:4223 , piece_id=LP74YFI2SRSYPYEIKREVINCOFMVCXLNDS2555YKUVFN63N6TBJDA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: <nil>
Jun 26 06:08:21.230245145 (68h56m48.156539832s) [noise=true , address=104.36.82.14:4222 , piece_id=U3MBMAKFBFTOPJZ3HS3FXLPDBLQTN577XNCZLUBHRO6IU4BE5VNA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: <nil>
Jun 26 06:08:21.230801067 (68h56m48.15709574s) [noise=true , address=104.36.82.6:4873 , piece_id=TIIMEQUSDLRFF3OK3WBB74F7SWJ5MEC4Z6IQNELDMRZLNTMQT7LA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: <nil>
Jun 26 06:08:21.231457382 (68h56m48.157752071s) [noise=true , address=104.36.82.6:4876 , piece_id=SSMGONONY34EHF5W6QMKO7YXB7RLO5LZVTJMMFKTP6GQNGYFYQ4A, seg_pos={0 0}, upload=55354]: Done uploading piece. err: <nil>
Jun 26 06:08:21.231563205 (68h56m48.157857893s) [noise=true , address=104.36.82.13:4186 , piece_id=5SVABN3KS6AUL6HILHNQCVF74BA75VPKTTAK7IQAPKRRME3MOZEA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: <nil>
Jun 26 06:08:21.239131119 (68h56m48.165425799s) [noise=true , address=104.36.82.7:4912 , piece_id=A5HPC7CEU66IFNS47HO3AQX22LUZ6AZ4L4XAMZ6T5NCVE4D3TAZA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: <nil>
Jun 26 06:08:21.534686665 (68h56m48.460981337s) [noise=true , address=104.36.82.15:4258 , piece_id=YXS4J7YSG6JV33JG4BZLK5AXKE3JXLBTGZJXPXZQ4LBYBT3MU2AA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: <nil>
Jun 26 06:08:51.073673533 (68h57m17.999968219s) [noise=true , address=45.91.116.9:5261 , piece_id=CZWB3XJVA7B7R7GWFYNW3RCGNVRXQ3E37XDXD4SUXTEMQFZ73HHA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi, address:45.91.116.9:5261): protocol: rpc: dial tcp 45.91.116.9:5261: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jun 26 06:08:51.073739594 (68h57m18.000034272s) [noise=true , address=45.91.116.10:4082 , piece_id=7NVEGA4QQOE7KYPHXUUX42SJPQL6LP46RGM37VD6A2AWN7PAOM3Q, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj, address:45.91.116.10:4082): protocol: rpc: dial tcp 45.91.116.10:4082: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jun 26 06:08:51.073789897 (68h57m18.000084584s) [noise=true , address=77.90.31.19:4842 , piece_id=G3AG375DATHB7MGEEWONE5GMMIC7QHG2VVDKF5AEVOSXV5N4S6AA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: rpc: dial tcp 77.90.31.19:4842: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jun 26 06:08:51.073849570 (68h57m18.000144246s) [noise=true , address=45.91.116.13:4154 , piece_id=STFPXN2LUJYUIAYROWZIRTCMT2Y3XET6KWA4GZWUHENXRJURP62Q, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN, address:45.91.116.13:4154): protocol: write tcp 159.223.143.39:35392->45.91.116.13:4154: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jun 26 06:08:51.073904716 (68h57m18.000199391s) [noise=true , address=77.90.31.13:4626 , piece_id=MXYBIEDFAFK53CVTE2C6CKEZWOL3MLW4S7LOHK4IHW6QEIZJL76Q, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: write tcp 159.223.143.39:58864->77.90.31.13:4626: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jun 26 06:08:51.073952402 (68h57m18.000247082s) [noise=true , address=77.90.31.10:4517 , piece_id=VJZ5T4QNVAPNUHNFNK24S6TAS4V4NKNFMQE2FZMCWWLCCB2ELXXQ, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, address:77.90.31.10:4517): protocol: write tcp 159.223.143.39:36382->77.90.31.10:4517: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jun 26 06:08:51.074022481 (68h57m18.000317156s) [noise=true , address=77.90.31.15:4697 , piece_id=CUXGVL7PS3H7ZQKUCSYXJCBR3ETYFDPZFYIEAWEYL7C2S3RJY3AA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5, address:77.90.31.15:4697): protocol: write tcp 159.223.143.39:52730->77.90.31.15:4697: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jun 26 06:08:51.074071824 (68h57m18.000366503s) [noise=true , address=77.90.31.11:4554 , piece_id=R2CIFLWS4LXTGSDSIMIRC4UYVJTXC7T35HPWZ2S7VYUTJ2AMY3BQ, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, address:77.90.31.11:4554): protocol: rpc: dial tcp 77.90.31.11:4554: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jun 26 06:08:51.074102177 (68h57m18.000396855s) [noise=true , address=77.90.31.14:4661 , piece_id=VXI2ZUSGR4N3H7TPIK75RIDFP3SBAOWVRGHNXTZOHFT5FNG4JWJA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue, address:77.90.31.14:4661): protocol: rpc: dial tcp 77.90.31.14:4661: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jun 26 06:08:51.074119276 (68h57m18.000413966s) [seg_pos={0 0}, upload=55354]: Done waiting for segment. successful: 7 optimal: 13 errs: <nil>
Jun 26 06:08:51.100079867 (68h57m18.026374541s) [upload=55354]: Done uploading segments...
Jun 26 06:08:51.100114505 (68h57m18.026409191s) [upload=55354]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 7)
Duration: 10.784806315s
core
uplink
us-select-1
|
checkers.AuthServiceHTTP{"https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us1)antiblip:
1 out of 60 is failed (threshold=3) [.................................X..........................] duration:
182.911275ms
FAIL (2025-06-26 06:09:03 UTC)
orbiter warning: unexpected status code "500 Internal Server Error" "\n"
Duration: 182.950369ms
edge
authservice
http
bgp
us1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [...............................X............................] duration:
1.081529533s
FAIL (2025-06-26 06:09:02 UTC)
orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded
Duration: 1.081562472s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
3 out of 60 is failed (threshold=3) [.........................................................XXX] duration:
30.02860262s
FAIL (2025-06-26 06:08:51 UTC)
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 7)
======LOGS======
pload=55354]: Got next segment. Inline: false
Jun 26 06:08:21.105827735 (68h56m48.032122411s) [upload=55354]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jun 26 06:08:21.105833588 (68h56m48.032128274s) [upload=55354]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jun 26 06:08:21.135318313 (68h56m48.061613009s) [seg_pos={0 0}, upload=55354]: Begin upload segment...
Jun 26 06:08:21.135366534 (68h56m48.061661214s) [seg_pos={0 0}, upload=55354]: Done begin upload segment.
Jun 26 06:08:21.135390342 (68h56m48.061685028s) [noise=true , address=45.91.116.9:5261 , piece_id=CZWB3XJVA7B7R7GWFYNW3RCGNVRXQ3E37XDXD4SUXTEMQFZ73HHA, seg_pos={0 0}, upload=55354]: Uploading piece...
Jun 26 06:08:21.135405801 (68h56m48.061700497s) [noise=true , address=104.36.82.6:4873 , piece_id=TIIMEQUSDLRFF3OK3WBB74F7SWJ5MEC4Z6IQNELDMRZLNTMQT7LA, seg_pos={0 0}, upload=55354]: Uploading piece...
Jun 26 06:08:21.135774729 (68h56m48.062069418s) [noise=true , address=77.90.31.13:4626 , piece_id=MXYBIEDFAFK53CVTE2C6CKEZWOL3MLW4S7LOHK4IHW6QEIZJL76Q, seg_pos={0 0}, upload=55354]: Uploading piece...
Jun 26 06:08:21.135944436 (68h56m48.06223913s) [noise=true , address=77.90.31.10:4517 , piece_id=VJZ5T4QNVAPNUHNFNK24S6TAS4V4NKNFMQE2FZMCWWLCCB2ELXXQ, seg_pos={0 0}, upload=55354]: Uploading piece...
Jun 26 06:08:21.136080402 (68h56m48.062375094s) [noise=true , address=77.90.31.15:4697 , piece_id=CUXGVL7PS3H7ZQKUCSYXJCBR3ETYFDPZFYIEAWEYL7C2S3RJY3AA, seg_pos={0 0}, upload=55354]: Uploading piece...
Jun 26 06:08:21.136382603 (68h56m48.062677289s) [noise=true , address=104.36.82.13:4186 , piece_id=5SVABN3KS6AUL6HILHNQCVF74BA75VPKTTAK7IQAPKRRME3MOZEA, seg_pos={0 0}, upload=55354]: Uploading piece...
Jun 26 06:08:21.136399304 (68h56m48.062693995s) [noise=true , address=104.36.82.14:4223 , piece_id=LP74YFI2SRSYPYEIKREVINCOFMVCXLNDS2555YKUVFN63N6TBJDA, seg_pos={0 0}, upload=55354]: Uploading piece...
Jun 26 06:08:21.136669576 (68h56m48.062964262s) [noise=true , address=104.36.82.6:4876 , piece_id=SSMGONONY34EHF5W6QMKO7YXB7RLO5LZVTJMMFKTP6GQNGYFYQ4A, seg_pos={0 0}, upload=55354]: Uploading piece...
Jun 26 06:08:21.136809784 (68h56m48.063104477s) [noise=true , address=104.36.82.15:4258 , piece_id=YXS4J7YSG6JV33JG4BZLK5AXKE3JXLBTGZJXPXZQ4LBYBT3MU2AA, seg_pos={0 0}, upload=55354]: Uploading piece...
Jun 26 06:08:21.136950556 (68h56m48.063245242s) [noise=true , address=77.90.31.11:4554 , piece_id=R2CIFLWS4LXTGSDSIMIRC4UYVJTXC7T35HPWZ2S7VYUTJ2AMY3BQ, seg_pos={0 0}, upload=55354]: Uploading piece...
Jun 26 06:08:21.137226497 (68h56m48.06352119s) [noise=true , address=77.90.31.14:4661 , piece_id=VXI2ZUSGR4N3H7TPIK75RIDFP3SBAOWVRGHNXTZOHFT5FNG4JWJA, seg_pos={0 0}, upload=55354]: Uploading piece...
Jun 26 06:08:21.138099107 (68h56m48.064393792s) [noise=true , address=45.91.116.10:4082 , piece_id=7NVEGA4QQOE7KYPHXUUX42SJPQL6LP46RGM37VD6A2AWN7PAOM3Q, seg_pos={0 0}, upload=55354]: Uploading piece...
Jun 26 06:08:21.139330448 (68h56m48.065625141s) [noise=true , address=104.36.82.14:4222 , piece_id=U3MBMAKFBFTOPJZ3HS3FXLPDBLQTN577XNCZLUBHRO6IU4BE5VNA, seg_pos={0 0}, upload=55354]: Uploading piece...
Jun 26 06:08:21.140580734 (68h56m48.06687543s) [noise=true , address=77.90.31.19:4842 , piece_id=G3AG375DATHB7MGEEWONE5GMMIC7QHG2VVDKF5AEVOSXV5N4S6AA, seg_pos={0 0}, upload=55354]: Uploading piece...
Jun 26 06:08:21.141452300 (68h56m48.067747027s) [noise=true , address=45.91.116.13:4154 , piece_id=STFPXN2LUJYUIAYROWZIRTCMT2Y3XET6KWA4GZWUHENXRJURP62Q, seg_pos={0 0}, upload=55354]: Uploading piece...
Jun 26 06:08:21.142917062 (68h56m48.069211739s) [noise=true , address=104.36.82.7:4912 , piece_id=A5HPC7CEU66IFNS47HO3AQX22LUZ6AZ4L4XAMZ6T5NCVE4D3TAZA, seg_pos={0 0}, upload=55354]: Uploading piece...
Jun 26 06:08:21.221824189 (68h56m48.148118865s) [upload=55354]: Next returned nil segment
Jun 26 06:08:21.221837958 (68h56m48.148132644s) [upload=55354]: Waiting for error group managing segments...
Jun 26 06:08:21.229550628 (68h56m48.155845315s) [noise=true , address=104.36.82.14:4223 , piece_id=LP74YFI2SRSYPYEIKREVINCOFMVCXLNDS2555YKUVFN63N6TBJDA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: <nil>
Jun 26 06:08:21.230245145 (68h56m48.156539832s) [noise=true , address=104.36.82.14:4222 , piece_id=U3MBMAKFBFTOPJZ3HS3FXLPDBLQTN577XNCZLUBHRO6IU4BE5VNA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: <nil>
Jun 26 06:08:21.230801067 (68h56m48.15709574s) [noise=true , address=104.36.82.6:4873 , piece_id=TIIMEQUSDLRFF3OK3WBB74F7SWJ5MEC4Z6IQNELDMRZLNTMQT7LA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: <nil>
Jun 26 06:08:21.231457382 (68h56m48.157752071s) [noise=true , address=104.36.82.6:4876 , piece_id=SSMGONONY34EHF5W6QMKO7YXB7RLO5LZVTJMMFKTP6GQNGYFYQ4A, seg_pos={0 0}, upload=55354]: Done uploading piece. err: <nil>
Jun 26 06:08:21.231563205 (68h56m48.157857893s) [noise=true , address=104.36.82.13:4186 , piece_id=5SVABN3KS6AUL6HILHNQCVF74BA75VPKTTAK7IQAPKRRME3MOZEA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: <nil>
Jun 26 06:08:21.239131119 (68h56m48.165425799s) [noise=true , address=104.36.82.7:4912 , piece_id=A5HPC7CEU66IFNS47HO3AQX22LUZ6AZ4L4XAMZ6T5NCVE4D3TAZA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: <nil>
Jun 26 06:08:21.534686665 (68h56m48.460981337s) [noise=true , address=104.36.82.15:4258 , piece_id=YXS4J7YSG6JV33JG4BZLK5AXKE3JXLBTGZJXPXZQ4LBYBT3MU2AA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: <nil>
Jun 26 06:08:51.073673533 (68h57m17.999968219s) [noise=true , address=45.91.116.9:5261 , piece_id=CZWB3XJVA7B7R7GWFYNW3RCGNVRXQ3E37XDXD4SUXTEMQFZ73HHA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi, address:45.91.116.9:5261): protocol: rpc: dial tcp 45.91.116.9:5261: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jun 26 06:08:51.073739594 (68h57m18.000034272s) [noise=true , address=45.91.116.10:4082 , piece_id=7NVEGA4QQOE7KYPHXUUX42SJPQL6LP46RGM37VD6A2AWN7PAOM3Q, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj, address:45.91.116.10:4082): protocol: rpc: dial tcp 45.91.116.10:4082: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jun 26 06:08:51.073789897 (68h57m18.000084584s) [noise=true , address=77.90.31.19:4842 , piece_id=G3AG375DATHB7MGEEWONE5GMMIC7QHG2VVDKF5AEVOSXV5N4S6AA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: rpc: dial tcp 77.90.31.19:4842: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jun 26 06:08:51.073849570 (68h57m18.000144246s) [noise=true , address=45.91.116.13:4154 , piece_id=STFPXN2LUJYUIAYROWZIRTCMT2Y3XET6KWA4GZWUHENXRJURP62Q, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN, address:45.91.116.13:4154): protocol: write tcp 159.223.143.39:35392->45.91.116.13:4154: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jun 26 06:08:51.073904716 (68h57m18.000199391s) [noise=true , address=77.90.31.13:4626 , piece_id=MXYBIEDFAFK53CVTE2C6CKEZWOL3MLW4S7LOHK4IHW6QEIZJL76Q, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: write tcp 159.223.143.39:58864->77.90.31.13:4626: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jun 26 06:08:51.073952402 (68h57m18.000247082s) [noise=true , address=77.90.31.10:4517 , piece_id=VJZ5T4QNVAPNUHNFNK24S6TAS4V4NKNFMQE2FZMCWWLCCB2ELXXQ, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, address:77.90.31.10:4517): protocol: write tcp 159.223.143.39:36382->77.90.31.10:4517: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jun 26 06:08:51.074022481 (68h57m18.000317156s) [noise=true , address=77.90.31.15:4697 , piece_id=CUXGVL7PS3H7ZQKUCSYXJCBR3ETYFDPZFYIEAWEYL7C2S3RJY3AA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5, address:77.90.31.15:4697): protocol: write tcp 159.223.143.39:52730->77.90.31.15:4697: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jun 26 06:08:51.074071824 (68h57m18.000366503s) [noise=true , address=77.90.31.11:4554 , piece_id=R2CIFLWS4LXTGSDSIMIRC4UYVJTXC7T35HPWZ2S7VYUTJ2AMY3BQ, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, address:77.90.31.11:4554): protocol: rpc: dial tcp 77.90.31.11:4554: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jun 26 06:08:51.074102177 (68h57m18.000396855s) [noise=true , address=77.90.31.14:4661 , piece_id=VXI2ZUSGR4N3H7TPIK75RIDFP3SBAOWVRGHNXTZOHFT5FNG4JWJA, seg_pos={0 0}, upload=55354]: Done uploading piece. err: ecclient: upload failed (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue, address:77.90.31.14:4661): protocol: rpc: dial tcp 77.90.31.14:4661: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jun 26 06:08:51.074119276 (68h57m18.000413966s) [seg_pos={0 0}, upload=55354]: Done waiting for segment. successful: 7 optimal: 13 errs: <nil>
Jun 26 06:08:51.100079867 (68h57m18.026374541s) [upload=55354]: Done uploading segments...
Jun 26 06:08:51.100114505 (68h57m18.026409191s) [upload=55354]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 7)
Duration: 30.028641443s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [.........................................................XX.] duration:
654.547644ms
FAIL (2025-06-26 06:08:39 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 654.572582ms
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [..........................................................XX] duration:
30.000652323s
FAIL (2025-06-26 06:08:20 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 30.000670547s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [..........................................................XX] duration:
30.031098661s
FAIL (2025-06-26 06:08:19 UTC)
orbiter warning:
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 4)
======LOGS======
iece_id=XCDEYSOUIIXRRK3U6KT4LRPLGRSMQ3VOPAF55K3I27JOYL4EDF7A, seg_pos={0 0}, upload=55347]: Uploading piece...
Jun 26 06:07:49.241458801 (68h56m16.167753487s) [noise=true , address=77.90.31.13:4626 , piece_id=C5N6PUVWR6E4NZGOBENZPXTMWMC7ADMUWC6Z2LHMQSWTNRQYJVSQ, seg_pos={0 0}, upload=55347]: Uploading piece...
Jun 26 06:07:49.241800555 (68h56m16.168095248s) [noise=true , address=77.90.31.17:4772 , piece_id=3WWORRAIQ55V6XNAYTJQ2INOXABI3UY4B2FWOH3TG5OSY5UE2XLA, seg_pos={0 0}, upload=55347]: Uploading piece...
Jun 26 06:07:49.241816528 (68h56m16.168111224s) [noise=true , address=45.91.116.6:4974 , piece_id=J7Z364IRCN53JESS5WUVEFMSR2GAJLF2CXSCIELPL4G6YGJ3HWYQ, seg_pos={0 0}, upload=55347]: Uploading piece...
Jun 26 06:07:49.242183501 (68h56m16.168478197s) [noise=true , address=104.36.82.3:4767 , piece_id=FADUUPOJIPRYWDQUBJGEBSOEKRY6MNJ5XVQLRNZPY6FOZWBPOXHQ, seg_pos={0 0}, upload=55347]: Uploading piece...
Jun 26 06:07:49.242364633 (68h56m16.168659325s) [noise=true , address=77.90.31.19:4843 , piece_id=EW7MTAHMW4QQKZ5O6RRKXZWD7HVY5T7I7O5QCTE3U7X5CAIGQXSA, seg_pos={0 0}, upload=55347]: Uploading piece...
Jun 26 06:07:49.242677644 (68h56m16.168972319s) [noise=true , address=45.91.116.13:4155 , piece_id=T5WDVSQDJGK33QHY3YNQU2DCYKKGTMP7SKSMUTINR777SATU7LCA, seg_pos={0 0}, upload=55347]: Uploading piece...
Jun 26 06:07:49.242808205 (68h56m16.169102894s) [noise=true , address=104.36.82.15:4259 , piece_id=DS4QQHVFUVHWTJ4HRU3WLPUH7IHCPYP3XFIQIZZNSBRLGFGC6NSQ, seg_pos={0 0}, upload=55347]: Uploading piece...
Jun 26 06:07:49.243262517 (68h56m16.169557203s) [noise=true , address=77.90.31.16:4734 , piece_id=B3LL363SGYXHBCMHXA5YIF5DKAZ7SDRYHEO6KL3XEJ63MWAHH6VQ, seg_pos={0 0}, upload=55347]: Uploading piece...
Jun 26 06:07:49.243745014 (68h56m16.170039706s) [noise=true , address=77.90.31.21:4878 , piece_id=NSFJJJTQWIF723LXBGVC2JLBW7PIHTDVB7AFELB7IETPHBKOTCEA, seg_pos={0 0}, upload=55347]: Uploading piece...
Jun 26 06:07:49.244186879 (68h56m16.170481568s) [noise=true , address=77.90.31.10:4520 , piece_id=RQCQK4L4LQJYXZF43K6Q5YHZEPSXBZ3CVBHMDPAG2DDJWVYMDFTA, seg_pos={0 0}, upload=55347]: Uploading piece...
Jun 26 06:07:49.244565443 (68h56m16.170860135s) [noise=true , address=45.91.116.10:4082 , piece_id=7S5UAYTUZNEJ22RXBTBICPYQQ6L3NRIMRBYDED4NQWULQGYYMYKQ, seg_pos={0 0}, upload=55347]: Uploading piece...
Jun 26 06:07:49.244973355 (68h56m16.171268044s) [noise=true , address=77.90.31.12:4592 , piece_id=V5YHUH5PX6EMNNE7BVQ3ULR2ADKMAXYHWCVJUIUK2P2BTW5MKVHA, seg_pos={0 0}, upload=55347]: Uploading piece...
Jun 26 06:07:49.245405608 (68h56m16.171700299s) [noise=true , address=104.36.82.12:4151 , piece_id=ONVL476GHECF6YM5FNA56YJPKUXPB4TEAUGD7A6QDY7I4Y2BSFIQ, seg_pos={0 0}, upload=55347]: Uploading piece...
Jun 26 06:07:49.248196877 (68h56m16.174491556s) [noise=true , address=77.90.31.18:4807 , piece_id=B43MHNVOAKQNY4RFN5REU5USIM7QBDB74AV423TWNDGML762L3ZA, seg_pos={0 0}, upload=55347]: Uploading piece...
Jun 26 06:07:49.332800764 (68h56m16.259095451s) [upload=55347]: Next returned nil segment
Jun 26 06:07:49.332815258 (68h56m16.259109944s) [upload=55347]: Waiting for error group managing segments...
Jun 26 06:07:49.339858293 (68h56m16.266152972s) [noise=true , address=104.36.82.15:4259 , piece_id=DS4QQHVFUVHWTJ4HRU3WLPUH7IHCPYP3XFIQIZZNSBRLGFGC6NSQ, seg_pos={0 0}, upload=55347]: Done uploading piece. err: <nil>
Jun 26 06:07:49.341015012 (68h56m16.26730969s) [noise=true , address=104.36.82.11:4113 , piece_id=XCDEYSOUIIXRRK3U6KT4LRPLGRSMQ3VOPAF55K3I27JOYL4EDF7A, seg_pos={0 0}, upload=55347]: Done uploading piece. err: <nil>
Jun 26 06:07:49.341626983 (68h56m16.267921662s) [noise=true , address=104.36.82.12:4151 , piece_id=ONVL476GHECF6YM5FNA56YJPKUXPB4TEAUGD7A6QDY7I4Y2BSFIQ, seg_pos={0 0}, upload=55347]: Done uploading piece. err: <nil>
Jun 26 06:07:49.342115651 (68h56m16.268410334s) [noise=true , address=104.36.82.3:4767 , piece_id=FADUUPOJIPRYWDQUBJGEBSOEKRY6MNJ5XVQLRNZPY6FOZWBPOXHQ, seg_pos={0 0}, upload=55347]: Done uploading piece. err: <nil>
Jun 26 06:08:19.135259338 (68h56m46.061554015s) [noise=true , address=77.90.31.12:4592 , piece_id=V5YHUH5PX6EMNNE7BVQ3ULR2ADKMAXYHWCVJUIUK2P2BTW5MKVHA, seg_pos={0 0}, upload=55347]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: rpc: dial tcp 77.90.31.12:4592: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jun 26 06:08:19.135337684 (68h56m46.061632359s) [noise=true , address=77.90.31.11:4554 , piece_id=3557RWJQ45UKR6CTRNLT2AV2T4RTSQCJANHUYBSOTQEPYGOVEKPQ, seg_pos={0 0}, upload=55347]: Done uploading piece. err: ecclient: upload failed (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, address:77.90.31.11:4554): protocol: rpc: dial tcp 77.90.31.11:4554: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jun 26 06:08:19.135413354 (68h56m46.061708031s) [noise=true , address=77.90.31.13:4626 , piece_id=C5N6PUVWR6E4NZGOBENZPXTMWMC7ADMUWC6Z2LHMQSWTNRQYJVSQ, seg_pos={0 0}, upload=55347]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: rpc: dial tcp 77.90.31.13:4626: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jun 26 06:08:19.135479186 (68h56m46.061773866s) [noise=true , address=77.90.31.19:4843 , piece_id=EW7MTAHMW4QQKZ5O6RRKXZWD7HVY5T7I7O5QCTE3U7X5CAIGQXSA, seg_pos={0 0}, upload=55347]: Done uploading piece. err: ecclient: upload failed (node:12akCPrMTihqq6CKv7VtDbHhXnWrXygXXhZKgZbftGbymRNzuMb, address:77.90.31.19:4843): protocol: rpc: dial tcp 77.90.31.19:4843: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jun 26 06:08:19.135553159 (68h56m46.061847836s) [noise=true , address=77.90.31.16:4734 , piece_id=B3LL363SGYXHBCMHXA5YIF5DKAZ7SDRYHEO6KL3XEJ63MWAHH6VQ, seg_pos={0 0}, upload=55347]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: write tcp 159.223.143.39:35196->77.90.31.16:4734: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jun 26 06:08:19.135606860 (68h56m46.061901536s) [noise=true , address=77.90.31.10:4520 , piece_id=RQCQK4L4LQJYXZF43K6Q5YHZEPSXBZ3CVBHMDPAG2DDJWVYMDFTA, seg_pos={0 0}, upload=55347]: Done uploading piece. err: ecclient: upload failed (node:1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp, address:77.90.31.10:4520): protocol: rpc: dial tcp 77.90.31.10:4520: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jun 26 06:08:19.135655149 (68h56m46.061949829s) [noise=true , address=77.90.31.21:4878 , piece_id=NSFJJJTQWIF723LXBGVC2JLBW7PIHTDVB7AFELB7IETPHBKOTCEA, seg_pos={0 0}, upload=55347]: Done uploading piece. err: ecclient: upload failed (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK, address:77.90.31.21:4878): protocol: write tcp 159.223.143.39:37892->77.90.31.21:4878: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jun 26 06:08:19.135686878 (68h56m46.061981568s) [noise=true , address=77.90.31.17:4772 , piece_id=3WWORRAIQ55V6XNAYTJQ2INOXABI3UY4B2FWOH3TG5OSY5UE2XLA, seg_pos={0 0}, upload=55347]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: write tcp 159.223.143.39:58138->77.90.31.17:4772: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jun 26 06:08:19.135720102 (68h56m46.062014778s) [noise=true , address=45.91.116.10:4082 , piece_id=7S5UAYTUZNEJ22RXBTBICPYQQ6L3NRIMRBYDED4NQWULQGYYMYKQ, seg_pos={0 0}, upload=55347]: Done uploading piece. err: ecclient: upload failed (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj, address:45.91.116.10:4082): protocol: write tcp 159.223.143.39:47532->45.91.116.10:4082: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jun 26 06:08:19.135757433 (68h56m46.062052106s) [noise=true , address=45.91.116.6:4974 , piece_id=J7Z364IRCN53JESS5WUVEFMSR2GAJLF2CXSCIELPL4G6YGJ3HWYQ, seg_pos={0 0}, upload=55347]: Done uploading piece. err: ecclient: upload failed (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ, address:45.91.116.6:4974): protocol: write tcp 159.223.143.39:45450->45.91.116.6:4974: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jun 26 06:08:19.135779990 (68h56m46.062074669s) [noise=true , address=77.90.31.18:4807 , piece_id=B43MHNVOAKQNY4RFN5REU5USIM7QBDB74AV423TWNDGML762L3ZA, seg_pos={0 0}, upload=55347]: Done uploading piece. err: ecclient: upload failed (node:12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS, address:77.90.31.18:4807): protocol: write tcp 159.223.143.39:53946->77.90.31.18:4807: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jun 26 06:08:19.135796777 (68h56m46.062091455s) [noise=true , address=45.91.116.13:4155 , piece_id=T5WDVSQDJGK33QHY3YNQU2DCYKKGTMP7SKSMUTINR777SATU7LCA, seg_pos={0 0}, upload=55347]: Done uploading piece. err: ecclient: upload failed (node:12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf, address:45.91.116.13:4155): protocol: write tcp 159.223.143.39:34020->45.91.116.13:4155: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jun 26 06:08:19.135837560 (68h56m46.062132242s) [seg_pos={0 0}, upload=55347]: Done waiting for segment. successful: 4 optimal: 13 errs: <nil>
Jun 26 06:08:19.164364757 (68h56m46.09065944s) [upload=55347]: Done uploading segments...
Jun 26 06:08:19.164412307 (68h56m46.090706994s) [upload=55347]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 4)
Duration: 30.031112601s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [................................X...........................] duration:
630.188399ms
FAIL (2025-06-26 06:07:58 UTC)
orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded
Duration: 630.220189ms
core
uplink
us-select-1
us1-placement-20
|
checkers.AuthServiceHTTP{"https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us1)antiblip:
1 out of 60 is failed (threshold=3) [..................................X.........................] duration:
177.111199ms
FAIL (2025-06-26 06:07:53 UTC)
orbiter warning: unexpected status code "500 Internal Server Error" "\n"
Duration: 177.136811ms
edge
authservice
http
bgp
us1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [...........................................................X] duration:
30.037774132s
FAIL (2025-06-26 06:07:49 UTC)
orbiter warning:
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 11)
======LOGS======
PA, seg_pos={0 0}, upload=55325]: Done uploading piece. err: <nil>
Jun 26 06:06:18.077213217 (68h54m45.003507897s) [seg_pos={0 0}, upload=55325]: Done waiting for segment. successful: 14 optimal: 13 errs: <nil>
Jun 26 06:06:18.077290751 (68h54m45.003585441s) [upload=55325]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jun 26 06:06:18.077295752 (68h54m45.003590445s) [upload=55325]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jun 26 06:06:18.143317897 (68h54m45.069612578s) [upload=55325]: Done uploading segments...
Jun 26 06:06:18.143336818 (68h54m45.069631495s) [upload=55325]: Upload finished. err: <nil>
Jun 26 06:07:19.035622859 (68h55m45.961917554s) [upload=55340]: Starting upload
Jun 26 06:07:19.035739692 (68h55m45.962034383s) [upload=55340]: Done starting upload
Jun 26 06:07:19.035842835 (68h55m45.962137577s) [upload=55340]: Uploading segments...
Jun 26 06:07:19.036194093 (68h55m45.962488784s) [upload=55340]: Got next segment. Inline: false
Jun 26 06:07:19.036269272 (68h55m45.962563951s) [upload=55340]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jun 26 06:07:19.036276108 (68h55m45.962570804s) [upload=55340]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jun 26 06:07:19.068289708 (68h55m45.994584413s) [seg_pos={0 0}, upload=55340]: Begin upload segment...
Jun 26 06:07:19.068360168 (68h55m45.994654864s) [seg_pos={0 0}, upload=55340]: Done begin upload segment.
Jun 26 06:07:19.068410263 (68h55m45.994704959s) [noise=true , address=104.36.82.2:4729 , piece_id=HAQYG2XGC2KUIJIFTU63IZGM66P6RPU5VIBIOD4VEFFIFWDCI3UA, seg_pos={0 0}, upload=55340]: Uploading piece...
Jun 26 06:07:19.068887226 (68h55m45.995181919s) [noise=true , address=104.36.82.15:4257 , piece_id=FZS4DV3UN5SFXZKMPZWJP5LMB363JFHRQPQLWZRDLCPWNFOKOSJQ, seg_pos={0 0}, upload=55340]: Uploading piece...
Jun 26 06:07:19.068956232 (68h55m45.995250929s) [noise=true , address=104.36.82.10:4077 , piece_id=MW5ISEZ5GOUOYBYIBMZO2VA4BKDBAX3TRJHQB3M6NMOMDZAMD5CA, seg_pos={0 0}, upload=55340]: Uploading piece...
Jun 26 06:07:19.069285875 (68h55m45.995580569s) [noise=true , address=104.36.82.12:4150 , piece_id=OQTRN3INEZJCXTHGMNYI6DISEAN44ZWNURC3QPET7YQ35VV3QITQ, seg_pos={0 0}, upload=55340]: Uploading piece...
Jun 26 06:07:19.069495525 (68h55m45.995790219s) [noise=true , address=104.36.82.6:4873 , piece_id=KQBAAC6MSJHQJNNLTR23ZT7R3X3TFN3ON7L52LASDPKOKWZH6EMQ, seg_pos={0 0}, upload=55340]: Uploading piece...
Jun 26 06:07:19.069666084 (68h55m45.995960786s) [noise=true , address=104.36.82.11:4113 , piece_id=67TFOQFYFTNZD4U7H23ROPKLJITJLEIBSH27YRHVVT7ZUL3RO2RA, seg_pos={0 0}, upload=55340]: Uploading piece...
Jun 26 06:07:19.069965522 (68h55m45.996260215s) [noise=true , address=77.90.31.15:4698 , piece_id=Z46O3RZZVGRLMCSLMUPICQZU5XAHMGDTF4YAQ6FWTKTAR64GP5GA, seg_pos={0 0}, upload=55340]: Uploading piece...
Jun 26 06:07:19.070065098 (68h55m45.996359794s) [noise=true , address=104.36.82.13:4187 , piece_id=IEUC24VTX65EM2FLHTQSTOZKMJKNGMARGBI7JRK2WREBZIWT5I4Q, seg_pos={0 0}, upload=55340]: Uploading piece...
Jun 26 06:07:19.070504022 (68h55m45.996798709s) [noise=true , address=104.36.82.14:4223 , piece_id=GL6CEWVGCXJBKKZYSJNJC6TYUOMCSRP7UVVF7LZSNCLLYKTJE3VQ, seg_pos={0 0}, upload=55340]: Uploading piece...
Jun 26 06:07:19.070525898 (68h55m45.996820598s) [noise=true , address=45.91.116.5:5298 , piece_id=AVSMEAQ3JGQGU2BBI5HC7PDXVBKJRGC4TNS2AQZQLHXMG2U3ZGFA, seg_pos={0 0}, upload=55340]: Uploading piece...
Jun 26 06:07:19.070936299 (68h55m45.997231103s) [noise=true , address=104.36.82.7:4912 , piece_id=XDPISLDHANLXGIWZMGMF4EHTSOZKAOYKTER6BHXWTNODTOI7ZZEA, seg_pos={0 0}, upload=55340]: Uploading piece...
Jun 26 06:07:19.070951453 (68h55m45.997246146s) [noise=true , address=45.91.116.12:5226 , piece_id=JM3RHVN6KG6YQYP7GTVYAG64GKUMUOO3TY6BKRAMSSONFVTJ5H7A, seg_pos={0 0}, upload=55340]: Uploading piece...
Jun 26 06:07:19.071383223 (68h55m45.997677926s) [noise=true , address=77.90.31.12:4589 , piece_id=GVFFI23232D4XAICJF5Z2JML3EWGACZXP56KE3MKMJWEXFMIZZDA, seg_pos={0 0}, upload=55340]: Uploading piece...
Jun 26 06:07:19.071414399 (68h55m45.997709089s) [noise=true , address=104.36.82.8:4969 , piece_id=CUA4QFNEI42SQOLYQILYIEPNJJ5V3HESAWLBMSDVJOZIH5NQEINA, seg_pos={0 0}, upload=55340]: Uploading piece...
Jun 26 06:07:19.071863449 (68h55m45.998158145s) [noise=true , address=104.36.82.3:4765 , piece_id=JI5YRABHVN5W7OE6C7SN4VMAFVNQWTNFZ5JIDRFKXDP7OR6BW37A, seg_pos={0 0}, upload=55340]: Uploading piece...
Jun 26 06:07:19.072258167 (68h55m45.99855286s) [noise=true , address=77.90.31.16:4735 , piece_id=I6AB2Z6MV4GSNS4Y2QNUVPNRN2TJQW5ZWNTH3LL6E22TDWFTVRFA, seg_pos={0 0}, upload=55340]: Uploading piece...
Jun 26 06:07:19.188330371 (68h55m46.114625094s) [upload=55340]: Next returned nil segment
Jun 26 06:07:19.188363363 (68h55m46.11465806s) [upload=55340]: Waiting for error group managing segments...
Jun 26 06:07:19.195212013 (68h55m46.12150671s) [noise=true , address=104.36.82.10:4077 , piece_id=MW5ISEZ5GOUOYBYIBMZO2VA4BKDBAX3TRJHQB3M6NMOMDZAMD5CA, seg_pos={0 0}, upload=55340]: Done uploading piece. err: <nil>
Jun 26 06:07:19.195811737 (68h55m46.122106411s) [noise=true , address=104.36.82.7:4912 , piece_id=XDPISLDHANLXGIWZMGMF4EHTSOZKAOYKTER6BHXWTNODTOI7ZZEA, seg_pos={0 0}, upload=55340]: Done uploading piece. err: <nil>
Jun 26 06:07:19.195858757 (68h55m46.122153431s) [noise=true , address=104.36.82.3:4765 , piece_id=JI5YRABHVN5W7OE6C7SN4VMAFVNQWTNFZ5JIDRFKXDP7OR6BW37A, seg_pos={0 0}, upload=55340]: Done uploading piece. err: <nil>
Jun 26 06:07:19.196319433 (68h55m46.122614109s) [noise=true , address=104.36.82.8:4969 , piece_id=CUA4QFNEI42SQOLYQILYIEPNJJ5V3HESAWLBMSDVJOZIH5NQEINA, seg_pos={0 0}, upload=55340]: Done uploading piece. err: <nil>
Jun 26 06:07:19.196448853 (68h55m46.122743539s) [noise=true , address=104.36.82.13:4187 , piece_id=IEUC24VTX65EM2FLHTQSTOZKMJKNGMARGBI7JRK2WREBZIWT5I4Q, seg_pos={0 0}, upload=55340]: Done uploading piece. err: <nil>
Jun 26 06:07:19.196873842 (68h55m46.12316853s) [noise=true , address=104.36.82.6:4873 , piece_id=KQBAAC6MSJHQJNNLTR23ZT7R3X3TFN3ON7L52LASDPKOKWZH6EMQ, seg_pos={0 0}, upload=55340]: Done uploading piece. err: <nil>
Jun 26 06:07:19.196935394 (68h55m46.123230067s) [noise=true , address=104.36.82.14:4223 , piece_id=GL6CEWVGCXJBKKZYSJNJC6TYUOMCSRP7UVVF7LZSNCLLYKTJE3VQ, seg_pos={0 0}, upload=55340]: Done uploading piece. err: <nil>
Jun 26 06:07:19.197457053 (68h55m46.123751726s) [noise=true , address=104.36.82.11:4113 , piece_id=67TFOQFYFTNZD4U7H23ROPKLJITJLEIBSH27YRHVVT7ZUL3RO2RA, seg_pos={0 0}, upload=55340]: Done uploading piece. err: <nil>
Jun 26 06:07:19.197510895 (68h55m46.123805585s) [noise=true , address=104.36.82.15:4257 , piece_id=FZS4DV3UN5SFXZKMPZWJP5LMB363JFHRQPQLWZRDLCPWNFOKOSJQ, seg_pos={0 0}, upload=55340]: Done uploading piece. err: <nil>
Jun 26 06:07:19.197925745 (68h55m46.124220417s) [noise=true , address=104.36.82.12:4150 , piece_id=OQTRN3INEZJCXTHGMNYI6DISEAN44ZWNURC3QPET7YQ35VV3QITQ, seg_pos={0 0}, upload=55340]: Done uploading piece. err: <nil>
Jun 26 06:07:19.198099537 (68h55m46.12439422s) [noise=true , address=104.36.82.2:4729 , piece_id=HAQYG2XGC2KUIJIFTU63IZGM66P6RPU5VIBIOD4VEFFIFWDCI3UA, seg_pos={0 0}, upload=55340]: Done uploading piece. err: <nil>
Jun 26 06:07:49.008944931 (68h56m15.935239631s) [noise=true , address=45.91.116.12:5226 , piece_id=JM3RHVN6KG6YQYP7GTVYAG64GKUMUOO3TY6BKRAMSSONFVTJ5H7A, seg_pos={0 0}, upload=55340]: Done uploading piece. err: ecclient: upload failed (node:12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1, address:45.91.116.12:5226): protocol: rpc: dial tcp 45.91.116.12:5226: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jun 26 06:07:49.009193698 (68h56m15.935488394s) [noise=true , address=77.90.31.16:4735 , piece_id=I6AB2Z6MV4GSNS4Y2QNUVPNRN2TJQW5ZWNTH3LL6E22TDWFTVRFA, seg_pos={0 0}, upload=55340]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: rpc: dial tcp 77.90.31.16:4735: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jun 26 06:07:49.009332694 (68h56m15.935627383s) [noise=true , address=45.91.116.5:5298 , piece_id=AVSMEAQ3JGQGU2BBI5HC7PDXVBKJRGC4TNS2AQZQLHXMG2U3ZGFA, seg_pos={0 0}, upload=55340]: Done uploading piece. err: ecclient: upload failed (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C, address:45.91.116.5:5298): protocol: write tcp 159.223.143.39:52494->45.91.116.5:5298: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jun 26 06:07:49.009461287 (68h56m15.93575596s) [noise=true , address=77.90.31.12:4589 , piece_id=GVFFI23232D4XAICJF5Z2JML3EWGACZXP56KE3MKMJWEXFMIZZDA, seg_pos={0 0}, upload=55340]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: write tcp 159.223.143.39:39176->77.90.31.12:4589: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jun 26 06:07:49.009509372 (68h56m15.935804045s) [noise=true , address=77.90.31.15:4698 , piece_id=Z46O3RZZVGRLMCSLMUPICQZU5XAHMGDTF4YAQ6FWTKTAR64GP5GA, seg_pos={0 0}, upload=55340]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: write tcp 159.223.143.39:57642->77.90.31.15:4698: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jun 26 06:07:49.009528353 (68h56m15.935823063s) [seg_pos={0 0}, upload=55340]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil>
Jun 26 06:07:49.041471570 (68h56m15.967766251s) [upload=55340]: Done uploading segments...
Jun 26 06:07:49.041559110 (68h56m15.96785381s) [upload=55340]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11)
Duration: 30.037800663s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [.................................X..........................] duration:
691.395803ms
FAIL (2025-06-26 06:07:34 UTC)
orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded
Duration: 691.968625ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [...........................................................X] duration:
30.001438767s
FAIL (2025-06-26 06:07:09 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 30.001466949s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [..................................X.........................] duration:
912.23403ms
FAIL (2025-06-26 06:07:03 UTC)
orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded
Duration: 912.258919ms
core
uplink
us-select-1
us1-placement-20
|
checkers.AuthServiceHTTP{"https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us1)antiblip:
1 out of 60 is failed (threshold=3) [...................................X........................] duration:
161.482433ms
FAIL (2025-06-26 06:06:43 UTC)
orbiter warning: unexpected status code "500 Internal Server Error" "\n"
Duration: 161.509405ms
edge
authservice
http
bgp
us1
|
checkers.AuthServiceHTTP{"https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us1)antiblip:
1 out of 60 is failed (threshold=3) [....................................X.......................] duration:
198.771598ms
FAIL (2025-06-26 06:06:17 UTC)
orbiter warning: unexpected status code "500 Internal Server Error" "\n"
Duration: 198.934ms
edge
authservice
http
bgp
us1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [...................................X........................] duration:
712.087757ms
FAIL (2025-06-26 06:06:14 UTC)
orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded
Duration: 712.140288ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [....................................X.......................] duration:
751.227462ms
FAIL (2025-06-26 06:05:28 UTC)
orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded
Duration: 751.253014ms
core
uplink
us-select-1
us1-placement-20
|
checkers.AuthServiceHTTP{"https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us1)antiblip:
1 out of 60 is failed (threshold=3) [.....................................X......................] duration:
134.708834ms
FAIL (2025-06-26 06:05:25 UTC)
orbiter warning: unexpected status code "500 Internal Server Error" "\n"
Duration: 134.740356ms
edge
authservice
http
bgp
us1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [.....................................X......................] duration:
894.678534ms
FAIL (2025-06-26 06:04:53 UTC)
orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded
Duration: 895.212241ms
core
uplink
us-select-1
us1-placement-20
|
checkers.AuthServiceHTTP{"https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us1)antiblip:
1 out of 60 is failed (threshold=3) [......................................X.....................] duration:
171.369284ms
FAIL (2025-06-26 06:04:30 UTC)
orbiter warning: unexpected status code "500 Internal Server Error" "\n"
Duration: 171.398177ms
edge
authservice
http
bgp
us1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [......................................X.....................] duration:
784.022944ms
FAIL (2025-06-26 06:04:13 UTC)
orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded
Duration: 784.073844ms
core
uplink
us-select-1
us1-placement-20
|