Orbiter: EU/US/AP | Code | Config | Playbook | Edge Deployment | Escalation Procedure
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
17 out of 60 is failed (threshold=3) [.X....XX.X.X.X.......XXX...XX.....X.XXX...............X..X..] duration:
639.521622ms
FAIL (2025-07-26 23:52:30 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 639.542183ms
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
18 out of 60 is failed (threshold=3) [X.X....XX.X.X.X.......XXX...XX.....X.XXX...............X..X.] duration:
24.692138661s
FAIL (2025-07-26 23:52:29 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 24.692159903s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)antiblip:
56 out of 60 is failed (threshold=3) [...XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.XXXXX] duration:
30.039936918s
FAIL (2025-07-26 23:52:12 UTC)
orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 9)
Duration: 30.039967944s
core
uplink
us1
vivint
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
13 out of 60 is failed (threshold=3) [..X..XX.X...X..X.......X...X....XX..............X.X........X] duration:
1m0.001315644s
FAIL (2025-07-26 23:52:09 UTC)
operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 1m0.001353991s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)antiblip:
54 out of 60 is failed (threshold=3) [...XXXX.XXXXX.X.XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration:
1m0.026904236s
FAIL (2025-07-26 23:51:51 UTC)
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 10)
======LOGS======
.959129282s) [seg_pos={0 0}, upload=1827]: Done begin upload segment.
Jul 26 23:50:51.178261382 (1h59m6.959194762s) [noise=true , address=104.36.82.56:4243 , piece_id=OF5JCTG3JM4HWRJFEVW7CH6RRYLAXULAYMSDEIJNANK5HFNYHLAA, seg_pos={0 0}, upload=1827]: Uploading piece...
Jul 26 23:50:51.179146680 (1h59m6.960080065s) [noise=true , address=79.127.205.234:4610 , piece_id=3TW5UIUDCZXV5VRIQAWSVA4NIGUHWXIRSYUFFZ6WXWS2C33ROHQA, seg_pos={0 0}, upload=1827]: Uploading piece...
Jul 26 23:50:51.179203137 (1h59m6.960136514s) [noise=true , address=156.146.43.225:4448 , piece_id=XWSP242FPYUME47HXF72SHNDK7YT6FWNSPD2RIZTR35JEESWMHJA, seg_pos={0 0}, upload=1827]: Uploading piece...
Jul 26 23:50:51.179565343 (1h59m6.960498725s) [noise=true , address=77.90.31.11:4569 , piece_id=FLMT4B76OUTAR5XAR6S2FDCPNUTM37IKETF6UBUG7MRGUV2VH75A, seg_pos={0 0}, upload=1827]: Uploading piece...
Jul 26 23:50:51.179884017 (1h59m6.960817394s) [noise=true , address=104.36.82.46:4481 , piece_id=JT3AXWC4H4XZHKQMLW5XHLSDKQWZOPB6V7DVTJ7NLM7E3QQDDOWQ, seg_pos={0 0}, upload=1827]: Uploading piece...
Jul 26 23:50:51.179914277 (1h59m6.960847652s) [noise=true , address=64.49.13.140:4514 , piece_id=OCC7KSCOS6BE6PEOBUS4Q7MZRAGYDI7CD6Y5MLLO5FFBY7RADHUQ, seg_pos={0 0}, upload=1827]: Uploading piece...
Jul 26 23:50:51.180443644 (1h59m6.961377028s) [noise=true , address=104.36.82.47:4517 , piece_id=K4B2XNXTV7DPLI3CJD6IUQKL27Y6KWGV22GS4FNAR762K2I2GGYA, seg_pos={0 0}, upload=1827]: Uploading piece...
Jul 26 23:50:51.180738898 (1h59m6.961672281s) [noise=true , address=77.90.31.16:4767 , piece_id=JTU7P7QS4WJIOCSBOUDWUOFYL7RIE4MX34HNKMW7CTXBTPHXV5OA, seg_pos={0 0}, upload=1827]: Uploading piece...
Jul 26 23:50:51.180876611 (1h59m6.961809999s) [noise=true , address=104.36.82.53:4121 , piece_id=IU2HT47XTNDQPGCCP3KFVWF6VCLX3DVISCZTUTWSAYESUVSM6PLQ, seg_pos={0 0}, upload=1827]: Uploading piece...
Jul 26 23:50:51.181245114 (1h59m6.962178497s) [noise=true , address=77.90.31.17:4792 , piece_id=653PY5EYIYPAJUSIR62IXOERWFB347DPZHCL5ZAJ6WSP5AHI7NJQ, seg_pos={0 0}, upload=1827]: Uploading piece...
Jul 26 23:50:51.181407995 (1h59m6.962341373s) [noise=true , address=79.127.205.246:4277 , piece_id=7IMNLDOGS4HJBQK6XTBCYGUIEOPMA7V6L33J6J4P2TEOHVZ6NVWQ, seg_pos={0 0}, upload=1827]: Uploading piece...
Jul 26 23:50:51.181593110 (1h59m6.962526489s) [noise=true , address=79.127.201.215:4356 , piece_id=7D4IMDRCZ5XFNPK2T7ZHAD3SNQGGVKRSPPQKG5FPXHKO77X2A7RQ, seg_pos={0 0}, upload=1827]: Uploading piece...
Jul 26 23:50:51.181968126 (1h59m6.962901512s) [noise=true , address=45.91.116.5:4306 , piece_id=V5IURFLBU6H65RGYQZFM23O6QYDXSFL3EXPZC73CM2DGKNQTVRAA, seg_pos={0 0}, upload=1827]: Uploading piece...
Jul 26 23:50:51.182043158 (1h59m6.962976545s) [noise=true , address=77.90.31.15:4722 , piece_id=MLP3H6AXPRGNDM6VEWLZ56ORDHMDGDQOVBHVFCDQR7X4XUMGSZ3A, seg_pos={0 0}, upload=1827]: Uploading piece...
Jul 26 23:50:51.182618477 (1h59m6.963551857s) [noise=true , address=64.49.13.138:4482 , piece_id=HGUIJCKFXCMKTRQHINT6UKT4LKWW7NATHX6ULX5Q42GIQ2WZ6E4A, seg_pos={0 0}, upload=1827]: Uploading piece...
Jul 26 23:50:51.183269947 (1h59m6.964203327s) [noise=true , address=104.36.82.55:4222 , piece_id=MVC7DX33LDG3UBNHLBHDUJVUGDHUCH772NDMDZAV5Q23V2PFGFOQ, seg_pos={0 0}, upload=1827]: Uploading piece...
Jul 26 23:50:51.183819898 (1h59m6.964753276s) [noise=true , address=79.127.220.98:4422 , piece_id=R4XQY356LY7N2U65JQFZIK6N4ZLZQVU74Z7HRIFBQAP4NTZ4NTQA, seg_pos={0 0}, upload=1827]: Uploading piece...
Jul 26 23:50:51.185669167 (1h59m6.966602543s) [noise=true , address=104.36.82.54:4167 , piece_id=66WKVVW54BKJ4ZWSWFL5UWCL34F2ZVSIS6UQPYZ4H42G6TA5BDOA, seg_pos={0 0}, upload=1827]: Uploading piece...
Jul 26 23:50:51.277784191 (1h59m7.058717576s) [upload=1827]: Next returned nil segment
Jul 26 23:50:51.277812777 (1h59m7.058746397s) [upload=1827]: Waiting for error group managing segments...
Jul 26 23:50:51.308303375 (1h59m7.089236761s) [noise=true , address=77.90.31.11:4569 , piece_id=FLMT4B76OUTAR5XAR6S2FDCPNUTM37IKETF6UBUG7MRGUV2VH75A, seg_pos={0 0}, upload=1827]: Done uploading piece. err: <nil>
Jul 26 23:50:51.702448121 (1h59m7.483381509s) [noise=true , address=45.91.116.5:4306 , piece_id=V5IURFLBU6H65RGYQZFM23O6QYDXSFL3EXPZC73CM2DGKNQTVRAA, seg_pos={0 0}, upload=1827]: Done uploading piece. err: <nil>
Jul 26 23:50:51.746084254 (1h59m7.527017688s) [noise=true , address=79.127.205.246:4277 , piece_id=7IMNLDOGS4HJBQK6XTBCYGUIEOPMA7V6L33J6J4P2TEOHVZ6NVWQ, seg_pos={0 0}, upload=1827]: Done uploading piece. err: <nil>
Jul 26 23:50:51.801116499 (1h59m7.582049889s) [noise=true , address=79.127.205.234:4610 , piece_id=3TW5UIUDCZXV5VRIQAWSVA4NIGUHWXIRSYUFFZ6WXWS2C33ROHQA, seg_pos={0 0}, upload=1827]: Done uploading piece. err: <nil>
Jul 26 23:50:51.971134786 (1h59m7.752068149s) [noise=true , address=64.49.13.138:4482 , piece_id=HGUIJCKFXCMKTRQHINT6UKT4LKWW7NATHX6ULX5Q42GIQ2WZ6E4A, seg_pos={0 0}, upload=1827]: Done uploading piece. err: <nil>
Jul 26 23:50:53.733508775 (1h59m9.514442145s) [noise=true , address=79.127.220.98:4422 , piece_id=R4XQY356LY7N2U65JQFZIK6N4ZLZQVU74Z7HRIFBQAP4NTZ4NTQA, seg_pos={0 0}, upload=1827]: Done uploading piece. err: <nil>
Jul 26 23:50:56.067135334 (1h59m11.848068711s) [noise=true , address=156.146.43.225:4448 , piece_id=XWSP242FPYUME47HXF72SHNDK7YT6FWNSPD2RIZTR35JEESWMHJA, seg_pos={0 0}, upload=1827]: Done uploading piece. err: <nil>
Jul 26 23:50:57.227642526 (1h59m13.008575902s) [noise=true , address=64.49.13.140:4514 , piece_id=OCC7KSCOS6BE6PEOBUS4Q7MZRAGYDI7CD6Y5MLLO5FFBY7RADHUQ, seg_pos={0 0}, upload=1827]: Done uploading piece. err: <nil>
Jul 26 23:50:58.197785065 (1h59m13.978718435s) [noise=true , address=77.90.31.16:4767 , piece_id=JTU7P7QS4WJIOCSBOUDWUOFYL7RIE4MX34HNKMW7CTXBTPHXV5OA, seg_pos={0 0}, upload=1827]: Done uploading piece. err: <nil>
Jul 26 23:51:17.923930239 (1h59m33.704863602s) [noise=true , address=79.127.201.215:4356 , piece_id=7D4IMDRCZ5XFNPK2T7ZHAD3SNQGGVKRSPPQKG5FPXHKO77X2A7RQ, seg_pos={0 0}, upload=1827]: Done uploading piece. err: <nil>
Jul 26 23:51:51.106833575 (2h0m6.88776696s) [noise=true , address=104.36.82.46:4481 , piece_id=JT3AXWC4H4XZHKQMLW5XHLSDKQWZOPB6V7DVTJ7NLM7E3QQDDOWQ, seg_pos={0 0}, upload=1827]: Done uploading piece. err: ecclient: upload failed (node:12bRK8WWmAnWUwzCfHu8JtZgrruYSJCvhnsYTKcaLccJcv5RQjp, address:104.36.82.46:4481): protocol: rpc: dial tcp 104.36.82.46:4481: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 26 23:51:51.106946732 (2h0m6.887880113s) [noise=true , address=77.90.31.15:4722 , piece_id=MLP3H6AXPRGNDM6VEWLZ56ORDHMDGDQOVBHVFCDQR7X4XUMGSZ3A, seg_pos={0 0}, upload=1827]: Done uploading piece. err: ecclient: upload failed (node:12vx19tv5KVo9mpt896aAivK3RCEEaVnt67oRipCoU7QLa4GYzM, address:77.90.31.15:4722): protocol: expected piece hash; context deadline exceeded; EOF
Jul 26 23:51:51.107112453 (2h0m6.888045842s) [noise=true , address=77.90.31.17:4792 , piece_id=653PY5EYIYPAJUSIR62IXOERWFB347DPZHCL5ZAJ6WSP5AHI7NJQ, seg_pos={0 0}, upload=1827]: Done uploading piece. err: ecclient: upload failed (node:12SPFgYZnbwjEupjaghfs61qwFM678KNPwpDmRNqV1j2opNgZCN, address:77.90.31.17:4792): protocol: rpc: dial tcp 77.90.31.17:4792: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 26 23:51:51.107473667 (2h0m6.888407051s) [noise=true , address=104.36.82.55:4222 , piece_id=MVC7DX33LDG3UBNHLBHDUJVUGDHUCH772NDMDZAV5Q23V2PFGFOQ, seg_pos={0 0}, upload=1827]: Done uploading piece. err: ecclient: upload failed (node:12ZKW35KvrsoYRefbYeUFZo8bh4XZYycNKGEvTFjzibezjXgj3e, address:104.36.82.55:4222): protocol: rpc: dial tcp 104.36.82.55:4222: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 26 23:51:51.107553162 (2h0m6.88848654s) [noise=true , address=104.36.82.56:4243 , piece_id=OF5JCTG3JM4HWRJFEVW7CH6RRYLAXULAYMSDEIJNANK5HFNYHLAA, seg_pos={0 0}, upload=1827]: Done uploading piece. err: ecclient: upload failed (node:12Fyogwa9NcRSUwxTPFTGL68Dbn1FEAqs8K48CNtTyKQEgVwY1c, address:104.36.82.56:4243): protocol: write tcp 159.223.143.39:35570->104.36.82.56:4243: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 26 23:51:51.107649947 (2h0m6.888583313s) [noise=true , address=104.36.82.47:4517 , piece_id=K4B2XNXTV7DPLI3CJD6IUQKL27Y6KWGV22GS4FNAR762K2I2GGYA, seg_pos={0 0}, upload=1827]: Done uploading piece. err: ecclient: upload failed (node:12HZjKTnjpeNU5B6muVHGrmWQS8GsyuYTW6cLxzNk5CGt3b8PJS, address:104.36.82.47:4517): protocol: write tcp 159.223.143.39:41120->104.36.82.47:4517: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 26 23:51:51.107743740 (2h0m6.888677116s) [noise=true , address=104.36.82.53:4121 , piece_id=IU2HT47XTNDQPGCCP3KFVWF6VCLX3DVISCZTUTWSAYESUVSM6PLQ, seg_pos={0 0}, upload=1827]: Done uploading piece. err: ecclient: upload failed (node:1QJBWV6xLXS7DcX8d26be79z1DH3r27oi4EcuiwVwR5Dn8JEGh, address:104.36.82.53:4121): protocol: rpc: dial tcp 104.36.82.53:4121: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 26 23:51:51.107819214 (2h0m6.888752583s) [noise=true , address=104.36.82.54:4167 , piece_id=66WKVVW54BKJ4ZWSWFL5UWCL34F2ZVSIS6UQPYZ4H42G6TA5BDOA, seg_pos={0 0}, upload=1827]: Done uploading piece. err: ecclient: upload failed (node:1hhT46NW5CQ7dWHRxy5MSjyKc3PiQqBifThT7dBmx2sgNvn8GZ, address:104.36.82.54:4167): protocol: write tcp 159.223.143.39:47212->104.36.82.54:4167: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 26 23:51:51.107858434 (2h0m6.888791823s) [seg_pos={0 0}, upload=1827]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil>
Jul 26 23:51:51.131672555 (2h0m6.912605924s) [upload=1827]: Done uploading segments...
Jul 26 23:51:51.131700105 (2h0m6.912633482s) [upload=1827]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10)
Duration: 1m0.02692711s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
19 out of 60 is failed (threshold=3) [XX.X....XX.X.X.X.......XXX...XX.....X.XXX...............X..X] duration:
30.001596798s
FAIL (2025-07-26 23:51:50 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 30.001623646s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
53 out of 60 is failed (threshold=3) [......XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.XXXXXXXXXXXXXX] duration:
30.0296708s
FAIL (2025-07-26 23:51:46 UTC)
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 8)
======LOGS======
26 23:51:16.261048963 (1h59m32.041982357s) [upload=1824]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9)
Jul 26 23:51:16.377982237 (1h59m32.158915605s) [upload=1832]: Starting upload
Jul 26 23:51:16.378077340 (1h59m32.159010729s) [upload=1832]: Done starting upload
Jul 26 23:51:16.378466007 (1h59m32.159399385s) [upload=1832]: Uploading segments...
Jul 26 23:51:16.378534938 (1h59m32.159468317s) [upload=1832]: Got next segment. Inline: false
Jul 26 23:51:16.378568467 (1h59m32.159501843s) [upload=1832]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jul 26 23:51:16.378573507 (1h59m32.159506884s) [upload=1832]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jul 26 23:51:16.412670039 (1h59m32.193603416s) [seg_pos={0 0}, upload=1832]: Begin upload segment...
Jul 26 23:51:16.412732682 (1h59m32.193666047s) [seg_pos={0 0}, upload=1832]: Done begin upload segment.
Jul 26 23:51:16.412762000 (1h59m32.193695379s) [noise=true , address=77.90.31.19:4841 , piece_id=BXTC5RAEVKXDPQNZZWWQA3ABWLGEICPT5IGTUQDBIAN4RFYGGKMQ, seg_pos={0 0}, upload=1832]: Uploading piece...
Jul 26 23:51:16.413212564 (1h59m32.194145943s) [noise=true , address=104.36.82.7:4912 , piece_id=2BGCACSENNV62Q6OC2DMYY3LTAR63AJRR7UYKRJQM2HRRUGWOZSA, seg_pos={0 0}, upload=1832]: Uploading piece...
Jul 26 23:51:16.413447576 (1h59m32.194380959s) [noise=true , address=45.91.116.4:4940 , piece_id=H7PFRUIEX3MROPO7LQ2N3X75FC35QIM4CNBB4LFDP5F7PC2T56WA, seg_pos={0 0}, upload=1832]: Uploading piece...
Jul 26 23:51:16.414208622 (1h59m32.195142009s) [noise=true , address=77.90.31.21:4878 , piece_id=ZWHN4LMK6L6ENQ7DWOCFXODAOBZABCPHS5PWQXBINMU6RZMSE2PA, seg_pos={0 0}, upload=1832]: Uploading piece...
Jul 26 23:51:16.414255649 (1h59m32.195189037s) [noise=true , address=45.91.116.14:4189 , piece_id=7IPZVQZPG4XO3ULSRBXWVAJ5QRJAUXDCVR52G3WISLLKJ7MYRAXQ, seg_pos={0 0}, upload=1832]: Uploading piece...
Jul 26 23:51:16.414735270 (1h59m32.195668648s) [noise=true , address=77.90.31.18:4805 , piece_id=QQENGNBKMLYC5BVNHIE34LOW3UU5TI52W7E5RUBI5G7S7QH3FY4Q, seg_pos={0 0}, upload=1832]: Uploading piece...
Jul 26 23:51:16.414776049 (1h59m32.195709429s) [noise=true , address=104.36.82.8:4969 , piece_id=LFJKW5Y2HQJ4NHDJ37Z265FB5FQMNNMOLMC5YTZ6MHTXSWRKDNPQ, seg_pos={0 0}, upload=1832]: Uploading piece...
Jul 26 23:51:16.415149906 (1h59m32.196083291s) [noise=true , address=104.36.82.8:4972 , piece_id=S6XF26HSSBRVJMHHNJ2KULQRUDS4S7FO2AHSKK4YKM5FRLLWFAHA, seg_pos={0 0}, upload=1832]: Uploading piece...
Jul 26 23:51:16.415330759 (1h59m32.196264149s) [noise=true , address=104.36.82.16:4294 , piece_id=DVSZEY64WXHTTOKQNNQDY64T4TIZS2C7NIRAFGIYTVDIQEB43UWQ, seg_pos={0 0}, upload=1832]: Uploading piece...
Jul 26 23:51:16.415622799 (1h59m32.196556187s) [noise=true , address=77.90.31.17:4772 , piece_id=BXCJ2CN5JN345BFIUGHMFXFD7O7AU5BRQS6HIY7PZTP7KO7NTVJA, seg_pos={0 0}, upload=1832]: Uploading piece...
Jul 26 23:51:16.415897740 (1h59m32.196831126s) [noise=true , address=104.36.82.15:4257 , piece_id=T2RH4BAFLUIYMX2HHNHIKMPUI7KSYPXYQHD2NYWISOLTCHWPWHQA, seg_pos={0 0}, upload=1832]: Uploading piece...
Jul 26 23:51:16.416022039 (1h59m32.196955426s) [noise=true , address=77.90.31.10:4520 , piece_id=TI5OHJ3M5MVUEZRZHOXX3AZ6MYTWW4YJ6SBQGA35YXLEN7F4CWJQ, seg_pos={0 0}, upload=1832]: Uploading piece...
Jul 26 23:51:16.416530132 (1h59m32.197463521s) [noise=true , address=104.36.82.13:4187 , piece_id=BP2Q373XFGHP5RVWY3H4Y3LNNK3SF4UCMZ6HACYRPTYEFW3JRTOA, seg_pos={0 0}, upload=1832]: Uploading piece...
Jul 26 23:51:16.416697932 (1h59m32.197631309s) [noise=true , address=45.91.116.5:4297 , piece_id=EFLU373IG6SWZSUQGZOYFNUNFQXF45V2CTHYM5I5QX7WEUU7GE5A, seg_pos={0 0}, upload=1832]: Uploading piece...
Jul 26 23:51:16.416910861 (1h59m32.197844249s) [noise=true , address=45.91.116.13:4153 , piece_id=B6WGKDV56QWEUU7PEODDXKUCZOG3SOQG2FBXO2OOI5WTHG3666AQ, seg_pos={0 0}, upload=1832]: Uploading piece...
Jul 26 23:51:16.417152076 (1h59m32.198085462s) [noise=true , address=45.91.116.12:4225 , piece_id=SLXVYTU2BM5CSU56FO6UWFQEY5CH4DKFLQCD3YQQ4R24QFVJ6T4Q, seg_pos={0 0}, upload=1832]: Uploading piece...
Jul 26 23:51:16.526433751 (1h59m32.307367145s) [upload=1832]: Next returned nil segment
Jul 26 23:51:16.526464196 (1h59m32.307397578s) [upload=1832]: Waiting for error group managing segments...
Jul 26 23:51:16.529936697 (1h59m32.310870065s) [noise=true , address=45.91.116.13:4153 , piece_id=B6WGKDV56QWEUU7PEODDXKUCZOG3SOQG2FBXO2OOI5WTHG3666AQ, seg_pos={0 0}, upload=1832]: Done uploading piece. err: <nil>
Jul 26 23:51:16.530001187 (1h59m32.310934559s) [noise=true , address=77.90.31.10:4520 , piece_id=TI5OHJ3M5MVUEZRZHOXX3AZ6MYTWW4YJ6SBQGA35YXLEN7F4CWJQ, seg_pos={0 0}, upload=1832]: Done uploading piece. err: <nil>
Jul 26 23:51:16.530937367 (1h59m32.311870735s) [noise=true , address=77.90.31.21:4878 , piece_id=ZWHN4LMK6L6ENQ7DWOCFXODAOBZABCPHS5PWQXBINMU6RZMSE2PA, seg_pos={0 0}, upload=1832]: Done uploading piece. err: <nil>
Jul 26 23:51:16.530959441 (1h59m32.311892825s) [noise=true , address=45.91.116.12:4225 , piece_id=SLXVYTU2BM5CSU56FO6UWFQEY5CH4DKFLQCD3YQQ4R24QFVJ6T4Q, seg_pos={0 0}, upload=1832]: Done uploading piece. err: <nil>
Jul 26 23:51:16.531544648 (1h59m32.312478019s) [noise=true , address=45.91.116.4:4940 , piece_id=H7PFRUIEX3MROPO7LQ2N3X75FC35QIM4CNBB4LFDP5F7PC2T56WA, seg_pos={0 0}, upload=1832]: Done uploading piece. err: <nil>
Jul 26 23:51:16.532489779 (1h59m32.313423159s) [noise=true , address=45.91.116.5:4297 , piece_id=EFLU373IG6SWZSUQGZOYFNUNFQXF45V2CTHYM5I5QX7WEUU7GE5A, seg_pos={0 0}, upload=1832]: Done uploading piece. err: <nil>
Jul 26 23:51:16.569773807 (1h59m32.350707174s) [noise=true , address=45.91.116.14:4189 , piece_id=7IPZVQZPG4XO3ULSRBXWVAJ5QRJAUXDCVR52G3WISLLKJ7MYRAXQ, seg_pos={0 0}, upload=1832]: Done uploading piece. err: <nil>
Jul 26 23:51:17.879349361 (1h59m33.660282739s) [noise=true , address=77.90.31.19:4841 , piece_id=BXTC5RAEVKXDPQNZZWWQA3ABWLGEICPT5IGTUQDBIAN4RFYGGKMQ, seg_pos={0 0}, upload=1832]: Done uploading piece. err: <nil>
Jul 26 23:51:46.348332716 (2h0m2.129266106s) [noise=true , address=104.36.82.15:4257 , piece_id=T2RH4BAFLUIYMX2HHNHIKMPUI7KSYPXYQHD2NYWISOLTCHWPWHQA, seg_pos={0 0}, upload=1832]: Done uploading piece. err: ecclient: upload failed (node:12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK, address:104.36.82.15:4257): protocol: rpc: dial tcp 104.36.82.15:4257: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 26 23:51:46.348427245 (2h0m2.129360623s) [noise=true , address=77.90.31.17:4772 , piece_id=BXCJ2CN5JN345BFIUGHMFXFD7O7AU5BRQS6HIY7PZTP7KO7NTVJA, seg_pos={0 0}, upload=1832]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: rpc: dial tcp 77.90.31.17:4772: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 26 23:51:46.348495376 (2h0m2.129428742s) [noise=true , address=104.36.82.13:4187 , piece_id=BP2Q373XFGHP5RVWY3H4Y3LNNK3SF4UCMZ6HACYRPTYEFW3JRTOA, seg_pos={0 0}, upload=1832]: Done uploading piece. err: ecclient: upload failed (node:12cQxBB2L2CyymYx3xWp1ALj6p8QWr4A34sqvtG5zyeT7sVWv7x, address:104.36.82.13:4187): protocol: rpc: dial tcp 104.36.82.13:4187: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 26 23:51:46.348553293 (2h0m2.129486668s) [noise=true , address=77.90.31.18:4805 , piece_id=QQENGNBKMLYC5BVNHIE34LOW3UU5TI52W7E5RUBI5G7S7QH3FY4Q, seg_pos={0 0}, upload=1832]: Done uploading piece. err: ecclient: upload failed (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, address:77.90.31.18:4805): protocol: rpc: dial tcp 77.90.31.18:4805: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 26 23:51:46.348616454 (2h0m2.129549829s) [noise=true , address=104.36.82.8:4969 , piece_id=LFJKW5Y2HQJ4NHDJ37Z265FB5FQMNNMOLMC5YTZ6MHTXSWRKDNPQ, seg_pos={0 0}, upload=1832]: Done uploading piece. err: ecclient: upload failed (node:1HxTyEaQTErcZJWMYLsND9Xa8Nwaa5e1f4x2bs7UQzSRiW7buX, address:104.36.82.8:4969): protocol: rpc: dial tcp 104.36.82.8:4969: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 26 23:51:46.348674420 (2h0m2.129607797s) [noise=true , address=104.36.82.8:4972 , piece_id=S6XF26HSSBRVJMHHNJ2KULQRUDS4S7FO2AHSKK4YKM5FRLLWFAHA, seg_pos={0 0}, upload=1832]: Done uploading piece. err: ecclient: upload failed (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX, address:104.36.82.8:4972): protocol: rpc: dial tcp 104.36.82.8:4972: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 26 23:51:46.348732514 (2h0m2.129665882s) [noise=true , address=104.36.82.16:4294 , piece_id=DVSZEY64WXHTTOKQNNQDY64T4TIZS2C7NIRAFGIYTVDIQEB43UWQ, seg_pos={0 0}, upload=1832]: Done uploading piece. err: ecclient: upload failed (node:12CbfkWK4H84My3e6qhDuAkTrHWsooysz7KE4bk6VoDRDWEgb9e, address:104.36.82.16:4294): protocol: rpc: dial tcp 104.36.82.16:4294: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 26 23:51:46.348787969 (2h0m2.129721345s) [noise=true , address=104.36.82.7:4912 , piece_id=2BGCACSENNV62Q6OC2DMYY3LTAR63AJRR7UYKRJQM2HRRUGWOZSA, seg_pos={0 0}, upload=1832]: Done uploading piece. err: ecclient: upload failed (node:1gwuDRBC5oJXgZbP6FqP9tc8UB6aJt2uexUgSuzHCovw27v1NR, address:104.36.82.7:4912): protocol: write tcp 159.223.143.39:40614->104.36.82.7:4912: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 26 23:51:46.348809952 (2h0m2.129743338s) [seg_pos={0 0}, upload=1832]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil>
Jul 26 23:51:46.375129894 (2h0m2.156063321s) [upload=1832]: Done uploading segments...
Jul 26 23:51:46.375165498 (2h0m2.156098876s) [upload=1832]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8)
Duration: 30.029689117s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
52 out of 60 is failed (threshold=3) [.......XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.XXXXXXXXXXXXX] duration:
30.028697262s
FAIL (2025-07-26 23:51:16 UTC)
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 9)
======LOGS======
Done waiting for segment. successful: 11 optimal: 13 errs: <nil>
Jul 26 23:50:14.774834627 (1h58m30.555768005s) [upload=1806]: Done uploading segments...
Jul 26 23:50:14.774880812 (1h58m30.555814191s) [upload=1806]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11)
Jul 26 23:50:46.263134063 (1h59m2.044067453s) [upload=1824]: Starting upload
Jul 26 23:50:46.263216917 (1h59m2.044150291s) [upload=1824]: Done starting upload
Jul 26 23:50:46.263526278 (1h59m2.044459648s) [upload=1824]: Uploading segments...
Jul 26 23:50:46.263581707 (1h59m2.04451509s) [upload=1824]: Got next segment. Inline: false
Jul 26 23:50:46.263621851 (1h59m2.044555235s) [upload=1824]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jul 26 23:50:46.263628197 (1h59m2.044561582s) [upload=1824]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jul 26 23:50:46.294400796 (1h59m2.075334183s) [seg_pos={0 0}, upload=1824]: Begin upload segment...
Jul 26 23:50:46.294464897 (1h59m2.075398281s) [seg_pos={0 0}, upload=1824]: Done begin upload segment.
Jul 26 23:50:46.294524059 (1h59m2.075457438s) [noise=true , address=104.36.82.13:4187 , piece_id=NCP7XFGLC4HR3JHAURMNVJ3KGOO5ZIC45ZM7447XBQJSG3ETRRNQ, seg_pos={0 0}, upload=1824]: Uploading piece...
Jul 26 23:50:46.295059351 (1h59m2.075992734s) [noise=true , address=45.91.116.6:4974 , piece_id=T2HM65HYT7ZMU4U26S3AAEUIZ7W5TCOMQ2I4H4SRAMBSA6SOTBHQ, seg_pos={0 0}, upload=1824]: Uploading piece...
Jul 26 23:50:46.295467438 (1h59m2.076400815s) [noise=true , address=45.91.116.8:4047 , piece_id=I6A3I7KVZKDNOFC75R5O24RIH2DQ3XHZPID7QVNKH6ZQHG234DVA, seg_pos={0 0}, upload=1824]: Uploading piece...
Jul 26 23:50:46.295861637 (1h59m2.076795017s) [noise=true , address=104.36.82.9:4005 , piece_id=XY4YQ5OI6TOMIZ6ZXDEB4J6VC4P5GRFDLOT7FKIJOYVWJXW6DL6Q, seg_pos={0 0}, upload=1824]: Uploading piece...
Jul 26 23:50:46.296432319 (1h59m2.077365704s) [noise=true , address=45.91.116.6:4975 , piece_id=CVZLNWPBWEJYLT4LBFYCBEKYXYH7NOY7GNJPVZW35E7BUPHCBYPQ, seg_pos={0 0}, upload=1824]: Uploading piece...
Jul 26 23:50:46.296884733 (1h59m2.077818119s) [noise=true , address=77.90.31.15:4697 , piece_id=OG7L57X66JSNP3DO52E6LTCAE4HCBUK3NBPGIOVUYC6QBLEX5OHQ, seg_pos={0 0}, upload=1824]: Uploading piece...
Jul 26 23:50:46.297301936 (1h59m2.078235321s) [noise=true , address=104.36.82.16:4294 , piece_id=SGX35GWGTDCM5FIRIGCHEEUOR46DLOU5ZRVMWYN5WVULXDRDGRMQ, seg_pos={0 0}, upload=1824]: Uploading piece...
Jul 26 23:50:46.297686767 (1h59m2.078620147s) [noise=true , address=104.36.82.9:4007 , piece_id=XH2DW4QTSPULNSQ2K3DAYU737ZH5VXBSSLCRQBZ7YSWC5ZAMUIMQ, seg_pos={0 0}, upload=1824]: Uploading piece...
Jul 26 23:50:46.298569556 (1h59m2.079502936s) [noise=true , address=104.36.82.5:4839 , piece_id=5RQC5COYAZ5WM3V76UE55QBH3QRCSPNQPX2MT572IOODFHNZ5EYQ, seg_pos={0 0}, upload=1824]: Uploading piece...
Jul 26 23:50:46.298946623 (1h59m2.079880004s) [noise=true , address=45.91.116.10:4083 , piece_id=NMWGYAVQVMVXABLZXZ7ENAZNTNWS45HEWBFQRXXSSMQVHNC35JPQ, seg_pos={0 0}, upload=1824]: Uploading piece...
Jul 26 23:50:46.299355342 (1h59m2.080288724s) [noise=true , address=45.91.116.9:4262 , piece_id=M5JDD66PKMFZBKFX2JMXASO5FMR6DMXY6WOLDC3MSLIVPUANI77Q, seg_pos={0 0}, upload=1824]: Uploading piece...
Jul 26 23:50:46.299724032 (1h59m2.080657419s) [noise=true , address=104.36.82.4:4801 , piece_id=A6RPYR5SNYJ3344DBDC4R3RTM3IKQ4DKHJVNFINPI5V6XZD2BVBA, seg_pos={0 0}, upload=1824]: Uploading piece...
Jul 26 23:50:46.300148671 (1h59m2.081082058s) [noise=true , address=45.91.116.4:4938 , piece_id=NV3AVDIDY7KAKY73SS4PM6NW2DVDQWJINKKTKAR4FNGBSJDJIXGQ, seg_pos={0 0}, upload=1824]: Uploading piece...
Jul 26 23:50:46.300543936 (1h59m2.081477321s) [noise=true , address=77.90.31.13:4628 , piece_id=6OIXFOZSSJHKWFHDCIA55FLKIHCC3BQYR7WTDSLX42Q6226FXM7A, seg_pos={0 0}, upload=1824]: Uploading piece...
Jul 26 23:50:46.300886438 (1h59m2.081819826s) [noise=true , address=104.36.82.2:4732 , piece_id=XMWHE6SDRP7AJZAZVUSXYZQJSR245STM6JPJQL3MSWUNFUF7D7HA, seg_pos={0 0}, upload=1824]: Uploading piece...
Jul 26 23:50:46.301238368 (1h59m2.082171752s) [noise=true , address=45.91.116.14:4189 , piece_id=252XE2CKXYGKMMHCACZHVQ4S6MX7YS6IBTFS3BJRJFSI36O2UOTQ, seg_pos={0 0}, upload=1824]: Uploading piece...
Jul 26 23:50:46.385635730 (1h59m2.166569107s) [upload=1824]: Next returned nil segment
Jul 26 23:50:46.385662191 (1h59m2.166595576s) [upload=1824]: Waiting for error group managing segments...
Jul 26 23:50:46.389429798 (1h59m2.170363167s) [noise=true , address=45.91.116.10:4083 , piece_id=NMWGYAVQVMVXABLZXZ7ENAZNTNWS45HEWBFQRXXSSMQVHNC35JPQ, seg_pos={0 0}, upload=1824]: Done uploading piece. err: <nil>
Jul 26 23:50:46.389975111 (1h59m2.170908472s) [noise=true , address=45.91.116.4:4938 , piece_id=NV3AVDIDY7KAKY73SS4PM6NW2DVDQWJINKKTKAR4FNGBSJDJIXGQ, seg_pos={0 0}, upload=1824]: Done uploading piece. err: <nil>
Jul 26 23:50:46.390112755 (1h59m2.171046131s) [noise=true , address=45.91.116.6:4974 , piece_id=T2HM65HYT7ZMU4U26S3AAEUIZ7W5TCOMQ2I4H4SRAMBSA6SOTBHQ, seg_pos={0 0}, upload=1824]: Done uploading piece. err: <nil>
Jul 26 23:50:46.390469691 (1h59m2.171403055s) [noise=true , address=45.91.116.6:4975 , piece_id=CVZLNWPBWEJYLT4LBFYCBEKYXYH7NOY7GNJPVZW35E7BUPHCBYPQ, seg_pos={0 0}, upload=1824]: Done uploading piece. err: <nil>
Jul 26 23:50:46.390677819 (1h59m2.171611206s) [noise=true , address=45.91.116.8:4047 , piece_id=I6A3I7KVZKDNOFC75R5O24RIH2DQ3XHZPID7QVNKH6ZQHG234DVA, seg_pos={0 0}, upload=1824]: Done uploading piece. err: <nil>
Jul 26 23:50:46.438128277 (1h59m2.219061658s) [noise=true , address=45.91.116.14:4189 , piece_id=252XE2CKXYGKMMHCACZHVQ4S6MX7YS6IBTFS3BJRJFSI36O2UOTQ, seg_pos={0 0}, upload=1824]: Done uploading piece. err: <nil>
Jul 26 23:50:46.503792335 (1h59m2.284725716s) [noise=true , address=45.91.116.9:4262 , piece_id=M5JDD66PKMFZBKFX2JMXASO5FMR6DMXY6WOLDC3MSLIVPUANI77Q, seg_pos={0 0}, upload=1824]: Done uploading piece. err: <nil>
Jul 26 23:50:47.774058502 (1h59m3.554991865s) [noise=true , address=77.90.31.13:4628 , piece_id=6OIXFOZSSJHKWFHDCIA55FLKIHCC3BQYR7WTDSLX42Q6226FXM7A, seg_pos={0 0}, upload=1824]: Done uploading piece. err: <nil>
Jul 26 23:50:47.793501082 (1h59m3.574434468s) [noise=true , address=77.90.31.15:4697 , piece_id=OG7L57X66JSNP3DO52E6LTCAE4HCBUK3NBPGIOVUYC6QBLEX5OHQ, seg_pos={0 0}, upload=1824]: Done uploading piece. err: <nil>
Jul 26 23:51:16.235726962 (1h59m32.016660353s) [noise=true , address=104.36.82.16:4294 , piece_id=SGX35GWGTDCM5FIRIGCHEEUOR46DLOU5ZRVMWYN5WVULXDRDGRMQ, seg_pos={0 0}, upload=1824]: Done uploading piece. err: ecclient: upload failed (node:12CbfkWK4H84My3e6qhDuAkTrHWsooysz7KE4bk6VoDRDWEgb9e, address:104.36.82.16:4294): protocol: write tcp 159.223.143.39:43906->104.36.82.16:4294: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 26 23:51:16.235813169 (1h59m32.016746548s) [noise=true , address=104.36.82.9:4007 , piece_id=XH2DW4QTSPULNSQ2K3DAYU737ZH5VXBSSLCRQBZ7YSWC5ZAMUIMQ, seg_pos={0 0}, upload=1824]: Done uploading piece. err: ecclient: upload failed (node:19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp, address:104.36.82.9:4007): protocol: write tcp 159.223.143.39:42212->104.36.82.9:4007: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 26 23:51:16.235877202 (1h59m32.016810581s) [noise=true , address=104.36.82.2:4732 , piece_id=XMWHE6SDRP7AJZAZVUSXYZQJSR245STM6JPJQL3MSWUNFUF7D7HA, seg_pos={0 0}, upload=1824]: Done uploading piece. err: ecclient: upload failed (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE, address:104.36.82.2:4732): protocol: rpc: dial tcp 104.36.82.2:4732: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 26 23:51:16.235943132 (1h59m32.016876512s) [noise=true , address=104.36.82.9:4005 , piece_id=XY4YQ5OI6TOMIZ6ZXDEB4J6VC4P5GRFDLOT7FKIJOYVWJXW6DL6Q, seg_pos={0 0}, upload=1824]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:104.36.82.9:4005): protocol: rpc: dial tcp 104.36.82.9:4005: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 26 23:51:16.236001838 (1h59m32.016935217s) [noise=true , address=104.36.82.13:4187 , piece_id=NCP7XFGLC4HR3JHAURMNVJ3KGOO5ZIC45ZM7447XBQJSG3ETRRNQ, seg_pos={0 0}, upload=1824]: Done uploading piece. err: ecclient: upload failed (node:12cQxBB2L2CyymYx3xWp1ALj6p8QWr4A34sqvtG5zyeT7sVWv7x, address:104.36.82.13:4187): protocol: write tcp 159.223.143.39:46578->104.36.82.13:4187: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 26 23:51:16.236085703 (1h59m32.017019086s) [noise=true , address=104.36.82.5:4839 , piece_id=5RQC5COYAZ5WM3V76UE55QBH3QRCSPNQPX2MT572IOODFHNZ5EYQ, seg_pos={0 0}, upload=1824]: Done uploading piece. err: ecclient: upload failed (node:1FcPS8yXfgdFvsxFofPrNCtFDLtebQMTaLu75335kVXeQxttuv, address:104.36.82.5:4839): protocol: rpc: dial tcp 104.36.82.5:4839: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 26 23:51:16.236156367 (1h59m32.017089747s) [noise=true , address=104.36.82.4:4801 , piece_id=A6RPYR5SNYJ3344DBDC4R3RTM3IKQ4DKHJVNFINPI5V6XZD2BVBA, seg_pos={0 0}, upload=1824]: Done uploading piece. err: ecclient: upload failed (node:12mj4gkN1YNsnfjc9svNEhUo8EAgEmHe7yj4kScHCasB9RwVNwZ, address:104.36.82.4:4801): protocol: rpc: dial tcp 104.36.82.4:4801: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 26 23:51:16.236176291 (1h59m32.017109681s) [seg_pos={0 0}, upload=1824]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil>
Jul 26 23:51:16.261018201 (1h59m32.041951593s) [upload=1824]: Done uploading segments...
Jul 26 23:51:16.261048963 (1h59m32.041982357s) [upload=1824]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9)
Duration: 30.028726733s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
12 out of 60 is failed (threshold=3) [...X..XX.X...X..X.......X...X....XX..............X.X........] duration:
26.918788473s
FAIL (2025-07-26 23:51:08 UTC)
context deadline exceeded
Duration: 26.918818339s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)antiblip:
55 out of 60 is failed (threshold=3) [....XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.XXXX] duration:
30.030561403s
FAIL (2025-07-26 23:51:04 UTC)
orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 9)
Duration: 30.030592445s
core
uplink
us1
vivint
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
19 out of 60 is failed (threshold=3) [XXX.X....XX.X.X.X.......XXX...XX.....X.XXX...............X..] duration:
1.889780951s
FAIL (2025-07-26 23:51:02 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 1.889810249s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)antiblip:
53 out of 60 is failed (threshold=3) [....XXXX.XXXXX.X.XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration:
1m0.033701335s
FAIL (2025-07-26 23:50:51 UTC)
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 10)
======LOGS======
...
Jul 26 23:49:51.062512474 (1h58m6.843445839s) [seg_pos={0 0}, upload=1810]: Done begin upload segment.
Jul 26 23:49:51.062567904 (1h58m6.843501287s) [noise=true , address=104.36.82.45:4910 , piece_id=2FPGI2PLU5S6Q5YDWSSDYT35OAKTCFHPUMRPHUMBDUU2PUOEH2YQ, seg_pos={0 0}, upload=1810]: Uploading piece...
Jul 26 23:49:51.062600660 (1h58m6.843534042s) [noise=true , address=79.127.201.215:4351 , piece_id=ZGV32GT6FLCDUF45JZLKJ7OMXH7RIJ5W7IINGHFTW33VBQEDMMOQ, seg_pos={0 0}, upload=1810]: Uploading piece...
Jul 26 23:49:51.063098862 (1h58m6.844032243s) [noise=true , address=77.90.31.15:4714 , piece_id=H66HQNWM2QUA36L2TWQBMBV62EJXUIT4BCXFPYRIMWQY3SOCQGZA, seg_pos={0 0}, upload=1810]: Uploading piece...
Jul 26 23:49:51.063249255 (1h58m6.844182645s) [noise=true , address=77.90.31.11:4562 , piece_id=YBE3OL6TWY7ZKD4JKVA3SM3SJTIUOL2FQNBH76L43GR22ZANTZRA, seg_pos={0 0}, upload=1810]: Uploading piece...
Jul 26 23:49:51.063530794 (1h58m6.844464162s) [noise=true , address=79.127.219.38:4015 , piece_id=P6QFOWJINWHOZSHXMRESNEKHVJDC7LKQN7DZBOA3EVO2CVNBUHDQ, seg_pos={0 0}, upload=1810]: Uploading piece...
Jul 26 23:49:51.063781607 (1h58m6.844714986s) [noise=true , address=79.127.205.234:4609 , piece_id=BU23SDJE7IA5SWDWPQ4YACDNOM667EUJL7WQTPXKNFVKIWBK2PRA, seg_pos={0 0}, upload=1810]: Uploading piece...
Jul 26 23:49:51.063958145 (1h58m6.84489164s) [noise=true , address=104.36.82.46:4482 , piece_id=DZCSV5P6GZZFBY5G6X6GCTCBTWPWO6L47AGJP4F6D2TGCQDEBXVQ, seg_pos={0 0}, upload=1810]: Uploading piece...
Jul 26 23:49:51.064564164 (1h58m6.845497545s) [noise=true , address=45.91.116.9:4277 , piece_id=EGV63SXH2PZ2VSUWPWHYMRJSCSH65GYQWD3L5NCNCM6EGDXOHXMA, seg_pos={0 0}, upload=1810]: Uploading piece...
Jul 26 23:49:51.065420599 (1h58m6.846353987s) [noise=true , address=104.36.82.55:4224 , piece_id=V76WAOWB6BNQ3ZRBVDFMVCKU3RQE5QMIMMBJWFUWRRDGDXIWJCHQ, seg_pos={0 0}, upload=1810]: Uploading piece...
Jul 26 23:49:51.066134773 (1h58m6.847068153s) [noise=true , address=64.49.13.154:4927 , piece_id=HYJ622FJCHC3LSMCZS6GGTCCPX6P6UIFGYTVGGSPKRVXJMFDRWYQ, seg_pos={0 0}, upload=1810]: Uploading piece...
Jul 26 23:49:51.067323522 (1h58m6.848256908s) [noise=true , address=79.127.205.245:4248 , piece_id=HWQLK367ZLE2OXRILUK53PSTJH6CPQ5VGR4L4OK2G7W43JHPFAXQ, seg_pos={0 0}, upload=1810]: Uploading piece...
Jul 26 23:49:51.067916192 (1h58m6.84884957s) [noise=true , address=104.36.82.5:4870 , piece_id=XBPXX2Y2SSS2EAHRFRN6RCZDFOKIU5ECYSDSSMT6I2WTKOEU7JMA, seg_pos={0 0}, upload=1810]: Uploading piece...
Jul 26 23:49:51.068486830 (1h58m6.849420209s) [noise=true , address=77.90.31.17:4787 , piece_id=GXREXTJ5NGAO3WGC2NECV32J4FLJMWKY662TAK2YWGZDEV5RRPHA, seg_pos={0 0}, upload=1810]: Uploading piece...
Jul 26 23:49:51.068920439 (1h58m6.849853821s) [noise=true , address=77.90.31.16:4755 , piece_id=2DVYODA3WJFOS2J3X6CRY2G7TRVS334B5TT3LWM37JW3XZLTI7CA, seg_pos={0 0}, upload=1810]: Uploading piece...
Jul 26 23:49:51.069381010 (1h58m6.850314395s) [noise=true , address=104.36.82.47:4526 , piece_id=4VPISVBP743IMA7KYBFJHPJ7P4VVQ655KHN3BX7FMMY3DZTB2CGQ, seg_pos={0 0}, upload=1810]: Uploading piece...
Jul 26 23:49:51.069780164 (1h58m6.850713553s) [noise=true , address=104.36.82.52:4113 , piece_id=TREYS247R3PIIB7ASLO5VONA2HXMAFDWE7MD25X6VBZJ2SAI3QOQ, seg_pos={0 0}, upload=1810]: Uploading piece...
Jul 26 23:49:51.070273180 (1h58m6.851206569s) [noise=true , address=104.36.82.54:4159 , piece_id=S6CHS7J4KRSHHW3KSCOTM6LQSNQX6D5HG5IRBNKYOFRMOAWO2PBA, seg_pos={0 0}, upload=1810]: Uploading piece...
Jul 26 23:49:51.070718840 (1h58m6.851652223s) [noise=true , address=64.49.13.140:4508 , piece_id=IMWKZIRVHMQXP2QA2T5RHIKIEZHEMPTWKADFRQHVWKIFVC75IKKQ, seg_pos={0 0}, upload=1810]: Uploading piece...
Jul 26 23:49:51.151447319 (1h58m6.932380704s) [upload=1810]: Next returned nil segment
Jul 26 23:49:51.151466710 (1h58m6.932400107s) [upload=1810]: Waiting for error group managing segments...
Jul 26 23:49:51.189817686 (1h58m6.970751063s) [noise=true , address=79.127.219.38:4015 , piece_id=P6QFOWJINWHOZSHXMRESNEKHVJDC7LKQN7DZBOA3EVO2CVNBUHDQ, seg_pos={0 0}, upload=1810]: Done uploading piece. err: <nil>
Jul 26 23:49:51.194075321 (1h58m6.975008695s) [noise=true , address=77.90.31.15:4714 , piece_id=H66HQNWM2QUA36L2TWQBMBV62EJXUIT4BCXFPYRIMWQY3SOCQGZA, seg_pos={0 0}, upload=1810]: Done uploading piece. err: <nil>
Jul 26 23:49:51.280470482 (1h58m7.061403851s) [noise=true , address=77.90.31.11:4562 , piece_id=YBE3OL6TWY7ZKD4JKVA3SM3SJTIUOL2FQNBH76L43GR22ZANTZRA, seg_pos={0 0}, upload=1810]: Done uploading piece. err: <nil>
Jul 26 23:49:51.520768266 (1h58m7.301701636s) [noise=true , address=64.49.13.154:4927 , piece_id=HYJ622FJCHC3LSMCZS6GGTCCPX6P6UIFGYTVGGSPKRVXJMFDRWYQ, seg_pos={0 0}, upload=1810]: Done uploading piece. err: ecclient: upload failed (node:12U8eKLnBHxvuutyABVGSgUJbG3kr9Cce4sDx59LQiJHT1W2U8k, address:64.49.13.154:4927): protocol: storage node overloaded, request limit: 2000
Jul 26 23:49:51.664984244 (1h58m7.445917613s) [noise=true , address=79.127.205.245:4248 , piece_id=HWQLK367ZLE2OXRILUK53PSTJH6CPQ5VGR4L4OK2G7W43JHPFAXQ, seg_pos={0 0}, upload=1810]: Done uploading piece. err: <nil>
Jul 26 23:49:51.682657078 (1h58m7.463590448s) [noise=true , address=79.127.205.234:4609 , piece_id=BU23SDJE7IA5SWDWPQ4YACDNOM667EUJL7WQTPXKNFVKIWBK2PRA, seg_pos={0 0}, upload=1810]: Done uploading piece. err: <nil>
Jul 26 23:49:56.803533811 (1h58m12.584467174s) [noise=true , address=77.90.31.17:4787 , piece_id=GXREXTJ5NGAO3WGC2NECV32J4FLJMWKY662TAK2YWGZDEV5RRPHA, seg_pos={0 0}, upload=1810]: Done uploading piece. err: <nil>
Jul 26 23:49:58.348839446 (1h58m14.129772827s) [noise=true , address=64.49.13.140:4508 , piece_id=IMWKZIRVHMQXP2QA2T5RHIKIEZHEMPTWKADFRQHVWKIFVC75IKKQ, seg_pos={0 0}, upload=1810]: Done uploading piece. err: <nil>
Jul 26 23:50:01.403142927 (1h58m17.18407632s) [noise=true , address=45.91.116.9:4277 , piece_id=EGV63SXH2PZ2VSUWPWHYMRJSCSH65GYQWD3L5NCNCM6EGDXOHXMA, seg_pos={0 0}, upload=1810]: Done uploading piece. err: <nil>
Jul 26 23:50:25.435534628 (1h58m41.216468005s) [noise=true , address=79.127.201.215:4351 , piece_id=ZGV32GT6FLCDUF45JZLKJ7OMXH7RIJ5W7IINGHFTW33VBQEDMMOQ, seg_pos={0 0}, upload=1810]: Done uploading piece. err: <nil>
Jul 26 23:50:47.537991050 (1h59m3.318924424s) [noise=true , address=77.90.31.16:4755 , piece_id=2DVYODA3WJFOS2J3X6CRY2G7TRVS334B5TT3LWM37JW3XZLTI7CA, seg_pos={0 0}, upload=1810]: Done uploading piece. err: <nil>
Jul 26 23:50:50.996366693 (1h59m6.777300069s) [noise=true , address=104.36.82.47:4526 , piece_id=4VPISVBP743IMA7KYBFJHPJ7P4VVQ655KHN3BX7FMMY3DZTB2CGQ, seg_pos={0 0}, upload=1810]: Done uploading piece. err: ecclient: upload failed (node:1djnDo1bR9DZLEAHGJGoaxg895sQ3tQpYmPgGXpmsAxnpAtQwk, address:104.36.82.47:4526): protocol: rpc: dial tcp 104.36.82.47:4526: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 26 23:50:50.996541596 (1h59m6.777474978s) [noise=true , address=104.36.82.55:4224 , piece_id=V76WAOWB6BNQ3ZRBVDFMVCKU3RQE5QMIMMBJWFUWRRDGDXIWJCHQ, seg_pos={0 0}, upload=1810]: Done uploading piece. err: ecclient: upload failed (node:12EjKmUiv4pxkVCB4WoiXim4uFpxrBPyfiSS4efTY3HKSE3jb5y, address:104.36.82.55:4224): protocol: write tcp 159.223.143.39:43862->104.36.82.55:4224: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 26 23:50:50.996618248 (1h59m6.77755163s) [noise=true , address=104.36.82.52:4113 , piece_id=TREYS247R3PIIB7ASLO5VONA2HXMAFDWE7MD25X6VBZJ2SAI3QOQ, seg_pos={0 0}, upload=1810]: Done uploading piece. err: ecclient: upload failed (node:1brZ4RtcK5J3xCPmnbsTccF9hrnj8tqhV1WyMqqrjuitgSGMou, address:104.36.82.52:4113): protocol: write tcp 159.223.143.39:59156->104.36.82.52:4113: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 26 23:50:50.996658963 (1h59m6.777592341s) [noise=true , address=104.36.82.5:4870 , piece_id=XBPXX2Y2SSS2EAHRFRN6RCZDFOKIU5ECYSDSSMT6I2WTKOEU7JMA, seg_pos={0 0}, upload=1810]: Done uploading piece. err: ecclient: upload failed (node:1MmdhnDp6kKktFb5btZUDeVQ2u3CNUKCZU7RfuEqiSQ2TEqwKX, address:104.36.82.5:4870): protocol: rpc: dial tcp 104.36.82.5:4870: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 26 23:50:50.996728285 (1h59m6.777661664s) [noise=true , address=104.36.82.45:4910 , piece_id=2FPGI2PLU5S6Q5YDWSSDYT35OAKTCFHPUMRPHUMBDUU2PUOEH2YQ, seg_pos={0 0}, upload=1810]: Done uploading piece. err: ecclient: upload failed (node:12bRYzNggkq7QWFTP9fB94dTrMMX8AYvKMt2zjmJBzT1VwqyPVZ, address:104.36.82.45:4910): protocol: rpc: dial tcp 104.36.82.45:4910: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 26 23:50:50.996766567 (1h59m6.777699929s) [noise=true , address=104.36.82.54:4159 , piece_id=S6CHS7J4KRSHHW3KSCOTM6LQSNQX6D5HG5IRBNKYOFRMOAWO2PBA, seg_pos={0 0}, upload=1810]: Done uploading piece. err: ecclient: upload failed (node:12ULUPKjJwZocjV1R6Dy7vmWSrZ4hKx3YAHLAihgtMKf8Uw2UgA, address:104.36.82.54:4159): protocol: rpc: dial tcp 104.36.82.54:4159: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 26 23:50:50.996780048 (1h59m6.777713411s) [noise=true , address=104.36.82.46:4482 , piece_id=DZCSV5P6GZZFBY5G6X6GCTCBTWPWO6L47AGJP4F6D2TGCQDEBXVQ, seg_pos={0 0}, upload=1810]: Done uploading piece. err: ecclient: upload failed (node:12mNCKAdMzWu87fmyrcwDKZvscMcu2CogjXicwAi5wLSFJa7oVV, address:104.36.82.46:4482): protocol: rpc: dial tcp 104.36.82.46:4482: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 26 23:50:50.996803866 (1h59m6.777737256s) [seg_pos={0 0}, upload=1810]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil>
Jul 26 23:50:51.027692779 (1h59m6.808626147s) [upload=1810]: Done uploading segments...
Jul 26 23:50:51.027740654 (1h59m6.808674035s) [upload=1810]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10)
Duration: 1m0.033726328s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
19 out of 60 is failed (threshold=3) [.XXX.X....XX.X.X.X.......XXX...XX.....X.XXX...............X.] duration:
27.641707061s
FAIL (2025-07-26 23:50:46 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 27.641799369s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
13 out of 60 is failed (threshold=3) [X...X..XX.X...X..X.......X...X....XX..............X.X.......] duration:
35.104111093s
FAIL (2025-07-26 23:50:38 UTC)
context deadline exceeded
Duration: 35.10414243s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)antiblip:
54 out of 60 is failed (threshold=3) [.....XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.XXX] duration:
30.028116557s
FAIL (2025-07-26 23:50:30 UTC)
orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 10)
Duration: 30.02814762s
core
uplink
us1
vivint
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
51 out of 60 is failed (threshold=3) [........XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.XXXXXXXXXXXX] duration:
30.028644475s
FAIL (2025-07-26 23:50:14 UTC)
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 11)
======LOGS======
15.718809995s) [noise=true , address=104.36.82.6:4874 , piece_id=GQDCX4I2PWKO76IAUPFMKZ6P34CUCHAFKP6VG2DZTV2H33M2KATQ, seg_pos={0 0}, upload=1791]: Done uploading piece. err: ecclient: upload failed (node:12ay3ZqUvNcaX19JYT1hEaqydsxP6VoDjPavWXBNB34cestcm71, address:104.36.82.6:4874): protocol: rpc: dial tcp 104.36.82.6:4874: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 26 23:48:59.937906032 (1h57m15.718839395s) [seg_pos={0 0}, upload=1791]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil>
Jul 26 23:48:59.961543129 (1h57m15.742476505s) [upload=1791]: Done uploading segments...
Jul 26 23:48:59.961589585 (1h57m15.742522967s) [upload=1791]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10)
Jul 26 23:49:44.783843222 (1h58m0.564776602s) [upload=1806]: Starting upload
Jul 26 23:49:44.783927151 (1h58m0.564860514s) [upload=1806]: Done starting upload
Jul 26 23:49:44.784027248 (1h58m0.564960628s) [upload=1806]: Uploading segments...
Jul 26 23:49:44.784417545 (1h58m0.565350923s) [upload=1806]: Got next segment. Inline: false
Jul 26 23:49:44.784467247 (1h58m0.565400624s) [upload=1806]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jul 26 23:49:44.784475657 (1h58m0.565409037s) [upload=1806]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jul 26 23:49:44.828827465 (1h58m0.609760831s) [seg_pos={0 0}, upload=1806]: Begin upload segment...
Jul 26 23:49:44.828899983 (1h58m0.609833352s) [seg_pos={0 0}, upload=1806]: Done begin upload segment.
Jul 26 23:49:44.828957355 (1h58m0.609890742s) [noise=true , address=77.90.31.14:4662 , piece_id=XYGW55XYY3Q5UP3TWQAA6GFZDVSGKOQOK44RRLZNBM2ODSMXKK4Q, seg_pos={0 0}, upload=1806]: Uploading piece...
Jul 26 23:49:44.829492531 (1h58m0.610425918s) [noise=true , address=104.36.82.15:4259 , piece_id=IP2ME27ARN24X6SQIWJ4VJZO3DRT3QR5X62D3PXAGPKJSZLJJBJQ, seg_pos={0 0}, upload=1806]: Uploading piece...
Jul 26 23:49:44.829932411 (1h58m0.610865795s) [noise=true , address=45.91.116.7:4010 , piece_id=FPSXDJGHNIJF6RHZ4TPIW3B4O6VT3VHE2RFECVTJCTBJVT57Y52A, seg_pos={0 0}, upload=1806]: Uploading piece...
Jul 26 23:49:44.830521335 (1h58m0.611454714s) [noise=true , address=77.90.31.12:4592 , piece_id=UFA3FAYUMNPK6W54LHM3S5CL5L7DD5GOABZ6XRGEY4WLQMSJIELQ, seg_pos={0 0}, upload=1806]: Uploading piece...
Jul 26 23:49:44.830943966 (1h58m0.611877345s) [noise=true , address=104.36.82.11:4113 , piece_id=O3OLP35I72UTEBMZUCUDLKPMYCZ4VN7AQNP7W2Z5DPS3UCL4EZLQ, seg_pos={0 0}, upload=1806]: Uploading piece...
Jul 26 23:49:44.831135649 (1h58m0.612069031s) [noise=true , address=45.91.116.9:4262 , piece_id=K5NV5HLMNZFHB7PEOE2STOFG44GB3DS7V56BTOIHFXPY2MYREWBA, seg_pos={0 0}, upload=1806]: Uploading piece...
Jul 26 23:49:44.831364431 (1h58m0.612297812s) [noise=true , address=77.90.31.21:4879 , piece_id=VVX5RS3HRCFW46VDPA6EWS5XPX4VNRPNEO4OA6ZOOXIACC5Y7GUQ, seg_pos={0 0}, upload=1806]: Uploading piece...
Jul 26 23:49:44.831667623 (1h58m0.612600997s) [noise=true , address=77.90.31.11:4553 , piece_id=P65FDAFZYLPPATJ5JNP7KQ4MVOR7K4HIVZVDZPRI4I32NFIETDWQ, seg_pos={0 0}, upload=1806]: Uploading piece...
Jul 26 23:49:44.831770441 (1h58m0.612703828s) [noise=true , address=77.90.31.15:4698 , piece_id=BJ36WSTZOGQWMIDB6NNGPSS66J5QA7VEQM5ZIVGCKRPRR6OXYT2Q, seg_pos={0 0}, upload=1806]: Uploading piece...
Jul 26 23:49:44.832218793 (1h58m0.613152189s) [noise=true , address=104.36.82.5:4839 , piece_id=FOFV7KSLZQRGIYF34ROV7QWSN2NZTUVHP3CKKGTBIPKRIQ3T3PCA, seg_pos={0 0}, upload=1806]: Uploading piece...
Jul 26 23:49:44.832294596 (1h58m0.613227973s) [noise=true , address=45.91.116.13:4155 , piece_id=UHF77UOL55YIPUQROVVXOOEMRBKMTMOCOGQZSHDUYXUKSUVQBFSQ, seg_pos={0 0}, upload=1806]: Uploading piece...
Jul 26 23:49:44.832609944 (1h58m0.613543321s) [noise=true , address=104.36.82.15:4257 , piece_id=FYEG6JRAG6GX4OC3TJO553FLTXVX4JUHJU4VAFZHYCCOKOGA5QLA, seg_pos={0 0}, upload=1806]: Uploading piece...
Jul 26 23:49:44.832988392 (1h58m0.613921772s) [noise=true , address=45.91.116.8:4048 , piece_id=PCNJHK3FGP2PO2DSHSTIHJ6PGR4ERWS6XW3WTWWIURBYGX4RYF3A, seg_pos={0 0}, upload=1806]: Uploading piece...
Jul 26 23:49:44.833364868 (1h58m0.614298252s) [noise=true , address=104.36.82.6:4875 , piece_id=4GPNO2DZZ2HLNE4DGZVT2AKYTYP5ODGF7VIXAXPFBN7OGTVKM5TQ, seg_pos={0 0}, upload=1806]: Uploading piece...
Jul 26 23:49:44.833711978 (1h58m0.614645355s) [noise=true , address=77.90.31.14:4661 , piece_id=GJGHG6U6KNCISVHE6R74LZV6FKUO65T4O6ENJSFOEIDQ6QU2VLAA, seg_pos={0 0}, upload=1806]: Uploading piece...
Jul 26 23:49:44.834092083 (1h58m0.615025466s) [noise=true , address=45.91.116.6:4973 , piece_id=LOUDCZH4QNZ6HNKFPWOGD3DHJCG45L5LRTT3A2CN6BOVE5OI7SEA, seg_pos={0 0}, upload=1806]: Uploading piece...
Jul 26 23:49:44.929003034 (1h58m0.709936406s) [upload=1806]: Next returned nil segment
Jul 26 23:49:44.929023757 (1h58m0.709957121s) [upload=1806]: Waiting for error group managing segments...
Jul 26 23:49:44.933782221 (1h58m0.714715583s) [noise=true , address=45.91.116.6:4973 , piece_id=LOUDCZH4QNZ6HNKFPWOGD3DHJCG45L5LRTT3A2CN6BOVE5OI7SEA, seg_pos={0 0}, upload=1806]: Done uploading piece. err: <nil>
Jul 26 23:49:44.934086051 (1h58m0.715019421s) [noise=true , address=45.91.116.8:4048 , piece_id=PCNJHK3FGP2PO2DSHSTIHJ6PGR4ERWS6XW3WTWWIURBYGX4RYF3A, seg_pos={0 0}, upload=1806]: Done uploading piece. err: <nil>
Jul 26 23:49:44.934364421 (1h58m0.715297801s) [noise=true , address=77.90.31.21:4879 , piece_id=VVX5RS3HRCFW46VDPA6EWS5XPX4VNRPNEO4OA6ZOOXIACC5Y7GUQ, seg_pos={0 0}, upload=1806]: Done uploading piece. err: <nil>
Jul 26 23:49:44.934597398 (1h58m0.715530763s) [noise=true , address=45.91.116.7:4010 , piece_id=FPSXDJGHNIJF6RHZ4TPIW3B4O6VT3VHE2RFECVTJCTBJVT57Y52A, seg_pos={0 0}, upload=1806]: Done uploading piece. err: <nil>
Jul 26 23:49:44.934876966 (1h58m0.715810333s) [noise=true , address=77.90.31.14:4661 , piece_id=GJGHG6U6KNCISVHE6R74LZV6FKUO65T4O6ENJSFOEIDQ6QU2VLAA, seg_pos={0 0}, upload=1806]: Done uploading piece. err: <nil>
Jul 26 23:49:44.935305336 (1h58m0.716238703s) [noise=true , address=77.90.31.14:4662 , piece_id=XYGW55XYY3Q5UP3TWQAA6GFZDVSGKOQOK44RRLZNBM2ODSMXKK4Q, seg_pos={0 0}, upload=1806]: Done uploading piece. err: <nil>
Jul 26 23:49:44.935406366 (1h58m0.716339729s) [noise=true , address=45.91.116.13:4155 , piece_id=UHF77UOL55YIPUQROVVXOOEMRBKMTMOCOGQZSHDUYXUKSUVQBFSQ, seg_pos={0 0}, upload=1806]: Done uploading piece. err: <nil>
Jul 26 23:49:44.936035430 (1h58m0.716968796s) [noise=true , address=77.90.31.15:4698 , piece_id=BJ36WSTZOGQWMIDB6NNGPSS66J5QA7VEQM5ZIVGCKRPRR6OXYT2Q, seg_pos={0 0}, upload=1806]: Done uploading piece. err: <nil>
Jul 26 23:49:44.937906062 (1h58m0.71883943s) [noise=true , address=77.90.31.11:4553 , piece_id=P65FDAFZYLPPATJ5JNP7KQ4MVOR7K4HIVZVDZPRI4I32NFIETDWQ, seg_pos={0 0}, upload=1806]: Done uploading piece. err: <nil>
Jul 26 23:49:45.005296059 (1h58m0.786229488s) [noise=true , address=45.91.116.9:4262 , piece_id=K5NV5HLMNZFHB7PEOE2STOFG44GB3DS7V56BTOIHFXPY2MYREWBA, seg_pos={0 0}, upload=1806]: Done uploading piece. err: <nil>
Jul 26 23:49:46.348890347 (1h58m2.129823708s) [noise=true , address=77.90.31.12:4592 , piece_id=UFA3FAYUMNPK6W54LHM3S5CL5L7DD5GOABZ6XRGEY4WLQMSJIELQ, seg_pos={0 0}, upload=1806]: Done uploading piece. err: <nil>
Jul 26 23:50:14.749321210 (1h58m30.530254587s) [noise=true , address=104.36.82.6:4875 , piece_id=4GPNO2DZZ2HLNE4DGZVT2AKYTYP5ODGF7VIXAXPFBN7OGTVKM5TQ, seg_pos={0 0}, upload=1806]: Done uploading piece. err: ecclient: upload failed (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2, address:104.36.82.6:4875): protocol: rpc: dial tcp 104.36.82.6:4875: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 26 23:50:14.749380296 (1h58m30.530313663s) [noise=true , address=104.36.82.15:4259 , piece_id=IP2ME27ARN24X6SQIWJ4VJZO3DRT3QR5X62D3PXAGPKJSZLJJBJQ, seg_pos={0 0}, upload=1806]: Done uploading piece. err: ecclient: upload failed (node:1ET8PaAfWq2RcPNjucMqDbmnkMZg8qnkTw9dszwLzz765oBr7y, address:104.36.82.15:4259): protocol: rpc: dial tcp 104.36.82.15:4259: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 26 23:50:14.749459499 (1h58m30.530392869s) [noise=true , address=104.36.82.15:4257 , piece_id=FYEG6JRAG6GX4OC3TJO553FLTXVX4JUHJU4VAFZHYCCOKOGA5QLA, seg_pos={0 0}, upload=1806]: Done uploading piece. err: ecclient: upload failed (node:12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK, address:104.36.82.15:4257): protocol: rpc: dial tcp 104.36.82.15:4257: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 26 23:50:14.749475904 (1h58m30.530409282s) [noise=true , address=104.36.82.5:4839 , piece_id=FOFV7KSLZQRGIYF34ROV7QWSN2NZTUVHP3CKKGTBIPKRIQ3T3PCA, seg_pos={0 0}, upload=1806]: Done uploading piece. err: ecclient: upload failed (node:1FcPS8yXfgdFvsxFofPrNCtFDLtebQMTaLu75335kVXeQxttuv, address:104.36.82.5:4839): protocol: write tcp 159.223.143.39:43694->104.36.82.5:4839: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 26 23:50:14.749526216 (1h58m30.530459584s) [noise=true , address=104.36.82.11:4113 , piece_id=O3OLP35I72UTEBMZUCUDLKPMYCZ4VN7AQNP7W2Z5DPS3UCL4EZLQ, seg_pos={0 0}, upload=1806]: Done uploading piece. err: ecclient: upload failed (node:1SQKETWozrz4PcDC3yhy6V5d8Ryo6NNiyWWEkE5xRcmUj9B6UV, address:104.36.82.11:4113): protocol: write tcp 159.223.143.39:43682->104.36.82.11:4113: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 26 23:50:14.749563181 (1h58m30.53049656s) [seg_pos={0 0}, upload=1806]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil>
Jul 26 23:50:14.774834627 (1h58m30.555768005s) [upload=1806]: Done uploading segments...
Jul 26 23:50:14.774880812 (1h58m30.555814191s) [upload=1806]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11)
Duration: 30.028673544s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
19 out of 60 is failed (threshold=3) [..XXX.X....XX.X.X.X.......XXX...XX.....X.XXX...............X] duration:
30.002197301s
FAIL (2025-07-26 23:50:00 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 30.002301338s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)antiblip:
53 out of 60 is failed (threshold=3) [X....XXXX.XXXXX.X.XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration:
1m0.029019682s
FAIL (2025-07-26 23:49:50 UTC)
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 9)
======LOGS======
1h57m6.739595004s) [noise=true , address=45.91.116.14:4206 , piece_id=X4OAWUEJY3QS45ZVQAHOCYN5ZQOR5ZMYR443QXRRSLT3H4VLI4AQ, seg_pos={0 0}, upload=1797]: Uploading piece...
Jul 26 23:48:50.958693285 (1h57m6.739626669s) [noise=true , address=77.90.31.19:4847 , piece_id=IHRYNTEY5SZRTUFVUFTEWKFR2SVHYLSAD3HYGVU6L2SHWSJUIHAA, seg_pos={0 0}, upload=1797]: Uploading piece...
Jul 26 23:48:50.959149382 (1h57m6.740082765s) [noise=true , address=64.49.13.139:4478 , piece_id=ISTDHHKCQ6EF7BRSQUHJLGZSBPPRCASFCWWFFZYYTON4MKJTTKZA, seg_pos={0 0}, upload=1797]: Uploading piece...
Jul 26 23:48:50.959265162 (1h57m6.740198551s) [noise=true , address=104.36.82.47:4543 , piece_id=ABZTEPO5KGDEBMQ3UWCSAY3D6JRRQAKQWBZRNLHJZKONULZPSW4A, seg_pos={0 0}, upload=1797]: Uploading piece...
Jul 26 23:48:50.959543419 (1h57m6.740476804s) [noise=true , address=79.127.201.215:4351 , piece_id=YIJLPXVXGF2D5WIGTQDV7EKH6AFESBCY5NYQ3S4P66BOGDZGKUYA, seg_pos={0 0}, upload=1797]: Uploading piece...
Jul 26 23:48:50.959832987 (1h57m6.740766372s) [noise=true , address=77.90.31.11:4576 , piece_id=2DMCVAKN6J2YQAOVEZR4S3CN7PF44MSOOG7BYMUWKV3XFQDFI7PQ, seg_pos={0 0}, upload=1797]: Uploading piece...
Jul 26 23:48:50.959904226 (1h57m6.740837611s) [noise=true , address=104.36.82.52:4090 , piece_id=WCOJD7FO6YJHAP4R637EXSQU44BLHTDOUO5EAFMTVPQI7GZG7RRQ, seg_pos={0 0}, upload=1797]: Uploading piece...
Jul 26 23:48:50.960384541 (1h57m6.741317922s) [noise=true , address=104.36.82.54:4160 , piece_id=ODTJMEQZWBCGFHQNQ72XGQXN2M3NL3N2B2LRTHD7XOUVIMY7BIEQ, seg_pos={0 0}, upload=1797]: Uploading piece...
Jul 26 23:48:50.960451153 (1h57m6.74138454s) [noise=true , address=104.36.82.56:4256 , piece_id=JJBUSYWPGBCIE3OBVMXBESRF626KV4NJCI2GTNEAPXJPTVIWGJLQ, seg_pos={0 0}, upload=1797]: Uploading piece...
Jul 26 23:48:50.960758939 (1h57m6.741692324s) [noise=true , address=77.90.31.17:4792 , piece_id=BMPWWVSWPTNBVAAMSIW634HRJMK5TKHTXCV4OF6H4RAJ7PKPVNWA, seg_pos={0 0}, upload=1797]: Uploading piece...
Jul 26 23:48:50.960957186 (1h57m6.741890574s) [noise=true , address=64.49.13.140:4508 , piece_id=6R2AGSOK7DH3JW3VHLICW6AMHHMEN2GZ6XA7WH5NAOZ2AC4TKEPQ, seg_pos={0 0}, upload=1797]: Uploading piece...
Jul 26 23:48:50.961128404 (1h57m6.742061786s) [noise=true , address=77.90.31.10:4533 , piece_id=2JYCWGX5GS2AXPH34WXSMGXBVLFHWL44P2XTLOXIXSSLDV7CGMYA, seg_pos={0 0}, upload=1797]: Uploading piece...
Jul 26 23:48:50.961427257 (1h57m6.742360641s) [noise=true , address=79.127.219.51:4210 , piece_id=CC5CHBXUKQLGSSPBGIXA65PGMMWHFV2SPRVBA2MUU2O5KMISQ2BA, seg_pos={0 0}, upload=1797]: Uploading piece...
Jul 26 23:48:50.961442268 (1h57m6.742375645s) [noise=true , address=104.36.82.46:4502 , piece_id=W3UD5L4UKN5QN455GL6FWVXRQR24P4DOSQ3IV6HDYUHAIDPNLWDQ, seg_pos={0 0}, upload=1797]: Uploading piece...
Jul 26 23:48:50.961932144 (1h57m6.74286552s) [noise=true , address=64.49.13.154:4927 , piece_id=AFYLWX5GM2YKFBFCVH5CGDBXNHAC63FGDG6HCGPMWO34DFNQC54Q, seg_pos={0 0}, upload=1797]: Uploading piece...
Jul 26 23:48:50.962412839 (1h57m6.743346239s) [noise=true , address=45.91.116.9:4277 , piece_id=WQGSIP6V3AD3LJGC6EGS7GCWVXNDXV5MXLBQ35GBQVTBHTMILU7A, seg_pos={0 0}, upload=1797]: Uploading piece...
Jul 26 23:48:50.962831440 (1h57m6.743764832s) [noise=true , address=104.36.82.53:4142 , piece_id=MVNJG7XUS56IPKO3QSJU7MULKLFO2JAFZHQJTSVSMJVA3VWEZY3A, seg_pos={0 0}, upload=1797]: Uploading piece...
Jul 26 23:48:50.964832151 (1h57m6.74576552s) [noise=true , address=64.49.13.137:4502 , piece_id=GAZZKUWSRW6MNOCVGDP7IHI2SJW3KN5TH4R26FRRQZ2CHFXMXJXQ, seg_pos={0 0}, upload=1797]: Uploading piece...
Jul 26 23:48:50.992825199 (1h57m6.773758575s) [noise=true , address=77.90.31.19:4847 , piece_id=IHRYNTEY5SZRTUFVUFTEWKFR2SVHYLSAD3HYGVU6L2SHWSJUIHAA, seg_pos={0 0}, upload=1797]: Done uploading piece. err: ecclient: upload failed (node:12RkRJoWfBB5tUsZScQgydJhY5XffVuQn1JUa1ZHThjcP3bUfJv, address:77.90.31.19:4847): protocol: storage node overloaded, request limit: 2000
Jul 26 23:48:51.026699454 (1h57m6.807632823s) [upload=1797]: Next returned nil segment
Jul 26 23:48:51.026720993 (1h57m6.807654367s) [upload=1797]: Waiting for error group managing segments...
Jul 26 23:48:51.032996258 (1h57m6.813929627s) [noise=true , address=77.90.31.10:4533 , piece_id=2JYCWGX5GS2AXPH34WXSMGXBVLFHWL44P2XTLOXIXSSLDV7CGMYA, seg_pos={0 0}, upload=1797]: Done uploading piece. err: <nil>
Jul 26 23:48:51.071786016 (1h57m6.852719395s) [noise=true , address=77.90.31.11:4576 , piece_id=2DMCVAKN6J2YQAOVEZR4S3CN7PF44MSOOG7BYMUWKV3XFQDFI7PQ, seg_pos={0 0}, upload=1797]: Done uploading piece. err: <nil>
Jul 26 23:48:51.107847016 (1h57m6.888780392s) [noise=true , address=77.90.31.17:4792 , piece_id=BMPWWVSWPTNBVAAMSIW634HRJMK5TKHTXCV4OF6H4RAJ7PKPVNWA, seg_pos={0 0}, upload=1797]: Done uploading piece. err: <nil>
Jul 26 23:48:51.663534169 (1h57m7.444467553s) [noise=true , address=64.49.13.137:4502 , piece_id=GAZZKUWSRW6MNOCVGDP7IHI2SJW3KN5TH4R26FRRQZ2CHFXMXJXQ, seg_pos={0 0}, upload=1797]: Done uploading piece. err: <nil>
Jul 26 23:48:52.608569918 (1h57m8.389503289s) [noise=true , address=79.127.219.51:4210 , piece_id=CC5CHBXUKQLGSSPBGIXA65PGMMWHFV2SPRVBA2MUU2O5KMISQ2BA, seg_pos={0 0}, upload=1797]: Done uploading piece. err: <nil>
Jul 26 23:48:52.659046744 (1h57m8.439980119s) [noise=true , address=64.49.13.154:4927 , piece_id=AFYLWX5GM2YKFBFCVH5CGDBXNHAC63FGDG6HCGPMWO34DFNQC54Q, seg_pos={0 0}, upload=1797]: Done uploading piece. err: ecclient: upload failed (node:12U8eKLnBHxvuutyABVGSgUJbG3kr9Cce4sDx59LQiJHT1W2U8k, address:64.49.13.154:4927): protocol: storage node overloaded, request limit: 2000
Jul 26 23:48:53.697277228 (1h57m9.478210607s) [noise=true , address=64.49.13.140:4508 , piece_id=6R2AGSOK7DH3JW3VHLICW6AMHHMEN2GZ6XA7WH5NAOZ2AC4TKEPQ, seg_pos={0 0}, upload=1797]: Done uploading piece. err: ecclient: upload failed (node:1diUU2skKzXPjYy3Mu3iDDjnG5W8Z2cxCRMxDCH9cQGg2TdEiR, address:64.49.13.140:4508): protocol: storage node overloaded, request limit: 2000
Jul 26 23:48:54.224036958 (1h57m10.004970335s) [noise=true , address=45.91.116.14:4206 , piece_id=X4OAWUEJY3QS45ZVQAHOCYN5ZQOR5ZMYR443QXRRSLT3H4VLI4AQ, seg_pos={0 0}, upload=1797]: Done uploading piece. err: <nil>
Jul 26 23:48:55.495228481 (1h57m11.276161862s) [noise=true , address=64.49.13.139:4478 , piece_id=ISTDHHKCQ6EF7BRSQUHJLGZSBPPRCASFCWWFFZYYTON4MKJTTKZA, seg_pos={0 0}, upload=1797]: Done uploading piece. err: <nil>
Jul 26 23:49:06.295320598 (1h57m22.076253981s) [noise=true , address=45.91.116.9:4277 , piece_id=WQGSIP6V3AD3LJGC6EGS7GCWVXNDXV5MXLBQ35GBQVTBHTMILU7A, seg_pos={0 0}, upload=1797]: Done uploading piece. err: <nil>
Jul 26 23:49:20.248961587 (1h57m36.029894955s) [noise=true , address=79.127.201.215:4351 , piece_id=YIJLPXVXGF2D5WIGTQDV7EKH6AFESBCY5NYQ3S4P66BOGDZGKUYA, seg_pos={0 0}, upload=1797]: Done uploading piece. err: <nil>
Jul 26 23:49:50.886255050 (1h58m6.66718844s) [noise=true , address=104.36.82.53:4142 , piece_id=MVNJG7XUS56IPKO3QSJU7MULKLFO2JAFZHQJTSVSMJVA3VWEZY3A, seg_pos={0 0}, upload=1797]: Done uploading piece. err: ecclient: upload failed (node:1yLPWJs1RfZgFisHUpW3wwiFsvV98yz7adKtAKAK3unq1KZCSK, address:104.36.82.53:4142): protocol: rpc: dial tcp 104.36.82.53:4142: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 26 23:49:50.886337272 (1h58m6.667270648s) [noise=true , address=104.36.82.52:4090 , piece_id=WCOJD7FO6YJHAP4R637EXSQU44BLHTDOUO5EAFMTVPQI7GZG7RRQ, seg_pos={0 0}, upload=1797]: Done uploading piece. err: ecclient: upload failed (node:12UUnoD9p5FC7rq9fMSzu1phkFpsnPsu4UYxC59MhpMjJ6wUC6F, address:104.36.82.52:4090): protocol: write tcp 159.223.143.39:59390->104.36.82.52:4090: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 26 23:49:50.886412042 (1h58m6.667345411s) [noise=true , address=104.36.82.54:4160 , piece_id=ODTJMEQZWBCGFHQNQ72XGQXN2M3NL3N2B2LRTHD7XOUVIMY7BIEQ, seg_pos={0 0}, upload=1797]: Done uploading piece. err: ecclient: upload failed (node:12qRsRx7vmb3S9m5J7gceK44gnuuYYMyorSAyxcpjpPeRJJFZ84, address:104.36.82.54:4160): protocol: rpc: dial tcp 104.36.82.54:4160: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 26 23:49:50.886447846 (1h58m6.667381211s) [noise=true , address=104.36.82.56:4256 , piece_id=JJBUSYWPGBCIE3OBVMXBESRF626KV4NJCI2GTNEAPXJPTVIWGJLQ, seg_pos={0 0}, upload=1797]: Done uploading piece. err: ecclient: upload failed (node:11N1BR6rgiYye7CnuM5xDJogUXoUp4FgXrXM41GdhfrRCbbgT8, address:104.36.82.56:4256): protocol: rpc: dial tcp 104.36.82.56:4256: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 26 23:49:50.886491267 (1h58m6.667424633s) [noise=true , address=104.36.82.47:4543 , piece_id=ABZTEPO5KGDEBMQ3UWCSAY3D6JRRQAKQWBZRNLHJZKONULZPSW4A, seg_pos={0 0}, upload=1797]: Done uploading piece. err: ecclient: upload failed (node:12KxnBTVxhbwA79Gd8V7Mh6bWybtqttqgo5agHoHewXHAey5y7D, address:104.36.82.47:4543): protocol: write tcp 159.223.143.39:45114->104.36.82.47:4543: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 26 23:49:50.886511678 (1h58m6.667445044s) [noise=true , address=104.36.82.46:4502 , piece_id=W3UD5L4UKN5QN455GL6FWVXRQR24P4DOSQ3IV6HDYUHAIDPNLWDQ, seg_pos={0 0}, upload=1797]: Done uploading piece. err: ecclient: upload failed (node:12HP5xuRAGgTsL11UQh7VRgfuFzwNdEMXgc8zPVv4QknStyv6Ta, address:104.36.82.46:4502): protocol: write tcp 159.223.143.39:49514->104.36.82.46:4502: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 26 23:49:50.886566922 (1h58m6.667500299s) [seg_pos={0 0}, upload=1797]: Done waiting for segment. successful: 9 optimal: 14 errs: <nil>
Jul 26 23:49:50.912933369 (1h58m6.693866744s) [upload=1797]: Done uploading segments...
Jul 26 23:49:50.912980684 (1h58m6.693914066s) [upload=1797]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 9)
Duration: 1m0.029042668s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
13 out of 60 is failed (threshold=3) [.X...X..XX.X...X..X.......X...X....XX..............X.X......] duration:
29.105241524s
FAIL (2025-07-26 23:49:29 UTC)
context deadline exceeded
Duration: 29.105268146s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)antiblip:
53 out of 60 is failed (threshold=3) [......XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.XX] duration:
30.032334886s
FAIL (2025-07-26 23:49:27 UTC)
orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 9)
Duration: 30.032367731s
core
uplink
us1
vivint
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
50 out of 60 is failed (threshold=3) [.........XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.XXXXXXXXXXX] duration:
30.026370872s
FAIL (2025-07-26 23:48:59 UTC)
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 10)
======LOGS======
iled (node:12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK, address:104.36.82.15:4257): protocol: write tcp 159.223.143.39:48126->104.36.82.15:4257: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 26 23:48:07.498548008 (1h56m23.279481386s) [seg_pos={0 0}, upload=1777]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil>
Jul 26 23:48:07.526079352 (1h56m23.307012724s) [upload=1777]: Done uploading segments...
Jul 26 23:48:07.526127435 (1h56m23.30706082s) [upload=1777]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8)
Jul 26 23:48:29.973052621 (1h56m45.753985989s) [upload=1791]: Starting upload
Jul 26 23:48:29.973154855 (1h56m45.754088241s) [upload=1791]: Done starting upload
Jul 26 23:48:29.973294734 (1h56m45.7542281s) [upload=1791]: Uploading segments...
Jul 26 23:48:29.973657027 (1h56m45.754590393s) [upload=1791]: Got next segment. Inline: false
Jul 26 23:48:29.973721230 (1h56m45.754654598s) [upload=1791]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jul 26 23:48:29.973728168 (1h56m45.75466153s) [upload=1791]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jul 26 23:48:30.001024890 (1h56m45.781958302s) [seg_pos={0 0}, upload=1791]: Begin upload segment...
Jul 26 23:48:30.001084651 (1h56m45.78201805s) [seg_pos={0 0}, upload=1791]: Done begin upload segment.
Jul 26 23:48:30.001117532 (1h56m45.782050935s) [noise=true , address=45.91.116.11:4117 , piece_id=D34BCNYYKIPKEOVK7NVX2FLZJL7HMK6BZNAIRE2MIG4OZ2F4J7MA, seg_pos={0 0}, upload=1791]: Uploading piece...
Jul 26 23:48:30.001651152 (1h56m45.782584553s) [noise=true , address=45.91.116.13:4153 , piece_id=GTVBZEX6OHDVD6HYQR3G6TR55V4YOQIGGMSTLSB56S33HJ2ETV5A, seg_pos={0 0}, upload=1791]: Uploading piece...
Jul 26 23:48:30.002016296 (1h56m45.78294969s) [noise=true , address=45.91.116.6:4973 , piece_id=SPO3XWPX5FUJFFSOEW5ARV2INMOS24APEZZWNYAOQDV6IH6MD2CQ, seg_pos={0 0}, upload=1791]: Uploading piece...
Jul 26 23:48:30.002367862 (1h56m45.783301264s) [noise=true , address=104.36.82.14:4221 , piece_id=HBYBZTSMBVVJWDDWOMFQCMIWAQJGSG2YLGHL22NVRR5BBG4WAB2A, seg_pos={0 0}, upload=1791]: Uploading piece...
Jul 26 23:48:30.002688708 (1h56m45.783622086s) [noise=true , address=104.36.82.8:4971 , piece_id=L3BE65HJ3WTOT7ZIYD5UUSWO5C5JS4UX7LBYA45KJBLJYRHML3ZQ, seg_pos={0 0}, upload=1791]: Uploading piece...
Jul 26 23:48:30.002989049 (1h56m45.783922431s) [noise=true , address=45.91.116.12:4227 , piece_id=C266SVGBNE4QKDPFAMPWNAJX4GOLGNRQWUP2SUX6CCMSNI3AVEIQ, seg_pos={0 0}, upload=1791]: Uploading piece...
Jul 26 23:48:30.003351875 (1h56m45.784285256s) [noise=true , address=45.91.116.10:4081 , piece_id=OTTOIN5DEVRQYWMJ4VF54US7IVBECSJZU5OSV22XKRVXMXZP6DRA, seg_pos={0 0}, upload=1791]: Uploading piece...
Jul 26 23:48:30.005317512 (1h56m45.786250893s) [noise=true , address=104.36.82.6:4874 , piece_id=GQDCX4I2PWKO76IAUPFMKZ6P34CUCHAFKP6VG2DZTV2H33M2KATQ, seg_pos={0 0}, upload=1791]: Uploading piece...
Jul 26 23:48:30.005821370 (1h56m45.78675475s) [noise=true , address=45.91.116.8:4045 , piece_id=V33YGB2H5OGN2O4IE4M5NJV4F5AUVTFHGOUD2BHUMEOBFX6VYVNA, seg_pos={0 0}, upload=1791]: Uploading piece...
Jul 26 23:48:30.006147011 (1h56m45.78708039s) [noise=true , address=104.36.82.9:4006 , piece_id=AEBIZNESPA42354VZIGS3EGUZKDKZ5EAYBBTKDHRJF7A6RHHUL4A, seg_pos={0 0}, upload=1791]: Uploading piece...
Jul 26 23:48:30.006457281 (1h56m45.787390664s) [noise=true , address=45.91.116.8:4048 , piece_id=SJ5LEGP3XQJ32O5HVB5XDYJHG32BWNVUZFTW3GL5UBBHFOW57UQA, seg_pos={0 0}, upload=1791]: Uploading piece...
Jul 26 23:48:30.006754649 (1h56m45.787688032s) [noise=true , address=45.91.116.5:4298 , piece_id=4OFZU66WCOPNSULSKZUZODLS4AIWYBYRBVQVL35SP3L6FCNBN2AA, seg_pos={0 0}, upload=1791]: Uploading piece...
Jul 26 23:48:30.007096765 (1h56m45.788030148s) [noise=true , address=77.90.31.14:4663 , piece_id=TQZDFBBCYQUIBKJTFCQ5TJXCGRPAZJRFGU5SELPWN2XTE4Y3BMTA, seg_pos={0 0}, upload=1791]: Uploading piece...
Jul 26 23:48:30.007491917 (1h56m45.788425299s) [noise=true , address=45.91.116.7:4010 , piece_id=5UL3B7UEM6EPYHU7AGMWSHU6TAGRDBNEU25MX4OUC4LZPP4OPBMA, seg_pos={0 0}, upload=1791]: Uploading piece...
Jul 26 23:48:30.009584814 (1h56m45.790518222s) [noise=true , address=104.36.82.3:4765 , piece_id=UGUHNPRLC56BYOYDFI466SYFRCFEDBUD4ICO2DVKI3CPO7QAQNUQ, seg_pos={0 0}, upload=1791]: Uploading piece...
Jul 26 23:48:30.010187976 (1h56m45.791121383s) [noise=true , address=104.36.82.13:4187 , piece_id=R6CHN46X2IXV3KMYOED2O7PP34TSB45ZWPBJHWNZT4PYHF7LECZQ, seg_pos={0 0}, upload=1791]: Uploading piece...
Jul 26 23:48:30.274819250 (1h56m46.055752635s) [upload=1791]: Next returned nil segment
Jul 26 23:48:30.274844994 (1h56m46.055778383s) [upload=1791]: Waiting for error group managing segments...
Jul 26 23:48:30.282908830 (1h56m46.063842198s) [noise=true , address=45.91.116.5:4298 , piece_id=4OFZU66WCOPNSULSKZUZODLS4AIWYBYRBVQVL35SP3L6FCNBN2AA, seg_pos={0 0}, upload=1791]: Done uploading piece. err: <nil>
Jul 26 23:48:30.284035898 (1h56m46.064969266s) [noise=true , address=45.91.116.10:4081 , piece_id=OTTOIN5DEVRQYWMJ4VF54US7IVBECSJZU5OSV22XKRVXMXZP6DRA, seg_pos={0 0}, upload=1791]: Done uploading piece. err: <nil>
Jul 26 23:48:30.285029686 (1h56m46.065963049s) [noise=true , address=45.91.116.8:4045 , piece_id=V33YGB2H5OGN2O4IE4M5NJV4F5AUVTFHGOUD2BHUMEOBFX6VYVNA, seg_pos={0 0}, upload=1791]: Done uploading piece. err: <nil>
Jul 26 23:48:30.285997064 (1h56m46.066930428s) [noise=true , address=45.91.116.12:4227 , piece_id=C266SVGBNE4QKDPFAMPWNAJX4GOLGNRQWUP2SUX6CCMSNI3AVEIQ, seg_pos={0 0}, upload=1791]: Done uploading piece. err: <nil>
Jul 26 23:48:30.286768904 (1h56m46.067702273s) [noise=true , address=45.91.116.8:4048 , piece_id=SJ5LEGP3XQJ32O5HVB5XDYJHG32BWNVUZFTW3GL5UBBHFOW57UQA, seg_pos={0 0}, upload=1791]: Done uploading piece. err: <nil>
Jul 26 23:48:30.287375432 (1h56m46.068308797s) [noise=true , address=77.90.31.14:4663 , piece_id=TQZDFBBCYQUIBKJTFCQ5TJXCGRPAZJRFGU5SELPWN2XTE4Y3BMTA, seg_pos={0 0}, upload=1791]: Done uploading piece. err: <nil>
Jul 26 23:48:30.287978896 (1h56m46.068912263s) [noise=true , address=45.91.116.6:4973 , piece_id=SPO3XWPX5FUJFFSOEW5ARV2INMOS24APEZZWNYAOQDV6IH6MD2CQ, seg_pos={0 0}, upload=1791]: Done uploading piece. err: <nil>
Jul 26 23:48:30.288573535 (1h56m46.069506915s) [noise=true , address=45.91.116.13:4153 , piece_id=GTVBZEX6OHDVD6HYQR3G6TR55V4YOQIGGMSTLSB56S33HJ2ETV5A, seg_pos={0 0}, upload=1791]: Done uploading piece. err: <nil>
Jul 26 23:48:30.292985744 (1h56m46.073919125s) [noise=true , address=45.91.116.7:4010 , piece_id=5UL3B7UEM6EPYHU7AGMWSHU6TAGRDBNEU25MX4OUC4LZPP4OPBMA, seg_pos={0 0}, upload=1791]: Done uploading piece. err: <nil>
Jul 26 23:48:30.297040331 (1h56m46.077973697s) [noise=true , address=45.91.116.11:4117 , piece_id=D34BCNYYKIPKEOVK7NVX2FLZJL7HMK6BZNAIRE2MIG4OZ2F4J7MA, seg_pos={0 0}, upload=1791]: Done uploading piece. err: <nil>
Jul 26 23:48:59.937478845 (1h57m15.718412224s) [noise=true , address=104.36.82.14:4221 , piece_id=HBYBZTSMBVVJWDDWOMFQCMIWAQJGSG2YLGHL22NVRR5BBG4WAB2A, seg_pos={0 0}, upload=1791]: Done uploading piece. err: ecclient: upload failed (node:129qMN1kX8C3x3H1Epfc4pUbtVGU1xPiZj43QSLByCTPKqwUoJR, address:104.36.82.14:4221): protocol: rpc: dial tcp 104.36.82.14:4221: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 26 23:48:59.937695728 (1h57m15.718629107s) [noise=true , address=104.36.82.3:4765 , piece_id=UGUHNPRLC56BYOYDFI466SYFRCFEDBUD4ICO2DVKI3CPO7QAQNUQ, seg_pos={0 0}, upload=1791]: Done uploading piece. err: ecclient: upload failed (node:1b1zhuhfXoTexYZUxKUAdCNhWJD8oGC17uXKRykA3gErEEMPWV, address:104.36.82.3:4765): protocol: rpc: dial tcp 104.36.82.3:4765: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 26 23:48:59.937763042 (1h57m15.718696419s) [noise=true , address=104.36.82.9:4006 , piece_id=AEBIZNESPA42354VZIGS3EGUZKDKZ5EAYBBTKDHRJF7A6RHHUL4A, seg_pos={0 0}, upload=1791]: Done uploading piece. err: ecclient: upload failed (node:1ig3zGgKZUH8jwXEojp2A6tYGVFCL33PUDkhAbEjS6affBLtmc, address:104.36.82.9:4006): protocol: context deadline exceeded; context deadline exceeded
Jul 26 23:48:59.937839038 (1h57m15.718772415s) [noise=true , address=104.36.82.13:4187 , piece_id=R6CHN46X2IXV3KMYOED2O7PP34TSB45ZWPBJHWNZT4PYHF7LECZQ, seg_pos={0 0}, upload=1791]: Done uploading piece. err: ecclient: upload failed (node:12cQxBB2L2CyymYx3xWp1ALj6p8QWr4A34sqvtG5zyeT7sVWv7x, address:104.36.82.13:4187): protocol: write tcp 159.223.143.39:38866->104.36.82.13:4187: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 26 23:48:59.937855239 (1h57m15.718788619s) [noise=true , address=104.36.82.8:4971 , piece_id=L3BE65HJ3WTOT7ZIYD5UUSWO5C5JS4UX7LBYA45KJBLJYRHML3ZQ, seg_pos={0 0}, upload=1791]: Done uploading piece. err: ecclient: upload failed (node:12Ut4J9xtQTDXFFmCSv9rk6sBSjNpEdnyxo6cZWMXcpAvuQtebo, address:104.36.82.8:4971): protocol: rpc: dial tcp 104.36.82.8:4971: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 26 23:48:59.937876625 (1h57m15.718809995s) [noise=true , address=104.36.82.6:4874 , piece_id=GQDCX4I2PWKO76IAUPFMKZ6P34CUCHAFKP6VG2DZTV2H33M2KATQ, seg_pos={0 0}, upload=1791]: Done uploading piece. err: ecclient: upload failed (node:12ay3ZqUvNcaX19JYT1hEaqydsxP6VoDjPavWXBNB34cestcm71, address:104.36.82.6:4874): protocol: rpc: dial tcp 104.36.82.6:4874: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 26 23:48:59.937906032 (1h57m15.718839395s) [seg_pos={0 0}, upload=1791]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil>
Jul 26 23:48:59.961543129 (1h57m15.742476505s) [upload=1791]: Done uploading segments...
Jul 26 23:48:59.961589585 (1h57m15.742522967s) [upload=1791]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10)
Duration: 30.02639934s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)antiblip:
53 out of 60 is failed (threshold=3) [X......XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.X] duration:
30.031329045s
FAIL (2025-07-26 23:48:52 UTC)
orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 9)
Duration: 30.031352418s
core
uplink
us1
vivint
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)antiblip:
53 out of 60 is failed (threshold=3) [XX....XXXX.XXXXX.X.XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration:
1m0.026877216s
FAIL (2025-07-26 23:48:50 UTC)
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 12)
======LOGS======
ts...
Jul 26 23:47:50.818338951 (1h56m6.599272339s) [upload=1781]: Got next segment. Inline: false
Jul 26 23:47:50.818427169 (1h56m6.599360546s) [upload=1781]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jul 26 23:47:50.818435467 (1h56m6.599368852s) [upload=1781]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jul 26 23:47:50.852575146 (1h56m6.633508512s) [seg_pos={0 0}, upload=1781]: Begin upload segment...
Jul 26 23:47:50.852632775 (1h56m6.633566153s) [seg_pos={0 0}, upload=1781]: Done begin upload segment.
Jul 26 23:47:50.852672162 (1h56m6.63360554s) [noise=true , address=77.90.31.16:4747 , piece_id=HRMAPCAJ4Q7JQPTALZQCHSKI43P6Q54D45IZJQXT6YP5U5FFOKRA, seg_pos={0 0}, upload=1781]: Uploading piece...
Jul 26 23:47:50.853143012 (1h56m6.634076403s) [noise=true , address=104.36.82.55:4198 , piece_id=SXGZYS3YKOZC5GJOOOI6BBWUVIKM4NE25C3GVUXNLEP5STQPIC5A, seg_pos={0 0}, upload=1781]: Uploading piece...
Jul 26 23:47:50.853673148 (1h56m6.634606532s) [noise=true , address=104.36.82.53:4138 , piece_id=RDOXJNWARUJ2Q5FWGQ7XNMPYSAQMRYWTECPSRV2DB4IKHNPEWSKQ, seg_pos={0 0}, upload=1781]: Uploading piece...
Jul 26 23:47:50.853845836 (1h56m6.634779214s) [noise=true , address=77.90.31.13:4644 , piece_id=QJLD7E6JMGETVW46TNJJ2ZJJNTESFUTWVOZ4V24VYBMYCPGVPE3Q, seg_pos={0 0}, upload=1781]: Uploading piece...
Jul 26 23:47:50.854729082 (1h56m6.635662462s) [noise=true , address=79.127.201.214:4289 , piece_id=5Q272LELGSMG73ITNPPWSU6KRZGJVUXSNJGJL3N2VJLH6E2XD47Q, seg_pos={0 0}, upload=1781]: Uploading piece...
Jul 26 23:47:50.855582594 (1h56m6.63651598s) [noise=true , address=79.127.205.245:4243 , piece_id=MGXEDD7MBPK6WHOZE272GP3AQUJNGD2PK22VTTVVB5GGZBMWZGNA, seg_pos={0 0}, upload=1781]: Uploading piece...
Jul 26 23:47:50.856754050 (1h56m6.637687434s) [noise=true , address=79.127.201.215:4359 , piece_id=W5TFF2OVPLMU5L5XIV6JSPLKL3HT7YM4EUX4M55VFUFIOKM7CFPQ, seg_pos={0 0}, upload=1781]: Uploading piece...
Jul 26 23:47:50.857246574 (1h56m6.638179959s) [noise=true , address=79.127.220.101:4048 , piece_id=5Y7GEA54Q5S7Y63J5Z5IUREPQIFPZGLHX3ZSXL3DJUCVXWQE2QTA, seg_pos={0 0}, upload=1781]: Uploading piece...
Jul 26 23:47:50.858576749 (1h56m6.639510153s) [noise=true , address=104.36.82.56:4233 , piece_id=MGTZAU7DY4UU3RPIF7PJ4IT6FOOH6YNQ4ULYC5HGGDRKIIDHZVNA, seg_pos={0 0}, upload=1781]: Uploading piece...
Jul 26 23:47:50.859163927 (1h56m6.640097308s) [noise=true , address=104.36.82.46:4507 , piece_id=FFRVQS72UBWHSNCTQWXCQU6VMRUGDA7FJPVUVPZW4G3MVW6WD3ZA, seg_pos={0 0}, upload=1781]: Uploading piece...
Jul 26 23:47:50.859536418 (1h56m6.640469796s) [noise=true , address=104.36.82.47:4544 , piece_id=42MEVXRAN3KXUJJWRWTDC3W2IY27B6CSKZSYMHWA6MDW6GATI67A, seg_pos={0 0}, upload=1781]: Uploading piece...
Jul 26 23:47:50.859853040 (1h56m6.640786418s) [noise=true , address=77.90.31.10:4533 , piece_id=TJKDZQQGYYGBMJVJDXUZMB2XRLVFEO6YXDZG6UEZWASL2OQCCLCQ, seg_pos={0 0}, upload=1781]: Uploading piece...
Jul 26 23:47:50.860218627 (1h56m6.641151992s) [noise=true , address=79.127.220.101:4041 , piece_id=PZ2QP3GERIHRGI2BGDNLASEKHKUJJ7NEYYD73IDKEH44PH6WWP4Q, seg_pos={0 0}, upload=1781]: Uploading piece...
Jul 26 23:47:50.860692805 (1h56m6.641626192s) [noise=true , address=64.49.13.154:4927 , piece_id=KXGJSF6PWXRFAPFYUDDO75ZHQEP524RFMJFZ6UYDVPJIAQMYYQBQ, seg_pos={0 0}, upload=1781]: Uploading piece...
Jul 26 23:47:50.861174941 (1h56m6.642108311s) [noise=true , address=79.127.205.235:4666 , piece_id=4LH3KM55VESEPSIXU6VGEBI4ZGMSPUVJ7KZXTFM4PNJEZCW2Q6RQ, seg_pos={0 0}, upload=1781]: Uploading piece...
Jul 26 23:47:50.861945352 (1h56m6.642878738s) [noise=true , address=79.127.219.38:4017 , piece_id=7USCBFN3SR5U434ES632SUKSEZFB44CXIXULUXP447CEROVTK4GA, seg_pos={0 0}, upload=1781]: Uploading piece...
Jul 26 23:47:50.864719645 (1h56m6.645653014s) [noise=true , address=45.91.116.5:4306 , piece_id=JTJQ3RMG3KQDDRKRWKQ2HEZOWKCWEKE2TBBLYZPLOQBWBCW5LEPQ, seg_pos={0 0}, upload=1781]: Uploading piece...
Jul 26 23:47:50.865188257 (1h56m6.646121617s) [noise=true , address=45.91.116.9:4267 , piece_id=NAGNQNRQVMOYPUOP5RJ7OFKTII6IV6KRB3KSYTRKIS3VTOZMWA5A, seg_pos={0 0}, upload=1781]: Uploading piece...
Jul 26 23:47:50.967850852 (1h56m6.74878424s) [upload=1781]: Next returned nil segment
Jul 26 23:47:50.967909997 (1h56m6.748843386s) [upload=1781]: Waiting for error group managing segments...
Jul 26 23:47:50.973153501 (1h56m6.754086867s) [noise=true , address=77.90.31.10:4533 , piece_id=TJKDZQQGYYGBMJVJDXUZMB2XRLVFEO6YXDZG6UEZWASL2OQCCLCQ, seg_pos={0 0}, upload=1781]: Done uploading piece. err: <nil>
Jul 26 23:47:50.973823257 (1h56m6.754756627s) [noise=true , address=79.127.219.38:4017 , piece_id=7USCBFN3SR5U434ES632SUKSEZFB44CXIXULUXP447CEROVTK4GA, seg_pos={0 0}, upload=1781]: Done uploading piece. err: <nil>
Jul 26 23:47:51.229992473 (1h56m7.010925843s) [noise=true , address=45.91.116.9:4267 , piece_id=NAGNQNRQVMOYPUOP5RJ7OFKTII6IV6KRB3KSYTRKIS3VTOZMWA5A, seg_pos={0 0}, upload=1781]: Done uploading piece. err: <nil>
Jul 26 23:47:51.522582029 (1h56m7.303515396s) [noise=true , address=64.49.13.154:4927 , piece_id=KXGJSF6PWXRFAPFYUDDO75ZHQEP524RFMJFZ6UYDVPJIAQMYYQBQ, seg_pos={0 0}, upload=1781]: Done uploading piece. err: <nil>
Jul 26 23:47:51.528512108 (1h56m7.309445491s) [noise=true , address=79.127.205.245:4243 , piece_id=MGXEDD7MBPK6WHOZE272GP3AQUJNGD2PK22VTTVVB5GGZBMWZGNA, seg_pos={0 0}, upload=1781]: Done uploading piece. err: <nil>
Jul 26 23:47:51.663423094 (1h56m7.444356473s) [noise=true , address=79.127.205.235:4666 , piece_id=4LH3KM55VESEPSIXU6VGEBI4ZGMSPUVJ7KZXTFM4PNJEZCW2Q6RQ, seg_pos={0 0}, upload=1781]: Done uploading piece. err: <nil>
Jul 26 23:47:54.327215338 (1h56m10.108148699s) [noise=true , address=77.90.31.13:4644 , piece_id=QJLD7E6JMGETVW46TNJJ2ZJJNTESFUTWVOZ4V24VYBMYCPGVPE3Q, seg_pos={0 0}, upload=1781]: Done uploading piece. err: <nil>
Jul 26 23:47:54.694537518 (1h56m10.475470891s) [noise=true , address=79.127.220.101:4048 , piece_id=5Y7GEA54Q5S7Y63J5Z5IUREPQIFPZGLHX3ZSXL3DJUCVXWQE2QTA, seg_pos={0 0}, upload=1781]: Done uploading piece. err: <nil>
Jul 26 23:47:56.141286870 (1h56m11.922220239s) [noise=true , address=45.91.116.5:4306 , piece_id=JTJQ3RMG3KQDDRKRWKQ2HEZOWKCWEKE2TBBLYZPLOQBWBCW5LEPQ, seg_pos={0 0}, upload=1781]: Done uploading piece. err: <nil>
Jul 26 23:47:56.627782361 (1h56m12.408715827s) [noise=true , address=79.127.220.101:4041 , piece_id=PZ2QP3GERIHRGI2BGDNLASEKHKUJJ7NEYYD73IDKEH44PH6WWP4Q, seg_pos={0 0}, upload=1781]: Done uploading piece. err: <nil>
Jul 26 23:47:57.625636065 (1h56m13.406569431s) [noise=true , address=79.127.201.214:4289 , piece_id=5Q272LELGSMG73ITNPPWSU6KRZGJVUXSNJGJL3N2VJLH6E2XD47Q, seg_pos={0 0}, upload=1781]: Done uploading piece. err: <nil>
Jul 26 23:48:32.739574388 (1h56m48.520507752s) [noise=true , address=79.127.201.215:4359 , piece_id=W5TFF2OVPLMU5L5XIV6JSPLKL3HT7YM4EUX4M55VFUFIOKM7CFPQ, seg_pos={0 0}, upload=1781]: Done uploading piece. err: <nil>
Jul 26 23:48:50.787580192 (1h57m6.568513574s) [noise=true , address=104.36.82.47:4544 , piece_id=42MEVXRAN3KXUJJWRWTDC3W2IY27B6CSKZSYMHWA6MDW6GATI67A, seg_pos={0 0}, upload=1781]: Done uploading piece. err: ecclient: upload failed (node:12RSxwxw1WXkNcu7b3dEgJhyNkFGYUfKF4QVKmK323L5g3EKFnG, address:104.36.82.47:4544): protocol: rpc: dial tcp 104.36.82.47:4544: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 26 23:48:50.787657420 (1h57m6.568590797s) [noise=true , address=104.36.82.55:4198 , piece_id=SXGZYS3YKOZC5GJOOOI6BBWUVIKM4NE25C3GVUXNLEP5STQPIC5A, seg_pos={0 0}, upload=1781]: Done uploading piece. err: ecclient: upload failed (node:12YxL4YR6mNE536w1SnBrMUZHiaRTuU68Zjg9LBvRURVx5s5HWL, address:104.36.82.55:4198): protocol: write tcp 159.223.143.39:47734->104.36.82.55:4198: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 26 23:48:50.787706646 (1h57m6.568640029s) [noise=true , address=77.90.31.16:4747 , piece_id=HRMAPCAJ4Q7JQPTALZQCHSKI43P6Q54D45IZJQXT6YP5U5FFOKRA, seg_pos={0 0}, upload=1781]: Done uploading piece. err: ecclient: upload failed (node:12ERdBm3XPw4s7JHeMWsVniwcqnfMtP4ZX8sBVjri6dzZtq8AF2, address:77.90.31.16:4747): protocol: rpc: dial tcp 77.90.31.16:4747: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 26 23:48:50.787812939 (1h57m6.568746314s) [noise=true , address=104.36.82.53:4138 , piece_id=RDOXJNWARUJ2Q5FWGQ7XNMPYSAQMRYWTECPSRV2DB4IKHNPEWSKQ, seg_pos={0 0}, upload=1781]: Done uploading piece. err: ecclient: upload failed (node:1YUGGJg6mWeQMmzSNWtLFWAaWwaKWfEf8wxXs1jwiZaA9nhsr3, address:104.36.82.53:4138): protocol: rpc: dial tcp 104.36.82.53:4138: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 26 23:48:50.787891428 (1h57m6.568824806s) [noise=true , address=104.36.82.56:4233 , piece_id=MGTZAU7DY4UU3RPIF7PJ4IT6FOOH6YNQ4ULYC5HGGDRKIIDHZVNA, seg_pos={0 0}, upload=1781]: Done uploading piece. err: ecclient: upload failed (node:1DmQeoQWTnFhQeJFseJoNYzFyo7vZjYKwDtQVJ6pomksWExsTb, address:104.36.82.56:4233): protocol: rpc: dial tcp 104.36.82.56:4233: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 26 23:48:50.787922971 (1h57m6.568856347s) [noise=true , address=104.36.82.46:4507 , piece_id=FFRVQS72UBWHSNCTQWXCQU6VMRUGDA7FJPVUVPZW4G3MVW6WD3ZA, seg_pos={0 0}, upload=1781]: Done uploading piece. err: ecclient: upload failed (node:14jurWWPpmxwiBS3XmZysEUwugsceYJqVXFj7BbYpx8UFcTR8R, address:104.36.82.46:4507): protocol: rpc: dial tcp 104.36.82.46:4507: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 26 23:48:50.787962855 (1h57m6.568896242s) [seg_pos={0 0}, upload=1781]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil>
Jul 26 23:48:50.812333363 (1h57m6.593266727s) [upload=1781]: Done uploading segments...
Jul 26 23:48:50.812374174 (1h57m6.593307549s) [upload=1781]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12)
Duration: 1m0.026896411s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
18 out of 60 is failed (threshold=3) [...XXX.X....XX.X.X.X.......XXX...XX.....X.XXX...............] duration:
1.931986371s
FAIL (2025-07-26 23:48:49 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 1.932012676s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
13 out of 60 is failed (threshold=3) [..X...X..XX.X...X..X.......X...X....XX..............X.X.....] duration:
35.447672495s
FAIL (2025-07-26 23:48:43 UTC)
context deadline exceeded
Duration: 35.447714062s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
18 out of 60 is failed (threshold=3) [....XXX.X....XX.X.X.X.......XXX...XX.....X.XXX..............] duration:
526.756777ms
FAIL (2025-07-26 23:48:08 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 526.782488ms
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
49 out of 60 is failed (threshold=3) [..........XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.XXXXXXXXXX] duration:
30.030261844s
FAIL (2025-07-26 23:48:07 UTC)
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 8)
======LOGS======
6]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Jul 26 23:47:37.527875473 (1h55m53.308808849s) [upload=1777]: Starting upload
Jul 26 23:47:37.527949599 (1h55m53.308882979s) [upload=1777]: Done starting upload
Jul 26 23:47:37.528017058 (1h55m53.308950424s) [upload=1777]: Uploading segments...
Jul 26 23:47:37.528294490 (1h55m53.309227861s) [upload=1777]: Got next segment. Inline: false
Jul 26 23:47:37.528338994 (1h55m53.309272369s) [upload=1777]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jul 26 23:47:37.528344859 (1h55m53.309278235s) [upload=1777]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jul 26 23:47:37.558168809 (1h55m53.339102191s) [seg_pos={0 0}, upload=1777]: Begin upload segment...
Jul 26 23:47:37.558235107 (1h55m53.339168486s) [seg_pos={0 0}, upload=1777]: Done begin upload segment.
Jul 26 23:47:37.558287934 (1h55m53.339221325s) [noise=true , address=45.91.116.5:4297 , piece_id=LF4VTRQS5KUBY5ZMDJIVCGS3BCMRV7ASOGJU33MFDEP4KIPPTXNQ, seg_pos={0 0}, upload=1777]: Uploading piece...
Jul 26 23:47:37.558942976 (1h55m53.339876363s) [noise=true , address=45.91.116.8:4047 , piece_id=O4IQUREEYILXI5DBT7NQWBPTBYLT4J4OGAOYDQ4AENELVIOYIEGA, seg_pos={0 0}, upload=1777]: Uploading piece...
Jul 26 23:47:37.559380729 (1h55m53.340314113s) [noise=true , address=104.36.82.5:4839 , piece_id=44Y5WSP5THZCBZEAAN2UAOEV6SWVKGJKNUWNB2XK4CBNRYOK6CDQ, seg_pos={0 0}, upload=1777]: Uploading piece...
Jul 26 23:47:37.559759875 (1h55m53.34069326s) [noise=true , address=77.90.31.17:4770 , piece_id=ASY7Z3FGRPLF3LZJSAFHFXGRGXSO2H6QFZ6MJ73IAXVWVYZ3X3MQ, seg_pos={0 0}, upload=1777]: Uploading piece...
Jul 26 23:47:37.560303225 (1h55m53.341236612s) [noise=true , address=104.36.82.11:4115 , piece_id=FFICM37JDSWEMPTX472OOBAJDJBL7DW6RSFRN3T2NAIT6ABFPV5Q, seg_pos={0 0}, upload=1777]: Uploading piece...
Jul 26 23:47:37.560683951 (1h55m53.341617338s) [noise=true , address=77.90.31.18:4807 , piece_id=TWZH6ZGFIHDNPB2VJCWFBUBXNS3E45FRJOO3HW2JIRUWNN5PTSMA, seg_pos={0 0}, upload=1777]: Uploading piece...
Jul 26 23:47:37.561045028 (1h55m53.341978413s) [noise=true , address=45.91.116.8:4048 , piece_id=A62MAJI7XDAJ7FV5Q6T6Z3QEYYAWRW4XLG5AFPL4GTSGNTJMIA2Q, seg_pos={0 0}, upload=1777]: Uploading piece...
Jul 26 23:47:37.561380481 (1h55m53.342313866s) [noise=true , address=104.36.82.8:4969 , piece_id=HKEPNFLA3FGNCJUL5INGDFRKGGZJHEF5YD7SULHHJP5SYIV3J6RQ, seg_pos={0 0}, upload=1777]: Uploading piece...
Jul 26 23:47:37.561750873 (1h55m53.34268426s) [noise=true , address=77.90.31.13:4627 , piece_id=MB6BZ2N7JEQSNT34DZKMQH7QW37SOXMCNKZATWK7WFOZMP4RFOXA, seg_pos={0 0}, upload=1777]: Uploading piece...
Jul 26 23:47:37.562132507 (1h55m53.343065893s) [noise=true , address=104.36.82.5:4837 , piece_id=4N6VJS5MXC2ZEX6K2OAGATLVGXGXAFJF3YGITIE6FMBJ4FRYI7LA, seg_pos={0 0}, upload=1777]: Uploading piece...
Jul 26 23:47:37.562492426 (1h55m53.343425809s) [noise=true , address=104.36.82.2:4732 , piece_id=SO72M3MPNP3P5LOHBSNDFLKH7JGA2FIU5FRAJ5GLQPVHYBOXVKYA, seg_pos={0 0}, upload=1777]: Uploading piece...
Jul 26 23:47:37.562816523 (1h55m53.343749907s) [noise=true , address=104.36.82.15:4257 , piece_id=YIBYEQOCFB2RQBIH25O76NKKCVWAO5YAY6Q6PG3KQPLKD4BIBGYQ, seg_pos={0 0}, upload=1777]: Uploading piece...
Jul 26 23:47:37.563122290 (1h55m53.344055671s) [noise=true , address=104.36.82.10:4079 , piece_id=WVQDLMLEMPQQLYV6ESENXILSBPZ4L2KSCESFOSCMPC5S4ATMSXRA, seg_pos={0 0}, upload=1777]: Uploading piece...
Jul 26 23:47:37.563350758 (1h55m53.344284144s) [noise=true , address=45.91.116.9:4261 , piece_id=TL3FY4IT6PSJLEJFNMFU5P7IU5SPYTY3ZZNYGXGXG4LBTNUNVC5A, seg_pos={0 0}, upload=1777]: Uploading piece...
Jul 26 23:47:37.563844153 (1h55m53.3447776s) [noise=true , address=45.91.116.6:4975 , piece_id=INLDPSR72KZI6UCE7XBBBM7MF3QVUYZK5QENI2MPQZUZX2Y7IW6A, seg_pos={0 0}, upload=1777]: Uploading piece...
Jul 26 23:47:37.564376257 (1h55m53.345309632s) [noise=true , address=45.91.116.12:4227 , piece_id=C7U3VM7JTAPXZL7DGSNAICDFSCVRHSQ3HALSIUDCGTEYAXMRVKFQ, seg_pos={0 0}, upload=1777]: Uploading piece...
Jul 26 23:47:37.641178717 (1h55m53.422112098s) [upload=1777]: Next returned nil segment
Jul 26 23:47:37.641199954 (1h55m53.42213334s) [upload=1777]: Waiting for error group managing segments...
Jul 26 23:47:37.645131321 (1h55m53.426064701s) [noise=true , address=45.91.116.8:4048 , piece_id=A62MAJI7XDAJ7FV5Q6T6Z3QEYYAWRW4XLG5AFPL4GTSGNTJMIA2Q, seg_pos={0 0}, upload=1777]: Done uploading piece. err: <nil>
Jul 26 23:47:37.645740259 (1h55m53.426673637s) [noise=true , address=45.91.116.8:4047 , piece_id=O4IQUREEYILXI5DBT7NQWBPTBYLT4J4OGAOYDQ4AENELVIOYIEGA, seg_pos={0 0}, upload=1777]: Done uploading piece. err: <nil>
Jul 26 23:47:37.646303572 (1h55m53.427236951s) [noise=true , address=45.91.116.12:4227 , piece_id=C7U3VM7JTAPXZL7DGSNAICDFSCVRHSQ3HALSIUDCGTEYAXMRVKFQ, seg_pos={0 0}, upload=1777]: Done uploading piece. err: <nil>
Jul 26 23:47:37.646915711 (1h55m53.427849087s) [noise=true , address=77.90.31.17:4770 , piece_id=ASY7Z3FGRPLF3LZJSAFHFXGRGXSO2H6QFZ6MJ73IAXVWVYZ3X3MQ, seg_pos={0 0}, upload=1777]: Done uploading piece. err: <nil>
Jul 26 23:47:37.647563748 (1h55m53.428497125s) [noise=true , address=45.91.116.6:4975 , piece_id=INLDPSR72KZI6UCE7XBBBM7MF3QVUYZK5QENI2MPQZUZX2Y7IW6A, seg_pos={0 0}, upload=1777]: Done uploading piece. err: <nil>
Jul 26 23:47:37.985755377 (1h55m53.766688752s) [noise=true , address=45.91.116.9:4261 , piece_id=TL3FY4IT6PSJLEJFNMFU5P7IU5SPYTY3ZZNYGXGXG4LBTNUNVC5A, seg_pos={0 0}, upload=1777]: Done uploading piece. err: <nil>
Jul 26 23:47:38.091323807 (1h55m53.872257177s) [noise=true , address=45.91.116.5:4297 , piece_id=LF4VTRQS5KUBY5ZMDJIVCGS3BCMRV7ASOGJU33MFDEP4KIPPTXNQ, seg_pos={0 0}, upload=1777]: Done uploading piece. err: <nil>
Jul 26 23:47:39.022083056 (1h55m54.803016433s) [noise=true , address=77.90.31.13:4627 , piece_id=MB6BZ2N7JEQSNT34DZKMQH7QW37SOXMCNKZATWK7WFOZMP4RFOXA, seg_pos={0 0}, upload=1777]: Done uploading piece. err: <nil>
Jul 26 23:48:07.498227396 (1h56m23.279160778s) [noise=true , address=104.36.82.5:4839 , piece_id=44Y5WSP5THZCBZEAAN2UAOEV6SWVKGJKNUWNB2XK4CBNRYOK6CDQ, seg_pos={0 0}, upload=1777]: Done uploading piece. err: ecclient: upload failed (node:1FcPS8yXfgdFvsxFofPrNCtFDLtebQMTaLu75335kVXeQxttuv, address:104.36.82.5:4839): protocol: rpc: dial tcp 104.36.82.5:4839: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 26 23:48:07.498315165 (1h56m23.279248531s) [noise=true , address=104.36.82.2:4732 , piece_id=SO72M3MPNP3P5LOHBSNDFLKH7JGA2FIU5FRAJ5GLQPVHYBOXVKYA, seg_pos={0 0}, upload=1777]: Done uploading piece. err: ecclient: upload failed (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE, address:104.36.82.2:4732): protocol: rpc: dial tcp 104.36.82.2:4732: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 26 23:48:07.498385439 (1h56m23.279318803s) [noise=true , address=104.36.82.11:4115 , piece_id=FFICM37JDSWEMPTX472OOBAJDJBL7DW6RSFRN3T2NAIT6ABFPV5Q, seg_pos={0 0}, upload=1777]: Done uploading piece. err: ecclient: upload failed (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR, address:104.36.82.11:4115): protocol: rpc: dial tcp 104.36.82.11:4115: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 26 23:48:07.498420911 (1h56m23.279354278s) [noise=true , address=104.36.82.10:4079 , piece_id=WVQDLMLEMPQQLYV6ESENXILSBPZ4L2KSCESFOSCMPC5S4ATMSXRA, seg_pos={0 0}, upload=1777]: Done uploading piece. err: ecclient: upload failed (node:12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud, address:104.36.82.10:4079): protocol: rpc: dial tcp 104.36.82.10:4079: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 26 23:48:07.498457186 (1h56m23.27939055s) [noise=true , address=77.90.31.18:4807 , piece_id=TWZH6ZGFIHDNPB2VJCWFBUBXNS3E45FRJOO3HW2JIRUWNN5PTSMA, seg_pos={0 0}, upload=1777]: Done uploading piece. err: ecclient: upload failed (node:12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS, address:77.90.31.18:4807): protocol: rpc: dial tcp 77.90.31.18:4807: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 26 23:48:07.498474409 (1h56m23.279407776s) [noise=true , address=104.36.82.8:4969 , piece_id=HKEPNFLA3FGNCJUL5INGDFRKGGZJHEF5YD7SULHHJP5SYIV3J6RQ, seg_pos={0 0}, upload=1777]: Done uploading piece. err: ecclient: upload failed (node:1HxTyEaQTErcZJWMYLsND9Xa8Nwaa5e1f4x2bs7UQzSRiW7buX, address:104.36.82.8:4969): protocol: write tcp 159.223.143.39:42522->104.36.82.8:4969: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 26 23:48:07.498484500 (1h56m23.279417863s) [noise=true , address=104.36.82.5:4837 , piece_id=4N6VJS5MXC2ZEX6K2OAGATLVGXGXAFJF3YGITIE6FMBJ4FRYI7LA, seg_pos={0 0}, upload=1777]: Done uploading piece. err: ecclient: upload failed (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV, address:104.36.82.5:4837): protocol: rpc: dial tcp 104.36.82.5:4837: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 26 23:48:07.498521916 (1h56m23.2794553s) [noise=true , address=104.36.82.15:4257 , piece_id=YIBYEQOCFB2RQBIH25O76NKKCVWAO5YAY6Q6PG3KQPLKD4BIBGYQ, seg_pos={0 0}, upload=1777]: Done uploading piece. err: ecclient: upload failed (node:12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK, address:104.36.82.15:4257): protocol: write tcp 159.223.143.39:48126->104.36.82.15:4257: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 26 23:48:07.498548008 (1h56m23.279481386s) [seg_pos={0 0}, upload=1777]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil>
Jul 26 23:48:07.526079352 (1h56m23.307012724s) [upload=1777]: Done uploading segments...
Jul 26 23:48:07.526127435 (1h56m23.30706082s) [upload=1777]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8)
Duration: 30.030287736s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
13 out of 60 is failed (threshold=3) [...X...X..XX.X...X..X.......X...X....XX..............X.X....] duration:
34.959670595s
FAIL (2025-07-26 23:48:07 UTC)
context deadline exceeded
Duration: 34.959695117s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)antiblip:
52 out of 60 is failed (threshold=3) [.XX....XXXX.XXXXX.X.XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration:
1m0.026385634s
FAIL (2025-07-26 23:47:50 UTC)
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 10)
======LOGS======
23:46:50.741258335 (1h55m6.522191701s) [seg_pos={0 0}, upload=1761]: Done begin upload segment.
Jul 26 23:46:50.741296624 (1h55m6.52223s) [noise=true , address=79.127.205.245:4223 , piece_id=XUYPCZKXBDRPPXBJZABJWFUDC4R3HMNWCXKEZB7GRC4AOPMAC53A, seg_pos={0 0}, upload=1761]: Uploading piece...
Jul 26 23:46:50.741863330 (1h55m6.522796711s) [noise=true , address=64.49.13.140:4514 , piece_id=B43VCZPYNZAVEPM73B6ZP4HA7VUTZVH6IWGQPRWZCU6I2XW3VC3A, seg_pos={0 0}, upload=1761]: Uploading piece...
Jul 26 23:46:50.742243093 (1h55m6.523176459s) [noise=true , address=104.36.82.55:4198 , piece_id=MOAJGCTK2HX352AFAV2DPXQB2ABF7TGMLG34EGOBNNZV2SJIR2HA, seg_pos={0 0}, upload=1761]: Uploading piece...
Jul 26 23:46:50.742713710 (1h55m6.523647075s) [noise=true , address=79.127.205.233:4639 , piece_id=IFHAJKNGZI7CXLLHIFZ6UU35OXKN4R6DRKTZOFUTSSWEFLPUD34A, seg_pos={0 0}, upload=1761]: Uploading piece...
Jul 26 23:46:50.742795025 (1h55m6.523728401s) [noise=true , address=79.127.205.235:4691 , piece_id=XD3PQQWMCIF4TCHGG4PCVEHLV46CE4RCBBHDJAM3KZOUFZMNZV5Q, seg_pos={0 0}, upload=1761]: Uploading piece...
Jul 26 23:46:50.743443815 (1h55m6.524377193s) [noise=true , address=64.49.13.138:4482 , piece_id=2T5EVLLGN7S3IKBNBYJPHM6ZOAOFVWJJ73UMN4M7U6M7D5EYRZZA, seg_pos={0 0}, upload=1761]: Uploading piece...
Jul 26 23:46:50.743489781 (1h55m6.524423156s) [noise=true , address=79.127.219.38:4009 , piece_id=MXCAEBV2EASVTXBV2ODL7PB3BTKYBLIBX2P6AMH6QLWBSONLEWPA, seg_pos={0 0}, upload=1761]: Uploading piece...
Jul 26 23:46:50.745542940 (1h55m6.526476328s) [noise=true , address=104.36.82.53:4155 , piece_id=VLRJ3PFP33L65VJBDGNDAFDC2OZLYQ3QXUFLNJZAVUWGRXWHQWLQ, seg_pos={0 0}, upload=1761]: Uploading piece...
Jul 26 23:46:50.747456571 (1h55m6.52838996s) [noise=true , address=77.90.31.16:4751 , piece_id=IKPXMQS4NK5SPH4OG7VVYEZPURDYOX42VNONP3NQKGVQBIASWYJA, seg_pos={0 0}, upload=1761]: Uploading piece...
Jul 26 23:46:50.748561713 (1h55m6.52949509s) [noise=true , address=77.90.31.11:4562 , piece_id=Q6XE2QDQHIN2V3FVP5ND5NJ775IOOFGPGG2KNMXCJOG3UDIS2QPA, seg_pos={0 0}, upload=1761]: Uploading piece...
Jul 26 23:46:50.751253569 (1h55m6.532186934s) [noise=true , address=104.36.82.47:4550 , piece_id=JRR64PPLW3XFFLKH6DH2TEQSJ3CEOKNJN4JNCLK6DN66KCNX6DAA, seg_pos={0 0}, upload=1761]: Uploading piece...
Jul 26 23:46:50.751855586 (1h55m6.532788966s) [noise=true , address=79.127.205.246:4282 , piece_id=T6QL3EUTWM764VCXJ66PE2SQLE26XOWFYGESUQRUVVXMRGJ4A3JA, seg_pos={0 0}, upload=1761]: Uploading piece...
Jul 26 23:46:50.752829775 (1h55m6.533763161s) [noise=true , address=104.36.82.3:4796 , piece_id=2E73WPDAPO7SHJQUH7NVBDAL37AKZCKJ5Z2DPLDSDGHYMSAXS7CQ, seg_pos={0 0}, upload=1761]: Uploading piece...
Jul 26 23:46:50.753702355 (1h55m6.534635741s) [noise=true , address=79.127.205.234:4615 , piece_id=PFH2OVDWWOIHDRMJLNEE5JQPZEB525CLCDHBL7XLNHL65ANJRZKQ, seg_pos={0 0}, upload=1761]: Uploading piece...
Jul 26 23:46:50.755846908 (1h55m6.536780292s) [noise=true , address=104.36.82.14:4236 , piece_id=OGTQF76VCS6ZZ37IFULCSOOY57IY52N6JSW5M7AYIVW7NREZR5VA, seg_pos={0 0}, upload=1761]: Uploading piece...
Jul 26 23:46:50.758149954 (1h55m6.539083344s) [noise=true , address=45.91.116.14:4194 , piece_id=ZN7V6Y6OXZSGI55LRZYGR7CCLMXFAI3QFBT6DFU27EFG7OYVZZYQ, seg_pos={0 0}, upload=1761]: Uploading piece...
Jul 26 23:46:50.762999165 (1h55m6.543932552s) [noise=true , address=104.36.82.56:4235 , piece_id=B5OKANQZ3S6MBNTNJBD75HLRJRFKZF6GVCJSEOTH4P4XTFBQ352A, seg_pos={0 0}, upload=1761]: Uploading piece...
Jul 26 23:46:50.768688898 (1h55m6.549622286s) [noise=true , address=104.36.82.46:4483 , piece_id=6T6VBFPV4WKYFHPFBMIM2QIJZZZ666PVQ7OCDBFGFRNIWXYTUH6A, seg_pos={0 0}, upload=1761]: Uploading piece...
Jul 26 23:46:50.830056987 (1h55m6.610990363s) [upload=1761]: Next returned nil segment
Jul 26 23:46:50.830077461 (1h55m6.61101084s) [upload=1761]: Waiting for error group managing segments...
Jul 26 23:46:50.835286105 (1h55m6.616219474s) [noise=true , address=79.127.219.38:4009 , piece_id=MXCAEBV2EASVTXBV2ODL7PB3BTKYBLIBX2P6AMH6QLWBSONLEWPA, seg_pos={0 0}, upload=1761]: Done uploading piece. err: <nil>
Jul 26 23:46:50.844324478 (1h55m6.625257848s) [noise=true , address=45.91.116.14:4194 , piece_id=ZN7V6Y6OXZSGI55LRZYGR7CCLMXFAI3QFBT6DFU27EFG7OYVZZYQ, seg_pos={0 0}, upload=1761]: Done uploading piece. err: <nil>
Jul 26 23:46:50.867851587 (1h55m6.64878497s) [noise=true , address=77.90.31.11:4562 , piece_id=Q6XE2QDQHIN2V3FVP5ND5NJ775IOOFGPGG2KNMXCJOG3UDIS2QPA, seg_pos={0 0}, upload=1761]: Done uploading piece. err: <nil>
Jul 26 23:46:51.306073912 (1h55m7.087007281s) [noise=true , address=79.127.205.245:4223 , piece_id=XUYPCZKXBDRPPXBJZABJWFUDC4R3HMNWCXKEZB7GRC4AOPMAC53A, seg_pos={0 0}, upload=1761]: Done uploading piece. err: <nil>
Jul 26 23:46:51.311970486 (1h55m7.092903867s) [noise=true , address=79.127.205.235:4691 , piece_id=XD3PQQWMCIF4TCHGG4PCVEHLV46CE4RCBBHDJAM3KZOUFZMNZV5Q, seg_pos={0 0}, upload=1761]: Done uploading piece. err: <nil>
Jul 26 23:46:51.320568779 (1h55m7.101502158s) [noise=true , address=79.127.205.233:4639 , piece_id=IFHAJKNGZI7CXLLHIFZ6UU35OXKN4R6DRKTZOFUTSSWEFLPUD34A, seg_pos={0 0}, upload=1761]: Done uploading piece. err: <nil>
Jul 26 23:46:51.338459952 (1h55m7.119393336s) [noise=true , address=79.127.205.246:4282 , piece_id=T6QL3EUTWM764VCXJ66PE2SQLE26XOWFYGESUQRUVVXMRGJ4A3JA, seg_pos={0 0}, upload=1761]: Done uploading piece. err: <nil>
Jul 26 23:46:51.344736585 (1h55m7.125669967s) [noise=true , address=79.127.205.234:4615 , piece_id=PFH2OVDWWOIHDRMJLNEE5JQPZEB525CLCDHBL7XLNHL65ANJRZKQ, seg_pos={0 0}, upload=1761]: Done uploading piece. err: <nil>
Jul 26 23:46:52.954574066 (1h55m8.735507434s) [noise=true , address=64.49.13.138:4482 , piece_id=2T5EVLLGN7S3IKBNBYJPHM6ZOAOFVWJJ73UMN4M7U6M7D5EYRZZA, seg_pos={0 0}, upload=1761]: Done uploading piece. err: <nil>
Jul 26 23:46:53.162653573 (1h55m8.94358695s) [noise=true , address=64.49.13.140:4514 , piece_id=B43VCZPYNZAVEPM73B6ZP4HA7VUTZVH6IWGQPRWZCU6I2XW3VC3A, seg_pos={0 0}, upload=1761]: Done uploading piece. err: <nil>
Jul 26 23:47:50.672326331 (1h56m6.453259701s) [noise=true , address=104.36.82.56:4235 , piece_id=B5OKANQZ3S6MBNTNJBD75HLRJRFKZF6GVCJSEOTH4P4XTFBQ352A, seg_pos={0 0}, upload=1761]: Done uploading piece. err: ecclient: upload failed (node:1a1sW1G1WPL5WQ2XxprWKQMsJkcb5ETSkk3ULp1TjRaZWptaT, address:104.36.82.56:4235): protocol: rpc: dial tcp 104.36.82.56:4235: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 26 23:47:50.672426801 (1h56m6.453360169s) [noise=true , address=104.36.82.3:4796 , piece_id=2E73WPDAPO7SHJQUH7NVBDAL37AKZCKJ5Z2DPLDSDGHYMSAXS7CQ, seg_pos={0 0}, upload=1761]: Done uploading piece. err: ecclient: upload failed (node:12nZwQ4Poz9M14ijFV6qpy5SCWEMeyHQpxN17uSBTH1ts1CFtVS, address:104.36.82.3:4796): protocol: rpc: dial tcp 104.36.82.3:4796: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 26 23:47:50.672518748 (1h56m6.453452113s) [noise=true , address=104.36.82.46:4483 , piece_id=6T6VBFPV4WKYFHPFBMIM2QIJZZZ666PVQ7OCDBFGFRNIWXYTUH6A, seg_pos={0 0}, upload=1761]: Done uploading piece. err: ecclient: upload failed (node:1igSs2ZGX3zWVsSdmm1pnwYd7JckcyBubTuNU1gJDxQsZ9nKaY, address:104.36.82.46:4483): protocol: rpc: dial tcp 104.36.82.46:4483: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 26 23:47:50.672594467 (1h56m6.453527833s) [noise=true , address=104.36.82.47:4550 , piece_id=JRR64PPLW3XFFLKH6DH2TEQSJ3CEOKNJN4JNCLK6DN66KCNX6DAA, seg_pos={0 0}, upload=1761]: Done uploading piece. err: ecclient: upload failed (node:1Fz8JANjsHWJkHj7Y2JAkr7NTKqDtcKjRW4kDaYGLJYwgWnxbS, address:104.36.82.47:4550): protocol: rpc: dial tcp 104.36.82.47:4550: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 26 23:47:50.672688561 (1h56m6.453621926s) [noise=true , address=104.36.82.14:4236 , piece_id=OGTQF76VCS6ZZ37IFULCSOOY57IY52N6JSW5M7AYIVW7NREZR5VA, seg_pos={0 0}, upload=1761]: Done uploading piece. err: ecclient: upload failed (node:1XaE29W27ym2RmuGdBvHeKcQZZEUbask1zsm9Bz8xLokZNGfcj, address:104.36.82.14:4236): protocol: rpc: dial tcp 104.36.82.14:4236: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 26 23:47:50.672761864 (1h56m6.453695229s) [noise=true , address=104.36.82.55:4198 , piece_id=MOAJGCTK2HX352AFAV2DPXQB2ABF7TGMLG34EGOBNNZV2SJIR2HA, seg_pos={0 0}, upload=1761]: Done uploading piece. err: ecclient: upload failed (node:12YxL4YR6mNE536w1SnBrMUZHiaRTuU68Zjg9LBvRURVx5s5HWL, address:104.36.82.55:4198): protocol: rpc: dial tcp 104.36.82.55:4198: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 26 23:47:50.672821216 (1h56m6.453754581s) [noise=true , address=104.36.82.53:4155 , piece_id=VLRJ3PFP33L65VJBDGNDAFDC2OZLYQ3QXUFLNJZAVUWGRXWHQWLQ, seg_pos={0 0}, upload=1761]: Done uploading piece. err: ecclient: upload failed (node:17wRBzmgMvpNyXy9SZEwEjwc4W5TfKzTj8vaa9c2HXnPNLMMYi, address:104.36.82.53:4155): protocol: rpc: dial tcp 104.36.82.53:4155: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 26 23:47:50.672904081 (1h56m6.453837448s) [noise=true , address=77.90.31.16:4751 , piece_id=IKPXMQS4NK5SPH4OG7VVYEZPURDYOX42VNONP3NQKGVQBIASWYJA, seg_pos={0 0}, upload=1761]: Done uploading piece. err: ecclient: upload failed (node:12Sj7nPQiL1EmwvnVCpRv1HEeDYFnknTXLNZTvhAUmHDcP2Wpuf, address:77.90.31.16:4751): protocol: write tcp 159.223.143.39:49890->77.90.31.16:4751: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 26 23:47:50.672923463 (1h56m6.453856845s) [seg_pos={0 0}, upload=1761]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil>
Jul 26 23:47:50.694930936 (1h56m6.475864306s) [upload=1761]: Done uploading segments...
Jul 26 23:47:50.694956257 (1h56m6.475889633s) [upload=1761]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10)
Duration: 1m0.026470506s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
18 out of 60 is failed (threshold=3) [.....XXX.X....XX.X.X.X.......XXX...XX.....X.XXX.............] duration:
4.005178328s
FAIL (2025-07-26 23:47:47 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 4.005211467s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
14 out of 60 is failed (threshold=3) [X...X...X..XX.X...X..X.......X...X....XX..............X.X...] duration:
27.778145579s
FAIL (2025-07-26 23:47:32 UTC)
context deadline exceeded
Duration: 27.778300915s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
48 out of 60 is failed (threshold=3) [...........XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.XXXXXXXXX] duration:
30.029294507s
FAIL (2025-07-26 23:47:31 UTC)
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
2.5:4838): protocol: rpc: dial tcp 104.36.82.5:4838: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 26 23:46:47.595347255 (1h55m3.376280637s) [noise=true , address=104.36.82.11:4113 , piece_id=VB2HSMCNMM2NCGSDCFZCOWDXWGGJUFM2FQTDF5LNEGHFXO4RQUAA, seg_pos={0 0}, upload=1754]: Done uploading piece. err: ecclient: upload failed (node:1SQKETWozrz4PcDC3yhy6V5d8Ryo6NNiyWWEkE5xRcmUj9B6UV, address:104.36.82.11:4113): protocol: write tcp 159.223.143.39:39866->104.36.82.11:4113: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 26 23:46:47.595370646 (1h55m3.376304037s) [seg_pos={0 0}, upload=1754]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil>
Jul 26 23:46:47.625410173 (1h55m3.406343561s) [upload=1754]: Done uploading segments...
Jul 26 23:46:47.625470564 (1h55m3.406403949s) [upload=1754]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9)
Jul 26 23:47:01.500955539 (1h55m17.281888924s) [upload=1766]: Starting upload
Jul 26 23:47:01.501081603 (1h55m17.28201498s) [upload=1766]: Done starting upload
Jul 26 23:47:01.501199722 (1h55m17.2821331s) [upload=1766]: Uploading segments...
Jul 26 23:47:01.501446727 (1h55m17.282380104s) [upload=1766]: Got next segment. Inline: false
Jul 26 23:47:01.501522183 (1h55m17.28245556s) [upload=1766]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jul 26 23:47:01.501530424 (1h55m17.282463802s) [upload=1766]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jul 26 23:47:01.533216618 (1h55m17.314149999s) [seg_pos={0 0}, upload=1766]: Begin upload segment...
Jul 26 23:47:01.533283292 (1h55m17.314216674s) [seg_pos={0 0}, upload=1766]: Done begin upload segment.
Jul 26 23:47:01.533318241 (1h55m17.314251628s) [noise=true , address=45.91.116.12:4225 , piece_id=REQ5G3CELHPZDOCWXX4INQRJLRPHQ4T4SQ2L5HJNGPTEVF5ULOCA, seg_pos={0 0}, upload=1766]: Uploading piece...
Jul 26 23:47:01.533330156 (1h55m17.314263542s) [noise=true , address=45.91.116.8:4045 , piece_id=WJR3LN7VDQ5CXYDJRGRQGH4MF62HS3FGEHJN47PCXGBFKXN3TIVQ, seg_pos={0 0}, upload=1766]: Uploading piece...
Jul 26 23:47:01.533842599 (1h55m17.314775989s) [noise=true , address=45.91.116.13:4155 , piece_id=D54ANATLN425LK3BYXRWHLF5ZURXMPGRNWRE52OIPCHQ672X7F2Q, seg_pos={0 0}, upload=1766]: Uploading piece...
Jul 26 23:47:01.533912175 (1h55m17.314845541s) [noise=true , address=77.90.31.12:4590 , piece_id=EGHHI56SSVPTYOVSBDCG2ZTZM65DLXCRK6YLPWZNSGCZFDLBTEZA, seg_pos={0 0}, upload=1766]: Uploading piece...
Jul 26 23:47:01.534276158 (1h55m17.315209546s) [noise=true , address=104.36.82.7:4912 , piece_id=IYGFHYQPGE54VZLRQ4X47OE63PENODSP3TCLF6AANGU5TVGCLYUQ, seg_pos={0 0}, upload=1766]: Uploading piece...
Jul 26 23:47:01.534421105 (1h55m17.315354485s) [noise=true , address=45.91.116.5:4298 , piece_id=DGIB22ZRWLDPO2MYS62MGFW6RHWE4BJOUDK7FVAQK3R2UJIAVVWQ, seg_pos={0 0}, upload=1766]: Uploading piece...
Jul 26 23:47:01.534670997 (1h55m17.315604381s) [noise=true , address=45.91.116.6:4974 , piece_id=OEWQAOPD3IFF37256ROD4L55YA7BXLDUCGRDG5LIWJWB56BDNUQA, seg_pos={0 0}, upload=1766]: Uploading piece...
Jul 26 23:47:01.536634078 (1h55m17.317567471s) [noise=true , address=45.91.116.10:4083 , piece_id=ZXKRY4D3AJ7SFNKRRHZJ233AAW3DVMDP7FQPZVJLIIW3CVPDGDDQ, seg_pos={0 0}, upload=1766]: Uploading piece...
Jul 26 23:47:01.537163937 (1h55m17.318097323s) [noise=true , address=45.91.116.14:4189 , piece_id=CZRCKXYEZ6MRBAE625SSZQCDRTCZJEKXAPSRX23ATT5UOF3HAO5A, seg_pos={0 0}, upload=1766]: Uploading piece...
Jul 26 23:47:01.537649133 (1h55m17.318582519s) [noise=true , address=77.90.31.15:4699 , piece_id=2FIJ6HIPV47LF6EVSBYRPJRY2WJ6XMUZWSRCDPNQT577EULTWFMQ, seg_pos={0 0}, upload=1766]: Uploading piece...
Jul 26 23:47:01.538233253 (1h55m17.319166637s) [noise=true , address=45.91.116.7:4010 , piece_id=SI3QWI2QNY7WLYAGIRSVX6GD5HZZZ5GRADPZUOAYZGO3DXZ7UOOQ, seg_pos={0 0}, upload=1766]: Uploading piece...
Jul 26 23:47:01.538862519 (1h55m17.319795901s) [noise=true , address=104.36.82.4:4803 , piece_id=NFHH4SZHNOMJ6ZFIRS5MHPONERX4EMNO6LZIT3M25LSSH57UDXFQ, seg_pos={0 0}, upload=1766]: Uploading piece...
Jul 26 23:47:01.539458233 (1h55m17.320391618s) [noise=true , address=45.91.116.11:4118 , piece_id=ERMOL62UJO4CHMCZXJ4ERWATPH6TP4JBHKEOSZC663IINNOLMDYA, seg_pos={0 0}, upload=1766]: Uploading piece...
Jul 26 23:47:01.540045777 (1h55m17.320979167s) [noise=true , address=104.36.82.7:4909 , piece_id=LSPB6NGZQTTAGNU43I3Z6ICMEAYM6IY6DXTRCKW56QEJ34BM5NHA, seg_pos={0 0}, upload=1766]: Uploading piece...
Jul 26 23:47:01.540705401 (1h55m17.32163879s) [noise=true , address=45.91.116.9:4263 , piece_id=MSJNPDAO4TLUNZQA3VIOT5NNJVBHTTK7BUTXPZ3NWSPWTSXYEURQ, seg_pos={0 0}, upload=1766]: Uploading piece...
Jul 26 23:47:01.541339312 (1h55m17.322272716s) [noise=true , address=104.36.82.6:4874 , piece_id=GOMUVRZAQGEIKTGBLFZK5IMIKAS4MARHGN6ACXEEHTSXKAT3Z4KQ, seg_pos={0 0}, upload=1766]: Uploading piece...
Jul 26 23:47:01.634889508 (1h55m17.415822882s) [upload=1766]: Next returned nil segment
Jul 26 23:47:01.634907916 (1h55m17.4158413s) [upload=1766]: Waiting for error group managing segments...
Jul 26 23:47:01.637150829 (1h55m17.418084191s) [noise=true , address=77.90.31.15:4699 , piece_id=2FIJ6HIPV47LF6EVSBYRPJRY2WJ6XMUZWSRCDPNQT577EULTWFMQ, seg_pos={0 0}, upload=1766]: Done uploading piece. err: <nil>
Jul 26 23:47:01.637919046 (1h55m17.418852414s) [noise=true , address=45.91.116.7:4010 , piece_id=SI3QWI2QNY7WLYAGIRSVX6GD5HZZZ5GRADPZUOAYZGO3DXZ7UOOQ, seg_pos={0 0}, upload=1766]: Done uploading piece. err: <nil>
Jul 26 23:47:01.638473473 (1h55m17.419406837s) [noise=true , address=45.91.116.10:4083 , piece_id=ZXKRY4D3AJ7SFNKRRHZJ233AAW3DVMDP7FQPZVJLIIW3CVPDGDDQ, seg_pos={0 0}, upload=1766]: Done uploading piece. err: <nil>
Jul 26 23:47:01.639102724 (1h55m17.420036091s) [noise=true , address=45.91.116.11:4118 , piece_id=ERMOL62UJO4CHMCZXJ4ERWATPH6TP4JBHKEOSZC663IINNOLMDYA, seg_pos={0 0}, upload=1766]: Done uploading piece. err: <nil>
Jul 26 23:47:01.639769712 (1h55m17.420703075s) [noise=true , address=45.91.116.8:4045 , piece_id=WJR3LN7VDQ5CXYDJRGRQGH4MF62HS3FGEHJN47PCXGBFKXN3TIVQ, seg_pos={0 0}, upload=1766]: Done uploading piece. err: <nil>
Jul 26 23:47:01.640400722 (1h55m17.421334092s) [noise=true , address=45.91.116.6:4974 , piece_id=OEWQAOPD3IFF37256ROD4L55YA7BXLDUCGRDG5LIWJWB56BDNUQA, seg_pos={0 0}, upload=1766]: Done uploading piece. err: <nil>
Jul 26 23:47:01.640867399 (1h55m17.42180077s) [noise=true , address=45.91.116.12:4225 , piece_id=REQ5G3CELHPZDOCWXX4INQRJLRPHQ4T4SQ2L5HJNGPTEVF5ULOCA, seg_pos={0 0}, upload=1766]: Done uploading piece. err: <nil>
Jul 26 23:47:01.641311046 (1h55m17.42224442s) [noise=true , address=45.91.116.14:4189 , piece_id=CZRCKXYEZ6MRBAE625SSZQCDRTCZJEKXAPSRX23ATT5UOF3HAO5A, seg_pos={0 0}, upload=1766]: Done uploading piece. err: <nil>
Jul 26 23:47:01.641798758 (1h55m17.422732148s) [noise=true , address=45.91.116.13:4155 , piece_id=D54ANATLN425LK3BYXRWHLF5ZURXMPGRNWRE52OIPCHQ672X7F2Q, seg_pos={0 0}, upload=1766]: Done uploading piece. err: <nil>
Jul 26 23:47:01.761950372 (1h55m17.542883754s) [noise=true , address=45.91.116.9:4263 , piece_id=MSJNPDAO4TLUNZQA3VIOT5NNJVBHTTK7BUTXPZ3NWSPWTSXYEURQ, seg_pos={0 0}, upload=1766]: Done uploading piece. err: <nil>
Jul 26 23:47:01.778401865 (1h55m17.559335235s) [noise=true , address=45.91.116.5:4298 , piece_id=DGIB22ZRWLDPO2MYS62MGFW6RHWE4BJOUDK7FVAQK3R2UJIAVVWQ, seg_pos={0 0}, upload=1766]: Done uploading piece. err: <nil>
Jul 26 23:47:03.029323184 (1h55m18.810256565s) [noise=true , address=77.90.31.12:4590 , piece_id=EGHHI56SSVPTYOVSBDCG2ZTZM65DLXCRK6YLPWZNSGCZFDLBTEZA, seg_pos={0 0}, upload=1766]: Done uploading piece. err: <nil>
Jul 26 23:47:31.471281253 (1h55m47.252214628s) [noise=true , address=104.36.82.7:4912 , piece_id=IYGFHYQPGE54VZLRQ4X47OE63PENODSP3TCLF6AANGU5TVGCLYUQ, seg_pos={0 0}, upload=1766]: Done uploading piece. err: ecclient: upload failed (node:1gwuDRBC5oJXgZbP6FqP9tc8UB6aJt2uexUgSuzHCovw27v1NR, address:104.36.82.7:4912): protocol: rpc: dial tcp 104.36.82.7:4912: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 26 23:47:31.471351103 (1h55m47.252284525s) [noise=true , address=104.36.82.4:4803 , piece_id=NFHH4SZHNOMJ6ZFIRS5MHPONERX4EMNO6LZIT3M25LSSH57UDXFQ, seg_pos={0 0}, upload=1766]: Done uploading piece. err: ecclient: upload failed (node:12HWjkru3nFUa9hvA27sHqLEmRHSyXTV45phS5kMHKWM163M4H7, address:104.36.82.4:4803): protocol: write tcp 159.223.143.39:37766->104.36.82.4:4803: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 26 23:47:31.471383566 (1h55m47.252316935s) [noise=true , address=104.36.82.7:4909 , piece_id=LSPB6NGZQTTAGNU43I3Z6ICMEAYM6IY6DXTRCKW56QEJ34BM5NHA, seg_pos={0 0}, upload=1766]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:104.36.82.7:4909): protocol: write tcp 159.223.143.39:39038->104.36.82.7:4909: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 26 23:47:31.471421220 (1h55m47.252354596s) [noise=true , address=104.36.82.6:4874 , piece_id=GOMUVRZAQGEIKTGBLFZK5IMIKAS4MARHGN6ACXEEHTSXKAT3Z4KQ, seg_pos={0 0}, upload=1766]: Done uploading piece. err: ecclient: upload failed (node:12ay3ZqUvNcaX19JYT1hEaqydsxP6VoDjPavWXBNB34cestcm71, address:104.36.82.6:4874): protocol: rpc: dial tcp 104.36.82.6:4874: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 26 23:47:31.471469597 (1h55m47.252402965s) [seg_pos={0 0}, upload=1766]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Jul 26 23:47:31.498930488 (1h55m47.279863862s) [upload=1766]: Done uploading segments...
Jul 26 23:47:31.498966456 (1h55m47.279899821s) [upload=1766]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 30.029322092s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)antiblip:
52 out of 60 is failed (threshold=3) [.X......XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.] duration:
862.465011ms
FAIL (2025-07-26 23:47:23 UTC)
orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 10)
Duration: 862.48758ms
core
uplink
us1
vivint
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)antiblip:
52 out of 60 is failed (threshold=3) [..X......XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration:
30.02723166s
FAIL (2025-07-26 23:47:22 UTC)
orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 10)
Duration: 30.027258108s
core
uplink
us1
vivint
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
15 out of 60 is failed (threshold=3) [XX...X...X..XX.X...X..X.......X...X....XX..............X.X..] duration:
21.26828273s
FAIL (2025-07-26 23:47:04 UTC)
context deadline exceeded
Duration: 21.268317173s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
18 out of 60 is failed (threshold=3) [......XXX.X....XX.X.X.X.......XXX...XX.....X.XXX............] duration:
1.94635662s
FAIL (2025-07-26 23:46:54 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 1.94640006s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
47 out of 60 is failed (threshold=3) [............XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.XXXXXXXX] duration:
30.039771731s
FAIL (2025-07-26 23:46:47 UTC)
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 9)
======LOGS======
s={0 0}, upload=1747]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil>
Jul 26 23:46:17.524551936 (1h54m33.305485313s) [upload=1747]: Done uploading segments...
Jul 26 23:46:17.524598794 (1h54m33.30553217s) [upload=1747]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8)
Jul 26 23:46:17.624596775 (1h54m33.40553016s) [upload=1754]: Starting upload
Jul 26 23:46:17.624681788 (1h54m33.405615175s) [upload=1754]: Done starting upload
Jul 26 23:46:17.625039922 (1h54m33.405973284s) [upload=1754]: Uploading segments...
Jul 26 23:46:17.625108200 (1h54m33.406041562s) [upload=1754]: Got next segment. Inline: false
Jul 26 23:46:17.625149000 (1h54m33.406082386s) [upload=1754]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jul 26 23:46:17.625155538 (1h54m33.406088925s) [upload=1754]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jul 26 23:46:17.655075907 (1h54m33.436009295s) [seg_pos={0 0}, upload=1754]: Begin upload segment...
Jul 26 23:46:17.655137255 (1h54m33.43607085s) [seg_pos={0 0}, upload=1754]: Done begin upload segment.
Jul 26 23:46:17.655170740 (1h54m33.436104127s) [noise=true , address=104.36.82.11:4115 , piece_id=7PO7SY2IFSUSKGCFSTEYHB7RDT26Q525P6IN5X5CFW53ZXPUHRDQ, seg_pos={0 0}, upload=1754]: Uploading piece...
Jul 26 23:46:17.655597579 (1h54m33.436530958s) [noise=true , address=45.91.116.7:4009 , piece_id=PDAZ2CTPP3LACPCZ4NPIAB2MAIIOB274NOS7YQX72VEJ5UMXN6GQ, seg_pos={0 0}, upload=1754]: Uploading piece...
Jul 26 23:46:17.655984837 (1h54m33.436918224s) [noise=true , address=45.91.116.10:4082 , piece_id=UOTJQFI33SQPSQZMNSLETPXE4JCW4WDJ4J334VQKUA425R2JAAAQ, seg_pos={0 0}, upload=1754]: Uploading piece...
Jul 26 23:46:17.656528096 (1h54m33.437461475s) [noise=true , address=45.91.116.4:4940 , piece_id=GDHFUEEC7D3Y6A4MMRBJWRTNJXRJ2JWGJFSGV575XHCRAHGUPIMA, seg_pos={0 0}, upload=1754]: Uploading piece...
Jul 26 23:46:17.656879091 (1h54m33.437812474s) [noise=true , address=104.36.82.14:4223 , piece_id=EOFC4DVGIVEC4YMI3WI4JCKCMGQPPEGZ6LBC76FZTMAMLQ7VF6IQ, seg_pos={0 0}, upload=1754]: Uploading piece...
Jul 26 23:46:17.657222660 (1h54m33.438156046s) [noise=true , address=104.36.82.3:4767 , piece_id=6H3PKDYKCPSGYZN5YV75ZTTV2FLOQYXI3RFO5HKH5QIZ74QFL3HQ, seg_pos={0 0}, upload=1754]: Uploading piece...
Jul 26 23:46:17.657578618 (1h54m33.438512005s) [noise=true , address=45.91.116.7:4010 , piece_id=4EGNACS2CUB2TDRFFXEGGPT4O4CUCPGZYTHMMRBBNGUK46QW4GRA, seg_pos={0 0}, upload=1754]: Uploading piece...
Jul 26 23:46:17.657902329 (1h54m33.438835713s) [noise=true , address=104.36.82.6:4874 , piece_id=TTCCDLXB3Y7J4OMKFSWEHH762ZRFQTDO7ULTM647HIOC4VZFE3VQ, seg_pos={0 0}, upload=1754]: Uploading piece...
Jul 26 23:46:17.658191074 (1h54m33.439124454s) [noise=true , address=104.36.82.5:4838 , piece_id=NCUDT2LX5WWLIAVEC66NBYAXROJGYAM2UYP5TNQPK6FSO4CWKPNQ, seg_pos={0 0}, upload=1754]: Uploading piece...
Jul 26 23:46:17.658492019 (1h54m33.439425405s) [noise=true , address=104.36.82.7:4912 , piece_id=E26RTIRTXF6AE4FD4QYSB4BDZAA7AH4MVB56QMSJAM46EVO6S5AQ, seg_pos={0 0}, upload=1754]: Uploading piece...
Jul 26 23:46:17.658782220 (1h54m33.4397156s) [noise=true , address=104.36.82.11:4113 , piece_id=VB2HSMCNMM2NCGSDCFZCOWDXWGGJUFM2FQTDF5LNEGHFXO4RQUAA, seg_pos={0 0}, upload=1754]: Uploading piece...
Jul 26 23:46:17.659321098 (1h54m33.440254475s) [noise=true , address=45.91.116.8:4047 , piece_id=ONF5J6732S6FLJMYO6A4MKQEIDTBOWBE2UZ5BCVXY56A7GZ64PIQ, seg_pos={0 0}, upload=1754]: Uploading piece...
Jul 26 23:46:17.659765533 (1h54m33.440698909s) [noise=true , address=45.91.116.5:4297 , piece_id=HQK3FUYO7SFJGQN33ADHIICOZBLUKSCCH42DGATPAFVNEVVF3GGQ, seg_pos={0 0}, upload=1754]: Uploading piece...
Jul 26 23:46:17.660232930 (1h54m33.441166306s) [noise=true , address=45.91.116.10:4083 , piece_id=4XZCNJRB7XPPQJFZDICZVLO5KNN7G7Z44S42HUBLXRG2NZUYDB5Q, seg_pos={0 0}, upload=1754]: Uploading piece...
Jul 26 23:46:17.660638792 (1h54m33.441572178s) [noise=true , address=77.90.31.10:4518 , piece_id=MEMHKZDGOL5F4WDBQNOBIUGVK33PQBH7CQZRTGOPHHKXBNVDR3NA, seg_pos={0 0}, upload=1754]: Uploading piece...
Jul 26 23:46:17.661014017 (1h54m33.441947393s) [noise=true , address=45.91.116.12:4225 , piece_id=N3KUTSOCDBJ73CS7XIJMS2NJFM2S74ZQ344MIRT3QSOT6BHP6NAQ, seg_pos={0 0}, upload=1754]: Uploading piece...
Jul 26 23:46:17.742771369 (1h54m33.523704749s) [upload=1754]: Next returned nil segment
Jul 26 23:46:17.742810540 (1h54m33.523743904s) [upload=1754]: Waiting for error group managing segments...
Jul 26 23:46:17.753482767 (1h54m33.53441613s) [noise=true , address=45.91.116.7:4009 , piece_id=PDAZ2CTPP3LACPCZ4NPIAB2MAIIOB274NOS7YQX72VEJ5UMXN6GQ, seg_pos={0 0}, upload=1754]: Done uploading piece. err: <nil>
Jul 26 23:46:17.753537613 (1h54m33.534470981s) [noise=true , address=45.91.116.10:4083 , piece_id=4XZCNJRB7XPPQJFZDICZVLO5KNN7G7Z44S42HUBLXRG2NZUYDB5Q, seg_pos={0 0}, upload=1754]: Done uploading piece. err: <nil>
Jul 26 23:46:17.754145497 (1h54m33.535078869s) [noise=true , address=45.91.116.8:4047 , piece_id=ONF5J6732S6FLJMYO6A4MKQEIDTBOWBE2UZ5BCVXY56A7GZ64PIQ, seg_pos={0 0}, upload=1754]: Done uploading piece. err: <nil>
Jul 26 23:46:17.754171756 (1h54m33.535105188s) [noise=true , address=77.90.31.10:4518 , piece_id=MEMHKZDGOL5F4WDBQNOBIUGVK33PQBH7CQZRTGOPHHKXBNVDR3NA, seg_pos={0 0}, upload=1754]: Done uploading piece. err: <nil>
Jul 26 23:46:17.754601308 (1h54m33.53553468s) [noise=true , address=45.91.116.10:4082 , piece_id=UOTJQFI33SQPSQZMNSLETPXE4JCW4WDJ4J334VQKUA425R2JAAAQ, seg_pos={0 0}, upload=1754]: Done uploading piece. err: <nil>
Jul 26 23:46:17.755125717 (1h54m33.53605909s) [noise=true , address=45.91.116.4:4940 , piece_id=GDHFUEEC7D3Y6A4MMRBJWRTNJXRJ2JWGJFSGV575XHCRAHGUPIMA, seg_pos={0 0}, upload=1754]: Done uploading piece. err: <nil>
Jul 26 23:46:17.755155528 (1h54m33.536088895s) [noise=true , address=45.91.116.12:4225 , piece_id=N3KUTSOCDBJ73CS7XIJMS2NJFM2S74ZQ344MIRT3QSOT6BHP6NAQ, seg_pos={0 0}, upload=1754]: Done uploading piece. err: <nil>
Jul 26 23:46:17.755212459 (1h54m33.536145837s) [noise=true , address=45.91.116.7:4010 , piece_id=4EGNACS2CUB2TDRFFXEGGPT4O4CUCPGZYTHMMRBBNGUK46QW4GRA, seg_pos={0 0}, upload=1754]: Done uploading piece. err: <nil>
Jul 26 23:46:17.908402062 (1h54m33.689335425s) [noise=true , address=45.91.116.5:4297 , piece_id=HQK3FUYO7SFJGQN33ADHIICOZBLUKSCCH42DGATPAFVNEVVF3GGQ, seg_pos={0 0}, upload=1754]: Done uploading piece. err: <nil>
Jul 26 23:46:47.594360102 (1h55m3.375293486s) [noise=true , address=104.36.82.6:4874 , piece_id=TTCCDLXB3Y7J4OMKFSWEHH762ZRFQTDO7ULTM647HIOC4VZFE3VQ, seg_pos={0 0}, upload=1754]: Done uploading piece. err: ecclient: upload failed (node:12ay3ZqUvNcaX19JYT1hEaqydsxP6VoDjPavWXBNB34cestcm71, address:104.36.82.6:4874): protocol: write tcp 159.223.143.39:48862->104.36.82.6:4874: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 26 23:46:47.595112735 (1h55m3.376046115s) [noise=true , address=104.36.82.7:4912 , piece_id=E26RTIRTXF6AE4FD4QYSB4BDZAA7AH4MVB56QMSJAM46EVO6S5AQ, seg_pos={0 0}, upload=1754]: Done uploading piece. err: ecclient: upload failed (node:1gwuDRBC5oJXgZbP6FqP9tc8UB6aJt2uexUgSuzHCovw27v1NR, address:104.36.82.7:4912): protocol: rpc: dial tcp 104.36.82.7:4912: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 26 23:46:47.595193825 (1h55m3.376127208s) [noise=true , address=104.36.82.11:4115 , piece_id=7PO7SY2IFSUSKGCFSTEYHB7RDT26Q525P6IN5X5CFW53ZXPUHRDQ, seg_pos={0 0}, upload=1754]: Done uploading piece. err: ecclient: upload failed (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR, address:104.36.82.11:4115): protocol: rpc: dial tcp 104.36.82.11:4115: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 26 23:46:47.595235841 (1h55m3.376169219s) [noise=true , address=104.36.82.14:4223 , piece_id=EOFC4DVGIVEC4YMI3WI4JCKCMGQPPEGZ6LBC76FZTMAMLQ7VF6IQ, seg_pos={0 0}, upload=1754]: Done uploading piece. err: ecclient: upload failed (node:12C59b8knr62ANoUWyaJZDHm2os1tFm3SKL8d34eKsdpZGHYQWE, address:104.36.82.14:4223): protocol: rpc: dial tcp 104.36.82.14:4223: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 26 23:46:47.595269175 (1h55m3.376202558s) [noise=true , address=104.36.82.3:4767 , piece_id=6H3PKDYKCPSGYZN5YV75ZTTV2FLOQYXI3RFO5HKH5QIZ74QFL3HQ, seg_pos={0 0}, upload=1754]: Done uploading piece. err: ecclient: upload failed (node:1TiDHs6xp3atLDHHv1i2564ibyPGLKfFyHx3bgZWGRRN5cwUi5, address:104.36.82.3:4767): protocol: rpc: dial tcp 104.36.82.3:4767: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 26 23:46:47.595302169 (1h55m3.376235549s) [noise=true , address=104.36.82.5:4838 , piece_id=NCUDT2LX5WWLIAVEC66NBYAXROJGYAM2UYP5TNQPK6FSO4CWKPNQ, seg_pos={0 0}, upload=1754]: Done uploading piece. err: ecclient: upload failed (node:1iSxSmovuEAgNmjE3yCqXKtiRijYbcEcTtUdNxU3iC1ozTxfwL, address:104.36.82.5:4838): protocol: rpc: dial tcp 104.36.82.5:4838: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 26 23:46:47.595347255 (1h55m3.376280637s) [noise=true , address=104.36.82.11:4113 , piece_id=VB2HSMCNMM2NCGSDCFZCOWDXWGGJUFM2FQTDF5LNEGHFXO4RQUAA, seg_pos={0 0}, upload=1754]: Done uploading piece. err: ecclient: upload failed (node:1SQKETWozrz4PcDC3yhy6V5d8Ryo6NNiyWWEkE5xRcmUj9B6UV, address:104.36.82.11:4113): protocol: write tcp 159.223.143.39:39866->104.36.82.11:4113: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 26 23:46:47.595370646 (1h55m3.376304037s) [seg_pos={0 0}, upload=1754]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil>
Jul 26 23:46:47.625410173 (1h55m3.406343561s) [upload=1754]: Done uploading segments...
Jul 26 23:46:47.625470564 (1h55m3.406403949s) [upload=1754]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9)
Duration: 30.039797328s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)antiblip:
52 out of 60 is failed (threshold=3) [X.XX....XXXX.XXXXX.X.XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration:
1m0.033872509s
FAIL (2025-07-26 23:46:38 UTC)
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 9)
======LOGS======
, piece_id=XPCWZYOKZAFB3BA6NDWY6RCIA3THYOCCRAOB7UINHCASKAVC37KQ, seg_pos={0 0}, upload=1745]: Uploading piece...
Jul 26 23:45:38.571912330 (1h53m54.352845712s) [noise=true , address=79.127.201.215:4355 , piece_id=LOREMUNEAOQ3ASXQLU4Q7ISP2LFVIINIGKF2Y54YIEA4XPMJ627A, seg_pos={0 0}, upload=1745]: Uploading piece...
Jul 26 23:45:38.572544233 (1h53m54.353477616s) [noise=true , address=104.36.82.47:4536 , piece_id=SR27WNQKZMNV53T65EA6ZDCQQTDL42NQ7NPIF3DU4DTFKXBHHIDA, seg_pos={0 0}, upload=1745]: Uploading piece...
Jul 26 23:45:38.572966807 (1h53m54.353900199s) [noise=true , address=77.90.31.17:4799 , piece_id=FYMB36VLWYJVN3XAIZBWB5KH4DJDJNIEUQDHCEGKZ5FGKB4NCDRA, seg_pos={0 0}, upload=1745]: Uploading piece...
Jul 26 23:45:38.573391378 (1h53m54.354324756s) [noise=true , address=104.36.82.56:4229 , piece_id=D3ITSMS7GM4NCMQV44JFE2IGVN5XI7SDAVNHRG6I6BYEDF5KXLZA, seg_pos={0 0}, upload=1745]: Uploading piece...
Jul 26 23:45:38.573845783 (1h53m54.35477916s) [noise=true , address=79.127.219.38:4004 , piece_id=7TVBMQKTFI554JKMI4WJJ2T4C4AZ3IVTDNTJZHBALPARD7JG4MEQ, seg_pos={0 0}, upload=1745]: Uploading piece...
Jul 26 23:45:38.574267669 (1h53m54.355201046s) [noise=true , address=104.36.82.46:4509 , piece_id=6QEQM3Y6IU5Y3CHGYFHSC4652TGPYEB6Q6GG4RT6OZHGLM6HEMKQ, seg_pos={0 0}, upload=1745]: Uploading piece...
Jul 26 23:45:38.574709734 (1h53m54.35564312s) [noise=true , address=104.36.82.53:4128 , piece_id=BTLRM2F7OZ2NZZCQJZXT2XJ2GF46YKHYXRVU4XOOZASTLO3YYZXA, seg_pos={0 0}, upload=1745]: Uploading piece...
Jul 26 23:45:38.575215208 (1h53m54.356148594s) [noise=true , address=64.49.13.140:4508 , piece_id=XV6DTKPFOHFO5765LYJ2PUMIEDBAQHO37BCOMEL5UVHK7OA6DYZA, seg_pos={0 0}, upload=1745]: Uploading piece...
Jul 26 23:45:38.575682258 (1h53m54.35661564s) [noise=true , address=79.127.205.246:4280 , piece_id=EVFHHFB7SBQFAP64QR56QU3X2KBMPCGNDTF6TCMN5EO6NQJKMXNA, seg_pos={0 0}, upload=1745]: Uploading piece...
Jul 26 23:45:38.576219733 (1h53m54.357153121s) [noise=true , address=79.127.220.97:4361 , piece_id=FTWC5PMPCTOUGNBWTFT6HVGSQ25CCWIXHVPOA246XOZPEBMEHP6A, seg_pos={0 0}, upload=1745]: Uploading piece...
Jul 26 23:45:38.576719265 (1h53m54.357652644s) [noise=true , address=77.90.31.21:4895 , piece_id=GIO2QXWPRLEB2HSEPASEUVLU4A3OKZIH4AME4Q5XMLPYUT6RMUAA, seg_pos={0 0}, upload=1745]: Uploading piece...
Jul 26 23:45:38.577249070 (1h53m54.358182456s) [noise=true , address=79.127.205.235:4687 , piece_id=DDSOVQH2T44BYM2FEFSO6EU4NDN3FJ3PU3D57SKEIFDUK2WV5YPA, seg_pos={0 0}, upload=1745]: Uploading piece...
Jul 26 23:45:38.580325096 (1h53m54.361258487s) [noise=true , address=77.90.31.15:4715 , piece_id=EAEXN7IZRW5KKXOLUPMGML3GXMIYJTOORCQA4TGJ7QYFD2ZOJU3A, seg_pos={0 0}, upload=1745]: Uploading piece...
Jul 26 23:45:38.580921670 (1h53m54.36185506s) [noise=true , address=104.36.82.54:4175 , piece_id=HIYPJX4N7N5K6FLVYA4327IWS7M55BUSSJKELUV7NG2IWTSOZKVA, seg_pos={0 0}, upload=1745]: Uploading piece...
Jul 26 23:45:38.581391842 (1h53m54.362325209s) [noise=true , address=64.49.13.138:4483 , piece_id=TZAKN36VPGXHMV5JTZ7GPX2FSY6XP2SEQ5UPFADYDFU5BE7YSNZQ, seg_pos={0 0}, upload=1745]: Uploading piece...
Jul 26 23:45:38.582279540 (1h53m54.363212928s) [noise=true , address=104.36.82.20:4440 , piece_id=VJOTKS3DPZCFVNK6AOY2NH32GWX3BMKDRDNPLVWKZRJX7HWRTTYA, seg_pos={0 0}, upload=1745]: Uploading piece...
Jul 26 23:45:38.709721926 (1h53m54.49065531s) [upload=1745]: Next returned nil segment
Jul 26 23:45:38.709742350 (1h53m54.490675716s) [upload=1745]: Waiting for error group managing segments...
Jul 26 23:45:38.717073278 (1h53m54.498006649s) [noise=true , address=79.127.219.38:4004 , piece_id=7TVBMQKTFI554JKMI4WJJ2T4C4AZ3IVTDNTJZHBALPARD7JG4MEQ, seg_pos={0 0}, upload=1745]: Done uploading piece. err: <nil>
Jul 26 23:45:38.753237211 (1h53m54.534170581s) [noise=true , address=77.90.31.17:4799 , piece_id=FYMB36VLWYJVN3XAIZBWB5KH4DJDJNIEUQDHCEGKZ5FGKB4NCDRA, seg_pos={0 0}, upload=1745]: Done uploading piece. err: <nil>
Jul 26 23:45:38.756214249 (1h53m54.537147611s) [noise=true , address=77.90.31.21:4895 , piece_id=GIO2QXWPRLEB2HSEPASEUVLU4A3OKZIH4AME4Q5XMLPYUT6RMUAA, seg_pos={0 0}, upload=1745]: Done uploading piece. err: <nil>
Jul 26 23:45:38.787695172 (1h53m54.568628534s) [noise=true , address=77.90.31.15:4715 , piece_id=EAEXN7IZRW5KKXOLUPMGML3GXMIYJTOORCQA4TGJ7QYFD2ZOJU3A, seg_pos={0 0}, upload=1745]: Done uploading piece. err: <nil>
Jul 26 23:45:39.210707133 (1h53m54.991640495s) [noise=true , address=79.127.205.246:4280 , piece_id=EVFHHFB7SBQFAP64QR56QU3X2KBMPCGNDTF6TCMN5EO6NQJKMXNA, seg_pos={0 0}, upload=1745]: Done uploading piece. err: <nil>
Jul 26 23:45:40.003715113 (1h53m55.784648494s) [noise=true , address=64.49.13.140:4508 , piece_id=XV6DTKPFOHFO5765LYJ2PUMIEDBAQHO37BCOMEL5UVHK7OA6DYZA, seg_pos={0 0}, upload=1745]: Done uploading piece. err: ecclient: upload failed (node:1diUU2skKzXPjYy3Mu3iDDjnG5W8Z2cxCRMxDCH9cQGg2TdEiR, address:64.49.13.140:4508): protocol: storage node overloaded, request limit: 2000
Jul 26 23:45:40.323250495 (1h53m56.104183865s) [noise=true , address=79.127.205.235:4687 , piece_id=DDSOVQH2T44BYM2FEFSO6EU4NDN3FJ3PU3D57SKEIFDUK2WV5YPA, seg_pos={0 0}, upload=1745]: Done uploading piece. err: <nil>
Jul 26 23:45:41.328958532 (1h53m57.109891907s) [noise=true , address=79.127.220.97:4361 , piece_id=FTWC5PMPCTOUGNBWTFT6HVGSQ25CCWIXHVPOA246XOZPEBMEHP6A, seg_pos={0 0}, upload=1745]: Done uploading piece. err: <nil>
Jul 26 23:45:42.814520097 (1h53m58.595453478s) [noise=true , address=64.49.13.138:4483 , piece_id=TZAKN36VPGXHMV5JTZ7GPX2FSY6XP2SEQ5UPFADYDFU5BE7YSNZQ, seg_pos={0 0}, upload=1745]: Done uploading piece. err: <nil>
Jul 26 23:46:19.496410067 (1h54m35.277343429s) [noise=true , address=79.127.201.215:4355 , piece_id=LOREMUNEAOQ3ASXQLU4Q7ISP2LFVIINIGKF2Y54YIEA4XPMJ627A, seg_pos={0 0}, upload=1745]: Done uploading piece. err: <nil>
Jul 26 23:46:38.468888642 (1h54m54.24982201s) [noise=true , address=104.36.82.54:4175 , piece_id=HIYPJX4N7N5K6FLVYA4327IWS7M55BUSSJKELUV7NG2IWTSOZKVA, seg_pos={0 0}, upload=1745]: Done uploading piece. err: ecclient: upload failed (node:12ou7jwFyJP3mNoALddniUTDT9duVyQKZjeq4xHCyx6sRWpwEPJ, address:104.36.82.54:4175): protocol: context deadline exceeded; context deadline exceeded
Jul 26 23:46:38.469217903 (1h54m54.250151384s) [noise=true , address=104.36.82.55:4194 , piece_id=XPCWZYOKZAFB3BA6NDWY6RCIA3THYOCCRAOB7UINHCASKAVC37KQ, seg_pos={0 0}, upload=1745]: Done uploading piece. err: ecclient: upload failed (node:12kta1F6RNDP9GJdkiUG3x1Nk6juWJ2tmSDr599DYomSq9C1BBh, address:104.36.82.55:4194): protocol: write tcp 159.223.143.39:33582->104.36.82.55:4194: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 26 23:46:38.469333760 (1h54m54.250267151s) [noise=true , address=104.36.82.46:4509 , piece_id=6QEQM3Y6IU5Y3CHGYFHSC4652TGPYEB6Q6GG4RT6OZHGLM6HEMKQ, seg_pos={0 0}, upload=1745]: Done uploading piece. err: ecclient: upload failed (node:1y2NmauNvCyrMJ5z66Lihb3h3C9nwdJYqfSfNmC5dwDGYQqUKh, address:104.36.82.46:4509): protocol: write tcp 159.223.143.39:52422->104.36.82.46:4509: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 26 23:46:38.469490069 (1h54m54.250423432s) [noise=true , address=104.36.82.53:4128 , piece_id=BTLRM2F7OZ2NZZCQJZXT2XJ2GF46YKHYXRVU4XOOZASTLO3YYZXA, seg_pos={0 0}, upload=1745]: Done uploading piece. err: ecclient: upload failed (node:125ZMc6ed9qgBRtBC5N4zNvGonip3GNAEBPt3auF6dBGteX46Pf, address:104.36.82.53:4128): protocol: write tcp 159.223.143.39:53512->104.36.82.53:4128: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 26 23:46:38.469648942 (1h54m54.250582307s) [noise=true , address=104.36.82.20:4457 , piece_id=77AJOFAEB4EAG7C7TVLZFATTRBLQABPZVJ2TQU2LEY6FGAFFCSQQ, seg_pos={0 0}, upload=1745]: Done uploading piece. err: ecclient: upload failed (node:16sDrSF9mwW9cQ2h8N5TfUKrnwPAwtEfed6AwCEAT95r5iY9vQ, address:104.36.82.20:4457): protocol: write tcp 159.223.143.39:60032->104.36.82.20:4457: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 26 23:46:38.469734893 (1h54m54.250668273s) [noise=true , address=104.36.82.47:4536 , piece_id=SR27WNQKZMNV53T65EA6ZDCQQTDL42NQ7NPIF3DU4DTFKXBHHIDA, seg_pos={0 0}, upload=1745]: Done uploading piece. err: ecclient: upload failed (node:1WoVNAp5RfVviaxN4RWCUDEhxLKsdGB8bgE6gcUcgZKcM4Cz1s, address:104.36.82.47:4536): protocol: write tcp 159.223.143.39:53182->104.36.82.47:4536: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 26 23:46:38.469870044 (1h54m54.250803421s) [noise=true , address=104.36.82.56:4229 , piece_id=D3ITSMS7GM4NCMQV44JFE2IGVN5XI7SDAVNHRG6I6BYEDF5KXLZA, seg_pos={0 0}, upload=1745]: Done uploading piece. err: ecclient: upload failed (node:12wWYnhWHSqqAFgje4p9Cr7WYaJ6dJWsCqvrx8ZqVFoakjuFeka, address:104.36.82.56:4229): protocol: write tcp 159.223.143.39:50170->104.36.82.56:4229: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 26 23:46:38.469894808 (1h54m54.25082819s) [noise=true , address=104.36.82.20:4440 , piece_id=VJOTKS3DPZCFVNK6AOY2NH32GWX3BMKDRDNPLVWKZRJX7HWRTTYA, seg_pos={0 0}, upload=1745]: Done uploading piece. err: ecclient: upload failed (node:12GfuKp6kJEy1NTHpo9i21UDm62hrAtKq7124Z6FUehNyyLQ33i, address:104.36.82.20:4440): protocol: write tcp 159.223.143.39:52280->104.36.82.20:4440: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 26 23:46:38.469935760 (1h54m54.250869146s) [seg_pos={0 0}, upload=1745]: Done waiting for segment. successful: 9 optimal: 14 errs: <nil>
Jul 26 23:46:38.500714241 (1h54m54.281647611s) [upload=1745]: Done uploading segments...
Jul 26 23:46:38.500770670 (1h54m54.281704048s) [upload=1745]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 9)
Duration: 1m0.033918276s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)antiblip:
51 out of 60 is failed (threshold=3) [...X......XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration:
30.070774948s
FAIL (2025-07-26 23:46:34 UTC)
orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 10)
Duration: 30.070813611s
core
uplink
us1
vivint
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
18 out of 60 is failed (threshold=3) [.......XXX.X....XX.X.X.X.......XXX...XX.....X.XXX...........] duration:
1.840566776s
FAIL (2025-07-26 23:46:31 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 1.840591407s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
46 out of 60 is failed (threshold=3) [.............XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.XXXXXXX] duration:
30.027787578s
FAIL (2025-07-26 23:46:17 UTC)
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 8)
======LOGS======
24.191083224s) [upload=1728]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 6)
Jul 26 23:45:47.528550027 (1h54m3.309483396s) [upload=1747]: Starting upload
Jul 26 23:45:47.528640839 (1h54m3.309574218s) [upload=1747]: Done starting upload
Jul 26 23:45:47.528745650 (1h54m3.309679015s) [upload=1747]: Uploading segments...
Jul 26 23:45:47.529023904 (1h54m3.309957269s) [upload=1747]: Got next segment. Inline: false
Jul 26 23:45:47.529113546 (1h54m3.310046926s) [upload=1747]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jul 26 23:45:47.529121441 (1h54m3.31005482s) [upload=1747]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jul 26 23:45:47.558220849 (1h54m3.339154215s) [seg_pos={0 0}, upload=1747]: Begin upload segment...
Jul 26 23:45:47.558283441 (1h54m3.33921682s) [seg_pos={0 0}, upload=1747]: Done begin upload segment.
Jul 26 23:45:47.558351476 (1h54m3.339284867s) [noise=true , address=104.36.82.11:4115 , piece_id=TFH32Q4SWNEBDGDUTIIBRF23IZIIR3HBPKJ2XXBECEL2ZAYJGEDQ, seg_pos={0 0}, upload=1747]: Uploading piece...
Jul 26 23:45:47.558369352 (1h54m3.339302731s) [noise=true , address=104.36.82.13:4187 , piece_id=62DZZ7VXJMTOEDKM3X4PCC62RT7DZFIYBAKJITPFJ33E62M4RTVQ, seg_pos={0 0}, upload=1747]: Uploading piece...
Jul 26 23:45:47.558827957 (1h54m3.339761335s) [noise=true , address=77.90.31.19:4842 , piece_id=4XPIOROXEUXWGFSSGILHJQUHEMSOBZWC7OM3YK5NKQTJ7UYJIHTQ, seg_pos={0 0}, upload=1747]: Uploading piece...
Jul 26 23:45:47.558903236 (1h54m3.33983662s) [noise=true , address=45.91.116.13:4154 , piece_id=5KPL33WBEOW26RPXI4ZO4UBD26NAIYM24L3WT2TE35NHZRGZXZVQ, seg_pos={0 0}, upload=1747]: Uploading piece...
Jul 26 23:45:47.559214896 (1h54m3.34014828s) [noise=true , address=104.36.82.12:4151 , piece_id=BYQLIFBJM2SV2GRDQSVX2DCBY5EDJY4ZOSPHFV7Y4QWF74QJMURA, seg_pos={0 0}, upload=1747]: Uploading piece...
Jul 26 23:45:47.559661333 (1h54m3.340594713s) [noise=true , address=45.91.116.12:4227 , piece_id=YAB7MCF4WHAYQUB7ORLXZRBGMKBJ4X3XPQUH26ULZTWQH7IY6KMQ, seg_pos={0 0}, upload=1747]: Uploading piece...
Jul 26 23:45:47.559999040 (1h54m3.340932417s) [noise=true , address=45.91.116.8:4047 , piece_id=57T5JDPBGBYXXLEHQPXWZ2QWNHL7MJQUIJ52CDKGCCRCOTB2MLRA, seg_pos={0 0}, upload=1747]: Uploading piece...
Jul 26 23:45:47.560036872 (1h54m3.340970257s) [noise=true , address=104.36.82.11:4114 , piece_id=MMP6F2OP57H57YPYHJVHKOTDKMPGW2UJL6GSN7VKTBYXWN2YHV7Q, seg_pos={0 0}, upload=1747]: Uploading piece...
Jul 26 23:45:47.560402073 (1h54m3.341335454s) [noise=true , address=104.36.82.15:4259 , piece_id=PG6G724ZDDZ7KUTJQGFEKWI6BLABYPOD4ZC3MUYNN5JKIIMCC2SA, seg_pos={0 0}, upload=1747]: Uploading piece...
Jul 26 23:45:47.560498278 (1h54m3.341431655s) [noise=true , address=104.36.82.14:4221 , piece_id=SLEEOKCQPOVD656KFD7QNFVYF4ZFVBKJYYMVFRDILH6Q3HSTKBVQ, seg_pos={0 0}, upload=1747]: Uploading piece...
Jul 26 23:45:47.560707268 (1h54m3.341640635s) [noise=true , address=104.36.82.12:4149 , piece_id=T6UFFEOQVPWMUJVBIW5C2ZMQOLNCS2QE4IL7KRFMRDYJ6J2WNE5A, seg_pos={0 0}, upload=1747]: Uploading piece...
Jul 26 23:45:47.560776189 (1h54m3.341709568s) [noise=true , address=77.90.31.13:4627 , piece_id=YSSOPU5BKDLVEDTNGYJLPOC3YH4NWNMLQCXEVGVXMNTB6E3EB2QA, seg_pos={0 0}, upload=1747]: Uploading piece...
Jul 26 23:45:47.561074124 (1h54m3.342007503s) [noise=true , address=77.90.31.14:4662 , piece_id=Y6QBRXDNYS56E5ZE37TWO3DNBQHKTSBME3IO2OQ6M3MCH2LBR4BA, seg_pos={0 0}, upload=1747]: Uploading piece...
Jul 26 23:45:47.561367507 (1h54m3.342300882s) [noise=true , address=45.91.116.6:4975 , piece_id=BSA3QO43FCGA7D6F74TXTFCSHSG7RXC7KZCUPCNLLEFMNHQ4TGHQ, seg_pos={0 0}, upload=1747]: Uploading piece...
Jul 26 23:45:47.561671774 (1h54m3.342605152s) [noise=true , address=45.91.116.5:4297 , piece_id=ZV6TFLM2X7VASLP2GEME7G7NYWNRLYYMWMT7IZRK5IEWHPAHJ3MQ, seg_pos={0 0}, upload=1747]: Uploading piece...
Jul 26 23:45:47.561996146 (1h54m3.342929522s) [noise=true , address=104.36.82.7:4909 , piece_id=WSHEAAFVGA63SPALM2XPNSNNPSRZPKK2J4VTP3PT3XOYBTFP6LSQ, seg_pos={0 0}, upload=1747]: Uploading piece...
Jul 26 23:45:47.649771810 (1h54m3.430705175s) [upload=1747]: Next returned nil segment
Jul 26 23:45:47.649792842 (1h54m3.430726211s) [upload=1747]: Waiting for error group managing segments...
Jul 26 23:45:47.653651879 (1h54m3.434585247s) [noise=true , address=45.91.116.6:4975 , piece_id=BSA3QO43FCGA7D6F74TXTFCSHSG7RXC7KZCUPCNLLEFMNHQ4TGHQ, seg_pos={0 0}, upload=1747]: Done uploading piece. err: <nil>
Jul 26 23:45:47.655490460 (1h54m3.436423829s) [noise=true , address=45.91.116.8:4047 , piece_id=57T5JDPBGBYXXLEHQPXWZ2QWNHL7MJQUIJ52CDKGCCRCOTB2MLRA, seg_pos={0 0}, upload=1747]: Done uploading piece. err: <nil>
Jul 26 23:45:47.656316568 (1h54m3.437249937s) [noise=true , address=45.91.116.13:4154 , piece_id=5KPL33WBEOW26RPXI4ZO4UBD26NAIYM24L3WT2TE35NHZRGZXZVQ, seg_pos={0 0}, upload=1747]: Done uploading piece. err: <nil>
Jul 26 23:45:47.658051626 (1h54m3.438984993s) [noise=true , address=77.90.31.14:4662 , piece_id=Y6QBRXDNYS56E5ZE37TWO3DNBQHKTSBME3IO2OQ6M3MCH2LBR4BA, seg_pos={0 0}, upload=1747]: Done uploading piece. err: <nil>
Jul 26 23:45:47.663908347 (1h54m3.444841723s) [noise=true , address=45.91.116.12:4227 , piece_id=YAB7MCF4WHAYQUB7ORLXZRBGMKBJ4X3XPQUH26ULZTWQH7IY6KMQ, seg_pos={0 0}, upload=1747]: Done uploading piece. err: <nil>
Jul 26 23:45:47.889974750 (1h54m3.67090812s) [noise=true , address=45.91.116.5:4297 , piece_id=ZV6TFLM2X7VASLP2GEME7G7NYWNRLYYMWMT7IZRK5IEWHPAHJ3MQ, seg_pos={0 0}, upload=1747]: Done uploading piece. err: <nil>
Jul 26 23:45:49.022796696 (1h54m4.803730077s) [noise=true , address=77.90.31.13:4627 , piece_id=YSSOPU5BKDLVEDTNGYJLPOC3YH4NWNMLQCXEVGVXMNTB6E3EB2QA, seg_pos={0 0}, upload=1747]: Done uploading piece. err: <nil>
Jul 26 23:45:49.105919749 (1h54m4.886853117s) [noise=true , address=77.90.31.19:4842 , piece_id=4XPIOROXEUXWGFSSGILHJQUHEMSOBZWC7OM3YK5NKQTJ7UYJIHTQ, seg_pos={0 0}, upload=1747]: Done uploading piece. err: <nil>
Jul 26 23:46:17.498482600 (1h54m33.279415977s) [noise=true , address=104.36.82.12:4149 , piece_id=T6UFFEOQVPWMUJVBIW5C2ZMQOLNCS2QE4IL7KRFMRDYJ6J2WNE5A, seg_pos={0 0}, upload=1747]: Done uploading piece. err: ecclient: upload failed (node:12qexRLJM6TY935UaEWsj1X9ViyBvk2BM1XGjnq8kapnXXXmgbm, address:104.36.82.12:4149): protocol: rpc: dial tcp 104.36.82.12:4149: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 26 23:46:17.498555419 (1h54m33.279488797s) [noise=true , address=104.36.82.13:4187 , piece_id=62DZZ7VXJMTOEDKM3X4PCC62RT7DZFIYBAKJITPFJ33E62M4RTVQ, seg_pos={0 0}, upload=1747]: Done uploading piece. err: ecclient: upload failed (node:12cQxBB2L2CyymYx3xWp1ALj6p8QWr4A34sqvtG5zyeT7sVWv7x, address:104.36.82.13:4187): protocol: rpc: dial tcp 104.36.82.13:4187: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 26 23:46:17.498588278 (1h54m33.279521641s) [noise=true , address=104.36.82.12:4151 , piece_id=BYQLIFBJM2SV2GRDQSVX2DCBY5EDJY4ZOSPHFV7Y4QWF74QJMURA, seg_pos={0 0}, upload=1747]: Done uploading piece. err: ecclient: upload failed (node:1w6pwZMdeQKjt92qx6WbTWU5VPYJto5gS87f6iKvCBZH6LfyeB, address:104.36.82.12:4151): protocol: rpc: dial tcp 104.36.82.12:4151: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 26 23:46:17.498659084 (1h54m33.27959247s) [noise=true , address=104.36.82.11:4115 , piece_id=TFH32Q4SWNEBDGDUTIIBRF23IZIIR3HBPKJ2XXBECEL2ZAYJGEDQ, seg_pos={0 0}, upload=1747]: Done uploading piece. err: ecclient: upload failed (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR, address:104.36.82.11:4115): protocol: rpc: dial tcp 104.36.82.11:4115: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 26 23:46:17.498675455 (1h54m33.279608817s) [noise=true , address=104.36.82.11:4114 , piece_id=MMP6F2OP57H57YPYHJVHKOTDKMPGW2UJL6GSN7VKTBYXWN2YHV7Q, seg_pos={0 0}, upload=1747]: Done uploading piece. err: ecclient: upload failed (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN, address:104.36.82.11:4114): protocol: write tcp 159.223.143.39:47826->104.36.82.11:4114: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 26 23:46:17.498689802 (1h54m33.279623172s) [noise=true , address=104.36.82.7:4909 , piece_id=WSHEAAFVGA63SPALM2XPNSNNPSRZPKK2J4VTP3PT3XOYBTFP6LSQ, seg_pos={0 0}, upload=1747]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:104.36.82.7:4909): protocol: rpc: dial tcp 104.36.82.7:4909: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 26 23:46:17.498733751 (1h54m33.279667257s) [noise=true , address=104.36.82.15:4259 , piece_id=PG6G724ZDDZ7KUTJQGFEKWI6BLABYPOD4ZC3MUYNN5JKIIMCC2SA, seg_pos={0 0}, upload=1747]: Done uploading piece. err: ecclient: upload failed (node:1ET8PaAfWq2RcPNjucMqDbmnkMZg8qnkTw9dszwLzz765oBr7y, address:104.36.82.15:4259): protocol: rpc: dial tcp 104.36.82.15:4259: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 26 23:46:17.498783558 (1h54m33.279716935s) [noise=true , address=104.36.82.14:4221 , piece_id=SLEEOKCQPOVD656KFD7QNFVYF4ZFVBKJYYMVFRDILH6Q3HSTKBVQ, seg_pos={0 0}, upload=1747]: Done uploading piece. err: ecclient: upload failed (node:129qMN1kX8C3x3H1Epfc4pUbtVGU1xPiZj43QSLByCTPKqwUoJR, address:104.36.82.14:4221): protocol: write tcp 159.223.143.39:33490->104.36.82.14:4221: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 26 23:46:17.498804857 (1h54m33.279738224s) [seg_pos={0 0}, upload=1747]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil>
Jul 26 23:46:17.524551936 (1h54m33.305485313s) [upload=1747]: Done uploading segments...
Jul 26 23:46:17.524598794 (1h54m33.30553217s) [upload=1747]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8)
Duration: 30.027876714s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
18 out of 60 is failed (threshold=3) [........XXX.X....XX.X.X.X.......XXX...XX.....X.XXX..........] duration:
2.834682599s
FAIL (2025-07-26 23:46:10 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 2.834709429s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
15 out of 60 is failed (threshold=3) [.XX...X...X..XX.X...X..X.......X...X....XX..............X.X.] duration:
19.422835539s
FAIL (2025-07-26 23:46:00 UTC)
context deadline exceeded
Duration: 19.422862753s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)antiblip:
50 out of 60 is failed (threshold=3) [....X......XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration:
30.034225081s
FAIL (2025-07-26 23:45:48 UTC)
orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 6)
Duration: 30.034250318s
core
uplink
us1
vivint
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
15 out of 60 is failed (threshold=3) [..XX...X...X..XX.X...X..X.......X...X....XX..............X.X] duration:
1m0.001079875s
FAIL (2025-07-26 23:45:41 UTC)
context deadline exceeded
Duration: 1m0.001168071s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)antiblip:
51 out of 60 is failed (threshold=3) [.X.XX....XXXX.XXXXX.X.XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration:
1m0.03371062s
FAIL (2025-07-26 23:45:35 UTC)
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 9)
======LOGS======
s) [seg_pos={0 0}, upload=1726]: Done begin upload segment.
Jul 26 23:44:35.488979313 (1h52m51.269912699s) [noise=true , address=104.36.82.53:4149 , piece_id=5I4OY4D4WBZA3FB7HLKCPWDQAI7TKPFDP3IECR5J6TVHQ7HQ2K5Q, seg_pos={0 0}, upload=1726]: Uploading piece...
Jul 26 23:44:35.489001662 (1h52m51.269935044s) [noise=true , address=79.127.205.235:4691 , piece_id=34QHDCOHKOGY6YAAX5C2U44STARV6AQJFODJB4WTPGLWPQ47J56A, seg_pos={0 0}, upload=1726]: Uploading piece...
Jul 26 23:44:35.489541468 (1h52m51.270474856s) [noise=true , address=64.49.13.140:4508 , piece_id=KXAO5GNBNBWWQ3KMC4VIE5KAVIXATL3DBVZM4EUAPUKUKXF2WRGA, seg_pos={0 0}, upload=1726]: Uploading piece...
Jul 26 23:44:35.489575066 (1h52m51.270508451s) [noise=true , address=77.90.31.21:4912 , piece_id=PIA7P2VPFVLBH4GFYRHQ5KP5PRSCJCDOYXJYNZEDGKDYHULGBUBQ, seg_pos={0 0}, upload=1726]: Uploading piece...
Jul 26 23:44:35.490050535 (1h52m51.270983919s) [noise=true , address=104.36.82.46:4490 , piece_id=QFKKQU7UEDERMIXEIWGAEAUMRCGUANOC5ZMM7HOWUBFBFZL6EQIA, seg_pos={0 0}, upload=1726]: Uploading piece...
Jul 26 23:44:35.490125109 (1h52m51.271058494s) [noise=true , address=104.36.82.56:4247 , piece_id=2FDHHAMOQO6RGUFXLQH4UKNHYPOFYZTPEABXDHST6RVWJC5CJYCA, seg_pos={0 0}, upload=1726]: Uploading piece...
Jul 26 23:44:35.490686708 (1h52m51.271620098s) [noise=true , address=64.49.13.138:4482 , piece_id=6KPYSZAM4AJTFBVHDNLPJY7CGLGJSIKPB7S7N4G5F4ZIRE4G3SGQ, seg_pos={0 0}, upload=1726]: Uploading piece...
Jul 26 23:44:35.490794891 (1h52m51.271728282s) [noise=true , address=64.49.13.139:4478 , piece_id=RQBUGIMT7XHLF2FBZGUA4RTRYUN6FL45RIZH2YEUFBKPLBR7P6UA, seg_pos={0 0}, upload=1726]: Uploading piece...
Jul 26 23:44:35.491316121 (1h52m51.272249506s) [noise=true , address=156.146.43.225:4448 , piece_id=LIAN4KFQUGNFOX4PLKNJ6GHICUDTCEIFRIULLRHYENJUYPR4TF2Q, seg_pos={0 0}, upload=1726]: Uploading piece...
Jul 26 23:44:35.491691604 (1h52m51.272624987s) [noise=true , address=156.146.43.225:4449 , piece_id=ZVXMQF7BR7FMGBFFFF5UPAPACIOIWNOUK3JCOG62VOG4D24FC3TQ, seg_pos={0 0}, upload=1726]: Uploading piece...
Jul 26 23:44:35.492241107 (1h52m51.273174493s) [noise=true , address=77.90.31.11:4564 , piece_id=W4XJ3TSB5PXNGV2ZIASXSVNVOHFH4SEPUE3MBWWSGINNPLXEFP2Q, seg_pos={0 0}, upload=1726]: Uploading piece...
Jul 26 23:44:35.492725186 (1h52m51.273658573s) [noise=true , address=104.36.82.52:4107 , piece_id=DQKOQZXURQQDTVR6ELRV5CRIFXFWJFSIMHCIJSGKDXZKV5A6C7PA, seg_pos={0 0}, upload=1726]: Uploading piece...
Jul 26 23:44:35.493214294 (1h52m51.274147682s) [noise=true , address=77.90.31.19:4871 , piece_id=7ZSMZ2RIR42ZVPSFQ3ILXTGA4LTSUIBTMINIGLFHWSD6YCH4ASRA, seg_pos={0 0}, upload=1726]: Uploading piece...
Jul 26 23:44:35.494016913 (1h52m51.274950301s) [noise=true , address=79.127.219.38:4001 , piece_id=4VIS7HGYCK4JHUIZHCRIKNI4A6UZEF6VE6Z2TPVYS7E2WOJYCVRQ, seg_pos={0 0}, upload=1726]: Uploading piece...
Jul 26 23:44:35.494443652 (1h52m51.275377043s) [noise=true , address=77.90.31.12:4608 , piece_id=27LCIXMJX4LBTBANJPOMVNA2W3LC6LRFWMM4PYS2PPU3XFUE7NNQ, seg_pos={0 0}, upload=1726]: Uploading piece...
Jul 26 23:44:35.495095329 (1h52m51.276028718s) [noise=true , address=104.36.82.13:4200 , piece_id=HSLZPW5YURREZAGGW5LQN7QTDIKB4WX54IDJI4MFHMV3WILAC6NQ, seg_pos={0 0}, upload=1726]: Uploading piece...
Jul 26 23:44:35.495535278 (1h52m51.276468675s) [noise=true , address=104.36.82.55:4199 , piece_id=RDOM7TE4TMKCIJF7CWLO6L2HOO7HNTEOLEWK3YLXXJ263CEZP4GQ, seg_pos={0 0}, upload=1726]: Uploading piece...
Jul 26 23:44:35.495982709 (1h52m51.276916088s) [noise=true , address=104.36.82.47:4522 , piece_id=VMPSGRVEATIN2IKU7MBPJTWW4DWKKGOWDBWYSLER7T3J3HDMPB3A, seg_pos={0 0}, upload=1726]: Uploading piece...
Jul 26 23:44:35.565482117 (1h52m51.346415485s) [upload=1726]: Next returned nil segment
Jul 26 23:44:35.565499405 (1h52m51.346432784s) [upload=1726]: Waiting for error group managing segments...
Jul 26 23:44:35.570357164 (1h52m51.351290541s) [noise=true , address=79.127.219.38:4001 , piece_id=4VIS7HGYCK4JHUIZHCRIKNI4A6UZEF6VE6Z2TPVYS7E2WOJYCVRQ, seg_pos={0 0}, upload=1726]: Done uploading piece. err: <nil>
Jul 26 23:44:35.589069819 (1h52m51.370003196s) [noise=true , address=77.90.31.12:4608 , piece_id=27LCIXMJX4LBTBANJPOMVNA2W3LC6LRFWMM4PYS2PPU3XFUE7NNQ, seg_pos={0 0}, upload=1726]: Done uploading piece. err: <nil>
Jul 26 23:44:35.615620249 (1h52m51.396553628s) [noise=true , address=77.90.31.21:4912 , piece_id=PIA7P2VPFVLBH4GFYRHQ5KP5PRSCJCDOYXJYNZEDGKDYHULGBUBQ, seg_pos={0 0}, upload=1726]: Done uploading piece. err: <nil>
Jul 26 23:44:35.666362835 (1h52m51.447296203s) [noise=true , address=77.90.31.11:4564 , piece_id=W4XJ3TSB5PXNGV2ZIASXSVNVOHFH4SEPUE3MBWWSGINNPLXEFP2Q, seg_pos={0 0}, upload=1726]: Done uploading piece. err: <nil>
Jul 26 23:44:35.843127855 (1h52m51.624061235s) [noise=true , address=64.49.13.139:4478 , piece_id=RQBUGIMT7XHLF2FBZGUA4RTRYUN6FL45RIZH2YEUFBKPLBR7P6UA, seg_pos={0 0}, upload=1726]: Done uploading piece. err: ecclient: upload failed (node:1kmGWoxQSmnEF7KAt2JqGDcmwTYnu82wpaCQUFaBAYUshTY98J, address:64.49.13.139:4478): protocol: storage node overloaded, request limit: 2000
Jul 26 23:44:36.095341527 (1h52m51.876274889s) [noise=true , address=79.127.205.235:4691 , piece_id=34QHDCOHKOGY6YAAX5C2U44STARV6AQJFODJB4WTPGLWPQ47J56A, seg_pos={0 0}, upload=1726]: Done uploading piece. err: <nil>
Jul 26 23:44:36.619796257 (1h52m52.400729624s) [noise=true , address=64.49.13.140:4508 , piece_id=KXAO5GNBNBWWQ3KMC4VIE5KAVIXATL3DBVZM4EUAPUKUKXF2WRGA, seg_pos={0 0}, upload=1726]: Done uploading piece. err: ecclient: upload failed (node:1diUU2skKzXPjYy3Mu3iDDjnG5W8Z2cxCRMxDCH9cQGg2TdEiR, address:64.49.13.140:4508): protocol: storage node overloaded, request limit: 2000
Jul 26 23:44:37.623736203 (1h52m53.404669582s) [noise=true , address=156.146.43.225:4448 , piece_id=LIAN4KFQUGNFOX4PLKNJ6GHICUDTCEIFRIULLRHYENJUYPR4TF2Q, seg_pos={0 0}, upload=1726]: Done uploading piece. err: <nil>
Jul 26 23:44:38.346038855 (1h52m54.126972234s) [noise=true , address=64.49.13.138:4482 , piece_id=6KPYSZAM4AJTFBVHDNLPJY7CGLGJSIKPB7S7N4G5F4ZIRE4G3SGQ, seg_pos={0 0}, upload=1726]: Done uploading piece. err: <nil>
Jul 26 23:44:38.748590410 (1h52m54.529523779s) [noise=true , address=77.90.31.19:4871 , piece_id=7ZSMZ2RIR42ZVPSFQ3ILXTGA4LTSUIBTMINIGLFHWSD6YCH4ASRA, seg_pos={0 0}, upload=1726]: Done uploading piece. err: <nil>
Jul 26 23:44:39.757730680 (1h52m55.538664049s) [noise=true , address=156.146.43.225:4449 , piece_id=ZVXMQF7BR7FMGBFFFF5UPAPACIOIWNOUK3JCOG62VOG4D24FC3TQ, seg_pos={0 0}, upload=1726]: Done uploading piece. err: <nil>
Jul 26 23:45:35.421284307 (1h53m51.202217675s) [noise=true , address=104.36.82.52:4107 , piece_id=DQKOQZXURQQDTVR6ELRV5CRIFXFWJFSIMHCIJSGKDXZKV5A6C7PA, seg_pos={0 0}, upload=1726]: Done uploading piece. err: ecclient: upload failed (node:12YVUtzGaKXr1Leb1URHZuaQXZv4zTtYwnv59j5GaAfd9zoK2Kj, address:104.36.82.52:4107): protocol: write tcp 159.223.143.39:45372->104.36.82.52:4107: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 26 23:45:35.421302414 (1h53m51.202235796s) [noise=true , address=104.36.82.55:4199 , piece_id=RDOM7TE4TMKCIJF7CWLO6L2HOO7HNTEOLEWK3YLXXJ263CEZP4GQ, seg_pos={0 0}, upload=1726]: Done uploading piece. err: ecclient: upload failed (node:1mCH3fNEMd64WztUYYRCE2KUdJgLxWGoPUfEe4KaF34YhUxUXx, address:104.36.82.55:4199): protocol: context deadline exceeded; context deadline exceeded
Jul 26 23:45:35.421406438 (1h53m51.202339807s) [noise=true , address=104.36.82.13:4200 , piece_id=HSLZPW5YURREZAGGW5LQN7QTDIKB4WX54IDJI4MFHMV3WILAC6NQ, seg_pos={0 0}, upload=1726]: Done uploading piece. err: ecclient: upload failed (node:12dfLumooVsDsWXqi5FQMDutkcexM9A5Kmu4Vo9s1TqfWxv5ozH, address:104.36.82.13:4200): protocol: context deadline exceeded; context deadline exceeded
Jul 26 23:45:35.421690535 (1h53m51.202623904s) [noise=true , address=104.36.82.47:4522 , piece_id=VMPSGRVEATIN2IKU7MBPJTWW4DWKKGOWDBWYSLER7T3J3HDMPB3A, seg_pos={0 0}, upload=1726]: Done uploading piece. err: ecclient: upload failed (node:1zGZe6e37uop4y622fU1GbspMPVEZURS3FbMrf7je6VT9TYGFr, address:104.36.82.47:4522): protocol: rpc: dial tcp 104.36.82.47:4522: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 26 23:45:35.421736944 (1h53m51.202670307s) [noise=true , address=104.36.82.53:4149 , piece_id=5I4OY4D4WBZA3FB7HLKCPWDQAI7TKPFDP3IECR5J6TVHQ7HQ2K5Q, seg_pos={0 0}, upload=1726]: Done uploading piece. err: ecclient: upload failed (node:1nDCYjyLUqt1zkfotDNVcor73Cdpq3fnyMiygujdc9V4tGNHLY, address:104.36.82.53:4149): protocol: rpc: dial tcp 104.36.82.53:4149: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 26 23:45:35.421795354 (1h53m51.20272872s) [noise=true , address=104.36.82.56:4247 , piece_id=2FDHHAMOQO6RGUFXLQH4UKNHYPOFYZTPEABXDHST6RVWJC5CJYCA, seg_pos={0 0}, upload=1726]: Done uploading piece. err: ecclient: upload failed (node:12PzKreuPT9X594ZTTYbXanNdaFVVJsnKL99fumjXymnRa9jeEq, address:104.36.82.56:4247): protocol: rpc: dial tcp 104.36.82.56:4247: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 26 23:45:35.421847011 (1h53m51.202780377s) [noise=true , address=104.36.82.46:4490 , piece_id=QFKKQU7UEDERMIXEIWGAEAUMRCGUANOC5ZMM7HOWUBFBFZL6EQIA, seg_pos={0 0}, upload=1726]: Done uploading piece. err: ecclient: upload failed (node:12PUxwz3cZkF8e2VkjTXnu1utrFzQRTxdohfd9EvDaLvyejpTji, address:104.36.82.46:4490): protocol: rpc: dial tcp 104.36.82.46:4490: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 26 23:45:35.421871433 (1h53m51.202804809s) [seg_pos={0 0}, upload=1726]: Done waiting for segment. successful: 9 optimal: 14 errs: <nil>
Jul 26 23:45:35.451883786 (1h53m51.232817154s) [upload=1726]: Done uploading segments...
Jul 26 23:45:35.451947357 (1h53m51.232880736s) [upload=1726]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 9)
Duration: 1m0.033817297s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)antiblip:
49 out of 60 is failed (threshold=3) [.....X......XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration:
30.032771051s
FAIL (2025-07-26 23:45:18 UTC)
orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 10)
Duration: 30.032795171s
core
uplink
us1
vivint
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
45 out of 60 is failed (threshold=3) [..............XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.XXXXXX] duration:
30.033294463s
FAIL (2025-07-26 23:45:08 UTC)
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 6)
======LOGS======
0552 (1h52m54.191933919s) [upload=1728]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jul 26 23:44:38.411008360 (1h52m54.191941737s) [upload=1728]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jul 26 23:44:38.447363024 (1h52m54.228296409s) [seg_pos={0 0}, upload=1728]: Begin upload segment...
Jul 26 23:44:38.447442037 (1h52m54.228375417s) [seg_pos={0 0}, upload=1728]: Done begin upload segment.
Jul 26 23:44:38.447472505 (1h52m54.228405885s) [noise=true , address=77.90.31.11:4553 , piece_id=ZO7U2H2GQZUVSXM52YIFFIRPLQMFCQ24N2PVLJBZU7XFLR4H37FQ, seg_pos={0 0}, upload=1728]: Uploading piece...
Jul 26 23:44:38.447530743 (1h52m54.228464133s) [noise=true , address=104.36.82.7:4912 , piece_id=JCJA4K6XVTUZY5UREEOTVGO4TKY5LCAWGCQWUBFY42W33CHCCLFQ, seg_pos={0 0}, upload=1728]: Uploading piece...
Jul 26 23:44:38.448168457 (1h52m54.229101841s) [noise=true , address=104.36.82.13:4187 , piece_id=6JBRSVVGZE335WKHZTOTI3YWORTCDJPYKSJHUMW3A2UER2YKOA5Q, seg_pos={0 0}, upload=1728]: Uploading piece...
Jul 26 23:44:38.448218182 (1h52m54.229151566s) [noise=true , address=45.91.116.7:4011 , piece_id=BSGIRDLDXT7QMTOE2AQXLL7SF5V5CD2DTXO7ZJMJBCHG7ZVBADIQ, seg_pos={0 0}, upload=1728]: Uploading piece...
Jul 26 23:44:38.448712998 (1h52m54.229646397s) [noise=true , address=104.36.82.12:4150 , piece_id=VXNVNZLNJTT3CGZC4MEI3NMYUVIZYN2JTW7YZRUP2LFI43CDLOAQ, seg_pos={0 0}, upload=1728]: Uploading piece...
Jul 26 23:44:38.448734552 (1h52m54.229668042s) [noise=true , address=45.91.116.10:4081 , piece_id=YPIRVCBM7YS3QFDUHK62GYEJJ56V4CIKJ5PCLIGZ4WNMNYTC6VOA, seg_pos={0 0}, upload=1728]: Uploading piece...
Jul 26 23:44:38.449124782 (1h52m54.230058176s) [noise=true , address=104.36.82.16:4294 , piece_id=OQJOQ5ZIV2GKUEOKJPRXFT7F2E37IGBLKV5AMTLD2B4QL5SPP7EQ, seg_pos={0 0}, upload=1728]: Uploading piece...
Jul 26 23:44:38.449223824 (1h52m54.230157203s) [noise=true , address=45.91.116.8:4047 , piece_id=V3ZDAKPXAYZPCHMPF2SWKJ22EPNF5JNSQA3JZQ5NXBAVHYTKRT2A, seg_pos={0 0}, upload=1728]: Uploading piece...
Jul 26 23:44:38.449621031 (1h52m54.23055442s) [noise=true , address=104.36.82.9:4005 , piece_id=GXYGGBP3AGHKXAADRPVIGWQPGHNQEJ4KKFLVRCN36GC3KWULZLVA, seg_pos={0 0}, upload=1728]: Uploading piece...
Jul 26 23:44:38.449737624 (1h52m54.23067101s) [noise=true , address=77.90.31.17:4770 , piece_id=NB6VD2P6PB3JIYL3DM4SCMYBVKKKLMABOCS7JOMMLTRQIAH4UXOQ, seg_pos={0 0}, upload=1728]: Uploading piece...
Jul 26 23:44:38.449998923 (1h52m54.230932312s) [noise=true , address=104.36.82.2:4731 , piece_id=5UTPIJXRGD6D4XA47KYE46SY432B6KJV2QQXG2Q2OBNPKRFT2EIA, seg_pos={0 0}, upload=1728]: Uploading piece...
Jul 26 23:44:38.450229755 (1h52m54.231163142s) [noise=true , address=104.36.82.3:4767 , piece_id=VYKOLINBOKDE44IKVZOSJKNRE76FBNQZ3JATRB2X2E2LBEQXORKQ, seg_pos={0 0}, upload=1728]: Uploading piece...
Jul 26 23:44:38.450770002 (1h52m54.23170343s) [noise=true , address=104.36.82.12:4149 , piece_id=XTIJK66IAMVTYWTJS6GY54LFTDEIEVQLXMEVEENCFIFAJ4MASDNA, seg_pos={0 0}, upload=1728]: Uploading piece...
Jul 26 23:44:38.451012447 (1h52m54.23194584s) [noise=true , address=104.36.82.11:4114 , piece_id=NFTK6Q7L32SGS3A2QF65B46NKNMB7C5EBHPZ2LHMN6ZKCAEOW5QA, seg_pos={0 0}, upload=1728]: Uploading piece...
Jul 26 23:44:38.451269660 (1h52m54.232203049s) [noise=true , address=77.90.31.16:4733 , piece_id=5ABNUXZUVCJVXRFNINT6PBS43EYM57LJLUGIMJ6Y5DIHYEJA2GUQ, seg_pos={0 0}, upload=1728]: Uploading piece...
Jul 26 23:44:38.451502954 (1h52m54.232436341s) [noise=true , address=45.91.116.12:4225 , piece_id=EYPXNTNWCFI7ZAOA5AN3FMHIBK6UDNJGBIEFKCY5NKMZVT25E2AQ, seg_pos={0 0}, upload=1728]: Uploading piece...
Jul 26 23:44:38.561793425 (1h52m54.342726809s) [upload=1728]: Next returned nil segment
Jul 26 23:44:38.561815173 (1h52m54.34274856s) [upload=1728]: Waiting for error group managing segments...
Jul 26 23:44:38.570459813 (1h52m54.351393192s) [noise=true , address=45.91.116.10:4081 , piece_id=YPIRVCBM7YS3QFDUHK62GYEJJ56V4CIKJ5PCLIGZ4WNMNYTC6VOA, seg_pos={0 0}, upload=1728]: Done uploading piece. err: <nil>
Jul 26 23:44:38.571277426 (1h52m54.352210805s) [noise=true , address=77.90.31.17:4770 , piece_id=NB6VD2P6PB3JIYL3DM4SCMYBVKKKLMABOCS7JOMMLTRQIAH4UXOQ, seg_pos={0 0}, upload=1728]: Done uploading piece. err: <nil>
Jul 26 23:44:38.571419049 (1h52m54.352352426s) [noise=true , address=45.91.116.12:4225 , piece_id=EYPXNTNWCFI7ZAOA5AN3FMHIBK6UDNJGBIEFKCY5NKMZVT25E2AQ, seg_pos={0 0}, upload=1728]: Done uploading piece. err: <nil>
Jul 26 23:44:38.571956093 (1h52m54.352889472s) [noise=true , address=77.90.31.11:4553 , piece_id=ZO7U2H2GQZUVSXM52YIFFIRPLQMFCQ24N2PVLJBZU7XFLR4H37FQ, seg_pos={0 0}, upload=1728]: Done uploading piece. err: <nil>
Jul 26 23:44:38.572036717 (1h52m54.352970086s) [noise=true , address=45.91.116.7:4011 , piece_id=BSGIRDLDXT7QMTOE2AQXLL7SF5V5CD2DTXO7ZJMJBCHG7ZVBADIQ, seg_pos={0 0}, upload=1728]: Done uploading piece. err: <nil>
Jul 26 23:44:38.572566661 (1h52m54.353500038s) [noise=true , address=45.91.116.8:4047 , piece_id=V3ZDAKPXAYZPCHMPF2SWKJ22EPNF5JNSQA3JZQ5NXBAVHYTKRT2A, seg_pos={0 0}, upload=1728]: Done uploading piece. err: <nil>
Jul 26 23:45:08.379149593 (1h53m24.16008296s) [noise=true , address=104.36.82.13:4187 , piece_id=6JBRSVVGZE335WKHZTOTI3YWORTCDJPYKSJHUMW3A2UER2YKOA5Q, seg_pos={0 0}, upload=1728]: Done uploading piece. err: ecclient: upload failed (node:12cQxBB2L2CyymYx3xWp1ALj6p8QWr4A34sqvtG5zyeT7sVWv7x, address:104.36.82.13:4187): protocol: rpc: dial tcp 104.36.82.13:4187: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 26 23:45:08.379256562 (1h53m24.160189944s) [noise=true , address=104.36.82.16:4294 , piece_id=OQJOQ5ZIV2GKUEOKJPRXFT7F2E37IGBLKV5AMTLD2B4QL5SPP7EQ, seg_pos={0 0}, upload=1728]: Done uploading piece. err: ecclient: upload failed (node:12CbfkWK4H84My3e6qhDuAkTrHWsooysz7KE4bk6VoDRDWEgb9e, address:104.36.82.16:4294): protocol: rpc: dial tcp 104.36.82.16:4294: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 26 23:45:08.379323343 (1h53m24.160256722s) [noise=true , address=104.36.82.9:4005 , piece_id=GXYGGBP3AGHKXAADRPVIGWQPGHNQEJ4KKFLVRCN36GC3KWULZLVA, seg_pos={0 0}, upload=1728]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:104.36.82.9:4005): protocol: rpc: dial tcp 104.36.82.9:4005: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 26 23:45:08.379393923 (1h53m24.160327286s) [noise=true , address=104.36.82.3:4767 , piece_id=VYKOLINBOKDE44IKVZOSJKNRE76FBNQZ3JATRB2X2E2LBEQXORKQ, seg_pos={0 0}, upload=1728]: Done uploading piece. err: ecclient: upload failed (node:1TiDHs6xp3atLDHHv1i2564ibyPGLKfFyHx3bgZWGRRN5cwUi5, address:104.36.82.3:4767): protocol: write tcp 159.223.143.39:46344->104.36.82.3:4767: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 26 23:45:08.379446117 (1h53m24.160379495s) [noise=true , address=104.36.82.7:4912 , piece_id=JCJA4K6XVTUZY5UREEOTVGO4TKY5LCAWGCQWUBFY42W33CHCCLFQ, seg_pos={0 0}, upload=1728]: Done uploading piece. err: ecclient: upload failed (node:1gwuDRBC5oJXgZbP6FqP9tc8UB6aJt2uexUgSuzHCovw27v1NR, address:104.36.82.7:4912): protocol: rpc: dial tcp 104.36.82.7:4912: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 26 23:45:08.379515407 (1h53m24.160448787s) [noise=true , address=104.36.82.12:4149 , piece_id=XTIJK66IAMVTYWTJS6GY54LFTDEIEVQLXMEVEENCFIFAJ4MASDNA, seg_pos={0 0}, upload=1728]: Done uploading piece. err: ecclient: upload failed (node:12qexRLJM6TY935UaEWsj1X9ViyBvk2BM1XGjnq8kapnXXXmgbm, address:104.36.82.12:4149): protocol: write tcp 159.223.143.39:49236->104.36.82.12:4149: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 26 23:45:08.379578632 (1h53m24.160511996s) [noise=true , address=104.36.82.11:4114 , piece_id=NFTK6Q7L32SGS3A2QF65B46NKNMB7C5EBHPZ2LHMN6ZKCAEOW5QA, seg_pos={0 0}, upload=1728]: Done uploading piece. err: ecclient: upload failed (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN, address:104.36.82.11:4114): protocol: rpc: dial tcp 104.36.82.11:4114: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 26 23:45:08.379653427 (1h53m24.160586806s) [noise=true , address=77.90.31.16:4733 , piece_id=5ABNUXZUVCJVXRFNINT6PBS43EYM57LJLUGIMJ6Y5DIHYEJA2GUQ, seg_pos={0 0}, upload=1728]: Done uploading piece. err: ecclient: upload failed (node:12TBBdm1hfKVn49KrFVNj3MRbqoanGxJW82ws1iJqMtjRdzJKVf, address:77.90.31.16:4733): protocol: rpc: dial tcp 77.90.31.16:4733: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 26 23:45:08.379668067 (1h53m24.160601453s) [noise=true , address=104.36.82.2:4731 , piece_id=5UTPIJXRGD6D4XA47KYE46SY432B6KJV2QQXG2Q2OBNPKRFT2EIA, seg_pos={0 0}, upload=1728]: Done uploading piece. err: ecclient: upload failed (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9, address:104.36.82.2:4731): protocol: rpc: dial tcp 104.36.82.2:4731: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 26 23:45:08.379736540 (1h53m24.160669921s) [noise=true , address=104.36.82.12:4150 , piece_id=VXNVNZLNJTT3CGZC4MEI3NMYUVIZYN2JTW7YZRUP2LFI43CDLOAQ, seg_pos={0 0}, upload=1728]: Done uploading piece. err: ecclient: upload failed (node:12YanYT7FkCt7SzDuqx15RP9zxaYmikVLzPkBp4i15WEShDpfpK, address:104.36.82.12:4150): protocol: rpc: dial tcp 104.36.82.12:4150: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 26 23:45:08.379804090 (1h53m24.160737483s) [seg_pos={0 0}, upload=1728]: Done waiting for segment. successful: 6 optimal: 13 errs: <nil>
Jul 26 23:45:08.410118785 (1h53m24.191052163s) [upload=1728]: Done uploading segments...
Jul 26 23:45:08.410149846 (1h53m24.191083224s) [upload=1728]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 6)
Duration: 30.033321456s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
18 out of 60 is failed (threshold=3) [.........XXX.X....XX.X.X.X.......XXX...XX.....X.XXX.........] duration:
2.641604534s
FAIL (2025-07-26 23:45:06 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 2.641768951s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
18 out of 60 is failed (threshold=3) [..........XXX.X....XX.X.X.X.......XXX...XX.....X.XXX........] duration:
4.863295244s
FAIL (2025-07-26 23:44:44 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 4.863320337s
edge
gateway
us-select-1
|