Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............................X............................] duration: 544.21019ms trace: https://jaeger.dev.storj.io/trace/7013ffb1a981d605 FAIL (2025-12-22 22:53:47 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 22 22:31:05.439510308 (415h3m6.2672187s) [upload=431917]: Starting upload Dec 22 22:31:05.439581993 (415h3m6.26729037s) [upload=431917]: Done starting upload Dec 22 22:31:05.439812792 (415h3m6.267521185s) [upload=431917]: Uploading segments... Dec 22 22:31:05.439942524 (415h3m6.267650918s) [upload=431917]: Got next segment. Inline: false Dec 22 22:31:05.440022744 (415h3m6.267731174s) [upload=431917]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 22 22:31:05.440029670 (415h3m6.267738072s) [upload=431917]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 22 22:31:05.467186372 (415h3m6.294894769s) [seg_pos={0 0}, upload=431917]: Begin upload segment... Dec 22 22:31:05.467265149 (415h3m6.294973535s) [seg_pos={0 0}, upload=431917]: Done begin upload segment. Dec 22 22:31:05.467296974 (415h3m6.295005373s) [noise=true , address=145.79.227.51:4409 , piece_id=ATH3F4DFOUUIFFJZRA2SSRRL7AKWZMQNQRYOZCVFFDLTFCFP236A, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.467389853 (415h3m6.295098247s) [noise=true , address=145.79.227.39:4013 , piece_id=JJVURGSLKY2UQ4FKNCLATA3ZBD5SDL32XJAG42KH2U6OYK3KTS6Q, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.467912104 (415h3m6.295620486s) [noise=true , address=145.79.227.36:4519 , piece_id=JUKRRIZWVGKEDD3GPJ7HMY2MEQZNMUCTKTXR6ZLSIERRONTZKHCQ, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.468324863 (415h3m6.296033263s) [noise=true , address=145.79.227.38:4979 , piece_id=HLMQC5OEB4OL4OZSRMCEVAWR5QM2T4MQUBPTHRVQLEAHPBWPQF6Q, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.468334764 (415h3m6.296043147s) [noise=true , address=145.79.227.40:4051 , piece_id=55RRAD7DQX6SR7RBLYAV7MD7XRWPNIWHTVVBQZ2ZL3OE2L7LRRIQ, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.468742525 (415h3m6.296450916s) [noise=true , address=45.91.116.4:4939 , piece_id=QYNKMKDLQJV7B6M2E34DU53Y7YI6VBCUW3C7ZLQOI4XWTBXZHKDQ, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.468915034 (415h3m6.296623414s) [noise=true , address=45.91.116.11:4117 , piece_id=WLMO5E3BIFFL3TOAUSSBGORIZ6DOCWFCMVT4AVFPPGPSKP2OKDAA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.469069393 (415h3m6.296777787s) [noise=true , address=145.79.227.32:4835 , piece_id=BRE3JH3XKPXB6FN7WB3W6ED7MOFES6HYLR34YGR6L5AX7RPIQ7IA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.469285789 (415h3m6.296994183s) [noise=true , address=145.79.227.29:4943 , piece_id=G4TMZ6IAIZBDON3UEDPH4LJQFHV3JLALEXSJ3VI4XN6WHIZOB57Q, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.469411962 (415h3m6.297120355s) [noise=true , address=145.79.227.48:4301 , piece_id=LBIYTJNOX5J4M3HDE7FGF5CSARBITW2AK3P2DSYFLFKUOIHZUKOQ, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.469749379 (415h3m6.297457773s) [noise=true , address=145.79.227.37:4565 , piece_id=554RHTYPG27UNCFPSRMOB245AX4QXL2QFIYECNTVWNNJMAL4BTYA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.470080828 (415h3m6.29778922s) [noise=true , address=145.79.227.43:4157 , piece_id=3W3LVTU37DNHWS52IGIIH7SXWXVNPGEY27RQNT272L3Z64GNR4KA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.470477660 (415h3m6.298186058s) [noise=true , address=45.91.116.14:4190 , piece_id=V4T6Z7K3C75LQEXNQIYZGYWW2CSQ7UPSET46QGDRT52RFGAROTTA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.470567977 (415h3m6.29827637s) [noise=true , address=145.79.227.15:4874 , piece_id=FNCCIMIVJTGZ2X5TOYOZH727JXHYM7HG3HBHBTZDAG67SHZLYYYA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.470980950 (415h3m6.298689349s) [noise=true , address=145.79.227.50:4373 , piece_id=4U55T5QDYCE6AZ3NCTPAVECPAELPQUKT3HV7GZL7JFRMWRN2AZUQ, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.471440098 (415h3m6.299148489s) [noise=true , address=145.79.227.47:4266 , piece_id=3NFYV2PWBNTAOSFHENM4J5PZ7XXE72UKTQVXLTVQRHGK4DUXAKDA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.560766475 (415h3m6.388474871s) [upload=431917]: Next returned nil segment Dec 22 22:31:05.560785161 (415h3m6.388493562s) [upload=431917]: Waiting for error group managing segments... Dec 22 22:31:05.566117878 (415h3m6.39382628s) [noise=true , address=45.91.116.4:4939 , piece_id=QYNKMKDLQJV7B6M2E34DU53Y7YI6VBCUW3C7ZLQOI4XWTBXZHKDQ, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.566723306 (415h3m6.394431687s) [noise=true , address=45.91.116.11:4117 , piece_id=WLMO5E3BIFFL3TOAUSSBGORIZ6DOCWFCMVT4AVFPPGPSKP2OKDAA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.566992667 (415h3m6.394701063s) [noise=true , address=145.79.227.36:4519 , piece_id=JUKRRIZWVGKEDD3GPJ7HMY2MEQZNMUCTKTXR6ZLSIERRONTZKHCQ, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.567464626 (415h3m6.395173019s) [noise=true , address=145.79.227.47:4266 , piece_id=3NFYV2PWBNTAOSFHENM4J5PZ7XXE72UKTQVXLTVQRHGK4DUXAKDA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.567924492 (415h3m6.395632883s) [noise=true , address=145.79.227.15:4874 , piece_id=FNCCIMIVJTGZ2X5TOYOZH727JXHYM7HG3HBHBTZDAG67SHZLYYYA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.568807294 (415h3m6.396515685s) [noise=true , address=145.79.227.51:4409 , piece_id=ATH3F4DFOUUIFFJZRA2SSRRL7AKWZMQNQRYOZCVFFDLTFCFP236A, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.570579823 (415h3m6.398288214s) [noise=true , address=145.79.227.48:4301 , piece_id=LBIYTJNOX5J4M3HDE7FGF5CSARBITW2AK3P2DSYFLFKUOIHZUKOQ, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.571632758 (415h3m6.399341149s) [noise=true , address=145.79.227.43:4157 , piece_id=3W3LVTU37DNHWS52IGIIH7SXWXVNPGEY27RQNT272L3Z64GNR4KA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.573168383 (415h3m6.400876774s) [noise=true , address=145.79.227.38:4979 , piece_id=HLMQC5OEB4OL4OZSRMCEVAWR5QM2T4MQUBPTHRVQLEAHPBWPQF6Q, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.579995420 (415h3m6.407703816s) [noise=true , address=45.91.116.14:4190 , piece_id=V4T6Z7K3C75LQEXNQIYZGYWW2CSQ7UPSET46QGDRT52RFGAROTTA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.592093172 (415h3m6.419801563s) [noise=true , address=145.79.227.37:4565 , piece_id=554RHTYPG27UNCFPSRMOB245AX4QXL2QFIYECNTVWNNJMAL4BTYA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:09.946500434 (415h3m10.774208816s) [noise=true , address=145.79.227.40:4051 , piece_id=55RRAD7DQX6SR7RBLYAV7MD7XRWPNIWHTVVBQZ2ZL3OE2L7LRRIQ, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:35.398235543 (415h3m36.225943943s) [noise=true , address=145.79.227.39:4013 , piece_id=JJVURGSLKY2UQ4FKNCLATA3ZBD5SDL32XJAG42KH2U6OYK3KTS6Q, seg_pos={0 0}, upload=431917]: Done uploading piece. err: ecclient: upload failed (node:121scLvKEW9zw2aEz22zpremcjiop67eQ8v2uqpmaD5aZzCAP7b, address:145.79.227.39:4013): protocol: write tcp 159.223.143.39:60842->145.79.227.39:4013: use of closed network connection; write tcp 159.223.143.39:60842->145.79.227.39:4013: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:60842->145.79.227.39:4013: use of closed network connection Dec 22 22:31:35.398562305 (415h3m36.226270699s) [noise=true , address=145.79.227.32:4835 , piece_id=BRE3JH3XKPXB6FN7WB3W6ED7MOFES6HYLR34YGR6L5AX7RPIQ7IA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: ecclient: upload failed (node:16vVfeoqCkbMYCRjMMWaKBnCvQBBJPMgwQtWTq9HKQrD9SS17y, address:145.79.227.32:4835): protocol: write tcp 159.223.143.39:45098->145.79.227.32:4835: use of closed network connection; write tcp 159.223.143.39:45098->145.79.227.32:4835: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:45098->145.79.227.32:4835: use of closed network connection Dec 22 22:31:35.399183028 (415h3m36.226891422s) [noise=true , address=145.79.227.50:4373 , piece_id=4U55T5QDYCE6AZ3NCTPAVECPAELPQUKT3HV7GZL7JFRMWRN2AZUQ, seg_pos={0 0}, upload=431917]: Done uploading piece. err: ecclient: upload failed (node:12AQGSkJuNJDxoeqhChUenWxHwF5yK41UESCF63FQz3XvabXXfz, address:145.79.227.50:4373): protocol: read tcp 159.223.143.39:56028->145.79.227.50:4373: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 22 22:31:35.399278021 (415h3m36.226986417s) [noise=true , address=145.79.227.29:4943 , piece_id=G4TMZ6IAIZBDON3UEDPH4LJQFHV3JLALEXSJ3VI4XN6WHIZOB57Q, seg_pos={0 0}, upload=431917]: Done uploading piece. err: ecclient: upload failed (node:12fc5Trj8XxacZhzxdksUVT5CbG6uTmiwdUPjreYi4UMPY1TXWz, address:145.79.227.29:4943): protocol: write tcp 159.223.143.39:50820->145.79.227.29:4943: use of closed network connection; write tcp 159.223.143.39:50820->145.79.227.29:4943: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:50820->145.79.227.29:4943: use of closed network connection Dec 22 22:31:35.399379485 (415h3m36.227087866s) [seg_pos={0 0}, upload=431917]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 22 22:31:35.421122544 (415h3m36.248830935s) [upload=431917]: Done uploading segments... Dec 22 22:31:35.421174369 (415h3m36.248882769s) [upload=431917]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 544.238819ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................................X...........................] duration: 392.884394ms trace: https://jaeger.dev.storj.io/trace/7013ffb1a981d605 FAIL (2025-12-22 22:52:47 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 22 22:31:05.439510308 (415h3m6.2672187s) [upload=431917]: Starting upload Dec 22 22:31:05.439581993 (415h3m6.26729037s) [upload=431917]: Done starting upload Dec 22 22:31:05.439812792 (415h3m6.267521185s) [upload=431917]: Uploading segments... Dec 22 22:31:05.439942524 (415h3m6.267650918s) [upload=431917]: Got next segment. Inline: false Dec 22 22:31:05.440022744 (415h3m6.267731174s) [upload=431917]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 22 22:31:05.440029670 (415h3m6.267738072s) [upload=431917]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 22 22:31:05.467186372 (415h3m6.294894769s) [seg_pos={0 0}, upload=431917]: Begin upload segment... Dec 22 22:31:05.467265149 (415h3m6.294973535s) [seg_pos={0 0}, upload=431917]: Done begin upload segment. Dec 22 22:31:05.467296974 (415h3m6.295005373s) [noise=true , address=145.79.227.51:4409 , piece_id=ATH3F4DFOUUIFFJZRA2SSRRL7AKWZMQNQRYOZCVFFDLTFCFP236A, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.467389853 (415h3m6.295098247s) [noise=true , address=145.79.227.39:4013 , piece_id=JJVURGSLKY2UQ4FKNCLATA3ZBD5SDL32XJAG42KH2U6OYK3KTS6Q, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.467912104 (415h3m6.295620486s) [noise=true , address=145.79.227.36:4519 , piece_id=JUKRRIZWVGKEDD3GPJ7HMY2MEQZNMUCTKTXR6ZLSIERRONTZKHCQ, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.468324863 (415h3m6.296033263s) [noise=true , address=145.79.227.38:4979 , piece_id=HLMQC5OEB4OL4OZSRMCEVAWR5QM2T4MQUBPTHRVQLEAHPBWPQF6Q, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.468334764 (415h3m6.296043147s) [noise=true , address=145.79.227.40:4051 , piece_id=55RRAD7DQX6SR7RBLYAV7MD7XRWPNIWHTVVBQZ2ZL3OE2L7LRRIQ, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.468742525 (415h3m6.296450916s) [noise=true , address=45.91.116.4:4939 , piece_id=QYNKMKDLQJV7B6M2E34DU53Y7YI6VBCUW3C7ZLQOI4XWTBXZHKDQ, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.468915034 (415h3m6.296623414s) [noise=true , address=45.91.116.11:4117 , piece_id=WLMO5E3BIFFL3TOAUSSBGORIZ6DOCWFCMVT4AVFPPGPSKP2OKDAA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.469069393 (415h3m6.296777787s) [noise=true , address=145.79.227.32:4835 , piece_id=BRE3JH3XKPXB6FN7WB3W6ED7MOFES6HYLR34YGR6L5AX7RPIQ7IA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.469285789 (415h3m6.296994183s) [noise=true , address=145.79.227.29:4943 , piece_id=G4TMZ6IAIZBDON3UEDPH4LJQFHV3JLALEXSJ3VI4XN6WHIZOB57Q, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.469411962 (415h3m6.297120355s) [noise=true , address=145.79.227.48:4301 , piece_id=LBIYTJNOX5J4M3HDE7FGF5CSARBITW2AK3P2DSYFLFKUOIHZUKOQ, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.469749379 (415h3m6.297457773s) [noise=true , address=145.79.227.37:4565 , piece_id=554RHTYPG27UNCFPSRMOB245AX4QXL2QFIYECNTVWNNJMAL4BTYA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.470080828 (415h3m6.29778922s) [noise=true , address=145.79.227.43:4157 , piece_id=3W3LVTU37DNHWS52IGIIH7SXWXVNPGEY27RQNT272L3Z64GNR4KA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.470477660 (415h3m6.298186058s) [noise=true , address=45.91.116.14:4190 , piece_id=V4T6Z7K3C75LQEXNQIYZGYWW2CSQ7UPSET46QGDRT52RFGAROTTA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.470567977 (415h3m6.29827637s) [noise=true , address=145.79.227.15:4874 , piece_id=FNCCIMIVJTGZ2X5TOYOZH727JXHYM7HG3HBHBTZDAG67SHZLYYYA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.470980950 (415h3m6.298689349s) [noise=true , address=145.79.227.50:4373 , piece_id=4U55T5QDYCE6AZ3NCTPAVECPAELPQUKT3HV7GZL7JFRMWRN2AZUQ, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.471440098 (415h3m6.299148489s) [noise=true , address=145.79.227.47:4266 , piece_id=3NFYV2PWBNTAOSFHENM4J5PZ7XXE72UKTQVXLTVQRHGK4DUXAKDA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.560766475 (415h3m6.388474871s) [upload=431917]: Next returned nil segment Dec 22 22:31:05.560785161 (415h3m6.388493562s) [upload=431917]: Waiting for error group managing segments... Dec 22 22:31:05.566117878 (415h3m6.39382628s) [noise=true , address=45.91.116.4:4939 , piece_id=QYNKMKDLQJV7B6M2E34DU53Y7YI6VBCUW3C7ZLQOI4XWTBXZHKDQ, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.566723306 (415h3m6.394431687s) [noise=true , address=45.91.116.11:4117 , piece_id=WLMO5E3BIFFL3TOAUSSBGORIZ6DOCWFCMVT4AVFPPGPSKP2OKDAA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.566992667 (415h3m6.394701063s) [noise=true , address=145.79.227.36:4519 , piece_id=JUKRRIZWVGKEDD3GPJ7HMY2MEQZNMUCTKTXR6ZLSIERRONTZKHCQ, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.567464626 (415h3m6.395173019s) [noise=true , address=145.79.227.47:4266 , piece_id=3NFYV2PWBNTAOSFHENM4J5PZ7XXE72UKTQVXLTVQRHGK4DUXAKDA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.567924492 (415h3m6.395632883s) [noise=true , address=145.79.227.15:4874 , piece_id=FNCCIMIVJTGZ2X5TOYOZH727JXHYM7HG3HBHBTZDAG67SHZLYYYA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.568807294 (415h3m6.396515685s) [noise=true , address=145.79.227.51:4409 , piece_id=ATH3F4DFOUUIFFJZRA2SSRRL7AKWZMQNQRYOZCVFFDLTFCFP236A, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.570579823 (415h3m6.398288214s) [noise=true , address=145.79.227.48:4301 , piece_id=LBIYTJNOX5J4M3HDE7FGF5CSARBITW2AK3P2DSYFLFKUOIHZUKOQ, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.571632758 (415h3m6.399341149s) [noise=true , address=145.79.227.43:4157 , piece_id=3W3LVTU37DNHWS52IGIIH7SXWXVNPGEY27RQNT272L3Z64GNR4KA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.573168383 (415h3m6.400876774s) [noise=true , address=145.79.227.38:4979 , piece_id=HLMQC5OEB4OL4OZSRMCEVAWR5QM2T4MQUBPTHRVQLEAHPBWPQF6Q, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.579995420 (415h3m6.407703816s) [noise=true , address=45.91.116.14:4190 , piece_id=V4T6Z7K3C75LQEXNQIYZGYWW2CSQ7UPSET46QGDRT52RFGAROTTA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.592093172 (415h3m6.419801563s) [noise=true , address=145.79.227.37:4565 , piece_id=554RHTYPG27UNCFPSRMOB245AX4QXL2QFIYECNTVWNNJMAL4BTYA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:09.946500434 (415h3m10.774208816s) [noise=true , address=145.79.227.40:4051 , piece_id=55RRAD7DQX6SR7RBLYAV7MD7XRWPNIWHTVVBQZ2ZL3OE2L7LRRIQ, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:35.398235543 (415h3m36.225943943s) [noise=true , address=145.79.227.39:4013 , piece_id=JJVURGSLKY2UQ4FKNCLATA3ZBD5SDL32XJAG42KH2U6OYK3KTS6Q, seg_pos={0 0}, upload=431917]: Done uploading piece. err: ecclient: upload failed (node:121scLvKEW9zw2aEz22zpremcjiop67eQ8v2uqpmaD5aZzCAP7b, address:145.79.227.39:4013): protocol: write tcp 159.223.143.39:60842->145.79.227.39:4013: use of closed network connection; write tcp 159.223.143.39:60842->145.79.227.39:4013: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:60842->145.79.227.39:4013: use of closed network connection Dec 22 22:31:35.398562305 (415h3m36.226270699s) [noise=true , address=145.79.227.32:4835 , piece_id=BRE3JH3XKPXB6FN7WB3W6ED7MOFES6HYLR34YGR6L5AX7RPIQ7IA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: ecclient: upload failed (node:16vVfeoqCkbMYCRjMMWaKBnCvQBBJPMgwQtWTq9HKQrD9SS17y, address:145.79.227.32:4835): protocol: write tcp 159.223.143.39:45098->145.79.227.32:4835: use of closed network connection; write tcp 159.223.143.39:45098->145.79.227.32:4835: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:45098->145.79.227.32:4835: use of closed network connection Dec 22 22:31:35.399183028 (415h3m36.226891422s) [noise=true , address=145.79.227.50:4373 , piece_id=4U55T5QDYCE6AZ3NCTPAVECPAELPQUKT3HV7GZL7JFRMWRN2AZUQ, seg_pos={0 0}, upload=431917]: Done uploading piece. err: ecclient: upload failed (node:12AQGSkJuNJDxoeqhChUenWxHwF5yK41UESCF63FQz3XvabXXfz, address:145.79.227.50:4373): protocol: read tcp 159.223.143.39:56028->145.79.227.50:4373: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 22 22:31:35.399278021 (415h3m36.226986417s) [noise=true , address=145.79.227.29:4943 , piece_id=G4TMZ6IAIZBDON3UEDPH4LJQFHV3JLALEXSJ3VI4XN6WHIZOB57Q, seg_pos={0 0}, upload=431917]: Done uploading piece. err: ecclient: upload failed (node:12fc5Trj8XxacZhzxdksUVT5CbG6uTmiwdUPjreYi4UMPY1TXWz, address:145.79.227.29:4943): protocol: write tcp 159.223.143.39:50820->145.79.227.29:4943: use of closed network connection; write tcp 159.223.143.39:50820->145.79.227.29:4943: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:50820->145.79.227.29:4943: use of closed network connection Dec 22 22:31:35.399379485 (415h3m36.227087866s) [seg_pos={0 0}, upload=431917]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 22 22:31:35.421122544 (415h3m36.248830935s) [upload=431917]: Done uploading segments... Dec 22 22:31:35.421174369 (415h3m36.248882769s) [upload=431917]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 392.907016ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................................X..........................] duration: 456.60347ms trace: https://jaeger.dev.storj.io/trace/7013ffb1a981d605 FAIL (2025-12-22 22:51:52 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 22 22:31:05.439510308 (415h3m6.2672187s) [upload=431917]: Starting upload Dec 22 22:31:05.439581993 (415h3m6.26729037s) [upload=431917]: Done starting upload Dec 22 22:31:05.439812792 (415h3m6.267521185s) [upload=431917]: Uploading segments... Dec 22 22:31:05.439942524 (415h3m6.267650918s) [upload=431917]: Got next segment. Inline: false Dec 22 22:31:05.440022744 (415h3m6.267731174s) [upload=431917]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 22 22:31:05.440029670 (415h3m6.267738072s) [upload=431917]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 22 22:31:05.467186372 (415h3m6.294894769s) [seg_pos={0 0}, upload=431917]: Begin upload segment... Dec 22 22:31:05.467265149 (415h3m6.294973535s) [seg_pos={0 0}, upload=431917]: Done begin upload segment. Dec 22 22:31:05.467296974 (415h3m6.295005373s) [noise=true , address=145.79.227.51:4409 , piece_id=ATH3F4DFOUUIFFJZRA2SSRRL7AKWZMQNQRYOZCVFFDLTFCFP236A, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.467389853 (415h3m6.295098247s) [noise=true , address=145.79.227.39:4013 , piece_id=JJVURGSLKY2UQ4FKNCLATA3ZBD5SDL32XJAG42KH2U6OYK3KTS6Q, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.467912104 (415h3m6.295620486s) [noise=true , address=145.79.227.36:4519 , piece_id=JUKRRIZWVGKEDD3GPJ7HMY2MEQZNMUCTKTXR6ZLSIERRONTZKHCQ, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.468324863 (415h3m6.296033263s) [noise=true , address=145.79.227.38:4979 , piece_id=HLMQC5OEB4OL4OZSRMCEVAWR5QM2T4MQUBPTHRVQLEAHPBWPQF6Q, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.468334764 (415h3m6.296043147s) [noise=true , address=145.79.227.40:4051 , piece_id=55RRAD7DQX6SR7RBLYAV7MD7XRWPNIWHTVVBQZ2ZL3OE2L7LRRIQ, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.468742525 (415h3m6.296450916s) [noise=true , address=45.91.116.4:4939 , piece_id=QYNKMKDLQJV7B6M2E34DU53Y7YI6VBCUW3C7ZLQOI4XWTBXZHKDQ, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.468915034 (415h3m6.296623414s) [noise=true , address=45.91.116.11:4117 , piece_id=WLMO5E3BIFFL3TOAUSSBGORIZ6DOCWFCMVT4AVFPPGPSKP2OKDAA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.469069393 (415h3m6.296777787s) [noise=true , address=145.79.227.32:4835 , piece_id=BRE3JH3XKPXB6FN7WB3W6ED7MOFES6HYLR34YGR6L5AX7RPIQ7IA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.469285789 (415h3m6.296994183s) [noise=true , address=145.79.227.29:4943 , piece_id=G4TMZ6IAIZBDON3UEDPH4LJQFHV3JLALEXSJ3VI4XN6WHIZOB57Q, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.469411962 (415h3m6.297120355s) [noise=true , address=145.79.227.48:4301 , piece_id=LBIYTJNOX5J4M3HDE7FGF5CSARBITW2AK3P2DSYFLFKUOIHZUKOQ, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.469749379 (415h3m6.297457773s) [noise=true , address=145.79.227.37:4565 , piece_id=554RHTYPG27UNCFPSRMOB245AX4QXL2QFIYECNTVWNNJMAL4BTYA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.470080828 (415h3m6.29778922s) [noise=true , address=145.79.227.43:4157 , piece_id=3W3LVTU37DNHWS52IGIIH7SXWXVNPGEY27RQNT272L3Z64GNR4KA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.470477660 (415h3m6.298186058s) [noise=true , address=45.91.116.14:4190 , piece_id=V4T6Z7K3C75LQEXNQIYZGYWW2CSQ7UPSET46QGDRT52RFGAROTTA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.470567977 (415h3m6.29827637s) [noise=true , address=145.79.227.15:4874 , piece_id=FNCCIMIVJTGZ2X5TOYOZH727JXHYM7HG3HBHBTZDAG67SHZLYYYA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.470980950 (415h3m6.298689349s) [noise=true , address=145.79.227.50:4373 , piece_id=4U55T5QDYCE6AZ3NCTPAVECPAELPQUKT3HV7GZL7JFRMWRN2AZUQ, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.471440098 (415h3m6.299148489s) [noise=true , address=145.79.227.47:4266 , piece_id=3NFYV2PWBNTAOSFHENM4J5PZ7XXE72UKTQVXLTVQRHGK4DUXAKDA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.560766475 (415h3m6.388474871s) [upload=431917]: Next returned nil segment Dec 22 22:31:05.560785161 (415h3m6.388493562s) [upload=431917]: Waiting for error group managing segments... Dec 22 22:31:05.566117878 (415h3m6.39382628s) [noise=true , address=45.91.116.4:4939 , piece_id=QYNKMKDLQJV7B6M2E34DU53Y7YI6VBCUW3C7ZLQOI4XWTBXZHKDQ, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.566723306 (415h3m6.394431687s) [noise=true , address=45.91.116.11:4117 , piece_id=WLMO5E3BIFFL3TOAUSSBGORIZ6DOCWFCMVT4AVFPPGPSKP2OKDAA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.566992667 (415h3m6.394701063s) [noise=true , address=145.79.227.36:4519 , piece_id=JUKRRIZWVGKEDD3GPJ7HMY2MEQZNMUCTKTXR6ZLSIERRONTZKHCQ, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.567464626 (415h3m6.395173019s) [noise=true , address=145.79.227.47:4266 , piece_id=3NFYV2PWBNTAOSFHENM4J5PZ7XXE72UKTQVXLTVQRHGK4DUXAKDA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.567924492 (415h3m6.395632883s) [noise=true , address=145.79.227.15:4874 , piece_id=FNCCIMIVJTGZ2X5TOYOZH727JXHYM7HG3HBHBTZDAG67SHZLYYYA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.568807294 (415h3m6.396515685s) [noise=true , address=145.79.227.51:4409 , piece_id=ATH3F4DFOUUIFFJZRA2SSRRL7AKWZMQNQRYOZCVFFDLTFCFP236A, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.570579823 (415h3m6.398288214s) [noise=true , address=145.79.227.48:4301 , piece_id=LBIYTJNOX5J4M3HDE7FGF5CSARBITW2AK3P2DSYFLFKUOIHZUKOQ, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.571632758 (415h3m6.399341149s) [noise=true , address=145.79.227.43:4157 , piece_id=3W3LVTU37DNHWS52IGIIH7SXWXVNPGEY27RQNT272L3Z64GNR4KA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.573168383 (415h3m6.400876774s) [noise=true , address=145.79.227.38:4979 , piece_id=HLMQC5OEB4OL4OZSRMCEVAWR5QM2T4MQUBPTHRVQLEAHPBWPQF6Q, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.579995420 (415h3m6.407703816s) [noise=true , address=45.91.116.14:4190 , piece_id=V4T6Z7K3C75LQEXNQIYZGYWW2CSQ7UPSET46QGDRT52RFGAROTTA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.592093172 (415h3m6.419801563s) [noise=true , address=145.79.227.37:4565 , piece_id=554RHTYPG27UNCFPSRMOB245AX4QXL2QFIYECNTVWNNJMAL4BTYA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:09.946500434 (415h3m10.774208816s) [noise=true , address=145.79.227.40:4051 , piece_id=55RRAD7DQX6SR7RBLYAV7MD7XRWPNIWHTVVBQZ2ZL3OE2L7LRRIQ, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:35.398235543 (415h3m36.225943943s) [noise=true , address=145.79.227.39:4013 , piece_id=JJVURGSLKY2UQ4FKNCLATA3ZBD5SDL32XJAG42KH2U6OYK3KTS6Q, seg_pos={0 0}, upload=431917]: Done uploading piece. err: ecclient: upload failed (node:121scLvKEW9zw2aEz22zpremcjiop67eQ8v2uqpmaD5aZzCAP7b, address:145.79.227.39:4013): protocol: write tcp 159.223.143.39:60842->145.79.227.39:4013: use of closed network connection; write tcp 159.223.143.39:60842->145.79.227.39:4013: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:60842->145.79.227.39:4013: use of closed network connection Dec 22 22:31:35.398562305 (415h3m36.226270699s) [noise=true , address=145.79.227.32:4835 , piece_id=BRE3JH3XKPXB6FN7WB3W6ED7MOFES6HYLR34YGR6L5AX7RPIQ7IA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: ecclient: upload failed (node:16vVfeoqCkbMYCRjMMWaKBnCvQBBJPMgwQtWTq9HKQrD9SS17y, address:145.79.227.32:4835): protocol: write tcp 159.223.143.39:45098->145.79.227.32:4835: use of closed network connection; write tcp 159.223.143.39:45098->145.79.227.32:4835: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:45098->145.79.227.32:4835: use of closed network connection Dec 22 22:31:35.399183028 (415h3m36.226891422s) [noise=true , address=145.79.227.50:4373 , piece_id=4U55T5QDYCE6AZ3NCTPAVECPAELPQUKT3HV7GZL7JFRMWRN2AZUQ, seg_pos={0 0}, upload=431917]: Done uploading piece. err: ecclient: upload failed (node:12AQGSkJuNJDxoeqhChUenWxHwF5yK41UESCF63FQz3XvabXXfz, address:145.79.227.50:4373): protocol: read tcp 159.223.143.39:56028->145.79.227.50:4373: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 22 22:31:35.399278021 (415h3m36.226986417s) [noise=true , address=145.79.227.29:4943 , piece_id=G4TMZ6IAIZBDON3UEDPH4LJQFHV3JLALEXSJ3VI4XN6WHIZOB57Q, seg_pos={0 0}, upload=431917]: Done uploading piece. err: ecclient: upload failed (node:12fc5Trj8XxacZhzxdksUVT5CbG6uTmiwdUPjreYi4UMPY1TXWz, address:145.79.227.29:4943): protocol: write tcp 159.223.143.39:50820->145.79.227.29:4943: use of closed network connection; write tcp 159.223.143.39:50820->145.79.227.29:4943: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:50820->145.79.227.29:4943: use of closed network connection Dec 22 22:31:35.399379485 (415h3m36.227087866s) [seg_pos={0 0}, upload=431917]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 22 22:31:35.421122544 (415h3m36.248830935s) [upload=431917]: Done uploading segments... Dec 22 22:31:35.421174369 (415h3m36.248882769s) [upload=431917]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 456.631125ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................................X.........................] duration: 408.155014ms trace: https://jaeger.dev.storj.io/trace/7013ffb1a981d605 FAIL (2025-12-22 22:51:17 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 22 22:31:05.439510308 (415h3m6.2672187s) [upload=431917]: Starting upload Dec 22 22:31:05.439581993 (415h3m6.26729037s) [upload=431917]: Done starting upload Dec 22 22:31:05.439812792 (415h3m6.267521185s) [upload=431917]: Uploading segments... Dec 22 22:31:05.439942524 (415h3m6.267650918s) [upload=431917]: Got next segment. Inline: false Dec 22 22:31:05.440022744 (415h3m6.267731174s) [upload=431917]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 22 22:31:05.440029670 (415h3m6.267738072s) [upload=431917]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 22 22:31:05.467186372 (415h3m6.294894769s) [seg_pos={0 0}, upload=431917]: Begin upload segment... Dec 22 22:31:05.467265149 (415h3m6.294973535s) [seg_pos={0 0}, upload=431917]: Done begin upload segment. Dec 22 22:31:05.467296974 (415h3m6.295005373s) [noise=true , address=145.79.227.51:4409 , piece_id=ATH3F4DFOUUIFFJZRA2SSRRL7AKWZMQNQRYOZCVFFDLTFCFP236A, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.467389853 (415h3m6.295098247s) [noise=true , address=145.79.227.39:4013 , piece_id=JJVURGSLKY2UQ4FKNCLATA3ZBD5SDL32XJAG42KH2U6OYK3KTS6Q, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.467912104 (415h3m6.295620486s) [noise=true , address=145.79.227.36:4519 , piece_id=JUKRRIZWVGKEDD3GPJ7HMY2MEQZNMUCTKTXR6ZLSIERRONTZKHCQ, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.468324863 (415h3m6.296033263s) [noise=true , address=145.79.227.38:4979 , piece_id=HLMQC5OEB4OL4OZSRMCEVAWR5QM2T4MQUBPTHRVQLEAHPBWPQF6Q, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.468334764 (415h3m6.296043147s) [noise=true , address=145.79.227.40:4051 , piece_id=55RRAD7DQX6SR7RBLYAV7MD7XRWPNIWHTVVBQZ2ZL3OE2L7LRRIQ, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.468742525 (415h3m6.296450916s) [noise=true , address=45.91.116.4:4939 , piece_id=QYNKMKDLQJV7B6M2E34DU53Y7YI6VBCUW3C7ZLQOI4XWTBXZHKDQ, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.468915034 (415h3m6.296623414s) [noise=true , address=45.91.116.11:4117 , piece_id=WLMO5E3BIFFL3TOAUSSBGORIZ6DOCWFCMVT4AVFPPGPSKP2OKDAA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.469069393 (415h3m6.296777787s) [noise=true , address=145.79.227.32:4835 , piece_id=BRE3JH3XKPXB6FN7WB3W6ED7MOFES6HYLR34YGR6L5AX7RPIQ7IA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.469285789 (415h3m6.296994183s) [noise=true , address=145.79.227.29:4943 , piece_id=G4TMZ6IAIZBDON3UEDPH4LJQFHV3JLALEXSJ3VI4XN6WHIZOB57Q, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.469411962 (415h3m6.297120355s) [noise=true , address=145.79.227.48:4301 , piece_id=LBIYTJNOX5J4M3HDE7FGF5CSARBITW2AK3P2DSYFLFKUOIHZUKOQ, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.469749379 (415h3m6.297457773s) [noise=true , address=145.79.227.37:4565 , piece_id=554RHTYPG27UNCFPSRMOB245AX4QXL2QFIYECNTVWNNJMAL4BTYA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.470080828 (415h3m6.29778922s) [noise=true , address=145.79.227.43:4157 , piece_id=3W3LVTU37DNHWS52IGIIH7SXWXVNPGEY27RQNT272L3Z64GNR4KA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.470477660 (415h3m6.298186058s) [noise=true , address=45.91.116.14:4190 , piece_id=V4T6Z7K3C75LQEXNQIYZGYWW2CSQ7UPSET46QGDRT52RFGAROTTA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.470567977 (415h3m6.29827637s) [noise=true , address=145.79.227.15:4874 , piece_id=FNCCIMIVJTGZ2X5TOYOZH727JXHYM7HG3HBHBTZDAG67SHZLYYYA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.470980950 (415h3m6.298689349s) [noise=true , address=145.79.227.50:4373 , piece_id=4U55T5QDYCE6AZ3NCTPAVECPAELPQUKT3HV7GZL7JFRMWRN2AZUQ, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.471440098 (415h3m6.299148489s) [noise=true , address=145.79.227.47:4266 , piece_id=3NFYV2PWBNTAOSFHENM4J5PZ7XXE72UKTQVXLTVQRHGK4DUXAKDA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.560766475 (415h3m6.388474871s) [upload=431917]: Next returned nil segment Dec 22 22:31:05.560785161 (415h3m6.388493562s) [upload=431917]: Waiting for error group managing segments... Dec 22 22:31:05.566117878 (415h3m6.39382628s) [noise=true , address=45.91.116.4:4939 , piece_id=QYNKMKDLQJV7B6M2E34DU53Y7YI6VBCUW3C7ZLQOI4XWTBXZHKDQ, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.566723306 (415h3m6.394431687s) [noise=true , address=45.91.116.11:4117 , piece_id=WLMO5E3BIFFL3TOAUSSBGORIZ6DOCWFCMVT4AVFPPGPSKP2OKDAA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.566992667 (415h3m6.394701063s) [noise=true , address=145.79.227.36:4519 , piece_id=JUKRRIZWVGKEDD3GPJ7HMY2MEQZNMUCTKTXR6ZLSIERRONTZKHCQ, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.567464626 (415h3m6.395173019s) [noise=true , address=145.79.227.47:4266 , piece_id=3NFYV2PWBNTAOSFHENM4J5PZ7XXE72UKTQVXLTVQRHGK4DUXAKDA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.567924492 (415h3m6.395632883s) [noise=true , address=145.79.227.15:4874 , piece_id=FNCCIMIVJTGZ2X5TOYOZH727JXHYM7HG3HBHBTZDAG67SHZLYYYA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.568807294 (415h3m6.396515685s) [noise=true , address=145.79.227.51:4409 , piece_id=ATH3F4DFOUUIFFJZRA2SSRRL7AKWZMQNQRYOZCVFFDLTFCFP236A, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.570579823 (415h3m6.398288214s) [noise=true , address=145.79.227.48:4301 , piece_id=LBIYTJNOX5J4M3HDE7FGF5CSARBITW2AK3P2DSYFLFKUOIHZUKOQ, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.571632758 (415h3m6.399341149s) [noise=true , address=145.79.227.43:4157 , piece_id=3W3LVTU37DNHWS52IGIIH7SXWXVNPGEY27RQNT272L3Z64GNR4KA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.573168383 (415h3m6.400876774s) [noise=true , address=145.79.227.38:4979 , piece_id=HLMQC5OEB4OL4OZSRMCEVAWR5QM2T4MQUBPTHRVQLEAHPBWPQF6Q, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.579995420 (415h3m6.407703816s) [noise=true , address=45.91.116.14:4190 , piece_id=V4T6Z7K3C75LQEXNQIYZGYWW2CSQ7UPSET46QGDRT52RFGAROTTA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.592093172 (415h3m6.419801563s) [noise=true , address=145.79.227.37:4565 , piece_id=554RHTYPG27UNCFPSRMOB245AX4QXL2QFIYECNTVWNNJMAL4BTYA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:09.946500434 (415h3m10.774208816s) [noise=true , address=145.79.227.40:4051 , piece_id=55RRAD7DQX6SR7RBLYAV7MD7XRWPNIWHTVVBQZ2ZL3OE2L7LRRIQ, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:35.398235543 (415h3m36.225943943s) [noise=true , address=145.79.227.39:4013 , piece_id=JJVURGSLKY2UQ4FKNCLATA3ZBD5SDL32XJAG42KH2U6OYK3KTS6Q, seg_pos={0 0}, upload=431917]: Done uploading piece. err: ecclient: upload failed (node:121scLvKEW9zw2aEz22zpremcjiop67eQ8v2uqpmaD5aZzCAP7b, address:145.79.227.39:4013): protocol: write tcp 159.223.143.39:60842->145.79.227.39:4013: use of closed network connection; write tcp 159.223.143.39:60842->145.79.227.39:4013: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:60842->145.79.227.39:4013: use of closed network connection Dec 22 22:31:35.398562305 (415h3m36.226270699s) [noise=true , address=145.79.227.32:4835 , piece_id=BRE3JH3XKPXB6FN7WB3W6ED7MOFES6HYLR34YGR6L5AX7RPIQ7IA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: ecclient: upload failed (node:16vVfeoqCkbMYCRjMMWaKBnCvQBBJPMgwQtWTq9HKQrD9SS17y, address:145.79.227.32:4835): protocol: write tcp 159.223.143.39:45098->145.79.227.32:4835: use of closed network connection; write tcp 159.223.143.39:45098->145.79.227.32:4835: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:45098->145.79.227.32:4835: use of closed network connection Dec 22 22:31:35.399183028 (415h3m36.226891422s) [noise=true , address=145.79.227.50:4373 , piece_id=4U55T5QDYCE6AZ3NCTPAVECPAELPQUKT3HV7GZL7JFRMWRN2AZUQ, seg_pos={0 0}, upload=431917]: Done uploading piece. err: ecclient: upload failed (node:12AQGSkJuNJDxoeqhChUenWxHwF5yK41UESCF63FQz3XvabXXfz, address:145.79.227.50:4373): protocol: read tcp 159.223.143.39:56028->145.79.227.50:4373: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 22 22:31:35.399278021 (415h3m36.226986417s) [noise=true , address=145.79.227.29:4943 , piece_id=G4TMZ6IAIZBDON3UEDPH4LJQFHV3JLALEXSJ3VI4XN6WHIZOB57Q, seg_pos={0 0}, upload=431917]: Done uploading piece. err: ecclient: upload failed (node:12fc5Trj8XxacZhzxdksUVT5CbG6uTmiwdUPjreYi4UMPY1TXWz, address:145.79.227.29:4943): protocol: write tcp 159.223.143.39:50820->145.79.227.29:4943: use of closed network connection; write tcp 159.223.143.39:50820->145.79.227.29:4943: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:50820->145.79.227.29:4943: use of closed network connection Dec 22 22:31:35.399379485 (415h3m36.227087866s) [seg_pos={0 0}, upload=431917]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 22 22:31:35.421122544 (415h3m36.248830935s) [upload=431917]: Done uploading segments... Dec 22 22:31:35.421174369 (415h3m36.248882769s) [upload=431917]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 408.201167ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................................X........................] duration: 466.125041ms trace: https://jaeger.dev.storj.io/trace/7013ffb1a981d605 FAIL (2025-12-22 22:50:34 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 22 22:31:05.439510308 (415h3m6.2672187s) [upload=431917]: Starting upload Dec 22 22:31:05.439581993 (415h3m6.26729037s) [upload=431917]: Done starting upload Dec 22 22:31:05.439812792 (415h3m6.267521185s) [upload=431917]: Uploading segments... Dec 22 22:31:05.439942524 (415h3m6.267650918s) [upload=431917]: Got next segment. Inline: false Dec 22 22:31:05.440022744 (415h3m6.267731174s) [upload=431917]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 22 22:31:05.440029670 (415h3m6.267738072s) [upload=431917]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 22 22:31:05.467186372 (415h3m6.294894769s) [seg_pos={0 0}, upload=431917]: Begin upload segment... Dec 22 22:31:05.467265149 (415h3m6.294973535s) [seg_pos={0 0}, upload=431917]: Done begin upload segment. Dec 22 22:31:05.467296974 (415h3m6.295005373s) [noise=true , address=145.79.227.51:4409 , piece_id=ATH3F4DFOUUIFFJZRA2SSRRL7AKWZMQNQRYOZCVFFDLTFCFP236A, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.467389853 (415h3m6.295098247s) [noise=true , address=145.79.227.39:4013 , piece_id=JJVURGSLKY2UQ4FKNCLATA3ZBD5SDL32XJAG42KH2U6OYK3KTS6Q, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.467912104 (415h3m6.295620486s) [noise=true , address=145.79.227.36:4519 , piece_id=JUKRRIZWVGKEDD3GPJ7HMY2MEQZNMUCTKTXR6ZLSIERRONTZKHCQ, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.468324863 (415h3m6.296033263s) [noise=true , address=145.79.227.38:4979 , piece_id=HLMQC5OEB4OL4OZSRMCEVAWR5QM2T4MQUBPTHRVQLEAHPBWPQF6Q, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.468334764 (415h3m6.296043147s) [noise=true , address=145.79.227.40:4051 , piece_id=55RRAD7DQX6SR7RBLYAV7MD7XRWPNIWHTVVBQZ2ZL3OE2L7LRRIQ, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.468742525 (415h3m6.296450916s) [noise=true , address=45.91.116.4:4939 , piece_id=QYNKMKDLQJV7B6M2E34DU53Y7YI6VBCUW3C7ZLQOI4XWTBXZHKDQ, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.468915034 (415h3m6.296623414s) [noise=true , address=45.91.116.11:4117 , piece_id=WLMO5E3BIFFL3TOAUSSBGORIZ6DOCWFCMVT4AVFPPGPSKP2OKDAA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.469069393 (415h3m6.296777787s) [noise=true , address=145.79.227.32:4835 , piece_id=BRE3JH3XKPXB6FN7WB3W6ED7MOFES6HYLR34YGR6L5AX7RPIQ7IA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.469285789 (415h3m6.296994183s) [noise=true , address=145.79.227.29:4943 , piece_id=G4TMZ6IAIZBDON3UEDPH4LJQFHV3JLALEXSJ3VI4XN6WHIZOB57Q, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.469411962 (415h3m6.297120355s) [noise=true , address=145.79.227.48:4301 , piece_id=LBIYTJNOX5J4M3HDE7FGF5CSARBITW2AK3P2DSYFLFKUOIHZUKOQ, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.469749379 (415h3m6.297457773s) [noise=true , address=145.79.227.37:4565 , piece_id=554RHTYPG27UNCFPSRMOB245AX4QXL2QFIYECNTVWNNJMAL4BTYA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.470080828 (415h3m6.29778922s) [noise=true , address=145.79.227.43:4157 , piece_id=3W3LVTU37DNHWS52IGIIH7SXWXVNPGEY27RQNT272L3Z64GNR4KA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.470477660 (415h3m6.298186058s) [noise=true , address=45.91.116.14:4190 , piece_id=V4T6Z7K3C75LQEXNQIYZGYWW2CSQ7UPSET46QGDRT52RFGAROTTA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.470567977 (415h3m6.29827637s) [noise=true , address=145.79.227.15:4874 , piece_id=FNCCIMIVJTGZ2X5TOYOZH727JXHYM7HG3HBHBTZDAG67SHZLYYYA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.470980950 (415h3m6.298689349s) [noise=true , address=145.79.227.50:4373 , piece_id=4U55T5QDYCE6AZ3NCTPAVECPAELPQUKT3HV7GZL7JFRMWRN2AZUQ, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.471440098 (415h3m6.299148489s) [noise=true , address=145.79.227.47:4266 , piece_id=3NFYV2PWBNTAOSFHENM4J5PZ7XXE72UKTQVXLTVQRHGK4DUXAKDA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.560766475 (415h3m6.388474871s) [upload=431917]: Next returned nil segment Dec 22 22:31:05.560785161 (415h3m6.388493562s) [upload=431917]: Waiting for error group managing segments... Dec 22 22:31:05.566117878 (415h3m6.39382628s) [noise=true , address=45.91.116.4:4939 , piece_id=QYNKMKDLQJV7B6M2E34DU53Y7YI6VBCUW3C7ZLQOI4XWTBXZHKDQ, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.566723306 (415h3m6.394431687s) [noise=true , address=45.91.116.11:4117 , piece_id=WLMO5E3BIFFL3TOAUSSBGORIZ6DOCWFCMVT4AVFPPGPSKP2OKDAA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.566992667 (415h3m6.394701063s) [noise=true , address=145.79.227.36:4519 , piece_id=JUKRRIZWVGKEDD3GPJ7HMY2MEQZNMUCTKTXR6ZLSIERRONTZKHCQ, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.567464626 (415h3m6.395173019s) [noise=true , address=145.79.227.47:4266 , piece_id=3NFYV2PWBNTAOSFHENM4J5PZ7XXE72UKTQVXLTVQRHGK4DUXAKDA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.567924492 (415h3m6.395632883s) [noise=true , address=145.79.227.15:4874 , piece_id=FNCCIMIVJTGZ2X5TOYOZH727JXHYM7HG3HBHBTZDAG67SHZLYYYA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.568807294 (415h3m6.396515685s) [noise=true , address=145.79.227.51:4409 , piece_id=ATH3F4DFOUUIFFJZRA2SSRRL7AKWZMQNQRYOZCVFFDLTFCFP236A, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.570579823 (415h3m6.398288214s) [noise=true , address=145.79.227.48:4301 , piece_id=LBIYTJNOX5J4M3HDE7FGF5CSARBITW2AK3P2DSYFLFKUOIHZUKOQ, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.571632758 (415h3m6.399341149s) [noise=true , address=145.79.227.43:4157 , piece_id=3W3LVTU37DNHWS52IGIIH7SXWXVNPGEY27RQNT272L3Z64GNR4KA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.573168383 (415h3m6.400876774s) [noise=true , address=145.79.227.38:4979 , piece_id=HLMQC5OEB4OL4OZSRMCEVAWR5QM2T4MQUBPTHRVQLEAHPBWPQF6Q, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.579995420 (415h3m6.407703816s) [noise=true , address=45.91.116.14:4190 , piece_id=V4T6Z7K3C75LQEXNQIYZGYWW2CSQ7UPSET46QGDRT52RFGAROTTA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.592093172 (415h3m6.419801563s) [noise=true , address=145.79.227.37:4565 , piece_id=554RHTYPG27UNCFPSRMOB245AX4QXL2QFIYECNTVWNNJMAL4BTYA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:09.946500434 (415h3m10.774208816s) [noise=true , address=145.79.227.40:4051 , piece_id=55RRAD7DQX6SR7RBLYAV7MD7XRWPNIWHTVVBQZ2ZL3OE2L7LRRIQ, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:35.398235543 (415h3m36.225943943s) [noise=true , address=145.79.227.39:4013 , piece_id=JJVURGSLKY2UQ4FKNCLATA3ZBD5SDL32XJAG42KH2U6OYK3KTS6Q, seg_pos={0 0}, upload=431917]: Done uploading piece. err: ecclient: upload failed (node:121scLvKEW9zw2aEz22zpremcjiop67eQ8v2uqpmaD5aZzCAP7b, address:145.79.227.39:4013): protocol: write tcp 159.223.143.39:60842->145.79.227.39:4013: use of closed network connection; write tcp 159.223.143.39:60842->145.79.227.39:4013: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:60842->145.79.227.39:4013: use of closed network connection Dec 22 22:31:35.398562305 (415h3m36.226270699s) [noise=true , address=145.79.227.32:4835 , piece_id=BRE3JH3XKPXB6FN7WB3W6ED7MOFES6HYLR34YGR6L5AX7RPIQ7IA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: ecclient: upload failed (node:16vVfeoqCkbMYCRjMMWaKBnCvQBBJPMgwQtWTq9HKQrD9SS17y, address:145.79.227.32:4835): protocol: write tcp 159.223.143.39:45098->145.79.227.32:4835: use of closed network connection; write tcp 159.223.143.39:45098->145.79.227.32:4835: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:45098->145.79.227.32:4835: use of closed network connection Dec 22 22:31:35.399183028 (415h3m36.226891422s) [noise=true , address=145.79.227.50:4373 , piece_id=4U55T5QDYCE6AZ3NCTPAVECPAELPQUKT3HV7GZL7JFRMWRN2AZUQ, seg_pos={0 0}, upload=431917]: Done uploading piece. err: ecclient: upload failed (node:12AQGSkJuNJDxoeqhChUenWxHwF5yK41UESCF63FQz3XvabXXfz, address:145.79.227.50:4373): protocol: read tcp 159.223.143.39:56028->145.79.227.50:4373: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 22 22:31:35.399278021 (415h3m36.226986417s) [noise=true , address=145.79.227.29:4943 , piece_id=G4TMZ6IAIZBDON3UEDPH4LJQFHV3JLALEXSJ3VI4XN6WHIZOB57Q, seg_pos={0 0}, upload=431917]: Done uploading piece. err: ecclient: upload failed (node:12fc5Trj8XxacZhzxdksUVT5CbG6uTmiwdUPjreYi4UMPY1TXWz, address:145.79.227.29:4943): protocol: write tcp 159.223.143.39:50820->145.79.227.29:4943: use of closed network connection; write tcp 159.223.143.39:50820->145.79.227.29:4943: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:50820->145.79.227.29:4943: use of closed network connection Dec 22 22:31:35.399379485 (415h3m36.227087866s) [seg_pos={0 0}, upload=431917]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 22 22:31:35.421122544 (415h3m36.248830935s) [upload=431917]: Done uploading segments... Dec 22 22:31:35.421174369 (415h3m36.248882769s) [upload=431917]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 466.156011ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................................X.......................] duration: 511.695585ms trace: https://jaeger.dev.storj.io/trace/7013ffb1a981d605 FAIL (2025-12-22 22:49:41 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 22 22:31:05.439510308 (415h3m6.2672187s) [upload=431917]: Starting upload Dec 22 22:31:05.439581993 (415h3m6.26729037s) [upload=431917]: Done starting upload Dec 22 22:31:05.439812792 (415h3m6.267521185s) [upload=431917]: Uploading segments... Dec 22 22:31:05.439942524 (415h3m6.267650918s) [upload=431917]: Got next segment. Inline: false Dec 22 22:31:05.440022744 (415h3m6.267731174s) [upload=431917]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 22 22:31:05.440029670 (415h3m6.267738072s) [upload=431917]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 22 22:31:05.467186372 (415h3m6.294894769s) [seg_pos={0 0}, upload=431917]: Begin upload segment... Dec 22 22:31:05.467265149 (415h3m6.294973535s) [seg_pos={0 0}, upload=431917]: Done begin upload segment. Dec 22 22:31:05.467296974 (415h3m6.295005373s) [noise=true , address=145.79.227.51:4409 , piece_id=ATH3F4DFOUUIFFJZRA2SSRRL7AKWZMQNQRYOZCVFFDLTFCFP236A, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.467389853 (415h3m6.295098247s) [noise=true , address=145.79.227.39:4013 , piece_id=JJVURGSLKY2UQ4FKNCLATA3ZBD5SDL32XJAG42KH2U6OYK3KTS6Q, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.467912104 (415h3m6.295620486s) [noise=true , address=145.79.227.36:4519 , piece_id=JUKRRIZWVGKEDD3GPJ7HMY2MEQZNMUCTKTXR6ZLSIERRONTZKHCQ, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.468324863 (415h3m6.296033263s) [noise=true , address=145.79.227.38:4979 , piece_id=HLMQC5OEB4OL4OZSRMCEVAWR5QM2T4MQUBPTHRVQLEAHPBWPQF6Q, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.468334764 (415h3m6.296043147s) [noise=true , address=145.79.227.40:4051 , piece_id=55RRAD7DQX6SR7RBLYAV7MD7XRWPNIWHTVVBQZ2ZL3OE2L7LRRIQ, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.468742525 (415h3m6.296450916s) [noise=true , address=45.91.116.4:4939 , piece_id=QYNKMKDLQJV7B6M2E34DU53Y7YI6VBCUW3C7ZLQOI4XWTBXZHKDQ, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.468915034 (415h3m6.296623414s) [noise=true , address=45.91.116.11:4117 , piece_id=WLMO5E3BIFFL3TOAUSSBGORIZ6DOCWFCMVT4AVFPPGPSKP2OKDAA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.469069393 (415h3m6.296777787s) [noise=true , address=145.79.227.32:4835 , piece_id=BRE3JH3XKPXB6FN7WB3W6ED7MOFES6HYLR34YGR6L5AX7RPIQ7IA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.469285789 (415h3m6.296994183s) [noise=true , address=145.79.227.29:4943 , piece_id=G4TMZ6IAIZBDON3UEDPH4LJQFHV3JLALEXSJ3VI4XN6WHIZOB57Q, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.469411962 (415h3m6.297120355s) [noise=true , address=145.79.227.48:4301 , piece_id=LBIYTJNOX5J4M3HDE7FGF5CSARBITW2AK3P2DSYFLFKUOIHZUKOQ, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.469749379 (415h3m6.297457773s) [noise=true , address=145.79.227.37:4565 , piece_id=554RHTYPG27UNCFPSRMOB245AX4QXL2QFIYECNTVWNNJMAL4BTYA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.470080828 (415h3m6.29778922s) [noise=true , address=145.79.227.43:4157 , piece_id=3W3LVTU37DNHWS52IGIIH7SXWXVNPGEY27RQNT272L3Z64GNR4KA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.470477660 (415h3m6.298186058s) [noise=true , address=45.91.116.14:4190 , piece_id=V4T6Z7K3C75LQEXNQIYZGYWW2CSQ7UPSET46QGDRT52RFGAROTTA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.470567977 (415h3m6.29827637s) [noise=true , address=145.79.227.15:4874 , piece_id=FNCCIMIVJTGZ2X5TOYOZH727JXHYM7HG3HBHBTZDAG67SHZLYYYA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.470980950 (415h3m6.298689349s) [noise=true , address=145.79.227.50:4373 , piece_id=4U55T5QDYCE6AZ3NCTPAVECPAELPQUKT3HV7GZL7JFRMWRN2AZUQ, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.471440098 (415h3m6.299148489s) [noise=true , address=145.79.227.47:4266 , piece_id=3NFYV2PWBNTAOSFHENM4J5PZ7XXE72UKTQVXLTVQRHGK4DUXAKDA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.560766475 (415h3m6.388474871s) [upload=431917]: Next returned nil segment Dec 22 22:31:05.560785161 (415h3m6.388493562s) [upload=431917]: Waiting for error group managing segments... Dec 22 22:31:05.566117878 (415h3m6.39382628s) [noise=true , address=45.91.116.4:4939 , piece_id=QYNKMKDLQJV7B6M2E34DU53Y7YI6VBCUW3C7ZLQOI4XWTBXZHKDQ, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.566723306 (415h3m6.394431687s) [noise=true , address=45.91.116.11:4117 , piece_id=WLMO5E3BIFFL3TOAUSSBGORIZ6DOCWFCMVT4AVFPPGPSKP2OKDAA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.566992667 (415h3m6.394701063s) [noise=true , address=145.79.227.36:4519 , piece_id=JUKRRIZWVGKEDD3GPJ7HMY2MEQZNMUCTKTXR6ZLSIERRONTZKHCQ, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.567464626 (415h3m6.395173019s) [noise=true , address=145.79.227.47:4266 , piece_id=3NFYV2PWBNTAOSFHENM4J5PZ7XXE72UKTQVXLTVQRHGK4DUXAKDA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.567924492 (415h3m6.395632883s) [noise=true , address=145.79.227.15:4874 , piece_id=FNCCIMIVJTGZ2X5TOYOZH727JXHYM7HG3HBHBTZDAG67SHZLYYYA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.568807294 (415h3m6.396515685s) [noise=true , address=145.79.227.51:4409 , piece_id=ATH3F4DFOUUIFFJZRA2SSRRL7AKWZMQNQRYOZCVFFDLTFCFP236A, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.570579823 (415h3m6.398288214s) [noise=true , address=145.79.227.48:4301 , piece_id=LBIYTJNOX5J4M3HDE7FGF5CSARBITW2AK3P2DSYFLFKUOIHZUKOQ, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.571632758 (415h3m6.399341149s) [noise=true , address=145.79.227.43:4157 , piece_id=3W3LVTU37DNHWS52IGIIH7SXWXVNPGEY27RQNT272L3Z64GNR4KA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.573168383 (415h3m6.400876774s) [noise=true , address=145.79.227.38:4979 , piece_id=HLMQC5OEB4OL4OZSRMCEVAWR5QM2T4MQUBPTHRVQLEAHPBWPQF6Q, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.579995420 (415h3m6.407703816s) [noise=true , address=45.91.116.14:4190 , piece_id=V4T6Z7K3C75LQEXNQIYZGYWW2CSQ7UPSET46QGDRT52RFGAROTTA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.592093172 (415h3m6.419801563s) [noise=true , address=145.79.227.37:4565 , piece_id=554RHTYPG27UNCFPSRMOB245AX4QXL2QFIYECNTVWNNJMAL4BTYA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:09.946500434 (415h3m10.774208816s) [noise=true , address=145.79.227.40:4051 , piece_id=55RRAD7DQX6SR7RBLYAV7MD7XRWPNIWHTVVBQZ2ZL3OE2L7LRRIQ, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:35.398235543 (415h3m36.225943943s) [noise=true , address=145.79.227.39:4013 , piece_id=JJVURGSLKY2UQ4FKNCLATA3ZBD5SDL32XJAG42KH2U6OYK3KTS6Q, seg_pos={0 0}, upload=431917]: Done uploading piece. err: ecclient: upload failed (node:121scLvKEW9zw2aEz22zpremcjiop67eQ8v2uqpmaD5aZzCAP7b, address:145.79.227.39:4013): protocol: write tcp 159.223.143.39:60842->145.79.227.39:4013: use of closed network connection; write tcp 159.223.143.39:60842->145.79.227.39:4013: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:60842->145.79.227.39:4013: use of closed network connection Dec 22 22:31:35.398562305 (415h3m36.226270699s) [noise=true , address=145.79.227.32:4835 , piece_id=BRE3JH3XKPXB6FN7WB3W6ED7MOFES6HYLR34YGR6L5AX7RPIQ7IA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: ecclient: upload failed (node:16vVfeoqCkbMYCRjMMWaKBnCvQBBJPMgwQtWTq9HKQrD9SS17y, address:145.79.227.32:4835): protocol: write tcp 159.223.143.39:45098->145.79.227.32:4835: use of closed network connection; write tcp 159.223.143.39:45098->145.79.227.32:4835: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:45098->145.79.227.32:4835: use of closed network connection Dec 22 22:31:35.399183028 (415h3m36.226891422s) [noise=true , address=145.79.227.50:4373 , piece_id=4U55T5QDYCE6AZ3NCTPAVECPAELPQUKT3HV7GZL7JFRMWRN2AZUQ, seg_pos={0 0}, upload=431917]: Done uploading piece. err: ecclient: upload failed (node:12AQGSkJuNJDxoeqhChUenWxHwF5yK41UESCF63FQz3XvabXXfz, address:145.79.227.50:4373): protocol: read tcp 159.223.143.39:56028->145.79.227.50:4373: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 22 22:31:35.399278021 (415h3m36.226986417s) [noise=true , address=145.79.227.29:4943 , piece_id=G4TMZ6IAIZBDON3UEDPH4LJQFHV3JLALEXSJ3VI4XN6WHIZOB57Q, seg_pos={0 0}, upload=431917]: Done uploading piece. err: ecclient: upload failed (node:12fc5Trj8XxacZhzxdksUVT5CbG6uTmiwdUPjreYi4UMPY1TXWz, address:145.79.227.29:4943): protocol: write tcp 159.223.143.39:50820->145.79.227.29:4943: use of closed network connection; write tcp 159.223.143.39:50820->145.79.227.29:4943: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:50820->145.79.227.29:4943: use of closed network connection Dec 22 22:31:35.399379485 (415h3m36.227087866s) [seg_pos={0 0}, upload=431917]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 22 22:31:35.421122544 (415h3m36.248830935s) [upload=431917]: Done uploading segments... Dec 22 22:31:35.421174369 (415h3m36.248882769s) [upload=431917]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 511.79667ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................................X......................] duration: 476.416567ms trace: https://jaeger.dev.storj.io/trace/7013ffb1a981d605 FAIL (2025-12-22 22:49:10 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 22 22:31:05.439510308 (415h3m6.2672187s) [upload=431917]: Starting upload Dec 22 22:31:05.439581993 (415h3m6.26729037s) [upload=431917]: Done starting upload Dec 22 22:31:05.439812792 (415h3m6.267521185s) [upload=431917]: Uploading segments... Dec 22 22:31:05.439942524 (415h3m6.267650918s) [upload=431917]: Got next segment. Inline: false Dec 22 22:31:05.440022744 (415h3m6.267731174s) [upload=431917]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 22 22:31:05.440029670 (415h3m6.267738072s) [upload=431917]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 22 22:31:05.467186372 (415h3m6.294894769s) [seg_pos={0 0}, upload=431917]: Begin upload segment... Dec 22 22:31:05.467265149 (415h3m6.294973535s) [seg_pos={0 0}, upload=431917]: Done begin upload segment. Dec 22 22:31:05.467296974 (415h3m6.295005373s) [noise=true , address=145.79.227.51:4409 , piece_id=ATH3F4DFOUUIFFJZRA2SSRRL7AKWZMQNQRYOZCVFFDLTFCFP236A, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.467389853 (415h3m6.295098247s) [noise=true , address=145.79.227.39:4013 , piece_id=JJVURGSLKY2UQ4FKNCLATA3ZBD5SDL32XJAG42KH2U6OYK3KTS6Q, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.467912104 (415h3m6.295620486s) [noise=true , address=145.79.227.36:4519 , piece_id=JUKRRIZWVGKEDD3GPJ7HMY2MEQZNMUCTKTXR6ZLSIERRONTZKHCQ, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.468324863 (415h3m6.296033263s) [noise=true , address=145.79.227.38:4979 , piece_id=HLMQC5OEB4OL4OZSRMCEVAWR5QM2T4MQUBPTHRVQLEAHPBWPQF6Q, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.468334764 (415h3m6.296043147s) [noise=true , address=145.79.227.40:4051 , piece_id=55RRAD7DQX6SR7RBLYAV7MD7XRWPNIWHTVVBQZ2ZL3OE2L7LRRIQ, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.468742525 (415h3m6.296450916s) [noise=true , address=45.91.116.4:4939 , piece_id=QYNKMKDLQJV7B6M2E34DU53Y7YI6VBCUW3C7ZLQOI4XWTBXZHKDQ, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.468915034 (415h3m6.296623414s) [noise=true , address=45.91.116.11:4117 , piece_id=WLMO5E3BIFFL3TOAUSSBGORIZ6DOCWFCMVT4AVFPPGPSKP2OKDAA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.469069393 (415h3m6.296777787s) [noise=true , address=145.79.227.32:4835 , piece_id=BRE3JH3XKPXB6FN7WB3W6ED7MOFES6HYLR34YGR6L5AX7RPIQ7IA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.469285789 (415h3m6.296994183s) [noise=true , address=145.79.227.29:4943 , piece_id=G4TMZ6IAIZBDON3UEDPH4LJQFHV3JLALEXSJ3VI4XN6WHIZOB57Q, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.469411962 (415h3m6.297120355s) [noise=true , address=145.79.227.48:4301 , piece_id=LBIYTJNOX5J4M3HDE7FGF5CSARBITW2AK3P2DSYFLFKUOIHZUKOQ, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.469749379 (415h3m6.297457773s) [noise=true , address=145.79.227.37:4565 , piece_id=554RHTYPG27UNCFPSRMOB245AX4QXL2QFIYECNTVWNNJMAL4BTYA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.470080828 (415h3m6.29778922s) [noise=true , address=145.79.227.43:4157 , piece_id=3W3LVTU37DNHWS52IGIIH7SXWXVNPGEY27RQNT272L3Z64GNR4KA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.470477660 (415h3m6.298186058s) [noise=true , address=45.91.116.14:4190 , piece_id=V4T6Z7K3C75LQEXNQIYZGYWW2CSQ7UPSET46QGDRT52RFGAROTTA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.470567977 (415h3m6.29827637s) [noise=true , address=145.79.227.15:4874 , piece_id=FNCCIMIVJTGZ2X5TOYOZH727JXHYM7HG3HBHBTZDAG67SHZLYYYA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.470980950 (415h3m6.298689349s) [noise=true , address=145.79.227.50:4373 , piece_id=4U55T5QDYCE6AZ3NCTPAVECPAELPQUKT3HV7GZL7JFRMWRN2AZUQ, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.471440098 (415h3m6.299148489s) [noise=true , address=145.79.227.47:4266 , piece_id=3NFYV2PWBNTAOSFHENM4J5PZ7XXE72UKTQVXLTVQRHGK4DUXAKDA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.560766475 (415h3m6.388474871s) [upload=431917]: Next returned nil segment Dec 22 22:31:05.560785161 (415h3m6.388493562s) [upload=431917]: Waiting for error group managing segments... Dec 22 22:31:05.566117878 (415h3m6.39382628s) [noise=true , address=45.91.116.4:4939 , piece_id=QYNKMKDLQJV7B6M2E34DU53Y7YI6VBCUW3C7ZLQOI4XWTBXZHKDQ, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.566723306 (415h3m6.394431687s) [noise=true , address=45.91.116.11:4117 , piece_id=WLMO5E3BIFFL3TOAUSSBGORIZ6DOCWFCMVT4AVFPPGPSKP2OKDAA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.566992667 (415h3m6.394701063s) [noise=true , address=145.79.227.36:4519 , piece_id=JUKRRIZWVGKEDD3GPJ7HMY2MEQZNMUCTKTXR6ZLSIERRONTZKHCQ, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.567464626 (415h3m6.395173019s) [noise=true , address=145.79.227.47:4266 , piece_id=3NFYV2PWBNTAOSFHENM4J5PZ7XXE72UKTQVXLTVQRHGK4DUXAKDA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.567924492 (415h3m6.395632883s) [noise=true , address=145.79.227.15:4874 , piece_id=FNCCIMIVJTGZ2X5TOYOZH727JXHYM7HG3HBHBTZDAG67SHZLYYYA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.568807294 (415h3m6.396515685s) [noise=true , address=145.79.227.51:4409 , piece_id=ATH3F4DFOUUIFFJZRA2SSRRL7AKWZMQNQRYOZCVFFDLTFCFP236A, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.570579823 (415h3m6.398288214s) [noise=true , address=145.79.227.48:4301 , piece_id=LBIYTJNOX5J4M3HDE7FGF5CSARBITW2AK3P2DSYFLFKUOIHZUKOQ, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.571632758 (415h3m6.399341149s) [noise=true , address=145.79.227.43:4157 , piece_id=3W3LVTU37DNHWS52IGIIH7SXWXVNPGEY27RQNT272L3Z64GNR4KA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.573168383 (415h3m6.400876774s) [noise=true , address=145.79.227.38:4979 , piece_id=HLMQC5OEB4OL4OZSRMCEVAWR5QM2T4MQUBPTHRVQLEAHPBWPQF6Q, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.579995420 (415h3m6.407703816s) [noise=true , address=45.91.116.14:4190 , piece_id=V4T6Z7K3C75LQEXNQIYZGYWW2CSQ7UPSET46QGDRT52RFGAROTTA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.592093172 (415h3m6.419801563s) [noise=true , address=145.79.227.37:4565 , piece_id=554RHTYPG27UNCFPSRMOB245AX4QXL2QFIYECNTVWNNJMAL4BTYA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:09.946500434 (415h3m10.774208816s) [noise=true , address=145.79.227.40:4051 , piece_id=55RRAD7DQX6SR7RBLYAV7MD7XRWPNIWHTVVBQZ2ZL3OE2L7LRRIQ, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:35.398235543 (415h3m36.225943943s) [noise=true , address=145.79.227.39:4013 , piece_id=JJVURGSLKY2UQ4FKNCLATA3ZBD5SDL32XJAG42KH2U6OYK3KTS6Q, seg_pos={0 0}, upload=431917]: Done uploading piece. err: ecclient: upload failed (node:121scLvKEW9zw2aEz22zpremcjiop67eQ8v2uqpmaD5aZzCAP7b, address:145.79.227.39:4013): protocol: write tcp 159.223.143.39:60842->145.79.227.39:4013: use of closed network connection; write tcp 159.223.143.39:60842->145.79.227.39:4013: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:60842->145.79.227.39:4013: use of closed network connection Dec 22 22:31:35.398562305 (415h3m36.226270699s) [noise=true , address=145.79.227.32:4835 , piece_id=BRE3JH3XKPXB6FN7WB3W6ED7MOFES6HYLR34YGR6L5AX7RPIQ7IA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: ecclient: upload failed (node:16vVfeoqCkbMYCRjMMWaKBnCvQBBJPMgwQtWTq9HKQrD9SS17y, address:145.79.227.32:4835): protocol: write tcp 159.223.143.39:45098->145.79.227.32:4835: use of closed network connection; write tcp 159.223.143.39:45098->145.79.227.32:4835: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:45098->145.79.227.32:4835: use of closed network connection Dec 22 22:31:35.399183028 (415h3m36.226891422s) [noise=true , address=145.79.227.50:4373 , piece_id=4U55T5QDYCE6AZ3NCTPAVECPAELPQUKT3HV7GZL7JFRMWRN2AZUQ, seg_pos={0 0}, upload=431917]: Done uploading piece. err: ecclient: upload failed (node:12AQGSkJuNJDxoeqhChUenWxHwF5yK41UESCF63FQz3XvabXXfz, address:145.79.227.50:4373): protocol: read tcp 159.223.143.39:56028->145.79.227.50:4373: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 22 22:31:35.399278021 (415h3m36.226986417s) [noise=true , address=145.79.227.29:4943 , piece_id=G4TMZ6IAIZBDON3UEDPH4LJQFHV3JLALEXSJ3VI4XN6WHIZOB57Q, seg_pos={0 0}, upload=431917]: Done uploading piece. err: ecclient: upload failed (node:12fc5Trj8XxacZhzxdksUVT5CbG6uTmiwdUPjreYi4UMPY1TXWz, address:145.79.227.29:4943): protocol: write tcp 159.223.143.39:50820->145.79.227.29:4943: use of closed network connection; write tcp 159.223.143.39:50820->145.79.227.29:4943: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:50820->145.79.227.29:4943: use of closed network connection Dec 22 22:31:35.399379485 (415h3m36.227087866s) [seg_pos={0 0}, upload=431917]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 22 22:31:35.421122544 (415h3m36.248830935s) [upload=431917]: Done uploading segments... Dec 22 22:31:35.421174369 (415h3m36.248882769s) [upload=431917]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 476.441836ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................................X.....................] duration: 440.504099ms trace: https://jaeger.dev.storj.io/trace/7013ffb1a981d605 FAIL (2025-12-22 22:48:18 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 22 22:31:05.439510308 (415h3m6.2672187s) [upload=431917]: Starting upload Dec 22 22:31:05.439581993 (415h3m6.26729037s) [upload=431917]: Done starting upload Dec 22 22:31:05.439812792 (415h3m6.267521185s) [upload=431917]: Uploading segments... Dec 22 22:31:05.439942524 (415h3m6.267650918s) [upload=431917]: Got next segment. Inline: false Dec 22 22:31:05.440022744 (415h3m6.267731174s) [upload=431917]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 22 22:31:05.440029670 (415h3m6.267738072s) [upload=431917]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 22 22:31:05.467186372 (415h3m6.294894769s) [seg_pos={0 0}, upload=431917]: Begin upload segment... Dec 22 22:31:05.467265149 (415h3m6.294973535s) [seg_pos={0 0}, upload=431917]: Done begin upload segment. Dec 22 22:31:05.467296974 (415h3m6.295005373s) [noise=true , address=145.79.227.51:4409 , piece_id=ATH3F4DFOUUIFFJZRA2SSRRL7AKWZMQNQRYOZCVFFDLTFCFP236A, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.467389853 (415h3m6.295098247s) [noise=true , address=145.79.227.39:4013 , piece_id=JJVURGSLKY2UQ4FKNCLATA3ZBD5SDL32XJAG42KH2U6OYK3KTS6Q, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.467912104 (415h3m6.295620486s) [noise=true , address=145.79.227.36:4519 , piece_id=JUKRRIZWVGKEDD3GPJ7HMY2MEQZNMUCTKTXR6ZLSIERRONTZKHCQ, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.468324863 (415h3m6.296033263s) [noise=true , address=145.79.227.38:4979 , piece_id=HLMQC5OEB4OL4OZSRMCEVAWR5QM2T4MQUBPTHRVQLEAHPBWPQF6Q, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.468334764 (415h3m6.296043147s) [noise=true , address=145.79.227.40:4051 , piece_id=55RRAD7DQX6SR7RBLYAV7MD7XRWPNIWHTVVBQZ2ZL3OE2L7LRRIQ, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.468742525 (415h3m6.296450916s) [noise=true , address=45.91.116.4:4939 , piece_id=QYNKMKDLQJV7B6M2E34DU53Y7YI6VBCUW3C7ZLQOI4XWTBXZHKDQ, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.468915034 (415h3m6.296623414s) [noise=true , address=45.91.116.11:4117 , piece_id=WLMO5E3BIFFL3TOAUSSBGORIZ6DOCWFCMVT4AVFPPGPSKP2OKDAA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.469069393 (415h3m6.296777787s) [noise=true , address=145.79.227.32:4835 , piece_id=BRE3JH3XKPXB6FN7WB3W6ED7MOFES6HYLR34YGR6L5AX7RPIQ7IA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.469285789 (415h3m6.296994183s) [noise=true , address=145.79.227.29:4943 , piece_id=G4TMZ6IAIZBDON3UEDPH4LJQFHV3JLALEXSJ3VI4XN6WHIZOB57Q, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.469411962 (415h3m6.297120355s) [noise=true , address=145.79.227.48:4301 , piece_id=LBIYTJNOX5J4M3HDE7FGF5CSARBITW2AK3P2DSYFLFKUOIHZUKOQ, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.469749379 (415h3m6.297457773s) [noise=true , address=145.79.227.37:4565 , piece_id=554RHTYPG27UNCFPSRMOB245AX4QXL2QFIYECNTVWNNJMAL4BTYA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.470080828 (415h3m6.29778922s) [noise=true , address=145.79.227.43:4157 , piece_id=3W3LVTU37DNHWS52IGIIH7SXWXVNPGEY27RQNT272L3Z64GNR4KA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.470477660 (415h3m6.298186058s) [noise=true , address=45.91.116.14:4190 , piece_id=V4T6Z7K3C75LQEXNQIYZGYWW2CSQ7UPSET46QGDRT52RFGAROTTA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.470567977 (415h3m6.29827637s) [noise=true , address=145.79.227.15:4874 , piece_id=FNCCIMIVJTGZ2X5TOYOZH727JXHYM7HG3HBHBTZDAG67SHZLYYYA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.470980950 (415h3m6.298689349s) [noise=true , address=145.79.227.50:4373 , piece_id=4U55T5QDYCE6AZ3NCTPAVECPAELPQUKT3HV7GZL7JFRMWRN2AZUQ, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.471440098 (415h3m6.299148489s) [noise=true , address=145.79.227.47:4266 , piece_id=3NFYV2PWBNTAOSFHENM4J5PZ7XXE72UKTQVXLTVQRHGK4DUXAKDA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.560766475 (415h3m6.388474871s) [upload=431917]: Next returned nil segment Dec 22 22:31:05.560785161 (415h3m6.388493562s) [upload=431917]: Waiting for error group managing segments... Dec 22 22:31:05.566117878 (415h3m6.39382628s) [noise=true , address=45.91.116.4:4939 , piece_id=QYNKMKDLQJV7B6M2E34DU53Y7YI6VBCUW3C7ZLQOI4XWTBXZHKDQ, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.566723306 (415h3m6.394431687s) [noise=true , address=45.91.116.11:4117 , piece_id=WLMO5E3BIFFL3TOAUSSBGORIZ6DOCWFCMVT4AVFPPGPSKP2OKDAA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.566992667 (415h3m6.394701063s) [noise=true , address=145.79.227.36:4519 , piece_id=JUKRRIZWVGKEDD3GPJ7HMY2MEQZNMUCTKTXR6ZLSIERRONTZKHCQ, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.567464626 (415h3m6.395173019s) [noise=true , address=145.79.227.47:4266 , piece_id=3NFYV2PWBNTAOSFHENM4J5PZ7XXE72UKTQVXLTVQRHGK4DUXAKDA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.567924492 (415h3m6.395632883s) [noise=true , address=145.79.227.15:4874 , piece_id=FNCCIMIVJTGZ2X5TOYOZH727JXHYM7HG3HBHBTZDAG67SHZLYYYA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.568807294 (415h3m6.396515685s) [noise=true , address=145.79.227.51:4409 , piece_id=ATH3F4DFOUUIFFJZRA2SSRRL7AKWZMQNQRYOZCVFFDLTFCFP236A, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.570579823 (415h3m6.398288214s) [noise=true , address=145.79.227.48:4301 , piece_id=LBIYTJNOX5J4M3HDE7FGF5CSARBITW2AK3P2DSYFLFKUOIHZUKOQ, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.571632758 (415h3m6.399341149s) [noise=true , address=145.79.227.43:4157 , piece_id=3W3LVTU37DNHWS52IGIIH7SXWXVNPGEY27RQNT272L3Z64GNR4KA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.573168383 (415h3m6.400876774s) [noise=true , address=145.79.227.38:4979 , piece_id=HLMQC5OEB4OL4OZSRMCEVAWR5QM2T4MQUBPTHRVQLEAHPBWPQF6Q, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.579995420 (415h3m6.407703816s) [noise=true , address=45.91.116.14:4190 , piece_id=V4T6Z7K3C75LQEXNQIYZGYWW2CSQ7UPSET46QGDRT52RFGAROTTA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.592093172 (415h3m6.419801563s) [noise=true , address=145.79.227.37:4565 , piece_id=554RHTYPG27UNCFPSRMOB245AX4QXL2QFIYECNTVWNNJMAL4BTYA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:09.946500434 (415h3m10.774208816s) [noise=true , address=145.79.227.40:4051 , piece_id=55RRAD7DQX6SR7RBLYAV7MD7XRWPNIWHTVVBQZ2ZL3OE2L7LRRIQ, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:35.398235543 (415h3m36.225943943s) [noise=true , address=145.79.227.39:4013 , piece_id=JJVURGSLKY2UQ4FKNCLATA3ZBD5SDL32XJAG42KH2U6OYK3KTS6Q, seg_pos={0 0}, upload=431917]: Done uploading piece. err: ecclient: upload failed (node:121scLvKEW9zw2aEz22zpremcjiop67eQ8v2uqpmaD5aZzCAP7b, address:145.79.227.39:4013): protocol: write tcp 159.223.143.39:60842->145.79.227.39:4013: use of closed network connection; write tcp 159.223.143.39:60842->145.79.227.39:4013: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:60842->145.79.227.39:4013: use of closed network connection Dec 22 22:31:35.398562305 (415h3m36.226270699s) [noise=true , address=145.79.227.32:4835 , piece_id=BRE3JH3XKPXB6FN7WB3W6ED7MOFES6HYLR34YGR6L5AX7RPIQ7IA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: ecclient: upload failed (node:16vVfeoqCkbMYCRjMMWaKBnCvQBBJPMgwQtWTq9HKQrD9SS17y, address:145.79.227.32:4835): protocol: write tcp 159.223.143.39:45098->145.79.227.32:4835: use of closed network connection; write tcp 159.223.143.39:45098->145.79.227.32:4835: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:45098->145.79.227.32:4835: use of closed network connection Dec 22 22:31:35.399183028 (415h3m36.226891422s) [noise=true , address=145.79.227.50:4373 , piece_id=4U55T5QDYCE6AZ3NCTPAVECPAELPQUKT3HV7GZL7JFRMWRN2AZUQ, seg_pos={0 0}, upload=431917]: Done uploading piece. err: ecclient: upload failed (node:12AQGSkJuNJDxoeqhChUenWxHwF5yK41UESCF63FQz3XvabXXfz, address:145.79.227.50:4373): protocol: read tcp 159.223.143.39:56028->145.79.227.50:4373: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 22 22:31:35.399278021 (415h3m36.226986417s) [noise=true , address=145.79.227.29:4943 , piece_id=G4TMZ6IAIZBDON3UEDPH4LJQFHV3JLALEXSJ3VI4XN6WHIZOB57Q, seg_pos={0 0}, upload=431917]: Done uploading piece. err: ecclient: upload failed (node:12fc5Trj8XxacZhzxdksUVT5CbG6uTmiwdUPjreYi4UMPY1TXWz, address:145.79.227.29:4943): protocol: write tcp 159.223.143.39:50820->145.79.227.29:4943: use of closed network connection; write tcp 159.223.143.39:50820->145.79.227.29:4943: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:50820->145.79.227.29:4943: use of closed network connection Dec 22 22:31:35.399379485 (415h3m36.227087866s) [seg_pos={0 0}, upload=431917]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 22 22:31:35.421122544 (415h3m36.248830935s) [upload=431917]: Done uploading segments... Dec 22 22:31:35.421174369 (415h3m36.248882769s) [upload=431917]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 440.611292ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................................X....................] duration: 448.465254ms trace: https://jaeger.dev.storj.io/trace/7013ffb1a981d605 FAIL (2025-12-22 22:47:39 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 22 22:31:05.439510308 (415h3m6.2672187s) [upload=431917]: Starting upload Dec 22 22:31:05.439581993 (415h3m6.26729037s) [upload=431917]: Done starting upload Dec 22 22:31:05.439812792 (415h3m6.267521185s) [upload=431917]: Uploading segments... Dec 22 22:31:05.439942524 (415h3m6.267650918s) [upload=431917]: Got next segment. Inline: false Dec 22 22:31:05.440022744 (415h3m6.267731174s) [upload=431917]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 22 22:31:05.440029670 (415h3m6.267738072s) [upload=431917]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 22 22:31:05.467186372 (415h3m6.294894769s) [seg_pos={0 0}, upload=431917]: Begin upload segment... Dec 22 22:31:05.467265149 (415h3m6.294973535s) [seg_pos={0 0}, upload=431917]: Done begin upload segment. Dec 22 22:31:05.467296974 (415h3m6.295005373s) [noise=true , address=145.79.227.51:4409 , piece_id=ATH3F4DFOUUIFFJZRA2SSRRL7AKWZMQNQRYOZCVFFDLTFCFP236A, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.467389853 (415h3m6.295098247s) [noise=true , address=145.79.227.39:4013 , piece_id=JJVURGSLKY2UQ4FKNCLATA3ZBD5SDL32XJAG42KH2U6OYK3KTS6Q, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.467912104 (415h3m6.295620486s) [noise=true , address=145.79.227.36:4519 , piece_id=JUKRRIZWVGKEDD3GPJ7HMY2MEQZNMUCTKTXR6ZLSIERRONTZKHCQ, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.468324863 (415h3m6.296033263s) [noise=true , address=145.79.227.38:4979 , piece_id=HLMQC5OEB4OL4OZSRMCEVAWR5QM2T4MQUBPTHRVQLEAHPBWPQF6Q, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.468334764 (415h3m6.296043147s) [noise=true , address=145.79.227.40:4051 , piece_id=55RRAD7DQX6SR7RBLYAV7MD7XRWPNIWHTVVBQZ2ZL3OE2L7LRRIQ, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.468742525 (415h3m6.296450916s) [noise=true , address=45.91.116.4:4939 , piece_id=QYNKMKDLQJV7B6M2E34DU53Y7YI6VBCUW3C7ZLQOI4XWTBXZHKDQ, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.468915034 (415h3m6.296623414s) [noise=true , address=45.91.116.11:4117 , piece_id=WLMO5E3BIFFL3TOAUSSBGORIZ6DOCWFCMVT4AVFPPGPSKP2OKDAA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.469069393 (415h3m6.296777787s) [noise=true , address=145.79.227.32:4835 , piece_id=BRE3JH3XKPXB6FN7WB3W6ED7MOFES6HYLR34YGR6L5AX7RPIQ7IA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.469285789 (415h3m6.296994183s) [noise=true , address=145.79.227.29:4943 , piece_id=G4TMZ6IAIZBDON3UEDPH4LJQFHV3JLALEXSJ3VI4XN6WHIZOB57Q, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.469411962 (415h3m6.297120355s) [noise=true , address=145.79.227.48:4301 , piece_id=LBIYTJNOX5J4M3HDE7FGF5CSARBITW2AK3P2DSYFLFKUOIHZUKOQ, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.469749379 (415h3m6.297457773s) [noise=true , address=145.79.227.37:4565 , piece_id=554RHTYPG27UNCFPSRMOB245AX4QXL2QFIYECNTVWNNJMAL4BTYA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.470080828 (415h3m6.29778922s) [noise=true , address=145.79.227.43:4157 , piece_id=3W3LVTU37DNHWS52IGIIH7SXWXVNPGEY27RQNT272L3Z64GNR4KA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.470477660 (415h3m6.298186058s) [noise=true , address=45.91.116.14:4190 , piece_id=V4T6Z7K3C75LQEXNQIYZGYWW2CSQ7UPSET46QGDRT52RFGAROTTA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.470567977 (415h3m6.29827637s) [noise=true , address=145.79.227.15:4874 , piece_id=FNCCIMIVJTGZ2X5TOYOZH727JXHYM7HG3HBHBTZDAG67SHZLYYYA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.470980950 (415h3m6.298689349s) [noise=true , address=145.79.227.50:4373 , piece_id=4U55T5QDYCE6AZ3NCTPAVECPAELPQUKT3HV7GZL7JFRMWRN2AZUQ, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.471440098 (415h3m6.299148489s) [noise=true , address=145.79.227.47:4266 , piece_id=3NFYV2PWBNTAOSFHENM4J5PZ7XXE72UKTQVXLTVQRHGK4DUXAKDA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.560766475 (415h3m6.388474871s) [upload=431917]: Next returned nil segment Dec 22 22:31:05.560785161 (415h3m6.388493562s) [upload=431917]: Waiting for error group managing segments... Dec 22 22:31:05.566117878 (415h3m6.39382628s) [noise=true , address=45.91.116.4:4939 , piece_id=QYNKMKDLQJV7B6M2E34DU53Y7YI6VBCUW3C7ZLQOI4XWTBXZHKDQ, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.566723306 (415h3m6.394431687s) [noise=true , address=45.91.116.11:4117 , piece_id=WLMO5E3BIFFL3TOAUSSBGORIZ6DOCWFCMVT4AVFPPGPSKP2OKDAA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.566992667 (415h3m6.394701063s) [noise=true , address=145.79.227.36:4519 , piece_id=JUKRRIZWVGKEDD3GPJ7HMY2MEQZNMUCTKTXR6ZLSIERRONTZKHCQ, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.567464626 (415h3m6.395173019s) [noise=true , address=145.79.227.47:4266 , piece_id=3NFYV2PWBNTAOSFHENM4J5PZ7XXE72UKTQVXLTVQRHGK4DUXAKDA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.567924492 (415h3m6.395632883s) [noise=true , address=145.79.227.15:4874 , piece_id=FNCCIMIVJTGZ2X5TOYOZH727JXHYM7HG3HBHBTZDAG67SHZLYYYA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.568807294 (415h3m6.396515685s) [noise=true , address=145.79.227.51:4409 , piece_id=ATH3F4DFOUUIFFJZRA2SSRRL7AKWZMQNQRYOZCVFFDLTFCFP236A, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.570579823 (415h3m6.398288214s) [noise=true , address=145.79.227.48:4301 , piece_id=LBIYTJNOX5J4M3HDE7FGF5CSARBITW2AK3P2DSYFLFKUOIHZUKOQ, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.571632758 (415h3m6.399341149s) [noise=true , address=145.79.227.43:4157 , piece_id=3W3LVTU37DNHWS52IGIIH7SXWXVNPGEY27RQNT272L3Z64GNR4KA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.573168383 (415h3m6.400876774s) [noise=true , address=145.79.227.38:4979 , piece_id=HLMQC5OEB4OL4OZSRMCEVAWR5QM2T4MQUBPTHRVQLEAHPBWPQF6Q, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.579995420 (415h3m6.407703816s) [noise=true , address=45.91.116.14:4190 , piece_id=V4T6Z7K3C75LQEXNQIYZGYWW2CSQ7UPSET46QGDRT52RFGAROTTA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.592093172 (415h3m6.419801563s) [noise=true , address=145.79.227.37:4565 , piece_id=554RHTYPG27UNCFPSRMOB245AX4QXL2QFIYECNTVWNNJMAL4BTYA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:09.946500434 (415h3m10.774208816s) [noise=true , address=145.79.227.40:4051 , piece_id=55RRAD7DQX6SR7RBLYAV7MD7XRWPNIWHTVVBQZ2ZL3OE2L7LRRIQ, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:35.398235543 (415h3m36.225943943s) [noise=true , address=145.79.227.39:4013 , piece_id=JJVURGSLKY2UQ4FKNCLATA3ZBD5SDL32XJAG42KH2U6OYK3KTS6Q, seg_pos={0 0}, upload=431917]: Done uploading piece. err: ecclient: upload failed (node:121scLvKEW9zw2aEz22zpremcjiop67eQ8v2uqpmaD5aZzCAP7b, address:145.79.227.39:4013): protocol: write tcp 159.223.143.39:60842->145.79.227.39:4013: use of closed network connection; write tcp 159.223.143.39:60842->145.79.227.39:4013: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:60842->145.79.227.39:4013: use of closed network connection Dec 22 22:31:35.398562305 (415h3m36.226270699s) [noise=true , address=145.79.227.32:4835 , piece_id=BRE3JH3XKPXB6FN7WB3W6ED7MOFES6HYLR34YGR6L5AX7RPIQ7IA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: ecclient: upload failed (node:16vVfeoqCkbMYCRjMMWaKBnCvQBBJPMgwQtWTq9HKQrD9SS17y, address:145.79.227.32:4835): protocol: write tcp 159.223.143.39:45098->145.79.227.32:4835: use of closed network connection; write tcp 159.223.143.39:45098->145.79.227.32:4835: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:45098->145.79.227.32:4835: use of closed network connection Dec 22 22:31:35.399183028 (415h3m36.226891422s) [noise=true , address=145.79.227.50:4373 , piece_id=4U55T5QDYCE6AZ3NCTPAVECPAELPQUKT3HV7GZL7JFRMWRN2AZUQ, seg_pos={0 0}, upload=431917]: Done uploading piece. err: ecclient: upload failed (node:12AQGSkJuNJDxoeqhChUenWxHwF5yK41UESCF63FQz3XvabXXfz, address:145.79.227.50:4373): protocol: read tcp 159.223.143.39:56028->145.79.227.50:4373: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 22 22:31:35.399278021 (415h3m36.226986417s) [noise=true , address=145.79.227.29:4943 , piece_id=G4TMZ6IAIZBDON3UEDPH4LJQFHV3JLALEXSJ3VI4XN6WHIZOB57Q, seg_pos={0 0}, upload=431917]: Done uploading piece. err: ecclient: upload failed (node:12fc5Trj8XxacZhzxdksUVT5CbG6uTmiwdUPjreYi4UMPY1TXWz, address:145.79.227.29:4943): protocol: write tcp 159.223.143.39:50820->145.79.227.29:4943: use of closed network connection; write tcp 159.223.143.39:50820->145.79.227.29:4943: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:50820->145.79.227.29:4943: use of closed network connection Dec 22 22:31:35.399379485 (415h3m36.227087866s) [seg_pos={0 0}, upload=431917]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 22 22:31:35.421122544 (415h3m36.248830935s) [upload=431917]: Done uploading segments... Dec 22 22:31:35.421174369 (415h3m36.248882769s) [upload=431917]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 448.496428ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................................X...................] duration: 503.357942ms trace: https://jaeger.dev.storj.io/trace/7013ffb1a981d605 FAIL (2025-12-22 22:46:29 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 22 22:31:05.439510308 (415h3m6.2672187s) [upload=431917]: Starting upload Dec 22 22:31:05.439581993 (415h3m6.26729037s) [upload=431917]: Done starting upload Dec 22 22:31:05.439812792 (415h3m6.267521185s) [upload=431917]: Uploading segments... Dec 22 22:31:05.439942524 (415h3m6.267650918s) [upload=431917]: Got next segment. Inline: false Dec 22 22:31:05.440022744 (415h3m6.267731174s) [upload=431917]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 22 22:31:05.440029670 (415h3m6.267738072s) [upload=431917]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 22 22:31:05.467186372 (415h3m6.294894769s) [seg_pos={0 0}, upload=431917]: Begin upload segment... Dec 22 22:31:05.467265149 (415h3m6.294973535s) [seg_pos={0 0}, upload=431917]: Done begin upload segment. Dec 22 22:31:05.467296974 (415h3m6.295005373s) [noise=true , address=145.79.227.51:4409 , piece_id=ATH3F4DFOUUIFFJZRA2SSRRL7AKWZMQNQRYOZCVFFDLTFCFP236A, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.467389853 (415h3m6.295098247s) [noise=true , address=145.79.227.39:4013 , piece_id=JJVURGSLKY2UQ4FKNCLATA3ZBD5SDL32XJAG42KH2U6OYK3KTS6Q, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.467912104 (415h3m6.295620486s) [noise=true , address=145.79.227.36:4519 , piece_id=JUKRRIZWVGKEDD3GPJ7HMY2MEQZNMUCTKTXR6ZLSIERRONTZKHCQ, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.468324863 (415h3m6.296033263s) [noise=true , address=145.79.227.38:4979 , piece_id=HLMQC5OEB4OL4OZSRMCEVAWR5QM2T4MQUBPTHRVQLEAHPBWPQF6Q, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.468334764 (415h3m6.296043147s) [noise=true , address=145.79.227.40:4051 , piece_id=55RRAD7DQX6SR7RBLYAV7MD7XRWPNIWHTVVBQZ2ZL3OE2L7LRRIQ, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.468742525 (415h3m6.296450916s) [noise=true , address=45.91.116.4:4939 , piece_id=QYNKMKDLQJV7B6M2E34DU53Y7YI6VBCUW3C7ZLQOI4XWTBXZHKDQ, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.468915034 (415h3m6.296623414s) [noise=true , address=45.91.116.11:4117 , piece_id=WLMO5E3BIFFL3TOAUSSBGORIZ6DOCWFCMVT4AVFPPGPSKP2OKDAA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.469069393 (415h3m6.296777787s) [noise=true , address=145.79.227.32:4835 , piece_id=BRE3JH3XKPXB6FN7WB3W6ED7MOFES6HYLR34YGR6L5AX7RPIQ7IA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.469285789 (415h3m6.296994183s) [noise=true , address=145.79.227.29:4943 , piece_id=G4TMZ6IAIZBDON3UEDPH4LJQFHV3JLALEXSJ3VI4XN6WHIZOB57Q, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.469411962 (415h3m6.297120355s) [noise=true , address=145.79.227.48:4301 , piece_id=LBIYTJNOX5J4M3HDE7FGF5CSARBITW2AK3P2DSYFLFKUOIHZUKOQ, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.469749379 (415h3m6.297457773s) [noise=true , address=145.79.227.37:4565 , piece_id=554RHTYPG27UNCFPSRMOB245AX4QXL2QFIYECNTVWNNJMAL4BTYA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.470080828 (415h3m6.29778922s) [noise=true , address=145.79.227.43:4157 , piece_id=3W3LVTU37DNHWS52IGIIH7SXWXVNPGEY27RQNT272L3Z64GNR4KA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.470477660 (415h3m6.298186058s) [noise=true , address=45.91.116.14:4190 , piece_id=V4T6Z7K3C75LQEXNQIYZGYWW2CSQ7UPSET46QGDRT52RFGAROTTA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.470567977 (415h3m6.29827637s) [noise=true , address=145.79.227.15:4874 , piece_id=FNCCIMIVJTGZ2X5TOYOZH727JXHYM7HG3HBHBTZDAG67SHZLYYYA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.470980950 (415h3m6.298689349s) [noise=true , address=145.79.227.50:4373 , piece_id=4U55T5QDYCE6AZ3NCTPAVECPAELPQUKT3HV7GZL7JFRMWRN2AZUQ, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.471440098 (415h3m6.299148489s) [noise=true , address=145.79.227.47:4266 , piece_id=3NFYV2PWBNTAOSFHENM4J5PZ7XXE72UKTQVXLTVQRHGK4DUXAKDA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.560766475 (415h3m6.388474871s) [upload=431917]: Next returned nil segment Dec 22 22:31:05.560785161 (415h3m6.388493562s) [upload=431917]: Waiting for error group managing segments... Dec 22 22:31:05.566117878 (415h3m6.39382628s) [noise=true , address=45.91.116.4:4939 , piece_id=QYNKMKDLQJV7B6M2E34DU53Y7YI6VBCUW3C7ZLQOI4XWTBXZHKDQ, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.566723306 (415h3m6.394431687s) [noise=true , address=45.91.116.11:4117 , piece_id=WLMO5E3BIFFL3TOAUSSBGORIZ6DOCWFCMVT4AVFPPGPSKP2OKDAA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.566992667 (415h3m6.394701063s) [noise=true , address=145.79.227.36:4519 , piece_id=JUKRRIZWVGKEDD3GPJ7HMY2MEQZNMUCTKTXR6ZLSIERRONTZKHCQ, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.567464626 (415h3m6.395173019s) [noise=true , address=145.79.227.47:4266 , piece_id=3NFYV2PWBNTAOSFHENM4J5PZ7XXE72UKTQVXLTVQRHGK4DUXAKDA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.567924492 (415h3m6.395632883s) [noise=true , address=145.79.227.15:4874 , piece_id=FNCCIMIVJTGZ2X5TOYOZH727JXHYM7HG3HBHBTZDAG67SHZLYYYA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.568807294 (415h3m6.396515685s) [noise=true , address=145.79.227.51:4409 , piece_id=ATH3F4DFOUUIFFJZRA2SSRRL7AKWZMQNQRYOZCVFFDLTFCFP236A, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.570579823 (415h3m6.398288214s) [noise=true , address=145.79.227.48:4301 , piece_id=LBIYTJNOX5J4M3HDE7FGF5CSARBITW2AK3P2DSYFLFKUOIHZUKOQ, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.571632758 (415h3m6.399341149s) [noise=true , address=145.79.227.43:4157 , piece_id=3W3LVTU37DNHWS52IGIIH7SXWXVNPGEY27RQNT272L3Z64GNR4KA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.573168383 (415h3m6.400876774s) [noise=true , address=145.79.227.38:4979 , piece_id=HLMQC5OEB4OL4OZSRMCEVAWR5QM2T4MQUBPTHRVQLEAHPBWPQF6Q, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.579995420 (415h3m6.407703816s) [noise=true , address=45.91.116.14:4190 , piece_id=V4T6Z7K3C75LQEXNQIYZGYWW2CSQ7UPSET46QGDRT52RFGAROTTA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.592093172 (415h3m6.419801563s) [noise=true , address=145.79.227.37:4565 , piece_id=554RHTYPG27UNCFPSRMOB245AX4QXL2QFIYECNTVWNNJMAL4BTYA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:09.946500434 (415h3m10.774208816s) [noise=true , address=145.79.227.40:4051 , piece_id=55RRAD7DQX6SR7RBLYAV7MD7XRWPNIWHTVVBQZ2ZL3OE2L7LRRIQ, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:35.398235543 (415h3m36.225943943s) [noise=true , address=145.79.227.39:4013 , piece_id=JJVURGSLKY2UQ4FKNCLATA3ZBD5SDL32XJAG42KH2U6OYK3KTS6Q, seg_pos={0 0}, upload=431917]: Done uploading piece. err: ecclient: upload failed (node:121scLvKEW9zw2aEz22zpremcjiop67eQ8v2uqpmaD5aZzCAP7b, address:145.79.227.39:4013): protocol: write tcp 159.223.143.39:60842->145.79.227.39:4013: use of closed network connection; write tcp 159.223.143.39:60842->145.79.227.39:4013: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:60842->145.79.227.39:4013: use of closed network connection Dec 22 22:31:35.398562305 (415h3m36.226270699s) [noise=true , address=145.79.227.32:4835 , piece_id=BRE3JH3XKPXB6FN7WB3W6ED7MOFES6HYLR34YGR6L5AX7RPIQ7IA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: ecclient: upload failed (node:16vVfeoqCkbMYCRjMMWaKBnCvQBBJPMgwQtWTq9HKQrD9SS17y, address:145.79.227.32:4835): protocol: write tcp 159.223.143.39:45098->145.79.227.32:4835: use of closed network connection; write tcp 159.223.143.39:45098->145.79.227.32:4835: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:45098->145.79.227.32:4835: use of closed network connection Dec 22 22:31:35.399183028 (415h3m36.226891422s) [noise=true , address=145.79.227.50:4373 , piece_id=4U55T5QDYCE6AZ3NCTPAVECPAELPQUKT3HV7GZL7JFRMWRN2AZUQ, seg_pos={0 0}, upload=431917]: Done uploading piece. err: ecclient: upload failed (node:12AQGSkJuNJDxoeqhChUenWxHwF5yK41UESCF63FQz3XvabXXfz, address:145.79.227.50:4373): protocol: read tcp 159.223.143.39:56028->145.79.227.50:4373: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 22 22:31:35.399278021 (415h3m36.226986417s) [noise=true , address=145.79.227.29:4943 , piece_id=G4TMZ6IAIZBDON3UEDPH4LJQFHV3JLALEXSJ3VI4XN6WHIZOB57Q, seg_pos={0 0}, upload=431917]: Done uploading piece. err: ecclient: upload failed (node:12fc5Trj8XxacZhzxdksUVT5CbG6uTmiwdUPjreYi4UMPY1TXWz, address:145.79.227.29:4943): protocol: write tcp 159.223.143.39:50820->145.79.227.29:4943: use of closed network connection; write tcp 159.223.143.39:50820->145.79.227.29:4943: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:50820->145.79.227.29:4943: use of closed network connection Dec 22 22:31:35.399379485 (415h3m36.227087866s) [seg_pos={0 0}, upload=431917]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 22 22:31:35.421122544 (415h3m36.248830935s) [upload=431917]: Done uploading segments... Dec 22 22:31:35.421174369 (415h3m36.248882769s) [upload=431917]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 503.383881ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................X..................] duration: 506.519912ms trace: https://jaeger.dev.storj.io/trace/7013ffb1a981d605 FAIL (2025-12-22 22:45:42 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 22 22:31:05.439510308 (415h3m6.2672187s) [upload=431917]: Starting upload Dec 22 22:31:05.439581993 (415h3m6.26729037s) [upload=431917]: Done starting upload Dec 22 22:31:05.439812792 (415h3m6.267521185s) [upload=431917]: Uploading segments... Dec 22 22:31:05.439942524 (415h3m6.267650918s) [upload=431917]: Got next segment. Inline: false Dec 22 22:31:05.440022744 (415h3m6.267731174s) [upload=431917]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 22 22:31:05.440029670 (415h3m6.267738072s) [upload=431917]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 22 22:31:05.467186372 (415h3m6.294894769s) [seg_pos={0 0}, upload=431917]: Begin upload segment... Dec 22 22:31:05.467265149 (415h3m6.294973535s) [seg_pos={0 0}, upload=431917]: Done begin upload segment. Dec 22 22:31:05.467296974 (415h3m6.295005373s) [noise=true , address=145.79.227.51:4409 , piece_id=ATH3F4DFOUUIFFJZRA2SSRRL7AKWZMQNQRYOZCVFFDLTFCFP236A, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.467389853 (415h3m6.295098247s) [noise=true , address=145.79.227.39:4013 , piece_id=JJVURGSLKY2UQ4FKNCLATA3ZBD5SDL32XJAG42KH2U6OYK3KTS6Q, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.467912104 (415h3m6.295620486s) [noise=true , address=145.79.227.36:4519 , piece_id=JUKRRIZWVGKEDD3GPJ7HMY2MEQZNMUCTKTXR6ZLSIERRONTZKHCQ, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.468324863 (415h3m6.296033263s) [noise=true , address=145.79.227.38:4979 , piece_id=HLMQC5OEB4OL4OZSRMCEVAWR5QM2T4MQUBPTHRVQLEAHPBWPQF6Q, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.468334764 (415h3m6.296043147s) [noise=true , address=145.79.227.40:4051 , piece_id=55RRAD7DQX6SR7RBLYAV7MD7XRWPNIWHTVVBQZ2ZL3OE2L7LRRIQ, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.468742525 (415h3m6.296450916s) [noise=true , address=45.91.116.4:4939 , piece_id=QYNKMKDLQJV7B6M2E34DU53Y7YI6VBCUW3C7ZLQOI4XWTBXZHKDQ, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.468915034 (415h3m6.296623414s) [noise=true , address=45.91.116.11:4117 , piece_id=WLMO5E3BIFFL3TOAUSSBGORIZ6DOCWFCMVT4AVFPPGPSKP2OKDAA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.469069393 (415h3m6.296777787s) [noise=true , address=145.79.227.32:4835 , piece_id=BRE3JH3XKPXB6FN7WB3W6ED7MOFES6HYLR34YGR6L5AX7RPIQ7IA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.469285789 (415h3m6.296994183s) [noise=true , address=145.79.227.29:4943 , piece_id=G4TMZ6IAIZBDON3UEDPH4LJQFHV3JLALEXSJ3VI4XN6WHIZOB57Q, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.469411962 (415h3m6.297120355s) [noise=true , address=145.79.227.48:4301 , piece_id=LBIYTJNOX5J4M3HDE7FGF5CSARBITW2AK3P2DSYFLFKUOIHZUKOQ, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.469749379 (415h3m6.297457773s) [noise=true , address=145.79.227.37:4565 , piece_id=554RHTYPG27UNCFPSRMOB245AX4QXL2QFIYECNTVWNNJMAL4BTYA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.470080828 (415h3m6.29778922s) [noise=true , address=145.79.227.43:4157 , piece_id=3W3LVTU37DNHWS52IGIIH7SXWXVNPGEY27RQNT272L3Z64GNR4KA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.470477660 (415h3m6.298186058s) [noise=true , address=45.91.116.14:4190 , piece_id=V4T6Z7K3C75LQEXNQIYZGYWW2CSQ7UPSET46QGDRT52RFGAROTTA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.470567977 (415h3m6.29827637s) [noise=true , address=145.79.227.15:4874 , piece_id=FNCCIMIVJTGZ2X5TOYOZH727JXHYM7HG3HBHBTZDAG67SHZLYYYA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.470980950 (415h3m6.298689349s) [noise=true , address=145.79.227.50:4373 , piece_id=4U55T5QDYCE6AZ3NCTPAVECPAELPQUKT3HV7GZL7JFRMWRN2AZUQ, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.471440098 (415h3m6.299148489s) [noise=true , address=145.79.227.47:4266 , piece_id=3NFYV2PWBNTAOSFHENM4J5PZ7XXE72UKTQVXLTVQRHGK4DUXAKDA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.560766475 (415h3m6.388474871s) [upload=431917]: Next returned nil segment Dec 22 22:31:05.560785161 (415h3m6.388493562s) [upload=431917]: Waiting for error group managing segments... Dec 22 22:31:05.566117878 (415h3m6.39382628s) [noise=true , address=45.91.116.4:4939 , piece_id=QYNKMKDLQJV7B6M2E34DU53Y7YI6VBCUW3C7ZLQOI4XWTBXZHKDQ, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.566723306 (415h3m6.394431687s) [noise=true , address=45.91.116.11:4117 , piece_id=WLMO5E3BIFFL3TOAUSSBGORIZ6DOCWFCMVT4AVFPPGPSKP2OKDAA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.566992667 (415h3m6.394701063s) [noise=true , address=145.79.227.36:4519 , piece_id=JUKRRIZWVGKEDD3GPJ7HMY2MEQZNMUCTKTXR6ZLSIERRONTZKHCQ, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.567464626 (415h3m6.395173019s) [noise=true , address=145.79.227.47:4266 , piece_id=3NFYV2PWBNTAOSFHENM4J5PZ7XXE72UKTQVXLTVQRHGK4DUXAKDA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.567924492 (415h3m6.395632883s) [noise=true , address=145.79.227.15:4874 , piece_id=FNCCIMIVJTGZ2X5TOYOZH727JXHYM7HG3HBHBTZDAG67SHZLYYYA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.568807294 (415h3m6.396515685s) [noise=true , address=145.79.227.51:4409 , piece_id=ATH3F4DFOUUIFFJZRA2SSRRL7AKWZMQNQRYOZCVFFDLTFCFP236A, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.570579823 (415h3m6.398288214s) [noise=true , address=145.79.227.48:4301 , piece_id=LBIYTJNOX5J4M3HDE7FGF5CSARBITW2AK3P2DSYFLFKUOIHZUKOQ, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.571632758 (415h3m6.399341149s) [noise=true , address=145.79.227.43:4157 , piece_id=3W3LVTU37DNHWS52IGIIH7SXWXVNPGEY27RQNT272L3Z64GNR4KA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.573168383 (415h3m6.400876774s) [noise=true , address=145.79.227.38:4979 , piece_id=HLMQC5OEB4OL4OZSRMCEVAWR5QM2T4MQUBPTHRVQLEAHPBWPQF6Q, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.579995420 (415h3m6.407703816s) [noise=true , address=45.91.116.14:4190 , piece_id=V4T6Z7K3C75LQEXNQIYZGYWW2CSQ7UPSET46QGDRT52RFGAROTTA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.592093172 (415h3m6.419801563s) [noise=true , address=145.79.227.37:4565 , piece_id=554RHTYPG27UNCFPSRMOB245AX4QXL2QFIYECNTVWNNJMAL4BTYA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:09.946500434 (415h3m10.774208816s) [noise=true , address=145.79.227.40:4051 , piece_id=55RRAD7DQX6SR7RBLYAV7MD7XRWPNIWHTVVBQZ2ZL3OE2L7LRRIQ, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:35.398235543 (415h3m36.225943943s) [noise=true , address=145.79.227.39:4013 , piece_id=JJVURGSLKY2UQ4FKNCLATA3ZBD5SDL32XJAG42KH2U6OYK3KTS6Q, seg_pos={0 0}, upload=431917]: Done uploading piece. err: ecclient: upload failed (node:121scLvKEW9zw2aEz22zpremcjiop67eQ8v2uqpmaD5aZzCAP7b, address:145.79.227.39:4013): protocol: write tcp 159.223.143.39:60842->145.79.227.39:4013: use of closed network connection; write tcp 159.223.143.39:60842->145.79.227.39:4013: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:60842->145.79.227.39:4013: use of closed network connection Dec 22 22:31:35.398562305 (415h3m36.226270699s) [noise=true , address=145.79.227.32:4835 , piece_id=BRE3JH3XKPXB6FN7WB3W6ED7MOFES6HYLR34YGR6L5AX7RPIQ7IA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: ecclient: upload failed (node:16vVfeoqCkbMYCRjMMWaKBnCvQBBJPMgwQtWTq9HKQrD9SS17y, address:145.79.227.32:4835): protocol: write tcp 159.223.143.39:45098->145.79.227.32:4835: use of closed network connection; write tcp 159.223.143.39:45098->145.79.227.32:4835: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:45098->145.79.227.32:4835: use of closed network connection Dec 22 22:31:35.399183028 (415h3m36.226891422s) [noise=true , address=145.79.227.50:4373 , piece_id=4U55T5QDYCE6AZ3NCTPAVECPAELPQUKT3HV7GZL7JFRMWRN2AZUQ, seg_pos={0 0}, upload=431917]: Done uploading piece. err: ecclient: upload failed (node:12AQGSkJuNJDxoeqhChUenWxHwF5yK41UESCF63FQz3XvabXXfz, address:145.79.227.50:4373): protocol: read tcp 159.223.143.39:56028->145.79.227.50:4373: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 22 22:31:35.399278021 (415h3m36.226986417s) [noise=true , address=145.79.227.29:4943 , piece_id=G4TMZ6IAIZBDON3UEDPH4LJQFHV3JLALEXSJ3VI4XN6WHIZOB57Q, seg_pos={0 0}, upload=431917]: Done uploading piece. err: ecclient: upload failed (node:12fc5Trj8XxacZhzxdksUVT5CbG6uTmiwdUPjreYi4UMPY1TXWz, address:145.79.227.29:4943): protocol: write tcp 159.223.143.39:50820->145.79.227.29:4943: use of closed network connection; write tcp 159.223.143.39:50820->145.79.227.29:4943: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:50820->145.79.227.29:4943: use of closed network connection Dec 22 22:31:35.399379485 (415h3m36.227087866s) [seg_pos={0 0}, upload=431917]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 22 22:31:35.421122544 (415h3m36.248830935s) [upload=431917]: Done uploading segments... Dec 22 22:31:35.421174369 (415h3m36.248882769s) [upload=431917]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 506.539806ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................X.................] duration: 515.745284ms trace: https://jaeger.dev.storj.io/trace/7013ffb1a981d605 FAIL (2025-12-22 22:45:08 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 22 22:31:05.439510308 (415h3m6.2672187s) [upload=431917]: Starting upload Dec 22 22:31:05.439581993 (415h3m6.26729037s) [upload=431917]: Done starting upload Dec 22 22:31:05.439812792 (415h3m6.267521185s) [upload=431917]: Uploading segments... Dec 22 22:31:05.439942524 (415h3m6.267650918s) [upload=431917]: Got next segment. Inline: false Dec 22 22:31:05.440022744 (415h3m6.267731174s) [upload=431917]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 22 22:31:05.440029670 (415h3m6.267738072s) [upload=431917]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 22 22:31:05.467186372 (415h3m6.294894769s) [seg_pos={0 0}, upload=431917]: Begin upload segment... Dec 22 22:31:05.467265149 (415h3m6.294973535s) [seg_pos={0 0}, upload=431917]: Done begin upload segment. Dec 22 22:31:05.467296974 (415h3m6.295005373s) [noise=true , address=145.79.227.51:4409 , piece_id=ATH3F4DFOUUIFFJZRA2SSRRL7AKWZMQNQRYOZCVFFDLTFCFP236A, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.467389853 (415h3m6.295098247s) [noise=true , address=145.79.227.39:4013 , piece_id=JJVURGSLKY2UQ4FKNCLATA3ZBD5SDL32XJAG42KH2U6OYK3KTS6Q, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.467912104 (415h3m6.295620486s) [noise=true , address=145.79.227.36:4519 , piece_id=JUKRRIZWVGKEDD3GPJ7HMY2MEQZNMUCTKTXR6ZLSIERRONTZKHCQ, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.468324863 (415h3m6.296033263s) [noise=true , address=145.79.227.38:4979 , piece_id=HLMQC5OEB4OL4OZSRMCEVAWR5QM2T4MQUBPTHRVQLEAHPBWPQF6Q, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.468334764 (415h3m6.296043147s) [noise=true , address=145.79.227.40:4051 , piece_id=55RRAD7DQX6SR7RBLYAV7MD7XRWPNIWHTVVBQZ2ZL3OE2L7LRRIQ, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.468742525 (415h3m6.296450916s) [noise=true , address=45.91.116.4:4939 , piece_id=QYNKMKDLQJV7B6M2E34DU53Y7YI6VBCUW3C7ZLQOI4XWTBXZHKDQ, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.468915034 (415h3m6.296623414s) [noise=true , address=45.91.116.11:4117 , piece_id=WLMO5E3BIFFL3TOAUSSBGORIZ6DOCWFCMVT4AVFPPGPSKP2OKDAA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.469069393 (415h3m6.296777787s) [noise=true , address=145.79.227.32:4835 , piece_id=BRE3JH3XKPXB6FN7WB3W6ED7MOFES6HYLR34YGR6L5AX7RPIQ7IA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.469285789 (415h3m6.296994183s) [noise=true , address=145.79.227.29:4943 , piece_id=G4TMZ6IAIZBDON3UEDPH4LJQFHV3JLALEXSJ3VI4XN6WHIZOB57Q, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.469411962 (415h3m6.297120355s) [noise=true , address=145.79.227.48:4301 , piece_id=LBIYTJNOX5J4M3HDE7FGF5CSARBITW2AK3P2DSYFLFKUOIHZUKOQ, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.469749379 (415h3m6.297457773s) [noise=true , address=145.79.227.37:4565 , piece_id=554RHTYPG27UNCFPSRMOB245AX4QXL2QFIYECNTVWNNJMAL4BTYA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.470080828 (415h3m6.29778922s) [noise=true , address=145.79.227.43:4157 , piece_id=3W3LVTU37DNHWS52IGIIH7SXWXVNPGEY27RQNT272L3Z64GNR4KA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.470477660 (415h3m6.298186058s) [noise=true , address=45.91.116.14:4190 , piece_id=V4T6Z7K3C75LQEXNQIYZGYWW2CSQ7UPSET46QGDRT52RFGAROTTA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.470567977 (415h3m6.29827637s) [noise=true , address=145.79.227.15:4874 , piece_id=FNCCIMIVJTGZ2X5TOYOZH727JXHYM7HG3HBHBTZDAG67SHZLYYYA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.470980950 (415h3m6.298689349s) [noise=true , address=145.79.227.50:4373 , piece_id=4U55T5QDYCE6AZ3NCTPAVECPAELPQUKT3HV7GZL7JFRMWRN2AZUQ, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.471440098 (415h3m6.299148489s) [noise=true , address=145.79.227.47:4266 , piece_id=3NFYV2PWBNTAOSFHENM4J5PZ7XXE72UKTQVXLTVQRHGK4DUXAKDA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.560766475 (415h3m6.388474871s) [upload=431917]: Next returned nil segment Dec 22 22:31:05.560785161 (415h3m6.388493562s) [upload=431917]: Waiting for error group managing segments... Dec 22 22:31:05.566117878 (415h3m6.39382628s) [noise=true , address=45.91.116.4:4939 , piece_id=QYNKMKDLQJV7B6M2E34DU53Y7YI6VBCUW3C7ZLQOI4XWTBXZHKDQ, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.566723306 (415h3m6.394431687s) [noise=true , address=45.91.116.11:4117 , piece_id=WLMO5E3BIFFL3TOAUSSBGORIZ6DOCWFCMVT4AVFPPGPSKP2OKDAA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.566992667 (415h3m6.394701063s) [noise=true , address=145.79.227.36:4519 , piece_id=JUKRRIZWVGKEDD3GPJ7HMY2MEQZNMUCTKTXR6ZLSIERRONTZKHCQ, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.567464626 (415h3m6.395173019s) [noise=true , address=145.79.227.47:4266 , piece_id=3NFYV2PWBNTAOSFHENM4J5PZ7XXE72UKTQVXLTVQRHGK4DUXAKDA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.567924492 (415h3m6.395632883s) [noise=true , address=145.79.227.15:4874 , piece_id=FNCCIMIVJTGZ2X5TOYOZH727JXHYM7HG3HBHBTZDAG67SHZLYYYA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.568807294 (415h3m6.396515685s) [noise=true , address=145.79.227.51:4409 , piece_id=ATH3F4DFOUUIFFJZRA2SSRRL7AKWZMQNQRYOZCVFFDLTFCFP236A, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.570579823 (415h3m6.398288214s) [noise=true , address=145.79.227.48:4301 , piece_id=LBIYTJNOX5J4M3HDE7FGF5CSARBITW2AK3P2DSYFLFKUOIHZUKOQ, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.571632758 (415h3m6.399341149s) [noise=true , address=145.79.227.43:4157 , piece_id=3W3LVTU37DNHWS52IGIIH7SXWXVNPGEY27RQNT272L3Z64GNR4KA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.573168383 (415h3m6.400876774s) [noise=true , address=145.79.227.38:4979 , piece_id=HLMQC5OEB4OL4OZSRMCEVAWR5QM2T4MQUBPTHRVQLEAHPBWPQF6Q, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.579995420 (415h3m6.407703816s) [noise=true , address=45.91.116.14:4190 , piece_id=V4T6Z7K3C75LQEXNQIYZGYWW2CSQ7UPSET46QGDRT52RFGAROTTA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.592093172 (415h3m6.419801563s) [noise=true , address=145.79.227.37:4565 , piece_id=554RHTYPG27UNCFPSRMOB245AX4QXL2QFIYECNTVWNNJMAL4BTYA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:09.946500434 (415h3m10.774208816s) [noise=true , address=145.79.227.40:4051 , piece_id=55RRAD7DQX6SR7RBLYAV7MD7XRWPNIWHTVVBQZ2ZL3OE2L7LRRIQ, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:35.398235543 (415h3m36.225943943s) [noise=true , address=145.79.227.39:4013 , piece_id=JJVURGSLKY2UQ4FKNCLATA3ZBD5SDL32XJAG42KH2U6OYK3KTS6Q, seg_pos={0 0}, upload=431917]: Done uploading piece. err: ecclient: upload failed (node:121scLvKEW9zw2aEz22zpremcjiop67eQ8v2uqpmaD5aZzCAP7b, address:145.79.227.39:4013): protocol: write tcp 159.223.143.39:60842->145.79.227.39:4013: use of closed network connection; write tcp 159.223.143.39:60842->145.79.227.39:4013: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:60842->145.79.227.39:4013: use of closed network connection Dec 22 22:31:35.398562305 (415h3m36.226270699s) [noise=true , address=145.79.227.32:4835 , piece_id=BRE3JH3XKPXB6FN7WB3W6ED7MOFES6HYLR34YGR6L5AX7RPIQ7IA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: ecclient: upload failed (node:16vVfeoqCkbMYCRjMMWaKBnCvQBBJPMgwQtWTq9HKQrD9SS17y, address:145.79.227.32:4835): protocol: write tcp 159.223.143.39:45098->145.79.227.32:4835: use of closed network connection; write tcp 159.223.143.39:45098->145.79.227.32:4835: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:45098->145.79.227.32:4835: use of closed network connection Dec 22 22:31:35.399183028 (415h3m36.226891422s) [noise=true , address=145.79.227.50:4373 , piece_id=4U55T5QDYCE6AZ3NCTPAVECPAELPQUKT3HV7GZL7JFRMWRN2AZUQ, seg_pos={0 0}, upload=431917]: Done uploading piece. err: ecclient: upload failed (node:12AQGSkJuNJDxoeqhChUenWxHwF5yK41UESCF63FQz3XvabXXfz, address:145.79.227.50:4373): protocol: read tcp 159.223.143.39:56028->145.79.227.50:4373: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 22 22:31:35.399278021 (415h3m36.226986417s) [noise=true , address=145.79.227.29:4943 , piece_id=G4TMZ6IAIZBDON3UEDPH4LJQFHV3JLALEXSJ3VI4XN6WHIZOB57Q, seg_pos={0 0}, upload=431917]: Done uploading piece. err: ecclient: upload failed (node:12fc5Trj8XxacZhzxdksUVT5CbG6uTmiwdUPjreYi4UMPY1TXWz, address:145.79.227.29:4943): protocol: write tcp 159.223.143.39:50820->145.79.227.29:4943: use of closed network connection; write tcp 159.223.143.39:50820->145.79.227.29:4943: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:50820->145.79.227.29:4943: use of closed network connection Dec 22 22:31:35.399379485 (415h3m36.227087866s) [seg_pos={0 0}, upload=431917]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 22 22:31:35.421122544 (415h3m36.248830935s) [upload=431917]: Done uploading segments... Dec 22 22:31:35.421174369 (415h3m36.248882769s) [upload=431917]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 515.769683ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................X................] duration: 407.281146ms trace: https://jaeger.dev.storj.io/trace/7013ffb1a981d605 FAIL (2025-12-22 22:44:30 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 22 22:31:05.439510308 (415h3m6.2672187s) [upload=431917]: Starting upload Dec 22 22:31:05.439581993 (415h3m6.26729037s) [upload=431917]: Done starting upload Dec 22 22:31:05.439812792 (415h3m6.267521185s) [upload=431917]: Uploading segments... Dec 22 22:31:05.439942524 (415h3m6.267650918s) [upload=431917]: Got next segment. Inline: false Dec 22 22:31:05.440022744 (415h3m6.267731174s) [upload=431917]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 22 22:31:05.440029670 (415h3m6.267738072s) [upload=431917]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 22 22:31:05.467186372 (415h3m6.294894769s) [seg_pos={0 0}, upload=431917]: Begin upload segment... Dec 22 22:31:05.467265149 (415h3m6.294973535s) [seg_pos={0 0}, upload=431917]: Done begin upload segment. Dec 22 22:31:05.467296974 (415h3m6.295005373s) [noise=true , address=145.79.227.51:4409 , piece_id=ATH3F4DFOUUIFFJZRA2SSRRL7AKWZMQNQRYOZCVFFDLTFCFP236A, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.467389853 (415h3m6.295098247s) [noise=true , address=145.79.227.39:4013 , piece_id=JJVURGSLKY2UQ4FKNCLATA3ZBD5SDL32XJAG42KH2U6OYK3KTS6Q, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.467912104 (415h3m6.295620486s) [noise=true , address=145.79.227.36:4519 , piece_id=JUKRRIZWVGKEDD3GPJ7HMY2MEQZNMUCTKTXR6ZLSIERRONTZKHCQ, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.468324863 (415h3m6.296033263s) [noise=true , address=145.79.227.38:4979 , piece_id=HLMQC5OEB4OL4OZSRMCEVAWR5QM2T4MQUBPTHRVQLEAHPBWPQF6Q, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.468334764 (415h3m6.296043147s) [noise=true , address=145.79.227.40:4051 , piece_id=55RRAD7DQX6SR7RBLYAV7MD7XRWPNIWHTVVBQZ2ZL3OE2L7LRRIQ, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.468742525 (415h3m6.296450916s) [noise=true , address=45.91.116.4:4939 , piece_id=QYNKMKDLQJV7B6M2E34DU53Y7YI6VBCUW3C7ZLQOI4XWTBXZHKDQ, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.468915034 (415h3m6.296623414s) [noise=true , address=45.91.116.11:4117 , piece_id=WLMO5E3BIFFL3TOAUSSBGORIZ6DOCWFCMVT4AVFPPGPSKP2OKDAA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.469069393 (415h3m6.296777787s) [noise=true , address=145.79.227.32:4835 , piece_id=BRE3JH3XKPXB6FN7WB3W6ED7MOFES6HYLR34YGR6L5AX7RPIQ7IA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.469285789 (415h3m6.296994183s) [noise=true , address=145.79.227.29:4943 , piece_id=G4TMZ6IAIZBDON3UEDPH4LJQFHV3JLALEXSJ3VI4XN6WHIZOB57Q, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.469411962 (415h3m6.297120355s) [noise=true , address=145.79.227.48:4301 , piece_id=LBIYTJNOX5J4M3HDE7FGF5CSARBITW2AK3P2DSYFLFKUOIHZUKOQ, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.469749379 (415h3m6.297457773s) [noise=true , address=145.79.227.37:4565 , piece_id=554RHTYPG27UNCFPSRMOB245AX4QXL2QFIYECNTVWNNJMAL4BTYA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.470080828 (415h3m6.29778922s) [noise=true , address=145.79.227.43:4157 , piece_id=3W3LVTU37DNHWS52IGIIH7SXWXVNPGEY27RQNT272L3Z64GNR4KA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.470477660 (415h3m6.298186058s) [noise=true , address=45.91.116.14:4190 , piece_id=V4T6Z7K3C75LQEXNQIYZGYWW2CSQ7UPSET46QGDRT52RFGAROTTA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.470567977 (415h3m6.29827637s) [noise=true , address=145.79.227.15:4874 , piece_id=FNCCIMIVJTGZ2X5TOYOZH727JXHYM7HG3HBHBTZDAG67SHZLYYYA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.470980950 (415h3m6.298689349s) [noise=true , address=145.79.227.50:4373 , piece_id=4U55T5QDYCE6AZ3NCTPAVECPAELPQUKT3HV7GZL7JFRMWRN2AZUQ, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.471440098 (415h3m6.299148489s) [noise=true , address=145.79.227.47:4266 , piece_id=3NFYV2PWBNTAOSFHENM4J5PZ7XXE72UKTQVXLTVQRHGK4DUXAKDA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.560766475 (415h3m6.388474871s) [upload=431917]: Next returned nil segment Dec 22 22:31:05.560785161 (415h3m6.388493562s) [upload=431917]: Waiting for error group managing segments... Dec 22 22:31:05.566117878 (415h3m6.39382628s) [noise=true , address=45.91.116.4:4939 , piece_id=QYNKMKDLQJV7B6M2E34DU53Y7YI6VBCUW3C7ZLQOI4XWTBXZHKDQ, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.566723306 (415h3m6.394431687s) [noise=true , address=45.91.116.11:4117 , piece_id=WLMO5E3BIFFL3TOAUSSBGORIZ6DOCWFCMVT4AVFPPGPSKP2OKDAA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.566992667 (415h3m6.394701063s) [noise=true , address=145.79.227.36:4519 , piece_id=JUKRRIZWVGKEDD3GPJ7HMY2MEQZNMUCTKTXR6ZLSIERRONTZKHCQ, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.567464626 (415h3m6.395173019s) [noise=true , address=145.79.227.47:4266 , piece_id=3NFYV2PWBNTAOSFHENM4J5PZ7XXE72UKTQVXLTVQRHGK4DUXAKDA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.567924492 (415h3m6.395632883s) [noise=true , address=145.79.227.15:4874 , piece_id=FNCCIMIVJTGZ2X5TOYOZH727JXHYM7HG3HBHBTZDAG67SHZLYYYA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.568807294 (415h3m6.396515685s) [noise=true , address=145.79.227.51:4409 , piece_id=ATH3F4DFOUUIFFJZRA2SSRRL7AKWZMQNQRYOZCVFFDLTFCFP236A, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.570579823 (415h3m6.398288214s) [noise=true , address=145.79.227.48:4301 , piece_id=LBIYTJNOX5J4M3HDE7FGF5CSARBITW2AK3P2DSYFLFKUOIHZUKOQ, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.571632758 (415h3m6.399341149s) [noise=true , address=145.79.227.43:4157 , piece_id=3W3LVTU37DNHWS52IGIIH7SXWXVNPGEY27RQNT272L3Z64GNR4KA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.573168383 (415h3m6.400876774s) [noise=true , address=145.79.227.38:4979 , piece_id=HLMQC5OEB4OL4OZSRMCEVAWR5QM2T4MQUBPTHRVQLEAHPBWPQF6Q, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.579995420 (415h3m6.407703816s) [noise=true , address=45.91.116.14:4190 , piece_id=V4T6Z7K3C75LQEXNQIYZGYWW2CSQ7UPSET46QGDRT52RFGAROTTA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.592093172 (415h3m6.419801563s) [noise=true , address=145.79.227.37:4565 , piece_id=554RHTYPG27UNCFPSRMOB245AX4QXL2QFIYECNTVWNNJMAL4BTYA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:09.946500434 (415h3m10.774208816s) [noise=true , address=145.79.227.40:4051 , piece_id=55RRAD7DQX6SR7RBLYAV7MD7XRWPNIWHTVVBQZ2ZL3OE2L7LRRIQ, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:35.398235543 (415h3m36.225943943s) [noise=true , address=145.79.227.39:4013 , piece_id=JJVURGSLKY2UQ4FKNCLATA3ZBD5SDL32XJAG42KH2U6OYK3KTS6Q, seg_pos={0 0}, upload=431917]: Done uploading piece. err: ecclient: upload failed (node:121scLvKEW9zw2aEz22zpremcjiop67eQ8v2uqpmaD5aZzCAP7b, address:145.79.227.39:4013): protocol: write tcp 159.223.143.39:60842->145.79.227.39:4013: use of closed network connection; write tcp 159.223.143.39:60842->145.79.227.39:4013: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:60842->145.79.227.39:4013: use of closed network connection Dec 22 22:31:35.398562305 (415h3m36.226270699s) [noise=true , address=145.79.227.32:4835 , piece_id=BRE3JH3XKPXB6FN7WB3W6ED7MOFES6HYLR34YGR6L5AX7RPIQ7IA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: ecclient: upload failed (node:16vVfeoqCkbMYCRjMMWaKBnCvQBBJPMgwQtWTq9HKQrD9SS17y, address:145.79.227.32:4835): protocol: write tcp 159.223.143.39:45098->145.79.227.32:4835: use of closed network connection; write tcp 159.223.143.39:45098->145.79.227.32:4835: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:45098->145.79.227.32:4835: use of closed network connection Dec 22 22:31:35.399183028 (415h3m36.226891422s) [noise=true , address=145.79.227.50:4373 , piece_id=4U55T5QDYCE6AZ3NCTPAVECPAELPQUKT3HV7GZL7JFRMWRN2AZUQ, seg_pos={0 0}, upload=431917]: Done uploading piece. err: ecclient: upload failed (node:12AQGSkJuNJDxoeqhChUenWxHwF5yK41UESCF63FQz3XvabXXfz, address:145.79.227.50:4373): protocol: read tcp 159.223.143.39:56028->145.79.227.50:4373: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 22 22:31:35.399278021 (415h3m36.226986417s) [noise=true , address=145.79.227.29:4943 , piece_id=G4TMZ6IAIZBDON3UEDPH4LJQFHV3JLALEXSJ3VI4XN6WHIZOB57Q, seg_pos={0 0}, upload=431917]: Done uploading piece. err: ecclient: upload failed (node:12fc5Trj8XxacZhzxdksUVT5CbG6uTmiwdUPjreYi4UMPY1TXWz, address:145.79.227.29:4943): protocol: write tcp 159.223.143.39:50820->145.79.227.29:4943: use of closed network connection; write tcp 159.223.143.39:50820->145.79.227.29:4943: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:50820->145.79.227.29:4943: use of closed network connection Dec 22 22:31:35.399379485 (415h3m36.227087866s) [seg_pos={0 0}, upload=431917]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 22 22:31:35.421122544 (415h3m36.248830935s) [upload=431917]: Done uploading segments... Dec 22 22:31:35.421174369 (415h3m36.248882769s) [upload=431917]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 407.329786ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............................................X...............] duration: 524.466856ms trace: https://jaeger.dev.storj.io/trace/7013ffb1a981d605 FAIL (2025-12-22 22:43:50 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 22 22:31:05.439510308 (415h3m6.2672187s) [upload=431917]: Starting upload Dec 22 22:31:05.439581993 (415h3m6.26729037s) [upload=431917]: Done starting upload Dec 22 22:31:05.439812792 (415h3m6.267521185s) [upload=431917]: Uploading segments... Dec 22 22:31:05.439942524 (415h3m6.267650918s) [upload=431917]: Got next segment. Inline: false Dec 22 22:31:05.440022744 (415h3m6.267731174s) [upload=431917]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 22 22:31:05.440029670 (415h3m6.267738072s) [upload=431917]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 22 22:31:05.467186372 (415h3m6.294894769s) [seg_pos={0 0}, upload=431917]: Begin upload segment... Dec 22 22:31:05.467265149 (415h3m6.294973535s) [seg_pos={0 0}, upload=431917]: Done begin upload segment. Dec 22 22:31:05.467296974 (415h3m6.295005373s) [noise=true , address=145.79.227.51:4409 , piece_id=ATH3F4DFOUUIFFJZRA2SSRRL7AKWZMQNQRYOZCVFFDLTFCFP236A, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.467389853 (415h3m6.295098247s) [noise=true , address=145.79.227.39:4013 , piece_id=JJVURGSLKY2UQ4FKNCLATA3ZBD5SDL32XJAG42KH2U6OYK3KTS6Q, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.467912104 (415h3m6.295620486s) [noise=true , address=145.79.227.36:4519 , piece_id=JUKRRIZWVGKEDD3GPJ7HMY2MEQZNMUCTKTXR6ZLSIERRONTZKHCQ, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.468324863 (415h3m6.296033263s) [noise=true , address=145.79.227.38:4979 , piece_id=HLMQC5OEB4OL4OZSRMCEVAWR5QM2T4MQUBPTHRVQLEAHPBWPQF6Q, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.468334764 (415h3m6.296043147s) [noise=true , address=145.79.227.40:4051 , piece_id=55RRAD7DQX6SR7RBLYAV7MD7XRWPNIWHTVVBQZ2ZL3OE2L7LRRIQ, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.468742525 (415h3m6.296450916s) [noise=true , address=45.91.116.4:4939 , piece_id=QYNKMKDLQJV7B6M2E34DU53Y7YI6VBCUW3C7ZLQOI4XWTBXZHKDQ, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.468915034 (415h3m6.296623414s) [noise=true , address=45.91.116.11:4117 , piece_id=WLMO5E3BIFFL3TOAUSSBGORIZ6DOCWFCMVT4AVFPPGPSKP2OKDAA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.469069393 (415h3m6.296777787s) [noise=true , address=145.79.227.32:4835 , piece_id=BRE3JH3XKPXB6FN7WB3W6ED7MOFES6HYLR34YGR6L5AX7RPIQ7IA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.469285789 (415h3m6.296994183s) [noise=true , address=145.79.227.29:4943 , piece_id=G4TMZ6IAIZBDON3UEDPH4LJQFHV3JLALEXSJ3VI4XN6WHIZOB57Q, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.469411962 (415h3m6.297120355s) [noise=true , address=145.79.227.48:4301 , piece_id=LBIYTJNOX5J4M3HDE7FGF5CSARBITW2AK3P2DSYFLFKUOIHZUKOQ, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.469749379 (415h3m6.297457773s) [noise=true , address=145.79.227.37:4565 , piece_id=554RHTYPG27UNCFPSRMOB245AX4QXL2QFIYECNTVWNNJMAL4BTYA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.470080828 (415h3m6.29778922s) [noise=true , address=145.79.227.43:4157 , piece_id=3W3LVTU37DNHWS52IGIIH7SXWXVNPGEY27RQNT272L3Z64GNR4KA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.470477660 (415h3m6.298186058s) [noise=true , address=45.91.116.14:4190 , piece_id=V4T6Z7K3C75LQEXNQIYZGYWW2CSQ7UPSET46QGDRT52RFGAROTTA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.470567977 (415h3m6.29827637s) [noise=true , address=145.79.227.15:4874 , piece_id=FNCCIMIVJTGZ2X5TOYOZH727JXHYM7HG3HBHBTZDAG67SHZLYYYA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.470980950 (415h3m6.298689349s) [noise=true , address=145.79.227.50:4373 , piece_id=4U55T5QDYCE6AZ3NCTPAVECPAELPQUKT3HV7GZL7JFRMWRN2AZUQ, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.471440098 (415h3m6.299148489s) [noise=true , address=145.79.227.47:4266 , piece_id=3NFYV2PWBNTAOSFHENM4J5PZ7XXE72UKTQVXLTVQRHGK4DUXAKDA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.560766475 (415h3m6.388474871s) [upload=431917]: Next returned nil segment Dec 22 22:31:05.560785161 (415h3m6.388493562s) [upload=431917]: Waiting for error group managing segments... Dec 22 22:31:05.566117878 (415h3m6.39382628s) [noise=true , address=45.91.116.4:4939 , piece_id=QYNKMKDLQJV7B6M2E34DU53Y7YI6VBCUW3C7ZLQOI4XWTBXZHKDQ, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.566723306 (415h3m6.394431687s) [noise=true , address=45.91.116.11:4117 , piece_id=WLMO5E3BIFFL3TOAUSSBGORIZ6DOCWFCMVT4AVFPPGPSKP2OKDAA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.566992667 (415h3m6.394701063s) [noise=true , address=145.79.227.36:4519 , piece_id=JUKRRIZWVGKEDD3GPJ7HMY2MEQZNMUCTKTXR6ZLSIERRONTZKHCQ, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.567464626 (415h3m6.395173019s) [noise=true , address=145.79.227.47:4266 , piece_id=3NFYV2PWBNTAOSFHENM4J5PZ7XXE72UKTQVXLTVQRHGK4DUXAKDA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.567924492 (415h3m6.395632883s) [noise=true , address=145.79.227.15:4874 , piece_id=FNCCIMIVJTGZ2X5TOYOZH727JXHYM7HG3HBHBTZDAG67SHZLYYYA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.568807294 (415h3m6.396515685s) [noise=true , address=145.79.227.51:4409 , piece_id=ATH3F4DFOUUIFFJZRA2SSRRL7AKWZMQNQRYOZCVFFDLTFCFP236A, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.570579823 (415h3m6.398288214s) [noise=true , address=145.79.227.48:4301 , piece_id=LBIYTJNOX5J4M3HDE7FGF5CSARBITW2AK3P2DSYFLFKUOIHZUKOQ, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.571632758 (415h3m6.399341149s) [noise=true , address=145.79.227.43:4157 , piece_id=3W3LVTU37DNHWS52IGIIH7SXWXVNPGEY27RQNT272L3Z64GNR4KA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.573168383 (415h3m6.400876774s) [noise=true , address=145.79.227.38:4979 , piece_id=HLMQC5OEB4OL4OZSRMCEVAWR5QM2T4MQUBPTHRVQLEAHPBWPQF6Q, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.579995420 (415h3m6.407703816s) [noise=true , address=45.91.116.14:4190 , piece_id=V4T6Z7K3C75LQEXNQIYZGYWW2CSQ7UPSET46QGDRT52RFGAROTTA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.592093172 (415h3m6.419801563s) [noise=true , address=145.79.227.37:4565 , piece_id=554RHTYPG27UNCFPSRMOB245AX4QXL2QFIYECNTVWNNJMAL4BTYA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:09.946500434 (415h3m10.774208816s) [noise=true , address=145.79.227.40:4051 , piece_id=55RRAD7DQX6SR7RBLYAV7MD7XRWPNIWHTVVBQZ2ZL3OE2L7LRRIQ, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:35.398235543 (415h3m36.225943943s) [noise=true , address=145.79.227.39:4013 , piece_id=JJVURGSLKY2UQ4FKNCLATA3ZBD5SDL32XJAG42KH2U6OYK3KTS6Q, seg_pos={0 0}, upload=431917]: Done uploading piece. err: ecclient: upload failed (node:121scLvKEW9zw2aEz22zpremcjiop67eQ8v2uqpmaD5aZzCAP7b, address:145.79.227.39:4013): protocol: write tcp 159.223.143.39:60842->145.79.227.39:4013: use of closed network connection; write tcp 159.223.143.39:60842->145.79.227.39:4013: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:60842->145.79.227.39:4013: use of closed network connection Dec 22 22:31:35.398562305 (415h3m36.226270699s) [noise=true , address=145.79.227.32:4835 , piece_id=BRE3JH3XKPXB6FN7WB3W6ED7MOFES6HYLR34YGR6L5AX7RPIQ7IA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: ecclient: upload failed (node:16vVfeoqCkbMYCRjMMWaKBnCvQBBJPMgwQtWTq9HKQrD9SS17y, address:145.79.227.32:4835): protocol: write tcp 159.223.143.39:45098->145.79.227.32:4835: use of closed network connection; write tcp 159.223.143.39:45098->145.79.227.32:4835: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:45098->145.79.227.32:4835: use of closed network connection Dec 22 22:31:35.399183028 (415h3m36.226891422s) [noise=true , address=145.79.227.50:4373 , piece_id=4U55T5QDYCE6AZ3NCTPAVECPAELPQUKT3HV7GZL7JFRMWRN2AZUQ, seg_pos={0 0}, upload=431917]: Done uploading piece. err: ecclient: upload failed (node:12AQGSkJuNJDxoeqhChUenWxHwF5yK41UESCF63FQz3XvabXXfz, address:145.79.227.50:4373): protocol: read tcp 159.223.143.39:56028->145.79.227.50:4373: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 22 22:31:35.399278021 (415h3m36.226986417s) [noise=true , address=145.79.227.29:4943 , piece_id=G4TMZ6IAIZBDON3UEDPH4LJQFHV3JLALEXSJ3VI4XN6WHIZOB57Q, seg_pos={0 0}, upload=431917]: Done uploading piece. err: ecclient: upload failed (node:12fc5Trj8XxacZhzxdksUVT5CbG6uTmiwdUPjreYi4UMPY1TXWz, address:145.79.227.29:4943): protocol: write tcp 159.223.143.39:50820->145.79.227.29:4943: use of closed network connection; write tcp 159.223.143.39:50820->145.79.227.29:4943: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:50820->145.79.227.29:4943: use of closed network connection Dec 22 22:31:35.399379485 (415h3m36.227087866s) [seg_pos={0 0}, upload=431917]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 22 22:31:35.421122544 (415h3m36.248830935s) [upload=431917]: Done uploading segments... Dec 22 22:31:35.421174369 (415h3m36.248882769s) [upload=431917]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 524.494076ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............................................X..............] duration: 477.861784ms trace: https://jaeger.dev.storj.io/trace/7013ffb1a981d605 FAIL (2025-12-22 22:42:55 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 22 22:31:05.439510308 (415h3m6.2672187s) [upload=431917]: Starting upload Dec 22 22:31:05.439581993 (415h3m6.26729037s) [upload=431917]: Done starting upload Dec 22 22:31:05.439812792 (415h3m6.267521185s) [upload=431917]: Uploading segments... Dec 22 22:31:05.439942524 (415h3m6.267650918s) [upload=431917]: Got next segment. Inline: false Dec 22 22:31:05.440022744 (415h3m6.267731174s) [upload=431917]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 22 22:31:05.440029670 (415h3m6.267738072s) [upload=431917]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 22 22:31:05.467186372 (415h3m6.294894769s) [seg_pos={0 0}, upload=431917]: Begin upload segment... Dec 22 22:31:05.467265149 (415h3m6.294973535s) [seg_pos={0 0}, upload=431917]: Done begin upload segment. Dec 22 22:31:05.467296974 (415h3m6.295005373s) [noise=true , address=145.79.227.51:4409 , piece_id=ATH3F4DFOUUIFFJZRA2SSRRL7AKWZMQNQRYOZCVFFDLTFCFP236A, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.467389853 (415h3m6.295098247s) [noise=true , address=145.79.227.39:4013 , piece_id=JJVURGSLKY2UQ4FKNCLATA3ZBD5SDL32XJAG42KH2U6OYK3KTS6Q, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.467912104 (415h3m6.295620486s) [noise=true , address=145.79.227.36:4519 , piece_id=JUKRRIZWVGKEDD3GPJ7HMY2MEQZNMUCTKTXR6ZLSIERRONTZKHCQ, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.468324863 (415h3m6.296033263s) [noise=true , address=145.79.227.38:4979 , piece_id=HLMQC5OEB4OL4OZSRMCEVAWR5QM2T4MQUBPTHRVQLEAHPBWPQF6Q, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.468334764 (415h3m6.296043147s) [noise=true , address=145.79.227.40:4051 , piece_id=55RRAD7DQX6SR7RBLYAV7MD7XRWPNIWHTVVBQZ2ZL3OE2L7LRRIQ, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.468742525 (415h3m6.296450916s) [noise=true , address=45.91.116.4:4939 , piece_id=QYNKMKDLQJV7B6M2E34DU53Y7YI6VBCUW3C7ZLQOI4XWTBXZHKDQ, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.468915034 (415h3m6.296623414s) [noise=true , address=45.91.116.11:4117 , piece_id=WLMO5E3BIFFL3TOAUSSBGORIZ6DOCWFCMVT4AVFPPGPSKP2OKDAA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.469069393 (415h3m6.296777787s) [noise=true , address=145.79.227.32:4835 , piece_id=BRE3JH3XKPXB6FN7WB3W6ED7MOFES6HYLR34YGR6L5AX7RPIQ7IA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.469285789 (415h3m6.296994183s) [noise=true , address=145.79.227.29:4943 , piece_id=G4TMZ6IAIZBDON3UEDPH4LJQFHV3JLALEXSJ3VI4XN6WHIZOB57Q, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.469411962 (415h3m6.297120355s) [noise=true , address=145.79.227.48:4301 , piece_id=LBIYTJNOX5J4M3HDE7FGF5CSARBITW2AK3P2DSYFLFKUOIHZUKOQ, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.469749379 (415h3m6.297457773s) [noise=true , address=145.79.227.37:4565 , piece_id=554RHTYPG27UNCFPSRMOB245AX4QXL2QFIYECNTVWNNJMAL4BTYA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.470080828 (415h3m6.29778922s) [noise=true , address=145.79.227.43:4157 , piece_id=3W3LVTU37DNHWS52IGIIH7SXWXVNPGEY27RQNT272L3Z64GNR4KA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.470477660 (415h3m6.298186058s) [noise=true , address=45.91.116.14:4190 , piece_id=V4T6Z7K3C75LQEXNQIYZGYWW2CSQ7UPSET46QGDRT52RFGAROTTA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.470567977 (415h3m6.29827637s) [noise=true , address=145.79.227.15:4874 , piece_id=FNCCIMIVJTGZ2X5TOYOZH727JXHYM7HG3HBHBTZDAG67SHZLYYYA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.470980950 (415h3m6.298689349s) [noise=true , address=145.79.227.50:4373 , piece_id=4U55T5QDYCE6AZ3NCTPAVECPAELPQUKT3HV7GZL7JFRMWRN2AZUQ, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.471440098 (415h3m6.299148489s) [noise=true , address=145.79.227.47:4266 , piece_id=3NFYV2PWBNTAOSFHENM4J5PZ7XXE72UKTQVXLTVQRHGK4DUXAKDA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.560766475 (415h3m6.388474871s) [upload=431917]: Next returned nil segment Dec 22 22:31:05.560785161 (415h3m6.388493562s) [upload=431917]: Waiting for error group managing segments... Dec 22 22:31:05.566117878 (415h3m6.39382628s) [noise=true , address=45.91.116.4:4939 , piece_id=QYNKMKDLQJV7B6M2E34DU53Y7YI6VBCUW3C7ZLQOI4XWTBXZHKDQ, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.566723306 (415h3m6.394431687s) [noise=true , address=45.91.116.11:4117 , piece_id=WLMO5E3BIFFL3TOAUSSBGORIZ6DOCWFCMVT4AVFPPGPSKP2OKDAA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.566992667 (415h3m6.394701063s) [noise=true , address=145.79.227.36:4519 , piece_id=JUKRRIZWVGKEDD3GPJ7HMY2MEQZNMUCTKTXR6ZLSIERRONTZKHCQ, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.567464626 (415h3m6.395173019s) [noise=true , address=145.79.227.47:4266 , piece_id=3NFYV2PWBNTAOSFHENM4J5PZ7XXE72UKTQVXLTVQRHGK4DUXAKDA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.567924492 (415h3m6.395632883s) [noise=true , address=145.79.227.15:4874 , piece_id=FNCCIMIVJTGZ2X5TOYOZH727JXHYM7HG3HBHBTZDAG67SHZLYYYA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.568807294 (415h3m6.396515685s) [noise=true , address=145.79.227.51:4409 , piece_id=ATH3F4DFOUUIFFJZRA2SSRRL7AKWZMQNQRYOZCVFFDLTFCFP236A, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.570579823 (415h3m6.398288214s) [noise=true , address=145.79.227.48:4301 , piece_id=LBIYTJNOX5J4M3HDE7FGF5CSARBITW2AK3P2DSYFLFKUOIHZUKOQ, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.571632758 (415h3m6.399341149s) [noise=true , address=145.79.227.43:4157 , piece_id=3W3LVTU37DNHWS52IGIIH7SXWXVNPGEY27RQNT272L3Z64GNR4KA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.573168383 (415h3m6.400876774s) [noise=true , address=145.79.227.38:4979 , piece_id=HLMQC5OEB4OL4OZSRMCEVAWR5QM2T4MQUBPTHRVQLEAHPBWPQF6Q, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.579995420 (415h3m6.407703816s) [noise=true , address=45.91.116.14:4190 , piece_id=V4T6Z7K3C75LQEXNQIYZGYWW2CSQ7UPSET46QGDRT52RFGAROTTA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.592093172 (415h3m6.419801563s) [noise=true , address=145.79.227.37:4565 , piece_id=554RHTYPG27UNCFPSRMOB245AX4QXL2QFIYECNTVWNNJMAL4BTYA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:09.946500434 (415h3m10.774208816s) [noise=true , address=145.79.227.40:4051 , piece_id=55RRAD7DQX6SR7RBLYAV7MD7XRWPNIWHTVVBQZ2ZL3OE2L7LRRIQ, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:35.398235543 (415h3m36.225943943s) [noise=true , address=145.79.227.39:4013 , piece_id=JJVURGSLKY2UQ4FKNCLATA3ZBD5SDL32XJAG42KH2U6OYK3KTS6Q, seg_pos={0 0}, upload=431917]: Done uploading piece. err: ecclient: upload failed (node:121scLvKEW9zw2aEz22zpremcjiop67eQ8v2uqpmaD5aZzCAP7b, address:145.79.227.39:4013): protocol: write tcp 159.223.143.39:60842->145.79.227.39:4013: use of closed network connection; write tcp 159.223.143.39:60842->145.79.227.39:4013: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:60842->145.79.227.39:4013: use of closed network connection Dec 22 22:31:35.398562305 (415h3m36.226270699s) [noise=true , address=145.79.227.32:4835 , piece_id=BRE3JH3XKPXB6FN7WB3W6ED7MOFES6HYLR34YGR6L5AX7RPIQ7IA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: ecclient: upload failed (node:16vVfeoqCkbMYCRjMMWaKBnCvQBBJPMgwQtWTq9HKQrD9SS17y, address:145.79.227.32:4835): protocol: write tcp 159.223.143.39:45098->145.79.227.32:4835: use of closed network connection; write tcp 159.223.143.39:45098->145.79.227.32:4835: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:45098->145.79.227.32:4835: use of closed network connection Dec 22 22:31:35.399183028 (415h3m36.226891422s) [noise=true , address=145.79.227.50:4373 , piece_id=4U55T5QDYCE6AZ3NCTPAVECPAELPQUKT3HV7GZL7JFRMWRN2AZUQ, seg_pos={0 0}, upload=431917]: Done uploading piece. err: ecclient: upload failed (node:12AQGSkJuNJDxoeqhChUenWxHwF5yK41UESCF63FQz3XvabXXfz, address:145.79.227.50:4373): protocol: read tcp 159.223.143.39:56028->145.79.227.50:4373: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 22 22:31:35.399278021 (415h3m36.226986417s) [noise=true , address=145.79.227.29:4943 , piece_id=G4TMZ6IAIZBDON3UEDPH4LJQFHV3JLALEXSJ3VI4XN6WHIZOB57Q, seg_pos={0 0}, upload=431917]: Done uploading piece. err: ecclient: upload failed (node:12fc5Trj8XxacZhzxdksUVT5CbG6uTmiwdUPjreYi4UMPY1TXWz, address:145.79.227.29:4943): protocol: write tcp 159.223.143.39:50820->145.79.227.29:4943: use of closed network connection; write tcp 159.223.143.39:50820->145.79.227.29:4943: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:50820->145.79.227.29:4943: use of closed network connection Dec 22 22:31:35.399379485 (415h3m36.227087866s) [seg_pos={0 0}, upload=431917]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 22 22:31:35.421122544 (415h3m36.248830935s) [upload=431917]: Done uploading segments... Dec 22 22:31:35.421174369 (415h3m36.248882769s) [upload=431917]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 477.893004ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............................................X.............] duration: 531.377185ms trace: https://jaeger.dev.storj.io/trace/7013ffb1a981d605 FAIL (2025-12-22 22:42:31 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 22 22:31:05.439510308 (415h3m6.2672187s) [upload=431917]: Starting upload Dec 22 22:31:05.439581993 (415h3m6.26729037s) [upload=431917]: Done starting upload Dec 22 22:31:05.439812792 (415h3m6.267521185s) [upload=431917]: Uploading segments... Dec 22 22:31:05.439942524 (415h3m6.267650918s) [upload=431917]: Got next segment. Inline: false Dec 22 22:31:05.440022744 (415h3m6.267731174s) [upload=431917]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 22 22:31:05.440029670 (415h3m6.267738072s) [upload=431917]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 22 22:31:05.467186372 (415h3m6.294894769s) [seg_pos={0 0}, upload=431917]: Begin upload segment... Dec 22 22:31:05.467265149 (415h3m6.294973535s) [seg_pos={0 0}, upload=431917]: Done begin upload segment. Dec 22 22:31:05.467296974 (415h3m6.295005373s) [noise=true , address=145.79.227.51:4409 , piece_id=ATH3F4DFOUUIFFJZRA2SSRRL7AKWZMQNQRYOZCVFFDLTFCFP236A, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.467389853 (415h3m6.295098247s) [noise=true , address=145.79.227.39:4013 , piece_id=JJVURGSLKY2UQ4FKNCLATA3ZBD5SDL32XJAG42KH2U6OYK3KTS6Q, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.467912104 (415h3m6.295620486s) [noise=true , address=145.79.227.36:4519 , piece_id=JUKRRIZWVGKEDD3GPJ7HMY2MEQZNMUCTKTXR6ZLSIERRONTZKHCQ, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.468324863 (415h3m6.296033263s) [noise=true , address=145.79.227.38:4979 , piece_id=HLMQC5OEB4OL4OZSRMCEVAWR5QM2T4MQUBPTHRVQLEAHPBWPQF6Q, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.468334764 (415h3m6.296043147s) [noise=true , address=145.79.227.40:4051 , piece_id=55RRAD7DQX6SR7RBLYAV7MD7XRWPNIWHTVVBQZ2ZL3OE2L7LRRIQ, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.468742525 (415h3m6.296450916s) [noise=true , address=45.91.116.4:4939 , piece_id=QYNKMKDLQJV7B6M2E34DU53Y7YI6VBCUW3C7ZLQOI4XWTBXZHKDQ, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.468915034 (415h3m6.296623414s) [noise=true , address=45.91.116.11:4117 , piece_id=WLMO5E3BIFFL3TOAUSSBGORIZ6DOCWFCMVT4AVFPPGPSKP2OKDAA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.469069393 (415h3m6.296777787s) [noise=true , address=145.79.227.32:4835 , piece_id=BRE3JH3XKPXB6FN7WB3W6ED7MOFES6HYLR34YGR6L5AX7RPIQ7IA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.469285789 (415h3m6.296994183s) [noise=true , address=145.79.227.29:4943 , piece_id=G4TMZ6IAIZBDON3UEDPH4LJQFHV3JLALEXSJ3VI4XN6WHIZOB57Q, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.469411962 (415h3m6.297120355s) [noise=true , address=145.79.227.48:4301 , piece_id=LBIYTJNOX5J4M3HDE7FGF5CSARBITW2AK3P2DSYFLFKUOIHZUKOQ, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.469749379 (415h3m6.297457773s) [noise=true , address=145.79.227.37:4565 , piece_id=554RHTYPG27UNCFPSRMOB245AX4QXL2QFIYECNTVWNNJMAL4BTYA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.470080828 (415h3m6.29778922s) [noise=true , address=145.79.227.43:4157 , piece_id=3W3LVTU37DNHWS52IGIIH7SXWXVNPGEY27RQNT272L3Z64GNR4KA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.470477660 (415h3m6.298186058s) [noise=true , address=45.91.116.14:4190 , piece_id=V4T6Z7K3C75LQEXNQIYZGYWW2CSQ7UPSET46QGDRT52RFGAROTTA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.470567977 (415h3m6.29827637s) [noise=true , address=145.79.227.15:4874 , piece_id=FNCCIMIVJTGZ2X5TOYOZH727JXHYM7HG3HBHBTZDAG67SHZLYYYA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.470980950 (415h3m6.298689349s) [noise=true , address=145.79.227.50:4373 , piece_id=4U55T5QDYCE6AZ3NCTPAVECPAELPQUKT3HV7GZL7JFRMWRN2AZUQ, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.471440098 (415h3m6.299148489s) [noise=true , address=145.79.227.47:4266 , piece_id=3NFYV2PWBNTAOSFHENM4J5PZ7XXE72UKTQVXLTVQRHGK4DUXAKDA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.560766475 (415h3m6.388474871s) [upload=431917]: Next returned nil segment Dec 22 22:31:05.560785161 (415h3m6.388493562s) [upload=431917]: Waiting for error group managing segments... Dec 22 22:31:05.566117878 (415h3m6.39382628s) [noise=true , address=45.91.116.4:4939 , piece_id=QYNKMKDLQJV7B6M2E34DU53Y7YI6VBCUW3C7ZLQOI4XWTBXZHKDQ, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.566723306 (415h3m6.394431687s) [noise=true , address=45.91.116.11:4117 , piece_id=WLMO5E3BIFFL3TOAUSSBGORIZ6DOCWFCMVT4AVFPPGPSKP2OKDAA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.566992667 (415h3m6.394701063s) [noise=true , address=145.79.227.36:4519 , piece_id=JUKRRIZWVGKEDD3GPJ7HMY2MEQZNMUCTKTXR6ZLSIERRONTZKHCQ, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.567464626 (415h3m6.395173019s) [noise=true , address=145.79.227.47:4266 , piece_id=3NFYV2PWBNTAOSFHENM4J5PZ7XXE72UKTQVXLTVQRHGK4DUXAKDA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.567924492 (415h3m6.395632883s) [noise=true , address=145.79.227.15:4874 , piece_id=FNCCIMIVJTGZ2X5TOYOZH727JXHYM7HG3HBHBTZDAG67SHZLYYYA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.568807294 (415h3m6.396515685s) [noise=true , address=145.79.227.51:4409 , piece_id=ATH3F4DFOUUIFFJZRA2SSRRL7AKWZMQNQRYOZCVFFDLTFCFP236A, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.570579823 (415h3m6.398288214s) [noise=true , address=145.79.227.48:4301 , piece_id=LBIYTJNOX5J4M3HDE7FGF5CSARBITW2AK3P2DSYFLFKUOIHZUKOQ, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.571632758 (415h3m6.399341149s) [noise=true , address=145.79.227.43:4157 , piece_id=3W3LVTU37DNHWS52IGIIH7SXWXVNPGEY27RQNT272L3Z64GNR4KA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.573168383 (415h3m6.400876774s) [noise=true , address=145.79.227.38:4979 , piece_id=HLMQC5OEB4OL4OZSRMCEVAWR5QM2T4MQUBPTHRVQLEAHPBWPQF6Q, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.579995420 (415h3m6.407703816s) [noise=true , address=45.91.116.14:4190 , piece_id=V4T6Z7K3C75LQEXNQIYZGYWW2CSQ7UPSET46QGDRT52RFGAROTTA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.592093172 (415h3m6.419801563s) [noise=true , address=145.79.227.37:4565 , piece_id=554RHTYPG27UNCFPSRMOB245AX4QXL2QFIYECNTVWNNJMAL4BTYA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:09.946500434 (415h3m10.774208816s) [noise=true , address=145.79.227.40:4051 , piece_id=55RRAD7DQX6SR7RBLYAV7MD7XRWPNIWHTVVBQZ2ZL3OE2L7LRRIQ, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:35.398235543 (415h3m36.225943943s) [noise=true , address=145.79.227.39:4013 , piece_id=JJVURGSLKY2UQ4FKNCLATA3ZBD5SDL32XJAG42KH2U6OYK3KTS6Q, seg_pos={0 0}, upload=431917]: Done uploading piece. err: ecclient: upload failed (node:121scLvKEW9zw2aEz22zpremcjiop67eQ8v2uqpmaD5aZzCAP7b, address:145.79.227.39:4013): protocol: write tcp 159.223.143.39:60842->145.79.227.39:4013: use of closed network connection; write tcp 159.223.143.39:60842->145.79.227.39:4013: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:60842->145.79.227.39:4013: use of closed network connection Dec 22 22:31:35.398562305 (415h3m36.226270699s) [noise=true , address=145.79.227.32:4835 , piece_id=BRE3JH3XKPXB6FN7WB3W6ED7MOFES6HYLR34YGR6L5AX7RPIQ7IA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: ecclient: upload failed (node:16vVfeoqCkbMYCRjMMWaKBnCvQBBJPMgwQtWTq9HKQrD9SS17y, address:145.79.227.32:4835): protocol: write tcp 159.223.143.39:45098->145.79.227.32:4835: use of closed network connection; write tcp 159.223.143.39:45098->145.79.227.32:4835: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:45098->145.79.227.32:4835: use of closed network connection Dec 22 22:31:35.399183028 (415h3m36.226891422s) [noise=true , address=145.79.227.50:4373 , piece_id=4U55T5QDYCE6AZ3NCTPAVECPAELPQUKT3HV7GZL7JFRMWRN2AZUQ, seg_pos={0 0}, upload=431917]: Done uploading piece. err: ecclient: upload failed (node:12AQGSkJuNJDxoeqhChUenWxHwF5yK41UESCF63FQz3XvabXXfz, address:145.79.227.50:4373): protocol: read tcp 159.223.143.39:56028->145.79.227.50:4373: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 22 22:31:35.399278021 (415h3m36.226986417s) [noise=true , address=145.79.227.29:4943 , piece_id=G4TMZ6IAIZBDON3UEDPH4LJQFHV3JLALEXSJ3VI4XN6WHIZOB57Q, seg_pos={0 0}, upload=431917]: Done uploading piece. err: ecclient: upload failed (node:12fc5Trj8XxacZhzxdksUVT5CbG6uTmiwdUPjreYi4UMPY1TXWz, address:145.79.227.29:4943): protocol: write tcp 159.223.143.39:50820->145.79.227.29:4943: use of closed network connection; write tcp 159.223.143.39:50820->145.79.227.29:4943: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:50820->145.79.227.29:4943: use of closed network connection Dec 22 22:31:35.399379485 (415h3m36.227087866s) [seg_pos={0 0}, upload=431917]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 22 22:31:35.421122544 (415h3m36.248830935s) [upload=431917]: Done uploading segments... Dec 22 22:31:35.421174369 (415h3m36.248882769s) [upload=431917]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 531.406053ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............................................X............] duration: 458.013511ms trace: https://jaeger.dev.storj.io/trace/7013ffb1a981d605 FAIL (2025-12-22 22:41:33 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 22 22:31:05.439510308 (415h3m6.2672187s) [upload=431917]: Starting upload Dec 22 22:31:05.439581993 (415h3m6.26729037s) [upload=431917]: Done starting upload Dec 22 22:31:05.439812792 (415h3m6.267521185s) [upload=431917]: Uploading segments... Dec 22 22:31:05.439942524 (415h3m6.267650918s) [upload=431917]: Got next segment. Inline: false Dec 22 22:31:05.440022744 (415h3m6.267731174s) [upload=431917]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 22 22:31:05.440029670 (415h3m6.267738072s) [upload=431917]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 22 22:31:05.467186372 (415h3m6.294894769s) [seg_pos={0 0}, upload=431917]: Begin upload segment... Dec 22 22:31:05.467265149 (415h3m6.294973535s) [seg_pos={0 0}, upload=431917]: Done begin upload segment. Dec 22 22:31:05.467296974 (415h3m6.295005373s) [noise=true , address=145.79.227.51:4409 , piece_id=ATH3F4DFOUUIFFJZRA2SSRRL7AKWZMQNQRYOZCVFFDLTFCFP236A, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.467389853 (415h3m6.295098247s) [noise=true , address=145.79.227.39:4013 , piece_id=JJVURGSLKY2UQ4FKNCLATA3ZBD5SDL32XJAG42KH2U6OYK3KTS6Q, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.467912104 (415h3m6.295620486s) [noise=true , address=145.79.227.36:4519 , piece_id=JUKRRIZWVGKEDD3GPJ7HMY2MEQZNMUCTKTXR6ZLSIERRONTZKHCQ, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.468324863 (415h3m6.296033263s) [noise=true , address=145.79.227.38:4979 , piece_id=HLMQC5OEB4OL4OZSRMCEVAWR5QM2T4MQUBPTHRVQLEAHPBWPQF6Q, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.468334764 (415h3m6.296043147s) [noise=true , address=145.79.227.40:4051 , piece_id=55RRAD7DQX6SR7RBLYAV7MD7XRWPNIWHTVVBQZ2ZL3OE2L7LRRIQ, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.468742525 (415h3m6.296450916s) [noise=true , address=45.91.116.4:4939 , piece_id=QYNKMKDLQJV7B6M2E34DU53Y7YI6VBCUW3C7ZLQOI4XWTBXZHKDQ, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.468915034 (415h3m6.296623414s) [noise=true , address=45.91.116.11:4117 , piece_id=WLMO5E3BIFFL3TOAUSSBGORIZ6DOCWFCMVT4AVFPPGPSKP2OKDAA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.469069393 (415h3m6.296777787s) [noise=true , address=145.79.227.32:4835 , piece_id=BRE3JH3XKPXB6FN7WB3W6ED7MOFES6HYLR34YGR6L5AX7RPIQ7IA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.469285789 (415h3m6.296994183s) [noise=true , address=145.79.227.29:4943 , piece_id=G4TMZ6IAIZBDON3UEDPH4LJQFHV3JLALEXSJ3VI4XN6WHIZOB57Q, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.469411962 (415h3m6.297120355s) [noise=true , address=145.79.227.48:4301 , piece_id=LBIYTJNOX5J4M3HDE7FGF5CSARBITW2AK3P2DSYFLFKUOIHZUKOQ, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.469749379 (415h3m6.297457773s) [noise=true , address=145.79.227.37:4565 , piece_id=554RHTYPG27UNCFPSRMOB245AX4QXL2QFIYECNTVWNNJMAL4BTYA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.470080828 (415h3m6.29778922s) [noise=true , address=145.79.227.43:4157 , piece_id=3W3LVTU37DNHWS52IGIIH7SXWXVNPGEY27RQNT272L3Z64GNR4KA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.470477660 (415h3m6.298186058s) [noise=true , address=45.91.116.14:4190 , piece_id=V4T6Z7K3C75LQEXNQIYZGYWW2CSQ7UPSET46QGDRT52RFGAROTTA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.470567977 (415h3m6.29827637s) [noise=true , address=145.79.227.15:4874 , piece_id=FNCCIMIVJTGZ2X5TOYOZH727JXHYM7HG3HBHBTZDAG67SHZLYYYA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.470980950 (415h3m6.298689349s) [noise=true , address=145.79.227.50:4373 , piece_id=4U55T5QDYCE6AZ3NCTPAVECPAELPQUKT3HV7GZL7JFRMWRN2AZUQ, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.471440098 (415h3m6.299148489s) [noise=true , address=145.79.227.47:4266 , piece_id=3NFYV2PWBNTAOSFHENM4J5PZ7XXE72UKTQVXLTVQRHGK4DUXAKDA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.560766475 (415h3m6.388474871s) [upload=431917]: Next returned nil segment Dec 22 22:31:05.560785161 (415h3m6.388493562s) [upload=431917]: Waiting for error group managing segments... Dec 22 22:31:05.566117878 (415h3m6.39382628s) [noise=true , address=45.91.116.4:4939 , piece_id=QYNKMKDLQJV7B6M2E34DU53Y7YI6VBCUW3C7ZLQOI4XWTBXZHKDQ, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.566723306 (415h3m6.394431687s) [noise=true , address=45.91.116.11:4117 , piece_id=WLMO5E3BIFFL3TOAUSSBGORIZ6DOCWFCMVT4AVFPPGPSKP2OKDAA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.566992667 (415h3m6.394701063s) [noise=true , address=145.79.227.36:4519 , piece_id=JUKRRIZWVGKEDD3GPJ7HMY2MEQZNMUCTKTXR6ZLSIERRONTZKHCQ, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.567464626 (415h3m6.395173019s) [noise=true , address=145.79.227.47:4266 , piece_id=3NFYV2PWBNTAOSFHENM4J5PZ7XXE72UKTQVXLTVQRHGK4DUXAKDA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.567924492 (415h3m6.395632883s) [noise=true , address=145.79.227.15:4874 , piece_id=FNCCIMIVJTGZ2X5TOYOZH727JXHYM7HG3HBHBTZDAG67SHZLYYYA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.568807294 (415h3m6.396515685s) [noise=true , address=145.79.227.51:4409 , piece_id=ATH3F4DFOUUIFFJZRA2SSRRL7AKWZMQNQRYOZCVFFDLTFCFP236A, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.570579823 (415h3m6.398288214s) [noise=true , address=145.79.227.48:4301 , piece_id=LBIYTJNOX5J4M3HDE7FGF5CSARBITW2AK3P2DSYFLFKUOIHZUKOQ, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.571632758 (415h3m6.399341149s) [noise=true , address=145.79.227.43:4157 , piece_id=3W3LVTU37DNHWS52IGIIH7SXWXVNPGEY27RQNT272L3Z64GNR4KA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.573168383 (415h3m6.400876774s) [noise=true , address=145.79.227.38:4979 , piece_id=HLMQC5OEB4OL4OZSRMCEVAWR5QM2T4MQUBPTHRVQLEAHPBWPQF6Q, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.579995420 (415h3m6.407703816s) [noise=true , address=45.91.116.14:4190 , piece_id=V4T6Z7K3C75LQEXNQIYZGYWW2CSQ7UPSET46QGDRT52RFGAROTTA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.592093172 (415h3m6.419801563s) [noise=true , address=145.79.227.37:4565 , piece_id=554RHTYPG27UNCFPSRMOB245AX4QXL2QFIYECNTVWNNJMAL4BTYA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:09.946500434 (415h3m10.774208816s) [noise=true , address=145.79.227.40:4051 , piece_id=55RRAD7DQX6SR7RBLYAV7MD7XRWPNIWHTVVBQZ2ZL3OE2L7LRRIQ, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:35.398235543 (415h3m36.225943943s) [noise=true , address=145.79.227.39:4013 , piece_id=JJVURGSLKY2UQ4FKNCLATA3ZBD5SDL32XJAG42KH2U6OYK3KTS6Q, seg_pos={0 0}, upload=431917]: Done uploading piece. err: ecclient: upload failed (node:121scLvKEW9zw2aEz22zpremcjiop67eQ8v2uqpmaD5aZzCAP7b, address:145.79.227.39:4013): protocol: write tcp 159.223.143.39:60842->145.79.227.39:4013: use of closed network connection; write tcp 159.223.143.39:60842->145.79.227.39:4013: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:60842->145.79.227.39:4013: use of closed network connection Dec 22 22:31:35.398562305 (415h3m36.226270699s) [noise=true , address=145.79.227.32:4835 , piece_id=BRE3JH3XKPXB6FN7WB3W6ED7MOFES6HYLR34YGR6L5AX7RPIQ7IA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: ecclient: upload failed (node:16vVfeoqCkbMYCRjMMWaKBnCvQBBJPMgwQtWTq9HKQrD9SS17y, address:145.79.227.32:4835): protocol: write tcp 159.223.143.39:45098->145.79.227.32:4835: use of closed network connection; write tcp 159.223.143.39:45098->145.79.227.32:4835: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:45098->145.79.227.32:4835: use of closed network connection Dec 22 22:31:35.399183028 (415h3m36.226891422s) [noise=true , address=145.79.227.50:4373 , piece_id=4U55T5QDYCE6AZ3NCTPAVECPAELPQUKT3HV7GZL7JFRMWRN2AZUQ, seg_pos={0 0}, upload=431917]: Done uploading piece. err: ecclient: upload failed (node:12AQGSkJuNJDxoeqhChUenWxHwF5yK41UESCF63FQz3XvabXXfz, address:145.79.227.50:4373): protocol: read tcp 159.223.143.39:56028->145.79.227.50:4373: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 22 22:31:35.399278021 (415h3m36.226986417s) [noise=true , address=145.79.227.29:4943 , piece_id=G4TMZ6IAIZBDON3UEDPH4LJQFHV3JLALEXSJ3VI4XN6WHIZOB57Q, seg_pos={0 0}, upload=431917]: Done uploading piece. err: ecclient: upload failed (node:12fc5Trj8XxacZhzxdksUVT5CbG6uTmiwdUPjreYi4UMPY1TXWz, address:145.79.227.29:4943): protocol: write tcp 159.223.143.39:50820->145.79.227.29:4943: use of closed network connection; write tcp 159.223.143.39:50820->145.79.227.29:4943: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:50820->145.79.227.29:4943: use of closed network connection Dec 22 22:31:35.399379485 (415h3m36.227087866s) [seg_pos={0 0}, upload=431917]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 22 22:31:35.421122544 (415h3m36.248830935s) [upload=431917]: Done uploading segments... Dec 22 22:31:35.421174369 (415h3m36.248882769s) [upload=431917]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 458.02808ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................................................X...........] duration: 490.327198ms trace: https://jaeger.dev.storj.io/trace/7013ffb1a981d605 FAIL (2025-12-22 22:40:21 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 22 22:31:05.439510308 (415h3m6.2672187s) [upload=431917]: Starting upload Dec 22 22:31:05.439581993 (415h3m6.26729037s) [upload=431917]: Done starting upload Dec 22 22:31:05.439812792 (415h3m6.267521185s) [upload=431917]: Uploading segments... Dec 22 22:31:05.439942524 (415h3m6.267650918s) [upload=431917]: Got next segment. Inline: false Dec 22 22:31:05.440022744 (415h3m6.267731174s) [upload=431917]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 22 22:31:05.440029670 (415h3m6.267738072s) [upload=431917]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 22 22:31:05.467186372 (415h3m6.294894769s) [seg_pos={0 0}, upload=431917]: Begin upload segment... Dec 22 22:31:05.467265149 (415h3m6.294973535s) [seg_pos={0 0}, upload=431917]: Done begin upload segment. Dec 22 22:31:05.467296974 (415h3m6.295005373s) [noise=true , address=145.79.227.51:4409 , piece_id=ATH3F4DFOUUIFFJZRA2SSRRL7AKWZMQNQRYOZCVFFDLTFCFP236A, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.467389853 (415h3m6.295098247s) [noise=true , address=145.79.227.39:4013 , piece_id=JJVURGSLKY2UQ4FKNCLATA3ZBD5SDL32XJAG42KH2U6OYK3KTS6Q, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.467912104 (415h3m6.295620486s) [noise=true , address=145.79.227.36:4519 , piece_id=JUKRRIZWVGKEDD3GPJ7HMY2MEQZNMUCTKTXR6ZLSIERRONTZKHCQ, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.468324863 (415h3m6.296033263s) [noise=true , address=145.79.227.38:4979 , piece_id=HLMQC5OEB4OL4OZSRMCEVAWR5QM2T4MQUBPTHRVQLEAHPBWPQF6Q, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.468334764 (415h3m6.296043147s) [noise=true , address=145.79.227.40:4051 , piece_id=55RRAD7DQX6SR7RBLYAV7MD7XRWPNIWHTVVBQZ2ZL3OE2L7LRRIQ, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.468742525 (415h3m6.296450916s) [noise=true , address=45.91.116.4:4939 , piece_id=QYNKMKDLQJV7B6M2E34DU53Y7YI6VBCUW3C7ZLQOI4XWTBXZHKDQ, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.468915034 (415h3m6.296623414s) [noise=true , address=45.91.116.11:4117 , piece_id=WLMO5E3BIFFL3TOAUSSBGORIZ6DOCWFCMVT4AVFPPGPSKP2OKDAA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.469069393 (415h3m6.296777787s) [noise=true , address=145.79.227.32:4835 , piece_id=BRE3JH3XKPXB6FN7WB3W6ED7MOFES6HYLR34YGR6L5AX7RPIQ7IA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.469285789 (415h3m6.296994183s) [noise=true , address=145.79.227.29:4943 , piece_id=G4TMZ6IAIZBDON3UEDPH4LJQFHV3JLALEXSJ3VI4XN6WHIZOB57Q, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.469411962 (415h3m6.297120355s) [noise=true , address=145.79.227.48:4301 , piece_id=LBIYTJNOX5J4M3HDE7FGF5CSARBITW2AK3P2DSYFLFKUOIHZUKOQ, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.469749379 (415h3m6.297457773s) [noise=true , address=145.79.227.37:4565 , piece_id=554RHTYPG27UNCFPSRMOB245AX4QXL2QFIYECNTVWNNJMAL4BTYA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.470080828 (415h3m6.29778922s) [noise=true , address=145.79.227.43:4157 , piece_id=3W3LVTU37DNHWS52IGIIH7SXWXVNPGEY27RQNT272L3Z64GNR4KA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.470477660 (415h3m6.298186058s) [noise=true , address=45.91.116.14:4190 , piece_id=V4T6Z7K3C75LQEXNQIYZGYWW2CSQ7UPSET46QGDRT52RFGAROTTA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.470567977 (415h3m6.29827637s) [noise=true , address=145.79.227.15:4874 , piece_id=FNCCIMIVJTGZ2X5TOYOZH727JXHYM7HG3HBHBTZDAG67SHZLYYYA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.470980950 (415h3m6.298689349s) [noise=true , address=145.79.227.50:4373 , piece_id=4U55T5QDYCE6AZ3NCTPAVECPAELPQUKT3HV7GZL7JFRMWRN2AZUQ, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.471440098 (415h3m6.299148489s) [noise=true , address=145.79.227.47:4266 , piece_id=3NFYV2PWBNTAOSFHENM4J5PZ7XXE72UKTQVXLTVQRHGK4DUXAKDA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.560766475 (415h3m6.388474871s) [upload=431917]: Next returned nil segment Dec 22 22:31:05.560785161 (415h3m6.388493562s) [upload=431917]: Waiting for error group managing segments... Dec 22 22:31:05.566117878 (415h3m6.39382628s) [noise=true , address=45.91.116.4:4939 , piece_id=QYNKMKDLQJV7B6M2E34DU53Y7YI6VBCUW3C7ZLQOI4XWTBXZHKDQ, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.566723306 (415h3m6.394431687s) [noise=true , address=45.91.116.11:4117 , piece_id=WLMO5E3BIFFL3TOAUSSBGORIZ6DOCWFCMVT4AVFPPGPSKP2OKDAA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.566992667 (415h3m6.394701063s) [noise=true , address=145.79.227.36:4519 , piece_id=JUKRRIZWVGKEDD3GPJ7HMY2MEQZNMUCTKTXR6ZLSIERRONTZKHCQ, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.567464626 (415h3m6.395173019s) [noise=true , address=145.79.227.47:4266 , piece_id=3NFYV2PWBNTAOSFHENM4J5PZ7XXE72UKTQVXLTVQRHGK4DUXAKDA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.567924492 (415h3m6.395632883s) [noise=true , address=145.79.227.15:4874 , piece_id=FNCCIMIVJTGZ2X5TOYOZH727JXHYM7HG3HBHBTZDAG67SHZLYYYA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.568807294 (415h3m6.396515685s) [noise=true , address=145.79.227.51:4409 , piece_id=ATH3F4DFOUUIFFJZRA2SSRRL7AKWZMQNQRYOZCVFFDLTFCFP236A, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.570579823 (415h3m6.398288214s) [noise=true , address=145.79.227.48:4301 , piece_id=LBIYTJNOX5J4M3HDE7FGF5CSARBITW2AK3P2DSYFLFKUOIHZUKOQ, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.571632758 (415h3m6.399341149s) [noise=true , address=145.79.227.43:4157 , piece_id=3W3LVTU37DNHWS52IGIIH7SXWXVNPGEY27RQNT272L3Z64GNR4KA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.573168383 (415h3m6.400876774s) [noise=true , address=145.79.227.38:4979 , piece_id=HLMQC5OEB4OL4OZSRMCEVAWR5QM2T4MQUBPTHRVQLEAHPBWPQF6Q, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.579995420 (415h3m6.407703816s) [noise=true , address=45.91.116.14:4190 , piece_id=V4T6Z7K3C75LQEXNQIYZGYWW2CSQ7UPSET46QGDRT52RFGAROTTA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.592093172 (415h3m6.419801563s) [noise=true , address=145.79.227.37:4565 , piece_id=554RHTYPG27UNCFPSRMOB245AX4QXL2QFIYECNTVWNNJMAL4BTYA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:09.946500434 (415h3m10.774208816s) [noise=true , address=145.79.227.40:4051 , piece_id=55RRAD7DQX6SR7RBLYAV7MD7XRWPNIWHTVVBQZ2ZL3OE2L7LRRIQ, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:35.398235543 (415h3m36.225943943s) [noise=true , address=145.79.227.39:4013 , piece_id=JJVURGSLKY2UQ4FKNCLATA3ZBD5SDL32XJAG42KH2U6OYK3KTS6Q, seg_pos={0 0}, upload=431917]: Done uploading piece. err: ecclient: upload failed (node:121scLvKEW9zw2aEz22zpremcjiop67eQ8v2uqpmaD5aZzCAP7b, address:145.79.227.39:4013): protocol: write tcp 159.223.143.39:60842->145.79.227.39:4013: use of closed network connection; write tcp 159.223.143.39:60842->145.79.227.39:4013: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:60842->145.79.227.39:4013: use of closed network connection Dec 22 22:31:35.398562305 (415h3m36.226270699s) [noise=true , address=145.79.227.32:4835 , piece_id=BRE3JH3XKPXB6FN7WB3W6ED7MOFES6HYLR34YGR6L5AX7RPIQ7IA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: ecclient: upload failed (node:16vVfeoqCkbMYCRjMMWaKBnCvQBBJPMgwQtWTq9HKQrD9SS17y, address:145.79.227.32:4835): protocol: write tcp 159.223.143.39:45098->145.79.227.32:4835: use of closed network connection; write tcp 159.223.143.39:45098->145.79.227.32:4835: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:45098->145.79.227.32:4835: use of closed network connection Dec 22 22:31:35.399183028 (415h3m36.226891422s) [noise=true , address=145.79.227.50:4373 , piece_id=4U55T5QDYCE6AZ3NCTPAVECPAELPQUKT3HV7GZL7JFRMWRN2AZUQ, seg_pos={0 0}, upload=431917]: Done uploading piece. err: ecclient: upload failed (node:12AQGSkJuNJDxoeqhChUenWxHwF5yK41UESCF63FQz3XvabXXfz, address:145.79.227.50:4373): protocol: read tcp 159.223.143.39:56028->145.79.227.50:4373: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 22 22:31:35.399278021 (415h3m36.226986417s) [noise=true , address=145.79.227.29:4943 , piece_id=G4TMZ6IAIZBDON3UEDPH4LJQFHV3JLALEXSJ3VI4XN6WHIZOB57Q, seg_pos={0 0}, upload=431917]: Done uploading piece. err: ecclient: upload failed (node:12fc5Trj8XxacZhzxdksUVT5CbG6uTmiwdUPjreYi4UMPY1TXWz, address:145.79.227.29:4943): protocol: write tcp 159.223.143.39:50820->145.79.227.29:4943: use of closed network connection; write tcp 159.223.143.39:50820->145.79.227.29:4943: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:50820->145.79.227.29:4943: use of closed network connection Dec 22 22:31:35.399379485 (415h3m36.227087866s) [seg_pos={0 0}, upload=431917]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 22 22:31:35.421122544 (415h3m36.248830935s) [upload=431917]: Done uploading segments... Dec 22 22:31:35.421174369 (415h3m36.248882769s) [upload=431917]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 490.349854ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [X................................................X..........] duration: 518.239569ms trace: https://jaeger.dev.storj.io/trace/7013ffb1a981d605 FAIL (2025-12-22 22:39:49 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 22 22:31:05.439510308 (415h3m6.2672187s) [upload=431917]: Starting upload Dec 22 22:31:05.439581993 (415h3m6.26729037s) [upload=431917]: Done starting upload Dec 22 22:31:05.439812792 (415h3m6.267521185s) [upload=431917]: Uploading segments... Dec 22 22:31:05.439942524 (415h3m6.267650918s) [upload=431917]: Got next segment. Inline: false Dec 22 22:31:05.440022744 (415h3m6.267731174s) [upload=431917]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 22 22:31:05.440029670 (415h3m6.267738072s) [upload=431917]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 22 22:31:05.467186372 (415h3m6.294894769s) [seg_pos={0 0}, upload=431917]: Begin upload segment... Dec 22 22:31:05.467265149 (415h3m6.294973535s) [seg_pos={0 0}, upload=431917]: Done begin upload segment. Dec 22 22:31:05.467296974 (415h3m6.295005373s) [noise=true , address=145.79.227.51:4409 , piece_id=ATH3F4DFOUUIFFJZRA2SSRRL7AKWZMQNQRYOZCVFFDLTFCFP236A, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.467389853 (415h3m6.295098247s) [noise=true , address=145.79.227.39:4013 , piece_id=JJVURGSLKY2UQ4FKNCLATA3ZBD5SDL32XJAG42KH2U6OYK3KTS6Q, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.467912104 (415h3m6.295620486s) [noise=true , address=145.79.227.36:4519 , piece_id=JUKRRIZWVGKEDD3GPJ7HMY2MEQZNMUCTKTXR6ZLSIERRONTZKHCQ, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.468324863 (415h3m6.296033263s) [noise=true , address=145.79.227.38:4979 , piece_id=HLMQC5OEB4OL4OZSRMCEVAWR5QM2T4MQUBPTHRVQLEAHPBWPQF6Q, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.468334764 (415h3m6.296043147s) [noise=true , address=145.79.227.40:4051 , piece_id=55RRAD7DQX6SR7RBLYAV7MD7XRWPNIWHTVVBQZ2ZL3OE2L7LRRIQ, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.468742525 (415h3m6.296450916s) [noise=true , address=45.91.116.4:4939 , piece_id=QYNKMKDLQJV7B6M2E34DU53Y7YI6VBCUW3C7ZLQOI4XWTBXZHKDQ, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.468915034 (415h3m6.296623414s) [noise=true , address=45.91.116.11:4117 , piece_id=WLMO5E3BIFFL3TOAUSSBGORIZ6DOCWFCMVT4AVFPPGPSKP2OKDAA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.469069393 (415h3m6.296777787s) [noise=true , address=145.79.227.32:4835 , piece_id=BRE3JH3XKPXB6FN7WB3W6ED7MOFES6HYLR34YGR6L5AX7RPIQ7IA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.469285789 (415h3m6.296994183s) [noise=true , address=145.79.227.29:4943 , piece_id=G4TMZ6IAIZBDON3UEDPH4LJQFHV3JLALEXSJ3VI4XN6WHIZOB57Q, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.469411962 (415h3m6.297120355s) [noise=true , address=145.79.227.48:4301 , piece_id=LBIYTJNOX5J4M3HDE7FGF5CSARBITW2AK3P2DSYFLFKUOIHZUKOQ, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.469749379 (415h3m6.297457773s) [noise=true , address=145.79.227.37:4565 , piece_id=554RHTYPG27UNCFPSRMOB245AX4QXL2QFIYECNTVWNNJMAL4BTYA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.470080828 (415h3m6.29778922s) [noise=true , address=145.79.227.43:4157 , piece_id=3W3LVTU37DNHWS52IGIIH7SXWXVNPGEY27RQNT272L3Z64GNR4KA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.470477660 (415h3m6.298186058s) [noise=true , address=45.91.116.14:4190 , piece_id=V4T6Z7K3C75LQEXNQIYZGYWW2CSQ7UPSET46QGDRT52RFGAROTTA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.470567977 (415h3m6.29827637s) [noise=true , address=145.79.227.15:4874 , piece_id=FNCCIMIVJTGZ2X5TOYOZH727JXHYM7HG3HBHBTZDAG67SHZLYYYA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.470980950 (415h3m6.298689349s) [noise=true , address=145.79.227.50:4373 , piece_id=4U55T5QDYCE6AZ3NCTPAVECPAELPQUKT3HV7GZL7JFRMWRN2AZUQ, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.471440098 (415h3m6.299148489s) [noise=true , address=145.79.227.47:4266 , piece_id=3NFYV2PWBNTAOSFHENM4J5PZ7XXE72UKTQVXLTVQRHGK4DUXAKDA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.560766475 (415h3m6.388474871s) [upload=431917]: Next returned nil segment Dec 22 22:31:05.560785161 (415h3m6.388493562s) [upload=431917]: Waiting for error group managing segments... Dec 22 22:31:05.566117878 (415h3m6.39382628s) [noise=true , address=45.91.116.4:4939 , piece_id=QYNKMKDLQJV7B6M2E34DU53Y7YI6VBCUW3C7ZLQOI4XWTBXZHKDQ, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.566723306 (415h3m6.394431687s) [noise=true , address=45.91.116.11:4117 , piece_id=WLMO5E3BIFFL3TOAUSSBGORIZ6DOCWFCMVT4AVFPPGPSKP2OKDAA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.566992667 (415h3m6.394701063s) [noise=true , address=145.79.227.36:4519 , piece_id=JUKRRIZWVGKEDD3GPJ7HMY2MEQZNMUCTKTXR6ZLSIERRONTZKHCQ, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.567464626 (415h3m6.395173019s) [noise=true , address=145.79.227.47:4266 , piece_id=3NFYV2PWBNTAOSFHENM4J5PZ7XXE72UKTQVXLTVQRHGK4DUXAKDA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.567924492 (415h3m6.395632883s) [noise=true , address=145.79.227.15:4874 , piece_id=FNCCIMIVJTGZ2X5TOYOZH727JXHYM7HG3HBHBTZDAG67SHZLYYYA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.568807294 (415h3m6.396515685s) [noise=true , address=145.79.227.51:4409 , piece_id=ATH3F4DFOUUIFFJZRA2SSRRL7AKWZMQNQRYOZCVFFDLTFCFP236A, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.570579823 (415h3m6.398288214s) [noise=true , address=145.79.227.48:4301 , piece_id=LBIYTJNOX5J4M3HDE7FGF5CSARBITW2AK3P2DSYFLFKUOIHZUKOQ, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.571632758 (415h3m6.399341149s) [noise=true , address=145.79.227.43:4157 , piece_id=3W3LVTU37DNHWS52IGIIH7SXWXVNPGEY27RQNT272L3Z64GNR4KA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.573168383 (415h3m6.400876774s) [noise=true , address=145.79.227.38:4979 , piece_id=HLMQC5OEB4OL4OZSRMCEVAWR5QM2T4MQUBPTHRVQLEAHPBWPQF6Q, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.579995420 (415h3m6.407703816s) [noise=true , address=45.91.116.14:4190 , piece_id=V4T6Z7K3C75LQEXNQIYZGYWW2CSQ7UPSET46QGDRT52RFGAROTTA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.592093172 (415h3m6.419801563s) [noise=true , address=145.79.227.37:4565 , piece_id=554RHTYPG27UNCFPSRMOB245AX4QXL2QFIYECNTVWNNJMAL4BTYA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:09.946500434 (415h3m10.774208816s) [noise=true , address=145.79.227.40:4051 , piece_id=55RRAD7DQX6SR7RBLYAV7MD7XRWPNIWHTVVBQZ2ZL3OE2L7LRRIQ, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:35.398235543 (415h3m36.225943943s) [noise=true , address=145.79.227.39:4013 , piece_id=JJVURGSLKY2UQ4FKNCLATA3ZBD5SDL32XJAG42KH2U6OYK3KTS6Q, seg_pos={0 0}, upload=431917]: Done uploading piece. err: ecclient: upload failed (node:121scLvKEW9zw2aEz22zpremcjiop67eQ8v2uqpmaD5aZzCAP7b, address:145.79.227.39:4013): protocol: write tcp 159.223.143.39:60842->145.79.227.39:4013: use of closed network connection; write tcp 159.223.143.39:60842->145.79.227.39:4013: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:60842->145.79.227.39:4013: use of closed network connection Dec 22 22:31:35.398562305 (415h3m36.226270699s) [noise=true , address=145.79.227.32:4835 , piece_id=BRE3JH3XKPXB6FN7WB3W6ED7MOFES6HYLR34YGR6L5AX7RPIQ7IA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: ecclient: upload failed (node:16vVfeoqCkbMYCRjMMWaKBnCvQBBJPMgwQtWTq9HKQrD9SS17y, address:145.79.227.32:4835): protocol: write tcp 159.223.143.39:45098->145.79.227.32:4835: use of closed network connection; write tcp 159.223.143.39:45098->145.79.227.32:4835: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:45098->145.79.227.32:4835: use of closed network connection Dec 22 22:31:35.399183028 (415h3m36.226891422s) [noise=true , address=145.79.227.50:4373 , piece_id=4U55T5QDYCE6AZ3NCTPAVECPAELPQUKT3HV7GZL7JFRMWRN2AZUQ, seg_pos={0 0}, upload=431917]: Done uploading piece. err: ecclient: upload failed (node:12AQGSkJuNJDxoeqhChUenWxHwF5yK41UESCF63FQz3XvabXXfz, address:145.79.227.50:4373): protocol: read tcp 159.223.143.39:56028->145.79.227.50:4373: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 22 22:31:35.399278021 (415h3m36.226986417s) [noise=true , address=145.79.227.29:4943 , piece_id=G4TMZ6IAIZBDON3UEDPH4LJQFHV3JLALEXSJ3VI4XN6WHIZOB57Q, seg_pos={0 0}, upload=431917]: Done uploading piece. err: ecclient: upload failed (node:12fc5Trj8XxacZhzxdksUVT5CbG6uTmiwdUPjreYi4UMPY1TXWz, address:145.79.227.29:4943): protocol: write tcp 159.223.143.39:50820->145.79.227.29:4943: use of closed network connection; write tcp 159.223.143.39:50820->145.79.227.29:4943: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:50820->145.79.227.29:4943: use of closed network connection Dec 22 22:31:35.399379485 (415h3m36.227087866s) [seg_pos={0 0}, upload=431917]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 22 22:31:35.421122544 (415h3m36.248830935s) [upload=431917]: Done uploading segments... Dec 22 22:31:35.421174369 (415h3m36.248882769s) [upload=431917]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 518.333905ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.X................................................X.........] duration: 441.56238ms trace: https://jaeger.dev.storj.io/trace/7013ffb1a981d605 FAIL (2025-12-22 22:39:25 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 22 22:31:05.439510308 (415h3m6.2672187s) [upload=431917]: Starting upload Dec 22 22:31:05.439581993 (415h3m6.26729037s) [upload=431917]: Done starting upload Dec 22 22:31:05.439812792 (415h3m6.267521185s) [upload=431917]: Uploading segments... Dec 22 22:31:05.439942524 (415h3m6.267650918s) [upload=431917]: Got next segment. Inline: false Dec 22 22:31:05.440022744 (415h3m6.267731174s) [upload=431917]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 22 22:31:05.440029670 (415h3m6.267738072s) [upload=431917]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 22 22:31:05.467186372 (415h3m6.294894769s) [seg_pos={0 0}, upload=431917]: Begin upload segment... Dec 22 22:31:05.467265149 (415h3m6.294973535s) [seg_pos={0 0}, upload=431917]: Done begin upload segment. Dec 22 22:31:05.467296974 (415h3m6.295005373s) [noise=true , address=145.79.227.51:4409 , piece_id=ATH3F4DFOUUIFFJZRA2SSRRL7AKWZMQNQRYOZCVFFDLTFCFP236A, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.467389853 (415h3m6.295098247s) [noise=true , address=145.79.227.39:4013 , piece_id=JJVURGSLKY2UQ4FKNCLATA3ZBD5SDL32XJAG42KH2U6OYK3KTS6Q, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.467912104 (415h3m6.295620486s) [noise=true , address=145.79.227.36:4519 , piece_id=JUKRRIZWVGKEDD3GPJ7HMY2MEQZNMUCTKTXR6ZLSIERRONTZKHCQ, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.468324863 (415h3m6.296033263s) [noise=true , address=145.79.227.38:4979 , piece_id=HLMQC5OEB4OL4OZSRMCEVAWR5QM2T4MQUBPTHRVQLEAHPBWPQF6Q, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.468334764 (415h3m6.296043147s) [noise=true , address=145.79.227.40:4051 , piece_id=55RRAD7DQX6SR7RBLYAV7MD7XRWPNIWHTVVBQZ2ZL3OE2L7LRRIQ, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.468742525 (415h3m6.296450916s) [noise=true , address=45.91.116.4:4939 , piece_id=QYNKMKDLQJV7B6M2E34DU53Y7YI6VBCUW3C7ZLQOI4XWTBXZHKDQ, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.468915034 (415h3m6.296623414s) [noise=true , address=45.91.116.11:4117 , piece_id=WLMO5E3BIFFL3TOAUSSBGORIZ6DOCWFCMVT4AVFPPGPSKP2OKDAA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.469069393 (415h3m6.296777787s) [noise=true , address=145.79.227.32:4835 , piece_id=BRE3JH3XKPXB6FN7WB3W6ED7MOFES6HYLR34YGR6L5AX7RPIQ7IA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.469285789 (415h3m6.296994183s) [noise=true , address=145.79.227.29:4943 , piece_id=G4TMZ6IAIZBDON3UEDPH4LJQFHV3JLALEXSJ3VI4XN6WHIZOB57Q, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.469411962 (415h3m6.297120355s) [noise=true , address=145.79.227.48:4301 , piece_id=LBIYTJNOX5J4M3HDE7FGF5CSARBITW2AK3P2DSYFLFKUOIHZUKOQ, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.469749379 (415h3m6.297457773s) [noise=true , address=145.79.227.37:4565 , piece_id=554RHTYPG27UNCFPSRMOB245AX4QXL2QFIYECNTVWNNJMAL4BTYA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.470080828 (415h3m6.29778922s) [noise=true , address=145.79.227.43:4157 , piece_id=3W3LVTU37DNHWS52IGIIH7SXWXVNPGEY27RQNT272L3Z64GNR4KA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.470477660 (415h3m6.298186058s) [noise=true , address=45.91.116.14:4190 , piece_id=V4T6Z7K3C75LQEXNQIYZGYWW2CSQ7UPSET46QGDRT52RFGAROTTA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.470567977 (415h3m6.29827637s) [noise=true , address=145.79.227.15:4874 , piece_id=FNCCIMIVJTGZ2X5TOYOZH727JXHYM7HG3HBHBTZDAG67SHZLYYYA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.470980950 (415h3m6.298689349s) [noise=true , address=145.79.227.50:4373 , piece_id=4U55T5QDYCE6AZ3NCTPAVECPAELPQUKT3HV7GZL7JFRMWRN2AZUQ, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.471440098 (415h3m6.299148489s) [noise=true , address=145.79.227.47:4266 , piece_id=3NFYV2PWBNTAOSFHENM4J5PZ7XXE72UKTQVXLTVQRHGK4DUXAKDA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.560766475 (415h3m6.388474871s) [upload=431917]: Next returned nil segment Dec 22 22:31:05.560785161 (415h3m6.388493562s) [upload=431917]: Waiting for error group managing segments... Dec 22 22:31:05.566117878 (415h3m6.39382628s) [noise=true , address=45.91.116.4:4939 , piece_id=QYNKMKDLQJV7B6M2E34DU53Y7YI6VBCUW3C7ZLQOI4XWTBXZHKDQ, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.566723306 (415h3m6.394431687s) [noise=true , address=45.91.116.11:4117 , piece_id=WLMO5E3BIFFL3TOAUSSBGORIZ6DOCWFCMVT4AVFPPGPSKP2OKDAA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.566992667 (415h3m6.394701063s) [noise=true , address=145.79.227.36:4519 , piece_id=JUKRRIZWVGKEDD3GPJ7HMY2MEQZNMUCTKTXR6ZLSIERRONTZKHCQ, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.567464626 (415h3m6.395173019s) [noise=true , address=145.79.227.47:4266 , piece_id=3NFYV2PWBNTAOSFHENM4J5PZ7XXE72UKTQVXLTVQRHGK4DUXAKDA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.567924492 (415h3m6.395632883s) [noise=true , address=145.79.227.15:4874 , piece_id=FNCCIMIVJTGZ2X5TOYOZH727JXHYM7HG3HBHBTZDAG67SHZLYYYA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.568807294 (415h3m6.396515685s) [noise=true , address=145.79.227.51:4409 , piece_id=ATH3F4DFOUUIFFJZRA2SSRRL7AKWZMQNQRYOZCVFFDLTFCFP236A, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.570579823 (415h3m6.398288214s) [noise=true , address=145.79.227.48:4301 , piece_id=LBIYTJNOX5J4M3HDE7FGF5CSARBITW2AK3P2DSYFLFKUOIHZUKOQ, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.571632758 (415h3m6.399341149s) [noise=true , address=145.79.227.43:4157 , piece_id=3W3LVTU37DNHWS52IGIIH7SXWXVNPGEY27RQNT272L3Z64GNR4KA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.573168383 (415h3m6.400876774s) [noise=true , address=145.79.227.38:4979 , piece_id=HLMQC5OEB4OL4OZSRMCEVAWR5QM2T4MQUBPTHRVQLEAHPBWPQF6Q, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.579995420 (415h3m6.407703816s) [noise=true , address=45.91.116.14:4190 , piece_id=V4T6Z7K3C75LQEXNQIYZGYWW2CSQ7UPSET46QGDRT52RFGAROTTA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.592093172 (415h3m6.419801563s) [noise=true , address=145.79.227.37:4565 , piece_id=554RHTYPG27UNCFPSRMOB245AX4QXL2QFIYECNTVWNNJMAL4BTYA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:09.946500434 (415h3m10.774208816s) [noise=true , address=145.79.227.40:4051 , piece_id=55RRAD7DQX6SR7RBLYAV7MD7XRWPNIWHTVVBQZ2ZL3OE2L7LRRIQ, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:35.398235543 (415h3m36.225943943s) [noise=true , address=145.79.227.39:4013 , piece_id=JJVURGSLKY2UQ4FKNCLATA3ZBD5SDL32XJAG42KH2U6OYK3KTS6Q, seg_pos={0 0}, upload=431917]: Done uploading piece. err: ecclient: upload failed (node:121scLvKEW9zw2aEz22zpremcjiop67eQ8v2uqpmaD5aZzCAP7b, address:145.79.227.39:4013): protocol: write tcp 159.223.143.39:60842->145.79.227.39:4013: use of closed network connection; write tcp 159.223.143.39:60842->145.79.227.39:4013: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:60842->145.79.227.39:4013: use of closed network connection Dec 22 22:31:35.398562305 (415h3m36.226270699s) [noise=true , address=145.79.227.32:4835 , piece_id=BRE3JH3XKPXB6FN7WB3W6ED7MOFES6HYLR34YGR6L5AX7RPIQ7IA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: ecclient: upload failed (node:16vVfeoqCkbMYCRjMMWaKBnCvQBBJPMgwQtWTq9HKQrD9SS17y, address:145.79.227.32:4835): protocol: write tcp 159.223.143.39:45098->145.79.227.32:4835: use of closed network connection; write tcp 159.223.143.39:45098->145.79.227.32:4835: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:45098->145.79.227.32:4835: use of closed network connection Dec 22 22:31:35.399183028 (415h3m36.226891422s) [noise=true , address=145.79.227.50:4373 , piece_id=4U55T5QDYCE6AZ3NCTPAVECPAELPQUKT3HV7GZL7JFRMWRN2AZUQ, seg_pos={0 0}, upload=431917]: Done uploading piece. err: ecclient: upload failed (node:12AQGSkJuNJDxoeqhChUenWxHwF5yK41UESCF63FQz3XvabXXfz, address:145.79.227.50:4373): protocol: read tcp 159.223.143.39:56028->145.79.227.50:4373: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 22 22:31:35.399278021 (415h3m36.226986417s) [noise=true , address=145.79.227.29:4943 , piece_id=G4TMZ6IAIZBDON3UEDPH4LJQFHV3JLALEXSJ3VI4XN6WHIZOB57Q, seg_pos={0 0}, upload=431917]: Done uploading piece. err: ecclient: upload failed (node:12fc5Trj8XxacZhzxdksUVT5CbG6uTmiwdUPjreYi4UMPY1TXWz, address:145.79.227.29:4943): protocol: write tcp 159.223.143.39:50820->145.79.227.29:4943: use of closed network connection; write tcp 159.223.143.39:50820->145.79.227.29:4943: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:50820->145.79.227.29:4943: use of closed network connection Dec 22 22:31:35.399379485 (415h3m36.227087866s) [seg_pos={0 0}, upload=431917]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 22 22:31:35.421122544 (415h3m36.248830935s) [upload=431917]: Done uploading segments... Dec 22 22:31:35.421174369 (415h3m36.248882769s) [upload=431917]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 441.61045ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..X................................................X........] duration: 449.735267ms trace: https://jaeger.dev.storj.io/trace/7013ffb1a981d605 FAIL (2025-12-22 22:38:20 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 22 22:31:05.439510308 (415h3m6.2672187s) [upload=431917]: Starting upload Dec 22 22:31:05.439581993 (415h3m6.26729037s) [upload=431917]: Done starting upload Dec 22 22:31:05.439812792 (415h3m6.267521185s) [upload=431917]: Uploading segments... Dec 22 22:31:05.439942524 (415h3m6.267650918s) [upload=431917]: Got next segment. Inline: false Dec 22 22:31:05.440022744 (415h3m6.267731174s) [upload=431917]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 22 22:31:05.440029670 (415h3m6.267738072s) [upload=431917]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 22 22:31:05.467186372 (415h3m6.294894769s) [seg_pos={0 0}, upload=431917]: Begin upload segment... Dec 22 22:31:05.467265149 (415h3m6.294973535s) [seg_pos={0 0}, upload=431917]: Done begin upload segment. Dec 22 22:31:05.467296974 (415h3m6.295005373s) [noise=true , address=145.79.227.51:4409 , piece_id=ATH3F4DFOUUIFFJZRA2SSRRL7AKWZMQNQRYOZCVFFDLTFCFP236A, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.467389853 (415h3m6.295098247s) [noise=true , address=145.79.227.39:4013 , piece_id=JJVURGSLKY2UQ4FKNCLATA3ZBD5SDL32XJAG42KH2U6OYK3KTS6Q, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.467912104 (415h3m6.295620486s) [noise=true , address=145.79.227.36:4519 , piece_id=JUKRRIZWVGKEDD3GPJ7HMY2MEQZNMUCTKTXR6ZLSIERRONTZKHCQ, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.468324863 (415h3m6.296033263s) [noise=true , address=145.79.227.38:4979 , piece_id=HLMQC5OEB4OL4OZSRMCEVAWR5QM2T4MQUBPTHRVQLEAHPBWPQF6Q, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.468334764 (415h3m6.296043147s) [noise=true , address=145.79.227.40:4051 , piece_id=55RRAD7DQX6SR7RBLYAV7MD7XRWPNIWHTVVBQZ2ZL3OE2L7LRRIQ, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.468742525 (415h3m6.296450916s) [noise=true , address=45.91.116.4:4939 , piece_id=QYNKMKDLQJV7B6M2E34DU53Y7YI6VBCUW3C7ZLQOI4XWTBXZHKDQ, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.468915034 (415h3m6.296623414s) [noise=true , address=45.91.116.11:4117 , piece_id=WLMO5E3BIFFL3TOAUSSBGORIZ6DOCWFCMVT4AVFPPGPSKP2OKDAA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.469069393 (415h3m6.296777787s) [noise=true , address=145.79.227.32:4835 , piece_id=BRE3JH3XKPXB6FN7WB3W6ED7MOFES6HYLR34YGR6L5AX7RPIQ7IA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.469285789 (415h3m6.296994183s) [noise=true , address=145.79.227.29:4943 , piece_id=G4TMZ6IAIZBDON3UEDPH4LJQFHV3JLALEXSJ3VI4XN6WHIZOB57Q, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.469411962 (415h3m6.297120355s) [noise=true , address=145.79.227.48:4301 , piece_id=LBIYTJNOX5J4M3HDE7FGF5CSARBITW2AK3P2DSYFLFKUOIHZUKOQ, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.469749379 (415h3m6.297457773s) [noise=true , address=145.79.227.37:4565 , piece_id=554RHTYPG27UNCFPSRMOB245AX4QXL2QFIYECNTVWNNJMAL4BTYA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.470080828 (415h3m6.29778922s) [noise=true , address=145.79.227.43:4157 , piece_id=3W3LVTU37DNHWS52IGIIH7SXWXVNPGEY27RQNT272L3Z64GNR4KA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.470477660 (415h3m6.298186058s) [noise=true , address=45.91.116.14:4190 , piece_id=V4T6Z7K3C75LQEXNQIYZGYWW2CSQ7UPSET46QGDRT52RFGAROTTA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.470567977 (415h3m6.29827637s) [noise=true , address=145.79.227.15:4874 , piece_id=FNCCIMIVJTGZ2X5TOYOZH727JXHYM7HG3HBHBTZDAG67SHZLYYYA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.470980950 (415h3m6.298689349s) [noise=true , address=145.79.227.50:4373 , piece_id=4U55T5QDYCE6AZ3NCTPAVECPAELPQUKT3HV7GZL7JFRMWRN2AZUQ, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.471440098 (415h3m6.299148489s) [noise=true , address=145.79.227.47:4266 , piece_id=3NFYV2PWBNTAOSFHENM4J5PZ7XXE72UKTQVXLTVQRHGK4DUXAKDA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.560766475 (415h3m6.388474871s) [upload=431917]: Next returned nil segment Dec 22 22:31:05.560785161 (415h3m6.388493562s) [upload=431917]: Waiting for error group managing segments... Dec 22 22:31:05.566117878 (415h3m6.39382628s) [noise=true , address=45.91.116.4:4939 , piece_id=QYNKMKDLQJV7B6M2E34DU53Y7YI6VBCUW3C7ZLQOI4XWTBXZHKDQ, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.566723306 (415h3m6.394431687s) [noise=true , address=45.91.116.11:4117 , piece_id=WLMO5E3BIFFL3TOAUSSBGORIZ6DOCWFCMVT4AVFPPGPSKP2OKDAA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.566992667 (415h3m6.394701063s) [noise=true , address=145.79.227.36:4519 , piece_id=JUKRRIZWVGKEDD3GPJ7HMY2MEQZNMUCTKTXR6ZLSIERRONTZKHCQ, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.567464626 (415h3m6.395173019s) [noise=true , address=145.79.227.47:4266 , piece_id=3NFYV2PWBNTAOSFHENM4J5PZ7XXE72UKTQVXLTVQRHGK4DUXAKDA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.567924492 (415h3m6.395632883s) [noise=true , address=145.79.227.15:4874 , piece_id=FNCCIMIVJTGZ2X5TOYOZH727JXHYM7HG3HBHBTZDAG67SHZLYYYA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.568807294 (415h3m6.396515685s) [noise=true , address=145.79.227.51:4409 , piece_id=ATH3F4DFOUUIFFJZRA2SSRRL7AKWZMQNQRYOZCVFFDLTFCFP236A, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.570579823 (415h3m6.398288214s) [noise=true , address=145.79.227.48:4301 , piece_id=LBIYTJNOX5J4M3HDE7FGF5CSARBITW2AK3P2DSYFLFKUOIHZUKOQ, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.571632758 (415h3m6.399341149s) [noise=true , address=145.79.227.43:4157 , piece_id=3W3LVTU37DNHWS52IGIIH7SXWXVNPGEY27RQNT272L3Z64GNR4KA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.573168383 (415h3m6.400876774s) [noise=true , address=145.79.227.38:4979 , piece_id=HLMQC5OEB4OL4OZSRMCEVAWR5QM2T4MQUBPTHRVQLEAHPBWPQF6Q, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.579995420 (415h3m6.407703816s) [noise=true , address=45.91.116.14:4190 , piece_id=V4T6Z7K3C75LQEXNQIYZGYWW2CSQ7UPSET46QGDRT52RFGAROTTA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.592093172 (415h3m6.419801563s) [noise=true , address=145.79.227.37:4565 , piece_id=554RHTYPG27UNCFPSRMOB245AX4QXL2QFIYECNTVWNNJMAL4BTYA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:09.946500434 (415h3m10.774208816s) [noise=true , address=145.79.227.40:4051 , piece_id=55RRAD7DQX6SR7RBLYAV7MD7XRWPNIWHTVVBQZ2ZL3OE2L7LRRIQ, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:35.398235543 (415h3m36.225943943s) [noise=true , address=145.79.227.39:4013 , piece_id=JJVURGSLKY2UQ4FKNCLATA3ZBD5SDL32XJAG42KH2U6OYK3KTS6Q, seg_pos={0 0}, upload=431917]: Done uploading piece. err: ecclient: upload failed (node:121scLvKEW9zw2aEz22zpremcjiop67eQ8v2uqpmaD5aZzCAP7b, address:145.79.227.39:4013): protocol: write tcp 159.223.143.39:60842->145.79.227.39:4013: use of closed network connection; write tcp 159.223.143.39:60842->145.79.227.39:4013: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:60842->145.79.227.39:4013: use of closed network connection Dec 22 22:31:35.398562305 (415h3m36.226270699s) [noise=true , address=145.79.227.32:4835 , piece_id=BRE3JH3XKPXB6FN7WB3W6ED7MOFES6HYLR34YGR6L5AX7RPIQ7IA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: ecclient: upload failed (node:16vVfeoqCkbMYCRjMMWaKBnCvQBBJPMgwQtWTq9HKQrD9SS17y, address:145.79.227.32:4835): protocol: write tcp 159.223.143.39:45098->145.79.227.32:4835: use of closed network connection; write tcp 159.223.143.39:45098->145.79.227.32:4835: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:45098->145.79.227.32:4835: use of closed network connection Dec 22 22:31:35.399183028 (415h3m36.226891422s) [noise=true , address=145.79.227.50:4373 , piece_id=4U55T5QDYCE6AZ3NCTPAVECPAELPQUKT3HV7GZL7JFRMWRN2AZUQ, seg_pos={0 0}, upload=431917]: Done uploading piece. err: ecclient: upload failed (node:12AQGSkJuNJDxoeqhChUenWxHwF5yK41UESCF63FQz3XvabXXfz, address:145.79.227.50:4373): protocol: read tcp 159.223.143.39:56028->145.79.227.50:4373: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 22 22:31:35.399278021 (415h3m36.226986417s) [noise=true , address=145.79.227.29:4943 , piece_id=G4TMZ6IAIZBDON3UEDPH4LJQFHV3JLALEXSJ3VI4XN6WHIZOB57Q, seg_pos={0 0}, upload=431917]: Done uploading piece. err: ecclient: upload failed (node:12fc5Trj8XxacZhzxdksUVT5CbG6uTmiwdUPjreYi4UMPY1TXWz, address:145.79.227.29:4943): protocol: write tcp 159.223.143.39:50820->145.79.227.29:4943: use of closed network connection; write tcp 159.223.143.39:50820->145.79.227.29:4943: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:50820->145.79.227.29:4943: use of closed network connection Dec 22 22:31:35.399379485 (415h3m36.227087866s) [seg_pos={0 0}, upload=431917]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 22 22:31:35.421122544 (415h3m36.248830935s) [upload=431917]: Done uploading segments... Dec 22 22:31:35.421174369 (415h3m36.248882769s) [upload=431917]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 449.817381ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...X................................................X.......] duration: 547.528381ms trace: https://jaeger.dev.storj.io/trace/7013ffb1a981d605 FAIL (2025-12-22 22:37:18 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 22 22:31:05.439510308 (415h3m6.2672187s) [upload=431917]: Starting upload Dec 22 22:31:05.439581993 (415h3m6.26729037s) [upload=431917]: Done starting upload Dec 22 22:31:05.439812792 (415h3m6.267521185s) [upload=431917]: Uploading segments... Dec 22 22:31:05.439942524 (415h3m6.267650918s) [upload=431917]: Got next segment. Inline: false Dec 22 22:31:05.440022744 (415h3m6.267731174s) [upload=431917]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 22 22:31:05.440029670 (415h3m6.267738072s) [upload=431917]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 22 22:31:05.467186372 (415h3m6.294894769s) [seg_pos={0 0}, upload=431917]: Begin upload segment... Dec 22 22:31:05.467265149 (415h3m6.294973535s) [seg_pos={0 0}, upload=431917]: Done begin upload segment. Dec 22 22:31:05.467296974 (415h3m6.295005373s) [noise=true , address=145.79.227.51:4409 , piece_id=ATH3F4DFOUUIFFJZRA2SSRRL7AKWZMQNQRYOZCVFFDLTFCFP236A, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.467389853 (415h3m6.295098247s) [noise=true , address=145.79.227.39:4013 , piece_id=JJVURGSLKY2UQ4FKNCLATA3ZBD5SDL32XJAG42KH2U6OYK3KTS6Q, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.467912104 (415h3m6.295620486s) [noise=true , address=145.79.227.36:4519 , piece_id=JUKRRIZWVGKEDD3GPJ7HMY2MEQZNMUCTKTXR6ZLSIERRONTZKHCQ, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.468324863 (415h3m6.296033263s) [noise=true , address=145.79.227.38:4979 , piece_id=HLMQC5OEB4OL4OZSRMCEVAWR5QM2T4MQUBPTHRVQLEAHPBWPQF6Q, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.468334764 (415h3m6.296043147s) [noise=true , address=145.79.227.40:4051 , piece_id=55RRAD7DQX6SR7RBLYAV7MD7XRWPNIWHTVVBQZ2ZL3OE2L7LRRIQ, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.468742525 (415h3m6.296450916s) [noise=true , address=45.91.116.4:4939 , piece_id=QYNKMKDLQJV7B6M2E34DU53Y7YI6VBCUW3C7ZLQOI4XWTBXZHKDQ, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.468915034 (415h3m6.296623414s) [noise=true , address=45.91.116.11:4117 , piece_id=WLMO5E3BIFFL3TOAUSSBGORIZ6DOCWFCMVT4AVFPPGPSKP2OKDAA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.469069393 (415h3m6.296777787s) [noise=true , address=145.79.227.32:4835 , piece_id=BRE3JH3XKPXB6FN7WB3W6ED7MOFES6HYLR34YGR6L5AX7RPIQ7IA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.469285789 (415h3m6.296994183s) [noise=true , address=145.79.227.29:4943 , piece_id=G4TMZ6IAIZBDON3UEDPH4LJQFHV3JLALEXSJ3VI4XN6WHIZOB57Q, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.469411962 (415h3m6.297120355s) [noise=true , address=145.79.227.48:4301 , piece_id=LBIYTJNOX5J4M3HDE7FGF5CSARBITW2AK3P2DSYFLFKUOIHZUKOQ, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.469749379 (415h3m6.297457773s) [noise=true , address=145.79.227.37:4565 , piece_id=554RHTYPG27UNCFPSRMOB245AX4QXL2QFIYECNTVWNNJMAL4BTYA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.470080828 (415h3m6.29778922s) [noise=true , address=145.79.227.43:4157 , piece_id=3W3LVTU37DNHWS52IGIIH7SXWXVNPGEY27RQNT272L3Z64GNR4KA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.470477660 (415h3m6.298186058s) [noise=true , address=45.91.116.14:4190 , piece_id=V4T6Z7K3C75LQEXNQIYZGYWW2CSQ7UPSET46QGDRT52RFGAROTTA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.470567977 (415h3m6.29827637s) [noise=true , address=145.79.227.15:4874 , piece_id=FNCCIMIVJTGZ2X5TOYOZH727JXHYM7HG3HBHBTZDAG67SHZLYYYA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.470980950 (415h3m6.298689349s) [noise=true , address=145.79.227.50:4373 , piece_id=4U55T5QDYCE6AZ3NCTPAVECPAELPQUKT3HV7GZL7JFRMWRN2AZUQ, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.471440098 (415h3m6.299148489s) [noise=true , address=145.79.227.47:4266 , piece_id=3NFYV2PWBNTAOSFHENM4J5PZ7XXE72UKTQVXLTVQRHGK4DUXAKDA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.560766475 (415h3m6.388474871s) [upload=431917]: Next returned nil segment Dec 22 22:31:05.560785161 (415h3m6.388493562s) [upload=431917]: Waiting for error group managing segments... Dec 22 22:31:05.566117878 (415h3m6.39382628s) [noise=true , address=45.91.116.4:4939 , piece_id=QYNKMKDLQJV7B6M2E34DU53Y7YI6VBCUW3C7ZLQOI4XWTBXZHKDQ, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.566723306 (415h3m6.394431687s) [noise=true , address=45.91.116.11:4117 , piece_id=WLMO5E3BIFFL3TOAUSSBGORIZ6DOCWFCMVT4AVFPPGPSKP2OKDAA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.566992667 (415h3m6.394701063s) [noise=true , address=145.79.227.36:4519 , piece_id=JUKRRIZWVGKEDD3GPJ7HMY2MEQZNMUCTKTXR6ZLSIERRONTZKHCQ, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.567464626 (415h3m6.395173019s) [noise=true , address=145.79.227.47:4266 , piece_id=3NFYV2PWBNTAOSFHENM4J5PZ7XXE72UKTQVXLTVQRHGK4DUXAKDA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.567924492 (415h3m6.395632883s) [noise=true , address=145.79.227.15:4874 , piece_id=FNCCIMIVJTGZ2X5TOYOZH727JXHYM7HG3HBHBTZDAG67SHZLYYYA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.568807294 (415h3m6.396515685s) [noise=true , address=145.79.227.51:4409 , piece_id=ATH3F4DFOUUIFFJZRA2SSRRL7AKWZMQNQRYOZCVFFDLTFCFP236A, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.570579823 (415h3m6.398288214s) [noise=true , address=145.79.227.48:4301 , piece_id=LBIYTJNOX5J4M3HDE7FGF5CSARBITW2AK3P2DSYFLFKUOIHZUKOQ, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.571632758 (415h3m6.399341149s) [noise=true , address=145.79.227.43:4157 , piece_id=3W3LVTU37DNHWS52IGIIH7SXWXVNPGEY27RQNT272L3Z64GNR4KA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.573168383 (415h3m6.400876774s) [noise=true , address=145.79.227.38:4979 , piece_id=HLMQC5OEB4OL4OZSRMCEVAWR5QM2T4MQUBPTHRVQLEAHPBWPQF6Q, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.579995420 (415h3m6.407703816s) [noise=true , address=45.91.116.14:4190 , piece_id=V4T6Z7K3C75LQEXNQIYZGYWW2CSQ7UPSET46QGDRT52RFGAROTTA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.592093172 (415h3m6.419801563s) [noise=true , address=145.79.227.37:4565 , piece_id=554RHTYPG27UNCFPSRMOB245AX4QXL2QFIYECNTVWNNJMAL4BTYA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:09.946500434 (415h3m10.774208816s) [noise=true , address=145.79.227.40:4051 , piece_id=55RRAD7DQX6SR7RBLYAV7MD7XRWPNIWHTVVBQZ2ZL3OE2L7LRRIQ, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:35.398235543 (415h3m36.225943943s) [noise=true , address=145.79.227.39:4013 , piece_id=JJVURGSLKY2UQ4FKNCLATA3ZBD5SDL32XJAG42KH2U6OYK3KTS6Q, seg_pos={0 0}, upload=431917]: Done uploading piece. err: ecclient: upload failed (node:121scLvKEW9zw2aEz22zpremcjiop67eQ8v2uqpmaD5aZzCAP7b, address:145.79.227.39:4013): protocol: write tcp 159.223.143.39:60842->145.79.227.39:4013: use of closed network connection; write tcp 159.223.143.39:60842->145.79.227.39:4013: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:60842->145.79.227.39:4013: use of closed network connection Dec 22 22:31:35.398562305 (415h3m36.226270699s) [noise=true , address=145.79.227.32:4835 , piece_id=BRE3JH3XKPXB6FN7WB3W6ED7MOFES6HYLR34YGR6L5AX7RPIQ7IA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: ecclient: upload failed (node:16vVfeoqCkbMYCRjMMWaKBnCvQBBJPMgwQtWTq9HKQrD9SS17y, address:145.79.227.32:4835): protocol: write tcp 159.223.143.39:45098->145.79.227.32:4835: use of closed network connection; write tcp 159.223.143.39:45098->145.79.227.32:4835: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:45098->145.79.227.32:4835: use of closed network connection Dec 22 22:31:35.399183028 (415h3m36.226891422s) [noise=true , address=145.79.227.50:4373 , piece_id=4U55T5QDYCE6AZ3NCTPAVECPAELPQUKT3HV7GZL7JFRMWRN2AZUQ, seg_pos={0 0}, upload=431917]: Done uploading piece. err: ecclient: upload failed (node:12AQGSkJuNJDxoeqhChUenWxHwF5yK41UESCF63FQz3XvabXXfz, address:145.79.227.50:4373): protocol: read tcp 159.223.143.39:56028->145.79.227.50:4373: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 22 22:31:35.399278021 (415h3m36.226986417s) [noise=true , address=145.79.227.29:4943 , piece_id=G4TMZ6IAIZBDON3UEDPH4LJQFHV3JLALEXSJ3VI4XN6WHIZOB57Q, seg_pos={0 0}, upload=431917]: Done uploading piece. err: ecclient: upload failed (node:12fc5Trj8XxacZhzxdksUVT5CbG6uTmiwdUPjreYi4UMPY1TXWz, address:145.79.227.29:4943): protocol: write tcp 159.223.143.39:50820->145.79.227.29:4943: use of closed network connection; write tcp 159.223.143.39:50820->145.79.227.29:4943: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:50820->145.79.227.29:4943: use of closed network connection Dec 22 22:31:35.399379485 (415h3m36.227087866s) [seg_pos={0 0}, upload=431917]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 22 22:31:35.421122544 (415h3m36.248830935s) [upload=431917]: Done uploading segments... Dec 22 22:31:35.421174369 (415h3m36.248882769s) [upload=431917]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 547.552722ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [....X................................................X......] duration: 444.394437ms trace: https://jaeger.dev.storj.io/trace/7013ffb1a981d605 FAIL (2025-12-22 22:36:26 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 22 22:31:05.439510308 (415h3m6.2672187s) [upload=431917]: Starting upload Dec 22 22:31:05.439581993 (415h3m6.26729037s) [upload=431917]: Done starting upload Dec 22 22:31:05.439812792 (415h3m6.267521185s) [upload=431917]: Uploading segments... Dec 22 22:31:05.439942524 (415h3m6.267650918s) [upload=431917]: Got next segment. Inline: false Dec 22 22:31:05.440022744 (415h3m6.267731174s) [upload=431917]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 22 22:31:05.440029670 (415h3m6.267738072s) [upload=431917]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 22 22:31:05.467186372 (415h3m6.294894769s) [seg_pos={0 0}, upload=431917]: Begin upload segment... Dec 22 22:31:05.467265149 (415h3m6.294973535s) [seg_pos={0 0}, upload=431917]: Done begin upload segment. Dec 22 22:31:05.467296974 (415h3m6.295005373s) [noise=true , address=145.79.227.51:4409 , piece_id=ATH3F4DFOUUIFFJZRA2SSRRL7AKWZMQNQRYOZCVFFDLTFCFP236A, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.467389853 (415h3m6.295098247s) [noise=true , address=145.79.227.39:4013 , piece_id=JJVURGSLKY2UQ4FKNCLATA3ZBD5SDL32XJAG42KH2U6OYK3KTS6Q, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.467912104 (415h3m6.295620486s) [noise=true , address=145.79.227.36:4519 , piece_id=JUKRRIZWVGKEDD3GPJ7HMY2MEQZNMUCTKTXR6ZLSIERRONTZKHCQ, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.468324863 (415h3m6.296033263s) [noise=true , address=145.79.227.38:4979 , piece_id=HLMQC5OEB4OL4OZSRMCEVAWR5QM2T4MQUBPTHRVQLEAHPBWPQF6Q, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.468334764 (415h3m6.296043147s) [noise=true , address=145.79.227.40:4051 , piece_id=55RRAD7DQX6SR7RBLYAV7MD7XRWPNIWHTVVBQZ2ZL3OE2L7LRRIQ, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.468742525 (415h3m6.296450916s) [noise=true , address=45.91.116.4:4939 , piece_id=QYNKMKDLQJV7B6M2E34DU53Y7YI6VBCUW3C7ZLQOI4XWTBXZHKDQ, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.468915034 (415h3m6.296623414s) [noise=true , address=45.91.116.11:4117 , piece_id=WLMO5E3BIFFL3TOAUSSBGORIZ6DOCWFCMVT4AVFPPGPSKP2OKDAA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.469069393 (415h3m6.296777787s) [noise=true , address=145.79.227.32:4835 , piece_id=BRE3JH3XKPXB6FN7WB3W6ED7MOFES6HYLR34YGR6L5AX7RPIQ7IA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.469285789 (415h3m6.296994183s) [noise=true , address=145.79.227.29:4943 , piece_id=G4TMZ6IAIZBDON3UEDPH4LJQFHV3JLALEXSJ3VI4XN6WHIZOB57Q, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.469411962 (415h3m6.297120355s) [noise=true , address=145.79.227.48:4301 , piece_id=LBIYTJNOX5J4M3HDE7FGF5CSARBITW2AK3P2DSYFLFKUOIHZUKOQ, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.469749379 (415h3m6.297457773s) [noise=true , address=145.79.227.37:4565 , piece_id=554RHTYPG27UNCFPSRMOB245AX4QXL2QFIYECNTVWNNJMAL4BTYA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.470080828 (415h3m6.29778922s) [noise=true , address=145.79.227.43:4157 , piece_id=3W3LVTU37DNHWS52IGIIH7SXWXVNPGEY27RQNT272L3Z64GNR4KA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.470477660 (415h3m6.298186058s) [noise=true , address=45.91.116.14:4190 , piece_id=V4T6Z7K3C75LQEXNQIYZGYWW2CSQ7UPSET46QGDRT52RFGAROTTA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.470567977 (415h3m6.29827637s) [noise=true , address=145.79.227.15:4874 , piece_id=FNCCIMIVJTGZ2X5TOYOZH727JXHYM7HG3HBHBTZDAG67SHZLYYYA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.470980950 (415h3m6.298689349s) [noise=true , address=145.79.227.50:4373 , piece_id=4U55T5QDYCE6AZ3NCTPAVECPAELPQUKT3HV7GZL7JFRMWRN2AZUQ, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.471440098 (415h3m6.299148489s) [noise=true , address=145.79.227.47:4266 , piece_id=3NFYV2PWBNTAOSFHENM4J5PZ7XXE72UKTQVXLTVQRHGK4DUXAKDA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.560766475 (415h3m6.388474871s) [upload=431917]: Next returned nil segment Dec 22 22:31:05.560785161 (415h3m6.388493562s) [upload=431917]: Waiting for error group managing segments... Dec 22 22:31:05.566117878 (415h3m6.39382628s) [noise=true , address=45.91.116.4:4939 , piece_id=QYNKMKDLQJV7B6M2E34DU53Y7YI6VBCUW3C7ZLQOI4XWTBXZHKDQ, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.566723306 (415h3m6.394431687s) [noise=true , address=45.91.116.11:4117 , piece_id=WLMO5E3BIFFL3TOAUSSBGORIZ6DOCWFCMVT4AVFPPGPSKP2OKDAA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.566992667 (415h3m6.394701063s) [noise=true , address=145.79.227.36:4519 , piece_id=JUKRRIZWVGKEDD3GPJ7HMY2MEQZNMUCTKTXR6ZLSIERRONTZKHCQ, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.567464626 (415h3m6.395173019s) [noise=true , address=145.79.227.47:4266 , piece_id=3NFYV2PWBNTAOSFHENM4J5PZ7XXE72UKTQVXLTVQRHGK4DUXAKDA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.567924492 (415h3m6.395632883s) [noise=true , address=145.79.227.15:4874 , piece_id=FNCCIMIVJTGZ2X5TOYOZH727JXHYM7HG3HBHBTZDAG67SHZLYYYA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.568807294 (415h3m6.396515685s) [noise=true , address=145.79.227.51:4409 , piece_id=ATH3F4DFOUUIFFJZRA2SSRRL7AKWZMQNQRYOZCVFFDLTFCFP236A, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.570579823 (415h3m6.398288214s) [noise=true , address=145.79.227.48:4301 , piece_id=LBIYTJNOX5J4M3HDE7FGF5CSARBITW2AK3P2DSYFLFKUOIHZUKOQ, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.571632758 (415h3m6.399341149s) [noise=true , address=145.79.227.43:4157 , piece_id=3W3LVTU37DNHWS52IGIIH7SXWXVNPGEY27RQNT272L3Z64GNR4KA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.573168383 (415h3m6.400876774s) [noise=true , address=145.79.227.38:4979 , piece_id=HLMQC5OEB4OL4OZSRMCEVAWR5QM2T4MQUBPTHRVQLEAHPBWPQF6Q, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.579995420 (415h3m6.407703816s) [noise=true , address=45.91.116.14:4190 , piece_id=V4T6Z7K3C75LQEXNQIYZGYWW2CSQ7UPSET46QGDRT52RFGAROTTA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.592093172 (415h3m6.419801563s) [noise=true , address=145.79.227.37:4565 , piece_id=554RHTYPG27UNCFPSRMOB245AX4QXL2QFIYECNTVWNNJMAL4BTYA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:09.946500434 (415h3m10.774208816s) [noise=true , address=145.79.227.40:4051 , piece_id=55RRAD7DQX6SR7RBLYAV7MD7XRWPNIWHTVVBQZ2ZL3OE2L7LRRIQ, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:35.398235543 (415h3m36.225943943s) [noise=true , address=145.79.227.39:4013 , piece_id=JJVURGSLKY2UQ4FKNCLATA3ZBD5SDL32XJAG42KH2U6OYK3KTS6Q, seg_pos={0 0}, upload=431917]: Done uploading piece. err: ecclient: upload failed (node:121scLvKEW9zw2aEz22zpremcjiop67eQ8v2uqpmaD5aZzCAP7b, address:145.79.227.39:4013): protocol: write tcp 159.223.143.39:60842->145.79.227.39:4013: use of closed network connection; write tcp 159.223.143.39:60842->145.79.227.39:4013: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:60842->145.79.227.39:4013: use of closed network connection Dec 22 22:31:35.398562305 (415h3m36.226270699s) [noise=true , address=145.79.227.32:4835 , piece_id=BRE3JH3XKPXB6FN7WB3W6ED7MOFES6HYLR34YGR6L5AX7RPIQ7IA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: ecclient: upload failed (node:16vVfeoqCkbMYCRjMMWaKBnCvQBBJPMgwQtWTq9HKQrD9SS17y, address:145.79.227.32:4835): protocol: write tcp 159.223.143.39:45098->145.79.227.32:4835: use of closed network connection; write tcp 159.223.143.39:45098->145.79.227.32:4835: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:45098->145.79.227.32:4835: use of closed network connection Dec 22 22:31:35.399183028 (415h3m36.226891422s) [noise=true , address=145.79.227.50:4373 , piece_id=4U55T5QDYCE6AZ3NCTPAVECPAELPQUKT3HV7GZL7JFRMWRN2AZUQ, seg_pos={0 0}, upload=431917]: Done uploading piece. err: ecclient: upload failed (node:12AQGSkJuNJDxoeqhChUenWxHwF5yK41UESCF63FQz3XvabXXfz, address:145.79.227.50:4373): protocol: read tcp 159.223.143.39:56028->145.79.227.50:4373: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 22 22:31:35.399278021 (415h3m36.226986417s) [noise=true , address=145.79.227.29:4943 , piece_id=G4TMZ6IAIZBDON3UEDPH4LJQFHV3JLALEXSJ3VI4XN6WHIZOB57Q, seg_pos={0 0}, upload=431917]: Done uploading piece. err: ecclient: upload failed (node:12fc5Trj8XxacZhzxdksUVT5CbG6uTmiwdUPjreYi4UMPY1TXWz, address:145.79.227.29:4943): protocol: write tcp 159.223.143.39:50820->145.79.227.29:4943: use of closed network connection; write tcp 159.223.143.39:50820->145.79.227.29:4943: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:50820->145.79.227.29:4943: use of closed network connection Dec 22 22:31:35.399379485 (415h3m36.227087866s) [seg_pos={0 0}, upload=431917]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 22 22:31:35.421122544 (415h3m36.248830935s) [upload=431917]: Done uploading segments... Dec 22 22:31:35.421174369 (415h3m36.248882769s) [upload=431917]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 444.426646ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.....X................................................X.....] duration: 431.709864ms trace: https://jaeger.dev.storj.io/trace/7013ffb1a981d605 FAIL (2025-12-22 22:35:21 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 22 22:31:05.439510308 (415h3m6.2672187s) [upload=431917]: Starting upload Dec 22 22:31:05.439581993 (415h3m6.26729037s) [upload=431917]: Done starting upload Dec 22 22:31:05.439812792 (415h3m6.267521185s) [upload=431917]: Uploading segments... Dec 22 22:31:05.439942524 (415h3m6.267650918s) [upload=431917]: Got next segment. Inline: false Dec 22 22:31:05.440022744 (415h3m6.267731174s) [upload=431917]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 22 22:31:05.440029670 (415h3m6.267738072s) [upload=431917]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 22 22:31:05.467186372 (415h3m6.294894769s) [seg_pos={0 0}, upload=431917]: Begin upload segment... Dec 22 22:31:05.467265149 (415h3m6.294973535s) [seg_pos={0 0}, upload=431917]: Done begin upload segment. Dec 22 22:31:05.467296974 (415h3m6.295005373s) [noise=true , address=145.79.227.51:4409 , piece_id=ATH3F4DFOUUIFFJZRA2SSRRL7AKWZMQNQRYOZCVFFDLTFCFP236A, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.467389853 (415h3m6.295098247s) [noise=true , address=145.79.227.39:4013 , piece_id=JJVURGSLKY2UQ4FKNCLATA3ZBD5SDL32XJAG42KH2U6OYK3KTS6Q, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.467912104 (415h3m6.295620486s) [noise=true , address=145.79.227.36:4519 , piece_id=JUKRRIZWVGKEDD3GPJ7HMY2MEQZNMUCTKTXR6ZLSIERRONTZKHCQ, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.468324863 (415h3m6.296033263s) [noise=true , address=145.79.227.38:4979 , piece_id=HLMQC5OEB4OL4OZSRMCEVAWR5QM2T4MQUBPTHRVQLEAHPBWPQF6Q, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.468334764 (415h3m6.296043147s) [noise=true , address=145.79.227.40:4051 , piece_id=55RRAD7DQX6SR7RBLYAV7MD7XRWPNIWHTVVBQZ2ZL3OE2L7LRRIQ, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.468742525 (415h3m6.296450916s) [noise=true , address=45.91.116.4:4939 , piece_id=QYNKMKDLQJV7B6M2E34DU53Y7YI6VBCUW3C7ZLQOI4XWTBXZHKDQ, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.468915034 (415h3m6.296623414s) [noise=true , address=45.91.116.11:4117 , piece_id=WLMO5E3BIFFL3TOAUSSBGORIZ6DOCWFCMVT4AVFPPGPSKP2OKDAA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.469069393 (415h3m6.296777787s) [noise=true , address=145.79.227.32:4835 , piece_id=BRE3JH3XKPXB6FN7WB3W6ED7MOFES6HYLR34YGR6L5AX7RPIQ7IA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.469285789 (415h3m6.296994183s) [noise=true , address=145.79.227.29:4943 , piece_id=G4TMZ6IAIZBDON3UEDPH4LJQFHV3JLALEXSJ3VI4XN6WHIZOB57Q, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.469411962 (415h3m6.297120355s) [noise=true , address=145.79.227.48:4301 , piece_id=LBIYTJNOX5J4M3HDE7FGF5CSARBITW2AK3P2DSYFLFKUOIHZUKOQ, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.469749379 (415h3m6.297457773s) [noise=true , address=145.79.227.37:4565 , piece_id=554RHTYPG27UNCFPSRMOB245AX4QXL2QFIYECNTVWNNJMAL4BTYA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.470080828 (415h3m6.29778922s) [noise=true , address=145.79.227.43:4157 , piece_id=3W3LVTU37DNHWS52IGIIH7SXWXVNPGEY27RQNT272L3Z64GNR4KA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.470477660 (415h3m6.298186058s) [noise=true , address=45.91.116.14:4190 , piece_id=V4T6Z7K3C75LQEXNQIYZGYWW2CSQ7UPSET46QGDRT52RFGAROTTA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.470567977 (415h3m6.29827637s) [noise=true , address=145.79.227.15:4874 , piece_id=FNCCIMIVJTGZ2X5TOYOZH727JXHYM7HG3HBHBTZDAG67SHZLYYYA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.470980950 (415h3m6.298689349s) [noise=true , address=145.79.227.50:4373 , piece_id=4U55T5QDYCE6AZ3NCTPAVECPAELPQUKT3HV7GZL7JFRMWRN2AZUQ, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.471440098 (415h3m6.299148489s) [noise=true , address=145.79.227.47:4266 , piece_id=3NFYV2PWBNTAOSFHENM4J5PZ7XXE72UKTQVXLTVQRHGK4DUXAKDA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.560766475 (415h3m6.388474871s) [upload=431917]: Next returned nil segment Dec 22 22:31:05.560785161 (415h3m6.388493562s) [upload=431917]: Waiting for error group managing segments... Dec 22 22:31:05.566117878 (415h3m6.39382628s) [noise=true , address=45.91.116.4:4939 , piece_id=QYNKMKDLQJV7B6M2E34DU53Y7YI6VBCUW3C7ZLQOI4XWTBXZHKDQ, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.566723306 (415h3m6.394431687s) [noise=true , address=45.91.116.11:4117 , piece_id=WLMO5E3BIFFL3TOAUSSBGORIZ6DOCWFCMVT4AVFPPGPSKP2OKDAA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.566992667 (415h3m6.394701063s) [noise=true , address=145.79.227.36:4519 , piece_id=JUKRRIZWVGKEDD3GPJ7HMY2MEQZNMUCTKTXR6ZLSIERRONTZKHCQ, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.567464626 (415h3m6.395173019s) [noise=true , address=145.79.227.47:4266 , piece_id=3NFYV2PWBNTAOSFHENM4J5PZ7XXE72UKTQVXLTVQRHGK4DUXAKDA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.567924492 (415h3m6.395632883s) [noise=true , address=145.79.227.15:4874 , piece_id=FNCCIMIVJTGZ2X5TOYOZH727JXHYM7HG3HBHBTZDAG67SHZLYYYA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.568807294 (415h3m6.396515685s) [noise=true , address=145.79.227.51:4409 , piece_id=ATH3F4DFOUUIFFJZRA2SSRRL7AKWZMQNQRYOZCVFFDLTFCFP236A, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.570579823 (415h3m6.398288214s) [noise=true , address=145.79.227.48:4301 , piece_id=LBIYTJNOX5J4M3HDE7FGF5CSARBITW2AK3P2DSYFLFKUOIHZUKOQ, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.571632758 (415h3m6.399341149s) [noise=true , address=145.79.227.43:4157 , piece_id=3W3LVTU37DNHWS52IGIIH7SXWXVNPGEY27RQNT272L3Z64GNR4KA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.573168383 (415h3m6.400876774s) [noise=true , address=145.79.227.38:4979 , piece_id=HLMQC5OEB4OL4OZSRMCEVAWR5QM2T4MQUBPTHRVQLEAHPBWPQF6Q, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.579995420 (415h3m6.407703816s) [noise=true , address=45.91.116.14:4190 , piece_id=V4T6Z7K3C75LQEXNQIYZGYWW2CSQ7UPSET46QGDRT52RFGAROTTA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.592093172 (415h3m6.419801563s) [noise=true , address=145.79.227.37:4565 , piece_id=554RHTYPG27UNCFPSRMOB245AX4QXL2QFIYECNTVWNNJMAL4BTYA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:09.946500434 (415h3m10.774208816s) [noise=true , address=145.79.227.40:4051 , piece_id=55RRAD7DQX6SR7RBLYAV7MD7XRWPNIWHTVVBQZ2ZL3OE2L7LRRIQ, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:35.398235543 (415h3m36.225943943s) [noise=true , address=145.79.227.39:4013 , piece_id=JJVURGSLKY2UQ4FKNCLATA3ZBD5SDL32XJAG42KH2U6OYK3KTS6Q, seg_pos={0 0}, upload=431917]: Done uploading piece. err: ecclient: upload failed (node:121scLvKEW9zw2aEz22zpremcjiop67eQ8v2uqpmaD5aZzCAP7b, address:145.79.227.39:4013): protocol: write tcp 159.223.143.39:60842->145.79.227.39:4013: use of closed network connection; write tcp 159.223.143.39:60842->145.79.227.39:4013: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:60842->145.79.227.39:4013: use of closed network connection Dec 22 22:31:35.398562305 (415h3m36.226270699s) [noise=true , address=145.79.227.32:4835 , piece_id=BRE3JH3XKPXB6FN7WB3W6ED7MOFES6HYLR34YGR6L5AX7RPIQ7IA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: ecclient: upload failed (node:16vVfeoqCkbMYCRjMMWaKBnCvQBBJPMgwQtWTq9HKQrD9SS17y, address:145.79.227.32:4835): protocol: write tcp 159.223.143.39:45098->145.79.227.32:4835: use of closed network connection; write tcp 159.223.143.39:45098->145.79.227.32:4835: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:45098->145.79.227.32:4835: use of closed network connection Dec 22 22:31:35.399183028 (415h3m36.226891422s) [noise=true , address=145.79.227.50:4373 , piece_id=4U55T5QDYCE6AZ3NCTPAVECPAELPQUKT3HV7GZL7JFRMWRN2AZUQ, seg_pos={0 0}, upload=431917]: Done uploading piece. err: ecclient: upload failed (node:12AQGSkJuNJDxoeqhChUenWxHwF5yK41UESCF63FQz3XvabXXfz, address:145.79.227.50:4373): protocol: read tcp 159.223.143.39:56028->145.79.227.50:4373: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 22 22:31:35.399278021 (415h3m36.226986417s) [noise=true , address=145.79.227.29:4943 , piece_id=G4TMZ6IAIZBDON3UEDPH4LJQFHV3JLALEXSJ3VI4XN6WHIZOB57Q, seg_pos={0 0}, upload=431917]: Done uploading piece. err: ecclient: upload failed (node:12fc5Trj8XxacZhzxdksUVT5CbG6uTmiwdUPjreYi4UMPY1TXWz, address:145.79.227.29:4943): protocol: write tcp 159.223.143.39:50820->145.79.227.29:4943: use of closed network connection; write tcp 159.223.143.39:50820->145.79.227.29:4943: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:50820->145.79.227.29:4943: use of closed network connection Dec 22 22:31:35.399379485 (415h3m36.227087866s) [seg_pos={0 0}, upload=431917]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 22 22:31:35.421122544 (415h3m36.248830935s) [upload=431917]: Done uploading segments... Dec 22 22:31:35.421174369 (415h3m36.248882769s) [upload=431917]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 431.739374ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [......X................................................X....] duration: 427.64273ms trace: https://jaeger.dev.storj.io/trace/7013ffb1a981d605 FAIL (2025-12-22 22:34:17 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 22 22:31:05.439510308 (415h3m6.2672187s) [upload=431917]: Starting upload Dec 22 22:31:05.439581993 (415h3m6.26729037s) [upload=431917]: Done starting upload Dec 22 22:31:05.439812792 (415h3m6.267521185s) [upload=431917]: Uploading segments... Dec 22 22:31:05.439942524 (415h3m6.267650918s) [upload=431917]: Got next segment. Inline: false Dec 22 22:31:05.440022744 (415h3m6.267731174s) [upload=431917]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 22 22:31:05.440029670 (415h3m6.267738072s) [upload=431917]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 22 22:31:05.467186372 (415h3m6.294894769s) [seg_pos={0 0}, upload=431917]: Begin upload segment... Dec 22 22:31:05.467265149 (415h3m6.294973535s) [seg_pos={0 0}, upload=431917]: Done begin upload segment. Dec 22 22:31:05.467296974 (415h3m6.295005373s) [noise=true , address=145.79.227.51:4409 , piece_id=ATH3F4DFOUUIFFJZRA2SSRRL7AKWZMQNQRYOZCVFFDLTFCFP236A, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.467389853 (415h3m6.295098247s) [noise=true , address=145.79.227.39:4013 , piece_id=JJVURGSLKY2UQ4FKNCLATA3ZBD5SDL32XJAG42KH2U6OYK3KTS6Q, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.467912104 (415h3m6.295620486s) [noise=true , address=145.79.227.36:4519 , piece_id=JUKRRIZWVGKEDD3GPJ7HMY2MEQZNMUCTKTXR6ZLSIERRONTZKHCQ, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.468324863 (415h3m6.296033263s) [noise=true , address=145.79.227.38:4979 , piece_id=HLMQC5OEB4OL4OZSRMCEVAWR5QM2T4MQUBPTHRVQLEAHPBWPQF6Q, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.468334764 (415h3m6.296043147s) [noise=true , address=145.79.227.40:4051 , piece_id=55RRAD7DQX6SR7RBLYAV7MD7XRWPNIWHTVVBQZ2ZL3OE2L7LRRIQ, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.468742525 (415h3m6.296450916s) [noise=true , address=45.91.116.4:4939 , piece_id=QYNKMKDLQJV7B6M2E34DU53Y7YI6VBCUW3C7ZLQOI4XWTBXZHKDQ, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.468915034 (415h3m6.296623414s) [noise=true , address=45.91.116.11:4117 , piece_id=WLMO5E3BIFFL3TOAUSSBGORIZ6DOCWFCMVT4AVFPPGPSKP2OKDAA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.469069393 (415h3m6.296777787s) [noise=true , address=145.79.227.32:4835 , piece_id=BRE3JH3XKPXB6FN7WB3W6ED7MOFES6HYLR34YGR6L5AX7RPIQ7IA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.469285789 (415h3m6.296994183s) [noise=true , address=145.79.227.29:4943 , piece_id=G4TMZ6IAIZBDON3UEDPH4LJQFHV3JLALEXSJ3VI4XN6WHIZOB57Q, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.469411962 (415h3m6.297120355s) [noise=true , address=145.79.227.48:4301 , piece_id=LBIYTJNOX5J4M3HDE7FGF5CSARBITW2AK3P2DSYFLFKUOIHZUKOQ, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.469749379 (415h3m6.297457773s) [noise=true , address=145.79.227.37:4565 , piece_id=554RHTYPG27UNCFPSRMOB245AX4QXL2QFIYECNTVWNNJMAL4BTYA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.470080828 (415h3m6.29778922s) [noise=true , address=145.79.227.43:4157 , piece_id=3W3LVTU37DNHWS52IGIIH7SXWXVNPGEY27RQNT272L3Z64GNR4KA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.470477660 (415h3m6.298186058s) [noise=true , address=45.91.116.14:4190 , piece_id=V4T6Z7K3C75LQEXNQIYZGYWW2CSQ7UPSET46QGDRT52RFGAROTTA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.470567977 (415h3m6.29827637s) [noise=true , address=145.79.227.15:4874 , piece_id=FNCCIMIVJTGZ2X5TOYOZH727JXHYM7HG3HBHBTZDAG67SHZLYYYA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.470980950 (415h3m6.298689349s) [noise=true , address=145.79.227.50:4373 , piece_id=4U55T5QDYCE6AZ3NCTPAVECPAELPQUKT3HV7GZL7JFRMWRN2AZUQ, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.471440098 (415h3m6.299148489s) [noise=true , address=145.79.227.47:4266 , piece_id=3NFYV2PWBNTAOSFHENM4J5PZ7XXE72UKTQVXLTVQRHGK4DUXAKDA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.560766475 (415h3m6.388474871s) [upload=431917]: Next returned nil segment Dec 22 22:31:05.560785161 (415h3m6.388493562s) [upload=431917]: Waiting for error group managing segments... Dec 22 22:31:05.566117878 (415h3m6.39382628s) [noise=true , address=45.91.116.4:4939 , piece_id=QYNKMKDLQJV7B6M2E34DU53Y7YI6VBCUW3C7ZLQOI4XWTBXZHKDQ, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.566723306 (415h3m6.394431687s) [noise=true , address=45.91.116.11:4117 , piece_id=WLMO5E3BIFFL3TOAUSSBGORIZ6DOCWFCMVT4AVFPPGPSKP2OKDAA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.566992667 (415h3m6.394701063s) [noise=true , address=145.79.227.36:4519 , piece_id=JUKRRIZWVGKEDD3GPJ7HMY2MEQZNMUCTKTXR6ZLSIERRONTZKHCQ, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.567464626 (415h3m6.395173019s) [noise=true , address=145.79.227.47:4266 , piece_id=3NFYV2PWBNTAOSFHENM4J5PZ7XXE72UKTQVXLTVQRHGK4DUXAKDA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.567924492 (415h3m6.395632883s) [noise=true , address=145.79.227.15:4874 , piece_id=FNCCIMIVJTGZ2X5TOYOZH727JXHYM7HG3HBHBTZDAG67SHZLYYYA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.568807294 (415h3m6.396515685s) [noise=true , address=145.79.227.51:4409 , piece_id=ATH3F4DFOUUIFFJZRA2SSRRL7AKWZMQNQRYOZCVFFDLTFCFP236A, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.570579823 (415h3m6.398288214s) [noise=true , address=145.79.227.48:4301 , piece_id=LBIYTJNOX5J4M3HDE7FGF5CSARBITW2AK3P2DSYFLFKUOIHZUKOQ, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.571632758 (415h3m6.399341149s) [noise=true , address=145.79.227.43:4157 , piece_id=3W3LVTU37DNHWS52IGIIH7SXWXVNPGEY27RQNT272L3Z64GNR4KA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.573168383 (415h3m6.400876774s) [noise=true , address=145.79.227.38:4979 , piece_id=HLMQC5OEB4OL4OZSRMCEVAWR5QM2T4MQUBPTHRVQLEAHPBWPQF6Q, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.579995420 (415h3m6.407703816s) [noise=true , address=45.91.116.14:4190 , piece_id=V4T6Z7K3C75LQEXNQIYZGYWW2CSQ7UPSET46QGDRT52RFGAROTTA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.592093172 (415h3m6.419801563s) [noise=true , address=145.79.227.37:4565 , piece_id=554RHTYPG27UNCFPSRMOB245AX4QXL2QFIYECNTVWNNJMAL4BTYA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:09.946500434 (415h3m10.774208816s) [noise=true , address=145.79.227.40:4051 , piece_id=55RRAD7DQX6SR7RBLYAV7MD7XRWPNIWHTVVBQZ2ZL3OE2L7LRRIQ, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:35.398235543 (415h3m36.225943943s) [noise=true , address=145.79.227.39:4013 , piece_id=JJVURGSLKY2UQ4FKNCLATA3ZBD5SDL32XJAG42KH2U6OYK3KTS6Q, seg_pos={0 0}, upload=431917]: Done uploading piece. err: ecclient: upload failed (node:121scLvKEW9zw2aEz22zpremcjiop67eQ8v2uqpmaD5aZzCAP7b, address:145.79.227.39:4013): protocol: write tcp 159.223.143.39:60842->145.79.227.39:4013: use of closed network connection; write tcp 159.223.143.39:60842->145.79.227.39:4013: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:60842->145.79.227.39:4013: use of closed network connection Dec 22 22:31:35.398562305 (415h3m36.226270699s) [noise=true , address=145.79.227.32:4835 , piece_id=BRE3JH3XKPXB6FN7WB3W6ED7MOFES6HYLR34YGR6L5AX7RPIQ7IA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: ecclient: upload failed (node:16vVfeoqCkbMYCRjMMWaKBnCvQBBJPMgwQtWTq9HKQrD9SS17y, address:145.79.227.32:4835): protocol: write tcp 159.223.143.39:45098->145.79.227.32:4835: use of closed network connection; write tcp 159.223.143.39:45098->145.79.227.32:4835: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:45098->145.79.227.32:4835: use of closed network connection Dec 22 22:31:35.399183028 (415h3m36.226891422s) [noise=true , address=145.79.227.50:4373 , piece_id=4U55T5QDYCE6AZ3NCTPAVECPAELPQUKT3HV7GZL7JFRMWRN2AZUQ, seg_pos={0 0}, upload=431917]: Done uploading piece. err: ecclient: upload failed (node:12AQGSkJuNJDxoeqhChUenWxHwF5yK41UESCF63FQz3XvabXXfz, address:145.79.227.50:4373): protocol: read tcp 159.223.143.39:56028->145.79.227.50:4373: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 22 22:31:35.399278021 (415h3m36.226986417s) [noise=true , address=145.79.227.29:4943 , piece_id=G4TMZ6IAIZBDON3UEDPH4LJQFHV3JLALEXSJ3VI4XN6WHIZOB57Q, seg_pos={0 0}, upload=431917]: Done uploading piece. err: ecclient: upload failed (node:12fc5Trj8XxacZhzxdksUVT5CbG6uTmiwdUPjreYi4UMPY1TXWz, address:145.79.227.29:4943): protocol: write tcp 159.223.143.39:50820->145.79.227.29:4943: use of closed network connection; write tcp 159.223.143.39:50820->145.79.227.29:4943: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:50820->145.79.227.29:4943: use of closed network connection Dec 22 22:31:35.399379485 (415h3m36.227087866s) [seg_pos={0 0}, upload=431917]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 22 22:31:35.421122544 (415h3m36.248830935s) [upload=431917]: Done uploading segments... Dec 22 22:31:35.421174369 (415h3m36.248882769s) [upload=431917]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 427.667561ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.......X................................................X...] duration: 512.422504ms trace: https://jaeger.dev.storj.io/trace/7013ffb1a981d605 FAIL (2025-12-22 22:33:57 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 22 22:31:05.439510308 (415h3m6.2672187s) [upload=431917]: Starting upload Dec 22 22:31:05.439581993 (415h3m6.26729037s) [upload=431917]: Done starting upload Dec 22 22:31:05.439812792 (415h3m6.267521185s) [upload=431917]: Uploading segments... Dec 22 22:31:05.439942524 (415h3m6.267650918s) [upload=431917]: Got next segment. Inline: false Dec 22 22:31:05.440022744 (415h3m6.267731174s) [upload=431917]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 22 22:31:05.440029670 (415h3m6.267738072s) [upload=431917]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 22 22:31:05.467186372 (415h3m6.294894769s) [seg_pos={0 0}, upload=431917]: Begin upload segment... Dec 22 22:31:05.467265149 (415h3m6.294973535s) [seg_pos={0 0}, upload=431917]: Done begin upload segment. Dec 22 22:31:05.467296974 (415h3m6.295005373s) [noise=true , address=145.79.227.51:4409 , piece_id=ATH3F4DFOUUIFFJZRA2SSRRL7AKWZMQNQRYOZCVFFDLTFCFP236A, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.467389853 (415h3m6.295098247s) [noise=true , address=145.79.227.39:4013 , piece_id=JJVURGSLKY2UQ4FKNCLATA3ZBD5SDL32XJAG42KH2U6OYK3KTS6Q, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.467912104 (415h3m6.295620486s) [noise=true , address=145.79.227.36:4519 , piece_id=JUKRRIZWVGKEDD3GPJ7HMY2MEQZNMUCTKTXR6ZLSIERRONTZKHCQ, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.468324863 (415h3m6.296033263s) [noise=true , address=145.79.227.38:4979 , piece_id=HLMQC5OEB4OL4OZSRMCEVAWR5QM2T4MQUBPTHRVQLEAHPBWPQF6Q, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.468334764 (415h3m6.296043147s) [noise=true , address=145.79.227.40:4051 , piece_id=55RRAD7DQX6SR7RBLYAV7MD7XRWPNIWHTVVBQZ2ZL3OE2L7LRRIQ, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.468742525 (415h3m6.296450916s) [noise=true , address=45.91.116.4:4939 , piece_id=QYNKMKDLQJV7B6M2E34DU53Y7YI6VBCUW3C7ZLQOI4XWTBXZHKDQ, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.468915034 (415h3m6.296623414s) [noise=true , address=45.91.116.11:4117 , piece_id=WLMO5E3BIFFL3TOAUSSBGORIZ6DOCWFCMVT4AVFPPGPSKP2OKDAA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.469069393 (415h3m6.296777787s) [noise=true , address=145.79.227.32:4835 , piece_id=BRE3JH3XKPXB6FN7WB3W6ED7MOFES6HYLR34YGR6L5AX7RPIQ7IA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.469285789 (415h3m6.296994183s) [noise=true , address=145.79.227.29:4943 , piece_id=G4TMZ6IAIZBDON3UEDPH4LJQFHV3JLALEXSJ3VI4XN6WHIZOB57Q, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.469411962 (415h3m6.297120355s) [noise=true , address=145.79.227.48:4301 , piece_id=LBIYTJNOX5J4M3HDE7FGF5CSARBITW2AK3P2DSYFLFKUOIHZUKOQ, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.469749379 (415h3m6.297457773s) [noise=true , address=145.79.227.37:4565 , piece_id=554RHTYPG27UNCFPSRMOB245AX4QXL2QFIYECNTVWNNJMAL4BTYA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.470080828 (415h3m6.29778922s) [noise=true , address=145.79.227.43:4157 , piece_id=3W3LVTU37DNHWS52IGIIH7SXWXVNPGEY27RQNT272L3Z64GNR4KA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.470477660 (415h3m6.298186058s) [noise=true , address=45.91.116.14:4190 , piece_id=V4T6Z7K3C75LQEXNQIYZGYWW2CSQ7UPSET46QGDRT52RFGAROTTA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.470567977 (415h3m6.29827637s) [noise=true , address=145.79.227.15:4874 , piece_id=FNCCIMIVJTGZ2X5TOYOZH727JXHYM7HG3HBHBTZDAG67SHZLYYYA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.470980950 (415h3m6.298689349s) [noise=true , address=145.79.227.50:4373 , piece_id=4U55T5QDYCE6AZ3NCTPAVECPAELPQUKT3HV7GZL7JFRMWRN2AZUQ, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.471440098 (415h3m6.299148489s) [noise=true , address=145.79.227.47:4266 , piece_id=3NFYV2PWBNTAOSFHENM4J5PZ7XXE72UKTQVXLTVQRHGK4DUXAKDA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.560766475 (415h3m6.388474871s) [upload=431917]: Next returned nil segment Dec 22 22:31:05.560785161 (415h3m6.388493562s) [upload=431917]: Waiting for error group managing segments... Dec 22 22:31:05.566117878 (415h3m6.39382628s) [noise=true , address=45.91.116.4:4939 , piece_id=QYNKMKDLQJV7B6M2E34DU53Y7YI6VBCUW3C7ZLQOI4XWTBXZHKDQ, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.566723306 (415h3m6.394431687s) [noise=true , address=45.91.116.11:4117 , piece_id=WLMO5E3BIFFL3TOAUSSBGORIZ6DOCWFCMVT4AVFPPGPSKP2OKDAA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.566992667 (415h3m6.394701063s) [noise=true , address=145.79.227.36:4519 , piece_id=JUKRRIZWVGKEDD3GPJ7HMY2MEQZNMUCTKTXR6ZLSIERRONTZKHCQ, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.567464626 (415h3m6.395173019s) [noise=true , address=145.79.227.47:4266 , piece_id=3NFYV2PWBNTAOSFHENM4J5PZ7XXE72UKTQVXLTVQRHGK4DUXAKDA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.567924492 (415h3m6.395632883s) [noise=true , address=145.79.227.15:4874 , piece_id=FNCCIMIVJTGZ2X5TOYOZH727JXHYM7HG3HBHBTZDAG67SHZLYYYA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.568807294 (415h3m6.396515685s) [noise=true , address=145.79.227.51:4409 , piece_id=ATH3F4DFOUUIFFJZRA2SSRRL7AKWZMQNQRYOZCVFFDLTFCFP236A, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.570579823 (415h3m6.398288214s) [noise=true , address=145.79.227.48:4301 , piece_id=LBIYTJNOX5J4M3HDE7FGF5CSARBITW2AK3P2DSYFLFKUOIHZUKOQ, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.571632758 (415h3m6.399341149s) [noise=true , address=145.79.227.43:4157 , piece_id=3W3LVTU37DNHWS52IGIIH7SXWXVNPGEY27RQNT272L3Z64GNR4KA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.573168383 (415h3m6.400876774s) [noise=true , address=145.79.227.38:4979 , piece_id=HLMQC5OEB4OL4OZSRMCEVAWR5QM2T4MQUBPTHRVQLEAHPBWPQF6Q, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.579995420 (415h3m6.407703816s) [noise=true , address=45.91.116.14:4190 , piece_id=V4T6Z7K3C75LQEXNQIYZGYWW2CSQ7UPSET46QGDRT52RFGAROTTA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.592093172 (415h3m6.419801563s) [noise=true , address=145.79.227.37:4565 , piece_id=554RHTYPG27UNCFPSRMOB245AX4QXL2QFIYECNTVWNNJMAL4BTYA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:09.946500434 (415h3m10.774208816s) [noise=true , address=145.79.227.40:4051 , piece_id=55RRAD7DQX6SR7RBLYAV7MD7XRWPNIWHTVVBQZ2ZL3OE2L7LRRIQ, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:35.398235543 (415h3m36.225943943s) [noise=true , address=145.79.227.39:4013 , piece_id=JJVURGSLKY2UQ4FKNCLATA3ZBD5SDL32XJAG42KH2U6OYK3KTS6Q, seg_pos={0 0}, upload=431917]: Done uploading piece. err: ecclient: upload failed (node:121scLvKEW9zw2aEz22zpremcjiop67eQ8v2uqpmaD5aZzCAP7b, address:145.79.227.39:4013): protocol: write tcp 159.223.143.39:60842->145.79.227.39:4013: use of closed network connection; write tcp 159.223.143.39:60842->145.79.227.39:4013: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:60842->145.79.227.39:4013: use of closed network connection Dec 22 22:31:35.398562305 (415h3m36.226270699s) [noise=true , address=145.79.227.32:4835 , piece_id=BRE3JH3XKPXB6FN7WB3W6ED7MOFES6HYLR34YGR6L5AX7RPIQ7IA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: ecclient: upload failed (node:16vVfeoqCkbMYCRjMMWaKBnCvQBBJPMgwQtWTq9HKQrD9SS17y, address:145.79.227.32:4835): protocol: write tcp 159.223.143.39:45098->145.79.227.32:4835: use of closed network connection; write tcp 159.223.143.39:45098->145.79.227.32:4835: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:45098->145.79.227.32:4835: use of closed network connection Dec 22 22:31:35.399183028 (415h3m36.226891422s) [noise=true , address=145.79.227.50:4373 , piece_id=4U55T5QDYCE6AZ3NCTPAVECPAELPQUKT3HV7GZL7JFRMWRN2AZUQ, seg_pos={0 0}, upload=431917]: Done uploading piece. err: ecclient: upload failed (node:12AQGSkJuNJDxoeqhChUenWxHwF5yK41UESCF63FQz3XvabXXfz, address:145.79.227.50:4373): protocol: read tcp 159.223.143.39:56028->145.79.227.50:4373: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 22 22:31:35.399278021 (415h3m36.226986417s) [noise=true , address=145.79.227.29:4943 , piece_id=G4TMZ6IAIZBDON3UEDPH4LJQFHV3JLALEXSJ3VI4XN6WHIZOB57Q, seg_pos={0 0}, upload=431917]: Done uploading piece. err: ecclient: upload failed (node:12fc5Trj8XxacZhzxdksUVT5CbG6uTmiwdUPjreYi4UMPY1TXWz, address:145.79.227.29:4943): protocol: write tcp 159.223.143.39:50820->145.79.227.29:4943: use of closed network connection; write tcp 159.223.143.39:50820->145.79.227.29:4943: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:50820->145.79.227.29:4943: use of closed network connection Dec 22 22:31:35.399379485 (415h3m36.227087866s) [seg_pos={0 0}, upload=431917]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 22 22:31:35.421122544 (415h3m36.248830935s) [upload=431917]: Done uploading segments... Dec 22 22:31:35.421174369 (415h3m36.248882769s) [upload=431917]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 512.464734ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [........X................................................X..] duration: 409.630349ms trace: https://jaeger.dev.storj.io/trace/7013ffb1a981d605 FAIL (2025-12-22 22:32:53 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 22 22:31:05.439510308 (415h3m6.2672187s) [upload=431917]: Starting upload Dec 22 22:31:05.439581993 (415h3m6.26729037s) [upload=431917]: Done starting upload Dec 22 22:31:05.439812792 (415h3m6.267521185s) [upload=431917]: Uploading segments... Dec 22 22:31:05.439942524 (415h3m6.267650918s) [upload=431917]: Got next segment. Inline: false Dec 22 22:31:05.440022744 (415h3m6.267731174s) [upload=431917]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 22 22:31:05.440029670 (415h3m6.267738072s) [upload=431917]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 22 22:31:05.467186372 (415h3m6.294894769s) [seg_pos={0 0}, upload=431917]: Begin upload segment... Dec 22 22:31:05.467265149 (415h3m6.294973535s) [seg_pos={0 0}, upload=431917]: Done begin upload segment. Dec 22 22:31:05.467296974 (415h3m6.295005373s) [noise=true , address=145.79.227.51:4409 , piece_id=ATH3F4DFOUUIFFJZRA2SSRRL7AKWZMQNQRYOZCVFFDLTFCFP236A, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.467389853 (415h3m6.295098247s) [noise=true , address=145.79.227.39:4013 , piece_id=JJVURGSLKY2UQ4FKNCLATA3ZBD5SDL32XJAG42KH2U6OYK3KTS6Q, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.467912104 (415h3m6.295620486s) [noise=true , address=145.79.227.36:4519 , piece_id=JUKRRIZWVGKEDD3GPJ7HMY2MEQZNMUCTKTXR6ZLSIERRONTZKHCQ, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.468324863 (415h3m6.296033263s) [noise=true , address=145.79.227.38:4979 , piece_id=HLMQC5OEB4OL4OZSRMCEVAWR5QM2T4MQUBPTHRVQLEAHPBWPQF6Q, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.468334764 (415h3m6.296043147s) [noise=true , address=145.79.227.40:4051 , piece_id=55RRAD7DQX6SR7RBLYAV7MD7XRWPNIWHTVVBQZ2ZL3OE2L7LRRIQ, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.468742525 (415h3m6.296450916s) [noise=true , address=45.91.116.4:4939 , piece_id=QYNKMKDLQJV7B6M2E34DU53Y7YI6VBCUW3C7ZLQOI4XWTBXZHKDQ, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.468915034 (415h3m6.296623414s) [noise=true , address=45.91.116.11:4117 , piece_id=WLMO5E3BIFFL3TOAUSSBGORIZ6DOCWFCMVT4AVFPPGPSKP2OKDAA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.469069393 (415h3m6.296777787s) [noise=true , address=145.79.227.32:4835 , piece_id=BRE3JH3XKPXB6FN7WB3W6ED7MOFES6HYLR34YGR6L5AX7RPIQ7IA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.469285789 (415h3m6.296994183s) [noise=true , address=145.79.227.29:4943 , piece_id=G4TMZ6IAIZBDON3UEDPH4LJQFHV3JLALEXSJ3VI4XN6WHIZOB57Q, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.469411962 (415h3m6.297120355s) [noise=true , address=145.79.227.48:4301 , piece_id=LBIYTJNOX5J4M3HDE7FGF5CSARBITW2AK3P2DSYFLFKUOIHZUKOQ, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.469749379 (415h3m6.297457773s) [noise=true , address=145.79.227.37:4565 , piece_id=554RHTYPG27UNCFPSRMOB245AX4QXL2QFIYECNTVWNNJMAL4BTYA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.470080828 (415h3m6.29778922s) [noise=true , address=145.79.227.43:4157 , piece_id=3W3LVTU37DNHWS52IGIIH7SXWXVNPGEY27RQNT272L3Z64GNR4KA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.470477660 (415h3m6.298186058s) [noise=true , address=45.91.116.14:4190 , piece_id=V4T6Z7K3C75LQEXNQIYZGYWW2CSQ7UPSET46QGDRT52RFGAROTTA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.470567977 (415h3m6.29827637s) [noise=true , address=145.79.227.15:4874 , piece_id=FNCCIMIVJTGZ2X5TOYOZH727JXHYM7HG3HBHBTZDAG67SHZLYYYA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.470980950 (415h3m6.298689349s) [noise=true , address=145.79.227.50:4373 , piece_id=4U55T5QDYCE6AZ3NCTPAVECPAELPQUKT3HV7GZL7JFRMWRN2AZUQ, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.471440098 (415h3m6.299148489s) [noise=true , address=145.79.227.47:4266 , piece_id=3NFYV2PWBNTAOSFHENM4J5PZ7XXE72UKTQVXLTVQRHGK4DUXAKDA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.560766475 (415h3m6.388474871s) [upload=431917]: Next returned nil segment Dec 22 22:31:05.560785161 (415h3m6.388493562s) [upload=431917]: Waiting for error group managing segments... Dec 22 22:31:05.566117878 (415h3m6.39382628s) [noise=true , address=45.91.116.4:4939 , piece_id=QYNKMKDLQJV7B6M2E34DU53Y7YI6VBCUW3C7ZLQOI4XWTBXZHKDQ, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.566723306 (415h3m6.394431687s) [noise=true , address=45.91.116.11:4117 , piece_id=WLMO5E3BIFFL3TOAUSSBGORIZ6DOCWFCMVT4AVFPPGPSKP2OKDAA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.566992667 (415h3m6.394701063s) [noise=true , address=145.79.227.36:4519 , piece_id=JUKRRIZWVGKEDD3GPJ7HMY2MEQZNMUCTKTXR6ZLSIERRONTZKHCQ, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.567464626 (415h3m6.395173019s) [noise=true , address=145.79.227.47:4266 , piece_id=3NFYV2PWBNTAOSFHENM4J5PZ7XXE72UKTQVXLTVQRHGK4DUXAKDA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.567924492 (415h3m6.395632883s) [noise=true , address=145.79.227.15:4874 , piece_id=FNCCIMIVJTGZ2X5TOYOZH727JXHYM7HG3HBHBTZDAG67SHZLYYYA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.568807294 (415h3m6.396515685s) [noise=true , address=145.79.227.51:4409 , piece_id=ATH3F4DFOUUIFFJZRA2SSRRL7AKWZMQNQRYOZCVFFDLTFCFP236A, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.570579823 (415h3m6.398288214s) [noise=true , address=145.79.227.48:4301 , piece_id=LBIYTJNOX5J4M3HDE7FGF5CSARBITW2AK3P2DSYFLFKUOIHZUKOQ, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.571632758 (415h3m6.399341149s) [noise=true , address=145.79.227.43:4157 , piece_id=3W3LVTU37DNHWS52IGIIH7SXWXVNPGEY27RQNT272L3Z64GNR4KA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.573168383 (415h3m6.400876774s) [noise=true , address=145.79.227.38:4979 , piece_id=HLMQC5OEB4OL4OZSRMCEVAWR5QM2T4MQUBPTHRVQLEAHPBWPQF6Q, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.579995420 (415h3m6.407703816s) [noise=true , address=45.91.116.14:4190 , piece_id=V4T6Z7K3C75LQEXNQIYZGYWW2CSQ7UPSET46QGDRT52RFGAROTTA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.592093172 (415h3m6.419801563s) [noise=true , address=145.79.227.37:4565 , piece_id=554RHTYPG27UNCFPSRMOB245AX4QXL2QFIYECNTVWNNJMAL4BTYA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:09.946500434 (415h3m10.774208816s) [noise=true , address=145.79.227.40:4051 , piece_id=55RRAD7DQX6SR7RBLYAV7MD7XRWPNIWHTVVBQZ2ZL3OE2L7LRRIQ, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:35.398235543 (415h3m36.225943943s) [noise=true , address=145.79.227.39:4013 , piece_id=JJVURGSLKY2UQ4FKNCLATA3ZBD5SDL32XJAG42KH2U6OYK3KTS6Q, seg_pos={0 0}, upload=431917]: Done uploading piece. err: ecclient: upload failed (node:121scLvKEW9zw2aEz22zpremcjiop67eQ8v2uqpmaD5aZzCAP7b, address:145.79.227.39:4013): protocol: write tcp 159.223.143.39:60842->145.79.227.39:4013: use of closed network connection; write tcp 159.223.143.39:60842->145.79.227.39:4013: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:60842->145.79.227.39:4013: use of closed network connection Dec 22 22:31:35.398562305 (415h3m36.226270699s) [noise=true , address=145.79.227.32:4835 , piece_id=BRE3JH3XKPXB6FN7WB3W6ED7MOFES6HYLR34YGR6L5AX7RPIQ7IA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: ecclient: upload failed (node:16vVfeoqCkbMYCRjMMWaKBnCvQBBJPMgwQtWTq9HKQrD9SS17y, address:145.79.227.32:4835): protocol: write tcp 159.223.143.39:45098->145.79.227.32:4835: use of closed network connection; write tcp 159.223.143.39:45098->145.79.227.32:4835: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:45098->145.79.227.32:4835: use of closed network connection Dec 22 22:31:35.399183028 (415h3m36.226891422s) [noise=true , address=145.79.227.50:4373 , piece_id=4U55T5QDYCE6AZ3NCTPAVECPAELPQUKT3HV7GZL7JFRMWRN2AZUQ, seg_pos={0 0}, upload=431917]: Done uploading piece. err: ecclient: upload failed (node:12AQGSkJuNJDxoeqhChUenWxHwF5yK41UESCF63FQz3XvabXXfz, address:145.79.227.50:4373): protocol: read tcp 159.223.143.39:56028->145.79.227.50:4373: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 22 22:31:35.399278021 (415h3m36.226986417s) [noise=true , address=145.79.227.29:4943 , piece_id=G4TMZ6IAIZBDON3UEDPH4LJQFHV3JLALEXSJ3VI4XN6WHIZOB57Q, seg_pos={0 0}, upload=431917]: Done uploading piece. err: ecclient: upload failed (node:12fc5Trj8XxacZhzxdksUVT5CbG6uTmiwdUPjreYi4UMPY1TXWz, address:145.79.227.29:4943): protocol: write tcp 159.223.143.39:50820->145.79.227.29:4943: use of closed network connection; write tcp 159.223.143.39:50820->145.79.227.29:4943: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:50820->145.79.227.29:4943: use of closed network connection Dec 22 22:31:35.399379485 (415h3m36.227087866s) [seg_pos={0 0}, upload=431917]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 22 22:31:35.421122544 (415h3m36.248830935s) [upload=431917]: Done uploading segments... Dec 22 22:31:35.421174369 (415h3m36.248882769s) [upload=431917]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 409.65536ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.........X................................................X.] duration: 469.588505ms trace: https://jaeger.dev.storj.io/trace/7013ffb1a981d605 FAIL (2025-12-22 22:32:14 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 22 22:31:05.439510308 (415h3m6.2672187s) [upload=431917]: Starting upload Dec 22 22:31:05.439581993 (415h3m6.26729037s) [upload=431917]: Done starting upload Dec 22 22:31:05.439812792 (415h3m6.267521185s) [upload=431917]: Uploading segments... Dec 22 22:31:05.439942524 (415h3m6.267650918s) [upload=431917]: Got next segment. Inline: false Dec 22 22:31:05.440022744 (415h3m6.267731174s) [upload=431917]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 22 22:31:05.440029670 (415h3m6.267738072s) [upload=431917]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 22 22:31:05.467186372 (415h3m6.294894769s) [seg_pos={0 0}, upload=431917]: Begin upload segment... Dec 22 22:31:05.467265149 (415h3m6.294973535s) [seg_pos={0 0}, upload=431917]: Done begin upload segment. Dec 22 22:31:05.467296974 (415h3m6.295005373s) [noise=true , address=145.79.227.51:4409 , piece_id=ATH3F4DFOUUIFFJZRA2SSRRL7AKWZMQNQRYOZCVFFDLTFCFP236A, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.467389853 (415h3m6.295098247s) [noise=true , address=145.79.227.39:4013 , piece_id=JJVURGSLKY2UQ4FKNCLATA3ZBD5SDL32XJAG42KH2U6OYK3KTS6Q, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.467912104 (415h3m6.295620486s) [noise=true , address=145.79.227.36:4519 , piece_id=JUKRRIZWVGKEDD3GPJ7HMY2MEQZNMUCTKTXR6ZLSIERRONTZKHCQ, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.468324863 (415h3m6.296033263s) [noise=true , address=145.79.227.38:4979 , piece_id=HLMQC5OEB4OL4OZSRMCEVAWR5QM2T4MQUBPTHRVQLEAHPBWPQF6Q, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.468334764 (415h3m6.296043147s) [noise=true , address=145.79.227.40:4051 , piece_id=55RRAD7DQX6SR7RBLYAV7MD7XRWPNIWHTVVBQZ2ZL3OE2L7LRRIQ, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.468742525 (415h3m6.296450916s) [noise=true , address=45.91.116.4:4939 , piece_id=QYNKMKDLQJV7B6M2E34DU53Y7YI6VBCUW3C7ZLQOI4XWTBXZHKDQ, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.468915034 (415h3m6.296623414s) [noise=true , address=45.91.116.11:4117 , piece_id=WLMO5E3BIFFL3TOAUSSBGORIZ6DOCWFCMVT4AVFPPGPSKP2OKDAA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.469069393 (415h3m6.296777787s) [noise=true , address=145.79.227.32:4835 , piece_id=BRE3JH3XKPXB6FN7WB3W6ED7MOFES6HYLR34YGR6L5AX7RPIQ7IA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.469285789 (415h3m6.296994183s) [noise=true , address=145.79.227.29:4943 , piece_id=G4TMZ6IAIZBDON3UEDPH4LJQFHV3JLALEXSJ3VI4XN6WHIZOB57Q, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.469411962 (415h3m6.297120355s) [noise=true , address=145.79.227.48:4301 , piece_id=LBIYTJNOX5J4M3HDE7FGF5CSARBITW2AK3P2DSYFLFKUOIHZUKOQ, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.469749379 (415h3m6.297457773s) [noise=true , address=145.79.227.37:4565 , piece_id=554RHTYPG27UNCFPSRMOB245AX4QXL2QFIYECNTVWNNJMAL4BTYA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.470080828 (415h3m6.29778922s) [noise=true , address=145.79.227.43:4157 , piece_id=3W3LVTU37DNHWS52IGIIH7SXWXVNPGEY27RQNT272L3Z64GNR4KA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.470477660 (415h3m6.298186058s) [noise=true , address=45.91.116.14:4190 , piece_id=V4T6Z7K3C75LQEXNQIYZGYWW2CSQ7UPSET46QGDRT52RFGAROTTA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.470567977 (415h3m6.29827637s) [noise=true , address=145.79.227.15:4874 , piece_id=FNCCIMIVJTGZ2X5TOYOZH727JXHYM7HG3HBHBTZDAG67SHZLYYYA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.470980950 (415h3m6.298689349s) [noise=true , address=145.79.227.50:4373 , piece_id=4U55T5QDYCE6AZ3NCTPAVECPAELPQUKT3HV7GZL7JFRMWRN2AZUQ, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.471440098 (415h3m6.299148489s) [noise=true , address=145.79.227.47:4266 , piece_id=3NFYV2PWBNTAOSFHENM4J5PZ7XXE72UKTQVXLTVQRHGK4DUXAKDA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.560766475 (415h3m6.388474871s) [upload=431917]: Next returned nil segment Dec 22 22:31:05.560785161 (415h3m6.388493562s) [upload=431917]: Waiting for error group managing segments... Dec 22 22:31:05.566117878 (415h3m6.39382628s) [noise=true , address=45.91.116.4:4939 , piece_id=QYNKMKDLQJV7B6M2E34DU53Y7YI6VBCUW3C7ZLQOI4XWTBXZHKDQ, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.566723306 (415h3m6.394431687s) [noise=true , address=45.91.116.11:4117 , piece_id=WLMO5E3BIFFL3TOAUSSBGORIZ6DOCWFCMVT4AVFPPGPSKP2OKDAA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.566992667 (415h3m6.394701063s) [noise=true , address=145.79.227.36:4519 , piece_id=JUKRRIZWVGKEDD3GPJ7HMY2MEQZNMUCTKTXR6ZLSIERRONTZKHCQ, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.567464626 (415h3m6.395173019s) [noise=true , address=145.79.227.47:4266 , piece_id=3NFYV2PWBNTAOSFHENM4J5PZ7XXE72UKTQVXLTVQRHGK4DUXAKDA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.567924492 (415h3m6.395632883s) [noise=true , address=145.79.227.15:4874 , piece_id=FNCCIMIVJTGZ2X5TOYOZH727JXHYM7HG3HBHBTZDAG67SHZLYYYA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.568807294 (415h3m6.396515685s) [noise=true , address=145.79.227.51:4409 , piece_id=ATH3F4DFOUUIFFJZRA2SSRRL7AKWZMQNQRYOZCVFFDLTFCFP236A, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.570579823 (415h3m6.398288214s) [noise=true , address=145.79.227.48:4301 , piece_id=LBIYTJNOX5J4M3HDE7FGF5CSARBITW2AK3P2DSYFLFKUOIHZUKOQ, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.571632758 (415h3m6.399341149s) [noise=true , address=145.79.227.43:4157 , piece_id=3W3LVTU37DNHWS52IGIIH7SXWXVNPGEY27RQNT272L3Z64GNR4KA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.573168383 (415h3m6.400876774s) [noise=true , address=145.79.227.38:4979 , piece_id=HLMQC5OEB4OL4OZSRMCEVAWR5QM2T4MQUBPTHRVQLEAHPBWPQF6Q, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.579995420 (415h3m6.407703816s) [noise=true , address=45.91.116.14:4190 , piece_id=V4T6Z7K3C75LQEXNQIYZGYWW2CSQ7UPSET46QGDRT52RFGAROTTA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.592093172 (415h3m6.419801563s) [noise=true , address=145.79.227.37:4565 , piece_id=554RHTYPG27UNCFPSRMOB245AX4QXL2QFIYECNTVWNNJMAL4BTYA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:09.946500434 (415h3m10.774208816s) [noise=true , address=145.79.227.40:4051 , piece_id=55RRAD7DQX6SR7RBLYAV7MD7XRWPNIWHTVVBQZ2ZL3OE2L7LRRIQ, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:35.398235543 (415h3m36.225943943s) [noise=true , address=145.79.227.39:4013 , piece_id=JJVURGSLKY2UQ4FKNCLATA3ZBD5SDL32XJAG42KH2U6OYK3KTS6Q, seg_pos={0 0}, upload=431917]: Done uploading piece. err: ecclient: upload failed (node:121scLvKEW9zw2aEz22zpremcjiop67eQ8v2uqpmaD5aZzCAP7b, address:145.79.227.39:4013): protocol: write tcp 159.223.143.39:60842->145.79.227.39:4013: use of closed network connection; write tcp 159.223.143.39:60842->145.79.227.39:4013: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:60842->145.79.227.39:4013: use of closed network connection Dec 22 22:31:35.398562305 (415h3m36.226270699s) [noise=true , address=145.79.227.32:4835 , piece_id=BRE3JH3XKPXB6FN7WB3W6ED7MOFES6HYLR34YGR6L5AX7RPIQ7IA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: ecclient: upload failed (node:16vVfeoqCkbMYCRjMMWaKBnCvQBBJPMgwQtWTq9HKQrD9SS17y, address:145.79.227.32:4835): protocol: write tcp 159.223.143.39:45098->145.79.227.32:4835: use of closed network connection; write tcp 159.223.143.39:45098->145.79.227.32:4835: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:45098->145.79.227.32:4835: use of closed network connection Dec 22 22:31:35.399183028 (415h3m36.226891422s) [noise=true , address=145.79.227.50:4373 , piece_id=4U55T5QDYCE6AZ3NCTPAVECPAELPQUKT3HV7GZL7JFRMWRN2AZUQ, seg_pos={0 0}, upload=431917]: Done uploading piece. err: ecclient: upload failed (node:12AQGSkJuNJDxoeqhChUenWxHwF5yK41UESCF63FQz3XvabXXfz, address:145.79.227.50:4373): protocol: read tcp 159.223.143.39:56028->145.79.227.50:4373: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 22 22:31:35.399278021 (415h3m36.226986417s) [noise=true , address=145.79.227.29:4943 , piece_id=G4TMZ6IAIZBDON3UEDPH4LJQFHV3JLALEXSJ3VI4XN6WHIZOB57Q, seg_pos={0 0}, upload=431917]: Done uploading piece. err: ecclient: upload failed (node:12fc5Trj8XxacZhzxdksUVT5CbG6uTmiwdUPjreYi4UMPY1TXWz, address:145.79.227.29:4943): protocol: write tcp 159.223.143.39:50820->145.79.227.29:4943: use of closed network connection; write tcp 159.223.143.39:50820->145.79.227.29:4943: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:50820->145.79.227.29:4943: use of closed network connection Dec 22 22:31:35.399379485 (415h3m36.227087866s) [seg_pos={0 0}, upload=431917]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 22 22:31:35.421122544 (415h3m36.248830935s) [upload=431917]: Done uploading segments... Dec 22 22:31:35.421174369 (415h3m36.248882769s) [upload=431917]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 469.659717ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..........X................................................X] duration: 30.024824241s trace: https://jaeger.dev.storj.io/trace/7013ffb1a981d605 FAIL (2025-12-22 22:31:35 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 22 22:31:05.439510308 (415h3m6.2672187s) [upload=431917]: Starting upload Dec 22 22:31:05.439581993 (415h3m6.26729037s) [upload=431917]: Done starting upload Dec 22 22:31:05.439812792 (415h3m6.267521185s) [upload=431917]: Uploading segments... Dec 22 22:31:05.439942524 (415h3m6.267650918s) [upload=431917]: Got next segment. Inline: false Dec 22 22:31:05.440022744 (415h3m6.267731174s) [upload=431917]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 22 22:31:05.440029670 (415h3m6.267738072s) [upload=431917]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 22 22:31:05.467186372 (415h3m6.294894769s) [seg_pos={0 0}, upload=431917]: Begin upload segment... Dec 22 22:31:05.467265149 (415h3m6.294973535s) [seg_pos={0 0}, upload=431917]: Done begin upload segment. Dec 22 22:31:05.467296974 (415h3m6.295005373s) [noise=true , address=145.79.227.51:4409 , piece_id=ATH3F4DFOUUIFFJZRA2SSRRL7AKWZMQNQRYOZCVFFDLTFCFP236A, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.467389853 (415h3m6.295098247s) [noise=true , address=145.79.227.39:4013 , piece_id=JJVURGSLKY2UQ4FKNCLATA3ZBD5SDL32XJAG42KH2U6OYK3KTS6Q, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.467912104 (415h3m6.295620486s) [noise=true , address=145.79.227.36:4519 , piece_id=JUKRRIZWVGKEDD3GPJ7HMY2MEQZNMUCTKTXR6ZLSIERRONTZKHCQ, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.468324863 (415h3m6.296033263s) [noise=true , address=145.79.227.38:4979 , piece_id=HLMQC5OEB4OL4OZSRMCEVAWR5QM2T4MQUBPTHRVQLEAHPBWPQF6Q, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.468334764 (415h3m6.296043147s) [noise=true , address=145.79.227.40:4051 , piece_id=55RRAD7DQX6SR7RBLYAV7MD7XRWPNIWHTVVBQZ2ZL3OE2L7LRRIQ, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.468742525 (415h3m6.296450916s) [noise=true , address=45.91.116.4:4939 , piece_id=QYNKMKDLQJV7B6M2E34DU53Y7YI6VBCUW3C7ZLQOI4XWTBXZHKDQ, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.468915034 (415h3m6.296623414s) [noise=true , address=45.91.116.11:4117 , piece_id=WLMO5E3BIFFL3TOAUSSBGORIZ6DOCWFCMVT4AVFPPGPSKP2OKDAA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.469069393 (415h3m6.296777787s) [noise=true , address=145.79.227.32:4835 , piece_id=BRE3JH3XKPXB6FN7WB3W6ED7MOFES6HYLR34YGR6L5AX7RPIQ7IA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.469285789 (415h3m6.296994183s) [noise=true , address=145.79.227.29:4943 , piece_id=G4TMZ6IAIZBDON3UEDPH4LJQFHV3JLALEXSJ3VI4XN6WHIZOB57Q, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.469411962 (415h3m6.297120355s) [noise=true , address=145.79.227.48:4301 , piece_id=LBIYTJNOX5J4M3HDE7FGF5CSARBITW2AK3P2DSYFLFKUOIHZUKOQ, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.469749379 (415h3m6.297457773s) [noise=true , address=145.79.227.37:4565 , piece_id=554RHTYPG27UNCFPSRMOB245AX4QXL2QFIYECNTVWNNJMAL4BTYA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.470080828 (415h3m6.29778922s) [noise=true , address=145.79.227.43:4157 , piece_id=3W3LVTU37DNHWS52IGIIH7SXWXVNPGEY27RQNT272L3Z64GNR4KA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.470477660 (415h3m6.298186058s) [noise=true , address=45.91.116.14:4190 , piece_id=V4T6Z7K3C75LQEXNQIYZGYWW2CSQ7UPSET46QGDRT52RFGAROTTA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.470567977 (415h3m6.29827637s) [noise=true , address=145.79.227.15:4874 , piece_id=FNCCIMIVJTGZ2X5TOYOZH727JXHYM7HG3HBHBTZDAG67SHZLYYYA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.470980950 (415h3m6.298689349s) [noise=true , address=145.79.227.50:4373 , piece_id=4U55T5QDYCE6AZ3NCTPAVECPAELPQUKT3HV7GZL7JFRMWRN2AZUQ, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.471440098 (415h3m6.299148489s) [noise=true , address=145.79.227.47:4266 , piece_id=3NFYV2PWBNTAOSFHENM4J5PZ7XXE72UKTQVXLTVQRHGK4DUXAKDA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.560766475 (415h3m6.388474871s) [upload=431917]: Next returned nil segment Dec 22 22:31:05.560785161 (415h3m6.388493562s) [upload=431917]: Waiting for error group managing segments... Dec 22 22:31:05.566117878 (415h3m6.39382628s) [noise=true , address=45.91.116.4:4939 , piece_id=QYNKMKDLQJV7B6M2E34DU53Y7YI6VBCUW3C7ZLQOI4XWTBXZHKDQ, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.566723306 (415h3m6.394431687s) [noise=true , address=45.91.116.11:4117 , piece_id=WLMO5E3BIFFL3TOAUSSBGORIZ6DOCWFCMVT4AVFPPGPSKP2OKDAA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.566992667 (415h3m6.394701063s) [noise=true , address=145.79.227.36:4519 , piece_id=JUKRRIZWVGKEDD3GPJ7HMY2MEQZNMUCTKTXR6ZLSIERRONTZKHCQ, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.567464626 (415h3m6.395173019s) [noise=true , address=145.79.227.47:4266 , piece_id=3NFYV2PWBNTAOSFHENM4J5PZ7XXE72UKTQVXLTVQRHGK4DUXAKDA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.567924492 (415h3m6.395632883s) [noise=true , address=145.79.227.15:4874 , piece_id=FNCCIMIVJTGZ2X5TOYOZH727JXHYM7HG3HBHBTZDAG67SHZLYYYA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.568807294 (415h3m6.396515685s) [noise=true , address=145.79.227.51:4409 , piece_id=ATH3F4DFOUUIFFJZRA2SSRRL7AKWZMQNQRYOZCVFFDLTFCFP236A, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.570579823 (415h3m6.398288214s) [noise=true , address=145.79.227.48:4301 , piece_id=LBIYTJNOX5J4M3HDE7FGF5CSARBITW2AK3P2DSYFLFKUOIHZUKOQ, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.571632758 (415h3m6.399341149s) [noise=true , address=145.79.227.43:4157 , piece_id=3W3LVTU37DNHWS52IGIIH7SXWXVNPGEY27RQNT272L3Z64GNR4KA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.573168383 (415h3m6.400876774s) [noise=true , address=145.79.227.38:4979 , piece_id=HLMQC5OEB4OL4OZSRMCEVAWR5QM2T4MQUBPTHRVQLEAHPBWPQF6Q, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.579995420 (415h3m6.407703816s) [noise=true , address=45.91.116.14:4190 , piece_id=V4T6Z7K3C75LQEXNQIYZGYWW2CSQ7UPSET46QGDRT52RFGAROTTA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.592093172 (415h3m6.419801563s) [noise=true , address=145.79.227.37:4565 , piece_id=554RHTYPG27UNCFPSRMOB245AX4QXL2QFIYECNTVWNNJMAL4BTYA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:09.946500434 (415h3m10.774208816s) [noise=true , address=145.79.227.40:4051 , piece_id=55RRAD7DQX6SR7RBLYAV7MD7XRWPNIWHTVVBQZ2ZL3OE2L7LRRIQ, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:35.398235543 (415h3m36.225943943s) [noise=true , address=145.79.227.39:4013 , piece_id=JJVURGSLKY2UQ4FKNCLATA3ZBD5SDL32XJAG42KH2U6OYK3KTS6Q, seg_pos={0 0}, upload=431917]: Done uploading piece. err: ecclient: upload failed (node:121scLvKEW9zw2aEz22zpremcjiop67eQ8v2uqpmaD5aZzCAP7b, address:145.79.227.39:4013): protocol: write tcp 159.223.143.39:60842->145.79.227.39:4013: use of closed network connection; write tcp 159.223.143.39:60842->145.79.227.39:4013: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:60842->145.79.227.39:4013: use of closed network connection Dec 22 22:31:35.398562305 (415h3m36.226270699s) [noise=true , address=145.79.227.32:4835 , piece_id=BRE3JH3XKPXB6FN7WB3W6ED7MOFES6HYLR34YGR6L5AX7RPIQ7IA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: ecclient: upload failed (node:16vVfeoqCkbMYCRjMMWaKBnCvQBBJPMgwQtWTq9HKQrD9SS17y, address:145.79.227.32:4835): protocol: write tcp 159.223.143.39:45098->145.79.227.32:4835: use of closed network connection; write tcp 159.223.143.39:45098->145.79.227.32:4835: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:45098->145.79.227.32:4835: use of closed network connection Dec 22 22:31:35.399183028 (415h3m36.226891422s) [noise=true , address=145.79.227.50:4373 , piece_id=4U55T5QDYCE6AZ3NCTPAVECPAELPQUKT3HV7GZL7JFRMWRN2AZUQ, seg_pos={0 0}, upload=431917]: Done uploading piece. err: ecclient: upload failed (node:12AQGSkJuNJDxoeqhChUenWxHwF5yK41UESCF63FQz3XvabXXfz, address:145.79.227.50:4373): protocol: read tcp 159.223.143.39:56028->145.79.227.50:4373: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 22 22:31:35.399278021 (415h3m36.226986417s) [noise=true , address=145.79.227.29:4943 , piece_id=G4TMZ6IAIZBDON3UEDPH4LJQFHV3JLALEXSJ3VI4XN6WHIZOB57Q, seg_pos={0 0}, upload=431917]: Done uploading piece. err: ecclient: upload failed (node:12fc5Trj8XxacZhzxdksUVT5CbG6uTmiwdUPjreYi4UMPY1TXWz, address:145.79.227.29:4943): protocol: write tcp 159.223.143.39:50820->145.79.227.29:4943: use of closed network connection; write tcp 159.223.143.39:50820->145.79.227.29:4943: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:50820->145.79.227.29:4943: use of closed network connection Dec 22 22:31:35.399379485 (415h3m36.227087866s) [seg_pos={0 0}, upload=431917]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 22 22:31:35.421122544 (415h3m36.248830935s) [upload=431917]: Done uploading segments... Dec 22 22:31:35.421174369 (415h3m36.248882769s) [upload=431917]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 30.0248467s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........X................................................] duration: 487.997493ms trace: https://jaeger.dev.storj.io/trace/17ef15eb0c5881c1 FAIL (2025-12-22 22:29:52 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 22 21:52:50.192390642 (414h24m51.020099033s) [upload=431251]: Starting upload Dec 22 21:52:50.192488257 (414h24m51.020196648s) [upload=431251]: Done starting upload Dec 22 21:52:50.192531865 (414h24m51.020240259s) [upload=431251]: Uploading segments... Dec 22 21:52:50.192783054 (414h24m51.020491446s) [upload=431251]: Got next segment. Inline: false Dec 22 21:52:50.192836048 (414h24m51.020544439s) [upload=431251]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 22 21:52:50.192841417 (414h24m51.020549808s) [upload=431251]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 22 21:52:50.220288008 (414h24m51.047996404s) [seg_pos={0 0}, upload=431251]: Begin upload segment... Dec 22 21:52:50.220342100 (414h24m51.048050497s) [seg_pos={0 0}, upload=431251]: Done begin upload segment. Dec 22 21:52:50.220384745 (414h24m51.048093145s) [noise=true , address=77.90.31.10:4517 , piece_id=UAIKGPQF75AM6EKYBV3XK45CNYS2N2PNHKBAKRKO6DDILYAUFBLA, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.220421097 (414h24m51.048129475s) [noise=true , address=145.79.227.45:4229 , piece_id=QYCGIYFFNWLGOBOKEZYY4EBLHKC3SFYWOBQJ34DETCLF6BTS54VQ, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.221068129 (414h24m51.048776527s) [noise=true , address=145.79.227.35:4483 , piece_id=EJIXZOBJWPIVL37GIYYPHQRZDMEMS3YSRPVJH4GLMB4A2CSSRCTQ, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.221771875 (414h24m51.049480268s) [noise=true , address=145.79.227.51:4411 , piece_id=O76ELDU3V3XZ2ZYC7VCLZC2CU5VUMRTFWJEV2ITS5GFAGRSXTOSA, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.222308205 (414h24m51.050016602s) [noise=true , address=145.79.227.39:4015 , piece_id=AZSOOT5RLGTAHSO47SVO27XDSNFPE65JU2I5UEZDPQ44PKB2YLYA, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.222424843 (414h24m51.050133244s) [noise=true , address=77.90.31.18:4805 , piece_id=YWS2P6F2X4PRKE7FVEA722G4FGDV2D4BXJILLHGQE4Q2HXRU2SAA, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.222721983 (414h24m51.050430374s) [noise=true , address=145.79.227.28:4727 , piece_id=Y5UCAOYDWNY5WPZC7LYUOT5QB325BSVZJUGSRVMR64ML23SEI4BQ, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.222910142 (414h24m51.050618542s) [noise=true , address=145.79.227.37:4566 , piece_id=EKCKTUJE37ZDSKNA6JM2ODSBLYAI2KFCA5RVFRNJPSCQ72ZWKY6A, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.223093651 (414h24m51.050802048s) [noise=true , address=145.79.227.50:4373 , piece_id=HESOK4IM7VRJNRL6NY5U4V52G7PDLSE3L3ZOF7DSTSQGFP6D2ZQA, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.223601176 (414h24m51.051309571s) [noise=true , address=145.79.227.15:4876 , piece_id=X5KN53INJAXSFQUEY2S37RYA3ELNBGN6ZFM6KKJ3A4ROOSUTBTRQ, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.224086995 (414h24m51.051795377s) [noise=true , address=145.79.227.43:4157 , piece_id=EKE6ZQ6YMYBBKZKQ2OIAAF5OI3OZRPJF4XDYCCU7QSKS33UXPDZQ, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.224569909 (414h24m51.052278304s) [noise=true , address=77.90.31.11:4553 , piece_id=MJFMBIGW3L6E75AAMZRCDONHZXPFLQSCK3VINMI4I2ZJC5HVHREA, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.225250077 (414h24m51.052958472s) [noise=true , address=145.79.227.48:4301 , piece_id=OCJPU5E7QLJWVY5K444J3J3ISMFY5E6MVXRMXBHIQRHXMIMRYC2A, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.226504914 (414h24m51.054213295s) [noise=true , address=145.79.227.47:4266 , piece_id=EJYUC5ODHWWQEA7QK5JVT3T3HQ76G5WC5OHPNGUSYMP4AW5BWLPA, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.227351850 (414h24m51.055060238s) [noise=true , address=145.79.227.38:4978 , piece_id=S26D5M7SYZEXTTNQEELKEFFQTOMNA5SG7RL5THCON6KCFY267IDQ, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.228140124 (414h24m51.055848511s) [noise=true , address=145.79.227.32:4834 , piece_id=KLWKSAGBRLFWDKNCHW6UWKCADPZ6D6UURDMNMD5UCZUUWAPDOKVQ, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.321476218 (414h24m51.149184612s) [upload=431251]: Next returned nil segment Dec 22 21:52:50.321492753 (414h24m51.149201151s) [upload=431251]: Waiting for error group managing segments... Dec 22 21:52:50.328186730 (414h24m51.155895125s) [noise=true , address=77.90.31.11:4553 , piece_id=MJFMBIGW3L6E75AAMZRCDONHZXPFLQSCK3VINMI4I2ZJC5HVHREA, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.328400062 (414h24m51.156108454s) [noise=true , address=145.79.227.47:4266 , piece_id=EJYUC5ODHWWQEA7QK5JVT3T3HQ76G5WC5OHPNGUSYMP4AW5BWLPA, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.328834800 (414h24m51.156543191s) [noise=true , address=145.79.227.37:4566 , piece_id=EKCKTUJE37ZDSKNA6JM2ODSBLYAI2KFCA5RVFRNJPSCQ72ZWKY6A, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.330369052 (414h24m51.158077447s) [noise=true , address=145.79.227.51:4411 , piece_id=O76ELDU3V3XZ2ZYC7VCLZC2CU5VUMRTFWJEV2ITS5GFAGRSXTOSA, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.330425878 (414h24m51.158134271s) [noise=true , address=145.79.227.28:4727 , piece_id=Y5UCAOYDWNY5WPZC7LYUOT5QB325BSVZJUGSRVMR64ML23SEI4BQ, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.331015687 (414h24m51.158724085s) [noise=true , address=145.79.227.45:4229 , piece_id=QYCGIYFFNWLGOBOKEZYY4EBLHKC3SFYWOBQJ34DETCLF6BTS54VQ, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.331368793 (414h24m51.159077188s) [noise=true , address=145.79.227.48:4301 , piece_id=OCJPU5E7QLJWVY5K444J3J3ISMFY5E6MVXRMXBHIQRHXMIMRYC2A, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.332233159 (414h24m51.159941554s) [noise=true , address=145.79.227.50:4373 , piece_id=HESOK4IM7VRJNRL6NY5U4V52G7PDLSE3L3ZOF7DSTSQGFP6D2ZQA, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.333040553 (414h24m51.160748944s) [noise=true , address=145.79.227.35:4483 , piece_id=EJIXZOBJWPIVL37GIYYPHQRZDMEMS3YSRPVJH4GLMB4A2CSSRCTQ, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.333195450 (414h24m51.160903848s) [noise=true , address=145.79.227.43:4157 , piece_id=EKE6ZQ6YMYBBKZKQ2OIAAF5OI3OZRPJF4XDYCCU7QSKS33UXPDZQ, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.334537739 (414h24m51.162246133s) [noise=true , address=145.79.227.39:4015 , piece_id=AZSOOT5RLGTAHSO47SVO27XDSNFPE65JU2I5UEZDPQ44PKB2YLYA, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.413477504 (414h24m51.2411859s) [noise=true , address=77.90.31.10:4517 , piece_id=UAIKGPQF75AM6EKYBV3XK45CNYS2N2PNHKBAKRKO6DDILYAUFBLA, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:53:20.161293124 (414h25m20.989001514s) [noise=true , address=77.90.31.18:4805 , piece_id=YWS2P6F2X4PRKE7FVEA722G4FGDV2D4BXJILLHGQE4Q2HXRU2SAA, seg_pos={0 0}, upload=431251]: Done uploading piece. err: ecclient: upload failed (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, address:77.90.31.18:4805): protocol: expected piece hash; context deadline exceeded; EOF Dec 22 21:53:20.161472578 (414h25m20.989180974s) [noise=true , address=145.79.227.38:4978 , piece_id=S26D5M7SYZEXTTNQEELKEFFQTOMNA5SG7RL5THCON6KCFY267IDQ, seg_pos={0 0}, upload=431251]: Done uploading piece. err: ecclient: upload failed (node:12PcGHVSm5UAXPh5vCfyRHCd7Bdb67oXF51AZcnRFUjXqVYzXLR, address:145.79.227.38:4978): protocol: expected piece hash; context deadline exceeded; EOF Dec 22 21:53:20.161591128 (414h25m20.989299529s) [noise=true , address=145.79.227.32:4834 , piece_id=KLWKSAGBRLFWDKNCHW6UWKCADPZ6D6UURDMNMD5UCZUUWAPDOKVQ, seg_pos={0 0}, upload=431251]: Done uploading piece. err: ecclient: upload failed (node:1gRnamgAa2ccZS6F44apRmzSkXC6jGvNreJ5J8Y9nciWPZCSsk, address:145.79.227.32:4834): protocol: expected piece hash; context deadline exceeded; EOF Dec 22 21:53:20.161797225 (414h25m20.989505624s) [noise=true , address=145.79.227.15:4876 , piece_id=X5KN53INJAXSFQUEY2S37RYA3ELNBGN6ZFM6KKJ3A4ROOSUTBTRQ, seg_pos={0 0}, upload=431251]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:145.79.227.15:4876): protocol: expected piece hash; context deadline exceeded; EOF Dec 22 21:53:20.161824434 (414h25m20.989532813s) [seg_pos={0 0}, upload=431251]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 22 21:53:20.183925866 (414h25m21.011634264s) [upload=431251]: Done uploading segments... Dec 22 21:53:20.184019830 (414h25m21.011728235s) [upload=431251]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 488.091813ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............X...............................................] duration: 456.554653ms trace: https://jaeger.dev.storj.io/trace/17ef15eb0c5881c1 FAIL (2025-12-22 22:29:16 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 22 21:52:50.192390642 (414h24m51.020099033s) [upload=431251]: Starting upload Dec 22 21:52:50.192488257 (414h24m51.020196648s) [upload=431251]: Done starting upload Dec 22 21:52:50.192531865 (414h24m51.020240259s) [upload=431251]: Uploading segments... Dec 22 21:52:50.192783054 (414h24m51.020491446s) [upload=431251]: Got next segment. Inline: false Dec 22 21:52:50.192836048 (414h24m51.020544439s) [upload=431251]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 22 21:52:50.192841417 (414h24m51.020549808s) [upload=431251]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 22 21:52:50.220288008 (414h24m51.047996404s) [seg_pos={0 0}, upload=431251]: Begin upload segment... Dec 22 21:52:50.220342100 (414h24m51.048050497s) [seg_pos={0 0}, upload=431251]: Done begin upload segment. Dec 22 21:52:50.220384745 (414h24m51.048093145s) [noise=true , address=77.90.31.10:4517 , piece_id=UAIKGPQF75AM6EKYBV3XK45CNYS2N2PNHKBAKRKO6DDILYAUFBLA, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.220421097 (414h24m51.048129475s) [noise=true , address=145.79.227.45:4229 , piece_id=QYCGIYFFNWLGOBOKEZYY4EBLHKC3SFYWOBQJ34DETCLF6BTS54VQ, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.221068129 (414h24m51.048776527s) [noise=true , address=145.79.227.35:4483 , piece_id=EJIXZOBJWPIVL37GIYYPHQRZDMEMS3YSRPVJH4GLMB4A2CSSRCTQ, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.221771875 (414h24m51.049480268s) [noise=true , address=145.79.227.51:4411 , piece_id=O76ELDU3V3XZ2ZYC7VCLZC2CU5VUMRTFWJEV2ITS5GFAGRSXTOSA, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.222308205 (414h24m51.050016602s) [noise=true , address=145.79.227.39:4015 , piece_id=AZSOOT5RLGTAHSO47SVO27XDSNFPE65JU2I5UEZDPQ44PKB2YLYA, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.222424843 (414h24m51.050133244s) [noise=true , address=77.90.31.18:4805 , piece_id=YWS2P6F2X4PRKE7FVEA722G4FGDV2D4BXJILLHGQE4Q2HXRU2SAA, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.222721983 (414h24m51.050430374s) [noise=true , address=145.79.227.28:4727 , piece_id=Y5UCAOYDWNY5WPZC7LYUOT5QB325BSVZJUGSRVMR64ML23SEI4BQ, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.222910142 (414h24m51.050618542s) [noise=true , address=145.79.227.37:4566 , piece_id=EKCKTUJE37ZDSKNA6JM2ODSBLYAI2KFCA5RVFRNJPSCQ72ZWKY6A, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.223093651 (414h24m51.050802048s) [noise=true , address=145.79.227.50:4373 , piece_id=HESOK4IM7VRJNRL6NY5U4V52G7PDLSE3L3ZOF7DSTSQGFP6D2ZQA, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.223601176 (414h24m51.051309571s) [noise=true , address=145.79.227.15:4876 , piece_id=X5KN53INJAXSFQUEY2S37RYA3ELNBGN6ZFM6KKJ3A4ROOSUTBTRQ, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.224086995 (414h24m51.051795377s) [noise=true , address=145.79.227.43:4157 , piece_id=EKE6ZQ6YMYBBKZKQ2OIAAF5OI3OZRPJF4XDYCCU7QSKS33UXPDZQ, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.224569909 (414h24m51.052278304s) [noise=true , address=77.90.31.11:4553 , piece_id=MJFMBIGW3L6E75AAMZRCDONHZXPFLQSCK3VINMI4I2ZJC5HVHREA, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.225250077 (414h24m51.052958472s) [noise=true , address=145.79.227.48:4301 , piece_id=OCJPU5E7QLJWVY5K444J3J3ISMFY5E6MVXRMXBHIQRHXMIMRYC2A, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.226504914 (414h24m51.054213295s) [noise=true , address=145.79.227.47:4266 , piece_id=EJYUC5ODHWWQEA7QK5JVT3T3HQ76G5WC5OHPNGUSYMP4AW5BWLPA, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.227351850 (414h24m51.055060238s) [noise=true , address=145.79.227.38:4978 , piece_id=S26D5M7SYZEXTTNQEELKEFFQTOMNA5SG7RL5THCON6KCFY267IDQ, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.228140124 (414h24m51.055848511s) [noise=true , address=145.79.227.32:4834 , piece_id=KLWKSAGBRLFWDKNCHW6UWKCADPZ6D6UURDMNMD5UCZUUWAPDOKVQ, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.321476218 (414h24m51.149184612s) [upload=431251]: Next returned nil segment Dec 22 21:52:50.321492753 (414h24m51.149201151s) [upload=431251]: Waiting for error group managing segments... Dec 22 21:52:50.328186730 (414h24m51.155895125s) [noise=true , address=77.90.31.11:4553 , piece_id=MJFMBIGW3L6E75AAMZRCDONHZXPFLQSCK3VINMI4I2ZJC5HVHREA, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.328400062 (414h24m51.156108454s) [noise=true , address=145.79.227.47:4266 , piece_id=EJYUC5ODHWWQEA7QK5JVT3T3HQ76G5WC5OHPNGUSYMP4AW5BWLPA, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.328834800 (414h24m51.156543191s) [noise=true , address=145.79.227.37:4566 , piece_id=EKCKTUJE37ZDSKNA6JM2ODSBLYAI2KFCA5RVFRNJPSCQ72ZWKY6A, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.330369052 (414h24m51.158077447s) [noise=true , address=145.79.227.51:4411 , piece_id=O76ELDU3V3XZ2ZYC7VCLZC2CU5VUMRTFWJEV2ITS5GFAGRSXTOSA, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.330425878 (414h24m51.158134271s) [noise=true , address=145.79.227.28:4727 , piece_id=Y5UCAOYDWNY5WPZC7LYUOT5QB325BSVZJUGSRVMR64ML23SEI4BQ, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.331015687 (414h24m51.158724085s) [noise=true , address=145.79.227.45:4229 , piece_id=QYCGIYFFNWLGOBOKEZYY4EBLHKC3SFYWOBQJ34DETCLF6BTS54VQ, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.331368793 (414h24m51.159077188s) [noise=true , address=145.79.227.48:4301 , piece_id=OCJPU5E7QLJWVY5K444J3J3ISMFY5E6MVXRMXBHIQRHXMIMRYC2A, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.332233159 (414h24m51.159941554s) [noise=true , address=145.79.227.50:4373 , piece_id=HESOK4IM7VRJNRL6NY5U4V52G7PDLSE3L3ZOF7DSTSQGFP6D2ZQA, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.333040553 (414h24m51.160748944s) [noise=true , address=145.79.227.35:4483 , piece_id=EJIXZOBJWPIVL37GIYYPHQRZDMEMS3YSRPVJH4GLMB4A2CSSRCTQ, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.333195450 (414h24m51.160903848s) [noise=true , address=145.79.227.43:4157 , piece_id=EKE6ZQ6YMYBBKZKQ2OIAAF5OI3OZRPJF4XDYCCU7QSKS33UXPDZQ, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.334537739 (414h24m51.162246133s) [noise=true , address=145.79.227.39:4015 , piece_id=AZSOOT5RLGTAHSO47SVO27XDSNFPE65JU2I5UEZDPQ44PKB2YLYA, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.413477504 (414h24m51.2411859s) [noise=true , address=77.90.31.10:4517 , piece_id=UAIKGPQF75AM6EKYBV3XK45CNYS2N2PNHKBAKRKO6DDILYAUFBLA, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:53:20.161293124 (414h25m20.989001514s) [noise=true , address=77.90.31.18:4805 , piece_id=YWS2P6F2X4PRKE7FVEA722G4FGDV2D4BXJILLHGQE4Q2HXRU2SAA, seg_pos={0 0}, upload=431251]: Done uploading piece. err: ecclient: upload failed (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, address:77.90.31.18:4805): protocol: expected piece hash; context deadline exceeded; EOF Dec 22 21:53:20.161472578 (414h25m20.989180974s) [noise=true , address=145.79.227.38:4978 , piece_id=S26D5M7SYZEXTTNQEELKEFFQTOMNA5SG7RL5THCON6KCFY267IDQ, seg_pos={0 0}, upload=431251]: Done uploading piece. err: ecclient: upload failed (node:12PcGHVSm5UAXPh5vCfyRHCd7Bdb67oXF51AZcnRFUjXqVYzXLR, address:145.79.227.38:4978): protocol: expected piece hash; context deadline exceeded; EOF Dec 22 21:53:20.161591128 (414h25m20.989299529s) [noise=true , address=145.79.227.32:4834 , piece_id=KLWKSAGBRLFWDKNCHW6UWKCADPZ6D6UURDMNMD5UCZUUWAPDOKVQ, seg_pos={0 0}, upload=431251]: Done uploading piece. err: ecclient: upload failed (node:1gRnamgAa2ccZS6F44apRmzSkXC6jGvNreJ5J8Y9nciWPZCSsk, address:145.79.227.32:4834): protocol: expected piece hash; context deadline exceeded; EOF Dec 22 21:53:20.161797225 (414h25m20.989505624s) [noise=true , address=145.79.227.15:4876 , piece_id=X5KN53INJAXSFQUEY2S37RYA3ELNBGN6ZFM6KKJ3A4ROOSUTBTRQ, seg_pos={0 0}, upload=431251]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:145.79.227.15:4876): protocol: expected piece hash; context deadline exceeded; EOF Dec 22 21:53:20.161824434 (414h25m20.989532813s) [seg_pos={0 0}, upload=431251]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 22 21:53:20.183925866 (414h25m21.011634264s) [upload=431251]: Done uploading segments... Dec 22 21:53:20.184019830 (414h25m21.011728235s) [upload=431251]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 456.604978ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............X..............................................] duration: 568.864918ms trace: https://jaeger.dev.storj.io/trace/17ef15eb0c5881c1 FAIL (2025-12-22 22:28:58 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 22 21:52:50.192390642 (414h24m51.020099033s) [upload=431251]: Starting upload Dec 22 21:52:50.192488257 (414h24m51.020196648s) [upload=431251]: Done starting upload Dec 22 21:52:50.192531865 (414h24m51.020240259s) [upload=431251]: Uploading segments... Dec 22 21:52:50.192783054 (414h24m51.020491446s) [upload=431251]: Got next segment. Inline: false Dec 22 21:52:50.192836048 (414h24m51.020544439s) [upload=431251]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 22 21:52:50.192841417 (414h24m51.020549808s) [upload=431251]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 22 21:52:50.220288008 (414h24m51.047996404s) [seg_pos={0 0}, upload=431251]: Begin upload segment... Dec 22 21:52:50.220342100 (414h24m51.048050497s) [seg_pos={0 0}, upload=431251]: Done begin upload segment. Dec 22 21:52:50.220384745 (414h24m51.048093145s) [noise=true , address=77.90.31.10:4517 , piece_id=UAIKGPQF75AM6EKYBV3XK45CNYS2N2PNHKBAKRKO6DDILYAUFBLA, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.220421097 (414h24m51.048129475s) [noise=true , address=145.79.227.45:4229 , piece_id=QYCGIYFFNWLGOBOKEZYY4EBLHKC3SFYWOBQJ34DETCLF6BTS54VQ, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.221068129 (414h24m51.048776527s) [noise=true , address=145.79.227.35:4483 , piece_id=EJIXZOBJWPIVL37GIYYPHQRZDMEMS3YSRPVJH4GLMB4A2CSSRCTQ, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.221771875 (414h24m51.049480268s) [noise=true , address=145.79.227.51:4411 , piece_id=O76ELDU3V3XZ2ZYC7VCLZC2CU5VUMRTFWJEV2ITS5GFAGRSXTOSA, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.222308205 (414h24m51.050016602s) [noise=true , address=145.79.227.39:4015 , piece_id=AZSOOT5RLGTAHSO47SVO27XDSNFPE65JU2I5UEZDPQ44PKB2YLYA, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.222424843 (414h24m51.050133244s) [noise=true , address=77.90.31.18:4805 , piece_id=YWS2P6F2X4PRKE7FVEA722G4FGDV2D4BXJILLHGQE4Q2HXRU2SAA, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.222721983 (414h24m51.050430374s) [noise=true , address=145.79.227.28:4727 , piece_id=Y5UCAOYDWNY5WPZC7LYUOT5QB325BSVZJUGSRVMR64ML23SEI4BQ, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.222910142 (414h24m51.050618542s) [noise=true , address=145.79.227.37:4566 , piece_id=EKCKTUJE37ZDSKNA6JM2ODSBLYAI2KFCA5RVFRNJPSCQ72ZWKY6A, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.223093651 (414h24m51.050802048s) [noise=true , address=145.79.227.50:4373 , piece_id=HESOK4IM7VRJNRL6NY5U4V52G7PDLSE3L3ZOF7DSTSQGFP6D2ZQA, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.223601176 (414h24m51.051309571s) [noise=true , address=145.79.227.15:4876 , piece_id=X5KN53INJAXSFQUEY2S37RYA3ELNBGN6ZFM6KKJ3A4ROOSUTBTRQ, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.224086995 (414h24m51.051795377s) [noise=true , address=145.79.227.43:4157 , piece_id=EKE6ZQ6YMYBBKZKQ2OIAAF5OI3OZRPJF4XDYCCU7QSKS33UXPDZQ, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.224569909 (414h24m51.052278304s) [noise=true , address=77.90.31.11:4553 , piece_id=MJFMBIGW3L6E75AAMZRCDONHZXPFLQSCK3VINMI4I2ZJC5HVHREA, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.225250077 (414h24m51.052958472s) [noise=true , address=145.79.227.48:4301 , piece_id=OCJPU5E7QLJWVY5K444J3J3ISMFY5E6MVXRMXBHIQRHXMIMRYC2A, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.226504914 (414h24m51.054213295s) [noise=true , address=145.79.227.47:4266 , piece_id=EJYUC5ODHWWQEA7QK5JVT3T3HQ76G5WC5OHPNGUSYMP4AW5BWLPA, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.227351850 (414h24m51.055060238s) [noise=true , address=145.79.227.38:4978 , piece_id=S26D5M7SYZEXTTNQEELKEFFQTOMNA5SG7RL5THCON6KCFY267IDQ, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.228140124 (414h24m51.055848511s) [noise=true , address=145.79.227.32:4834 , piece_id=KLWKSAGBRLFWDKNCHW6UWKCADPZ6D6UURDMNMD5UCZUUWAPDOKVQ, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.321476218 (414h24m51.149184612s) [upload=431251]: Next returned nil segment Dec 22 21:52:50.321492753 (414h24m51.149201151s) [upload=431251]: Waiting for error group managing segments... Dec 22 21:52:50.328186730 (414h24m51.155895125s) [noise=true , address=77.90.31.11:4553 , piece_id=MJFMBIGW3L6E75AAMZRCDONHZXPFLQSCK3VINMI4I2ZJC5HVHREA, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.328400062 (414h24m51.156108454s) [noise=true , address=145.79.227.47:4266 , piece_id=EJYUC5ODHWWQEA7QK5JVT3T3HQ76G5WC5OHPNGUSYMP4AW5BWLPA, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.328834800 (414h24m51.156543191s) [noise=true , address=145.79.227.37:4566 , piece_id=EKCKTUJE37ZDSKNA6JM2ODSBLYAI2KFCA5RVFRNJPSCQ72ZWKY6A, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.330369052 (414h24m51.158077447s) [noise=true , address=145.79.227.51:4411 , piece_id=O76ELDU3V3XZ2ZYC7VCLZC2CU5VUMRTFWJEV2ITS5GFAGRSXTOSA, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.330425878 (414h24m51.158134271s) [noise=true , address=145.79.227.28:4727 , piece_id=Y5UCAOYDWNY5WPZC7LYUOT5QB325BSVZJUGSRVMR64ML23SEI4BQ, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.331015687 (414h24m51.158724085s) [noise=true , address=145.79.227.45:4229 , piece_id=QYCGIYFFNWLGOBOKEZYY4EBLHKC3SFYWOBQJ34DETCLF6BTS54VQ, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.331368793 (414h24m51.159077188s) [noise=true , address=145.79.227.48:4301 , piece_id=OCJPU5E7QLJWVY5K444J3J3ISMFY5E6MVXRMXBHIQRHXMIMRYC2A, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.332233159 (414h24m51.159941554s) [noise=true , address=145.79.227.50:4373 , piece_id=HESOK4IM7VRJNRL6NY5U4V52G7PDLSE3L3ZOF7DSTSQGFP6D2ZQA, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.333040553 (414h24m51.160748944s) [noise=true , address=145.79.227.35:4483 , piece_id=EJIXZOBJWPIVL37GIYYPHQRZDMEMS3YSRPVJH4GLMB4A2CSSRCTQ, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.333195450 (414h24m51.160903848s) [noise=true , address=145.79.227.43:4157 , piece_id=EKE6ZQ6YMYBBKZKQ2OIAAF5OI3OZRPJF4XDYCCU7QSKS33UXPDZQ, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.334537739 (414h24m51.162246133s) [noise=true , address=145.79.227.39:4015 , piece_id=AZSOOT5RLGTAHSO47SVO27XDSNFPE65JU2I5UEZDPQ44PKB2YLYA, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.413477504 (414h24m51.2411859s) [noise=true , address=77.90.31.10:4517 , piece_id=UAIKGPQF75AM6EKYBV3XK45CNYS2N2PNHKBAKRKO6DDILYAUFBLA, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:53:20.161293124 (414h25m20.989001514s) [noise=true , address=77.90.31.18:4805 , piece_id=YWS2P6F2X4PRKE7FVEA722G4FGDV2D4BXJILLHGQE4Q2HXRU2SAA, seg_pos={0 0}, upload=431251]: Done uploading piece. err: ecclient: upload failed (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, address:77.90.31.18:4805): protocol: expected piece hash; context deadline exceeded; EOF Dec 22 21:53:20.161472578 (414h25m20.989180974s) [noise=true , address=145.79.227.38:4978 , piece_id=S26D5M7SYZEXTTNQEELKEFFQTOMNA5SG7RL5THCON6KCFY267IDQ, seg_pos={0 0}, upload=431251]: Done uploading piece. err: ecclient: upload failed (node:12PcGHVSm5UAXPh5vCfyRHCd7Bdb67oXF51AZcnRFUjXqVYzXLR, address:145.79.227.38:4978): protocol: expected piece hash; context deadline exceeded; EOF Dec 22 21:53:20.161591128 (414h25m20.989299529s) [noise=true , address=145.79.227.32:4834 , piece_id=KLWKSAGBRLFWDKNCHW6UWKCADPZ6D6UURDMNMD5UCZUUWAPDOKVQ, seg_pos={0 0}, upload=431251]: Done uploading piece. err: ecclient: upload failed (node:1gRnamgAa2ccZS6F44apRmzSkXC6jGvNreJ5J8Y9nciWPZCSsk, address:145.79.227.32:4834): protocol: expected piece hash; context deadline exceeded; EOF Dec 22 21:53:20.161797225 (414h25m20.989505624s) [noise=true , address=145.79.227.15:4876 , piece_id=X5KN53INJAXSFQUEY2S37RYA3ELNBGN6ZFM6KKJ3A4ROOSUTBTRQ, seg_pos={0 0}, upload=431251]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:145.79.227.15:4876): protocol: expected piece hash; context deadline exceeded; EOF Dec 22 21:53:20.161824434 (414h25m20.989532813s) [seg_pos={0 0}, upload=431251]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 22 21:53:20.183925866 (414h25m21.011634264s) [upload=431251]: Done uploading segments... Dec 22 21:53:20.184019830 (414h25m21.011728235s) [upload=431251]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 568.890366ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............X.............................................] duration: 451.890603ms trace: https://jaeger.dev.storj.io/trace/17ef15eb0c5881c1 FAIL (2025-12-22 22:28:03 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 22 21:52:50.192390642 (414h24m51.020099033s) [upload=431251]: Starting upload Dec 22 21:52:50.192488257 (414h24m51.020196648s) [upload=431251]: Done starting upload Dec 22 21:52:50.192531865 (414h24m51.020240259s) [upload=431251]: Uploading segments... Dec 22 21:52:50.192783054 (414h24m51.020491446s) [upload=431251]: Got next segment. Inline: false Dec 22 21:52:50.192836048 (414h24m51.020544439s) [upload=431251]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 22 21:52:50.192841417 (414h24m51.020549808s) [upload=431251]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 22 21:52:50.220288008 (414h24m51.047996404s) [seg_pos={0 0}, upload=431251]: Begin upload segment... Dec 22 21:52:50.220342100 (414h24m51.048050497s) [seg_pos={0 0}, upload=431251]: Done begin upload segment. Dec 22 21:52:50.220384745 (414h24m51.048093145s) [noise=true , address=77.90.31.10:4517 , piece_id=UAIKGPQF75AM6EKYBV3XK45CNYS2N2PNHKBAKRKO6DDILYAUFBLA, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.220421097 (414h24m51.048129475s) [noise=true , address=145.79.227.45:4229 , piece_id=QYCGIYFFNWLGOBOKEZYY4EBLHKC3SFYWOBQJ34DETCLF6BTS54VQ, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.221068129 (414h24m51.048776527s) [noise=true , address=145.79.227.35:4483 , piece_id=EJIXZOBJWPIVL37GIYYPHQRZDMEMS3YSRPVJH4GLMB4A2CSSRCTQ, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.221771875 (414h24m51.049480268s) [noise=true , address=145.79.227.51:4411 , piece_id=O76ELDU3V3XZ2ZYC7VCLZC2CU5VUMRTFWJEV2ITS5GFAGRSXTOSA, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.222308205 (414h24m51.050016602s) [noise=true , address=145.79.227.39:4015 , piece_id=AZSOOT5RLGTAHSO47SVO27XDSNFPE65JU2I5UEZDPQ44PKB2YLYA, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.222424843 (414h24m51.050133244s) [noise=true , address=77.90.31.18:4805 , piece_id=YWS2P6F2X4PRKE7FVEA722G4FGDV2D4BXJILLHGQE4Q2HXRU2SAA, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.222721983 (414h24m51.050430374s) [noise=true , address=145.79.227.28:4727 , piece_id=Y5UCAOYDWNY5WPZC7LYUOT5QB325BSVZJUGSRVMR64ML23SEI4BQ, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.222910142 (414h24m51.050618542s) [noise=true , address=145.79.227.37:4566 , piece_id=EKCKTUJE37ZDSKNA6JM2ODSBLYAI2KFCA5RVFRNJPSCQ72ZWKY6A, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.223093651 (414h24m51.050802048s) [noise=true , address=145.79.227.50:4373 , piece_id=HESOK4IM7VRJNRL6NY5U4V52G7PDLSE3L3ZOF7DSTSQGFP6D2ZQA, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.223601176 (414h24m51.051309571s) [noise=true , address=145.79.227.15:4876 , piece_id=X5KN53INJAXSFQUEY2S37RYA3ELNBGN6ZFM6KKJ3A4ROOSUTBTRQ, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.224086995 (414h24m51.051795377s) [noise=true , address=145.79.227.43:4157 , piece_id=EKE6ZQ6YMYBBKZKQ2OIAAF5OI3OZRPJF4XDYCCU7QSKS33UXPDZQ, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.224569909 (414h24m51.052278304s) [noise=true , address=77.90.31.11:4553 , piece_id=MJFMBIGW3L6E75AAMZRCDONHZXPFLQSCK3VINMI4I2ZJC5HVHREA, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.225250077 (414h24m51.052958472s) [noise=true , address=145.79.227.48:4301 , piece_id=OCJPU5E7QLJWVY5K444J3J3ISMFY5E6MVXRMXBHIQRHXMIMRYC2A, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.226504914 (414h24m51.054213295s) [noise=true , address=145.79.227.47:4266 , piece_id=EJYUC5ODHWWQEA7QK5JVT3T3HQ76G5WC5OHPNGUSYMP4AW5BWLPA, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.227351850 (414h24m51.055060238s) [noise=true , address=145.79.227.38:4978 , piece_id=S26D5M7SYZEXTTNQEELKEFFQTOMNA5SG7RL5THCON6KCFY267IDQ, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.228140124 (414h24m51.055848511s) [noise=true , address=145.79.227.32:4834 , piece_id=KLWKSAGBRLFWDKNCHW6UWKCADPZ6D6UURDMNMD5UCZUUWAPDOKVQ, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.321476218 (414h24m51.149184612s) [upload=431251]: Next returned nil segment Dec 22 21:52:50.321492753 (414h24m51.149201151s) [upload=431251]: Waiting for error group managing segments... Dec 22 21:52:50.328186730 (414h24m51.155895125s) [noise=true , address=77.90.31.11:4553 , piece_id=MJFMBIGW3L6E75AAMZRCDONHZXPFLQSCK3VINMI4I2ZJC5HVHREA, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.328400062 (414h24m51.156108454s) [noise=true , address=145.79.227.47:4266 , piece_id=EJYUC5ODHWWQEA7QK5JVT3T3HQ76G5WC5OHPNGUSYMP4AW5BWLPA, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.328834800 (414h24m51.156543191s) [noise=true , address=145.79.227.37:4566 , piece_id=EKCKTUJE37ZDSKNA6JM2ODSBLYAI2KFCA5RVFRNJPSCQ72ZWKY6A, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.330369052 (414h24m51.158077447s) [noise=true , address=145.79.227.51:4411 , piece_id=O76ELDU3V3XZ2ZYC7VCLZC2CU5VUMRTFWJEV2ITS5GFAGRSXTOSA, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.330425878 (414h24m51.158134271s) [noise=true , address=145.79.227.28:4727 , piece_id=Y5UCAOYDWNY5WPZC7LYUOT5QB325BSVZJUGSRVMR64ML23SEI4BQ, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.331015687 (414h24m51.158724085s) [noise=true , address=145.79.227.45:4229 , piece_id=QYCGIYFFNWLGOBOKEZYY4EBLHKC3SFYWOBQJ34DETCLF6BTS54VQ, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.331368793 (414h24m51.159077188s) [noise=true , address=145.79.227.48:4301 , piece_id=OCJPU5E7QLJWVY5K444J3J3ISMFY5E6MVXRMXBHIQRHXMIMRYC2A, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.332233159 (414h24m51.159941554s) [noise=true , address=145.79.227.50:4373 , piece_id=HESOK4IM7VRJNRL6NY5U4V52G7PDLSE3L3ZOF7DSTSQGFP6D2ZQA, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.333040553 (414h24m51.160748944s) [noise=true , address=145.79.227.35:4483 , piece_id=EJIXZOBJWPIVL37GIYYPHQRZDMEMS3YSRPVJH4GLMB4A2CSSRCTQ, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.333195450 (414h24m51.160903848s) [noise=true , address=145.79.227.43:4157 , piece_id=EKE6ZQ6YMYBBKZKQ2OIAAF5OI3OZRPJF4XDYCCU7QSKS33UXPDZQ, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.334537739 (414h24m51.162246133s) [noise=true , address=145.79.227.39:4015 , piece_id=AZSOOT5RLGTAHSO47SVO27XDSNFPE65JU2I5UEZDPQ44PKB2YLYA, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.413477504 (414h24m51.2411859s) [noise=true , address=77.90.31.10:4517 , piece_id=UAIKGPQF75AM6EKYBV3XK45CNYS2N2PNHKBAKRKO6DDILYAUFBLA, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:53:20.161293124 (414h25m20.989001514s) [noise=true , address=77.90.31.18:4805 , piece_id=YWS2P6F2X4PRKE7FVEA722G4FGDV2D4BXJILLHGQE4Q2HXRU2SAA, seg_pos={0 0}, upload=431251]: Done uploading piece. err: ecclient: upload failed (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, address:77.90.31.18:4805): protocol: expected piece hash; context deadline exceeded; EOF Dec 22 21:53:20.161472578 (414h25m20.989180974s) [noise=true , address=145.79.227.38:4978 , piece_id=S26D5M7SYZEXTTNQEELKEFFQTOMNA5SG7RL5THCON6KCFY267IDQ, seg_pos={0 0}, upload=431251]: Done uploading piece. err: ecclient: upload failed (node:12PcGHVSm5UAXPh5vCfyRHCd7Bdb67oXF51AZcnRFUjXqVYzXLR, address:145.79.227.38:4978): protocol: expected piece hash; context deadline exceeded; EOF Dec 22 21:53:20.161591128 (414h25m20.989299529s) [noise=true , address=145.79.227.32:4834 , piece_id=KLWKSAGBRLFWDKNCHW6UWKCADPZ6D6UURDMNMD5UCZUUWAPDOKVQ, seg_pos={0 0}, upload=431251]: Done uploading piece. err: ecclient: upload failed (node:1gRnamgAa2ccZS6F44apRmzSkXC6jGvNreJ5J8Y9nciWPZCSsk, address:145.79.227.32:4834): protocol: expected piece hash; context deadline exceeded; EOF Dec 22 21:53:20.161797225 (414h25m20.989505624s) [noise=true , address=145.79.227.15:4876 , piece_id=X5KN53INJAXSFQUEY2S37RYA3ELNBGN6ZFM6KKJ3A4ROOSUTBTRQ, seg_pos={0 0}, upload=431251]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:145.79.227.15:4876): protocol: expected piece hash; context deadline exceeded; EOF Dec 22 21:53:20.161824434 (414h25m20.989532813s) [seg_pos={0 0}, upload=431251]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 22 21:53:20.183925866 (414h25m21.011634264s) [upload=431251]: Done uploading segments... Dec 22 21:53:20.184019830 (414h25m21.011728235s) [upload=431251]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 451.909794ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............X............................................] duration: 479.194519ms trace: https://jaeger.dev.storj.io/trace/17ef15eb0c5881c1 FAIL (2025-12-22 22:26:53 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 22 21:52:50.192390642 (414h24m51.020099033s) [upload=431251]: Starting upload Dec 22 21:52:50.192488257 (414h24m51.020196648s) [upload=431251]: Done starting upload Dec 22 21:52:50.192531865 (414h24m51.020240259s) [upload=431251]: Uploading segments... Dec 22 21:52:50.192783054 (414h24m51.020491446s) [upload=431251]: Got next segment. Inline: false Dec 22 21:52:50.192836048 (414h24m51.020544439s) [upload=431251]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 22 21:52:50.192841417 (414h24m51.020549808s) [upload=431251]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 22 21:52:50.220288008 (414h24m51.047996404s) [seg_pos={0 0}, upload=431251]: Begin upload segment... Dec 22 21:52:50.220342100 (414h24m51.048050497s) [seg_pos={0 0}, upload=431251]: Done begin upload segment. Dec 22 21:52:50.220384745 (414h24m51.048093145s) [noise=true , address=77.90.31.10:4517 , piece_id=UAIKGPQF75AM6EKYBV3XK45CNYS2N2PNHKBAKRKO6DDILYAUFBLA, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.220421097 (414h24m51.048129475s) [noise=true , address=145.79.227.45:4229 , piece_id=QYCGIYFFNWLGOBOKEZYY4EBLHKC3SFYWOBQJ34DETCLF6BTS54VQ, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.221068129 (414h24m51.048776527s) [noise=true , address=145.79.227.35:4483 , piece_id=EJIXZOBJWPIVL37GIYYPHQRZDMEMS3YSRPVJH4GLMB4A2CSSRCTQ, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.221771875 (414h24m51.049480268s) [noise=true , address=145.79.227.51:4411 , piece_id=O76ELDU3V3XZ2ZYC7VCLZC2CU5VUMRTFWJEV2ITS5GFAGRSXTOSA, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.222308205 (414h24m51.050016602s) [noise=true , address=145.79.227.39:4015 , piece_id=AZSOOT5RLGTAHSO47SVO27XDSNFPE65JU2I5UEZDPQ44PKB2YLYA, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.222424843 (414h24m51.050133244s) [noise=true , address=77.90.31.18:4805 , piece_id=YWS2P6F2X4PRKE7FVEA722G4FGDV2D4BXJILLHGQE4Q2HXRU2SAA, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.222721983 (414h24m51.050430374s) [noise=true , address=145.79.227.28:4727 , piece_id=Y5UCAOYDWNY5WPZC7LYUOT5QB325BSVZJUGSRVMR64ML23SEI4BQ, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.222910142 (414h24m51.050618542s) [noise=true , address=145.79.227.37:4566 , piece_id=EKCKTUJE37ZDSKNA6JM2ODSBLYAI2KFCA5RVFRNJPSCQ72ZWKY6A, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.223093651 (414h24m51.050802048s) [noise=true , address=145.79.227.50:4373 , piece_id=HESOK4IM7VRJNRL6NY5U4V52G7PDLSE3L3ZOF7DSTSQGFP6D2ZQA, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.223601176 (414h24m51.051309571s) [noise=true , address=145.79.227.15:4876 , piece_id=X5KN53INJAXSFQUEY2S37RYA3ELNBGN6ZFM6KKJ3A4ROOSUTBTRQ, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.224086995 (414h24m51.051795377s) [noise=true , address=145.79.227.43:4157 , piece_id=EKE6ZQ6YMYBBKZKQ2OIAAF5OI3OZRPJF4XDYCCU7QSKS33UXPDZQ, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.224569909 (414h24m51.052278304s) [noise=true , address=77.90.31.11:4553 , piece_id=MJFMBIGW3L6E75AAMZRCDONHZXPFLQSCK3VINMI4I2ZJC5HVHREA, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.225250077 (414h24m51.052958472s) [noise=true , address=145.79.227.48:4301 , piece_id=OCJPU5E7QLJWVY5K444J3J3ISMFY5E6MVXRMXBHIQRHXMIMRYC2A, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.226504914 (414h24m51.054213295s) [noise=true , address=145.79.227.47:4266 , piece_id=EJYUC5ODHWWQEA7QK5JVT3T3HQ76G5WC5OHPNGUSYMP4AW5BWLPA, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.227351850 (414h24m51.055060238s) [noise=true , address=145.79.227.38:4978 , piece_id=S26D5M7SYZEXTTNQEELKEFFQTOMNA5SG7RL5THCON6KCFY267IDQ, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.228140124 (414h24m51.055848511s) [noise=true , address=145.79.227.32:4834 , piece_id=KLWKSAGBRLFWDKNCHW6UWKCADPZ6D6UURDMNMD5UCZUUWAPDOKVQ, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.321476218 (414h24m51.149184612s) [upload=431251]: Next returned nil segment Dec 22 21:52:50.321492753 (414h24m51.149201151s) [upload=431251]: Waiting for error group managing segments... Dec 22 21:52:50.328186730 (414h24m51.155895125s) [noise=true , address=77.90.31.11:4553 , piece_id=MJFMBIGW3L6E75AAMZRCDONHZXPFLQSCK3VINMI4I2ZJC5HVHREA, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.328400062 (414h24m51.156108454s) [noise=true , address=145.79.227.47:4266 , piece_id=EJYUC5ODHWWQEA7QK5JVT3T3HQ76G5WC5OHPNGUSYMP4AW5BWLPA, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.328834800 (414h24m51.156543191s) [noise=true , address=145.79.227.37:4566 , piece_id=EKCKTUJE37ZDSKNA6JM2ODSBLYAI2KFCA5RVFRNJPSCQ72ZWKY6A, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.330369052 (414h24m51.158077447s) [noise=true , address=145.79.227.51:4411 , piece_id=O76ELDU3V3XZ2ZYC7VCLZC2CU5VUMRTFWJEV2ITS5GFAGRSXTOSA, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.330425878 (414h24m51.158134271s) [noise=true , address=145.79.227.28:4727 , piece_id=Y5UCAOYDWNY5WPZC7LYUOT5QB325BSVZJUGSRVMR64ML23SEI4BQ, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.331015687 (414h24m51.158724085s) [noise=true , address=145.79.227.45:4229 , piece_id=QYCGIYFFNWLGOBOKEZYY4EBLHKC3SFYWOBQJ34DETCLF6BTS54VQ, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.331368793 (414h24m51.159077188s) [noise=true , address=145.79.227.48:4301 , piece_id=OCJPU5E7QLJWVY5K444J3J3ISMFY5E6MVXRMXBHIQRHXMIMRYC2A, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.332233159 (414h24m51.159941554s) [noise=true , address=145.79.227.50:4373 , piece_id=HESOK4IM7VRJNRL6NY5U4V52G7PDLSE3L3ZOF7DSTSQGFP6D2ZQA, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.333040553 (414h24m51.160748944s) [noise=true , address=145.79.227.35:4483 , piece_id=EJIXZOBJWPIVL37GIYYPHQRZDMEMS3YSRPVJH4GLMB4A2CSSRCTQ, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.333195450 (414h24m51.160903848s) [noise=true , address=145.79.227.43:4157 , piece_id=EKE6ZQ6YMYBBKZKQ2OIAAF5OI3OZRPJF4XDYCCU7QSKS33UXPDZQ, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.334537739 (414h24m51.162246133s) [noise=true , address=145.79.227.39:4015 , piece_id=AZSOOT5RLGTAHSO47SVO27XDSNFPE65JU2I5UEZDPQ44PKB2YLYA, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.413477504 (414h24m51.2411859s) [noise=true , address=77.90.31.10:4517 , piece_id=UAIKGPQF75AM6EKYBV3XK45CNYS2N2PNHKBAKRKO6DDILYAUFBLA, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:53:20.161293124 (414h25m20.989001514s) [noise=true , address=77.90.31.18:4805 , piece_id=YWS2P6F2X4PRKE7FVEA722G4FGDV2D4BXJILLHGQE4Q2HXRU2SAA, seg_pos={0 0}, upload=431251]: Done uploading piece. err: ecclient: upload failed (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, address:77.90.31.18:4805): protocol: expected piece hash; context deadline exceeded; EOF Dec 22 21:53:20.161472578 (414h25m20.989180974s) [noise=true , address=145.79.227.38:4978 , piece_id=S26D5M7SYZEXTTNQEELKEFFQTOMNA5SG7RL5THCON6KCFY267IDQ, seg_pos={0 0}, upload=431251]: Done uploading piece. err: ecclient: upload failed (node:12PcGHVSm5UAXPh5vCfyRHCd7Bdb67oXF51AZcnRFUjXqVYzXLR, address:145.79.227.38:4978): protocol: expected piece hash; context deadline exceeded; EOF Dec 22 21:53:20.161591128 (414h25m20.989299529s) [noise=true , address=145.79.227.32:4834 , piece_id=KLWKSAGBRLFWDKNCHW6UWKCADPZ6D6UURDMNMD5UCZUUWAPDOKVQ, seg_pos={0 0}, upload=431251]: Done uploading piece. err: ecclient: upload failed (node:1gRnamgAa2ccZS6F44apRmzSkXC6jGvNreJ5J8Y9nciWPZCSsk, address:145.79.227.32:4834): protocol: expected piece hash; context deadline exceeded; EOF Dec 22 21:53:20.161797225 (414h25m20.989505624s) [noise=true , address=145.79.227.15:4876 , piece_id=X5KN53INJAXSFQUEY2S37RYA3ELNBGN6ZFM6KKJ3A4ROOSUTBTRQ, seg_pos={0 0}, upload=431251]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:145.79.227.15:4876): protocol: expected piece hash; context deadline exceeded; EOF Dec 22 21:53:20.161824434 (414h25m20.989532813s) [seg_pos={0 0}, upload=431251]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 22 21:53:20.183925866 (414h25m21.011634264s) [upload=431251]: Done uploading segments... Dec 22 21:53:20.184019830 (414h25m21.011728235s) [upload=431251]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 479.224276ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................X...........................................] duration: 450.48154ms trace: https://jaeger.dev.storj.io/trace/17ef15eb0c5881c1 FAIL (2025-12-22 22:25:38 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 22 21:52:50.192390642 (414h24m51.020099033s) [upload=431251]: Starting upload Dec 22 21:52:50.192488257 (414h24m51.020196648s) [upload=431251]: Done starting upload Dec 22 21:52:50.192531865 (414h24m51.020240259s) [upload=431251]: Uploading segments... Dec 22 21:52:50.192783054 (414h24m51.020491446s) [upload=431251]: Got next segment. Inline: false Dec 22 21:52:50.192836048 (414h24m51.020544439s) [upload=431251]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 22 21:52:50.192841417 (414h24m51.020549808s) [upload=431251]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 22 21:52:50.220288008 (414h24m51.047996404s) [seg_pos={0 0}, upload=431251]: Begin upload segment... Dec 22 21:52:50.220342100 (414h24m51.048050497s) [seg_pos={0 0}, upload=431251]: Done begin upload segment. Dec 22 21:52:50.220384745 (414h24m51.048093145s) [noise=true , address=77.90.31.10:4517 , piece_id=UAIKGPQF75AM6EKYBV3XK45CNYS2N2PNHKBAKRKO6DDILYAUFBLA, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.220421097 (414h24m51.048129475s) [noise=true , address=145.79.227.45:4229 , piece_id=QYCGIYFFNWLGOBOKEZYY4EBLHKC3SFYWOBQJ34DETCLF6BTS54VQ, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.221068129 (414h24m51.048776527s) [noise=true , address=145.79.227.35:4483 , piece_id=EJIXZOBJWPIVL37GIYYPHQRZDMEMS3YSRPVJH4GLMB4A2CSSRCTQ, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.221771875 (414h24m51.049480268s) [noise=true , address=145.79.227.51:4411 , piece_id=O76ELDU3V3XZ2ZYC7VCLZC2CU5VUMRTFWJEV2ITS5GFAGRSXTOSA, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.222308205 (414h24m51.050016602s) [noise=true , address=145.79.227.39:4015 , piece_id=AZSOOT5RLGTAHSO47SVO27XDSNFPE65JU2I5UEZDPQ44PKB2YLYA, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.222424843 (414h24m51.050133244s) [noise=true , address=77.90.31.18:4805 , piece_id=YWS2P6F2X4PRKE7FVEA722G4FGDV2D4BXJILLHGQE4Q2HXRU2SAA, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.222721983 (414h24m51.050430374s) [noise=true , address=145.79.227.28:4727 , piece_id=Y5UCAOYDWNY5WPZC7LYUOT5QB325BSVZJUGSRVMR64ML23SEI4BQ, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.222910142 (414h24m51.050618542s) [noise=true , address=145.79.227.37:4566 , piece_id=EKCKTUJE37ZDSKNA6JM2ODSBLYAI2KFCA5RVFRNJPSCQ72ZWKY6A, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.223093651 (414h24m51.050802048s) [noise=true , address=145.79.227.50:4373 , piece_id=HESOK4IM7VRJNRL6NY5U4V52G7PDLSE3L3ZOF7DSTSQGFP6D2ZQA, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.223601176 (414h24m51.051309571s) [noise=true , address=145.79.227.15:4876 , piece_id=X5KN53INJAXSFQUEY2S37RYA3ELNBGN6ZFM6KKJ3A4ROOSUTBTRQ, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.224086995 (414h24m51.051795377s) [noise=true , address=145.79.227.43:4157 , piece_id=EKE6ZQ6YMYBBKZKQ2OIAAF5OI3OZRPJF4XDYCCU7QSKS33UXPDZQ, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.224569909 (414h24m51.052278304s) [noise=true , address=77.90.31.11:4553 , piece_id=MJFMBIGW3L6E75AAMZRCDONHZXPFLQSCK3VINMI4I2ZJC5HVHREA, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.225250077 (414h24m51.052958472s) [noise=true , address=145.79.227.48:4301 , piece_id=OCJPU5E7QLJWVY5K444J3J3ISMFY5E6MVXRMXBHIQRHXMIMRYC2A, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.226504914 (414h24m51.054213295s) [noise=true , address=145.79.227.47:4266 , piece_id=EJYUC5ODHWWQEA7QK5JVT3T3HQ76G5WC5OHPNGUSYMP4AW5BWLPA, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.227351850 (414h24m51.055060238s) [noise=true , address=145.79.227.38:4978 , piece_id=S26D5M7SYZEXTTNQEELKEFFQTOMNA5SG7RL5THCON6KCFY267IDQ, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.228140124 (414h24m51.055848511s) [noise=true , address=145.79.227.32:4834 , piece_id=KLWKSAGBRLFWDKNCHW6UWKCADPZ6D6UURDMNMD5UCZUUWAPDOKVQ, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.321476218 (414h24m51.149184612s) [upload=431251]: Next returned nil segment Dec 22 21:52:50.321492753 (414h24m51.149201151s) [upload=431251]: Waiting for error group managing segments... Dec 22 21:52:50.328186730 (414h24m51.155895125s) [noise=true , address=77.90.31.11:4553 , piece_id=MJFMBIGW3L6E75AAMZRCDONHZXPFLQSCK3VINMI4I2ZJC5HVHREA, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.328400062 (414h24m51.156108454s) [noise=true , address=145.79.227.47:4266 , piece_id=EJYUC5ODHWWQEA7QK5JVT3T3HQ76G5WC5OHPNGUSYMP4AW5BWLPA, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.328834800 (414h24m51.156543191s) [noise=true , address=145.79.227.37:4566 , piece_id=EKCKTUJE37ZDSKNA6JM2ODSBLYAI2KFCA5RVFRNJPSCQ72ZWKY6A, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.330369052 (414h24m51.158077447s) [noise=true , address=145.79.227.51:4411 , piece_id=O76ELDU3V3XZ2ZYC7VCLZC2CU5VUMRTFWJEV2ITS5GFAGRSXTOSA, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.330425878 (414h24m51.158134271s) [noise=true , address=145.79.227.28:4727 , piece_id=Y5UCAOYDWNY5WPZC7LYUOT5QB325BSVZJUGSRVMR64ML23SEI4BQ, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.331015687 (414h24m51.158724085s) [noise=true , address=145.79.227.45:4229 , piece_id=QYCGIYFFNWLGOBOKEZYY4EBLHKC3SFYWOBQJ34DETCLF6BTS54VQ, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.331368793 (414h24m51.159077188s) [noise=true , address=145.79.227.48:4301 , piece_id=OCJPU5E7QLJWVY5K444J3J3ISMFY5E6MVXRMXBHIQRHXMIMRYC2A, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.332233159 (414h24m51.159941554s) [noise=true , address=145.79.227.50:4373 , piece_id=HESOK4IM7VRJNRL6NY5U4V52G7PDLSE3L3ZOF7DSTSQGFP6D2ZQA, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.333040553 (414h24m51.160748944s) [noise=true , address=145.79.227.35:4483 , piece_id=EJIXZOBJWPIVL37GIYYPHQRZDMEMS3YSRPVJH4GLMB4A2CSSRCTQ, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.333195450 (414h24m51.160903848s) [noise=true , address=145.79.227.43:4157 , piece_id=EKE6ZQ6YMYBBKZKQ2OIAAF5OI3OZRPJF4XDYCCU7QSKS33UXPDZQ, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.334537739 (414h24m51.162246133s) [noise=true , address=145.79.227.39:4015 , piece_id=AZSOOT5RLGTAHSO47SVO27XDSNFPE65JU2I5UEZDPQ44PKB2YLYA, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.413477504 (414h24m51.2411859s) [noise=true , address=77.90.31.10:4517 , piece_id=UAIKGPQF75AM6EKYBV3XK45CNYS2N2PNHKBAKRKO6DDILYAUFBLA, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:53:20.161293124 (414h25m20.989001514s) [noise=true , address=77.90.31.18:4805 , piece_id=YWS2P6F2X4PRKE7FVEA722G4FGDV2D4BXJILLHGQE4Q2HXRU2SAA, seg_pos={0 0}, upload=431251]: Done uploading piece. err: ecclient: upload failed (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, address:77.90.31.18:4805): protocol: expected piece hash; context deadline exceeded; EOF Dec 22 21:53:20.161472578 (414h25m20.989180974s) [noise=true , address=145.79.227.38:4978 , piece_id=S26D5M7SYZEXTTNQEELKEFFQTOMNA5SG7RL5THCON6KCFY267IDQ, seg_pos={0 0}, upload=431251]: Done uploading piece. err: ecclient: upload failed (node:12PcGHVSm5UAXPh5vCfyRHCd7Bdb67oXF51AZcnRFUjXqVYzXLR, address:145.79.227.38:4978): protocol: expected piece hash; context deadline exceeded; EOF Dec 22 21:53:20.161591128 (414h25m20.989299529s) [noise=true , address=145.79.227.32:4834 , piece_id=KLWKSAGBRLFWDKNCHW6UWKCADPZ6D6UURDMNMD5UCZUUWAPDOKVQ, seg_pos={0 0}, upload=431251]: Done uploading piece. err: ecclient: upload failed (node:1gRnamgAa2ccZS6F44apRmzSkXC6jGvNreJ5J8Y9nciWPZCSsk, address:145.79.227.32:4834): protocol: expected piece hash; context deadline exceeded; EOF Dec 22 21:53:20.161797225 (414h25m20.989505624s) [noise=true , address=145.79.227.15:4876 , piece_id=X5KN53INJAXSFQUEY2S37RYA3ELNBGN6ZFM6KKJ3A4ROOSUTBTRQ, seg_pos={0 0}, upload=431251]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:145.79.227.15:4876): protocol: expected piece hash; context deadline exceeded; EOF Dec 22 21:53:20.161824434 (414h25m20.989532813s) [seg_pos={0 0}, upload=431251]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 22 21:53:20.183925866 (414h25m21.011634264s) [upload=431251]: Done uploading segments... Dec 22 21:53:20.184019830 (414h25m21.011728235s) [upload=431251]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 450.508713ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................X..........................................] duration: 475.066799ms trace: https://jaeger.dev.storj.io/trace/17ef15eb0c5881c1 FAIL (2025-12-22 22:25:17 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 22 21:52:50.192390642 (414h24m51.020099033s) [upload=431251]: Starting upload Dec 22 21:52:50.192488257 (414h24m51.020196648s) [upload=431251]: Done starting upload Dec 22 21:52:50.192531865 (414h24m51.020240259s) [upload=431251]: Uploading segments... Dec 22 21:52:50.192783054 (414h24m51.020491446s) [upload=431251]: Got next segment. Inline: false Dec 22 21:52:50.192836048 (414h24m51.020544439s) [upload=431251]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 22 21:52:50.192841417 (414h24m51.020549808s) [upload=431251]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 22 21:52:50.220288008 (414h24m51.047996404s) [seg_pos={0 0}, upload=431251]: Begin upload segment... Dec 22 21:52:50.220342100 (414h24m51.048050497s) [seg_pos={0 0}, upload=431251]: Done begin upload segment. Dec 22 21:52:50.220384745 (414h24m51.048093145s) [noise=true , address=77.90.31.10:4517 , piece_id=UAIKGPQF75AM6EKYBV3XK45CNYS2N2PNHKBAKRKO6DDILYAUFBLA, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.220421097 (414h24m51.048129475s) [noise=true , address=145.79.227.45:4229 , piece_id=QYCGIYFFNWLGOBOKEZYY4EBLHKC3SFYWOBQJ34DETCLF6BTS54VQ, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.221068129 (414h24m51.048776527s) [noise=true , address=145.79.227.35:4483 , piece_id=EJIXZOBJWPIVL37GIYYPHQRZDMEMS3YSRPVJH4GLMB4A2CSSRCTQ, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.221771875 (414h24m51.049480268s) [noise=true , address=145.79.227.51:4411 , piece_id=O76ELDU3V3XZ2ZYC7VCLZC2CU5VUMRTFWJEV2ITS5GFAGRSXTOSA, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.222308205 (414h24m51.050016602s) [noise=true , address=145.79.227.39:4015 , piece_id=AZSOOT5RLGTAHSO47SVO27XDSNFPE65JU2I5UEZDPQ44PKB2YLYA, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.222424843 (414h24m51.050133244s) [noise=true , address=77.90.31.18:4805 , piece_id=YWS2P6F2X4PRKE7FVEA722G4FGDV2D4BXJILLHGQE4Q2HXRU2SAA, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.222721983 (414h24m51.050430374s) [noise=true , address=145.79.227.28:4727 , piece_id=Y5UCAOYDWNY5WPZC7LYUOT5QB325BSVZJUGSRVMR64ML23SEI4BQ, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.222910142 (414h24m51.050618542s) [noise=true , address=145.79.227.37:4566 , piece_id=EKCKTUJE37ZDSKNA6JM2ODSBLYAI2KFCA5RVFRNJPSCQ72ZWKY6A, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.223093651 (414h24m51.050802048s) [noise=true , address=145.79.227.50:4373 , piece_id=HESOK4IM7VRJNRL6NY5U4V52G7PDLSE3L3ZOF7DSTSQGFP6D2ZQA, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.223601176 (414h24m51.051309571s) [noise=true , address=145.79.227.15:4876 , piece_id=X5KN53INJAXSFQUEY2S37RYA3ELNBGN6ZFM6KKJ3A4ROOSUTBTRQ, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.224086995 (414h24m51.051795377s) [noise=true , address=145.79.227.43:4157 , piece_id=EKE6ZQ6YMYBBKZKQ2OIAAF5OI3OZRPJF4XDYCCU7QSKS33UXPDZQ, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.224569909 (414h24m51.052278304s) [noise=true , address=77.90.31.11:4553 , piece_id=MJFMBIGW3L6E75AAMZRCDONHZXPFLQSCK3VINMI4I2ZJC5HVHREA, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.225250077 (414h24m51.052958472s) [noise=true , address=145.79.227.48:4301 , piece_id=OCJPU5E7QLJWVY5K444J3J3ISMFY5E6MVXRMXBHIQRHXMIMRYC2A, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.226504914 (414h24m51.054213295s) [noise=true , address=145.79.227.47:4266 , piece_id=EJYUC5ODHWWQEA7QK5JVT3T3HQ76G5WC5OHPNGUSYMP4AW5BWLPA, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.227351850 (414h24m51.055060238s) [noise=true , address=145.79.227.38:4978 , piece_id=S26D5M7SYZEXTTNQEELKEFFQTOMNA5SG7RL5THCON6KCFY267IDQ, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.228140124 (414h24m51.055848511s) [noise=true , address=145.79.227.32:4834 , piece_id=KLWKSAGBRLFWDKNCHW6UWKCADPZ6D6UURDMNMD5UCZUUWAPDOKVQ, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.321476218 (414h24m51.149184612s) [upload=431251]: Next returned nil segment Dec 22 21:52:50.321492753 (414h24m51.149201151s) [upload=431251]: Waiting for error group managing segments... Dec 22 21:52:50.328186730 (414h24m51.155895125s) [noise=true , address=77.90.31.11:4553 , piece_id=MJFMBIGW3L6E75AAMZRCDONHZXPFLQSCK3VINMI4I2ZJC5HVHREA, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.328400062 (414h24m51.156108454s) [noise=true , address=145.79.227.47:4266 , piece_id=EJYUC5ODHWWQEA7QK5JVT3T3HQ76G5WC5OHPNGUSYMP4AW5BWLPA, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.328834800 (414h24m51.156543191s) [noise=true , address=145.79.227.37:4566 , piece_id=EKCKTUJE37ZDSKNA6JM2ODSBLYAI2KFCA5RVFRNJPSCQ72ZWKY6A, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.330369052 (414h24m51.158077447s) [noise=true , address=145.79.227.51:4411 , piece_id=O76ELDU3V3XZ2ZYC7VCLZC2CU5VUMRTFWJEV2ITS5GFAGRSXTOSA, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.330425878 (414h24m51.158134271s) [noise=true , address=145.79.227.28:4727 , piece_id=Y5UCAOYDWNY5WPZC7LYUOT5QB325BSVZJUGSRVMR64ML23SEI4BQ, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.331015687 (414h24m51.158724085s) [noise=true , address=145.79.227.45:4229 , piece_id=QYCGIYFFNWLGOBOKEZYY4EBLHKC3SFYWOBQJ34DETCLF6BTS54VQ, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.331368793 (414h24m51.159077188s) [noise=true , address=145.79.227.48:4301 , piece_id=OCJPU5E7QLJWVY5K444J3J3ISMFY5E6MVXRMXBHIQRHXMIMRYC2A, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.332233159 (414h24m51.159941554s) [noise=true , address=145.79.227.50:4373 , piece_id=HESOK4IM7VRJNRL6NY5U4V52G7PDLSE3L3ZOF7DSTSQGFP6D2ZQA, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.333040553 (414h24m51.160748944s) [noise=true , address=145.79.227.35:4483 , piece_id=EJIXZOBJWPIVL37GIYYPHQRZDMEMS3YSRPVJH4GLMB4A2CSSRCTQ, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.333195450 (414h24m51.160903848s) [noise=true , address=145.79.227.43:4157 , piece_id=EKE6ZQ6YMYBBKZKQ2OIAAF5OI3OZRPJF4XDYCCU7QSKS33UXPDZQ, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.334537739 (414h24m51.162246133s) [noise=true , address=145.79.227.39:4015 , piece_id=AZSOOT5RLGTAHSO47SVO27XDSNFPE65JU2I5UEZDPQ44PKB2YLYA, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.413477504 (414h24m51.2411859s) [noise=true , address=77.90.31.10:4517 , piece_id=UAIKGPQF75AM6EKYBV3XK45CNYS2N2PNHKBAKRKO6DDILYAUFBLA, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:53:20.161293124 (414h25m20.989001514s) [noise=true , address=77.90.31.18:4805 , piece_id=YWS2P6F2X4PRKE7FVEA722G4FGDV2D4BXJILLHGQE4Q2HXRU2SAA, seg_pos={0 0}, upload=431251]: Done uploading piece. err: ecclient: upload failed (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, address:77.90.31.18:4805): protocol: expected piece hash; context deadline exceeded; EOF Dec 22 21:53:20.161472578 (414h25m20.989180974s) [noise=true , address=145.79.227.38:4978 , piece_id=S26D5M7SYZEXTTNQEELKEFFQTOMNA5SG7RL5THCON6KCFY267IDQ, seg_pos={0 0}, upload=431251]: Done uploading piece. err: ecclient: upload failed (node:12PcGHVSm5UAXPh5vCfyRHCd7Bdb67oXF51AZcnRFUjXqVYzXLR, address:145.79.227.38:4978): protocol: expected piece hash; context deadline exceeded; EOF Dec 22 21:53:20.161591128 (414h25m20.989299529s) [noise=true , address=145.79.227.32:4834 , piece_id=KLWKSAGBRLFWDKNCHW6UWKCADPZ6D6UURDMNMD5UCZUUWAPDOKVQ, seg_pos={0 0}, upload=431251]: Done uploading piece. err: ecclient: upload failed (node:1gRnamgAa2ccZS6F44apRmzSkXC6jGvNreJ5J8Y9nciWPZCSsk, address:145.79.227.32:4834): protocol: expected piece hash; context deadline exceeded; EOF Dec 22 21:53:20.161797225 (414h25m20.989505624s) [noise=true , address=145.79.227.15:4876 , piece_id=X5KN53INJAXSFQUEY2S37RYA3ELNBGN6ZFM6KKJ3A4ROOSUTBTRQ, seg_pos={0 0}, upload=431251]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:145.79.227.15:4876): protocol: expected piece hash; context deadline exceeded; EOF Dec 22 21:53:20.161824434 (414h25m20.989532813s) [seg_pos={0 0}, upload=431251]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 22 21:53:20.183925866 (414h25m21.011634264s) [upload=431251]: Done uploading segments... Dec 22 21:53:20.184019830 (414h25m21.011728235s) [upload=431251]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 475.093187ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................X.........................................] duration: 445.497447ms trace: https://jaeger.dev.storj.io/trace/17ef15eb0c5881c1 FAIL (2025-12-22 22:24:32 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 22 21:52:50.192390642 (414h24m51.020099033s) [upload=431251]: Starting upload Dec 22 21:52:50.192488257 (414h24m51.020196648s) [upload=431251]: Done starting upload Dec 22 21:52:50.192531865 (414h24m51.020240259s) [upload=431251]: Uploading segments... Dec 22 21:52:50.192783054 (414h24m51.020491446s) [upload=431251]: Got next segment. Inline: false Dec 22 21:52:50.192836048 (414h24m51.020544439s) [upload=431251]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 22 21:52:50.192841417 (414h24m51.020549808s) [upload=431251]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 22 21:52:50.220288008 (414h24m51.047996404s) [seg_pos={0 0}, upload=431251]: Begin upload segment... Dec 22 21:52:50.220342100 (414h24m51.048050497s) [seg_pos={0 0}, upload=431251]: Done begin upload segment. Dec 22 21:52:50.220384745 (414h24m51.048093145s) [noise=true , address=77.90.31.10:4517 , piece_id=UAIKGPQF75AM6EKYBV3XK45CNYS2N2PNHKBAKRKO6DDILYAUFBLA, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.220421097 (414h24m51.048129475s) [noise=true , address=145.79.227.45:4229 , piece_id=QYCGIYFFNWLGOBOKEZYY4EBLHKC3SFYWOBQJ34DETCLF6BTS54VQ, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.221068129 (414h24m51.048776527s) [noise=true , address=145.79.227.35:4483 , piece_id=EJIXZOBJWPIVL37GIYYPHQRZDMEMS3YSRPVJH4GLMB4A2CSSRCTQ, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.221771875 (414h24m51.049480268s) [noise=true , address=145.79.227.51:4411 , piece_id=O76ELDU3V3XZ2ZYC7VCLZC2CU5VUMRTFWJEV2ITS5GFAGRSXTOSA, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.222308205 (414h24m51.050016602s) [noise=true , address=145.79.227.39:4015 , piece_id=AZSOOT5RLGTAHSO47SVO27XDSNFPE65JU2I5UEZDPQ44PKB2YLYA, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.222424843 (414h24m51.050133244s) [noise=true , address=77.90.31.18:4805 , piece_id=YWS2P6F2X4PRKE7FVEA722G4FGDV2D4BXJILLHGQE4Q2HXRU2SAA, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.222721983 (414h24m51.050430374s) [noise=true , address=145.79.227.28:4727 , piece_id=Y5UCAOYDWNY5WPZC7LYUOT5QB325BSVZJUGSRVMR64ML23SEI4BQ, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.222910142 (414h24m51.050618542s) [noise=true , address=145.79.227.37:4566 , piece_id=EKCKTUJE37ZDSKNA6JM2ODSBLYAI2KFCA5RVFRNJPSCQ72ZWKY6A, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.223093651 (414h24m51.050802048s) [noise=true , address=145.79.227.50:4373 , piece_id=HESOK4IM7VRJNRL6NY5U4V52G7PDLSE3L3ZOF7DSTSQGFP6D2ZQA, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.223601176 (414h24m51.051309571s) [noise=true , address=145.79.227.15:4876 , piece_id=X5KN53INJAXSFQUEY2S37RYA3ELNBGN6ZFM6KKJ3A4ROOSUTBTRQ, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.224086995 (414h24m51.051795377s) [noise=true , address=145.79.227.43:4157 , piece_id=EKE6ZQ6YMYBBKZKQ2OIAAF5OI3OZRPJF4XDYCCU7QSKS33UXPDZQ, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.224569909 (414h24m51.052278304s) [noise=true , address=77.90.31.11:4553 , piece_id=MJFMBIGW3L6E75AAMZRCDONHZXPFLQSCK3VINMI4I2ZJC5HVHREA, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.225250077 (414h24m51.052958472s) [noise=true , address=145.79.227.48:4301 , piece_id=OCJPU5E7QLJWVY5K444J3J3ISMFY5E6MVXRMXBHIQRHXMIMRYC2A, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.226504914 (414h24m51.054213295s) [noise=true , address=145.79.227.47:4266 , piece_id=EJYUC5ODHWWQEA7QK5JVT3T3HQ76G5WC5OHPNGUSYMP4AW5BWLPA, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.227351850 (414h24m51.055060238s) [noise=true , address=145.79.227.38:4978 , piece_id=S26D5M7SYZEXTTNQEELKEFFQTOMNA5SG7RL5THCON6KCFY267IDQ, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.228140124 (414h24m51.055848511s) [noise=true , address=145.79.227.32:4834 , piece_id=KLWKSAGBRLFWDKNCHW6UWKCADPZ6D6UURDMNMD5UCZUUWAPDOKVQ, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.321476218 (414h24m51.149184612s) [upload=431251]: Next returned nil segment Dec 22 21:52:50.321492753 (414h24m51.149201151s) [upload=431251]: Waiting for error group managing segments... Dec 22 21:52:50.328186730 (414h24m51.155895125s) [noise=true , address=77.90.31.11:4553 , piece_id=MJFMBIGW3L6E75AAMZRCDONHZXPFLQSCK3VINMI4I2ZJC5HVHREA, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.328400062 (414h24m51.156108454s) [noise=true , address=145.79.227.47:4266 , piece_id=EJYUC5ODHWWQEA7QK5JVT3T3HQ76G5WC5OHPNGUSYMP4AW5BWLPA, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.328834800 (414h24m51.156543191s) [noise=true , address=145.79.227.37:4566 , piece_id=EKCKTUJE37ZDSKNA6JM2ODSBLYAI2KFCA5RVFRNJPSCQ72ZWKY6A, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.330369052 (414h24m51.158077447s) [noise=true , address=145.79.227.51:4411 , piece_id=O76ELDU3V3XZ2ZYC7VCLZC2CU5VUMRTFWJEV2ITS5GFAGRSXTOSA, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.330425878 (414h24m51.158134271s) [noise=true , address=145.79.227.28:4727 , piece_id=Y5UCAOYDWNY5WPZC7LYUOT5QB325BSVZJUGSRVMR64ML23SEI4BQ, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.331015687 (414h24m51.158724085s) [noise=true , address=145.79.227.45:4229 , piece_id=QYCGIYFFNWLGOBOKEZYY4EBLHKC3SFYWOBQJ34DETCLF6BTS54VQ, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.331368793 (414h24m51.159077188s) [noise=true , address=145.79.227.48:4301 , piece_id=OCJPU5E7QLJWVY5K444J3J3ISMFY5E6MVXRMXBHIQRHXMIMRYC2A, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.332233159 (414h24m51.159941554s) [noise=true , address=145.79.227.50:4373 , piece_id=HESOK4IM7VRJNRL6NY5U4V52G7PDLSE3L3ZOF7DSTSQGFP6D2ZQA, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.333040553 (414h24m51.160748944s) [noise=true , address=145.79.227.35:4483 , piece_id=EJIXZOBJWPIVL37GIYYPHQRZDMEMS3YSRPVJH4GLMB4A2CSSRCTQ, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.333195450 (414h24m51.160903848s) [noise=true , address=145.79.227.43:4157 , piece_id=EKE6ZQ6YMYBBKZKQ2OIAAF5OI3OZRPJF4XDYCCU7QSKS33UXPDZQ, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.334537739 (414h24m51.162246133s) [noise=true , address=145.79.227.39:4015 , piece_id=AZSOOT5RLGTAHSO47SVO27XDSNFPE65JU2I5UEZDPQ44PKB2YLYA, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.413477504 (414h24m51.2411859s) [noise=true , address=77.90.31.10:4517 , piece_id=UAIKGPQF75AM6EKYBV3XK45CNYS2N2PNHKBAKRKO6DDILYAUFBLA, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:53:20.161293124 (414h25m20.989001514s) [noise=true , address=77.90.31.18:4805 , piece_id=YWS2P6F2X4PRKE7FVEA722G4FGDV2D4BXJILLHGQE4Q2HXRU2SAA, seg_pos={0 0}, upload=431251]: Done uploading piece. err: ecclient: upload failed (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, address:77.90.31.18:4805): protocol: expected piece hash; context deadline exceeded; EOF Dec 22 21:53:20.161472578 (414h25m20.989180974s) [noise=true , address=145.79.227.38:4978 , piece_id=S26D5M7SYZEXTTNQEELKEFFQTOMNA5SG7RL5THCON6KCFY267IDQ, seg_pos={0 0}, upload=431251]: Done uploading piece. err: ecclient: upload failed (node:12PcGHVSm5UAXPh5vCfyRHCd7Bdb67oXF51AZcnRFUjXqVYzXLR, address:145.79.227.38:4978): protocol: expected piece hash; context deadline exceeded; EOF Dec 22 21:53:20.161591128 (414h25m20.989299529s) [noise=true , address=145.79.227.32:4834 , piece_id=KLWKSAGBRLFWDKNCHW6UWKCADPZ6D6UURDMNMD5UCZUUWAPDOKVQ, seg_pos={0 0}, upload=431251]: Done uploading piece. err: ecclient: upload failed (node:1gRnamgAa2ccZS6F44apRmzSkXC6jGvNreJ5J8Y9nciWPZCSsk, address:145.79.227.32:4834): protocol: expected piece hash; context deadline exceeded; EOF Dec 22 21:53:20.161797225 (414h25m20.989505624s) [noise=true , address=145.79.227.15:4876 , piece_id=X5KN53INJAXSFQUEY2S37RYA3ELNBGN6ZFM6KKJ3A4ROOSUTBTRQ, seg_pos={0 0}, upload=431251]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:145.79.227.15:4876): protocol: expected piece hash; context deadline exceeded; EOF Dec 22 21:53:20.161824434 (414h25m20.989532813s) [seg_pos={0 0}, upload=431251]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 22 21:53:20.183925866 (414h25m21.011634264s) [upload=431251]: Done uploading segments... Dec 22 21:53:20.184019830 (414h25m21.011728235s) [upload=431251]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 445.527696ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................X........................................] duration: 565.529109ms trace: https://jaeger.dev.storj.io/trace/17ef15eb0c5881c1 FAIL (2025-12-22 22:23:27 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 22 21:52:50.192390642 (414h24m51.020099033s) [upload=431251]: Starting upload Dec 22 21:52:50.192488257 (414h24m51.020196648s) [upload=431251]: Done starting upload Dec 22 21:52:50.192531865 (414h24m51.020240259s) [upload=431251]: Uploading segments... Dec 22 21:52:50.192783054 (414h24m51.020491446s) [upload=431251]: Got next segment. Inline: false Dec 22 21:52:50.192836048 (414h24m51.020544439s) [upload=431251]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 22 21:52:50.192841417 (414h24m51.020549808s) [upload=431251]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 22 21:52:50.220288008 (414h24m51.047996404s) [seg_pos={0 0}, upload=431251]: Begin upload segment... Dec 22 21:52:50.220342100 (414h24m51.048050497s) [seg_pos={0 0}, upload=431251]: Done begin upload segment. Dec 22 21:52:50.220384745 (414h24m51.048093145s) [noise=true , address=77.90.31.10:4517 , piece_id=UAIKGPQF75AM6EKYBV3XK45CNYS2N2PNHKBAKRKO6DDILYAUFBLA, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.220421097 (414h24m51.048129475s) [noise=true , address=145.79.227.45:4229 , piece_id=QYCGIYFFNWLGOBOKEZYY4EBLHKC3SFYWOBQJ34DETCLF6BTS54VQ, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.221068129 (414h24m51.048776527s) [noise=true , address=145.79.227.35:4483 , piece_id=EJIXZOBJWPIVL37GIYYPHQRZDMEMS3YSRPVJH4GLMB4A2CSSRCTQ, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.221771875 (414h24m51.049480268s) [noise=true , address=145.79.227.51:4411 , piece_id=O76ELDU3V3XZ2ZYC7VCLZC2CU5VUMRTFWJEV2ITS5GFAGRSXTOSA, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.222308205 (414h24m51.050016602s) [noise=true , address=145.79.227.39:4015 , piece_id=AZSOOT5RLGTAHSO47SVO27XDSNFPE65JU2I5UEZDPQ44PKB2YLYA, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.222424843 (414h24m51.050133244s) [noise=true , address=77.90.31.18:4805 , piece_id=YWS2P6F2X4PRKE7FVEA722G4FGDV2D4BXJILLHGQE4Q2HXRU2SAA, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.222721983 (414h24m51.050430374s) [noise=true , address=145.79.227.28:4727 , piece_id=Y5UCAOYDWNY5WPZC7LYUOT5QB325BSVZJUGSRVMR64ML23SEI4BQ, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.222910142 (414h24m51.050618542s) [noise=true , address=145.79.227.37:4566 , piece_id=EKCKTUJE37ZDSKNA6JM2ODSBLYAI2KFCA5RVFRNJPSCQ72ZWKY6A, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.223093651 (414h24m51.050802048s) [noise=true , address=145.79.227.50:4373 , piece_id=HESOK4IM7VRJNRL6NY5U4V52G7PDLSE3L3ZOF7DSTSQGFP6D2ZQA, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.223601176 (414h24m51.051309571s) [noise=true , address=145.79.227.15:4876 , piece_id=X5KN53INJAXSFQUEY2S37RYA3ELNBGN6ZFM6KKJ3A4ROOSUTBTRQ, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.224086995 (414h24m51.051795377s) [noise=true , address=145.79.227.43:4157 , piece_id=EKE6ZQ6YMYBBKZKQ2OIAAF5OI3OZRPJF4XDYCCU7QSKS33UXPDZQ, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.224569909 (414h24m51.052278304s) [noise=true , address=77.90.31.11:4553 , piece_id=MJFMBIGW3L6E75AAMZRCDONHZXPFLQSCK3VINMI4I2ZJC5HVHREA, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.225250077 (414h24m51.052958472s) [noise=true , address=145.79.227.48:4301 , piece_id=OCJPU5E7QLJWVY5K444J3J3ISMFY5E6MVXRMXBHIQRHXMIMRYC2A, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.226504914 (414h24m51.054213295s) [noise=true , address=145.79.227.47:4266 , piece_id=EJYUC5ODHWWQEA7QK5JVT3T3HQ76G5WC5OHPNGUSYMP4AW5BWLPA, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.227351850 (414h24m51.055060238s) [noise=true , address=145.79.227.38:4978 , piece_id=S26D5M7SYZEXTTNQEELKEFFQTOMNA5SG7RL5THCON6KCFY267IDQ, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.228140124 (414h24m51.055848511s) [noise=true , address=145.79.227.32:4834 , piece_id=KLWKSAGBRLFWDKNCHW6UWKCADPZ6D6UURDMNMD5UCZUUWAPDOKVQ, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.321476218 (414h24m51.149184612s) [upload=431251]: Next returned nil segment Dec 22 21:52:50.321492753 (414h24m51.149201151s) [upload=431251]: Waiting for error group managing segments... Dec 22 21:52:50.328186730 (414h24m51.155895125s) [noise=true , address=77.90.31.11:4553 , piece_id=MJFMBIGW3L6E75AAMZRCDONHZXPFLQSCK3VINMI4I2ZJC5HVHREA, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.328400062 (414h24m51.156108454s) [noise=true , address=145.79.227.47:4266 , piece_id=EJYUC5ODHWWQEA7QK5JVT3T3HQ76G5WC5OHPNGUSYMP4AW5BWLPA, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.328834800 (414h24m51.156543191s) [noise=true , address=145.79.227.37:4566 , piece_id=EKCKTUJE37ZDSKNA6JM2ODSBLYAI2KFCA5RVFRNJPSCQ72ZWKY6A, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.330369052 (414h24m51.158077447s) [noise=true , address=145.79.227.51:4411 , piece_id=O76ELDU3V3XZ2ZYC7VCLZC2CU5VUMRTFWJEV2ITS5GFAGRSXTOSA, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.330425878 (414h24m51.158134271s) [noise=true , address=145.79.227.28:4727 , piece_id=Y5UCAOYDWNY5WPZC7LYUOT5QB325BSVZJUGSRVMR64ML23SEI4BQ, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.331015687 (414h24m51.158724085s) [noise=true , address=145.79.227.45:4229 , piece_id=QYCGIYFFNWLGOBOKEZYY4EBLHKC3SFYWOBQJ34DETCLF6BTS54VQ, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.331368793 (414h24m51.159077188s) [noise=true , address=145.79.227.48:4301 , piece_id=OCJPU5E7QLJWVY5K444J3J3ISMFY5E6MVXRMXBHIQRHXMIMRYC2A, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.332233159 (414h24m51.159941554s) [noise=true , address=145.79.227.50:4373 , piece_id=HESOK4IM7VRJNRL6NY5U4V52G7PDLSE3L3ZOF7DSTSQGFP6D2ZQA, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.333040553 (414h24m51.160748944s) [noise=true , address=145.79.227.35:4483 , piece_id=EJIXZOBJWPIVL37GIYYPHQRZDMEMS3YSRPVJH4GLMB4A2CSSRCTQ, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.333195450 (414h24m51.160903848s) [noise=true , address=145.79.227.43:4157 , piece_id=EKE6ZQ6YMYBBKZKQ2OIAAF5OI3OZRPJF4XDYCCU7QSKS33UXPDZQ, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.334537739 (414h24m51.162246133s) [noise=true , address=145.79.227.39:4015 , piece_id=AZSOOT5RLGTAHSO47SVO27XDSNFPE65JU2I5UEZDPQ44PKB2YLYA, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.413477504 (414h24m51.2411859s) [noise=true , address=77.90.31.10:4517 , piece_id=UAIKGPQF75AM6EKYBV3XK45CNYS2N2PNHKBAKRKO6DDILYAUFBLA, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:53:20.161293124 (414h25m20.989001514s) [noise=true , address=77.90.31.18:4805 , piece_id=YWS2P6F2X4PRKE7FVEA722G4FGDV2D4BXJILLHGQE4Q2HXRU2SAA, seg_pos={0 0}, upload=431251]: Done uploading piece. err: ecclient: upload failed (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, address:77.90.31.18:4805): protocol: expected piece hash; context deadline exceeded; EOF Dec 22 21:53:20.161472578 (414h25m20.989180974s) [noise=true , address=145.79.227.38:4978 , piece_id=S26D5M7SYZEXTTNQEELKEFFQTOMNA5SG7RL5THCON6KCFY267IDQ, seg_pos={0 0}, upload=431251]: Done uploading piece. err: ecclient: upload failed (node:12PcGHVSm5UAXPh5vCfyRHCd7Bdb67oXF51AZcnRFUjXqVYzXLR, address:145.79.227.38:4978): protocol: expected piece hash; context deadline exceeded; EOF Dec 22 21:53:20.161591128 (414h25m20.989299529s) [noise=true , address=145.79.227.32:4834 , piece_id=KLWKSAGBRLFWDKNCHW6UWKCADPZ6D6UURDMNMD5UCZUUWAPDOKVQ, seg_pos={0 0}, upload=431251]: Done uploading piece. err: ecclient: upload failed (node:1gRnamgAa2ccZS6F44apRmzSkXC6jGvNreJ5J8Y9nciWPZCSsk, address:145.79.227.32:4834): protocol: expected piece hash; context deadline exceeded; EOF Dec 22 21:53:20.161797225 (414h25m20.989505624s) [noise=true , address=145.79.227.15:4876 , piece_id=X5KN53INJAXSFQUEY2S37RYA3ELNBGN6ZFM6KKJ3A4ROOSUTBTRQ, seg_pos={0 0}, upload=431251]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:145.79.227.15:4876): protocol: expected piece hash; context deadline exceeded; EOF Dec 22 21:53:20.161824434 (414h25m20.989532813s) [seg_pos={0 0}, upload=431251]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 22 21:53:20.183925866 (414h25m21.011634264s) [upload=431251]: Done uploading segments... Dec 22 21:53:20.184019830 (414h25m21.011728235s) [upload=431251]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 565.550301ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................X.......................................] duration: 548.081435ms trace: https://jaeger.dev.storj.io/trace/17ef15eb0c5881c1 FAIL (2025-12-22 22:22:53 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 22 21:52:50.192390642 (414h24m51.020099033s) [upload=431251]: Starting upload Dec 22 21:52:50.192488257 (414h24m51.020196648s) [upload=431251]: Done starting upload Dec 22 21:52:50.192531865 (414h24m51.020240259s) [upload=431251]: Uploading segments... Dec 22 21:52:50.192783054 (414h24m51.020491446s) [upload=431251]: Got next segment. Inline: false Dec 22 21:52:50.192836048 (414h24m51.020544439s) [upload=431251]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 22 21:52:50.192841417 (414h24m51.020549808s) [upload=431251]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 22 21:52:50.220288008 (414h24m51.047996404s) [seg_pos={0 0}, upload=431251]: Begin upload segment... Dec 22 21:52:50.220342100 (414h24m51.048050497s) [seg_pos={0 0}, upload=431251]: Done begin upload segment. Dec 22 21:52:50.220384745 (414h24m51.048093145s) [noise=true , address=77.90.31.10:4517 , piece_id=UAIKGPQF75AM6EKYBV3XK45CNYS2N2PNHKBAKRKO6DDILYAUFBLA, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.220421097 (414h24m51.048129475s) [noise=true , address=145.79.227.45:4229 , piece_id=QYCGIYFFNWLGOBOKEZYY4EBLHKC3SFYWOBQJ34DETCLF6BTS54VQ, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.221068129 (414h24m51.048776527s) [noise=true , address=145.79.227.35:4483 , piece_id=EJIXZOBJWPIVL37GIYYPHQRZDMEMS3YSRPVJH4GLMB4A2CSSRCTQ, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.221771875 (414h24m51.049480268s) [noise=true , address=145.79.227.51:4411 , piece_id=O76ELDU3V3XZ2ZYC7VCLZC2CU5VUMRTFWJEV2ITS5GFAGRSXTOSA, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.222308205 (414h24m51.050016602s) [noise=true , address=145.79.227.39:4015 , piece_id=AZSOOT5RLGTAHSO47SVO27XDSNFPE65JU2I5UEZDPQ44PKB2YLYA, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.222424843 (414h24m51.050133244s) [noise=true , address=77.90.31.18:4805 , piece_id=YWS2P6F2X4PRKE7FVEA722G4FGDV2D4BXJILLHGQE4Q2HXRU2SAA, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.222721983 (414h24m51.050430374s) [noise=true , address=145.79.227.28:4727 , piece_id=Y5UCAOYDWNY5WPZC7LYUOT5QB325BSVZJUGSRVMR64ML23SEI4BQ, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.222910142 (414h24m51.050618542s) [noise=true , address=145.79.227.37:4566 , piece_id=EKCKTUJE37ZDSKNA6JM2ODSBLYAI2KFCA5RVFRNJPSCQ72ZWKY6A, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.223093651 (414h24m51.050802048s) [noise=true , address=145.79.227.50:4373 , piece_id=HESOK4IM7VRJNRL6NY5U4V52G7PDLSE3L3ZOF7DSTSQGFP6D2ZQA, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.223601176 (414h24m51.051309571s) [noise=true , address=145.79.227.15:4876 , piece_id=X5KN53INJAXSFQUEY2S37RYA3ELNBGN6ZFM6KKJ3A4ROOSUTBTRQ, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.224086995 (414h24m51.051795377s) [noise=true , address=145.79.227.43:4157 , piece_id=EKE6ZQ6YMYBBKZKQ2OIAAF5OI3OZRPJF4XDYCCU7QSKS33UXPDZQ, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.224569909 (414h24m51.052278304s) [noise=true , address=77.90.31.11:4553 , piece_id=MJFMBIGW3L6E75AAMZRCDONHZXPFLQSCK3VINMI4I2ZJC5HVHREA, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.225250077 (414h24m51.052958472s) [noise=true , address=145.79.227.48:4301 , piece_id=OCJPU5E7QLJWVY5K444J3J3ISMFY5E6MVXRMXBHIQRHXMIMRYC2A, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.226504914 (414h24m51.054213295s) [noise=true , address=145.79.227.47:4266 , piece_id=EJYUC5ODHWWQEA7QK5JVT3T3HQ76G5WC5OHPNGUSYMP4AW5BWLPA, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.227351850 (414h24m51.055060238s) [noise=true , address=145.79.227.38:4978 , piece_id=S26D5M7SYZEXTTNQEELKEFFQTOMNA5SG7RL5THCON6KCFY267IDQ, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.228140124 (414h24m51.055848511s) [noise=true , address=145.79.227.32:4834 , piece_id=KLWKSAGBRLFWDKNCHW6UWKCADPZ6D6UURDMNMD5UCZUUWAPDOKVQ, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.321476218 (414h24m51.149184612s) [upload=431251]: Next returned nil segment Dec 22 21:52:50.321492753 (414h24m51.149201151s) [upload=431251]: Waiting for error group managing segments... Dec 22 21:52:50.328186730 (414h24m51.155895125s) [noise=true , address=77.90.31.11:4553 , piece_id=MJFMBIGW3L6E75AAMZRCDONHZXPFLQSCK3VINMI4I2ZJC5HVHREA, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.328400062 (414h24m51.156108454s) [noise=true , address=145.79.227.47:4266 , piece_id=EJYUC5ODHWWQEA7QK5JVT3T3HQ76G5WC5OHPNGUSYMP4AW5BWLPA, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.328834800 (414h24m51.156543191s) [noise=true , address=145.79.227.37:4566 , piece_id=EKCKTUJE37ZDSKNA6JM2ODSBLYAI2KFCA5RVFRNJPSCQ72ZWKY6A, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.330369052 (414h24m51.158077447s) [noise=true , address=145.79.227.51:4411 , piece_id=O76ELDU3V3XZ2ZYC7VCLZC2CU5VUMRTFWJEV2ITS5GFAGRSXTOSA, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.330425878 (414h24m51.158134271s) [noise=true , address=145.79.227.28:4727 , piece_id=Y5UCAOYDWNY5WPZC7LYUOT5QB325BSVZJUGSRVMR64ML23SEI4BQ, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.331015687 (414h24m51.158724085s) [noise=true , address=145.79.227.45:4229 , piece_id=QYCGIYFFNWLGOBOKEZYY4EBLHKC3SFYWOBQJ34DETCLF6BTS54VQ, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.331368793 (414h24m51.159077188s) [noise=true , address=145.79.227.48:4301 , piece_id=OCJPU5E7QLJWVY5K444J3J3ISMFY5E6MVXRMXBHIQRHXMIMRYC2A, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.332233159 (414h24m51.159941554s) [noise=true , address=145.79.227.50:4373 , piece_id=HESOK4IM7VRJNRL6NY5U4V52G7PDLSE3L3ZOF7DSTSQGFP6D2ZQA, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.333040553 (414h24m51.160748944s) [noise=true , address=145.79.227.35:4483 , piece_id=EJIXZOBJWPIVL37GIYYPHQRZDMEMS3YSRPVJH4GLMB4A2CSSRCTQ, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.333195450 (414h24m51.160903848s) [noise=true , address=145.79.227.43:4157 , piece_id=EKE6ZQ6YMYBBKZKQ2OIAAF5OI3OZRPJF4XDYCCU7QSKS33UXPDZQ, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.334537739 (414h24m51.162246133s) [noise=true , address=145.79.227.39:4015 , piece_id=AZSOOT5RLGTAHSO47SVO27XDSNFPE65JU2I5UEZDPQ44PKB2YLYA, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.413477504 (414h24m51.2411859s) [noise=true , address=77.90.31.10:4517 , piece_id=UAIKGPQF75AM6EKYBV3XK45CNYS2N2PNHKBAKRKO6DDILYAUFBLA, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:53:20.161293124 (414h25m20.989001514s) [noise=true , address=77.90.31.18:4805 , piece_id=YWS2P6F2X4PRKE7FVEA722G4FGDV2D4BXJILLHGQE4Q2HXRU2SAA, seg_pos={0 0}, upload=431251]: Done uploading piece. err: ecclient: upload failed (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, address:77.90.31.18:4805): protocol: expected piece hash; context deadline exceeded; EOF Dec 22 21:53:20.161472578 (414h25m20.989180974s) [noise=true , address=145.79.227.38:4978 , piece_id=S26D5M7SYZEXTTNQEELKEFFQTOMNA5SG7RL5THCON6KCFY267IDQ, seg_pos={0 0}, upload=431251]: Done uploading piece. err: ecclient: upload failed (node:12PcGHVSm5UAXPh5vCfyRHCd7Bdb67oXF51AZcnRFUjXqVYzXLR, address:145.79.227.38:4978): protocol: expected piece hash; context deadline exceeded; EOF Dec 22 21:53:20.161591128 (414h25m20.989299529s) [noise=true , address=145.79.227.32:4834 , piece_id=KLWKSAGBRLFWDKNCHW6UWKCADPZ6D6UURDMNMD5UCZUUWAPDOKVQ, seg_pos={0 0}, upload=431251]: Done uploading piece. err: ecclient: upload failed (node:1gRnamgAa2ccZS6F44apRmzSkXC6jGvNreJ5J8Y9nciWPZCSsk, address:145.79.227.32:4834): protocol: expected piece hash; context deadline exceeded; EOF Dec 22 21:53:20.161797225 (414h25m20.989505624s) [noise=true , address=145.79.227.15:4876 , piece_id=X5KN53INJAXSFQUEY2S37RYA3ELNBGN6ZFM6KKJ3A4ROOSUTBTRQ, seg_pos={0 0}, upload=431251]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:145.79.227.15:4876): protocol: expected piece hash; context deadline exceeded; EOF Dec 22 21:53:20.161824434 (414h25m20.989532813s) [seg_pos={0 0}, upload=431251]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 22 21:53:20.183925866 (414h25m21.011634264s) [upload=431251]: Done uploading segments... Dec 22 21:53:20.184019830 (414h25m21.011728235s) [upload=431251]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 548.156294ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................X......................................] duration: 673.023445ms trace: https://jaeger.dev.storj.io/trace/17ef15eb0c5881c1 FAIL (2025-12-22 22:22:04 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 22 21:52:50.192390642 (414h24m51.020099033s) [upload=431251]: Starting upload Dec 22 21:52:50.192488257 (414h24m51.020196648s) [upload=431251]: Done starting upload Dec 22 21:52:50.192531865 (414h24m51.020240259s) [upload=431251]: Uploading segments... Dec 22 21:52:50.192783054 (414h24m51.020491446s) [upload=431251]: Got next segment. Inline: false Dec 22 21:52:50.192836048 (414h24m51.020544439s) [upload=431251]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 22 21:52:50.192841417 (414h24m51.020549808s) [upload=431251]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 22 21:52:50.220288008 (414h24m51.047996404s) [seg_pos={0 0}, upload=431251]: Begin upload segment... Dec 22 21:52:50.220342100 (414h24m51.048050497s) [seg_pos={0 0}, upload=431251]: Done begin upload segment. Dec 22 21:52:50.220384745 (414h24m51.048093145s) [noise=true , address=77.90.31.10:4517 , piece_id=UAIKGPQF75AM6EKYBV3XK45CNYS2N2PNHKBAKRKO6DDILYAUFBLA, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.220421097 (414h24m51.048129475s) [noise=true , address=145.79.227.45:4229 , piece_id=QYCGIYFFNWLGOBOKEZYY4EBLHKC3SFYWOBQJ34DETCLF6BTS54VQ, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.221068129 (414h24m51.048776527s) [noise=true , address=145.79.227.35:4483 , piece_id=EJIXZOBJWPIVL37GIYYPHQRZDMEMS3YSRPVJH4GLMB4A2CSSRCTQ, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.221771875 (414h24m51.049480268s) [noise=true , address=145.79.227.51:4411 , piece_id=O76ELDU3V3XZ2ZYC7VCLZC2CU5VUMRTFWJEV2ITS5GFAGRSXTOSA, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.222308205 (414h24m51.050016602s) [noise=true , address=145.79.227.39:4015 , piece_id=AZSOOT5RLGTAHSO47SVO27XDSNFPE65JU2I5UEZDPQ44PKB2YLYA, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.222424843 (414h24m51.050133244s) [noise=true , address=77.90.31.18:4805 , piece_id=YWS2P6F2X4PRKE7FVEA722G4FGDV2D4BXJILLHGQE4Q2HXRU2SAA, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.222721983 (414h24m51.050430374s) [noise=true , address=145.79.227.28:4727 , piece_id=Y5UCAOYDWNY5WPZC7LYUOT5QB325BSVZJUGSRVMR64ML23SEI4BQ, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.222910142 (414h24m51.050618542s) [noise=true , address=145.79.227.37:4566 , piece_id=EKCKTUJE37ZDSKNA6JM2ODSBLYAI2KFCA5RVFRNJPSCQ72ZWKY6A, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.223093651 (414h24m51.050802048s) [noise=true , address=145.79.227.50:4373 , piece_id=HESOK4IM7VRJNRL6NY5U4V52G7PDLSE3L3ZOF7DSTSQGFP6D2ZQA, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.223601176 (414h24m51.051309571s) [noise=true , address=145.79.227.15:4876 , piece_id=X5KN53INJAXSFQUEY2S37RYA3ELNBGN6ZFM6KKJ3A4ROOSUTBTRQ, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.224086995 (414h24m51.051795377s) [noise=true , address=145.79.227.43:4157 , piece_id=EKE6ZQ6YMYBBKZKQ2OIAAF5OI3OZRPJF4XDYCCU7QSKS33UXPDZQ, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.224569909 (414h24m51.052278304s) [noise=true , address=77.90.31.11:4553 , piece_id=MJFMBIGW3L6E75AAMZRCDONHZXPFLQSCK3VINMI4I2ZJC5HVHREA, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.225250077 (414h24m51.052958472s) [noise=true , address=145.79.227.48:4301 , piece_id=OCJPU5E7QLJWVY5K444J3J3ISMFY5E6MVXRMXBHIQRHXMIMRYC2A, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.226504914 (414h24m51.054213295s) [noise=true , address=145.79.227.47:4266 , piece_id=EJYUC5ODHWWQEA7QK5JVT3T3HQ76G5WC5OHPNGUSYMP4AW5BWLPA, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.227351850 (414h24m51.055060238s) [noise=true , address=145.79.227.38:4978 , piece_id=S26D5M7SYZEXTTNQEELKEFFQTOMNA5SG7RL5THCON6KCFY267IDQ, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.228140124 (414h24m51.055848511s) [noise=true , address=145.79.227.32:4834 , piece_id=KLWKSAGBRLFWDKNCHW6UWKCADPZ6D6UURDMNMD5UCZUUWAPDOKVQ, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.321476218 (414h24m51.149184612s) [upload=431251]: Next returned nil segment Dec 22 21:52:50.321492753 (414h24m51.149201151s) [upload=431251]: Waiting for error group managing segments... Dec 22 21:52:50.328186730 (414h24m51.155895125s) [noise=true , address=77.90.31.11:4553 , piece_id=MJFMBIGW3L6E75AAMZRCDONHZXPFLQSCK3VINMI4I2ZJC5HVHREA, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.328400062 (414h24m51.156108454s) [noise=true , address=145.79.227.47:4266 , piece_id=EJYUC5ODHWWQEA7QK5JVT3T3HQ76G5WC5OHPNGUSYMP4AW5BWLPA, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.328834800 (414h24m51.156543191s) [noise=true , address=145.79.227.37:4566 , piece_id=EKCKTUJE37ZDSKNA6JM2ODSBLYAI2KFCA5RVFRNJPSCQ72ZWKY6A, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.330369052 (414h24m51.158077447s) [noise=true , address=145.79.227.51:4411 , piece_id=O76ELDU3V3XZ2ZYC7VCLZC2CU5VUMRTFWJEV2ITS5GFAGRSXTOSA, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.330425878 (414h24m51.158134271s) [noise=true , address=145.79.227.28:4727 , piece_id=Y5UCAOYDWNY5WPZC7LYUOT5QB325BSVZJUGSRVMR64ML23SEI4BQ, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.331015687 (414h24m51.158724085s) [noise=true , address=145.79.227.45:4229 , piece_id=QYCGIYFFNWLGOBOKEZYY4EBLHKC3SFYWOBQJ34DETCLF6BTS54VQ, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.331368793 (414h24m51.159077188s) [noise=true , address=145.79.227.48:4301 , piece_id=OCJPU5E7QLJWVY5K444J3J3ISMFY5E6MVXRMXBHIQRHXMIMRYC2A, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.332233159 (414h24m51.159941554s) [noise=true , address=145.79.227.50:4373 , piece_id=HESOK4IM7VRJNRL6NY5U4V52G7PDLSE3L3ZOF7DSTSQGFP6D2ZQA, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.333040553 (414h24m51.160748944s) [noise=true , address=145.79.227.35:4483 , piece_id=EJIXZOBJWPIVL37GIYYPHQRZDMEMS3YSRPVJH4GLMB4A2CSSRCTQ, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.333195450 (414h24m51.160903848s) [noise=true , address=145.79.227.43:4157 , piece_id=EKE6ZQ6YMYBBKZKQ2OIAAF5OI3OZRPJF4XDYCCU7QSKS33UXPDZQ, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.334537739 (414h24m51.162246133s) [noise=true , address=145.79.227.39:4015 , piece_id=AZSOOT5RLGTAHSO47SVO27XDSNFPE65JU2I5UEZDPQ44PKB2YLYA, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.413477504 (414h24m51.2411859s) [noise=true , address=77.90.31.10:4517 , piece_id=UAIKGPQF75AM6EKYBV3XK45CNYS2N2PNHKBAKRKO6DDILYAUFBLA, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:53:20.161293124 (414h25m20.989001514s) [noise=true , address=77.90.31.18:4805 , piece_id=YWS2P6F2X4PRKE7FVEA722G4FGDV2D4BXJILLHGQE4Q2HXRU2SAA, seg_pos={0 0}, upload=431251]: Done uploading piece. err: ecclient: upload failed (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, address:77.90.31.18:4805): protocol: expected piece hash; context deadline exceeded; EOF Dec 22 21:53:20.161472578 (414h25m20.989180974s) [noise=true , address=145.79.227.38:4978 , piece_id=S26D5M7SYZEXTTNQEELKEFFQTOMNA5SG7RL5THCON6KCFY267IDQ, seg_pos={0 0}, upload=431251]: Done uploading piece. err: ecclient: upload failed (node:12PcGHVSm5UAXPh5vCfyRHCd7Bdb67oXF51AZcnRFUjXqVYzXLR, address:145.79.227.38:4978): protocol: expected piece hash; context deadline exceeded; EOF Dec 22 21:53:20.161591128 (414h25m20.989299529s) [noise=true , address=145.79.227.32:4834 , piece_id=KLWKSAGBRLFWDKNCHW6UWKCADPZ6D6UURDMNMD5UCZUUWAPDOKVQ, seg_pos={0 0}, upload=431251]: Done uploading piece. err: ecclient: upload failed (node:1gRnamgAa2ccZS6F44apRmzSkXC6jGvNreJ5J8Y9nciWPZCSsk, address:145.79.227.32:4834): protocol: expected piece hash; context deadline exceeded; EOF Dec 22 21:53:20.161797225 (414h25m20.989505624s) [noise=true , address=145.79.227.15:4876 , piece_id=X5KN53INJAXSFQUEY2S37RYA3ELNBGN6ZFM6KKJ3A4ROOSUTBTRQ, seg_pos={0 0}, upload=431251]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:145.79.227.15:4876): protocol: expected piece hash; context deadline exceeded; EOF Dec 22 21:53:20.161824434 (414h25m20.989532813s) [seg_pos={0 0}, upload=431251]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 22 21:53:20.183925866 (414h25m21.011634264s) [upload=431251]: Done uploading segments... Dec 22 21:53:20.184019830 (414h25m21.011728235s) [upload=431251]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 673.052394ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................X.....................................] duration: 471.295992ms trace: https://jaeger.dev.storj.io/trace/17ef15eb0c5881c1 FAIL (2025-12-22 22:21:03 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 22 21:52:50.192390642 (414h24m51.020099033s) [upload=431251]: Starting upload Dec 22 21:52:50.192488257 (414h24m51.020196648s) [upload=431251]: Done starting upload Dec 22 21:52:50.192531865 (414h24m51.020240259s) [upload=431251]: Uploading segments... Dec 22 21:52:50.192783054 (414h24m51.020491446s) [upload=431251]: Got next segment. Inline: false Dec 22 21:52:50.192836048 (414h24m51.020544439s) [upload=431251]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 22 21:52:50.192841417 (414h24m51.020549808s) [upload=431251]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 22 21:52:50.220288008 (414h24m51.047996404s) [seg_pos={0 0}, upload=431251]: Begin upload segment... Dec 22 21:52:50.220342100 (414h24m51.048050497s) [seg_pos={0 0}, upload=431251]: Done begin upload segment. Dec 22 21:52:50.220384745 (414h24m51.048093145s) [noise=true , address=77.90.31.10:4517 , piece_id=UAIKGPQF75AM6EKYBV3XK45CNYS2N2PNHKBAKRKO6DDILYAUFBLA, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.220421097 (414h24m51.048129475s) [noise=true , address=145.79.227.45:4229 , piece_id=QYCGIYFFNWLGOBOKEZYY4EBLHKC3SFYWOBQJ34DETCLF6BTS54VQ, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.221068129 (414h24m51.048776527s) [noise=true , address=145.79.227.35:4483 , piece_id=EJIXZOBJWPIVL37GIYYPHQRZDMEMS3YSRPVJH4GLMB4A2CSSRCTQ, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.221771875 (414h24m51.049480268s) [noise=true , address=145.79.227.51:4411 , piece_id=O76ELDU3V3XZ2ZYC7VCLZC2CU5VUMRTFWJEV2ITS5GFAGRSXTOSA, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.222308205 (414h24m51.050016602s) [noise=true , address=145.79.227.39:4015 , piece_id=AZSOOT5RLGTAHSO47SVO27XDSNFPE65JU2I5UEZDPQ44PKB2YLYA, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.222424843 (414h24m51.050133244s) [noise=true , address=77.90.31.18:4805 , piece_id=YWS2P6F2X4PRKE7FVEA722G4FGDV2D4BXJILLHGQE4Q2HXRU2SAA, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.222721983 (414h24m51.050430374s) [noise=true , address=145.79.227.28:4727 , piece_id=Y5UCAOYDWNY5WPZC7LYUOT5QB325BSVZJUGSRVMR64ML23SEI4BQ, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.222910142 (414h24m51.050618542s) [noise=true , address=145.79.227.37:4566 , piece_id=EKCKTUJE37ZDSKNA6JM2ODSBLYAI2KFCA5RVFRNJPSCQ72ZWKY6A, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.223093651 (414h24m51.050802048s) [noise=true , address=145.79.227.50:4373 , piece_id=HESOK4IM7VRJNRL6NY5U4V52G7PDLSE3L3ZOF7DSTSQGFP6D2ZQA, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.223601176 (414h24m51.051309571s) [noise=true , address=145.79.227.15:4876 , piece_id=X5KN53INJAXSFQUEY2S37RYA3ELNBGN6ZFM6KKJ3A4ROOSUTBTRQ, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.224086995 (414h24m51.051795377s) [noise=true , address=145.79.227.43:4157 , piece_id=EKE6ZQ6YMYBBKZKQ2OIAAF5OI3OZRPJF4XDYCCU7QSKS33UXPDZQ, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.224569909 (414h24m51.052278304s) [noise=true , address=77.90.31.11:4553 , piece_id=MJFMBIGW3L6E75AAMZRCDONHZXPFLQSCK3VINMI4I2ZJC5HVHREA, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.225250077 (414h24m51.052958472s) [noise=true , address=145.79.227.48:4301 , piece_id=OCJPU5E7QLJWVY5K444J3J3ISMFY5E6MVXRMXBHIQRHXMIMRYC2A, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.226504914 (414h24m51.054213295s) [noise=true , address=145.79.227.47:4266 , piece_id=EJYUC5ODHWWQEA7QK5JVT3T3HQ76G5WC5OHPNGUSYMP4AW5BWLPA, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.227351850 (414h24m51.055060238s) [noise=true , address=145.79.227.38:4978 , piece_id=S26D5M7SYZEXTTNQEELKEFFQTOMNA5SG7RL5THCON6KCFY267IDQ, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.228140124 (414h24m51.055848511s) [noise=true , address=145.79.227.32:4834 , piece_id=KLWKSAGBRLFWDKNCHW6UWKCADPZ6D6UURDMNMD5UCZUUWAPDOKVQ, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.321476218 (414h24m51.149184612s) [upload=431251]: Next returned nil segment Dec 22 21:52:50.321492753 (414h24m51.149201151s) [upload=431251]: Waiting for error group managing segments... Dec 22 21:52:50.328186730 (414h24m51.155895125s) [noise=true , address=77.90.31.11:4553 , piece_id=MJFMBIGW3L6E75AAMZRCDONHZXPFLQSCK3VINMI4I2ZJC5HVHREA, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.328400062 (414h24m51.156108454s) [noise=true , address=145.79.227.47:4266 , piece_id=EJYUC5ODHWWQEA7QK5JVT3T3HQ76G5WC5OHPNGUSYMP4AW5BWLPA, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.328834800 (414h24m51.156543191s) [noise=true , address=145.79.227.37:4566 , piece_id=EKCKTUJE37ZDSKNA6JM2ODSBLYAI2KFCA5RVFRNJPSCQ72ZWKY6A, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.330369052 (414h24m51.158077447s) [noise=true , address=145.79.227.51:4411 , piece_id=O76ELDU3V3XZ2ZYC7VCLZC2CU5VUMRTFWJEV2ITS5GFAGRSXTOSA, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.330425878 (414h24m51.158134271s) [noise=true , address=145.79.227.28:4727 , piece_id=Y5UCAOYDWNY5WPZC7LYUOT5QB325BSVZJUGSRVMR64ML23SEI4BQ, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.331015687 (414h24m51.158724085s) [noise=true , address=145.79.227.45:4229 , piece_id=QYCGIYFFNWLGOBOKEZYY4EBLHKC3SFYWOBQJ34DETCLF6BTS54VQ, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.331368793 (414h24m51.159077188s) [noise=true , address=145.79.227.48:4301 , piece_id=OCJPU5E7QLJWVY5K444J3J3ISMFY5E6MVXRMXBHIQRHXMIMRYC2A, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.332233159 (414h24m51.159941554s) [noise=true , address=145.79.227.50:4373 , piece_id=HESOK4IM7VRJNRL6NY5U4V52G7PDLSE3L3ZOF7DSTSQGFP6D2ZQA, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.333040553 (414h24m51.160748944s) [noise=true , address=145.79.227.35:4483 , piece_id=EJIXZOBJWPIVL37GIYYPHQRZDMEMS3YSRPVJH4GLMB4A2CSSRCTQ, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.333195450 (414h24m51.160903848s) [noise=true , address=145.79.227.43:4157 , piece_id=EKE6ZQ6YMYBBKZKQ2OIAAF5OI3OZRPJF4XDYCCU7QSKS33UXPDZQ, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.334537739 (414h24m51.162246133s) [noise=true , address=145.79.227.39:4015 , piece_id=AZSOOT5RLGTAHSO47SVO27XDSNFPE65JU2I5UEZDPQ44PKB2YLYA, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.413477504 (414h24m51.2411859s) [noise=true , address=77.90.31.10:4517 , piece_id=UAIKGPQF75AM6EKYBV3XK45CNYS2N2PNHKBAKRKO6DDILYAUFBLA, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:53:20.161293124 (414h25m20.989001514s) [noise=true , address=77.90.31.18:4805 , piece_id=YWS2P6F2X4PRKE7FVEA722G4FGDV2D4BXJILLHGQE4Q2HXRU2SAA, seg_pos={0 0}, upload=431251]: Done uploading piece. err: ecclient: upload failed (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, address:77.90.31.18:4805): protocol: expected piece hash; context deadline exceeded; EOF Dec 22 21:53:20.161472578 (414h25m20.989180974s) [noise=true , address=145.79.227.38:4978 , piece_id=S26D5M7SYZEXTTNQEELKEFFQTOMNA5SG7RL5THCON6KCFY267IDQ, seg_pos={0 0}, upload=431251]: Done uploading piece. err: ecclient: upload failed (node:12PcGHVSm5UAXPh5vCfyRHCd7Bdb67oXF51AZcnRFUjXqVYzXLR, address:145.79.227.38:4978): protocol: expected piece hash; context deadline exceeded; EOF Dec 22 21:53:20.161591128 (414h25m20.989299529s) [noise=true , address=145.79.227.32:4834 , piece_id=KLWKSAGBRLFWDKNCHW6UWKCADPZ6D6UURDMNMD5UCZUUWAPDOKVQ, seg_pos={0 0}, upload=431251]: Done uploading piece. err: ecclient: upload failed (node:1gRnamgAa2ccZS6F44apRmzSkXC6jGvNreJ5J8Y9nciWPZCSsk, address:145.79.227.32:4834): protocol: expected piece hash; context deadline exceeded; EOF Dec 22 21:53:20.161797225 (414h25m20.989505624s) [noise=true , address=145.79.227.15:4876 , piece_id=X5KN53INJAXSFQUEY2S37RYA3ELNBGN6ZFM6KKJ3A4ROOSUTBTRQ, seg_pos={0 0}, upload=431251]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:145.79.227.15:4876): protocol: expected piece hash; context deadline exceeded; EOF Dec 22 21:53:20.161824434 (414h25m20.989532813s) [seg_pos={0 0}, upload=431251]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 22 21:53:20.183925866 (414h25m21.011634264s) [upload=431251]: Done uploading segments... Dec 22 21:53:20.184019830 (414h25m21.011728235s) [upload=431251]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 471.328287ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................X....................................] duration: 544.834482ms trace: https://jaeger.dev.storj.io/trace/17ef15eb0c5881c1 FAIL (2025-12-22 22:20:28 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 22 21:52:50.192390642 (414h24m51.020099033s) [upload=431251]: Starting upload Dec 22 21:52:50.192488257 (414h24m51.020196648s) [upload=431251]: Done starting upload Dec 22 21:52:50.192531865 (414h24m51.020240259s) [upload=431251]: Uploading segments... Dec 22 21:52:50.192783054 (414h24m51.020491446s) [upload=431251]: Got next segment. Inline: false Dec 22 21:52:50.192836048 (414h24m51.020544439s) [upload=431251]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 22 21:52:50.192841417 (414h24m51.020549808s) [upload=431251]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 22 21:52:50.220288008 (414h24m51.047996404s) [seg_pos={0 0}, upload=431251]: Begin upload segment... Dec 22 21:52:50.220342100 (414h24m51.048050497s) [seg_pos={0 0}, upload=431251]: Done begin upload segment. Dec 22 21:52:50.220384745 (414h24m51.048093145s) [noise=true , address=77.90.31.10:4517 , piece_id=UAIKGPQF75AM6EKYBV3XK45CNYS2N2PNHKBAKRKO6DDILYAUFBLA, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.220421097 (414h24m51.048129475s) [noise=true , address=145.79.227.45:4229 , piece_id=QYCGIYFFNWLGOBOKEZYY4EBLHKC3SFYWOBQJ34DETCLF6BTS54VQ, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.221068129 (414h24m51.048776527s) [noise=true , address=145.79.227.35:4483 , piece_id=EJIXZOBJWPIVL37GIYYPHQRZDMEMS3YSRPVJH4GLMB4A2CSSRCTQ, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.221771875 (414h24m51.049480268s) [noise=true , address=145.79.227.51:4411 , piece_id=O76ELDU3V3XZ2ZYC7VCLZC2CU5VUMRTFWJEV2ITS5GFAGRSXTOSA, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.222308205 (414h24m51.050016602s) [noise=true , address=145.79.227.39:4015 , piece_id=AZSOOT5RLGTAHSO47SVO27XDSNFPE65JU2I5UEZDPQ44PKB2YLYA, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.222424843 (414h24m51.050133244s) [noise=true , address=77.90.31.18:4805 , piece_id=YWS2P6F2X4PRKE7FVEA722G4FGDV2D4BXJILLHGQE4Q2HXRU2SAA, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.222721983 (414h24m51.050430374s) [noise=true , address=145.79.227.28:4727 , piece_id=Y5UCAOYDWNY5WPZC7LYUOT5QB325BSVZJUGSRVMR64ML23SEI4BQ, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.222910142 (414h24m51.050618542s) [noise=true , address=145.79.227.37:4566 , piece_id=EKCKTUJE37ZDSKNA6JM2ODSBLYAI2KFCA5RVFRNJPSCQ72ZWKY6A, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.223093651 (414h24m51.050802048s) [noise=true , address=145.79.227.50:4373 , piece_id=HESOK4IM7VRJNRL6NY5U4V52G7PDLSE3L3ZOF7DSTSQGFP6D2ZQA, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.223601176 (414h24m51.051309571s) [noise=true , address=145.79.227.15:4876 , piece_id=X5KN53INJAXSFQUEY2S37RYA3ELNBGN6ZFM6KKJ3A4ROOSUTBTRQ, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.224086995 (414h24m51.051795377s) [noise=true , address=145.79.227.43:4157 , piece_id=EKE6ZQ6YMYBBKZKQ2OIAAF5OI3OZRPJF4XDYCCU7QSKS33UXPDZQ, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.224569909 (414h24m51.052278304s) [noise=true , address=77.90.31.11:4553 , piece_id=MJFMBIGW3L6E75AAMZRCDONHZXPFLQSCK3VINMI4I2ZJC5HVHREA, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.225250077 (414h24m51.052958472s) [noise=true , address=145.79.227.48:4301 , piece_id=OCJPU5E7QLJWVY5K444J3J3ISMFY5E6MVXRMXBHIQRHXMIMRYC2A, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.226504914 (414h24m51.054213295s) [noise=true , address=145.79.227.47:4266 , piece_id=EJYUC5ODHWWQEA7QK5JVT3T3HQ76G5WC5OHPNGUSYMP4AW5BWLPA, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.227351850 (414h24m51.055060238s) [noise=true , address=145.79.227.38:4978 , piece_id=S26D5M7SYZEXTTNQEELKEFFQTOMNA5SG7RL5THCON6KCFY267IDQ, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.228140124 (414h24m51.055848511s) [noise=true , address=145.79.227.32:4834 , piece_id=KLWKSAGBRLFWDKNCHW6UWKCADPZ6D6UURDMNMD5UCZUUWAPDOKVQ, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.321476218 (414h24m51.149184612s) [upload=431251]: Next returned nil segment Dec 22 21:52:50.321492753 (414h24m51.149201151s) [upload=431251]: Waiting for error group managing segments... Dec 22 21:52:50.328186730 (414h24m51.155895125s) [noise=true , address=77.90.31.11:4553 , piece_id=MJFMBIGW3L6E75AAMZRCDONHZXPFLQSCK3VINMI4I2ZJC5HVHREA, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.328400062 (414h24m51.156108454s) [noise=true , address=145.79.227.47:4266 , piece_id=EJYUC5ODHWWQEA7QK5JVT3T3HQ76G5WC5OHPNGUSYMP4AW5BWLPA, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.328834800 (414h24m51.156543191s) [noise=true , address=145.79.227.37:4566 , piece_id=EKCKTUJE37ZDSKNA6JM2ODSBLYAI2KFCA5RVFRNJPSCQ72ZWKY6A, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.330369052 (414h24m51.158077447s) [noise=true , address=145.79.227.51:4411 , piece_id=O76ELDU3V3XZ2ZYC7VCLZC2CU5VUMRTFWJEV2ITS5GFAGRSXTOSA, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.330425878 (414h24m51.158134271s) [noise=true , address=145.79.227.28:4727 , piece_id=Y5UCAOYDWNY5WPZC7LYUOT5QB325BSVZJUGSRVMR64ML23SEI4BQ, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.331015687 (414h24m51.158724085s) [noise=true , address=145.79.227.45:4229 , piece_id=QYCGIYFFNWLGOBOKEZYY4EBLHKC3SFYWOBQJ34DETCLF6BTS54VQ, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.331368793 (414h24m51.159077188s) [noise=true , address=145.79.227.48:4301 , piece_id=OCJPU5E7QLJWVY5K444J3J3ISMFY5E6MVXRMXBHIQRHXMIMRYC2A, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.332233159 (414h24m51.159941554s) [noise=true , address=145.79.227.50:4373 , piece_id=HESOK4IM7VRJNRL6NY5U4V52G7PDLSE3L3ZOF7DSTSQGFP6D2ZQA, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.333040553 (414h24m51.160748944s) [noise=true , address=145.79.227.35:4483 , piece_id=EJIXZOBJWPIVL37GIYYPHQRZDMEMS3YSRPVJH4GLMB4A2CSSRCTQ, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.333195450 (414h24m51.160903848s) [noise=true , address=145.79.227.43:4157 , piece_id=EKE6ZQ6YMYBBKZKQ2OIAAF5OI3OZRPJF4XDYCCU7QSKS33UXPDZQ, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.334537739 (414h24m51.162246133s) [noise=true , address=145.79.227.39:4015 , piece_id=AZSOOT5RLGTAHSO47SVO27XDSNFPE65JU2I5UEZDPQ44PKB2YLYA, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.413477504 (414h24m51.2411859s) [noise=true , address=77.90.31.10:4517 , piece_id=UAIKGPQF75AM6EKYBV3XK45CNYS2N2PNHKBAKRKO6DDILYAUFBLA, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:53:20.161293124 (414h25m20.989001514s) [noise=true , address=77.90.31.18:4805 , piece_id=YWS2P6F2X4PRKE7FVEA722G4FGDV2D4BXJILLHGQE4Q2HXRU2SAA, seg_pos={0 0}, upload=431251]: Done uploading piece. err: ecclient: upload failed (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, address:77.90.31.18:4805): protocol: expected piece hash; context deadline exceeded; EOF Dec 22 21:53:20.161472578 (414h25m20.989180974s) [noise=true , address=145.79.227.38:4978 , piece_id=S26D5M7SYZEXTTNQEELKEFFQTOMNA5SG7RL5THCON6KCFY267IDQ, seg_pos={0 0}, upload=431251]: Done uploading piece. err: ecclient: upload failed (node:12PcGHVSm5UAXPh5vCfyRHCd7Bdb67oXF51AZcnRFUjXqVYzXLR, address:145.79.227.38:4978): protocol: expected piece hash; context deadline exceeded; EOF Dec 22 21:53:20.161591128 (414h25m20.989299529s) [noise=true , address=145.79.227.32:4834 , piece_id=KLWKSAGBRLFWDKNCHW6UWKCADPZ6D6UURDMNMD5UCZUUWAPDOKVQ, seg_pos={0 0}, upload=431251]: Done uploading piece. err: ecclient: upload failed (node:1gRnamgAa2ccZS6F44apRmzSkXC6jGvNreJ5J8Y9nciWPZCSsk, address:145.79.227.32:4834): protocol: expected piece hash; context deadline exceeded; EOF Dec 22 21:53:20.161797225 (414h25m20.989505624s) [noise=true , address=145.79.227.15:4876 , piece_id=X5KN53INJAXSFQUEY2S37RYA3ELNBGN6ZFM6KKJ3A4ROOSUTBTRQ, seg_pos={0 0}, upload=431251]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:145.79.227.15:4876): protocol: expected piece hash; context deadline exceeded; EOF Dec 22 21:53:20.161824434 (414h25m20.989532813s) [seg_pos={0 0}, upload=431251]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 22 21:53:20.183925866 (414h25m21.011634264s) [upload=431251]: Done uploading segments... Dec 22 21:53:20.184019830 (414h25m21.011728235s) [upload=431251]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 544.923733ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................X...................................] duration: 797.021425ms trace: https://jaeger.dev.storj.io/trace/17ef15eb0c5881c1 FAIL (2025-12-22 22:20:02 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 22 21:52:50.192390642 (414h24m51.020099033s) [upload=431251]: Starting upload Dec 22 21:52:50.192488257 (414h24m51.020196648s) [upload=431251]: Done starting upload Dec 22 21:52:50.192531865 (414h24m51.020240259s) [upload=431251]: Uploading segments... Dec 22 21:52:50.192783054 (414h24m51.020491446s) [upload=431251]: Got next segment. Inline: false Dec 22 21:52:50.192836048 (414h24m51.020544439s) [upload=431251]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 22 21:52:50.192841417 (414h24m51.020549808s) [upload=431251]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 22 21:52:50.220288008 (414h24m51.047996404s) [seg_pos={0 0}, upload=431251]: Begin upload segment... Dec 22 21:52:50.220342100 (414h24m51.048050497s) [seg_pos={0 0}, upload=431251]: Done begin upload segment. Dec 22 21:52:50.220384745 (414h24m51.048093145s) [noise=true , address=77.90.31.10:4517 , piece_id=UAIKGPQF75AM6EKYBV3XK45CNYS2N2PNHKBAKRKO6DDILYAUFBLA, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.220421097 (414h24m51.048129475s) [noise=true , address=145.79.227.45:4229 , piece_id=QYCGIYFFNWLGOBOKEZYY4EBLHKC3SFYWOBQJ34DETCLF6BTS54VQ, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.221068129 (414h24m51.048776527s) [noise=true , address=145.79.227.35:4483 , piece_id=EJIXZOBJWPIVL37GIYYPHQRZDMEMS3YSRPVJH4GLMB4A2CSSRCTQ, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.221771875 (414h24m51.049480268s) [noise=true , address=145.79.227.51:4411 , piece_id=O76ELDU3V3XZ2ZYC7VCLZC2CU5VUMRTFWJEV2ITS5GFAGRSXTOSA, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.222308205 (414h24m51.050016602s) [noise=true , address=145.79.227.39:4015 , piece_id=AZSOOT5RLGTAHSO47SVO27XDSNFPE65JU2I5UEZDPQ44PKB2YLYA, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.222424843 (414h24m51.050133244s) [noise=true , address=77.90.31.18:4805 , piece_id=YWS2P6F2X4PRKE7FVEA722G4FGDV2D4BXJILLHGQE4Q2HXRU2SAA, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.222721983 (414h24m51.050430374s) [noise=true , address=145.79.227.28:4727 , piece_id=Y5UCAOYDWNY5WPZC7LYUOT5QB325BSVZJUGSRVMR64ML23SEI4BQ, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.222910142 (414h24m51.050618542s) [noise=true , address=145.79.227.37:4566 , piece_id=EKCKTUJE37ZDSKNA6JM2ODSBLYAI2KFCA5RVFRNJPSCQ72ZWKY6A, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.223093651 (414h24m51.050802048s) [noise=true , address=145.79.227.50:4373 , piece_id=HESOK4IM7VRJNRL6NY5U4V52G7PDLSE3L3ZOF7DSTSQGFP6D2ZQA, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.223601176 (414h24m51.051309571s) [noise=true , address=145.79.227.15:4876 , piece_id=X5KN53INJAXSFQUEY2S37RYA3ELNBGN6ZFM6KKJ3A4ROOSUTBTRQ, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.224086995 (414h24m51.051795377s) [noise=true , address=145.79.227.43:4157 , piece_id=EKE6ZQ6YMYBBKZKQ2OIAAF5OI3OZRPJF4XDYCCU7QSKS33UXPDZQ, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.224569909 (414h24m51.052278304s) [noise=true , address=77.90.31.11:4553 , piece_id=MJFMBIGW3L6E75AAMZRCDONHZXPFLQSCK3VINMI4I2ZJC5HVHREA, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.225250077 (414h24m51.052958472s) [noise=true , address=145.79.227.48:4301 , piece_id=OCJPU5E7QLJWVY5K444J3J3ISMFY5E6MVXRMXBHIQRHXMIMRYC2A, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.226504914 (414h24m51.054213295s) [noise=true , address=145.79.227.47:4266 , piece_id=EJYUC5ODHWWQEA7QK5JVT3T3HQ76G5WC5OHPNGUSYMP4AW5BWLPA, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.227351850 (414h24m51.055060238s) [noise=true , address=145.79.227.38:4978 , piece_id=S26D5M7SYZEXTTNQEELKEFFQTOMNA5SG7RL5THCON6KCFY267IDQ, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.228140124 (414h24m51.055848511s) [noise=true , address=145.79.227.32:4834 , piece_id=KLWKSAGBRLFWDKNCHW6UWKCADPZ6D6UURDMNMD5UCZUUWAPDOKVQ, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.321476218 (414h24m51.149184612s) [upload=431251]: Next returned nil segment Dec 22 21:52:50.321492753 (414h24m51.149201151s) [upload=431251]: Waiting for error group managing segments... Dec 22 21:52:50.328186730 (414h24m51.155895125s) [noise=true , address=77.90.31.11:4553 , piece_id=MJFMBIGW3L6E75AAMZRCDONHZXPFLQSCK3VINMI4I2ZJC5HVHREA, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.328400062 (414h24m51.156108454s) [noise=true , address=145.79.227.47:4266 , piece_id=EJYUC5ODHWWQEA7QK5JVT3T3HQ76G5WC5OHPNGUSYMP4AW5BWLPA, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.328834800 (414h24m51.156543191s) [noise=true , address=145.79.227.37:4566 , piece_id=EKCKTUJE37ZDSKNA6JM2ODSBLYAI2KFCA5RVFRNJPSCQ72ZWKY6A, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.330369052 (414h24m51.158077447s) [noise=true , address=145.79.227.51:4411 , piece_id=O76ELDU3V3XZ2ZYC7VCLZC2CU5VUMRTFWJEV2ITS5GFAGRSXTOSA, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.330425878 (414h24m51.158134271s) [noise=true , address=145.79.227.28:4727 , piece_id=Y5UCAOYDWNY5WPZC7LYUOT5QB325BSVZJUGSRVMR64ML23SEI4BQ, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.331015687 (414h24m51.158724085s) [noise=true , address=145.79.227.45:4229 , piece_id=QYCGIYFFNWLGOBOKEZYY4EBLHKC3SFYWOBQJ34DETCLF6BTS54VQ, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.331368793 (414h24m51.159077188s) [noise=true , address=145.79.227.48:4301 , piece_id=OCJPU5E7QLJWVY5K444J3J3ISMFY5E6MVXRMXBHIQRHXMIMRYC2A, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.332233159 (414h24m51.159941554s) [noise=true , address=145.79.227.50:4373 , piece_id=HESOK4IM7VRJNRL6NY5U4V52G7PDLSE3L3ZOF7DSTSQGFP6D2ZQA, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.333040553 (414h24m51.160748944s) [noise=true , address=145.79.227.35:4483 , piece_id=EJIXZOBJWPIVL37GIYYPHQRZDMEMS3YSRPVJH4GLMB4A2CSSRCTQ, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.333195450 (414h24m51.160903848s) [noise=true , address=145.79.227.43:4157 , piece_id=EKE6ZQ6YMYBBKZKQ2OIAAF5OI3OZRPJF4XDYCCU7QSKS33UXPDZQ, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.334537739 (414h24m51.162246133s) [noise=true , address=145.79.227.39:4015 , piece_id=AZSOOT5RLGTAHSO47SVO27XDSNFPE65JU2I5UEZDPQ44PKB2YLYA, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.413477504 (414h24m51.2411859s) [noise=true , address=77.90.31.10:4517 , piece_id=UAIKGPQF75AM6EKYBV3XK45CNYS2N2PNHKBAKRKO6DDILYAUFBLA, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:53:20.161293124 (414h25m20.989001514s) [noise=true , address=77.90.31.18:4805 , piece_id=YWS2P6F2X4PRKE7FVEA722G4FGDV2D4BXJILLHGQE4Q2HXRU2SAA, seg_pos={0 0}, upload=431251]: Done uploading piece. err: ecclient: upload failed (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, address:77.90.31.18:4805): protocol: expected piece hash; context deadline exceeded; EOF Dec 22 21:53:20.161472578 (414h25m20.989180974s) [noise=true , address=145.79.227.38:4978 , piece_id=S26D5M7SYZEXTTNQEELKEFFQTOMNA5SG7RL5THCON6KCFY267IDQ, seg_pos={0 0}, upload=431251]: Done uploading piece. err: ecclient: upload failed (node:12PcGHVSm5UAXPh5vCfyRHCd7Bdb67oXF51AZcnRFUjXqVYzXLR, address:145.79.227.38:4978): protocol: expected piece hash; context deadline exceeded; EOF Dec 22 21:53:20.161591128 (414h25m20.989299529s) [noise=true , address=145.79.227.32:4834 , piece_id=KLWKSAGBRLFWDKNCHW6UWKCADPZ6D6UURDMNMD5UCZUUWAPDOKVQ, seg_pos={0 0}, upload=431251]: Done uploading piece. err: ecclient: upload failed (node:1gRnamgAa2ccZS6F44apRmzSkXC6jGvNreJ5J8Y9nciWPZCSsk, address:145.79.227.32:4834): protocol: expected piece hash; context deadline exceeded; EOF Dec 22 21:53:20.161797225 (414h25m20.989505624s) [noise=true , address=145.79.227.15:4876 , piece_id=X5KN53INJAXSFQUEY2S37RYA3ELNBGN6ZFM6KKJ3A4ROOSUTBTRQ, seg_pos={0 0}, upload=431251]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:145.79.227.15:4876): protocol: expected piece hash; context deadline exceeded; EOF Dec 22 21:53:20.161824434 (414h25m20.989532813s) [seg_pos={0 0}, upload=431251]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 22 21:53:20.183925866 (414h25m21.011634264s) [upload=431251]: Done uploading segments... Dec 22 21:53:20.184019830 (414h25m21.011728235s) [upload=431251]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 797.057313ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................X..................................] duration: 594.035073ms trace: https://jaeger.dev.storj.io/trace/17ef15eb0c5881c1 FAIL (2025-12-22 22:19:15 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 22 21:52:50.192390642 (414h24m51.020099033s) [upload=431251]: Starting upload Dec 22 21:52:50.192488257 (414h24m51.020196648s) [upload=431251]: Done starting upload Dec 22 21:52:50.192531865 (414h24m51.020240259s) [upload=431251]: Uploading segments... Dec 22 21:52:50.192783054 (414h24m51.020491446s) [upload=431251]: Got next segment. Inline: false Dec 22 21:52:50.192836048 (414h24m51.020544439s) [upload=431251]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 22 21:52:50.192841417 (414h24m51.020549808s) [upload=431251]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 22 21:52:50.220288008 (414h24m51.047996404s) [seg_pos={0 0}, upload=431251]: Begin upload segment... Dec 22 21:52:50.220342100 (414h24m51.048050497s) [seg_pos={0 0}, upload=431251]: Done begin upload segment. Dec 22 21:52:50.220384745 (414h24m51.048093145s) [noise=true , address=77.90.31.10:4517 , piece_id=UAIKGPQF75AM6EKYBV3XK45CNYS2N2PNHKBAKRKO6DDILYAUFBLA, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.220421097 (414h24m51.048129475s) [noise=true , address=145.79.227.45:4229 , piece_id=QYCGIYFFNWLGOBOKEZYY4EBLHKC3SFYWOBQJ34DETCLF6BTS54VQ, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.221068129 (414h24m51.048776527s) [noise=true , address=145.79.227.35:4483 , piece_id=EJIXZOBJWPIVL37GIYYPHQRZDMEMS3YSRPVJH4GLMB4A2CSSRCTQ, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.221771875 (414h24m51.049480268s) [noise=true , address=145.79.227.51:4411 , piece_id=O76ELDU3V3XZ2ZYC7VCLZC2CU5VUMRTFWJEV2ITS5GFAGRSXTOSA, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.222308205 (414h24m51.050016602s) [noise=true , address=145.79.227.39:4015 , piece_id=AZSOOT5RLGTAHSO47SVO27XDSNFPE65JU2I5UEZDPQ44PKB2YLYA, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.222424843 (414h24m51.050133244s) [noise=true , address=77.90.31.18:4805 , piece_id=YWS2P6F2X4PRKE7FVEA722G4FGDV2D4BXJILLHGQE4Q2HXRU2SAA, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.222721983 (414h24m51.050430374s) [noise=true , address=145.79.227.28:4727 , piece_id=Y5UCAOYDWNY5WPZC7LYUOT5QB325BSVZJUGSRVMR64ML23SEI4BQ, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.222910142 (414h24m51.050618542s) [noise=true , address=145.79.227.37:4566 , piece_id=EKCKTUJE37ZDSKNA6JM2ODSBLYAI2KFCA5RVFRNJPSCQ72ZWKY6A, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.223093651 (414h24m51.050802048s) [noise=true , address=145.79.227.50:4373 , piece_id=HESOK4IM7VRJNRL6NY5U4V52G7PDLSE3L3ZOF7DSTSQGFP6D2ZQA, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.223601176 (414h24m51.051309571s) [noise=true , address=145.79.227.15:4876 , piece_id=X5KN53INJAXSFQUEY2S37RYA3ELNBGN6ZFM6KKJ3A4ROOSUTBTRQ, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.224086995 (414h24m51.051795377s) [noise=true , address=145.79.227.43:4157 , piece_id=EKE6ZQ6YMYBBKZKQ2OIAAF5OI3OZRPJF4XDYCCU7QSKS33UXPDZQ, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.224569909 (414h24m51.052278304s) [noise=true , address=77.90.31.11:4553 , piece_id=MJFMBIGW3L6E75AAMZRCDONHZXPFLQSCK3VINMI4I2ZJC5HVHREA, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.225250077 (414h24m51.052958472s) [noise=true , address=145.79.227.48:4301 , piece_id=OCJPU5E7QLJWVY5K444J3J3ISMFY5E6MVXRMXBHIQRHXMIMRYC2A, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.226504914 (414h24m51.054213295s) [noise=true , address=145.79.227.47:4266 , piece_id=EJYUC5ODHWWQEA7QK5JVT3T3HQ76G5WC5OHPNGUSYMP4AW5BWLPA, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.227351850 (414h24m51.055060238s) [noise=true , address=145.79.227.38:4978 , piece_id=S26D5M7SYZEXTTNQEELKEFFQTOMNA5SG7RL5THCON6KCFY267IDQ, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.228140124 (414h24m51.055848511s) [noise=true , address=145.79.227.32:4834 , piece_id=KLWKSAGBRLFWDKNCHW6UWKCADPZ6D6UURDMNMD5UCZUUWAPDOKVQ, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.321476218 (414h24m51.149184612s) [upload=431251]: Next returned nil segment Dec 22 21:52:50.321492753 (414h24m51.149201151s) [upload=431251]: Waiting for error group managing segments... Dec 22 21:52:50.328186730 (414h24m51.155895125s) [noise=true , address=77.90.31.11:4553 , piece_id=MJFMBIGW3L6E75AAMZRCDONHZXPFLQSCK3VINMI4I2ZJC5HVHREA, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.328400062 (414h24m51.156108454s) [noise=true , address=145.79.227.47:4266 , piece_id=EJYUC5ODHWWQEA7QK5JVT3T3HQ76G5WC5OHPNGUSYMP4AW5BWLPA, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.328834800 (414h24m51.156543191s) [noise=true , address=145.79.227.37:4566 , piece_id=EKCKTUJE37ZDSKNA6JM2ODSBLYAI2KFCA5RVFRNJPSCQ72ZWKY6A, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.330369052 (414h24m51.158077447s) [noise=true , address=145.79.227.51:4411 , piece_id=O76ELDU3V3XZ2ZYC7VCLZC2CU5VUMRTFWJEV2ITS5GFAGRSXTOSA, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.330425878 (414h24m51.158134271s) [noise=true , address=145.79.227.28:4727 , piece_id=Y5UCAOYDWNY5WPZC7LYUOT5QB325BSVZJUGSRVMR64ML23SEI4BQ, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.331015687 (414h24m51.158724085s) [noise=true , address=145.79.227.45:4229 , piece_id=QYCGIYFFNWLGOBOKEZYY4EBLHKC3SFYWOBQJ34DETCLF6BTS54VQ, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.331368793 (414h24m51.159077188s) [noise=true , address=145.79.227.48:4301 , piece_id=OCJPU5E7QLJWVY5K444J3J3ISMFY5E6MVXRMXBHIQRHXMIMRYC2A, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.332233159 (414h24m51.159941554s) [noise=true , address=145.79.227.50:4373 , piece_id=HESOK4IM7VRJNRL6NY5U4V52G7PDLSE3L3ZOF7DSTSQGFP6D2ZQA, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.333040553 (414h24m51.160748944s) [noise=true , address=145.79.227.35:4483 , piece_id=EJIXZOBJWPIVL37GIYYPHQRZDMEMS3YSRPVJH4GLMB4A2CSSRCTQ, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.333195450 (414h24m51.160903848s) [noise=true , address=145.79.227.43:4157 , piece_id=EKE6ZQ6YMYBBKZKQ2OIAAF5OI3OZRPJF4XDYCCU7QSKS33UXPDZQ, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.334537739 (414h24m51.162246133s) [noise=true , address=145.79.227.39:4015 , piece_id=AZSOOT5RLGTAHSO47SVO27XDSNFPE65JU2I5UEZDPQ44PKB2YLYA, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.413477504 (414h24m51.2411859s) [noise=true , address=77.90.31.10:4517 , piece_id=UAIKGPQF75AM6EKYBV3XK45CNYS2N2PNHKBAKRKO6DDILYAUFBLA, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:53:20.161293124 (414h25m20.989001514s) [noise=true , address=77.90.31.18:4805 , piece_id=YWS2P6F2X4PRKE7FVEA722G4FGDV2D4BXJILLHGQE4Q2HXRU2SAA, seg_pos={0 0}, upload=431251]: Done uploading piece. err: ecclient: upload failed (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, address:77.90.31.18:4805): protocol: expected piece hash; context deadline exceeded; EOF Dec 22 21:53:20.161472578 (414h25m20.989180974s) [noise=true , address=145.79.227.38:4978 , piece_id=S26D5M7SYZEXTTNQEELKEFFQTOMNA5SG7RL5THCON6KCFY267IDQ, seg_pos={0 0}, upload=431251]: Done uploading piece. err: ecclient: upload failed (node:12PcGHVSm5UAXPh5vCfyRHCd7Bdb67oXF51AZcnRFUjXqVYzXLR, address:145.79.227.38:4978): protocol: expected piece hash; context deadline exceeded; EOF Dec 22 21:53:20.161591128 (414h25m20.989299529s) [noise=true , address=145.79.227.32:4834 , piece_id=KLWKSAGBRLFWDKNCHW6UWKCADPZ6D6UURDMNMD5UCZUUWAPDOKVQ, seg_pos={0 0}, upload=431251]: Done uploading piece. err: ecclient: upload failed (node:1gRnamgAa2ccZS6F44apRmzSkXC6jGvNreJ5J8Y9nciWPZCSsk, address:145.79.227.32:4834): protocol: expected piece hash; context deadline exceeded; EOF Dec 22 21:53:20.161797225 (414h25m20.989505624s) [noise=true , address=145.79.227.15:4876 , piece_id=X5KN53INJAXSFQUEY2S37RYA3ELNBGN6ZFM6KKJ3A4ROOSUTBTRQ, seg_pos={0 0}, upload=431251]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:145.79.227.15:4876): protocol: expected piece hash; context deadline exceeded; EOF Dec 22 21:53:20.161824434 (414h25m20.989532813s) [seg_pos={0 0}, upload=431251]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 22 21:53:20.183925866 (414h25m21.011634264s) [upload=431251]: Done uploading segments... Dec 22 21:53:20.184019830 (414h25m21.011728235s) [upload=431251]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 594.065258ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................X.................................] duration: 470.646843ms trace: https://jaeger.dev.storj.io/trace/17ef15eb0c5881c1 FAIL (2025-12-22 22:18:44 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 22 21:52:50.192390642 (414h24m51.020099033s) [upload=431251]: Starting upload Dec 22 21:52:50.192488257 (414h24m51.020196648s) [upload=431251]: Done starting upload Dec 22 21:52:50.192531865 (414h24m51.020240259s) [upload=431251]: Uploading segments... Dec 22 21:52:50.192783054 (414h24m51.020491446s) [upload=431251]: Got next segment. Inline: false Dec 22 21:52:50.192836048 (414h24m51.020544439s) [upload=431251]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 22 21:52:50.192841417 (414h24m51.020549808s) [upload=431251]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 22 21:52:50.220288008 (414h24m51.047996404s) [seg_pos={0 0}, upload=431251]: Begin upload segment... Dec 22 21:52:50.220342100 (414h24m51.048050497s) [seg_pos={0 0}, upload=431251]: Done begin upload segment. Dec 22 21:52:50.220384745 (414h24m51.048093145s) [noise=true , address=77.90.31.10:4517 , piece_id=UAIKGPQF75AM6EKYBV3XK45CNYS2N2PNHKBAKRKO6DDILYAUFBLA, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.220421097 (414h24m51.048129475s) [noise=true , address=145.79.227.45:4229 , piece_id=QYCGIYFFNWLGOBOKEZYY4EBLHKC3SFYWOBQJ34DETCLF6BTS54VQ, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.221068129 (414h24m51.048776527s) [noise=true , address=145.79.227.35:4483 , piece_id=EJIXZOBJWPIVL37GIYYPHQRZDMEMS3YSRPVJH4GLMB4A2CSSRCTQ, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.221771875 (414h24m51.049480268s) [noise=true , address=145.79.227.51:4411 , piece_id=O76ELDU3V3XZ2ZYC7VCLZC2CU5VUMRTFWJEV2ITS5GFAGRSXTOSA, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.222308205 (414h24m51.050016602s) [noise=true , address=145.79.227.39:4015 , piece_id=AZSOOT5RLGTAHSO47SVO27XDSNFPE65JU2I5UEZDPQ44PKB2YLYA, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.222424843 (414h24m51.050133244s) [noise=true , address=77.90.31.18:4805 , piece_id=YWS2P6F2X4PRKE7FVEA722G4FGDV2D4BXJILLHGQE4Q2HXRU2SAA, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.222721983 (414h24m51.050430374s) [noise=true , address=145.79.227.28:4727 , piece_id=Y5UCAOYDWNY5WPZC7LYUOT5QB325BSVZJUGSRVMR64ML23SEI4BQ, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.222910142 (414h24m51.050618542s) [noise=true , address=145.79.227.37:4566 , piece_id=EKCKTUJE37ZDSKNA6JM2ODSBLYAI2KFCA5RVFRNJPSCQ72ZWKY6A, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.223093651 (414h24m51.050802048s) [noise=true , address=145.79.227.50:4373 , piece_id=HESOK4IM7VRJNRL6NY5U4V52G7PDLSE3L3ZOF7DSTSQGFP6D2ZQA, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.223601176 (414h24m51.051309571s) [noise=true , address=145.79.227.15:4876 , piece_id=X5KN53INJAXSFQUEY2S37RYA3ELNBGN6ZFM6KKJ3A4ROOSUTBTRQ, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.224086995 (414h24m51.051795377s) [noise=true , address=145.79.227.43:4157 , piece_id=EKE6ZQ6YMYBBKZKQ2OIAAF5OI3OZRPJF4XDYCCU7QSKS33UXPDZQ, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.224569909 (414h24m51.052278304s) [noise=true , address=77.90.31.11:4553 , piece_id=MJFMBIGW3L6E75AAMZRCDONHZXPFLQSCK3VINMI4I2ZJC5HVHREA, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.225250077 (414h24m51.052958472s) [noise=true , address=145.79.227.48:4301 , piece_id=OCJPU5E7QLJWVY5K444J3J3ISMFY5E6MVXRMXBHIQRHXMIMRYC2A, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.226504914 (414h24m51.054213295s) [noise=true , address=145.79.227.47:4266 , piece_id=EJYUC5ODHWWQEA7QK5JVT3T3HQ76G5WC5OHPNGUSYMP4AW5BWLPA, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.227351850 (414h24m51.055060238s) [noise=true , address=145.79.227.38:4978 , piece_id=S26D5M7SYZEXTTNQEELKEFFQTOMNA5SG7RL5THCON6KCFY267IDQ, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.228140124 (414h24m51.055848511s) [noise=true , address=145.79.227.32:4834 , piece_id=KLWKSAGBRLFWDKNCHW6UWKCADPZ6D6UURDMNMD5UCZUUWAPDOKVQ, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.321476218 (414h24m51.149184612s) [upload=431251]: Next returned nil segment Dec 22 21:52:50.321492753 (414h24m51.149201151s) [upload=431251]: Waiting for error group managing segments... Dec 22 21:52:50.328186730 (414h24m51.155895125s) [noise=true , address=77.90.31.11:4553 , piece_id=MJFMBIGW3L6E75AAMZRCDONHZXPFLQSCK3VINMI4I2ZJC5HVHREA, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.328400062 (414h24m51.156108454s) [noise=true , address=145.79.227.47:4266 , piece_id=EJYUC5ODHWWQEA7QK5JVT3T3HQ76G5WC5OHPNGUSYMP4AW5BWLPA, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.328834800 (414h24m51.156543191s) [noise=true , address=145.79.227.37:4566 , piece_id=EKCKTUJE37ZDSKNA6JM2ODSBLYAI2KFCA5RVFRNJPSCQ72ZWKY6A, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.330369052 (414h24m51.158077447s) [noise=true , address=145.79.227.51:4411 , piece_id=O76ELDU3V3XZ2ZYC7VCLZC2CU5VUMRTFWJEV2ITS5GFAGRSXTOSA, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.330425878 (414h24m51.158134271s) [noise=true , address=145.79.227.28:4727 , piece_id=Y5UCAOYDWNY5WPZC7LYUOT5QB325BSVZJUGSRVMR64ML23SEI4BQ, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.331015687 (414h24m51.158724085s) [noise=true , address=145.79.227.45:4229 , piece_id=QYCGIYFFNWLGOBOKEZYY4EBLHKC3SFYWOBQJ34DETCLF6BTS54VQ, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.331368793 (414h24m51.159077188s) [noise=true , address=145.79.227.48:4301 , piece_id=OCJPU5E7QLJWVY5K444J3J3ISMFY5E6MVXRMXBHIQRHXMIMRYC2A, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.332233159 (414h24m51.159941554s) [noise=true , address=145.79.227.50:4373 , piece_id=HESOK4IM7VRJNRL6NY5U4V52G7PDLSE3L3ZOF7DSTSQGFP6D2ZQA, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.333040553 (414h24m51.160748944s) [noise=true , address=145.79.227.35:4483 , piece_id=EJIXZOBJWPIVL37GIYYPHQRZDMEMS3YSRPVJH4GLMB4A2CSSRCTQ, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.333195450 (414h24m51.160903848s) [noise=true , address=145.79.227.43:4157 , piece_id=EKE6ZQ6YMYBBKZKQ2OIAAF5OI3OZRPJF4XDYCCU7QSKS33UXPDZQ, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.334537739 (414h24m51.162246133s) [noise=true , address=145.79.227.39:4015 , piece_id=AZSOOT5RLGTAHSO47SVO27XDSNFPE65JU2I5UEZDPQ44PKB2YLYA, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.413477504 (414h24m51.2411859s) [noise=true , address=77.90.31.10:4517 , piece_id=UAIKGPQF75AM6EKYBV3XK45CNYS2N2PNHKBAKRKO6DDILYAUFBLA, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:53:20.161293124 (414h25m20.989001514s) [noise=true , address=77.90.31.18:4805 , piece_id=YWS2P6F2X4PRKE7FVEA722G4FGDV2D4BXJILLHGQE4Q2HXRU2SAA, seg_pos={0 0}, upload=431251]: Done uploading piece. err: ecclient: upload failed (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, address:77.90.31.18:4805): protocol: expected piece hash; context deadline exceeded; EOF Dec 22 21:53:20.161472578 (414h25m20.989180974s) [noise=true , address=145.79.227.38:4978 , piece_id=S26D5M7SYZEXTTNQEELKEFFQTOMNA5SG7RL5THCON6KCFY267IDQ, seg_pos={0 0}, upload=431251]: Done uploading piece. err: ecclient: upload failed (node:12PcGHVSm5UAXPh5vCfyRHCd7Bdb67oXF51AZcnRFUjXqVYzXLR, address:145.79.227.38:4978): protocol: expected piece hash; context deadline exceeded; EOF Dec 22 21:53:20.161591128 (414h25m20.989299529s) [noise=true , address=145.79.227.32:4834 , piece_id=KLWKSAGBRLFWDKNCHW6UWKCADPZ6D6UURDMNMD5UCZUUWAPDOKVQ, seg_pos={0 0}, upload=431251]: Done uploading piece. err: ecclient: upload failed (node:1gRnamgAa2ccZS6F44apRmzSkXC6jGvNreJ5J8Y9nciWPZCSsk, address:145.79.227.32:4834): protocol: expected piece hash; context deadline exceeded; EOF Dec 22 21:53:20.161797225 (414h25m20.989505624s) [noise=true , address=145.79.227.15:4876 , piece_id=X5KN53INJAXSFQUEY2S37RYA3ELNBGN6ZFM6KKJ3A4ROOSUTBTRQ, seg_pos={0 0}, upload=431251]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:145.79.227.15:4876): protocol: expected piece hash; context deadline exceeded; EOF Dec 22 21:53:20.161824434 (414h25m20.989532813s) [seg_pos={0 0}, upload=431251]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 22 21:53:20.183925866 (414h25m21.011634264s) [upload=431251]: Done uploading segments... Dec 22 21:53:20.184019830 (414h25m21.011728235s) [upload=431251]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 470.679589ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................X................................] duration: 497.172147ms trace: https://jaeger.dev.storj.io/trace/17ef15eb0c5881c1 FAIL (2025-12-22 22:18:14 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 22 21:52:50.192390642 (414h24m51.020099033s) [upload=431251]: Starting upload Dec 22 21:52:50.192488257 (414h24m51.020196648s) [upload=431251]: Done starting upload Dec 22 21:52:50.192531865 (414h24m51.020240259s) [upload=431251]: Uploading segments... Dec 22 21:52:50.192783054 (414h24m51.020491446s) [upload=431251]: Got next segment. Inline: false Dec 22 21:52:50.192836048 (414h24m51.020544439s) [upload=431251]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 22 21:52:50.192841417 (414h24m51.020549808s) [upload=431251]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 22 21:52:50.220288008 (414h24m51.047996404s) [seg_pos={0 0}, upload=431251]: Begin upload segment... Dec 22 21:52:50.220342100 (414h24m51.048050497s) [seg_pos={0 0}, upload=431251]: Done begin upload segment. Dec 22 21:52:50.220384745 (414h24m51.048093145s) [noise=true , address=77.90.31.10:4517 , piece_id=UAIKGPQF75AM6EKYBV3XK45CNYS2N2PNHKBAKRKO6DDILYAUFBLA, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.220421097 (414h24m51.048129475s) [noise=true , address=145.79.227.45:4229 , piece_id=QYCGIYFFNWLGOBOKEZYY4EBLHKC3SFYWOBQJ34DETCLF6BTS54VQ, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.221068129 (414h24m51.048776527s) [noise=true , address=145.79.227.35:4483 , piece_id=EJIXZOBJWPIVL37GIYYPHQRZDMEMS3YSRPVJH4GLMB4A2CSSRCTQ, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.221771875 (414h24m51.049480268s) [noise=true , address=145.79.227.51:4411 , piece_id=O76ELDU3V3XZ2ZYC7VCLZC2CU5VUMRTFWJEV2ITS5GFAGRSXTOSA, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.222308205 (414h24m51.050016602s) [noise=true , address=145.79.227.39:4015 , piece_id=AZSOOT5RLGTAHSO47SVO27XDSNFPE65JU2I5UEZDPQ44PKB2YLYA, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.222424843 (414h24m51.050133244s) [noise=true , address=77.90.31.18:4805 , piece_id=YWS2P6F2X4PRKE7FVEA722G4FGDV2D4BXJILLHGQE4Q2HXRU2SAA, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.222721983 (414h24m51.050430374s) [noise=true , address=145.79.227.28:4727 , piece_id=Y5UCAOYDWNY5WPZC7LYUOT5QB325BSVZJUGSRVMR64ML23SEI4BQ, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.222910142 (414h24m51.050618542s) [noise=true , address=145.79.227.37:4566 , piece_id=EKCKTUJE37ZDSKNA6JM2ODSBLYAI2KFCA5RVFRNJPSCQ72ZWKY6A, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.223093651 (414h24m51.050802048s) [noise=true , address=145.79.227.50:4373 , piece_id=HESOK4IM7VRJNRL6NY5U4V52G7PDLSE3L3ZOF7DSTSQGFP6D2ZQA, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.223601176 (414h24m51.051309571s) [noise=true , address=145.79.227.15:4876 , piece_id=X5KN53INJAXSFQUEY2S37RYA3ELNBGN6ZFM6KKJ3A4ROOSUTBTRQ, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.224086995 (414h24m51.051795377s) [noise=true , address=145.79.227.43:4157 , piece_id=EKE6ZQ6YMYBBKZKQ2OIAAF5OI3OZRPJF4XDYCCU7QSKS33UXPDZQ, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.224569909 (414h24m51.052278304s) [noise=true , address=77.90.31.11:4553 , piece_id=MJFMBIGW3L6E75AAMZRCDONHZXPFLQSCK3VINMI4I2ZJC5HVHREA, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.225250077 (414h24m51.052958472s) [noise=true , address=145.79.227.48:4301 , piece_id=OCJPU5E7QLJWVY5K444J3J3ISMFY5E6MVXRMXBHIQRHXMIMRYC2A, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.226504914 (414h24m51.054213295s) [noise=true , address=145.79.227.47:4266 , piece_id=EJYUC5ODHWWQEA7QK5JVT3T3HQ76G5WC5OHPNGUSYMP4AW5BWLPA, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.227351850 (414h24m51.055060238s) [noise=true , address=145.79.227.38:4978 , piece_id=S26D5M7SYZEXTTNQEELKEFFQTOMNA5SG7RL5THCON6KCFY267IDQ, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.228140124 (414h24m51.055848511s) [noise=true , address=145.79.227.32:4834 , piece_id=KLWKSAGBRLFWDKNCHW6UWKCADPZ6D6UURDMNMD5UCZUUWAPDOKVQ, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.321476218 (414h24m51.149184612s) [upload=431251]: Next returned nil segment Dec 22 21:52:50.321492753 (414h24m51.149201151s) [upload=431251]: Waiting for error group managing segments... Dec 22 21:52:50.328186730 (414h24m51.155895125s) [noise=true , address=77.90.31.11:4553 , piece_id=MJFMBIGW3L6E75AAMZRCDONHZXPFLQSCK3VINMI4I2ZJC5HVHREA, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.328400062 (414h24m51.156108454s) [noise=true , address=145.79.227.47:4266 , piece_id=EJYUC5ODHWWQEA7QK5JVT3T3HQ76G5WC5OHPNGUSYMP4AW5BWLPA, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.328834800 (414h24m51.156543191s) [noise=true , address=145.79.227.37:4566 , piece_id=EKCKTUJE37ZDSKNA6JM2ODSBLYAI2KFCA5RVFRNJPSCQ72ZWKY6A, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.330369052 (414h24m51.158077447s) [noise=true , address=145.79.227.51:4411 , piece_id=O76ELDU3V3XZ2ZYC7VCLZC2CU5VUMRTFWJEV2ITS5GFAGRSXTOSA, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.330425878 (414h24m51.158134271s) [noise=true , address=145.79.227.28:4727 , piece_id=Y5UCAOYDWNY5WPZC7LYUOT5QB325BSVZJUGSRVMR64ML23SEI4BQ, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.331015687 (414h24m51.158724085s) [noise=true , address=145.79.227.45:4229 , piece_id=QYCGIYFFNWLGOBOKEZYY4EBLHKC3SFYWOBQJ34DETCLF6BTS54VQ, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.331368793 (414h24m51.159077188s) [noise=true , address=145.79.227.48:4301 , piece_id=OCJPU5E7QLJWVY5K444J3J3ISMFY5E6MVXRMXBHIQRHXMIMRYC2A, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.332233159 (414h24m51.159941554s) [noise=true , address=145.79.227.50:4373 , piece_id=HESOK4IM7VRJNRL6NY5U4V52G7PDLSE3L3ZOF7DSTSQGFP6D2ZQA, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.333040553 (414h24m51.160748944s) [noise=true , address=145.79.227.35:4483 , piece_id=EJIXZOBJWPIVL37GIYYPHQRZDMEMS3YSRPVJH4GLMB4A2CSSRCTQ, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.333195450 (414h24m51.160903848s) [noise=true , address=145.79.227.43:4157 , piece_id=EKE6ZQ6YMYBBKZKQ2OIAAF5OI3OZRPJF4XDYCCU7QSKS33UXPDZQ, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.334537739 (414h24m51.162246133s) [noise=true , address=145.79.227.39:4015 , piece_id=AZSOOT5RLGTAHSO47SVO27XDSNFPE65JU2I5UEZDPQ44PKB2YLYA, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.413477504 (414h24m51.2411859s) [noise=true , address=77.90.31.10:4517 , piece_id=UAIKGPQF75AM6EKYBV3XK45CNYS2N2PNHKBAKRKO6DDILYAUFBLA, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:53:20.161293124 (414h25m20.989001514s) [noise=true , address=77.90.31.18:4805 , piece_id=YWS2P6F2X4PRKE7FVEA722G4FGDV2D4BXJILLHGQE4Q2HXRU2SAA, seg_pos={0 0}, upload=431251]: Done uploading piece. err: ecclient: upload failed (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, address:77.90.31.18:4805): protocol: expected piece hash; context deadline exceeded; EOF Dec 22 21:53:20.161472578 (414h25m20.989180974s) [noise=true , address=145.79.227.38:4978 , piece_id=S26D5M7SYZEXTTNQEELKEFFQTOMNA5SG7RL5THCON6KCFY267IDQ, seg_pos={0 0}, upload=431251]: Done uploading piece. err: ecclient: upload failed (node:12PcGHVSm5UAXPh5vCfyRHCd7Bdb67oXF51AZcnRFUjXqVYzXLR, address:145.79.227.38:4978): protocol: expected piece hash; context deadline exceeded; EOF Dec 22 21:53:20.161591128 (414h25m20.989299529s) [noise=true , address=145.79.227.32:4834 , piece_id=KLWKSAGBRLFWDKNCHW6UWKCADPZ6D6UURDMNMD5UCZUUWAPDOKVQ, seg_pos={0 0}, upload=431251]: Done uploading piece. err: ecclient: upload failed (node:1gRnamgAa2ccZS6F44apRmzSkXC6jGvNreJ5J8Y9nciWPZCSsk, address:145.79.227.32:4834): protocol: expected piece hash; context deadline exceeded; EOF Dec 22 21:53:20.161797225 (414h25m20.989505624s) [noise=true , address=145.79.227.15:4876 , piece_id=X5KN53INJAXSFQUEY2S37RYA3ELNBGN6ZFM6KKJ3A4ROOSUTBTRQ, seg_pos={0 0}, upload=431251]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:145.79.227.15:4876): protocol: expected piece hash; context deadline exceeded; EOF Dec 22 21:53:20.161824434 (414h25m20.989532813s) [seg_pos={0 0}, upload=431251]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 22 21:53:20.183925866 (414h25m21.011634264s) [upload=431251]: Done uploading segments... Dec 22 21:53:20.184019830 (414h25m21.011728235s) [upload=431251]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 497.281856ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............................X...............................] duration: 497.291222ms trace: https://jaeger.dev.storj.io/trace/17ef15eb0c5881c1 FAIL (2025-12-22 22:17:45 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 22 21:52:50.192390642 (414h24m51.020099033s) [upload=431251]: Starting upload Dec 22 21:52:50.192488257 (414h24m51.020196648s) [upload=431251]: Done starting upload Dec 22 21:52:50.192531865 (414h24m51.020240259s) [upload=431251]: Uploading segments... Dec 22 21:52:50.192783054 (414h24m51.020491446s) [upload=431251]: Got next segment. Inline: false Dec 22 21:52:50.192836048 (414h24m51.020544439s) [upload=431251]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 22 21:52:50.192841417 (414h24m51.020549808s) [upload=431251]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 22 21:52:50.220288008 (414h24m51.047996404s) [seg_pos={0 0}, upload=431251]: Begin upload segment... Dec 22 21:52:50.220342100 (414h24m51.048050497s) [seg_pos={0 0}, upload=431251]: Done begin upload segment. Dec 22 21:52:50.220384745 (414h24m51.048093145s) [noise=true , address=77.90.31.10:4517 , piece_id=UAIKGPQF75AM6EKYBV3XK45CNYS2N2PNHKBAKRKO6DDILYAUFBLA, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.220421097 (414h24m51.048129475s) [noise=true , address=145.79.227.45:4229 , piece_id=QYCGIYFFNWLGOBOKEZYY4EBLHKC3SFYWOBQJ34DETCLF6BTS54VQ, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.221068129 (414h24m51.048776527s) [noise=true , address=145.79.227.35:4483 , piece_id=EJIXZOBJWPIVL37GIYYPHQRZDMEMS3YSRPVJH4GLMB4A2CSSRCTQ, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.221771875 (414h24m51.049480268s) [noise=true , address=145.79.227.51:4411 , piece_id=O76ELDU3V3XZ2ZYC7VCLZC2CU5VUMRTFWJEV2ITS5GFAGRSXTOSA, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.222308205 (414h24m51.050016602s) [noise=true , address=145.79.227.39:4015 , piece_id=AZSOOT5RLGTAHSO47SVO27XDSNFPE65JU2I5UEZDPQ44PKB2YLYA, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.222424843 (414h24m51.050133244s) [noise=true , address=77.90.31.18:4805 , piece_id=YWS2P6F2X4PRKE7FVEA722G4FGDV2D4BXJILLHGQE4Q2HXRU2SAA, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.222721983 (414h24m51.050430374s) [noise=true , address=145.79.227.28:4727 , piece_id=Y5UCAOYDWNY5WPZC7LYUOT5QB325BSVZJUGSRVMR64ML23SEI4BQ, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.222910142 (414h24m51.050618542s) [noise=true , address=145.79.227.37:4566 , piece_id=EKCKTUJE37ZDSKNA6JM2ODSBLYAI2KFCA5RVFRNJPSCQ72ZWKY6A, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.223093651 (414h24m51.050802048s) [noise=true , address=145.79.227.50:4373 , piece_id=HESOK4IM7VRJNRL6NY5U4V52G7PDLSE3L3ZOF7DSTSQGFP6D2ZQA, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.223601176 (414h24m51.051309571s) [noise=true , address=145.79.227.15:4876 , piece_id=X5KN53INJAXSFQUEY2S37RYA3ELNBGN6ZFM6KKJ3A4ROOSUTBTRQ, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.224086995 (414h24m51.051795377s) [noise=true , address=145.79.227.43:4157 , piece_id=EKE6ZQ6YMYBBKZKQ2OIAAF5OI3OZRPJF4XDYCCU7QSKS33UXPDZQ, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.224569909 (414h24m51.052278304s) [noise=true , address=77.90.31.11:4553 , piece_id=MJFMBIGW3L6E75AAMZRCDONHZXPFLQSCK3VINMI4I2ZJC5HVHREA, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.225250077 (414h24m51.052958472s) [noise=true , address=145.79.227.48:4301 , piece_id=OCJPU5E7QLJWVY5K444J3J3ISMFY5E6MVXRMXBHIQRHXMIMRYC2A, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.226504914 (414h24m51.054213295s) [noise=true , address=145.79.227.47:4266 , piece_id=EJYUC5ODHWWQEA7QK5JVT3T3HQ76G5WC5OHPNGUSYMP4AW5BWLPA, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.227351850 (414h24m51.055060238s) [noise=true , address=145.79.227.38:4978 , piece_id=S26D5M7SYZEXTTNQEELKEFFQTOMNA5SG7RL5THCON6KCFY267IDQ, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.228140124 (414h24m51.055848511s) [noise=true , address=145.79.227.32:4834 , piece_id=KLWKSAGBRLFWDKNCHW6UWKCADPZ6D6UURDMNMD5UCZUUWAPDOKVQ, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.321476218 (414h24m51.149184612s) [upload=431251]: Next returned nil segment Dec 22 21:52:50.321492753 (414h24m51.149201151s) [upload=431251]: Waiting for error group managing segments... Dec 22 21:52:50.328186730 (414h24m51.155895125s) [noise=true , address=77.90.31.11:4553 , piece_id=MJFMBIGW3L6E75AAMZRCDONHZXPFLQSCK3VINMI4I2ZJC5HVHREA, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.328400062 (414h24m51.156108454s) [noise=true , address=145.79.227.47:4266 , piece_id=EJYUC5ODHWWQEA7QK5JVT3T3HQ76G5WC5OHPNGUSYMP4AW5BWLPA, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.328834800 (414h24m51.156543191s) [noise=true , address=145.79.227.37:4566 , piece_id=EKCKTUJE37ZDSKNA6JM2ODSBLYAI2KFCA5RVFRNJPSCQ72ZWKY6A, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.330369052 (414h24m51.158077447s) [noise=true , address=145.79.227.51:4411 , piece_id=O76ELDU3V3XZ2ZYC7VCLZC2CU5VUMRTFWJEV2ITS5GFAGRSXTOSA, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.330425878 (414h24m51.158134271s) [noise=true , address=145.79.227.28:4727 , piece_id=Y5UCAOYDWNY5WPZC7LYUOT5QB325BSVZJUGSRVMR64ML23SEI4BQ, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.331015687 (414h24m51.158724085s) [noise=true , address=145.79.227.45:4229 , piece_id=QYCGIYFFNWLGOBOKEZYY4EBLHKC3SFYWOBQJ34DETCLF6BTS54VQ, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.331368793 (414h24m51.159077188s) [noise=true , address=145.79.227.48:4301 , piece_id=OCJPU5E7QLJWVY5K444J3J3ISMFY5E6MVXRMXBHIQRHXMIMRYC2A, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.332233159 (414h24m51.159941554s) [noise=true , address=145.79.227.50:4373 , piece_id=HESOK4IM7VRJNRL6NY5U4V52G7PDLSE3L3ZOF7DSTSQGFP6D2ZQA, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.333040553 (414h24m51.160748944s) [noise=true , address=145.79.227.35:4483 , piece_id=EJIXZOBJWPIVL37GIYYPHQRZDMEMS3YSRPVJH4GLMB4A2CSSRCTQ, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.333195450 (414h24m51.160903848s) [noise=true , address=145.79.227.43:4157 , piece_id=EKE6ZQ6YMYBBKZKQ2OIAAF5OI3OZRPJF4XDYCCU7QSKS33UXPDZQ, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.334537739 (414h24m51.162246133s) [noise=true , address=145.79.227.39:4015 , piece_id=AZSOOT5RLGTAHSO47SVO27XDSNFPE65JU2I5UEZDPQ44PKB2YLYA, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.413477504 (414h24m51.2411859s) [noise=true , address=77.90.31.10:4517 , piece_id=UAIKGPQF75AM6EKYBV3XK45CNYS2N2PNHKBAKRKO6DDILYAUFBLA, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:53:20.161293124 (414h25m20.989001514s) [noise=true , address=77.90.31.18:4805 , piece_id=YWS2P6F2X4PRKE7FVEA722G4FGDV2D4BXJILLHGQE4Q2HXRU2SAA, seg_pos={0 0}, upload=431251]: Done uploading piece. err: ecclient: upload failed (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, address:77.90.31.18:4805): protocol: expected piece hash; context deadline exceeded; EOF Dec 22 21:53:20.161472578 (414h25m20.989180974s) [noise=true , address=145.79.227.38:4978 , piece_id=S26D5M7SYZEXTTNQEELKEFFQTOMNA5SG7RL5THCON6KCFY267IDQ, seg_pos={0 0}, upload=431251]: Done uploading piece. err: ecclient: upload failed (node:12PcGHVSm5UAXPh5vCfyRHCd7Bdb67oXF51AZcnRFUjXqVYzXLR, address:145.79.227.38:4978): protocol: expected piece hash; context deadline exceeded; EOF Dec 22 21:53:20.161591128 (414h25m20.989299529s) [noise=true , address=145.79.227.32:4834 , piece_id=KLWKSAGBRLFWDKNCHW6UWKCADPZ6D6UURDMNMD5UCZUUWAPDOKVQ, seg_pos={0 0}, upload=431251]: Done uploading piece. err: ecclient: upload failed (node:1gRnamgAa2ccZS6F44apRmzSkXC6jGvNreJ5J8Y9nciWPZCSsk, address:145.79.227.32:4834): protocol: expected piece hash; context deadline exceeded; EOF Dec 22 21:53:20.161797225 (414h25m20.989505624s) [noise=true , address=145.79.227.15:4876 , piece_id=X5KN53INJAXSFQUEY2S37RYA3ELNBGN6ZFM6KKJ3A4ROOSUTBTRQ, seg_pos={0 0}, upload=431251]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:145.79.227.15:4876): protocol: expected piece hash; context deadline exceeded; EOF Dec 22 21:53:20.161824434 (414h25m20.989532813s) [seg_pos={0 0}, upload=431251]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 22 21:53:20.183925866 (414h25m21.011634264s) [upload=431251]: Done uploading segments... Dec 22 21:53:20.184019830 (414h25m21.011728235s) [upload=431251]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 497.316624ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............................X..............................] duration: 442.276538ms trace: https://jaeger.dev.storj.io/trace/17ef15eb0c5881c1 FAIL (2025-12-22 22:16:58 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 22 21:52:50.192390642 (414h24m51.020099033s) [upload=431251]: Starting upload Dec 22 21:52:50.192488257 (414h24m51.020196648s) [upload=431251]: Done starting upload Dec 22 21:52:50.192531865 (414h24m51.020240259s) [upload=431251]: Uploading segments... Dec 22 21:52:50.192783054 (414h24m51.020491446s) [upload=431251]: Got next segment. Inline: false Dec 22 21:52:50.192836048 (414h24m51.020544439s) [upload=431251]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 22 21:52:50.192841417 (414h24m51.020549808s) [upload=431251]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 22 21:52:50.220288008 (414h24m51.047996404s) [seg_pos={0 0}, upload=431251]: Begin upload segment... Dec 22 21:52:50.220342100 (414h24m51.048050497s) [seg_pos={0 0}, upload=431251]: Done begin upload segment. Dec 22 21:52:50.220384745 (414h24m51.048093145s) [noise=true , address=77.90.31.10:4517 , piece_id=UAIKGPQF75AM6EKYBV3XK45CNYS2N2PNHKBAKRKO6DDILYAUFBLA, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.220421097 (414h24m51.048129475s) [noise=true , address=145.79.227.45:4229 , piece_id=QYCGIYFFNWLGOBOKEZYY4EBLHKC3SFYWOBQJ34DETCLF6BTS54VQ, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.221068129 (414h24m51.048776527s) [noise=true , address=145.79.227.35:4483 , piece_id=EJIXZOBJWPIVL37GIYYPHQRZDMEMS3YSRPVJH4GLMB4A2CSSRCTQ, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.221771875 (414h24m51.049480268s) [noise=true , address=145.79.227.51:4411 , piece_id=O76ELDU3V3XZ2ZYC7VCLZC2CU5VUMRTFWJEV2ITS5GFAGRSXTOSA, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.222308205 (414h24m51.050016602s) [noise=true , address=145.79.227.39:4015 , piece_id=AZSOOT5RLGTAHSO47SVO27XDSNFPE65JU2I5UEZDPQ44PKB2YLYA, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.222424843 (414h24m51.050133244s) [noise=true , address=77.90.31.18:4805 , piece_id=YWS2P6F2X4PRKE7FVEA722G4FGDV2D4BXJILLHGQE4Q2HXRU2SAA, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.222721983 (414h24m51.050430374s) [noise=true , address=145.79.227.28:4727 , piece_id=Y5UCAOYDWNY5WPZC7LYUOT5QB325BSVZJUGSRVMR64ML23SEI4BQ, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.222910142 (414h24m51.050618542s) [noise=true , address=145.79.227.37:4566 , piece_id=EKCKTUJE37ZDSKNA6JM2ODSBLYAI2KFCA5RVFRNJPSCQ72ZWKY6A, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.223093651 (414h24m51.050802048s) [noise=true , address=145.79.227.50:4373 , piece_id=HESOK4IM7VRJNRL6NY5U4V52G7PDLSE3L3ZOF7DSTSQGFP6D2ZQA, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.223601176 (414h24m51.051309571s) [noise=true , address=145.79.227.15:4876 , piece_id=X5KN53INJAXSFQUEY2S37RYA3ELNBGN6ZFM6KKJ3A4ROOSUTBTRQ, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.224086995 (414h24m51.051795377s) [noise=true , address=145.79.227.43:4157 , piece_id=EKE6ZQ6YMYBBKZKQ2OIAAF5OI3OZRPJF4XDYCCU7QSKS33UXPDZQ, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.224569909 (414h24m51.052278304s) [noise=true , address=77.90.31.11:4553 , piece_id=MJFMBIGW3L6E75AAMZRCDONHZXPFLQSCK3VINMI4I2ZJC5HVHREA, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.225250077 (414h24m51.052958472s) [noise=true , address=145.79.227.48:4301 , piece_id=OCJPU5E7QLJWVY5K444J3J3ISMFY5E6MVXRMXBHIQRHXMIMRYC2A, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.226504914 (414h24m51.054213295s) [noise=true , address=145.79.227.47:4266 , piece_id=EJYUC5ODHWWQEA7QK5JVT3T3HQ76G5WC5OHPNGUSYMP4AW5BWLPA, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.227351850 (414h24m51.055060238s) [noise=true , address=145.79.227.38:4978 , piece_id=S26D5M7SYZEXTTNQEELKEFFQTOMNA5SG7RL5THCON6KCFY267IDQ, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.228140124 (414h24m51.055848511s) [noise=true , address=145.79.227.32:4834 , piece_id=KLWKSAGBRLFWDKNCHW6UWKCADPZ6D6UURDMNMD5UCZUUWAPDOKVQ, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.321476218 (414h24m51.149184612s) [upload=431251]: Next returned nil segment Dec 22 21:52:50.321492753 (414h24m51.149201151s) [upload=431251]: Waiting for error group managing segments... Dec 22 21:52:50.328186730 (414h24m51.155895125s) [noise=true , address=77.90.31.11:4553 , piece_id=MJFMBIGW3L6E75AAMZRCDONHZXPFLQSCK3VINMI4I2ZJC5HVHREA, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.328400062 (414h24m51.156108454s) [noise=true , address=145.79.227.47:4266 , piece_id=EJYUC5ODHWWQEA7QK5JVT3T3HQ76G5WC5OHPNGUSYMP4AW5BWLPA, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.328834800 (414h24m51.156543191s) [noise=true , address=145.79.227.37:4566 , piece_id=EKCKTUJE37ZDSKNA6JM2ODSBLYAI2KFCA5RVFRNJPSCQ72ZWKY6A, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.330369052 (414h24m51.158077447s) [noise=true , address=145.79.227.51:4411 , piece_id=O76ELDU3V3XZ2ZYC7VCLZC2CU5VUMRTFWJEV2ITS5GFAGRSXTOSA, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.330425878 (414h24m51.158134271s) [noise=true , address=145.79.227.28:4727 , piece_id=Y5UCAOYDWNY5WPZC7LYUOT5QB325BSVZJUGSRVMR64ML23SEI4BQ, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.331015687 (414h24m51.158724085s) [noise=true , address=145.79.227.45:4229 , piece_id=QYCGIYFFNWLGOBOKEZYY4EBLHKC3SFYWOBQJ34DETCLF6BTS54VQ, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.331368793 (414h24m51.159077188s) [noise=true , address=145.79.227.48:4301 , piece_id=OCJPU5E7QLJWVY5K444J3J3ISMFY5E6MVXRMXBHIQRHXMIMRYC2A, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.332233159 (414h24m51.159941554s) [noise=true , address=145.79.227.50:4373 , piece_id=HESOK4IM7VRJNRL6NY5U4V52G7PDLSE3L3ZOF7DSTSQGFP6D2ZQA, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.333040553 (414h24m51.160748944s) [noise=true , address=145.79.227.35:4483 , piece_id=EJIXZOBJWPIVL37GIYYPHQRZDMEMS3YSRPVJH4GLMB4A2CSSRCTQ, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.333195450 (414h24m51.160903848s) [noise=true , address=145.79.227.43:4157 , piece_id=EKE6ZQ6YMYBBKZKQ2OIAAF5OI3OZRPJF4XDYCCU7QSKS33UXPDZQ, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.334537739 (414h24m51.162246133s) [noise=true , address=145.79.227.39:4015 , piece_id=AZSOOT5RLGTAHSO47SVO27XDSNFPE65JU2I5UEZDPQ44PKB2YLYA, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.413477504 (414h24m51.2411859s) [noise=true , address=77.90.31.10:4517 , piece_id=UAIKGPQF75AM6EKYBV3XK45CNYS2N2PNHKBAKRKO6DDILYAUFBLA, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:53:20.161293124 (414h25m20.989001514s) [noise=true , address=77.90.31.18:4805 , piece_id=YWS2P6F2X4PRKE7FVEA722G4FGDV2D4BXJILLHGQE4Q2HXRU2SAA, seg_pos={0 0}, upload=431251]: Done uploading piece. err: ecclient: upload failed (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, address:77.90.31.18:4805): protocol: expected piece hash; context deadline exceeded; EOF Dec 22 21:53:20.161472578 (414h25m20.989180974s) [noise=true , address=145.79.227.38:4978 , piece_id=S26D5M7SYZEXTTNQEELKEFFQTOMNA5SG7RL5THCON6KCFY267IDQ, seg_pos={0 0}, upload=431251]: Done uploading piece. err: ecclient: upload failed (node:12PcGHVSm5UAXPh5vCfyRHCd7Bdb67oXF51AZcnRFUjXqVYzXLR, address:145.79.227.38:4978): protocol: expected piece hash; context deadline exceeded; EOF Dec 22 21:53:20.161591128 (414h25m20.989299529s) [noise=true , address=145.79.227.32:4834 , piece_id=KLWKSAGBRLFWDKNCHW6UWKCADPZ6D6UURDMNMD5UCZUUWAPDOKVQ, seg_pos={0 0}, upload=431251]: Done uploading piece. err: ecclient: upload failed (node:1gRnamgAa2ccZS6F44apRmzSkXC6jGvNreJ5J8Y9nciWPZCSsk, address:145.79.227.32:4834): protocol: expected piece hash; context deadline exceeded; EOF Dec 22 21:53:20.161797225 (414h25m20.989505624s) [noise=true , address=145.79.227.15:4876 , piece_id=X5KN53INJAXSFQUEY2S37RYA3ELNBGN6ZFM6KKJ3A4ROOSUTBTRQ, seg_pos={0 0}, upload=431251]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:145.79.227.15:4876): protocol: expected piece hash; context deadline exceeded; EOF Dec 22 21:53:20.161824434 (414h25m20.989532813s) [seg_pos={0 0}, upload=431251]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 22 21:53:20.183925866 (414h25m21.011634264s) [upload=431251]: Done uploading segments... Dec 22 21:53:20.184019830 (414h25m21.011728235s) [upload=431251]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 442.307642ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............................X.............................] duration: 536.383291ms trace: https://jaeger.dev.storj.io/trace/17ef15eb0c5881c1 FAIL (2025-12-22 22:16:21 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 22 21:52:50.192390642 (414h24m51.020099033s) [upload=431251]: Starting upload Dec 22 21:52:50.192488257 (414h24m51.020196648s) [upload=431251]: Done starting upload Dec 22 21:52:50.192531865 (414h24m51.020240259s) [upload=431251]: Uploading segments... Dec 22 21:52:50.192783054 (414h24m51.020491446s) [upload=431251]: Got next segment. Inline: false Dec 22 21:52:50.192836048 (414h24m51.020544439s) [upload=431251]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 22 21:52:50.192841417 (414h24m51.020549808s) [upload=431251]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 22 21:52:50.220288008 (414h24m51.047996404s) [seg_pos={0 0}, upload=431251]: Begin upload segment... Dec 22 21:52:50.220342100 (414h24m51.048050497s) [seg_pos={0 0}, upload=431251]: Done begin upload segment. Dec 22 21:52:50.220384745 (414h24m51.048093145s) [noise=true , address=77.90.31.10:4517 , piece_id=UAIKGPQF75AM6EKYBV3XK45CNYS2N2PNHKBAKRKO6DDILYAUFBLA, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.220421097 (414h24m51.048129475s) [noise=true , address=145.79.227.45:4229 , piece_id=QYCGIYFFNWLGOBOKEZYY4EBLHKC3SFYWOBQJ34DETCLF6BTS54VQ, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.221068129 (414h24m51.048776527s) [noise=true , address=145.79.227.35:4483 , piece_id=EJIXZOBJWPIVL37GIYYPHQRZDMEMS3YSRPVJH4GLMB4A2CSSRCTQ, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.221771875 (414h24m51.049480268s) [noise=true , address=145.79.227.51:4411 , piece_id=O76ELDU3V3XZ2ZYC7VCLZC2CU5VUMRTFWJEV2ITS5GFAGRSXTOSA, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.222308205 (414h24m51.050016602s) [noise=true , address=145.79.227.39:4015 , piece_id=AZSOOT5RLGTAHSO47SVO27XDSNFPE65JU2I5UEZDPQ44PKB2YLYA, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.222424843 (414h24m51.050133244s) [noise=true , address=77.90.31.18:4805 , piece_id=YWS2P6F2X4PRKE7FVEA722G4FGDV2D4BXJILLHGQE4Q2HXRU2SAA, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.222721983 (414h24m51.050430374s) [noise=true , address=145.79.227.28:4727 , piece_id=Y5UCAOYDWNY5WPZC7LYUOT5QB325BSVZJUGSRVMR64ML23SEI4BQ, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.222910142 (414h24m51.050618542s) [noise=true , address=145.79.227.37:4566 , piece_id=EKCKTUJE37ZDSKNA6JM2ODSBLYAI2KFCA5RVFRNJPSCQ72ZWKY6A, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.223093651 (414h24m51.050802048s) [noise=true , address=145.79.227.50:4373 , piece_id=HESOK4IM7VRJNRL6NY5U4V52G7PDLSE3L3ZOF7DSTSQGFP6D2ZQA, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.223601176 (414h24m51.051309571s) [noise=true , address=145.79.227.15:4876 , piece_id=X5KN53INJAXSFQUEY2S37RYA3ELNBGN6ZFM6KKJ3A4ROOSUTBTRQ, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.224086995 (414h24m51.051795377s) [noise=true , address=145.79.227.43:4157 , piece_id=EKE6ZQ6YMYBBKZKQ2OIAAF5OI3OZRPJF4XDYCCU7QSKS33UXPDZQ, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.224569909 (414h24m51.052278304s) [noise=true , address=77.90.31.11:4553 , piece_id=MJFMBIGW3L6E75AAMZRCDONHZXPFLQSCK3VINMI4I2ZJC5HVHREA, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.225250077 (414h24m51.052958472s) [noise=true , address=145.79.227.48:4301 , piece_id=OCJPU5E7QLJWVY5K444J3J3ISMFY5E6MVXRMXBHIQRHXMIMRYC2A, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.226504914 (414h24m51.054213295s) [noise=true , address=145.79.227.47:4266 , piece_id=EJYUC5ODHWWQEA7QK5JVT3T3HQ76G5WC5OHPNGUSYMP4AW5BWLPA, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.227351850 (414h24m51.055060238s) [noise=true , address=145.79.227.38:4978 , piece_id=S26D5M7SYZEXTTNQEELKEFFQTOMNA5SG7RL5THCON6KCFY267IDQ, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.228140124 (414h24m51.055848511s) [noise=true , address=145.79.227.32:4834 , piece_id=KLWKSAGBRLFWDKNCHW6UWKCADPZ6D6UURDMNMD5UCZUUWAPDOKVQ, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.321476218 (414h24m51.149184612s) [upload=431251]: Next returned nil segment Dec 22 21:52:50.321492753 (414h24m51.149201151s) [upload=431251]: Waiting for error group managing segments... Dec 22 21:52:50.328186730 (414h24m51.155895125s) [noise=true , address=77.90.31.11:4553 , piece_id=MJFMBIGW3L6E75AAMZRCDONHZXPFLQSCK3VINMI4I2ZJC5HVHREA, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.328400062 (414h24m51.156108454s) [noise=true , address=145.79.227.47:4266 , piece_id=EJYUC5ODHWWQEA7QK5JVT3T3HQ76G5WC5OHPNGUSYMP4AW5BWLPA, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.328834800 (414h24m51.156543191s) [noise=true , address=145.79.227.37:4566 , piece_id=EKCKTUJE37ZDSKNA6JM2ODSBLYAI2KFCA5RVFRNJPSCQ72ZWKY6A, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.330369052 (414h24m51.158077447s) [noise=true , address=145.79.227.51:4411 , piece_id=O76ELDU3V3XZ2ZYC7VCLZC2CU5VUMRTFWJEV2ITS5GFAGRSXTOSA, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.330425878 (414h24m51.158134271s) [noise=true , address=145.79.227.28:4727 , piece_id=Y5UCAOYDWNY5WPZC7LYUOT5QB325BSVZJUGSRVMR64ML23SEI4BQ, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.331015687 (414h24m51.158724085s) [noise=true , address=145.79.227.45:4229 , piece_id=QYCGIYFFNWLGOBOKEZYY4EBLHKC3SFYWOBQJ34DETCLF6BTS54VQ, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.331368793 (414h24m51.159077188s) [noise=true , address=145.79.227.48:4301 , piece_id=OCJPU5E7QLJWVY5K444J3J3ISMFY5E6MVXRMXBHIQRHXMIMRYC2A, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.332233159 (414h24m51.159941554s) [noise=true , address=145.79.227.50:4373 , piece_id=HESOK4IM7VRJNRL6NY5U4V52G7PDLSE3L3ZOF7DSTSQGFP6D2ZQA, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.333040553 (414h24m51.160748944s) [noise=true , address=145.79.227.35:4483 , piece_id=EJIXZOBJWPIVL37GIYYPHQRZDMEMS3YSRPVJH4GLMB4A2CSSRCTQ, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.333195450 (414h24m51.160903848s) [noise=true , address=145.79.227.43:4157 , piece_id=EKE6ZQ6YMYBBKZKQ2OIAAF5OI3OZRPJF4XDYCCU7QSKS33UXPDZQ, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.334537739 (414h24m51.162246133s) [noise=true , address=145.79.227.39:4015 , piece_id=AZSOOT5RLGTAHSO47SVO27XDSNFPE65JU2I5UEZDPQ44PKB2YLYA, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.413477504 (414h24m51.2411859s) [noise=true , address=77.90.31.10:4517 , piece_id=UAIKGPQF75AM6EKYBV3XK45CNYS2N2PNHKBAKRKO6DDILYAUFBLA, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:53:20.161293124 (414h25m20.989001514s) [noise=true , address=77.90.31.18:4805 , piece_id=YWS2P6F2X4PRKE7FVEA722G4FGDV2D4BXJILLHGQE4Q2HXRU2SAA, seg_pos={0 0}, upload=431251]: Done uploading piece. err: ecclient: upload failed (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, address:77.90.31.18:4805): protocol: expected piece hash; context deadline exceeded; EOF Dec 22 21:53:20.161472578 (414h25m20.989180974s) [noise=true , address=145.79.227.38:4978 , piece_id=S26D5M7SYZEXTTNQEELKEFFQTOMNA5SG7RL5THCON6KCFY267IDQ, seg_pos={0 0}, upload=431251]: Done uploading piece. err: ecclient: upload failed (node:12PcGHVSm5UAXPh5vCfyRHCd7Bdb67oXF51AZcnRFUjXqVYzXLR, address:145.79.227.38:4978): protocol: expected piece hash; context deadline exceeded; EOF Dec 22 21:53:20.161591128 (414h25m20.989299529s) [noise=true , address=145.79.227.32:4834 , piece_id=KLWKSAGBRLFWDKNCHW6UWKCADPZ6D6UURDMNMD5UCZUUWAPDOKVQ, seg_pos={0 0}, upload=431251]: Done uploading piece. err: ecclient: upload failed (node:1gRnamgAa2ccZS6F44apRmzSkXC6jGvNreJ5J8Y9nciWPZCSsk, address:145.79.227.32:4834): protocol: expected piece hash; context deadline exceeded; EOF Dec 22 21:53:20.161797225 (414h25m20.989505624s) [noise=true , address=145.79.227.15:4876 , piece_id=X5KN53INJAXSFQUEY2S37RYA3ELNBGN6ZFM6KKJ3A4ROOSUTBTRQ, seg_pos={0 0}, upload=431251]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:145.79.227.15:4876): protocol: expected piece hash; context deadline exceeded; EOF Dec 22 21:53:20.161824434 (414h25m20.989532813s) [seg_pos={0 0}, upload=431251]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 22 21:53:20.183925866 (414h25m21.011634264s) [upload=431251]: Done uploading segments... Dec 22 21:53:20.184019830 (414h25m21.011728235s) [upload=431251]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 536.412516ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............................X............................] duration: 469.66945ms trace: https://jaeger.dev.storj.io/trace/17ef15eb0c5881c1 FAIL (2025-12-22 22:15:13 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 22 21:52:50.192390642 (414h24m51.020099033s) [upload=431251]: Starting upload Dec 22 21:52:50.192488257 (414h24m51.020196648s) [upload=431251]: Done starting upload Dec 22 21:52:50.192531865 (414h24m51.020240259s) [upload=431251]: Uploading segments... Dec 22 21:52:50.192783054 (414h24m51.020491446s) [upload=431251]: Got next segment. Inline: false Dec 22 21:52:50.192836048 (414h24m51.020544439s) [upload=431251]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 22 21:52:50.192841417 (414h24m51.020549808s) [upload=431251]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 22 21:52:50.220288008 (414h24m51.047996404s) [seg_pos={0 0}, upload=431251]: Begin upload segment... Dec 22 21:52:50.220342100 (414h24m51.048050497s) [seg_pos={0 0}, upload=431251]: Done begin upload segment. Dec 22 21:52:50.220384745 (414h24m51.048093145s) [noise=true , address=77.90.31.10:4517 , piece_id=UAIKGPQF75AM6EKYBV3XK45CNYS2N2PNHKBAKRKO6DDILYAUFBLA, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.220421097 (414h24m51.048129475s) [noise=true , address=145.79.227.45:4229 , piece_id=QYCGIYFFNWLGOBOKEZYY4EBLHKC3SFYWOBQJ34DETCLF6BTS54VQ, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.221068129 (414h24m51.048776527s) [noise=true , address=145.79.227.35:4483 , piece_id=EJIXZOBJWPIVL37GIYYPHQRZDMEMS3YSRPVJH4GLMB4A2CSSRCTQ, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.221771875 (414h24m51.049480268s) [noise=true , address=145.79.227.51:4411 , piece_id=O76ELDU3V3XZ2ZYC7VCLZC2CU5VUMRTFWJEV2ITS5GFAGRSXTOSA, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.222308205 (414h24m51.050016602s) [noise=true , address=145.79.227.39:4015 , piece_id=AZSOOT5RLGTAHSO47SVO27XDSNFPE65JU2I5UEZDPQ44PKB2YLYA, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.222424843 (414h24m51.050133244s) [noise=true , address=77.90.31.18:4805 , piece_id=YWS2P6F2X4PRKE7FVEA722G4FGDV2D4BXJILLHGQE4Q2HXRU2SAA, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.222721983 (414h24m51.050430374s) [noise=true , address=145.79.227.28:4727 , piece_id=Y5UCAOYDWNY5WPZC7LYUOT5QB325BSVZJUGSRVMR64ML23SEI4BQ, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.222910142 (414h24m51.050618542s) [noise=true , address=145.79.227.37:4566 , piece_id=EKCKTUJE37ZDSKNA6JM2ODSBLYAI2KFCA5RVFRNJPSCQ72ZWKY6A, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.223093651 (414h24m51.050802048s) [noise=true , address=145.79.227.50:4373 , piece_id=HESOK4IM7VRJNRL6NY5U4V52G7PDLSE3L3ZOF7DSTSQGFP6D2ZQA, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.223601176 (414h24m51.051309571s) [noise=true , address=145.79.227.15:4876 , piece_id=X5KN53INJAXSFQUEY2S37RYA3ELNBGN6ZFM6KKJ3A4ROOSUTBTRQ, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.224086995 (414h24m51.051795377s) [noise=true , address=145.79.227.43:4157 , piece_id=EKE6ZQ6YMYBBKZKQ2OIAAF5OI3OZRPJF4XDYCCU7QSKS33UXPDZQ, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.224569909 (414h24m51.052278304s) [noise=true , address=77.90.31.11:4553 , piece_id=MJFMBIGW3L6E75AAMZRCDONHZXPFLQSCK3VINMI4I2ZJC5HVHREA, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.225250077 (414h24m51.052958472s) [noise=true , address=145.79.227.48:4301 , piece_id=OCJPU5E7QLJWVY5K444J3J3ISMFY5E6MVXRMXBHIQRHXMIMRYC2A, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.226504914 (414h24m51.054213295s) [noise=true , address=145.79.227.47:4266 , piece_id=EJYUC5ODHWWQEA7QK5JVT3T3HQ76G5WC5OHPNGUSYMP4AW5BWLPA, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.227351850 (414h24m51.055060238s) [noise=true , address=145.79.227.38:4978 , piece_id=S26D5M7SYZEXTTNQEELKEFFQTOMNA5SG7RL5THCON6KCFY267IDQ, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.228140124 (414h24m51.055848511s) [noise=true , address=145.79.227.32:4834 , piece_id=KLWKSAGBRLFWDKNCHW6UWKCADPZ6D6UURDMNMD5UCZUUWAPDOKVQ, seg_pos={0 0}, upload=431251]: Uploading piece... Dec 22 21:52:50.321476218 (414h24m51.149184612s) [upload=431251]: Next returned nil segment Dec 22 21:52:50.321492753 (414h24m51.149201151s) [upload=431251]: Waiting for error group managing segments... Dec 22 21:52:50.328186730 (414h24m51.155895125s) [noise=true , address=77.90.31.11:4553 , piece_id=MJFMBIGW3L6E75AAMZRCDONHZXPFLQSCK3VINMI4I2ZJC5HVHREA, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.328400062 (414h24m51.156108454s) [noise=true , address=145.79.227.47:4266 , piece_id=EJYUC5ODHWWQEA7QK5JVT3T3HQ76G5WC5OHPNGUSYMP4AW5BWLPA, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.328834800 (414h24m51.156543191s) [noise=true , address=145.79.227.37:4566 , piece_id=EKCKTUJE37ZDSKNA6JM2ODSBLYAI2KFCA5RVFRNJPSCQ72ZWKY6A, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.330369052 (414h24m51.158077447s) [noise=true , address=145.79.227.51:4411 , piece_id=O76ELDU3V3XZ2ZYC7VCLZC2CU5VUMRTFWJEV2ITS5GFAGRSXTOSA, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.330425878 (414h24m51.158134271s) [noise=true , address=145.79.227.28:4727 , piece_id=Y5UCAOYDWNY5WPZC7LYUOT5QB325BSVZJUGSRVMR64ML23SEI4BQ, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.331015687 (414h24m51.158724085s) [noise=true , address=145.79.227.45:4229 , piece_id=QYCGIYFFNWLGOBOKEZYY4EBLHKC3SFYWOBQJ34DETCLF6BTS54VQ, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.331368793 (414h24m51.159077188s) [noise=true , address=145.79.227.48:4301 , piece_id=OCJPU5E7QLJWVY5K444J3J3ISMFY5E6MVXRMXBHIQRHXMIMRYC2A, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.332233159 (414h24m51.159941554s) [noise=true , address=145.79.227.50:4373 , piece_id=HESOK4IM7VRJNRL6NY5U4V52G7PDLSE3L3ZOF7DSTSQGFP6D2ZQA, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.333040553 (414h24m51.160748944s) [noise=true , address=145.79.227.35:4483 , piece_id=EJIXZOBJWPIVL37GIYYPHQRZDMEMS3YSRPVJH4GLMB4A2CSSRCTQ, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.333195450 (414h24m51.160903848s) [noise=true , address=145.79.227.43:4157 , piece_id=EKE6ZQ6YMYBBKZKQ2OIAAF5OI3OZRPJF4XDYCCU7QSKS33UXPDZQ, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.334537739 (414h24m51.162246133s) [noise=true , address=145.79.227.39:4015 , piece_id=AZSOOT5RLGTAHSO47SVO27XDSNFPE65JU2I5UEZDPQ44PKB2YLYA, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:52:50.413477504 (414h24m51.2411859s) [noise=true , address=77.90.31.10:4517 , piece_id=UAIKGPQF75AM6EKYBV3XK45CNYS2N2PNHKBAKRKO6DDILYAUFBLA, seg_pos={0 0}, upload=431251]: Done uploading piece. err: <nil> Dec 22 21:53:20.161293124 (414h25m20.989001514s) [noise=true , address=77.90.31.18:4805 , piece_id=YWS2P6F2X4PRKE7FVEA722G4FGDV2D4BXJILLHGQE4Q2HXRU2SAA, seg_pos={0 0}, upload=431251]: Done uploading piece. err: ecclient: upload failed (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, address:77.90.31.18:4805): protocol: expected piece hash; context deadline exceeded; EOF Dec 22 21:53:20.161472578 (414h25m20.989180974s) [noise=true , address=145.79.227.38:4978 , piece_id=S26D5M7SYZEXTTNQEELKEFFQTOMNA5SG7RL5THCON6KCFY267IDQ, seg_pos={0 0}, upload=431251]: Done uploading piece. err: ecclient: upload failed (node:12PcGHVSm5UAXPh5vCfyRHCd7Bdb67oXF51AZcnRFUjXqVYzXLR, address:145.79.227.38:4978): protocol: expected piece hash; context deadline exceeded; EOF Dec 22 21:53:20.161591128 (414h25m20.989299529s) [noise=true , address=145.79.227.32:4834 , piece_id=KLWKSAGBRLFWDKNCHW6UWKCADPZ6D6UURDMNMD5UCZUUWAPDOKVQ, seg_pos={0 0}, upload=431251]: Done uploading piece. err: ecclient: upload failed (node:1gRnamgAa2ccZS6F44apRmzSkXC6jGvNreJ5J8Y9nciWPZCSsk, address:145.79.227.32:4834): protocol: expected piece hash; context deadline exceeded; EOF Dec 22 21:53:20.161797225 (414h25m20.989505624s) [noise=true , address=145.79.227.15:4876 , piece_id=X5KN53INJAXSFQUEY2S37RYA3ELNBGN6ZFM6KKJ3A4ROOSUTBTRQ, seg_pos={0 0}, upload=431251]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:145.79.227.15:4876): protocol: expected piece hash; context deadline exceeded; EOF Dec 22 21:53:20.161824434 (414h25m20.989532813s) [seg_pos={0 0}, upload=431251]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 22 21:53:20.183925866 (414h25m21.011634264s) [upload=431251]: Done uploading segments... Dec 22 21:53:20.184019830 (414h25m21.011728235s) [upload=431251]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 469.774204ms core uplink us-select-1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)