Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [X...........................................................] duration: 491.521405ms trace: https://jaeger.dev.storj.io/trace/7013ffb1a981d605 FAIL (2025-12-22 23:16:21 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 22 22:31:05.439510308 (415h3m6.2672187s) [upload=431917]: Starting upload Dec 22 22:31:05.439581993 (415h3m6.26729037s) [upload=431917]: Done starting upload Dec 22 22:31:05.439812792 (415h3m6.267521185s) [upload=431917]: Uploading segments... Dec 22 22:31:05.439942524 (415h3m6.267650918s) [upload=431917]: Got next segment. Inline: false Dec 22 22:31:05.440022744 (415h3m6.267731174s) [upload=431917]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 22 22:31:05.440029670 (415h3m6.267738072s) [upload=431917]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 22 22:31:05.467186372 (415h3m6.294894769s) [seg_pos={0 0}, upload=431917]: Begin upload segment... Dec 22 22:31:05.467265149 (415h3m6.294973535s) [seg_pos={0 0}, upload=431917]: Done begin upload segment. Dec 22 22:31:05.467296974 (415h3m6.295005373s) [noise=true , address=145.79.227.51:4409 , piece_id=ATH3F4DFOUUIFFJZRA2SSRRL7AKWZMQNQRYOZCVFFDLTFCFP236A, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.467389853 (415h3m6.295098247s) [noise=true , address=145.79.227.39:4013 , piece_id=JJVURGSLKY2UQ4FKNCLATA3ZBD5SDL32XJAG42KH2U6OYK3KTS6Q, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.467912104 (415h3m6.295620486s) [noise=true , address=145.79.227.36:4519 , piece_id=JUKRRIZWVGKEDD3GPJ7HMY2MEQZNMUCTKTXR6ZLSIERRONTZKHCQ, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.468324863 (415h3m6.296033263s) [noise=true , address=145.79.227.38:4979 , piece_id=HLMQC5OEB4OL4OZSRMCEVAWR5QM2T4MQUBPTHRVQLEAHPBWPQF6Q, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.468334764 (415h3m6.296043147s) [noise=true , address=145.79.227.40:4051 , piece_id=55RRAD7DQX6SR7RBLYAV7MD7XRWPNIWHTVVBQZ2ZL3OE2L7LRRIQ, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.468742525 (415h3m6.296450916s) [noise=true , address=45.91.116.4:4939 , piece_id=QYNKMKDLQJV7B6M2E34DU53Y7YI6VBCUW3C7ZLQOI4XWTBXZHKDQ, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.468915034 (415h3m6.296623414s) [noise=true , address=45.91.116.11:4117 , piece_id=WLMO5E3BIFFL3TOAUSSBGORIZ6DOCWFCMVT4AVFPPGPSKP2OKDAA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.469069393 (415h3m6.296777787s) [noise=true , address=145.79.227.32:4835 , piece_id=BRE3JH3XKPXB6FN7WB3W6ED7MOFES6HYLR34YGR6L5AX7RPIQ7IA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.469285789 (415h3m6.296994183s) [noise=true , address=145.79.227.29:4943 , piece_id=G4TMZ6IAIZBDON3UEDPH4LJQFHV3JLALEXSJ3VI4XN6WHIZOB57Q, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.469411962 (415h3m6.297120355s) [noise=true , address=145.79.227.48:4301 , piece_id=LBIYTJNOX5J4M3HDE7FGF5CSARBITW2AK3P2DSYFLFKUOIHZUKOQ, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.469749379 (415h3m6.297457773s) [noise=true , address=145.79.227.37:4565 , piece_id=554RHTYPG27UNCFPSRMOB245AX4QXL2QFIYECNTVWNNJMAL4BTYA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.470080828 (415h3m6.29778922s) [noise=true , address=145.79.227.43:4157 , piece_id=3W3LVTU37DNHWS52IGIIH7SXWXVNPGEY27RQNT272L3Z64GNR4KA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.470477660 (415h3m6.298186058s) [noise=true , address=45.91.116.14:4190 , piece_id=V4T6Z7K3C75LQEXNQIYZGYWW2CSQ7UPSET46QGDRT52RFGAROTTA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.470567977 (415h3m6.29827637s) [noise=true , address=145.79.227.15:4874 , piece_id=FNCCIMIVJTGZ2X5TOYOZH727JXHYM7HG3HBHBTZDAG67SHZLYYYA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.470980950 (415h3m6.298689349s) [noise=true , address=145.79.227.50:4373 , piece_id=4U55T5QDYCE6AZ3NCTPAVECPAELPQUKT3HV7GZL7JFRMWRN2AZUQ, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.471440098 (415h3m6.299148489s) [noise=true , address=145.79.227.47:4266 , piece_id=3NFYV2PWBNTAOSFHENM4J5PZ7XXE72UKTQVXLTVQRHGK4DUXAKDA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.560766475 (415h3m6.388474871s) [upload=431917]: Next returned nil segment Dec 22 22:31:05.560785161 (415h3m6.388493562s) [upload=431917]: Waiting for error group managing segments... Dec 22 22:31:05.566117878 (415h3m6.39382628s) [noise=true , address=45.91.116.4:4939 , piece_id=QYNKMKDLQJV7B6M2E34DU53Y7YI6VBCUW3C7ZLQOI4XWTBXZHKDQ, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.566723306 (415h3m6.394431687s) [noise=true , address=45.91.116.11:4117 , piece_id=WLMO5E3BIFFL3TOAUSSBGORIZ6DOCWFCMVT4AVFPPGPSKP2OKDAA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.566992667 (415h3m6.394701063s) [noise=true , address=145.79.227.36:4519 , piece_id=JUKRRIZWVGKEDD3GPJ7HMY2MEQZNMUCTKTXR6ZLSIERRONTZKHCQ, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.567464626 (415h3m6.395173019s) [noise=true , address=145.79.227.47:4266 , piece_id=3NFYV2PWBNTAOSFHENM4J5PZ7XXE72UKTQVXLTVQRHGK4DUXAKDA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.567924492 (415h3m6.395632883s) [noise=true , address=145.79.227.15:4874 , piece_id=FNCCIMIVJTGZ2X5TOYOZH727JXHYM7HG3HBHBTZDAG67SHZLYYYA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.568807294 (415h3m6.396515685s) [noise=true , address=145.79.227.51:4409 , piece_id=ATH3F4DFOUUIFFJZRA2SSRRL7AKWZMQNQRYOZCVFFDLTFCFP236A, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.570579823 (415h3m6.398288214s) [noise=true , address=145.79.227.48:4301 , piece_id=LBIYTJNOX5J4M3HDE7FGF5CSARBITW2AK3P2DSYFLFKUOIHZUKOQ, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.571632758 (415h3m6.399341149s) [noise=true , address=145.79.227.43:4157 , piece_id=3W3LVTU37DNHWS52IGIIH7SXWXVNPGEY27RQNT272L3Z64GNR4KA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.573168383 (415h3m6.400876774s) [noise=true , address=145.79.227.38:4979 , piece_id=HLMQC5OEB4OL4OZSRMCEVAWR5QM2T4MQUBPTHRVQLEAHPBWPQF6Q, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.579995420 (415h3m6.407703816s) [noise=true , address=45.91.116.14:4190 , piece_id=V4T6Z7K3C75LQEXNQIYZGYWW2CSQ7UPSET46QGDRT52RFGAROTTA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.592093172 (415h3m6.419801563s) [noise=true , address=145.79.227.37:4565 , piece_id=554RHTYPG27UNCFPSRMOB245AX4QXL2QFIYECNTVWNNJMAL4BTYA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:09.946500434 (415h3m10.774208816s) [noise=true , address=145.79.227.40:4051 , piece_id=55RRAD7DQX6SR7RBLYAV7MD7XRWPNIWHTVVBQZ2ZL3OE2L7LRRIQ, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:35.398235543 (415h3m36.225943943s) [noise=true , address=145.79.227.39:4013 , piece_id=JJVURGSLKY2UQ4FKNCLATA3ZBD5SDL32XJAG42KH2U6OYK3KTS6Q, seg_pos={0 0}, upload=431917]: Done uploading piece. err: ecclient: upload failed (node:121scLvKEW9zw2aEz22zpremcjiop67eQ8v2uqpmaD5aZzCAP7b, address:145.79.227.39:4013): protocol: write tcp 159.223.143.39:60842->145.79.227.39:4013: use of closed network connection; write tcp 159.223.143.39:60842->145.79.227.39:4013: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:60842->145.79.227.39:4013: use of closed network connection Dec 22 22:31:35.398562305 (415h3m36.226270699s) [noise=true , address=145.79.227.32:4835 , piece_id=BRE3JH3XKPXB6FN7WB3W6ED7MOFES6HYLR34YGR6L5AX7RPIQ7IA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: ecclient: upload failed (node:16vVfeoqCkbMYCRjMMWaKBnCvQBBJPMgwQtWTq9HKQrD9SS17y, address:145.79.227.32:4835): protocol: write tcp 159.223.143.39:45098->145.79.227.32:4835: use of closed network connection; write tcp 159.223.143.39:45098->145.79.227.32:4835: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:45098->145.79.227.32:4835: use of closed network connection Dec 22 22:31:35.399183028 (415h3m36.226891422s) [noise=true , address=145.79.227.50:4373 , piece_id=4U55T5QDYCE6AZ3NCTPAVECPAELPQUKT3HV7GZL7JFRMWRN2AZUQ, seg_pos={0 0}, upload=431917]: Done uploading piece. err: ecclient: upload failed (node:12AQGSkJuNJDxoeqhChUenWxHwF5yK41UESCF63FQz3XvabXXfz, address:145.79.227.50:4373): protocol: read tcp 159.223.143.39:56028->145.79.227.50:4373: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 22 22:31:35.399278021 (415h3m36.226986417s) [noise=true , address=145.79.227.29:4943 , piece_id=G4TMZ6IAIZBDON3UEDPH4LJQFHV3JLALEXSJ3VI4XN6WHIZOB57Q, seg_pos={0 0}, upload=431917]: Done uploading piece. err: ecclient: upload failed (node:12fc5Trj8XxacZhzxdksUVT5CbG6uTmiwdUPjreYi4UMPY1TXWz, address:145.79.227.29:4943): protocol: write tcp 159.223.143.39:50820->145.79.227.29:4943: use of closed network connection; write tcp 159.223.143.39:50820->145.79.227.29:4943: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:50820->145.79.227.29:4943: use of closed network connection Dec 22 22:31:35.399379485 (415h3m36.227087866s) [seg_pos={0 0}, upload=431917]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 22 22:31:35.421122544 (415h3m36.248830935s) [upload=431917]: Done uploading segments... Dec 22 22:31:35.421174369 (415h3m36.248882769s) [upload=431917]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 491.571171ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.X..........................................................] duration: 553.281854ms trace: https://jaeger.dev.storj.io/trace/7013ffb1a981d605 FAIL (2025-12-22 23:15:20 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 22 22:31:05.439510308 (415h3m6.2672187s) [upload=431917]: Starting upload Dec 22 22:31:05.439581993 (415h3m6.26729037s) [upload=431917]: Done starting upload Dec 22 22:31:05.439812792 (415h3m6.267521185s) [upload=431917]: Uploading segments... Dec 22 22:31:05.439942524 (415h3m6.267650918s) [upload=431917]: Got next segment. Inline: false Dec 22 22:31:05.440022744 (415h3m6.267731174s) [upload=431917]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 22 22:31:05.440029670 (415h3m6.267738072s) [upload=431917]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 22 22:31:05.467186372 (415h3m6.294894769s) [seg_pos={0 0}, upload=431917]: Begin upload segment... Dec 22 22:31:05.467265149 (415h3m6.294973535s) [seg_pos={0 0}, upload=431917]: Done begin upload segment. Dec 22 22:31:05.467296974 (415h3m6.295005373s) [noise=true , address=145.79.227.51:4409 , piece_id=ATH3F4DFOUUIFFJZRA2SSRRL7AKWZMQNQRYOZCVFFDLTFCFP236A, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.467389853 (415h3m6.295098247s) [noise=true , address=145.79.227.39:4013 , piece_id=JJVURGSLKY2UQ4FKNCLATA3ZBD5SDL32XJAG42KH2U6OYK3KTS6Q, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.467912104 (415h3m6.295620486s) [noise=true , address=145.79.227.36:4519 , piece_id=JUKRRIZWVGKEDD3GPJ7HMY2MEQZNMUCTKTXR6ZLSIERRONTZKHCQ, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.468324863 (415h3m6.296033263s) [noise=true , address=145.79.227.38:4979 , piece_id=HLMQC5OEB4OL4OZSRMCEVAWR5QM2T4MQUBPTHRVQLEAHPBWPQF6Q, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.468334764 (415h3m6.296043147s) [noise=true , address=145.79.227.40:4051 , piece_id=55RRAD7DQX6SR7RBLYAV7MD7XRWPNIWHTVVBQZ2ZL3OE2L7LRRIQ, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.468742525 (415h3m6.296450916s) [noise=true , address=45.91.116.4:4939 , piece_id=QYNKMKDLQJV7B6M2E34DU53Y7YI6VBCUW3C7ZLQOI4XWTBXZHKDQ, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.468915034 (415h3m6.296623414s) [noise=true , address=45.91.116.11:4117 , piece_id=WLMO5E3BIFFL3TOAUSSBGORIZ6DOCWFCMVT4AVFPPGPSKP2OKDAA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.469069393 (415h3m6.296777787s) [noise=true , address=145.79.227.32:4835 , piece_id=BRE3JH3XKPXB6FN7WB3W6ED7MOFES6HYLR34YGR6L5AX7RPIQ7IA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.469285789 (415h3m6.296994183s) [noise=true , address=145.79.227.29:4943 , piece_id=G4TMZ6IAIZBDON3UEDPH4LJQFHV3JLALEXSJ3VI4XN6WHIZOB57Q, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.469411962 (415h3m6.297120355s) [noise=true , address=145.79.227.48:4301 , piece_id=LBIYTJNOX5J4M3HDE7FGF5CSARBITW2AK3P2DSYFLFKUOIHZUKOQ, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.469749379 (415h3m6.297457773s) [noise=true , address=145.79.227.37:4565 , piece_id=554RHTYPG27UNCFPSRMOB245AX4QXL2QFIYECNTVWNNJMAL4BTYA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.470080828 (415h3m6.29778922s) [noise=true , address=145.79.227.43:4157 , piece_id=3W3LVTU37DNHWS52IGIIH7SXWXVNPGEY27RQNT272L3Z64GNR4KA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.470477660 (415h3m6.298186058s) [noise=true , address=45.91.116.14:4190 , piece_id=V4T6Z7K3C75LQEXNQIYZGYWW2CSQ7UPSET46QGDRT52RFGAROTTA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.470567977 (415h3m6.29827637s) [noise=true , address=145.79.227.15:4874 , piece_id=FNCCIMIVJTGZ2X5TOYOZH727JXHYM7HG3HBHBTZDAG67SHZLYYYA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.470980950 (415h3m6.298689349s) [noise=true , address=145.79.227.50:4373 , piece_id=4U55T5QDYCE6AZ3NCTPAVECPAELPQUKT3HV7GZL7JFRMWRN2AZUQ, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.471440098 (415h3m6.299148489s) [noise=true , address=145.79.227.47:4266 , piece_id=3NFYV2PWBNTAOSFHENM4J5PZ7XXE72UKTQVXLTVQRHGK4DUXAKDA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.560766475 (415h3m6.388474871s) [upload=431917]: Next returned nil segment Dec 22 22:31:05.560785161 (415h3m6.388493562s) [upload=431917]: Waiting for error group managing segments... Dec 22 22:31:05.566117878 (415h3m6.39382628s) [noise=true , address=45.91.116.4:4939 , piece_id=QYNKMKDLQJV7B6M2E34DU53Y7YI6VBCUW3C7ZLQOI4XWTBXZHKDQ, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.566723306 (415h3m6.394431687s) [noise=true , address=45.91.116.11:4117 , piece_id=WLMO5E3BIFFL3TOAUSSBGORIZ6DOCWFCMVT4AVFPPGPSKP2OKDAA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.566992667 (415h3m6.394701063s) [noise=true , address=145.79.227.36:4519 , piece_id=JUKRRIZWVGKEDD3GPJ7HMY2MEQZNMUCTKTXR6ZLSIERRONTZKHCQ, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.567464626 (415h3m6.395173019s) [noise=true , address=145.79.227.47:4266 , piece_id=3NFYV2PWBNTAOSFHENM4J5PZ7XXE72UKTQVXLTVQRHGK4DUXAKDA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.567924492 (415h3m6.395632883s) [noise=true , address=145.79.227.15:4874 , piece_id=FNCCIMIVJTGZ2X5TOYOZH727JXHYM7HG3HBHBTZDAG67SHZLYYYA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.568807294 (415h3m6.396515685s) [noise=true , address=145.79.227.51:4409 , piece_id=ATH3F4DFOUUIFFJZRA2SSRRL7AKWZMQNQRYOZCVFFDLTFCFP236A, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.570579823 (415h3m6.398288214s) [noise=true , address=145.79.227.48:4301 , piece_id=LBIYTJNOX5J4M3HDE7FGF5CSARBITW2AK3P2DSYFLFKUOIHZUKOQ, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.571632758 (415h3m6.399341149s) [noise=true , address=145.79.227.43:4157 , piece_id=3W3LVTU37DNHWS52IGIIH7SXWXVNPGEY27RQNT272L3Z64GNR4KA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.573168383 (415h3m6.400876774s) [noise=true , address=145.79.227.38:4979 , piece_id=HLMQC5OEB4OL4OZSRMCEVAWR5QM2T4MQUBPTHRVQLEAHPBWPQF6Q, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.579995420 (415h3m6.407703816s) [noise=true , address=45.91.116.14:4190 , piece_id=V4T6Z7K3C75LQEXNQIYZGYWW2CSQ7UPSET46QGDRT52RFGAROTTA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.592093172 (415h3m6.419801563s) [noise=true , address=145.79.227.37:4565 , piece_id=554RHTYPG27UNCFPSRMOB245AX4QXL2QFIYECNTVWNNJMAL4BTYA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:09.946500434 (415h3m10.774208816s) [noise=true , address=145.79.227.40:4051 , piece_id=55RRAD7DQX6SR7RBLYAV7MD7XRWPNIWHTVVBQZ2ZL3OE2L7LRRIQ, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:35.398235543 (415h3m36.225943943s) [noise=true , address=145.79.227.39:4013 , piece_id=JJVURGSLKY2UQ4FKNCLATA3ZBD5SDL32XJAG42KH2U6OYK3KTS6Q, seg_pos={0 0}, upload=431917]: Done uploading piece. err: ecclient: upload failed (node:121scLvKEW9zw2aEz22zpremcjiop67eQ8v2uqpmaD5aZzCAP7b, address:145.79.227.39:4013): protocol: write tcp 159.223.143.39:60842->145.79.227.39:4013: use of closed network connection; write tcp 159.223.143.39:60842->145.79.227.39:4013: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:60842->145.79.227.39:4013: use of closed network connection Dec 22 22:31:35.398562305 (415h3m36.226270699s) [noise=true , address=145.79.227.32:4835 , piece_id=BRE3JH3XKPXB6FN7WB3W6ED7MOFES6HYLR34YGR6L5AX7RPIQ7IA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: ecclient: upload failed (node:16vVfeoqCkbMYCRjMMWaKBnCvQBBJPMgwQtWTq9HKQrD9SS17y, address:145.79.227.32:4835): protocol: write tcp 159.223.143.39:45098->145.79.227.32:4835: use of closed network connection; write tcp 159.223.143.39:45098->145.79.227.32:4835: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:45098->145.79.227.32:4835: use of closed network connection Dec 22 22:31:35.399183028 (415h3m36.226891422s) [noise=true , address=145.79.227.50:4373 , piece_id=4U55T5QDYCE6AZ3NCTPAVECPAELPQUKT3HV7GZL7JFRMWRN2AZUQ, seg_pos={0 0}, upload=431917]: Done uploading piece. err: ecclient: upload failed (node:12AQGSkJuNJDxoeqhChUenWxHwF5yK41UESCF63FQz3XvabXXfz, address:145.79.227.50:4373): protocol: read tcp 159.223.143.39:56028->145.79.227.50:4373: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 22 22:31:35.399278021 (415h3m36.226986417s) [noise=true , address=145.79.227.29:4943 , piece_id=G4TMZ6IAIZBDON3UEDPH4LJQFHV3JLALEXSJ3VI4XN6WHIZOB57Q, seg_pos={0 0}, upload=431917]: Done uploading piece. err: ecclient: upload failed (node:12fc5Trj8XxacZhzxdksUVT5CbG6uTmiwdUPjreYi4UMPY1TXWz, address:145.79.227.29:4943): protocol: write tcp 159.223.143.39:50820->145.79.227.29:4943: use of closed network connection; write tcp 159.223.143.39:50820->145.79.227.29:4943: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:50820->145.79.227.29:4943: use of closed network connection Dec 22 22:31:35.399379485 (415h3m36.227087866s) [seg_pos={0 0}, upload=431917]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 22 22:31:35.421122544 (415h3m36.248830935s) [upload=431917]: Done uploading segments... Dec 22 22:31:35.421174369 (415h3m36.248882769s) [upload=431917]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 553.3137ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..X.........................................................] duration: 446.716365ms trace: https://jaeger.dev.storj.io/trace/7013ffb1a981d605 FAIL (2025-12-22 23:14:32 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 22 22:31:05.439510308 (415h3m6.2672187s) [upload=431917]: Starting upload Dec 22 22:31:05.439581993 (415h3m6.26729037s) [upload=431917]: Done starting upload Dec 22 22:31:05.439812792 (415h3m6.267521185s) [upload=431917]: Uploading segments... Dec 22 22:31:05.439942524 (415h3m6.267650918s) [upload=431917]: Got next segment. Inline: false Dec 22 22:31:05.440022744 (415h3m6.267731174s) [upload=431917]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 22 22:31:05.440029670 (415h3m6.267738072s) [upload=431917]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 22 22:31:05.467186372 (415h3m6.294894769s) [seg_pos={0 0}, upload=431917]: Begin upload segment... Dec 22 22:31:05.467265149 (415h3m6.294973535s) [seg_pos={0 0}, upload=431917]: Done begin upload segment. Dec 22 22:31:05.467296974 (415h3m6.295005373s) [noise=true , address=145.79.227.51:4409 , piece_id=ATH3F4DFOUUIFFJZRA2SSRRL7AKWZMQNQRYOZCVFFDLTFCFP236A, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.467389853 (415h3m6.295098247s) [noise=true , address=145.79.227.39:4013 , piece_id=JJVURGSLKY2UQ4FKNCLATA3ZBD5SDL32XJAG42KH2U6OYK3KTS6Q, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.467912104 (415h3m6.295620486s) [noise=true , address=145.79.227.36:4519 , piece_id=JUKRRIZWVGKEDD3GPJ7HMY2MEQZNMUCTKTXR6ZLSIERRONTZKHCQ, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.468324863 (415h3m6.296033263s) [noise=true , address=145.79.227.38:4979 , piece_id=HLMQC5OEB4OL4OZSRMCEVAWR5QM2T4MQUBPTHRVQLEAHPBWPQF6Q, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.468334764 (415h3m6.296043147s) [noise=true , address=145.79.227.40:4051 , piece_id=55RRAD7DQX6SR7RBLYAV7MD7XRWPNIWHTVVBQZ2ZL3OE2L7LRRIQ, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.468742525 (415h3m6.296450916s) [noise=true , address=45.91.116.4:4939 , piece_id=QYNKMKDLQJV7B6M2E34DU53Y7YI6VBCUW3C7ZLQOI4XWTBXZHKDQ, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.468915034 (415h3m6.296623414s) [noise=true , address=45.91.116.11:4117 , piece_id=WLMO5E3BIFFL3TOAUSSBGORIZ6DOCWFCMVT4AVFPPGPSKP2OKDAA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.469069393 (415h3m6.296777787s) [noise=true , address=145.79.227.32:4835 , piece_id=BRE3JH3XKPXB6FN7WB3W6ED7MOFES6HYLR34YGR6L5AX7RPIQ7IA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.469285789 (415h3m6.296994183s) [noise=true , address=145.79.227.29:4943 , piece_id=G4TMZ6IAIZBDON3UEDPH4LJQFHV3JLALEXSJ3VI4XN6WHIZOB57Q, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.469411962 (415h3m6.297120355s) [noise=true , address=145.79.227.48:4301 , piece_id=LBIYTJNOX5J4M3HDE7FGF5CSARBITW2AK3P2DSYFLFKUOIHZUKOQ, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.469749379 (415h3m6.297457773s) [noise=true , address=145.79.227.37:4565 , piece_id=554RHTYPG27UNCFPSRMOB245AX4QXL2QFIYECNTVWNNJMAL4BTYA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.470080828 (415h3m6.29778922s) [noise=true , address=145.79.227.43:4157 , piece_id=3W3LVTU37DNHWS52IGIIH7SXWXVNPGEY27RQNT272L3Z64GNR4KA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.470477660 (415h3m6.298186058s) [noise=true , address=45.91.116.14:4190 , piece_id=V4T6Z7K3C75LQEXNQIYZGYWW2CSQ7UPSET46QGDRT52RFGAROTTA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.470567977 (415h3m6.29827637s) [noise=true , address=145.79.227.15:4874 , piece_id=FNCCIMIVJTGZ2X5TOYOZH727JXHYM7HG3HBHBTZDAG67SHZLYYYA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.470980950 (415h3m6.298689349s) [noise=true , address=145.79.227.50:4373 , piece_id=4U55T5QDYCE6AZ3NCTPAVECPAELPQUKT3HV7GZL7JFRMWRN2AZUQ, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.471440098 (415h3m6.299148489s) [noise=true , address=145.79.227.47:4266 , piece_id=3NFYV2PWBNTAOSFHENM4J5PZ7XXE72UKTQVXLTVQRHGK4DUXAKDA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.560766475 (415h3m6.388474871s) [upload=431917]: Next returned nil segment Dec 22 22:31:05.560785161 (415h3m6.388493562s) [upload=431917]: Waiting for error group managing segments... Dec 22 22:31:05.566117878 (415h3m6.39382628s) [noise=true , address=45.91.116.4:4939 , piece_id=QYNKMKDLQJV7B6M2E34DU53Y7YI6VBCUW3C7ZLQOI4XWTBXZHKDQ, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.566723306 (415h3m6.394431687s) [noise=true , address=45.91.116.11:4117 , piece_id=WLMO5E3BIFFL3TOAUSSBGORIZ6DOCWFCMVT4AVFPPGPSKP2OKDAA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.566992667 (415h3m6.394701063s) [noise=true , address=145.79.227.36:4519 , piece_id=JUKRRIZWVGKEDD3GPJ7HMY2MEQZNMUCTKTXR6ZLSIERRONTZKHCQ, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.567464626 (415h3m6.395173019s) [noise=true , address=145.79.227.47:4266 , piece_id=3NFYV2PWBNTAOSFHENM4J5PZ7XXE72UKTQVXLTVQRHGK4DUXAKDA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.567924492 (415h3m6.395632883s) [noise=true , address=145.79.227.15:4874 , piece_id=FNCCIMIVJTGZ2X5TOYOZH727JXHYM7HG3HBHBTZDAG67SHZLYYYA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.568807294 (415h3m6.396515685s) [noise=true , address=145.79.227.51:4409 , piece_id=ATH3F4DFOUUIFFJZRA2SSRRL7AKWZMQNQRYOZCVFFDLTFCFP236A, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.570579823 (415h3m6.398288214s) [noise=true , address=145.79.227.48:4301 , piece_id=LBIYTJNOX5J4M3HDE7FGF5CSARBITW2AK3P2DSYFLFKUOIHZUKOQ, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.571632758 (415h3m6.399341149s) [noise=true , address=145.79.227.43:4157 , piece_id=3W3LVTU37DNHWS52IGIIH7SXWXVNPGEY27RQNT272L3Z64GNR4KA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.573168383 (415h3m6.400876774s) [noise=true , address=145.79.227.38:4979 , piece_id=HLMQC5OEB4OL4OZSRMCEVAWR5QM2T4MQUBPTHRVQLEAHPBWPQF6Q, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.579995420 (415h3m6.407703816s) [noise=true , address=45.91.116.14:4190 , piece_id=V4T6Z7K3C75LQEXNQIYZGYWW2CSQ7UPSET46QGDRT52RFGAROTTA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.592093172 (415h3m6.419801563s) [noise=true , address=145.79.227.37:4565 , piece_id=554RHTYPG27UNCFPSRMOB245AX4QXL2QFIYECNTVWNNJMAL4BTYA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:09.946500434 (415h3m10.774208816s) [noise=true , address=145.79.227.40:4051 , piece_id=55RRAD7DQX6SR7RBLYAV7MD7XRWPNIWHTVVBQZ2ZL3OE2L7LRRIQ, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:35.398235543 (415h3m36.225943943s) [noise=true , address=145.79.227.39:4013 , piece_id=JJVURGSLKY2UQ4FKNCLATA3ZBD5SDL32XJAG42KH2U6OYK3KTS6Q, seg_pos={0 0}, upload=431917]: Done uploading piece. err: ecclient: upload failed (node:121scLvKEW9zw2aEz22zpremcjiop67eQ8v2uqpmaD5aZzCAP7b, address:145.79.227.39:4013): protocol: write tcp 159.223.143.39:60842->145.79.227.39:4013: use of closed network connection; write tcp 159.223.143.39:60842->145.79.227.39:4013: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:60842->145.79.227.39:4013: use of closed network connection Dec 22 22:31:35.398562305 (415h3m36.226270699s) [noise=true , address=145.79.227.32:4835 , piece_id=BRE3JH3XKPXB6FN7WB3W6ED7MOFES6HYLR34YGR6L5AX7RPIQ7IA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: ecclient: upload failed (node:16vVfeoqCkbMYCRjMMWaKBnCvQBBJPMgwQtWTq9HKQrD9SS17y, address:145.79.227.32:4835): protocol: write tcp 159.223.143.39:45098->145.79.227.32:4835: use of closed network connection; write tcp 159.223.143.39:45098->145.79.227.32:4835: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:45098->145.79.227.32:4835: use of closed network connection Dec 22 22:31:35.399183028 (415h3m36.226891422s) [noise=true , address=145.79.227.50:4373 , piece_id=4U55T5QDYCE6AZ3NCTPAVECPAELPQUKT3HV7GZL7JFRMWRN2AZUQ, seg_pos={0 0}, upload=431917]: Done uploading piece. err: ecclient: upload failed (node:12AQGSkJuNJDxoeqhChUenWxHwF5yK41UESCF63FQz3XvabXXfz, address:145.79.227.50:4373): protocol: read tcp 159.223.143.39:56028->145.79.227.50:4373: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 22 22:31:35.399278021 (415h3m36.226986417s) [noise=true , address=145.79.227.29:4943 , piece_id=G4TMZ6IAIZBDON3UEDPH4LJQFHV3JLALEXSJ3VI4XN6WHIZOB57Q, seg_pos={0 0}, upload=431917]: Done uploading piece. err: ecclient: upload failed (node:12fc5Trj8XxacZhzxdksUVT5CbG6uTmiwdUPjreYi4UMPY1TXWz, address:145.79.227.29:4943): protocol: write tcp 159.223.143.39:50820->145.79.227.29:4943: use of closed network connection; write tcp 159.223.143.39:50820->145.79.227.29:4943: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:50820->145.79.227.29:4943: use of closed network connection Dec 22 22:31:35.399379485 (415h3m36.227087866s) [seg_pos={0 0}, upload=431917]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 22 22:31:35.421122544 (415h3m36.248830935s) [upload=431917]: Done uploading segments... Dec 22 22:31:35.421174369 (415h3m36.248882769s) [upload=431917]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 446.752203ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...X........................................................] duration: 474.420777ms trace: https://jaeger.dev.storj.io/trace/7013ffb1a981d605 FAIL (2025-12-22 23:13:46 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 22 22:31:05.439510308 (415h3m6.2672187s) [upload=431917]: Starting upload Dec 22 22:31:05.439581993 (415h3m6.26729037s) [upload=431917]: Done starting upload Dec 22 22:31:05.439812792 (415h3m6.267521185s) [upload=431917]: Uploading segments... Dec 22 22:31:05.439942524 (415h3m6.267650918s) [upload=431917]: Got next segment. Inline: false Dec 22 22:31:05.440022744 (415h3m6.267731174s) [upload=431917]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 22 22:31:05.440029670 (415h3m6.267738072s) [upload=431917]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 22 22:31:05.467186372 (415h3m6.294894769s) [seg_pos={0 0}, upload=431917]: Begin upload segment... Dec 22 22:31:05.467265149 (415h3m6.294973535s) [seg_pos={0 0}, upload=431917]: Done begin upload segment. Dec 22 22:31:05.467296974 (415h3m6.295005373s) [noise=true , address=145.79.227.51:4409 , piece_id=ATH3F4DFOUUIFFJZRA2SSRRL7AKWZMQNQRYOZCVFFDLTFCFP236A, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.467389853 (415h3m6.295098247s) [noise=true , address=145.79.227.39:4013 , piece_id=JJVURGSLKY2UQ4FKNCLATA3ZBD5SDL32XJAG42KH2U6OYK3KTS6Q, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.467912104 (415h3m6.295620486s) [noise=true , address=145.79.227.36:4519 , piece_id=JUKRRIZWVGKEDD3GPJ7HMY2MEQZNMUCTKTXR6ZLSIERRONTZKHCQ, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.468324863 (415h3m6.296033263s) [noise=true , address=145.79.227.38:4979 , piece_id=HLMQC5OEB4OL4OZSRMCEVAWR5QM2T4MQUBPTHRVQLEAHPBWPQF6Q, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.468334764 (415h3m6.296043147s) [noise=true , address=145.79.227.40:4051 , piece_id=55RRAD7DQX6SR7RBLYAV7MD7XRWPNIWHTVVBQZ2ZL3OE2L7LRRIQ, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.468742525 (415h3m6.296450916s) [noise=true , address=45.91.116.4:4939 , piece_id=QYNKMKDLQJV7B6M2E34DU53Y7YI6VBCUW3C7ZLQOI4XWTBXZHKDQ, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.468915034 (415h3m6.296623414s) [noise=true , address=45.91.116.11:4117 , piece_id=WLMO5E3BIFFL3TOAUSSBGORIZ6DOCWFCMVT4AVFPPGPSKP2OKDAA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.469069393 (415h3m6.296777787s) [noise=true , address=145.79.227.32:4835 , piece_id=BRE3JH3XKPXB6FN7WB3W6ED7MOFES6HYLR34YGR6L5AX7RPIQ7IA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.469285789 (415h3m6.296994183s) [noise=true , address=145.79.227.29:4943 , piece_id=G4TMZ6IAIZBDON3UEDPH4LJQFHV3JLALEXSJ3VI4XN6WHIZOB57Q, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.469411962 (415h3m6.297120355s) [noise=true , address=145.79.227.48:4301 , piece_id=LBIYTJNOX5J4M3HDE7FGF5CSARBITW2AK3P2DSYFLFKUOIHZUKOQ, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.469749379 (415h3m6.297457773s) [noise=true , address=145.79.227.37:4565 , piece_id=554RHTYPG27UNCFPSRMOB245AX4QXL2QFIYECNTVWNNJMAL4BTYA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.470080828 (415h3m6.29778922s) [noise=true , address=145.79.227.43:4157 , piece_id=3W3LVTU37DNHWS52IGIIH7SXWXVNPGEY27RQNT272L3Z64GNR4KA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.470477660 (415h3m6.298186058s) [noise=true , address=45.91.116.14:4190 , piece_id=V4T6Z7K3C75LQEXNQIYZGYWW2CSQ7UPSET46QGDRT52RFGAROTTA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.470567977 (415h3m6.29827637s) [noise=true , address=145.79.227.15:4874 , piece_id=FNCCIMIVJTGZ2X5TOYOZH727JXHYM7HG3HBHBTZDAG67SHZLYYYA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.470980950 (415h3m6.298689349s) [noise=true , address=145.79.227.50:4373 , piece_id=4U55T5QDYCE6AZ3NCTPAVECPAELPQUKT3HV7GZL7JFRMWRN2AZUQ, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.471440098 (415h3m6.299148489s) [noise=true , address=145.79.227.47:4266 , piece_id=3NFYV2PWBNTAOSFHENM4J5PZ7XXE72UKTQVXLTVQRHGK4DUXAKDA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.560766475 (415h3m6.388474871s) [upload=431917]: Next returned nil segment Dec 22 22:31:05.560785161 (415h3m6.388493562s) [upload=431917]: Waiting for error group managing segments... Dec 22 22:31:05.566117878 (415h3m6.39382628s) [noise=true , address=45.91.116.4:4939 , piece_id=QYNKMKDLQJV7B6M2E34DU53Y7YI6VBCUW3C7ZLQOI4XWTBXZHKDQ, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.566723306 (415h3m6.394431687s) [noise=true , address=45.91.116.11:4117 , piece_id=WLMO5E3BIFFL3TOAUSSBGORIZ6DOCWFCMVT4AVFPPGPSKP2OKDAA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.566992667 (415h3m6.394701063s) [noise=true , address=145.79.227.36:4519 , piece_id=JUKRRIZWVGKEDD3GPJ7HMY2MEQZNMUCTKTXR6ZLSIERRONTZKHCQ, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.567464626 (415h3m6.395173019s) [noise=true , address=145.79.227.47:4266 , piece_id=3NFYV2PWBNTAOSFHENM4J5PZ7XXE72UKTQVXLTVQRHGK4DUXAKDA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.567924492 (415h3m6.395632883s) [noise=true , address=145.79.227.15:4874 , piece_id=FNCCIMIVJTGZ2X5TOYOZH727JXHYM7HG3HBHBTZDAG67SHZLYYYA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.568807294 (415h3m6.396515685s) [noise=true , address=145.79.227.51:4409 , piece_id=ATH3F4DFOUUIFFJZRA2SSRRL7AKWZMQNQRYOZCVFFDLTFCFP236A, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.570579823 (415h3m6.398288214s) [noise=true , address=145.79.227.48:4301 , piece_id=LBIYTJNOX5J4M3HDE7FGF5CSARBITW2AK3P2DSYFLFKUOIHZUKOQ, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.571632758 (415h3m6.399341149s) [noise=true , address=145.79.227.43:4157 , piece_id=3W3LVTU37DNHWS52IGIIH7SXWXVNPGEY27RQNT272L3Z64GNR4KA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.573168383 (415h3m6.400876774s) [noise=true , address=145.79.227.38:4979 , piece_id=HLMQC5OEB4OL4OZSRMCEVAWR5QM2T4MQUBPTHRVQLEAHPBWPQF6Q, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.579995420 (415h3m6.407703816s) [noise=true , address=45.91.116.14:4190 , piece_id=V4T6Z7K3C75LQEXNQIYZGYWW2CSQ7UPSET46QGDRT52RFGAROTTA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.592093172 (415h3m6.419801563s) [noise=true , address=145.79.227.37:4565 , piece_id=554RHTYPG27UNCFPSRMOB245AX4QXL2QFIYECNTVWNNJMAL4BTYA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:09.946500434 (415h3m10.774208816s) [noise=true , address=145.79.227.40:4051 , piece_id=55RRAD7DQX6SR7RBLYAV7MD7XRWPNIWHTVVBQZ2ZL3OE2L7LRRIQ, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:35.398235543 (415h3m36.225943943s) [noise=true , address=145.79.227.39:4013 , piece_id=JJVURGSLKY2UQ4FKNCLATA3ZBD5SDL32XJAG42KH2U6OYK3KTS6Q, seg_pos={0 0}, upload=431917]: Done uploading piece. err: ecclient: upload failed (node:121scLvKEW9zw2aEz22zpremcjiop67eQ8v2uqpmaD5aZzCAP7b, address:145.79.227.39:4013): protocol: write tcp 159.223.143.39:60842->145.79.227.39:4013: use of closed network connection; write tcp 159.223.143.39:60842->145.79.227.39:4013: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:60842->145.79.227.39:4013: use of closed network connection Dec 22 22:31:35.398562305 (415h3m36.226270699s) [noise=true , address=145.79.227.32:4835 , piece_id=BRE3JH3XKPXB6FN7WB3W6ED7MOFES6HYLR34YGR6L5AX7RPIQ7IA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: ecclient: upload failed (node:16vVfeoqCkbMYCRjMMWaKBnCvQBBJPMgwQtWTq9HKQrD9SS17y, address:145.79.227.32:4835): protocol: write tcp 159.223.143.39:45098->145.79.227.32:4835: use of closed network connection; write tcp 159.223.143.39:45098->145.79.227.32:4835: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:45098->145.79.227.32:4835: use of closed network connection Dec 22 22:31:35.399183028 (415h3m36.226891422s) [noise=true , address=145.79.227.50:4373 , piece_id=4U55T5QDYCE6AZ3NCTPAVECPAELPQUKT3HV7GZL7JFRMWRN2AZUQ, seg_pos={0 0}, upload=431917]: Done uploading piece. err: ecclient: upload failed (node:12AQGSkJuNJDxoeqhChUenWxHwF5yK41UESCF63FQz3XvabXXfz, address:145.79.227.50:4373): protocol: read tcp 159.223.143.39:56028->145.79.227.50:4373: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 22 22:31:35.399278021 (415h3m36.226986417s) [noise=true , address=145.79.227.29:4943 , piece_id=G4TMZ6IAIZBDON3UEDPH4LJQFHV3JLALEXSJ3VI4XN6WHIZOB57Q, seg_pos={0 0}, upload=431917]: Done uploading piece. err: ecclient: upload failed (node:12fc5Trj8XxacZhzxdksUVT5CbG6uTmiwdUPjreYi4UMPY1TXWz, address:145.79.227.29:4943): protocol: write tcp 159.223.143.39:50820->145.79.227.29:4943: use of closed network connection; write tcp 159.223.143.39:50820->145.79.227.29:4943: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:50820->145.79.227.29:4943: use of closed network connection Dec 22 22:31:35.399379485 (415h3m36.227087866s) [seg_pos={0 0}, upload=431917]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 22 22:31:35.421122544 (415h3m36.248830935s) [upload=431917]: Done uploading segments... Dec 22 22:31:35.421174369 (415h3m36.248882769s) [upload=431917]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 474.443374ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....X.......................................................] duration: 498.596607ms trace: https://jaeger.dev.storj.io/trace/7013ffb1a981d605 FAIL (2025-12-22 23:13:26 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 22 22:31:05.439510308 (415h3m6.2672187s) [upload=431917]: Starting upload Dec 22 22:31:05.439581993 (415h3m6.26729037s) [upload=431917]: Done starting upload Dec 22 22:31:05.439812792 (415h3m6.267521185s) [upload=431917]: Uploading segments... Dec 22 22:31:05.439942524 (415h3m6.267650918s) [upload=431917]: Got next segment. Inline: false Dec 22 22:31:05.440022744 (415h3m6.267731174s) [upload=431917]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 22 22:31:05.440029670 (415h3m6.267738072s) [upload=431917]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 22 22:31:05.467186372 (415h3m6.294894769s) [seg_pos={0 0}, upload=431917]: Begin upload segment... Dec 22 22:31:05.467265149 (415h3m6.294973535s) [seg_pos={0 0}, upload=431917]: Done begin upload segment. Dec 22 22:31:05.467296974 (415h3m6.295005373s) [noise=true , address=145.79.227.51:4409 , piece_id=ATH3F4DFOUUIFFJZRA2SSRRL7AKWZMQNQRYOZCVFFDLTFCFP236A, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.467389853 (415h3m6.295098247s) [noise=true , address=145.79.227.39:4013 , piece_id=JJVURGSLKY2UQ4FKNCLATA3ZBD5SDL32XJAG42KH2U6OYK3KTS6Q, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.467912104 (415h3m6.295620486s) [noise=true , address=145.79.227.36:4519 , piece_id=JUKRRIZWVGKEDD3GPJ7HMY2MEQZNMUCTKTXR6ZLSIERRONTZKHCQ, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.468324863 (415h3m6.296033263s) [noise=true , address=145.79.227.38:4979 , piece_id=HLMQC5OEB4OL4OZSRMCEVAWR5QM2T4MQUBPTHRVQLEAHPBWPQF6Q, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.468334764 (415h3m6.296043147s) [noise=true , address=145.79.227.40:4051 , piece_id=55RRAD7DQX6SR7RBLYAV7MD7XRWPNIWHTVVBQZ2ZL3OE2L7LRRIQ, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.468742525 (415h3m6.296450916s) [noise=true , address=45.91.116.4:4939 , piece_id=QYNKMKDLQJV7B6M2E34DU53Y7YI6VBCUW3C7ZLQOI4XWTBXZHKDQ, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.468915034 (415h3m6.296623414s) [noise=true , address=45.91.116.11:4117 , piece_id=WLMO5E3BIFFL3TOAUSSBGORIZ6DOCWFCMVT4AVFPPGPSKP2OKDAA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.469069393 (415h3m6.296777787s) [noise=true , address=145.79.227.32:4835 , piece_id=BRE3JH3XKPXB6FN7WB3W6ED7MOFES6HYLR34YGR6L5AX7RPIQ7IA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.469285789 (415h3m6.296994183s) [noise=true , address=145.79.227.29:4943 , piece_id=G4TMZ6IAIZBDON3UEDPH4LJQFHV3JLALEXSJ3VI4XN6WHIZOB57Q, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.469411962 (415h3m6.297120355s) [noise=true , address=145.79.227.48:4301 , piece_id=LBIYTJNOX5J4M3HDE7FGF5CSARBITW2AK3P2DSYFLFKUOIHZUKOQ, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.469749379 (415h3m6.297457773s) [noise=true , address=145.79.227.37:4565 , piece_id=554RHTYPG27UNCFPSRMOB245AX4QXL2QFIYECNTVWNNJMAL4BTYA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.470080828 (415h3m6.29778922s) [noise=true , address=145.79.227.43:4157 , piece_id=3W3LVTU37DNHWS52IGIIH7SXWXVNPGEY27RQNT272L3Z64GNR4KA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.470477660 (415h3m6.298186058s) [noise=true , address=45.91.116.14:4190 , piece_id=V4T6Z7K3C75LQEXNQIYZGYWW2CSQ7UPSET46QGDRT52RFGAROTTA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.470567977 (415h3m6.29827637s) [noise=true , address=145.79.227.15:4874 , piece_id=FNCCIMIVJTGZ2X5TOYOZH727JXHYM7HG3HBHBTZDAG67SHZLYYYA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.470980950 (415h3m6.298689349s) [noise=true , address=145.79.227.50:4373 , piece_id=4U55T5QDYCE6AZ3NCTPAVECPAELPQUKT3HV7GZL7JFRMWRN2AZUQ, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.471440098 (415h3m6.299148489s) [noise=true , address=145.79.227.47:4266 , piece_id=3NFYV2PWBNTAOSFHENM4J5PZ7XXE72UKTQVXLTVQRHGK4DUXAKDA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.560766475 (415h3m6.388474871s) [upload=431917]: Next returned nil segment Dec 22 22:31:05.560785161 (415h3m6.388493562s) [upload=431917]: Waiting for error group managing segments... Dec 22 22:31:05.566117878 (415h3m6.39382628s) [noise=true , address=45.91.116.4:4939 , piece_id=QYNKMKDLQJV7B6M2E34DU53Y7YI6VBCUW3C7ZLQOI4XWTBXZHKDQ, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.566723306 (415h3m6.394431687s) [noise=true , address=45.91.116.11:4117 , piece_id=WLMO5E3BIFFL3TOAUSSBGORIZ6DOCWFCMVT4AVFPPGPSKP2OKDAA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.566992667 (415h3m6.394701063s) [noise=true , address=145.79.227.36:4519 , piece_id=JUKRRIZWVGKEDD3GPJ7HMY2MEQZNMUCTKTXR6ZLSIERRONTZKHCQ, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.567464626 (415h3m6.395173019s) [noise=true , address=145.79.227.47:4266 , piece_id=3NFYV2PWBNTAOSFHENM4J5PZ7XXE72UKTQVXLTVQRHGK4DUXAKDA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.567924492 (415h3m6.395632883s) [noise=true , address=145.79.227.15:4874 , piece_id=FNCCIMIVJTGZ2X5TOYOZH727JXHYM7HG3HBHBTZDAG67SHZLYYYA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.568807294 (415h3m6.396515685s) [noise=true , address=145.79.227.51:4409 , piece_id=ATH3F4DFOUUIFFJZRA2SSRRL7AKWZMQNQRYOZCVFFDLTFCFP236A, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.570579823 (415h3m6.398288214s) [noise=true , address=145.79.227.48:4301 , piece_id=LBIYTJNOX5J4M3HDE7FGF5CSARBITW2AK3P2DSYFLFKUOIHZUKOQ, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.571632758 (415h3m6.399341149s) [noise=true , address=145.79.227.43:4157 , piece_id=3W3LVTU37DNHWS52IGIIH7SXWXVNPGEY27RQNT272L3Z64GNR4KA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.573168383 (415h3m6.400876774s) [noise=true , address=145.79.227.38:4979 , piece_id=HLMQC5OEB4OL4OZSRMCEVAWR5QM2T4MQUBPTHRVQLEAHPBWPQF6Q, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.579995420 (415h3m6.407703816s) [noise=true , address=45.91.116.14:4190 , piece_id=V4T6Z7K3C75LQEXNQIYZGYWW2CSQ7UPSET46QGDRT52RFGAROTTA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.592093172 (415h3m6.419801563s) [noise=true , address=145.79.227.37:4565 , piece_id=554RHTYPG27UNCFPSRMOB245AX4QXL2QFIYECNTVWNNJMAL4BTYA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:09.946500434 (415h3m10.774208816s) [noise=true , address=145.79.227.40:4051 , piece_id=55RRAD7DQX6SR7RBLYAV7MD7XRWPNIWHTVVBQZ2ZL3OE2L7LRRIQ, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:35.398235543 (415h3m36.225943943s) [noise=true , address=145.79.227.39:4013 , piece_id=JJVURGSLKY2UQ4FKNCLATA3ZBD5SDL32XJAG42KH2U6OYK3KTS6Q, seg_pos={0 0}, upload=431917]: Done uploading piece. err: ecclient: upload failed (node:121scLvKEW9zw2aEz22zpremcjiop67eQ8v2uqpmaD5aZzCAP7b, address:145.79.227.39:4013): protocol: write tcp 159.223.143.39:60842->145.79.227.39:4013: use of closed network connection; write tcp 159.223.143.39:60842->145.79.227.39:4013: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:60842->145.79.227.39:4013: use of closed network connection Dec 22 22:31:35.398562305 (415h3m36.226270699s) [noise=true , address=145.79.227.32:4835 , piece_id=BRE3JH3XKPXB6FN7WB3W6ED7MOFES6HYLR34YGR6L5AX7RPIQ7IA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: ecclient: upload failed (node:16vVfeoqCkbMYCRjMMWaKBnCvQBBJPMgwQtWTq9HKQrD9SS17y, address:145.79.227.32:4835): protocol: write tcp 159.223.143.39:45098->145.79.227.32:4835: use of closed network connection; write tcp 159.223.143.39:45098->145.79.227.32:4835: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:45098->145.79.227.32:4835: use of closed network connection Dec 22 22:31:35.399183028 (415h3m36.226891422s) [noise=true , address=145.79.227.50:4373 , piece_id=4U55T5QDYCE6AZ3NCTPAVECPAELPQUKT3HV7GZL7JFRMWRN2AZUQ, seg_pos={0 0}, upload=431917]: Done uploading piece. err: ecclient: upload failed (node:12AQGSkJuNJDxoeqhChUenWxHwF5yK41UESCF63FQz3XvabXXfz, address:145.79.227.50:4373): protocol: read tcp 159.223.143.39:56028->145.79.227.50:4373: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 22 22:31:35.399278021 (415h3m36.226986417s) [noise=true , address=145.79.227.29:4943 , piece_id=G4TMZ6IAIZBDON3UEDPH4LJQFHV3JLALEXSJ3VI4XN6WHIZOB57Q, seg_pos={0 0}, upload=431917]: Done uploading piece. err: ecclient: upload failed (node:12fc5Trj8XxacZhzxdksUVT5CbG6uTmiwdUPjreYi4UMPY1TXWz, address:145.79.227.29:4943): protocol: write tcp 159.223.143.39:50820->145.79.227.29:4943: use of closed network connection; write tcp 159.223.143.39:50820->145.79.227.29:4943: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:50820->145.79.227.29:4943: use of closed network connection Dec 22 22:31:35.399379485 (415h3m36.227087866s) [seg_pos={0 0}, upload=431917]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 22 22:31:35.421122544 (415h3m36.248830935s) [upload=431917]: Done uploading segments... Dec 22 22:31:35.421174369 (415h3m36.248882769s) [upload=431917]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 498.62295ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....X......................................................] duration: 446.015916ms trace: https://jaeger.dev.storj.io/trace/7013ffb1a981d605 FAIL (2025-12-22 23:12:23 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 22 22:31:05.439510308 (415h3m6.2672187s) [upload=431917]: Starting upload Dec 22 22:31:05.439581993 (415h3m6.26729037s) [upload=431917]: Done starting upload Dec 22 22:31:05.439812792 (415h3m6.267521185s) [upload=431917]: Uploading segments... Dec 22 22:31:05.439942524 (415h3m6.267650918s) [upload=431917]: Got next segment. Inline: false Dec 22 22:31:05.440022744 (415h3m6.267731174s) [upload=431917]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 22 22:31:05.440029670 (415h3m6.267738072s) [upload=431917]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 22 22:31:05.467186372 (415h3m6.294894769s) [seg_pos={0 0}, upload=431917]: Begin upload segment... Dec 22 22:31:05.467265149 (415h3m6.294973535s) [seg_pos={0 0}, upload=431917]: Done begin upload segment. Dec 22 22:31:05.467296974 (415h3m6.295005373s) [noise=true , address=145.79.227.51:4409 , piece_id=ATH3F4DFOUUIFFJZRA2SSRRL7AKWZMQNQRYOZCVFFDLTFCFP236A, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.467389853 (415h3m6.295098247s) [noise=true , address=145.79.227.39:4013 , piece_id=JJVURGSLKY2UQ4FKNCLATA3ZBD5SDL32XJAG42KH2U6OYK3KTS6Q, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.467912104 (415h3m6.295620486s) [noise=true , address=145.79.227.36:4519 , piece_id=JUKRRIZWVGKEDD3GPJ7HMY2MEQZNMUCTKTXR6ZLSIERRONTZKHCQ, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.468324863 (415h3m6.296033263s) [noise=true , address=145.79.227.38:4979 , piece_id=HLMQC5OEB4OL4OZSRMCEVAWR5QM2T4MQUBPTHRVQLEAHPBWPQF6Q, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.468334764 (415h3m6.296043147s) [noise=true , address=145.79.227.40:4051 , piece_id=55RRAD7DQX6SR7RBLYAV7MD7XRWPNIWHTVVBQZ2ZL3OE2L7LRRIQ, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.468742525 (415h3m6.296450916s) [noise=true , address=45.91.116.4:4939 , piece_id=QYNKMKDLQJV7B6M2E34DU53Y7YI6VBCUW3C7ZLQOI4XWTBXZHKDQ, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.468915034 (415h3m6.296623414s) [noise=true , address=45.91.116.11:4117 , piece_id=WLMO5E3BIFFL3TOAUSSBGORIZ6DOCWFCMVT4AVFPPGPSKP2OKDAA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.469069393 (415h3m6.296777787s) [noise=true , address=145.79.227.32:4835 , piece_id=BRE3JH3XKPXB6FN7WB3W6ED7MOFES6HYLR34YGR6L5AX7RPIQ7IA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.469285789 (415h3m6.296994183s) [noise=true , address=145.79.227.29:4943 , piece_id=G4TMZ6IAIZBDON3UEDPH4LJQFHV3JLALEXSJ3VI4XN6WHIZOB57Q, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.469411962 (415h3m6.297120355s) [noise=true , address=145.79.227.48:4301 , piece_id=LBIYTJNOX5J4M3HDE7FGF5CSARBITW2AK3P2DSYFLFKUOIHZUKOQ, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.469749379 (415h3m6.297457773s) [noise=true , address=145.79.227.37:4565 , piece_id=554RHTYPG27UNCFPSRMOB245AX4QXL2QFIYECNTVWNNJMAL4BTYA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.470080828 (415h3m6.29778922s) [noise=true , address=145.79.227.43:4157 , piece_id=3W3LVTU37DNHWS52IGIIH7SXWXVNPGEY27RQNT272L3Z64GNR4KA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.470477660 (415h3m6.298186058s) [noise=true , address=45.91.116.14:4190 , piece_id=V4T6Z7K3C75LQEXNQIYZGYWW2CSQ7UPSET46QGDRT52RFGAROTTA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.470567977 (415h3m6.29827637s) [noise=true , address=145.79.227.15:4874 , piece_id=FNCCIMIVJTGZ2X5TOYOZH727JXHYM7HG3HBHBTZDAG67SHZLYYYA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.470980950 (415h3m6.298689349s) [noise=true , address=145.79.227.50:4373 , piece_id=4U55T5QDYCE6AZ3NCTPAVECPAELPQUKT3HV7GZL7JFRMWRN2AZUQ, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.471440098 (415h3m6.299148489s) [noise=true , address=145.79.227.47:4266 , piece_id=3NFYV2PWBNTAOSFHENM4J5PZ7XXE72UKTQVXLTVQRHGK4DUXAKDA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.560766475 (415h3m6.388474871s) [upload=431917]: Next returned nil segment Dec 22 22:31:05.560785161 (415h3m6.388493562s) [upload=431917]: Waiting for error group managing segments... Dec 22 22:31:05.566117878 (415h3m6.39382628s) [noise=true , address=45.91.116.4:4939 , piece_id=QYNKMKDLQJV7B6M2E34DU53Y7YI6VBCUW3C7ZLQOI4XWTBXZHKDQ, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.566723306 (415h3m6.394431687s) [noise=true , address=45.91.116.11:4117 , piece_id=WLMO5E3BIFFL3TOAUSSBGORIZ6DOCWFCMVT4AVFPPGPSKP2OKDAA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.566992667 (415h3m6.394701063s) [noise=true , address=145.79.227.36:4519 , piece_id=JUKRRIZWVGKEDD3GPJ7HMY2MEQZNMUCTKTXR6ZLSIERRONTZKHCQ, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.567464626 (415h3m6.395173019s) [noise=true , address=145.79.227.47:4266 , piece_id=3NFYV2PWBNTAOSFHENM4J5PZ7XXE72UKTQVXLTVQRHGK4DUXAKDA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.567924492 (415h3m6.395632883s) [noise=true , address=145.79.227.15:4874 , piece_id=FNCCIMIVJTGZ2X5TOYOZH727JXHYM7HG3HBHBTZDAG67SHZLYYYA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.568807294 (415h3m6.396515685s) [noise=true , address=145.79.227.51:4409 , piece_id=ATH3F4DFOUUIFFJZRA2SSRRL7AKWZMQNQRYOZCVFFDLTFCFP236A, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.570579823 (415h3m6.398288214s) [noise=true , address=145.79.227.48:4301 , piece_id=LBIYTJNOX5J4M3HDE7FGF5CSARBITW2AK3P2DSYFLFKUOIHZUKOQ, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.571632758 (415h3m6.399341149s) [noise=true , address=145.79.227.43:4157 , piece_id=3W3LVTU37DNHWS52IGIIH7SXWXVNPGEY27RQNT272L3Z64GNR4KA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.573168383 (415h3m6.400876774s) [noise=true , address=145.79.227.38:4979 , piece_id=HLMQC5OEB4OL4OZSRMCEVAWR5QM2T4MQUBPTHRVQLEAHPBWPQF6Q, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.579995420 (415h3m6.407703816s) [noise=true , address=45.91.116.14:4190 , piece_id=V4T6Z7K3C75LQEXNQIYZGYWW2CSQ7UPSET46QGDRT52RFGAROTTA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.592093172 (415h3m6.419801563s) [noise=true , address=145.79.227.37:4565 , piece_id=554RHTYPG27UNCFPSRMOB245AX4QXL2QFIYECNTVWNNJMAL4BTYA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:09.946500434 (415h3m10.774208816s) [noise=true , address=145.79.227.40:4051 , piece_id=55RRAD7DQX6SR7RBLYAV7MD7XRWPNIWHTVVBQZ2ZL3OE2L7LRRIQ, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:35.398235543 (415h3m36.225943943s) [noise=true , address=145.79.227.39:4013 , piece_id=JJVURGSLKY2UQ4FKNCLATA3ZBD5SDL32XJAG42KH2U6OYK3KTS6Q, seg_pos={0 0}, upload=431917]: Done uploading piece. err: ecclient: upload failed (node:121scLvKEW9zw2aEz22zpremcjiop67eQ8v2uqpmaD5aZzCAP7b, address:145.79.227.39:4013): protocol: write tcp 159.223.143.39:60842->145.79.227.39:4013: use of closed network connection; write tcp 159.223.143.39:60842->145.79.227.39:4013: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:60842->145.79.227.39:4013: use of closed network connection Dec 22 22:31:35.398562305 (415h3m36.226270699s) [noise=true , address=145.79.227.32:4835 , piece_id=BRE3JH3XKPXB6FN7WB3W6ED7MOFES6HYLR34YGR6L5AX7RPIQ7IA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: ecclient: upload failed (node:16vVfeoqCkbMYCRjMMWaKBnCvQBBJPMgwQtWTq9HKQrD9SS17y, address:145.79.227.32:4835): protocol: write tcp 159.223.143.39:45098->145.79.227.32:4835: use of closed network connection; write tcp 159.223.143.39:45098->145.79.227.32:4835: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:45098->145.79.227.32:4835: use of closed network connection Dec 22 22:31:35.399183028 (415h3m36.226891422s) [noise=true , address=145.79.227.50:4373 , piece_id=4U55T5QDYCE6AZ3NCTPAVECPAELPQUKT3HV7GZL7JFRMWRN2AZUQ, seg_pos={0 0}, upload=431917]: Done uploading piece. err: ecclient: upload failed (node:12AQGSkJuNJDxoeqhChUenWxHwF5yK41UESCF63FQz3XvabXXfz, address:145.79.227.50:4373): protocol: read tcp 159.223.143.39:56028->145.79.227.50:4373: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 22 22:31:35.399278021 (415h3m36.226986417s) [noise=true , address=145.79.227.29:4943 , piece_id=G4TMZ6IAIZBDON3UEDPH4LJQFHV3JLALEXSJ3VI4XN6WHIZOB57Q, seg_pos={0 0}, upload=431917]: Done uploading piece. err: ecclient: upload failed (node:12fc5Trj8XxacZhzxdksUVT5CbG6uTmiwdUPjreYi4UMPY1TXWz, address:145.79.227.29:4943): protocol: write tcp 159.223.143.39:50820->145.79.227.29:4943: use of closed network connection; write tcp 159.223.143.39:50820->145.79.227.29:4943: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:50820->145.79.227.29:4943: use of closed network connection Dec 22 22:31:35.399379485 (415h3m36.227087866s) [seg_pos={0 0}, upload=431917]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 22 22:31:35.421122544 (415h3m36.248830935s) [upload=431917]: Done uploading segments... Dec 22 22:31:35.421174369 (415h3m36.248882769s) [upload=431917]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 446.040296ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......X.....................................................] duration: 437.384373ms trace: https://jaeger.dev.storj.io/trace/7013ffb1a981d605 FAIL (2025-12-22 23:11:52 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 22 22:31:05.439510308 (415h3m6.2672187s) [upload=431917]: Starting upload Dec 22 22:31:05.439581993 (415h3m6.26729037s) [upload=431917]: Done starting upload Dec 22 22:31:05.439812792 (415h3m6.267521185s) [upload=431917]: Uploading segments... Dec 22 22:31:05.439942524 (415h3m6.267650918s) [upload=431917]: Got next segment. Inline: false Dec 22 22:31:05.440022744 (415h3m6.267731174s) [upload=431917]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 22 22:31:05.440029670 (415h3m6.267738072s) [upload=431917]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 22 22:31:05.467186372 (415h3m6.294894769s) [seg_pos={0 0}, upload=431917]: Begin upload segment... Dec 22 22:31:05.467265149 (415h3m6.294973535s) [seg_pos={0 0}, upload=431917]: Done begin upload segment. Dec 22 22:31:05.467296974 (415h3m6.295005373s) [noise=true , address=145.79.227.51:4409 , piece_id=ATH3F4DFOUUIFFJZRA2SSRRL7AKWZMQNQRYOZCVFFDLTFCFP236A, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.467389853 (415h3m6.295098247s) [noise=true , address=145.79.227.39:4013 , piece_id=JJVURGSLKY2UQ4FKNCLATA3ZBD5SDL32XJAG42KH2U6OYK3KTS6Q, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.467912104 (415h3m6.295620486s) [noise=true , address=145.79.227.36:4519 , piece_id=JUKRRIZWVGKEDD3GPJ7HMY2MEQZNMUCTKTXR6ZLSIERRONTZKHCQ, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.468324863 (415h3m6.296033263s) [noise=true , address=145.79.227.38:4979 , piece_id=HLMQC5OEB4OL4OZSRMCEVAWR5QM2T4MQUBPTHRVQLEAHPBWPQF6Q, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.468334764 (415h3m6.296043147s) [noise=true , address=145.79.227.40:4051 , piece_id=55RRAD7DQX6SR7RBLYAV7MD7XRWPNIWHTVVBQZ2ZL3OE2L7LRRIQ, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.468742525 (415h3m6.296450916s) [noise=true , address=45.91.116.4:4939 , piece_id=QYNKMKDLQJV7B6M2E34DU53Y7YI6VBCUW3C7ZLQOI4XWTBXZHKDQ, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.468915034 (415h3m6.296623414s) [noise=true , address=45.91.116.11:4117 , piece_id=WLMO5E3BIFFL3TOAUSSBGORIZ6DOCWFCMVT4AVFPPGPSKP2OKDAA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.469069393 (415h3m6.296777787s) [noise=true , address=145.79.227.32:4835 , piece_id=BRE3JH3XKPXB6FN7WB3W6ED7MOFES6HYLR34YGR6L5AX7RPIQ7IA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.469285789 (415h3m6.296994183s) [noise=true , address=145.79.227.29:4943 , piece_id=G4TMZ6IAIZBDON3UEDPH4LJQFHV3JLALEXSJ3VI4XN6WHIZOB57Q, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.469411962 (415h3m6.297120355s) [noise=true , address=145.79.227.48:4301 , piece_id=LBIYTJNOX5J4M3HDE7FGF5CSARBITW2AK3P2DSYFLFKUOIHZUKOQ, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.469749379 (415h3m6.297457773s) [noise=true , address=145.79.227.37:4565 , piece_id=554RHTYPG27UNCFPSRMOB245AX4QXL2QFIYECNTVWNNJMAL4BTYA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.470080828 (415h3m6.29778922s) [noise=true , address=145.79.227.43:4157 , piece_id=3W3LVTU37DNHWS52IGIIH7SXWXVNPGEY27RQNT272L3Z64GNR4KA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.470477660 (415h3m6.298186058s) [noise=true , address=45.91.116.14:4190 , piece_id=V4T6Z7K3C75LQEXNQIYZGYWW2CSQ7UPSET46QGDRT52RFGAROTTA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.470567977 (415h3m6.29827637s) [noise=true , address=145.79.227.15:4874 , piece_id=FNCCIMIVJTGZ2X5TOYOZH727JXHYM7HG3HBHBTZDAG67SHZLYYYA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.470980950 (415h3m6.298689349s) [noise=true , address=145.79.227.50:4373 , piece_id=4U55T5QDYCE6AZ3NCTPAVECPAELPQUKT3HV7GZL7JFRMWRN2AZUQ, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.471440098 (415h3m6.299148489s) [noise=true , address=145.79.227.47:4266 , piece_id=3NFYV2PWBNTAOSFHENM4J5PZ7XXE72UKTQVXLTVQRHGK4DUXAKDA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.560766475 (415h3m6.388474871s) [upload=431917]: Next returned nil segment Dec 22 22:31:05.560785161 (415h3m6.388493562s) [upload=431917]: Waiting for error group managing segments... Dec 22 22:31:05.566117878 (415h3m6.39382628s) [noise=true , address=45.91.116.4:4939 , piece_id=QYNKMKDLQJV7B6M2E34DU53Y7YI6VBCUW3C7ZLQOI4XWTBXZHKDQ, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.566723306 (415h3m6.394431687s) [noise=true , address=45.91.116.11:4117 , piece_id=WLMO5E3BIFFL3TOAUSSBGORIZ6DOCWFCMVT4AVFPPGPSKP2OKDAA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.566992667 (415h3m6.394701063s) [noise=true , address=145.79.227.36:4519 , piece_id=JUKRRIZWVGKEDD3GPJ7HMY2MEQZNMUCTKTXR6ZLSIERRONTZKHCQ, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.567464626 (415h3m6.395173019s) [noise=true , address=145.79.227.47:4266 , piece_id=3NFYV2PWBNTAOSFHENM4J5PZ7XXE72UKTQVXLTVQRHGK4DUXAKDA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.567924492 (415h3m6.395632883s) [noise=true , address=145.79.227.15:4874 , piece_id=FNCCIMIVJTGZ2X5TOYOZH727JXHYM7HG3HBHBTZDAG67SHZLYYYA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.568807294 (415h3m6.396515685s) [noise=true , address=145.79.227.51:4409 , piece_id=ATH3F4DFOUUIFFJZRA2SSRRL7AKWZMQNQRYOZCVFFDLTFCFP236A, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.570579823 (415h3m6.398288214s) [noise=true , address=145.79.227.48:4301 , piece_id=LBIYTJNOX5J4M3HDE7FGF5CSARBITW2AK3P2DSYFLFKUOIHZUKOQ, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.571632758 (415h3m6.399341149s) [noise=true , address=145.79.227.43:4157 , piece_id=3W3LVTU37DNHWS52IGIIH7SXWXVNPGEY27RQNT272L3Z64GNR4KA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.573168383 (415h3m6.400876774s) [noise=true , address=145.79.227.38:4979 , piece_id=HLMQC5OEB4OL4OZSRMCEVAWR5QM2T4MQUBPTHRVQLEAHPBWPQF6Q, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.579995420 (415h3m6.407703816s) [noise=true , address=45.91.116.14:4190 , piece_id=V4T6Z7K3C75LQEXNQIYZGYWW2CSQ7UPSET46QGDRT52RFGAROTTA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.592093172 (415h3m6.419801563s) [noise=true , address=145.79.227.37:4565 , piece_id=554RHTYPG27UNCFPSRMOB245AX4QXL2QFIYECNTVWNNJMAL4BTYA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:09.946500434 (415h3m10.774208816s) [noise=true , address=145.79.227.40:4051 , piece_id=55RRAD7DQX6SR7RBLYAV7MD7XRWPNIWHTVVBQZ2ZL3OE2L7LRRIQ, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:35.398235543 (415h3m36.225943943s) [noise=true , address=145.79.227.39:4013 , piece_id=JJVURGSLKY2UQ4FKNCLATA3ZBD5SDL32XJAG42KH2U6OYK3KTS6Q, seg_pos={0 0}, upload=431917]: Done uploading piece. err: ecclient: upload failed (node:121scLvKEW9zw2aEz22zpremcjiop67eQ8v2uqpmaD5aZzCAP7b, address:145.79.227.39:4013): protocol: write tcp 159.223.143.39:60842->145.79.227.39:4013: use of closed network connection; write tcp 159.223.143.39:60842->145.79.227.39:4013: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:60842->145.79.227.39:4013: use of closed network connection Dec 22 22:31:35.398562305 (415h3m36.226270699s) [noise=true , address=145.79.227.32:4835 , piece_id=BRE3JH3XKPXB6FN7WB3W6ED7MOFES6HYLR34YGR6L5AX7RPIQ7IA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: ecclient: upload failed (node:16vVfeoqCkbMYCRjMMWaKBnCvQBBJPMgwQtWTq9HKQrD9SS17y, address:145.79.227.32:4835): protocol: write tcp 159.223.143.39:45098->145.79.227.32:4835: use of closed network connection; write tcp 159.223.143.39:45098->145.79.227.32:4835: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:45098->145.79.227.32:4835: use of closed network connection Dec 22 22:31:35.399183028 (415h3m36.226891422s) [noise=true , address=145.79.227.50:4373 , piece_id=4U55T5QDYCE6AZ3NCTPAVECPAELPQUKT3HV7GZL7JFRMWRN2AZUQ, seg_pos={0 0}, upload=431917]: Done uploading piece. err: ecclient: upload failed (node:12AQGSkJuNJDxoeqhChUenWxHwF5yK41UESCF63FQz3XvabXXfz, address:145.79.227.50:4373): protocol: read tcp 159.223.143.39:56028->145.79.227.50:4373: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 22 22:31:35.399278021 (415h3m36.226986417s) [noise=true , address=145.79.227.29:4943 , piece_id=G4TMZ6IAIZBDON3UEDPH4LJQFHV3JLALEXSJ3VI4XN6WHIZOB57Q, seg_pos={0 0}, upload=431917]: Done uploading piece. err: ecclient: upload failed (node:12fc5Trj8XxacZhzxdksUVT5CbG6uTmiwdUPjreYi4UMPY1TXWz, address:145.79.227.29:4943): protocol: write tcp 159.223.143.39:50820->145.79.227.29:4943: use of closed network connection; write tcp 159.223.143.39:50820->145.79.227.29:4943: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:50820->145.79.227.29:4943: use of closed network connection Dec 22 22:31:35.399379485 (415h3m36.227087866s) [seg_pos={0 0}, upload=431917]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 22 22:31:35.421122544 (415h3m36.248830935s) [upload=431917]: Done uploading segments... Dec 22 22:31:35.421174369 (415h3m36.248882769s) [upload=431917]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 437.415174ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......X....................................................] duration: 473.987509ms trace: https://jaeger.dev.storj.io/trace/7013ffb1a981d605 FAIL (2025-12-22 23:11:15 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 22 22:31:05.439510308 (415h3m6.2672187s) [upload=431917]: Starting upload Dec 22 22:31:05.439581993 (415h3m6.26729037s) [upload=431917]: Done starting upload Dec 22 22:31:05.439812792 (415h3m6.267521185s) [upload=431917]: Uploading segments... Dec 22 22:31:05.439942524 (415h3m6.267650918s) [upload=431917]: Got next segment. Inline: false Dec 22 22:31:05.440022744 (415h3m6.267731174s) [upload=431917]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 22 22:31:05.440029670 (415h3m6.267738072s) [upload=431917]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 22 22:31:05.467186372 (415h3m6.294894769s) [seg_pos={0 0}, upload=431917]: Begin upload segment... Dec 22 22:31:05.467265149 (415h3m6.294973535s) [seg_pos={0 0}, upload=431917]: Done begin upload segment. Dec 22 22:31:05.467296974 (415h3m6.295005373s) [noise=true , address=145.79.227.51:4409 , piece_id=ATH3F4DFOUUIFFJZRA2SSRRL7AKWZMQNQRYOZCVFFDLTFCFP236A, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.467389853 (415h3m6.295098247s) [noise=true , address=145.79.227.39:4013 , piece_id=JJVURGSLKY2UQ4FKNCLATA3ZBD5SDL32XJAG42KH2U6OYK3KTS6Q, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.467912104 (415h3m6.295620486s) [noise=true , address=145.79.227.36:4519 , piece_id=JUKRRIZWVGKEDD3GPJ7HMY2MEQZNMUCTKTXR6ZLSIERRONTZKHCQ, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.468324863 (415h3m6.296033263s) [noise=true , address=145.79.227.38:4979 , piece_id=HLMQC5OEB4OL4OZSRMCEVAWR5QM2T4MQUBPTHRVQLEAHPBWPQF6Q, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.468334764 (415h3m6.296043147s) [noise=true , address=145.79.227.40:4051 , piece_id=55RRAD7DQX6SR7RBLYAV7MD7XRWPNIWHTVVBQZ2ZL3OE2L7LRRIQ, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.468742525 (415h3m6.296450916s) [noise=true , address=45.91.116.4:4939 , piece_id=QYNKMKDLQJV7B6M2E34DU53Y7YI6VBCUW3C7ZLQOI4XWTBXZHKDQ, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.468915034 (415h3m6.296623414s) [noise=true , address=45.91.116.11:4117 , piece_id=WLMO5E3BIFFL3TOAUSSBGORIZ6DOCWFCMVT4AVFPPGPSKP2OKDAA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.469069393 (415h3m6.296777787s) [noise=true , address=145.79.227.32:4835 , piece_id=BRE3JH3XKPXB6FN7WB3W6ED7MOFES6HYLR34YGR6L5AX7RPIQ7IA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.469285789 (415h3m6.296994183s) [noise=true , address=145.79.227.29:4943 , piece_id=G4TMZ6IAIZBDON3UEDPH4LJQFHV3JLALEXSJ3VI4XN6WHIZOB57Q, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.469411962 (415h3m6.297120355s) [noise=true , address=145.79.227.48:4301 , piece_id=LBIYTJNOX5J4M3HDE7FGF5CSARBITW2AK3P2DSYFLFKUOIHZUKOQ, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.469749379 (415h3m6.297457773s) [noise=true , address=145.79.227.37:4565 , piece_id=554RHTYPG27UNCFPSRMOB245AX4QXL2QFIYECNTVWNNJMAL4BTYA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.470080828 (415h3m6.29778922s) [noise=true , address=145.79.227.43:4157 , piece_id=3W3LVTU37DNHWS52IGIIH7SXWXVNPGEY27RQNT272L3Z64GNR4KA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.470477660 (415h3m6.298186058s) [noise=true , address=45.91.116.14:4190 , piece_id=V4T6Z7K3C75LQEXNQIYZGYWW2CSQ7UPSET46QGDRT52RFGAROTTA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.470567977 (415h3m6.29827637s) [noise=true , address=145.79.227.15:4874 , piece_id=FNCCIMIVJTGZ2X5TOYOZH727JXHYM7HG3HBHBTZDAG67SHZLYYYA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.470980950 (415h3m6.298689349s) [noise=true , address=145.79.227.50:4373 , piece_id=4U55T5QDYCE6AZ3NCTPAVECPAELPQUKT3HV7GZL7JFRMWRN2AZUQ, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.471440098 (415h3m6.299148489s) [noise=true , address=145.79.227.47:4266 , piece_id=3NFYV2PWBNTAOSFHENM4J5PZ7XXE72UKTQVXLTVQRHGK4DUXAKDA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.560766475 (415h3m6.388474871s) [upload=431917]: Next returned nil segment Dec 22 22:31:05.560785161 (415h3m6.388493562s) [upload=431917]: Waiting for error group managing segments... Dec 22 22:31:05.566117878 (415h3m6.39382628s) [noise=true , address=45.91.116.4:4939 , piece_id=QYNKMKDLQJV7B6M2E34DU53Y7YI6VBCUW3C7ZLQOI4XWTBXZHKDQ, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.566723306 (415h3m6.394431687s) [noise=true , address=45.91.116.11:4117 , piece_id=WLMO5E3BIFFL3TOAUSSBGORIZ6DOCWFCMVT4AVFPPGPSKP2OKDAA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.566992667 (415h3m6.394701063s) [noise=true , address=145.79.227.36:4519 , piece_id=JUKRRIZWVGKEDD3GPJ7HMY2MEQZNMUCTKTXR6ZLSIERRONTZKHCQ, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.567464626 (415h3m6.395173019s) [noise=true , address=145.79.227.47:4266 , piece_id=3NFYV2PWBNTAOSFHENM4J5PZ7XXE72UKTQVXLTVQRHGK4DUXAKDA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.567924492 (415h3m6.395632883s) [noise=true , address=145.79.227.15:4874 , piece_id=FNCCIMIVJTGZ2X5TOYOZH727JXHYM7HG3HBHBTZDAG67SHZLYYYA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.568807294 (415h3m6.396515685s) [noise=true , address=145.79.227.51:4409 , piece_id=ATH3F4DFOUUIFFJZRA2SSRRL7AKWZMQNQRYOZCVFFDLTFCFP236A, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.570579823 (415h3m6.398288214s) [noise=true , address=145.79.227.48:4301 , piece_id=LBIYTJNOX5J4M3HDE7FGF5CSARBITW2AK3P2DSYFLFKUOIHZUKOQ, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.571632758 (415h3m6.399341149s) [noise=true , address=145.79.227.43:4157 , piece_id=3W3LVTU37DNHWS52IGIIH7SXWXVNPGEY27RQNT272L3Z64GNR4KA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.573168383 (415h3m6.400876774s) [noise=true , address=145.79.227.38:4979 , piece_id=HLMQC5OEB4OL4OZSRMCEVAWR5QM2T4MQUBPTHRVQLEAHPBWPQF6Q, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.579995420 (415h3m6.407703816s) [noise=true , address=45.91.116.14:4190 , piece_id=V4T6Z7K3C75LQEXNQIYZGYWW2CSQ7UPSET46QGDRT52RFGAROTTA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.592093172 (415h3m6.419801563s) [noise=true , address=145.79.227.37:4565 , piece_id=554RHTYPG27UNCFPSRMOB245AX4QXL2QFIYECNTVWNNJMAL4BTYA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:09.946500434 (415h3m10.774208816s) [noise=true , address=145.79.227.40:4051 , piece_id=55RRAD7DQX6SR7RBLYAV7MD7XRWPNIWHTVVBQZ2ZL3OE2L7LRRIQ, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:35.398235543 (415h3m36.225943943s) [noise=true , address=145.79.227.39:4013 , piece_id=JJVURGSLKY2UQ4FKNCLATA3ZBD5SDL32XJAG42KH2U6OYK3KTS6Q, seg_pos={0 0}, upload=431917]: Done uploading piece. err: ecclient: upload failed (node:121scLvKEW9zw2aEz22zpremcjiop67eQ8v2uqpmaD5aZzCAP7b, address:145.79.227.39:4013): protocol: write tcp 159.223.143.39:60842->145.79.227.39:4013: use of closed network connection; write tcp 159.223.143.39:60842->145.79.227.39:4013: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:60842->145.79.227.39:4013: use of closed network connection Dec 22 22:31:35.398562305 (415h3m36.226270699s) [noise=true , address=145.79.227.32:4835 , piece_id=BRE3JH3XKPXB6FN7WB3W6ED7MOFES6HYLR34YGR6L5AX7RPIQ7IA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: ecclient: upload failed (node:16vVfeoqCkbMYCRjMMWaKBnCvQBBJPMgwQtWTq9HKQrD9SS17y, address:145.79.227.32:4835): protocol: write tcp 159.223.143.39:45098->145.79.227.32:4835: use of closed network connection; write tcp 159.223.143.39:45098->145.79.227.32:4835: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:45098->145.79.227.32:4835: use of closed network connection Dec 22 22:31:35.399183028 (415h3m36.226891422s) [noise=true , address=145.79.227.50:4373 , piece_id=4U55T5QDYCE6AZ3NCTPAVECPAELPQUKT3HV7GZL7JFRMWRN2AZUQ, seg_pos={0 0}, upload=431917]: Done uploading piece. err: ecclient: upload failed (node:12AQGSkJuNJDxoeqhChUenWxHwF5yK41UESCF63FQz3XvabXXfz, address:145.79.227.50:4373): protocol: read tcp 159.223.143.39:56028->145.79.227.50:4373: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 22 22:31:35.399278021 (415h3m36.226986417s) [noise=true , address=145.79.227.29:4943 , piece_id=G4TMZ6IAIZBDON3UEDPH4LJQFHV3JLALEXSJ3VI4XN6WHIZOB57Q, seg_pos={0 0}, upload=431917]: Done uploading piece. err: ecclient: upload failed (node:12fc5Trj8XxacZhzxdksUVT5CbG6uTmiwdUPjreYi4UMPY1TXWz, address:145.79.227.29:4943): protocol: write tcp 159.223.143.39:50820->145.79.227.29:4943: use of closed network connection; write tcp 159.223.143.39:50820->145.79.227.29:4943: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:50820->145.79.227.29:4943: use of closed network connection Dec 22 22:31:35.399379485 (415h3m36.227087866s) [seg_pos={0 0}, upload=431917]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 22 22:31:35.421122544 (415h3m36.248830935s) [upload=431917]: Done uploading segments... Dec 22 22:31:35.421174369 (415h3m36.248882769s) [upload=431917]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 474.013015ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........X...................................................] duration: 623.891876ms trace: https://jaeger.dev.storj.io/trace/7013ffb1a981d605 FAIL (2025-12-22 23:10:06 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 22 22:31:05.439510308 (415h3m6.2672187s) [upload=431917]: Starting upload Dec 22 22:31:05.439581993 (415h3m6.26729037s) [upload=431917]: Done starting upload Dec 22 22:31:05.439812792 (415h3m6.267521185s) [upload=431917]: Uploading segments... Dec 22 22:31:05.439942524 (415h3m6.267650918s) [upload=431917]: Got next segment. Inline: false Dec 22 22:31:05.440022744 (415h3m6.267731174s) [upload=431917]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 22 22:31:05.440029670 (415h3m6.267738072s) [upload=431917]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 22 22:31:05.467186372 (415h3m6.294894769s) [seg_pos={0 0}, upload=431917]: Begin upload segment... Dec 22 22:31:05.467265149 (415h3m6.294973535s) [seg_pos={0 0}, upload=431917]: Done begin upload segment. Dec 22 22:31:05.467296974 (415h3m6.295005373s) [noise=true , address=145.79.227.51:4409 , piece_id=ATH3F4DFOUUIFFJZRA2SSRRL7AKWZMQNQRYOZCVFFDLTFCFP236A, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.467389853 (415h3m6.295098247s) [noise=true , address=145.79.227.39:4013 , piece_id=JJVURGSLKY2UQ4FKNCLATA3ZBD5SDL32XJAG42KH2U6OYK3KTS6Q, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.467912104 (415h3m6.295620486s) [noise=true , address=145.79.227.36:4519 , piece_id=JUKRRIZWVGKEDD3GPJ7HMY2MEQZNMUCTKTXR6ZLSIERRONTZKHCQ, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.468324863 (415h3m6.296033263s) [noise=true , address=145.79.227.38:4979 , piece_id=HLMQC5OEB4OL4OZSRMCEVAWR5QM2T4MQUBPTHRVQLEAHPBWPQF6Q, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.468334764 (415h3m6.296043147s) [noise=true , address=145.79.227.40:4051 , piece_id=55RRAD7DQX6SR7RBLYAV7MD7XRWPNIWHTVVBQZ2ZL3OE2L7LRRIQ, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.468742525 (415h3m6.296450916s) [noise=true , address=45.91.116.4:4939 , piece_id=QYNKMKDLQJV7B6M2E34DU53Y7YI6VBCUW3C7ZLQOI4XWTBXZHKDQ, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.468915034 (415h3m6.296623414s) [noise=true , address=45.91.116.11:4117 , piece_id=WLMO5E3BIFFL3TOAUSSBGORIZ6DOCWFCMVT4AVFPPGPSKP2OKDAA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.469069393 (415h3m6.296777787s) [noise=true , address=145.79.227.32:4835 , piece_id=BRE3JH3XKPXB6FN7WB3W6ED7MOFES6HYLR34YGR6L5AX7RPIQ7IA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.469285789 (415h3m6.296994183s) [noise=true , address=145.79.227.29:4943 , piece_id=G4TMZ6IAIZBDON3UEDPH4LJQFHV3JLALEXSJ3VI4XN6WHIZOB57Q, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.469411962 (415h3m6.297120355s) [noise=true , address=145.79.227.48:4301 , piece_id=LBIYTJNOX5J4M3HDE7FGF5CSARBITW2AK3P2DSYFLFKUOIHZUKOQ, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.469749379 (415h3m6.297457773s) [noise=true , address=145.79.227.37:4565 , piece_id=554RHTYPG27UNCFPSRMOB245AX4QXL2QFIYECNTVWNNJMAL4BTYA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.470080828 (415h3m6.29778922s) [noise=true , address=145.79.227.43:4157 , piece_id=3W3LVTU37DNHWS52IGIIH7SXWXVNPGEY27RQNT272L3Z64GNR4KA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.470477660 (415h3m6.298186058s) [noise=true , address=45.91.116.14:4190 , piece_id=V4T6Z7K3C75LQEXNQIYZGYWW2CSQ7UPSET46QGDRT52RFGAROTTA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.470567977 (415h3m6.29827637s) [noise=true , address=145.79.227.15:4874 , piece_id=FNCCIMIVJTGZ2X5TOYOZH727JXHYM7HG3HBHBTZDAG67SHZLYYYA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.470980950 (415h3m6.298689349s) [noise=true , address=145.79.227.50:4373 , piece_id=4U55T5QDYCE6AZ3NCTPAVECPAELPQUKT3HV7GZL7JFRMWRN2AZUQ, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.471440098 (415h3m6.299148489s) [noise=true , address=145.79.227.47:4266 , piece_id=3NFYV2PWBNTAOSFHENM4J5PZ7XXE72UKTQVXLTVQRHGK4DUXAKDA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.560766475 (415h3m6.388474871s) [upload=431917]: Next returned nil segment Dec 22 22:31:05.560785161 (415h3m6.388493562s) [upload=431917]: Waiting for error group managing segments... Dec 22 22:31:05.566117878 (415h3m6.39382628s) [noise=true , address=45.91.116.4:4939 , piece_id=QYNKMKDLQJV7B6M2E34DU53Y7YI6VBCUW3C7ZLQOI4XWTBXZHKDQ, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.566723306 (415h3m6.394431687s) [noise=true , address=45.91.116.11:4117 , piece_id=WLMO5E3BIFFL3TOAUSSBGORIZ6DOCWFCMVT4AVFPPGPSKP2OKDAA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.566992667 (415h3m6.394701063s) [noise=true , address=145.79.227.36:4519 , piece_id=JUKRRIZWVGKEDD3GPJ7HMY2MEQZNMUCTKTXR6ZLSIERRONTZKHCQ, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.567464626 (415h3m6.395173019s) [noise=true , address=145.79.227.47:4266 , piece_id=3NFYV2PWBNTAOSFHENM4J5PZ7XXE72UKTQVXLTVQRHGK4DUXAKDA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.567924492 (415h3m6.395632883s) [noise=true , address=145.79.227.15:4874 , piece_id=FNCCIMIVJTGZ2X5TOYOZH727JXHYM7HG3HBHBTZDAG67SHZLYYYA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.568807294 (415h3m6.396515685s) [noise=true , address=145.79.227.51:4409 , piece_id=ATH3F4DFOUUIFFJZRA2SSRRL7AKWZMQNQRYOZCVFFDLTFCFP236A, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.570579823 (415h3m6.398288214s) [noise=true , address=145.79.227.48:4301 , piece_id=LBIYTJNOX5J4M3HDE7FGF5CSARBITW2AK3P2DSYFLFKUOIHZUKOQ, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.571632758 (415h3m6.399341149s) [noise=true , address=145.79.227.43:4157 , piece_id=3W3LVTU37DNHWS52IGIIH7SXWXVNPGEY27RQNT272L3Z64GNR4KA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.573168383 (415h3m6.400876774s) [noise=true , address=145.79.227.38:4979 , piece_id=HLMQC5OEB4OL4OZSRMCEVAWR5QM2T4MQUBPTHRVQLEAHPBWPQF6Q, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.579995420 (415h3m6.407703816s) [noise=true , address=45.91.116.14:4190 , piece_id=V4T6Z7K3C75LQEXNQIYZGYWW2CSQ7UPSET46QGDRT52RFGAROTTA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.592093172 (415h3m6.419801563s) [noise=true , address=145.79.227.37:4565 , piece_id=554RHTYPG27UNCFPSRMOB245AX4QXL2QFIYECNTVWNNJMAL4BTYA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:09.946500434 (415h3m10.774208816s) [noise=true , address=145.79.227.40:4051 , piece_id=55RRAD7DQX6SR7RBLYAV7MD7XRWPNIWHTVVBQZ2ZL3OE2L7LRRIQ, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:35.398235543 (415h3m36.225943943s) [noise=true , address=145.79.227.39:4013 , piece_id=JJVURGSLKY2UQ4FKNCLATA3ZBD5SDL32XJAG42KH2U6OYK3KTS6Q, seg_pos={0 0}, upload=431917]: Done uploading piece. err: ecclient: upload failed (node:121scLvKEW9zw2aEz22zpremcjiop67eQ8v2uqpmaD5aZzCAP7b, address:145.79.227.39:4013): protocol: write tcp 159.223.143.39:60842->145.79.227.39:4013: use of closed network connection; write tcp 159.223.143.39:60842->145.79.227.39:4013: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:60842->145.79.227.39:4013: use of closed network connection Dec 22 22:31:35.398562305 (415h3m36.226270699s) [noise=true , address=145.79.227.32:4835 , piece_id=BRE3JH3XKPXB6FN7WB3W6ED7MOFES6HYLR34YGR6L5AX7RPIQ7IA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: ecclient: upload failed (node:16vVfeoqCkbMYCRjMMWaKBnCvQBBJPMgwQtWTq9HKQrD9SS17y, address:145.79.227.32:4835): protocol: write tcp 159.223.143.39:45098->145.79.227.32:4835: use of closed network connection; write tcp 159.223.143.39:45098->145.79.227.32:4835: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:45098->145.79.227.32:4835: use of closed network connection Dec 22 22:31:35.399183028 (415h3m36.226891422s) [noise=true , address=145.79.227.50:4373 , piece_id=4U55T5QDYCE6AZ3NCTPAVECPAELPQUKT3HV7GZL7JFRMWRN2AZUQ, seg_pos={0 0}, upload=431917]: Done uploading piece. err: ecclient: upload failed (node:12AQGSkJuNJDxoeqhChUenWxHwF5yK41UESCF63FQz3XvabXXfz, address:145.79.227.50:4373): protocol: read tcp 159.223.143.39:56028->145.79.227.50:4373: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 22 22:31:35.399278021 (415h3m36.226986417s) [noise=true , address=145.79.227.29:4943 , piece_id=G4TMZ6IAIZBDON3UEDPH4LJQFHV3JLALEXSJ3VI4XN6WHIZOB57Q, seg_pos={0 0}, upload=431917]: Done uploading piece. err: ecclient: upload failed (node:12fc5Trj8XxacZhzxdksUVT5CbG6uTmiwdUPjreYi4UMPY1TXWz, address:145.79.227.29:4943): protocol: write tcp 159.223.143.39:50820->145.79.227.29:4943: use of closed network connection; write tcp 159.223.143.39:50820->145.79.227.29:4943: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:50820->145.79.227.29:4943: use of closed network connection Dec 22 22:31:35.399379485 (415h3m36.227087866s) [seg_pos={0 0}, upload=431917]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 22 22:31:35.421122544 (415h3m36.248830935s) [upload=431917]: Done uploading segments... Dec 22 22:31:35.421174369 (415h3m36.248882769s) [upload=431917]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 623.92638ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........X..................................................] duration: 417.557092ms trace: https://jaeger.dev.storj.io/trace/7013ffb1a981d605 FAIL (2025-12-22 23:09:35 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 22 22:31:05.439510308 (415h3m6.2672187s) [upload=431917]: Starting upload Dec 22 22:31:05.439581993 (415h3m6.26729037s) [upload=431917]: Done starting upload Dec 22 22:31:05.439812792 (415h3m6.267521185s) [upload=431917]: Uploading segments... Dec 22 22:31:05.439942524 (415h3m6.267650918s) [upload=431917]: Got next segment. Inline: false Dec 22 22:31:05.440022744 (415h3m6.267731174s) [upload=431917]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 22 22:31:05.440029670 (415h3m6.267738072s) [upload=431917]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 22 22:31:05.467186372 (415h3m6.294894769s) [seg_pos={0 0}, upload=431917]: Begin upload segment... Dec 22 22:31:05.467265149 (415h3m6.294973535s) [seg_pos={0 0}, upload=431917]: Done begin upload segment. Dec 22 22:31:05.467296974 (415h3m6.295005373s) [noise=true , address=145.79.227.51:4409 , piece_id=ATH3F4DFOUUIFFJZRA2SSRRL7AKWZMQNQRYOZCVFFDLTFCFP236A, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.467389853 (415h3m6.295098247s) [noise=true , address=145.79.227.39:4013 , piece_id=JJVURGSLKY2UQ4FKNCLATA3ZBD5SDL32XJAG42KH2U6OYK3KTS6Q, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.467912104 (415h3m6.295620486s) [noise=true , address=145.79.227.36:4519 , piece_id=JUKRRIZWVGKEDD3GPJ7HMY2MEQZNMUCTKTXR6ZLSIERRONTZKHCQ, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.468324863 (415h3m6.296033263s) [noise=true , address=145.79.227.38:4979 , piece_id=HLMQC5OEB4OL4OZSRMCEVAWR5QM2T4MQUBPTHRVQLEAHPBWPQF6Q, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.468334764 (415h3m6.296043147s) [noise=true , address=145.79.227.40:4051 , piece_id=55RRAD7DQX6SR7RBLYAV7MD7XRWPNIWHTVVBQZ2ZL3OE2L7LRRIQ, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.468742525 (415h3m6.296450916s) [noise=true , address=45.91.116.4:4939 , piece_id=QYNKMKDLQJV7B6M2E34DU53Y7YI6VBCUW3C7ZLQOI4XWTBXZHKDQ, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.468915034 (415h3m6.296623414s) [noise=true , address=45.91.116.11:4117 , piece_id=WLMO5E3BIFFL3TOAUSSBGORIZ6DOCWFCMVT4AVFPPGPSKP2OKDAA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.469069393 (415h3m6.296777787s) [noise=true , address=145.79.227.32:4835 , piece_id=BRE3JH3XKPXB6FN7WB3W6ED7MOFES6HYLR34YGR6L5AX7RPIQ7IA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.469285789 (415h3m6.296994183s) [noise=true , address=145.79.227.29:4943 , piece_id=G4TMZ6IAIZBDON3UEDPH4LJQFHV3JLALEXSJ3VI4XN6WHIZOB57Q, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.469411962 (415h3m6.297120355s) [noise=true , address=145.79.227.48:4301 , piece_id=LBIYTJNOX5J4M3HDE7FGF5CSARBITW2AK3P2DSYFLFKUOIHZUKOQ, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.469749379 (415h3m6.297457773s) [noise=true , address=145.79.227.37:4565 , piece_id=554RHTYPG27UNCFPSRMOB245AX4QXL2QFIYECNTVWNNJMAL4BTYA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.470080828 (415h3m6.29778922s) [noise=true , address=145.79.227.43:4157 , piece_id=3W3LVTU37DNHWS52IGIIH7SXWXVNPGEY27RQNT272L3Z64GNR4KA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.470477660 (415h3m6.298186058s) [noise=true , address=45.91.116.14:4190 , piece_id=V4T6Z7K3C75LQEXNQIYZGYWW2CSQ7UPSET46QGDRT52RFGAROTTA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.470567977 (415h3m6.29827637s) [noise=true , address=145.79.227.15:4874 , piece_id=FNCCIMIVJTGZ2X5TOYOZH727JXHYM7HG3HBHBTZDAG67SHZLYYYA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.470980950 (415h3m6.298689349s) [noise=true , address=145.79.227.50:4373 , piece_id=4U55T5QDYCE6AZ3NCTPAVECPAELPQUKT3HV7GZL7JFRMWRN2AZUQ, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.471440098 (415h3m6.299148489s) [noise=true , address=145.79.227.47:4266 , piece_id=3NFYV2PWBNTAOSFHENM4J5PZ7XXE72UKTQVXLTVQRHGK4DUXAKDA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.560766475 (415h3m6.388474871s) [upload=431917]: Next returned nil segment Dec 22 22:31:05.560785161 (415h3m6.388493562s) [upload=431917]: Waiting for error group managing segments... Dec 22 22:31:05.566117878 (415h3m6.39382628s) [noise=true , address=45.91.116.4:4939 , piece_id=QYNKMKDLQJV7B6M2E34DU53Y7YI6VBCUW3C7ZLQOI4XWTBXZHKDQ, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.566723306 (415h3m6.394431687s) [noise=true , address=45.91.116.11:4117 , piece_id=WLMO5E3BIFFL3TOAUSSBGORIZ6DOCWFCMVT4AVFPPGPSKP2OKDAA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.566992667 (415h3m6.394701063s) [noise=true , address=145.79.227.36:4519 , piece_id=JUKRRIZWVGKEDD3GPJ7HMY2MEQZNMUCTKTXR6ZLSIERRONTZKHCQ, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.567464626 (415h3m6.395173019s) [noise=true , address=145.79.227.47:4266 , piece_id=3NFYV2PWBNTAOSFHENM4J5PZ7XXE72UKTQVXLTVQRHGK4DUXAKDA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.567924492 (415h3m6.395632883s) [noise=true , address=145.79.227.15:4874 , piece_id=FNCCIMIVJTGZ2X5TOYOZH727JXHYM7HG3HBHBTZDAG67SHZLYYYA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.568807294 (415h3m6.396515685s) [noise=true , address=145.79.227.51:4409 , piece_id=ATH3F4DFOUUIFFJZRA2SSRRL7AKWZMQNQRYOZCVFFDLTFCFP236A, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.570579823 (415h3m6.398288214s) [noise=true , address=145.79.227.48:4301 , piece_id=LBIYTJNOX5J4M3HDE7FGF5CSARBITW2AK3P2DSYFLFKUOIHZUKOQ, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.571632758 (415h3m6.399341149s) [noise=true , address=145.79.227.43:4157 , piece_id=3W3LVTU37DNHWS52IGIIH7SXWXVNPGEY27RQNT272L3Z64GNR4KA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.573168383 (415h3m6.400876774s) [noise=true , address=145.79.227.38:4979 , piece_id=HLMQC5OEB4OL4OZSRMCEVAWR5QM2T4MQUBPTHRVQLEAHPBWPQF6Q, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.579995420 (415h3m6.407703816s) [noise=true , address=45.91.116.14:4190 , piece_id=V4T6Z7K3C75LQEXNQIYZGYWW2CSQ7UPSET46QGDRT52RFGAROTTA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.592093172 (415h3m6.419801563s) [noise=true , address=145.79.227.37:4565 , piece_id=554RHTYPG27UNCFPSRMOB245AX4QXL2QFIYECNTVWNNJMAL4BTYA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:09.946500434 (415h3m10.774208816s) [noise=true , address=145.79.227.40:4051 , piece_id=55RRAD7DQX6SR7RBLYAV7MD7XRWPNIWHTVVBQZ2ZL3OE2L7LRRIQ, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:35.398235543 (415h3m36.225943943s) [noise=true , address=145.79.227.39:4013 , piece_id=JJVURGSLKY2UQ4FKNCLATA3ZBD5SDL32XJAG42KH2U6OYK3KTS6Q, seg_pos={0 0}, upload=431917]: Done uploading piece. err: ecclient: upload failed (node:121scLvKEW9zw2aEz22zpremcjiop67eQ8v2uqpmaD5aZzCAP7b, address:145.79.227.39:4013): protocol: write tcp 159.223.143.39:60842->145.79.227.39:4013: use of closed network connection; write tcp 159.223.143.39:60842->145.79.227.39:4013: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:60842->145.79.227.39:4013: use of closed network connection Dec 22 22:31:35.398562305 (415h3m36.226270699s) [noise=true , address=145.79.227.32:4835 , piece_id=BRE3JH3XKPXB6FN7WB3W6ED7MOFES6HYLR34YGR6L5AX7RPIQ7IA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: ecclient: upload failed (node:16vVfeoqCkbMYCRjMMWaKBnCvQBBJPMgwQtWTq9HKQrD9SS17y, address:145.79.227.32:4835): protocol: write tcp 159.223.143.39:45098->145.79.227.32:4835: use of closed network connection; write tcp 159.223.143.39:45098->145.79.227.32:4835: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:45098->145.79.227.32:4835: use of closed network connection Dec 22 22:31:35.399183028 (415h3m36.226891422s) [noise=true , address=145.79.227.50:4373 , piece_id=4U55T5QDYCE6AZ3NCTPAVECPAELPQUKT3HV7GZL7JFRMWRN2AZUQ, seg_pos={0 0}, upload=431917]: Done uploading piece. err: ecclient: upload failed (node:12AQGSkJuNJDxoeqhChUenWxHwF5yK41UESCF63FQz3XvabXXfz, address:145.79.227.50:4373): protocol: read tcp 159.223.143.39:56028->145.79.227.50:4373: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 22 22:31:35.399278021 (415h3m36.226986417s) [noise=true , address=145.79.227.29:4943 , piece_id=G4TMZ6IAIZBDON3UEDPH4LJQFHV3JLALEXSJ3VI4XN6WHIZOB57Q, seg_pos={0 0}, upload=431917]: Done uploading piece. err: ecclient: upload failed (node:12fc5Trj8XxacZhzxdksUVT5CbG6uTmiwdUPjreYi4UMPY1TXWz, address:145.79.227.29:4943): protocol: write tcp 159.223.143.39:50820->145.79.227.29:4943: use of closed network connection; write tcp 159.223.143.39:50820->145.79.227.29:4943: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:50820->145.79.227.29:4943: use of closed network connection Dec 22 22:31:35.399379485 (415h3m36.227087866s) [seg_pos={0 0}, upload=431917]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 22 22:31:35.421122544 (415h3m36.248830935s) [upload=431917]: Done uploading segments... Dec 22 22:31:35.421174369 (415h3m36.248882769s) [upload=431917]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 417.584248ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........X.................................................] duration: 551.411189ms trace: https://jaeger.dev.storj.io/trace/7013ffb1a981d605 FAIL (2025-12-22 23:08:26 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 22 22:31:05.439510308 (415h3m6.2672187s) [upload=431917]: Starting upload Dec 22 22:31:05.439581993 (415h3m6.26729037s) [upload=431917]: Done starting upload Dec 22 22:31:05.439812792 (415h3m6.267521185s) [upload=431917]: Uploading segments... Dec 22 22:31:05.439942524 (415h3m6.267650918s) [upload=431917]: Got next segment. Inline: false Dec 22 22:31:05.440022744 (415h3m6.267731174s) [upload=431917]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 22 22:31:05.440029670 (415h3m6.267738072s) [upload=431917]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 22 22:31:05.467186372 (415h3m6.294894769s) [seg_pos={0 0}, upload=431917]: Begin upload segment... Dec 22 22:31:05.467265149 (415h3m6.294973535s) [seg_pos={0 0}, upload=431917]: Done begin upload segment. Dec 22 22:31:05.467296974 (415h3m6.295005373s) [noise=true , address=145.79.227.51:4409 , piece_id=ATH3F4DFOUUIFFJZRA2SSRRL7AKWZMQNQRYOZCVFFDLTFCFP236A, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.467389853 (415h3m6.295098247s) [noise=true , address=145.79.227.39:4013 , piece_id=JJVURGSLKY2UQ4FKNCLATA3ZBD5SDL32XJAG42KH2U6OYK3KTS6Q, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.467912104 (415h3m6.295620486s) [noise=true , address=145.79.227.36:4519 , piece_id=JUKRRIZWVGKEDD3GPJ7HMY2MEQZNMUCTKTXR6ZLSIERRONTZKHCQ, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.468324863 (415h3m6.296033263s) [noise=true , address=145.79.227.38:4979 , piece_id=HLMQC5OEB4OL4OZSRMCEVAWR5QM2T4MQUBPTHRVQLEAHPBWPQF6Q, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.468334764 (415h3m6.296043147s) [noise=true , address=145.79.227.40:4051 , piece_id=55RRAD7DQX6SR7RBLYAV7MD7XRWPNIWHTVVBQZ2ZL3OE2L7LRRIQ, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.468742525 (415h3m6.296450916s) [noise=true , address=45.91.116.4:4939 , piece_id=QYNKMKDLQJV7B6M2E34DU53Y7YI6VBCUW3C7ZLQOI4XWTBXZHKDQ, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.468915034 (415h3m6.296623414s) [noise=true , address=45.91.116.11:4117 , piece_id=WLMO5E3BIFFL3TOAUSSBGORIZ6DOCWFCMVT4AVFPPGPSKP2OKDAA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.469069393 (415h3m6.296777787s) [noise=true , address=145.79.227.32:4835 , piece_id=BRE3JH3XKPXB6FN7WB3W6ED7MOFES6HYLR34YGR6L5AX7RPIQ7IA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.469285789 (415h3m6.296994183s) [noise=true , address=145.79.227.29:4943 , piece_id=G4TMZ6IAIZBDON3UEDPH4LJQFHV3JLALEXSJ3VI4XN6WHIZOB57Q, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.469411962 (415h3m6.297120355s) [noise=true , address=145.79.227.48:4301 , piece_id=LBIYTJNOX5J4M3HDE7FGF5CSARBITW2AK3P2DSYFLFKUOIHZUKOQ, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.469749379 (415h3m6.297457773s) [noise=true , address=145.79.227.37:4565 , piece_id=554RHTYPG27UNCFPSRMOB245AX4QXL2QFIYECNTVWNNJMAL4BTYA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.470080828 (415h3m6.29778922s) [noise=true , address=145.79.227.43:4157 , piece_id=3W3LVTU37DNHWS52IGIIH7SXWXVNPGEY27RQNT272L3Z64GNR4KA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.470477660 (415h3m6.298186058s) [noise=true , address=45.91.116.14:4190 , piece_id=V4T6Z7K3C75LQEXNQIYZGYWW2CSQ7UPSET46QGDRT52RFGAROTTA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.470567977 (415h3m6.29827637s) [noise=true , address=145.79.227.15:4874 , piece_id=FNCCIMIVJTGZ2X5TOYOZH727JXHYM7HG3HBHBTZDAG67SHZLYYYA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.470980950 (415h3m6.298689349s) [noise=true , address=145.79.227.50:4373 , piece_id=4U55T5QDYCE6AZ3NCTPAVECPAELPQUKT3HV7GZL7JFRMWRN2AZUQ, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.471440098 (415h3m6.299148489s) [noise=true , address=145.79.227.47:4266 , piece_id=3NFYV2PWBNTAOSFHENM4J5PZ7XXE72UKTQVXLTVQRHGK4DUXAKDA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.560766475 (415h3m6.388474871s) [upload=431917]: Next returned nil segment Dec 22 22:31:05.560785161 (415h3m6.388493562s) [upload=431917]: Waiting for error group managing segments... Dec 22 22:31:05.566117878 (415h3m6.39382628s) [noise=true , address=45.91.116.4:4939 , piece_id=QYNKMKDLQJV7B6M2E34DU53Y7YI6VBCUW3C7ZLQOI4XWTBXZHKDQ, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.566723306 (415h3m6.394431687s) [noise=true , address=45.91.116.11:4117 , piece_id=WLMO5E3BIFFL3TOAUSSBGORIZ6DOCWFCMVT4AVFPPGPSKP2OKDAA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.566992667 (415h3m6.394701063s) [noise=true , address=145.79.227.36:4519 , piece_id=JUKRRIZWVGKEDD3GPJ7HMY2MEQZNMUCTKTXR6ZLSIERRONTZKHCQ, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.567464626 (415h3m6.395173019s) [noise=true , address=145.79.227.47:4266 , piece_id=3NFYV2PWBNTAOSFHENM4J5PZ7XXE72UKTQVXLTVQRHGK4DUXAKDA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.567924492 (415h3m6.395632883s) [noise=true , address=145.79.227.15:4874 , piece_id=FNCCIMIVJTGZ2X5TOYOZH727JXHYM7HG3HBHBTZDAG67SHZLYYYA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.568807294 (415h3m6.396515685s) [noise=true , address=145.79.227.51:4409 , piece_id=ATH3F4DFOUUIFFJZRA2SSRRL7AKWZMQNQRYOZCVFFDLTFCFP236A, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.570579823 (415h3m6.398288214s) [noise=true , address=145.79.227.48:4301 , piece_id=LBIYTJNOX5J4M3HDE7FGF5CSARBITW2AK3P2DSYFLFKUOIHZUKOQ, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.571632758 (415h3m6.399341149s) [noise=true , address=145.79.227.43:4157 , piece_id=3W3LVTU37DNHWS52IGIIH7SXWXVNPGEY27RQNT272L3Z64GNR4KA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.573168383 (415h3m6.400876774s) [noise=true , address=145.79.227.38:4979 , piece_id=HLMQC5OEB4OL4OZSRMCEVAWR5QM2T4MQUBPTHRVQLEAHPBWPQF6Q, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.579995420 (415h3m6.407703816s) [noise=true , address=45.91.116.14:4190 , piece_id=V4T6Z7K3C75LQEXNQIYZGYWW2CSQ7UPSET46QGDRT52RFGAROTTA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.592093172 (415h3m6.419801563s) [noise=true , address=145.79.227.37:4565 , piece_id=554RHTYPG27UNCFPSRMOB245AX4QXL2QFIYECNTVWNNJMAL4BTYA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:09.946500434 (415h3m10.774208816s) [noise=true , address=145.79.227.40:4051 , piece_id=55RRAD7DQX6SR7RBLYAV7MD7XRWPNIWHTVVBQZ2ZL3OE2L7LRRIQ, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:35.398235543 (415h3m36.225943943s) [noise=true , address=145.79.227.39:4013 , piece_id=JJVURGSLKY2UQ4FKNCLATA3ZBD5SDL32XJAG42KH2U6OYK3KTS6Q, seg_pos={0 0}, upload=431917]: Done uploading piece. err: ecclient: upload failed (node:121scLvKEW9zw2aEz22zpremcjiop67eQ8v2uqpmaD5aZzCAP7b, address:145.79.227.39:4013): protocol: write tcp 159.223.143.39:60842->145.79.227.39:4013: use of closed network connection; write tcp 159.223.143.39:60842->145.79.227.39:4013: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:60842->145.79.227.39:4013: use of closed network connection Dec 22 22:31:35.398562305 (415h3m36.226270699s) [noise=true , address=145.79.227.32:4835 , piece_id=BRE3JH3XKPXB6FN7WB3W6ED7MOFES6HYLR34YGR6L5AX7RPIQ7IA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: ecclient: upload failed (node:16vVfeoqCkbMYCRjMMWaKBnCvQBBJPMgwQtWTq9HKQrD9SS17y, address:145.79.227.32:4835): protocol: write tcp 159.223.143.39:45098->145.79.227.32:4835: use of closed network connection; write tcp 159.223.143.39:45098->145.79.227.32:4835: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:45098->145.79.227.32:4835: use of closed network connection Dec 22 22:31:35.399183028 (415h3m36.226891422s) [noise=true , address=145.79.227.50:4373 , piece_id=4U55T5QDYCE6AZ3NCTPAVECPAELPQUKT3HV7GZL7JFRMWRN2AZUQ, seg_pos={0 0}, upload=431917]: Done uploading piece. err: ecclient: upload failed (node:12AQGSkJuNJDxoeqhChUenWxHwF5yK41UESCF63FQz3XvabXXfz, address:145.79.227.50:4373): protocol: read tcp 159.223.143.39:56028->145.79.227.50:4373: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 22 22:31:35.399278021 (415h3m36.226986417s) [noise=true , address=145.79.227.29:4943 , piece_id=G4TMZ6IAIZBDON3UEDPH4LJQFHV3JLALEXSJ3VI4XN6WHIZOB57Q, seg_pos={0 0}, upload=431917]: Done uploading piece. err: ecclient: upload failed (node:12fc5Trj8XxacZhzxdksUVT5CbG6uTmiwdUPjreYi4UMPY1TXWz, address:145.79.227.29:4943): protocol: write tcp 159.223.143.39:50820->145.79.227.29:4943: use of closed network connection; write tcp 159.223.143.39:50820->145.79.227.29:4943: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:50820->145.79.227.29:4943: use of closed network connection Dec 22 22:31:35.399379485 (415h3m36.227087866s) [seg_pos={0 0}, upload=431917]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 22 22:31:35.421122544 (415h3m36.248830935s) [upload=431917]: Done uploading segments... Dec 22 22:31:35.421174369 (415h3m36.248882769s) [upload=431917]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 551.438142ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........X................................................] duration: 515.633272ms trace: https://jaeger.dev.storj.io/trace/7013ffb1a981d605 FAIL (2025-12-22 23:07:41 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 22 22:31:05.439510308 (415h3m6.2672187s) [upload=431917]: Starting upload Dec 22 22:31:05.439581993 (415h3m6.26729037s) [upload=431917]: Done starting upload Dec 22 22:31:05.439812792 (415h3m6.267521185s) [upload=431917]: Uploading segments... Dec 22 22:31:05.439942524 (415h3m6.267650918s) [upload=431917]: Got next segment. Inline: false Dec 22 22:31:05.440022744 (415h3m6.267731174s) [upload=431917]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 22 22:31:05.440029670 (415h3m6.267738072s) [upload=431917]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 22 22:31:05.467186372 (415h3m6.294894769s) [seg_pos={0 0}, upload=431917]: Begin upload segment... Dec 22 22:31:05.467265149 (415h3m6.294973535s) [seg_pos={0 0}, upload=431917]: Done begin upload segment. Dec 22 22:31:05.467296974 (415h3m6.295005373s) [noise=true , address=145.79.227.51:4409 , piece_id=ATH3F4DFOUUIFFJZRA2SSRRL7AKWZMQNQRYOZCVFFDLTFCFP236A, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.467389853 (415h3m6.295098247s) [noise=true , address=145.79.227.39:4013 , piece_id=JJVURGSLKY2UQ4FKNCLATA3ZBD5SDL32XJAG42KH2U6OYK3KTS6Q, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.467912104 (415h3m6.295620486s) [noise=true , address=145.79.227.36:4519 , piece_id=JUKRRIZWVGKEDD3GPJ7HMY2MEQZNMUCTKTXR6ZLSIERRONTZKHCQ, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.468324863 (415h3m6.296033263s) [noise=true , address=145.79.227.38:4979 , piece_id=HLMQC5OEB4OL4OZSRMCEVAWR5QM2T4MQUBPTHRVQLEAHPBWPQF6Q, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.468334764 (415h3m6.296043147s) [noise=true , address=145.79.227.40:4051 , piece_id=55RRAD7DQX6SR7RBLYAV7MD7XRWPNIWHTVVBQZ2ZL3OE2L7LRRIQ, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.468742525 (415h3m6.296450916s) [noise=true , address=45.91.116.4:4939 , piece_id=QYNKMKDLQJV7B6M2E34DU53Y7YI6VBCUW3C7ZLQOI4XWTBXZHKDQ, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.468915034 (415h3m6.296623414s) [noise=true , address=45.91.116.11:4117 , piece_id=WLMO5E3BIFFL3TOAUSSBGORIZ6DOCWFCMVT4AVFPPGPSKP2OKDAA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.469069393 (415h3m6.296777787s) [noise=true , address=145.79.227.32:4835 , piece_id=BRE3JH3XKPXB6FN7WB3W6ED7MOFES6HYLR34YGR6L5AX7RPIQ7IA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.469285789 (415h3m6.296994183s) [noise=true , address=145.79.227.29:4943 , piece_id=G4TMZ6IAIZBDON3UEDPH4LJQFHV3JLALEXSJ3VI4XN6WHIZOB57Q, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.469411962 (415h3m6.297120355s) [noise=true , address=145.79.227.48:4301 , piece_id=LBIYTJNOX5J4M3HDE7FGF5CSARBITW2AK3P2DSYFLFKUOIHZUKOQ, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.469749379 (415h3m6.297457773s) [noise=true , address=145.79.227.37:4565 , piece_id=554RHTYPG27UNCFPSRMOB245AX4QXL2QFIYECNTVWNNJMAL4BTYA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.470080828 (415h3m6.29778922s) [noise=true , address=145.79.227.43:4157 , piece_id=3W3LVTU37DNHWS52IGIIH7SXWXVNPGEY27RQNT272L3Z64GNR4KA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.470477660 (415h3m6.298186058s) [noise=true , address=45.91.116.14:4190 , piece_id=V4T6Z7K3C75LQEXNQIYZGYWW2CSQ7UPSET46QGDRT52RFGAROTTA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.470567977 (415h3m6.29827637s) [noise=true , address=145.79.227.15:4874 , piece_id=FNCCIMIVJTGZ2X5TOYOZH727JXHYM7HG3HBHBTZDAG67SHZLYYYA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.470980950 (415h3m6.298689349s) [noise=true , address=145.79.227.50:4373 , piece_id=4U55T5QDYCE6AZ3NCTPAVECPAELPQUKT3HV7GZL7JFRMWRN2AZUQ, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.471440098 (415h3m6.299148489s) [noise=true , address=145.79.227.47:4266 , piece_id=3NFYV2PWBNTAOSFHENM4J5PZ7XXE72UKTQVXLTVQRHGK4DUXAKDA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.560766475 (415h3m6.388474871s) [upload=431917]: Next returned nil segment Dec 22 22:31:05.560785161 (415h3m6.388493562s) [upload=431917]: Waiting for error group managing segments... Dec 22 22:31:05.566117878 (415h3m6.39382628s) [noise=true , address=45.91.116.4:4939 , piece_id=QYNKMKDLQJV7B6M2E34DU53Y7YI6VBCUW3C7ZLQOI4XWTBXZHKDQ, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.566723306 (415h3m6.394431687s) [noise=true , address=45.91.116.11:4117 , piece_id=WLMO5E3BIFFL3TOAUSSBGORIZ6DOCWFCMVT4AVFPPGPSKP2OKDAA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.566992667 (415h3m6.394701063s) [noise=true , address=145.79.227.36:4519 , piece_id=JUKRRIZWVGKEDD3GPJ7HMY2MEQZNMUCTKTXR6ZLSIERRONTZKHCQ, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.567464626 (415h3m6.395173019s) [noise=true , address=145.79.227.47:4266 , piece_id=3NFYV2PWBNTAOSFHENM4J5PZ7XXE72UKTQVXLTVQRHGK4DUXAKDA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.567924492 (415h3m6.395632883s) [noise=true , address=145.79.227.15:4874 , piece_id=FNCCIMIVJTGZ2X5TOYOZH727JXHYM7HG3HBHBTZDAG67SHZLYYYA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.568807294 (415h3m6.396515685s) [noise=true , address=145.79.227.51:4409 , piece_id=ATH3F4DFOUUIFFJZRA2SSRRL7AKWZMQNQRYOZCVFFDLTFCFP236A, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.570579823 (415h3m6.398288214s) [noise=true , address=145.79.227.48:4301 , piece_id=LBIYTJNOX5J4M3HDE7FGF5CSARBITW2AK3P2DSYFLFKUOIHZUKOQ, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.571632758 (415h3m6.399341149s) [noise=true , address=145.79.227.43:4157 , piece_id=3W3LVTU37DNHWS52IGIIH7SXWXVNPGEY27RQNT272L3Z64GNR4KA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.573168383 (415h3m6.400876774s) [noise=true , address=145.79.227.38:4979 , piece_id=HLMQC5OEB4OL4OZSRMCEVAWR5QM2T4MQUBPTHRVQLEAHPBWPQF6Q, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.579995420 (415h3m6.407703816s) [noise=true , address=45.91.116.14:4190 , piece_id=V4T6Z7K3C75LQEXNQIYZGYWW2CSQ7UPSET46QGDRT52RFGAROTTA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.592093172 (415h3m6.419801563s) [noise=true , address=145.79.227.37:4565 , piece_id=554RHTYPG27UNCFPSRMOB245AX4QXL2QFIYECNTVWNNJMAL4BTYA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:09.946500434 (415h3m10.774208816s) [noise=true , address=145.79.227.40:4051 , piece_id=55RRAD7DQX6SR7RBLYAV7MD7XRWPNIWHTVVBQZ2ZL3OE2L7LRRIQ, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:35.398235543 (415h3m36.225943943s) [noise=true , address=145.79.227.39:4013 , piece_id=JJVURGSLKY2UQ4FKNCLATA3ZBD5SDL32XJAG42KH2U6OYK3KTS6Q, seg_pos={0 0}, upload=431917]: Done uploading piece. err: ecclient: upload failed (node:121scLvKEW9zw2aEz22zpremcjiop67eQ8v2uqpmaD5aZzCAP7b, address:145.79.227.39:4013): protocol: write tcp 159.223.143.39:60842->145.79.227.39:4013: use of closed network connection; write tcp 159.223.143.39:60842->145.79.227.39:4013: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:60842->145.79.227.39:4013: use of closed network connection Dec 22 22:31:35.398562305 (415h3m36.226270699s) [noise=true , address=145.79.227.32:4835 , piece_id=BRE3JH3XKPXB6FN7WB3W6ED7MOFES6HYLR34YGR6L5AX7RPIQ7IA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: ecclient: upload failed (node:16vVfeoqCkbMYCRjMMWaKBnCvQBBJPMgwQtWTq9HKQrD9SS17y, address:145.79.227.32:4835): protocol: write tcp 159.223.143.39:45098->145.79.227.32:4835: use of closed network connection; write tcp 159.223.143.39:45098->145.79.227.32:4835: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:45098->145.79.227.32:4835: use of closed network connection Dec 22 22:31:35.399183028 (415h3m36.226891422s) [noise=true , address=145.79.227.50:4373 , piece_id=4U55T5QDYCE6AZ3NCTPAVECPAELPQUKT3HV7GZL7JFRMWRN2AZUQ, seg_pos={0 0}, upload=431917]: Done uploading piece. err: ecclient: upload failed (node:12AQGSkJuNJDxoeqhChUenWxHwF5yK41UESCF63FQz3XvabXXfz, address:145.79.227.50:4373): protocol: read tcp 159.223.143.39:56028->145.79.227.50:4373: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 22 22:31:35.399278021 (415h3m36.226986417s) [noise=true , address=145.79.227.29:4943 , piece_id=G4TMZ6IAIZBDON3UEDPH4LJQFHV3JLALEXSJ3VI4XN6WHIZOB57Q, seg_pos={0 0}, upload=431917]: Done uploading piece. err: ecclient: upload failed (node:12fc5Trj8XxacZhzxdksUVT5CbG6uTmiwdUPjreYi4UMPY1TXWz, address:145.79.227.29:4943): protocol: write tcp 159.223.143.39:50820->145.79.227.29:4943: use of closed network connection; write tcp 159.223.143.39:50820->145.79.227.29:4943: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:50820->145.79.227.29:4943: use of closed network connection Dec 22 22:31:35.399379485 (415h3m36.227087866s) [seg_pos={0 0}, upload=431917]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 22 22:31:35.421122544 (415h3m36.248830935s) [upload=431917]: Done uploading segments... Dec 22 22:31:35.421174369 (415h3m36.248882769s) [upload=431917]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 515.662782ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............X...............................................] duration: 546.837264ms trace: https://jaeger.dev.storj.io/trace/7013ffb1a981d605 FAIL (2025-12-22 23:07:23 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 22 22:31:05.439510308 (415h3m6.2672187s) [upload=431917]: Starting upload Dec 22 22:31:05.439581993 (415h3m6.26729037s) [upload=431917]: Done starting upload Dec 22 22:31:05.439812792 (415h3m6.267521185s) [upload=431917]: Uploading segments... Dec 22 22:31:05.439942524 (415h3m6.267650918s) [upload=431917]: Got next segment. Inline: false Dec 22 22:31:05.440022744 (415h3m6.267731174s) [upload=431917]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 22 22:31:05.440029670 (415h3m6.267738072s) [upload=431917]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 22 22:31:05.467186372 (415h3m6.294894769s) [seg_pos={0 0}, upload=431917]: Begin upload segment... Dec 22 22:31:05.467265149 (415h3m6.294973535s) [seg_pos={0 0}, upload=431917]: Done begin upload segment. Dec 22 22:31:05.467296974 (415h3m6.295005373s) [noise=true , address=145.79.227.51:4409 , piece_id=ATH3F4DFOUUIFFJZRA2SSRRL7AKWZMQNQRYOZCVFFDLTFCFP236A, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.467389853 (415h3m6.295098247s) [noise=true , address=145.79.227.39:4013 , piece_id=JJVURGSLKY2UQ4FKNCLATA3ZBD5SDL32XJAG42KH2U6OYK3KTS6Q, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.467912104 (415h3m6.295620486s) [noise=true , address=145.79.227.36:4519 , piece_id=JUKRRIZWVGKEDD3GPJ7HMY2MEQZNMUCTKTXR6ZLSIERRONTZKHCQ, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.468324863 (415h3m6.296033263s) [noise=true , address=145.79.227.38:4979 , piece_id=HLMQC5OEB4OL4OZSRMCEVAWR5QM2T4MQUBPTHRVQLEAHPBWPQF6Q, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.468334764 (415h3m6.296043147s) [noise=true , address=145.79.227.40:4051 , piece_id=55RRAD7DQX6SR7RBLYAV7MD7XRWPNIWHTVVBQZ2ZL3OE2L7LRRIQ, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.468742525 (415h3m6.296450916s) [noise=true , address=45.91.116.4:4939 , piece_id=QYNKMKDLQJV7B6M2E34DU53Y7YI6VBCUW3C7ZLQOI4XWTBXZHKDQ, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.468915034 (415h3m6.296623414s) [noise=true , address=45.91.116.11:4117 , piece_id=WLMO5E3BIFFL3TOAUSSBGORIZ6DOCWFCMVT4AVFPPGPSKP2OKDAA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.469069393 (415h3m6.296777787s) [noise=true , address=145.79.227.32:4835 , piece_id=BRE3JH3XKPXB6FN7WB3W6ED7MOFES6HYLR34YGR6L5AX7RPIQ7IA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.469285789 (415h3m6.296994183s) [noise=true , address=145.79.227.29:4943 , piece_id=G4TMZ6IAIZBDON3UEDPH4LJQFHV3JLALEXSJ3VI4XN6WHIZOB57Q, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.469411962 (415h3m6.297120355s) [noise=true , address=145.79.227.48:4301 , piece_id=LBIYTJNOX5J4M3HDE7FGF5CSARBITW2AK3P2DSYFLFKUOIHZUKOQ, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.469749379 (415h3m6.297457773s) [noise=true , address=145.79.227.37:4565 , piece_id=554RHTYPG27UNCFPSRMOB245AX4QXL2QFIYECNTVWNNJMAL4BTYA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.470080828 (415h3m6.29778922s) [noise=true , address=145.79.227.43:4157 , piece_id=3W3LVTU37DNHWS52IGIIH7SXWXVNPGEY27RQNT272L3Z64GNR4KA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.470477660 (415h3m6.298186058s) [noise=true , address=45.91.116.14:4190 , piece_id=V4T6Z7K3C75LQEXNQIYZGYWW2CSQ7UPSET46QGDRT52RFGAROTTA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.470567977 (415h3m6.29827637s) [noise=true , address=145.79.227.15:4874 , piece_id=FNCCIMIVJTGZ2X5TOYOZH727JXHYM7HG3HBHBTZDAG67SHZLYYYA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.470980950 (415h3m6.298689349s) [noise=true , address=145.79.227.50:4373 , piece_id=4U55T5QDYCE6AZ3NCTPAVECPAELPQUKT3HV7GZL7JFRMWRN2AZUQ, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.471440098 (415h3m6.299148489s) [noise=true , address=145.79.227.47:4266 , piece_id=3NFYV2PWBNTAOSFHENM4J5PZ7XXE72UKTQVXLTVQRHGK4DUXAKDA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.560766475 (415h3m6.388474871s) [upload=431917]: Next returned nil segment Dec 22 22:31:05.560785161 (415h3m6.388493562s) [upload=431917]: Waiting for error group managing segments... Dec 22 22:31:05.566117878 (415h3m6.39382628s) [noise=true , address=45.91.116.4:4939 , piece_id=QYNKMKDLQJV7B6M2E34DU53Y7YI6VBCUW3C7ZLQOI4XWTBXZHKDQ, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.566723306 (415h3m6.394431687s) [noise=true , address=45.91.116.11:4117 , piece_id=WLMO5E3BIFFL3TOAUSSBGORIZ6DOCWFCMVT4AVFPPGPSKP2OKDAA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.566992667 (415h3m6.394701063s) [noise=true , address=145.79.227.36:4519 , piece_id=JUKRRIZWVGKEDD3GPJ7HMY2MEQZNMUCTKTXR6ZLSIERRONTZKHCQ, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.567464626 (415h3m6.395173019s) [noise=true , address=145.79.227.47:4266 , piece_id=3NFYV2PWBNTAOSFHENM4J5PZ7XXE72UKTQVXLTVQRHGK4DUXAKDA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.567924492 (415h3m6.395632883s) [noise=true , address=145.79.227.15:4874 , piece_id=FNCCIMIVJTGZ2X5TOYOZH727JXHYM7HG3HBHBTZDAG67SHZLYYYA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.568807294 (415h3m6.396515685s) [noise=true , address=145.79.227.51:4409 , piece_id=ATH3F4DFOUUIFFJZRA2SSRRL7AKWZMQNQRYOZCVFFDLTFCFP236A, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.570579823 (415h3m6.398288214s) [noise=true , address=145.79.227.48:4301 , piece_id=LBIYTJNOX5J4M3HDE7FGF5CSARBITW2AK3P2DSYFLFKUOIHZUKOQ, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.571632758 (415h3m6.399341149s) [noise=true , address=145.79.227.43:4157 , piece_id=3W3LVTU37DNHWS52IGIIH7SXWXVNPGEY27RQNT272L3Z64GNR4KA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.573168383 (415h3m6.400876774s) [noise=true , address=145.79.227.38:4979 , piece_id=HLMQC5OEB4OL4OZSRMCEVAWR5QM2T4MQUBPTHRVQLEAHPBWPQF6Q, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.579995420 (415h3m6.407703816s) [noise=true , address=45.91.116.14:4190 , piece_id=V4T6Z7K3C75LQEXNQIYZGYWW2CSQ7UPSET46QGDRT52RFGAROTTA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.592093172 (415h3m6.419801563s) [noise=true , address=145.79.227.37:4565 , piece_id=554RHTYPG27UNCFPSRMOB245AX4QXL2QFIYECNTVWNNJMAL4BTYA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:09.946500434 (415h3m10.774208816s) [noise=true , address=145.79.227.40:4051 , piece_id=55RRAD7DQX6SR7RBLYAV7MD7XRWPNIWHTVVBQZ2ZL3OE2L7LRRIQ, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:35.398235543 (415h3m36.225943943s) [noise=true , address=145.79.227.39:4013 , piece_id=JJVURGSLKY2UQ4FKNCLATA3ZBD5SDL32XJAG42KH2U6OYK3KTS6Q, seg_pos={0 0}, upload=431917]: Done uploading piece. err: ecclient: upload failed (node:121scLvKEW9zw2aEz22zpremcjiop67eQ8v2uqpmaD5aZzCAP7b, address:145.79.227.39:4013): protocol: write tcp 159.223.143.39:60842->145.79.227.39:4013: use of closed network connection; write tcp 159.223.143.39:60842->145.79.227.39:4013: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:60842->145.79.227.39:4013: use of closed network connection Dec 22 22:31:35.398562305 (415h3m36.226270699s) [noise=true , address=145.79.227.32:4835 , piece_id=BRE3JH3XKPXB6FN7WB3W6ED7MOFES6HYLR34YGR6L5AX7RPIQ7IA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: ecclient: upload failed (node:16vVfeoqCkbMYCRjMMWaKBnCvQBBJPMgwQtWTq9HKQrD9SS17y, address:145.79.227.32:4835): protocol: write tcp 159.223.143.39:45098->145.79.227.32:4835: use of closed network connection; write tcp 159.223.143.39:45098->145.79.227.32:4835: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:45098->145.79.227.32:4835: use of closed network connection Dec 22 22:31:35.399183028 (415h3m36.226891422s) [noise=true , address=145.79.227.50:4373 , piece_id=4U55T5QDYCE6AZ3NCTPAVECPAELPQUKT3HV7GZL7JFRMWRN2AZUQ, seg_pos={0 0}, upload=431917]: Done uploading piece. err: ecclient: upload failed (node:12AQGSkJuNJDxoeqhChUenWxHwF5yK41UESCF63FQz3XvabXXfz, address:145.79.227.50:4373): protocol: read tcp 159.223.143.39:56028->145.79.227.50:4373: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 22 22:31:35.399278021 (415h3m36.226986417s) [noise=true , address=145.79.227.29:4943 , piece_id=G4TMZ6IAIZBDON3UEDPH4LJQFHV3JLALEXSJ3VI4XN6WHIZOB57Q, seg_pos={0 0}, upload=431917]: Done uploading piece. err: ecclient: upload failed (node:12fc5Trj8XxacZhzxdksUVT5CbG6uTmiwdUPjreYi4UMPY1TXWz, address:145.79.227.29:4943): protocol: write tcp 159.223.143.39:50820->145.79.227.29:4943: use of closed network connection; write tcp 159.223.143.39:50820->145.79.227.29:4943: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:50820->145.79.227.29:4943: use of closed network connection Dec 22 22:31:35.399379485 (415h3m36.227087866s) [seg_pos={0 0}, upload=431917]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 22 22:31:35.421122544 (415h3m36.248830935s) [upload=431917]: Done uploading segments... Dec 22 22:31:35.421174369 (415h3m36.248882769s) [upload=431917]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 546.859996ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............X..............................................] duration: 433.905112ms trace: https://jaeger.dev.storj.io/trace/7013ffb1a981d605 FAIL (2025-12-22 23:06:47 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 22 22:31:05.439510308 (415h3m6.2672187s) [upload=431917]: Starting upload Dec 22 22:31:05.439581993 (415h3m6.26729037s) [upload=431917]: Done starting upload Dec 22 22:31:05.439812792 (415h3m6.267521185s) [upload=431917]: Uploading segments... Dec 22 22:31:05.439942524 (415h3m6.267650918s) [upload=431917]: Got next segment. Inline: false Dec 22 22:31:05.440022744 (415h3m6.267731174s) [upload=431917]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 22 22:31:05.440029670 (415h3m6.267738072s) [upload=431917]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 22 22:31:05.467186372 (415h3m6.294894769s) [seg_pos={0 0}, upload=431917]: Begin upload segment... Dec 22 22:31:05.467265149 (415h3m6.294973535s) [seg_pos={0 0}, upload=431917]: Done begin upload segment. Dec 22 22:31:05.467296974 (415h3m6.295005373s) [noise=true , address=145.79.227.51:4409 , piece_id=ATH3F4DFOUUIFFJZRA2SSRRL7AKWZMQNQRYOZCVFFDLTFCFP236A, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.467389853 (415h3m6.295098247s) [noise=true , address=145.79.227.39:4013 , piece_id=JJVURGSLKY2UQ4FKNCLATA3ZBD5SDL32XJAG42KH2U6OYK3KTS6Q, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.467912104 (415h3m6.295620486s) [noise=true , address=145.79.227.36:4519 , piece_id=JUKRRIZWVGKEDD3GPJ7HMY2MEQZNMUCTKTXR6ZLSIERRONTZKHCQ, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.468324863 (415h3m6.296033263s) [noise=true , address=145.79.227.38:4979 , piece_id=HLMQC5OEB4OL4OZSRMCEVAWR5QM2T4MQUBPTHRVQLEAHPBWPQF6Q, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.468334764 (415h3m6.296043147s) [noise=true , address=145.79.227.40:4051 , piece_id=55RRAD7DQX6SR7RBLYAV7MD7XRWPNIWHTVVBQZ2ZL3OE2L7LRRIQ, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.468742525 (415h3m6.296450916s) [noise=true , address=45.91.116.4:4939 , piece_id=QYNKMKDLQJV7B6M2E34DU53Y7YI6VBCUW3C7ZLQOI4XWTBXZHKDQ, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.468915034 (415h3m6.296623414s) [noise=true , address=45.91.116.11:4117 , piece_id=WLMO5E3BIFFL3TOAUSSBGORIZ6DOCWFCMVT4AVFPPGPSKP2OKDAA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.469069393 (415h3m6.296777787s) [noise=true , address=145.79.227.32:4835 , piece_id=BRE3JH3XKPXB6FN7WB3W6ED7MOFES6HYLR34YGR6L5AX7RPIQ7IA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.469285789 (415h3m6.296994183s) [noise=true , address=145.79.227.29:4943 , piece_id=G4TMZ6IAIZBDON3UEDPH4LJQFHV3JLALEXSJ3VI4XN6WHIZOB57Q, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.469411962 (415h3m6.297120355s) [noise=true , address=145.79.227.48:4301 , piece_id=LBIYTJNOX5J4M3HDE7FGF5CSARBITW2AK3P2DSYFLFKUOIHZUKOQ, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.469749379 (415h3m6.297457773s) [noise=true , address=145.79.227.37:4565 , piece_id=554RHTYPG27UNCFPSRMOB245AX4QXL2QFIYECNTVWNNJMAL4BTYA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.470080828 (415h3m6.29778922s) [noise=true , address=145.79.227.43:4157 , piece_id=3W3LVTU37DNHWS52IGIIH7SXWXVNPGEY27RQNT272L3Z64GNR4KA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.470477660 (415h3m6.298186058s) [noise=true , address=45.91.116.14:4190 , piece_id=V4T6Z7K3C75LQEXNQIYZGYWW2CSQ7UPSET46QGDRT52RFGAROTTA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.470567977 (415h3m6.29827637s) [noise=true , address=145.79.227.15:4874 , piece_id=FNCCIMIVJTGZ2X5TOYOZH727JXHYM7HG3HBHBTZDAG67SHZLYYYA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.470980950 (415h3m6.298689349s) [noise=true , address=145.79.227.50:4373 , piece_id=4U55T5QDYCE6AZ3NCTPAVECPAELPQUKT3HV7GZL7JFRMWRN2AZUQ, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.471440098 (415h3m6.299148489s) [noise=true , address=145.79.227.47:4266 , piece_id=3NFYV2PWBNTAOSFHENM4J5PZ7XXE72UKTQVXLTVQRHGK4DUXAKDA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.560766475 (415h3m6.388474871s) [upload=431917]: Next returned nil segment Dec 22 22:31:05.560785161 (415h3m6.388493562s) [upload=431917]: Waiting for error group managing segments... Dec 22 22:31:05.566117878 (415h3m6.39382628s) [noise=true , address=45.91.116.4:4939 , piece_id=QYNKMKDLQJV7B6M2E34DU53Y7YI6VBCUW3C7ZLQOI4XWTBXZHKDQ, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.566723306 (415h3m6.394431687s) [noise=true , address=45.91.116.11:4117 , piece_id=WLMO5E3BIFFL3TOAUSSBGORIZ6DOCWFCMVT4AVFPPGPSKP2OKDAA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.566992667 (415h3m6.394701063s) [noise=true , address=145.79.227.36:4519 , piece_id=JUKRRIZWVGKEDD3GPJ7HMY2MEQZNMUCTKTXR6ZLSIERRONTZKHCQ, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.567464626 (415h3m6.395173019s) [noise=true , address=145.79.227.47:4266 , piece_id=3NFYV2PWBNTAOSFHENM4J5PZ7XXE72UKTQVXLTVQRHGK4DUXAKDA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.567924492 (415h3m6.395632883s) [noise=true , address=145.79.227.15:4874 , piece_id=FNCCIMIVJTGZ2X5TOYOZH727JXHYM7HG3HBHBTZDAG67SHZLYYYA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.568807294 (415h3m6.396515685s) [noise=true , address=145.79.227.51:4409 , piece_id=ATH3F4DFOUUIFFJZRA2SSRRL7AKWZMQNQRYOZCVFFDLTFCFP236A, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.570579823 (415h3m6.398288214s) [noise=true , address=145.79.227.48:4301 , piece_id=LBIYTJNOX5J4M3HDE7FGF5CSARBITW2AK3P2DSYFLFKUOIHZUKOQ, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.571632758 (415h3m6.399341149s) [noise=true , address=145.79.227.43:4157 , piece_id=3W3LVTU37DNHWS52IGIIH7SXWXVNPGEY27RQNT272L3Z64GNR4KA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.573168383 (415h3m6.400876774s) [noise=true , address=145.79.227.38:4979 , piece_id=HLMQC5OEB4OL4OZSRMCEVAWR5QM2T4MQUBPTHRVQLEAHPBWPQF6Q, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.579995420 (415h3m6.407703816s) [noise=true , address=45.91.116.14:4190 , piece_id=V4T6Z7K3C75LQEXNQIYZGYWW2CSQ7UPSET46QGDRT52RFGAROTTA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.592093172 (415h3m6.419801563s) [noise=true , address=145.79.227.37:4565 , piece_id=554RHTYPG27UNCFPSRMOB245AX4QXL2QFIYECNTVWNNJMAL4BTYA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:09.946500434 (415h3m10.774208816s) [noise=true , address=145.79.227.40:4051 , piece_id=55RRAD7DQX6SR7RBLYAV7MD7XRWPNIWHTVVBQZ2ZL3OE2L7LRRIQ, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:35.398235543 (415h3m36.225943943s) [noise=true , address=145.79.227.39:4013 , piece_id=JJVURGSLKY2UQ4FKNCLATA3ZBD5SDL32XJAG42KH2U6OYK3KTS6Q, seg_pos={0 0}, upload=431917]: Done uploading piece. err: ecclient: upload failed (node:121scLvKEW9zw2aEz22zpremcjiop67eQ8v2uqpmaD5aZzCAP7b, address:145.79.227.39:4013): protocol: write tcp 159.223.143.39:60842->145.79.227.39:4013: use of closed network connection; write tcp 159.223.143.39:60842->145.79.227.39:4013: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:60842->145.79.227.39:4013: use of closed network connection Dec 22 22:31:35.398562305 (415h3m36.226270699s) [noise=true , address=145.79.227.32:4835 , piece_id=BRE3JH3XKPXB6FN7WB3W6ED7MOFES6HYLR34YGR6L5AX7RPIQ7IA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: ecclient: upload failed (node:16vVfeoqCkbMYCRjMMWaKBnCvQBBJPMgwQtWTq9HKQrD9SS17y, address:145.79.227.32:4835): protocol: write tcp 159.223.143.39:45098->145.79.227.32:4835: use of closed network connection; write tcp 159.223.143.39:45098->145.79.227.32:4835: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:45098->145.79.227.32:4835: use of closed network connection Dec 22 22:31:35.399183028 (415h3m36.226891422s) [noise=true , address=145.79.227.50:4373 , piece_id=4U55T5QDYCE6AZ3NCTPAVECPAELPQUKT3HV7GZL7JFRMWRN2AZUQ, seg_pos={0 0}, upload=431917]: Done uploading piece. err: ecclient: upload failed (node:12AQGSkJuNJDxoeqhChUenWxHwF5yK41UESCF63FQz3XvabXXfz, address:145.79.227.50:4373): protocol: read tcp 159.223.143.39:56028->145.79.227.50:4373: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 22 22:31:35.399278021 (415h3m36.226986417s) [noise=true , address=145.79.227.29:4943 , piece_id=G4TMZ6IAIZBDON3UEDPH4LJQFHV3JLALEXSJ3VI4XN6WHIZOB57Q, seg_pos={0 0}, upload=431917]: Done uploading piece. err: ecclient: upload failed (node:12fc5Trj8XxacZhzxdksUVT5CbG6uTmiwdUPjreYi4UMPY1TXWz, address:145.79.227.29:4943): protocol: write tcp 159.223.143.39:50820->145.79.227.29:4943: use of closed network connection; write tcp 159.223.143.39:50820->145.79.227.29:4943: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:50820->145.79.227.29:4943: use of closed network connection Dec 22 22:31:35.399379485 (415h3m36.227087866s) [seg_pos={0 0}, upload=431917]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 22 22:31:35.421122544 (415h3m36.248830935s) [upload=431917]: Done uploading segments... Dec 22 22:31:35.421174369 (415h3m36.248882769s) [upload=431917]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 433.93652ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............X.............................................] duration: 504.327727ms trace: https://jaeger.dev.storj.io/trace/7013ffb1a981d605 FAIL (2025-12-22 23:05:33 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 22 22:31:05.439510308 (415h3m6.2672187s) [upload=431917]: Starting upload Dec 22 22:31:05.439581993 (415h3m6.26729037s) [upload=431917]: Done starting upload Dec 22 22:31:05.439812792 (415h3m6.267521185s) [upload=431917]: Uploading segments... Dec 22 22:31:05.439942524 (415h3m6.267650918s) [upload=431917]: Got next segment. Inline: false Dec 22 22:31:05.440022744 (415h3m6.267731174s) [upload=431917]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 22 22:31:05.440029670 (415h3m6.267738072s) [upload=431917]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 22 22:31:05.467186372 (415h3m6.294894769s) [seg_pos={0 0}, upload=431917]: Begin upload segment... Dec 22 22:31:05.467265149 (415h3m6.294973535s) [seg_pos={0 0}, upload=431917]: Done begin upload segment. Dec 22 22:31:05.467296974 (415h3m6.295005373s) [noise=true , address=145.79.227.51:4409 , piece_id=ATH3F4DFOUUIFFJZRA2SSRRL7AKWZMQNQRYOZCVFFDLTFCFP236A, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.467389853 (415h3m6.295098247s) [noise=true , address=145.79.227.39:4013 , piece_id=JJVURGSLKY2UQ4FKNCLATA3ZBD5SDL32XJAG42KH2U6OYK3KTS6Q, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.467912104 (415h3m6.295620486s) [noise=true , address=145.79.227.36:4519 , piece_id=JUKRRIZWVGKEDD3GPJ7HMY2MEQZNMUCTKTXR6ZLSIERRONTZKHCQ, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.468324863 (415h3m6.296033263s) [noise=true , address=145.79.227.38:4979 , piece_id=HLMQC5OEB4OL4OZSRMCEVAWR5QM2T4MQUBPTHRVQLEAHPBWPQF6Q, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.468334764 (415h3m6.296043147s) [noise=true , address=145.79.227.40:4051 , piece_id=55RRAD7DQX6SR7RBLYAV7MD7XRWPNIWHTVVBQZ2ZL3OE2L7LRRIQ, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.468742525 (415h3m6.296450916s) [noise=true , address=45.91.116.4:4939 , piece_id=QYNKMKDLQJV7B6M2E34DU53Y7YI6VBCUW3C7ZLQOI4XWTBXZHKDQ, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.468915034 (415h3m6.296623414s) [noise=true , address=45.91.116.11:4117 , piece_id=WLMO5E3BIFFL3TOAUSSBGORIZ6DOCWFCMVT4AVFPPGPSKP2OKDAA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.469069393 (415h3m6.296777787s) [noise=true , address=145.79.227.32:4835 , piece_id=BRE3JH3XKPXB6FN7WB3W6ED7MOFES6HYLR34YGR6L5AX7RPIQ7IA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.469285789 (415h3m6.296994183s) [noise=true , address=145.79.227.29:4943 , piece_id=G4TMZ6IAIZBDON3UEDPH4LJQFHV3JLALEXSJ3VI4XN6WHIZOB57Q, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.469411962 (415h3m6.297120355s) [noise=true , address=145.79.227.48:4301 , piece_id=LBIYTJNOX5J4M3HDE7FGF5CSARBITW2AK3P2DSYFLFKUOIHZUKOQ, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.469749379 (415h3m6.297457773s) [noise=true , address=145.79.227.37:4565 , piece_id=554RHTYPG27UNCFPSRMOB245AX4QXL2QFIYECNTVWNNJMAL4BTYA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.470080828 (415h3m6.29778922s) [noise=true , address=145.79.227.43:4157 , piece_id=3W3LVTU37DNHWS52IGIIH7SXWXVNPGEY27RQNT272L3Z64GNR4KA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.470477660 (415h3m6.298186058s) [noise=true , address=45.91.116.14:4190 , piece_id=V4T6Z7K3C75LQEXNQIYZGYWW2CSQ7UPSET46QGDRT52RFGAROTTA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.470567977 (415h3m6.29827637s) [noise=true , address=145.79.227.15:4874 , piece_id=FNCCIMIVJTGZ2X5TOYOZH727JXHYM7HG3HBHBTZDAG67SHZLYYYA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.470980950 (415h3m6.298689349s) [noise=true , address=145.79.227.50:4373 , piece_id=4U55T5QDYCE6AZ3NCTPAVECPAELPQUKT3HV7GZL7JFRMWRN2AZUQ, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.471440098 (415h3m6.299148489s) [noise=true , address=145.79.227.47:4266 , piece_id=3NFYV2PWBNTAOSFHENM4J5PZ7XXE72UKTQVXLTVQRHGK4DUXAKDA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.560766475 (415h3m6.388474871s) [upload=431917]: Next returned nil segment Dec 22 22:31:05.560785161 (415h3m6.388493562s) [upload=431917]: Waiting for error group managing segments... Dec 22 22:31:05.566117878 (415h3m6.39382628s) [noise=true , address=45.91.116.4:4939 , piece_id=QYNKMKDLQJV7B6M2E34DU53Y7YI6VBCUW3C7ZLQOI4XWTBXZHKDQ, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.566723306 (415h3m6.394431687s) [noise=true , address=45.91.116.11:4117 , piece_id=WLMO5E3BIFFL3TOAUSSBGORIZ6DOCWFCMVT4AVFPPGPSKP2OKDAA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.566992667 (415h3m6.394701063s) [noise=true , address=145.79.227.36:4519 , piece_id=JUKRRIZWVGKEDD3GPJ7HMY2MEQZNMUCTKTXR6ZLSIERRONTZKHCQ, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.567464626 (415h3m6.395173019s) [noise=true , address=145.79.227.47:4266 , piece_id=3NFYV2PWBNTAOSFHENM4J5PZ7XXE72UKTQVXLTVQRHGK4DUXAKDA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.567924492 (415h3m6.395632883s) [noise=true , address=145.79.227.15:4874 , piece_id=FNCCIMIVJTGZ2X5TOYOZH727JXHYM7HG3HBHBTZDAG67SHZLYYYA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.568807294 (415h3m6.396515685s) [noise=true , address=145.79.227.51:4409 , piece_id=ATH3F4DFOUUIFFJZRA2SSRRL7AKWZMQNQRYOZCVFFDLTFCFP236A, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.570579823 (415h3m6.398288214s) [noise=true , address=145.79.227.48:4301 , piece_id=LBIYTJNOX5J4M3HDE7FGF5CSARBITW2AK3P2DSYFLFKUOIHZUKOQ, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.571632758 (415h3m6.399341149s) [noise=true , address=145.79.227.43:4157 , piece_id=3W3LVTU37DNHWS52IGIIH7SXWXVNPGEY27RQNT272L3Z64GNR4KA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.573168383 (415h3m6.400876774s) [noise=true , address=145.79.227.38:4979 , piece_id=HLMQC5OEB4OL4OZSRMCEVAWR5QM2T4MQUBPTHRVQLEAHPBWPQF6Q, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.579995420 (415h3m6.407703816s) [noise=true , address=45.91.116.14:4190 , piece_id=V4T6Z7K3C75LQEXNQIYZGYWW2CSQ7UPSET46QGDRT52RFGAROTTA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.592093172 (415h3m6.419801563s) [noise=true , address=145.79.227.37:4565 , piece_id=554RHTYPG27UNCFPSRMOB245AX4QXL2QFIYECNTVWNNJMAL4BTYA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:09.946500434 (415h3m10.774208816s) [noise=true , address=145.79.227.40:4051 , piece_id=55RRAD7DQX6SR7RBLYAV7MD7XRWPNIWHTVVBQZ2ZL3OE2L7LRRIQ, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:35.398235543 (415h3m36.225943943s) [noise=true , address=145.79.227.39:4013 , piece_id=JJVURGSLKY2UQ4FKNCLATA3ZBD5SDL32XJAG42KH2U6OYK3KTS6Q, seg_pos={0 0}, upload=431917]: Done uploading piece. err: ecclient: upload failed (node:121scLvKEW9zw2aEz22zpremcjiop67eQ8v2uqpmaD5aZzCAP7b, address:145.79.227.39:4013): protocol: write tcp 159.223.143.39:60842->145.79.227.39:4013: use of closed network connection; write tcp 159.223.143.39:60842->145.79.227.39:4013: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:60842->145.79.227.39:4013: use of closed network connection Dec 22 22:31:35.398562305 (415h3m36.226270699s) [noise=true , address=145.79.227.32:4835 , piece_id=BRE3JH3XKPXB6FN7WB3W6ED7MOFES6HYLR34YGR6L5AX7RPIQ7IA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: ecclient: upload failed (node:16vVfeoqCkbMYCRjMMWaKBnCvQBBJPMgwQtWTq9HKQrD9SS17y, address:145.79.227.32:4835): protocol: write tcp 159.223.143.39:45098->145.79.227.32:4835: use of closed network connection; write tcp 159.223.143.39:45098->145.79.227.32:4835: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:45098->145.79.227.32:4835: use of closed network connection Dec 22 22:31:35.399183028 (415h3m36.226891422s) [noise=true , address=145.79.227.50:4373 , piece_id=4U55T5QDYCE6AZ3NCTPAVECPAELPQUKT3HV7GZL7JFRMWRN2AZUQ, seg_pos={0 0}, upload=431917]: Done uploading piece. err: ecclient: upload failed (node:12AQGSkJuNJDxoeqhChUenWxHwF5yK41UESCF63FQz3XvabXXfz, address:145.79.227.50:4373): protocol: read tcp 159.223.143.39:56028->145.79.227.50:4373: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 22 22:31:35.399278021 (415h3m36.226986417s) [noise=true , address=145.79.227.29:4943 , piece_id=G4TMZ6IAIZBDON3UEDPH4LJQFHV3JLALEXSJ3VI4XN6WHIZOB57Q, seg_pos={0 0}, upload=431917]: Done uploading piece. err: ecclient: upload failed (node:12fc5Trj8XxacZhzxdksUVT5CbG6uTmiwdUPjreYi4UMPY1TXWz, address:145.79.227.29:4943): protocol: write tcp 159.223.143.39:50820->145.79.227.29:4943: use of closed network connection; write tcp 159.223.143.39:50820->145.79.227.29:4943: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:50820->145.79.227.29:4943: use of closed network connection Dec 22 22:31:35.399379485 (415h3m36.227087866s) [seg_pos={0 0}, upload=431917]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 22 22:31:35.421122544 (415h3m36.248830935s) [upload=431917]: Done uploading segments... Dec 22 22:31:35.421174369 (415h3m36.248882769s) [upload=431917]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 504.354298ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............X............................................] duration: 418.658014ms trace: https://jaeger.dev.storj.io/trace/7013ffb1a981d605 FAIL (2025-12-22 23:05:08 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 22 22:31:05.439510308 (415h3m6.2672187s) [upload=431917]: Starting upload Dec 22 22:31:05.439581993 (415h3m6.26729037s) [upload=431917]: Done starting upload Dec 22 22:31:05.439812792 (415h3m6.267521185s) [upload=431917]: Uploading segments... Dec 22 22:31:05.439942524 (415h3m6.267650918s) [upload=431917]: Got next segment. Inline: false Dec 22 22:31:05.440022744 (415h3m6.267731174s) [upload=431917]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 22 22:31:05.440029670 (415h3m6.267738072s) [upload=431917]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 22 22:31:05.467186372 (415h3m6.294894769s) [seg_pos={0 0}, upload=431917]: Begin upload segment... Dec 22 22:31:05.467265149 (415h3m6.294973535s) [seg_pos={0 0}, upload=431917]: Done begin upload segment. Dec 22 22:31:05.467296974 (415h3m6.295005373s) [noise=true , address=145.79.227.51:4409 , piece_id=ATH3F4DFOUUIFFJZRA2SSRRL7AKWZMQNQRYOZCVFFDLTFCFP236A, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.467389853 (415h3m6.295098247s) [noise=true , address=145.79.227.39:4013 , piece_id=JJVURGSLKY2UQ4FKNCLATA3ZBD5SDL32XJAG42KH2U6OYK3KTS6Q, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.467912104 (415h3m6.295620486s) [noise=true , address=145.79.227.36:4519 , piece_id=JUKRRIZWVGKEDD3GPJ7HMY2MEQZNMUCTKTXR6ZLSIERRONTZKHCQ, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.468324863 (415h3m6.296033263s) [noise=true , address=145.79.227.38:4979 , piece_id=HLMQC5OEB4OL4OZSRMCEVAWR5QM2T4MQUBPTHRVQLEAHPBWPQF6Q, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.468334764 (415h3m6.296043147s) [noise=true , address=145.79.227.40:4051 , piece_id=55RRAD7DQX6SR7RBLYAV7MD7XRWPNIWHTVVBQZ2ZL3OE2L7LRRIQ, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.468742525 (415h3m6.296450916s) [noise=true , address=45.91.116.4:4939 , piece_id=QYNKMKDLQJV7B6M2E34DU53Y7YI6VBCUW3C7ZLQOI4XWTBXZHKDQ, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.468915034 (415h3m6.296623414s) [noise=true , address=45.91.116.11:4117 , piece_id=WLMO5E3BIFFL3TOAUSSBGORIZ6DOCWFCMVT4AVFPPGPSKP2OKDAA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.469069393 (415h3m6.296777787s) [noise=true , address=145.79.227.32:4835 , piece_id=BRE3JH3XKPXB6FN7WB3W6ED7MOFES6HYLR34YGR6L5AX7RPIQ7IA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.469285789 (415h3m6.296994183s) [noise=true , address=145.79.227.29:4943 , piece_id=G4TMZ6IAIZBDON3UEDPH4LJQFHV3JLALEXSJ3VI4XN6WHIZOB57Q, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.469411962 (415h3m6.297120355s) [noise=true , address=145.79.227.48:4301 , piece_id=LBIYTJNOX5J4M3HDE7FGF5CSARBITW2AK3P2DSYFLFKUOIHZUKOQ, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.469749379 (415h3m6.297457773s) [noise=true , address=145.79.227.37:4565 , piece_id=554RHTYPG27UNCFPSRMOB245AX4QXL2QFIYECNTVWNNJMAL4BTYA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.470080828 (415h3m6.29778922s) [noise=true , address=145.79.227.43:4157 , piece_id=3W3LVTU37DNHWS52IGIIH7SXWXVNPGEY27RQNT272L3Z64GNR4KA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.470477660 (415h3m6.298186058s) [noise=true , address=45.91.116.14:4190 , piece_id=V4T6Z7K3C75LQEXNQIYZGYWW2CSQ7UPSET46QGDRT52RFGAROTTA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.470567977 (415h3m6.29827637s) [noise=true , address=145.79.227.15:4874 , piece_id=FNCCIMIVJTGZ2X5TOYOZH727JXHYM7HG3HBHBTZDAG67SHZLYYYA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.470980950 (415h3m6.298689349s) [noise=true , address=145.79.227.50:4373 , piece_id=4U55T5QDYCE6AZ3NCTPAVECPAELPQUKT3HV7GZL7JFRMWRN2AZUQ, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.471440098 (415h3m6.299148489s) [noise=true , address=145.79.227.47:4266 , piece_id=3NFYV2PWBNTAOSFHENM4J5PZ7XXE72UKTQVXLTVQRHGK4DUXAKDA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.560766475 (415h3m6.388474871s) [upload=431917]: Next returned nil segment Dec 22 22:31:05.560785161 (415h3m6.388493562s) [upload=431917]: Waiting for error group managing segments... Dec 22 22:31:05.566117878 (415h3m6.39382628s) [noise=true , address=45.91.116.4:4939 , piece_id=QYNKMKDLQJV7B6M2E34DU53Y7YI6VBCUW3C7ZLQOI4XWTBXZHKDQ, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.566723306 (415h3m6.394431687s) [noise=true , address=45.91.116.11:4117 , piece_id=WLMO5E3BIFFL3TOAUSSBGORIZ6DOCWFCMVT4AVFPPGPSKP2OKDAA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.566992667 (415h3m6.394701063s) [noise=true , address=145.79.227.36:4519 , piece_id=JUKRRIZWVGKEDD3GPJ7HMY2MEQZNMUCTKTXR6ZLSIERRONTZKHCQ, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.567464626 (415h3m6.395173019s) [noise=true , address=145.79.227.47:4266 , piece_id=3NFYV2PWBNTAOSFHENM4J5PZ7XXE72UKTQVXLTVQRHGK4DUXAKDA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.567924492 (415h3m6.395632883s) [noise=true , address=145.79.227.15:4874 , piece_id=FNCCIMIVJTGZ2X5TOYOZH727JXHYM7HG3HBHBTZDAG67SHZLYYYA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.568807294 (415h3m6.396515685s) [noise=true , address=145.79.227.51:4409 , piece_id=ATH3F4DFOUUIFFJZRA2SSRRL7AKWZMQNQRYOZCVFFDLTFCFP236A, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.570579823 (415h3m6.398288214s) [noise=true , address=145.79.227.48:4301 , piece_id=LBIYTJNOX5J4M3HDE7FGF5CSARBITW2AK3P2DSYFLFKUOIHZUKOQ, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.571632758 (415h3m6.399341149s) [noise=true , address=145.79.227.43:4157 , piece_id=3W3LVTU37DNHWS52IGIIH7SXWXVNPGEY27RQNT272L3Z64GNR4KA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.573168383 (415h3m6.400876774s) [noise=true , address=145.79.227.38:4979 , piece_id=HLMQC5OEB4OL4OZSRMCEVAWR5QM2T4MQUBPTHRVQLEAHPBWPQF6Q, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.579995420 (415h3m6.407703816s) [noise=true , address=45.91.116.14:4190 , piece_id=V4T6Z7K3C75LQEXNQIYZGYWW2CSQ7UPSET46QGDRT52RFGAROTTA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.592093172 (415h3m6.419801563s) [noise=true , address=145.79.227.37:4565 , piece_id=554RHTYPG27UNCFPSRMOB245AX4QXL2QFIYECNTVWNNJMAL4BTYA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:09.946500434 (415h3m10.774208816s) [noise=true , address=145.79.227.40:4051 , piece_id=55RRAD7DQX6SR7RBLYAV7MD7XRWPNIWHTVVBQZ2ZL3OE2L7LRRIQ, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:35.398235543 (415h3m36.225943943s) [noise=true , address=145.79.227.39:4013 , piece_id=JJVURGSLKY2UQ4FKNCLATA3ZBD5SDL32XJAG42KH2U6OYK3KTS6Q, seg_pos={0 0}, upload=431917]: Done uploading piece. err: ecclient: upload failed (node:121scLvKEW9zw2aEz22zpremcjiop67eQ8v2uqpmaD5aZzCAP7b, address:145.79.227.39:4013): protocol: write tcp 159.223.143.39:60842->145.79.227.39:4013: use of closed network connection; write tcp 159.223.143.39:60842->145.79.227.39:4013: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:60842->145.79.227.39:4013: use of closed network connection Dec 22 22:31:35.398562305 (415h3m36.226270699s) [noise=true , address=145.79.227.32:4835 , piece_id=BRE3JH3XKPXB6FN7WB3W6ED7MOFES6HYLR34YGR6L5AX7RPIQ7IA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: ecclient: upload failed (node:16vVfeoqCkbMYCRjMMWaKBnCvQBBJPMgwQtWTq9HKQrD9SS17y, address:145.79.227.32:4835): protocol: write tcp 159.223.143.39:45098->145.79.227.32:4835: use of closed network connection; write tcp 159.223.143.39:45098->145.79.227.32:4835: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:45098->145.79.227.32:4835: use of closed network connection Dec 22 22:31:35.399183028 (415h3m36.226891422s) [noise=true , address=145.79.227.50:4373 , piece_id=4U55T5QDYCE6AZ3NCTPAVECPAELPQUKT3HV7GZL7JFRMWRN2AZUQ, seg_pos={0 0}, upload=431917]: Done uploading piece. err: ecclient: upload failed (node:12AQGSkJuNJDxoeqhChUenWxHwF5yK41UESCF63FQz3XvabXXfz, address:145.79.227.50:4373): protocol: read tcp 159.223.143.39:56028->145.79.227.50:4373: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 22 22:31:35.399278021 (415h3m36.226986417s) [noise=true , address=145.79.227.29:4943 , piece_id=G4TMZ6IAIZBDON3UEDPH4LJQFHV3JLALEXSJ3VI4XN6WHIZOB57Q, seg_pos={0 0}, upload=431917]: Done uploading piece. err: ecclient: upload failed (node:12fc5Trj8XxacZhzxdksUVT5CbG6uTmiwdUPjreYi4UMPY1TXWz, address:145.79.227.29:4943): protocol: write tcp 159.223.143.39:50820->145.79.227.29:4943: use of closed network connection; write tcp 159.223.143.39:50820->145.79.227.29:4943: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:50820->145.79.227.29:4943: use of closed network connection Dec 22 22:31:35.399379485 (415h3m36.227087866s) [seg_pos={0 0}, upload=431917]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 22 22:31:35.421122544 (415h3m36.248830935s) [upload=431917]: Done uploading segments... Dec 22 22:31:35.421174369 (415h3m36.248882769s) [upload=431917]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 418.686084ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................X...........................................] duration: 419.355024ms trace: https://jaeger.dev.storj.io/trace/7013ffb1a981d605 FAIL (2025-12-22 23:04:33 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 22 22:31:05.439510308 (415h3m6.2672187s) [upload=431917]: Starting upload Dec 22 22:31:05.439581993 (415h3m6.26729037s) [upload=431917]: Done starting upload Dec 22 22:31:05.439812792 (415h3m6.267521185s) [upload=431917]: Uploading segments... Dec 22 22:31:05.439942524 (415h3m6.267650918s) [upload=431917]: Got next segment. Inline: false Dec 22 22:31:05.440022744 (415h3m6.267731174s) [upload=431917]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 22 22:31:05.440029670 (415h3m6.267738072s) [upload=431917]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 22 22:31:05.467186372 (415h3m6.294894769s) [seg_pos={0 0}, upload=431917]: Begin upload segment... Dec 22 22:31:05.467265149 (415h3m6.294973535s) [seg_pos={0 0}, upload=431917]: Done begin upload segment. Dec 22 22:31:05.467296974 (415h3m6.295005373s) [noise=true , address=145.79.227.51:4409 , piece_id=ATH3F4DFOUUIFFJZRA2SSRRL7AKWZMQNQRYOZCVFFDLTFCFP236A, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.467389853 (415h3m6.295098247s) [noise=true , address=145.79.227.39:4013 , piece_id=JJVURGSLKY2UQ4FKNCLATA3ZBD5SDL32XJAG42KH2U6OYK3KTS6Q, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.467912104 (415h3m6.295620486s) [noise=true , address=145.79.227.36:4519 , piece_id=JUKRRIZWVGKEDD3GPJ7HMY2MEQZNMUCTKTXR6ZLSIERRONTZKHCQ, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.468324863 (415h3m6.296033263s) [noise=true , address=145.79.227.38:4979 , piece_id=HLMQC5OEB4OL4OZSRMCEVAWR5QM2T4MQUBPTHRVQLEAHPBWPQF6Q, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.468334764 (415h3m6.296043147s) [noise=true , address=145.79.227.40:4051 , piece_id=55RRAD7DQX6SR7RBLYAV7MD7XRWPNIWHTVVBQZ2ZL3OE2L7LRRIQ, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.468742525 (415h3m6.296450916s) [noise=true , address=45.91.116.4:4939 , piece_id=QYNKMKDLQJV7B6M2E34DU53Y7YI6VBCUW3C7ZLQOI4XWTBXZHKDQ, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.468915034 (415h3m6.296623414s) [noise=true , address=45.91.116.11:4117 , piece_id=WLMO5E3BIFFL3TOAUSSBGORIZ6DOCWFCMVT4AVFPPGPSKP2OKDAA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.469069393 (415h3m6.296777787s) [noise=true , address=145.79.227.32:4835 , piece_id=BRE3JH3XKPXB6FN7WB3W6ED7MOFES6HYLR34YGR6L5AX7RPIQ7IA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.469285789 (415h3m6.296994183s) [noise=true , address=145.79.227.29:4943 , piece_id=G4TMZ6IAIZBDON3UEDPH4LJQFHV3JLALEXSJ3VI4XN6WHIZOB57Q, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.469411962 (415h3m6.297120355s) [noise=true , address=145.79.227.48:4301 , piece_id=LBIYTJNOX5J4M3HDE7FGF5CSARBITW2AK3P2DSYFLFKUOIHZUKOQ, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.469749379 (415h3m6.297457773s) [noise=true , address=145.79.227.37:4565 , piece_id=554RHTYPG27UNCFPSRMOB245AX4QXL2QFIYECNTVWNNJMAL4BTYA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.470080828 (415h3m6.29778922s) [noise=true , address=145.79.227.43:4157 , piece_id=3W3LVTU37DNHWS52IGIIH7SXWXVNPGEY27RQNT272L3Z64GNR4KA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.470477660 (415h3m6.298186058s) [noise=true , address=45.91.116.14:4190 , piece_id=V4T6Z7K3C75LQEXNQIYZGYWW2CSQ7UPSET46QGDRT52RFGAROTTA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.470567977 (415h3m6.29827637s) [noise=true , address=145.79.227.15:4874 , piece_id=FNCCIMIVJTGZ2X5TOYOZH727JXHYM7HG3HBHBTZDAG67SHZLYYYA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.470980950 (415h3m6.298689349s) [noise=true , address=145.79.227.50:4373 , piece_id=4U55T5QDYCE6AZ3NCTPAVECPAELPQUKT3HV7GZL7JFRMWRN2AZUQ, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.471440098 (415h3m6.299148489s) [noise=true , address=145.79.227.47:4266 , piece_id=3NFYV2PWBNTAOSFHENM4J5PZ7XXE72UKTQVXLTVQRHGK4DUXAKDA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.560766475 (415h3m6.388474871s) [upload=431917]: Next returned nil segment Dec 22 22:31:05.560785161 (415h3m6.388493562s) [upload=431917]: Waiting for error group managing segments... Dec 22 22:31:05.566117878 (415h3m6.39382628s) [noise=true , address=45.91.116.4:4939 , piece_id=QYNKMKDLQJV7B6M2E34DU53Y7YI6VBCUW3C7ZLQOI4XWTBXZHKDQ, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.566723306 (415h3m6.394431687s) [noise=true , address=45.91.116.11:4117 , piece_id=WLMO5E3BIFFL3TOAUSSBGORIZ6DOCWFCMVT4AVFPPGPSKP2OKDAA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.566992667 (415h3m6.394701063s) [noise=true , address=145.79.227.36:4519 , piece_id=JUKRRIZWVGKEDD3GPJ7HMY2MEQZNMUCTKTXR6ZLSIERRONTZKHCQ, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.567464626 (415h3m6.395173019s) [noise=true , address=145.79.227.47:4266 , piece_id=3NFYV2PWBNTAOSFHENM4J5PZ7XXE72UKTQVXLTVQRHGK4DUXAKDA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.567924492 (415h3m6.395632883s) [noise=true , address=145.79.227.15:4874 , piece_id=FNCCIMIVJTGZ2X5TOYOZH727JXHYM7HG3HBHBTZDAG67SHZLYYYA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.568807294 (415h3m6.396515685s) [noise=true , address=145.79.227.51:4409 , piece_id=ATH3F4DFOUUIFFJZRA2SSRRL7AKWZMQNQRYOZCVFFDLTFCFP236A, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.570579823 (415h3m6.398288214s) [noise=true , address=145.79.227.48:4301 , piece_id=LBIYTJNOX5J4M3HDE7FGF5CSARBITW2AK3P2DSYFLFKUOIHZUKOQ, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.571632758 (415h3m6.399341149s) [noise=true , address=145.79.227.43:4157 , piece_id=3W3LVTU37DNHWS52IGIIH7SXWXVNPGEY27RQNT272L3Z64GNR4KA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.573168383 (415h3m6.400876774s) [noise=true , address=145.79.227.38:4979 , piece_id=HLMQC5OEB4OL4OZSRMCEVAWR5QM2T4MQUBPTHRVQLEAHPBWPQF6Q, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.579995420 (415h3m6.407703816s) [noise=true , address=45.91.116.14:4190 , piece_id=V4T6Z7K3C75LQEXNQIYZGYWW2CSQ7UPSET46QGDRT52RFGAROTTA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.592093172 (415h3m6.419801563s) [noise=true , address=145.79.227.37:4565 , piece_id=554RHTYPG27UNCFPSRMOB245AX4QXL2QFIYECNTVWNNJMAL4BTYA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:09.946500434 (415h3m10.774208816s) [noise=true , address=145.79.227.40:4051 , piece_id=55RRAD7DQX6SR7RBLYAV7MD7XRWPNIWHTVVBQZ2ZL3OE2L7LRRIQ, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:35.398235543 (415h3m36.225943943s) [noise=true , address=145.79.227.39:4013 , piece_id=JJVURGSLKY2UQ4FKNCLATA3ZBD5SDL32XJAG42KH2U6OYK3KTS6Q, seg_pos={0 0}, upload=431917]: Done uploading piece. err: ecclient: upload failed (node:121scLvKEW9zw2aEz22zpremcjiop67eQ8v2uqpmaD5aZzCAP7b, address:145.79.227.39:4013): protocol: write tcp 159.223.143.39:60842->145.79.227.39:4013: use of closed network connection; write tcp 159.223.143.39:60842->145.79.227.39:4013: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:60842->145.79.227.39:4013: use of closed network connection Dec 22 22:31:35.398562305 (415h3m36.226270699s) [noise=true , address=145.79.227.32:4835 , piece_id=BRE3JH3XKPXB6FN7WB3W6ED7MOFES6HYLR34YGR6L5AX7RPIQ7IA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: ecclient: upload failed (node:16vVfeoqCkbMYCRjMMWaKBnCvQBBJPMgwQtWTq9HKQrD9SS17y, address:145.79.227.32:4835): protocol: write tcp 159.223.143.39:45098->145.79.227.32:4835: use of closed network connection; write tcp 159.223.143.39:45098->145.79.227.32:4835: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:45098->145.79.227.32:4835: use of closed network connection Dec 22 22:31:35.399183028 (415h3m36.226891422s) [noise=true , address=145.79.227.50:4373 , piece_id=4U55T5QDYCE6AZ3NCTPAVECPAELPQUKT3HV7GZL7JFRMWRN2AZUQ, seg_pos={0 0}, upload=431917]: Done uploading piece. err: ecclient: upload failed (node:12AQGSkJuNJDxoeqhChUenWxHwF5yK41UESCF63FQz3XvabXXfz, address:145.79.227.50:4373): protocol: read tcp 159.223.143.39:56028->145.79.227.50:4373: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 22 22:31:35.399278021 (415h3m36.226986417s) [noise=true , address=145.79.227.29:4943 , piece_id=G4TMZ6IAIZBDON3UEDPH4LJQFHV3JLALEXSJ3VI4XN6WHIZOB57Q, seg_pos={0 0}, upload=431917]: Done uploading piece. err: ecclient: upload failed (node:12fc5Trj8XxacZhzxdksUVT5CbG6uTmiwdUPjreYi4UMPY1TXWz, address:145.79.227.29:4943): protocol: write tcp 159.223.143.39:50820->145.79.227.29:4943: use of closed network connection; write tcp 159.223.143.39:50820->145.79.227.29:4943: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:50820->145.79.227.29:4943: use of closed network connection Dec 22 22:31:35.399379485 (415h3m36.227087866s) [seg_pos={0 0}, upload=431917]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 22 22:31:35.421122544 (415h3m36.248830935s) [upload=431917]: Done uploading segments... Dec 22 22:31:35.421174369 (415h3m36.248882769s) [upload=431917]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 419.375675ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................X..........................................] duration: 540.770341ms trace: https://jaeger.dev.storj.io/trace/7013ffb1a981d605 FAIL (2025-12-22 23:04:09 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 22 22:31:05.439510308 (415h3m6.2672187s) [upload=431917]: Starting upload Dec 22 22:31:05.439581993 (415h3m6.26729037s) [upload=431917]: Done starting upload Dec 22 22:31:05.439812792 (415h3m6.267521185s) [upload=431917]: Uploading segments... Dec 22 22:31:05.439942524 (415h3m6.267650918s) [upload=431917]: Got next segment. Inline: false Dec 22 22:31:05.440022744 (415h3m6.267731174s) [upload=431917]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 22 22:31:05.440029670 (415h3m6.267738072s) [upload=431917]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 22 22:31:05.467186372 (415h3m6.294894769s) [seg_pos={0 0}, upload=431917]: Begin upload segment... Dec 22 22:31:05.467265149 (415h3m6.294973535s) [seg_pos={0 0}, upload=431917]: Done begin upload segment. Dec 22 22:31:05.467296974 (415h3m6.295005373s) [noise=true , address=145.79.227.51:4409 , piece_id=ATH3F4DFOUUIFFJZRA2SSRRL7AKWZMQNQRYOZCVFFDLTFCFP236A, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.467389853 (415h3m6.295098247s) [noise=true , address=145.79.227.39:4013 , piece_id=JJVURGSLKY2UQ4FKNCLATA3ZBD5SDL32XJAG42KH2U6OYK3KTS6Q, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.467912104 (415h3m6.295620486s) [noise=true , address=145.79.227.36:4519 , piece_id=JUKRRIZWVGKEDD3GPJ7HMY2MEQZNMUCTKTXR6ZLSIERRONTZKHCQ, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.468324863 (415h3m6.296033263s) [noise=true , address=145.79.227.38:4979 , piece_id=HLMQC5OEB4OL4OZSRMCEVAWR5QM2T4MQUBPTHRVQLEAHPBWPQF6Q, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.468334764 (415h3m6.296043147s) [noise=true , address=145.79.227.40:4051 , piece_id=55RRAD7DQX6SR7RBLYAV7MD7XRWPNIWHTVVBQZ2ZL3OE2L7LRRIQ, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.468742525 (415h3m6.296450916s) [noise=true , address=45.91.116.4:4939 , piece_id=QYNKMKDLQJV7B6M2E34DU53Y7YI6VBCUW3C7ZLQOI4XWTBXZHKDQ, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.468915034 (415h3m6.296623414s) [noise=true , address=45.91.116.11:4117 , piece_id=WLMO5E3BIFFL3TOAUSSBGORIZ6DOCWFCMVT4AVFPPGPSKP2OKDAA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.469069393 (415h3m6.296777787s) [noise=true , address=145.79.227.32:4835 , piece_id=BRE3JH3XKPXB6FN7WB3W6ED7MOFES6HYLR34YGR6L5AX7RPIQ7IA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.469285789 (415h3m6.296994183s) [noise=true , address=145.79.227.29:4943 , piece_id=G4TMZ6IAIZBDON3UEDPH4LJQFHV3JLALEXSJ3VI4XN6WHIZOB57Q, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.469411962 (415h3m6.297120355s) [noise=true , address=145.79.227.48:4301 , piece_id=LBIYTJNOX5J4M3HDE7FGF5CSARBITW2AK3P2DSYFLFKUOIHZUKOQ, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.469749379 (415h3m6.297457773s) [noise=true , address=145.79.227.37:4565 , piece_id=554RHTYPG27UNCFPSRMOB245AX4QXL2QFIYECNTVWNNJMAL4BTYA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.470080828 (415h3m6.29778922s) [noise=true , address=145.79.227.43:4157 , piece_id=3W3LVTU37DNHWS52IGIIH7SXWXVNPGEY27RQNT272L3Z64GNR4KA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.470477660 (415h3m6.298186058s) [noise=true , address=45.91.116.14:4190 , piece_id=V4T6Z7K3C75LQEXNQIYZGYWW2CSQ7UPSET46QGDRT52RFGAROTTA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.470567977 (415h3m6.29827637s) [noise=true , address=145.79.227.15:4874 , piece_id=FNCCIMIVJTGZ2X5TOYOZH727JXHYM7HG3HBHBTZDAG67SHZLYYYA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.470980950 (415h3m6.298689349s) [noise=true , address=145.79.227.50:4373 , piece_id=4U55T5QDYCE6AZ3NCTPAVECPAELPQUKT3HV7GZL7JFRMWRN2AZUQ, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.471440098 (415h3m6.299148489s) [noise=true , address=145.79.227.47:4266 , piece_id=3NFYV2PWBNTAOSFHENM4J5PZ7XXE72UKTQVXLTVQRHGK4DUXAKDA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.560766475 (415h3m6.388474871s) [upload=431917]: Next returned nil segment Dec 22 22:31:05.560785161 (415h3m6.388493562s) [upload=431917]: Waiting for error group managing segments... Dec 22 22:31:05.566117878 (415h3m6.39382628s) [noise=true , address=45.91.116.4:4939 , piece_id=QYNKMKDLQJV7B6M2E34DU53Y7YI6VBCUW3C7ZLQOI4XWTBXZHKDQ, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.566723306 (415h3m6.394431687s) [noise=true , address=45.91.116.11:4117 , piece_id=WLMO5E3BIFFL3TOAUSSBGORIZ6DOCWFCMVT4AVFPPGPSKP2OKDAA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.566992667 (415h3m6.394701063s) [noise=true , address=145.79.227.36:4519 , piece_id=JUKRRIZWVGKEDD3GPJ7HMY2MEQZNMUCTKTXR6ZLSIERRONTZKHCQ, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.567464626 (415h3m6.395173019s) [noise=true , address=145.79.227.47:4266 , piece_id=3NFYV2PWBNTAOSFHENM4J5PZ7XXE72UKTQVXLTVQRHGK4DUXAKDA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.567924492 (415h3m6.395632883s) [noise=true , address=145.79.227.15:4874 , piece_id=FNCCIMIVJTGZ2X5TOYOZH727JXHYM7HG3HBHBTZDAG67SHZLYYYA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.568807294 (415h3m6.396515685s) [noise=true , address=145.79.227.51:4409 , piece_id=ATH3F4DFOUUIFFJZRA2SSRRL7AKWZMQNQRYOZCVFFDLTFCFP236A, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.570579823 (415h3m6.398288214s) [noise=true , address=145.79.227.48:4301 , piece_id=LBIYTJNOX5J4M3HDE7FGF5CSARBITW2AK3P2DSYFLFKUOIHZUKOQ, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.571632758 (415h3m6.399341149s) [noise=true , address=145.79.227.43:4157 , piece_id=3W3LVTU37DNHWS52IGIIH7SXWXVNPGEY27RQNT272L3Z64GNR4KA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.573168383 (415h3m6.400876774s) [noise=true , address=145.79.227.38:4979 , piece_id=HLMQC5OEB4OL4OZSRMCEVAWR5QM2T4MQUBPTHRVQLEAHPBWPQF6Q, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.579995420 (415h3m6.407703816s) [noise=true , address=45.91.116.14:4190 , piece_id=V4T6Z7K3C75LQEXNQIYZGYWW2CSQ7UPSET46QGDRT52RFGAROTTA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.592093172 (415h3m6.419801563s) [noise=true , address=145.79.227.37:4565 , piece_id=554RHTYPG27UNCFPSRMOB245AX4QXL2QFIYECNTVWNNJMAL4BTYA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:09.946500434 (415h3m10.774208816s) [noise=true , address=145.79.227.40:4051 , piece_id=55RRAD7DQX6SR7RBLYAV7MD7XRWPNIWHTVVBQZ2ZL3OE2L7LRRIQ, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:35.398235543 (415h3m36.225943943s) [noise=true , address=145.79.227.39:4013 , piece_id=JJVURGSLKY2UQ4FKNCLATA3ZBD5SDL32XJAG42KH2U6OYK3KTS6Q, seg_pos={0 0}, upload=431917]: Done uploading piece. err: ecclient: upload failed (node:121scLvKEW9zw2aEz22zpremcjiop67eQ8v2uqpmaD5aZzCAP7b, address:145.79.227.39:4013): protocol: write tcp 159.223.143.39:60842->145.79.227.39:4013: use of closed network connection; write tcp 159.223.143.39:60842->145.79.227.39:4013: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:60842->145.79.227.39:4013: use of closed network connection Dec 22 22:31:35.398562305 (415h3m36.226270699s) [noise=true , address=145.79.227.32:4835 , piece_id=BRE3JH3XKPXB6FN7WB3W6ED7MOFES6HYLR34YGR6L5AX7RPIQ7IA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: ecclient: upload failed (node:16vVfeoqCkbMYCRjMMWaKBnCvQBBJPMgwQtWTq9HKQrD9SS17y, address:145.79.227.32:4835): protocol: write tcp 159.223.143.39:45098->145.79.227.32:4835: use of closed network connection; write tcp 159.223.143.39:45098->145.79.227.32:4835: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:45098->145.79.227.32:4835: use of closed network connection Dec 22 22:31:35.399183028 (415h3m36.226891422s) [noise=true , address=145.79.227.50:4373 , piece_id=4U55T5QDYCE6AZ3NCTPAVECPAELPQUKT3HV7GZL7JFRMWRN2AZUQ, seg_pos={0 0}, upload=431917]: Done uploading piece. err: ecclient: upload failed (node:12AQGSkJuNJDxoeqhChUenWxHwF5yK41UESCF63FQz3XvabXXfz, address:145.79.227.50:4373): protocol: read tcp 159.223.143.39:56028->145.79.227.50:4373: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 22 22:31:35.399278021 (415h3m36.226986417s) [noise=true , address=145.79.227.29:4943 , piece_id=G4TMZ6IAIZBDON3UEDPH4LJQFHV3JLALEXSJ3VI4XN6WHIZOB57Q, seg_pos={0 0}, upload=431917]: Done uploading piece. err: ecclient: upload failed (node:12fc5Trj8XxacZhzxdksUVT5CbG6uTmiwdUPjreYi4UMPY1TXWz, address:145.79.227.29:4943): protocol: write tcp 159.223.143.39:50820->145.79.227.29:4943: use of closed network connection; write tcp 159.223.143.39:50820->145.79.227.29:4943: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:50820->145.79.227.29:4943: use of closed network connection Dec 22 22:31:35.399379485 (415h3m36.227087866s) [seg_pos={0 0}, upload=431917]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 22 22:31:35.421122544 (415h3m36.248830935s) [upload=431917]: Done uploading segments... Dec 22 22:31:35.421174369 (415h3m36.248882769s) [upload=431917]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 540.869673ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................X.........................................] duration: 541.52798ms trace: https://jaeger.dev.storj.io/trace/7013ffb1a981d605 FAIL (2025-12-22 23:03:09 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 22 22:31:05.439510308 (415h3m6.2672187s) [upload=431917]: Starting upload Dec 22 22:31:05.439581993 (415h3m6.26729037s) [upload=431917]: Done starting upload Dec 22 22:31:05.439812792 (415h3m6.267521185s) [upload=431917]: Uploading segments... Dec 22 22:31:05.439942524 (415h3m6.267650918s) [upload=431917]: Got next segment. Inline: false Dec 22 22:31:05.440022744 (415h3m6.267731174s) [upload=431917]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 22 22:31:05.440029670 (415h3m6.267738072s) [upload=431917]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 22 22:31:05.467186372 (415h3m6.294894769s) [seg_pos={0 0}, upload=431917]: Begin upload segment... Dec 22 22:31:05.467265149 (415h3m6.294973535s) [seg_pos={0 0}, upload=431917]: Done begin upload segment. Dec 22 22:31:05.467296974 (415h3m6.295005373s) [noise=true , address=145.79.227.51:4409 , piece_id=ATH3F4DFOUUIFFJZRA2SSRRL7AKWZMQNQRYOZCVFFDLTFCFP236A, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.467389853 (415h3m6.295098247s) [noise=true , address=145.79.227.39:4013 , piece_id=JJVURGSLKY2UQ4FKNCLATA3ZBD5SDL32XJAG42KH2U6OYK3KTS6Q, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.467912104 (415h3m6.295620486s) [noise=true , address=145.79.227.36:4519 , piece_id=JUKRRIZWVGKEDD3GPJ7HMY2MEQZNMUCTKTXR6ZLSIERRONTZKHCQ, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.468324863 (415h3m6.296033263s) [noise=true , address=145.79.227.38:4979 , piece_id=HLMQC5OEB4OL4OZSRMCEVAWR5QM2T4MQUBPTHRVQLEAHPBWPQF6Q, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.468334764 (415h3m6.296043147s) [noise=true , address=145.79.227.40:4051 , piece_id=55RRAD7DQX6SR7RBLYAV7MD7XRWPNIWHTVVBQZ2ZL3OE2L7LRRIQ, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.468742525 (415h3m6.296450916s) [noise=true , address=45.91.116.4:4939 , piece_id=QYNKMKDLQJV7B6M2E34DU53Y7YI6VBCUW3C7ZLQOI4XWTBXZHKDQ, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.468915034 (415h3m6.296623414s) [noise=true , address=45.91.116.11:4117 , piece_id=WLMO5E3BIFFL3TOAUSSBGORIZ6DOCWFCMVT4AVFPPGPSKP2OKDAA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.469069393 (415h3m6.296777787s) [noise=true , address=145.79.227.32:4835 , piece_id=BRE3JH3XKPXB6FN7WB3W6ED7MOFES6HYLR34YGR6L5AX7RPIQ7IA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.469285789 (415h3m6.296994183s) [noise=true , address=145.79.227.29:4943 , piece_id=G4TMZ6IAIZBDON3UEDPH4LJQFHV3JLALEXSJ3VI4XN6WHIZOB57Q, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.469411962 (415h3m6.297120355s) [noise=true , address=145.79.227.48:4301 , piece_id=LBIYTJNOX5J4M3HDE7FGF5CSARBITW2AK3P2DSYFLFKUOIHZUKOQ, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.469749379 (415h3m6.297457773s) [noise=true , address=145.79.227.37:4565 , piece_id=554RHTYPG27UNCFPSRMOB245AX4QXL2QFIYECNTVWNNJMAL4BTYA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.470080828 (415h3m6.29778922s) [noise=true , address=145.79.227.43:4157 , piece_id=3W3LVTU37DNHWS52IGIIH7SXWXVNPGEY27RQNT272L3Z64GNR4KA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.470477660 (415h3m6.298186058s) [noise=true , address=45.91.116.14:4190 , piece_id=V4T6Z7K3C75LQEXNQIYZGYWW2CSQ7UPSET46QGDRT52RFGAROTTA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.470567977 (415h3m6.29827637s) [noise=true , address=145.79.227.15:4874 , piece_id=FNCCIMIVJTGZ2X5TOYOZH727JXHYM7HG3HBHBTZDAG67SHZLYYYA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.470980950 (415h3m6.298689349s) [noise=true , address=145.79.227.50:4373 , piece_id=4U55T5QDYCE6AZ3NCTPAVECPAELPQUKT3HV7GZL7JFRMWRN2AZUQ, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.471440098 (415h3m6.299148489s) [noise=true , address=145.79.227.47:4266 , piece_id=3NFYV2PWBNTAOSFHENM4J5PZ7XXE72UKTQVXLTVQRHGK4DUXAKDA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.560766475 (415h3m6.388474871s) [upload=431917]: Next returned nil segment Dec 22 22:31:05.560785161 (415h3m6.388493562s) [upload=431917]: Waiting for error group managing segments... Dec 22 22:31:05.566117878 (415h3m6.39382628s) [noise=true , address=45.91.116.4:4939 , piece_id=QYNKMKDLQJV7B6M2E34DU53Y7YI6VBCUW3C7ZLQOI4XWTBXZHKDQ, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.566723306 (415h3m6.394431687s) [noise=true , address=45.91.116.11:4117 , piece_id=WLMO5E3BIFFL3TOAUSSBGORIZ6DOCWFCMVT4AVFPPGPSKP2OKDAA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.566992667 (415h3m6.394701063s) [noise=true , address=145.79.227.36:4519 , piece_id=JUKRRIZWVGKEDD3GPJ7HMY2MEQZNMUCTKTXR6ZLSIERRONTZKHCQ, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.567464626 (415h3m6.395173019s) [noise=true , address=145.79.227.47:4266 , piece_id=3NFYV2PWBNTAOSFHENM4J5PZ7XXE72UKTQVXLTVQRHGK4DUXAKDA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.567924492 (415h3m6.395632883s) [noise=true , address=145.79.227.15:4874 , piece_id=FNCCIMIVJTGZ2X5TOYOZH727JXHYM7HG3HBHBTZDAG67SHZLYYYA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.568807294 (415h3m6.396515685s) [noise=true , address=145.79.227.51:4409 , piece_id=ATH3F4DFOUUIFFJZRA2SSRRL7AKWZMQNQRYOZCVFFDLTFCFP236A, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.570579823 (415h3m6.398288214s) [noise=true , address=145.79.227.48:4301 , piece_id=LBIYTJNOX5J4M3HDE7FGF5CSARBITW2AK3P2DSYFLFKUOIHZUKOQ, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.571632758 (415h3m6.399341149s) [noise=true , address=145.79.227.43:4157 , piece_id=3W3LVTU37DNHWS52IGIIH7SXWXVNPGEY27RQNT272L3Z64GNR4KA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.573168383 (415h3m6.400876774s) [noise=true , address=145.79.227.38:4979 , piece_id=HLMQC5OEB4OL4OZSRMCEVAWR5QM2T4MQUBPTHRVQLEAHPBWPQF6Q, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.579995420 (415h3m6.407703816s) [noise=true , address=45.91.116.14:4190 , piece_id=V4T6Z7K3C75LQEXNQIYZGYWW2CSQ7UPSET46QGDRT52RFGAROTTA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.592093172 (415h3m6.419801563s) [noise=true , address=145.79.227.37:4565 , piece_id=554RHTYPG27UNCFPSRMOB245AX4QXL2QFIYECNTVWNNJMAL4BTYA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:09.946500434 (415h3m10.774208816s) [noise=true , address=145.79.227.40:4051 , piece_id=55RRAD7DQX6SR7RBLYAV7MD7XRWPNIWHTVVBQZ2ZL3OE2L7LRRIQ, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:35.398235543 (415h3m36.225943943s) [noise=true , address=145.79.227.39:4013 , piece_id=JJVURGSLKY2UQ4FKNCLATA3ZBD5SDL32XJAG42KH2U6OYK3KTS6Q, seg_pos={0 0}, upload=431917]: Done uploading piece. err: ecclient: upload failed (node:121scLvKEW9zw2aEz22zpremcjiop67eQ8v2uqpmaD5aZzCAP7b, address:145.79.227.39:4013): protocol: write tcp 159.223.143.39:60842->145.79.227.39:4013: use of closed network connection; write tcp 159.223.143.39:60842->145.79.227.39:4013: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:60842->145.79.227.39:4013: use of closed network connection Dec 22 22:31:35.398562305 (415h3m36.226270699s) [noise=true , address=145.79.227.32:4835 , piece_id=BRE3JH3XKPXB6FN7WB3W6ED7MOFES6HYLR34YGR6L5AX7RPIQ7IA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: ecclient: upload failed (node:16vVfeoqCkbMYCRjMMWaKBnCvQBBJPMgwQtWTq9HKQrD9SS17y, address:145.79.227.32:4835): protocol: write tcp 159.223.143.39:45098->145.79.227.32:4835: use of closed network connection; write tcp 159.223.143.39:45098->145.79.227.32:4835: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:45098->145.79.227.32:4835: use of closed network connection Dec 22 22:31:35.399183028 (415h3m36.226891422s) [noise=true , address=145.79.227.50:4373 , piece_id=4U55T5QDYCE6AZ3NCTPAVECPAELPQUKT3HV7GZL7JFRMWRN2AZUQ, seg_pos={0 0}, upload=431917]: Done uploading piece. err: ecclient: upload failed (node:12AQGSkJuNJDxoeqhChUenWxHwF5yK41UESCF63FQz3XvabXXfz, address:145.79.227.50:4373): protocol: read tcp 159.223.143.39:56028->145.79.227.50:4373: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 22 22:31:35.399278021 (415h3m36.226986417s) [noise=true , address=145.79.227.29:4943 , piece_id=G4TMZ6IAIZBDON3UEDPH4LJQFHV3JLALEXSJ3VI4XN6WHIZOB57Q, seg_pos={0 0}, upload=431917]: Done uploading piece. err: ecclient: upload failed (node:12fc5Trj8XxacZhzxdksUVT5CbG6uTmiwdUPjreYi4UMPY1TXWz, address:145.79.227.29:4943): protocol: write tcp 159.223.143.39:50820->145.79.227.29:4943: use of closed network connection; write tcp 159.223.143.39:50820->145.79.227.29:4943: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:50820->145.79.227.29:4943: use of closed network connection Dec 22 22:31:35.399379485 (415h3m36.227087866s) [seg_pos={0 0}, upload=431917]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 22 22:31:35.421122544 (415h3m36.248830935s) [upload=431917]: Done uploading segments... Dec 22 22:31:35.421174369 (415h3m36.248882769s) [upload=431917]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 541.565843ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................X........................................] duration: 487.175254ms trace: https://jaeger.dev.storj.io/trace/7013ffb1a981d605 FAIL (2025-12-22 23:02:12 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 22 22:31:05.439510308 (415h3m6.2672187s) [upload=431917]: Starting upload Dec 22 22:31:05.439581993 (415h3m6.26729037s) [upload=431917]: Done starting upload Dec 22 22:31:05.439812792 (415h3m6.267521185s) [upload=431917]: Uploading segments... Dec 22 22:31:05.439942524 (415h3m6.267650918s) [upload=431917]: Got next segment. Inline: false Dec 22 22:31:05.440022744 (415h3m6.267731174s) [upload=431917]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 22 22:31:05.440029670 (415h3m6.267738072s) [upload=431917]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 22 22:31:05.467186372 (415h3m6.294894769s) [seg_pos={0 0}, upload=431917]: Begin upload segment... Dec 22 22:31:05.467265149 (415h3m6.294973535s) [seg_pos={0 0}, upload=431917]: Done begin upload segment. Dec 22 22:31:05.467296974 (415h3m6.295005373s) [noise=true , address=145.79.227.51:4409 , piece_id=ATH3F4DFOUUIFFJZRA2SSRRL7AKWZMQNQRYOZCVFFDLTFCFP236A, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.467389853 (415h3m6.295098247s) [noise=true , address=145.79.227.39:4013 , piece_id=JJVURGSLKY2UQ4FKNCLATA3ZBD5SDL32XJAG42KH2U6OYK3KTS6Q, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.467912104 (415h3m6.295620486s) [noise=true , address=145.79.227.36:4519 , piece_id=JUKRRIZWVGKEDD3GPJ7HMY2MEQZNMUCTKTXR6ZLSIERRONTZKHCQ, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.468324863 (415h3m6.296033263s) [noise=true , address=145.79.227.38:4979 , piece_id=HLMQC5OEB4OL4OZSRMCEVAWR5QM2T4MQUBPTHRVQLEAHPBWPQF6Q, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.468334764 (415h3m6.296043147s) [noise=true , address=145.79.227.40:4051 , piece_id=55RRAD7DQX6SR7RBLYAV7MD7XRWPNIWHTVVBQZ2ZL3OE2L7LRRIQ, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.468742525 (415h3m6.296450916s) [noise=true , address=45.91.116.4:4939 , piece_id=QYNKMKDLQJV7B6M2E34DU53Y7YI6VBCUW3C7ZLQOI4XWTBXZHKDQ, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.468915034 (415h3m6.296623414s) [noise=true , address=45.91.116.11:4117 , piece_id=WLMO5E3BIFFL3TOAUSSBGORIZ6DOCWFCMVT4AVFPPGPSKP2OKDAA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.469069393 (415h3m6.296777787s) [noise=true , address=145.79.227.32:4835 , piece_id=BRE3JH3XKPXB6FN7WB3W6ED7MOFES6HYLR34YGR6L5AX7RPIQ7IA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.469285789 (415h3m6.296994183s) [noise=true , address=145.79.227.29:4943 , piece_id=G4TMZ6IAIZBDON3UEDPH4LJQFHV3JLALEXSJ3VI4XN6WHIZOB57Q, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.469411962 (415h3m6.297120355s) [noise=true , address=145.79.227.48:4301 , piece_id=LBIYTJNOX5J4M3HDE7FGF5CSARBITW2AK3P2DSYFLFKUOIHZUKOQ, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.469749379 (415h3m6.297457773s) [noise=true , address=145.79.227.37:4565 , piece_id=554RHTYPG27UNCFPSRMOB245AX4QXL2QFIYECNTVWNNJMAL4BTYA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.470080828 (415h3m6.29778922s) [noise=true , address=145.79.227.43:4157 , piece_id=3W3LVTU37DNHWS52IGIIH7SXWXVNPGEY27RQNT272L3Z64GNR4KA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.470477660 (415h3m6.298186058s) [noise=true , address=45.91.116.14:4190 , piece_id=V4T6Z7K3C75LQEXNQIYZGYWW2CSQ7UPSET46QGDRT52RFGAROTTA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.470567977 (415h3m6.29827637s) [noise=true , address=145.79.227.15:4874 , piece_id=FNCCIMIVJTGZ2X5TOYOZH727JXHYM7HG3HBHBTZDAG67SHZLYYYA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.470980950 (415h3m6.298689349s) [noise=true , address=145.79.227.50:4373 , piece_id=4U55T5QDYCE6AZ3NCTPAVECPAELPQUKT3HV7GZL7JFRMWRN2AZUQ, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.471440098 (415h3m6.299148489s) [noise=true , address=145.79.227.47:4266 , piece_id=3NFYV2PWBNTAOSFHENM4J5PZ7XXE72UKTQVXLTVQRHGK4DUXAKDA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.560766475 (415h3m6.388474871s) [upload=431917]: Next returned nil segment Dec 22 22:31:05.560785161 (415h3m6.388493562s) [upload=431917]: Waiting for error group managing segments... Dec 22 22:31:05.566117878 (415h3m6.39382628s) [noise=true , address=45.91.116.4:4939 , piece_id=QYNKMKDLQJV7B6M2E34DU53Y7YI6VBCUW3C7ZLQOI4XWTBXZHKDQ, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.566723306 (415h3m6.394431687s) [noise=true , address=45.91.116.11:4117 , piece_id=WLMO5E3BIFFL3TOAUSSBGORIZ6DOCWFCMVT4AVFPPGPSKP2OKDAA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.566992667 (415h3m6.394701063s) [noise=true , address=145.79.227.36:4519 , piece_id=JUKRRIZWVGKEDD3GPJ7HMY2MEQZNMUCTKTXR6ZLSIERRONTZKHCQ, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.567464626 (415h3m6.395173019s) [noise=true , address=145.79.227.47:4266 , piece_id=3NFYV2PWBNTAOSFHENM4J5PZ7XXE72UKTQVXLTVQRHGK4DUXAKDA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.567924492 (415h3m6.395632883s) [noise=true , address=145.79.227.15:4874 , piece_id=FNCCIMIVJTGZ2X5TOYOZH727JXHYM7HG3HBHBTZDAG67SHZLYYYA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.568807294 (415h3m6.396515685s) [noise=true , address=145.79.227.51:4409 , piece_id=ATH3F4DFOUUIFFJZRA2SSRRL7AKWZMQNQRYOZCVFFDLTFCFP236A, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.570579823 (415h3m6.398288214s) [noise=true , address=145.79.227.48:4301 , piece_id=LBIYTJNOX5J4M3HDE7FGF5CSARBITW2AK3P2DSYFLFKUOIHZUKOQ, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.571632758 (415h3m6.399341149s) [noise=true , address=145.79.227.43:4157 , piece_id=3W3LVTU37DNHWS52IGIIH7SXWXVNPGEY27RQNT272L3Z64GNR4KA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.573168383 (415h3m6.400876774s) [noise=true , address=145.79.227.38:4979 , piece_id=HLMQC5OEB4OL4OZSRMCEVAWR5QM2T4MQUBPTHRVQLEAHPBWPQF6Q, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.579995420 (415h3m6.407703816s) [noise=true , address=45.91.116.14:4190 , piece_id=V4T6Z7K3C75LQEXNQIYZGYWW2CSQ7UPSET46QGDRT52RFGAROTTA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.592093172 (415h3m6.419801563s) [noise=true , address=145.79.227.37:4565 , piece_id=554RHTYPG27UNCFPSRMOB245AX4QXL2QFIYECNTVWNNJMAL4BTYA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:09.946500434 (415h3m10.774208816s) [noise=true , address=145.79.227.40:4051 , piece_id=55RRAD7DQX6SR7RBLYAV7MD7XRWPNIWHTVVBQZ2ZL3OE2L7LRRIQ, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:35.398235543 (415h3m36.225943943s) [noise=true , address=145.79.227.39:4013 , piece_id=JJVURGSLKY2UQ4FKNCLATA3ZBD5SDL32XJAG42KH2U6OYK3KTS6Q, seg_pos={0 0}, upload=431917]: Done uploading piece. err: ecclient: upload failed (node:121scLvKEW9zw2aEz22zpremcjiop67eQ8v2uqpmaD5aZzCAP7b, address:145.79.227.39:4013): protocol: write tcp 159.223.143.39:60842->145.79.227.39:4013: use of closed network connection; write tcp 159.223.143.39:60842->145.79.227.39:4013: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:60842->145.79.227.39:4013: use of closed network connection Dec 22 22:31:35.398562305 (415h3m36.226270699s) [noise=true , address=145.79.227.32:4835 , piece_id=BRE3JH3XKPXB6FN7WB3W6ED7MOFES6HYLR34YGR6L5AX7RPIQ7IA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: ecclient: upload failed (node:16vVfeoqCkbMYCRjMMWaKBnCvQBBJPMgwQtWTq9HKQrD9SS17y, address:145.79.227.32:4835): protocol: write tcp 159.223.143.39:45098->145.79.227.32:4835: use of closed network connection; write tcp 159.223.143.39:45098->145.79.227.32:4835: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:45098->145.79.227.32:4835: use of closed network connection Dec 22 22:31:35.399183028 (415h3m36.226891422s) [noise=true , address=145.79.227.50:4373 , piece_id=4U55T5QDYCE6AZ3NCTPAVECPAELPQUKT3HV7GZL7JFRMWRN2AZUQ, seg_pos={0 0}, upload=431917]: Done uploading piece. err: ecclient: upload failed (node:12AQGSkJuNJDxoeqhChUenWxHwF5yK41UESCF63FQz3XvabXXfz, address:145.79.227.50:4373): protocol: read tcp 159.223.143.39:56028->145.79.227.50:4373: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 22 22:31:35.399278021 (415h3m36.226986417s) [noise=true , address=145.79.227.29:4943 , piece_id=G4TMZ6IAIZBDON3UEDPH4LJQFHV3JLALEXSJ3VI4XN6WHIZOB57Q, seg_pos={0 0}, upload=431917]: Done uploading piece. err: ecclient: upload failed (node:12fc5Trj8XxacZhzxdksUVT5CbG6uTmiwdUPjreYi4UMPY1TXWz, address:145.79.227.29:4943): protocol: write tcp 159.223.143.39:50820->145.79.227.29:4943: use of closed network connection; write tcp 159.223.143.39:50820->145.79.227.29:4943: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:50820->145.79.227.29:4943: use of closed network connection Dec 22 22:31:35.399379485 (415h3m36.227087866s) [seg_pos={0 0}, upload=431917]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 22 22:31:35.421122544 (415h3m36.248830935s) [upload=431917]: Done uploading segments... Dec 22 22:31:35.421174369 (415h3m36.248882769s) [upload=431917]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 487.203488ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................X.......................................] duration: 547.937283ms trace: https://jaeger.dev.storj.io/trace/7013ffb1a981d605 FAIL (2025-12-22 23:01:57 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 22 22:31:05.439510308 (415h3m6.2672187s) [upload=431917]: Starting upload Dec 22 22:31:05.439581993 (415h3m6.26729037s) [upload=431917]: Done starting upload Dec 22 22:31:05.439812792 (415h3m6.267521185s) [upload=431917]: Uploading segments... Dec 22 22:31:05.439942524 (415h3m6.267650918s) [upload=431917]: Got next segment. Inline: false Dec 22 22:31:05.440022744 (415h3m6.267731174s) [upload=431917]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 22 22:31:05.440029670 (415h3m6.267738072s) [upload=431917]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 22 22:31:05.467186372 (415h3m6.294894769s) [seg_pos={0 0}, upload=431917]: Begin upload segment... Dec 22 22:31:05.467265149 (415h3m6.294973535s) [seg_pos={0 0}, upload=431917]: Done begin upload segment. Dec 22 22:31:05.467296974 (415h3m6.295005373s) [noise=true , address=145.79.227.51:4409 , piece_id=ATH3F4DFOUUIFFJZRA2SSRRL7AKWZMQNQRYOZCVFFDLTFCFP236A, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.467389853 (415h3m6.295098247s) [noise=true , address=145.79.227.39:4013 , piece_id=JJVURGSLKY2UQ4FKNCLATA3ZBD5SDL32XJAG42KH2U6OYK3KTS6Q, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.467912104 (415h3m6.295620486s) [noise=true , address=145.79.227.36:4519 , piece_id=JUKRRIZWVGKEDD3GPJ7HMY2MEQZNMUCTKTXR6ZLSIERRONTZKHCQ, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.468324863 (415h3m6.296033263s) [noise=true , address=145.79.227.38:4979 , piece_id=HLMQC5OEB4OL4OZSRMCEVAWR5QM2T4MQUBPTHRVQLEAHPBWPQF6Q, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.468334764 (415h3m6.296043147s) [noise=true , address=145.79.227.40:4051 , piece_id=55RRAD7DQX6SR7RBLYAV7MD7XRWPNIWHTVVBQZ2ZL3OE2L7LRRIQ, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.468742525 (415h3m6.296450916s) [noise=true , address=45.91.116.4:4939 , piece_id=QYNKMKDLQJV7B6M2E34DU53Y7YI6VBCUW3C7ZLQOI4XWTBXZHKDQ, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.468915034 (415h3m6.296623414s) [noise=true , address=45.91.116.11:4117 , piece_id=WLMO5E3BIFFL3TOAUSSBGORIZ6DOCWFCMVT4AVFPPGPSKP2OKDAA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.469069393 (415h3m6.296777787s) [noise=true , address=145.79.227.32:4835 , piece_id=BRE3JH3XKPXB6FN7WB3W6ED7MOFES6HYLR34YGR6L5AX7RPIQ7IA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.469285789 (415h3m6.296994183s) [noise=true , address=145.79.227.29:4943 , piece_id=G4TMZ6IAIZBDON3UEDPH4LJQFHV3JLALEXSJ3VI4XN6WHIZOB57Q, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.469411962 (415h3m6.297120355s) [noise=true , address=145.79.227.48:4301 , piece_id=LBIYTJNOX5J4M3HDE7FGF5CSARBITW2AK3P2DSYFLFKUOIHZUKOQ, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.469749379 (415h3m6.297457773s) [noise=true , address=145.79.227.37:4565 , piece_id=554RHTYPG27UNCFPSRMOB245AX4QXL2QFIYECNTVWNNJMAL4BTYA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.470080828 (415h3m6.29778922s) [noise=true , address=145.79.227.43:4157 , piece_id=3W3LVTU37DNHWS52IGIIH7SXWXVNPGEY27RQNT272L3Z64GNR4KA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.470477660 (415h3m6.298186058s) [noise=true , address=45.91.116.14:4190 , piece_id=V4T6Z7K3C75LQEXNQIYZGYWW2CSQ7UPSET46QGDRT52RFGAROTTA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.470567977 (415h3m6.29827637s) [noise=true , address=145.79.227.15:4874 , piece_id=FNCCIMIVJTGZ2X5TOYOZH727JXHYM7HG3HBHBTZDAG67SHZLYYYA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.470980950 (415h3m6.298689349s) [noise=true , address=145.79.227.50:4373 , piece_id=4U55T5QDYCE6AZ3NCTPAVECPAELPQUKT3HV7GZL7JFRMWRN2AZUQ, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.471440098 (415h3m6.299148489s) [noise=true , address=145.79.227.47:4266 , piece_id=3NFYV2PWBNTAOSFHENM4J5PZ7XXE72UKTQVXLTVQRHGK4DUXAKDA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.560766475 (415h3m6.388474871s) [upload=431917]: Next returned nil segment Dec 22 22:31:05.560785161 (415h3m6.388493562s) [upload=431917]: Waiting for error group managing segments... Dec 22 22:31:05.566117878 (415h3m6.39382628s) [noise=true , address=45.91.116.4:4939 , piece_id=QYNKMKDLQJV7B6M2E34DU53Y7YI6VBCUW3C7ZLQOI4XWTBXZHKDQ, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.566723306 (415h3m6.394431687s) [noise=true , address=45.91.116.11:4117 , piece_id=WLMO5E3BIFFL3TOAUSSBGORIZ6DOCWFCMVT4AVFPPGPSKP2OKDAA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.566992667 (415h3m6.394701063s) [noise=true , address=145.79.227.36:4519 , piece_id=JUKRRIZWVGKEDD3GPJ7HMY2MEQZNMUCTKTXR6ZLSIERRONTZKHCQ, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.567464626 (415h3m6.395173019s) [noise=true , address=145.79.227.47:4266 , piece_id=3NFYV2PWBNTAOSFHENM4J5PZ7XXE72UKTQVXLTVQRHGK4DUXAKDA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.567924492 (415h3m6.395632883s) [noise=true , address=145.79.227.15:4874 , piece_id=FNCCIMIVJTGZ2X5TOYOZH727JXHYM7HG3HBHBTZDAG67SHZLYYYA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.568807294 (415h3m6.396515685s) [noise=true , address=145.79.227.51:4409 , piece_id=ATH3F4DFOUUIFFJZRA2SSRRL7AKWZMQNQRYOZCVFFDLTFCFP236A, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.570579823 (415h3m6.398288214s) [noise=true , address=145.79.227.48:4301 , piece_id=LBIYTJNOX5J4M3HDE7FGF5CSARBITW2AK3P2DSYFLFKUOIHZUKOQ, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.571632758 (415h3m6.399341149s) [noise=true , address=145.79.227.43:4157 , piece_id=3W3LVTU37DNHWS52IGIIH7SXWXVNPGEY27RQNT272L3Z64GNR4KA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.573168383 (415h3m6.400876774s) [noise=true , address=145.79.227.38:4979 , piece_id=HLMQC5OEB4OL4OZSRMCEVAWR5QM2T4MQUBPTHRVQLEAHPBWPQF6Q, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.579995420 (415h3m6.407703816s) [noise=true , address=45.91.116.14:4190 , piece_id=V4T6Z7K3C75LQEXNQIYZGYWW2CSQ7UPSET46QGDRT52RFGAROTTA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.592093172 (415h3m6.419801563s) [noise=true , address=145.79.227.37:4565 , piece_id=554RHTYPG27UNCFPSRMOB245AX4QXL2QFIYECNTVWNNJMAL4BTYA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:09.946500434 (415h3m10.774208816s) [noise=true , address=145.79.227.40:4051 , piece_id=55RRAD7DQX6SR7RBLYAV7MD7XRWPNIWHTVVBQZ2ZL3OE2L7LRRIQ, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:35.398235543 (415h3m36.225943943s) [noise=true , address=145.79.227.39:4013 , piece_id=JJVURGSLKY2UQ4FKNCLATA3ZBD5SDL32XJAG42KH2U6OYK3KTS6Q, seg_pos={0 0}, upload=431917]: Done uploading piece. err: ecclient: upload failed (node:121scLvKEW9zw2aEz22zpremcjiop67eQ8v2uqpmaD5aZzCAP7b, address:145.79.227.39:4013): protocol: write tcp 159.223.143.39:60842->145.79.227.39:4013: use of closed network connection; write tcp 159.223.143.39:60842->145.79.227.39:4013: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:60842->145.79.227.39:4013: use of closed network connection Dec 22 22:31:35.398562305 (415h3m36.226270699s) [noise=true , address=145.79.227.32:4835 , piece_id=BRE3JH3XKPXB6FN7WB3W6ED7MOFES6HYLR34YGR6L5AX7RPIQ7IA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: ecclient: upload failed (node:16vVfeoqCkbMYCRjMMWaKBnCvQBBJPMgwQtWTq9HKQrD9SS17y, address:145.79.227.32:4835): protocol: write tcp 159.223.143.39:45098->145.79.227.32:4835: use of closed network connection; write tcp 159.223.143.39:45098->145.79.227.32:4835: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:45098->145.79.227.32:4835: use of closed network connection Dec 22 22:31:35.399183028 (415h3m36.226891422s) [noise=true , address=145.79.227.50:4373 , piece_id=4U55T5QDYCE6AZ3NCTPAVECPAELPQUKT3HV7GZL7JFRMWRN2AZUQ, seg_pos={0 0}, upload=431917]: Done uploading piece. err: ecclient: upload failed (node:12AQGSkJuNJDxoeqhChUenWxHwF5yK41UESCF63FQz3XvabXXfz, address:145.79.227.50:4373): protocol: read tcp 159.223.143.39:56028->145.79.227.50:4373: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 22 22:31:35.399278021 (415h3m36.226986417s) [noise=true , address=145.79.227.29:4943 , piece_id=G4TMZ6IAIZBDON3UEDPH4LJQFHV3JLALEXSJ3VI4XN6WHIZOB57Q, seg_pos={0 0}, upload=431917]: Done uploading piece. err: ecclient: upload failed (node:12fc5Trj8XxacZhzxdksUVT5CbG6uTmiwdUPjreYi4UMPY1TXWz, address:145.79.227.29:4943): protocol: write tcp 159.223.143.39:50820->145.79.227.29:4943: use of closed network connection; write tcp 159.223.143.39:50820->145.79.227.29:4943: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:50820->145.79.227.29:4943: use of closed network connection Dec 22 22:31:35.399379485 (415h3m36.227087866s) [seg_pos={0 0}, upload=431917]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 22 22:31:35.421122544 (415h3m36.248830935s) [upload=431917]: Done uploading segments... Dec 22 22:31:35.421174369 (415h3m36.248882769s) [upload=431917]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 547.962548ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................X......................................] duration: 426.23237ms trace: https://jaeger.dev.storj.io/trace/7013ffb1a981d605 FAIL (2025-12-22 23:01:00 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 22 22:31:05.439510308 (415h3m6.2672187s) [upload=431917]: Starting upload Dec 22 22:31:05.439581993 (415h3m6.26729037s) [upload=431917]: Done starting upload Dec 22 22:31:05.439812792 (415h3m6.267521185s) [upload=431917]: Uploading segments... Dec 22 22:31:05.439942524 (415h3m6.267650918s) [upload=431917]: Got next segment. Inline: false Dec 22 22:31:05.440022744 (415h3m6.267731174s) [upload=431917]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 22 22:31:05.440029670 (415h3m6.267738072s) [upload=431917]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 22 22:31:05.467186372 (415h3m6.294894769s) [seg_pos={0 0}, upload=431917]: Begin upload segment... Dec 22 22:31:05.467265149 (415h3m6.294973535s) [seg_pos={0 0}, upload=431917]: Done begin upload segment. Dec 22 22:31:05.467296974 (415h3m6.295005373s) [noise=true , address=145.79.227.51:4409 , piece_id=ATH3F4DFOUUIFFJZRA2SSRRL7AKWZMQNQRYOZCVFFDLTFCFP236A, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.467389853 (415h3m6.295098247s) [noise=true , address=145.79.227.39:4013 , piece_id=JJVURGSLKY2UQ4FKNCLATA3ZBD5SDL32XJAG42KH2U6OYK3KTS6Q, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.467912104 (415h3m6.295620486s) [noise=true , address=145.79.227.36:4519 , piece_id=JUKRRIZWVGKEDD3GPJ7HMY2MEQZNMUCTKTXR6ZLSIERRONTZKHCQ, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.468324863 (415h3m6.296033263s) [noise=true , address=145.79.227.38:4979 , piece_id=HLMQC5OEB4OL4OZSRMCEVAWR5QM2T4MQUBPTHRVQLEAHPBWPQF6Q, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.468334764 (415h3m6.296043147s) [noise=true , address=145.79.227.40:4051 , piece_id=55RRAD7DQX6SR7RBLYAV7MD7XRWPNIWHTVVBQZ2ZL3OE2L7LRRIQ, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.468742525 (415h3m6.296450916s) [noise=true , address=45.91.116.4:4939 , piece_id=QYNKMKDLQJV7B6M2E34DU53Y7YI6VBCUW3C7ZLQOI4XWTBXZHKDQ, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.468915034 (415h3m6.296623414s) [noise=true , address=45.91.116.11:4117 , piece_id=WLMO5E3BIFFL3TOAUSSBGORIZ6DOCWFCMVT4AVFPPGPSKP2OKDAA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.469069393 (415h3m6.296777787s) [noise=true , address=145.79.227.32:4835 , piece_id=BRE3JH3XKPXB6FN7WB3W6ED7MOFES6HYLR34YGR6L5AX7RPIQ7IA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.469285789 (415h3m6.296994183s) [noise=true , address=145.79.227.29:4943 , piece_id=G4TMZ6IAIZBDON3UEDPH4LJQFHV3JLALEXSJ3VI4XN6WHIZOB57Q, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.469411962 (415h3m6.297120355s) [noise=true , address=145.79.227.48:4301 , piece_id=LBIYTJNOX5J4M3HDE7FGF5CSARBITW2AK3P2DSYFLFKUOIHZUKOQ, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.469749379 (415h3m6.297457773s) [noise=true , address=145.79.227.37:4565 , piece_id=554RHTYPG27UNCFPSRMOB245AX4QXL2QFIYECNTVWNNJMAL4BTYA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.470080828 (415h3m6.29778922s) [noise=true , address=145.79.227.43:4157 , piece_id=3W3LVTU37DNHWS52IGIIH7SXWXVNPGEY27RQNT272L3Z64GNR4KA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.470477660 (415h3m6.298186058s) [noise=true , address=45.91.116.14:4190 , piece_id=V4T6Z7K3C75LQEXNQIYZGYWW2CSQ7UPSET46QGDRT52RFGAROTTA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.470567977 (415h3m6.29827637s) [noise=true , address=145.79.227.15:4874 , piece_id=FNCCIMIVJTGZ2X5TOYOZH727JXHYM7HG3HBHBTZDAG67SHZLYYYA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.470980950 (415h3m6.298689349s) [noise=true , address=145.79.227.50:4373 , piece_id=4U55T5QDYCE6AZ3NCTPAVECPAELPQUKT3HV7GZL7JFRMWRN2AZUQ, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.471440098 (415h3m6.299148489s) [noise=true , address=145.79.227.47:4266 , piece_id=3NFYV2PWBNTAOSFHENM4J5PZ7XXE72UKTQVXLTVQRHGK4DUXAKDA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.560766475 (415h3m6.388474871s) [upload=431917]: Next returned nil segment Dec 22 22:31:05.560785161 (415h3m6.388493562s) [upload=431917]: Waiting for error group managing segments... Dec 22 22:31:05.566117878 (415h3m6.39382628s) [noise=true , address=45.91.116.4:4939 , piece_id=QYNKMKDLQJV7B6M2E34DU53Y7YI6VBCUW3C7ZLQOI4XWTBXZHKDQ, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.566723306 (415h3m6.394431687s) [noise=true , address=45.91.116.11:4117 , piece_id=WLMO5E3BIFFL3TOAUSSBGORIZ6DOCWFCMVT4AVFPPGPSKP2OKDAA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.566992667 (415h3m6.394701063s) [noise=true , address=145.79.227.36:4519 , piece_id=JUKRRIZWVGKEDD3GPJ7HMY2MEQZNMUCTKTXR6ZLSIERRONTZKHCQ, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.567464626 (415h3m6.395173019s) [noise=true , address=145.79.227.47:4266 , piece_id=3NFYV2PWBNTAOSFHENM4J5PZ7XXE72UKTQVXLTVQRHGK4DUXAKDA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.567924492 (415h3m6.395632883s) [noise=true , address=145.79.227.15:4874 , piece_id=FNCCIMIVJTGZ2X5TOYOZH727JXHYM7HG3HBHBTZDAG67SHZLYYYA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.568807294 (415h3m6.396515685s) [noise=true , address=145.79.227.51:4409 , piece_id=ATH3F4DFOUUIFFJZRA2SSRRL7AKWZMQNQRYOZCVFFDLTFCFP236A, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.570579823 (415h3m6.398288214s) [noise=true , address=145.79.227.48:4301 , piece_id=LBIYTJNOX5J4M3HDE7FGF5CSARBITW2AK3P2DSYFLFKUOIHZUKOQ, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.571632758 (415h3m6.399341149s) [noise=true , address=145.79.227.43:4157 , piece_id=3W3LVTU37DNHWS52IGIIH7SXWXVNPGEY27RQNT272L3Z64GNR4KA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.573168383 (415h3m6.400876774s) [noise=true , address=145.79.227.38:4979 , piece_id=HLMQC5OEB4OL4OZSRMCEVAWR5QM2T4MQUBPTHRVQLEAHPBWPQF6Q, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.579995420 (415h3m6.407703816s) [noise=true , address=45.91.116.14:4190 , piece_id=V4T6Z7K3C75LQEXNQIYZGYWW2CSQ7UPSET46QGDRT52RFGAROTTA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.592093172 (415h3m6.419801563s) [noise=true , address=145.79.227.37:4565 , piece_id=554RHTYPG27UNCFPSRMOB245AX4QXL2QFIYECNTVWNNJMAL4BTYA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:09.946500434 (415h3m10.774208816s) [noise=true , address=145.79.227.40:4051 , piece_id=55RRAD7DQX6SR7RBLYAV7MD7XRWPNIWHTVVBQZ2ZL3OE2L7LRRIQ, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:35.398235543 (415h3m36.225943943s) [noise=true , address=145.79.227.39:4013 , piece_id=JJVURGSLKY2UQ4FKNCLATA3ZBD5SDL32XJAG42KH2U6OYK3KTS6Q, seg_pos={0 0}, upload=431917]: Done uploading piece. err: ecclient: upload failed (node:121scLvKEW9zw2aEz22zpremcjiop67eQ8v2uqpmaD5aZzCAP7b, address:145.79.227.39:4013): protocol: write tcp 159.223.143.39:60842->145.79.227.39:4013: use of closed network connection; write tcp 159.223.143.39:60842->145.79.227.39:4013: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:60842->145.79.227.39:4013: use of closed network connection Dec 22 22:31:35.398562305 (415h3m36.226270699s) [noise=true , address=145.79.227.32:4835 , piece_id=BRE3JH3XKPXB6FN7WB3W6ED7MOFES6HYLR34YGR6L5AX7RPIQ7IA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: ecclient: upload failed (node:16vVfeoqCkbMYCRjMMWaKBnCvQBBJPMgwQtWTq9HKQrD9SS17y, address:145.79.227.32:4835): protocol: write tcp 159.223.143.39:45098->145.79.227.32:4835: use of closed network connection; write tcp 159.223.143.39:45098->145.79.227.32:4835: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:45098->145.79.227.32:4835: use of closed network connection Dec 22 22:31:35.399183028 (415h3m36.226891422s) [noise=true , address=145.79.227.50:4373 , piece_id=4U55T5QDYCE6AZ3NCTPAVECPAELPQUKT3HV7GZL7JFRMWRN2AZUQ, seg_pos={0 0}, upload=431917]: Done uploading piece. err: ecclient: upload failed (node:12AQGSkJuNJDxoeqhChUenWxHwF5yK41UESCF63FQz3XvabXXfz, address:145.79.227.50:4373): protocol: read tcp 159.223.143.39:56028->145.79.227.50:4373: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 22 22:31:35.399278021 (415h3m36.226986417s) [noise=true , address=145.79.227.29:4943 , piece_id=G4TMZ6IAIZBDON3UEDPH4LJQFHV3JLALEXSJ3VI4XN6WHIZOB57Q, seg_pos={0 0}, upload=431917]: Done uploading piece. err: ecclient: upload failed (node:12fc5Trj8XxacZhzxdksUVT5CbG6uTmiwdUPjreYi4UMPY1TXWz, address:145.79.227.29:4943): protocol: write tcp 159.223.143.39:50820->145.79.227.29:4943: use of closed network connection; write tcp 159.223.143.39:50820->145.79.227.29:4943: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:50820->145.79.227.29:4943: use of closed network connection Dec 22 22:31:35.399379485 (415h3m36.227087866s) [seg_pos={0 0}, upload=431917]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 22 22:31:35.421122544 (415h3m36.248830935s) [upload=431917]: Done uploading segments... Dec 22 22:31:35.421174369 (415h3m36.248882769s) [upload=431917]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 426.256299ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................X.....................................] duration: 601.923656ms trace: https://jaeger.dev.storj.io/trace/7013ffb1a981d605 FAIL (2025-12-22 23:00:15 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 22 22:31:05.439510308 (415h3m6.2672187s) [upload=431917]: Starting upload Dec 22 22:31:05.439581993 (415h3m6.26729037s) [upload=431917]: Done starting upload Dec 22 22:31:05.439812792 (415h3m6.267521185s) [upload=431917]: Uploading segments... Dec 22 22:31:05.439942524 (415h3m6.267650918s) [upload=431917]: Got next segment. Inline: false Dec 22 22:31:05.440022744 (415h3m6.267731174s) [upload=431917]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 22 22:31:05.440029670 (415h3m6.267738072s) [upload=431917]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 22 22:31:05.467186372 (415h3m6.294894769s) [seg_pos={0 0}, upload=431917]: Begin upload segment... Dec 22 22:31:05.467265149 (415h3m6.294973535s) [seg_pos={0 0}, upload=431917]: Done begin upload segment. Dec 22 22:31:05.467296974 (415h3m6.295005373s) [noise=true , address=145.79.227.51:4409 , piece_id=ATH3F4DFOUUIFFJZRA2SSRRL7AKWZMQNQRYOZCVFFDLTFCFP236A, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.467389853 (415h3m6.295098247s) [noise=true , address=145.79.227.39:4013 , piece_id=JJVURGSLKY2UQ4FKNCLATA3ZBD5SDL32XJAG42KH2U6OYK3KTS6Q, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.467912104 (415h3m6.295620486s) [noise=true , address=145.79.227.36:4519 , piece_id=JUKRRIZWVGKEDD3GPJ7HMY2MEQZNMUCTKTXR6ZLSIERRONTZKHCQ, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.468324863 (415h3m6.296033263s) [noise=true , address=145.79.227.38:4979 , piece_id=HLMQC5OEB4OL4OZSRMCEVAWR5QM2T4MQUBPTHRVQLEAHPBWPQF6Q, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.468334764 (415h3m6.296043147s) [noise=true , address=145.79.227.40:4051 , piece_id=55RRAD7DQX6SR7RBLYAV7MD7XRWPNIWHTVVBQZ2ZL3OE2L7LRRIQ, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.468742525 (415h3m6.296450916s) [noise=true , address=45.91.116.4:4939 , piece_id=QYNKMKDLQJV7B6M2E34DU53Y7YI6VBCUW3C7ZLQOI4XWTBXZHKDQ, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.468915034 (415h3m6.296623414s) [noise=true , address=45.91.116.11:4117 , piece_id=WLMO5E3BIFFL3TOAUSSBGORIZ6DOCWFCMVT4AVFPPGPSKP2OKDAA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.469069393 (415h3m6.296777787s) [noise=true , address=145.79.227.32:4835 , piece_id=BRE3JH3XKPXB6FN7WB3W6ED7MOFES6HYLR34YGR6L5AX7RPIQ7IA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.469285789 (415h3m6.296994183s) [noise=true , address=145.79.227.29:4943 , piece_id=G4TMZ6IAIZBDON3UEDPH4LJQFHV3JLALEXSJ3VI4XN6WHIZOB57Q, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.469411962 (415h3m6.297120355s) [noise=true , address=145.79.227.48:4301 , piece_id=LBIYTJNOX5J4M3HDE7FGF5CSARBITW2AK3P2DSYFLFKUOIHZUKOQ, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.469749379 (415h3m6.297457773s) [noise=true , address=145.79.227.37:4565 , piece_id=554RHTYPG27UNCFPSRMOB245AX4QXL2QFIYECNTVWNNJMAL4BTYA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.470080828 (415h3m6.29778922s) [noise=true , address=145.79.227.43:4157 , piece_id=3W3LVTU37DNHWS52IGIIH7SXWXVNPGEY27RQNT272L3Z64GNR4KA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.470477660 (415h3m6.298186058s) [noise=true , address=45.91.116.14:4190 , piece_id=V4T6Z7K3C75LQEXNQIYZGYWW2CSQ7UPSET46QGDRT52RFGAROTTA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.470567977 (415h3m6.29827637s) [noise=true , address=145.79.227.15:4874 , piece_id=FNCCIMIVJTGZ2X5TOYOZH727JXHYM7HG3HBHBTZDAG67SHZLYYYA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.470980950 (415h3m6.298689349s) [noise=true , address=145.79.227.50:4373 , piece_id=4U55T5QDYCE6AZ3NCTPAVECPAELPQUKT3HV7GZL7JFRMWRN2AZUQ, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.471440098 (415h3m6.299148489s) [noise=true , address=145.79.227.47:4266 , piece_id=3NFYV2PWBNTAOSFHENM4J5PZ7XXE72UKTQVXLTVQRHGK4DUXAKDA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.560766475 (415h3m6.388474871s) [upload=431917]: Next returned nil segment Dec 22 22:31:05.560785161 (415h3m6.388493562s) [upload=431917]: Waiting for error group managing segments... Dec 22 22:31:05.566117878 (415h3m6.39382628s) [noise=true , address=45.91.116.4:4939 , piece_id=QYNKMKDLQJV7B6M2E34DU53Y7YI6VBCUW3C7ZLQOI4XWTBXZHKDQ, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.566723306 (415h3m6.394431687s) [noise=true , address=45.91.116.11:4117 , piece_id=WLMO5E3BIFFL3TOAUSSBGORIZ6DOCWFCMVT4AVFPPGPSKP2OKDAA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.566992667 (415h3m6.394701063s) [noise=true , address=145.79.227.36:4519 , piece_id=JUKRRIZWVGKEDD3GPJ7HMY2MEQZNMUCTKTXR6ZLSIERRONTZKHCQ, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.567464626 (415h3m6.395173019s) [noise=true , address=145.79.227.47:4266 , piece_id=3NFYV2PWBNTAOSFHENM4J5PZ7XXE72UKTQVXLTVQRHGK4DUXAKDA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.567924492 (415h3m6.395632883s) [noise=true , address=145.79.227.15:4874 , piece_id=FNCCIMIVJTGZ2X5TOYOZH727JXHYM7HG3HBHBTZDAG67SHZLYYYA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.568807294 (415h3m6.396515685s) [noise=true , address=145.79.227.51:4409 , piece_id=ATH3F4DFOUUIFFJZRA2SSRRL7AKWZMQNQRYOZCVFFDLTFCFP236A, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.570579823 (415h3m6.398288214s) [noise=true , address=145.79.227.48:4301 , piece_id=LBIYTJNOX5J4M3HDE7FGF5CSARBITW2AK3P2DSYFLFKUOIHZUKOQ, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.571632758 (415h3m6.399341149s) [noise=true , address=145.79.227.43:4157 , piece_id=3W3LVTU37DNHWS52IGIIH7SXWXVNPGEY27RQNT272L3Z64GNR4KA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.573168383 (415h3m6.400876774s) [noise=true , address=145.79.227.38:4979 , piece_id=HLMQC5OEB4OL4OZSRMCEVAWR5QM2T4MQUBPTHRVQLEAHPBWPQF6Q, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.579995420 (415h3m6.407703816s) [noise=true , address=45.91.116.14:4190 , piece_id=V4T6Z7K3C75LQEXNQIYZGYWW2CSQ7UPSET46QGDRT52RFGAROTTA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.592093172 (415h3m6.419801563s) [noise=true , address=145.79.227.37:4565 , piece_id=554RHTYPG27UNCFPSRMOB245AX4QXL2QFIYECNTVWNNJMAL4BTYA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:09.946500434 (415h3m10.774208816s) [noise=true , address=145.79.227.40:4051 , piece_id=55RRAD7DQX6SR7RBLYAV7MD7XRWPNIWHTVVBQZ2ZL3OE2L7LRRIQ, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:35.398235543 (415h3m36.225943943s) [noise=true , address=145.79.227.39:4013 , piece_id=JJVURGSLKY2UQ4FKNCLATA3ZBD5SDL32XJAG42KH2U6OYK3KTS6Q, seg_pos={0 0}, upload=431917]: Done uploading piece. err: ecclient: upload failed (node:121scLvKEW9zw2aEz22zpremcjiop67eQ8v2uqpmaD5aZzCAP7b, address:145.79.227.39:4013): protocol: write tcp 159.223.143.39:60842->145.79.227.39:4013: use of closed network connection; write tcp 159.223.143.39:60842->145.79.227.39:4013: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:60842->145.79.227.39:4013: use of closed network connection Dec 22 22:31:35.398562305 (415h3m36.226270699s) [noise=true , address=145.79.227.32:4835 , piece_id=BRE3JH3XKPXB6FN7WB3W6ED7MOFES6HYLR34YGR6L5AX7RPIQ7IA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: ecclient: upload failed (node:16vVfeoqCkbMYCRjMMWaKBnCvQBBJPMgwQtWTq9HKQrD9SS17y, address:145.79.227.32:4835): protocol: write tcp 159.223.143.39:45098->145.79.227.32:4835: use of closed network connection; write tcp 159.223.143.39:45098->145.79.227.32:4835: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:45098->145.79.227.32:4835: use of closed network connection Dec 22 22:31:35.399183028 (415h3m36.226891422s) [noise=true , address=145.79.227.50:4373 , piece_id=4U55T5QDYCE6AZ3NCTPAVECPAELPQUKT3HV7GZL7JFRMWRN2AZUQ, seg_pos={0 0}, upload=431917]: Done uploading piece. err: ecclient: upload failed (node:12AQGSkJuNJDxoeqhChUenWxHwF5yK41UESCF63FQz3XvabXXfz, address:145.79.227.50:4373): protocol: read tcp 159.223.143.39:56028->145.79.227.50:4373: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 22 22:31:35.399278021 (415h3m36.226986417s) [noise=true , address=145.79.227.29:4943 , piece_id=G4TMZ6IAIZBDON3UEDPH4LJQFHV3JLALEXSJ3VI4XN6WHIZOB57Q, seg_pos={0 0}, upload=431917]: Done uploading piece. err: ecclient: upload failed (node:12fc5Trj8XxacZhzxdksUVT5CbG6uTmiwdUPjreYi4UMPY1TXWz, address:145.79.227.29:4943): protocol: write tcp 159.223.143.39:50820->145.79.227.29:4943: use of closed network connection; write tcp 159.223.143.39:50820->145.79.227.29:4943: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:50820->145.79.227.29:4943: use of closed network connection Dec 22 22:31:35.399379485 (415h3m36.227087866s) [seg_pos={0 0}, upload=431917]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 22 22:31:35.421122544 (415h3m36.248830935s) [upload=431917]: Done uploading segments... Dec 22 22:31:35.421174369 (415h3m36.248882769s) [upload=431917]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 601.965912ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................X....................................] duration: 424.231581ms trace: https://jaeger.dev.storj.io/trace/7013ffb1a981d605 FAIL (2025-12-22 22:59:49 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 22 22:31:05.439510308 (415h3m6.2672187s) [upload=431917]: Starting upload Dec 22 22:31:05.439581993 (415h3m6.26729037s) [upload=431917]: Done starting upload Dec 22 22:31:05.439812792 (415h3m6.267521185s) [upload=431917]: Uploading segments... Dec 22 22:31:05.439942524 (415h3m6.267650918s) [upload=431917]: Got next segment. Inline: false Dec 22 22:31:05.440022744 (415h3m6.267731174s) [upload=431917]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 22 22:31:05.440029670 (415h3m6.267738072s) [upload=431917]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 22 22:31:05.467186372 (415h3m6.294894769s) [seg_pos={0 0}, upload=431917]: Begin upload segment... Dec 22 22:31:05.467265149 (415h3m6.294973535s) [seg_pos={0 0}, upload=431917]: Done begin upload segment. Dec 22 22:31:05.467296974 (415h3m6.295005373s) [noise=true , address=145.79.227.51:4409 , piece_id=ATH3F4DFOUUIFFJZRA2SSRRL7AKWZMQNQRYOZCVFFDLTFCFP236A, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.467389853 (415h3m6.295098247s) [noise=true , address=145.79.227.39:4013 , piece_id=JJVURGSLKY2UQ4FKNCLATA3ZBD5SDL32XJAG42KH2U6OYK3KTS6Q, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.467912104 (415h3m6.295620486s) [noise=true , address=145.79.227.36:4519 , piece_id=JUKRRIZWVGKEDD3GPJ7HMY2MEQZNMUCTKTXR6ZLSIERRONTZKHCQ, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.468324863 (415h3m6.296033263s) [noise=true , address=145.79.227.38:4979 , piece_id=HLMQC5OEB4OL4OZSRMCEVAWR5QM2T4MQUBPTHRVQLEAHPBWPQF6Q, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.468334764 (415h3m6.296043147s) [noise=true , address=145.79.227.40:4051 , piece_id=55RRAD7DQX6SR7RBLYAV7MD7XRWPNIWHTVVBQZ2ZL3OE2L7LRRIQ, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.468742525 (415h3m6.296450916s) [noise=true , address=45.91.116.4:4939 , piece_id=QYNKMKDLQJV7B6M2E34DU53Y7YI6VBCUW3C7ZLQOI4XWTBXZHKDQ, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.468915034 (415h3m6.296623414s) [noise=true , address=45.91.116.11:4117 , piece_id=WLMO5E3BIFFL3TOAUSSBGORIZ6DOCWFCMVT4AVFPPGPSKP2OKDAA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.469069393 (415h3m6.296777787s) [noise=true , address=145.79.227.32:4835 , piece_id=BRE3JH3XKPXB6FN7WB3W6ED7MOFES6HYLR34YGR6L5AX7RPIQ7IA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.469285789 (415h3m6.296994183s) [noise=true , address=145.79.227.29:4943 , piece_id=G4TMZ6IAIZBDON3UEDPH4LJQFHV3JLALEXSJ3VI4XN6WHIZOB57Q, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.469411962 (415h3m6.297120355s) [noise=true , address=145.79.227.48:4301 , piece_id=LBIYTJNOX5J4M3HDE7FGF5CSARBITW2AK3P2DSYFLFKUOIHZUKOQ, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.469749379 (415h3m6.297457773s) [noise=true , address=145.79.227.37:4565 , piece_id=554RHTYPG27UNCFPSRMOB245AX4QXL2QFIYECNTVWNNJMAL4BTYA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.470080828 (415h3m6.29778922s) [noise=true , address=145.79.227.43:4157 , piece_id=3W3LVTU37DNHWS52IGIIH7SXWXVNPGEY27RQNT272L3Z64GNR4KA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.470477660 (415h3m6.298186058s) [noise=true , address=45.91.116.14:4190 , piece_id=V4T6Z7K3C75LQEXNQIYZGYWW2CSQ7UPSET46QGDRT52RFGAROTTA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.470567977 (415h3m6.29827637s) [noise=true , address=145.79.227.15:4874 , piece_id=FNCCIMIVJTGZ2X5TOYOZH727JXHYM7HG3HBHBTZDAG67SHZLYYYA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.470980950 (415h3m6.298689349s) [noise=true , address=145.79.227.50:4373 , piece_id=4U55T5QDYCE6AZ3NCTPAVECPAELPQUKT3HV7GZL7JFRMWRN2AZUQ, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.471440098 (415h3m6.299148489s) [noise=true , address=145.79.227.47:4266 , piece_id=3NFYV2PWBNTAOSFHENM4J5PZ7XXE72UKTQVXLTVQRHGK4DUXAKDA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.560766475 (415h3m6.388474871s) [upload=431917]: Next returned nil segment Dec 22 22:31:05.560785161 (415h3m6.388493562s) [upload=431917]: Waiting for error group managing segments... Dec 22 22:31:05.566117878 (415h3m6.39382628s) [noise=true , address=45.91.116.4:4939 , piece_id=QYNKMKDLQJV7B6M2E34DU53Y7YI6VBCUW3C7ZLQOI4XWTBXZHKDQ, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.566723306 (415h3m6.394431687s) [noise=true , address=45.91.116.11:4117 , piece_id=WLMO5E3BIFFL3TOAUSSBGORIZ6DOCWFCMVT4AVFPPGPSKP2OKDAA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.566992667 (415h3m6.394701063s) [noise=true , address=145.79.227.36:4519 , piece_id=JUKRRIZWVGKEDD3GPJ7HMY2MEQZNMUCTKTXR6ZLSIERRONTZKHCQ, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.567464626 (415h3m6.395173019s) [noise=true , address=145.79.227.47:4266 , piece_id=3NFYV2PWBNTAOSFHENM4J5PZ7XXE72UKTQVXLTVQRHGK4DUXAKDA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.567924492 (415h3m6.395632883s) [noise=true , address=145.79.227.15:4874 , piece_id=FNCCIMIVJTGZ2X5TOYOZH727JXHYM7HG3HBHBTZDAG67SHZLYYYA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.568807294 (415h3m6.396515685s) [noise=true , address=145.79.227.51:4409 , piece_id=ATH3F4DFOUUIFFJZRA2SSRRL7AKWZMQNQRYOZCVFFDLTFCFP236A, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.570579823 (415h3m6.398288214s) [noise=true , address=145.79.227.48:4301 , piece_id=LBIYTJNOX5J4M3HDE7FGF5CSARBITW2AK3P2DSYFLFKUOIHZUKOQ, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.571632758 (415h3m6.399341149s) [noise=true , address=145.79.227.43:4157 , piece_id=3W3LVTU37DNHWS52IGIIH7SXWXVNPGEY27RQNT272L3Z64GNR4KA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.573168383 (415h3m6.400876774s) [noise=true , address=145.79.227.38:4979 , piece_id=HLMQC5OEB4OL4OZSRMCEVAWR5QM2T4MQUBPTHRVQLEAHPBWPQF6Q, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.579995420 (415h3m6.407703816s) [noise=true , address=45.91.116.14:4190 , piece_id=V4T6Z7K3C75LQEXNQIYZGYWW2CSQ7UPSET46QGDRT52RFGAROTTA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.592093172 (415h3m6.419801563s) [noise=true , address=145.79.227.37:4565 , piece_id=554RHTYPG27UNCFPSRMOB245AX4QXL2QFIYECNTVWNNJMAL4BTYA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:09.946500434 (415h3m10.774208816s) [noise=true , address=145.79.227.40:4051 , piece_id=55RRAD7DQX6SR7RBLYAV7MD7XRWPNIWHTVVBQZ2ZL3OE2L7LRRIQ, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:35.398235543 (415h3m36.225943943s) [noise=true , address=145.79.227.39:4013 , piece_id=JJVURGSLKY2UQ4FKNCLATA3ZBD5SDL32XJAG42KH2U6OYK3KTS6Q, seg_pos={0 0}, upload=431917]: Done uploading piece. err: ecclient: upload failed (node:121scLvKEW9zw2aEz22zpremcjiop67eQ8v2uqpmaD5aZzCAP7b, address:145.79.227.39:4013): protocol: write tcp 159.223.143.39:60842->145.79.227.39:4013: use of closed network connection; write tcp 159.223.143.39:60842->145.79.227.39:4013: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:60842->145.79.227.39:4013: use of closed network connection Dec 22 22:31:35.398562305 (415h3m36.226270699s) [noise=true , address=145.79.227.32:4835 , piece_id=BRE3JH3XKPXB6FN7WB3W6ED7MOFES6HYLR34YGR6L5AX7RPIQ7IA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: ecclient: upload failed (node:16vVfeoqCkbMYCRjMMWaKBnCvQBBJPMgwQtWTq9HKQrD9SS17y, address:145.79.227.32:4835): protocol: write tcp 159.223.143.39:45098->145.79.227.32:4835: use of closed network connection; write tcp 159.223.143.39:45098->145.79.227.32:4835: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:45098->145.79.227.32:4835: use of closed network connection Dec 22 22:31:35.399183028 (415h3m36.226891422s) [noise=true , address=145.79.227.50:4373 , piece_id=4U55T5QDYCE6AZ3NCTPAVECPAELPQUKT3HV7GZL7JFRMWRN2AZUQ, seg_pos={0 0}, upload=431917]: Done uploading piece. err: ecclient: upload failed (node:12AQGSkJuNJDxoeqhChUenWxHwF5yK41UESCF63FQz3XvabXXfz, address:145.79.227.50:4373): protocol: read tcp 159.223.143.39:56028->145.79.227.50:4373: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 22 22:31:35.399278021 (415h3m36.226986417s) [noise=true , address=145.79.227.29:4943 , piece_id=G4TMZ6IAIZBDON3UEDPH4LJQFHV3JLALEXSJ3VI4XN6WHIZOB57Q, seg_pos={0 0}, upload=431917]: Done uploading piece. err: ecclient: upload failed (node:12fc5Trj8XxacZhzxdksUVT5CbG6uTmiwdUPjreYi4UMPY1TXWz, address:145.79.227.29:4943): protocol: write tcp 159.223.143.39:50820->145.79.227.29:4943: use of closed network connection; write tcp 159.223.143.39:50820->145.79.227.29:4943: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:50820->145.79.227.29:4943: use of closed network connection Dec 22 22:31:35.399379485 (415h3m36.227087866s) [seg_pos={0 0}, upload=431917]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 22 22:31:35.421122544 (415h3m36.248830935s) [upload=431917]: Done uploading segments... Dec 22 22:31:35.421174369 (415h3m36.248882769s) [upload=431917]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 424.260363ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................X...................................] duration: 412.035994ms trace: https://jaeger.dev.storj.io/trace/7013ffb1a981d605 FAIL (2025-12-22 22:58:42 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 22 22:31:05.439510308 (415h3m6.2672187s) [upload=431917]: Starting upload Dec 22 22:31:05.439581993 (415h3m6.26729037s) [upload=431917]: Done starting upload Dec 22 22:31:05.439812792 (415h3m6.267521185s) [upload=431917]: Uploading segments... Dec 22 22:31:05.439942524 (415h3m6.267650918s) [upload=431917]: Got next segment. Inline: false Dec 22 22:31:05.440022744 (415h3m6.267731174s) [upload=431917]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 22 22:31:05.440029670 (415h3m6.267738072s) [upload=431917]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 22 22:31:05.467186372 (415h3m6.294894769s) [seg_pos={0 0}, upload=431917]: Begin upload segment... Dec 22 22:31:05.467265149 (415h3m6.294973535s) [seg_pos={0 0}, upload=431917]: Done begin upload segment. Dec 22 22:31:05.467296974 (415h3m6.295005373s) [noise=true , address=145.79.227.51:4409 , piece_id=ATH3F4DFOUUIFFJZRA2SSRRL7AKWZMQNQRYOZCVFFDLTFCFP236A, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.467389853 (415h3m6.295098247s) [noise=true , address=145.79.227.39:4013 , piece_id=JJVURGSLKY2UQ4FKNCLATA3ZBD5SDL32XJAG42KH2U6OYK3KTS6Q, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.467912104 (415h3m6.295620486s) [noise=true , address=145.79.227.36:4519 , piece_id=JUKRRIZWVGKEDD3GPJ7HMY2MEQZNMUCTKTXR6ZLSIERRONTZKHCQ, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.468324863 (415h3m6.296033263s) [noise=true , address=145.79.227.38:4979 , piece_id=HLMQC5OEB4OL4OZSRMCEVAWR5QM2T4MQUBPTHRVQLEAHPBWPQF6Q, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.468334764 (415h3m6.296043147s) [noise=true , address=145.79.227.40:4051 , piece_id=55RRAD7DQX6SR7RBLYAV7MD7XRWPNIWHTVVBQZ2ZL3OE2L7LRRIQ, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.468742525 (415h3m6.296450916s) [noise=true , address=45.91.116.4:4939 , piece_id=QYNKMKDLQJV7B6M2E34DU53Y7YI6VBCUW3C7ZLQOI4XWTBXZHKDQ, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.468915034 (415h3m6.296623414s) [noise=true , address=45.91.116.11:4117 , piece_id=WLMO5E3BIFFL3TOAUSSBGORIZ6DOCWFCMVT4AVFPPGPSKP2OKDAA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.469069393 (415h3m6.296777787s) [noise=true , address=145.79.227.32:4835 , piece_id=BRE3JH3XKPXB6FN7WB3W6ED7MOFES6HYLR34YGR6L5AX7RPIQ7IA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.469285789 (415h3m6.296994183s) [noise=true , address=145.79.227.29:4943 , piece_id=G4TMZ6IAIZBDON3UEDPH4LJQFHV3JLALEXSJ3VI4XN6WHIZOB57Q, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.469411962 (415h3m6.297120355s) [noise=true , address=145.79.227.48:4301 , piece_id=LBIYTJNOX5J4M3HDE7FGF5CSARBITW2AK3P2DSYFLFKUOIHZUKOQ, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.469749379 (415h3m6.297457773s) [noise=true , address=145.79.227.37:4565 , piece_id=554RHTYPG27UNCFPSRMOB245AX4QXL2QFIYECNTVWNNJMAL4BTYA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.470080828 (415h3m6.29778922s) [noise=true , address=145.79.227.43:4157 , piece_id=3W3LVTU37DNHWS52IGIIH7SXWXVNPGEY27RQNT272L3Z64GNR4KA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.470477660 (415h3m6.298186058s) [noise=true , address=45.91.116.14:4190 , piece_id=V4T6Z7K3C75LQEXNQIYZGYWW2CSQ7UPSET46QGDRT52RFGAROTTA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.470567977 (415h3m6.29827637s) [noise=true , address=145.79.227.15:4874 , piece_id=FNCCIMIVJTGZ2X5TOYOZH727JXHYM7HG3HBHBTZDAG67SHZLYYYA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.470980950 (415h3m6.298689349s) [noise=true , address=145.79.227.50:4373 , piece_id=4U55T5QDYCE6AZ3NCTPAVECPAELPQUKT3HV7GZL7JFRMWRN2AZUQ, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.471440098 (415h3m6.299148489s) [noise=true , address=145.79.227.47:4266 , piece_id=3NFYV2PWBNTAOSFHENM4J5PZ7XXE72UKTQVXLTVQRHGK4DUXAKDA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.560766475 (415h3m6.388474871s) [upload=431917]: Next returned nil segment Dec 22 22:31:05.560785161 (415h3m6.388493562s) [upload=431917]: Waiting for error group managing segments... Dec 22 22:31:05.566117878 (415h3m6.39382628s) [noise=true , address=45.91.116.4:4939 , piece_id=QYNKMKDLQJV7B6M2E34DU53Y7YI6VBCUW3C7ZLQOI4XWTBXZHKDQ, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.566723306 (415h3m6.394431687s) [noise=true , address=45.91.116.11:4117 , piece_id=WLMO5E3BIFFL3TOAUSSBGORIZ6DOCWFCMVT4AVFPPGPSKP2OKDAA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.566992667 (415h3m6.394701063s) [noise=true , address=145.79.227.36:4519 , piece_id=JUKRRIZWVGKEDD3GPJ7HMY2MEQZNMUCTKTXR6ZLSIERRONTZKHCQ, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.567464626 (415h3m6.395173019s) [noise=true , address=145.79.227.47:4266 , piece_id=3NFYV2PWBNTAOSFHENM4J5PZ7XXE72UKTQVXLTVQRHGK4DUXAKDA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.567924492 (415h3m6.395632883s) [noise=true , address=145.79.227.15:4874 , piece_id=FNCCIMIVJTGZ2X5TOYOZH727JXHYM7HG3HBHBTZDAG67SHZLYYYA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.568807294 (415h3m6.396515685s) [noise=true , address=145.79.227.51:4409 , piece_id=ATH3F4DFOUUIFFJZRA2SSRRL7AKWZMQNQRYOZCVFFDLTFCFP236A, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.570579823 (415h3m6.398288214s) [noise=true , address=145.79.227.48:4301 , piece_id=LBIYTJNOX5J4M3HDE7FGF5CSARBITW2AK3P2DSYFLFKUOIHZUKOQ, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.571632758 (415h3m6.399341149s) [noise=true , address=145.79.227.43:4157 , piece_id=3W3LVTU37DNHWS52IGIIH7SXWXVNPGEY27RQNT272L3Z64GNR4KA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.573168383 (415h3m6.400876774s) [noise=true , address=145.79.227.38:4979 , piece_id=HLMQC5OEB4OL4OZSRMCEVAWR5QM2T4MQUBPTHRVQLEAHPBWPQF6Q, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.579995420 (415h3m6.407703816s) [noise=true , address=45.91.116.14:4190 , piece_id=V4T6Z7K3C75LQEXNQIYZGYWW2CSQ7UPSET46QGDRT52RFGAROTTA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.592093172 (415h3m6.419801563s) [noise=true , address=145.79.227.37:4565 , piece_id=554RHTYPG27UNCFPSRMOB245AX4QXL2QFIYECNTVWNNJMAL4BTYA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:09.946500434 (415h3m10.774208816s) [noise=true , address=145.79.227.40:4051 , piece_id=55RRAD7DQX6SR7RBLYAV7MD7XRWPNIWHTVVBQZ2ZL3OE2L7LRRIQ, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:35.398235543 (415h3m36.225943943s) [noise=true , address=145.79.227.39:4013 , piece_id=JJVURGSLKY2UQ4FKNCLATA3ZBD5SDL32XJAG42KH2U6OYK3KTS6Q, seg_pos={0 0}, upload=431917]: Done uploading piece. err: ecclient: upload failed (node:121scLvKEW9zw2aEz22zpremcjiop67eQ8v2uqpmaD5aZzCAP7b, address:145.79.227.39:4013): protocol: write tcp 159.223.143.39:60842->145.79.227.39:4013: use of closed network connection; write tcp 159.223.143.39:60842->145.79.227.39:4013: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:60842->145.79.227.39:4013: use of closed network connection Dec 22 22:31:35.398562305 (415h3m36.226270699s) [noise=true , address=145.79.227.32:4835 , piece_id=BRE3JH3XKPXB6FN7WB3W6ED7MOFES6HYLR34YGR6L5AX7RPIQ7IA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: ecclient: upload failed (node:16vVfeoqCkbMYCRjMMWaKBnCvQBBJPMgwQtWTq9HKQrD9SS17y, address:145.79.227.32:4835): protocol: write tcp 159.223.143.39:45098->145.79.227.32:4835: use of closed network connection; write tcp 159.223.143.39:45098->145.79.227.32:4835: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:45098->145.79.227.32:4835: use of closed network connection Dec 22 22:31:35.399183028 (415h3m36.226891422s) [noise=true , address=145.79.227.50:4373 , piece_id=4U55T5QDYCE6AZ3NCTPAVECPAELPQUKT3HV7GZL7JFRMWRN2AZUQ, seg_pos={0 0}, upload=431917]: Done uploading piece. err: ecclient: upload failed (node:12AQGSkJuNJDxoeqhChUenWxHwF5yK41UESCF63FQz3XvabXXfz, address:145.79.227.50:4373): protocol: read tcp 159.223.143.39:56028->145.79.227.50:4373: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 22 22:31:35.399278021 (415h3m36.226986417s) [noise=true , address=145.79.227.29:4943 , piece_id=G4TMZ6IAIZBDON3UEDPH4LJQFHV3JLALEXSJ3VI4XN6WHIZOB57Q, seg_pos={0 0}, upload=431917]: Done uploading piece. err: ecclient: upload failed (node:12fc5Trj8XxacZhzxdksUVT5CbG6uTmiwdUPjreYi4UMPY1TXWz, address:145.79.227.29:4943): protocol: write tcp 159.223.143.39:50820->145.79.227.29:4943: use of closed network connection; write tcp 159.223.143.39:50820->145.79.227.29:4943: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:50820->145.79.227.29:4943: use of closed network connection Dec 22 22:31:35.399379485 (415h3m36.227087866s) [seg_pos={0 0}, upload=431917]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 22 22:31:35.421122544 (415h3m36.248830935s) [upload=431917]: Done uploading segments... Dec 22 22:31:35.421174369 (415h3m36.248882769s) [upload=431917]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 412.1147ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................X..................................] duration: 469.717089ms trace: https://jaeger.dev.storj.io/trace/7013ffb1a981d605 FAIL (2025-12-22 22:57:30 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 22 22:31:05.439510308 (415h3m6.2672187s) [upload=431917]: Starting upload Dec 22 22:31:05.439581993 (415h3m6.26729037s) [upload=431917]: Done starting upload Dec 22 22:31:05.439812792 (415h3m6.267521185s) [upload=431917]: Uploading segments... Dec 22 22:31:05.439942524 (415h3m6.267650918s) [upload=431917]: Got next segment. Inline: false Dec 22 22:31:05.440022744 (415h3m6.267731174s) [upload=431917]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 22 22:31:05.440029670 (415h3m6.267738072s) [upload=431917]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 22 22:31:05.467186372 (415h3m6.294894769s) [seg_pos={0 0}, upload=431917]: Begin upload segment... Dec 22 22:31:05.467265149 (415h3m6.294973535s) [seg_pos={0 0}, upload=431917]: Done begin upload segment. Dec 22 22:31:05.467296974 (415h3m6.295005373s) [noise=true , address=145.79.227.51:4409 , piece_id=ATH3F4DFOUUIFFJZRA2SSRRL7AKWZMQNQRYOZCVFFDLTFCFP236A, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.467389853 (415h3m6.295098247s) [noise=true , address=145.79.227.39:4013 , piece_id=JJVURGSLKY2UQ4FKNCLATA3ZBD5SDL32XJAG42KH2U6OYK3KTS6Q, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.467912104 (415h3m6.295620486s) [noise=true , address=145.79.227.36:4519 , piece_id=JUKRRIZWVGKEDD3GPJ7HMY2MEQZNMUCTKTXR6ZLSIERRONTZKHCQ, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.468324863 (415h3m6.296033263s) [noise=true , address=145.79.227.38:4979 , piece_id=HLMQC5OEB4OL4OZSRMCEVAWR5QM2T4MQUBPTHRVQLEAHPBWPQF6Q, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.468334764 (415h3m6.296043147s) [noise=true , address=145.79.227.40:4051 , piece_id=55RRAD7DQX6SR7RBLYAV7MD7XRWPNIWHTVVBQZ2ZL3OE2L7LRRIQ, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.468742525 (415h3m6.296450916s) [noise=true , address=45.91.116.4:4939 , piece_id=QYNKMKDLQJV7B6M2E34DU53Y7YI6VBCUW3C7ZLQOI4XWTBXZHKDQ, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.468915034 (415h3m6.296623414s) [noise=true , address=45.91.116.11:4117 , piece_id=WLMO5E3BIFFL3TOAUSSBGORIZ6DOCWFCMVT4AVFPPGPSKP2OKDAA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.469069393 (415h3m6.296777787s) [noise=true , address=145.79.227.32:4835 , piece_id=BRE3JH3XKPXB6FN7WB3W6ED7MOFES6HYLR34YGR6L5AX7RPIQ7IA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.469285789 (415h3m6.296994183s) [noise=true , address=145.79.227.29:4943 , piece_id=G4TMZ6IAIZBDON3UEDPH4LJQFHV3JLALEXSJ3VI4XN6WHIZOB57Q, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.469411962 (415h3m6.297120355s) [noise=true , address=145.79.227.48:4301 , piece_id=LBIYTJNOX5J4M3HDE7FGF5CSARBITW2AK3P2DSYFLFKUOIHZUKOQ, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.469749379 (415h3m6.297457773s) [noise=true , address=145.79.227.37:4565 , piece_id=554RHTYPG27UNCFPSRMOB245AX4QXL2QFIYECNTVWNNJMAL4BTYA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.470080828 (415h3m6.29778922s) [noise=true , address=145.79.227.43:4157 , piece_id=3W3LVTU37DNHWS52IGIIH7SXWXVNPGEY27RQNT272L3Z64GNR4KA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.470477660 (415h3m6.298186058s) [noise=true , address=45.91.116.14:4190 , piece_id=V4T6Z7K3C75LQEXNQIYZGYWW2CSQ7UPSET46QGDRT52RFGAROTTA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.470567977 (415h3m6.29827637s) [noise=true , address=145.79.227.15:4874 , piece_id=FNCCIMIVJTGZ2X5TOYOZH727JXHYM7HG3HBHBTZDAG67SHZLYYYA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.470980950 (415h3m6.298689349s) [noise=true , address=145.79.227.50:4373 , piece_id=4U55T5QDYCE6AZ3NCTPAVECPAELPQUKT3HV7GZL7JFRMWRN2AZUQ, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.471440098 (415h3m6.299148489s) [noise=true , address=145.79.227.47:4266 , piece_id=3NFYV2PWBNTAOSFHENM4J5PZ7XXE72UKTQVXLTVQRHGK4DUXAKDA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.560766475 (415h3m6.388474871s) [upload=431917]: Next returned nil segment Dec 22 22:31:05.560785161 (415h3m6.388493562s) [upload=431917]: Waiting for error group managing segments... Dec 22 22:31:05.566117878 (415h3m6.39382628s) [noise=true , address=45.91.116.4:4939 , piece_id=QYNKMKDLQJV7B6M2E34DU53Y7YI6VBCUW3C7ZLQOI4XWTBXZHKDQ, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.566723306 (415h3m6.394431687s) [noise=true , address=45.91.116.11:4117 , piece_id=WLMO5E3BIFFL3TOAUSSBGORIZ6DOCWFCMVT4AVFPPGPSKP2OKDAA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.566992667 (415h3m6.394701063s) [noise=true , address=145.79.227.36:4519 , piece_id=JUKRRIZWVGKEDD3GPJ7HMY2MEQZNMUCTKTXR6ZLSIERRONTZKHCQ, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.567464626 (415h3m6.395173019s) [noise=true , address=145.79.227.47:4266 , piece_id=3NFYV2PWBNTAOSFHENM4J5PZ7XXE72UKTQVXLTVQRHGK4DUXAKDA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.567924492 (415h3m6.395632883s) [noise=true , address=145.79.227.15:4874 , piece_id=FNCCIMIVJTGZ2X5TOYOZH727JXHYM7HG3HBHBTZDAG67SHZLYYYA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.568807294 (415h3m6.396515685s) [noise=true , address=145.79.227.51:4409 , piece_id=ATH3F4DFOUUIFFJZRA2SSRRL7AKWZMQNQRYOZCVFFDLTFCFP236A, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.570579823 (415h3m6.398288214s) [noise=true , address=145.79.227.48:4301 , piece_id=LBIYTJNOX5J4M3HDE7FGF5CSARBITW2AK3P2DSYFLFKUOIHZUKOQ, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.571632758 (415h3m6.399341149s) [noise=true , address=145.79.227.43:4157 , piece_id=3W3LVTU37DNHWS52IGIIH7SXWXVNPGEY27RQNT272L3Z64GNR4KA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.573168383 (415h3m6.400876774s) [noise=true , address=145.79.227.38:4979 , piece_id=HLMQC5OEB4OL4OZSRMCEVAWR5QM2T4MQUBPTHRVQLEAHPBWPQF6Q, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.579995420 (415h3m6.407703816s) [noise=true , address=45.91.116.14:4190 , piece_id=V4T6Z7K3C75LQEXNQIYZGYWW2CSQ7UPSET46QGDRT52RFGAROTTA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.592093172 (415h3m6.419801563s) [noise=true , address=145.79.227.37:4565 , piece_id=554RHTYPG27UNCFPSRMOB245AX4QXL2QFIYECNTVWNNJMAL4BTYA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:09.946500434 (415h3m10.774208816s) [noise=true , address=145.79.227.40:4051 , piece_id=55RRAD7DQX6SR7RBLYAV7MD7XRWPNIWHTVVBQZ2ZL3OE2L7LRRIQ, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:35.398235543 (415h3m36.225943943s) [noise=true , address=145.79.227.39:4013 , piece_id=JJVURGSLKY2UQ4FKNCLATA3ZBD5SDL32XJAG42KH2U6OYK3KTS6Q, seg_pos={0 0}, upload=431917]: Done uploading piece. err: ecclient: upload failed (node:121scLvKEW9zw2aEz22zpremcjiop67eQ8v2uqpmaD5aZzCAP7b, address:145.79.227.39:4013): protocol: write tcp 159.223.143.39:60842->145.79.227.39:4013: use of closed network connection; write tcp 159.223.143.39:60842->145.79.227.39:4013: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:60842->145.79.227.39:4013: use of closed network connection Dec 22 22:31:35.398562305 (415h3m36.226270699s) [noise=true , address=145.79.227.32:4835 , piece_id=BRE3JH3XKPXB6FN7WB3W6ED7MOFES6HYLR34YGR6L5AX7RPIQ7IA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: ecclient: upload failed (node:16vVfeoqCkbMYCRjMMWaKBnCvQBBJPMgwQtWTq9HKQrD9SS17y, address:145.79.227.32:4835): protocol: write tcp 159.223.143.39:45098->145.79.227.32:4835: use of closed network connection; write tcp 159.223.143.39:45098->145.79.227.32:4835: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:45098->145.79.227.32:4835: use of closed network connection Dec 22 22:31:35.399183028 (415h3m36.226891422s) [noise=true , address=145.79.227.50:4373 , piece_id=4U55T5QDYCE6AZ3NCTPAVECPAELPQUKT3HV7GZL7JFRMWRN2AZUQ, seg_pos={0 0}, upload=431917]: Done uploading piece. err: ecclient: upload failed (node:12AQGSkJuNJDxoeqhChUenWxHwF5yK41UESCF63FQz3XvabXXfz, address:145.79.227.50:4373): protocol: read tcp 159.223.143.39:56028->145.79.227.50:4373: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 22 22:31:35.399278021 (415h3m36.226986417s) [noise=true , address=145.79.227.29:4943 , piece_id=G4TMZ6IAIZBDON3UEDPH4LJQFHV3JLALEXSJ3VI4XN6WHIZOB57Q, seg_pos={0 0}, upload=431917]: Done uploading piece. err: ecclient: upload failed (node:12fc5Trj8XxacZhzxdksUVT5CbG6uTmiwdUPjreYi4UMPY1TXWz, address:145.79.227.29:4943): protocol: write tcp 159.223.143.39:50820->145.79.227.29:4943: use of closed network connection; write tcp 159.223.143.39:50820->145.79.227.29:4943: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:50820->145.79.227.29:4943: use of closed network connection Dec 22 22:31:35.399379485 (415h3m36.227087866s) [seg_pos={0 0}, upload=431917]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 22 22:31:35.421122544 (415h3m36.248830935s) [upload=431917]: Done uploading segments... Dec 22 22:31:35.421174369 (415h3m36.248882769s) [upload=431917]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 469.7467ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................X.................................] duration: 492.020585ms trace: https://jaeger.dev.storj.io/trace/7013ffb1a981d605 FAIL (2025-12-22 22:56:42 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 22 22:31:05.439510308 (415h3m6.2672187s) [upload=431917]: Starting upload Dec 22 22:31:05.439581993 (415h3m6.26729037s) [upload=431917]: Done starting upload Dec 22 22:31:05.439812792 (415h3m6.267521185s) [upload=431917]: Uploading segments... Dec 22 22:31:05.439942524 (415h3m6.267650918s) [upload=431917]: Got next segment. Inline: false Dec 22 22:31:05.440022744 (415h3m6.267731174s) [upload=431917]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 22 22:31:05.440029670 (415h3m6.267738072s) [upload=431917]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 22 22:31:05.467186372 (415h3m6.294894769s) [seg_pos={0 0}, upload=431917]: Begin upload segment... Dec 22 22:31:05.467265149 (415h3m6.294973535s) [seg_pos={0 0}, upload=431917]: Done begin upload segment. Dec 22 22:31:05.467296974 (415h3m6.295005373s) [noise=true , address=145.79.227.51:4409 , piece_id=ATH3F4DFOUUIFFJZRA2SSRRL7AKWZMQNQRYOZCVFFDLTFCFP236A, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.467389853 (415h3m6.295098247s) [noise=true , address=145.79.227.39:4013 , piece_id=JJVURGSLKY2UQ4FKNCLATA3ZBD5SDL32XJAG42KH2U6OYK3KTS6Q, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.467912104 (415h3m6.295620486s) [noise=true , address=145.79.227.36:4519 , piece_id=JUKRRIZWVGKEDD3GPJ7HMY2MEQZNMUCTKTXR6ZLSIERRONTZKHCQ, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.468324863 (415h3m6.296033263s) [noise=true , address=145.79.227.38:4979 , piece_id=HLMQC5OEB4OL4OZSRMCEVAWR5QM2T4MQUBPTHRVQLEAHPBWPQF6Q, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.468334764 (415h3m6.296043147s) [noise=true , address=145.79.227.40:4051 , piece_id=55RRAD7DQX6SR7RBLYAV7MD7XRWPNIWHTVVBQZ2ZL3OE2L7LRRIQ, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.468742525 (415h3m6.296450916s) [noise=true , address=45.91.116.4:4939 , piece_id=QYNKMKDLQJV7B6M2E34DU53Y7YI6VBCUW3C7ZLQOI4XWTBXZHKDQ, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.468915034 (415h3m6.296623414s) [noise=true , address=45.91.116.11:4117 , piece_id=WLMO5E3BIFFL3TOAUSSBGORIZ6DOCWFCMVT4AVFPPGPSKP2OKDAA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.469069393 (415h3m6.296777787s) [noise=true , address=145.79.227.32:4835 , piece_id=BRE3JH3XKPXB6FN7WB3W6ED7MOFES6HYLR34YGR6L5AX7RPIQ7IA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.469285789 (415h3m6.296994183s) [noise=true , address=145.79.227.29:4943 , piece_id=G4TMZ6IAIZBDON3UEDPH4LJQFHV3JLALEXSJ3VI4XN6WHIZOB57Q, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.469411962 (415h3m6.297120355s) [noise=true , address=145.79.227.48:4301 , piece_id=LBIYTJNOX5J4M3HDE7FGF5CSARBITW2AK3P2DSYFLFKUOIHZUKOQ, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.469749379 (415h3m6.297457773s) [noise=true , address=145.79.227.37:4565 , piece_id=554RHTYPG27UNCFPSRMOB245AX4QXL2QFIYECNTVWNNJMAL4BTYA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.470080828 (415h3m6.29778922s) [noise=true , address=145.79.227.43:4157 , piece_id=3W3LVTU37DNHWS52IGIIH7SXWXVNPGEY27RQNT272L3Z64GNR4KA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.470477660 (415h3m6.298186058s) [noise=true , address=45.91.116.14:4190 , piece_id=V4T6Z7K3C75LQEXNQIYZGYWW2CSQ7UPSET46QGDRT52RFGAROTTA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.470567977 (415h3m6.29827637s) [noise=true , address=145.79.227.15:4874 , piece_id=FNCCIMIVJTGZ2X5TOYOZH727JXHYM7HG3HBHBTZDAG67SHZLYYYA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.470980950 (415h3m6.298689349s) [noise=true , address=145.79.227.50:4373 , piece_id=4U55T5QDYCE6AZ3NCTPAVECPAELPQUKT3HV7GZL7JFRMWRN2AZUQ, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.471440098 (415h3m6.299148489s) [noise=true , address=145.79.227.47:4266 , piece_id=3NFYV2PWBNTAOSFHENM4J5PZ7XXE72UKTQVXLTVQRHGK4DUXAKDA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.560766475 (415h3m6.388474871s) [upload=431917]: Next returned nil segment Dec 22 22:31:05.560785161 (415h3m6.388493562s) [upload=431917]: Waiting for error group managing segments... Dec 22 22:31:05.566117878 (415h3m6.39382628s) [noise=true , address=45.91.116.4:4939 , piece_id=QYNKMKDLQJV7B6M2E34DU53Y7YI6VBCUW3C7ZLQOI4XWTBXZHKDQ, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.566723306 (415h3m6.394431687s) [noise=true , address=45.91.116.11:4117 , piece_id=WLMO5E3BIFFL3TOAUSSBGORIZ6DOCWFCMVT4AVFPPGPSKP2OKDAA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.566992667 (415h3m6.394701063s) [noise=true , address=145.79.227.36:4519 , piece_id=JUKRRIZWVGKEDD3GPJ7HMY2MEQZNMUCTKTXR6ZLSIERRONTZKHCQ, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.567464626 (415h3m6.395173019s) [noise=true , address=145.79.227.47:4266 , piece_id=3NFYV2PWBNTAOSFHENM4J5PZ7XXE72UKTQVXLTVQRHGK4DUXAKDA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.567924492 (415h3m6.395632883s) [noise=true , address=145.79.227.15:4874 , piece_id=FNCCIMIVJTGZ2X5TOYOZH727JXHYM7HG3HBHBTZDAG67SHZLYYYA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.568807294 (415h3m6.396515685s) [noise=true , address=145.79.227.51:4409 , piece_id=ATH3F4DFOUUIFFJZRA2SSRRL7AKWZMQNQRYOZCVFFDLTFCFP236A, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.570579823 (415h3m6.398288214s) [noise=true , address=145.79.227.48:4301 , piece_id=LBIYTJNOX5J4M3HDE7FGF5CSARBITW2AK3P2DSYFLFKUOIHZUKOQ, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.571632758 (415h3m6.399341149s) [noise=true , address=145.79.227.43:4157 , piece_id=3W3LVTU37DNHWS52IGIIH7SXWXVNPGEY27RQNT272L3Z64GNR4KA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.573168383 (415h3m6.400876774s) [noise=true , address=145.79.227.38:4979 , piece_id=HLMQC5OEB4OL4OZSRMCEVAWR5QM2T4MQUBPTHRVQLEAHPBWPQF6Q, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.579995420 (415h3m6.407703816s) [noise=true , address=45.91.116.14:4190 , piece_id=V4T6Z7K3C75LQEXNQIYZGYWW2CSQ7UPSET46QGDRT52RFGAROTTA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.592093172 (415h3m6.419801563s) [noise=true , address=145.79.227.37:4565 , piece_id=554RHTYPG27UNCFPSRMOB245AX4QXL2QFIYECNTVWNNJMAL4BTYA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:09.946500434 (415h3m10.774208816s) [noise=true , address=145.79.227.40:4051 , piece_id=55RRAD7DQX6SR7RBLYAV7MD7XRWPNIWHTVVBQZ2ZL3OE2L7LRRIQ, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:35.398235543 (415h3m36.225943943s) [noise=true , address=145.79.227.39:4013 , piece_id=JJVURGSLKY2UQ4FKNCLATA3ZBD5SDL32XJAG42KH2U6OYK3KTS6Q, seg_pos={0 0}, upload=431917]: Done uploading piece. err: ecclient: upload failed (node:121scLvKEW9zw2aEz22zpremcjiop67eQ8v2uqpmaD5aZzCAP7b, address:145.79.227.39:4013): protocol: write tcp 159.223.143.39:60842->145.79.227.39:4013: use of closed network connection; write tcp 159.223.143.39:60842->145.79.227.39:4013: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:60842->145.79.227.39:4013: use of closed network connection Dec 22 22:31:35.398562305 (415h3m36.226270699s) [noise=true , address=145.79.227.32:4835 , piece_id=BRE3JH3XKPXB6FN7WB3W6ED7MOFES6HYLR34YGR6L5AX7RPIQ7IA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: ecclient: upload failed (node:16vVfeoqCkbMYCRjMMWaKBnCvQBBJPMgwQtWTq9HKQrD9SS17y, address:145.79.227.32:4835): protocol: write tcp 159.223.143.39:45098->145.79.227.32:4835: use of closed network connection; write tcp 159.223.143.39:45098->145.79.227.32:4835: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:45098->145.79.227.32:4835: use of closed network connection Dec 22 22:31:35.399183028 (415h3m36.226891422s) [noise=true , address=145.79.227.50:4373 , piece_id=4U55T5QDYCE6AZ3NCTPAVECPAELPQUKT3HV7GZL7JFRMWRN2AZUQ, seg_pos={0 0}, upload=431917]: Done uploading piece. err: ecclient: upload failed (node:12AQGSkJuNJDxoeqhChUenWxHwF5yK41UESCF63FQz3XvabXXfz, address:145.79.227.50:4373): protocol: read tcp 159.223.143.39:56028->145.79.227.50:4373: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 22 22:31:35.399278021 (415h3m36.226986417s) [noise=true , address=145.79.227.29:4943 , piece_id=G4TMZ6IAIZBDON3UEDPH4LJQFHV3JLALEXSJ3VI4XN6WHIZOB57Q, seg_pos={0 0}, upload=431917]: Done uploading piece. err: ecclient: upload failed (node:12fc5Trj8XxacZhzxdksUVT5CbG6uTmiwdUPjreYi4UMPY1TXWz, address:145.79.227.29:4943): protocol: write tcp 159.223.143.39:50820->145.79.227.29:4943: use of closed network connection; write tcp 159.223.143.39:50820->145.79.227.29:4943: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:50820->145.79.227.29:4943: use of closed network connection Dec 22 22:31:35.399379485 (415h3m36.227087866s) [seg_pos={0 0}, upload=431917]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 22 22:31:35.421122544 (415h3m36.248830935s) [upload=431917]: Done uploading segments... Dec 22 22:31:35.421174369 (415h3m36.248882769s) [upload=431917]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 492.046357ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................X................................] duration: 514.756365ms trace: https://jaeger.dev.storj.io/trace/7013ffb1a981d605 FAIL (2025-12-22 22:56:13 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 22 22:31:05.439510308 (415h3m6.2672187s) [upload=431917]: Starting upload Dec 22 22:31:05.439581993 (415h3m6.26729037s) [upload=431917]: Done starting upload Dec 22 22:31:05.439812792 (415h3m6.267521185s) [upload=431917]: Uploading segments... Dec 22 22:31:05.439942524 (415h3m6.267650918s) [upload=431917]: Got next segment. Inline: false Dec 22 22:31:05.440022744 (415h3m6.267731174s) [upload=431917]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 22 22:31:05.440029670 (415h3m6.267738072s) [upload=431917]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 22 22:31:05.467186372 (415h3m6.294894769s) [seg_pos={0 0}, upload=431917]: Begin upload segment... Dec 22 22:31:05.467265149 (415h3m6.294973535s) [seg_pos={0 0}, upload=431917]: Done begin upload segment. Dec 22 22:31:05.467296974 (415h3m6.295005373s) [noise=true , address=145.79.227.51:4409 , piece_id=ATH3F4DFOUUIFFJZRA2SSRRL7AKWZMQNQRYOZCVFFDLTFCFP236A, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.467389853 (415h3m6.295098247s) [noise=true , address=145.79.227.39:4013 , piece_id=JJVURGSLKY2UQ4FKNCLATA3ZBD5SDL32XJAG42KH2U6OYK3KTS6Q, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.467912104 (415h3m6.295620486s) [noise=true , address=145.79.227.36:4519 , piece_id=JUKRRIZWVGKEDD3GPJ7HMY2MEQZNMUCTKTXR6ZLSIERRONTZKHCQ, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.468324863 (415h3m6.296033263s) [noise=true , address=145.79.227.38:4979 , piece_id=HLMQC5OEB4OL4OZSRMCEVAWR5QM2T4MQUBPTHRVQLEAHPBWPQF6Q, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.468334764 (415h3m6.296043147s) [noise=true , address=145.79.227.40:4051 , piece_id=55RRAD7DQX6SR7RBLYAV7MD7XRWPNIWHTVVBQZ2ZL3OE2L7LRRIQ, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.468742525 (415h3m6.296450916s) [noise=true , address=45.91.116.4:4939 , piece_id=QYNKMKDLQJV7B6M2E34DU53Y7YI6VBCUW3C7ZLQOI4XWTBXZHKDQ, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.468915034 (415h3m6.296623414s) [noise=true , address=45.91.116.11:4117 , piece_id=WLMO5E3BIFFL3TOAUSSBGORIZ6DOCWFCMVT4AVFPPGPSKP2OKDAA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.469069393 (415h3m6.296777787s) [noise=true , address=145.79.227.32:4835 , piece_id=BRE3JH3XKPXB6FN7WB3W6ED7MOFES6HYLR34YGR6L5AX7RPIQ7IA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.469285789 (415h3m6.296994183s) [noise=true , address=145.79.227.29:4943 , piece_id=G4TMZ6IAIZBDON3UEDPH4LJQFHV3JLALEXSJ3VI4XN6WHIZOB57Q, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.469411962 (415h3m6.297120355s) [noise=true , address=145.79.227.48:4301 , piece_id=LBIYTJNOX5J4M3HDE7FGF5CSARBITW2AK3P2DSYFLFKUOIHZUKOQ, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.469749379 (415h3m6.297457773s) [noise=true , address=145.79.227.37:4565 , piece_id=554RHTYPG27UNCFPSRMOB245AX4QXL2QFIYECNTVWNNJMAL4BTYA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.470080828 (415h3m6.29778922s) [noise=true , address=145.79.227.43:4157 , piece_id=3W3LVTU37DNHWS52IGIIH7SXWXVNPGEY27RQNT272L3Z64GNR4KA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.470477660 (415h3m6.298186058s) [noise=true , address=45.91.116.14:4190 , piece_id=V4T6Z7K3C75LQEXNQIYZGYWW2CSQ7UPSET46QGDRT52RFGAROTTA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.470567977 (415h3m6.29827637s) [noise=true , address=145.79.227.15:4874 , piece_id=FNCCIMIVJTGZ2X5TOYOZH727JXHYM7HG3HBHBTZDAG67SHZLYYYA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.470980950 (415h3m6.298689349s) [noise=true , address=145.79.227.50:4373 , piece_id=4U55T5QDYCE6AZ3NCTPAVECPAELPQUKT3HV7GZL7JFRMWRN2AZUQ, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.471440098 (415h3m6.299148489s) [noise=true , address=145.79.227.47:4266 , piece_id=3NFYV2PWBNTAOSFHENM4J5PZ7XXE72UKTQVXLTVQRHGK4DUXAKDA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.560766475 (415h3m6.388474871s) [upload=431917]: Next returned nil segment Dec 22 22:31:05.560785161 (415h3m6.388493562s) [upload=431917]: Waiting for error group managing segments... Dec 22 22:31:05.566117878 (415h3m6.39382628s) [noise=true , address=45.91.116.4:4939 , piece_id=QYNKMKDLQJV7B6M2E34DU53Y7YI6VBCUW3C7ZLQOI4XWTBXZHKDQ, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.566723306 (415h3m6.394431687s) [noise=true , address=45.91.116.11:4117 , piece_id=WLMO5E3BIFFL3TOAUSSBGORIZ6DOCWFCMVT4AVFPPGPSKP2OKDAA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.566992667 (415h3m6.394701063s) [noise=true , address=145.79.227.36:4519 , piece_id=JUKRRIZWVGKEDD3GPJ7HMY2MEQZNMUCTKTXR6ZLSIERRONTZKHCQ, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.567464626 (415h3m6.395173019s) [noise=true , address=145.79.227.47:4266 , piece_id=3NFYV2PWBNTAOSFHENM4J5PZ7XXE72UKTQVXLTVQRHGK4DUXAKDA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.567924492 (415h3m6.395632883s) [noise=true , address=145.79.227.15:4874 , piece_id=FNCCIMIVJTGZ2X5TOYOZH727JXHYM7HG3HBHBTZDAG67SHZLYYYA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.568807294 (415h3m6.396515685s) [noise=true , address=145.79.227.51:4409 , piece_id=ATH3F4DFOUUIFFJZRA2SSRRL7AKWZMQNQRYOZCVFFDLTFCFP236A, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.570579823 (415h3m6.398288214s) [noise=true , address=145.79.227.48:4301 , piece_id=LBIYTJNOX5J4M3HDE7FGF5CSARBITW2AK3P2DSYFLFKUOIHZUKOQ, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.571632758 (415h3m6.399341149s) [noise=true , address=145.79.227.43:4157 , piece_id=3W3LVTU37DNHWS52IGIIH7SXWXVNPGEY27RQNT272L3Z64GNR4KA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.573168383 (415h3m6.400876774s) [noise=true , address=145.79.227.38:4979 , piece_id=HLMQC5OEB4OL4OZSRMCEVAWR5QM2T4MQUBPTHRVQLEAHPBWPQF6Q, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.579995420 (415h3m6.407703816s) [noise=true , address=45.91.116.14:4190 , piece_id=V4T6Z7K3C75LQEXNQIYZGYWW2CSQ7UPSET46QGDRT52RFGAROTTA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.592093172 (415h3m6.419801563s) [noise=true , address=145.79.227.37:4565 , piece_id=554RHTYPG27UNCFPSRMOB245AX4QXL2QFIYECNTVWNNJMAL4BTYA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:09.946500434 (415h3m10.774208816s) [noise=true , address=145.79.227.40:4051 , piece_id=55RRAD7DQX6SR7RBLYAV7MD7XRWPNIWHTVVBQZ2ZL3OE2L7LRRIQ, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:35.398235543 (415h3m36.225943943s) [noise=true , address=145.79.227.39:4013 , piece_id=JJVURGSLKY2UQ4FKNCLATA3ZBD5SDL32XJAG42KH2U6OYK3KTS6Q, seg_pos={0 0}, upload=431917]: Done uploading piece. err: ecclient: upload failed (node:121scLvKEW9zw2aEz22zpremcjiop67eQ8v2uqpmaD5aZzCAP7b, address:145.79.227.39:4013): protocol: write tcp 159.223.143.39:60842->145.79.227.39:4013: use of closed network connection; write tcp 159.223.143.39:60842->145.79.227.39:4013: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:60842->145.79.227.39:4013: use of closed network connection Dec 22 22:31:35.398562305 (415h3m36.226270699s) [noise=true , address=145.79.227.32:4835 , piece_id=BRE3JH3XKPXB6FN7WB3W6ED7MOFES6HYLR34YGR6L5AX7RPIQ7IA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: ecclient: upload failed (node:16vVfeoqCkbMYCRjMMWaKBnCvQBBJPMgwQtWTq9HKQrD9SS17y, address:145.79.227.32:4835): protocol: write tcp 159.223.143.39:45098->145.79.227.32:4835: use of closed network connection; write tcp 159.223.143.39:45098->145.79.227.32:4835: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:45098->145.79.227.32:4835: use of closed network connection Dec 22 22:31:35.399183028 (415h3m36.226891422s) [noise=true , address=145.79.227.50:4373 , piece_id=4U55T5QDYCE6AZ3NCTPAVECPAELPQUKT3HV7GZL7JFRMWRN2AZUQ, seg_pos={0 0}, upload=431917]: Done uploading piece. err: ecclient: upload failed (node:12AQGSkJuNJDxoeqhChUenWxHwF5yK41UESCF63FQz3XvabXXfz, address:145.79.227.50:4373): protocol: read tcp 159.223.143.39:56028->145.79.227.50:4373: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 22 22:31:35.399278021 (415h3m36.226986417s) [noise=true , address=145.79.227.29:4943 , piece_id=G4TMZ6IAIZBDON3UEDPH4LJQFHV3JLALEXSJ3VI4XN6WHIZOB57Q, seg_pos={0 0}, upload=431917]: Done uploading piece. err: ecclient: upload failed (node:12fc5Trj8XxacZhzxdksUVT5CbG6uTmiwdUPjreYi4UMPY1TXWz, address:145.79.227.29:4943): protocol: write tcp 159.223.143.39:50820->145.79.227.29:4943: use of closed network connection; write tcp 159.223.143.39:50820->145.79.227.29:4943: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:50820->145.79.227.29:4943: use of closed network connection Dec 22 22:31:35.399379485 (415h3m36.227087866s) [seg_pos={0 0}, upload=431917]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 22 22:31:35.421122544 (415h3m36.248830935s) [upload=431917]: Done uploading segments... Dec 22 22:31:35.421174369 (415h3m36.248882769s) [upload=431917]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 514.780683ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............................X...............................] duration: 404.888217ms trace: https://jaeger.dev.storj.io/trace/7013ffb1a981d605 FAIL (2025-12-22 22:55:32 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 22 22:31:05.439510308 (415h3m6.2672187s) [upload=431917]: Starting upload Dec 22 22:31:05.439581993 (415h3m6.26729037s) [upload=431917]: Done starting upload Dec 22 22:31:05.439812792 (415h3m6.267521185s) [upload=431917]: Uploading segments... Dec 22 22:31:05.439942524 (415h3m6.267650918s) [upload=431917]: Got next segment. Inline: false Dec 22 22:31:05.440022744 (415h3m6.267731174s) [upload=431917]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 22 22:31:05.440029670 (415h3m6.267738072s) [upload=431917]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 22 22:31:05.467186372 (415h3m6.294894769s) [seg_pos={0 0}, upload=431917]: Begin upload segment... Dec 22 22:31:05.467265149 (415h3m6.294973535s) [seg_pos={0 0}, upload=431917]: Done begin upload segment. Dec 22 22:31:05.467296974 (415h3m6.295005373s) [noise=true , address=145.79.227.51:4409 , piece_id=ATH3F4DFOUUIFFJZRA2SSRRL7AKWZMQNQRYOZCVFFDLTFCFP236A, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.467389853 (415h3m6.295098247s) [noise=true , address=145.79.227.39:4013 , piece_id=JJVURGSLKY2UQ4FKNCLATA3ZBD5SDL32XJAG42KH2U6OYK3KTS6Q, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.467912104 (415h3m6.295620486s) [noise=true , address=145.79.227.36:4519 , piece_id=JUKRRIZWVGKEDD3GPJ7HMY2MEQZNMUCTKTXR6ZLSIERRONTZKHCQ, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.468324863 (415h3m6.296033263s) [noise=true , address=145.79.227.38:4979 , piece_id=HLMQC5OEB4OL4OZSRMCEVAWR5QM2T4MQUBPTHRVQLEAHPBWPQF6Q, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.468334764 (415h3m6.296043147s) [noise=true , address=145.79.227.40:4051 , piece_id=55RRAD7DQX6SR7RBLYAV7MD7XRWPNIWHTVVBQZ2ZL3OE2L7LRRIQ, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.468742525 (415h3m6.296450916s) [noise=true , address=45.91.116.4:4939 , piece_id=QYNKMKDLQJV7B6M2E34DU53Y7YI6VBCUW3C7ZLQOI4XWTBXZHKDQ, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.468915034 (415h3m6.296623414s) [noise=true , address=45.91.116.11:4117 , piece_id=WLMO5E3BIFFL3TOAUSSBGORIZ6DOCWFCMVT4AVFPPGPSKP2OKDAA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.469069393 (415h3m6.296777787s) [noise=true , address=145.79.227.32:4835 , piece_id=BRE3JH3XKPXB6FN7WB3W6ED7MOFES6HYLR34YGR6L5AX7RPIQ7IA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.469285789 (415h3m6.296994183s) [noise=true , address=145.79.227.29:4943 , piece_id=G4TMZ6IAIZBDON3UEDPH4LJQFHV3JLALEXSJ3VI4XN6WHIZOB57Q, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.469411962 (415h3m6.297120355s) [noise=true , address=145.79.227.48:4301 , piece_id=LBIYTJNOX5J4M3HDE7FGF5CSARBITW2AK3P2DSYFLFKUOIHZUKOQ, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.469749379 (415h3m6.297457773s) [noise=true , address=145.79.227.37:4565 , piece_id=554RHTYPG27UNCFPSRMOB245AX4QXL2QFIYECNTVWNNJMAL4BTYA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.470080828 (415h3m6.29778922s) [noise=true , address=145.79.227.43:4157 , piece_id=3W3LVTU37DNHWS52IGIIH7SXWXVNPGEY27RQNT272L3Z64GNR4KA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.470477660 (415h3m6.298186058s) [noise=true , address=45.91.116.14:4190 , piece_id=V4T6Z7K3C75LQEXNQIYZGYWW2CSQ7UPSET46QGDRT52RFGAROTTA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.470567977 (415h3m6.29827637s) [noise=true , address=145.79.227.15:4874 , piece_id=FNCCIMIVJTGZ2X5TOYOZH727JXHYM7HG3HBHBTZDAG67SHZLYYYA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.470980950 (415h3m6.298689349s) [noise=true , address=145.79.227.50:4373 , piece_id=4U55T5QDYCE6AZ3NCTPAVECPAELPQUKT3HV7GZL7JFRMWRN2AZUQ, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.471440098 (415h3m6.299148489s) [noise=true , address=145.79.227.47:4266 , piece_id=3NFYV2PWBNTAOSFHENM4J5PZ7XXE72UKTQVXLTVQRHGK4DUXAKDA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.560766475 (415h3m6.388474871s) [upload=431917]: Next returned nil segment Dec 22 22:31:05.560785161 (415h3m6.388493562s) [upload=431917]: Waiting for error group managing segments... Dec 22 22:31:05.566117878 (415h3m6.39382628s) [noise=true , address=45.91.116.4:4939 , piece_id=QYNKMKDLQJV7B6M2E34DU53Y7YI6VBCUW3C7ZLQOI4XWTBXZHKDQ, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.566723306 (415h3m6.394431687s) [noise=true , address=45.91.116.11:4117 , piece_id=WLMO5E3BIFFL3TOAUSSBGORIZ6DOCWFCMVT4AVFPPGPSKP2OKDAA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.566992667 (415h3m6.394701063s) [noise=true , address=145.79.227.36:4519 , piece_id=JUKRRIZWVGKEDD3GPJ7HMY2MEQZNMUCTKTXR6ZLSIERRONTZKHCQ, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.567464626 (415h3m6.395173019s) [noise=true , address=145.79.227.47:4266 , piece_id=3NFYV2PWBNTAOSFHENM4J5PZ7XXE72UKTQVXLTVQRHGK4DUXAKDA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.567924492 (415h3m6.395632883s) [noise=true , address=145.79.227.15:4874 , piece_id=FNCCIMIVJTGZ2X5TOYOZH727JXHYM7HG3HBHBTZDAG67SHZLYYYA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.568807294 (415h3m6.396515685s) [noise=true , address=145.79.227.51:4409 , piece_id=ATH3F4DFOUUIFFJZRA2SSRRL7AKWZMQNQRYOZCVFFDLTFCFP236A, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.570579823 (415h3m6.398288214s) [noise=true , address=145.79.227.48:4301 , piece_id=LBIYTJNOX5J4M3HDE7FGF5CSARBITW2AK3P2DSYFLFKUOIHZUKOQ, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.571632758 (415h3m6.399341149s) [noise=true , address=145.79.227.43:4157 , piece_id=3W3LVTU37DNHWS52IGIIH7SXWXVNPGEY27RQNT272L3Z64GNR4KA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.573168383 (415h3m6.400876774s) [noise=true , address=145.79.227.38:4979 , piece_id=HLMQC5OEB4OL4OZSRMCEVAWR5QM2T4MQUBPTHRVQLEAHPBWPQF6Q, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.579995420 (415h3m6.407703816s) [noise=true , address=45.91.116.14:4190 , piece_id=V4T6Z7K3C75LQEXNQIYZGYWW2CSQ7UPSET46QGDRT52RFGAROTTA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.592093172 (415h3m6.419801563s) [noise=true , address=145.79.227.37:4565 , piece_id=554RHTYPG27UNCFPSRMOB245AX4QXL2QFIYECNTVWNNJMAL4BTYA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:09.946500434 (415h3m10.774208816s) [noise=true , address=145.79.227.40:4051 , piece_id=55RRAD7DQX6SR7RBLYAV7MD7XRWPNIWHTVVBQZ2ZL3OE2L7LRRIQ, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:35.398235543 (415h3m36.225943943s) [noise=true , address=145.79.227.39:4013 , piece_id=JJVURGSLKY2UQ4FKNCLATA3ZBD5SDL32XJAG42KH2U6OYK3KTS6Q, seg_pos={0 0}, upload=431917]: Done uploading piece. err: ecclient: upload failed (node:121scLvKEW9zw2aEz22zpremcjiop67eQ8v2uqpmaD5aZzCAP7b, address:145.79.227.39:4013): protocol: write tcp 159.223.143.39:60842->145.79.227.39:4013: use of closed network connection; write tcp 159.223.143.39:60842->145.79.227.39:4013: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:60842->145.79.227.39:4013: use of closed network connection Dec 22 22:31:35.398562305 (415h3m36.226270699s) [noise=true , address=145.79.227.32:4835 , piece_id=BRE3JH3XKPXB6FN7WB3W6ED7MOFES6HYLR34YGR6L5AX7RPIQ7IA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: ecclient: upload failed (node:16vVfeoqCkbMYCRjMMWaKBnCvQBBJPMgwQtWTq9HKQrD9SS17y, address:145.79.227.32:4835): protocol: write tcp 159.223.143.39:45098->145.79.227.32:4835: use of closed network connection; write tcp 159.223.143.39:45098->145.79.227.32:4835: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:45098->145.79.227.32:4835: use of closed network connection Dec 22 22:31:35.399183028 (415h3m36.226891422s) [noise=true , address=145.79.227.50:4373 , piece_id=4U55T5QDYCE6AZ3NCTPAVECPAELPQUKT3HV7GZL7JFRMWRN2AZUQ, seg_pos={0 0}, upload=431917]: Done uploading piece. err: ecclient: upload failed (node:12AQGSkJuNJDxoeqhChUenWxHwF5yK41UESCF63FQz3XvabXXfz, address:145.79.227.50:4373): protocol: read tcp 159.223.143.39:56028->145.79.227.50:4373: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 22 22:31:35.399278021 (415h3m36.226986417s) [noise=true , address=145.79.227.29:4943 , piece_id=G4TMZ6IAIZBDON3UEDPH4LJQFHV3JLALEXSJ3VI4XN6WHIZOB57Q, seg_pos={0 0}, upload=431917]: Done uploading piece. err: ecclient: upload failed (node:12fc5Trj8XxacZhzxdksUVT5CbG6uTmiwdUPjreYi4UMPY1TXWz, address:145.79.227.29:4943): protocol: write tcp 159.223.143.39:50820->145.79.227.29:4943: use of closed network connection; write tcp 159.223.143.39:50820->145.79.227.29:4943: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:50820->145.79.227.29:4943: use of closed network connection Dec 22 22:31:35.399379485 (415h3m36.227087866s) [seg_pos={0 0}, upload=431917]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 22 22:31:35.421122544 (415h3m36.248830935s) [upload=431917]: Done uploading segments... Dec 22 22:31:35.421174369 (415h3m36.248882769s) [upload=431917]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 404.915689ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............................X..............................] duration: 495.097203ms trace: https://jaeger.dev.storj.io/trace/7013ffb1a981d605 FAIL (2025-12-22 22:54:59 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 22 22:31:05.439510308 (415h3m6.2672187s) [upload=431917]: Starting upload Dec 22 22:31:05.439581993 (415h3m6.26729037s) [upload=431917]: Done starting upload Dec 22 22:31:05.439812792 (415h3m6.267521185s) [upload=431917]: Uploading segments... Dec 22 22:31:05.439942524 (415h3m6.267650918s) [upload=431917]: Got next segment. Inline: false Dec 22 22:31:05.440022744 (415h3m6.267731174s) [upload=431917]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 22 22:31:05.440029670 (415h3m6.267738072s) [upload=431917]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 22 22:31:05.467186372 (415h3m6.294894769s) [seg_pos={0 0}, upload=431917]: Begin upload segment... Dec 22 22:31:05.467265149 (415h3m6.294973535s) [seg_pos={0 0}, upload=431917]: Done begin upload segment. Dec 22 22:31:05.467296974 (415h3m6.295005373s) [noise=true , address=145.79.227.51:4409 , piece_id=ATH3F4DFOUUIFFJZRA2SSRRL7AKWZMQNQRYOZCVFFDLTFCFP236A, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.467389853 (415h3m6.295098247s) [noise=true , address=145.79.227.39:4013 , piece_id=JJVURGSLKY2UQ4FKNCLATA3ZBD5SDL32XJAG42KH2U6OYK3KTS6Q, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.467912104 (415h3m6.295620486s) [noise=true , address=145.79.227.36:4519 , piece_id=JUKRRIZWVGKEDD3GPJ7HMY2MEQZNMUCTKTXR6ZLSIERRONTZKHCQ, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.468324863 (415h3m6.296033263s) [noise=true , address=145.79.227.38:4979 , piece_id=HLMQC5OEB4OL4OZSRMCEVAWR5QM2T4MQUBPTHRVQLEAHPBWPQF6Q, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.468334764 (415h3m6.296043147s) [noise=true , address=145.79.227.40:4051 , piece_id=55RRAD7DQX6SR7RBLYAV7MD7XRWPNIWHTVVBQZ2ZL3OE2L7LRRIQ, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.468742525 (415h3m6.296450916s) [noise=true , address=45.91.116.4:4939 , piece_id=QYNKMKDLQJV7B6M2E34DU53Y7YI6VBCUW3C7ZLQOI4XWTBXZHKDQ, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.468915034 (415h3m6.296623414s) [noise=true , address=45.91.116.11:4117 , piece_id=WLMO5E3BIFFL3TOAUSSBGORIZ6DOCWFCMVT4AVFPPGPSKP2OKDAA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.469069393 (415h3m6.296777787s) [noise=true , address=145.79.227.32:4835 , piece_id=BRE3JH3XKPXB6FN7WB3W6ED7MOFES6HYLR34YGR6L5AX7RPIQ7IA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.469285789 (415h3m6.296994183s) [noise=true , address=145.79.227.29:4943 , piece_id=G4TMZ6IAIZBDON3UEDPH4LJQFHV3JLALEXSJ3VI4XN6WHIZOB57Q, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.469411962 (415h3m6.297120355s) [noise=true , address=145.79.227.48:4301 , piece_id=LBIYTJNOX5J4M3HDE7FGF5CSARBITW2AK3P2DSYFLFKUOIHZUKOQ, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.469749379 (415h3m6.297457773s) [noise=true , address=145.79.227.37:4565 , piece_id=554RHTYPG27UNCFPSRMOB245AX4QXL2QFIYECNTVWNNJMAL4BTYA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.470080828 (415h3m6.29778922s) [noise=true , address=145.79.227.43:4157 , piece_id=3W3LVTU37DNHWS52IGIIH7SXWXVNPGEY27RQNT272L3Z64GNR4KA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.470477660 (415h3m6.298186058s) [noise=true , address=45.91.116.14:4190 , piece_id=V4T6Z7K3C75LQEXNQIYZGYWW2CSQ7UPSET46QGDRT52RFGAROTTA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.470567977 (415h3m6.29827637s) [noise=true , address=145.79.227.15:4874 , piece_id=FNCCIMIVJTGZ2X5TOYOZH727JXHYM7HG3HBHBTZDAG67SHZLYYYA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.470980950 (415h3m6.298689349s) [noise=true , address=145.79.227.50:4373 , piece_id=4U55T5QDYCE6AZ3NCTPAVECPAELPQUKT3HV7GZL7JFRMWRN2AZUQ, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.471440098 (415h3m6.299148489s) [noise=true , address=145.79.227.47:4266 , piece_id=3NFYV2PWBNTAOSFHENM4J5PZ7XXE72UKTQVXLTVQRHGK4DUXAKDA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.560766475 (415h3m6.388474871s) [upload=431917]: Next returned nil segment Dec 22 22:31:05.560785161 (415h3m6.388493562s) [upload=431917]: Waiting for error group managing segments... Dec 22 22:31:05.566117878 (415h3m6.39382628s) [noise=true , address=45.91.116.4:4939 , piece_id=QYNKMKDLQJV7B6M2E34DU53Y7YI6VBCUW3C7ZLQOI4XWTBXZHKDQ, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.566723306 (415h3m6.394431687s) [noise=true , address=45.91.116.11:4117 , piece_id=WLMO5E3BIFFL3TOAUSSBGORIZ6DOCWFCMVT4AVFPPGPSKP2OKDAA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.566992667 (415h3m6.394701063s) [noise=true , address=145.79.227.36:4519 , piece_id=JUKRRIZWVGKEDD3GPJ7HMY2MEQZNMUCTKTXR6ZLSIERRONTZKHCQ, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.567464626 (415h3m6.395173019s) [noise=true , address=145.79.227.47:4266 , piece_id=3NFYV2PWBNTAOSFHENM4J5PZ7XXE72UKTQVXLTVQRHGK4DUXAKDA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.567924492 (415h3m6.395632883s) [noise=true , address=145.79.227.15:4874 , piece_id=FNCCIMIVJTGZ2X5TOYOZH727JXHYM7HG3HBHBTZDAG67SHZLYYYA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.568807294 (415h3m6.396515685s) [noise=true , address=145.79.227.51:4409 , piece_id=ATH3F4DFOUUIFFJZRA2SSRRL7AKWZMQNQRYOZCVFFDLTFCFP236A, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.570579823 (415h3m6.398288214s) [noise=true , address=145.79.227.48:4301 , piece_id=LBIYTJNOX5J4M3HDE7FGF5CSARBITW2AK3P2DSYFLFKUOIHZUKOQ, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.571632758 (415h3m6.399341149s) [noise=true , address=145.79.227.43:4157 , piece_id=3W3LVTU37DNHWS52IGIIH7SXWXVNPGEY27RQNT272L3Z64GNR4KA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.573168383 (415h3m6.400876774s) [noise=true , address=145.79.227.38:4979 , piece_id=HLMQC5OEB4OL4OZSRMCEVAWR5QM2T4MQUBPTHRVQLEAHPBWPQF6Q, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.579995420 (415h3m6.407703816s) [noise=true , address=45.91.116.14:4190 , piece_id=V4T6Z7K3C75LQEXNQIYZGYWW2CSQ7UPSET46QGDRT52RFGAROTTA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.592093172 (415h3m6.419801563s) [noise=true , address=145.79.227.37:4565 , piece_id=554RHTYPG27UNCFPSRMOB245AX4QXL2QFIYECNTVWNNJMAL4BTYA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:09.946500434 (415h3m10.774208816s) [noise=true , address=145.79.227.40:4051 , piece_id=55RRAD7DQX6SR7RBLYAV7MD7XRWPNIWHTVVBQZ2ZL3OE2L7LRRIQ, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:35.398235543 (415h3m36.225943943s) [noise=true , address=145.79.227.39:4013 , piece_id=JJVURGSLKY2UQ4FKNCLATA3ZBD5SDL32XJAG42KH2U6OYK3KTS6Q, seg_pos={0 0}, upload=431917]: Done uploading piece. err: ecclient: upload failed (node:121scLvKEW9zw2aEz22zpremcjiop67eQ8v2uqpmaD5aZzCAP7b, address:145.79.227.39:4013): protocol: write tcp 159.223.143.39:60842->145.79.227.39:4013: use of closed network connection; write tcp 159.223.143.39:60842->145.79.227.39:4013: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:60842->145.79.227.39:4013: use of closed network connection Dec 22 22:31:35.398562305 (415h3m36.226270699s) [noise=true , address=145.79.227.32:4835 , piece_id=BRE3JH3XKPXB6FN7WB3W6ED7MOFES6HYLR34YGR6L5AX7RPIQ7IA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: ecclient: upload failed (node:16vVfeoqCkbMYCRjMMWaKBnCvQBBJPMgwQtWTq9HKQrD9SS17y, address:145.79.227.32:4835): protocol: write tcp 159.223.143.39:45098->145.79.227.32:4835: use of closed network connection; write tcp 159.223.143.39:45098->145.79.227.32:4835: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:45098->145.79.227.32:4835: use of closed network connection Dec 22 22:31:35.399183028 (415h3m36.226891422s) [noise=true , address=145.79.227.50:4373 , piece_id=4U55T5QDYCE6AZ3NCTPAVECPAELPQUKT3HV7GZL7JFRMWRN2AZUQ, seg_pos={0 0}, upload=431917]: Done uploading piece. err: ecclient: upload failed (node:12AQGSkJuNJDxoeqhChUenWxHwF5yK41UESCF63FQz3XvabXXfz, address:145.79.227.50:4373): protocol: read tcp 159.223.143.39:56028->145.79.227.50:4373: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 22 22:31:35.399278021 (415h3m36.226986417s) [noise=true , address=145.79.227.29:4943 , piece_id=G4TMZ6IAIZBDON3UEDPH4LJQFHV3JLALEXSJ3VI4XN6WHIZOB57Q, seg_pos={0 0}, upload=431917]: Done uploading piece. err: ecclient: upload failed (node:12fc5Trj8XxacZhzxdksUVT5CbG6uTmiwdUPjreYi4UMPY1TXWz, address:145.79.227.29:4943): protocol: write tcp 159.223.143.39:50820->145.79.227.29:4943: use of closed network connection; write tcp 159.223.143.39:50820->145.79.227.29:4943: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:50820->145.79.227.29:4943: use of closed network connection Dec 22 22:31:35.399379485 (415h3m36.227087866s) [seg_pos={0 0}, upload=431917]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 22 22:31:35.421122544 (415h3m36.248830935s) [upload=431917]: Done uploading segments... Dec 22 22:31:35.421174369 (415h3m36.248882769s) [upload=431917]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 495.137331ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............................X.............................] duration: 390.748645ms trace: https://jaeger.dev.storj.io/trace/7013ffb1a981d605 FAIL (2025-12-22 22:54:12 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 22 22:31:05.439510308 (415h3m6.2672187s) [upload=431917]: Starting upload Dec 22 22:31:05.439581993 (415h3m6.26729037s) [upload=431917]: Done starting upload Dec 22 22:31:05.439812792 (415h3m6.267521185s) [upload=431917]: Uploading segments... Dec 22 22:31:05.439942524 (415h3m6.267650918s) [upload=431917]: Got next segment. Inline: false Dec 22 22:31:05.440022744 (415h3m6.267731174s) [upload=431917]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 22 22:31:05.440029670 (415h3m6.267738072s) [upload=431917]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 22 22:31:05.467186372 (415h3m6.294894769s) [seg_pos={0 0}, upload=431917]: Begin upload segment... Dec 22 22:31:05.467265149 (415h3m6.294973535s) [seg_pos={0 0}, upload=431917]: Done begin upload segment. Dec 22 22:31:05.467296974 (415h3m6.295005373s) [noise=true , address=145.79.227.51:4409 , piece_id=ATH3F4DFOUUIFFJZRA2SSRRL7AKWZMQNQRYOZCVFFDLTFCFP236A, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.467389853 (415h3m6.295098247s) [noise=true , address=145.79.227.39:4013 , piece_id=JJVURGSLKY2UQ4FKNCLATA3ZBD5SDL32XJAG42KH2U6OYK3KTS6Q, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.467912104 (415h3m6.295620486s) [noise=true , address=145.79.227.36:4519 , piece_id=JUKRRIZWVGKEDD3GPJ7HMY2MEQZNMUCTKTXR6ZLSIERRONTZKHCQ, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.468324863 (415h3m6.296033263s) [noise=true , address=145.79.227.38:4979 , piece_id=HLMQC5OEB4OL4OZSRMCEVAWR5QM2T4MQUBPTHRVQLEAHPBWPQF6Q, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.468334764 (415h3m6.296043147s) [noise=true , address=145.79.227.40:4051 , piece_id=55RRAD7DQX6SR7RBLYAV7MD7XRWPNIWHTVVBQZ2ZL3OE2L7LRRIQ, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.468742525 (415h3m6.296450916s) [noise=true , address=45.91.116.4:4939 , piece_id=QYNKMKDLQJV7B6M2E34DU53Y7YI6VBCUW3C7ZLQOI4XWTBXZHKDQ, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.468915034 (415h3m6.296623414s) [noise=true , address=45.91.116.11:4117 , piece_id=WLMO5E3BIFFL3TOAUSSBGORIZ6DOCWFCMVT4AVFPPGPSKP2OKDAA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.469069393 (415h3m6.296777787s) [noise=true , address=145.79.227.32:4835 , piece_id=BRE3JH3XKPXB6FN7WB3W6ED7MOFES6HYLR34YGR6L5AX7RPIQ7IA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.469285789 (415h3m6.296994183s) [noise=true , address=145.79.227.29:4943 , piece_id=G4TMZ6IAIZBDON3UEDPH4LJQFHV3JLALEXSJ3VI4XN6WHIZOB57Q, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.469411962 (415h3m6.297120355s) [noise=true , address=145.79.227.48:4301 , piece_id=LBIYTJNOX5J4M3HDE7FGF5CSARBITW2AK3P2DSYFLFKUOIHZUKOQ, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.469749379 (415h3m6.297457773s) [noise=true , address=145.79.227.37:4565 , piece_id=554RHTYPG27UNCFPSRMOB245AX4QXL2QFIYECNTVWNNJMAL4BTYA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.470080828 (415h3m6.29778922s) [noise=true , address=145.79.227.43:4157 , piece_id=3W3LVTU37DNHWS52IGIIH7SXWXVNPGEY27RQNT272L3Z64GNR4KA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.470477660 (415h3m6.298186058s) [noise=true , address=45.91.116.14:4190 , piece_id=V4T6Z7K3C75LQEXNQIYZGYWW2CSQ7UPSET46QGDRT52RFGAROTTA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.470567977 (415h3m6.29827637s) [noise=true , address=145.79.227.15:4874 , piece_id=FNCCIMIVJTGZ2X5TOYOZH727JXHYM7HG3HBHBTZDAG67SHZLYYYA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.470980950 (415h3m6.298689349s) [noise=true , address=145.79.227.50:4373 , piece_id=4U55T5QDYCE6AZ3NCTPAVECPAELPQUKT3HV7GZL7JFRMWRN2AZUQ, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.471440098 (415h3m6.299148489s) [noise=true , address=145.79.227.47:4266 , piece_id=3NFYV2PWBNTAOSFHENM4J5PZ7XXE72UKTQVXLTVQRHGK4DUXAKDA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.560766475 (415h3m6.388474871s) [upload=431917]: Next returned nil segment Dec 22 22:31:05.560785161 (415h3m6.388493562s) [upload=431917]: Waiting for error group managing segments... Dec 22 22:31:05.566117878 (415h3m6.39382628s) [noise=true , address=45.91.116.4:4939 , piece_id=QYNKMKDLQJV7B6M2E34DU53Y7YI6VBCUW3C7ZLQOI4XWTBXZHKDQ, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.566723306 (415h3m6.394431687s) [noise=true , address=45.91.116.11:4117 , piece_id=WLMO5E3BIFFL3TOAUSSBGORIZ6DOCWFCMVT4AVFPPGPSKP2OKDAA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.566992667 (415h3m6.394701063s) [noise=true , address=145.79.227.36:4519 , piece_id=JUKRRIZWVGKEDD3GPJ7HMY2MEQZNMUCTKTXR6ZLSIERRONTZKHCQ, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.567464626 (415h3m6.395173019s) [noise=true , address=145.79.227.47:4266 , piece_id=3NFYV2PWBNTAOSFHENM4J5PZ7XXE72UKTQVXLTVQRHGK4DUXAKDA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.567924492 (415h3m6.395632883s) [noise=true , address=145.79.227.15:4874 , piece_id=FNCCIMIVJTGZ2X5TOYOZH727JXHYM7HG3HBHBTZDAG67SHZLYYYA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.568807294 (415h3m6.396515685s) [noise=true , address=145.79.227.51:4409 , piece_id=ATH3F4DFOUUIFFJZRA2SSRRL7AKWZMQNQRYOZCVFFDLTFCFP236A, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.570579823 (415h3m6.398288214s) [noise=true , address=145.79.227.48:4301 , piece_id=LBIYTJNOX5J4M3HDE7FGF5CSARBITW2AK3P2DSYFLFKUOIHZUKOQ, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.571632758 (415h3m6.399341149s) [noise=true , address=145.79.227.43:4157 , piece_id=3W3LVTU37DNHWS52IGIIH7SXWXVNPGEY27RQNT272L3Z64GNR4KA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.573168383 (415h3m6.400876774s) [noise=true , address=145.79.227.38:4979 , piece_id=HLMQC5OEB4OL4OZSRMCEVAWR5QM2T4MQUBPTHRVQLEAHPBWPQF6Q, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.579995420 (415h3m6.407703816s) [noise=true , address=45.91.116.14:4190 , piece_id=V4T6Z7K3C75LQEXNQIYZGYWW2CSQ7UPSET46QGDRT52RFGAROTTA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.592093172 (415h3m6.419801563s) [noise=true , address=145.79.227.37:4565 , piece_id=554RHTYPG27UNCFPSRMOB245AX4QXL2QFIYECNTVWNNJMAL4BTYA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:09.946500434 (415h3m10.774208816s) [noise=true , address=145.79.227.40:4051 , piece_id=55RRAD7DQX6SR7RBLYAV7MD7XRWPNIWHTVVBQZ2ZL3OE2L7LRRIQ, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:35.398235543 (415h3m36.225943943s) [noise=true , address=145.79.227.39:4013 , piece_id=JJVURGSLKY2UQ4FKNCLATA3ZBD5SDL32XJAG42KH2U6OYK3KTS6Q, seg_pos={0 0}, upload=431917]: Done uploading piece. err: ecclient: upload failed (node:121scLvKEW9zw2aEz22zpremcjiop67eQ8v2uqpmaD5aZzCAP7b, address:145.79.227.39:4013): protocol: write tcp 159.223.143.39:60842->145.79.227.39:4013: use of closed network connection; write tcp 159.223.143.39:60842->145.79.227.39:4013: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:60842->145.79.227.39:4013: use of closed network connection Dec 22 22:31:35.398562305 (415h3m36.226270699s) [noise=true , address=145.79.227.32:4835 , piece_id=BRE3JH3XKPXB6FN7WB3W6ED7MOFES6HYLR34YGR6L5AX7RPIQ7IA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: ecclient: upload failed (node:16vVfeoqCkbMYCRjMMWaKBnCvQBBJPMgwQtWTq9HKQrD9SS17y, address:145.79.227.32:4835): protocol: write tcp 159.223.143.39:45098->145.79.227.32:4835: use of closed network connection; write tcp 159.223.143.39:45098->145.79.227.32:4835: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:45098->145.79.227.32:4835: use of closed network connection Dec 22 22:31:35.399183028 (415h3m36.226891422s) [noise=true , address=145.79.227.50:4373 , piece_id=4U55T5QDYCE6AZ3NCTPAVECPAELPQUKT3HV7GZL7JFRMWRN2AZUQ, seg_pos={0 0}, upload=431917]: Done uploading piece. err: ecclient: upload failed (node:12AQGSkJuNJDxoeqhChUenWxHwF5yK41UESCF63FQz3XvabXXfz, address:145.79.227.50:4373): protocol: read tcp 159.223.143.39:56028->145.79.227.50:4373: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 22 22:31:35.399278021 (415h3m36.226986417s) [noise=true , address=145.79.227.29:4943 , piece_id=G4TMZ6IAIZBDON3UEDPH4LJQFHV3JLALEXSJ3VI4XN6WHIZOB57Q, seg_pos={0 0}, upload=431917]: Done uploading piece. err: ecclient: upload failed (node:12fc5Trj8XxacZhzxdksUVT5CbG6uTmiwdUPjreYi4UMPY1TXWz, address:145.79.227.29:4943): protocol: write tcp 159.223.143.39:50820->145.79.227.29:4943: use of closed network connection; write tcp 159.223.143.39:50820->145.79.227.29:4943: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:50820->145.79.227.29:4943: use of closed network connection Dec 22 22:31:35.399379485 (415h3m36.227087866s) [seg_pos={0 0}, upload=431917]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 22 22:31:35.421122544 (415h3m36.248830935s) [upload=431917]: Done uploading segments... Dec 22 22:31:35.421174369 (415h3m36.248882769s) [upload=431917]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 390.773115ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............................X............................] duration: 544.21019ms trace: https://jaeger.dev.storj.io/trace/7013ffb1a981d605 FAIL (2025-12-22 22:53:47 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 22 22:31:05.439510308 (415h3m6.2672187s) [upload=431917]: Starting upload Dec 22 22:31:05.439581993 (415h3m6.26729037s) [upload=431917]: Done starting upload Dec 22 22:31:05.439812792 (415h3m6.267521185s) [upload=431917]: Uploading segments... Dec 22 22:31:05.439942524 (415h3m6.267650918s) [upload=431917]: Got next segment. Inline: false Dec 22 22:31:05.440022744 (415h3m6.267731174s) [upload=431917]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 22 22:31:05.440029670 (415h3m6.267738072s) [upload=431917]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 22 22:31:05.467186372 (415h3m6.294894769s) [seg_pos={0 0}, upload=431917]: Begin upload segment... Dec 22 22:31:05.467265149 (415h3m6.294973535s) [seg_pos={0 0}, upload=431917]: Done begin upload segment. Dec 22 22:31:05.467296974 (415h3m6.295005373s) [noise=true , address=145.79.227.51:4409 , piece_id=ATH3F4DFOUUIFFJZRA2SSRRL7AKWZMQNQRYOZCVFFDLTFCFP236A, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.467389853 (415h3m6.295098247s) [noise=true , address=145.79.227.39:4013 , piece_id=JJVURGSLKY2UQ4FKNCLATA3ZBD5SDL32XJAG42KH2U6OYK3KTS6Q, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.467912104 (415h3m6.295620486s) [noise=true , address=145.79.227.36:4519 , piece_id=JUKRRIZWVGKEDD3GPJ7HMY2MEQZNMUCTKTXR6ZLSIERRONTZKHCQ, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.468324863 (415h3m6.296033263s) [noise=true , address=145.79.227.38:4979 , piece_id=HLMQC5OEB4OL4OZSRMCEVAWR5QM2T4MQUBPTHRVQLEAHPBWPQF6Q, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.468334764 (415h3m6.296043147s) [noise=true , address=145.79.227.40:4051 , piece_id=55RRAD7DQX6SR7RBLYAV7MD7XRWPNIWHTVVBQZ2ZL3OE2L7LRRIQ, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.468742525 (415h3m6.296450916s) [noise=true , address=45.91.116.4:4939 , piece_id=QYNKMKDLQJV7B6M2E34DU53Y7YI6VBCUW3C7ZLQOI4XWTBXZHKDQ, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.468915034 (415h3m6.296623414s) [noise=true , address=45.91.116.11:4117 , piece_id=WLMO5E3BIFFL3TOAUSSBGORIZ6DOCWFCMVT4AVFPPGPSKP2OKDAA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.469069393 (415h3m6.296777787s) [noise=true , address=145.79.227.32:4835 , piece_id=BRE3JH3XKPXB6FN7WB3W6ED7MOFES6HYLR34YGR6L5AX7RPIQ7IA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.469285789 (415h3m6.296994183s) [noise=true , address=145.79.227.29:4943 , piece_id=G4TMZ6IAIZBDON3UEDPH4LJQFHV3JLALEXSJ3VI4XN6WHIZOB57Q, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.469411962 (415h3m6.297120355s) [noise=true , address=145.79.227.48:4301 , piece_id=LBIYTJNOX5J4M3HDE7FGF5CSARBITW2AK3P2DSYFLFKUOIHZUKOQ, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.469749379 (415h3m6.297457773s) [noise=true , address=145.79.227.37:4565 , piece_id=554RHTYPG27UNCFPSRMOB245AX4QXL2QFIYECNTVWNNJMAL4BTYA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.470080828 (415h3m6.29778922s) [noise=true , address=145.79.227.43:4157 , piece_id=3W3LVTU37DNHWS52IGIIH7SXWXVNPGEY27RQNT272L3Z64GNR4KA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.470477660 (415h3m6.298186058s) [noise=true , address=45.91.116.14:4190 , piece_id=V4T6Z7K3C75LQEXNQIYZGYWW2CSQ7UPSET46QGDRT52RFGAROTTA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.470567977 (415h3m6.29827637s) [noise=true , address=145.79.227.15:4874 , piece_id=FNCCIMIVJTGZ2X5TOYOZH727JXHYM7HG3HBHBTZDAG67SHZLYYYA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.470980950 (415h3m6.298689349s) [noise=true , address=145.79.227.50:4373 , piece_id=4U55T5QDYCE6AZ3NCTPAVECPAELPQUKT3HV7GZL7JFRMWRN2AZUQ, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.471440098 (415h3m6.299148489s) [noise=true , address=145.79.227.47:4266 , piece_id=3NFYV2PWBNTAOSFHENM4J5PZ7XXE72UKTQVXLTVQRHGK4DUXAKDA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.560766475 (415h3m6.388474871s) [upload=431917]: Next returned nil segment Dec 22 22:31:05.560785161 (415h3m6.388493562s) [upload=431917]: Waiting for error group managing segments... Dec 22 22:31:05.566117878 (415h3m6.39382628s) [noise=true , address=45.91.116.4:4939 , piece_id=QYNKMKDLQJV7B6M2E34DU53Y7YI6VBCUW3C7ZLQOI4XWTBXZHKDQ, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.566723306 (415h3m6.394431687s) [noise=true , address=45.91.116.11:4117 , piece_id=WLMO5E3BIFFL3TOAUSSBGORIZ6DOCWFCMVT4AVFPPGPSKP2OKDAA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.566992667 (415h3m6.394701063s) [noise=true , address=145.79.227.36:4519 , piece_id=JUKRRIZWVGKEDD3GPJ7HMY2MEQZNMUCTKTXR6ZLSIERRONTZKHCQ, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.567464626 (415h3m6.395173019s) [noise=true , address=145.79.227.47:4266 , piece_id=3NFYV2PWBNTAOSFHENM4J5PZ7XXE72UKTQVXLTVQRHGK4DUXAKDA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.567924492 (415h3m6.395632883s) [noise=true , address=145.79.227.15:4874 , piece_id=FNCCIMIVJTGZ2X5TOYOZH727JXHYM7HG3HBHBTZDAG67SHZLYYYA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.568807294 (415h3m6.396515685s) [noise=true , address=145.79.227.51:4409 , piece_id=ATH3F4DFOUUIFFJZRA2SSRRL7AKWZMQNQRYOZCVFFDLTFCFP236A, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.570579823 (415h3m6.398288214s) [noise=true , address=145.79.227.48:4301 , piece_id=LBIYTJNOX5J4M3HDE7FGF5CSARBITW2AK3P2DSYFLFKUOIHZUKOQ, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.571632758 (415h3m6.399341149s) [noise=true , address=145.79.227.43:4157 , piece_id=3W3LVTU37DNHWS52IGIIH7SXWXVNPGEY27RQNT272L3Z64GNR4KA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.573168383 (415h3m6.400876774s) [noise=true , address=145.79.227.38:4979 , piece_id=HLMQC5OEB4OL4OZSRMCEVAWR5QM2T4MQUBPTHRVQLEAHPBWPQF6Q, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.579995420 (415h3m6.407703816s) [noise=true , address=45.91.116.14:4190 , piece_id=V4T6Z7K3C75LQEXNQIYZGYWW2CSQ7UPSET46QGDRT52RFGAROTTA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.592093172 (415h3m6.419801563s) [noise=true , address=145.79.227.37:4565 , piece_id=554RHTYPG27UNCFPSRMOB245AX4QXL2QFIYECNTVWNNJMAL4BTYA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:09.946500434 (415h3m10.774208816s) [noise=true , address=145.79.227.40:4051 , piece_id=55RRAD7DQX6SR7RBLYAV7MD7XRWPNIWHTVVBQZ2ZL3OE2L7LRRIQ, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:35.398235543 (415h3m36.225943943s) [noise=true , address=145.79.227.39:4013 , piece_id=JJVURGSLKY2UQ4FKNCLATA3ZBD5SDL32XJAG42KH2U6OYK3KTS6Q, seg_pos={0 0}, upload=431917]: Done uploading piece. err: ecclient: upload failed (node:121scLvKEW9zw2aEz22zpremcjiop67eQ8v2uqpmaD5aZzCAP7b, address:145.79.227.39:4013): protocol: write tcp 159.223.143.39:60842->145.79.227.39:4013: use of closed network connection; write tcp 159.223.143.39:60842->145.79.227.39:4013: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:60842->145.79.227.39:4013: use of closed network connection Dec 22 22:31:35.398562305 (415h3m36.226270699s) [noise=true , address=145.79.227.32:4835 , piece_id=BRE3JH3XKPXB6FN7WB3W6ED7MOFES6HYLR34YGR6L5AX7RPIQ7IA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: ecclient: upload failed (node:16vVfeoqCkbMYCRjMMWaKBnCvQBBJPMgwQtWTq9HKQrD9SS17y, address:145.79.227.32:4835): protocol: write tcp 159.223.143.39:45098->145.79.227.32:4835: use of closed network connection; write tcp 159.223.143.39:45098->145.79.227.32:4835: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:45098->145.79.227.32:4835: use of closed network connection Dec 22 22:31:35.399183028 (415h3m36.226891422s) [noise=true , address=145.79.227.50:4373 , piece_id=4U55T5QDYCE6AZ3NCTPAVECPAELPQUKT3HV7GZL7JFRMWRN2AZUQ, seg_pos={0 0}, upload=431917]: Done uploading piece. err: ecclient: upload failed (node:12AQGSkJuNJDxoeqhChUenWxHwF5yK41UESCF63FQz3XvabXXfz, address:145.79.227.50:4373): protocol: read tcp 159.223.143.39:56028->145.79.227.50:4373: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 22 22:31:35.399278021 (415h3m36.226986417s) [noise=true , address=145.79.227.29:4943 , piece_id=G4TMZ6IAIZBDON3UEDPH4LJQFHV3JLALEXSJ3VI4XN6WHIZOB57Q, seg_pos={0 0}, upload=431917]: Done uploading piece. err: ecclient: upload failed (node:12fc5Trj8XxacZhzxdksUVT5CbG6uTmiwdUPjreYi4UMPY1TXWz, address:145.79.227.29:4943): protocol: write tcp 159.223.143.39:50820->145.79.227.29:4943: use of closed network connection; write tcp 159.223.143.39:50820->145.79.227.29:4943: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:50820->145.79.227.29:4943: use of closed network connection Dec 22 22:31:35.399379485 (415h3m36.227087866s) [seg_pos={0 0}, upload=431917]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 22 22:31:35.421122544 (415h3m36.248830935s) [upload=431917]: Done uploading segments... Dec 22 22:31:35.421174369 (415h3m36.248882769s) [upload=431917]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 544.238819ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................................X...........................] duration: 392.884394ms trace: https://jaeger.dev.storj.io/trace/7013ffb1a981d605 FAIL (2025-12-22 22:52:47 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 22 22:31:05.439510308 (415h3m6.2672187s) [upload=431917]: Starting upload Dec 22 22:31:05.439581993 (415h3m6.26729037s) [upload=431917]: Done starting upload Dec 22 22:31:05.439812792 (415h3m6.267521185s) [upload=431917]: Uploading segments... Dec 22 22:31:05.439942524 (415h3m6.267650918s) [upload=431917]: Got next segment. Inline: false Dec 22 22:31:05.440022744 (415h3m6.267731174s) [upload=431917]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 22 22:31:05.440029670 (415h3m6.267738072s) [upload=431917]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 22 22:31:05.467186372 (415h3m6.294894769s) [seg_pos={0 0}, upload=431917]: Begin upload segment... Dec 22 22:31:05.467265149 (415h3m6.294973535s) [seg_pos={0 0}, upload=431917]: Done begin upload segment. Dec 22 22:31:05.467296974 (415h3m6.295005373s) [noise=true , address=145.79.227.51:4409 , piece_id=ATH3F4DFOUUIFFJZRA2SSRRL7AKWZMQNQRYOZCVFFDLTFCFP236A, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.467389853 (415h3m6.295098247s) [noise=true , address=145.79.227.39:4013 , piece_id=JJVURGSLKY2UQ4FKNCLATA3ZBD5SDL32XJAG42KH2U6OYK3KTS6Q, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.467912104 (415h3m6.295620486s) [noise=true , address=145.79.227.36:4519 , piece_id=JUKRRIZWVGKEDD3GPJ7HMY2MEQZNMUCTKTXR6ZLSIERRONTZKHCQ, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.468324863 (415h3m6.296033263s) [noise=true , address=145.79.227.38:4979 , piece_id=HLMQC5OEB4OL4OZSRMCEVAWR5QM2T4MQUBPTHRVQLEAHPBWPQF6Q, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.468334764 (415h3m6.296043147s) [noise=true , address=145.79.227.40:4051 , piece_id=55RRAD7DQX6SR7RBLYAV7MD7XRWPNIWHTVVBQZ2ZL3OE2L7LRRIQ, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.468742525 (415h3m6.296450916s) [noise=true , address=45.91.116.4:4939 , piece_id=QYNKMKDLQJV7B6M2E34DU53Y7YI6VBCUW3C7ZLQOI4XWTBXZHKDQ, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.468915034 (415h3m6.296623414s) [noise=true , address=45.91.116.11:4117 , piece_id=WLMO5E3BIFFL3TOAUSSBGORIZ6DOCWFCMVT4AVFPPGPSKP2OKDAA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.469069393 (415h3m6.296777787s) [noise=true , address=145.79.227.32:4835 , piece_id=BRE3JH3XKPXB6FN7WB3W6ED7MOFES6HYLR34YGR6L5AX7RPIQ7IA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.469285789 (415h3m6.296994183s) [noise=true , address=145.79.227.29:4943 , piece_id=G4TMZ6IAIZBDON3UEDPH4LJQFHV3JLALEXSJ3VI4XN6WHIZOB57Q, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.469411962 (415h3m6.297120355s) [noise=true , address=145.79.227.48:4301 , piece_id=LBIYTJNOX5J4M3HDE7FGF5CSARBITW2AK3P2DSYFLFKUOIHZUKOQ, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.469749379 (415h3m6.297457773s) [noise=true , address=145.79.227.37:4565 , piece_id=554RHTYPG27UNCFPSRMOB245AX4QXL2QFIYECNTVWNNJMAL4BTYA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.470080828 (415h3m6.29778922s) [noise=true , address=145.79.227.43:4157 , piece_id=3W3LVTU37DNHWS52IGIIH7SXWXVNPGEY27RQNT272L3Z64GNR4KA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.470477660 (415h3m6.298186058s) [noise=true , address=45.91.116.14:4190 , piece_id=V4T6Z7K3C75LQEXNQIYZGYWW2CSQ7UPSET46QGDRT52RFGAROTTA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.470567977 (415h3m6.29827637s) [noise=true , address=145.79.227.15:4874 , piece_id=FNCCIMIVJTGZ2X5TOYOZH727JXHYM7HG3HBHBTZDAG67SHZLYYYA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.470980950 (415h3m6.298689349s) [noise=true , address=145.79.227.50:4373 , piece_id=4U55T5QDYCE6AZ3NCTPAVECPAELPQUKT3HV7GZL7JFRMWRN2AZUQ, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.471440098 (415h3m6.299148489s) [noise=true , address=145.79.227.47:4266 , piece_id=3NFYV2PWBNTAOSFHENM4J5PZ7XXE72UKTQVXLTVQRHGK4DUXAKDA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.560766475 (415h3m6.388474871s) [upload=431917]: Next returned nil segment Dec 22 22:31:05.560785161 (415h3m6.388493562s) [upload=431917]: Waiting for error group managing segments... Dec 22 22:31:05.566117878 (415h3m6.39382628s) [noise=true , address=45.91.116.4:4939 , piece_id=QYNKMKDLQJV7B6M2E34DU53Y7YI6VBCUW3C7ZLQOI4XWTBXZHKDQ, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.566723306 (415h3m6.394431687s) [noise=true , address=45.91.116.11:4117 , piece_id=WLMO5E3BIFFL3TOAUSSBGORIZ6DOCWFCMVT4AVFPPGPSKP2OKDAA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.566992667 (415h3m6.394701063s) [noise=true , address=145.79.227.36:4519 , piece_id=JUKRRIZWVGKEDD3GPJ7HMY2MEQZNMUCTKTXR6ZLSIERRONTZKHCQ, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.567464626 (415h3m6.395173019s) [noise=true , address=145.79.227.47:4266 , piece_id=3NFYV2PWBNTAOSFHENM4J5PZ7XXE72UKTQVXLTVQRHGK4DUXAKDA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.567924492 (415h3m6.395632883s) [noise=true , address=145.79.227.15:4874 , piece_id=FNCCIMIVJTGZ2X5TOYOZH727JXHYM7HG3HBHBTZDAG67SHZLYYYA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.568807294 (415h3m6.396515685s) [noise=true , address=145.79.227.51:4409 , piece_id=ATH3F4DFOUUIFFJZRA2SSRRL7AKWZMQNQRYOZCVFFDLTFCFP236A, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.570579823 (415h3m6.398288214s) [noise=true , address=145.79.227.48:4301 , piece_id=LBIYTJNOX5J4M3HDE7FGF5CSARBITW2AK3P2DSYFLFKUOIHZUKOQ, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.571632758 (415h3m6.399341149s) [noise=true , address=145.79.227.43:4157 , piece_id=3W3LVTU37DNHWS52IGIIH7SXWXVNPGEY27RQNT272L3Z64GNR4KA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.573168383 (415h3m6.400876774s) [noise=true , address=145.79.227.38:4979 , piece_id=HLMQC5OEB4OL4OZSRMCEVAWR5QM2T4MQUBPTHRVQLEAHPBWPQF6Q, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.579995420 (415h3m6.407703816s) [noise=true , address=45.91.116.14:4190 , piece_id=V4T6Z7K3C75LQEXNQIYZGYWW2CSQ7UPSET46QGDRT52RFGAROTTA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.592093172 (415h3m6.419801563s) [noise=true , address=145.79.227.37:4565 , piece_id=554RHTYPG27UNCFPSRMOB245AX4QXL2QFIYECNTVWNNJMAL4BTYA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:09.946500434 (415h3m10.774208816s) [noise=true , address=145.79.227.40:4051 , piece_id=55RRAD7DQX6SR7RBLYAV7MD7XRWPNIWHTVVBQZ2ZL3OE2L7LRRIQ, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:35.398235543 (415h3m36.225943943s) [noise=true , address=145.79.227.39:4013 , piece_id=JJVURGSLKY2UQ4FKNCLATA3ZBD5SDL32XJAG42KH2U6OYK3KTS6Q, seg_pos={0 0}, upload=431917]: Done uploading piece. err: ecclient: upload failed (node:121scLvKEW9zw2aEz22zpremcjiop67eQ8v2uqpmaD5aZzCAP7b, address:145.79.227.39:4013): protocol: write tcp 159.223.143.39:60842->145.79.227.39:4013: use of closed network connection; write tcp 159.223.143.39:60842->145.79.227.39:4013: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:60842->145.79.227.39:4013: use of closed network connection Dec 22 22:31:35.398562305 (415h3m36.226270699s) [noise=true , address=145.79.227.32:4835 , piece_id=BRE3JH3XKPXB6FN7WB3W6ED7MOFES6HYLR34YGR6L5AX7RPIQ7IA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: ecclient: upload failed (node:16vVfeoqCkbMYCRjMMWaKBnCvQBBJPMgwQtWTq9HKQrD9SS17y, address:145.79.227.32:4835): protocol: write tcp 159.223.143.39:45098->145.79.227.32:4835: use of closed network connection; write tcp 159.223.143.39:45098->145.79.227.32:4835: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:45098->145.79.227.32:4835: use of closed network connection Dec 22 22:31:35.399183028 (415h3m36.226891422s) [noise=true , address=145.79.227.50:4373 , piece_id=4U55T5QDYCE6AZ3NCTPAVECPAELPQUKT3HV7GZL7JFRMWRN2AZUQ, seg_pos={0 0}, upload=431917]: Done uploading piece. err: ecclient: upload failed (node:12AQGSkJuNJDxoeqhChUenWxHwF5yK41UESCF63FQz3XvabXXfz, address:145.79.227.50:4373): protocol: read tcp 159.223.143.39:56028->145.79.227.50:4373: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 22 22:31:35.399278021 (415h3m36.226986417s) [noise=true , address=145.79.227.29:4943 , piece_id=G4TMZ6IAIZBDON3UEDPH4LJQFHV3JLALEXSJ3VI4XN6WHIZOB57Q, seg_pos={0 0}, upload=431917]: Done uploading piece. err: ecclient: upload failed (node:12fc5Trj8XxacZhzxdksUVT5CbG6uTmiwdUPjreYi4UMPY1TXWz, address:145.79.227.29:4943): protocol: write tcp 159.223.143.39:50820->145.79.227.29:4943: use of closed network connection; write tcp 159.223.143.39:50820->145.79.227.29:4943: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:50820->145.79.227.29:4943: use of closed network connection Dec 22 22:31:35.399379485 (415h3m36.227087866s) [seg_pos={0 0}, upload=431917]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 22 22:31:35.421122544 (415h3m36.248830935s) [upload=431917]: Done uploading segments... Dec 22 22:31:35.421174369 (415h3m36.248882769s) [upload=431917]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 392.907016ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................................X..........................] duration: 456.60347ms trace: https://jaeger.dev.storj.io/trace/7013ffb1a981d605 FAIL (2025-12-22 22:51:52 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 22 22:31:05.439510308 (415h3m6.2672187s) [upload=431917]: Starting upload Dec 22 22:31:05.439581993 (415h3m6.26729037s) [upload=431917]: Done starting upload Dec 22 22:31:05.439812792 (415h3m6.267521185s) [upload=431917]: Uploading segments... Dec 22 22:31:05.439942524 (415h3m6.267650918s) [upload=431917]: Got next segment. Inline: false Dec 22 22:31:05.440022744 (415h3m6.267731174s) [upload=431917]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 22 22:31:05.440029670 (415h3m6.267738072s) [upload=431917]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 22 22:31:05.467186372 (415h3m6.294894769s) [seg_pos={0 0}, upload=431917]: Begin upload segment... Dec 22 22:31:05.467265149 (415h3m6.294973535s) [seg_pos={0 0}, upload=431917]: Done begin upload segment. Dec 22 22:31:05.467296974 (415h3m6.295005373s) [noise=true , address=145.79.227.51:4409 , piece_id=ATH3F4DFOUUIFFJZRA2SSRRL7AKWZMQNQRYOZCVFFDLTFCFP236A, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.467389853 (415h3m6.295098247s) [noise=true , address=145.79.227.39:4013 , piece_id=JJVURGSLKY2UQ4FKNCLATA3ZBD5SDL32XJAG42KH2U6OYK3KTS6Q, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.467912104 (415h3m6.295620486s) [noise=true , address=145.79.227.36:4519 , piece_id=JUKRRIZWVGKEDD3GPJ7HMY2MEQZNMUCTKTXR6ZLSIERRONTZKHCQ, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.468324863 (415h3m6.296033263s) [noise=true , address=145.79.227.38:4979 , piece_id=HLMQC5OEB4OL4OZSRMCEVAWR5QM2T4MQUBPTHRVQLEAHPBWPQF6Q, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.468334764 (415h3m6.296043147s) [noise=true , address=145.79.227.40:4051 , piece_id=55RRAD7DQX6SR7RBLYAV7MD7XRWPNIWHTVVBQZ2ZL3OE2L7LRRIQ, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.468742525 (415h3m6.296450916s) [noise=true , address=45.91.116.4:4939 , piece_id=QYNKMKDLQJV7B6M2E34DU53Y7YI6VBCUW3C7ZLQOI4XWTBXZHKDQ, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.468915034 (415h3m6.296623414s) [noise=true , address=45.91.116.11:4117 , piece_id=WLMO5E3BIFFL3TOAUSSBGORIZ6DOCWFCMVT4AVFPPGPSKP2OKDAA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.469069393 (415h3m6.296777787s) [noise=true , address=145.79.227.32:4835 , piece_id=BRE3JH3XKPXB6FN7WB3W6ED7MOFES6HYLR34YGR6L5AX7RPIQ7IA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.469285789 (415h3m6.296994183s) [noise=true , address=145.79.227.29:4943 , piece_id=G4TMZ6IAIZBDON3UEDPH4LJQFHV3JLALEXSJ3VI4XN6WHIZOB57Q, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.469411962 (415h3m6.297120355s) [noise=true , address=145.79.227.48:4301 , piece_id=LBIYTJNOX5J4M3HDE7FGF5CSARBITW2AK3P2DSYFLFKUOIHZUKOQ, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.469749379 (415h3m6.297457773s) [noise=true , address=145.79.227.37:4565 , piece_id=554RHTYPG27UNCFPSRMOB245AX4QXL2QFIYECNTVWNNJMAL4BTYA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.470080828 (415h3m6.29778922s) [noise=true , address=145.79.227.43:4157 , piece_id=3W3LVTU37DNHWS52IGIIH7SXWXVNPGEY27RQNT272L3Z64GNR4KA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.470477660 (415h3m6.298186058s) [noise=true , address=45.91.116.14:4190 , piece_id=V4T6Z7K3C75LQEXNQIYZGYWW2CSQ7UPSET46QGDRT52RFGAROTTA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.470567977 (415h3m6.29827637s) [noise=true , address=145.79.227.15:4874 , piece_id=FNCCIMIVJTGZ2X5TOYOZH727JXHYM7HG3HBHBTZDAG67SHZLYYYA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.470980950 (415h3m6.298689349s) [noise=true , address=145.79.227.50:4373 , piece_id=4U55T5QDYCE6AZ3NCTPAVECPAELPQUKT3HV7GZL7JFRMWRN2AZUQ, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.471440098 (415h3m6.299148489s) [noise=true , address=145.79.227.47:4266 , piece_id=3NFYV2PWBNTAOSFHENM4J5PZ7XXE72UKTQVXLTVQRHGK4DUXAKDA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.560766475 (415h3m6.388474871s) [upload=431917]: Next returned nil segment Dec 22 22:31:05.560785161 (415h3m6.388493562s) [upload=431917]: Waiting for error group managing segments... Dec 22 22:31:05.566117878 (415h3m6.39382628s) [noise=true , address=45.91.116.4:4939 , piece_id=QYNKMKDLQJV7B6M2E34DU53Y7YI6VBCUW3C7ZLQOI4XWTBXZHKDQ, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.566723306 (415h3m6.394431687s) [noise=true , address=45.91.116.11:4117 , piece_id=WLMO5E3BIFFL3TOAUSSBGORIZ6DOCWFCMVT4AVFPPGPSKP2OKDAA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.566992667 (415h3m6.394701063s) [noise=true , address=145.79.227.36:4519 , piece_id=JUKRRIZWVGKEDD3GPJ7HMY2MEQZNMUCTKTXR6ZLSIERRONTZKHCQ, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.567464626 (415h3m6.395173019s) [noise=true , address=145.79.227.47:4266 , piece_id=3NFYV2PWBNTAOSFHENM4J5PZ7XXE72UKTQVXLTVQRHGK4DUXAKDA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.567924492 (415h3m6.395632883s) [noise=true , address=145.79.227.15:4874 , piece_id=FNCCIMIVJTGZ2X5TOYOZH727JXHYM7HG3HBHBTZDAG67SHZLYYYA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.568807294 (415h3m6.396515685s) [noise=true , address=145.79.227.51:4409 , piece_id=ATH3F4DFOUUIFFJZRA2SSRRL7AKWZMQNQRYOZCVFFDLTFCFP236A, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.570579823 (415h3m6.398288214s) [noise=true , address=145.79.227.48:4301 , piece_id=LBIYTJNOX5J4M3HDE7FGF5CSARBITW2AK3P2DSYFLFKUOIHZUKOQ, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.571632758 (415h3m6.399341149s) [noise=true , address=145.79.227.43:4157 , piece_id=3W3LVTU37DNHWS52IGIIH7SXWXVNPGEY27RQNT272L3Z64GNR4KA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.573168383 (415h3m6.400876774s) [noise=true , address=145.79.227.38:4979 , piece_id=HLMQC5OEB4OL4OZSRMCEVAWR5QM2T4MQUBPTHRVQLEAHPBWPQF6Q, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.579995420 (415h3m6.407703816s) [noise=true , address=45.91.116.14:4190 , piece_id=V4T6Z7K3C75LQEXNQIYZGYWW2CSQ7UPSET46QGDRT52RFGAROTTA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.592093172 (415h3m6.419801563s) [noise=true , address=145.79.227.37:4565 , piece_id=554RHTYPG27UNCFPSRMOB245AX4QXL2QFIYECNTVWNNJMAL4BTYA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:09.946500434 (415h3m10.774208816s) [noise=true , address=145.79.227.40:4051 , piece_id=55RRAD7DQX6SR7RBLYAV7MD7XRWPNIWHTVVBQZ2ZL3OE2L7LRRIQ, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:35.398235543 (415h3m36.225943943s) [noise=true , address=145.79.227.39:4013 , piece_id=JJVURGSLKY2UQ4FKNCLATA3ZBD5SDL32XJAG42KH2U6OYK3KTS6Q, seg_pos={0 0}, upload=431917]: Done uploading piece. err: ecclient: upload failed (node:121scLvKEW9zw2aEz22zpremcjiop67eQ8v2uqpmaD5aZzCAP7b, address:145.79.227.39:4013): protocol: write tcp 159.223.143.39:60842->145.79.227.39:4013: use of closed network connection; write tcp 159.223.143.39:60842->145.79.227.39:4013: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:60842->145.79.227.39:4013: use of closed network connection Dec 22 22:31:35.398562305 (415h3m36.226270699s) [noise=true , address=145.79.227.32:4835 , piece_id=BRE3JH3XKPXB6FN7WB3W6ED7MOFES6HYLR34YGR6L5AX7RPIQ7IA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: ecclient: upload failed (node:16vVfeoqCkbMYCRjMMWaKBnCvQBBJPMgwQtWTq9HKQrD9SS17y, address:145.79.227.32:4835): protocol: write tcp 159.223.143.39:45098->145.79.227.32:4835: use of closed network connection; write tcp 159.223.143.39:45098->145.79.227.32:4835: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:45098->145.79.227.32:4835: use of closed network connection Dec 22 22:31:35.399183028 (415h3m36.226891422s) [noise=true , address=145.79.227.50:4373 , piece_id=4U55T5QDYCE6AZ3NCTPAVECPAELPQUKT3HV7GZL7JFRMWRN2AZUQ, seg_pos={0 0}, upload=431917]: Done uploading piece. err: ecclient: upload failed (node:12AQGSkJuNJDxoeqhChUenWxHwF5yK41UESCF63FQz3XvabXXfz, address:145.79.227.50:4373): protocol: read tcp 159.223.143.39:56028->145.79.227.50:4373: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 22 22:31:35.399278021 (415h3m36.226986417s) [noise=true , address=145.79.227.29:4943 , piece_id=G4TMZ6IAIZBDON3UEDPH4LJQFHV3JLALEXSJ3VI4XN6WHIZOB57Q, seg_pos={0 0}, upload=431917]: Done uploading piece. err: ecclient: upload failed (node:12fc5Trj8XxacZhzxdksUVT5CbG6uTmiwdUPjreYi4UMPY1TXWz, address:145.79.227.29:4943): protocol: write tcp 159.223.143.39:50820->145.79.227.29:4943: use of closed network connection; write tcp 159.223.143.39:50820->145.79.227.29:4943: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:50820->145.79.227.29:4943: use of closed network connection Dec 22 22:31:35.399379485 (415h3m36.227087866s) [seg_pos={0 0}, upload=431917]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 22 22:31:35.421122544 (415h3m36.248830935s) [upload=431917]: Done uploading segments... Dec 22 22:31:35.421174369 (415h3m36.248882769s) [upload=431917]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 456.631125ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................................X.........................] duration: 408.155014ms trace: https://jaeger.dev.storj.io/trace/7013ffb1a981d605 FAIL (2025-12-22 22:51:17 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 22 22:31:05.439510308 (415h3m6.2672187s) [upload=431917]: Starting upload Dec 22 22:31:05.439581993 (415h3m6.26729037s) [upload=431917]: Done starting upload Dec 22 22:31:05.439812792 (415h3m6.267521185s) [upload=431917]: Uploading segments... Dec 22 22:31:05.439942524 (415h3m6.267650918s) [upload=431917]: Got next segment. Inline: false Dec 22 22:31:05.440022744 (415h3m6.267731174s) [upload=431917]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 22 22:31:05.440029670 (415h3m6.267738072s) [upload=431917]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 22 22:31:05.467186372 (415h3m6.294894769s) [seg_pos={0 0}, upload=431917]: Begin upload segment... Dec 22 22:31:05.467265149 (415h3m6.294973535s) [seg_pos={0 0}, upload=431917]: Done begin upload segment. Dec 22 22:31:05.467296974 (415h3m6.295005373s) [noise=true , address=145.79.227.51:4409 , piece_id=ATH3F4DFOUUIFFJZRA2SSRRL7AKWZMQNQRYOZCVFFDLTFCFP236A, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.467389853 (415h3m6.295098247s) [noise=true , address=145.79.227.39:4013 , piece_id=JJVURGSLKY2UQ4FKNCLATA3ZBD5SDL32XJAG42KH2U6OYK3KTS6Q, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.467912104 (415h3m6.295620486s) [noise=true , address=145.79.227.36:4519 , piece_id=JUKRRIZWVGKEDD3GPJ7HMY2MEQZNMUCTKTXR6ZLSIERRONTZKHCQ, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.468324863 (415h3m6.296033263s) [noise=true , address=145.79.227.38:4979 , piece_id=HLMQC5OEB4OL4OZSRMCEVAWR5QM2T4MQUBPTHRVQLEAHPBWPQF6Q, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.468334764 (415h3m6.296043147s) [noise=true , address=145.79.227.40:4051 , piece_id=55RRAD7DQX6SR7RBLYAV7MD7XRWPNIWHTVVBQZ2ZL3OE2L7LRRIQ, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.468742525 (415h3m6.296450916s) [noise=true , address=45.91.116.4:4939 , piece_id=QYNKMKDLQJV7B6M2E34DU53Y7YI6VBCUW3C7ZLQOI4XWTBXZHKDQ, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.468915034 (415h3m6.296623414s) [noise=true , address=45.91.116.11:4117 , piece_id=WLMO5E3BIFFL3TOAUSSBGORIZ6DOCWFCMVT4AVFPPGPSKP2OKDAA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.469069393 (415h3m6.296777787s) [noise=true , address=145.79.227.32:4835 , piece_id=BRE3JH3XKPXB6FN7WB3W6ED7MOFES6HYLR34YGR6L5AX7RPIQ7IA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.469285789 (415h3m6.296994183s) [noise=true , address=145.79.227.29:4943 , piece_id=G4TMZ6IAIZBDON3UEDPH4LJQFHV3JLALEXSJ3VI4XN6WHIZOB57Q, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.469411962 (415h3m6.297120355s) [noise=true , address=145.79.227.48:4301 , piece_id=LBIYTJNOX5J4M3HDE7FGF5CSARBITW2AK3P2DSYFLFKUOIHZUKOQ, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.469749379 (415h3m6.297457773s) [noise=true , address=145.79.227.37:4565 , piece_id=554RHTYPG27UNCFPSRMOB245AX4QXL2QFIYECNTVWNNJMAL4BTYA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.470080828 (415h3m6.29778922s) [noise=true , address=145.79.227.43:4157 , piece_id=3W3LVTU37DNHWS52IGIIH7SXWXVNPGEY27RQNT272L3Z64GNR4KA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.470477660 (415h3m6.298186058s) [noise=true , address=45.91.116.14:4190 , piece_id=V4T6Z7K3C75LQEXNQIYZGYWW2CSQ7UPSET46QGDRT52RFGAROTTA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.470567977 (415h3m6.29827637s) [noise=true , address=145.79.227.15:4874 , piece_id=FNCCIMIVJTGZ2X5TOYOZH727JXHYM7HG3HBHBTZDAG67SHZLYYYA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.470980950 (415h3m6.298689349s) [noise=true , address=145.79.227.50:4373 , piece_id=4U55T5QDYCE6AZ3NCTPAVECPAELPQUKT3HV7GZL7JFRMWRN2AZUQ, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.471440098 (415h3m6.299148489s) [noise=true , address=145.79.227.47:4266 , piece_id=3NFYV2PWBNTAOSFHENM4J5PZ7XXE72UKTQVXLTVQRHGK4DUXAKDA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.560766475 (415h3m6.388474871s) [upload=431917]: Next returned nil segment Dec 22 22:31:05.560785161 (415h3m6.388493562s) [upload=431917]: Waiting for error group managing segments... Dec 22 22:31:05.566117878 (415h3m6.39382628s) [noise=true , address=45.91.116.4:4939 , piece_id=QYNKMKDLQJV7B6M2E34DU53Y7YI6VBCUW3C7ZLQOI4XWTBXZHKDQ, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.566723306 (415h3m6.394431687s) [noise=true , address=45.91.116.11:4117 , piece_id=WLMO5E3BIFFL3TOAUSSBGORIZ6DOCWFCMVT4AVFPPGPSKP2OKDAA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.566992667 (415h3m6.394701063s) [noise=true , address=145.79.227.36:4519 , piece_id=JUKRRIZWVGKEDD3GPJ7HMY2MEQZNMUCTKTXR6ZLSIERRONTZKHCQ, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.567464626 (415h3m6.395173019s) [noise=true , address=145.79.227.47:4266 , piece_id=3NFYV2PWBNTAOSFHENM4J5PZ7XXE72UKTQVXLTVQRHGK4DUXAKDA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.567924492 (415h3m6.395632883s) [noise=true , address=145.79.227.15:4874 , piece_id=FNCCIMIVJTGZ2X5TOYOZH727JXHYM7HG3HBHBTZDAG67SHZLYYYA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.568807294 (415h3m6.396515685s) [noise=true , address=145.79.227.51:4409 , piece_id=ATH3F4DFOUUIFFJZRA2SSRRL7AKWZMQNQRYOZCVFFDLTFCFP236A, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.570579823 (415h3m6.398288214s) [noise=true , address=145.79.227.48:4301 , piece_id=LBIYTJNOX5J4M3HDE7FGF5CSARBITW2AK3P2DSYFLFKUOIHZUKOQ, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.571632758 (415h3m6.399341149s) [noise=true , address=145.79.227.43:4157 , piece_id=3W3LVTU37DNHWS52IGIIH7SXWXVNPGEY27RQNT272L3Z64GNR4KA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.573168383 (415h3m6.400876774s) [noise=true , address=145.79.227.38:4979 , piece_id=HLMQC5OEB4OL4OZSRMCEVAWR5QM2T4MQUBPTHRVQLEAHPBWPQF6Q, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.579995420 (415h3m6.407703816s) [noise=true , address=45.91.116.14:4190 , piece_id=V4T6Z7K3C75LQEXNQIYZGYWW2CSQ7UPSET46QGDRT52RFGAROTTA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.592093172 (415h3m6.419801563s) [noise=true , address=145.79.227.37:4565 , piece_id=554RHTYPG27UNCFPSRMOB245AX4QXL2QFIYECNTVWNNJMAL4BTYA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:09.946500434 (415h3m10.774208816s) [noise=true , address=145.79.227.40:4051 , piece_id=55RRAD7DQX6SR7RBLYAV7MD7XRWPNIWHTVVBQZ2ZL3OE2L7LRRIQ, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:35.398235543 (415h3m36.225943943s) [noise=true , address=145.79.227.39:4013 , piece_id=JJVURGSLKY2UQ4FKNCLATA3ZBD5SDL32XJAG42KH2U6OYK3KTS6Q, seg_pos={0 0}, upload=431917]: Done uploading piece. err: ecclient: upload failed (node:121scLvKEW9zw2aEz22zpremcjiop67eQ8v2uqpmaD5aZzCAP7b, address:145.79.227.39:4013): protocol: write tcp 159.223.143.39:60842->145.79.227.39:4013: use of closed network connection; write tcp 159.223.143.39:60842->145.79.227.39:4013: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:60842->145.79.227.39:4013: use of closed network connection Dec 22 22:31:35.398562305 (415h3m36.226270699s) [noise=true , address=145.79.227.32:4835 , piece_id=BRE3JH3XKPXB6FN7WB3W6ED7MOFES6HYLR34YGR6L5AX7RPIQ7IA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: ecclient: upload failed (node:16vVfeoqCkbMYCRjMMWaKBnCvQBBJPMgwQtWTq9HKQrD9SS17y, address:145.79.227.32:4835): protocol: write tcp 159.223.143.39:45098->145.79.227.32:4835: use of closed network connection; write tcp 159.223.143.39:45098->145.79.227.32:4835: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:45098->145.79.227.32:4835: use of closed network connection Dec 22 22:31:35.399183028 (415h3m36.226891422s) [noise=true , address=145.79.227.50:4373 , piece_id=4U55T5QDYCE6AZ3NCTPAVECPAELPQUKT3HV7GZL7JFRMWRN2AZUQ, seg_pos={0 0}, upload=431917]: Done uploading piece. err: ecclient: upload failed (node:12AQGSkJuNJDxoeqhChUenWxHwF5yK41UESCF63FQz3XvabXXfz, address:145.79.227.50:4373): protocol: read tcp 159.223.143.39:56028->145.79.227.50:4373: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 22 22:31:35.399278021 (415h3m36.226986417s) [noise=true , address=145.79.227.29:4943 , piece_id=G4TMZ6IAIZBDON3UEDPH4LJQFHV3JLALEXSJ3VI4XN6WHIZOB57Q, seg_pos={0 0}, upload=431917]: Done uploading piece. err: ecclient: upload failed (node:12fc5Trj8XxacZhzxdksUVT5CbG6uTmiwdUPjreYi4UMPY1TXWz, address:145.79.227.29:4943): protocol: write tcp 159.223.143.39:50820->145.79.227.29:4943: use of closed network connection; write tcp 159.223.143.39:50820->145.79.227.29:4943: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:50820->145.79.227.29:4943: use of closed network connection Dec 22 22:31:35.399379485 (415h3m36.227087866s) [seg_pos={0 0}, upload=431917]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 22 22:31:35.421122544 (415h3m36.248830935s) [upload=431917]: Done uploading segments... Dec 22 22:31:35.421174369 (415h3m36.248882769s) [upload=431917]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 408.201167ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................................X........................] duration: 466.125041ms trace: https://jaeger.dev.storj.io/trace/7013ffb1a981d605 FAIL (2025-12-22 22:50:34 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 22 22:31:05.439510308 (415h3m6.2672187s) [upload=431917]: Starting upload Dec 22 22:31:05.439581993 (415h3m6.26729037s) [upload=431917]: Done starting upload Dec 22 22:31:05.439812792 (415h3m6.267521185s) [upload=431917]: Uploading segments... Dec 22 22:31:05.439942524 (415h3m6.267650918s) [upload=431917]: Got next segment. Inline: false Dec 22 22:31:05.440022744 (415h3m6.267731174s) [upload=431917]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 22 22:31:05.440029670 (415h3m6.267738072s) [upload=431917]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 22 22:31:05.467186372 (415h3m6.294894769s) [seg_pos={0 0}, upload=431917]: Begin upload segment... Dec 22 22:31:05.467265149 (415h3m6.294973535s) [seg_pos={0 0}, upload=431917]: Done begin upload segment. Dec 22 22:31:05.467296974 (415h3m6.295005373s) [noise=true , address=145.79.227.51:4409 , piece_id=ATH3F4DFOUUIFFJZRA2SSRRL7AKWZMQNQRYOZCVFFDLTFCFP236A, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.467389853 (415h3m6.295098247s) [noise=true , address=145.79.227.39:4013 , piece_id=JJVURGSLKY2UQ4FKNCLATA3ZBD5SDL32XJAG42KH2U6OYK3KTS6Q, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.467912104 (415h3m6.295620486s) [noise=true , address=145.79.227.36:4519 , piece_id=JUKRRIZWVGKEDD3GPJ7HMY2MEQZNMUCTKTXR6ZLSIERRONTZKHCQ, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.468324863 (415h3m6.296033263s) [noise=true , address=145.79.227.38:4979 , piece_id=HLMQC5OEB4OL4OZSRMCEVAWR5QM2T4MQUBPTHRVQLEAHPBWPQF6Q, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.468334764 (415h3m6.296043147s) [noise=true , address=145.79.227.40:4051 , piece_id=55RRAD7DQX6SR7RBLYAV7MD7XRWPNIWHTVVBQZ2ZL3OE2L7LRRIQ, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.468742525 (415h3m6.296450916s) [noise=true , address=45.91.116.4:4939 , piece_id=QYNKMKDLQJV7B6M2E34DU53Y7YI6VBCUW3C7ZLQOI4XWTBXZHKDQ, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.468915034 (415h3m6.296623414s) [noise=true , address=45.91.116.11:4117 , piece_id=WLMO5E3BIFFL3TOAUSSBGORIZ6DOCWFCMVT4AVFPPGPSKP2OKDAA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.469069393 (415h3m6.296777787s) [noise=true , address=145.79.227.32:4835 , piece_id=BRE3JH3XKPXB6FN7WB3W6ED7MOFES6HYLR34YGR6L5AX7RPIQ7IA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.469285789 (415h3m6.296994183s) [noise=true , address=145.79.227.29:4943 , piece_id=G4TMZ6IAIZBDON3UEDPH4LJQFHV3JLALEXSJ3VI4XN6WHIZOB57Q, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.469411962 (415h3m6.297120355s) [noise=true , address=145.79.227.48:4301 , piece_id=LBIYTJNOX5J4M3HDE7FGF5CSARBITW2AK3P2DSYFLFKUOIHZUKOQ, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.469749379 (415h3m6.297457773s) [noise=true , address=145.79.227.37:4565 , piece_id=554RHTYPG27UNCFPSRMOB245AX4QXL2QFIYECNTVWNNJMAL4BTYA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.470080828 (415h3m6.29778922s) [noise=true , address=145.79.227.43:4157 , piece_id=3W3LVTU37DNHWS52IGIIH7SXWXVNPGEY27RQNT272L3Z64GNR4KA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.470477660 (415h3m6.298186058s) [noise=true , address=45.91.116.14:4190 , piece_id=V4T6Z7K3C75LQEXNQIYZGYWW2CSQ7UPSET46QGDRT52RFGAROTTA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.470567977 (415h3m6.29827637s) [noise=true , address=145.79.227.15:4874 , piece_id=FNCCIMIVJTGZ2X5TOYOZH727JXHYM7HG3HBHBTZDAG67SHZLYYYA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.470980950 (415h3m6.298689349s) [noise=true , address=145.79.227.50:4373 , piece_id=4U55T5QDYCE6AZ3NCTPAVECPAELPQUKT3HV7GZL7JFRMWRN2AZUQ, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.471440098 (415h3m6.299148489s) [noise=true , address=145.79.227.47:4266 , piece_id=3NFYV2PWBNTAOSFHENM4J5PZ7XXE72UKTQVXLTVQRHGK4DUXAKDA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.560766475 (415h3m6.388474871s) [upload=431917]: Next returned nil segment Dec 22 22:31:05.560785161 (415h3m6.388493562s) [upload=431917]: Waiting for error group managing segments... Dec 22 22:31:05.566117878 (415h3m6.39382628s) [noise=true , address=45.91.116.4:4939 , piece_id=QYNKMKDLQJV7B6M2E34DU53Y7YI6VBCUW3C7ZLQOI4XWTBXZHKDQ, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.566723306 (415h3m6.394431687s) [noise=true , address=45.91.116.11:4117 , piece_id=WLMO5E3BIFFL3TOAUSSBGORIZ6DOCWFCMVT4AVFPPGPSKP2OKDAA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.566992667 (415h3m6.394701063s) [noise=true , address=145.79.227.36:4519 , piece_id=JUKRRIZWVGKEDD3GPJ7HMY2MEQZNMUCTKTXR6ZLSIERRONTZKHCQ, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.567464626 (415h3m6.395173019s) [noise=true , address=145.79.227.47:4266 , piece_id=3NFYV2PWBNTAOSFHENM4J5PZ7XXE72UKTQVXLTVQRHGK4DUXAKDA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.567924492 (415h3m6.395632883s) [noise=true , address=145.79.227.15:4874 , piece_id=FNCCIMIVJTGZ2X5TOYOZH727JXHYM7HG3HBHBTZDAG67SHZLYYYA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.568807294 (415h3m6.396515685s) [noise=true , address=145.79.227.51:4409 , piece_id=ATH3F4DFOUUIFFJZRA2SSRRL7AKWZMQNQRYOZCVFFDLTFCFP236A, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.570579823 (415h3m6.398288214s) [noise=true , address=145.79.227.48:4301 , piece_id=LBIYTJNOX5J4M3HDE7FGF5CSARBITW2AK3P2DSYFLFKUOIHZUKOQ, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.571632758 (415h3m6.399341149s) [noise=true , address=145.79.227.43:4157 , piece_id=3W3LVTU37DNHWS52IGIIH7SXWXVNPGEY27RQNT272L3Z64GNR4KA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.573168383 (415h3m6.400876774s) [noise=true , address=145.79.227.38:4979 , piece_id=HLMQC5OEB4OL4OZSRMCEVAWR5QM2T4MQUBPTHRVQLEAHPBWPQF6Q, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.579995420 (415h3m6.407703816s) [noise=true , address=45.91.116.14:4190 , piece_id=V4T6Z7K3C75LQEXNQIYZGYWW2CSQ7UPSET46QGDRT52RFGAROTTA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.592093172 (415h3m6.419801563s) [noise=true , address=145.79.227.37:4565 , piece_id=554RHTYPG27UNCFPSRMOB245AX4QXL2QFIYECNTVWNNJMAL4BTYA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:09.946500434 (415h3m10.774208816s) [noise=true , address=145.79.227.40:4051 , piece_id=55RRAD7DQX6SR7RBLYAV7MD7XRWPNIWHTVVBQZ2ZL3OE2L7LRRIQ, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:35.398235543 (415h3m36.225943943s) [noise=true , address=145.79.227.39:4013 , piece_id=JJVURGSLKY2UQ4FKNCLATA3ZBD5SDL32XJAG42KH2U6OYK3KTS6Q, seg_pos={0 0}, upload=431917]: Done uploading piece. err: ecclient: upload failed (node:121scLvKEW9zw2aEz22zpremcjiop67eQ8v2uqpmaD5aZzCAP7b, address:145.79.227.39:4013): protocol: write tcp 159.223.143.39:60842->145.79.227.39:4013: use of closed network connection; write tcp 159.223.143.39:60842->145.79.227.39:4013: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:60842->145.79.227.39:4013: use of closed network connection Dec 22 22:31:35.398562305 (415h3m36.226270699s) [noise=true , address=145.79.227.32:4835 , piece_id=BRE3JH3XKPXB6FN7WB3W6ED7MOFES6HYLR34YGR6L5AX7RPIQ7IA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: ecclient: upload failed (node:16vVfeoqCkbMYCRjMMWaKBnCvQBBJPMgwQtWTq9HKQrD9SS17y, address:145.79.227.32:4835): protocol: write tcp 159.223.143.39:45098->145.79.227.32:4835: use of closed network connection; write tcp 159.223.143.39:45098->145.79.227.32:4835: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:45098->145.79.227.32:4835: use of closed network connection Dec 22 22:31:35.399183028 (415h3m36.226891422s) [noise=true , address=145.79.227.50:4373 , piece_id=4U55T5QDYCE6AZ3NCTPAVECPAELPQUKT3HV7GZL7JFRMWRN2AZUQ, seg_pos={0 0}, upload=431917]: Done uploading piece. err: ecclient: upload failed (node:12AQGSkJuNJDxoeqhChUenWxHwF5yK41UESCF63FQz3XvabXXfz, address:145.79.227.50:4373): protocol: read tcp 159.223.143.39:56028->145.79.227.50:4373: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 22 22:31:35.399278021 (415h3m36.226986417s) [noise=true , address=145.79.227.29:4943 , piece_id=G4TMZ6IAIZBDON3UEDPH4LJQFHV3JLALEXSJ3VI4XN6WHIZOB57Q, seg_pos={0 0}, upload=431917]: Done uploading piece. err: ecclient: upload failed (node:12fc5Trj8XxacZhzxdksUVT5CbG6uTmiwdUPjreYi4UMPY1TXWz, address:145.79.227.29:4943): protocol: write tcp 159.223.143.39:50820->145.79.227.29:4943: use of closed network connection; write tcp 159.223.143.39:50820->145.79.227.29:4943: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:50820->145.79.227.29:4943: use of closed network connection Dec 22 22:31:35.399379485 (415h3m36.227087866s) [seg_pos={0 0}, upload=431917]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 22 22:31:35.421122544 (415h3m36.248830935s) [upload=431917]: Done uploading segments... Dec 22 22:31:35.421174369 (415h3m36.248882769s) [upload=431917]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 466.156011ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................................X.......................] duration: 511.695585ms trace: https://jaeger.dev.storj.io/trace/7013ffb1a981d605 FAIL (2025-12-22 22:49:41 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 22 22:31:05.439510308 (415h3m6.2672187s) [upload=431917]: Starting upload Dec 22 22:31:05.439581993 (415h3m6.26729037s) [upload=431917]: Done starting upload Dec 22 22:31:05.439812792 (415h3m6.267521185s) [upload=431917]: Uploading segments... Dec 22 22:31:05.439942524 (415h3m6.267650918s) [upload=431917]: Got next segment. Inline: false Dec 22 22:31:05.440022744 (415h3m6.267731174s) [upload=431917]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 22 22:31:05.440029670 (415h3m6.267738072s) [upload=431917]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 22 22:31:05.467186372 (415h3m6.294894769s) [seg_pos={0 0}, upload=431917]: Begin upload segment... Dec 22 22:31:05.467265149 (415h3m6.294973535s) [seg_pos={0 0}, upload=431917]: Done begin upload segment. Dec 22 22:31:05.467296974 (415h3m6.295005373s) [noise=true , address=145.79.227.51:4409 , piece_id=ATH3F4DFOUUIFFJZRA2SSRRL7AKWZMQNQRYOZCVFFDLTFCFP236A, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.467389853 (415h3m6.295098247s) [noise=true , address=145.79.227.39:4013 , piece_id=JJVURGSLKY2UQ4FKNCLATA3ZBD5SDL32XJAG42KH2U6OYK3KTS6Q, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.467912104 (415h3m6.295620486s) [noise=true , address=145.79.227.36:4519 , piece_id=JUKRRIZWVGKEDD3GPJ7HMY2MEQZNMUCTKTXR6ZLSIERRONTZKHCQ, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.468324863 (415h3m6.296033263s) [noise=true , address=145.79.227.38:4979 , piece_id=HLMQC5OEB4OL4OZSRMCEVAWR5QM2T4MQUBPTHRVQLEAHPBWPQF6Q, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.468334764 (415h3m6.296043147s) [noise=true , address=145.79.227.40:4051 , piece_id=55RRAD7DQX6SR7RBLYAV7MD7XRWPNIWHTVVBQZ2ZL3OE2L7LRRIQ, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.468742525 (415h3m6.296450916s) [noise=true , address=45.91.116.4:4939 , piece_id=QYNKMKDLQJV7B6M2E34DU53Y7YI6VBCUW3C7ZLQOI4XWTBXZHKDQ, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.468915034 (415h3m6.296623414s) [noise=true , address=45.91.116.11:4117 , piece_id=WLMO5E3BIFFL3TOAUSSBGORIZ6DOCWFCMVT4AVFPPGPSKP2OKDAA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.469069393 (415h3m6.296777787s) [noise=true , address=145.79.227.32:4835 , piece_id=BRE3JH3XKPXB6FN7WB3W6ED7MOFES6HYLR34YGR6L5AX7RPIQ7IA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.469285789 (415h3m6.296994183s) [noise=true , address=145.79.227.29:4943 , piece_id=G4TMZ6IAIZBDON3UEDPH4LJQFHV3JLALEXSJ3VI4XN6WHIZOB57Q, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.469411962 (415h3m6.297120355s) [noise=true , address=145.79.227.48:4301 , piece_id=LBIYTJNOX5J4M3HDE7FGF5CSARBITW2AK3P2DSYFLFKUOIHZUKOQ, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.469749379 (415h3m6.297457773s) [noise=true , address=145.79.227.37:4565 , piece_id=554RHTYPG27UNCFPSRMOB245AX4QXL2QFIYECNTVWNNJMAL4BTYA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.470080828 (415h3m6.29778922s) [noise=true , address=145.79.227.43:4157 , piece_id=3W3LVTU37DNHWS52IGIIH7SXWXVNPGEY27RQNT272L3Z64GNR4KA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.470477660 (415h3m6.298186058s) [noise=true , address=45.91.116.14:4190 , piece_id=V4T6Z7K3C75LQEXNQIYZGYWW2CSQ7UPSET46QGDRT52RFGAROTTA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.470567977 (415h3m6.29827637s) [noise=true , address=145.79.227.15:4874 , piece_id=FNCCIMIVJTGZ2X5TOYOZH727JXHYM7HG3HBHBTZDAG67SHZLYYYA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.470980950 (415h3m6.298689349s) [noise=true , address=145.79.227.50:4373 , piece_id=4U55T5QDYCE6AZ3NCTPAVECPAELPQUKT3HV7GZL7JFRMWRN2AZUQ, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.471440098 (415h3m6.299148489s) [noise=true , address=145.79.227.47:4266 , piece_id=3NFYV2PWBNTAOSFHENM4J5PZ7XXE72UKTQVXLTVQRHGK4DUXAKDA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.560766475 (415h3m6.388474871s) [upload=431917]: Next returned nil segment Dec 22 22:31:05.560785161 (415h3m6.388493562s) [upload=431917]: Waiting for error group managing segments... Dec 22 22:31:05.566117878 (415h3m6.39382628s) [noise=true , address=45.91.116.4:4939 , piece_id=QYNKMKDLQJV7B6M2E34DU53Y7YI6VBCUW3C7ZLQOI4XWTBXZHKDQ, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.566723306 (415h3m6.394431687s) [noise=true , address=45.91.116.11:4117 , piece_id=WLMO5E3BIFFL3TOAUSSBGORIZ6DOCWFCMVT4AVFPPGPSKP2OKDAA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.566992667 (415h3m6.394701063s) [noise=true , address=145.79.227.36:4519 , piece_id=JUKRRIZWVGKEDD3GPJ7HMY2MEQZNMUCTKTXR6ZLSIERRONTZKHCQ, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.567464626 (415h3m6.395173019s) [noise=true , address=145.79.227.47:4266 , piece_id=3NFYV2PWBNTAOSFHENM4J5PZ7XXE72UKTQVXLTVQRHGK4DUXAKDA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.567924492 (415h3m6.395632883s) [noise=true , address=145.79.227.15:4874 , piece_id=FNCCIMIVJTGZ2X5TOYOZH727JXHYM7HG3HBHBTZDAG67SHZLYYYA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.568807294 (415h3m6.396515685s) [noise=true , address=145.79.227.51:4409 , piece_id=ATH3F4DFOUUIFFJZRA2SSRRL7AKWZMQNQRYOZCVFFDLTFCFP236A, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.570579823 (415h3m6.398288214s) [noise=true , address=145.79.227.48:4301 , piece_id=LBIYTJNOX5J4M3HDE7FGF5CSARBITW2AK3P2DSYFLFKUOIHZUKOQ, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.571632758 (415h3m6.399341149s) [noise=true , address=145.79.227.43:4157 , piece_id=3W3LVTU37DNHWS52IGIIH7SXWXVNPGEY27RQNT272L3Z64GNR4KA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.573168383 (415h3m6.400876774s) [noise=true , address=145.79.227.38:4979 , piece_id=HLMQC5OEB4OL4OZSRMCEVAWR5QM2T4MQUBPTHRVQLEAHPBWPQF6Q, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.579995420 (415h3m6.407703816s) [noise=true , address=45.91.116.14:4190 , piece_id=V4T6Z7K3C75LQEXNQIYZGYWW2CSQ7UPSET46QGDRT52RFGAROTTA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.592093172 (415h3m6.419801563s) [noise=true , address=145.79.227.37:4565 , piece_id=554RHTYPG27UNCFPSRMOB245AX4QXL2QFIYECNTVWNNJMAL4BTYA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:09.946500434 (415h3m10.774208816s) [noise=true , address=145.79.227.40:4051 , piece_id=55RRAD7DQX6SR7RBLYAV7MD7XRWPNIWHTVVBQZ2ZL3OE2L7LRRIQ, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:35.398235543 (415h3m36.225943943s) [noise=true , address=145.79.227.39:4013 , piece_id=JJVURGSLKY2UQ4FKNCLATA3ZBD5SDL32XJAG42KH2U6OYK3KTS6Q, seg_pos={0 0}, upload=431917]: Done uploading piece. err: ecclient: upload failed (node:121scLvKEW9zw2aEz22zpremcjiop67eQ8v2uqpmaD5aZzCAP7b, address:145.79.227.39:4013): protocol: write tcp 159.223.143.39:60842->145.79.227.39:4013: use of closed network connection; write tcp 159.223.143.39:60842->145.79.227.39:4013: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:60842->145.79.227.39:4013: use of closed network connection Dec 22 22:31:35.398562305 (415h3m36.226270699s) [noise=true , address=145.79.227.32:4835 , piece_id=BRE3JH3XKPXB6FN7WB3W6ED7MOFES6HYLR34YGR6L5AX7RPIQ7IA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: ecclient: upload failed (node:16vVfeoqCkbMYCRjMMWaKBnCvQBBJPMgwQtWTq9HKQrD9SS17y, address:145.79.227.32:4835): protocol: write tcp 159.223.143.39:45098->145.79.227.32:4835: use of closed network connection; write tcp 159.223.143.39:45098->145.79.227.32:4835: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:45098->145.79.227.32:4835: use of closed network connection Dec 22 22:31:35.399183028 (415h3m36.226891422s) [noise=true , address=145.79.227.50:4373 , piece_id=4U55T5QDYCE6AZ3NCTPAVECPAELPQUKT3HV7GZL7JFRMWRN2AZUQ, seg_pos={0 0}, upload=431917]: Done uploading piece. err: ecclient: upload failed (node:12AQGSkJuNJDxoeqhChUenWxHwF5yK41UESCF63FQz3XvabXXfz, address:145.79.227.50:4373): protocol: read tcp 159.223.143.39:56028->145.79.227.50:4373: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 22 22:31:35.399278021 (415h3m36.226986417s) [noise=true , address=145.79.227.29:4943 , piece_id=G4TMZ6IAIZBDON3UEDPH4LJQFHV3JLALEXSJ3VI4XN6WHIZOB57Q, seg_pos={0 0}, upload=431917]: Done uploading piece. err: ecclient: upload failed (node:12fc5Trj8XxacZhzxdksUVT5CbG6uTmiwdUPjreYi4UMPY1TXWz, address:145.79.227.29:4943): protocol: write tcp 159.223.143.39:50820->145.79.227.29:4943: use of closed network connection; write tcp 159.223.143.39:50820->145.79.227.29:4943: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:50820->145.79.227.29:4943: use of closed network connection Dec 22 22:31:35.399379485 (415h3m36.227087866s) [seg_pos={0 0}, upload=431917]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 22 22:31:35.421122544 (415h3m36.248830935s) [upload=431917]: Done uploading segments... Dec 22 22:31:35.421174369 (415h3m36.248882769s) [upload=431917]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 511.79667ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................................X......................] duration: 476.416567ms trace: https://jaeger.dev.storj.io/trace/7013ffb1a981d605 FAIL (2025-12-22 22:49:10 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 22 22:31:05.439510308 (415h3m6.2672187s) [upload=431917]: Starting upload Dec 22 22:31:05.439581993 (415h3m6.26729037s) [upload=431917]: Done starting upload Dec 22 22:31:05.439812792 (415h3m6.267521185s) [upload=431917]: Uploading segments... Dec 22 22:31:05.439942524 (415h3m6.267650918s) [upload=431917]: Got next segment. Inline: false Dec 22 22:31:05.440022744 (415h3m6.267731174s) [upload=431917]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 22 22:31:05.440029670 (415h3m6.267738072s) [upload=431917]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 22 22:31:05.467186372 (415h3m6.294894769s) [seg_pos={0 0}, upload=431917]: Begin upload segment... Dec 22 22:31:05.467265149 (415h3m6.294973535s) [seg_pos={0 0}, upload=431917]: Done begin upload segment. Dec 22 22:31:05.467296974 (415h3m6.295005373s) [noise=true , address=145.79.227.51:4409 , piece_id=ATH3F4DFOUUIFFJZRA2SSRRL7AKWZMQNQRYOZCVFFDLTFCFP236A, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.467389853 (415h3m6.295098247s) [noise=true , address=145.79.227.39:4013 , piece_id=JJVURGSLKY2UQ4FKNCLATA3ZBD5SDL32XJAG42KH2U6OYK3KTS6Q, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.467912104 (415h3m6.295620486s) [noise=true , address=145.79.227.36:4519 , piece_id=JUKRRIZWVGKEDD3GPJ7HMY2MEQZNMUCTKTXR6ZLSIERRONTZKHCQ, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.468324863 (415h3m6.296033263s) [noise=true , address=145.79.227.38:4979 , piece_id=HLMQC5OEB4OL4OZSRMCEVAWR5QM2T4MQUBPTHRVQLEAHPBWPQF6Q, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.468334764 (415h3m6.296043147s) [noise=true , address=145.79.227.40:4051 , piece_id=55RRAD7DQX6SR7RBLYAV7MD7XRWPNIWHTVVBQZ2ZL3OE2L7LRRIQ, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.468742525 (415h3m6.296450916s) [noise=true , address=45.91.116.4:4939 , piece_id=QYNKMKDLQJV7B6M2E34DU53Y7YI6VBCUW3C7ZLQOI4XWTBXZHKDQ, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.468915034 (415h3m6.296623414s) [noise=true , address=45.91.116.11:4117 , piece_id=WLMO5E3BIFFL3TOAUSSBGORIZ6DOCWFCMVT4AVFPPGPSKP2OKDAA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.469069393 (415h3m6.296777787s) [noise=true , address=145.79.227.32:4835 , piece_id=BRE3JH3XKPXB6FN7WB3W6ED7MOFES6HYLR34YGR6L5AX7RPIQ7IA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.469285789 (415h3m6.296994183s) [noise=true , address=145.79.227.29:4943 , piece_id=G4TMZ6IAIZBDON3UEDPH4LJQFHV3JLALEXSJ3VI4XN6WHIZOB57Q, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.469411962 (415h3m6.297120355s) [noise=true , address=145.79.227.48:4301 , piece_id=LBIYTJNOX5J4M3HDE7FGF5CSARBITW2AK3P2DSYFLFKUOIHZUKOQ, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.469749379 (415h3m6.297457773s) [noise=true , address=145.79.227.37:4565 , piece_id=554RHTYPG27UNCFPSRMOB245AX4QXL2QFIYECNTVWNNJMAL4BTYA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.470080828 (415h3m6.29778922s) [noise=true , address=145.79.227.43:4157 , piece_id=3W3LVTU37DNHWS52IGIIH7SXWXVNPGEY27RQNT272L3Z64GNR4KA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.470477660 (415h3m6.298186058s) [noise=true , address=45.91.116.14:4190 , piece_id=V4T6Z7K3C75LQEXNQIYZGYWW2CSQ7UPSET46QGDRT52RFGAROTTA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.470567977 (415h3m6.29827637s) [noise=true , address=145.79.227.15:4874 , piece_id=FNCCIMIVJTGZ2X5TOYOZH727JXHYM7HG3HBHBTZDAG67SHZLYYYA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.470980950 (415h3m6.298689349s) [noise=true , address=145.79.227.50:4373 , piece_id=4U55T5QDYCE6AZ3NCTPAVECPAELPQUKT3HV7GZL7JFRMWRN2AZUQ, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.471440098 (415h3m6.299148489s) [noise=true , address=145.79.227.47:4266 , piece_id=3NFYV2PWBNTAOSFHENM4J5PZ7XXE72UKTQVXLTVQRHGK4DUXAKDA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.560766475 (415h3m6.388474871s) [upload=431917]: Next returned nil segment Dec 22 22:31:05.560785161 (415h3m6.388493562s) [upload=431917]: Waiting for error group managing segments... Dec 22 22:31:05.566117878 (415h3m6.39382628s) [noise=true , address=45.91.116.4:4939 , piece_id=QYNKMKDLQJV7B6M2E34DU53Y7YI6VBCUW3C7ZLQOI4XWTBXZHKDQ, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.566723306 (415h3m6.394431687s) [noise=true , address=45.91.116.11:4117 , piece_id=WLMO5E3BIFFL3TOAUSSBGORIZ6DOCWFCMVT4AVFPPGPSKP2OKDAA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.566992667 (415h3m6.394701063s) [noise=true , address=145.79.227.36:4519 , piece_id=JUKRRIZWVGKEDD3GPJ7HMY2MEQZNMUCTKTXR6ZLSIERRONTZKHCQ, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.567464626 (415h3m6.395173019s) [noise=true , address=145.79.227.47:4266 , piece_id=3NFYV2PWBNTAOSFHENM4J5PZ7XXE72UKTQVXLTVQRHGK4DUXAKDA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.567924492 (415h3m6.395632883s) [noise=true , address=145.79.227.15:4874 , piece_id=FNCCIMIVJTGZ2X5TOYOZH727JXHYM7HG3HBHBTZDAG67SHZLYYYA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.568807294 (415h3m6.396515685s) [noise=true , address=145.79.227.51:4409 , piece_id=ATH3F4DFOUUIFFJZRA2SSRRL7AKWZMQNQRYOZCVFFDLTFCFP236A, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.570579823 (415h3m6.398288214s) [noise=true , address=145.79.227.48:4301 , piece_id=LBIYTJNOX5J4M3HDE7FGF5CSARBITW2AK3P2DSYFLFKUOIHZUKOQ, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.571632758 (415h3m6.399341149s) [noise=true , address=145.79.227.43:4157 , piece_id=3W3LVTU37DNHWS52IGIIH7SXWXVNPGEY27RQNT272L3Z64GNR4KA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.573168383 (415h3m6.400876774s) [noise=true , address=145.79.227.38:4979 , piece_id=HLMQC5OEB4OL4OZSRMCEVAWR5QM2T4MQUBPTHRVQLEAHPBWPQF6Q, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.579995420 (415h3m6.407703816s) [noise=true , address=45.91.116.14:4190 , piece_id=V4T6Z7K3C75LQEXNQIYZGYWW2CSQ7UPSET46QGDRT52RFGAROTTA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.592093172 (415h3m6.419801563s) [noise=true , address=145.79.227.37:4565 , piece_id=554RHTYPG27UNCFPSRMOB245AX4QXL2QFIYECNTVWNNJMAL4BTYA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:09.946500434 (415h3m10.774208816s) [noise=true , address=145.79.227.40:4051 , piece_id=55RRAD7DQX6SR7RBLYAV7MD7XRWPNIWHTVVBQZ2ZL3OE2L7LRRIQ, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:35.398235543 (415h3m36.225943943s) [noise=true , address=145.79.227.39:4013 , piece_id=JJVURGSLKY2UQ4FKNCLATA3ZBD5SDL32XJAG42KH2U6OYK3KTS6Q, seg_pos={0 0}, upload=431917]: Done uploading piece. err: ecclient: upload failed (node:121scLvKEW9zw2aEz22zpremcjiop67eQ8v2uqpmaD5aZzCAP7b, address:145.79.227.39:4013): protocol: write tcp 159.223.143.39:60842->145.79.227.39:4013: use of closed network connection; write tcp 159.223.143.39:60842->145.79.227.39:4013: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:60842->145.79.227.39:4013: use of closed network connection Dec 22 22:31:35.398562305 (415h3m36.226270699s) [noise=true , address=145.79.227.32:4835 , piece_id=BRE3JH3XKPXB6FN7WB3W6ED7MOFES6HYLR34YGR6L5AX7RPIQ7IA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: ecclient: upload failed (node:16vVfeoqCkbMYCRjMMWaKBnCvQBBJPMgwQtWTq9HKQrD9SS17y, address:145.79.227.32:4835): protocol: write tcp 159.223.143.39:45098->145.79.227.32:4835: use of closed network connection; write tcp 159.223.143.39:45098->145.79.227.32:4835: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:45098->145.79.227.32:4835: use of closed network connection Dec 22 22:31:35.399183028 (415h3m36.226891422s) [noise=true , address=145.79.227.50:4373 , piece_id=4U55T5QDYCE6AZ3NCTPAVECPAELPQUKT3HV7GZL7JFRMWRN2AZUQ, seg_pos={0 0}, upload=431917]: Done uploading piece. err: ecclient: upload failed (node:12AQGSkJuNJDxoeqhChUenWxHwF5yK41UESCF63FQz3XvabXXfz, address:145.79.227.50:4373): protocol: read tcp 159.223.143.39:56028->145.79.227.50:4373: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 22 22:31:35.399278021 (415h3m36.226986417s) [noise=true , address=145.79.227.29:4943 , piece_id=G4TMZ6IAIZBDON3UEDPH4LJQFHV3JLALEXSJ3VI4XN6WHIZOB57Q, seg_pos={0 0}, upload=431917]: Done uploading piece. err: ecclient: upload failed (node:12fc5Trj8XxacZhzxdksUVT5CbG6uTmiwdUPjreYi4UMPY1TXWz, address:145.79.227.29:4943): protocol: write tcp 159.223.143.39:50820->145.79.227.29:4943: use of closed network connection; write tcp 159.223.143.39:50820->145.79.227.29:4943: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:50820->145.79.227.29:4943: use of closed network connection Dec 22 22:31:35.399379485 (415h3m36.227087866s) [seg_pos={0 0}, upload=431917]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 22 22:31:35.421122544 (415h3m36.248830935s) [upload=431917]: Done uploading segments... Dec 22 22:31:35.421174369 (415h3m36.248882769s) [upload=431917]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 476.441836ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................................X.....................] duration: 440.504099ms trace: https://jaeger.dev.storj.io/trace/7013ffb1a981d605 FAIL (2025-12-22 22:48:18 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 22 22:31:05.439510308 (415h3m6.2672187s) [upload=431917]: Starting upload Dec 22 22:31:05.439581993 (415h3m6.26729037s) [upload=431917]: Done starting upload Dec 22 22:31:05.439812792 (415h3m6.267521185s) [upload=431917]: Uploading segments... Dec 22 22:31:05.439942524 (415h3m6.267650918s) [upload=431917]: Got next segment. Inline: false Dec 22 22:31:05.440022744 (415h3m6.267731174s) [upload=431917]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 22 22:31:05.440029670 (415h3m6.267738072s) [upload=431917]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 22 22:31:05.467186372 (415h3m6.294894769s) [seg_pos={0 0}, upload=431917]: Begin upload segment... Dec 22 22:31:05.467265149 (415h3m6.294973535s) [seg_pos={0 0}, upload=431917]: Done begin upload segment. Dec 22 22:31:05.467296974 (415h3m6.295005373s) [noise=true , address=145.79.227.51:4409 , piece_id=ATH3F4DFOUUIFFJZRA2SSRRL7AKWZMQNQRYOZCVFFDLTFCFP236A, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.467389853 (415h3m6.295098247s) [noise=true , address=145.79.227.39:4013 , piece_id=JJVURGSLKY2UQ4FKNCLATA3ZBD5SDL32XJAG42KH2U6OYK3KTS6Q, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.467912104 (415h3m6.295620486s) [noise=true , address=145.79.227.36:4519 , piece_id=JUKRRIZWVGKEDD3GPJ7HMY2MEQZNMUCTKTXR6ZLSIERRONTZKHCQ, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.468324863 (415h3m6.296033263s) [noise=true , address=145.79.227.38:4979 , piece_id=HLMQC5OEB4OL4OZSRMCEVAWR5QM2T4MQUBPTHRVQLEAHPBWPQF6Q, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.468334764 (415h3m6.296043147s) [noise=true , address=145.79.227.40:4051 , piece_id=55RRAD7DQX6SR7RBLYAV7MD7XRWPNIWHTVVBQZ2ZL3OE2L7LRRIQ, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.468742525 (415h3m6.296450916s) [noise=true , address=45.91.116.4:4939 , piece_id=QYNKMKDLQJV7B6M2E34DU53Y7YI6VBCUW3C7ZLQOI4XWTBXZHKDQ, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.468915034 (415h3m6.296623414s) [noise=true , address=45.91.116.11:4117 , piece_id=WLMO5E3BIFFL3TOAUSSBGORIZ6DOCWFCMVT4AVFPPGPSKP2OKDAA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.469069393 (415h3m6.296777787s) [noise=true , address=145.79.227.32:4835 , piece_id=BRE3JH3XKPXB6FN7WB3W6ED7MOFES6HYLR34YGR6L5AX7RPIQ7IA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.469285789 (415h3m6.296994183s) [noise=true , address=145.79.227.29:4943 , piece_id=G4TMZ6IAIZBDON3UEDPH4LJQFHV3JLALEXSJ3VI4XN6WHIZOB57Q, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.469411962 (415h3m6.297120355s) [noise=true , address=145.79.227.48:4301 , piece_id=LBIYTJNOX5J4M3HDE7FGF5CSARBITW2AK3P2DSYFLFKUOIHZUKOQ, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.469749379 (415h3m6.297457773s) [noise=true , address=145.79.227.37:4565 , piece_id=554RHTYPG27UNCFPSRMOB245AX4QXL2QFIYECNTVWNNJMAL4BTYA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.470080828 (415h3m6.29778922s) [noise=true , address=145.79.227.43:4157 , piece_id=3W3LVTU37DNHWS52IGIIH7SXWXVNPGEY27RQNT272L3Z64GNR4KA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.470477660 (415h3m6.298186058s) [noise=true , address=45.91.116.14:4190 , piece_id=V4T6Z7K3C75LQEXNQIYZGYWW2CSQ7UPSET46QGDRT52RFGAROTTA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.470567977 (415h3m6.29827637s) [noise=true , address=145.79.227.15:4874 , piece_id=FNCCIMIVJTGZ2X5TOYOZH727JXHYM7HG3HBHBTZDAG67SHZLYYYA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.470980950 (415h3m6.298689349s) [noise=true , address=145.79.227.50:4373 , piece_id=4U55T5QDYCE6AZ3NCTPAVECPAELPQUKT3HV7GZL7JFRMWRN2AZUQ, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.471440098 (415h3m6.299148489s) [noise=true , address=145.79.227.47:4266 , piece_id=3NFYV2PWBNTAOSFHENM4J5PZ7XXE72UKTQVXLTVQRHGK4DUXAKDA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.560766475 (415h3m6.388474871s) [upload=431917]: Next returned nil segment Dec 22 22:31:05.560785161 (415h3m6.388493562s) [upload=431917]: Waiting for error group managing segments... Dec 22 22:31:05.566117878 (415h3m6.39382628s) [noise=true , address=45.91.116.4:4939 , piece_id=QYNKMKDLQJV7B6M2E34DU53Y7YI6VBCUW3C7ZLQOI4XWTBXZHKDQ, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.566723306 (415h3m6.394431687s) [noise=true , address=45.91.116.11:4117 , piece_id=WLMO5E3BIFFL3TOAUSSBGORIZ6DOCWFCMVT4AVFPPGPSKP2OKDAA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.566992667 (415h3m6.394701063s) [noise=true , address=145.79.227.36:4519 , piece_id=JUKRRIZWVGKEDD3GPJ7HMY2MEQZNMUCTKTXR6ZLSIERRONTZKHCQ, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.567464626 (415h3m6.395173019s) [noise=true , address=145.79.227.47:4266 , piece_id=3NFYV2PWBNTAOSFHENM4J5PZ7XXE72UKTQVXLTVQRHGK4DUXAKDA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.567924492 (415h3m6.395632883s) [noise=true , address=145.79.227.15:4874 , piece_id=FNCCIMIVJTGZ2X5TOYOZH727JXHYM7HG3HBHBTZDAG67SHZLYYYA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.568807294 (415h3m6.396515685s) [noise=true , address=145.79.227.51:4409 , piece_id=ATH3F4DFOUUIFFJZRA2SSRRL7AKWZMQNQRYOZCVFFDLTFCFP236A, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.570579823 (415h3m6.398288214s) [noise=true , address=145.79.227.48:4301 , piece_id=LBIYTJNOX5J4M3HDE7FGF5CSARBITW2AK3P2DSYFLFKUOIHZUKOQ, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.571632758 (415h3m6.399341149s) [noise=true , address=145.79.227.43:4157 , piece_id=3W3LVTU37DNHWS52IGIIH7SXWXVNPGEY27RQNT272L3Z64GNR4KA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.573168383 (415h3m6.400876774s) [noise=true , address=145.79.227.38:4979 , piece_id=HLMQC5OEB4OL4OZSRMCEVAWR5QM2T4MQUBPTHRVQLEAHPBWPQF6Q, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.579995420 (415h3m6.407703816s) [noise=true , address=45.91.116.14:4190 , piece_id=V4T6Z7K3C75LQEXNQIYZGYWW2CSQ7UPSET46QGDRT52RFGAROTTA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.592093172 (415h3m6.419801563s) [noise=true , address=145.79.227.37:4565 , piece_id=554RHTYPG27UNCFPSRMOB245AX4QXL2QFIYECNTVWNNJMAL4BTYA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:09.946500434 (415h3m10.774208816s) [noise=true , address=145.79.227.40:4051 , piece_id=55RRAD7DQX6SR7RBLYAV7MD7XRWPNIWHTVVBQZ2ZL3OE2L7LRRIQ, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:35.398235543 (415h3m36.225943943s) [noise=true , address=145.79.227.39:4013 , piece_id=JJVURGSLKY2UQ4FKNCLATA3ZBD5SDL32XJAG42KH2U6OYK3KTS6Q, seg_pos={0 0}, upload=431917]: Done uploading piece. err: ecclient: upload failed (node:121scLvKEW9zw2aEz22zpremcjiop67eQ8v2uqpmaD5aZzCAP7b, address:145.79.227.39:4013): protocol: write tcp 159.223.143.39:60842->145.79.227.39:4013: use of closed network connection; write tcp 159.223.143.39:60842->145.79.227.39:4013: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:60842->145.79.227.39:4013: use of closed network connection Dec 22 22:31:35.398562305 (415h3m36.226270699s) [noise=true , address=145.79.227.32:4835 , piece_id=BRE3JH3XKPXB6FN7WB3W6ED7MOFES6HYLR34YGR6L5AX7RPIQ7IA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: ecclient: upload failed (node:16vVfeoqCkbMYCRjMMWaKBnCvQBBJPMgwQtWTq9HKQrD9SS17y, address:145.79.227.32:4835): protocol: write tcp 159.223.143.39:45098->145.79.227.32:4835: use of closed network connection; write tcp 159.223.143.39:45098->145.79.227.32:4835: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:45098->145.79.227.32:4835: use of closed network connection Dec 22 22:31:35.399183028 (415h3m36.226891422s) [noise=true , address=145.79.227.50:4373 , piece_id=4U55T5QDYCE6AZ3NCTPAVECPAELPQUKT3HV7GZL7JFRMWRN2AZUQ, seg_pos={0 0}, upload=431917]: Done uploading piece. err: ecclient: upload failed (node:12AQGSkJuNJDxoeqhChUenWxHwF5yK41UESCF63FQz3XvabXXfz, address:145.79.227.50:4373): protocol: read tcp 159.223.143.39:56028->145.79.227.50:4373: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 22 22:31:35.399278021 (415h3m36.226986417s) [noise=true , address=145.79.227.29:4943 , piece_id=G4TMZ6IAIZBDON3UEDPH4LJQFHV3JLALEXSJ3VI4XN6WHIZOB57Q, seg_pos={0 0}, upload=431917]: Done uploading piece. err: ecclient: upload failed (node:12fc5Trj8XxacZhzxdksUVT5CbG6uTmiwdUPjreYi4UMPY1TXWz, address:145.79.227.29:4943): protocol: write tcp 159.223.143.39:50820->145.79.227.29:4943: use of closed network connection; write tcp 159.223.143.39:50820->145.79.227.29:4943: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:50820->145.79.227.29:4943: use of closed network connection Dec 22 22:31:35.399379485 (415h3m36.227087866s) [seg_pos={0 0}, upload=431917]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 22 22:31:35.421122544 (415h3m36.248830935s) [upload=431917]: Done uploading segments... Dec 22 22:31:35.421174369 (415h3m36.248882769s) [upload=431917]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 440.611292ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................................X....................] duration: 448.465254ms trace: https://jaeger.dev.storj.io/trace/7013ffb1a981d605 FAIL (2025-12-22 22:47:39 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 22 22:31:05.439510308 (415h3m6.2672187s) [upload=431917]: Starting upload Dec 22 22:31:05.439581993 (415h3m6.26729037s) [upload=431917]: Done starting upload Dec 22 22:31:05.439812792 (415h3m6.267521185s) [upload=431917]: Uploading segments... Dec 22 22:31:05.439942524 (415h3m6.267650918s) [upload=431917]: Got next segment. Inline: false Dec 22 22:31:05.440022744 (415h3m6.267731174s) [upload=431917]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 22 22:31:05.440029670 (415h3m6.267738072s) [upload=431917]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 22 22:31:05.467186372 (415h3m6.294894769s) [seg_pos={0 0}, upload=431917]: Begin upload segment... Dec 22 22:31:05.467265149 (415h3m6.294973535s) [seg_pos={0 0}, upload=431917]: Done begin upload segment. Dec 22 22:31:05.467296974 (415h3m6.295005373s) [noise=true , address=145.79.227.51:4409 , piece_id=ATH3F4DFOUUIFFJZRA2SSRRL7AKWZMQNQRYOZCVFFDLTFCFP236A, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.467389853 (415h3m6.295098247s) [noise=true , address=145.79.227.39:4013 , piece_id=JJVURGSLKY2UQ4FKNCLATA3ZBD5SDL32XJAG42KH2U6OYK3KTS6Q, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.467912104 (415h3m6.295620486s) [noise=true , address=145.79.227.36:4519 , piece_id=JUKRRIZWVGKEDD3GPJ7HMY2MEQZNMUCTKTXR6ZLSIERRONTZKHCQ, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.468324863 (415h3m6.296033263s) [noise=true , address=145.79.227.38:4979 , piece_id=HLMQC5OEB4OL4OZSRMCEVAWR5QM2T4MQUBPTHRVQLEAHPBWPQF6Q, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.468334764 (415h3m6.296043147s) [noise=true , address=145.79.227.40:4051 , piece_id=55RRAD7DQX6SR7RBLYAV7MD7XRWPNIWHTVVBQZ2ZL3OE2L7LRRIQ, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.468742525 (415h3m6.296450916s) [noise=true , address=45.91.116.4:4939 , piece_id=QYNKMKDLQJV7B6M2E34DU53Y7YI6VBCUW3C7ZLQOI4XWTBXZHKDQ, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.468915034 (415h3m6.296623414s) [noise=true , address=45.91.116.11:4117 , piece_id=WLMO5E3BIFFL3TOAUSSBGORIZ6DOCWFCMVT4AVFPPGPSKP2OKDAA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.469069393 (415h3m6.296777787s) [noise=true , address=145.79.227.32:4835 , piece_id=BRE3JH3XKPXB6FN7WB3W6ED7MOFES6HYLR34YGR6L5AX7RPIQ7IA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.469285789 (415h3m6.296994183s) [noise=true , address=145.79.227.29:4943 , piece_id=G4TMZ6IAIZBDON3UEDPH4LJQFHV3JLALEXSJ3VI4XN6WHIZOB57Q, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.469411962 (415h3m6.297120355s) [noise=true , address=145.79.227.48:4301 , piece_id=LBIYTJNOX5J4M3HDE7FGF5CSARBITW2AK3P2DSYFLFKUOIHZUKOQ, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.469749379 (415h3m6.297457773s) [noise=true , address=145.79.227.37:4565 , piece_id=554RHTYPG27UNCFPSRMOB245AX4QXL2QFIYECNTVWNNJMAL4BTYA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.470080828 (415h3m6.29778922s) [noise=true , address=145.79.227.43:4157 , piece_id=3W3LVTU37DNHWS52IGIIH7SXWXVNPGEY27RQNT272L3Z64GNR4KA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.470477660 (415h3m6.298186058s) [noise=true , address=45.91.116.14:4190 , piece_id=V4T6Z7K3C75LQEXNQIYZGYWW2CSQ7UPSET46QGDRT52RFGAROTTA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.470567977 (415h3m6.29827637s) [noise=true , address=145.79.227.15:4874 , piece_id=FNCCIMIVJTGZ2X5TOYOZH727JXHYM7HG3HBHBTZDAG67SHZLYYYA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.470980950 (415h3m6.298689349s) [noise=true , address=145.79.227.50:4373 , piece_id=4U55T5QDYCE6AZ3NCTPAVECPAELPQUKT3HV7GZL7JFRMWRN2AZUQ, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.471440098 (415h3m6.299148489s) [noise=true , address=145.79.227.47:4266 , piece_id=3NFYV2PWBNTAOSFHENM4J5PZ7XXE72UKTQVXLTVQRHGK4DUXAKDA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.560766475 (415h3m6.388474871s) [upload=431917]: Next returned nil segment Dec 22 22:31:05.560785161 (415h3m6.388493562s) [upload=431917]: Waiting for error group managing segments... Dec 22 22:31:05.566117878 (415h3m6.39382628s) [noise=true , address=45.91.116.4:4939 , piece_id=QYNKMKDLQJV7B6M2E34DU53Y7YI6VBCUW3C7ZLQOI4XWTBXZHKDQ, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.566723306 (415h3m6.394431687s) [noise=true , address=45.91.116.11:4117 , piece_id=WLMO5E3BIFFL3TOAUSSBGORIZ6DOCWFCMVT4AVFPPGPSKP2OKDAA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.566992667 (415h3m6.394701063s) [noise=true , address=145.79.227.36:4519 , piece_id=JUKRRIZWVGKEDD3GPJ7HMY2MEQZNMUCTKTXR6ZLSIERRONTZKHCQ, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.567464626 (415h3m6.395173019s) [noise=true , address=145.79.227.47:4266 , piece_id=3NFYV2PWBNTAOSFHENM4J5PZ7XXE72UKTQVXLTVQRHGK4DUXAKDA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.567924492 (415h3m6.395632883s) [noise=true , address=145.79.227.15:4874 , piece_id=FNCCIMIVJTGZ2X5TOYOZH727JXHYM7HG3HBHBTZDAG67SHZLYYYA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.568807294 (415h3m6.396515685s) [noise=true , address=145.79.227.51:4409 , piece_id=ATH3F4DFOUUIFFJZRA2SSRRL7AKWZMQNQRYOZCVFFDLTFCFP236A, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.570579823 (415h3m6.398288214s) [noise=true , address=145.79.227.48:4301 , piece_id=LBIYTJNOX5J4M3HDE7FGF5CSARBITW2AK3P2DSYFLFKUOIHZUKOQ, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.571632758 (415h3m6.399341149s) [noise=true , address=145.79.227.43:4157 , piece_id=3W3LVTU37DNHWS52IGIIH7SXWXVNPGEY27RQNT272L3Z64GNR4KA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.573168383 (415h3m6.400876774s) [noise=true , address=145.79.227.38:4979 , piece_id=HLMQC5OEB4OL4OZSRMCEVAWR5QM2T4MQUBPTHRVQLEAHPBWPQF6Q, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.579995420 (415h3m6.407703816s) [noise=true , address=45.91.116.14:4190 , piece_id=V4T6Z7K3C75LQEXNQIYZGYWW2CSQ7UPSET46QGDRT52RFGAROTTA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.592093172 (415h3m6.419801563s) [noise=true , address=145.79.227.37:4565 , piece_id=554RHTYPG27UNCFPSRMOB245AX4QXL2QFIYECNTVWNNJMAL4BTYA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:09.946500434 (415h3m10.774208816s) [noise=true , address=145.79.227.40:4051 , piece_id=55RRAD7DQX6SR7RBLYAV7MD7XRWPNIWHTVVBQZ2ZL3OE2L7LRRIQ, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:35.398235543 (415h3m36.225943943s) [noise=true , address=145.79.227.39:4013 , piece_id=JJVURGSLKY2UQ4FKNCLATA3ZBD5SDL32XJAG42KH2U6OYK3KTS6Q, seg_pos={0 0}, upload=431917]: Done uploading piece. err: ecclient: upload failed (node:121scLvKEW9zw2aEz22zpremcjiop67eQ8v2uqpmaD5aZzCAP7b, address:145.79.227.39:4013): protocol: write tcp 159.223.143.39:60842->145.79.227.39:4013: use of closed network connection; write tcp 159.223.143.39:60842->145.79.227.39:4013: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:60842->145.79.227.39:4013: use of closed network connection Dec 22 22:31:35.398562305 (415h3m36.226270699s) [noise=true , address=145.79.227.32:4835 , piece_id=BRE3JH3XKPXB6FN7WB3W6ED7MOFES6HYLR34YGR6L5AX7RPIQ7IA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: ecclient: upload failed (node:16vVfeoqCkbMYCRjMMWaKBnCvQBBJPMgwQtWTq9HKQrD9SS17y, address:145.79.227.32:4835): protocol: write tcp 159.223.143.39:45098->145.79.227.32:4835: use of closed network connection; write tcp 159.223.143.39:45098->145.79.227.32:4835: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:45098->145.79.227.32:4835: use of closed network connection Dec 22 22:31:35.399183028 (415h3m36.226891422s) [noise=true , address=145.79.227.50:4373 , piece_id=4U55T5QDYCE6AZ3NCTPAVECPAELPQUKT3HV7GZL7JFRMWRN2AZUQ, seg_pos={0 0}, upload=431917]: Done uploading piece. err: ecclient: upload failed (node:12AQGSkJuNJDxoeqhChUenWxHwF5yK41UESCF63FQz3XvabXXfz, address:145.79.227.50:4373): protocol: read tcp 159.223.143.39:56028->145.79.227.50:4373: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 22 22:31:35.399278021 (415h3m36.226986417s) [noise=true , address=145.79.227.29:4943 , piece_id=G4TMZ6IAIZBDON3UEDPH4LJQFHV3JLALEXSJ3VI4XN6WHIZOB57Q, seg_pos={0 0}, upload=431917]: Done uploading piece. err: ecclient: upload failed (node:12fc5Trj8XxacZhzxdksUVT5CbG6uTmiwdUPjreYi4UMPY1TXWz, address:145.79.227.29:4943): protocol: write tcp 159.223.143.39:50820->145.79.227.29:4943: use of closed network connection; write tcp 159.223.143.39:50820->145.79.227.29:4943: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:50820->145.79.227.29:4943: use of closed network connection Dec 22 22:31:35.399379485 (415h3m36.227087866s) [seg_pos={0 0}, upload=431917]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 22 22:31:35.421122544 (415h3m36.248830935s) [upload=431917]: Done uploading segments... Dec 22 22:31:35.421174369 (415h3m36.248882769s) [upload=431917]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 448.496428ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................................X...................] duration: 503.357942ms trace: https://jaeger.dev.storj.io/trace/7013ffb1a981d605 FAIL (2025-12-22 22:46:29 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 22 22:31:05.439510308 (415h3m6.2672187s) [upload=431917]: Starting upload Dec 22 22:31:05.439581993 (415h3m6.26729037s) [upload=431917]: Done starting upload Dec 22 22:31:05.439812792 (415h3m6.267521185s) [upload=431917]: Uploading segments... Dec 22 22:31:05.439942524 (415h3m6.267650918s) [upload=431917]: Got next segment. Inline: false Dec 22 22:31:05.440022744 (415h3m6.267731174s) [upload=431917]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 22 22:31:05.440029670 (415h3m6.267738072s) [upload=431917]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 22 22:31:05.467186372 (415h3m6.294894769s) [seg_pos={0 0}, upload=431917]: Begin upload segment... Dec 22 22:31:05.467265149 (415h3m6.294973535s) [seg_pos={0 0}, upload=431917]: Done begin upload segment. Dec 22 22:31:05.467296974 (415h3m6.295005373s) [noise=true , address=145.79.227.51:4409 , piece_id=ATH3F4DFOUUIFFJZRA2SSRRL7AKWZMQNQRYOZCVFFDLTFCFP236A, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.467389853 (415h3m6.295098247s) [noise=true , address=145.79.227.39:4013 , piece_id=JJVURGSLKY2UQ4FKNCLATA3ZBD5SDL32XJAG42KH2U6OYK3KTS6Q, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.467912104 (415h3m6.295620486s) [noise=true , address=145.79.227.36:4519 , piece_id=JUKRRIZWVGKEDD3GPJ7HMY2MEQZNMUCTKTXR6ZLSIERRONTZKHCQ, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.468324863 (415h3m6.296033263s) [noise=true , address=145.79.227.38:4979 , piece_id=HLMQC5OEB4OL4OZSRMCEVAWR5QM2T4MQUBPTHRVQLEAHPBWPQF6Q, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.468334764 (415h3m6.296043147s) [noise=true , address=145.79.227.40:4051 , piece_id=55RRAD7DQX6SR7RBLYAV7MD7XRWPNIWHTVVBQZ2ZL3OE2L7LRRIQ, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.468742525 (415h3m6.296450916s) [noise=true , address=45.91.116.4:4939 , piece_id=QYNKMKDLQJV7B6M2E34DU53Y7YI6VBCUW3C7ZLQOI4XWTBXZHKDQ, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.468915034 (415h3m6.296623414s) [noise=true , address=45.91.116.11:4117 , piece_id=WLMO5E3BIFFL3TOAUSSBGORIZ6DOCWFCMVT4AVFPPGPSKP2OKDAA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.469069393 (415h3m6.296777787s) [noise=true , address=145.79.227.32:4835 , piece_id=BRE3JH3XKPXB6FN7WB3W6ED7MOFES6HYLR34YGR6L5AX7RPIQ7IA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.469285789 (415h3m6.296994183s) [noise=true , address=145.79.227.29:4943 , piece_id=G4TMZ6IAIZBDON3UEDPH4LJQFHV3JLALEXSJ3VI4XN6WHIZOB57Q, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.469411962 (415h3m6.297120355s) [noise=true , address=145.79.227.48:4301 , piece_id=LBIYTJNOX5J4M3HDE7FGF5CSARBITW2AK3P2DSYFLFKUOIHZUKOQ, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.469749379 (415h3m6.297457773s) [noise=true , address=145.79.227.37:4565 , piece_id=554RHTYPG27UNCFPSRMOB245AX4QXL2QFIYECNTVWNNJMAL4BTYA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.470080828 (415h3m6.29778922s) [noise=true , address=145.79.227.43:4157 , piece_id=3W3LVTU37DNHWS52IGIIH7SXWXVNPGEY27RQNT272L3Z64GNR4KA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.470477660 (415h3m6.298186058s) [noise=true , address=45.91.116.14:4190 , piece_id=V4T6Z7K3C75LQEXNQIYZGYWW2CSQ7UPSET46QGDRT52RFGAROTTA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.470567977 (415h3m6.29827637s) [noise=true , address=145.79.227.15:4874 , piece_id=FNCCIMIVJTGZ2X5TOYOZH727JXHYM7HG3HBHBTZDAG67SHZLYYYA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.470980950 (415h3m6.298689349s) [noise=true , address=145.79.227.50:4373 , piece_id=4U55T5QDYCE6AZ3NCTPAVECPAELPQUKT3HV7GZL7JFRMWRN2AZUQ, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.471440098 (415h3m6.299148489s) [noise=true , address=145.79.227.47:4266 , piece_id=3NFYV2PWBNTAOSFHENM4J5PZ7XXE72UKTQVXLTVQRHGK4DUXAKDA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.560766475 (415h3m6.388474871s) [upload=431917]: Next returned nil segment Dec 22 22:31:05.560785161 (415h3m6.388493562s) [upload=431917]: Waiting for error group managing segments... Dec 22 22:31:05.566117878 (415h3m6.39382628s) [noise=true , address=45.91.116.4:4939 , piece_id=QYNKMKDLQJV7B6M2E34DU53Y7YI6VBCUW3C7ZLQOI4XWTBXZHKDQ, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.566723306 (415h3m6.394431687s) [noise=true , address=45.91.116.11:4117 , piece_id=WLMO5E3BIFFL3TOAUSSBGORIZ6DOCWFCMVT4AVFPPGPSKP2OKDAA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.566992667 (415h3m6.394701063s) [noise=true , address=145.79.227.36:4519 , piece_id=JUKRRIZWVGKEDD3GPJ7HMY2MEQZNMUCTKTXR6ZLSIERRONTZKHCQ, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.567464626 (415h3m6.395173019s) [noise=true , address=145.79.227.47:4266 , piece_id=3NFYV2PWBNTAOSFHENM4J5PZ7XXE72UKTQVXLTVQRHGK4DUXAKDA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.567924492 (415h3m6.395632883s) [noise=true , address=145.79.227.15:4874 , piece_id=FNCCIMIVJTGZ2X5TOYOZH727JXHYM7HG3HBHBTZDAG67SHZLYYYA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.568807294 (415h3m6.396515685s) [noise=true , address=145.79.227.51:4409 , piece_id=ATH3F4DFOUUIFFJZRA2SSRRL7AKWZMQNQRYOZCVFFDLTFCFP236A, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.570579823 (415h3m6.398288214s) [noise=true , address=145.79.227.48:4301 , piece_id=LBIYTJNOX5J4M3HDE7FGF5CSARBITW2AK3P2DSYFLFKUOIHZUKOQ, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.571632758 (415h3m6.399341149s) [noise=true , address=145.79.227.43:4157 , piece_id=3W3LVTU37DNHWS52IGIIH7SXWXVNPGEY27RQNT272L3Z64GNR4KA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.573168383 (415h3m6.400876774s) [noise=true , address=145.79.227.38:4979 , piece_id=HLMQC5OEB4OL4OZSRMCEVAWR5QM2T4MQUBPTHRVQLEAHPBWPQF6Q, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.579995420 (415h3m6.407703816s) [noise=true , address=45.91.116.14:4190 , piece_id=V4T6Z7K3C75LQEXNQIYZGYWW2CSQ7UPSET46QGDRT52RFGAROTTA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.592093172 (415h3m6.419801563s) [noise=true , address=145.79.227.37:4565 , piece_id=554RHTYPG27UNCFPSRMOB245AX4QXL2QFIYECNTVWNNJMAL4BTYA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:09.946500434 (415h3m10.774208816s) [noise=true , address=145.79.227.40:4051 , piece_id=55RRAD7DQX6SR7RBLYAV7MD7XRWPNIWHTVVBQZ2ZL3OE2L7LRRIQ, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:35.398235543 (415h3m36.225943943s) [noise=true , address=145.79.227.39:4013 , piece_id=JJVURGSLKY2UQ4FKNCLATA3ZBD5SDL32XJAG42KH2U6OYK3KTS6Q, seg_pos={0 0}, upload=431917]: Done uploading piece. err: ecclient: upload failed (node:121scLvKEW9zw2aEz22zpremcjiop67eQ8v2uqpmaD5aZzCAP7b, address:145.79.227.39:4013): protocol: write tcp 159.223.143.39:60842->145.79.227.39:4013: use of closed network connection; write tcp 159.223.143.39:60842->145.79.227.39:4013: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:60842->145.79.227.39:4013: use of closed network connection Dec 22 22:31:35.398562305 (415h3m36.226270699s) [noise=true , address=145.79.227.32:4835 , piece_id=BRE3JH3XKPXB6FN7WB3W6ED7MOFES6HYLR34YGR6L5AX7RPIQ7IA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: ecclient: upload failed (node:16vVfeoqCkbMYCRjMMWaKBnCvQBBJPMgwQtWTq9HKQrD9SS17y, address:145.79.227.32:4835): protocol: write tcp 159.223.143.39:45098->145.79.227.32:4835: use of closed network connection; write tcp 159.223.143.39:45098->145.79.227.32:4835: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:45098->145.79.227.32:4835: use of closed network connection Dec 22 22:31:35.399183028 (415h3m36.226891422s) [noise=true , address=145.79.227.50:4373 , piece_id=4U55T5QDYCE6AZ3NCTPAVECPAELPQUKT3HV7GZL7JFRMWRN2AZUQ, seg_pos={0 0}, upload=431917]: Done uploading piece. err: ecclient: upload failed (node:12AQGSkJuNJDxoeqhChUenWxHwF5yK41UESCF63FQz3XvabXXfz, address:145.79.227.50:4373): protocol: read tcp 159.223.143.39:56028->145.79.227.50:4373: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 22 22:31:35.399278021 (415h3m36.226986417s) [noise=true , address=145.79.227.29:4943 , piece_id=G4TMZ6IAIZBDON3UEDPH4LJQFHV3JLALEXSJ3VI4XN6WHIZOB57Q, seg_pos={0 0}, upload=431917]: Done uploading piece. err: ecclient: upload failed (node:12fc5Trj8XxacZhzxdksUVT5CbG6uTmiwdUPjreYi4UMPY1TXWz, address:145.79.227.29:4943): protocol: write tcp 159.223.143.39:50820->145.79.227.29:4943: use of closed network connection; write tcp 159.223.143.39:50820->145.79.227.29:4943: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:50820->145.79.227.29:4943: use of closed network connection Dec 22 22:31:35.399379485 (415h3m36.227087866s) [seg_pos={0 0}, upload=431917]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 22 22:31:35.421122544 (415h3m36.248830935s) [upload=431917]: Done uploading segments... Dec 22 22:31:35.421174369 (415h3m36.248882769s) [upload=431917]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 503.383881ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................X..................] duration: 506.519912ms trace: https://jaeger.dev.storj.io/trace/7013ffb1a981d605 FAIL (2025-12-22 22:45:42 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 22 22:31:05.439510308 (415h3m6.2672187s) [upload=431917]: Starting upload Dec 22 22:31:05.439581993 (415h3m6.26729037s) [upload=431917]: Done starting upload Dec 22 22:31:05.439812792 (415h3m6.267521185s) [upload=431917]: Uploading segments... Dec 22 22:31:05.439942524 (415h3m6.267650918s) [upload=431917]: Got next segment. Inline: false Dec 22 22:31:05.440022744 (415h3m6.267731174s) [upload=431917]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 22 22:31:05.440029670 (415h3m6.267738072s) [upload=431917]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 22 22:31:05.467186372 (415h3m6.294894769s) [seg_pos={0 0}, upload=431917]: Begin upload segment... Dec 22 22:31:05.467265149 (415h3m6.294973535s) [seg_pos={0 0}, upload=431917]: Done begin upload segment. Dec 22 22:31:05.467296974 (415h3m6.295005373s) [noise=true , address=145.79.227.51:4409 , piece_id=ATH3F4DFOUUIFFJZRA2SSRRL7AKWZMQNQRYOZCVFFDLTFCFP236A, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.467389853 (415h3m6.295098247s) [noise=true , address=145.79.227.39:4013 , piece_id=JJVURGSLKY2UQ4FKNCLATA3ZBD5SDL32XJAG42KH2U6OYK3KTS6Q, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.467912104 (415h3m6.295620486s) [noise=true , address=145.79.227.36:4519 , piece_id=JUKRRIZWVGKEDD3GPJ7HMY2MEQZNMUCTKTXR6ZLSIERRONTZKHCQ, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.468324863 (415h3m6.296033263s) [noise=true , address=145.79.227.38:4979 , piece_id=HLMQC5OEB4OL4OZSRMCEVAWR5QM2T4MQUBPTHRVQLEAHPBWPQF6Q, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.468334764 (415h3m6.296043147s) [noise=true , address=145.79.227.40:4051 , piece_id=55RRAD7DQX6SR7RBLYAV7MD7XRWPNIWHTVVBQZ2ZL3OE2L7LRRIQ, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.468742525 (415h3m6.296450916s) [noise=true , address=45.91.116.4:4939 , piece_id=QYNKMKDLQJV7B6M2E34DU53Y7YI6VBCUW3C7ZLQOI4XWTBXZHKDQ, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.468915034 (415h3m6.296623414s) [noise=true , address=45.91.116.11:4117 , piece_id=WLMO5E3BIFFL3TOAUSSBGORIZ6DOCWFCMVT4AVFPPGPSKP2OKDAA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.469069393 (415h3m6.296777787s) [noise=true , address=145.79.227.32:4835 , piece_id=BRE3JH3XKPXB6FN7WB3W6ED7MOFES6HYLR34YGR6L5AX7RPIQ7IA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.469285789 (415h3m6.296994183s) [noise=true , address=145.79.227.29:4943 , piece_id=G4TMZ6IAIZBDON3UEDPH4LJQFHV3JLALEXSJ3VI4XN6WHIZOB57Q, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.469411962 (415h3m6.297120355s) [noise=true , address=145.79.227.48:4301 , piece_id=LBIYTJNOX5J4M3HDE7FGF5CSARBITW2AK3P2DSYFLFKUOIHZUKOQ, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.469749379 (415h3m6.297457773s) [noise=true , address=145.79.227.37:4565 , piece_id=554RHTYPG27UNCFPSRMOB245AX4QXL2QFIYECNTVWNNJMAL4BTYA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.470080828 (415h3m6.29778922s) [noise=true , address=145.79.227.43:4157 , piece_id=3W3LVTU37DNHWS52IGIIH7SXWXVNPGEY27RQNT272L3Z64GNR4KA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.470477660 (415h3m6.298186058s) [noise=true , address=45.91.116.14:4190 , piece_id=V4T6Z7K3C75LQEXNQIYZGYWW2CSQ7UPSET46QGDRT52RFGAROTTA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.470567977 (415h3m6.29827637s) [noise=true , address=145.79.227.15:4874 , piece_id=FNCCIMIVJTGZ2X5TOYOZH727JXHYM7HG3HBHBTZDAG67SHZLYYYA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.470980950 (415h3m6.298689349s) [noise=true , address=145.79.227.50:4373 , piece_id=4U55T5QDYCE6AZ3NCTPAVECPAELPQUKT3HV7GZL7JFRMWRN2AZUQ, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.471440098 (415h3m6.299148489s) [noise=true , address=145.79.227.47:4266 , piece_id=3NFYV2PWBNTAOSFHENM4J5PZ7XXE72UKTQVXLTVQRHGK4DUXAKDA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.560766475 (415h3m6.388474871s) [upload=431917]: Next returned nil segment Dec 22 22:31:05.560785161 (415h3m6.388493562s) [upload=431917]: Waiting for error group managing segments... Dec 22 22:31:05.566117878 (415h3m6.39382628s) [noise=true , address=45.91.116.4:4939 , piece_id=QYNKMKDLQJV7B6M2E34DU53Y7YI6VBCUW3C7ZLQOI4XWTBXZHKDQ, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.566723306 (415h3m6.394431687s) [noise=true , address=45.91.116.11:4117 , piece_id=WLMO5E3BIFFL3TOAUSSBGORIZ6DOCWFCMVT4AVFPPGPSKP2OKDAA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.566992667 (415h3m6.394701063s) [noise=true , address=145.79.227.36:4519 , piece_id=JUKRRIZWVGKEDD3GPJ7HMY2MEQZNMUCTKTXR6ZLSIERRONTZKHCQ, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.567464626 (415h3m6.395173019s) [noise=true , address=145.79.227.47:4266 , piece_id=3NFYV2PWBNTAOSFHENM4J5PZ7XXE72UKTQVXLTVQRHGK4DUXAKDA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.567924492 (415h3m6.395632883s) [noise=true , address=145.79.227.15:4874 , piece_id=FNCCIMIVJTGZ2X5TOYOZH727JXHYM7HG3HBHBTZDAG67SHZLYYYA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.568807294 (415h3m6.396515685s) [noise=true , address=145.79.227.51:4409 , piece_id=ATH3F4DFOUUIFFJZRA2SSRRL7AKWZMQNQRYOZCVFFDLTFCFP236A, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.570579823 (415h3m6.398288214s) [noise=true , address=145.79.227.48:4301 , piece_id=LBIYTJNOX5J4M3HDE7FGF5CSARBITW2AK3P2DSYFLFKUOIHZUKOQ, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.571632758 (415h3m6.399341149s) [noise=true , address=145.79.227.43:4157 , piece_id=3W3LVTU37DNHWS52IGIIH7SXWXVNPGEY27RQNT272L3Z64GNR4KA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.573168383 (415h3m6.400876774s) [noise=true , address=145.79.227.38:4979 , piece_id=HLMQC5OEB4OL4OZSRMCEVAWR5QM2T4MQUBPTHRVQLEAHPBWPQF6Q, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.579995420 (415h3m6.407703816s) [noise=true , address=45.91.116.14:4190 , piece_id=V4T6Z7K3C75LQEXNQIYZGYWW2CSQ7UPSET46QGDRT52RFGAROTTA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.592093172 (415h3m6.419801563s) [noise=true , address=145.79.227.37:4565 , piece_id=554RHTYPG27UNCFPSRMOB245AX4QXL2QFIYECNTVWNNJMAL4BTYA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:09.946500434 (415h3m10.774208816s) [noise=true , address=145.79.227.40:4051 , piece_id=55RRAD7DQX6SR7RBLYAV7MD7XRWPNIWHTVVBQZ2ZL3OE2L7LRRIQ, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:35.398235543 (415h3m36.225943943s) [noise=true , address=145.79.227.39:4013 , piece_id=JJVURGSLKY2UQ4FKNCLATA3ZBD5SDL32XJAG42KH2U6OYK3KTS6Q, seg_pos={0 0}, upload=431917]: Done uploading piece. err: ecclient: upload failed (node:121scLvKEW9zw2aEz22zpremcjiop67eQ8v2uqpmaD5aZzCAP7b, address:145.79.227.39:4013): protocol: write tcp 159.223.143.39:60842->145.79.227.39:4013: use of closed network connection; write tcp 159.223.143.39:60842->145.79.227.39:4013: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:60842->145.79.227.39:4013: use of closed network connection Dec 22 22:31:35.398562305 (415h3m36.226270699s) [noise=true , address=145.79.227.32:4835 , piece_id=BRE3JH3XKPXB6FN7WB3W6ED7MOFES6HYLR34YGR6L5AX7RPIQ7IA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: ecclient: upload failed (node:16vVfeoqCkbMYCRjMMWaKBnCvQBBJPMgwQtWTq9HKQrD9SS17y, address:145.79.227.32:4835): protocol: write tcp 159.223.143.39:45098->145.79.227.32:4835: use of closed network connection; write tcp 159.223.143.39:45098->145.79.227.32:4835: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:45098->145.79.227.32:4835: use of closed network connection Dec 22 22:31:35.399183028 (415h3m36.226891422s) [noise=true , address=145.79.227.50:4373 , piece_id=4U55T5QDYCE6AZ3NCTPAVECPAELPQUKT3HV7GZL7JFRMWRN2AZUQ, seg_pos={0 0}, upload=431917]: Done uploading piece. err: ecclient: upload failed (node:12AQGSkJuNJDxoeqhChUenWxHwF5yK41UESCF63FQz3XvabXXfz, address:145.79.227.50:4373): protocol: read tcp 159.223.143.39:56028->145.79.227.50:4373: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 22 22:31:35.399278021 (415h3m36.226986417s) [noise=true , address=145.79.227.29:4943 , piece_id=G4TMZ6IAIZBDON3UEDPH4LJQFHV3JLALEXSJ3VI4XN6WHIZOB57Q, seg_pos={0 0}, upload=431917]: Done uploading piece. err: ecclient: upload failed (node:12fc5Trj8XxacZhzxdksUVT5CbG6uTmiwdUPjreYi4UMPY1TXWz, address:145.79.227.29:4943): protocol: write tcp 159.223.143.39:50820->145.79.227.29:4943: use of closed network connection; write tcp 159.223.143.39:50820->145.79.227.29:4943: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:50820->145.79.227.29:4943: use of closed network connection Dec 22 22:31:35.399379485 (415h3m36.227087866s) [seg_pos={0 0}, upload=431917]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 22 22:31:35.421122544 (415h3m36.248830935s) [upload=431917]: Done uploading segments... Dec 22 22:31:35.421174369 (415h3m36.248882769s) [upload=431917]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 506.539806ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................X.................] duration: 515.745284ms trace: https://jaeger.dev.storj.io/trace/7013ffb1a981d605 FAIL (2025-12-22 22:45:08 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 22 22:31:05.439510308 (415h3m6.2672187s) [upload=431917]: Starting upload Dec 22 22:31:05.439581993 (415h3m6.26729037s) [upload=431917]: Done starting upload Dec 22 22:31:05.439812792 (415h3m6.267521185s) [upload=431917]: Uploading segments... Dec 22 22:31:05.439942524 (415h3m6.267650918s) [upload=431917]: Got next segment. Inline: false Dec 22 22:31:05.440022744 (415h3m6.267731174s) [upload=431917]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 22 22:31:05.440029670 (415h3m6.267738072s) [upload=431917]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 22 22:31:05.467186372 (415h3m6.294894769s) [seg_pos={0 0}, upload=431917]: Begin upload segment... Dec 22 22:31:05.467265149 (415h3m6.294973535s) [seg_pos={0 0}, upload=431917]: Done begin upload segment. Dec 22 22:31:05.467296974 (415h3m6.295005373s) [noise=true , address=145.79.227.51:4409 , piece_id=ATH3F4DFOUUIFFJZRA2SSRRL7AKWZMQNQRYOZCVFFDLTFCFP236A, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.467389853 (415h3m6.295098247s) [noise=true , address=145.79.227.39:4013 , piece_id=JJVURGSLKY2UQ4FKNCLATA3ZBD5SDL32XJAG42KH2U6OYK3KTS6Q, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.467912104 (415h3m6.295620486s) [noise=true , address=145.79.227.36:4519 , piece_id=JUKRRIZWVGKEDD3GPJ7HMY2MEQZNMUCTKTXR6ZLSIERRONTZKHCQ, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.468324863 (415h3m6.296033263s) [noise=true , address=145.79.227.38:4979 , piece_id=HLMQC5OEB4OL4OZSRMCEVAWR5QM2T4MQUBPTHRVQLEAHPBWPQF6Q, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.468334764 (415h3m6.296043147s) [noise=true , address=145.79.227.40:4051 , piece_id=55RRAD7DQX6SR7RBLYAV7MD7XRWPNIWHTVVBQZ2ZL3OE2L7LRRIQ, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.468742525 (415h3m6.296450916s) [noise=true , address=45.91.116.4:4939 , piece_id=QYNKMKDLQJV7B6M2E34DU53Y7YI6VBCUW3C7ZLQOI4XWTBXZHKDQ, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.468915034 (415h3m6.296623414s) [noise=true , address=45.91.116.11:4117 , piece_id=WLMO5E3BIFFL3TOAUSSBGORIZ6DOCWFCMVT4AVFPPGPSKP2OKDAA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.469069393 (415h3m6.296777787s) [noise=true , address=145.79.227.32:4835 , piece_id=BRE3JH3XKPXB6FN7WB3W6ED7MOFES6HYLR34YGR6L5AX7RPIQ7IA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.469285789 (415h3m6.296994183s) [noise=true , address=145.79.227.29:4943 , piece_id=G4TMZ6IAIZBDON3UEDPH4LJQFHV3JLALEXSJ3VI4XN6WHIZOB57Q, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.469411962 (415h3m6.297120355s) [noise=true , address=145.79.227.48:4301 , piece_id=LBIYTJNOX5J4M3HDE7FGF5CSARBITW2AK3P2DSYFLFKUOIHZUKOQ, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.469749379 (415h3m6.297457773s) [noise=true , address=145.79.227.37:4565 , piece_id=554RHTYPG27UNCFPSRMOB245AX4QXL2QFIYECNTVWNNJMAL4BTYA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.470080828 (415h3m6.29778922s) [noise=true , address=145.79.227.43:4157 , piece_id=3W3LVTU37DNHWS52IGIIH7SXWXVNPGEY27RQNT272L3Z64GNR4KA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.470477660 (415h3m6.298186058s) [noise=true , address=45.91.116.14:4190 , piece_id=V4T6Z7K3C75LQEXNQIYZGYWW2CSQ7UPSET46QGDRT52RFGAROTTA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.470567977 (415h3m6.29827637s) [noise=true , address=145.79.227.15:4874 , piece_id=FNCCIMIVJTGZ2X5TOYOZH727JXHYM7HG3HBHBTZDAG67SHZLYYYA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.470980950 (415h3m6.298689349s) [noise=true , address=145.79.227.50:4373 , piece_id=4U55T5QDYCE6AZ3NCTPAVECPAELPQUKT3HV7GZL7JFRMWRN2AZUQ, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.471440098 (415h3m6.299148489s) [noise=true , address=145.79.227.47:4266 , piece_id=3NFYV2PWBNTAOSFHENM4J5PZ7XXE72UKTQVXLTVQRHGK4DUXAKDA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.560766475 (415h3m6.388474871s) [upload=431917]: Next returned nil segment Dec 22 22:31:05.560785161 (415h3m6.388493562s) [upload=431917]: Waiting for error group managing segments... Dec 22 22:31:05.566117878 (415h3m6.39382628s) [noise=true , address=45.91.116.4:4939 , piece_id=QYNKMKDLQJV7B6M2E34DU53Y7YI6VBCUW3C7ZLQOI4XWTBXZHKDQ, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.566723306 (415h3m6.394431687s) [noise=true , address=45.91.116.11:4117 , piece_id=WLMO5E3BIFFL3TOAUSSBGORIZ6DOCWFCMVT4AVFPPGPSKP2OKDAA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.566992667 (415h3m6.394701063s) [noise=true , address=145.79.227.36:4519 , piece_id=JUKRRIZWVGKEDD3GPJ7HMY2MEQZNMUCTKTXR6ZLSIERRONTZKHCQ, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.567464626 (415h3m6.395173019s) [noise=true , address=145.79.227.47:4266 , piece_id=3NFYV2PWBNTAOSFHENM4J5PZ7XXE72UKTQVXLTVQRHGK4DUXAKDA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.567924492 (415h3m6.395632883s) [noise=true , address=145.79.227.15:4874 , piece_id=FNCCIMIVJTGZ2X5TOYOZH727JXHYM7HG3HBHBTZDAG67SHZLYYYA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.568807294 (415h3m6.396515685s) [noise=true , address=145.79.227.51:4409 , piece_id=ATH3F4DFOUUIFFJZRA2SSRRL7AKWZMQNQRYOZCVFFDLTFCFP236A, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.570579823 (415h3m6.398288214s) [noise=true , address=145.79.227.48:4301 , piece_id=LBIYTJNOX5J4M3HDE7FGF5CSARBITW2AK3P2DSYFLFKUOIHZUKOQ, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.571632758 (415h3m6.399341149s) [noise=true , address=145.79.227.43:4157 , piece_id=3W3LVTU37DNHWS52IGIIH7SXWXVNPGEY27RQNT272L3Z64GNR4KA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.573168383 (415h3m6.400876774s) [noise=true , address=145.79.227.38:4979 , piece_id=HLMQC5OEB4OL4OZSRMCEVAWR5QM2T4MQUBPTHRVQLEAHPBWPQF6Q, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.579995420 (415h3m6.407703816s) [noise=true , address=45.91.116.14:4190 , piece_id=V4T6Z7K3C75LQEXNQIYZGYWW2CSQ7UPSET46QGDRT52RFGAROTTA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.592093172 (415h3m6.419801563s) [noise=true , address=145.79.227.37:4565 , piece_id=554RHTYPG27UNCFPSRMOB245AX4QXL2QFIYECNTVWNNJMAL4BTYA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:09.946500434 (415h3m10.774208816s) [noise=true , address=145.79.227.40:4051 , piece_id=55RRAD7DQX6SR7RBLYAV7MD7XRWPNIWHTVVBQZ2ZL3OE2L7LRRIQ, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:35.398235543 (415h3m36.225943943s) [noise=true , address=145.79.227.39:4013 , piece_id=JJVURGSLKY2UQ4FKNCLATA3ZBD5SDL32XJAG42KH2U6OYK3KTS6Q, seg_pos={0 0}, upload=431917]: Done uploading piece. err: ecclient: upload failed (node:121scLvKEW9zw2aEz22zpremcjiop67eQ8v2uqpmaD5aZzCAP7b, address:145.79.227.39:4013): protocol: write tcp 159.223.143.39:60842->145.79.227.39:4013: use of closed network connection; write tcp 159.223.143.39:60842->145.79.227.39:4013: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:60842->145.79.227.39:4013: use of closed network connection Dec 22 22:31:35.398562305 (415h3m36.226270699s) [noise=true , address=145.79.227.32:4835 , piece_id=BRE3JH3XKPXB6FN7WB3W6ED7MOFES6HYLR34YGR6L5AX7RPIQ7IA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: ecclient: upload failed (node:16vVfeoqCkbMYCRjMMWaKBnCvQBBJPMgwQtWTq9HKQrD9SS17y, address:145.79.227.32:4835): protocol: write tcp 159.223.143.39:45098->145.79.227.32:4835: use of closed network connection; write tcp 159.223.143.39:45098->145.79.227.32:4835: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:45098->145.79.227.32:4835: use of closed network connection Dec 22 22:31:35.399183028 (415h3m36.226891422s) [noise=true , address=145.79.227.50:4373 , piece_id=4U55T5QDYCE6AZ3NCTPAVECPAELPQUKT3HV7GZL7JFRMWRN2AZUQ, seg_pos={0 0}, upload=431917]: Done uploading piece. err: ecclient: upload failed (node:12AQGSkJuNJDxoeqhChUenWxHwF5yK41UESCF63FQz3XvabXXfz, address:145.79.227.50:4373): protocol: read tcp 159.223.143.39:56028->145.79.227.50:4373: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 22 22:31:35.399278021 (415h3m36.226986417s) [noise=true , address=145.79.227.29:4943 , piece_id=G4TMZ6IAIZBDON3UEDPH4LJQFHV3JLALEXSJ3VI4XN6WHIZOB57Q, seg_pos={0 0}, upload=431917]: Done uploading piece. err: ecclient: upload failed (node:12fc5Trj8XxacZhzxdksUVT5CbG6uTmiwdUPjreYi4UMPY1TXWz, address:145.79.227.29:4943): protocol: write tcp 159.223.143.39:50820->145.79.227.29:4943: use of closed network connection; write tcp 159.223.143.39:50820->145.79.227.29:4943: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:50820->145.79.227.29:4943: use of closed network connection Dec 22 22:31:35.399379485 (415h3m36.227087866s) [seg_pos={0 0}, upload=431917]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 22 22:31:35.421122544 (415h3m36.248830935s) [upload=431917]: Done uploading segments... Dec 22 22:31:35.421174369 (415h3m36.248882769s) [upload=431917]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 515.769683ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................X................] duration: 407.281146ms trace: https://jaeger.dev.storj.io/trace/7013ffb1a981d605 FAIL (2025-12-22 22:44:30 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 22 22:31:05.439510308 (415h3m6.2672187s) [upload=431917]: Starting upload Dec 22 22:31:05.439581993 (415h3m6.26729037s) [upload=431917]: Done starting upload Dec 22 22:31:05.439812792 (415h3m6.267521185s) [upload=431917]: Uploading segments... Dec 22 22:31:05.439942524 (415h3m6.267650918s) [upload=431917]: Got next segment. Inline: false Dec 22 22:31:05.440022744 (415h3m6.267731174s) [upload=431917]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 22 22:31:05.440029670 (415h3m6.267738072s) [upload=431917]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 22 22:31:05.467186372 (415h3m6.294894769s) [seg_pos={0 0}, upload=431917]: Begin upload segment... Dec 22 22:31:05.467265149 (415h3m6.294973535s) [seg_pos={0 0}, upload=431917]: Done begin upload segment. Dec 22 22:31:05.467296974 (415h3m6.295005373s) [noise=true , address=145.79.227.51:4409 , piece_id=ATH3F4DFOUUIFFJZRA2SSRRL7AKWZMQNQRYOZCVFFDLTFCFP236A, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.467389853 (415h3m6.295098247s) [noise=true , address=145.79.227.39:4013 , piece_id=JJVURGSLKY2UQ4FKNCLATA3ZBD5SDL32XJAG42KH2U6OYK3KTS6Q, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.467912104 (415h3m6.295620486s) [noise=true , address=145.79.227.36:4519 , piece_id=JUKRRIZWVGKEDD3GPJ7HMY2MEQZNMUCTKTXR6ZLSIERRONTZKHCQ, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.468324863 (415h3m6.296033263s) [noise=true , address=145.79.227.38:4979 , piece_id=HLMQC5OEB4OL4OZSRMCEVAWR5QM2T4MQUBPTHRVQLEAHPBWPQF6Q, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.468334764 (415h3m6.296043147s) [noise=true , address=145.79.227.40:4051 , piece_id=55RRAD7DQX6SR7RBLYAV7MD7XRWPNIWHTVVBQZ2ZL3OE2L7LRRIQ, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.468742525 (415h3m6.296450916s) [noise=true , address=45.91.116.4:4939 , piece_id=QYNKMKDLQJV7B6M2E34DU53Y7YI6VBCUW3C7ZLQOI4XWTBXZHKDQ, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.468915034 (415h3m6.296623414s) [noise=true , address=45.91.116.11:4117 , piece_id=WLMO5E3BIFFL3TOAUSSBGORIZ6DOCWFCMVT4AVFPPGPSKP2OKDAA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.469069393 (415h3m6.296777787s) [noise=true , address=145.79.227.32:4835 , piece_id=BRE3JH3XKPXB6FN7WB3W6ED7MOFES6HYLR34YGR6L5AX7RPIQ7IA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.469285789 (415h3m6.296994183s) [noise=true , address=145.79.227.29:4943 , piece_id=G4TMZ6IAIZBDON3UEDPH4LJQFHV3JLALEXSJ3VI4XN6WHIZOB57Q, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.469411962 (415h3m6.297120355s) [noise=true , address=145.79.227.48:4301 , piece_id=LBIYTJNOX5J4M3HDE7FGF5CSARBITW2AK3P2DSYFLFKUOIHZUKOQ, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.469749379 (415h3m6.297457773s) [noise=true , address=145.79.227.37:4565 , piece_id=554RHTYPG27UNCFPSRMOB245AX4QXL2QFIYECNTVWNNJMAL4BTYA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.470080828 (415h3m6.29778922s) [noise=true , address=145.79.227.43:4157 , piece_id=3W3LVTU37DNHWS52IGIIH7SXWXVNPGEY27RQNT272L3Z64GNR4KA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.470477660 (415h3m6.298186058s) [noise=true , address=45.91.116.14:4190 , piece_id=V4T6Z7K3C75LQEXNQIYZGYWW2CSQ7UPSET46QGDRT52RFGAROTTA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.470567977 (415h3m6.29827637s) [noise=true , address=145.79.227.15:4874 , piece_id=FNCCIMIVJTGZ2X5TOYOZH727JXHYM7HG3HBHBTZDAG67SHZLYYYA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.470980950 (415h3m6.298689349s) [noise=true , address=145.79.227.50:4373 , piece_id=4U55T5QDYCE6AZ3NCTPAVECPAELPQUKT3HV7GZL7JFRMWRN2AZUQ, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.471440098 (415h3m6.299148489s) [noise=true , address=145.79.227.47:4266 , piece_id=3NFYV2PWBNTAOSFHENM4J5PZ7XXE72UKTQVXLTVQRHGK4DUXAKDA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.560766475 (415h3m6.388474871s) [upload=431917]: Next returned nil segment Dec 22 22:31:05.560785161 (415h3m6.388493562s) [upload=431917]: Waiting for error group managing segments... Dec 22 22:31:05.566117878 (415h3m6.39382628s) [noise=true , address=45.91.116.4:4939 , piece_id=QYNKMKDLQJV7B6M2E34DU53Y7YI6VBCUW3C7ZLQOI4XWTBXZHKDQ, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.566723306 (415h3m6.394431687s) [noise=true , address=45.91.116.11:4117 , piece_id=WLMO5E3BIFFL3TOAUSSBGORIZ6DOCWFCMVT4AVFPPGPSKP2OKDAA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.566992667 (415h3m6.394701063s) [noise=true , address=145.79.227.36:4519 , piece_id=JUKRRIZWVGKEDD3GPJ7HMY2MEQZNMUCTKTXR6ZLSIERRONTZKHCQ, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.567464626 (415h3m6.395173019s) [noise=true , address=145.79.227.47:4266 , piece_id=3NFYV2PWBNTAOSFHENM4J5PZ7XXE72UKTQVXLTVQRHGK4DUXAKDA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.567924492 (415h3m6.395632883s) [noise=true , address=145.79.227.15:4874 , piece_id=FNCCIMIVJTGZ2X5TOYOZH727JXHYM7HG3HBHBTZDAG67SHZLYYYA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.568807294 (415h3m6.396515685s) [noise=true , address=145.79.227.51:4409 , piece_id=ATH3F4DFOUUIFFJZRA2SSRRL7AKWZMQNQRYOZCVFFDLTFCFP236A, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.570579823 (415h3m6.398288214s) [noise=true , address=145.79.227.48:4301 , piece_id=LBIYTJNOX5J4M3HDE7FGF5CSARBITW2AK3P2DSYFLFKUOIHZUKOQ, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.571632758 (415h3m6.399341149s) [noise=true , address=145.79.227.43:4157 , piece_id=3W3LVTU37DNHWS52IGIIH7SXWXVNPGEY27RQNT272L3Z64GNR4KA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.573168383 (415h3m6.400876774s) [noise=true , address=145.79.227.38:4979 , piece_id=HLMQC5OEB4OL4OZSRMCEVAWR5QM2T4MQUBPTHRVQLEAHPBWPQF6Q, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.579995420 (415h3m6.407703816s) [noise=true , address=45.91.116.14:4190 , piece_id=V4T6Z7K3C75LQEXNQIYZGYWW2CSQ7UPSET46QGDRT52RFGAROTTA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.592093172 (415h3m6.419801563s) [noise=true , address=145.79.227.37:4565 , piece_id=554RHTYPG27UNCFPSRMOB245AX4QXL2QFIYECNTVWNNJMAL4BTYA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:09.946500434 (415h3m10.774208816s) [noise=true , address=145.79.227.40:4051 , piece_id=55RRAD7DQX6SR7RBLYAV7MD7XRWPNIWHTVVBQZ2ZL3OE2L7LRRIQ, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:35.398235543 (415h3m36.225943943s) [noise=true , address=145.79.227.39:4013 , piece_id=JJVURGSLKY2UQ4FKNCLATA3ZBD5SDL32XJAG42KH2U6OYK3KTS6Q, seg_pos={0 0}, upload=431917]: Done uploading piece. err: ecclient: upload failed (node:121scLvKEW9zw2aEz22zpremcjiop67eQ8v2uqpmaD5aZzCAP7b, address:145.79.227.39:4013): protocol: write tcp 159.223.143.39:60842->145.79.227.39:4013: use of closed network connection; write tcp 159.223.143.39:60842->145.79.227.39:4013: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:60842->145.79.227.39:4013: use of closed network connection Dec 22 22:31:35.398562305 (415h3m36.226270699s) [noise=true , address=145.79.227.32:4835 , piece_id=BRE3JH3XKPXB6FN7WB3W6ED7MOFES6HYLR34YGR6L5AX7RPIQ7IA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: ecclient: upload failed (node:16vVfeoqCkbMYCRjMMWaKBnCvQBBJPMgwQtWTq9HKQrD9SS17y, address:145.79.227.32:4835): protocol: write tcp 159.223.143.39:45098->145.79.227.32:4835: use of closed network connection; write tcp 159.223.143.39:45098->145.79.227.32:4835: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:45098->145.79.227.32:4835: use of closed network connection Dec 22 22:31:35.399183028 (415h3m36.226891422s) [noise=true , address=145.79.227.50:4373 , piece_id=4U55T5QDYCE6AZ3NCTPAVECPAELPQUKT3HV7GZL7JFRMWRN2AZUQ, seg_pos={0 0}, upload=431917]: Done uploading piece. err: ecclient: upload failed (node:12AQGSkJuNJDxoeqhChUenWxHwF5yK41UESCF63FQz3XvabXXfz, address:145.79.227.50:4373): protocol: read tcp 159.223.143.39:56028->145.79.227.50:4373: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 22 22:31:35.399278021 (415h3m36.226986417s) [noise=true , address=145.79.227.29:4943 , piece_id=G4TMZ6IAIZBDON3UEDPH4LJQFHV3JLALEXSJ3VI4XN6WHIZOB57Q, seg_pos={0 0}, upload=431917]: Done uploading piece. err: ecclient: upload failed (node:12fc5Trj8XxacZhzxdksUVT5CbG6uTmiwdUPjreYi4UMPY1TXWz, address:145.79.227.29:4943): protocol: write tcp 159.223.143.39:50820->145.79.227.29:4943: use of closed network connection; write tcp 159.223.143.39:50820->145.79.227.29:4943: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:50820->145.79.227.29:4943: use of closed network connection Dec 22 22:31:35.399379485 (415h3m36.227087866s) [seg_pos={0 0}, upload=431917]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 22 22:31:35.421122544 (415h3m36.248830935s) [upload=431917]: Done uploading segments... Dec 22 22:31:35.421174369 (415h3m36.248882769s) [upload=431917]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 407.329786ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............................................X...............] duration: 524.466856ms trace: https://jaeger.dev.storj.io/trace/7013ffb1a981d605 FAIL (2025-12-22 22:43:50 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 22 22:31:05.439510308 (415h3m6.2672187s) [upload=431917]: Starting upload Dec 22 22:31:05.439581993 (415h3m6.26729037s) [upload=431917]: Done starting upload Dec 22 22:31:05.439812792 (415h3m6.267521185s) [upload=431917]: Uploading segments... Dec 22 22:31:05.439942524 (415h3m6.267650918s) [upload=431917]: Got next segment. Inline: false Dec 22 22:31:05.440022744 (415h3m6.267731174s) [upload=431917]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 22 22:31:05.440029670 (415h3m6.267738072s) [upload=431917]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 22 22:31:05.467186372 (415h3m6.294894769s) [seg_pos={0 0}, upload=431917]: Begin upload segment... Dec 22 22:31:05.467265149 (415h3m6.294973535s) [seg_pos={0 0}, upload=431917]: Done begin upload segment. Dec 22 22:31:05.467296974 (415h3m6.295005373s) [noise=true , address=145.79.227.51:4409 , piece_id=ATH3F4DFOUUIFFJZRA2SSRRL7AKWZMQNQRYOZCVFFDLTFCFP236A, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.467389853 (415h3m6.295098247s) [noise=true , address=145.79.227.39:4013 , piece_id=JJVURGSLKY2UQ4FKNCLATA3ZBD5SDL32XJAG42KH2U6OYK3KTS6Q, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.467912104 (415h3m6.295620486s) [noise=true , address=145.79.227.36:4519 , piece_id=JUKRRIZWVGKEDD3GPJ7HMY2MEQZNMUCTKTXR6ZLSIERRONTZKHCQ, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.468324863 (415h3m6.296033263s) [noise=true , address=145.79.227.38:4979 , piece_id=HLMQC5OEB4OL4OZSRMCEVAWR5QM2T4MQUBPTHRVQLEAHPBWPQF6Q, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.468334764 (415h3m6.296043147s) [noise=true , address=145.79.227.40:4051 , piece_id=55RRAD7DQX6SR7RBLYAV7MD7XRWPNIWHTVVBQZ2ZL3OE2L7LRRIQ, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.468742525 (415h3m6.296450916s) [noise=true , address=45.91.116.4:4939 , piece_id=QYNKMKDLQJV7B6M2E34DU53Y7YI6VBCUW3C7ZLQOI4XWTBXZHKDQ, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.468915034 (415h3m6.296623414s) [noise=true , address=45.91.116.11:4117 , piece_id=WLMO5E3BIFFL3TOAUSSBGORIZ6DOCWFCMVT4AVFPPGPSKP2OKDAA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.469069393 (415h3m6.296777787s) [noise=true , address=145.79.227.32:4835 , piece_id=BRE3JH3XKPXB6FN7WB3W6ED7MOFES6HYLR34YGR6L5AX7RPIQ7IA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.469285789 (415h3m6.296994183s) [noise=true , address=145.79.227.29:4943 , piece_id=G4TMZ6IAIZBDON3UEDPH4LJQFHV3JLALEXSJ3VI4XN6WHIZOB57Q, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.469411962 (415h3m6.297120355s) [noise=true , address=145.79.227.48:4301 , piece_id=LBIYTJNOX5J4M3HDE7FGF5CSARBITW2AK3P2DSYFLFKUOIHZUKOQ, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.469749379 (415h3m6.297457773s) [noise=true , address=145.79.227.37:4565 , piece_id=554RHTYPG27UNCFPSRMOB245AX4QXL2QFIYECNTVWNNJMAL4BTYA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.470080828 (415h3m6.29778922s) [noise=true , address=145.79.227.43:4157 , piece_id=3W3LVTU37DNHWS52IGIIH7SXWXVNPGEY27RQNT272L3Z64GNR4KA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.470477660 (415h3m6.298186058s) [noise=true , address=45.91.116.14:4190 , piece_id=V4T6Z7K3C75LQEXNQIYZGYWW2CSQ7UPSET46QGDRT52RFGAROTTA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.470567977 (415h3m6.29827637s) [noise=true , address=145.79.227.15:4874 , piece_id=FNCCIMIVJTGZ2X5TOYOZH727JXHYM7HG3HBHBTZDAG67SHZLYYYA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.470980950 (415h3m6.298689349s) [noise=true , address=145.79.227.50:4373 , piece_id=4U55T5QDYCE6AZ3NCTPAVECPAELPQUKT3HV7GZL7JFRMWRN2AZUQ, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.471440098 (415h3m6.299148489s) [noise=true , address=145.79.227.47:4266 , piece_id=3NFYV2PWBNTAOSFHENM4J5PZ7XXE72UKTQVXLTVQRHGK4DUXAKDA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.560766475 (415h3m6.388474871s) [upload=431917]: Next returned nil segment Dec 22 22:31:05.560785161 (415h3m6.388493562s) [upload=431917]: Waiting for error group managing segments... Dec 22 22:31:05.566117878 (415h3m6.39382628s) [noise=true , address=45.91.116.4:4939 , piece_id=QYNKMKDLQJV7B6M2E34DU53Y7YI6VBCUW3C7ZLQOI4XWTBXZHKDQ, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.566723306 (415h3m6.394431687s) [noise=true , address=45.91.116.11:4117 , piece_id=WLMO5E3BIFFL3TOAUSSBGORIZ6DOCWFCMVT4AVFPPGPSKP2OKDAA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.566992667 (415h3m6.394701063s) [noise=true , address=145.79.227.36:4519 , piece_id=JUKRRIZWVGKEDD3GPJ7HMY2MEQZNMUCTKTXR6ZLSIERRONTZKHCQ, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.567464626 (415h3m6.395173019s) [noise=true , address=145.79.227.47:4266 , piece_id=3NFYV2PWBNTAOSFHENM4J5PZ7XXE72UKTQVXLTVQRHGK4DUXAKDA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.567924492 (415h3m6.395632883s) [noise=true , address=145.79.227.15:4874 , piece_id=FNCCIMIVJTGZ2X5TOYOZH727JXHYM7HG3HBHBTZDAG67SHZLYYYA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.568807294 (415h3m6.396515685s) [noise=true , address=145.79.227.51:4409 , piece_id=ATH3F4DFOUUIFFJZRA2SSRRL7AKWZMQNQRYOZCVFFDLTFCFP236A, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.570579823 (415h3m6.398288214s) [noise=true , address=145.79.227.48:4301 , piece_id=LBIYTJNOX5J4M3HDE7FGF5CSARBITW2AK3P2DSYFLFKUOIHZUKOQ, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.571632758 (415h3m6.399341149s) [noise=true , address=145.79.227.43:4157 , piece_id=3W3LVTU37DNHWS52IGIIH7SXWXVNPGEY27RQNT272L3Z64GNR4KA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.573168383 (415h3m6.400876774s) [noise=true , address=145.79.227.38:4979 , piece_id=HLMQC5OEB4OL4OZSRMCEVAWR5QM2T4MQUBPTHRVQLEAHPBWPQF6Q, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.579995420 (415h3m6.407703816s) [noise=true , address=45.91.116.14:4190 , piece_id=V4T6Z7K3C75LQEXNQIYZGYWW2CSQ7UPSET46QGDRT52RFGAROTTA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.592093172 (415h3m6.419801563s) [noise=true , address=145.79.227.37:4565 , piece_id=554RHTYPG27UNCFPSRMOB245AX4QXL2QFIYECNTVWNNJMAL4BTYA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:09.946500434 (415h3m10.774208816s) [noise=true , address=145.79.227.40:4051 , piece_id=55RRAD7DQX6SR7RBLYAV7MD7XRWPNIWHTVVBQZ2ZL3OE2L7LRRIQ, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:35.398235543 (415h3m36.225943943s) [noise=true , address=145.79.227.39:4013 , piece_id=JJVURGSLKY2UQ4FKNCLATA3ZBD5SDL32XJAG42KH2U6OYK3KTS6Q, seg_pos={0 0}, upload=431917]: Done uploading piece. err: ecclient: upload failed (node:121scLvKEW9zw2aEz22zpremcjiop67eQ8v2uqpmaD5aZzCAP7b, address:145.79.227.39:4013): protocol: write tcp 159.223.143.39:60842->145.79.227.39:4013: use of closed network connection; write tcp 159.223.143.39:60842->145.79.227.39:4013: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:60842->145.79.227.39:4013: use of closed network connection Dec 22 22:31:35.398562305 (415h3m36.226270699s) [noise=true , address=145.79.227.32:4835 , piece_id=BRE3JH3XKPXB6FN7WB3W6ED7MOFES6HYLR34YGR6L5AX7RPIQ7IA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: ecclient: upload failed (node:16vVfeoqCkbMYCRjMMWaKBnCvQBBJPMgwQtWTq9HKQrD9SS17y, address:145.79.227.32:4835): protocol: write tcp 159.223.143.39:45098->145.79.227.32:4835: use of closed network connection; write tcp 159.223.143.39:45098->145.79.227.32:4835: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:45098->145.79.227.32:4835: use of closed network connection Dec 22 22:31:35.399183028 (415h3m36.226891422s) [noise=true , address=145.79.227.50:4373 , piece_id=4U55T5QDYCE6AZ3NCTPAVECPAELPQUKT3HV7GZL7JFRMWRN2AZUQ, seg_pos={0 0}, upload=431917]: Done uploading piece. err: ecclient: upload failed (node:12AQGSkJuNJDxoeqhChUenWxHwF5yK41UESCF63FQz3XvabXXfz, address:145.79.227.50:4373): protocol: read tcp 159.223.143.39:56028->145.79.227.50:4373: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 22 22:31:35.399278021 (415h3m36.226986417s) [noise=true , address=145.79.227.29:4943 , piece_id=G4TMZ6IAIZBDON3UEDPH4LJQFHV3JLALEXSJ3VI4XN6WHIZOB57Q, seg_pos={0 0}, upload=431917]: Done uploading piece. err: ecclient: upload failed (node:12fc5Trj8XxacZhzxdksUVT5CbG6uTmiwdUPjreYi4UMPY1TXWz, address:145.79.227.29:4943): protocol: write tcp 159.223.143.39:50820->145.79.227.29:4943: use of closed network connection; write tcp 159.223.143.39:50820->145.79.227.29:4943: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:50820->145.79.227.29:4943: use of closed network connection Dec 22 22:31:35.399379485 (415h3m36.227087866s) [seg_pos={0 0}, upload=431917]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 22 22:31:35.421122544 (415h3m36.248830935s) [upload=431917]: Done uploading segments... Dec 22 22:31:35.421174369 (415h3m36.248882769s) [upload=431917]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 524.494076ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............................................X..............] duration: 477.861784ms trace: https://jaeger.dev.storj.io/trace/7013ffb1a981d605 FAIL (2025-12-22 22:42:55 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 22 22:31:05.439510308 (415h3m6.2672187s) [upload=431917]: Starting upload Dec 22 22:31:05.439581993 (415h3m6.26729037s) [upload=431917]: Done starting upload Dec 22 22:31:05.439812792 (415h3m6.267521185s) [upload=431917]: Uploading segments... Dec 22 22:31:05.439942524 (415h3m6.267650918s) [upload=431917]: Got next segment. Inline: false Dec 22 22:31:05.440022744 (415h3m6.267731174s) [upload=431917]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 22 22:31:05.440029670 (415h3m6.267738072s) [upload=431917]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 22 22:31:05.467186372 (415h3m6.294894769s) [seg_pos={0 0}, upload=431917]: Begin upload segment... Dec 22 22:31:05.467265149 (415h3m6.294973535s) [seg_pos={0 0}, upload=431917]: Done begin upload segment. Dec 22 22:31:05.467296974 (415h3m6.295005373s) [noise=true , address=145.79.227.51:4409 , piece_id=ATH3F4DFOUUIFFJZRA2SSRRL7AKWZMQNQRYOZCVFFDLTFCFP236A, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.467389853 (415h3m6.295098247s) [noise=true , address=145.79.227.39:4013 , piece_id=JJVURGSLKY2UQ4FKNCLATA3ZBD5SDL32XJAG42KH2U6OYK3KTS6Q, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.467912104 (415h3m6.295620486s) [noise=true , address=145.79.227.36:4519 , piece_id=JUKRRIZWVGKEDD3GPJ7HMY2MEQZNMUCTKTXR6ZLSIERRONTZKHCQ, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.468324863 (415h3m6.296033263s) [noise=true , address=145.79.227.38:4979 , piece_id=HLMQC5OEB4OL4OZSRMCEVAWR5QM2T4MQUBPTHRVQLEAHPBWPQF6Q, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.468334764 (415h3m6.296043147s) [noise=true , address=145.79.227.40:4051 , piece_id=55RRAD7DQX6SR7RBLYAV7MD7XRWPNIWHTVVBQZ2ZL3OE2L7LRRIQ, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.468742525 (415h3m6.296450916s) [noise=true , address=45.91.116.4:4939 , piece_id=QYNKMKDLQJV7B6M2E34DU53Y7YI6VBCUW3C7ZLQOI4XWTBXZHKDQ, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.468915034 (415h3m6.296623414s) [noise=true , address=45.91.116.11:4117 , piece_id=WLMO5E3BIFFL3TOAUSSBGORIZ6DOCWFCMVT4AVFPPGPSKP2OKDAA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.469069393 (415h3m6.296777787s) [noise=true , address=145.79.227.32:4835 , piece_id=BRE3JH3XKPXB6FN7WB3W6ED7MOFES6HYLR34YGR6L5AX7RPIQ7IA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.469285789 (415h3m6.296994183s) [noise=true , address=145.79.227.29:4943 , piece_id=G4TMZ6IAIZBDON3UEDPH4LJQFHV3JLALEXSJ3VI4XN6WHIZOB57Q, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.469411962 (415h3m6.297120355s) [noise=true , address=145.79.227.48:4301 , piece_id=LBIYTJNOX5J4M3HDE7FGF5CSARBITW2AK3P2DSYFLFKUOIHZUKOQ, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.469749379 (415h3m6.297457773s) [noise=true , address=145.79.227.37:4565 , piece_id=554RHTYPG27UNCFPSRMOB245AX4QXL2QFIYECNTVWNNJMAL4BTYA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.470080828 (415h3m6.29778922s) [noise=true , address=145.79.227.43:4157 , piece_id=3W3LVTU37DNHWS52IGIIH7SXWXVNPGEY27RQNT272L3Z64GNR4KA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.470477660 (415h3m6.298186058s) [noise=true , address=45.91.116.14:4190 , piece_id=V4T6Z7K3C75LQEXNQIYZGYWW2CSQ7UPSET46QGDRT52RFGAROTTA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.470567977 (415h3m6.29827637s) [noise=true , address=145.79.227.15:4874 , piece_id=FNCCIMIVJTGZ2X5TOYOZH727JXHYM7HG3HBHBTZDAG67SHZLYYYA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.470980950 (415h3m6.298689349s) [noise=true , address=145.79.227.50:4373 , piece_id=4U55T5QDYCE6AZ3NCTPAVECPAELPQUKT3HV7GZL7JFRMWRN2AZUQ, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.471440098 (415h3m6.299148489s) [noise=true , address=145.79.227.47:4266 , piece_id=3NFYV2PWBNTAOSFHENM4J5PZ7XXE72UKTQVXLTVQRHGK4DUXAKDA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.560766475 (415h3m6.388474871s) [upload=431917]: Next returned nil segment Dec 22 22:31:05.560785161 (415h3m6.388493562s) [upload=431917]: Waiting for error group managing segments... Dec 22 22:31:05.566117878 (415h3m6.39382628s) [noise=true , address=45.91.116.4:4939 , piece_id=QYNKMKDLQJV7B6M2E34DU53Y7YI6VBCUW3C7ZLQOI4XWTBXZHKDQ, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.566723306 (415h3m6.394431687s) [noise=true , address=45.91.116.11:4117 , piece_id=WLMO5E3BIFFL3TOAUSSBGORIZ6DOCWFCMVT4AVFPPGPSKP2OKDAA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.566992667 (415h3m6.394701063s) [noise=true , address=145.79.227.36:4519 , piece_id=JUKRRIZWVGKEDD3GPJ7HMY2MEQZNMUCTKTXR6ZLSIERRONTZKHCQ, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.567464626 (415h3m6.395173019s) [noise=true , address=145.79.227.47:4266 , piece_id=3NFYV2PWBNTAOSFHENM4J5PZ7XXE72UKTQVXLTVQRHGK4DUXAKDA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.567924492 (415h3m6.395632883s) [noise=true , address=145.79.227.15:4874 , piece_id=FNCCIMIVJTGZ2X5TOYOZH727JXHYM7HG3HBHBTZDAG67SHZLYYYA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.568807294 (415h3m6.396515685s) [noise=true , address=145.79.227.51:4409 , piece_id=ATH3F4DFOUUIFFJZRA2SSRRL7AKWZMQNQRYOZCVFFDLTFCFP236A, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.570579823 (415h3m6.398288214s) [noise=true , address=145.79.227.48:4301 , piece_id=LBIYTJNOX5J4M3HDE7FGF5CSARBITW2AK3P2DSYFLFKUOIHZUKOQ, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.571632758 (415h3m6.399341149s) [noise=true , address=145.79.227.43:4157 , piece_id=3W3LVTU37DNHWS52IGIIH7SXWXVNPGEY27RQNT272L3Z64GNR4KA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.573168383 (415h3m6.400876774s) [noise=true , address=145.79.227.38:4979 , piece_id=HLMQC5OEB4OL4OZSRMCEVAWR5QM2T4MQUBPTHRVQLEAHPBWPQF6Q, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.579995420 (415h3m6.407703816s) [noise=true , address=45.91.116.14:4190 , piece_id=V4T6Z7K3C75LQEXNQIYZGYWW2CSQ7UPSET46QGDRT52RFGAROTTA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.592093172 (415h3m6.419801563s) [noise=true , address=145.79.227.37:4565 , piece_id=554RHTYPG27UNCFPSRMOB245AX4QXL2QFIYECNTVWNNJMAL4BTYA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:09.946500434 (415h3m10.774208816s) [noise=true , address=145.79.227.40:4051 , piece_id=55RRAD7DQX6SR7RBLYAV7MD7XRWPNIWHTVVBQZ2ZL3OE2L7LRRIQ, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:35.398235543 (415h3m36.225943943s) [noise=true , address=145.79.227.39:4013 , piece_id=JJVURGSLKY2UQ4FKNCLATA3ZBD5SDL32XJAG42KH2U6OYK3KTS6Q, seg_pos={0 0}, upload=431917]: Done uploading piece. err: ecclient: upload failed (node:121scLvKEW9zw2aEz22zpremcjiop67eQ8v2uqpmaD5aZzCAP7b, address:145.79.227.39:4013): protocol: write tcp 159.223.143.39:60842->145.79.227.39:4013: use of closed network connection; write tcp 159.223.143.39:60842->145.79.227.39:4013: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:60842->145.79.227.39:4013: use of closed network connection Dec 22 22:31:35.398562305 (415h3m36.226270699s) [noise=true , address=145.79.227.32:4835 , piece_id=BRE3JH3XKPXB6FN7WB3W6ED7MOFES6HYLR34YGR6L5AX7RPIQ7IA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: ecclient: upload failed (node:16vVfeoqCkbMYCRjMMWaKBnCvQBBJPMgwQtWTq9HKQrD9SS17y, address:145.79.227.32:4835): protocol: write tcp 159.223.143.39:45098->145.79.227.32:4835: use of closed network connection; write tcp 159.223.143.39:45098->145.79.227.32:4835: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:45098->145.79.227.32:4835: use of closed network connection Dec 22 22:31:35.399183028 (415h3m36.226891422s) [noise=true , address=145.79.227.50:4373 , piece_id=4U55T5QDYCE6AZ3NCTPAVECPAELPQUKT3HV7GZL7JFRMWRN2AZUQ, seg_pos={0 0}, upload=431917]: Done uploading piece. err: ecclient: upload failed (node:12AQGSkJuNJDxoeqhChUenWxHwF5yK41UESCF63FQz3XvabXXfz, address:145.79.227.50:4373): protocol: read tcp 159.223.143.39:56028->145.79.227.50:4373: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 22 22:31:35.399278021 (415h3m36.226986417s) [noise=true , address=145.79.227.29:4943 , piece_id=G4TMZ6IAIZBDON3UEDPH4LJQFHV3JLALEXSJ3VI4XN6WHIZOB57Q, seg_pos={0 0}, upload=431917]: Done uploading piece. err: ecclient: upload failed (node:12fc5Trj8XxacZhzxdksUVT5CbG6uTmiwdUPjreYi4UMPY1TXWz, address:145.79.227.29:4943): protocol: write tcp 159.223.143.39:50820->145.79.227.29:4943: use of closed network connection; write tcp 159.223.143.39:50820->145.79.227.29:4943: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:50820->145.79.227.29:4943: use of closed network connection Dec 22 22:31:35.399379485 (415h3m36.227087866s) [seg_pos={0 0}, upload=431917]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 22 22:31:35.421122544 (415h3m36.248830935s) [upload=431917]: Done uploading segments... Dec 22 22:31:35.421174369 (415h3m36.248882769s) [upload=431917]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 477.893004ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............................................X.............] duration: 531.377185ms trace: https://jaeger.dev.storj.io/trace/7013ffb1a981d605 FAIL (2025-12-22 22:42:31 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 22 22:31:05.439510308 (415h3m6.2672187s) [upload=431917]: Starting upload Dec 22 22:31:05.439581993 (415h3m6.26729037s) [upload=431917]: Done starting upload Dec 22 22:31:05.439812792 (415h3m6.267521185s) [upload=431917]: Uploading segments... Dec 22 22:31:05.439942524 (415h3m6.267650918s) [upload=431917]: Got next segment. Inline: false Dec 22 22:31:05.440022744 (415h3m6.267731174s) [upload=431917]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 22 22:31:05.440029670 (415h3m6.267738072s) [upload=431917]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 22 22:31:05.467186372 (415h3m6.294894769s) [seg_pos={0 0}, upload=431917]: Begin upload segment... Dec 22 22:31:05.467265149 (415h3m6.294973535s) [seg_pos={0 0}, upload=431917]: Done begin upload segment. Dec 22 22:31:05.467296974 (415h3m6.295005373s) [noise=true , address=145.79.227.51:4409 , piece_id=ATH3F4DFOUUIFFJZRA2SSRRL7AKWZMQNQRYOZCVFFDLTFCFP236A, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.467389853 (415h3m6.295098247s) [noise=true , address=145.79.227.39:4013 , piece_id=JJVURGSLKY2UQ4FKNCLATA3ZBD5SDL32XJAG42KH2U6OYK3KTS6Q, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.467912104 (415h3m6.295620486s) [noise=true , address=145.79.227.36:4519 , piece_id=JUKRRIZWVGKEDD3GPJ7HMY2MEQZNMUCTKTXR6ZLSIERRONTZKHCQ, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.468324863 (415h3m6.296033263s) [noise=true , address=145.79.227.38:4979 , piece_id=HLMQC5OEB4OL4OZSRMCEVAWR5QM2T4MQUBPTHRVQLEAHPBWPQF6Q, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.468334764 (415h3m6.296043147s) [noise=true , address=145.79.227.40:4051 , piece_id=55RRAD7DQX6SR7RBLYAV7MD7XRWPNIWHTVVBQZ2ZL3OE2L7LRRIQ, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.468742525 (415h3m6.296450916s) [noise=true , address=45.91.116.4:4939 , piece_id=QYNKMKDLQJV7B6M2E34DU53Y7YI6VBCUW3C7ZLQOI4XWTBXZHKDQ, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.468915034 (415h3m6.296623414s) [noise=true , address=45.91.116.11:4117 , piece_id=WLMO5E3BIFFL3TOAUSSBGORIZ6DOCWFCMVT4AVFPPGPSKP2OKDAA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.469069393 (415h3m6.296777787s) [noise=true , address=145.79.227.32:4835 , piece_id=BRE3JH3XKPXB6FN7WB3W6ED7MOFES6HYLR34YGR6L5AX7RPIQ7IA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.469285789 (415h3m6.296994183s) [noise=true , address=145.79.227.29:4943 , piece_id=G4TMZ6IAIZBDON3UEDPH4LJQFHV3JLALEXSJ3VI4XN6WHIZOB57Q, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.469411962 (415h3m6.297120355s) [noise=true , address=145.79.227.48:4301 , piece_id=LBIYTJNOX5J4M3HDE7FGF5CSARBITW2AK3P2DSYFLFKUOIHZUKOQ, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.469749379 (415h3m6.297457773s) [noise=true , address=145.79.227.37:4565 , piece_id=554RHTYPG27UNCFPSRMOB245AX4QXL2QFIYECNTVWNNJMAL4BTYA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.470080828 (415h3m6.29778922s) [noise=true , address=145.79.227.43:4157 , piece_id=3W3LVTU37DNHWS52IGIIH7SXWXVNPGEY27RQNT272L3Z64GNR4KA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.470477660 (415h3m6.298186058s) [noise=true , address=45.91.116.14:4190 , piece_id=V4T6Z7K3C75LQEXNQIYZGYWW2CSQ7UPSET46QGDRT52RFGAROTTA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.470567977 (415h3m6.29827637s) [noise=true , address=145.79.227.15:4874 , piece_id=FNCCIMIVJTGZ2X5TOYOZH727JXHYM7HG3HBHBTZDAG67SHZLYYYA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.470980950 (415h3m6.298689349s) [noise=true , address=145.79.227.50:4373 , piece_id=4U55T5QDYCE6AZ3NCTPAVECPAELPQUKT3HV7GZL7JFRMWRN2AZUQ, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.471440098 (415h3m6.299148489s) [noise=true , address=145.79.227.47:4266 , piece_id=3NFYV2PWBNTAOSFHENM4J5PZ7XXE72UKTQVXLTVQRHGK4DUXAKDA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.560766475 (415h3m6.388474871s) [upload=431917]: Next returned nil segment Dec 22 22:31:05.560785161 (415h3m6.388493562s) [upload=431917]: Waiting for error group managing segments... Dec 22 22:31:05.566117878 (415h3m6.39382628s) [noise=true , address=45.91.116.4:4939 , piece_id=QYNKMKDLQJV7B6M2E34DU53Y7YI6VBCUW3C7ZLQOI4XWTBXZHKDQ, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.566723306 (415h3m6.394431687s) [noise=true , address=45.91.116.11:4117 , piece_id=WLMO5E3BIFFL3TOAUSSBGORIZ6DOCWFCMVT4AVFPPGPSKP2OKDAA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.566992667 (415h3m6.394701063s) [noise=true , address=145.79.227.36:4519 , piece_id=JUKRRIZWVGKEDD3GPJ7HMY2MEQZNMUCTKTXR6ZLSIERRONTZKHCQ, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.567464626 (415h3m6.395173019s) [noise=true , address=145.79.227.47:4266 , piece_id=3NFYV2PWBNTAOSFHENM4J5PZ7XXE72UKTQVXLTVQRHGK4DUXAKDA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.567924492 (415h3m6.395632883s) [noise=true , address=145.79.227.15:4874 , piece_id=FNCCIMIVJTGZ2X5TOYOZH727JXHYM7HG3HBHBTZDAG67SHZLYYYA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.568807294 (415h3m6.396515685s) [noise=true , address=145.79.227.51:4409 , piece_id=ATH3F4DFOUUIFFJZRA2SSRRL7AKWZMQNQRYOZCVFFDLTFCFP236A, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.570579823 (415h3m6.398288214s) [noise=true , address=145.79.227.48:4301 , piece_id=LBIYTJNOX5J4M3HDE7FGF5CSARBITW2AK3P2DSYFLFKUOIHZUKOQ, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.571632758 (415h3m6.399341149s) [noise=true , address=145.79.227.43:4157 , piece_id=3W3LVTU37DNHWS52IGIIH7SXWXVNPGEY27RQNT272L3Z64GNR4KA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.573168383 (415h3m6.400876774s) [noise=true , address=145.79.227.38:4979 , piece_id=HLMQC5OEB4OL4OZSRMCEVAWR5QM2T4MQUBPTHRVQLEAHPBWPQF6Q, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.579995420 (415h3m6.407703816s) [noise=true , address=45.91.116.14:4190 , piece_id=V4T6Z7K3C75LQEXNQIYZGYWW2CSQ7UPSET46QGDRT52RFGAROTTA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.592093172 (415h3m6.419801563s) [noise=true , address=145.79.227.37:4565 , piece_id=554RHTYPG27UNCFPSRMOB245AX4QXL2QFIYECNTVWNNJMAL4BTYA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:09.946500434 (415h3m10.774208816s) [noise=true , address=145.79.227.40:4051 , piece_id=55RRAD7DQX6SR7RBLYAV7MD7XRWPNIWHTVVBQZ2ZL3OE2L7LRRIQ, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:35.398235543 (415h3m36.225943943s) [noise=true , address=145.79.227.39:4013 , piece_id=JJVURGSLKY2UQ4FKNCLATA3ZBD5SDL32XJAG42KH2U6OYK3KTS6Q, seg_pos={0 0}, upload=431917]: Done uploading piece. err: ecclient: upload failed (node:121scLvKEW9zw2aEz22zpremcjiop67eQ8v2uqpmaD5aZzCAP7b, address:145.79.227.39:4013): protocol: write tcp 159.223.143.39:60842->145.79.227.39:4013: use of closed network connection; write tcp 159.223.143.39:60842->145.79.227.39:4013: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:60842->145.79.227.39:4013: use of closed network connection Dec 22 22:31:35.398562305 (415h3m36.226270699s) [noise=true , address=145.79.227.32:4835 , piece_id=BRE3JH3XKPXB6FN7WB3W6ED7MOFES6HYLR34YGR6L5AX7RPIQ7IA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: ecclient: upload failed (node:16vVfeoqCkbMYCRjMMWaKBnCvQBBJPMgwQtWTq9HKQrD9SS17y, address:145.79.227.32:4835): protocol: write tcp 159.223.143.39:45098->145.79.227.32:4835: use of closed network connection; write tcp 159.223.143.39:45098->145.79.227.32:4835: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:45098->145.79.227.32:4835: use of closed network connection Dec 22 22:31:35.399183028 (415h3m36.226891422s) [noise=true , address=145.79.227.50:4373 , piece_id=4U55T5QDYCE6AZ3NCTPAVECPAELPQUKT3HV7GZL7JFRMWRN2AZUQ, seg_pos={0 0}, upload=431917]: Done uploading piece. err: ecclient: upload failed (node:12AQGSkJuNJDxoeqhChUenWxHwF5yK41UESCF63FQz3XvabXXfz, address:145.79.227.50:4373): protocol: read tcp 159.223.143.39:56028->145.79.227.50:4373: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 22 22:31:35.399278021 (415h3m36.226986417s) [noise=true , address=145.79.227.29:4943 , piece_id=G4TMZ6IAIZBDON3UEDPH4LJQFHV3JLALEXSJ3VI4XN6WHIZOB57Q, seg_pos={0 0}, upload=431917]: Done uploading piece. err: ecclient: upload failed (node:12fc5Trj8XxacZhzxdksUVT5CbG6uTmiwdUPjreYi4UMPY1TXWz, address:145.79.227.29:4943): protocol: write tcp 159.223.143.39:50820->145.79.227.29:4943: use of closed network connection; write tcp 159.223.143.39:50820->145.79.227.29:4943: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:50820->145.79.227.29:4943: use of closed network connection Dec 22 22:31:35.399379485 (415h3m36.227087866s) [seg_pos={0 0}, upload=431917]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 22 22:31:35.421122544 (415h3m36.248830935s) [upload=431917]: Done uploading segments... Dec 22 22:31:35.421174369 (415h3m36.248882769s) [upload=431917]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 531.406053ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............................................X............] duration: 458.013511ms trace: https://jaeger.dev.storj.io/trace/7013ffb1a981d605 FAIL (2025-12-22 22:41:33 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 22 22:31:05.439510308 (415h3m6.2672187s) [upload=431917]: Starting upload Dec 22 22:31:05.439581993 (415h3m6.26729037s) [upload=431917]: Done starting upload Dec 22 22:31:05.439812792 (415h3m6.267521185s) [upload=431917]: Uploading segments... Dec 22 22:31:05.439942524 (415h3m6.267650918s) [upload=431917]: Got next segment. Inline: false Dec 22 22:31:05.440022744 (415h3m6.267731174s) [upload=431917]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 22 22:31:05.440029670 (415h3m6.267738072s) [upload=431917]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 22 22:31:05.467186372 (415h3m6.294894769s) [seg_pos={0 0}, upload=431917]: Begin upload segment... Dec 22 22:31:05.467265149 (415h3m6.294973535s) [seg_pos={0 0}, upload=431917]: Done begin upload segment. Dec 22 22:31:05.467296974 (415h3m6.295005373s) [noise=true , address=145.79.227.51:4409 , piece_id=ATH3F4DFOUUIFFJZRA2SSRRL7AKWZMQNQRYOZCVFFDLTFCFP236A, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.467389853 (415h3m6.295098247s) [noise=true , address=145.79.227.39:4013 , piece_id=JJVURGSLKY2UQ4FKNCLATA3ZBD5SDL32XJAG42KH2U6OYK3KTS6Q, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.467912104 (415h3m6.295620486s) [noise=true , address=145.79.227.36:4519 , piece_id=JUKRRIZWVGKEDD3GPJ7HMY2MEQZNMUCTKTXR6ZLSIERRONTZKHCQ, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.468324863 (415h3m6.296033263s) [noise=true , address=145.79.227.38:4979 , piece_id=HLMQC5OEB4OL4OZSRMCEVAWR5QM2T4MQUBPTHRVQLEAHPBWPQF6Q, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.468334764 (415h3m6.296043147s) [noise=true , address=145.79.227.40:4051 , piece_id=55RRAD7DQX6SR7RBLYAV7MD7XRWPNIWHTVVBQZ2ZL3OE2L7LRRIQ, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.468742525 (415h3m6.296450916s) [noise=true , address=45.91.116.4:4939 , piece_id=QYNKMKDLQJV7B6M2E34DU53Y7YI6VBCUW3C7ZLQOI4XWTBXZHKDQ, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.468915034 (415h3m6.296623414s) [noise=true , address=45.91.116.11:4117 , piece_id=WLMO5E3BIFFL3TOAUSSBGORIZ6DOCWFCMVT4AVFPPGPSKP2OKDAA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.469069393 (415h3m6.296777787s) [noise=true , address=145.79.227.32:4835 , piece_id=BRE3JH3XKPXB6FN7WB3W6ED7MOFES6HYLR34YGR6L5AX7RPIQ7IA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.469285789 (415h3m6.296994183s) [noise=true , address=145.79.227.29:4943 , piece_id=G4TMZ6IAIZBDON3UEDPH4LJQFHV3JLALEXSJ3VI4XN6WHIZOB57Q, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.469411962 (415h3m6.297120355s) [noise=true , address=145.79.227.48:4301 , piece_id=LBIYTJNOX5J4M3HDE7FGF5CSARBITW2AK3P2DSYFLFKUOIHZUKOQ, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.469749379 (415h3m6.297457773s) [noise=true , address=145.79.227.37:4565 , piece_id=554RHTYPG27UNCFPSRMOB245AX4QXL2QFIYECNTVWNNJMAL4BTYA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.470080828 (415h3m6.29778922s) [noise=true , address=145.79.227.43:4157 , piece_id=3W3LVTU37DNHWS52IGIIH7SXWXVNPGEY27RQNT272L3Z64GNR4KA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.470477660 (415h3m6.298186058s) [noise=true , address=45.91.116.14:4190 , piece_id=V4T6Z7K3C75LQEXNQIYZGYWW2CSQ7UPSET46QGDRT52RFGAROTTA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.470567977 (415h3m6.29827637s) [noise=true , address=145.79.227.15:4874 , piece_id=FNCCIMIVJTGZ2X5TOYOZH727JXHYM7HG3HBHBTZDAG67SHZLYYYA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.470980950 (415h3m6.298689349s) [noise=true , address=145.79.227.50:4373 , piece_id=4U55T5QDYCE6AZ3NCTPAVECPAELPQUKT3HV7GZL7JFRMWRN2AZUQ, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.471440098 (415h3m6.299148489s) [noise=true , address=145.79.227.47:4266 , piece_id=3NFYV2PWBNTAOSFHENM4J5PZ7XXE72UKTQVXLTVQRHGK4DUXAKDA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.560766475 (415h3m6.388474871s) [upload=431917]: Next returned nil segment Dec 22 22:31:05.560785161 (415h3m6.388493562s) [upload=431917]: Waiting for error group managing segments... Dec 22 22:31:05.566117878 (415h3m6.39382628s) [noise=true , address=45.91.116.4:4939 , piece_id=QYNKMKDLQJV7B6M2E34DU53Y7YI6VBCUW3C7ZLQOI4XWTBXZHKDQ, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.566723306 (415h3m6.394431687s) [noise=true , address=45.91.116.11:4117 , piece_id=WLMO5E3BIFFL3TOAUSSBGORIZ6DOCWFCMVT4AVFPPGPSKP2OKDAA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.566992667 (415h3m6.394701063s) [noise=true , address=145.79.227.36:4519 , piece_id=JUKRRIZWVGKEDD3GPJ7HMY2MEQZNMUCTKTXR6ZLSIERRONTZKHCQ, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.567464626 (415h3m6.395173019s) [noise=true , address=145.79.227.47:4266 , piece_id=3NFYV2PWBNTAOSFHENM4J5PZ7XXE72UKTQVXLTVQRHGK4DUXAKDA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.567924492 (415h3m6.395632883s) [noise=true , address=145.79.227.15:4874 , piece_id=FNCCIMIVJTGZ2X5TOYOZH727JXHYM7HG3HBHBTZDAG67SHZLYYYA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.568807294 (415h3m6.396515685s) [noise=true , address=145.79.227.51:4409 , piece_id=ATH3F4DFOUUIFFJZRA2SSRRL7AKWZMQNQRYOZCVFFDLTFCFP236A, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.570579823 (415h3m6.398288214s) [noise=true , address=145.79.227.48:4301 , piece_id=LBIYTJNOX5J4M3HDE7FGF5CSARBITW2AK3P2DSYFLFKUOIHZUKOQ, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.571632758 (415h3m6.399341149s) [noise=true , address=145.79.227.43:4157 , piece_id=3W3LVTU37DNHWS52IGIIH7SXWXVNPGEY27RQNT272L3Z64GNR4KA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.573168383 (415h3m6.400876774s) [noise=true , address=145.79.227.38:4979 , piece_id=HLMQC5OEB4OL4OZSRMCEVAWR5QM2T4MQUBPTHRVQLEAHPBWPQF6Q, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.579995420 (415h3m6.407703816s) [noise=true , address=45.91.116.14:4190 , piece_id=V4T6Z7K3C75LQEXNQIYZGYWW2CSQ7UPSET46QGDRT52RFGAROTTA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.592093172 (415h3m6.419801563s) [noise=true , address=145.79.227.37:4565 , piece_id=554RHTYPG27UNCFPSRMOB245AX4QXL2QFIYECNTVWNNJMAL4BTYA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:09.946500434 (415h3m10.774208816s) [noise=true , address=145.79.227.40:4051 , piece_id=55RRAD7DQX6SR7RBLYAV7MD7XRWPNIWHTVVBQZ2ZL3OE2L7LRRIQ, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:35.398235543 (415h3m36.225943943s) [noise=true , address=145.79.227.39:4013 , piece_id=JJVURGSLKY2UQ4FKNCLATA3ZBD5SDL32XJAG42KH2U6OYK3KTS6Q, seg_pos={0 0}, upload=431917]: Done uploading piece. err: ecclient: upload failed (node:121scLvKEW9zw2aEz22zpremcjiop67eQ8v2uqpmaD5aZzCAP7b, address:145.79.227.39:4013): protocol: write tcp 159.223.143.39:60842->145.79.227.39:4013: use of closed network connection; write tcp 159.223.143.39:60842->145.79.227.39:4013: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:60842->145.79.227.39:4013: use of closed network connection Dec 22 22:31:35.398562305 (415h3m36.226270699s) [noise=true , address=145.79.227.32:4835 , piece_id=BRE3JH3XKPXB6FN7WB3W6ED7MOFES6HYLR34YGR6L5AX7RPIQ7IA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: ecclient: upload failed (node:16vVfeoqCkbMYCRjMMWaKBnCvQBBJPMgwQtWTq9HKQrD9SS17y, address:145.79.227.32:4835): protocol: write tcp 159.223.143.39:45098->145.79.227.32:4835: use of closed network connection; write tcp 159.223.143.39:45098->145.79.227.32:4835: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:45098->145.79.227.32:4835: use of closed network connection Dec 22 22:31:35.399183028 (415h3m36.226891422s) [noise=true , address=145.79.227.50:4373 , piece_id=4U55T5QDYCE6AZ3NCTPAVECPAELPQUKT3HV7GZL7JFRMWRN2AZUQ, seg_pos={0 0}, upload=431917]: Done uploading piece. err: ecclient: upload failed (node:12AQGSkJuNJDxoeqhChUenWxHwF5yK41UESCF63FQz3XvabXXfz, address:145.79.227.50:4373): protocol: read tcp 159.223.143.39:56028->145.79.227.50:4373: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 22 22:31:35.399278021 (415h3m36.226986417s) [noise=true , address=145.79.227.29:4943 , piece_id=G4TMZ6IAIZBDON3UEDPH4LJQFHV3JLALEXSJ3VI4XN6WHIZOB57Q, seg_pos={0 0}, upload=431917]: Done uploading piece. err: ecclient: upload failed (node:12fc5Trj8XxacZhzxdksUVT5CbG6uTmiwdUPjreYi4UMPY1TXWz, address:145.79.227.29:4943): protocol: write tcp 159.223.143.39:50820->145.79.227.29:4943: use of closed network connection; write tcp 159.223.143.39:50820->145.79.227.29:4943: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:50820->145.79.227.29:4943: use of closed network connection Dec 22 22:31:35.399379485 (415h3m36.227087866s) [seg_pos={0 0}, upload=431917]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 22 22:31:35.421122544 (415h3m36.248830935s) [upload=431917]: Done uploading segments... Dec 22 22:31:35.421174369 (415h3m36.248882769s) [upload=431917]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 458.02808ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................................................X...........] duration: 490.327198ms trace: https://jaeger.dev.storj.io/trace/7013ffb1a981d605 FAIL (2025-12-22 22:40:21 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 22 22:31:05.439510308 (415h3m6.2672187s) [upload=431917]: Starting upload Dec 22 22:31:05.439581993 (415h3m6.26729037s) [upload=431917]: Done starting upload Dec 22 22:31:05.439812792 (415h3m6.267521185s) [upload=431917]: Uploading segments... Dec 22 22:31:05.439942524 (415h3m6.267650918s) [upload=431917]: Got next segment. Inline: false Dec 22 22:31:05.440022744 (415h3m6.267731174s) [upload=431917]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 22 22:31:05.440029670 (415h3m6.267738072s) [upload=431917]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 22 22:31:05.467186372 (415h3m6.294894769s) [seg_pos={0 0}, upload=431917]: Begin upload segment... Dec 22 22:31:05.467265149 (415h3m6.294973535s) [seg_pos={0 0}, upload=431917]: Done begin upload segment. Dec 22 22:31:05.467296974 (415h3m6.295005373s) [noise=true , address=145.79.227.51:4409 , piece_id=ATH3F4DFOUUIFFJZRA2SSRRL7AKWZMQNQRYOZCVFFDLTFCFP236A, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.467389853 (415h3m6.295098247s) [noise=true , address=145.79.227.39:4013 , piece_id=JJVURGSLKY2UQ4FKNCLATA3ZBD5SDL32XJAG42KH2U6OYK3KTS6Q, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.467912104 (415h3m6.295620486s) [noise=true , address=145.79.227.36:4519 , piece_id=JUKRRIZWVGKEDD3GPJ7HMY2MEQZNMUCTKTXR6ZLSIERRONTZKHCQ, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.468324863 (415h3m6.296033263s) [noise=true , address=145.79.227.38:4979 , piece_id=HLMQC5OEB4OL4OZSRMCEVAWR5QM2T4MQUBPTHRVQLEAHPBWPQF6Q, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.468334764 (415h3m6.296043147s) [noise=true , address=145.79.227.40:4051 , piece_id=55RRAD7DQX6SR7RBLYAV7MD7XRWPNIWHTVVBQZ2ZL3OE2L7LRRIQ, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.468742525 (415h3m6.296450916s) [noise=true , address=45.91.116.4:4939 , piece_id=QYNKMKDLQJV7B6M2E34DU53Y7YI6VBCUW3C7ZLQOI4XWTBXZHKDQ, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.468915034 (415h3m6.296623414s) [noise=true , address=45.91.116.11:4117 , piece_id=WLMO5E3BIFFL3TOAUSSBGORIZ6DOCWFCMVT4AVFPPGPSKP2OKDAA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.469069393 (415h3m6.296777787s) [noise=true , address=145.79.227.32:4835 , piece_id=BRE3JH3XKPXB6FN7WB3W6ED7MOFES6HYLR34YGR6L5AX7RPIQ7IA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.469285789 (415h3m6.296994183s) [noise=true , address=145.79.227.29:4943 , piece_id=G4TMZ6IAIZBDON3UEDPH4LJQFHV3JLALEXSJ3VI4XN6WHIZOB57Q, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.469411962 (415h3m6.297120355s) [noise=true , address=145.79.227.48:4301 , piece_id=LBIYTJNOX5J4M3HDE7FGF5CSARBITW2AK3P2DSYFLFKUOIHZUKOQ, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.469749379 (415h3m6.297457773s) [noise=true , address=145.79.227.37:4565 , piece_id=554RHTYPG27UNCFPSRMOB245AX4QXL2QFIYECNTVWNNJMAL4BTYA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.470080828 (415h3m6.29778922s) [noise=true , address=145.79.227.43:4157 , piece_id=3W3LVTU37DNHWS52IGIIH7SXWXVNPGEY27RQNT272L3Z64GNR4KA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.470477660 (415h3m6.298186058s) [noise=true , address=45.91.116.14:4190 , piece_id=V4T6Z7K3C75LQEXNQIYZGYWW2CSQ7UPSET46QGDRT52RFGAROTTA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.470567977 (415h3m6.29827637s) [noise=true , address=145.79.227.15:4874 , piece_id=FNCCIMIVJTGZ2X5TOYOZH727JXHYM7HG3HBHBTZDAG67SHZLYYYA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.470980950 (415h3m6.298689349s) [noise=true , address=145.79.227.50:4373 , piece_id=4U55T5QDYCE6AZ3NCTPAVECPAELPQUKT3HV7GZL7JFRMWRN2AZUQ, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.471440098 (415h3m6.299148489s) [noise=true , address=145.79.227.47:4266 , piece_id=3NFYV2PWBNTAOSFHENM4J5PZ7XXE72UKTQVXLTVQRHGK4DUXAKDA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.560766475 (415h3m6.388474871s) [upload=431917]: Next returned nil segment Dec 22 22:31:05.560785161 (415h3m6.388493562s) [upload=431917]: Waiting for error group managing segments... Dec 22 22:31:05.566117878 (415h3m6.39382628s) [noise=true , address=45.91.116.4:4939 , piece_id=QYNKMKDLQJV7B6M2E34DU53Y7YI6VBCUW3C7ZLQOI4XWTBXZHKDQ, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.566723306 (415h3m6.394431687s) [noise=true , address=45.91.116.11:4117 , piece_id=WLMO5E3BIFFL3TOAUSSBGORIZ6DOCWFCMVT4AVFPPGPSKP2OKDAA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.566992667 (415h3m6.394701063s) [noise=true , address=145.79.227.36:4519 , piece_id=JUKRRIZWVGKEDD3GPJ7HMY2MEQZNMUCTKTXR6ZLSIERRONTZKHCQ, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.567464626 (415h3m6.395173019s) [noise=true , address=145.79.227.47:4266 , piece_id=3NFYV2PWBNTAOSFHENM4J5PZ7XXE72UKTQVXLTVQRHGK4DUXAKDA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.567924492 (415h3m6.395632883s) [noise=true , address=145.79.227.15:4874 , piece_id=FNCCIMIVJTGZ2X5TOYOZH727JXHYM7HG3HBHBTZDAG67SHZLYYYA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.568807294 (415h3m6.396515685s) [noise=true , address=145.79.227.51:4409 , piece_id=ATH3F4DFOUUIFFJZRA2SSRRL7AKWZMQNQRYOZCVFFDLTFCFP236A, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.570579823 (415h3m6.398288214s) [noise=true , address=145.79.227.48:4301 , piece_id=LBIYTJNOX5J4M3HDE7FGF5CSARBITW2AK3P2DSYFLFKUOIHZUKOQ, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.571632758 (415h3m6.399341149s) [noise=true , address=145.79.227.43:4157 , piece_id=3W3LVTU37DNHWS52IGIIH7SXWXVNPGEY27RQNT272L3Z64GNR4KA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.573168383 (415h3m6.400876774s) [noise=true , address=145.79.227.38:4979 , piece_id=HLMQC5OEB4OL4OZSRMCEVAWR5QM2T4MQUBPTHRVQLEAHPBWPQF6Q, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.579995420 (415h3m6.407703816s) [noise=true , address=45.91.116.14:4190 , piece_id=V4T6Z7K3C75LQEXNQIYZGYWW2CSQ7UPSET46QGDRT52RFGAROTTA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.592093172 (415h3m6.419801563s) [noise=true , address=145.79.227.37:4565 , piece_id=554RHTYPG27UNCFPSRMOB245AX4QXL2QFIYECNTVWNNJMAL4BTYA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:09.946500434 (415h3m10.774208816s) [noise=true , address=145.79.227.40:4051 , piece_id=55RRAD7DQX6SR7RBLYAV7MD7XRWPNIWHTVVBQZ2ZL3OE2L7LRRIQ, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:35.398235543 (415h3m36.225943943s) [noise=true , address=145.79.227.39:4013 , piece_id=JJVURGSLKY2UQ4FKNCLATA3ZBD5SDL32XJAG42KH2U6OYK3KTS6Q, seg_pos={0 0}, upload=431917]: Done uploading piece. err: ecclient: upload failed (node:121scLvKEW9zw2aEz22zpremcjiop67eQ8v2uqpmaD5aZzCAP7b, address:145.79.227.39:4013): protocol: write tcp 159.223.143.39:60842->145.79.227.39:4013: use of closed network connection; write tcp 159.223.143.39:60842->145.79.227.39:4013: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:60842->145.79.227.39:4013: use of closed network connection Dec 22 22:31:35.398562305 (415h3m36.226270699s) [noise=true , address=145.79.227.32:4835 , piece_id=BRE3JH3XKPXB6FN7WB3W6ED7MOFES6HYLR34YGR6L5AX7RPIQ7IA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: ecclient: upload failed (node:16vVfeoqCkbMYCRjMMWaKBnCvQBBJPMgwQtWTq9HKQrD9SS17y, address:145.79.227.32:4835): protocol: write tcp 159.223.143.39:45098->145.79.227.32:4835: use of closed network connection; write tcp 159.223.143.39:45098->145.79.227.32:4835: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:45098->145.79.227.32:4835: use of closed network connection Dec 22 22:31:35.399183028 (415h3m36.226891422s) [noise=true , address=145.79.227.50:4373 , piece_id=4U55T5QDYCE6AZ3NCTPAVECPAELPQUKT3HV7GZL7JFRMWRN2AZUQ, seg_pos={0 0}, upload=431917]: Done uploading piece. err: ecclient: upload failed (node:12AQGSkJuNJDxoeqhChUenWxHwF5yK41UESCF63FQz3XvabXXfz, address:145.79.227.50:4373): protocol: read tcp 159.223.143.39:56028->145.79.227.50:4373: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 22 22:31:35.399278021 (415h3m36.226986417s) [noise=true , address=145.79.227.29:4943 , piece_id=G4TMZ6IAIZBDON3UEDPH4LJQFHV3JLALEXSJ3VI4XN6WHIZOB57Q, seg_pos={0 0}, upload=431917]: Done uploading piece. err: ecclient: upload failed (node:12fc5Trj8XxacZhzxdksUVT5CbG6uTmiwdUPjreYi4UMPY1TXWz, address:145.79.227.29:4943): protocol: write tcp 159.223.143.39:50820->145.79.227.29:4943: use of closed network connection; write tcp 159.223.143.39:50820->145.79.227.29:4943: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:50820->145.79.227.29:4943: use of closed network connection Dec 22 22:31:35.399379485 (415h3m36.227087866s) [seg_pos={0 0}, upload=431917]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 22 22:31:35.421122544 (415h3m36.248830935s) [upload=431917]: Done uploading segments... Dec 22 22:31:35.421174369 (415h3m36.248882769s) [upload=431917]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 490.349854ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [X................................................X..........] duration: 518.239569ms trace: https://jaeger.dev.storj.io/trace/7013ffb1a981d605 FAIL (2025-12-22 22:39:49 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 22 22:31:05.439510308 (415h3m6.2672187s) [upload=431917]: Starting upload Dec 22 22:31:05.439581993 (415h3m6.26729037s) [upload=431917]: Done starting upload Dec 22 22:31:05.439812792 (415h3m6.267521185s) [upload=431917]: Uploading segments... Dec 22 22:31:05.439942524 (415h3m6.267650918s) [upload=431917]: Got next segment. Inline: false Dec 22 22:31:05.440022744 (415h3m6.267731174s) [upload=431917]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 22 22:31:05.440029670 (415h3m6.267738072s) [upload=431917]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 22 22:31:05.467186372 (415h3m6.294894769s) [seg_pos={0 0}, upload=431917]: Begin upload segment... Dec 22 22:31:05.467265149 (415h3m6.294973535s) [seg_pos={0 0}, upload=431917]: Done begin upload segment. Dec 22 22:31:05.467296974 (415h3m6.295005373s) [noise=true , address=145.79.227.51:4409 , piece_id=ATH3F4DFOUUIFFJZRA2SSRRL7AKWZMQNQRYOZCVFFDLTFCFP236A, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.467389853 (415h3m6.295098247s) [noise=true , address=145.79.227.39:4013 , piece_id=JJVURGSLKY2UQ4FKNCLATA3ZBD5SDL32XJAG42KH2U6OYK3KTS6Q, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.467912104 (415h3m6.295620486s) [noise=true , address=145.79.227.36:4519 , piece_id=JUKRRIZWVGKEDD3GPJ7HMY2MEQZNMUCTKTXR6ZLSIERRONTZKHCQ, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.468324863 (415h3m6.296033263s) [noise=true , address=145.79.227.38:4979 , piece_id=HLMQC5OEB4OL4OZSRMCEVAWR5QM2T4MQUBPTHRVQLEAHPBWPQF6Q, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.468334764 (415h3m6.296043147s) [noise=true , address=145.79.227.40:4051 , piece_id=55RRAD7DQX6SR7RBLYAV7MD7XRWPNIWHTVVBQZ2ZL3OE2L7LRRIQ, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.468742525 (415h3m6.296450916s) [noise=true , address=45.91.116.4:4939 , piece_id=QYNKMKDLQJV7B6M2E34DU53Y7YI6VBCUW3C7ZLQOI4XWTBXZHKDQ, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.468915034 (415h3m6.296623414s) [noise=true , address=45.91.116.11:4117 , piece_id=WLMO5E3BIFFL3TOAUSSBGORIZ6DOCWFCMVT4AVFPPGPSKP2OKDAA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.469069393 (415h3m6.296777787s) [noise=true , address=145.79.227.32:4835 , piece_id=BRE3JH3XKPXB6FN7WB3W6ED7MOFES6HYLR34YGR6L5AX7RPIQ7IA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.469285789 (415h3m6.296994183s) [noise=true , address=145.79.227.29:4943 , piece_id=G4TMZ6IAIZBDON3UEDPH4LJQFHV3JLALEXSJ3VI4XN6WHIZOB57Q, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.469411962 (415h3m6.297120355s) [noise=true , address=145.79.227.48:4301 , piece_id=LBIYTJNOX5J4M3HDE7FGF5CSARBITW2AK3P2DSYFLFKUOIHZUKOQ, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.469749379 (415h3m6.297457773s) [noise=true , address=145.79.227.37:4565 , piece_id=554RHTYPG27UNCFPSRMOB245AX4QXL2QFIYECNTVWNNJMAL4BTYA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.470080828 (415h3m6.29778922s) [noise=true , address=145.79.227.43:4157 , piece_id=3W3LVTU37DNHWS52IGIIH7SXWXVNPGEY27RQNT272L3Z64GNR4KA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.470477660 (415h3m6.298186058s) [noise=true , address=45.91.116.14:4190 , piece_id=V4T6Z7K3C75LQEXNQIYZGYWW2CSQ7UPSET46QGDRT52RFGAROTTA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.470567977 (415h3m6.29827637s) [noise=true , address=145.79.227.15:4874 , piece_id=FNCCIMIVJTGZ2X5TOYOZH727JXHYM7HG3HBHBTZDAG67SHZLYYYA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.470980950 (415h3m6.298689349s) [noise=true , address=145.79.227.50:4373 , piece_id=4U55T5QDYCE6AZ3NCTPAVECPAELPQUKT3HV7GZL7JFRMWRN2AZUQ, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.471440098 (415h3m6.299148489s) [noise=true , address=145.79.227.47:4266 , piece_id=3NFYV2PWBNTAOSFHENM4J5PZ7XXE72UKTQVXLTVQRHGK4DUXAKDA, seg_pos={0 0}, upload=431917]: Uploading piece... Dec 22 22:31:05.560766475 (415h3m6.388474871s) [upload=431917]: Next returned nil segment Dec 22 22:31:05.560785161 (415h3m6.388493562s) [upload=431917]: Waiting for error group managing segments... Dec 22 22:31:05.566117878 (415h3m6.39382628s) [noise=true , address=45.91.116.4:4939 , piece_id=QYNKMKDLQJV7B6M2E34DU53Y7YI6VBCUW3C7ZLQOI4XWTBXZHKDQ, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.566723306 (415h3m6.394431687s) [noise=true , address=45.91.116.11:4117 , piece_id=WLMO5E3BIFFL3TOAUSSBGORIZ6DOCWFCMVT4AVFPPGPSKP2OKDAA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.566992667 (415h3m6.394701063s) [noise=true , address=145.79.227.36:4519 , piece_id=JUKRRIZWVGKEDD3GPJ7HMY2MEQZNMUCTKTXR6ZLSIERRONTZKHCQ, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.567464626 (415h3m6.395173019s) [noise=true , address=145.79.227.47:4266 , piece_id=3NFYV2PWBNTAOSFHENM4J5PZ7XXE72UKTQVXLTVQRHGK4DUXAKDA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.567924492 (415h3m6.395632883s) [noise=true , address=145.79.227.15:4874 , piece_id=FNCCIMIVJTGZ2X5TOYOZH727JXHYM7HG3HBHBTZDAG67SHZLYYYA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.568807294 (415h3m6.396515685s) [noise=true , address=145.79.227.51:4409 , piece_id=ATH3F4DFOUUIFFJZRA2SSRRL7AKWZMQNQRYOZCVFFDLTFCFP236A, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.570579823 (415h3m6.398288214s) [noise=true , address=145.79.227.48:4301 , piece_id=LBIYTJNOX5J4M3HDE7FGF5CSARBITW2AK3P2DSYFLFKUOIHZUKOQ, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.571632758 (415h3m6.399341149s) [noise=true , address=145.79.227.43:4157 , piece_id=3W3LVTU37DNHWS52IGIIH7SXWXVNPGEY27RQNT272L3Z64GNR4KA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.573168383 (415h3m6.400876774s) [noise=true , address=145.79.227.38:4979 , piece_id=HLMQC5OEB4OL4OZSRMCEVAWR5QM2T4MQUBPTHRVQLEAHPBWPQF6Q, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.579995420 (415h3m6.407703816s) [noise=true , address=45.91.116.14:4190 , piece_id=V4T6Z7K3C75LQEXNQIYZGYWW2CSQ7UPSET46QGDRT52RFGAROTTA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:05.592093172 (415h3m6.419801563s) [noise=true , address=145.79.227.37:4565 , piece_id=554RHTYPG27UNCFPSRMOB245AX4QXL2QFIYECNTVWNNJMAL4BTYA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:09.946500434 (415h3m10.774208816s) [noise=true , address=145.79.227.40:4051 , piece_id=55RRAD7DQX6SR7RBLYAV7MD7XRWPNIWHTVVBQZ2ZL3OE2L7LRRIQ, seg_pos={0 0}, upload=431917]: Done uploading piece. err: <nil> Dec 22 22:31:35.398235543 (415h3m36.225943943s) [noise=true , address=145.79.227.39:4013 , piece_id=JJVURGSLKY2UQ4FKNCLATA3ZBD5SDL32XJAG42KH2U6OYK3KTS6Q, seg_pos={0 0}, upload=431917]: Done uploading piece. err: ecclient: upload failed (node:121scLvKEW9zw2aEz22zpremcjiop67eQ8v2uqpmaD5aZzCAP7b, address:145.79.227.39:4013): protocol: write tcp 159.223.143.39:60842->145.79.227.39:4013: use of closed network connection; write tcp 159.223.143.39:60842->145.79.227.39:4013: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:60842->145.79.227.39:4013: use of closed network connection Dec 22 22:31:35.398562305 (415h3m36.226270699s) [noise=true , address=145.79.227.32:4835 , piece_id=BRE3JH3XKPXB6FN7WB3W6ED7MOFES6HYLR34YGR6L5AX7RPIQ7IA, seg_pos={0 0}, upload=431917]: Done uploading piece. err: ecclient: upload failed (node:16vVfeoqCkbMYCRjMMWaKBnCvQBBJPMgwQtWTq9HKQrD9SS17y, address:145.79.227.32:4835): protocol: write tcp 159.223.143.39:45098->145.79.227.32:4835: use of closed network connection; write tcp 159.223.143.39:45098->145.79.227.32:4835: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:45098->145.79.227.32:4835: use of closed network connection Dec 22 22:31:35.399183028 (415h3m36.226891422s) [noise=true , address=145.79.227.50:4373 , piece_id=4U55T5QDYCE6AZ3NCTPAVECPAELPQUKT3HV7GZL7JFRMWRN2AZUQ, seg_pos={0 0}, upload=431917]: Done uploading piece. err: ecclient: upload failed (node:12AQGSkJuNJDxoeqhChUenWxHwF5yK41UESCF63FQz3XvabXXfz, address:145.79.227.50:4373): protocol: read tcp 159.223.143.39:56028->145.79.227.50:4373: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 22 22:31:35.399278021 (415h3m36.226986417s) [noise=true , address=145.79.227.29:4943 , piece_id=G4TMZ6IAIZBDON3UEDPH4LJQFHV3JLALEXSJ3VI4XN6WHIZOB57Q, seg_pos={0 0}, upload=431917]: Done uploading piece. err: ecclient: upload failed (node:12fc5Trj8XxacZhzxdksUVT5CbG6uTmiwdUPjreYi4UMPY1TXWz, address:145.79.227.29:4943): protocol: write tcp 159.223.143.39:50820->145.79.227.29:4943: use of closed network connection; write tcp 159.223.143.39:50820->145.79.227.29:4943: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:50820->145.79.227.29:4943: use of closed network connection Dec 22 22:31:35.399379485 (415h3m36.227087866s) [seg_pos={0 0}, upload=431917]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 22 22:31:35.421122544 (415h3m36.248830935s) [upload=431917]: Done uploading segments... Dec 22 22:31:35.421174369 (415h3m36.248882769s) [upload=431917]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 518.333905ms core uplink us-select-1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)