Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [X...........................................................] duration: 510.360111ms trace: https://jaeger.dev.storj.io/trace/25fec5475908806a FAIL (2025-12-14 11:20:40 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 14 10:39:45.775689210 (211h11m46.603397602s) [upload=219598]: Starting upload Dec 14 10:39:45.775750055 (211h11m46.603458455s) [upload=219598]: Done starting upload Dec 14 10:39:45.775957558 (211h11m46.603665941s) [upload=219598]: Uploading segments... Dec 14 10:39:45.776074427 (211h11m46.603782809s) [upload=219598]: Got next segment. Inline: false Dec 14 10:39:45.776158162 (211h11m46.603866553s) [upload=219598]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 14 10:39:45.776165965 (211h11m46.603874362s) [upload=219598]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 14 10:39:45.801257283 (211h11m46.628965678s) [seg_pos={0 0}, upload=219598]: Begin upload segment... Dec 14 10:39:45.801373021 (211h11m46.629081422s) [noise=true , address=145.79.227.43:4159 , piece_id=ULRDDEYCZXA26UPSFZQRQWFPU7MUCGJJSM7YWIR7HPXDCPBOQXNA, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.801409865 (211h11m46.629118247s) [seg_pos={0 0}, upload=219598]: Done begin upload segment. Dec 14 10:39:45.801481233 (211h11m46.629189635s) [noise=true , address=145.79.227.30:4763 , piece_id=CLMX6DHN5HP3OWMVR4R5MIVRH3GHCELUVTYJHBE7HXZQMHRQESIQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.802050295 (211h11m46.629758695s) [noise=true , address=145.79.227.32:4836 , piece_id=IQRH4CRVBOG72M3BE3G5FTIEMBTLD2KFUEYU6NCTK3N6PUONMY5A, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.802573060 (211h11m46.630281451s) [noise=true , address=77.90.31.14:4661 , piece_id=LLI4CDLWJ32TFXQ2YEU3NRP4QIRSNNUODA5RRPWU5AA45MJEW2LQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.803112997 (211h11m46.630821425s) [noise=true , address=145.79.227.44:4194 , piece_id=ZWOCXZ746BWDZ2LWYHSQHSG2O3WJC4R6FDL5E3JP2SXRCX3AWT2Q, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.803148596 (211h11m46.630856991s) [noise=true , address=145.79.227.46:4603 , piece_id=3X2DPA75MR7RX6LCOO2U3UREG4QSYHRRKRE57JUYUBINXTJN5IEA, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.803615052 (211h11m46.631323455s) [noise=true , address=145.79.227.28:4727 , piece_id=42ILJYV7IJRXSV3E3RWXWNUNYQHHQFNOF2JZGZKDEGPXRXZ3H2YA, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.803722621 (211h11m46.631431034s) [noise=true , address=77.90.31.16:4735 , piece_id=XKJJK6K7XF3TOFWMIU2BUVQTESTNV4FRYINSXJS4MC7KK7ZJU7SQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.804120053 (211h11m46.631828444s) [noise=true , address=145.79.227.7:4293 , piece_id=56YULKIWDM4P4WZM6KGMPUK3B3AK3RRVF4H6RJ7H6IQISPLXGBCQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.804570407 (211h11m46.632278798s) [noise=true , address=145.79.227.47:4267 , piece_id=VY46BNDUYVGBXDRIOEFK4USNOOJH2KXZH5Z2NHCT6SDJC5SFUYMQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.805083798 (211h11m46.632792196s) [noise=true , address=145.79.227.50:4375 , piece_id=RKVBJHHIF3KJLKB6ULTA4E5SHOBPCLPMWJCA6YTKF5XDNVEZMYXQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.805547276 (211h11m46.633255677s) [noise=true , address=77.90.31.16:4734 , piece_id=4JO2RBACNY5AM3KG7XUW7JTRGDCXGQDXSZ3L23ASAEJ4YFCICVAQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.805916642 (211h11m46.633625023s) [noise=true , address=145.79.227.39:4014 , piece_id=WKYZGQX5SP22ML66ZQZIHBS324EONJ7AARTYAXK2SCV577ZCQ5OQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.806440130 (211h11m46.634148525s) [noise=true , address=77.90.31.13:4626 , piece_id=DCKUBRB3DR755GN67RLP6NUMREAWIB3AITGPIMA7MM35BRPZW7IA, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.806976882 (211h11m46.634685284s) [noise=true , address=77.90.31.10:4519 , piece_id=ACPHBL7LF6NCNNH4U5GUXKW7RCL6XIEQVWYDCYUTXEC5RQE7RZBA, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.807729329 (211h11m46.63543772s) [noise=true , address=77.90.31.10:4518 , piece_id=ZE3SFGBCM2QV3CGGS3VEYT7NKDTDE2OJ6T6CVSGGYACCFX5WN5ZQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.841457783 (211h11m46.66916618s) [noise=true , address=77.90.31.13:4626 , piece_id=DCKUBRB3DR755GN67RLP6NUMREAWIB3AITGPIMA7MM35BRPZW7IA, seg_pos={0 0}, upload=219598]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: storage node overloaded, request limit: 2000 Dec 14 10:39:45.850206751 (211h11m46.677915151s) [noise=true , address=77.90.31.16:4734 , piece_id=4JO2RBACNY5AM3KG7XUW7JTRGDCXGQDXSZ3L23ASAEJ4YFCICVAQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: storage node overloaded, request limit: 2000 Dec 14 10:39:45.859644247 (211h11m46.68735265s) [noise=true , address=145.79.227.30:4763 , piece_id=CLMX6DHN5HP3OWMVR4R5MIVRH3GHCELUVTYJHBE7HXZQMHRQESIQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: storage node overloaded, request limit: 2000 Dec 14 10:39:45.874109133 (211h11m46.701817525s) [upload=219598]: Next returned nil segment Dec 14 10:39:45.874123719 (211h11m46.701832113s) [upload=219598]: Waiting for error group managing segments... Dec 14 10:39:45.880971826 (211h11m46.708680217s) [noise=true , address=77.90.31.10:4518 , piece_id=ZE3SFGBCM2QV3CGGS3VEYT7NKDTDE2OJ6T6CVSGGYACCFX5WN5ZQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.882649818 (211h11m46.710358212s) [noise=true , address=77.90.31.10:4519 , piece_id=ACPHBL7LF6NCNNH4U5GUXKW7RCL6XIEQVWYDCYUTXEC5RQE7RZBA, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.883088160 (211h11m46.71079655s) [noise=true , address=77.90.31.14:4661 , piece_id=LLI4CDLWJ32TFXQ2YEU3NRP4QIRSNNUODA5RRPWU5AA45MJEW2LQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.883693953 (211h11m46.711402345s) [noise=true , address=145.79.227.39:4014 , piece_id=WKYZGQX5SP22ML66ZQZIHBS324EONJ7AARTYAXK2SCV577ZCQ5OQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.884112448 (211h11m46.711820839s) [noise=true , address=145.79.227.32:4836 , piece_id=IQRH4CRVBOG72M3BE3G5FTIEMBTLD2KFUEYU6NCTK3N6PUONMY5A, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.884619811 (211h11m46.712328205s) [noise=true , address=145.79.227.43:4159 , piece_id=ULRDDEYCZXA26UPSFZQRQWFPU7MUCGJJSM7YWIR7HPXDCPBOQXNA, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.885012115 (211h11m46.712720506s) [noise=true , address=145.79.227.47:4267 , piece_id=VY46BNDUYVGBXDRIOEFK4USNOOJH2KXZH5Z2NHCT6SDJC5SFUYMQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.887669218 (211h11m46.71537761s) [noise=true , address=145.79.227.50:4375 , piece_id=RKVBJHHIF3KJLKB6ULTA4E5SHOBPCLPMWJCA6YTKF5XDNVEZMYXQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.890935473 (211h11m46.718643851s) [noise=true , address=145.79.227.46:4603 , piece_id=3X2DPA75MR7RX6LCOO2U3UREG4QSYHRRKRE57JUYUBINXTJN5IEA, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.891096730 (211h11m46.718805121s) [noise=true , address=145.79.227.28:4727 , piece_id=42ILJYV7IJRXSV3E3RWXWNUNYQHHQFNOF2JZGZKDEGPXRXZ3H2YA, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.893061640 (211h11m46.720770031s) [noise=true , address=145.79.227.7:4293 , piece_id=56YULKIWDM4P4WZM6KGMPUK3B3AK3RRVF4H6RJ7H6IQISPLXGBCQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:48.683585502 (211h11m49.511293883s) [noise=true , address=145.79.227.44:4194 , piece_id=ZWOCXZ746BWDZ2LWYHSQHSG2O3WJC4R6FDL5E3JP2SXRCX3AWT2Q, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:40:15.740699026 (211h12m16.568407426s) [noise=true , address=77.90.31.16:4735 , piece_id=XKJJK6K7XF3TOFWMIU2BUVQTESTNV4FRYINSXJS4MC7KK7ZJU7SQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: expected piece hash; context deadline exceeded; EOF Dec 14 10:40:15.740740404 (211h12m16.568448788s) [seg_pos={0 0}, upload=219598]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 14 10:40:15.762950810 (211h12m16.590659203s) [upload=219598]: Done uploading segments... Dec 14 10:40:15.762987936 (211h12m16.59069633s) [upload=219598]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 510.388196ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.X..........................................................] duration: 388.748071ms trace: https://jaeger.dev.storj.io/trace/25fec5475908806a FAIL (2025-12-14 11:20:09 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 14 10:39:45.775689210 (211h11m46.603397602s) [upload=219598]: Starting upload Dec 14 10:39:45.775750055 (211h11m46.603458455s) [upload=219598]: Done starting upload Dec 14 10:39:45.775957558 (211h11m46.603665941s) [upload=219598]: Uploading segments... Dec 14 10:39:45.776074427 (211h11m46.603782809s) [upload=219598]: Got next segment. Inline: false Dec 14 10:39:45.776158162 (211h11m46.603866553s) [upload=219598]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 14 10:39:45.776165965 (211h11m46.603874362s) [upload=219598]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 14 10:39:45.801257283 (211h11m46.628965678s) [seg_pos={0 0}, upload=219598]: Begin upload segment... Dec 14 10:39:45.801373021 (211h11m46.629081422s) [noise=true , address=145.79.227.43:4159 , piece_id=ULRDDEYCZXA26UPSFZQRQWFPU7MUCGJJSM7YWIR7HPXDCPBOQXNA, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.801409865 (211h11m46.629118247s) [seg_pos={0 0}, upload=219598]: Done begin upload segment. Dec 14 10:39:45.801481233 (211h11m46.629189635s) [noise=true , address=145.79.227.30:4763 , piece_id=CLMX6DHN5HP3OWMVR4R5MIVRH3GHCELUVTYJHBE7HXZQMHRQESIQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.802050295 (211h11m46.629758695s) [noise=true , address=145.79.227.32:4836 , piece_id=IQRH4CRVBOG72M3BE3G5FTIEMBTLD2KFUEYU6NCTK3N6PUONMY5A, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.802573060 (211h11m46.630281451s) [noise=true , address=77.90.31.14:4661 , piece_id=LLI4CDLWJ32TFXQ2YEU3NRP4QIRSNNUODA5RRPWU5AA45MJEW2LQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.803112997 (211h11m46.630821425s) [noise=true , address=145.79.227.44:4194 , piece_id=ZWOCXZ746BWDZ2LWYHSQHSG2O3WJC4R6FDL5E3JP2SXRCX3AWT2Q, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.803148596 (211h11m46.630856991s) [noise=true , address=145.79.227.46:4603 , piece_id=3X2DPA75MR7RX6LCOO2U3UREG4QSYHRRKRE57JUYUBINXTJN5IEA, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.803615052 (211h11m46.631323455s) [noise=true , address=145.79.227.28:4727 , piece_id=42ILJYV7IJRXSV3E3RWXWNUNYQHHQFNOF2JZGZKDEGPXRXZ3H2YA, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.803722621 (211h11m46.631431034s) [noise=true , address=77.90.31.16:4735 , piece_id=XKJJK6K7XF3TOFWMIU2BUVQTESTNV4FRYINSXJS4MC7KK7ZJU7SQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.804120053 (211h11m46.631828444s) [noise=true , address=145.79.227.7:4293 , piece_id=56YULKIWDM4P4WZM6KGMPUK3B3AK3RRVF4H6RJ7H6IQISPLXGBCQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.804570407 (211h11m46.632278798s) [noise=true , address=145.79.227.47:4267 , piece_id=VY46BNDUYVGBXDRIOEFK4USNOOJH2KXZH5Z2NHCT6SDJC5SFUYMQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.805083798 (211h11m46.632792196s) [noise=true , address=145.79.227.50:4375 , piece_id=RKVBJHHIF3KJLKB6ULTA4E5SHOBPCLPMWJCA6YTKF5XDNVEZMYXQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.805547276 (211h11m46.633255677s) [noise=true , address=77.90.31.16:4734 , piece_id=4JO2RBACNY5AM3KG7XUW7JTRGDCXGQDXSZ3L23ASAEJ4YFCICVAQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.805916642 (211h11m46.633625023s) [noise=true , address=145.79.227.39:4014 , piece_id=WKYZGQX5SP22ML66ZQZIHBS324EONJ7AARTYAXK2SCV577ZCQ5OQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.806440130 (211h11m46.634148525s) [noise=true , address=77.90.31.13:4626 , piece_id=DCKUBRB3DR755GN67RLP6NUMREAWIB3AITGPIMA7MM35BRPZW7IA, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.806976882 (211h11m46.634685284s) [noise=true , address=77.90.31.10:4519 , piece_id=ACPHBL7LF6NCNNH4U5GUXKW7RCL6XIEQVWYDCYUTXEC5RQE7RZBA, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.807729329 (211h11m46.63543772s) [noise=true , address=77.90.31.10:4518 , piece_id=ZE3SFGBCM2QV3CGGS3VEYT7NKDTDE2OJ6T6CVSGGYACCFX5WN5ZQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.841457783 (211h11m46.66916618s) [noise=true , address=77.90.31.13:4626 , piece_id=DCKUBRB3DR755GN67RLP6NUMREAWIB3AITGPIMA7MM35BRPZW7IA, seg_pos={0 0}, upload=219598]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: storage node overloaded, request limit: 2000 Dec 14 10:39:45.850206751 (211h11m46.677915151s) [noise=true , address=77.90.31.16:4734 , piece_id=4JO2RBACNY5AM3KG7XUW7JTRGDCXGQDXSZ3L23ASAEJ4YFCICVAQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: storage node overloaded, request limit: 2000 Dec 14 10:39:45.859644247 (211h11m46.68735265s) [noise=true , address=145.79.227.30:4763 , piece_id=CLMX6DHN5HP3OWMVR4R5MIVRH3GHCELUVTYJHBE7HXZQMHRQESIQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: storage node overloaded, request limit: 2000 Dec 14 10:39:45.874109133 (211h11m46.701817525s) [upload=219598]: Next returned nil segment Dec 14 10:39:45.874123719 (211h11m46.701832113s) [upload=219598]: Waiting for error group managing segments... Dec 14 10:39:45.880971826 (211h11m46.708680217s) [noise=true , address=77.90.31.10:4518 , piece_id=ZE3SFGBCM2QV3CGGS3VEYT7NKDTDE2OJ6T6CVSGGYACCFX5WN5ZQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.882649818 (211h11m46.710358212s) [noise=true , address=77.90.31.10:4519 , piece_id=ACPHBL7LF6NCNNH4U5GUXKW7RCL6XIEQVWYDCYUTXEC5RQE7RZBA, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.883088160 (211h11m46.71079655s) [noise=true , address=77.90.31.14:4661 , piece_id=LLI4CDLWJ32TFXQ2YEU3NRP4QIRSNNUODA5RRPWU5AA45MJEW2LQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.883693953 (211h11m46.711402345s) [noise=true , address=145.79.227.39:4014 , piece_id=WKYZGQX5SP22ML66ZQZIHBS324EONJ7AARTYAXK2SCV577ZCQ5OQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.884112448 (211h11m46.711820839s) [noise=true , address=145.79.227.32:4836 , piece_id=IQRH4CRVBOG72M3BE3G5FTIEMBTLD2KFUEYU6NCTK3N6PUONMY5A, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.884619811 (211h11m46.712328205s) [noise=true , address=145.79.227.43:4159 , piece_id=ULRDDEYCZXA26UPSFZQRQWFPU7MUCGJJSM7YWIR7HPXDCPBOQXNA, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.885012115 (211h11m46.712720506s) [noise=true , address=145.79.227.47:4267 , piece_id=VY46BNDUYVGBXDRIOEFK4USNOOJH2KXZH5Z2NHCT6SDJC5SFUYMQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.887669218 (211h11m46.71537761s) [noise=true , address=145.79.227.50:4375 , piece_id=RKVBJHHIF3KJLKB6ULTA4E5SHOBPCLPMWJCA6YTKF5XDNVEZMYXQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.890935473 (211h11m46.718643851s) [noise=true , address=145.79.227.46:4603 , piece_id=3X2DPA75MR7RX6LCOO2U3UREG4QSYHRRKRE57JUYUBINXTJN5IEA, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.891096730 (211h11m46.718805121s) [noise=true , address=145.79.227.28:4727 , piece_id=42ILJYV7IJRXSV3E3RWXWNUNYQHHQFNOF2JZGZKDEGPXRXZ3H2YA, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.893061640 (211h11m46.720770031s) [noise=true , address=145.79.227.7:4293 , piece_id=56YULKIWDM4P4WZM6KGMPUK3B3AK3RRVF4H6RJ7H6IQISPLXGBCQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:48.683585502 (211h11m49.511293883s) [noise=true , address=145.79.227.44:4194 , piece_id=ZWOCXZ746BWDZ2LWYHSQHSG2O3WJC4R6FDL5E3JP2SXRCX3AWT2Q, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:40:15.740699026 (211h12m16.568407426s) [noise=true , address=77.90.31.16:4735 , piece_id=XKJJK6K7XF3TOFWMIU2BUVQTESTNV4FRYINSXJS4MC7KK7ZJU7SQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: expected piece hash; context deadline exceeded; EOF Dec 14 10:40:15.740740404 (211h12m16.568448788s) [seg_pos={0 0}, upload=219598]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 14 10:40:15.762950810 (211h12m16.590659203s) [upload=219598]: Done uploading segments... Dec 14 10:40:15.762987936 (211h12m16.59069633s) [upload=219598]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 388.779257ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..X.........................................................] duration: 423.772315ms trace: https://jaeger.dev.storj.io/trace/25fec5475908806a FAIL (2025-12-14 11:19:28 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 14 10:39:45.775689210 (211h11m46.603397602s) [upload=219598]: Starting upload Dec 14 10:39:45.775750055 (211h11m46.603458455s) [upload=219598]: Done starting upload Dec 14 10:39:45.775957558 (211h11m46.603665941s) [upload=219598]: Uploading segments... Dec 14 10:39:45.776074427 (211h11m46.603782809s) [upload=219598]: Got next segment. Inline: false Dec 14 10:39:45.776158162 (211h11m46.603866553s) [upload=219598]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 14 10:39:45.776165965 (211h11m46.603874362s) [upload=219598]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 14 10:39:45.801257283 (211h11m46.628965678s) [seg_pos={0 0}, upload=219598]: Begin upload segment... Dec 14 10:39:45.801373021 (211h11m46.629081422s) [noise=true , address=145.79.227.43:4159 , piece_id=ULRDDEYCZXA26UPSFZQRQWFPU7MUCGJJSM7YWIR7HPXDCPBOQXNA, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.801409865 (211h11m46.629118247s) [seg_pos={0 0}, upload=219598]: Done begin upload segment. Dec 14 10:39:45.801481233 (211h11m46.629189635s) [noise=true , address=145.79.227.30:4763 , piece_id=CLMX6DHN5HP3OWMVR4R5MIVRH3GHCELUVTYJHBE7HXZQMHRQESIQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.802050295 (211h11m46.629758695s) [noise=true , address=145.79.227.32:4836 , piece_id=IQRH4CRVBOG72M3BE3G5FTIEMBTLD2KFUEYU6NCTK3N6PUONMY5A, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.802573060 (211h11m46.630281451s) [noise=true , address=77.90.31.14:4661 , piece_id=LLI4CDLWJ32TFXQ2YEU3NRP4QIRSNNUODA5RRPWU5AA45MJEW2LQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.803112997 (211h11m46.630821425s) [noise=true , address=145.79.227.44:4194 , piece_id=ZWOCXZ746BWDZ2LWYHSQHSG2O3WJC4R6FDL5E3JP2SXRCX3AWT2Q, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.803148596 (211h11m46.630856991s) [noise=true , address=145.79.227.46:4603 , piece_id=3X2DPA75MR7RX6LCOO2U3UREG4QSYHRRKRE57JUYUBINXTJN5IEA, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.803615052 (211h11m46.631323455s) [noise=true , address=145.79.227.28:4727 , piece_id=42ILJYV7IJRXSV3E3RWXWNUNYQHHQFNOF2JZGZKDEGPXRXZ3H2YA, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.803722621 (211h11m46.631431034s) [noise=true , address=77.90.31.16:4735 , piece_id=XKJJK6K7XF3TOFWMIU2BUVQTESTNV4FRYINSXJS4MC7KK7ZJU7SQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.804120053 (211h11m46.631828444s) [noise=true , address=145.79.227.7:4293 , piece_id=56YULKIWDM4P4WZM6KGMPUK3B3AK3RRVF4H6RJ7H6IQISPLXGBCQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.804570407 (211h11m46.632278798s) [noise=true , address=145.79.227.47:4267 , piece_id=VY46BNDUYVGBXDRIOEFK4USNOOJH2KXZH5Z2NHCT6SDJC5SFUYMQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.805083798 (211h11m46.632792196s) [noise=true , address=145.79.227.50:4375 , piece_id=RKVBJHHIF3KJLKB6ULTA4E5SHOBPCLPMWJCA6YTKF5XDNVEZMYXQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.805547276 (211h11m46.633255677s) [noise=true , address=77.90.31.16:4734 , piece_id=4JO2RBACNY5AM3KG7XUW7JTRGDCXGQDXSZ3L23ASAEJ4YFCICVAQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.805916642 (211h11m46.633625023s) [noise=true , address=145.79.227.39:4014 , piece_id=WKYZGQX5SP22ML66ZQZIHBS324EONJ7AARTYAXK2SCV577ZCQ5OQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.806440130 (211h11m46.634148525s) [noise=true , address=77.90.31.13:4626 , piece_id=DCKUBRB3DR755GN67RLP6NUMREAWIB3AITGPIMA7MM35BRPZW7IA, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.806976882 (211h11m46.634685284s) [noise=true , address=77.90.31.10:4519 , piece_id=ACPHBL7LF6NCNNH4U5GUXKW7RCL6XIEQVWYDCYUTXEC5RQE7RZBA, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.807729329 (211h11m46.63543772s) [noise=true , address=77.90.31.10:4518 , piece_id=ZE3SFGBCM2QV3CGGS3VEYT7NKDTDE2OJ6T6CVSGGYACCFX5WN5ZQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.841457783 (211h11m46.66916618s) [noise=true , address=77.90.31.13:4626 , piece_id=DCKUBRB3DR755GN67RLP6NUMREAWIB3AITGPIMA7MM35BRPZW7IA, seg_pos={0 0}, upload=219598]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: storage node overloaded, request limit: 2000 Dec 14 10:39:45.850206751 (211h11m46.677915151s) [noise=true , address=77.90.31.16:4734 , piece_id=4JO2RBACNY5AM3KG7XUW7JTRGDCXGQDXSZ3L23ASAEJ4YFCICVAQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: storage node overloaded, request limit: 2000 Dec 14 10:39:45.859644247 (211h11m46.68735265s) [noise=true , address=145.79.227.30:4763 , piece_id=CLMX6DHN5HP3OWMVR4R5MIVRH3GHCELUVTYJHBE7HXZQMHRQESIQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: storage node overloaded, request limit: 2000 Dec 14 10:39:45.874109133 (211h11m46.701817525s) [upload=219598]: Next returned nil segment Dec 14 10:39:45.874123719 (211h11m46.701832113s) [upload=219598]: Waiting for error group managing segments... Dec 14 10:39:45.880971826 (211h11m46.708680217s) [noise=true , address=77.90.31.10:4518 , piece_id=ZE3SFGBCM2QV3CGGS3VEYT7NKDTDE2OJ6T6CVSGGYACCFX5WN5ZQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.882649818 (211h11m46.710358212s) [noise=true , address=77.90.31.10:4519 , piece_id=ACPHBL7LF6NCNNH4U5GUXKW7RCL6XIEQVWYDCYUTXEC5RQE7RZBA, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.883088160 (211h11m46.71079655s) [noise=true , address=77.90.31.14:4661 , piece_id=LLI4CDLWJ32TFXQ2YEU3NRP4QIRSNNUODA5RRPWU5AA45MJEW2LQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.883693953 (211h11m46.711402345s) [noise=true , address=145.79.227.39:4014 , piece_id=WKYZGQX5SP22ML66ZQZIHBS324EONJ7AARTYAXK2SCV577ZCQ5OQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.884112448 (211h11m46.711820839s) [noise=true , address=145.79.227.32:4836 , piece_id=IQRH4CRVBOG72M3BE3G5FTIEMBTLD2KFUEYU6NCTK3N6PUONMY5A, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.884619811 (211h11m46.712328205s) [noise=true , address=145.79.227.43:4159 , piece_id=ULRDDEYCZXA26UPSFZQRQWFPU7MUCGJJSM7YWIR7HPXDCPBOQXNA, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.885012115 (211h11m46.712720506s) [noise=true , address=145.79.227.47:4267 , piece_id=VY46BNDUYVGBXDRIOEFK4USNOOJH2KXZH5Z2NHCT6SDJC5SFUYMQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.887669218 (211h11m46.71537761s) [noise=true , address=145.79.227.50:4375 , piece_id=RKVBJHHIF3KJLKB6ULTA4E5SHOBPCLPMWJCA6YTKF5XDNVEZMYXQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.890935473 (211h11m46.718643851s) [noise=true , address=145.79.227.46:4603 , piece_id=3X2DPA75MR7RX6LCOO2U3UREG4QSYHRRKRE57JUYUBINXTJN5IEA, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.891096730 (211h11m46.718805121s) [noise=true , address=145.79.227.28:4727 , piece_id=42ILJYV7IJRXSV3E3RWXWNUNYQHHQFNOF2JZGZKDEGPXRXZ3H2YA, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.893061640 (211h11m46.720770031s) [noise=true , address=145.79.227.7:4293 , piece_id=56YULKIWDM4P4WZM6KGMPUK3B3AK3RRVF4H6RJ7H6IQISPLXGBCQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:48.683585502 (211h11m49.511293883s) [noise=true , address=145.79.227.44:4194 , piece_id=ZWOCXZ746BWDZ2LWYHSQHSG2O3WJC4R6FDL5E3JP2SXRCX3AWT2Q, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:40:15.740699026 (211h12m16.568407426s) [noise=true , address=77.90.31.16:4735 , piece_id=XKJJK6K7XF3TOFWMIU2BUVQTESTNV4FRYINSXJS4MC7KK7ZJU7SQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: expected piece hash; context deadline exceeded; EOF Dec 14 10:40:15.740740404 (211h12m16.568448788s) [seg_pos={0 0}, upload=219598]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 14 10:40:15.762950810 (211h12m16.590659203s) [upload=219598]: Done uploading segments... Dec 14 10:40:15.762987936 (211h12m16.59069633s) [upload=219598]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 423.804626ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...X........................................................] duration: 390.00864ms trace: https://jaeger.dev.storj.io/trace/25fec5475908806a FAIL (2025-12-14 11:18:44 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 14 10:39:45.775689210 (211h11m46.603397602s) [upload=219598]: Starting upload Dec 14 10:39:45.775750055 (211h11m46.603458455s) [upload=219598]: Done starting upload Dec 14 10:39:45.775957558 (211h11m46.603665941s) [upload=219598]: Uploading segments... Dec 14 10:39:45.776074427 (211h11m46.603782809s) [upload=219598]: Got next segment. Inline: false Dec 14 10:39:45.776158162 (211h11m46.603866553s) [upload=219598]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 14 10:39:45.776165965 (211h11m46.603874362s) [upload=219598]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 14 10:39:45.801257283 (211h11m46.628965678s) [seg_pos={0 0}, upload=219598]: Begin upload segment... Dec 14 10:39:45.801373021 (211h11m46.629081422s) [noise=true , address=145.79.227.43:4159 , piece_id=ULRDDEYCZXA26UPSFZQRQWFPU7MUCGJJSM7YWIR7HPXDCPBOQXNA, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.801409865 (211h11m46.629118247s) [seg_pos={0 0}, upload=219598]: Done begin upload segment. Dec 14 10:39:45.801481233 (211h11m46.629189635s) [noise=true , address=145.79.227.30:4763 , piece_id=CLMX6DHN5HP3OWMVR4R5MIVRH3GHCELUVTYJHBE7HXZQMHRQESIQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.802050295 (211h11m46.629758695s) [noise=true , address=145.79.227.32:4836 , piece_id=IQRH4CRVBOG72M3BE3G5FTIEMBTLD2KFUEYU6NCTK3N6PUONMY5A, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.802573060 (211h11m46.630281451s) [noise=true , address=77.90.31.14:4661 , piece_id=LLI4CDLWJ32TFXQ2YEU3NRP4QIRSNNUODA5RRPWU5AA45MJEW2LQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.803112997 (211h11m46.630821425s) [noise=true , address=145.79.227.44:4194 , piece_id=ZWOCXZ746BWDZ2LWYHSQHSG2O3WJC4R6FDL5E3JP2SXRCX3AWT2Q, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.803148596 (211h11m46.630856991s) [noise=true , address=145.79.227.46:4603 , piece_id=3X2DPA75MR7RX6LCOO2U3UREG4QSYHRRKRE57JUYUBINXTJN5IEA, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.803615052 (211h11m46.631323455s) [noise=true , address=145.79.227.28:4727 , piece_id=42ILJYV7IJRXSV3E3RWXWNUNYQHHQFNOF2JZGZKDEGPXRXZ3H2YA, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.803722621 (211h11m46.631431034s) [noise=true , address=77.90.31.16:4735 , piece_id=XKJJK6K7XF3TOFWMIU2BUVQTESTNV4FRYINSXJS4MC7KK7ZJU7SQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.804120053 (211h11m46.631828444s) [noise=true , address=145.79.227.7:4293 , piece_id=56YULKIWDM4P4WZM6KGMPUK3B3AK3RRVF4H6RJ7H6IQISPLXGBCQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.804570407 (211h11m46.632278798s) [noise=true , address=145.79.227.47:4267 , piece_id=VY46BNDUYVGBXDRIOEFK4USNOOJH2KXZH5Z2NHCT6SDJC5SFUYMQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.805083798 (211h11m46.632792196s) [noise=true , address=145.79.227.50:4375 , piece_id=RKVBJHHIF3KJLKB6ULTA4E5SHOBPCLPMWJCA6YTKF5XDNVEZMYXQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.805547276 (211h11m46.633255677s) [noise=true , address=77.90.31.16:4734 , piece_id=4JO2RBACNY5AM3KG7XUW7JTRGDCXGQDXSZ3L23ASAEJ4YFCICVAQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.805916642 (211h11m46.633625023s) [noise=true , address=145.79.227.39:4014 , piece_id=WKYZGQX5SP22ML66ZQZIHBS324EONJ7AARTYAXK2SCV577ZCQ5OQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.806440130 (211h11m46.634148525s) [noise=true , address=77.90.31.13:4626 , piece_id=DCKUBRB3DR755GN67RLP6NUMREAWIB3AITGPIMA7MM35BRPZW7IA, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.806976882 (211h11m46.634685284s) [noise=true , address=77.90.31.10:4519 , piece_id=ACPHBL7LF6NCNNH4U5GUXKW7RCL6XIEQVWYDCYUTXEC5RQE7RZBA, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.807729329 (211h11m46.63543772s) [noise=true , address=77.90.31.10:4518 , piece_id=ZE3SFGBCM2QV3CGGS3VEYT7NKDTDE2OJ6T6CVSGGYACCFX5WN5ZQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.841457783 (211h11m46.66916618s) [noise=true , address=77.90.31.13:4626 , piece_id=DCKUBRB3DR755GN67RLP6NUMREAWIB3AITGPIMA7MM35BRPZW7IA, seg_pos={0 0}, upload=219598]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: storage node overloaded, request limit: 2000 Dec 14 10:39:45.850206751 (211h11m46.677915151s) [noise=true , address=77.90.31.16:4734 , piece_id=4JO2RBACNY5AM3KG7XUW7JTRGDCXGQDXSZ3L23ASAEJ4YFCICVAQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: storage node overloaded, request limit: 2000 Dec 14 10:39:45.859644247 (211h11m46.68735265s) [noise=true , address=145.79.227.30:4763 , piece_id=CLMX6DHN5HP3OWMVR4R5MIVRH3GHCELUVTYJHBE7HXZQMHRQESIQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: storage node overloaded, request limit: 2000 Dec 14 10:39:45.874109133 (211h11m46.701817525s) [upload=219598]: Next returned nil segment Dec 14 10:39:45.874123719 (211h11m46.701832113s) [upload=219598]: Waiting for error group managing segments... Dec 14 10:39:45.880971826 (211h11m46.708680217s) [noise=true , address=77.90.31.10:4518 , piece_id=ZE3SFGBCM2QV3CGGS3VEYT7NKDTDE2OJ6T6CVSGGYACCFX5WN5ZQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.882649818 (211h11m46.710358212s) [noise=true , address=77.90.31.10:4519 , piece_id=ACPHBL7LF6NCNNH4U5GUXKW7RCL6XIEQVWYDCYUTXEC5RQE7RZBA, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.883088160 (211h11m46.71079655s) [noise=true , address=77.90.31.14:4661 , piece_id=LLI4CDLWJ32TFXQ2YEU3NRP4QIRSNNUODA5RRPWU5AA45MJEW2LQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.883693953 (211h11m46.711402345s) [noise=true , address=145.79.227.39:4014 , piece_id=WKYZGQX5SP22ML66ZQZIHBS324EONJ7AARTYAXK2SCV577ZCQ5OQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.884112448 (211h11m46.711820839s) [noise=true , address=145.79.227.32:4836 , piece_id=IQRH4CRVBOG72M3BE3G5FTIEMBTLD2KFUEYU6NCTK3N6PUONMY5A, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.884619811 (211h11m46.712328205s) [noise=true , address=145.79.227.43:4159 , piece_id=ULRDDEYCZXA26UPSFZQRQWFPU7MUCGJJSM7YWIR7HPXDCPBOQXNA, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.885012115 (211h11m46.712720506s) [noise=true , address=145.79.227.47:4267 , piece_id=VY46BNDUYVGBXDRIOEFK4USNOOJH2KXZH5Z2NHCT6SDJC5SFUYMQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.887669218 (211h11m46.71537761s) [noise=true , address=145.79.227.50:4375 , piece_id=RKVBJHHIF3KJLKB6ULTA4E5SHOBPCLPMWJCA6YTKF5XDNVEZMYXQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.890935473 (211h11m46.718643851s) [noise=true , address=145.79.227.46:4603 , piece_id=3X2DPA75MR7RX6LCOO2U3UREG4QSYHRRKRE57JUYUBINXTJN5IEA, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.891096730 (211h11m46.718805121s) [noise=true , address=145.79.227.28:4727 , piece_id=42ILJYV7IJRXSV3E3RWXWNUNYQHHQFNOF2JZGZKDEGPXRXZ3H2YA, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.893061640 (211h11m46.720770031s) [noise=true , address=145.79.227.7:4293 , piece_id=56YULKIWDM4P4WZM6KGMPUK3B3AK3RRVF4H6RJ7H6IQISPLXGBCQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:48.683585502 (211h11m49.511293883s) [noise=true , address=145.79.227.44:4194 , piece_id=ZWOCXZ746BWDZ2LWYHSQHSG2O3WJC4R6FDL5E3JP2SXRCX3AWT2Q, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:40:15.740699026 (211h12m16.568407426s) [noise=true , address=77.90.31.16:4735 , piece_id=XKJJK6K7XF3TOFWMIU2BUVQTESTNV4FRYINSXJS4MC7KK7ZJU7SQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: expected piece hash; context deadline exceeded; EOF Dec 14 10:40:15.740740404 (211h12m16.568448788s) [seg_pos={0 0}, upload=219598]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 14 10:40:15.762950810 (211h12m16.590659203s) [upload=219598]: Done uploading segments... Dec 14 10:40:15.762987936 (211h12m16.59069633s) [upload=219598]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 390.043362ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....X.......................................................] duration: 397.120313ms trace: https://jaeger.dev.storj.io/trace/25fec5475908806a FAIL (2025-12-14 11:18:23 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 14 10:39:45.775689210 (211h11m46.603397602s) [upload=219598]: Starting upload Dec 14 10:39:45.775750055 (211h11m46.603458455s) [upload=219598]: Done starting upload Dec 14 10:39:45.775957558 (211h11m46.603665941s) [upload=219598]: Uploading segments... Dec 14 10:39:45.776074427 (211h11m46.603782809s) [upload=219598]: Got next segment. Inline: false Dec 14 10:39:45.776158162 (211h11m46.603866553s) [upload=219598]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 14 10:39:45.776165965 (211h11m46.603874362s) [upload=219598]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 14 10:39:45.801257283 (211h11m46.628965678s) [seg_pos={0 0}, upload=219598]: Begin upload segment... Dec 14 10:39:45.801373021 (211h11m46.629081422s) [noise=true , address=145.79.227.43:4159 , piece_id=ULRDDEYCZXA26UPSFZQRQWFPU7MUCGJJSM7YWIR7HPXDCPBOQXNA, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.801409865 (211h11m46.629118247s) [seg_pos={0 0}, upload=219598]: Done begin upload segment. Dec 14 10:39:45.801481233 (211h11m46.629189635s) [noise=true , address=145.79.227.30:4763 , piece_id=CLMX6DHN5HP3OWMVR4R5MIVRH3GHCELUVTYJHBE7HXZQMHRQESIQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.802050295 (211h11m46.629758695s) [noise=true , address=145.79.227.32:4836 , piece_id=IQRH4CRVBOG72M3BE3G5FTIEMBTLD2KFUEYU6NCTK3N6PUONMY5A, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.802573060 (211h11m46.630281451s) [noise=true , address=77.90.31.14:4661 , piece_id=LLI4CDLWJ32TFXQ2YEU3NRP4QIRSNNUODA5RRPWU5AA45MJEW2LQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.803112997 (211h11m46.630821425s) [noise=true , address=145.79.227.44:4194 , piece_id=ZWOCXZ746BWDZ2LWYHSQHSG2O3WJC4R6FDL5E3JP2SXRCX3AWT2Q, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.803148596 (211h11m46.630856991s) [noise=true , address=145.79.227.46:4603 , piece_id=3X2DPA75MR7RX6LCOO2U3UREG4QSYHRRKRE57JUYUBINXTJN5IEA, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.803615052 (211h11m46.631323455s) [noise=true , address=145.79.227.28:4727 , piece_id=42ILJYV7IJRXSV3E3RWXWNUNYQHHQFNOF2JZGZKDEGPXRXZ3H2YA, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.803722621 (211h11m46.631431034s) [noise=true , address=77.90.31.16:4735 , piece_id=XKJJK6K7XF3TOFWMIU2BUVQTESTNV4FRYINSXJS4MC7KK7ZJU7SQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.804120053 (211h11m46.631828444s) [noise=true , address=145.79.227.7:4293 , piece_id=56YULKIWDM4P4WZM6KGMPUK3B3AK3RRVF4H6RJ7H6IQISPLXGBCQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.804570407 (211h11m46.632278798s) [noise=true , address=145.79.227.47:4267 , piece_id=VY46BNDUYVGBXDRIOEFK4USNOOJH2KXZH5Z2NHCT6SDJC5SFUYMQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.805083798 (211h11m46.632792196s) [noise=true , address=145.79.227.50:4375 , piece_id=RKVBJHHIF3KJLKB6ULTA4E5SHOBPCLPMWJCA6YTKF5XDNVEZMYXQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.805547276 (211h11m46.633255677s) [noise=true , address=77.90.31.16:4734 , piece_id=4JO2RBACNY5AM3KG7XUW7JTRGDCXGQDXSZ3L23ASAEJ4YFCICVAQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.805916642 (211h11m46.633625023s) [noise=true , address=145.79.227.39:4014 , piece_id=WKYZGQX5SP22ML66ZQZIHBS324EONJ7AARTYAXK2SCV577ZCQ5OQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.806440130 (211h11m46.634148525s) [noise=true , address=77.90.31.13:4626 , piece_id=DCKUBRB3DR755GN67RLP6NUMREAWIB3AITGPIMA7MM35BRPZW7IA, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.806976882 (211h11m46.634685284s) [noise=true , address=77.90.31.10:4519 , piece_id=ACPHBL7LF6NCNNH4U5GUXKW7RCL6XIEQVWYDCYUTXEC5RQE7RZBA, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.807729329 (211h11m46.63543772s) [noise=true , address=77.90.31.10:4518 , piece_id=ZE3SFGBCM2QV3CGGS3VEYT7NKDTDE2OJ6T6CVSGGYACCFX5WN5ZQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.841457783 (211h11m46.66916618s) [noise=true , address=77.90.31.13:4626 , piece_id=DCKUBRB3DR755GN67RLP6NUMREAWIB3AITGPIMA7MM35BRPZW7IA, seg_pos={0 0}, upload=219598]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: storage node overloaded, request limit: 2000 Dec 14 10:39:45.850206751 (211h11m46.677915151s) [noise=true , address=77.90.31.16:4734 , piece_id=4JO2RBACNY5AM3KG7XUW7JTRGDCXGQDXSZ3L23ASAEJ4YFCICVAQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: storage node overloaded, request limit: 2000 Dec 14 10:39:45.859644247 (211h11m46.68735265s) [noise=true , address=145.79.227.30:4763 , piece_id=CLMX6DHN5HP3OWMVR4R5MIVRH3GHCELUVTYJHBE7HXZQMHRQESIQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: storage node overloaded, request limit: 2000 Dec 14 10:39:45.874109133 (211h11m46.701817525s) [upload=219598]: Next returned nil segment Dec 14 10:39:45.874123719 (211h11m46.701832113s) [upload=219598]: Waiting for error group managing segments... Dec 14 10:39:45.880971826 (211h11m46.708680217s) [noise=true , address=77.90.31.10:4518 , piece_id=ZE3SFGBCM2QV3CGGS3VEYT7NKDTDE2OJ6T6CVSGGYACCFX5WN5ZQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.882649818 (211h11m46.710358212s) [noise=true , address=77.90.31.10:4519 , piece_id=ACPHBL7LF6NCNNH4U5GUXKW7RCL6XIEQVWYDCYUTXEC5RQE7RZBA, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.883088160 (211h11m46.71079655s) [noise=true , address=77.90.31.14:4661 , piece_id=LLI4CDLWJ32TFXQ2YEU3NRP4QIRSNNUODA5RRPWU5AA45MJEW2LQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.883693953 (211h11m46.711402345s) [noise=true , address=145.79.227.39:4014 , piece_id=WKYZGQX5SP22ML66ZQZIHBS324EONJ7AARTYAXK2SCV577ZCQ5OQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.884112448 (211h11m46.711820839s) [noise=true , address=145.79.227.32:4836 , piece_id=IQRH4CRVBOG72M3BE3G5FTIEMBTLD2KFUEYU6NCTK3N6PUONMY5A, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.884619811 (211h11m46.712328205s) [noise=true , address=145.79.227.43:4159 , piece_id=ULRDDEYCZXA26UPSFZQRQWFPU7MUCGJJSM7YWIR7HPXDCPBOQXNA, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.885012115 (211h11m46.712720506s) [noise=true , address=145.79.227.47:4267 , piece_id=VY46BNDUYVGBXDRIOEFK4USNOOJH2KXZH5Z2NHCT6SDJC5SFUYMQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.887669218 (211h11m46.71537761s) [noise=true , address=145.79.227.50:4375 , piece_id=RKVBJHHIF3KJLKB6ULTA4E5SHOBPCLPMWJCA6YTKF5XDNVEZMYXQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.890935473 (211h11m46.718643851s) [noise=true , address=145.79.227.46:4603 , piece_id=3X2DPA75MR7RX6LCOO2U3UREG4QSYHRRKRE57JUYUBINXTJN5IEA, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.891096730 (211h11m46.718805121s) [noise=true , address=145.79.227.28:4727 , piece_id=42ILJYV7IJRXSV3E3RWXWNUNYQHHQFNOF2JZGZKDEGPXRXZ3H2YA, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.893061640 (211h11m46.720770031s) [noise=true , address=145.79.227.7:4293 , piece_id=56YULKIWDM4P4WZM6KGMPUK3B3AK3RRVF4H6RJ7H6IQISPLXGBCQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:48.683585502 (211h11m49.511293883s) [noise=true , address=145.79.227.44:4194 , piece_id=ZWOCXZ746BWDZ2LWYHSQHSG2O3WJC4R6FDL5E3JP2SXRCX3AWT2Q, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:40:15.740699026 (211h12m16.568407426s) [noise=true , address=77.90.31.16:4735 , piece_id=XKJJK6K7XF3TOFWMIU2BUVQTESTNV4FRYINSXJS4MC7KK7ZJU7SQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: expected piece hash; context deadline exceeded; EOF Dec 14 10:40:15.740740404 (211h12m16.568448788s) [seg_pos={0 0}, upload=219598]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 14 10:40:15.762950810 (211h12m16.590659203s) [upload=219598]: Done uploading segments... Dec 14 10:40:15.762987936 (211h12m16.59069633s) [upload=219598]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 397.14737ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....X......................................................] duration: 401.153573ms trace: https://jaeger.dev.storj.io/trace/25fec5475908806a FAIL (2025-12-14 11:17:52 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 14 10:39:45.775689210 (211h11m46.603397602s) [upload=219598]: Starting upload Dec 14 10:39:45.775750055 (211h11m46.603458455s) [upload=219598]: Done starting upload Dec 14 10:39:45.775957558 (211h11m46.603665941s) [upload=219598]: Uploading segments... Dec 14 10:39:45.776074427 (211h11m46.603782809s) [upload=219598]: Got next segment. Inline: false Dec 14 10:39:45.776158162 (211h11m46.603866553s) [upload=219598]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 14 10:39:45.776165965 (211h11m46.603874362s) [upload=219598]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 14 10:39:45.801257283 (211h11m46.628965678s) [seg_pos={0 0}, upload=219598]: Begin upload segment... Dec 14 10:39:45.801373021 (211h11m46.629081422s) [noise=true , address=145.79.227.43:4159 , piece_id=ULRDDEYCZXA26UPSFZQRQWFPU7MUCGJJSM7YWIR7HPXDCPBOQXNA, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.801409865 (211h11m46.629118247s) [seg_pos={0 0}, upload=219598]: Done begin upload segment. Dec 14 10:39:45.801481233 (211h11m46.629189635s) [noise=true , address=145.79.227.30:4763 , piece_id=CLMX6DHN5HP3OWMVR4R5MIVRH3GHCELUVTYJHBE7HXZQMHRQESIQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.802050295 (211h11m46.629758695s) [noise=true , address=145.79.227.32:4836 , piece_id=IQRH4CRVBOG72M3BE3G5FTIEMBTLD2KFUEYU6NCTK3N6PUONMY5A, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.802573060 (211h11m46.630281451s) [noise=true , address=77.90.31.14:4661 , piece_id=LLI4CDLWJ32TFXQ2YEU3NRP4QIRSNNUODA5RRPWU5AA45MJEW2LQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.803112997 (211h11m46.630821425s) [noise=true , address=145.79.227.44:4194 , piece_id=ZWOCXZ746BWDZ2LWYHSQHSG2O3WJC4R6FDL5E3JP2SXRCX3AWT2Q, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.803148596 (211h11m46.630856991s) [noise=true , address=145.79.227.46:4603 , piece_id=3X2DPA75MR7RX6LCOO2U3UREG4QSYHRRKRE57JUYUBINXTJN5IEA, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.803615052 (211h11m46.631323455s) [noise=true , address=145.79.227.28:4727 , piece_id=42ILJYV7IJRXSV3E3RWXWNUNYQHHQFNOF2JZGZKDEGPXRXZ3H2YA, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.803722621 (211h11m46.631431034s) [noise=true , address=77.90.31.16:4735 , piece_id=XKJJK6K7XF3TOFWMIU2BUVQTESTNV4FRYINSXJS4MC7KK7ZJU7SQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.804120053 (211h11m46.631828444s) [noise=true , address=145.79.227.7:4293 , piece_id=56YULKIWDM4P4WZM6KGMPUK3B3AK3RRVF4H6RJ7H6IQISPLXGBCQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.804570407 (211h11m46.632278798s) [noise=true , address=145.79.227.47:4267 , piece_id=VY46BNDUYVGBXDRIOEFK4USNOOJH2KXZH5Z2NHCT6SDJC5SFUYMQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.805083798 (211h11m46.632792196s) [noise=true , address=145.79.227.50:4375 , piece_id=RKVBJHHIF3KJLKB6ULTA4E5SHOBPCLPMWJCA6YTKF5XDNVEZMYXQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.805547276 (211h11m46.633255677s) [noise=true , address=77.90.31.16:4734 , piece_id=4JO2RBACNY5AM3KG7XUW7JTRGDCXGQDXSZ3L23ASAEJ4YFCICVAQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.805916642 (211h11m46.633625023s) [noise=true , address=145.79.227.39:4014 , piece_id=WKYZGQX5SP22ML66ZQZIHBS324EONJ7AARTYAXK2SCV577ZCQ5OQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.806440130 (211h11m46.634148525s) [noise=true , address=77.90.31.13:4626 , piece_id=DCKUBRB3DR755GN67RLP6NUMREAWIB3AITGPIMA7MM35BRPZW7IA, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.806976882 (211h11m46.634685284s) [noise=true , address=77.90.31.10:4519 , piece_id=ACPHBL7LF6NCNNH4U5GUXKW7RCL6XIEQVWYDCYUTXEC5RQE7RZBA, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.807729329 (211h11m46.63543772s) [noise=true , address=77.90.31.10:4518 , piece_id=ZE3SFGBCM2QV3CGGS3VEYT7NKDTDE2OJ6T6CVSGGYACCFX5WN5ZQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.841457783 (211h11m46.66916618s) [noise=true , address=77.90.31.13:4626 , piece_id=DCKUBRB3DR755GN67RLP6NUMREAWIB3AITGPIMA7MM35BRPZW7IA, seg_pos={0 0}, upload=219598]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: storage node overloaded, request limit: 2000 Dec 14 10:39:45.850206751 (211h11m46.677915151s) [noise=true , address=77.90.31.16:4734 , piece_id=4JO2RBACNY5AM3KG7XUW7JTRGDCXGQDXSZ3L23ASAEJ4YFCICVAQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: storage node overloaded, request limit: 2000 Dec 14 10:39:45.859644247 (211h11m46.68735265s) [noise=true , address=145.79.227.30:4763 , piece_id=CLMX6DHN5HP3OWMVR4R5MIVRH3GHCELUVTYJHBE7HXZQMHRQESIQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: storage node overloaded, request limit: 2000 Dec 14 10:39:45.874109133 (211h11m46.701817525s) [upload=219598]: Next returned nil segment Dec 14 10:39:45.874123719 (211h11m46.701832113s) [upload=219598]: Waiting for error group managing segments... Dec 14 10:39:45.880971826 (211h11m46.708680217s) [noise=true , address=77.90.31.10:4518 , piece_id=ZE3SFGBCM2QV3CGGS3VEYT7NKDTDE2OJ6T6CVSGGYACCFX5WN5ZQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.882649818 (211h11m46.710358212s) [noise=true , address=77.90.31.10:4519 , piece_id=ACPHBL7LF6NCNNH4U5GUXKW7RCL6XIEQVWYDCYUTXEC5RQE7RZBA, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.883088160 (211h11m46.71079655s) [noise=true , address=77.90.31.14:4661 , piece_id=LLI4CDLWJ32TFXQ2YEU3NRP4QIRSNNUODA5RRPWU5AA45MJEW2LQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.883693953 (211h11m46.711402345s) [noise=true , address=145.79.227.39:4014 , piece_id=WKYZGQX5SP22ML66ZQZIHBS324EONJ7AARTYAXK2SCV577ZCQ5OQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.884112448 (211h11m46.711820839s) [noise=true , address=145.79.227.32:4836 , piece_id=IQRH4CRVBOG72M3BE3G5FTIEMBTLD2KFUEYU6NCTK3N6PUONMY5A, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.884619811 (211h11m46.712328205s) [noise=true , address=145.79.227.43:4159 , piece_id=ULRDDEYCZXA26UPSFZQRQWFPU7MUCGJJSM7YWIR7HPXDCPBOQXNA, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.885012115 (211h11m46.712720506s) [noise=true , address=145.79.227.47:4267 , piece_id=VY46BNDUYVGBXDRIOEFK4USNOOJH2KXZH5Z2NHCT6SDJC5SFUYMQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.887669218 (211h11m46.71537761s) [noise=true , address=145.79.227.50:4375 , piece_id=RKVBJHHIF3KJLKB6ULTA4E5SHOBPCLPMWJCA6YTKF5XDNVEZMYXQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.890935473 (211h11m46.718643851s) [noise=true , address=145.79.227.46:4603 , piece_id=3X2DPA75MR7RX6LCOO2U3UREG4QSYHRRKRE57JUYUBINXTJN5IEA, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.891096730 (211h11m46.718805121s) [noise=true , address=145.79.227.28:4727 , piece_id=42ILJYV7IJRXSV3E3RWXWNUNYQHHQFNOF2JZGZKDEGPXRXZ3H2YA, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.893061640 (211h11m46.720770031s) [noise=true , address=145.79.227.7:4293 , piece_id=56YULKIWDM4P4WZM6KGMPUK3B3AK3RRVF4H6RJ7H6IQISPLXGBCQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:48.683585502 (211h11m49.511293883s) [noise=true , address=145.79.227.44:4194 , piece_id=ZWOCXZ746BWDZ2LWYHSQHSG2O3WJC4R6FDL5E3JP2SXRCX3AWT2Q, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:40:15.740699026 (211h12m16.568407426s) [noise=true , address=77.90.31.16:4735 , piece_id=XKJJK6K7XF3TOFWMIU2BUVQTESTNV4FRYINSXJS4MC7KK7ZJU7SQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: expected piece hash; context deadline exceeded; EOF Dec 14 10:40:15.740740404 (211h12m16.568448788s) [seg_pos={0 0}, upload=219598]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 14 10:40:15.762950810 (211h12m16.590659203s) [upload=219598]: Done uploading segments... Dec 14 10:40:15.762987936 (211h12m16.59069633s) [upload=219598]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 401.182335ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......X.....................................................] duration: 539.151888ms trace: https://jaeger.dev.storj.io/trace/25fec5475908806a FAIL (2025-12-14 11:16:53 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 14 10:39:45.775689210 (211h11m46.603397602s) [upload=219598]: Starting upload Dec 14 10:39:45.775750055 (211h11m46.603458455s) [upload=219598]: Done starting upload Dec 14 10:39:45.775957558 (211h11m46.603665941s) [upload=219598]: Uploading segments... Dec 14 10:39:45.776074427 (211h11m46.603782809s) [upload=219598]: Got next segment. Inline: false Dec 14 10:39:45.776158162 (211h11m46.603866553s) [upload=219598]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 14 10:39:45.776165965 (211h11m46.603874362s) [upload=219598]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 14 10:39:45.801257283 (211h11m46.628965678s) [seg_pos={0 0}, upload=219598]: Begin upload segment... Dec 14 10:39:45.801373021 (211h11m46.629081422s) [noise=true , address=145.79.227.43:4159 , piece_id=ULRDDEYCZXA26UPSFZQRQWFPU7MUCGJJSM7YWIR7HPXDCPBOQXNA, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.801409865 (211h11m46.629118247s) [seg_pos={0 0}, upload=219598]: Done begin upload segment. Dec 14 10:39:45.801481233 (211h11m46.629189635s) [noise=true , address=145.79.227.30:4763 , piece_id=CLMX6DHN5HP3OWMVR4R5MIVRH3GHCELUVTYJHBE7HXZQMHRQESIQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.802050295 (211h11m46.629758695s) [noise=true , address=145.79.227.32:4836 , piece_id=IQRH4CRVBOG72M3BE3G5FTIEMBTLD2KFUEYU6NCTK3N6PUONMY5A, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.802573060 (211h11m46.630281451s) [noise=true , address=77.90.31.14:4661 , piece_id=LLI4CDLWJ32TFXQ2YEU3NRP4QIRSNNUODA5RRPWU5AA45MJEW2LQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.803112997 (211h11m46.630821425s) [noise=true , address=145.79.227.44:4194 , piece_id=ZWOCXZ746BWDZ2LWYHSQHSG2O3WJC4R6FDL5E3JP2SXRCX3AWT2Q, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.803148596 (211h11m46.630856991s) [noise=true , address=145.79.227.46:4603 , piece_id=3X2DPA75MR7RX6LCOO2U3UREG4QSYHRRKRE57JUYUBINXTJN5IEA, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.803615052 (211h11m46.631323455s) [noise=true , address=145.79.227.28:4727 , piece_id=42ILJYV7IJRXSV3E3RWXWNUNYQHHQFNOF2JZGZKDEGPXRXZ3H2YA, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.803722621 (211h11m46.631431034s) [noise=true , address=77.90.31.16:4735 , piece_id=XKJJK6K7XF3TOFWMIU2BUVQTESTNV4FRYINSXJS4MC7KK7ZJU7SQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.804120053 (211h11m46.631828444s) [noise=true , address=145.79.227.7:4293 , piece_id=56YULKIWDM4P4WZM6KGMPUK3B3AK3RRVF4H6RJ7H6IQISPLXGBCQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.804570407 (211h11m46.632278798s) [noise=true , address=145.79.227.47:4267 , piece_id=VY46BNDUYVGBXDRIOEFK4USNOOJH2KXZH5Z2NHCT6SDJC5SFUYMQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.805083798 (211h11m46.632792196s) [noise=true , address=145.79.227.50:4375 , piece_id=RKVBJHHIF3KJLKB6ULTA4E5SHOBPCLPMWJCA6YTKF5XDNVEZMYXQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.805547276 (211h11m46.633255677s) [noise=true , address=77.90.31.16:4734 , piece_id=4JO2RBACNY5AM3KG7XUW7JTRGDCXGQDXSZ3L23ASAEJ4YFCICVAQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.805916642 (211h11m46.633625023s) [noise=true , address=145.79.227.39:4014 , piece_id=WKYZGQX5SP22ML66ZQZIHBS324EONJ7AARTYAXK2SCV577ZCQ5OQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.806440130 (211h11m46.634148525s) [noise=true , address=77.90.31.13:4626 , piece_id=DCKUBRB3DR755GN67RLP6NUMREAWIB3AITGPIMA7MM35BRPZW7IA, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.806976882 (211h11m46.634685284s) [noise=true , address=77.90.31.10:4519 , piece_id=ACPHBL7LF6NCNNH4U5GUXKW7RCL6XIEQVWYDCYUTXEC5RQE7RZBA, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.807729329 (211h11m46.63543772s) [noise=true , address=77.90.31.10:4518 , piece_id=ZE3SFGBCM2QV3CGGS3VEYT7NKDTDE2OJ6T6CVSGGYACCFX5WN5ZQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.841457783 (211h11m46.66916618s) [noise=true , address=77.90.31.13:4626 , piece_id=DCKUBRB3DR755GN67RLP6NUMREAWIB3AITGPIMA7MM35BRPZW7IA, seg_pos={0 0}, upload=219598]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: storage node overloaded, request limit: 2000 Dec 14 10:39:45.850206751 (211h11m46.677915151s) [noise=true , address=77.90.31.16:4734 , piece_id=4JO2RBACNY5AM3KG7XUW7JTRGDCXGQDXSZ3L23ASAEJ4YFCICVAQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: storage node overloaded, request limit: 2000 Dec 14 10:39:45.859644247 (211h11m46.68735265s) [noise=true , address=145.79.227.30:4763 , piece_id=CLMX6DHN5HP3OWMVR4R5MIVRH3GHCELUVTYJHBE7HXZQMHRQESIQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: storage node overloaded, request limit: 2000 Dec 14 10:39:45.874109133 (211h11m46.701817525s) [upload=219598]: Next returned nil segment Dec 14 10:39:45.874123719 (211h11m46.701832113s) [upload=219598]: Waiting for error group managing segments... Dec 14 10:39:45.880971826 (211h11m46.708680217s) [noise=true , address=77.90.31.10:4518 , piece_id=ZE3SFGBCM2QV3CGGS3VEYT7NKDTDE2OJ6T6CVSGGYACCFX5WN5ZQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.882649818 (211h11m46.710358212s) [noise=true , address=77.90.31.10:4519 , piece_id=ACPHBL7LF6NCNNH4U5GUXKW7RCL6XIEQVWYDCYUTXEC5RQE7RZBA, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.883088160 (211h11m46.71079655s) [noise=true , address=77.90.31.14:4661 , piece_id=LLI4CDLWJ32TFXQ2YEU3NRP4QIRSNNUODA5RRPWU5AA45MJEW2LQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.883693953 (211h11m46.711402345s) [noise=true , address=145.79.227.39:4014 , piece_id=WKYZGQX5SP22ML66ZQZIHBS324EONJ7AARTYAXK2SCV577ZCQ5OQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.884112448 (211h11m46.711820839s) [noise=true , address=145.79.227.32:4836 , piece_id=IQRH4CRVBOG72M3BE3G5FTIEMBTLD2KFUEYU6NCTK3N6PUONMY5A, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.884619811 (211h11m46.712328205s) [noise=true , address=145.79.227.43:4159 , piece_id=ULRDDEYCZXA26UPSFZQRQWFPU7MUCGJJSM7YWIR7HPXDCPBOQXNA, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.885012115 (211h11m46.712720506s) [noise=true , address=145.79.227.47:4267 , piece_id=VY46BNDUYVGBXDRIOEFK4USNOOJH2KXZH5Z2NHCT6SDJC5SFUYMQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.887669218 (211h11m46.71537761s) [noise=true , address=145.79.227.50:4375 , piece_id=RKVBJHHIF3KJLKB6ULTA4E5SHOBPCLPMWJCA6YTKF5XDNVEZMYXQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.890935473 (211h11m46.718643851s) [noise=true , address=145.79.227.46:4603 , piece_id=3X2DPA75MR7RX6LCOO2U3UREG4QSYHRRKRE57JUYUBINXTJN5IEA, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.891096730 (211h11m46.718805121s) [noise=true , address=145.79.227.28:4727 , piece_id=42ILJYV7IJRXSV3E3RWXWNUNYQHHQFNOF2JZGZKDEGPXRXZ3H2YA, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.893061640 (211h11m46.720770031s) [noise=true , address=145.79.227.7:4293 , piece_id=56YULKIWDM4P4WZM6KGMPUK3B3AK3RRVF4H6RJ7H6IQISPLXGBCQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:48.683585502 (211h11m49.511293883s) [noise=true , address=145.79.227.44:4194 , piece_id=ZWOCXZ746BWDZ2LWYHSQHSG2O3WJC4R6FDL5E3JP2SXRCX3AWT2Q, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:40:15.740699026 (211h12m16.568407426s) [noise=true , address=77.90.31.16:4735 , piece_id=XKJJK6K7XF3TOFWMIU2BUVQTESTNV4FRYINSXJS4MC7KK7ZJU7SQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: expected piece hash; context deadline exceeded; EOF Dec 14 10:40:15.740740404 (211h12m16.568448788s) [seg_pos={0 0}, upload=219598]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 14 10:40:15.762950810 (211h12m16.590659203s) [upload=219598]: Done uploading segments... Dec 14 10:40:15.762987936 (211h12m16.59069633s) [upload=219598]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 539.203947ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......X....................................................] duration: 431.567266ms trace: https://jaeger.dev.storj.io/trace/25fec5475908806a FAIL (2025-12-14 11:16:05 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 14 10:39:45.775689210 (211h11m46.603397602s) [upload=219598]: Starting upload Dec 14 10:39:45.775750055 (211h11m46.603458455s) [upload=219598]: Done starting upload Dec 14 10:39:45.775957558 (211h11m46.603665941s) [upload=219598]: Uploading segments... Dec 14 10:39:45.776074427 (211h11m46.603782809s) [upload=219598]: Got next segment. Inline: false Dec 14 10:39:45.776158162 (211h11m46.603866553s) [upload=219598]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 14 10:39:45.776165965 (211h11m46.603874362s) [upload=219598]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 14 10:39:45.801257283 (211h11m46.628965678s) [seg_pos={0 0}, upload=219598]: Begin upload segment... Dec 14 10:39:45.801373021 (211h11m46.629081422s) [noise=true , address=145.79.227.43:4159 , piece_id=ULRDDEYCZXA26UPSFZQRQWFPU7MUCGJJSM7YWIR7HPXDCPBOQXNA, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.801409865 (211h11m46.629118247s) [seg_pos={0 0}, upload=219598]: Done begin upload segment. Dec 14 10:39:45.801481233 (211h11m46.629189635s) [noise=true , address=145.79.227.30:4763 , piece_id=CLMX6DHN5HP3OWMVR4R5MIVRH3GHCELUVTYJHBE7HXZQMHRQESIQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.802050295 (211h11m46.629758695s) [noise=true , address=145.79.227.32:4836 , piece_id=IQRH4CRVBOG72M3BE3G5FTIEMBTLD2KFUEYU6NCTK3N6PUONMY5A, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.802573060 (211h11m46.630281451s) [noise=true , address=77.90.31.14:4661 , piece_id=LLI4CDLWJ32TFXQ2YEU3NRP4QIRSNNUODA5RRPWU5AA45MJEW2LQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.803112997 (211h11m46.630821425s) [noise=true , address=145.79.227.44:4194 , piece_id=ZWOCXZ746BWDZ2LWYHSQHSG2O3WJC4R6FDL5E3JP2SXRCX3AWT2Q, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.803148596 (211h11m46.630856991s) [noise=true , address=145.79.227.46:4603 , piece_id=3X2DPA75MR7RX6LCOO2U3UREG4QSYHRRKRE57JUYUBINXTJN5IEA, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.803615052 (211h11m46.631323455s) [noise=true , address=145.79.227.28:4727 , piece_id=42ILJYV7IJRXSV3E3RWXWNUNYQHHQFNOF2JZGZKDEGPXRXZ3H2YA, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.803722621 (211h11m46.631431034s) [noise=true , address=77.90.31.16:4735 , piece_id=XKJJK6K7XF3TOFWMIU2BUVQTESTNV4FRYINSXJS4MC7KK7ZJU7SQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.804120053 (211h11m46.631828444s) [noise=true , address=145.79.227.7:4293 , piece_id=56YULKIWDM4P4WZM6KGMPUK3B3AK3RRVF4H6RJ7H6IQISPLXGBCQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.804570407 (211h11m46.632278798s) [noise=true , address=145.79.227.47:4267 , piece_id=VY46BNDUYVGBXDRIOEFK4USNOOJH2KXZH5Z2NHCT6SDJC5SFUYMQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.805083798 (211h11m46.632792196s) [noise=true , address=145.79.227.50:4375 , piece_id=RKVBJHHIF3KJLKB6ULTA4E5SHOBPCLPMWJCA6YTKF5XDNVEZMYXQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.805547276 (211h11m46.633255677s) [noise=true , address=77.90.31.16:4734 , piece_id=4JO2RBACNY5AM3KG7XUW7JTRGDCXGQDXSZ3L23ASAEJ4YFCICVAQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.805916642 (211h11m46.633625023s) [noise=true , address=145.79.227.39:4014 , piece_id=WKYZGQX5SP22ML66ZQZIHBS324EONJ7AARTYAXK2SCV577ZCQ5OQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.806440130 (211h11m46.634148525s) [noise=true , address=77.90.31.13:4626 , piece_id=DCKUBRB3DR755GN67RLP6NUMREAWIB3AITGPIMA7MM35BRPZW7IA, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.806976882 (211h11m46.634685284s) [noise=true , address=77.90.31.10:4519 , piece_id=ACPHBL7LF6NCNNH4U5GUXKW7RCL6XIEQVWYDCYUTXEC5RQE7RZBA, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.807729329 (211h11m46.63543772s) [noise=true , address=77.90.31.10:4518 , piece_id=ZE3SFGBCM2QV3CGGS3VEYT7NKDTDE2OJ6T6CVSGGYACCFX5WN5ZQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.841457783 (211h11m46.66916618s) [noise=true , address=77.90.31.13:4626 , piece_id=DCKUBRB3DR755GN67RLP6NUMREAWIB3AITGPIMA7MM35BRPZW7IA, seg_pos={0 0}, upload=219598]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: storage node overloaded, request limit: 2000 Dec 14 10:39:45.850206751 (211h11m46.677915151s) [noise=true , address=77.90.31.16:4734 , piece_id=4JO2RBACNY5AM3KG7XUW7JTRGDCXGQDXSZ3L23ASAEJ4YFCICVAQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: storage node overloaded, request limit: 2000 Dec 14 10:39:45.859644247 (211h11m46.68735265s) [noise=true , address=145.79.227.30:4763 , piece_id=CLMX6DHN5HP3OWMVR4R5MIVRH3GHCELUVTYJHBE7HXZQMHRQESIQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: storage node overloaded, request limit: 2000 Dec 14 10:39:45.874109133 (211h11m46.701817525s) [upload=219598]: Next returned nil segment Dec 14 10:39:45.874123719 (211h11m46.701832113s) [upload=219598]: Waiting for error group managing segments... Dec 14 10:39:45.880971826 (211h11m46.708680217s) [noise=true , address=77.90.31.10:4518 , piece_id=ZE3SFGBCM2QV3CGGS3VEYT7NKDTDE2OJ6T6CVSGGYACCFX5WN5ZQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.882649818 (211h11m46.710358212s) [noise=true , address=77.90.31.10:4519 , piece_id=ACPHBL7LF6NCNNH4U5GUXKW7RCL6XIEQVWYDCYUTXEC5RQE7RZBA, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.883088160 (211h11m46.71079655s) [noise=true , address=77.90.31.14:4661 , piece_id=LLI4CDLWJ32TFXQ2YEU3NRP4QIRSNNUODA5RRPWU5AA45MJEW2LQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.883693953 (211h11m46.711402345s) [noise=true , address=145.79.227.39:4014 , piece_id=WKYZGQX5SP22ML66ZQZIHBS324EONJ7AARTYAXK2SCV577ZCQ5OQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.884112448 (211h11m46.711820839s) [noise=true , address=145.79.227.32:4836 , piece_id=IQRH4CRVBOG72M3BE3G5FTIEMBTLD2KFUEYU6NCTK3N6PUONMY5A, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.884619811 (211h11m46.712328205s) [noise=true , address=145.79.227.43:4159 , piece_id=ULRDDEYCZXA26UPSFZQRQWFPU7MUCGJJSM7YWIR7HPXDCPBOQXNA, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.885012115 (211h11m46.712720506s) [noise=true , address=145.79.227.47:4267 , piece_id=VY46BNDUYVGBXDRIOEFK4USNOOJH2KXZH5Z2NHCT6SDJC5SFUYMQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.887669218 (211h11m46.71537761s) [noise=true , address=145.79.227.50:4375 , piece_id=RKVBJHHIF3KJLKB6ULTA4E5SHOBPCLPMWJCA6YTKF5XDNVEZMYXQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.890935473 (211h11m46.718643851s) [noise=true , address=145.79.227.46:4603 , piece_id=3X2DPA75MR7RX6LCOO2U3UREG4QSYHRRKRE57JUYUBINXTJN5IEA, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.891096730 (211h11m46.718805121s) [noise=true , address=145.79.227.28:4727 , piece_id=42ILJYV7IJRXSV3E3RWXWNUNYQHHQFNOF2JZGZKDEGPXRXZ3H2YA, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.893061640 (211h11m46.720770031s) [noise=true , address=145.79.227.7:4293 , piece_id=56YULKIWDM4P4WZM6KGMPUK3B3AK3RRVF4H6RJ7H6IQISPLXGBCQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:48.683585502 (211h11m49.511293883s) [noise=true , address=145.79.227.44:4194 , piece_id=ZWOCXZ746BWDZ2LWYHSQHSG2O3WJC4R6FDL5E3JP2SXRCX3AWT2Q, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:40:15.740699026 (211h12m16.568407426s) [noise=true , address=77.90.31.16:4735 , piece_id=XKJJK6K7XF3TOFWMIU2BUVQTESTNV4FRYINSXJS4MC7KK7ZJU7SQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: expected piece hash; context deadline exceeded; EOF Dec 14 10:40:15.740740404 (211h12m16.568448788s) [seg_pos={0 0}, upload=219598]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 14 10:40:15.762950810 (211h12m16.590659203s) [upload=219598]: Done uploading segments... Dec 14 10:40:15.762987936 (211h12m16.59069633s) [upload=219598]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 431.605288ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........X...................................................] duration: 423.684375ms trace: https://jaeger.dev.storj.io/trace/25fec5475908806a FAIL (2025-12-14 11:15:19 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 14 10:39:45.775689210 (211h11m46.603397602s) [upload=219598]: Starting upload Dec 14 10:39:45.775750055 (211h11m46.603458455s) [upload=219598]: Done starting upload Dec 14 10:39:45.775957558 (211h11m46.603665941s) [upload=219598]: Uploading segments... Dec 14 10:39:45.776074427 (211h11m46.603782809s) [upload=219598]: Got next segment. Inline: false Dec 14 10:39:45.776158162 (211h11m46.603866553s) [upload=219598]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 14 10:39:45.776165965 (211h11m46.603874362s) [upload=219598]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 14 10:39:45.801257283 (211h11m46.628965678s) [seg_pos={0 0}, upload=219598]: Begin upload segment... Dec 14 10:39:45.801373021 (211h11m46.629081422s) [noise=true , address=145.79.227.43:4159 , piece_id=ULRDDEYCZXA26UPSFZQRQWFPU7MUCGJJSM7YWIR7HPXDCPBOQXNA, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.801409865 (211h11m46.629118247s) [seg_pos={0 0}, upload=219598]: Done begin upload segment. Dec 14 10:39:45.801481233 (211h11m46.629189635s) [noise=true , address=145.79.227.30:4763 , piece_id=CLMX6DHN5HP3OWMVR4R5MIVRH3GHCELUVTYJHBE7HXZQMHRQESIQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.802050295 (211h11m46.629758695s) [noise=true , address=145.79.227.32:4836 , piece_id=IQRH4CRVBOG72M3BE3G5FTIEMBTLD2KFUEYU6NCTK3N6PUONMY5A, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.802573060 (211h11m46.630281451s) [noise=true , address=77.90.31.14:4661 , piece_id=LLI4CDLWJ32TFXQ2YEU3NRP4QIRSNNUODA5RRPWU5AA45MJEW2LQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.803112997 (211h11m46.630821425s) [noise=true , address=145.79.227.44:4194 , piece_id=ZWOCXZ746BWDZ2LWYHSQHSG2O3WJC4R6FDL5E3JP2SXRCX3AWT2Q, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.803148596 (211h11m46.630856991s) [noise=true , address=145.79.227.46:4603 , piece_id=3X2DPA75MR7RX6LCOO2U3UREG4QSYHRRKRE57JUYUBINXTJN5IEA, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.803615052 (211h11m46.631323455s) [noise=true , address=145.79.227.28:4727 , piece_id=42ILJYV7IJRXSV3E3RWXWNUNYQHHQFNOF2JZGZKDEGPXRXZ3H2YA, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.803722621 (211h11m46.631431034s) [noise=true , address=77.90.31.16:4735 , piece_id=XKJJK6K7XF3TOFWMIU2BUVQTESTNV4FRYINSXJS4MC7KK7ZJU7SQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.804120053 (211h11m46.631828444s) [noise=true , address=145.79.227.7:4293 , piece_id=56YULKIWDM4P4WZM6KGMPUK3B3AK3RRVF4H6RJ7H6IQISPLXGBCQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.804570407 (211h11m46.632278798s) [noise=true , address=145.79.227.47:4267 , piece_id=VY46BNDUYVGBXDRIOEFK4USNOOJH2KXZH5Z2NHCT6SDJC5SFUYMQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.805083798 (211h11m46.632792196s) [noise=true , address=145.79.227.50:4375 , piece_id=RKVBJHHIF3KJLKB6ULTA4E5SHOBPCLPMWJCA6YTKF5XDNVEZMYXQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.805547276 (211h11m46.633255677s) [noise=true , address=77.90.31.16:4734 , piece_id=4JO2RBACNY5AM3KG7XUW7JTRGDCXGQDXSZ3L23ASAEJ4YFCICVAQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.805916642 (211h11m46.633625023s) [noise=true , address=145.79.227.39:4014 , piece_id=WKYZGQX5SP22ML66ZQZIHBS324EONJ7AARTYAXK2SCV577ZCQ5OQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.806440130 (211h11m46.634148525s) [noise=true , address=77.90.31.13:4626 , piece_id=DCKUBRB3DR755GN67RLP6NUMREAWIB3AITGPIMA7MM35BRPZW7IA, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.806976882 (211h11m46.634685284s) [noise=true , address=77.90.31.10:4519 , piece_id=ACPHBL7LF6NCNNH4U5GUXKW7RCL6XIEQVWYDCYUTXEC5RQE7RZBA, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.807729329 (211h11m46.63543772s) [noise=true , address=77.90.31.10:4518 , piece_id=ZE3SFGBCM2QV3CGGS3VEYT7NKDTDE2OJ6T6CVSGGYACCFX5WN5ZQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.841457783 (211h11m46.66916618s) [noise=true , address=77.90.31.13:4626 , piece_id=DCKUBRB3DR755GN67RLP6NUMREAWIB3AITGPIMA7MM35BRPZW7IA, seg_pos={0 0}, upload=219598]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: storage node overloaded, request limit: 2000 Dec 14 10:39:45.850206751 (211h11m46.677915151s) [noise=true , address=77.90.31.16:4734 , piece_id=4JO2RBACNY5AM3KG7XUW7JTRGDCXGQDXSZ3L23ASAEJ4YFCICVAQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: storage node overloaded, request limit: 2000 Dec 14 10:39:45.859644247 (211h11m46.68735265s) [noise=true , address=145.79.227.30:4763 , piece_id=CLMX6DHN5HP3OWMVR4R5MIVRH3GHCELUVTYJHBE7HXZQMHRQESIQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: storage node overloaded, request limit: 2000 Dec 14 10:39:45.874109133 (211h11m46.701817525s) [upload=219598]: Next returned nil segment Dec 14 10:39:45.874123719 (211h11m46.701832113s) [upload=219598]: Waiting for error group managing segments... Dec 14 10:39:45.880971826 (211h11m46.708680217s) [noise=true , address=77.90.31.10:4518 , piece_id=ZE3SFGBCM2QV3CGGS3VEYT7NKDTDE2OJ6T6CVSGGYACCFX5WN5ZQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.882649818 (211h11m46.710358212s) [noise=true , address=77.90.31.10:4519 , piece_id=ACPHBL7LF6NCNNH4U5GUXKW7RCL6XIEQVWYDCYUTXEC5RQE7RZBA, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.883088160 (211h11m46.71079655s) [noise=true , address=77.90.31.14:4661 , piece_id=LLI4CDLWJ32TFXQ2YEU3NRP4QIRSNNUODA5RRPWU5AA45MJEW2LQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.883693953 (211h11m46.711402345s) [noise=true , address=145.79.227.39:4014 , piece_id=WKYZGQX5SP22ML66ZQZIHBS324EONJ7AARTYAXK2SCV577ZCQ5OQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.884112448 (211h11m46.711820839s) [noise=true , address=145.79.227.32:4836 , piece_id=IQRH4CRVBOG72M3BE3G5FTIEMBTLD2KFUEYU6NCTK3N6PUONMY5A, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.884619811 (211h11m46.712328205s) [noise=true , address=145.79.227.43:4159 , piece_id=ULRDDEYCZXA26UPSFZQRQWFPU7MUCGJJSM7YWIR7HPXDCPBOQXNA, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.885012115 (211h11m46.712720506s) [noise=true , address=145.79.227.47:4267 , piece_id=VY46BNDUYVGBXDRIOEFK4USNOOJH2KXZH5Z2NHCT6SDJC5SFUYMQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.887669218 (211h11m46.71537761s) [noise=true , address=145.79.227.50:4375 , piece_id=RKVBJHHIF3KJLKB6ULTA4E5SHOBPCLPMWJCA6YTKF5XDNVEZMYXQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.890935473 (211h11m46.718643851s) [noise=true , address=145.79.227.46:4603 , piece_id=3X2DPA75MR7RX6LCOO2U3UREG4QSYHRRKRE57JUYUBINXTJN5IEA, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.891096730 (211h11m46.718805121s) [noise=true , address=145.79.227.28:4727 , piece_id=42ILJYV7IJRXSV3E3RWXWNUNYQHHQFNOF2JZGZKDEGPXRXZ3H2YA, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.893061640 (211h11m46.720770031s) [noise=true , address=145.79.227.7:4293 , piece_id=56YULKIWDM4P4WZM6KGMPUK3B3AK3RRVF4H6RJ7H6IQISPLXGBCQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:48.683585502 (211h11m49.511293883s) [noise=true , address=145.79.227.44:4194 , piece_id=ZWOCXZ746BWDZ2LWYHSQHSG2O3WJC4R6FDL5E3JP2SXRCX3AWT2Q, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:40:15.740699026 (211h12m16.568407426s) [noise=true , address=77.90.31.16:4735 , piece_id=XKJJK6K7XF3TOFWMIU2BUVQTESTNV4FRYINSXJS4MC7KK7ZJU7SQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: expected piece hash; context deadline exceeded; EOF Dec 14 10:40:15.740740404 (211h12m16.568448788s) [seg_pos={0 0}, upload=219598]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 14 10:40:15.762950810 (211h12m16.590659203s) [upload=219598]: Done uploading segments... Dec 14 10:40:15.762987936 (211h12m16.59069633s) [upload=219598]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 423.710964ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........X..................................................] duration: 537.600521ms trace: https://jaeger.dev.storj.io/trace/25fec5475908806a FAIL (2025-12-14 11:14:59 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 14 10:39:45.775689210 (211h11m46.603397602s) [upload=219598]: Starting upload Dec 14 10:39:45.775750055 (211h11m46.603458455s) [upload=219598]: Done starting upload Dec 14 10:39:45.775957558 (211h11m46.603665941s) [upload=219598]: Uploading segments... Dec 14 10:39:45.776074427 (211h11m46.603782809s) [upload=219598]: Got next segment. Inline: false Dec 14 10:39:45.776158162 (211h11m46.603866553s) [upload=219598]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 14 10:39:45.776165965 (211h11m46.603874362s) [upload=219598]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 14 10:39:45.801257283 (211h11m46.628965678s) [seg_pos={0 0}, upload=219598]: Begin upload segment... Dec 14 10:39:45.801373021 (211h11m46.629081422s) [noise=true , address=145.79.227.43:4159 , piece_id=ULRDDEYCZXA26UPSFZQRQWFPU7MUCGJJSM7YWIR7HPXDCPBOQXNA, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.801409865 (211h11m46.629118247s) [seg_pos={0 0}, upload=219598]: Done begin upload segment. Dec 14 10:39:45.801481233 (211h11m46.629189635s) [noise=true , address=145.79.227.30:4763 , piece_id=CLMX6DHN5HP3OWMVR4R5MIVRH3GHCELUVTYJHBE7HXZQMHRQESIQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.802050295 (211h11m46.629758695s) [noise=true , address=145.79.227.32:4836 , piece_id=IQRH4CRVBOG72M3BE3G5FTIEMBTLD2KFUEYU6NCTK3N6PUONMY5A, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.802573060 (211h11m46.630281451s) [noise=true , address=77.90.31.14:4661 , piece_id=LLI4CDLWJ32TFXQ2YEU3NRP4QIRSNNUODA5RRPWU5AA45MJEW2LQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.803112997 (211h11m46.630821425s) [noise=true , address=145.79.227.44:4194 , piece_id=ZWOCXZ746BWDZ2LWYHSQHSG2O3WJC4R6FDL5E3JP2SXRCX3AWT2Q, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.803148596 (211h11m46.630856991s) [noise=true , address=145.79.227.46:4603 , piece_id=3X2DPA75MR7RX6LCOO2U3UREG4QSYHRRKRE57JUYUBINXTJN5IEA, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.803615052 (211h11m46.631323455s) [noise=true , address=145.79.227.28:4727 , piece_id=42ILJYV7IJRXSV3E3RWXWNUNYQHHQFNOF2JZGZKDEGPXRXZ3H2YA, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.803722621 (211h11m46.631431034s) [noise=true , address=77.90.31.16:4735 , piece_id=XKJJK6K7XF3TOFWMIU2BUVQTESTNV4FRYINSXJS4MC7KK7ZJU7SQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.804120053 (211h11m46.631828444s) [noise=true , address=145.79.227.7:4293 , piece_id=56YULKIWDM4P4WZM6KGMPUK3B3AK3RRVF4H6RJ7H6IQISPLXGBCQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.804570407 (211h11m46.632278798s) [noise=true , address=145.79.227.47:4267 , piece_id=VY46BNDUYVGBXDRIOEFK4USNOOJH2KXZH5Z2NHCT6SDJC5SFUYMQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.805083798 (211h11m46.632792196s) [noise=true , address=145.79.227.50:4375 , piece_id=RKVBJHHIF3KJLKB6ULTA4E5SHOBPCLPMWJCA6YTKF5XDNVEZMYXQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.805547276 (211h11m46.633255677s) [noise=true , address=77.90.31.16:4734 , piece_id=4JO2RBACNY5AM3KG7XUW7JTRGDCXGQDXSZ3L23ASAEJ4YFCICVAQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.805916642 (211h11m46.633625023s) [noise=true , address=145.79.227.39:4014 , piece_id=WKYZGQX5SP22ML66ZQZIHBS324EONJ7AARTYAXK2SCV577ZCQ5OQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.806440130 (211h11m46.634148525s) [noise=true , address=77.90.31.13:4626 , piece_id=DCKUBRB3DR755GN67RLP6NUMREAWIB3AITGPIMA7MM35BRPZW7IA, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.806976882 (211h11m46.634685284s) [noise=true , address=77.90.31.10:4519 , piece_id=ACPHBL7LF6NCNNH4U5GUXKW7RCL6XIEQVWYDCYUTXEC5RQE7RZBA, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.807729329 (211h11m46.63543772s) [noise=true , address=77.90.31.10:4518 , piece_id=ZE3SFGBCM2QV3CGGS3VEYT7NKDTDE2OJ6T6CVSGGYACCFX5WN5ZQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.841457783 (211h11m46.66916618s) [noise=true , address=77.90.31.13:4626 , piece_id=DCKUBRB3DR755GN67RLP6NUMREAWIB3AITGPIMA7MM35BRPZW7IA, seg_pos={0 0}, upload=219598]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: storage node overloaded, request limit: 2000 Dec 14 10:39:45.850206751 (211h11m46.677915151s) [noise=true , address=77.90.31.16:4734 , piece_id=4JO2RBACNY5AM3KG7XUW7JTRGDCXGQDXSZ3L23ASAEJ4YFCICVAQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: storage node overloaded, request limit: 2000 Dec 14 10:39:45.859644247 (211h11m46.68735265s) [noise=true , address=145.79.227.30:4763 , piece_id=CLMX6DHN5HP3OWMVR4R5MIVRH3GHCELUVTYJHBE7HXZQMHRQESIQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: storage node overloaded, request limit: 2000 Dec 14 10:39:45.874109133 (211h11m46.701817525s) [upload=219598]: Next returned nil segment Dec 14 10:39:45.874123719 (211h11m46.701832113s) [upload=219598]: Waiting for error group managing segments... Dec 14 10:39:45.880971826 (211h11m46.708680217s) [noise=true , address=77.90.31.10:4518 , piece_id=ZE3SFGBCM2QV3CGGS3VEYT7NKDTDE2OJ6T6CVSGGYACCFX5WN5ZQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.882649818 (211h11m46.710358212s) [noise=true , address=77.90.31.10:4519 , piece_id=ACPHBL7LF6NCNNH4U5GUXKW7RCL6XIEQVWYDCYUTXEC5RQE7RZBA, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.883088160 (211h11m46.71079655s) [noise=true , address=77.90.31.14:4661 , piece_id=LLI4CDLWJ32TFXQ2YEU3NRP4QIRSNNUODA5RRPWU5AA45MJEW2LQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.883693953 (211h11m46.711402345s) [noise=true , address=145.79.227.39:4014 , piece_id=WKYZGQX5SP22ML66ZQZIHBS324EONJ7AARTYAXK2SCV577ZCQ5OQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.884112448 (211h11m46.711820839s) [noise=true , address=145.79.227.32:4836 , piece_id=IQRH4CRVBOG72M3BE3G5FTIEMBTLD2KFUEYU6NCTK3N6PUONMY5A, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.884619811 (211h11m46.712328205s) [noise=true , address=145.79.227.43:4159 , piece_id=ULRDDEYCZXA26UPSFZQRQWFPU7MUCGJJSM7YWIR7HPXDCPBOQXNA, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.885012115 (211h11m46.712720506s) [noise=true , address=145.79.227.47:4267 , piece_id=VY46BNDUYVGBXDRIOEFK4USNOOJH2KXZH5Z2NHCT6SDJC5SFUYMQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.887669218 (211h11m46.71537761s) [noise=true , address=145.79.227.50:4375 , piece_id=RKVBJHHIF3KJLKB6ULTA4E5SHOBPCLPMWJCA6YTKF5XDNVEZMYXQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.890935473 (211h11m46.718643851s) [noise=true , address=145.79.227.46:4603 , piece_id=3X2DPA75MR7RX6LCOO2U3UREG4QSYHRRKRE57JUYUBINXTJN5IEA, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.891096730 (211h11m46.718805121s) [noise=true , address=145.79.227.28:4727 , piece_id=42ILJYV7IJRXSV3E3RWXWNUNYQHHQFNOF2JZGZKDEGPXRXZ3H2YA, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.893061640 (211h11m46.720770031s) [noise=true , address=145.79.227.7:4293 , piece_id=56YULKIWDM4P4WZM6KGMPUK3B3AK3RRVF4H6RJ7H6IQISPLXGBCQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:48.683585502 (211h11m49.511293883s) [noise=true , address=145.79.227.44:4194 , piece_id=ZWOCXZ746BWDZ2LWYHSQHSG2O3WJC4R6FDL5E3JP2SXRCX3AWT2Q, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:40:15.740699026 (211h12m16.568407426s) [noise=true , address=77.90.31.16:4735 , piece_id=XKJJK6K7XF3TOFWMIU2BUVQTESTNV4FRYINSXJS4MC7KK7ZJU7SQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: expected piece hash; context deadline exceeded; EOF Dec 14 10:40:15.740740404 (211h12m16.568448788s) [seg_pos={0 0}, upload=219598]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 14 10:40:15.762950810 (211h12m16.590659203s) [upload=219598]: Done uploading segments... Dec 14 10:40:15.762987936 (211h12m16.59069633s) [upload=219598]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 537.631544ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........X.................................................] duration: 483.078848ms trace: https://jaeger.dev.storj.io/trace/25fec5475908806a FAIL (2025-12-14 11:13:51 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 14 10:39:45.775689210 (211h11m46.603397602s) [upload=219598]: Starting upload Dec 14 10:39:45.775750055 (211h11m46.603458455s) [upload=219598]: Done starting upload Dec 14 10:39:45.775957558 (211h11m46.603665941s) [upload=219598]: Uploading segments... Dec 14 10:39:45.776074427 (211h11m46.603782809s) [upload=219598]: Got next segment. Inline: false Dec 14 10:39:45.776158162 (211h11m46.603866553s) [upload=219598]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 14 10:39:45.776165965 (211h11m46.603874362s) [upload=219598]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 14 10:39:45.801257283 (211h11m46.628965678s) [seg_pos={0 0}, upload=219598]: Begin upload segment... Dec 14 10:39:45.801373021 (211h11m46.629081422s) [noise=true , address=145.79.227.43:4159 , piece_id=ULRDDEYCZXA26UPSFZQRQWFPU7MUCGJJSM7YWIR7HPXDCPBOQXNA, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.801409865 (211h11m46.629118247s) [seg_pos={0 0}, upload=219598]: Done begin upload segment. Dec 14 10:39:45.801481233 (211h11m46.629189635s) [noise=true , address=145.79.227.30:4763 , piece_id=CLMX6DHN5HP3OWMVR4R5MIVRH3GHCELUVTYJHBE7HXZQMHRQESIQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.802050295 (211h11m46.629758695s) [noise=true , address=145.79.227.32:4836 , piece_id=IQRH4CRVBOG72M3BE3G5FTIEMBTLD2KFUEYU6NCTK3N6PUONMY5A, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.802573060 (211h11m46.630281451s) [noise=true , address=77.90.31.14:4661 , piece_id=LLI4CDLWJ32TFXQ2YEU3NRP4QIRSNNUODA5RRPWU5AA45MJEW2LQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.803112997 (211h11m46.630821425s) [noise=true , address=145.79.227.44:4194 , piece_id=ZWOCXZ746BWDZ2LWYHSQHSG2O3WJC4R6FDL5E3JP2SXRCX3AWT2Q, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.803148596 (211h11m46.630856991s) [noise=true , address=145.79.227.46:4603 , piece_id=3X2DPA75MR7RX6LCOO2U3UREG4QSYHRRKRE57JUYUBINXTJN5IEA, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.803615052 (211h11m46.631323455s) [noise=true , address=145.79.227.28:4727 , piece_id=42ILJYV7IJRXSV3E3RWXWNUNYQHHQFNOF2JZGZKDEGPXRXZ3H2YA, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.803722621 (211h11m46.631431034s) [noise=true , address=77.90.31.16:4735 , piece_id=XKJJK6K7XF3TOFWMIU2BUVQTESTNV4FRYINSXJS4MC7KK7ZJU7SQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.804120053 (211h11m46.631828444s) [noise=true , address=145.79.227.7:4293 , piece_id=56YULKIWDM4P4WZM6KGMPUK3B3AK3RRVF4H6RJ7H6IQISPLXGBCQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.804570407 (211h11m46.632278798s) [noise=true , address=145.79.227.47:4267 , piece_id=VY46BNDUYVGBXDRIOEFK4USNOOJH2KXZH5Z2NHCT6SDJC5SFUYMQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.805083798 (211h11m46.632792196s) [noise=true , address=145.79.227.50:4375 , piece_id=RKVBJHHIF3KJLKB6ULTA4E5SHOBPCLPMWJCA6YTKF5XDNVEZMYXQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.805547276 (211h11m46.633255677s) [noise=true , address=77.90.31.16:4734 , piece_id=4JO2RBACNY5AM3KG7XUW7JTRGDCXGQDXSZ3L23ASAEJ4YFCICVAQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.805916642 (211h11m46.633625023s) [noise=true , address=145.79.227.39:4014 , piece_id=WKYZGQX5SP22ML66ZQZIHBS324EONJ7AARTYAXK2SCV577ZCQ5OQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.806440130 (211h11m46.634148525s) [noise=true , address=77.90.31.13:4626 , piece_id=DCKUBRB3DR755GN67RLP6NUMREAWIB3AITGPIMA7MM35BRPZW7IA, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.806976882 (211h11m46.634685284s) [noise=true , address=77.90.31.10:4519 , piece_id=ACPHBL7LF6NCNNH4U5GUXKW7RCL6XIEQVWYDCYUTXEC5RQE7RZBA, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.807729329 (211h11m46.63543772s) [noise=true , address=77.90.31.10:4518 , piece_id=ZE3SFGBCM2QV3CGGS3VEYT7NKDTDE2OJ6T6CVSGGYACCFX5WN5ZQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.841457783 (211h11m46.66916618s) [noise=true , address=77.90.31.13:4626 , piece_id=DCKUBRB3DR755GN67RLP6NUMREAWIB3AITGPIMA7MM35BRPZW7IA, seg_pos={0 0}, upload=219598]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: storage node overloaded, request limit: 2000 Dec 14 10:39:45.850206751 (211h11m46.677915151s) [noise=true , address=77.90.31.16:4734 , piece_id=4JO2RBACNY5AM3KG7XUW7JTRGDCXGQDXSZ3L23ASAEJ4YFCICVAQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: storage node overloaded, request limit: 2000 Dec 14 10:39:45.859644247 (211h11m46.68735265s) [noise=true , address=145.79.227.30:4763 , piece_id=CLMX6DHN5HP3OWMVR4R5MIVRH3GHCELUVTYJHBE7HXZQMHRQESIQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: storage node overloaded, request limit: 2000 Dec 14 10:39:45.874109133 (211h11m46.701817525s) [upload=219598]: Next returned nil segment Dec 14 10:39:45.874123719 (211h11m46.701832113s) [upload=219598]: Waiting for error group managing segments... Dec 14 10:39:45.880971826 (211h11m46.708680217s) [noise=true , address=77.90.31.10:4518 , piece_id=ZE3SFGBCM2QV3CGGS3VEYT7NKDTDE2OJ6T6CVSGGYACCFX5WN5ZQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.882649818 (211h11m46.710358212s) [noise=true , address=77.90.31.10:4519 , piece_id=ACPHBL7LF6NCNNH4U5GUXKW7RCL6XIEQVWYDCYUTXEC5RQE7RZBA, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.883088160 (211h11m46.71079655s) [noise=true , address=77.90.31.14:4661 , piece_id=LLI4CDLWJ32TFXQ2YEU3NRP4QIRSNNUODA5RRPWU5AA45MJEW2LQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.883693953 (211h11m46.711402345s) [noise=true , address=145.79.227.39:4014 , piece_id=WKYZGQX5SP22ML66ZQZIHBS324EONJ7AARTYAXK2SCV577ZCQ5OQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.884112448 (211h11m46.711820839s) [noise=true , address=145.79.227.32:4836 , piece_id=IQRH4CRVBOG72M3BE3G5FTIEMBTLD2KFUEYU6NCTK3N6PUONMY5A, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.884619811 (211h11m46.712328205s) [noise=true , address=145.79.227.43:4159 , piece_id=ULRDDEYCZXA26UPSFZQRQWFPU7MUCGJJSM7YWIR7HPXDCPBOQXNA, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.885012115 (211h11m46.712720506s) [noise=true , address=145.79.227.47:4267 , piece_id=VY46BNDUYVGBXDRIOEFK4USNOOJH2KXZH5Z2NHCT6SDJC5SFUYMQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.887669218 (211h11m46.71537761s) [noise=true , address=145.79.227.50:4375 , piece_id=RKVBJHHIF3KJLKB6ULTA4E5SHOBPCLPMWJCA6YTKF5XDNVEZMYXQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.890935473 (211h11m46.718643851s) [noise=true , address=145.79.227.46:4603 , piece_id=3X2DPA75MR7RX6LCOO2U3UREG4QSYHRRKRE57JUYUBINXTJN5IEA, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.891096730 (211h11m46.718805121s) [noise=true , address=145.79.227.28:4727 , piece_id=42ILJYV7IJRXSV3E3RWXWNUNYQHHQFNOF2JZGZKDEGPXRXZ3H2YA, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.893061640 (211h11m46.720770031s) [noise=true , address=145.79.227.7:4293 , piece_id=56YULKIWDM4P4WZM6KGMPUK3B3AK3RRVF4H6RJ7H6IQISPLXGBCQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:48.683585502 (211h11m49.511293883s) [noise=true , address=145.79.227.44:4194 , piece_id=ZWOCXZ746BWDZ2LWYHSQHSG2O3WJC4R6FDL5E3JP2SXRCX3AWT2Q, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:40:15.740699026 (211h12m16.568407426s) [noise=true , address=77.90.31.16:4735 , piece_id=XKJJK6K7XF3TOFWMIU2BUVQTESTNV4FRYINSXJS4MC7KK7ZJU7SQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: expected piece hash; context deadline exceeded; EOF Dec 14 10:40:15.740740404 (211h12m16.568448788s) [seg_pos={0 0}, upload=219598]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 14 10:40:15.762950810 (211h12m16.590659203s) [upload=219598]: Done uploading segments... Dec 14 10:40:15.762987936 (211h12m16.59069633s) [upload=219598]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 483.173738ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........X................................................] duration: 437.267517ms trace: https://jaeger.dev.storj.io/trace/25fec5475908806a FAIL (2025-12-14 11:13:22 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 14 10:39:45.775689210 (211h11m46.603397602s) [upload=219598]: Starting upload Dec 14 10:39:45.775750055 (211h11m46.603458455s) [upload=219598]: Done starting upload Dec 14 10:39:45.775957558 (211h11m46.603665941s) [upload=219598]: Uploading segments... Dec 14 10:39:45.776074427 (211h11m46.603782809s) [upload=219598]: Got next segment. Inline: false Dec 14 10:39:45.776158162 (211h11m46.603866553s) [upload=219598]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 14 10:39:45.776165965 (211h11m46.603874362s) [upload=219598]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 14 10:39:45.801257283 (211h11m46.628965678s) [seg_pos={0 0}, upload=219598]: Begin upload segment... Dec 14 10:39:45.801373021 (211h11m46.629081422s) [noise=true , address=145.79.227.43:4159 , piece_id=ULRDDEYCZXA26UPSFZQRQWFPU7MUCGJJSM7YWIR7HPXDCPBOQXNA, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.801409865 (211h11m46.629118247s) [seg_pos={0 0}, upload=219598]: Done begin upload segment. Dec 14 10:39:45.801481233 (211h11m46.629189635s) [noise=true , address=145.79.227.30:4763 , piece_id=CLMX6DHN5HP3OWMVR4R5MIVRH3GHCELUVTYJHBE7HXZQMHRQESIQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.802050295 (211h11m46.629758695s) [noise=true , address=145.79.227.32:4836 , piece_id=IQRH4CRVBOG72M3BE3G5FTIEMBTLD2KFUEYU6NCTK3N6PUONMY5A, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.802573060 (211h11m46.630281451s) [noise=true , address=77.90.31.14:4661 , piece_id=LLI4CDLWJ32TFXQ2YEU3NRP4QIRSNNUODA5RRPWU5AA45MJEW2LQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.803112997 (211h11m46.630821425s) [noise=true , address=145.79.227.44:4194 , piece_id=ZWOCXZ746BWDZ2LWYHSQHSG2O3WJC4R6FDL5E3JP2SXRCX3AWT2Q, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.803148596 (211h11m46.630856991s) [noise=true , address=145.79.227.46:4603 , piece_id=3X2DPA75MR7RX6LCOO2U3UREG4QSYHRRKRE57JUYUBINXTJN5IEA, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.803615052 (211h11m46.631323455s) [noise=true , address=145.79.227.28:4727 , piece_id=42ILJYV7IJRXSV3E3RWXWNUNYQHHQFNOF2JZGZKDEGPXRXZ3H2YA, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.803722621 (211h11m46.631431034s) [noise=true , address=77.90.31.16:4735 , piece_id=XKJJK6K7XF3TOFWMIU2BUVQTESTNV4FRYINSXJS4MC7KK7ZJU7SQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.804120053 (211h11m46.631828444s) [noise=true , address=145.79.227.7:4293 , piece_id=56YULKIWDM4P4WZM6KGMPUK3B3AK3RRVF4H6RJ7H6IQISPLXGBCQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.804570407 (211h11m46.632278798s) [noise=true , address=145.79.227.47:4267 , piece_id=VY46BNDUYVGBXDRIOEFK4USNOOJH2KXZH5Z2NHCT6SDJC5SFUYMQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.805083798 (211h11m46.632792196s) [noise=true , address=145.79.227.50:4375 , piece_id=RKVBJHHIF3KJLKB6ULTA4E5SHOBPCLPMWJCA6YTKF5XDNVEZMYXQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.805547276 (211h11m46.633255677s) [noise=true , address=77.90.31.16:4734 , piece_id=4JO2RBACNY5AM3KG7XUW7JTRGDCXGQDXSZ3L23ASAEJ4YFCICVAQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.805916642 (211h11m46.633625023s) [noise=true , address=145.79.227.39:4014 , piece_id=WKYZGQX5SP22ML66ZQZIHBS324EONJ7AARTYAXK2SCV577ZCQ5OQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.806440130 (211h11m46.634148525s) [noise=true , address=77.90.31.13:4626 , piece_id=DCKUBRB3DR755GN67RLP6NUMREAWIB3AITGPIMA7MM35BRPZW7IA, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.806976882 (211h11m46.634685284s) [noise=true , address=77.90.31.10:4519 , piece_id=ACPHBL7LF6NCNNH4U5GUXKW7RCL6XIEQVWYDCYUTXEC5RQE7RZBA, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.807729329 (211h11m46.63543772s) [noise=true , address=77.90.31.10:4518 , piece_id=ZE3SFGBCM2QV3CGGS3VEYT7NKDTDE2OJ6T6CVSGGYACCFX5WN5ZQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.841457783 (211h11m46.66916618s) [noise=true , address=77.90.31.13:4626 , piece_id=DCKUBRB3DR755GN67RLP6NUMREAWIB3AITGPIMA7MM35BRPZW7IA, seg_pos={0 0}, upload=219598]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: storage node overloaded, request limit: 2000 Dec 14 10:39:45.850206751 (211h11m46.677915151s) [noise=true , address=77.90.31.16:4734 , piece_id=4JO2RBACNY5AM3KG7XUW7JTRGDCXGQDXSZ3L23ASAEJ4YFCICVAQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: storage node overloaded, request limit: 2000 Dec 14 10:39:45.859644247 (211h11m46.68735265s) [noise=true , address=145.79.227.30:4763 , piece_id=CLMX6DHN5HP3OWMVR4R5MIVRH3GHCELUVTYJHBE7HXZQMHRQESIQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: storage node overloaded, request limit: 2000 Dec 14 10:39:45.874109133 (211h11m46.701817525s) [upload=219598]: Next returned nil segment Dec 14 10:39:45.874123719 (211h11m46.701832113s) [upload=219598]: Waiting for error group managing segments... Dec 14 10:39:45.880971826 (211h11m46.708680217s) [noise=true , address=77.90.31.10:4518 , piece_id=ZE3SFGBCM2QV3CGGS3VEYT7NKDTDE2OJ6T6CVSGGYACCFX5WN5ZQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.882649818 (211h11m46.710358212s) [noise=true , address=77.90.31.10:4519 , piece_id=ACPHBL7LF6NCNNH4U5GUXKW7RCL6XIEQVWYDCYUTXEC5RQE7RZBA, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.883088160 (211h11m46.71079655s) [noise=true , address=77.90.31.14:4661 , piece_id=LLI4CDLWJ32TFXQ2YEU3NRP4QIRSNNUODA5RRPWU5AA45MJEW2LQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.883693953 (211h11m46.711402345s) [noise=true , address=145.79.227.39:4014 , piece_id=WKYZGQX5SP22ML66ZQZIHBS324EONJ7AARTYAXK2SCV577ZCQ5OQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.884112448 (211h11m46.711820839s) [noise=true , address=145.79.227.32:4836 , piece_id=IQRH4CRVBOG72M3BE3G5FTIEMBTLD2KFUEYU6NCTK3N6PUONMY5A, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.884619811 (211h11m46.712328205s) [noise=true , address=145.79.227.43:4159 , piece_id=ULRDDEYCZXA26UPSFZQRQWFPU7MUCGJJSM7YWIR7HPXDCPBOQXNA, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.885012115 (211h11m46.712720506s) [noise=true , address=145.79.227.47:4267 , piece_id=VY46BNDUYVGBXDRIOEFK4USNOOJH2KXZH5Z2NHCT6SDJC5SFUYMQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.887669218 (211h11m46.71537761s) [noise=true , address=145.79.227.50:4375 , piece_id=RKVBJHHIF3KJLKB6ULTA4E5SHOBPCLPMWJCA6YTKF5XDNVEZMYXQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.890935473 (211h11m46.718643851s) [noise=true , address=145.79.227.46:4603 , piece_id=3X2DPA75MR7RX6LCOO2U3UREG4QSYHRRKRE57JUYUBINXTJN5IEA, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.891096730 (211h11m46.718805121s) [noise=true , address=145.79.227.28:4727 , piece_id=42ILJYV7IJRXSV3E3RWXWNUNYQHHQFNOF2JZGZKDEGPXRXZ3H2YA, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.893061640 (211h11m46.720770031s) [noise=true , address=145.79.227.7:4293 , piece_id=56YULKIWDM4P4WZM6KGMPUK3B3AK3RRVF4H6RJ7H6IQISPLXGBCQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:48.683585502 (211h11m49.511293883s) [noise=true , address=145.79.227.44:4194 , piece_id=ZWOCXZ746BWDZ2LWYHSQHSG2O3WJC4R6FDL5E3JP2SXRCX3AWT2Q, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:40:15.740699026 (211h12m16.568407426s) [noise=true , address=77.90.31.16:4735 , piece_id=XKJJK6K7XF3TOFWMIU2BUVQTESTNV4FRYINSXJS4MC7KK7ZJU7SQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: expected piece hash; context deadline exceeded; EOF Dec 14 10:40:15.740740404 (211h12m16.568448788s) [seg_pos={0 0}, upload=219598]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 14 10:40:15.762950810 (211h12m16.590659203s) [upload=219598]: Done uploading segments... Dec 14 10:40:15.762987936 (211h12m16.59069633s) [upload=219598]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 437.289441ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............X...............................................] duration: 454.391543ms trace: https://jaeger.dev.storj.io/trace/25fec5475908806a FAIL (2025-12-14 11:12:35 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 14 10:39:45.775689210 (211h11m46.603397602s) [upload=219598]: Starting upload Dec 14 10:39:45.775750055 (211h11m46.603458455s) [upload=219598]: Done starting upload Dec 14 10:39:45.775957558 (211h11m46.603665941s) [upload=219598]: Uploading segments... Dec 14 10:39:45.776074427 (211h11m46.603782809s) [upload=219598]: Got next segment. Inline: false Dec 14 10:39:45.776158162 (211h11m46.603866553s) [upload=219598]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 14 10:39:45.776165965 (211h11m46.603874362s) [upload=219598]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 14 10:39:45.801257283 (211h11m46.628965678s) [seg_pos={0 0}, upload=219598]: Begin upload segment... Dec 14 10:39:45.801373021 (211h11m46.629081422s) [noise=true , address=145.79.227.43:4159 , piece_id=ULRDDEYCZXA26UPSFZQRQWFPU7MUCGJJSM7YWIR7HPXDCPBOQXNA, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.801409865 (211h11m46.629118247s) [seg_pos={0 0}, upload=219598]: Done begin upload segment. Dec 14 10:39:45.801481233 (211h11m46.629189635s) [noise=true , address=145.79.227.30:4763 , piece_id=CLMX6DHN5HP3OWMVR4R5MIVRH3GHCELUVTYJHBE7HXZQMHRQESIQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.802050295 (211h11m46.629758695s) [noise=true , address=145.79.227.32:4836 , piece_id=IQRH4CRVBOG72M3BE3G5FTIEMBTLD2KFUEYU6NCTK3N6PUONMY5A, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.802573060 (211h11m46.630281451s) [noise=true , address=77.90.31.14:4661 , piece_id=LLI4CDLWJ32TFXQ2YEU3NRP4QIRSNNUODA5RRPWU5AA45MJEW2LQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.803112997 (211h11m46.630821425s) [noise=true , address=145.79.227.44:4194 , piece_id=ZWOCXZ746BWDZ2LWYHSQHSG2O3WJC4R6FDL5E3JP2SXRCX3AWT2Q, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.803148596 (211h11m46.630856991s) [noise=true , address=145.79.227.46:4603 , piece_id=3X2DPA75MR7RX6LCOO2U3UREG4QSYHRRKRE57JUYUBINXTJN5IEA, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.803615052 (211h11m46.631323455s) [noise=true , address=145.79.227.28:4727 , piece_id=42ILJYV7IJRXSV3E3RWXWNUNYQHHQFNOF2JZGZKDEGPXRXZ3H2YA, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.803722621 (211h11m46.631431034s) [noise=true , address=77.90.31.16:4735 , piece_id=XKJJK6K7XF3TOFWMIU2BUVQTESTNV4FRYINSXJS4MC7KK7ZJU7SQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.804120053 (211h11m46.631828444s) [noise=true , address=145.79.227.7:4293 , piece_id=56YULKIWDM4P4WZM6KGMPUK3B3AK3RRVF4H6RJ7H6IQISPLXGBCQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.804570407 (211h11m46.632278798s) [noise=true , address=145.79.227.47:4267 , piece_id=VY46BNDUYVGBXDRIOEFK4USNOOJH2KXZH5Z2NHCT6SDJC5SFUYMQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.805083798 (211h11m46.632792196s) [noise=true , address=145.79.227.50:4375 , piece_id=RKVBJHHIF3KJLKB6ULTA4E5SHOBPCLPMWJCA6YTKF5XDNVEZMYXQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.805547276 (211h11m46.633255677s) [noise=true , address=77.90.31.16:4734 , piece_id=4JO2RBACNY5AM3KG7XUW7JTRGDCXGQDXSZ3L23ASAEJ4YFCICVAQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.805916642 (211h11m46.633625023s) [noise=true , address=145.79.227.39:4014 , piece_id=WKYZGQX5SP22ML66ZQZIHBS324EONJ7AARTYAXK2SCV577ZCQ5OQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.806440130 (211h11m46.634148525s) [noise=true , address=77.90.31.13:4626 , piece_id=DCKUBRB3DR755GN67RLP6NUMREAWIB3AITGPIMA7MM35BRPZW7IA, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.806976882 (211h11m46.634685284s) [noise=true , address=77.90.31.10:4519 , piece_id=ACPHBL7LF6NCNNH4U5GUXKW7RCL6XIEQVWYDCYUTXEC5RQE7RZBA, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.807729329 (211h11m46.63543772s) [noise=true , address=77.90.31.10:4518 , piece_id=ZE3SFGBCM2QV3CGGS3VEYT7NKDTDE2OJ6T6CVSGGYACCFX5WN5ZQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.841457783 (211h11m46.66916618s) [noise=true , address=77.90.31.13:4626 , piece_id=DCKUBRB3DR755GN67RLP6NUMREAWIB3AITGPIMA7MM35BRPZW7IA, seg_pos={0 0}, upload=219598]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: storage node overloaded, request limit: 2000 Dec 14 10:39:45.850206751 (211h11m46.677915151s) [noise=true , address=77.90.31.16:4734 , piece_id=4JO2RBACNY5AM3KG7XUW7JTRGDCXGQDXSZ3L23ASAEJ4YFCICVAQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: storage node overloaded, request limit: 2000 Dec 14 10:39:45.859644247 (211h11m46.68735265s) [noise=true , address=145.79.227.30:4763 , piece_id=CLMX6DHN5HP3OWMVR4R5MIVRH3GHCELUVTYJHBE7HXZQMHRQESIQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: storage node overloaded, request limit: 2000 Dec 14 10:39:45.874109133 (211h11m46.701817525s) [upload=219598]: Next returned nil segment Dec 14 10:39:45.874123719 (211h11m46.701832113s) [upload=219598]: Waiting for error group managing segments... Dec 14 10:39:45.880971826 (211h11m46.708680217s) [noise=true , address=77.90.31.10:4518 , piece_id=ZE3SFGBCM2QV3CGGS3VEYT7NKDTDE2OJ6T6CVSGGYACCFX5WN5ZQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.882649818 (211h11m46.710358212s) [noise=true , address=77.90.31.10:4519 , piece_id=ACPHBL7LF6NCNNH4U5GUXKW7RCL6XIEQVWYDCYUTXEC5RQE7RZBA, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.883088160 (211h11m46.71079655s) [noise=true , address=77.90.31.14:4661 , piece_id=LLI4CDLWJ32TFXQ2YEU3NRP4QIRSNNUODA5RRPWU5AA45MJEW2LQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.883693953 (211h11m46.711402345s) [noise=true , address=145.79.227.39:4014 , piece_id=WKYZGQX5SP22ML66ZQZIHBS324EONJ7AARTYAXK2SCV577ZCQ5OQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.884112448 (211h11m46.711820839s) [noise=true , address=145.79.227.32:4836 , piece_id=IQRH4CRVBOG72M3BE3G5FTIEMBTLD2KFUEYU6NCTK3N6PUONMY5A, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.884619811 (211h11m46.712328205s) [noise=true , address=145.79.227.43:4159 , piece_id=ULRDDEYCZXA26UPSFZQRQWFPU7MUCGJJSM7YWIR7HPXDCPBOQXNA, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.885012115 (211h11m46.712720506s) [noise=true , address=145.79.227.47:4267 , piece_id=VY46BNDUYVGBXDRIOEFK4USNOOJH2KXZH5Z2NHCT6SDJC5SFUYMQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.887669218 (211h11m46.71537761s) [noise=true , address=145.79.227.50:4375 , piece_id=RKVBJHHIF3KJLKB6ULTA4E5SHOBPCLPMWJCA6YTKF5XDNVEZMYXQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.890935473 (211h11m46.718643851s) [noise=true , address=145.79.227.46:4603 , piece_id=3X2DPA75MR7RX6LCOO2U3UREG4QSYHRRKRE57JUYUBINXTJN5IEA, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.891096730 (211h11m46.718805121s) [noise=true , address=145.79.227.28:4727 , piece_id=42ILJYV7IJRXSV3E3RWXWNUNYQHHQFNOF2JZGZKDEGPXRXZ3H2YA, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.893061640 (211h11m46.720770031s) [noise=true , address=145.79.227.7:4293 , piece_id=56YULKIWDM4P4WZM6KGMPUK3B3AK3RRVF4H6RJ7H6IQISPLXGBCQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:48.683585502 (211h11m49.511293883s) [noise=true , address=145.79.227.44:4194 , piece_id=ZWOCXZ746BWDZ2LWYHSQHSG2O3WJC4R6FDL5E3JP2SXRCX3AWT2Q, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:40:15.740699026 (211h12m16.568407426s) [noise=true , address=77.90.31.16:4735 , piece_id=XKJJK6K7XF3TOFWMIU2BUVQTESTNV4FRYINSXJS4MC7KK7ZJU7SQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: expected piece hash; context deadline exceeded; EOF Dec 14 10:40:15.740740404 (211h12m16.568448788s) [seg_pos={0 0}, upload=219598]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 14 10:40:15.762950810 (211h12m16.590659203s) [upload=219598]: Done uploading segments... Dec 14 10:40:15.762987936 (211h12m16.59069633s) [upload=219598]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 454.425464ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............X..............................................] duration: 418.912122ms trace: https://jaeger.dev.storj.io/trace/25fec5475908806a FAIL (2025-12-14 11:11:21 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 14 10:39:45.775689210 (211h11m46.603397602s) [upload=219598]: Starting upload Dec 14 10:39:45.775750055 (211h11m46.603458455s) [upload=219598]: Done starting upload Dec 14 10:39:45.775957558 (211h11m46.603665941s) [upload=219598]: Uploading segments... Dec 14 10:39:45.776074427 (211h11m46.603782809s) [upload=219598]: Got next segment. Inline: false Dec 14 10:39:45.776158162 (211h11m46.603866553s) [upload=219598]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 14 10:39:45.776165965 (211h11m46.603874362s) [upload=219598]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 14 10:39:45.801257283 (211h11m46.628965678s) [seg_pos={0 0}, upload=219598]: Begin upload segment... Dec 14 10:39:45.801373021 (211h11m46.629081422s) [noise=true , address=145.79.227.43:4159 , piece_id=ULRDDEYCZXA26UPSFZQRQWFPU7MUCGJJSM7YWIR7HPXDCPBOQXNA, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.801409865 (211h11m46.629118247s) [seg_pos={0 0}, upload=219598]: Done begin upload segment. Dec 14 10:39:45.801481233 (211h11m46.629189635s) [noise=true , address=145.79.227.30:4763 , piece_id=CLMX6DHN5HP3OWMVR4R5MIVRH3GHCELUVTYJHBE7HXZQMHRQESIQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.802050295 (211h11m46.629758695s) [noise=true , address=145.79.227.32:4836 , piece_id=IQRH4CRVBOG72M3BE3G5FTIEMBTLD2KFUEYU6NCTK3N6PUONMY5A, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.802573060 (211h11m46.630281451s) [noise=true , address=77.90.31.14:4661 , piece_id=LLI4CDLWJ32TFXQ2YEU3NRP4QIRSNNUODA5RRPWU5AA45MJEW2LQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.803112997 (211h11m46.630821425s) [noise=true , address=145.79.227.44:4194 , piece_id=ZWOCXZ746BWDZ2LWYHSQHSG2O3WJC4R6FDL5E3JP2SXRCX3AWT2Q, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.803148596 (211h11m46.630856991s) [noise=true , address=145.79.227.46:4603 , piece_id=3X2DPA75MR7RX6LCOO2U3UREG4QSYHRRKRE57JUYUBINXTJN5IEA, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.803615052 (211h11m46.631323455s) [noise=true , address=145.79.227.28:4727 , piece_id=42ILJYV7IJRXSV3E3RWXWNUNYQHHQFNOF2JZGZKDEGPXRXZ3H2YA, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.803722621 (211h11m46.631431034s) [noise=true , address=77.90.31.16:4735 , piece_id=XKJJK6K7XF3TOFWMIU2BUVQTESTNV4FRYINSXJS4MC7KK7ZJU7SQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.804120053 (211h11m46.631828444s) [noise=true , address=145.79.227.7:4293 , piece_id=56YULKIWDM4P4WZM6KGMPUK3B3AK3RRVF4H6RJ7H6IQISPLXGBCQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.804570407 (211h11m46.632278798s) [noise=true , address=145.79.227.47:4267 , piece_id=VY46BNDUYVGBXDRIOEFK4USNOOJH2KXZH5Z2NHCT6SDJC5SFUYMQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.805083798 (211h11m46.632792196s) [noise=true , address=145.79.227.50:4375 , piece_id=RKVBJHHIF3KJLKB6ULTA4E5SHOBPCLPMWJCA6YTKF5XDNVEZMYXQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.805547276 (211h11m46.633255677s) [noise=true , address=77.90.31.16:4734 , piece_id=4JO2RBACNY5AM3KG7XUW7JTRGDCXGQDXSZ3L23ASAEJ4YFCICVAQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.805916642 (211h11m46.633625023s) [noise=true , address=145.79.227.39:4014 , piece_id=WKYZGQX5SP22ML66ZQZIHBS324EONJ7AARTYAXK2SCV577ZCQ5OQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.806440130 (211h11m46.634148525s) [noise=true , address=77.90.31.13:4626 , piece_id=DCKUBRB3DR755GN67RLP6NUMREAWIB3AITGPIMA7MM35BRPZW7IA, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.806976882 (211h11m46.634685284s) [noise=true , address=77.90.31.10:4519 , piece_id=ACPHBL7LF6NCNNH4U5GUXKW7RCL6XIEQVWYDCYUTXEC5RQE7RZBA, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.807729329 (211h11m46.63543772s) [noise=true , address=77.90.31.10:4518 , piece_id=ZE3SFGBCM2QV3CGGS3VEYT7NKDTDE2OJ6T6CVSGGYACCFX5WN5ZQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.841457783 (211h11m46.66916618s) [noise=true , address=77.90.31.13:4626 , piece_id=DCKUBRB3DR755GN67RLP6NUMREAWIB3AITGPIMA7MM35BRPZW7IA, seg_pos={0 0}, upload=219598]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: storage node overloaded, request limit: 2000 Dec 14 10:39:45.850206751 (211h11m46.677915151s) [noise=true , address=77.90.31.16:4734 , piece_id=4JO2RBACNY5AM3KG7XUW7JTRGDCXGQDXSZ3L23ASAEJ4YFCICVAQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: storage node overloaded, request limit: 2000 Dec 14 10:39:45.859644247 (211h11m46.68735265s) [noise=true , address=145.79.227.30:4763 , piece_id=CLMX6DHN5HP3OWMVR4R5MIVRH3GHCELUVTYJHBE7HXZQMHRQESIQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: storage node overloaded, request limit: 2000 Dec 14 10:39:45.874109133 (211h11m46.701817525s) [upload=219598]: Next returned nil segment Dec 14 10:39:45.874123719 (211h11m46.701832113s) [upload=219598]: Waiting for error group managing segments... Dec 14 10:39:45.880971826 (211h11m46.708680217s) [noise=true , address=77.90.31.10:4518 , piece_id=ZE3SFGBCM2QV3CGGS3VEYT7NKDTDE2OJ6T6CVSGGYACCFX5WN5ZQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.882649818 (211h11m46.710358212s) [noise=true , address=77.90.31.10:4519 , piece_id=ACPHBL7LF6NCNNH4U5GUXKW7RCL6XIEQVWYDCYUTXEC5RQE7RZBA, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.883088160 (211h11m46.71079655s) [noise=true , address=77.90.31.14:4661 , piece_id=LLI4CDLWJ32TFXQ2YEU3NRP4QIRSNNUODA5RRPWU5AA45MJEW2LQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.883693953 (211h11m46.711402345s) [noise=true , address=145.79.227.39:4014 , piece_id=WKYZGQX5SP22ML66ZQZIHBS324EONJ7AARTYAXK2SCV577ZCQ5OQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.884112448 (211h11m46.711820839s) [noise=true , address=145.79.227.32:4836 , piece_id=IQRH4CRVBOG72M3BE3G5FTIEMBTLD2KFUEYU6NCTK3N6PUONMY5A, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.884619811 (211h11m46.712328205s) [noise=true , address=145.79.227.43:4159 , piece_id=ULRDDEYCZXA26UPSFZQRQWFPU7MUCGJJSM7YWIR7HPXDCPBOQXNA, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.885012115 (211h11m46.712720506s) [noise=true , address=145.79.227.47:4267 , piece_id=VY46BNDUYVGBXDRIOEFK4USNOOJH2KXZH5Z2NHCT6SDJC5SFUYMQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.887669218 (211h11m46.71537761s) [noise=true , address=145.79.227.50:4375 , piece_id=RKVBJHHIF3KJLKB6ULTA4E5SHOBPCLPMWJCA6YTKF5XDNVEZMYXQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.890935473 (211h11m46.718643851s) [noise=true , address=145.79.227.46:4603 , piece_id=3X2DPA75MR7RX6LCOO2U3UREG4QSYHRRKRE57JUYUBINXTJN5IEA, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.891096730 (211h11m46.718805121s) [noise=true , address=145.79.227.28:4727 , piece_id=42ILJYV7IJRXSV3E3RWXWNUNYQHHQFNOF2JZGZKDEGPXRXZ3H2YA, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.893061640 (211h11m46.720770031s) [noise=true , address=145.79.227.7:4293 , piece_id=56YULKIWDM4P4WZM6KGMPUK3B3AK3RRVF4H6RJ7H6IQISPLXGBCQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:48.683585502 (211h11m49.511293883s) [noise=true , address=145.79.227.44:4194 , piece_id=ZWOCXZ746BWDZ2LWYHSQHSG2O3WJC4R6FDL5E3JP2SXRCX3AWT2Q, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:40:15.740699026 (211h12m16.568407426s) [noise=true , address=77.90.31.16:4735 , piece_id=XKJJK6K7XF3TOFWMIU2BUVQTESTNV4FRYINSXJS4MC7KK7ZJU7SQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: expected piece hash; context deadline exceeded; EOF Dec 14 10:40:15.740740404 (211h12m16.568448788s) [seg_pos={0 0}, upload=219598]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 14 10:40:15.762950810 (211h12m16.590659203s) [upload=219598]: Done uploading segments... Dec 14 10:40:15.762987936 (211h12m16.59069633s) [upload=219598]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 418.938601ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............X.............................................] duration: 533.103116ms trace: https://jaeger.dev.storj.io/trace/25fec5475908806a FAIL (2025-12-14 11:11:04 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 14 10:39:45.775689210 (211h11m46.603397602s) [upload=219598]: Starting upload Dec 14 10:39:45.775750055 (211h11m46.603458455s) [upload=219598]: Done starting upload Dec 14 10:39:45.775957558 (211h11m46.603665941s) [upload=219598]: Uploading segments... Dec 14 10:39:45.776074427 (211h11m46.603782809s) [upload=219598]: Got next segment. Inline: false Dec 14 10:39:45.776158162 (211h11m46.603866553s) [upload=219598]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 14 10:39:45.776165965 (211h11m46.603874362s) [upload=219598]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 14 10:39:45.801257283 (211h11m46.628965678s) [seg_pos={0 0}, upload=219598]: Begin upload segment... Dec 14 10:39:45.801373021 (211h11m46.629081422s) [noise=true , address=145.79.227.43:4159 , piece_id=ULRDDEYCZXA26UPSFZQRQWFPU7MUCGJJSM7YWIR7HPXDCPBOQXNA, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.801409865 (211h11m46.629118247s) [seg_pos={0 0}, upload=219598]: Done begin upload segment. Dec 14 10:39:45.801481233 (211h11m46.629189635s) [noise=true , address=145.79.227.30:4763 , piece_id=CLMX6DHN5HP3OWMVR4R5MIVRH3GHCELUVTYJHBE7HXZQMHRQESIQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.802050295 (211h11m46.629758695s) [noise=true , address=145.79.227.32:4836 , piece_id=IQRH4CRVBOG72M3BE3G5FTIEMBTLD2KFUEYU6NCTK3N6PUONMY5A, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.802573060 (211h11m46.630281451s) [noise=true , address=77.90.31.14:4661 , piece_id=LLI4CDLWJ32TFXQ2YEU3NRP4QIRSNNUODA5RRPWU5AA45MJEW2LQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.803112997 (211h11m46.630821425s) [noise=true , address=145.79.227.44:4194 , piece_id=ZWOCXZ746BWDZ2LWYHSQHSG2O3WJC4R6FDL5E3JP2SXRCX3AWT2Q, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.803148596 (211h11m46.630856991s) [noise=true , address=145.79.227.46:4603 , piece_id=3X2DPA75MR7RX6LCOO2U3UREG4QSYHRRKRE57JUYUBINXTJN5IEA, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.803615052 (211h11m46.631323455s) [noise=true , address=145.79.227.28:4727 , piece_id=42ILJYV7IJRXSV3E3RWXWNUNYQHHQFNOF2JZGZKDEGPXRXZ3H2YA, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.803722621 (211h11m46.631431034s) [noise=true , address=77.90.31.16:4735 , piece_id=XKJJK6K7XF3TOFWMIU2BUVQTESTNV4FRYINSXJS4MC7KK7ZJU7SQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.804120053 (211h11m46.631828444s) [noise=true , address=145.79.227.7:4293 , piece_id=56YULKIWDM4P4WZM6KGMPUK3B3AK3RRVF4H6RJ7H6IQISPLXGBCQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.804570407 (211h11m46.632278798s) [noise=true , address=145.79.227.47:4267 , piece_id=VY46BNDUYVGBXDRIOEFK4USNOOJH2KXZH5Z2NHCT6SDJC5SFUYMQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.805083798 (211h11m46.632792196s) [noise=true , address=145.79.227.50:4375 , piece_id=RKVBJHHIF3KJLKB6ULTA4E5SHOBPCLPMWJCA6YTKF5XDNVEZMYXQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.805547276 (211h11m46.633255677s) [noise=true , address=77.90.31.16:4734 , piece_id=4JO2RBACNY5AM3KG7XUW7JTRGDCXGQDXSZ3L23ASAEJ4YFCICVAQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.805916642 (211h11m46.633625023s) [noise=true , address=145.79.227.39:4014 , piece_id=WKYZGQX5SP22ML66ZQZIHBS324EONJ7AARTYAXK2SCV577ZCQ5OQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.806440130 (211h11m46.634148525s) [noise=true , address=77.90.31.13:4626 , piece_id=DCKUBRB3DR755GN67RLP6NUMREAWIB3AITGPIMA7MM35BRPZW7IA, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.806976882 (211h11m46.634685284s) [noise=true , address=77.90.31.10:4519 , piece_id=ACPHBL7LF6NCNNH4U5GUXKW7RCL6XIEQVWYDCYUTXEC5RQE7RZBA, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.807729329 (211h11m46.63543772s) [noise=true , address=77.90.31.10:4518 , piece_id=ZE3SFGBCM2QV3CGGS3VEYT7NKDTDE2OJ6T6CVSGGYACCFX5WN5ZQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.841457783 (211h11m46.66916618s) [noise=true , address=77.90.31.13:4626 , piece_id=DCKUBRB3DR755GN67RLP6NUMREAWIB3AITGPIMA7MM35BRPZW7IA, seg_pos={0 0}, upload=219598]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: storage node overloaded, request limit: 2000 Dec 14 10:39:45.850206751 (211h11m46.677915151s) [noise=true , address=77.90.31.16:4734 , piece_id=4JO2RBACNY5AM3KG7XUW7JTRGDCXGQDXSZ3L23ASAEJ4YFCICVAQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: storage node overloaded, request limit: 2000 Dec 14 10:39:45.859644247 (211h11m46.68735265s) [noise=true , address=145.79.227.30:4763 , piece_id=CLMX6DHN5HP3OWMVR4R5MIVRH3GHCELUVTYJHBE7HXZQMHRQESIQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: storage node overloaded, request limit: 2000 Dec 14 10:39:45.874109133 (211h11m46.701817525s) [upload=219598]: Next returned nil segment Dec 14 10:39:45.874123719 (211h11m46.701832113s) [upload=219598]: Waiting for error group managing segments... Dec 14 10:39:45.880971826 (211h11m46.708680217s) [noise=true , address=77.90.31.10:4518 , piece_id=ZE3SFGBCM2QV3CGGS3VEYT7NKDTDE2OJ6T6CVSGGYACCFX5WN5ZQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.882649818 (211h11m46.710358212s) [noise=true , address=77.90.31.10:4519 , piece_id=ACPHBL7LF6NCNNH4U5GUXKW7RCL6XIEQVWYDCYUTXEC5RQE7RZBA, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.883088160 (211h11m46.71079655s) [noise=true , address=77.90.31.14:4661 , piece_id=LLI4CDLWJ32TFXQ2YEU3NRP4QIRSNNUODA5RRPWU5AA45MJEW2LQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.883693953 (211h11m46.711402345s) [noise=true , address=145.79.227.39:4014 , piece_id=WKYZGQX5SP22ML66ZQZIHBS324EONJ7AARTYAXK2SCV577ZCQ5OQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.884112448 (211h11m46.711820839s) [noise=true , address=145.79.227.32:4836 , piece_id=IQRH4CRVBOG72M3BE3G5FTIEMBTLD2KFUEYU6NCTK3N6PUONMY5A, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.884619811 (211h11m46.712328205s) [noise=true , address=145.79.227.43:4159 , piece_id=ULRDDEYCZXA26UPSFZQRQWFPU7MUCGJJSM7YWIR7HPXDCPBOQXNA, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.885012115 (211h11m46.712720506s) [noise=true , address=145.79.227.47:4267 , piece_id=VY46BNDUYVGBXDRIOEFK4USNOOJH2KXZH5Z2NHCT6SDJC5SFUYMQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.887669218 (211h11m46.71537761s) [noise=true , address=145.79.227.50:4375 , piece_id=RKVBJHHIF3KJLKB6ULTA4E5SHOBPCLPMWJCA6YTKF5XDNVEZMYXQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.890935473 (211h11m46.718643851s) [noise=true , address=145.79.227.46:4603 , piece_id=3X2DPA75MR7RX6LCOO2U3UREG4QSYHRRKRE57JUYUBINXTJN5IEA, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.891096730 (211h11m46.718805121s) [noise=true , address=145.79.227.28:4727 , piece_id=42ILJYV7IJRXSV3E3RWXWNUNYQHHQFNOF2JZGZKDEGPXRXZ3H2YA, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.893061640 (211h11m46.720770031s) [noise=true , address=145.79.227.7:4293 , piece_id=56YULKIWDM4P4WZM6KGMPUK3B3AK3RRVF4H6RJ7H6IQISPLXGBCQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:48.683585502 (211h11m49.511293883s) [noise=true , address=145.79.227.44:4194 , piece_id=ZWOCXZ746BWDZ2LWYHSQHSG2O3WJC4R6FDL5E3JP2SXRCX3AWT2Q, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:40:15.740699026 (211h12m16.568407426s) [noise=true , address=77.90.31.16:4735 , piece_id=XKJJK6K7XF3TOFWMIU2BUVQTESTNV4FRYINSXJS4MC7KK7ZJU7SQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: expected piece hash; context deadline exceeded; EOF Dec 14 10:40:15.740740404 (211h12m16.568448788s) [seg_pos={0 0}, upload=219598]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 14 10:40:15.762950810 (211h12m16.590659203s) [upload=219598]: Done uploading segments... Dec 14 10:40:15.762987936 (211h12m16.59069633s) [upload=219598]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 533.128731ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............X............................................] duration: 406.427207ms trace: https://jaeger.dev.storj.io/trace/25fec5475908806a FAIL (2025-12-14 11:09:52 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 14 10:39:45.775689210 (211h11m46.603397602s) [upload=219598]: Starting upload Dec 14 10:39:45.775750055 (211h11m46.603458455s) [upload=219598]: Done starting upload Dec 14 10:39:45.775957558 (211h11m46.603665941s) [upload=219598]: Uploading segments... Dec 14 10:39:45.776074427 (211h11m46.603782809s) [upload=219598]: Got next segment. Inline: false Dec 14 10:39:45.776158162 (211h11m46.603866553s) [upload=219598]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 14 10:39:45.776165965 (211h11m46.603874362s) [upload=219598]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 14 10:39:45.801257283 (211h11m46.628965678s) [seg_pos={0 0}, upload=219598]: Begin upload segment... Dec 14 10:39:45.801373021 (211h11m46.629081422s) [noise=true , address=145.79.227.43:4159 , piece_id=ULRDDEYCZXA26UPSFZQRQWFPU7MUCGJJSM7YWIR7HPXDCPBOQXNA, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.801409865 (211h11m46.629118247s) [seg_pos={0 0}, upload=219598]: Done begin upload segment. Dec 14 10:39:45.801481233 (211h11m46.629189635s) [noise=true , address=145.79.227.30:4763 , piece_id=CLMX6DHN5HP3OWMVR4R5MIVRH3GHCELUVTYJHBE7HXZQMHRQESIQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.802050295 (211h11m46.629758695s) [noise=true , address=145.79.227.32:4836 , piece_id=IQRH4CRVBOG72M3BE3G5FTIEMBTLD2KFUEYU6NCTK3N6PUONMY5A, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.802573060 (211h11m46.630281451s) [noise=true , address=77.90.31.14:4661 , piece_id=LLI4CDLWJ32TFXQ2YEU3NRP4QIRSNNUODA5RRPWU5AA45MJEW2LQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.803112997 (211h11m46.630821425s) [noise=true , address=145.79.227.44:4194 , piece_id=ZWOCXZ746BWDZ2LWYHSQHSG2O3WJC4R6FDL5E3JP2SXRCX3AWT2Q, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.803148596 (211h11m46.630856991s) [noise=true , address=145.79.227.46:4603 , piece_id=3X2DPA75MR7RX6LCOO2U3UREG4QSYHRRKRE57JUYUBINXTJN5IEA, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.803615052 (211h11m46.631323455s) [noise=true , address=145.79.227.28:4727 , piece_id=42ILJYV7IJRXSV3E3RWXWNUNYQHHQFNOF2JZGZKDEGPXRXZ3H2YA, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.803722621 (211h11m46.631431034s) [noise=true , address=77.90.31.16:4735 , piece_id=XKJJK6K7XF3TOFWMIU2BUVQTESTNV4FRYINSXJS4MC7KK7ZJU7SQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.804120053 (211h11m46.631828444s) [noise=true , address=145.79.227.7:4293 , piece_id=56YULKIWDM4P4WZM6KGMPUK3B3AK3RRVF4H6RJ7H6IQISPLXGBCQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.804570407 (211h11m46.632278798s) [noise=true , address=145.79.227.47:4267 , piece_id=VY46BNDUYVGBXDRIOEFK4USNOOJH2KXZH5Z2NHCT6SDJC5SFUYMQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.805083798 (211h11m46.632792196s) [noise=true , address=145.79.227.50:4375 , piece_id=RKVBJHHIF3KJLKB6ULTA4E5SHOBPCLPMWJCA6YTKF5XDNVEZMYXQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.805547276 (211h11m46.633255677s) [noise=true , address=77.90.31.16:4734 , piece_id=4JO2RBACNY5AM3KG7XUW7JTRGDCXGQDXSZ3L23ASAEJ4YFCICVAQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.805916642 (211h11m46.633625023s) [noise=true , address=145.79.227.39:4014 , piece_id=WKYZGQX5SP22ML66ZQZIHBS324EONJ7AARTYAXK2SCV577ZCQ5OQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.806440130 (211h11m46.634148525s) [noise=true , address=77.90.31.13:4626 , piece_id=DCKUBRB3DR755GN67RLP6NUMREAWIB3AITGPIMA7MM35BRPZW7IA, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.806976882 (211h11m46.634685284s) [noise=true , address=77.90.31.10:4519 , piece_id=ACPHBL7LF6NCNNH4U5GUXKW7RCL6XIEQVWYDCYUTXEC5RQE7RZBA, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.807729329 (211h11m46.63543772s) [noise=true , address=77.90.31.10:4518 , piece_id=ZE3SFGBCM2QV3CGGS3VEYT7NKDTDE2OJ6T6CVSGGYACCFX5WN5ZQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.841457783 (211h11m46.66916618s) [noise=true , address=77.90.31.13:4626 , piece_id=DCKUBRB3DR755GN67RLP6NUMREAWIB3AITGPIMA7MM35BRPZW7IA, seg_pos={0 0}, upload=219598]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: storage node overloaded, request limit: 2000 Dec 14 10:39:45.850206751 (211h11m46.677915151s) [noise=true , address=77.90.31.16:4734 , piece_id=4JO2RBACNY5AM3KG7XUW7JTRGDCXGQDXSZ3L23ASAEJ4YFCICVAQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: storage node overloaded, request limit: 2000 Dec 14 10:39:45.859644247 (211h11m46.68735265s) [noise=true , address=145.79.227.30:4763 , piece_id=CLMX6DHN5HP3OWMVR4R5MIVRH3GHCELUVTYJHBE7HXZQMHRQESIQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: storage node overloaded, request limit: 2000 Dec 14 10:39:45.874109133 (211h11m46.701817525s) [upload=219598]: Next returned nil segment Dec 14 10:39:45.874123719 (211h11m46.701832113s) [upload=219598]: Waiting for error group managing segments... Dec 14 10:39:45.880971826 (211h11m46.708680217s) [noise=true , address=77.90.31.10:4518 , piece_id=ZE3SFGBCM2QV3CGGS3VEYT7NKDTDE2OJ6T6CVSGGYACCFX5WN5ZQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.882649818 (211h11m46.710358212s) [noise=true , address=77.90.31.10:4519 , piece_id=ACPHBL7LF6NCNNH4U5GUXKW7RCL6XIEQVWYDCYUTXEC5RQE7RZBA, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.883088160 (211h11m46.71079655s) [noise=true , address=77.90.31.14:4661 , piece_id=LLI4CDLWJ32TFXQ2YEU3NRP4QIRSNNUODA5RRPWU5AA45MJEW2LQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.883693953 (211h11m46.711402345s) [noise=true , address=145.79.227.39:4014 , piece_id=WKYZGQX5SP22ML66ZQZIHBS324EONJ7AARTYAXK2SCV577ZCQ5OQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.884112448 (211h11m46.711820839s) [noise=true , address=145.79.227.32:4836 , piece_id=IQRH4CRVBOG72M3BE3G5FTIEMBTLD2KFUEYU6NCTK3N6PUONMY5A, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.884619811 (211h11m46.712328205s) [noise=true , address=145.79.227.43:4159 , piece_id=ULRDDEYCZXA26UPSFZQRQWFPU7MUCGJJSM7YWIR7HPXDCPBOQXNA, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.885012115 (211h11m46.712720506s) [noise=true , address=145.79.227.47:4267 , piece_id=VY46BNDUYVGBXDRIOEFK4USNOOJH2KXZH5Z2NHCT6SDJC5SFUYMQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.887669218 (211h11m46.71537761s) [noise=true , address=145.79.227.50:4375 , piece_id=RKVBJHHIF3KJLKB6ULTA4E5SHOBPCLPMWJCA6YTKF5XDNVEZMYXQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.890935473 (211h11m46.718643851s) [noise=true , address=145.79.227.46:4603 , piece_id=3X2DPA75MR7RX6LCOO2U3UREG4QSYHRRKRE57JUYUBINXTJN5IEA, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.891096730 (211h11m46.718805121s) [noise=true , address=145.79.227.28:4727 , piece_id=42ILJYV7IJRXSV3E3RWXWNUNYQHHQFNOF2JZGZKDEGPXRXZ3H2YA, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.893061640 (211h11m46.720770031s) [noise=true , address=145.79.227.7:4293 , piece_id=56YULKIWDM4P4WZM6KGMPUK3B3AK3RRVF4H6RJ7H6IQISPLXGBCQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:48.683585502 (211h11m49.511293883s) [noise=true , address=145.79.227.44:4194 , piece_id=ZWOCXZ746BWDZ2LWYHSQHSG2O3WJC4R6FDL5E3JP2SXRCX3AWT2Q, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:40:15.740699026 (211h12m16.568407426s) [noise=true , address=77.90.31.16:4735 , piece_id=XKJJK6K7XF3TOFWMIU2BUVQTESTNV4FRYINSXJS4MC7KK7ZJU7SQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: expected piece hash; context deadline exceeded; EOF Dec 14 10:40:15.740740404 (211h12m16.568448788s) [seg_pos={0 0}, upload=219598]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 14 10:40:15.762950810 (211h12m16.590659203s) [upload=219598]: Done uploading segments... Dec 14 10:40:15.762987936 (211h12m16.59069633s) [upload=219598]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 406.459105ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................X...........................................] duration: 386.10147ms trace: https://jaeger.dev.storj.io/trace/25fec5475908806a FAIL (2025-12-14 11:09:33 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 14 10:39:45.775689210 (211h11m46.603397602s) [upload=219598]: Starting upload Dec 14 10:39:45.775750055 (211h11m46.603458455s) [upload=219598]: Done starting upload Dec 14 10:39:45.775957558 (211h11m46.603665941s) [upload=219598]: Uploading segments... Dec 14 10:39:45.776074427 (211h11m46.603782809s) [upload=219598]: Got next segment. Inline: false Dec 14 10:39:45.776158162 (211h11m46.603866553s) [upload=219598]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 14 10:39:45.776165965 (211h11m46.603874362s) [upload=219598]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 14 10:39:45.801257283 (211h11m46.628965678s) [seg_pos={0 0}, upload=219598]: Begin upload segment... Dec 14 10:39:45.801373021 (211h11m46.629081422s) [noise=true , address=145.79.227.43:4159 , piece_id=ULRDDEYCZXA26UPSFZQRQWFPU7MUCGJJSM7YWIR7HPXDCPBOQXNA, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.801409865 (211h11m46.629118247s) [seg_pos={0 0}, upload=219598]: Done begin upload segment. Dec 14 10:39:45.801481233 (211h11m46.629189635s) [noise=true , address=145.79.227.30:4763 , piece_id=CLMX6DHN5HP3OWMVR4R5MIVRH3GHCELUVTYJHBE7HXZQMHRQESIQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.802050295 (211h11m46.629758695s) [noise=true , address=145.79.227.32:4836 , piece_id=IQRH4CRVBOG72M3BE3G5FTIEMBTLD2KFUEYU6NCTK3N6PUONMY5A, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.802573060 (211h11m46.630281451s) [noise=true , address=77.90.31.14:4661 , piece_id=LLI4CDLWJ32TFXQ2YEU3NRP4QIRSNNUODA5RRPWU5AA45MJEW2LQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.803112997 (211h11m46.630821425s) [noise=true , address=145.79.227.44:4194 , piece_id=ZWOCXZ746BWDZ2LWYHSQHSG2O3WJC4R6FDL5E3JP2SXRCX3AWT2Q, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.803148596 (211h11m46.630856991s) [noise=true , address=145.79.227.46:4603 , piece_id=3X2DPA75MR7RX6LCOO2U3UREG4QSYHRRKRE57JUYUBINXTJN5IEA, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.803615052 (211h11m46.631323455s) [noise=true , address=145.79.227.28:4727 , piece_id=42ILJYV7IJRXSV3E3RWXWNUNYQHHQFNOF2JZGZKDEGPXRXZ3H2YA, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.803722621 (211h11m46.631431034s) [noise=true , address=77.90.31.16:4735 , piece_id=XKJJK6K7XF3TOFWMIU2BUVQTESTNV4FRYINSXJS4MC7KK7ZJU7SQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.804120053 (211h11m46.631828444s) [noise=true , address=145.79.227.7:4293 , piece_id=56YULKIWDM4P4WZM6KGMPUK3B3AK3RRVF4H6RJ7H6IQISPLXGBCQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.804570407 (211h11m46.632278798s) [noise=true , address=145.79.227.47:4267 , piece_id=VY46BNDUYVGBXDRIOEFK4USNOOJH2KXZH5Z2NHCT6SDJC5SFUYMQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.805083798 (211h11m46.632792196s) [noise=true , address=145.79.227.50:4375 , piece_id=RKVBJHHIF3KJLKB6ULTA4E5SHOBPCLPMWJCA6YTKF5XDNVEZMYXQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.805547276 (211h11m46.633255677s) [noise=true , address=77.90.31.16:4734 , piece_id=4JO2RBACNY5AM3KG7XUW7JTRGDCXGQDXSZ3L23ASAEJ4YFCICVAQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.805916642 (211h11m46.633625023s) [noise=true , address=145.79.227.39:4014 , piece_id=WKYZGQX5SP22ML66ZQZIHBS324EONJ7AARTYAXK2SCV577ZCQ5OQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.806440130 (211h11m46.634148525s) [noise=true , address=77.90.31.13:4626 , piece_id=DCKUBRB3DR755GN67RLP6NUMREAWIB3AITGPIMA7MM35BRPZW7IA, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.806976882 (211h11m46.634685284s) [noise=true , address=77.90.31.10:4519 , piece_id=ACPHBL7LF6NCNNH4U5GUXKW7RCL6XIEQVWYDCYUTXEC5RQE7RZBA, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.807729329 (211h11m46.63543772s) [noise=true , address=77.90.31.10:4518 , piece_id=ZE3SFGBCM2QV3CGGS3VEYT7NKDTDE2OJ6T6CVSGGYACCFX5WN5ZQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.841457783 (211h11m46.66916618s) [noise=true , address=77.90.31.13:4626 , piece_id=DCKUBRB3DR755GN67RLP6NUMREAWIB3AITGPIMA7MM35BRPZW7IA, seg_pos={0 0}, upload=219598]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: storage node overloaded, request limit: 2000 Dec 14 10:39:45.850206751 (211h11m46.677915151s) [noise=true , address=77.90.31.16:4734 , piece_id=4JO2RBACNY5AM3KG7XUW7JTRGDCXGQDXSZ3L23ASAEJ4YFCICVAQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: storage node overloaded, request limit: 2000 Dec 14 10:39:45.859644247 (211h11m46.68735265s) [noise=true , address=145.79.227.30:4763 , piece_id=CLMX6DHN5HP3OWMVR4R5MIVRH3GHCELUVTYJHBE7HXZQMHRQESIQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: storage node overloaded, request limit: 2000 Dec 14 10:39:45.874109133 (211h11m46.701817525s) [upload=219598]: Next returned nil segment Dec 14 10:39:45.874123719 (211h11m46.701832113s) [upload=219598]: Waiting for error group managing segments... Dec 14 10:39:45.880971826 (211h11m46.708680217s) [noise=true , address=77.90.31.10:4518 , piece_id=ZE3SFGBCM2QV3CGGS3VEYT7NKDTDE2OJ6T6CVSGGYACCFX5WN5ZQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.882649818 (211h11m46.710358212s) [noise=true , address=77.90.31.10:4519 , piece_id=ACPHBL7LF6NCNNH4U5GUXKW7RCL6XIEQVWYDCYUTXEC5RQE7RZBA, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.883088160 (211h11m46.71079655s) [noise=true , address=77.90.31.14:4661 , piece_id=LLI4CDLWJ32TFXQ2YEU3NRP4QIRSNNUODA5RRPWU5AA45MJEW2LQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.883693953 (211h11m46.711402345s) [noise=true , address=145.79.227.39:4014 , piece_id=WKYZGQX5SP22ML66ZQZIHBS324EONJ7AARTYAXK2SCV577ZCQ5OQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.884112448 (211h11m46.711820839s) [noise=true , address=145.79.227.32:4836 , piece_id=IQRH4CRVBOG72M3BE3G5FTIEMBTLD2KFUEYU6NCTK3N6PUONMY5A, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.884619811 (211h11m46.712328205s) [noise=true , address=145.79.227.43:4159 , piece_id=ULRDDEYCZXA26UPSFZQRQWFPU7MUCGJJSM7YWIR7HPXDCPBOQXNA, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.885012115 (211h11m46.712720506s) [noise=true , address=145.79.227.47:4267 , piece_id=VY46BNDUYVGBXDRIOEFK4USNOOJH2KXZH5Z2NHCT6SDJC5SFUYMQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.887669218 (211h11m46.71537761s) [noise=true , address=145.79.227.50:4375 , piece_id=RKVBJHHIF3KJLKB6ULTA4E5SHOBPCLPMWJCA6YTKF5XDNVEZMYXQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.890935473 (211h11m46.718643851s) [noise=true , address=145.79.227.46:4603 , piece_id=3X2DPA75MR7RX6LCOO2U3UREG4QSYHRRKRE57JUYUBINXTJN5IEA, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.891096730 (211h11m46.718805121s) [noise=true , address=145.79.227.28:4727 , piece_id=42ILJYV7IJRXSV3E3RWXWNUNYQHHQFNOF2JZGZKDEGPXRXZ3H2YA, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.893061640 (211h11m46.720770031s) [noise=true , address=145.79.227.7:4293 , piece_id=56YULKIWDM4P4WZM6KGMPUK3B3AK3RRVF4H6RJ7H6IQISPLXGBCQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:48.683585502 (211h11m49.511293883s) [noise=true , address=145.79.227.44:4194 , piece_id=ZWOCXZ746BWDZ2LWYHSQHSG2O3WJC4R6FDL5E3JP2SXRCX3AWT2Q, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:40:15.740699026 (211h12m16.568407426s) [noise=true , address=77.90.31.16:4735 , piece_id=XKJJK6K7XF3TOFWMIU2BUVQTESTNV4FRYINSXJS4MC7KK7ZJU7SQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: expected piece hash; context deadline exceeded; EOF Dec 14 10:40:15.740740404 (211h12m16.568448788s) [seg_pos={0 0}, upload=219598]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 14 10:40:15.762950810 (211h12m16.590659203s) [upload=219598]: Done uploading segments... Dec 14 10:40:15.762987936 (211h12m16.59069633s) [upload=219598]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 386.129017ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................X..........................................] duration: 661.615418ms trace: https://jaeger.dev.storj.io/trace/25fec5475908806a FAIL (2025-12-14 11:08:39 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 14 10:39:45.775689210 (211h11m46.603397602s) [upload=219598]: Starting upload Dec 14 10:39:45.775750055 (211h11m46.603458455s) [upload=219598]: Done starting upload Dec 14 10:39:45.775957558 (211h11m46.603665941s) [upload=219598]: Uploading segments... Dec 14 10:39:45.776074427 (211h11m46.603782809s) [upload=219598]: Got next segment. Inline: false Dec 14 10:39:45.776158162 (211h11m46.603866553s) [upload=219598]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 14 10:39:45.776165965 (211h11m46.603874362s) [upload=219598]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 14 10:39:45.801257283 (211h11m46.628965678s) [seg_pos={0 0}, upload=219598]: Begin upload segment... Dec 14 10:39:45.801373021 (211h11m46.629081422s) [noise=true , address=145.79.227.43:4159 , piece_id=ULRDDEYCZXA26UPSFZQRQWFPU7MUCGJJSM7YWIR7HPXDCPBOQXNA, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.801409865 (211h11m46.629118247s) [seg_pos={0 0}, upload=219598]: Done begin upload segment. Dec 14 10:39:45.801481233 (211h11m46.629189635s) [noise=true , address=145.79.227.30:4763 , piece_id=CLMX6DHN5HP3OWMVR4R5MIVRH3GHCELUVTYJHBE7HXZQMHRQESIQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.802050295 (211h11m46.629758695s) [noise=true , address=145.79.227.32:4836 , piece_id=IQRH4CRVBOG72M3BE3G5FTIEMBTLD2KFUEYU6NCTK3N6PUONMY5A, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.802573060 (211h11m46.630281451s) [noise=true , address=77.90.31.14:4661 , piece_id=LLI4CDLWJ32TFXQ2YEU3NRP4QIRSNNUODA5RRPWU5AA45MJEW2LQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.803112997 (211h11m46.630821425s) [noise=true , address=145.79.227.44:4194 , piece_id=ZWOCXZ746BWDZ2LWYHSQHSG2O3WJC4R6FDL5E3JP2SXRCX3AWT2Q, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.803148596 (211h11m46.630856991s) [noise=true , address=145.79.227.46:4603 , piece_id=3X2DPA75MR7RX6LCOO2U3UREG4QSYHRRKRE57JUYUBINXTJN5IEA, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.803615052 (211h11m46.631323455s) [noise=true , address=145.79.227.28:4727 , piece_id=42ILJYV7IJRXSV3E3RWXWNUNYQHHQFNOF2JZGZKDEGPXRXZ3H2YA, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.803722621 (211h11m46.631431034s) [noise=true , address=77.90.31.16:4735 , piece_id=XKJJK6K7XF3TOFWMIU2BUVQTESTNV4FRYINSXJS4MC7KK7ZJU7SQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.804120053 (211h11m46.631828444s) [noise=true , address=145.79.227.7:4293 , piece_id=56YULKIWDM4P4WZM6KGMPUK3B3AK3RRVF4H6RJ7H6IQISPLXGBCQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.804570407 (211h11m46.632278798s) [noise=true , address=145.79.227.47:4267 , piece_id=VY46BNDUYVGBXDRIOEFK4USNOOJH2KXZH5Z2NHCT6SDJC5SFUYMQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.805083798 (211h11m46.632792196s) [noise=true , address=145.79.227.50:4375 , piece_id=RKVBJHHIF3KJLKB6ULTA4E5SHOBPCLPMWJCA6YTKF5XDNVEZMYXQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.805547276 (211h11m46.633255677s) [noise=true , address=77.90.31.16:4734 , piece_id=4JO2RBACNY5AM3KG7XUW7JTRGDCXGQDXSZ3L23ASAEJ4YFCICVAQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.805916642 (211h11m46.633625023s) [noise=true , address=145.79.227.39:4014 , piece_id=WKYZGQX5SP22ML66ZQZIHBS324EONJ7AARTYAXK2SCV577ZCQ5OQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.806440130 (211h11m46.634148525s) [noise=true , address=77.90.31.13:4626 , piece_id=DCKUBRB3DR755GN67RLP6NUMREAWIB3AITGPIMA7MM35BRPZW7IA, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.806976882 (211h11m46.634685284s) [noise=true , address=77.90.31.10:4519 , piece_id=ACPHBL7LF6NCNNH4U5GUXKW7RCL6XIEQVWYDCYUTXEC5RQE7RZBA, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.807729329 (211h11m46.63543772s) [noise=true , address=77.90.31.10:4518 , piece_id=ZE3SFGBCM2QV3CGGS3VEYT7NKDTDE2OJ6T6CVSGGYACCFX5WN5ZQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.841457783 (211h11m46.66916618s) [noise=true , address=77.90.31.13:4626 , piece_id=DCKUBRB3DR755GN67RLP6NUMREAWIB3AITGPIMA7MM35BRPZW7IA, seg_pos={0 0}, upload=219598]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: storage node overloaded, request limit: 2000 Dec 14 10:39:45.850206751 (211h11m46.677915151s) [noise=true , address=77.90.31.16:4734 , piece_id=4JO2RBACNY5AM3KG7XUW7JTRGDCXGQDXSZ3L23ASAEJ4YFCICVAQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: storage node overloaded, request limit: 2000 Dec 14 10:39:45.859644247 (211h11m46.68735265s) [noise=true , address=145.79.227.30:4763 , piece_id=CLMX6DHN5HP3OWMVR4R5MIVRH3GHCELUVTYJHBE7HXZQMHRQESIQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: storage node overloaded, request limit: 2000 Dec 14 10:39:45.874109133 (211h11m46.701817525s) [upload=219598]: Next returned nil segment Dec 14 10:39:45.874123719 (211h11m46.701832113s) [upload=219598]: Waiting for error group managing segments... Dec 14 10:39:45.880971826 (211h11m46.708680217s) [noise=true , address=77.90.31.10:4518 , piece_id=ZE3SFGBCM2QV3CGGS3VEYT7NKDTDE2OJ6T6CVSGGYACCFX5WN5ZQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.882649818 (211h11m46.710358212s) [noise=true , address=77.90.31.10:4519 , piece_id=ACPHBL7LF6NCNNH4U5GUXKW7RCL6XIEQVWYDCYUTXEC5RQE7RZBA, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.883088160 (211h11m46.71079655s) [noise=true , address=77.90.31.14:4661 , piece_id=LLI4CDLWJ32TFXQ2YEU3NRP4QIRSNNUODA5RRPWU5AA45MJEW2LQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.883693953 (211h11m46.711402345s) [noise=true , address=145.79.227.39:4014 , piece_id=WKYZGQX5SP22ML66ZQZIHBS324EONJ7AARTYAXK2SCV577ZCQ5OQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.884112448 (211h11m46.711820839s) [noise=true , address=145.79.227.32:4836 , piece_id=IQRH4CRVBOG72M3BE3G5FTIEMBTLD2KFUEYU6NCTK3N6PUONMY5A, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.884619811 (211h11m46.712328205s) [noise=true , address=145.79.227.43:4159 , piece_id=ULRDDEYCZXA26UPSFZQRQWFPU7MUCGJJSM7YWIR7HPXDCPBOQXNA, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.885012115 (211h11m46.712720506s) [noise=true , address=145.79.227.47:4267 , piece_id=VY46BNDUYVGBXDRIOEFK4USNOOJH2KXZH5Z2NHCT6SDJC5SFUYMQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.887669218 (211h11m46.71537761s) [noise=true , address=145.79.227.50:4375 , piece_id=RKVBJHHIF3KJLKB6ULTA4E5SHOBPCLPMWJCA6YTKF5XDNVEZMYXQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.890935473 (211h11m46.718643851s) [noise=true , address=145.79.227.46:4603 , piece_id=3X2DPA75MR7RX6LCOO2U3UREG4QSYHRRKRE57JUYUBINXTJN5IEA, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.891096730 (211h11m46.718805121s) [noise=true , address=145.79.227.28:4727 , piece_id=42ILJYV7IJRXSV3E3RWXWNUNYQHHQFNOF2JZGZKDEGPXRXZ3H2YA, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.893061640 (211h11m46.720770031s) [noise=true , address=145.79.227.7:4293 , piece_id=56YULKIWDM4P4WZM6KGMPUK3B3AK3RRVF4H6RJ7H6IQISPLXGBCQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:48.683585502 (211h11m49.511293883s) [noise=true , address=145.79.227.44:4194 , piece_id=ZWOCXZ746BWDZ2LWYHSQHSG2O3WJC4R6FDL5E3JP2SXRCX3AWT2Q, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:40:15.740699026 (211h12m16.568407426s) [noise=true , address=77.90.31.16:4735 , piece_id=XKJJK6K7XF3TOFWMIU2BUVQTESTNV4FRYINSXJS4MC7KK7ZJU7SQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: expected piece hash; context deadline exceeded; EOF Dec 14 10:40:15.740740404 (211h12m16.568448788s) [seg_pos={0 0}, upload=219598]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 14 10:40:15.762950810 (211h12m16.590659203s) [upload=219598]: Done uploading segments... Dec 14 10:40:15.762987936 (211h12m16.59069633s) [upload=219598]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 661.641236ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................X.........................................] duration: 418.811536ms trace: https://jaeger.dev.storj.io/trace/25fec5475908806a FAIL (2025-12-14 11:08:19 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 14 10:39:45.775689210 (211h11m46.603397602s) [upload=219598]: Starting upload Dec 14 10:39:45.775750055 (211h11m46.603458455s) [upload=219598]: Done starting upload Dec 14 10:39:45.775957558 (211h11m46.603665941s) [upload=219598]: Uploading segments... Dec 14 10:39:45.776074427 (211h11m46.603782809s) [upload=219598]: Got next segment. Inline: false Dec 14 10:39:45.776158162 (211h11m46.603866553s) [upload=219598]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 14 10:39:45.776165965 (211h11m46.603874362s) [upload=219598]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 14 10:39:45.801257283 (211h11m46.628965678s) [seg_pos={0 0}, upload=219598]: Begin upload segment... Dec 14 10:39:45.801373021 (211h11m46.629081422s) [noise=true , address=145.79.227.43:4159 , piece_id=ULRDDEYCZXA26UPSFZQRQWFPU7MUCGJJSM7YWIR7HPXDCPBOQXNA, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.801409865 (211h11m46.629118247s) [seg_pos={0 0}, upload=219598]: Done begin upload segment. Dec 14 10:39:45.801481233 (211h11m46.629189635s) [noise=true , address=145.79.227.30:4763 , piece_id=CLMX6DHN5HP3OWMVR4R5MIVRH3GHCELUVTYJHBE7HXZQMHRQESIQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.802050295 (211h11m46.629758695s) [noise=true , address=145.79.227.32:4836 , piece_id=IQRH4CRVBOG72M3BE3G5FTIEMBTLD2KFUEYU6NCTK3N6PUONMY5A, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.802573060 (211h11m46.630281451s) [noise=true , address=77.90.31.14:4661 , piece_id=LLI4CDLWJ32TFXQ2YEU3NRP4QIRSNNUODA5RRPWU5AA45MJEW2LQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.803112997 (211h11m46.630821425s) [noise=true , address=145.79.227.44:4194 , piece_id=ZWOCXZ746BWDZ2LWYHSQHSG2O3WJC4R6FDL5E3JP2SXRCX3AWT2Q, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.803148596 (211h11m46.630856991s) [noise=true , address=145.79.227.46:4603 , piece_id=3X2DPA75MR7RX6LCOO2U3UREG4QSYHRRKRE57JUYUBINXTJN5IEA, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.803615052 (211h11m46.631323455s) [noise=true , address=145.79.227.28:4727 , piece_id=42ILJYV7IJRXSV3E3RWXWNUNYQHHQFNOF2JZGZKDEGPXRXZ3H2YA, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.803722621 (211h11m46.631431034s) [noise=true , address=77.90.31.16:4735 , piece_id=XKJJK6K7XF3TOFWMIU2BUVQTESTNV4FRYINSXJS4MC7KK7ZJU7SQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.804120053 (211h11m46.631828444s) [noise=true , address=145.79.227.7:4293 , piece_id=56YULKIWDM4P4WZM6KGMPUK3B3AK3RRVF4H6RJ7H6IQISPLXGBCQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.804570407 (211h11m46.632278798s) [noise=true , address=145.79.227.47:4267 , piece_id=VY46BNDUYVGBXDRIOEFK4USNOOJH2KXZH5Z2NHCT6SDJC5SFUYMQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.805083798 (211h11m46.632792196s) [noise=true , address=145.79.227.50:4375 , piece_id=RKVBJHHIF3KJLKB6ULTA4E5SHOBPCLPMWJCA6YTKF5XDNVEZMYXQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.805547276 (211h11m46.633255677s) [noise=true , address=77.90.31.16:4734 , piece_id=4JO2RBACNY5AM3KG7XUW7JTRGDCXGQDXSZ3L23ASAEJ4YFCICVAQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.805916642 (211h11m46.633625023s) [noise=true , address=145.79.227.39:4014 , piece_id=WKYZGQX5SP22ML66ZQZIHBS324EONJ7AARTYAXK2SCV577ZCQ5OQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.806440130 (211h11m46.634148525s) [noise=true , address=77.90.31.13:4626 , piece_id=DCKUBRB3DR755GN67RLP6NUMREAWIB3AITGPIMA7MM35BRPZW7IA, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.806976882 (211h11m46.634685284s) [noise=true , address=77.90.31.10:4519 , piece_id=ACPHBL7LF6NCNNH4U5GUXKW7RCL6XIEQVWYDCYUTXEC5RQE7RZBA, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.807729329 (211h11m46.63543772s) [noise=true , address=77.90.31.10:4518 , piece_id=ZE3SFGBCM2QV3CGGS3VEYT7NKDTDE2OJ6T6CVSGGYACCFX5WN5ZQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.841457783 (211h11m46.66916618s) [noise=true , address=77.90.31.13:4626 , piece_id=DCKUBRB3DR755GN67RLP6NUMREAWIB3AITGPIMA7MM35BRPZW7IA, seg_pos={0 0}, upload=219598]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: storage node overloaded, request limit: 2000 Dec 14 10:39:45.850206751 (211h11m46.677915151s) [noise=true , address=77.90.31.16:4734 , piece_id=4JO2RBACNY5AM3KG7XUW7JTRGDCXGQDXSZ3L23ASAEJ4YFCICVAQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: storage node overloaded, request limit: 2000 Dec 14 10:39:45.859644247 (211h11m46.68735265s) [noise=true , address=145.79.227.30:4763 , piece_id=CLMX6DHN5HP3OWMVR4R5MIVRH3GHCELUVTYJHBE7HXZQMHRQESIQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: storage node overloaded, request limit: 2000 Dec 14 10:39:45.874109133 (211h11m46.701817525s) [upload=219598]: Next returned nil segment Dec 14 10:39:45.874123719 (211h11m46.701832113s) [upload=219598]: Waiting for error group managing segments... Dec 14 10:39:45.880971826 (211h11m46.708680217s) [noise=true , address=77.90.31.10:4518 , piece_id=ZE3SFGBCM2QV3CGGS3VEYT7NKDTDE2OJ6T6CVSGGYACCFX5WN5ZQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.882649818 (211h11m46.710358212s) [noise=true , address=77.90.31.10:4519 , piece_id=ACPHBL7LF6NCNNH4U5GUXKW7RCL6XIEQVWYDCYUTXEC5RQE7RZBA, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.883088160 (211h11m46.71079655s) [noise=true , address=77.90.31.14:4661 , piece_id=LLI4CDLWJ32TFXQ2YEU3NRP4QIRSNNUODA5RRPWU5AA45MJEW2LQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.883693953 (211h11m46.711402345s) [noise=true , address=145.79.227.39:4014 , piece_id=WKYZGQX5SP22ML66ZQZIHBS324EONJ7AARTYAXK2SCV577ZCQ5OQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.884112448 (211h11m46.711820839s) [noise=true , address=145.79.227.32:4836 , piece_id=IQRH4CRVBOG72M3BE3G5FTIEMBTLD2KFUEYU6NCTK3N6PUONMY5A, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.884619811 (211h11m46.712328205s) [noise=true , address=145.79.227.43:4159 , piece_id=ULRDDEYCZXA26UPSFZQRQWFPU7MUCGJJSM7YWIR7HPXDCPBOQXNA, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.885012115 (211h11m46.712720506s) [noise=true , address=145.79.227.47:4267 , piece_id=VY46BNDUYVGBXDRIOEFK4USNOOJH2KXZH5Z2NHCT6SDJC5SFUYMQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.887669218 (211h11m46.71537761s) [noise=true , address=145.79.227.50:4375 , piece_id=RKVBJHHIF3KJLKB6ULTA4E5SHOBPCLPMWJCA6YTKF5XDNVEZMYXQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.890935473 (211h11m46.718643851s) [noise=true , address=145.79.227.46:4603 , piece_id=3X2DPA75MR7RX6LCOO2U3UREG4QSYHRRKRE57JUYUBINXTJN5IEA, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.891096730 (211h11m46.718805121s) [noise=true , address=145.79.227.28:4727 , piece_id=42ILJYV7IJRXSV3E3RWXWNUNYQHHQFNOF2JZGZKDEGPXRXZ3H2YA, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.893061640 (211h11m46.720770031s) [noise=true , address=145.79.227.7:4293 , piece_id=56YULKIWDM4P4WZM6KGMPUK3B3AK3RRVF4H6RJ7H6IQISPLXGBCQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:48.683585502 (211h11m49.511293883s) [noise=true , address=145.79.227.44:4194 , piece_id=ZWOCXZ746BWDZ2LWYHSQHSG2O3WJC4R6FDL5E3JP2SXRCX3AWT2Q, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:40:15.740699026 (211h12m16.568407426s) [noise=true , address=77.90.31.16:4735 , piece_id=XKJJK6K7XF3TOFWMIU2BUVQTESTNV4FRYINSXJS4MC7KK7ZJU7SQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: expected piece hash; context deadline exceeded; EOF Dec 14 10:40:15.740740404 (211h12m16.568448788s) [seg_pos={0 0}, upload=219598]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 14 10:40:15.762950810 (211h12m16.590659203s) [upload=219598]: Done uploading segments... Dec 14 10:40:15.762987936 (211h12m16.59069633s) [upload=219598]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 418.846936ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................X........................................] duration: 414.485418ms trace: https://jaeger.dev.storj.io/trace/25fec5475908806a FAIL (2025-12-14 11:07:28 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 14 10:39:45.775689210 (211h11m46.603397602s) [upload=219598]: Starting upload Dec 14 10:39:45.775750055 (211h11m46.603458455s) [upload=219598]: Done starting upload Dec 14 10:39:45.775957558 (211h11m46.603665941s) [upload=219598]: Uploading segments... Dec 14 10:39:45.776074427 (211h11m46.603782809s) [upload=219598]: Got next segment. Inline: false Dec 14 10:39:45.776158162 (211h11m46.603866553s) [upload=219598]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 14 10:39:45.776165965 (211h11m46.603874362s) [upload=219598]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 14 10:39:45.801257283 (211h11m46.628965678s) [seg_pos={0 0}, upload=219598]: Begin upload segment... Dec 14 10:39:45.801373021 (211h11m46.629081422s) [noise=true , address=145.79.227.43:4159 , piece_id=ULRDDEYCZXA26UPSFZQRQWFPU7MUCGJJSM7YWIR7HPXDCPBOQXNA, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.801409865 (211h11m46.629118247s) [seg_pos={0 0}, upload=219598]: Done begin upload segment. Dec 14 10:39:45.801481233 (211h11m46.629189635s) [noise=true , address=145.79.227.30:4763 , piece_id=CLMX6DHN5HP3OWMVR4R5MIVRH3GHCELUVTYJHBE7HXZQMHRQESIQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.802050295 (211h11m46.629758695s) [noise=true , address=145.79.227.32:4836 , piece_id=IQRH4CRVBOG72M3BE3G5FTIEMBTLD2KFUEYU6NCTK3N6PUONMY5A, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.802573060 (211h11m46.630281451s) [noise=true , address=77.90.31.14:4661 , piece_id=LLI4CDLWJ32TFXQ2YEU3NRP4QIRSNNUODA5RRPWU5AA45MJEW2LQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.803112997 (211h11m46.630821425s) [noise=true , address=145.79.227.44:4194 , piece_id=ZWOCXZ746BWDZ2LWYHSQHSG2O3WJC4R6FDL5E3JP2SXRCX3AWT2Q, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.803148596 (211h11m46.630856991s) [noise=true , address=145.79.227.46:4603 , piece_id=3X2DPA75MR7RX6LCOO2U3UREG4QSYHRRKRE57JUYUBINXTJN5IEA, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.803615052 (211h11m46.631323455s) [noise=true , address=145.79.227.28:4727 , piece_id=42ILJYV7IJRXSV3E3RWXWNUNYQHHQFNOF2JZGZKDEGPXRXZ3H2YA, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.803722621 (211h11m46.631431034s) [noise=true , address=77.90.31.16:4735 , piece_id=XKJJK6K7XF3TOFWMIU2BUVQTESTNV4FRYINSXJS4MC7KK7ZJU7SQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.804120053 (211h11m46.631828444s) [noise=true , address=145.79.227.7:4293 , piece_id=56YULKIWDM4P4WZM6KGMPUK3B3AK3RRVF4H6RJ7H6IQISPLXGBCQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.804570407 (211h11m46.632278798s) [noise=true , address=145.79.227.47:4267 , piece_id=VY46BNDUYVGBXDRIOEFK4USNOOJH2KXZH5Z2NHCT6SDJC5SFUYMQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.805083798 (211h11m46.632792196s) [noise=true , address=145.79.227.50:4375 , piece_id=RKVBJHHIF3KJLKB6ULTA4E5SHOBPCLPMWJCA6YTKF5XDNVEZMYXQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.805547276 (211h11m46.633255677s) [noise=true , address=77.90.31.16:4734 , piece_id=4JO2RBACNY5AM3KG7XUW7JTRGDCXGQDXSZ3L23ASAEJ4YFCICVAQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.805916642 (211h11m46.633625023s) [noise=true , address=145.79.227.39:4014 , piece_id=WKYZGQX5SP22ML66ZQZIHBS324EONJ7AARTYAXK2SCV577ZCQ5OQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.806440130 (211h11m46.634148525s) [noise=true , address=77.90.31.13:4626 , piece_id=DCKUBRB3DR755GN67RLP6NUMREAWIB3AITGPIMA7MM35BRPZW7IA, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.806976882 (211h11m46.634685284s) [noise=true , address=77.90.31.10:4519 , piece_id=ACPHBL7LF6NCNNH4U5GUXKW7RCL6XIEQVWYDCYUTXEC5RQE7RZBA, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.807729329 (211h11m46.63543772s) [noise=true , address=77.90.31.10:4518 , piece_id=ZE3SFGBCM2QV3CGGS3VEYT7NKDTDE2OJ6T6CVSGGYACCFX5WN5ZQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.841457783 (211h11m46.66916618s) [noise=true , address=77.90.31.13:4626 , piece_id=DCKUBRB3DR755GN67RLP6NUMREAWIB3AITGPIMA7MM35BRPZW7IA, seg_pos={0 0}, upload=219598]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: storage node overloaded, request limit: 2000 Dec 14 10:39:45.850206751 (211h11m46.677915151s) [noise=true , address=77.90.31.16:4734 , piece_id=4JO2RBACNY5AM3KG7XUW7JTRGDCXGQDXSZ3L23ASAEJ4YFCICVAQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: storage node overloaded, request limit: 2000 Dec 14 10:39:45.859644247 (211h11m46.68735265s) [noise=true , address=145.79.227.30:4763 , piece_id=CLMX6DHN5HP3OWMVR4R5MIVRH3GHCELUVTYJHBE7HXZQMHRQESIQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: storage node overloaded, request limit: 2000 Dec 14 10:39:45.874109133 (211h11m46.701817525s) [upload=219598]: Next returned nil segment Dec 14 10:39:45.874123719 (211h11m46.701832113s) [upload=219598]: Waiting for error group managing segments... Dec 14 10:39:45.880971826 (211h11m46.708680217s) [noise=true , address=77.90.31.10:4518 , piece_id=ZE3SFGBCM2QV3CGGS3VEYT7NKDTDE2OJ6T6CVSGGYACCFX5WN5ZQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.882649818 (211h11m46.710358212s) [noise=true , address=77.90.31.10:4519 , piece_id=ACPHBL7LF6NCNNH4U5GUXKW7RCL6XIEQVWYDCYUTXEC5RQE7RZBA, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.883088160 (211h11m46.71079655s) [noise=true , address=77.90.31.14:4661 , piece_id=LLI4CDLWJ32TFXQ2YEU3NRP4QIRSNNUODA5RRPWU5AA45MJEW2LQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.883693953 (211h11m46.711402345s) [noise=true , address=145.79.227.39:4014 , piece_id=WKYZGQX5SP22ML66ZQZIHBS324EONJ7AARTYAXK2SCV577ZCQ5OQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.884112448 (211h11m46.711820839s) [noise=true , address=145.79.227.32:4836 , piece_id=IQRH4CRVBOG72M3BE3G5FTIEMBTLD2KFUEYU6NCTK3N6PUONMY5A, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.884619811 (211h11m46.712328205s) [noise=true , address=145.79.227.43:4159 , piece_id=ULRDDEYCZXA26UPSFZQRQWFPU7MUCGJJSM7YWIR7HPXDCPBOQXNA, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.885012115 (211h11m46.712720506s) [noise=true , address=145.79.227.47:4267 , piece_id=VY46BNDUYVGBXDRIOEFK4USNOOJH2KXZH5Z2NHCT6SDJC5SFUYMQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.887669218 (211h11m46.71537761s) [noise=true , address=145.79.227.50:4375 , piece_id=RKVBJHHIF3KJLKB6ULTA4E5SHOBPCLPMWJCA6YTKF5XDNVEZMYXQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.890935473 (211h11m46.718643851s) [noise=true , address=145.79.227.46:4603 , piece_id=3X2DPA75MR7RX6LCOO2U3UREG4QSYHRRKRE57JUYUBINXTJN5IEA, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.891096730 (211h11m46.718805121s) [noise=true , address=145.79.227.28:4727 , piece_id=42ILJYV7IJRXSV3E3RWXWNUNYQHHQFNOF2JZGZKDEGPXRXZ3H2YA, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.893061640 (211h11m46.720770031s) [noise=true , address=145.79.227.7:4293 , piece_id=56YULKIWDM4P4WZM6KGMPUK3B3AK3RRVF4H6RJ7H6IQISPLXGBCQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:48.683585502 (211h11m49.511293883s) [noise=true , address=145.79.227.44:4194 , piece_id=ZWOCXZ746BWDZ2LWYHSQHSG2O3WJC4R6FDL5E3JP2SXRCX3AWT2Q, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:40:15.740699026 (211h12m16.568407426s) [noise=true , address=77.90.31.16:4735 , piece_id=XKJJK6K7XF3TOFWMIU2BUVQTESTNV4FRYINSXJS4MC7KK7ZJU7SQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: expected piece hash; context deadline exceeded; EOF Dec 14 10:40:15.740740404 (211h12m16.568448788s) [seg_pos={0 0}, upload=219598]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 14 10:40:15.762950810 (211h12m16.590659203s) [upload=219598]: Done uploading segments... Dec 14 10:40:15.762987936 (211h12m16.59069633s) [upload=219598]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 414.520981ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................X.......................................] duration: 535.620601ms trace: https://jaeger.dev.storj.io/trace/25fec5475908806a FAIL (2025-12-14 11:06:39 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 14 10:39:45.775689210 (211h11m46.603397602s) [upload=219598]: Starting upload Dec 14 10:39:45.775750055 (211h11m46.603458455s) [upload=219598]: Done starting upload Dec 14 10:39:45.775957558 (211h11m46.603665941s) [upload=219598]: Uploading segments... Dec 14 10:39:45.776074427 (211h11m46.603782809s) [upload=219598]: Got next segment. Inline: false Dec 14 10:39:45.776158162 (211h11m46.603866553s) [upload=219598]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 14 10:39:45.776165965 (211h11m46.603874362s) [upload=219598]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 14 10:39:45.801257283 (211h11m46.628965678s) [seg_pos={0 0}, upload=219598]: Begin upload segment... Dec 14 10:39:45.801373021 (211h11m46.629081422s) [noise=true , address=145.79.227.43:4159 , piece_id=ULRDDEYCZXA26UPSFZQRQWFPU7MUCGJJSM7YWIR7HPXDCPBOQXNA, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.801409865 (211h11m46.629118247s) [seg_pos={0 0}, upload=219598]: Done begin upload segment. Dec 14 10:39:45.801481233 (211h11m46.629189635s) [noise=true , address=145.79.227.30:4763 , piece_id=CLMX6DHN5HP3OWMVR4R5MIVRH3GHCELUVTYJHBE7HXZQMHRQESIQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.802050295 (211h11m46.629758695s) [noise=true , address=145.79.227.32:4836 , piece_id=IQRH4CRVBOG72M3BE3G5FTIEMBTLD2KFUEYU6NCTK3N6PUONMY5A, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.802573060 (211h11m46.630281451s) [noise=true , address=77.90.31.14:4661 , piece_id=LLI4CDLWJ32TFXQ2YEU3NRP4QIRSNNUODA5RRPWU5AA45MJEW2LQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.803112997 (211h11m46.630821425s) [noise=true , address=145.79.227.44:4194 , piece_id=ZWOCXZ746BWDZ2LWYHSQHSG2O3WJC4R6FDL5E3JP2SXRCX3AWT2Q, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.803148596 (211h11m46.630856991s) [noise=true , address=145.79.227.46:4603 , piece_id=3X2DPA75MR7RX6LCOO2U3UREG4QSYHRRKRE57JUYUBINXTJN5IEA, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.803615052 (211h11m46.631323455s) [noise=true , address=145.79.227.28:4727 , piece_id=42ILJYV7IJRXSV3E3RWXWNUNYQHHQFNOF2JZGZKDEGPXRXZ3H2YA, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.803722621 (211h11m46.631431034s) [noise=true , address=77.90.31.16:4735 , piece_id=XKJJK6K7XF3TOFWMIU2BUVQTESTNV4FRYINSXJS4MC7KK7ZJU7SQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.804120053 (211h11m46.631828444s) [noise=true , address=145.79.227.7:4293 , piece_id=56YULKIWDM4P4WZM6KGMPUK3B3AK3RRVF4H6RJ7H6IQISPLXGBCQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.804570407 (211h11m46.632278798s) [noise=true , address=145.79.227.47:4267 , piece_id=VY46BNDUYVGBXDRIOEFK4USNOOJH2KXZH5Z2NHCT6SDJC5SFUYMQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.805083798 (211h11m46.632792196s) [noise=true , address=145.79.227.50:4375 , piece_id=RKVBJHHIF3KJLKB6ULTA4E5SHOBPCLPMWJCA6YTKF5XDNVEZMYXQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.805547276 (211h11m46.633255677s) [noise=true , address=77.90.31.16:4734 , piece_id=4JO2RBACNY5AM3KG7XUW7JTRGDCXGQDXSZ3L23ASAEJ4YFCICVAQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.805916642 (211h11m46.633625023s) [noise=true , address=145.79.227.39:4014 , piece_id=WKYZGQX5SP22ML66ZQZIHBS324EONJ7AARTYAXK2SCV577ZCQ5OQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.806440130 (211h11m46.634148525s) [noise=true , address=77.90.31.13:4626 , piece_id=DCKUBRB3DR755GN67RLP6NUMREAWIB3AITGPIMA7MM35BRPZW7IA, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.806976882 (211h11m46.634685284s) [noise=true , address=77.90.31.10:4519 , piece_id=ACPHBL7LF6NCNNH4U5GUXKW7RCL6XIEQVWYDCYUTXEC5RQE7RZBA, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.807729329 (211h11m46.63543772s) [noise=true , address=77.90.31.10:4518 , piece_id=ZE3SFGBCM2QV3CGGS3VEYT7NKDTDE2OJ6T6CVSGGYACCFX5WN5ZQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.841457783 (211h11m46.66916618s) [noise=true , address=77.90.31.13:4626 , piece_id=DCKUBRB3DR755GN67RLP6NUMREAWIB3AITGPIMA7MM35BRPZW7IA, seg_pos={0 0}, upload=219598]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: storage node overloaded, request limit: 2000 Dec 14 10:39:45.850206751 (211h11m46.677915151s) [noise=true , address=77.90.31.16:4734 , piece_id=4JO2RBACNY5AM3KG7XUW7JTRGDCXGQDXSZ3L23ASAEJ4YFCICVAQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: storage node overloaded, request limit: 2000 Dec 14 10:39:45.859644247 (211h11m46.68735265s) [noise=true , address=145.79.227.30:4763 , piece_id=CLMX6DHN5HP3OWMVR4R5MIVRH3GHCELUVTYJHBE7HXZQMHRQESIQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: storage node overloaded, request limit: 2000 Dec 14 10:39:45.874109133 (211h11m46.701817525s) [upload=219598]: Next returned nil segment Dec 14 10:39:45.874123719 (211h11m46.701832113s) [upload=219598]: Waiting for error group managing segments... Dec 14 10:39:45.880971826 (211h11m46.708680217s) [noise=true , address=77.90.31.10:4518 , piece_id=ZE3SFGBCM2QV3CGGS3VEYT7NKDTDE2OJ6T6CVSGGYACCFX5WN5ZQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.882649818 (211h11m46.710358212s) [noise=true , address=77.90.31.10:4519 , piece_id=ACPHBL7LF6NCNNH4U5GUXKW7RCL6XIEQVWYDCYUTXEC5RQE7RZBA, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.883088160 (211h11m46.71079655s) [noise=true , address=77.90.31.14:4661 , piece_id=LLI4CDLWJ32TFXQ2YEU3NRP4QIRSNNUODA5RRPWU5AA45MJEW2LQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.883693953 (211h11m46.711402345s) [noise=true , address=145.79.227.39:4014 , piece_id=WKYZGQX5SP22ML66ZQZIHBS324EONJ7AARTYAXK2SCV577ZCQ5OQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.884112448 (211h11m46.711820839s) [noise=true , address=145.79.227.32:4836 , piece_id=IQRH4CRVBOG72M3BE3G5FTIEMBTLD2KFUEYU6NCTK3N6PUONMY5A, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.884619811 (211h11m46.712328205s) [noise=true , address=145.79.227.43:4159 , piece_id=ULRDDEYCZXA26UPSFZQRQWFPU7MUCGJJSM7YWIR7HPXDCPBOQXNA, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.885012115 (211h11m46.712720506s) [noise=true , address=145.79.227.47:4267 , piece_id=VY46BNDUYVGBXDRIOEFK4USNOOJH2KXZH5Z2NHCT6SDJC5SFUYMQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.887669218 (211h11m46.71537761s) [noise=true , address=145.79.227.50:4375 , piece_id=RKVBJHHIF3KJLKB6ULTA4E5SHOBPCLPMWJCA6YTKF5XDNVEZMYXQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.890935473 (211h11m46.718643851s) [noise=true , address=145.79.227.46:4603 , piece_id=3X2DPA75MR7RX6LCOO2U3UREG4QSYHRRKRE57JUYUBINXTJN5IEA, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.891096730 (211h11m46.718805121s) [noise=true , address=145.79.227.28:4727 , piece_id=42ILJYV7IJRXSV3E3RWXWNUNYQHHQFNOF2JZGZKDEGPXRXZ3H2YA, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.893061640 (211h11m46.720770031s) [noise=true , address=145.79.227.7:4293 , piece_id=56YULKIWDM4P4WZM6KGMPUK3B3AK3RRVF4H6RJ7H6IQISPLXGBCQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:48.683585502 (211h11m49.511293883s) [noise=true , address=145.79.227.44:4194 , piece_id=ZWOCXZ746BWDZ2LWYHSQHSG2O3WJC4R6FDL5E3JP2SXRCX3AWT2Q, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:40:15.740699026 (211h12m16.568407426s) [noise=true , address=77.90.31.16:4735 , piece_id=XKJJK6K7XF3TOFWMIU2BUVQTESTNV4FRYINSXJS4MC7KK7ZJU7SQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: expected piece hash; context deadline exceeded; EOF Dec 14 10:40:15.740740404 (211h12m16.568448788s) [seg_pos={0 0}, upload=219598]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 14 10:40:15.762950810 (211h12m16.590659203s) [upload=219598]: Done uploading segments... Dec 14 10:40:15.762987936 (211h12m16.59069633s) [upload=219598]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 535.649228ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................X......................................] duration: 598.764803ms trace: https://jaeger.dev.storj.io/trace/25fec5475908806a FAIL (2025-12-14 11:05:50 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 14 10:39:45.775689210 (211h11m46.603397602s) [upload=219598]: Starting upload Dec 14 10:39:45.775750055 (211h11m46.603458455s) [upload=219598]: Done starting upload Dec 14 10:39:45.775957558 (211h11m46.603665941s) [upload=219598]: Uploading segments... Dec 14 10:39:45.776074427 (211h11m46.603782809s) [upload=219598]: Got next segment. Inline: false Dec 14 10:39:45.776158162 (211h11m46.603866553s) [upload=219598]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 14 10:39:45.776165965 (211h11m46.603874362s) [upload=219598]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 14 10:39:45.801257283 (211h11m46.628965678s) [seg_pos={0 0}, upload=219598]: Begin upload segment... Dec 14 10:39:45.801373021 (211h11m46.629081422s) [noise=true , address=145.79.227.43:4159 , piece_id=ULRDDEYCZXA26UPSFZQRQWFPU7MUCGJJSM7YWIR7HPXDCPBOQXNA, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.801409865 (211h11m46.629118247s) [seg_pos={0 0}, upload=219598]: Done begin upload segment. Dec 14 10:39:45.801481233 (211h11m46.629189635s) [noise=true , address=145.79.227.30:4763 , piece_id=CLMX6DHN5HP3OWMVR4R5MIVRH3GHCELUVTYJHBE7HXZQMHRQESIQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.802050295 (211h11m46.629758695s) [noise=true , address=145.79.227.32:4836 , piece_id=IQRH4CRVBOG72M3BE3G5FTIEMBTLD2KFUEYU6NCTK3N6PUONMY5A, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.802573060 (211h11m46.630281451s) [noise=true , address=77.90.31.14:4661 , piece_id=LLI4CDLWJ32TFXQ2YEU3NRP4QIRSNNUODA5RRPWU5AA45MJEW2LQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.803112997 (211h11m46.630821425s) [noise=true , address=145.79.227.44:4194 , piece_id=ZWOCXZ746BWDZ2LWYHSQHSG2O3WJC4R6FDL5E3JP2SXRCX3AWT2Q, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.803148596 (211h11m46.630856991s) [noise=true , address=145.79.227.46:4603 , piece_id=3X2DPA75MR7RX6LCOO2U3UREG4QSYHRRKRE57JUYUBINXTJN5IEA, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.803615052 (211h11m46.631323455s) [noise=true , address=145.79.227.28:4727 , piece_id=42ILJYV7IJRXSV3E3RWXWNUNYQHHQFNOF2JZGZKDEGPXRXZ3H2YA, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.803722621 (211h11m46.631431034s) [noise=true , address=77.90.31.16:4735 , piece_id=XKJJK6K7XF3TOFWMIU2BUVQTESTNV4FRYINSXJS4MC7KK7ZJU7SQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.804120053 (211h11m46.631828444s) [noise=true , address=145.79.227.7:4293 , piece_id=56YULKIWDM4P4WZM6KGMPUK3B3AK3RRVF4H6RJ7H6IQISPLXGBCQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.804570407 (211h11m46.632278798s) [noise=true , address=145.79.227.47:4267 , piece_id=VY46BNDUYVGBXDRIOEFK4USNOOJH2KXZH5Z2NHCT6SDJC5SFUYMQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.805083798 (211h11m46.632792196s) [noise=true , address=145.79.227.50:4375 , piece_id=RKVBJHHIF3KJLKB6ULTA4E5SHOBPCLPMWJCA6YTKF5XDNVEZMYXQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.805547276 (211h11m46.633255677s) [noise=true , address=77.90.31.16:4734 , piece_id=4JO2RBACNY5AM3KG7XUW7JTRGDCXGQDXSZ3L23ASAEJ4YFCICVAQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.805916642 (211h11m46.633625023s) [noise=true , address=145.79.227.39:4014 , piece_id=WKYZGQX5SP22ML66ZQZIHBS324EONJ7AARTYAXK2SCV577ZCQ5OQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.806440130 (211h11m46.634148525s) [noise=true , address=77.90.31.13:4626 , piece_id=DCKUBRB3DR755GN67RLP6NUMREAWIB3AITGPIMA7MM35BRPZW7IA, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.806976882 (211h11m46.634685284s) [noise=true , address=77.90.31.10:4519 , piece_id=ACPHBL7LF6NCNNH4U5GUXKW7RCL6XIEQVWYDCYUTXEC5RQE7RZBA, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.807729329 (211h11m46.63543772s) [noise=true , address=77.90.31.10:4518 , piece_id=ZE3SFGBCM2QV3CGGS3VEYT7NKDTDE2OJ6T6CVSGGYACCFX5WN5ZQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.841457783 (211h11m46.66916618s) [noise=true , address=77.90.31.13:4626 , piece_id=DCKUBRB3DR755GN67RLP6NUMREAWIB3AITGPIMA7MM35BRPZW7IA, seg_pos={0 0}, upload=219598]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: storage node overloaded, request limit: 2000 Dec 14 10:39:45.850206751 (211h11m46.677915151s) [noise=true , address=77.90.31.16:4734 , piece_id=4JO2RBACNY5AM3KG7XUW7JTRGDCXGQDXSZ3L23ASAEJ4YFCICVAQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: storage node overloaded, request limit: 2000 Dec 14 10:39:45.859644247 (211h11m46.68735265s) [noise=true , address=145.79.227.30:4763 , piece_id=CLMX6DHN5HP3OWMVR4R5MIVRH3GHCELUVTYJHBE7HXZQMHRQESIQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: storage node overloaded, request limit: 2000 Dec 14 10:39:45.874109133 (211h11m46.701817525s) [upload=219598]: Next returned nil segment Dec 14 10:39:45.874123719 (211h11m46.701832113s) [upload=219598]: Waiting for error group managing segments... Dec 14 10:39:45.880971826 (211h11m46.708680217s) [noise=true , address=77.90.31.10:4518 , piece_id=ZE3SFGBCM2QV3CGGS3VEYT7NKDTDE2OJ6T6CVSGGYACCFX5WN5ZQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.882649818 (211h11m46.710358212s) [noise=true , address=77.90.31.10:4519 , piece_id=ACPHBL7LF6NCNNH4U5GUXKW7RCL6XIEQVWYDCYUTXEC5RQE7RZBA, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.883088160 (211h11m46.71079655s) [noise=true , address=77.90.31.14:4661 , piece_id=LLI4CDLWJ32TFXQ2YEU3NRP4QIRSNNUODA5RRPWU5AA45MJEW2LQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.883693953 (211h11m46.711402345s) [noise=true , address=145.79.227.39:4014 , piece_id=WKYZGQX5SP22ML66ZQZIHBS324EONJ7AARTYAXK2SCV577ZCQ5OQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.884112448 (211h11m46.711820839s) [noise=true , address=145.79.227.32:4836 , piece_id=IQRH4CRVBOG72M3BE3G5FTIEMBTLD2KFUEYU6NCTK3N6PUONMY5A, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.884619811 (211h11m46.712328205s) [noise=true , address=145.79.227.43:4159 , piece_id=ULRDDEYCZXA26UPSFZQRQWFPU7MUCGJJSM7YWIR7HPXDCPBOQXNA, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.885012115 (211h11m46.712720506s) [noise=true , address=145.79.227.47:4267 , piece_id=VY46BNDUYVGBXDRIOEFK4USNOOJH2KXZH5Z2NHCT6SDJC5SFUYMQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.887669218 (211h11m46.71537761s) [noise=true , address=145.79.227.50:4375 , piece_id=RKVBJHHIF3KJLKB6ULTA4E5SHOBPCLPMWJCA6YTKF5XDNVEZMYXQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.890935473 (211h11m46.718643851s) [noise=true , address=145.79.227.46:4603 , piece_id=3X2DPA75MR7RX6LCOO2U3UREG4QSYHRRKRE57JUYUBINXTJN5IEA, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.891096730 (211h11m46.718805121s) [noise=true , address=145.79.227.28:4727 , piece_id=42ILJYV7IJRXSV3E3RWXWNUNYQHHQFNOF2JZGZKDEGPXRXZ3H2YA, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.893061640 (211h11m46.720770031s) [noise=true , address=145.79.227.7:4293 , piece_id=56YULKIWDM4P4WZM6KGMPUK3B3AK3RRVF4H6RJ7H6IQISPLXGBCQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:48.683585502 (211h11m49.511293883s) [noise=true , address=145.79.227.44:4194 , piece_id=ZWOCXZ746BWDZ2LWYHSQHSG2O3WJC4R6FDL5E3JP2SXRCX3AWT2Q, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:40:15.740699026 (211h12m16.568407426s) [noise=true , address=77.90.31.16:4735 , piece_id=XKJJK6K7XF3TOFWMIU2BUVQTESTNV4FRYINSXJS4MC7KK7ZJU7SQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: expected piece hash; context deadline exceeded; EOF Dec 14 10:40:15.740740404 (211h12m16.568448788s) [seg_pos={0 0}, upload=219598]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 14 10:40:15.762950810 (211h12m16.590659203s) [upload=219598]: Done uploading segments... Dec 14 10:40:15.762987936 (211h12m16.59069633s) [upload=219598]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 598.790501ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................X.....................................] duration: 458.168456ms trace: https://jaeger.dev.storj.io/trace/25fec5475908806a FAIL (2025-12-14 11:05:24 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 14 10:39:45.775689210 (211h11m46.603397602s) [upload=219598]: Starting upload Dec 14 10:39:45.775750055 (211h11m46.603458455s) [upload=219598]: Done starting upload Dec 14 10:39:45.775957558 (211h11m46.603665941s) [upload=219598]: Uploading segments... Dec 14 10:39:45.776074427 (211h11m46.603782809s) [upload=219598]: Got next segment. Inline: false Dec 14 10:39:45.776158162 (211h11m46.603866553s) [upload=219598]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 14 10:39:45.776165965 (211h11m46.603874362s) [upload=219598]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 14 10:39:45.801257283 (211h11m46.628965678s) [seg_pos={0 0}, upload=219598]: Begin upload segment... Dec 14 10:39:45.801373021 (211h11m46.629081422s) [noise=true , address=145.79.227.43:4159 , piece_id=ULRDDEYCZXA26UPSFZQRQWFPU7MUCGJJSM7YWIR7HPXDCPBOQXNA, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.801409865 (211h11m46.629118247s) [seg_pos={0 0}, upload=219598]: Done begin upload segment. Dec 14 10:39:45.801481233 (211h11m46.629189635s) [noise=true , address=145.79.227.30:4763 , piece_id=CLMX6DHN5HP3OWMVR4R5MIVRH3GHCELUVTYJHBE7HXZQMHRQESIQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.802050295 (211h11m46.629758695s) [noise=true , address=145.79.227.32:4836 , piece_id=IQRH4CRVBOG72M3BE3G5FTIEMBTLD2KFUEYU6NCTK3N6PUONMY5A, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.802573060 (211h11m46.630281451s) [noise=true , address=77.90.31.14:4661 , piece_id=LLI4CDLWJ32TFXQ2YEU3NRP4QIRSNNUODA5RRPWU5AA45MJEW2LQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.803112997 (211h11m46.630821425s) [noise=true , address=145.79.227.44:4194 , piece_id=ZWOCXZ746BWDZ2LWYHSQHSG2O3WJC4R6FDL5E3JP2SXRCX3AWT2Q, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.803148596 (211h11m46.630856991s) [noise=true , address=145.79.227.46:4603 , piece_id=3X2DPA75MR7RX6LCOO2U3UREG4QSYHRRKRE57JUYUBINXTJN5IEA, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.803615052 (211h11m46.631323455s) [noise=true , address=145.79.227.28:4727 , piece_id=42ILJYV7IJRXSV3E3RWXWNUNYQHHQFNOF2JZGZKDEGPXRXZ3H2YA, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.803722621 (211h11m46.631431034s) [noise=true , address=77.90.31.16:4735 , piece_id=XKJJK6K7XF3TOFWMIU2BUVQTESTNV4FRYINSXJS4MC7KK7ZJU7SQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.804120053 (211h11m46.631828444s) [noise=true , address=145.79.227.7:4293 , piece_id=56YULKIWDM4P4WZM6KGMPUK3B3AK3RRVF4H6RJ7H6IQISPLXGBCQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.804570407 (211h11m46.632278798s) [noise=true , address=145.79.227.47:4267 , piece_id=VY46BNDUYVGBXDRIOEFK4USNOOJH2KXZH5Z2NHCT6SDJC5SFUYMQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.805083798 (211h11m46.632792196s) [noise=true , address=145.79.227.50:4375 , piece_id=RKVBJHHIF3KJLKB6ULTA4E5SHOBPCLPMWJCA6YTKF5XDNVEZMYXQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.805547276 (211h11m46.633255677s) [noise=true , address=77.90.31.16:4734 , piece_id=4JO2RBACNY5AM3KG7XUW7JTRGDCXGQDXSZ3L23ASAEJ4YFCICVAQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.805916642 (211h11m46.633625023s) [noise=true , address=145.79.227.39:4014 , piece_id=WKYZGQX5SP22ML66ZQZIHBS324EONJ7AARTYAXK2SCV577ZCQ5OQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.806440130 (211h11m46.634148525s) [noise=true , address=77.90.31.13:4626 , piece_id=DCKUBRB3DR755GN67RLP6NUMREAWIB3AITGPIMA7MM35BRPZW7IA, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.806976882 (211h11m46.634685284s) [noise=true , address=77.90.31.10:4519 , piece_id=ACPHBL7LF6NCNNH4U5GUXKW7RCL6XIEQVWYDCYUTXEC5RQE7RZBA, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.807729329 (211h11m46.63543772s) [noise=true , address=77.90.31.10:4518 , piece_id=ZE3SFGBCM2QV3CGGS3VEYT7NKDTDE2OJ6T6CVSGGYACCFX5WN5ZQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.841457783 (211h11m46.66916618s) [noise=true , address=77.90.31.13:4626 , piece_id=DCKUBRB3DR755GN67RLP6NUMREAWIB3AITGPIMA7MM35BRPZW7IA, seg_pos={0 0}, upload=219598]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: storage node overloaded, request limit: 2000 Dec 14 10:39:45.850206751 (211h11m46.677915151s) [noise=true , address=77.90.31.16:4734 , piece_id=4JO2RBACNY5AM3KG7XUW7JTRGDCXGQDXSZ3L23ASAEJ4YFCICVAQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: storage node overloaded, request limit: 2000 Dec 14 10:39:45.859644247 (211h11m46.68735265s) [noise=true , address=145.79.227.30:4763 , piece_id=CLMX6DHN5HP3OWMVR4R5MIVRH3GHCELUVTYJHBE7HXZQMHRQESIQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: storage node overloaded, request limit: 2000 Dec 14 10:39:45.874109133 (211h11m46.701817525s) [upload=219598]: Next returned nil segment Dec 14 10:39:45.874123719 (211h11m46.701832113s) [upload=219598]: Waiting for error group managing segments... Dec 14 10:39:45.880971826 (211h11m46.708680217s) [noise=true , address=77.90.31.10:4518 , piece_id=ZE3SFGBCM2QV3CGGS3VEYT7NKDTDE2OJ6T6CVSGGYACCFX5WN5ZQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.882649818 (211h11m46.710358212s) [noise=true , address=77.90.31.10:4519 , piece_id=ACPHBL7LF6NCNNH4U5GUXKW7RCL6XIEQVWYDCYUTXEC5RQE7RZBA, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.883088160 (211h11m46.71079655s) [noise=true , address=77.90.31.14:4661 , piece_id=LLI4CDLWJ32TFXQ2YEU3NRP4QIRSNNUODA5RRPWU5AA45MJEW2LQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.883693953 (211h11m46.711402345s) [noise=true , address=145.79.227.39:4014 , piece_id=WKYZGQX5SP22ML66ZQZIHBS324EONJ7AARTYAXK2SCV577ZCQ5OQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.884112448 (211h11m46.711820839s) [noise=true , address=145.79.227.32:4836 , piece_id=IQRH4CRVBOG72M3BE3G5FTIEMBTLD2KFUEYU6NCTK3N6PUONMY5A, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.884619811 (211h11m46.712328205s) [noise=true , address=145.79.227.43:4159 , piece_id=ULRDDEYCZXA26UPSFZQRQWFPU7MUCGJJSM7YWIR7HPXDCPBOQXNA, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.885012115 (211h11m46.712720506s) [noise=true , address=145.79.227.47:4267 , piece_id=VY46BNDUYVGBXDRIOEFK4USNOOJH2KXZH5Z2NHCT6SDJC5SFUYMQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.887669218 (211h11m46.71537761s) [noise=true , address=145.79.227.50:4375 , piece_id=RKVBJHHIF3KJLKB6ULTA4E5SHOBPCLPMWJCA6YTKF5XDNVEZMYXQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.890935473 (211h11m46.718643851s) [noise=true , address=145.79.227.46:4603 , piece_id=3X2DPA75MR7RX6LCOO2U3UREG4QSYHRRKRE57JUYUBINXTJN5IEA, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.891096730 (211h11m46.718805121s) [noise=true , address=145.79.227.28:4727 , piece_id=42ILJYV7IJRXSV3E3RWXWNUNYQHHQFNOF2JZGZKDEGPXRXZ3H2YA, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.893061640 (211h11m46.720770031s) [noise=true , address=145.79.227.7:4293 , piece_id=56YULKIWDM4P4WZM6KGMPUK3B3AK3RRVF4H6RJ7H6IQISPLXGBCQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:48.683585502 (211h11m49.511293883s) [noise=true , address=145.79.227.44:4194 , piece_id=ZWOCXZ746BWDZ2LWYHSQHSG2O3WJC4R6FDL5E3JP2SXRCX3AWT2Q, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:40:15.740699026 (211h12m16.568407426s) [noise=true , address=77.90.31.16:4735 , piece_id=XKJJK6K7XF3TOFWMIU2BUVQTESTNV4FRYINSXJS4MC7KK7ZJU7SQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: expected piece hash; context deadline exceeded; EOF Dec 14 10:40:15.740740404 (211h12m16.568448788s) [seg_pos={0 0}, upload=219598]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 14 10:40:15.762950810 (211h12m16.590659203s) [upload=219598]: Done uploading segments... Dec 14 10:40:15.762987936 (211h12m16.59069633s) [upload=219598]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 458.24625ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................X....................................] duration: 390.42851ms trace: https://jaeger.dev.storj.io/trace/25fec5475908806a FAIL (2025-12-14 11:05:08 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 14 10:39:45.775689210 (211h11m46.603397602s) [upload=219598]: Starting upload Dec 14 10:39:45.775750055 (211h11m46.603458455s) [upload=219598]: Done starting upload Dec 14 10:39:45.775957558 (211h11m46.603665941s) [upload=219598]: Uploading segments... Dec 14 10:39:45.776074427 (211h11m46.603782809s) [upload=219598]: Got next segment. Inline: false Dec 14 10:39:45.776158162 (211h11m46.603866553s) [upload=219598]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 14 10:39:45.776165965 (211h11m46.603874362s) [upload=219598]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 14 10:39:45.801257283 (211h11m46.628965678s) [seg_pos={0 0}, upload=219598]: Begin upload segment... Dec 14 10:39:45.801373021 (211h11m46.629081422s) [noise=true , address=145.79.227.43:4159 , piece_id=ULRDDEYCZXA26UPSFZQRQWFPU7MUCGJJSM7YWIR7HPXDCPBOQXNA, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.801409865 (211h11m46.629118247s) [seg_pos={0 0}, upload=219598]: Done begin upload segment. Dec 14 10:39:45.801481233 (211h11m46.629189635s) [noise=true , address=145.79.227.30:4763 , piece_id=CLMX6DHN5HP3OWMVR4R5MIVRH3GHCELUVTYJHBE7HXZQMHRQESIQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.802050295 (211h11m46.629758695s) [noise=true , address=145.79.227.32:4836 , piece_id=IQRH4CRVBOG72M3BE3G5FTIEMBTLD2KFUEYU6NCTK3N6PUONMY5A, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.802573060 (211h11m46.630281451s) [noise=true , address=77.90.31.14:4661 , piece_id=LLI4CDLWJ32TFXQ2YEU3NRP4QIRSNNUODA5RRPWU5AA45MJEW2LQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.803112997 (211h11m46.630821425s) [noise=true , address=145.79.227.44:4194 , piece_id=ZWOCXZ746BWDZ2LWYHSQHSG2O3WJC4R6FDL5E3JP2SXRCX3AWT2Q, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.803148596 (211h11m46.630856991s) [noise=true , address=145.79.227.46:4603 , piece_id=3X2DPA75MR7RX6LCOO2U3UREG4QSYHRRKRE57JUYUBINXTJN5IEA, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.803615052 (211h11m46.631323455s) [noise=true , address=145.79.227.28:4727 , piece_id=42ILJYV7IJRXSV3E3RWXWNUNYQHHQFNOF2JZGZKDEGPXRXZ3H2YA, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.803722621 (211h11m46.631431034s) [noise=true , address=77.90.31.16:4735 , piece_id=XKJJK6K7XF3TOFWMIU2BUVQTESTNV4FRYINSXJS4MC7KK7ZJU7SQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.804120053 (211h11m46.631828444s) [noise=true , address=145.79.227.7:4293 , piece_id=56YULKIWDM4P4WZM6KGMPUK3B3AK3RRVF4H6RJ7H6IQISPLXGBCQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.804570407 (211h11m46.632278798s) [noise=true , address=145.79.227.47:4267 , piece_id=VY46BNDUYVGBXDRIOEFK4USNOOJH2KXZH5Z2NHCT6SDJC5SFUYMQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.805083798 (211h11m46.632792196s) [noise=true , address=145.79.227.50:4375 , piece_id=RKVBJHHIF3KJLKB6ULTA4E5SHOBPCLPMWJCA6YTKF5XDNVEZMYXQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.805547276 (211h11m46.633255677s) [noise=true , address=77.90.31.16:4734 , piece_id=4JO2RBACNY5AM3KG7XUW7JTRGDCXGQDXSZ3L23ASAEJ4YFCICVAQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.805916642 (211h11m46.633625023s) [noise=true , address=145.79.227.39:4014 , piece_id=WKYZGQX5SP22ML66ZQZIHBS324EONJ7AARTYAXK2SCV577ZCQ5OQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.806440130 (211h11m46.634148525s) [noise=true , address=77.90.31.13:4626 , piece_id=DCKUBRB3DR755GN67RLP6NUMREAWIB3AITGPIMA7MM35BRPZW7IA, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.806976882 (211h11m46.634685284s) [noise=true , address=77.90.31.10:4519 , piece_id=ACPHBL7LF6NCNNH4U5GUXKW7RCL6XIEQVWYDCYUTXEC5RQE7RZBA, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.807729329 (211h11m46.63543772s) [noise=true , address=77.90.31.10:4518 , piece_id=ZE3SFGBCM2QV3CGGS3VEYT7NKDTDE2OJ6T6CVSGGYACCFX5WN5ZQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.841457783 (211h11m46.66916618s) [noise=true , address=77.90.31.13:4626 , piece_id=DCKUBRB3DR755GN67RLP6NUMREAWIB3AITGPIMA7MM35BRPZW7IA, seg_pos={0 0}, upload=219598]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: storage node overloaded, request limit: 2000 Dec 14 10:39:45.850206751 (211h11m46.677915151s) [noise=true , address=77.90.31.16:4734 , piece_id=4JO2RBACNY5AM3KG7XUW7JTRGDCXGQDXSZ3L23ASAEJ4YFCICVAQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: storage node overloaded, request limit: 2000 Dec 14 10:39:45.859644247 (211h11m46.68735265s) [noise=true , address=145.79.227.30:4763 , piece_id=CLMX6DHN5HP3OWMVR4R5MIVRH3GHCELUVTYJHBE7HXZQMHRQESIQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: storage node overloaded, request limit: 2000 Dec 14 10:39:45.874109133 (211h11m46.701817525s) [upload=219598]: Next returned nil segment Dec 14 10:39:45.874123719 (211h11m46.701832113s) [upload=219598]: Waiting for error group managing segments... Dec 14 10:39:45.880971826 (211h11m46.708680217s) [noise=true , address=77.90.31.10:4518 , piece_id=ZE3SFGBCM2QV3CGGS3VEYT7NKDTDE2OJ6T6CVSGGYACCFX5WN5ZQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.882649818 (211h11m46.710358212s) [noise=true , address=77.90.31.10:4519 , piece_id=ACPHBL7LF6NCNNH4U5GUXKW7RCL6XIEQVWYDCYUTXEC5RQE7RZBA, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.883088160 (211h11m46.71079655s) [noise=true , address=77.90.31.14:4661 , piece_id=LLI4CDLWJ32TFXQ2YEU3NRP4QIRSNNUODA5RRPWU5AA45MJEW2LQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.883693953 (211h11m46.711402345s) [noise=true , address=145.79.227.39:4014 , piece_id=WKYZGQX5SP22ML66ZQZIHBS324EONJ7AARTYAXK2SCV577ZCQ5OQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.884112448 (211h11m46.711820839s) [noise=true , address=145.79.227.32:4836 , piece_id=IQRH4CRVBOG72M3BE3G5FTIEMBTLD2KFUEYU6NCTK3N6PUONMY5A, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.884619811 (211h11m46.712328205s) [noise=true , address=145.79.227.43:4159 , piece_id=ULRDDEYCZXA26UPSFZQRQWFPU7MUCGJJSM7YWIR7HPXDCPBOQXNA, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.885012115 (211h11m46.712720506s) [noise=true , address=145.79.227.47:4267 , piece_id=VY46BNDUYVGBXDRIOEFK4USNOOJH2KXZH5Z2NHCT6SDJC5SFUYMQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.887669218 (211h11m46.71537761s) [noise=true , address=145.79.227.50:4375 , piece_id=RKVBJHHIF3KJLKB6ULTA4E5SHOBPCLPMWJCA6YTKF5XDNVEZMYXQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.890935473 (211h11m46.718643851s) [noise=true , address=145.79.227.46:4603 , piece_id=3X2DPA75MR7RX6LCOO2U3UREG4QSYHRRKRE57JUYUBINXTJN5IEA, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.891096730 (211h11m46.718805121s) [noise=true , address=145.79.227.28:4727 , piece_id=42ILJYV7IJRXSV3E3RWXWNUNYQHHQFNOF2JZGZKDEGPXRXZ3H2YA, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.893061640 (211h11m46.720770031s) [noise=true , address=145.79.227.7:4293 , piece_id=56YULKIWDM4P4WZM6KGMPUK3B3AK3RRVF4H6RJ7H6IQISPLXGBCQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:48.683585502 (211h11m49.511293883s) [noise=true , address=145.79.227.44:4194 , piece_id=ZWOCXZ746BWDZ2LWYHSQHSG2O3WJC4R6FDL5E3JP2SXRCX3AWT2Q, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:40:15.740699026 (211h12m16.568407426s) [noise=true , address=77.90.31.16:4735 , piece_id=XKJJK6K7XF3TOFWMIU2BUVQTESTNV4FRYINSXJS4MC7KK7ZJU7SQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: expected piece hash; context deadline exceeded; EOF Dec 14 10:40:15.740740404 (211h12m16.568448788s) [seg_pos={0 0}, upload=219598]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 14 10:40:15.762950810 (211h12m16.590659203s) [upload=219598]: Done uploading segments... Dec 14 10:40:15.762987936 (211h12m16.59069633s) [upload=219598]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 390.455756ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................X...................................] duration: 467.054285ms trace: https://jaeger.dev.storj.io/trace/25fec5475908806a FAIL (2025-12-14 11:04:01 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 14 10:39:45.775689210 (211h11m46.603397602s) [upload=219598]: Starting upload Dec 14 10:39:45.775750055 (211h11m46.603458455s) [upload=219598]: Done starting upload Dec 14 10:39:45.775957558 (211h11m46.603665941s) [upload=219598]: Uploading segments... Dec 14 10:39:45.776074427 (211h11m46.603782809s) [upload=219598]: Got next segment. Inline: false Dec 14 10:39:45.776158162 (211h11m46.603866553s) [upload=219598]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 14 10:39:45.776165965 (211h11m46.603874362s) [upload=219598]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 14 10:39:45.801257283 (211h11m46.628965678s) [seg_pos={0 0}, upload=219598]: Begin upload segment... Dec 14 10:39:45.801373021 (211h11m46.629081422s) [noise=true , address=145.79.227.43:4159 , piece_id=ULRDDEYCZXA26UPSFZQRQWFPU7MUCGJJSM7YWIR7HPXDCPBOQXNA, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.801409865 (211h11m46.629118247s) [seg_pos={0 0}, upload=219598]: Done begin upload segment. Dec 14 10:39:45.801481233 (211h11m46.629189635s) [noise=true , address=145.79.227.30:4763 , piece_id=CLMX6DHN5HP3OWMVR4R5MIVRH3GHCELUVTYJHBE7HXZQMHRQESIQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.802050295 (211h11m46.629758695s) [noise=true , address=145.79.227.32:4836 , piece_id=IQRH4CRVBOG72M3BE3G5FTIEMBTLD2KFUEYU6NCTK3N6PUONMY5A, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.802573060 (211h11m46.630281451s) [noise=true , address=77.90.31.14:4661 , piece_id=LLI4CDLWJ32TFXQ2YEU3NRP4QIRSNNUODA5RRPWU5AA45MJEW2LQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.803112997 (211h11m46.630821425s) [noise=true , address=145.79.227.44:4194 , piece_id=ZWOCXZ746BWDZ2LWYHSQHSG2O3WJC4R6FDL5E3JP2SXRCX3AWT2Q, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.803148596 (211h11m46.630856991s) [noise=true , address=145.79.227.46:4603 , piece_id=3X2DPA75MR7RX6LCOO2U3UREG4QSYHRRKRE57JUYUBINXTJN5IEA, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.803615052 (211h11m46.631323455s) [noise=true , address=145.79.227.28:4727 , piece_id=42ILJYV7IJRXSV3E3RWXWNUNYQHHQFNOF2JZGZKDEGPXRXZ3H2YA, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.803722621 (211h11m46.631431034s) [noise=true , address=77.90.31.16:4735 , piece_id=XKJJK6K7XF3TOFWMIU2BUVQTESTNV4FRYINSXJS4MC7KK7ZJU7SQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.804120053 (211h11m46.631828444s) [noise=true , address=145.79.227.7:4293 , piece_id=56YULKIWDM4P4WZM6KGMPUK3B3AK3RRVF4H6RJ7H6IQISPLXGBCQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.804570407 (211h11m46.632278798s) [noise=true , address=145.79.227.47:4267 , piece_id=VY46BNDUYVGBXDRIOEFK4USNOOJH2KXZH5Z2NHCT6SDJC5SFUYMQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.805083798 (211h11m46.632792196s) [noise=true , address=145.79.227.50:4375 , piece_id=RKVBJHHIF3KJLKB6ULTA4E5SHOBPCLPMWJCA6YTKF5XDNVEZMYXQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.805547276 (211h11m46.633255677s) [noise=true , address=77.90.31.16:4734 , piece_id=4JO2RBACNY5AM3KG7XUW7JTRGDCXGQDXSZ3L23ASAEJ4YFCICVAQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.805916642 (211h11m46.633625023s) [noise=true , address=145.79.227.39:4014 , piece_id=WKYZGQX5SP22ML66ZQZIHBS324EONJ7AARTYAXK2SCV577ZCQ5OQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.806440130 (211h11m46.634148525s) [noise=true , address=77.90.31.13:4626 , piece_id=DCKUBRB3DR755GN67RLP6NUMREAWIB3AITGPIMA7MM35BRPZW7IA, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.806976882 (211h11m46.634685284s) [noise=true , address=77.90.31.10:4519 , piece_id=ACPHBL7LF6NCNNH4U5GUXKW7RCL6XIEQVWYDCYUTXEC5RQE7RZBA, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.807729329 (211h11m46.63543772s) [noise=true , address=77.90.31.10:4518 , piece_id=ZE3SFGBCM2QV3CGGS3VEYT7NKDTDE2OJ6T6CVSGGYACCFX5WN5ZQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.841457783 (211h11m46.66916618s) [noise=true , address=77.90.31.13:4626 , piece_id=DCKUBRB3DR755GN67RLP6NUMREAWIB3AITGPIMA7MM35BRPZW7IA, seg_pos={0 0}, upload=219598]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: storage node overloaded, request limit: 2000 Dec 14 10:39:45.850206751 (211h11m46.677915151s) [noise=true , address=77.90.31.16:4734 , piece_id=4JO2RBACNY5AM3KG7XUW7JTRGDCXGQDXSZ3L23ASAEJ4YFCICVAQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: storage node overloaded, request limit: 2000 Dec 14 10:39:45.859644247 (211h11m46.68735265s) [noise=true , address=145.79.227.30:4763 , piece_id=CLMX6DHN5HP3OWMVR4R5MIVRH3GHCELUVTYJHBE7HXZQMHRQESIQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: storage node overloaded, request limit: 2000 Dec 14 10:39:45.874109133 (211h11m46.701817525s) [upload=219598]: Next returned nil segment Dec 14 10:39:45.874123719 (211h11m46.701832113s) [upload=219598]: Waiting for error group managing segments... Dec 14 10:39:45.880971826 (211h11m46.708680217s) [noise=true , address=77.90.31.10:4518 , piece_id=ZE3SFGBCM2QV3CGGS3VEYT7NKDTDE2OJ6T6CVSGGYACCFX5WN5ZQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.882649818 (211h11m46.710358212s) [noise=true , address=77.90.31.10:4519 , piece_id=ACPHBL7LF6NCNNH4U5GUXKW7RCL6XIEQVWYDCYUTXEC5RQE7RZBA, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.883088160 (211h11m46.71079655s) [noise=true , address=77.90.31.14:4661 , piece_id=LLI4CDLWJ32TFXQ2YEU3NRP4QIRSNNUODA5RRPWU5AA45MJEW2LQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.883693953 (211h11m46.711402345s) [noise=true , address=145.79.227.39:4014 , piece_id=WKYZGQX5SP22ML66ZQZIHBS324EONJ7AARTYAXK2SCV577ZCQ5OQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.884112448 (211h11m46.711820839s) [noise=true , address=145.79.227.32:4836 , piece_id=IQRH4CRVBOG72M3BE3G5FTIEMBTLD2KFUEYU6NCTK3N6PUONMY5A, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.884619811 (211h11m46.712328205s) [noise=true , address=145.79.227.43:4159 , piece_id=ULRDDEYCZXA26UPSFZQRQWFPU7MUCGJJSM7YWIR7HPXDCPBOQXNA, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.885012115 (211h11m46.712720506s) [noise=true , address=145.79.227.47:4267 , piece_id=VY46BNDUYVGBXDRIOEFK4USNOOJH2KXZH5Z2NHCT6SDJC5SFUYMQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.887669218 (211h11m46.71537761s) [noise=true , address=145.79.227.50:4375 , piece_id=RKVBJHHIF3KJLKB6ULTA4E5SHOBPCLPMWJCA6YTKF5XDNVEZMYXQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.890935473 (211h11m46.718643851s) [noise=true , address=145.79.227.46:4603 , piece_id=3X2DPA75MR7RX6LCOO2U3UREG4QSYHRRKRE57JUYUBINXTJN5IEA, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.891096730 (211h11m46.718805121s) [noise=true , address=145.79.227.28:4727 , piece_id=42ILJYV7IJRXSV3E3RWXWNUNYQHHQFNOF2JZGZKDEGPXRXZ3H2YA, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.893061640 (211h11m46.720770031s) [noise=true , address=145.79.227.7:4293 , piece_id=56YULKIWDM4P4WZM6KGMPUK3B3AK3RRVF4H6RJ7H6IQISPLXGBCQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:48.683585502 (211h11m49.511293883s) [noise=true , address=145.79.227.44:4194 , piece_id=ZWOCXZ746BWDZ2LWYHSQHSG2O3WJC4R6FDL5E3JP2SXRCX3AWT2Q, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:40:15.740699026 (211h12m16.568407426s) [noise=true , address=77.90.31.16:4735 , piece_id=XKJJK6K7XF3TOFWMIU2BUVQTESTNV4FRYINSXJS4MC7KK7ZJU7SQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: expected piece hash; context deadline exceeded; EOF Dec 14 10:40:15.740740404 (211h12m16.568448788s) [seg_pos={0 0}, upload=219598]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 14 10:40:15.762950810 (211h12m16.590659203s) [upload=219598]: Done uploading segments... Dec 14 10:40:15.762987936 (211h12m16.59069633s) [upload=219598]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 467.078363ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................X..................................] duration: 411.611414ms trace: https://jaeger.dev.storj.io/trace/25fec5475908806a FAIL (2025-12-14 11:03:14 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 14 10:39:45.775689210 (211h11m46.603397602s) [upload=219598]: Starting upload Dec 14 10:39:45.775750055 (211h11m46.603458455s) [upload=219598]: Done starting upload Dec 14 10:39:45.775957558 (211h11m46.603665941s) [upload=219598]: Uploading segments... Dec 14 10:39:45.776074427 (211h11m46.603782809s) [upload=219598]: Got next segment. Inline: false Dec 14 10:39:45.776158162 (211h11m46.603866553s) [upload=219598]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 14 10:39:45.776165965 (211h11m46.603874362s) [upload=219598]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 14 10:39:45.801257283 (211h11m46.628965678s) [seg_pos={0 0}, upload=219598]: Begin upload segment... Dec 14 10:39:45.801373021 (211h11m46.629081422s) [noise=true , address=145.79.227.43:4159 , piece_id=ULRDDEYCZXA26UPSFZQRQWFPU7MUCGJJSM7YWIR7HPXDCPBOQXNA, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.801409865 (211h11m46.629118247s) [seg_pos={0 0}, upload=219598]: Done begin upload segment. Dec 14 10:39:45.801481233 (211h11m46.629189635s) [noise=true , address=145.79.227.30:4763 , piece_id=CLMX6DHN5HP3OWMVR4R5MIVRH3GHCELUVTYJHBE7HXZQMHRQESIQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.802050295 (211h11m46.629758695s) [noise=true , address=145.79.227.32:4836 , piece_id=IQRH4CRVBOG72M3BE3G5FTIEMBTLD2KFUEYU6NCTK3N6PUONMY5A, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.802573060 (211h11m46.630281451s) [noise=true , address=77.90.31.14:4661 , piece_id=LLI4CDLWJ32TFXQ2YEU3NRP4QIRSNNUODA5RRPWU5AA45MJEW2LQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.803112997 (211h11m46.630821425s) [noise=true , address=145.79.227.44:4194 , piece_id=ZWOCXZ746BWDZ2LWYHSQHSG2O3WJC4R6FDL5E3JP2SXRCX3AWT2Q, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.803148596 (211h11m46.630856991s) [noise=true , address=145.79.227.46:4603 , piece_id=3X2DPA75MR7RX6LCOO2U3UREG4QSYHRRKRE57JUYUBINXTJN5IEA, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.803615052 (211h11m46.631323455s) [noise=true , address=145.79.227.28:4727 , piece_id=42ILJYV7IJRXSV3E3RWXWNUNYQHHQFNOF2JZGZKDEGPXRXZ3H2YA, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.803722621 (211h11m46.631431034s) [noise=true , address=77.90.31.16:4735 , piece_id=XKJJK6K7XF3TOFWMIU2BUVQTESTNV4FRYINSXJS4MC7KK7ZJU7SQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.804120053 (211h11m46.631828444s) [noise=true , address=145.79.227.7:4293 , piece_id=56YULKIWDM4P4WZM6KGMPUK3B3AK3RRVF4H6RJ7H6IQISPLXGBCQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.804570407 (211h11m46.632278798s) [noise=true , address=145.79.227.47:4267 , piece_id=VY46BNDUYVGBXDRIOEFK4USNOOJH2KXZH5Z2NHCT6SDJC5SFUYMQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.805083798 (211h11m46.632792196s) [noise=true , address=145.79.227.50:4375 , piece_id=RKVBJHHIF3KJLKB6ULTA4E5SHOBPCLPMWJCA6YTKF5XDNVEZMYXQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.805547276 (211h11m46.633255677s) [noise=true , address=77.90.31.16:4734 , piece_id=4JO2RBACNY5AM3KG7XUW7JTRGDCXGQDXSZ3L23ASAEJ4YFCICVAQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.805916642 (211h11m46.633625023s) [noise=true , address=145.79.227.39:4014 , piece_id=WKYZGQX5SP22ML66ZQZIHBS324EONJ7AARTYAXK2SCV577ZCQ5OQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.806440130 (211h11m46.634148525s) [noise=true , address=77.90.31.13:4626 , piece_id=DCKUBRB3DR755GN67RLP6NUMREAWIB3AITGPIMA7MM35BRPZW7IA, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.806976882 (211h11m46.634685284s) [noise=true , address=77.90.31.10:4519 , piece_id=ACPHBL7LF6NCNNH4U5GUXKW7RCL6XIEQVWYDCYUTXEC5RQE7RZBA, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.807729329 (211h11m46.63543772s) [noise=true , address=77.90.31.10:4518 , piece_id=ZE3SFGBCM2QV3CGGS3VEYT7NKDTDE2OJ6T6CVSGGYACCFX5WN5ZQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.841457783 (211h11m46.66916618s) [noise=true , address=77.90.31.13:4626 , piece_id=DCKUBRB3DR755GN67RLP6NUMREAWIB3AITGPIMA7MM35BRPZW7IA, seg_pos={0 0}, upload=219598]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: storage node overloaded, request limit: 2000 Dec 14 10:39:45.850206751 (211h11m46.677915151s) [noise=true , address=77.90.31.16:4734 , piece_id=4JO2RBACNY5AM3KG7XUW7JTRGDCXGQDXSZ3L23ASAEJ4YFCICVAQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: storage node overloaded, request limit: 2000 Dec 14 10:39:45.859644247 (211h11m46.68735265s) [noise=true , address=145.79.227.30:4763 , piece_id=CLMX6DHN5HP3OWMVR4R5MIVRH3GHCELUVTYJHBE7HXZQMHRQESIQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: storage node overloaded, request limit: 2000 Dec 14 10:39:45.874109133 (211h11m46.701817525s) [upload=219598]: Next returned nil segment Dec 14 10:39:45.874123719 (211h11m46.701832113s) [upload=219598]: Waiting for error group managing segments... Dec 14 10:39:45.880971826 (211h11m46.708680217s) [noise=true , address=77.90.31.10:4518 , piece_id=ZE3SFGBCM2QV3CGGS3VEYT7NKDTDE2OJ6T6CVSGGYACCFX5WN5ZQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.882649818 (211h11m46.710358212s) [noise=true , address=77.90.31.10:4519 , piece_id=ACPHBL7LF6NCNNH4U5GUXKW7RCL6XIEQVWYDCYUTXEC5RQE7RZBA, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.883088160 (211h11m46.71079655s) [noise=true , address=77.90.31.14:4661 , piece_id=LLI4CDLWJ32TFXQ2YEU3NRP4QIRSNNUODA5RRPWU5AA45MJEW2LQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.883693953 (211h11m46.711402345s) [noise=true , address=145.79.227.39:4014 , piece_id=WKYZGQX5SP22ML66ZQZIHBS324EONJ7AARTYAXK2SCV577ZCQ5OQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.884112448 (211h11m46.711820839s) [noise=true , address=145.79.227.32:4836 , piece_id=IQRH4CRVBOG72M3BE3G5FTIEMBTLD2KFUEYU6NCTK3N6PUONMY5A, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.884619811 (211h11m46.712328205s) [noise=true , address=145.79.227.43:4159 , piece_id=ULRDDEYCZXA26UPSFZQRQWFPU7MUCGJJSM7YWIR7HPXDCPBOQXNA, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.885012115 (211h11m46.712720506s) [noise=true , address=145.79.227.47:4267 , piece_id=VY46BNDUYVGBXDRIOEFK4USNOOJH2KXZH5Z2NHCT6SDJC5SFUYMQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.887669218 (211h11m46.71537761s) [noise=true , address=145.79.227.50:4375 , piece_id=RKVBJHHIF3KJLKB6ULTA4E5SHOBPCLPMWJCA6YTKF5XDNVEZMYXQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.890935473 (211h11m46.718643851s) [noise=true , address=145.79.227.46:4603 , piece_id=3X2DPA75MR7RX6LCOO2U3UREG4QSYHRRKRE57JUYUBINXTJN5IEA, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.891096730 (211h11m46.718805121s) [noise=true , address=145.79.227.28:4727 , piece_id=42ILJYV7IJRXSV3E3RWXWNUNYQHHQFNOF2JZGZKDEGPXRXZ3H2YA, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.893061640 (211h11m46.720770031s) [noise=true , address=145.79.227.7:4293 , piece_id=56YULKIWDM4P4WZM6KGMPUK3B3AK3RRVF4H6RJ7H6IQISPLXGBCQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:48.683585502 (211h11m49.511293883s) [noise=true , address=145.79.227.44:4194 , piece_id=ZWOCXZ746BWDZ2LWYHSQHSG2O3WJC4R6FDL5E3JP2SXRCX3AWT2Q, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:40:15.740699026 (211h12m16.568407426s) [noise=true , address=77.90.31.16:4735 , piece_id=XKJJK6K7XF3TOFWMIU2BUVQTESTNV4FRYINSXJS4MC7KK7ZJU7SQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: expected piece hash; context deadline exceeded; EOF Dec 14 10:40:15.740740404 (211h12m16.568448788s) [seg_pos={0 0}, upload=219598]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 14 10:40:15.762950810 (211h12m16.590659203s) [upload=219598]: Done uploading segments... Dec 14 10:40:15.762987936 (211h12m16.59069633s) [upload=219598]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 411.643723ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................X.................................] duration: 616.180032ms trace: https://jaeger.dev.storj.io/trace/25fec5475908806a FAIL (2025-12-14 11:02:34 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 14 10:39:45.775689210 (211h11m46.603397602s) [upload=219598]: Starting upload Dec 14 10:39:45.775750055 (211h11m46.603458455s) [upload=219598]: Done starting upload Dec 14 10:39:45.775957558 (211h11m46.603665941s) [upload=219598]: Uploading segments... Dec 14 10:39:45.776074427 (211h11m46.603782809s) [upload=219598]: Got next segment. Inline: false Dec 14 10:39:45.776158162 (211h11m46.603866553s) [upload=219598]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 14 10:39:45.776165965 (211h11m46.603874362s) [upload=219598]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 14 10:39:45.801257283 (211h11m46.628965678s) [seg_pos={0 0}, upload=219598]: Begin upload segment... Dec 14 10:39:45.801373021 (211h11m46.629081422s) [noise=true , address=145.79.227.43:4159 , piece_id=ULRDDEYCZXA26UPSFZQRQWFPU7MUCGJJSM7YWIR7HPXDCPBOQXNA, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.801409865 (211h11m46.629118247s) [seg_pos={0 0}, upload=219598]: Done begin upload segment. Dec 14 10:39:45.801481233 (211h11m46.629189635s) [noise=true , address=145.79.227.30:4763 , piece_id=CLMX6DHN5HP3OWMVR4R5MIVRH3GHCELUVTYJHBE7HXZQMHRQESIQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.802050295 (211h11m46.629758695s) [noise=true , address=145.79.227.32:4836 , piece_id=IQRH4CRVBOG72M3BE3G5FTIEMBTLD2KFUEYU6NCTK3N6PUONMY5A, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.802573060 (211h11m46.630281451s) [noise=true , address=77.90.31.14:4661 , piece_id=LLI4CDLWJ32TFXQ2YEU3NRP4QIRSNNUODA5RRPWU5AA45MJEW2LQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.803112997 (211h11m46.630821425s) [noise=true , address=145.79.227.44:4194 , piece_id=ZWOCXZ746BWDZ2LWYHSQHSG2O3WJC4R6FDL5E3JP2SXRCX3AWT2Q, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.803148596 (211h11m46.630856991s) [noise=true , address=145.79.227.46:4603 , piece_id=3X2DPA75MR7RX6LCOO2U3UREG4QSYHRRKRE57JUYUBINXTJN5IEA, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.803615052 (211h11m46.631323455s) [noise=true , address=145.79.227.28:4727 , piece_id=42ILJYV7IJRXSV3E3RWXWNUNYQHHQFNOF2JZGZKDEGPXRXZ3H2YA, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.803722621 (211h11m46.631431034s) [noise=true , address=77.90.31.16:4735 , piece_id=XKJJK6K7XF3TOFWMIU2BUVQTESTNV4FRYINSXJS4MC7KK7ZJU7SQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.804120053 (211h11m46.631828444s) [noise=true , address=145.79.227.7:4293 , piece_id=56YULKIWDM4P4WZM6KGMPUK3B3AK3RRVF4H6RJ7H6IQISPLXGBCQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.804570407 (211h11m46.632278798s) [noise=true , address=145.79.227.47:4267 , piece_id=VY46BNDUYVGBXDRIOEFK4USNOOJH2KXZH5Z2NHCT6SDJC5SFUYMQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.805083798 (211h11m46.632792196s) [noise=true , address=145.79.227.50:4375 , piece_id=RKVBJHHIF3KJLKB6ULTA4E5SHOBPCLPMWJCA6YTKF5XDNVEZMYXQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.805547276 (211h11m46.633255677s) [noise=true , address=77.90.31.16:4734 , piece_id=4JO2RBACNY5AM3KG7XUW7JTRGDCXGQDXSZ3L23ASAEJ4YFCICVAQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.805916642 (211h11m46.633625023s) [noise=true , address=145.79.227.39:4014 , piece_id=WKYZGQX5SP22ML66ZQZIHBS324EONJ7AARTYAXK2SCV577ZCQ5OQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.806440130 (211h11m46.634148525s) [noise=true , address=77.90.31.13:4626 , piece_id=DCKUBRB3DR755GN67RLP6NUMREAWIB3AITGPIMA7MM35BRPZW7IA, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.806976882 (211h11m46.634685284s) [noise=true , address=77.90.31.10:4519 , piece_id=ACPHBL7LF6NCNNH4U5GUXKW7RCL6XIEQVWYDCYUTXEC5RQE7RZBA, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.807729329 (211h11m46.63543772s) [noise=true , address=77.90.31.10:4518 , piece_id=ZE3SFGBCM2QV3CGGS3VEYT7NKDTDE2OJ6T6CVSGGYACCFX5WN5ZQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.841457783 (211h11m46.66916618s) [noise=true , address=77.90.31.13:4626 , piece_id=DCKUBRB3DR755GN67RLP6NUMREAWIB3AITGPIMA7MM35BRPZW7IA, seg_pos={0 0}, upload=219598]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: storage node overloaded, request limit: 2000 Dec 14 10:39:45.850206751 (211h11m46.677915151s) [noise=true , address=77.90.31.16:4734 , piece_id=4JO2RBACNY5AM3KG7XUW7JTRGDCXGQDXSZ3L23ASAEJ4YFCICVAQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: storage node overloaded, request limit: 2000 Dec 14 10:39:45.859644247 (211h11m46.68735265s) [noise=true , address=145.79.227.30:4763 , piece_id=CLMX6DHN5HP3OWMVR4R5MIVRH3GHCELUVTYJHBE7HXZQMHRQESIQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: storage node overloaded, request limit: 2000 Dec 14 10:39:45.874109133 (211h11m46.701817525s) [upload=219598]: Next returned nil segment Dec 14 10:39:45.874123719 (211h11m46.701832113s) [upload=219598]: Waiting for error group managing segments... Dec 14 10:39:45.880971826 (211h11m46.708680217s) [noise=true , address=77.90.31.10:4518 , piece_id=ZE3SFGBCM2QV3CGGS3VEYT7NKDTDE2OJ6T6CVSGGYACCFX5WN5ZQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.882649818 (211h11m46.710358212s) [noise=true , address=77.90.31.10:4519 , piece_id=ACPHBL7LF6NCNNH4U5GUXKW7RCL6XIEQVWYDCYUTXEC5RQE7RZBA, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.883088160 (211h11m46.71079655s) [noise=true , address=77.90.31.14:4661 , piece_id=LLI4CDLWJ32TFXQ2YEU3NRP4QIRSNNUODA5RRPWU5AA45MJEW2LQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.883693953 (211h11m46.711402345s) [noise=true , address=145.79.227.39:4014 , piece_id=WKYZGQX5SP22ML66ZQZIHBS324EONJ7AARTYAXK2SCV577ZCQ5OQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.884112448 (211h11m46.711820839s) [noise=true , address=145.79.227.32:4836 , piece_id=IQRH4CRVBOG72M3BE3G5FTIEMBTLD2KFUEYU6NCTK3N6PUONMY5A, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.884619811 (211h11m46.712328205s) [noise=true , address=145.79.227.43:4159 , piece_id=ULRDDEYCZXA26UPSFZQRQWFPU7MUCGJJSM7YWIR7HPXDCPBOQXNA, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.885012115 (211h11m46.712720506s) [noise=true , address=145.79.227.47:4267 , piece_id=VY46BNDUYVGBXDRIOEFK4USNOOJH2KXZH5Z2NHCT6SDJC5SFUYMQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.887669218 (211h11m46.71537761s) [noise=true , address=145.79.227.50:4375 , piece_id=RKVBJHHIF3KJLKB6ULTA4E5SHOBPCLPMWJCA6YTKF5XDNVEZMYXQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.890935473 (211h11m46.718643851s) [noise=true , address=145.79.227.46:4603 , piece_id=3X2DPA75MR7RX6LCOO2U3UREG4QSYHRRKRE57JUYUBINXTJN5IEA, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.891096730 (211h11m46.718805121s) [noise=true , address=145.79.227.28:4727 , piece_id=42ILJYV7IJRXSV3E3RWXWNUNYQHHQFNOF2JZGZKDEGPXRXZ3H2YA, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.893061640 (211h11m46.720770031s) [noise=true , address=145.79.227.7:4293 , piece_id=56YULKIWDM4P4WZM6KGMPUK3B3AK3RRVF4H6RJ7H6IQISPLXGBCQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:48.683585502 (211h11m49.511293883s) [noise=true , address=145.79.227.44:4194 , piece_id=ZWOCXZ746BWDZ2LWYHSQHSG2O3WJC4R6FDL5E3JP2SXRCX3AWT2Q, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:40:15.740699026 (211h12m16.568407426s) [noise=true , address=77.90.31.16:4735 , piece_id=XKJJK6K7XF3TOFWMIU2BUVQTESTNV4FRYINSXJS4MC7KK7ZJU7SQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: expected piece hash; context deadline exceeded; EOF Dec 14 10:40:15.740740404 (211h12m16.568448788s) [seg_pos={0 0}, upload=219598]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 14 10:40:15.762950810 (211h12m16.590659203s) [upload=219598]: Done uploading segments... Dec 14 10:40:15.762987936 (211h12m16.59069633s) [upload=219598]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 616.202926ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................X................................] duration: 499.723096ms trace: https://jaeger.dev.storj.io/trace/25fec5475908806a FAIL (2025-12-14 11:01:45 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 14 10:39:45.775689210 (211h11m46.603397602s) [upload=219598]: Starting upload Dec 14 10:39:45.775750055 (211h11m46.603458455s) [upload=219598]: Done starting upload Dec 14 10:39:45.775957558 (211h11m46.603665941s) [upload=219598]: Uploading segments... Dec 14 10:39:45.776074427 (211h11m46.603782809s) [upload=219598]: Got next segment. Inline: false Dec 14 10:39:45.776158162 (211h11m46.603866553s) [upload=219598]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 14 10:39:45.776165965 (211h11m46.603874362s) [upload=219598]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 14 10:39:45.801257283 (211h11m46.628965678s) [seg_pos={0 0}, upload=219598]: Begin upload segment... Dec 14 10:39:45.801373021 (211h11m46.629081422s) [noise=true , address=145.79.227.43:4159 , piece_id=ULRDDEYCZXA26UPSFZQRQWFPU7MUCGJJSM7YWIR7HPXDCPBOQXNA, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.801409865 (211h11m46.629118247s) [seg_pos={0 0}, upload=219598]: Done begin upload segment. Dec 14 10:39:45.801481233 (211h11m46.629189635s) [noise=true , address=145.79.227.30:4763 , piece_id=CLMX6DHN5HP3OWMVR4R5MIVRH3GHCELUVTYJHBE7HXZQMHRQESIQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.802050295 (211h11m46.629758695s) [noise=true , address=145.79.227.32:4836 , piece_id=IQRH4CRVBOG72M3BE3G5FTIEMBTLD2KFUEYU6NCTK3N6PUONMY5A, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.802573060 (211h11m46.630281451s) [noise=true , address=77.90.31.14:4661 , piece_id=LLI4CDLWJ32TFXQ2YEU3NRP4QIRSNNUODA5RRPWU5AA45MJEW2LQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.803112997 (211h11m46.630821425s) [noise=true , address=145.79.227.44:4194 , piece_id=ZWOCXZ746BWDZ2LWYHSQHSG2O3WJC4R6FDL5E3JP2SXRCX3AWT2Q, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.803148596 (211h11m46.630856991s) [noise=true , address=145.79.227.46:4603 , piece_id=3X2DPA75MR7RX6LCOO2U3UREG4QSYHRRKRE57JUYUBINXTJN5IEA, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.803615052 (211h11m46.631323455s) [noise=true , address=145.79.227.28:4727 , piece_id=42ILJYV7IJRXSV3E3RWXWNUNYQHHQFNOF2JZGZKDEGPXRXZ3H2YA, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.803722621 (211h11m46.631431034s) [noise=true , address=77.90.31.16:4735 , piece_id=XKJJK6K7XF3TOFWMIU2BUVQTESTNV4FRYINSXJS4MC7KK7ZJU7SQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.804120053 (211h11m46.631828444s) [noise=true , address=145.79.227.7:4293 , piece_id=56YULKIWDM4P4WZM6KGMPUK3B3AK3RRVF4H6RJ7H6IQISPLXGBCQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.804570407 (211h11m46.632278798s) [noise=true , address=145.79.227.47:4267 , piece_id=VY46BNDUYVGBXDRIOEFK4USNOOJH2KXZH5Z2NHCT6SDJC5SFUYMQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.805083798 (211h11m46.632792196s) [noise=true , address=145.79.227.50:4375 , piece_id=RKVBJHHIF3KJLKB6ULTA4E5SHOBPCLPMWJCA6YTKF5XDNVEZMYXQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.805547276 (211h11m46.633255677s) [noise=true , address=77.90.31.16:4734 , piece_id=4JO2RBACNY5AM3KG7XUW7JTRGDCXGQDXSZ3L23ASAEJ4YFCICVAQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.805916642 (211h11m46.633625023s) [noise=true , address=145.79.227.39:4014 , piece_id=WKYZGQX5SP22ML66ZQZIHBS324EONJ7AARTYAXK2SCV577ZCQ5OQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.806440130 (211h11m46.634148525s) [noise=true , address=77.90.31.13:4626 , piece_id=DCKUBRB3DR755GN67RLP6NUMREAWIB3AITGPIMA7MM35BRPZW7IA, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.806976882 (211h11m46.634685284s) [noise=true , address=77.90.31.10:4519 , piece_id=ACPHBL7LF6NCNNH4U5GUXKW7RCL6XIEQVWYDCYUTXEC5RQE7RZBA, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.807729329 (211h11m46.63543772s) [noise=true , address=77.90.31.10:4518 , piece_id=ZE3SFGBCM2QV3CGGS3VEYT7NKDTDE2OJ6T6CVSGGYACCFX5WN5ZQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.841457783 (211h11m46.66916618s) [noise=true , address=77.90.31.13:4626 , piece_id=DCKUBRB3DR755GN67RLP6NUMREAWIB3AITGPIMA7MM35BRPZW7IA, seg_pos={0 0}, upload=219598]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: storage node overloaded, request limit: 2000 Dec 14 10:39:45.850206751 (211h11m46.677915151s) [noise=true , address=77.90.31.16:4734 , piece_id=4JO2RBACNY5AM3KG7XUW7JTRGDCXGQDXSZ3L23ASAEJ4YFCICVAQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: storage node overloaded, request limit: 2000 Dec 14 10:39:45.859644247 (211h11m46.68735265s) [noise=true , address=145.79.227.30:4763 , piece_id=CLMX6DHN5HP3OWMVR4R5MIVRH3GHCELUVTYJHBE7HXZQMHRQESIQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: storage node overloaded, request limit: 2000 Dec 14 10:39:45.874109133 (211h11m46.701817525s) [upload=219598]: Next returned nil segment Dec 14 10:39:45.874123719 (211h11m46.701832113s) [upload=219598]: Waiting for error group managing segments... Dec 14 10:39:45.880971826 (211h11m46.708680217s) [noise=true , address=77.90.31.10:4518 , piece_id=ZE3SFGBCM2QV3CGGS3VEYT7NKDTDE2OJ6T6CVSGGYACCFX5WN5ZQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.882649818 (211h11m46.710358212s) [noise=true , address=77.90.31.10:4519 , piece_id=ACPHBL7LF6NCNNH4U5GUXKW7RCL6XIEQVWYDCYUTXEC5RQE7RZBA, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.883088160 (211h11m46.71079655s) [noise=true , address=77.90.31.14:4661 , piece_id=LLI4CDLWJ32TFXQ2YEU3NRP4QIRSNNUODA5RRPWU5AA45MJEW2LQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.883693953 (211h11m46.711402345s) [noise=true , address=145.79.227.39:4014 , piece_id=WKYZGQX5SP22ML66ZQZIHBS324EONJ7AARTYAXK2SCV577ZCQ5OQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.884112448 (211h11m46.711820839s) [noise=true , address=145.79.227.32:4836 , piece_id=IQRH4CRVBOG72M3BE3G5FTIEMBTLD2KFUEYU6NCTK3N6PUONMY5A, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.884619811 (211h11m46.712328205s) [noise=true , address=145.79.227.43:4159 , piece_id=ULRDDEYCZXA26UPSFZQRQWFPU7MUCGJJSM7YWIR7HPXDCPBOQXNA, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.885012115 (211h11m46.712720506s) [noise=true , address=145.79.227.47:4267 , piece_id=VY46BNDUYVGBXDRIOEFK4USNOOJH2KXZH5Z2NHCT6SDJC5SFUYMQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.887669218 (211h11m46.71537761s) [noise=true , address=145.79.227.50:4375 , piece_id=RKVBJHHIF3KJLKB6ULTA4E5SHOBPCLPMWJCA6YTKF5XDNVEZMYXQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.890935473 (211h11m46.718643851s) [noise=true , address=145.79.227.46:4603 , piece_id=3X2DPA75MR7RX6LCOO2U3UREG4QSYHRRKRE57JUYUBINXTJN5IEA, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.891096730 (211h11m46.718805121s) [noise=true , address=145.79.227.28:4727 , piece_id=42ILJYV7IJRXSV3E3RWXWNUNYQHHQFNOF2JZGZKDEGPXRXZ3H2YA, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.893061640 (211h11m46.720770031s) [noise=true , address=145.79.227.7:4293 , piece_id=56YULKIWDM4P4WZM6KGMPUK3B3AK3RRVF4H6RJ7H6IQISPLXGBCQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:48.683585502 (211h11m49.511293883s) [noise=true , address=145.79.227.44:4194 , piece_id=ZWOCXZ746BWDZ2LWYHSQHSG2O3WJC4R6FDL5E3JP2SXRCX3AWT2Q, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:40:15.740699026 (211h12m16.568407426s) [noise=true , address=77.90.31.16:4735 , piece_id=XKJJK6K7XF3TOFWMIU2BUVQTESTNV4FRYINSXJS4MC7KK7ZJU7SQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: expected piece hash; context deadline exceeded; EOF Dec 14 10:40:15.740740404 (211h12m16.568448788s) [seg_pos={0 0}, upload=219598]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 14 10:40:15.762950810 (211h12m16.590659203s) [upload=219598]: Done uploading segments... Dec 14 10:40:15.762987936 (211h12m16.59069633s) [upload=219598]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 499.752519ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............................X...............................] duration: 399.608113ms trace: https://jaeger.dev.storj.io/trace/25fec5475908806a FAIL (2025-12-14 11:00:36 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 14 10:39:45.775689210 (211h11m46.603397602s) [upload=219598]: Starting upload Dec 14 10:39:45.775750055 (211h11m46.603458455s) [upload=219598]: Done starting upload Dec 14 10:39:45.775957558 (211h11m46.603665941s) [upload=219598]: Uploading segments... Dec 14 10:39:45.776074427 (211h11m46.603782809s) [upload=219598]: Got next segment. Inline: false Dec 14 10:39:45.776158162 (211h11m46.603866553s) [upload=219598]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 14 10:39:45.776165965 (211h11m46.603874362s) [upload=219598]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 14 10:39:45.801257283 (211h11m46.628965678s) [seg_pos={0 0}, upload=219598]: Begin upload segment... Dec 14 10:39:45.801373021 (211h11m46.629081422s) [noise=true , address=145.79.227.43:4159 , piece_id=ULRDDEYCZXA26UPSFZQRQWFPU7MUCGJJSM7YWIR7HPXDCPBOQXNA, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.801409865 (211h11m46.629118247s) [seg_pos={0 0}, upload=219598]: Done begin upload segment. Dec 14 10:39:45.801481233 (211h11m46.629189635s) [noise=true , address=145.79.227.30:4763 , piece_id=CLMX6DHN5HP3OWMVR4R5MIVRH3GHCELUVTYJHBE7HXZQMHRQESIQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.802050295 (211h11m46.629758695s) [noise=true , address=145.79.227.32:4836 , piece_id=IQRH4CRVBOG72M3BE3G5FTIEMBTLD2KFUEYU6NCTK3N6PUONMY5A, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.802573060 (211h11m46.630281451s) [noise=true , address=77.90.31.14:4661 , piece_id=LLI4CDLWJ32TFXQ2YEU3NRP4QIRSNNUODA5RRPWU5AA45MJEW2LQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.803112997 (211h11m46.630821425s) [noise=true , address=145.79.227.44:4194 , piece_id=ZWOCXZ746BWDZ2LWYHSQHSG2O3WJC4R6FDL5E3JP2SXRCX3AWT2Q, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.803148596 (211h11m46.630856991s) [noise=true , address=145.79.227.46:4603 , piece_id=3X2DPA75MR7RX6LCOO2U3UREG4QSYHRRKRE57JUYUBINXTJN5IEA, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.803615052 (211h11m46.631323455s) [noise=true , address=145.79.227.28:4727 , piece_id=42ILJYV7IJRXSV3E3RWXWNUNYQHHQFNOF2JZGZKDEGPXRXZ3H2YA, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.803722621 (211h11m46.631431034s) [noise=true , address=77.90.31.16:4735 , piece_id=XKJJK6K7XF3TOFWMIU2BUVQTESTNV4FRYINSXJS4MC7KK7ZJU7SQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.804120053 (211h11m46.631828444s) [noise=true , address=145.79.227.7:4293 , piece_id=56YULKIWDM4P4WZM6KGMPUK3B3AK3RRVF4H6RJ7H6IQISPLXGBCQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.804570407 (211h11m46.632278798s) [noise=true , address=145.79.227.47:4267 , piece_id=VY46BNDUYVGBXDRIOEFK4USNOOJH2KXZH5Z2NHCT6SDJC5SFUYMQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.805083798 (211h11m46.632792196s) [noise=true , address=145.79.227.50:4375 , piece_id=RKVBJHHIF3KJLKB6ULTA4E5SHOBPCLPMWJCA6YTKF5XDNVEZMYXQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.805547276 (211h11m46.633255677s) [noise=true , address=77.90.31.16:4734 , piece_id=4JO2RBACNY5AM3KG7XUW7JTRGDCXGQDXSZ3L23ASAEJ4YFCICVAQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.805916642 (211h11m46.633625023s) [noise=true , address=145.79.227.39:4014 , piece_id=WKYZGQX5SP22ML66ZQZIHBS324EONJ7AARTYAXK2SCV577ZCQ5OQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.806440130 (211h11m46.634148525s) [noise=true , address=77.90.31.13:4626 , piece_id=DCKUBRB3DR755GN67RLP6NUMREAWIB3AITGPIMA7MM35BRPZW7IA, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.806976882 (211h11m46.634685284s) [noise=true , address=77.90.31.10:4519 , piece_id=ACPHBL7LF6NCNNH4U5GUXKW7RCL6XIEQVWYDCYUTXEC5RQE7RZBA, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.807729329 (211h11m46.63543772s) [noise=true , address=77.90.31.10:4518 , piece_id=ZE3SFGBCM2QV3CGGS3VEYT7NKDTDE2OJ6T6CVSGGYACCFX5WN5ZQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.841457783 (211h11m46.66916618s) [noise=true , address=77.90.31.13:4626 , piece_id=DCKUBRB3DR755GN67RLP6NUMREAWIB3AITGPIMA7MM35BRPZW7IA, seg_pos={0 0}, upload=219598]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: storage node overloaded, request limit: 2000 Dec 14 10:39:45.850206751 (211h11m46.677915151s) [noise=true , address=77.90.31.16:4734 , piece_id=4JO2RBACNY5AM3KG7XUW7JTRGDCXGQDXSZ3L23ASAEJ4YFCICVAQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: storage node overloaded, request limit: 2000 Dec 14 10:39:45.859644247 (211h11m46.68735265s) [noise=true , address=145.79.227.30:4763 , piece_id=CLMX6DHN5HP3OWMVR4R5MIVRH3GHCELUVTYJHBE7HXZQMHRQESIQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: storage node overloaded, request limit: 2000 Dec 14 10:39:45.874109133 (211h11m46.701817525s) [upload=219598]: Next returned nil segment Dec 14 10:39:45.874123719 (211h11m46.701832113s) [upload=219598]: Waiting for error group managing segments... Dec 14 10:39:45.880971826 (211h11m46.708680217s) [noise=true , address=77.90.31.10:4518 , piece_id=ZE3SFGBCM2QV3CGGS3VEYT7NKDTDE2OJ6T6CVSGGYACCFX5WN5ZQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.882649818 (211h11m46.710358212s) [noise=true , address=77.90.31.10:4519 , piece_id=ACPHBL7LF6NCNNH4U5GUXKW7RCL6XIEQVWYDCYUTXEC5RQE7RZBA, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.883088160 (211h11m46.71079655s) [noise=true , address=77.90.31.14:4661 , piece_id=LLI4CDLWJ32TFXQ2YEU3NRP4QIRSNNUODA5RRPWU5AA45MJEW2LQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.883693953 (211h11m46.711402345s) [noise=true , address=145.79.227.39:4014 , piece_id=WKYZGQX5SP22ML66ZQZIHBS324EONJ7AARTYAXK2SCV577ZCQ5OQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.884112448 (211h11m46.711820839s) [noise=true , address=145.79.227.32:4836 , piece_id=IQRH4CRVBOG72M3BE3G5FTIEMBTLD2KFUEYU6NCTK3N6PUONMY5A, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.884619811 (211h11m46.712328205s) [noise=true , address=145.79.227.43:4159 , piece_id=ULRDDEYCZXA26UPSFZQRQWFPU7MUCGJJSM7YWIR7HPXDCPBOQXNA, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.885012115 (211h11m46.712720506s) [noise=true , address=145.79.227.47:4267 , piece_id=VY46BNDUYVGBXDRIOEFK4USNOOJH2KXZH5Z2NHCT6SDJC5SFUYMQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.887669218 (211h11m46.71537761s) [noise=true , address=145.79.227.50:4375 , piece_id=RKVBJHHIF3KJLKB6ULTA4E5SHOBPCLPMWJCA6YTKF5XDNVEZMYXQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.890935473 (211h11m46.718643851s) [noise=true , address=145.79.227.46:4603 , piece_id=3X2DPA75MR7RX6LCOO2U3UREG4QSYHRRKRE57JUYUBINXTJN5IEA, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.891096730 (211h11m46.718805121s) [noise=true , address=145.79.227.28:4727 , piece_id=42ILJYV7IJRXSV3E3RWXWNUNYQHHQFNOF2JZGZKDEGPXRXZ3H2YA, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.893061640 (211h11m46.720770031s) [noise=true , address=145.79.227.7:4293 , piece_id=56YULKIWDM4P4WZM6KGMPUK3B3AK3RRVF4H6RJ7H6IQISPLXGBCQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:48.683585502 (211h11m49.511293883s) [noise=true , address=145.79.227.44:4194 , piece_id=ZWOCXZ746BWDZ2LWYHSQHSG2O3WJC4R6FDL5E3JP2SXRCX3AWT2Q, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:40:15.740699026 (211h12m16.568407426s) [noise=true , address=77.90.31.16:4735 , piece_id=XKJJK6K7XF3TOFWMIU2BUVQTESTNV4FRYINSXJS4MC7KK7ZJU7SQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: expected piece hash; context deadline exceeded; EOF Dec 14 10:40:15.740740404 (211h12m16.568448788s) [seg_pos={0 0}, upload=219598]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 14 10:40:15.762950810 (211h12m16.590659203s) [upload=219598]: Done uploading segments... Dec 14 10:40:15.762987936 (211h12m16.59069633s) [upload=219598]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 399.635178ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............................X..............................] duration: 385.878983ms trace: https://jaeger.dev.storj.io/trace/25fec5475908806a FAIL (2025-12-14 11:00:19 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 14 10:39:45.775689210 (211h11m46.603397602s) [upload=219598]: Starting upload Dec 14 10:39:45.775750055 (211h11m46.603458455s) [upload=219598]: Done starting upload Dec 14 10:39:45.775957558 (211h11m46.603665941s) [upload=219598]: Uploading segments... Dec 14 10:39:45.776074427 (211h11m46.603782809s) [upload=219598]: Got next segment. Inline: false Dec 14 10:39:45.776158162 (211h11m46.603866553s) [upload=219598]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 14 10:39:45.776165965 (211h11m46.603874362s) [upload=219598]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 14 10:39:45.801257283 (211h11m46.628965678s) [seg_pos={0 0}, upload=219598]: Begin upload segment... Dec 14 10:39:45.801373021 (211h11m46.629081422s) [noise=true , address=145.79.227.43:4159 , piece_id=ULRDDEYCZXA26UPSFZQRQWFPU7MUCGJJSM7YWIR7HPXDCPBOQXNA, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.801409865 (211h11m46.629118247s) [seg_pos={0 0}, upload=219598]: Done begin upload segment. Dec 14 10:39:45.801481233 (211h11m46.629189635s) [noise=true , address=145.79.227.30:4763 , piece_id=CLMX6DHN5HP3OWMVR4R5MIVRH3GHCELUVTYJHBE7HXZQMHRQESIQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.802050295 (211h11m46.629758695s) [noise=true , address=145.79.227.32:4836 , piece_id=IQRH4CRVBOG72M3BE3G5FTIEMBTLD2KFUEYU6NCTK3N6PUONMY5A, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.802573060 (211h11m46.630281451s) [noise=true , address=77.90.31.14:4661 , piece_id=LLI4CDLWJ32TFXQ2YEU3NRP4QIRSNNUODA5RRPWU5AA45MJEW2LQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.803112997 (211h11m46.630821425s) [noise=true , address=145.79.227.44:4194 , piece_id=ZWOCXZ746BWDZ2LWYHSQHSG2O3WJC4R6FDL5E3JP2SXRCX3AWT2Q, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.803148596 (211h11m46.630856991s) [noise=true , address=145.79.227.46:4603 , piece_id=3X2DPA75MR7RX6LCOO2U3UREG4QSYHRRKRE57JUYUBINXTJN5IEA, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.803615052 (211h11m46.631323455s) [noise=true , address=145.79.227.28:4727 , piece_id=42ILJYV7IJRXSV3E3RWXWNUNYQHHQFNOF2JZGZKDEGPXRXZ3H2YA, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.803722621 (211h11m46.631431034s) [noise=true , address=77.90.31.16:4735 , piece_id=XKJJK6K7XF3TOFWMIU2BUVQTESTNV4FRYINSXJS4MC7KK7ZJU7SQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.804120053 (211h11m46.631828444s) [noise=true , address=145.79.227.7:4293 , piece_id=56YULKIWDM4P4WZM6KGMPUK3B3AK3RRVF4H6RJ7H6IQISPLXGBCQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.804570407 (211h11m46.632278798s) [noise=true , address=145.79.227.47:4267 , piece_id=VY46BNDUYVGBXDRIOEFK4USNOOJH2KXZH5Z2NHCT6SDJC5SFUYMQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.805083798 (211h11m46.632792196s) [noise=true , address=145.79.227.50:4375 , piece_id=RKVBJHHIF3KJLKB6ULTA4E5SHOBPCLPMWJCA6YTKF5XDNVEZMYXQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.805547276 (211h11m46.633255677s) [noise=true , address=77.90.31.16:4734 , piece_id=4JO2RBACNY5AM3KG7XUW7JTRGDCXGQDXSZ3L23ASAEJ4YFCICVAQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.805916642 (211h11m46.633625023s) [noise=true , address=145.79.227.39:4014 , piece_id=WKYZGQX5SP22ML66ZQZIHBS324EONJ7AARTYAXK2SCV577ZCQ5OQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.806440130 (211h11m46.634148525s) [noise=true , address=77.90.31.13:4626 , piece_id=DCKUBRB3DR755GN67RLP6NUMREAWIB3AITGPIMA7MM35BRPZW7IA, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.806976882 (211h11m46.634685284s) [noise=true , address=77.90.31.10:4519 , piece_id=ACPHBL7LF6NCNNH4U5GUXKW7RCL6XIEQVWYDCYUTXEC5RQE7RZBA, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.807729329 (211h11m46.63543772s) [noise=true , address=77.90.31.10:4518 , piece_id=ZE3SFGBCM2QV3CGGS3VEYT7NKDTDE2OJ6T6CVSGGYACCFX5WN5ZQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.841457783 (211h11m46.66916618s) [noise=true , address=77.90.31.13:4626 , piece_id=DCKUBRB3DR755GN67RLP6NUMREAWIB3AITGPIMA7MM35BRPZW7IA, seg_pos={0 0}, upload=219598]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: storage node overloaded, request limit: 2000 Dec 14 10:39:45.850206751 (211h11m46.677915151s) [noise=true , address=77.90.31.16:4734 , piece_id=4JO2RBACNY5AM3KG7XUW7JTRGDCXGQDXSZ3L23ASAEJ4YFCICVAQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: storage node overloaded, request limit: 2000 Dec 14 10:39:45.859644247 (211h11m46.68735265s) [noise=true , address=145.79.227.30:4763 , piece_id=CLMX6DHN5HP3OWMVR4R5MIVRH3GHCELUVTYJHBE7HXZQMHRQESIQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: storage node overloaded, request limit: 2000 Dec 14 10:39:45.874109133 (211h11m46.701817525s) [upload=219598]: Next returned nil segment Dec 14 10:39:45.874123719 (211h11m46.701832113s) [upload=219598]: Waiting for error group managing segments... Dec 14 10:39:45.880971826 (211h11m46.708680217s) [noise=true , address=77.90.31.10:4518 , piece_id=ZE3SFGBCM2QV3CGGS3VEYT7NKDTDE2OJ6T6CVSGGYACCFX5WN5ZQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.882649818 (211h11m46.710358212s) [noise=true , address=77.90.31.10:4519 , piece_id=ACPHBL7LF6NCNNH4U5GUXKW7RCL6XIEQVWYDCYUTXEC5RQE7RZBA, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.883088160 (211h11m46.71079655s) [noise=true , address=77.90.31.14:4661 , piece_id=LLI4CDLWJ32TFXQ2YEU3NRP4QIRSNNUODA5RRPWU5AA45MJEW2LQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.883693953 (211h11m46.711402345s) [noise=true , address=145.79.227.39:4014 , piece_id=WKYZGQX5SP22ML66ZQZIHBS324EONJ7AARTYAXK2SCV577ZCQ5OQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.884112448 (211h11m46.711820839s) [noise=true , address=145.79.227.32:4836 , piece_id=IQRH4CRVBOG72M3BE3G5FTIEMBTLD2KFUEYU6NCTK3N6PUONMY5A, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.884619811 (211h11m46.712328205s) [noise=true , address=145.79.227.43:4159 , piece_id=ULRDDEYCZXA26UPSFZQRQWFPU7MUCGJJSM7YWIR7HPXDCPBOQXNA, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.885012115 (211h11m46.712720506s) [noise=true , address=145.79.227.47:4267 , piece_id=VY46BNDUYVGBXDRIOEFK4USNOOJH2KXZH5Z2NHCT6SDJC5SFUYMQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.887669218 (211h11m46.71537761s) [noise=true , address=145.79.227.50:4375 , piece_id=RKVBJHHIF3KJLKB6ULTA4E5SHOBPCLPMWJCA6YTKF5XDNVEZMYXQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.890935473 (211h11m46.718643851s) [noise=true , address=145.79.227.46:4603 , piece_id=3X2DPA75MR7RX6LCOO2U3UREG4QSYHRRKRE57JUYUBINXTJN5IEA, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.891096730 (211h11m46.718805121s) [noise=true , address=145.79.227.28:4727 , piece_id=42ILJYV7IJRXSV3E3RWXWNUNYQHHQFNOF2JZGZKDEGPXRXZ3H2YA, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.893061640 (211h11m46.720770031s) [noise=true , address=145.79.227.7:4293 , piece_id=56YULKIWDM4P4WZM6KGMPUK3B3AK3RRVF4H6RJ7H6IQISPLXGBCQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:48.683585502 (211h11m49.511293883s) [noise=true , address=145.79.227.44:4194 , piece_id=ZWOCXZ746BWDZ2LWYHSQHSG2O3WJC4R6FDL5E3JP2SXRCX3AWT2Q, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:40:15.740699026 (211h12m16.568407426s) [noise=true , address=77.90.31.16:4735 , piece_id=XKJJK6K7XF3TOFWMIU2BUVQTESTNV4FRYINSXJS4MC7KK7ZJU7SQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: expected piece hash; context deadline exceeded; EOF Dec 14 10:40:15.740740404 (211h12m16.568448788s) [seg_pos={0 0}, upload=219598]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 14 10:40:15.762950810 (211h12m16.590659203s) [upload=219598]: Done uploading segments... Dec 14 10:40:15.762987936 (211h12m16.59069633s) [upload=219598]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 385.9017ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............................X.............................] duration: 438.835665ms trace: https://jaeger.dev.storj.io/trace/25fec5475908806a FAIL (2025-12-14 10:59:12 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 14 10:39:45.775689210 (211h11m46.603397602s) [upload=219598]: Starting upload Dec 14 10:39:45.775750055 (211h11m46.603458455s) [upload=219598]: Done starting upload Dec 14 10:39:45.775957558 (211h11m46.603665941s) [upload=219598]: Uploading segments... Dec 14 10:39:45.776074427 (211h11m46.603782809s) [upload=219598]: Got next segment. Inline: false Dec 14 10:39:45.776158162 (211h11m46.603866553s) [upload=219598]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 14 10:39:45.776165965 (211h11m46.603874362s) [upload=219598]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 14 10:39:45.801257283 (211h11m46.628965678s) [seg_pos={0 0}, upload=219598]: Begin upload segment... Dec 14 10:39:45.801373021 (211h11m46.629081422s) [noise=true , address=145.79.227.43:4159 , piece_id=ULRDDEYCZXA26UPSFZQRQWFPU7MUCGJJSM7YWIR7HPXDCPBOQXNA, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.801409865 (211h11m46.629118247s) [seg_pos={0 0}, upload=219598]: Done begin upload segment. Dec 14 10:39:45.801481233 (211h11m46.629189635s) [noise=true , address=145.79.227.30:4763 , piece_id=CLMX6DHN5HP3OWMVR4R5MIVRH3GHCELUVTYJHBE7HXZQMHRQESIQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.802050295 (211h11m46.629758695s) [noise=true , address=145.79.227.32:4836 , piece_id=IQRH4CRVBOG72M3BE3G5FTIEMBTLD2KFUEYU6NCTK3N6PUONMY5A, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.802573060 (211h11m46.630281451s) [noise=true , address=77.90.31.14:4661 , piece_id=LLI4CDLWJ32TFXQ2YEU3NRP4QIRSNNUODA5RRPWU5AA45MJEW2LQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.803112997 (211h11m46.630821425s) [noise=true , address=145.79.227.44:4194 , piece_id=ZWOCXZ746BWDZ2LWYHSQHSG2O3WJC4R6FDL5E3JP2SXRCX3AWT2Q, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.803148596 (211h11m46.630856991s) [noise=true , address=145.79.227.46:4603 , piece_id=3X2DPA75MR7RX6LCOO2U3UREG4QSYHRRKRE57JUYUBINXTJN5IEA, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.803615052 (211h11m46.631323455s) [noise=true , address=145.79.227.28:4727 , piece_id=42ILJYV7IJRXSV3E3RWXWNUNYQHHQFNOF2JZGZKDEGPXRXZ3H2YA, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.803722621 (211h11m46.631431034s) [noise=true , address=77.90.31.16:4735 , piece_id=XKJJK6K7XF3TOFWMIU2BUVQTESTNV4FRYINSXJS4MC7KK7ZJU7SQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.804120053 (211h11m46.631828444s) [noise=true , address=145.79.227.7:4293 , piece_id=56YULKIWDM4P4WZM6KGMPUK3B3AK3RRVF4H6RJ7H6IQISPLXGBCQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.804570407 (211h11m46.632278798s) [noise=true , address=145.79.227.47:4267 , piece_id=VY46BNDUYVGBXDRIOEFK4USNOOJH2KXZH5Z2NHCT6SDJC5SFUYMQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.805083798 (211h11m46.632792196s) [noise=true , address=145.79.227.50:4375 , piece_id=RKVBJHHIF3KJLKB6ULTA4E5SHOBPCLPMWJCA6YTKF5XDNVEZMYXQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.805547276 (211h11m46.633255677s) [noise=true , address=77.90.31.16:4734 , piece_id=4JO2RBACNY5AM3KG7XUW7JTRGDCXGQDXSZ3L23ASAEJ4YFCICVAQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.805916642 (211h11m46.633625023s) [noise=true , address=145.79.227.39:4014 , piece_id=WKYZGQX5SP22ML66ZQZIHBS324EONJ7AARTYAXK2SCV577ZCQ5OQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.806440130 (211h11m46.634148525s) [noise=true , address=77.90.31.13:4626 , piece_id=DCKUBRB3DR755GN67RLP6NUMREAWIB3AITGPIMA7MM35BRPZW7IA, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.806976882 (211h11m46.634685284s) [noise=true , address=77.90.31.10:4519 , piece_id=ACPHBL7LF6NCNNH4U5GUXKW7RCL6XIEQVWYDCYUTXEC5RQE7RZBA, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.807729329 (211h11m46.63543772s) [noise=true , address=77.90.31.10:4518 , piece_id=ZE3SFGBCM2QV3CGGS3VEYT7NKDTDE2OJ6T6CVSGGYACCFX5WN5ZQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.841457783 (211h11m46.66916618s) [noise=true , address=77.90.31.13:4626 , piece_id=DCKUBRB3DR755GN67RLP6NUMREAWIB3AITGPIMA7MM35BRPZW7IA, seg_pos={0 0}, upload=219598]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: storage node overloaded, request limit: 2000 Dec 14 10:39:45.850206751 (211h11m46.677915151s) [noise=true , address=77.90.31.16:4734 , piece_id=4JO2RBACNY5AM3KG7XUW7JTRGDCXGQDXSZ3L23ASAEJ4YFCICVAQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: storage node overloaded, request limit: 2000 Dec 14 10:39:45.859644247 (211h11m46.68735265s) [noise=true , address=145.79.227.30:4763 , piece_id=CLMX6DHN5HP3OWMVR4R5MIVRH3GHCELUVTYJHBE7HXZQMHRQESIQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: storage node overloaded, request limit: 2000 Dec 14 10:39:45.874109133 (211h11m46.701817525s) [upload=219598]: Next returned nil segment Dec 14 10:39:45.874123719 (211h11m46.701832113s) [upload=219598]: Waiting for error group managing segments... Dec 14 10:39:45.880971826 (211h11m46.708680217s) [noise=true , address=77.90.31.10:4518 , piece_id=ZE3SFGBCM2QV3CGGS3VEYT7NKDTDE2OJ6T6CVSGGYACCFX5WN5ZQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.882649818 (211h11m46.710358212s) [noise=true , address=77.90.31.10:4519 , piece_id=ACPHBL7LF6NCNNH4U5GUXKW7RCL6XIEQVWYDCYUTXEC5RQE7RZBA, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.883088160 (211h11m46.71079655s) [noise=true , address=77.90.31.14:4661 , piece_id=LLI4CDLWJ32TFXQ2YEU3NRP4QIRSNNUODA5RRPWU5AA45MJEW2LQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.883693953 (211h11m46.711402345s) [noise=true , address=145.79.227.39:4014 , piece_id=WKYZGQX5SP22ML66ZQZIHBS324EONJ7AARTYAXK2SCV577ZCQ5OQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.884112448 (211h11m46.711820839s) [noise=true , address=145.79.227.32:4836 , piece_id=IQRH4CRVBOG72M3BE3G5FTIEMBTLD2KFUEYU6NCTK3N6PUONMY5A, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.884619811 (211h11m46.712328205s) [noise=true , address=145.79.227.43:4159 , piece_id=ULRDDEYCZXA26UPSFZQRQWFPU7MUCGJJSM7YWIR7HPXDCPBOQXNA, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.885012115 (211h11m46.712720506s) [noise=true , address=145.79.227.47:4267 , piece_id=VY46BNDUYVGBXDRIOEFK4USNOOJH2KXZH5Z2NHCT6SDJC5SFUYMQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.887669218 (211h11m46.71537761s) [noise=true , address=145.79.227.50:4375 , piece_id=RKVBJHHIF3KJLKB6ULTA4E5SHOBPCLPMWJCA6YTKF5XDNVEZMYXQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.890935473 (211h11m46.718643851s) [noise=true , address=145.79.227.46:4603 , piece_id=3X2DPA75MR7RX6LCOO2U3UREG4QSYHRRKRE57JUYUBINXTJN5IEA, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.891096730 (211h11m46.718805121s) [noise=true , address=145.79.227.28:4727 , piece_id=42ILJYV7IJRXSV3E3RWXWNUNYQHHQFNOF2JZGZKDEGPXRXZ3H2YA, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.893061640 (211h11m46.720770031s) [noise=true , address=145.79.227.7:4293 , piece_id=56YULKIWDM4P4WZM6KGMPUK3B3AK3RRVF4H6RJ7H6IQISPLXGBCQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:48.683585502 (211h11m49.511293883s) [noise=true , address=145.79.227.44:4194 , piece_id=ZWOCXZ746BWDZ2LWYHSQHSG2O3WJC4R6FDL5E3JP2SXRCX3AWT2Q, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:40:15.740699026 (211h12m16.568407426s) [noise=true , address=77.90.31.16:4735 , piece_id=XKJJK6K7XF3TOFWMIU2BUVQTESTNV4FRYINSXJS4MC7KK7ZJU7SQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: expected piece hash; context deadline exceeded; EOF Dec 14 10:40:15.740740404 (211h12m16.568448788s) [seg_pos={0 0}, upload=219598]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 14 10:40:15.762950810 (211h12m16.590659203s) [upload=219598]: Done uploading segments... Dec 14 10:40:15.762987936 (211h12m16.59069633s) [upload=219598]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 438.849247ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............................X............................] duration: 406.127544ms trace: https://jaeger.dev.storj.io/trace/25fec5475908806a FAIL (2025-12-14 10:58:35 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 14 10:39:45.775689210 (211h11m46.603397602s) [upload=219598]: Starting upload Dec 14 10:39:45.775750055 (211h11m46.603458455s) [upload=219598]: Done starting upload Dec 14 10:39:45.775957558 (211h11m46.603665941s) [upload=219598]: Uploading segments... Dec 14 10:39:45.776074427 (211h11m46.603782809s) [upload=219598]: Got next segment. Inline: false Dec 14 10:39:45.776158162 (211h11m46.603866553s) [upload=219598]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 14 10:39:45.776165965 (211h11m46.603874362s) [upload=219598]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 14 10:39:45.801257283 (211h11m46.628965678s) [seg_pos={0 0}, upload=219598]: Begin upload segment... Dec 14 10:39:45.801373021 (211h11m46.629081422s) [noise=true , address=145.79.227.43:4159 , piece_id=ULRDDEYCZXA26UPSFZQRQWFPU7MUCGJJSM7YWIR7HPXDCPBOQXNA, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.801409865 (211h11m46.629118247s) [seg_pos={0 0}, upload=219598]: Done begin upload segment. Dec 14 10:39:45.801481233 (211h11m46.629189635s) [noise=true , address=145.79.227.30:4763 , piece_id=CLMX6DHN5HP3OWMVR4R5MIVRH3GHCELUVTYJHBE7HXZQMHRQESIQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.802050295 (211h11m46.629758695s) [noise=true , address=145.79.227.32:4836 , piece_id=IQRH4CRVBOG72M3BE3G5FTIEMBTLD2KFUEYU6NCTK3N6PUONMY5A, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.802573060 (211h11m46.630281451s) [noise=true , address=77.90.31.14:4661 , piece_id=LLI4CDLWJ32TFXQ2YEU3NRP4QIRSNNUODA5RRPWU5AA45MJEW2LQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.803112997 (211h11m46.630821425s) [noise=true , address=145.79.227.44:4194 , piece_id=ZWOCXZ746BWDZ2LWYHSQHSG2O3WJC4R6FDL5E3JP2SXRCX3AWT2Q, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.803148596 (211h11m46.630856991s) [noise=true , address=145.79.227.46:4603 , piece_id=3X2DPA75MR7RX6LCOO2U3UREG4QSYHRRKRE57JUYUBINXTJN5IEA, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.803615052 (211h11m46.631323455s) [noise=true , address=145.79.227.28:4727 , piece_id=42ILJYV7IJRXSV3E3RWXWNUNYQHHQFNOF2JZGZKDEGPXRXZ3H2YA, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.803722621 (211h11m46.631431034s) [noise=true , address=77.90.31.16:4735 , piece_id=XKJJK6K7XF3TOFWMIU2BUVQTESTNV4FRYINSXJS4MC7KK7ZJU7SQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.804120053 (211h11m46.631828444s) [noise=true , address=145.79.227.7:4293 , piece_id=56YULKIWDM4P4WZM6KGMPUK3B3AK3RRVF4H6RJ7H6IQISPLXGBCQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.804570407 (211h11m46.632278798s) [noise=true , address=145.79.227.47:4267 , piece_id=VY46BNDUYVGBXDRIOEFK4USNOOJH2KXZH5Z2NHCT6SDJC5SFUYMQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.805083798 (211h11m46.632792196s) [noise=true , address=145.79.227.50:4375 , piece_id=RKVBJHHIF3KJLKB6ULTA4E5SHOBPCLPMWJCA6YTKF5XDNVEZMYXQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.805547276 (211h11m46.633255677s) [noise=true , address=77.90.31.16:4734 , piece_id=4JO2RBACNY5AM3KG7XUW7JTRGDCXGQDXSZ3L23ASAEJ4YFCICVAQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.805916642 (211h11m46.633625023s) [noise=true , address=145.79.227.39:4014 , piece_id=WKYZGQX5SP22ML66ZQZIHBS324EONJ7AARTYAXK2SCV577ZCQ5OQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.806440130 (211h11m46.634148525s) [noise=true , address=77.90.31.13:4626 , piece_id=DCKUBRB3DR755GN67RLP6NUMREAWIB3AITGPIMA7MM35BRPZW7IA, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.806976882 (211h11m46.634685284s) [noise=true , address=77.90.31.10:4519 , piece_id=ACPHBL7LF6NCNNH4U5GUXKW7RCL6XIEQVWYDCYUTXEC5RQE7RZBA, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.807729329 (211h11m46.63543772s) [noise=true , address=77.90.31.10:4518 , piece_id=ZE3SFGBCM2QV3CGGS3VEYT7NKDTDE2OJ6T6CVSGGYACCFX5WN5ZQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.841457783 (211h11m46.66916618s) [noise=true , address=77.90.31.13:4626 , piece_id=DCKUBRB3DR755GN67RLP6NUMREAWIB3AITGPIMA7MM35BRPZW7IA, seg_pos={0 0}, upload=219598]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: storage node overloaded, request limit: 2000 Dec 14 10:39:45.850206751 (211h11m46.677915151s) [noise=true , address=77.90.31.16:4734 , piece_id=4JO2RBACNY5AM3KG7XUW7JTRGDCXGQDXSZ3L23ASAEJ4YFCICVAQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: storage node overloaded, request limit: 2000 Dec 14 10:39:45.859644247 (211h11m46.68735265s) [noise=true , address=145.79.227.30:4763 , piece_id=CLMX6DHN5HP3OWMVR4R5MIVRH3GHCELUVTYJHBE7HXZQMHRQESIQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: storage node overloaded, request limit: 2000 Dec 14 10:39:45.874109133 (211h11m46.701817525s) [upload=219598]: Next returned nil segment Dec 14 10:39:45.874123719 (211h11m46.701832113s) [upload=219598]: Waiting for error group managing segments... Dec 14 10:39:45.880971826 (211h11m46.708680217s) [noise=true , address=77.90.31.10:4518 , piece_id=ZE3SFGBCM2QV3CGGS3VEYT7NKDTDE2OJ6T6CVSGGYACCFX5WN5ZQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.882649818 (211h11m46.710358212s) [noise=true , address=77.90.31.10:4519 , piece_id=ACPHBL7LF6NCNNH4U5GUXKW7RCL6XIEQVWYDCYUTXEC5RQE7RZBA, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.883088160 (211h11m46.71079655s) [noise=true , address=77.90.31.14:4661 , piece_id=LLI4CDLWJ32TFXQ2YEU3NRP4QIRSNNUODA5RRPWU5AA45MJEW2LQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.883693953 (211h11m46.711402345s) [noise=true , address=145.79.227.39:4014 , piece_id=WKYZGQX5SP22ML66ZQZIHBS324EONJ7AARTYAXK2SCV577ZCQ5OQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.884112448 (211h11m46.711820839s) [noise=true , address=145.79.227.32:4836 , piece_id=IQRH4CRVBOG72M3BE3G5FTIEMBTLD2KFUEYU6NCTK3N6PUONMY5A, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.884619811 (211h11m46.712328205s) [noise=true , address=145.79.227.43:4159 , piece_id=ULRDDEYCZXA26UPSFZQRQWFPU7MUCGJJSM7YWIR7HPXDCPBOQXNA, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.885012115 (211h11m46.712720506s) [noise=true , address=145.79.227.47:4267 , piece_id=VY46BNDUYVGBXDRIOEFK4USNOOJH2KXZH5Z2NHCT6SDJC5SFUYMQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.887669218 (211h11m46.71537761s) [noise=true , address=145.79.227.50:4375 , piece_id=RKVBJHHIF3KJLKB6ULTA4E5SHOBPCLPMWJCA6YTKF5XDNVEZMYXQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.890935473 (211h11m46.718643851s) [noise=true , address=145.79.227.46:4603 , piece_id=3X2DPA75MR7RX6LCOO2U3UREG4QSYHRRKRE57JUYUBINXTJN5IEA, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.891096730 (211h11m46.718805121s) [noise=true , address=145.79.227.28:4727 , piece_id=42ILJYV7IJRXSV3E3RWXWNUNYQHHQFNOF2JZGZKDEGPXRXZ3H2YA, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.893061640 (211h11m46.720770031s) [noise=true , address=145.79.227.7:4293 , piece_id=56YULKIWDM4P4WZM6KGMPUK3B3AK3RRVF4H6RJ7H6IQISPLXGBCQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:48.683585502 (211h11m49.511293883s) [noise=true , address=145.79.227.44:4194 , piece_id=ZWOCXZ746BWDZ2LWYHSQHSG2O3WJC4R6FDL5E3JP2SXRCX3AWT2Q, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:40:15.740699026 (211h12m16.568407426s) [noise=true , address=77.90.31.16:4735 , piece_id=XKJJK6K7XF3TOFWMIU2BUVQTESTNV4FRYINSXJS4MC7KK7ZJU7SQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: expected piece hash; context deadline exceeded; EOF Dec 14 10:40:15.740740404 (211h12m16.568448788s) [seg_pos={0 0}, upload=219598]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 14 10:40:15.762950810 (211h12m16.590659203s) [upload=219598]: Done uploading segments... Dec 14 10:40:15.762987936 (211h12m16.59069633s) [upload=219598]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 406.192244ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................................X...........................] duration: 385.360098ms trace: https://jaeger.dev.storj.io/trace/25fec5475908806a FAIL (2025-12-14 10:58:07 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 14 10:39:45.775689210 (211h11m46.603397602s) [upload=219598]: Starting upload Dec 14 10:39:45.775750055 (211h11m46.603458455s) [upload=219598]: Done starting upload Dec 14 10:39:45.775957558 (211h11m46.603665941s) [upload=219598]: Uploading segments... Dec 14 10:39:45.776074427 (211h11m46.603782809s) [upload=219598]: Got next segment. Inline: false Dec 14 10:39:45.776158162 (211h11m46.603866553s) [upload=219598]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 14 10:39:45.776165965 (211h11m46.603874362s) [upload=219598]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 14 10:39:45.801257283 (211h11m46.628965678s) [seg_pos={0 0}, upload=219598]: Begin upload segment... Dec 14 10:39:45.801373021 (211h11m46.629081422s) [noise=true , address=145.79.227.43:4159 , piece_id=ULRDDEYCZXA26UPSFZQRQWFPU7MUCGJJSM7YWIR7HPXDCPBOQXNA, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.801409865 (211h11m46.629118247s) [seg_pos={0 0}, upload=219598]: Done begin upload segment. Dec 14 10:39:45.801481233 (211h11m46.629189635s) [noise=true , address=145.79.227.30:4763 , piece_id=CLMX6DHN5HP3OWMVR4R5MIVRH3GHCELUVTYJHBE7HXZQMHRQESIQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.802050295 (211h11m46.629758695s) [noise=true , address=145.79.227.32:4836 , piece_id=IQRH4CRVBOG72M3BE3G5FTIEMBTLD2KFUEYU6NCTK3N6PUONMY5A, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.802573060 (211h11m46.630281451s) [noise=true , address=77.90.31.14:4661 , piece_id=LLI4CDLWJ32TFXQ2YEU3NRP4QIRSNNUODA5RRPWU5AA45MJEW2LQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.803112997 (211h11m46.630821425s) [noise=true , address=145.79.227.44:4194 , piece_id=ZWOCXZ746BWDZ2LWYHSQHSG2O3WJC4R6FDL5E3JP2SXRCX3AWT2Q, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.803148596 (211h11m46.630856991s) [noise=true , address=145.79.227.46:4603 , piece_id=3X2DPA75MR7RX6LCOO2U3UREG4QSYHRRKRE57JUYUBINXTJN5IEA, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.803615052 (211h11m46.631323455s) [noise=true , address=145.79.227.28:4727 , piece_id=42ILJYV7IJRXSV3E3RWXWNUNYQHHQFNOF2JZGZKDEGPXRXZ3H2YA, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.803722621 (211h11m46.631431034s) [noise=true , address=77.90.31.16:4735 , piece_id=XKJJK6K7XF3TOFWMIU2BUVQTESTNV4FRYINSXJS4MC7KK7ZJU7SQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.804120053 (211h11m46.631828444s) [noise=true , address=145.79.227.7:4293 , piece_id=56YULKIWDM4P4WZM6KGMPUK3B3AK3RRVF4H6RJ7H6IQISPLXGBCQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.804570407 (211h11m46.632278798s) [noise=true , address=145.79.227.47:4267 , piece_id=VY46BNDUYVGBXDRIOEFK4USNOOJH2KXZH5Z2NHCT6SDJC5SFUYMQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.805083798 (211h11m46.632792196s) [noise=true , address=145.79.227.50:4375 , piece_id=RKVBJHHIF3KJLKB6ULTA4E5SHOBPCLPMWJCA6YTKF5XDNVEZMYXQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.805547276 (211h11m46.633255677s) [noise=true , address=77.90.31.16:4734 , piece_id=4JO2RBACNY5AM3KG7XUW7JTRGDCXGQDXSZ3L23ASAEJ4YFCICVAQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.805916642 (211h11m46.633625023s) [noise=true , address=145.79.227.39:4014 , piece_id=WKYZGQX5SP22ML66ZQZIHBS324EONJ7AARTYAXK2SCV577ZCQ5OQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.806440130 (211h11m46.634148525s) [noise=true , address=77.90.31.13:4626 , piece_id=DCKUBRB3DR755GN67RLP6NUMREAWIB3AITGPIMA7MM35BRPZW7IA, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.806976882 (211h11m46.634685284s) [noise=true , address=77.90.31.10:4519 , piece_id=ACPHBL7LF6NCNNH4U5GUXKW7RCL6XIEQVWYDCYUTXEC5RQE7RZBA, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.807729329 (211h11m46.63543772s) [noise=true , address=77.90.31.10:4518 , piece_id=ZE3SFGBCM2QV3CGGS3VEYT7NKDTDE2OJ6T6CVSGGYACCFX5WN5ZQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.841457783 (211h11m46.66916618s) [noise=true , address=77.90.31.13:4626 , piece_id=DCKUBRB3DR755GN67RLP6NUMREAWIB3AITGPIMA7MM35BRPZW7IA, seg_pos={0 0}, upload=219598]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: storage node overloaded, request limit: 2000 Dec 14 10:39:45.850206751 (211h11m46.677915151s) [noise=true , address=77.90.31.16:4734 , piece_id=4JO2RBACNY5AM3KG7XUW7JTRGDCXGQDXSZ3L23ASAEJ4YFCICVAQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: storage node overloaded, request limit: 2000 Dec 14 10:39:45.859644247 (211h11m46.68735265s) [noise=true , address=145.79.227.30:4763 , piece_id=CLMX6DHN5HP3OWMVR4R5MIVRH3GHCELUVTYJHBE7HXZQMHRQESIQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: storage node overloaded, request limit: 2000 Dec 14 10:39:45.874109133 (211h11m46.701817525s) [upload=219598]: Next returned nil segment Dec 14 10:39:45.874123719 (211h11m46.701832113s) [upload=219598]: Waiting for error group managing segments... Dec 14 10:39:45.880971826 (211h11m46.708680217s) [noise=true , address=77.90.31.10:4518 , piece_id=ZE3SFGBCM2QV3CGGS3VEYT7NKDTDE2OJ6T6CVSGGYACCFX5WN5ZQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.882649818 (211h11m46.710358212s) [noise=true , address=77.90.31.10:4519 , piece_id=ACPHBL7LF6NCNNH4U5GUXKW7RCL6XIEQVWYDCYUTXEC5RQE7RZBA, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.883088160 (211h11m46.71079655s) [noise=true , address=77.90.31.14:4661 , piece_id=LLI4CDLWJ32TFXQ2YEU3NRP4QIRSNNUODA5RRPWU5AA45MJEW2LQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.883693953 (211h11m46.711402345s) [noise=true , address=145.79.227.39:4014 , piece_id=WKYZGQX5SP22ML66ZQZIHBS324EONJ7AARTYAXK2SCV577ZCQ5OQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.884112448 (211h11m46.711820839s) [noise=true , address=145.79.227.32:4836 , piece_id=IQRH4CRVBOG72M3BE3G5FTIEMBTLD2KFUEYU6NCTK3N6PUONMY5A, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.884619811 (211h11m46.712328205s) [noise=true , address=145.79.227.43:4159 , piece_id=ULRDDEYCZXA26UPSFZQRQWFPU7MUCGJJSM7YWIR7HPXDCPBOQXNA, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.885012115 (211h11m46.712720506s) [noise=true , address=145.79.227.47:4267 , piece_id=VY46BNDUYVGBXDRIOEFK4USNOOJH2KXZH5Z2NHCT6SDJC5SFUYMQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.887669218 (211h11m46.71537761s) [noise=true , address=145.79.227.50:4375 , piece_id=RKVBJHHIF3KJLKB6ULTA4E5SHOBPCLPMWJCA6YTKF5XDNVEZMYXQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.890935473 (211h11m46.718643851s) [noise=true , address=145.79.227.46:4603 , piece_id=3X2DPA75MR7RX6LCOO2U3UREG4QSYHRRKRE57JUYUBINXTJN5IEA, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.891096730 (211h11m46.718805121s) [noise=true , address=145.79.227.28:4727 , piece_id=42ILJYV7IJRXSV3E3RWXWNUNYQHHQFNOF2JZGZKDEGPXRXZ3H2YA, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.893061640 (211h11m46.720770031s) [noise=true , address=145.79.227.7:4293 , piece_id=56YULKIWDM4P4WZM6KGMPUK3B3AK3RRVF4H6RJ7H6IQISPLXGBCQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:48.683585502 (211h11m49.511293883s) [noise=true , address=145.79.227.44:4194 , piece_id=ZWOCXZ746BWDZ2LWYHSQHSG2O3WJC4R6FDL5E3JP2SXRCX3AWT2Q, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:40:15.740699026 (211h12m16.568407426s) [noise=true , address=77.90.31.16:4735 , piece_id=XKJJK6K7XF3TOFWMIU2BUVQTESTNV4FRYINSXJS4MC7KK7ZJU7SQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: expected piece hash; context deadline exceeded; EOF Dec 14 10:40:15.740740404 (211h12m16.568448788s) [seg_pos={0 0}, upload=219598]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 14 10:40:15.762950810 (211h12m16.590659203s) [upload=219598]: Done uploading segments... Dec 14 10:40:15.762987936 (211h12m16.59069633s) [upload=219598]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 385.383284ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................................X..........................] duration: 387.623265ms trace: https://jaeger.dev.storj.io/trace/25fec5475908806a FAIL (2025-12-14 10:57:50 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 14 10:39:45.775689210 (211h11m46.603397602s) [upload=219598]: Starting upload Dec 14 10:39:45.775750055 (211h11m46.603458455s) [upload=219598]: Done starting upload Dec 14 10:39:45.775957558 (211h11m46.603665941s) [upload=219598]: Uploading segments... Dec 14 10:39:45.776074427 (211h11m46.603782809s) [upload=219598]: Got next segment. Inline: false Dec 14 10:39:45.776158162 (211h11m46.603866553s) [upload=219598]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 14 10:39:45.776165965 (211h11m46.603874362s) [upload=219598]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 14 10:39:45.801257283 (211h11m46.628965678s) [seg_pos={0 0}, upload=219598]: Begin upload segment... Dec 14 10:39:45.801373021 (211h11m46.629081422s) [noise=true , address=145.79.227.43:4159 , piece_id=ULRDDEYCZXA26UPSFZQRQWFPU7MUCGJJSM7YWIR7HPXDCPBOQXNA, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.801409865 (211h11m46.629118247s) [seg_pos={0 0}, upload=219598]: Done begin upload segment. Dec 14 10:39:45.801481233 (211h11m46.629189635s) [noise=true , address=145.79.227.30:4763 , piece_id=CLMX6DHN5HP3OWMVR4R5MIVRH3GHCELUVTYJHBE7HXZQMHRQESIQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.802050295 (211h11m46.629758695s) [noise=true , address=145.79.227.32:4836 , piece_id=IQRH4CRVBOG72M3BE3G5FTIEMBTLD2KFUEYU6NCTK3N6PUONMY5A, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.802573060 (211h11m46.630281451s) [noise=true , address=77.90.31.14:4661 , piece_id=LLI4CDLWJ32TFXQ2YEU3NRP4QIRSNNUODA5RRPWU5AA45MJEW2LQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.803112997 (211h11m46.630821425s) [noise=true , address=145.79.227.44:4194 , piece_id=ZWOCXZ746BWDZ2LWYHSQHSG2O3WJC4R6FDL5E3JP2SXRCX3AWT2Q, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.803148596 (211h11m46.630856991s) [noise=true , address=145.79.227.46:4603 , piece_id=3X2DPA75MR7RX6LCOO2U3UREG4QSYHRRKRE57JUYUBINXTJN5IEA, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.803615052 (211h11m46.631323455s) [noise=true , address=145.79.227.28:4727 , piece_id=42ILJYV7IJRXSV3E3RWXWNUNYQHHQFNOF2JZGZKDEGPXRXZ3H2YA, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.803722621 (211h11m46.631431034s) [noise=true , address=77.90.31.16:4735 , piece_id=XKJJK6K7XF3TOFWMIU2BUVQTESTNV4FRYINSXJS4MC7KK7ZJU7SQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.804120053 (211h11m46.631828444s) [noise=true , address=145.79.227.7:4293 , piece_id=56YULKIWDM4P4WZM6KGMPUK3B3AK3RRVF4H6RJ7H6IQISPLXGBCQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.804570407 (211h11m46.632278798s) [noise=true , address=145.79.227.47:4267 , piece_id=VY46BNDUYVGBXDRIOEFK4USNOOJH2KXZH5Z2NHCT6SDJC5SFUYMQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.805083798 (211h11m46.632792196s) [noise=true , address=145.79.227.50:4375 , piece_id=RKVBJHHIF3KJLKB6ULTA4E5SHOBPCLPMWJCA6YTKF5XDNVEZMYXQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.805547276 (211h11m46.633255677s) [noise=true , address=77.90.31.16:4734 , piece_id=4JO2RBACNY5AM3KG7XUW7JTRGDCXGQDXSZ3L23ASAEJ4YFCICVAQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.805916642 (211h11m46.633625023s) [noise=true , address=145.79.227.39:4014 , piece_id=WKYZGQX5SP22ML66ZQZIHBS324EONJ7AARTYAXK2SCV577ZCQ5OQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.806440130 (211h11m46.634148525s) [noise=true , address=77.90.31.13:4626 , piece_id=DCKUBRB3DR755GN67RLP6NUMREAWIB3AITGPIMA7MM35BRPZW7IA, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.806976882 (211h11m46.634685284s) [noise=true , address=77.90.31.10:4519 , piece_id=ACPHBL7LF6NCNNH4U5GUXKW7RCL6XIEQVWYDCYUTXEC5RQE7RZBA, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.807729329 (211h11m46.63543772s) [noise=true , address=77.90.31.10:4518 , piece_id=ZE3SFGBCM2QV3CGGS3VEYT7NKDTDE2OJ6T6CVSGGYACCFX5WN5ZQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.841457783 (211h11m46.66916618s) [noise=true , address=77.90.31.13:4626 , piece_id=DCKUBRB3DR755GN67RLP6NUMREAWIB3AITGPIMA7MM35BRPZW7IA, seg_pos={0 0}, upload=219598]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: storage node overloaded, request limit: 2000 Dec 14 10:39:45.850206751 (211h11m46.677915151s) [noise=true , address=77.90.31.16:4734 , piece_id=4JO2RBACNY5AM3KG7XUW7JTRGDCXGQDXSZ3L23ASAEJ4YFCICVAQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: storage node overloaded, request limit: 2000 Dec 14 10:39:45.859644247 (211h11m46.68735265s) [noise=true , address=145.79.227.30:4763 , piece_id=CLMX6DHN5HP3OWMVR4R5MIVRH3GHCELUVTYJHBE7HXZQMHRQESIQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: storage node overloaded, request limit: 2000 Dec 14 10:39:45.874109133 (211h11m46.701817525s) [upload=219598]: Next returned nil segment Dec 14 10:39:45.874123719 (211h11m46.701832113s) [upload=219598]: Waiting for error group managing segments... Dec 14 10:39:45.880971826 (211h11m46.708680217s) [noise=true , address=77.90.31.10:4518 , piece_id=ZE3SFGBCM2QV3CGGS3VEYT7NKDTDE2OJ6T6CVSGGYACCFX5WN5ZQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.882649818 (211h11m46.710358212s) [noise=true , address=77.90.31.10:4519 , piece_id=ACPHBL7LF6NCNNH4U5GUXKW7RCL6XIEQVWYDCYUTXEC5RQE7RZBA, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.883088160 (211h11m46.71079655s) [noise=true , address=77.90.31.14:4661 , piece_id=LLI4CDLWJ32TFXQ2YEU3NRP4QIRSNNUODA5RRPWU5AA45MJEW2LQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.883693953 (211h11m46.711402345s) [noise=true , address=145.79.227.39:4014 , piece_id=WKYZGQX5SP22ML66ZQZIHBS324EONJ7AARTYAXK2SCV577ZCQ5OQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.884112448 (211h11m46.711820839s) [noise=true , address=145.79.227.32:4836 , piece_id=IQRH4CRVBOG72M3BE3G5FTIEMBTLD2KFUEYU6NCTK3N6PUONMY5A, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.884619811 (211h11m46.712328205s) [noise=true , address=145.79.227.43:4159 , piece_id=ULRDDEYCZXA26UPSFZQRQWFPU7MUCGJJSM7YWIR7HPXDCPBOQXNA, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.885012115 (211h11m46.712720506s) [noise=true , address=145.79.227.47:4267 , piece_id=VY46BNDUYVGBXDRIOEFK4USNOOJH2KXZH5Z2NHCT6SDJC5SFUYMQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.887669218 (211h11m46.71537761s) [noise=true , address=145.79.227.50:4375 , piece_id=RKVBJHHIF3KJLKB6ULTA4E5SHOBPCLPMWJCA6YTKF5XDNVEZMYXQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.890935473 (211h11m46.718643851s) [noise=true , address=145.79.227.46:4603 , piece_id=3X2DPA75MR7RX6LCOO2U3UREG4QSYHRRKRE57JUYUBINXTJN5IEA, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.891096730 (211h11m46.718805121s) [noise=true , address=145.79.227.28:4727 , piece_id=42ILJYV7IJRXSV3E3RWXWNUNYQHHQFNOF2JZGZKDEGPXRXZ3H2YA, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.893061640 (211h11m46.720770031s) [noise=true , address=145.79.227.7:4293 , piece_id=56YULKIWDM4P4WZM6KGMPUK3B3AK3RRVF4H6RJ7H6IQISPLXGBCQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:48.683585502 (211h11m49.511293883s) [noise=true , address=145.79.227.44:4194 , piece_id=ZWOCXZ746BWDZ2LWYHSQHSG2O3WJC4R6FDL5E3JP2SXRCX3AWT2Q, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:40:15.740699026 (211h12m16.568407426s) [noise=true , address=77.90.31.16:4735 , piece_id=XKJJK6K7XF3TOFWMIU2BUVQTESTNV4FRYINSXJS4MC7KK7ZJU7SQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: expected piece hash; context deadline exceeded; EOF Dec 14 10:40:15.740740404 (211h12m16.568448788s) [seg_pos={0 0}, upload=219598]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 14 10:40:15.762950810 (211h12m16.590659203s) [upload=219598]: Done uploading segments... Dec 14 10:40:15.762987936 (211h12m16.59069633s) [upload=219598]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 387.650265ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................................X.........................] duration: 414.140385ms trace: https://jaeger.dev.storj.io/trace/25fec5475908806a FAIL (2025-12-14 10:57:04 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 14 10:39:45.775689210 (211h11m46.603397602s) [upload=219598]: Starting upload Dec 14 10:39:45.775750055 (211h11m46.603458455s) [upload=219598]: Done starting upload Dec 14 10:39:45.775957558 (211h11m46.603665941s) [upload=219598]: Uploading segments... Dec 14 10:39:45.776074427 (211h11m46.603782809s) [upload=219598]: Got next segment. Inline: false Dec 14 10:39:45.776158162 (211h11m46.603866553s) [upload=219598]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 14 10:39:45.776165965 (211h11m46.603874362s) [upload=219598]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 14 10:39:45.801257283 (211h11m46.628965678s) [seg_pos={0 0}, upload=219598]: Begin upload segment... Dec 14 10:39:45.801373021 (211h11m46.629081422s) [noise=true , address=145.79.227.43:4159 , piece_id=ULRDDEYCZXA26UPSFZQRQWFPU7MUCGJJSM7YWIR7HPXDCPBOQXNA, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.801409865 (211h11m46.629118247s) [seg_pos={0 0}, upload=219598]: Done begin upload segment. Dec 14 10:39:45.801481233 (211h11m46.629189635s) [noise=true , address=145.79.227.30:4763 , piece_id=CLMX6DHN5HP3OWMVR4R5MIVRH3GHCELUVTYJHBE7HXZQMHRQESIQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.802050295 (211h11m46.629758695s) [noise=true , address=145.79.227.32:4836 , piece_id=IQRH4CRVBOG72M3BE3G5FTIEMBTLD2KFUEYU6NCTK3N6PUONMY5A, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.802573060 (211h11m46.630281451s) [noise=true , address=77.90.31.14:4661 , piece_id=LLI4CDLWJ32TFXQ2YEU3NRP4QIRSNNUODA5RRPWU5AA45MJEW2LQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.803112997 (211h11m46.630821425s) [noise=true , address=145.79.227.44:4194 , piece_id=ZWOCXZ746BWDZ2LWYHSQHSG2O3WJC4R6FDL5E3JP2SXRCX3AWT2Q, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.803148596 (211h11m46.630856991s) [noise=true , address=145.79.227.46:4603 , piece_id=3X2DPA75MR7RX6LCOO2U3UREG4QSYHRRKRE57JUYUBINXTJN5IEA, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.803615052 (211h11m46.631323455s) [noise=true , address=145.79.227.28:4727 , piece_id=42ILJYV7IJRXSV3E3RWXWNUNYQHHQFNOF2JZGZKDEGPXRXZ3H2YA, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.803722621 (211h11m46.631431034s) [noise=true , address=77.90.31.16:4735 , piece_id=XKJJK6K7XF3TOFWMIU2BUVQTESTNV4FRYINSXJS4MC7KK7ZJU7SQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.804120053 (211h11m46.631828444s) [noise=true , address=145.79.227.7:4293 , piece_id=56YULKIWDM4P4WZM6KGMPUK3B3AK3RRVF4H6RJ7H6IQISPLXGBCQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.804570407 (211h11m46.632278798s) [noise=true , address=145.79.227.47:4267 , piece_id=VY46BNDUYVGBXDRIOEFK4USNOOJH2KXZH5Z2NHCT6SDJC5SFUYMQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.805083798 (211h11m46.632792196s) [noise=true , address=145.79.227.50:4375 , piece_id=RKVBJHHIF3KJLKB6ULTA4E5SHOBPCLPMWJCA6YTKF5XDNVEZMYXQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.805547276 (211h11m46.633255677s) [noise=true , address=77.90.31.16:4734 , piece_id=4JO2RBACNY5AM3KG7XUW7JTRGDCXGQDXSZ3L23ASAEJ4YFCICVAQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.805916642 (211h11m46.633625023s) [noise=true , address=145.79.227.39:4014 , piece_id=WKYZGQX5SP22ML66ZQZIHBS324EONJ7AARTYAXK2SCV577ZCQ5OQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.806440130 (211h11m46.634148525s) [noise=true , address=77.90.31.13:4626 , piece_id=DCKUBRB3DR755GN67RLP6NUMREAWIB3AITGPIMA7MM35BRPZW7IA, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.806976882 (211h11m46.634685284s) [noise=true , address=77.90.31.10:4519 , piece_id=ACPHBL7LF6NCNNH4U5GUXKW7RCL6XIEQVWYDCYUTXEC5RQE7RZBA, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.807729329 (211h11m46.63543772s) [noise=true , address=77.90.31.10:4518 , piece_id=ZE3SFGBCM2QV3CGGS3VEYT7NKDTDE2OJ6T6CVSGGYACCFX5WN5ZQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.841457783 (211h11m46.66916618s) [noise=true , address=77.90.31.13:4626 , piece_id=DCKUBRB3DR755GN67RLP6NUMREAWIB3AITGPIMA7MM35BRPZW7IA, seg_pos={0 0}, upload=219598]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: storage node overloaded, request limit: 2000 Dec 14 10:39:45.850206751 (211h11m46.677915151s) [noise=true , address=77.90.31.16:4734 , piece_id=4JO2RBACNY5AM3KG7XUW7JTRGDCXGQDXSZ3L23ASAEJ4YFCICVAQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: storage node overloaded, request limit: 2000 Dec 14 10:39:45.859644247 (211h11m46.68735265s) [noise=true , address=145.79.227.30:4763 , piece_id=CLMX6DHN5HP3OWMVR4R5MIVRH3GHCELUVTYJHBE7HXZQMHRQESIQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: storage node overloaded, request limit: 2000 Dec 14 10:39:45.874109133 (211h11m46.701817525s) [upload=219598]: Next returned nil segment Dec 14 10:39:45.874123719 (211h11m46.701832113s) [upload=219598]: Waiting for error group managing segments... Dec 14 10:39:45.880971826 (211h11m46.708680217s) [noise=true , address=77.90.31.10:4518 , piece_id=ZE3SFGBCM2QV3CGGS3VEYT7NKDTDE2OJ6T6CVSGGYACCFX5WN5ZQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.882649818 (211h11m46.710358212s) [noise=true , address=77.90.31.10:4519 , piece_id=ACPHBL7LF6NCNNH4U5GUXKW7RCL6XIEQVWYDCYUTXEC5RQE7RZBA, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.883088160 (211h11m46.71079655s) [noise=true , address=77.90.31.14:4661 , piece_id=LLI4CDLWJ32TFXQ2YEU3NRP4QIRSNNUODA5RRPWU5AA45MJEW2LQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.883693953 (211h11m46.711402345s) [noise=true , address=145.79.227.39:4014 , piece_id=WKYZGQX5SP22ML66ZQZIHBS324EONJ7AARTYAXK2SCV577ZCQ5OQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.884112448 (211h11m46.711820839s) [noise=true , address=145.79.227.32:4836 , piece_id=IQRH4CRVBOG72M3BE3G5FTIEMBTLD2KFUEYU6NCTK3N6PUONMY5A, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.884619811 (211h11m46.712328205s) [noise=true , address=145.79.227.43:4159 , piece_id=ULRDDEYCZXA26UPSFZQRQWFPU7MUCGJJSM7YWIR7HPXDCPBOQXNA, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.885012115 (211h11m46.712720506s) [noise=true , address=145.79.227.47:4267 , piece_id=VY46BNDUYVGBXDRIOEFK4USNOOJH2KXZH5Z2NHCT6SDJC5SFUYMQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.887669218 (211h11m46.71537761s) [noise=true , address=145.79.227.50:4375 , piece_id=RKVBJHHIF3KJLKB6ULTA4E5SHOBPCLPMWJCA6YTKF5XDNVEZMYXQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.890935473 (211h11m46.718643851s) [noise=true , address=145.79.227.46:4603 , piece_id=3X2DPA75MR7RX6LCOO2U3UREG4QSYHRRKRE57JUYUBINXTJN5IEA, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.891096730 (211h11m46.718805121s) [noise=true , address=145.79.227.28:4727 , piece_id=42ILJYV7IJRXSV3E3RWXWNUNYQHHQFNOF2JZGZKDEGPXRXZ3H2YA, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.893061640 (211h11m46.720770031s) [noise=true , address=145.79.227.7:4293 , piece_id=56YULKIWDM4P4WZM6KGMPUK3B3AK3RRVF4H6RJ7H6IQISPLXGBCQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:48.683585502 (211h11m49.511293883s) [noise=true , address=145.79.227.44:4194 , piece_id=ZWOCXZ746BWDZ2LWYHSQHSG2O3WJC4R6FDL5E3JP2SXRCX3AWT2Q, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:40:15.740699026 (211h12m16.568407426s) [noise=true , address=77.90.31.16:4735 , piece_id=XKJJK6K7XF3TOFWMIU2BUVQTESTNV4FRYINSXJS4MC7KK7ZJU7SQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: expected piece hash; context deadline exceeded; EOF Dec 14 10:40:15.740740404 (211h12m16.568448788s) [seg_pos={0 0}, upload=219598]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 14 10:40:15.762950810 (211h12m16.590659203s) [upload=219598]: Done uploading segments... Dec 14 10:40:15.762987936 (211h12m16.59069633s) [upload=219598]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 414.184267ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................................X........................] duration: 522.239696ms trace: https://jaeger.dev.storj.io/trace/25fec5475908806a FAIL (2025-12-14 10:56:42 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 14 10:39:45.775689210 (211h11m46.603397602s) [upload=219598]: Starting upload Dec 14 10:39:45.775750055 (211h11m46.603458455s) [upload=219598]: Done starting upload Dec 14 10:39:45.775957558 (211h11m46.603665941s) [upload=219598]: Uploading segments... Dec 14 10:39:45.776074427 (211h11m46.603782809s) [upload=219598]: Got next segment. Inline: false Dec 14 10:39:45.776158162 (211h11m46.603866553s) [upload=219598]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 14 10:39:45.776165965 (211h11m46.603874362s) [upload=219598]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 14 10:39:45.801257283 (211h11m46.628965678s) [seg_pos={0 0}, upload=219598]: Begin upload segment... Dec 14 10:39:45.801373021 (211h11m46.629081422s) [noise=true , address=145.79.227.43:4159 , piece_id=ULRDDEYCZXA26UPSFZQRQWFPU7MUCGJJSM7YWIR7HPXDCPBOQXNA, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.801409865 (211h11m46.629118247s) [seg_pos={0 0}, upload=219598]: Done begin upload segment. Dec 14 10:39:45.801481233 (211h11m46.629189635s) [noise=true , address=145.79.227.30:4763 , piece_id=CLMX6DHN5HP3OWMVR4R5MIVRH3GHCELUVTYJHBE7HXZQMHRQESIQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.802050295 (211h11m46.629758695s) [noise=true , address=145.79.227.32:4836 , piece_id=IQRH4CRVBOG72M3BE3G5FTIEMBTLD2KFUEYU6NCTK3N6PUONMY5A, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.802573060 (211h11m46.630281451s) [noise=true , address=77.90.31.14:4661 , piece_id=LLI4CDLWJ32TFXQ2YEU3NRP4QIRSNNUODA5RRPWU5AA45MJEW2LQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.803112997 (211h11m46.630821425s) [noise=true , address=145.79.227.44:4194 , piece_id=ZWOCXZ746BWDZ2LWYHSQHSG2O3WJC4R6FDL5E3JP2SXRCX3AWT2Q, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.803148596 (211h11m46.630856991s) [noise=true , address=145.79.227.46:4603 , piece_id=3X2DPA75MR7RX6LCOO2U3UREG4QSYHRRKRE57JUYUBINXTJN5IEA, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.803615052 (211h11m46.631323455s) [noise=true , address=145.79.227.28:4727 , piece_id=42ILJYV7IJRXSV3E3RWXWNUNYQHHQFNOF2JZGZKDEGPXRXZ3H2YA, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.803722621 (211h11m46.631431034s) [noise=true , address=77.90.31.16:4735 , piece_id=XKJJK6K7XF3TOFWMIU2BUVQTESTNV4FRYINSXJS4MC7KK7ZJU7SQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.804120053 (211h11m46.631828444s) [noise=true , address=145.79.227.7:4293 , piece_id=56YULKIWDM4P4WZM6KGMPUK3B3AK3RRVF4H6RJ7H6IQISPLXGBCQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.804570407 (211h11m46.632278798s) [noise=true , address=145.79.227.47:4267 , piece_id=VY46BNDUYVGBXDRIOEFK4USNOOJH2KXZH5Z2NHCT6SDJC5SFUYMQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.805083798 (211h11m46.632792196s) [noise=true , address=145.79.227.50:4375 , piece_id=RKVBJHHIF3KJLKB6ULTA4E5SHOBPCLPMWJCA6YTKF5XDNVEZMYXQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.805547276 (211h11m46.633255677s) [noise=true , address=77.90.31.16:4734 , piece_id=4JO2RBACNY5AM3KG7XUW7JTRGDCXGQDXSZ3L23ASAEJ4YFCICVAQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.805916642 (211h11m46.633625023s) [noise=true , address=145.79.227.39:4014 , piece_id=WKYZGQX5SP22ML66ZQZIHBS324EONJ7AARTYAXK2SCV577ZCQ5OQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.806440130 (211h11m46.634148525s) [noise=true , address=77.90.31.13:4626 , piece_id=DCKUBRB3DR755GN67RLP6NUMREAWIB3AITGPIMA7MM35BRPZW7IA, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.806976882 (211h11m46.634685284s) [noise=true , address=77.90.31.10:4519 , piece_id=ACPHBL7LF6NCNNH4U5GUXKW7RCL6XIEQVWYDCYUTXEC5RQE7RZBA, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.807729329 (211h11m46.63543772s) [noise=true , address=77.90.31.10:4518 , piece_id=ZE3SFGBCM2QV3CGGS3VEYT7NKDTDE2OJ6T6CVSGGYACCFX5WN5ZQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.841457783 (211h11m46.66916618s) [noise=true , address=77.90.31.13:4626 , piece_id=DCKUBRB3DR755GN67RLP6NUMREAWIB3AITGPIMA7MM35BRPZW7IA, seg_pos={0 0}, upload=219598]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: storage node overloaded, request limit: 2000 Dec 14 10:39:45.850206751 (211h11m46.677915151s) [noise=true , address=77.90.31.16:4734 , piece_id=4JO2RBACNY5AM3KG7XUW7JTRGDCXGQDXSZ3L23ASAEJ4YFCICVAQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: storage node overloaded, request limit: 2000 Dec 14 10:39:45.859644247 (211h11m46.68735265s) [noise=true , address=145.79.227.30:4763 , piece_id=CLMX6DHN5HP3OWMVR4R5MIVRH3GHCELUVTYJHBE7HXZQMHRQESIQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: storage node overloaded, request limit: 2000 Dec 14 10:39:45.874109133 (211h11m46.701817525s) [upload=219598]: Next returned nil segment Dec 14 10:39:45.874123719 (211h11m46.701832113s) [upload=219598]: Waiting for error group managing segments... Dec 14 10:39:45.880971826 (211h11m46.708680217s) [noise=true , address=77.90.31.10:4518 , piece_id=ZE3SFGBCM2QV3CGGS3VEYT7NKDTDE2OJ6T6CVSGGYACCFX5WN5ZQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.882649818 (211h11m46.710358212s) [noise=true , address=77.90.31.10:4519 , piece_id=ACPHBL7LF6NCNNH4U5GUXKW7RCL6XIEQVWYDCYUTXEC5RQE7RZBA, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.883088160 (211h11m46.71079655s) [noise=true , address=77.90.31.14:4661 , piece_id=LLI4CDLWJ32TFXQ2YEU3NRP4QIRSNNUODA5RRPWU5AA45MJEW2LQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.883693953 (211h11m46.711402345s) [noise=true , address=145.79.227.39:4014 , piece_id=WKYZGQX5SP22ML66ZQZIHBS324EONJ7AARTYAXK2SCV577ZCQ5OQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.884112448 (211h11m46.711820839s) [noise=true , address=145.79.227.32:4836 , piece_id=IQRH4CRVBOG72M3BE3G5FTIEMBTLD2KFUEYU6NCTK3N6PUONMY5A, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.884619811 (211h11m46.712328205s) [noise=true , address=145.79.227.43:4159 , piece_id=ULRDDEYCZXA26UPSFZQRQWFPU7MUCGJJSM7YWIR7HPXDCPBOQXNA, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.885012115 (211h11m46.712720506s) [noise=true , address=145.79.227.47:4267 , piece_id=VY46BNDUYVGBXDRIOEFK4USNOOJH2KXZH5Z2NHCT6SDJC5SFUYMQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.887669218 (211h11m46.71537761s) [noise=true , address=145.79.227.50:4375 , piece_id=RKVBJHHIF3KJLKB6ULTA4E5SHOBPCLPMWJCA6YTKF5XDNVEZMYXQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.890935473 (211h11m46.718643851s) [noise=true , address=145.79.227.46:4603 , piece_id=3X2DPA75MR7RX6LCOO2U3UREG4QSYHRRKRE57JUYUBINXTJN5IEA, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.891096730 (211h11m46.718805121s) [noise=true , address=145.79.227.28:4727 , piece_id=42ILJYV7IJRXSV3E3RWXWNUNYQHHQFNOF2JZGZKDEGPXRXZ3H2YA, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.893061640 (211h11m46.720770031s) [noise=true , address=145.79.227.7:4293 , piece_id=56YULKIWDM4P4WZM6KGMPUK3B3AK3RRVF4H6RJ7H6IQISPLXGBCQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:48.683585502 (211h11m49.511293883s) [noise=true , address=145.79.227.44:4194 , piece_id=ZWOCXZ746BWDZ2LWYHSQHSG2O3WJC4R6FDL5E3JP2SXRCX3AWT2Q, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:40:15.740699026 (211h12m16.568407426s) [noise=true , address=77.90.31.16:4735 , piece_id=XKJJK6K7XF3TOFWMIU2BUVQTESTNV4FRYINSXJS4MC7KK7ZJU7SQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: expected piece hash; context deadline exceeded; EOF Dec 14 10:40:15.740740404 (211h12m16.568448788s) [seg_pos={0 0}, upload=219598]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 14 10:40:15.762950810 (211h12m16.590659203s) [upload=219598]: Done uploading segments... Dec 14 10:40:15.762987936 (211h12m16.59069633s) [upload=219598]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 522.27486ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................................X.......................] duration: 381.660884ms trace: https://jaeger.dev.storj.io/trace/25fec5475908806a FAIL (2025-12-14 10:56:10 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 14 10:39:45.775689210 (211h11m46.603397602s) [upload=219598]: Starting upload Dec 14 10:39:45.775750055 (211h11m46.603458455s) [upload=219598]: Done starting upload Dec 14 10:39:45.775957558 (211h11m46.603665941s) [upload=219598]: Uploading segments... Dec 14 10:39:45.776074427 (211h11m46.603782809s) [upload=219598]: Got next segment. Inline: false Dec 14 10:39:45.776158162 (211h11m46.603866553s) [upload=219598]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 14 10:39:45.776165965 (211h11m46.603874362s) [upload=219598]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 14 10:39:45.801257283 (211h11m46.628965678s) [seg_pos={0 0}, upload=219598]: Begin upload segment... Dec 14 10:39:45.801373021 (211h11m46.629081422s) [noise=true , address=145.79.227.43:4159 , piece_id=ULRDDEYCZXA26UPSFZQRQWFPU7MUCGJJSM7YWIR7HPXDCPBOQXNA, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.801409865 (211h11m46.629118247s) [seg_pos={0 0}, upload=219598]: Done begin upload segment. Dec 14 10:39:45.801481233 (211h11m46.629189635s) [noise=true , address=145.79.227.30:4763 , piece_id=CLMX6DHN5HP3OWMVR4R5MIVRH3GHCELUVTYJHBE7HXZQMHRQESIQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.802050295 (211h11m46.629758695s) [noise=true , address=145.79.227.32:4836 , piece_id=IQRH4CRVBOG72M3BE3G5FTIEMBTLD2KFUEYU6NCTK3N6PUONMY5A, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.802573060 (211h11m46.630281451s) [noise=true , address=77.90.31.14:4661 , piece_id=LLI4CDLWJ32TFXQ2YEU3NRP4QIRSNNUODA5RRPWU5AA45MJEW2LQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.803112997 (211h11m46.630821425s) [noise=true , address=145.79.227.44:4194 , piece_id=ZWOCXZ746BWDZ2LWYHSQHSG2O3WJC4R6FDL5E3JP2SXRCX3AWT2Q, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.803148596 (211h11m46.630856991s) [noise=true , address=145.79.227.46:4603 , piece_id=3X2DPA75MR7RX6LCOO2U3UREG4QSYHRRKRE57JUYUBINXTJN5IEA, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.803615052 (211h11m46.631323455s) [noise=true , address=145.79.227.28:4727 , piece_id=42ILJYV7IJRXSV3E3RWXWNUNYQHHQFNOF2JZGZKDEGPXRXZ3H2YA, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.803722621 (211h11m46.631431034s) [noise=true , address=77.90.31.16:4735 , piece_id=XKJJK6K7XF3TOFWMIU2BUVQTESTNV4FRYINSXJS4MC7KK7ZJU7SQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.804120053 (211h11m46.631828444s) [noise=true , address=145.79.227.7:4293 , piece_id=56YULKIWDM4P4WZM6KGMPUK3B3AK3RRVF4H6RJ7H6IQISPLXGBCQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.804570407 (211h11m46.632278798s) [noise=true , address=145.79.227.47:4267 , piece_id=VY46BNDUYVGBXDRIOEFK4USNOOJH2KXZH5Z2NHCT6SDJC5SFUYMQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.805083798 (211h11m46.632792196s) [noise=true , address=145.79.227.50:4375 , piece_id=RKVBJHHIF3KJLKB6ULTA4E5SHOBPCLPMWJCA6YTKF5XDNVEZMYXQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.805547276 (211h11m46.633255677s) [noise=true , address=77.90.31.16:4734 , piece_id=4JO2RBACNY5AM3KG7XUW7JTRGDCXGQDXSZ3L23ASAEJ4YFCICVAQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.805916642 (211h11m46.633625023s) [noise=true , address=145.79.227.39:4014 , piece_id=WKYZGQX5SP22ML66ZQZIHBS324EONJ7AARTYAXK2SCV577ZCQ5OQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.806440130 (211h11m46.634148525s) [noise=true , address=77.90.31.13:4626 , piece_id=DCKUBRB3DR755GN67RLP6NUMREAWIB3AITGPIMA7MM35BRPZW7IA, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.806976882 (211h11m46.634685284s) [noise=true , address=77.90.31.10:4519 , piece_id=ACPHBL7LF6NCNNH4U5GUXKW7RCL6XIEQVWYDCYUTXEC5RQE7RZBA, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.807729329 (211h11m46.63543772s) [noise=true , address=77.90.31.10:4518 , piece_id=ZE3SFGBCM2QV3CGGS3VEYT7NKDTDE2OJ6T6CVSGGYACCFX5WN5ZQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.841457783 (211h11m46.66916618s) [noise=true , address=77.90.31.13:4626 , piece_id=DCKUBRB3DR755GN67RLP6NUMREAWIB3AITGPIMA7MM35BRPZW7IA, seg_pos={0 0}, upload=219598]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: storage node overloaded, request limit: 2000 Dec 14 10:39:45.850206751 (211h11m46.677915151s) [noise=true , address=77.90.31.16:4734 , piece_id=4JO2RBACNY5AM3KG7XUW7JTRGDCXGQDXSZ3L23ASAEJ4YFCICVAQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: storage node overloaded, request limit: 2000 Dec 14 10:39:45.859644247 (211h11m46.68735265s) [noise=true , address=145.79.227.30:4763 , piece_id=CLMX6DHN5HP3OWMVR4R5MIVRH3GHCELUVTYJHBE7HXZQMHRQESIQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: storage node overloaded, request limit: 2000 Dec 14 10:39:45.874109133 (211h11m46.701817525s) [upload=219598]: Next returned nil segment Dec 14 10:39:45.874123719 (211h11m46.701832113s) [upload=219598]: Waiting for error group managing segments... Dec 14 10:39:45.880971826 (211h11m46.708680217s) [noise=true , address=77.90.31.10:4518 , piece_id=ZE3SFGBCM2QV3CGGS3VEYT7NKDTDE2OJ6T6CVSGGYACCFX5WN5ZQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.882649818 (211h11m46.710358212s) [noise=true , address=77.90.31.10:4519 , piece_id=ACPHBL7LF6NCNNH4U5GUXKW7RCL6XIEQVWYDCYUTXEC5RQE7RZBA, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.883088160 (211h11m46.71079655s) [noise=true , address=77.90.31.14:4661 , piece_id=LLI4CDLWJ32TFXQ2YEU3NRP4QIRSNNUODA5RRPWU5AA45MJEW2LQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.883693953 (211h11m46.711402345s) [noise=true , address=145.79.227.39:4014 , piece_id=WKYZGQX5SP22ML66ZQZIHBS324EONJ7AARTYAXK2SCV577ZCQ5OQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.884112448 (211h11m46.711820839s) [noise=true , address=145.79.227.32:4836 , piece_id=IQRH4CRVBOG72M3BE3G5FTIEMBTLD2KFUEYU6NCTK3N6PUONMY5A, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.884619811 (211h11m46.712328205s) [noise=true , address=145.79.227.43:4159 , piece_id=ULRDDEYCZXA26UPSFZQRQWFPU7MUCGJJSM7YWIR7HPXDCPBOQXNA, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.885012115 (211h11m46.712720506s) [noise=true , address=145.79.227.47:4267 , piece_id=VY46BNDUYVGBXDRIOEFK4USNOOJH2KXZH5Z2NHCT6SDJC5SFUYMQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.887669218 (211h11m46.71537761s) [noise=true , address=145.79.227.50:4375 , piece_id=RKVBJHHIF3KJLKB6ULTA4E5SHOBPCLPMWJCA6YTKF5XDNVEZMYXQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.890935473 (211h11m46.718643851s) [noise=true , address=145.79.227.46:4603 , piece_id=3X2DPA75MR7RX6LCOO2U3UREG4QSYHRRKRE57JUYUBINXTJN5IEA, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.891096730 (211h11m46.718805121s) [noise=true , address=145.79.227.28:4727 , piece_id=42ILJYV7IJRXSV3E3RWXWNUNYQHHQFNOF2JZGZKDEGPXRXZ3H2YA, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.893061640 (211h11m46.720770031s) [noise=true , address=145.79.227.7:4293 , piece_id=56YULKIWDM4P4WZM6KGMPUK3B3AK3RRVF4H6RJ7H6IQISPLXGBCQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:48.683585502 (211h11m49.511293883s) [noise=true , address=145.79.227.44:4194 , piece_id=ZWOCXZ746BWDZ2LWYHSQHSG2O3WJC4R6FDL5E3JP2SXRCX3AWT2Q, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:40:15.740699026 (211h12m16.568407426s) [noise=true , address=77.90.31.16:4735 , piece_id=XKJJK6K7XF3TOFWMIU2BUVQTESTNV4FRYINSXJS4MC7KK7ZJU7SQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: expected piece hash; context deadline exceeded; EOF Dec 14 10:40:15.740740404 (211h12m16.568448788s) [seg_pos={0 0}, upload=219598]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 14 10:40:15.762950810 (211h12m16.590659203s) [upload=219598]: Done uploading segments... Dec 14 10:40:15.762987936 (211h12m16.59069633s) [upload=219598]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 381.681478ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................................X......................] duration: 387.008823ms trace: https://jaeger.dev.storj.io/trace/25fec5475908806a FAIL (2025-12-14 10:55:49 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 14 10:39:45.775689210 (211h11m46.603397602s) [upload=219598]: Starting upload Dec 14 10:39:45.775750055 (211h11m46.603458455s) [upload=219598]: Done starting upload Dec 14 10:39:45.775957558 (211h11m46.603665941s) [upload=219598]: Uploading segments... Dec 14 10:39:45.776074427 (211h11m46.603782809s) [upload=219598]: Got next segment. Inline: false Dec 14 10:39:45.776158162 (211h11m46.603866553s) [upload=219598]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 14 10:39:45.776165965 (211h11m46.603874362s) [upload=219598]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 14 10:39:45.801257283 (211h11m46.628965678s) [seg_pos={0 0}, upload=219598]: Begin upload segment... Dec 14 10:39:45.801373021 (211h11m46.629081422s) [noise=true , address=145.79.227.43:4159 , piece_id=ULRDDEYCZXA26UPSFZQRQWFPU7MUCGJJSM7YWIR7HPXDCPBOQXNA, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.801409865 (211h11m46.629118247s) [seg_pos={0 0}, upload=219598]: Done begin upload segment. Dec 14 10:39:45.801481233 (211h11m46.629189635s) [noise=true , address=145.79.227.30:4763 , piece_id=CLMX6DHN5HP3OWMVR4R5MIVRH3GHCELUVTYJHBE7HXZQMHRQESIQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.802050295 (211h11m46.629758695s) [noise=true , address=145.79.227.32:4836 , piece_id=IQRH4CRVBOG72M3BE3G5FTIEMBTLD2KFUEYU6NCTK3N6PUONMY5A, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.802573060 (211h11m46.630281451s) [noise=true , address=77.90.31.14:4661 , piece_id=LLI4CDLWJ32TFXQ2YEU3NRP4QIRSNNUODA5RRPWU5AA45MJEW2LQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.803112997 (211h11m46.630821425s) [noise=true , address=145.79.227.44:4194 , piece_id=ZWOCXZ746BWDZ2LWYHSQHSG2O3WJC4R6FDL5E3JP2SXRCX3AWT2Q, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.803148596 (211h11m46.630856991s) [noise=true , address=145.79.227.46:4603 , piece_id=3X2DPA75MR7RX6LCOO2U3UREG4QSYHRRKRE57JUYUBINXTJN5IEA, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.803615052 (211h11m46.631323455s) [noise=true , address=145.79.227.28:4727 , piece_id=42ILJYV7IJRXSV3E3RWXWNUNYQHHQFNOF2JZGZKDEGPXRXZ3H2YA, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.803722621 (211h11m46.631431034s) [noise=true , address=77.90.31.16:4735 , piece_id=XKJJK6K7XF3TOFWMIU2BUVQTESTNV4FRYINSXJS4MC7KK7ZJU7SQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.804120053 (211h11m46.631828444s) [noise=true , address=145.79.227.7:4293 , piece_id=56YULKIWDM4P4WZM6KGMPUK3B3AK3RRVF4H6RJ7H6IQISPLXGBCQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.804570407 (211h11m46.632278798s) [noise=true , address=145.79.227.47:4267 , piece_id=VY46BNDUYVGBXDRIOEFK4USNOOJH2KXZH5Z2NHCT6SDJC5SFUYMQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.805083798 (211h11m46.632792196s) [noise=true , address=145.79.227.50:4375 , piece_id=RKVBJHHIF3KJLKB6ULTA4E5SHOBPCLPMWJCA6YTKF5XDNVEZMYXQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.805547276 (211h11m46.633255677s) [noise=true , address=77.90.31.16:4734 , piece_id=4JO2RBACNY5AM3KG7XUW7JTRGDCXGQDXSZ3L23ASAEJ4YFCICVAQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.805916642 (211h11m46.633625023s) [noise=true , address=145.79.227.39:4014 , piece_id=WKYZGQX5SP22ML66ZQZIHBS324EONJ7AARTYAXK2SCV577ZCQ5OQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.806440130 (211h11m46.634148525s) [noise=true , address=77.90.31.13:4626 , piece_id=DCKUBRB3DR755GN67RLP6NUMREAWIB3AITGPIMA7MM35BRPZW7IA, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.806976882 (211h11m46.634685284s) [noise=true , address=77.90.31.10:4519 , piece_id=ACPHBL7LF6NCNNH4U5GUXKW7RCL6XIEQVWYDCYUTXEC5RQE7RZBA, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.807729329 (211h11m46.63543772s) [noise=true , address=77.90.31.10:4518 , piece_id=ZE3SFGBCM2QV3CGGS3VEYT7NKDTDE2OJ6T6CVSGGYACCFX5WN5ZQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.841457783 (211h11m46.66916618s) [noise=true , address=77.90.31.13:4626 , piece_id=DCKUBRB3DR755GN67RLP6NUMREAWIB3AITGPIMA7MM35BRPZW7IA, seg_pos={0 0}, upload=219598]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: storage node overloaded, request limit: 2000 Dec 14 10:39:45.850206751 (211h11m46.677915151s) [noise=true , address=77.90.31.16:4734 , piece_id=4JO2RBACNY5AM3KG7XUW7JTRGDCXGQDXSZ3L23ASAEJ4YFCICVAQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: storage node overloaded, request limit: 2000 Dec 14 10:39:45.859644247 (211h11m46.68735265s) [noise=true , address=145.79.227.30:4763 , piece_id=CLMX6DHN5HP3OWMVR4R5MIVRH3GHCELUVTYJHBE7HXZQMHRQESIQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: storage node overloaded, request limit: 2000 Dec 14 10:39:45.874109133 (211h11m46.701817525s) [upload=219598]: Next returned nil segment Dec 14 10:39:45.874123719 (211h11m46.701832113s) [upload=219598]: Waiting for error group managing segments... Dec 14 10:39:45.880971826 (211h11m46.708680217s) [noise=true , address=77.90.31.10:4518 , piece_id=ZE3SFGBCM2QV3CGGS3VEYT7NKDTDE2OJ6T6CVSGGYACCFX5WN5ZQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.882649818 (211h11m46.710358212s) [noise=true , address=77.90.31.10:4519 , piece_id=ACPHBL7LF6NCNNH4U5GUXKW7RCL6XIEQVWYDCYUTXEC5RQE7RZBA, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.883088160 (211h11m46.71079655s) [noise=true , address=77.90.31.14:4661 , piece_id=LLI4CDLWJ32TFXQ2YEU3NRP4QIRSNNUODA5RRPWU5AA45MJEW2LQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.883693953 (211h11m46.711402345s) [noise=true , address=145.79.227.39:4014 , piece_id=WKYZGQX5SP22ML66ZQZIHBS324EONJ7AARTYAXK2SCV577ZCQ5OQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.884112448 (211h11m46.711820839s) [noise=true , address=145.79.227.32:4836 , piece_id=IQRH4CRVBOG72M3BE3G5FTIEMBTLD2KFUEYU6NCTK3N6PUONMY5A, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.884619811 (211h11m46.712328205s) [noise=true , address=145.79.227.43:4159 , piece_id=ULRDDEYCZXA26UPSFZQRQWFPU7MUCGJJSM7YWIR7HPXDCPBOQXNA, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.885012115 (211h11m46.712720506s) [noise=true , address=145.79.227.47:4267 , piece_id=VY46BNDUYVGBXDRIOEFK4USNOOJH2KXZH5Z2NHCT6SDJC5SFUYMQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.887669218 (211h11m46.71537761s) [noise=true , address=145.79.227.50:4375 , piece_id=RKVBJHHIF3KJLKB6ULTA4E5SHOBPCLPMWJCA6YTKF5XDNVEZMYXQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.890935473 (211h11m46.718643851s) [noise=true , address=145.79.227.46:4603 , piece_id=3X2DPA75MR7RX6LCOO2U3UREG4QSYHRRKRE57JUYUBINXTJN5IEA, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.891096730 (211h11m46.718805121s) [noise=true , address=145.79.227.28:4727 , piece_id=42ILJYV7IJRXSV3E3RWXWNUNYQHHQFNOF2JZGZKDEGPXRXZ3H2YA, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.893061640 (211h11m46.720770031s) [noise=true , address=145.79.227.7:4293 , piece_id=56YULKIWDM4P4WZM6KGMPUK3B3AK3RRVF4H6RJ7H6IQISPLXGBCQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:48.683585502 (211h11m49.511293883s) [noise=true , address=145.79.227.44:4194 , piece_id=ZWOCXZ746BWDZ2LWYHSQHSG2O3WJC4R6FDL5E3JP2SXRCX3AWT2Q, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:40:15.740699026 (211h12m16.568407426s) [noise=true , address=77.90.31.16:4735 , piece_id=XKJJK6K7XF3TOFWMIU2BUVQTESTNV4FRYINSXJS4MC7KK7ZJU7SQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: expected piece hash; context deadline exceeded; EOF Dec 14 10:40:15.740740404 (211h12m16.568448788s) [seg_pos={0 0}, upload=219598]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 14 10:40:15.762950810 (211h12m16.590659203s) [upload=219598]: Done uploading segments... Dec 14 10:40:15.762987936 (211h12m16.59069633s) [upload=219598]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 387.031932ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................................X.....................] duration: 579.383721ms trace: https://jaeger.dev.storj.io/trace/25fec5475908806a FAIL (2025-12-14 10:54:47 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 14 10:39:45.775689210 (211h11m46.603397602s) [upload=219598]: Starting upload Dec 14 10:39:45.775750055 (211h11m46.603458455s) [upload=219598]: Done starting upload Dec 14 10:39:45.775957558 (211h11m46.603665941s) [upload=219598]: Uploading segments... Dec 14 10:39:45.776074427 (211h11m46.603782809s) [upload=219598]: Got next segment. Inline: false Dec 14 10:39:45.776158162 (211h11m46.603866553s) [upload=219598]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 14 10:39:45.776165965 (211h11m46.603874362s) [upload=219598]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 14 10:39:45.801257283 (211h11m46.628965678s) [seg_pos={0 0}, upload=219598]: Begin upload segment... Dec 14 10:39:45.801373021 (211h11m46.629081422s) [noise=true , address=145.79.227.43:4159 , piece_id=ULRDDEYCZXA26UPSFZQRQWFPU7MUCGJJSM7YWIR7HPXDCPBOQXNA, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.801409865 (211h11m46.629118247s) [seg_pos={0 0}, upload=219598]: Done begin upload segment. Dec 14 10:39:45.801481233 (211h11m46.629189635s) [noise=true , address=145.79.227.30:4763 , piece_id=CLMX6DHN5HP3OWMVR4R5MIVRH3GHCELUVTYJHBE7HXZQMHRQESIQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.802050295 (211h11m46.629758695s) [noise=true , address=145.79.227.32:4836 , piece_id=IQRH4CRVBOG72M3BE3G5FTIEMBTLD2KFUEYU6NCTK3N6PUONMY5A, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.802573060 (211h11m46.630281451s) [noise=true , address=77.90.31.14:4661 , piece_id=LLI4CDLWJ32TFXQ2YEU3NRP4QIRSNNUODA5RRPWU5AA45MJEW2LQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.803112997 (211h11m46.630821425s) [noise=true , address=145.79.227.44:4194 , piece_id=ZWOCXZ746BWDZ2LWYHSQHSG2O3WJC4R6FDL5E3JP2SXRCX3AWT2Q, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.803148596 (211h11m46.630856991s) [noise=true , address=145.79.227.46:4603 , piece_id=3X2DPA75MR7RX6LCOO2U3UREG4QSYHRRKRE57JUYUBINXTJN5IEA, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.803615052 (211h11m46.631323455s) [noise=true , address=145.79.227.28:4727 , piece_id=42ILJYV7IJRXSV3E3RWXWNUNYQHHQFNOF2JZGZKDEGPXRXZ3H2YA, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.803722621 (211h11m46.631431034s) [noise=true , address=77.90.31.16:4735 , piece_id=XKJJK6K7XF3TOFWMIU2BUVQTESTNV4FRYINSXJS4MC7KK7ZJU7SQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.804120053 (211h11m46.631828444s) [noise=true , address=145.79.227.7:4293 , piece_id=56YULKIWDM4P4WZM6KGMPUK3B3AK3RRVF4H6RJ7H6IQISPLXGBCQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.804570407 (211h11m46.632278798s) [noise=true , address=145.79.227.47:4267 , piece_id=VY46BNDUYVGBXDRIOEFK4USNOOJH2KXZH5Z2NHCT6SDJC5SFUYMQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.805083798 (211h11m46.632792196s) [noise=true , address=145.79.227.50:4375 , piece_id=RKVBJHHIF3KJLKB6ULTA4E5SHOBPCLPMWJCA6YTKF5XDNVEZMYXQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.805547276 (211h11m46.633255677s) [noise=true , address=77.90.31.16:4734 , piece_id=4JO2RBACNY5AM3KG7XUW7JTRGDCXGQDXSZ3L23ASAEJ4YFCICVAQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.805916642 (211h11m46.633625023s) [noise=true , address=145.79.227.39:4014 , piece_id=WKYZGQX5SP22ML66ZQZIHBS324EONJ7AARTYAXK2SCV577ZCQ5OQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.806440130 (211h11m46.634148525s) [noise=true , address=77.90.31.13:4626 , piece_id=DCKUBRB3DR755GN67RLP6NUMREAWIB3AITGPIMA7MM35BRPZW7IA, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.806976882 (211h11m46.634685284s) [noise=true , address=77.90.31.10:4519 , piece_id=ACPHBL7LF6NCNNH4U5GUXKW7RCL6XIEQVWYDCYUTXEC5RQE7RZBA, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.807729329 (211h11m46.63543772s) [noise=true , address=77.90.31.10:4518 , piece_id=ZE3SFGBCM2QV3CGGS3VEYT7NKDTDE2OJ6T6CVSGGYACCFX5WN5ZQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.841457783 (211h11m46.66916618s) [noise=true , address=77.90.31.13:4626 , piece_id=DCKUBRB3DR755GN67RLP6NUMREAWIB3AITGPIMA7MM35BRPZW7IA, seg_pos={0 0}, upload=219598]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: storage node overloaded, request limit: 2000 Dec 14 10:39:45.850206751 (211h11m46.677915151s) [noise=true , address=77.90.31.16:4734 , piece_id=4JO2RBACNY5AM3KG7XUW7JTRGDCXGQDXSZ3L23ASAEJ4YFCICVAQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: storage node overloaded, request limit: 2000 Dec 14 10:39:45.859644247 (211h11m46.68735265s) [noise=true , address=145.79.227.30:4763 , piece_id=CLMX6DHN5HP3OWMVR4R5MIVRH3GHCELUVTYJHBE7HXZQMHRQESIQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: storage node overloaded, request limit: 2000 Dec 14 10:39:45.874109133 (211h11m46.701817525s) [upload=219598]: Next returned nil segment Dec 14 10:39:45.874123719 (211h11m46.701832113s) [upload=219598]: Waiting for error group managing segments... Dec 14 10:39:45.880971826 (211h11m46.708680217s) [noise=true , address=77.90.31.10:4518 , piece_id=ZE3SFGBCM2QV3CGGS3VEYT7NKDTDE2OJ6T6CVSGGYACCFX5WN5ZQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.882649818 (211h11m46.710358212s) [noise=true , address=77.90.31.10:4519 , piece_id=ACPHBL7LF6NCNNH4U5GUXKW7RCL6XIEQVWYDCYUTXEC5RQE7RZBA, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.883088160 (211h11m46.71079655s) [noise=true , address=77.90.31.14:4661 , piece_id=LLI4CDLWJ32TFXQ2YEU3NRP4QIRSNNUODA5RRPWU5AA45MJEW2LQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.883693953 (211h11m46.711402345s) [noise=true , address=145.79.227.39:4014 , piece_id=WKYZGQX5SP22ML66ZQZIHBS324EONJ7AARTYAXK2SCV577ZCQ5OQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.884112448 (211h11m46.711820839s) [noise=true , address=145.79.227.32:4836 , piece_id=IQRH4CRVBOG72M3BE3G5FTIEMBTLD2KFUEYU6NCTK3N6PUONMY5A, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.884619811 (211h11m46.712328205s) [noise=true , address=145.79.227.43:4159 , piece_id=ULRDDEYCZXA26UPSFZQRQWFPU7MUCGJJSM7YWIR7HPXDCPBOQXNA, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.885012115 (211h11m46.712720506s) [noise=true , address=145.79.227.47:4267 , piece_id=VY46BNDUYVGBXDRIOEFK4USNOOJH2KXZH5Z2NHCT6SDJC5SFUYMQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.887669218 (211h11m46.71537761s) [noise=true , address=145.79.227.50:4375 , piece_id=RKVBJHHIF3KJLKB6ULTA4E5SHOBPCLPMWJCA6YTKF5XDNVEZMYXQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.890935473 (211h11m46.718643851s) [noise=true , address=145.79.227.46:4603 , piece_id=3X2DPA75MR7RX6LCOO2U3UREG4QSYHRRKRE57JUYUBINXTJN5IEA, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.891096730 (211h11m46.718805121s) [noise=true , address=145.79.227.28:4727 , piece_id=42ILJYV7IJRXSV3E3RWXWNUNYQHHQFNOF2JZGZKDEGPXRXZ3H2YA, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.893061640 (211h11m46.720770031s) [noise=true , address=145.79.227.7:4293 , piece_id=56YULKIWDM4P4WZM6KGMPUK3B3AK3RRVF4H6RJ7H6IQISPLXGBCQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:48.683585502 (211h11m49.511293883s) [noise=true , address=145.79.227.44:4194 , piece_id=ZWOCXZ746BWDZ2LWYHSQHSG2O3WJC4R6FDL5E3JP2SXRCX3AWT2Q, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:40:15.740699026 (211h12m16.568407426s) [noise=true , address=77.90.31.16:4735 , piece_id=XKJJK6K7XF3TOFWMIU2BUVQTESTNV4FRYINSXJS4MC7KK7ZJU7SQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: expected piece hash; context deadline exceeded; EOF Dec 14 10:40:15.740740404 (211h12m16.568448788s) [seg_pos={0 0}, upload=219598]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 14 10:40:15.762950810 (211h12m16.590659203s) [upload=219598]: Done uploading segments... Dec 14 10:40:15.762987936 (211h12m16.59069633s) [upload=219598]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 579.503067ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................................X....................] duration: 408.477918ms trace: https://jaeger.dev.storj.io/trace/25fec5475908806a FAIL (2025-12-14 10:53:38 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 14 10:39:45.775689210 (211h11m46.603397602s) [upload=219598]: Starting upload Dec 14 10:39:45.775750055 (211h11m46.603458455s) [upload=219598]: Done starting upload Dec 14 10:39:45.775957558 (211h11m46.603665941s) [upload=219598]: Uploading segments... Dec 14 10:39:45.776074427 (211h11m46.603782809s) [upload=219598]: Got next segment. Inline: false Dec 14 10:39:45.776158162 (211h11m46.603866553s) [upload=219598]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 14 10:39:45.776165965 (211h11m46.603874362s) [upload=219598]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 14 10:39:45.801257283 (211h11m46.628965678s) [seg_pos={0 0}, upload=219598]: Begin upload segment... Dec 14 10:39:45.801373021 (211h11m46.629081422s) [noise=true , address=145.79.227.43:4159 , piece_id=ULRDDEYCZXA26UPSFZQRQWFPU7MUCGJJSM7YWIR7HPXDCPBOQXNA, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.801409865 (211h11m46.629118247s) [seg_pos={0 0}, upload=219598]: Done begin upload segment. Dec 14 10:39:45.801481233 (211h11m46.629189635s) [noise=true , address=145.79.227.30:4763 , piece_id=CLMX6DHN5HP3OWMVR4R5MIVRH3GHCELUVTYJHBE7HXZQMHRQESIQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.802050295 (211h11m46.629758695s) [noise=true , address=145.79.227.32:4836 , piece_id=IQRH4CRVBOG72M3BE3G5FTIEMBTLD2KFUEYU6NCTK3N6PUONMY5A, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.802573060 (211h11m46.630281451s) [noise=true , address=77.90.31.14:4661 , piece_id=LLI4CDLWJ32TFXQ2YEU3NRP4QIRSNNUODA5RRPWU5AA45MJEW2LQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.803112997 (211h11m46.630821425s) [noise=true , address=145.79.227.44:4194 , piece_id=ZWOCXZ746BWDZ2LWYHSQHSG2O3WJC4R6FDL5E3JP2SXRCX3AWT2Q, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.803148596 (211h11m46.630856991s) [noise=true , address=145.79.227.46:4603 , piece_id=3X2DPA75MR7RX6LCOO2U3UREG4QSYHRRKRE57JUYUBINXTJN5IEA, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.803615052 (211h11m46.631323455s) [noise=true , address=145.79.227.28:4727 , piece_id=42ILJYV7IJRXSV3E3RWXWNUNYQHHQFNOF2JZGZKDEGPXRXZ3H2YA, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.803722621 (211h11m46.631431034s) [noise=true , address=77.90.31.16:4735 , piece_id=XKJJK6K7XF3TOFWMIU2BUVQTESTNV4FRYINSXJS4MC7KK7ZJU7SQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.804120053 (211h11m46.631828444s) [noise=true , address=145.79.227.7:4293 , piece_id=56YULKIWDM4P4WZM6KGMPUK3B3AK3RRVF4H6RJ7H6IQISPLXGBCQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.804570407 (211h11m46.632278798s) [noise=true , address=145.79.227.47:4267 , piece_id=VY46BNDUYVGBXDRIOEFK4USNOOJH2KXZH5Z2NHCT6SDJC5SFUYMQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.805083798 (211h11m46.632792196s) [noise=true , address=145.79.227.50:4375 , piece_id=RKVBJHHIF3KJLKB6ULTA4E5SHOBPCLPMWJCA6YTKF5XDNVEZMYXQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.805547276 (211h11m46.633255677s) [noise=true , address=77.90.31.16:4734 , piece_id=4JO2RBACNY5AM3KG7XUW7JTRGDCXGQDXSZ3L23ASAEJ4YFCICVAQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.805916642 (211h11m46.633625023s) [noise=true , address=145.79.227.39:4014 , piece_id=WKYZGQX5SP22ML66ZQZIHBS324EONJ7AARTYAXK2SCV577ZCQ5OQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.806440130 (211h11m46.634148525s) [noise=true , address=77.90.31.13:4626 , piece_id=DCKUBRB3DR755GN67RLP6NUMREAWIB3AITGPIMA7MM35BRPZW7IA, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.806976882 (211h11m46.634685284s) [noise=true , address=77.90.31.10:4519 , piece_id=ACPHBL7LF6NCNNH4U5GUXKW7RCL6XIEQVWYDCYUTXEC5RQE7RZBA, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.807729329 (211h11m46.63543772s) [noise=true , address=77.90.31.10:4518 , piece_id=ZE3SFGBCM2QV3CGGS3VEYT7NKDTDE2OJ6T6CVSGGYACCFX5WN5ZQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.841457783 (211h11m46.66916618s) [noise=true , address=77.90.31.13:4626 , piece_id=DCKUBRB3DR755GN67RLP6NUMREAWIB3AITGPIMA7MM35BRPZW7IA, seg_pos={0 0}, upload=219598]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: storage node overloaded, request limit: 2000 Dec 14 10:39:45.850206751 (211h11m46.677915151s) [noise=true , address=77.90.31.16:4734 , piece_id=4JO2RBACNY5AM3KG7XUW7JTRGDCXGQDXSZ3L23ASAEJ4YFCICVAQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: storage node overloaded, request limit: 2000 Dec 14 10:39:45.859644247 (211h11m46.68735265s) [noise=true , address=145.79.227.30:4763 , piece_id=CLMX6DHN5HP3OWMVR4R5MIVRH3GHCELUVTYJHBE7HXZQMHRQESIQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: storage node overloaded, request limit: 2000 Dec 14 10:39:45.874109133 (211h11m46.701817525s) [upload=219598]: Next returned nil segment Dec 14 10:39:45.874123719 (211h11m46.701832113s) [upload=219598]: Waiting for error group managing segments... Dec 14 10:39:45.880971826 (211h11m46.708680217s) [noise=true , address=77.90.31.10:4518 , piece_id=ZE3SFGBCM2QV3CGGS3VEYT7NKDTDE2OJ6T6CVSGGYACCFX5WN5ZQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.882649818 (211h11m46.710358212s) [noise=true , address=77.90.31.10:4519 , piece_id=ACPHBL7LF6NCNNH4U5GUXKW7RCL6XIEQVWYDCYUTXEC5RQE7RZBA, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.883088160 (211h11m46.71079655s) [noise=true , address=77.90.31.14:4661 , piece_id=LLI4CDLWJ32TFXQ2YEU3NRP4QIRSNNUODA5RRPWU5AA45MJEW2LQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.883693953 (211h11m46.711402345s) [noise=true , address=145.79.227.39:4014 , piece_id=WKYZGQX5SP22ML66ZQZIHBS324EONJ7AARTYAXK2SCV577ZCQ5OQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.884112448 (211h11m46.711820839s) [noise=true , address=145.79.227.32:4836 , piece_id=IQRH4CRVBOG72M3BE3G5FTIEMBTLD2KFUEYU6NCTK3N6PUONMY5A, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.884619811 (211h11m46.712328205s) [noise=true , address=145.79.227.43:4159 , piece_id=ULRDDEYCZXA26UPSFZQRQWFPU7MUCGJJSM7YWIR7HPXDCPBOQXNA, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.885012115 (211h11m46.712720506s) [noise=true , address=145.79.227.47:4267 , piece_id=VY46BNDUYVGBXDRIOEFK4USNOOJH2KXZH5Z2NHCT6SDJC5SFUYMQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.887669218 (211h11m46.71537761s) [noise=true , address=145.79.227.50:4375 , piece_id=RKVBJHHIF3KJLKB6ULTA4E5SHOBPCLPMWJCA6YTKF5XDNVEZMYXQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.890935473 (211h11m46.718643851s) [noise=true , address=145.79.227.46:4603 , piece_id=3X2DPA75MR7RX6LCOO2U3UREG4QSYHRRKRE57JUYUBINXTJN5IEA, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.891096730 (211h11m46.718805121s) [noise=true , address=145.79.227.28:4727 , piece_id=42ILJYV7IJRXSV3E3RWXWNUNYQHHQFNOF2JZGZKDEGPXRXZ3H2YA, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.893061640 (211h11m46.720770031s) [noise=true , address=145.79.227.7:4293 , piece_id=56YULKIWDM4P4WZM6KGMPUK3B3AK3RRVF4H6RJ7H6IQISPLXGBCQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:48.683585502 (211h11m49.511293883s) [noise=true , address=145.79.227.44:4194 , piece_id=ZWOCXZ746BWDZ2LWYHSQHSG2O3WJC4R6FDL5E3JP2SXRCX3AWT2Q, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:40:15.740699026 (211h12m16.568407426s) [noise=true , address=77.90.31.16:4735 , piece_id=XKJJK6K7XF3TOFWMIU2BUVQTESTNV4FRYINSXJS4MC7KK7ZJU7SQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: expected piece hash; context deadline exceeded; EOF Dec 14 10:40:15.740740404 (211h12m16.568448788s) [seg_pos={0 0}, upload=219598]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 14 10:40:15.762950810 (211h12m16.590659203s) [upload=219598]: Done uploading segments... Dec 14 10:40:15.762987936 (211h12m16.59069633s) [upload=219598]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 408.529888ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................................X...................] duration: 386.344278ms trace: https://jaeger.dev.storj.io/trace/25fec5475908806a FAIL (2025-12-14 10:53:00 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 14 10:39:45.775689210 (211h11m46.603397602s) [upload=219598]: Starting upload Dec 14 10:39:45.775750055 (211h11m46.603458455s) [upload=219598]: Done starting upload Dec 14 10:39:45.775957558 (211h11m46.603665941s) [upload=219598]: Uploading segments... Dec 14 10:39:45.776074427 (211h11m46.603782809s) [upload=219598]: Got next segment. Inline: false Dec 14 10:39:45.776158162 (211h11m46.603866553s) [upload=219598]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 14 10:39:45.776165965 (211h11m46.603874362s) [upload=219598]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 14 10:39:45.801257283 (211h11m46.628965678s) [seg_pos={0 0}, upload=219598]: Begin upload segment... Dec 14 10:39:45.801373021 (211h11m46.629081422s) [noise=true , address=145.79.227.43:4159 , piece_id=ULRDDEYCZXA26UPSFZQRQWFPU7MUCGJJSM7YWIR7HPXDCPBOQXNA, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.801409865 (211h11m46.629118247s) [seg_pos={0 0}, upload=219598]: Done begin upload segment. Dec 14 10:39:45.801481233 (211h11m46.629189635s) [noise=true , address=145.79.227.30:4763 , piece_id=CLMX6DHN5HP3OWMVR4R5MIVRH3GHCELUVTYJHBE7HXZQMHRQESIQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.802050295 (211h11m46.629758695s) [noise=true , address=145.79.227.32:4836 , piece_id=IQRH4CRVBOG72M3BE3G5FTIEMBTLD2KFUEYU6NCTK3N6PUONMY5A, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.802573060 (211h11m46.630281451s) [noise=true , address=77.90.31.14:4661 , piece_id=LLI4CDLWJ32TFXQ2YEU3NRP4QIRSNNUODA5RRPWU5AA45MJEW2LQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.803112997 (211h11m46.630821425s) [noise=true , address=145.79.227.44:4194 , piece_id=ZWOCXZ746BWDZ2LWYHSQHSG2O3WJC4R6FDL5E3JP2SXRCX3AWT2Q, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.803148596 (211h11m46.630856991s) [noise=true , address=145.79.227.46:4603 , piece_id=3X2DPA75MR7RX6LCOO2U3UREG4QSYHRRKRE57JUYUBINXTJN5IEA, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.803615052 (211h11m46.631323455s) [noise=true , address=145.79.227.28:4727 , piece_id=42ILJYV7IJRXSV3E3RWXWNUNYQHHQFNOF2JZGZKDEGPXRXZ3H2YA, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.803722621 (211h11m46.631431034s) [noise=true , address=77.90.31.16:4735 , piece_id=XKJJK6K7XF3TOFWMIU2BUVQTESTNV4FRYINSXJS4MC7KK7ZJU7SQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.804120053 (211h11m46.631828444s) [noise=true , address=145.79.227.7:4293 , piece_id=56YULKIWDM4P4WZM6KGMPUK3B3AK3RRVF4H6RJ7H6IQISPLXGBCQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.804570407 (211h11m46.632278798s) [noise=true , address=145.79.227.47:4267 , piece_id=VY46BNDUYVGBXDRIOEFK4USNOOJH2KXZH5Z2NHCT6SDJC5SFUYMQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.805083798 (211h11m46.632792196s) [noise=true , address=145.79.227.50:4375 , piece_id=RKVBJHHIF3KJLKB6ULTA4E5SHOBPCLPMWJCA6YTKF5XDNVEZMYXQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.805547276 (211h11m46.633255677s) [noise=true , address=77.90.31.16:4734 , piece_id=4JO2RBACNY5AM3KG7XUW7JTRGDCXGQDXSZ3L23ASAEJ4YFCICVAQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.805916642 (211h11m46.633625023s) [noise=true , address=145.79.227.39:4014 , piece_id=WKYZGQX5SP22ML66ZQZIHBS324EONJ7AARTYAXK2SCV577ZCQ5OQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.806440130 (211h11m46.634148525s) [noise=true , address=77.90.31.13:4626 , piece_id=DCKUBRB3DR755GN67RLP6NUMREAWIB3AITGPIMA7MM35BRPZW7IA, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.806976882 (211h11m46.634685284s) [noise=true , address=77.90.31.10:4519 , piece_id=ACPHBL7LF6NCNNH4U5GUXKW7RCL6XIEQVWYDCYUTXEC5RQE7RZBA, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.807729329 (211h11m46.63543772s) [noise=true , address=77.90.31.10:4518 , piece_id=ZE3SFGBCM2QV3CGGS3VEYT7NKDTDE2OJ6T6CVSGGYACCFX5WN5ZQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.841457783 (211h11m46.66916618s) [noise=true , address=77.90.31.13:4626 , piece_id=DCKUBRB3DR755GN67RLP6NUMREAWIB3AITGPIMA7MM35BRPZW7IA, seg_pos={0 0}, upload=219598]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: storage node overloaded, request limit: 2000 Dec 14 10:39:45.850206751 (211h11m46.677915151s) [noise=true , address=77.90.31.16:4734 , piece_id=4JO2RBACNY5AM3KG7XUW7JTRGDCXGQDXSZ3L23ASAEJ4YFCICVAQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: storage node overloaded, request limit: 2000 Dec 14 10:39:45.859644247 (211h11m46.68735265s) [noise=true , address=145.79.227.30:4763 , piece_id=CLMX6DHN5HP3OWMVR4R5MIVRH3GHCELUVTYJHBE7HXZQMHRQESIQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: storage node overloaded, request limit: 2000 Dec 14 10:39:45.874109133 (211h11m46.701817525s) [upload=219598]: Next returned nil segment Dec 14 10:39:45.874123719 (211h11m46.701832113s) [upload=219598]: Waiting for error group managing segments... Dec 14 10:39:45.880971826 (211h11m46.708680217s) [noise=true , address=77.90.31.10:4518 , piece_id=ZE3SFGBCM2QV3CGGS3VEYT7NKDTDE2OJ6T6CVSGGYACCFX5WN5ZQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.882649818 (211h11m46.710358212s) [noise=true , address=77.90.31.10:4519 , piece_id=ACPHBL7LF6NCNNH4U5GUXKW7RCL6XIEQVWYDCYUTXEC5RQE7RZBA, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.883088160 (211h11m46.71079655s) [noise=true , address=77.90.31.14:4661 , piece_id=LLI4CDLWJ32TFXQ2YEU3NRP4QIRSNNUODA5RRPWU5AA45MJEW2LQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.883693953 (211h11m46.711402345s) [noise=true , address=145.79.227.39:4014 , piece_id=WKYZGQX5SP22ML66ZQZIHBS324EONJ7AARTYAXK2SCV577ZCQ5OQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.884112448 (211h11m46.711820839s) [noise=true , address=145.79.227.32:4836 , piece_id=IQRH4CRVBOG72M3BE3G5FTIEMBTLD2KFUEYU6NCTK3N6PUONMY5A, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.884619811 (211h11m46.712328205s) [noise=true , address=145.79.227.43:4159 , piece_id=ULRDDEYCZXA26UPSFZQRQWFPU7MUCGJJSM7YWIR7HPXDCPBOQXNA, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.885012115 (211h11m46.712720506s) [noise=true , address=145.79.227.47:4267 , piece_id=VY46BNDUYVGBXDRIOEFK4USNOOJH2KXZH5Z2NHCT6SDJC5SFUYMQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.887669218 (211h11m46.71537761s) [noise=true , address=145.79.227.50:4375 , piece_id=RKVBJHHIF3KJLKB6ULTA4E5SHOBPCLPMWJCA6YTKF5XDNVEZMYXQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.890935473 (211h11m46.718643851s) [noise=true , address=145.79.227.46:4603 , piece_id=3X2DPA75MR7RX6LCOO2U3UREG4QSYHRRKRE57JUYUBINXTJN5IEA, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.891096730 (211h11m46.718805121s) [noise=true , address=145.79.227.28:4727 , piece_id=42ILJYV7IJRXSV3E3RWXWNUNYQHHQFNOF2JZGZKDEGPXRXZ3H2YA, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.893061640 (211h11m46.720770031s) [noise=true , address=145.79.227.7:4293 , piece_id=56YULKIWDM4P4WZM6KGMPUK3B3AK3RRVF4H6RJ7H6IQISPLXGBCQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:48.683585502 (211h11m49.511293883s) [noise=true , address=145.79.227.44:4194 , piece_id=ZWOCXZ746BWDZ2LWYHSQHSG2O3WJC4R6FDL5E3JP2SXRCX3AWT2Q, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:40:15.740699026 (211h12m16.568407426s) [noise=true , address=77.90.31.16:4735 , piece_id=XKJJK6K7XF3TOFWMIU2BUVQTESTNV4FRYINSXJS4MC7KK7ZJU7SQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: expected piece hash; context deadline exceeded; EOF Dec 14 10:40:15.740740404 (211h12m16.568448788s) [seg_pos={0 0}, upload=219598]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 14 10:40:15.762950810 (211h12m16.590659203s) [upload=219598]: Done uploading segments... Dec 14 10:40:15.762987936 (211h12m16.59069633s) [upload=219598]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 386.368955ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................X..................] duration: 362.677564ms trace: https://jaeger.dev.storj.io/trace/25fec5475908806a FAIL (2025-12-14 10:52:43 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 14 10:39:45.775689210 (211h11m46.603397602s) [upload=219598]: Starting upload Dec 14 10:39:45.775750055 (211h11m46.603458455s) [upload=219598]: Done starting upload Dec 14 10:39:45.775957558 (211h11m46.603665941s) [upload=219598]: Uploading segments... Dec 14 10:39:45.776074427 (211h11m46.603782809s) [upload=219598]: Got next segment. Inline: false Dec 14 10:39:45.776158162 (211h11m46.603866553s) [upload=219598]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 14 10:39:45.776165965 (211h11m46.603874362s) [upload=219598]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 14 10:39:45.801257283 (211h11m46.628965678s) [seg_pos={0 0}, upload=219598]: Begin upload segment... Dec 14 10:39:45.801373021 (211h11m46.629081422s) [noise=true , address=145.79.227.43:4159 , piece_id=ULRDDEYCZXA26UPSFZQRQWFPU7MUCGJJSM7YWIR7HPXDCPBOQXNA, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.801409865 (211h11m46.629118247s) [seg_pos={0 0}, upload=219598]: Done begin upload segment. Dec 14 10:39:45.801481233 (211h11m46.629189635s) [noise=true , address=145.79.227.30:4763 , piece_id=CLMX6DHN5HP3OWMVR4R5MIVRH3GHCELUVTYJHBE7HXZQMHRQESIQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.802050295 (211h11m46.629758695s) [noise=true , address=145.79.227.32:4836 , piece_id=IQRH4CRVBOG72M3BE3G5FTIEMBTLD2KFUEYU6NCTK3N6PUONMY5A, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.802573060 (211h11m46.630281451s) [noise=true , address=77.90.31.14:4661 , piece_id=LLI4CDLWJ32TFXQ2YEU3NRP4QIRSNNUODA5RRPWU5AA45MJEW2LQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.803112997 (211h11m46.630821425s) [noise=true , address=145.79.227.44:4194 , piece_id=ZWOCXZ746BWDZ2LWYHSQHSG2O3WJC4R6FDL5E3JP2SXRCX3AWT2Q, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.803148596 (211h11m46.630856991s) [noise=true , address=145.79.227.46:4603 , piece_id=3X2DPA75MR7RX6LCOO2U3UREG4QSYHRRKRE57JUYUBINXTJN5IEA, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.803615052 (211h11m46.631323455s) [noise=true , address=145.79.227.28:4727 , piece_id=42ILJYV7IJRXSV3E3RWXWNUNYQHHQFNOF2JZGZKDEGPXRXZ3H2YA, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.803722621 (211h11m46.631431034s) [noise=true , address=77.90.31.16:4735 , piece_id=XKJJK6K7XF3TOFWMIU2BUVQTESTNV4FRYINSXJS4MC7KK7ZJU7SQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.804120053 (211h11m46.631828444s) [noise=true , address=145.79.227.7:4293 , piece_id=56YULKIWDM4P4WZM6KGMPUK3B3AK3RRVF4H6RJ7H6IQISPLXGBCQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.804570407 (211h11m46.632278798s) [noise=true , address=145.79.227.47:4267 , piece_id=VY46BNDUYVGBXDRIOEFK4USNOOJH2KXZH5Z2NHCT6SDJC5SFUYMQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.805083798 (211h11m46.632792196s) [noise=true , address=145.79.227.50:4375 , piece_id=RKVBJHHIF3KJLKB6ULTA4E5SHOBPCLPMWJCA6YTKF5XDNVEZMYXQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.805547276 (211h11m46.633255677s) [noise=true , address=77.90.31.16:4734 , piece_id=4JO2RBACNY5AM3KG7XUW7JTRGDCXGQDXSZ3L23ASAEJ4YFCICVAQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.805916642 (211h11m46.633625023s) [noise=true , address=145.79.227.39:4014 , piece_id=WKYZGQX5SP22ML66ZQZIHBS324EONJ7AARTYAXK2SCV577ZCQ5OQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.806440130 (211h11m46.634148525s) [noise=true , address=77.90.31.13:4626 , piece_id=DCKUBRB3DR755GN67RLP6NUMREAWIB3AITGPIMA7MM35BRPZW7IA, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.806976882 (211h11m46.634685284s) [noise=true , address=77.90.31.10:4519 , piece_id=ACPHBL7LF6NCNNH4U5GUXKW7RCL6XIEQVWYDCYUTXEC5RQE7RZBA, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.807729329 (211h11m46.63543772s) [noise=true , address=77.90.31.10:4518 , piece_id=ZE3SFGBCM2QV3CGGS3VEYT7NKDTDE2OJ6T6CVSGGYACCFX5WN5ZQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.841457783 (211h11m46.66916618s) [noise=true , address=77.90.31.13:4626 , piece_id=DCKUBRB3DR755GN67RLP6NUMREAWIB3AITGPIMA7MM35BRPZW7IA, seg_pos={0 0}, upload=219598]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: storage node overloaded, request limit: 2000 Dec 14 10:39:45.850206751 (211h11m46.677915151s) [noise=true , address=77.90.31.16:4734 , piece_id=4JO2RBACNY5AM3KG7XUW7JTRGDCXGQDXSZ3L23ASAEJ4YFCICVAQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: storage node overloaded, request limit: 2000 Dec 14 10:39:45.859644247 (211h11m46.68735265s) [noise=true , address=145.79.227.30:4763 , piece_id=CLMX6DHN5HP3OWMVR4R5MIVRH3GHCELUVTYJHBE7HXZQMHRQESIQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: storage node overloaded, request limit: 2000 Dec 14 10:39:45.874109133 (211h11m46.701817525s) [upload=219598]: Next returned nil segment Dec 14 10:39:45.874123719 (211h11m46.701832113s) [upload=219598]: Waiting for error group managing segments... Dec 14 10:39:45.880971826 (211h11m46.708680217s) [noise=true , address=77.90.31.10:4518 , piece_id=ZE3SFGBCM2QV3CGGS3VEYT7NKDTDE2OJ6T6CVSGGYACCFX5WN5ZQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.882649818 (211h11m46.710358212s) [noise=true , address=77.90.31.10:4519 , piece_id=ACPHBL7LF6NCNNH4U5GUXKW7RCL6XIEQVWYDCYUTXEC5RQE7RZBA, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.883088160 (211h11m46.71079655s) [noise=true , address=77.90.31.14:4661 , piece_id=LLI4CDLWJ32TFXQ2YEU3NRP4QIRSNNUODA5RRPWU5AA45MJEW2LQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.883693953 (211h11m46.711402345s) [noise=true , address=145.79.227.39:4014 , piece_id=WKYZGQX5SP22ML66ZQZIHBS324EONJ7AARTYAXK2SCV577ZCQ5OQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.884112448 (211h11m46.711820839s) [noise=true , address=145.79.227.32:4836 , piece_id=IQRH4CRVBOG72M3BE3G5FTIEMBTLD2KFUEYU6NCTK3N6PUONMY5A, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.884619811 (211h11m46.712328205s) [noise=true , address=145.79.227.43:4159 , piece_id=ULRDDEYCZXA26UPSFZQRQWFPU7MUCGJJSM7YWIR7HPXDCPBOQXNA, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.885012115 (211h11m46.712720506s) [noise=true , address=145.79.227.47:4267 , piece_id=VY46BNDUYVGBXDRIOEFK4USNOOJH2KXZH5Z2NHCT6SDJC5SFUYMQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.887669218 (211h11m46.71537761s) [noise=true , address=145.79.227.50:4375 , piece_id=RKVBJHHIF3KJLKB6ULTA4E5SHOBPCLPMWJCA6YTKF5XDNVEZMYXQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.890935473 (211h11m46.718643851s) [noise=true , address=145.79.227.46:4603 , piece_id=3X2DPA75MR7RX6LCOO2U3UREG4QSYHRRKRE57JUYUBINXTJN5IEA, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.891096730 (211h11m46.718805121s) [noise=true , address=145.79.227.28:4727 , piece_id=42ILJYV7IJRXSV3E3RWXWNUNYQHHQFNOF2JZGZKDEGPXRXZ3H2YA, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.893061640 (211h11m46.720770031s) [noise=true , address=145.79.227.7:4293 , piece_id=56YULKIWDM4P4WZM6KGMPUK3B3AK3RRVF4H6RJ7H6IQISPLXGBCQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:48.683585502 (211h11m49.511293883s) [noise=true , address=145.79.227.44:4194 , piece_id=ZWOCXZ746BWDZ2LWYHSQHSG2O3WJC4R6FDL5E3JP2SXRCX3AWT2Q, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:40:15.740699026 (211h12m16.568407426s) [noise=true , address=77.90.31.16:4735 , piece_id=XKJJK6K7XF3TOFWMIU2BUVQTESTNV4FRYINSXJS4MC7KK7ZJU7SQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: expected piece hash; context deadline exceeded; EOF Dec 14 10:40:15.740740404 (211h12m16.568448788s) [seg_pos={0 0}, upload=219598]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 14 10:40:15.762950810 (211h12m16.590659203s) [upload=219598]: Done uploading segments... Dec 14 10:40:15.762987936 (211h12m16.59069633s) [upload=219598]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 362.713356ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................X.................] duration: 411.705696ms trace: https://jaeger.dev.storj.io/trace/25fec5475908806a FAIL (2025-12-14 10:51:53 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 14 10:39:45.775689210 (211h11m46.603397602s) [upload=219598]: Starting upload Dec 14 10:39:45.775750055 (211h11m46.603458455s) [upload=219598]: Done starting upload Dec 14 10:39:45.775957558 (211h11m46.603665941s) [upload=219598]: Uploading segments... Dec 14 10:39:45.776074427 (211h11m46.603782809s) [upload=219598]: Got next segment. Inline: false Dec 14 10:39:45.776158162 (211h11m46.603866553s) [upload=219598]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 14 10:39:45.776165965 (211h11m46.603874362s) [upload=219598]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 14 10:39:45.801257283 (211h11m46.628965678s) [seg_pos={0 0}, upload=219598]: Begin upload segment... Dec 14 10:39:45.801373021 (211h11m46.629081422s) [noise=true , address=145.79.227.43:4159 , piece_id=ULRDDEYCZXA26UPSFZQRQWFPU7MUCGJJSM7YWIR7HPXDCPBOQXNA, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.801409865 (211h11m46.629118247s) [seg_pos={0 0}, upload=219598]: Done begin upload segment. Dec 14 10:39:45.801481233 (211h11m46.629189635s) [noise=true , address=145.79.227.30:4763 , piece_id=CLMX6DHN5HP3OWMVR4R5MIVRH3GHCELUVTYJHBE7HXZQMHRQESIQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.802050295 (211h11m46.629758695s) [noise=true , address=145.79.227.32:4836 , piece_id=IQRH4CRVBOG72M3BE3G5FTIEMBTLD2KFUEYU6NCTK3N6PUONMY5A, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.802573060 (211h11m46.630281451s) [noise=true , address=77.90.31.14:4661 , piece_id=LLI4CDLWJ32TFXQ2YEU3NRP4QIRSNNUODA5RRPWU5AA45MJEW2LQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.803112997 (211h11m46.630821425s) [noise=true , address=145.79.227.44:4194 , piece_id=ZWOCXZ746BWDZ2LWYHSQHSG2O3WJC4R6FDL5E3JP2SXRCX3AWT2Q, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.803148596 (211h11m46.630856991s) [noise=true , address=145.79.227.46:4603 , piece_id=3X2DPA75MR7RX6LCOO2U3UREG4QSYHRRKRE57JUYUBINXTJN5IEA, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.803615052 (211h11m46.631323455s) [noise=true , address=145.79.227.28:4727 , piece_id=42ILJYV7IJRXSV3E3RWXWNUNYQHHQFNOF2JZGZKDEGPXRXZ3H2YA, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.803722621 (211h11m46.631431034s) [noise=true , address=77.90.31.16:4735 , piece_id=XKJJK6K7XF3TOFWMIU2BUVQTESTNV4FRYINSXJS4MC7KK7ZJU7SQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.804120053 (211h11m46.631828444s) [noise=true , address=145.79.227.7:4293 , piece_id=56YULKIWDM4P4WZM6KGMPUK3B3AK3RRVF4H6RJ7H6IQISPLXGBCQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.804570407 (211h11m46.632278798s) [noise=true , address=145.79.227.47:4267 , piece_id=VY46BNDUYVGBXDRIOEFK4USNOOJH2KXZH5Z2NHCT6SDJC5SFUYMQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.805083798 (211h11m46.632792196s) [noise=true , address=145.79.227.50:4375 , piece_id=RKVBJHHIF3KJLKB6ULTA4E5SHOBPCLPMWJCA6YTKF5XDNVEZMYXQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.805547276 (211h11m46.633255677s) [noise=true , address=77.90.31.16:4734 , piece_id=4JO2RBACNY5AM3KG7XUW7JTRGDCXGQDXSZ3L23ASAEJ4YFCICVAQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.805916642 (211h11m46.633625023s) [noise=true , address=145.79.227.39:4014 , piece_id=WKYZGQX5SP22ML66ZQZIHBS324EONJ7AARTYAXK2SCV577ZCQ5OQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.806440130 (211h11m46.634148525s) [noise=true , address=77.90.31.13:4626 , piece_id=DCKUBRB3DR755GN67RLP6NUMREAWIB3AITGPIMA7MM35BRPZW7IA, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.806976882 (211h11m46.634685284s) [noise=true , address=77.90.31.10:4519 , piece_id=ACPHBL7LF6NCNNH4U5GUXKW7RCL6XIEQVWYDCYUTXEC5RQE7RZBA, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.807729329 (211h11m46.63543772s) [noise=true , address=77.90.31.10:4518 , piece_id=ZE3SFGBCM2QV3CGGS3VEYT7NKDTDE2OJ6T6CVSGGYACCFX5WN5ZQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.841457783 (211h11m46.66916618s) [noise=true , address=77.90.31.13:4626 , piece_id=DCKUBRB3DR755GN67RLP6NUMREAWIB3AITGPIMA7MM35BRPZW7IA, seg_pos={0 0}, upload=219598]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: storage node overloaded, request limit: 2000 Dec 14 10:39:45.850206751 (211h11m46.677915151s) [noise=true , address=77.90.31.16:4734 , piece_id=4JO2RBACNY5AM3KG7XUW7JTRGDCXGQDXSZ3L23ASAEJ4YFCICVAQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: storage node overloaded, request limit: 2000 Dec 14 10:39:45.859644247 (211h11m46.68735265s) [noise=true , address=145.79.227.30:4763 , piece_id=CLMX6DHN5HP3OWMVR4R5MIVRH3GHCELUVTYJHBE7HXZQMHRQESIQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: storage node overloaded, request limit: 2000 Dec 14 10:39:45.874109133 (211h11m46.701817525s) [upload=219598]: Next returned nil segment Dec 14 10:39:45.874123719 (211h11m46.701832113s) [upload=219598]: Waiting for error group managing segments... Dec 14 10:39:45.880971826 (211h11m46.708680217s) [noise=true , address=77.90.31.10:4518 , piece_id=ZE3SFGBCM2QV3CGGS3VEYT7NKDTDE2OJ6T6CVSGGYACCFX5WN5ZQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.882649818 (211h11m46.710358212s) [noise=true , address=77.90.31.10:4519 , piece_id=ACPHBL7LF6NCNNH4U5GUXKW7RCL6XIEQVWYDCYUTXEC5RQE7RZBA, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.883088160 (211h11m46.71079655s) [noise=true , address=77.90.31.14:4661 , piece_id=LLI4CDLWJ32TFXQ2YEU3NRP4QIRSNNUODA5RRPWU5AA45MJEW2LQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.883693953 (211h11m46.711402345s) [noise=true , address=145.79.227.39:4014 , piece_id=WKYZGQX5SP22ML66ZQZIHBS324EONJ7AARTYAXK2SCV577ZCQ5OQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.884112448 (211h11m46.711820839s) [noise=true , address=145.79.227.32:4836 , piece_id=IQRH4CRVBOG72M3BE3G5FTIEMBTLD2KFUEYU6NCTK3N6PUONMY5A, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.884619811 (211h11m46.712328205s) [noise=true , address=145.79.227.43:4159 , piece_id=ULRDDEYCZXA26UPSFZQRQWFPU7MUCGJJSM7YWIR7HPXDCPBOQXNA, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.885012115 (211h11m46.712720506s) [noise=true , address=145.79.227.47:4267 , piece_id=VY46BNDUYVGBXDRIOEFK4USNOOJH2KXZH5Z2NHCT6SDJC5SFUYMQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.887669218 (211h11m46.71537761s) [noise=true , address=145.79.227.50:4375 , piece_id=RKVBJHHIF3KJLKB6ULTA4E5SHOBPCLPMWJCA6YTKF5XDNVEZMYXQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.890935473 (211h11m46.718643851s) [noise=true , address=145.79.227.46:4603 , piece_id=3X2DPA75MR7RX6LCOO2U3UREG4QSYHRRKRE57JUYUBINXTJN5IEA, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.891096730 (211h11m46.718805121s) [noise=true , address=145.79.227.28:4727 , piece_id=42ILJYV7IJRXSV3E3RWXWNUNYQHHQFNOF2JZGZKDEGPXRXZ3H2YA, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.893061640 (211h11m46.720770031s) [noise=true , address=145.79.227.7:4293 , piece_id=56YULKIWDM4P4WZM6KGMPUK3B3AK3RRVF4H6RJ7H6IQISPLXGBCQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:48.683585502 (211h11m49.511293883s) [noise=true , address=145.79.227.44:4194 , piece_id=ZWOCXZ746BWDZ2LWYHSQHSG2O3WJC4R6FDL5E3JP2SXRCX3AWT2Q, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:40:15.740699026 (211h12m16.568407426s) [noise=true , address=77.90.31.16:4735 , piece_id=XKJJK6K7XF3TOFWMIU2BUVQTESTNV4FRYINSXJS4MC7KK7ZJU7SQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: expected piece hash; context deadline exceeded; EOF Dec 14 10:40:15.740740404 (211h12m16.568448788s) [seg_pos={0 0}, upload=219598]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 14 10:40:15.762950810 (211h12m16.590659203s) [upload=219598]: Done uploading segments... Dec 14 10:40:15.762987936 (211h12m16.59069633s) [upload=219598]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 411.73203ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................X................] duration: 376.532734ms trace: https://jaeger.dev.storj.io/trace/25fec5475908806a FAIL (2025-12-14 10:51:03 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 14 10:39:45.775689210 (211h11m46.603397602s) [upload=219598]: Starting upload Dec 14 10:39:45.775750055 (211h11m46.603458455s) [upload=219598]: Done starting upload Dec 14 10:39:45.775957558 (211h11m46.603665941s) [upload=219598]: Uploading segments... Dec 14 10:39:45.776074427 (211h11m46.603782809s) [upload=219598]: Got next segment. Inline: false Dec 14 10:39:45.776158162 (211h11m46.603866553s) [upload=219598]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 14 10:39:45.776165965 (211h11m46.603874362s) [upload=219598]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 14 10:39:45.801257283 (211h11m46.628965678s) [seg_pos={0 0}, upload=219598]: Begin upload segment... Dec 14 10:39:45.801373021 (211h11m46.629081422s) [noise=true , address=145.79.227.43:4159 , piece_id=ULRDDEYCZXA26UPSFZQRQWFPU7MUCGJJSM7YWIR7HPXDCPBOQXNA, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.801409865 (211h11m46.629118247s) [seg_pos={0 0}, upload=219598]: Done begin upload segment. Dec 14 10:39:45.801481233 (211h11m46.629189635s) [noise=true , address=145.79.227.30:4763 , piece_id=CLMX6DHN5HP3OWMVR4R5MIVRH3GHCELUVTYJHBE7HXZQMHRQESIQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.802050295 (211h11m46.629758695s) [noise=true , address=145.79.227.32:4836 , piece_id=IQRH4CRVBOG72M3BE3G5FTIEMBTLD2KFUEYU6NCTK3N6PUONMY5A, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.802573060 (211h11m46.630281451s) [noise=true , address=77.90.31.14:4661 , piece_id=LLI4CDLWJ32TFXQ2YEU3NRP4QIRSNNUODA5RRPWU5AA45MJEW2LQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.803112997 (211h11m46.630821425s) [noise=true , address=145.79.227.44:4194 , piece_id=ZWOCXZ746BWDZ2LWYHSQHSG2O3WJC4R6FDL5E3JP2SXRCX3AWT2Q, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.803148596 (211h11m46.630856991s) [noise=true , address=145.79.227.46:4603 , piece_id=3X2DPA75MR7RX6LCOO2U3UREG4QSYHRRKRE57JUYUBINXTJN5IEA, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.803615052 (211h11m46.631323455s) [noise=true , address=145.79.227.28:4727 , piece_id=42ILJYV7IJRXSV3E3RWXWNUNYQHHQFNOF2JZGZKDEGPXRXZ3H2YA, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.803722621 (211h11m46.631431034s) [noise=true , address=77.90.31.16:4735 , piece_id=XKJJK6K7XF3TOFWMIU2BUVQTESTNV4FRYINSXJS4MC7KK7ZJU7SQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.804120053 (211h11m46.631828444s) [noise=true , address=145.79.227.7:4293 , piece_id=56YULKIWDM4P4WZM6KGMPUK3B3AK3RRVF4H6RJ7H6IQISPLXGBCQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.804570407 (211h11m46.632278798s) [noise=true , address=145.79.227.47:4267 , piece_id=VY46BNDUYVGBXDRIOEFK4USNOOJH2KXZH5Z2NHCT6SDJC5SFUYMQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.805083798 (211h11m46.632792196s) [noise=true , address=145.79.227.50:4375 , piece_id=RKVBJHHIF3KJLKB6ULTA4E5SHOBPCLPMWJCA6YTKF5XDNVEZMYXQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.805547276 (211h11m46.633255677s) [noise=true , address=77.90.31.16:4734 , piece_id=4JO2RBACNY5AM3KG7XUW7JTRGDCXGQDXSZ3L23ASAEJ4YFCICVAQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.805916642 (211h11m46.633625023s) [noise=true , address=145.79.227.39:4014 , piece_id=WKYZGQX5SP22ML66ZQZIHBS324EONJ7AARTYAXK2SCV577ZCQ5OQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.806440130 (211h11m46.634148525s) [noise=true , address=77.90.31.13:4626 , piece_id=DCKUBRB3DR755GN67RLP6NUMREAWIB3AITGPIMA7MM35BRPZW7IA, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.806976882 (211h11m46.634685284s) [noise=true , address=77.90.31.10:4519 , piece_id=ACPHBL7LF6NCNNH4U5GUXKW7RCL6XIEQVWYDCYUTXEC5RQE7RZBA, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.807729329 (211h11m46.63543772s) [noise=true , address=77.90.31.10:4518 , piece_id=ZE3SFGBCM2QV3CGGS3VEYT7NKDTDE2OJ6T6CVSGGYACCFX5WN5ZQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.841457783 (211h11m46.66916618s) [noise=true , address=77.90.31.13:4626 , piece_id=DCKUBRB3DR755GN67RLP6NUMREAWIB3AITGPIMA7MM35BRPZW7IA, seg_pos={0 0}, upload=219598]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: storage node overloaded, request limit: 2000 Dec 14 10:39:45.850206751 (211h11m46.677915151s) [noise=true , address=77.90.31.16:4734 , piece_id=4JO2RBACNY5AM3KG7XUW7JTRGDCXGQDXSZ3L23ASAEJ4YFCICVAQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: storage node overloaded, request limit: 2000 Dec 14 10:39:45.859644247 (211h11m46.68735265s) [noise=true , address=145.79.227.30:4763 , piece_id=CLMX6DHN5HP3OWMVR4R5MIVRH3GHCELUVTYJHBE7HXZQMHRQESIQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: storage node overloaded, request limit: 2000 Dec 14 10:39:45.874109133 (211h11m46.701817525s) [upload=219598]: Next returned nil segment Dec 14 10:39:45.874123719 (211h11m46.701832113s) [upload=219598]: Waiting for error group managing segments... Dec 14 10:39:45.880971826 (211h11m46.708680217s) [noise=true , address=77.90.31.10:4518 , piece_id=ZE3SFGBCM2QV3CGGS3VEYT7NKDTDE2OJ6T6CVSGGYACCFX5WN5ZQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.882649818 (211h11m46.710358212s) [noise=true , address=77.90.31.10:4519 , piece_id=ACPHBL7LF6NCNNH4U5GUXKW7RCL6XIEQVWYDCYUTXEC5RQE7RZBA, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.883088160 (211h11m46.71079655s) [noise=true , address=77.90.31.14:4661 , piece_id=LLI4CDLWJ32TFXQ2YEU3NRP4QIRSNNUODA5RRPWU5AA45MJEW2LQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.883693953 (211h11m46.711402345s) [noise=true , address=145.79.227.39:4014 , piece_id=WKYZGQX5SP22ML66ZQZIHBS324EONJ7AARTYAXK2SCV577ZCQ5OQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.884112448 (211h11m46.711820839s) [noise=true , address=145.79.227.32:4836 , piece_id=IQRH4CRVBOG72M3BE3G5FTIEMBTLD2KFUEYU6NCTK3N6PUONMY5A, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.884619811 (211h11m46.712328205s) [noise=true , address=145.79.227.43:4159 , piece_id=ULRDDEYCZXA26UPSFZQRQWFPU7MUCGJJSM7YWIR7HPXDCPBOQXNA, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.885012115 (211h11m46.712720506s) [noise=true , address=145.79.227.47:4267 , piece_id=VY46BNDUYVGBXDRIOEFK4USNOOJH2KXZH5Z2NHCT6SDJC5SFUYMQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.887669218 (211h11m46.71537761s) [noise=true , address=145.79.227.50:4375 , piece_id=RKVBJHHIF3KJLKB6ULTA4E5SHOBPCLPMWJCA6YTKF5XDNVEZMYXQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.890935473 (211h11m46.718643851s) [noise=true , address=145.79.227.46:4603 , piece_id=3X2DPA75MR7RX6LCOO2U3UREG4QSYHRRKRE57JUYUBINXTJN5IEA, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.891096730 (211h11m46.718805121s) [noise=true , address=145.79.227.28:4727 , piece_id=42ILJYV7IJRXSV3E3RWXWNUNYQHHQFNOF2JZGZKDEGPXRXZ3H2YA, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.893061640 (211h11m46.720770031s) [noise=true , address=145.79.227.7:4293 , piece_id=56YULKIWDM4P4WZM6KGMPUK3B3AK3RRVF4H6RJ7H6IQISPLXGBCQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:48.683585502 (211h11m49.511293883s) [noise=true , address=145.79.227.44:4194 , piece_id=ZWOCXZ746BWDZ2LWYHSQHSG2O3WJC4R6FDL5E3JP2SXRCX3AWT2Q, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:40:15.740699026 (211h12m16.568407426s) [noise=true , address=77.90.31.16:4735 , piece_id=XKJJK6K7XF3TOFWMIU2BUVQTESTNV4FRYINSXJS4MC7KK7ZJU7SQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: expected piece hash; context deadline exceeded; EOF Dec 14 10:40:15.740740404 (211h12m16.568448788s) [seg_pos={0 0}, upload=219598]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 14 10:40:15.762950810 (211h12m16.590659203s) [upload=219598]: Done uploading segments... Dec 14 10:40:15.762987936 (211h12m16.59069633s) [upload=219598]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 376.56548ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............................................X...............] duration: 389.572904ms trace: https://jaeger.dev.storj.io/trace/25fec5475908806a FAIL (2025-12-14 10:50:47 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 14 10:39:45.775689210 (211h11m46.603397602s) [upload=219598]: Starting upload Dec 14 10:39:45.775750055 (211h11m46.603458455s) [upload=219598]: Done starting upload Dec 14 10:39:45.775957558 (211h11m46.603665941s) [upload=219598]: Uploading segments... Dec 14 10:39:45.776074427 (211h11m46.603782809s) [upload=219598]: Got next segment. Inline: false Dec 14 10:39:45.776158162 (211h11m46.603866553s) [upload=219598]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 14 10:39:45.776165965 (211h11m46.603874362s) [upload=219598]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 14 10:39:45.801257283 (211h11m46.628965678s) [seg_pos={0 0}, upload=219598]: Begin upload segment... Dec 14 10:39:45.801373021 (211h11m46.629081422s) [noise=true , address=145.79.227.43:4159 , piece_id=ULRDDEYCZXA26UPSFZQRQWFPU7MUCGJJSM7YWIR7HPXDCPBOQXNA, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.801409865 (211h11m46.629118247s) [seg_pos={0 0}, upload=219598]: Done begin upload segment. Dec 14 10:39:45.801481233 (211h11m46.629189635s) [noise=true , address=145.79.227.30:4763 , piece_id=CLMX6DHN5HP3OWMVR4R5MIVRH3GHCELUVTYJHBE7HXZQMHRQESIQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.802050295 (211h11m46.629758695s) [noise=true , address=145.79.227.32:4836 , piece_id=IQRH4CRVBOG72M3BE3G5FTIEMBTLD2KFUEYU6NCTK3N6PUONMY5A, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.802573060 (211h11m46.630281451s) [noise=true , address=77.90.31.14:4661 , piece_id=LLI4CDLWJ32TFXQ2YEU3NRP4QIRSNNUODA5RRPWU5AA45MJEW2LQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.803112997 (211h11m46.630821425s) [noise=true , address=145.79.227.44:4194 , piece_id=ZWOCXZ746BWDZ2LWYHSQHSG2O3WJC4R6FDL5E3JP2SXRCX3AWT2Q, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.803148596 (211h11m46.630856991s) [noise=true , address=145.79.227.46:4603 , piece_id=3X2DPA75MR7RX6LCOO2U3UREG4QSYHRRKRE57JUYUBINXTJN5IEA, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.803615052 (211h11m46.631323455s) [noise=true , address=145.79.227.28:4727 , piece_id=42ILJYV7IJRXSV3E3RWXWNUNYQHHQFNOF2JZGZKDEGPXRXZ3H2YA, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.803722621 (211h11m46.631431034s) [noise=true , address=77.90.31.16:4735 , piece_id=XKJJK6K7XF3TOFWMIU2BUVQTESTNV4FRYINSXJS4MC7KK7ZJU7SQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.804120053 (211h11m46.631828444s) [noise=true , address=145.79.227.7:4293 , piece_id=56YULKIWDM4P4WZM6KGMPUK3B3AK3RRVF4H6RJ7H6IQISPLXGBCQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.804570407 (211h11m46.632278798s) [noise=true , address=145.79.227.47:4267 , piece_id=VY46BNDUYVGBXDRIOEFK4USNOOJH2KXZH5Z2NHCT6SDJC5SFUYMQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.805083798 (211h11m46.632792196s) [noise=true , address=145.79.227.50:4375 , piece_id=RKVBJHHIF3KJLKB6ULTA4E5SHOBPCLPMWJCA6YTKF5XDNVEZMYXQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.805547276 (211h11m46.633255677s) [noise=true , address=77.90.31.16:4734 , piece_id=4JO2RBACNY5AM3KG7XUW7JTRGDCXGQDXSZ3L23ASAEJ4YFCICVAQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.805916642 (211h11m46.633625023s) [noise=true , address=145.79.227.39:4014 , piece_id=WKYZGQX5SP22ML66ZQZIHBS324EONJ7AARTYAXK2SCV577ZCQ5OQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.806440130 (211h11m46.634148525s) [noise=true , address=77.90.31.13:4626 , piece_id=DCKUBRB3DR755GN67RLP6NUMREAWIB3AITGPIMA7MM35BRPZW7IA, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.806976882 (211h11m46.634685284s) [noise=true , address=77.90.31.10:4519 , piece_id=ACPHBL7LF6NCNNH4U5GUXKW7RCL6XIEQVWYDCYUTXEC5RQE7RZBA, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.807729329 (211h11m46.63543772s) [noise=true , address=77.90.31.10:4518 , piece_id=ZE3SFGBCM2QV3CGGS3VEYT7NKDTDE2OJ6T6CVSGGYACCFX5WN5ZQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.841457783 (211h11m46.66916618s) [noise=true , address=77.90.31.13:4626 , piece_id=DCKUBRB3DR755GN67RLP6NUMREAWIB3AITGPIMA7MM35BRPZW7IA, seg_pos={0 0}, upload=219598]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: storage node overloaded, request limit: 2000 Dec 14 10:39:45.850206751 (211h11m46.677915151s) [noise=true , address=77.90.31.16:4734 , piece_id=4JO2RBACNY5AM3KG7XUW7JTRGDCXGQDXSZ3L23ASAEJ4YFCICVAQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: storage node overloaded, request limit: 2000 Dec 14 10:39:45.859644247 (211h11m46.68735265s) [noise=true , address=145.79.227.30:4763 , piece_id=CLMX6DHN5HP3OWMVR4R5MIVRH3GHCELUVTYJHBE7HXZQMHRQESIQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: storage node overloaded, request limit: 2000 Dec 14 10:39:45.874109133 (211h11m46.701817525s) [upload=219598]: Next returned nil segment Dec 14 10:39:45.874123719 (211h11m46.701832113s) [upload=219598]: Waiting for error group managing segments... Dec 14 10:39:45.880971826 (211h11m46.708680217s) [noise=true , address=77.90.31.10:4518 , piece_id=ZE3SFGBCM2QV3CGGS3VEYT7NKDTDE2OJ6T6CVSGGYACCFX5WN5ZQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.882649818 (211h11m46.710358212s) [noise=true , address=77.90.31.10:4519 , piece_id=ACPHBL7LF6NCNNH4U5GUXKW7RCL6XIEQVWYDCYUTXEC5RQE7RZBA, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.883088160 (211h11m46.71079655s) [noise=true , address=77.90.31.14:4661 , piece_id=LLI4CDLWJ32TFXQ2YEU3NRP4QIRSNNUODA5RRPWU5AA45MJEW2LQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.883693953 (211h11m46.711402345s) [noise=true , address=145.79.227.39:4014 , piece_id=WKYZGQX5SP22ML66ZQZIHBS324EONJ7AARTYAXK2SCV577ZCQ5OQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.884112448 (211h11m46.711820839s) [noise=true , address=145.79.227.32:4836 , piece_id=IQRH4CRVBOG72M3BE3G5FTIEMBTLD2KFUEYU6NCTK3N6PUONMY5A, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.884619811 (211h11m46.712328205s) [noise=true , address=145.79.227.43:4159 , piece_id=ULRDDEYCZXA26UPSFZQRQWFPU7MUCGJJSM7YWIR7HPXDCPBOQXNA, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.885012115 (211h11m46.712720506s) [noise=true , address=145.79.227.47:4267 , piece_id=VY46BNDUYVGBXDRIOEFK4USNOOJH2KXZH5Z2NHCT6SDJC5SFUYMQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.887669218 (211h11m46.71537761s) [noise=true , address=145.79.227.50:4375 , piece_id=RKVBJHHIF3KJLKB6ULTA4E5SHOBPCLPMWJCA6YTKF5XDNVEZMYXQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.890935473 (211h11m46.718643851s) [noise=true , address=145.79.227.46:4603 , piece_id=3X2DPA75MR7RX6LCOO2U3UREG4QSYHRRKRE57JUYUBINXTJN5IEA, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.891096730 (211h11m46.718805121s) [noise=true , address=145.79.227.28:4727 , piece_id=42ILJYV7IJRXSV3E3RWXWNUNYQHHQFNOF2JZGZKDEGPXRXZ3H2YA, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.893061640 (211h11m46.720770031s) [noise=true , address=145.79.227.7:4293 , piece_id=56YULKIWDM4P4WZM6KGMPUK3B3AK3RRVF4H6RJ7H6IQISPLXGBCQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:48.683585502 (211h11m49.511293883s) [noise=true , address=145.79.227.44:4194 , piece_id=ZWOCXZ746BWDZ2LWYHSQHSG2O3WJC4R6FDL5E3JP2SXRCX3AWT2Q, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:40:15.740699026 (211h12m16.568407426s) [noise=true , address=77.90.31.16:4735 , piece_id=XKJJK6K7XF3TOFWMIU2BUVQTESTNV4FRYINSXJS4MC7KK7ZJU7SQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: expected piece hash; context deadline exceeded; EOF Dec 14 10:40:15.740740404 (211h12m16.568448788s) [seg_pos={0 0}, upload=219598]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 14 10:40:15.762950810 (211h12m16.590659203s) [upload=219598]: Done uploading segments... Dec 14 10:40:15.762987936 (211h12m16.59069633s) [upload=219598]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 389.59989ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............................................X..............] duration: 558.34557ms trace: https://jaeger.dev.storj.io/trace/25fec5475908806a FAIL (2025-12-14 10:50:11 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 14 10:39:45.775689210 (211h11m46.603397602s) [upload=219598]: Starting upload Dec 14 10:39:45.775750055 (211h11m46.603458455s) [upload=219598]: Done starting upload Dec 14 10:39:45.775957558 (211h11m46.603665941s) [upload=219598]: Uploading segments... Dec 14 10:39:45.776074427 (211h11m46.603782809s) [upload=219598]: Got next segment. Inline: false Dec 14 10:39:45.776158162 (211h11m46.603866553s) [upload=219598]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 14 10:39:45.776165965 (211h11m46.603874362s) [upload=219598]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 14 10:39:45.801257283 (211h11m46.628965678s) [seg_pos={0 0}, upload=219598]: Begin upload segment... Dec 14 10:39:45.801373021 (211h11m46.629081422s) [noise=true , address=145.79.227.43:4159 , piece_id=ULRDDEYCZXA26UPSFZQRQWFPU7MUCGJJSM7YWIR7HPXDCPBOQXNA, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.801409865 (211h11m46.629118247s) [seg_pos={0 0}, upload=219598]: Done begin upload segment. Dec 14 10:39:45.801481233 (211h11m46.629189635s) [noise=true , address=145.79.227.30:4763 , piece_id=CLMX6DHN5HP3OWMVR4R5MIVRH3GHCELUVTYJHBE7HXZQMHRQESIQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.802050295 (211h11m46.629758695s) [noise=true , address=145.79.227.32:4836 , piece_id=IQRH4CRVBOG72M3BE3G5FTIEMBTLD2KFUEYU6NCTK3N6PUONMY5A, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.802573060 (211h11m46.630281451s) [noise=true , address=77.90.31.14:4661 , piece_id=LLI4CDLWJ32TFXQ2YEU3NRP4QIRSNNUODA5RRPWU5AA45MJEW2LQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.803112997 (211h11m46.630821425s) [noise=true , address=145.79.227.44:4194 , piece_id=ZWOCXZ746BWDZ2LWYHSQHSG2O3WJC4R6FDL5E3JP2SXRCX3AWT2Q, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.803148596 (211h11m46.630856991s) [noise=true , address=145.79.227.46:4603 , piece_id=3X2DPA75MR7RX6LCOO2U3UREG4QSYHRRKRE57JUYUBINXTJN5IEA, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.803615052 (211h11m46.631323455s) [noise=true , address=145.79.227.28:4727 , piece_id=42ILJYV7IJRXSV3E3RWXWNUNYQHHQFNOF2JZGZKDEGPXRXZ3H2YA, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.803722621 (211h11m46.631431034s) [noise=true , address=77.90.31.16:4735 , piece_id=XKJJK6K7XF3TOFWMIU2BUVQTESTNV4FRYINSXJS4MC7KK7ZJU7SQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.804120053 (211h11m46.631828444s) [noise=true , address=145.79.227.7:4293 , piece_id=56YULKIWDM4P4WZM6KGMPUK3B3AK3RRVF4H6RJ7H6IQISPLXGBCQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.804570407 (211h11m46.632278798s) [noise=true , address=145.79.227.47:4267 , piece_id=VY46BNDUYVGBXDRIOEFK4USNOOJH2KXZH5Z2NHCT6SDJC5SFUYMQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.805083798 (211h11m46.632792196s) [noise=true , address=145.79.227.50:4375 , piece_id=RKVBJHHIF3KJLKB6ULTA4E5SHOBPCLPMWJCA6YTKF5XDNVEZMYXQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.805547276 (211h11m46.633255677s) [noise=true , address=77.90.31.16:4734 , piece_id=4JO2RBACNY5AM3KG7XUW7JTRGDCXGQDXSZ3L23ASAEJ4YFCICVAQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.805916642 (211h11m46.633625023s) [noise=true , address=145.79.227.39:4014 , piece_id=WKYZGQX5SP22ML66ZQZIHBS324EONJ7AARTYAXK2SCV577ZCQ5OQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.806440130 (211h11m46.634148525s) [noise=true , address=77.90.31.13:4626 , piece_id=DCKUBRB3DR755GN67RLP6NUMREAWIB3AITGPIMA7MM35BRPZW7IA, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.806976882 (211h11m46.634685284s) [noise=true , address=77.90.31.10:4519 , piece_id=ACPHBL7LF6NCNNH4U5GUXKW7RCL6XIEQVWYDCYUTXEC5RQE7RZBA, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.807729329 (211h11m46.63543772s) [noise=true , address=77.90.31.10:4518 , piece_id=ZE3SFGBCM2QV3CGGS3VEYT7NKDTDE2OJ6T6CVSGGYACCFX5WN5ZQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.841457783 (211h11m46.66916618s) [noise=true , address=77.90.31.13:4626 , piece_id=DCKUBRB3DR755GN67RLP6NUMREAWIB3AITGPIMA7MM35BRPZW7IA, seg_pos={0 0}, upload=219598]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: storage node overloaded, request limit: 2000 Dec 14 10:39:45.850206751 (211h11m46.677915151s) [noise=true , address=77.90.31.16:4734 , piece_id=4JO2RBACNY5AM3KG7XUW7JTRGDCXGQDXSZ3L23ASAEJ4YFCICVAQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: storage node overloaded, request limit: 2000 Dec 14 10:39:45.859644247 (211h11m46.68735265s) [noise=true , address=145.79.227.30:4763 , piece_id=CLMX6DHN5HP3OWMVR4R5MIVRH3GHCELUVTYJHBE7HXZQMHRQESIQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: storage node overloaded, request limit: 2000 Dec 14 10:39:45.874109133 (211h11m46.701817525s) [upload=219598]: Next returned nil segment Dec 14 10:39:45.874123719 (211h11m46.701832113s) [upload=219598]: Waiting for error group managing segments... Dec 14 10:39:45.880971826 (211h11m46.708680217s) [noise=true , address=77.90.31.10:4518 , piece_id=ZE3SFGBCM2QV3CGGS3VEYT7NKDTDE2OJ6T6CVSGGYACCFX5WN5ZQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.882649818 (211h11m46.710358212s) [noise=true , address=77.90.31.10:4519 , piece_id=ACPHBL7LF6NCNNH4U5GUXKW7RCL6XIEQVWYDCYUTXEC5RQE7RZBA, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.883088160 (211h11m46.71079655s) [noise=true , address=77.90.31.14:4661 , piece_id=LLI4CDLWJ32TFXQ2YEU3NRP4QIRSNNUODA5RRPWU5AA45MJEW2LQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.883693953 (211h11m46.711402345s) [noise=true , address=145.79.227.39:4014 , piece_id=WKYZGQX5SP22ML66ZQZIHBS324EONJ7AARTYAXK2SCV577ZCQ5OQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.884112448 (211h11m46.711820839s) [noise=true , address=145.79.227.32:4836 , piece_id=IQRH4CRVBOG72M3BE3G5FTIEMBTLD2KFUEYU6NCTK3N6PUONMY5A, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.884619811 (211h11m46.712328205s) [noise=true , address=145.79.227.43:4159 , piece_id=ULRDDEYCZXA26UPSFZQRQWFPU7MUCGJJSM7YWIR7HPXDCPBOQXNA, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.885012115 (211h11m46.712720506s) [noise=true , address=145.79.227.47:4267 , piece_id=VY46BNDUYVGBXDRIOEFK4USNOOJH2KXZH5Z2NHCT6SDJC5SFUYMQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.887669218 (211h11m46.71537761s) [noise=true , address=145.79.227.50:4375 , piece_id=RKVBJHHIF3KJLKB6ULTA4E5SHOBPCLPMWJCA6YTKF5XDNVEZMYXQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.890935473 (211h11m46.718643851s) [noise=true , address=145.79.227.46:4603 , piece_id=3X2DPA75MR7RX6LCOO2U3UREG4QSYHRRKRE57JUYUBINXTJN5IEA, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.891096730 (211h11m46.718805121s) [noise=true , address=145.79.227.28:4727 , piece_id=42ILJYV7IJRXSV3E3RWXWNUNYQHHQFNOF2JZGZKDEGPXRXZ3H2YA, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.893061640 (211h11m46.720770031s) [noise=true , address=145.79.227.7:4293 , piece_id=56YULKIWDM4P4WZM6KGMPUK3B3AK3RRVF4H6RJ7H6IQISPLXGBCQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:48.683585502 (211h11m49.511293883s) [noise=true , address=145.79.227.44:4194 , piece_id=ZWOCXZ746BWDZ2LWYHSQHSG2O3WJC4R6FDL5E3JP2SXRCX3AWT2Q, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:40:15.740699026 (211h12m16.568407426s) [noise=true , address=77.90.31.16:4735 , piece_id=XKJJK6K7XF3TOFWMIU2BUVQTESTNV4FRYINSXJS4MC7KK7ZJU7SQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: expected piece hash; context deadline exceeded; EOF Dec 14 10:40:15.740740404 (211h12m16.568448788s) [seg_pos={0 0}, upload=219598]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 14 10:40:15.762950810 (211h12m16.590659203s) [upload=219598]: Done uploading segments... Dec 14 10:40:15.762987936 (211h12m16.59069633s) [upload=219598]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 558.430111ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............................................X.............] duration: 383.865807ms trace: https://jaeger.dev.storj.io/trace/25fec5475908806a FAIL (2025-12-14 10:49:26 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 14 10:39:45.775689210 (211h11m46.603397602s) [upload=219598]: Starting upload Dec 14 10:39:45.775750055 (211h11m46.603458455s) [upload=219598]: Done starting upload Dec 14 10:39:45.775957558 (211h11m46.603665941s) [upload=219598]: Uploading segments... Dec 14 10:39:45.776074427 (211h11m46.603782809s) [upload=219598]: Got next segment. Inline: false Dec 14 10:39:45.776158162 (211h11m46.603866553s) [upload=219598]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 14 10:39:45.776165965 (211h11m46.603874362s) [upload=219598]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 14 10:39:45.801257283 (211h11m46.628965678s) [seg_pos={0 0}, upload=219598]: Begin upload segment... Dec 14 10:39:45.801373021 (211h11m46.629081422s) [noise=true , address=145.79.227.43:4159 , piece_id=ULRDDEYCZXA26UPSFZQRQWFPU7MUCGJJSM7YWIR7HPXDCPBOQXNA, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.801409865 (211h11m46.629118247s) [seg_pos={0 0}, upload=219598]: Done begin upload segment. Dec 14 10:39:45.801481233 (211h11m46.629189635s) [noise=true , address=145.79.227.30:4763 , piece_id=CLMX6DHN5HP3OWMVR4R5MIVRH3GHCELUVTYJHBE7HXZQMHRQESIQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.802050295 (211h11m46.629758695s) [noise=true , address=145.79.227.32:4836 , piece_id=IQRH4CRVBOG72M3BE3G5FTIEMBTLD2KFUEYU6NCTK3N6PUONMY5A, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.802573060 (211h11m46.630281451s) [noise=true , address=77.90.31.14:4661 , piece_id=LLI4CDLWJ32TFXQ2YEU3NRP4QIRSNNUODA5RRPWU5AA45MJEW2LQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.803112997 (211h11m46.630821425s) [noise=true , address=145.79.227.44:4194 , piece_id=ZWOCXZ746BWDZ2LWYHSQHSG2O3WJC4R6FDL5E3JP2SXRCX3AWT2Q, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.803148596 (211h11m46.630856991s) [noise=true , address=145.79.227.46:4603 , piece_id=3X2DPA75MR7RX6LCOO2U3UREG4QSYHRRKRE57JUYUBINXTJN5IEA, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.803615052 (211h11m46.631323455s) [noise=true , address=145.79.227.28:4727 , piece_id=42ILJYV7IJRXSV3E3RWXWNUNYQHHQFNOF2JZGZKDEGPXRXZ3H2YA, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.803722621 (211h11m46.631431034s) [noise=true , address=77.90.31.16:4735 , piece_id=XKJJK6K7XF3TOFWMIU2BUVQTESTNV4FRYINSXJS4MC7KK7ZJU7SQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.804120053 (211h11m46.631828444s) [noise=true , address=145.79.227.7:4293 , piece_id=56YULKIWDM4P4WZM6KGMPUK3B3AK3RRVF4H6RJ7H6IQISPLXGBCQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.804570407 (211h11m46.632278798s) [noise=true , address=145.79.227.47:4267 , piece_id=VY46BNDUYVGBXDRIOEFK4USNOOJH2KXZH5Z2NHCT6SDJC5SFUYMQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.805083798 (211h11m46.632792196s) [noise=true , address=145.79.227.50:4375 , piece_id=RKVBJHHIF3KJLKB6ULTA4E5SHOBPCLPMWJCA6YTKF5XDNVEZMYXQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.805547276 (211h11m46.633255677s) [noise=true , address=77.90.31.16:4734 , piece_id=4JO2RBACNY5AM3KG7XUW7JTRGDCXGQDXSZ3L23ASAEJ4YFCICVAQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.805916642 (211h11m46.633625023s) [noise=true , address=145.79.227.39:4014 , piece_id=WKYZGQX5SP22ML66ZQZIHBS324EONJ7AARTYAXK2SCV577ZCQ5OQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.806440130 (211h11m46.634148525s) [noise=true , address=77.90.31.13:4626 , piece_id=DCKUBRB3DR755GN67RLP6NUMREAWIB3AITGPIMA7MM35BRPZW7IA, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.806976882 (211h11m46.634685284s) [noise=true , address=77.90.31.10:4519 , piece_id=ACPHBL7LF6NCNNH4U5GUXKW7RCL6XIEQVWYDCYUTXEC5RQE7RZBA, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.807729329 (211h11m46.63543772s) [noise=true , address=77.90.31.10:4518 , piece_id=ZE3SFGBCM2QV3CGGS3VEYT7NKDTDE2OJ6T6CVSGGYACCFX5WN5ZQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.841457783 (211h11m46.66916618s) [noise=true , address=77.90.31.13:4626 , piece_id=DCKUBRB3DR755GN67RLP6NUMREAWIB3AITGPIMA7MM35BRPZW7IA, seg_pos={0 0}, upload=219598]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: storage node overloaded, request limit: 2000 Dec 14 10:39:45.850206751 (211h11m46.677915151s) [noise=true , address=77.90.31.16:4734 , piece_id=4JO2RBACNY5AM3KG7XUW7JTRGDCXGQDXSZ3L23ASAEJ4YFCICVAQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: storage node overloaded, request limit: 2000 Dec 14 10:39:45.859644247 (211h11m46.68735265s) [noise=true , address=145.79.227.30:4763 , piece_id=CLMX6DHN5HP3OWMVR4R5MIVRH3GHCELUVTYJHBE7HXZQMHRQESIQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: storage node overloaded, request limit: 2000 Dec 14 10:39:45.874109133 (211h11m46.701817525s) [upload=219598]: Next returned nil segment Dec 14 10:39:45.874123719 (211h11m46.701832113s) [upload=219598]: Waiting for error group managing segments... Dec 14 10:39:45.880971826 (211h11m46.708680217s) [noise=true , address=77.90.31.10:4518 , piece_id=ZE3SFGBCM2QV3CGGS3VEYT7NKDTDE2OJ6T6CVSGGYACCFX5WN5ZQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.882649818 (211h11m46.710358212s) [noise=true , address=77.90.31.10:4519 , piece_id=ACPHBL7LF6NCNNH4U5GUXKW7RCL6XIEQVWYDCYUTXEC5RQE7RZBA, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.883088160 (211h11m46.71079655s) [noise=true , address=77.90.31.14:4661 , piece_id=LLI4CDLWJ32TFXQ2YEU3NRP4QIRSNNUODA5RRPWU5AA45MJEW2LQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.883693953 (211h11m46.711402345s) [noise=true , address=145.79.227.39:4014 , piece_id=WKYZGQX5SP22ML66ZQZIHBS324EONJ7AARTYAXK2SCV577ZCQ5OQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.884112448 (211h11m46.711820839s) [noise=true , address=145.79.227.32:4836 , piece_id=IQRH4CRVBOG72M3BE3G5FTIEMBTLD2KFUEYU6NCTK3N6PUONMY5A, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.884619811 (211h11m46.712328205s) [noise=true , address=145.79.227.43:4159 , piece_id=ULRDDEYCZXA26UPSFZQRQWFPU7MUCGJJSM7YWIR7HPXDCPBOQXNA, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.885012115 (211h11m46.712720506s) [noise=true , address=145.79.227.47:4267 , piece_id=VY46BNDUYVGBXDRIOEFK4USNOOJH2KXZH5Z2NHCT6SDJC5SFUYMQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.887669218 (211h11m46.71537761s) [noise=true , address=145.79.227.50:4375 , piece_id=RKVBJHHIF3KJLKB6ULTA4E5SHOBPCLPMWJCA6YTKF5XDNVEZMYXQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.890935473 (211h11m46.718643851s) [noise=true , address=145.79.227.46:4603 , piece_id=3X2DPA75MR7RX6LCOO2U3UREG4QSYHRRKRE57JUYUBINXTJN5IEA, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.891096730 (211h11m46.718805121s) [noise=true , address=145.79.227.28:4727 , piece_id=42ILJYV7IJRXSV3E3RWXWNUNYQHHQFNOF2JZGZKDEGPXRXZ3H2YA, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.893061640 (211h11m46.720770031s) [noise=true , address=145.79.227.7:4293 , piece_id=56YULKIWDM4P4WZM6KGMPUK3B3AK3RRVF4H6RJ7H6IQISPLXGBCQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:48.683585502 (211h11m49.511293883s) [noise=true , address=145.79.227.44:4194 , piece_id=ZWOCXZ746BWDZ2LWYHSQHSG2O3WJC4R6FDL5E3JP2SXRCX3AWT2Q, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:40:15.740699026 (211h12m16.568407426s) [noise=true , address=77.90.31.16:4735 , piece_id=XKJJK6K7XF3TOFWMIU2BUVQTESTNV4FRYINSXJS4MC7KK7ZJU7SQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: expected piece hash; context deadline exceeded; EOF Dec 14 10:40:15.740740404 (211h12m16.568448788s) [seg_pos={0 0}, upload=219598]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 14 10:40:15.762950810 (211h12m16.590659203s) [upload=219598]: Done uploading segments... Dec 14 10:40:15.762987936 (211h12m16.59069633s) [upload=219598]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 383.913884ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............................................X............] duration: 567.688558ms trace: https://jaeger.dev.storj.io/trace/25fec5475908806a FAIL (2025-12-14 10:48:37 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 14 10:39:45.775689210 (211h11m46.603397602s) [upload=219598]: Starting upload Dec 14 10:39:45.775750055 (211h11m46.603458455s) [upload=219598]: Done starting upload Dec 14 10:39:45.775957558 (211h11m46.603665941s) [upload=219598]: Uploading segments... Dec 14 10:39:45.776074427 (211h11m46.603782809s) [upload=219598]: Got next segment. Inline: false Dec 14 10:39:45.776158162 (211h11m46.603866553s) [upload=219598]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 14 10:39:45.776165965 (211h11m46.603874362s) [upload=219598]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 14 10:39:45.801257283 (211h11m46.628965678s) [seg_pos={0 0}, upload=219598]: Begin upload segment... Dec 14 10:39:45.801373021 (211h11m46.629081422s) [noise=true , address=145.79.227.43:4159 , piece_id=ULRDDEYCZXA26UPSFZQRQWFPU7MUCGJJSM7YWIR7HPXDCPBOQXNA, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.801409865 (211h11m46.629118247s) [seg_pos={0 0}, upload=219598]: Done begin upload segment. Dec 14 10:39:45.801481233 (211h11m46.629189635s) [noise=true , address=145.79.227.30:4763 , piece_id=CLMX6DHN5HP3OWMVR4R5MIVRH3GHCELUVTYJHBE7HXZQMHRQESIQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.802050295 (211h11m46.629758695s) [noise=true , address=145.79.227.32:4836 , piece_id=IQRH4CRVBOG72M3BE3G5FTIEMBTLD2KFUEYU6NCTK3N6PUONMY5A, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.802573060 (211h11m46.630281451s) [noise=true , address=77.90.31.14:4661 , piece_id=LLI4CDLWJ32TFXQ2YEU3NRP4QIRSNNUODA5RRPWU5AA45MJEW2LQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.803112997 (211h11m46.630821425s) [noise=true , address=145.79.227.44:4194 , piece_id=ZWOCXZ746BWDZ2LWYHSQHSG2O3WJC4R6FDL5E3JP2SXRCX3AWT2Q, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.803148596 (211h11m46.630856991s) [noise=true , address=145.79.227.46:4603 , piece_id=3X2DPA75MR7RX6LCOO2U3UREG4QSYHRRKRE57JUYUBINXTJN5IEA, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.803615052 (211h11m46.631323455s) [noise=true , address=145.79.227.28:4727 , piece_id=42ILJYV7IJRXSV3E3RWXWNUNYQHHQFNOF2JZGZKDEGPXRXZ3H2YA, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.803722621 (211h11m46.631431034s) [noise=true , address=77.90.31.16:4735 , piece_id=XKJJK6K7XF3TOFWMIU2BUVQTESTNV4FRYINSXJS4MC7KK7ZJU7SQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.804120053 (211h11m46.631828444s) [noise=true , address=145.79.227.7:4293 , piece_id=56YULKIWDM4P4WZM6KGMPUK3B3AK3RRVF4H6RJ7H6IQISPLXGBCQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.804570407 (211h11m46.632278798s) [noise=true , address=145.79.227.47:4267 , piece_id=VY46BNDUYVGBXDRIOEFK4USNOOJH2KXZH5Z2NHCT6SDJC5SFUYMQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.805083798 (211h11m46.632792196s) [noise=true , address=145.79.227.50:4375 , piece_id=RKVBJHHIF3KJLKB6ULTA4E5SHOBPCLPMWJCA6YTKF5XDNVEZMYXQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.805547276 (211h11m46.633255677s) [noise=true , address=77.90.31.16:4734 , piece_id=4JO2RBACNY5AM3KG7XUW7JTRGDCXGQDXSZ3L23ASAEJ4YFCICVAQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.805916642 (211h11m46.633625023s) [noise=true , address=145.79.227.39:4014 , piece_id=WKYZGQX5SP22ML66ZQZIHBS324EONJ7AARTYAXK2SCV577ZCQ5OQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.806440130 (211h11m46.634148525s) [noise=true , address=77.90.31.13:4626 , piece_id=DCKUBRB3DR755GN67RLP6NUMREAWIB3AITGPIMA7MM35BRPZW7IA, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.806976882 (211h11m46.634685284s) [noise=true , address=77.90.31.10:4519 , piece_id=ACPHBL7LF6NCNNH4U5GUXKW7RCL6XIEQVWYDCYUTXEC5RQE7RZBA, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.807729329 (211h11m46.63543772s) [noise=true , address=77.90.31.10:4518 , piece_id=ZE3SFGBCM2QV3CGGS3VEYT7NKDTDE2OJ6T6CVSGGYACCFX5WN5ZQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.841457783 (211h11m46.66916618s) [noise=true , address=77.90.31.13:4626 , piece_id=DCKUBRB3DR755GN67RLP6NUMREAWIB3AITGPIMA7MM35BRPZW7IA, seg_pos={0 0}, upload=219598]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: storage node overloaded, request limit: 2000 Dec 14 10:39:45.850206751 (211h11m46.677915151s) [noise=true , address=77.90.31.16:4734 , piece_id=4JO2RBACNY5AM3KG7XUW7JTRGDCXGQDXSZ3L23ASAEJ4YFCICVAQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: storage node overloaded, request limit: 2000 Dec 14 10:39:45.859644247 (211h11m46.68735265s) [noise=true , address=145.79.227.30:4763 , piece_id=CLMX6DHN5HP3OWMVR4R5MIVRH3GHCELUVTYJHBE7HXZQMHRQESIQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: storage node overloaded, request limit: 2000 Dec 14 10:39:45.874109133 (211h11m46.701817525s) [upload=219598]: Next returned nil segment Dec 14 10:39:45.874123719 (211h11m46.701832113s) [upload=219598]: Waiting for error group managing segments... Dec 14 10:39:45.880971826 (211h11m46.708680217s) [noise=true , address=77.90.31.10:4518 , piece_id=ZE3SFGBCM2QV3CGGS3VEYT7NKDTDE2OJ6T6CVSGGYACCFX5WN5ZQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.882649818 (211h11m46.710358212s) [noise=true , address=77.90.31.10:4519 , piece_id=ACPHBL7LF6NCNNH4U5GUXKW7RCL6XIEQVWYDCYUTXEC5RQE7RZBA, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.883088160 (211h11m46.71079655s) [noise=true , address=77.90.31.14:4661 , piece_id=LLI4CDLWJ32TFXQ2YEU3NRP4QIRSNNUODA5RRPWU5AA45MJEW2LQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.883693953 (211h11m46.711402345s) [noise=true , address=145.79.227.39:4014 , piece_id=WKYZGQX5SP22ML66ZQZIHBS324EONJ7AARTYAXK2SCV577ZCQ5OQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.884112448 (211h11m46.711820839s) [noise=true , address=145.79.227.32:4836 , piece_id=IQRH4CRVBOG72M3BE3G5FTIEMBTLD2KFUEYU6NCTK3N6PUONMY5A, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.884619811 (211h11m46.712328205s) [noise=true , address=145.79.227.43:4159 , piece_id=ULRDDEYCZXA26UPSFZQRQWFPU7MUCGJJSM7YWIR7HPXDCPBOQXNA, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.885012115 (211h11m46.712720506s) [noise=true , address=145.79.227.47:4267 , piece_id=VY46BNDUYVGBXDRIOEFK4USNOOJH2KXZH5Z2NHCT6SDJC5SFUYMQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.887669218 (211h11m46.71537761s) [noise=true , address=145.79.227.50:4375 , piece_id=RKVBJHHIF3KJLKB6ULTA4E5SHOBPCLPMWJCA6YTKF5XDNVEZMYXQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.890935473 (211h11m46.718643851s) [noise=true , address=145.79.227.46:4603 , piece_id=3X2DPA75MR7RX6LCOO2U3UREG4QSYHRRKRE57JUYUBINXTJN5IEA, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.891096730 (211h11m46.718805121s) [noise=true , address=145.79.227.28:4727 , piece_id=42ILJYV7IJRXSV3E3RWXWNUNYQHHQFNOF2JZGZKDEGPXRXZ3H2YA, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.893061640 (211h11m46.720770031s) [noise=true , address=145.79.227.7:4293 , piece_id=56YULKIWDM4P4WZM6KGMPUK3B3AK3RRVF4H6RJ7H6IQISPLXGBCQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:48.683585502 (211h11m49.511293883s) [noise=true , address=145.79.227.44:4194 , piece_id=ZWOCXZ746BWDZ2LWYHSQHSG2O3WJC4R6FDL5E3JP2SXRCX3AWT2Q, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:40:15.740699026 (211h12m16.568407426s) [noise=true , address=77.90.31.16:4735 , piece_id=XKJJK6K7XF3TOFWMIU2BUVQTESTNV4FRYINSXJS4MC7KK7ZJU7SQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: expected piece hash; context deadline exceeded; EOF Dec 14 10:40:15.740740404 (211h12m16.568448788s) [seg_pos={0 0}, upload=219598]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 14 10:40:15.762950810 (211h12m16.590659203s) [upload=219598]: Done uploading segments... Dec 14 10:40:15.762987936 (211h12m16.59069633s) [upload=219598]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 567.719114ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................................................X...........] duration: 427.315133ms trace: https://jaeger.dev.storj.io/trace/25fec5475908806a FAIL (2025-12-14 10:48:16 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 14 10:39:45.775689210 (211h11m46.603397602s) [upload=219598]: Starting upload Dec 14 10:39:45.775750055 (211h11m46.603458455s) [upload=219598]: Done starting upload Dec 14 10:39:45.775957558 (211h11m46.603665941s) [upload=219598]: Uploading segments... Dec 14 10:39:45.776074427 (211h11m46.603782809s) [upload=219598]: Got next segment. Inline: false Dec 14 10:39:45.776158162 (211h11m46.603866553s) [upload=219598]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 14 10:39:45.776165965 (211h11m46.603874362s) [upload=219598]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 14 10:39:45.801257283 (211h11m46.628965678s) [seg_pos={0 0}, upload=219598]: Begin upload segment... Dec 14 10:39:45.801373021 (211h11m46.629081422s) [noise=true , address=145.79.227.43:4159 , piece_id=ULRDDEYCZXA26UPSFZQRQWFPU7MUCGJJSM7YWIR7HPXDCPBOQXNA, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.801409865 (211h11m46.629118247s) [seg_pos={0 0}, upload=219598]: Done begin upload segment. Dec 14 10:39:45.801481233 (211h11m46.629189635s) [noise=true , address=145.79.227.30:4763 , piece_id=CLMX6DHN5HP3OWMVR4R5MIVRH3GHCELUVTYJHBE7HXZQMHRQESIQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.802050295 (211h11m46.629758695s) [noise=true , address=145.79.227.32:4836 , piece_id=IQRH4CRVBOG72M3BE3G5FTIEMBTLD2KFUEYU6NCTK3N6PUONMY5A, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.802573060 (211h11m46.630281451s) [noise=true , address=77.90.31.14:4661 , piece_id=LLI4CDLWJ32TFXQ2YEU3NRP4QIRSNNUODA5RRPWU5AA45MJEW2LQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.803112997 (211h11m46.630821425s) [noise=true , address=145.79.227.44:4194 , piece_id=ZWOCXZ746BWDZ2LWYHSQHSG2O3WJC4R6FDL5E3JP2SXRCX3AWT2Q, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.803148596 (211h11m46.630856991s) [noise=true , address=145.79.227.46:4603 , piece_id=3X2DPA75MR7RX6LCOO2U3UREG4QSYHRRKRE57JUYUBINXTJN5IEA, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.803615052 (211h11m46.631323455s) [noise=true , address=145.79.227.28:4727 , piece_id=42ILJYV7IJRXSV3E3RWXWNUNYQHHQFNOF2JZGZKDEGPXRXZ3H2YA, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.803722621 (211h11m46.631431034s) [noise=true , address=77.90.31.16:4735 , piece_id=XKJJK6K7XF3TOFWMIU2BUVQTESTNV4FRYINSXJS4MC7KK7ZJU7SQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.804120053 (211h11m46.631828444s) [noise=true , address=145.79.227.7:4293 , piece_id=56YULKIWDM4P4WZM6KGMPUK3B3AK3RRVF4H6RJ7H6IQISPLXGBCQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.804570407 (211h11m46.632278798s) [noise=true , address=145.79.227.47:4267 , piece_id=VY46BNDUYVGBXDRIOEFK4USNOOJH2KXZH5Z2NHCT6SDJC5SFUYMQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.805083798 (211h11m46.632792196s) [noise=true , address=145.79.227.50:4375 , piece_id=RKVBJHHIF3KJLKB6ULTA4E5SHOBPCLPMWJCA6YTKF5XDNVEZMYXQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.805547276 (211h11m46.633255677s) [noise=true , address=77.90.31.16:4734 , piece_id=4JO2RBACNY5AM3KG7XUW7JTRGDCXGQDXSZ3L23ASAEJ4YFCICVAQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.805916642 (211h11m46.633625023s) [noise=true , address=145.79.227.39:4014 , piece_id=WKYZGQX5SP22ML66ZQZIHBS324EONJ7AARTYAXK2SCV577ZCQ5OQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.806440130 (211h11m46.634148525s) [noise=true , address=77.90.31.13:4626 , piece_id=DCKUBRB3DR755GN67RLP6NUMREAWIB3AITGPIMA7MM35BRPZW7IA, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.806976882 (211h11m46.634685284s) [noise=true , address=77.90.31.10:4519 , piece_id=ACPHBL7LF6NCNNH4U5GUXKW7RCL6XIEQVWYDCYUTXEC5RQE7RZBA, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.807729329 (211h11m46.63543772s) [noise=true , address=77.90.31.10:4518 , piece_id=ZE3SFGBCM2QV3CGGS3VEYT7NKDTDE2OJ6T6CVSGGYACCFX5WN5ZQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.841457783 (211h11m46.66916618s) [noise=true , address=77.90.31.13:4626 , piece_id=DCKUBRB3DR755GN67RLP6NUMREAWIB3AITGPIMA7MM35BRPZW7IA, seg_pos={0 0}, upload=219598]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: storage node overloaded, request limit: 2000 Dec 14 10:39:45.850206751 (211h11m46.677915151s) [noise=true , address=77.90.31.16:4734 , piece_id=4JO2RBACNY5AM3KG7XUW7JTRGDCXGQDXSZ3L23ASAEJ4YFCICVAQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: storage node overloaded, request limit: 2000 Dec 14 10:39:45.859644247 (211h11m46.68735265s) [noise=true , address=145.79.227.30:4763 , piece_id=CLMX6DHN5HP3OWMVR4R5MIVRH3GHCELUVTYJHBE7HXZQMHRQESIQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: storage node overloaded, request limit: 2000 Dec 14 10:39:45.874109133 (211h11m46.701817525s) [upload=219598]: Next returned nil segment Dec 14 10:39:45.874123719 (211h11m46.701832113s) [upload=219598]: Waiting for error group managing segments... Dec 14 10:39:45.880971826 (211h11m46.708680217s) [noise=true , address=77.90.31.10:4518 , piece_id=ZE3SFGBCM2QV3CGGS3VEYT7NKDTDE2OJ6T6CVSGGYACCFX5WN5ZQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.882649818 (211h11m46.710358212s) [noise=true , address=77.90.31.10:4519 , piece_id=ACPHBL7LF6NCNNH4U5GUXKW7RCL6XIEQVWYDCYUTXEC5RQE7RZBA, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.883088160 (211h11m46.71079655s) [noise=true , address=77.90.31.14:4661 , piece_id=LLI4CDLWJ32TFXQ2YEU3NRP4QIRSNNUODA5RRPWU5AA45MJEW2LQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.883693953 (211h11m46.711402345s) [noise=true , address=145.79.227.39:4014 , piece_id=WKYZGQX5SP22ML66ZQZIHBS324EONJ7AARTYAXK2SCV577ZCQ5OQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.884112448 (211h11m46.711820839s) [noise=true , address=145.79.227.32:4836 , piece_id=IQRH4CRVBOG72M3BE3G5FTIEMBTLD2KFUEYU6NCTK3N6PUONMY5A, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.884619811 (211h11m46.712328205s) [noise=true , address=145.79.227.43:4159 , piece_id=ULRDDEYCZXA26UPSFZQRQWFPU7MUCGJJSM7YWIR7HPXDCPBOQXNA, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.885012115 (211h11m46.712720506s) [noise=true , address=145.79.227.47:4267 , piece_id=VY46BNDUYVGBXDRIOEFK4USNOOJH2KXZH5Z2NHCT6SDJC5SFUYMQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.887669218 (211h11m46.71537761s) [noise=true , address=145.79.227.50:4375 , piece_id=RKVBJHHIF3KJLKB6ULTA4E5SHOBPCLPMWJCA6YTKF5XDNVEZMYXQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.890935473 (211h11m46.718643851s) [noise=true , address=145.79.227.46:4603 , piece_id=3X2DPA75MR7RX6LCOO2U3UREG4QSYHRRKRE57JUYUBINXTJN5IEA, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.891096730 (211h11m46.718805121s) [noise=true , address=145.79.227.28:4727 , piece_id=42ILJYV7IJRXSV3E3RWXWNUNYQHHQFNOF2JZGZKDEGPXRXZ3H2YA, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.893061640 (211h11m46.720770031s) [noise=true , address=145.79.227.7:4293 , piece_id=56YULKIWDM4P4WZM6KGMPUK3B3AK3RRVF4H6RJ7H6IQISPLXGBCQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:48.683585502 (211h11m49.511293883s) [noise=true , address=145.79.227.44:4194 , piece_id=ZWOCXZ746BWDZ2LWYHSQHSG2O3WJC4R6FDL5E3JP2SXRCX3AWT2Q, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:40:15.740699026 (211h12m16.568407426s) [noise=true , address=77.90.31.16:4735 , piece_id=XKJJK6K7XF3TOFWMIU2BUVQTESTNV4FRYINSXJS4MC7KK7ZJU7SQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: expected piece hash; context deadline exceeded; EOF Dec 14 10:40:15.740740404 (211h12m16.568448788s) [seg_pos={0 0}, upload=219598]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 14 10:40:15.762950810 (211h12m16.590659203s) [upload=219598]: Done uploading segments... Dec 14 10:40:15.762987936 (211h12m16.59069633s) [upload=219598]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 427.358331ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................................................X..........] duration: 386.041152ms trace: https://jaeger.dev.storj.io/trace/25fec5475908806a FAIL (2025-12-14 10:47:49 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 14 10:39:45.775689210 (211h11m46.603397602s) [upload=219598]: Starting upload Dec 14 10:39:45.775750055 (211h11m46.603458455s) [upload=219598]: Done starting upload Dec 14 10:39:45.775957558 (211h11m46.603665941s) [upload=219598]: Uploading segments... Dec 14 10:39:45.776074427 (211h11m46.603782809s) [upload=219598]: Got next segment. Inline: false Dec 14 10:39:45.776158162 (211h11m46.603866553s) [upload=219598]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 14 10:39:45.776165965 (211h11m46.603874362s) [upload=219598]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 14 10:39:45.801257283 (211h11m46.628965678s) [seg_pos={0 0}, upload=219598]: Begin upload segment... Dec 14 10:39:45.801373021 (211h11m46.629081422s) [noise=true , address=145.79.227.43:4159 , piece_id=ULRDDEYCZXA26UPSFZQRQWFPU7MUCGJJSM7YWIR7HPXDCPBOQXNA, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.801409865 (211h11m46.629118247s) [seg_pos={0 0}, upload=219598]: Done begin upload segment. Dec 14 10:39:45.801481233 (211h11m46.629189635s) [noise=true , address=145.79.227.30:4763 , piece_id=CLMX6DHN5HP3OWMVR4R5MIVRH3GHCELUVTYJHBE7HXZQMHRQESIQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.802050295 (211h11m46.629758695s) [noise=true , address=145.79.227.32:4836 , piece_id=IQRH4CRVBOG72M3BE3G5FTIEMBTLD2KFUEYU6NCTK3N6PUONMY5A, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.802573060 (211h11m46.630281451s) [noise=true , address=77.90.31.14:4661 , piece_id=LLI4CDLWJ32TFXQ2YEU3NRP4QIRSNNUODA5RRPWU5AA45MJEW2LQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.803112997 (211h11m46.630821425s) [noise=true , address=145.79.227.44:4194 , piece_id=ZWOCXZ746BWDZ2LWYHSQHSG2O3WJC4R6FDL5E3JP2SXRCX3AWT2Q, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.803148596 (211h11m46.630856991s) [noise=true , address=145.79.227.46:4603 , piece_id=3X2DPA75MR7RX6LCOO2U3UREG4QSYHRRKRE57JUYUBINXTJN5IEA, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.803615052 (211h11m46.631323455s) [noise=true , address=145.79.227.28:4727 , piece_id=42ILJYV7IJRXSV3E3RWXWNUNYQHHQFNOF2JZGZKDEGPXRXZ3H2YA, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.803722621 (211h11m46.631431034s) [noise=true , address=77.90.31.16:4735 , piece_id=XKJJK6K7XF3TOFWMIU2BUVQTESTNV4FRYINSXJS4MC7KK7ZJU7SQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.804120053 (211h11m46.631828444s) [noise=true , address=145.79.227.7:4293 , piece_id=56YULKIWDM4P4WZM6KGMPUK3B3AK3RRVF4H6RJ7H6IQISPLXGBCQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.804570407 (211h11m46.632278798s) [noise=true , address=145.79.227.47:4267 , piece_id=VY46BNDUYVGBXDRIOEFK4USNOOJH2KXZH5Z2NHCT6SDJC5SFUYMQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.805083798 (211h11m46.632792196s) [noise=true , address=145.79.227.50:4375 , piece_id=RKVBJHHIF3KJLKB6ULTA4E5SHOBPCLPMWJCA6YTKF5XDNVEZMYXQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.805547276 (211h11m46.633255677s) [noise=true , address=77.90.31.16:4734 , piece_id=4JO2RBACNY5AM3KG7XUW7JTRGDCXGQDXSZ3L23ASAEJ4YFCICVAQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.805916642 (211h11m46.633625023s) [noise=true , address=145.79.227.39:4014 , piece_id=WKYZGQX5SP22ML66ZQZIHBS324EONJ7AARTYAXK2SCV577ZCQ5OQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.806440130 (211h11m46.634148525s) [noise=true , address=77.90.31.13:4626 , piece_id=DCKUBRB3DR755GN67RLP6NUMREAWIB3AITGPIMA7MM35BRPZW7IA, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.806976882 (211h11m46.634685284s) [noise=true , address=77.90.31.10:4519 , piece_id=ACPHBL7LF6NCNNH4U5GUXKW7RCL6XIEQVWYDCYUTXEC5RQE7RZBA, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.807729329 (211h11m46.63543772s) [noise=true , address=77.90.31.10:4518 , piece_id=ZE3SFGBCM2QV3CGGS3VEYT7NKDTDE2OJ6T6CVSGGYACCFX5WN5ZQ, seg_pos={0 0}, upload=219598]: Uploading piece... Dec 14 10:39:45.841457783 (211h11m46.66916618s) [noise=true , address=77.90.31.13:4626 , piece_id=DCKUBRB3DR755GN67RLP6NUMREAWIB3AITGPIMA7MM35BRPZW7IA, seg_pos={0 0}, upload=219598]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: storage node overloaded, request limit: 2000 Dec 14 10:39:45.850206751 (211h11m46.677915151s) [noise=true , address=77.90.31.16:4734 , piece_id=4JO2RBACNY5AM3KG7XUW7JTRGDCXGQDXSZ3L23ASAEJ4YFCICVAQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: storage node overloaded, request limit: 2000 Dec 14 10:39:45.859644247 (211h11m46.68735265s) [noise=true , address=145.79.227.30:4763 , piece_id=CLMX6DHN5HP3OWMVR4R5MIVRH3GHCELUVTYJHBE7HXZQMHRQESIQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: storage node overloaded, request limit: 2000 Dec 14 10:39:45.874109133 (211h11m46.701817525s) [upload=219598]: Next returned nil segment Dec 14 10:39:45.874123719 (211h11m46.701832113s) [upload=219598]: Waiting for error group managing segments... Dec 14 10:39:45.880971826 (211h11m46.708680217s) [noise=true , address=77.90.31.10:4518 , piece_id=ZE3SFGBCM2QV3CGGS3VEYT7NKDTDE2OJ6T6CVSGGYACCFX5WN5ZQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.882649818 (211h11m46.710358212s) [noise=true , address=77.90.31.10:4519 , piece_id=ACPHBL7LF6NCNNH4U5GUXKW7RCL6XIEQVWYDCYUTXEC5RQE7RZBA, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.883088160 (211h11m46.71079655s) [noise=true , address=77.90.31.14:4661 , piece_id=LLI4CDLWJ32TFXQ2YEU3NRP4QIRSNNUODA5RRPWU5AA45MJEW2LQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.883693953 (211h11m46.711402345s) [noise=true , address=145.79.227.39:4014 , piece_id=WKYZGQX5SP22ML66ZQZIHBS324EONJ7AARTYAXK2SCV577ZCQ5OQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.884112448 (211h11m46.711820839s) [noise=true , address=145.79.227.32:4836 , piece_id=IQRH4CRVBOG72M3BE3G5FTIEMBTLD2KFUEYU6NCTK3N6PUONMY5A, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.884619811 (211h11m46.712328205s) [noise=true , address=145.79.227.43:4159 , piece_id=ULRDDEYCZXA26UPSFZQRQWFPU7MUCGJJSM7YWIR7HPXDCPBOQXNA, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.885012115 (211h11m46.712720506s) [noise=true , address=145.79.227.47:4267 , piece_id=VY46BNDUYVGBXDRIOEFK4USNOOJH2KXZH5Z2NHCT6SDJC5SFUYMQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.887669218 (211h11m46.71537761s) [noise=true , address=145.79.227.50:4375 , piece_id=RKVBJHHIF3KJLKB6ULTA4E5SHOBPCLPMWJCA6YTKF5XDNVEZMYXQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.890935473 (211h11m46.718643851s) [noise=true , address=145.79.227.46:4603 , piece_id=3X2DPA75MR7RX6LCOO2U3UREG4QSYHRRKRE57JUYUBINXTJN5IEA, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.891096730 (211h11m46.718805121s) [noise=true , address=145.79.227.28:4727 , piece_id=42ILJYV7IJRXSV3E3RWXWNUNYQHHQFNOF2JZGZKDEGPXRXZ3H2YA, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:45.893061640 (211h11m46.720770031s) [noise=true , address=145.79.227.7:4293 , piece_id=56YULKIWDM4P4WZM6KGMPUK3B3AK3RRVF4H6RJ7H6IQISPLXGBCQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:39:48.683585502 (211h11m49.511293883s) [noise=true , address=145.79.227.44:4194 , piece_id=ZWOCXZ746BWDZ2LWYHSQHSG2O3WJC4R6FDL5E3JP2SXRCX3AWT2Q, seg_pos={0 0}, upload=219598]: Done uploading piece. err: <nil> Dec 14 10:40:15.740699026 (211h12m16.568407426s) [noise=true , address=77.90.31.16:4735 , piece_id=XKJJK6K7XF3TOFWMIU2BUVQTESTNV4FRYINSXJS4MC7KK7ZJU7SQ, seg_pos={0 0}, upload=219598]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: expected piece hash; context deadline exceeded; EOF Dec 14 10:40:15.740740404 (211h12m16.568448788s) [seg_pos={0 0}, upload=219598]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 14 10:40:15.762950810 (211h12m16.590659203s) [upload=219598]: Done uploading segments... Dec 14 10:40:15.762987936 (211h12m16.59069633s) [upload=219598]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 386.070733ms core uplink us-select-1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)